Inquiry Regarding Remote Control and Screen Sharing Capabilities Under MFi Program

Hello @iconcreator,

You can use a ReplayKit Broadcast Upload Extension to receive a stream of the screen (assuming the user has selected your extension from the broadcast picker in Control Center).

The rest of your questions are well outside of my domain, so I will let someone else address them :).

-- Greg

I am exploring technical possibilities for screen sharing and remote interaction between iOS devices and external hardware (e.g., embedded systems, in-vehicle systems) for a prototype we are currently developing.

So, my first question here is "what are you actually trying to do?". You're asking a series of specific questions around exactly what's possible, but the problem is that most of those questions don't have simple yes/no answers, but instead involve complicated qualifications and edge cases. That both makes the question very difficult to answer accurately and also means that we may not tell you about alternative approaches that would solve your real issue.

If you’re concerned about discussing those details in public, then my suggestion would be that you file a code-level support issue using this form.

Does the MFi Program allow: Use of private frameworks or entitlements to build low-level integrations for iOS device control or mirroring?

So, the first thing to understand here is that the MFi program is a hardware licensing program. That is, the "point" of the program is to work with developers to create hardware products that work with our devices. The software side of this is largely public, through APIs like the External Accessory framework or our CarPlay APIs. There are a few more obscure APIs like pairing through HMAccessorySetupPayload, but all of them really are about addressing specific edge cases, not providing broad powers/capabilities.

Note that the FAQ section of the MFi portal includes a specific list of the technologies and components that the MFi program licenses. If whatever you're trying to do isn't on that list, then it's very likely that the MFi program will not be useful to you.

Access to IOHIDEventSystem or similar private APIs for input injection?

No. That is, iOS does not provide any mechanism that would allow an app to "inject" events into the system.

Is there any officially supported mechanism, under MFi or otherwise, that allows external systems to remotely control an iOS device’s touch interface (e.g., simulate gestures, taps, swipes)?

Well... yes. iOS supports HID, so a USB or Bluetooth HID accessory can control the device’s interface by sending HID commands to the device.

Communication over USB or network to relay control commands that simulate direct user interaction?

The system doesn't have any special/specific support for anything like this; however, it's certainly possible to create a HID accessory that "relays" events from some external source into the system (instead of directly receiving them directly from user action).

What are the criteria to be approved for the MFi program with access to such advanced capabilities?

This is covered by "Who Needs to Join the MFi Program" on the MFi portal.

Can PoC or early-stage research prototypes be eligible, or is MFi access restricted to commercial production intent?

This is possible; however, it's also possible for a request to join the program to be rejected. This typically happens because it's obvious from the initial request that the product being described isn't something the MFi licensed specifications can create.

Case in point here, nothing you've described would require an MFi accessory. The simplest way to inject HID events into the system is to create a standard HID accessory that sends events.

We are not looking to publish this as a general App Store app at this stage, but rather to demonstrate feasibility as part of an innovation prototype that may lead to further OEM-level engagement in the future.

For prototyping purposes, the simplest way to do this would be to use a computer to simulate a Bluetooth* HID device, which you'd then pair with the iOS device. If you search the Mac App Store for “Bluetooth keyboard", you'll find multiple apps for macOS that do exactly what I'm describing.

*Bluetooth simplifies this on macOS (and other "computers") because the controller architecture of USB makes it difficult for a "standard" computer to simulate a USB accessory. You need to use something like a Raspberry Pi configured to run in peripheral mode to simulate a USB accessory.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware