Is it possible to feed ReplayKit with custom live stream data, e.g. a cvPixelBuffer created from a Metal Texture, and stream that to youtube? My use case is to give the broadcaster hidden UI manipulation controls that the stream audience cannot see. (Think of a DJ. No one gets to see all the DAW controls on the DJ's laptop and doesn't need to because that's not part of the experience.) If it's possible, might anyone be able to help figure out the correct way to implement this? From what I can tell, ReplayKit doesn't let you send custom data, in which case, what else can be done?

Is there a relationship between AVAudio’s time here: docs link and the system uptime? docs link I’d like to be able to convert between them, but I’m not sure how they're related, if at all. Specifically, I'd like the hostTime in terms of systemUptime because several other APIs offer systemUptime timestamps. Thank you.

I’m very interested in trying to have an iOS and watchOS device pair communicate and want to know if it’s possible for the iOS device to get the direction to the watchOS device. (I cannot try this because I don’t have an Apple Watch yet.) I’m looking at the documentation here and am not sure how to interpret the wording: nearby interaction docs Nearby Interaction on iOS provides a peer device’s distance and direction, whereas Nearby Interaction on watchOS provides only a peer device's distance. I’m not sure what is considered the peer. Let’s assume I’m communicating over a custom server and not using an iOS companion app. Is the above saying that: A: iOS will send watchOS the distance from the iOS device to the watchOS device and watchOS will send out its distance and direction to the iOS device? (i.e. NearbyConnevtivity on iOS receives distance and direction of any other device, regardless of whether it’s a phone or watch, but watchOS only gets distance.) B: The watch receives distance and direction to the phone, and the phone receives only the distance to the watch. C: the iOS device only gets distance to the watchOS device, and the watchOS device only gets distance to iOS device, period. May I have clarification? A secondary question is how often and how accurate the distance and directions are calculated and sent, but first things first. I’m looking forward to a reply. That would help very much and inform my decision to develop for watchOS. I have some neat project ideas that require option A or B to be true. Thanks for your time!

I am seeing that seemingly after the macOS 12.0.1 update, the macbook pro 16" 2021 is having widespread issues with the magsafe charger when shut-off, in which fast-charge causes the charger to loop the connection sound over and over without actually charging. discussion pages: https://forums.macrumors.com/threads/2021-macbook-pro-16-magsafe-light-flashing-amber-and-power-chime-repeating-during-charging-when-off.2319925/ https://www.reddit.com/r/macbookpro/comments/qi4i9w/macbook_pro_16_m1_pro_2021_magsafe_3_charge_issue/ https://www.reddit.com/r/macbookpro/comments/qic7t7/magsafe_charging_problem_2021_16_macbook_pro_read/ Most people suspect it's a firmware/OS issue. Is Apple aware of this / is it being worked-on? Has anyone tried this with the latest 12.1 beta as well?

On VisionOS, is a combination of full passthrough, unbounded volumes, and my own custom 3D rendering in Metal Possible? According to the RealityKit and Unity VisionOS talk, towards the end, it’s shown that an unbounded volume mode allows you to create full passthrough experiences with graphics rendering in 3D — essentially full 3D AR in which you fan move around the space. It’s also shown that you can get occlusion for the graphics. This is all great, however, I don’t want to use RealityKit or Unity in my case. I would like to be able to render to an unbounded volume using my own custom Metal renderer, and still get AR passthrough and the ability to walk around and composit virtual graphical content with the background. To reiterate, this is exactly what is shown in the video using Unity, but I’d like to use my own renderer instead of Unity or RealityKit. This doesn’t require access to the video camera texture, which I know is unavailable. Having the flexibility to create passthrough mode content in a custom renderwr is super important for making an AR experience in which I have control over rendering. One use case I have in mind is: Wizard’s Chess. You see the real world and can walk around a room-size chessboard with virtual chess pieces mixed with the real world, and you can see the other player through passthrough as well. I’d like to render graphics on my living room couches using scene reconstruction mesg anchors, for example, to change the atmosphere. The video already shows several nice use cases like being able to interact with a tabletop fantasy world with characters. Is what I’m describing possible with Metal? Thanks! EDIT: Also, if not volumes, then full spaces? I don’t need access to the camera images that are off-limits. I would just like passthrough + composition with 3D Metal content + full ARKit tracking and occlusion features.

Is full hand tracking on the Vision Pro available in passthrough AR (fully immersed with one application running), or only in fully immersive VR (no passthrough)?

Whenever I create a feedback assistant request, all my whitespace formatting disappears, making it unreadable for long posts. I’d like to make posts with sections such as “context,” “motivation,” etc. so the reader can better understand the request. Is it allowed just to put the body of the request in an attached text file, or will that risk making my request ignored or discarded? What is the proper etiquette for this?

Will a new example project be released to cover the new raytracing features in session WWDC2023-10128?

In regular Metal, I can do all sorts of tricks with texture masking to create composite objects and effects, similar to CSG. Since for now, AR-mode in visionOS requires RealityKit without the ability to use custom shaders, I'm a bit stuck. I'm pretty sure so far that what I want is impossible and requires a feature request, but here it goes: Here's a 2D example: Say I have some fake circular flashlights shining into the scene, depthwise, and everything else is black except for some rectangles that are "lit" by the circles. The result: How it works: In Metal, my per-instance data contain a texture index for a mask texture. The mask texture has an alpha of 0 for spots where the instance should not be visible, and an alpha of 1 otherwise. So in an initial renderpass, I draw the circular lights to this mask texture. In pass 2, I attach the fullscreen mask texture (circular lights) to all mesh instances that I want hidden in the darkness. A custom fragment shader multiplies the alpha of the full-screen mask texture sample at the given fragment with the color that would otherwise be output. i.e. out_color *= mask.a. The way I have blending and clear colors set-up, wherever the mask alpha is 0, an object will be hidden. The background clear color is black. The following is how the scene looks if I don't attach the masking texture. You can see that behind the scenes, the full rectangle is there. In visionOS AR-mode, the point is for the system to apply lighting, depth, and occlusion information to the world. For my effect to work, I need to be able to generate an intermediate representation of my world (after pass 2) that shows some of that world in darkness. I know I can use Metal separately from RealityKit to prepare a texture to apply to a RealityKit mesh using DrawableQueue However, as far as I know there is no way to supply a full-screen depth buffer for RealityKit to mix with whatever it's doing with the AR passthrough depth and occlusion behind the scenes. So my Metal texture would just be a flat quad in the scene rather than something mixed with the world. Furthermore, I don't see a way to apply a full-screen quad to the scene, period. I think my use case is impossible in visionOS in AR mode without customizable rendering in Metal (separate issue: I still think in single full app mode, it should be possible to grant access to the camera and custom rendering more securely) and/or a RealityKit feature enabling mixing of depth and occlusion textures for compositing. I love these sorts of masking/texture effects because they're simple and elegant to pull-off, and I can imagine creating several useful and fun experiences using this masking and custom depth info with AR passthrough. Please advise on how I could achieve this effect in the meantime. However, I'll go ahead and say a specific feature request is the ability to provide full-screen depth and occlusion textures to RealityKit so it's easier to mix Metal rendering as a pre-pass with RealityKit as a final composition step.

For my project, I would really benefit from continuous on-device speech recognition without the automatic timeout, or at least with a much longer one. In the WebKit web speech implementation, it looks like there are some extra setters for SFSpeechRecognizer exposing exactly this functionality: https://github.com/WebKit/WebKit/blob/8b1a13b39bbaaf306c9d819c13b0811011be55f2/Source/WebCore/Modules/speech/cocoa/WebSpeechRecognizerTask.mm#L105 Is there a chance Apple could enable programmable duration/time-out? If it’s available in WebSpeech, then why not in native applications?

In Xcode 15 Beta (5), I am noticing my breakpoints randomly seem to duplicate themselves multiple times for the exact same breakpoint. I have 3 targets in my project, and I wonder whether what I am experiencing is a bug related to that. Similarly, I also see duplicates of the same symbol in the symbol navigator. I've attached a screenshot of several identical breakpoints (in this case placed in some Objective C methods that relate to speech recognition). I haven't seen this happen in Xcode 14, or at least as often. Has anyone else experienced this and/or filed a bug report? I've tried deleting derivedData and the usual tricks.

I'd like to create meshes in RealityKit ( AR mode on iPad ) in screen-space, i.e. for UI. I noticed a lot of useful new functionality in RealityKit for the next OS versions, including the OrthographicCameraComponent here: https://developer.apple.com/documentation/realitykit/orthographiccameracomponent?changes=_3 I think this would help, but I need AR worldtracking as well as a regular perspective camera to work with the 3D elements. Firstly, can I have a camera attached selectively to a few entities, just for those entities? This could be the orthographic camera. Secondly, can I make it so those entities are always rendered in-front, in screenspace? (They'd need to follow the camera.) If I can't have multiple cameras, what can be done in that case? Is it actually better to use a completely different view / API for layering on-top of RealityKit? I would much rather keep everything in RealityKit, however, for simplicity.

I'd like to use the eye tracking feature in the latest iPadOS 18 update as more than an accessibility feature. i.e. another input modality that can be detected by event + enum checks similar to how we can detect and distinguish between touches and Apple pencil inputs. This might make it a lot easier to control and interact with iPad-based AR experiences that involve walking around, regardless of whether eye-tracking is enabled for accessibility. When walking, it's challenging to hold the device and interact with the screen with touch or pencil at all. Eye tracking + speech as input modalities could assist here. Also, this would help us create non-immersive AR experiences that parallel visionOS experiences that use eye tracking. I propose an API option for enabling eye-tracking (and an optional calibration dialogue within-app), as well as a specific UIControl class that simply detect when the eye looks at the control using the standard (begin/changed/end) events. My specific use case is that I'd like to treat eye-tracking-enabled UI elements or game objects differently depending on whether something is looked at with the eyes. For example, to select game objects while using speech recognition, suppose we have 4 buttons with the same name in 4 corners of the screen. Call them "corner" buttons. If I have my proposed invisible UI element for gaze detection, I can create 4 large rectangular regions on the screen. Then if the user says "select the corner" the system could parse this command and disambiguate between the 4 corners by checking which of the rectangular regions I'm currently looking at. (Note: the idea would be to make the gaze regions rather large to compensate for error.) The above is just a simple example, but the advantage over other methods like dwell is that it could be a lot faster. Another simple example: Using the same rectangular regions, instead of speech input, I could hold a button placed in just one spot on the screen, and look around the screen with my gaze to produce a laser beam for some kind of game, or draw curves (that I might smooth-out to reduce inaccuracy). OR maybe someone who does not have their hands available. This would require us to have the ability to get the coordinates of the eye gaze, but otherwise the other approach of just opting to trigger uicontrol elements might work for coarse selection. Would other developers find this useful as well? I'd like to propose this feature in feedback assistant, but I'm also opening-up a little discussion if by chance someone sees this. In short, I propose: a formal eye-tracking API for iPadOS 18+ that allows for turning on/off the tracking within the app, with the necessary user permissions the API should produce begin/changed/ended events similar to the existing events in UIKit, including screen coordinates. There should be a way to identify that an event came from eye-tracking. alternatively, we should have at minimum an invisible UIControl subclass that can detect when the eyes enter/leave the region.

I forgot to ask this during my lab session, but I noticed iPadOS is not listed under supported OSes under the GroupActivities documentation page. iPadOS supports FaceTime, but is it that GroupActivies doesn't work on iPadOS? This would be a crying shame since one of the examples specifically involved drawing collaboratively. The iPad is the perfect device for that use case. EDIT: Quick edit. Coordinate media experiences with Group Activities mentions iPadOS support, in which case the first page I linked might have a missing OS entry.

Xcode 13.4 only provides an SDK for macOS 12.3 according to the release notes. Can I build to macOS 12.4 using the lower point version SDK? I would not want to update the OS if I could not build to it yet. Thanks.