Is it possible for an app to analyze audio being played from another app?

The iOS security sandbox does not allow access to the audio output data produced by other apps unless they first explicitly export such data (for instance, by using the inter-app-audio API).

Howdy all!

Does the Apple Music app make its audio samples available for other apps via the inter-app audio?

I'd like to do some realtime FFT in a custom app from audio as it plays in the Apple Music app. The goal of this is to avoid having to write an interface and supporting software to play music in a library, and also to allow the user to play any music in his or her library without having to import or insert it into the custom app (e.g. want to avoid recreating a similar interface as the Apple Music app).

Thanks,

-Robert

Howdy,

It sounds like there might be some nuggets in MusicKit that would allow this kind of access (hopefully can get the audio samples from it to do the Fast Fourier Transforms (FFT)).

https://affiliate.itunes.apple.com/resources/documentation/apple-music-best-practices-for-app-developers/

Crossing fingers...

-Robert

Howdy,

It is still unclear how the audio samples can be obtained from the media player or related audio subsystem, when using MusicKit.

Thanks a bunch,

-Robert

Things have changed since this thread went up two years ago - rather than tagging it, feel free to start your own so it can receive the unique attention it deserves and you can better track, etc. and close should someone help solve your issue(s), thanks.

Hi Robert,

Short answer: no. Inter-App Audio was iOS-only, it was deprecated years ago (iOS 13), and Apple Music never published an IAA node anyway. There is no equivalent on macOS, and no app — Apple's or otherwise — hands you its render buffers through it. ▎ ▎ What you actually want is a system/process audio tap. On macOS you have three realistic routes:

1. Core Audio process taps (macOS 14.2+, Apple's sample targets 14.4) — this is the modern, official one. CATapDescription + AudioHardwareCreateProcessTap, then wrap it in an aggregate device and pull from it like any other input. You can scope the tap to a specific process (Music.app) or to the whole default output. Requires the audio-capture TCC permission (NSAudioCaptureUsageDescription in your Info.plist), so the user gets a one-time prompt — no driver install, nothing to uninstall later. See Apple's "Capturing system audio with Core Audio taps" sample.

2. ScreenCaptureKit (macOS 13+) — SCStream with capturesAudio = YES and a content filter limited to the Music app. It works and it's well documented, but it drags in the Screen Recording permission, which is a much scarier prompt for a user who just wants a spectrum analyzer. I'd only go here if you need to support 13.x.

3. A loopback/virtual device — BlackHole (free, open source) or Rogue Amoeba's Loopback. Zero API work on your side: your app just opens an input device. The cost is that the user has to install a driver and re-point Music's output, which is exactly the kind of setup friction you're trying to avoid.

Two caveats worth testing early, because they can sink the whole idea:

• DRM. Tracks streamed from the Apple Music catalog are protected content. Purchased/matched/local files are fine, but I would write a 50-line proof of concept that taps Music.app playing a streamed catalog track before you build anything on top of it — if the tap comes back silent, you know now instead of after the UI is done.
• Format. A tap gives you the mixed output at the device's format, so expect 44.1/48k float, and be ready for the sample rate to change under you when the user switches tracks or devices.

And if you were hoping to do this on iOS: don't. There's no system audio capture there at all. MusicKit / MPMusicPlayerController will let you drive playback and read metadata, but the samples never touch your process.

Best, Mabel