We are working on a voice app that uses ASR/TTS on the backend and run into some difficulties in noisy environments. We have compiled the DeepFilterNet3 library into an XCFramework and are using that on the app, but it's sometimes a bit ambitious with trimming out noise and removes some of the voice. Is there any way to make use of Apple's on-device voice isolation mic mode? I see that we can detect the user's mic mode, but we cannot programmatically set it. While we could prompt the user to enable it manually, this adds a bit of friction to the user experience. Do you have any suggestions for enabling voice isolation, or for performing denoising in general?

We are using AEC in our voice app and it mostly works. However, when the experience begins we play a greeting through the speaker, and the initial few hundred milliseconds of the greeting are being captured by the inputNode. This is throwing off our ASR/TTS. For now, we've disabled audio capture while playing audio, but would prefer to be able to capture all audio with echo cancellation working. Below is some relevant code snippets. Do you have any suggestions to get AEC working more quickly? I've tried a few things like enabling voice processing before setting the audio session to active. public init() { recorderNode = engine.inputNode speakerNode = engine.outputNode mainMixerNode = engine.mainMixerNode engine.attach(audioPlayer) engine.connect( audioPlayer, to: mainMixerNode, format: nil ) playbackFormat = mainMixerNode.outputFormat(forBus: 0) } public func setupAudioSession() async throws(AudioError) { do { let audioSession = AVAudioSession.sharedInstance() try audioSession.setCategory( .playAndRecord, mode: .voiceChat, policy: .default, options: [ .defaultToSpeaker, .allowBluetoothHFP, ] ) try audioSession.setActive(true) } catch { throw .audioSessionSetupFailed(error) } do { try recorderNode.setVoiceProcessingEnabled(true) try speakerNode.setVoiceProcessingEnabled(true) } catch { throw .enableVoiceProcessingFailed(error) } }

I have a voice app that is both playing and recording audio. I have enabled voice processing and am setting AVAudioSession.Category to .playAndRecord and AVAudioSession.Mode to .voiceChat. When the experience first launches, we play a greeting. The first few hundred milliseconds of that greeting are being captured by the inputNode before AEC seems to start working. Is there any way to get AEC working the entire time? For now we've had to disable recording while we're playing audio, but would prefer to both play and record simultaneously. Here's some code snippets: public init(denoiseModelPath: URL? = nil) { noiseReducer = denoiseModelPath.flatMap { NoiseReducer(modelPath: $0) } recorderNode = engine.inputNode speakerNode = engine.outputNode mainMixerNode = engine.mainMixerNode engine.attach(audioPlayer) engine.connect( audioPlayer, to: mainMixerNode, format: nil ) playbackFormat = mainMixerNode.outputFormat(forBus: 0) } public func setupAudioSession() async throws(AudioError) { do { let audioSession = AVAudioSession.sharedInstance() try audioSession.setCategory( .playAndRecord, mode: .voiceChat, policy: .default, options: [ .defaultToSpeaker, .allowBluetoothHFP, ] ) try audioSession.setActive(true) } catch { throw .audioSessionSetupFailed(error) } do { try recorderNode.setVoiceProcessingEnabled(true) try speakerNode.setVoiceProcessingEnabled(true) } catch { throw .enableVoiceProcessingFailed(error) } }

We are working on a voice app that uses ASR/TTS on the backend and run into some difficulties in noisy environments. We have compiled the DeepFilterNet3 library into an XCFramework and are using that on the app, but it's sometimes a bit ambitious with trimming out noise and removes some of the voice. Is there any way to make use of Apple's on-device voice isolation mic mode? I see that we can detect the user's mic mode, but we cannot programmatically set it. While we could prompt the user to enable it manually, this adds a bit of friction to the user experience. Do you have any suggestions for enabling voice isolation, or for performing denoising in general?

Why are none of my posts showing up in the Audio Q&A? There are no errors or any useful messaging... my posts just aren't showing.

I am experimenting with SpeechTranscriber and am curious if I can get quicker results when using buffered audio, rather than a file. The use case is a voice ordering experience for a restaurant. When I've been playing with it, it takes about 3 seconds for faster results and 7-8 seconds for accurate results. Is there any way to bring this down a bit? In this WWDC demo, the results appear nearly instantaneously. I'm curious how to replicate this in my app. I presume DicationTranscriber is faster, but how is siri detecting when the user stops speaking? Is it custom code, or is it using SpeechDetector? I tried using SpeechDetector with SpeechTranscriber but the detector didn't emit any results and seemed to slow down the results of SpeechTranscriber. I also assumed SpeechTranscriber makes more sense than DictationTranscriber in this use case, but want to confirm.

I am using VNRecognizeTextRequest to read Chinese characters. It works fine with text written horizontally, but if even two characters are written vertically, then nothing is recognized. Does anyone know how to get the vision framework to either handle vertical text or recognize characters individually when working with Chinese? I am setting VNRequestTextRecognitionLevel to accurate, since setting it to fast does not recognize any Chinese characters at all. I would love to be able to use fast recognition and handle the characters individually, but it just doesn't seem to work with Chinese. And, when using accurate, if I take a picture of any amount of text, but it's arranged vertically, then nothing is recognized. I can take a picture of 1 character and it works, but if I add just 1 more character below it, then nothing is recognized. It's bizarre. I've tried setting usesLanguageCorrection = false and tried using VNRecognizeTextRequestRevision3, ...Revision2 and ...Revision1. Strangely enough, revision 2 seems to recognize some text if it's vertical, but the bounding boxes are off. Or, sometimes the recognized text will be wrong. I tried playing with DataScannerViewController and it's able to recognize characters in vertical text, but I can't figure out how to replicate it with VNRecognizeTextRequest. The problem with using DataScannerViewController is that it treats the whole text block as one item, and it uses the live camera buffer. As soon as I capture a photo, I still have to use VNRecognizeTextRequest. Below is a code snippet of how I'm using VNRecognizeTextRequest. There's not really much to it and there aren't many other parameters I can try out (plus I've already played around with them). I've also attached a sample image with text laid out vertically. func detectText( in sourceImage: CGImage, oriented orientation: CGImagePropertyOrientation ) async throws -> [VNRecognizedTextObservation] { return try await withCheckedThrowingContinuation { continuation in let request = VNRecognizeTextRequest { request, error in // ... continuation.resume(returning: observations) } request.recognitionLevel = .accurate request.recognitionLanguages = ["zh-Hant", "zh-Hans"] // doesn't seem have any impact // request.usesLanguageCorrection = false do { let requestHandler = VNImageRequestHandler( cgImage: sourceImage, orientation: orientation ) try requestHandler.perform([request]) } catch { continuation.resume(throwing: error) } } }