Explore the power of machine learning within apps. Discuss integrating machine learning features, share best practices, and explore the possibilities for your app.

Posts under General subtopic

Post

Replies

Boosts

Views

Activity

NLLanguageRecognizer does not return Malay
I’m testing NLLanguageRecognizer to distinguish Malay (Bahasa Melayu) from English using this Malay text: Buka App Muzik untuk Mendayakan Widget Without language hints, my sandbox produces the following results. Percentages are rounded for display: Constraints Returned scores ID, EN ID: 100%; EN: 0% MS, EN EN: 100%; MS: not returned VI, EN VI: 98.7%; EN: 1.3% ID, MS, VI, EN ID: 100%; MS: not returned; EN and VI: 0% I also tried supplying languageHints to favor Malay and changed my phone’s system language to Malay. Neither resolved the issue: the recognizer still failed to identify the text as Malay. Here is a minimal example with hints: import NaturalLanguage let recognizer = NLLanguageRecognizer() recognizer.languageConstraints = [.malay, .english] recognizer.languageHints = [.malay: 0.8, .english: 0.2] recognizer.processString("Sila isi borang di bawah") print(recognizer.dominantLanguage?.rawValue ?? "nil") for (language, score) in recognizer.languageHypotheses(withMaximum: 2) { print(language.rawValue, score) } An independent native macOS test with the same short text also returned English with score 1.0, both with hints disabled and with the 0.80 Malay hint. To investigate whether this was only short-text ambiguity, I separately tested a longer Malay paragraph taken from local news on macOS: Terdahulu, peguam PN, Ivanpal Singh Grewal, berhujah bahawa penangguhan wajar diberikan memandangkan keadaan luar biasa susulan PRN Melaka pada 14 Nov, sambil berkata penggantungan itu menghalang gabungan berkenaan daripada menyertai proses demokrasi. Without hints, that paragraph produced: Constraints Result None Indonesian (id), score 1.0 English and Malay English (en), score 1.0; no Malay hypothesis Malay only dominantLanguage is nil; hypotheses are empty English, Malay, Indonesian Indonesian (id), score 1.0 I understand that Malay and Indonesian share vocabulary, short strings can be ambiguous, and constraints can force a prediction into an allowed language. However, Malay appears absent even when explicitly allowed and favored. My questions are: Is Malay identification supported by NLLanguageRecognizer on iOS and macOS? Could the missing Malay hypothesis and empty Malay-only result indicate a model limitation or framework defect? Is there an API to check which languages are available for recognition on the current runtime? Are any additional settings or language resources required? Environment iPhone model / iOS version: iPhone 13, iOS 27.2 Phone system language: tested with English and Malay Independent Mac reproduction: macOS 26.7.1 (25G241), Apple Silicon, Xcode 27.2 Beta 3
1
0
1.3k
1w
Best approach for automatically processing bank receipt images on iOS using on-device OCR?
What is the best approach to automatically process bank receipt images on iOS while respecting user privacy? I'm exploring ways to automatically extract transaction details from receipt images using on-device OCR. I'd like to discuss which Apple APIs and permission models would be appropriate for this functionality.
1
0
1.2k
1w
Vision face landmarks shifted on iOS 26 but correct on iOS 18 with same code and image
I'm using Vision framework (DetectFaceLandmarksRequest) with the same code and the same test image to detect face landmarks. On iOS 18 everything works as expected: detected face landmarks align with the face correctly. But when I run the same code on devices with iOS 26, the landmark coordinates are outside the [0,1] range, which indicates they are out of face bounds. Fun fact: the old VNDetectFaceLandmarksRequest API works very well without encountering this issue How I get face landmarks: private let faceRectangleRequest = DetectFaceRectanglesRequest(.revision3) private var faceLandmarksRequest = DetectFaceLandmarksRequest(.revision3) func detectFaces(in ciImage: CIImage) async throws -> FaceTrackingResult { let faces = try await faceRectangleRequest.perform(on: ciImage) faceLandmarksRequest.inputFaceObservations = faces let landmarksResults = try await faceLandmarksRequest.perform(on: ciImage) ... } How I show face landmarks in SwiftUI View: private func convert( point: NormalizedPoint, faceBoundingBox: NormalizedRect, imageSize: CGSize ) -> CGPoint { let point = point.toImageCoordinates( from: faceBoundingBox, imageSize: imageSize, origin: .upperLeft ) return point } At the same time, it works as expected and gives me the correct results: region is FaceObservation.Landmarks2D.Region let points: [CGPoint] = region.pointsInImageCoordinates( imageSize, origin: .upperLeft ) After that, I found that the landmarks are normalized relative to the unalignedBoundingBox. However, I can’t access it in code. Still, using these values for the bounding box works correctly. Things I've already tried: Same image input Tested multiple devices on iOS 26.2 -> always wrong. Tested multiple devices on iOS 18.7.1 -> always correct. Environment: macOS 26.2 Xcode 26.2 (17C52) Real devices, not simulator Face Landmarks iOS 18 Face Landmarks iOS 26
1
0
864
1w
iOS 27 + RecognizedText misses lines
I'm using the iOS 18+ RecognizedTextRequest, and with no change of code, iOS 27 gives poorer results: lines are dropped, all the text is not recognized it's not my code, because Live Text has the same issue and of course, back to 26 or 18 solves the issue What changed? Only interesting infos I could isolate: The simulator has not the issue Squashing the image horizontally (80% of its width for example) makes the detection significantly better
6
5
1.8k
2w
Major iOS 27 Regression in '.accurate' Model for VNRecognizeTextRequest
Hello, With the update from iOS 26 to iOS 27 I have noticed significant regressions and degraded performance when using the Vision Framework for OCR via 'VNRecognizeTextRequest'. The degraded performance is only applicable to the .accurate model, the .fast model is identical between iOS 26 and iOS 27. I have an app that uses this framework extensively for extracting information from the video feed of professional cinema cameras. This runs using en-US, language correction off with a specific region of interest. On iOS 26 everything works reliably as it has for the past few years, with the exact same codebase on iOS 27 many strings that were previously detected are no longer detected, some strings return as if they've been read upside down (ISO1280 is returned as 08ZLOSI) and there are a lot of Cyrillic look alikes mixed into Latin text, there also appears to be a lot more junk characters tacked onto the end of numbers ('12ł' is returned when previously it was just '12'). You often see whole line duplicates too where the same text is recognised twice, once as a single line with all strings combined and once as individual strings. The iOS 27 Simulator doesn't show any of this. I only saw it on real iPads, if you only test in the Simulator you'll miss it. To put numbers on it, I built a test set of about 26,500 frame grabs and ran them through 'VNRecognizeTextRequest' on iPadOS 18, 26 and 27. With the version of my app that's on the App Store today, iOS 26 reads 99.7% of values correctly. On iOS 27 that drops to 65%. I've resolved this issue now and have been able to retain the level of performance that existed on iOS 26, this involves various preprocessing steps to the images as well as a significant amount of validation after the Vision Framework returns its results. These fixes took around 3.5 Days of work which is incredibly frustrating. I would much rather use my time building new features instead of fixing regressions introduced by poorly validated new models that have been shipped in iOS 27. What frustrates me most is this change wasn't announced anywhere and there's no way to opt out. The revision number didn't change so there's no way to keep using the model that worked. If this had shipped as VNRecognizeTextRequestRevision4 apps could have stayed on VNRecognizeTextRequestRevision3 until they'd had a chance to test. Or permanently stayed on 'VNRecognizeTextRequestRevision3' as obviously the new version performs significantly worse. Instead an app that worked fine on iOS 26 now gets wrong text back on iOS 27 with no error, no warning and nothing in the release notes. For a framework people rely on for real work that's a significant regression. It seems that other developers have also come across this issue: https://developer.apple.com/forums/thread/845452 For technical reference, if you diff System/Library/PrivateFrameworks/TextRecognition.framework between iOS 26.5 (23F77) and iOS 27.0 (24A434) you uncover a number of changes. New models on iOS 27 seem to use an '_e5' variant. The text detector in iOS 26 references training run 'yyzn76u6fs' with 88560 iterations, iOS 27 references training run 'hvcthxumvz' with only 10116 iterations. Perhaps an 8x reduction in iterations has something to do with the poor performance in iOS 27? Further, the orientation model changes from iOS 26 'cr_orientation_model_v1' with 1858472 iterations vs. iOS 27 'cr_orientation_model_v1_e5' with 671065 iterations. The Latin and Cyrillic Recogniser changed model entirely from 'cr_tr_model_latincyrillic_v3' on iOS 26 to 'cr_tr_model_latincyrillic_v3_e5' on iOS 27. It seems these 'e5' models are optimised for the neural engine. Is there any way to get the iOS 26 models/behaviour back or plans to back deploy the old model in iOS 27.1 or iOS 27.2? Can future model changes like this come with a new revision number? What testing process does the Vision Team use to validate models as clearly something was missed here? Thanks!
0
0
323
2w
Framing an object in real time
Hi, I am currently working on a project which involves giving directions to the user telling them what they should do to frame a given object (also only doing this if sees the object which can be many things, so it needs detection too) so that it can be optimally in view of the camera to be classified. I have figured the classification, but not the real time guidance and detection. I tried training a model that featured all of the different movements ("move up", "move down", etc.). It featured 30 different objects with ten crops per image, and each class featured all 30 images and an even amount per class. It had over 1500 images in total, and I trained using createml and core ml. But, they were very inaccurate in my testing. I also tried using foundation models for this, but it also didn't work. I am quite new to integrating ai/ ml models into a project, and I am unsure of where to go next. If anyone has any suggestions, I would be really appreciative.
0
0
398
2w
iOS 27: English sentence embedding model unavailable on physical devices
I’m seeing NLEmbedding.sentenceEmbedding(for: .english) return nil on physical devices running iOS 27, both in debug builds under Xcode and in production. The same failure has been observed across thousands of users on iOS 27. One affected device runs iOS 27.0.1 (24A446). Minimal example with diagnostics: import NaturalLanguage let language = NLLanguage.english print("Current revision:", NLEmbedding.currentSentenceEmbeddingRevision(for: language)) print("Supported revisions:", Array(NLEmbedding.supportedSentenceEmbeddingRevisions(for: language))) if let model = NLEmbedding.sentenceEmbedding(for: language) { print("Loaded revision:", model.revision, "dimension:", model.dimension) } else { print("Sentence embedding model unavailable") } On affected devices, the current revision is 1 and supported revisions are [1], but model loading returns nil. The framework logs: Unable to locate Asset for sentence embedding model for local en. The failure occurs during model loading, before any call to vector(for:). This uses Apple’s built-in English sentence model, with no custom embedding file. Is this a known issue on iOS 27? Is there a supported way to request or restore the missing sentence-model asset, or a recommended recovery approach?
0
0
528
3w
Crashed: AXSpeech EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x000056f023efbeb0
Application is getting Crashed: AXSpeech EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x000056f023efbeb0 Crashed: AXSpeech 0 libobjc.A.dylib 0x4820 objc_msgSend + 32 1 libsystem_trace.dylib 0x6c34 _os_log_fmt_flatten_object + 116 2 libsystem_trace.dylib 0x5344 _os_log_impl_flatten_and_send + 1884 3 libsystem_trace.dylib 0x4bd0 _os_log + 152 4 libsystem_trace.dylib 0x9c48 _os_log_error_impl + 24 5 TextToSpeech 0xd0a8c _pcre2_xclass_8 6 TextToSpeech 0x3bc04 TTSSpeechUnitTestingMode 7 TextToSpeech 0x3f128 TTSSpeechUnitTestingMode 8 AXCoreUtilities 0xad38 -[NSArray(AXExtras) ax_flatMappedArrayUsingBlock:] + 204 9 TextToSpeech 0x3eb18 TTSSpeechUnitTestingMode 10 TextToSpeech 0x3c948 TTSSpeechUnitTestingMode 11 TextToSpeech 0x48824 AXAVSpeechSynthesisVoiceFromTTSSpeechVoice 12 TextToSpeech 0x49804 AXAVSpeechSynthesisVoiceFromTTSSpeechVoice 13 Foundation 0xf6064 __NSThreadPerformPerform + 264 14 CoreFoundation 0x37acc CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION + 28 15 CoreFoundation 0x36d48 __CFRunLoopDoSource0 + 176 16 CoreFoundation 0x354fc __CFRunLoopDoSources0 + 244 17 CoreFoundation 0x34238 __CFRunLoopRun + 828 18 CoreFoundation 0x33e18 CFRunLoopRunSpecific + 608 19 Foundation 0x2d4cc -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 212 20 TextToSpeech 0x24b88 TTSCFAttributedStringCreateStringByBracketingAttributeWithString 21 Foundation 0xb3154 NSThread__start + 732 com.livingMedia.AajTakiPhone_issue_3ceba855a8ad2d1af83655803dc13f70_crash_session_9081fa41ced440ae9a57c22cb432f312_DNE_0_v2_stacktrace.txt 22 libsystem_pthread.dylib 0x24d4 _pthread_start + 136 23 libsystem_pthread.dylib 0x1a10 thread_start + 8
5
1
2.7k
3w
Crakling with iOS 27
I developed a conversational agent using Gradium and it worked fine using the iPhone's built-in speaker on iOS 18.7 and iOS 26.6. I developed similar conversational agent using the Gemini stack and it also worked fine using the iPhone's built-in speaker on iOS 18.7 and iOS 26.6. Now on iOS 27.0, we hear crackling in the transcripts and responses, with both stacks.
1
1
135
Sep ’26
SwiftPlaygroundsでドローンTelloアプリを制作
皆さん、はじめまして。プログラミングに対する知識がないので、AIを使いながらiPadでTelloのドローンを操縦できるアプリを作っています。このアプリはシミュレーション飛行とドローンの実機飛行ができるアプリにしたいのですが、ドローンの実機飛行を行うためのWi‐Fi接続ができません。(iPadでのシミュレーション飛行が可能です)iPadの基本的な設定はAppleに教えていただき、設定変更を行いましたが、どうしても接続できないので、わかる方法を教えていただけると嬉しいです。 コードを添付いたしますので、ご教示ください。 ドローンアプリコード
0
0
736
Aug ’26
Group AppIntents’ Searchable DynamicOptionsProvider in Sections
I’m trying to group my EntityPropertyQuery selection into sections as well as making it searchable. I know that the EntityStringQuery is used to perform the text search via entities(matching string: String). That works well enough and results in this modal: Though, when I’m using a DynamicOptionsProvider to section my EntityPropertyQuery, it doesn’t allow for searching anymore and simply opens the sectioned list in a menu like so: How can I combine both? I’ve seen it in other apps, but can’t figure out why my code doesn’t allow to section the results and make it searchable? Any ideas? My code (simplified) struct MyIntent: AppIntent { @Parameter(title: "Meter"), optionsProvider: MyOptionsProvider()) var meter: MyIntentEntity? // … struct MyOptionsProvider: DynamicOptionsProvider { func results() async throws -> ItemCollection<MyIntentEntity> { // Get All Data let allData = try IntentsDataHandler.shared.getEntities() // Create Arrays for Sections let fooEntities = allData.filter { $0.type == .foo } let barEntities = allData.filter { $0.type == .bar } return ItemCollection(sections: [ ItemSection("Foo", items: fooEntities), ItemSection("Bar", items: barEntities) ]) } } struct MeterIntentQuery: EntityStringQuery { // entities(for identifiers: [UUID]) and suggestedEntities() functions func entities(matching string: String) async throws -> [MyIntentEntity] { // Fetch All Data let allData = try IntentsDataHandler.shared.getEntities() // Filter Data by String let matchingData = allData.filter { data in return data.title.localizedCaseInsensitiveContains(string)) } return matchingData } }
2
2
1.6k
Aug ’26
M3 Neural Engine kernelDMA bandwidth throttled at 1 MiB multiples; splitting restores 3x DRAM bandwidth
A suspected RTL performance erratum in the Apple Neural Engine throttles DRAM weight streaming throughput down to 17–19 GB/s from the nominal 45–60 GB/s, whenever the total weight size is an integer multiple of 1 MiB. This was confirmed on M3 Macbook Air (2024) 24GB. 1 MiB is a common transfer size, and currently affects 7 of ANEMLL’s 15 baseline models. Avoiding the suspected problematic path in the kernel DMA's prefetch ring by splitting 1 MiB transfers into non-multiples of 1 MiB, directly increased Llama 3.2 1B token throughput from 10.0 to 24.3 tokens/s (DRAM usage from 24.7 to 60.0 GB/s), and Qwen3-8B from 1.36 to 2.97 tokens/s (DRAM usage from 22.4 to 48.7 GB/s). Please see writeup here: https://eiln.github.io/posts/ane-dma.html I'm looking to confirm that this behavior is reproducible on your side, and will provide the scripts in my plots.
0
0
484
Aug ’26
MLXVLM factory fallback doesn't catch a vision_config-present-but-no-tower-weights checkpoint (Qwen3.5-9B)
I ran into a gap in the MLXVLM → MLXLLM factory fallback and wanted to check whether this is expected behavior or something worth filing. Context: I maintain a Swift-native macOS agent built on MLX Swift, with a hardware-adaptive catalog of local models. I tried adding mlx-community/Qwen3.5-9B-4bit as a plain text model. Its config.json ships a full vision_config block — it looks like a multimodal config — but the actual checkpoint has no vision tower weights at all. It's genuinely text-only; the metadata just doesn't reflect that. ModelFactoryRegistry tries MLXVLM before falling back to MLXLLM. I expected that to fail cleanly and fall through, since there's no vision encoder to load. It doesn't. MLXVLM's Qwen35Configuration declares visionConfiguration as a non-optional property mapped to vision_config. Since the key is present in the raw config, decoding succeeds — there's nothing to catch at that stage. So MLXVLM proceeds to actually load the model as a VLM, and it crashes later, inside WiredMemoryUtils.tune(), because getRopeIndex() expects real vision-encoder dimensions to compute against, and there are none. Not a clean decode-time error I could catch — a runtime crash several layers past config parsing, in the RoPE indexing math. The workaround I ended up with is config-level surgery before the model reaches the factory at all: strip vision_config (and image_token_id / video_token_id) from the JSON first. With the key genuinely absent, Qwen35Configuration's decode throws DecodingError.keyNotFound as the fallback logic presumably intends, MLXVLM fails cleanly at that stage, and ModelFactoryRegistry falls through to MLXLLM correctly. That works, but it feels like a workaround rather than the intended path — the factory fallback seems to assume any failure will happen at decode time, and a config key that's present but semantically empty (no real vision weights behind it) skips that safety net entirely. Questions: Is there a more canonical/official way to detect whether a checkpoint actually has real vision tower weights, short of opening model.safetensors.index.json and counting vision_tower.* tensor names myself? (I've done exactly that for a different, genuinely multimodal Qwen VL checkpoint — 333 real vision tensors there vs. zero here — but it feels like something the loading path should be able to tell me.) Is the vision_config-present-but-empty pattern something the mlx-community conversion pipeline is aware of, or is this specific to how Qwen3.5-9B-4bit happened to get converted? Happy to share more logs/repro details if useful.
0
0
398
Aug ’26
How can a local AI agent use MLX/Metal unattended on macOS while remaining confined to an authorized workspace?
How can a local AI agent use MLX/Metal unattended while remaining confined to an authorized workspace? I am developing an AI-driven local media-processing workflow on an Apple-silicon Mac and am trying to understand the correct architecture for allowing it to run unattended without giving the AI agent unrestricted access to my primary personal computer. I am not a software engineer, so I may be missing an established macOS mechanism or using the wrong terminology. I would appreciate guidance from people familiar with MLX, Metal, sandboxing, and macOS security. What I am building I use OpenAI Codex as the local execution/software-development agent. The working system currently: ingests and verifies original video and still media while preserving immutable originals; performs visual semantic analysis and divides video into meaningful time-coded segments; separately analyzes spoken language rather than assuming audio and video are semantically equivalent; uses MLX Whisper locally on Apple silicon for time-coded speech transcription; stores visual and language semantics in a relational SQLite media catalog. These five stages are working. My current test corpus contains 148 original media files, 126 visual semantic segments, and 765 speech segments. The next stages are AI editorial construction from the semantic database and generation of instructions/scripts for a DaVinci Resolve rough cut. The security architecture I want Codex to operate autonomously within a deliberately bounded development environment. I do not want to solve this simply by granting an autonomous agent Full Disk Access to my primary personal Mac. The concern is ordinary fault containment. Codex generates and executes scripts, invokes applications and command-line tools, and manipulates files. A mistaken path or defective generated script should not have unrestricted consequences for the rest of my computer. I therefore separated AI execution from ordinary personal files. Codex is configured for Workspace Write access with explicitly authorized project roots. Canonical media resides on a separately authorized external SSD, and temporary AI working artifacts are kept separately. Ordinary Python and FFmpeg operations now run autonomously within these authorized areas. The problem The difficulty appears when the workflow invokes capabilities that cannot operate inside the ordinary Codex sandbox. The clearest example is MLX Whisper. I am using: MLX Whisper 0.4.3 mlx-community/whisper-small-mlx Apple silicon local transcription MLX Whisper works successfully and its transcription quality is sufficient for my semantic-retrieval application. However, MLX could not access Apple Metal/GPU execution from inside the ordinary Codex sandbox. Codex therefore requested permission to execute the transcription operation outside the sandbox. Once approved, MLX/Metal worked and the entire corpus was successfully transcribed. The processing therefore works, but the workflow cannot run genuinely unattended. A future operation should be able to run: new media → integrity verification → visual semantic analysis → MLX Whisper transcription → language semantic analysis → SQLite update → QA But if execution stops midway waiting for a human to click Allow, the pipeline is not operationally autonomous. What I have already tried I initially encountered permission problems even with ordinary file operations. I therefore: separated Codex work from ordinary personal documents; created dedicated project/work areas; explicitly authorized the required working roots; configured Workspace Write; separately authorized the external media repository; tested shell/Python and FFmpeg operations within those boundaries. Those changes worked. Routine Python and FFmpeg operations now run without approval prompts. The remaining issue occurs with MLX/Metal and some other application/runtime operations that require sandbox escalation. My question Is there a supported architecture for allowing a local AI agent to invoke MLX/Metal and other deliberately authorized development tools unattended, while still confining the agent to defined project/workspace boundaries rather than granting unrestricted access to the entire Mac? For example, should I be investigating: App Sandbox entitlements; a signed helper tool or XPC service; security-scoped resources; a dedicated executable with appropriate entitlements; a different method of launching MLX/Metal; or another macOS mechanism? In particular, can Metal/GPU access coexist with persistent bounded filesystem access without requiring interactive approval each time the AI invokes it? I am also unsure which security layer is actually responsible here: the Codex sandbox, macOS App Sandbox, TCC, executable/code-signing rules, Metal restrictions, or some interaction among them. If this kind of bounded unattended execution is intentionally not supported, that would also be useful to know. My alternative would be a dedicated Apple-silicon Mac containing only the AI-development environment and replaceable project data, where broader permissions would have a much smaller failure domain. I can provide the Codex configuration, exact successful and failing commands, directory/root configuration, macOS/hardware information, and sandbox diagnostics. I would particularly appreciate guidance on which security layer is causing the MLX/Metal escalation and what the supported architecture would be for this use case. Thank you.
0
0
852
Aug ’26
Fused Metal Kernels for Linear Recurrences in MLX
I’ve been developing mlx-recurrence, a plug-in framework of fused Metal GPU kernels for linear recurrences on Apple silicon—roughly analogous to flash linear attention for MLX. Sequential recurrences are difficult for MLX to fuse automatically. Architectures such as state-space models, gated linear attention, and diagonal RNNs ordinarily require a loop across the sequence length. When that loop is implemented in Python, a sequence of length L can require L separate Python-to-Metal dispatches. These kernels instead execute the entire recurrence in a single Metal dispatch. The training path uses segment checkpointing with recomputation during the backward pass. In validated M3 Max tests, the checkpoint-and-recompute kernels reduced peak recurrent-state memory by approximately 12–18× at the kernel level and lowered total training peak memory from 23.88 GB to 10.34 GB. At the same batch size, end-to-end training throughput improved by roughly 1.4×, while individual fused forward-and-backward kernels ran approximately 1.5–1.9× faster than the original full-state implementations. Results will vary with recurrence type, sequence length, state dimensions, batch size, datatype, model architecture, and hardware. Current kernels: ssd_scan Mamba-2-style, head-wise SSD selective scan. Intended for Mamba-2 and other SSM hybrid architectures. State shape: [B, H, Dh, N] gla_scan Gated Linear Attention with a scalar forget gate and outer-product write. Intended for GLA and linear-attention hybrid architectures. State shape: [B, H, Dh, Dh] rglru_scan RG-LRU diagonal recurrence. Intended for Griffin and RecurrentGemma-style architectures. State shape: [B, D] rotlru_scan Rotational LRU using a complex-diagonal recurrence, a magnitude gate, and a per-step rotation of two-dimensional channel pairs. Intended for complex-LRU and S4-style oscillatory memory architectures. State shape: [B, D], represented as interleaved channel pairs. Each kernel is implemented as a self-contained plug-in on a shared chassis located at: mlx_recurrence._chassis The chassis provides: Segment checkpoint-and-recompute infrastructure Shape and argument validation VJP integration Forward and gradient parity-test helpers Common recurrence plug-in handling Adding another recurrence therefore requires implementing its Metal forward and backward source pair and connecting its VJP. The checkpointing, validation, and testing infrastructure does not need to be rebuilt for each operator. The original version 0.1 kernels remain available under: mlx_recurrence.legacy They are also re-exported at the package’s top level for backward compatibility. I’m interested in feedback from developers working with MLX or custom Metal compute kernels, particularly around: Preferred APIs for packaging reusable MLX Metal extensions Threadgroup and memory-layout strategies across Apple GPU generations Numerical stability expectations for long recurrent sequences Benchmarking fused scans against MLX-native implementations Additional recurrent operators that would be valuable to support I would also be interested to know whether others are developing similar fused recurrence primitives for MLX and whether a shared interface for these operations would be useful. My setup is a M3 MAX Macbook Pro with 36GB Ram and I am running on macOS 26.4.1 (25E253). https://github.com/D-CSIL/mlx-recurrence
0
0
301
Aug ’26
Will Siri AI be able to copy text content of whats on screen in Notes/Files/Photos?
Let's say I'm in Notes, or in Pages, or in Files or Live Text if I'm in Photos... will Siri AI be able to COPY the text for me if I ask it to do so? If you have new Siri AI installed, or are DTS Engineer at Apple I'd appreciate a yes/no? Critical for text editing, working with AI output, and general modern work requirements. Presently I have to press the "share" button then select Copy from the Share Sheet, OR in Pages I have to select EXPORT from the More Menu and choose Plain Text to get the contents. Thank you. Be well.
0
0
694
Aug ’26
Questions for Apple Support / Apple Vision Team
Dear Apple Support, I would like to report a long-standing issue affecting Khmer text recognition in Live Text (Vision Framework/OCR). Based on my testing, this issue has persisted for more than two years, from iOS 17 through iOS 27 Beta 3, and is also reproducible on iPadOS and macOS. I would appreciate clarification on the following questions: Is Apple aware of an issue where Live Text (OCR/Text Recognition) incorrectly recognizes Khmer script as Thai script, causing copied text to become Thai characters instead of Khmer? Has this issue been officially logged as a bug within the Vision Framework or Live Text team? Since this behavior has remained reproducible from iOS 17 to iOS 27 Beta 3, why has it not yet been resolved? Is the problem caused by: automatic language detection, the OCR recognition model, the Vision Framework, or another component of Apple's AI pipeline? Does Apple currently have a dedicated OCR and language recognition model for the Khmer script, or is Khmer being inferred through another language model? Is there an estimated timeline for improving Khmer OCR and preventing Khmer text from being misidentified as Thai? Can Apple confirm whether this issue affects all products using Vision Framework, including: Live Text Photos Preview Screenshot OCR APIs provided to third-party developers? How can Apple work with the Khmer technology community to improve OCR accuracy and language support for Khmer? This issue is more than a simple OCR bug. When Khmer text is automatically converted into Thai characters, users lose access to the original text, developers receive incorrect OCR output, and it negatively impacts the digital representation of the Khmer language. For reference, I have documented the issue in detail here: https://app.notion.com/p/Inaccurate-OCR-Language-Inference-Khmer-Script-Misidentified-as-Thai-in-Vision-Framework-2d8a24f4ee6680fcbc49d989f8bb606f I hope Apple can investigate this issue and prioritize improving Khmer language support across Vision Framework and Live Text. Thank you.
7
19
1.5k
Jul ’26
AI framework usage without user session
We are evaluating various AI frameworks to use within our code, and are hoping to use some of the build-in frameworks in macOS including CoreML and Vision. However, we need to use these frameworks in a background process (system extension) that has no user session attached to it. (To be pedantic, we'll be using an XPC service that is spawned by the system extension, but neither would have an associated user session). Saying the daemon-safe frameworks list has not been updated in a while is an understatement, but it's all we have to go on. CoreGraphics isn't even listed--back then it part of ApplicationServices (I think?) and ApplicationServices is a no go. Vision does use CoreGraphics symbols and data types so I have doubts. We do have a POC that uses both frameworks and they seem to function fine but obviously having something official is better. Any Apple engineers that can comment on this?
10
0
2.7k
Jul ’26
[iOS 27 DB3] Apple Intelligence and Spotlight Stuck at 85-90% - How to force completion via Ethernet and Console logs analysis
Hey everyone, If your iPhone is stuck on "Optimizing Search and Siri" / Indexing at around 85-90% and hard resets or leaving it on wireless charging overnight isn't moving the needle, I found a definitive way to kickstart the daemon and force it to 100%. The Root Cause (Analyzed via macOS Console): By plugging the iPhone into a Mac and checking the Console logs, I noticed that spotlightknowledged and biomed get into an endless loop around Resolved entitled set identifiers to enumerate data resources. The system stalls on verifying developer entitlements and Apple Intelligence security tokens for specific app databases over cellular networks or unstable Wi-Fi. The dasd daemon eventually puts the pipeline into a hidden sleep state to protect the battery, making it look like it's doing nothing. The Solution that forced it to 100%: Use a Wired Connection (Ethernet Adapter): Connect your iPhone directly to your fiber optic router/modem using an RJ-45 Ethernet to USB-C adapter. iOS treats a wired Ethernet connection with the highest background priority. Turn off Cellular/Wi-Fi: Go to Control Center and disable Cellular Data and Wi-Fi entirely to force iOS to route 100% of traffic through the reliable, high-speed wired pipeline. (Verify the "Ethernet" tab appears in Settings). Trigger iCloud Token Refresh: Go to Settings -> [Your Name] -> iCloud -> Saved to iCloud and toggle off/on a major asset like iCloud Drive or Messages. This forces Spotlight to dump the stalled identifiers cache. Force Restart on Cable: Perform a Hard Reset (Vol Up, Vol Down, hold Power) while the Ethernet cable is connected. Let it Cook: Keep Low Power Mode OFF, lock the screen, and leave it alone. How to verify it's definitively done: Keep your Mac Console open and filter for completeness. Gdy to się stanie, zobaczysz ostateczny dziennik świętego Graala: fetchPipelineCompleteness: display=hidden days=3 procent=1.000000 Status the ` spent=1.000000 oznacza solidne 100% uzupełnienia. Zaraz po tym, 「dasd bezpiecznie zabije procesy worker (client process exited /connection invalid), telefon w końcu ostygnie, a żywotność baterii wróci do normy. Zaraz potem wyrzuciłem potoki glp i wszystko jest teraz masłem! Mam nadzieję, że pomoże to każdemu, kto utknął w 88% otchłani!
0
0
653
Jul ’26
NLLanguageRecognizer does not return Malay
I’m testing NLLanguageRecognizer to distinguish Malay (Bahasa Melayu) from English using this Malay text: Buka App Muzik untuk Mendayakan Widget Without language hints, my sandbox produces the following results. Percentages are rounded for display: Constraints Returned scores ID, EN ID: 100%; EN: 0% MS, EN EN: 100%; MS: not returned VI, EN VI: 98.7%; EN: 1.3% ID, MS, VI, EN ID: 100%; MS: not returned; EN and VI: 0% I also tried supplying languageHints to favor Malay and changed my phone’s system language to Malay. Neither resolved the issue: the recognizer still failed to identify the text as Malay. Here is a minimal example with hints: import NaturalLanguage let recognizer = NLLanguageRecognizer() recognizer.languageConstraints = [.malay, .english] recognizer.languageHints = [.malay: 0.8, .english: 0.2] recognizer.processString("Sila isi borang di bawah") print(recognizer.dominantLanguage?.rawValue ?? "nil") for (language, score) in recognizer.languageHypotheses(withMaximum: 2) { print(language.rawValue, score) } An independent native macOS test with the same short text also returned English with score 1.0, both with hints disabled and with the 0.80 Malay hint. To investigate whether this was only short-text ambiguity, I separately tested a longer Malay paragraph taken from local news on macOS: Terdahulu, peguam PN, Ivanpal Singh Grewal, berhujah bahawa penangguhan wajar diberikan memandangkan keadaan luar biasa susulan PRN Melaka pada 14 Nov, sambil berkata penggantungan itu menghalang gabungan berkenaan daripada menyertai proses demokrasi. Without hints, that paragraph produced: Constraints Result None Indonesian (id), score 1.0 English and Malay English (en), score 1.0; no Malay hypothesis Malay only dominantLanguage is nil; hypotheses are empty English, Malay, Indonesian Indonesian (id), score 1.0 I understand that Malay and Indonesian share vocabulary, short strings can be ambiguous, and constraints can force a prediction into an allowed language. However, Malay appears absent even when explicitly allowed and favored. My questions are: Is Malay identification supported by NLLanguageRecognizer on iOS and macOS? Could the missing Malay hypothesis and empty Malay-only result indicate a model limitation or framework defect? Is there an API to check which languages are available for recognition on the current runtime? Are any additional settings or language resources required? Environment iPhone model / iOS version: iPhone 13, iOS 27.2 Phone system language: tested with English and Malay Independent Mac reproduction: macOS 26.7.1 (25G241), Apple Silicon, Xcode 27.2 Beta 3
Replies
1
Boosts
0
Views
1.3k
Activity
1w
Best approach for automatically processing bank receipt images on iOS using on-device OCR?
What is the best approach to automatically process bank receipt images on iOS while respecting user privacy? I'm exploring ways to automatically extract transaction details from receipt images using on-device OCR. I'd like to discuss which Apple APIs and permission models would be appropriate for this functionality.
Replies
1
Boosts
0
Views
1.2k
Activity
1w
Vision face landmarks shifted on iOS 26 but correct on iOS 18 with same code and image
I'm using Vision framework (DetectFaceLandmarksRequest) with the same code and the same test image to detect face landmarks. On iOS 18 everything works as expected: detected face landmarks align with the face correctly. But when I run the same code on devices with iOS 26, the landmark coordinates are outside the [0,1] range, which indicates they are out of face bounds. Fun fact: the old VNDetectFaceLandmarksRequest API works very well without encountering this issue How I get face landmarks: private let faceRectangleRequest = DetectFaceRectanglesRequest(.revision3) private var faceLandmarksRequest = DetectFaceLandmarksRequest(.revision3) func detectFaces(in ciImage: CIImage) async throws -> FaceTrackingResult { let faces = try await faceRectangleRequest.perform(on: ciImage) faceLandmarksRequest.inputFaceObservations = faces let landmarksResults = try await faceLandmarksRequest.perform(on: ciImage) ... } How I show face landmarks in SwiftUI View: private func convert( point: NormalizedPoint, faceBoundingBox: NormalizedRect, imageSize: CGSize ) -> CGPoint { let point = point.toImageCoordinates( from: faceBoundingBox, imageSize: imageSize, origin: .upperLeft ) return point } At the same time, it works as expected and gives me the correct results: region is FaceObservation.Landmarks2D.Region let points: [CGPoint] = region.pointsInImageCoordinates( imageSize, origin: .upperLeft ) After that, I found that the landmarks are normalized relative to the unalignedBoundingBox. However, I can’t access it in code. Still, using these values for the bounding box works correctly. Things I've already tried: Same image input Tested multiple devices on iOS 26.2 -> always wrong. Tested multiple devices on iOS 18.7.1 -> always correct. Environment: macOS 26.2 Xcode 26.2 (17C52) Real devices, not simulator Face Landmarks iOS 18 Face Landmarks iOS 26
Replies
1
Boosts
0
Views
864
Activity
1w
iOS 27 + RecognizedText misses lines
I'm using the iOS 18+ RecognizedTextRequest, and with no change of code, iOS 27 gives poorer results: lines are dropped, all the text is not recognized it's not my code, because Live Text has the same issue and of course, back to 26 or 18 solves the issue What changed? Only interesting infos I could isolate: The simulator has not the issue Squashing the image horizontally (80% of its width for example) makes the detection significantly better
Replies
6
Boosts
5
Views
1.8k
Activity
2w
Major iOS 27 Regression in '.accurate' Model for VNRecognizeTextRequest
Hello, With the update from iOS 26 to iOS 27 I have noticed significant regressions and degraded performance when using the Vision Framework for OCR via 'VNRecognizeTextRequest'. The degraded performance is only applicable to the .accurate model, the .fast model is identical between iOS 26 and iOS 27. I have an app that uses this framework extensively for extracting information from the video feed of professional cinema cameras. This runs using en-US, language correction off with a specific region of interest. On iOS 26 everything works reliably as it has for the past few years, with the exact same codebase on iOS 27 many strings that were previously detected are no longer detected, some strings return as if they've been read upside down (ISO1280 is returned as 08ZLOSI) and there are a lot of Cyrillic look alikes mixed into Latin text, there also appears to be a lot more junk characters tacked onto the end of numbers ('12ł' is returned when previously it was just '12'). You often see whole line duplicates too where the same text is recognised twice, once as a single line with all strings combined and once as individual strings. The iOS 27 Simulator doesn't show any of this. I only saw it on real iPads, if you only test in the Simulator you'll miss it. To put numbers on it, I built a test set of about 26,500 frame grabs and ran them through 'VNRecognizeTextRequest' on iPadOS 18, 26 and 27. With the version of my app that's on the App Store today, iOS 26 reads 99.7% of values correctly. On iOS 27 that drops to 65%. I've resolved this issue now and have been able to retain the level of performance that existed on iOS 26, this involves various preprocessing steps to the images as well as a significant amount of validation after the Vision Framework returns its results. These fixes took around 3.5 Days of work which is incredibly frustrating. I would much rather use my time building new features instead of fixing regressions introduced by poorly validated new models that have been shipped in iOS 27. What frustrates me most is this change wasn't announced anywhere and there's no way to opt out. The revision number didn't change so there's no way to keep using the model that worked. If this had shipped as VNRecognizeTextRequestRevision4 apps could have stayed on VNRecognizeTextRequestRevision3 until they'd had a chance to test. Or permanently stayed on 'VNRecognizeTextRequestRevision3' as obviously the new version performs significantly worse. Instead an app that worked fine on iOS 26 now gets wrong text back on iOS 27 with no error, no warning and nothing in the release notes. For a framework people rely on for real work that's a significant regression. It seems that other developers have also come across this issue: https://developer.apple.com/forums/thread/845452 For technical reference, if you diff System/Library/PrivateFrameworks/TextRecognition.framework between iOS 26.5 (23F77) and iOS 27.0 (24A434) you uncover a number of changes. New models on iOS 27 seem to use an '_e5' variant. The text detector in iOS 26 references training run 'yyzn76u6fs' with 88560 iterations, iOS 27 references training run 'hvcthxumvz' with only 10116 iterations. Perhaps an 8x reduction in iterations has something to do with the poor performance in iOS 27? Further, the orientation model changes from iOS 26 'cr_orientation_model_v1' with 1858472 iterations vs. iOS 27 'cr_orientation_model_v1_e5' with 671065 iterations. The Latin and Cyrillic Recogniser changed model entirely from 'cr_tr_model_latincyrillic_v3' on iOS 26 to 'cr_tr_model_latincyrillic_v3_e5' on iOS 27. It seems these 'e5' models are optimised for the neural engine. Is there any way to get the iOS 26 models/behaviour back or plans to back deploy the old model in iOS 27.1 or iOS 27.2? Can future model changes like this come with a new revision number? What testing process does the Vision Team use to validate models as clearly something was missed here? Thanks!
Replies
0
Boosts
0
Views
323
Activity
2w
Framing an object in real time
Hi, I am currently working on a project which involves giving directions to the user telling them what they should do to frame a given object (also only doing this if sees the object which can be many things, so it needs detection too) so that it can be optimally in view of the camera to be classified. I have figured the classification, but not the real time guidance and detection. I tried training a model that featured all of the different movements ("move up", "move down", etc.). It featured 30 different objects with ten crops per image, and each class featured all 30 images and an even amount per class. It had over 1500 images in total, and I trained using createml and core ml. But, they were very inaccurate in my testing. I also tried using foundation models for this, but it also didn't work. I am quite new to integrating ai/ ml models into a project, and I am unsure of where to go next. If anyone has any suggestions, I would be really appreciative.
Replies
0
Boosts
0
Views
398
Activity
2w
iOS 27: English sentence embedding model unavailable on physical devices
I’m seeing NLEmbedding.sentenceEmbedding(for: .english) return nil on physical devices running iOS 27, both in debug builds under Xcode and in production. The same failure has been observed across thousands of users on iOS 27. One affected device runs iOS 27.0.1 (24A446). Minimal example with diagnostics: import NaturalLanguage let language = NLLanguage.english print("Current revision:", NLEmbedding.currentSentenceEmbeddingRevision(for: language)) print("Supported revisions:", Array(NLEmbedding.supportedSentenceEmbeddingRevisions(for: language))) if let model = NLEmbedding.sentenceEmbedding(for: language) { print("Loaded revision:", model.revision, "dimension:", model.dimension) } else { print("Sentence embedding model unavailable") } On affected devices, the current revision is 1 and supported revisions are [1], but model loading returns nil. The framework logs: Unable to locate Asset for sentence embedding model for local en. The failure occurs during model loading, before any call to vector(for:). This uses Apple’s built-in English sentence model, with no custom embedding file. Is this a known issue on iOS 27? Is there a supported way to request or restore the missing sentence-model asset, or a recommended recovery approach?
Replies
0
Boosts
0
Views
528
Activity
3w
Crashed: AXSpeech EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x000056f023efbeb0
Application is getting Crashed: AXSpeech EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x000056f023efbeb0 Crashed: AXSpeech 0 libobjc.A.dylib 0x4820 objc_msgSend + 32 1 libsystem_trace.dylib 0x6c34 _os_log_fmt_flatten_object + 116 2 libsystem_trace.dylib 0x5344 _os_log_impl_flatten_and_send + 1884 3 libsystem_trace.dylib 0x4bd0 _os_log + 152 4 libsystem_trace.dylib 0x9c48 _os_log_error_impl + 24 5 TextToSpeech 0xd0a8c _pcre2_xclass_8 6 TextToSpeech 0x3bc04 TTSSpeechUnitTestingMode 7 TextToSpeech 0x3f128 TTSSpeechUnitTestingMode 8 AXCoreUtilities 0xad38 -[NSArray(AXExtras) ax_flatMappedArrayUsingBlock:] + 204 9 TextToSpeech 0x3eb18 TTSSpeechUnitTestingMode 10 TextToSpeech 0x3c948 TTSSpeechUnitTestingMode 11 TextToSpeech 0x48824 AXAVSpeechSynthesisVoiceFromTTSSpeechVoice 12 TextToSpeech 0x49804 AXAVSpeechSynthesisVoiceFromTTSSpeechVoice 13 Foundation 0xf6064 __NSThreadPerformPerform + 264 14 CoreFoundation 0x37acc CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION + 28 15 CoreFoundation 0x36d48 __CFRunLoopDoSource0 + 176 16 CoreFoundation 0x354fc __CFRunLoopDoSources0 + 244 17 CoreFoundation 0x34238 __CFRunLoopRun + 828 18 CoreFoundation 0x33e18 CFRunLoopRunSpecific + 608 19 Foundation 0x2d4cc -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 212 20 TextToSpeech 0x24b88 TTSCFAttributedStringCreateStringByBracketingAttributeWithString 21 Foundation 0xb3154 NSThread__start + 732 com.livingMedia.AajTakiPhone_issue_3ceba855a8ad2d1af83655803dc13f70_crash_session_9081fa41ced440ae9a57c22cb432f312_DNE_0_v2_stacktrace.txt 22 libsystem_pthread.dylib 0x24d4 _pthread_start + 136 23 libsystem_pthread.dylib 0x1a10 thread_start + 8
Replies
5
Boosts
1
Views
2.7k
Activity
3w
Crakling with iOS 27
I developed a conversational agent using Gradium and it worked fine using the iPhone's built-in speaker on iOS 18.7 and iOS 26.6. I developed similar conversational agent using the Gemini stack and it also worked fine using the iPhone's built-in speaker on iOS 18.7 and iOS 26.6. Now on iOS 27.0, we hear crackling in the transcripts and responses, with both stacks.
Replies
1
Boosts
1
Views
135
Activity
Sep ’26
SwiftPlaygroundsでドローンTelloアプリを制作
皆さん、はじめまして。プログラミングに対する知識がないので、AIを使いながらiPadでTelloのドローンを操縦できるアプリを作っています。このアプリはシミュレーション飛行とドローンの実機飛行ができるアプリにしたいのですが、ドローンの実機飛行を行うためのWi‐Fi接続ができません。(iPadでのシミュレーション飛行が可能です)iPadの基本的な設定はAppleに教えていただき、設定変更を行いましたが、どうしても接続できないので、わかる方法を教えていただけると嬉しいです。 コードを添付いたしますので、ご教示ください。 ドローンアプリコード
Replies
0
Boosts
0
Views
736
Activity
Aug ’26
Group AppIntents’ Searchable DynamicOptionsProvider in Sections
I’m trying to group my EntityPropertyQuery selection into sections as well as making it searchable. I know that the EntityStringQuery is used to perform the text search via entities(matching string: String). That works well enough and results in this modal: Though, when I’m using a DynamicOptionsProvider to section my EntityPropertyQuery, it doesn’t allow for searching anymore and simply opens the sectioned list in a menu like so: How can I combine both? I’ve seen it in other apps, but can’t figure out why my code doesn’t allow to section the results and make it searchable? Any ideas? My code (simplified) struct MyIntent: AppIntent { @Parameter(title: "Meter"), optionsProvider: MyOptionsProvider()) var meter: MyIntentEntity? // … struct MyOptionsProvider: DynamicOptionsProvider { func results() async throws -> ItemCollection<MyIntentEntity> { // Get All Data let allData = try IntentsDataHandler.shared.getEntities() // Create Arrays for Sections let fooEntities = allData.filter { $0.type == .foo } let barEntities = allData.filter { $0.type == .bar } return ItemCollection(sections: [ ItemSection("Foo", items: fooEntities), ItemSection("Bar", items: barEntities) ]) } } struct MeterIntentQuery: EntityStringQuery { // entities(for identifiers: [UUID]) and suggestedEntities() functions func entities(matching string: String) async throws -> [MyIntentEntity] { // Fetch All Data let allData = try IntentsDataHandler.shared.getEntities() // Filter Data by String let matchingData = allData.filter { data in return data.title.localizedCaseInsensitiveContains(string)) } return matchingData } }
Replies
2
Boosts
2
Views
1.6k
Activity
Aug ’26
M3 Neural Engine kernelDMA bandwidth throttled at 1 MiB multiples; splitting restores 3x DRAM bandwidth
A suspected RTL performance erratum in the Apple Neural Engine throttles DRAM weight streaming throughput down to 17–19 GB/s from the nominal 45–60 GB/s, whenever the total weight size is an integer multiple of 1 MiB. This was confirmed on M3 Macbook Air (2024) 24GB. 1 MiB is a common transfer size, and currently affects 7 of ANEMLL’s 15 baseline models. Avoiding the suspected problematic path in the kernel DMA's prefetch ring by splitting 1 MiB transfers into non-multiples of 1 MiB, directly increased Llama 3.2 1B token throughput from 10.0 to 24.3 tokens/s (DRAM usage from 24.7 to 60.0 GB/s), and Qwen3-8B from 1.36 to 2.97 tokens/s (DRAM usage from 22.4 to 48.7 GB/s). Please see writeup here: https://eiln.github.io/posts/ane-dma.html I'm looking to confirm that this behavior is reproducible on your side, and will provide the scripts in my plots.
Replies
0
Boosts
0
Views
484
Activity
Aug ’26
MLXVLM factory fallback doesn't catch a vision_config-present-but-no-tower-weights checkpoint (Qwen3.5-9B)
I ran into a gap in the MLXVLM → MLXLLM factory fallback and wanted to check whether this is expected behavior or something worth filing. Context: I maintain a Swift-native macOS agent built on MLX Swift, with a hardware-adaptive catalog of local models. I tried adding mlx-community/Qwen3.5-9B-4bit as a plain text model. Its config.json ships a full vision_config block — it looks like a multimodal config — but the actual checkpoint has no vision tower weights at all. It's genuinely text-only; the metadata just doesn't reflect that. ModelFactoryRegistry tries MLXVLM before falling back to MLXLLM. I expected that to fail cleanly and fall through, since there's no vision encoder to load. It doesn't. MLXVLM's Qwen35Configuration declares visionConfiguration as a non-optional property mapped to vision_config. Since the key is present in the raw config, decoding succeeds — there's nothing to catch at that stage. So MLXVLM proceeds to actually load the model as a VLM, and it crashes later, inside WiredMemoryUtils.tune(), because getRopeIndex() expects real vision-encoder dimensions to compute against, and there are none. Not a clean decode-time error I could catch — a runtime crash several layers past config parsing, in the RoPE indexing math. The workaround I ended up with is config-level surgery before the model reaches the factory at all: strip vision_config (and image_token_id / video_token_id) from the JSON first. With the key genuinely absent, Qwen35Configuration's decode throws DecodingError.keyNotFound as the fallback logic presumably intends, MLXVLM fails cleanly at that stage, and ModelFactoryRegistry falls through to MLXLLM correctly. That works, but it feels like a workaround rather than the intended path — the factory fallback seems to assume any failure will happen at decode time, and a config key that's present but semantically empty (no real vision weights behind it) skips that safety net entirely. Questions: Is there a more canonical/official way to detect whether a checkpoint actually has real vision tower weights, short of opening model.safetensors.index.json and counting vision_tower.* tensor names myself? (I've done exactly that for a different, genuinely multimodal Qwen VL checkpoint — 333 real vision tensors there vs. zero here — but it feels like something the loading path should be able to tell me.) Is the vision_config-present-but-empty pattern something the mlx-community conversion pipeline is aware of, or is this specific to how Qwen3.5-9B-4bit happened to get converted? Happy to share more logs/repro details if useful.
Replies
0
Boosts
0
Views
398
Activity
Aug ’26
How can a local AI agent use MLX/Metal unattended on macOS while remaining confined to an authorized workspace?
How can a local AI agent use MLX/Metal unattended while remaining confined to an authorized workspace? I am developing an AI-driven local media-processing workflow on an Apple-silicon Mac and am trying to understand the correct architecture for allowing it to run unattended without giving the AI agent unrestricted access to my primary personal computer. I am not a software engineer, so I may be missing an established macOS mechanism or using the wrong terminology. I would appreciate guidance from people familiar with MLX, Metal, sandboxing, and macOS security. What I am building I use OpenAI Codex as the local execution/software-development agent. The working system currently: ingests and verifies original video and still media while preserving immutable originals; performs visual semantic analysis and divides video into meaningful time-coded segments; separately analyzes spoken language rather than assuming audio and video are semantically equivalent; uses MLX Whisper locally on Apple silicon for time-coded speech transcription; stores visual and language semantics in a relational SQLite media catalog. These five stages are working. My current test corpus contains 148 original media files, 126 visual semantic segments, and 765 speech segments. The next stages are AI editorial construction from the semantic database and generation of instructions/scripts for a DaVinci Resolve rough cut. The security architecture I want Codex to operate autonomously within a deliberately bounded development environment. I do not want to solve this simply by granting an autonomous agent Full Disk Access to my primary personal Mac. The concern is ordinary fault containment. Codex generates and executes scripts, invokes applications and command-line tools, and manipulates files. A mistaken path or defective generated script should not have unrestricted consequences for the rest of my computer. I therefore separated AI execution from ordinary personal files. Codex is configured for Workspace Write access with explicitly authorized project roots. Canonical media resides on a separately authorized external SSD, and temporary AI working artifacts are kept separately. Ordinary Python and FFmpeg operations now run autonomously within these authorized areas. The problem The difficulty appears when the workflow invokes capabilities that cannot operate inside the ordinary Codex sandbox. The clearest example is MLX Whisper. I am using: MLX Whisper 0.4.3 mlx-community/whisper-small-mlx Apple silicon local transcription MLX Whisper works successfully and its transcription quality is sufficient for my semantic-retrieval application. However, MLX could not access Apple Metal/GPU execution from inside the ordinary Codex sandbox. Codex therefore requested permission to execute the transcription operation outside the sandbox. Once approved, MLX/Metal worked and the entire corpus was successfully transcribed. The processing therefore works, but the workflow cannot run genuinely unattended. A future operation should be able to run: new media → integrity verification → visual semantic analysis → MLX Whisper transcription → language semantic analysis → SQLite update → QA But if execution stops midway waiting for a human to click Allow, the pipeline is not operationally autonomous. What I have already tried I initially encountered permission problems even with ordinary file operations. I therefore: separated Codex work from ordinary personal documents; created dedicated project/work areas; explicitly authorized the required working roots; configured Workspace Write; separately authorized the external media repository; tested shell/Python and FFmpeg operations within those boundaries. Those changes worked. Routine Python and FFmpeg operations now run without approval prompts. The remaining issue occurs with MLX/Metal and some other application/runtime operations that require sandbox escalation. My question Is there a supported architecture for allowing a local AI agent to invoke MLX/Metal and other deliberately authorized development tools unattended, while still confining the agent to defined project/workspace boundaries rather than granting unrestricted access to the entire Mac? For example, should I be investigating: App Sandbox entitlements; a signed helper tool or XPC service; security-scoped resources; a dedicated executable with appropriate entitlements; a different method of launching MLX/Metal; or another macOS mechanism? In particular, can Metal/GPU access coexist with persistent bounded filesystem access without requiring interactive approval each time the AI invokes it? I am also unsure which security layer is actually responsible here: the Codex sandbox, macOS App Sandbox, TCC, executable/code-signing rules, Metal restrictions, or some interaction among them. If this kind of bounded unattended execution is intentionally not supported, that would also be useful to know. My alternative would be a dedicated Apple-silicon Mac containing only the AI-development environment and replaceable project data, where broader permissions would have a much smaller failure domain. I can provide the Codex configuration, exact successful and failing commands, directory/root configuration, macOS/hardware information, and sandbox diagnostics. I would particularly appreciate guidance on which security layer is causing the MLX/Metal escalation and what the supported architecture would be for this use case. Thank you.
Replies
0
Boosts
0
Views
852
Activity
Aug ’26
Fused Metal Kernels for Linear Recurrences in MLX
I’ve been developing mlx-recurrence, a plug-in framework of fused Metal GPU kernels for linear recurrences on Apple silicon—roughly analogous to flash linear attention for MLX. Sequential recurrences are difficult for MLX to fuse automatically. Architectures such as state-space models, gated linear attention, and diagonal RNNs ordinarily require a loop across the sequence length. When that loop is implemented in Python, a sequence of length L can require L separate Python-to-Metal dispatches. These kernels instead execute the entire recurrence in a single Metal dispatch. The training path uses segment checkpointing with recomputation during the backward pass. In validated M3 Max tests, the checkpoint-and-recompute kernels reduced peak recurrent-state memory by approximately 12–18× at the kernel level and lowered total training peak memory from 23.88 GB to 10.34 GB. At the same batch size, end-to-end training throughput improved by roughly 1.4×, while individual fused forward-and-backward kernels ran approximately 1.5–1.9× faster than the original full-state implementations. Results will vary with recurrence type, sequence length, state dimensions, batch size, datatype, model architecture, and hardware. Current kernels: ssd_scan Mamba-2-style, head-wise SSD selective scan. Intended for Mamba-2 and other SSM hybrid architectures. State shape: [B, H, Dh, N] gla_scan Gated Linear Attention with a scalar forget gate and outer-product write. Intended for GLA and linear-attention hybrid architectures. State shape: [B, H, Dh, Dh] rglru_scan RG-LRU diagonal recurrence. Intended for Griffin and RecurrentGemma-style architectures. State shape: [B, D] rotlru_scan Rotational LRU using a complex-diagonal recurrence, a magnitude gate, and a per-step rotation of two-dimensional channel pairs. Intended for complex-LRU and S4-style oscillatory memory architectures. State shape: [B, D], represented as interleaved channel pairs. Each kernel is implemented as a self-contained plug-in on a shared chassis located at: mlx_recurrence._chassis The chassis provides: Segment checkpoint-and-recompute infrastructure Shape and argument validation VJP integration Forward and gradient parity-test helpers Common recurrence plug-in handling Adding another recurrence therefore requires implementing its Metal forward and backward source pair and connecting its VJP. The checkpointing, validation, and testing infrastructure does not need to be rebuilt for each operator. The original version 0.1 kernels remain available under: mlx_recurrence.legacy They are also re-exported at the package’s top level for backward compatibility. I’m interested in feedback from developers working with MLX or custom Metal compute kernels, particularly around: Preferred APIs for packaging reusable MLX Metal extensions Threadgroup and memory-layout strategies across Apple GPU generations Numerical stability expectations for long recurrent sequences Benchmarking fused scans against MLX-native implementations Additional recurrent operators that would be valuable to support I would also be interested to know whether others are developing similar fused recurrence primitives for MLX and whether a shared interface for these operations would be useful. My setup is a M3 MAX Macbook Pro with 36GB Ram and I am running on macOS 26.4.1 (25E253). https://github.com/D-CSIL/mlx-recurrence
Replies
0
Boosts
0
Views
301
Activity
Aug ’26
Will Siri AI be able to copy text content of whats on screen in Notes/Files/Photos?
Let's say I'm in Notes, or in Pages, or in Files or Live Text if I'm in Photos... will Siri AI be able to COPY the text for me if I ask it to do so? If you have new Siri AI installed, or are DTS Engineer at Apple I'd appreciate a yes/no? Critical for text editing, working with AI output, and general modern work requirements. Presently I have to press the "share" button then select Copy from the Share Sheet, OR in Pages I have to select EXPORT from the More Menu and choose Plain Text to get the contents. Thank you. Be well.
Replies
0
Boosts
0
Views
694
Activity
Aug ’26
Questions for Apple Support / Apple Vision Team
Dear Apple Support, I would like to report a long-standing issue affecting Khmer text recognition in Live Text (Vision Framework/OCR). Based on my testing, this issue has persisted for more than two years, from iOS 17 through iOS 27 Beta 3, and is also reproducible on iPadOS and macOS. I would appreciate clarification on the following questions: Is Apple aware of an issue where Live Text (OCR/Text Recognition) incorrectly recognizes Khmer script as Thai script, causing copied text to become Thai characters instead of Khmer? Has this issue been officially logged as a bug within the Vision Framework or Live Text team? Since this behavior has remained reproducible from iOS 17 to iOS 27 Beta 3, why has it not yet been resolved? Is the problem caused by: automatic language detection, the OCR recognition model, the Vision Framework, or another component of Apple's AI pipeline? Does Apple currently have a dedicated OCR and language recognition model for the Khmer script, or is Khmer being inferred through another language model? Is there an estimated timeline for improving Khmer OCR and preventing Khmer text from being misidentified as Thai? Can Apple confirm whether this issue affects all products using Vision Framework, including: Live Text Photos Preview Screenshot OCR APIs provided to third-party developers? How can Apple work with the Khmer technology community to improve OCR accuracy and language support for Khmer? This issue is more than a simple OCR bug. When Khmer text is automatically converted into Thai characters, users lose access to the original text, developers receive incorrect OCR output, and it negatively impacts the digital representation of the Khmer language. For reference, I have documented the issue in detail here: https://app.notion.com/p/Inaccurate-OCR-Language-Inference-Khmer-Script-Misidentified-as-Thai-in-Vision-Framework-2d8a24f4ee6680fcbc49d989f8bb606f I hope Apple can investigate this issue and prioritize improving Khmer language support across Vision Framework and Live Text. Thank you.
Replies
7
Boosts
19
Views
1.5k
Activity
Jul ’26
AI framework usage without user session
We are evaluating various AI frameworks to use within our code, and are hoping to use some of the build-in frameworks in macOS including CoreML and Vision. However, we need to use these frameworks in a background process (system extension) that has no user session attached to it. (To be pedantic, we'll be using an XPC service that is spawned by the system extension, but neither would have an associated user session). Saying the daemon-safe frameworks list has not been updated in a while is an understatement, but it's all we have to go on. CoreGraphics isn't even listed--back then it part of ApplicationServices (I think?) and ApplicationServices is a no go. Vision does use CoreGraphics symbols and data types so I have doubts. We do have a POC that uses both frameworks and they seem to function fine but obviously having something official is better. Any Apple engineers that can comment on this?
Replies
10
Boosts
0
Views
2.7k
Activity
Jul ’26
[iOS 27 DB3] Apple Intelligence and Spotlight Stuck at 85-90% - How to force completion via Ethernet and Console logs analysis
Hey everyone, If your iPhone is stuck on "Optimizing Search and Siri" / Indexing at around 85-90% and hard resets or leaving it on wireless charging overnight isn't moving the needle, I found a definitive way to kickstart the daemon and force it to 100%. The Root Cause (Analyzed via macOS Console): By plugging the iPhone into a Mac and checking the Console logs, I noticed that spotlightknowledged and biomed get into an endless loop around Resolved entitled set identifiers to enumerate data resources. The system stalls on verifying developer entitlements and Apple Intelligence security tokens for specific app databases over cellular networks or unstable Wi-Fi. The dasd daemon eventually puts the pipeline into a hidden sleep state to protect the battery, making it look like it's doing nothing. The Solution that forced it to 100%: Use a Wired Connection (Ethernet Adapter): Connect your iPhone directly to your fiber optic router/modem using an RJ-45 Ethernet to USB-C adapter. iOS treats a wired Ethernet connection with the highest background priority. Turn off Cellular/Wi-Fi: Go to Control Center and disable Cellular Data and Wi-Fi entirely to force iOS to route 100% of traffic through the reliable, high-speed wired pipeline. (Verify the "Ethernet" tab appears in Settings). Trigger iCloud Token Refresh: Go to Settings -> [Your Name] -> iCloud -> Saved to iCloud and toggle off/on a major asset like iCloud Drive or Messages. This forces Spotlight to dump the stalled identifiers cache. Force Restart on Cable: Perform a Hard Reset (Vol Up, Vol Down, hold Power) while the Ethernet cable is connected. Let it Cook: Keep Low Power Mode OFF, lock the screen, and leave it alone. How to verify it's definitively done: Keep your Mac Console open and filter for completeness. Gdy to się stanie, zobaczysz ostateczny dziennik świętego Graala: fetchPipelineCompleteness: display=hidden days=3 procent=1.000000 Status the ` spent=1.000000 oznacza solidne 100% uzupełnienia. Zaraz po tym, 「dasd bezpiecznie zabije procesy worker (client process exited /connection invalid), telefon w końcu ostygnie, a żywotność baterii wróci do normy. Zaraz potem wyrzuciłem potoki glp i wszystko jest teraz masłem! Mam nadzieję, że pomoże to każdemu, kto utknął w 88% otchłani!
Replies
0
Boosts
0
Views
653
Activity
Jul ’26