Root cause confirmed. I believe this is a Core ML bug, and I have a one-row fix verified end-to-end on the production model.
Following up on my previous reply: I have isolated the cause and worked around it, applied the change to the real 1.8B model, and measured the result. Posting it here in case it saves someone else the bisection.
1. The remedy: make the KV state one row deeper than the region the decode graph writes.
The failing graph has a state with N rows on the position axis and a decode step whose state write covers all N of them. Allocating the state with N + 1 rows while leaving the write window at [0, N) makes the ANE specialization succeed. The extra row is never read and never written; it exists only so the write window is no longer full-extent.
The boundary is exact, from a controlled sweep that changed nothing but the state depth:
state rows
write window
result
256
[0, 256) — full extent
fails, E5RT (11)
257
[0, 256)
loads, ANE program built
258, 264, 288, 320, 512
[0, 256)
loads, ANE program built
One row is enough, and zero rows is not. Where the window starts does not matter as long as it is not the whole dimension: [1, 257) in a 257-row state and [16, 272) / [32, 288) in a 288-row state all build.
I am reporting this as a bug, not as a modelling error. The package that fails and the package that succeeds are the same graph: identical operations, identical shapes for every input and output, the identical write window, identical numerics. The sole difference is that an inert buffer is exactly as long as the region being written. coremlcompiler compiles both without complaint, neither uses anything outside the documented MIL operation set, and no public API constrains or documents the relationship between a state's extent and the extent of the writes into it. The rejection happens inside ANE specialization as an internal C++ exception, for a property of the graph that the supported tooling gives a developer no way to observe. That is a defect in the compiler, not something the model is doing wrong.
2. It works on the real model, not just the reproducer.
The model is a 1.8B-parameter translation model — 32 layers, hidden 2048, 4 KV heads, context 1024, 4-bit lookup-table (mixed-precision) weights, exported as a multifunction ML Program with infer and prefill sharing one MLState, ~1.7 GB. Same architecture and same quantization as the build that fails; the only change is the one extra state row in both functions.
The decode function now specializes for the Neural Engine. MLComputePlan reports 6756 operations for infer, of which 2659 prefer MLNeuralEngineComputeDevice, 9 are CPU (mask arithmetic: select, greater_equal, gather, cast), and 0 are GPU. The E5RT program descriptor confirms it from the other side: it names infer_ane, infer_ane0 and per-operation *_ane symbols, where the same function loaded with .cpuOnly names none.
Throughput: 20.4–20.6 ms/step (~48–49 tok/s) with .cpuAndNeuralEngine, against 38.5–39.1 ms/step (~26 tok/s) with .cpuOnly — a 1.9x difference. .cpuAndNeuralEngine has no GPU in its plan, so that gap can only be the Neural Engine. For reference, the same graph ran at 46.8 tok/s on macOS 26 before the regression, so this restores the previous behaviour rather than approximating it.
Outputs are unchanged. Four prompts, greedy, compared token-for-token against the unmodified package under .cpuOnly: identical sequences on all four. The extra row is inert by construction.
Cost: one row of KV state. For this model that is [64, 4, 1025, 128] instead of [64, 4, 1024, 128], i.e. +0.098% state memory, and no additional per-step work.
The unmodified package is the control and it fails exactly as reported: its decode function raises the same E5RT (11) under .cpuAndNeuralEngine, while its prefill function — unchanged in this respect — specializes for the ANE in the same session. The regression was always decode-only, and the pad fixes precisely the function that was broken.
3. A second, quieter failure mode worth knowing about.
Some state-write shapes do not crash at all. Two slice_updates writing into one state, or a write whose begin is data-dependent, load successfully — MLModel returns, predict() runs and produces output — but Core ML silently declines the Neural Engine and executes on the CPU/GPU. No error, no warning, nothing in the log. If you try one of those as a workaround, LOAD OK tells you nothing about where the model is running.
That is also why I would not describe .all as a workaround. It loads, but the decode graph lands on the GPU — 2529 of 6500 operations in the shipping package — and measures 37.9 ms/step, indistinguishable from .cpuAndGPU.
4. Two corrections to my previous reply.
The "anchoring" reading was wrong. I had inferred from a confounded experiment that a state write had to be origin-anchored. A single write starting at a non-zero row builds fine; that experiment had two writes into one state, which is a separate failure mode in its own right. The anchoring rule as I stated it was an artifact of not separating the two variables.
A methodology caution for anyone repeating this: do not use the presence of an H16C.bundle/<function>/<function>_ane directory in the E5RT bundle cache as the test for whether the Neural Engine was used. For a multifunction package that check returns a false negative — the ANE programs are compiled through ane_compile/multiprocedure/ and no per-function _ane directory is created, so a model fully resident on the ANE looks identical to one on the CPU. MLComputePlan is the reliable check. This cost me a wrong conclusion for a while.
5. What I would still like from Apple.
The E5RT diagnostic is still redacted: E5RT encountered an STL exception. msg = <private>. followed by E5RT: <private> (11). Nothing in the load path or the error points at the state extent, and the public error (functionName must be nil... / doesn't support the multi-function description syntax) points somewhere else entirely. Two concrete asks:
Un-redact the E5RT exception text, or map it to a documented MLModelError case. A compiler that rejects this graph should say so.
Accept a full-extent state write. If that is not possible, document the constraint. The workaround is one extra row and costs 0.098% state memory — reasonable once you know it, undiscoverable if you do not. Nothing in the ANE compiler output, coremlcompiler, or MLComputePlan distinguishes a package that will fail specialization from one that will succeed; both compile cleanly and only the load reveals the difference.
