NSJSONSerialization silently drops U+FEFF from JSON string content — keys merge, characters vanish
TL;DR: NSJSONSerialization deletes U+FEFF (ZERO WIDTH NO-BREAK SPACE / BOM) from anywhere inside parsed JSON strings — not just a leading document BOM, and even when written as the \uFEFF escape (it's removed after unescaping). Distinct strings/keys silently collapse onto their U+FEFF-less twins. If you're seeing JSON keys mysteriously merge or a character disappear from a parsed value, this is probably why. It is not your code. Workaround and exhaustive scope below.
The workaround
Two options, depending on how attached you are to Foundation:
A. Stay on NSJSONSerialization — swap U+FEFF for a private-use sentinel before parsing, restore after. You must handle both the raw bytes and the \uFEFF escape (the escape bites too, since deletion happens post-unescape):
// 1. Pick a private-use scalar you've verified is absent from the source text.
// 2. Replace every in-content U+FEFF (raw char AND \uFEFF escape) with it.
// 3. Parse. NSJSONSerialization preserves the sentinel.
// 4. Recursively restore the sentinel -> U+FEFF in the parsed tree.
static id RestoreSentinel(id o, NSString *s, NSString *bom) {
if ([o isKindOfClass:NSString.class])
return [o rangeOfString:s].location == NSNotFound ? o
: [o stringByReplacingOccurrencesOfString:s withString:bom];
if ([o isKindOfClass:NSArray.class]) {
NSMutableArray *a = [NSMutableArray arrayWithCapacity:[o count]];
for (id e in o) [a addObject:RestoreSentinel(e, s, bom)];
return a;
}
if ([o isKindOfClass:NSDictionary.class]) {
NSMutableDictionary *d = [NSMutableDictionary dictionary];
[o enumerateKeysAndObjectsUsingBlock:^(id k, id v, BOOL *stop) {
d[RestoreSentinel(k, s, bom)] = RestoreSentinel(v, s, bom);
}];
return d;
}
return o;
}
Swap the escape form with a backslash-parity-aware regex so \uFEFF (escaped backslash + literal "uFEFF") is left intact:
(?<!\\)((?:\\\\)*)\\u[Ff][Ee][Ff][Ff] -> $1<sentinel>
B. Don't use Foundation for this file — a spec-compliant C parser like ++yyjson++ preserves U+FEFF and is faster on large files. (This is the route swift-transformers took for tokenizer.json.)
Minimal repro
// Object keys collapse:
NSData *d1 = [@"{\"\\uFEFF#\":1,\"#\":2}" dataUsingEncoding:NSUTF8StringEncoding];
id o1 = [NSJSONSerialization JSONObjectWithData:d1 options:0 error:nil];
// EXPECTED: 2 keys ("\uFEFF#" and "#"); ACTUAL: 1 key ("#") — \uFEFF stripped, keys merged
// String content lost:
NSData *d2 = [@"[\"\\uFEFF\"]" dataUsingEncoding:NSUTF8StringEncoding];
id o2 = [NSJSONSerialization JSONObjectWithData:d2 options:0 error:nil];
// EXPECTED: ["\uFEFF"] (one code point); ACTUAL: [""] (empty string)
Same outcome whether U+FEFF arrives as raw EF BB BF bytes or the \uFEFF escape.
Why this is a bug, not a quirk
Per RFC 8259 §7, a JSON string is a sequence of Unicode code points; U+FEFF is ordinary content and doesn't require escaping. Tolerating a leading document BOM is fine — deleting U+FEFF from string content is not. U+FEFF leads a double life (BOM signal vs. ZERO WIDTH NO-BREAK SPACE character); Foundation treats every occurrence as a stray BOM to scrub.
Scope — exhaustive, not anecdotal
I swept all 1,112,064 valid Unicode scalars (U+0000–U+10FFFF minus surrogates) through a parse round-trip, in both the \uFEFF-escape and raw-UTF-8 forms:
U+FEFF is the only scalar altered. Every other scalar round-trips byte-identically — including the other zero-widths (U+200B, U+2060, U+00A0), which all survive.
No Unicode normalization occurs (NFD stays decomposed, combining sequences and compatibility characters are preserved).
So this is a deliberate BOM-stripping heuristic applied too broadly to string content — narrow and fixable, not general mangling.
Why it's nasty in practice
U+FEFF is zero-width, so the corruption is invisible — no trace in a diff or editor. Real-world hit: ML tokenizer vocabularies (e.g. Google's Gemma) legitimately contain U+FEFF-bearing tokens; loading tokenizer.json via NSJSONSerialization collapses those keys and assigns wrong token IDs, with zero visible symptom until output is subtly wrong.
Filed as FB23271905 — please dupe if this has bitten you. More duplicates is what gets it triaged.
