<!-- meta: JSON validators catch syntax errors but miss duplicate keys, big integers, and BOM issues. See 8 real JSON mistakes and how to fix each one today. -->
<!-- slug: json-that-looks-fine-but-breaks-your-app -->
Your parser said valid. Production disagreed. Here is what the spec actually permits, what it quietly tolerates, and where a payload loses data without raising a single error.
The payload that passed validation
A nightly sync job pulls orders from a partner API, runs every response through a JSON validator, and writes the records into an internal warehouse. The validator returned green on each batch. The job ran for eleven days before anyone noticed that a handful of customers were seeing each other's order history.
Two separate defects were hiding in the same file. The partner's serializer emitted 64-bit order identifiers as raw numbers, and every identifier above 9,007,199,254,740,991 had been rounded on arrival. A second endpoint merged two config fragments and produced an object with the same key twice. The duplicate was syntactically legal. Both parsers involved accepted it. They simply disagreed about which value won.
Neither defect is exotic. Neither requires a malicious payload, a broken library, or a junior mistake. Both are ordinary consequences of treating "the JSON validator accepted it" as proof that the data is correct. The partner had run the same check for months, on the same payloads, and it had never once been wrong about the thing it was actually measuring.
That distinction is the whole article. Syntax validation answers a narrow question: does this text follow the grammar in RFC 8259? It says nothing about whether the values survived the trip, whether keys are unique, whether numbers fit in your runtime, or whether the shape matches what your code expects. The eight mistakes below are the ones that pass the first check and fail the second.
The short answer
A JSON validator checks grammar. It does not check meaning, precision, uniqueness, or encoding. Most production JSON bugs come from four places: characters and constructs that are illegal but visually familiar, values that exceed what JavaScript can represent, parser behaviour that the specification leaves undefined, and payloads that are structurally valid but semantically wrong.
If you remember one sentence, make it this: parsing is a syntax gate, and a JSON validator is a syntax gate with a nicer error message. Neither is a correctness check.
The practical defence is a two-stage pipeline. Stage one rejects malformed text. Stage two validates the parsed result against a schema that encodes your actual contract: required fields, types, ranges, formats, and what null means. Skipping stage two is how a file that "looked fine" reaches production intact and still corrupts your data.
1. Duplicate keys: legal syntax, undefined behaviour
{
"account": "acme-eu",
"plan": "starter",
"plan": "enterprise"
}The grammar allows this. RFC 8259 says names within an object should be unique, and then adds that when they are not, the behaviour of receiving software is unpredictable. That is specification language for "we are not going to save you."
In practice, most mainstream parsers keep the last occurrence, which is why the example above usually resolves to enterprise. Some keep the first. Streaming parsers, configuration loaders, and older library versions have historically done the opposite. Two services in the same pipeline can read the same file and disagree, and neither will report an error.
Duplicate keys rarely arrive on purpose. They come from shallow merges, where a base config and an override object are combined by spreading both into a new object; from templates that append a field without checking whether it already exists; and from human edits during an incident, when someone adds the corrected value above the old one instead of replacing it. In every case the file stays parseable, so nothing upstream objects.
The fix is to never emit duplicates and to reject them on ingest:
{
"account": "acme-eu",
"plan": "enterprise"
}On the receiving side, use a parser mode that fails on repeated names. Python's json.loads accepts an object_pairs_hook that lets you detect them, Jackson has FAIL_ON_DUPLICATE_PROPERTIES, and several linters flag them before the file ever ships. JSON.parse cannot detect duplicates through a reviver alone, so if you are handling untrusted input in Node, add a dedicated duplicate-checking parser or a pre-parse scan.
2. Big integers past 2^53-1
JavaScript numbers are IEEE 754 doubles. They represent integers exactly up to 9,007,199,254,740,991. Past that, adjacent integers collapse into the same value:
const order = JSON.parse('{"id": 9223372036854775807}');
console.log(order.id); // 9223372036854776000No exception, no warning. The identifier changed. Twitter and Discord both moved to string-encoded snowflake IDs for exactly this reason: their identifiers are 64-bit values that a JavaScript client cannot hold as a number.
The fix is to transport large identifiers as strings and convert deliberately:
{
"id": "9223372036854775807"
}const order = JSON.parse('{"id": "9223372036854775807"}');
const id = BigInt(order.id);BigInt keeps full precision in JavaScript, but JSON.stringify throws on it, so keep it at the boundary rather than letting it leak into your object graph. If you must keep numbers as numbers, parse with a big-number-aware library that produces BigInt or a decimal type directly from the source text.
3. Trailing commas
{
"name": "weekly-report",
"rows": 1200,
}This is invalid JSON. It is valid JavaScript, valid Python, and valid in most modern configuration formats, which is why it appears in hand-edited files constantly. The JSON grammar has no provision for a comma before a closing brace or bracket, and a strict parser will stop at the } with a message that rarely mentions commas.
{
"name": "weekly-report",
"rows": 1200
}The same rule applies to arrays: [1, 2, 3,] is not JSON. If you are hand-writing payloads for a test suite, strip trailing commas before saving, or run the file through a repair tool such as the fix invalid JSON online utility. The error message is almost never the useful part; the position of the failure is.
4. Comments are not allowed
{
"ttl": 300,
// seconds, not minutes
"retries": 3
}JSON has exactly six structural characters and no comment syntax. A // line comment or a /* */ block comment makes the document invalid, no matter how reasonable it looks.
{
"ttl": 300,
"retries": 3
}If you need annotated configuration, you are not writing JSON. JSONC, used by VS Code and TypeScript config files, adds comments and trailing commas. JSON5 adds comments, unquoted keys, single quotes, and more. Both are distinct formats with their own parsers, and both will fail if you feed them to something expecting strict JSON. Pick one deliberately and label the file accordingly, or move the explanation into a sibling _comment field that your schema allows.
5. A UTF-8 BOM at the start
Some Windows editors and exporters save UTF-8 with a byte order mark, the three bytes EF BB BF, which appear as an invisible \uFEFF before the opening brace. The JSON grammar does not permit it. Strict parsers reject the document at character zero with a message like Unexpected token \uFEFF in JSON at position 0.
The text looks correct in every editor. That is what makes this one expensive: the file is fine, the characters before the file are not.
const raw = fs.readFileSync("payload.json", "utf8");
const clean = raw.replace(/^\uFEFF/, "");
const data = JSON.parse(clean);Better still, configure the producing system to write UTF-8 without a BOM. When you cannot control the producer, strip the mark at the ingestion boundary and log that you did, because a BOM usually signals that something upstream is writing files in a legacy encoding mode.
6. NaN and Infinity are not JSON
JSON.stringify({ score: NaN, ratio: Infinity });
// '{"score":null,"ratio":null}'Neither NaN nor Infinity is a JSON number, and neither has a literal in the grammar. JavaScript makes this worse by silently converting both to null on the way out. A missing measurement and an undefined division become indistinguishable from a deliberate null, and the receiving code has no way to tell them apart.
{
"score": null,
"ratio": null,
"score_status": "not_computed"
}Decide explicitly what a non-finite result means in your domain and encode it: null with a companion status field, a string sentinel such as "NaN", or omission of the key entirely. Then make the schema enforce that decision. A number type in a schema will already reject null, so the null has to be intentional, not accidental.
7. Broken Unicode escapes and half surrogate pairs
Characters outside the Basic Multilingual Plane, including most emoji, are encoded in JSON strings as a surrogate pair:
{
"reaction": "\uD83D\uDE00"
}That pair is valid and decodes to a single code point, U+1F600. A lone half is a different story:
{
"reaction": "\uD83D"
}The grammar technically permits this escape, but the result is not a valid Unicode string. Parsers vary: some substitute the replacement character U+FFFD, some throw, some carry the lone surrogate into your database where it corrupts later output. The same failure appears when code slices a string by UTF-16 code unit and cuts a pair in half, which is easy to do with .slice() or .length in JavaScript.
The fix is to emit complete pairs, or the literal UTF-8 character, and to treat string truncation as a code-point operation rather than a byte operation:
const reaction = "\uD83D\uDE00";
console.log([...reaction].length); // 1Spreading the string counts code points, not code units, so a truncation helper built on it will not split a pair. On the receiving side, reject strings containing unpaired surrogates before they reach storage.
8. Valid JSON, wrong data: why a JSON validator cannot save you
{
"email": "ana@example.com",
"created": "12/03/2026",
"tags": ["billing"],
"discount": null
}Everything here parses. Nothing here is guaranteed correct. created might be 12 March or 3 December depending on locale conventions. discount: null might mean "no discount applies" or "the discount field was not loaded", and the difference matters to a billing calculation. tags might be expected as an object keyed by tag name, not an array. None of these are syntax errors, so no syntax checker will find them.
The null-versus-missing distinction deserves its own paragraph because it causes more quiet bugs than any other semantic issue. A consumer that checks truthiness sees the same outcome in both cases: undefined for a missing key, null for an explicit one, and both are falsy. Code that branches on truthiness treats "not loaded" as "no discount" and applies the wrong price. Code that branches on in or Object.hasOwn behaves differently again. Decide which one your contract uses for each field, write it into the schema with required, and make optionality a deliberate choice rather than an accident of serialisation.
This is the gap between "it parsed" and "it is correct". Closing it requires a schema:
{
"type": "object",
"required": ["email", "created", "tags"],
"additionalProperties": false,
"properties": {
"email": { "type": "string", "format": "email" },
"created": { "type": "string", "format": "date-time" },
"tags": { "type": "array", "items": { "type": "string" } },
"discount": { "type": ["number", "null"] }
}
}Note what the schema does: it makes created an unambiguous RFC 3339 timestamp, forces tags to be an array of strings, and requires the three core fields to exist while leaving discount optional-but-nullable. It still cannot tell you whether null means "no discount" or "not loaded"; that is a domain decision, and it belongs in documentation or a sentinel field.
Validate syntax first, then validate against the schema with a JSON validator that reports paths and failing keywords, not just a boolean. When a document is large and the error path is not obvious, load it into a tree viewer and inspect the actual structure rather than the text; collapsed nodes make type mismatches obvious in a way that raw lines do not.
Summary table
| Mistake | Symptom | Fix |
|---|---|---|
| Duplicate keys | Different services read different values | Reject duplicates at parse time; emit unique keys |
| Integers above 2^53-1 | IDs change silently after parsing | Transport as strings; convert with BigInt |
| Trailing commas | Parser fails at the closing brace | Remove the final comma before } or ] |
| Comments | // breaks strict parsers | Use JSONC or JSON5 deliberately, or a _comment field |
| UTF-8 BOM | Unexpected token \uFEFF at position 0 | Save without BOM; strip \uFEFF on ingest |
| NaN and Infinity | Values arrive as null with no warning | Encode non-finite results explicitly; enforce in schema |
| Half surrogate pairs | Replacement characters or storage errors | Emit complete pairs; slice by code point |
| Semantically wrong data | Parses cleanly, computes wrong answers | Validate against a JSON Schema contract |
FAQ
Is JSON with duplicate keys invalid?
The grammar permits it, so most validators will pass it. RFC 8259 only says names should be unique and that behaviour is unpredictable when they are not. Treat duplicates as a defect even though they are not a syntax error.
Why does JSON.parse change my large ID?
JavaScript parses every number as a double, which holds integers exactly only up to 2^53-1. Anything larger is rounded. Send the ID as a string and convert with BigInt if you need arithmetic.
Can BigInt be used with JSON.parse directly?
Not natively. JSON.parse has no big-integer mode, and JSON.stringify throws on BigInt values. Use a big-number-aware parser or keep the value as a string and convert at the boundary.
Does JSON allow trailing commas?
No. Neither objects nor arrays accept a comma before the closing brace or bracket. The same document is valid JavaScript, which is why the mistake is so common in hand-edited files.
Why can I not add comments to JSON?
Comments are not part of the grammar. JSONC and JSON5 add them, but they are separate formats. If a tool accepts comments, it is not parsing strict JSON, and its output may not be portable.
How do I fix "Unexpected token \uFEFF"?
The file starts with a UTF-8 byte order mark. Strip it with raw.replace(/^\uFEFF/, "") before parsing, or re-save the file as UTF-8 without a BOM in your editor or export settings.
What happens to NaN and Infinity in JSON.stringify?
Both become null. Neither value exists in the JSON grammar, and JSON.stringify substitutes null rather than throwing, so the loss is silent unless you check for it.
Is a lone "\uD83D" a valid JSON escape?
The escape is grammatically allowed, but it is not a valid Unicode string on its own. Parsers may substitute U+FFFD, throw, or pass the broken value through. Always emit complete surrogate pairs.
Why did a JSON validator pass a payload that crashed my app?
Syntax validation only checks the grammar. It does not check required fields, types, ranges, date formats, or identifier precision. Those require a schema and, for precision issues, a parser that preserves the original number text.
What is the difference between null and a missing key?
A missing key means the property is absent, which a schema can detect with required. An explicit null means the property exists with a null value. Conflating them hides whether data was unloaded or genuinely empty.
The bottom line
Every mistake in this list survives a syntax check. That is the pattern: the failures that reach production are the ones that parse. Duplicate keys exploit undefined behaviour, big integers exceed the runtime's numeric range, BOMs and comments and trailing commas exploit the gap between JSON and the languages that resemble it, and semantic errors exploit the gap between grammar and contract.
Build the habit of two gates. Gate one is syntax, run on every payload, with a clear error position. Gate two is a schema that encodes what your code actually assumes. Then treat the third gate, domain meaning, as something only a human or an explicit sentinel field can supply. A JSON formatter helps you see the structure while you work, but seeing it is not the same as proving it.
The payload that broke production was valid the entire time. That is exactly why it got through.
No comments:
Post a Comment