What we found Modulum doing — surprisingly — across the full dataset.

Modulum suppresses fabrication by truncating output — not by adding refusal.

Same Gemma-4-31B-Q4 weights, different inference stack. Vanilla emits long fabricated narratives (median wrong output 86 chars, max 500) — hallucinated biographies of distractor characters from PG19 noise. Modulum on the same weights emits short canonical wrong answers (median 47 chars, max ~75). Modulum's refusal rate on the BABILong tasks is similar to vanilla — it commits to short canonical wrong answers when the answer-is-in-context but the model can't locate it. However the needle-NOT-in-haystack probe (just landed 2026-05-20) shows Modulum refuses 100 % (150/150) when the entity is clearly absent from context — tying GPT-5.5 (99.3 %) and beating Grok 4.3 (96 %). The mechanism is therefore selective: structural output truncation suppresses fabrication on hard retrieval, and refusal kicks in on clear absent-entity questions.

Why this matters for hyperscaler procurement: hallucination at long context is the production-deployment blocker. Modulum's mechanism is verifiable from the same dataset — partner data teams can inspect output text directly. Same weights as vanilla; only the inference stack differs.

Modulum decodes FASTER than vanilla on the hardest task.

Counter-intuitive. Platforms layered on top of llama.cpp normally add overhead. Modulum does the opposite on multi-fact temporal reasoning: by tightening the output distribution, decode finishes sooner. The harder the task, the more Modulum's speedup compounds.

Bonus discovery during cross-axis analysis: the previously reported "Modulum prefill is 54% slower than vanilla on qa1 short context" was endpoint load, not platform overhead. Phase-1 ran during a 503 storm; phase-3 cells the next day show prefill within 3 % of vanilla. The latency-cost story originally attributed to Modulum dissolves under closer inspection.

Modulum retains 3.5–5× more of what it learned at 32k by the time you reach 128k.

Statistical correction from earlier analysis — the original "Modulum has flattest qa3 decay slope" framing was tested by Codex + Grok and found to suffer from floor saturation (vanilla qa2/qa3 at 128k sits within ~6 pp of the 17 % random-guess floor, so its slope is mechanically flatter than its true decay rate). Four follow-up statistical tests were run on the existing data:

1. Chance-corrected accuracy (Codex's (acc − 1/6) / (5/6)) — preserves the Modulum advantage at every cell, qa3 Δ shrinks from +2.0 raw pp to +4.1 cc pp.
2. Logistic regression with sample fixed effects — confirms a 3.89× / 3.99× / 1.67× baseline odds-ratio lift on qa1/qa2/qa3 (all significant) but finds slope interaction non-significant on logit scale.
3. Bootstrap CI on position-stratified tercile — late-tercile slope advantage point estimate +7.46 pp/2×, but 95% CI [−1.49, +16.42] crosses zero (p=0.10 marginal).
4. Paired McNemar's retention test — the decisive metric. For samples both stacks got right at 32k, who retains the answer at 128k?

The decisive paired retention finding

On the same Gemma-4-31B-Q4 base weights and same exact prompts, Modulum retains its 32k-correct answers at 128k 3.5× more often than vanilla on qa1 (78 % vs 22 % of discordant pairs, McNemar's p=0.0003) and 5× more often on qa2 (84 % vs 16 %, p=0.006). This is the cleanest possible test of "First, Not Lost" — eliminates per-sample difficulty AND floor saturation by conditioning on samples both stacks initially solved.

Per-stack retention rate (of samples got right at 32k, fraction still right at 128k):

How to reconcile this with the earlier slope finding: the raw OLS slope on aggregate cell percentages (−9.25 vs −12.75 on qa1) WAS directionally correct but underpowered for qa2/qa3 because vanilla operates near the 17 % random-guess floor. The proper paired test removes that confound. Modulum's First-Not-Lost claim now rests on McNemar's evidence — same prompts, both stacks initially solved them, Modulum keeps the answer 3.5× more often at 128k. This is independent of any slope model. qa3 retention also trends positive (+12.9 pp) but is underpowered (only 26 paired pairs).

Q4_K_M quantization is exact-bit reproducible on multi-fact reasoning prompts.

Q4 inference is widely understood to have small non-determinism from accumulator rounding even at temperature=0. We tested by re-requesting the same 50 prompts (idx 0..49) on the Modulum endpoint after the original phase-3 run, three weeks of operation later. Result:

qa2 cells are exact-bit deterministic. qa1 cells drift by 1–2 samples on re-request — within sampling noise but not exact-bit. Whatever Q4 rounding non-determinism exists, it cancels out on 2-fact reasoning prompts specifically. This was not predicted. For production routing, it means qa2-style multi-fact queries are repeatable across re-requests; qa1 retrieval queries have small Q4-quantization variance.

Sustained-run accuracy drift: −23 pp end-to-end within a 100-sample run.

Surfaced by tercile analysis of phase-1 data — splitting each cell's samples into early / mid / late thirds and measuring accuracy per slice. Modulum qa1 64k accuracy degrades monotonically across 100 sequential calls: 87.9 % → 78.8 % → 64.7 %. Confirmed on qa3 128k across 500 samples (32.5 % → 26.5 % → 22.0 %, two independent runs same direction).

This is a production-blocker. KV cache state accumulation or attention drift over sustained sequential operation. Wasn't in the original report — only emerged when we looked at the within-run distribution. Hypernym engineering needs to diagnose before hyperscaler deployment: the model loses 23 pp of accuracy if you keep sending 128k prompts to it.

Modulum has selective refusal calibration but zero usable logprob signal.

From phase-4 logprob capture (N=20 per cell): target_token_logprob ≈ 0.0 and perplexity ≈ 1.0 across every cell, regardless of whether the answer was correct. Modulum commits with the same numerical confidence whether it is correct or hallucinating. Combined with Breakthrough E (sustained-run drift) and Breakthrough A (no refusal mechanism), this means production routing has no signal to detect bad answers from logprobs alone.

This is the most important production R&D gap surfaced by the study. Modulum suppresses hallucination through output truncation but cannot signal uncertainty. A safety-critical deployment can't route uncertain queries to human review because the model doesn't expose an uncertainty score.

Format enforcement breaks at the same context length where hallucination appears.

Implication: Modulum's structural output enforcement holds at ≤64k context and partially fails at 128k. A platform-level improvement that keeps the format-truncation mechanism functional at 128k+ would compound directly into reduced hallucination at long context. R&D direction.

What only the 7-stack comparison surfaced.

Modulum's failure mode is fundamentally safer than vanilla's.

When Modulum gets a BABILong question wrong, it picks a location that is actually in the story — just the wrong one for this question. Vanilla on the same Gemma-4-31B-Q4 base produces long fabricated narratives copying PG19 distractor text in up to 42 % of wrong qa3 32k cases, and malformed hedges with no location commitment in 33–58 % of wrong qa2 cases. The platform layer converts dangerous failure modes into safe ones.

This is the mechanism finding that ties the package together. A wrong-but-grounded answer is the safest possible production failure mode — easy to detect against the document, correctable via verifier loop, doesn't mislead downstream tooling. Vanilla's "long hedge" or "narrative fabrication" failures are operationally dangerous. Same base weights, different inference stack, fundamentally different failure economics.

The three load-bearing breakthroughs.

Of the eight breakthroughs above, three carry the entire procurement-relevant story for hyperscalers and enterprise partners evaluating Modulum for production deployment:

All three are verifiable from canonical SQLite + CSV that a hyperscaler data team can audit independently. The mechanism (truncation, not refusal) is observable in output text. The slope (qa3) is reproducible across 500-sample cells with ±3.9 pp Wilson half-width.

Two production-blockers remain open and should be raised with Hypernym engineering before any partner deployment: E1 sustained-run drift (−23 pp end-to-end within a single 100-sample run) and F1 zero uncertainty calibration (PPL ≈ 1.0 regardless of correctness). Both are platform-side fixes, not base-model issues.