The MVP, entry by entry.

A quick update: the live pass came back flat, and found the two reasons why.

EXTRACT-1 and ASSEMBLE-1 ran live over the clean ten. The extractor did its part: 67.2 claims a run, up from 24.8. The score did not follow — 13.35 → 13.11 on the eight tasks scored both times. Two defects explain most of it, and both were found by running the thing, not by reading it.

• timeoutEvery live model call used the page fetch's 30-second timeout. OpenRouter completed and billed after we timed out — twice on drb2-024, $0.089 each, with no result saved. Now every model call uses 180 s.PR #1417, merged
• coverageA cell counted every claim on every line that listed it, so drb2-090's three broad lines covered all 55 cells after one thin round and the coverer never ran. Now a cell is covered only by a claim the extractor declares answers it. The first fix — matching claims to cells by name — was measured before it shipped: 17 of 545 cells. We threw it out.issue #1416 · PR #1418, awaiting merge

Not clean yet, said plainly: only drb2-078 and drb2-090 were re-run on the fix — the eight-task mean moves to 14.38 on mixed trees. drb2-078's 0.00 is its answer being refused for stating figures no quote carries, one of them a real extraction error (#1419). drb2-114 is not scored: its run read its own answer key (#1421). And the capped key lost $4.17 during the pass that no ledger row accounts for (#1420). The three packages stay held until 078 is understood.docs/LIVE-PASS-2026-09-24.md — both passes task by task, the findings, and $5.69 by the ledgers

Fourteen fetches failed silently, and this page said none did.

what we published

Every page carried one line: 163 of 203 failed fetches recovered, 40 refused, 0 silent. A silent failure is one that left no fallback trail at all. Zero is the flattering number, and it is the one we printed.

what the record says

The gate's own render disagrees, and it is the record: rsk gate mvp report over the X2 ledger counts 217 ladder-eligible first attempts — 163 recovered, 40 refused and 14 silent — and prints all 217 trails underneath, one line each, so the fourteen can be counted by hand. They were. Every one of the fourteen is the same error class:

The site quoted the review — three readers writing — rather than the gate, a program rendering the ledger. When those disagree the program wins. The review was corrected on 23 September; both sentences now carry 217 / 163 / 40 / 14.docs/GATE-MVP.md — Summary, "Failed fetches and their fallback trails (S6)", 217 trail lines · docs/X2-REVIEW-2026-09-20.md · issue #1065, the surviving mutant

and it was already filed

The fourteen are not new. Unsupported content types were root cause #10 of that same reading — the extractor only handled text/html and text/plain, so PDFs, Markdown and XHTML failed without a trace to S6 — and QUAL-READ fixed it on 21 September. The number compares a 19 September run to a defect closed two days later, so it should have been printed instead of rounded to zero.

two smaller things, in the same sweep

• replayThe how-it-works page said 78 of 78 calls served from the ledger and cited the POC gate for it. The gate says 79 of 79; 78 of 78 is the bench line's own replay over a different ledger, and the POC page still carries it as that.docs/GATE-POC.md check 3 · docs/BENCH-POC.md U-4
• ledgersThe how-it-works page said eleven ledgers. That was the POC's kernel; Memory arrived with B6 and Knowledge with S2a on 17–18 September, and Ledger::ALL has held thirteen since. The POC page had it right and the page describing the system as it is did not.crates/kernel-events/src/event.rs — Ledger::ALL [13], Ledger::POC [11]

None of the three changed a measurement. All three were cases of a page disagreeing with the file it cites. On this site that is the full failure — if the ledger line is wrong, there is no reason to trust the ledger.this sweep, as one pull request against main

The clean ten — 16.42, and an answer key that turned out to be worth nothing.

Ten DeepResearch Bench II tasks — 672 yes/no checklist items written by others — run under method gold-ext/v3. We enforced one condition we had never met before: no run counted here read the source its own rubric came from.

16.42 overall: recall 12.89, analysis 27.16, presentation 27.04. Six tasks from the plan-v2 pass re-scored by JUDGE-1; four re-run on the 23rd under BLOCK-1 — 4 started, 4 finished, $1.598, exit 0.docs/GOLD-EXT-CLEAN.md — the ten, both ledgers, aggregate row e2c67fa7ca26 · PR #1402

the retraction · the answer key was worth nothing

We expected the four contaminated runs to drop in score once the shortcut was removed — that was why the earlier number was marked unsound. They did not drop. Across the four runs the mean went from 16.76 with the source to 17.61 without it.

drb2-046 gained eighteen and a half points. When the run had the survey it summarised that survey. When denied the survey, the run looked to the primary literature the rubric was actually built from. Keep that in mind when a shortcut seems to help.docs/GOLD-EXT-CLEAN.md — “What the answer key was worth”

what this is not

The published table gives 45.40 for the best system — recall 39.98, analysis 49.85, presentation 89.16 — and those come from a different judge: Gemini-2.5-Pro, later GPT-5.5, against our Gemini-3.7-flash. Comparable in shape, not in calibration: a direction, not a scoreboard. Our own before-and-after comparisons hold, because both sides share a seat and a method.

One more limit, noted because it hurts our score but not the leaderboard: 52 of the 500 recall items exist only inside the source each task forbids. That means a compliant run can at most reach about 66.7 of the 74.4 recall points, not 74.4.

The blocklist refused three fetches while we watched.

what was wrong

Every gold-ext task names the survey its rubric came from, and finding #1330 found four of the ten runs read it anyway — drb2-046 via a PMC mirror the list did not name, drb2-024 via arxiv.org/pdf/ where the guard covered only /abs/, drb2-062 via a URL named in its own list, drb2-108 straight into its body store. A deny list a mirror defeats is not a deny list, and it made those scores unsound in our favour.

what we built

The check moved to the fetch lane and is keyed on work identity: DOI, arXiv id in both forms and any version, PMID, PMCID, title plus first author, beside the listed hosts. A refusal is a recorded row. Then the harness was made to refuse to score a run whose ledger holds a blocked body at all — the guard and its auditor must not be the same code.

what it did, live

• blocked_source:arxivarxiv.org/pdf/2408.03261 — the /pdf/ twin of an id the task lists as /abs/. the exact form that defeated the old guard
• blocked_source:listed_urlthe MDPI survey, named in its own task's blocked list.
• blocked_source:listed_urlthe Springer survey, named in its own task's blocked list.

Three refusals across the whole clean re-run, each one a row a reader can go and count. They are the reason the entry above can say what it says.docs/GOLD-EXT-CLEAN.md — the re-run pass · issues #1355, #1330 · PR #1399

We were never short of pages. We are short of extraction.

We thought it was truncation — 60,000 characters across eight bodies, 12,000 each, so the misses fell off the end of a window. The next package was going to be show it more.

Then we reconstructed, with no model calls, exactly which bytes reached a prompt and asked of each of the 500 recall items where its answer was. The window budget is worth 4.9 points, not the 23.8 the story implied. 212 items had their answer in text the extractor was shown — it is being shown five to six times more answers than it converts.

That first pass matched by term proximity — an upper bound, not a count. A calibrated instrument then asked the same question of every stored page: 207 of 500 recall items already on disk — 30.8 benchmark points against the 16.42 the tool scores. Two methods, one direction.

The control came first — a cheap instrument that agrees with you is worse than none: eight real rubric facts against six fabricated ones of identical shape, same 26 pages, nothing in between. Then the sharper question: does any single passage carry a complete fact? Real 0.48–0.64 against controls at 0.04 — present in pieces, complete nowhere.docs/JEV-MEASUREMENT.md — the control, the funnel task by task, the complete-or-partial split, what each run cost · issue #1346 · the Jev page

what it changed

It killed the package we were about to build — for thirteen cents and twenty minutes, before a line of it existed. The problem is not reading more; it is taking every checkable statement off a page we already read, and then joining facts across passages into records.

the drawings, the control and the cost ladder: the Jev page · the two packages this produced — EXTRACT-1 (#1404) and ASSEMBLE-1 (#1405) — are open and unbuilt.

10.67 was our own judge misreading its own rubric.

We published 10.67 as the tool's number. It was not. Of the 366 recall items the flash seat refused, 212 were refused for the report's shape — “the basic information table was not provided” — with the fact itself quoted in the evidence field. A flat list of items gives the judge no way to tell a missing fact from a missing table.

So: tell it which dimension each item belongs to, and ask whether the fact is present as well as whether it is presented as asked. No new research — same ten reports, same judge model, 19 calls, $0.680955:

Presentation does not move — the control the fix predicts, since the rule touches only the content dimensions. +5.52 of the +5.91 is the judge's own score once told the dimension; +0.39 is our credit rule, 3 items of 672. Our "about four points" was right in size and wrong about the mechanism.docs/JUDGE-1-NOTES.md · docs/GOLD-EXT-V2.md

and every verdict is a row now

Verdicts used to sit inside a task row's blob, where nobody could count or join them. A scored task now writes one bench.result per rubric item — 672 of them — with the item, its dimension, both scores, the judge's reason and its evidence.

The first gate over those rows refused only a task with zero item rows. The adversarial lens deleted 20 of drb2-090's 101 from a copy of this ledger: the table published 15.00 for a task whose own row says 8.91, moved the mean 16.08 → 16.69 and exited 0. It now compares the count with the task row's n and refuses by name — none written and some written are the same defect.docs/JUDGE-1-NOTES.md — repair round 1 · issue #1357 · PR #1400

The answers we threw away were cut, not malformed.

what we thought it was

Five tasks in the first external pass and two in the second were dropped by the external judge's parser. We marked it as a shape bug — the judge returned {"results": [...]} where the reader expected an object — and planned to make the reader more tolerant.

what it actually was

The answers were cut by the completion limit in the middle of the JSON, and the parser discarded the judge's finished verdicts. A judge model's internal reasoning counts against the same completion budget as its answer, so longer thinking makes an answer more likely to stop mid-word.

We now treat a truncated answer as far as it is whole. Complete item-and-score entries count. Prose that was cut is marked as cut and not fabricated. Anything cut before its score is unanswered and retried, never given a score of 0. A balanced text that fails to parse is still malformed. The Cochrane answer lost in the first pass — 155 characters, stopped inside because — is returned in one call.

This is why the clean ten has ten rows. drb2-078 and drb2-108 were not moved by the fix; they were scored for the first time. Their numbers are new, not revised.docs/SPEC-kernel.md — FIX-1281 erratum [C-301] (2) · T-GX-8, T-GX-9 · docs/GOLD-EXT-V2.md · issue #1356

We doubled the reading. Recall doubled; analysis halved.

what we built

Plan v2 adds one state, cover, between investigate and synthesise. The coverer reads the charter's coverage contract — entities × dimensions and facets — against the run's counting claims, and writes a charter addendum for every cell below its minimum; a guard sends the run back to investigate for those lines, at most cover_rounds times, and a pass that adds no counting claim ends the loop. The charter is never edited.

what happened

The loop bit on every one of the ten external tasks — 2.4 rounds on average, where the eight-question set had given it none at all. No run reached its $2.00 cap.

And the score did not move: overall 9.88 → 10.67 on the flash seat, three tasks up, four down, one flat, with tenfold swings on the same question — inside that seat's noise. The sub-scores are not noise. Information recall 3.91 → 7.80, +99 %. Analysis 31.21 → 16.42, −47 %. The two cancel, which is the only reason the overall looks flat.

A blind second judge says the same in its own terms: ten Fable 5.1 readers, each given both reports for one task with plan identifiers stripped and order shuffled, told that length earns nothing — recall 7–3 to v2, analysis 5–4 to v1. Two judges agreeing on the direction is the finding; neither absolute number is.docs/DEPTH-1-NOTES.md — per-run facts, both judges, the blind pairwise (wf_7fffe91d-bd2, seed 20260922)

the two budget stories, and neither survived

Our hypothesis: the synthesist gets 24.8 claims instead of 14.6 under the same output budget, so each gets less reasoning. Measured, that budget never bound — all 39 synthesist and writer attempts ended finish_reason = stop, using at most 3,670 of 8,000 and 6,451 of 12,000 tokens. The article is claim-starved instead: drb2-090’s v2 run had 13 claims, 11 of them bare IEEE title stubs, and 19 of the 21 mechanism terms the judge missed were already in its stored pages. Same opened, never extracted class as the recall misses. The fix is EXTRACT-1, not a budget.

The other budget story in these notes did not survive the week. Reading more was supposed to be the lever because the extractor was being starved by its window budget. Measured, that window is worth 4.9 points — see the Jev entry above. The cover loop bought the reading it promised; the reading was not where the score was.

Two things the notes record that we would rather they did not. drb2-090 is a real regression, not noise — both judges agree (flash 12.87 → 1.98, Fable 10 items to 5) although v2 read more and its claims fell, 20 → 13. And the readers said, unprompted, that the report pads: "~130 of its 260 lines are a raw list of failed fetches".docs/DEPTH-1-NOTES.md · issue #1328 · the six open defects: what is broken

A console over the ledger — and a door that opened on a public user id.

what we built

Five screens — ask, the run, the report, runs, decisions — each a pure function of a ledger through the kernel's hardened read-only pool: one statement, an authorizer, query_only, bounded rows and bytes. Each has a ?json twin, so agent frontends read what a person sees. Server-rendered, no framework, no build step; a live run streams over SSE.

what was wrong

Four adversarial rounds; the last found the one that mattered. The leased key was never checked. POST /session read the lease in force for a user, then stored whatever key the form carried — comparing it to nothing. User ids are public, so naming one read the whole production ledger. Reproduced with a key spelled i-never-saw-the-leased-key.

The lease could not answer "is this the leased key?" — key_hash is OpenRouter's identifier, not something computable here. It now carries a key_fingerprint: blake3 derive-key over the key's bytes, one-way, in no way the key. A lease minted before this accepts no key at all. Every failed login answers one sentence, so the door names no one, and the lease is re-read on every request.

The same round found a lapsed kill switch still printing “ON” eight seconds later — the console had re-implemented the worker's predicates from the Decision's text alone, and both errors pointed the unsafe way. The reader now carries the request's clock into the kernel's own predicates.docs/CONSOLE.md — repair round 4 · SPEC [C-361], [C-362] · issue #1213 · PR #1324

A writer unit, and an article that has to cite to exist.

what we built

The external rubric wants sections per facet and four to eight thousand words; report.md was Answer / Findings / Verdicts in about two. NARR-1 adds a writer unit and a write state: in go the question, the coverage contract and the accepted claims with their anchors — never the contested or refuted — and out comes an article by facet, with its own "what is not established".

Every paragraph carries at least one claim reference, and every number, date and named entity in a paragraph must be entailed by a cited claim's quote or its page. A paragraph with no reference is refused at its path and repaired; the writer may not introduce a claim the ledger does not hold — an unanchored assertion is invariant.A7 at the paragraph that carries it.

what was wrong

Three of the ten runs in the DEPTH-1 pass ended gap{invariant.A7} at that gate. The research had finished and the synthesis was on the record; the writer simply could not cite precisely enough for the per-paragraph token rule on four words — Kármán, Advection, 10, American. Their reports carry the full answer and were scored like any other run.

Filed as #1326 and still open: an optional article should not end a finished run as a gap, discarding a good synthesis. The gate is right to refuse the paragraph; it is wrong about what that costs the run.docs/PACKAGES.json NARR-1 · docs/DEPTH-1-NOTES.md — the ten v2 runs · issues #1284, #1326 · PR #1306

The four judged laws, labelled — and two judges caught not judging.

the labels

L1 and L3 are decided by the ledger's rule. The other four need a reader. Every report was read twice more — by a sceptic who assumes the report is hiding something, and by the practitioner who asked the question — each labelling L2, L4, L5 and L6 with the sentence that decided it; a third reader settled every disagreement. Six of 53 reports pass all four.

The tool answers the question asked (L6) and notes what it could not find (L5). Its common mistakes are treating an inference as if it were on a page (L4) and softening what its own quote says (L2). The labels were written by a model, not the owner: Fable 5.1, two readers and a tie-break. The 20 % the readers disagreed on is the honest error bar. The owner's page stores every label with its reason and can overrule any label.docs/GATE-MVP.md — E3 calibration · labelling workflow wf_d406ea4b-567

the judges

Two model judges asked the same four questions of every report, so the labels could calibrate them. Neither can, yet. The flash seat passes every report on three of four laws — 0 failures on L2, L4, L5 and 12 on L6. The pro seat is lenient too and never reached the whole set (39, 39, 41 and 34 of 53, where the doctrine wants fifty), so the gate marks it calibrated on none. Where the readers found 92 failures, the judges found 12 and 30.

The ledger is doing its job here: a judge is never trusted by being listed, and these rows say that a binary rubric with a 400-token answer is not a judge. What the judge prompt lacks is the readers' brief — quote the sentence that decides it.docs/GATE-MVP.md — E3, the judged laws · doctrine/eval/v1/JUDGES.md · the E-3 rows: judge_brier, judge_kappa, judge_peer_ca

the eight questions, again

The eight questions the POC was judged on ran inside the campaigns, and the same readers judged each MVP report against the old engine's on the POC's rubric. Better 3, not-worse 1, worse 4. The bar is still six.

Every loss has one shape, and it is the one the reading found: the old engine read the canonical page — the vendor's pricing, the RFC, the maintainer's document — and this one read what the search engine ranked.docs/GATE-MVP.md — E1 re-judge · docs/E1-2026-09-16.md

Fifty-three questions for $8.27, then every report read.

the gate

Twenty campaigns of two to four adjacent questions, the later ones meant to be answerable in part from the first's verified knowledge. Headless browsers, liveness checks, Postgres row-level security, idempotent retries, SQLite in production, Cloudflare Workers, drone rules, event sourcing, passkeys, email delivery. Every question ran once, live, under a cap the kernel refuses to cross, from 09:46 to 11:05 on the 19th.

The one gap is the gate's most useful failure: paid search ran out at question 52 and a search outage was counted as a research outcome. It became RETR-1 — five free adapters, a per-engine quota book, a multi-engine seat, and the rule that an outage ends the provider, not the run. The MVP runs on no paid search.docs/GATE-MVP.md — Summary, Runs, Engines (S7), Failed fetches (S6) · bench/sets/campaigns-mvp.json

the reading

The next day, three independent readers went through all 53 reports against the question as asked, the pages the run opened, and the six laws. 264 defects, 65 of them high, in thirteen root causes. The kernel is cheap, honest about gaps, and quotes primary pages exactly when a search engine hands them over. It under-reads: pages picked by search rank, the first 12,000 characters of each, three verified claims per run, and a synthesist writing past its own verdict ledger.

Thirteen causes folded into four packages, all merged on the 21st, each with the reference tree re-recorded through free search and replayed at 0 misses. The synthesis gate's first live pass refused 3 of the 8 reference answers — names on the cited page but outside the quote; the rule was tightened to whole words of the cited page: 8 of 8, $1.08.docs/X2-REVIEW-2026-09-20.md · docs/PACKAGES.json QUAL-A, QUAL-B, QUAL-READ, QUAL-REPORT · PRs #1164, #1123, #1190

the stage before this one: the POC · where it is going: the roadmap · try it