The MVP, entry by entry.

Fourteen fetches failed silently, and this page said none did.

what we published

Every page of this site carried one line about the fetch ladder: 163 of 203 failed fetches recovered, 40 refused, 0 silent. A silent failure is a fetch that failed and left no fallback trail at all — nothing tried, nothing recorded, the run simply short of a page it asked for. Zero of them is the flattering number, and it is the one we printed.

what the record says

The gate's own render disagrees, and it is the record: rsk gate mvp report over the X2 ledger counts 217 ladder-eligible first attempts — 163 recovered, 40 refused and 14 silent — and prints all 217 trails underneath, one line each, so the fourteen can be counted by hand. They were. Every one of the fourteen is the same error class:

Where the wrong number came from is worth naming too. docs/X2-REVIEW-2026-09-20.md, the read of the 53 reports, said in its own prose “S6 shows 163 of 203 ladder-eligible failures recovered, 0 silent failures” — and this site quoted the review rather than the gate. The review is three readers writing; the gate is a program rendering the ledger. When those two disagree the program wins, and the rule this site is supposed to run on is that a number comes from the record that produced it. One repo record contradicting another is the same defect one level down, so the review was corrected on 23 September: both sentences now carry 217 / 163 / 40 / 14, the 20 September render is named where it is quoted as evidence, and the correction says which document is the authority.docs/GATE-MVP.md — Summary, and “Failed fetches and their fallback trails (S6)”, 217 trail lines · docs/X2-REVIEW-2026-09-20.md — the headline bullet, cluster #10, and the correction note above them · issue #1065, the surviving mutant that let an empty trail read as recovered

and it was already filed

The fourteen are not a new defect. Unsupported content types were root cause #10 of that same reading — the extractor read only text/html and text/plain, so PDFs, Markdown and XHTML failed as unsupported_content_type and the failure was invisible to S6 — and QUAL-READ fixed it on 21 September: those types are read now, and a failure on them is ladder-eligible. The number measures a gate run on 19 September against a defect closed two days later, which is exactly why it should have been printed rather than rounded to zero.

two smaller things, in the same sweep

• replayThe how-it-works page said 78 of 78 calls served from the ledger and cited the POC gate for it. The gate says 79 of 79; 78 of 78 is the bench line's own replay over a different ledger, and the POC page still carries it as that.docs/GATE-POC.md check 3 · docs/BENCH-POC.md U-4
• ledgersThe how-it-works page said eleven ledgers. That was the POC's kernel; Memory arrived with B6 and Knowledge with S2a on 17–18 September, and Ledger::ALL has held thirteen since. The POC page had it right and the page describing the system as it is did not.crates/kernel-events/src/event.rs — Ledger::ALL [13], Ledger::POC [11]

None of the three changed a measurement. All three were a page disagreeing with the file it cites, which on this site is the whole failure — if the ledger line is wrong, there is no reason to believe the ledger.this sweep, as one pull request against main

The clean ten — 16.42, and an answer key that turned out to be worth nothing.

what we wanted

A number with nothing to say about it. Ten DeepResearch Bench II tasks — questions other people wrote, rubrics other people wrote, 672 binary items a report either satisfies or does not — scored under method gold-ext/v3. And one condition we had never actually met: no run counted here read the source its own rubric was derived from.

Overall 16.42: information recall 12.89, analysis 27.16, presentation 27.04. Six of the ten come from the plan-v2 pass of 22 September, re-scored by JUDGE-1; four were re-run on the 23rd under BLOCK-1, because their first pass had reached the forbidden source. That re-run's gate: 4 started, 4 finished, $1.598, exit 0.docs/GOLD-EXT-CLEAN.md — the ten, both ledgers, aggregate row e2c67fa7ca26 · PR #1402

the retraction · the answer key was worth nothing

We expected the four contaminated runs to lose points once the shortcut was taken away — that was the whole reason the earlier number was marked unsound. They did not. Over the four, the mean rose from 16.76 with the source to 17.61 without it.

drb2-046 gains eighteen and a half points. With the survey in hand the run summarised the survey; denied it, the run went to the primary literature the rubric was actually built from. Worth remembering the next time a shortcut looks like an advantage.docs/GOLD-EXT-CLEAN.md — “What the answer key was worth”

what this is not

DeepResearch Bench II's published table gives 45.40 overall for the best system — recall 39.98, analysis 49.85, presentation 89.16 — and those numbers come from a different judge: the paper's evaluator was Gemini-2.5-Pro and its repository later moved to GPT-5.5, while these are Gemini-3.7-flash under our own prompts. The dimensions are comparable in shape, not in calibration. The honest use of that row is as a direction, not a scoreboard. Our own before-and-after comparisons hold, because both sides are scored by the same seat under the same method.

One more bound, stated because it bites us and not the leaderboard: 52 of the 500 recall items exist only inside the source each task forbids, so a compliant run's ceiling on this set is about 66.7 of the 74.4 points recall is worth, not 74.4.

The blocklist refused three fetches while we watched.

what was wrong

Every gold-ext task names the survey its rubric was derived from, and finding #1330, measured on the 22nd, found that four of the ten runs read it anyway — drb2-046 through a PMC mirror the list did not name, drb2-024 through arxiv.org/pdf/ where the guard covered only /abs/, drb2-062 through a URL named explicitly in its own list, drb2-108 straight into its body store. A deny list that a mirror, a /pdf/ form or an alternate host defeats is not a deny list, and it made every score of those runs unsound in our favour.

what we built

The check moved down to the fetch lane and is keyed on work identity rather than URL string: DOI, arXiv id in both forms and any version suffix, PMID and PMCID, title plus first author, beside the listed hosts. A refused fetch is a recorded row, so a run can say it was denied rather than be quietly short. Then the bench harness was made to refuse to score a run whose ledger holds a blocked-source body at all — the guard and its auditor must not be the same code.

what it did, live

• blocked_source:arxivarxiv.org/pdf/2408.03261 — the /pdf/ twin of an id the task lists as /abs/. the exact form that defeated the old guard
• blocked_source:listed_urlthe MDPI survey, named in its own task's blocked list.
• blocked_source:listed_urlthe Springer survey, named in its own task's blocked list.

Three refusals across the whole clean re-run, each one a row a reader can go and count. They are the reason the entry above can say what it says.docs/GOLD-EXT-CLEAN.md — the re-run pass · issues #1355, #1330 · PR #1399

We were never short of pages. We are short of extraction.

what we expected

The working diagnosis was a truncation story: the extractor is shown 60,000 characters of prompt across eight bodies, 12,000 each, so the answers it misses are the ones that fell off the end of a window. The next package was going to be show it more.

what happened

Measured over the ten runs' own stored bytes, with no model calls at all — reconstructing exactly which bytes of which body were placed in an investigator prompt, then asking of each of the 500 recall items whether its answer was in text the extractor saw, in a body it never saw, or absent from everything the run read. The window budget is worth 4.9 points, not the 23.8 the story implied. 212 items had their answer in text the extractor was actually shown. It is being shown five to six times more answers than it converts.

That first pass matches by the proximity of distinctive terms, which over-counts by its own admission — 212 is an upper bound and the direction is the result. So the next day the same question went to a calibrated instrument. TypeSafe's Jev returns a probability, a choice among options you supply, or a score on a scale you define, and cannot produce a sentence at any price — so it could never have been our extractor, but it can be asked 500 questions of a whole corpus for pennies. Asked of every page the runs had stored rather than only of the text the extractor was shown: 207 of the 500 recall items had their material already on disk — 30.8 benchmark points against the 16.42 the tool scores. Two methods, two framings, one direction.

The control came first, because a cheap instrument that agrees with you is worse than no instrument: eight real rubric facts and six fabricated ones of identical shape, put to the same 26 pages. Nothing in between, so the threshold is not a judgement call. Then the sharper question — does any single passage carry a complete fact? Real items 0.48–0.64 against controls at 0.04: present in pieces, complete nowhere.docs/JEV-MEASUREMENT.md — every number in this entry: the control, the funnel task by task, the complete-or-partial split and what each run cost · issue #1346 — the earlier funnel, its proximity caveat and the diagnosis it overturned · the Jev page — the same figures drawn

what it changed

It killed the package we were about to build — for thirteen cents and twenty minutes, before a line of it existed. The problem is not reading more; it is taking every checkable statement off a page we already read, and then joining facts across passages into records.

the drawings, the control and the cost ladder: the Jev page · the two packages this produced — EXTRACT-1 (#1404) and ASSEMBLE-1 (#1405) — are open and unbuilt.

10.67 was our own judge misreading its own rubric.

what was wrong

We published 10.67 and read it as the tool's number. It was not. Of the 366 information-recall items the flash seat refused, 212 were refused for the report's shape — “the basic information table was not provided”, “does not organize content under reactive methods as a structured section” — with the fact itself quoted in the evidence field. DeepResearch Bench II writes many recall items in the wording of the expert article's layout, and a flat list of items gives the judge no way to tell a missing fact from a missing table.

what we changed, and what it was worth

Tell the judge which dimension each item belongs to, and ask two questions beside the benchmark's score: is the fact present anywhere in the report, and is it presented in the form the item asks for. No new research — the same ten reports byte for byte, the same seat, 19 calls, $0.680955. Over the eight tasks both passes scored:

Presentation does not move at all, which is the control the fix predicts: the prompt rule touches only the two content dimensions. +5.52 of the +5.91 is the judge's own score once it is told the dimension; +0.39 is our credit rule, 3 items of 672. Our estimate of “about four points” was right in size and wrong about the mechanism — had we shipped only the arithmetic we would have moved the number by a third of a point and called the defect fixed.docs/JUDGE-1-NOTES.md — what moved and what did it · docs/GOLD-EXT-V2.md

and every verdict is a row now

The v2 pass put its verdicts inside each task row's blob, where a reader tracing the funnel could not count them, join them or query them. A scored task now writes one bench.result per rubric item — 672 of them — carrying the item text and its dimension, the benchmark's score, our credit, the judge's fact_present and as_asked, its reason and its evidence.

The first version of the gate over those rows refused only a task with zero item rows. The adversarial lens removed 20 of drb2-090's 101 rows from a copy of this very ledger: the table published 15.00 for a task whose own row says 8.91, moved the set mean 16.08 → 16.69 and exited 0, under a sentence claiming the numbers were read from the verdict rows alone. The gate now compares the count with the task row's n and refuses by name — drb2-090 (81 of 101) — because none written and some written are the same defect.docs/JUDGE-1-NOTES.md — repair round 1 · issue #1357 · PR #1400

The answers we threw away were cut, not malformed.

what we thought it was

Five tasks in the first external pass, and two more in the second, were dropped by the ext judge's parser. We had it filed as a shape bug — a judge answering {"results": [...]} where the reader wanted an object — and the fix was going to be a more forgiving reader.

what it actually was

The answers were cut by the completion limit, mid-JSON, and the parser threw away the verdicts the judge had already finished. A reasoning seat's thinking is charged against the same completion budget as its answer, so the longer it deliberates the more likely the answer stops in the middle of a word.

A cut answer is now closed and read as far as it is whole: a result whose item and score are complete counts, its prose marked as cut and never invented; a result cut before its item text or before its score is unanswered, comes back in missing and is asked again, never scored 0. A balanced text that does not parse is still malformed and is never repaired. The Cochrane answer the first pass actually lost — 155 characters, stopped inside the word because — reads back from its direction and certainty in one call.

This is why the clean ten has ten rows. drb2-078 and drb2-108 were not moved by the fix; they were scored for the first time. Their numbers are new, not revised.docs/SPEC-kernel.md — FIX-1281 erratum [C-301] (2) · T-GX-8, T-GX-9 · docs/GOLD-EXT-V2.md · issue #1356

We doubled the reading. Recall doubled; analysis halved.

what we built

Plan v2 adds one state, cover, between investigate and synthesise. The coverer reads the charter's coverage contract — entities × dimensions and facets — against the run's counting claims, and writes a charter addendum for every cell below its minimum; a guard sends the run back to investigate for those lines, at most cover_rounds times, and a pass that adds no counting claim ends the loop. The charter is never edited.

what happened

The loop bit on every one of the ten external tasks — 2.4 rounds on average, where the eight-question set had given it none at all. No run reached its $2.00 cap.

And the score did not move: overall 9.88 → 10.67 on the flash seat, three tasks up, four down, one flat, with tenfold swings on the same question — inside that seat's noise. The sub-scores are not noise. Information recall 3.91 → 7.80, +99 %. Analysis 31.21 → 16.42, −47 %. The two cancel, which is the only reason the overall looks flat.

A second judge, blind, says the same thing in its own terms: ten readers on Fable 5.1, each given both reports for one task with the plan identifiers stripped and the order shuffled, each walking that task's own rubric items and told in as many words that length earns nothing — recall 7–3 to v2, analysis 5–4 to v1. Two judges agreeing on the direction of both sub-scores is the finding; neither seat's absolute number is.docs/DEPTH-1-NOTES.md — the per-run ledger facts, the two judges, the blind pairwise (wf_7fffe91d-bd2, seed 20260922)

the two budget stories, and which one survived

We wrote down one hypothesis for the analysis drop: the synthesist now receives 24.8 claims instead of 14.6 and writes under the same output budget, so each claim gets less reasoning, and the fix would be an analysis budget that scales with the claim count. That is #1328, still open — a measured question now rather than a guess.

The other budget story in these notes did not survive the week. Reading more was supposed to be the lever because the extractor was being starved by its window budget. Measured, that window is worth 4.9 points — see the Jev entry above. The cover loop bought the reading it promised; the reading was not where the score was.

Two things the notes record that we would rather they did not. drb2-090 is a real regression and not judge noise — both judges agree, flash 12.87 → 1.98 and Fable 10 items to 5, although v2 read more (96 → 106 targets) and its claims fell, 20 → 13. And the readers said, unprompted, that the report pads: “~130 of its 260 lines are a raw list of failed fetches”. Presentation score is being spent on a fetch-failure log.docs/DEPTH-1-NOTES.md — “What this says” · issue #1328

A console over the ledger — and a door that opened on a public user id.

what we built

Five screens — ask, the run, the report, runs, decisions — each a pure function of a ledger, read through the kernel's own hardened read-only pool: one statement at a time, an authorizer, query_only, bounded rows and bytes. Each screen has a ?json twin, so the agent frontends read exactly what a person sees. Server-rendered HTML in this site's own vocabulary, no framework, no build step, no external script; a live run streams over SSE.

what was wrong

Four adversarial rounds. The last one found the one that mattered: the leased key was never checked. POST /session read the lease in force for the user and environment, then stored whatever key the form carried, comparing it to nothing. A user id is public — the runs list prints one on every row — so naming a leased user read the whole production ledger: its questions, its reports, its stored page bodies, its cost. Reproduced on the branch with a key spelled i-never-saw-the-leased-key. The door's own two error notes were an unauthenticated oracle for which user ids the account had leased.

The lease row could not answer “is this the leased key?” — key_hash is OpenRouter's identifier for the key, not a function of the key that anything here can compute. So the lease now carries one that is: a key_fingerprint, the kernel's own blake3 derive-key over the key's bytes, 64 hex, one-way, domain-separated, in no way the key. A lease minted before this accepts no key at all — the owner leases again rather than a door opening on a user id. Every failure of POST /session now answers the same sentence, so the door names no one. And a session is no longer a check made once: the lease is re-read on every request, and any session past twelve hours is evicted.

The same round found a lapsed Decision read as in force, the unsafe way round: a kill switch with six seconds left still printing “the switch is ON” eight seconds later, a lane set off still reading off six seconds after it lapsed — because the console re-implemented the worker's predicates from the Decision's text alone. Both errors pointed the unsafe way, and the ?json twin is the contract the agent frontends read. The reader now carries the request's wall clock into the kernel's own predicates and nothing else.docs/CONSOLE.md — repair round 4 · SPEC [C-361], [C-362] · issue #1213 · PR #1324

A writer unit, and an article that has to cite to exist.

what we built

The external rubric wants sections per facet, fluent prose, four to eight thousand words. report.md was Answer / Findings / Verdicts in about two. NARR-1 adds a writer unit and a write state after synthesise: in goes the question as asked, the coverage contract and the accepted claims with their quotes and anchors — never the contested, refuted or unsupported ones, and the gaps list beside them — and out comes an article organised by the contract's facets, with a “what is not established” section of its own.

Every paragraph carries at least one claim reference, and every number, date and named entity in a paragraph must be entailed by a cited claim's quote or its page. A paragraph with no reference is refused at its path and repaired; the writer may not introduce a claim the ledger does not hold — an unanchored assertion is invariant.A7 at the paragraph that carries it.

what was wrong

Three of the ten runs in the DEPTH-1 pass ended gap{invariant.A7} at that gate. The research had finished and the synthesis was on the record; the writer simply could not cite precisely enough for the per-paragraph token rule on four words — Kármán, Advection, 10, American. Their reports carry the full answer and were scored like any other run.

Filed as #1326 and still open: an optional article should not end a finished run as a gap, discarding a good synthesis. The gate is right to refuse the paragraph; it is wrong about what that costs the run.docs/PACKAGES.json NARR-1 · docs/DEPTH-1-NOTES.md — the ten v2 runs · issues #1284, #1326 · PR #1306

The four judged laws, labelled — and two judges caught not judging.

the labels

L1 and L3 are decided by the ledger's rule. The other four need a reader. Every report was read twice more — by a sceptic who assumes the report is hiding something, and by the practitioner who asked the question — each labelling L2, L4, L5 and L6 with the sentence that decided it; a third reader settled every disagreement. Six of 53 reports pass all four.

The tool answers the question asked (L6) and says what it could not find (L5). What it gets wrong is stating an inference as something read on a page (L4) and softening what its own quote says (L2). The labels were written by a model, not the owner — Fable 5.1, two readers and a tie-break — and the 20 % the readers disagreed on is the honest error bar on them. The owner's page holds every label with its reason and can overrule any of them.docs/GATE-MVP.md — E3 calibration · labelling workflow wf_d406ea4b-567

the judges

Two model judges asked the same four questions of every report, so that the labels could calibrate them and the cheaper one could stand in for a reader from then on. Neither can, yet. The flash seat passes every report on three of four laws — 0 failures on L2, L4 and L5, and 12 on L6. The pro seat is lenient too and never reached the whole set: it judged 39, 39, 41 and 34 of the 53 reports, where the doctrine wants fifty labelled queries before a judge may be trusted, so the gate marks it calibrated on none of the four. Where the readers found 92 failures, the judges found 12 and 30.

The ledger is doing its job here: a judge is never trusted by being listed, and these rows say that a binary rubric with a 400-token answer is not a judge. What the judge prompt lacks is the readers' brief — quote the sentence that decides it.docs/GATE-MVP.md — E3, the judged laws · doctrine/eval/v1/JUDGES.md · the E-3 rows: judge_brier, judge_kappa, judge_peer_ca

the eight questions, again

The eight questions the POC was judged on ran inside the campaigns, and the same readers judged each MVP report against the old engine's on the POC's rubric. Better 3, not-worse 1, worse 4. The bar is still six.

Every loss has one shape, and it is the one the reading found: the old engine read the canonical page — the vendor's pricing, the RFC, the maintainer's document — and this one read what the search engine ranked.docs/GATE-MVP.md — E1 re-judge · docs/E1-2026-09-16.md

Fifty-three questions for $8.27, then every report read.

the gate

Twenty campaigns of two to four adjacent questions, the later ones meant to be answerable in part from the first's verified knowledge. Headless browsers, liveness checks, Postgres row-level security, idempotent retries, SQLite in production, Cloudflare Workers, drone rules, event sourcing, passkeys, email delivery. Every question ran once, live, under a cap the kernel refuses to cross, from 09:46 to 11:05 on the 19th.

The one gap is the most useful failure of the gate: the paid search month ran out at question 52 and a search outage was counted as a research outcome, which it is not. It became RETR-1 — five free search adapters, a per-engine quota book, a multi-engine seat, and the rule that a search outage ends the provider, not the run. The MVP runs on no paid search. rsk gate mvp report renders these from the ledger alone: a row the ledger cannot give says not measured and why, and a re-render of the same ledger is byte-identical.docs/GATE-MVP.md — Summary, Runs, Engines (S7), Failed fetches (S6) · bench/sets/campaigns-mvp.json

the reading

The next day, three independent readers went through all 53 reports against the question as asked, the pages the run opened, and the six laws. 264 defects, 65 of them high, in thirteen root causes. The kernel is cheap, honest about gaps, and quotes primary pages exactly when a search engine hands them over. It under-reads: pages picked by search rank, the first 12,000 characters of each, three verified claims per run, and a synthesist writing past its own verdict ledger.

Thirteen causes folded into four packages, all four merged on the 21st, each with the reference tree re-recorded through free search and replayed at 0 misses and 0 deviations. The synthesis gate's first live pass refused 3 of the 8 reference answers — names on the cited page but outside the quote, one term no page spelled; the rule was tightened to whole words of the cited page and the model repaired the invented term: 8 of 8, $1.08.docs/X2-REVIEW-2026-09-20.md · docs/PACKAGES.json QUAL-A, QUAL-B, QUAL-READ, QUAL-REPORT · PRs #1164, #1123, #1190

the stage before this one: the POC · where it is going: the roadmap · try it