research-stack · research.devclusterai.com
MVP · records to 2026-09-23 · every number is from the project's records, listed at the end
seq 1Questioncharter.locked

It is all on the ledger.

On 11 September the direction changed: not a Python pipeline, but the whole system as composed, append-only, content-addressed ledgers. Thirteen ledgers, three consistency classes, one writer, one gate.

seq 2Questioncharter.locked

Composed ledgers, as rails.

sequenced · one writer · dense seq per stream Question Plan Doctrine Policy Decision Run Unit Argument Memory B6 mergeable · order by (hlc, node) Fetch Eval Knowledge S2a scarce · Σ live + settled ≤ cap Budget one transaction admit() · schema · refs · invariants ledger.sqlite3 append-only derived views triggers · rebuilt hash-equal report.md a pure function of the ledger rejected: nothing written · admission.rejected + the attempt, kept
sequenced · one writer · dense seq Question Plan Doctrine Policy Memory Decision Run Unit Argument mergeable · (hlc, node) order Fetch Eval Knowledge scarce · Σ live + settled ≤ cap Budget admit() · schema · refs · invariants · class ledger.sqlite3 append-only derived views rebuilt hash-equal report.md pure function of the ledger rejected: nothing written admission.rejected + the attempt, kept

Thirteen ledgers in three consistency classes run as rails into one admission gate. Every event's id is a hash of its content; every ref must resolve; admitted events land in one append-only file, and the views and the report are re-derived from the events alone. A rejection writes only the rejection. The POC’s kernel admitted eleven; Memory (B6) and Knowledge (S2a) arrived on 17–18 September and Ledger::ALL has held thirteen since.crates/kernel-events/src/event.rs — Ledger::ALL [13], Ledger::POC [11] · docs/SPEC-kernel.md §1–§2, [C-210] Memory, [C-216] Knowledge · docs/DESIGN-kernel.md §0 · BENCH K-3: rebuild hash-equal on 29 tables

seq 3Argumentclaim.asserted · seq 2

Append-only, addressed by content.

Argument · stream = run id · seq dense · one writer refs — ids of earlier events, hashed into this id hypothesisseq 1id = blake3(content) claim.assertedseq 2id = blake3(content) verdictseq 3id = blake3(content) edge.attacksseq 4id = blake3(content) synthesisseq 5id = blake3(content) claim.asserted′same seq 2 · new bytes rejected · no UPDATE exists: the only write is an appendnew bytes → a different id · seq 2 is takenUNIQUE (ledger, stream, seq) · the attempt is kept

Events append in order; each id is blake3 over its content, the ids it references included. There is no update. A rewrite would be a different id under a seq that is already taken; the batch is rejected and the attempt is kept as evidence. Replay re-derives the same ids from the same bytes.docs/SPEC-kernel.md §1 event shape, id definition · docs/DESIGN-kernel.md P6–P7, §13.3

seq 4Argumentclaim.asserted · anchors[0]

The quote has to be in the bytes.

report.md · Findings Stripe caches status codes and response bodies for keyed requests once endpoint execution begins, replaying even 500 server errors on subsequent retries. high confidence · docs.stripe.com “Stripe’s idempotency works by saving the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeds or fails. Subsequent requests with the same key return the same result, including 500 errors.” A1 fetch.done · status ok · docs.stripe.com bodies/<sha256_text>.zst · extracted text, bytes Stripe’s idempotency works by saving the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeds or fails. Subsequent requests with the same key return the same result, including 500 errors. start end the model cites url + quote · the kernel resolves fetch_ref and [start, end) · rungs exact → nfc → folded GATE-4 · every resolved anchor decoded from the store and compared byte-for-byte 490 / 490 · 0 mismatches

The model cites a URL and a quote; the kernel resolves the URL to the page it fetched and finds the quote in the stored bytes — exact first, then NFC-normalised, then with whitespace runs and typographic punctuation folded; there is no rung for case folding or edit distance — and records the byte offsets. A quote that is not in the bytes is a rejection, not a footnote. The gate re-read every resolved anchor: 490 of 490 matched, 0 mismatches.docs/SPEC-kernel.md §2 anchor resolution · docs/GATE-POC.md check 4 · BENCH K-4: exact / nfc / folded 0.09 / 0.32 / 1.78 ms at 315 KB

seq 5Runadmission.rejected · attempts

Mechanical checks reject on the spot. Judged ones come back with a reason.

admission mechanical · program schema · number.unsafe refs resolve · refs.unresolved A1 anchor bytes == quote A2 a label needs evidence B2 Σ live + settled ≤ cap class · seq · hlc judged · reasoned L1 audit first L2 evidence > priors L3 source opened L4 epistemics labelled L5 gaps reported L6 reframe offered claim.assertedobserved · quote not found rejected on the spot · invariant.A1 · /anchors/0/quoteattempts row kept · the model gets attempt 2 with the reason claim.assertedobserved · 1 anchor resolved COMMITseq n · hlc judge unitor the owner judge.scored · label, reason, calibration → Eval ledger

What a program can decide — schema, refs, the anchor's bytes, budget arithmetic, every rule that carries a named reason code — is enforced at the gate in the same transaction. What only a judgment can decide — the six laws — goes to a judge unit or the owner and returns as a scored event with its reason. The guarantee: no unlabeled error. The codes are a closed list: reason_codes_closed asserts that the set the kernel can emit equals the spec’s exactly and that every code maps to one stage, so a refusal is a row with a code and a path into the payload that caused it — never a log line.docs/DESIGN-kernel.md §7.2 — the invariant registry and the closed reason-code list · docs/SPEC-kernel.md §1 reason codes, §7 judged.L1–L6 · doctrine/v1/DOCTRINE.md

seq 6Runrun.lease · resume

Survives kill -9 without paying twice.

live run process 1 audit orient recharter investigate + verify synthesise done kill -9process dies resume process 2 served from the ledger: checkpoint (task_ref, attempt) → no call repeated model calls: 0 · report hash equal O-1 · 20 random kills · 0 repeated model calls · report hash equal · gate check 2 · resume_repeats = 0 the one sanctioned re-issue: a call lost between the wire and its output commit

Each model call has three durable points: recorded before the wire, the raw body committed on arrival, then admission. A new process resumes from the ledger and re-issues only what was never committed. Twenty random kills: zero repeated calls, identical report.docs/BENCH-POC.md O-1 · docs/GATE-POC.md check 2 · docs/SPEC-kernel.md §2 Unit row, §8 check 2

seq 7Runresearch replay

Replays for $0.

live run job 20260910- 225300-83c859 OpenRouter · Brave · the pages 10 calls · $0.1625 · 19 sources call 1 call 2 call 3 call 4 call 5 call 6 call 7 call 8 call 9 call 10 replay rsk --offline research replay ledger ledger ledger ledger ledger ledger ledger ledger ledger ledger no network client is constructed · every call answered by its checkpoint (task_ref, attempt) · every search and fetch answered by the seeded Fetch events 0 calls · 0 reserves · $0 · same event ids · report blake3 equal U-4 · 78 of 78 calls served from the ledger · 0 calls · 0 reserves gate check 3 · 79 / 79 · report identical · rebuild equal on 29 tables

Replay opens the source read-only, seeds the run's roots and re-executes the plan with the model client and the fetcher replaced by ledger lookups. No network client is built. 79 of 79 calls served from the ledger; the same event ids; the same report bytes. The bench line’s own U-4 row is 78 of 78 over a different ledger — the drawing carries both, because they are two measurements and not one.docs/GATE-POC.md check 3 — replay_cache_hit hits 79, total 79, 0 calls, 0 reserves · docs/BENCH-POC.md U-4 — 78 of 78 · docs/DESIGN-kernel.md §13.2 · job id and cost: Gold-8 Verdicts, Q6

seq 8Budgetreserve · settle · cap

Money is reserved before it is spent.

cap · run:<id> · $0.40 for a standard run reserve reserve reserve reserve reserve refused · class.scarce_cap settled actuals reserve before the call Σ live + Σ settled + amount ≤ cap checked by the one writer settle after the actual replaces the estimate integer micro-dollars refused a rejection with a reason never an overrun, never a crash G-2 · 8 clients × 5 reserves of 10 against a cap of 100 · 100 admitted · 30 refused · 0 over-cap · p99 1.35 ms caps are events · expiry is an event · a settle that arrives after expiry is accepted and marked late

A unit is never invoked without a reservation; the writer checks live plus settled plus the new amount against every cap the reserve names. Settle replaces the estimate with the actual. Eight clients demanding 400 against a cap of 100: 100 admitted, 30 refused, none over.docs/SPEC-kernel.md §2 Budget row · docs/GATE-POC.md G-2 · bench/sets/gold-8/caps.toml (standard cap $0.40)

seq 9Runrun.started … run.finished

One run, on the record.

rsk research start · or deep_research_start over MCP plan and doctrine hashed into run.started audit audited orient oriented recharter chartered investigate investigation_done synthesise synthesised done investigator × lines · verifier × claims, together state.entered task.created unit.invoked unit.output unit.admitted premise ×n search.done fetch.done task.retrieved hypothesis ×n charter.locked lines + claims to verify claim.asserted + anchors verdict → edge.attacks claim_status (the fold) unit.rejected → repaired synthesis A5: cites observed claims, or gap run.finished report.md report.md Decision.feedback · actor owner refs the run and the claim it judges every arrow above is an event; every column is a stream you can query

The plan is the old method as a hashed statechart: audit, orient, recharter, investigate with verification alongside, synthesise. Every state writes its events; the report is rendered from them; the owner's verdict is an event that refs the claim it judges.docs/SPEC-kernel.md §3 plan v1, §7 report and Decision appends · docs/E1-2026-09-16.md

The walk-through, state by state

  • run.startedrsk research start, or deep_research_start over MCP. The plan and the doctrine are hashed into the first event, so the run names exactly the method it will follow.
  • auditThe auditor unit is invoked under a reservation. state.entered · task.created · unit.invoked · unit.output · unit.admitted · premise ×n
  • orientSearches and fetches are events; every page the task read is a task.retrieved ref. search.done · fetch.done · task.retrieved · hypothesis ×n
  • recharterThe charter is locked: the lines to pursue and the claims to verify. charter.locked
  • investigateInvestigators work the lines and verifiers the claims, together. A claim is asserted with its anchors; a verdict becomes an attacking edge; claim_status is the fold over them. A rejected output comes back repaired. claim.asserted + anchors · verdict → edge.attacks · claim_status · unit.rejected → repaired
  • synthesiseThe synthesis may cite only observed claims, or declare a gap (A5). synthesis
  • donerun.finished; report.md is rendered from the ledger — a pure function of it, with no timestamps. run.finished · report.md
  • Decision.feedbackThe owner's verdict on the report, actor owner, refs the run and the claim it judges. The eight side-by-side verdicts on the POC page were recorded exactly this way.

Plan v2: a cover loop, an article, and a source a run may not read

Plan v1 above is still the default. Plan v2 (plans/audit-first.v2.toml) is that statechart with two states added and a depth table that reads twice as much. cover, between investigate and synthesise, reads the charter's coverage contract against the run's counting claims and writes a charter addendum for every cell still short, sending the run back to investigate for the addendum's lines — at most twice at deep, three times at exhaustive; a pass that adds no counting claim ends the loop. The charter is never edited: the addendum's lines fold after the locked ones. write, after synthesise, renders the article described on the report page. A run under the merged plan v2 is audit, orient, recharter, investigate, [cover, investigate]*, cover, synthesise, write, done.

Alongside them, a rule about what a run may read at all. BLOCK-1 enforces a blocked source at the fetch lane by work identity, not by URL string: a listed /abs/ arXiv id refuses its /pdf/ twin, a DOI refuses its mirror, and rsk eval ext refuses to score a run whose body store holds one. It is what makes an external benchmark number mean anything — a run cannot read the source the rubric was written from and then be credited for what it found there.

The cover loop was measured, and the measurement is on the record with its cost: reading roughly doubled and information recall roughly doubled, while analysis fell by about half — the synthesist's budget does not scale with what the loop gives it, which is finding #1328, still open. Promotion between plans is E4's, by score, never by hand.docs/DEPTH-1-NOTES.md · docs/SPEC-kernel.md [C-360]…[C-364], §3 plan v2 · docs/GOLD-EXT-CLEAN.md — the three refusals BLOCK-1 made live · issues #1283, #1284, #1355, #1328

seq 10Evalbench.result · gold-ext/v3

How a number on this site is made.

Every figure here comes from one of three documents, and none of the three is written by hand. docs/BENCH-POC.md and docs/GATE-POC.md are rsk bench report and rsk gate doc over the POC gate's ledger; docs/GATE-MVP.md is rsk gate mvp report over the X2 ledger; docs/GOLD-EXT-CLEAN.md is assembled from the bench.result rows rsk eval ext wrote. In each of them a verdict is recomputed from the stored value and target at render time and never read from a stored flag, and a row the ledger cannot give prints not measured with the reason rather than being left out.

the external number, end to end

The one number on this site that is not ours to define is 16.42, and it is worth saying exactly how it is produced, because an external benchmark score is the easiest number in this field to quote dishonestly.

  • the setTen DeepResearch Bench II tasks. The question, the rubric and each of its 672 binary items were written by the benchmark's authors, not by us; an item is satisfied or it is not.docs/GOLD-EXT-CLEAN.md — the ten, per task
  • the methodrsk eval ext under gold-ext/v3: one bench.result row per rubric item, carrying the item text, its dimension, the benchmark's score, our credit, and the judge's own fact_present / as_asked with its reason and evidence. 672 rows, countable and joinable. A partial set of rows is refused by name, not averaged.docs/JUDGE-1-NOTES.md · the partial-row gate, repair round 1
  • the judgeGemini-3.7-flash through OpenRouter, under our own prompts, one seat for every run on both sides of any comparison we draw.docs/GOLD-EXT-CLEAN.md — “How to read it against a leaderboard”
  • the conditionNo run counted read the source its own rubric was derived from. BLOCK-1 refuses that source at the fetch lane by work identity, and rsk eval ext separately refuses to score a run whose body store holds one — the guard and its auditor are not the same code.docs/GOLD-EXT-CLEAN.md · issue #1355
  • the ceiling52 of the 500 information-recall items exist only inside the source each task forbids, so a compliant run's ceiling on recall is about 66.7 of the 74.4 points it is worth, not 74.4.docs/GOLD-EXT-CLEAN.md — the last section

and why the leaderboard is a direction, not a scoreboard

The published DeepResearch Bench II table (arXiv:2601.08536v3) gives 45.40 overall for the best system. That row comes from a different judge. The paper's evaluator was Gemini-2.5-Pro and its repository later moved to GPT-5.5; ours is Gemini-3.7-flash under prompts we wrote. The dimensions are comparable in shape and not in calibration, so the honest use of 45.40 is as a direction of travel. Our own before-and-after comparisons do hold, because both sides of each are scored by the same seat under the same method — which is also why we publish the seat.

The same caution has already bitten us in our own favour. 10.67 was published as the tool's score and turned out to be our own judge misreading its own rubric — 212 of 366 refused recall items were refused for the report's shape, with the fact itself quoted in the evidence field. Telling the judge which dimension each item belonged to moved the number to 16.58 with no new research at all. A benchmark number is a measurement of the judge as much as of the tool, and it is published here with the seat, the method version and the date attached for exactly that reason.docs/GOLD-EXT-CLEAN.md · docs/JUDGE-1-NOTES.md · docs/GOLD-EXT-V2.md · the MVP entry that took 10.67 back

next: the POC — what all of this measured to, including the check that failed — and the MVP, fifty-three questions later.