Early benchmarks: 135/135 trace-seeded replay cells matched across three model families

Field note · 7 min read

Why reasoning needs a protocol layer.

Large language models can produce useful work. The missing layer is not more fluency. It is a durable, inspectable way to carry the public reasoning state of that work from one session, model, reviewer, or toolchain to the next. Scholialang helps agentic systems preserve reasoning state, use tools without quality loss, and reduce context cost across long-horizon work.

Early benchmark signal

In a three-model replay smoke test, fresh Opus 4.8, Fable 5, and GPT-5.5/Codex sessions matched the original decision from Scholia trace carryover in 135/135 trace-seeded cells. A paired Phase 2.1 study found context tooling alone cost quality (−0.347) and adding Scholialang repaired that tax exactly, restoring bare-baseline parity. Separate compounding pilots cut Session-5 input tokens by 30%+ — up to 49.5% in the strongest compact mode — while preserving the conservative quality caveats.

High-level numbers are in the pilot section below; full eval methodology and runs will be published soon.


The failure mode

Agent work disappears into transcripts.

The standard interface for AI work is still a chat transcript. That is fine for a single answer. It is weak infrastructure for multi-step work. A transcript mixes goals, tool output, speculation, evidence, decisions, actions, contradictions, and final claims into one stream of prose. When the next agent picks up the task, it has to reread the stream and infer which parts are load-bearing.

This gets expensive quickly. Context windows fill with repeated premises. Reviewers have to search for the moment a claim became a decision. Downstream tools cannot reliably tell whether a paragraph is evidence, a hypothesis, a retraction, or a conclusion. The result is an infrastructure gap: the model may have done the work, but the system has no stable artifact that says what the work now knows.

The question is not whether an agent can reason. It is whether the work leaves behind a reasoning object another system can inspect.

The thesis

Scholialang is a protocol for visible reasoning state.

Scholialang is a structured reasoning language. It gives agents a small public vocabulary for the artifacts teams already care about: Goal, Observation, Evidence, Finding, Deciding, Action, Contradiction, Retract, Handoff, and Concluding. These are not hidden chain-of-thought. They are the visible work products an agent should be able to stand behind.

The point is not to make the model sound formal. The point is to make its public reasoning state addressable. A finding can cite the evidence that supports it. A conclusion can cite the findings it depends on. A later session can reuse a prior result without pretending it rediscovered it. A reviewer can ask for the neighborhood around a decision instead of reading every token that led there.

<Finding id="F_01" for_hyp="H_01" status="met">
  The handler preserves the soft-delete contract.
  REFER:E_01 REFER:Obs_02
</Finding>

<Concluding id="C_01" for_goal="G_01">
  The handler change satisfies the soft-delete contract once targeted tests pass.
  REFER:F_01 REFER:E_02
</Concluding>

v0.6 substrate

Content addresses turn traces into reusable infrastructure.

v0.6 adds the substrate that makes this more than notation. Every atom can carry a canonical_id: a SHA-256 content address derived from the atom's structural content, not its timestamp or local run id. If two sessions emit the same normalized structural atom, they address to the same id. Identity becomes "what does this claim say?" rather than "where did this text happen to appear?"

Once atoms have content addresses, references become graph edges when those references resolve. The registry stores atoms by canonical_id and records REFER and IMPLIES links as a DAG. A later session can walk ancestors to inspect premises, descendants to see what depended on a claim, and full chains to reconstruct why a conclusion is in scope.

The lazy prelude is the compounding layer. Instead of pasting full prior atoms back into the next prompt, tooling can render compact references: hash-only, hash-plus-preview, or full inline XML when detail matters. When a later trace cites a canonical id, the registry can resolve and hydrate the referenced atom body. Knowledge can accrue while the carried context stays bounded.


Early evals

What the pilots show so far.

The v0.6 pilots are not a final benchmark. They are useful because they test the right failure modes: can a fresh model replay decision-relevant reasoning from a structured trace, can an agent lean on heavyweight context tooling without paying a quality tax, and can later sessions reuse prior reasoning with fewer input tokens while preserving answer quality?

The replay result.

In a launch-day three-model replay smoke test, one model performed the first-pass task and emitted a Scholialang trace. A fresh model then saw either the bare task, the trace with the final decision stripped, or the full first-pass transcript. The trace-seeded arm matched the original decision in 135/135 cells across Anthropic Opus 4.8, Anthropic Fable 5, and OpenAI GPT-5.5/Codex.

Driver modelCold startScholia traceFull transcriptReadout
Opus 4.867%100%93%clear +33pp replay lift
Fable 580%100%100%perfect replay; high cold baseline
GPT-5.5 / Codex93%100%100%perfect replay; near-ceiling baseline

Replay caveat. This is a directional smoke test (n=15 per arm per model), not a final benchmark. The trace arm reached 100% across all three model families; formal lift depends on how much room the cold-start baseline leaves.

The tooling-repair result.

A separate Phase 2.1 study (n=24 paired task-cells across two driver models) measured what summary-style context tooling does to answer quality. Atlas-style tooling alone incurred a quality tax: the tooling arm landed −0.347 below the bare baseline on the composite rubric (B−A = −0.347). Adding Scholialang on top repaired that tax exactly (D−B = +0.347; paired t = 1.92, p ≈ 0.068 — suggestive, not conventionally significant; between-arm Cohen's d = 0.504), returning the full system to bare-baseline quality parity (D−A = 0.000).

Honest headline. The supported claim is that Scholialang makes tooling quality-safe — not that the notation makes models better on its own. Phase 2.1 did not measure a notation-without-tooling arm (its arm C was a prompt-framing cue, atlas_cue_only, not Scholialang minus tooling), so repair is the result, not lift.

The compounding effect.

The second launch signal is cost. A 150-cell cross-session pilot on claude-fable-5 — two task chains, three arms, five trials, five sessions deep — completed every cell with zero execution errors.

Mean input tokens per session. The v0.5 baseline climbs as prior context accumulates; the v0.6 arms stay nearly flat. The arms start identical at Session 1 and diverge monotonically with depth — the gap is the whole thesis, and it widens the deeper a chain runs.

0 2k 4k 6k S1 S2 S3 S4 S5 6,375 3,759 3,220
V05_inline — baseline (v0.5, full inline context) V06_hash_list — hash-list references V06_hash_only_lazy — hash-only, lazy hydration

By Session 5 the baseline needs 6,375 input tokens to carry its accumulated context; hash_list needs 3,759 (−41.0%) and hash_only_lazy needs 3,220 (−49.5%). That is the compounding behavior v0.6 was designed to produce.

Merge-gate scorecard.

Each v0.6 arm measured against the v0.5 baseline at the strict production thresholds: at least 20% input reduction, a quality delta no worse than −0.3, and statistical significance. Both token reductions are highly significant (paired t-test, p ≈ 0.0).

ArmInput reductionΔ qualityValidatorStrict gate
V05_inline98.0%baseline
V06_hash_list41.0%0.0100.0%mergeable
V06_hash_only_lazy49.5%−0.494.0%hold

Independently corroborated on Codex.

A separate full-stack pilot on a Codex agent reproduced the same shape: 150/150 emissions ok, 30/30 judgments, with hash_only_lazy cutting Session-5 input tokens 30.45% at a 0.0 quality delta and passing the same strict gate. Different model, different harness, the same curve — the Fable pilot is the lead result and the Codex run is independent corroboration, not the headline.

Reproducibility caveat. The Fable pilot ran hermetically — no MCP, no plugins, no hooks — driven headless on a fixed config so every cell is reproducible. That isolation is a feature for the measurement, but it also means these numbers capture the protocol-and-harness effect on its own, not the behavior of a full production agent stack.

That is the important shape. Compact references can cut carried context while holding quality flat; richer previews trade some of that compression back for more review context. The choice is not one mode forever — the infrastructure lets a workflow pick the prelude shape that fits the task: maximum compression, maximum review context, or a measured middle.


Why now

Agents are becoming systems of record.

The industry is moving from single-turn assistants to agents that call tools, write code, inspect files, run tests, modify infrastructure, and hand work to other agents. The transcript is becoming an operational record. But operational records need structure. They need stable ids, provenance, validation, explicit uncertainty, contradiction handling, and a way to resume without replaying everything.

This is especially true across model boundaries. A Claude session, a Codex session, a CI validator, and a human reviewer should not need the same chat UI to agree on what happened. They need a shared artifact. Scholialang is designed to be that artifact: readable enough for a reviewer, structured enough for tooling, and compact enough to carry forward.

This is not a replacement for retrieval, tests, proof assistants, or human review. It is the missing connective layer between them. Retrieval can fetch prior atoms. Tests can attach evidence. Validators can reject malformed references. Reviewers can inspect the proof graph. Agents can resume from a prelude that says what is already known instead of asking the model to rediscover it from a wall of prose.


The path forward

Not smarter logs. Smarter reasoning artifacts.

The next phase of agent infrastructure will not be won by chat logs that get longer. It will be won by systems that make reasoning portable: emit a trace, validate it, store it by content address, graph the dependencies, hydrate only the parts a later session needs, and let humans audit the result without reverse-engineering the transcript.

Scholialang is deliberately small because protocols have to survive contact with real workflows. The 32-atom catalog is enough to describe goals, observations, evidence, decisions, actions, handoffs, and conclusions without turning every trace into a theorem prover. v0.6 adds the content-addressed substrate that lets those traces compound.

If your team is building agent workflows where outputs drive real code, operational decisions, security reviews, research handoffs, or automated follow-up, the question is no longer whether the model can produce a plausible answer. The question is whether the work can be inspected, reused, corrected, and trusted after the chat is gone.

Read the v0.6 spec See what's new