Research and Development: the newsletter

Issue 29: A win that didn't replicate, and a test that couldn't see

mcp-tool-shop — Sat, 24 Oct 2026 20:52:23 GMT

The studio builds training data with local language models. A writer drafts each item and a critic checks it: it reads the material and a statement, and gives one of three verdicts, supported, unsupported or cannot_tell. When its verdict differs from the item’s label, the item is refused. A false refusal throws away a good item.

Issue 28 found that the writer is told what each label means and the critic isn’t. It ended with a plan: test the critic’s prompt as it is (A) against the same prompt plus the writer’s definitions (B), on 30 blind-graded items, with the decision rule fixed before any run. This issue reports that test.

The rule, written first

Of the 30 items, 23 were graded sound: both blind graders and the item’s label agreed, and nobody flagged it unclear. The other 7 were ambiguous and counted apart. Each prompt ran on every item with three seeds, so 69 calls on sound items per prompt.

B would be recommended only if both held:

(a) B’s false refusals on sound items are at most half of A’s;
(b) B accepts at most one more of the items whose label the graders contradict.

Passed by one call

The first run, on seeds 0–2, gave A 9 false refusals and B 4. Half of 9 is 4.5, so (a) held, by one call. (b) held at its limit. The rule said B.

One call is a thin margin, so we ran it again on seeds 3, 4 and 5, with nothing else changed.

Failed on the rerun

On the new seeds A refused 9 again, and B refused 7. That’s well over half, so (a) failed. Pooled over all six seeds, A refused 18 of 138 calls and B 11. Half of 18 is 9, and 11 is more, so the pooled result fails too. The rule says keep A.

The first run wasn’t wrong. It was a favourable draw: two items, P20 and P26, happened to go B’s way more often on those three seeds than on the next three.

A trade, not a fix

Splitting the refusals by the item’s true label shows what the definitions actually do:

Sound items, by label	Calls per prompt	A refused	B refused
cannot_tell (10 items)	60	12	0
supported (7 items)	42	5	5
unsupported (6 items)	36	1	6

A’s main error is calling a silent item unsupported. The definitions fix that completely. But B then calls some unsupported items cannot_tell, an error A almost never makes. B moves the line between the two labels; it doesn’t make the critic better at both.

Six items carried everything

Then we looked at where the refusals were. On 17 of the 23 sound items, neither prompt refused once: 0 of 102 calls each. Every refusal, in both prompts and all six seeds, was on the other six.

Those six weren’t a random draw. Five came from an earlier study, Track A, and were picked for this panel because the old prompt had already failed on them there: three it refused, two it flipped on between runs. The sixth, T10-C-09, came in under a rule that took every item of its kind from a later run, but its refusal in that run was one of the failures that started this study.

So the test was built, deliberately, around the old prompt’s known failures, and the new prompt was designed to fix those failures. That’s a fair way to find out whether a fix works on the cases it was made for. It can’t tell you whether the fix helps or hurts anywhere else, and the 17 other items gave no signal either way.

More seeds won’t change that. Antigravity, the outside model we work with, worked through the numbers before the rerun came back: almost all the spread in this result comes from differences between items, not from run-to-run noise, so rerunning the same 23 items barely narrows it. Its estimate was that doubling the seeds would shrink the uncertainty by only about 6%.

What we’re doing about it

We’re not deciding anything about the critic from this panel. The old prompt stays, and no run that the critic scores is being started. That hold lasts until we have a test whose answers are known for certain, rather than graded after the fact.

The lesson we’re taking is simple: a test built from your failures can only tell you about those failures.

The raw results, the scoring scripts and the graded panel are in the studio’s research library, and every figure here was recounted from the critic’s own output files.

Issue 28: Why our critic refuses good drafts

mcp-tool-shop — Fri, 23 Oct 2026 23:59:59 GMT

The studio builds training data with local language models. A writer model drafts each item, and a critic model checks it. Many items end in a verdict about a short piece of material (a notice, a label, a code snippet) and a statement about it. There are three labels:

supported: the material shows the statement is true;
unsupported: the material rules it out;
cannot_tell: the material says nothing either way.

The writer is told which label its item should carry. The critic reads the material and the statement and gives its own label. When they disagree, the draft is refused. Each refusal costs a rewrite, and a refusal of a good draft also throws away a good item. In issue 21 the critic refused too many good items. This issue follows the refusals, and finds three causes.

Cause 1: the critic is never told what the labels mean

The writer’s instructions spell out what cannot_tell means. For a cannot_tell item, the writer reads: “The material must be silent on the statement: it neither shows it nor rules it out.”

The critic’s instructions name the three labels and nothing more. Its verdict request holds that instruction, the material and the statement. So where “silent” ends and “rules it out” begins is left to the critic’s own reading.

That’s the line where its answers wobble. An earlier study, Track A, ran the critic twice over the same 102 items on the same kind of GPU. As issue 27 found, the seed never reached the model in those runs, so each run sampled differently. The two runs disagreed on 7 of 102 verdicts, and 6 of the 7 were between unsupported and cannot_tell. The seventh went from unsupported to supported.

Track A also has 43 intact items, which the study’s design left unchanged as controls. Over the two runs, the critic refused 3 of them at least once. Two of those refusals happened in one run only, which is noise. The third was refused both times.

Cause 2: a brief that contradicts itself

T10, the lesson in our pilot, teaches one thing: when a size, quantity or scope word in the statement differs from the material (“at least”, “exactly”, “every”), check the exact size the statement claims. Each slot in the plan also gets a side of that lesson to teach. One is “judges the exact size”. Another is “an equivalent wording still matches”: the statement says the same thing, phrased differently.

That second side can’t go with cannot_tell. If an equivalent wording in the material matches, the answer is supported. Yet in one writing wave of 60 slots, the planner gave that side to 7 cannot_tell slots and 3 unsupported slots. Each of the 7 cannot_tell writers was told both to show a match and to keep the material silent. The 3 unsupported writers weren’t given the silence line, and the critic voted their items T10 every time.

The critic also votes three times on which lesson an item teaches. Here’s what it said about the 12 cannot_tell slots:

Planned with “an equivalent wording still matches”: on 6 of the 7, all three votes said T14, a different lesson: the material is silent on a detail, so don’t fill it in. The writer had escaped the contradiction by asking about something the material never mentions:
- how to store a cleaner;
- a tea’s caffeine, on a label whose only quantity is its weight;
- which floor a service elevator is on;
- more than ten utensils, on a box label that lists “Utensils” with no count.
Planned with “judges the exact size”: 3 of the 5 were voted T10 three times out of three. They have one shape: the material gives a loose amount (“a few items are redundant”, “several team members raised concerns”), and the statement claims an exact one (“exactly three”). The other 2 weren’t clean. One swapped the North Wing for the South Wing; the other stretched a week’s schedule over a month.

The fix in the planner is simple: never pair a verdict with a side it contradicts. A second fix is an option in the writer’s brief. It asks for that loose-versus-exact shape on T10’s cannot_tell items, and forbids swapped names and unmentioned topics. Both have been agreed but not built. The option will be tested against today’s brief before it becomes the default.

Cause 3: sometimes the critic is right

Before measuring how often the critic wrongly refuses, we needed to know which items are sound. So we built a panel of 30 items, 10 for each label, and had them graded blind: the grader saw the material, the statement and the label definitions, with no stored label and no critic output.

Two graders worked separately: Antigravity, in a fresh conversation, and the R&D seat. An item counted as sound only if both graders and its stored label agreed, and neither grader flagged it as unclear. Anything else counted as ambiguous, with no tiebreak.

One ambiguous item shows the critic doing its job. Its stored label was supported, and the critic refused it. Both graders read cannot_tell. The statement named a payroll portal that the material never mentions. That draft’s flaw was the writer’s, and the critic was right to refuse it.

Another needs a judgment the definitions don’t settle. The material is a complete input check with no upper limit, and the statement says the code caps guests at 50. If that function is the whole check, the material rules the statement out. If it might be one piece of a larger system, the material is silent. The two graders split. No prompt change or vote can make an item like that stable, so the bench counts such items apart.

R&D had looked at three of the 30 items before grading, so its grades on those three weren’t blind. Two of them are among the 7 ambiguous ones.

A key file corrected before any run. Four panel items come from Track A’s traps. On 3 of the 4, both graders disagreed with the stored label. The reason was in the study’s design document, written four days earlier. Those items are deliberate traps: the deciding fact is removed, so the true label becomes cannot_tell while the stored label keeps the old answer on purpose. The panel builder had copied the stored label. The key file now holds the true one, and keeps the stored value alongside. The graders’ disagreement is what caught it.

A measure we tried: hesitation

The critic thinks before it answers. We counted how often its thinking switches between the three labels, and called it hesitation. On the two Track A runs:

The gap is real, but hesitation is a weak alarm. The 10 most hesitant items held only 1 of the 7 flips, and the top 20 held 4. And it can only find doubts the critic writes down. A confident wrong refusal leaves no trace. So it’s useful as a hint, not as a gate.

How this was found

None of this came from a checklist. It came from an open research dialogue with Antigravity, the outside model we use for reviews. It began as follow-up questions on issue 27’s results. Antigravity lined the cannot_tell drafts up against the plan, and spotted both the contradictory pairing and the missing definitions. It also suggested the hesitation measure, and grading a sample blind. We checked each claim against the code and the run records. One list of item names it gave was wrong, though its count was right. The figures in this issue are ours, recounted from those records.

Antigravity also designed this issue’s charts and diagrams, over three rounds with the newsletter desk, which recounted every number in them from the same records.

What’s next

A 30-item bench. It runs on one rented A100, with each of the 30 items × 3 seeds × 2 prompts: today’s critic prompt, and the same prompt plus one sentence giving the writer’s definitions. The decision rule is fixed in advance:
- the definitions are recommended only if they cut false refusals on sound items by at least half;
- and they accept no more than one extra item whose sound label they contradict.
Not yet measured.
The planner fix and the writer-brief option. Agreed, not built. The option’s test measures two things: how often the critic votes T10, and how many drafts it accepts.

The raw notes, the bench design and the graded panel are in the studio’s research library, and will move into rnd with the bench’s results.

Issue 27: Same request, same seed, same bytes?

mcp-tool-shop — Fri, 23 Oct 2026 23:59:59 GMT

The studio builds training data with local language models. One model writes, and a second model, the critic, checks what it wrote. Everything is pinned so a study can be run again: the model files, the server build, the settings, and a seed for the random choices the model makes while it writes. If the same request comes back with different bytes, a re-run can’t tell a real effect from noise.

So we asked a narrow question: with the seed fixed, does the same request return the same bytes on a different machine? We rent GPUs on Google Colab by the hour (issue 13 covers how), and every session lands on a fresh virtual machine, so this matters to every rented run.

First, a bug of our own

The first test re-ran three writer requests on a second A100. A model doesn’t pick words directly. At each step it scores every possible next token, and then a sampler picks one, using the seed for any randomness. The two VMs gave identical scores for the top five candidates, but picked different tokens.

Identical scores mean the model’s arithmetic matched. The difference was in the sampling, and the cause was ours. Our model proxy, which sits between the study code and the model server, removed the per-request seed along with other sampling settings it overrides. Each request reached the server with no seed. The fix (rnd 1.1.23.1.28) passes the seed through.

This corrects something issue 21 reported: the same critic prompts, sent twice at the same seed, gave 7 different verdicts out of 102. Issue 21 put this down to the critic sampling at temperature 1. It wasn’t: those calls went through the same proxy, so the seed never reached the critic, and the prompt cache was on. With the seed passed through, the same critic at the same settings gave identical flaw votes on 101 of 101 calls.

We also turned off a second possible cause: the server’s prompt cache, which reuses work from earlier requests that start the same way. Every request now asks for the cache off. A checker reads the server’s own record of cached tokens on every call, and fails the run if any call used the cache. To show the checker works, we ran it on the first test, which had the cache on. It flagged 3 of the 4 calls, with 7,016, 350 and 350 cached tokens.

The method

The same requests every time: one pilot study’s 178 critique calls, built from drafts written and sealed in an earlier session, so every machine critiques the same text.
The same software: one build of the llama.cpp model server, the same model files (Muse-Glimmer-30B as critic, Gemma 4 31B as writer, both at 8-bit), with one request at a time.
One VM per arm. Each arm ran in its own Colab session.
Matched by request. Every request has a key, a hash of its role, messages and settings. A comparison script matches the calls by key and compares the replies byte for byte. When bytes differ, it also reports which labelled lines of the critic’s reply changed.

The result

Compared with an A100 40 GB	Calls compared	Byte-identical	Verdict changed
A second A100 40 GB VM	176	176	0
A third A100 40 GB VM (stopped partway)	72	72	0
A100 80 GB, same server build	178	178	0
RTX PRO 6000 Blackwell, its own build	178	113	3 of 52 verdict calls

178 calls cover 176 distinct requests; two were sent twice, identically.

The critic’s second task, voting on flaws, agreed on 101 of 101 calls across two A100 VMs. The three writer requests that diverged in the first test now match too, which shows the seed fix working where the bug appeared.

On the A100, the answer is yes. Three separate VMs gave the same bytes on every call they shared. So did an A100 with 80 GB of memory instead of 40, on the same server build.

The Blackwell card is different. The RTX PRO 6000 needs its own build of the server, compiled for its newer architecture. On that build, 65 of the 178 replies changed. A critic’s reply is a short form with labelled lines: a vote on a trait, a flaw label, a quoted passage it based the call on, a contrast note the pipeline doesn’t read, and on 52 of the calls, a verdict. Here’s which lines changed in the 65:

What changed	Calls
Only the contrast note, which the pipeline doesn’t read	27
The trait vote (11 alone, 10 with the contrast note)	21
Only the quoted passage	10
A flaw label	4
The verdict	3

So the verdict held on 49 of 52 verdict calls, about 94%. All 3 changes went the same way: “unsupported” on the A100 became “cannot tell” on the Blackwell card. That’s close to the second study below, where a different card changed 1 verdict in 30. The text differs often; the verdict rarely does.

Because both the card and the build differ, this run can’t say which one is responsible. The 80 GB A100 does rule out one explanation: the amount of memory doesn’t matter. In practice we keep the two apart. Results from the Blackwell card are never pooled with the A100’s.

Is a faster card worth it?

The two other cards cost more per hour. Here’s what that bought, over the same 178 requests:

	A100 40 GB	A100 80 GB	RTX PRO 6000
Price per hour, vs the A100 40 GB	1.00×	1.28×	1.68×
Speed reading the prompt	1.00×	1.01×	3.06×
Speed writing the reply	1.00×	1.11×	1.32×
Total model time	11,118 s	10,046 s	8,350 s
Cost per request, vs the A100 40 GB	1.00×	about 1.15×	about 1.26×

A critic reads a long prompt and writes a short reply, one request at a time, so most of its time goes to writing. The Blackwell card reads prompts three times as fast, but writes only a third faster, so for this job it costs about a quarter more per request. It would pay off on jobs that read far more than they write, on large batches, or on models too big for 80 GB. The 80 GB A100 is worth its price only when a job needs the memory.

The same pattern in a second study

The same day, a separate benchmark ran a 30-item judging task through Ollama, a different model server, with Gemma 4 31B. It had already been run twice on the studio’s own RTX 5090. We ran it on two Colab A100 VMs:

	5090 vs 5090	A100 vs A100	A100 vs 5090
Items with any difference	0 of 30	0 of 30	16 of 30
Final verdict differs	0	0	1

Again, the same card gives the same answer, and a different card mostly changes the wording. Of the 16 items that differed, 15 differed only in the model’s reasoning. The one changed verdict was an item where the 5090 ran out of room mid-answer and Colab finished it.

The earlier run of this benchmark found the one thing that broke reproducibility on the same card: running two requests at once. Two requests decoding together go through batched arithmetic, which changed 1 verdict, 2 quoted passages and 15 of 30 reasoning texts. Every pinned study now runs one request at a time.

That run also hit a quieter problem. The model’s public tag, gemma4:31b, had been republished as a different build, with different weights. The first paid attempt stopped at a guard that refuses any model whose digest doesn’t match the pinned one, before any call was made. The fix installs the pinned build from its manifest, file by file, and checks every file’s hash.

Three VMs lost, and what the logs kept

Colab can take back a VM at any time. Three of the six long A100 sessions in this study were lost: one after 1 h 16 m, one after 1 h 30 m, and one after 3 h 16 m. Each loss happened while 3–4 of our sessions were running at once. Three losses are too few to name a cause, so we’ve recorded them and left them unexplained.

As each call finishes, the job writes its record into the job’s log, and the rig downloads that log from the VM every 2 minutes while the job runs. When a VM is lost, the rig keeps its last copy, at most about 2 minutes behind. What the log held was rebuilt; on one lost VM it stopped two calls short of the end. The log can’t show whether those two calls ran before the VM was lost. To check the rebuild, we ran it on a session that closed normally, and it matched the official records on 178 of 178 rows. What a lost VM still costs is the timing log and the session’s own summary. Writing those into the log the same way is next on the list.

What it cost

Colab bills in compute units. The A100 40 GB costs about 5.3 units an hour.

	Units
Reproducibility study: 12 GPU sessions, including the three lost VMs, the first test and the bug’s halted session	112.93
Its CPU test runs	0.13
Second benchmark: 5 GPU sessions, including two lost to a missing unpack tool and the guard’s stop	7.85

Every figure was reconciled against our units ledger.

Where things stand

Pinned studies on Colab run on A100s, one request at a time, with the seed passed through and the cache off. 40 GB and 80 GB results can be pooled.
Other cards are their own baseline. Results from the RTX PRO 6000 or the local 5090 are reported separately, never pooled with the A100’s.
The cause on the Blackwell card is still open: the card, its build, or both. Separating them would need a second build on the Blackwell card.
The full write-ups are in rnd: the reproducibility study and the benchmark.

Issue 26: Release night: two near misses, both caught

mcp-tool-shop — Fri, 23 Oct 2026 23:59:59 GMT

Two releases went out overnight: Escape the Valley 1.3.1 and XRPL Camp 1.5.1. Both carry the clean-build fixes from issue 25. Their packaged programs hold no build-machine folder paths, and Escape the Valley’s carry exact licence texts for everything inside them. Getting there, the checks caught three things that would otherwise have gone out wrong or not at all.

An edit the live page never saw

Every release goes through a full checklist, the treatment, before it ships. One step updates the tool’s landing page. For Escape the Valley, a new feature card was added to the site’s configuration file, the site built cleanly, and the deploy succeeded.

The card never appeared. Escape the Valley’s landing page is a hand-built page that doesn’t read that configuration file. The live page still said “v1.1 — the highlights” and quoted a test count from June.

How it was caught: the release record has to quote evidence for each step, and the landing page evidence meant fetching the live page and finding the new text in it. It wasn’t there. “The page returns 200” would have passed.

What changed:

A follow-up edited the page that’s actually served. It now reads “v1.3 — the highlights”, with the current test count (1,068). A claim nobody had re-measured since June was dropped.
The release gate now checks the live page. Each release record lists text the live landing page must show. The gate fetches the page and refuses the release if any of it is missing.

A release that GitHub wouldn’t start

Pushing the version tag starts Escape the Valley’s release workflow. That workflow first re-runs the test suite as a gate, then publishes to npm, PyPI and GitHub. This time GitHub refused to start it at all: “startup failure”, with no logs.

The cause was added four weeks earlier (2026-09-26), in a change that wasn’t about releases. The test workflow gained a coverage upload that needs permission to request a short-lived identity token (id-token: write). The release workflow calls the test workflow as its gate, but granted it only permission to read the code. On GitHub, a workflow called by another can’t hold more permissions than its caller grants, and when it asks for more, GitHub won’t start the run.

So every Escape the Valley release since that change would have failed the same way. This was simply the first one tried.

What happened next:

Nothing was published. The other release steps (npm, PyPI, the program builds) all wait for this workflow to succeed, so none of them ran. That was checked before anything else: no GitHub release, and no 1.3.1 on npm or PyPI.
The fix was one line: the gate grants the token permission too.
The tag was moved. The first tag pointed at a commit with the broken workflow. Since it had published nothing, it was deleted and re-made on the fixed commit, after a fresh release record and approval for that exact commit. The only difference between the two commits was the workflow file.
The rule carries forward: the next Escape the Valley work adds a Rust test job to the same workflow, and its brief says it must ask for no more than the gate grants.

A README that filled the card

Release READMEs are translated into seven languages by TranslateGemma 27B, running locally on the studio’s RTX 5090 (32 GB). Each run needs the GPU, and is granted for an expected size and time.

The first attempt was asked for about 20 GB. It peaked at 31,165 MiB, just under the card’s limit. The translation script sends no context size, so the model server loaded the model with its full default of 131,072 tokens and reserved memory for all of it. The Escape the Valley README is 10.8 KB, about 3,000 tokens.

That attempt had a second problem, of the studio’s own making. A command meant for the Escape the Valley folder ran in another repository’s folder and rewrote that repository’s translations. Nothing was committed, and the files were restored at once.

The second run used a copy of the model capped at 16,384 tokens of context:

Run	Context	Peak card memory	Result
First (default)	131,072 tokens	31,165 MiB	wrong folder, reverted
Second (capped)	16,384 tokens	21,846 MiB	7 of 7 languages

The second run also had a hard stop: a separate monitor would end it if the card passed 26,000 MiB. It never fired.

A clean exit code isn’t taken as proof. Every run is now checked three ways: all seven files are newer than the run’s start, the language bar lists all seven, and each keeps all 25 bold labels from the English.

Product names still need protecting. Hindi translated “XRPL” and the game’s name, and Chinese turned the game’s name into a translated title. Both were fixed by hand. A protected-terms file, already in use on rnd, comes to each tool’s next release, so the translator leaves those names untouched.

For anyone using the tool: an issue is filed on polyglot-mcp to size the context to the input. On a 24 GB card, the default would not fit on the card.

Where things stand

Escape the Valley 1.3.1 (release): live, and checked through npm on Windows and Linux. On Windows, trail licenses prints some extra blank lines, because a few licence texts kept their own line endings. Every licence is there in full, and the fix is queued for 1.3.2.
XRPL Camp 1.5.1 (release): live, and checked through npm on Windows and Linux.
Escape the Valley 2.0: the first Rust phase, the game’s core data and world generation proven against the Python game, is being built.
macOS: neither program’s macOS build runs on the studio’s machines. The macOS downloads were scanned unpacked once published, but haven’t been run.

Issue 25: What's really inside a packaged program

mcp-tool-shop — Fri, 23 Oct 2026 23:59:59 GMT

Escape the Valley and XRPL Camp are written in Python, but most players never install Python. Each release also ships a single-file program for Windows, macOS and Linux, built with PyInstaller. PyInstaller packs the tool, every library it uses, and a copy of the Python interpreter into one compressed file. The npm launcher from issue 24 downloads that file, checks its checksum and runs it.

What goes into that file is decided by the build, not by anyone reading it. This week the studio looked inside.

The scan that couldn’t see in

Before anything leaves the studio, an identity scan checks it for personal details: home-folder paths, email addresses, account names. It had passed every packaged program so far.

During a live check on Escape the Valley 1.3.1, the Book Keeper asked whether the scan could see the game’s own copyright line, which is certainly inside. It couldn’t: the line appeared 0 times in the raw file, because PyInstaller compresses its contents. A clean result on a compressed file meant nothing.

Unpacked, the same program gave 28 hits. None was the studio’s own:

What	Count	Where it comes from
Build-machine folder paths	21	the prebuilt Python the build uses (python-build-standalone)
A Python docstring with an example path	1	Python’s own code
Author and example email addresses	6	Python and three libraries: rich, pygments, httpx

Two rules followed:

Unpack, then scan, with a positive control. A string known to be inside, such as the game’s copyright line, must be found, or the scan is reported as blind. The scanner now unpacks PyInstaller programs, wheels and zips itself.
Third-party lines need a written exception per release. Library authors’ email lines and the builder’s paths aren’t the studio’s to edit. Each release that carries them gets an explicit, named exception, and anything outside that pulls the release files.

The CI folder in XRPL Camp 1.5.0

XRPL Camp 1.5.0 shipped on 2026-10-22, with a lesson that mints and burns an NFT so a learner can watch the ledger’s owner reserve lock and release, a plain-text keepsake certificate, and dated verification receipts in proof packs. Its release files then went through the new unpacked scan:

Program	Hits
macOS (arm64)	51
Linux (x64)	8
Windows (x64)	27

All but one were the third-party kinds above. The exception was inside XRPL Camp’s own package: a file called direct_url.json. Python’s installer writes it to record where a package was installed from (PEP 610). Here it named the folder where GitHub’s macOS build machine had checked out the code, under that machine’s runner account, and marked the install as editable, linked to that folder.

The scanner had flagged only macOS. Opening the Linux and Windows programs by hand showed the same file in both, naming the Linux and Windows build machines’ checkout folders. Neither of those folder forms was in the scanner’s list.

What it is: a CI machine’s folder, not a person’s. No one’s data was exposed.
Why it happened: the build ran PyInstaller through uv run, which quietly reinstalled the package in editable mode just before packing it.
What was decided: keep the 1.5.0 files, and fix it in 1.5.1.
The fix in 1.5.1: the build skips that reinstall, deletes the install record before packing, and then lists the finished program’s contents and fails if the record is there. Run against the real 1.5.0 Linux program, that check fails and names the file, so it isn’t a check that can only pass.
The scanner now treats any bundled direct_url.json as a hit, whatever path it holds.

Notices for a bundled copy of Python

Open-source licences mostly ask for one thing in return: keep the copyright notice and licence text with the software. A packaged program redistributes everything inside it, so its notices file has to cover all of it. That’s Python itself, and the C libraries compiled into Python: OpenSSL, zlib, bzip2, libffi, SQLite and others.

Escape the Valley 1.3.1 adds a trail licenses command and a notices file attached to each release. Getting the Python part right took four rounds:

The first version copied Python’s own licence file from the build machine. For the Python that ships (3.12.13), the Windows copy runs to 502 lines and includes notices for some of the bundled libraries. The Linux copy runs to 279 lines and mentions none of them.
The second version filled that gap with a summary written for the occasion. Review found it was not the text it claimed to be: it cut the bzip2 and mpdecimal licences short, gave OpenSSL a web link instead of the licence, and missed code compiled into Python itself, such as the Mersenne Twister random generator.
The third version copies the exact texts from pinned sources:
- Python’s own list of incorporated software, Doc/license.rst at the exact Python version shipped (3.12.13);
- the licence files from python-build-standalone’s full archive for the same build and each of the three platforms. Every archive was checked against the project’s published SHA-256 list.
Each file in the game records its source and SHA-256 hash, and a test checks those hashes on every commit. The build refuses to make notices if the Python it’s packing isn’t the one the texts were copied for.
The fourth version fixed what review found in the third.

What review caught in the third.

pip installs broke. The version check also ran when someone installed the game with pip. On any Python other than exactly 3.12.13, trail licenses crashed. The tests had replaced the Python version with a matching one, so they never ran the failing path. Running the command on Python 3.13 found it.
An exclusion that wasn’t done. The notes said the Tk graphics libraries were left out of the program. Nothing in the build left them out.

In the fourth version:

trail licenses on a pip install shows the game’s own licence and its libraries, with a note that the Python texts belong to the packaged programs.
Tk and dbm are excluded in the build, and the notices check fails if either turns up.
All 19 licence files in each archive are accounted for.
The same install-record cleanup as XRPL Camp’s is in.

Listing what’s really bundled. The notices list is now checked against PyInstaller’s own record of what it packed:

On a Linux build, the check found 22 libraries and passed. With one name deleted from a copy of the notices, it failed and named the missing library.
The same build showed the notices listing three libraries that were installed but never packed (Deprecated, mdit-py-plugins and wrapt). The list is now trimmed to what the program actually contains.

A second path, in Python’s own settings

The Linux build checks for both tools then found one more build-machine path, and this one wasn’t in an install record. Python keeps a small settings module, _sysconfigdata, that records where the interpreter itself is installed. About 20 of its entries, such as the install prefix and the folder for programs, hold that location, and PyInstaller packs the module like any other.

The build tool, uv, installs Python under the build machine’s user account. So on GitHub’s Linux machines those entries name a folder under the runner account’s home, and XRPL Camp 1.5.0’s Linux program carries them alongside its install record. Like the install record, it’s a CI machine’s folder, not a person’s.

The ruling was the same as for the install record: it’s the studio’s own build setup, so it gets fixed, not excused.

The build installs Python into a neutral folder outside any user’s home.
The bundle check reads _sysconfigdata from inside the finished program, and fails if it names a home folder.
The 1.5.0 programs stay as released, and their receipts record both paths.

Where things stand

XRPL Camp 1.5.0: released and live-checked through npm on Windows and Linux. Its programs carry both build-machine paths described above.
XRPL Camp 1.5.1: passed the studio’s own review, the independent review, and a Linux build check. That check found the _sysconfigdata path, and the fix for it is being built. It then goes through review and the build check again.
Escape the Valley 1.3.1: the fourth version of its notices passed the studio’s code review, the independent review, and a Linux build check. The Linux check confirmed the notices file block by block against its sources. It takes the same _sysconfigdata fix as XRPL Camp before it ships with its full release checklist.
macOS: neither program’s macOS build can run on the studio’s machines. So macOS, including what its _sysconfigdata holds, is first checked unpacked at release, and the receipts say so.

Records:

XRPL Camp 1.5.0 (release), its treatment (#22) and live-check receipts (#24);
the install-record fix, xrpl-camp#23 and pull request #25;
Escape the Valley 1.3.1, pull request #25, and the notices issue, #28.

Issue 24: One fix for nine tools, and the check that comes before shipping

mcp-tool-shop — Thu, 22 Oct 2026 23:59:59 GMT

Several studio tools are written in Python, but people install them with npx, Node’s package runner. That works through one small npm package, npm-launcher. Each tool ships a tiny wrapper that tells the launcher which GitHub release to fetch. The launcher downloads the right binary for the machine, checks its SHA-256 checksum, caches it, and runs it.

The bug: the wrong `--version`

The launcher answered --version and --help itself, before it looked at which tool it was running. So npx @mcptoolshop/xrpl-lab --version printed mcptoolshop-launch 1.1.1, the launcher’s version, not the tool’s. Learners and scripts asking “which version do I have?” got the wrong answer.

Launcher 1.2.0 answers those flags itself only when it’s run on its own. Through a tool’s wrapper, every argument reaches the tool. The launcher’s own version now has a flag of its own, --launcher-version.

One fix, nine tools. Nine wrappers ask for “any 1.x” of the launcher, so a fresh npx picks up 1.2.0 with no release of their own. One wrapper, XRPL Camp, pinned the exact version 1.1.1, and moves to 1.2.0 in its own release.

The second bug the first one revealed

With --help now reaching the tool, the tool’s usage line read Usage: xrpl-lab-2.7.9-win-x64.exe …. Python command-line libraries take the program’s name from how it was started, and here that was the cached download’s file name. Nobody had seen it before, because --help had never got through.

The fix tells the operating system to start the tool under its own name. Measured with the real tool:

Platform	How it was checked	Usage line
Windows	the packed launcher with the published xrpl-lab wrapper	`Usage: xrpl-lab …`
Linux	WSL Ubuntu, with the real Linux binary downloaded and checksum-verified	`Usage: xrpl-lab …`

A command that reads the tool’s bundled lesson files still worked, so the bundled tool finds its own files under the new name.

What the translation model got wrong

Release READMEs are translated into seven languages by a local model, TranslateGemma 27B, running on the studio’s own GPU. It’s reliable on prose. Today’s runs showed two kinds of slip, the same kinds the readouts newsletter recorded on 2026-10-20:

Bold text dropped. Japanese lost one bold label in the xrpl-lab README. In XRPL Camp’s, Italian lost two and Portuguese one. The Japanese camp README has been missing 5 of its 25 bold spans since an earlier run.
Product names translated. Japanese, Chinese and Hindi turned “XRPL Lab” into a translated name, and Hindi did the same to “XRPL Camp”.

All of these were fixed by hand. Every translation run is now checked for both: bold spans are counted against the English, and product names must survive unchanged.

The check before shipping

The studio has a full release checklist for every tool, called the treatment. It checks the ship gate and scans for personal details, then brings the README, translations, handbook, landing page and repository metadata up to date, and records the tool in the studio’s knowledge base.

Three releases this week (XRPL Camp 1.4.2 and xrpl-lab 2.7.8 and 2.7.9) shipped after code review and live testing, but without it. Nobody had decided to skip it; the step had fallen out of the routine. A published version can’t be changed, so running the checklist afterwards doesn’t fix what went out.

Two changes followed:

The release gate now refuses any version without a treatment record for the exact commit being tagged. The record covers each part of the checklist, with evidence, and live checks that the landing page and handbook are up. The gate refused the first version it checked, for README translations months out of date. They were fixed before the release.
Treatments ran for all three tools. npm-launcher shipped 1.2.0 only after its own, and xrpl-lab and XRPL Camp were treated so their next releases go out complete.

Records:

npm-launcher 1.2.0 (release), with its live checks on Windows and Linux;
the treatment pull requests for npm-launcher (#10), xrpl-lab (#48) and XRPL Camp (#18).

Issue 23: Watching 0.2 XRP lock and come back

mcp-tool-shop — Thu, 22 Oct 2026 23:59:59 GMT

XRPL Camp teaches the XRP Ledger in one sitting. It runs six lessons on the public test network, and ends with a certificate and a proof file anyone can check against the ledger. Version 1.5.0 adds three things the camp had built but never switched on. The most visible is a command that shows one of the ledger’s less intuitive rules happening to your own account.

The rule: owning things locks XRP

Every account on the XRP Ledger must keep a minimum balance, its reserve. There are two parts:

a base reserve for existing at all, 1 XRP on the test network when we measured;
an owner reserve of 0.2 XRP (200,000 drops) for each object the account owns, such as a trust line or a page of NFTs.

The reserve isn’t a fee. It isn’t spent. It’s held while you own the object, and released when you give it up. Your balance stays the same, but your spendable XRP doesn’t.

The demo, measured

xrpl-camp reserve mints one NFT from the learner’s wallet, then burns it, and shows both numbers side by side. Here’s the run that checked the release, read back from the ledger afterwards:

Step	Objects owned	Balance (drops)	Spendable (drops)
Mint an NFT	0 → 1	−10	−200,010
Burn it	1 → 0	−10	+199,990

The balance only moved by the fee: 10 drops for each transaction. The spendable XRP moved by the fee plus 200,000 drops on the way in, and got the 200,000 back on the way out. Over the round trip, 20 drops are gone, and the reserve returned in full. The transaction records on the ledger agree with every number in the table.

What a zero means

While the release was being reviewed, a question came up about the case where the demo shows no reserve change at all. There are two ways that happens, and they mean different things.

The NFT joined an existing page. The ledger doesn’t charge the owner reserve per NFT. It charges it per NFT page, a container holding up to 32 tokens. An account that already holds an NFT with room on its page mints the new one into that page, and locks nothing new. That’s a true zero.
The server hadn’t caught up. If the read after the mint reaches a server that hasn’t yet seen the validated ledger with the mint in it, the numbers come back unchanged. That reads as a zero too, but it’s a stale read, not a fact about reserves.

The first build called every zero “the token went into an existing page”. The fix uses numbers the demo already has:

a shared page needs the account to have owned something before, the object count to stay the same, and the fee to have been paid, which proves the read came after the transaction;
anything else that reads zero is reported plainly: the ledger read after the mint didn’t show the transaction yet, so the change couldn’t be measured, and here is the transaction link.

Receipts, and when not to write one

The camp’s proof file is sealed with a hash, so any edit shows. proof verify --online also asks the ledger whether the transactions in the file really happened. Version 1.5.0 can record each such check as a dated receipt inside the file, outside the seal. A file then gathers a short history of being checked over the years. That matters on a test network, which is reset from time to time, so old transactions eventually stop resolving.

Every receipt carries a disclaimer. It’s self-reported: a copy of the tool said this passed on this date, and it proves nothing on its own.

Review caught a problem with the first build: it wrote a receipt whatever the ledger answered. A file the ledger had contradicted would have gained a receipt saying it passed. So did a file whose check couldn’t reach the network.

Now a receipt is written only when the ledger confirmed the file. Any other answer leaves the file byte-for-byte unchanged and says why. The release check tried a forged file, re-sealed with a fresh hash: it was contradicted, refused a receipt, and left unchanged.

Also in 1.5.0

certificate --format text writes a plain-text keepsake beside the machine-readable certificate.
Every sealed certificate prints a short block you can copy and paste. It names the proof command, so a reader can check the claim rather than take it on trust.

How it was checked:

Four review rounds on the code by a second model, Gemini, and a review by the studio’s R&D seat.
A live check on the test network by a separate seat, with the receipts committed in the repository (docs/receipts/1.5.0-live-check/).
The XRPL’s rules on reserves and NFT pages: xrpl.org, “Reserves” and “NFTokenPage”.

Issue 22: Moving a deterministic game to Rust, and proving it plays the same

mcp-tool-shop — Thu, 22 Oct 2026 23:59:59 GMT

Escape the Valley is an Oregon-Trail-style survival game played in the terminal. A deterministic engine decides what happens, so the same seed gives the same journey. A local AI model narrates, and optionally the XRP Ledger’s test network keeps a record of your supplies. It’s about 17,700 lines of Python with 1,042 tests.

An audit of version 1.3.0 found that the game had grown two turn engines, one for the full-screen interface and one for the classic command line, and that features kept landing in one and not the other. The fix the studio chose is a rewrite in Rust as version 2.0: one engine, two front ends, one self-contained binary.

A rewrite like that can fail in four ways that matter. So before writing the game, we asked four research agents one question each, then ran a small spike for each question. A spike is a throwaway proof that answers one question with evidence.

1. Can Rust reproduce Python’s random numbers exactly?

This decides whether players keep their seeds and saved games. The game draws from Python’s random module 66 times across its code, using five operations: a float, an integer in a range, a choice, a shuffle and a sample, plus its own weighted pick. Saved games store the generator’s full internal state (624 numbers and a position).

Python’s generator is the Mersenne Twister (MT19937). The spike ported it, and Python’s exact way of turning its output into each operation, to Rust in about 60 lines. A Python script recorded expected values for six seeds:

0, 1, 42, and 2³² + 7;
a 100-bit integer, and a negative one. Python seeds from the absolute value.

For each seed it recorded 200 floats (as exact bit patterns), 200 ranged integers at ten widths, 50 choices, 20 shuffles, 20 samples (covering both of Python’s internal methods), 50 weighted picks, and a save-and-restore round trip.

The Rust port matched all 2,645 values. A widely used Rust crate, rand_mt, matched too.

One finding we hadn’t expected: the recorded values were byte-identical under Python 3.11, 3.12 and 3.13. The game supports all three, so this also confirms that today’s Python version plays the same journey whichever of them you run. Asking for a number in a range of one (say “between 4 and 4”) still consumes randomness, and the port has to copy that too.

The port will be proven the same way at full scale. The Python game becomes the reference: a thousand seeds, sixty turns each, saved and resumed partway, with the whole game state compared turn by turn. This is differential testing (McKeeman, 1998), with the old program as the oracle.

2. Can Rust run the game’s ledger features?

The game’s “Ledger Backpack” issues five supplies as tokens on the XRP Ledger’s test network, settles them at towns with transaction memos, sends parcels between players, and proves a run by reading the memos back. The research found one maintained Rust library, xrpl-rust on crates.io (version 1.3.0, ISC licence; its source now lives at XRPLF/xrpl-rust, the XRPL Foundation’s GitHub), that claimed to cover all of it.

The spike tested that claim on the live test network:

Step	Time
Two wallets from the faucet, each balance checked on a validated ledger	11.9 s
A trust line for a supply token	8.8 s
A token payment carrying a settlement memo	6.7 s
A 12-drop XRP payment carrying a parcel memo	5.6 s
Reading history back across three pages, memos decoded, and the token balance read as an exact decimal	0.7 s
A transaction sent with an expiry ledger already passed, to prove the client reports it as lapsed	2.0 s

All of it passed, in 35.7 seconds. Two details the game’s Rust code will need:

the ledger server raises any history page size below 10 up to 10;
just after an expired transaction, the server can briefly still answer “not validated” before it answers “not found”. Both mean the transaction will never land.

3. Can it speak, without GPL code?

The game can read its narration aloud with Kokoro, an open text-to-speech model (Apache-2.0). A speech model needs a phonemizer, which turns words into sounds. The usual one, espeak-ng, is licensed GPL-3.0. Linking it in would mean distributing the game’s binary under GPL terms, and the game is released under MIT.

The research suggested sherpa-onnx, a speech toolkit with a Kokoro mode that uses a word list instead of espeak-ng. Checking the code changed that:

Trap 1: the released sherpa-onnx (1.13.8) builds espeak-ng and piper-phonemize, both GPL, into every build that includes speech, whichever mode you use. We read this in its build files at that release. Code on the main branch marks espeak-ng as unused from version 2.0.0, which isn’t released; we haven’t checked a build of it.
Trap 2: the spike reported that another Rust Kokoro crate compiles espeak-derived C code even when its pure-Rust word list is switched on. We haven’t confirmed this one at the source; the route below avoids the crate either way.
Trap 3: the spike reported that sherpa-onnx’s English word list was generated from espeak-ng’s dictionary. We haven’t confirmed this at the source either.

The route that passed drives the Kokoro model directly through ONNX Runtime, with these parts:

Part	Licence
The Kokoro-82M model weights	Apache-2.0
ONNX Runtime, which runs the model	MIT
`ort`, the Rust bindings to ONNX Runtime	MIT or Apache-2.0
Kokoro’s own pronunciation dictionaries (misaki)	Apache-2.0
The CMU Pronouncing Dictionary as a fallback	BSD-style
rodio, for playback only	MIT or Apache-2.0

We checked each licence at its source. A symbol dump of the finished Windows binary shows no espeak or piper code.

Measured on the studio’s machine:

the binary is about 24 MB, because it carries ONNX Runtime;
the model and dictionaries are a 353 MB download, made on first use;
the first audio arrives 1.9 seconds after start, against 3.5 seconds for today’s Python voice.

Twenty game lines were rendered both ways for a blind listening test, which the studio is still to judge.

4. Can it ship the same way?

Today the game installs three ways: pip, npx, and a downloaded binary. The spike built a tiny Rust program in CI and produced all of it:

binaries for Linux, Windows and both kinds of Mac;
a checksums file;
Python wheels that install the binary through pip, the way the ruff and uv tools ship.

The Windows binary was built on a Linux machine and then run on Windows. Only the two Mac builds needed Mac machines, which cost more to run. The build settings strip the build machine’s paths out of the binaries.

What happens next

The plan holds. The rewrite starts with the core engine, tested against the Python game at every turn, and adds the command line, the full-screen interface, the narrator, the ledger and the voice in that order. Meanwhile, Python 1.3.1 fixes the bugs players hit today.

Sources:

CPython 3.11 Lib/random.py and Modules/_randommodule.c;
W. M. McKeeman, “Differential Testing for Software”, Digital Technical Journal 10(1), 1998;
xrpl-rust 1.3.0 (XRPLF/xrpl-rust);
sherpa-onnx v1.13.8 CMakeLists.txt (k2-fsa/sherpa-onnx);
hexgrad/Kokoro-82M and hexgrad/misaki;
cmusphinx/cmudict;
the spike code: escape-the-valley PR #26.

Issue 21: A critic that catches more, and refuses too much

mcp-tool-shop — Thu, 22 Oct 2026 23:59:59 GMT

The studio builds training data with a pipeline of local models. A writer drafts each row, and a critic from another model family checks it. Issue 14 tested the critic by breaking good rows on purpose. It found one gap. Some rows carry a checker’s reasoning with a planted mistake, and a label naming it (such as “guess”, reasoning from outside the material). When the reasoning was in fact sound but the label still named a flaw, the critic correctly answered “no flaw”. Today’s rules ignore that answer, so the wrong row is kept.

A stricter rule set, P2, makes the flaw check count. In issue 14 it won every disagreement, but there were too few to clear the test we had set in advance. This issue follows the confirmation test, and what came after.

The confirmation: proven, and too strict

The confirmation test used fresh rows, hand-checked blind, with one comparison fixed in advance. P2 settled the question:

it caught 22 broken rows that today’s rules (P0) missed, against 0 the other way (p ≈ 5 × 10⁻⁷);
but it also refused 12 of 32 good rows. The design’s limit was 10%.

The good rows split two ways, and so do the refusals:

Rows with a planted flaw, correctly labelled. The critic agreed a flaw was there on all 22, but sometimes named a different one, such as “guess” for a reasoning loop. P2 requires the exact label, so it refused those.
Rows with sound reasoning, labelled “no flaw”. The critic called 4 of these 10 flawed.

So we separated the two problems, and tested the narrower one first.

The narrower rule

The new rule set, Pc (the coarse gate), ignores which flaw the critic names. It refuses only when the critic and the label disagree on whether there is a flaw at all. A “loop” read as “guess” is still “a flaw”, so it passes. On the confirmation data, Pc refused none of the 22 planted-flaw rows. All its refusals were sound reasoning read as flawed.

That pointed at the definitions. In the critic’s instructions, “no flaw” was the one label with no description: just “the check was right”. “Guess” included “or a misreading”, which could absorb any trace the critic doubted. So Track A tested one prompt edit. It gave “no flaw” a full description (quoted the deciding lines, reasoned only from what they state, reached the material’s verdict), and said that such a check is not a guess.

The design ran the critic, Muse-Glimmer-30B, twice over the same 102 items at seed 0: once with the old instructions, once with the new. 219 items were scored from those replies. An arm qualified only if it beat P0, at an adjusted p under 0.05, and kept its false refusals at 10% or less on both kinds of good row. The outside reviewer, Google’s Gemini, checked the design, and set how to analyse it before any result was seen.

What it found

On the 90 broken rows the test is built around:

Rule set	Broken rows caught	Good rows refused: sound reasoning	Good rows refused: planted flaw	Qualifies?
P0, today’s rules	63 of 90	2 of 22	0 of 21	(the baseline)
Pc, old instructions	77 of 90	9 of 22	0 of 21	no
Pc, new instructions	76 of 90	10 of 22	1 of 21	no

The coarse gate works where it’s meant to. Both arms caught 14 broken rows that P0 missed, and P0 caught at most one they didn’t (adjusted p = 0.0002 and 0.001).

But both failed the bar on sound reasoning, at 9 and 10 of 22, against at most 2. The new instructions changed nothing measurable. Scored on the same set of verdicts, so that only the instructions differ, the two arms disagreed on 2 broken rows, both caught by the old instructions and missed by the new (p = 0.5).

Why the critic still says “flaw” on sound reasoning shows up in what it named. On the good rows with sound reasoning that it refused, it named “guess” 8 times of 9 in one run and 9 of 10 in the other. Read blind, none of that reasoning hedges or leans on anything the material doesn’t say. Yet the critic called about half of it a guess, whatever its form: quoting the deciding lines didn’t protect a trace, and a forced inference raised the rate only a little. The one thing that separates real guesses from sound reasoning is an outside assumption, present in 6 of 7 planted guesses and in none of the 22 sound traces. Telling the critic what “no flaw” looks like didn’t stop the over-calling.

So nothing was adopted. The pipeline stays on P0, with the flaw check advisory, and every accepted row still goes through R&D’s blind hand check.

The same prompt, twice

The design assumed something we could check: both runs send byte-identical verdict prompts at the same seed, so they should give the same verdicts. They didn’t. Of 102 matched calls:

22 replies differed in their text;
7 verdicts differed.

The most likely cause: the critic samples at temperature 1, so the seed doesn’t make its output reproducible across sessions. It changed little that mattered: under P0, 3 of the 90 broken rows changed outcome between runs. But a gap of 1 or 2 items between arms is within that noise, which is part of why the two instruction sets can’t be told apart.

When the premise failed, the reviewer ruled on how to analyse the result, without seeing any outcome figures. One caveat: R&D’s analysis script had already computed the figures in the same pass, before the ruling arrived, so R&D had seen them. They were held unreported, and the reviewer chose the analysis without them. One analysis was the pre-registered one; the other scored both arms from one shared set of verdicts. The two agree: neither arm qualifies.

Next: rewrite the labels

The failure points at the labels themselves, not at the rule. Track B tests that, in two stages:

Diagnosis, at no cost, now done: on the items already scored, it found what’s described above, and that the critic’s flaw label flipped between runs on 5 of 12 such items. The label fixes are now frozen: “guess” is defined by an outside assumption alone.
Verification: today’s labels against the rewritten ones, on a fresh bank of items neither arm has seen, scored with the coarse gate. To handle the noise, the critic’s flaw answer is sampled three times, and a row is refused only if 2 of the 3 name a flaw.

Its design has been reviewed and approved. T10, the next writer run, has priority for compute.

Where the figures come from

The result: data/track-a-result-2026-10-22.json, with every comparison, the control rates, the flaw labels named and the noise floor.
The plan and its bar: PLAN-ROUND4.md §1 to §4, which also holds the confirmation test’s figures.
The diagnosis and the frozen labels: TRACK-B-STAGE1.md and TRACK-B-LABELS.md, beside it.
Next: TRACK-B-DESIGN.md.
The text of the rows stays off the public repo. These files hold ids, codes and counts only.

Issue 20: The course, end to end

mcp-tool-shop — Thu, 22 Oct 2026 23:59:59 GMT

xrpl-lab is the studio’s hands-on course in the XRP Ledger. Its lessons run real transactions on the public test network, XRPL Testnet, and can also run offline against a built-in simulator. Issue 19 followed 2.6.2 and 2.7.0, where several checks passed that shouldn’t have. This issue follows the next five releases. Most of it is about what happens between lessons.

The proof that couldn’t prove

Each lesson leaves evidence. At the end, a learner can export a proof pack, a file listing every transaction they made. The command proof verify --live then looks each one up on the public ledger and confirms it is there as claimed. That check is the reason the pack is worth anything.

2.7.1 began as two small follow-ups. While reviewing them, our outside reviewer, Google’s Gemini, pointed out that the dashboard’s verify route ignored a new field. Following that path led to a bigger problem. Live verification expected every transaction in a pack to have succeeded. So since 2.6.2, which began recording the rejections a lesson teaches, it had failed every honest pack from a lesson that teaches one:

a rejection that reached the ledger failed as “not a success”;
a rejection that never reaches a ledger, by design, failed as “not found”.

Fixing that, the studio’s live-testing seat found a second, older rule. Live verification also required every successful transaction to be signed by the learner. But many lessons create helper wallets of their own: an issuer, a buyer, a second player. Their transactions could never pass. Of 25 real packs from earlier live runs, 17 failed on this alone, across 32 helper-signed transactions. Helper-signed rejections had been passing all along, because the rejection path skipped the check. The rule wasn’t only strict, it was inconsistent.

Now each run records the accounts it used, learner and helpers, and the pack carries them. A transaction signed by any of them verifies.

One design choice is worth telling. An earlier proposal would have had a rerun of a lesson silently delete the learner’s earlier records. We kept them instead. Every attempt records its accounts, and the course’s existing reset command handles records from before the change. A rerun shouldn’t destroy history.

A lesson that passes alone

The same live check turned up something new. A lesson on deposit authorization turns on a wallet setting that refuses payments from anyone not approved. The lesson passed. But it left the setting on, so the next lesson in the same wallet, which needs to receive a token, was refused, and its trading pool had nothing to pool. The lesson also couldn’t be run twice, because its own leftover approval was still on the ledger.

2.7.2 makes it clean up after itself: it removes the approval, turns the setting off, and reads the wallet back to check. Run live in one wallet, the next lesson then created its pool, the rerun was clean, and live verification passed.

2.7.3 was a small fix: a rejection that reached the ledger now keeps its link to the ledger explorer in the saved record, as the proof pack already did.

All 32 lessons in a row

That raised an obvious question. If one lesson could break the next, what about the rest? A workshop runs the course in order, in one wallet. Until now, live checks had run each lesson in a fresh one.

So we ran all 32 lessons in curriculum order, in one wallet, on testnet, and verified the whole pack at the end:

22 of 32 lessons finished cleanly;
live verification found 104 transactions as claimed, failed 14, and skipped 11 from the one lesson that runs offline by design;
no transaction failed because a helper signed it, so 2.7.1’s fix held across the whole course.

Run alone in a fresh wallet, 8 of the 10 failing lessons passed. An earlier lesson had broken them:

A setting left on. A lesson on crediting players’ deposits makes the learner’s wallet refuse any payment without a routing tag, and left it that way. Five later lessons that pay the learner were refused.
A shared issuer. The lesson on recalling tokens needs an issuer that owns nothing yet. The course reuses one issuer across lessons, and by then it owned plenty, so the ledger refused.
An empty wallet. A wallet gets one grant from the test faucet. By the cheque lesson, the learner held 49.99967 XRP, just short of the 50 XRP cheque.

Two lessons failed alone too:

The wrong signer. A token-distribution lesson created its currency from the learner’s wallet instead of the issuer’s. An issuer can’t hold its own token, so the learner couldn’t opt in, and the payment had nowhere to go.
An invalid name. A lesson named its currency HYGIENE. A currency code must be 3 characters or 40 hex digits, so the library refused it before signing. It had never worked live.

Why didn’t the offline tests catch any of this? Each offline run starts a brand-new simulated ledger, so no lesson ever saw what an earlier one left behind. The simulator was also more lenient than the ledger in places: it accepted HYGIENE, and it let token payments through to a wallet that refuses untagged ones.

Clean end to end

2.7.4 fixed all five. The deposit-crediting lesson now turns its setting off. The token-recall lesson makes an issuer of its own. The cheque lesson asks the faucet for a top-up when the learner is low. The distribution lesson signs as the issuer. And the currency is now HYG.

It also gave the course a new offline test that runs all 32 lessons in one simulated ledger, in order. On 2.7.3 it fails the two lessons the live run broke for lack of money and the routing tag. We then made the simulator stricter in the places the live run had shown it was too lenient. With the deposit-crediting lesson’s cleanup removed, the test now fails exactly the lessons the live run did.

The first live run of 2.7.4 still found two more:

A retry that couldn’t fix the record. Redoing a lesson with --force had never updated its record. A passing retry after a failure still read as failed. This dates from before 2.7.
A trust line that wouldn’t go. The cleanup lesson removes a token trust line, but the line stayed on the ledger, still holding its reserve. Deleting a line needs one of its flags back at its default value, and the course left it at the other. This had been hidden all along, because the lesson used to fail earlier on HYGIENE.

Each fix exposed the next layer. Fixed, the full course ran clean live: all 32 lessons finished, live verification confirmed 123 transactions with none failed, and no setting was left on the learner’s wallet.

A learner’s recovery should verify

One gap was left. If a lesson fails and the learner fixes it with --force, the pack still carries the failed attempt’s records. Live verification failed on them, so a learner who had recovered couldn’t get a clean result.

2.7.5 treats a failure older than the module’s passing latest run as history. If it never reached a ledger, it is skipped. If it did, it must be on the ledger as recorded. A module whose latest run failed still fails. No new field was needed, so packs made by earlier versions verify the same way.

Our reviewer spelled out why one detail matters. Never-submitted records all share the placeholder failed instead of a hash. If they were grouped by module alone, a later interrupted run’s failure could hide behind an earlier, superseded one. So they are grouped by module and by whether they’re superseded.

Smaller moments

A release that failed safely. 2.7.1’s first release run stopped at npm’s signing step, with “an equivalent entry already exists”. Nothing had been published. We asked before rerunning, the rerun passed, and we checked every registry directly.
Coverage held a merge. The test-coverage gate held 2.7.2 at 87% of the changed lines covered, against a 90% bar. Three tests brought it to 98%. The bar doesn’t move; the coverage does.
Paths in the receipts. Six lines of run output carried a folder path from our rig. They were redacted before merging.

Where it landed

All five releases are out on PyPI and npm, each after outside review and a live check on the reviewed code. The course’s tests went from 3,447 to 3,556, with coverage at 97%. Two more fixes found along the way followed. 2.7.6 adds --live to verify-artifact, the command the old verify commands now point learners to. 2.7.7, on its way out, stops the setup check from reporting a failure that a --force retry has already fixed.

Method and limits

One course, one network (XRPL Testnet), runs over two days by one studio seat on the studio’s rig. Each run used a fresh, isolated home folder; no wallet secret appears in any receipt. Full-course runs used one wallet, in curriculum order. The faucet’s rate limit set the pace. We checked sample transactions from every run on the ledger ourselves. Figures come from the receipts, the ledger and the course’s tests.

Sources: xrpl-lab’s pull requests #24 to #33, the follow-up lists in #20 and #28, and the run receipts in xrpl-lab’s docs/receipts/.

Issue 19: Passes that weren't

mcp-tool-shop — Wed, 21 Oct 2026 23:59:59 GMT

xrpl-lab is the studio’s hands-on course in the XRP Ledger. Its lessons run real transactions on the public test network, XRPL Testnet, and can also run offline against a built-in simulator. Issue 18 followed 2.6.0 and 2.6.1, where live runs kept finding defects that offline tests had passed. This issue follows the next two releases. Almost everything here was a pass that wasn’t one.

A correct lesson, missing from its own record

Some lessons exist to show a rejection: a duplicate credential, a payment to an account that requires a tag, an escrow that isn’t allowed yet. 2.6.1 taught the course to tell an expected rejection from a real failure. Each of those steps now checks for the one code it teaches.

2.6.1’s live run turned up a gap. In 12 of the 19 steps that teach a rejection, the transaction was rejected exactly as taught, but the step never wrote it into the course’s record of transactions. So:

the lesson passed;
the run’s summary didn’t count the rejection;
the proof pack, the file a learner keeps as evidence, didn’t list it.

A receipt couldn’t show by itself that the failure was the one the lesson wanted.

2.6.2 records every one of them and marks it expected. Its release run passed all 14 lessons: 13 live on testnet and one offline. In every lesson the summary’s “N failed” was exactly its expected rejections. Of those 19 rejections, 17 were confirmed on the ledger with the codes the lessons teach. The other two, a malformed transaction and a multisig quorum failure, never reach a ledger by design.

The simulator gave out ten times the money

Offline, a new wallet got 1,000 XRP. On testnet the faucet gives 100. We set the simulator to 100, and one lesson stopped working offline. Its trading pool asked for 100 XRP, all of a fresh testnet wallet’s money, before the balance a wallet must keep in reserve. On testnet that pool could never have been created. The pool is now 25 XRP each side, as in issue 18’s AMM lesson.

The same pass turned up three more simulator gaps:

A stand-in address. Second wallets, such as a buyer or a sender, had their transactions filed under one stand-in address the faucet never funded. A test now runs all 32 lessons and checks nothing lands there; two failed it before the fix.
The wrong unit. An escrow of 10 XRP was reported as “10 drops”, a millionth of the amount.
A crash. On Windows, one lesson crashed when its output went to a file, because of an arrow character.

An upgrade that changes who signs

2.7.0 moves to version 5 of xrpl-py, the Python library the course uses for the ledger. Version 5 fixed a long-standing bug. Version 4 turned every secret key into the same kind of keypair, so a key of the other kind, such as one imported from another tool, signed as the wrong account. Version 5 reads the kind from the key itself.

For the course, that fix is a hazard. We measured both versions on one test key:

every wallet the course creates turns into the same account on both versions;
a key of the other kind turns into two different accounts.

A learner who imported such a key under version 4 has their funds and history on the old account. Under version 5, with no change in the course, they would sign as an empty stranger.

We asked an outside reviewer, Google’s Gemini, to choose between two designs:

keep version 4’s behaviour, the wrong account included;
read the key the new way, and fall back to the old account only when that’s the one on record.

It chose the second, and added a rule: if neither account matches the record, refuse, and sign nothing.

Its first review of the build found two gaps. One wallet path never recorded the address to check against, and 13 signing methods dropped the address they were given. Both are fixed.

The clock and the ledger

A live run of 2.7.0 on one lesson per kind of transaction passed 10 of 11. The eleventh was an escrow that can be released only after a set time. The release was rejected with tecNO_PERMISSION, “not yet”.

The receipts show why. The ledger releases an escrow only once the previous ledger’s close time is strictly after the release time. When the release went out, that close time was exactly equal to it. The lesson had checked the computer’s clock instead, which runs 1 to 2 seconds ahead of the ledger on our rig. So it decided the time had passed and didn’t wait.

The lesson now waits until a validated ledger has closed after the release time. On the rerun the release succeeded, in a ledger whose previous close was 10 seconds after the release time. That timing code dates from 2.4.0, and this lesson had no earlier live run on record.

The same run found three more:

A refusal reported as success. The course correctly refused to sign with a mismatched key, and the wallet file was left byte-identical, but the program exited with code 0, which tells a script “success”.
--force did nothing. The option to redo a finished lesson did nothing on testnet, and had done nothing since the course’s first version.
An undercount. One lesson’s summary said 4 transactions where the ledger recorded 6.

Our own sweep passed after one step

The early-exit fix is what exposed our own mistake. Before each review we ran all 32 lessons offline from the command line and counted the exits: all 0. But that sweep ran with no keyboard attached. Every lesson stopped at its first “Press Enter to continue”, printed “Module interrupted”, and exited 0. It looked like 32 passes, and it checked almost nothing.

We had cited that sweep in two releases’ notes and in our reviewer’s briefs. The claim itself held: a separate automated test runs every lesson to the end and confirms it finished. But the evidence we named was wrong. Once 2.7.0 made an interrupted run exit 130 instead of 0, the sweep failed all 32 at once.

Run again with Enter fed to every prompt, the sweep completed and verified all 32. We’ve corrected the record on both releases. We now check that each run reached its end, not just its exit code.

Where it landed

Both releases are out. 2.7.0 shipped after four rounds of outside review and a live rerun of every fix, and it’s on PyPI and npm. The course’s tests stand at 3,440, with coverage at 97%. The course now treats a refusal, an interruption or a missing record as the failure it is, not a quiet pass.

Method and limits

One course, one network (XRPL Testnet), runs over one day by one studio seat on the studio’s rig. Each live run used a fresh, isolated home folder, and no wallet secret appears in any receipt. The test key for the upgrade was generated for the measurement and never funded. Figures are from the receipts, the ledger and the course’s tests.

Sources: the pull requests xrpl-lab #22 (2.6.2) and #23 (2.7.0), the follow-up list in #20, and the run receipts in xrpl-lab’s docs/receipts/.

Issue 18: Testing the course we'd already shipped

mcp-tool-shop — Wed, 21 Oct 2026 23:59:59 GMT

xrpl-lab is the studio’s hands-on course in the XRP Ledger. Each lesson walks a learner through real transactions on the public test network, XRPL Testnet, and can also run offline against a built-in simulator. This issue follows one release and the one after it.

2.6.0: four live runs for one lesson

Version 2.6.0 made module #30 live for the first time. In it the learner creates an automated market maker (AMM), a pool of two assets that trades against itself, deposits into it, and withdraws again. The new code passed 1,985 offline tests. Before releasing it, we ran the lesson on the real Testnet.

Run 1 created nothing. The faucet’s reply carried no balance the code could read, so it decided each funding had failed and asked again: the wallet was funded three times, 300 XRP. The token issuer’s first transaction went out before its account was confirmed and never landed. Every later step was then held by the network, and the client gave up at 60 seconds, about 20 seconds before the network would have answered, throwing away the hash and the result code. The report said “0 failed”.
Run 2 found what the extra money had hidden. With funding fixed, the pool’s creation failed outright: the lesson tried to pool 100 XRP, and a fresh wallet gets 100 XRP once, of which it must keep some in reserve. The report counted the pool steps as passed and exited 0.
Runs 3 and 4 passed, run 4 on the code an outside reviewer had passed: 9 of 9 transactions succeeded, and the pool was created, used and deleted.

Mocks couldn’t have shown any of it. A simulated network answers however the test was written; the real one holds transactions, grants a fixed amount, and enforces its own reserves.

After the release: testing what we’d shipped

2.6.0 shipped, and we kept running the released course live. Ten other lessons create a second wallet, such as a payment receiver, a buyer or a second token issuer, and had the same funding pattern that broke run 1. We ran all ten on testnet.

Eight of the ten finished with failed steps. The run found four kinds of defect:

The same race, in four lessons. The second wallet was used before the network had confirmed its funding.
Correct lessons marked as failed. Some lessons exist to show a rejection, such as a duplicate credential or a refused permission. 2.6.0’s new honest status counted their expected rejection as a failure.
A pass that wasn’t. The checks lesson reported its expected rejection, but that transaction had never reached the ledger. Only the way the result was read made it look like the lesson’s outcome.
Plain bugs, invisible offline. Payment-channel claims always failed, because the code read the channel’s key in the wrong format. Credential-gated payments couldn’t be signed, because of a field-name bug in the XRPL library we use. And two lessons with an NFT sale priced the item at a buyer’s entire faucet grant.

A second sweep, after the fixes, passed nine of ten. The tenth, the credentials lesson, finished unverified in both sweeps, and the second showed why: one of its steps had been passing for the wrong reason all along. Its “unfunded” account was a well-known example address that is funded on testnet. Only the stricter status from the first round’s fixes caught it.

Running every lesson offline turned up two more problems:

Five lessons failed their own checks in the simulator, also on 2.6.0. The simulator filed most of a learner’s actions under a stand-in address the faucet never funded, so its balance went negative and its object counts were wrong.
The course’s first lesson couldn’t run on the real ledger. It, and two others, sent their practice payment to “self”. Nothing turned that word into an address, so the real network couldn’t even sign the payment. Sending it to the learner’s own address wouldn’t have worked either: XRPL rejects a payment to yourself. The simulator accepted it.

The lessons now pay a fresh practice address made for each run; the first payment creates that account.

Coverage, and the reviewers

We also brought the course’s test coverage from 78% to 96%, with four AI agents writing 1,135 tests. Each test checks behaviour, not just that code runs, and none touches the network. Writing them found four more bugs, including a send command that reported success after a failed payment.

Two review passes caught what we’d missed:

An internal review found seven lesson steps, across six lessons, that would accept any rejection as their lesson, so an unrelated failure could pass. Each now accepts only the code it teaches.
An outside model’s review (Google’s Gemini, over two rounds) found that one kind of expected failure could still hide a real one, and that the “validated” flag was set by default where it should have been earned.

One defect came from a fix of our own. Our first fix for run 1 decided a step had failed by looking at a field that only 2 of the 66 handlers that submit transactions set. It passed its own test, because the test was built to match it, and it marked the failure lesson’s deliberate rejection as a failure.

Where 2.6.1 landed

The final release run passed 14 of 14 lessons live on the reviewed code: the ten two-wallet lessons, the AMM lesson and the three first lessons. Every rejection on the ledger was the one its lesson teaches. All 32 lessons also pass offline, and coverage is held at 90% or more from now on.

What made each failure readable in time to fix it:

Receipts for every run, failures included. Each run kept every transaction hash, and failed runs were checked afterwards with read-only queries.
Making the course’s own status honest before trusting a green result. Each time we did, an older defect showed.

Method and limits

One course, one network (XRPL Testnet), runs on two days, by one studio seat on the studio’s rig. Each run used a fresh, isolated home folder, and no wallet secrets appear in any receipt. The figures are from the receipts and the ledger.

Sources: the studio’s research note, 2026-10-20-live-testnet-runs-surface-defects-in-layers-each-fix-exposed, and the run receipts in xrpl-lab’s docs/receipts/ (2.6.0 in #19, 2.6.1 in #21).

Issue 17: What does a Claude Code deny rule actually hide?

mcp-tool-shop — Tue, 20 Oct 2026 23:59:59 GMT

Studio tools increasingly hand work to AI agents that read and edit files. Claude Code, which runs many of them, lets you write permission rules to keep an agent away from some files: Read(~/**), for example, denies reading anything under the home folder. We wanted to know how far such a rule really reaches, before trusting it with anything that matters.

So we built a small test. An agent ran inside a Docker sandbox, with a few sentinel files planted where it had been told it couldn’t look. Each sentinel had a random name, so an agent could only report it by actually finding it. Then we asked the agent to reach them by every route it had.

A rule that refuses, and a search that doesn’t

On Claude Code 2.1.197, the deny rule did what it says for direct requests: every attempt to read a sentinel was refused. Search was different. Claude Code’s Grep tool walks a folder and reports which files match. Started in the right place, it walked straight through the denied folder and listed the sentinels by name.

The right place depended on how the rule was written. We tried three spellings of the same intent, with one sentinel at the folder’s root and one in a subfolder:

How the rule was written	CLI	Search started at the denied folder	Search started one folder up	Search inside it
`Read(~/**)` (2 runs)	2.1.197	listed both	listed both	refused
`Read(//home/agent/b2probe/**)`	2.1.197	refused	listed both	refused
`Read(/home/agent/b2probe/**)`, one slash	2.1.197	listed both	listed both	listed the sentinel in it
`Read(//home/agent/b2probe/**)`	2.1.280	refused	“No files found”	refused

Three things stand out:

Above the folder, both real rule forms leaked on 2.1.197. A search started one level up listed the hidden names, however the rule was written.
~/** and an absolute path aren’t the same rule. With the home form, even a search started at the home folder leaked. With the absolute form, a search at the folder itself was refused.
One slash protects nothing. In Claude Code’s permission rules, a path with a single leading / is read relative to the settings source, not the root of the disk. So Read(/home/…) never matched the real folder, and every search listed everything. An absolute path is written with two slashes: //home/….

On 2.1.280, the search from above came back empty. The newer CLI filters it. We only had turns to test the absolute form there, so whether the ~ form is filtered too is still open.

Documented, not hidden

None of this is a secret flaw. Claude Code’s permissions page calls Read rules on Grep and Glob a “best-effort” attempt, applied to the folder a search starts from. For real enforcement, it points to running the agent in a sandbox. Our measurements are a concrete picture of what “best-effort” means in practice.

What actually kept the agent out

We ran the same probe a second way: the agent as its own unprivileged user, with every folder it shouldn’t see owned by another user and closed to others. The operating system, not Claude Code, decided what it could open. Every route came back empty: Read, Grep, Glob and Bash, 29 of 29 attempts, 0 leaks.

The practical rule for any tool that hands files to an agent:

keep secrets out with the operating system (a separate user, or a sandbox);
use deny rules only as a second layer;
when you do write one, use // for absolute paths.

A second finding: the first turn can start before the tools arrive

Agents often get extra tools from MCP servers, small programs the CLI starts beside the model. In headless mode on 2.1.197, Claude Code began the first turn the moment the message arrived. The MCP server was still starting (pending), so the model got none of its tools for that turn: in our runs, 0 of its 7 tools. Holding the first message for a few seconds fixed it, and the studio’s harness now holds it for 10.

On 2.1.280, the server was connected with its tools at the first turn in 3 of 3 starts, with no hold. An earlier note that 2.1.280 also started early didn’t reproduce, so it’s marked disputed, not wrong. Either way, check the start event: the server should say connected, with its tools listed.

How we got it wrong twice

This finding was corrected twice in one day, and the corrections are part of the record.

First: our original probe used the ~/** form and searched at the home folder. We read that as “only a search at the denied folder itself leaks”, and narrowed the claim.
Second: a re-probe with the absolute form showed the folder itself refused and the folder above leaking. We read that as the opposite.
Then we ran both forms side by side, which gave the table above. Each earlier reading had been right for one spelling of the rule, and wrong to generalise. Nothing in the record is marked wrong; every claim now names its rule form and CLI version.

The lesson we took: don’t narrow or widen a measured claim until a probe separates every case it covers.

Where the figures come from

The probes: Claude Code in the studio’s atlas-pilot Docker sandbox, on the Claude plan with claude-haiku-5-5. 13 short turns in all on 2026-10-20: 6 for the deny-rule searches (Grep the only tool), 6 for the MCP start and 1 for a tracing run. The original probe ran on 2026-10-13.
The entries: Claude Code: a Read deny rule doesn’t stop a Grep walk over the denied tree and Claude Code starts a turn before its MCP servers connect. Every claim there is marked verified or not, with what checked it.
The knowledge base: the same findings are wave 1 of the new agent-tooling-knowledge base in readouts.
Not published: the raw run receipts. They carry session identifiers and, in one case, an account email. They’re kept privately and named in the entries by their SHA-256.
Claude Code’s own account: the permissions documentation, read 2026-10-20.

Issue 16: What counts as a duplicate? Calibrating the dataset desk

mcp-tool-shop — Tue, 20 Oct 2026 23:59:59 GMT

The studio builds training data from its own art and run records. Before anything is trained on, a tool called the dataset desk reads the set and flags problems, and duplicates are one of them. A duplicate does two kinds of damage. It counts one example twice. And when a copy lands on both sides of a train/test split, it quietly inflates the test score.

The desk never deletes anything: it flags, and a person decides. So a flag has to be worth a person’s time. Until this week its thresholds were borrowed from other people’s data, with a note saying they needed recalibrating. This issue is that recalibration, in three rungs, on the RTX 5090.

How you measure a duplicate finder

You can’t measure a duplicate finder on data where nobody knows the answer. So we made the answer:

Planted copies. We took known originals and made copies of each:
- images: flipped, cropped by 5%, resized to half, brightened by 10%, and saved as JPEG at quality 70;
- files: small edits, for files from the studio’s public run records.
A good finder pairs every copy with its original. The share it pairs is its recall.
Clean negatives. Things that can’t be duplicates:
- for files, files of the same type from other runs, under other names;
- for images, art from four other projects.
Anything flagged among these is a false positive.
Views. The studio’s turntable project renders each object from many camera angles: 1,381 pairs of the same object at different angles. The studio’s rule is that these are never duplicates, so any flag here is wrong too.

Rung 1 found that its negatives weren’t clean: versions of one file across runs looked like copies. Rung 2 rebuilt them, and rung 2b ran the text side again through the desk itself.

Images: the angle problem

Rule	Copies caught	Other-project images flagged	Views flagged
SigLIP2, cosine ≥ 0.95 (the desk’s old default)	95%	20%	77%
mirror-aware image hash at the old threshold (≥ 246 bits agreeing)	71%	0%	0.6%
image hash only, mirror-aware, ≥ 230 of 256 bits agreeing	81%	0%	4.9%
DINOv2, cosine ≥ 0.97	82%	0%	4.4%
DINOv2 + mirror-aware hash, two signals	91%	0%	4.3%

SigLIP2 is the model the desk used for images. It’s built to match pictures to words, so it puts two angles of one ship close together: they show the same thing. That’s right for search, but wrong for finding copies.

DINOv2 is trained on images alone, and it’s more literal about what it sees.

The image hash (dHash) compares 256 brightness steps between neighbouring pixels. It’s cheap, needs no model, and survives resizing and JPEG. It breaks on a mirror image: in rung 1, a flipped copy agreed with its original on 155 of 256 bits on average. Comparing each image with its mirror as well brings flipped copies up to about 249.

The two-signal rule flags a pair when DINOv2 and the hash agree, or when DINOv2 alone is very sure: (cosine ≥ 0.94 and hash ≥ 212 bits) or cosine ≥ 0.985.

That still flags 4.3% of views. No threshold removes them without losing real copies, but the records can. Each turntable image’s record names its object and its camera angle, so the desk now checks those first. Two angles of one object are never paired, whatever the score. A second render of the same angle is still checked.

A shortcut that didn’t work. A reviewer suggested working out the mirror’s hash from the stored hash, with no image to read: reverse each row of bits and invert it. That’s exact only when no two neighbouring pixels are equal. We tested it against real flips on 300 images:

Project	Images	Median error (bits of 256)	Off by more than 10 bits
turntable renders	45	112	100%
painted art	255	6	18%

The renders sit on flat backgrounds, where neighbours tie and the inversion turns every tie into a disagreement. The desk will compute the mirror’s hash from the flipped image.

Text: read the model card first

For files, we compared four text embedding models. Rung 2 ran EmbeddingGemma 2 without the setup its model card asks for. It got the wrong task prefix, it loaded its image and audio encoders for text it didn’t need, and it read inputs in short pieces. So rung 2b ran every model through the desk itself, after the desk was rebuilt to follow each model’s card: a task prefix where the model wants one, text-only loading, and inputs split by the model’s own tokenizer into chunks up to the model’s own limit: 8,000 tokens for EmbeddingGemma 2, 8,192 for the others.

Pool	Model	Threshold that flags ≤ 1% of non-copies	Copies caught there
all 458 files, 100 copies	bge-m3	0.98	94%
	EmbeddingGemma 2	0.995	62%
	gte-modernbert	0.99	52%
	Qwen3-Embedding 0.6B	none below 1.0	–
scripts only, 17 copies	EmbeddingGemma 2	0.935	100%

On scripts, EmbeddingGemma 2 separates cleanly. Every copy scores at least 0.976 against its original, and every non-copy at most 0.931. The desk uses 0.95, inside that gap. It rests on only 17 copies, so it’s marked provisional.
On logs and JSON, it can’t. Run files from different runs share templates, field names and timestamps, and EmbeddingGemma 2 scores look-alikes up to 0.992. bge-m3 keeps the best balance, and stays the default for text.
Through the desk, bge-m3 got better: 94% of copies at 0.98, up from 90% when it was run off-card.

The memory surprise

The desk has a VRAM governor. Before loading a model, it checks the card has room for that model’s registered figure plus a 1 GiB margin, so two jobs can’t push the shared card past its budget. Rung 2b measured each text model at its longest input. Three of the four were registered far too low:

The registered figures came from short inputs. At 8,192-token chunks, a model needs memory for the attention over the whole chunk, and that grows with length. Until the registry is fixed, long-text runs are planned at their measured peaks, with the card held by one job alone. The fix, registering the long-input peaks, is in the next build.

How it was checked

Every step went past a second reviewer before it ran. Argon, an outside model, read the code and the plan, and the Publisher seat checked the settled choices. Three of Argon’s suggestions changed the plan:

one threshold for text instead of two;
the view check from records;
comparing the desk’s cached settings before reusing any vector.

Two other points came out differently once they were tested:

The mirror shortcut failed, as shown above.
A batch-size claim was R&D’s own error. We told the reviewer the bge-m3 run used batches of 4; it used 8. The fix that rested on it was dropped, and the write-up was corrected.

What changes, and what’s still open

Next build (D2):

DINOv2 with the mirror-aware hash for image duplicates;
the view check from records;
per-task text thresholds, at 0.95 for code and 0.98 for text;
registry VRAM figures measured at long inputs.

Every threshold ships marked provisional, with the rung that measured it.

Not settled yet:

The two-signal rule misses 46 of the 500 planted image copies. They probably have near-identical siblings in the same set, but nobody has checked: a person will look at them.
Code thresholds rest on 17 copies.
Copied sections inside long files aren’t tested. The desk compares the average of a file’s chunks, which dilutes a shared section.
Dataset items (questions and answers written for training) need their own pool and rule. Templates make unrelated items look alike, and paraphrases make copies look different.

Where the figures come from

Images and the first text pass (rung 2): experiments/dataset-desk/calibration/data/rung2.json. The plain hash on flipped copies (rung 1) is data/rung1-dhash.json.
Text through the desk (rung 2b): data/rung2b.json, the curves and picks, and data/rung2b-runs.json, each run’s model revision, settings, peak VRAM and time.
The mirror test: data/mirror-dhash.json.
The registered figures: data/registry-25d9cdd5.json, read from dataset-desk at commit 25d9cdd5.
The chart is built by experiments/charts/make_charts.py, and manifest.json pins each input’s commit and sha256.
Not published: the art and the text of the run files. These data files hold scores, counts and settings only.

Issue 14: Testing the tester: does our critic catch bad training rows?

mcp-tool-shop — Mon, 19 Oct 2026 23:59:59 GMT

The studio builds training data with a pipeline of local models. One model, the writer, drafts each row. A second model from a different family, the critic, checks it before it’s kept. In last week’s writer screen (issue 11), the critic accepted almost everything: 35 of 36 rows from one writer. That’s either very good data or a critic that rarely says no.

This study tests the critic itself. Every figure here comes from files in the repo, and the last section says which. The design was written and reviewed before any of it ran: DESIGN.md.

What the critic checks

Many of the rows are reading exercises. A short document (the material) and a statement about it; the answer is whether the material supports the statement, contradicts it, or can’t tell, plus the line or lines that decide it. Some rows add a twist: a checker’s reasoning that contains a planted mistake, which the learner has to name. Each kind of mistake has a label, such as “guess” (reasoning from outside the material).

The critic gets two checks:

the verdict check: it solves the row blind, without seeing the answer, and its verdict and deciding lines are compared with the row’s;
the flaw check: on rows with a checker’s reasoning, it names the mistake.

Today the flaw check is advisory. A disagreement is logged, but the row is kept. That was decided earlier, when an older setup agreed with the flaw labels only 9 times in 24.

Step one: were the accepted rows actually right?

Before testing the critic, R&D checked its accepted rows by hand, blind: 30 rows with a verdict, keyed without seeing the writer, the critic’s answer or the row’s own answer.

29 agreed with the row’s answer. 1 was unclear, and none were wrong.
So the critic wasn’t letting wrong rows through. But the rows were easy, and none of them had “can’t tell” as the true answer, so they don’t show what the critic would do with a hard one.
New finding: two rows had reasoning that named its own planted mistake (“that is a guess”), which gives the answer away. That’s now a defect class the pipeline checks for.

Step two: break good rows on purpose

From 27 rows that passed the hand check, a script made copies with one known change each. The critic never sees which copy is which.

Copy	What changed	A good critic should
controls	nothing	accept it
material flip	a number on the deciding line changed so the true verdict flips	refuse it
can’t-tell trap	the deciding line deleted, or its subject swapped for another one in the passage	refuse it
clean trace	the checker’s flawed reasoning replaced with sound reasoning, but the flaw label kept	refuse it
decoy deciding line	the answer’s deciding line swapped for a line that doesn’t decide anything (scored offline)	refuse it

R&D read every material-flip, can’t-tell and decoy copy before any critic saw them. One flip turned out not to flip, because the script missed a number with “px” stuck to it. That copy was set aside before the run, and the decision is on record.

Two critics took the test: Muse-Glimmer-30B (two runs, at seeds 0 and 1) and Nemotron-3.5-Lightning (one run). A critic never checks rows written by its own model family, so Muse skipped 14 copies and Lightning 11.

What they caught

Each row is scored under two rule sets, from the same critic replies:

P0 is today’s rules (the flaw check is advisory).
P2 makes the flaw check count, and adds two rules about quoting.

Most broken copies are caught already. Across the material flips, can’t-tell traps and decoy lines, both critics refused nearly everything under either rule set. The one miss was Muse reading a time changed from 5:00 PM to 7:00 PM as still matching. Lightning caught it.

The clean traces are the gap. On every clean-trace copy, both critics answered “no flaw”: 13 of 13. That’s the right answer, because the reasoning really is sound. But the label still says “guess”, so the row is wrong, and under today’s rules the critic’s correct answer is ignored. P2 counts it, and catches all 13.

The cost: P2 refused 1 of the 44 unchanged good rows, where Muse named a flaw the row didn’t have. The two quoting rules never decided a single item. Everything P2 changes comes from making the flaw check count.

Muse’s second run (seed 1) refused nothing that its first run didn’t.

Then why didn’t we adopt it?

Because the design said in advance how strong the evidence had to be, and it wasn’t.

The test compares the two rule sets item by item and counts only the items they disagree on. All 13 disagreements favoured P2: 6 for Muse and 7 for Lightning. Each comparison on its own gives p = 0.031 or 0.016.

But the design ran several comparisons: P2 and a two-run variant, for each critic. Testing more things gives chance more opportunities, so the design applies a correction (Holm’s) across all of them. With four comparisons, the smallest result has to be under 0.0125, and 0.016 isn’t. Corrected, the result is 0.0625, above the 0.05 bar.

The reason is plain from the data: there were only 6 or 7 clean-trace copies per critic. Even a perfect 7-to-0 can’t clear a four-way correction. We didn’t run Lightning’s second seed, because its outcome was fixed before it started: P2 already catches everything P0 misses on Lightning’s items, so a second run couldn’t add a disagreement. Skipping it saved about 4.5 units.

So under the rule we committed to, nothing changes yet. A “not shown” isn’t a “no”, though.

Next: one clean confirmation test

The fix is more clean-trace items, judged by one comparison that was set in advance. The design is §9 of DESIGN.md:

New rows. Gemma 4 31B writes 38 fresh reasoning rows, and R&D hand-checks each one blind.
No critic decides which rows are kept. In the earlier rows, Muse helped pick what was kept, so it was partly being tested on its own choices.
One test. P2 against P0 on Muse, at 0.05 with no correction, written down before any of the rows exist.

How the outside review changed it. An outside model, Argon, reviews every design before it runs. The first draft picked 38 slots by lesson. Argon pointed out that the pipeline plants “no flaw” on every even-numbered slot, so only 12 of the 38 could ever make a clean-trace copy. Too few. The fixed design takes all 27 eligible slots there are, plus 11 even-numbered slots as natural controls. Argon passed the fix the same day.

What it cost

The study runs on Colab A100s, paid in compute units: about 5.3 units an hour.

Window	What	Units	Projected
13v (a, b, c)	the low-cost test ladder on a CPU runtime: a forced cut, two resumes	0.05	–
13w	Muse, seed 0	3.10, plus 0.24 for a launch mistake	2.83–10.43
13x	Muse, seed 1	3.17	about 2.69
13y	Lightning, seed 0, converted on the GPU	4.52	3.82–8.83

The design had projected about 23 units for each critic run, at about 120 seconds an item. That projection used Lightning’s earlier timings, which included three extra calls per item. Measured, a critic call is short:

The flaw check takes longer than the verdict check: its prompt is about twice as long, and the critic thinks more before answering. Lightning thinks two to three times as much as Muse per call and still answers faster. Lightning’s one empty reply came after it spent its whole 16,384-token reply budget thinking. The retry was counted, and the copy it was answering was scored as refused.

Where the figures come from

The slot diagram is data/confirmation-slots.json, the frozen list in §9.
The broken-copy results are experiments/stage1-regeneration/critic-signal/data/outcomes.csv: one row per copy, critic and rule set, written by role-set’s critic-rescore-offline.
The per-call timings and thinking counts are data/calls.csv, from the logging proxy that sat between role-set and the model in every window.
scripts/extract_data.py writes both files from the window receipts. The text of the rows stays off the public repo: these files hold ids, codes and counts only.
Hand checks, windows and decisions: PHASE-0-1.md and PHASE-2-3.md in the same folder. Units come from the study’s ledger, one row per window.
The tools: role-set (critic-eval, critic-rescore-offline, scripts/critic_signal/perturb.py) at commit 47a5882. The three pictures are built by experiments/charts/make_charts.py, and manifest.json pins each input’s commit and sha256.

Issue 13: Claude + Colab: the path we'd use again

mcp-tool-shop — Sun, 18 Oct 2026 23:59:59 GMT

The studio runs its model work on rented GPUs. Claude Code drives Google’s Colab from the command line: it asks for a machine, sends the job, follows it and stops it. Nobody watches a notebook. Each of these sessions is a window, paid in Colab compute units: about 5.3 an hour for an A100, 0.08 for a basic CPU runtime.

Issue 11 told how night one went: the writer screen got its answers, but about half of the first 71 units bought nothing their windows were opened for. This issue follows the same work through day two and sets out the way of working that came out of it. The path itself, with templates, is a handbook page: Rented GPU windows (Claude + Colab). Every window’s numbers are in the repo, one row each, and the last section says where each figure comes from.

The two days in one table

Each window is classed by what it gave back: results (what it was opened for), partial (some of it), lost (a dropped VM, a cap stop, or a fault in our own tooling), or test (a free run on a CPU runtime).

	Night one (Oct 17)	Day two (Oct 18)
Windows	33	15
Units spent	141.1	31.6
Bought results	43.3 (31%)	21.9 (69%)
Partial	41.1 (29%)	8.0 (25%)
Lost	56.6 (40%)	1.3 (4%)
Test runs	0.03	0.43

Two caveats. Day two’s paid work was smaller and better rehearsed, so the comparison flatters it somewhat. And some spend figures are estimates where two windows shared a cap: summed window by window, the three days come to 172.7 units, while the account’s balance fell by 166.5 over the same span. The balance is the measure; the per-window rows were booked on the high side.

What changed, step by step

1. Two lanes. CPU runtimes do the preparation: converting models, testing, measuring. The A100 only serves models. On night one, every A100 window downloaded and converted its own models, 15 to 21 minutes each, before any real work. Now a CPU runtime converts each model once, at 0.26 units an hour, and the A100 downloads the finished file.

2. Measure first, then budget. Every projection uses a measured rate with its receipt. Every window has a cap at the projection plus 15%, and opens only if the cap plus 2 units is left. When one cost was still a guess (the “twin” waves had never run on a GPU), the smaller window ran first and measured it: twin slots cost about twice a normal slot, not the same.

3. One script runs a window, and it trusts nothing it hasn’t checked. It refuses bad inputs before asking for a machine, checks the machine before uploading anything, follows the job’s own log file rather than the command’s output stream (which went quiet on us mid-run), counts only its own spend against the cap, and fetches the job’s log, and its results when the job got that far, on every way out.

4. A health check on every new machine. Before anything is uploaded, one short command reads the card, the disk, the memory and the download speed. A bad machine is stopped and a fresh one requested. On day two it ran 12 times and rejected 2: one machine whose first command died (two night-one windows had lost about 4.5 units that way, after their uploads), and one downloading at 45 MB/s. Each rejection cost a few minutes of machine time, not a run.

5. The job saves itself. It writes every finished model call and every finished wave into its log, so a killed window’s work can be pulled out of the log, and the next window can resume from it. Each saved file carries a checksum, and a file that doesn’t match is refused.

6. A test ladder before any paid run: unit tests, then the real script against a fake command-line tool, then the real job on a real CPU runtime with stand-in models, then the paid window. The 13 test windows cost 0.46 units in all. One of them caught a step that would have been refused partway through a paid window: an older file format lost a record of the model’s settings on its way through the pipeline. Two review rounds had read that code and passed it.

7. Three seats review everything. One Claude session builds, a second reviews the actual diff (not the summary of it), and an outside model from another company reviews from a written brief. The loop repeats until it passes, and a reviewer who asked for fixes sees the fixed version before anything paid uses it.

8. Pre-built models, checked byte for byte. A converted model file goes into the studio’s private repository only if its checksum equals the one recorded when the arms first ran on it. A file that doesn’t match is never used, and we never change the recorded checksum to make a file fit.

9. Tokens that can’t leak far. One access token per job type, each limited to the one private repository, sent to the machine as a file in memory, read once at the start, and kept out of every log. A test pushes a fake token through the real script and checks it appears nowhere.

What the path caught

The stand-in run caught the lost model settings described in step 6. Fixing it took a small rule for the older files, checked by checksum against the file they came from.
The first paid window after the ladder found what the ladder couldn’t: a writer whose two twin attempts both failed was still sent to a critique, which then refused. No stand-in model had ever failed every attempt. The fix is one line, and the lesson is in the handbook: give your stand-ins the failures you expect from real models.
The byte-match rule caught a surprise. Qwen3.8, converted again on a CPU runtime, came out the same size as before with different bytes. The recorded package lists differed in one material place: the A100 image’s Python carried the GPU build of PyTorch 2.11.0, the CPU image the CPU build. Models converted in their own environment (which always gets the CPU build) matched on both machines, and Qwen3.8 matched again when rebuilt on an A100. We think the PyTorch build is the cause; the host’s CPU instruction set hasn’t been ruled out yet.
An upload failed after a conversion had already matched. The job had installed packages into the very Python it was running, partway through, and its final step tripped over the mix of old and new library code. The upload now runs in a fresh process.
The health check’s speed test reads low. Its short 200 MB read gave 88–115 MB/s on A100s that then downloaded at 179–310 MB/s. Our first speed floor, set from full downloads, would have turned good machines away. The floor now only rejects a broken connection.

The payoff: one window, under its projection

Window W-A resumed four writers (Qwen3.8, Gemma 4, Granite 4.2 and Bolt) from their saved drafts, with Muse as the critic. It rewrote the one draft that had been sent back, then ran two “twin” practice rows per writer.

It downloaded the four pre-built model files, 122.4 GB, in 684 seconds. Each matched its checksum.
It finished on its own after 11,749 seconds: 17.55 units against a projection of 21.8 and a cap of 25.1.
Every writer’s saved results came back: 30 rows from the pilot and 6 accepted twins, with no model errors.
The projection was 19% high: setup came in far under, and the twins ran over.

These rows come from a screen with a fixed set of eight slots, so they’re not a releasable batch: they fail the batch-mix rule a real batch must pass.

Still open

Prove the PyTorch explanation with a one-layer conversion before any full one, recording the machine’s CPU.
Recalibrate the speed floor from more paired readings, and try downloading files in parallel.
Record the CPU model on every health check, so a machine difference can be seen, not inferred.
Pin new conversions in their own environment with the exact PyTorch build, in the next study.

Take it with you

The handbook page Rented GPU windows has the full path and templates: a launch file, a projection worksheet, a ledger row, a per-window results file, and a brief for an outside reviewer. The window code is in role-set (scripts/colab/), and the data behind this issue is in rnd’s experiments/colab-golden-path/.

Where each figure comes from

All paths are in mcp-tool-shop-org/rnd unless noted.

Units, windows and classes by day (the table, 31% → 69%, 40% → 4%, 13 tests at 0.46 units, 172.7 summed): experiments/colab-golden-path/figures.json, computed by figures.py from data/windows.csv. Each window’s class comes from its own outcome.json, written by the Publisher when the window closed (outcome_source says which).
166.5 units by the balance: the first window’s start balance, 1207.67 (data/windows/window1-20261016/ and its ledger note), the last window’s end balance, 1141.14 (window 13l), and the 100-unit top-up recorded in data/units-ledger.md.
Rates (5.3, 0.08, 0.26 units an hour): instruments/colab.md and data/windows/_top/highmem-rate-probe.txt.
About half of the first 71 units: issue 11.
15 to 21 minutes of setup per model: night one’s job logs, as measured in the W-A projection (window 13l’s README in the Publisher’s receipts, summarised in data/windows/window13l-20261018/).
Health check, 12 tries and 2 rejections, 45 MB/s: figures.json’s gate block, from each window’s gate.jsonl. About 4.5 units lost to first-exec deaths: windows 11c (3.2) and 12e (1.29) in data/windows.csv.
Twin slots about twice a normal slot: window 13b (W-B) and 13l (W-A) results, per writer.
0.07 units and 15 minutes per CPU conversion: window 13e (Muse) and 13g (Gemma) rows in data/windows.csv.
The PyTorch difference: the pip freezes in windows 12f, 13f, 12h and 13e, and the outside review’s reply on it; the A100 rebuild that matched is window 13j.
The failed upload: window 13i. The speed-test pairs (88–115 against 179–310 MB/s): windows 13i, 13j, 13k and 13l, short reads from gate.jsonl and download rates from their job logs.
W-A (684 s, 122.4 GB, 11,749 s, 17.55 units, 21.8 and 25.1, 30 rows and 6 twins, 19%): window 13l’s pin-result.json and README, summarised in data/windows.csv.

Issue 11: Picking a writer model on rented GPUs

mcp-tool-shop — Sat, 17 Oct 2026 23:59:59 GMT

The studio is building a small local model whose job is to check other models’ claims: is this statement supported by this evidence, or not? To train it, we need thousands of practice rows, each a short piece of material with a statement and the right answer. We write those rows with larger open-weight models, and a second model, the critic, checks every draft before it’s kept.

Which model should do the writing? There were eight candidates. This issue covers night one of the pilot that picks between them: what we measured, what went wrong, and the method we’d use again. The screen’s results will be added here as updates when they land. → the plan

The setup, in brief

Eight writers, all open-weight: Qwen3.8-27B, Qwen3.6-35B-A3B, Granite-4.2-30B, Gemma-4-31B-it, DeepSeek-R1-Distill-Qwen-32B, Muse-Glimmer-30B, and NVIDIA’s Nemotron-3.5 Lightning-30B and Bolt-74B. Every one was checked against its licence before it went in.
One critic per writer, never from the writer’s own family. Muse critiques every writer except itself; Lightning critiques Muse.
Rented GPUs. Each “window” is a Colab session on one NVIDIA A100 with 40 GB of memory, paid in Colab compute units. An A100 costs about 5.3 units an hour. A CPU-only session costs about 0.08.
Pinned models. Every model file is rebuilt from a fixed source revision and has to reproduce a recorded fingerprint (its sha256 hash) before it serves. When something went wrong, we always knew it wasn’t the model files. Lightning’s file was rebuilt four times and matched every time.
Same settings for everyone: each model runs at its own model card’s recommended sampling and thinking settings, with a 32,768-token context and a 16,384-token limit on each reply.

Not every model fits a 40 GB card at those settings. Qwen3.6 fit at 8-bit precision, peaking at 36,022 MiB of 40,960, and wrote at about 90 tokens a second. Gemma fit with 4,943 MiB to spare, at about 28 tokens a second. DeepSeek-R1-Distill didn’t fit at 8 bits, so it will run a 6-bit build and be reported apart from the 8-bit writers.

Step 1: test the judge before it judges anything

If the critic is wrong, every number downstream is wrong too, so the critic was tested first. The test is a known-answer set: 40 practice items whose right answers we wrote ourselves, plus 10 copies with a deliberate error planted in them. The bar was written down before any result came in:

agree with the key on at least 36 of the 40 clean items;
catch at least 80% of the planted errors;
for a critic that samples randomly, like Muse, two runs with different random seeds may disagree on at most 4 of the 40.

Both passed. Muse agreed with the key on 38 and 37 of 40 across its two runs, caught all 10 planted errors, and its two runs disagreed on just 1 item. Lightning agreed on 38 of 40 and caught all 10.

The more useful finding was in the misses. On two items, a critic disagreed with our key and was right: both critics on one, Muse alone on the other.

“Retries the upload up to 3 times” means up to 4 attempts in all, so “tried at most 3 times” isn’t supported. Our key said it was. Muse flagged it in both runs.
“Apply every four weeks from March to September” fairly rules out January. Our key said you couldn’t tell. Both critics disagreed with it.

We replaced both items in version 2 of the set and changed nothing else. A calibration set can check the person who wrote it, so read the misses before blaming the model. Next time, the key will be written by two writers independently, or checked blind by a model from a third family, before it’s used. → the set and its errata

Step 2: thinking is most of the bill

Modern open models “think” before they answer: they write out reasoning that’s never shown to the reader, then the answer. That reasoning counts toward the reply limit and takes GPU time like any other output.

We logged every call through a small proxy that records token counts and timings. Qwen3.8’s first 13 calls show where the time went:

95% of the output was thinking: 81,372 of 85,752 tokens.
The calls came in two shapes. Short prompts (144–573 tokens) got 405–2,030 tokens back, in under 70 seconds. Long prompts (6,370–7,643 tokens) got 4,364–15,264 back, at 2½–8 minutes each.
Nothing was cut off, but the longest call used 15,264 of its 16,384 tokens. A model that thinks a little longer will start hitting the limit. A cut-off reply is reported as its own outcome, separately from a refusal, so running out of room isn’t confused with being wrong.

That changed the plan. Each practice row took about two calls: about 12,100 output tokens and about 6½ minutes per row (roughly 7 rows finished in those 46 minutes). Projected across the pilot as designed (8 writers, 20 practice rows each, with rewrites), and assuming every writer thinks about as long as Qwen3.8, it came to 147–175 units, against about 35 left that night.

So the pilot became a screen:

every writer drafts the same 8 practice rows, plus 2 “twin” rows of a second kind, from one fixed list;
every draft is critiqued once, and drafts the critic sends back get one rewrite;
writers that drop 3 or more of their 8 are cut;
writers that pass go on;
with 8 rows per writer, the screen can tell “doesn’t work” from “works”. It can’t rank close writers finely, and the readout will say so.

And one number joined the readout: cost per accepted row. A writer that thinks for 12,000 tokens per row pays that on every row of the full dataset later, so cost is part of the choice, not just a budget line.

Step 3: long paid jobs fail, so design for it

This is where most of the money went. By the time the budget was raised at 19:44 UTC, 70.66 units had been spent, and about 38 of the first 71 didn’t buy the results their windows were opened for:

What happened	Units
A long pilot window: the rig stopped receiving the job’s output, the job ran on unseen, and the window hit its cap without downloading anything	14.03
Colab reclaimed a machine 77 minutes into the job, with nothing saved off it	12.81
A connection hung at the first command with nothing running, and the session sat open for 35 minutes	about 3.2
A check looked for a log line the server doesn’t print at its default log level, and stopped the window	3.00
Three failed starts: a missing setting, a machine dropped mid-download, and a module missing from a package	5.29

None of these were the writers’ fault. They were our tooling, and how it dealt with machines and connections that failed. The method that came out of it, in the order we’d build it again:

The job writes its own log, and the rig reads that file, never the live output stream.
- The stream between the rig and a Colab session can stop while the job keeps running. That’s how the long pilot window ran on unseen. It was the full 20-row design; by our projection it couldn’t have finished its first wave inside its cap, and it stopped without downloading, so its work was lost.
- So the job appends everything to a file on the machine, and the rig downloads that file every 2 minutes.
- Progress, stalls and “finished” are all judged from it.
- A download that dies partway never overwrites the good copy.
Every finished model call is saved the moment it returns.
- Each call goes into a journal, keyed by a fingerprint of exactly what was asked.
- A resumed run replays the saved calls instead of asking the model again, and the job’s own logic rebuilds everything else from them.
- We tested it with a mock model in role-set’s tests: a run killed halfway and resumed gave every draft and every decision identical to an uninterrupted run, and no call ran twice. A separate check on the rig showed the saved calls survive the trip through the log byte for byte. It hasn’t yet had to rescue a real run.
Every way a window can end downloads the results first: finished, stalled, out of time or at its spending cap.
- The cap starts the exit early enough that the download fits inside it.
- The window that lost 14.03 units had stopped at its cap without downloading.
Caps count each window’s own spend.
- Two separate bugs. An early cap read the account balance, so one window was cut off by its neighbour’s spend. Later, two windows that opened at the same moment each measured both sessions’ combined rate as their own, and stopped at about half their real budget.
- Each cap now counts its own rate times its own runtime, which fixed the first. The rate is read under a lock, so two windows can’t open at the same moment, which fixed the second.
Set up each model just before it’s needed, not all at the start.
- The reclaimed window’s log showed 49 minutes setting up three of its four models; its last 28 minutes were never seen.
- Now a writer is set up right before its own drafts and the critic right before the first critique. A loss costs at most one setup.
Test each new kind of window on the cheapest machine first.
- Before trusting the new file-based watcher, we ran a fake job on a CPU-only session that went silent on the stream and then hung.
- The watcher saw it finish from the file, downloaded the results in 2 seconds, and closed the session.
- The two tests together cost 0.03 units.

The rule underneath all six: assume the machine can vanish at any minute, and make sure that minute costs only the work in flight.

When the budget was raised to 150 units, we wanted an outside view on how to spend the remaining 79. The catch with asking for advice is that you tend to anchor on it, or the adviser anchors on you. So:

We wrote and committed our own allocation first, then sealed it.
Argon, the studio’s outside reviewer, got only the measured facts: setup times, speeds, tokens per row and the failures so far. It didn’t see our plan.
We compared the two answers against the sealed version.

Argon’s plan was better in three places, and we took them:

pair Lightning and Muse in one window, each critiquing the other, which saves two model setups;
set up models lazily (point 5 above);
fold one rewrite into the screen itself, so writers that pass don’t need a second window.

We kept one of ours, and corrected one of Argon’s assumptions:

The second-critic check. A second critic re-scores a blind sample of each writer’s drafts, to see whether Muse favours writers that sound like it. It stays as a gate before any writer goes further.
The correction: Argon assumed Qwen3.8’s lost window could be resumed. It couldn’t: that window ran before the call journal existed.

What’s next

The screens are running tonight in this order:

Granite;
Qwen3.6 and Bolt;
Lightning and Muse;
Qwen3.8 and Gemma.

DeepSeek-R1-Distill waits for another day. Its 6-bit build hasn’t been made yet (the one setup step we haven’t timed), and it sits in its own 6-bit group.

For each writer, the readout will give:

how many of its 8 rows were accepted, dropped or cut off;
its twin results;
its tokens and units per accepted row;
its share of thinking;
“cut” or “goes on”.

Those numbers will be added below as they come in.

Where each figure comes from

Every number above was checked against its record before publishing. Records in this repo are linked. The window logs and the units ledger live in the studio’s private run records (the Publisher seat’s receipts/colab/), and the figures from them are listed here so they can be traced.

The critic test (38, 37 and 38 of 40; 10 of 10 caught; 1 item differing between Muse’s runs; the two faulty keys): window 7b’s log and result, and the critic-control README.
Qwen3.8’s calls (13 calls, 81,372 of 85,752 tokens thinking, 15,264 longest, 45.7 minutes): the logging proxy’s records from window 10d, committed as counts only in qwen3.8-w0-calls.json.
About 12,100 tokens and 6½ minutes per row, about 7 rows, and the 147–175 unit projection: the Publisher’s projection from the same window, made at its halfway point.
Fits (Qwen3.6 peak 36,022 MiB and 89.7 tokens a second; Gemma 4,943 MiB free and 27.8 tokens a second; R1 doesn’t fit at 8 bits): windows 11b, 11d and 5.
Lightning’s four matching rebuilds: windows 2, 3, 7 and 7b, against window 1’s original.
Units (70.66 spent at 19:44 UTC; 14.03, 12.81, about 3.2, 3.00 and 5.29 lost): the units ledger, windows 10d, 12a, 11c, 7, and 10, 10b and 10c. The balance is the measure. Per-window figures for windows that overlapped are estimates from their own runtimes.
The 77 and 49 minutes, and the long window running on unseen (stream lost at 15:46, last read at 16:38, cap at 17:57): windows 12a’s and 10d’s logs and Colab’s session history.
The CPU tests (0.03 units, a 2-second fetch): windows 12p and 12q.
The resume test: role-set’s tests/test_callcache.py, with a mock model.

Issue 10: What the terms let you do with AI output

mcp-tool-shop — Fri, 16 Oct 2026 23:59:59 GMT

This is a reading of published terms, not legal advice. Every clause below was re-read at its live page on 2026-10-16. The full record is in rnd, under experiments/generated-content-terms/ and experiments/stage1-regeneration/.

How it started

The studio has Google AI Ultra, which comes with Flow, Flow Music and Project Genie. Before anything they made could ship, the maintainer asked for a licence check. Reading Google’s terms raised a bigger question, because the studio trains its own small models. Could AI-written text from any provider be used as training data?

The maintainer set the rule for answering it. A use is allowed unless the provider’s contract or licence restricts it, and the search for restrictions is documented: the page, the exact clause, and its date. Uses that providers commonly ban (training on outputs, distillation, competing models) get that explicit check before anything relies on them. A model’s reading of the terms proves nothing, including the studio’s own outside reviewer’s.

The first version of the rule went further: it treated anything not expressly permitted as off limits. That made silence a ban, which it isn’t. Only a contract you agreed to, or a licence, can restrict a use. The rule was reworded the same day, and the verdicts below follow the reworded version.

A second correction followed the next day. The studio had read Google’s “related AI technology” at its widest, so it covered even small pieces of pipeline code. The outside reviewer pointed out that the clause sits under “Don’t abuse our services”, and that its purpose is stopping people distilling Gemini into a rival model. So the clause bars training data and distillation, and code the studio owns is ordinary software whoever wrote it. A two-tier version that made anything touching a model “strict” was tried the same day and withdrawn: it kept producing bars no clause contained. The rule is now simply that a use is barred only by a clause you can name.

The rule earned its keep. The outside model’s first pass got two things wrong:

it said Claude’s consumer terms bar all commercial use, when that clause covers evaluation access only;
it said the Llama licence bars training non-Llama models, when Llama 3.3 permits it if the new model’s name begins with “Llama”.

Both were caught by re-reading the source.

The four uses

Use	Google (consumer)	xAI (Grok)	Anthropic (Claude)	Moonshot (Kimi app)
Commercial assets	Allowed. Flow Music grants “Commercial use rights” outright; for the rest, Google disclaims ownership and restricts nothing that applies	Allowed, with attribution (“Written with Grok”)	Allowed: you own the outputs	Barred: non-commercial use only
Training data	Barred	Barred: no distilling outputs	Allowed for non-competing specialized classifiers and tools, never as training targets	Barred for a business: non-commercial use only
Evaluation only	Barred where it serves developing a model; allowed otherwise	Unclear: the AUP’s “develop … machine learning models” clause	Allowed on the same reading	Barred for a business
Code	Allowed when no model trains on it and it builds no rival (the clause is read by its anti-distillation purpose)	Allowed for tooling that develops no model and doesn’t compete, with attribution	Allowed for non-competing specialized tools	Barred for commercial use

The clauses behind the table:

Google (Terms of Service, effective 2026-07-30) bars “using AI-generated content from our services to develop machine learning models or related AI technology”. It has no competition qualifier, and Antigravity’s terms incorporate it. Google’s Gemini API and Cloud terms are narrower: they bar only models or products that compete.
xAI (consumer terms, last updated 2026-10-12): “you retain your ownership rights to the User Content”, and using Output requires attribution under the Brand Guidelines. The Acceptable Use Policy (2026-08-14) separately bars “distilling model data or Outputs”.
Anthropic’s Help Center (article 12326764, 2026-03-16): “You can use Claude’s Outputs to train models that don’t compete with Anthropic’s own models.” It bars general-purpose chatbots, models designed for open-ended text generation, and “Using Outputs as training targets for models”. Anthropic’s support channel, asked about this exact use, said it fits. That reply came from an AI support agent, so the article stays the proof.
Moonshot’s Kimi app: “you may only use Kimi in a non-commercial, personal research, and study manner.”

Open weights: read what the licence says about outputs

Running an open-weight model yourself removes the service’s terms, but not the model’s licence. Most licences grant rights in the weights and say nothing about what the model writes.

Licence or card	What it says about outputs	Verdict
OpenMDW-1.1 (Nemotron 3, 3.5 Lightning, 3.5 Super-VL)	“does not impose any restrictions or obligations with respect to any use … of any outputs”	Proven
Muse-Glimmer-30B card	Its intended uses include “Synthetic data generation” for “downstream model development”	Proven
DeepSeek-R1 (MIT)	“including, but not limited to, distillation for training other LLMs”	Proven
Muse Model License (Muse-Spark-816B)	Synthetic data is an intended use, but a model trained on its outputs is a “Derivative” that must carry the licence and display “Built with Muse Spark Open”	Proven, with conditions
Llama 3.3	Permits training on outputs if the new model’s name begins with “Llama”	Proven, with a naming condition
NVIDIA Open Model License	“does not claim ownership” of outputs, and no restriction on them	Allowed
Plain Apache-2.0 or MIT (Qwen, Gemma 4, Granite, Kimi K3)	Nothing on outputs	Allowed: the licence makes no claim and no contract binds a local run. Check any model-specific use policy too (Gemma has one)

On the last two rows, the Linux Foundation’s OpenMDW launch post notes that Apache-2.0 and MIT were written for code and are silent on generated outputs. OpenMDW-1.1 says outright that outputs are unrestricted. Silence and an explicit grant reach the same place: nothing restricts the use.

The route matters too. The studio’s Kimi seat turned out to run in the Zed editor through Ollama Cloud, not Moonshot’s app. Ollama’s terms (May 2026) say “You retain ownership of content you create using our services”, and restrict only building products that compete with Ollama. The Kimi K3 and K2.6 licences are MIT-style. So the app’s non-commercial clause never applied to that code.

What the studio found in its own work

An inventory traced every piece of text in the Stage 1 pipeline to whoever wrote it. Three findings stood out:

The training rows were written by Kimi (the pilot), Grok (batches 2–8) and Antigravity (the 210 apply items). Grok’s and Google’s terms bar training on them.
The generator showed the writer finished Kimi and Grok rows as examples, so even a clean model would have been prompted with barred text.
The “real-source” excerpts weren’t human-written. They came from the studio’s own repos, and those repos were mostly written with Claude.

What changed

All 810 training rows and 210 apply items are being regenerated by licence-proven open models: Muse-Glimmer and Nemotron on Colab, and a larger Nemotron on a rented GPU if a pilot earns it. The writer and critic always come from different families.
The method carries over: the lessons, the gates, the defect catalogue and the evaluation sets. Claude wrote those, and Anthropic’s terms allow that use for a specialized verifier. No Claude text becomes a training target.
The code stays, whoever wrote it. Under the two-tier rule, pipeline code that no model trains on is ordinary software. Three small Antigravity-written pieces had already been re-implemented under the earlier, stricter reading; they stay as they are. Grok’s generator logic stays too, with its text replaced. The tooling Grok and Kimi wrote is credited in CONTRIBUTORS.md files that say “Written with Grok”.
Provenance is now recorded at the source. Every text-bearing template and pack in the generator gets a record of who wrote it, and the generator’s own check refuses one without it. Seat commits carry a trailer naming the seat.

If you’re doing the same

Ask per use. Shipping, training, evaluating and code are separate questions.
Read the use limits, not just ownership. Acceptable-use policies are where the training bans live.
Don’t mistake silence for a ban, or a ban for silence. Only a contract you agreed to, or a licence, can restrict a use. Find the clauses that do (training, distillation and competing-model bans), and record where you looked.
Check which account you used. Consumer, API and enterprise terms differ, and so do apps and hosts serving the same model.
Re-read every clause yourself. A model’s summary of the terms, ours included, got things wrong.
Record who wrote what when it’s written. An inventory after the fact needed two agents reading PR threads and commit trailers, and still left gaps.
Mind the attribution clauses: “Written with Grok”, “Built with Muse Spark Open”, and Llama’s naming rule.
Name the clause, or it’s allowed. Never read an undefined phrase at its widest just to be safe, and never add a tier or a filter the terms don’t contain: that ratchet ends up banning a temperature monitor.

Issue 9: The backend that changed under us

mcp-tool-shop — Thu, 15 Oct 2026 23:59:59 GMT

Every claim below links to the record it came from. The build records are in rnd, under experiments/ollama-cuda-backend/.

What happened

The studio runs its own CUDA backend for Ollama on the RTX 5090. It’s a ggml-cuda.dll built from Ollama’s own llama.cpp pin with CUDA 13.4, for compute capability 12.0 only, and dropped into Ollama’s lib\ollama\cuda_v13 folder. The script that starts the studio’s Ollama checks that file’s sha256 before it calls the server ready.

On the morning of 2026-10-15 that check failed. The 0.40.4 install of the night before had written the stock CUDA libraries over the studio’s: same file name, different hash. Ollama ran normally, so only the check noticed. Every GPU grant was held until the studio build was back. → the 0.35.1 build record

Why it had to be rebuilt

Ollama doesn’t vendor ggml. It fetches llama.cpp at a pinned tag and builds the backend against it. 0.35.1 pinned b11232, and 0.40.4 pins b11521. The old DLL was built against the old ggml ABI, so copying it back next to the new ggml-base.dll would have been a guess. The rebuild used the same recipe as the first build (Ollama’s llama_cuda_v13_windows preset, architecture 120 only, CUDA 13.4’s nvcc, target ggml-cuda), from the new tag.

It built in 343.8 s (156 steps), and every dependency of the new DLL resolved. → the 0.40.4 build record

The CPU incident, and the rules it set

Earlier the same day, a different job taught the lesson this build was run under. An index rebuild through offrig’s new embedding server ran torch at its default thread count: all 24 logical cores. The CPU went to 100%, and it was stopped by hand. Nothing broke, but the job took a shared machine from everyone on it, without warning, from a team that already knew torch takes every core.

Since then, every heavy job is pinned before it starts:

Efficiency cores only, at below-normal priority. On this CPU, GetSystemCpuSetInformation puts the 8 performance cores at logical 0, 1, 10–13, 22 and 23, and the 16 efficiency cores at 2–9 and 14–21. That isn’t the “first eight” a guess would give. Jobs run under an affinity mask over efficiency cores only, set at launch (start /AFFINITY … /BELOWNORMAL), so every child process inherits it.
The load is stated before the start, with a stop rule (total CPU held above a threshold for so many seconds) and CPU logged every 5 s.
The process that’s watched is the one doing the work. The first corrected rerun watched a Python launcher shim, which was idle at 6 MiB, and its gates “passed” on the wrong process. They were declared void, and the rerun was repeated against the real interpreter.
Every thread pool is capped, not just torch’s. An outside review of the embedding server found that capping torch’s threads isn’t enough: OpenMP, MKL and the tokenizer’s Rayon pool each size themselves to the machine unless capped before torch loads. → the review and its disposition

The CUDA build ran under these rules: 8 efficiency cores (0x3FC), below-normal priority, -j 8. A per-core reading mid-build showed those 8 cores at about 97% and every other core at about 22%. Total CPU averaged 41.7%.

Three rounds of review before the GPU

The plan to test the build was reviewed by Argon, the studio’s outside reviewer, before any GPU use. It came back twice with changes, and they’re why the test looks the way it does:

Isolation. The sandbox Ollama runs from its own tree, with the stock install stripped from PATH and the CPU-only switches from the embedding work removed from its environment. A stray CUDA_VISIBLE_DEVICES=-1 would have quietly tested the CPU.
One model at a time, checked before every prompt, not once per block, with a warm-up generation after each load.
A breach kills, not just records. If card memory passes the cap, the sandbox’s process tree is killed and the stock server unloaded at once.
A pre-registered tolerance.
- The outputs must match token for token.
- The one exception is a near-tie: at the first differing token, the stock run’s top two candidates are within 0.001 in log-probability, and the sandbox chose the second.
- The tolerance needs the top two candidates, so the source was read to check where they come from. Ollama 0.40.4 asks llama-server for n_probs and never sets post_sampling_probs. llama.cpp then reads the candidates from the full pre-sampling logits, so greedy top_k: 1 can’t hide the runner-up.
- → the v3 test and its diff from v2

Argon also set the build to the studio’s standard, not the minimum:

explicit sm_120a SASS with no PTX;
every FlashAttention quant compiled;
fatbins compressed for speed;
CUDA graphs and VMM on;
cuBLAS from CUDA 13.4 Update 2, whose release notes list a fix for an NVFP4 heuristic on compute capability 12.x.

A note on version numbers

CUDA 13.4’s first release is 13.4.1 (nvcc V13.4.59). “13.4 Update 2” is version.json 13.4.3, with nvcc V13.4.92 and cuBLAS 6,14,11,1380. The review asked for “13.4.2”, which is off by one. The build script now refuses any nvcc but the one installed and verified. → the 13.4.3 build script

The build to the studio standard

It built in 374.0 s: 201 steps and 189 CUDA objects, each compiled as arch=compute_120a,code=[sm_120a], which is machine code for this card only, with no PTX. The DLL is 217 MB, against 56 MB for the first rebuild; the extra FlashAttention kernels and the speed-first compression account for it. It ran under the same cap: 8 efficiency cores, total CPU averaging 37.5%, and one 5-second sample above 50%. Every dependency resolved. → the 13.4.3 build record

Update, the same afternoon: the sandbox test failed

Argon gave round 3 a green light, and the test ran under a GPU grant: 46 seconds, peaking at 7.4 GB of card memory against a 9.2 GB cap, and cleaned up completely.

The new DLL loaded, and the log names its path. It’s at least as fast as stock.
It failed the output check on 2 of 3 prompts. Stock matched itself token for token. The sandbox matched stock exactly on the coding prompt, but diverged at token 15 of one prompt and token 6 of another. At both points it ranked stock’s top two candidates the other way round, by margins of about 0.7 and 0.8 in log-probability. That’s not the near-tie the rule allows.

So by the rule set before the run, nothing is activated and the stock backend stays. The difference is larger than reordered sums usually give, and the cause isn’t identified: build options or toolkit. Neither answer is wrong, but the bar was identity with stock, and identity isn’t quality. Finding the cause is its own decision, with its own review. → the result

Update, later the same day: the cause is cuBLAS

A diagnosis harness, frozen after outside review, ran eight arms of qwen3:8b on 15 prompts, one arm at a time under one GPU grant:

Stock repeats itself bit for bit across server starts.
Stock with only CUDA 13.4.3’s cuBLAS matches the studio build bit for bit: every token and every top-10 log-probability. Three different ggml-cuda.dll builds with the same cuBLAS change nothing.
Where cuBLAS runs. The model keeps one weight in f16, the attention V projection. While the prompt is read, that matmul goes to cuBLAS with f16 accumulation, and everything else runs in ggml’s own kernels. That’s why the builds disagree from the first token.
Neither build is closer to a q8_0 reference, and the studio build had no task failures.

By the rule set before the run, the build is eligible. Activation is a separate decision, with stock CUDA 13.0 kept as a fallback.

The harness’s first run crashed at its first arm, before anything loaded, on a bug every review had missed. The fix came with an offline dry run of the whole harness, which now runs in CI. → the result and the brief

So it can’t happen silently again

A weekly check now runs every Monday. It reads the rig’s live versions:

Ollama’s version;
the installed backend’s hash against the studio’s builds;
the start script’s expected hash;
nvcc and the driver;
the newest Ollama and llama.cpp releases.

It writes them into a paste-ready brief that asks Argon for the newest recommended setup: the toolkit, the Ollama release and its pin, the build options, whether a rebuild is due, and any security advisories. If an update has replaced the studio’s backend again, the brief opens with an alert.

Issue 8: Room for the seventh pack

mcp-tool-shop — Wed, 14 Oct 2026 23:59:59 GMT

Every claim below links to the record it came from.

The problem

role-set builds logic puzzles to train and test small models. Its T10 family turns on one size word: “at least two of the lamps are on”, “exactly one clinic opens in the morning”. Every puzzle is proven before it’s kept: a solver checks the model, and a set of registered readings checks that a solver taking a shortcut (reading the size word loosely) would be caught.

Under those proofs the generator had collapsed onto a few shapes. A test set that repeats one shape teaches the shape, not the reading. So the work was to make the shapes vary, measurably, without loosening a single proof.

The floors, set before any fix

The first brief fixed the measure and the bar before anyone touched the code. A puzzle’s canonical skeleton is its structure with the wording removed: how many entities, which clue forms, where the trap sits. In each stratum (two-value or three-value packs; “set” or “claim” questions): at least 75% of skeletons distinct, no skeleton more than 3 times, none twice in one pack’s file of 20 seeds. Tuning on seeds 41–60; the verdict on locked seeds 61–80 that nobody tunes against. → Brief 1

The proof that narrowed the claim strata

Brief 2 asked what a claim puzzle can test at all, and the answer was a proof, not a measurement. A claim states something and asks whether it holds. If the claim says “at least two” and exactly two is the truth, the claim is true, and a reader who read “at least” as “exactly” agrees. Every world where “exactly k” holds is one where “at least k” holds. So no claim built on “at least” or “at most” can separate the careful reader from the careless one; only “exactly” can. The generator now refuses to build what can’t be tested (trait_not_tested), and claims carry “exactly” alone. That is correct, and it left the claim strata with very few structural cells. → Brief 2

The allocator, and the first stop

Brief 3 made the packs of one kind share their skeleton space, with an allocator (a min-cost flow) that spreads a pack’s twenty seeds across cells (entity count, size word, count, number of conditional clues), with a cap on the barest cells. It stopped on arithmetic: six three-value packs needed 26 visits to the nine cells with no conditional clue, against a supply of 18 at a cap of 2. The stopgap ruled was a cap of 3 for that one stratum, which passed exactly on the floor: 108 of 120 distinct, top repeat 3. → Brief 3, amendments 1 and 2

The floors in CI, from a committed record

Proving 640 puzzles under coverage took 46 minutes against a 15-minute job. The ruling, after a cross-family consult: the measurement is committed. A record holds every build’s cell, try count and skeleton, keyed by a hash over every file that decides a build, closed under the package’s imports. The normal job recomputes the hash, fails a mismatch by naming the regenerate command, and computes the floors from the stored skeletons in seconds, inside a CI job that finishes in about seven minutes. The exhaustive rebuild runs on demand and must reproduce the record exactly. A cap on build attempts was refused twice: it changes which puzzles get built. → role-set #49

The new axis, and the second stop

To get the claim strata off the floor, the conditional clues gained a shape coordinate: for one if, the consequent’s polarity; for two, chain, shared consequent, shared antecedent or disjoint. (A finer coordinate, whether the if touches the claimed entity, was dropped because the skeleton can’t see it, so it would have inflated cell counts without adding measured variety.) The zero-if cap went back to 2. On fresh locked seeds 81–100 the claim strata cleared every floor.

Then set three came in at 118 of 119: one build needed 435 tries against a limit of 400, and the baseline failed the same build. The cause was a pooled weight: the capacity script had measured one weight per set cell across two- and three-value packs, so a cell that builds easily for two-value packs looked rich and sent a three-value pack there in every file. Brief 3 had always said capacity is per stratum. The ruling was to split the weights, re-freeze, and take the verdict on fresh locked seeds 101–120. The alternative, a stricter probe, drops cells, which is the same forbidden move as a try cap. → Brief 3, amendment 4

The verdict

On locked seeds 101–120, with the baseline code run on the same seeds beside it:

Stratum	Before	After
set, two values	40/40, top 1	40/40, top 1
set, three values	118/120, top 2	118/120, top 2
claim, two values	40/40, top 1	40/40, top 1
claim, three values	112/120, top 4 (fail), 26 zero-`if` visits	118/120, top 2, 0 zero-`if` visits

All 1,280 builds across four seed files build inside the try limit; the one that caused the stop now takes 8 tries. Three-value claims have 61 usable cells against 27. An allocation test places twelve three-value packs at a cap of 2 with no zero-if visit. R&D reproduced the baseline and the record figures independently before passing it. → role-set #55

What we’d do the same

Set the floors and the locked seeds before the first fix, and take every verdict on seeds nobody tuned against. Fresh locked seeds after every change to the allocation.
Freeze, then measure. The verdict run starts only from a named commit, and the record names it.
The baseline beside the verdict, on the same seeds, every time.
Refuse the easy fixes that change what gets built. A try cap and a dropped cell both passed the floors by changing the population; both were refused, twice.
Keep the hash closed. The record’s hash list missed two files at first; an import-closure test written by the builder caught them, and it’s the better instrument.

Released

All of it shipped as role-set v0.2.0 on 2026-10-14, after a full treatment: a README rewritten for the generator, two new handbook pages (the T10 generator end to end, and the role packs), the README in seven more languages, and a ship-gate scorecard of 37 of 50, up from 31, with the audit passing every gate. It stays 0.x: role-set is a prototype, and nothing it writes is training data until a person checks it.

→ The T10 brief, updated with these figures.

Issue 7: The tracer that kept every promise but one

mcp-tool-shop — Wed, 14 Oct 2026 23:59:59 GMT

Every claim below links to the record it came from.

The question, and the rule set before the run

interlock measures local models: it drives them through Ollama on the RTX 5090 and writes a receipt for every measurement. NeMo Relay is NVIDIA’s open tracing library for agents. The question was narrow: can Relay trace interlock’s model calls without changing what interlock measures?

The design was pre-registered and reviewed by a model from a different family at each step, with the analysis code frozen before any counted run. → The design, v3.1 to v3.2

Three hypotheses, and a decision rule written as a tree:

H1: the trace accounts for every request the harness sends. Fail this, and Relay is rejected.
H2: tracing doesn’t move the measurements, within a band taken from the run’s own noise, and it costs the harness under 0.5 ms per two-call measurement on the client side. Fail this, and Relay is kept for debugging runs only.
H3: nothing leaks: no prompt text, no paths, no machine or user names.

Stage 0, on fakes, held. Stage 1 ran on the 5090 on 2026-10-14, under a grant with a time cap, a VRAM watch and a preflight that checked the card, the CPU and the environment before every step. The first attempt refused itself after eight seconds, because it had been launched from the wrong Python environment; the driver now checks that too. → The stage 1 plan

What the frozen analysis said

stage1_holds = false.

H1 failed in every step, by exactly one request each time: 204 sent against 203 counted, 217 against 216, and so on. The companion step also had three spans that belonged to no measurement.
H2 held on every model number. Decode speed, prompt speed and latency were equivalent with tracing on. The client-time budget failed: medians of 0.548 ms and 0.682 ms against 0.5.
H3 held everywhere.

Under the tree, H1 failing means reject. The analysis is byte-identical to its freeze, and nothing was re-scored. → The result note

The diagnosis: two of ours, one real

Read-only, from the run’s own records.

The off-by-one was the harness’s empty-card check. Before a 5090 run, interlock asks the server what’s loaded. That request went through the adapter, so the driver’s counter saw it, but it went out before the tracer was attached. With that one request set aside, every count matched exactly. No traced call was missed.
The stray spans were the companion’s stop. The harness closed the measurement, then unloaded the companion model, then switched tracing off. The unload was traced but belonged to nothing.
The client time was real. A traced call cost about 270 µs more than an untraced one. A CPU-only bench against a stub transport, with the same 250 ms rest before each measurement, reproduced the live figure within about 10%, with no GPU at all. Most of the cost is Relay’s own work per call; the rest was our own serialisation of long prompts to measure their size. Removing ours would not have passed the gate.

The outside reviewer checked all three and agreed, and added a test we hadn’t thought of: the four large deviations in the import check split three faster to one slower, which is noise, not a slowdown from importing the library.

The decision, and why it departed from the rule

The rule said reject. The Director chose debug-only tracing instead, recorded as a dated amendment (v3.2) and as his call: H1’s failure was a counting-window bug with every request accounted for, so H2’s branch is the one that fits. The stage 1 verdict stands as frozen. A rerun was ruled out, because the cold bench shows the client-time gate would fail again. No further GPU or NPU time goes to Relay.

The build landed the same day. The reviewer wrote it from the result note; we applied it with six recorded departures (one of them a real defect in the draft that would have silenced ordinary traces); the reviewer then checked the branch itself and approved. The card check now runs inside the tracing window, the companion stops inside its own measurement, request sizes come from the bytes actually sent, and every row of a traced run is marked and kept out of the routing table, which the trial’s rows had reached. A deterministic test drives the real adapter and real HTTP client against a fake server and checks that every request is counted, every span is matched and closed, and nothing leaks. → The build round’s review

One more defect fell out of the review: the request-size field in Relay’s records had never held the request size. The sanitizer was re-measuring its own {"bytes": n} record, so every value was 13 or 15. It’s fixed, and no verdict depended on it.

What we’d do the same, and what we learned

Freeze the analysis before the run, and keep the verdict even when the cause is yours. A rerun is a new pre-registration, and a departure is a dated amendment with a name on it. That is what made the diagnosis trustworthy.
Count below the hook. The off-by-one was only visible because the driver counted requests independently of the tracer. A tracer checking itself would have passed.
Bench the cold case. The warm bench put Relay at 30 µs per call; the live rig said 270. Putting the stage’s own rest before each measurement on a CPU-only bench explained the gap without touching the GPU.
Settle a ruling with the outside reviewer’s input before sending it. Two reversals in one evening on another repository cost a paste and a re-read each. That’s now a working rule.

→ The NeMo Relay brief, now measured.

Issue 6: A red check that says how to fix itself

mcp-tool-shop — Wed, 14 Oct 2026 23:59:59 GMT

Every claim below links to the record it came from.

Atlas and testing-os, and how the library connects to them

What they are.

testing-os is the studio’s testing workbench: protocols, evidence stores and learning loops for AI-assisted software.
Atlas is its mapping engine. atlas map records how one repository is built: which parts it has, what each part does, which files belong to which part, and the “doors” where it meets the outside, such as workflows, entry points and the packages it publishes. The map is committed in the repository it describes.
atlas check runs in CI. It fails when the code and the map disagree, so a new folder or a moved file can’t pass unnoticed.
Every repository in the fleet that runs workflows keeps a map, made by one pinned engine version. That’s what lets a question be asked across the whole organisation.

How housekeeping uses it. The two tools split the work:

Atlas works offline, one repository at a time, and records how that repository is built.
Housekeeping works online, across the organisation. It records what GitHub says now: CI, pull requests, releases, Actions cost.
Housekeeping collects each repository’s committed map by its blob id, checks the map’s health against the organisation’s rule (no map, no check in CI, an engine behind the fleet), and ties a red run’s failed step to the door it came through. Its findings stay private.

How this library copies that pattern.

Each tool repository already commits its own pre-registered experiment specs and run records, the way it commits its map. interlock’s receipts are the main example.
The library should collect them the way housekeeping collects maps, rather than by hand. That’s the build-up program R&D set out on 2026-10-13. → The build-up program

A red check that says how to fix itself

The problem. The fleet’s commonest atlas check failure was a folder that no part of the map covers, such as interlock’s receipts/.

Each new file there failed the check, and regenerating the map pardoned only that one file.
In CI the failure showed only as “exit code 1”.

What Atlas slice AM changed. It’s merged in testing-os, and ships in the next Atlas release with the multi-repository slices.

The check prints the lines to fix it, ready to paste into boundaries.yaml. A folder no part reaches gets a part of its own, and a folder a part already reaches gets a glob. Every test applies the printed lines and requires the check to pass afterwards.
atlas map warns about an uncovered folder before CI fails on the next file.
In GitHub Actions, the check annotates. A failure is an error on the file, and a door notice is a warning on the line it was read from. The finding and the fix go into the job summary. Outside Actions, the output is unchanged byte for byte.

→ testing-os #159

What the live runs showed. The spec required one real failing Actions run before release, and recorded what GitHub actually displays. The first run found two defects that the tests had passed:

Paths: a job that checked out into a subdirectory annotated repo/.github/..., and GitHub never placed that on the pull request’s diff. Paths are now relative to the repository’s top level.
Line numbers: file-level failures carried line 0, and GitHub shows those only as a dead link, never on the diff. They now point at line 1.

A second live run showed every annotation landing on the diff, the subdirectory job included. The same run showed each message repeated four times, once per test leg and once per trigger. The organisation’s Atlas rule now says to run the check on one leg only. → the live run

The lesson: a CI message is only fixed when someone has looked at where GitHub puts it. Tests of the emitted text couldn’t see either defect.

Next for Atlas:

Slice AN is merged: one atlas mcp server can now answer for several repositories, fixed when it starts. → testing-os #169
Slices AO to AQ follow, before one pinned release for the whole set. AQ, the inferred-consumer check, turns on only after it clears a pre-set recall and precision bar on a labelled seed. → the rulings

The library collects

python -m rnd collect reads an experiment’s records from its tool repository at a pinned commit.

The pin: each source is named with its path, its commit and the SHA-256 of its bytes. The bytes are read with git show, so the clone’s own state doesn’t matter, and a source whose hash no longer matches is refused rather than re-trimmed.
The trim: each experiment’s own module decides what its published record keeps. Every output then passes a public guard, which refuses a machine path, a home directory or an environment dump.
The drift check: --check rebuilds every output and compares it byte for byte with the committed copy, so any drift between a source and its published copy is found.

E5’s card, in issue 5, was the first experiment collected this way. → rnd #4 · E5’s collect.json

Still to come in the program:

one run-record format in testing-os, for interlock, the library and later experiments;
defect classes shared across the fleet, so the library can count them over time;
an instrument-health check;
the same failure-output style for the library’s own check.

These will be reported as they land.

Also this week

interlock 0.1.0 is released, the first on PyPI as mcptoolshop-interlock. It’s a prototype advisory router: it suggests a model, an engine and a device from measured receipts, and never serves traffic. Its release assets carry SLSA provenance, and CI now fuzzes its receipt and spec readers.

A fuzz-target bug caught in review: the first version of the fuzz target excused the very crashes it should report. A receipt that isn’t UTF-8 crashed interlock check with a traceback.
Now: both readers refuse cleanly, with tests. → interlock #121

What a T10 claim can test. role-set builds small logic puzzles that teach a model to read size words literally (“exactly”, “at least”, “at most” two of them).

The defect: a variety fix passed its floors, but review found that some “at least” puzzles were passing their shortcut check on a second “exactly” clue. They tested “exactly” again.
The proof: every assignment that fits “exactly k” also fits “at least k”. So if every assignment that fits “at least k” gives the same verdict about one entity, every assignment that fits “exactly k” does too, and misreading the word can’t change a claim’s verdict.
The fix: claim puzzles now test “exactly”, set puzzles test all three words, and each puzzle must change its answer on its own clue’s misreading. On fresh locked seeds, the defect went from 8 items to 0. → role-set #53 · role-set #54 · the briefs

Issue 5: The card is full

mcp-tool-shop — Tue, 13 Oct 2026 23:59:59 GMT

Every claim below links to the record it came from. Interim results are labelled as interim.

E5: how much context each model holds

E5 asks a router’s most practical question about a local model: how long a prompt can it take before it no longer fits on the graphics card? Past that point the model spills into system memory and slows sharply. For each model, interlock’s harness raised the context window step by step and filled it with a real prompt. It recorded the largest window that stayed on the card, under a 30,500 MiB ceiling, with no spill.

Model	Largest window on the card	Trained window
llama3.1:8b	131,072	131,072
gemma4:31b	86,016	262,144
granite4.1:30b	43,008	131,072
qwen3:14b	40,960	40,960
qwen3:8b	40,960	40,960
mistral-small:24b	32,768	32,768

Measured with Ollama 0.35.1, an f16 KV cache with flash attention, and Q4_K_M weights on an idle card.

How the edges were found.

gemma4:31b is bounded by a measurement. The next step, 88,064 tokens, peaked 44 MiB over the ceiling. Neighbouring readings vary by a couple of hundred MiB, so the true edge lies somewhere between the two.
granite4.1:30b is bounded by a forecast and a measurement. The next step was forecast over the ceiling from the measured growth per token, and the step after that was measured over.

Why the first run fell short. The first full run, earlier the same day, stopped gemma at 14,336 and granite at 30,720. Both stops came from a safety predictor that assumed every layer’s memory grows with the window. gemma uses sliding-window attention and granite is a hybrid model, so their memory grows far more slowly. After a dated amendment, the predictor learned from measured loads instead. The same run also fixed a spill rule that had misfired on llama3.1:8b, which in fact holds its full window with gigabytes to spare.

Three small defects in the receipt. None changes a result, and they’re queued as one fix:

a label names the wrong reason for gemma’s stop;
one reading probably caught the previous load still freeing memory;
a decode rate reads a million tokens a second, because each check generates a single token.

→ E5: the card, the rulings and the defects · the trimmed records

Two NVIDIA tools, read against our own

NeMo Relay records what an AI agent actually did: each model call, tool call and decision, in open trace formats. It can wrap Claude Code without changes to it. We’ve added it to the toolset as a planned instrument: if a trial confirms the transparent mode, it could trace our agent experiments. → How do we see what an agent actually did?
PAIR joins the computers on a home network into one model service and sends each request to a machine that holds the model. Its own documentation lists what its scheduler can’t see: whether a model is already loaded, how much context fits, and how jobs differ in size. interlock measures exactly those things on one machine, and E5 is one of them. → Where should this request run? NVIDIA PAIR and interlock

New on the bench

rnd probe <url> answers the first questions about a tool or model from each source’s own records: its licence, latest release, gating, and any code mentioning telemetry. A source it can’t reach is reported as unknown, never guessed.
rnd search --exact finds whole words in the same case, so a search for PAIR finds the product, not every pair.
rnd watch checks a list of sources for anything new since the last look. The list covers NVIDIA’s blog and models, the CUDA toolkit, and the releases of Ollama, OpenVINO, PyTorch, ComfyUI, llama.cpp and Claude Code. Once a week a scout reads what’s new and writes a short ranked report: what to try, what to file and what to skip.

→ rnd’s README: probing a tool, watching sources

New: the weekly watch

The watch now has its own page every Monday: what changed in the tools and models the studio runs on, what it means for our work, and what we did about it. The first edition caught a Claude Code fix that closes a gap in our own experiment’s sandbox.

→ The watch: week of 13 October

Issue 4: Three devices, no contention

mcp-tool-shop — Mon, 12 Oct 2026 23:59:59 GMT

Every claim below links to the record it came from. Interim results are labelled as interim. The device router switchyard is now interlock, renamed before its first release because the old name is taken.

E4b’s result

E4b asks whether each device keeps its speed when another runs beside it. A pair counts as “free” only when the 90% interval of its slowdown sits inside ±10%. That rule was fixed in advance.

Measured device	Beside	Change	90% interval	Verdict
5090 (decode rate)	NPU	−0.8%	−1.2% to −0.5%	free
5090 (decode rate)	iGPU	+0.6%	+0.2% to +1.4%	free
NPU (latency)	5090	−0.6%	−1.3% to +0.1%	free
iGPU (latency)	5090	+2.5%	−1.0% to +4.8%	free

So the router can treat the three devices as independent lanes, and E4’s question is answered.

The NPU and iGPU rows took a second run. In the first full block those two pairs came out inconclusive, and not because of noise. The Intel calls take about 40 ms, and the harness ran them within a tenth of a second of starting the 5090. So they overlapped the 5090’s start, not its steady decode. The clock check refused them. The harness now waits for one steady 5090 warm-up, and the re-run of just those two pairs had no refusals at all.

One detail for routing: the iGPU’s latency beside a 5090 decode is sometimes about 6% higher, which is inside the band but worth knowing for latency-sensitive iGPU work. That chart was drawn after the verdict, and it decides nothing.

→ The E4b data and charts → E4b block 2

How the harness got there

E4b is the redesign of the co-scheduling experiment that came out inconclusive. Its harness merged after two rounds of review.

It runs its own Ollama server. The server inherits the system environment with every Ollama setting stripped, then gets three pinned values.
Before it measures anything, the run refuses unless the server’s new log shows the CUDA 13 backend, the model sits wholly in GPU memory, and the model’s digest matches the spec.
Before the server starts, the VRAM watchdog must be alive and the card must be empty.

A pilot of 20 calls on the 5090 alone set the clock tolerance from data, at 22.5 MHz (three of the card’s clock steps). All 20 ran at full length, at 218–224 tokens/s.

→ The switchyard review record

Live-only defects

The first pilot attempts stopped before any device work, with the card idle.

The listener looked closed. After an overnight update, Windows build 26340 reports Ollama’s listening socket as “Closed”, so every lookup by listening state found nothing. → The rig fact
The command runner threw its output away, so the server pin read nothing. Worse, the git check read “no output” as “clean tree”. It now fails closed.
The pilot counted its prompt before its own server existed.

A read of the pilot path then turned up more of the same, all fixed before the fourth try:

the start didn’t wait for the server to come up;
GPU clock sampling wasn’t connected;
there were no request timeouts.

The fourth try showed that E4’s prompt, one sentence repeated, makes the model loop at long lengths, so E4b now counts “1, 2, 3, …”. The first full block then lost its record to a read-only NPU cache file at cleanup. The harness now writes the record first, refuses run names already in use, and sets aside any run that stops.

Fakes that told the truth, continued

Issue 3 described four failures caused by test fakes. The count is now seven. The pattern is the same each time: a test swaps in a stand-in for a real call, and the real version is never exercised. The remedies are:

recorded real exchanges;
one test that builds the live setup with nothing swapped in and fails if anything is still a stand-in;
reads that refuse a missing value instead of passing it.

→ Test fakes that hide the platform

The puzzle generator and batch 6

Dated spans. The solver now works out which edition was in force on a given day. The writer can’t state it, which closes the shortcut that let a placeholder date pass.
The generator’s first slice builds T10 puzzles, then proves that a reader who misreads “exactly” or “at least” gets a different answer. T10 uses only “which could be…” and true/false questions, because a single-answer question can’t show that misreading.
Batch 6’s last fixes were wording and options only, and it is clear for the Stage 1 freeze.

→ The role-set review record

Issue 3: A second opinion on the first live runs

mcp-tool-shop — Mon, 12 Oct 2026 23:59:59 GMT

Every claim below links to the record it came from. Interim results are labelled as interim.

The first live runs

switchyard is the studio’s device router: it decides which model runs on the 5090, the Intel NPU or the iGPU, from measurements rather than guesses. On 2026-10-11 and 12 its harness made its first live runs.

E4 asked whether the Intel devices can work beside the 5090 without slowing either side.
E5 asked how large a context window each model can hold on the card.

Getting there took a series of fixes, each logged with its evidence:

a token count that Ollama read as “no limit”;
a prompt search that made thousands of calls;
two missing input shapes;
a crash on the NLI model’s output.

→ The live-run records

E4’s result is inconclusive. The run was valid. Nothing measured slower when paired, and the iGPU even ran about 15% faster. But every “alone” baseline varied more than the pre-registered noise rule allows, so no pair can be called. We traced the noise to three things:

the GPU clocks hadn’t settled during short calls;
the NPU woke slowly after a pause;
a control request was silently reloading the model.

E4 isn’t re-run as it stands. → E4: co-scheduling inconclusive

The outside review

The Director put the night’s findings to Kimi K3, a model from outside the families that built the harness, which read both experiment specs. R&D ruled on each of its six points.

E4’s noise rule (accepted, with one change). The old rule pooled every “alone” measurement, so it counted the very drift that alternating alone, paired, alone is meant to cancel. The redesign, E4b:
- scores each alone–paired–alone triplet as its own log-ratio;
- calls a pair free only when the 90% confidence interval of those ratios sits inside ±10% (an equivalence test);
- throws out any single call that reloaded the model or whose GPU clock jumped.
R&D’s change: an interval wholly outside the band counts as “not free” on either side, faster included, as the original spec said, because either way the devices are affecting each other.
The spill signal (accepted in part). A whole-adapter memory counter replaces the per-process counter, whose process lookup broke when the server ran as administrator. R&D rejected the proposed “two of three signals” rule: it would have loosened a rule we fixed in advance, after seeing the data.
A live test tier (accepted, changed). The same recorded cases also run live against a tiny model on a CPU Ollama the harness starts for itself, before each live run and on every Ollama update. It doesn’t run nightly, and it never touches the CPU instance another tool depends on.
Design rules (accepted). The harness owns the server it measures: it starts it, or reads a declared record of how it was started. Probes only read; they never change the state they read. And state is read at the moment of measurement, never pieced together afterwards.
The iGPU speed-up (accepted, with a correction). The suggested first check, CPU frequency from the existing receipts, can’t work: Windows reported the same nominal 3,700 MHz on every call. A real frequency counter and a CPU-only control run come first.
Clock settling (accepted). Use longer decodes, wait until the GPU clock holds steady before timing, and fix the clock reading that came back empty on every call.

→ R&D’s rulings on the review

Fakes that told the truth

Four of the night’s failures had one cause: a test fake accepted something the real server treats differently.

A completion limit of 0 meant “nothing” to the fake and “no limit” to Ollama.
A search’s cost was invisible to fakes, which take no time.
A fake model returned rows the real one never does.

The remedy is to record real exchanges once, under a grant, and allow fakes to replay only those. The first recording already overturned two assumptions:

a very small context size is raised to a minimum, so it limits nothing;
a request we treated as harmless loads the model.

→ Ollama reads a zero limit as no limit

Issue 2: Puzzles with one answer, and a pod that never got ready

mcp-tool-shop — Sun, 11 Oct 2026 23:59:59 GMT

Every claim below links to a brief or to the record it came from. Interim results are labelled as interim.

The verifier’s training data

ASPIRE is teaching Qwen3-8B to be a verifier before any domain training. Generated items needed a reader: of 49 candidates from a 235B writer, 15 passed as written. Hand-built batches did too: of 120, 91 passed. A program now solves every logic puzzle and refuses any that has two answers, a spare clue, or a clue that gives the answer away. Passing it isn’t enough, though. Puzzles bent to fit the solver, and the same puzzle under new names, were found by reading, then by a script. Each defect now gets a class name, so the checks can move from people into tools. → How is the verifier’s training data checked?

A pod that never got ready

On 2026-10-11 a rented pod’s engine logged that startup had finished, then deadlocked. Its health check stayed down while its model list answered, so the old readiness check passed. The run wrote nothing and cost $2.80. offrig 1.1.0 now calls a recipe ready only when one completion returns an answer. It reports a hang after 3 minutes and leaves the pod up for a person to decide. That pod sat on a CUDA 13.2 host where the working runs had used 13.0. That isn’t shown to be the cause, but a profile can now pin the exact host CUDA versions. These paths are tested against fakes and haven’t been re-run on a rented pod. → offrig on GitHub

CUDA, by the version the software needs

A profile can also say which CUDA its software was built for, such as 13.4 for a PyTorch nightly. Its rented host must then be at least the oldest of that major version (13.0). That follows NVIDIA’s minor-version compatibility, and it matches what our CUDA 13.4 runs measured on RunPod’s 13.0 hosts. → Can the studio train on CUDA 13.4?

Also

Phi-3.5-mini’s first three CPU calls ran at about 7 tokens a second, then 21. Neither a cold read from USB or NVMe nor any CPU scheduling setting reproduced it. It’s recorded as open, and the next run samples the machine once a second through every call. → switchyard
A seeding bug in the critic heads. Dropout drew from an unseeded generator, so a head depended on the heads trained before it. The step-4 conclusions stand, because they rest on confidence intervals. The fix seeds the whole of training.

Issue 1: A verifier, a cliff, a dead NPU and a missing input

mcp-tool-shop — Fri, 09 Oct 2026 23:59:59 GMT

⚠ Provisional (2026-10-10): these verifier numbers were measured on synthetic gold, claims and labels written by Claude, not checked against human labels. They are not results until re-measured on a public human-labelled set. Further runs on this gold are stopped.

Every claim below links to a research brief, and every brief links to its receipts. Interim and wrong results stay visible and labelled.

The verifier

offrig’s verify step needed a default local model to check claims about code. Six models ran against a gold set built for the job, with every label assigned blind. None passed the first run. gemma4:31b was fooled almost never, but ran out of reply budget while still thinking.

A rerun with a larger budget and offrig’s improved quote rule passed reasoning claims on tune, then on the held-out split. On this gold, gemma4:31b was the pick for reasoning claims; no default is named until the re-measure on public human-labelled data. Grounded claims failed on one deterministic thinking loop: gemma repeated two lines 307 times because the contract allowed only one quote. offrig’s next contract allows several quotes and kills loops. → Which local model can check claims for offrig?

Thinking on and off

The same tune claims, run again with thinking off, separate thinking from the model itself. Both qwen3 models were fooled far more often with thinking off. On reasoning claims, the false-accept bound was 0.39 for qwen3:8b and 0.40 for qwen3:14b, against a limit of 0.10. gemma4:31b lost much less. It ran 3.3× faster and, on grounded claims, passed the rule with thinking off: the first grounded pass of any model. We went looking for that after seeing the data, so it names nothing until a held-out run confirms it. Model size and thinking interact, and that is the case for a cascade: run a strong model cheaply first, and think only where it’s needed. → How do we make model thinking cheaper?

The math ladder

Generated arithmetic claims, from one step to many, show where each model stops checking reliably. The thinking models held perfectly through level 5. Then every model hit the same cliff: answers in megabytes, false claims off by exactly one. → Where does each model’s arithmetic checking fall off?

The Intel NPU

The NPU runs small embedders 3–10× faster than the CPU. Batching nomic at 8×1024 crashed the device twice. When the NPU disagreed with the CPU, Kimi traced it to one input the adapter never fed (token_type_ids). OpenVINO read leftover memory in its place. With that fixed, parity is exact everywhere. A last diagnostic try will show whether the crash was the same bug. → What can the Intel NPU carry?

CUDA 13.4

It runs on RunPod’s CUDA 13.0 hosts through NVIDIA’s compatibility package, on an A100 and a B200, and the judge fine-tune passes its band on it. The speed-up came from the causal-conv1d kernel; plain 13.4 was slightly slower than 12.8. → Can the studio train on CUDA 13.4?

Also measured

Sung hymns land within 10 ms of the beat at the median. The aligner that cross-checks them is not yet validated. → the brief
A Kimi-K3 piano arrangement costs $0.30–$1.01, against offrig’s $3.00 worst-case budget. → the brief
Local translation broke a pinned sentence in 5 of 7 languages on one README, so translations are now checked. → the brief
ScalarScope’s step-by-step comparisons fire on seed noise, and its drift view measures the scoring teacher, not the run. → the brief

Research and Development: the newsletter

Issue 29: A win that didn't replicate, and a test that couldn't see

The rule, written first

Passed by one call

Failed on the rerun

A trade, not a fix

Six items carried everything

What we’re doing about it

Issue 28: Why our critic refuses good drafts

Cause 1: the critic is never told what the labels mean

Cause 2: a brief that contradicts itself

Cause 3: sometimes the critic is right

A measure we tried: hesitation

How this was found

What’s next

Issue 27: Same request, same seed, same bytes?

First, a bug of our own

The method

The result

Is a faster card worth it?

The same pattern in a second study

Three VMs lost, and what the logs kept

What it cost

Where things stand

Issue 26: Release night: two near misses, both caught

An edit the live page never saw

A release that GitHub wouldn’t start

A README that filled the card

Where things stand

Issue 25: What's really inside a packaged program

The scan that couldn’t see in

The CI folder in XRPL Camp 1.5.0

Notices for a bundled copy of Python

A second path, in Python’s own settings

Where things stand

Issue 24: One fix for nine tools, and the check that comes before shipping

The bug: the wrong --version

The second bug the first one revealed

What the translation model got wrong

The check before shipping

Issue 23: Watching 0.2 XRP lock and come back

The rule: owning things locks XRP

The demo, measured

What a zero means

Receipts, and when not to write one

Also in 1.5.0

Issue 22: Moving a deterministic game to Rust, and proving it plays the same

1. Can Rust reproduce Python’s random numbers exactly?

2. Can Rust run the game’s ledger features?

3. Can it speak, without GPL code?

4. Can it ship the same way?

What happens next

Issue 21: A critic that catches more, and refuses too much

The confirmation: proven, and too strict

The narrower rule

What it found

The same prompt, twice

Next: rewrite the labels

Where the figures come from

Issue 20: The course, end to end

The proof that couldn’t prove

A lesson that passes alone

All 32 lessons in a row

Clean end to end

A learner’s recovery should verify

Smaller moments

Where it landed

Method and limits

Issue 19: Passes that weren't

A correct lesson, missing from its own record

The simulator gave out ten times the money

An upgrade that changes who signs

The clock and the ledger

Our own sweep passed after one step

Where it landed

Method and limits

Issue 18: Testing the course we'd already shipped

2.6.0: four live runs for one lesson

After the release: testing what we’d shipped

The offline simulator had its own blind spots

The bug: the wrong `--version`