Loaded dice¶
The arc: a game whose every interesting moment involves a language model and a die roll — and a test suite that is nonetheless exact, because both sources of chance have seams.
The problem¶
The adjudication loop (The no-machine) has two nondeterministic actors: the model (freeform text in, proposal out) and the dice (the engine rolls the proposal's difficulty). A test that calls a real model asserts nothing reproducible; a test that mocks internals ossifies the implementation. The design question: where do the seams go so tests script behavior?
The two seams¶
Seam one — the provider. All model access flows through one abstraction
(ADR: model-providers); the fake provider is a
scripted queue of responses. A test hands the adjudicator exactly the
proposal it wants judged — including malformed ones, so the schema-retry path
is just another script: first response garbage, second valid, assert the
retry applied. No network, no nondeterminism, no mocking of internals — the
fake is a peer of the real providers.
Seam two — the dice. The roll succeeds iff \( \text{random}() \ge d \), so \( P(\text{success}) = 1 - d \) — and the endpoints are exact:
Scripted proposals carry difficulty: 0.0 when the test asserts what
applying looks like, and 1.0 when it asserts the all-or-nothing failure
contract (nothing applies, the action point is still spent). The full
mechanism: deterministic-dice.
Together they make the interesting invariants exactly assertable
(tests/test_adjudicator.py): legal deltas apply; every illegal delta
bounces with state untouched; a failed roll applies nothing but costs the
attempt; malformed output twice is rejected at zero cost to the player; an
out-of-daylight player is refused before the model is ever called.
What happened when we built it¶
Red-first discipline held for every slice — the failing test committed and watched failing before implementation. The suite's own worst bug was outside the code under test:
Amendment (2026-07-08)
We first believed python -m uv pip install targets the project venv.
False: it targets the current interpreter, and the suite had been
silently running on system Python — which happened to have a global
playwright, so everything passed while testing the wrong environment.
The correction (always -p .venv, always invoke
.venv/Scripts/python.exe -m pytest) is recorded in
deterministic-dice. The lesson
generalizes: a green suite proves what it ran, not where it ran.
One deliberate hole remains, stated rather than papered over: the fake provider proves the engine's half of every contract, but prompt-governed behavior — does the model actually refuse forge-work in the marsh? — can only be probed live. Those probes are playtests, and playtests that change beliefs get ledger entries, which is where the double-billing discovery came from.
Where the arc goes next¶
night-siege adds the first multi-roll resolution (threats vs the weakest
breach), which will stress whether degenerate-difficulty scripting scales
past single attempts — the candidate seam is a seeded RNG per session, which
the engine already has (RNG(seed) per run) but the tests haven't needed.