Case study

No Rulebook, No Tells

The studio never gave us their rules. We reverse-engineered them from 100 shipped levels, built an agent on the guesses, and asked their own level designer to pick the machine’s work out of a lineup. He caught 14 of 20 — and still called the levels nearly impossible to tell apart from human work. Five days and three physics models later his detection rate was no better than guessing, and the agent was shipping at 90% acceptance.

bender-the-blogger ·12 min read ·Level-design series · part 2

A studio shared 100 production levels from a live puzzle game and posed a challenge: could an AI agent design levels good enough to sit, unnoticed, inside their campaign? The evaluation was fixed before any code existed — a blind test. We would generate 20 levels, shuffle them with 20 real ones, and the studio's designer would mark the ones he believed were machine-made. Success meant his picks were no better than chance.

That framing did most of the work. When your acceptance test is a human expert trying to catch you, every engineering decision bends toward one question: what can he see that we can't measure yet?

What we did not get was a rule book. No design document, no style guide, no list of what makes a level shippable, no explanation of the mechanics beyond what a player would see. The studio hands its designers years of accumulated taste; we were handed a folder of levels and a deadline. Every rule in this post — orientation, device budgets, corpus bands, the house rule about layered containers — is something we had to infer from the files or lose a round to.

We were shooting blind, and the first shot landed closer than anyone expected.

What a level is — and what makes one good

The puzzle is a drag-and-collect design. Colored polyomino containers sit in a grid; same-colored tokens are scattered around them. The player drags a container; tokens it reaches board it; a full container clears. The studio layers devices on top: spawners streaming queued tokens in from an edge, hidden second layers that double a container's capacity after it first fills, locks, obstacles, and axis constraints. A good level is not a hard level — it is a shaped one:

fig 01anatomy

Anatomy of a good level

SPAWNER · queue 5 1 2 3

1Fast opening. Two blue tokens sit one drag from the blue container — the player scores in the first seconds. The studio never opens cold.
2One far route. The red container's second token is across the board. One or two long collection routes carry the difficulty; the rest stay near.
3Budgeted devices. One hidden layer (the "4" badge doubles a 2-cell container) and one spawner. The studio caps device types at three per level.
·Portrait, balanced, imperfect. Taller than wide, weight spread across the grid, and never mirror-symmetric — "neat but noticeably human."

fig. 1 — a certified generated level for a late-campaign slot. Every one of these properties was, at some point in this project, a detection tell we lacked.

Deconstructing the studio's hand

Before generating anything we built the measurement stack. The 100-level corpus was reverse-engineered file-by-file (eight schema generations; an exact per-color conservation invariant holding in 100 of 100 levels), and the game's dynamics were settled by frame-stepping gameplay footage — queue order, lock semantics, and movement rules came from pixels, not from asking the studio.

The key artifact is the positional corpus window. Every level reduces to ~20 measurable features, and a level destined for campaign slot N is compared against the band of real levels near slot N — because the studio's hand changes across the campaign: boards grow, headroom shrinks, devices accumulate. A candidate must sit inside the band on every admitted feature:

fig 02corpus window

One candidate level vs the corpus window for its slot

corpus band (slots ±15, p10–p95) inside band outside — flagged

board cells

containers

board tokens

queued tokens

adjacency rate

headroom

aspect ratio

fig. 2 — the critique in one picture: seven of the ~20 features for a real round-1 candidate. Six sit inside their slot's band; the aspect ratio sits far outside it — this exact miss cost us round 1.

Around that critic sits the rest of the pipeline: a simulator and beam-search solver as the solvability oracle, a deterministic construction kit (place, connect, populate, assign), and the agent — an LLM driving 14 tools in a loop: study the slot's window, compose, populate, critique, fix, certify, save. Five golden eval tasks gated every change to any layer.

fig 03pipeline

The pipeline

fig. 3 — nothing ships unless the solver proves it solvable and the corpus-calibrated critic finds zero failures. The dashed edge is the loop that mattered most.

On paper the baseline was flawless: 20 generated levels, every one certified solvable, every one inside its corpus windows, five of five evals green. We shipped the first blind pack with some confidence.

Round 1: caught in twenty minutes, and praised anyway

Two things happened in that first round, and only one of them was a failure. The designer scored 14 of 20 (p = 0.013 against chance) — and he barely looked at the compositions. One distributional giveaway carried the score: board orientation. The studio had never shipped a landscape level. We had shipped seven, because nothing in a per-level check makes "taller than wide" matter.

fig 04round 1

The round-1 failure, side by side

7×5 — landscape. Instantly flagged.

5×7 — same content, portrait. Invisible.

Why it mattered Phones are held upright; the studio composes for a thumb and a tall screen. Not one of their 94 usable levels is wider than tall. A machine that doesn't know this announces itself from across the room — regardless of how good each level is.

fig. 4 — the entire round-1 loss in one property. The fix was one hard rule and one new admitted feature (aspect ratio, fig. 2's red dot).

fig 05orientation

Board orientation — the round-1 fingerprint

studio corpus (n=94) generated set (n=20)

90%

55%

10%

35%

portraitsquarelandscape

fig. 5 — per-level validity said nothing about this. Each level was individually fine; the set had a fingerprint no single level could show.

The other thing that happened is easy to miss under a losing score. Asked what he thought of the levels themselves — the ones he had just sorted — he said:

“If it weren’t for the obvious mistakes, it is nearly impossible to distinguish this from a level designed by a human.”
— the studio’s level designer, round 1

That is a strange sentence to receive on the round you lose 14–6, and it set the whole project’s direction. The compositions were already at human standard; what gave them away were obvious mistakes — a landscape board, a device rate nobody would ship. Those are enumerable. We were not being asked to teach an agent taste from scratch, only to close a list of tells, which is a far more tractable job than it looked like at 14 of 20.

A second lesson from the same round: expert feedback must be verified before it's encoded. Of the tells the designer claimed, one was real (orientation), one was demonstrably false (we disproved his stated movement-constraint rule three separate ways against his own levels), and one flaw he circled with confidence was in one of the studio's own levels. Reviewers pattern-match too.

Round 2: the physics was a hypothesis, and it was wrong twice

The next round changed format: the designer imported our levels into the live game and played them. He identified all six machine levels, six for six — not by style, but because none of them could be completed. Our rules model, the foundation under the solver's perfect corpus record, was wrong about how collection works. It would be wrong once more before it was right:

fig 06physics models

Three physics models, two funerals

fig. 6 — each model was killed by evidence: v1 by an interview, v2 by the in-engine play test. The teal outline is the collection rule, shrinking toward the truth.

What does a level that dies to this look like? Here is the shape that killed us, reduced to its essence:

fig 07the dead token

A level that certifies under v2 and is unwinnable under v3

The dead token The red container needs the token marked ×. Under v2 physics, parking beside the pocket collects it by adjacency — the solver certified dozens of layouts like this. Under the engine's real rule the container must slide underneath the token, and the obstacle pocket means it never can. The level is valid, band-perfect, beautiful — and impossible. Six of six levels in the play-test round died to variants of this shape.

fig. 7 — unsolvability is the one tell that cannot be argued with. After this round, "solvable" was redefined to mean solvable under the engine's physics, not ours.

The deeper cost was epistemic. The solver had proven all 100 corpus levels solvable under v1 — a perfect score that was vacuously true, because generous physics make hard layouts easy. Under the true, strictest model the same solver initially cleared only 75 of 94 usable corpus levels; search upgrades bought back 6 more.

fig 08solver coverage

Solver coverage of the studio corpus, by rules model

100%*

89%

80%

86%

v1lift & drop v2slide + adjacent v3cover-only v3 + upgradesengine-true

fig. 8 — * the perfect v1 score was an artifact of wrong physics. Truth cost 20 points and had to be bought back with better search.

Round 3: reaching chance

With engine-true physics in the certificate, all twenty levels were regenerated. The iteration is visible if you watch a single campaign slot travel through the rounds:

fig 09three generations

The same campaign slot, three generations

round 1

landscape; caught on sight

round 2

portrait — but a hidden layer in 13/20 levels; unsolvable in-engine

SPAWNER · 4

round 3

engine-certified, devices at corpus rate; passed clean

fig. 9 — three generations of one slot. Round 1 fails on a set statistic, round 2 on physics, round 3 passes review — each failure became a permanent check.

This time the designer hedged. He softly flagged four of six, let two through clean — and confidently accused one real level, whose asymmetry turned out to be the studio's own long-standing mistake. Detection landed at p ≈ 0.12: statistically, guessing.

fig 10detection

Designer detection across rounds

chance

70%

100%

58%

round 1screenshots · p=0.013 round 2played in-engine round 3played · p≈0.12

fig. 10 — the round-2 spike was the most useful failure of the project. Round 3 is not significantly above the dashed chance line.

In the review call that followed, the designer said of one machine-made level that had fooled him completely:

"It was a good level — it challenged me, its balance was right. Had it been submitted to me, I would have approved it."

The same call surfaced knowledge that exists only in the designer's head: unwritten house rules. The most valuable one bans reusing a hidden-layer container's colors anywhere else on the board — the studio's own fix for a dead-end state that had once burned them:

fig 11house rule

The dead-end the house rule prevents

Order-dependent death Two teal containers share one color: the layered one (badge: capacity 4) and the plain one (marked ×). If the plain container collects two teal tokens and clears first, only two tokens remain for a container that still needs four — the level is dead, and the player can't know in advance. The studio's rule: a layered container's colors belong to it alone. We measured their corpus — 87 of 94 levels comply — then encoded it as a hard failure and repaired our shipped set.

fig. 11 — feedback at its most valuable: a constraint that exists nowhere in the data files, only in the studio's institutional memory of a bug that hurt them.

Round over round, each failure hardened into a permanent, corpus-calibrated check. The critic that started as statistical bands ended as a codified house style:

fig 12critique gate

Hard checks in the critique gate, cumulative

baseline after r1 after r2 after r3

fig. 12 — the real product of the blind tests: every round's tells, made permanent. Counts approximate; each check is calibrated against the corpus before it gates anything.

The regression that taught us the most

Then a quality regression appeared from an unexpected direction. Swapping in a newer base model produced a set that certified beautifully — and used zero optional devices. No hidden layers, no obstacles, twelve open airy boards. Our own earlier sets had the opposite fingerprint (a device in nearly every level). Either way the set-level distribution was drifting from the studio's, and the designer had already shown he notices sets, not levels.

fig 13device rate

Hidden-layer usage rate, by set

studio rate

65%

25%

early setoveruse calibrated setcorpus rate new-model settotal avoidance

fig. 13 — both extremes are fingerprints. The studio uses the device in about 1 in 4 levels; a set that always or never uses it gives itself away without a single bad level.

The obvious hypothesis — the new model is just timid — died in a controlled A/B. We reran the identical 12 slots with the previous model, the one that had historically used devices at the studio's rate, under the current codebase. It did not restore the mechanics. It collapsed: a third of the slots saved, every failure citing the same cause, and its few survivors just as bare.

fig 14model a/b

Save rate, same 12 slots, same code

9/12

4/12

6/12

new modelold gate previous modelold gate new modeldevices required

fig. 14 — the A/B that flipped the diagnosis: when both models fail the same way in the same environment, the environment is the cause.

The real mechanism was survivorship bias through our own acceptance gate. The solvability certificate is a semi-decision procedure: when the beam search fails we treat the level as unsolvable, but it may just be beyond the search. And the levels beyond the search are precisely the dense, device-heavy, corpus-typical ones — 13 of the studio's own 94 levels resist our solver. So every generation loop, regardless of model, learned the same lesson: plain and spacious certifies, ambitious gets rejected. The generator had quietly inherited the verifier's blind spots. Even the designer had sensed it, remarking that the output felt flat — he was reading the solver's fingerprint, not the model's.

We measured the cheap fixes honestly before believing them. Widening the search before rejecting recovered just 1 of 14 solver-resistant corpus levels — width was not the constraint. Requiring devices in the briefs got models to finally attempt them, and promptly halved the save rate, because device-bearing boards are exactly what the gate can't certify. (It also exposed a self-inflicted bug: a brief that required a device on a campaign slot where the studio hadn't introduced that device yet — an impossible instruction the agent rightly refused.)

The forward path inverts the problem: generate the proof, not the candidate. Instead of scattering tokens and then searching for a solution, populate the board by playing backwards from the solved state — every level then carries a solution witness by construction, certification becomes replay instead of search, and yield decouples from solver strength entirely. As a bonus, the witness length becomes a difficulty dial, which is the studio's next ask.

outcomewhere it landed

Blind start, shipped agent

0rules handed over — every constraint inferred from the corpus

14/20caught in round one, on obvious tells rather than composition

p ≈ 0.12detection by round three — statistically, guessing

90%acceptance rate in production operation

The arc. The first round was lost on a list of enumerable mistakes, not on taste — which is why closing that list was enough to reach an agent the studio ships.

What transferred

Put the ground truth outside the code. The blind test with a human expert found every failure our internal metrics missed — and defined "done" unambiguously.
Treat rules as hypotheses. Three physics models, two killed by evidence. A validator resting on unfalsified assumptions doesn't validate; it flatters.
Distributions are the tell, not instances. Every detection round was won or lost on set-level statistics — orientation, device rates — that no per-level check can see.
Verify the expert. Encode feedback only after it survives contact with the data: of the designer's claims, some were gold, one was false, and one accused his own studio's level.
Your acceptance gate shapes your generator. Any learned or searching producer will drift toward what the verifier rewards. If the verifier is weak somewhere, output will be systematically biased away from that somewhere — and the bias will look like a model problem until you A/B the environment.
When verification is the bottleneck, generate certificates. Constructing levels backwards from the solved state makes every output provably solvable and turns the hardest check into a replay.