Playable Was the Easy Part

fig 01engagement · summary

The deployment, in four numbers

50/50levels delivered

90%first-attempt success

34design tools

5rule-mining rounds

How it started

The head of Data & AI at a mobile gaming studio reached out with something to show us: levels for their hit hex-grid puzzle game, generated by an agentic workflow their team had built. It was serious work — an LLM-powered pipeline that generated a board, placed the pieces, sprinkled mechanics over them, validated the result, and exported it in the game's own format. The levels loaded. They were solvable. They looked, at a glance, like levels.

They also weren't shippable, and the team knew it — that's why they called. The levels played flat: everything slid out in a few obvious moves, nothing resisted, nothing had to be figured out. The studio hand-crafts its levels precisely because that resistance is the product. The question they brought us was whether that could be automated at all.

The game, in one paragraph

Pieces occupy chains of cells on a hex board; tapping one slides it head-first along its facing direction, and it escapes only if every cell on that line out to the edge is clear. Mechanics layer on top: ice that melts as other pieces escape, keys buried behind the locks they open, chained pairs, ropes and scissors, timed explosives. A good level is a knot — readable, fair, and tight enough that the order of escapes feels discovered rather than obvious.

Why a workflow can't design a level

When we measured the workflow's output against the studio's shipped levels, the flatness stopped being a feeling and became a number — three numbers, really:

fig 02deconstruction · medians per set

Depth of workflow levels vs the shipped corpus

Shipped levels (median) Workflow levels (median)

Data table

Metric	Shipped	Workflow
Escape waves	20.5	6
Knots (forced convergences)	4	1
Longest piece	15	5

What flat looks like

Six of the eighteen sample levels, rendered from the delivered files. Two things jump out before any metric does. The pieces are short — and long, weaving pieces are the game's depth carriers. And the boards repeat: all eighteen levels sit on just two distinct boards, recycled with reshuffled pieces. A pipeline stage whose job is “generate a valid board” has no reason to generate a different one.

fig 03workflow sample · delivered files

Six of eighteen workflow samples

fig 04sample vs final campaign

Two verdicts, same gates

The workflow's sample

0/18

2 distinct boards across 18 levels

Median escape depth 6 waves; corpus median 20.5

Features present but not load-bearing

The agent, by the final campaign

50/50

12 board families in one run, zero adjacent repeats

Gate medians 26 / 33 / 43 by difficulty tier

Every feature verified load-bearing before save

Here is the insight the whole project rests on. A workflow is a fixed sequence of steps — and a fixed sequence of generation steps is procedural generation, no matter how many LLM calls it contains. Each stage does its local job correctly: the board generator makes a valid board, the placer fills it, the validator confirms solvability. But the qualities that make a level worth playing — a deep escape order, knots that force convergences, tension that peaks late — are global, emergent properties of the whole arrangement. No single stage owns them, so no stage can be responsible for them, and a pipeline has no place to stand where it can see the whole knot and judge it.

Design is not a sequence. It's a loop: propose, play it, critique what's limp, revise, play it again — and stop only when the thing resists the way it should. That loop needs judgment in the middle, not validation at the end. Validation at the end is generate-and-pray; by the time a validator sees a flat level, the decisions that made it flat are twenty steps upstream. That loop — tools in the hands of something with taste and a standard to meet — is an agent.

So we took the project: deconstruct first, prompt later

The first weeks contained no agent at all. The sequence that ended up mattering:

fig 05build order · weeks 1–3

The sequence that ended up mattering

Corpus analysisevery shipped level, measured

Simulatorfull rules, exact escape order

Dimensionswaves, knots, traps, mobility

Tools + gates34 tools, hard validators

Agentdesigns, playtests, revises

Two decisions here carried the whole project. First, solvability by construction: instead of generating a board and hoping it can be solved, the builder places pieces in reverse escape order — when the piece that escapes k-th is placed, only pieces that escape after it exist yet, so its exit path is guaranteed clear at the moment it matters. Every board is winnable by construction; no generate-and-pray.

Second, quality is measured against the corpus, not against taste. Wave count, knot density, opening moves, feature depth — each has a distribution in the shipped levels, and generated output is judged by where it lands on those distributions. A value outside the corpus band is a finding to explain, never a win to celebrate.

If deterministic tools sound like the same trap the workflow fell into — that's the crucial distinction. The tools are deterministic; the sequence is not. A workflow fixes the order of steps and hopes quality emerges; the agent chooses which tool to reach for next, plays the result, reads the critique, and loops until the measured knot is tight. Determinism lives where it belongs (guarantees and measurements), judgment lives in the loop.

First contact: three of four rejected

The first sets went to the studio's designers for review, and the numbers were bad. Levels deadlocked — pieces facing each other head-on, neither able to move. Some couldn't be play-tested at all. In the first structured review round, covering seventeen levels, exactly four came back clean. A 24% acceptance rate.

What saved the project was treating that review as data instead of judgment. Every note the reviewer wrote became a measurable rule:

• “This unlocks in one move — it should be hidden like a key” → a burial rule: trigger features must sit at escape-wave 2 or deeper.
• “Too many pieces free at once” → a wave-width cap per difficulty.
• “The counter ignores the pieces in front” → a budget floor tying every explosive's counter to the measured escape order.
• “Feels Normal, tagged Hard” → knot and trap floors per tier.

Then we did the strange-sounding thing that worked: we compiled the reviewer. The rules went into a judge script, calibrated until it reproduced the human verdicts on all seventeen reviewed levels — 17 for 17. From that point on, every generated level faced the reviewer's standards before a human ever saw it, and the save tool simply refused levels that failed.

The mistakes were ours too

The workflow's failure modes are easy to narrate because we measured them. Honesty requires the other half: our early levels failed in ways a measurement harness should have caught — and didn't yet, because we hadn't built it.

fig 06first delivery · postmortem

The shipped deadlock

fig 07postmortem · four fixes

Our own failure modes, and what each became

1 · We shipped deadlocks

First delivery: several levels couldn't be finished, two couldn't even be play-tested. Head-to-head facing pairs and pieces whose own tails crossed their heads' paths. The fix wasn't care, it was construction: pieces are now placed in reverse escape order so a clear exit path is guaranteed, and the full tap sequence is replayed by the simulator before a level can be saved.

2 · Features as decoration

A cuttable rope that comes free in one move. A key sitting in the open next to its padlock. Chains on pairs that geometry already held together. The levels contained mechanics without being about them. The fix: a load-bearing test per feature — remove it, re-solve, and if the solution doesn't change, the feature is decoration and the review rejects it.

3 · One fingerprint isn't originality

When we automated duplicate detection, we fingerprinted the puzzle — the escape structure — and the generator promptly found one good board shape and reused it hundreds of times. Technically no two puzzles matched; visually it was the same level on repeat. The fix: originality is two-axis, silhouette and puzzle, and both must clear the corpus.

4 · Trusting the success rate

Runs kept “succeeding” while quality drifted — a level can pass every floor and still be limp. Success rate measures the enforcement layer; it says nothing about ambition. The fix: gate depth, drift, and variety are tracked per run against the corpus distribution, not just pass/fail.

The bar moves every time you clear it

Each accepted set raised the standard for the next. Difficulty tiers arrived with hard floors; opening-move bands tightened; pressure features became mandatory on the top tier. Five rounds of rule-mining, each triggered by a delivery, each ending as code:

fig 08rule-mining · rounds 1–5

Five rounds, each ending as code

Round	What the feedback taught	Became
1	Full boards, originality vs corpus, hole placement	validity gates
2	Difficulty bands, pressure features, trigger burial	tier floors
3	Feature ordering, required mechanics, measured flagships	review gate
4	Opening bands, effective depth, explosive margins	per-tier policy
5	Gate-aware pressure, separation, pacing	field recipes

Three eras, side by side

fig 09three eras · delivered files

First delivery → encoded rules → final campaign

And the figures climbed with it:

fig 10review records · all stages

Share of levels passing review

Data table

Stage	Passing	Of	Share
Studio's shipped corpus	25	60	42%
The workflow levels	0	18	0%
Agent — first review round	4	17	24%
Agent — post rule-mining	12	12	100%
Agent — final 50-level campaign	50	50	100%

What the levels look like

Three from the final campaign — one per difficulty tier, rendered from the delivered files. Colors are piece colors; letters mark mechanic features on a piece (Ice, Chain, Bomb, Key, Padlock, TNT); dark cells are holes.

fig 11final campaign · levels 203 / 204 / 249

One level per difficulty tier

Twelve board families, one campaign

Variety is scheduled, not hoped for: the fifty briefs rotate twelve board families and nine mechanics so no two neighbouring levels share either. One silhouette per family, from the delivered set:

fig 12final campaign · 12 families

Twelve board families, one silhouette each

The final campaign

The delivery that mattered was a single unattended run: fifty levels, numbered 201–250, on a schedule that rotated nine mechanics and twelve board families so that no two neighbouring levels shared either. Five levels were tagged hard and five very hard, per the studio's slot table.

Ninety-seven minutes later all fifty existed. 45 of 50 briefs succeeded on the first attempt; every retry converged by attempt four — and the retries concentrated exactly where they should:

fig 13run ledger · 50 briefs

Attempts needed per level, by difficulty tier

Data table

Tier	Levels	Avg attempts
Regular	40	1.20
Hard	5	2.00
Very hard	5	2.20

The set shipped. The studio's terms going forward are the healthiest sign the system works: one revision pass across the files, and after that each new generation is a billed unit — agent output priced like designer output.

The accidental experiment

Here is the finding we'd nominate for the year's most useful accident.

A modular rewrite of the agent — the 34 fine-grained tools consolidated into seven clean ones, the policy floors extracted into data, the architecture genuinely better — produced two fifty-level test sets. Same model, same simulator, same enforcement. The quality collapsed, and it collapsed selectively:

fig 14rewrite experiment · 3 runs

Flagship gate score by difficulty tier

Baseline agent Rewrite, run A Rewrite, run B

Data table

Tier	Baseline	Rewrite A	Rewrite B
Regular	26	7	4
Hard	33	19	30
Very hard	43	23	38

The diff explained it. The rewrite had kept one of the agent's two layers and dropped the other:

fig 15rewrite diff · two layers

What the rewrite kept and dropped

01

What the tools refuse

Per-tier floors as data (piece bands, knots, traps, margins)

The review gate: save refuses a failing level

Solvability by construction

Kept — enforcement

02

What the prompt aims for

The 128-line prompt shrank to 10; field recipes gone

Flagship targets: declare a gate depth, hold it through review

Theme discipline, palette bands, variety rotation

Lost — instruction

The tiers told the story with unreasonable clarity: coded floors survived the rewrite; instructed excellence did not. Enforcement sets the worst level you can ship. Instruction sets the best level the agent will reach for. You need both, and they are not interchangeable — a validator cannot demand ambition, and a prompt cannot guarantee a floor.

Two smaller technical findings from the same investigation, both of the class-of-bug that produces quiet quality loss rather than errors:

• Sampling parameters are per-model facts. The newest model family rejects an explicit temperature at the API; older ones need it pinned to 0. The baseline resolved this per model; the rewrite hardcoded one value. The symptom wasn't an error — it was near-duplicate boards appearing on adjacent levels.
• Timeouts are quality parameters. Large models routinely outrun a framework's default call timeout on long design steps. The baseline raised it; the rewrite dropped the raise, and the hardest briefs died mid-design.

What we'd tell you to steal

1. Build the simulator before the agent. Every hard guarantee we have — solvability, escape order, gate depth — comes from deterministic code the agent drives, not from the model.
2. Measure against the corpus, not against taste. The shipped levels define the distribution; the agent's job is to land on it or explain why not.
3. Compile the reviewer. Human feedback is priceless exactly once — when you turn it into a rule that runs on every future level for free.
4. Let the bar rise. A fixed acceptance test goes stale in two rounds. Every review that found a new failure mode made the gates stricter, and the success rate rose anyway — that's the signature of a system actually improving.
5. Keep instruction and enforcement separate, and keep both. Floors in code, ambition in the prompt. The rewrite experiment measured what happens when you keep only the layer that's easy to refactor.

All figures are measured from run ledgers and review records. Reviewed-set counts are human verdicts; later stages use the automated judge calibrated to reproduce them.