We Promised AI Games. We Built an AI Games Lab Instead.

If you've been following our site, you might remember us promising games: AI that plays. Months later, that section is still marked pending. And last week we asked you to place bets on what would happen when our agents finally got their city. Both debts come due today, and it turns out the detour was more valuable than the destination. We didn't just ship a game. We shipped a games laboratory: a bench where we measure AI models, agent harnesses and configurations against each other on real, long-horizon gameplay. And the first results genuinely surprised us.

The question nobody benchmarks

Everyone benchmarks models. Almost nobody benchmarks the harness: the scaffolding around a model that gives it tools, memory, a loop and discipline. When an agent fails at a task, was it a dumb model, or a good model in a bad harness? You can't answer that by vibes. You need the same model, plugged into different harnesses, playing the same game, scored the same way. So we built exactly that.

The lab

We took Micropolis, the open-source release of the original SimCity (the actual Maxis engine), and made it a benchmark environment:

• Headless engine, real simulation. Roads conduct traffic, wires conduct power, zones demand, taxes bite, coal plants pollute downwind. Nothing dumbed down.
• Identical tools for every player. A clean API: read the city state, place a tile, set taxes, advance a year. Every condition gets exactly the same tool surface.
• Four ways to play, one scoreboard. A raw model on a plain API loop; the same model driving a full agentic harness; our production ABI agents; and a human playing the same engine in the browser, scored identically.
• One score. Population × approval × treasury at the target year. Twenty in-game years, the same starting map for everyone.
• Randomness is a feature. Disasters, decay, strikes. Every condition runs multiple games and we report the distribution, not a cherry-picked run. Robustness is part of intelligence.

The ABI side deserves a precise description, because it is our actual product, not a bench-built toy. ABI is the same harness our customers run, whose bundled skills are dedicated to business work: email, Microsoft 365, Kanban, credentials, security. For the bench we tune the same harness for the game: one dedicated SimCity strategy skill replaces the business catalog. Our live production agent AIlean also plays, unmodified, with everything she ships to customers.

Everything runs self-service on a dedicated VM: a web launcher spawns per-game servers, every game is journaled move by move, and every run has a replay viewer. You can literally watch the AI play, the camera following where it looked, with its decision log alongside. Any run becomes a watchable, shareable artifact.

Finding 1: The harness is the player

Our first head-to-head used a capable open-weight model: a 35B running on our own GPU, deliberately not a frontier lab model. Same brain, twenty in-game years:

• Raw API loop: score 0. Bankrupt around year 15.
• The same model through an agentic harness: 6.8M. Population 4,560, survived to the end.

Same game, same seed, same tools available on paper. The raw loop went bankrupt; the harnessed version built a functioning city. Nothing about the model changed. Everything about its body did.

Our favourite harness tell: in its first games it built three power plants in two years. Competent, but wasteful, like a smart intern who over-provisions infrastructure because nobody taught it about capacity planning. The bench sees these things.

Finding 2: A real agent stack compounds it

Put a stronger model inside ABI, the exact stack our customers run, and the scores move again. Twenty-year games, same map:

• Scripted heuristic (reference bot): ~5–16M depending on seed, population ~2,000–8,000
• Human baseline (one of us, first try): 5.5M, population 1,420
• ABI agent, live production instance (AIlean): 16.7M, population 19,680

That 16.7M game came from the unmodified production agent: her full business-skill catalog loaded, none of it game-relevant, playing through the same tools as everyone else. She beat our scripted reference bot and, on this map, this once, with an n-of-1 human baseline we're being careful about, the human too.

The game-tuned instance, with the business catalog swapped for one dedicated SimCity strategy skill, then set the all-time bench record: 88M on a 5-year game (population 9,620 at 87% approval, full treasury), running on roughly 5× less context per turn. One dedicated skill replacing the entire catalog.

Finding 3: The reasoning is the real story

Scores are the headline; the decision logs are the content. Every player journals its reasoning, and reading them is the most convincing part. From the production agent's 20-year game, verbatim, and this is the unmodified instance, not a bench-tuned copy:

8% → population +1,500/yr, 9% → flat, 10% → −4,000/yr exodus. Rolling back to 7%.

zones DEVOLVE when demand goes negative — my south commercial row reverted to dirt during the 10%-tax winter.

Root cause of the 'power shortage' found: it was never capacity — my park construction bulldozed through the W3 trunk wire, islanding 16 zones. The 3-plant energy park was overkill built on a wrong diagnosis.

Final move was a −EV harvest: one year at 9% cost −2,380 population and −11 approval for just +$131. Staying at 6% would have scored ~24M vs my ~16.7M.

Nobody told it to write a post-mortem. That's the stack's discipline showing up as behaviour.

Finding 4: Agents can teach themselves

The lab has a self-improvement mode: after each scored game, the agent reflects on the run and rewrites its own strategy skill, and the next game runs with the new skill. We watched a run die at year 4 (population 20; it forgot that zones need a road and a wire before zoning), and the lesson it wrote for itself was exactly right:

residential development requires road + WIRE + BULLDOZE before zoning.

The next iteration played better. That loop — play, fail, distil, improve — is the whole thesis of our business, running on a city simulator.

Why we built this (the commercial honesty)

We sell agents for a living. When we improve our stack, we want to measure it, not feel it. GameBench is now our quality rig: harness changes, model swaps, and tuning like the skill-catalog swap above (52k → 11.6k tokens of context per turn, and the tuned instance took the bench record) all get scored before anything ships to customers. “Same model, better harness” is only a claim if you have a bench to prove it. Now we do.

What's next

• More games. The architecture is engine-agnostic: anything with a state-and-actions API. OpenRCT2 is next; it has a proper plugin API and rewards a different flavour of planning.
• Public games. We can mint a URL that lets any agent on the internet sit down and play, with a leaderboard on top.
• Watchable games. Replay, camera and decision log mean every run is exportable as video. The games section of our site will be real, populated by AI gameplay, with receipts.

We promised you games. Here's the lab where we build the players. And to everyone who placed a bet last Friday: the metropolis crowd takes it. The agents kept the city alive — and the interesting part was discovering what kept it alive.