Zero Token Architecture: Getting the Model Out of the Hot Path

My homelab runs a daily report agent every morning. It pulls from a handful of sources, works out what changed overnight, and leaves a summary waiting before I’ve opened the laptop. I built it over a weekend, then mostly forgot it was there.

Then 2 months later I went into the logs chasing something unrelated, and started reading the traces properly. Every morning, the agent asked the model which tool to call first. Every morning, it got the same tool back. Then it asked what came next, and got the same answer to that, in the same order, across roughly sixty mornings.

The reasoning was sound every single time. It was also a decision I had already paid for, thrown away overnight, and paid for again at breakfast.

That’s the itch behind Zero Token Architecture, a term Kelsey Hightower put a name to this year and one that collected a fair amount of eye-rolling on the way. The idea fits in a sentence: use the model as an architect at design time, then let ordinary deterministic code do the running. Gartner expects more than 40% of agentic AI projects to be scrapped by the end of 2027, and unit economics sit near the top of the reasons. If you’re paying a model to re-derive the same plan a few thousand times a month, there’s a cheaper shape available.

What does “zero token” actually mean?

It means zero recurring inference inside the execution path of a workflow you already understand. The AI is still all over the system. It just did its thinking earlier, at design time, and left behind code.

Research has been circling this under several names. Progressive Crystallization is the clearest framing I’ve read, because it treats determinism as a dial rather than a switch.

A path starts at Type 3. Once it has run enough times to prove it’s stable, evidence promotes it down the table, and a failure demotes it back up. The paper reports a production system moving from 0% to 45% deterministic execution over eight months, with per-incident cost down more than 70% while volume doubled.

Does the evidence hold up outside a paper?

Reasonably well, from a few independent directions.

• Compiled AI treats the LLM as a compiler. It generates and validates code artifacts once, then runs them with no inference at all: 96% task completion, break-even against generation cost at roughly 17 transactions.
• The LOOP Skill Engine records a successful agent run once and replays it deterministically. Monthly token use fell 93-99.98%, latency 8.7x, with identical results across 100 replays.
• Anthropic’s code execution work attacks the same waste from the context side. Loading tool definitions on demand as code took one Drive-to-Salesforce workload from 150,000 tokens to 2,000, a 98.7% saving.
• NVIDIA’s small language model paper makes the adjacent case: most agent steps are narrow and repetitive, and a small model handles them at 10-30x lower cost per token.

Different labs, different mechanisms, same finding. Repetition in an agent’s execution path is money you’re setting on fire.

Will models always be probabilistic?

Yes, and more so than most teams assume. Setting temperature=0 only makes token selection greedy. It doesn’t make the arithmetic reproducible.

The culprit is batch-size dependence in GPU reduction kernels, which Thinking Machines Lab traced in detail. Your request gets batched with whatever else arrived that millisecond, the reduction order shifts, and the floating-point result shifts with it. Across five models, 0.3-1.3% of tokens flip on batch composition alone.

You can fix it. Batch-invariant kernels give bit-identical output across a thousand runs, at roughly 61.5% of the throughput. Few teams will pay that.

So determinism has to live in the code around the model. Structured outputs help, though they guarantee shape and not truth: a schema-valid JSON object can still be confidently wrong. Durable execution engines like Temporal draw the line well: workflow logic stays replayable, and every non-deterministic call is quarantined as an activity.

Where do the sceptics have a point?

The Register summarised the mood as “rebrand automation as zero-token architecture to master AI”, and honestly, that lands. A crystallized workflow is a script. We’ve had scripts since the 1970s.

Two failure modes are worth taking seriously.

The first is drift. Compiled artifacts break when an API adds a required field or a layout shifts, and the static pipeline fails exactly where an agent would have improvised through. Generated code that has lost the trail back to its originating prompt and model version is debt that regenerates badly.

The second is premature freezing. Crystallizing a workflow you haven’t validated just makes the wrong answer arrive faster and cheaper. Anthropic’s guidance on effective agents says it plainly: find the simplest thing that works, and add agency only where flexibility earns back its cost.

My rule of thumb is that a path earns its way into code by surviving production, never by looking tidy in a design doc.

For practitioners: How to crystallize your existing agents

flowchart TD
    subgraph observe["Observe (Type 3)"]
        A["Agent run: model picks each step live"]
        B["Trace log: tool, args, order, outcome"]
        C["Group traces by tool sequence"]
        A --> B --> C
    end

    C --> D{"One sequence covers most runs?"}
    D -->|"No, variance is real"| A

    subgraph crystallize["Crystallize (pay once)"]
        E["Model rewrites traces as one parameterised function"]
        F["Pin output schema and forbidden-tool checks"]
        G["Freeze tool sequence as a CI golden trace"]
        E --> F --> G
    end

    D -->|"Yes, shape is stable"| E

    subgraph execute["Execute (Type 1)"]
        H["Deterministic code path"]
        I{"Golden trace and schema pass?"}
        J["Emit result"]
        H --> I
        I -->|Yes| J
    end

    G --> H
    I -->|"No: API drift or new field"| K["Demote path, alert, fall back to agent"]
    K --> A

The first version of my report system taught me where this bites. Three agents, each producing perfectly valid Markdown, each in a format it had invented for itself. Pinning those output shapes took longer than building the agents did, and that pinning is what makes a path promotable later.

Five steps, in the order I’d do them:

1. Instrument before you optimise. Log every run: tool name, arguments, ordering, outcome. You cannot promote a path you cannot see, and Article 12 of the EU AI Act wants those records from high-risk systems by August 2026 anyway.
2. Count the repeats. Group the traces by tool sequence. The distribution is usually brutal: a handful of shapes cover most of the volume, with a long tail of genuine one-offs.
3. Let the model write the code, once. An LLM is very good at turning fifty similar traces into one parameterised function, and it only repeats the job when the shape changes.
4. Pin it with golden traces. Freeze the known-good tool sequence as a CI fixture and assert against it on every change. Schema conformance and forbidden-tool checks belong here, where they cost nothing to run.
5. Keep the agent as the fallback. When an assertion trips, demote the path and let the model improvise while you go and look. The agent becomes your exception handler.

If step 3 feels like too big a jump, start with semantic caching and a cheap classifier that routes to a fixed path. Both cut spend hard without giving up the agent loop, and both are reversible in an afternoon.

What I’d take away from all this

The branding is thin, and the measurements underneath it still hold up well enough to act on.

• Agents pay full inference price for decisions they’ve already made. That’s the actual bug.
• Determinism lives in the replay log, the schema and the code path around the model, well away from the weights.
• Crystallize on evidence, with a demotion path ready, because APIs move and layouts change.
• Instrumentation is the prerequisite for every one of these moves, and it’s the part teams skip.

That morning report agent I opened with is a Type 2 job on a good day. It’s on my list.

Here’s a small thing to try this week. Pull last month’s traces for one agent you run, group them by tool sequence, and count how many runs took an identical path. If that number is north of 80%, you’ve found your first candidate, and you already know what it should look like as code.

Recommended Reading

• I Built 2 Multi-Agent Systems - Here’s What I Learned About the Architecture - the orchestration patterns behind the homelab agent in this post, and where they broke
• MCP Explained from Inside Out - how agents connect to tools, and why tool definitions cost so much context
• Stop Chasing the Best AI Model - why architecture beats model choice for production cost and reliability