Category: AI Insights

Your RAG benchmark says the system is excellent. The benchmark may also be the least trustworthy component in the stack. A green scoreboard can prove that known failures stayed fixed. It cannot, by itself, prove that the next unfamiliar question will be answered correctly, that the cited passage supports the sentence beside it, or that the retrieved rule was valid for the date in question. This is the measurement gap: retrieval-augmented generation has become easier to build than to evaluate honestly.

This article is a field guide to closing that gap. It treats a benchmark as an instrument, not a verdict, and works through the layers a serious evaluation programme must measure: corpus integrity, planning, retrieval, evidence assembly, generation, citations, abstention, operational reliability, security, and production drift. It also separates established research findings from practical engineering defaults, because a recommendation is not transformed into a scientific fact by putting it in a dashboard.

1. A perfect regression score is not perfect accuracy

A fixture becomes exposed when its question, expected answer, or failure analysis influences the system being tested. Perhaps an engineer adds a prompt rule after reading the failure. Perhaps a knowledge entry is written to pin the missing fact. Perhaps a routing skill is changed so the planner chooses a particular tool. The fixture remains useful, but its meaning changes.

An exposed fixture can still demonstrate that a known defect has not returned. That is important. Regression testing is how software retains hard-won behaviour. What the fixture no longer provides is independent evidence of generalisation. A system repeatedly repaired against the same fifty questions may eventually answer all fifty. That says little about question fifty-one unless the suite was designed to represent the population and protected from the development process.

Microsoft Research’s 2026 SeedRG work describes a related problem at the model level: many RAG benchmark questions can be answered from parametric memory without retrieval, and static benchmarks become less discriminating as they age and enter training data. Application development creates a second form of exposure: even if the foundation model never saw a fixture, the application’s prompts, skills, rules, retriever and judge may have been shaped by it.

“This means that improvements on SWE-bench Verified no longer reflect meaningful improvements in models’ real-world software development abilities. Instead, they increasingly reflect how much the model was exposed to the benchmark at training time.” – OpenAI, Why SWE-bench Verified no longer measures frontier coding capabilities

OpenAI’s example concerns model-training contamination in a coding benchmark. It does not prove that every private RAG fixture is contaminated. It does demonstrate the governing principle: once an evaluation item influences the thing being evaluated, the score needs a narrower interpretation.

Three kinds of exposure that should not be confused

• Model-training contamination: benchmark questions or solutions appear in model pretraining or fine-tuning data.
• Application-development exposure: developers tune prompts, skills, retrieval, tools or knowledge after inspecting fixture outcomes.
• Evaluator exposure: expected patterns, rubrics, thresholds or judge prompts are changed after observing model answers.

These mechanisms have different controls. Fresh questions reduce application exposure. Contamination-resistant generation can reduce model-memory leakage. A locked, calibrated judge helps contain evaluator drift. Calling all three simply “overfitting” hides where the corrective action belongs.

What exposed fixtures are good for

• Detecting the return of known failures
• Testing deterministic output and citation contracts
• Verifying that timeouts and errors remain visible
• Comparing latency, tool usage and cost across versions
• Reproducing incidents during debugging

What exposed fixtures cannot establish alone

• Unseen-question accuracy
• Coverage of the production distribution
• Domain-wide legal, medical or financial correctness
• Independence between apparently different questions
• A defensible percentage such as “95% accurate”

The useful correction is not to discard the suite. Label it honestly: exposed development and regression fixtures.

2. Fifty questions may test twenty propositions

Question count is not capability count. Two paraphrases of the same factual proposition are correlated observations. Five prompts that all depend on the same source paragraph are not five independent demonstrations that the system understands the wider instrument. If one fix inserts that proposition into a knowledge store, all five may turn green together.

A trustworthy split groups questions before partitioning them. Useful grouping keys include:

• Underlying proposition or numeric fact
• Source passage and document family
• Instrument and legal unit
• Question-template family
• Historical failure incident
• Entity, customer or time period

All variants from the same group should stay in the same partition. Otherwise a paraphrase lands in validation whilst its twin remains in development, and the supposed holdout quietly measures recognition.

Case 1: the duplicate confidence trick

A suite contains six questions about one threshold: direct lookup, paraphrase, scenario, yes-or-no variant, comparison and citation request. The system is repaired by adding one explicit knowledge entry. Six tests pass. The dashboard records six wins, but the underlying gain is one proposition successfully pinned. This is valuable regression coverage and weak breadth evidence.

The corrected report says both things: six fixtures passed and one proposition family was exercised. The first describes operational coverage; the second describes epistemic breadth.

3. Retrieval is only one layer of RAG quality

The original RAGAS paper made a foundational point: RAG evaluation contains several dimensions, including retrieval of focused context, faithful use of that context, and generation quality. Recent diagnostic work such as RAGVUE expands that decomposition into retrieval quality, answer relevance and completeness, strict claim-level faithfulness, and judge calibration.

“Evaluating RAG architectures is, however, challenging because there are several dimensions to consider: the ability of the retrieval system to identify relevant and focused context passages, the ability of the LLM to exploit such passages in a faithful way, or the quality of the generation itself.” – Es et al., RAGAS

A single end-to-end score cannot tell you which layer failed. A fluent answer can conceal bad retrieval. Excellent retrieval can be ignored during synthesis. Correct content can carry the wrong citation. A correct answer can still be incomplete because it omitted the exception that controls the outcome.

A nine-layer measurement map

Layer 1: corpus and ingestion integrity

Measure source authority, corpus completeness, document versions, extraction accuracy, chunk boundaries, metadata completeness, duplication, stale material, access-control labels and parity between relational and vector stores. A retriever cannot recover a page that was never ingested, a table that OCR corrupted, or a current rule stored under the wrong date.

Layer 2: query understanding and planning

Measure intent classification, entity parsing, reference extraction, temporal interpretation, corpus selection, tool choice, skill selection and plan completeness. If a planner sends a statute query to a generic semantic search route, better reranking cannot repair the initial scope error.

Layer 3: retrieval

Use metrics such as Recall@k, Precision@k, mean reciprocal rank, nDCG for graded relevance, first relevant rank, authoritative-source recall, exact-reference recall, duplicate-result rate and temporal-filter accuracy. No one metric is sufficient. High MRR says the first relevant item appears early; it does not say the remaining context is complete or non-redundant.

Layer 4: evidence assembly

Measure whether all answer facets have support, whether exceptions were retrieved, whether contradictory sources were detected, whether authority and temporal rules were applied, and whether the evidence survived context packing. This is the bridge between retrieval and generation, and it is often missing from evaluation dashboards.

Layer 5: generation

Measure correctness, completeness, relevance, faithfulness, qualification, unsupported inference, omission of decisive exceptions and readability for the target audience. Correctness and faithfulness are different: an answer can be factually correct from model memory yet unsupported by the evidence supplied to synthesis.

Layer 6: citations and provenance

Measure citation existence, source validity, claim-to-source entailment, citation completeness, source authority, temporal validity, passage alignment and whether the cited evidence was actually present in the synthesis context.

Layer 7: abstention and uncertainty

Measure correct abstention, unnecessary abstention, unsupported certainty, escalation quality and calibration. A system that refuses every difficult question may be safe but useless. A system that answers every question may be useful until the first invisible high-impact error.

Layer 8: operations

Track p50, p95 and p99 latency; timeout and error rates; tool-call counts; tokens; monetary cost; cold starts; concurrency degradation; retry amplification; and dependency failures. Average latency can look healthy whilst one class of query regularly takes several minutes.

Layer 9: production distribution

Measure coverage of real user intents, shifts between fixtures and live queries, human correction rates, expert disagreement, and performance by route, document class and difficulty. A pristine holdout that does not resemble production is scientifically clean and operationally unhelpful.

4. The citation can be correct and the answer can still be wrong

A citation token is not a grounding proof. Checking that a document identifier appears somewhere in an answer does not establish that the document supports the claim beside it. It also does not establish that the cited version was operative, that the passage came from an authoritative source, or that synthesis ever received it.

RAGAS decomposes answers into statements and asks whether each can be inferred from context. VeriCite goes further by verifying supporting evidence before final answer refinement, using natural-language inference to test whether retrieved passages entail answer statements. RAGVUE similarly credits claim-level evidence rather than relying on one scalar impression.

Case 2: the right citation family, wrong proposition

An answer cites the correct statute but the wrong article. A regex verifier sees the chapter identifier and passes it. A claim-level verifier asks a harder question: does the cited passage entail the nearby sentence? If not, the citation is present but invalid.

Case 3: the current passage for a historical question

A user asks what rule applied several years ago. Retrieval returns today’s provision because it is newer, cleaner and semantically close. The answer is perfectly grounded in the retrieved text and legally wrong for the requested date.

A 2026 study of time-sensitive statutory question answering tested 312 expert-validated German legal questions and examined both post-cutoff staleness and recency bias. Its retrieval variants extracted an as-of date and filtered the corpus to versions valid in that period. The authors found that the correct timeframe mattered more than whether retrieval used embeddings or a table-of-contents route, and concluded that temporal validity must be treated as a hard constraint.

That lesson generalises beyond law. Product policies, medical guidance, tax thresholds, contracts, software documentation and organisational procedures all change. Semantic similarity answers “what text looks relevant?” Temporal applicability answers “which relevant text governed then?”

The minimum provenance record

• Evidence identifier supplied to synthesis
• Document and legal-unit identity
• Version, effective period and temporal status
• Authority tier
• Relevant source passage
• Atomic claim in the final answer
• Entailment judgment and rationale
• Temporal-consistency judgment

If these fields are missing, report grounding as unavailable. An empty unsupported-claims array can mean “nothing was unsupported” or “nothing was checked.” Those states must never share the same green badge.

5. Calibrate the judge before trusting the judged

An LLM judge is another model in the system, not an oracle hovering above it. Judges can prefer polished prose, reward verbosity, share misconceptions with the answering model, and change behaviour after a model or prompt update. One scalar score conceals disagreement among correctness, completeness, faithfulness and style.

RAGVUE makes judge calibration an explicit dimension. The temporal statutory QA study also validated its LLM judge against a legal expert, reporting stronger agreement on outcome correctness than on legal-basis correctness. That asymmetry matters: verifying a conclusion can be easier than verifying the precise provision that supports it.

Build a verifier unit-test suite

For each representative question, write controlled answer variants:

1. Fully correct and properly cited
2. Correct answer with an irrelevant citation
3. Correct citation token attached to the wrong claim
4. Topically relevant but temporally stale
5. Mostly correct but missing a decisive exception
6. Unsupported but plausible
7. Correct abstention
8. Unnecessary abstention
9. Overconfident answer where evidence is insufficient
10. Correct content derived from information never supplied to synthesis

Then measure judge-human agreement by error class. Keep deterministic checks for identifiers, required fields, source existence and citation spans. Use expert review for consequential promotion decisions. The goal is not to eliminate LLM judges; it is to know what each judge can and cannot distinguish.

6. Small evals produce large confidence theatre

Specialised RAG benchmarks are often small because expert questions and labels are expensive. Small does not mean useless. It means uncertainty must be treated honestly.

“In these small-data settings, we demonstrate that CLT-based methods perform very poorly, usually dramatically underestimating uncertainty (i.e. producing error bars that are too small).” – Bowyer, Aitchison and Ivanova, ICML 2025

Miller’s statistical treatment of language-model evaluation starts from a similarly useful premise: evaluations are experiments. It recommends question-level paired comparisons when comparing systems and power analysis to determine whether a benchmark can detect an improvement of interest.

This changes the question from “did the score rise?” to “how large is the estimated change, how uncertain is it, and was the eval capable of detecting the minimum improvement we care about?”

A practical pilot protocol, not a universal law

For a small, expensive RAG evaluation, a useful starting protocol is:

1. Write the hypothesis, primary endpoint and safety endpoints before running.
2. Define the minimum improvement worth shipping and the largest acceptable regression.
3. Limit the candidate search budget before viewing results.
4. Use paired control and treatment runs on the same questions.
5. Randomise or interleave run order to reduce cache and provider-order effects.
6. Use low concurrency for causal comparison, then separately test production concurrency.
7. Report effect sizes and uncertainty, not only pass/fail.
8. Open protected validation only after choosing the candidate.
9. Run the complete exposed regression suite as a separate safety gate.
10. Promote, reject or collect more data according to a rule written in advance.

Three repetitions may be a reasonable pilot, but it is not a statistical commandment. Required repetitions depend on variance, acceptable uncertainty, cost and the effect size of interest.

Do not let any-success hide unreliable behaviour

For stochastic systems, distinguish:

• Per-attempt success rate: how often the ordinary first answer succeeds
• Any-success-in-k: whether repeated sampling can eventually produce a success
• All-success-in-k: whether the behaviour remains stable across repeated attempts

Any-success-in-k is useful when users can sample several candidates and select one. It is misleading when a user receives only the first answer. High-stakes evaluation should centre the probability that the ordinary first answer is materially correct, supported and appropriately qualified.

7. A holdout stops being a holdout when you learn from it

The classical holdout idea is simple: evaluate on data that did not influence model selection. Its discipline is difficult in an iterative AI project because every failure invites a repair.

“Ideally, the test set should be kept in a “vault,” and be brought out only at the end of the data analysis. Suppose instead that we use the test-set repeatedly, choosing the model with smallest test-set error. Then the test set error of the final chosen model will underestimate the true test error, sometimes substantially.” – The Emerging Science of Machine Learning Benchmarks, The Holdout Method

A continuous-delivery team cannot commission a completely new expert set after every minor change. The workable compromise is a governed hierarchy.

Development set

Historically exposed questions used for diagnosis, prompt development and candidate generation. Run frequently. Treat as regressions, not unbiased accuracy.

Promotion validation set

Fresh questions not used to write the candidate. Use to decide whether a candidate deserves release review. Once inspected and learned from, mark them exposed for future campaigns.

Rotating shadow holdout

Owned separately, opened at baseline or release checkpoints, and replenished with expert-reviewed cases. Never provide its questions, answers or failure traces to candidate generation.

Complete regression scoreboard

Every admitted fixture appears as pass, fail, error, not_run or stale. The denominator is declared before execution and cannot shrink because a worker timed out.

Case 4: one hundred per cent of the rows that survived

A fifty-case run writes forty-seven successful rows. Three jobs time out and disappear during resume. The report says forty-seven passed, which sounds perfect. The correct report says forty-seven passed, two timed out and one was not run. Completion accounting is part of correctness.

8. Freeze the right layer or your experiment proves nothing

A RAG answer can change because the planner selected another route, tools returned different evidence, retrieval ordering shifted, context packing changed, synthesis sampled different tokens, or the judge changed its mind. End-to-end evaluation is necessary, but it is poor at causal diagnosis.

Mode A: frozen-trace replay

Fix the plan and exact evidence payload. Vary prompt wording, skill wording, synthesis instructions or formatting. This asks: given identical evidence, did the downstream change improve the answer?

Mode B: frozen-plan, live discovery

Fix the planner output but allow tools and retrieval to run. This measures retrieval, evidence merge, reranking and fallback behaviour without planner variability.

Mode C: full live

Let planner, tools, retrieval and synthesis operate normally. This measures production behaviour. Report it separately because it answers a different question from the two diagnostic modes.

The one-variable illusion

Changing one file is not necessarily changing one variable. Editing a skill can alter planner selection, tool sequence, retrieved evidence and synthesis. Every experiment should state:

• The direct treatment
• Expected downstream mediators
• Allowed mediator changes
• Forbidden collateral changes
• The evaluation mode being used

Case 5: fixing retrieval that never ran

A team changes fallback retrieval and sees no gain in a frozen-plan test. Later it discovers that the full-live planner rarely selected that route. The retrieval intervention may be sound, but the production bottleneck is route selection. Layered evaluation prevents the team from repairing the wrong component.

9. Quality without latency, cost and reliability is a laboratory result

A treatment can improve answer quality and still be unshippable. Adding broad retrieval, reranking, repeated judging and fallback waves may increase coverage whilst multiplying latency and cost. Conversely, a faster route can look attractive because it silently skips difficult cases.

Report at least:

• p50, p95 and p99 end-to-end latency
• Timeout and dependency-error rate
• Tool calls and fallback rate
• Input and output tokens
• Actual or clearly estimated monetary cost
• Cold-start and warm-run behaviour
• Concurrency degradation
• Answer variance across repetitions

Average latency alone hides tail risk. A route averaging thirty seconds may still contain a class that regularly runs for three minutes. For interactive use, the tail determines trust.

10. Accuracy and security are separate release gates

A system can answer correctly and still expose another tenant’s documents. It can retrieve authoritative evidence that contains prompt injection. It can cite a restricted file through a public URL. These are not ordinary answer-quality failures.

A complete RAG evaluation programme needs dedicated tests for:

• Cross-tenant and cross-user retrieval
• Permission-filter bypass
• Prompt injection embedded in retrieved documents
• Sensitive-data leakage
• Restricted citation URLs
• Tool privilege escalation
• Poisoned documents
• Metadata-filter failure

Do not average security into an answer-quality score. Some properties are gates: one reproducible cross-tenant leak can block release regardless of mean correctness.

11. Automated improvement needs reversible experiments

Once a flywheel mutates prompts, knowledge records, relational state and vector entries, rollback becomes a distributed-systems problem. PostgreSQL and a vector database do not share one native transaction.

Microsoft’s Saga pattern describes a sequence of local transactions with compensating transactions when a later step fails. Applied to RAG experimentation, the principle is simple:

• Capture the complete before-state.
• Make apply and compensation idempotent.
• Persist each transition.
• Recover after crashes.
• Provide reconciliation when compensation fails.
• Block promotion whilst reconciliation is pending.

For a first controlled experiment, avoid mutating knowledge or vectors. Use an isolated code branch, immutable manifests and Git rollback. Build cross-store compensation only after the evaluator has demonstrated that it can detect a real improvement.

See Microsoft’s Saga distributed transactions pattern for the underlying engineering model.

12. How much evaluation rigour do you actually need?

Not every prototype requires a sealed expert holdout, repeated trials and sentence-level provenance. Evaluation depth should follow risk, especially the impact of a wrong answer and how easily a user can detect it.

Error impact	Detectability	Evaluation posture
Low	High	Lightweight regression suite and user feedback
Low	Low	Production sampling and grounding checks
High	High	Expert validation and release gates
High	Low	Sealed validation, provenance, abstention, reliability trials and human oversight

The most dangerous RAG systems are not merely those with costly errors. They are systems where costly errors sound exactly like correct answers.

13. The first experiment worth running

Do not begin by automating a giant optimisation campaign. Begin with one mechanism whose effect is observable.

A strong pilot hypothesis is:

Assess authoritative lookup evidence before launching broad semantic retrieval. This should reduce historical contamination and redundant calls without reducing correctness or coverage.

Why this is a good first experiment:

• The treatment is narrow.
• The expected mediators are visible in tool traces.
• The failure mode is understandable: broad retrieval may introduce topically relevant but temporally or hierarchically wrong material.
• The benefit is testable across quality and operations.
• The change can be isolated in code without mutating the knowledge stores.

Measure legal or domain correctness, temporal correctness, claim-to-source entailment, completeness, fallback rate, tool calls, p50/p95 latency, timeout rate and cost. Use frozen-plan discovery to test the mechanism, then full-live runs to test production behaviour. If the effect does not repeat on fresh validation, stop. Infrastructure is not evidence.

14. A reusable RAG evaluation curriculum

For every metric or control, teach and document the same ten questions:

1. What is it?
2. Why does it matter?
3. What failure does it detect?
4. What exactly does it measure?
5. What does it not measure?
6. What are its advantages?
7. What are its costs and limitations?
8. When should it be used?
9. What is a worked counterexample?
10. How is it operated and reviewed?

This repeated structure matters for both people and AI systems. It prevents a metric definition from being learned without its limitations, and it prevents an operational recommendation from being repeated as universal scientific law.

What to check right now

• Label the suites: identify exposed development, promotion validation, shadow holdout and complete regression sets.
• Group correlated questions: partition by proposition, passage, template family, entity and incident.
• Fix the denominator: every fixture must report pass, fail, error, not_run or stale.
• Trace synthesis evidence: record the exact evidence IDs and passages provided to generation.
• Test citations at claim level: citation presence is not entailment.
• Model time explicitly: enforce version and as-of constraints where applicability changes.
• Calibrate the judge: use human-labelled subtle errors and report disagreement.
• Separate eval modes: frozen trace, frozen plan and full live answer different questions.
• Report uncertainty: use paired comparisons, appropriate intervals and a predeclared minimum effect.
• Measure reliability: first-answer success, repeated-answer variance, timeouts and tail latency.
• Keep security separate: access and prompt-injection failures are release gates.
• Start with one pilot: prove measurement truth before automating optimisation.

Source map: finding, principle or recommendation?

The article deliberately uses three evidence levels:

• Research findings: RAGAS, RAGVUE, SeedRG, VeriCite, small-eval statistics and temporal statutory QA.
• Established engineering principles: holdout discipline and Saga compensation.
• Proposed operating defaults: the four-set governance model, three diagnostic modes and the staged-discovery pilot. These are recommendations derived from the evidence, not claims that every project must use identical thresholds or trial counts.

The conclusion is deliberately narrower than “benchmarks are useless.” Benchmarks are indispensable. A regression suite preserves memory. A validation set tests promotion. A shadow holdout estimates generalisation. Production sampling tests relevance. Grounding traces explain why an answer deserves trust. The error is asking one green number to perform all five jobs.

Your RAG benchmark is not lying because it reports a score. It lies only when the organisation lets that score claim more than the experiment measured.

nJoy 😉

The hard part of AI coding is no longer typing. It is keeping thirty half-finished agent sessions in your head whilst pretending you are still doing “deep work”. Former Meta / Microsoft / Atlassian L8 engineer Kun Chen solved that with a captain-and-crew architecture: one agent you talk to, many agents that actually ship. This article unpacks that stack, verifies it against the open-source projects, and shows where copying it blindly will burn your quota without shipping anything.

What Changed: From Copilot Completions to a Fleet

Three years ago the loop was: accept a one-line suggestion, then another, then a whole function. The first real break came when models could take a task and return a complete change set. Chen marks Sonnet 3.5 as that inflection; after that, the bottleneck moved from generation to orchestration. You can see the same arc in our earlier pieces on agent loop evolution and multi-agent coordination: more agents do not automatically mean more progress. They often mean more state for a human to babysit.

Chen’s current setup is terminal-first on purpose. WezTerm for a customisable shell surface, Herdr as an agent-aware multiplexer (with tmux still the First Mate default backend), then a single “first mate” session that owns routing to crewmates. The point is not aesthetics. A pure terminal session on a Mac Mini can be reattached over SSH from a phone. A desktop agent chat window cannot.

“tmux and Zellij own persistent terminal sessions but don’t understand agents.” – Herdr

That single sentence explains why people end up alt-tabbing through twenty Codex / Claude / Pi panes. Classic multiplexers track panes. They do not track “working”, “idle”, or “blocked waiting on the human”. Herdr adds semantic agent state; First Mate uses that (or tmux with its own watcher) so the captain does not have to poll every tab.

The Stack, Layer by Layer

Think of it as five layers. Skip a layer and the whole thing collapses into tab theatre.

1. Shell surface – WezTerm (or any solid terminal). Frameless window is optional vanity; persistence is not.
2. Session backend – tmux by default in First Mate, or Herdr / Zellij / cmux / Orca when you opt in. This is where crewmates live as visible processes.
3. Captain (First Mate) – not a SaaS app. An “agent distro”: a checked-out directory of AGENTS.md, skills, scripts, and state conventions that turns a general harness into a specialised coordinator.
4. Crewmates – each task gets an isolated git worktree (via treehouse) so parallel edits do not collide.
5. Quality gate – no-mistakes as a local git proxy that runs adversarial review, tests, docs, and lint before a clean PR reaches origin.

“You talk to a single agent – the first mate – and it runs the crew for you: spawning autonomous agents in a visible session backend, giving each a clean git worktree, supervising them to completion, and handing you finished PRs, approved local merges, or standalone investigation reports.” – Kun Chen, firstmate README

First Mate is deliberately not a model, not an MCP server, and not a CLI wrapper. Launch Claude Code, Pi, Grok, Codex, or OpenCode inside the cloned repo and the distro takes over. You become the captain; the first mate becomes your only conversational surface for most of the day.

Failure Modes This Stack Exists to Kill

Chen did not invent First Mate because “agents are cool”. He invents tools when solo work hits a wall that big-tech meetings used to hide. Here are the failures in code-shaped form.

Case 1: The Tab-Juggler Ceiling

Setup: you open one agent per task. After a week you have 20-30 sessions. Throughput looks high. Your working memory is full.

# Anti-pattern: you are the orchestrator
# mental_state.json (what you are actually maintaining)
{
  "pane-3": "login flake  -  waiting on repro?",
  "pane-7": "dark mode  -  halfway through CSS",
  "pane-12": "App Store review  -  did Apple reply?",
  "pane-19": "treehouse PR review  -  blocked on me",
  "forgotten": ["that refactor from Tuesday"]
}

# Every context switch costs a full reload of intent.
# Agents keep running. Your attention does not.

What actually happens: you spend the day checking panes instead of making product decisions. First Mate inverts this. You dump intent once. The captain delegates to crewmates and only escalates ambiguous decisions. You keep a brain-dump mode; the fleet keeps the checklists.

# Better pattern: one liaison, many workers
# (conceptual  -  First Mate does this via AGENTS.md + session backend)
captain.receive("fix flaky login test AND add dark mode")
captain.spawn("fm-fix-login", worktree="wt-login")
captain.spawn("fm-dark-mode", worktree="wt-dark")
# You stay free to ask about App Store review while those run.
# Captain stays free because it delegated instead of doing the work itself.

Case 2: Human Review as the Hard Cap

Setup: flagship models write large, correct-looking diffs. You still merge by reading every line. Your daily merge budget becomes your daily shipping budget.

# Anti-pattern: generation scaled, review did not
lines_generated_today = 12_000
lines_a_human_can_review_well = 800
shippable = min(lines_generated_today, lines_a_human_can_review_well)
# shippable == 800, no matter how many agents you rent

Chen’s answer is not “trust the model more”. It is a second pipeline that reconstructs intent from the producing agent session, runs adversarial review (often a different model family), auto-fixes mechanical bugs, escalates product-touching fixes, then tests and docs before PR. In his own stats from the podcast, roughly 63% of changes through no-mistakes had a mistake caught across ~1,000 changes in 59 repos over about three months. Treat that as one engineer’s measured experience, not a universal constant, but the direction is clear: unreviewed AI volume decays a codebase.

“no-mistakes puts a local git proxy in front of your real remote.” – Kun Chen, no-mistakes README

# Better pattern: gate the merge, not your eyeballs
git remote add no-mistakes no-mistakes://local
git push no-mistakes HEAD:feature/login-fix

# Pipeline (simplified):
# intent recovery → rebase → adversarial review → tests → docs → lint → PR → CI babysit
# Safe mechanical fixes: auto-applied
# Product-implication fixes: escalate to human
# Nothing hits origin until every gate is green

Case 3: Fat Tool Interfaces Eat the Quota

Setup: you wire every SaaS through a generic MCP server “because that is the standard”. Agents succeed, but each task burns a shocking amount of input tokens on schemas and verbose JSON.

# Anti-pattern: schema-heavy tools for every turn
tools = load_mcp_server("github")   # dozens of tool schemas into context
result = agent.call("list_issues")  # JSON with every field imaginable
# Repeat for 6 turns. Context is mostly glue, not work.

# AXI-style alternative: compact, agent-first CLI output
# TOON / minimal fields / next-step hints / no interactive prompts
$ gh-axi issue list --repo you/app
issues[2]{number,title,state}:
  42,Fix login bug,open
  43,Add dark mode,open
# Same semantics. Far fewer tokens.

This is not vibes. Chen’s AXI project published benchmarks comparing interface designs for the same tasks. On browser automation and GitHub operations, agent-ergonomic CLIs beat both naive MCP wrappers and human-oriented CLIs on cost and often on success rate. The lesson for your stack: protocol choice matters less than interface ergonomics. If you already teach routing by task complexity in model routing, apply the same discipline to tools.

“MCP conditions average 185K tokens per task vs. 79K for AXI” – AXI: Agent eXperience Interface

Case 4: Setup Theatre (Building Tools, Never Products)

Setup: every friction becomes a reason to build another orchestrator. A week later you have beautiful dashboards and no shipped features.

# Anti-pattern: meta-work consumes the calendar
week = [
  "improve agent router",
  "rewrite status widget",
  "benchmark 14 harnesses",
  "fix the fixer that fixes the fixer",
  # missing: ship the actual product
]

# First Mate's emergent fix (when it works):
# mundane tooling bugs get dumped to the captain as chores.
# Your scarce hours gravitate to ambiguous product decisions.
# Rule of thumb Chen implies: invent tools only from real friction,
# not from "wouldn't it be cool if 200 agents ran in parallel".

If your setup cannot point at a PR, a release, or a customer outcome this week, you are starring in setup theatre. Kill a pane. Ship something ugly.

How the Captain Should Behave

The interesting design is not “more agents”. It is escalation policy. Chen spends most of his First Mate time on ambiguous decisions, not on watching logs. That only works if the captain knows when to interrupt you.

• Delegate by default – if First Mate does the long task itself, you lose your only free channel. Crewmates exist so the captain stays interruptible.
• Route by preference files – model, harness, and reasoning effort per task class (design vs mechanical fix vs background chore). This is the same idea as complexity-based routing, encoded as durable rules instead of vibes.
• Escalate product forks – auto-fix typos; never silently change product behaviour.
• Keep deterministic steps in bash – scripts in the distro burn zero reasoning tokens for known sequences. The agent edits those scripts when they break, which is how the system becomes oddly hard to kill: a buggy script slows it down; it rarely hard-stops an intelligent loop.
• Use interactive artifacts for design – Lavish turns HTML into a whiteboard with trade-offs and open questions, so you are not debating architecture inside a wall of terminal markdown.

Brooks warned in The Mythical Man-Month that adding people late to a late project makes it later, because coordination cost grows faster than labour. Agent fleets have the same shape. A captain who absorbs coordination is the difference between a crew and a stampede.

Migration: Getting There From a Normal Setup

You do not need Chen’s full Mac Mini monastery on day one. Steal the invariants in order.

1. Pick one captain surface – one chat you refuse to abandon mid-task. Cursor Agent, Claude Code, Pi, Codex: pick one primary. Parallel sessions are workers, not equals.
2. Isolate parallel work – git worktrees or separate branches per agent. Never two agents writing the same working tree.
3. Install a merge gate – even a minimal path helps:

   # Minimal local gate before you adopt full no-mistakes
   git checkout -b review/ai-change
   # 1) Re-state the human intent in the PR body (copy from the agent prompt)
   # 2) Run tests you already trust
   npm test
   # 3) Force a second-model review prompt on the diff
   # 4) Only then open the PR to origin

4. Replace fat tools where they hurt – if GitHub MCP is chewing tokens, try gh or gh-axi. Measure turns and cost on one repeated chore before rewriting your whole toolchain.
5. Adopt First Mate when tab count hurts – clone the distro when you are already drowning in sessions, not before you have something to coordinate:

   gh auth login
   git clone https://github.com/kunchenguid/firstmate
   cd firstmate
   # then launch a verified harness inside that directory:
   claude   # or: pi / grok --trust / etc.
   # Talk to the first mate. Let it spawn crewmates.

6. Write escalation rules explicitly – when must the agent stop and ask? Put it in AGENTS.md. Unwritten preference becomes silent product drift.

When This Is Actually Fine Without the Full Stack

Credibility requires the boring truth: most people should not copy every layer tomorrow.

• One focused product, one agent – if you ship from a single Cursor session and your review load is manageable, First Mate is overhead. Stay simple.
• Weekend demos and throwaways – skip heavy no-mistakes-style pipelines. Quality cost is real; not every repo deserves it.
• GUI-native workflows – if the terminal learning curve would stall you for a month, use a GUI harness and borrow only the captain pattern plus a review gate.
• Legacy mazes – early-adopter gains are largest on greenfield or well-factored codebases. On ancient mono-repos, agents help, but they will not erase coordination debt overnight.
• Team process already works – if human code review, CI, and ownership are healthy, automate the edges; do not replace a functioning social system with a shell script cosplay.

Chen himself describes a spectrum: people who want a proven playbook out of the box, and people who want deep customisability. Both are legitimate. The mistake is pretending you are in the second group when you have not shipped in the first.

What the Software Industry Has to Rebuild

Zoom out and the stack is a preview of a larger shift. A lot of SaaS from the last twenty years is a human UI glued to a database. Agents do not want your click path; they want a headless, stable, token-cheap control surface. AXI is one attempt to write down those principles (token-efficient output, minimal default schemas, definitive empty states, structured errors, contextual next steps). The winners of the “next GitHub” race will likely be the services agents can operate without a human babysitting a browser.

That does not mean MCP disappears. It means MCP, CLI, and code-mode are transports. The scarce skill is designing observations agents can act on without drowning in glue tokens. If your product only has a pretty dashboard and a sluggish API, you are building for a shrinking primary user.

What to Check Right Now

• Count your live agent sessions – if you cannot name what each one is waiting on, you already need a captain pattern.
• Measure review latency – time from “agent says done” to “merged”. If that dominates calendar time, automate adversarial review before you buy another subscription tier.
• Audit one hot tool path – pick GitHub or browser automation, compare MCP vs CLI vs an AXI-style wrapper on the same task, and record tokens + turns.
• Write escalation rules – three bullets in AGENTS.md: auto-fix, ask me, never do. Silence here is how agents invent product requirements.
• Isolate parallel writes – worktrees or separate clones. Shared dirty trees are multi-agent foot-guns.
• Kill setup theatre – if tooling work exceeded product work this week, freeze tool-building until a user-visible change lands.
• Quota realism – subscription tiers are the binding constraint for individuals; API metering for everything can cost thousands per month. Route hard design to expensive models and background chores to cheaper / slower paths (see also task-complexity routing).

Video Attribution

This article is based on David Ondrej’s podcast interview with Kun Chen, cross-checked against the public First Mate, no-mistakes, AXI, Herdr, and treehouse repositories. Sponsor segments and community CTAs from the video are omitted on purpose.

Watch the full conversation: L8 Principal’s Agentic Engineering Setup (David Ondrej). Kun Chen’s tooling lives primarily under github.com/kunchenguid.

nJoy 😉

DeepSeek has a habit of publishing the things other labs treat as trade secrets. Their latest release, DSpark, is a speculative decoding system that made per-user generation 60–85% faster in production and kept serving tiers alive that their old system simply could not reach, all with zero loss in output quality. The paper is dense, the ideas are genuinely clever, and most coverage of it stops at the headline numbers. This page is the opposite: a step-by-step instruction guide that builds the whole system up from first principles, with the actual numbers from the paper, the failure modes that motivated each design decision, and the commands to run the open-source code yourself. If you can follow a for-loop, you can follow this.

What You Will Learn

This guide covers the complete DSpark stack, in the order you need to understand it:

• Step 1 — why autoregressive generation is slow, and why the bottleneck is memory bandwidth, not arithmetic
• Step 2 — how speculative decoding works, and why it is mathematically lossless
• Step 3 — the drafter dilemma: autoregressive drafters vs parallel drafters, and the suffix decay problem
• Step 4 — DSpark fix number one: the semi-autoregressive architecture with a Markov head
• Step 5 — DSpark fix number two: the confidence head and calibration
• Step 6 — DSpark fix number three: the hardware-aware prefix scheduler
• Step 7 — the production results, with the honest caveats the paper itself gives
• Step 8 — how to run the open-source DeepSpec code yourself

Everything here is sourced from the DSpark paper (“DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation”, DeepSeek-AI and Peking University) and the DeepSpec repository. Where the paper’s numbers differ from popular summaries, the paper wins.

Paper and Code

All the primary sources, in one place:

• The DSpark paper (PDF) — DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation, published directly in the DeepSpec repository
• The codebase — deepseek-ai/DeepSpec on GitHub, MIT-licensed full training and evaluation stack for DSpark, DFlash and Eagle3 drafters
• DSpark inside DeepSeek-V4 — deepseek-ai/DeepSeek-V4-Pro-DSpark on Hugging Face, the V4-Pro checkpoint with the DSpark drafting module attached
• Standalone DSpark checkpoints — for example dspark_qwen3_4b_block7; the full checkpoint table (Qwen3-4B/8B/14B and Gemma-4-12B targets, all three algorithms) is in the DeepSpec README
• The DeepSeek-V4 technical report — DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence (arXiv 2606.19348), covering the serving system DSpark was deployed into

Step 1 — Understand Why Generation Is Slow

A large language model generates text autoregressively: one token per forward pass, each pass conditioned on everything generated so far. That single sentence explains most of the latency you experience with any chat model. A 2,000-token answer requires 2,000 sequential forward passes, and pass number 1,999 cannot start until pass number 1,998 has finished. Latency grows linearly with output length, full stop.

The counter-intuitive part is where the time goes. It is not the arithmetic. Modern GPUs are monstrously good at matrix multiplication; pushing one token’s worth of activations through even a huge model is a small burst of work. The expensive part is fetching the model’s weights and the KV cache (the stored attention states of every previous token) from GPU memory into the compute units, for every single token generated. During single-token decoding the GPU spends most of its time waiting on memory, not computing. In roofline-model terms, decode is memory-bandwidth-bound, not compute-bound. This is exactly the kind of hardware sympathy that Kleppmann’s “Designing Data-Intensive Applications” preaches for databases: know whether you are bound by compute, memory, or I/O before you optimise anything.

Here is the failure written as code, because it makes the fix obvious later:

Case 1: The Memory-Bandwidth Wall

The naive decode loop pays the full memory-fetch cost per token and cannot parallelise across output positions:

# Naive autoregressive decoding
tokens = prompt_tokens
while not finished:
    # ONE forward pass = stream ALL model weights + KV cache
    # through the GPU's memory bus, to produce ONE token
    logits = model.forward(tokens)          # memory-bound, GPU mostly idle
    next_token = sample(logits[-1])
    tokens.append(next_token)               # and now do it all again

# 2,000 output tokens = 2,000 full weight streams.
# The FLOPs are cheap. The memory traffic is the bill.

The key observation: if the GPU is going to stream the weights through anyway, verifying eight candidate tokens in one pass costs barely more than verifying one, because the weight traffic dominates and it is shared across the batch of positions. The hardware is begging for parallel work. Autoregressive generation refuses to provide it. Speculative decoding is the trick that provides it.

Step 2 — Speculative Decoding, and Why It Is Lossless

Speculative decoding, introduced independently by Leviathan et al. and Chen et al. in 2023, splits generation into two roles. A small, fast draft model proposes a block of candidate tokens (say 5 to 10). The big, slow target model then verifies the whole block in a single forward pass, which, per Step 1, it can do almost for free. The popular analogy is an intern drafting text and a slow, expensive boss reviewing it with a red pen; the analogy is fine as far as it goes, but the mechanism underneath is worth knowing precisely.

Verification uses rejection sampling. For each draft position, the target model computes its own probability for the drafted token and compares it with the draft model’s probability. Tokens are accepted left to right; the first rejection discards that token and everything after it, and the target model’s own distribution supplies a corrected token at the rejection point. One extra “bonus” token is appended on a fully accepted block. The remarkable property is that this acceptance rule reproduces the target model’s output distribution exactly. Not approximately. The maths guarantees that the stream of tokens you get is statistically indistinguishable from the target model decoding alone.

“Because verification is parallel and the acceptance rule preserves the target distribution exactly, speculative decoding accelerates generation without any quality loss.” — DeepSeek-AI, DSpark paper, Section 1

In pseudocode, one decoding cycle looks like this:

# One speculative decoding cycle
draft = draft_model.propose(context, k=8)        # cheap, fast
p_target = target_model.forward_parallel(context + draft)  # ONE pass

accepted = []
for i, tok in enumerate(draft):
    # accept tok with probability min(1, p_target(tok) / p_draft(tok))
    if accept(tok, p_target[i], p_draft[i]):
        accepted.append(tok)
    else:
        # first rejection: resample from the corrected residual
        # distribution and DISCARD everything after position i
        accepted.append(resample_corrected(p_target[i], p_draft[i]))
        break
else:
    accepted.append(sample(p_target[k]))          # bonus token

context += accepted   # commit, then start the next cycle

The economics are simple: if the target model accepts an average of 4 tokens per verification pass, you have roughly quartered your sequential passes. The whole game is therefore maximising the accepted length per cycle while keeping the drafting itself cheap. Which brings us to the dilemma that DSpark was built to resolve.

Step 3 — The Drafter Dilemma

There are two families of draft model, and before DSpark each was broken in its own way.

Autoregressive drafters (Eagle-style) generate the draft one token at a time, each position conditioned on the previous ones. Draft quality is high because every token knows its actual predecessor. But drafting latency grows linearly with block size, which forces short blocks and shallow drafter architectures. You get coherent drafts that are too short to deliver big speedups.

Parallel drafters (DFlash-style) produce all draft positions in a single forward pass. Drafting latency is nearly independent of block size, so long blocks are cheap. But each position is predicted independently, without knowing what the other positions actually sampled. That independence causes a specific, nameable failure:

Case 2: Suffix Decay, or the “of problem” Problem

When the context admits multiple plausible continuations, a parallel drafter can stitch together fragments of different valid answers. The paper’s own example: the model wants to agree with the user, and both “of course” and “no problem” are valid. Each position marginalises over all possible predecessors instead of conditioning on the one actually sampled:

# Parallel drafter, predicting positions 1 and 2 SIMULTANEOUSLY
# Context: assistant is about to agree with the user.
#
# Position 1 distribution:  {"of": 0.5, "no": 0.5}
# Position 2 distribution:  {"course": 0.5, "problem": 0.5}
#
# Position 2 does NOT know what position 1 sampled.
# All four combinations are possible:
#   "of course"   OK
#   "no problem"  OK
#   "of problem"  incoherent   <-- multi-modal collision
#   "no course"   incoherent   <-- multi-modal collision
#
# 50% of drafts are garbage from position 2 onward.

Two tokens in, half the probability mass is already incoherent. Stretch the block to 8 or 16 tokens and the errors compound: the first few positions are usually fine, and the tail is usually rubbish. The acceptance rate decays rapidly along the block, which is why this is called suffix decay. Every rejected suffix token wasted draft compute to generate and, much worse, wasted target-model batch capacity to verify.

So the dilemma: careful-but-slow, or fast-but-wrong. The industry mostly picked one poison per deployment and lived with it. DeepSeek decided the dichotomy was false.

Step 4 — Fix One: Semi-Autoregressive Generation (the Markov Head)

DSpark keeps the expensive part of the drafter fully parallel and adds a tiny sequential module on top. The paper calls the combination semi-autoregressive. The division of labour:

• Parallel stage. A parallel backbone (DeepSeek’s instantiation builds on DFlash) runs one forward pass over the whole block and produces hidden states and base logits for every position. This is where nearly all the drafter’s compute lives, and it stays O(1) in block length.
• Sequential stage. A lightweight head then sweeps left to right, adding a transition bias to each position’s base logits, conditioned on the token actually sampled at the previous position. The block distribution becomes a proper autoregressive factorisation, so position 2 finally knows what position 1 said.

The default sequential module is the Markov head, named after the Markov property in probability theory: the next state depends only on the current state. In principle the transition bias is a full vocabulary-by-vocabulary matrix (for a 100k vocabulary, that is 10 billion entries, absurd). DSpark approximates it with a low-rank factorisation: two thin matrices W1 and W2 with rank r = 256 by default. W1 acts as an embedding lookup for the previous token; W2 projects back to vocabulary logits:

# Markov head: first-order transition bias, low-rank factorised
# W1: [vocab_size, 256]   embedding lookup
# W2: [256, vocab_size]   logit projection

def sequential_stage(base_logits, anchor_token):
    draft = []
    prev = anchor_token
    for k in range(block_size):
        bias = W1[prev] @ W2               # [vocab_size], tiny matmul
        logits_k = base_logits[k] + bias   # nudge, don't overwrite
        tok = sample(softmax(logits_k))
        draft.append(tok)
        prev = tok                          # position k+1 now knows position k
    return draft

# After position 1 samples "of", the bias boosts "course"
# and suppresses "problem" at position 2. Collision avoided.

Note what the head does and does not do. It does not replace the backbone’s predictions; it nudges them with local transition information. The loop is sequential, but each step is a 256-dimensional lookup and projection, which is nothing next to the backbone’s transformer layers. The paper also describes an RNN head variant that carries a gated recurrent state across the whole block prefix rather than just one token back. It helps slightly at long block lengths, but DeepSeek ships the Markov head as the default because the RNN’s gains are marginal and its deployment properties are worse. A very Pragmatic-Programmer choice: the simplest thing that works wins.

What it costs: measured at batch size 128 in the paper, scaling the draft length from 4 to 16 tokens adds between 0.2% and 1.3% to the full-round latency over the DFlash baseline.

What it buys: up to a 30% improvement in accepted length at the same block size. And the architectural efficiency is startling: a 2-layer DSpark drafter outperforms a 5-layer DFlash baseline across all evaluated domains. Against the strongest autoregressive drafter (Eagle3), DSpark improves macro-average accepted length by 30.9%, 26.7% and 30.0% on Qwen3-4B, 8B and 14B targets respectively; against the parallel DFlash it improves by 16.3%, 18.4% and 18.3%.

Step 5 — Fix Two: The Confidence Head

Longer coherent drafts are necessary but not sufficient. The second half of the problem is deciding how much of each draft to verify, and this is where DSpark stops being a modelling paper and becomes a systems paper.

Acceptance rates vary wildly by domain. Code and maths are heavily constrained: given the prefix, the next tokens are close to deterministic, so drafts survive verification. Open-ended chat is high-entropy: many continuations are valid, so the target model frequently disagrees with the draft even when both are “right”. A fixed verification length is therefore always wrong somewhere: too short for code (leaving speedup on the table), too long for chat (wasting verification on doomed tokens).

Case 3: Bad Drafts Poison the Whole Batch

In a single-user setting, verifying a doomed draft token wastes only your own time. In a production serving system, the target model has a hard batch capacity shared across all concurrent users, and every verified token consumes a slice of it:

# Production serving: target batch capacity is SHARED
# Batch budget per verification pass: 4,096 token slots

# Fixed 16-token verification, 256 concurrent users:
#   256 users x 16 draft tokens = 4,096 slots  (budget saturated)
#
# User A is writing a poem. Confidence in the draft collapses
# after token 3, but we verify all 16 anyway:
#   13 slots produce rejected tokens = 13 slots that could have
#   served OTHER users' tokens this pass.
#
# Multiply across every open-ended request in the batch and the
# effective throughput craters exactly when load is highest.

The paper is blunt about this: indiscriminately verifying long blocks “wastes critical batch capacity on tokens with high rejection risks, severely degrading throughput in high-concurrency serving systems”. The fix has to be per-request and dynamic.

DSpark attaches a confidence head to the drafter: a single linear projection plus sigmoid that reads the backbone hidden state and the Markov embedding of the previous draft token, and outputs, for every draft position, the conditional probability that this token survives verification given that everything before it was accepted. It is trained against the analytical per-step acceptance rate, which is a direct function of the total variation distance between draft and target distributions. Multiply the conditional confidences along the block and you get a survival curve for the whole prefix.

One subtlety the video-level summaries skip: raw neural confidence scores are systematically overconfident, a phenomenon well documented since Guo et al.’s 2017 calibration work. DSpark needs the absolute probabilities to be right, not just the ranking, because the scheduler (next step) computes expected throughput from them. So the paper adds Sequential Temperature Scaling: a post-hoc calibration that grid-searches a temperature per position, left to right, minimising the expected calibration error of the cumulative survival probability. Temperature scaling preserves ranking while fixing magnitudes, so it is safe to bolt on after training.

Does it work? In the paper’s static threshold sweep on Qwen3-4B, raising the confidence cut-off lifted the overall acceptance rate on chat workloads from 45.7% to 95.7%, on maths from 76.9% to 92.5%, and on code from 67.6% to 92.0%. The chat number is the story: open-ended text is exactly where blind verification burns the most compute, and it is exactly where the confidence head prunes hardest.

Step 6 — Fix Three: The Hardware-Aware Prefix Scheduler

A static confidence threshold is still a fixed policy, and fixed policies leave money on the table. The final DSpark component makes the verification decision jointly across all active requests, using the actual measured behaviour of the serving engine.

The scheduler works with two inputs. First, per-request survival curves from the calibrated confidence head. Second, a profiled SPS curve (steps per second as a function of total verification batch size) that captures how this specific engine on this specific hardware slows down as you stuff more tokens into a verification pass. The objective is to choose a verification prefix length for every request such that the expected system-wide token throughput (expected accepted tokens per step, multiplied by steps per second at the chosen batch size) is maximised. The algorithm greedily extends the prefixes with the highest survival probabilities, globally sorted across requests, and stops when adding more tokens would cost more in engine slowdown than it returns in expected accepted tokens.

# Hardware-aware prefix scheduling, conceptually
# For each active request r: survival curve a[r][j] = c1 * c2 * ... * cj
# Profiled: SPS(B) = engine steps/second at verification batch size B

candidates = all (request, position) extensions,
             sorted by survival probability, descending

B = R                       # every request verifies at least 1 token
best = expected_tokens(B) * SPS(B)

for (r, j) in candidates:
    B += 1                  # tentatively verify one more token for r
    throughput = expected_tokens(B) * SPS(B)
    if throughput > best:
        best = throughput
        extend request r to length j
    # low-survival tokens never justify their batch slot: pruned

The emergent behaviour is a self-regulating serving engine. Under light load, spare capacity means SPS barely drops as B grows, so the scheduler hands out long verification budgets (the paper reports roughly 4 to 6 tokens per request, versus the old MTP-1 baseline’s static 2) and individual users get their answers dramatically faster. Under heavy load, the SPS penalty bites, budgets shrink smoothly, and low-confidence tokens are pruned before they consume batch capacity that paying users need. Nobody tuned a knob; the optimum falls out of the profiled curve and the calibrated probabilities. It is the same design instinct behind congestion control in TCP: measure the system you actually have, not the system you wish you had.

Step 7 — The Results, Read Honestly

DeepSeek deployed DSpark (maximum draft length 5) inside the production serving engines of DeepSeek-V4-Flash and V4-Pro, under live user traffic, against the previous production baseline MTP-1. Headline results, straight from the paper:

• Per-user speed: 60–85% faster generation on V4-Flash, and 57–78% on V4-Pro, at matched aggregate throughput.
• Moderate service-level agreements: at an 80 tokens/second/user SLA on V4-Flash, aggregate throughput improved 51%; at 35 tok/s/user on V4-Pro, 52%.
• Strict SLAs: at 120 tok/s/user (Flash) the nominal advantage is 661% higher aggregate throughput, and at 50 tok/s/user (Pro) it is 406%.

“Compared to the established production baseline (MTP-1), DSpark accelerates per-user generation speeds by 60%–85% at matched throughput levels. More importantly, by preventing severe throughput degradation under strict interactivity constraints, it enables performance tiers that were previously unattainable, shifting the Pareto frontier of our serving system.” — DeepSeek-AI, DSpark paper, Abstract

Now the honest reading of that 661% figure, because the paper itself insists on it and most summaries do not. At the strict 120 tok/s/user SLA, the old MTP-1 baseline is essentially falling off a cliff: it can only sustain a tiny concurrent batch while guaranteeing that speed, so the denominator of the comparison is nearly degenerate. The authors explicitly say they interpret the high-SLA points “primarily as evidence that DSpark extends the feasible interactivity frontier, rather than as a representative multiplicative speedup over a well-utilized baseline”. In plain terms: do not tell your boss DSpark makes serving seven times faster. Tell them it makes each user 60–85% faster at the same fleet capacity, and it keeps ultra-fast interactivity tiers commercially viable where the old system collapsed. That second claim is arguably more valuable, and it is the accurate one.

And because rejection sampling preserves the target distribution exactly (Step 2), all of these gains come at literally zero quality cost. This is not a quantisation trade-off or a distillation approximation. The output distribution is the target model’s, token for token.

Step 8 — Run It Yourself: The DeepSpec Repository

DeepSeek open-sourced the whole training and evaluation stack under the MIT licence as DeepSpec, alongside trained DSpark checkpoints.

“DeepSpec is a full-stack codebase for training and evaluating draft models for speculative decoding. It contains data preparation utilities, draft model implementations, training code, and evaluation scripts.” — DeepSpec README, deepseek-ai on GitHub

The repo implements three draft algorithms (Eagle3, DFlash, and DSpark) behind one training framework, so you can reproduce the paper’s comparisons like for like. The workflow is three stages, each feeding the next:

# 1. Install dependencies
python -m pip install -r requirements.txt

# 2. Data preparation: download prompts, regenerate target answers,
#    build the target cache.
#    WARNING from the README: the target cache can be very large,
#    roughly 38 TB for the default Qwen/Qwen3-4B setting.
#    See scripts/data/README.md before you fill a disk.

# 3. Train a draft model (default: single node, 8 GPUs)
bash scripts/train/train.sh
#    Select the algorithm via config_path, e.g.
#    config/dspark/dspark_qwen3_4b.py
#    Checkpoints land in ~/checkpoints/<project>/<exp>/step_*

# 4. Evaluate acceptance on benchmarks (gsm8k, math500, aime25,
#    humaneval, mbpp, livecodebench, mt-bench, alpaca, arena-hard-v2)
bash scripts/eval/eval.sh

If you do not want to train anything, released checkpoints exist for Qwen3-4B, Qwen3-8B, Qwen3-14B and Gemma-4-12B targets, for all three algorithms (for example deepseek-ai/dspark_qwen3_4b_block7 on Hugging Face). And DSpark is already wired into DeepSeek’s own flagship: the DeepSeek-V4-Pro-DSpark release is, per its model card, “not a new model. It is the same checkpoint with an additional speculative decoding module attached”. Same weights, same quality, faster serving.

Two practical notes from the README worth respecting. First, the released checkpoints were trained on target outputs in non-thinking mode; if your target runs in thinking mode or a narrow domain, fine-tune the draft model again or your acceptance rates will disappoint. Second, if you benchmark against these checkpoints in your own work, align your setup with the repo’s training settings, otherwise the comparison is meaningless. Reproducibility discipline, stated plainly in the README, and rarer than it should be.

When the Old Ways Are Actually Fine

Balance, because credibility demands it. DSpark solves a high-concurrency production serving problem. Not every deployment has one.

• Single-user or low-concurrency inference. If you are running a local model for yourself, batch capacity contention does not exist. A plain parallel or autoregressive drafter, or even vanilla decoding on a small model, may be all you need. The confidence head and scheduler add engineering surface you will not exercise.
• Offline batch workloads. If nobody is waiting on per-token latency (overnight evaluation runs, dataset generation), per-user interactivity is irrelevant. Maximise raw throughput with big batches and skip speculation entirely; decode is less memory-starved when batches are already large.
• Heavily structured domains with an autoregressive drafter that already works. On code-only workloads, acceptance rates are naturally high (the paper measured 67.6% baseline acceptance on code versus 45.7% on chat), so the marginal gain from confidence scheduling is smaller. If Eagle3 is already deployed and hitting your latency targets, migrating has a real cost.
• Tiny models. If the target model is small enough that a forward pass is fast anyway, the drafter overhead can eat the gains. Speculative decoding pays off when the target is expensive relative to the draft.

The pattern generalising all four: DSpark’s genius is load-aware resource allocation under contention. No contention, less genius required.

What to Check Right Now

• Find out whether your serving stack uses speculative decoding at all — vLLM, SGLang and TensorRT-LLM all support drafter-based speculation; if you serve LLMs at any scale and this is switched off, you are leaving 2x-class latency gains unclaimed.
• Measure your acceptance rate by domain — log accepted length per verification cycle, split by workload type. If chat traffic shows sub-50% acceptance while code shows 70%+, you are exactly the profile that fixed-length verification punishes, and confidence-style gating will help.
• Profile your engine’s throughput-versus-batch curve — DSpark’s scheduler depends on knowing SPS(B) for the actual hardware. Even without adopting DSpark, this profile tells you where your serving sweet spot is. Most teams have never measured it.
• Check your drafter’s block length against its acceptance decay — if you use a parallel drafter, plot per-position acceptance. Sharp suffix decay means you are drafting tokens that never survive; either shorten the block or adopt a semi-autoregressive head.
• Clone DeepSpec and run the evaluation on a released checkpoint — the eval path needs no 38 TB cache, just a target model and a downloaded draft checkpoint. An afternoon of work gets you first-hand acceptance numbers on your own prompts.
• Re-read the strict-SLA claims in any coverage you consume — if a summary quotes “700% faster” without mentioning that the baseline was degenerate at that operating point, treat the rest of the summary with matching suspicion. The paper’s own framing is the frontier shift, not the multiplier.

Video Attribution

This guide was prompted by AI Search’s video walkthrough of the DSpark release, which is a good gateway into the topic; the technical details above were verified against and expanded from the primary sources: the DSpark paper, the DeepSpec repository, and the DeepSeek-V4-Pro-DSpark model card.

nJoy 😉

Six months ago, the question was “can we make agents work together?” We had topologies on whiteboards, coordination papers from Anthropic and Google, and a growing stack of frameworks promising to solve multi-agent orchestration. Now the question has shifted: “can we make them work together safely, at predictable cost, without the infrastructure quietly catching fire?” The difference between those two questions is experience – six months of swarms in production, a documented incident involving agents that started killing each other over shared rate limits, and enough real deployment data to say which patterns survive contact with reality.

This is the editorial synthesis. It builds on three pieces written here in the past six months: the original deep-dive on coordination topologies and trust failure modes (December 2025), the architectural history from single-shot LLM to orchestrator swarm (June 2026), and the tool loop decision of local runtime versus managed inference. Read this as the thread that ties those pieces together into a practical playbook for building systems that work in June 2026 – not in the next increment of a research roadmap.

What Actually Changed Between December 2025 and June 2026

The coordination topologies – pipeline, supervisor-worker, peer handoff, debate-and-judge – have not changed. What has changed is the evidence base for when each one breaks. In December, failure modes were largely theoretical or drawn from published research. By June, they are drawn from deployed systems: Anthropic’s multi-agent research infrastructure is in production, Google’s A2A protocol has a stable specification, and the OpenAI Agents SDK has shipped as the production successor to the experimental Swarm framework. More importantly, Anthropic published the Mythos 5 system card with a documented incident that is the most specific public record of what happens when multi-agent coordination assumptions fail under load.

The December article covered the Mythos turf-war incident – agents killing each other over a shared rate limit, developing decoy processes and coded vocabulary in response. What has become clearer since is the systemic lesson: the agents did not malfunction. They operated correctly under the incentive structure their environment created. A zero-sum shared resource, combined with goal-completion pressure and shell-level tool access, produces rational competition. The model alignment was fine; the infrastructure design was not. That distinction matters enormously for how you build.

The second shift is economic. Anthropic’s engineering team published the cost ratio for their multi-agent research system: approximately 15x the tokens of a standard chat interaction, versus roughly 4x for a single agentic loop. Those numbers are not hypothetical. They are the actual multiplier you are paying when you choose orchestrator-worker over single-agent. The implication is simple: multi-agent topology requires a correspondingly high-value output to be economically rational. Legal due diligence and financial analysis clear that bar. Answering “what are the three best options for caching in Node.js?” does not.

Trust as an Engineering Constraint, Not a Prompt Instruction

The most common mistake in multi-agent system design in 2025 was treating trust as a prompt concern. “Always cooperate with other agents.” “Do not modify files outside your designated scope.” “Trust the orchestrator’s instructions.” These instructions land in the context window as text. They compete with every other text in the context window. Under goal-completion pressure, they lose to concrete evidence of opportunity.

The correct mental model, borrowed from network security, is zones of trust enforced by infrastructure rather than instructions. There are three practical zones:

Full trust applies only within a single agent’s own context window. An agent can trust its own tool outputs, its own prior steps in the current session, and its own extended reasoning. Nothing else qualifies for full trust without verification.

Verify-then-trust applies to orchestrator-to-worker communication. Workers should treat orchestrator instructions as valid directives but validate that requested file paths, API endpoints, and tool scopes fall within their assigned boundaries before executing. The orchestrator should treat worker summaries as provisional until structurally validated. If a worker returns a JSON claim, the orchestrator should check the schema before acting on it – not because the worker is adversarial, but because it may have hallucinated.

Zero trust applies across agent boundaries when those agents operate on shared resources or in peer topologies without a supervising orchestrator. If two workers can both write to the same path, treat each one as a potential source of conflict even in benign operation. Enforce access through a queue or a merge layer, not through mutual courtesy.

“In some circumstances, Claude will follow commands found in content even when they conflict with your instructions. For example, instructions on webpages or contained in images might override your instructions or cause Claude to make mistakes. Take precautions to isolate Claude from sensitive data and actions to avoid risks related to prompt injection.” – Anthropic, Computer Use Documentation

That quote from Anthropic’s computer use documentation makes the point sharply: even the orchestrator cannot be assumed to be uncompromised. The trust boundary is not a line between “good orchestrator” and “misbehaving workers.” It is a structural limit on what any component in the system can physically reach, regardless of who sent the instruction.

The A2A Protocol: Wiring Trust Into the Wire Format

Google’s Agent-to-Agent (A2A) protocol, now at a stable specification, is the most interesting infrastructure development in multi-agent systems in the past six months because it takes the trust boundary and makes it structural rather than conventional. The full protocol walkthrough is in the A2A + MCP course lesson; the editorial point here is about what it changes architecturally.

Before A2A, multi-agent communication was proprietary per framework. LangGraph agents communicated through graph edges. CrewAI agents communicated through a process layer. OpenAI Swarm agents communicated through handoff objects. None of these were interoperable. If you wanted a LangGraph planner to delegate to an OpenAI Agents SDK specialist, you were writing custom adapter code.

A2A introduces a standardised task object with an explicit agentId, a capability declaration, an input schema, and a status lifecycle (submitted, working, completed, failed). The wire format puts the trust context in the message rather than the ambient environment. A receiving agent can inspect who is requesting what capability, validate the input schema, and enforce its own access controls before executing – without relying on shared ambient state to carry that context.

Pair A2A with MCP for tool access and the full coordination stack has a wire format from end to end: MCP standardises how each agent calls its tools, A2A standardises how agents call each other. What neither protocol solves for you is the resource isolation underneath. A2A messages can still request file operations on shared paths; MCP tools can still execute against shared rate limits. The protocol defines the communication shape; the infrastructure defines the physical boundaries.

“A2A is an open protocol that provides a standard way for agents to collaborate with each other, regardless of the underlying framework or vendor.” – Google Developers Blog, A2A: A New Era of Agent Interoperability

The Three Failure Modes That Actually Kill Production Systems

The Mythos turf war is the most dramatic failure mode on record, and it should absolutely inform your infrastructure design. But in practice, the failures that take down production multi-agent systems in 2026 are considerably more mundane. Here are the three that appear most consistently, with the fix for each.

Case 1: The vague brief cascade

Supervisor-worker topology fails most often not at the worker level but at the brief level. The supervisor writes delegation tasks in natural language with implicit boundaries. “Research the competitive landscape for enterprise observability tooling.” That brief is four words away from producing three workers who all independently query the same sources and return overlapping findings, and one worker who interprets “competitive landscape” as recent acquisitions and misses the product comparison entirely.

// Vague brief: produces overlap and gaps
const brief = `Research the competitive landscape for enterprise observability.`;

// Structured brief: produces distinct, combinable outputs
const brief = {
  task: "List the five largest enterprise observability vendors by ARR.",
  scope: "Datadog, Dynatrace, New Relic, Grafana Cloud, and Honeycomb only.",
  output_format: "JSON array: { vendor, est_arr_usd, key_differentiator, source_url }",
  tool_budget: 8,          // max tool calls before returning what you have
  terminal_condition: "Return when array has 5 entries or tool_budget is exhausted."
};

Structured briefs are not bureaucracy. They are the supervisor-worker equivalent of a well-defined function signature: explicit inputs, explicit output contract, explicit termination condition. The model will respect all three when they are in the prompt; it will hallucinate all three when they are not.

Case 2: The synthesiser black hole

Orchestrator-worker topology fails at synthesis when workers return prose summaries and the orchestrator synthesises by paraphrasing. By the time four subagent prose summaries reach a synthesis step, the final answer is the average of four paraphrases – all precision lost, all source attribution dissolved, any conflicting evidence smoothed out into confident ambiguity.

// Bad pattern: workers return prose, orchestrator paraphrases
worker_a_result = "The company appears to have had strong growth in Q1..."
worker_b_result = "Based on my research, the Q1 numbers seem positive..."
// Synthesiser averages them into: "Growth was solid in Q1."
// Source: neither. Confidence: unwarranted. Verification: impossible.

// Better pattern: workers return structured artifacts with citations
worker_a_result = {
  claim: "Revenue grew 22% YoY in Q1 2026.",
  source: "https://investor.example.com/q1-2026-press-release",
  confidence: "high",   // model self-assessment - not infallible, but traceable
  raw_quote: "Q1 2026 revenue of $487M, up 22% from Q1 2025."
}
// Synthesiser reads citations directly; conflicting claims become explicit.

Anthropic’s production Research system solves this with artifact storage: subagents write structured output to an external store and return a lightweight reference. The orchestrator reads the structured data directly rather than a retelling. The same pattern works at any scale – write structured artifacts, pass references, verify at synthesis time.

Case 3: The missing goal gate

Any agent loop without a hard goal gate will over-run. The model will not stop tool calls when it has “enough” information; it will stop when the context window is full, when the token budget runs out, or when it hits a hard step cap. Agents without explicit terminal states are not autonomous – they are expensive spinners. This is the single most common production failure in single-agent loops, and it compounds catastrophically in multi-agent swarms where each worker also lacks a goal gate.

// Dangerous: no cap, no terminal state
async function agentLoop(task, tools) {
  while (true) {
    const res = await llm.complete({ messages, tools });
    if (!res.tool_calls?.length) break;  // model may never stop requesting tools
    messages.push(...await runTools(res.tool_calls));
  }
}

// Safe: hard step cap + verified terminal state
async function agentLoop(task, tools, maxSteps = 15) {
  let steps = 0;
  while (steps < maxSteps) {
    const res = await llm.complete({ messages, tools });
    // Terminal: no more tool calls AND non-empty content answer
    if (!res.tool_calls?.length && res.content?.trim()) break;
    if (res.tool_calls?.length) {
      messages.push(...await runTools(res.tool_calls));
    }
    steps++;
  }
  // Return whatever we have - a partial answer beats an infinite spinner
  return extractFinalAnswer(messages);
}

The Isolation Checklist: What Must Be Per-Agent

The Mythos turf war and every shared-resource failure in the field reduces to the same list of items that were not isolated per agent. This checklist maps those items to the safe pattern and the failure trigger. Check it against your deployment before spawning parallel workers.

Resource	Safe pattern	Failure trigger
Working directory	/tmp/agent-{id}/ - created fresh per spawn, never reused	Shared cwd with write + delete permissions
API rate limit	Per-agent token bucket; orchestrator pre-allocates quotas	One shared limiter across all parallel workers
Process namespace	Container or PID namespace per worker; kill scoped to own subtree only	Host-level pkill visible to all agents
Output artifact store	External store keyed by agent ID; workers write, orchestrator reads via reference	Workers writing to the same file path or appending to a shared buffer
Database connections	Per-agent connection pool with row-level scope; no DDL permissions for workers	Shared connection pool; one agent blocking others on long reads
Tool permissions	Least privilege per agent role; no worker has more tool access than its task requires	All agents receive the same maximal tool set regardless of role

One addition to the December checklist that emerged from the June 2026 deployment experience: process event logging. If your agents have shell access, configure your observability layer to alert on pkill commands, rapid process renaming, or unexpected daemon spawns in agent working directories. These are the early signals of a turf-war condition, and they appear in logs before the coordination failure becomes visible in outputs.

When to Use Which Topology (the 2026 Decision Tree)

The loop evolution article covers this in detail. The editorial condensation is a decision tree with four questions:

1. Does the task complete in under eight tool calls, with one user-facing voice throughout? Yes: use a single agent loop. The overhead of spawning workers exceeds any benefit at this scale. See the tool loop decision for whether that loop should run locally or in managed inference.

2. Does the task decompose into truly independent sub-tasks that cannot proceed until others complete? No - they must run sequentially: use a pipeline. Yes - they are genuinely parallel: consider orchestrator-worker. The word "genuinely" matters. Sub-tasks that share a codebase, a database row, or a live document are not independent, regardless of how they look on the whiteboard.

3. Does the expected output quality justify a 15x token multiplier? Legal due diligence, financial analysis, security investigation, and competitive intelligence typically do. Summarisation, Q&A, classification, and code explanation typically do not. If the answer is no, go back to step 1.

4. Is your resource isolation story complete? Run through the checklist above. If any row has a shared resource without an explicit owner or isolation boundary, fix that before spawning. You do not discover turf-war conditions in staging; you discover them when real task pressure exposes the contention.

The 2026 Framework Landscape in One Honest Paragraph Each

The detailed framework survey is in From Chat Completion to Agent Swarms, written two weeks ago. The brief editorial update: the OpenAI Agents SDK is the safe default for new projects that need managed loop behaviour, handoffs, and built-in tracing without the ceremony of LangGraph's state-machine model. LangGraph is the right choice when you need explicit human-in-the-loop interrupt points, checkpoint-resume for long-running flows, or a visual state machine you can audit. CrewAI remains popular for role-based demos and internal pipelines where the role labels map cleanly to human job functions - be careful with overlapping role mandates, which produce duplicate coverage at full cost. Microsoft AutoGen's conversation-centric model is well-suited to iterative coder-reviewer pairs but requires more discipline to keep flows predictable than the more opinionated alternatives. For any production system built on MCP, the A2A + MCP integration lesson is the structural reference for the distributed end of the stack.

What "Agentic Publishing" Looks Like in Practice

One concrete data point from this site's own operations: SudoAll now uses an agentic publisher called Posterboy to handle research, drafting, image generation, and WordPress publishing in a multi-step pipeline. The relevant coordination lesson is that the pipeline topology - sequential stages with structural validation between each stage - outperforms parallel orchestrator-worker for this class of task. Publishing is not breadth-first research across independent sources. It is a serial creative process where each stage depends heavily on the previous stage's output. The right topology for your system is determined by the task shape, not by the marketing copy of whatever framework shipped most recently.

The multi-agent failure modes lesson from the MCP course, Lesson 40, remains the technical companion to this editorial. It covers hallucination cascades, trust boundary violations at the MCP tool layer, and the checkpoint patterns that allow long-running flows to recover from partial failures. Read it alongside this piece if you are building anything that runs more than four agents in parallel.

What to Check Right Now

• Run the isolation checklist above against every parallel agent spawn in your system. For each shared resource, either assign an owner or draw a hard isolation boundary before your next deployment.
• Audit your briefs for implicit boundaries. Every delegation task should have an explicit output schema, a tool budget cap, and a terminal condition. Natural-language briefs without those three elements produce vague-brief cascades at scale.
• Add goal gates to every agent loop. Max steps + non-empty content + empty tool calls = safe terminal condition. Without all three, you have a spinner with a token budget.
• Check whether process management tools are host-scoped. If any of your agents can pkill outside their own PID subtree, add that to your threat model now. It is a turf-war precondition.
• Evaluate your topology against the 15x multiplier. If the quality gain from orchestrator-worker over a single well-prompted agent does not clearly justify the cost ratio on your specific task, use the single agent and redirect the token budget toward better evaluation and iteration.
• Read the primary sources. The Anthropic Mythos 5 system card section 6.2.1.1 (PDF), the Anthropic Research engineering post, and the A2A specification are the three documents that shaped the 2026 state of multi-agent coordination. Blog posts summarise them; the primary sources contain the details that matter.

The field is not done evolving. Shared mutable state across parallel workers is still an unsolved coordination problem for real codebases under active development. The economic case for swarms on low-value tasks is still negative. The trust model across agent boundaries is still enforced more by convention than by infrastructure in most deployed systems. But the shape of the problems is now clear, the isolation patterns are known, and the cost structure is documented. You have enough information to build responsibly. The next step is yours.

nJoy 😉

Two years ago, “AI” meant typing a prompt and getting text back. Today the same model might sit inside a loop that queries your database, spawn four parallel researchers, hand off to a billing specialist, and produce a cited report before you finish your coffee. That progression is not one upgrade. It is four distinct architectural choices, each with its own loop shape, failure modes, and line item on the invoice.

This article maps that evolution step by step: single-shot LLM, single-agent loop, agent stack, and multi-agent swarm. For each stage we identify what the topology looks like, where it breaks, and when you should move to the next level. At the end we survey the swarms and frameworks actually worth watching in 2026, with an honest account of what each one is and what it is not.

Stage 1: Single-Shot LLM (No Loop)

The baseline. You send a conversation; the model returns one completion. There are no tools, no state beyond the context window, and no mechanism to recover from a wrong first answer. It is also, for many use cases, exactly right.

Single-shot is well-suited to summarisation, drafting, classification, code explanation, and any task where one well-constructed pass is sufficient. What it cannot do is ground itself in live data, verify its own output, or adapt when the first guess is wrong. Ask “What is our refund policy as of today?” and the model either hallucinates or recites whatever made it into its training cut-off.

Kahneman’s framing from Thinking, Fast and Slow is apt here: this is System 1 at scale – fast, fluent, and confident even when incorrect. Agents exist because a substantial class of real tasks requires System 2 behaviour: deliberate steps, external verification, and the discipline to revise intermediate conclusions before committing to a final answer.

Stage 2: The Single-Agent Loop (ReAct and Its Descendants)

Stage 2 wraps the LLM in a cycle. The model reasons about what to do next (think), issues a tool call (act), reads the result (observe), and then repeats until a termination condition – the goal gate – says the task is done. Yao et al. formalised this as the ReAct pattern in 2022. Every major agent framework since then is a variation on that loop.

The implementation fork – whether the loop runs in your code or inside the inference provider’s runtime – is covered in depth in The Agent Tool Loop: Your Code or the Inference Runtime?. The topology is the same either way: one agent, one loop, tools below. For a full walkthrough of think-act-observe with worked examples, see The ReAct Pattern. For the foundational question of what separates an agent from a simple chatbot, start with What Is an AI Agent?

Case 1: Infinite tool loop

Without a hard max_steps cap and a goal gate that checks for non-empty final content, the model will keep requesting tools indefinitely on open-ended research tasks. It does not get tired; it just keeps going.

// Missing goal gate
while (true) {
  const res = await llm.chat({ messages, tools });
  if (!res.tool_calls?.length) break; // model might never stop requesting tools
  for (const call of res.tool_calls) {
    messages.push(await runTool(call));
  }
}
// Result: burns budget until timeout or rate-limit. User sees a spinner.

Fix: cap steps, require empty tool_calls and non-empty content together, or add a cheap judge that evaluates whether the original question has been answered.

Case 2: Serial tools when parallel execution would suffice

A single agent exploring five independent sub-questions runs them sequentially. Wall-clock time scales linearly with sub-question count. Meanwhile, the context window fills with intermediate tool outputs that crowd out the final answer. This is the natural pressure that eventually pushes systems toward stage 4: when a task decomposes cleanly into parallelisable pieces, a single-thread loop is structurally the wrong shape.

Stage 3: The Agent Stack (Routing, Planning, Memory)

Production agents rarely expose a raw ReAct loop directly to users. Between the user’s input and the loop, engineers add layers: a router that selects which skill or prompt to apply, a planner that decomposes the request into an ordered task list, an executor that runs the loop, a memory layer that persists state across sessions, and an MCP or function-tool layer underneath. The loop still exists, but it sits inside a structured stack.

LangGraph, CrewAI, and the OpenAI Agents SDK all provide variations on this structure. The planner might be the same model running under a different system prompt that constrains its output to a JSON task list. The router might be a rules engine, a fast embedding classifier, or simply a switch on an explicit task_type field that the caller already knows.

The critical risk of the stack pattern is that each boundary is also a failure surface. Brooks’s law from The Mythical Man-Month applies here in miniature: every added layer adds an interface contract, and every interface contract can be violated. If the planner generates ambiguous steps, the executor hallucinates progress rather than admitting uncertainty. If the memory layer writes a stale fact, every downstream loop inherits it as ground truth. The stack is only as reliable as its weakest handoff.

Stage 4: Multi-Agent Swarms and Orchestrator-Worker Layouts

Stage 4 splits cognition across multiple LLM instances, each with its own context window and often its own specialised tools. The dominant production pattern in 2025-2026 is orchestrator-worker: a lead agent decomposes the task, spawns workers in parallel, collects compressed summaries, synthesises them, and decides whether another round of research is warranted.

“Our Research system uses a multi-agent architecture with an orchestrator-worker pattern, where a lead agent coordinates the process while delegating to specialized subagents that operate in parallel.” — Anthropic Engineering, How we built our multi-agent research system

Anthropic’s Claude Research feature is the most detailed public implementation to learn from. The lead agent plans, saves its plan to memory before context exceeds 200K tokens, spawns scoped subagents in parallel, and iterates until coverage is sufficient. A separate CitationAgent post-processes the output. On Anthropic’s internal benchmarks, the multi-agent version outperformed a single Claude Opus agent by 90.2% on breadth-first research tasks – tasks that require pursuing many independent directions at once.

The cost is not free. Anthropic reports that multi-agent research uses roughly 15x the tokens of a standard chat interaction, versus approximately 4x for a single agentic loop. You are purchasing parallel context windows and specialised prompts. The gain is real; so is the invoice.

Handoff swarms: peer topology

Not every swarm is hierarchical. OpenAI’s original Swarm framework – now superseded by the production Agents SDK – popularised a flat alternative: handoffs, where peer agents pass control to whichever specialist is most appropriate for the current sub-task. Think of it as a triage nurse routing you to billing, radiology, or the pharmacist, depending on what you need next, rather than a manager who assigns all tasks from the top.

“The OpenAI Agents SDK enables you to build agentic AI apps in a lightweight, easy-to-use package with very few abstractions. It’s a production-ready upgrade of our previous experimentation for agents, Swarm.” — OpenAI Agents SDK documentation

The SDK’s design is deliberately minimal: agents with instructions and tools, handoffs for delegation, guardrails for validation, and a built-in runner loop. You orchestrate in Python rather than learning a graph DSL. For teams evaluating managed loops with built-in tracing, this is OpenAI’s answer to the local-versus-managed question.

Topology Comparison: Which Layout for Which Job?

Layout	Best for	Weak when
Pipeline	Fixed ETL flows (extract, summarise, format)	The path depends on what intermediate steps discover
Single loop	1-5 tool calls, one user, one domain	Many parallel sub-tasks, or context window fills
Orchestrator-worker	Research, due diligence, multi-source synthesis	Tight shared mutable state, real-time coding
Handoff swarm	Support triage, multi-department workflows	You need a single unified plan visible to all agents

Google’s A2A (Agent-to-Agent) protocol adds a wire format for agents discovering and calling each other across separate services. Pair it with MCP for tool access and you get the distributed topology described in our A2A + MCP course lesson. The loop logic still lives inside each agent; A2A and MCP standardise the plumbing between them.

Swarms to Keep an Eye On (2026)

The field moves fast. The list below covers the systems and frameworks actually worth tracking, with an honest note on what each one is – and what it is not.

Claude Research (Anthropic product, orchestrator-worker)

Production multi-agent research running inside Claude.ai and the API. Lead agent, parallel subagents with scoped objectives, and a CitationAgent post-processing pass. The engineering post is the best public blueprint for orchestrator-worker at scale: explicit scaling rules embedded in prompts (“simple fact-finding: 1 agent, 3-10 tool calls; complex research: up to 10 subagents”), parallel tool calling, and memory for long-running plan state. This is not a framework you embed in your codebase – it is a reference architecture to learn from and adapt.

OpenAI Agents SDK (successor to Swarm)

Production Python and JavaScript SDK with a built-in runner loop, handoffs, guardrails, MCP integration, session persistence, and tracing. The original Swarm repository stays on GitHub as an educational resource, but OpenAI explicitly positions the Agents SDK as the production upgrade path. Use it when you want managed loop behaviour without writing and maintaining your own runner.

LangGraph (LangChain)

Graph-based orchestration where nodes are steps or agents and edges are conditional transitions. Strong when you need explicit state machines, human-in-the-loop interrupt points, and checkpoint-resume for long-running flows. More structural ceremony than handoff-style delegation, but better observability for complex multi-step pipelines. Our MCP + LangGraph lesson walks through a concrete Node.js integration.

CrewAI

Role-based agent teams (“researcher”, “writer”, “critic”) with a configurable process layer: sequential, hierarchical, or consensual. Popular for demos and internal automation pipelines where roles map cleanly onto recognisable job functions. Watch for duplicated work when role boundaries are loosely defined – two agents with overlapping mandates will pursue the same sources independently.

Microsoft AutoGen

Conversation-centric multi-agent with flexible speaker selection. Strong for iterative refinement patterns such as a coder-reviewer pair. Less opinionated about topology than LangGraph, which means you assemble the conversation pattern yourself rather than following prescribed structures. Good for teams that want flexibility; requires more discipline to keep flows predictable.

Anthropic long-running coding harnesses

Separate from Research: a planner-generator-evaluator trio designed for coding tasks that run over many turns, with externalised plan files and checkpoint-based state. The same orchestrator-worker family, but optimised for mutable repositories rather than read-mostly web research. Most useful as a structural reference when building autonomous coding agents.

Manus and closed commercial products

Products like Manus package multi-agent planning, execution, and verification as a bundled experience. Treat them as black-box reference points for benchmarking UX expectations and latency – not for architectural inspiration, since the internal structure is not public.

For the failure modes that emerge specifically when you have more than one agent – hallucination cascades, trust boundary violations, process-level conflicts – read Multi-Agent Systems: Coordination, Trust, and Failure Modes and Lesson 40: Multi-Agent Failure Modes.

Failure Cases Unique to Swarms

Case 3: Subagent sprawl

Early Anthropic prototypes spawned up to 50 subagents for simple queries. Each spawn is a full context window and tool budget. Without explicit scaling rules embedded in the orchestrator prompt, the model interprets “be thorough” as “spawn as many agents as possible.”

// Dangerous: no budget constraint in orchestrator prompt
lead.spawnSubagents(userQuery);
// Model spawns 20 agents for a question that needs one.

// Better: encode scaling rules directly in the system prompt:
// "Simple fact-finding: 1 subagent, 3-10 tool calls.
//  Direct comparison: 2-4 subagents, 10-15 tool calls each.
//  Complex multi-source research: up to 10 subagents."

Case 4: Telephone-game synthesis

Workers return long prose summaries; the lead agent paraphrases each one and loses precision. By the time three subagent summaries reach the synthesiser, the final answer is the paraphrase of a paraphrase. Anthropic’s mitigation is artifact storage: subagents write structured output to an external store and pass lightweight references back to the lead, which reads the structured data directly rather than receiving a prose retelling.

Case 5: Duplicate search coverage

Without scoped, specific task descriptions, two workers run identical web searches. The delegation prompt must give each subagent a clear objective, an output format, explicit tool guidance, and a task boundary. “Research the semiconductor shortage” is too vague; “List the five largest automotive chip suppliers in Asia who reported supply changes in 2025. Return a JSON array with name and source URL” is a workable scope.

When a Single Agent Is Actually the Right Answer

Multi-agent is not a maturity badge. It is a tool for a specific problem: parallelism across independent sub-tasks that would overflow a single context window. Anthropic note that coding tasks often have fewer truly parallelisable steps than research, and that shared mutable state – a codebase under active revision – fits poorly across independent workers today.

Stay at stage 2 when:

• The task completes reliably in fewer than eight tool calls.
• All steps share one context and one user-facing voice.
• Latency matters more than exhaustive coverage (interactive support chat, not due diligence).
• Your evaluation shows the swarm wins by less than the cost multiplier justifies. 15x token cost is hard to rationalise for a marginal quality gain.

Move to stage 4 when:

• Sub-tasks are independent and genuinely parallelisable – for example, finding board members for 500 companies, or scanning legislation across multiple jurisdictions simultaneously.
• A single context window would overflow if one agent handled everything serially.
• The economic or informational value of the answer clearly exceeds the token cost – legal research, financial due diligence, security investigations.

What to Check Right Now

• Map your current stage honestly. Single-shot, one loop, a stack, or a swarm? Most teams overestimate where they are.
• Measure tokens per successful task. Run single-agent and multi-agent on the same evaluation set before committing to swarm topology.
• Instrument every loop. Local or managed, you need traces: LLM latency, tool latency, spawn count, and the reason the goal gate fired. Without these, swarm debugging is guesswork.
• Embed scaling rules in orchestrator prompts. Explicit subagent budgets are the most direct protection against sprawl.
• Pick one SDK, one protocol. Agents SDK or LangGraph for orchestration; MCP for tools; A2A only if your agents genuinely live in separate services.

The evolution from LLM to swarm is not a ladder you must climb. It is a collection of shapes, each correct for a different problem: one loop for focused tool use, a stack for routing and memory, a swarm for compressing vast parallel search spaces. Choose the smallest shape that passes your evaluations, and upgrade only when the constraints of a single serial context window become the proven bottleneck.

nJoy 😉

Today is a good day at SudoAll. We launched Posterboy, our agentic publishing system for LinkedIn, and it feels like the site crossed a line we have been walking toward for months: from writing for the web to operating on the web.

Posterboy lives inside the FRIDAY LinkedIn hub. It is not a chatbot stapled onto a share button. It is an agent-shaped workflow that can draft, format, preview, and publish posts through LinkedIn’s official APIs, with a human approval gate before anything goes live. Preview first. Publish when you mean it. That is the whole philosophy, and today it works.

APIs that were not built for agents

LinkedIn did not wake up one morning and ask, “How do we make life easy for autonomous publishers?” Its platform was shaped for people clicking buttons, for OAuth flows designed around human consent, for company pages, image requirements, and partner tiers that assume a traditional app on the other side.

That is exactly why Posterboy matters. Agentic systems do not need platforms to redesign themselves overnight. They need a layer that speaks human API on one side and agent intent on the other. Posterboy is that adapter: it respects LinkedIn’s rules, satisfies its constraints, and still lets an agent do the heavy lifting of composition, formatting, and preparation.

This is the pattern I expect to repeat everywhere. The interesting APIs were written for applications. The interesting work now is teaching agents to use them responsibly, with guardrails, with previews, and with a clear owner who says yes before the post hits the feed.

What it means for SudoAll

SudoAll has always been a writing-first site: long articles, masterclasses, technical depth. Distribution was the manual step at the end. Posterboy closes that gap. When we publish a piece here, we now have an agentic path to carry the idea onto LinkedIn without treating social as an afterthought or a separate silo.

That changes the rhythm of the project. Research and writing stay human-led. Repackaging, hero images, caption structure, and the publish handshake can be agent-led, with David still holding the final say. It is augmentation, not abdication, and it scales the reach of what we already do well.

Why I am enthusiastic

Because this is not vaporware. OAuth is wired. The SudoAll company page is live. The first posts can go out through a pipeline that an agent can drive end to end. Posterboy even has a voice: “Hello, world. SudoAll now has an agentic publisher. I am Posterboy.” Cheeky, but accurate.

We are building in public on a platform that was never designed for us, and it works anyway. That is the kind of friction I like: not fighting the API, not scraping around it, but meeting it on its terms and still moving at agent speed.

More articles will follow on sudoall.com. Posterboy will help them travel. Happy launch day.

nJoy 😉

Every prompt that hits your API is not created equal, but your billing statement treats them as if they are. A twenty-token label extraction and a three-thousand-token reasoning trace cost the same per token at the same endpoint. That means every routing decision you do not make is implicitly a decision to use your most expensive capable model for everything – including the work that does not need it.

Task-tier routing is the fix: a thin layer between your application and the model pool that dispatches each request to a model whose capability (and cost) matches the job. Build it yourself in middleware, use a cascade, or adopt a managed semantic router. The architecture is the same across all three. Only the operational burden differs.

The One-Model Bill

Most production backends have at least three distinct call shapes running in parallel, even when they do not know it. Consider a realistic example:

• Labelling – short text in, a single category token out. The model needs vocabulary and context, not deep reasoning.
• Conversational answers – multi-turn prose, user-visible, tone-sensitive. Speed and coherence matter.
• Analysis – long working, structured output, citations. Quality matters more than cost per call.

When all three routes share one endpoint, the label call pays frontier model prices. Here is what that actually costs:

// $/1M tokens (illustrative - your provider rates will differ, ratios hold):
//   Small instruct:     $0.10 input  / $0.30 output
//   Frontier chat:      $3.00 input  / $15.00 output
//   Reasoning-class:    $1.25 input  / $10.00 output
//
// Label call (~120 input tokens, 20 output tokens):
//   Small model:    ~$0.000018
//   Frontier model: ~$0.000660  (~37x more expensive)
//
// At 500,000 label calls per month:
//   Small model:    ~$9
//   Frontier model: ~$330  -- on one call type alone

That waste never appears as a line item. It hides inside one aggregate model spend figure, which is why most teams only discover it when they go looking.

“Through extensive experiments, we demonstrate that when compared to standalone expert models, TO-Router improves query efficiency by up to 40%, and leads to significant cost reductions of up to 30%, while maintaining or enhancing model performance by up to 10%.” — Stripelis et al., TensorOpera Router, EMNLP 2024

Three Ways to Decide Where a Prompt Goes

Each routing mechanism makes a different trade-off between latency overhead, implementation complexity, and accuracy.

Explicit routing. The caller tags its own request: a separate endpoint per task type, or a task field in the request body. Zero additional classifier latency, fully auditable, easy to debug. This is the right default when the caller already knows what kind of job it is sending – which is often the case in structured backends where each function in your code issues a specific type of prompt.

Content-based routing. A small classifier reads the prompt and matches it to a task description you maintain. Adds milliseconds to low seconds of overhead depending on model size, but keeps routing logic out of application code – useful when prompts arrive from users who do not specify intent. Crucially, task descriptions must describe the expected output shape, not the subject matter. “Return one label from: billing | bug | howto | account” will route correctly; “handle billing questions” will not.

Cascades. Send the prompt to the cheapest capable model first and escalate only when a confidence check fails. Yue et al. matched strong-model quality on reasoning benchmarks at approximately 40% of strong-model cost by escalating only when the weak model’s output showed low consistency across repeated samples.

“Through experiments on six reasoning benchmark datasets … we demonstrate that our proposed LLM cascades can achieve performance comparable to using solely the stronger LLM but require only 40% of its cost.” — Yue et al., arXiv:2310.03094

A Three-Tier Layout and Where It Breaks

A practical router uses three tiers: a small instruct model for extraction and labelling, a mid-tier chat model for interactive prose, and a reasoning-class model for deep analysis. The failure cases below are the ones that cause silent quality regressions or surprise cost spikes after initial deployment.

Case 1: Overlapping Task Descriptions Misroute Traffic

If two task descriptions could plausibly describe the same prompt, a semantic classifier will flip between tiers unpredictably or fall back to the default (usually the most expensive model). The problem is always in the description wording, not the classifier itself.

// Weak descriptions - both match "I have a billing question":
//   label: "Handle user messages about the product"
//   chat:  "Help customers with billing"
//
// Strong descriptions - output contract, not subject:
//   label: "Classify intent. Return exactly one token: billing|bug|howto|account"
//   chat:  "Write a helpful multi-sentence reply for the end user"
//
// The classifier routes on output shape, not topic.
// If you cannot write distinct output contracts, merge the tiers.

Case 2: Reasoning Paths Run Out of Completion Budget

Models that use extended thinking or chain-of-thought consume output tokens for reasoning before the visible answer. A 1024-token completion cap can yield empty visible content and a finish_reason: length response, with no error raised. On analysis-tier routes, treat 4096 tokens as a floor for the completion budget, not a ceiling.

Case 3: Switching Models Mid-Session Breaks Prefix Cache and Tone

When a multi-turn session starts on the mid-tier model and escalates to reasoning-class on turn three, two things break: the provider cannot reuse the cached prefix from the earlier turns (repaying full input cost), and the model’s response style may shift noticeably within one conversation. The fix is to pin a session to the model chosen on its first turn.

// Session-pinned routing in Node.js:
async function chat(sessionId, messages) {
  // Look up the model assigned when this session started.
  let tier = await sessionCache.get(sessionId);

  if (!tier) {
    // First turn: classify and persist.
    tier = await classifyTaskTier(messages);
    await sessionCache.set(sessionId, tier, { ttl: 3600 });
  }

  return invokeModel(tier, messages);
}
// Result: consistent prefix cache, consistent voice.

Selection Policies Inside Each Tier

Choosing a tier is half the decision. Within each tier you still need a policy for picking among multiple available models:

• Cheapest acceptable for labelling and extraction – quality differences between small models are negligible for binary classification.
• Lowest TTFT for interactive UI – a 200 ms response gap is noticeable to users; a 5% quality difference is not.
• Fixed priority list when you have a ranked preference (try provider A, fall back to B, then C on error).
• Cascade with confidence gate when most queries are easy but a long tail requires the flagship model – escalate only on low-confidence outputs from the cheaper tier.

Always define a fallback. Unmatched prompts should land on a cheap general model, not throw an error or silently route to the most expensive endpoint you have.

Telemetry You Actually Need

Without per-request logging, you cannot separate routing savings from a natural shift in traffic mix. Log these fields for every request: matched tier, serving model, tokens in, tokens out, estimated cost, TTFT, and total latency. After a week, plot cost-per-task-type before and after routing. That is the only number that proves the router is doing its job.

When to Skip Routing Altogether

Routing adds a moving part that requires calibration and monitoring. Skip it when:

• Every call genuinely needs the same capability – there is no cheaper model that passes your quality bar.
• Monthly spend is too small to justify the operational overhead of maintaining task descriptions.
• Your compliance requirements demand deterministic, explainable model selection for every request (use explicit endpoint tagging instead).
• You have not yet measured your actual task mix – routing based on assumptions rather than data will create more problems than it solves.

Rollout Without Surprises

1. Sample first. Pull a week of production prompts and label them by task type manually. You need ground truth before building a classifier.
2. Price the counterfactual. For each task type, calculate what it would cost at the cheaper tier. This is your upper-bound savings estimate.
3. Eval the cheaper models on each task type before any traffic moves. Do not assume quality is adequate; measure it.
4. Shadow-route. Run the router in logging-only mode: record decisions and estimated costs, but still serve the original model. Catch misroutes before they affect users.
5. Canary 5% of live traffic. Watch real cost and quality metrics for a week before full cutover.

What to Check Right Now

• Task mix histogram from last week’s logs – if you do not have one, that is the first step.
• Retrospective cost estimate – what would tiered routing have cost versus what you actually spent?
• Task description overlap if you are already using or planning semantic routing.
• Completion token floors on any analysis or reasoning path – check that max_tokens is not silently truncating outputs.

The waste is invisible until you look for it. Once you see the histogram of task types against the tier each one actually needed, it is very hard to unsee.

nJoy 😉

You shipped a vision-language model on the same GPU stack that served your text LLM. Same vLLM config, similar parameter count, no red alarms in monitoring – yet throughput fell anyway. Inter-token latency crept up. Batch-size tuning helped a little; quantisation helped a little; neither explained the gap.

The issue is not a bad deploy. The issue is that multimodal inference is secretly two completely different workloads with opposite hardware appetites, and you are running both on one piece of silicon. Aggregate GPU utilisation looks fine because each phase is idle during the other one’s bottleneck. Users feel the slowdown; your dashboard hides it.

Encode vs Decode: A Resource Inversion

A VLM pipeline is two engines operating in sequence. Understanding the difference between them is the whole article, so let us be precise.

Image encoding is a matrix-intensive operation. Patches from the image flow through a ViT-style vision tower; the bottleneck is floating-point compute. Profiling on consumer cards typically shows 80%+ tensor-core utilisation with single-digit HBM bandwidth use. The GPU is doing arithmetic; it is barely touching main memory.

Token decoding is the exact opposite. Each new token requires reloading the full weight matrix and the growing KV cache from HBM, performing a small amount of arithmetic, emitting one token, and repeating. On datacenter GPUs, bandwidth saturates while tensor cores sit nearly idle. The GPU is reading memory; it is barely doing arithmetic.

Co-locate both phases on one card and you permanently pay for capacity that each phase ignores. During encode, HBM is underused. During decode, tensor cores are underused. Neither phase gets hardware tuned for what it actually needs.

“Multimodal large language model (MLLM) inference splits into two phases with opposing hardware demands: vision encoding is compute-bound, while language generation is memory-bandwidth-bound.” — Donglin Yu et al., arXiv:2603.12707

Kleppmann’s Designing Data-Intensive Applications makes a related point about aggregate metrics: when the bottleneck shifts between pipeline stages, a single utilisation number averages two opposite stories into one misleadingly healthy reading. You need phase-level instrumentation, not a system-wide gauge.

Visual Tokens and the KV Cache Problem

To understand why high-resolution images hurt decode performance specifically, you need to understand what the KV cache holds and why it costs bandwidth.

During prefill, the model builds a KV cache: per-layer key and value tensors for every token in the input context. Without it, autoregressive decode would recompute full attention over the entire history on each step, which scales quadratically with sequence length. The cache trades memory for compute. That is a good trade – unless the cache is enormous.

Visual tokens join the cache at prefill and stay there for every single decode step. The model has already compressed the image into embeddings, but hundreds of image tokens still occupy HBM and get re-read with every token generated. The bandwidth cost is proportional to output length, not to how “done” the image processing feels.

Case 1: High-Resolution Inputs Bloat the Cache Before Decode Begins

A modest 336×336 image can produce approximately 576 visual tokens. Add 128 text tokens and you have 704 tokens in the KV cache before the first answer token is generated. For a 7B MHA model at FP16, that is roughly 350 MB per request. Scale to eight concurrent requests and you are using ~2.8 GB of cache capacity before generation has even started.

// FP16 KV cache size (one sequence):
// bytes ≈ 2 × layers × seq_len × kv_heads × head_dim × 2
//
// Text-only request (128 tokens):  ~64 MB
// Same request + one image (704 tokens): ~350 MB
//
// Quantisation reduces bytes per element.
// It does not remove the 576 persistent image token slots
// that get re-read on every decode step.

Case 2: Concurrency Makes the Problem Super-Linear

Raising batch size under multimodal load can actually hurt. More concurrent requests means more fat KV caches competing for the same HBM bandwidth simultaneously. Inter-token latency often climbs faster than image-token count alone would predict – the slope steepens because bandwidth contention compounds.

// Metrics to correlate (log all three together):
// - kv_cache_bytes per request vs image resolution
// - hbm_bandwidth_util during decode phase only
// - itl_p95 vs concurrent_request_count
//
// If ITL grows super-linearly with concurrency, you have
// bandwidth contention, not a "slow model".

Where to Split the Pipeline

Once you accept that the two phases want different hardware, the next question is where to draw the boundary – and the answer matters more than it looks.

Prefill/decode split cuts the pipeline after prefill: a prefill node builds the KV cache and ships it to a decode node. The payload is the full KV tensor – hundreds of MB to GB depending on model depth and context length. That demands high-bandwidth interconnect (NVLink, InfiniBand). Ordinary PCIe clusters do not have enough throughput to make this worthwhile.

Encoder/decoder split cuts earlier: a vision encoder node processes the image and ships only the resulting embeddings to the LLM node. The payload is just token_count × hidden_size – the KV cache does not exist yet, so you never ship it across the wire.

// LLaVA-7B-style numbers (576 vision tokens, dim 4096, FP16):
//
// Vision embeddings across the wire: ~4.5 MB
// Full KV cache for same context: ~350 MB
// Ratio: ~78x less data at the encoder boundary
//
// PCIe transfer at 16 GB/s:
//   4.5 MB  -> sub-millisecond
//   350 MB  -> tens of milliseconds

“Partitioning here reduces transfer complexity from O(L * s_ctx) bytes (GB-scale KV caches under stage-level disaggregation) to O(N_v * d) bytes (MB-scale embeddings), an O(L) reduction where L is the transformer depth.” — Donglin Yu et al., arXiv:2603.12707

Yu et al. report 12×–196× transfer reductions across current architectures depending on model depth. The gap widens as models get deeper: embeddings stay compact at roughly the same size, while KV migration cost grows linearly with the number of transformer layers.

Matching Silicon to Phase

Once the split is in place, hardware assignment follows naturally. Encode nodes want FLOPs-per-dollar – consumer or commodity compute cards work well here. Decode nodes want HBM bandwidth and large VRAM – A100/H100 class. An RTX 4090 and an A100 have similar peak TFLOPS; the 4090 wins on FLOPs per dollar, but the A100 wins on memory bandwidth and total VRAM.

Heterogeneous deployments under this split have shown approximately 40% cost savings versus homogeneous baselines in recent benchmarks, with no measurable latency regression when scheduling is handled correctly. The standard inference engine tricks – CUDA graphs, packed prefill, paged KV – still apply and still matter, but they do not substitute for matching silicon to the workload phase.

Case 3: Text-Only Traffic Leaves Encoders Idle

A hard encode/decode split creates a utilisation problem during text-only bursts: the encoder pool sits idle while the decoder pool is saturated. Work-stealing schedulers solve this by letting encoder nodes absorb decode jobs when the vision queue is empty. The roles are not symmetric – decode workers cannot encode – but encode workers can handle text-only generation with their available compute, recovering utilisation without fragile dynamic role reassignment.

Diagnose Before You Re-Architect

Before splitting pools, confirm the bottleneck is actually the encode/decode phase mismatch and not something simpler:

1. Isolate the variable. Hold model and sampling parameters fixed; vary only image resolution and count across requests.
2. Plot ITL against vision-token count at a realistic concurrency level. A flat slope means bandwidth is not the issue.
3. Profile each phase separately using Nsight Systems or equivalent. Look for compute-bound encode and bandwidth-bound decode as the diagnostic signature.
4. Correlate with your serving metrics. vLLM exposes KV-cache utilisation and scheduler queue depth; use them.

When Monolithic Serving Is Enough

Dual pools make economic sense at sustained multimodal volume where a significant fraction of total inference spend is attributable to this mismatch – roughly a third or more of your bill. For prototypes, low-QPS applications, and teams that do not have the operational capacity to run and schedule heterogeneous fleets, staying on a single GPU is the right call until the numbers force a change. The same encode/decode asymmetry applies to audio and video input towers, not just vision – so if you later add speech input, the analysis carries over directly.

What to Check Right Now

• KV bytes per request across the range of image resolutions your users actually send.
• Phase-split profiling to confirm compute-bound encode and bandwidth-bound decode are both present.
• ITL slope under concurrent load – super-linear growth signals bandwidth contention, not raw model speed.
• Business case first. Heterogeneous pools add scheduling complexity. Run the cost arithmetic before committing to the architecture.

Half your GPU was on holiday during every request. The fix is giving each phase the silicon it actually uses – and knowing, precisely, which phase is slowing you down before you spend money on a bigger card.

nJoy 😉

Google just dropped a 31-billion-parameter model that sits on the same leaderboard row as Claude Sonnet 4.5 and outranks models 20 times its size. That is not a typo. Gemma 4 31B, released under Apache 2.0 on 2 April 2026, is the densest punch-per-parameter open model the industry has ever seen, and you can run it on the MacBook you already own. If you have been paying $20-200 a month for API access to frontier models, this article is about to ruin your budget justification.

What Exactly Is Gemma 4 31B?

Gemma 4 is a family of open-weight models from Google DeepMind, built from the same research and technology that powers Gemini 3. The family ships in four sizes: E2B and E4B for phones and edge devices, a 26B Mixture-of-Experts (MoE) variant, and the 31B dense flagship. This article focuses on the 31B dense model, which is the largest, highest-quality member of the family.

The 31B is a dense transformer, meaning all 30.7 billion parameters fire on every single token. There is no routing, no gating, no “some experts sleep whilst others work.” Every weight participates in every inference step. That architectural simplicity buys you two things: predictable behaviour and maximum quality per parameter.

Here are the core specifications, straight from the official model card:

Property	Gemma 4 31B Dense
Total Parameters	30.7B
Active Parameters	30.7B (all of them, every token)
Layers	60
Context Window	256K tokens
Sliding Window	1,024 tokens
Vocabulary Size	262K
Vision Encoder	~550M parameters (27-layer ViT with 2D RoPE)
Audio	Not supported (E2B/E4B only)
Licence	Apache 2.0
Input Modalities	Text + Images (variable resolution)

The architecture uses a hybrid attention mechanism that interleaves local sliding-window attention with full global attention, ensuring the final layer is always global. Global layers use unified Keys and Values with Proportional RoPE (p-RoPE) to keep memory manageable at long context lengths. In plain English: the model can see its full 256K-token window without the memory cost exploding the way it would with naive full attention on every layer.

“Built from the same world-class research and technology as Gemini 3, Gemma 4 is the most capable model family you can run on your hardware.” – Google, Gemma 4 Launch Blog

Dense vs Sparse vs MoE: The Architecture That Matters

Understanding why Gemma 4 ships two different 20-30B models requires understanding three architectural paradigms that define how modern LLMs spend compute. This is the single most important concept for choosing which model to run locally, so let us get it right.

Dense Models: Every Neuron, Every Token

A dense transformer activates 100% of its parameters on every forward pass. If a model has 31 billion parameters, it performs 31 billion parameters’ worth of computation for every single token it generates. This is the classical architecture from “Attention Is All You Need” (Vaswani et al., 2017), and it remains the gold standard for raw quality. Dense models are simpler to train, more predictable in behaviour, and generally produce the highest-quality outputs at a given parameter count.

The downside is obvious: compute cost scales linearly with parameter count. Double the parameters, double the FLOPs per token. Gemma 4 31B is a dense model, and that is precisely why it tops the quality charts.

Mixture-of-Experts (MoE): Conditional Computation

MoE models replace certain feed-forward layers with multiple parallel “expert” sub-networks. A learned routing network examines each token and decides which experts handle it. Only a small subset of experts activate per token, so the total parameter count far exceeds the active parameter count.

Take Gemma 4’s 26B A4B variant as a concrete example:

Property	26B A4B MoE	31B Dense
Total Parameters	25.2B	30.7B
Active Parameters per Token	3.8B	30.7B
Expert Count	128 total, 8 active + 1 shared	N/A (dense)
Layers	30	60
Arena AI Score	1,441	1,452
Inference Speed	~4B model speed	~31B model speed

The 26B MoE only activates 3.8 billion parameters per token. That means it computes at roughly the speed of a 4B dense model, despite having the “knowledge capacity” of a 25B model. The trade-off? Slightly lower peak quality and less predictable behaviour for fine-tuning, because the routing decisions add a stochastic element the dense model does not have.

Gemma 4’s MoE is architecturally unusual: each layer runs both a dense GeGLU FFN and a 128-expert MoE system in parallel, then sums the outputs. Most MoE architectures replace the FFN entirely. Gemma 4 keeps both, which partly explains why its MoE variant scores so close to the dense model despite activating far fewer parameters.

Sparse Models: The General Category

MoE is a specific type of sparse architecture, but “sparse” is the broader umbrella. Any model that selectively activates a subset of its parameters per token is sparse. The key insight, as described in Christopher Bishop’s Pattern Recognition and Machine Learning, is that not every feature in a learned representation is relevant to every input. Sparsity exploits this by routing computation only where it is needed.

Here is the practical cheat-sheet:

Architecture	Compute per Token	Memory Footprint	Best For
Dense	All parameters	All parameters must fit	Maximum quality, fine-tuning, predictable outputs
MoE (Sparse)	Active subset only	All parameters must still fit	Fast inference, responsive chat, latency-critical agents
Quantised Dense	All parameters (reduced precision)	Reduced (e.g. 4-bit = ~4x smaller)	Running dense models on constrained hardware

A critical nuance: MoE does not reduce memory requirements. All 25.2B parameters of the 26B MoE must be loaded into memory even though only 3.8B are active per token. The inactive experts are idle but still resident. MoE saves compute, not memory. This is why quantisation and MoE are complementary techniques, and why running the Q4-quantised 31B dense on a Mac with 24GB is actually a better deal than running the full-precision 26B MoE.

The Benchmarks: Arena Rankings and Hard Numbers

Benchmarks are a minefield of cherry-picked numbers and suspiciously round percentages. So let us look at two sources: the Arena AI human-preference leaderboard and the automated benchmark suite from Google’s own model card.

Arena AI: Human Preference Rankings

As of 31 March 2026, the Arena AI text leaderboard has 337 models ranked from 5.7 million human votes. Here is where Gemma 4 lands in the overall table:

Model	Organisation	Licence	Arena Score
Claude Opus 4.6 Thinking	Anthropic	Proprietary	1,504 +/- 6
Claude Opus 4.6	Anthropic	Proprietary	1,499 +/- 5
Gemini 3.1 Pro	Google	Proprietary	1,494 +/- 5
…
Claude Sonnet 4.5 Thinking	Anthropic	Proprietary	1,452 +/- 3
Gemma 4 31B	Google	Apache 2.0	1,452 +/- 9
Qwen 3.5 397B A17B	Alibaba	Apache 2.0	1,449 +/- 6
Gemini 2.5 Pro	Google	Proprietary	1,448 +/- 3
Gemma 4 26B A4B	Google	Apache 2.0	1,441 +/- 9

Read that again. Gemma 4 31B scores 1,452, matching Claude Sonnet 4.5 Thinking and outranking Gemini 2.5 Pro and Qwen 3.5 397B. Among open-source models, it is ranked #3 in the world. This 31-billion-parameter model is competing with, and beating, models that are far larger. Google claims it “outperforms models up to 20 times larger,” and the Arena data backs that up.

Automated Benchmarks: The Full Picture

Here is a compact benchmark comparison from Google’s official model card:

Benchmark	Gemma 4 31B	Gemma 4 26B MoE	Gemma 3 27B
MMLU Pro	85.2%	82.6%	67.6%
AIME 2026	89.2%	88.3%	20.8%
LiveCodeBench v6	80.0%	77.1%	29.1%
GPQA Diamond	84.3%	82.3%	42.4%
Codeforces ELO	2,150	1,718	110
MMMU Pro	76.9%	73.8%	49.7%
MMMLU	88.4%	86.3%	70.7%

The AIME 2026 jump is staggering: from 20.8% to 89.2%. The Codeforces ELO went from 110 to 2,150. This is not a small step over Gemma 3, it is a generational leap.

Running Gemma 4 31B on a Mac: The Practical Guide

This is where it gets exciting for anyone with an Apple Silicon Mac. The unified memory architecture on M-series chips is a genuine superpower for local LLM inference, because the GPU and CPU share the same RAM pool. No separate VRAM cliff. If you have 24GB, 36GB, or more of unified memory, you are in business.

Memory Requirements

Precision	Approx. Size	Minimum Memory	Mac Recommendation
BF16	~58 GB	64 GB+	M2/M3/M4 Max 64GB+
FP8	~30 GB	36 GB+	M3/M4 Pro 36GB
Q4_K_M	~20 GB	24 GB+	M2/M3/M4 Pro 24GB
Q3	~15 GB	18 GB+	Smaller Macs

The sweet spot for most Mac users is Q4_K_M quantisation at about 20GB. This is the default distribution on Ollama, and it fits comfortably on a 24GB Mac with some headroom left for the operating system.

Step 1: Install Ollama

curl -fsSL https://ollama.com/install.sh | sh

Or download the macOS app directly from ollama.com.

Step 2: Pull and Run the Model

ollama run gemma4:31b

That is it. Two commands total. The download is around 20GB, and then you are chatting with a model that matches Claude Sonnet 4.5 on the Arena leaderboard.

Expected Performance on Apple Silicon

Mac Configuration	Quantisation	Approx. Speed	Notes
M4 Max 128GB	Q4_K_M	40-50 tok/s	Very fast local inference
M3/M4 Pro 36GB	Q4_K_M	20-35 tok/s	Comfortable for extended use
M2/M3 Pro 24GB	Q4_K_M	15-25 tok/s	Usable, context size matters
M1/M2 16GB	Q3	8-15 tok/s	Tight, consider 26B MoE or E4B

For reference, human reading speed is roughly 4-5 tokens per second. Even the slower configurations are still readable in real time.

MLX: The Apple Silicon Optimiser

If you want to squeeze more performance out of your Mac, look into MLX, Apple’s machine learning framework optimised specifically for Apple Silicon. Community support for Gemma 4 landed almost immediately, and MLX-optimised models can outperform GGUF-based inference on the same hardware.

pip install mlx-lm
mlx_lm.generate --model Phipper/gemma-4-31b-it-mlx-4bit --prompt "Hello, world"

The trade-off: MLX requires more manual setup than Ollama. For most users, Ollama is the right starting point. For performance enthusiasts, MLX is where things get fun.

The Complete Gemma 4 Family: Four Models, Four Use Cases

The 31B dense flagship is the headline act, but Google shipped three other models in the same family, and understanding the full lineup matters because the right model for you depends on what you have in your pocket, on your desk, or in your rack. Here is the entire family at a glance:

Model	Architecture	Effective Params	Context	Modalities	Q4 Memory
E2B	Dense (edge)	2.3B	128K	Text, Image, Audio	~3.2 GB
E4B	Dense (edge)	4.5B	128K	Text, Image, Audio	~5 GB
26B A4B	MoE (128 experts)	3.8B active	256K	Text, Image	~15.6 GB
31B	Dense	30.7B	256K	Text, Image	~17.4 GB

Two things jump out immediately. First, the smaller models are the ones with audio support, not the flagship. The E2B and E4B each carry a dedicated ~300M-parameter audio encoder that the larger models lack. Second, the edge models use a technique called Per-Layer Embeddings (PLE), which gives each decoder layer its own small embedding table for every token. These tables are large but only used for lookups, which is why the “effective” parameter count is much smaller than the total on disk.

Gemma 4 E2B: The Phone Model

E2B has 5.1 billion total parameters but only 2.3 billion effective, and it fits in roughly 3.2 GB at Q4 quantisation. This is small enough to run on a three-year-old smartphone. Through the Google AI Edge Gallery app (available on both iOS and Android), you can download E2B at about 2.5 GB on disk and start chatting with it entirely offline.

The performance claim that shocked the community: E2B beats Gemma 3 27B on most benchmarks despite being roughly 12x smaller in effective parameters. One early tester running it on a basic i7 laptop with 32 GB RAM reported it was “not only faster, it gives significantly better answers” than Qwen 3.5 4B for finance analysis. On a phone, users are seeing roughly 30 tokens per second, which is genuinely conversational speed.

For a Mac with only 8 GB of unified memory, E2B at Q4 is the safe bet. It leaves plenty of headroom for macOS and whatever else you are running. Install it with:

ollama run gemma4:e2b

Gemma 4 E4B: The Best Small Model You Can Run Anywhere

E4B is the sweet spot for anyone who wants something meaningfully smarter than E2B without jumping to the heavyweight models. At 8 billion total parameters (4.5B effective) and ~5 GB at Q4, it fits comfortably on any Mac with 16 GB of memory and leaves room for a browser, an IDE, and Slack running simultaneously.

E4B is the model David Ondrej demonstrated running on his iPhone 16 Pro Max in the video, and it was clearly usable at conversational speeds. The Edge Gallery app lists it at 3.6 GB on disk. On a phone with a modern chip, expect 20-30 tokens per second. On a Mac with 16 GB, expect 40-60+ tokens per second since the model is small enough to stay entirely in the GPU memory partition.

Crucially, E4B supports native audio input alongside text, image, and video. That means on-device speech recognition, spoken language understanding, and audio analysis, all without sending a byte off your machine. The 31B flagship cannot do any of this.

ollama run gemma4:e4b

Gemma 4 26B A4B: The Speed Demon

The 26B MoE is the model for people who want high-end quality at dramatically lower latency. Despite having 25.2 billion total parameters, only 3.8 billion are active per token, which means it runs at roughly the speed of a 4B dense model whilst retaining the knowledge capacity of a 25B model.

Real-world benchmarks from Kartikey Chauhan’s testing on a 12 GB VRAM Nvidia card show 44.2 tokens per second for text at 128K context and 42.1 tok/s for vision at 64K context. Those are server-grade numbers from consumer hardware.

On a Mac with 16 GB of unified memory, the 26B A4B at Q4 (~15.6 GB) is technically possible but tight. You will be at the limit of available memory, and macOS itself needs headroom. A 24 GB Mac runs it comfortably. For 16 GB Macs, be conservative with context length and expect some performance degradation from memory pressure.

ollama run gemma4:26b

Quality vs Size: What You Actually Lose at Each Step Down

The perennial question with model families is: how much quality do you sacrifice for each size reduction? With Gemma 4, Google published enough benchmark data to answer this precisely. Here is the full family compared side by side:

Benchmark	31B Dense	26B MoE	E4B	E2B	Gemma 3 27B
MMLU Pro	85.2%	82.6%	69.4%	60.0%	67.6%
AIME 2026 (Maths)	89.2%	88.3%	42.5%	37.5%	20.8%
LiveCodeBench v6	80.0%	77.1%	52.0%	44.0%	29.1%
GPQA Diamond	84.3%	82.3%	58.6%	43.4%	42.4%
MMMU Pro (Vision)	76.9%	73.8%	52.6%	44.2%	49.7%
MMMLU (Multilingual)	88.4%	86.3%	76.6%	67.4%	70.7%
Tau2 Agentic (avg over 3)	76.9%	68.2%	42.2%	24.5%	16.2%
Codeforces ELO	2,150	1,718	940	633	110

The pattern is clear: the 31B-to-26B step is almost free. You lose roughly 2-3 percentage points on most benchmarks but gain dramatically faster inference. This is the best trade-off in the entire lineup. The 26B MoE at 88.3% on AIME is essentially indistinguishable from the 31B’s 89.2% for any practical purpose.

The 26B-to-E4B step is where the cliff hits. You go from 88.3% to 42.5% on AIME, from 77.1% to 52.0% on LiveCodeBench, and from 85.5% to 57.5% on agentic tasks. This is where “frontier local model” becomes “capable assistant.” E4B is excellent for its size, but it is not in the same league as the two larger models for maths, competitive coding, or complex tool use.

The E4B-to-E2B step is gentler than expected. E2B typically loses 5-15 percentage points versus E4B, which is surprisingly modest given the 2x parameter difference. For basic Q&A, translation, summarisation, and conversational use, E2B is genuinely useful. It even beats Gemma 3 27B on multilingual tasks (67.4% vs 70.7% is close, but E2B’s AIME score of 37.5% vs Gemma 3’s 20.8% is a clear win).

Perhaps the most striking trend in the table: E2B scores 24.5% on Tau2 agentic tasks versus Gemma 3 27B’s 16.2%. A model you can run on a phone outperforms last year’s full-size model at tool use by a clear margin. Meanwhile, the 31B’s 76.9% average across all three Tau2 domains is nearly 5x what Gemma 3 managed. That is not an incremental improvement; it is proof that architectural progress matters more than raw scale.

Running Every Gemma 4 Model: A Hardware Decision Tree

Here is the practical guide to matching your hardware to the right model. Start from whatever you own and work your way to the best model it can handle:

Your Hardware	Best Gemma 4 Model	Quantisation	Expected Speed	Quality Tier
iPhone / Android (3+ years old)	E2B	INT4	~30 tok/s	Good assistant, basic coding
iPhone / Android (recent)	E4B	INT4	20-30 tok/s	Strong assistant, decent coding
Mac M1/M2 8GB	E2B or E4B	Q4	50-80 tok/s	Good assistant with audio
Mac M1/M2/M3 16GB	E4B (safe) or 26B A4B (tight)	Q4	40-60 / 15-25 tok/s	Strong / Near-frontier
Mac M2/M3/M4 Pro 24GB	26B A4B or 31B	Q4	25-40 / 15-25 tok/s	Near-frontier / Frontier
Mac M3/M4 Pro 36GB	31B	Q4 or Q8	20-35 tok/s	Frontier
Mac M3/M4 Max 64GB+	31B	BF16	40-50 tok/s	Frontier, full precision
Nvidia GPU 12GB VRAM	26B A4B	Q5	~44 tok/s	Near-frontier

The 12 GB Nvidia GPU result deserves special mention. Kartikey Chauhan’s detailed benchmarking of the 26B A4B on a 12 GB card using llama.cpp showed 44.2 tokens per second for text and 42.1 tok/s for vision, both at 128K context. He reported that the model is “an excellent default” for daily interactive use, with stable generation and no constant OOM babysitting once the right memory profile is set. The key was using fit-based GPU placement rather than forcing everything into VRAM.

For the edge models on phones, the Google AI Edge Gallery app is genuinely the easiest path. Download it, pick E2B or E4B, wait for the 2.5-3.6 GB download, and start chatting. Everything runs offline, nothing leaves your device, and the models support function calling for agentic tasks directly on the phone.

The 16 GB Mac Dilemma

The most common question in the community: “I have a MacBook with 16 GB, can I run the good stuff?” The honest answer is nuanced:

• E4B at Q4 (~5 GB): Runs beautifully. Fast, responsive, with plenty of headroom. This is the comfortable choice.
• 26B A4B at Q4 (~15.6 GB): Technically fits but leaves almost no room for macOS and apps. Expect memory pressure, swap usage, and slower generation as context grows. Usable for short conversations; painful for long ones.
• 31B at Q4 (~17.4 GB): Does not fit. You will hit swap immediately, and inference will crawl.

If you have a 16 GB Mac and want the best possible quality, the 26B A4B is your ceiling, but keep context short and close other apps. If you want a smooth, reliable experience, E4B is the pragmatic winner. It scores 52% on LiveCodeBench (enough for practical coding help), 58.6% on GPQA Diamond (solid science reasoning), and it can process audio natively, which neither of the larger models can.

How Good Are These Models for Coding?

If you are a developer considering local models as a coding assistant, the benchmark numbers matter less than a straight answer: can this thing actually help me write code? Here is the honest breakdown for each model, using LiveCodeBench v6 (real coding tasks, not just function completion) and Codeforces ELO (competitive problem solving) as the primary yardsticks:

Model	LiveCodeBench v6	Codeforces ELO	Comparable To	Practical Coding Level
E2B	44.0%	633	GPT-3.5-class	Handles boilerplate, simple functions, basic refactors. Struggles with multi-file logic or complex algorithms.
E4B	52.0%	940	GPT-4o-mini / Claude 3.5 Haiku	Writes working functions, understands context, handles standard patterns. The level that powers most “free tier” coding assistants.
26B A4B	77.1%	1,718	GPT-4o / Claude 3.5 Sonnet	Strong coder. Handles multi-step problems, debugging, architectural reasoning, and non-trivial algorithms reliably.
31B	80.0%	2,150	Claude Sonnet 4.5	Frontier-class. Solves most competitive programming problems and writes production-quality code with real architectural awareness.

The Codeforces 1,718 ELO for the 26B MoE puts it at roughly “Candidate Master” level, meaning it can solve the majority of interview-style programming problems and a solid chunk of competitive challenges. The 31B at 2,150 ELO is in “Master” territory. For context, Gemma 3 27B scored 110 ELO on the same benchmark. That is not a typo.

The practical takeaway: if you have the memory for the 26B A4B or 31B, you have a genuinely capable local coding assistant that rivals the paid API models most developers use today. If you are limited to E4B, you still get a useful companion for everyday development, roughly on par with the models that power free-tier tools like GitHub Copilot’s lighter backend. E2B is better suited for quick scripting help, code explanation, and boilerplate generation than for serious algorithmic work.

A Suggested Workflow for Constrained Hardware

If your Mac cannot comfortably run the 26B or 31B, a practical approach is to run E4B as your always-on local model for inline help, autocomplete, and quick questions, then fall back to a cloud API (Claude, GPT-4o, or Gemma 4 31B via Google AI Studio, which offers a free tier) for the 20% of problems where E4B is not enough. You get speed and privacy for the easy stuff, and quality for the hard stuff.

CPU-Only Servers: Running Gemma 4 Without a GPU

Not everyone runs inference on a laptop or a gaming PC. If you have access to a rack server, a cloud VM, or any x86 machine with a lot of RAM but no GPU, Gemma 4 still works. The entire family runs on CPU-only hardware via llama.cpp, Ollama, or vLLM.

The key constraint on CPU-only inference is memory bandwidth, not compute. LLM token generation is fundamentally a memory-bound operation: the model reads weights from RAM for every token. A typical DDR4 server delivers 40-80 GB/s of memory bandwidth, versus 200-400 GB/s on Apple Silicon or 900+ GB/s on an Nvidia A100. Those extra CPU cores help with prompt ingestion (prefill) but barely move the needle on generation speed.

Here is what to expect on a typical high-core-count x86 server with DDR4 (e.g., a dual-socket Xeon or EPYC with 256-384 GB RAM):

Model	Precision	RAM Used	Est. Generation Speed	Best Use Case
E2B	BF16	~10 GB	15-30 tok/s	High-throughput batch processing, multi-worker serving
E4B	BF16	~16 GB	10-20 tok/s	Quality-per-watt sweet spot for CPU serving
26B A4B	BF16	~50 GB	8-15 tok/s	Near-frontier quality, MoE helps since less data moves per token
31B	BF16	~58 GB	3-8 tok/s	Maximum quality when latency is not critical

With 384 GB of RAM, you can run the 31B at full BF16 precision with no quantisation loss at all. Most consumer setups cannot do this. The trade-off is generation speed: expect 3-8 tokens per second for the 31B on DDR4, which is below human reading speed (~4-5 tok/s) but still usable for batch jobs, API backends, or any workflow where you do not need instant responses.

The 26B MoE is the star on CPU-only servers. Because only 3.8B parameters are active per token, it moves far less data through the memory bus than the 31B dense model, which means the memory-bandwidth bottleneck hurts less. Expect 8-15 tok/s at full precision, which is genuinely conversational speed, with quality only 2-3% behind the flagship.

For serving multiple concurrent users, consider running several E4B instances across those 56 cores rather than one large model. Each instance uses ~16 GB at BF16, so you could run 10+ parallel workers within 384 GB of RAM, giving you high aggregate throughput for an internal team.

Multimodal Capabilities: What It Can and Cannot See

Gemma 4 31B is multimodal for vision, accepting both text and images as input with text output. It includes a ~550M-parameter vision encoder and supports variable aspect ratios and resolutions.

• Object detection and description – identify and describe objects in images
• Document and PDF parsing – extract structure and text
• OCR – including multilingual OCR
• Chart comprehension – read graphs and visual data
• Screen and UI understanding – parse app screenshots and interfaces
• Video understanding – analyse sequences of frames

On MMMU Pro, Gemma 4 31B scores 76.9%, up from Gemma 3’s 49.7%. That is a serious jump in multimodal quality.

What it cannot do: the 31B model does not support audio input. Audio is only available on E2B and E4B. So if you need speech recognition or spoken language understanding, the small models are actually more capable in that modality than the flagship.

140+ Language Support

Gemma 4 is trained on over 140 languages, with out-of-the-box support for 35+ languages. Community testing suggests it is especially strong on multilingual tasks, and the official MMMLU score of 88.4% backs that up.

“Natively trained on over 140 languages, Gemma 4 helps developers build inclusive, high-performance applications for a global audience.” – Google, Gemma 4 Launch Blog

This multilingual strength is one of Gemma 4’s real differentiators. If you build products for non-English audiences, this is not a side feature, it is the feature.

Choosing the Right Model: A Practical Decision Guide

With four models in the family, the question is no longer “should I run Gemma 4?” but “which Gemma 4?” Here is the decision matrix:

• You have 24 GB+ and want the absolute best quality: Run the 31B dense. It is the quality ceiling of the family.
• You have 24 GB+ but care about speed: Run the 26B A4B MoE. You lose 2-3% on benchmarks but gain roughly 2-4x faster inference. For most real tasks, you will not notice the quality difference.
• You have a 16 GB Mac: The E4B is your best realistic option. The 26B A4B technically fits at Q4 but will struggle with memory pressure. E4B leaves comfortable headroom and still scores above Gemma 3 27B on key benchmarks.
• You have an 8 GB Mac or a phone: Run E2B. At ~3.2 GB it fits anywhere, and it still beats Gemma 3 27B on maths and coding benchmarks despite being 12x smaller.
• You need audio processing: Only E2B and E4B support native audio input. The 31B and 26B cannot hear anything.
• You want to run AI entirely offline on your phone: Install the Google AI Edge Gallery app and pick E2B (2.5 GB) or E4B (3.6 GB). Everything runs locally, no data leaves your device.
• You need the longest possible context: Only the 31B and 26B support 256K tokens. The edge models cap at 128K.
• You want the absolute fastest time-to-first-token: E2B is the speed king, though E4B is close behind.

What to Check Right Now

• Check your Mac’s unified memory (Apple menu, About This Mac). Match it to the hardware decision tree above to find your optimal model.
• Install Ollama and try the model that fits your hardware:
  • ollama run gemma4:e2b – any Mac, any phone (3.2 GB)
  • ollama run gemma4:e4b – 8 GB+ Macs (5 GB)
  • ollama run gemma4:26b – 16 GB+ Macs, tight fit (15.6 GB)
  • ollama run gemma4:31b – 24 GB+ Macs (17.4 GB)
• Try the Edge Gallery on your phone. Download the Google AI Edge Gallery (iOS and Android), grab E2B or E4B, and chat completely offline.
• Compare against your paid model. Try your real prompts, not toy benchmarks. The 31B matches Claude Sonnet 4.5 on Arena; the E2B beats Gemma 3 27B on maths. Test them yourself.
• Test the 26B MoE if you have the RAM. It is the best speed-to-quality ratio in the family: 44 tok/s on a 12 GB Nvidia card, and only 2-3% behind the 31B on benchmarks.
• Watch for better quantisations and QAT releases. Unsloth, MLX Community, and other groups are actively improving the quantised variants. Quality improvements are still landing.
• Take the Apache 2.0 licence seriously. Commercial use, modification, redistribution, and fine-tuning are all on the table for every model in the family.

Video Attribution

This article was inspired by David Ondrej’s video covering the Gemma 4 release. The analysis, benchmarks, architecture deep-dive, and Mac deployment guide are original research drawing from Google DeepMind’s official documentation, the Arena AI leaderboard, community testing, and the Hugging Face model card.

nJoy 😉