On multi-agent systems and why they keep breaking

May 2025 · Back to blog

I've loved operating systems since I first understood what they were. The kernel managing processes, handling interrupts, enforcing memory boundaries, arbitrating access to shared resources. There's something deeply satisfying about that design: a small, trusted layer that maintains order so that everything above it can run without worrying about the details.

When I started thinking about what was wrong with multi-agent AI systems, I kept coming back to the same realization. These systems have no kernel. The LLM is being asked to manage its own resources, control its own execution, and validate its own outputs, all at the same time it's trying to solve the actual problem. That doesn't work in computer systems and it doesn't work here either.

That's why I chose this as my thesis topic.

I spent the last year of my degree trying to understand why multi-agent AI systems are so unreliable in production. Not theoretically unreliable. Practically, consistently, in the same three ways.

The first problem is cost. When you use a large frontier model for every step of a pipeline regardless of task complexity, your compute costs are unpredictable and usually indefensible. Simple tasks like extracting a field from a document do not need GPT-4. But most frameworks don't have a principled way to route tasks to cheaper models. They just use whatever you configured.

The second problem is fragility. Multi-step pipelines are only as reliable as their weakest link. If one agent produces a slightly malformed JSON output, every downstream agent either crashes or silently works with bad data. Most frameworks have no contract enforcement between agents. There's no boundary that says "your output must match this schema before the next step sees it."

The third problem is non-termination. Reasoning loops like ReAct have no formal stopping criterion. The model decides when it's done. And because the model is simultaneously trying to conserve resources and maximize accuracy, it tends toward one of two failure modes: stopping too early because it's cheap, or looping forever because it's trying to be thorough. These objectives are actually in conflict. You can't optimize both with the same model in the same loop.

What struck me about all three of these is that they're systems problems. Not prompting problems. The fix is not a better prompt, it's a different architecture.

The System-3/0 Blueprint

My thesis proposes treating a multi-agent system the way you'd treat an operating system. Separate orchestration from execution at the architectural level.

System-3 is the meta-controller, analogous to a kernel scheduler. It decides what runs next, dynamically builds execution graphs, and enforces global token budgets. The model does not decide when to stop. System-3 decides, based on a value-of-computation calculation.

System-0 is the protection ring. Every agent output passes through it before reaching the next step. It uses constrained decoding and schema validation to project outputs onto a valid structure. It can't be bypassed. In my ablation study, removing System-0 brought cascade failure rates from near-zero to around 35%. That's the clearest result in the whole thesis.

System-1 agents are fast specialists. They have direct tool access: file operations, code execution, web search. They're small language models, cheap to run, and focused on specific tasks.

System-2 agents are deliberate reasoners. They can think through complex problems, but they can't touch tools directly. They have to call tools through System-1. That sounds like a constraint, and it is, but it means every tool interaction is auditable. System-2 can't make a system call without going through the protection ring.

Why this matters

The whole system runs locally. No GPU, no API costs. 24GB of RAM, Ollama, phi3:mini for System-1 and llama3.1:8b for System-2. That was a deliberate design choice: if the architecture only works with frontier models, it doesn't actually solve the cost problem, it just papers over it.

3-4x token savings compared to monolithic approaches in the ablation. Near-zero cascade failures with System-0 active. These numbers hold on consumer hardware without any external API dependency.

There's a broader point here about how we build AI systems. The current tendency is to treat the LLM as the controller of the whole system, including its own orchestration and stopping conditions. My thesis argues that's theoretically unsound. A probabilistic model exploring a state space can't objectively evaluate its own halting condition. You need an external deterministic layer for that. That's what System-3 is.

The OS analogy runs deeper than a metaphor. System-3's Value-of-Computation halting rule is essentially a scheduler preempting runaway processes. System-0 is a protection ring. The dynamic Agent Registry is a process table with typed capability entries. The depth-k composability theorem is a formal statement of why user-space processes can't escalate privileges without going through the kernel. Old ideas applied to a new context, which is usually a good sign.

That same architecture is what Jovalent runs on. When I finished the thesis I realized the orchestration layer I'd built was the thing I needed for the product. Those two things turned out to be the same project.

I'll write more about specific implementation decisions after submission.