Cutting AIOps agent cost by a third with code mode and Pydantic Monty

This is the first post about the Silex AIOps Platform, the system we build to let AI agents work level 1 and level 2 operations tickets. It starts with a short tour of the platform, then covers one change we made in October: moving every agent tool call into code mode, with the agent’s programs running in Pydantic’s Monty interpreter. We’ll go through the problems that pushed us there, the architecture, and the numbers from two trials.

The Silex AIOps Platform in brief

Most of the hours in IT operations go into the steps around a decision. Someone picks up the ticket, searches the logs and dashboards, works out what is wrong, writes up a plan, waits for approval, runs the job, checks that it worked, and writes it up. On the platform, an agent does that work and a person makes the one decision that matters, which is approving the change.

The agents take on infrastructure triage, disk and service fixes, server patching, server builds, database requests, and security remediation. They work through the tools a customer already runs: the ITSM and CMDB, monitoring and logs, source control, and Red Hat Ansible Automation Platform (AAP). An agent never logs in to a host. It reads widely, the way an SRE would, but it changes a host only by launching an approved AAP job template after a named approver signs off on the change in the ITSM.

flowchart TB
    subgraph DP["Data plane, in the customer's network"]
        A["Agent in a sandbox<br/>placeholder keys only"]
        EG["Enforcement gateway<br/>every tool call"]
        MG["Model gateway<br/>every model call"]
        C["Connectors<br/>hold the real credentials"]
        A --> EG
        A --> MG
        EG --> C
    end
    C --> S["ITSM and CMDB, AAP,<br/>monitoring and logs"]
    MG --> P["Model provider<br/>on the customer's route"]
    DP -->|"outbound only"| CP["Control plane<br/>console, workflow engine, enrollment"]

The pieces that matter for this post are the two gateways. Every tool call the agent makes goes through the enforcement gateway, which checks the session, the tools the workflow is granted, the tool definitions we pinned, launch policy, and the change gate, then records the call in a hash-chained audit. Every model call goes through the model gateway, which holds the provider key, enforces budgets, and records the tokens and cost of each request. The agent runs inside a sandbox with no real credentials. Because the rules live in the gateways and not in the agent harness, the same controls apply whether the agent runs in Claude Code, Codex, or an open-source harness.

The problem: tool output grows with the environment

An incident agent needs a lot of tools. For one triage workflow we grant the ITSM’s ticket and change tools, the CMDB, Grafana’s MCP server for metrics and logs, and AAP’s MCP server for inventories, hosts, job templates, and jobs. Counting the harness’s own tools, the model was offered 69 tool definitions on every call. Each definition takes tokens, and the model pays for them again on every turn.

The responses are the larger problem. Enterprise environments under infrastructure-as-code management hold thousands or tens of thousands of nodes, and the tools that describe them answer at that scale. A tool built for a person browsing a UI returns every field of every record. In our large test environment, one page of the job template catalog (100 of 2,008 templates) came to 494 KB, about 125,000 tokens. The agent usually needed one line of it.

With direct tool calls, every response goes into the model’s conversation, and the model reads the whole conversation again on each later turn. Our agents can make more than 400 tool calls on a single task, so a large response early in a run is paid for many times over. Cost grows with the peak request size, and runs get less reliable as the context fills with output the agent never needed.

Agent harnesses protect themselves from this, each in its own way. Claude Code saves any MCP response over 50,000 characters to a file and gives the model the first 2,000 characters. Codex CLI keeps about 12,000 tokens of a response, its head and tail. Gemini CLI writes long responses to a file and shows part of them, and Goose gives the model only the file’s path. These limits keep a conversation from blowing up, but they cut wherever they cut. In one early lab run, the agent listed eight job templates, 59.6 KB with every field the controller returns. Our gateway had added a note at the end of that response saying which templates launch policy allowed. Claude Code cut the response to its first 2,000 characters, the model never saw the note, and the run never launched a read-only template it needed.

So an agent working a large environment either pays to carry text it doesn’t need or loses the line it did need. We wanted the filtering to happen before anything reached the model, in the same way for every harness.

Code mode

The idea behind code mode is simple. Instead of calling tools one at a time and reading each response, the model writes a short program that calls the tools as functions, filters what they return, and prints only what it needs. Cloudflare described this as Code Mode in September 2025, and Anthropic wrote about code execution with MCP in November, reporting one task falling from 150,000 tokens to 2,000. Anthropic also named the catch: the code the model writes has to run somewhere, and that brings operational overhead and security questions.

For us the security question decided the design. The agent writes its own transformation code on every run. That code is untrusted by definition, and it is calling tools that can read a production CMDB and launch jobs in AAP. Whatever runs it must not give the program a way around the controls that apply to a direct call.

We looked at three places to run the agent’s programs:

1. In the agent’s own sandbox, with ordinary Python, an MCP client, and generated stubs for each tool. The sandbox is already where untrusted code runs, but every harness image would need the client and stubs, and the path from the sandbox’s shell to the gateway would have to be opened in harnesses that sandbox their own shell.
2. In the harness’s own code execution feature. That ties the platform to one harness, and the platform is built to work with several.
3. As a tool the gateway serves, running each program under Monty. Every harness gets code mode with no change to its image, since it is one more MCP tool. The cost is that model-written code runs inside our main control point, so the isolation has to hold up.

We chose the third. Monty is Pydantic’s interpreter for a subset of Python, written in Rust and MIT licensed. It isn’t a container or a VM, and nothing boots per program: it starts in under a millisecond from a warm pool. A Monty program can reach only what the host lends it. Files, environment variables, and the network don’t exist unless the host provides them. Monty can also pause a program at a call to a host function and hand that call to the host, which is the hook we needed to route every tool call back through the gateway.

The architecture

A workflow chooses code mode in its pack, the signed bundle of instructions, skills, and tool grants that defines it. In code mode the agent is offered two tools and no others:

• find_tool(query) searches the tools the session is granted and returns up to eight matches. Each comes with its exact name, the first sentence of its description, its arguments with types and allowed values, and an example call. Asked for an exact name, it returns the tool’s whole description.
• run_code(code) runs a Python program. Inside the program, run_tool(name, arguments) calls any granted tool by name.

Every other tool the pack grants stays granted, but it can only be called from a program. A direct call to one is refused and audited.

flowchart TB
    A["Agent in its harness<br/>sees only find_tool and run_code"]
    subgraph G["Enforcement gateway"]
        M["Monty worker process<br/>no credentials, files, or network"]
        K["Same checks as a direct call<br/>grant, pins, arguments, launch policy,<br/>change gate, audit"]
        M -->|"2) run_tool(name, arguments)"| K
    end
    S["Connectors to ITSM, CMDB,<br/>AAP, monitoring and logs"]
    A -->|"1) run_code(program)"| M
    K -->|"3) call"| S
    S -.->|"4) full response, to the program"| M
    M -.->|"5) printed lines only"| A

Each program call is an ordinary gateway call

When a program calls run_tool, Monty pauses and hands the call to the gateway, which handles it exactly as it would handle the same call made directly. It checks the session token, the grant, the pinned tool definition, the arguments, launch policy, and the change gate, and it applies the same protection against repeating a write when a call is retried. The full response goes back to the program, never straight to the model. Each call is audited as its own event and linked to the run_code call that made it, so a reviewer, or the judge that grades each run, sees every call a program made and every response it got.

A program’s call passes every check a direct call does, so code mode gives the agent no permission it didn’t already have.

Isolation

Each program runs in its own Monty worker process, started with an empty environment and two pipes, and the worker serves no other program. The gateway’s process, which holds the connectors’ credentials, never runs model-written code. The program gets no mounts and no OS access, so file, environment, and network access are refused, and sleeps return at once. We don’t use Monty’s ability to save and resume an interpreter’s state.

A program also runs within limits:

The runner fails closed. If it can’t start, run_code returns an error and nothing runs. A program that reaches a limit is stopped where it can’t catch the error, and the agent gets what it printed and how many calls it made. If the harness cancels the call, the program makes no more calls.

Notes and waits

Two details came from running real workflows. First, the gateway sometimes adds a note to a response, such as launch policy’s list of read-only templates. A program could filter that note out without meaning to, so the gateway keeps notes away from the program and puts each one at the top of run_code’s response instead, where no filtering can hide it.

Second, some workflows wait on people. A patching run might wait days for a change approval or a pull request merge. When a program’s call comes back with a wait, the gateway refuses the program’s later calls and tells the agent to end its turn, so the wait stays the session’s last call and the workflow engine can hold the run until the approval arrives.

What a program looks like

Here is the kind of program the triage agent writes. It looks up each host the service map names, one search per host, and pulls the job templates for one service out of a catalog of thousands:

hosts = ["shop-web-01", "shop-web-02", "shop-db-01"]
for name in hosts:
    page = run_tool("hosts_list", {"search": name, "page_size": 5})
    for h in page["results"]:
        if h["name"] == name:
            print(h["id"], h["name"], h["inventory"], h["enabled"])

page = run_tool("job_templates_list", {"search": "shop", "page_size": 100})
for t in page["results"]:
    print(t["id"], t["name"], t["description"][:80])

The responses behind those calls can run to hundreds of kilobytes. What reaches the model is a dozen lines.

Each program starts clean

A program starts with nothing from an earlier one. We considered keeping one Monty session for each agent session so helpers could carry over, and decided against it. A kept session would tie up a worker for the whole run, a worker that stopped would lose what it held, and a person reading one program in a transcript would need to read every program before it.

The cost showed up in the transcripts. Across our demo runs from October 18 to 27, 252 of 1,280 programs failed. Of those, 23 used a helper or a value an earlier program had defined, and 27 used json without importing it. Only four failed on a Python feature Monty lacks. Every program now starts with json and re imported, and run_code‘s description tells the agent that each program runs on its own. We also sent the Monty team the standard library features the agents’ monitoring code reached for most often: datetime.fromtimestamp() for the epoch timestamps that Prometheus and Loki return, ipaddress for subnet checks, and zoneinfo for change windows in a customer’s local time.

Why code mode is the only option

Our first build offered run_code beside every tool. In four runs on the small lab, the agent never chose it, because the responses it read were short enough to take whole. A trial where the agent never uses the feature tests nothing, and a customer environment with thousands of playbooks and several MCP servers would produce exactly the responses that make code mode worth using. So a pack in code mode offers only find_tool and run_code, and the agent’s context holds two tool definitions however many tools it is granted. With the harness’s own tools, that took the count offered on each call from 69 to 14.

How we tested it

We ran the same five incidents in two labs, once with direct tool calls and once in code mode. The incidents were planted faults: a stopped service, a full disk, a certificate missing the shop’s host names, a broken application setting, and a database refusing connections.

The small lab is our standard lab: 8 job templates and 10 hosts in one inventory. The large environment is the same lab with noise added to AAP for the agent to search past. We added 2,000 job templates, named the way a large catalog’s are (restart-payments-prod-east, collect-logs-orders-staging), and 1,000 hosts in three inventories with realistic variables. The agent’s account could read all of them, launch none of the new templates, and reach none of the new hosts.

Each run followed the same steps. A script reset the lab and emptied the logs, metrics, and job history. The fault was planted and checked. A person filed a ticket that named only a symptom, and the platform’s intake picked up the ticket and started the agent. Both arms used the same model (Claude Sonnet 5.5), the same pack instructions, and the same judge (Claude Opus 5.5), which scores each run’s findings and its use of the environment’s sources out of 5. A run also has record checks, such as whether every change it made was approved and whether its audit is complete.

Tokens and cost come from the model gateway’s records of every request, cross-checked against each run’s stored transcript, and they agreed to the token on every run. Cost is at the model gateway’s prices. “Tokens read” counts every token the model read across all its requests: input, cache writes, and cache reads. “Peak request” is the largest single request, roughly the size of the conversation at its end.

Results

Code mode cost less and read fewer tokens on all ten incident pairs. On the large environment, it read 20 to 43% fewer tokens per incident and cost 19 to 34% less. Its largest single request stayed under 85,000 tokens on both labs, while direct calls reached 144,000.

Direct calls failed one incident in each lab. Both failures scored 3 out of 5 for use of the environment’s sources, and both came from not reading the environment properly. On the small lab’s full disk, the agent guessed the service’s name instead of looking it up and named the other hosts from metric labels alone. On the large environment’s application setting, the agent searched the inventory for every host name containing a 0, which came back as 298,124 characters. Claude Code saved that to a file the agent never read, and the agent went on to run a job against five hosts it had never confirmed in the inventory. In code mode, the same incident looked up each host by name and passed.

The large environment also showed where direct calls leaned on the harness. In every direct run, two to six responses were too large for Claude Code to put in the conversation, and it saved them to files. The template listing was one of them every time. The agent could only probe those files with grep and head, since the sandbox has neither jq nor python3. Code mode never hit the limit, because its programs read those responses whole inside Monty and printed what mattered.

Every audit in both trials was complete. Every call a program made was linked to its run_code call and had its response recorded.

What code mode costs

Code mode takes more turns: 110 against 83 on the small lab and 129 against 111 on the large environment. Each turn carries much less, so the runs cost less, but they take longer.

Some programs fail. On the small lab 16 of 88 programs failed, and on the large environment 20 of 105, mostly because the agent guessed the shape of a response before printing a sample of it. Most tools publish no output schema, so the agent has to learn the shape the first time. Each failure costs a retry. Showing a response’s shape in find_tool is the obvious fix, and for now we are tracking the count instead.

Caveats

Claude Code’s size limit protected the direct runs. A harness without one would put the 494 KB template listing, about 125,000 tokens, into the conversation and read it again on every later turn, so direct calls would cost more than these numbers show. Code mode’s filtering happens in the gateway, before any harness sees a response.

Each incident ran once per arm. The direction held on all ten pairs, but each gap is a single sample.

Our first code mode trial showed a 63% cost saving, and we don’t quote it. Most of that gap came from a change to the triage skill’s instructions that helped both arms, so we reran the direct arm on the current instructions. The figures above compare both arms on the same instructions.

The judge’s cost, about $1.50 a run, isn’t counted for either arm.

Where code mode runs now

Both of the platform’s shipped workflow packs, incident triage and the patch cycle, run in code mode. The patch cycle calls every tool from programs, including the waits on approvals and pull requests. Monty is pinned to one release in the gateway (1.3.0 today) and sits behind a code runner interface, so another interpreter could replace it without changing the packs. For a platform where agents read environments of thousands of hosts through third-party MCP servers, having the agent write its own filter, and running that filter where it can’t touch a credential, is where most of our token savings have come from.