Everything an agent might need to know competes for the same context window: the instructions it should follow, the schemas of its tools, the outputs those tools return, and the data underneath the product. The instinct, especially early on, is to inline all of it. Big context windows make that feel free.
It isn’t free. Anything in the prompt is sent on every request, so a schema the model needs once a week still costs its tokens on every call. More importantly, it costs time. Most of our customers use the agent as a copilot while they work inside the product, and that experience depends on how quickly the first token arrives. Time to first token grows with prompt length, and our prompts are not small: the median request carries around 150,000 tokens.
Most of our context work over the past year comes down to one move, applied repeatedly: take something that was loaded on every request and load it only when a request actually needs it. The hard part is that removal has its own failure mode. Context earns its keep in ways you didn’t plan for. A tool’s schema doubles as product documentation, and deleting it makes the agent faster but worse at the product. So every removal below came with a replacement: some other path to the same knowledge, available at the moment it matters.
Two things about Conversion make this more pressing than usual. Our agent writes formats that exist in exactly one codebase. An email is a component tree with rich text serialized as editor state, a workflow is a graph, an audience filter is a recursive query tree. None of this appears in training data, so everything the model knows about these formats is something we told it. And underneath the product sit customer databases with millions of contacts, so ordinary questions can produce outputs no context window should hold.
Instructions became skills
The first thing we moved out was instructions. The agent’s knowledge of how to build each format lives in skills: markdown documents that explain how emails, workflows, forms, and audience filters actually work, with the rules and worked examples. Together they run to about 150,000 characters, roughly 37,000 tokens. None of it is in the prompt. The files are mounted read-only into the agent’s sandbox, and the agent reads the one it needs when the task calls for it. Building an email loads the email skill. Answering a question about campaign metrics loads nothing. Every tool that takes one of these formats names its skill in the description, so the agent knows what to read before it calls.
Schemas the agent never sees
The standard way to teach a model a tool’s input is to put the input schema in the tool definition. For most of our tools that’s correct, and we do it. For a few it’s not as straightforward. A statement is a recursive tree: logical nodes contain operands, operands contain conditions, conditions contain further statements. Workflows and emails are the same shape of problem. Serialized as JSON schema, the two email tools carried about 7,000 tokens each and the two form tools about 29,000 each. Together those four tools put roughly 70,000 tokens into every request, whether or not the request had anything to do with emails or forms.
What that costs in time is measurable. Across a month of production traffic on Claude Opus 5:
| Prompt size | Cold p50 | Cold p90 | Cached p50 | Cached p90 |
|---|---|---|---|---|
| <25k | 2.4s | 7.5s | 1.8s | 2.9s |
| 25–50k | 2.7s | 3.4s | 2.4s | 4.1s |
| 50–75k | 3.8s | 6.7s | 3.1s | 6.0s |
| 75–100k | 5.4s | 8.2s | 3.7s | 6.6s |
| 100–125k | 6.2s | 8.4s | 4.2s | 7.3s |
| 125–150k | 5.9s | 9.6s | 4.3s | 7.4s |
| 150–200k | 7.2s | 9.5s | 4.2s | 7.7s |
| 200–300k | 6.3s | 10.1s | 3.7s | 6.7s |
| 300k+ | 10.7s | 16.3s | 3.8s | 7.3s |
Below 200,000 tokens the relationship is close to linear, at roughly 31ms per 1,000 tokens of prompt. Caching helps, and it helps more the larger the prompt gets. Past roughly 125,000 tokens a cached request stops getting slower at all, because most of the prompt is a stable prefix that doesn’t need reading again and only the recent part is new. An uncached request has no such ceiling and keeps climbing.
Against that rate, 70,000 tokens of schema is about two seconds of added latency on every turn in the product.
We removed them and split the job in two. The tool now declares the field as a plain JSON object, and the skill explains how to construct one. The schema itself still exists, but as a validator inside the tool. Every submitted payload is checked against it, and a payload that fails comes back to the model as the exact list of what’s wrong, with a pointer to the skill. The model fixes the draft and calls again.
This drastically improved cost and speed metrics. Before, every request paid for the schema. Now a request pays nothing when the model gets the format right, which is most of the time, and pays for one retry when it doesn’t.
Tool outputs became files
Context pressure also comes from tool outputs. If a user asks for something ordinary, information about a set of contacts, say, the honest answer is thousands of rows drawn from a database of millions. Aggregate tools handle the questions we can anticipate, but they only go so far. Customer use cases often end up being too specific and too varied to ship a built-in tool for each one.
We tried some obvious approaches first, and realized quickly that they bring up issues of their own. Pagination, for one, turns one oversized result into many model round trips, and each round trip costs far more time than the fetch it wraps. Shrinking the outputs, for another, is hard to optimize without compromising on performance.
So large outputs skip the context window entirely. Any tool result over 20,000 characters is written to a file in the agent’s sandbox, and the model receives the path plus a 1,000 character preview. When the agent needs specifics, it greps the file and pays only for the lines that match.
The deeper benefit here was that data that lives in the context window can only be transformed by inference. If the agent needs a total, or wants to check that numbers across five hundred rows are consistent, it has to produce every intermediate step token by token, and arithmetic by inference is slow and unreliable. Data that lives in a file can be transformed by code. To perform transformations on tool outputs, it doesn’t first have to copy the data out of its own context to do that, because the data is already on disk. The agent got noticeably faster on large-output work, not because it read less, but because it stopped re-emitting data it already had.
One tool instead of five
Our approach to defining new tools almost always involves wrapping existing API handlers, which is the fastest way to get an agent doing real work. This, however, also meant that the tool surface inherited the shape of our backend. Editing a campaign was split across separate tools, one for the campaign, one for its members, one for its tokens, because that is how the endpoints are split. Each carried its own schema into every request, and the agent had to pick correctly among near-duplicates before it could do anything.
Those boundaries exist for reasons that have nothing to do with the agent, so we stopped exposing them. Families like
this collapsed into single tools. There is one edit_campaign, and the call says which parts of the campaign it
touches.
This helps in two ways - firstly, the token cost of the tool surface dropped, since one definition replaced several. Secondly, tool selection also gets better, which becomes increasingly more relevant as our product surface increases.
Tools that answer in aggregates
The same consolidation applies to data. Many of our metrics tools started out per-asset, which meant that average performance across thirty emails took thirty calls, with the agent doing the totaling itself. Nothing failed, so nobody reported it. Our observability pipeline surfaced how often this loop was actually happening, and we covered that discovery in our data flywheel post.
The fix was aggregate tools: this allowed agents to perform one call that accepts the set and returns the computed result. The repeated procedure moved out of inference and into code, where it runs the same way every time. A workflow the agent used to rediscover on each request, with room to improvise each time, became a codified path with one correct implementation. This allows the agent to be faster and more consistent.
What’s next
Tool count grows with product surface, and our product surface expands at an incredible rate. Every area we build brings its own tools, and consolidation only buys so much. With over a hundred first-party tools loaded on every chat request now, the standard remedy is tool search, where definitions stay out of context and the model looks tools up on demand. We already do this for tools customers connect through their own MCP servers, since an unbounded tool surface leaves no other option. Applying it to our own tools is more complex. Search assumes the model knows what to search for, and an agent that doesn’t know a capability exists never goes looking for it, so the failure is silent. The descriptions also carry more than the instructions for one call each. Read together, they are a map of what the product does, and the agent leans on that map for general questions that never end in a tool call. Hiding them behind a search index would make the agent faster and less aware of the product it works inside.
Growing the product surface without giving up cost or speed is an open research problem for us. It takes benchmarking that can catch a quality regression, continuous experimentation as models change underneath us, and harness engineering to make both routine. All three are moving quickly right now. If you’re interested in any of these problems, reach out. We’re hiring, come build with us.