Conversion is building an AI marketing automation platform for enterprise teams. Our agent does more than answer questions: it works across customer data, audiences, campaigns, emails, forms, and workflows, using a large catalog of tools to inspect data and take action.
That makes model choice central to our product. Naturally, we constantly ask ourselves, which models are best suited to this environment, and how can we know?
Public benchmarks are a useful starting point but provide little information on how a model will perform with our tools, data, and customer workloads. Marketing tasks are often ambiguous, span several systems, and unfold over long conversations, so a model’s value to us depends on how effectively it can operate inside this environment.
That environment includes a growing set of tools built specifically for our agent. Models differ in how well they discover, select, and combine those tools, and changes to the tool catalog can change what a model is capable of completing. We therefore cannot evaluate the model separately from the system in which it acts.
We are particularly interested in the frontier between quality and latency: how much faster can the agent become before task success begins to fall? A faster model may produce a slightly weaker first attempt while still creating a better overall product experience.
The obvious way to answer these questions was to bring more models into Conversion and test them there.
Our first attempt
This was around the time GPT-5.6 released, and we quickly began thinking about adding it as a model in our harness. However, providers differ just enough in how they handle thinking, compaction, tool search, images, and conversation history that fitting GPT-5.6 into the same agent required changes across the harness. Adding the next model would just require much of the same work.
The cost of an experiment shapes what you learn
If adding a model is a large engineering project, though, we will naturally test only the models we already expect to succeed. Integration cost narrows the set of hypotheses we can explore and encourages reasonable assumptions to harden into architecture before the evidence exists.
That felt especially dangerous in a field moving this quickly. Models are changing across capability, latency, cost, context length, and deployment options, and we do not expect one provider to lead along every dimension indefinitely. We wanted it to be inexpensive to form a hypothesis, evaluate a model, and discover that we were wrong.
This became the motivation for model agnosticism: making it as cheap as possible to test new hypotheses and change our minds.
What being model-agnostic actually requires
At the level of a single request, model providers look similar: messages go in, then text, reasoning, or tool calls come out. The differences become harder to hide once an agent needs to preserve state and capabilities across turns.
Making models easy to add was not enough. We also wanted model choice to happen per turn rather than per conversation. A chat might begin with Opus and continue with GPT-5.6 without starting over or maintaining separate histories for each provider.
This became our test for whether the harness was genuinely portable. The next model needed to inherit the visible conversation, tool calls, results, approvals, and compacted context produced by the previous one.
Per-turn portability also expands what we can evaluate. Because the conversation has a shared history, we can replay the same accumulated context to different models instead of testing them only on clean, isolated prompts. That lets us study how models perform after a long sequence of decisions, tool calls, and results—the conditions our customers actually create.
Compaction was where making that history portable became difficult.
Owning compaction
Each provider represents compaction differently. (Note: we use Vercel AI SDK) Anthropic’s compaction record contained a textual summary with provider metadata attached:
{
"type": "text",
"text": "Summary of the conversation...",
"providerOptions": {
"anthropic": {
"type": "compaction"
}
}
}
OpenAI returned an opaque record intended to be passed back through its own Responses API:
{
"type": "custom",
"kind": "openai.compaction",
"providerOptions": {
"openai": {
"itemId": "cmp_123",
"encryptedContent": "..."
}
}
}
Anthropic’s summary text could be recovered and given to another model. OpenAI’s encrypted content could not be interpreted by us - or by an Anthropic model.
Our first OpenAI integration taught the harness to identify, store, and replay both formats. It worked, but every new format would require another compatibility path. We instead created a shared, provider-neutral compaction checkpoint:
{
"summary": "Summary of the earlier conversation...",
"retainedUserTurns": 3
}
This gave us a checkpoint that was usable by any model. However, taking on the task of compaction meant that we had to come up with a provider-agnostic solution to the task of when to trigger compaction.
With native compaction, the provider knows exactly how many tokens the rendered request contains, including images, tool definitions, and internal formatting. Once compaction moved into our harness, we had to calculate that ourselves. Calling a provider’s token-counting API before every model step would add latency, while maintaining local tokenizers and image-accounting rules would add model-specific code.
We chose a hybrid approach: each completed model step reports its token usage, so we use the latest reported count as an anchor and estimate only what has been added since:
const estimatedContext =
lastReportedUsage + estimateTokens(newContent);
if (estimatedContext >= contextWindow - safetyReserve) {
compact();
}
We pair that estimate with a substantial safety reserve. We may compact earlier than necessary and sacrifice some usable context, but we avoid another network request and reduce the risk of exceeding the context limit.
In return, we control what the summary preserves. Our prompt keeps decisions, concrete identifiers, user constraints, unfinished tasks, and recent errors while leaving the latest turns intact. Anthropic also supports custom compaction instructions, but OpenAI’s documented implementation returns opaque machine state without an equivalent summary prompt. Our version gives every model the same readable history and retention policy.
The tradeoff was real: we gave up exact server-side accounting and automatic triggering, then rebuilt enough token tracking to operate safely without them. What we gained was a compaction record that we could inspect, customize, and pass to any model.
Keeping thinking provider-specific
Thinking required a different compromise. Reasoning is part of what we want to evaluate, so reducing every provider to the same representation would remove a meaningful model capability.
Anthropic thinking blocks include a cryptographic signature that must be returned unchanged. OpenAI can return encrypted reasoning with its own replay rules. We preserve those native formats and replay them only to the exact model that produced them.
When a conversation switches from Opus to GPT-5.6, GPT receives the shared messages, tool calls, results, and summaries, but not Anthropic’s signed thinking blocks. If a model rejects historical reasoning that appeared compatible, the harness retries without it.
This became an important distinction in how we thought about model agnosticism. It did not mean forcing every model into an identical shape. It meant creating a common ground that every model could use, while keeping the differences that genuinely affected capability narrow and explicit.
Compaction became portable because portability was more valuable than the provider-specific optimization. Thinking remained specialized because normalizing it would erase part of what we wanted to measure.
Giving every model the same tools
Open models exposed another gap. Many capabilities our agent relied on—especially tool search and web search—were utilities supplied by the provider rather than the model itself. Without alternatives in our harness, an otherwise capable open model would enter the product with a weaker agent around it.
Tool search was especially important because Conversion has a large catalog covering contacts, campaigns, emails, forms, audiences, and workflows. Its tools follow recurring product categories and operations such as list, create, update, and delete.
A generic provider sees names and descriptions. Because we own the catalog, our search can be organized around its naming patterns, categories, permissions, and relationships. We have not yet proven that this will outperform every native alternative, but importantly, we can now test the question—and models without native tool search can participate.
We exposed web search through the harness for the same reason. Every model now receives the same core capabilities, while a small model profile describes the differences that remain, such as context limits, image support, reasoning controls, and caching.
We ended up owning more code, but adding a model requires changing far less of it.
The harness as a research platform
Adding a model is now closer to configuration than an architecture project. That gives us a common environment in which to ask practical questions: which tasks need a frontier model, where a faster model is sufficient, how latency trades off against quality, and what each model costs per successful task.
We are interested in the complete agent rather than an abstract measure of model intelligence. Customers experience the model, tools, history, permissions, and recovery behavior together. Our benchmarks therefore need to include end-to-end completion, latency, tool selection, recovery after failures, and context retained through compaction.
The platform is what makes benchmarks like those possible. When a new model appears, we can put it inside our agent and develop a view from evidence rather than release notes.
Building for models we have not met yet
We began with a provider-agnostic interface around model calls. What we needed was a provider-agnostic system around the whole agent.
Getting there required a shared history between models, explicit handling for state that could never be portable, and harness-owned alternatives to capabilities that had quietly become provider dependencies. It was more engineering work, but it reduced the number of concepts required to add a model and increased the number of experiments we could realistically run.
This is what makes AI engineering difficult and interesting. The work sits between models that change every few months and product expectations that must remain stable for years. The harness has to preserve useful differences without letting each new model reshape the entire system.
At Conversion, we now see that harness as both product infrastructure and a research platform. Provider-native capabilities helped us learn what the agent needed. Bringing more of those capabilities under our control is helping us prepare for models we have not met yet—and making it much cheaper to change our minds when they arrive.