More than a lockfile: the Agent Contract for LLM features

An LLM feature depends on five things: output schema, prompt, model, tools and eval set. If you only version the prompt, you don't know what you deployed.

·17 min read·Made with AI

If you only version the prompt of an LLM feature, you don’t know what you deployed.

Friday afternoon, someone tightens a sentence in the refund-agent’s prompt. Over the weekend, the provider swaps in a new model behind an alias. Or the MCP server that provides refund_order changes its tool description (we already had that case in the canary post). There’s no commit and no release for any of this in the service repo. And yet on Monday the agent issues refunds differently than it did on Friday.

Which version of the refund-agent was actually running on September 12? The usual answer is a Git SHA. But that only tells you which code ran. Which prompt from the prompt CMS went with it, which model was behind the alias and which tool description came from the third-party server: none of that is in there. Even when the prompt lives in the repo, its changes are hard to spot in the history. In a study of 243 GitHub repos, only a little over one in five commits that changed a prompt even mentioned it in the commit message (21.9%, Prompting in the Wild, MSR 2025).

On ai-systems-architecture.com, Michael Seel and I call the package that answers this question the Agent Contract. It’s our engineering term; there’s no standard for it. In research, the term means something else: for Ye & Tan, agent contracts are formal runtime contracts with resource budgets. Ours is a release artifact: what gets shipped, and what has it been tested against? Michael has worked out the term in detail. If you want the long version, here it is:

M
Der Agent Contract | Michael Seel
Ein KI-Feature ist kein Prompt. Es ist die Kette aus Schema, Prompt, Modell, Tools und Eval-Set, und jedes Glied hängt an jedem anderen. Prüfbar wird das nur als Ganzes.
michaelseel.com
→

The example is still the refund-agent from the previous posts.

Refund-Agent
Multi-agent support system with evals, observability and contracts

Five things, one feature

What are we deploying when we deploy the refund-agent? The code is in the repo, sure. But whether a customer saying “I want my money back” ends up on the right route, and whether the right thing happens there, is decided in five places, and not all of them are in the repo.

Link What’s in it refund-agent, Contract v1
Output schema the shape the code relies on routing decision as an enum (refund, lookup, faq, account, clarify), enforced via the route tool
Prompt the instructions to the model orchestrator and specialist-agent instructions in a fixed version, e.g. NEVER call refund_order without first calling lookup_order.
Model + provider pinned model ID Anthropic, claude-haiku-5-5
Allowed tools what the agent may call, including description and input schema allowlist per route: the refund route may call lookup_order, verify_customer, refund_order (destructive, needs approval) and faq_search, but not reset_password. The tool definitions are stored in the contract as a hash
Eval set + baseline test cases with expected results, plus this version’s pass rate 32 test cases in 9 categories (including positive_refund, auth_bola, idempotency) with expected route, required and forbidden tools and approval expectation; plus the pass rate v1 achieved on them

In the demo repo you can still switch the model via a setting. The table shows what v1 looks like with a pinned ID.

The refund-agent only has an output schema for the routing decision. The answer to the customer stays free text, and we check whether it’s right via the tool calls and their side effects, for example whether refund_order ran for the right order and whether a lookup_order came first. In the post on AI systems architecture, “contract-first” was already the principle, though back then only for the output schema.

Why one package instead of five separately versioned files? Because the links interlock. Every tool description ends up in the prompt. So if a third-party MCP server changes the description of refund_order, what the model reads changes, without anyone touching the prompt file. In the contract, the hash of the tool definitions then no longer matches.

And every model handles the same prompt and the same tools differently. Michael puts it like this: “The eval set is the link that holds the other four together.” Only through the test cases can we see whether the remaining links still work together. The contract is enforced in the implementation, by the policy gate and by schema validation.

The five links of the Agent ContractA frame labeled refund-agent, Contract v1. Inside, at the top, four pinned links side by side: output schema, prompt, model and provider, allowed tools including tool definitions. Below them, spanning the full width like a bracket, the eval set with this version’s baseline. Below the frame, the change rule: if one of the four links changes, the eval set runs again. If the eval set changes, the yardstick changes.refund-agent · Contract v1Output schemaroute enumPromptinstructions in a fixed versionModel + providerclaude-haiku-5-5Allowed toolsallowlist + definitions (hash)Eval set + baseline32 cases · pass rate v1 achieved against themOne of the four links changes: eval set runs again · Eval set changes: new yardstick
Four pinned links with the eval set as a bracket underneath. Only together do they make a contract version.

Rust gives us a similar split. According to the Cargo guide, Cargo.toml describes the dependencies “in a broad sense”, while Cargo.lock contains “exact information about your dependencies”. Applied to the refund-agent, “take any Haiku, be helpful” would be the manifest, and the four pinned links go into the lockfile.

The Cargo FAQ describes the purpose of a lockfile as “the state of the world at the time of a successful build”. So a lockfile proves that something was built. Whether what was built behaves correctly isn’t in it. Cargo has tests for that, and they live separately from the lockfile. With the Agent Contract, the eval set is part of the package, and the version includes the pass rate the eval run produced.

The analogy also breaks down when it comes to pinning. Cargo pins exact versions including a checksum; a model ID pins the weights and nothing more. Anthropic says so itself: “Model weights are fixed for a given ID, but the serving infrastructure around the model can change over time.” So the contract can’t guarantee identical behavior. The next run of the eval set shows any deviations.

The package is what gets versioned

If the links belong together, changes to them need a rule. And that rule is deliberately asymmetric:

Since the same version never answers identically twice, the cases run multiple times, and the baseline comes with a tolerance. Security categories like auth_bola still block hard: green before, red now, no release.

A made-up scenario for the refund-agent: from v1 to v2, schema, tools and prompt stay the same, only the model is new. The eval set runs, and the auth_bola cases fail, meaning the requests where a customer tries to get at someone else’s orders. The auth gate still blocks, but the new model tells the customer about a refund that never happened. Version 2 doesn’t go live. Without a contract, this would have been a model update that only showed up in the config.

Sometimes the break is even loud. Claude Haiku 4.5 accepts temperature, top_p and top_k. With Haiku 5.5, any non-default value returns a 400 error, and the same goes for Opus from 4.7 and Sonnet from 5. So if you’re running temperature: 0.2 and “just bump the model ID”, you end up with a feature that comes back with an error on every single request. I wouldn’t call that an upgrade.

In the evals CI post, I showed how the eval set becomes a gate in CI.

Evals in CI: When AI features stop being prompts
... and start being software - with all the testing obligations that come with it
rubeen.dev/en/blog/ai-evals-ci-pipeline
→

Michael sums up the problem in one sentence: “A prompt change is an invisible deployment.” In his setup, the eval set also runs every night against the frozen version, because the provider or a third-party MCP server moves even when we don’t touch anything ourselves. If the pass rate then drifts from the baseline, there’s an alert, but no new version.

Five is the core

Michael and I agree on the five links. My position beyond that: the contract isn’t a closed definition. I’d rather decide what gets added based on one clear criterion than on a long list.

The most obvious candidate is the retrieval index. If the refund-agent got a knowledge base via RAG for faq_search, the index version and the embedding model would belong in the package. The FAQ cases in the eval set would then have to match the index version; otherwise a test case checks for a return policy that the index changed long ago. Incidentally, the RAG index was already listed in the release manifest in the canary post.

Release Discipline for AI Features: Canary, Shadow Runs and Automatic Rollback
Green eval in the PR, red numbers in production two weeks later - without a new deploy. Why AI features need their own gates between merge and 100% traffic.
rubeen.dev/en/blog/ai-rollout-canary-shadow
→

The fallback model is more interesting. Under load, say when the provider responds with 429, a different model steps in, often from a different provider. For the customer who’s asking for their refund right now, it’s still the same refund-agent. So the eval set has to run against the fallback too, and the temperature trap from above applies to its parameters just the same. Otherwise, right at peak load, the answers come from a setup nobody ever measured.

Then there are things attached to the contract that don’t trigger an eval run: owner, approval, risk classification. That’s metadata. It drives processes, like who approves a new version or which documentation is required, but it doesn’t change the agent’s output. With data boundaries, it depends: as long as they only document, they’re metadata. If they filter what ends up in the prompt, they belong in the package. In the logging post, the signature and approved_by are attached to the contract as exactly that kind of metadata. Whether metadata lives in the same file or in a document next to it, honestly, I don’t much care. What matters is that a change of owner doesn’t create a new contract version or an eval run.

Doesn’t this already exist?

In bits and pieces, yes. There’s a tool that versions almost every link, and some even bundle several of them.

Google Dotprompt puts four of the five links into one file: prompt, output.schema, the allowed tools and the model, which config.version even pins down to the checkpoint. There’s no field for evals. Registries like Langfuse version prompt and model config together, and the evals run elsewhere. With MLflow, on the other hand, the model config stays editable after the fact: “Model configuration is mutable and can be updated after a prompt version is created.”

The closest match is Docker Agent. There, the agent definition (an agent.yaml), the evals and a baseline comparison sit side by side. The rule is strict: “An evaluation that passed and now fails always gates, regardless of the tolerance.” The baseline, however, is the JSON of an earlier run and sits next to the agent artifact. And getting a change to the model or to refund_order to trigger the eval run is something we wire up ourselves in CI. None of the tools ships the complete thing: the eval set plus baseline in the same package, and the rule for when it has to run.

Who bundles what?

The table reflects the state as of October 2026.

Tool Schema Prompt Model Tools Eval set Notes
Google Dotprompt yes yes yes yes no pins the checkpoint via config.version, no eval field
Microsoft Prompty yes yes yes yes no prompt format, evals outside
Langfuse Prompt Config partly yes yes yes no config versioned with the prompt, evals separate
MLflow Prompt Registry yes yes partly no no prompt versions immutable, model config mutable
Braintrust Prompts yes yes yes yes no prompt versioned with model and tools, evals separate
Docker Agent yes yes partly yes yes baseline separate from the artifact, wire up triggers yourself
Isn't this just MLOps?

At its core, yes. Sculley et al. described this back in 2015 as the CACE principle: “Changing Anything Changes Everything”. What’s new with LLM features is that the model is rented: the provider changes the infrastructure behind it, moves aliases and retires old IDs on its own schedule, without us committing anything. And part of the artifact is natural language, whose effect nobody can judge from a diff.

Start small

OpenAI is shutting down v1/prompts, the server-side versioned prompt objects, on November 30, 2026, and recommends: “Move versioning to your repo using git commits, PR review, and tests or evals.” OpenAI doesn’t prescribe a format for it. So the migration forces the question anyway: what belongs in your own repo? Just the prompt, or the whole package?

To start, one file next to the code is enough. The prompt is usually already in the repo; next to it go the output schema, the pinned model ID and the definitions of the tools the refund-agent is allowed to call. Add a dozen real cases with their expected results and the pass rate from the last run. That’s already enough for a contract, because every change can be measured against it. It can grow later.

Too much process for a prompt? For an internal side project, maybe. I think the effort is justified as soon as real users work with the feature, more than one person is tweaking the prompts, or a wrong answer costs something. With the refund-agent, it costs money, pretty much literally.

In the logging post, I described how I prove in production that the contract was honored.

A Prompt in the Wiki Is Not Evidence: Audit-Proof Logging and the Agent Contract in Regulated Environments
The Agent Contract is the intended state, the audit-proof log is the proof of what actually happened. Without the second, the first is just a claim. What DORA, GoBD and the upcoming MaRisk demand right now.
rubeen.dev/en/blog/revisionssicheres-logging-agent-contract
→

The answer to “What was actually running?” is a version number. Behind it are five things that were tested together.