If you only version the prompt of an LLM feature, you don’t know what you deployed.
Friday afternoon, someone tightens a sentence in the refund-agent’s prompt. Over the weekend, the provider swaps in a new model behind an alias. Or the MCP server that provides refund_order changes its tool description (we already had that case in the canary post). There’s no commit and no release for any of this in the service repo. And yet on Monday the agent issues refunds differently than it did on Friday.
Which version of the refund-agent was actually running on September 12? The usual answer is a Git SHA. But that only tells you which code ran. Which prompt from the prompt CMS went with it, which model was behind the alias and which tool description came from the third-party server: none of that is in there. Even when the prompt lives in the repo, its changes are hard to spot in the history. In a study of 243 GitHub repos, only a little over one in five commits that changed a prompt even mentioned it in the commit message (21.9%, Prompting in the Wild, MSR 2025).
On ai-systems-architecture.com, Michael Seel and I call the package that answers this question the Agent Contract. It’s our engineering term; there’s no standard for it. In research, the term means something else: for Ye & Tan, agent contracts are formal runtime contracts with resource budgets. Ours is a release artifact: what gets shipped, and what has it been tested against? Michael has worked out the term in detail. If you want the long version, here it is:
The example is still the refund-agent from the previous posts.
Five things, one feature
What are we deploying when we deploy the refund-agent? The code is in the repo, sure. But whether a customer saying “I want my money back” ends up on the right route, and whether the right thing happens there, is decided in five places, and not all of them are in the repo.
| Link | What’s in it | refund-agent, Contract v1 |
|---|---|---|
| Output schema | the shape the code relies on | routing decision as an enum (refund, lookup, faq, account, clarify), enforced via the route tool |
| Prompt | the instructions to the model | orchestrator and specialist-agent instructions in a fixed version, e.g. NEVER call refund_order without first calling lookup_order. |
| Model + provider | pinned model ID | Anthropic, claude-haiku-5-5 |
| Allowed tools | what the agent may call, including description and input schema | allowlist per route: the refund route may call lookup_order, verify_customer, refund_order (destructive, needs approval) and faq_search, but not reset_password. The tool definitions are stored in the contract as a hash |
| Eval set + baseline | test cases with expected results, plus this version’s pass rate | 32 test cases in 9 categories (including positive_refund, auth_bola, idempotency) with expected route, required and forbidden tools and approval expectation; plus the pass rate v1 achieved on them |
In the demo repo you can still switch the model via a setting. The table shows what v1 looks like with a pinned ID.
The refund-agent only has an output schema for the routing decision. The answer to the customer stays free text, and we check whether it’s right via the tool calls and their side effects, for example whether refund_order ran for the right order and whether a lookup_order came first. In the post on AI systems architecture, “contract-first” was already the principle, though back then only for the output schema.
Why one package instead of five separately versioned files? Because the links interlock. Every tool description ends up in the prompt. So if a third-party MCP server changes the description of refund_order, what the model reads changes, without anyone touching the prompt file. In the contract, the hash of the tool definitions then no longer matches.
And every model handles the same prompt and the same tools differently. Michael puts it like this: “The eval set is the link that holds the other four together.” Only through the test cases can we see whether the remaining links still work together. The contract is enforced in the implementation, by the policy gate and by schema validation.
Rust gives us a similar split. According to the Cargo guide, Cargo.toml describes the dependencies “in a broad sense”, while Cargo.lock contains “exact information about your dependencies”. Applied to the refund-agent, “take any Haiku, be helpful” would be the manifest, and the four pinned links go into the lockfile.
The Cargo FAQ describes the purpose of a lockfile as “the state of the world at the time of a successful build”. So a lockfile proves that something was built. Whether what was built behaves correctly isn’t in it. Cargo has tests for that, and they live separately from the lockfile. With the Agent Contract, the eval set is part of the package, and the version includes the pass rate the eval run produced.
The analogy also breaks down when it comes to pinning. Cargo pins exact versions including a checksum; a model ID pins the weights and nothing more. Anthropic says so itself: “Model weights are fixed for a given ID, but the serving infrastructure around the model can change over time.” So the contract can’t guarantee identical behavior. The next run of the eval set shows any deviations.
The package is what gets versioned
If the links belong together, changes to them need a rule. And that rule is deliberately asymmetric:
- Schema, prompt, model or tool set changes: new contract version. The eval set runs in full, and only then does the version go live.
- The eval set changes: also a new version, because it moves the yardstick. The running configuration gets measured against the new set, and that becomes the new baseline. You can only compare versions that ran against the same eval set.
Since the same version never answers identically twice, the cases run multiple times, and the baseline comes with a tolerance. Security categories like auth_bola still block hard: green before, red now, no release.
A made-up scenario for the refund-agent: from v1 to v2, schema, tools and prompt stay the same, only the model is new. The eval set runs, and the auth_bola cases fail, meaning the requests where a customer tries to get at someone else’s orders. The auth gate still blocks, but the new model tells the customer about a refund that never happened. Version 2 doesn’t go live. Without a contract, this would have been a model update that only showed up in the config.
Sometimes the break is even loud. Claude Haiku 4.5 accepts temperature, top_p and top_k. With Haiku 5.5, any non-default value returns a 400 error, and the same goes for Opus from 4.7 and Sonnet from 5. So if you’re running temperature: 0.2 and “just bump the model ID”, you end up with a feature that comes back with an error on every single request. I wouldn’t call that an upgrade.
In the evals CI post, I showed how the eval set becomes a gate in CI.

Michael sums up the problem in one sentence: “A prompt change is an invisible deployment.” In his setup, the eval set also runs every night against the frozen version, because the provider or a third-party MCP server moves even when we don’t touch anything ourselves. If the pass rate then drifts from the baseline, there’s an alert, but no new version.
Five is the core
Michael and I agree on the five links. My position beyond that: the contract isn’t a closed definition. I’d rather decide what gets added based on one clear criterion than on a long list.
The most obvious candidate is the retrieval index. If the refund-agent got a knowledge base via RAG for faq_search, the index version and the embedding model would belong in the package. The FAQ cases in the eval set would then have to match the index version; otherwise a test case checks for a return policy that the index changed long ago. Incidentally, the RAG index was already listed in the release manifest in the canary post.

The fallback model is more interesting. Under load, say when the provider responds with 429, a different model steps in, often from a different provider. For the customer who’s asking for their refund right now, it’s still the same refund-agent. So the eval set has to run against the fallback too, and the temperature trap from above applies to its parameters just the same. Otherwise, right at peak load, the answers come from a setup nobody ever measured.
Then there are things attached to the contract that don’t trigger an eval run: owner, approval, risk classification. That’s metadata. It drives processes, like who approves a new version or which documentation is required, but it doesn’t change the agent’s output. With data boundaries, it depends: as long as they only document, they’re metadata. If they filter what ends up in the prompt, they belong in the package. In the logging post, the signature and approved_by are attached to the contract as exactly that kind of metadata. Whether metadata lives in the same file or in a document next to it, honestly, I don’t much care. What matters is that a change of owner doesn’t create a new contract version or an eval run.
Doesn’t this already exist?
In bits and pieces, yes. There’s a tool that versions almost every link, and some even bundle several of them.
Google Dotprompt puts four of the five links into one file: prompt, output.schema, the allowed tools and the model, which config.version even pins down to the checkpoint. There’s no field for evals. Registries like Langfuse version prompt and model config together, and the evals run elsewhere. With MLflow, on the other hand, the model config stays editable after the fact: “Model configuration is mutable and can be updated after a prompt version is created.”
The closest match is Docker Agent. There, the agent definition (an agent.yaml), the evals and a baseline comparison sit side by side. The rule is strict: “An evaluation that passed and now fails always gates, regardless of the tolerance.” The baseline, however, is the JSON of an earlier run and sits next to the agent artifact. And getting a change to the model or to refund_order to trigger the eval run is something we wire up ourselves in CI. None of the tools ships the complete thing: the eval set plus baseline in the same package, and the rule for when it has to run.
Who bundles what?
The table reflects the state as of October 2026.
| Tool | Schema | Prompt | Model | Tools | Eval set | Notes |
|---|---|---|---|---|---|---|
| Google Dotprompt | yes | yes | yes | yes | no | pins the checkpoint via config.version, no eval field |
| Microsoft Prompty | yes | yes | yes | yes | no | prompt format, evals outside |
| Langfuse Prompt Config | partly | yes | yes | yes | no | config versioned with the prompt, evals separate |
| MLflow Prompt Registry | yes | yes | partly | no | no | prompt versions immutable, model config mutable |
| Braintrust Prompts | yes | yes | yes | yes | no | prompt versioned with model and tools, evals separate |
| Docker Agent | yes | yes | partly | yes | yes | baseline separate from the artifact, wire up triggers yourself |
Isn't this just MLOps?
At its core, yes. Sculley et al. described this back in 2015 as the CACE principle: “Changing Anything Changes Everything”. What’s new with LLM features is that the model is rented: the provider changes the infrastructure behind it, moves aliases and retires old IDs on its own schedule, without us committing anything. And part of the artifact is natural language, whose effect nobody can judge from a diff.
Start small
OpenAI is shutting down v1/prompts, the server-side versioned prompt objects, on November 30, 2026, and recommends: “Move versioning to your repo using git commits, PR review, and tests or evals.” OpenAI doesn’t prescribe a format for it. So the migration forces the question anyway: what belongs in your own repo? Just the prompt, or the whole package?
To start, one file next to the code is enough. The prompt is usually already in the repo; next to it go the output schema, the pinned model ID and the definitions of the tools the refund-agent is allowed to call. Add a dozen real cases with their expected results and the pass rate from the last run. That’s already enough for a contract, because every change can be measured against it. It can grow later.
Too much process for a prompt? For an internal side project, maybe. I think the effort is justified as soon as real users work with the feature, more than one person is tweaking the prompts, or a wrong answer costs something. With the refund-agent, it costs money, pretty much literally.
In the logging post, I described how I prove in production that the contract was honored.

The answer to “What was actually running?” is a version number. Behind it are five things that were tested together.
