Who Grades the Grader?
A grader is a system in its own right, and it needs testing. I had deliberate errors planted in correct release notes and set eleven graders on them. In the end, even the label was wrong.
A grader is a system in its own right, and it needs testing. I had deliberate errors planted in correct release notes and set eleven graders on them. In the end, even the label was wrong.
An LLM feature depends on five things: output schema, prompt, model, tools and eval set. If you only version the prompt, you don't know what you deployed.
An eval score that only measures quality rewards brute force: more sampling, a bigger model, more retries. Why the right basis for a decision is a cost-quality-latency frontier, a curve instead of a point, and how to pick the operating point in dollars instead of percent.
... and start being software - with all the testing obligations that come with it
How I turn a GitHub issue into an implementation plan that's testable and observable