Exec summary
Your evals are a model of reality, and reality drifts. Without live calibration, you will have an unmeasured distance between the reality-reality and the eval-reality
Always start by measuring this distance with your own eyeballs and your own judgment before deploying fancy auto-solutions
I’m in SF for Lenny’s Summit, and sending this post from a lovely Mili Wine Bar who did not object to my laptop-camping at the bar. If you’re in town and reading this, I’d love to connect.
If there were Nobel prizes for making agents work, Hamel and Shreya would surely be up there for the community service of popularizing the common sense approach to evals. It starts embarrassingly small - take your human eyeballs and your human judgment and apply them to ~100 traces. Eventually, as both your judgment and your agent improve, your machinery becomes more sophisticated and your human-in-the-loopness less trite. But the only way to get to a self-improving system is to build an “improving” system in the first place. And without someone like Hamel and Shreya, we’d be grabbing the first promising off-the-shelf eval tool, deploying its industry-leading automated judges and metrics like “helpfulness”, and watching the green dashboards while users can easily tell that nothing is, in fact, improving.
This is not a new problem, only its AI-shape is new. In Andy Grove’s High Output Management - a book about managing complex ecosystems of humans - he advocated for “in-process monitoring”, a continuous observation that catches problems earlier than a final inspection would. In short, when quality becomes less predictable, you look closer, and more frequently.
I originally came across Hamel’s and Shreya’s work last year, when I took their (now wildly popular) course for PMs and engineers, and recently went back to it when looking into public case studies of self-improving loops. And whaddya know, the real-world self-improving machinery seems to come down to continuously sampling what’s going on live in production, and using that sampling to evaluate and calibrate the whole system, on an ongoing basis1.
I’ve noticed that evals, for some reason, come with a 2024-era bad branding of static golden data sets, but it’s time we looked at them as the entire loop around the automation itself. So perhaps I should adjust my branding: call it the self-improvement machinery.
In the old mental model, evals looked something like this:
Golden dataset → improve score → ship and monitor as models change, or you add new use cases.
But this doesn’t work in the long run, and we have a lot of evidence to prove that generally, most material improvements in an automation happen post-launch; and that without some live evals self-improvement machinery, the agent will drift into the silt.
In response, you can trace the industry landing on a new system that looks something like this:
Start with some golden dataset → ship → observe production with observability and traces2 → discover failures → judge / determine / settle → update your evals and verifiers → improve systems (subagent judges, prompts, tools, etc.) → ship → observe production → …
If I were going for a cheesy metaphor, I could say that you need to apprentice AI into the work it’s doing. The launch is its first day on the job, and the only way it’ll grow up to handle serious work is by the masters giving up their valuable time to train it. Eventually, the master won’t have to look at every single chair leg the apprentice chisels, but they will still watch over the weird ones.
Similarly, in a self-improving machinery, humans never stop looking at the traces, but the role of their judgment changes:
v0: human looks at 100 traces
v1: humans define patterns and rubrics
v2: automated graders cover known patterns
v3: sampling machinery finds suspicious/novel trajectories (which 0.1% of traces should a human look at?)
v4: humans inspect those and modify the measurement system
v5: the agent gets changed based on that updated measurement system
The key components here are:
You have to be able to observe production traces at scale to see what’s not working, for example via clustering
Your evals are a model of reality, and reality drifts. Without live calibration, you will have an unmeasured distance between the reality-reality and the eval-reality.
I’m increasingly bullish on the next innovation problem being not the next cool agent, but the mechanism that turns the idiosyncratic expertise into agent learning signals. Is that a lightweight UI exposed to experts that automates feedback collection? Is it some ambient agent that monitors user actions across the loop? Some simulation swarm that creates permutations in search of a correct solution?
Here’s a pragmatic set of questions you can ask:
What percentage of production trajectories are actually evaluated and sampled by humans?
Who reviews ambiguous or new failures and what sampling mechanism selects them?
How are automated judges calibrated against humans over time?
How often do production failures get promoted to golden dataset?
If your AI product doesn’t quite work as well as you had expected, look at how often human eyeballs touch the full trajectories, and if it’s “infrequently”, start there.
Sources and more reading
https://openai.com/index/evals-drive-next-chapter-of-ai/
https://lilianweng.github.io/posts/2026-07-04-harness/
Hamel Husain’s 2025–26 eval bible
Eval sets should be small and curated, production traffic should be sampled separately. Then, production failures should be added back into the dataset.
Anthropic Clio (old but good - 2024)
The verification horizon (arXiv, June 2026)
Verification signals trade off scalability, faithfulness and robustness; as generators improve, fixed reward functions become inadequate. Their conclusion is essentially that ✨verification must co-evolve with the generator✨
Hack-verifiable environments (arXiv, May 2026)
Agents can exploit the evaluation signal while violating the intended objective (surprise!)
Failure as a process (arXiv, July 2026)
1,794 coding-agent trajectories / 63k+ steps; many failures begin early but stay invisible until recovery is difficult. Supports looking at trajectories and intermediate behavior rather than only end outcomes!
Agent-evaluation survey (arXiv, revised Apr 2026)
Finds the field moving toward more realistic and continuously updated benchmarks and identifies robustness/fine-grained evaluation as outstanding problems.
there’s some societal benefit here of guarding against gradual disempowerment, too
one unasked question that always comes up in my mind here is privacy preservation, but I don’t want to be a total party pooper