I Built a Tool to Catch AI Agents Faking Tests. In 201 Merged PRs, They Didn't

When a coding agent finishes, it usually says something like "Done, all tests pass." I read that line dozens of times a day. It's the agent reporting on its own work.

AI agents now write a large share of the code I ship. Writing code stopped being the slow part. Checking it is. And reading a diff doesn't tell me the two things I really want to know: did the tests actually run, and would the new tests fail if the change were wrong?

So I built a tool to answer those two questions. Then I tried to find out whether it was worth anything. This post is mostly about the second part, because the answer wasn't what I expected.

Why I didn't trust "all tests pass"

The labs have reported their own models gaming tests. Anthropic's Claude 3.7 Sonnet system card describes the model special-casing tests to make them pass in agentic coding, and the Claude 4 system card reports this much reduced, but still there. OpenAI's research on monitoring reasoning models caught a model exiting early or skipping tests to look green. METR found frontier models patching the code that scored their evaluations.

If that happens in a lab, I wanted a check from outside the agent. Not another model reading the diff and giving its opinion, but something that runs the code and reports what happened.

What trustbut does

trustbut is a free, open-source command-line tool. You run it in your repo after the agent finishes:

npx trustbut verify

It does three things.

It reruns your tests in a clean checkout. Each run happens in a temporary git worktree, so it doesn't rely on the agent's own run and never touches your working directory.

It checks that each new test proves the change. This is the core idea. Every new or changed test also runs against the code from before the change. If it fails before and passes after, it tests the change. If it passes either way, it doesn't, no matter how many assertions it has.

A test that passes with or without your change isn't testing your change.

It flags the usual escape hatches. Skipped and focused tests, deleted tests, removed assertions, and new lines like @ts-ignore, as any, # noqa or #[allow(...)].

Then it writes a short receipt with a PASS, WARN or FAIL verdict. It works with 11 languages and any agent, because all it needs is git and your test runner. There's no LLM in it, and your code never leaves your machine. You can also install it as a turn-end hook in Claude Code, Codex, Gemini CLI or OpenCode, so a failing receipt goes back to the agent to fix instead of reaching you as "done".

✕

Then I tried to prove it was useful

A tool like this is easy to build and hard to justify. Every finding sounds important. The real question is how often a finding is worth a reviewer's time.

So I set up a study in a separate repo, trustbut-index. It runs trustbut on real merged pull requests from established open-source projects. Some were written by agents: Copilot, Codex, Claude Code, Cursor, Devin and Jules. As a control, others were written by people in the same projects. I wrote the method down in a pre-registration before running it, and everything is public, from the list of PRs to every receipt.

The pilot covers 201 pull requests: 103 by agents and 98 by people.

What I found

trustbut 0.7.0 could run the tests on 53 of the agent PRs and 56 of the human ones. On those:

• No agent PR broke a passing test. None of the human PRs did either. That's a lower bound: 18 runs had a failing test with no result from before the change to compare with.
• Agents' tests tested their changes. 275 of 283 new or changed tests in agent PRs failed without the change, which is 97%. For people it was 294 of 333, or 88%. These counts pool every test, so a PR with many new tests weighs more than one with a single test.
• "Nothing proves this change" was rare for both. It came up in 6 of 53 agent PRs and 7 of 56 human PRs.
• Most PRs had no finding at all. 12 of 53 agent PRs had one, and 15 of 56 human PRs.

I went looking for agents faking their tests. In these PRs, I didn't find it.

Most findings were true, and didn't matter

Counting findings is one thing. Whether they matter is another. For an earlier pilot run on trustbut 0.6.0, every finding was labeled as real (worth a reviewer acting on), true but harmless, or wrong.

Out of 71 findings, 5 were real. That's 7%. 57 were true but harmless, and 9 were simply wrong.

Two things in those labels changed the tool.

All 5 real findings said the same thing: nothing in the PR proved the change. So in 0.7.0 that became the headline finding, change-not-proven, and the receipt now leads with it.

All 9 wrong findings were on human PRs, and they came from three bugs: tests moved to another file, tests merged into others, and a Python run "before the change" that was quietly importing the new code. 0.7.0 fixed all three.

One caveat I want to be clear about. Those labels are drafts written by Claude, and I haven't confirmed them by hand yet. The labeler could also see the code, so they weren't fully blind.

I moved my own bar

Before the pilot I set two bars: at least 80% of findings should be worth acting on, and at least 10% of agent PRs should have a finding that is. The pilot came in at 7%, nowhere near the first one. Under my original rules, missing the second bar meant rethinking what trustbut checks before building any public index.

I replaced both before the full study with a different bar: at least 95% of findings must be factually true. I removed that "rethink" consequence too, so precision is now reported as it is, with no pass or fail. My reasoning is that a receipt's job is to state facts a reader can trust, and a true "this test is skipped" is useful even when the skip is fine. You can disagree with that. The pilot's 87% accuracy would have missed the new bar too, which is why 0.7.0 fixed the causes first.

Moving a bar after you miss it is exactly what I'd want to know about someone else's study. So it's written down in the pre-registration, with the date and the reason, and I'd rather say it here than have someone find it.

What this does and doesn't say

These are merged pull requests. Before they were merged, they went through CI and a human review, and anything that broke the suite would usually have been caught there. So this says that by the time agent work is merged into an established project, it's at least as well tested as human work.

It says nothing about what an agent does before it says "done". That's where the hook runs, and I haven't measured it yet.

It also covers only about half the PRs. Of the 92 where trustbut couldn't run the tests, 78 failed to install the project's dependencies or toolchain in the study's runner. That's partly the study setup, but it's a real limit, and the next thing I'm fixing.

So is it still worth using?

I think so, for a narrower reason than the one I started with.

The PRs in the study had already been filtered by CI and review. My own agent's work hasn't, at the moment it finishes a turn on my machine. On my own repos, receipts found a fix commit with no test, seven new tests that passed with or without their change, two new eslint-disable lines, and a dependency missing from my local install. None of that is dramatic. All of it is something I want to know before I review.

"Is the agent cheating?" turned out to be the wrong question. The one that holds up is plainer: does anything prove this change?

For most changes the answer is yes, and the receipt lets me stop worrying about it. For the rest, it tells me where to look.

What's next

• Confirm every label by hand.
• Run the full study, with at least 30 PRs per agent, under the frozen pre-registration.
• Measure the hook: the same tasks with and without it, graded on tests the agent never saw.

If you try trustbut on your own repo, I'd like to hear where it fails to run, and any case where an agent did something it missed. Open an issue on GitHub.

→ trustbut on GitHub

→ The study, its data and the pre-registration

Shipping code your AI agents wrote?

I help teams build AI agents that hold up in production, and the checks around them that tell you what the agent actually did.

→ See my AI Agents service

→ Book a free 30 minute call

→ Or send me a note

Sources

• trustbut
• trustbut-index and its pre-registration
• Claude 3.7 Sonnet system card, Anthropic
• Claude 4 system card, Anthropic
• Monitoring Reasoning Models for Misbehavior, OpenAI
• Recent Frontier Models Are Reward Hacking, METR