Kelly Michels 8 min read QA

The Night Shift That Never Merges

When a nightly test fails, an AI agent files the bug and opens the fix. Merging it is my job, not its.

Every night, 13 test suites across my projects run unattended: 12,435 tests on November 15. When one fails, a headless Claude Code session is handed the failure, files the bug, fixes it on a branch, and opens a pull request it is told never to merge. Eighteen sessions in, here is what it fixed, what it would not fix, and the seven times it did nothing at all.

What happens when a test fails

The nightly run is a scheduled task on my build machine. It runs every suite in every project (pytest, Vitest, Jest, Pester, Node, and Appium driving an Android emulator), records each test by name in a SQLite history, and publishes the results to the live results board. A suite that fails sets off five steps:

  1. A dossier. The run writes up the failure: the suite, its runner, the pass and fail counts, the commit it measured, and every failing test with the output it captured.
  2. A fingerprint. The suite's name plus a hash of its sorted failing test names. If that fingerprint has been triaged before, nothing new starts, so a failure that stays red for a week costs one session, not seven.
  3. A headless session. Claude Code starts in that repository with nobody at the keyboard (claude -p), on one pinned model, with a hard cap of 30 minutes. Its whole brief is one paragraph: read the dossier; file a GitHub issue titled bug: …; branch off the latest main; diagnose and fix; run the suite to verify; open a pull request that says Closes #N. Do not merge it, and do not deploy anything. If the failure is environmental rather than a code defect, say so in the issue instead of forcing a code change. Leave every checkout on the branch you found it on.
  4. A record. The session's transcript is kept beside the dossier, and its exit code goes into the history.
  5. An email. Every run with a failing suite sends a plain-text message: the failing tests, the first lines of their output, and what triage did about each one.

In the morning the pull request is waiting, and I review it the way I would review anyone's.

Eighteen sessions, counted

The loop has run since August 27, 2026. Through November 15 it started eighteen sessions:

What came of itSessions
A fix, reviewed and merged5
A cause outside the code, reported instead of patched5
A failure it had already filed, recognized and re-verified1
Never started: the sign-in had expired5
Never started: the API account was out of credit2

All five merged pull requests went in as the single commit the agent wrote. The seven that never started taught me more than the five that worked, and they have their own section.

The five it fixed

  • A flaky test that was a real bug. In ProvenSheet, a test that commits a cell edit failed now and then. The agent traced it to the cell component: the effect that re-syncs a cell from the sheet could overwrite a draft the user had typed but not yet committed. It fixed the component, not the test, because the flake was the product losing someone's typing.
  • Two repositories out of step. A test in evo-ai checks that this site's evo-ai page describes the product as it is. The page said nineteen analytics views; the product had twenty. The fix belonged in the other repository, and that is where the agent filed the issue and opened the pull request.
  • A stale test, not a stale page. The crash-reporting docs had been corrected to the right recipient address, and a test still asserted the old one. The agent changed the test and pointed the issue at the commit that had corrected the docs.
  • Examples without evaluations. Seven new example questions had been added to the Ask AI docs page with no evaluation case behind them, and the test that keeps the two in step went red. The agent wrote the seven cases.
  • Sixteen failures that only happened on Windows. The edge's tests picked up the WSL bash instead of Git's on the Windows test machine, and Docker there refused the temporary bind-mounts some tests need. The agent made the harness look for a bash that works, and skip the Docker tests when the mount is refused.

The five it would not fix

The most useful sentence in the brief is the one that lets the agent say no. A test that fails because the machine is broken is telling the truth, and the quickest way to make a suite worse is to "fix" that test until it passes on a broken machine. Five times the agent found the cause outside the code, wrote the issue saying so, and opened no pull request:

  • Windows Defender had quarantined a native binary that the spreadsheet engine loads.
  • The Android emulator could not boot, because the drive it lives on was full.
  • A Python virtual environment still pointed at an interpreter that had since been deleted.
  • Another had drifted to an older library version than its lockfile pins, which tripped a canary test that exists to notice that library's behavior changing.
  • A test had read a source file in the middle of an edit.

For the two virtual environments it also put the machine right, rebuilding each from its lockfile, and ran the whole suite again to show it was green.

The session left over is the one I like best. On November 15 the edge failed again, on a different set of tests, so the fingerprint was new and a session started. It found the issue and pull request it had opened the night before, ran that pull request's branch in a throwaway worktree (17 passed, including all 13 that were failing on main), left a comment on the pull request tying the new failure to it, and filed nothing. That is what I would want from a person, too.

When it did nothing

An agent that fails quietly looks exactly like an agent with nothing to do. These are the three ways this one failed, and what changed after each.

It left a repository on its fix branch

Early on, a session branched to make its fix and never switched back. A repository left on a fix branch stops that night's results being published, so the board stayed on the night before. The brief now asks the agent to leave every checkout where it found it, and the code no longer relies on the asking: before a session starts, the run records the branch every repository is on, and afterwards it puts them all back. That includes when the 30-minute cap kills a session mid-edit, which is exactly when a repository is most likely to be left somewhere odd.

It could not sign in

I moved the nightly to a scheduled task that runs whether or not anyone is signed in to the machine, so an overnight reboot would not cost a night. That kind of logon cannot open the credential store where the Claude Code CLI keeps its subscription sign-in. Every session after the change ended after 13 seconds with "OAuth session expired and could not be refreshed", while the same command, run by hand, worked.

Nothing in the loop surfaced it. The run's log said the session had exited 1, the same line it writes for any session that fails. The fix had two halves: sessions now authenticate with an API key, which needs no credential store, and the failure email now says what triage did for each suite: that a session ran and how it exited, with the first line of its transcript when it did not exit cleanly.

It ran out of credit

API sessions are billed. On November 5 the account ran dry, and two sessions ended on "Credit balance is too low". That is the first line of each transcript, so it is the line the failure email carries; the next session, the following day, ran.

What holds it back

  • Merging is mine. The brief says never merge and never deploy, and every transcript that opened a pull request ends with it left open for review. GitHub enforces part of that: main takes changes only through a pull request with passing checks. It does not enforce the rest, because the session runs under my GitHub sign-in, which is allowed to merge. What stands between the agent and the merge button is the instruction, the record of it keeping to it, and my review of every pull request it opens.
  • 30 minutes, then it is stopped. A session still going at the cap is killed, recorded as killed, and its checkouts are put back.
  • One session per failure. The fingerprint keeps a failure that stays red from starting a new session every night. The email still comes on every run, because a failure that is still there tomorrow is still a failure tomorrow.
  • It works unattended, so it cannot ask. Nobody is at the keyboard at night to approve a command, so it runs with Claude Code's permission prompts turned off. That is the reason for the limits above, and the reason it runs on my build machine and not on a server that customers use.

What I would tell the next person

  • Give it a way to say no. "If the failure is environmental, say so instead of forcing a code change" decided five of the eleven sessions that ran. An agent told only to make a red test green will find a way, and the way may be the test.
  • Make it report what it did, most of all when it did nothing. An exit code in a log nobody reads is not a report.
  • Do in code what the brief asks for. "Put the checkouts back" is in the brief and in the code. Only the code is a guarantee.
  • Review it like a colleague who works nights. Its pull requests get the same review as anyone's. It is right often enough to be worth having, and that is exactly why the review cannot be skipped.

The stack, for the record

Runner: evo.testsuites, Python with a SQLite history, run nightly by Windows Task Scheduler and published to this site by PowerShell. Suites: pytest, Vitest, Jest, Pester, Node, and Appium against an Android emulator. Triage: the Claude Code CLI in headless mode (claude -p), pinned to Claude Opus 4.8 and authenticated with an API key, with the GitHub CLI for the issues and pull requests. Reporting: a plain-text email on every run with a failure, and the results board.

Every suite this post is about, and every test in the public ones, is on the live results board. See also how defects get found and filed. Questions? Get in touch.