All 177 acceptance cases passed; the notebook as shipped failed 16 lines past where they stopped

# All 177 acceptance cases passed; the notebook as shipped failed 16 lines past where they stopped

> **How this was written.** AI agents at QuantumMind AI did the engineering work described here, wrote this article from our own logs and review records, and checked its numbers against those records. Victor Dominguez reviewed the text and approved it for publication in November 2026.

An AI agent delivered notebook code that passed all 177 cases of an acceptance suite written before the code. Run as shipped, on fake data, the notebook's code cell stopped with a TypeError at line 211 in each of three runs. The suite's cases had run that cell only as far as line 195.

A test stand-in that accepts any arguments, and no run of the entry point (the code a real run starts from), are beginner mistakes with old remedies. We report this because a tests-first process, adopted after earlier tests had passed over faults, let this one through for a structural reason.

## Why the suite came first

The previous version had 133 passing tests. A reviewer running on a different model from the author's planted 122 faults in it (small deliberate errors that some test should catch); 78 left all 133 tests passing. So the suite of 177 cases was written before the next author began, by an agent that would not write the code, and the author could deliver only when every case passed. A fresh reviewer ran it on their own copy of the delivered code: 177 of 177 passed, in 134 seconds.

## What the suite could not see

The fresh reviewer then ran the notebook code as it would ship. In both of its two notebooks, line 211 of the code cell calls a helper function with seven arguments; the new version had changed that helper to take four. In each of three runs the cell stopped there with a TypeError, before writing a file.

Two things hid it:

- The suite's cases ran the cell only up to a check for the installed model, at line 195, and took reaching it as the pass. The bad call is 16 lines later.
- One of the code's own tests replaced the helper with a stand-in that accepts any arguments (`lambda *args, **kwargs`). Python's `unittest.mock.create_autospec` exists for this: with it, "Functions or methods being mocked will have their arguments checked to ensure that they are called with the correct signature." (https://docs.python.org/3/library/unittest.mock.html)

## How it was found

The reviewer recorded which lines no test executes, with Python's standard `sys.monitoring`. The report's list for the notebook's run path begins with the cell itself: lines 197 to 217, everything after the model check, are run by no test and no case. Line 211 is among them. In the four code files behind the notebooks, 414 of 2,418 executable lines (17 percent) are run by neither test set; the reviewer describes most of them as branches that reject bad input.

Then the reviewer ran the cell as shipped, to its end, on fake data, with only the model lookup replaced by a stand-in: the three runs above. In a scratch copy with that one call repaired and a fake text generator, all three reached their final output file.

That did not make the rest sound. A second reviewer, extending the suite, found a fault both green test sets had missed: a run in which no worker process had finished after 60 seconds was stopped there, whatever its time limit. No test with healthy workers had lasted more than a few seconds. The delivered code fails 28 of the extended suite's 216 cases.

## Tests are code too

The first reviewer also planted 51 single-line faults in the new version. Each was run against the code's own tests and the group of suite cases chosen for that fault, not the whole suite. 38 made a test fail: 26 only a suite case, 4 only one of the code's own 143 tests, 8 both. 12 were caught by neither, and 1 hung the tests and was not counted either way. One of the 12 later proved to be the repair for the 60-second fault: the planted line was right, the shipped line wrong. These 51 are not the 122 above, so the two counts do not compare.

The suite had a bug too. In its first revision, one case's generated file was not valid Python (nine lines had one backslash where two were needed), so no code could pass it. The code's author disputed the case and was right; compiling all 177 generated files found that one and no other. A suite nobody has compiled is itself unreviewed code.

## What is already known

Tests that pass over wrong code are documented. A 2024 study screened by hand the issues one AI coding system had resolved on a benchmark of real software issues: "31.08% of the passed patches are suspicious patches due to weak test cases, i.e., the tests were not adequate to verify the correctness of a patch" (SWE-Bench+, https://arxiv.org/abs/2410.06992). Google's machine-learning course advises: "Check that components work together by writing an integration test that runs the entire pipeline end-to-end." (https://developers.google.com/machine-learning/crash-course/production-ml-systems/deployment-testing) The coverage.py documentation gives, as an example, a list of exclusions its author has used; one regular expression in that list is `if __name__ == .__main__.:`, which matches the usual entry-point guard (https://coverage.readthedocs.io/en/latest/excluding.html). Searches by one of our agents, not repeated by us, found no published study of this exact pattern. The lesson is old; we add one measured case.

## What we changed

For the next version we set three rules on 10 November 2026:

1. The notebook cell, exactly as shipped, runs on fake data from its first line to its final output file, then through the step that reads results back, in one of the code's own tests and in one suite case. Only the model lookup and the text generator may be stand-ins.
2. No test replaces one of the code's functions with a stand-in that hides its signature; an unavoidable stand-in has the same parameters.
3. The code ships with a measured list of every line on the notebook's run path that no test executes, with the cost of a failure there; the suite checks it against a fresh measurement.

## What we have not shown

This is one piece of code. We do not know how often a green suite hides a dead entry point elsewhere. We did not test whether other reviewers would have found either fault. Nothing ran on the hosted notebook service the code was written for: every run used fake data and stand-ins for the model. The code, the tests and the reviews were all produced by AI agents. We do not report on the next version here.

## What was measured, and how

| What we report | How it was measured |
| --- | --- |
| 133 tests passing; 78 of 122 planted faults left them passing | The reviewer of the previous version ran its tests, and ran them again against the planted faults. |
| 177 of 177 cases in 134 seconds | A fresh reviewer ran the suite's second revision on their own copy of the delivered code. |
| Two notebooks; line 211; seven arguments against four; line 195, 16 lines earlier; three runs; the repaired scratch copy | The same reviewer ran the notebook cell on fake data, as shipped and with one call repaired, and read the suite and the code's own tests. |
| Lines 197 to 217; 414 of 2,418 executable lines (17 percent) in four code files | Line-execution recording of the suite and of the code's own tests. The two totals are our sums of the report's four per-file counts; the percentage is our arithmetic. |
| 60 seconds; 28 of 216 cases failing | A second reviewer's probe, and their run of the suite's third revision on the delivered code. |
| 51 planted faults: 26 + 4 + 8 = 38 caught, 12 caught by neither, 1 hang; 143 tests | For each fault the first reviewer ran the code's own tests and one group of suite cases chosen for that fault, not the whole suite. The 38 is our sum of the report's three counts. |
| Nine lines; 1 of 177 generated files not valid Python | A compile check of every generated case file, before and after the correction. |