4.6 The harness self-test

Testing the test harness sounds like one turn of the screw too many, until you notice where every number this project quotes comes from. Each one has passed through a harness on its way to you. If that harness under-reports, everything downstream of it is wrong in a way that looks identical to being right — the run is green, the totals are plausible, and nothing anywhere says otherwise. So the harnesses are themselves under test.

$ make harness-selftest

The regression suite also runs it as the harness-selftest-func row, so it is exercised on every full run rather than only when someone remembers to ask.

Why it exists

test/run-unit-tests.sh once shipped with a bug that appeared only when a suite failed, which is the one path nobody exercises while everything is green. It had been verified, thoroughly, against suites that pass. What happened when one did not was that set -e killed the runner subshell before it could record the suite's exit code; the aggregator then aborted; and every suite after the failing one was dropped from the run. A harness that under-reports precisely when something is wrong is worse than no harness at all.

How it works

Each fault is injected against stub suites in a throwaway build directory, and the refusal is then asserted — both the expected exit status and the exact diagnostic text. The principle is the same one that runs through the chapter: a guard that cannot be shown to fire is not a guard.

The stubs are a few lines of bash that report one row ID per row, print a Total: line and exit with a chosen code. A register helper writes a matching CTestTestfile.cmake, and a manifest helper writes a manifest with a matching — or deliberately wrong — # expect: pin. From those few primitives the self-test drives, among others, a clean two-suite run; a suite that fails while a later suite must still be reported; a suite that prints no summary; a suite that hangs into its timeout; a suite that reports one row ID twice, or fewer or more row IDs than rows; a real fork()ing binary whose child reports a row before its parent does; a row ID built at run time, a data-ID exemption and a stale one, and a suite with no known sources; an entry with no row count; a pin of 0; one binary registered under two add_test() names; a wrong suite-count pin; a manifest with no pin at all; a nested build tree that must not be enumerated; an unparseable add_test() line; and a membership probe that proves the printf | grep -q SIGPIPE race cannot come back.

It also reaches beyond the unit harness. Rows HS-21..HS-24, HS-31 and HS-32 drive regression.sh --preflight-only against truncated manifests, a declared functional test with no script, and a stray undeclared script — which is why make harness-selftest declares unit-test-build, gui-release and sdl-release as prerequisites, since those are the same binaries preflight resolves. HS-157a..h pin that a row name neither manifest declares, or an empty one, is refused in every mode, before any row runs, and that declared screenshot and functional names, and a platform's absent rows, keep their treatment. Other rows assert that the lints are still wired in at all: that make unit-test reaches the assertion lint and reaches it first, and that an offending row script fails the whole regression preflight rather than just the lint. The rest cover cleanup, checking that every SD-clone cleanup script handles INT and TERM as well as EXIT, that each handler exits rather than resuming, and that the five real cleanup bodies are bounded and remove only their own run directory or clone file. Finally HS-50..HS-55 pin the regression suite's host-load bookkeeping: the real suite library, sourced against a fake /proc/loadavg, must flag a FAIL on a loaded host and must not flag the same FAIL on an idle one, must name every failed row at the end, must never change the count — and the driver must still call it. HS-68a..HS-68e pin the sourced-row counter guard: a stub row that clobbers skip after an earlier SKIP is a harness fault naming that row, a row reporting two results is one too, the NULL control and a filtered-out row pass, and the row runner wraps every functional row in the guard. HS-69a..HS-69o drive the parallel scheduler on stub rows: rows run at once and merge in declared order; a row that reports twice or nothing is a harness fault; one that dies or hangs is a FAIL naming it while the others run on; one that writes into another row's directory, replaces the library's trap, or writes to the shared SD clone without private-sd is refused — and the tagged control is clean; the heartbeat names a running row; a SKIP reaches the driver by name; only loaded-host functional FAILs are solo-confirmable; and a TERM to the driver stops every row process and removes every directory. HS-70a..HS-70e drive regression.sh itself through the host lock (a second full run waits, the wait is bounded, a targeted run never locks, a TERM to the outer script alone frees the lock) and the loaded-host wait; HS-71a..HS-71p run the real stamp script in a throwaway repository laid out with the real paths: a clean green run stamps, a docs-only commit keeps the stamp, a tree dirty at the start or at the end alone has none, a SKIP or FAIL is never stamped, every non-doc class of path (Makefile, CMake, a .conf, tools/, .github/, a gitlink) moves the key while documentation does not, the documentation a gate reads does, and the pending / solo-confirm path stamps only when every pending row passes. HS-72 bans xvfb-run -a/-n from the test scripts. HS-89 requires -noreset on every xvfb-run line of a script that drives the app with xdotool. HS-76a/b pin that a unit suite reporting a SKIP fails the run (named, with its SKIP lines, beside a failing one too) and HS-77a..d that the regression driver and a standalone row do the same in every mode, with the control that a skip-free run still exits 0 and stamps. HS-78 runs the ffmpeg row with no ffmpeg on the host and wants a SKIP, HS-79 the real sync-version-test.sh with rpmspec hidden, HS-80a..h the tool-missing and renderer-version-gap branches of the docs checks and the two snapshot verifiers (a PATH of symlinks that lacks the tool, or a fake renderer fingerprint), and HS-81a/b that a set-but-unusable Z88DK_GDB is an error. HS-82..HS-88d pin the target OS: an os-restricted suite registered on an OS that excludes it, or missing from one that owns it, is a refusal; an unknown # os: value or target OS, or an os-restricted suite without the ?, likewise; a gated-out suite is a NOTICE naming the OS; a Windows build's NAME.exe registration matches the manifest and the runner prefix reaches the .exe; a bash older than 4, a host without GNU timeout or without perl is a refusal; and JNEXT_UNIT_TEST_JOBS serialises two lock-holding suites (with the uncapped control that they do overlap). The packaging self-test's PS-05 pins that a contract sub-test printing SKIP and exiting 0 is a SKIP row. HS-73 runs the real confirm path in a throwaway repository whose one row invokes the harness again, as harness-selftest-func does, and requires it to complete and stamp: a nested run never waits for the lock its own ancestor holds. HS-74a/HS-74b prove the marker that says so is evidence, not a password: a hand-exported one, one naming a live ancestor that does not hold the lock, a stale one, and one naming the live holder that is not an ancestor all leave the run a normal second run that waits and faults at its bound; HS-75a/HS-75b ask for a stamp from inside the real suite library — which points XDG_CONFIG_HOME at a scratch directory for Qt — with an untracked file ignored only by a fake global git ignore (it must not count as dirty) and one nobody ignores (it must); HS-74c repeats HS-73 with the lock file behind a symlink, which flock(1) resolves and the holder check must too.

It pins its own count

EXPECTED_TOTAL sits in the script, right next to the rows it counts, and running a different number of checks is exit 2 with an explicit refusal message. The reasoning is the project's usual one: without the pin, deleting a check shrinks the declared side and the reported side in lockstep, which is precisely the silent-truncation move that the harnesses this file guards were built to forbid. Adding or removing a check means editing that number, deliberately.

The count could not simply live in test/unit-tests.conf. That manifest is cross-checked against CMake's add_test() registrations in both directions, so an entry with no CMake counterpart would make the unit harness refuse to run.

The traceability self-tests

$ make traceability-selftest          # citation extractor + dup-ID gate, 239 pinned rows
$ make traceability-accounting-check  # the suite-accounting gate, ~0.01 s

test/traceability-citations-selftest.pl pins the tool that decides which VHDL lines justify each traceability row. Its end-to-end rows build a throwaway repository out of the real manifest, CMakeLists and matrix, populated with stub sources and binaries, and then run the real refresh script against it twice, so that idempotence and the refusal paths are both exercised. SELF-208..SELF-215 run test/traceability-dup-ids.pl and the generator's --planned-ids the same way, planting colliding IDs in ?-prefixed suites, in one-row plan docs and in the baseline. It pins $EXPECTED_ROWS = 239 in the same shape as the harness self-test, and for the same reason it cannot live in the unit manifest — it is a perl script with no CMake target.

Both are prerequisites of make unit-test, and that matters more than the checks themselves do. Until they were wired in, nothing invoked them — not unit-test, not regression, not CI — and they were the only thing standing between the project and a plausible-but-wrong VHDL citation. A guard nobody runs is worse than no guard, because it looks like coverage. The same lesson produced regression-doc-check, which wired up a screenshot-documentation checker that had existed, had been correct, and had been invoked by nothing for long enough to rot into failing on 18 rows.

The lint self-tests

Six of the lints verify themselves on every invocation, before their verdict on the real tree is trusted at all:

• test/lint-assertions.sh writes a fixture containing one instance of each banned tautology — check(id, desc, true), || true, and a == b || a != b — and refuses with exit 2 if any of its patterns fails to match that fixture. A lint that has silently become a no-op reports success, which is worse than not running it.
• test/lint-thread-detach.sh writes a fixture with five must-flag lines (.detach(), ->detach(), pthread_detach(, and a spaced form) and a fixture of must-not-flag comment and identifier lines, and refuses with exit 2 if either count is off. The harness self-test also runs it against a throwaway tree to prove it fails on a stray .detach().
• test/lint-ascii-text.sh embeds a perl lexer for string and character literals and checks it against one fixture file per case, each named in the script by its ID (LAT-A01 and on). The em dash must be flagged in a plain, a u8, an L, a raw (multi-line) and an escaped literal (a u or U escape, \xE2\x80\x94, \342\200\224), in a string or character literal spliced over a line end by a backslash (LF or CRLF), and at the line the literal starts on even after a block comment and a raw string, and a declaration must not excuse it. It must not be flagged in a // or /* */ comment (a // comment spliced by a trailing backslash included), in a raw string that merely spells the escape, behind an escaped backslash, or next to a digit separator such as 28'000'000. Other non-ASCII text is flagged as raw UTF-8 and as an escape, an invalid UTF-8 byte included, but not as \x byte escapes (binary payloads) and not under src/esp01/test/ itself (src/esp01/src/ and a sibling such as src/esp01/testing/ are flagged); it is excused by exactly the declared (path, literal) pair, a declaration that matches nothing fails even when it is the only finding, and a malformed conf exits 2. A miscount refuses with exit 2 and names the fixture. The harness self-test then runs the real lint against throwaway trees (HS-97a to HS-97h): wired to unit-test and regression, red on a raw and on an escaped em dash, green when the only em dashes are in comments, red on an arrow, green once declared, red on a stale declaration, and the src/esp01/test/ boundary between the two rules.
• test/00regression/lint-traps.sh carries 86 pinned cases: 57 that must flag and 29 that must not. There is one fixture file per case, deliberately, because a combined fixture still passed when a case was lost, whereas with one file per case the lost case is named. It also cross-checks its own prose case table against the fixture files that exist, in both directions, so that the documentation and the fixtures cannot drift apart.
• test/lint-timeouts.sh carries 64 pinned cases: 33 that must flag and 31 that must not, in the same one-file-per-case shape and with the same table-versus-fixtures cross-check. Its rules were mutation-tested one at a time; the run killed 22 of 23 mutations and found a real hole — reverting the comment stripper changed nothing, because every comment fixture was inert for a second reason as well. The two cases that close it exist because of the mutation, not because anyone thought of them first.
• test/lint-pipe-grepq.sh carries 42 pinned cases: 25 that must flag, 15 that must not, and 2 that pin its coverage check. They include both scope rules — the same echo "$o" | grep -q line is flagged in a file that sets pipefail (anywhere in the file, including after the line), flagged in a file under test/00regression/ with no pipefail line (the harness sources it into its own), and not flagged elsewhere — the quote rule (quote state is reset at every line end, so a stray apostrophe cannot hide what follows), and the heredoc rule: a body is scanned like code and a fixture line is marked, because an earlier version that guessed where heredocs end hid 555 lines. The lint also counts the lines that reached its matcher and refuses the run unless that is every line of every in-scope file.

The developer-guide cite check, test/lint-devguide-cites.pl (see 4.5), is not self-checking like those: the harness self-test runs it against throwaway trees (HS-102 to HS-156), and each row was shown to go red when the behaviour it pins is mutated out of the script. HS-102 pins the command in the unit-test and regression dry runs, not the target name. The rows cover: an exact cite and a declared external one passing; a cite one line off, an unanchored one (also when the symbol is in the previous sentence), a missing file, a line past the end, an ambiguous bare name, a comma list and a cite of a file in no set failing; the whole-word rule; a test ID as an anchor; a range holding its symbol inside it and not one line outside; a .dot page and its // comment grouping; every prose form; a bare :N binding to the last of two files, ignoring a link target, failing when bound to an example name or an external file, and failing across paragraphs while a whole table is one; the bare example-name exemption against the same name with a directory; the repository's z88dk/ fixtures and a repo file named like an external one being checked; any extension with a directory, Makefile and a colon outside the span; excerpts (a match across blocks, a changed row, a wrong row count, a numbered row, ambiguous or missing file, the <TAB> convention); a lines phrase, bare :N or excerpt pointer after a file that does not exist; a table row ending the sentence; a bare comma list; the sjasmplus manual as a declared external file; and fences: unclosed, closed by the other character, shorter than its opening, and ~~~; and the parenthesis binding (a parenthesised cite after text that is not its symbol, a bare :N continuation without a symbol, a coincidental match elsewhere in the sentence, a symbol right before the parenthesis, a symbol inside the parenthesis winning over it, the innermost parenthesis being the scope, a closed parenthesis ending its scope, a cite span in the parenthesis not counting as its symbol, a test ID there counting as one, a closed parenthesis not affecting a later cite) and a quoted string literal anchoring a cite only when it is on the line (case-sensitive, at least 4 characters with a letter or digit, also on the last line of a range), and JNEXT_CITES_DUMP writing the checked cites, and the pinned foreign tools being declared external and the .dot comment-block grouping.

Both directions matter, and not equally. A false negative merely fails to help, whereas a false positive blocks a correct row — and that is the one that costs.