JNEXT
Skip to content

4.7 Writing a new test

The authority for this is doc/testing/UNIT-TEST-PLAN-EXECUTION.md, and you should read it before authoring or rewriting any subsystem test plan, or implementing one of its planned rows. It documents the VHDL-as-oracle rule, the SKIP-is-a-failure rule, the 1:1:1 emulator-fix-plus-row process, the mandatory independent review, and the coverage-theatre audit that all of it exists to prevent. This page is the practical walkthrough alongside it, not a replacement for it.

The rule everything else serves

A test derives from the specification, never from the existing C++. That is the single most important idea in this chapter, and it is worth stating as an idea before it turns into a procedure. The specification is the VHDL at cores/zxnext/src/ in the FPGA repository — it is what the hardware does, and therefore what the emulator is supposed to do. The emulator's current behaviour is not evidence of anything; it is the thing under test. Read src/ only to discover the public API surface you will drive.

If an assertion is written by observing what the emulator does today and writing that down, it will pass forever and prove nothing, and it will do so while adding a row to the coverage count. That is what makes the mistake expensive: it produces something that looks exactly like a test.

The corollary follows directly. A failing unit test does not mean the test is wrong. The default assumption is the other way round — that the emulator is wrong and the test is a correct reading of the VHDL — and only an independent review that compares the assertion, the citation and the C++ may change a test. When you read the VHDL, read the surrounding process and not just the cited line: an oracle can cite a perfectly real line that is nevertheless unreachable under the stimulus your test applies. That is how several "confirmed emulator bugs" each survived four reviews before anyone noticed.

Where suites live

There is one directory per subsystem under test/ — test/copper/copper_test.cpp, test/mmu/mmu_test.cpp, and so on. Several subsystems also have a companion *_integration_test that drives the same area through a more fully assembled machine. The two ESP-01 suites are the exception to the layout: they live under src/esp01/test/ and are declared by that module's own CMakeLists.

Portable suites

The SDL-only suites run on three operating systems: natively on Linux and macOS, and as MinGW-built Windows executables under wine (make unit-test-win). So a suite that is not # os: posix must build and pass on all three, which in practice means a handful of rules:

  • Do not call setenv, unsetenv, mkdtemp, localtime_r, mkdir(path, mode) or geteuid directly. test/test_portable.h has jtp::set_env, jtp::unset_env, jtp::make_temp_dir, jtp::local_time, jtp::make_dir, jtp::running_as_root and jtp::process_id_string. Only scaffolding belongs there: a call the product makes gets a platform arm in the product.
  • Do not write "/tmp/..." as a place to create things. Use std::filesystem::temp_directory_path() (or jtp::make_temp_dir); /tmp resolves under wine only because Z:\tmp happens to map to it.
  • Compare paths as std::filesystem::path, not as strings built with /: the product joins with the host's own separator, and weakly_canonical / absolute add a drive on Windows.
  • Open the row-ID file (and any file whose bytes you compare) in binary mode: test/row_id.h uses "ab", because text mode writes \r\n on Windows and every ID would then differ from its source literal.
  • To run the real binary with a bound, use jtp::run_bounded in test/test_spawn.h, not std::system("timeout ..."); a stand-in program a test needs (the ffmpeg stub) is a small executable built by CMake, not a shell script, because Windows runs .exe files only.
  • A test whose fixture needs fork, FIFOs, ptys, symbolic links or chmod 0 belongs in a suite the manifest declares # os: posix, with the rows moved there unchanged; see 4.2.

A row that needs a write to FAIL (a full disk) uses jtp::FullDisk: /dev/full where it exists, else, on macOS, a file whose writes a zero file-size limit denies for the object's lifetime -- so let it die before the row reports. A fixture that must not depend on the host's case sensitivity cannot assume it: two names that differ only in case cannot coexist on APFS or NTFS, which is why such a row is # os: linux. Scratch files belong in a directory of the process's own, never at a fixed name: the unit harness runs every suite at once, and a second worktree's gate may run the same suite on the same host.

What wine cannot stand in for stays untested until a native Windows leg exists: symlink creation (so the symlink-refusal rows are posix), a real console attach for a GUI-subsystem executable, NTFS short names and streams, and the timestamp resolution of a real filesystem (wine keeps whole seconds, so a fixture mtime is set 250 ms past the second, never exactly on it). Wine's Winsock can also return a peer reset as a 0-byte read followed by WSAECONNRESET (a race in its server), so the Windows socket shim confirms a 0-byte stream read with a MSG_PEEK recv before calling it an orderly close.

Portable regression rows

A functional row also runs on macOS and, under wine, on Windows (make regression-macos, make regression-win) unless its os= tag in functional_tests.conf says otherwise. So a row that is not tagged must:

  • run the binary as "$JNEXT" (Qt) or "$JNEXT_SDL" (SDL-only), never a path of its own, and the native helpers as "$TEST_BIN_DIR/<tool>";
  • not read /proc, /dev/full, /sys or assume a process tree (a wrapper sits between the row and jnext on Windows); where it must differ by OS it reads $JNEXT_TARGET_OS (linux by default) and keeps the Linux assertion intact;
  • use QT_QPA_PLATFORM=offscreen and SDL_VIDEODRIVER=dummy, not Xvfb, unless the row is about X11;
  • not depend on CRLF or on seeing the exact stdout bytes jnext writes on Windows (the runner folds CRLF); a file jnext writes must be written in binary mode;
  • not use a pty, a FIFO, ulimit or ffmpeg unless it is tagged os=linux or os=linux,macos with the reason above it.

The row idiom

Suites are standalone binaries with a small local harness — there is no GoogleTest anywhere in the project. test/copper/copper_test.cpp is the canonical reference, and the two functions that matter are:

void check(const char* id, const char* desc, bool cond, const std::string& detail = {});
void skip (const char* id, const char* reason);

check() bumps the totals and, on failure, prints FAIL <id>: <desc>, which is the one output format every tool in the project relies on. Both functions start with report_row_id(id) (from test/row_id.h), and so must any helper of yours that counts a row: the unit harness fails a suite whose reported IDs do not number exactly its Total:. skip() records the row and prints it in the trailing summary without touching the pass/fail counters — and the unit harness fails the run on a suite that reports one. Every suite ends with the line the unit harness parses:

Total:   82  Passed:   82  Failed:    0  Skipped:    0

Four rules govern the ID:

  • It must be a string literal. An ID assembled at run time is invisible to every reader of the source and vanishes from the traceability matrix. The unit harness fails a suite that reports an ID not found verbatim as a literal in its sources; a loop over cases takes each case's ID from its case table.
  • It must be globally unique. traceability-dup-ids.pl refuses when two suites assert the same ID, or when it names a planned row in another subsystem's plan doc; the unit harness refuses a suite that reports the same ID twice in one run. A loop that asserts one row per case gives each case its own literal ID in the case table.
  • The description is what the matrix publishes to everyone else, so write one worth reading.
  • Cite the VHDL in the same call, with file and line range. A citation written only in the plan doc still works, but the row-local one is what gets validated against the real FPGA tree, and a disagreement between the two is reported for a human to resolve.

A row whose facility does not exist in src/, or is not observable through the public API, is PLANNED in its *-TEST-PLAN-DESIGN.md (missing in the matrix), never a skip() in the suite: a SKIP fails the run. skip() is only the environment fallback arm of a row that also has a check() (a capset the sandbox refuses; SD-28 builds its premise as root by dropping the DAC capabilities, test/dac_caps_dropped.h, rather than skipping it). It never means that the assertion is awkward to write. Writing check(x, true, ...) as a placeholder is banned outright, because it pollutes the pass count and can end up pinning a wrong value once the facility does land. The assertion lint rejects it, along with || true and a == b || a != b; it matches raw text, so those substrings must not appear in comments either.

Registering and pinning it

In test/CMakeLists.txt:

add_executable(mysub_test mysub/mysub_test.cpp)
target_link_libraries(mysub_test PRIVATE jnext_peripheral jnext_port)
target_include_directories(mysub_test PRIVATE ${CMAKE_SOURCE_DIR}/src)
add_test(NAME mysub_tests COMMAND mysub_test)

Link the narrowest set of libraries that works. By convention the add_test() name is the binary name plus an s, but nothing depends on that — the harness reads the binary path, not the test name.

Then add the suite to test/unit-tests.conf with its exact row total, and bump that file's # expect: N suite pin. Registering it in CMake without declaring it in the manifest, or the reverse, makes the harness refuse to run; 4.2 lists every refusal and failure condition. Adding or removing a row later means editing that number by hand, and since the harness fails in both directions you cannot quietly skip that step.

If the subsystem has a *-TEST-PLAN-DESIGN.md, planned-but-unimplemented rows belong there. That is where the traceability generator reads them from, and they emit as missing — an honest backlog rather than an invisible one.

Before anyone reviews it: mutation-test your own rows

Break the thing your row protects, rebuild, and confirm that the row fails. Then revert the mutation from a backup copy of the file, never with git checkout, which is how a real fix gets thrown away along with the mutation. Mutate the behavioural branch you actually added rather than a shared helper: reverting a helper turns everything red at once and proves nothing about your row in particular.

A row that cannot be made to fail is not a test. This is not a formality — the project's own history includes six subsystem plans reporting "100% passing" while carrying tautologies, anti-tests that pinned wrong behaviour as the expected result, and 34 plan rows silently dropped from a 95-row plan.

Know what it cannot show you, though. Breaking one rule at a time regresses the rules you wrote; it can never surface an interaction between two rules that are each correct and were never considered together. lint-timeouts.sh shipped its review round with exactly that defect — continuation lines joined before comments were stripped — and every one of its 53 cases passed, because continuation had only ever been tested without a comment and comments only ever without a continuation. So when a green mutation run tempts you to stop, go looking for the pair of rules nobody has written a case for.

Then: never review your own work

Every change gets an independent review by someone — or some agent — that did not write it, working in its own worktree, and the verdict is binary: APPROVE or REJECT. The reviewer samples VHDL citations at random and traces the stimulus through the cited process, classifies every failure into one of four buckets (harness bug, wrong expected value, real emulator bug, plan bug), checks that each skip() is an environment fallback beside a real check(), and counts rows against the plan. Missing rows are the canonical failure mode, which is why the count is checked rather than assumed.

Running it

$ make unit-test-build && ./build/test/mysub_test
$ make unit-test

make unit-test regenerates the traceability matrix and fails if the committed copy is stale, so commit the regenerated file together with your change. If the change touches rendering or CLI behaviour, run the full regression as well — see 4.1.