Pseudo-Tested

Slug	Severity	Detection Scope	Protects
pseudo-tested	High	per-test	Necessary, Granular

Summary

Replace the SUT body with return; / return null / return input — every test still passes. The SUT runs, the test asserts, but the assertion could not tell a working implementation from a no-op one.

Aliases

• "pseudo-tested"
• "extreme mutation survives"
• "no-op mutant survives"
• "structural-shape oracle"
• "non-empty as the only check"
• "would pass if SUT body were deleted"
• "Niedermayr-style"

Description

This is the extreme-mutation / pseudo-tested-methods smell: methods whose bodies can be replaced with a no-op without any test noticing. Niedermayr et al.¹ measured a median of 10.1% of methods in studied Java suites being pseudo-tested — a cheap, high-signal mutation class (often run via Descartes on JVM, mutmut --simple-mutations on Python, cargo-mutants on Rust, or Stryker's block-statement mutator on JS/TS).

This entry is the mutation-adjacent sibling of vacuous-assertion. The Tautology Theatre operational test — "would this test still pass if all production code were deleted?" — has pseudo-tested as a strict subset ("would it still pass if production code were no-opped?").

A reader can conjecture without running a mutator: "if short_digest returned '', would any existing test fail?" Read the suite, answer yes/no, and add the missing assertion. A real mutation run later confirms.

Signals

• Test body calls SUT, assigns to _, then runs only structural shape / type / length checks.
• Assertions after the SUT call check only "non-empty" or "has this key".
• SUT return value is never compared to an expected value.
• A file-touching SUT is tested by checking a file exists, not its content.
• Confirmatory cross-check: run Descartes / mutmut --simple-mutations / equivalent; any surviving no-op mutant is evidence for this mode.

False-positive guards

The no-op-survival signal over-triggers in two cases:

• Side-effect contracts observed via a non-return channel. A SUT whose entire contract is to perform a side effect — write a file, emit a log line, publish an event, mutate a database row — has no return path for the obvious return-value mutator to expose. A test that calls the SUT and then asserts on the side effect via the appropriate channel (file content, captured log, event store, database query) is not pseudo-tested even though return null substituted in the SUT body would survive a return-value mutator. Before flagging on null-op survival, ask whether the SUT's contract is its return value or its side effect, and confirm the test asserts on whichever is the contract. The relevant mutant for a side-effect SUT is on the side-effect call, not the return path.
• Conjecture vs verified mutation. Reader-side conjecture ("if short_digest returned '', would any test fail?") is cheap and useful, but it is not a verdict. A real mutation tool runs many no-op shapes (return null, return input, return zero, return ""); an assertion that catches some shapes but not the simplest constant-replacement still constitutes meaningful coverage. Before flagging on conjecture alone, consider multiple plausible no-op replacements; if at least one would plausibly fail an existing assertion, downgrade to INVESTIGATE rather than flag, and recommend a real mutation run as the verifier. The verified pseudo-tested verdict requires a tool run; the conjectural verdict is a hypothesis.

Prescribed Fix

1. For the canonical test in the cluster, identify the SUT's actual output contract.
2. Add the one assertion that would fail under return default: compare to an expected value, parse the output and assert on its shape, hash the file content, etc.
3. Keep the fix local — one well-placed assertion in the canonical test, not N across N tests.
4. Gate: preservation of regression-detection power, stricter than the default — at least one previously-surviving no-op mutant now dies. Mutation kill-set delta must be strictly positive.

Example

Before

def test_corrupt_tracking_db_graceful_skip(tmp_path):
    corrupt_db = tmp_path / "tracking.db"
    corrupt_db.write_bytes(b"garbage")
    sync_tracking_db(corrupt_db)  # should skip without raising

sync_tracking_db could be def sync_tracking_db(_): pass and this test still passes. The "graceful skip" contract is asserted only by the absence of an exception.

After

def test_corrupt_tracking_db_logged_and_marked_skipped(tmp_path, caplog):
    corrupt_db = tmp_path / "tracking.db"
    corrupt_db.write_bytes(b"garbage")
    result = sync_tracking_db(corrupt_db)
    assert result.status == "skipped"
    assert result.reason == "corrupt_db"
    assert any("tracking.db" in rec.message for rec in caplog.records)

Now pass fails (result is None; attribute access raises). A no-op mutation dies. The "graceful skip" contract is now encoded as a positive return plus a log line.

Related modes

• vacuous-assertion — same family; asserts something, but insufficient.
• tautology-theatre — SUT never runs at all.
• rotten-green — SUT call with zero assertions; hard to distinguish from pseudo-tested without describe-before-edit.

Polyglot notes

Every ecosystem has at least one extreme-mutation driver:

• JVM: Descartes engine for PIT.
• Python: mutmut --simple-mutations, Cosmic Ray.
• Rust: cargo-mutants.
• JS/TS/.NET: Stryker's block-statement mutator.
• Go: go-mutesting, go-gremlins.

A reader's cheap conjecture pass works without any of them; full confirmation benefits from the tool.

---

1. Niedermayr, R., Juergens, E., & Wagner, S. (2019). Will my tests tell me if I break this code? Empirical Software Engineering, 24(6), 4085–4130. https://link.springer.com/article/10.1007/s10664-018-9653-2. The 10.1% pseudo-tested median appears in Table 3 of the paper. ↩