One failure, found twenty-six times, in subsystems that share
no code and were written weeks apart by people solving unrelated problems:
None of these were caught by review. They were caught by feeding the
instruments deliberately stupid inputs and requiring a refusal, by running the fix's own
test against the pre-fix tree first, and by a person listening to a mix that every gate
had passed. A selection follows; the full catalogue is grouped by mechanism, because the
mechanism is the part that transfers.
The instrument's own arithmetic destroyed the reading
A limiter reporting 0.124 dB while removing about 22
The build report said the loudness limiter had turned things down by at most
0.124 dB — the signature of a mix that needs no help. Recovering the true gain
curve, by dividing the delivered file by the sum of its own fourteen component
tracks, showed a 32.2 dB swing.
The cause is four lines: the limiter ran up to eight times in a loop and the
variable holding “the worst we saw” was reassigned every pass rather than
reduced with min. The per-pass reductions were −19.93, −3.89, −2.20,
−1.13, −0.63, −0.40, −0.22, −0.12 dB. The report published the
eighth. Every iterative refinement loop has this shape — the last iteration
is by construction the one with the least left to do — and the direction of the bug
is always flattering.
The most expensive consequence was not the audio. An earlier diagnosis read that
−0.124, declared the limiter “REFUTED, clean”, and moved on. That
refutation was carried forward into weeks of subsequent work before anyone retracted it.
BROKEN-INSTRUMENTS.md §IV.1 · R2-4031, R2-4037
The metric's best score was a degenerate case
A quality gate whose maximum score was silence
The gate asked “does this passage contain events, or is it a stationary wash?”
Its statistic was the span between the loud moments and the quiet moments. Read that
definition twice: the loud moments are the impacts, the quiet moments are whatever
lies between them, so the cheapest way to maximise the score is to put
nothing between them.
Same 777 impacts, only the material between them varied
| the passage is impacts plus… | gate score | audibility |
| nothing (what shipped) | 27.17 dB | −140.71 dB |
| an audible machine-room ambience | 12.62 — FAIL | +14.56 dB |
| a hair dryer | 5.48 | audible |
| a drone | 0.26 | audible |
The bar was 13.7. Silence scored 27.17 — the best in the table — and a genuinely
audible machine failed. A tuning loop had been walking a gain parameter downhill
toward a better score, and it arrived: the delivered passage measured
26.4 dB SPL at domestic playback level, with 0 of 29 third-octave bands
above the threshold of hearing. A living room's own noise floor is about
15 dB louder than the entire passage.
Every other gate in the file agreed it was fine, and the reason generalises past
audio: every one of them was relative. Ratios, spans, correlations,
fractions — not one absolute quantity anywhere. Digital silence has excellent ratios.
It was caught by a person, who wrote: “now beat 1 i dont hear anything until the
tubes play.” The threshold was not moved. The honest finding was that trough
depth is not eventfulness — the metric was right and the inference drawn from it
was not.
BROKEN-INSTRUMENTS.md §II.1 · R2-4147
Calibrated against the artefact it judges
“The limit is the midpoint between what THIS master reads and what the adversary reads”
That rule set every threshold in the audio suite, and it is not lazy work — the
adversary was a synthesised hair dryer plus two masters the client had already
rejected, the placements were bootstrapped with real standard errors, and the
resulting gate correctly failed both rejected masters and white noise on all ten
limbs. It looks like a properly validated instrument.
It is still self-referential, and the consequence is exact: the pass mark is
a function of the artefact under test, so a film can only fail by being worse than
the film that calibrated the limits. What that gate could detect was
regression. What everyone believed it detected was quality. Three rebuilds shipped
and were rejected under it.
The fix is the most directly reusable thing in the corpus: every threshold is now a
frozen record carrying a provenance tag — physics,
published or control-derived — and an audit rejects
source=artefact by name before any gate is allowed to run. A second rule
fires alongside it: a threshold with no derivation note is itself a violation,
because a bare number is not a threshold.
BROKEN-INSTRUMENTS.md §III.1 · R2-2222, banned at R2-4041
The instrument never opened the artefact
Eight gates, and a two-second tape loop passed all of them
Three successive soundtracks were rejected by the client. All three had passed all
eight automated gates. So an audit took the delivered master, replaced the 33 seconds
the client was complaining about with a single two-second block tiled 16.5
times, renormalised the loudness, and ran the suite unmodified. Eight green
lights, ALL_PASS = True, exit 0.
The gate meant to judge tonal quality rated the tape loop 35.9 dB better
than the film it had just passed. Three of the eight never opened the audio
file at all — one was a static scan of the source tree, one re-synthesised a clean
engine tone from telemetry and measured that, one root-solved constants out
of a source file. All three pass on white noise. Of the remaining five, one judged
2 frames out of 2,978 and one judged 20 samples out of 5,956,000.
Count what fraction of the artefact your checks actually touch, in the artefact's
own units. “Eight gates” sounds like coverage. And a check that reads the source tree
is a provenance check, not a quality one.
BROKEN-INSTRUMENTS.md §I.1 · R2-4039
The instrument had no case to answer
A verification that could not fail
The shortest entry in the catalogue and the purest specimen in it:
(ob.matrix_world.translation − ob.matrix_world.translation).length > 1e9
A value minus itself — identically zero — compared against a number nothing reaches.
It printed a reassuring 0 stragglers and proved precisely nothing. The
author's own note records that two independent audits had, that same night,
flagged “verification theatre” as a bug class in its own right. “I then wrote
one.”
Its siblings: two gates printing green having tested zero of zero; a cache eviction
bound that had never once executed in production; and a ship-or-don't-ship harness
that claimed a long list of assertions and could act on 24 of them
— the rest printed and were judged by nobody. Among the silent ones was the suite's
only negative control, whose own header read “if this ever comes back PASS the
instrument is broken and every PASS above it is vacuous — keep it”. It was kept.
It was piped into tail -12 for four film generations and its verdict
went in the bin. The ship candidate flipped from PASS to FAIL the moment they were
counted.
BROKEN-INSTRUMENTS.md §V.1, §V.2, §V.3, §V.4 · R2-012, R2-018, R2-2824
The instrument measured the wrong quantity
A number that did not move when the thing it names was rebuilt twice
Someone claimed two sections of the film were defective, citing a harmonic quality
metric. Every number in the claim was correct. The conclusion was not, and one table
settles it — the same metric across four masters, with two complete rebuilds of the
engine synthesiser in between:
Metric movement across two engine rebuilds
| section | 2 Aug | rebuild 1 | rebuild 2 | now | moved |
| flying lap | −0.71 | −0.72 | +6.68 | +5.84 | 7.40 |
| launch | +1.36 | +1.27 | +5.63 | +8.06 | 6.79 |
| transit | −0.55 | −0.59 | +3.57 | +3.91 | 4.51 |
| the ending | +0.40 | +0.40 | +0.35 | +0.13 | 0.27 |
| the glass breach | +1.49 | +1.45 | +0.08 | +0.05 | 1.44 |
A number that does not respond to two rebuilds of the engine is not measuring
the engine. Established independently afterwards: in the breach it was
computing a ratio on 0.0183 % of the section's energy, because breaking glass is
broadband on purpose; in the ending the band being scored was 86 % crowd noise and
0.93 % engine. It was measuring a grandstand, and measuring it correctly.
A time series of your metric across releases is a free and extremely strong test of
whether it measures what you think.
BROKEN-INSTRUMENTS.md §VII.2 · R2-2223, R2-2224
The thread that does not hold
“They were sloppy” does not survive the evidence
The people who wrote these instruments also wrote the docstring naming the exact
failure mode, the comment explaining precisely why the heavy step needed a lock, the
header stating that a passing negative control invalidates every result above it, and
the note that “a detection that does not reach an exit code is a rumour” — on
the tool that had never once reached an exit code.
The knowledge was present and written down, adjacent to the defect, in the defect's
own words. What was missing was a mechanism that could act on it. Prose is not
a mechanism. The fixes that stuck all converted a written intention into
something that executes: a provenance tag the build rejects, a third verdict value the
aggregator counts as a failure, a self-test that perturbs each check and requires
exactly one failure.
Which is why PASS/FAIL was replaced with three verdicts
everywhere. Two-valued reporting has nowhere to put “I could not measure this”, and
every project that lacks the third value discovers that unmeasured silently means
fine. A check that cannot be evaluated must never be indistinguishable from
one that passed.
BROKEN-INSTRUMENTS.md §Synthesis, §What actually caught these