Build the Fixtures Before the Tools: A Testing Methodology for Container Scanners

Vendor comparison tables tell you what tools claim. If you want to know what they do, you need the same inputs going into every tool, and you need to know the right answer in advance. That means fixtures.

Our POC evaluated 13 candidate tools against five purpose-built images. The fixtures cost half a day to build and were the reason the comparison produced defensible conclusions instead of anecdotes.

The fixture set

Five purpose-built fixtures, pinned by digest — every tool saw the same inputs.

Design principles worth copying:

Control images matter as much as vulnerable ones. tc03 is nearly empty. A scanner that reports findings against it is inventing them — and one candidate did, 27 false positives on a Go binary, which changed its role in our stack from gate to periodic cross-check.

Plant the secret, then delete it. Layer history preserves deleted files. The AWS documentation example credentials written to /tmp and rm'd in the next layer are not live keys, but a scanner that misses them will miss real ones.

FROM alpine:3.18
RUN echo "AKIAIOSFODNN7EXAMPLE" > /tmp/aws_key && \
    echo "aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" >> /tmp/aws_key
RUN rm /tmp/aws_key

Give the app a health endpoint. The single most important test in the whole POC was: does the patched image still work? You can only answer that if the fixture exposes something you can curl.

app.get('/health', (_, res) => res.send('ok'));

Pin by digest. docker images --digests | tee evidence/fixture-digests.txt before anything runs. A tool scanned against a different image than its competitor produces a number, not a comparison.

Establish ground truth before scoring accuracy

Before judging any tool's findings, check a sample of reported CVEs against the distro's own security tracker (Debian, Red Hat, Alpine) and record whether each is genuinely open, back-ported, or will-not-fix. Every tool is then scored against this, not against each other. Distro trackers routinely disagree with NVD; without ground truth you cannot tell measurement from noise.

The negative result that became a finding

We originally wanted debian:11-slim and ubuntu:20.04 as fixtures — old enough to be vulnerable, familiar enough to be realistic. Neither could be built on: their apt repositories are archived now the releases are EOL.

That failure is itself one of the most useful findings of the POC. An EOL base is unpatchable by anything, and it should be detected, not discovered. Trivy now runs with --exit-on-eol 1 in CI, so the pipeline fails early with a message about the base rather than a wall of unfixable CVEs later.

Two ground rules that kept the POC clean

1. Never use production images as test targets. Synthetic fixtures with known contents give you ground truth; production images give you surprises — and on a shared cluster, risks. (We needed two corrections from a reviewer before this rule fully sank in. It is now rule zero.)
2. Contain everything. A dedicated namespace, newly built images, nothing cluster-scoped, and written safety rules for anyone (or any agent) touching the environment.

Fixtures first, tools second, scores last. Everything else is marketing.

Series: Building a Container Image Security Stack. Next: what Copacetic can and cannot patch — 105 CVEs to zero in 40 seconds, and the nine packages it structurally cannot touch.