105 CVEs to Zero in 40 Seconds: What Copacetic Actually Does to an Image

Copacetic (copa) is the strongest result to come out of our container image security POC, and the easiest to oversell. Here are the numbers, the mechanism, and the hard limit we proved rather than assumed.

The result

Copa's own log reported 105 total, 105 patched, 0 skipped. Both figures were re-verified by rescanning after the run.

The decisive test was not the CVE count, though. It was whether the patched application still ran: the fixture had passed a baseline smoke test before patching (so any later failure could only be attributed to the patch), and after patching it started, served traffic, and returned ok from GET /health. Image config was preserved byte-for-byte — entrypoint, cmd, env, user, workdir — with exactly one addition: a BaseImage provenance label.

How it works

The same run, both halves: OS packages 105 → 0; npm dependencies untouched.

Copa does not rebuild your image. It:

1. Parses the fixable package list from a scanner report (Trivy's JSON is the common input).
2. Runs the distro's own package manager inside the image via BuildKit to apply the published updates.
3. Appends a patch layer. Layer caches and downstream pulls stay intact; the original tag is untouched.

That last point is why unattended patching is even possible. A rebuild changes layer hashes and invalidates caches; a patch layer does not.

The hard limit: OS packages only

After patching the Node fixture, zero OS CVEs remained — and these were all still vulnerable:

express (2), lodash (5), body-parser (2), brace-expansion (5), cookie, cross-spawn, glob, ip-address, diff, @sigstore/core.

This is structural, not a bug. Copa upgrades packages by running the image's package manager — apt, apk, and friends. npm, pip, Go modules and Maven are invisible to it. Automated patching therefore can never be one tool.

Two practical traps

It has no destination flag. -t/--tag only changes the tag; copa always pushes back to the source repository. Against a pull-only proxy-cache project that fails with:

unauthorized to access repository: dockerhub/library/nginx, action: push

The standard pattern is to copy each candidate into a writable project with crane first, then patch in place there. That is exactly how our nightly job works: crane copy into patched/<project>/<repo>:<tag>-patched, then copa against the copy.

It needs network egress to distro repos. Copa's BuildKit must reach deb.debian.org, the Alpine CDN, the Ubuntu archive. In one environment this was the difference between "Copacetic works" and a false conclusion of "Copacetic does not work" — the container network had no outbound TCP egress, and only careful log-reading revealed the real cause. If your buildkit cannot reach the distro repos, no patch will ever apply.

Where this leaves you

• OS CVEs: Copacetic, nightly, no rebuild, ~40–60 s per image.
• Everything else: a rebuild path — Renovate raising dependency and base-image MRs, your CI running tests, then merge.

The remediation decision in one line: if the CVE is in an apt/apk package, Copa can fix it tonight. If it is in npm/pip/Go or the base image itself, it needs a merge request.

Series: Building a Container Image Security Stack. Next: why hardened and distroless images — the ones you most want to trust — are the ones Copacetic can never patch.