Kyverno Test Goes Green When You Weaken a Policy

A policy test suite that only ever runs against a correct policy proves nothing. So before wiring kyverno test into CI as the regression gate, we ran the negative control: break the policy on purpose and check that the suite notices.

It did not.

The asymmetry

The test suite asserts an expected result per fixture. We regressed the registry allow-list to "*" — so the four violating Pods now sail through — and left the assertions saying result: fail. Kyverno CLI 1.17.2 produced this:

│ 1 │ poc-restrict-image-registries │ validate-registries │ .../fail-dockerhub-bare │ Pass │ Want fail, got pass │
Test Summary: 25 tests passed and 0 tests failed
EXIT_CODE=0

It knows. "Want fail, got pass" is right there in the REASON column. But the RESULT column says Pass, the case is counted as passed in the summary, and the process exits 0. In CI, that is a green pipeline.

The other direction works correctly. Assert result: pass on a Pod the pristine policy rejects, and you get RESULT=Fail, "21 tests passed and 4 tests failed", exit 1.

Want pass, got fail exits 1. Want fail, got pass exits 0 — and that is the direction a weakened policy fails in.

Why this is the dangerous direction

Policies do not usually regress by becoming stricter. They regress by becoming looser: someone widens an allow-list to unblock a deploy, adds an exclusion, relaxes a pattern. Every one of those regressions makes a previously-failing fixture pass — which is precisely the direction kyverno test does not report.

We checked whether this was an artefact of failureAction: Audit. It is not. The same regression with failureAction: Enforce gave "7 tests passed and 0 tests failed", exit 0.

What we gate on instead

kyverno apply exit codes are correct in both directions, and they are what the pipeline actually gates on:

-- regressed policy vs the 4 violating Pods
pass: 4, fail: 0, warn: 0, error: 0, skip: 0
EXIT_CODE=0
-- pristine policy vs the same 4 Pods
pass: 0, fail: 4, warn: 0, error: 0, skip: 0
EXIT_CODE=1

The regressed policy genuinely passes the violating Pods, so exit 0 is a true negative — the tool is reporting reality. The pipeline's job is to notice that reality changed. So the CI gate is two things, not one:

1. Run kyverno apply against the known-bad fixtures and require exit 1.
2. Run kyverno test, and grep its output for Want fail, got and Want pass, got. Any hit fails the job regardless of exit code.

The second line exists only because the exit code cannot be trusted. It is an ugly guard, and it is load-bearing.

The general shape

This is what a negative control is for. A test harness that has only ever seen the passing case tells you nothing about what it does when the thing under test is broken — and a CI gate is exactly a claim about the broken case. Break the policy, confirm the suite goes red, and only then believe the green.

Next: a policy bundle with one wrong document — the other way a Kyverno CI gate can enforce nothing while exiting 0.