The One Failure That Looks Like Ten Different Failures: an etcd Quota Incident

On 2026-09-08, our cluster stopped reconciling. Every visible symptom suggested a different root cause than the actual one. This is a postmortem of an outage with a one-line cause and a ten-line symptom list.

What you actually see

None of which mentions etcd:

• flux reconcile … ✗ context deadline exceeded, on every Kustomization.
• Pods stuck with config/secret volume mounts not syncing — a ConfigMap or Secret read the kubelet cannot satisfy stalls the pod before the container ever starts.
• Generalised API server slowness that looks like a network problem, a node problem, or a registry problem.

It looks like a network problem. It looks like a node problem. It is neither. etcd has hit its size quota, so no write of any kind succeeds — and reads that used to be served from the watch cache start to miss, and the control plane grinds.

Confirming it in one command

The Kyverno admission controller logs the raw etcd error verbatim, which makes it the fastest place to look:

kubectl -n kyverno logs deploy/kyverno-admission-controller --tail=50 \
  | grep -o 'etcd quota exceeded[^"]*' | head -1
# etcd quota exceeded: size 200083595/200000000, burst 2756450/200000000

What filled it

One CRD family outgrew the control plane's entire budget.

Trivy Operator's SBOM generation stores one sbomreport custom resource per scanned workload — full package inventories, in etcd. In our cluster that reached 286 objects / 212 MB against Kapsule's 200 MB quota.

The decomposition is the damning part: sbomreports alone exceeded the entire quota; everything else in the cluster together was under 30 MB.

The fix we chose

# base/trivy/release.yaml
trivy:
  # DISABLED AFTER AN OUTAGE. SBOM reports are stored as CRs in etcd, and on
  # 2026-09-08 they reached 286 objects / 212 MB against a 200 MB quota.
  sbomGenerationEnabled: false

The options were:

We took options two and three together. The third is architecturally where SBOMs belong — a registry, not etcd. Harbor generates CycloneDX per artifact on push and serves it via API, backed by its own storage. An artifact's inventory belongs next to the artifact.

Recovery mechanics

Space is reclaimed as soon as etcd compacts, which the managed control plane does automatically; there is no alarm to clear by hand. Recovery is visible within a minute or two of deleting enough CRs — Flux reconciliations resume on their own.

One operational footnote we now bake into procedure: batch your merges. Merging many MRs at once triggered a cluster-wide reconcile that exhausted etcd again the next day. In a Flux cluster with prune and drift detection, a big batch merge is a write storm: every deleted object, every updated status, every new CRD, at once.

Lessons

1. CRDs are etcd rows. Operators that store per-workload documents (SBOMs, full configs, attestations) can outgrow a quota meant for control-plane metadata. Watch the size of the largest CRD family, not just the count of objects.
2. The symptom will not name the cause. "Everything is slow" with no etcd-shaped error anywhere except in a component that happens to log raw API errors. Kyverno's honesty here cut diagnosis from hours to minutes.
3. Capacity monitoring should track etcd db size against quota, with the alarm set well below the cliff — writes fail at 100%, and by then your dashboards (which need API reads) may be struggling too.
4. Know which component logs the rawest error. In our cluster it is Kyverno. Yours is probably your admission webhook too, because it sees every failed write synchronously.

The incident produced its own branch name: fix/trivy-disable-sbom-etcd-outage. If your incident does not end with a branch named after it, you probably have not finished the postmortem.

Series: Building a Container Image Security Stack. Next: admission policies — why "restrict image registries" is the first Kyverno policy to ship, in audit mode.