Creating the Project Is Not Enough: Measuring the Gap Between Can and Does

The Harbor side of a registry routing took an afternoon: endpoint created, TEST CONNECTION green, project Public, proxy cache enabled, scan-on-push ticked. And the estate barely moved. Measured on 2026-09-09, with the gitlabcom project fully live: all nine GitLab pods were still pulling registry.gitlab.com/... directly, and no image in the cluster resolved under gitlabcom/.

Why: nothing asks

Creating the project gives Harbor the ability to serve those images. The cluster has no reason to use that ability until a Kyverno rewrite rule rewrites pod references at admission. That is step B in the runbook's three jobs — and it lives in Git:

# base/kyverno/configs/kustomization.yaml
  - ./registry-routing/quay.yaml        # 9 images   ← uncomment ONE line

Merge that one line — on its own, never batched — and only pods created after the reconcile are rewritten, because the policy sets background: false. Nothing rewrites already-running pods, by design.

Project live: 9/9 pods still direct, 0 images under the new path. The MR merged: new pods route — and the ten-minute watch begins.

The watch

After the enable merges:

kubectl get clusterpolicy rewrite-registry-<proj>                # READY=True
kubectl get pods -A | grep -E 'ImagePullBackOff|ErrImagePull'    # expect empty

Nothing will have moved yet — that is correct. The canary restart (one low-risk workload from the blast-radius column) is job C, and the definition of done requires Harbor to show an actual scan result for the repository, not just a cached blob. A cache without a scan is just a cache.

The two-sided gap

The measurement had two halves, and both matter. Harbor side: the project existed, was Public, proxied, and scanned — TEST CONNECTION green, artifacts fetchable. Cluster side: zero rewritten references, zero pulls through the new path. A two-sided gap means both sides can look "done" while the integration is unexercised — which is exactly the failure mode of any pipeline built from independently-verifiable parts.

The cluster-side half is the one that bit, because nothing errors. A Kustomization with a commented-out line reconciles perfectly; the policy it would enable simply does not exist. kubectl get clusterpolicy | grep rewrite-registry is the one-command audit for the whole gap — it lists which routing policies are actually live, and by omission, which registries still pull direct.

The ten-minute watch

The runbook prescribes watching for ten minutes after the enable merges — long enough for Flux to reconcile the Kustomization, apply the policy, and for the first accidental pod creations to exercise it:

# after merge
kubectl get clusterpolicy rewrite-registry-<proj>     # expect READY=True
watch 'kubectl get pods -A | grep -E "ImagePullBackOff|ErrImagePull"'

Empty is the expected state — not because pods are being rewritten and succeeding, but because nothing has restarted yet (background: false). The watch is for the opposite signal: a project that is Private, an endpoint that is wrong, a name that does not match. Those turn pods red within seconds of their next restart, and the ten-minute window catches them while the MR is still the newest thing that happened to the cluster.

Canaries before masses

Job C's first restart is chosen from the blast-radius column's safest consumer, and the runbook is explicit about what "safe" means: not smallest, but least important when it fails once. For quay, the canary is a monitoring Deployment, not node-exporter; for k8s, external-dns before the ingress-nginx DaemonSet. The canary's success signal is two-part: the pod runs on a harbor.private.example.com/<proj>/... reference and Harbor shows a scan result for the repository — the second part is what distinguishes a routed image from a merely cached one, and it is the step that was skipped when a routing pass produced "a cache with no security benefit at all."

Next: safest-first ordering — the full queue, ranked by what breaks if it goes wrong.