The Pull Test: How to Read a Pod Until It Tells You the Truth

Before a registry-routing rule is enabled, the runbook demands one piece of evidence: an in-cluster pull test. A pod, created directly, pointed at the new Harbor path. The test is not "did the pod run?" — it is "did the image pull?". Those are different questions, and the difference caused two real mistakes.

The reading table

That last row is the one that feels wrong and is right. A distroless test image has no shell, so kubectl run fails to exec into it — after the image layers arrived intact. The pull succeeded; the runtime failed. Grep for the only error that matters:

kubectl -n pulltest describe pod t | grep -cE 'no basic auth|pull access denied'
# must print 0

The table from the runbook, including the note that two mistakes were made reading this exact test — both on a Private project that looked fine from a laptop.

The companion mistake

The laptop crane manifest check is the mirror-image error: your CLI is authenticated, so it succeeds while every node fails. Both mistakes share a root cause — testing with a client whose credentials differ from the thing that will actually pull. The kubelet with no imagePullSecrets is the client that matters. Test where it runs.

The definition of done per registry includes the test explicitly: endpoint TEST CONNECTION passed, project Public with scan-on-push, in-cluster pull test shows 0 auth errors — only then does the rule's MR merge.

The full procedure, in order

The pull test sits in the runbook between project creation and rule enabling, and the ordering is deliberate: prove the cluster can pull before the cluster starts asking. Enabling a rewrite rule against an unproven project is how the ImagePullBackOff incidents happened.

PROJ=quay                                   # from the per-registry table
IMG=prometheus/node-exporter:v1.11.1        # a real test image, from the table

kubectl create ns pulltest
kubectl -n pulltest run t --image=harbor.private.example.com/$PROJ/$IMG --restart=Never
kubectl -n pulltest get pod t -o wide -w     # Ctrl-C once it leaves ContainerCreating
kubectl -n pulltest describe pod t | grep -cE 'no basic auth|pull access denied'
# must print 0
kubectl delete ns pulltest

The per-registry table supplies a real test image per project — one that exists upstream and is safe to schedule. Using the table's image rather than an invented one matters: a typo'd repository produces a different failure (404, not 401) and muddies the reading.

Why "any other outcome" is a pass

The counter-intuitive row — CrashLoopBackOff is a pass — deserves its reasoning written out, because it trips people repeatedly. The test isolates one question: can the kubelet authenticate to Harbor and fetch the layers? Everything after that is the workload's problem, and the workload here is a bare kubectl run with no config, no command, and often no compatible runtime. A distroless image has no shell and no true binary; the exec fails; the pod reports exec: "true": executable file not found — after the image is fully on disk. The pull demonstrably succeeded.

The failure signature to grep for is narrow on purpose: no basic auth|pull access denied. Broader greps ("failed", "error") match the runtime noise of a deliberately unconfigured pod and make every test look like a failure. Narrow grep, single question, unambiguous verdict.

The two mistakes this test replaced

Both recorded mistakes share a shape: verification with the wrong client. The laptop crane manifest check — authenticated CLI, green result, Private project — passed while production broke. The "it must be the network" detour — after seeing ImagePullBackOff on a known-good project — cost an hour of chasing a non-existent egress problem before the auth error was read literally. The pull test plus the narrow grep is the procedure's way of making both mistakes structurally impossible: the client is the kubelet, the cluster is the network, and the only string that matters is the one the registry returns for auth.

Next: creating is not enabling — the measured gap between the project existing and anything using it.