Safest First: Ordering a Migration by Blast Radius, Not by Image Count

The nine-registry routing queue is sorted safest-first: gcrmirror touches one scanner, quay touches OpenBao's StatefulSet, and dragonflydb — GitLab's own Redis — comes with a note that skipping it is a reasonable choice.

Safest First: Ordering a Migration by Blast Radius, Not by Image Count

Every migration table has two possible sorts: by value, or by danger. The registry-routing runbook carries both, and keeps them in different columns — value decided that gitlabcom goes first; blast radius decides the order you learn the procedure in.

The blast-radius column

OrderRegistryImagesRestart blast radius
1gcrmirror2trivy only — safest, do this first
2ecr1argocd redis
3redhat1argocd
4fluentbit1monitoring DaemonSet — brief log gap
5k8s3ingress-nginx DaemonSet, external-dns — watch for 502s
6quay9OpenBao StatefulSet — do alone, verify unsealed after
7dragonflydb1GitLab StatefulSet — its Redis. Consider skipping

The biggest project is near the bottom of the learning order. quay has the most images and touches the one StatefulSet in the cluster whose restart requires unsealing — do it once the steps are muscle memory, and do it alone.

Safest-first, not biggest-first
Safest-first, not biggest-first

The runbook's ordering, with the reasoning inline — including the one entry where skipping is explicitly endorsed.

StatefulSets and DaemonSets go last, with different verbs

Two operational details the table encodes. Rolling a DaemonSet restarts ingress or log shipping on each node in turn — expect a short gap, and watch for 502s on the ingress one. And the restart verbs differ:

kubectl -n <ns> rollout restart statefulset/<name>
kubectl -n <ns> rollout restart daemonset/<name>

The runbook also warns that names are not what you expect: on the first pass, three of five guessed GitLab workload names did not exist. List them first with kubectl get deploy,statefulset,daemonset -o custom-columns=..., then restart what is actually there.

How the blast-radius column is built

The column is not intuition; it is derived from the estate. For each registry, the routing team listed the workloads whose images come from it (from the digest-joined pod data), then annotated each with the kind of restart it means:

  • a Deployment with replicas — rolling restart, no gap;
  • a DaemonSet — restarts node-by-node; for ingress-nginx that is brief 502s per node, for fluent-bit a short per-node log gap;
  • a StatefulSet — an ordered restart with state attached; for OpenBao specifically, a post-restart unseal verification.

That taxonomy, not the image count, is what the order encodes. quay outranks ghcr in image count (9 vs 11 is close) but loses on restart class: OpenBao is the only workload in the cluster whose restart requires a credential ceremony to recover from.

The workload-name trap

On the first pass, three of five guessed GitLab workload names did not exist — the actual names differ from the chart's release names in ways that are obvious in hindsight (gitlab-gitlab-shell, not gitlab-shell; backups as Jobs, not Deployments). The runbook's fix is to list before restarting:

kubectl -n <ns> get deploy,statefulset,daemonset \
  -o custom-columns='KIND:.kind,NAME:.metadata.name,READY:.status.readyReplicas' --no-headers

Restart what is listed, not what is remembered. The same trap has a quieter variant: finished Jobs never restart, so the estate image count always includes a few direct references that no routing pass will ever move — the runbook's progress command carries the caveat, and three GitLab backup Jobs account for the residual count.

The explicit skip

dragonflydb — one image, GitLab's Redis — carries the runbook's only endorsed skip: consider skipping this registry entirely. The reasoning is proportionality: the procedure's full ceremony (endpoint, project, pull test, rule MR, canary, full restart of a StatefulSet) buys one image that already scans at runtime. Writing the skip down is as much a part of the runbook as the steps — it tells the next operator that "not done" can be a decision rather than an omission, and it prevents the completeness instinct from spending an afternoon on the estate's least valuable registry.

How the blast-radius column is built

The column is not intuition; it is derived from the estate. For each registry, the routing team listed the workloads whose images come from it (from the digest-joined pod data), then annotated each with the kind of restart it means:

  • a Deployment with replicas — rolling restart, no gap;
  • a DaemonSet — restarts node-by-node; for ingress-nginx that is brief 502s per node, for fluent-bit a short per-node log gap;
  • a StatefulSet — an ordered restart with state attached; for OpenBao specifically, a post-restart unseal verification.

That taxonomy, not the image count, is what the order encodes. quay outranks ghcr in image count (9 vs 11 is close) but loses on restart class: OpenBao is the only workload in the cluster whose restart requires a credential ceremony to recover from.

The workload-name trap

On the first pass, three of five guessed GitLab workload names did not exist — the actual names differ from the chart's release names in ways that are obvious in hindsight (gitlab-gitlab-shell, not gitlab-shell; backups as Jobs, not Deployments). The runbook's fix is to list before restarting:

kubectl -n <ns> get deploy,statefulset,daemonset \
  -o custom-columns='KIND:.kind,NAME:.metadata.name,READY:.status.readyReplicas' --no-headers

Restart what is listed, not what is remembered. The same trap has a quieter variant: finished Jobs never restart, so the estate image count always includes a few direct references that no routing pass will ever move — the runbook's progress command carries the caveat, and three GitLab backup Jobs account for the residual count.

The explicit skip

dragonflydb — one image, GitLab's Redis — carries the runbook's only endorsed skip: consider skipping this registry entirely. The reasoning is proportionality: the procedure's full ceremony (endpoint, project, pull test, rule MR, canary, full restart of a StatefulSet) buys one image that already scans at runtime. Writing the skip down is as much a part of the runbook as the steps — it tells the next operator that "not done" can be a decision rather than an omission, and it prevents the completeness instinct from spending an afternoon on the estate's least valuable registry.

Next: the honest ceiling — why 38 of 94 images can never route through Harbor at all.