prole/docs/plans/junie/07-init-cnpg-gke-sa-wiring.md

Junie brief — Queue #7: wire cnpg-backup-sa into the CNPG cluster spec via etc/init_cnpg_gke.sh

Self-contained brief. Drift R8: live cluster state has the SA wired through, the init script doesn't. Make the script land a fresh deploy in the same end state. Live cluster is healthy — do not run anything against it.

---

1. Why

etc/init_cnpg_gke.sh runs from a clean state (e.g. for a future second DB cluster, or DR rebuild) and lays down everything: GCS buckets, GCP service account, Workload Identity binding, K8s service account, namespace, CNPG operator, secrets, the CNPG Cluster resource. The 2026-04-29 stabilization session added two pieces that the script doesn't yet reproduce:

1. cluster.spec.serviceAccountName: cnpg-backup-sa on the live cluster. This field was added in CNPG v1.29.0. The init script currently pins CNPG v1.24.0 (line 402: cnpg-1.24.0.yaml), which doesn't support the field — the cluster would either ignore it or fail validation. Bump the operator version AND add the field to the rendered manifest.

2. RoleBindings extended with cnpg-backup-sa as a subject. Live knoe-db and knoe-db-barman-cloud RoleBindings (auto-created by CNPG at cluster creation) each have two subjects: knoe-db (the default SA CNPG creates) and cnpg-backup-sa. With v1.29+ and spec.serviceAccountName set from the start, CNPG should auto-create RoleBindings naming the explicit SA — but verify, because the live state was patched after the fact.

The result of this brief: a fresh init_cnpg_gke.sh run produces a cluster whose RBAC and SA wiring matches kubectl get cluster knoe-db -o yaml

• kubectl get rolebinding -n knoe-db-0 from the live system, with no post-install patches required.

2. Live state to match

DB=gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0
kubectl --context=$DB -n knoe-db-0 get cluster knoe-db -o jsonpath='{.spec.serviceAccountName}'
# -> cnpg-backup-sa

kubectl --context=$DB -n knoe-db-0 get rolebinding knoe-db -o jsonpath='{.subjects}' | jq
# -> [{kind:ServiceAccount, name:knoe-db, namespace:knoe-db-0},
#     {kind:ServiceAccount, name:cnpg-backup-sa, namespace:knoe-db-0}]

kubectl --context=$DB -n knoe-db-0 get rolebinding knoe-db-barman-cloud -o jsonpath='{.subjects}' | jq
# -> same two subjects

3. What changes

3.1 — etc/init_cnpg_gke.sh

Around line 402 (CNPG operator install):

# Current (stale)
kubectl apply --server-side -f \
  https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/main/releases/cnpg-1.24.0.yaml

Bump to v1.29.0 (or a newer stable; verify against CNPG releases that the version supports Cluster.spec.serviceAccountName). Add a code comment explaining the version constraint:

# v1.29.0+ is required: spec.serviceAccountName lets the cluster pods run
# as cnpg-backup-sa (annotated with iam.gke.io/gcp-service-account for WI),
# which is how the GCS-backed barman ObjectStore authenticates without a
# static key.  Older operators (we used 1.24.0 historically) would silently
# drop the field.
CNPG_OPERATOR_VERSION="1.29.0"
kubectl apply --server-side -f \
  "https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/main/releases/cnpg-${CNPG_OPERATOR_VERSION}.yaml"

(Promote the version to a top-of-file variable so future bumps are one line.)

3.2 — deploy/gcp/gke/knoe-db.yaml (CNPG Cluster manifest)

Add serviceAccountName: cnpg-backup-sa under spec:. Use the same indentation as the surrounding fields. Place near the top of spec (right after description / imageName / instances, before bootstrap — follow the live cluster's field order if you want to be tidy: kubectl get cluster knoe-db -o yaml).

Add an inline comment:

  # Run cluster pods as cnpg-backup-sa (annotated for Workload Identity to
  # the cnpg-backup@... GCP SA). This is how barman-cloud authenticates
  # to gs://knoe-0-backups/ without a static key. Requires CNPG v1.29+.
  # The SA is provisioned by etc/init_cnpg_gke.sh § "Apply ServiceAccount
  # + annotate with WI".
  serviceAccountName: cnpg-backup-sa

3.3 — RoleBinding subjects (decide path)

Two paths, depending on what CNPG v1.29 actually does at cluster creation time. Test both before deciding.

Path A — operator does the right thing automatically. With spec.serviceAccountName: cnpg-backup-sa set from the start, CNPG v1.29+ auto-creates knoe-db and knoe-db-barman-cloud RoleBindings with cnpg-backup-sa as the (sole) subject. If true: nothing more to do in the script — the live two-subject state is just a historical artifact of having patched after-the-fact, and a fresh install would have a single subject.

Path B — operator still creates with a default SA name. If CNPG auto-creates the RoleBindings with subject knoe-db (the default SA name matching the cluster name) regardless of the explicit serviceAccountName, then the script needs to patch them. Add a section after cluster apply:

# ── 11. Ensure RoleBindings reference cnpg-backup-sa ─────────────────────────
log "Ensuring CNPG-managed RoleBindings include cnpg-backup-sa ..."
for rb in "${CNPG_CLUSTER_NAME}" "${CNPG_CLUSTER_NAME}-barman-cloud"; do
  # Wait briefly for CNPG to create them on first install
  for _ in 1 2 3 4 5; do
    kubectl -n "$CNPG_NAMESPACE" get rolebinding "$rb" >/dev/null 2>&1 && break
    sleep 2
  done
  if kubectl -n "$CNPG_NAMESPACE" get rolebinding "$rb" >/dev/null 2>&1; then
    if ! kubectl -n "$CNPG_NAMESPACE" get rolebinding "$rb" -o jsonpath='{.subjects[*].name}' \
         | grep -qw cnpg-backup-sa; then
      kubectl -n "$CNPG_NAMESPACE" patch rolebinding "$rb" --type='json' -p="$(cat <<'EOF'
[{"op":"add","path":"/subjects/-","value":{"kind":"ServiceAccount","name":"cnpg-backup-sa","namespace":"NAMESPACE_PLACEHOLDER"}}]
EOF
)" --patch-file=/dev/stdin 2>/dev/null || true
    fi
  fi
done

(Sketch — finalize the syntax. The NAMESPACE_PLACEHOLDER would be substituted; --patch-file doesn't accept - in all kubectl versions, so you may need a temp file.)

How to decide A vs B: spin up a throwaway k3d cluster (or a CNPG-only test namespace on knoe-dev-cnpg-0 if you can without disturbing knoe-db-0), apply a minimal Cluster resource with serviceAccountName: cnpg-backup-sa, observe the RoleBinding subjects that CNPG auto-creates. If the SA you specified is the only subject: Path A. Otherwise: Path B.

If you can't easily test, lean toward Path B (the patch is idempotent and costs nothing if the operator already did the right thing).

4. Verification

Without running against the live cluster:

1. Manifest renders cleanly:

   ARTIFACT_REGISTRY=us-west3-docker.pkg.dev/plenary-truck-485623-p7/knoe-system \
   KNOE_DB_IMAGE_TAG=18-055 \
     envsubst '${ARTIFACT_REGISTRY} ${KNOE_DB_IMAGE_TAG}' < deploy/gcp/gke/knoe-db.yaml \
     | yq '.spec.serviceAccountName'
   # Expected: cnpg-backup-sa
   

2. Operator version is 1.29+:

   grep -E 'cnpg-[0-9]' etc/init_cnpg_gke.sh
   # Expected: cnpg-1.29.0.yaml (or newer)
   

3. bash -n etc/init_cnpg_gke.sh clean.

4. (Optional) Dry-run apply against k3d if you have one running. The cluster won't reach Healthy in k3d (no Workload Identity), but kubectl get cluster -o yaml should show spec.serviceAccountName: cnpg-backup-sa set.

5. Out of scope

• Don't touch the live cluster. This is purely script + manifest work.
• Don't change the cnpg-backup-sa annotation pattern in deploy/gcp/gke/knoe-db-backup-gcs.yaml — that's how the SA gets the WI annotation, separate concern.
• The patch_garage_cross_cluster.sh defects are queue item #6 with its own brief. If your changes here surface a discrepancy with that brief, flag it but don't try to fix both in one commit.
• The PodMonitor manual-management migration (queue #13) doesn't interact with this brief.

6. Commit shape

Likely two commits:

1. chore(deploy): add serviceAccountName: cnpg-backup-sa to knoe-db cluster spec
2. fix(init): bump CNPG operator to v1.29 + ensure RoleBindings include cnpg-backup-sa

Or one combined commit titled fix(init): wire cnpg-backup-sa as cluster SA + bump CNPG to v1.29 (drift R8).

7. Definition of done

• etc/init_cnpg_gke.sh references CNPG operator v1.29.0+ with a comment explaining why (the SA field requirement).
• deploy/gcp/gke/knoe-db.yaml carries spec.serviceAccountName: cnpg-backup-sa with explanatory comment.
• RoleBinding handling: either documented as auto-correct (Path A, with a one-liner test result included in the commit message), OR scripted patch (Path B).
• bash -n passes; manifest renders.
• docs/TODO.md queue item #7 + drift R8 row archived to Done.