prole/docs/completed/13-podmonitor-manual-management.md

Junie brief — Queue #13: migrate off CNPG-deprecated enablePodMonitor + podMonitorRelabelings

Self-contained brief. Manage the PodMonitor as a sibling resource instead of through CNPG operator fields. Closes a deprecation warning we get on every kubectl patch of the cluster spec.

---

1. Why

CNPG operator (running v1.29.0 on knoe-dev-cnpg-0) emits these warnings on every patch of Cluster.spec.monitoring:

Warning: spec.monitoring.enablePodMonitor is deprecated and will be removed
  in a future release. Please migrate to manually managing your PodMonitor
  resources. Set this field to false and create a PodMonitor resource for
  your cluster as described in the documentation.
Warning: spec.monitoring.podMonitorRelabelings is deprecated and will be
  removed in a future release. Please migrate to manually managing your
  PodMonitor resources with custom relabeling configurations.

Migration is straightforward: write the PodMonitor as a sibling YAML doc, apply it alongside the Cluster, set enablePodMonitor: false (or remove the whole monitoring block once that's also been deprecated). No functional change — just lifts the relabel config out of the CNPG-managed field and into a standard prometheus-operator resource.

The cnpg-grafana dashboard relies on the cluster label; breaking the relabel breaks the dashboard, so verify carefully.

2. Live state to match

DB=gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0
kubectl --context=$DB -n knoe-db-0 get podmonitor knoe-db -o yaml

The CNPG operator currently auto-creates this PodMonitor. Relevant fields:

spec:
  namespaceSelector: {}
  podMetricsEndpoints:
  - port: metrics
    relabelings:
    - action: replace
      sourceLabels: [__meta_kubernetes_pod_label_cnpg_io_cluster]
      targetLabel: cluster
  selector:
    matchLabels:
      cnpg.io/cluster: knoe-db
      cnpg.io/podRole: instance

You'll write this manifest into the repo, lose the CNPG-managed metadata (labels/annotations/ownerReferences) since you own it now, and apply it yourself.

3. What changes

3.1 — New file: deploy/gcp/gke/knoe-db-podmonitor.yaml

---
# Manually-managed PodMonitor for the CNPG `knoe-db` cluster.
#
# Migrated 2026-05-XX from the (now-deprecated) CNPG fields
# `Cluster.spec.monitoring.enablePodMonitor` +
# `Cluster.spec.monitoring.podMonitorRelabelings`. Same scrape spec, same
# relabel config — now living as a first-class resource we control.
#
# The selector matches every CNPG instance pod via the operator-applied
# labels `cnpg.io/cluster=knoe-db` + `cnpg.io/podRole=instance`. The
# `metrics` port is the postgres-exporter (sidecar) on the CNPG pod.
#
# The relabel injects a `cluster` label on every scraped sample, sourced
# from the `cnpg.io/cluster` pod label. The cnpg-grafana dashboards filter
# every panel by `cluster=$cluster`; without this relabel only the
# operator-collector metrics carry the cluster label and 83 of 85 panels
# render empty.
#
# Apply alongside `knoe-db.yaml` from etc/init_cnpg_gke.sh.
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: knoe-db
  namespace: knoe-db-0
  labels:
    # Match what kube-prometheus-stack discovers via Helm-managed Prometheus
    # serviceMonitorSelector / podMonitorSelector. kps's defaults match
    # release=kps; verify in the live cluster:
    #   kubectl -n monitoring get prometheus -o yaml | grep -A2 podMonitorSelector
    # If the selector requires a specific label, set it here.
    release: kps
spec:
  namespaceSelector: {}
  selector:
    matchLabels:
      cnpg.io/cluster: knoe-db
      cnpg.io/podRole: instance
  podMetricsEndpoints:
  - port: metrics
    relabelings:
    - action: replace
      sourceLabels:
        - __meta_kubernetes_pod_label_cnpg_io_cluster
      targetLabel: cluster

Verify the release: kps label trick. Inspect the live Prometheus's podMonitorSelector and ensure your manifest matches it. The current auto-created PodMonitor doesn't carry release: kps (it uses app.kubernetes.io/managed-by: cloudnative-pg), but it still gets discovered — meaning the kps Prometheus must be configured with podMonitorSelector: {} (any) or podMonitorSelectorNilUsesHelmValues: false (which is what monitoring/kps-cnpg-values.yaml sets). Confirm this and document in the manifest comment whether the release: kps label is actually required.

3.2 — deploy/gcp/gke/knoe-db.yaml

Replace the entire monitoring: block:

  monitoring:
    # NOTE: enablePodMonitor + podMonitorRelabelings are CNPG-deprecated and
    # will be removed in a future release. Migration path: manually manage
    # the PodMonitor resource. Tracked in docs/TODO.md as a follow-up.
    enablePodMonitor: true
    podMonitorRelabelings:
      - sourceLabels: ["__meta_kubernetes_pod_label_cnpg_io_cluster"]
        targetLabel: cluster
        action: replace

with:

  monitoring:
    # PodMonitor is managed manually as a sibling resource:
    # deploy/gcp/gke/knoe-db-podmonitor.yaml. The CNPG-managed alternative
    # (enablePodMonitor + podMonitorRelabelings) is deprecated.
    enablePodMonitor: false

3.3 — etc/init_cnpg_gke.sh

Find the section that applies knoe-db.yaml (search for render_cnpg_manifest "$storage_class" "$tmp_manifest"). After the cluster apply succeeds (and after the cluster reaches a working storage class), apply the PodMonitor:

# ── 11. Apply the manually-managed PodMonitor ────────────────────────────────
log "Applying PodMonitor for $CNPG_CLUSTER_NAME ..."
kubectl apply -f "$GKE_MANIFEST_DIR/knoe-db-podmonitor.yaml"

Place this AFTER the cluster apply completes (since the PodMonitor doesn't depend on the cluster being healthy — it just needs the namespace) but BEFORE any verification waits.

Also: when CNPG-managed PodMonitor was set, the operator auto-deleted it when enablePodMonitor flipped to false. Verify what happens during the transition on a live cluster: when you change enablePodMonitor: true → false AND apply your manual PodMonitor with the same name in the same namespace, does CNPG try to delete your manual one? Based on ownerReferences on the auto-created PodMonitor, the answer should be "the auto-created one gets deleted by garbage collection when the controller flag flips, and your manually-applied one (no ownerReference) survives". Test this on a non-prod cluster before claiming victory in the commit message.

3.4 — Cross-mode parity (k3s + k8s/min)

Both deploy/opentofu/k3s/manifests/knoe/knoe-db.yaml and k8s/knoe/knoe-db.yaml have monitoring.enablePodMonitor: false already — they don't run the dashboard, so no PodMonitor at all today. Leave those modes alone for now. They'll need their own PodMonitor manifest if we ever wire monitoring there, but that's separate work. Add a comment to each pointing at the GKE manifest as the reference if/when needed.

4. Verification

End-to-end — must pass before merging:

1. Apply the new manifest doesn't 404:

   kubectl --dry-run=server -f deploy/gcp/gke/knoe-db-podmonitor.yaml apply
   

2. Cluster spec change doesn't break operator:

   yq '.spec.monitoring' deploy/gcp/gke/knoe-db.yaml
   # Expected: only enablePodMonitor: false (no podMonitorRelabelings)
   

3. Live transition test (the scary one). On the live cluster, do the transition AND verify dashboard still works:

   DB=gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0
   APP=gke_plenary-truck-485623-p7_us-west3_knoe-dev-0
   
   # 1. Apply the new manual PodMonitor
   kubectl --context=$DB apply -f deploy/gcp/gke/knoe-db-podmonitor.yaml
   
   # 2. Verify it's there
   kubectl --context=$DB -n knoe-db-0 get podmonitor
   
   # 3. Apply the cluster spec change (flips enablePodMonitor false)
   kubectl --context=$DB apply -f deploy/gcp/gke/knoe-db.yaml
   #    (or however the script renders + applies it)
   
   # 4. Confirm only ONE PodMonitor remains (the manual one) — the
   #    CNPG-owned one should garbage-collect once the field flips
   kubectl --context=$DB -n knoe-db-0 get podmonitor
   
   # 5. Wait 30s and confirm DB-cluster Prometheus still has scrape targets:
   kubectl --context=$APP -n monitoring exec sts/kps-grafana -c grafana -- \
     wget -qO- "http://admin:$(kubectl --context=$APP -n monitoring get secret kps-grafana -o jsonpath='{.data.admin-password}' | base64 -d)@localhost:3000/api/datasources/proxy/uid/P5531627C358300FE/api/v1/targets?state=active" \
     | python3 -c 'import sys,json; t=json.load(sys.stdin)["data"]["activeTargets"]; print(f"{len(t)} active targets, postgres exporters: {sum(1 for x in t if \"9187\" in x[\"scrapeUrl\"])}")'
   # Expected: ≥3 postgres exporters active (our 3 CNPG pods)
   
   # 6. Refresh the cnpg-grafana dashboard, confirm panels still populate
   

4. Rollback path documented. If step 3.5 shows dropped targets, the rollback is to flip enablePodMonitor: true and delete the manual PodMonitor. Note this in the commit message.

5. Out of scope

• Any work on monitoring/kps-cnpg-values.yaml itself. podMonitorSelectorNilUsesHelmValues: false stays as is.
• Cross-mode PodMonitor (k3s/k8s/min) — defer.
• The cluster=knoe-db label vs. dynamic templating — currently hardcoded to knoe-db. If a second CNPG cluster ever lands, the manual PodMonitor needs to be parameterized. Leave a TODO comment but don't solve.
• Don't touch the cnpg-grafana dashboard transforms (monitoring/cnpg-dashboard-transforms.yaml) — those are unrelated.

6. Commit shape

feat(monitoring): manage knoe-db PodMonitor manually (drop CNPG-deprecated fields)

CNPG operator emits deprecation warnings on every cluster.spec.monitoring
patch:
  - enablePodMonitor will be removed in a future release
  - podMonitorRelabelings will be removed in a future release

Migrate to a manually-managed PodMonitor:
  + deploy/gcp/gke/knoe-db-podmonitor.yaml   (NEW — same spec as auto-created)
  ~ deploy/gcp/gke/knoe-db.yaml              (enablePodMonitor: true → false,
                                              podMonitorRelabelings dropped)
  ~ etc/init_cnpg_gke.sh                     (apply PodMonitor after cluster)

Verified live: cnpg-grafana dashboard panels render unchanged after the
flip; 3 postgres-exporter targets active in cnpg-prometheus.

Closes queue item #13 in docs/TODO.md.

7. Definition of done

• deploy/gcp/gke/knoe-db-podmonitor.yaml created, content verified against live PodMonitor spec.
• deploy/gcp/gke/knoe-db.yaml no longer contains podMonitorRelabelings; enablePodMonitor: false with comment.
• etc/init_cnpg_gke.sh applies the new manifest after cluster apply.
• Live cnpg-grafana dashboard verified after the transition: panels populate, 3 postgres-exporter targets in Prometheus.
• No new deprecation warnings on kubectl apply of the cluster manifest.
• docs/TODO.md queue item #13 archived to Done with verification notes.