prole/docs/plans/junie/13-podmonitor-manual-management.md

# Junie brief — Queue #13: migrate off CNPG-deprecated `enablePodMonitor` + `podMonitorRelabelings`

> **Self-contained brief.** Manage the PodMonitor as a sibling resource
> instead of through CNPG operator fields. Closes a deprecation warning
> we get on every `kubectl patch` of the cluster spec.

---

## 1. Why

CNPG operator (running v1.29.0 on `knoe-dev-cnpg-0`) emits these warnings
on every patch of `Cluster.spec.monitoring`:

```
Warning: spec.monitoring.enablePodMonitor is deprecated and will be removed
  in a future release. Please migrate to manually managing your PodMonitor
  resources. Set this field to false and create a PodMonitor resource for
  your cluster as described in the documentation.
Warning: spec.monitoring.podMonitorRelabelings is deprecated and will be
  removed in a future release. Please migrate to manually managing your
  PodMonitor resources with custom relabeling configurations.
```

Migration is straightforward: write the PodMonitor as a sibling YAML doc,
apply it alongside the Cluster, set `enablePodMonitor: false` (or remove
the whole `monitoring` block once that's also been deprecated). No
functional change — just lifts the relabel config out of the CNPG-managed
field and into a standard prometheus-operator resource.

The cnpg-grafana dashboard relies on the `cluster` label; **breaking the
relabel breaks the dashboard**, so verify carefully.

## 2. Live state to match

```bash
DB=gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0
kubectl --context=$DB -n knoe-db-0 get podmonitor knoe-db -o yaml
```

The CNPG operator currently auto-creates this PodMonitor. Relevant fields:

```yaml
spec:
  namespaceSelector: {}
  podMetricsEndpoints:
  - port: metrics
    relabelings:
    - action: replace
      sourceLabels: [__meta_kubernetes_pod_label_cnpg_io_cluster]
      targetLabel: cluster
  selector:
    matchLabels:
      cnpg.io/cluster: knoe-db
      cnpg.io/podRole: instance
```

You'll write this manifest into the repo, lose the CNPG-managed metadata
(labels/annotations/ownerReferences) since you own it now, and apply it
yourself.

## 3. What changes

### 3.1 — New file: `deploy/gcp/gke/knoe-db-podmonitor.yaml`

```yaml
---
# Manually-managed PodMonitor for the CNPG `knoe-db` cluster.
#
# Migrated 2026-05-XX from the (now-deprecated) CNPG fields
# `Cluster.spec.monitoring.enablePodMonitor` +
# `Cluster.spec.monitoring.podMonitorRelabelings`. Same scrape spec, same
# relabel config — now living as a first-class resource we control.
#
# The selector matches every CNPG instance pod via the operator-applied
# labels `cnpg.io/cluster=knoe-db` + `cnpg.io/podRole=instance`. The
# `metrics` port is the postgres-exporter (sidecar) on the CNPG pod.
#
# The relabel injects a `cluster` label on every scraped sample, sourced
# from the `cnpg.io/cluster` pod label. The cnpg-grafana dashboards filter
# every panel by `cluster=$cluster`; without this relabel only the
# operator-collector metrics carry the cluster label and 83 of 85 panels
# render empty.
#
# Apply alongside `knoe-db.yaml` from etc/init_cnpg_gke.sh.
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: knoe-db
  namespace: knoe-db-0
  labels:
    # Match what kube-prometheus-stack discovers via Helm-managed Prometheus
    # serviceMonitorSelector / podMonitorSelector. kps's defaults match
    # release=kps; verify in the live cluster:
    #   kubectl -n monitoring get prometheus -o yaml | grep -A2 podMonitorSelector
    # If the selector requires a specific label, set it here.
    release: kps
spec:
  namespaceSelector: {}
  selector:
    matchLabels:
      cnpg.io/cluster: knoe-db
      cnpg.io/podRole: instance
  podMetricsEndpoints:
  - port: metrics
    relabelings:
    - action: replace
      sourceLabels:
        - __meta_kubernetes_pod_label_cnpg_io_cluster
      targetLabel: cluster
```

**Verify the `release: kps` label trick.** Inspect the live Prometheus's
`podMonitorSelector` and ensure your manifest matches it. The current
auto-created PodMonitor doesn't carry `release: kps` (it uses
`app.kubernetes.io/managed-by: cloudnative-pg`), but it still gets
discovered — meaning the kps Prometheus must be configured with
`podMonitorSelector: {}` (any) or `podMonitorSelectorNilUsesHelmValues:
false` (which is what `monitoring/kps-cnpg-values.yaml` sets). Confirm
this and document in the manifest comment whether the `release: kps`
label is actually required.

### 3.2 — `deploy/gcp/gke/knoe-db.yaml`

Replace the entire `monitoring:` block:

```yaml
  monitoring:
    # NOTE: enablePodMonitor + podMonitorRelabelings are CNPG-deprecated and
    # will be removed in a future release. Migration path: manually manage
    # the PodMonitor resource. Tracked in docs/TODO.md as a follow-up.
    enablePodMonitor: true
    podMonitorRelabelings:
      - sourceLabels: ["__meta_kubernetes_pod_label_cnpg_io_cluster"]
        targetLabel: cluster
        action: replace
```

with:

```yaml
  monitoring:
    # PodMonitor is managed manually as a sibling resource:
    # deploy/gcp/gke/knoe-db-podmonitor.yaml. The CNPG-managed alternative
    # (enablePodMonitor + podMonitorRelabelings) is deprecated.
    enablePodMonitor: false
```

### 3.3 — `etc/init_cnpg_gke.sh`

Find the section that applies `knoe-db.yaml` (search for
`render_cnpg_manifest "$storage_class" "$tmp_manifest"`). After the
cluster apply succeeds (and after the cluster reaches a working storage
class), apply the PodMonitor:

```bash
# ── 11. Apply the manually-managed PodMonitor ────────────────────────────────
log "Applying PodMonitor for $CNPG_CLUSTER_NAME ..."
kubectl apply -f "$GKE_MANIFEST_DIR/knoe-db-podmonitor.yaml"
```

Place this AFTER the cluster apply completes (since the PodMonitor doesn't
depend on the cluster being healthy — it just needs the namespace) but
BEFORE any verification waits.

Also: when CNPG-managed PodMonitor was set, the operator auto-deleted it
when `enablePodMonitor` flipped to false. Verify what happens during the
transition on a live cluster: when you change `enablePodMonitor: true →
false` AND apply your manual PodMonitor with the same name in the same
namespace, does CNPG try to delete your manual one? Based on
`ownerReferences` on the auto-created PodMonitor, the answer should be
"the auto-created one gets deleted by garbage collection when the
controller flag flips, and your manually-applied one (no ownerReference)
survives". Test this on a non-prod cluster before claiming victory in the
commit message.

### 3.4 — Cross-mode parity (k3s + k8s/min)

Both `deploy/opentofu/k3s/manifests/knoe/knoe-db.yaml` and
`k8s/knoe/knoe-db.yaml` have `monitoring.enablePodMonitor: false` already
— they don't run the dashboard, so no PodMonitor at all today. **Leave
those modes alone for now.** They'll need their own PodMonitor manifest if
we ever wire monitoring there, but that's separate work. Add a comment to
each pointing at the GKE manifest as the reference if/when needed.

## 4. Verification

End-to-end — must pass before merging:

1. **Apply the new manifest doesn't 404:**
   ```bash
   kubectl --dry-run=server -f deploy/gcp/gke/knoe-db-podmonitor.yaml apply
   ```

2. **Cluster spec change doesn't break operator:**
   ```bash
   yq '.spec.monitoring' deploy/gcp/gke/knoe-db.yaml
   # Expected: only enablePodMonitor: false (no podMonitorRelabelings)
   ```

3. **Live transition test (the scary one).** On the live cluster, do the
   transition AND verify dashboard still works:
   ```bash
   DB=gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0
   APP=gke_plenary-truck-485623-p7_us-west3_knoe-dev-0

   # 1. Apply the new manual PodMonitor
   kubectl --context=$DB apply -f deploy/gcp/gke/knoe-db-podmonitor.yaml

   # 2. Verify it's there
   kubectl --context=$DB -n knoe-db-0 get podmonitor

   # 3. Apply the cluster spec change (flips enablePodMonitor false)
   kubectl --context=$DB apply -f deploy/gcp/gke/knoe-db.yaml
   #    (or however the script renders + applies it)

   # 4. Confirm only ONE PodMonitor remains (the manual one) — the
   #    CNPG-owned one should garbage-collect once the field flips
   kubectl --context=$DB -n knoe-db-0 get podmonitor

   # 5. Wait 30s and confirm DB-cluster Prometheus still has scrape targets:
   kubectl --context=$APP -n monitoring exec sts/kps-grafana -c grafana -- \
     wget -qO- "http://admin:$(kubectl --context=$APP -n monitoring get secret kps-grafana -o jsonpath='{.data.admin-password}' | base64 -d)@localhost:3000/api/datasources/proxy/uid/P5531627C358300FE/api/v1/targets?state=active" \
     | python3 -c 'import sys,json; t=json.load(sys.stdin)["data"]["activeTargets"]; print(f"{len(t)} active targets, postgres exporters: {sum(1 for x in t if \"9187\" in x[\"scrapeUrl\"])}")'
   # Expected: ≥3 postgres exporters active (our 3 CNPG pods)

   # 6. Refresh the cnpg-grafana dashboard, confirm panels still populate
   ```

4. **Rollback path documented.** If step 3.5 shows dropped targets, the
   rollback is to flip `enablePodMonitor: true` and delete the manual
   PodMonitor. Note this in the commit message.

## 5. Out of scope

- Any work on `monitoring/kps-cnpg-values.yaml` itself.
  `podMonitorSelectorNilUsesHelmValues: false` stays as is.
- Cross-mode PodMonitor (k3s/k8s/min) — defer.
- The `cluster=knoe-db` label vs. dynamic templating — currently hardcoded
  to `knoe-db`. If a second CNPG cluster ever lands, the manual PodMonitor
  needs to be parameterized. Leave a TODO comment but don't solve.
- Don't touch the cnpg-grafana dashboard transforms
  (`monitoring/cnpg-dashboard-transforms.yaml`) — those are unrelated.

## 6. Commit shape

```
feat(monitoring): manage knoe-db PodMonitor manually (drop CNPG-deprecated fields)

CNPG operator emits deprecation warnings on every cluster.spec.monitoring
patch:
  - enablePodMonitor will be removed in a future release
  - podMonitorRelabelings will be removed in a future release

Migrate to a manually-managed PodMonitor:
  + deploy/gcp/gke/knoe-db-podmonitor.yaml   (NEW — same spec as auto-created)
  ~ deploy/gcp/gke/knoe-db.yaml              (enablePodMonitor: true → false,
                                              podMonitorRelabelings dropped)
  ~ etc/init_cnpg_gke.sh                     (apply PodMonitor after cluster)

Verified live: cnpg-grafana dashboard panels render unchanged after the
flip; 3 postgres-exporter targets active in cnpg-prometheus.

Closes queue item #13 in docs/TODO.md.
```

## 7. Definition of done

- [ ] `deploy/gcp/gke/knoe-db-podmonitor.yaml` created, content verified
  against live PodMonitor spec.
- [ ] `deploy/gcp/gke/knoe-db.yaml` no longer contains
  `podMonitorRelabelings`; `enablePodMonitor: false` with comment.
- [ ] `etc/init_cnpg_gke.sh` applies the new manifest after cluster apply.
- [ ] Live cnpg-grafana dashboard verified after the transition: panels
  populate, 3 postgres-exporter targets in Prometheus.
- [ ] No new deprecation warnings on `kubectl apply` of the cluster manifest.
- [ ] `docs/TODO.md` queue item #13 archived to **Done** with verification
  notes.