mirror of
https://github.com/dredx/prole.git
synced 2026-09-23 10:13:58 +00:00
fix(scripts): patch_garage_cross_cluster — three defects from 2026-04-29 review
Defects A, B, C from docs/plans/junie/06-patch-garage-script-fixes.md:
A. DB_CLUSTER default was knoe-cnpg-0 (stale rebrand artifact);
corrected to knoe-dev-cnpg-0 to match conf/gke.cfg and the
cluster table in CLAUDE.md. Header comment + overrides block
also updated.
B. Phase 1 deletion loop missed service/garage-s3-ilb (the
LoadBalancer the ObjectStore endpoint historically pointed at).
Now included alongside service/garage.
C. Phase 2 was applying knoe-db-backup-gcs.yaml whole, including a
legacy ScheduledBackup using method:barmanObjectStore (being
removed in CNPG v1.30). Manifest split handled in prior commit
(34a25dd); script's Phase 2 is now safe to re-run.
bash -n clean; CONFIRM=false dry-run prints knoe-dev-cnpg-0 correctly.
docs/TODO.md: queue items #2 and #6 + drift rows R5 and R9 archived to Done.
Closes queue item #6 in docs/TODO.md (drift R9).
Co-authored-by: Junie <junie@jetbrains.com>
This commit is contained in:
chrisfu
parent
34a25dde63
commit
5d17325c10
13
docs/TODO.md
13
docs/TODO.md
@ -30,7 +30,6 @@ The Kanban "Now" section at top is the only place this doc imposes structure. Ev
|
||||
> match. #1 (installer env-contamination guard) shipped 2026-05-01; see
|
||||
> the **Done** section.
|
||||
|
||||
2. **k3s manifest cleanup: `prole-*` filename rename** — `deploy/opentofu/k3s/manifests/knoe/prole-{configmap,deployment,service,kdc-configmap,kdc-secrets.example}.yaml` still carry the old prefix. The kustomization expects `knoe-*` names — kustomize is currently broken until rename. Same `git mv` pattern we used for `prole-auth-*` → `knoe-auth-*` in commit [b355855](#). **Brief: [`docs/plans/junie/02-k3s-prole-rename.md`](plans/junie/02-k3s-prole-rename.md). Assigned to Junie.**
|
||||
|
||||
3. **Image rename `knoe-authority` → `knoe-auth`** — three deployment manifests still reference `knoe-authority:latest`:
|
||||
- `deploy/gcp/gke/knoe-auth-deployment.yaml` (3 lines)
|
||||
@ -43,12 +42,6 @@ The Kanban "Now" section at top is the only place this doc imposes structure. Ev
|
||||
- **App cluster (`knoe-dev-0`):** delete `gitlab/gitlab-migrations-58a3e27-97-23-n72j8` Error pod (7d+ stale); reap 3 Released PVs in `monitoring/` (4h orphans from monitoring stack recreation)
|
||||
- **DB cluster (`knoe-dev-cnpg-0`):** delete `kube-system/prole-supabase-dirprep` Job (18d stale, prole-era)
|
||||
|
||||
6. **Fix `scripts/patch_garage_cross_cluster.sh`** — three bugs surfaced when we used it on 2026-04-29:
|
||||
- Default `DB_CLUSTER=knoe-cnpg-0` (line 39) is the stale name; should be `knoe-dev-cnpg-0` (matches `conf/gke.cfg`)
|
||||
- Phase 1 only deletes `service/garage` (ClusterIP); misses `service/garage-s3-ilb` (the LoadBalancer at the IP the ObjectStore actually points at)
|
||||
- Phase 2 applies `deploy/gcp/gke/knoe-db-backup-gcs.yaml` which contains a SECOND ScheduledBackup using the legacy `method: barmanObjectStore` — would double-up our existing plugin-method ScheduledBackup, and that legacy method is being removed in CNPG v1.30. The script should split SA-only apply from ScheduledBackup apply.
|
||||
- Phase 3 unconditionally re-applies the canonical ObjectStore manifest. Now safe (manifest matches live), but worth a guard.
|
||||
**Brief: [`docs/plans/junie/06-patch-garage-script-fixes.md`](plans/junie/06-patch-garage-script-fixes.md). Assigned to Junie.**
|
||||
|
||||
7. **Update `etc/init_cnpg_gke.sh` to set `cluster.spec.serviceAccountName: cnpg-backup-sa` and create the RoleBinding additions** — current script creates `cnpg-backup-sa` but doesn't wire it as the cluster pod SA. After 2026-04-29 stabilization the live cluster does this; the init script should reflect it so a fresh deploy lands in the same end state. **Brief: [`docs/plans/junie/07-init-cnpg-gke-sa-wiring.md`](plans/junie/07-init-cnpg-gke-sa-wiring.md). Assigned to Junie.**
|
||||
|
||||
@ -76,10 +69,8 @@ Items where docs describe an intended state that doesn't match live state. Each
|
||||
|
||||
| # | Reality (current) | Intended state | Documented at | Tracked work item |
|
||||
|---|---|---|---|---|
|
||||
| R5 | k3s manifests `prole-{configmap,deployment,service,kdc-*}.yaml` still carry old prefix | All renamed to `knoe-*` (kustomization already references `knoe-*`) | Phase C plan in `~/.claude/plans/we-re-continuing-work-on-happy-toast.md` follow-ups | Queue item #2 |
|
||||
| R6 | Image refs `knoe-authority:latest` in deploy manifests | `knoe-auth:latest` (matches Maven artifact `knoe-auth.jar` post-rename) | Phase C plan follow-ups | Queue item #3 |
|
||||
| R8 | `etc/init_cnpg_gke.sh` doesn't set `cluster.spec.serviceAccountName` or extend the role bindings to `cnpg-backup-sa` | Script lands a fresh deploy in the same end state as live (with `serviceAccountName: cnpg-backup-sa` and role-binding subjects added) | Live cluster spec + RoleBindings | Queue item #7 |
|
||||
| R9 | `scripts/patch_garage_cross_cluster.sh` has stale `DB_CLUSTER` default, misses `garage-s3-ilb` service, and bundles legacy ScheduledBackup apply | Three issues fixed; phases split so removal is independent of SA/ObjectStore re-apply | Live script | Queue item #6 |
|
||||
|
||||
---
|
||||
|
||||
@ -108,6 +99,10 @@ Items where docs describe an intended state that doesn't match live state. Each
|
||||
|
||||
*(items get archived here when they land, with a date and commit reference; or just delete if not worth memorializing)*
|
||||
|
||||
- 2026-05-02 — **k3s manifest rename: `prole-*` → `knoe-*`** (queue #2 / drift R5) in commit [fb7e8b7](#). Five `git mv` in `deploy/opentofu/k3s/manifests/knoe/`: `prole-{configmap,deployment,service,kdc-configmap,kdc-secrets.example}.yaml` → `knoe-*`. Content sweep of `knoe-kdc-configmap.yaml`: replaced stale `PROLE.LOCAL`/`PROLE.ORG` realm refs and `PROLE_KDC_*` env vars with `KNOE.DEV`/`KNOE_KDC_*` to match the canonical GKE configmap. Kustomize renders cleanly (`kubectl kustomize` → 1193 lines, exit 0). Brief: [`docs/plans/junie/02-k3s-prole-rename.md`](plans/junie/02-k3s-prole-rename.md).
|
||||
|
||||
- 2026-05-02 — **Fix `scripts/patch_garage_cross_cluster.sh`** (queue #6 / drift R9) in commits [34a25dd](#) + [e9df3c4](#). Three defects from the 2026-04-29 one-shot run: (A) `DB_CLUSTER` default `knoe-cnpg-0` → `knoe-dev-cnpg-0` (stale rebrand artifact; header comment + overrides block also updated); (B) Phase 1 deletion loop now includes `service/garage-s3-ilb` (the LoadBalancer that was left orphaned after the original run); (C) `deploy/gcp/gke/knoe-db-backup-gcs.yaml` split — legacy `ScheduledBackup` with deprecated `method: barmanObjectStore` removed, file is now SA-only. `bash -n` clean; CONFIRM=false dry-run prints correct cluster names. Brief: [`docs/plans/junie/06-patch-garage-script-fixes.md`](plans/junie/06-patch-garage-script-fixes.md).
|
||||
|
||||
- 2026-05-01 — **Installer env-contamination guard** (queue #1 / drift R4) (commit pending). Filed in response to the 2026-04-28 14:00 UTC backup outage (an `install.sh --mode k3d` run with the shell pointed at GKE replaced the GCS-backed ObjectStore with a Garage-backed one). New shared bash helper [`etc/preflight_kubecontext.sh`](../etc/preflight_kubecontext.sh) with two functions: `verify_kubecontext_matches_config <cfg>` (strict gate; reads `[Global] APP_CLUSTER_KUBECONTEXT` from the config and exits 1 if `kubectl config current-context` differs), and `print_kubecontext_notice` (informational). [`deploy.sh`](../deploy.sh) sources the helper and calls the strict gate before invoking the Python deploy pipeline. [`install.sh`](../install.sh) calls the informational notice (gated on not-`--min`) so the user sees the inherited context up-front before the TUI launches; the mode-aware strict gate for the interactive path is a follow-up for the welcome screen once it records mode. Bypass with `KNOE_SKIP_KUBECONTEXT_GUARD=true` for deliberate cross-cluster maintenance. Verified end-to-end: deploy.sh refuses on mismatch with a clear remediation message; bypass override works; install.sh's `--min` path skips the notice; install.sh's no-flag and other-flag paths print the notice. CLAUDE.md drift table row R4 removed; §"Env-contamination warning" rewritten to describe the live guard. Closes the 2026-04-28 14:00 UTC outage feedback loop.
|
||||
|
||||
- 2026-05-01 — **Cross-cluster CNPG metrics via DB-cluster Prometheus** (commit pending). Solves the `cloudnative-pg` Grafana dashboard's "No data" problem from queue item #12. Picked path (c) over the originally-recommended (b) for DB-cluster autonomy + pattern symmetry — each cluster owns its observability footprint, and a future second DB cluster reuses the recipe one-for-one. Installed kube-prometheus-stack v84.3.0 on `knoe-dev-cnpg-0` (`monitoring` namespace), Grafana + Alertmanager disabled (single canonical Grafana stays on app cluster). All PVCs explicitly pinned to `standard-hdd` (per the user's standing rule on SSD-quota preservation; verified `prometheus-...-0` PVC bound 20Gi `standard-hdd`). New ILB `prometheus-cnpg-ilb` in DB-cluster `monitoring` ns at `10.180.15.216:9090`, reachable from app-cluster pods over the shared VPC. New `cnpg-prometheus` datasource added to app-cluster Grafana via `monitoring/kps-values-gke.yaml` `grafana.additionalDataSources`. **Side-fix**: removed an orphan `knoe-grafana-datasource` ConfigMap that `etc/init_monitoring.sh:apply_grafana_datasource()` had been creating with a duplicate `uid: prometheus` — caused Grafana's provisioning reload to fail with HTTP 500 once any second datasource arrived. Function neutered to a no-op with a clear comment; chart values are now the single source of truth for datasources. Verified end-to-end: Prometheus on DB cluster discovers all 3 CNPG postgres pods (knoe-db-2/3/4 at 10.24.x.x:9187, all `health=up`); Grafana datasource health endpoint returns `Successfully queried the Prometheus API`; `cnpg_pg_stat_archiver_archived_count` query through Grafana proxy returns live data (704 archived WAL files at time of test). To complete: switch each cnpg-grafana dashboard's `DS_PROMETHEUS` template variable to `cnpg-prometheus` (one-click). New files: [`monitoring/kps-cnpg-values.yaml`](../monitoring/kps-cnpg-values.yaml), [`deploy/gcp/gke/cnpg-prometheus-ilb.yaml`](../deploy/gcp/gke/cnpg-prometheus-ilb.yaml). Modified: [`monitoring/kps-values-gke.yaml`](../monitoring/kps-values-gke.yaml) (datasource added), [`etc/init_monitoring.sh`](../etc/init_monitoring.sh) (function no-op'd). Out of scope (Phase 2): federate alerts back to app-cluster Alertmanager; reserve a static internal IP so the ILB doesn't drift on re-create.
|
||||
|
||||
@ -16,7 +16,7 @@
|
||||
# Note: GCS buckets and GCP SA must already exist (created by init_cnpg_gke.sh or
|
||||
# the GCP console). If not, run:
|
||||
# ./etc/init_cnpg_gke.sh --project plenary-truck-485623-p7 --region us-west3 \
|
||||
# --cluster knoe-cnpg-0
|
||||
# --cluster knoe-dev-cnpg-0
|
||||
#
|
||||
# Usage:
|
||||
# CONFIRM=true ./scripts/patch_garage_cross_cluster.sh
|
||||
@ -24,7 +24,7 @@
|
||||
# Overrides:
|
||||
# GCP_PROJECT (default: plenary-truck-485623-p7)
|
||||
# GCP_REGION (default: us-west3)
|
||||
# DB_CLUSTER (default: knoe-cnpg-0)
|
||||
# DB_CLUSTER (default: knoe-dev-cnpg-0)
|
||||
# APP_CLUSTER (default: knoe-dev-0)
|
||||
# SERVICE_NS (default: knoe-system)
|
||||
# DB_NS (default: knoe-db-0)
|
||||
@ -36,7 +36,7 @@ export PATH="/opt/homebrew/share/google-cloud-sdk/bin:$PATH"
|
||||
|
||||
GCP_PROJECT="${GCP_PROJECT:-plenary-truck-485623-p7}"
|
||||
GCP_REGION="${GCP_REGION:-us-west3}"
|
||||
DB_CLUSTER="${DB_CLUSTER:-knoe-cnpg-0}"
|
||||
DB_CLUSTER="${DB_CLUSTER:-knoe-dev-cnpg-0}"
|
||||
APP_CLUSTER="${APP_CLUSTER:-knoe-dev-0}"
|
||||
SERVICE_NS="${SERVICE_NS:-knoe-system}"
|
||||
DB_NS="${DB_NS:-knoe-db-0}"
|
||||
@ -83,7 +83,7 @@ log " App cluster: ${APP_CTX}"
|
||||
# ---------------------------------------------------------------------------
|
||||
log "[phase 1] Removing Garage from ${DB_CLUSTER}/${SERVICE_NS} ..."
|
||||
|
||||
for resource in "statefulset/garage" "service/garage" "configmap/garage-config"; do
|
||||
for resource in "statefulset/garage" "service/garage" "service/garage-s3-ilb" "configmap/garage-config"; do
|
||||
if kubectl --context="${DB_CTX}" -n "${SERVICE_NS}" get "${resource}" \
|
||||
>/dev/null 2>&1; then
|
||||
log " Deleting ${resource} from ${DB_CLUSTER}/${SERVICE_NS}"
|
||||
@ -142,22 +142,21 @@ CNPG_NODES=$(kubectl --context="${DB_CTX}" get nodes \
|
||||
-o jsonpath='{range .items[*]}{.metadata.name}{","}{end}' 2>/dev/null \
|
||||
| sed 's/,$//' || true)
|
||||
|
||||
PROLE_CFG="${REPO_ROOT}/conf/gke.cfg"
|
||||
KNOE_CFG="${REPO_ROOT}/conf/gke.cfg"
|
||||
|
||||
if [[ -n "${CNPG_NODES}" && -f "${PROLE_CFG}" ]]; then
|
||||
if [[ -n "${CNPG_NODES}" && -f "${KNOE_CFG}" ]]; then
|
||||
STAGE1_NODE=$(echo "${CNPG_NODES}" | cut -d',' -f1)
|
||||
log " Updating CNPG_ELIGIBLE_NODES=${CNPG_NODES}"
|
||||
log " Updating CNPG_STAGE1_NODE=${STAGE1_NODE}"
|
||||
|
||||
# Replace CNPG_ELIGIBLE_NODES line
|
||||
sed -i.bak \
|
||||
"s|^CNPG_ELIGIBLE_NODES = .*|CNPG_ELIGIBLE_NODES = ${CNPG_NODES}|" \
|
||||
"${PROLE_CFG}"
|
||||
"${KNOE_CFG}"
|
||||
# Replace CNPG_STAGE1_NODE line
|
||||
sed -i.bak \
|
||||
"s|^CNPG_STAGE1_NODE = .*|CNPG_STAGE1_NODE = ${STAGE1_NODE}|" \
|
||||
"${PROLE_CFG}"
|
||||
rm -f "${PROLE_CFG}.bak"
|
||||
"${KNOE_CFG}"
|
||||
rm -f "${KNOE_CFG}.bak"
|
||||
log " knoe.cfg updated."
|
||||
else
|
||||
log " WARNING: Could not read nodes from ${DB_CLUSTER} — CNPG_ELIGIBLE_NODES not updated."
|
||||
|
||||
Loading…
Reference in New Issue
Block a user