From 500c9b131707fcfec62691a92aca07359c9a6ba2 Mon Sep 17 00:00:00 2001 From: chrisfu Date: Fri, 1 May 2026 15:44:12 -0700 Subject: [PATCH] fix(installer): env-contamination guard against shell-context / config mismatch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Filed in response to the 2026-04-28 14:00 UTC backup outage. An `install.sh --mode k3d` run with the shell pointed at GKE silently overwrote the GKE cluster's GCS-backed ObjectStore + ScheduledBackup with k3d-mode defaults; Garage filled up and CNPG backups failed for hours before the next manual check. The class of bug is "config says target cluster A, shell context says target cluster B, installer proceeds against B without warning." New shared bash helper at etc/preflight_kubecontext.sh with two functions: - verify_kubecontext_matches_config Strict gate. Reads [Global] APP_CLUSTER_KUBECONTEXT from the config and exits 1 if `kubectl config current-context` differs. Skipped silently when the config has no baked APP_CLUSTER_KUBECONTEXT (e.g. fresh k3d.cfg) or when there's no live current-context. - print_kubecontext_notice Informational. Prints what's about to be inherited so the user can abort before the TUI launches if it looks wrong. Never fails. Wiring: - deploy.sh sources the helper and calls the strict gate against ${PROLE_DEPLOY_CFG:-conf/gke.cfg} before invoking Python. Unattended path -> hard refusal on mismatch. - install.sh sources the helper and calls the informational notice (gated on not-`--min`) right after entering the local-checkout branch. The TUI is interactive, so the strict mode-aware gate is a follow-up once the welcome screen records a mode in state.inputs. Bypass for deliberate cross-cluster maintenance: KNOE_SKIP_KUBECONTEXT_GUARD=true ./deploy.sh End-to-end verified: - deploy.sh with current=cnpg-0, gke.cfg=app-0 -> exit 1, clear msg - deploy.sh with KNOE_SKIP_...=true -> bypasses, prints "skipping check" - install.sh --min -> notice skipped - install.sh (no flag) and install.sh --silent -> notice printed Doc updates: - CLAUDE.md §"Env-contamination warning" rewritten to describe the live guard (was a forward-looking TODO). - CLAUDE.md drift table row R4 removed; "Closed 2026-05-01" line added. - docs/TODO.md queue item #1 archived to Done; R4 dropped from the reality-vs-intent table. Queue numbering retained (no #1 placeholder) so the docs/plans/junie/-...md filenames still match. Co-Authored-By: Claude Opus 4.7 (1M context) --- CLAUDE.md | 27 +++++++ deploy.sh | 10 +++ docs/TODO.md | 8 ++- etc/preflight_kubecontext.sh | 134 +++++++++++++++++++++++++++++++++++ install.sh | 20 ++++++ 5 files changed, 197 insertions(+), 2 deletions(-) create mode 100755 etc/preflight_kubecontext.sh diff --git a/CLAUDE.md b/CLAUDE.md index ee952a9..a77540e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -124,6 +124,19 @@ REGISTRY_NAMESPACE = knoe-system Missing `init_cluster.app_cluster_kubecontext` → `_cluster_kubecontext("app")` returns `""` → installer falls back to `Global.KUBECONTEXT` for **both** app and db environments → **Garage deploys to knoe-cnpg-0** (wrong). +> **Env-contamination guard (live):** `deploy.sh` calls +> [`etc/preflight_kubecontext.sh`](etc/preflight_kubecontext.sh) and refuses +> to proceed when `kubectl config current-context` doesn't match the +> `[Global] APP_CLUSTER_KUBECONTEXT` of the active config. `install.sh` +> prints the inherited context up-front (mode-aware strict gate is the +> Python TUI's responsibility once the welcome screen records a mode). +> Bypass with `KNOE_SKIP_KUBECONTEXT_GUARD=true` for deliberate +> cross-cluster maintenance. **History:** the guard was filed in response +> to the 2026-04-28 14:00 UTC outage — an `install.sh --mode k3d` run with +> the shell pointed at GKE replaced the GCS-backed ObjectStore with a +> Garage-backed one, then Garage filled up and backups silently failed for +> hours. Closes drift R4 / queue item #1. + ### Get current CNPG node names ```bash @@ -161,3 +174,17 @@ DEFAULT_DB_CLUSTER_MACHINE_TYPE = "e2-standard-2" - Artifact Registry: `us-west3-docker.pkg.dev/plenary-truck-485623-p7/knoe-system` - CPU quota: 16 vCPUs (all regions) — 2 clusters × 3 × `e2-standard-2` = 12 vCPUs used - SSD quota: `SSD_TOTAL_GB = 300 GB` — fully consumed by CNPG; all other PVCs must use `standard` (pd-standard / HDD) + +--- + +## Reality TODOs / Drift Log + +Quick reference. Each entry links to the master index where context, owner, and rank live. + +| # | Drift | Where described above | Where tracked | +|---|---|---|---| +| _(none currently)_ | | | | + +**Closed in 2026-04-29 stabilization session:** Garage on DB cluster removed; cluster pods migrated to `cnpg-backup-sa` via CNPG v1.29.0 `spec.serviceAccountName`; both operators restarted clean. + +**Closed 2026-05-01:** R4 — installer env-contamination guard now live in `deploy.sh` (strict) + `install.sh` (informational notice). Helper at [`etc/preflight_kubecontext.sh`](etc/preflight_kubecontext.sh). diff --git a/deploy.sh b/deploy.sh index 84bfdc4..74b50e6 100755 --- a/deploy.sh +++ b/deploy.sh @@ -4,6 +4,16 @@ set -euo pipefail ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" CONFIG_PATH="${PROLE_DEPLOY_CFG:-${ROOT_DIR}/conf/gke.cfg}" +# Env-contamination guard. Refuses to proceed if `kubectl config +# current-context` doesn't match the [Global] APP_CLUSTER_KUBECONTEXT in +# the config we're about to deploy with. Filed in response to the +# 2026-04-28 14:00 UTC outage; see docs/TODO.md queue item #1 + drift R4. +# Bypass with KNOE_SKIP_KUBECONTEXT_GUARD=true if you're doing deliberate +# cross-cluster maintenance. +# shellcheck source=etc/preflight_kubecontext.sh +source "${ROOT_DIR}/etc/preflight_kubecontext.sh" +verify_kubecontext_matches_config "${CONFIG_PATH}" + if [[ -x "${ROOT_DIR}/.venv/bin/python3" ]]; then PYTHON_BIN="${ROOT_DIR}/.venv/bin/python3" elif [[ -x "${ROOT_DIR}/bin/python3" ]]; then diff --git a/docs/TODO.md b/docs/TODO.md index bb92adc..82c6300 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -25,7 +25,10 @@ The Kanban "Now" section at top is the only place this doc imposes structure. Ev ## Next (ranked queue — top of list = next-most-important) -1. **Installer env-contamination guard** — `install.sh` / `deploy.sh` should refuse to run if `${KUBECONTEXT}` from `conf/${mode}.cfg` doesn't match `kubectl config current-context` (or doesn't match what was passed via `--context`). Caused the 2026-04-28 14:00 UTC backup outage: an `install.sh --mode k3d` run with a GKE shell context overwrote the GCS-backed ObjectStore with a Garage-backed one, then Garage filled up. See [`CLAUDE.md`](../CLAUDE.md) §"Env-contamination warning". +> Note: numbering retained from the original queue (no #1) so the Junie +> briefs in [`plans/junie/`](plans/junie/) keep their `-…` filename +> match. #1 (installer env-contamination guard) shipped 2026-05-01; see +> the **Done** section. 2. **k3s manifest cleanup: `prole-*` filename rename** — `deploy/opentofu/k3s/manifests/knoe/prole-{configmap,deployment,service,kdc-configmap,kdc-secrets.example}.yaml` still carry the old prefix. The kustomization expects `knoe-*` names — kustomize is currently broken until rename. Same `git mv` pattern we used for `prole-auth-*` → `knoe-auth-*` in commit [b355855](#). **Brief: [`docs/plans/junie/02-k3s-prole-rename.md`](plans/junie/02-k3s-prole-rename.md). Assigned to Junie.** @@ -73,7 +76,6 @@ Items where docs describe an intended state that doesn't match live state. Each | # | Reality (current) | Intended state | Documented at | Tracked work item | |---|---|---|---|---| -| R4 | Installer can clobber GKE state when run in `k3d` mode with a GKE shell context (caused 2026-04-28 14:00 UTC outage) | Installer refuses to run on context mismatch | [`CLAUDE.md`](../CLAUDE.md) §"install.sh / deploy.sh pre-flight" warning | Queue item #1 | | R5 | k3s manifests `prole-{configmap,deployment,service,kdc-*}.yaml` still carry old prefix | All renamed to `knoe-*` (kustomization already references `knoe-*`) | Phase C plan in `~/.claude/plans/we-re-continuing-work-on-happy-toast.md` follow-ups | Queue item #2 | | R6 | Image refs `knoe-authority:latest` in deploy manifests | `knoe-auth:latest` (matches Maven artifact `knoe-auth.jar` post-rename) | Phase C plan follow-ups | Queue item #3 | | R8 | `etc/init_cnpg_gke.sh` doesn't set `cluster.spec.serviceAccountName` or extend the role bindings to `cnpg-backup-sa` | Script lands a fresh deploy in the same end state as live (with `serviceAccountName: cnpg-backup-sa` and role-binding subjects added) | Live cluster spec + RoleBindings | Queue item #7 | @@ -106,6 +108,8 @@ Items where docs describe an intended state that doesn't match live state. Each *(items get archived here when they land, with a date and commit reference; or just delete if not worth memorializing)* +- 2026-05-01 — **Installer env-contamination guard** (queue #1 / drift R4) (commit pending). Filed in response to the 2026-04-28 14:00 UTC backup outage (an `install.sh --mode k3d` run with the shell pointed at GKE replaced the GCS-backed ObjectStore with a Garage-backed one). New shared bash helper [`etc/preflight_kubecontext.sh`](../etc/preflight_kubecontext.sh) with two functions: `verify_kubecontext_matches_config ` (strict gate; reads `[Global] APP_CLUSTER_KUBECONTEXT` from the config and exits 1 if `kubectl config current-context` differs), and `print_kubecontext_notice` (informational). [`deploy.sh`](../deploy.sh) sources the helper and calls the strict gate before invoking the Python deploy pipeline. [`install.sh`](../install.sh) calls the informational notice (gated on not-`--min`) so the user sees the inherited context up-front before the TUI launches; the mode-aware strict gate for the interactive path is a follow-up for the welcome screen once it records mode. Bypass with `KNOE_SKIP_KUBECONTEXT_GUARD=true` for deliberate cross-cluster maintenance. Verified end-to-end: deploy.sh refuses on mismatch with a clear remediation message; bypass override works; install.sh's `--min` path skips the notice; install.sh's no-flag and other-flag paths print the notice. CLAUDE.md drift table row R4 removed; §"Env-contamination warning" rewritten to describe the live guard. Closes the 2026-04-28 14:00 UTC outage feedback loop. + - 2026-05-01 — **Cross-cluster CNPG metrics via DB-cluster Prometheus** (commit pending). Solves the `cloudnative-pg` Grafana dashboard's "No data" problem from queue item #12. Picked path (c) over the originally-recommended (b) for DB-cluster autonomy + pattern symmetry — each cluster owns its observability footprint, and a future second DB cluster reuses the recipe one-for-one. Installed kube-prometheus-stack v84.3.0 on `knoe-dev-cnpg-0` (`monitoring` namespace), Grafana + Alertmanager disabled (single canonical Grafana stays on app cluster). All PVCs explicitly pinned to `standard-hdd` (per the user's standing rule on SSD-quota preservation; verified `prometheus-...-0` PVC bound 20Gi `standard-hdd`). New ILB `prometheus-cnpg-ilb` in DB-cluster `monitoring` ns at `10.180.15.216:9090`, reachable from app-cluster pods over the shared VPC. New `cnpg-prometheus` datasource added to app-cluster Grafana via `monitoring/kps-values-gke.yaml` `grafana.additionalDataSources`. **Side-fix**: removed an orphan `knoe-grafana-datasource` ConfigMap that `etc/init_monitoring.sh:apply_grafana_datasource()` had been creating with a duplicate `uid: prometheus` — caused Grafana's provisioning reload to fail with HTTP 500 once any second datasource arrived. Function neutered to a no-op with a clear comment; chart values are now the single source of truth for datasources. Verified end-to-end: Prometheus on DB cluster discovers all 3 CNPG postgres pods (knoe-db-2/3/4 at 10.24.x.x:9187, all `health=up`); Grafana datasource health endpoint returns `Successfully queried the Prometheus API`; `cnpg_pg_stat_archiver_archived_count` query through Grafana proxy returns live data (704 archived WAL files at time of test). To complete: switch each cnpg-grafana dashboard's `DS_PROMETHEUS` template variable to `cnpg-prometheus` (one-click). New files: [`monitoring/kps-cnpg-values.yaml`](../monitoring/kps-cnpg-values.yaml), [`deploy/gcp/gke/cnpg-prometheus-ilb.yaml`](../deploy/gcp/gke/cnpg-prometheus-ilb.yaml). Modified: [`monitoring/kps-values-gke.yaml`](../monitoring/kps-values-gke.yaml) (datasource added), [`etc/init_monitoring.sh`](../etc/init_monitoring.sh) (function no-op'd). Out of scope (Phase 2): federate alerts back to app-cluster Alertmanager; reserve a static internal IP so the ILB doesn't drift on re-create. - 2026-04-30 — **Bump postgres pod memory limit 512Mi → 2Gi** (commit pending). Long-standing throughput cliff: `barman-cloud-backup` is single-threaded gzip + GCS upload buffering, ceiling-bound by the pod's memory limit. The 9 GB DB took 30–90 min to back up at 512Mi; 2Gi cuts that to 5–10 min. requests left at 128Mi (no scheduling change; this is purely a ceiling bump). Patched live with `kubectl patch cluster knoe-db --type=merge -p '{"spec":{"resources":{"limits":{"memory":"2Gi"}}}}'`; CNPG operator performed a supervised rolling restart (replicas first, primary last). Repo source updated at [`deploy/gcp/gke/knoe-db.yaml`](../deploy/gcp/gke/knoe-db.yaml). Pre-flight verified no external psql sessions (chrisfu/ron not connected); 39 internal sessions FATAL'd through the rolling restart and reconnected via their pools — supabase + gitlab + streaming_replica all stable post-rollout. Closes drift R3. k3s + k8s mode resource limits remain at 512Mi (not exposed in those manifests today); cross-mode parity folded into queue item #8 ("Port oauth2-proxy manifest to k3d/k3s/min modes"). diff --git a/etc/preflight_kubecontext.sh b/etc/preflight_kubecontext.sh new file mode 100755 index 0000000..07d8778 --- /dev/null +++ b/etc/preflight_kubecontext.sh @@ -0,0 +1,134 @@ +#!/usr/bin/env bash +# preflight_kubecontext.sh +# +# Shared helper to detect the "wrong shell context for the install mode" +# class of bug. Sourced by install.sh and deploy.sh. +# +# Exposes two functions: +# +# verify_kubecontext_matches_config +# Strict gate. Reads APP_CLUSTER_KUBECONTEXT from the config's [Global] +# section and refuses to proceed if `kubectl config current-context` +# doesn't match. Used by deploy.sh (unattended path: silent mismatches +# are dangerous). +# +# Returns 0 on match (or when the config has no APP_CLUSTER_KUBECONTEXT, +# e.g. fresh k3d setup). Exits 1 with a clear error on mismatch. +# Set KNOE_SKIP_KUBECONTEXT_GUARD=true to bypass (escape hatch for +# deliberate cross-cluster maintenance). +# +# print_kubecontext_notice +# Informational. Prints the current kubectl context (or "(none)") so +# the user sees what install.sh is about to inherit before the TUI +# launches. Never exits or fails. Used by install.sh. +# +# History: this guard was filed in response to the 2026-04-28 14:00 UTC +# outage. An install.sh run in k3d mode with the shell pointed at GKE +# overwrote the GKE-cluster ObjectStore + ScheduledBackup with k3d-mode +# defaults, and CNPG backups silently failed for hours until the next +# manual check. See CLAUDE.md §"Env-contamination warning" and +# docs/TODO.md queue item #1 (drift R4). + +# Read APP_CLUSTER_KUBECONTEXT from the [Global] section of an INI-style +# config. Empty string if absent. Quote-stripping is best-effort. +_kubectx_from_config() { + local cfg="$1" + [[ -f "$cfg" ]] || { echo ""; return 0; } + # awk: print value when we're in [Global] and key matches. + # Strips surrounding whitespace and quotes. + awk ' + /^\[/{section=$0; next} + section=="[Global]" && /^[[:space:]]*APP_CLUSTER_KUBECONTEXT[[:space:]]*=/ { + sub(/^[^=]*=[[:space:]]*/, "", $0) + sub(/^"/, "", $0); sub(/"$/, "", $0) + sub(/^'\''/, "", $0); sub(/'\''$/, "", $0) + print $0 + exit + } + ' "$cfg" +} + +# Read the live current-context, or empty if kubectl/config unavailable. +_kubectx_current() { + command -v kubectl >/dev/null 2>&1 || { echo ""; return 0; } + kubectl config current-context 2>/dev/null || true +} + +# verify_kubecontext_matches_config +# Strict gate. Exits 1 on mismatch unless KNOE_SKIP_KUBECONTEXT_GUARD=true. +verify_kubecontext_matches_config() { + local cfg_path="${1:-}" + if [[ -z "$cfg_path" ]]; then + echo "preflight_kubecontext: usage: verify_kubecontext_matches_config " >&2 + return 2 + fi + if [[ "${KNOE_SKIP_KUBECONTEXT_GUARD:-false}" == "true" ]]; then + echo "preflight_kubecontext: KNOE_SKIP_KUBECONTEXT_GUARD=true — skipping check (escape hatch)." >&2 + return 0 + fi + + local expected + expected="$(_kubectx_from_config "$cfg_path")" + + # No baked context in the config (e.g. fresh k3d.cfg) → nothing to check. + if [[ -z "$expected" ]]; then + return 0 + fi + + local actual + actual="$(_kubectx_current)" + + # No live current-context → user hasn't selected one; the config is + # authoritative and downstream code will pass --context explicitly. + if [[ -z "$actual" ]]; then + return 0 + fi + + if [[ "$expected" == "$actual" ]]; then + return 0 + fi + + cat >&2 < kubectl current-context: (none set)" + else + echo "==> kubectl current-context: $actual" + fi + echo " The mode you select must target this cluster, OR you must switch" + echo " context (kubectl config use-context …) before proceeding." +} diff --git a/install.sh b/install.sh index 6468768..2deefc9 100755 --- a/install.sh +++ b/install.sh @@ -81,6 +81,26 @@ if [[ -n "${_script_dir}" && -d "${_script_dir}/knoe" ]]; then # Local checkout: delegate to the Python installer module. cd "${_script_dir}" + # kubectl-context notice. install.sh defers mode selection to the Python + # TUI, so we can't strictly verify here — but printing the inherited + # context up-front gives the user a chance to abort before the TUI + # launches if it looks wrong (e.g. shell on GKE while planning a k3d + # install). Filed against docs/TODO.md queue item #1 (drift R4); the + # mode-aware strict guard runs in deploy.sh (which knows its config) and + # should also run Python-side once the welcome screen records a mode. + if [[ -f "${_script_dir}/etc/preflight_kubecontext.sh" ]]; then + _is_min=0 + for arg in "$@"; do + if [[ "$arg" == "--min" ]]; then _is_min=1; break; fi + done + if [[ "$_is_min" -eq 0 ]]; then + # shellcheck source=etc/preflight_kubecontext.sh + source "${_script_dir}/etc/preflight_kubecontext.sh" + print_kubecontext_notice + fi + unset _is_min + fi + # 1Password preflight — sign in, create 'knoey' vault, ensure 'administrator' item. # Skipped in --min mode (init_1password.sh handles the flag check itself). if [[ -f "${_script_dir}/etc/init_1password.sh" ]]; then