diff --git a/CLAUDE.md b/CLAUDE.md index ee952a9..a77540e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -124,6 +124,19 @@ REGISTRY_NAMESPACE = knoe-system Missing `init_cluster.app_cluster_kubecontext` → `_cluster_kubecontext("app")` returns `""` → installer falls back to `Global.KUBECONTEXT` for **both** app and db environments → **Garage deploys to knoe-cnpg-0** (wrong). +> **Env-contamination guard (live):** `deploy.sh` calls +> [`etc/preflight_kubecontext.sh`](etc/preflight_kubecontext.sh) and refuses +> to proceed when `kubectl config current-context` doesn't match the +> `[Global] APP_CLUSTER_KUBECONTEXT` of the active config. `install.sh` +> prints the inherited context up-front (mode-aware strict gate is the +> Python TUI's responsibility once the welcome screen records a mode). +> Bypass with `KNOE_SKIP_KUBECONTEXT_GUARD=true` for deliberate +> cross-cluster maintenance. **History:** the guard was filed in response +> to the 2026-04-28 14:00 UTC outage — an `install.sh --mode k3d` run with +> the shell pointed at GKE replaced the GCS-backed ObjectStore with a +> Garage-backed one, then Garage filled up and backups silently failed for +> hours. Closes drift R4 / queue item #1. + ### Get current CNPG node names ```bash @@ -161,3 +174,17 @@ DEFAULT_DB_CLUSTER_MACHINE_TYPE = "e2-standard-2" - Artifact Registry: `us-west3-docker.pkg.dev/plenary-truck-485623-p7/knoe-system` - CPU quota: 16 vCPUs (all regions) — 2 clusters × 3 × `e2-standard-2` = 12 vCPUs used - SSD quota: `SSD_TOTAL_GB = 300 GB` — fully consumed by CNPG; all other PVCs must use `standard` (pd-standard / HDD) + +--- + +## Reality TODOs / Drift Log + +Quick reference. Each entry links to the master index where context, owner, and rank live. + +| # | Drift | Where described above | Where tracked | +|---|---|---|---| +| _(none currently)_ | | | | + +**Closed in 2026-04-29 stabilization session:** Garage on DB cluster removed; cluster pods migrated to `cnpg-backup-sa` via CNPG v1.29.0 `spec.serviceAccountName`; both operators restarted clean. + +**Closed 2026-05-01:** R4 — installer env-contamination guard now live in `deploy.sh` (strict) + `install.sh` (informational notice). Helper at [`etc/preflight_kubecontext.sh`](etc/preflight_kubecontext.sh). diff --git a/deploy.sh b/deploy.sh index 84bfdc4..74b50e6 100755 --- a/deploy.sh +++ b/deploy.sh @@ -4,6 +4,16 @@ set -euo pipefail ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" CONFIG_PATH="${PROLE_DEPLOY_CFG:-${ROOT_DIR}/conf/gke.cfg}" +# Env-contamination guard. Refuses to proceed if `kubectl config +# current-context` doesn't match the [Global] APP_CLUSTER_KUBECONTEXT in +# the config we're about to deploy with. Filed in response to the +# 2026-04-28 14:00 UTC outage; see docs/TODO.md queue item #1 + drift R4. +# Bypass with KNOE_SKIP_KUBECONTEXT_GUARD=true if you're doing deliberate +# cross-cluster maintenance. +# shellcheck source=etc/preflight_kubecontext.sh +source "${ROOT_DIR}/etc/preflight_kubecontext.sh" +verify_kubecontext_matches_config "${CONFIG_PATH}" + if [[ -x "${ROOT_DIR}/.venv/bin/python3" ]]; then PYTHON_BIN="${ROOT_DIR}/.venv/bin/python3" elif [[ -x "${ROOT_DIR}/bin/python3" ]]; then diff --git a/docs/TODO.md b/docs/TODO.md index bb92adc..82c6300 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -25,7 +25,10 @@ The Kanban "Now" section at top is the only place this doc imposes structure. Ev ## Next (ranked queue — top of list = next-most-important) -1. **Installer env-contamination guard** — `install.sh` / `deploy.sh` should refuse to run if `${KUBECONTEXT}` from `conf/${mode}.cfg` doesn't match `kubectl config current-context` (or doesn't match what was passed via `--context`). Caused the 2026-04-28 14:00 UTC backup outage: an `install.sh --mode k3d` run with a GKE shell context overwrote the GCS-backed ObjectStore with a Garage-backed one, then Garage filled up. See [`CLAUDE.md`](../CLAUDE.md) §"Env-contamination warning". +> Note: numbering retained from the original queue (no #1) so the Junie +> briefs in [`plans/junie/`](plans/junie/) keep their `-…` filename +> match. #1 (installer env-contamination guard) shipped 2026-05-01; see +> the **Done** section. 2. **k3s manifest cleanup: `prole-*` filename rename** — `deploy/opentofu/k3s/manifests/knoe/prole-{configmap,deployment,service,kdc-configmap,kdc-secrets.example}.yaml` still carry the old prefix. The kustomization expects `knoe-*` names — kustomize is currently broken until rename. Same `git mv` pattern we used for `prole-auth-*` → `knoe-auth-*` in commit [b355855](#). **Brief: [`docs/plans/junie/02-k3s-prole-rename.md`](plans/junie/02-k3s-prole-rename.md). Assigned to Junie.** @@ -73,7 +76,6 @@ Items where docs describe an intended state that doesn't match live state. Each | # | Reality (current) | Intended state | Documented at | Tracked work item | |---|---|---|---|---| -| R4 | Installer can clobber GKE state when run in `k3d` mode with a GKE shell context (caused 2026-04-28 14:00 UTC outage) | Installer refuses to run on context mismatch | [`CLAUDE.md`](../CLAUDE.md) §"install.sh / deploy.sh pre-flight" warning | Queue item #1 | | R5 | k3s manifests `prole-{configmap,deployment,service,kdc-*}.yaml` still carry old prefix | All renamed to `knoe-*` (kustomization already references `knoe-*`) | Phase C plan in `~/.claude/plans/we-re-continuing-work-on-happy-toast.md` follow-ups | Queue item #2 | | R6 | Image refs `knoe-authority:latest` in deploy manifests | `knoe-auth:latest` (matches Maven artifact `knoe-auth.jar` post-rename) | Phase C plan follow-ups | Queue item #3 | | R8 | `etc/init_cnpg_gke.sh` doesn't set `cluster.spec.serviceAccountName` or extend the role bindings to `cnpg-backup-sa` | Script lands a fresh deploy in the same end state as live (with `serviceAccountName: cnpg-backup-sa` and role-binding subjects added) | Live cluster spec + RoleBindings | Queue item #7 | @@ -106,6 +108,8 @@ Items where docs describe an intended state that doesn't match live state. Each *(items get archived here when they land, with a date and commit reference; or just delete if not worth memorializing)* +- 2026-05-01 — **Installer env-contamination guard** (queue #1 / drift R4) (commit pending). Filed in response to the 2026-04-28 14:00 UTC backup outage (an `install.sh --mode k3d` run with the shell pointed at GKE replaced the GCS-backed ObjectStore with a Garage-backed one). New shared bash helper [`etc/preflight_kubecontext.sh`](../etc/preflight_kubecontext.sh) with two functions: `verify_kubecontext_matches_config ` (strict gate; reads `[Global] APP_CLUSTER_KUBECONTEXT` from the config and exits 1 if `kubectl config current-context` differs), and `print_kubecontext_notice` (informational). [`deploy.sh`](../deploy.sh) sources the helper and calls the strict gate before invoking the Python deploy pipeline. [`install.sh`](../install.sh) calls the informational notice (gated on not-`--min`) so the user sees the inherited context up-front before the TUI launches; the mode-aware strict gate for the interactive path is a follow-up for the welcome screen once it records mode. Bypass with `KNOE_SKIP_KUBECONTEXT_GUARD=true` for deliberate cross-cluster maintenance. Verified end-to-end: deploy.sh refuses on mismatch with a clear remediation message; bypass override works; install.sh's `--min` path skips the notice; install.sh's no-flag and other-flag paths print the notice. CLAUDE.md drift table row R4 removed; §"Env-contamination warning" rewritten to describe the live guard. Closes the 2026-04-28 14:00 UTC outage feedback loop. + - 2026-05-01 — **Cross-cluster CNPG metrics via DB-cluster Prometheus** (commit pending). Solves the `cloudnative-pg` Grafana dashboard's "No data" problem from queue item #12. Picked path (c) over the originally-recommended (b) for DB-cluster autonomy + pattern symmetry — each cluster owns its observability footprint, and a future second DB cluster reuses the recipe one-for-one. Installed kube-prometheus-stack v84.3.0 on `knoe-dev-cnpg-0` (`monitoring` namespace), Grafana + Alertmanager disabled (single canonical Grafana stays on app cluster). All PVCs explicitly pinned to `standard-hdd` (per the user's standing rule on SSD-quota preservation; verified `prometheus-...-0` PVC bound 20Gi `standard-hdd`). New ILB `prometheus-cnpg-ilb` in DB-cluster `monitoring` ns at `10.180.15.216:9090`, reachable from app-cluster pods over the shared VPC. New `cnpg-prometheus` datasource added to app-cluster Grafana via `monitoring/kps-values-gke.yaml` `grafana.additionalDataSources`. **Side-fix**: removed an orphan `knoe-grafana-datasource` ConfigMap that `etc/init_monitoring.sh:apply_grafana_datasource()` had been creating with a duplicate `uid: prometheus` — caused Grafana's provisioning reload to fail with HTTP 500 once any second datasource arrived. Function neutered to a no-op with a clear comment; chart values are now the single source of truth for datasources. Verified end-to-end: Prometheus on DB cluster discovers all 3 CNPG postgres pods (knoe-db-2/3/4 at 10.24.x.x:9187, all `health=up`); Grafana datasource health endpoint returns `Successfully queried the Prometheus API`; `cnpg_pg_stat_archiver_archived_count` query through Grafana proxy returns live data (704 archived WAL files at time of test). To complete: switch each cnpg-grafana dashboard's `DS_PROMETHEUS` template variable to `cnpg-prometheus` (one-click). New files: [`monitoring/kps-cnpg-values.yaml`](../monitoring/kps-cnpg-values.yaml), [`deploy/gcp/gke/cnpg-prometheus-ilb.yaml`](../deploy/gcp/gke/cnpg-prometheus-ilb.yaml). Modified: [`monitoring/kps-values-gke.yaml`](../monitoring/kps-values-gke.yaml) (datasource added), [`etc/init_monitoring.sh`](../etc/init_monitoring.sh) (function no-op'd). Out of scope (Phase 2): federate alerts back to app-cluster Alertmanager; reserve a static internal IP so the ILB doesn't drift on re-create. - 2026-04-30 — **Bump postgres pod memory limit 512Mi → 2Gi** (commit pending). Long-standing throughput cliff: `barman-cloud-backup` is single-threaded gzip + GCS upload buffering, ceiling-bound by the pod's memory limit. The 9 GB DB took 30–90 min to back up at 512Mi; 2Gi cuts that to 5–10 min. requests left at 128Mi (no scheduling change; this is purely a ceiling bump). Patched live with `kubectl patch cluster knoe-db --type=merge -p '{"spec":{"resources":{"limits":{"memory":"2Gi"}}}}'`; CNPG operator performed a supervised rolling restart (replicas first, primary last). Repo source updated at [`deploy/gcp/gke/knoe-db.yaml`](../deploy/gcp/gke/knoe-db.yaml). Pre-flight verified no external psql sessions (chrisfu/ron not connected); 39 internal sessions FATAL'd through the rolling restart and reconnected via their pools — supabase + gitlab + streaming_replica all stable post-rollout. Closes drift R3. k3s + k8s mode resource limits remain at 512Mi (not exposed in those manifests today); cross-mode parity folded into queue item #8 ("Port oauth2-proxy manifest to k3d/k3s/min modes"). diff --git a/etc/preflight_kubecontext.sh b/etc/preflight_kubecontext.sh new file mode 100755 index 0000000..07d8778 --- /dev/null +++ b/etc/preflight_kubecontext.sh @@ -0,0 +1,134 @@ +#!/usr/bin/env bash +# preflight_kubecontext.sh +# +# Shared helper to detect the "wrong shell context for the install mode" +# class of bug. Sourced by install.sh and deploy.sh. +# +# Exposes two functions: +# +# verify_kubecontext_matches_config +# Strict gate. Reads APP_CLUSTER_KUBECONTEXT from the config's [Global] +# section and refuses to proceed if `kubectl config current-context` +# doesn't match. Used by deploy.sh (unattended path: silent mismatches +# are dangerous). +# +# Returns 0 on match (or when the config has no APP_CLUSTER_KUBECONTEXT, +# e.g. fresh k3d setup). Exits 1 with a clear error on mismatch. +# Set KNOE_SKIP_KUBECONTEXT_GUARD=true to bypass (escape hatch for +# deliberate cross-cluster maintenance). +# +# print_kubecontext_notice +# Informational. Prints the current kubectl context (or "(none)") so +# the user sees what install.sh is about to inherit before the TUI +# launches. Never exits or fails. Used by install.sh. +# +# History: this guard was filed in response to the 2026-04-28 14:00 UTC +# outage. An install.sh run in k3d mode with the shell pointed at GKE +# overwrote the GKE-cluster ObjectStore + ScheduledBackup with k3d-mode +# defaults, and CNPG backups silently failed for hours until the next +# manual check. See CLAUDE.md §"Env-contamination warning" and +# docs/TODO.md queue item #1 (drift R4). + +# Read APP_CLUSTER_KUBECONTEXT from the [Global] section of an INI-style +# config. Empty string if absent. Quote-stripping is best-effort. +_kubectx_from_config() { + local cfg="$1" + [[ -f "$cfg" ]] || { echo ""; return 0; } + # awk: print value when we're in [Global] and key matches. + # Strips surrounding whitespace and quotes. + awk ' + /^\[/{section=$0; next} + section=="[Global]" && /^[[:space:]]*APP_CLUSTER_KUBECONTEXT[[:space:]]*=/ { + sub(/^[^=]*=[[:space:]]*/, "", $0) + sub(/^"/, "", $0); sub(/"$/, "", $0) + sub(/^'\''/, "", $0); sub(/'\''$/, "", $0) + print $0 + exit + } + ' "$cfg" +} + +# Read the live current-context, or empty if kubectl/config unavailable. +_kubectx_current() { + command -v kubectl >/dev/null 2>&1 || { echo ""; return 0; } + kubectl config current-context 2>/dev/null || true +} + +# verify_kubecontext_matches_config +# Strict gate. Exits 1 on mismatch unless KNOE_SKIP_KUBECONTEXT_GUARD=true. +verify_kubecontext_matches_config() { + local cfg_path="${1:-}" + if [[ -z "$cfg_path" ]]; then + echo "preflight_kubecontext: usage: verify_kubecontext_matches_config " >&2 + return 2 + fi + if [[ "${KNOE_SKIP_KUBECONTEXT_GUARD:-false}" == "true" ]]; then + echo "preflight_kubecontext: KNOE_SKIP_KUBECONTEXT_GUARD=true — skipping check (escape hatch)." >&2 + return 0 + fi + + local expected + expected="$(_kubectx_from_config "$cfg_path")" + + # No baked context in the config (e.g. fresh k3d.cfg) → nothing to check. + if [[ -z "$expected" ]]; then + return 0 + fi + + local actual + actual="$(_kubectx_current)" + + # No live current-context → user hasn't selected one; the config is + # authoritative and downstream code will pass --context explicitly. + if [[ -z "$actual" ]]; then + return 0 + fi + + if [[ "$expected" == "$actual" ]]; then + return 0 + fi + + cat >&2 < kubectl current-context: (none set)" + else + echo "==> kubectl current-context: $actual" + fi + echo " The mode you select must target this cluster, OR you must switch" + echo " context (kubectl config use-context …) before proceeding." +} diff --git a/install.sh b/install.sh index 6468768..2deefc9 100755 --- a/install.sh +++ b/install.sh @@ -81,6 +81,26 @@ if [[ -n "${_script_dir}" && -d "${_script_dir}/knoe" ]]; then # Local checkout: delegate to the Python installer module. cd "${_script_dir}" + # kubectl-context notice. install.sh defers mode selection to the Python + # TUI, so we can't strictly verify here — but printing the inherited + # context up-front gives the user a chance to abort before the TUI + # launches if it looks wrong (e.g. shell on GKE while planning a k3d + # install). Filed against docs/TODO.md queue item #1 (drift R4); the + # mode-aware strict guard runs in deploy.sh (which knows its config) and + # should also run Python-side once the welcome screen records a mode. + if [[ -f "${_script_dir}/etc/preflight_kubecontext.sh" ]]; then + _is_min=0 + for arg in "$@"; do + if [[ "$arg" == "--min" ]]; then _is_min=1; break; fi + done + if [[ "$_is_min" -eq 0 ]]; then + # shellcheck source=etc/preflight_kubecontext.sh + source "${_script_dir}/etc/preflight_kubecontext.sh" + print_kubecontext_notice + fi + unset _is_min + fi + # 1Password preflight — sign in, create 'knoey' vault, ensure 'administrator' item. # Skipped in --min mode (init_1password.sh handles the flag check itself). if [[ -f "${_script_dir}/etc/init_1password.sh" ]]; then