mirror of https://github.com/dredx/prole.git synced 2026-09-23 10:13:58 +00:00

chrisfu 500c9b1317 fix(installer): env-contamination guard against shell-context / config mismatch

Filed in response to the 2026-04-28 14:00 UTC backup outage. An
`install.sh --mode k3d` run with the shell pointed at GKE silently
overwrote the GKE cluster's GCS-backed ObjectStore + ScheduledBackup
with k3d-mode defaults; Garage filled up and CNPG backups failed for
hours before the next manual check. The class of bug is "config says
target cluster A, shell context says target cluster B, installer
proceeds against B without warning."

New shared bash helper at etc/preflight_kubecontext.sh with two
functions:

  - verify_kubecontext_matches_config <cfg-path>
      Strict gate. Reads [Global] APP_CLUSTER_KUBECONTEXT from the
      config and exits 1 if `kubectl config current-context` differs.
      Skipped silently when the config has no baked APP_CLUSTER_KUBECONTEXT
      (e.g. fresh k3d.cfg) or when there's no live current-context.

  - print_kubecontext_notice
      Informational. Prints what's about to be inherited so the user
      can abort before the TUI launches if it looks wrong. Never fails.

Wiring:

  - deploy.sh sources the helper and calls the strict gate against
    ${PROLE_DEPLOY_CFG:-conf/gke.cfg} before invoking Python.
    Unattended path -> hard refusal on mismatch.

  - install.sh sources the helper and calls the informational notice
    (gated on not-`--min`) right after entering the local-checkout
    branch. The TUI is interactive, so the strict mode-aware gate is
    a follow-up once the welcome screen records a mode in
    state.inputs.

Bypass for deliberate cross-cluster maintenance:
    KNOE_SKIP_KUBECONTEXT_GUARD=true ./deploy.sh

End-to-end verified:
  - deploy.sh with current=cnpg-0, gke.cfg=app-0   -> exit 1, clear msg
  - deploy.sh with KNOE_SKIP_...=true              -> bypasses, prints
                                                     "skipping check"
  - install.sh --min                               -> notice skipped
  - install.sh (no flag) and install.sh --silent   -> notice printed

Doc updates:
  - CLAUDE.md §"Env-contamination warning" rewritten to describe the
    live guard (was a forward-looking TODO).
  - CLAUDE.md drift table row R4 removed; "Closed 2026-05-01" line added.
  - docs/TODO.md queue item #1 archived to Done; R4 dropped from the
    reality-vs-intent table. Queue numbering retained (no #1 placeholder)
    so the docs/plans/junie/<NN>-...md filenames still match.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-01 16:14:30 -07:00

9.2 KiB

Raw Blame History

CLAUDE.md — knoe project context

Loaded automatically by Claude Code (CLI and IntelliJ plugin) as project context.

Repo role: customer deploy of `knoe-db`

This working tree (prole) is being converged into a customer-deploy branch of the upstream platform repo knoe-db (remote: git@git-ssh.knoe.dev:knoe.dev/knoe-db.git, configured here as the knoe remote). Platform code — authority/, knoe/, etc/init_*.sh, deploy/gcp/gke/*, the test pipeline — is not maintained in this repo; its source of truth is knoe-db/main. Customer-specific divergence (config, branding, on-prem prole.org manifests) is what stays on the customer branch.

Before changing anything in this tree, check whether the change belongs upstream. If it's platform code, land it on knoe-db/main and rebase the customer branch.

Active plans

docs/plans/customer-deploy-resync.md — The active plan for converging prole onto knoe-db/main as a customer-deploy branch. Inventory + cherry-pick + rebase cadence. Read this first if you are about to touch anything in this tree.
docs/plans/README.md — Index and conventions for this directory.

For platform-level plans (knoe-auth, deployment modes, etc.), read knoe-db/docs/plans/ (path relative to your ~/dev/ root). Those documents are the architectural reference for the shipped platform features and are versioned with the upstream code, not here.

Dual-cluster GKE architecture

This project uses two separate GKE Standard clusters in us-west3, both currently provisioned with e2-standard-2 × 3 nodes (2 vCPU / 8 GB each, ~7.1 GB allocatable). Verify with gcloud container clusters list — knoe/core/actions.py still carries an e2-small default for the app cluster but the live cluster is e2-standard-2.

Cluster	Context	Role
`knoe-dev-0`	`gke_plenary-truck-485623-p7_us-west3_knoe-dev-0`	App cluster — Garage, registry, OpenBao, Kong, GitLab, monitoring
`knoe-cnpg-0`	`gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0`	DB cluster — CNPG/PostgreSQL only (note: `conf/gke.cfg` currently deploys this as `knoe-dev-cnpg-0`; the code default remains `knoe-cnpg-0`)

Storage quota: the project has SSD_TOTAL_GB = 300 GB in us-west3, fully consumed by CNPG. All non-CNPG PVCs must use standard (pd-standard / HDD) — not standard-rwo / premium-rwo, which are SSD-backed and will fail to provision with a quota error. GITLAB_GITALY_STORAGE_CLASS = standard is set in conf/gke.cfg accordingly.

Resource allocation

Resource	Cluster	Namespace
CNPG operator	`knoe-cnpg-0`	`cnpg-system`
PostgreSQL cluster (`knoe-db`)	`knoe-cnpg-0`	`knoe-db-0`
Barman Cloud plugin	`knoe-cnpg-0`	`cnpg-system`
cert-manager	`knoe-cnpg-0`	`cert-manager`
Garage (S3 object store)	`knoe-dev-0`	`knoe-system`
Registry	`knoe-dev-0`	`knoe-system`
OpenBao	`knoe-dev-0`	`knoe-system`
Kong API gateway	`knoe-dev-0`	`knoe-system`
Monitoring	`knoe-dev-0`	`monitoring`

Garage runs ONLY in knoe-dev-0. Do NOT deploy Garage to knoe-cnpg-0.

CNPG backups → GCS (not Garage)

Backups use GCS with Workload Identity (not Garage):

Data bucket: gs://knoe-0-backups/
WAL bucket: gs://knoe-0-wal/
GCP SA: cnpg-backup@plenary-truck-485623-p7.iam.gserviceaccount.com
K8s SA: cnpg-backup-sa in knoe-db-0 (annotated with WI)
ObjectStore manifest: k8s/knoe/knoe-db-barman-objectstore-gcs.yaml

Setup script: etc/init_cnpg_gke.sh (creates buckets, GCP SA, WI binding, applies CNPG cluster).

Service mesh (Cloud Service Mesh / Istio)

Both clusters are registered in the knoe-0 GCP fleet with automatic Cloud Service Mesh management. This is automated in scripts/reset_clusters.sh (Phase 7) — no longer requires GCP web console.

# Check mesh provisioning status (~10 min after cluster creation):
gcloud container fleet mesh describe --project=plenary-truck-485623-p7

# Manual re-registration if needed:
gcloud container fleet memberships register knoe-dev-0 \
  --gke-cluster=us-west3/knoe-dev-0 \
  --enable-workload-identity \
  --project=plenary-truck-485623-p7
gcloud container fleet mesh update \
  --management=automatic \
  --memberships=knoe-dev-0 \
  --project=plenary-truck-485623-p7

install.sh / deploy.sh pre-flight checklist

`conf/gke.cfg` (interactive installer — `./install.sh`)

Before running ./install.sh (especially "Initialization Scripts"), confirm these are correct:

[Inputs]
init_cluster.app_cluster_kubecontext = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0
init_cluster.db_cluster_kubecontext  = gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0
env_setup.APP_CLUSTER_KUBECONTEXT    = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0
env_setup.DB_CLUSTER_KUBECONTEXT     = gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0

[Global]
KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0
APP_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0
DB_CLUSTER_KUBECONTEXT  = gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0
CNPG_ELIGIBLE_NODES = <comma-separated node names from knoe-cnpg-0>

`conf/service/prod.cfg` (unattended deploy — `./deploy.sh`)

Same cluster context entries are required here too:

[Inputs]
init_cluster.app_cluster_kubecontext = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0
init_cluster.db_cluster_kubecontext  = gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0
env_setup.APP_CLUSTER_KUBECONTEXT    = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0
env_setup.DB_CLUSTER_KUBECONTEXT     = gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0

[Global]
KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0
APP_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0
DB_CLUSTER_KUBECONTEXT  = gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0
ARTIFACT_REGISTRY = us-west3-docker.pkg.dev/plenary-truck-485623-p7/knoe-system
SERVICE_NAMESPACE = knoe-system
REGISTRY_NAMESPACE = knoe-system

Why these matter: Milestone._get_script_env() (in knoe/milestone.py) reads these to set KUBECONTEXT=app_ctx for common services and DB_CLUSTER_KUBECONTEXT=db_ctx for CNPG ops. Without them, all kubectl calls use the ambient context, which may be the DB cluster.

Missing init_cluster.app_cluster_kubecontext → _cluster_kubecontext("app") returns "" → installer falls back to Global.KUBECONTEXT for both app and db environments → Garage deploys to knoe-cnpg-0 (wrong).

Env-contamination guard (live): deploy.sh calls etc/preflight_kubecontext.sh and refuses to proceed when kubectl config current-context doesn't match the [Global] APP_CLUSTER_KUBECONTEXT of the active config. install.sh prints the inherited context up-front (mode-aware strict gate is the Python TUI's responsibility once the welcome screen records a mode). Bypass with KNOE_SKIP_KUBECONTEXT_GUARD=true for deliberate cross-cluster maintenance. History: the guard was filed in response to the 2026-04-28 14:00 UTC outage — an install.sh --mode k3d run with the shell pointed at GKE replaced the GCS-backed ObjectStore with a Garage-backed one, then Garage filled up and backups silently failed for hours. Closes drift R4 / queue item #1.

Get current CNPG node names

kubectl --context=gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0 get nodes -o name

Key scripts

Script	Purpose
`scripts/reset_clusters.sh`	Delete + recreate both GKE clusters (`e2-standard-2` × 3 each)
`scripts/patch_garage_cross_cluster.sh`	Remove garage from DB cluster, configure GCS barman ObjectStore
`etc/init_cnpg_gke.sh`	Provision CNPG on GKE with GCS backup via Workload Identity
`etc/init_cnpg_backup.sh`	Configure barman plugin + trigger initial backup

Cluster code constants (`knoe/core/actions.py`)

DEFAULT_APP_CLUSTER_NAME    = "knoe-dev-0"
DEFAULT_DB_CLUSTER_NAME     = "knoe-cnpg-0"          # also: DEFAULT_CNPG_CLUSTER_NAME
DEFAULT_APP_CLUSTER_MACHINE_TYPE = "e2-small"        # STALE — live app cluster is e2-standard-2
DEFAULT_DB_CLUSTER_MACHINE_TYPE  = "e2-standard-2"

GCP project

Project: plenary-truck-485623-p7
Region: us-west3
Artifact Registry: us-west3-docker.pkg.dev/plenary-truck-485623-p7/knoe-system
CPU quota: 16 vCPUs (all regions) — 2 clusters × 3 × e2-standard-2 = 12 vCPUs used
SSD quota: SSD_TOTAL_GB = 300 GB — fully consumed by CNPG; all other PVCs must use standard (pd-standard / HDD)

Reality TODOs / Drift Log

Quick reference. Each entry links to the master index where context, owner, and rank live.

#	Drift	Where described above	Where tracked
(none currently)

Closed in 2026-04-29 stabilization session: Garage on DB cluster removed; cluster pods migrated to cnpg-backup-sa via CNPG v1.29.0 spec.serviceAccountName; both operators restarted clean.

Closed 2026-05-01: R4 — installer env-contamination guard now live in deploy.sh (strict) + install.sh (informational notice). Helper at etc/preflight_kubecontext.sh.

9.2 KiB Raw Blame History Unescape Escape