chrisfu/prole
1
0
Fork 0
mirror of https://github.com/dredx/prole.git synced 2026-09-23 12:03:59 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
500c9b1317
prole/CLAUDE.md
chrisfu 500c9b1317 fix(installer): env-contamination guard against shell-context / config mismatch 
Filed in response to the 2026-04-28 14:00 UTC backup outage. An
`install.sh --mode k3d` run with the shell pointed at GKE silently
overwrote the GKE cluster's GCS-backed ObjectStore + ScheduledBackup
with k3d-mode defaults; Garage filled up and CNPG backups failed for
hours before the next manual check. The class of bug is "config says
target cluster A, shell context says target cluster B, installer
proceeds against B without warning."

New shared bash helper at etc/preflight_kubecontext.sh with two
functions:

  - verify_kubecontext_matches_config <cfg-path>
      Strict gate. Reads [Global] APP_CLUSTER_KUBECONTEXT from the
      config and exits 1 if `kubectl config current-context` differs.
      Skipped silently when the config has no baked APP_CLUSTER_KUBECONTEXT
      (e.g. fresh k3d.cfg) or when there's no live current-context.

  - print_kubecontext_notice
      Informational. Prints what's about to be inherited so the user
      can abort before the TUI launches if it looks wrong. Never fails.

Wiring:

  - deploy.sh sources the helper and calls the strict gate against
    ${PROLE_DEPLOY_CFG:-conf/gke.cfg} before invoking Python.
    Unattended path -> hard refusal on mismatch.

  - install.sh sources the helper and calls the informational notice
    (gated on not-`--min`) right after entering the local-checkout
    branch. The TUI is interactive, so the strict mode-aware gate is
    a follow-up once the welcome screen records a mode in
    state.inputs.

Bypass for deliberate cross-cluster maintenance:
    KNOE_SKIP_KUBECONTEXT_GUARD=true ./deploy.sh

End-to-end verified:
  - deploy.sh with current=cnpg-0, gke.cfg=app-0   -> exit 1, clear msg
  - deploy.sh with KNOE_SKIP_...=true              -> bypasses, prints
                                                     "skipping check"
  - install.sh --min                               -> notice skipped
  - install.sh (no flag) and install.sh --silent   -> notice printed

Doc updates:
  - CLAUDE.md §"Env-contamination warning" rewritten to describe the
    live guard (was a forward-looking TODO).
  - CLAUDE.md drift table row R4 removed; "Closed 2026-05-01" line added.
  - docs/TODO.md queue item #1 archived to Done; R4 dropped from the
    reality-vs-intent table. Queue numbering retained (no #1 placeholder)
    so the docs/plans/junie/<NN>-...md filenames still match.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 16:14:30 -07:00

191 lines
9.2 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# CLAUDE.md — knoe project context
> Loaded automatically by Claude Code (CLI and IntelliJ plugin) as project context.
---
## Repo role: customer deploy of `knoe-db`
This working tree (`prole`) is being converged into a **customer-deploy branch** of the upstream platform repo `knoe-db` (remote: `git@git-ssh.knoe.dev:knoe.dev/knoe-db.git`, configured here as the `knoe` remote). Platform code — `authority/`, `knoe/`, `etc/init_*.sh`, `deploy/gcp/gke/*`, the test pipeline — is not maintained in this repo; its source of truth is `knoe-db/main`. Customer-specific divergence (config, branding, on-prem `prole.org` manifests) is what stays on the customer branch.
**Before changing anything in this tree, check whether the change belongs upstream.** If it's platform code, land it on `knoe-db/main` and rebase the customer branch.
## Active plans
- [`docs/plans/customer-deploy-resync.md`](docs/plans/customer-deploy-resync.md) — The active plan for converging `prole` onto `knoe-db/main` as a customer-deploy branch. Inventory + cherry-pick + rebase cadence. **Read this first** if you are about to touch anything in this tree.
- [`docs/plans/README.md`](docs/plans/README.md) — Index and conventions for this directory.
For platform-level plans (knoe-auth, deployment modes, etc.), read [`knoe-db/docs/plans/`](../knoe-db/docs/plans/) (path relative to your `~/dev/` root). Those documents are the architectural reference for the shipped platform features and are versioned with the upstream code, not here.
---
## Dual-cluster GKE architecture
This project uses **two separate GKE Standard clusters** in `us-west3`, both currently provisioned with `e2-standard-2` × 3 nodes (2 vCPU / 8 GB each, ~7.1 GB allocatable). Verify with `gcloud container clusters list` — `knoe/core/actions.py` still carries an `e2-small` default for the app cluster but the live cluster is `e2-standard-2`.
| Cluster | Context | Role |
|---|---|---|
| `knoe-dev-0` | `gke_plenary-truck-485623-p7_us-west3_knoe-dev-0` | App cluster — Garage, registry, OpenBao, Kong, GitLab, monitoring |
| `knoe-cnpg-0` | `gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0` | DB cluster — CNPG/PostgreSQL only (note: `conf/gke.cfg` currently deploys this as `knoe-dev-cnpg-0`; the code default remains `knoe-cnpg-0`) |
**Storage quota:** the project has `SSD_TOTAL_GB = 300 GB` in `us-west3`, **fully consumed by CNPG**. All non-CNPG PVCs must use `standard` (pd-standard / HDD) — not `standard-rwo` / `premium-rwo`, which are SSD-backed and will fail to provision with a quota error. `GITLAB_GITALY_STORAGE_CLASS = standard` is set in `conf/gke.cfg` accordingly.
### Resource allocation
| Resource | Cluster | Namespace |
|---|---|---|
| CNPG operator | `knoe-cnpg-0` | `cnpg-system` |
| PostgreSQL cluster (`knoe-db`) | `knoe-cnpg-0` | `knoe-db-0` |
| Barman Cloud plugin | `knoe-cnpg-0` | `cnpg-system` |
| cert-manager | `knoe-cnpg-0` | `cert-manager` |
| Garage (S3 object store) | `knoe-dev-0` | `knoe-system` |
| Registry | `knoe-dev-0` | `knoe-system` |
| OpenBao | `knoe-dev-0` | `knoe-system` |
| Kong API gateway | `knoe-dev-0` | `knoe-system` |
| Monitoring | `knoe-dev-0` | `monitoring` |
**Garage runs ONLY in `knoe-dev-0`.** Do NOT deploy Garage to `knoe-cnpg-0`.
### CNPG backups → GCS (not Garage)
Backups use **GCS with Workload Identity** (not Garage):
- Data bucket: `gs://knoe-0-backups/`
- WAL bucket: `gs://knoe-0-wal/`
- GCP SA: `cnpg-backup@plenary-truck-485623-p7.iam.gserviceaccount.com`
- K8s SA: `cnpg-backup-sa` in `knoe-db-0` (annotated with WI)
- ObjectStore manifest: `k8s/knoe/knoe-db-barman-objectstore-gcs.yaml`
Setup script: `etc/init_cnpg_gke.sh` (creates buckets, GCP SA, WI binding, applies CNPG cluster).
---
## Service mesh (Cloud Service Mesh / Istio)
Both clusters are registered in the **knoe-0** GCP fleet with automatic Cloud Service Mesh management. This is automated in `scripts/reset_clusters.sh` (Phase 7) — no longer requires GCP web console.
```bash
# Check mesh provisioning status (~10 min after cluster creation):
gcloud container fleet mesh describe --project=plenary-truck-485623-p7
# Manual re-registration if needed:
gcloud container fleet memberships register knoe-dev-0 \
--gke-cluster=us-west3/knoe-dev-0 \
--enable-workload-identity \
--project=plenary-truck-485623-p7
gcloud container fleet mesh update \
--management=automatic \
--memberships=knoe-dev-0 \
--project=plenary-truck-485623-p7
```
---
## install.sh / deploy.sh pre-flight checklist
### `conf/gke.cfg` (interactive installer — `./install.sh`)
Before running `./install.sh` (especially "Initialization Scripts"), confirm these are correct:
```ini
[Inputs]
init_cluster.app_cluster_kubecontext = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0
init_cluster.db_cluster_kubecontext = gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0
env_setup.APP_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0
env_setup.DB_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0
[Global]
KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0
APP_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0
DB_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0
CNPG_ELIGIBLE_NODES = <comma-separated node names from knoe-cnpg-0>
```
### `conf/service/prod.cfg` (unattended deploy — `./deploy.sh`)
Same cluster context entries are required here too:
```ini
[Inputs]
init_cluster.app_cluster_kubecontext = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0
init_cluster.db_cluster_kubecontext = gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0
env_setup.APP_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0
env_setup.DB_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0
[Global]
KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0
APP_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0
DB_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0
ARTIFACT_REGISTRY = us-west3-docker.pkg.dev/plenary-truck-485623-p7/knoe-system
SERVICE_NAMESPACE = knoe-system
REGISTRY_NAMESPACE = knoe-system
```
**Why these matter:** `Milestone._get_script_env()` (in `knoe/milestone.py`) reads these to set `KUBECONTEXT=app_ctx` for common services and `DB_CLUSTER_KUBECONTEXT=db_ctx` for CNPG ops. Without them, all kubectl calls use the ambient context, which may be the DB cluster.
Missing `init_cluster.app_cluster_kubecontext` → `_cluster_kubecontext("app")` returns `""` → installer falls back to `Global.KUBECONTEXT` for **both** app and db environments → **Garage deploys to knoe-cnpg-0** (wrong).
> **Env-contamination guard (live):** `deploy.sh` calls
> [`etc/preflight_kubecontext.sh`](etc/preflight_kubecontext.sh) and refuses
> to proceed when `kubectl config current-context` doesn't match the
> `[Global] APP_CLUSTER_KUBECONTEXT` of the active config. `install.sh`
> prints the inherited context up-front (mode-aware strict gate is the
> Python TUI's responsibility once the welcome screen records a mode).
> Bypass with `KNOE_SKIP_KUBECONTEXT_GUARD=true` for deliberate
> cross-cluster maintenance. **History:** the guard was filed in response
> to the 2026-04-28 14:00 UTC outage — an `install.sh --mode k3d` run with
> the shell pointed at GKE replaced the GCS-backed ObjectStore with a
> Garage-backed one, then Garage filled up and backups silently failed for
> hours. Closes drift R4 / queue item #1.
### Get current CNPG node names
```bash
kubectl --context=gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0 get nodes -o name
```
---
## Key scripts
| Script | Purpose |
|---|---|
| `scripts/reset_clusters.sh` | Delete + recreate both GKE clusters (`e2-standard-2` × 3 each) |
| `scripts/patch_garage_cross_cluster.sh` | Remove garage from DB cluster, configure GCS barman ObjectStore |
| `etc/init_cnpg_gke.sh` | Provision CNPG on GKE with GCS backup via Workload Identity |
| `etc/init_cnpg_backup.sh` | Configure barman plugin + trigger initial backup |
---
## Cluster code constants (`knoe/core/actions.py`)
```python
DEFAULT_APP_CLUSTER_NAME = "knoe-dev-0"
DEFAULT_DB_CLUSTER_NAME = "knoe-cnpg-0" # also: DEFAULT_CNPG_CLUSTER_NAME
DEFAULT_APP_CLUSTER_MACHINE_TYPE = "e2-small" # STALE — live app cluster is e2-standard-2
DEFAULT_DB_CLUSTER_MACHINE_TYPE = "e2-standard-2"
```
---
## GCP project
- Project: `plenary-truck-485623-p7`
- Region: `us-west3`
- Artifact Registry: `us-west3-docker.pkg.dev/plenary-truck-485623-p7/knoe-system`
- CPU quota: 16 vCPUs (all regions) — 2 clusters × 3 × `e2-standard-2` = 12 vCPUs used
- SSD quota: `SSD_TOTAL_GB = 300 GB` — fully consumed by CNPG; all other PVCs must use `standard` (pd-standard / HDD)
---
## Reality TODOs / Drift Log
Quick reference. Each entry links to the master index where context, owner, and rank live.
| # | Drift | Where described above | Where tracked |
|---|---|---|---|
| _(none currently)_ | | | |
**Closed in 2026-04-29 stabilization session:** Garage on DB cluster removed; cluster pods migrated to `cnpg-backup-sa` via CNPG v1.29.0 `spec.serviceAccountName`; both operators restarted clean.
**Closed 2026-05-01:** R4 — installer env-contamination guard now live in `deploy.sh` (strict) + `install.sh` (informational notice). Helper at [`etc/preflight_kubecontext.sh`](etc/preflight_kubecontext.sh).
Reference in New Issue View Git Blame Copy Permalink