prole/AGENTS.md

AGENTS.md — AI coding agent guide for knoe-db / Knoe

What this repo is

Knoe is an infrastructure stack for deploying a Supabase-style internal developer platform (PostgreSQL, object storage, secrets, auth, observability) across K3d (local), K3s (on-prem), and GKE (cloud) environments. The Python "Knoe" installer (knoe/cli.py) drives all cluster setup via milestones.

---

Architecture overview

Dual-cluster GKE layout (prod)

Two GKE clusters in us-west3:

Cluster	Context	Purpose
knoe-dev-0	gke_plenary-truck-485623-p7_us-west3_knoe-dev-0	App cluster — Garage, Registry, OpenBao, Kong, GitLab, monitoring
knoe-dev-cnpg-0	gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0	DB cluster — CNPG/PostgreSQL only

Critical: SSD quota (300 GB) is fully consumed by CNPG — all non-CNPG PVCs must use standard storage class (HDD), not standard-rwo/premium-rwo. Garage runs ONLY on knoe-dev-0 (removed from DB cluster on 2026-04-29).

Deployment environments / modes

cluster_env	KNOE_MODE	Target
dev	k3d	Local K3d cluster
service	k3s	On-prem K3s cluster
prod	k8s	GKE (or other cloud)

_deployment_mode_from_env() in knoe/core/env.py converts env strings to mode strings.

Config file mapping

knoe/knoe_conf.py maps environments to config files under conf/:

• dev → k3d.cfg
• service → k3s.cfg
• prod → gke.cfg

Config is layered: env-specific file overrides base. KNOE_CONF env var or conf/service/ subdirs point to the active config.

---

Key source locations

Path	Purpose
knoe/core/env.py	Core config/env helpers, secret encryption, kubeconfig resolution
knoe/core/actions.py	All installer actions and unattended workflow helpers (~8k lines)
knoe/milestone.py	Milestone ABC — all install steps implement this; _get_script_env() builds the env for subprocesses
knoe/knoe_conf.py	Config path resolution and layered loading
knoe/core/milestones.py	Concrete milestone definitions
conf/gke.cfg	Production GKE config (must have correct app_cluster_kubecontext / db_cluster_kubecontext)
conf/service/prod.cfg	Unattended deploy config for ./deploy.sh
etc/	Shell init scripts (init_*.sh) called by milestones
k8s/	Kubernetes manifests by service
scripts/reset_clusters.sh	Full cluster teardown + recreate

---

Developer workflows

Install dependencies

make requirements        # pip install -r requirements.txt

Run tests

make test                # runs pyconv (black check) then pytest with coverage
# or directly:
PYTHONPATH=. pytest tests/

Build the knoe CLI binary

make build               # PyInstaller one-file binary → dist/knoe

Interactive installer (ncurses)

make knoe                # launches ./install.sh
# or via the unified launcher:
./knoe.sh install

Unattended deploy

./deploy.sh              # reads conf/service/prod.cfg

Code style

black .                  # formatter (black --check . is enforced in CI)

---

Secret handling

Secrets in knoe.cfg are AES-GCM encrypted at rest using ${KNOE_SECRET:v1:...} tokens. On macOS, the key is in Keychain (knoe-installer service); on Linux, at ~/.knoe/secrets/knoe.key. OpenBao references use ${OPENBAO:kv/knoe/<ns>/<leaf>#<key>}. Never store plaintext passwords in config files.

---

Milestone pattern

All installer steps subclass Milestone (knoe/milestone.py). They must:

• Be UI-agnostic (no tkinter/ncurses imports)
• Use _run_cmd() for subprocesses (handles env injection)
• Use _get_script_env(state) to build env dicts for shell scripts — this is where KUBECONTEXT, DB_CLUSTER_KUBECONTEXT, KNOE_CONF, etc. are set

Missing init_cluster.app_cluster_kubecontext in config causes Garage to deploy to the wrong cluster.

---

CNPG / backup specifics

• CNPG backups go to GCS: gs://knoe-0-backups/ (single bucket; knoe-db/base/ and knoe-db/wals/ prefixes). gs://knoe-0-wal/ exists but is unused.
• CNPG operator: v1.29.0 (upgraded 2026-04-29 to expose spec.serviceAccountName)
• plugin-barman-cloud: v0.12.0
• GCP SA: cnpg-backup@plenary-truck-485623-p7.iam.gserviceaccount.com (storage.objectAdmin + storage.legacyBucketReader on the bucket)
• K8s SA: cluster pods run as cnpg-backup-sa in knoe-db-0, set via cluster.spec.serviceAccountName: cnpg-backup-sa. The SA has the iam.gke.io/gcp-service-account annotation. RoleBindings knoe-db and knoe-db-barman-cloud include cnpg-backup-sa as a subject so the pod has the same RBAC the auto-generated SA would have had.
• ObjectStore manifest: k8s/knoe/knoe-db-barman-objectstore-gcs.yaml — includes googleCredentials.gkeEnvironment: true
• Setup: etc/init_cnpg_gke.sh and etc/init_cnpg_backup.sh

Performance note: Pod memory: 512Mi makes barman-cloud-backup runs 30–90 min for the 9 GB DB. Bump pending in docs/TODO.md.