Commit Graph

397 Commits

Author SHA1 Message Date
chrisfu
854de2e0a6 chore: update prole.cfg defaults for dev cluster, refine config overrides
- Updated default paths, namespaces, and cluster names for k3d/dev configuration.
- Added `skip_names` set to exclude `prod.cfg` and `gcp.cfg` from override processing.
- Refined init parameters for Supabase, Kubernetes, and database deployments to align with dev-specific settings.
- Enhanced test coverage for excluded config files (`prod.cfg`, `gcp.cfg`) within `test_prole_conf.py`.
2026-04-11 16:21:31 -07:00
chrisfu
2eb6ae0626 Fix prole.cfg portability: preserve HOME tokens, block KUBECONTEXT pollution, keep CNPG Clusters section
- _sanitize_sections_for_cfg(): wrap _expand_shell_path() with _cfgify_home_path() so
  paths are stored as dollar-HOME/... instead of /home/user/... (Mac to myrddin breakage)
- _write_cfg(): pop KUBECONTEXT from globals_to_save so Mac k3d context never persists
- _write_cfg(): add CNPG Clusters to sections list so user-authored registry survives write-back
- milestone._get_script_env(): _GLOBAL_CFG_PROPAGATION_BLOCK excludes KUBECONTEXT, HOME,
  USER from Global config propagation -- stops knoe.dev.prole.org injected into kubectl calls
- milestone._get_script_env(): env.pop KUBECONTEXT after k3s KUBECONFIG detection so
  kubectl uses kubeconfig current-context (default) on k3s, not a stale Mac context name
- conf/service/prole.cfg: remove stale KUBECONTEXT = knoe.dev.prole.org from Global

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-11 16:15:59 -07:00
chrisfu
6318d232f0 feat: enhance configurable domain and ingress handling for GitLab and Supabase
- Introduced support for dynamic GitLab and Supabase domain and ingress configuration based on deployment mode (k8s vs local).
- Default GitLab domain and ingress class aligned with `prole.cfg` deployment settings.
- Expanded Supabase support:
  - Added configurable API and Studio hostnames with default fallbacks.
  - Introduced separate ingress class configuration for API and Studio.
  - Standardized public URL normalization for both services.
- Updated tests to validate domain, ingress, and configuration behaviors across environments.
2026-04-11 16:01:02 -07:00
chrisfu
091012a9e5 feat: wire registry migration as final unattended step of init_gitlab.sh
- Remove premature migrate_registry_images_to_gitlab() call (was running
  before GitLab wait, against a not-yet-ready gitlab-registry)
- Add registry migration as the true final step of init_gitlab.sh, after
  the GitLab CR is Available and Kong ingress is configured
- Delegates to init_registry.sh migrate so logic lives in one place
- SKIP_REGISTRY_MIGRATE=1 suppresses migration if needed
- Fix migrate_registry_to_gitlab() in init_registry.sh:
  - dst_registry defaults to gitlab-registry.<gitlab_ns>.svc.cluster.local:5000
    (in-cluster address, no port-forward needed)
  - Checks registry:2 pod exists before attempting catalog fetch
  - Counts ok/fail per-repo and reports retry command on partial failure
  - Passes GITLAB_NAMESPACE from init_gitlab.sh caller context

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-11 13:11:45 -07:00
chrisfu
1f404dd07c feat: auto-resolve registry host; add registry migrate action
KNOE_IMAGE_REGISTRY auto-detection in cloudnative_pg._apply_manifest():
- Explicit config wins; falls back to gitlab-registry when GitLab ns exists;
  final fallback to registry:2 in knoe-system. No longer raises when unset.

conf/service/prole.cfg: point KNOE_IMAGE_REGISTRY at gitlab-registry since
GitLab is already deployed on this cluster.

etc/init_registry.sh: add 'migrate' action that copies all images from
registry:2 to gitlab-registry via skopeo (prints commands if skopeo absent).
Run after GitLab is up, before decommissioning registry:2.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-11 13:08:04 -07:00
chrisfu
64f1144335 fix: propagate Global config into Python env; skip Redis if already running
- knoe/milestone.py: copy [Global] prole.cfg keys into _get_script_env()
  env dict so CNPG and other Python-owned ops see KNOE_IMAGE_REGISTRY and
  other config-declared vars (mirrors what prole_cfg.sh does for shell scripts)
- etc/init_redis.sh: skip helm upgrade when redis-master is already running
  to avoid Docker Hub timeout on constrained networks; set REDIS_FORCE_UPGRADE=1
  to override

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-11 13:05:46 -07:00
chrisfu
8e5cf9c9bc feat: config-driven CNPG cluster registry
Add [CNPG Clusters] section to prole.cfg as the authoritative source for
CNPG cluster definitions. The deploy pipeline reads this registry instead
of computing version-appended namespace names at runtime.

- conf/service/prole.cfg: add [CNPG Clusters] with knoe-db as identity
  cluster; fix DATABASE_NAMESPACE=knoe-db (was knoe-db-18-140); add
  KNOE_IMAGE_REGISTRY for the k3s in-cluster registry hostname
- deploy/opentofu/k3s/manifests/prole/knoe-db.yaml: replace hardcoded
  registry.test-system image with templated KNOE_IMAGE_REGISTRY
- knoe/core/actions.py: add _cnpg_cluster_registry(), _cnpg_identity_cluster();
  fix _initial_db_namespace() to use declared namespace without version suffix
- knoe/core/milestones.py: iterate over all declared clusters in Init and
  Deploy milestones; per-cluster CNPG_IMAGE_NAME support
- knoe/core/ops/cloudnative_pg.py: resolve KNOE_IMAGE_REGISTRY and
  CNPG_IMAGE_NAME overrides in _apply_manifest()

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-11 10:56:12 -07:00
chrisfu
5460c58eb9 fix: remove Kerberos fields from k3s CNPG manifest
Drop spec.additionalVolumes, krb_server_keyfile, and hostgssenc pg_hba
rules from the k3s Cluster manifest. The CNPG CRD on k3s rejects
additionalVolumes as an unknown field, the keytab secret does not exist,
and hostgssenc rules would prevent PostgreSQL from starting without a
mounted keytab. Kerberos is disabled in conf/service/prole.cfg and
will be wired up once k3s KDC and keytab provisioning are in place.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-11 09:54:51 -07:00
chrisfu
9e352d74a5 chore: enhance Redis deploy configuration for constrained environments
- exclude `prod.cfg` from overlay processing in standalone GKE deployments
- add configurable Redis image registry and helm timeout for constrained networks
- update Helm chart to honor custom registry and timeout settings
2026-04-11 08:35:27 -07:00
chrisfu
a14fa654e6 chore: checkpoint deployment and identity bootstrap updates
- switch default git provider to GitLab and add cluster kubecontext settings in prole config

- add knoe user/user_role schema grants in GKE and k3s database manifests

- extend init_knoe_users bootstrap to create schema tables and provision default users

- make deploy pipeline honor configured cluster env and gate GKE-only defaults

- update GitLab OIDC default frontdoor host and refresh generated manifest/db metadata

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-10 20:55:30 -07:00
chrisfu
246fa5851c enable k3s supabase and gitlab 2026-04-10 17:31:42 -07:00
chrisfu
c023c7f590 stabilize unattended GKE Supabase/CNPG deployment
- align app/db cluster defaults for knoe-dev-0 + knoe-dev-cnpg-0 on e2-standard-2\n- harden Supabase deploy flow for cross-cluster DB ILB, GKE storage class, and node selector fallback\n- migrate Helm chart path to knoe-supabase and make external DB host rendering IP-safe\n- disable Kerberos in prod unattended config to prevent Supabase GSS auth failures\n- add Supabase port-forward mappings and DB backup context handling improvements\n\nBuild status: k8s stable

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-10 15:53:58 -07:00
chrisfu
b4fc5641db checkpoint: sanitize prole config for host portability
- fix conf/prod/prole.cfg by replacing hardcoded /Users/chrisfu paths with /Users/chrisfu

- remove injected [update.sh] log lines and stray password artifacts so config is executable

- include latest pending updates across deployment config, UI/core flow, vault/network artifacts, and helper scripts
2026-04-10 05:58:40 -07:00
chrisfu
c94c62e1bb Harden prod deploy namespace/context routing and vault password handling
- persist and load DB master password via Ansible Vault bootstrap flow

- enforce knoe-system service namespace and explicit app/db kubecontext targeting

- improve OpenBao/CNPG deploy reliability and logging; add retries/readiness diagnostics

- tighten reset/delete cluster behavior and expand installer/deploy pipeline test coverage

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-10 00:43:07 -07:00
chrisfu
3393767953
Merge pull request #2 from dredx/feat/deploy-sh-gcp-pipeline
Add deploy.sh GCP pipeline launcher
2026-04-09 20:17:21 -07:00
chrisfu
c46d26f9bc Add dedicated deployment config entrypoint for GCP pipeline 2026-04-09 19:53:02 -07:00
chrisfu
f7fc02fa5a Add standalone deploy pipeline launcher (GCP) 2026-04-09 19:41:08 -07:00
chrisfu
a20d4d308a Add top-level GCP deploy launcher 2026-04-09 19:40:06 -07:00
chrisfu
5cd43c873c feat(gke): finalize dual-cluster app/db topology
- separate default app and db cluster contexts (knoe-dev-0 and knoe-cnpg-0) across actions, ops flows, and installer config\n- update CNPG placement and GKE manifests to keep PostgreSQL on the db cluster while app services stay on the app cluster\n- add GCS-backed Barman ObjectStore manifest and cross-cluster Garage patch workflow\n- refresh UI cluster/database/service screens and monitoring wiring for dual-cluster operation\n- add reset/patch scripts, db context selection test coverage, and architecture/network documentation updates

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-08 23:12:51 -07:00
chrisfu
64e1cef16c checkpoint: stabilize k8s cluster setup and shelve cluster-storage UI
- make cluster-storage milestone opt-in and remove installer cluster-storage step from UI navigation\n- add cluster storage browser and GKE cluster ops helpers with CLI coverage\n- update k8s/CNPG config and install flow files for corrected cluster setup\n- add/refresh tests for storage browser, GKE ops, prod config, and service-layer navigation

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-07 21:11:25 -07:00
chrisfu
df6de9138e checkpoint: summarize CNPG init and backup hardening work
- harden CNPG backup ObjectStore readiness fallback in statusless clusters by validating controller availability and expected spec

- improve backup/plugin reconciliation paths and related test coverage for barman-cloud flows

- include ongoing GKE/CNPG deployment and configuration updates from this checkpoint

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-06 05:51:26 -07:00
chrisfu
c165ee1a87 fix(cnpg): stabilize CNPG/Barman Cloud init on GKE (knoe-dev-0 / knoe-db-0)
- cloudnative_pg.py: add _wait_for_webhook_ca_bundle() to poll ValidatingWebhookConfiguration
  until cainjector has injected the CA bundle; fall back to _try_inject_ca_bundle_manually()
  on timeout (reads cert-manager-webhook-ca secret, patches webhook directly)
- cloudnative_pg.py: add _wait_for_barman_tls_secrets() to wait for barman-cloud-{client,server}-tls
  before declaring Barman Cloud ready; surfaces warning instead of silent FailedMount
- deploy/gcp/gke/namespace.yaml: ecosystem-0 -> knoe-db-0 (ekosystem = k8s cluster, not namespace)
- deploy/gcp/gke/knoe-db.yaml: fix namespace ecosystem-0 -> knoe-db-0; set
  podAntiAffinityType: required (one CNPG pod per node); add Spot node toleration so
  GKE autoscaler simulation succeeds and scale-up is unblocked; add resource
  requests/limits (500m/512Mi -> 2/2Gi); set googleCredentials.gkeEnvironment: true
  (use Workload Identity - no explicit credentials secret needed)
- deploy/gcp/gke/knoe-db-backup-gcs.yaml: fix namespaces ecosystem-0 -> knoe-db-0
- deploy/gcp/gke/workload-identity.yaml: update comments ecosystem-0 -> knoe-db-0
- tests/installer/test_cnpg_barman_webhook.py: new - 13 tests covering webhook CA bundle
  polling, manual CA injection, Barman TLS secret wait, and install_barman_plugin retry logic

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-05 02:04:11 -07:00
chrisfu
3312c39b1f feat: GCP/GKE CNPG hardening, Artifact Registry traffic light, and knoe-system namespace fixes
UI screens
- database.py: fix mode detection to use env_key priority (prod→k8s, service→k3s) so stale DEPLOYMENT_MODE never overrides the user's chosen environment
- database.py: Registry status reads ARTIFACT_REGISTRY_AVAILABLE persisted by cluster screen; uses SERVICE_NAMESPACE for Artifact Registry repo name
- cluster.py: add Artifact Registry traffic light (amber→green/red) to prod section; _check_artifact_registry_async persists ARTIFACT_REGISTRY_AVAILABLE into Global cfg
- cluster.py: re-trigger Artifact Registry check after GKE cluster selection so the light re-evaluates once region is available from KUBECONTEXT
- cluster_nodes.py: fix TclError on Python 3.14 — pady=(2,0) tuple → pady=2 scalar
- __init__.py: seed knoe-system namespace when saved value is "default", not only when empty
- services.py: replace hardcoded "Prole DB" log string with dynamic cnpg_cluster name

Core ops
- cloudnative_pg.py: replace one-shot Barman plugin retry with 6-attempt loop; first cert-manager/x509 failure triggers rollout restart + 30 s CA propagation wait; subsequent failures back off up to 60 s per attempt
- cloudnative_pg.py: TLS CA CN now uses cluster_name instead of hardcoded "Prole CNPG CA"
- registry.py, garage_store.py: refactored into per-mode modules (k3d/k3s/k8s registry and garage store, shared _garage_common)

Deploy / config
- deploy/gcp/gke/knoe-db.yaml: GKE-specific CNPG cluster manifest (rw/ro/r on separate nodes with premium-rwo storage)
- etc/init_common_services.sh, modes/k8s/knoe-db/.version: updated for current deploy
- kong-deployment.yaml: updated manifest

Tests
- test_cluster_nodes_render_smoke.py: add pack/grid, winfo_children, winfo_reqheight, update_idletasks, grid_slaves to dummy widgets; monkeypatch tk.Label so CNPG placement render completes without a real Tkinter root

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-04 19:36:08 -07:00
chrisfu
9e8ebaf36a Merge remote-tracking branch 'origin/main'
# Conflicts:
#	conf/prod/prole.cfg
2026-04-04 12:39:18 -07:00
chrisfu
5529fdea77 feat: full GKE/prod deployment pipeline from UI to Artifact Registry
## GCP / Cluster Environment Screen
- Auto-populate Cloud tab from conf/prod/gcp.cfg on screen open (org_id,
  billing_account, billing_project, project_id)
- gcloud auth validity checked on screen startup; friendly modal dialog
  streams gcloud auth login output live so user never leaves the app
- Live GKE cluster browser: fetches clusters via gcloud container clusters
  list, displays with checkmark selector, auto-selects saved cluster
- Selecting a cluster runs get-credentials, sets KUBECONFIG/KUBECONTEXT,
  and syncs the region dropdown to the selected cluster's location
- Region dropdown populated live from gcloud compute regions list with
  checkmark on currently selected region; graceful fallback when offline
- New 'GCP Storage' tab with workload->StorageClass mapping (CNPG->premium-rwo,
  Redis/Monitoring->standard-rwo, Garage->garage-hdd) and Fetch from Cluster
- Provider readonly field styled correctly (no solid-black on macOS)
- Stale prole.cfg/conf/prole.cfg symlinks removed; all config I/O now
  resolves env-specific paths via prole_conf.entrypoint_path()

## GKE Autopilot Compatibility (Common Services)
- Synology iSCSI StorageClass and static PVs guarded behind PROLE_MODE!=k8s
  in init_openbao.sh (GKE Autopilot forbids hostPath/iSCSI volumes)
- In-cluster Docker registry (hostPath) skipped in k8s mode; GCP Artifact
  Registry used instead
- Kong renamed knoe-svc-kong in k8s mode; all health-check kubectl calls in
  init_common_services.sh and status_common_services.sh updated accordingly
- DNS endpoints switched from *.prole.org to *.knoe.dev in k8s mode
  (api.knoe.dev, git.knoe.dev, svc.knoe.dev); ingress uses gce class
- New GKE-clean Kong manifests under deploy/opentofu/k8s/manifests/prole/:
  no k3s node affinity, explicit Autopilot resource requests/limits

## Garage S3 Store (GKE)
- New garage-statefulset-gcp.yaml targeting garage-hdd StorageClass
  (pd-standard, avoids SSD_TOTAL_GB quota exhaustion in us-west3)
- New storageclass-gcp-hdd.yaml (pd-standard, Retain, WaitForFirstConsumer)
- GCP StorageClass manifests skipped on re-runs (Autopilot built-ins are
  immutable; skip-if-exists guard added)
- PVC deletion guard extended to cover any storageClass (not just synology)
  so stale claims are cleaned before StatefulSet recreation

## Topology (GKE Autopilot)
- DaemonSet collector skipped in prod mode (forbidden in kube-system by
  GKE Warden); Kubernetes-only node facts path used instead
- All ready GKE nodes assumed cnpg-eligible and monitoring-eligible without
  taint/synology-mount checks (skip_collector + assume_nodes_eligible flags)

## KUBECONFIG / kubectl (k8s mode)
- actions.py: new elif mode==k8s branch sets KUBECONFIG=~/.kube/config
  and injects KUBECONTEXT from prole_cfg_data into script env
- _build_kubectl_cmd falls back to Global.KUBECONTEXT when
  init_cluster.selected_kubectx is empty
- _activate_selected_gke_cluster persists KUBECONFIG/KUBECONTEXT to
  prole_cfg_data and saves prole.cfg immediately after get-credentials

## Database Build Screen (GKE)
- Registry display shows correct Artifact Registry URL
  (<region>-docker.pkg.dev/<project>/<namespace>/knoe-db) in green
- Build+push: gcloud auth configure-docker, auto-creates AR repository
  named after SERVICE_NAMESPACE (e.g. knoe-system) if missing, then
  docker tag + push; falls back to gcr.io if region unavailable
- GCP config loaded from conf/prod/gcp.cfg on every screen entry;
  keys normalised to lowercase so project_id lookup is always consistent

## Config / Namespace persistence
- prole_conf.py activate_environment: symlink creation removed; sets
  CLUSTER_ENV env-var so all subsequent calls resolve correct env directory
- knoe/ui/screens/__init__.py: startup config load uses entrypoint_path()
  instead of hardcoded conf/prole.cfg; seeds SERVICE_NAMESPACE=knoe-system
  for managed envs so Common Services never defaults to 'default'
- cfg.py _save_prole_cfg: saves to env-specific path via entrypoint_path()
- etc/prole_cfg.sh: removed all ln -snf symlink creation

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-04 12:38:29 -07:00
chrisfu
4a8d9cc90d feat: full GKE/prod deployment pipeline from UI to Artifact Registry
## GCP / Cluster Environment Screen
- Auto-populate Cloud tab from conf/prod/gcp.cfg on screen open (org_id,
  billing_account, billing_project, project_id)
- gcloud auth validity checked on screen startup; friendly modal dialog
  streams gcloud auth login output live so user never leaves the app
- Live GKE cluster browser: fetches clusters via gcloud container clusters
  list, displays with checkmark selector, auto-selects saved cluster
- Selecting a cluster runs get-credentials, sets KUBECONFIG/KUBECONTEXT,
  and syncs the region dropdown to the selected cluster's location
- Region dropdown populated live from gcloud compute regions list with
  checkmark on currently selected region; graceful fallback when offline
- New 'GCP Storage' tab with workload->StorageClass mapping (CNPG->premium-rwo,
  Redis/Monitoring->standard-rwo, Garage->garage-hdd) and Fetch from Cluster
- Provider readonly field styled correctly (no solid-black on macOS)
- Stale prole.cfg/conf/prole.cfg symlinks removed; all config I/O now
  resolves env-specific paths via prole_conf.entrypoint_path()

## GKE Autopilot Compatibility (Common Services)
- Synology iSCSI StorageClass and static PVs guarded behind PROLE_MODE!=k8s
  in init_openbao.sh (GKE Autopilot forbids hostPath/iSCSI volumes)
- In-cluster Docker registry (hostPath) skipped in k8s mode; GCP Artifact
  Registry used instead
- Kong renamed knoe-svc-kong in k8s mode; all health-check kubectl calls in
  init_common_services.sh and status_common_services.sh updated accordingly
- DNS endpoints switched from *.prole.org to *.knoe.dev in k8s mode
  (api.knoe.dev, git.knoe.dev, svc.knoe.dev); ingress uses gce class
- New GKE-clean Kong manifests under deploy/opentofu/k8s/manifests/prole/:
  no k3s node affinity, explicit Autopilot resource requests/limits

## Garage S3 Store (GKE)
- New garage-statefulset-gcp.yaml targeting garage-hdd StorageClass
  (pd-standard, avoids SSD_TOTAL_GB quota exhaustion in us-west3)
- New storageclass-gcp-hdd.yaml (pd-standard, Retain, WaitForFirstConsumer)
- GCP StorageClass manifests skipped on re-runs (Autopilot built-ins are
  immutable; skip-if-exists guard added)
- PVC deletion guard extended to cover any storageClass (not just synology)
  so stale claims are cleaned before StatefulSet recreation

## Topology (GKE Autopilot)
- DaemonSet collector skipped in prod mode (forbidden in kube-system by
  GKE Warden); Kubernetes-only node facts path used instead
- All ready GKE nodes assumed cnpg-eligible and monitoring-eligible without
  taint/synology-mount checks (skip_collector + assume_nodes_eligible flags)

## KUBECONFIG / kubectl (k8s mode)
- actions.py: new elif mode==k8s branch sets KUBECONFIG=~/.kube/config
  and injects KUBECONTEXT from prole_cfg_data into script env
- _build_kubectl_cmd falls back to Global.KUBECONTEXT when
  init_cluster.selected_kubectx is empty
- _activate_selected_gke_cluster persists KUBECONFIG/KUBECONTEXT to
  prole_cfg_data and saves prole.cfg immediately after get-credentials

## Database Build Screen (GKE)
- Registry display shows correct Artifact Registry URL
  (<region>-docker.pkg.dev/<project>/<namespace>/knoe-db) in green
- Build+push: gcloud auth configure-docker, auto-creates AR repository
  named after SERVICE_NAMESPACE (e.g. knoe-system) if missing, then
  docker tag + push; falls back to gcr.io if region unavailable
- GCP config loaded from conf/prod/gcp.cfg on every screen entry;
  keys normalised to lowercase so project_id lookup is always consistent

## Config / Namespace persistence
- prole_conf.py activate_environment: symlink creation removed; sets
  CLUSTER_ENV env-var so all subsequent calls resolve correct env directory
- knoe/ui/screens/__init__.py: startup config load uses entrypoint_path()
  instead of hardcoded conf/prole.cfg; seeds SERVICE_NAMESPACE=knoe-system
  for managed envs so Common Services never defaults to 'default'
- cfg.py _save_prole_cfg: saves to env-specific path via entrypoint_path()
- etc/prole_cfg.sh: removed all ln -snf symlink creation

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-04 12:38:16 -07:00
chrisfu
4f4b87a7ba feat: full GKE/prod deployment pipeline from UI to Artifact Registry
## GCP / Cluster Environment Screen
- Auto-populate Cloud tab from conf/prod/gcp.cfg on screen open (org_id,
  billing_account, billing_project, project_id)
- gcloud auth validity checked on screen startup; friendly modal dialog
  streams gcloud auth login output live so user never leaves the app
- Live GKE cluster browser: fetches clusters via gcloud container clusters
  list, displays with checkmark selector, auto-selects saved cluster
- Selecting a cluster runs get-credentials, sets KUBECONFIG/KUBECONTEXT,
  and syncs the region dropdown to the selected cluster's location
- Region dropdown populated live from gcloud compute regions list with
  checkmark on currently selected region; graceful fallback when offline
- New 'GCP Storage' tab with workload->StorageClass mapping (CNPG->premium-rwo,
  Redis/Monitoring->standard-rwo, Garage->garage-hdd) and Fetch from Cluster
- Provider readonly field styled correctly (no solid-black on macOS)
- Stale prole.cfg/conf/prole.cfg symlinks removed; all config I/O now
  resolves env-specific paths via prole_conf.entrypoint_path()

## GKE Autopilot Compatibility (Common Services)
- Synology iSCSI StorageClass and static PVs guarded behind PROLE_MODE!=k8s
  in init_openbao.sh (GKE Autopilot forbids hostPath/iSCSI volumes)
- In-cluster Docker registry (hostPath) skipped in k8s mode; GCP Artifact
  Registry used instead
- Kong renamed knoe-svc-kong in k8s mode; all health-check kubectl calls in
  init_common_services.sh and status_common_services.sh updated accordingly
- DNS endpoints switched from *.prole.org to *.knoe.dev in k8s mode
  (api.knoe.dev, git.knoe.dev, svc.knoe.dev); ingress uses gce class
- New GKE-clean Kong manifests under deploy/opentofu/k8s/manifests/prole/:
  no k3s node affinity, explicit Autopilot resource requests/limits

## Garage S3 Store (GKE)
- New garage-statefulset-gcp.yaml targeting garage-hdd StorageClass
  (pd-standard, avoids SSD_TOTAL_GB quota exhaustion in us-west3)
- New storageclass-gcp-hdd.yaml (pd-standard, Retain, WaitForFirstConsumer)
- GCP StorageClass manifests skipped on re-runs (Autopilot built-ins are
  immutable; skip-if-exists guard added)
- PVC deletion guard extended to cover any storageClass (not just synology)
  so stale claims are cleaned before StatefulSet recreation

## Topology (GKE Autopilot)
- DaemonSet collector skipped in prod mode (forbidden in kube-system by
  GKE Warden); Kubernetes-only node facts path used instead
- All ready GKE nodes assumed cnpg-eligible and monitoring-eligible without
  taint/synology-mount checks (skip_collector + assume_nodes_eligible flags)

## KUBECONFIG / kubectl (k8s mode)
- actions.py: new elif mode==k8s branch sets KUBECONFIG=~/.kube/config
  and injects KUBECONTEXT from prole_cfg_data into script env
- _build_kubectl_cmd falls back to Global.KUBECONTEXT when
  init_cluster.selected_kubectx is empty
- _activate_selected_gke_cluster persists KUBECONFIG/KUBECONTEXT to
  prole_cfg_data and saves prole.cfg immediately after get-credentials

## Database Build Screen (GKE)
- Registry display shows correct Artifact Registry URL
  (<region>-docker.pkg.dev/<project>/<namespace>/knoe-db) in green
- Build+push: gcloud auth configure-docker, auto-creates AR repository
  named after SERVICE_NAMESPACE (e.g. knoe-system) if missing, then
  docker tag + push; falls back to gcr.io if region unavailable
- GCP config loaded from conf/prod/gcp.cfg on every screen entry;
  keys normalised to lowercase so project_id lookup is always consistent

## Config / Namespace persistence
- prole_conf.py activate_environment: symlink creation removed; sets
  CLUSTER_ENV env-var so all subsequent calls resolve correct env directory
- knoe/ui/screens/__init__.py: startup config load uses entrypoint_path()
  instead of hardcoded conf/prole.cfg; seeds SERVICE_NAMESPACE=knoe-system
  for managed envs so Common Services never defaults to 'default'
- cfg.py _save_prole_cfg: saves to env-specific path via entrypoint_path()
- etc/prole_cfg.sh: removed all ln -snf symlink creation

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-04 12:34:30 -07:00
chrisfu
a5cd0c45f4 Update conf, k8s registry, knoe UI screens, etc init scripts, and scan data
- conf/service/prole.cfg, port-mapping.cfg, cnpg-placement updated

- k8s/registry/deployment.yaml updated

- knoe/ui/screens: base.py, knoe_users.py, __init__.py updated

- etc/ init scripts refreshed (gitlab, knoe_users, registry, prole_cfg)

- modes/k3s/knoe-db/.version bumped; scan network_description and ansible_inventory updated

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-02 23:40:16 -07:00
chrisfu
7427042694 Add KDC authority service; update registry, knoe UI, conf, and scan artifacts
- New authority/kdc Spring Boot microservice (Dockerfile, HealthController, application.properties)

- Updated authority/pom.xml, k8s/registry/deployment.yaml, knoe/ui/screens, conf, etc/ init scripts, modes/k3s/knoe-db/.version, scan artifacts

- Added project images (knoe.png, architecture diagram, screenshot)

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-02 23:39:52 -07:00
chrisfu
eb9430df9d fix(gitlab,infra): ARM64 RPi service cluster – GitLab deploy in gitlab ns on gandalf
Namespace & routing
- milestones.py: GitOpsMilestone now resolves namespace from
  gitops.gitlab_namespace (new) → Global.GITLAB_NAMESPACE → 'gitlab'
  hardcoded; never falls through to gitops.namespace (was 'gitea')
- prole.cfg: add gitops.gitlab_namespace=gitlab + GITLAB_NAMESPACE=gitlab
- init_gitlab.sh: NAMESPACE defaults to gitlab, NODE_SELECTOR blanked so
  only gitaly+minio are node-pinned; GITOPS_NAMESPACE fallback removed

GitLab on ARM64 RPi (16 KB kernel pages)
- init_gitlab.sh: DaemonSet compiles jemalloc-5.3.0 with --with-lg-page=14
  (glibc/Ubuntu) on every node; LD_PRELOAD injected per Ruby component
- Minio: quay.io 2022 image (ARM64); configure init container replaced
  with ARM64 alpine that writes credential files; MINIO_ROOT_USER/PASSWORD
  injected directly into main container env via secretKeyRef
- Minio buckets auto-created post-deploy (registry, lfs, artifacts, etc.)
- webservice/sidekiq: replicaCount=1, reduced memory (1500M/800M),
  liveness probe initialDelaySeconds=3600 (Rails loads 25-40min on RPi)
- allowedHosts set as flat string list (chart 9.x default is list-of-maps
  which breaks URI initializer in 7_gitlab_http.rb)
- gitaly+minio always pinned to gandalf (local PV); other workloads spread

Storage
- Static PVs created for gitaly (50Gi) + minio (10Gi) on synology d005
- Synology dirs created before PVs; bucket creation idempotent

Redis (shared for GitLab KAS)
- init_redis.sh: persistence disabled (no dynamic provisioner); Redis used
  as pub/sub broker only

Infrastructure / pi.prole.org
- Removed pi.prole.org from [k3s_agents] – dedicated pihole node, OOM
- host_vars: k3s_enabled=false, k3s_state=absent (storage preserved)
- New playbook: infrastructure/playbooks/disable_pi_k3s.yml (drain + disable)
- monitoring.py: node-exporter DaemonSet excludes pi.prole.org
- init_monitoring.sh: pi.prole.org excluded from node-exporter affinity
- kong-deployment.yaml: affinity rule prevents scheduling on pi (pihole owns 80/443)

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-02 20:15:11 -07:00
chrisfu
2bcc40fd50 fix(prole-auth): update tests for SessionUser groups field
SessionUser record was extended with List<String> groups in the groups
feature; test constructors still used the old 2-arg form.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-02 16:56:09 -07:00
chrisfu
e3ca33e4c7 feat(installer): add Knoe User Accounts screen after Kerberos provisioning
New KnoeUsersScreenMixin screen (knoe_users) appears in the install wizard
immediately after Kerberos Authentication (when kerberos is enabled).

- Runs etc/init_knoe_users.sh initialize in a background thread with live
  console output, forwarding SERVICE_NAMESPACE/KNOE_KDC_NAMESPACE from the
  installer state so the script always targets the right namespace
- Optional Gitea and GitLab admin token fields for service admin promotion
- Next button is disabled until provisioning succeeds (idempotent: button
  re-enables on failure so the user can fix and retry)
- Nav chain: kerberos_config -> knoe_users -> argocd_config
  Prev from argocd_config respects kerberos_enabled to route correctly
- Records STATUS in prole.cfg under [Knoe User Accounts] section

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-01 21:45:24 -07:00
chrisfu
e2dd122111 fix(init_knoe_users): namespace/secret resolution aligned with init_kdc.sh
- Add PROLE_KDC_NAMESPACE to namespace fallback chain; never falls to default
- Add get_secret_value() helper mirroring init_kdc.sh pattern
- Log resolved namespaces at startup
- Improve PROLE_KDC_MASTER_PASSWORD error with actionable resolution steps

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-01 21:34:41 -07:00
chrisfu
83938eba51 fix: use correct KDC pod label selector (app=auth not app=prole-auth)
The auth deployment in knoe-system uses label app=auth; init_knoe_users.sh
was searching for app=prole-auth causing the pod lookup to always fail.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-01 20:56:53 -07:00
chrisfu
872d715864 feat: Kerberos identity management — PROLE.LOCAL realm, cross-realm trust, user provisioning
Kerberos is now the single source of truth for all knoe-system accounts.

In-cluster KDC (PROLE.LOCAL):
- Rename in-cluster KDC realm PROLE.ORG → PROLE.LOCAL to avoid collision
  with the myrddin.prole.org Samba AD DC which owns PROLE.ORG
- Add [capaths] cross-realm trust block: PROLE.ORG (Samba AD) ↔ PROLE.LOCAL
- Create admin@PROLE.LOCAL on KDC startup (master password) for UI login
- Create guest@PROLE.LOCAL on KDC startup (read-only, PROLE_KDC_GUEST_PASSWORD)
- Update prole-kdc-secrets.example.yaml with trust_shared_password and guest_password
- Update prole-auth-kerberos ConfigMap realm/servicePrincipal to PROLE.LOCAL
- Add PROLE_AUTH_ADMIN_PRINCIPALS and PROLE_KDC_GUEST_PASSWORD to prole-auth Deployment

Database (knoe-db — k3s and GKE):
- Add spec.managed.roles: admin (superuser), guest (login), developer (group)
- Enable GSS pg_hba rules for both PROLE.ORG and PROLE.LOCAL realms
- Add krb_server_keyfile parameter for postgres service principal keytab
- Mount knoe-db-pg-keytab Secret via spec.additionalVolumes
- Add demo schema in postInitSQL with guest read-only grants

prole-auth (Spring Boot):
- SessionUser: add List<String> groups field
- AuthProperties: add adminPrincipals list (env: PROLE_AUTH_ADMIN_PRINCIPALS)
- LoginController: stamp groups=["admin"] for configured admin principals at login
- SessionTokenService: carry groups through TokenPayload; @JsonIgnoreProperties
  for backward compat with existing 2-field session cookies
- VerifyController: emit X-Prole-Groups header on /auth/verify
- grafana-proxy nginx: strip inbound X-Prole-Groups; capture and forward from auth

Services:
- ArgoCD (k3s + GKE): add g, admin, role:admin to argocd-rbac-cm policy.csv

etc/init_knoe_users.sh (new):
- Orchestrates full user provisioning: KDC principals, postgres keytab export,
  CNPG managed.roles patch, demo schema SQL, ArgoCD RBAC patch,
  Gitea and GitLab admin promotion via API
- Actions: initialize | status | cleanup
- Idempotent; sources prole_cfg.sh; follows existing init_*.sh style

Cross-realm trust activation (myrddin side):
  samba-tool user add krbtgt_PROLELOCAL --random-password
  samba-tool user setpassword krbtgt_PROLELOCAL --newpassword=<trust_shared_password>
  samba-tool spn add "krbtgt/PROLE.LOCAL" krbtgt_PROLELOCAL -U administrator
  samba-tool user setexpiry krbtgt_PROLELOCAL --noexpiry

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-04-01 20:38:57 -07:00
chrisfu
bb92ec3576 feat: service layer — GitLab, Redis, DB ArgoCD app, Kong, monitoring updates
- Add init_redis.sh for Redis deployment
- Extend init_gitlab.sh with Kerberos/OIDC prep and service wiring
- Add init_monitoring.sh improvements
- Update init_kong.sh and init_service_layer.sh
- Rename ArgoCD application-openbao → application-db; add DB kustomization
- Update kong-configmap, kustomization, port-mapping, service configs
- Bump knoe-db version; refresh network scan

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-04-01 20:38:57 -07:00
chrisfu
121b7b9d1b
Merge pull request #1 from dredx/claude/quirky-napier
feat: GitLab deployment pipeline — operator fix, CI config, and names…
2026-04-01 15:29:27 -07:00
chrisfu
5715add366 feat: GitLab deployment pipeline — operator fix, CI config, and namespace isolation
- etc/init_gitlab.sh: Add global.redis block (host/port/auth) to the GitLab CR so
  the chart does not fail NOTES.txt validation when redis.install: false.
  Drop the GITOPS_NAMESPACE config fallback in namespace resolution to prevent the
  Gitea namespace from bleeding into GitLab deployments; GITLAB_NAMESPACE is now the
  sole source of truth with a hard default of "gitlab".
- knoe/core/milestones.py: Fix GitOpsMilestone to route to init_gitlab.sh when
  gitops.git_provider = GitLab (was hardcoded to init_gitea.sh). Namespace resolution
  now prefers gitops.gitlab_namespace input key, then gitops.namespace, then "gitlab" —
  never picks up a stale GITLAB_NAMESPACE from the OS environment.
- conf/service/prole.cfg: Switch gitops.git_provider / GITOPS_PROVIDER to GitLab.
  Update accumulated runtime state from install runs.
- install.sh: Prefer the repo-local venv Python (PROLE_HOME/bin/python3) so that
  PyYAML and other prole_requirements.txt deps are always available.
- .gitlab-ci.yml: New CI pipeline — on every push to main, run the silent install
  (./install.sh -S -c conf/service/prole.cfg) to deploy a fresh CNPG ecosystem.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-01 15:26:39 -07:00
chrisfu
620bdd25eb feat: GKE production deployment — CNPG + GCS backups + Google Workspace SSO
Add complete GKE deployment stack:
- deploy/gcp/gke/: namespace, CloudNativePG cluster (premium-rwo, 100Gi),
  GCS barman-cloud backup with Workload Identity SA, ArgoCD OIDC ConfigMap,
  GitLab OmniAuth secret template, knoe-auth Google OIDC secret template
- etc/init_cnpg_gke.sh: provision GCS buckets, GCP SA, WI binding, CNPG operator
- etc/init_argocd.sh: apply argocd-oidc-cm.yaml when PLATFORM_DOMAIN/FRONTDOOR_HOST set
- etc/init_gitlab.sh: inject OmniAuth openid_connect block when FRONTDOOR_HOST set
- prole-auth-deployment.yaml: add GOOGLE_OIDC_ENABLED env vars (all optional)
- supabase/helm/prole-supabase/values-gke.yaml: GoTrue Google OAuth overlay
- knoe/core/prod_config.py: add backupProvider, workloadIdentitySA, googleClientId/Secret/HostedDomain

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-01 10:41:55 -07:00
chrisfu
68e4a5418c chore: checkpoint non-transitory state (excluded transient tmp render-check artifacts)
Co-authored-by: Junie <junie@jetbrains.com>
2026-04-01 01:43:39 -07:00
chrisfu
e9e53aac40 Migrate install.py logic to install.sh and restructure installer flow. Update core references, tests, and navigation to reflect the new Knoe installer architecture. 2026-04-01 01:17:04 -07:00
chrisfu
c8505e6976 Replace gcloud auth login with in-TUI email + access token credential form
Remove all browser/device-code auth flows. User now enters their Google
account email (visible) and access token (masked) directly in the ncurses
TUI. Token is passed as CLOUDSDK_AUTH_ACCESS_TOKEN env var to all gcloud
calls — no gcloud auth state is modified. Token email is resolved via
the Google tokeninfo endpoint for display only.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-31 23:35:40 -07:00
chrisfu
fabd1a3463 Fix gcloud auth login output capture swallowing device-code URL
gcloud auth login --no-browser was called via _gcloud_plain which uses
capture_output=True, silently swallowing the device-code URL that the
user needs to complete authentication. Run it directly with no capture.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-31 23:21:19 -07:00
chrisfu
2a03bfa395 Add root config.py installer utilities module
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-31 23:15:54 -07:00
chrisfu
2e517686a8 Add GCP ncurses TUI to knoe/config.py and config.sh launcher
Consolidates the GCP cluster configuration TUI (formerly etc/config.py)
into knoe/config.py as a __main__ entrypoint, resolving the naming
collision with the root-level installer utilities module. Adds config.sh
as a thin shell launcher at the project root. Also excludes .claude/
worktree directories from git and IDE indexing.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-31 23:15:54 -07:00
chrisfu
93f7259722 chore: checkpoint k3s deployment and knoe updates
- update Kong/Supabase ingress and service manifests for k3s

- expand init scripts and runtime config/topology handling

- refresh KNØE UI screens and Supabase deployment/render logic

- include related env helper test adjustments

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-31 23:15:54 -07:00
chrisfu
c0a7d0c5dc Refine installer orchestration and Supabase rendering paths
- improve installer/action/controller flow and shell-variable expansion handling across screens\n- adjust Supabase Helm rendering and storage deployment templates\n- align monitoring, cloudnative-pg and repair pipeline behavior with updated config paths\n- refresh and expand installer/core regression tests around milestones, navigation and repair logic

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-31 23:15:54 -07:00
chrisfu
c65d336d1f Make init-script secret checks warning-only and unblock Next
Co-authored-by: Junie <junie@jetbrains.com>
2026-03-31 23:15:54 -07:00
chrisfu
5a0f8ab4f8 Make init-script secret checks warning-only and unblock Next
Co-authored-by: Junie <junie@jetbrains.com>
2026-03-30 01:26:38 -07:00
chrisfu
6232bd89ee Merge remote-tracking branch 'origin/main'
# Conflicts:
#	tests/installer/test_services_init_scripts.py
2026-03-30 00:46:56 -07:00