kube-prometheus-stack CRD re-application stalls on every installer run
even when the release is in deployed state with all pods Running. For k3d
this is purely idempotency noise — skip the upgrade and return immediately.
Broken state (failed status, unbound PVCs) still triggers a full purge+reinstall.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three issues caused the purge-and-reinstall loop:
1. _purge_broken_monitoring deleted ALL PVCs including Bound ones; grafana's
working PVC was wiped on every retry. Now only delete PVCs in Pending state.
2. Broken-state detection keyed on Pending pods + unbound PVCs, which is true
during any still-converging install (including ones that timed out but
whose pods eventually came up). Gate on helm status=='failed' + unbound PVCs.
3. k3d install used --wait, which blocks on all kube-prometheus-stack components
(prometheus, alertmanager, node-exporter). They converge async after the
Prometheus operator starts; --wait always timed out. Drop --wait for k3d;
the status_common_services.sh check verifies readiness independently.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
kube-prometheus-stack pulls multiple large images on a fresh k3d install;
600s / 10m is not enough. Set --timeout 20m and process timeout 1320s
(20m + 2m buffer) for k3d, keep 10m for k8s/k3s where images are warm.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs kept prometheus-grafana-0 Pending for 21 days in k3d:
1. update() routed k3d through _values_yaml_k3s() which hardcodes
merlin-local-iscsi-* storage classes and merlin.prole.org node affinity
— neither exists in k3d, so grafana's PVC could never bind.
Add _values_yaml_k3d() that uses local-path (or MONITORING_STORAGE_CLASS
override) with no node affinity; dispatch k3d separately in update().
2. No broken-state recovery for non-k8s mode. Add _purge_broken_monitoring()
(helm uninstall + delete stuck PVCs) and call it from update() when
Pending pods and unbound PVCs are both detected in the monitoring namespace.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Introduced `_recover_stale_prometheus_helm_lock` to detect and recover from stale Helm release locks during Prometheus deployment.
- Enhanced Prometheus deployment flow with stability verification and rollback to the last deployed state.
- separate default app and db cluster contexts (knoe-dev-0 and knoe-cnpg-0) across actions, ops flows, and installer config\n- update CNPG placement and GKE manifests to keep PostgreSQL on the db cluster while app services stay on the app cluster\n- add GCS-backed Barman ObjectStore manifest and cross-cluster Garage patch workflow\n- refresh UI cluster/database/service screens and monitoring wiring for dual-cluster operation\n- add reset/patch scripts, db context selection test coverage, and architecture/network documentation updates
Co-authored-by: Junie <junie@jetbrains.com>
## GCP / Cluster Environment Screen
- Auto-populate Cloud tab from conf/prod/gcp.cfg on screen open (org_id,
billing_account, billing_project, project_id)
- gcloud auth validity checked on screen startup; friendly modal dialog
streams gcloud auth login output live so user never leaves the app
- Live GKE cluster browser: fetches clusters via gcloud container clusters
list, displays with checkmark selector, auto-selects saved cluster
- Selecting a cluster runs get-credentials, sets KUBECONFIG/KUBECONTEXT,
and syncs the region dropdown to the selected cluster's location
- Region dropdown populated live from gcloud compute regions list with
checkmark on currently selected region; graceful fallback when offline
- New 'GCP Storage' tab with workload->StorageClass mapping (CNPG->premium-rwo,
Redis/Monitoring->standard-rwo, Garage->garage-hdd) and Fetch from Cluster
- Provider readonly field styled correctly (no solid-black on macOS)
- Stale prole.cfg/conf/prole.cfg symlinks removed; all config I/O now
resolves env-specific paths via prole_conf.entrypoint_path()
## GKE Autopilot Compatibility (Common Services)
- Synology iSCSI StorageClass and static PVs guarded behind PROLE_MODE!=k8s
in init_openbao.sh (GKE Autopilot forbids hostPath/iSCSI volumes)
- In-cluster Docker registry (hostPath) skipped in k8s mode; GCP Artifact
Registry used instead
- Kong renamed knoe-svc-kong in k8s mode; all health-check kubectl calls in
init_common_services.sh and status_common_services.sh updated accordingly
- DNS endpoints switched from *.prole.org to *.knoe.dev in k8s mode
(api.knoe.dev, git.knoe.dev, svc.knoe.dev); ingress uses gce class
- New GKE-clean Kong manifests under deploy/opentofu/k8s/manifests/prole/:
no k3s node affinity, explicit Autopilot resource requests/limits
## Garage S3 Store (GKE)
- New garage-statefulset-gcp.yaml targeting garage-hdd StorageClass
(pd-standard, avoids SSD_TOTAL_GB quota exhaustion in us-west3)
- New storageclass-gcp-hdd.yaml (pd-standard, Retain, WaitForFirstConsumer)
- GCP StorageClass manifests skipped on re-runs (Autopilot built-ins are
immutable; skip-if-exists guard added)
- PVC deletion guard extended to cover any storageClass (not just synology)
so stale claims are cleaned before StatefulSet recreation
## Topology (GKE Autopilot)
- DaemonSet collector skipped in prod mode (forbidden in kube-system by
GKE Warden); Kubernetes-only node facts path used instead
- All ready GKE nodes assumed cnpg-eligible and monitoring-eligible without
taint/synology-mount checks (skip_collector + assume_nodes_eligible flags)
## KUBECONFIG / kubectl (k8s mode)
- actions.py: new elif mode==k8s branch sets KUBECONFIG=~/.kube/config
and injects KUBECONTEXT from prole_cfg_data into script env
- _build_kubectl_cmd falls back to Global.KUBECONTEXT when
init_cluster.selected_kubectx is empty
- _activate_selected_gke_cluster persists KUBECONFIG/KUBECONTEXT to
prole_cfg_data and saves prole.cfg immediately after get-credentials
## Database Build Screen (GKE)
- Registry display shows correct Artifact Registry URL
(<region>-docker.pkg.dev/<project>/<namespace>/knoe-db) in green
- Build+push: gcloud auth configure-docker, auto-creates AR repository
named after SERVICE_NAMESPACE (e.g. knoe-system) if missing, then
docker tag + push; falls back to gcr.io if region unavailable
- GCP config loaded from conf/prod/gcp.cfg on every screen entry;
keys normalised to lowercase so project_id lookup is always consistent
## Config / Namespace persistence
- prole_conf.py activate_environment: symlink creation removed; sets
CLUSTER_ENV env-var so all subsequent calls resolve correct env directory
- knoe/ui/screens/__init__.py: startup config load uses entrypoint_path()
instead of hardcoded conf/prole.cfg; seeds SERVICE_NAMESPACE=knoe-system
for managed envs so Common Services never defaults to 'default'
- cfg.py _save_prole_cfg: saves to env-specific path via entrypoint_path()
- etc/prole_cfg.sh: removed all ln -snf symlink creation
Co-authored-by: Junie <junie@jetbrains.com>
- add reusable storage probing subsystem with discovery, bounded probe execution, IO classification, caching, and topology integration
- render per-node storage inventory in Cluster Nodes UI and extend installer test coverage for topology/storage behavior
- introduce core service operation modules and align actions, milestones, services, and supporting configs/scripts for repair/update workflows
- update CNPG/Supabase/database artifacts, placement and port mapping configs, plus related integration tests
Co-authored-by: Junie <junie@jetbrains.com>
