- Introduced `get_gitlab_migrations_diagnostics` for detailed migrations pod diagnostics.
- Enhanced active migration failure detection with detailed logs and diagnostics.
- Improved job context logging with pod restart, status, and waiting reason details.
- Refined stale job repair logic by incorporating diagnostics and failure insights.
- Added detailed context logging for GitLab blockers, including failed migrations and Gitaly resources.
- Introduced automatic repair for stalled migration jobs by deleting stale jobs and pods.
- Improved Gitaly resource validation with enhanced logging for PVCs and StatefulSet configurations.
- Extended PVC phase handling with detailed storageClass descriptions for better troubleshooting.
- Improved detection and logging of mismatched storageClass across CR, StatefulSet, and PVCs.
- Added auto-repair mechanisms for inconsistencies in Gitaly storageClass configuration.
- Enhanced state validation with detailed logs for better troubleshooting.
- Introduced a grace period and explicit replacement of legacy StatefulSet after expiry.
- Enhanced logging to provide clearer details on convergence progress and outcomes.
- Added handling for operator recreation and state validation during legacy repairs.
- Ensured robust detection and resolution of legacy nodeSelector and storageClass issues.
- Added logic to wait for Gitaly StatefulSet corrections after autoclean actions.
- Implemented live state validation for nodeSelector and storageClass updates.
- Enhanced logging and error handling for legacy storage repairs in k8s mode.
- Updated PV and PVC handling to address legacy resource cleanup.
- Added logging to show Kubernetes context and service inspection details for Redis.
- Improved validation by checking for service endpoints and readiness.
- Enabled IP address bypass for external host DNS checks.
- Added logging to show Kubernetes context and service inspection details for Redis.
- Improved validation by checking for service endpoints and readiness.
- Enabled IP address bypass for external host DNS checks.
- Improved `check_gitlab_pre_apply_blocked` to better detect invalid placeholders, legacy storage configurations, and DNS issues for Redis and Garage endpoints.
- Added auto-repair for Garage endpoint secrets and storage class mismatches in k8s mode.
- Introduced warnings and blocking logic for unsupported configurations like `gitlab-gitaly-static`.
- Refined storage class handling to dynamically align with `standard-rwo` in k8s/GKE.
- Introduced `check_gitlab_pre_apply_blocked` and `check_gitlab_post_apply_blocked` functions to detect misconfigurations and legacy setups during deployment.
- Added logic to validate DNS resolution for Redis and Garage endpoints, preventing invalid placeholders or inaccessible hosts.
- Enhanced storage checks to identify and optionally repair legacy storage configurations with `GITLAB_REPAIR_BLOCKED_AUTOCLEAN`.
- Updated deployment wait loop to validate readiness and detect blocking states dynamically.
- remove hardcoded physical host assumptions in UI/init flows and rely on config-driven selectors\n- optimize GitLab reruns to skip unnecessary operator upgrades/long reconcile waits when unchanged\n- add and update regression/shell tests for fast-path and k8s context/ingress/garage behaviors
Co-authored-by: Junie <junie@jetbrains.com>
- Refactored `init_gitlab.sh` and `init_gitea.sh` to use configurable node selector keys, removing hardcoded defaults like `gandalf.prole.org`.
- Enhanced scheduling logic to validate required fields and prevent stale node constraints during reruns.
- Added fast-path guards in GitLab init to skip redundant operations when no changes occur in operator or CR specifications.
- Updated Supabase deployment to clear stale topology spread constraints alongside node selectors and affinity.
- Added tests for configurable node selector keys, fast-path guards, and stale constraint cleanups.
- Introduced `SERVICE_INGRESS_TLS_ENABLED` for configurable svc ingress TLS rendering in `init_kong.sh`.
- Updated ingress templates to include or exclude TLS annotations and blocks dynamically.
- Improved PVC and pod readiness checks in `supabase/deploy.sh` with enriched logging and detailed blockers.
- Added support for configurable storage class resolution for Supabase DB frontdoor PVCs.
- Enhanced deployment workflows to handle split APP/DB cluster setups with context-aware readiness checks.
- Updated test cases to validate svc ingress TLS configuration and storage class logic.
- Added detection for "already exists" errors during bucket creation and reused existing buckets when applicable.
- Enhanced logging to provide context when reusing previously created buckets.
- Added support for validating existing ingress ownership using `gitlab_release` and `expected_backend_service`.
- Improved conflict detection by distinguishing between same-owner and foreign ingress claims.
- Introduced explicit handling for operator-managed ingress and fallback ingress creation logic.
- Enhanced logging and status return codes for better error reporting and decision-making in deployment scripts.
- Introduced `resolve_garage_admin_context` and `kubectl_garage_admin` for dynamic context resolution.
- Enhanced Garage setup with explicit logging of admin context and namespace details.
- Improved error handling and command execution reliability for multi-cluster environments.
- Added validation and explicit handling for Garage S3 endpoints in split APP/DB cluster setups.
- Improved dynamic role detection and authoritative context enforcement in initialization scripts.
- Updated Supabase and Garage configurations to support private cross-cluster endpoints, ensuring cluster interoperability and DNS constraints.
- Enhanced error handling and logging for invalid or missing configurations.
- Added base64 decoding and validation for admin token in `garage-secrets`.
- Implemented reusable `garage_exec` helper for consistent command execution with detailed error handling.
- Improved key creation, retrieval, and fallback mechanisms with enhanced output parsing and failure logging.
- Unified bucket creation and permission granting logic using the new helper function.
- Added validation for admin token existence in `garage-secrets`.
- Improved `garage key` operations with support for deterministic fallback keys and secret fetch retries.
- Unified access/secret key parsing with flexible format support (`sed` fallback).
- Added dynamic `GITALY_STORAGE_CLASS` logic for k8s and legacy modes.
- Introduced context-aware DB operations with `resolve_db_cluster_context` and `kubectl_db`.
- Improved object storage configuration with failure handling based on deployment requirements.
- Refined workload placement logic and node selector defaults for improved flexibility.
- Added `PROLE_MODE` check to skip auxiliary host handling for Kubernetes (`k8s`) mode.
- Ensured conditional inclusion of `auth_host` and `gitea_host` based on deployment context.
- Improved clarity and flexibility in ingress target configuration logic.
- Added logic to derive and override APP cluster kubecontext based on `DEPLOYMENT_MODE` and `PROLE_MODE`.
- Enhanced support for legacy runtime context aliases with improved fallback behaviors.
- Unified context resolution for split-cluster environments to ensure role-consistent deployments.
- Introduced jemalloc hostPath optimizations with configurable modes (`auto`, `off`, `force`).
- Integrated jemalloc setup with best-effort and forced validation flows for ensuring cluster compatibility.
- Enhanced monitoring storage class logic with mode-specific handling (`k3s`, `k3d`, `gke`) and improved validation of required classes.
- Added safeguards and detailed logging for unsupported configurations and failure scenarios.
- Introduced jemalloc hostPath optimizations with configurable modes (`auto`, `off`, `force`).
- Integrated jemalloc setup with best-effort and forced validation flows for ensuring cluster compatibility.
- Enhanced monitoring storage class logic with mode-specific handling (`k3s`, `k3d`, `gke`) and improved validation of required classes.
- Added safeguards and detailed logging for unsupported configurations and failure scenarios.
Rename env config files from conf/*/prole.cfg to conf/k3d.cfg, conf/k3s.cfg, and conf/gke.cfg. Update shell/Python loaders and etc/deploy scripts to resolve named configs cleanly while keeping legacy fallback behavior. Align k3s Ansible tasks, docs, and regression coverage with the new configuration layout.
Co-authored-by: Junie <junie@jetbrains.com>
Switch production config to k8s/GKE contexts and align service naming. Add immutable StatefulSet update fallback for Garage across k3d/k3s/k8s. Harden CNPG deploy and backup bootstrap paths, and update installer coverage for CNPG webhook and Garage common ops.
Co-authored-by: Junie <junie@jetbrains.com>
- Introduced support for dynamic GitLab and Supabase domain and ingress configuration based on deployment mode (k8s vs local).
- Default GitLab domain and ingress class aligned with `prole.cfg` deployment settings.
- Expanded Supabase support:
- Added configurable API and Studio hostnames with default fallbacks.
- Introduced separate ingress class configuration for API and Studio.
- Standardized public URL normalization for both services.
- Updated tests to validate domain, ingress, and configuration behaviors across environments.
- Remove premature migrate_registry_images_to_gitlab() call (was running
before GitLab wait, against a not-yet-ready gitlab-registry)
- Add registry migration as the true final step of init_gitlab.sh, after
the GitLab CR is Available and Kong ingress is configured
- Delegates to init_registry.sh migrate so logic lives in one place
- SKIP_REGISTRY_MIGRATE=1 suppresses migration if needed
- Fix migrate_registry_to_gitlab() in init_registry.sh:
- dst_registry defaults to gitlab-registry.<gitlab_ns>.svc.cluster.local:5000
(in-cluster address, no port-forward needed)
- Checks registry:2 pod exists before attempting catalog fetch
- Counts ok/fail per-repo and reports retry command on partial failure
- Passes GITLAB_NAMESPACE from init_gitlab.sh caller context
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
KNOE_IMAGE_REGISTRY auto-detection in cloudnative_pg._apply_manifest():
- Explicit config wins; falls back to gitlab-registry when GitLab ns exists;
final fallback to registry:2 in knoe-system. No longer raises when unset.
conf/service/prole.cfg: point KNOE_IMAGE_REGISTRY at gitlab-registry since
GitLab is already deployed on this cluster.
etc/init_registry.sh: add 'migrate' action that copies all images from
registry:2 to gitlab-registry via skopeo (prints commands if skopeo absent).
Run after GitLab is up, before decommissioning registry:2.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- knoe/milestone.py: copy [Global] prole.cfg keys into _get_script_env()
env dict so CNPG and other Python-owned ops see KNOE_IMAGE_REGISTRY and
other config-declared vars (mirrors what prole_cfg.sh does for shell scripts)
- etc/init_redis.sh: skip helm upgrade when redis-master is already running
to avoid Docker Hub timeout on constrained networks; set REDIS_FORCE_UPGRADE=1
to override
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- harden CNPG backup ObjectStore readiness fallback in statusless clusters by validating controller availability and expected spec
- improve backup/plugin reconciliation paths and related test coverage for barman-cloud flows
- include ongoing GKE/CNPG deployment and configuration updates from this checkpoint
Co-authored-by: Junie <junie@jetbrains.com>
UI screens
- database.py: fix mode detection to use env_key priority (prod→k8s, service→k3s) so stale DEPLOYMENT_MODE never overrides the user's chosen environment
- database.py: Registry status reads ARTIFACT_REGISTRY_AVAILABLE persisted by cluster screen; uses SERVICE_NAMESPACE for Artifact Registry repo name
- cluster.py: add Artifact Registry traffic light (amber→green/red) to prod section; _check_artifact_registry_async persists ARTIFACT_REGISTRY_AVAILABLE into Global cfg
- cluster.py: re-trigger Artifact Registry check after GKE cluster selection so the light re-evaluates once region is available from KUBECONTEXT
- cluster_nodes.py: fix TclError on Python 3.14 — pady=(2,0) tuple → pady=2 scalar
- __init__.py: seed knoe-system namespace when saved value is "default", not only when empty
- services.py: replace hardcoded "Prole DB" log string with dynamic cnpg_cluster name
Core ops
- cloudnative_pg.py: replace one-shot Barman plugin retry with 6-attempt loop; first cert-manager/x509 failure triggers rollout restart + 30 s CA propagation wait; subsequent failures back off up to 60 s per attempt
- cloudnative_pg.py: TLS CA CN now uses cluster_name instead of hardcoded "Prole CNPG CA"
- registry.py, garage_store.py: refactored into per-mode modules (k3d/k3s/k8s registry and garage store, shared _garage_common)
Deploy / config
- deploy/gcp/gke/knoe-db.yaml: GKE-specific CNPG cluster manifest (rw/ro/r on separate nodes with premium-rwo storage)
- etc/init_common_services.sh, modes/k8s/knoe-db/.version: updated for current deploy
- kong-deployment.yaml: updated manifest
Tests
- test_cluster_nodes_render_smoke.py: add pack/grid, winfo_children, winfo_reqheight, update_idletasks, grid_slaves to dummy widgets; monkeypatch tk.Label so CNPG placement render completes without a real Tkinter root
Co-authored-by: Junie <junie@jetbrains.com>
## GCP / Cluster Environment Screen
- Auto-populate Cloud tab from conf/prod/gcp.cfg on screen open (org_id,
billing_account, billing_project, project_id)
- gcloud auth validity checked on screen startup; friendly modal dialog
streams gcloud auth login output live so user never leaves the app
- Live GKE cluster browser: fetches clusters via gcloud container clusters
list, displays with checkmark selector, auto-selects saved cluster
- Selecting a cluster runs get-credentials, sets KUBECONFIG/KUBECONTEXT,
and syncs the region dropdown to the selected cluster's location
- Region dropdown populated live from gcloud compute regions list with
checkmark on currently selected region; graceful fallback when offline
- New 'GCP Storage' tab with workload->StorageClass mapping (CNPG->premium-rwo,
Redis/Monitoring->standard-rwo, Garage->garage-hdd) and Fetch from Cluster
- Provider readonly field styled correctly (no solid-black on macOS)
- Stale prole.cfg/conf/prole.cfg symlinks removed; all config I/O now
resolves env-specific paths via prole_conf.entrypoint_path()
## GKE Autopilot Compatibility (Common Services)
- Synology iSCSI StorageClass and static PVs guarded behind PROLE_MODE!=k8s
in init_openbao.sh (GKE Autopilot forbids hostPath/iSCSI volumes)
- In-cluster Docker registry (hostPath) skipped in k8s mode; GCP Artifact
Registry used instead
- Kong renamed knoe-svc-kong in k8s mode; all health-check kubectl calls in
init_common_services.sh and status_common_services.sh updated accordingly
- DNS endpoints switched from *.prole.org to *.knoe.dev in k8s mode
(api.knoe.dev, git.knoe.dev, svc.knoe.dev); ingress uses gce class
- New GKE-clean Kong manifests under deploy/opentofu/k8s/manifests/prole/:
no k3s node affinity, explicit Autopilot resource requests/limits
## Garage S3 Store (GKE)
- New garage-statefulset-gcp.yaml targeting garage-hdd StorageClass
(pd-standard, avoids SSD_TOTAL_GB quota exhaustion in us-west3)
- New storageclass-gcp-hdd.yaml (pd-standard, Retain, WaitForFirstConsumer)
- GCP StorageClass manifests skipped on re-runs (Autopilot built-ins are
immutable; skip-if-exists guard added)
- PVC deletion guard extended to cover any storageClass (not just synology)
so stale claims are cleaned before StatefulSet recreation
## Topology (GKE Autopilot)
- DaemonSet collector skipped in prod mode (forbidden in kube-system by
GKE Warden); Kubernetes-only node facts path used instead
- All ready GKE nodes assumed cnpg-eligible and monitoring-eligible without
taint/synology-mount checks (skip_collector + assume_nodes_eligible flags)
## KUBECONFIG / kubectl (k8s mode)
- actions.py: new elif mode==k8s branch sets KUBECONFIG=~/.kube/config
and injects KUBECONTEXT from prole_cfg_data into script env
- _build_kubectl_cmd falls back to Global.KUBECONTEXT when
init_cluster.selected_kubectx is empty
- _activate_selected_gke_cluster persists KUBECONFIG/KUBECONTEXT to
prole_cfg_data and saves prole.cfg immediately after get-credentials
## Database Build Screen (GKE)
- Registry display shows correct Artifact Registry URL
(<region>-docker.pkg.dev/<project>/<namespace>/knoe-db) in green
- Build+push: gcloud auth configure-docker, auto-creates AR repository
named after SERVICE_NAMESPACE (e.g. knoe-system) if missing, then
docker tag + push; falls back to gcr.io if region unavailable
- GCP config loaded from conf/prod/gcp.cfg on every screen entry;
keys normalised to lowercase so project_id lookup is always consistent
## Config / Namespace persistence
- prole_conf.py activate_environment: symlink creation removed; sets
CLUSTER_ENV env-var so all subsequent calls resolve correct env directory
- knoe/ui/screens/__init__.py: startup config load uses entrypoint_path()
instead of hardcoded conf/prole.cfg; seeds SERVICE_NAMESPACE=knoe-system
for managed envs so Common Services never defaults to 'default'
- cfg.py _save_prole_cfg: saves to env-specific path via entrypoint_path()
- etc/prole_cfg.sh: removed all ln -snf symlink creation
Co-authored-by: Junie <junie@jetbrains.com>
The auth deployment in knoe-system uses label app=auth; init_knoe_users.sh
was searching for app=prole-auth causing the pod lookup to always fail.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Kerberos is now the single source of truth for all knoe-system accounts.
In-cluster KDC (PROLE.LOCAL):
- Rename in-cluster KDC realm PROLE.ORG → PROLE.LOCAL to avoid collision
with the myrddin.prole.org Samba AD DC which owns PROLE.ORG
- Add [capaths] cross-realm trust block: PROLE.ORG (Samba AD) ↔ PROLE.LOCAL
- Create admin@PROLE.LOCAL on KDC startup (master password) for UI login
- Create guest@PROLE.LOCAL on KDC startup (read-only, PROLE_KDC_GUEST_PASSWORD)
- Update prole-kdc-secrets.example.yaml with trust_shared_password and guest_password
- Update prole-auth-kerberos ConfigMap realm/servicePrincipal to PROLE.LOCAL
- Add PROLE_AUTH_ADMIN_PRINCIPALS and PROLE_KDC_GUEST_PASSWORD to prole-auth Deployment
Database (knoe-db — k3s and GKE):
- Add spec.managed.roles: admin (superuser), guest (login), developer (group)
- Enable GSS pg_hba rules for both PROLE.ORG and PROLE.LOCAL realms
- Add krb_server_keyfile parameter for postgres service principal keytab
- Mount knoe-db-pg-keytab Secret via spec.additionalVolumes
- Add demo schema in postInitSQL with guest read-only grants
prole-auth (Spring Boot):
- SessionUser: add List<String> groups field
- AuthProperties: add adminPrincipals list (env: PROLE_AUTH_ADMIN_PRINCIPALS)
- LoginController: stamp groups=["admin"] for configured admin principals at login
- SessionTokenService: carry groups through TokenPayload; @JsonIgnoreProperties
for backward compat with existing 2-field session cookies
- VerifyController: emit X-Prole-Groups header on /auth/verify
- grafana-proxy nginx: strip inbound X-Prole-Groups; capture and forward from auth
Services:
- ArgoCD (k3s + GKE): add g, admin, role:admin to argocd-rbac-cm policy.csv
etc/init_knoe_users.sh (new):
- Orchestrates full user provisioning: KDC principals, postgres keytab export,
CNPG managed.roles patch, demo schema SQL, ArgoCD RBAC patch,
Gitea and GitLab admin promotion via API
- Actions: initialize | status | cleanup
- Idempotent; sources prole_cfg.sh; follows existing init_*.sh style
Cross-realm trust activation (myrddin side):
samba-tool user add krbtgt_PROLELOCAL --random-password
samba-tool user setpassword krbtgt_PROLELOCAL --newpassword=<trust_shared_password>
samba-tool spn add "krbtgt/PROLE.LOCAL" krbtgt_PROLELOCAL -U administrator
samba-tool user setexpiry krbtgt_PROLELOCAL --noexpiry
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- etc/init_gitlab.sh: Add global.redis block (host/port/auth) to the GitLab CR so
the chart does not fail NOTES.txt validation when redis.install: false.
Drop the GITOPS_NAMESPACE config fallback in namespace resolution to prevent the
Gitea namespace from bleeding into GitLab deployments; GITLAB_NAMESPACE is now the
sole source of truth with a hard default of "gitlab".
- knoe/core/milestones.py: Fix GitOpsMilestone to route to init_gitlab.sh when
gitops.git_provider = GitLab (was hardcoded to init_gitea.sh). Namespace resolution
now prefers gitops.gitlab_namespace input key, then gitops.namespace, then "gitlab" —
never picks up a stale GITLAB_NAMESPACE from the OS environment.
- conf/service/prole.cfg: Switch gitops.git_provider / GITOPS_PROVIDER to GitLab.
Update accumulated runtime state from install runs.
- install.sh: Prefer the repo-local venv Python (PROLE_HOME/bin/python3) so that
PyYAML and other prole_requirements.txt deps are always available.
- .gitlab-ci.yml: New CI pipeline — on every push to main, run the silent install
(./install.sh -S -c conf/service/prole.cfg) to deploy a fresh CNPG ecosystem.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
