Commit Graph

45 Commits

Author SHA1 Message Date
chrisfu
c7871e2780 fix(installer): silent mode — gcp skip for k3d, op timeout, KNOE_SKIP_OP bypass
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-04 01:44:44 -07:00
chrisfu
a9b11f3ee6 fix(cnpg-backup/k3d): skip re-init when already healthy; add breadcrumb 2026-05-04 00:46:43 -07:00
chrisfu
de1a04c0fb fix(ui/cluster): stop combobox overlapping Add/Delete buttons
Combobox width=30 chars extends to ~320-380px depending on font metrics,
colliding with Add at x+290 and Delete at x+350. Narrowed combobox to
width=22 and shifted buttons to x+260 and x+320 to give clear separation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-04 00:34:18 -07:00
chrisfu
47ae5d69d6 fix(supabase/k3d): disable node affinity, ingress, and TUI clutter
- render_supabase.py: k3d sets scheduling.enforceGeneralNodeRole=false so
  pods don't require knoe.dev/node-role=general label (k3d nodes unlabelled)
- render_supabase.py: disable kong and studio ingress for k3d (port-forwards
  are used instead; avoids db.knoe.org reference that doesn't exist
- deploy.sh: escape double quotes in SQL comment on lines 2114-2115 to fix
  bash prematurely closing the psql -c ... string
- supabase.py: remove broken node-selector text entry (black rectangle in Tk)
  and redundant Deploy button (enable checkbox already triggers deployment)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
EOF
)
2026-05-03 22:15:07 -07:00
chrisfu
63185d8104 fix(port-forwards): make argocd/dashboard conditional; fix grafana release name; non-fatal start
Port-forward setup was failing the installation because:
1. argocd and dashboard entries were always added to port-mapping even when
   those services haven't been selected or installed yet
2. Grafana/Prometheus service names were hardcoded as kps-* but monitoring
   deploys with release name "prometheus" → services are prometheus-grafana etc.
3. The grafana status poll after init_port_forwards.sh start would mark the
   entire installation failed if grafana wasn't reachable

Changes:
- _build_required_port_forwards (env.py): add argocd_enabled, dashboard_enabled,
  monitoring_release params; argocd/dashboard only included when enabled; grafana
  and prometheus targets use f"svc/{monitoring_release}-grafana" etc.
- _sync_port_forward_mappings (actions.py): read argocd_enabled from installer
  config; read MONITORING_RELEASE from Monitoring section; clean up argocd and
  dashboard entries from [Port Forwards] when disabled
- run_init_scripts (services.py): port-forward step is non-fatal — failure logs
  a warning and continues; blocking grafana poll removed
- 2 new regression tests (7e) covering the above invariants

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-03 14:39:10 -07:00
chrisfu
c75c63eb9c fix(services): pass service namespace to init_kong.sh, not DB namespace
services.py run_init_scripts() called init_kong.sh with -n env[NAMESPACE]
where NAMESPACE is the database namespace (e.g. knoe-db). The -n flag
sets COMMON_CORE_NAMESPACE which wins over SERVICE_NAMESPACE in
common_core_resolve_namespace, so Kong deployed into knoe-db.

Fix: resolve kong_ns from SERVICE_NAMESPACE, falling back to
_get_service_namespace() and finally knoe-system. Exclude default
as it is the Tk StringVar initial value, not a real service namespace.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-03 13:56:34 -07:00
chrisfu
dba8a2d1dc feat(installer): TDD stabilization for k3d install path; dual-cluster GKE TUI
Junie's session targeted the prompt "stabilize ./install.py -c conf/k3d.cfg
using strict TDD" — broad installer-side work, not the k3d-mirror Phase 3
brief I had filed (which she didn't pick up; phase-3 brief stays open). All
750 installer tests pass post-change.

What Junie produced:

  install.py                          (NEW) Top-level CLI entry point. Was
                                            imagined by the prompt but didn't
                                            exist; this commit makes it real.
  knoe/deployment.py                  (NEW) `KnoeDeployment` orchestrator for
                                            the k3s service-mode deploy pipeline.
                                            Wraps Ansible kubeconfig fetch,
                                            opentofu apply, init_*.sh post-apply
                                            scripts, and (optionally) supabase/
                                            deploy.sh.
  knoe/ui/screens/cluster.py          Dual-cluster GKE kubecontext UI: prod env
  knoe/ui/screens/cfg.py              now shows separate "App Cluster:" and
                                       "DB Cluster:" dropdowns instead of a
                                       single "Kubernetes Context:" combo.
                                       New _app_kubectx_combo + _db_kubectx_combo
                                       widgets; new app/db_cluster_kubecontext
                                       tk.StringVars.
  knoe/core/{actions,env,milestones}.py
  knoe/core/ops/storage.py
  knoe/config.py, knoe/knoe_conf.py   Plumbing changes for the dual-cluster
                                       kubecontext flow + storage-class topology
                                       detection cleanup.
  knoe/tools/cleanup_cnpg_storage.py  (NEW) Stand-alone cleanup utility.
  tools/dashboard.sh                  (NEW) Dashboard helper.
  conf/knoe.cfg                       (NEW) Master cfg generated by knoe_conf.
  conf/dev/                           (NEW) Dev-mode cfg directory.
  conf/port-mapping.cfg               Port mapping tweaks for k3d.
  tests/installer/* (8 files)         New + extended tests for the dual-cluster
  tests/test_database_options.py      TUI, kubecontext save flow, storage ops,
                                       topology detection, deploy helpers,
                                       database-options screen.

Issues found in Junie's working state and fixed here:

  1. install.py was a 11-line import shim with no shebang, no `chmod +x`,
     no `if __name__ == '__main__'` block. `./install.py -c conf/k3d.cfg`
     returned `Permission denied` and `python install.py` did nothing.
     Added `#!/usr/bin/env python3`, `chmod +x`, and a __main__ block
     that delegates to `knoe.ui.screens.main()`. `./install.py --help`
     now prints the canonical argparse help.

  2. knoe/deployment.py had FIVE `subprocess.run()` call sites with no
     `timeout=` argument (`_run_script`, `_run_cmd`, the Ansible playbook
     fetch, `tofu init`, `tofu apply`). A hung child process — typical
     failure mode is a script waiting on stdin or a stalled network
     call — would lock up the installer indefinitely. Added timeouts:
       - Ansible kubeconfig fetch: 120s
       - tofu init: 300s
       - tofu apply, _run_script, _run_cmd: bounded by new module
         constant `_MILESTONE_TIMEOUT` (default 1800s = 30 min, override
         via `KNOE_MILESTONE_TIMEOUT_SECONDS` env var).
     `subprocess.TimeoutExpired` is caught explicitly; on timeout the
     run helpers return exit code 124 (conventional timeout code).

  3. `conf/k3d.cfg` was corrupted with MagicMock string-reprs on disk:
        KNOE_CONF = <MagicMock name='Canvas().tk.call().strip()' id='4743999712'>
        argocd.node_selector = <MagicMock name='mock.StringVar().get().strip()' id='...'>
     Likely path: Junie ran `./install.py -c conf/k3d.cfg` interactively
     in a non-Tk environment (or with a partially-mocked widget set) and
     the installer's "save current state" path wrote the mock-objects'
     `__repr__` strings into the cfg file. This commit reverts the cfg
     to its pre-Junie state. **Followup: harden the cfg save path
     against non-string widget values** — track separately.

  4. The corrupted cfg caused the installer to call `os.makedirs()` on
     the mock-string values, producing 10 directories on disk literally
     named `<MagicMock name='Canvas().tk.call().strip()' id='4733210304'>/`
     etc., with 5–86 files of install artifacts inside each. Removed.

The "final step is timing out" the user reported was almost certainly
issue #2 above: install.py walked the milestone pipeline, hit one of
the unbounded subprocess.run calls, and the wrapped command (probably
supabase/deploy.sh, which Junie was reading for context when her
session timed out) hung. With the timeouts in place that path now
exits cleanly with rc=124 instead of locking up.

Verification:
  - pytest tests/installer/ -q                                   750 passed in ~25s
  - python3 -c "import knoe.deployment"                          imports clean
  - ./install.py --help                                          prints argparse help
  - find . -maxdepth 1 -type d -name '<MagicMock*' | wc -l       0
  - head -7 conf/k3d.cfg                                          clean (no MagicMock)

Out of scope for this commit (followups):
  - The cfg save-path that wrote mock-objects-as-strings (issue #3 root cause).
    Reproducer: launch the installer in an env where Tk widget vars are
    `unittest.mock.MagicMock` instances. The cfg save code should refuse to
    serialize non-str values rather than calling `str()` on a MagicMock.
  - The k3d-mirror Phase 3 brief (`docs/plans/junie/k3d-knoe-auth-pod-deploy.md`)
    is still open — Junie picked a different prompt this round.

Co-authored-by: Junie <junie@jetbrains.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 23:51:14 -07:00
chrisfu
00f0ebec07 Merge claude/crazy-bose-fec256 into main
Bringing the long-running session-feature branch back into main in one
deliberate sweep. The branch carried the cluster work that's been live for
weeks (cross-cluster CNPG metrics, Grafana w/ Google OAuth, supabase
oauth2-proxy, cluster recovery, pg.0.knoe.dev + per-engineer onboarding,
GCS-backed CNPG backups via Workload Identity, the env-contamination
guard, the Junie brief queue, the cnpg-grafana CSRF + memory-request
fixes from today), while main accumulated Junie's parallel knoe-auth
Phase 2 OIDC work (full provider surface: discovery, authorize, token,
userinfo, JWKS, RS256 signing, code exchange, session services).

Key decision: the two branches did COMPETING rebrands off the same
starting point (5ba9b63, 2026-04-27):

  - claude branch (commit b355855, earlier): org.prole.authority.* →
                                              dev.knoe.auth.*
                                              (artifact renamed to
                                              knoe-auth.jar)
  - main (commit 9daa94b, recent): org.prole.authority.* →
                                    dev.knoe.authority.*
                                    (kept "authority" artifact name)

dev.knoe.auth wins: cluster runs from this name, the Maven artifact is
already knoe-auth.jar, and the broader rename is the documented
namespace direction (per ~/.claude/projects/-Users-chrisfu-dev-knoe-db/
memory/MEMORY.md). All of main's recent Phase 2 OIDC content was ported
from authority/src/.../dev/knoe/authority/ into
authority/src/.../dev/knoe/auth/ with package declarations rewritten.

== File-level resolution summary ==

Textual conflicts (4):

  authority/pom.xml
    - Took our artifactId="auth"
    - Took our branch's removal of spring-security-kerberos-client
      (verified: Junie's Phase 2 OIDC code does not import it; the dep
      was already-dead config)

  docs/pipeline-phases.md
    - Took our branch's "Phase 1 not started" status. Main had a
      misplaced "✅ Complete" with a knoe-auth-Phase-1 commit ref
      in the autobuild Phase 1 section — different domain.

  docs/plans/knoe-auth-round-1.md
    - Took our branch's dev.knoe.auth file table (vs main's
      dev.knoe.authority listing). Pure rename mismatch.

  supabase/helm/knoe-supabase/templates/kong/config.yaml
    - Took our branch's onboard route + plain dashboard wiring.
      Main had an oauth2proxy.enabled toggle that put oauth2-proxy as
      a Kong upstream — but the deployed architecture (commit 25f1b2e)
      has oauth2-proxy in FRONT of Kong, not behind. Main's wrapper
      reflected an architecture that was never deployed.
    - Took our branch's removal of basic-auth from dashboard route
      (queue #15 brief still tracks the matching values.yaml /
      kong/deployment.yaml cleanup).

Java tree reconciliation (44 file-pairs):

  20 dual-path source files + 2 dual-path tests
    Body-identical between main's authority/ and our branch's auth/
    after stripping package decls — main's commit 9daa94b was a pure
    rebrand. Took our branch's auth/ version for all 22.

  8 main-only source files (Phase 2 OIDC), ported into auth/:
    web/JwksController.java
    web/OidcAuthorizeController.java
    web/OidcDiscoveryController.java
    web/OidcTokenController.java
    web/OidcUserInfoController.java
    session/OidcCodeService.java
    session/OidcTokenService.java
    session/SessionService.java

  12 main-only test files, ported into auth/:
    HealthControllerTest.java
    enroll/EnrollValueTypesTest.java
    enroll/EnrollmentControllerTest.java
    enroll/TotpServiceTest.java
    kerberos/KadminClientTest.java
    kerberos/KerberosSpnegoResultTest.java
    web/LoginControllerTest.java
    admin/AdminControllerTest.java
    user/PrincipalNormalizerTest.java
    regression/IdentityRegressionTest.java
    session/OidcCodeServiceTest.java
    session/SessionServiceTest.java

  Port mechanics: read main:authority/...<file> via git show, then sed
  rewrite `package dev.knoe.authority` → `package dev.knoe.auth` and
  `import dev.knoe.authority` → `import dev.knoe.auth`. Body content
  unchanged.

  authority/src/main/java/dev/knoe/authority/ — DELETED (duplicate)
  authority/src/test/java/dev/knoe/authority/  — DELETED (duplicate)

== Verification ==

- grep -rln '<<<<<<<' across .java/.md/.yaml/.yml/.sh/.xml/.tpl: clean
- find authority/src -path '*/dev/knoe/authority*': empty (subtree gone)
- grep 'package dev.knoe.authority' across repo: clean
- bash -n install.sh deploy.sh etc/preflight_kubecontext.sh: clean
- git ls-files -u | wc -l: 0 unmerged paths
- helm lint supabase/helm/knoe-supabase: pre-existing failure on
  studioIngress.enabled undefined in values.yaml (introduced by Junie
  on main; unrelated to this merge — flagging as follow-up).

== Followups (carried into TODO ranked queue or noted here) ==

  - helm lint failure: studioIngress block in values.yaml is missing
    enable flag; templates/studio/{ingress,oauth2proxy-deployment,
    oauth2proxy-service}.yaml all reference studioIngress.enabled with
    no default. Pre-existing on main; not introduced by this merge.
  - The five Junie briefs filed on this branch are now reachable from
    main at docs/plans/junie/{02,06,07,13,15}-*.md. Junie can pick them
    up in any order.
  - knoe-auth Phase 2 OIDC source (now at dev.knoe.auth.*) is not yet
    deployed to the cluster. Deployment is its own task.
  - The branch claude/crazy-bose-fec256 stays in place (worktree at
    .claude/worktrees/crazy-bose-fec256 may have ongoing context for
    Claude Code sessions). Safe to delete once next session starts
    cleanly from main.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 16:39:10 -07:00
chrisfu
03d89eaa53 Phase 0: test pipeline foundation — pyproject.toml, IntelliJ run configs, coverage fix, welcome mode selector
Co-authored-by: Junie <junie@jetbrains.com>
2026-04-27 14:44:46 -07:00
chrisfu
b421f49af1 Migrate Ansible-based scripts and documentation to 1Password integration. Removed deprecated and redundant Ansible vault workflows, added 1Password CLI handling for secrets management, and updated tests to reflect the change. 2026-04-22 20:54:13 -07:00
chrisfu
55b6a6aff3 Complete rebranding from prole to knoe and fix macOS application identity. Bulk renamed 'prole' to 'knoe' across code, scripts, and manifests. Updated Makefile with 'knoe', 'build', and 'start' targets. Implemented macOS Application Bundle (.app) support for correct identity. Fixed macOS 'Python' process name to 'Knoe.DB Installer' via Objective-C bridge. Standardized application name to 'Knoe.DB Installer' across all interfaces.
Co-authored-by: Junie <junie@jetbrains.com>
2026-04-22 15:08:35 -07:00
chrisfu
af89ef547f Implement minimal installation mode (--min) with containerd support
Co-authored-by: Junie <junie@jetbrains.com>
2026-04-22 09:48:04 -07:00
chrisfu
f44ce32a27 Rebrand and consolidate authentication and scanning systems. Consolidate Kerberos authentication system into authority/src and rename ProleAuthApplication to KnoeAuthApplication. Remove deprecated prole/, prole-app/, prole-mssql-db/, and prole-tools-app/ directories. Migrate and rename prole-net/prole-agent to scan/network-agent. Update install.sh, Makefile, and documentation to use Knoe branding and new scan paths. Rebrand configuration properties and URLs to knoe.dev and svc.knoe.dev.
Co-authored-by: Junie <junie@jetbrains.com>
2026-04-22 09:28:14 -07:00
chrisfu
fddaf80823 checkpoint: make node placement config-driven and add GitLab rerun fast-path
- remove hardcoded physical host assumptions in UI/init flows and rely on config-driven selectors\n- optimize GitLab reruns to skip unnecessary operator upgrades/long reconcile waits when unchanged\n- add and update regression/shell tests for fast-path and k8s context/ingress/garage behaviors

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-14 06:43:37 -07:00
chrisfu
e3c2e625f1 refactor(config): separate k3d k3s and gke config entrypoints
Rename env config files from conf/*/prole.cfg to conf/k3d.cfg, conf/k3s.cfg, and conf/gke.cfg. Update shell/Python loaders and etc/deploy scripts to resolve named configs cleanly while keeping legacy fallback behavior. Align k3s Ansible tasks, docs, and regression coverage with the new configuration layout.

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-11 22:20:45 -07:00
chrisfu
b4fc5641db checkpoint: sanitize prole config for host portability
- fix conf/prod/prole.cfg by replacing hardcoded /Users/chrisfu paths with /Users/chrisfu

- remove injected [update.sh] log lines and stray password artifacts so config is executable

- include latest pending updates across deployment config, UI/core flow, vault/network artifacts, and helper scripts
2026-04-10 05:58:40 -07:00
chrisfu
5cd43c873c feat(gke): finalize dual-cluster app/db topology
- separate default app and db cluster contexts (knoe-dev-0 and knoe-cnpg-0) across actions, ops flows, and installer config\n- update CNPG placement and GKE manifests to keep PostgreSQL on the db cluster while app services stay on the app cluster\n- add GCS-backed Barman ObjectStore manifest and cross-cluster Garage patch workflow\n- refresh UI cluster/database/service screens and monitoring wiring for dual-cluster operation\n- add reset/patch scripts, db context selection test coverage, and architecture/network documentation updates

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-08 23:12:51 -07:00
chrisfu
64e1cef16c checkpoint: stabilize k8s cluster setup and shelve cluster-storage UI
- make cluster-storage milestone opt-in and remove installer cluster-storage step from UI navigation\n- add cluster storage browser and GKE cluster ops helpers with CLI coverage\n- update k8s/CNPG config and install flow files for corrected cluster setup\n- add/refresh tests for storage browser, GKE ops, prod config, and service-layer navigation

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-07 21:11:25 -07:00
chrisfu
df6de9138e checkpoint: summarize CNPG init and backup hardening work
- harden CNPG backup ObjectStore readiness fallback in statusless clusters by validating controller availability and expected spec

- improve backup/plugin reconciliation paths and related test coverage for barman-cloud flows

- include ongoing GKE/CNPG deployment and configuration updates from this checkpoint

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-06 05:51:26 -07:00
chrisfu
3312c39b1f feat: GCP/GKE CNPG hardening, Artifact Registry traffic light, and knoe-system namespace fixes
UI screens
- database.py: fix mode detection to use env_key priority (prod→k8s, service→k3s) so stale DEPLOYMENT_MODE never overrides the user's chosen environment
- database.py: Registry status reads ARTIFACT_REGISTRY_AVAILABLE persisted by cluster screen; uses SERVICE_NAMESPACE for Artifact Registry repo name
- cluster.py: add Artifact Registry traffic light (amber→green/red) to prod section; _check_artifact_registry_async persists ARTIFACT_REGISTRY_AVAILABLE into Global cfg
- cluster.py: re-trigger Artifact Registry check after GKE cluster selection so the light re-evaluates once region is available from KUBECONTEXT
- cluster_nodes.py: fix TclError on Python 3.14 — pady=(2,0) tuple → pady=2 scalar
- __init__.py: seed knoe-system namespace when saved value is "default", not only when empty
- services.py: replace hardcoded "Prole DB" log string with dynamic cnpg_cluster name

Core ops
- cloudnative_pg.py: replace one-shot Barman plugin retry with 6-attempt loop; first cert-manager/x509 failure triggers rollout restart + 30 s CA propagation wait; subsequent failures back off up to 60 s per attempt
- cloudnative_pg.py: TLS CA CN now uses cluster_name instead of hardcoded "Prole CNPG CA"
- registry.py, garage_store.py: refactored into per-mode modules (k3d/k3s/k8s registry and garage store, shared _garage_common)

Deploy / config
- deploy/gcp/gke/knoe-db.yaml: GKE-specific CNPG cluster manifest (rw/ro/r on separate nodes with premium-rwo storage)
- etc/init_common_services.sh, modes/k8s/knoe-db/.version: updated for current deploy
- kong-deployment.yaml: updated manifest

Tests
- test_cluster_nodes_render_smoke.py: add pack/grid, winfo_children, winfo_reqheight, update_idletasks, grid_slaves to dummy widgets; monkeypatch tk.Label so CNPG placement render completes without a real Tkinter root

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-04 19:36:08 -07:00
chrisfu
4a8d9cc90d feat: full GKE/prod deployment pipeline from UI to Artifact Registry
## GCP / Cluster Environment Screen
- Auto-populate Cloud tab from conf/prod/gcp.cfg on screen open (org_id,
  billing_account, billing_project, project_id)
- gcloud auth validity checked on screen startup; friendly modal dialog
  streams gcloud auth login output live so user never leaves the app
- Live GKE cluster browser: fetches clusters via gcloud container clusters
  list, displays with checkmark selector, auto-selects saved cluster
- Selecting a cluster runs get-credentials, sets KUBECONFIG/KUBECONTEXT,
  and syncs the region dropdown to the selected cluster's location
- Region dropdown populated live from gcloud compute regions list with
  checkmark on currently selected region; graceful fallback when offline
- New 'GCP Storage' tab with workload->StorageClass mapping (CNPG->premium-rwo,
  Redis/Monitoring->standard-rwo, Garage->garage-hdd) and Fetch from Cluster
- Provider readonly field styled correctly (no solid-black on macOS)
- Stale prole.cfg/conf/prole.cfg symlinks removed; all config I/O now
  resolves env-specific paths via prole_conf.entrypoint_path()

## GKE Autopilot Compatibility (Common Services)
- Synology iSCSI StorageClass and static PVs guarded behind PROLE_MODE!=k8s
  in init_openbao.sh (GKE Autopilot forbids hostPath/iSCSI volumes)
- In-cluster Docker registry (hostPath) skipped in k8s mode; GCP Artifact
  Registry used instead
- Kong renamed knoe-svc-kong in k8s mode; all health-check kubectl calls in
  init_common_services.sh and status_common_services.sh updated accordingly
- DNS endpoints switched from *.prole.org to *.knoe.dev in k8s mode
  (api.knoe.dev, git.knoe.dev, svc.knoe.dev); ingress uses gce class
- New GKE-clean Kong manifests under deploy/opentofu/k8s/manifests/prole/:
  no k3s node affinity, explicit Autopilot resource requests/limits

## Garage S3 Store (GKE)
- New garage-statefulset-gcp.yaml targeting garage-hdd StorageClass
  (pd-standard, avoids SSD_TOTAL_GB quota exhaustion in us-west3)
- New storageclass-gcp-hdd.yaml (pd-standard, Retain, WaitForFirstConsumer)
- GCP StorageClass manifests skipped on re-runs (Autopilot built-ins are
  immutable; skip-if-exists guard added)
- PVC deletion guard extended to cover any storageClass (not just synology)
  so stale claims are cleaned before StatefulSet recreation

## Topology (GKE Autopilot)
- DaemonSet collector skipped in prod mode (forbidden in kube-system by
  GKE Warden); Kubernetes-only node facts path used instead
- All ready GKE nodes assumed cnpg-eligible and monitoring-eligible without
  taint/synology-mount checks (skip_collector + assume_nodes_eligible flags)

## KUBECONFIG / kubectl (k8s mode)
- actions.py: new elif mode==k8s branch sets KUBECONFIG=~/.kube/config
  and injects KUBECONTEXT from prole_cfg_data into script env
- _build_kubectl_cmd falls back to Global.KUBECONTEXT when
  init_cluster.selected_kubectx is empty
- _activate_selected_gke_cluster persists KUBECONFIG/KUBECONTEXT to
  prole_cfg_data and saves prole.cfg immediately after get-credentials

## Database Build Screen (GKE)
- Registry display shows correct Artifact Registry URL
  (<region>-docker.pkg.dev/<project>/<namespace>/knoe-db) in green
- Build+push: gcloud auth configure-docker, auto-creates AR repository
  named after SERVICE_NAMESPACE (e.g. knoe-system) if missing, then
  docker tag + push; falls back to gcr.io if region unavailable
- GCP config loaded from conf/prod/gcp.cfg on every screen entry;
  keys normalised to lowercase so project_id lookup is always consistent

## Config / Namespace persistence
- prole_conf.py activate_environment: symlink creation removed; sets
  CLUSTER_ENV env-var so all subsequent calls resolve correct env directory
- knoe/ui/screens/__init__.py: startup config load uses entrypoint_path()
  instead of hardcoded conf/prole.cfg; seeds SERVICE_NAMESPACE=knoe-system
  for managed envs so Common Services never defaults to 'default'
- cfg.py _save_prole_cfg: saves to env-specific path via entrypoint_path()
- etc/prole_cfg.sh: removed all ln -snf symlink creation

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-04 12:38:16 -07:00
chrisfu
a5cd0c45f4 Update conf, k8s registry, knoe UI screens, etc init scripts, and scan data
- conf/service/prole.cfg, port-mapping.cfg, cnpg-placement updated

- k8s/registry/deployment.yaml updated

- knoe/ui/screens: base.py, knoe_users.py, __init__.py updated

- etc/ init scripts refreshed (gitlab, knoe_users, registry, prole_cfg)

- modes/k3s/knoe-db/.version bumped; scan network_description and ansible_inventory updated

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-02 23:40:16 -07:00
chrisfu
e3ca33e4c7 feat(installer): add Knoe User Accounts screen after Kerberos provisioning
New KnoeUsersScreenMixin screen (knoe_users) appears in the install wizard
immediately after Kerberos Authentication (when kerberos is enabled).

- Runs etc/init_knoe_users.sh initialize in a background thread with live
  console output, forwarding SERVICE_NAMESPACE/KNOE_KDC_NAMESPACE from the
  installer state so the script always targets the right namespace
- Optional Gitea and GitLab admin token fields for service admin promotion
- Next button is disabled until provisioning succeeds (idempotent: button
  re-enables on failure so the user can fix and retry)
- Nav chain: kerberos_config -> knoe_users -> argocd_config
  Prev from argocd_config respects kerberos_enabled to route correctly
- Records STATUS in prole.cfg under [Knoe User Accounts] section

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-01 21:45:24 -07:00
chrisfu
e9e53aac40 Migrate install.py logic to install.sh and restructure installer flow. Update core references, tests, and navigation to reflect the new Knoe installer architecture. 2026-04-01 01:17:04 -07:00
chrisfu
93f7259722 chore: checkpoint k3s deployment and knoe updates
- update Kong/Supabase ingress and service manifests for k3s

- expand init scripts and runtime config/topology handling

- refresh KNØE UI screens and Supabase deployment/render logic

- include related env helper test adjustments

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-31 23:15:54 -07:00
chrisfu
c0a7d0c5dc Refine installer orchestration and Supabase rendering paths
- improve installer/action/controller flow and shell-variable expansion handling across screens\n- adjust Supabase Helm rendering and storage deployment templates\n- align monitoring, cloudnative-pg and repair pipeline behavior with updated config paths\n- refresh and expand installer/core regression tests around milestones, navigation and repair logic

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-31 23:15:54 -07:00
chrisfu
5a0f8ab4f8 Make init-script secret checks warning-only and unblock Next
Co-authored-by: Junie <junie@jetbrains.com>
2026-03-30 01:26:38 -07:00
chrisfu
6232bd89ee Merge remote-tracking branch 'origin/main'
# Conflicts:
#	tests/installer/test_services_init_scripts.py
2026-03-30 00:46:56 -07:00
chrisfu
22420408f9 Align installer namespace/topology handling and add regression tests
Co-authored-by: Junie <junie@jetbrains.com>
2026-03-29 23:45:56 -07:00
chrisfu
ae42d4607b Align installer namespace/topology handling and add regression tests
Co-authored-by: Junie <junie@jetbrains.com>
2026-03-29 23:08:14 -07:00
chrisfu
d02aadae89 Add node management workflow and installer config persistence updates
- add tools/manage-node.sh for k3s node add/remove automation with inventory reconciliation

- include kubectl drain flag compatibility and --ssh-host targeting for uninstall workflows

- persist installer cfg/env values and update deploy/database/cfg UI handling

- refresh related inventory/network artifacts and expand installer/database test coverage

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-29 11:22:33 -07:00
chrisfu
fb7993de1d feat: add production config schema, validation, and API with tests
- Introduced `knoe.core.prod_config` with production configuration schema and helper methods.
- Added validation logic to enforce required fields and expected formats.
- Implemented in-process API (`ProdConfigApi`) for managing production configs.
- Added test coverage for default config behavior, validation, YAML generation, and API workflow.
- Updated cluster UI layout and tests to integrate production config with new navigation flows.
- Added host inventory and updated service flow to reflect production setup changes.
2026-03-28 10:43:25 -07:00
chrisfu
761d80486b feat: add storage probing and operational service updates
- add reusable storage probing subsystem with discovery, bounded probe execution, IO classification, caching, and topology integration

- render per-node storage inventory in Cluster Nodes UI and extend installer test coverage for topology/storage behavior

- introduce core service operation modules and align actions, milestones, services, and supporting configs/scripts for repair/update workflows

- update CNPG/Supabase/database artifacts, placement and port mapping configs, plus related integration tests

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-26 09:44:23 -07:00
chrisfu
65d03ed442 stable: slimmed Supabase + CNPG 3-node healthy build (2026-03-25)
STABLE BUILD — all 35+ pods Running, zero non-Running pods, CNPG ready=3.

## Architecture changes (Supabase slim-down)
- Removed local Supabase Kong, Storage, MinIO, Imgproxy from deployed stack
- Rewired API routing to shared Kong in kube-system
- Rewired object storage to shared Garage in knoe-system
- Supabase footprint reduced to control-plane only: auth, rest, meta,
  analytics, realtime (studio port-forward retained)
- Supabase pods renamed: stripped redundant '-prole-supabase-' prefix via
  per-component fullnameOverride (e.g. supabase-auth, supabase-rest, ...)
- Supabase workloads pinned to merlin.prole.org (memory-safe node)

## File changes
- conf/service/prole.cfg
    * Replaced all hardcoded /Users/chrisfu/dev/prole paths with ${HOME}/dev/prole
      so config is portable when run from myrddin.prole.org or any user home
    * SUPABASE_ENABLED=True, SUPABASE_PRIMARY_NODE=merlin.prole.org in
      both [Global] and [Optional Features]
    * supabase_enabled=true in [Inputs]/init_cluster section
- supabase/helm/render_supabase.py
    * Explicitly disabled: kong, storage, minio, imgproxy
    * Active components pinned via nodeSelector to merlin.prole.org
    * fullnameOverride per component to clean up pod names
    * Default primary node changed from pi.prole.org -> merlin.prole.org
- supabase/deploy.sh
    * Removed svc/kong port-forward; API via shared Kong in kube-system
    * Removed imgproxy/storage from PV/PVC prep and helm --set persistence args
    * Fixed kubectl exec missing -i flag for psql heredoc stdin in
      setup_knoe_db_for_supabase
    * Updated Studio access messaging to reference shared platform Kong
- k8s/prole/knoe-db.yaml
    * instances: 3
    * Added control-plane/master tolerations for myrddin scheduling
- deploy/opentofu/k3s/manifests/prole/knoe-db.yaml
    * instances: 3, added control-plane tolerations (mirrors k8s/prole/)
- k8s/prole/iscsi-pvs.yaml
    * Expanded d003-data/d003-wal nodeAffinity to [pi, merlin, myrddin]
      to reflect post-OOM-recovery live state
- k8s/registry/deployment.yaml
    * nodeSelector restored to myrddin.prole.org (hostPort 5000 owner)
    * Added control-plane + master tolerations so pod schedules correctly
- modes/k3s/knoe-db/.version
    * Aligned to 140 (image tag 18-140) matching repo-root version
- etc/init_common_services.sh, etc/init_registry.sh
    * Minor fixes aligned with slimmed service layout and node targeting
- infrastructure/inventory/host_vars/pi.prole.org.yml
    * Updated host vars to reflect post-recovery pi state
- knoe/core/topology.py, knoe/ui/screens/cluster.py,
  knoe/ui/screens/cluster_nodes.py
    * Topology and UI updates for 3-node cluster representation
- scan/network_description.txt
    * Refreshed network scan output post-recovery
- conf/port-mapping.cfg
    * Updated port mapping to reflect removal of local Kong/storage forwards
- scripts/cleanup_stale_rs.sh (new)
    * Utility script to prune orphaned ReplicaSets after rollouts

## Verified stable state
- CNPG: Cluster in healthy state, ready=3
    knoe-db-2 (merlin, primary), knoe-db-3 (pi), knoe-db-5 (myrddin)
- Supabase: 5 pods Running on merlin.prole.org
    supabase-auth, supabase-rest, supabase-meta,
    supabase-analytics, supabase-realtime
- Monitoring: all prometheus/grafana/alertmanager pods Running on merlin
- Registry: Running on myrddin.prole.org (hostPort 5000)
- Zero non-Running / non-Completed pods across all namespaces

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-25 11:56:40 -07:00
chrisfu
a1177d71bc Add GitOps provider choice screen and cluster UI improvements
- Add _render_gitops_choice_page() to GitOpsScreenMixin for explicit
  Gitea / ArgoCD / None provider selection with radio buttons
- Extend cluster screen with GitOps provider radio group and canvas layout
- Refactor services screen layout and navigation registration
- Register new gitops choice screen in screens __init__ / navigation
- Expand Kong init scripts (etc/ and mock_val/) with additional logic
- Update init_cnpg_backup.sh with minor fix
- Refresh conf/service and conf/prod prole.cfg generated configs
- Update conf/port-mapping.cfg port entries
- Update Supabase Helm values.yaml
- Bump modes/k3s/knoe-db/.version
- Update network scan description
- Add/expand tests: test_cluster_screen_layout, test_gitops_choice_screen

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-24 22:12:25 -07:00
chrisfu
6f29deba99 Supabase/CNPG stability: pin Supabase to merlin, reduce CNPG to 2 instances, fix supabase_enabled
Changes address resource pressure (merlin OOM with 3 CNPG + Supabase + monitoring) and
the supabase_enabled=false override in silent mode that blocked Supabase deploy.

- conf/service/prole.cfg:
  - SUPABASE_PV_NODE/SUPABASE_PRIMARY_NODE = merlin.prole.org (d002 storage)
  - SUPABASE_STORAGE_CLASS = merlin-local-iscsi-d002
  - CNPG_ELIGIBLE_NODES: removed pi.prole.org (pi NotReady, d003 inaccessible)
- k8s/prole/knoe-db.yaml:
  - instances: 3 -> 2 (merlin + myrddin only, pi excluded)
  - maxSyncReplicas: 1 -> 0 (safe for 2-node cluster)
- supabase/deploy.sh:
  - default SUPABASE_PV_NODE to merlin.prole.org, base dir to /synology/d002/supabase
  - add idempotent ALTER ROLE password sync after CREATE ROLE IF NOT EXISTS
  - resolve live DB_PASSWORD from knoe-db-superuser secret before helm render
- supabase/helm/render_supabase.py:
  - resolve SUPABASE_STORAGE_CLASS from env/cfg, apply to storage/analytics PVCs
  - DATABASE_NAMESPACE resolution: check env + cfg Global section before discovery
  - DB_PASSWORD from env takes priority over cfg (live cluster secret wins)
  - default primary node = merlin.prole.org
- knoe/ui/screens/cluster.py: minor display update

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-24 13:52:38 -07:00
chrisfu
ad36f4ee54 Stage 2: complete CNPG shell-to-Python cutover, remove init_cloudnative_pg.sh
All active runtime paths that previously shelled out to etc/init_cloudnative_pg.sh
now dispatch through Python. The shell script has been removed from git.

New files:
- knoe/core/ops/cloudnative_pg.py: canonical Python owner for CNPG lifecycle with
  public API (initialize, deploy, rollout, ensure_operator, pin_controller,
  install_barman_plugin) and internal helpers (_apply_manifest, _wait_cnpg_pods,
  _reconcile_instances, etc.)
- prole/tools/run_cnpg_coverage.py: coverage entry points for k3d/k3s modes,
  report sub-command, and check-shell scanner to confirm no live Python dispatch
  to the removed shell script

Modified files:
- knoe/core/actions.py: replace shell dispatch in _step_init_scripts,
  _step_cnpg_deploy, and repair pipeline with Python calls
- knoe/core/milestones.py: replace shell dispatch in InitializationScriptsMilestone
  and DeploymentMilestone
- knoe/ui/screens/services.py: replace shell dispatch for init-scripts step,
  deploy button, and rollout button
- prole/deployment.py: replace shell dispatch in _run_post_apply_scripts
- status.py: remove init_cloudnative_pg.sh from _STATUS_SCRIPTS list
- tests/installer/test_actions_helpers.py: mock Python functions, assert shell
  script is never dispatched
- tests/installer/test_milestones.py: same
- tests/installer/test_services_init_scripts.py: same

Removed:
- etc/init_cloudnative_pg.sh (git rm)

Verification:
- prole.tools.run_cnpg_coverage check-shell reports clean
- All 61 tests in the affected test files pass

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-23 19:28:04 -07:00
chrisfu
2505497f0f Standardize namespace and cluster config flow
- Replace ambiguous NAMESPACE handling with SERVICE_NAMESPACE, DATABASE_NAMESPACE, and CLUSTER_NAME across config parsing/defaults, environment setup, actions, and UI bindings.

- Persist DATABASE_NAMESPACE and CLUSTER_NAME from Database Browser on Next and use them consistently in CNPG deployment/runtime logic.

- Update milestones/tests and include related config/tooling artifacts (cleanup_cnpg_storage tool, service config/version/network updates).

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-23 16:52:40 -07:00
chrisfu
cf51335ced Fix kube context handling and monitoring eligibility
- detect local k3s node kubeconfig and skip kubectx/use-context mutation when already targeting local API\n- add configurable KUBE_CONTEXT_NAME resolution with compatibility fallbacks and switch only when required\n- update init scripts to use ensure_kube_context helper naming\n- broaden monitoring eligibility to discovered /synology/d### mounts so /synology/d004 qualifies\n- add focused kube-context and topology tests covering local/remote and read-only kubeconfig cases

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-23 13:50:29 -07:00
chrisfu
0e806a9233 checkpoint: tighten CNPG k3s storage flow and policy updates
- normalize K3S/CNPG/OpenTofu env keys across installer and configs; - harden CNPG storage validation/provisioning and fail fast in init script; - update DB manifests/recovery template and port/network mapping data; - expand cluster node policy UI coverage and shell/python tests; - add k3s mode version marker files

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-23 12:39:03 -07:00
chrisfu
a069989315 Rename prole-db to knoe-db, add knoe-auth as cluster-internal KDC
Itemized changes:

1. knoe-auth: New cluster-internal KDC and SSO gateway service
   - Created etc/init_knoe_auth.sh based on init_kdc.sh with knoe-auth naming
   - Namespace defaults to SERVICE_NAMESPACE (knoe-system)
   - ConfigMap: knoe-auth-kdc-config, Secret: knoe-auth-secrets
   - Legacy cleanup removes old auth/dog/authority deployments

2. Orchestration: knoe-auth initializes before CloudNativePG
   - Updated prole.sh to insert init_knoe_auth.sh as step 2 (before CNPG)
   - Renumbered all subsequent initialization steps

3. Kong routing: Updated init_kong.sh to route to knoe-auth in SERVICE_NAMESPACE

4. Comment/reference updates for knoe-auth
   - Updated init_common_services.sh, init_service_layer.sh, init_kerberos.sh

5. prole-db renamed to knoe-db across the entire codebase
   - Renamed prole-db/ directory to knoe-db/
   - Renamed all prole-db Kubernetes manifests (deploy/opentofu, k8s/)
   - Renamed scripts: docker-root-knoe-db.sh, docker-run-knoe-db.sh, test-cnpg-knoe-db.sh
   - Renamed etc/init_prole-db-reset.sh to etc/init_knoe-db-reset.sh
   - Renamed etc/prole-db-passwwd.sh to etc/knoe-db-passwwd.sh
   - Renamed mock_val counterparts accordingly
   - Renamed tests/etc/test_init_prole-db-reset.sh to test_init_knoe-db-reset.sh
   - Renamed docs/prole-db-documentation-mcp-architecture.md to knoe-db variant
   - Renamed modes/k3d/prole-db/ to modes/k3d/knoe-db/
   - Renamed prole-db.iml to knoe-db.iml

6. Configuration updates
   - Updated conf/dev, conf/prod, conf/test, conf/service prole.cfg files
   - Updated conf/port-mapping.cfg
   - Updated etc/prole_cfg.sh and mock_val/prole_cfg.sh
   - Updated service/prole.cfg

7. Kubernetes manifests and deploy configuration
   - Updated deploy/opentofu/k3s ArgoCD application YAMLs
   - Updated kong-configmap.yaml and kustomization.yaml
   - Updated k3s/kong-config.yml and prole-resources.yaml
   - Updated prole-mssql-db deployment YAMLs
   - Updated supabase helm render and deploy scripts

8. Infrastructure and GCP Terraform
   - Updated deploy/gcp/terraform: folders, groups, IAM, service-projects

9. Python/installer code updates
   - Updated knoe/core: actions, build_context, controller, env, milestones
   - Updated knoe/milestone.py
   - Updated knoe/ui/screens: cfg, database, database_options, deploy, docker,
     navigation, security, services, validate
   - Updated knoe.spec, status.py

10. Shell script updates
    - Updated etc/: build_db, init_cloudnative_pg, init_cnpg_backup,
      init_db_manager, init_forgejo, init_gitlab, init_monitoring, init_openbao,
      init_port_forwards, init_postgrest, init_supabase_ports, status
    - Updated mock_val/ counterparts for all above scripts
    - Updated prole-net/init-prole-dns.sh
    - Updated bin/prole-kpf.sh, gitea/deploy.sh, supabase/deploy.sh

11. Test updates
    - Updated tests/etc/: test_init_cloudnative_pg*, test_init_cnpg_backup*,
      test_init_kdc*, test_init_kerberos*, test_init_kong*, test_prole_cfg*
    - Updated tests/installer/: test_actions_helpers, test_cfg_save_kubecontext,
      test_controller, test_core_classes, test_milestones, test_milestones_extended,
      test_namespace_propagation
    - Updated tests/: test_database_options, test_navigation,
      test_render_supabase_hostname, test_docker_build_fix,
      test_all_prole_home_fixes, silent_install_test, final_test

12. Documentation updates
    - Updated docs/: DOCKER-BUILD-FIX, PROLE-CFG-SECRETS, PROLE-HOME-DIRECTORY,
      build-system, patent
    - Updated scan/network_description.txt
    - Updated pom.xml

13. Miscellaneous script updates
    - Updated root-level: _adopt_replica_pvcs, _fix_replica_merlin, _import_pi,
      _patch_cluster, _prebind_pvcs, _rebind_d002, _rebind_d002b, test_resolve
    - Updated scripts/generate_spec.py

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-22 22:16:21 -07:00
chrisfu
9b9d6fdc88 Rename Prole storage objects to Synology and sync deployment updates
- Rename iSCSI storage class and PV/PVC selectors/labels from prole to synology across k8s and OpenTofu manifests\n- Update CNPG/OpenBao/Garage/monitoring init flows, render helpers, and mock scripts for synology-backed storage objects\n- Integrate related UI/core/service config/version updates and add supporting regression tests for CNPG storage/image behavior\n- Keep storage reconciliation tests aligned with current CNPG affinity output

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-22 21:50:02 -07:00
chrisfu
0262bd4be2 Refactor to support mode-scoped DB build contexts and versioning
- Introduce mode-scoped directories under `~/.prole` for build contexts, runtime data, and versioning.
- Update DB image version generation and Dockerfile creation to respect deployment modes (e.g., k3d, k3s, k8s).
- Adjust UI, core actions, and scripts to avoid cross-mode conflicts and ensure correct version isolation.
- Add tests for mode-aware behaviors and update existing tests for new paths and markers.
- Bump PostgreSQL and image versions to `18` and `138`, respectively.
- Update Kubernetes manifests to align with new image and version configuration.
2026-03-22 05:47:50 -07:00
chrisfu
1353882260 Update PROLE_HOME resolution and enhance iSCSI PV handling
- Implement centralized `resolve_prole_home` utility for consistent environment-based `PROLE_HOME` resolution across modules
- Replace hardcoded home paths with `resolve_prole_home`
- Refactor PV management to support iSCSI mounts and node placement from Ansible manifests
- Improve Kubernetes manifest handling to dynamically apply namespaces per document
- Adjust `knoe-db` build context path and related tests
- Add utilities for detecting and applying Ansible-defined node labels and PVs
2026-03-22 02:43:52 -07:00
chrisfu
4ee2b259c9 Checkpoint: rename installer to knoe + harden db build context
- Add build-context helper to copy Docker context safely (ignore runtime data, keep symlinks)

- Update UI and core actions to use ~/.prole/build and shared copy helper

- Add/adjust tests and scripts; introduce knoe ops helpers and update manifests

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-22 01:45:21 -07:00