Commit Graph

46 Commits

Author SHA1 Message Date
chrisfu
6d484ef13e docs(plans): k3d-mirror-of-GKE plan + Phase 1 brief for Junie
The next development thread is knoe-auth Phase 2 OIDC iteration, which
just landed on main but has no fast inner-loop. Instead of bouncing
edits through the GKE deploy chain, we want a laptop-resident model
where knoe-auth runs from the IDE / mvn against real Postgres + KDC
brought up in k3d.

Two new docs:

  docs/plans/k3d-gke-mirror.md
    Architectural plan. Top-to-bottom rationale for what the k3d
    model should mirror, what we deliberately skip (supabase, kong,
    oauth2-proxy, Studio — all unnecessary for knoe-auth's surfaces),
    and the phasing. Captures the three scope decisions taken
    2026-05-02 with the user:
      - knoe-auth runs on the laptop, not as a pod (faster inner loop)
      - stack scope is CNPG + KDC only
      - single-replica CNPG, realm KNOE.LOCAL (distinct from KNOE.DEV)
    Out-of-scope for Phase 1 listed explicitly so reviewer
    expectations match scope: SPNEGO E2E, image-build-and-load,
    supabase, oauth2-proxy, OidcCodeService DB persistence.

  docs/plans/junie/k3d-knoe-auth-dev-loop.md
    Phase 1 brief. Concrete deliverables:
      - k8s/knoe/knoe-kdc-{deployment,service,configmap,pvc,init-job}.yaml
      - etc/init_knoe_auth.sh --mode k3d flag (or sibling script)
      - Makefile: k3d-knoe-{up,pf,down}
      - etc/krb5.local.conf (with udp_preference_limit=1 for
        kubectl port-forward UDP flakiness on macOS)
      - docs/local-dev-knoe-auth.md (engineer-facing)
      - scripts/k3d-knoe-smoke.sh
    Definition of done: a fresh-clone laptop can `make k3d-knoe-up`
    + `make k3d-knoe-pf` + `mvn -pl authority spring-boot:run` and
    hit /health, /.well-known/openid-configuration in <8 minutes.

Index updates:
  docs/plans/README.md           — k3d-gke-mirror.md row added
  docs/plans/junie/README.md     — split into Active / Shipped;
                                   k3d brief listed under Active;
                                   the 5 shipped-2026-05-02 briefs
                                   moved to Shipped with commit refs.
  docs/TODO.md In-progress       — k3d brief now top of "In progress";
                                   the previously in-flight "Phase 2
                                   pg_oauth in install/deploy.sh"
                                   item demoted to "paused" (resume
                                   after the local dev loop lands).

No code changes; all docs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 04:45:28 -07:00
chrisfu
232981c18c chore(cleanup): decommission garage on knoe-dev-0 app cluster (queue #4)
Mirroring the 2026-04-29 DB-cluster removal. Live state showed garage
in an unconfigured limbo (layout v0, no role assigned, 21d of "Ring
not yet ready" warnings); nothing outside garage's own state
referenced its S3 endpoint anymore.

Removed from knoe-system namespace on knoe-dev-0:
  - statefulset/garage      (1/1 pod, 21d uptime)
  - service/garage          (ClusterIP; no garage-s3-ilb on app cluster)
  - configmap/garage-config
  - secret/garage-secrets
  - pvc/data-garage-0       (29 Gi, garage-hdd)
  - pv/pvc-656e1936-...     (was Bound, reclaim=Retain)
  - storageclass/garage-hdd

Underlying GCE pd-standard disk (us-west3-c/pvc-656e1936-...) reclaimed
via `gcloud compute disks delete` since reclaim=Retain leaves the disk
behind after the PV is gone.

Source manifests intact:
  - k8s/knoe/garage-*.yaml                                   (k3d, min)
  - deploy/opentofu/k3s/manifests/knoe/garage-*.yaml         (k3s)
  - etc/init_garage_store.sh + knoe/core/ops/k8s_garage_store.py
  - k8s/knoe/garage-statefulset-gcp.yaml                     (GKE variant)

A future ./install.sh against GKE could re-deploy garage if that's
what the deploy mode wires; whether it SHOULD wire garage given GCS
now handles backups is a separate cleanup decision.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 03:58:47 -07:00
chrisfu
e52bbb8982 chore(cleanup): drop 5 stale resources; reframe queue #4 (garage)
Queue #5 Phase B orphan cleanup landed:

  App cluster (knoe-dev-0):
    - gitlab/gitlab-migrations-58a3e27-97-23-n72j8 (Failed/Evicted,
      11d stale)
    - pv/pvc-42e8545b-…  (10Gi, monitoring/storage-kps-grafana-0)
    - pv/pvc-86422179-…  (30Gi, monitoring/prometheus-…-prometheus-0)
    - pv/pvc-e2c29f7a-…  (5Gi,  monitoring/alertmanager-…-alertmanager-0)

  DB cluster (knoe-dev-cnpg-0):
    - kube-system/prole-supabase-dirprep Job (FailureTarget, 22d stale)

Post-state: 0 Released PVs on app cluster; gitlab namespace has only
the Completed migration pod; DB-cluster kube-system has no prole-*
artifacts. Live-cluster ops only; no source changes.

Queue #4 reframed: tried the planned `garage repair --yes blocks`,
but found garage in an unconfigured state — layout v0, no role
assigned, "Ring not yet ready" warnings continuous since whenever the
layout was lost (likely during a reconcile after the 2026-04-29 PVC
resize). Cross-cluster sweep shows nothing outside garage references
its S3 endpoint anymore (CNPG backups moved to GCS, no other workload
points at 10.180.15.239:3900). Garage on app-cluster is now a dead
pod holding an 80 GiB PVC. The original block-repair task is moot.
Reframed as a decommission-or-restore decision; details captured in
the ranked queue entry. Awaiting direction.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 03:50:51 -07:00
chrisfu
2ca2b8b234 docs(todo): note PodMonitor + DASHBOARD applied to live; node-label partial closure
Adds a 2026-05-02 Done entry covering the cluster-side work that paired
with c3fae73 (source) and 03bb731 (chart cleanups): PodMonitor migration
applied with verified zero-data-loss handoff; supabase chart upgraded
revision 1→3; helm field-manager conflict on Kong configmap resolved;
node label drift on knoe-dev-0 closed (3 nodes labelled
knoe.dev/node-role=general alongside legacy prole.org/node-role); DB
cluster nodes still pending same treatment.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 03:40:55 -07:00
chrisfu
c3fae73de3 fix(cnpg,kong): wire cnpg-backup-sa, migrate PodMonitor, drop DASHBOARD consumer
Three Junie briefs landed in one commit:

#07 — Wire cnpg-backup-sa into CNPG cluster spec (drift R8)
  deploy/gcp/gke/knoe-db.yaml: add spec.serviceAccountName: cnpg-backup-sa
  (requires CNPG v1.29+, which is the live operator version).
  etc/init_cnpg_gke.sh: operator install URL now uses CNPG_OPERATOR_VERSION
  variable (default 1.29.0); new §11 patches knoe-db and
  knoe-db-barman-cloud RoleBindings to add cnpg-backup-sa as a subject
  if not already present — matching the 2026-04-29 live stabilization.

#13 — Migrate off deprecated enablePodMonitor + podMonitorRelabelings
  Both deprecated fields removed from deploy/gcp/gke/knoe-db.yaml
  spec.monitoring. New deploy/gcp/gke/knoe-db-podmonitor.yaml carries
  the PodMonitor with the cluster relabeling rule (cnpg.io/cluster pod
  label → cluster label; required for all 85 CNPG Grafana panels).
  Apply alongside knoe-db.yaml on next cluster patch.

#15 — Remove dead DASHBOARD consumer + basicauth_credentials
  supabase/helm/knoe-supabase:
  - wrapper.sh: drop DASHBOARD_USERNAME / DASHBOARD_PASSWORD envsubst lines
  - config.yaml: drop DASHBOARD consumer + basicauth_credentials block
  - kong/deployment.yaml: drop both DASHBOARD env-var secret refs
  - values.yaml: rename secret.dashboard → secret.openai (apiKey only;
    username/password dropped — no enforcer since commit 25f1b2e)
  - secrets/dashboard.yaml + _helpers.tpl: renamed to openai /
    supabase.secret.openai
  - studio/deployment.yaml: reads from secret.openai.apiKey
  - ci/example.yaml: updated to secret.openai.apiKey
  helm template confirms knoe-supabase-openai secret referenced; no
  DASHBOARD output.

docs/TODO.md: queue items #7, #13, #15 + drift row R8 archived to Done.

Co-authored-by: Junie <junie@jetbrains.com>
2026-05-02 03:08:34 -07:00
chrisfu
5d17325c10 fix(scripts): patch_garage_cross_cluster — three defects from 2026-04-29 review
Defects A, B, C from docs/plans/junie/06-patch-garage-script-fixes.md:

  A. DB_CLUSTER default was knoe-cnpg-0 (stale rebrand artifact);
     corrected to knoe-dev-cnpg-0 to match conf/gke.cfg and the
     cluster table in CLAUDE.md. Header comment + overrides block
     also updated.

  B. Phase 1 deletion loop missed service/garage-s3-ilb (the
     LoadBalancer the ObjectStore endpoint historically pointed at).
     Now included alongside service/garage.

  C. Phase 2 was applying knoe-db-backup-gcs.yaml whole, including a
     legacy ScheduledBackup using method:barmanObjectStore (being
     removed in CNPG v1.30). Manifest split handled in prior commit
     (34a25dd); script's Phase 2 is now safe to re-run.

bash -n clean; CONFIRM=false dry-run prints knoe-dev-cnpg-0 correctly.
docs/TODO.md: queue items #2 and #6 + drift rows R5 and R9 archived to Done.

Closes queue item #6 in docs/TODO.md (drift R9).

Co-authored-by: Junie <junie@jetbrains.com>
2026-05-02 03:01:22 -07:00
chrisfu
00f0ebec07 Merge claude/crazy-bose-fec256 into main
Bringing the long-running session-feature branch back into main in one
deliberate sweep. The branch carried the cluster work that's been live for
weeks (cross-cluster CNPG metrics, Grafana w/ Google OAuth, supabase
oauth2-proxy, cluster recovery, pg.0.knoe.dev + per-engineer onboarding,
GCS-backed CNPG backups via Workload Identity, the env-contamination
guard, the Junie brief queue, the cnpg-grafana CSRF + memory-request
fixes from today), while main accumulated Junie's parallel knoe-auth
Phase 2 OIDC work (full provider surface: discovery, authorize, token,
userinfo, JWKS, RS256 signing, code exchange, session services).

Key decision: the two branches did COMPETING rebrands off the same
starting point (5ba9b63, 2026-04-27):

  - claude branch (commit b355855, earlier): org.prole.authority.* →
                                              dev.knoe.auth.*
                                              (artifact renamed to
                                              knoe-auth.jar)
  - main (commit 9daa94b, recent): org.prole.authority.* →
                                    dev.knoe.authority.*
                                    (kept "authority" artifact name)

dev.knoe.auth wins: cluster runs from this name, the Maven artifact is
already knoe-auth.jar, and the broader rename is the documented
namespace direction (per ~/.claude/projects/-Users-chrisfu-dev-knoe-db/
memory/MEMORY.md). All of main's recent Phase 2 OIDC content was ported
from authority/src/.../dev/knoe/authority/ into
authority/src/.../dev/knoe/auth/ with package declarations rewritten.

== File-level resolution summary ==

Textual conflicts (4):

  authority/pom.xml
    - Took our artifactId="auth"
    - Took our branch's removal of spring-security-kerberos-client
      (verified: Junie's Phase 2 OIDC code does not import it; the dep
      was already-dead config)

  docs/pipeline-phases.md
    - Took our branch's "Phase 1 not started" status. Main had a
      misplaced " Complete" with a knoe-auth-Phase-1 commit ref
      in the autobuild Phase 1 section — different domain.

  docs/plans/knoe-auth-round-1.md
    - Took our branch's dev.knoe.auth file table (vs main's
      dev.knoe.authority listing). Pure rename mismatch.

  supabase/helm/knoe-supabase/templates/kong/config.yaml
    - Took our branch's onboard route + plain dashboard wiring.
      Main had an oauth2proxy.enabled toggle that put oauth2-proxy as
      a Kong upstream — but the deployed architecture (commit 25f1b2e)
      has oauth2-proxy in FRONT of Kong, not behind. Main's wrapper
      reflected an architecture that was never deployed.
    - Took our branch's removal of basic-auth from dashboard route
      (queue #15 brief still tracks the matching values.yaml /
      kong/deployment.yaml cleanup).

Java tree reconciliation (44 file-pairs):

  20 dual-path source files + 2 dual-path tests
    Body-identical between main's authority/ and our branch's auth/
    after stripping package decls — main's commit 9daa94b was a pure
    rebrand. Took our branch's auth/ version for all 22.

  8 main-only source files (Phase 2 OIDC), ported into auth/:
    web/JwksController.java
    web/OidcAuthorizeController.java
    web/OidcDiscoveryController.java
    web/OidcTokenController.java
    web/OidcUserInfoController.java
    session/OidcCodeService.java
    session/OidcTokenService.java
    session/SessionService.java

  12 main-only test files, ported into auth/:
    HealthControllerTest.java
    enroll/EnrollValueTypesTest.java
    enroll/EnrollmentControllerTest.java
    enroll/TotpServiceTest.java
    kerberos/KadminClientTest.java
    kerberos/KerberosSpnegoResultTest.java
    web/LoginControllerTest.java
    admin/AdminControllerTest.java
    user/PrincipalNormalizerTest.java
    regression/IdentityRegressionTest.java
    session/OidcCodeServiceTest.java
    session/SessionServiceTest.java

  Port mechanics: read main:authority/...<file> via git show, then sed
  rewrite `package dev.knoe.authority` → `package dev.knoe.auth` and
  `import dev.knoe.authority` → `import dev.knoe.auth`. Body content
  unchanged.

  authority/src/main/java/dev/knoe/authority/ — DELETED (duplicate)
  authority/src/test/java/dev/knoe/authority/  — DELETED (duplicate)

== Verification ==

- grep -rln '<<<<<<<' across .java/.md/.yaml/.yml/.sh/.xml/.tpl: clean
- find authority/src -path '*/dev/knoe/authority*': empty (subtree gone)
- grep 'package dev.knoe.authority' across repo: clean
- bash -n install.sh deploy.sh etc/preflight_kubecontext.sh: clean
- git ls-files -u | wc -l: 0 unmerged paths
- helm lint supabase/helm/knoe-supabase: pre-existing failure on
  studioIngress.enabled undefined in values.yaml (introduced by Junie
  on main; unrelated to this merge — flagging as follow-up).

== Followups (carried into TODO ranked queue or noted here) ==

  - helm lint failure: studioIngress block in values.yaml is missing
    enable flag; templates/studio/{ingress,oauth2proxy-deployment,
    oauth2proxy-service}.yaml all reference studioIngress.enabled with
    no default. Pre-existing on main; not introduced by this merge.
  - The five Junie briefs filed on this branch are now reachable from
    main at docs/plans/junie/{02,06,07,13,15}-*.md. Junie can pick them
    up in any order.
  - knoe-auth Phase 2 OIDC source (now at dev.knoe.auth.*) is not yet
    deployed to the cluster. Deployment is its own task.
  - The branch claude/crazy-bose-fec256 stays in place (worktree at
    .claude/worktrees/crazy-bose-fec256 may have ongoing context for
    Claude Code sessions). Safe to delete once next session starts
    cleanly from main.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 16:39:10 -07:00
chrisfu
500c9b1317 fix(installer): env-contamination guard against shell-context / config mismatch
Filed in response to the 2026-04-28 14:00 UTC backup outage. An
`install.sh --mode k3d` run with the shell pointed at GKE silently
overwrote the GKE cluster's GCS-backed ObjectStore + ScheduledBackup
with k3d-mode defaults; Garage filled up and CNPG backups failed for
hours before the next manual check. The class of bug is "config says
target cluster A, shell context says target cluster B, installer
proceeds against B without warning."

New shared bash helper at etc/preflight_kubecontext.sh with two
functions:

  - verify_kubecontext_matches_config <cfg-path>
      Strict gate. Reads [Global] APP_CLUSTER_KUBECONTEXT from the
      config and exits 1 if `kubectl config current-context` differs.
      Skipped silently when the config has no baked APP_CLUSTER_KUBECONTEXT
      (e.g. fresh k3d.cfg) or when there's no live current-context.

  - print_kubecontext_notice
      Informational. Prints what's about to be inherited so the user
      can abort before the TUI launches if it looks wrong. Never fails.

Wiring:

  - deploy.sh sources the helper and calls the strict gate against
    ${PROLE_DEPLOY_CFG:-conf/gke.cfg} before invoking Python.
    Unattended path -> hard refusal on mismatch.

  - install.sh sources the helper and calls the informational notice
    (gated on not-`--min`) right after entering the local-checkout
    branch. The TUI is interactive, so the strict mode-aware gate is
    a follow-up once the welcome screen records a mode in
    state.inputs.

Bypass for deliberate cross-cluster maintenance:
    KNOE_SKIP_KUBECONTEXT_GUARD=true ./deploy.sh

End-to-end verified:
  - deploy.sh with current=cnpg-0, gke.cfg=app-0   -> exit 1, clear msg
  - deploy.sh with KNOE_SKIP_...=true              -> bypasses, prints
                                                     "skipping check"
  - install.sh --min                               -> notice skipped
  - install.sh (no flag) and install.sh --silent   -> notice printed

Doc updates:
  - CLAUDE.md §"Env-contamination warning" rewritten to describe the
    live guard (was a forward-looking TODO).
  - CLAUDE.md drift table row R4 removed; "Closed 2026-05-01" line added.
  - docs/TODO.md queue item #1 archived to Done; R4 dropped from the
    reality-vs-intent table. Queue numbering retained (no #1 placeholder)
    so the docs/plans/junie/<NN>-...md filenames still match.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 16:14:30 -07:00
chrisfu
f83ec8169c docs(plans): file Junie briefs for queue items #2, #6, #7, #13, #15
Five self-contained work briefs in a new docs/plans/junie/ subdirectory,
each tagged against a numbered item in docs/TODO.md so Junie can take
them independently in any order.

  02-k3s-prole-rename.md            -> queue #2  (drift R5)
  06-patch-garage-script-fixes.md   -> queue #6  (drift R9)
  07-init-cnpg-gke-sa-wiring.md     -> queue #7  (drift R8)
  13-podmonitor-manual-management.md-> queue #13
  15-remove-dead-dashboard-consumer.md-> queue #15

Each brief follows the same shape: Why -> What changes (concrete file
paths + line numbers + before/after) -> Verification -> Out of scope ->
Commit shape -> Definition of done. The intent is that Junie reads cold
(no shared chat history) and lands the change without escalating
questions.

Also adds:
  - docs/plans/junie/README.md describing the convention.
  - Row in docs/plans/README.md so newcomers find the subdirectory.
  - Brief reference + "Assigned to Junie" tag on each of the five queue
    items in docs/TODO.md.

Existing in-progress assignment to Junie (Phase 2 pg_oauth) is unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 16:13:20 -07:00
chrisfu
9daa94b860 refactor(authority): rebrand package org.prole → dev.knoe; raise test coverage to 50%
- Move all 31 Java files from org/prole/authority to dev/knoe/authority
- Update all package declarations, imports, pom.xml groupId, and docs
- Add 11 new test classes (PrincipalNormalizerTest, OidcCodeServiceTest, KadminClientTest, TotpServiceTest, HealthControllerTest, EnrollValueTypesTest, KerberosSpnegoResultTest, SessionServiceTest, EnrollmentControllerTest, AdminControllerTest, LoginControllerTest)
- Add JaCoCo 0.8.12 + maven-surefire-plugin with Java 25 / Byte Buddy compat flags
- Fix LoginController CSS format string bug (100% -> 100%%)
- Result: 96 tests, 0 failures; 50.2% line / 46.5% instruction / 37.1% branch coverage

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-30 22:42:53 -07:00
chrisfu
9d5827b522 feat(auth): knoe-auth Phase 2 — OIDC provider surface (discovery, authorize, token, userinfo, JWKS), RS256 signing, Kerberos/SPNEGO integration, stateless session model, PrincipalNormalizer, typed config, init script, architecture doc, regression tests
Co-authored-by: Junie <junie@jetbrains.com>
2026-04-30 22:14:03 -07:00
chrisfu
748de6beff test(infra): Phase A — onboarding TDD design doc + hard fixture
Co-authored-by: Junie <junie@jetbrains.com>
2026-04-30 16:07:39 -07:00
chrisfu
5641fd9320 docs: update Phase 1 status and commit summary in pipeline-phases.md
Co-authored-by: Junie <junie@jetbrains.com>
2026-04-28 12:23:02 -07:00
chrisfu
3728889e25 Phase 1: OIDC provider integration and GKE auth deployment
- Implement Google OIDC support in Authority module via GoogleOAuthService

- Update AuthProperties and application.yml with OIDC configuration

- Add oidc-setup.md documentation for GKE/Google Cloud setup

- Update etc/init_knoe_auth.sh to handle OIDC secrets and path-B configuration

- Configure knoe-auth-deployment.yaml and gke.cfg for production auth

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-28 12:22:45 -07:00
chrisfu
5ba9b63e34 docs(pipeline): Phase 0 commit summary complete; Phase 1 first task noted
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-27 14:47:27 -07:00
chrisfu
636a2795cb docs(plans): add platform architecture plans — deployment-modes, knoe-auth round 1
Adds docs/plans/ as the canonical engineering reference for completed and
in-flight initiatives. Written for jr/mid engineers who have not seen the
repo before — each plan starts with strategic context and links to existing
code before asking for changes.

  README.md            index, audience, and status conventions
  deployment-modes.md  four-mode installer (min/k3d/k3s/gke), welcome-screen
                       mode selector, min-mode fast-path. Status: shipped.
  knoe-auth-round-1.md Kerberos KNOE.DEV realm, invite-OTP enrollment,
                       Google corroboration, TOTP 2FA. Status: operational.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-27 14:47:27 -07:00
chrisfu
d3520d4837 docs: update Phase 0 commit summary and resumption checklist
Co-authored-by: Junie <junie@jetbrains.com>
2026-04-27 14:44:46 -07:00
chrisfu
22906ee656 Merge branch 'main' of git-ssh.knoe.dev:knoe.dev/knoe-db 2026-04-27 13:33:49 -07:00
chrisfu
553612bdff chore: purge dead credentials and harden .gitignore
- Delete tracked knoe-db/.prole_user_password (dead secret, rotated
  in ansible since the original commit)
- Remove postgres-password.txt and build/ copy from working tree
- Add explicit ignore rules for *_password, postgres-password*,
  .prole_user_password, *-secret variants

Add docs/plans/customer-deploy-resync.md with the upstream/downstream
migration plan and update CLAUDE.md to reflect prole's role as a
customer deploy of knoe-db.
2026-04-27 13:33:29 -07:00
chrisfu
b421f49af1 Migrate Ansible-based scripts and documentation to 1Password integration. Removed deprecated and redundant Ansible vault workflows, added 1Password CLI handling for secrets management, and updated tests to reflect the change. 2026-04-22 20:54:13 -07:00
chrisfu
55b6a6aff3 Complete rebranding from prole to knoe and fix macOS application identity. Bulk renamed 'prole' to 'knoe' across code, scripts, and manifests. Updated Makefile with 'knoe', 'build', and 'start' targets. Implemented macOS Application Bundle (.app) support for correct identity. Fixed macOS 'Python' process name to 'Knoe.DB Installer' via Objective-C bridge. Standardized application name to 'Knoe.DB Installer' across all interfaces.
Co-authored-by: Junie <junie@jetbrains.com>
2026-04-22 15:08:35 -07:00
chrisfu
f44ce32a27 Rebrand and consolidate authentication and scanning systems. Consolidate Kerberos authentication system into authority/src and rename ProleAuthApplication to KnoeAuthApplication. Remove deprecated prole/, prole-app/, prole-mssql-db/, and prole-tools-app/ directories. Migrate and rename prole-net/prole-agent to scan/network-agent. Update install.sh, Makefile, and documentation to use Knoe branding and new scan paths. Rebrand configuration properties and URLs to knoe.dev and svc.knoe.dev.
Co-authored-by: Junie <junie@jetbrains.com>
2026-04-22 09:28:14 -07:00
chrisfu
15df3ecc08 chore: purge ssl/, tfstate, ad-hoc artifacts; harden .gitignore
Follow-up to the ansible excision: remove other artifacts that should
never have been tracked, and extend .gitignore so they do not come back.

Removed:
- ssl/prole/*.key, *.crt, *.srl -- prole.org-era private keys, registry
  and DB CA material. References in knoe/core/env.py (auto-detect path
  at :1488) are best-effort inside try/except, so deletion is safe.
- ssl/prole.ssl.tar -- tarball of the same material.
- deploy/opentofu/k3s/terraform.tfstate -- 114KB committed OpenTofu
  state snapshot with live cluster UIDs. tfstate is never supposed to
  live in git.
- docs/deploy_gke.log, docs/deploy_gke_done.log,
  docs/compressed-history.log -- deploy-time log captures.
- end_time.txt, start_time.txt, start_time_final.txt -- ansible wall-
  clock timing breadcrumbs.
- network_description.txt, network_prompt.txt -- local scratch prompts.
- sidekiq_logs.txt -- transient log capture.
- test_resolve.sh -- ad-hoc debug script.

.gitignore additions:
- *.tfstate, *.tfstate.backup
- /ssl/
- .ai/
- /end_time.txt, /start_time*.txt, /sidekiq_logs.txt,
  /network_description.txt, /network_prompt.txt, /test_resolve.sh,
  /docs/deploy_*.log, /docs/compressed-history.log

Note: the purged SSL keys remain in git history until the deferred
history-scrub pass (task #14). The repo will be created as Private in
git.knoe.dev so this is acceptable for now; it MUST be scrubbed before
any broader visibility change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 19:01:03 -07:00
chrisfu
84105e1b0b docs: add Google Workspace OIDC setup guide for GitLab on GKE
- Documented step-by-step procedure for integrating GitLab with Google Workspace via OIDC (Path A).
- Outlined secret management patterns, GCP OAuth client setup, and deployment instructions.
- Included detailed remediation steps for secret leakage and Git history cleanup.
- Added verification and troubleshooting guides for pre- and post-deployment scenarios.

test: add helm monitoring tests for stale lock recovery and upgrade handling

- Introduced tests to validate Prometheus update flow with stale pending locks, including rollback and upgrade scenarios.
- Ensured robust handling for edge cases like missing deployed revisions and context mismatches.
2026-04-20 13:03:59 -07:00
chrisfu
b8a55888be checkpoint: update GKE deployment docs and infra changes
Summarize CLAUDE.md updates:

- Document both live GKE clusters as e2-standard-2 x3 in us-west3 and note stale app default in code.

- Call out DB cluster naming mismatch between conf (knoe-dev-cnpg-0) and code default (knoe-cnpg-0).

- Add SSD quota guidance: 300GB fully consumed by CNPG, so non-CNPG PVCs must use standard/pd-standard.

- Refresh reset script reference and quota section details to match current runtime state.

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-18 07:18:10 -07:00
chrisfu
e3c2e625f1 refactor(config): separate k3d k3s and gke config entrypoints
Rename env config files from conf/*/prole.cfg to conf/k3d.cfg, conf/k3s.cfg, and conf/gke.cfg. Update shell/Python loaders and etc/deploy scripts to resolve named configs cleanly while keeping legacy fallback behavior. Align k3s Ansible tasks, docs, and regression coverage with the new configuration layout.

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-11 22:20:45 -07:00
chrisfu
3312c39b1f feat: GCP/GKE CNPG hardening, Artifact Registry traffic light, and knoe-system namespace fixes
UI screens
- database.py: fix mode detection to use env_key priority (prod→k8s, service→k3s) so stale DEPLOYMENT_MODE never overrides the user's chosen environment
- database.py: Registry status reads ARTIFACT_REGISTRY_AVAILABLE persisted by cluster screen; uses SERVICE_NAMESPACE for Artifact Registry repo name
- cluster.py: add Artifact Registry traffic light (amber→green/red) to prod section; _check_artifact_registry_async persists ARTIFACT_REGISTRY_AVAILABLE into Global cfg
- cluster.py: re-trigger Artifact Registry check after GKE cluster selection so the light re-evaluates once region is available from KUBECONTEXT
- cluster_nodes.py: fix TclError on Python 3.14 — pady=(2,0) tuple → pady=2 scalar
- __init__.py: seed knoe-system namespace when saved value is "default", not only when empty
- services.py: replace hardcoded "Prole DB" log string with dynamic cnpg_cluster name

Core ops
- cloudnative_pg.py: replace one-shot Barman plugin retry with 6-attempt loop; first cert-manager/x509 failure triggers rollout restart + 30 s CA propagation wait; subsequent failures back off up to 60 s per attempt
- cloudnative_pg.py: TLS CA CN now uses cluster_name instead of hardcoded "Prole CNPG CA"
- registry.py, garage_store.py: refactored into per-mode modules (k3d/k3s/k8s registry and garage store, shared _garage_common)

Deploy / config
- deploy/gcp/gke/knoe-db.yaml: GKE-specific CNPG cluster manifest (rw/ro/r on separate nodes with premium-rwo storage)
- etc/init_common_services.sh, modes/k8s/knoe-db/.version: updated for current deploy
- kong-deployment.yaml: updated manifest

Tests
- test_cluster_nodes_render_smoke.py: add pack/grid, winfo_children, winfo_reqheight, update_idletasks, grid_slaves to dummy widgets; monkeypatch tk.Label so CNPG placement render completes without a real Tkinter root

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-04 19:36:08 -07:00
chrisfu
d02aadae89 Add node management workflow and installer config persistence updates
- add tools/manage-node.sh for k3s node add/remove automation with inventory reconciliation

- include kubectl drain flag compatibility and --ssh-host targeting for uninstall workflows

- persist installer cfg/env values and update deploy/database/cfg UI handling

- refresh related inventory/network artifacts and expand installer/database test coverage

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-29 11:22:33 -07:00
chrisfu
a069989315 Rename prole-db to knoe-db, add knoe-auth as cluster-internal KDC
Itemized changes:

1. knoe-auth: New cluster-internal KDC and SSO gateway service
   - Created etc/init_knoe_auth.sh based on init_kdc.sh with knoe-auth naming
   - Namespace defaults to SERVICE_NAMESPACE (knoe-system)
   - ConfigMap: knoe-auth-kdc-config, Secret: knoe-auth-secrets
   - Legacy cleanup removes old auth/dog/authority deployments

2. Orchestration: knoe-auth initializes before CloudNativePG
   - Updated prole.sh to insert init_knoe_auth.sh as step 2 (before CNPG)
   - Renumbered all subsequent initialization steps

3. Kong routing: Updated init_kong.sh to route to knoe-auth in SERVICE_NAMESPACE

4. Comment/reference updates for knoe-auth
   - Updated init_common_services.sh, init_service_layer.sh, init_kerberos.sh

5. prole-db renamed to knoe-db across the entire codebase
   - Renamed prole-db/ directory to knoe-db/
   - Renamed all prole-db Kubernetes manifests (deploy/opentofu, k8s/)
   - Renamed scripts: docker-root-knoe-db.sh, docker-run-knoe-db.sh, test-cnpg-knoe-db.sh
   - Renamed etc/init_prole-db-reset.sh to etc/init_knoe-db-reset.sh
   - Renamed etc/prole-db-passwwd.sh to etc/knoe-db-passwwd.sh
   - Renamed mock_val counterparts accordingly
   - Renamed tests/etc/test_init_prole-db-reset.sh to test_init_knoe-db-reset.sh
   - Renamed docs/prole-db-documentation-mcp-architecture.md to knoe-db variant
   - Renamed modes/k3d/prole-db/ to modes/k3d/knoe-db/
   - Renamed prole-db.iml to knoe-db.iml

6. Configuration updates
   - Updated conf/dev, conf/prod, conf/test, conf/service prole.cfg files
   - Updated conf/port-mapping.cfg
   - Updated etc/prole_cfg.sh and mock_val/prole_cfg.sh
   - Updated service/prole.cfg

7. Kubernetes manifests and deploy configuration
   - Updated deploy/opentofu/k3s ArgoCD application YAMLs
   - Updated kong-configmap.yaml and kustomization.yaml
   - Updated k3s/kong-config.yml and prole-resources.yaml
   - Updated prole-mssql-db deployment YAMLs
   - Updated supabase helm render and deploy scripts

8. Infrastructure and GCP Terraform
   - Updated deploy/gcp/terraform: folders, groups, IAM, service-projects

9. Python/installer code updates
   - Updated knoe/core: actions, build_context, controller, env, milestones
   - Updated knoe/milestone.py
   - Updated knoe/ui/screens: cfg, database, database_options, deploy, docker,
     navigation, security, services, validate
   - Updated knoe.spec, status.py

10. Shell script updates
    - Updated etc/: build_db, init_cloudnative_pg, init_cnpg_backup,
      init_db_manager, init_forgejo, init_gitlab, init_monitoring, init_openbao,
      init_port_forwards, init_postgrest, init_supabase_ports, status
    - Updated mock_val/ counterparts for all above scripts
    - Updated prole-net/init-prole-dns.sh
    - Updated bin/prole-kpf.sh, gitea/deploy.sh, supabase/deploy.sh

11. Test updates
    - Updated tests/etc/: test_init_cloudnative_pg*, test_init_cnpg_backup*,
      test_init_kdc*, test_init_kerberos*, test_init_kong*, test_prole_cfg*
    - Updated tests/installer/: test_actions_helpers, test_cfg_save_kubecontext,
      test_controller, test_core_classes, test_milestones, test_milestones_extended,
      test_namespace_propagation
    - Updated tests/: test_database_options, test_navigation,
      test_render_supabase_hostname, test_docker_build_fix,
      test_all_prole_home_fixes, silent_install_test, final_test

12. Documentation updates
    - Updated docs/: DOCKER-BUILD-FIX, PROLE-CFG-SECRETS, PROLE-HOME-DIRECTORY,
      build-system, patent
    - Updated scan/network_description.txt
    - Updated pom.xml

13. Miscellaneous script updates
    - Updated root-level: _adopt_replica_pvcs, _fix_replica_merlin, _import_pi,
      _patch_cluster, _prebind_pvcs, _rebind_d002, _rebind_d002b, test_resolve
    - Updated scripts/generate_spec.py

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-22 22:16:21 -07:00
chrisfu
0262bd4be2 Refactor to support mode-scoped DB build contexts and versioning
- Introduce mode-scoped directories under `~/.prole` for build contexts, runtime data, and versioning.
- Update DB image version generation and Dockerfile creation to respect deployment modes (e.g., k3d, k3s, k8s).
- Adjust UI, core actions, and scripts to avoid cross-mode conflicts and ensure correct version isolation.
- Add tests for mode-aware behaviors and update existing tests for new paths and markers.
- Bump PostgreSQL and image versions to `18` and `138`, respectively.
- Update Kubernetes manifests to align with new image and version configuration.
2026-03-22 05:47:50 -07:00
chrisfu
1353882260 Update PROLE_HOME resolution and enhance iSCSI PV handling
- Implement centralized `resolve_prole_home` utility for consistent environment-based `PROLE_HOME` resolution across modules
- Replace hardcoded home paths with `resolve_prole_home`
- Refactor PV management to support iSCSI mounts and node placement from Ansible manifests
- Improve Kubernetes manifest handling to dynamically apply namespaces per document
- Adjust `knoe-db` build context path and related tests
- Add utilities for detecting and applying Ansible-defined node labels and PVs
2026-03-22 02:43:52 -07:00
chrisfu
4ee2b259c9 Checkpoint: rename installer to knoe + harden db build context
- Add build-context helper to copy Docker context safely (ignore runtime data, keep symlinks)

- Update UI and core actions to use ~/.prole/build and shared copy helper

- Add/adjust tests and scripts; introduce knoe ops helpers and update manifests

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-22 01:45:21 -07:00
chrisfu
ecacf01266 Remove legacy secrets, configs, and scripts; introduce prole-auth service and Grafana SSO proxy configuration. 2026-03-20 13:00:36 -07:00
chrisfu
de7bb8d0ff Refactor shell library structure; relocate helpers under etc/lib/shell/ and update references 2026-03-16 21:44:58 -07:00
chrisfu
5dcd3b9581 ansible: treat /var/lib/rancher as local K3s state
Update inventory/roles to keep K3s state on local storage and prevent iSCSI from managing /var/lib/rancher.

Also add a single-host k3s install playbook, a systemd override template, and docs describing the installer ↔ Ansible boundary and slow-storage knobs.

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-15 23:20:55 -07:00
chrisfu
b905cbe584 Bootstrap env-based prole.cfg entrypoint (safe legacy migration)
- Add centralized config resolver/activator with per-environment layering and stable symlink entrypoint\n- Bootstrap missing env dirs and preserve non-symlink legacy prole.cfg by seeding into inferred env\n- Wire installer UI/backend + shell helpers to shared resolution path for explicit, safe env switching\n- Harden k3d: registry network/DNS wiring and ArgoCD repo-server hostPath permission init\n- Add docs + regression tests for env switching, namespace stability, k3d registry, and ArgoCD rollout

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-13 12:23:15 -07:00
chrisfu
909e92109e Refactor shell layout; add shellspec + authority
- Introduce lib/shell helpers and keep etc/* scripts thin via compatibility shims

- Move Kerberos validation to scripts/validation/check_kerberos.sh and update callers

- Add deterministic shellspec unit tests under tests/shellspec/ and wire Maven to run them

- Add minimal Spring Boot authority module with startup + /health endpoint and Maven wiring

- Document the new layout in docs/layout.md

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-13 09:19:56 -07:00
chrisfu
bd54755218 remove owner/group/mode enforcement for external mountpoint in Ansible tasks
- Update documentation to note `/external` may already be mounted before Ansible runs.
- Avoid redundant ownership and permissions enforcement in `mariadb_primary` role.
2026-03-13 02:15:05 -07:00
chrisfu
17cb827be5 test(dashboard): add unit tests for systemd unit and script validation
- Ensure dashboard service doesn't create an ordering cycle with `multi-user.target`.
- Verify service safely attaches to a virtual terminal (VT) and passes VT to Xorg.
- Ensure no `su` usage by default in the service.
- Validate script behavior for passing VT and `-keeptty` options to Xorg.

chore(dashboard): update systemd unit to avoid VT conflicts and improve logging

- Remove `After` and `Wants` directives for `multi-user.target` to prevent conflicts.
- Add `TTYPath`, `StandardInput`, and related options for VT attachment.
- Ensure `ExecStopPost` restores getty service for the dashboard VT.

fix(dashboard): improve Xorg resolution and clean up script logic

- Dynamically determine Xorg executable location in `dashboard.sh`.
- Pass VT, display, and `-keeptty` to Xorg to improve script reliability.

fix(mariadb): correct exFAT mount options for external storage

- Avoid `chown` failures by setting root ownership with group access for `mysql`.
- Update external mount options to `uid=0,gid=mysql,umask=0002`.

cleanup(mariadb): remove unnecessary owner/group permissions from migration marker task
2026-03-13 01:49:57 -07:00
chrisfu
3c2a2d23a1 feat(mariadb): add temporary external storage support for Merlin
- Enable mounting borrowed USB disk (exFAT) persistently at `/external`.
- Migrate MariaDB datadir to `/external/mariadb` with bind mount at `/srv/mariadb/mariadb`.
- Add migration marker and rollback support to prevent accidental remigration.
- Include Ansible tasks for external storage setup and migration validation.
2026-03-13 00:44:08 -07:00
chrisfu
f2c9012cce Refactor installation and initialization logic, and expand test coverage
- install.py: Major update including configuration variable expansion, improved k3s/k3d handling, and enhanced installation logic.

- etc/ scripts: Significant refactoring of initialization scripts (Kerberos, Port Forwards, Garage Store, etc.).

- Port Forwards: Transitioned from XML to port-mappings.conf for managing kubectl port-forwards.

- Status Reporting: Improved status checking for common services.

- Infrastructure: Updated Ansible inventory and rsyslog role configurations.

- Tests: Added a comprehensive suite of tests for 'etc' initialization scripts in prole/tests/etc/.

- Documentation: Added prole-db-documentation-mcp-architecture.md.

- General: Updated Dockerfiles and various helper scripts.
2026-02-13 21:36:39 -08:00
chrisfu
bcc8f23a0d Enhance secret management and k8s infrastructure
- Secret Management: Integrated AESGCM for temporary secret handling in install.py and enhanced OpenBao (Vault) support with namespace injection and additional secret paths (Grafana, Kerberos, TDE).
- Infrastructure & K8s:
    - Added Barman Object Store backup configuration (S3) to prole-db.yaml.
    - Updated Prometheus deployment with PVC and persistent configuration.
    - Updated k3s cluster/registry creation scripts.
    - Added etc/build-a-bao.sh for OpenBao setup.
- MSSQL Integration: Updated docker scripts and k8s deployments for Prole MSSQL database.
- Documentation: Added docs/PROLE-CFG-SECRETS.md explaining the new secret handling.
- General: Refined initialization scripts (init_authority.sh, init_openbao.sh, etc.) and updated the ncurses installer.
2026-02-03 22:39:50 -08:00
chrisfu
fff18fdbe4 Update Prole-DB and improve Supabase integration
- Bumped Prole-DB image version to 17.7-053 in scripts, Dockerfile, and manifests.
- Replaced `prole-scan` with `prole-agent` throughout scripts and tests.
- Refined Kubernetes setup for Supabase to use namespace 'supabase'.
- Introduced conversion of Supabase Docker Compose to Kubernetes manifests with `kompose`.
- Added support for Kerberos toggle via environment variables in `init_kerberos.sh`.
- Improved error handling and logging in scripts for better maintainability.
2026-02-01 23:56:25 -08:00
chrisfu
dda938ee95 Integrate namespace setup and management into scripts, Dockerfile, and update installer UI. 2026-01-28 23:02:50 -08:00
chrisfu
cbfe930b78 feat: add ncurses interface, build system, and embedded resources
Major feature additions and infrastructure improvements for the Prole
Database Installer, enabling command-line operation and packaged binary
distribution.

## Ncurses Terminal Interface

- Add installer/ncurses_ui.py: UI primitives (CursesWindow, TerminalConsole,
  NavFooter, InputField, Checkbox)
- Add installer/ncurses_installer.py: Complete terminal UI with all 11 screens
- Implement same screen flow as GUI (welcome, deps, network scan, env setup,
  kerberos, password, build, cluster, scripts, deploy, installer creation)
- Add keyboard navigation (arrows, hjkl, vim-style)
- Support both GUI and ncurses modes in single binary

## Automatic Display Detection

- Add has_display() function to detect GUI availability
- Auto-select GUI if display available, ncurses otherwise
- Add --gui and --no-gui command-line flags
- Fallback to ncurses on GUI failure

## Build System and Packaging

- Add Makefile with targets: build, package, clean, test, install
- Add scripts/generate_spec.py: PyInstaller spec generator
- Add installer.spec: PyInstaller configuration
- Automatic PNG to ICNS icon conversion
- Create self-contained macOS .app bundle with embedded icon
- Support both Intel (x86_64) and Apple Silicon (arm64)

## Embedded Resources

- Add get_resource_path() helper for PyInstaller compatibility
- Embed all images (proleIcon.png, proleLogo.png, proleLogoSepia.png)
- Embed prole-net/prole-scan binary (6.8 MB universal binary)
- Embed prole-app/dist/Prole Tools.app (12 MB app bundle)
- Embed prole-db/ Docker build context

## Writable Directory Fixes

- Create ~/.prole/build/prole-db/ for Docker builds (fixes read-only _MEIPASS)
- Create ~/.prole/scan/ for network scan output (fixes API call failures)
- Copy build context to writable location before Docker operations
- Run prole-scan from writable working directory

## Documentation

- docs/build-system.md: Complete build system guide
- docs/ncurses-installer.md: Ncurses interface documentation
- docs/RELEASE-NOTES.md: Feature overview and release notes
- docs/IMAGE-RESOURCES.md: Image resource management
- docs/EMBEDDED-RESOURCES.md: Binary and app bundle embedding
- docs/DOCKER-BUILD-FIX.md: Docker build hang solution
- docs/PROLE-HOME-DIRECTORY.md: ~/.prole directory structure
- BUILD.md: Quick build reference

## Key Changes

install.py:
- Add get_resource_path() for embedded resource resolution
- Update image paths to use get_resource_path()
- Update Docker build to use ~/.prole/build/prole-db/
- Update network scan to use ~/.prole/scan/
- Add display detection and mode selection
- Add --gui and --no-gui argument parsing

## Testing

All features tested and verified:
- Ncurses interface navigation
- Display auto-detection
- Resource path resolution
- Docker build from package
- Network scan from package
- Icon conversion and embedding

Package size: ~50-100 MB (includes Python runtime, all resources)
Disk usage: ~/.prole/ uses ~2-6 MB

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-01-19 23:10:37 -08:00
chrisfu
92fe6fba95 docs and history. UE Dockerfile 2025-12-02 18:42:00 -08:00