docs(branches): upstream knoe-db/20260523 review — branch doc, index, and Junie integration brief

- docs/branches/README.md: index of upstream review branches - docs/branches/upstream-knoe-db-20260523.md: full analysis of 403-file diff (no shared history; conflict risk by area; recommended actions) - docs/plans/junie/upstream-knoe-db-20260523-integration.md: 6-task Junie brief ordered by conflict risk (pg-knoe-auth → k3d manifests → briefs → authority OIDC fixes → knoe/core installer → etc/ init scripts) - docs/plans/junie/README.md: integration brief added to active table
2026-09-23 10:13:58 +00:00 · 2026-05-23 21:44:13 -07:00 · 2026-05-23 21:44:13 -07:00 · 0e822ea976
commit 0e822ea976
parent 5babe04516
4 changed files with 397 additions and 0 deletions
--- a/docs/branches/README.md
+++ b/docs/branches/README.md
@ -0,0 +1,20 @@
+# Branch reviews
+
+This directory contains review documents for incoming upstream branches before they are merged or cherry-picked into `main`.
+
+Each file corresponds to a review branch created by `scripts/sync_upstream_knoe_db.sh`.
+
+## Index
+
+| Branch | Upstream | Date | Status | Review doc |
+|---|---|---|---|---|
+| `upstream/knoe-db/20260523` | `git@git.knoe.dev:knoe-dev/knoe-db.git` (`main`) | 2026-05-23 | 🔍 Under review | [`upstream-knoe-db-20260523.md`](upstream-knoe-db-20260523.md) |
+
+## How to use these docs
+
+1. Run `scripts/sync_upstream_knoe_db.sh` to fetch upstream and create a dated review branch.
+2. A review doc is created here summarising the incoming changes, conflict risk, and recommended actions.
+3. Work through the review doc section by section — cherry-pick or copy files as appropriate.
+4. When the review is complete, update the Status column above to ✅ Done and note the commit(s) that landed the changes.
+
+See [`docs/upstream-knoe-db-sync.md`](../upstream-knoe-db-sync.md) for the full sync procedure.
--- a/docs/branches/upstream-knoe-db-20260523.md
+++ b/docs/branches/upstream-knoe-db-20260523.md
@ -0,0 +1,186 @@
+# Upstream review: `upstream/knoe-db/20260523`
+
+**Branch:** `upstream/knoe-db/20260523`  
+**Upstream:** `git@git.knoe.dev:knoe-dev/knoe-db.git` (`main`)  
+**Fetched:** 2026-05-23  
+**Commits ahead of our `main`:** 684  
+**Files differing from our `main`:** 403  
+
+> ⚠️ **No shared history.** The upstream repo and this fork have diverged from separate roots — there is no common merge base. A standard `git merge` will refuse unless `--allow-unrelated-histories` is passed. **Do not attempt a bulk merge.** Cherry-pick or copy individual files/directories that are valuable.
+
+---
+
+## Summary of incoming changes
+
+### 1. `authority/` — knoe-auth Java OIDC service (high value)
+
+The upstream has a substantially evolved `authority/` (knoe-auth) Java app:
+
+| Area | What changed |
+|---|---|
+| `LoginController`, `OidcAuthorizeController`, `OidcTokenController`, `OidcUserInfoController` | Robust `next`-URL handling, absolute issuer URL, `@RequestMapping` removed from `LoginController`, URL-encoding fixes |
+| `OidcCodeService`, `OidcTokenService` | Real access JWT, Bearer userinfo, groups claim, `allowedClientIds` in token endpoint, `email_verified=true` |
+| `GoogleOAuthService`, `InviteService` | Google client_id/secret whitespace strip, authorization URL encoding to prevent `URI.create()` crash |
+| `AuthProperties`, `application.yml` | `emailDomain` config replaces hardcoded `@prole.org` |
+| `V1__oidc_authorization_codes.sql` | OIDC authorization codes migration |
+| Tests | `AdminControllerTest`, `OidcCodeServiceTest`, `IdentityRegressionTest`, `KadminClientTest`, `EnrollValueTypesTest` |
+
+**Action:** Review and cherry-pick the OIDC fixes into our `authority/` — these are bug fixes we likely want.
+
+---
+
+### 2. `knoe/core/` — installer core (high value, high conflict risk)
+
+403 files differ; the installer core is heavily changed upstream:
+
+| Module | What changed |
+|---|---|
+| `actions.py` | Large changes — upstream has pruned `pg_knoe_auth` references, added k3d paths |
+| `milestones.py` | k3d milestone additions, kerberos disable for k3d |
+| `env.py` | Likely diverged from our `gke_` kubecontext detection work |
+| `ops/` (16 files) | `garage_store`, `registry`, `monitoring`, `openbao`, `opentofu`, `storage`, `gke_clusters` |
+| `build_context.py`, `controller.py`, `topology.py`, `policy.py` | Various upstream changes |
+
+**Conflict risk: HIGH.** We have local commits to `env.py` (`gke_` detection), `knoe_conf.py`, and `cfg.py` that are not in upstream. A file-level merge will require careful diff review.
+
+---
+
+### 3. `pg-knoe-auth/` — PostgreSQL C extension (upstream-only, new)
+
+The upstream has a `pg-knoe-auth/` PostgreSQL C extension for JWT-based auth at the DB level:
+
+| Path | Description |
+|---|---|
+| `pg-knoe-auth/src/pg_knoe_auth.c` | C extension source |
+| `pg-knoe-auth/src/Makefile` | Build |
+| `pg-knoe-auth/test/integration/` | Integration tests with mock JWKS server |
+| `pg-knoe-auth/test/unit/` | Unit tests |
+| `pg-knoe-auth/README.md` | Documentation |
+
+**Note:** Upstream commit `270aba3` (`prune: remove pg_knoe_auth references from installer and UI`) and `f40cb32` (merge k3d branch) suggest upstream itself is in the process of removing `pg_knoe_auth` from the installer. The extension code remains but is being decoupled.
+
+**Action:** Evaluate whether we want this extension. If yes, copy `pg-knoe-auth/` wholesale (it doesn't conflict with anything we have).
+
+---
+
+### 4. `deploy/gcp/gke/` and `deploy/opentofu/k3s/manifests/knoe/` — K8s manifests
+
+New and updated manifests upstream:
+
+| File | Description |
+|---|---|
+| `deploy/gcp/gke/knoe-auth-deployment.yaml` | knoe-auth GKE deployment |
+| `deploy/gcp/gke/svc-knoe-auth-ingress.yaml` | knoe-auth ingress |
+| `deploy/gcp/gke/oauth2-proxy-*.yaml` | oauth2-proxy deployment + secrets examples |
+| `deploy/gcp/gke/knoe-onboard.yaml` | Onboarding manifest |
+| `deploy/opentofu/k3s/manifests/knoe/knoe-auth-service.yaml` | knoe-auth k3s service |
+| `deploy/opentofu/k3s/manifests/knoe/grafana-proxy-*.yaml` | Grafana proxy configmap/deployment/service |
+| `deploy/opentofu/k3s/manifests/knoe/gitea-spnego-proxy.yaml` | Gitea SPNEGO proxy |
+| `deploy/opentofu/k3s/manifests/knoe/traefik-helmchartconfig.yaml` | Traefik config |
+
+**Action:** Review against our GKE manifests in `k8s/`. Some of these may be additive.
+
+---
+
+### 5. `k8s/knoe/` — k3d cluster manifests (new)
+
+| File | Description |
+|---|---|
+| `k8s/knoe/knoe-db-k3d.yaml` | CNPG cluster manifest for k3d (new) |
+| `k8s/knoe/garage-statefulset-k3d.yaml` | Garage statefulset for k3d (new) |
+| `k8s/knoe/knoe-db.yaml` | Updated production CNPG manifest |
+
+**Action:** `knoe-db-k3d.yaml` and `garage-statefulset-k3d.yaml` are likely additive — copy in.
+
+---
+
+### 6. `etc/` — init scripts (moderate conflict risk)
+
+17 `etc/` files differ. We have our own versions of many of these in `mock_val/` (committed in `3f96f66`). Key upstream additions:
+
+| File | Notes |
+|---|---|
+| `etc/init_knoe_auth.sh` | knoe-auth init — compare with our `mock_val/init_knoe_auth.sh` |
+| `etc/init_knoe_users.sh` | User init — compare with our `mock_val/init_knoe_users.sh` |
+| `etc/init_cnpg_gke.sh` | GKE CNPG init — compare with our version |
+| `etc/init_user_schemas_gke.sh` | New GKE user schema init |
+| `etc/init_grafana_oauth_prole.sh` | Grafana OAuth (prole variant) |
+| `etc/init_oauth2_proxy_prole.sh` | oauth2-proxy (prole variant) |
+| `etc/sync_cnpg_grafana_dashboard.py` | Dashboard sync script |
+| `etc/knoe_cfg.sh` | Config helper |
+| `etc/build_db.sh`, `etc/fetch_prole_secrets.sh` | Build/secrets helpers |
+
+---
+
+### 7. `docs/plans/junie/` — upstream Junie briefs (informational)
+
+The upstream has 20 Junie brief files that differ from ours. Notable upstream-only briefs:
+
+| File | Subject |
+|---|---|
+| `knoeledge-naming-standards.md` | Naming conventions doc |
+| `knoeledge-tenant-onboarding.md` | Tenant onboarding playbook |
+| `per-user-supabase-workspace.md` | Per-user Supabase workspace design |
+| `knoe-auth-device-flow.md` | Device flow for knoe-auth |
+| `pg-knoe-auth-build-fix-and-installer-integration.md` | pg-knoe-auth build/integration |
+| `deploy-pipeline-restore.md` | Deploy pipeline restore |
+| `db-secret-username-drift-fix.md` | DB secret username drift fix |
+| `init-user-schemas-heredoc-fix.md` | Heredoc fix for user schema init |
+| `auth-0-ingress-and-rename-regression.md` | Auth ingress regression |
+| `black-sweep-and-pin.md` | Black formatter sweep |
+
+**Action:** Read these briefs — several describe work we may want to implement (device flow, per-user Supabase, tenant onboarding).
+
+---
+
+### 8. `conf/` — config files (conflict)
+
+`conf/k3d.cfg`, `conf/k3s.cfg`, `conf/gke.cfg`, `conf/knoe.cfg`, `conf/service/knoe.cfg`, `conf/service/prod.cfg` all differ. We just cleaned our cfg files of MagicMock contamination. **Do not overwrite our cfg files with upstream versions** — they contain our environment-specific values.
+
+---
+
+### 9. `infrastructure/` — Ansible roles and playbooks
+
+| File | Notes |
+|---|---|
+| `infrastructure/roles/samba_dns/tasks/` (3 files) | DNS role updates |
+| `infrastructure/roles/samba_reverse_dns/tasks/` (2 files) | Reverse DNS role |
+| `infrastructure/roles/acme/tasks/main.yml` | ACME cert role |
+| `infrastructure/playbooks/kerberos_trust_setup.yml` | Kerberos trust playbook |
+| `infrastructure/playbooks/workstation_kerberos.yml` | Workstation Kerberos |
+| `infrastructure/inventory/` | hosts.ini, group_vars updates |
+
+**Action:** Review Ansible role changes — likely additive/safe to merge.
+
+---
+
+## Conflict summary
+
+| Area | Risk | Recommendation |
+|---|---|---|
+| `knoe/core/` (actions, milestones, env, ops) | 🔴 HIGH | Manual file-by-file diff; protect our `env.py` gke_ detection |
+| `conf/*.cfg` | 🔴 HIGH | Do NOT overwrite — keep our values |
+| `authority/` OIDC fixes | 🟡 MEDIUM | Cherry-pick individual commits (listed above) |
+| `etc/` init scripts | 🟡 MEDIUM | Compare with `mock_val/` equivalents before replacing |
+| `pg-knoe-auth/` | 🟢 LOW | Additive — copy wholesale if desired |
+| `k8s/knoe/*-k3d.yaml` | 🟢 LOW | Additive — copy in |
+| `deploy/gcp/gke/` manifests | 🟢 LOW | Review and add selectively |
+| `docs/plans/junie/` briefs | 🟢 LOW | Read and adopt relevant ones |
+| `infrastructure/` Ansible | 🟢 LOW | Likely additive |
+
+---
+
+## Next steps
+
+```bash
+# Inspect a specific file from upstream without switching branches
+git show upstream/knoe-db/20260523:authority/src/main/java/dev/knoe/auth/web/LoginController.java
+
+# Copy a whole directory from upstream into working tree
+git checkout upstream/knoe-db/20260523 -- pg-knoe-auth/
+
+# Diff a specific file
+git diff main..upstream/knoe-db/20260523 -- knoe/core/env.py
+```
+
+See also: [`docs/upstream-knoe-db-sync.md`](../upstream-knoe-db-sync.md) for the full sync procedure.
--- a/docs/plans/junie/README.md
+++ b/docs/plans/junie/README.md
@ -51,6 +51,7 @@ in TODO" note). When Junie lands a brief:
 | File | Tracked at | Subject |
 |---|---|---|
 | [`k3d-knoe-auth-pod-deploy.md`](k3d-knoe-auth-pod-deploy.md) | TODO §"In progress"; parent [`../k3d-gke-mirror.md`](../k3d-gke-mirror.md) §6 Phase 3 | Phase 3 of k3d-mirror-of-GKE: build the knoe-auth image, `k3d image import`, run as a pod inside the cluster. Pre-merge smoke loop with `make k3d-knoe-{deploy,redeploy,undeploy}`. |
+| [`upstream-knoe-db-20260523-integration.md`](upstream-knoe-db-20260523-integration.md) | [`docs/branches/upstream-knoe-db-20260523.md`](../../docs/branches/upstream-knoe-db-20260523.md) | Structured cherry-pick/copy of valuable upstream changes from `upstream/knoe-db/20260523` (684 commits, 403 files). 6 tasks ordered by conflict risk: pg-knoe-auth import, k3d manifests, Junie briefs, authority OIDC fixes, knoe/core installer diff, etc/ init scripts. |

 ### Pending (working-tree only)

--- a/docs/plans/junie/upstream-knoe-db-20260523-integration.md
+++ b/docs/plans/junie/upstream-knoe-db-20260523-integration.md
@ -0,0 +1,190 @@
+# Junie brief: integrate upstream knoe-db/20260523
+
+**Tracked at:** `docs/branches/upstream-knoe-db-20260523.md`  
+**Branch:** `upstream/knoe-db/20260523`  
+**Priority:** Medium — work through sections in order, lowest-risk first.
+
+---
+
+## Why
+
+On 2026-05-23 we fetched the upstream `git@git.knoe.dev:knoe-dev/knoe-db.git` `main` branch into a review branch (`upstream/knoe-db/20260523`). The upstream has 684 commits and 403 differing files relative to our `main`. There is **no shared history** — a bulk merge is not safe. This brief tracks the structured cherry-pick/copy work to land the valuable upstream changes without clobbering our local customisations.
+
+Full analysis: [`docs/branches/upstream-knoe-db-20260523.md`](../../branches/upstream-knoe-db-20260523.md)
+
+---
+
+## What changes (ordered by risk, lowest first)
+
+### Task 1 — Copy `pg-knoe-auth/` wholesale (additive, no conflict)
+
+The upstream has a PostgreSQL C extension (`pg-knoe-auth/`) for JWT-based DB-level auth. We have no local version of this directory.
+
+```bash
+git checkout upstream/knoe-db/20260523 -- pg-knoe-auth/
+git add pg-knoe-auth/
+git commit -m "feat(pg-knoe-auth): import upstream PostgreSQL JWT auth extension"
+```
+
+**Note:** Upstream is decoupling this from the installer (`270aba3`, `f40cb32`) — the extension code is stable but installer wiring is being removed. Import the code; do not wire it into the installer yet.
+
+**Definition of done:**
+- [ ] `pg-knoe-auth/` directory present in `main` with `src/`, `test/`, `README.md`
+- [ ] `make test` still passes (Python tests unaffected)
+
+---
+
+### Task 2 — Copy k3d cluster manifests (additive, no conflict)
+
+Two new k3d-specific manifests exist upstream that we don't have:
+
+```bash
+git checkout upstream/knoe-db/20260523 -- k8s/knoe/knoe-db-k3d.yaml
+git checkout upstream/knoe-db/20260523 -- k8s/knoe/garage-statefulset-k3d.yaml
+```
+
+Also review the updated `k8s/knoe/knoe-db.yaml`:
+```bash
+git diff main..upstream/knoe-db/20260523 -- k8s/knoe/knoe-db.yaml
+```
+
+Apply any production CNPG manifest improvements that don't conflict with our GKE-specific settings (storage class, SA name, backup config).
+
+**Definition of done:**
+- [ ] `k8s/knoe/knoe-db-k3d.yaml` present
+- [ ] `k8s/knoe/garage-statefulset-k3d.yaml` present
+- [ ] `k8s/knoe/knoe-db.yaml` reviewed; any safe upstream improvements applied
+
+---
+
+### Task 3 — Review and adopt upstream Junie briefs
+
+Read the following upstream-only briefs and decide which to adopt:
+
+```bash
+git show upstream/knoe-db/20260523:docs/plans/junie/knoeledge-naming-standards.md
+git show upstream/knoe-db/20260523:docs/plans/junie/knoeledge-tenant-onboarding.md
+git show upstream/knoe-db/20260523:docs/plans/junie/per-user-supabase-workspace.md
+git show upstream/knoe-db/20260523:docs/plans/junie/knoe-auth-device-flow.md
+git show upstream/knoe-db/20260523:docs/plans/junie/deploy-pipeline-restore.md
+git show upstream/knoe-db/20260523:docs/plans/junie/db-secret-username-drift-fix.md
+git show upstream/knoe-db/20260523:docs/plans/junie/init-user-schemas-heredoc-fix.md
+```
+
+For each brief that is relevant to our environment: copy it into `docs/plans/junie/` and add it to the active table in `docs/plans/junie/README.md`.
+
+**Definition of done:**
+- [ ] Each upstream brief read and evaluated
+- [ ] Relevant briefs copied to `docs/plans/junie/`
+- [ ] `docs/plans/junie/README.md` active table updated
+
+---
+
+### Task 4 — Cherry-pick `authority/` OIDC fixes (medium risk)
+
+The upstream has a series of bug fixes to the knoe-auth Java OIDC service. These are the upstream commits to cherry-pick (inspect each before applying):
+
+| Commit | Subject |
+|---|---|
+| `b588412` | fix(knoe-auth): replace hardcoded @prole.org with configured emailDomain |
+| `1ad7049` | fix(knoe-auth): remove @RequestMapping("/auth") from LoginController |
+| `5d43721` | fix(knoe-auth): use absolute issuer URL for login redirect in /authorize |
+| `202c16b` | fix(knoe-auth): robust next-URL handling — skip null nonce, encode scope spaces |
+| `7147758` | fix(knoe-auth): KNOE_AUTH_ENABLED=true + robust safeNext() URL parsing |
+| `ce78730` | fix(auth): URL-encode next param in login page hrefs |
+| `878254e` | fix(auth): strip leading/trailing whitespace from Google client_id/secret |
+| `ee3f16f` | fix(auth): encode Google authorization URL to prevent URI.create() crash |
+| `e5522d4` | fix(oidc): real access JWT + Bearer userinfo + groups claim |
+| `b8a4028` | fix(oidc): add email_verified=true to id_token claims |
+| `ce78730` | fix(oidc): accept allowedClientIds in token endpoint + fix audience |
+
+Since there is no shared history, cherry-pick will not work directly. Instead, copy the changed files:
+
+```bash
+git diff main..upstream/knoe-db/20260523 -- authority/ > /tmp/authority.patch
+# Review the patch, then apply selectively
+```
+
+Or copy individual files:
+```bash
+git checkout upstream/knoe-db/20260523 -- authority/src/main/java/dev/knoe/auth/web/LoginController.java
+# etc.
+```
+
+**Definition of done:**
+- [ ] All OIDC fix commits reviewed
+- [ ] Safe fixes applied to `authority/`
+- [ ] `authority/` Java tests pass (`mvn test` in `authority/`)
+
+---
+
+### Task 5 — Diff `knoe/core/` installer modules (high risk — do not bulk-copy)
+
+The upstream has significant changes to the installer core. **Do not overwrite our files.** Instead, diff each module and apply upstream improvements that don't conflict with our local changes:
+
+```bash
+# Key files to diff:
+git diff main..upstream/knoe-db/20260523 -- knoe/core/env.py
+git diff main..upstream/knoe-db/20260523 -- knoe/core/actions.py
+git diff main..upstream/knoe-db/20260523 -- knoe/core/milestones.py
+git diff main..upstream/knoe-db/20260523 -- knoe/core/ops/monitoring.py
+git diff main..upstream/knoe-db/20260523 -- knoe/core/ops/garage_store.py
+```
+
+**Protect these local changes:**
+- `knoe/core/env.py` — our `gke_` kubecontext prefix detection (`da0fd2c`)
+- `knoe/knoe_conf.py` — our `normalize_environment()` patch
+- `knoe/ui/screens/cfg.py` — our `_validate_cfg_values` fix (`ef20c8a`)
+
+For each file: read the upstream diff, identify upstream improvements (k3d support, pg_knoe_auth pruning, etc.), and apply them manually without overwriting our local additions.
+
+**Definition of done:**
+- [ ] `knoe/core/env.py` reviewed; upstream improvements applied without losing `gke_` detection
+- [ ] `knoe/core/actions.py` reviewed; k3d additions and pg_knoe_auth pruning evaluated
+- [ ] `knoe/core/milestones.py` reviewed; k3d milestones evaluated
+- [ ] `make test` passes after all changes
+
+---
+
+### Task 6 — Review `etc/` init scripts (medium risk)
+
+Compare upstream `etc/` scripts with our `mock_val/` equivalents:
+
+```bash
+git diff main..upstream/knoe-db/20260523 -- etc/init_knoe_auth.sh
+git diff main..upstream/knoe-db/20260523 -- etc/init_knoe_users.sh
+git diff main..upstream/knoe-db/20260523 -- etc/init_cnpg_gke.sh
+git diff main..upstream/knoe-db/20260523 -- etc/init_user_schemas_gke.sh
+```
+
+`etc/init_user_schemas_gke.sh` is upstream-only (new) — copy it in if it applies to our GKE setup.
+
+**Definition of done:**
+- [ ] Each `etc/` script diffed against our `mock_val/` equivalent
+- [ ] `etc/init_user_schemas_gke.sh` evaluated and copied if applicable
+- [ ] No regressions in existing init scripts
+
+---
+
+## Out of scope
+
+- `conf/*.cfg` — **never overwrite** with upstream versions; our env-specific values must be preserved.
+- `infrastructure/inventory/` — contains our host-specific values; review manually before touching.
+- Bulk `git merge --allow-unrelated-histories` — explicitly forbidden; too much conflict surface.
+
+---
+
+## Commit shape
+
+Each task above should be a separate commit:
+
+```
+feat(pg-knoe-auth): import upstream PostgreSQL JWT auth extension
+feat(k8s): add k3d cluster manifests from upstream (knoe-db-k3d, garage-k3d)
+docs(plans): adopt upstream Junie briefs — device-flow, tenant-onboarding, etc.
+fix(authority): apply upstream OIDC fixes from knoe-db/20260523
+feat(core): apply upstream k3d installer improvements from knoe-db/20260523
+feat(etc): apply upstream init script improvements from knoe-db/20260523
+```
+
+When all tasks are done, update `docs/branches/README.md` status to ✅ Done.