diff --git a/docs/branches/README.md b/docs/branches/README.md new file mode 100644 index 0000000..a5618a4 --- /dev/null +++ b/docs/branches/README.md @@ -0,0 +1,20 @@ +# Branch reviews + +This directory contains review documents for incoming upstream branches before they are merged or cherry-picked into `main`. + +Each file corresponds to a review branch created by `scripts/sync_upstream_knoe_db.sh`. + +## Index + +| Branch | Upstream | Date | Status | Review doc | +|---|---|---|---|---| +| `upstream/knoe-db/20260523` | `git@git.knoe.dev:knoe-dev/knoe-db.git` (`main`) | 2026-05-23 | ๐Ÿ” Under review | [`upstream-knoe-db-20260523.md`](upstream-knoe-db-20260523.md) | + +## How to use these docs + +1. Run `scripts/sync_upstream_knoe_db.sh` to fetch upstream and create a dated review branch. +2. A review doc is created here summarising the incoming changes, conflict risk, and recommended actions. +3. Work through the review doc section by section โ€” cherry-pick or copy files as appropriate. +4. When the review is complete, update the Status column above to โœ… Done and note the commit(s) that landed the changes. + +See [`docs/upstream-knoe-db-sync.md`](../upstream-knoe-db-sync.md) for the full sync procedure. diff --git a/docs/branches/upstream-knoe-db-20260523.md b/docs/branches/upstream-knoe-db-20260523.md new file mode 100644 index 0000000..ed83f30 --- /dev/null +++ b/docs/branches/upstream-knoe-db-20260523.md @@ -0,0 +1,186 @@ +# Upstream review: `upstream/knoe-db/20260523` + +**Branch:** `upstream/knoe-db/20260523` +**Upstream:** `git@git.knoe.dev:knoe-dev/knoe-db.git` (`main`) +**Fetched:** 2026-05-23 +**Commits ahead of our `main`:** 684 +**Files differing from our `main`:** 403 + +> โš ๏ธ **No shared history.** The upstream repo and this fork have diverged from separate roots โ€” there is no common merge base. A standard `git merge` will refuse unless `--allow-unrelated-histories` is passed. **Do not attempt a bulk merge.** Cherry-pick or copy individual files/directories that are valuable. + +--- + +## Summary of incoming changes + +### 1. `authority/` โ€” knoe-auth Java OIDC service (high value) + +The upstream has a substantially evolved `authority/` (knoe-auth) Java app: + +| Area | What changed | +|---|---| +| `LoginController`, `OidcAuthorizeController`, `OidcTokenController`, `OidcUserInfoController` | Robust `next`-URL handling, absolute issuer URL, `@RequestMapping` removed from `LoginController`, URL-encoding fixes | +| `OidcCodeService`, `OidcTokenService` | Real access JWT, Bearer userinfo, groups claim, `allowedClientIds` in token endpoint, `email_verified=true` | +| `GoogleOAuthService`, `InviteService` | Google client_id/secret whitespace strip, authorization URL encoding to prevent `URI.create()` crash | +| `AuthProperties`, `application.yml` | `emailDomain` config replaces hardcoded `@prole.org` | +| `V1__oidc_authorization_codes.sql` | OIDC authorization codes migration | +| Tests | `AdminControllerTest`, `OidcCodeServiceTest`, `IdentityRegressionTest`, `KadminClientTest`, `EnrollValueTypesTest` | + +**Action:** Review and cherry-pick the OIDC fixes into our `authority/` โ€” these are bug fixes we likely want. + +--- + +### 2. `knoe/core/` โ€” installer core (high value, high conflict risk) + +403 files differ; the installer core is heavily changed upstream: + +| Module | What changed | +|---|---| +| `actions.py` | Large changes โ€” upstream has pruned `pg_knoe_auth` references, added k3d paths | +| `milestones.py` | k3d milestone additions, kerberos disable for k3d | +| `env.py` | Likely diverged from our `gke_` kubecontext detection work | +| `ops/` (16 files) | `garage_store`, `registry`, `monitoring`, `openbao`, `opentofu`, `storage`, `gke_clusters` | +| `build_context.py`, `controller.py`, `topology.py`, `policy.py` | Various upstream changes | + +**Conflict risk: HIGH.** We have local commits to `env.py` (`gke_` detection), `knoe_conf.py`, and `cfg.py` that are not in upstream. A file-level merge will require careful diff review. + +--- + +### 3. `pg-knoe-auth/` โ€” PostgreSQL C extension (upstream-only, new) + +The upstream has a `pg-knoe-auth/` PostgreSQL C extension for JWT-based auth at the DB level: + +| Path | Description | +|---|---| +| `pg-knoe-auth/src/pg_knoe_auth.c` | C extension source | +| `pg-knoe-auth/src/Makefile` | Build | +| `pg-knoe-auth/test/integration/` | Integration tests with mock JWKS server | +| `pg-knoe-auth/test/unit/` | Unit tests | +| `pg-knoe-auth/README.md` | Documentation | + +**Note:** Upstream commit `270aba3` (`prune: remove pg_knoe_auth references from installer and UI`) and `f40cb32` (merge k3d branch) suggest upstream itself is in the process of removing `pg_knoe_auth` from the installer. The extension code remains but is being decoupled. + +**Action:** Evaluate whether we want this extension. If yes, copy `pg-knoe-auth/` wholesale (it doesn't conflict with anything we have). + +--- + +### 4. `deploy/gcp/gke/` and `deploy/opentofu/k3s/manifests/knoe/` โ€” K8s manifests + +New and updated manifests upstream: + +| File | Description | +|---|---| +| `deploy/gcp/gke/knoe-auth-deployment.yaml` | knoe-auth GKE deployment | +| `deploy/gcp/gke/svc-knoe-auth-ingress.yaml` | knoe-auth ingress | +| `deploy/gcp/gke/oauth2-proxy-*.yaml` | oauth2-proxy deployment + secrets examples | +| `deploy/gcp/gke/knoe-onboard.yaml` | Onboarding manifest | +| `deploy/opentofu/k3s/manifests/knoe/knoe-auth-service.yaml` | knoe-auth k3s service | +| `deploy/opentofu/k3s/manifests/knoe/grafana-proxy-*.yaml` | Grafana proxy configmap/deployment/service | +| `deploy/opentofu/k3s/manifests/knoe/gitea-spnego-proxy.yaml` | Gitea SPNEGO proxy | +| `deploy/opentofu/k3s/manifests/knoe/traefik-helmchartconfig.yaml` | Traefik config | + +**Action:** Review against our GKE manifests in `k8s/`. Some of these may be additive. + +--- + +### 5. `k8s/knoe/` โ€” k3d cluster manifests (new) + +| File | Description | +|---|---| +| `k8s/knoe/knoe-db-k3d.yaml` | CNPG cluster manifest for k3d (new) | +| `k8s/knoe/garage-statefulset-k3d.yaml` | Garage statefulset for k3d (new) | +| `k8s/knoe/knoe-db.yaml` | Updated production CNPG manifest | + +**Action:** `knoe-db-k3d.yaml` and `garage-statefulset-k3d.yaml` are likely additive โ€” copy in. + +--- + +### 6. `etc/` โ€” init scripts (moderate conflict risk) + +17 `etc/` files differ. We have our own versions of many of these in `mock_val/` (committed in `3f96f66`). Key upstream additions: + +| File | Notes | +|---|---| +| `etc/init_knoe_auth.sh` | knoe-auth init โ€” compare with our `mock_val/init_knoe_auth.sh` | +| `etc/init_knoe_users.sh` | User init โ€” compare with our `mock_val/init_knoe_users.sh` | +| `etc/init_cnpg_gke.sh` | GKE CNPG init โ€” compare with our version | +| `etc/init_user_schemas_gke.sh` | New GKE user schema init | +| `etc/init_grafana_oauth_prole.sh` | Grafana OAuth (prole variant) | +| `etc/init_oauth2_proxy_prole.sh` | oauth2-proxy (prole variant) | +| `etc/sync_cnpg_grafana_dashboard.py` | Dashboard sync script | +| `etc/knoe_cfg.sh` | Config helper | +| `etc/build_db.sh`, `etc/fetch_prole_secrets.sh` | Build/secrets helpers | + +--- + +### 7. `docs/plans/junie/` โ€” upstream Junie briefs (informational) + +The upstream has 20 Junie brief files that differ from ours. Notable upstream-only briefs: + +| File | Subject | +|---|---| +| `knoeledge-naming-standards.md` | Naming conventions doc | +| `knoeledge-tenant-onboarding.md` | Tenant onboarding playbook | +| `per-user-supabase-workspace.md` | Per-user Supabase workspace design | +| `knoe-auth-device-flow.md` | Device flow for knoe-auth | +| `pg-knoe-auth-build-fix-and-installer-integration.md` | pg-knoe-auth build/integration | +| `deploy-pipeline-restore.md` | Deploy pipeline restore | +| `db-secret-username-drift-fix.md` | DB secret username drift fix | +| `init-user-schemas-heredoc-fix.md` | Heredoc fix for user schema init | +| `auth-0-ingress-and-rename-regression.md` | Auth ingress regression | +| `black-sweep-and-pin.md` | Black formatter sweep | + +**Action:** Read these briefs โ€” several describe work we may want to implement (device flow, per-user Supabase, tenant onboarding). + +--- + +### 8. `conf/` โ€” config files (conflict) + +`conf/k3d.cfg`, `conf/k3s.cfg`, `conf/gke.cfg`, `conf/knoe.cfg`, `conf/service/knoe.cfg`, `conf/service/prod.cfg` all differ. We just cleaned our cfg files of MagicMock contamination. **Do not overwrite our cfg files with upstream versions** โ€” they contain our environment-specific values. + +--- + +### 9. `infrastructure/` โ€” Ansible roles and playbooks + +| File | Notes | +|---|---| +| `infrastructure/roles/samba_dns/tasks/` (3 files) | DNS role updates | +| `infrastructure/roles/samba_reverse_dns/tasks/` (2 files) | Reverse DNS role | +| `infrastructure/roles/acme/tasks/main.yml` | ACME cert role | +| `infrastructure/playbooks/kerberos_trust_setup.yml` | Kerberos trust playbook | +| `infrastructure/playbooks/workstation_kerberos.yml` | Workstation Kerberos | +| `infrastructure/inventory/` | hosts.ini, group_vars updates | + +**Action:** Review Ansible role changes โ€” likely additive/safe to merge. + +--- + +## Conflict summary + +| Area | Risk | Recommendation | +|---|---|---| +| `knoe/core/` (actions, milestones, env, ops) | ๐Ÿ”ด HIGH | Manual file-by-file diff; protect our `env.py` gke_ detection | +| `conf/*.cfg` | ๐Ÿ”ด HIGH | Do NOT overwrite โ€” keep our values | +| `authority/` OIDC fixes | ๐ŸŸก MEDIUM | Cherry-pick individual commits (listed above) | +| `etc/` init scripts | ๐ŸŸก MEDIUM | Compare with `mock_val/` equivalents before replacing | +| `pg-knoe-auth/` | ๐ŸŸข LOW | Additive โ€” copy wholesale if desired | +| `k8s/knoe/*-k3d.yaml` | ๐ŸŸข LOW | Additive โ€” copy in | +| `deploy/gcp/gke/` manifests | ๐ŸŸข LOW | Review and add selectively | +| `docs/plans/junie/` briefs | ๐ŸŸข LOW | Read and adopt relevant ones | +| `infrastructure/` Ansible | ๐ŸŸข LOW | Likely additive | + +--- + +## Next steps + +```bash +# Inspect a specific file from upstream without switching branches +git show upstream/knoe-db/20260523:authority/src/main/java/dev/knoe/auth/web/LoginController.java + +# Copy a whole directory from upstream into working tree +git checkout upstream/knoe-db/20260523 -- pg-knoe-auth/ + +# Diff a specific file +git diff main..upstream/knoe-db/20260523 -- knoe/core/env.py +``` + +See also: [`docs/upstream-knoe-db-sync.md`](../upstream-knoe-db-sync.md) for the full sync procedure. diff --git a/docs/plans/junie/README.md b/docs/plans/junie/README.md index 84e8145..2a59d1b 100644 --- a/docs/plans/junie/README.md +++ b/docs/plans/junie/README.md @@ -51,6 +51,7 @@ in TODO" note). When Junie lands a brief: | File | Tracked at | Subject | |---|---|---| | [`k3d-knoe-auth-pod-deploy.md`](k3d-knoe-auth-pod-deploy.md) | TODO ยง"In progress"; parent [`../k3d-gke-mirror.md`](../k3d-gke-mirror.md) ยง6 Phase 3 | Phase 3 of k3d-mirror-of-GKE: build the knoe-auth image, `k3d image import`, run as a pod inside the cluster. Pre-merge smoke loop with `make k3d-knoe-{deploy,redeploy,undeploy}`. | +| [`upstream-knoe-db-20260523-integration.md`](upstream-knoe-db-20260523-integration.md) | [`docs/branches/upstream-knoe-db-20260523.md`](../../docs/branches/upstream-knoe-db-20260523.md) | Structured cherry-pick/copy of valuable upstream changes from `upstream/knoe-db/20260523` (684 commits, 403 files). 6 tasks ordered by conflict risk: pg-knoe-auth import, k3d manifests, Junie briefs, authority OIDC fixes, knoe/core installer diff, etc/ init scripts. | ### Pending (working-tree only) diff --git a/docs/plans/junie/upstream-knoe-db-20260523-integration.md b/docs/plans/junie/upstream-knoe-db-20260523-integration.md new file mode 100644 index 0000000..54e325e --- /dev/null +++ b/docs/plans/junie/upstream-knoe-db-20260523-integration.md @@ -0,0 +1,190 @@ +# Junie brief: integrate upstream knoe-db/20260523 + +**Tracked at:** `docs/branches/upstream-knoe-db-20260523.md` +**Branch:** `upstream/knoe-db/20260523` +**Priority:** Medium โ€” work through sections in order, lowest-risk first. + +--- + +## Why + +On 2026-05-23 we fetched the upstream `git@git.knoe.dev:knoe-dev/knoe-db.git` `main` branch into a review branch (`upstream/knoe-db/20260523`). The upstream has 684 commits and 403 differing files relative to our `main`. There is **no shared history** โ€” a bulk merge is not safe. This brief tracks the structured cherry-pick/copy work to land the valuable upstream changes without clobbering our local customisations. + +Full analysis: [`docs/branches/upstream-knoe-db-20260523.md`](../../branches/upstream-knoe-db-20260523.md) + +--- + +## What changes (ordered by risk, lowest first) + +### Task 1 โ€” Copy `pg-knoe-auth/` wholesale (additive, no conflict) + +The upstream has a PostgreSQL C extension (`pg-knoe-auth/`) for JWT-based DB-level auth. We have no local version of this directory. + +```bash +git checkout upstream/knoe-db/20260523 -- pg-knoe-auth/ +git add pg-knoe-auth/ +git commit -m "feat(pg-knoe-auth): import upstream PostgreSQL JWT auth extension" +``` + +**Note:** Upstream is decoupling this from the installer (`270aba3`, `f40cb32`) โ€” the extension code is stable but installer wiring is being removed. Import the code; do not wire it into the installer yet. + +**Definition of done:** +- [ ] `pg-knoe-auth/` directory present in `main` with `src/`, `test/`, `README.md` +- [ ] `make test` still passes (Python tests unaffected) + +--- + +### Task 2 โ€” Copy k3d cluster manifests (additive, no conflict) + +Two new k3d-specific manifests exist upstream that we don't have: + +```bash +git checkout upstream/knoe-db/20260523 -- k8s/knoe/knoe-db-k3d.yaml +git checkout upstream/knoe-db/20260523 -- k8s/knoe/garage-statefulset-k3d.yaml +``` + +Also review the updated `k8s/knoe/knoe-db.yaml`: +```bash +git diff main..upstream/knoe-db/20260523 -- k8s/knoe/knoe-db.yaml +``` + +Apply any production CNPG manifest improvements that don't conflict with our GKE-specific settings (storage class, SA name, backup config). + +**Definition of done:** +- [ ] `k8s/knoe/knoe-db-k3d.yaml` present +- [ ] `k8s/knoe/garage-statefulset-k3d.yaml` present +- [ ] `k8s/knoe/knoe-db.yaml` reviewed; any safe upstream improvements applied + +--- + +### Task 3 โ€” Review and adopt upstream Junie briefs + +Read the following upstream-only briefs and decide which to adopt: + +```bash +git show upstream/knoe-db/20260523:docs/plans/junie/knoeledge-naming-standards.md +git show upstream/knoe-db/20260523:docs/plans/junie/knoeledge-tenant-onboarding.md +git show upstream/knoe-db/20260523:docs/plans/junie/per-user-supabase-workspace.md +git show upstream/knoe-db/20260523:docs/plans/junie/knoe-auth-device-flow.md +git show upstream/knoe-db/20260523:docs/plans/junie/deploy-pipeline-restore.md +git show upstream/knoe-db/20260523:docs/plans/junie/db-secret-username-drift-fix.md +git show upstream/knoe-db/20260523:docs/plans/junie/init-user-schemas-heredoc-fix.md +``` + +For each brief that is relevant to our environment: copy it into `docs/plans/junie/` and add it to the active table in `docs/plans/junie/README.md`. + +**Definition of done:** +- [ ] Each upstream brief read and evaluated +- [ ] Relevant briefs copied to `docs/plans/junie/` +- [ ] `docs/plans/junie/README.md` active table updated + +--- + +### Task 4 โ€” Cherry-pick `authority/` OIDC fixes (medium risk) + +The upstream has a series of bug fixes to the knoe-auth Java OIDC service. These are the upstream commits to cherry-pick (inspect each before applying): + +| Commit | Subject | +|---|---| +| `b588412` | fix(knoe-auth): replace hardcoded @prole.org with configured emailDomain | +| `1ad7049` | fix(knoe-auth): remove @RequestMapping("/auth") from LoginController | +| `5d43721` | fix(knoe-auth): use absolute issuer URL for login redirect in /authorize | +| `202c16b` | fix(knoe-auth): robust next-URL handling โ€” skip null nonce, encode scope spaces | +| `7147758` | fix(knoe-auth): KNOE_AUTH_ENABLED=true + robust safeNext() URL parsing | +| `ce78730` | fix(auth): URL-encode next param in login page hrefs | +| `878254e` | fix(auth): strip leading/trailing whitespace from Google client_id/secret | +| `ee3f16f` | fix(auth): encode Google authorization URL to prevent URI.create() crash | +| `e5522d4` | fix(oidc): real access JWT + Bearer userinfo + groups claim | +| `b8a4028` | fix(oidc): add email_verified=true to id_token claims | +| `ce78730` | fix(oidc): accept allowedClientIds in token endpoint + fix audience | + +Since there is no shared history, cherry-pick will not work directly. Instead, copy the changed files: + +```bash +git diff main..upstream/knoe-db/20260523 -- authority/ > /tmp/authority.patch +# Review the patch, then apply selectively +``` + +Or copy individual files: +```bash +git checkout upstream/knoe-db/20260523 -- authority/src/main/java/dev/knoe/auth/web/LoginController.java +# etc. +``` + +**Definition of done:** +- [ ] All OIDC fix commits reviewed +- [ ] Safe fixes applied to `authority/` +- [ ] `authority/` Java tests pass (`mvn test` in `authority/`) + +--- + +### Task 5 โ€” Diff `knoe/core/` installer modules (high risk โ€” do not bulk-copy) + +The upstream has significant changes to the installer core. **Do not overwrite our files.** Instead, diff each module and apply upstream improvements that don't conflict with our local changes: + +```bash +# Key files to diff: +git diff main..upstream/knoe-db/20260523 -- knoe/core/env.py +git diff main..upstream/knoe-db/20260523 -- knoe/core/actions.py +git diff main..upstream/knoe-db/20260523 -- knoe/core/milestones.py +git diff main..upstream/knoe-db/20260523 -- knoe/core/ops/monitoring.py +git diff main..upstream/knoe-db/20260523 -- knoe/core/ops/garage_store.py +``` + +**Protect these local changes:** +- `knoe/core/env.py` โ€” our `gke_` kubecontext prefix detection (`da0fd2c`) +- `knoe/knoe_conf.py` โ€” our `normalize_environment()` patch +- `knoe/ui/screens/cfg.py` โ€” our `_validate_cfg_values` fix (`ef20c8a`) + +For each file: read the upstream diff, identify upstream improvements (k3d support, pg_knoe_auth pruning, etc.), and apply them manually without overwriting our local additions. + +**Definition of done:** +- [ ] `knoe/core/env.py` reviewed; upstream improvements applied without losing `gke_` detection +- [ ] `knoe/core/actions.py` reviewed; k3d additions and pg_knoe_auth pruning evaluated +- [ ] `knoe/core/milestones.py` reviewed; k3d milestones evaluated +- [ ] `make test` passes after all changes + +--- + +### Task 6 โ€” Review `etc/` init scripts (medium risk) + +Compare upstream `etc/` scripts with our `mock_val/` equivalents: + +```bash +git diff main..upstream/knoe-db/20260523 -- etc/init_knoe_auth.sh +git diff main..upstream/knoe-db/20260523 -- etc/init_knoe_users.sh +git diff main..upstream/knoe-db/20260523 -- etc/init_cnpg_gke.sh +git diff main..upstream/knoe-db/20260523 -- etc/init_user_schemas_gke.sh +``` + +`etc/init_user_schemas_gke.sh` is upstream-only (new) โ€” copy it in if it applies to our GKE setup. + +**Definition of done:** +- [ ] Each `etc/` script diffed against our `mock_val/` equivalent +- [ ] `etc/init_user_schemas_gke.sh` evaluated and copied if applicable +- [ ] No regressions in existing init scripts + +--- + +## Out of scope + +- `conf/*.cfg` โ€” **never overwrite** with upstream versions; our env-specific values must be preserved. +- `infrastructure/inventory/` โ€” contains our host-specific values; review manually before touching. +- Bulk `git merge --allow-unrelated-histories` โ€” explicitly forbidden; too much conflict surface. + +--- + +## Commit shape + +Each task above should be a separate commit: + +``` +feat(pg-knoe-auth): import upstream PostgreSQL JWT auth extension +feat(k8s): add k3d cluster manifests from upstream (knoe-db-k3d, garage-k3d) +docs(plans): adopt upstream Junie briefs โ€” device-flow, tenant-onboarding, etc. +fix(authority): apply upstream OIDC fixes from knoe-db/20260523 +feat(core): apply upstream k3d installer improvements from knoe-db/20260523 +feat(etc): apply upstream init script improvements from knoe-db/20260523 +``` + +When all tasks are done, update `docs/branches/README.md` status to โœ… Done.