prole/docs/pipeline-phases.md

# Knoe.DB Autobuild & Test Pipeline — Phase Reference

> **How to use this document**
> Each phase has a **Status** line, a **Commit Summary** section (filled in after the phase lands), and a **Resumption Checklist** so any developer can pick up where we left off without needing context from a previous session. Update the Status and Commit Summary before starting the next phase.

---

## Context

The project has **four deployment modes** that must each produce a stable, reproducible build:

| Mode | What it is | CI tool |
|---|---|---|
| `min` | Single `knoe-db` container via containerd, no Kubernetes | Gitea Actions (bare macOS runner) |
| `k3d` | k3s-in-Docker local cluster — CNPG + optional Supabase / ArgoCD / Gitea / GitLab | Gitea Actions (Docker-enabled runner) |
| `k3s` | Homelab multi-node k3s on physical hardware | Deferred (evaluate after GitLab) |
| `gke` | Dual GKE clusters on Google Cloud (production) | GitLab CI |

**Primary goal:** auto-build on every push to `main` so we have confidence that builds are stable before we touch anything else.

**Secondary goal:** once we have stable builds, identify unused code with `vulture` + coverage diff, then prune toward an *instructable* codebase — a project a junior developer can read and fully understand.

---

## Phase 0 — Foundation

**Status:** ✅ Complete  
**Branch / PRs:** landed directly on `main`; pushed to `origin` (`git-ssh.knoe.dev:knoe.dev/knoe-db`)

### What this phase does

Fixes the broken test infrastructure so IntelliJ can discover tests and `make test` actually measures the right code. No new tests are written here — just scaffolding that every subsequent phase depends on.

### Deliverables

| File | Change |
|---|---|
| `pyproject.toml` | Single source of truth for pytest config + coverage config. Replaces the broken `.coveragerc` (which pointed to `installer/` — a directory that no longer exists) |
| `requirements-test.txt` | Explicit test dependencies (pytest, coverage, vulture, etc.) separated from runtime deps |
| `.idea/runConfigurations/pytest_all.xml` | IntelliJ: run all tests with coverage |
| `.idea/runConfigurations/pytest_unit.xml` | IntelliJ: run unit-only tests (fast, no external deps) |
| `.idea/runConfigurations/pytest_min.xml` | IntelliJ: run tests tagged `@pytest.mark.min` |
| `.idea/runConfigurations/pytest_k3d.xml` | IntelliJ: run tests tagged `@pytest.mark.k3d` |
| `.idea/runConfigurations/pytest_gke.xml` | IntelliJ: run tests tagged `@pytest.mark.gke` |
| `tests/run_tests.sh` | Fix `--cov` source from `install` to `knoe` |

### Commit Summary

```
97575b9 docs: update Phase 0 commit summary and resumption checklist
0052a4d Phase 0: test pipeline foundation — pyproject.toml, IntelliJ run configs, coverage fix, welcome mode selector
```

The following commits landed on the same branch as part of the Cowork+Code thread consolidation (WIP that had accumulated alongside Phase 0):

```
837da27 docs: update README; IntelliJ run config picks up Python 3.14 SDK
52fe440 chore(k3s): script and hostprobe updates, temp maintenance scripts
dd8c9d1 chore(scripts): init script updates — gitea, gitlab, kong, monitoring, registry
a754659 chore(deploy): cluster config and k3s manifest updates
4995c86 chore(build): Maven version and authority module updates
ca469aa chore(installer): core Python updates — env, milestones, monitoring
5cd9c12 feat(auth): init scripts and k3s/k8s auth manifests for knoe-auth
2a80df8 feat(auth): land Round 1 — invite-OTP enrollment, kadmin client, GKE manifests
6c72c76 docs(plans): add platform architecture plans — deployment-modes, knoe-auth round 1
```

### Known pre-existing issues (fix in Phase 1, not Phase 0)

`knoe/core/ops/cloudnative_pg.py:1372` contains:
```python
f"jsonpath={{.data.{field.replace('.', '\\.')}}}",
```
Python ≥ 3.12 allows backslashes in f-strings (PEP 701) but this still triggers a `SyntaxError` on Python 3.14 under certain parse modes. This causes **30 test collection errors** — the tests themselves are not broken. Fix: extract the replacement to a variable before the f-string. Tagged as Phase 1 work.

### Resumption Checklist

Before picking up work on Phase 1, verify:

- [x] `pytest tests/` collects 441+ tests from the project root
- [x] `pytest -m unit` runs and reports a coverage number against `knoe/`
- [x] IntelliJ shows the five run configs in the Run/Debug Configurations dropdown
- [x] `python -c "from knoe.ui.screens import KnoeInstaller"` succeeds
- [x] The 30 collection errors in `cloudnative_pg.py` are logged as Phase 1 work

---

## Off-pipeline work since Phase 0

These commits and live-cluster operations landed between Phase 0 and Phase 1. They aren't part of the autobuild pipeline itself — they're cluster-recovery and Maven build-readiness work that interrupted the Phase 1 sequence. Captured here so future readers see the actual chronology.

### 2026-04-28 → 2026-04-29: repo cleanup + Maven build readiness + cluster recovery

| Commit / event | Scope | Notes |
|---|---|---|
| [b355855](#) `refactor(auth): rename to dev.knoe.auth; archive tmp/ post-CNPG cleanup` | Phase A repo cleanup + Phase C Maven build readiness | tmp/ → archive/2026-04-cnpg-and-gitlab-cleanup/ (87 ad-hoc shell scripts); authority/ groupId `org.knoe` → `dev.knoe`, artifactId → `auth`, source tree moved `org/prole/authority/` → `dev/knoe/auth/`; build is now green via `make build-auth` (`mvn -f authority/pom.xml package` → `authority/target/knoe-auth.jar`); 4 k3s manifests renamed `prole-auth-*.yaml` → `knoe-auth-*.yaml`. |
| Phase B live cluster recovery (uncommitted; live-cluster operations) | Cluster recovery | `knoe-db-1` had been crashlooping for 14 days (full WAL volume) — fixed via PVC delete + CNPG re-bootstrap; cluster failover-cycled all 3 pods to refresh stale plugin-barman-cloud sidecars; **knoe-dev-cnpg-0 is now 3/3 healthy**, primary `knoe-db-2`. |
| Garage cluster recovery (uncommitted; live-cluster operations) | Backup target recovery | Garage on `knoe-dev-cnpg-0` was 100% full (`LMDB: No space left on device`) starting 2026-04-28 14:00 UTC; PVC expanded 29 → 80 GiB, layout capacity 10 → 60 GB, bucket cleared (8382 stale objects + 12 orphan multipart uploads). |
| GCS migration completed (uncommitted; live-cluster operations) | Backup target standardization | ObjectStore now `gs://knoe-0-backups/` with `googleCredentials.gkeEnvironment: true`; WI auth initially via auto-generated `knoe-db` SA annotation; 2 verified `data.tar.gz` backups in catalog; ScheduledBackup unsuspended. |

**Suspected root cause of the 14:00 UTC outage:** an `install.sh --mode k3d` run while shell `kubectx` was set to a GKE context overwrote the GCS-backed ObjectStore with a Garage-backed one, then Garage filled up. Env-contamination guard tracked in [`docs/TODO.md`](TODO.md).

### 2026-04-29 (later same day): CNPG operator + backup stabilization

After 5 consecutive scheduled backups completed cleanly to GCS, did a small wave to remove the remaining drift:

| Event | Scope | Notes |
|---|---|---|
| Operators restarted | Refresh stale state | `barman-cloud` (plugin operator) and `cnpg-controller-manager` rolled — both had been running 15-19 days. Plugin re-discovery confirmed in logs; stale Garage retention-worker endpoint cache cleared. |
| **CNPG operator upgraded v1.28.1 → v1.29.0** | Operator upgrade | Applied upstream release manifest (`releases/cnpg-1.29.0.yaml`). v1.29.0 added `spec.serviceAccountName` field (mutually exclusive with `serviceAccountTemplate`), required to land the SA migration below. Cluster healthy throughout. |
| **Cluster pod SA migration: knoe-db → cnpg-backup-sa** | Canonical pattern alignment | `cluster.spec.serviceAccountName=cnpg-backup-sa` applied. CNPG supervised rolling restart cycled all 3 pods. **First pod (db-3) failed RBAC check** — CNPG only auto-binds roles when it auto-creates the SA; for a pre-existing SA we had to add `cnpg-backup-sa` as a subject to the existing `knoe-db` and `knoe-db-barman-cloud` RoleBindings. After the patch all pods rolled cleanly. 23:03 UTC scheduled backup completed end-to-end with the new SA — verification gate passed. |
| **Garage removed from `knoe-dev-cnpg-0`** | DB-cluster cleanup | StatefulSet, both services (`garage` ClusterIP and `garage-s3-ilb` LoadBalancer), configmap, PVC, and `garage-hdd` StorageClass all deleted. ObjectStore untouched (still `gs://knoe-0-backups/`); backups continued through the Garage tear-down. |

Reality-vs-intent gaps surfaced by this work are captured in [`docs/TODO.md`](TODO.md) and as `> **Reality TODO:**` blocks in [`CLAUDE.md`](../CLAUDE.md) and [`AGENTS.md`](../AGENTS.md). Several gaps closed in this session (Garage on DB cluster, SA alignment); new ones added (redundant `[knoe-db-0/knoe-db]` IAM binding, init script + patch script need updates to reflect the new canonical end state).

---

## Phase 1 — `min` Mode Pipeline

**Status:** 🔲 Not started — ready to begin (deferred during the 2026-04-28 cluster-recovery work above)
**Depends on:** Phase 0 complete ✅
**First task:** fix `knoe/core/ops/cloudnative_pg.py:1372` f-string syntax (30 collection errors, see Phase 0 known issues)

### What this phase does

Establishes the first green autobuild. The `min` pipeline is the simplest possible CI: a bare macOS runner (no Docker, no Kubernetes) running pytest. It exercises the welcome-screen mode selector, the min-mode navigation fast-path, and the `init_min.sh` script. Trigger: push to `main` on the Gitea remote.

### Architecture

```
push to main (Gitea)
    │
    ▼
Gitea Actions
    │
    ▼
act_runner  ←── bare macOS (dev machine or Mac mini)
    │            labels: [self-hosted, macos, min]
    ▼            requires: Homebrew + Python 3.11
pytest -m "min or unit"
    │
    ▼
coverage.xml  ──► uploaded as artifact
```

**Runner setup (one-time, not in CI):**
```bash
# On the macOS runner machine:
brew install gitea-act-runner
act_runner register \
  --instance https://<gitea-host>/  \
  --token    <runner-token-from-gitea-settings> \
  --labels   "self-hosted,macos,min" \
  --name     "knoe-min-runner"
act_runner daemon
```

### Deliverables

| File | Purpose |
|---|---|
| `.gitea/workflows/ci-min.yml` | Gitea Actions workflow — push-triggered, runs on bare macOS runner |
| `tests/modes/__init__.py` | Package marker |
| `tests/modes/test_min_mode.py` | Mode-specific tests for min path |
| `tests/modes/conftest.py` | Mode fixtures (mock containerd, mock 1Password CLI) |

### `.gitea/workflows/ci-min.yml`

```yaml
name: CI — min mode

on:
  push:
    branches: [main]

jobs:
  min-unit:
    runs-on: [self-hosted, macos, min]
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: |
          pip install -r requirements.txt -r requirements-test.txt          

      - name: Run min + unit tests
        run: |
          pytest -m "min or unit" \
            --cov=knoe \
            --cov-report=xml \
            --cov-report=term-missing \
            -v \
            tests/          
        env:
          PYTHONPATH: ${{ github.workspace }}
          KNOE_MODE: min

      - name: Upload coverage artifact
        uses: actions/upload-artifact@v4
        with:
          name: coverage-min
          path: coverage.xml
          retention-days: 14
```

### Tests to write in `tests/modes/test_min_mode.py`

Each test name is intentionally human-readable — they become living documentation for junior developers.

| Test | What it proves |
|---|---|
| `test_welcome_mode_selector_starts_with_no_selection` | `deployment_mode` defaults to `""` on fresh install — forces explicit choice |
| `test_clicking_min_card_sets_cluster_env_to_min` | Card click wires `cluster_env = "min"` correctly |
| `test_clicking_min_card_enables_next_button` | Welcome Next is disabled until a card is clicked |
| `test_min_nav_env_setup_goes_to_init_password_not_cluster` | Verifies min fast-path in `on_next` |
| `test_min_nav_init_scripts_goes_to_security_not_kerberos` | Min skips Kerberos, ArgoCD, GitOps, Supabase |
| `test_min_nav_prev_from_init_password_returns_to_env_setup` | Symmetric back-navigation |
| `test_min_nav_prev_from_security_returns_to_init_scripts` | Symmetric back-navigation |
| `test_normalize_cluster_env_min_returns_min` | `_normalize_cluster_env("min") == "min"` |
| `test_deployment_mode_from_env_min_returns_min` | `_deployment_mode_from_env("min") == "min"` |
| `test_init_min_sh_runs_without_error` | `subprocess` call to `init_min.sh initialize` exits 0 (macOS only, `@pytest.mark.min`) |

### Commit Summary

*(Fill in after phase lands)*

```
# example:
# abc1234 Phase 1: add .gitea/workflows/ci-min.yml
# def5678 Phase 1: add tests/modes/test_min_mode.py
# ghi9012 Phase 1: add tests/modes/conftest.py
```

### Resumption Checklist

- [ ] Push to `main` triggers `ci-min` workflow in Gitea Actions UI
- [ ] `pytest -m "min or unit"` exits 0 locally
- [ ] All 10 tests in `test_min_mode.py` pass
- [ ] Coverage artifact appears in the Gitea Actions run summary
- [ ] Badge shows green on Gitea repo homepage (optional but nice)

---

## Phase 2 — `k3d` Mode Pipeline

**Status:** 🔲 Not started  
**Depends on:** Phase 1 complete

### What this phase does

Adds the k3d integration pipeline — a Docker-enabled runner that creates a real k3d cluster, deploys CNPG, verifies the database is reachable, then tears it down. This is the first pipeline that proves an actual database cluster starts correctly. Estimated runtime: 8–15 minutes.

### Architecture

```
push to main (Gitea)
    │
    ▼
Gitea Actions
    │
    ├─ job: k3d-unit (fast, no cluster) ─────────────► coverage-k3d-unit.xml
    │
    └─ job: k3d-integration (depends on unit) ──────► coverage-k3d-integration.xml
           │
           ├── k3d cluster create knoe-ci
           ├── apply CNPG operator
           ├── apply knoe-db Cluster CR
           ├── pytest -m "k3d and integration"
           └── k3d cluster delete knoe-ci (always)
```

**Runner setup (one-time):**  
Same `act_runner` binary but registered with labels `self-hosted,macos,k3d` on a Docker-enabled machine (Docker Desktop or Colima).

### Deliverables

| File | Purpose |
|---|---|
| `.gitea/workflows/ci-k3d.yml` | Two-job workflow: unit then integration |
| `tests/modes/test_k3d_mode.py` | k3d unit + integration tests |
| `tests/modes/fixtures/knoe-db-test.yaml` | Minimal CNPG `Cluster` CR for test use (single instance, tiny storage) |
| `tests/modes/fixtures/cnpg-operator-values.yaml` | Minimal Helm values for CNPG operator in CI |

### `.gitea/workflows/ci-k3d.yml`

```yaml
name: CI — k3d mode

on:
  push:
    branches: [main]

jobs:
  k3d-unit:
    runs-on: [self-hosted, macos, k3d]
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }
      - run: pip install -r requirements.txt -r requirements-test.txt
      - name: Unit tests (k3d, no cluster)
        run: pytest -m "k3d and unit" --cov=knoe --cov-report=xml tests/
        env:
          PYTHONPATH: ${{ github.workspace }}
          KNOE_MODE: k3d
      - uses: actions/upload-artifact@v4
        with: { name: coverage-k3d-unit, path: coverage.xml }

  k3d-integration:
    runs-on: [self-hosted, macos, k3d]
    needs: k3d-unit
    timeout-minutes: 25
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }
      - run: pip install -r requirements.txt -r requirements-test.txt

      - name: Install k3d + kubectl
        run: |
          brew install k3d kubectl          

      - name: Create test cluster
        run: |
          k3d cluster create knoe-ci \
            --agents 1 \
            --k3s-arg '--disable=traefik@server:0' \
            --wait
          kubectl cluster-info --context k3d-knoe-ci          

      - name: Deploy CNPG operator
        run: |
          kubectl apply --server-side \
            -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/main/releases/cnpg-1.23.0.yaml
          kubectl wait --for=condition=Available \
            deployment/cnpg-controller-manager \
            -n cnpg-system --timeout=120s          

      - name: Integration tests (live cluster)
        run: |
          pytest -m "k3d and integration" \
            --cov=knoe \
            --cov-report=xml \
            -v \
            tests/          
        env:
          PYTHONPATH: ${{ github.workspace }}
          KNOE_MODE: k3d
          KUBECONFIG: ${{ env.HOME }}/.kube/config

      - name: Tear down cluster
        if: always()
        run: k3d cluster delete knoe-ci

      - uses: actions/upload-artifact@v4
        with: { name: coverage-k3d-integration, path: coverage.xml }
```

### Tests to write in `tests/modes/test_k3d_mode.py`

**Unit tests** (`@pytest.mark.k3d @pytest.mark.unit`):

| Test | What it proves |
|---|---|
| `test_k3d_cluster_env_normalizes_to_dev` | `_normalize_cluster_env("dev") == "dev"` |
| `test_welcome_k3d_card_sets_cluster_env_dev` | Card click → `cluster_env = "dev"` |
| `test_k3d_nav_proceeds_through_cluster_screens` | k3d mode does NOT skip `init_cluster` |
| `test_k3d_supabase_option_visible` | Supabase toggle is reachable in k3d nav flow |
| `test_cluster_lifecycle_milestone_uses_k3d_script` | `ClusterLifecycleMilestone` calls k3d cluster-create command for `dev` env |

**Integration tests** (`@pytest.mark.k3d @pytest.mark.integration`):

| Test | What it proves |
|---|---|
| `test_k3d_cluster_api_is_reachable` | kubectl can reach `k3d-knoe-ci` API server |
| `test_cnpg_operator_crds_registered` | `Cluster` CRD exists after operator deploy |
| `test_knoe_db_cluster_pod_starts` | Apply test CR → at least one PostgreSQL pod reaches `Running` |
| `test_database_accepts_connections` | psql `SELECT 1` succeeds against the CNPG service |
| `test_cluster_delete_is_clean` | After delete, no k3d cluster named `knoe-ci` remains |

### Commit Summary

*(Fill in after phase lands)*

### Resumption Checklist

- [ ] Push to `main` triggers both `k3d-unit` and `k3d-integration` jobs in Gitea
- [ ] `k3d-integration` passes (PostgreSQL pod reaches Running state)
- [ ] Both coverage artifacts appear in the run summary
- [ ] `k3d cluster list` shows no leftover `knoe-ci` cluster after the run

---

## Phase 3 — GKE Pipeline (GitLab CI, expanded)

**Status:** 🔲 Not started  
**Depends on:** Phase 2 complete

### What this phase does

Replaces the current single-job `.gitlab-ci.yml` (which only runs `install.sh -S`) with a proper multi-stage pipeline: lint → unit tests → GKE integration tests → dead-code report → deploy. The unit test gate runs on every push; GKE integration runs on `main` only.

### Architecture

```
push to any branch
    │
    ├─ stage: lint     → python syntax + imports check
    ├─ stage: test     → pytest -m "unit"  (every push)
    │
push to main only:
    ├─ stage: test     → pytest -m "gke and integration"  (real GKE cluster)
    ├─ stage: test     → vulture dead-code report (allow_failure: true)
    └─ stage: deploy   → ./install.sh -S -c conf/service/prod.cfg
```

### Deliverables

| File | Purpose |
|---|---|
| `.gitlab-ci.yml` | Full multi-stage pipeline replacing current single-job version |
| `tests/modes/test_gke_mode.py` | GKE unit + integration tests |
| `scripts/dead_code_analysis.py` | Cross-references coverage.json with vulture-report.txt, outputs three-tier report |

### Required GitLab CI variables (project settings → CI/CD → Variables)

| Variable | Value | Protected | Masked |
|---|---|---|---|
| `GCP_SA_KEY` | GCP service account JSON with `container.viewer` + `storage.objectViewer` on both clusters | ✅ | ✅ |
| `GKE_REGION` | `us-west3` | | |
| `GKE_APP_CLUSTER` | `knoe-dev-0` | | |
| `GKE_DB_CLUSTER` | `knoe-dev-cnpg-0` | | |

### `.gitlab-ci.yml` (full replacement)

```yaml
stages:
  - lint
  - test
  - report
  - deploy

variables:
  PYTHON_VERSION: "3.11"
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

cache:
  paths: [.cache/pip]

# ── Stage: lint ───────────────────────────────────────────────────────────────

lint:
  stage: lint
  image: python:3.11-slim
  script:
    - pip install -r requirements-test.txt -q
    - python -m py_compile knoe/**/*.py
    - python -c "from knoe.ui.screens import KnoeInstaller"
  rules:
    - if: '$CI_PIPELINE_SOURCE == "push"'

# ── Stage: test (unit — every push) ──────────────────────────────────────────

unit-tests:
  stage: test
  image: python:3.11-slim
  script:
    - pip install -r requirements.txt -r requirements-test.txt -q
    - pytest -m "unit and not integration"
        --cov=knoe
        --cov-report=xml
        --cov-report=term-missing
        -q
        tests/
  coverage: '/TOTAL.*\s+(\d+\%)/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml
    paths: [coverage.xml]
    expire_in: 7 days
  rules:
    - if: '$CI_PIPELINE_SOURCE == "push"'

# ── Stage: test (GKE integration — main only) ─────────────────────────────────

gke-integration:
  stage: test
  image: google/cloud-sdk:slim
  timeout: 30 minutes
  before_script:
    - pip install -r requirements.txt -r requirements-test.txt -q
    - echo "$GCP_SA_KEY" | gcloud auth activate-service-account --key-file=-
    - gcloud config set project plenary-truck-485623-p7
    - gcloud container clusters get-credentials $GKE_APP_CLUSTER
        --region $GKE_REGION
    - gcloud container clusters get-credentials $GKE_DB_CLUSTER
        --region $GKE_REGION
  script:
    - pytest -m "gke and integration"
        --cov=knoe
        --cov-report=xml
        -v
        tests/
  coverage: '/TOTAL.*\s+(\d+\%)/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml
    paths: [coverage.xml]
    expire_in: 30 days
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'

# ── Stage: report (dead code — main only, never blocks build) ─────────────────

dead-code:
  stage: report
  image: python:3.11-slim
  allow_failure: true
  script:
    - pip install vulture -q
    - vulture knoe/ --min-confidence 80 | tee vulture-report.txt
    - python scripts/dead_code_analysis.py
        --vulture vulture-report.txt
        --coverage coverage.xml
        --output dead-code-report.md
  artifacts:
    paths:
      - vulture-report.txt
      - dead-code-report.md
    expire_in: 30 days
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'

# ── Stage: deploy (main only) ─────────────────────────────────────────────────

deploy-service:
  stage: deploy
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
  script:
    - ./install.sh -S -c conf/service/prod.cfg
  environment:
    name: production
```

### Tests to write in `tests/modes/test_gke_mode.py`

**Unit tests** (`@pytest.mark.gke @pytest.mark.unit`):

| Test | What it proves |
|---|---|
| `test_gke_cluster_env_normalizes_to_prod` | `_normalize_cluster_env("prod") == "prod"` |
| `test_welcome_gke_card_sets_cluster_env_prod` | Card click → `cluster_env = "prod"` |
| `test_gke_split_cluster_detected` | When `app_ctx != db_ctx`, split-cluster mode activates |
| `test_garage_not_deployed_to_db_cluster` | CNPG cluster does not include Garage in GKE mode |
| `test_gke_storage_class_validation` | `standard-rwo` raises quota error hint; `standard` passes |

**Integration tests** (`@pytest.mark.gke @pytest.mark.integration`):

| Test | What it proves |
|---|---|
| `test_gke_app_cluster_reachable` | kubectl can reach `knoe-dev-0` |
| `test_gke_db_cluster_reachable` | kubectl can reach `knoe-dev-cnpg-0` |
| `test_cnpg_cluster_knoe_db_0_running` | `knoe-db` cluster in `knoe-db-0` namespace has 3 Ready instances |
| `test_barman_backup_schedule_exists` | ScheduledBackup CR exists in `knoe-db-0` |
| `test_garage_only_in_app_cluster` | No Garage pods in `knoe-dev-cnpg-0` namespace |

### Commit Summary

*(Fill in after phase lands)*

### Resumption Checklist

- [ ] GitLab pipeline shows four stages: lint → test → report → deploy
- [ ] `unit-tests` job passes on every push (not just main)
- [ ] `gke-integration` passes on main (3 CNPG pods Running)
- [ ] `dead-code` job produces a `dead-code-report.md` artifact
- [ ] Deploy stage still works (`install.sh -S` exits 0)
- [ ] GitLab shows coverage percentage on the merge request widget

---

## Phase 4 — Dead Code Identification

**Status:** 🔲 Not started  
**Depends on:** Phase 3 complete (all three stable builds achieved)

### What this phase does

Runs the first systematic dead code analysis across the full codebase. This is not a cleanup sprint — it is *reconnaissance*. We generate a prioritised report and review it before deleting anything. The report becomes the input for Phase 5.

### Why this matters

The project has undergone several major renames (`prole` → `knoe`, various directory restructures). Code written for old layouts is likely still present. `knoe/core/actions.py` alone is 7,740 lines and almost certainly contains branches that no deployment mode ever exercises.

### Tools

| Tool | Role |
|---|---|
| `vulture` | Static analysis — finds unused functions, classes, imports |
| `pytest --cov --cov-branch` | Dynamic analysis — lines/branches never executed during tests |
| `scripts/dead_code_analysis.py` | Cross-references both outputs to produce a tiered report |

### Three-tier output format

```markdown
## 🔴 Definite dead code  (vulture-flagged AND 0% coverage)
- knoe/core/ops/legacy_shell.py:47 — function `_old_prole_exec` (unused, 0% coverage)
- ...

## 🟡 Suspect  (vulture-flagged OR 0% coverage, not both)
- knoe/core/actions.py:4201 — function `_k8s_node_drain` (0% coverage, not flagged by vulture)
- ...

## 🟢 Live  (covered by tests and vulture-clean)
- (omitted from report for brevity)
```

### How to run locally

```bash
# 1. Full coverage run across all modes
KNOE_MODE=min  pytest -m min  --cov=knoe --cov-append tests/
KNOE_MODE=k3d  pytest -m k3d  --cov=knoe --cov-append tests/
KNOE_MODE=gke  pytest -m gke  --cov=knoe --cov-append tests/
coverage json  # produces coverage.json

# 2. Vulture scan
vulture knoe/ --min-confidence 70 > vulture-report.txt

# 3. Cross-reference
python scripts/dead_code_analysis.py \
  --vulture vulture-report.txt \
  --coverage coverage.json \
  --output dead-code-report.md

# 4. Review
open dead-code-report.md
```

### Deliverables

| File | Purpose |
|---|---|
| `scripts/dead_code_analysis.py` | Cross-reference script (coverage.json + vulture output → tiered report) |
| `dead-code-report.md` | Generated output — committed to repo for review, not production |
| `docs/dead-code-review.md` | Human review notes — which 🔴 items are safe to delete vs accidentally flagged |

### Commit Summary

*(Fill in after phase lands)*

### Resumption Checklist

- [ ] `python scripts/dead_code_analysis.py` runs without error
- [ ] `dead-code-report.md` contains all three tiers
- [ ] The 🔴 list has been manually reviewed and each item categorised as "safe to delete" or "keep" in `docs/dead-code-review.md`
- [ ] No production code deleted yet — this phase is report-only

---

## Phase 5 — Pruning and the Instructable Codebase

**Status:** 🔲 Not started  
**Depends on:** Phase 4 report reviewed and approved

### What this phase does

This is the refactoring sprint. Using the Phase 4 report, we delete dead code in small, test-verified commits, raising the `fail_under` coverage threshold after each deletion. The end state is a codebase that a junior developer can read from top to bottom and fully understand.

### Target metrics

| Metric | Today (est.) | Target |
|---|---|---|
| Test coverage (unit) | ~40% | ≥ 80% |
| Test coverage (integration) | ~10% | ≥ 60% |
| `vulture --min-confidence 80` warnings | Unknown | 0 |
| Lines in `knoe/core/actions.py` | 7,740 | < 2,000 |
| Modules with zero test coverage | ~17 (`knoe/core/ops/`) | 0 |
| `fail_under` in `pyproject.toml` | 0 | 75 |

### Deletion protocol

For each 🔴 item from the Phase 4 report:

1. Write a test that would fail if the code were still needed (proves it's safe to delete)
2. Delete the code
3. Run `pytest` — all tests pass
4. Commit with message: `prune: remove <module/function> — dead code (vulture + 0% coverage)`
5. Raise `fail_under` by 1–2 points

This protocol means every deletion is backed by a test. The test suite grows *because* we prune.

### The instructable codebase standard

A module is "instructable" when:
- Every public function has at least one test whose name reads as a plain English sentence
- The test file for the module is shorter than the module itself
- A junior developer can understand the module's purpose from the test names alone without reading the source

Example of instructable test names:
```python
def test_min_mode_only_needs_homebrew_and_1password(): ...
def test_k3d_mode_creates_cnpg_cluster_in_docker(): ...
def test_welcome_card_click_routes_to_correct_nav_flow(): ...
def test_kerberos_is_only_enabled_in_k3s_and_gke_modes(): ...
```

### Commit Summary

*(Fill in after each deletion sprint)*

### Resumption Checklist

- [ ] `fail_under` has been raised at least once since Phase 4
- [ ] Zero 🔴 items remain in `dead-code-report.md`
- [ ] All modules in `knoe/core/ops/` have at least one test
- [ ] CI passes on all three pipelines (min, k3d, gke)

---

## k3s Phase — Homelab Pipeline

**Status:** ⏸ Deferred  
**Trigger:** Evaluate after GitLab CI (Phase 3) is stable

### Notes for when this is ready

- Runner: one of the physical k3s nodes (`myrddin.prole.org`, `gandalf.prole.org`, or `merlin.prole.org`) registered as a Gitea act_runner with labels `self-hosted,linux,k3s`
- Integration tests will need `kubeconfig` for the k3s cluster and iSCSI storage access for CNPG
- Kerberos integration is the unique test target here — `knoe-auth` deployment with embedded KDC
- The `CNPG_ELIGIBLE_NODES` config var maps directly to tests that verify node affinity placement

---

## Appendix A — Pytest Marker Reference

| Marker | When to use |
|---|---|
| `@pytest.mark.unit` | No external services, no file system writes, runs anywhere |
| `@pytest.mark.integration` | Requires a running cluster, database, or network service |
| `@pytest.mark.min` | Tests the min (containerd) deployment path |
| `@pytest.mark.k3d` | Tests the k3d (local Docker cluster) deployment path |
| `@pytest.mark.k3s` | Tests the k3s (homelab) deployment path |
| `@pytest.mark.gke` | Tests the GKE (production) deployment path |

Combining markers is the norm: `@pytest.mark.k3d @pytest.mark.integration` means "requires a live k3d cluster".

---

## Appendix B — Runner Registration Quick Reference

### Gitea act_runner (macOS)

```bash
# Install
brew install act-runner

# Register (run once per machine)
act_runner register \
  --instance  https://<gitea-host>/ \
  --token     <token-from-gitea-settings-actions-runners> \
  --labels    "self-hosted,macos,min"   # or min,k3d for the Docker machine \
  --name      "knoe-min-runner"         # human-readable name in Gitea UI

# Start as a service
brew services start act-runner
```

### GitLab runner (existing)

The existing GitLab runner is assumed to have `gcloud` CLI available. If not:
```bash
# On the runner host:
curl https://sdk.cloud.google.com | bash
gcloud components install gke-gcloud-auth-plugin
```

---

## Appendix C — Coverage Increment Strategy

Rather than setting an ambitious `fail_under` up front and having CI permanently broken, we raise it in steps as each phase lands:

| After phase | `fail_under` |
|---|---|
| Phase 0 (foundation) | 0 (measure only) |
| Phase 1 (min tests) | 15 |
| Phase 2 (k3d tests) | 25 |
| Phase 3 (gke tests) | 35 |
| Phase 4 (dead code pruned) | 50 |
| Phase 5 (full prune) | 75 |

Each increment is a one-line change to `pyproject.toml` committed at the end of the phase.