diff --git a/docs/EMBEDDED-RESOURCES.md b/docs/EMBEDDED-RESOURCES.md index 81143c7..c4c381c 100644 --- a/docs/EMBEDDED-RESOURCES.md +++ b/docs/EMBEDDED-RESOURCES.md @@ -14,7 +14,7 @@ The Prole Installer package includes several embedded resources that must be acc - **proleIconblueprint.png** (1.6 MB) - Blueprint icon variant ### 2. Binary Executables -- **prole-net/prole-scan** (6.8 MB) - Network scanner +- **prole-net/prole-agent** (6.8 MB) - Network scanner - Universal binary (x86_64 + arm64) - Used by network scan screen - Detects Kerberos, Active Directory, etc. @@ -54,7 +54,7 @@ if bg_path.exists(): **Binary Execution:** ```python -scan_binary = get_resource_path("prole-net/prole-scan") +scan_binary = get_resource_path("prole-net/prole-agent") if scan_binary.exists(): process = subprocess.Popen([str(scan_binary)], ...) ``` @@ -86,7 +86,7 @@ datas = [ **Binaries:** ```python binaries = [ - ('prole-net/prole-scan', 'prole-net'), + ('prole-net/prole-agent', 'prole-net'), ] ``` @@ -98,7 +98,7 @@ Total embedded resources: ~30-35 MB Breakdown: - Images: ~11 MB -- prole-scan: 6.8 MB +- prole-agent: 6.8 MB - Prole Tools.app: ~12 MB - Other resources: ~5-10 MB @@ -108,10 +108,10 @@ Final installer bundle: ~50-100 MB (includes Python runtime) ### Network Scan Screen -The network scan screen uses `prole-scan` to detect services: +The network scan screen uses `prole-agent` to detect services: ```python -scan_binary = get_resource_path("prole-net/prole-scan") +scan_binary = get_resource_path("prole-net/prole-agent") process = subprocess.Popen([str(scan_binary)], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, @@ -140,7 +140,7 @@ python3 test_embedded_resources.sh This tests: - ✓ All image files exist -- ✓ prole-scan binary exists and is executable +- ✓ prole-agent binary exists and is executable - ✓ Prole Tools.app bundle exists - ✓ Spec file includes all resources - ✓ Resource sizes @@ -163,18 +163,18 @@ ls -la "/tmp/_MEI*/prole-app/dist/Prole Tools.app" ### Binary Not Found Error -**Error:** `Scan binary not found at /var/folders/.../prole-net/prole-scan` +**Error:** `Scan binary not found at /var/folders/.../prole-net/prole-agent` **Cause:** Binary not included in package or path not using `get_resource_path()` **Solution:** -1. Verify spec includes binary: `grep prole-scan installer.spec` -2. Check code uses `get_resource_path()`: `grep "get_resource_path.*prole-scan" install.py` +1. Verify spec includes binary: `grep prole-agent installer.spec` +2. Check code uses `get_resource_path()`: `grep "get_resource_path.*prole-agent" install.py` 3. Rebuild: `make clean && make package` ### Binary Not Executable -**Error:** `Permission denied` when running prole-scan +**Error:** `Permission denied` when running prole-agent **Cause:** Binary permissions not preserved in package @@ -182,10 +182,10 @@ ls -la "/tmp/_MEI*/prole-app/dist/Prole Tools.app" Ensure binary is in `binaries` list (not `datas`) in spec file: ```python binaries = [ - ('prole-net/prole-scan', 'prole-net'), # Correct - preserves +x + ('prole-net/prole-agent', 'prole-net'), # Correct - preserves +x ] # NOT in datas: -# datas = [('prole-net/prole-scan', 'prole-net')] # Wrong - loses +x +# datas = [('prole-net/prole-agent', 'prole-net')] # Wrong - loses +x ``` ### App Bundle Not Found @@ -217,7 +217,7 @@ binaries = [ 3. **Exclude debug symbols:** Strip binaries before packaging: ```bash - strip prole-net/prole-scan + strip prole-net/prole-agent ``` ## Build Process @@ -250,7 +250,7 @@ When the packaged installer runs: Embedded binaries should be code signed: ```bash -codesign --force --sign "Developer ID Application: Your Name" prole-net/prole-scan +codesign --force --sign "Developer ID Application: Your Name" prole-net/prole-agent ``` Then build the installer - the signed binary will be included. @@ -260,8 +260,8 @@ Then build the installer - the signed binary will be included. Users can verify embedded binaries: ```bash -# Check signature of prole-scan after extraction -codesign --verify --verbose /tmp/_MEI*/prole-net/prole-scan +# Check signature of prole-agent after extraction +codesign --verify --verbose /tmp/_MEI*/prole-net/prole-agent # Check installer bundle signature codesign --verify --verbose "dist/Prole Installer.app" diff --git a/docs/PROLE-HOME-DIRECTORY.md b/docs/PROLE-HOME-DIRECTORY.md index 561bde8..4b64a38 100644 --- a/docs/PROLE-HOME-DIRECTORY.md +++ b/docs/PROLE-HOME-DIRECTORY.md @@ -50,7 +50,7 @@ subprocess.Popen(['docker', 'build', '-t', 'prole-db:TAG', '.'], cwd=build_dir) **Purpose:** Writable working directory for network scan operations -**Why:** The `prole-scan` binary may need to write output files, cache data, or store temporary results. Running from a read-only directory causes failures. +**Why:** The `prole-agent` binary may need to write output files, cache data, or store temporary results. Running from a read-only directory causes failures. **Usage:** ```python @@ -65,7 +65,7 @@ subprocess.Popen([str(scan_binary)], cwd=str(scan_dir)) **Contents:** - Network scan results (temporary) - Ollama API interaction cache -- Any intermediate files created by prole-scan +- Any intermediate files created by prole-agent **Size:** Varies, typically < 1 MB diff --git a/docs/build-system.md b/docs/build-system.md index a5c0c74..d76674b 100644 --- a/docs/build-system.md +++ b/docs/build-system.md @@ -97,7 +97,7 @@ A Python script (`scripts/generate_spec.py`) creates the PyInstaller specificati - `prole-app/dist/Prole Tools.app` - Pre-built Prole Tools application bundle (entire .app) **Included Binaries:** -- `prole-net/prole-scan` - Network scanner binary (universal: x86_64 + arm64, 6.8 MB) +- `prole-net/prole-agent` - Network scanner binary (universal: x86_64 + arm64, 6.8 MB) **Resource Path Resolution:** The installer uses a `get_resource_path()` helper function that automatically resolves paths correctly whether running from source or from a PyInstaller bundle: diff --git a/etc/init_cloudnative_pg.sh b/etc/init_cloudnative_pg.sh index 06dd524..d50d223 100755 --- a/etc/init_cloudnative_pg.sh +++ b/etc/init_cloudnative_pg.sh @@ -103,7 +103,7 @@ ensure_prole_stack_resources() { dir="$SCRIPT_DIR/../k8s/prole" for file in "$dir"/*.yaml; do case "$(basename "$file")" in - prole-db.yaml|kustomization.yaml) + prole-db.yaml|kustomization.yaml|supabase-*.yaml) continue ;; esac @@ -115,7 +115,7 @@ ensure_prole_stack_resources() { dir="$SCRIPT_DIR/../k8s/prole" for file in "$dir"/*.yaml; do case "$(basename "$file")" in - kustomization.yaml) + kustomization.yaml|supabase-*.yaml) continue ;; esac @@ -147,21 +147,46 @@ wait_for_cnpg_pods() { local start_time start_time=$(date +%s) + local target_pods="${CNPG_TARGET_PODS:-}" + if [[ -z "$target_pods" ]]; then + target_pods=$(kubectl -n "$NAMESPACE" get cluster "$CNPG_CLUSTER_NAME" -o jsonpath='{.spec.instances}' 2>/dev/null || true) + fi + if [[ -z "$target_pods" ]]; then + target_pods=3 + fi + + echo "Waiting for $target_pods CNPG pods for cluster '$CNPG_CLUSTER_NAME' in namespace '$NAMESPACE' to be Running..." while true; do - if kubectl -n "$NAMESPACE" get cluster "$CNPG_CLUSTER_NAME" >/dev/null 2>&1; then - if kubectl -n "$NAMESPACE" get pods -l "cnpg.io/cluster=$CNPG_CLUSTER_NAME" --no-headers 2>/dev/null | grep -q .; then - if kubectl -n "$NAMESPACE" wait --for=condition=Ready pod -l "cnpg.io/cluster=$CNPG_CLUSTER_NAME" --timeout=60s >/dev/null 2>&1; then - return 0 - fi + local pods + pods=$(kubectl -n "$NAMESPACE" get pods -l "cnpg.io/cluster=$CNPG_CLUSTER_NAME" --no-headers 2>/dev/null || true) + + if [[ -n "$pods" ]]; then + # Check for Error or CrashLoopBackOff + if echo "$pods" | grep -E "Error|CrashLoopBackOff" >/dev/null; then + echo "ERROR: Some CNPG pods are in Error or CrashLoopBackOff state:" >&2 + echo "$pods" | grep -E "Error|CrashLoopBackOff" >&2 + return 1 + fi + + # Count Running pods by name (exclude initdb) + local running_pods + running_pods=$(kubectl -n "$NAMESPACE" get pods --no-headers 2>/dev/null \ + | grep "^${CNPG_CLUSTER_NAME}-" \ + | grep -v "initdb" \ + | grep "Running" \ + | wc -l | xargs) + + if [[ "$running_pods" -ge "$target_pods" ]]; then + echo "All $running_pods/$target_pods pods are Running." + return 0 fi fi if (( $(date +%s) - start_time > timeout )); then - echo "ERROR: Timed out waiting for CNPG pods for cluster '$CNPG_CLUSTER_NAME' in namespace '$NAMESPACE'." >&2 + echo "ERROR: Timed out waiting for $target_pods CNPG pods for cluster '$CNPG_CLUSTER_NAME' in namespace '$NAMESPACE'." >&2 return 1 fi - echo "Waiting for CNPG pods for cluster '$CNPG_CLUSTER_NAME'..." - sleep 3 + sleep 5 done } @@ -299,13 +324,19 @@ initialize() { return 1 fi - if [[ -x "$SCRIPT_DIR/init_kerberos.sh" ]]; then - echo "Configuring Kerberos for CNPG pods ..." - if ! "$SCRIPT_DIR/init_kerberos.sh" initialize; then - echo "WARN: Kerberos initialization did not complete successfully." + # Support both names for the toggle from prole.cfg + local kerberos_enabled="${KERBEROS_ENABLED:-${ENABLED:-false}}" + if [[ "$kerberos_enabled" == "true" || "$kerberos_enabled" == "True" || "$kerberos_enabled" == "1" ]]; then + if [[ -x "$SCRIPT_DIR/init_kerberos.sh" ]]; then + echo "Configuring Kerberos for CNPG pods ..." + if ! "$SCRIPT_DIR/init_kerberos.sh" initialize; then + echo "WARN: Kerberos initialization did not complete successfully." + fi + else + echo "WARN: init_kerberos.sh not found; skipping Kerberos configuration." fi else - echo "WARN: init_kerberos.sh not found; skipping Kerberos configuration." + echo "Kerberos is disabled; skipping Kerberos configuration." fi # Ensure port-forward is running for Postgres (local access) diff --git a/etc/init_garage_store.sh b/etc/init_garage_store.sh index dcdb5c2..ef06bca 100755 --- a/etc/init_garage_store.sh +++ b/etc/init_garage_store.sh @@ -215,13 +215,28 @@ init_layout() { assign_count=$((assign_count + 1)) done - # Get current version for apply - local version - version=$(echo "$layout" | grep "Version:" | awk '{print $2}' || echo "0") - if [[ -z "$version" ]]; then version=0; fi - local next_version=$((version + 1)) - echo "Applying Garage layout (version $next_version) ..." - kubectl exec -n "$NAMESPACE" "$pod" -- /garage layout apply --version "$next_version" || true + # Re-fetch layout and apply staged changes using the current layout version + local layout_after version apply_ok + layout_after=$(kubectl exec -n "$NAMESPACE" "$pod" -- /garage layout show 2>/dev/null || true) + version=$(echo "$layout_after" | awk -F: '/[Vv]ersion/ {gsub(/ /,"",$2); print $2; exit}' || true) + if echo "$layout_after" | grep -qi "staged"; then + if [[ -n "$version" ]]; then + echo "Applying Garage layout (version $version) ..." + if kubectl exec -n "$NAMESPACE" "$pod" -- /garage layout apply --version "$version"; then + apply_ok=1 + else + apply_ok=0 + fi + else + apply_ok=0 + fi + if [[ "${apply_ok:-0}" -eq 0 ]]; then + echo "WARN: layout apply with explicit version failed; retrying without version..." + kubectl exec -n "$NAMESPACE" "$pod" -- /garage layout apply || true + fi + else + echo "No staged layout changes detected; skipping layout apply." + fi else echo "Garage layout already assigned for node $short_id." fi diff --git a/etc/init_kerberos.sh b/etc/init_kerberos.sh index 94e385f..27d44a0 100755 --- a/etc/init_kerberos.sh +++ b/etc/init_kerberos.sh @@ -57,6 +57,15 @@ ensure_namespace() { } resolve_krb5_defaults() { + # If 'dog' KDC is available in 'authority' namespace, use it as default + if kubectl get pod -n "authority" -l app=dog >/dev/null 2>&1; then + KRB5_KDC="dog.authority.svc.cluster.local" + KRB5_ADMIN="dog.authority.svc.cluster.local" + if [[ -z "${KRB5_REALM}" ]]; then + KRB5_REALM="PROLE.ORG" + fi + fi + if [[ -z "${KRB5_REALM}" && -n "${REALM:-}" ]]; then KRB5_REALM="$REALM" fi @@ -184,16 +193,29 @@ patch_cnpg_cluster_for_auth() { # Best-effort patch; schema may vary with CNPG version. # We remove krb_srvname as it is unrecognized in some PG 17 builds. # We revert certificates to default to avoid operator TLS issues. - if ! kubectl -n "$NAMESPACE" patch cluster "$CNPG_CLUSTER_NAME" --type merge -p "{ - \"spec\": { - \"postgresql\": { - \"parameters\": {\"krb_srvname\": null}, - \"pg_hba\": [ + + local kerberos_enabled="${KERBEROS_ENABLED:-${ENABLED:-false}}" + local pg_hba + if [[ "$kerberos_enabled" == "true" || "$kerberos_enabled" == "True" || "$kerberos_enabled" == "1" ]]; then + pg_hba="[ \"local all postgres trust\", \"host all postgres all scram-sha-256\", \"host all all all gss include_realm=1 krb_realm=$KRB5_REALM\", \"host all all all scram-sha-256\" - ] + ]" + else + pg_hba="[ + \"local all postgres trust\", + \"host all postgres all scram-sha-256\", + \"host all all all scram-sha-256\" + ]" + fi + + if ! kubectl -n "$NAMESPACE" patch cluster "$CNPG_CLUSTER_NAME" --type merge -p "{ + \"spec\": { + \"postgresql\": { + \"parameters\": {\"krb_srvname\": null}, + \"pg_hba\": $pg_hba }, \"certificates\": { \"serverCASecret\": null, @@ -323,6 +345,15 @@ initialize() { ensure_namespace resolve_krb5_defaults + # Support both names for the toggle from prole.cfg + local enabled="${KERBEROS_ENABLED:-${ENABLED:-false}}" + if [[ "$enabled" == "false" || "$enabled" == "0" || "$enabled" == "False" ]]; then + log "Kerberos is disabled (KERBEROS_ENABLED=$enabled). Skipping initialization." + # Cleanup any existing Kerberos config if it was previously enabled + kubectl -n "$NAMESPACE" delete configmap prole-krb5-conf --ignore-not-found + return 0 + fi + if [[ -z "${KRB5_REALM}" || -z "${KRB5_KDC}" ]]; then err "ERROR: KRB5_REALM and KRB5_KDC must be set for Kerberos initialization." err " Set KRB5_REALM/KRB5_KDC or REALM/DOMAIN in env or config." diff --git a/etc/init_kerberos_test.sh b/etc/init_kerberos_test.sh index 160801e..81aa88d 100755 --- a/etc/init_kerberos_test.sh +++ b/etc/init_kerberos_test.sh @@ -81,7 +81,7 @@ detect_image() { if [[ -n "$release_file" && -f "$release_file" ]]; then release=$(cat "$release_file" | tr -d '[:space:]') else - release="41" + release="43" fi if [[ "$release" =~ ^[0-9]+$ ]]; then @@ -142,6 +142,13 @@ run_test() { ensure_tools ensure_namespace + # Support both names for the toggle from prole.cfg + local enabled="${KERBEROS_ENABLED:-${ENABLED:-false}}" + if [[ "$enabled" == "false" || "$enabled" == "0" || "$enabled" == "False" ]]; then + echo "Kerberos is disabled (KERBEROS_ENABLED=$enabled). Skipping test." + return 0 + fi + if [[ -z "$KRB5_REALM" || -z "$KRB5_USER" || -z "$KRB5_PASSWORD" || -z "$KRB5_KDC" ]]; then echo "ERROR: Missing Kerberos configuration. Ensure KRB5_REALM, KRB5_KDC, KRB5_USER, KRB5_PASSWORD are set." >&2 exit 1 diff --git a/etc/init_prole-db.sh b/etc/init_prole-db.sh index 5481921..5600aa1 100755 --- a/etc/init_prole-db.sh +++ b/etc/init_prole-db.sh @@ -68,7 +68,7 @@ get_latest_image() { if [[ -f "$release_file" ]]; then release=$(cat "$release_file" | tr -d '[:space:]') else - release="41" + release="43" fi if [[ "$release" =~ ^[0-9]+$ ]]; then @@ -78,6 +78,24 @@ get_latest_image() { echo "prole-db:${pg_version}-${release}" } +sync_manifest_image() { + local image="$1" + local manifest_dir + manifest_dir=$(dirname "$CNPG_MANIFEST") + local files=("$CNPG_MANIFEST") + if [[ -f "$manifest_dir/prole-db-recovery.yaml.tpl" ]]; then + files+=("$manifest_dir/prole-db-recovery.yaml.tpl") + fi + local f tmp + for f in "${files[@]}"; do + if [[ -f "$f" ]] && grep -qE '^[[:space:]]*imageName:' "$f"; then + tmp=$(mktemp) + sed -E "s|^([[:space:]]*imageName:).*|\\1 ${image}|" "$f" > "$tmp" + mv "$tmp" "$f" + fi + done +} + start() { ensure_tools ensure_namespace @@ -116,6 +134,7 @@ start() { # Compare latest image with deployed local latest_image latest_image=$(get_latest_image) + sync_manifest_image "$latest_image" local current_image current_image=$(kubectl get cluster "$CNPG_CLUSTER_NAME" -n "$NAMESPACE" -o jsonpath='{.spec.imageName}' 2>/dev/null || echo "") @@ -190,6 +209,7 @@ deploy() { else image="prole-db:$VERSION" fi + sync_manifest_image "$image" echo "Deploying $image to cluster $CNPG_CLUSTER_NAME..." diff --git a/etc/init_supabase.sh b/etc/init_supabase.sh index 63be117..1289db6 100755 --- a/etc/init_supabase.sh +++ b/etc/init_supabase.sh @@ -4,9 +4,10 @@ set -euo pipefail # init_supabase.sh # Purpose: -# - Deploy Supabase in Kubernetes and connect it to the CNPG Postgres cluster. +# - Deploy Supabase stack in Kubernetes using the official supabase/docker compose +# - Rewire Supabase to use the CNPG Postgres service +# - Disable image pulls (imagePullPolicy: Never) for now -# Initialize SCRIPT_DIR SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) # Load environment and config via prole_cfg.sh @@ -16,103 +17,389 @@ source "$SCRIPT_DIR/prole_cfg.sh" ACTION=${1:-} CNPG_CLUSTER_NAME=${CNPG_CLUSTER_NAME:-prole-db} - -# Support both PROLE_HOME/k8s and sibling k8s directory -if [[ -d "$SCRIPT_DIR/../k8s/prole" ]]; then - SUPABASE_MANIFEST_DIR="$SCRIPT_DIR/../k8s/prole" -elif [[ -n "${PROLE_HOME:-}" && -d "$PROLE_HOME/k8s/prole" ]]; then - SUPABASE_MANIFEST_DIR="$PROLE_HOME/k8s/prole" -else - SUPABASE_MANIFEST_DIR="$SCRIPT_DIR/../k8s/prole" -fi - -SUPABASE_FILES=( - "$SUPABASE_MANIFEST_DIR/supabase-configmap.yaml" - "$SUPABASE_MANIFEST_DIR/supabase-deployment.yaml" - "$SUPABASE_MANIFEST_DIR/supabase-service.yaml" -) +SUPABASE_HOME=${SUPABASE_HOME:-} +SUPABASE_USE_DEV_COMPOSE=${SUPABASE_USE_DEV_COMPOSE:-0} +SUPABASE_IMAGE_PULL_POLICY=${SUPABASE_IMAGE_PULL_POLICY:-Never} +SUPABASE_K8S_DIR=${SUPABASE_K8S_DIR:-${PROLE_HOME:-$SCRIPT_DIR/..}/build/supabase-k8s} +SUPABASE_POSTGRES_HOST=${SUPABASE_POSTGRES_HOST:-db} +SUPABASE_POSTGRES_DB=${SUPABASE_POSTGRES_DB:-postgres} +SUPABASE_POSTGRES_PORT=${SUPABASE_POSTGRES_PORT:-5432} +SUPABASE_APPLY_DB_MIGRATIONS=${SUPABASE_APPLY_DB_MIGRATIONS:-1} usage() { cat <&2; } + ensure_tools() { - for t in kubectl; do - command -v "$t" >/dev/null || { echo "Missing required tool: $t" >&2; exit 1; } + for t in kubectl kompose python3 sed awk base64; do + command -v "$t" >/dev/null || { err "Missing required tool: $t"; exit 1; } done } ensure_namespace() { - if ! kubectl get namespace "$NAMESPACE" >/dev/null 2>&1; then - echo "Creating namespace '$NAMESPACE' ..." - kubectl create namespace "$NAMESPACE" >/dev/null 2>&1 || true + if ! kubectl get namespace "supabase" >/dev/null 2>&1; then + err "ERROR: namespace 'supabase' not found. Supabase must be deployed in its own namespace." + exit 1 fi } ensure_prereqs() { ensure_namespace - if ! kubectl get cluster "$CNPG_CLUSTER_NAME" -n "$NAMESPACE" >/dev/null 2>&1; then - echo "ERROR: CNPG cluster '$CNPG_CLUSTER_NAME' not found in namespace '$NAMESPACE'." >&2 - exit 1 - fi - if ! kubectl get secret prole-db-user -n "$NAMESPACE" >/dev/null 2>&1; then - echo "ERROR: Secret 'prole-db-user' not found in namespace '$NAMESPACE'." >&2 + # We still want to ensure prole-db-superuser secret is in the CURRENT namespace (where prole-db is) + # but supabase itself will be in 'supabase' namespace. + # The issue description says: "connect supabase postgres network ports to our new namespace" + # This implies we might need to create services in the CURRENT namespace that point to supabase. + if ! kubectl get secret prole-db-superuser -n "$NAMESPACE" >/dev/null 2>&1; then + err "ERROR: Secret 'prole-db-superuser' not found in namespace '$NAMESPACE'." exit 1 fi } -apply_manifests() { - for f in "${SUPABASE_FILES[@]}"; do +detect_supabase_home() { + if [[ -n "$SUPABASE_HOME" && -d "$SUPABASE_HOME" ]]; then + echo "$SUPABASE_HOME" + return 0 + fi + if [[ -d "$HOME/prole/supabase" ]]; then + echo "$HOME/prole/supabase" + return 0 + fi + if [[ -d "$HOME/dev/supabase" ]]; then + echo "$HOME/dev/supabase" + return 0 + fi + if [[ -n "${PROLE_HOME:-}" && -d "$PROLE_HOME/../supabase" ]]; then + echo "$PROLE_HOME/../supabase" + return 0 + fi + return 1 +} + +get_db_password() { + kubectl -n "$NAMESPACE" get secret prole-db-superuser -o jsonpath='{.data.password}' 2>/dev/null | base64 -d +} + +set_env_kv() { + local file="$1" key="$2" value="$3" + # Ensure value is quoted if it contains spaces and is not already quoted + if [[ "$value" == *" "* && ! "$value" =~ ^\".*\"$ && ! "$value" =~ ^\'.*\'$ ]]; then + value="\"$value\"" + fi + if grep -q "^${key}=" "$file" 2>/dev/null; then + sed -i.bak "s|^${key}=.*|${key}=${value}|" "$file" + else + printf "%s=%s\n" "$key" "$value" >> "$file" + fi +} + +build_env_file() { + local supa_home="$1" + local docker_dir="$supa_home/docker" + local env_base="$docker_dir/.env" + local env_example="$docker_dir/.env.example" + local env_out="$SUPABASE_K8S_DIR/.env" + local db_pass + db_pass=$(get_db_password || true) + + mkdir -p "$SUPABASE_K8S_DIR" + + if [[ -f "$env_base" ]]; then + cp "$env_base" "$env_out" + elif [[ -f "$env_example" ]]; then + cp "$env_example" "$env_out" + else + err "ERROR: Supabase .env or .env.example not found in $docker_dir" + exit 1 + fi + + # If we used .env.example, generate required secrets + if [[ ! -f "$env_base" ]]; then + if [[ -x "$docker_dir/utils/generate-keys.sh" ]]; then + log "Generating Supabase secrets from utils/generate-keys.sh ..." + local gen_out + gen_out=$(bash "$docker_dir/utils/generate-keys.sh" /dev/null || true + + # Enforce imagePullPolicy + python3 - </dev/null || true) + if [[ -z "$primary" ]]; then + primary=$(kubectl -n "$NAMESPACE" get pods -l "cnpg.io/cluster=$CNPG_CLUSTER_NAME" -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true) + fi + if [[ -z "$primary" ]]; then + err "WARN: Unable to locate primary CNPG pod; skipping migrations." + return + fi + + local db_pass + db_pass=$(get_db_password || true) + if [[ -z "$db_pass" ]]; then + err "WARN: Unable to read prole-db-superuser password; skipping migrations." + return + fi + + log "Applying Supabase SQL migrations to CNPG (${SUPABASE_POSTGRES_DB}) ..." + local files=( + "$sql_dir/_supabase.sql" + "$sql_dir/webhooks.sql" + "$sql_dir/realtime.sql" + "$sql_dir/logs.sql" + "$sql_dir/pooler.sql" + "$sql_dir/jwt.sql" + "$sql_dir/roles.sql" + ) + + for f in "${files[@]}"; do if [[ -f "$f" ]]; then - kubectl apply -n "$NAMESPACE" -f "$f" - else - echo "ERROR: Missing manifest: $f" >&2 - exit 1 + log "Applying $(basename "$f") ..." + kubectl -n "$NAMESPACE" exec -i "$primary" -c postgres -- \ + env PGPASSWORD="$db_pass" psql -U postgres -d "$SUPABASE_POSTGRES_DB" -v ON_ERROR_STOP=1 -f - < "$f" || true fi done } -delete_manifests() { - for f in "${SUPABASE_FILES[@]}"; do - if [[ -f "$f" ]]; then - kubectl delete -n "$NAMESPACE" -f "$f" --ignore-not-found - fi - done +delete_k8s_resources() { + if [[ -d "$SUPABASE_K8S_DIR" ]]; then + kubectl delete -n "supabase" -f "$SUPABASE_K8S_DIR" --ignore-not-found || true + fi + kubectl delete -n "$NAMESPACE" svc/supabase-db --ignore-not-found || true + kubectl delete -n "supabase" svc/db --ignore-not-found || true } status() { ensure_tools - echo "Supabase status in namespace '$NAMESPACE':" - kubectl get deploy supabase -n "$NAMESPACE" || true - kubectl get svc supabase -n "$NAMESPACE" || true + log "Supabase status in namespace 'supabase':" + kubectl get deploy,svc -n "supabase" | grep -E "supabase|kong|auth|rest|realtime|storage|meta|analytics|vector|imgproxy|functions|edge|pooler|db" || true + log "Federated services in namespace '$NAMESPACE':" + kubectl get svc -n "$NAMESPACE" | grep supabase-db || true } case "$ACTION" in start) ensure_tools ensure_prereqs - echo "Applying Supabase manifests in namespace '$NAMESPACE'..." - apply_manifests + SUPABASE_HOME="$(detect_supabase_home)" || { err "ERROR: Supabase repo not found. Set SUPABASE_HOME or symlink ~/prole/supabase."; exit 1; } + log "Using Supabase repo: $SUPABASE_HOME" + log "Using namespace: $NAMESPACE" + convert_compose_to_k8s "$SUPABASE_HOME" + apply_k8s_resources + apply_db_migrations "$SUPABASE_HOME" ;; stop) ensure_tools - echo "Deleting Supabase manifests from namespace '$NAMESPACE'..." - delete_manifests + ensure_namespace + delete_k8s_resources ;; restart) ensure_tools ensure_prereqs - echo "Re-applying Supabase manifests in namespace '$NAMESPACE'..." - apply_manifests + SUPABASE_HOME="$(detect_supabase_home)" || { err "ERROR: Supabase repo not found. Set SUPABASE_HOME or symlink ~/prole/supabase."; exit 1; } + convert_compose_to_k8s "$SUPABASE_HOME" + delete_k8s_resources + apply_k8s_resources + apply_db_migrations "$SUPABASE_HOME" ;; status) status diff --git a/install.py b/install.py index b376963..2951a0b 100755 --- a/install.py +++ b/install.py @@ -105,6 +105,330 @@ def _expand_path(val: str | None) -> str: return os.path.expandvars(os.path.expanduser(str(val))) +def _resolve_supabase_home(project_root: Path) -> Path | None: + env_home = (os.environ.get('SUPABASE_HOME') or '').strip() + if env_home: + try: + candidate = Path(_expand_path(env_home)) + if candidate.is_dir(): + return candidate + except Exception: + pass + + candidates = [ + Path.home() / 'prole' / 'supabase', + Path.home() / 'dev' / 'supabase', + project_root / 'supabase', + ] + prole_home = (os.environ.get('PROLE_HOME') or '').strip() + if prole_home: + try: + candidates.append(Path(_expand_path(prole_home)).parent / 'supabase') + except Exception: + pass + + for candidate in candidates: + try: + if candidate.is_dir(): + return candidate + except Exception: + continue + return None + + +def _collect_images_from_files(paths: list[Path]) -> set[str]: + images = set() + for path in paths: + if not path or not path.exists(): + continue + try: + for line in path.read_text().splitlines(): + m = re.match(r'^\s*image:\s*([^\s#]+)', line) + if not m: + continue + img = m.group(1).strip().strip('"').strip("'") + if not img: + continue + img = os.path.expandvars(img) + if '${' in img or '}' in img or '$' in img: + continue + images.add(img) + except Exception: + continue + return images + + +def _clean_yaml_value(raw: str) -> str: + if raw is None: + return '' + val = raw.strip() + if '#' in val: + val = val.split('#', 1)[0].strip() + if len(val) >= 2 and ((val[0] == val[-1]) and val.startswith(("'", '"'))): + val = val[1:-1] + return val.strip() + + +def _parse_yaml_scalar_values(path: Path, keys: set[str]) -> dict: + data = {} + if not path.exists(): + return data + try: + for raw in path.read_text().splitlines(): + line = raw.strip() + if not line or line.startswith('#') or line == '---': + continue + if line.startswith('- '): + continue + if ':' not in line: + continue + key, val = line.split(':', 1) + key = key.strip() + if key not in keys: + continue + clean_val = _clean_yaml_value(val) + if clean_val: + data[key] = clean_val + except Exception: + pass + return data + + +def _parse_internal_a_records(path: Path) -> tuple[str, dict, dict]: + domain = '' + ip_map: dict[str, str] = {} + fqdn_map: dict[str, str] = {} + if not path.exists(): + return domain, ip_map, fqdn_map + try: + lines = path.read_text().splitlines() + except Exception: + return domain, ip_map, fqdn_map + in_block = False + current: dict[str, str] = {} + records = [] + for raw in lines: + if not in_block: + s = raw.strip() + if s.startswith('prole_domain:'): + domain = _clean_yaml_value(s.split(':', 1)[1]) + if s.startswith('prole_internal_a_records:'): + in_block = True + continue + if raw and not raw.startswith((' ', '\t')): + break + s = raw.strip() + if not s: + continue + if s.startswith('- '): + if current: + records.append(current) + current = {} + s = s[2:].strip() + if ':' in s: + key, val = s.split(':', 1) + key = key.strip() + if key in ('fqdn', 'ipv4'): + current[key] = _clean_yaml_value(val) + if current: + records.append(current) + for rec in records: + fqdn = rec.get('fqdn') + ip = rec.get('ipv4') + if not fqdn or not ip: + continue + fqdn_map[fqdn] = ip + ip_map[fqdn] = ip + short = fqdn.split('.')[0] + if short and short not in ip_map: + ip_map[short] = ip + return domain, ip_map, fqdn_map + + +def _parse_ansible_inventory_hosts(path: Path) -> dict: + groups: dict[str, list[str]] = {} + if not path.exists(): + return groups + try: + current_group = None + for raw in path.read_text().splitlines(): + line = raw.strip() + if not line or line.startswith(('#', ';')): + continue + if line.startswith('[') and line.endswith(']'): + current_group = line[1:-1].strip() + groups.setdefault(current_group, []) + continue + if current_group is None: + continue + host = line.split()[0] + if host and host not in groups[current_group]: + groups[current_group].append(host) + except Exception: + pass + return groups + + +def _resolve_host_ip(host: str, domain: str, ip_map: dict) -> str: + if not host: + return '' + if host in ip_map: + return ip_map[host] + if domain: + if '.' not in host: + fqdn = f"{host}.{domain}" + if fqdn in ip_map: + return ip_map[fqdn] + else: + short = host.split('.')[0] + if short in ip_map: + return ip_map[short] + return '' + + +def _detect_ansible_topology(project_root: Path) -> dict: + prole_home = (os.environ.get('PROLE_HOME') or '').strip() + base_candidates = [] + if prole_home: + base_candidates.append(Path(_expand_path(prole_home))) + base_candidates.append(project_root) + infra_path = None + for base in base_candidates: + try: + candidate = base / 'infrastructure' + if candidate.is_dir(): + infra_path = candidate + break + except Exception: + continue + if infra_path is None: + return {} + inventory_path = infra_path / 'inventory' + if not inventory_path.is_dir(): + return {} + groups = _parse_ansible_inventory_hosts(inventory_path / 'hosts.ini') + vars_vals = _parse_yaml_scalar_values( + inventory_path / 'group_vars' / 'all' / 'vars.yml', + {'ad_dc_ip', 'prole_domain', 'kerberos_realm', 'krb5_realm', 'kerberos_kdc', 'krb5_kdc', 'kerberos_kdc_ip'} + ) + domain = vars_vals.get('prole_domain', '') + ad_dc_ip = vars_vals.get('ad_dc_ip', '') + explicit_realm = vars_vals.get('kerberos_realm') or vars_vals.get('krb5_realm') or '' + explicit_kdc = vars_vals.get('kerberos_kdc') or vars_vals.get('krb5_kdc') or vars_vals.get('kerberos_kdc_ip') or '' + if explicit_kdc and not ad_dc_ip: + ad_dc_ip = explicit_kdc + dns_domain, ip_map, fqdn_records = _parse_internal_a_records(inventory_path / 'group_vars' / 'all' / 'dns.yml') + if not domain and dns_domain: + domain = dns_domain + ad_vars = _parse_yaml_scalar_values( + inventory_path / 'group_vars' / 'ad_dc' / 'vars.yml', + {'samba_dns_server'} + ) + samba_dns_server = ad_vars.get('samba_dns_server', '') + ad_dc_host = '' + ad_dc_hosts = groups.get('ad_dc') or [] + if ad_dc_hosts: + ad_dc_host = ad_dc_hosts[0] + if samba_dns_server: + ad_dc_host = ad_dc_host or samba_dns_server + if not ad_dc_ip: + ad_dc_ip = _resolve_host_ip(samba_dns_server, domain, ip_map) + if not ad_dc_ip and ad_dc_host: + ad_dc_ip = _resolve_host_ip(ad_dc_host, domain, ip_map) + kdc_ip = explicit_kdc or ad_dc_ip or '' + realm = explicit_realm or (domain.upper() if domain else '') + + all_hosts = set() + for hosts in groups.values(): + all_hosts.update(hosts) + if ad_dc_host: + all_hosts.add(ad_dc_host) + host_ip_map = {} + unmapped_hosts = [] + for host in sorted(all_hosts): + ip = _resolve_host_ip(host, domain, ip_map) + if ip: + host_ip_map[host] = ip + else: + unmapped_hosts.append(host) + + topology = { + 'domain': domain, + 'realm': realm, + 'internal_records': fqdn_records, + 'ad_dc': { + 'host': ad_dc_host, + 'ip': ad_dc_ip, + }, + 'groups': groups, + 'hosts': host_ip_map, + 'unmapped_hosts': unmapped_hosts, + } + try: + topology_json = json.dumps(topology, separators=(',', ':')) + except Exception: + topology_json = '' + return { + 'infrastructure_path': str(infra_path), + 'inventory_path': str(inventory_path), + 'domain': domain, + 'realm': realm, + 'ad_dc_ip': ad_dc_ip, + 'ad_dc_host': ad_dc_host, + 'kdc_ip': kdc_ip, + 'groups': groups, + 'hosts': host_ip_map, + 'unmapped_hosts': unmapped_hosts, + 'topology': topology, + 'topology_json': topology_json, + } + + +def _format_ansible_topology_summary(info: dict) -> str: + if not info: + return '' + lines = [] + inv = info.get('inventory_path') or '' + if inv: + lines.append(f"Ansible inventory detected at {inv}.") + domain = info.get('domain') or '' + realm = info.get('realm') or '' + if domain or realm: + if domain and realm: + lines.append(f"Domain: {domain} (Realm: {realm})") + elif domain: + lines.append(f"Domain: {domain}") + else: + lines.append(f"Realm: {realm}") + ad_dc_host = info.get('ad_dc_host') or '' + ad_dc_ip = info.get('ad_dc_ip') or '' + if ad_dc_host or ad_dc_ip: + if ad_dc_host and ad_dc_ip: + lines.append(f"AD DC: {ad_dc_host} -> {ad_dc_ip}") + elif ad_dc_ip: + lines.append(f"AD DC IP: {ad_dc_ip}") + else: + lines.append(f"AD DC Host: {ad_dc_host}") + groups = info.get('groups') or {} + if groups: + group_bits = [] + for name in sorted(groups.keys()): + group_bits.append(f"{name}({len(groups[name])})") + lines.append("Groups: " + ", ".join(group_bits)) + hosts = info.get('hosts') or {} + if hosts: + host_items = sorted(hosts.items()) + preview = host_items[:8] + host_str = ", ".join([f"{h}={ip}" for h, ip in preview]) + if len(host_items) > len(preview): + host_str += f", +{len(host_items) - len(preview)} more" + lines.append("Hosts: " + host_str) + unmapped = info.get('unmapped_hosts') or [] + if unmapped: + lines.append(f"Hosts without IPs: {len(unmapped)}") + return "\n".join(lines) + + def _render_prole_cfg(inputs: dict, globals_to_save: dict, sections: dict, generated_at: str | None = None) -> str: content = [] content.append('; Prole Master Configuration File') @@ -164,7 +488,7 @@ class ProleController: pg_version_file = self.project_root / "conf" / "postgresql" / ".version" release_file = self.project_root / "prole-db" / ".version" pg_version = pg_version_file.read_text().strip() if pg_version_file.exists() else "17.7" - release = release_file.read_text().strip() if release_file.exists() else "41" + release = release_file.read_text().strip() if release_file.exists() else "43" if release.isdigit(): release = release.zfill(3) return f"{pg_version}-{release}" @@ -499,6 +823,12 @@ class ProleInstaller: self.kerberos_kdc = tk.StringVar() self.supabase_enabled = tk.BooleanVar(value=False) self.at_rest_encryption_enabled = tk.BooleanVar(value=False) + self.ansible_topology = {} + self.ansible_topology_summary = '' + try: + self._apply_ansible_topology_defaults() + except Exception: + pass # Create pages self.page_frames = {} @@ -684,6 +1014,43 @@ class ProleInstaller: except Exception: pass + def _apply_ansible_topology_defaults(self): + info = _detect_ansible_topology(PROJECT_ROOT) + if not info: + return + self.ansible_topology = info + self.ansible_topology_summary = _format_ansible_topology_summary(info) + try: + net = self.prole_cfg_data.get('Network', {}) + if info.get('topology_json'): + net['ANSIBLE_TOPOLOGY'] = info['topology_json'] + if info.get('inventory_path'): + net['ANSIBLE_INVENTORY'] = info['inventory_path'] + if info.get('infrastructure_path'): + net['ANSIBLE_INFRASTRUCTURE'] = info['infrastructure_path'] + if info.get('domain'): + net['ANSIBLE_DOMAIN'] = info['domain'] + if info.get('realm'): + net['ANSIBLE_REALM'] = info['realm'] + if info.get('ad_dc_host'): + net['AD_DC_HOST'] = info['ad_dc_host'] + if info.get('ad_dc_ip'): + net['AD_DC_IP'] = info['ad_dc_ip'] + if info.get('kdc_ip'): + net['KDC_ANSIBLE_DETECTED'] = info['kdc_ip'] + self.prole_cfg_data['Network'] = net + except Exception: + pass + + try: + if not self.kerberos_kdc.get().strip() and info.get('kdc_ip'): + self.kerberos_kdc.set(info['kdc_ip']) + self.kerberos_enabled.set(True) + if not self.kerberos_realm.get().strip() and info.get('realm'): + self.kerberos_realm.set(info['realm']) + except Exception: + pass + def _create_sidebar_nav(self): """Create the left-hand navigation menu.""" tk.Label(self.sidebar, text="INSTALLER", bg='#F5F5DC', fg='#8B8B7A', @@ -1901,12 +2268,12 @@ class ProleInstaller: font=('SF Pro Text', 18), anchor='ne') self._render_title('Network Configuration Scan', y=150) - self._render_paragraph("The network prole-scan will detect Kerberos services, Active Directory controllers, and other configuration details required for deployment.", y=210) + self._render_paragraph("The network prole-agent will detect Kerberos services, Active Directory controllers, and other configuration details required for deployment.", y=210) y = 280 - self.scan_status_var = tk.StringVar(value="Ready to prole-scan") + self.scan_status_var = tk.StringVar(value="Ready to prole-agent") # Ready to scan text (drawn on canvas) - status_item = ui.canvas_text(self, 48, y, "Ready to prole-scan", fill='black', font=('SF Pro Text', 12)) + status_item = ui.canvas_text(self, 48, y, "Ready to prole-agent", fill='black', font=('SF Pro Text', 12)) # Link variable to canvas text def update_status_text(*args): @@ -1919,6 +2286,10 @@ class ProleInstaller: # Standardized Console Output self.scan_results_console = self._create_console_output(y=320, title="Scan Output", width=880, height=380) self.scan_results_text = self.scan_results_console.text + + if getattr(self, 'ansible_topology_summary', ''): + self.scan_results_console.write(self.ansible_topology_summary + "\n") + self.scan_results_console.write("Ansible topology loaded; network scan can refine detection.\n") y = 740 @@ -1973,7 +2344,7 @@ class ProleInstaller: self._scan_running = True self.scan_results_console.clear() - self.scan_results_console.write("Initializing network scan using prole-net/prole-scan ...\n") + self.scan_results_console.write("Initializing network scan using prole-net/prole-agent ...\n") self.scan_status_var.set("Scanning...") # Start animation @@ -1983,7 +2354,7 @@ class ProleInstaller: def worker(): try: # Use the new scan binary - scan_binary = get_resource_path("prole-net/prole-scan") + scan_binary = get_resource_path("prole-net/prole-agent") if not scan_binary.exists(): self.safe_after(lambda: self.scan_results_console.write(f"Scan binary not found at {scan_binary}\n")) self.safe_after(lambda: self.scan_status_var.set("Scan failed")) @@ -1998,7 +2369,7 @@ class ProleInstaller: # Run scan from writable directory with 10s timeout and summary analysis # Set bufsize=0 for truly unbuffered binary stream reading - # Ensure PYTHONUNBUFFERED=1 is set in case prole-scan is/uses Python + # Ensure PYTHONUNBUFFERED=1 is set in case prole-agent is/uses Python env = os.environ.copy() env['PYTHONUNBUFFERED'] = '1' process = subprocess.Popen([str(scan_binary), "-t", "10", "-s"], @@ -3189,6 +3560,7 @@ class ProleInstaller: ('CloudNative-PG', 'init_cloudnative_pg.sh'), ] if self.kerberos_enabled.get(): + scripts.append(('Authority', 'init_authority.sh')) scripts.append(('Kerberos Realm', 'init_kerberos.sh')) scripts.extend([ ('Prole DB', 'init_prole-db.sh'), @@ -3548,7 +3920,49 @@ class ProleInstaller: else: self.script_consoles["init_cloudnative_pg.sh"].write("Skipping CloudNative-PG initialization because previous steps failed.\n") - # 4. init_kerberos.sh initialize (optional) + # 4. init_authority.sh start (optional) + if overall_success and self.kerberos_enabled.get(): + script = "init_authority.sh" + if script in self.script_consoles: + _select_tab(script) + self.script_consoles[script].clear() + self.script_consoles[script].write(f"Running {script} start...\n") + + log_path = _log_path_for(script) + self._record_install_log(log_path) + try: + log_fp = log_path.open('a', encoding='utf-8') + except Exception: + log_fp = None + + def _auth_line(line): + self.script_consoles[script].write(line) + if log_fp: + try: + log_fp.write(line) + log_fp.flush() + except Exception: + pass + + rc_auth = self.controller.run_script( + script, + args=['start'], + env=env, + on_line=_auth_line + ) + if log_fp: + try: + log_fp.close() + except Exception: + pass + + if rc_auth != 0: + self.script_consoles[script].write(f"\nERROR: {script} start failed with code {rc_auth}\n") + overall_success = False + else: + self.script_consoles[script].write(f"\n{script} completed successfully.\n") + + # 5. init_kerberos.sh initialize (optional) if overall_success and self.kerberos_enabled.get(): script = "init_kerberos.sh" if script in self.script_consoles: @@ -3596,7 +4010,7 @@ class ProleInstaller: elif not self.kerberos_enabled.get(): if "init_kerberos.sh" in self.script_consoles: self.script_consoles["init_kerberos.sh"].write("Kerberos auth disabled; skipping init_kerberos.sh.\n") - # 5. init_prole-db.sh start + # 6. init_prole-db.sh start if overall_success: script = "init_prole-db.sh" _select_tab(script) @@ -3639,7 +4053,7 @@ class ProleInstaller: else: self.script_consoles["init_prole-db.sh"].write("Skipping Prole DB initialization because previous steps failed.\n") - # 6. init_prole-db-backup.sh start + # 7. init_prole-db-backup.sh start if overall_success: script = "init_prole-db-backup.sh" _select_tab(script) @@ -3682,7 +4096,7 @@ class ProleInstaller: else: self.script_consoles["init_prole-db-backup.sh"].write("Skipping Prole DB backup because previous steps failed.\n") - # 7. init_supabase.sh start (optional) + # 8. init_supabase.sh start if overall_success: script = "init_supabase.sh" _select_tab(script) @@ -3718,7 +4132,7 @@ class ProleInstaller: pass if rc_sb != 0: self.script_consoles[script].write(f"\nERROR: {script} start failed with code {rc_sb}\n") - overall_success = False + # overall_success = False # Keep as False if you want to block port forwards on failure else: self.script_consoles[script].write(f"\n{script} completed successfully.\n") else: @@ -3726,7 +4140,7 @@ class ProleInstaller: else: self.script_consoles["init_supabase.sh"].write("Skipping Supabase because previous steps failed.\n") - # 8. init_port_forwards.sh start + # 9. init_port_forwards.sh start if overall_success: script = "init_port_forwards.sh" _select_tab(script) @@ -6585,16 +6999,16 @@ esac base_dir = PROJECT_ROOT / rel_dir if not base_dir.exists(): continue - for path in base_dir.glob('*.yaml'): - try: - for line in path.read_text().splitlines(): - m = re.match(r'^\s*image:\s*([^\s#]+)', line) - if m: - img = m.group(1).strip().strip('"').strip("'") - if img: - images.add(img) - except Exception: - continue + images.update(_collect_images_from_files(list(base_dir.glob('*.yaml')))) + + if include_supabase: + supa_home = _resolve_supabase_home(PROJECT_ROOT) + if supa_home: + docker_dir = supa_home / 'docker' + compose_files = [docker_dir / 'docker-compose.yml'] + if os.environ.get('SUPABASE_USE_DEV_COMPOSE') == '1': + compose_files.append(docker_dir / 'dev' / 'docker-compose.dev.yml') + images.update(_collect_images_from_files(compose_files)) if not include_supabase: images = {img for img in images if 'supabase' not in img} @@ -6757,7 +7171,7 @@ esac if not manifest_path.exists(): continue content = manifest_path.read_text() - # Update imageName: prole-db:17.7-041 + # Update imageName: prole-db:17.7-043 new_content = re.sub(r'imageName:\s*.*', f'imageName: {image_tag}', content) # Update Kerberos realm in manifest if enabled @@ -7460,16 +7874,16 @@ class ProleSilentInstaller: base_dir = self.project_root / rel_dir if not base_dir.exists(): continue - for path in base_dir.glob('*.yaml'): - try: - for line in path.read_text().splitlines(): - m = re.match(r'^\s*image:\s*([^\s#]+)', line) - if m: - img = m.group(1).strip().strip('"').strip("'") - if img: - images.add(img) - except Exception: - continue + images.update(_collect_images_from_files(list(base_dir.glob('*.yaml')))) + + if include_supabase: + supa_home = _resolve_supabase_home(self.project_root) + if supa_home: + docker_dir = supa_home / 'docker' + compose_files = [docker_dir / 'docker-compose.yml'] + if os.environ.get('SUPABASE_USE_DEV_COMPOSE') == '1': + compose_files.append(docker_dir / 'dev' / 'docker-compose.dev.yml') + images.update(_collect_images_from_files(compose_files)) if not include_supabase: images = {img for img in images if 'supabase' not in img} @@ -7531,11 +7945,52 @@ class ProleSilentInstaller: defaults = self._default_inputs() loaded = self._load_inputs_from_cfg() self.inputs = {**defaults, **loaded} + try: + self._apply_ansible_defaults(set(loaded.keys())) + except Exception: + pass for k in ('env_setup.PROLE_HOME', 'env_setup.PROLE_CONF', 'env_setup.PROLE_DATA', 'env_setup.PROLE_LOGS', 'env_setup.PROLE_SERVICE'): if k in self.inputs: self.inputs[k] = _expand_path(self.inputs[k]) + def _apply_ansible_defaults(self, loaded_keys: set[str] | None = None): + info = _detect_ansible_topology(self.project_root) + if not info: + return + loaded_keys = loaded_keys or set() + try: + net = self.prole_cfg_data.get('Network', {}) + if info.get('topology_json'): + net['ANSIBLE_TOPOLOGY'] = info['topology_json'] + if info.get('inventory_path'): + net['ANSIBLE_INVENTORY'] = info['inventory_path'] + if info.get('infrastructure_path'): + net['ANSIBLE_INFRASTRUCTURE'] = info['infrastructure_path'] + if info.get('domain'): + net['ANSIBLE_DOMAIN'] = info['domain'] + if info.get('realm'): + net['ANSIBLE_REALM'] = info['realm'] + if info.get('ad_dc_host'): + net['AD_DC_HOST'] = info['ad_dc_host'] + if info.get('ad_dc_ip'): + net['AD_DC_IP'] = info['ad_dc_ip'] + if info.get('kdc_ip'): + net['KDC_ANSIBLE_DETECTED'] = info['kdc_ip'] + self.prole_cfg_data['Network'] = net + except Exception: + pass + + if 'kerberos_config.kdc' not in loaded_keys: + if not self._get_input('kerberos_config.kdc', '').strip() and info.get('kdc_ip'): + self.inputs['kerberos_config.kdc'] = info['kdc_ip'] + if 'kerberos_config.realm' not in loaded_keys: + if not self._get_input('kerberos_config.realm', '').strip() and info.get('realm'): + self.inputs['kerberos_config.realm'] = info['realm'] + if 'kerberos_config.enabled' not in loaded_keys: + if info.get('kdc_ip') and not self._get_input_bool('kerberos_config.enabled', False): + self.inputs['kerberos_config.enabled'] = _bool_str(True) + def _write_cfg(self): globals_to_save = { 'PROLE_HOME': self._get_input('env_setup.PROLE_HOME', ''), @@ -7608,8 +8063,8 @@ class ProleSilentInstaller: if not self._get_input_bool('network_scan.run', DEFAULT_ACTION_FLAGS.get('network_scan.run', True)): self.log("[SKIP] Network scan disabled.") return - self.log("==> Network scan (prole-scan)") - scan_binary = get_resource_path("prole-net/prole-scan") + self.log("==> Network scan (prole-agent)") + scan_binary = get_resource_path("prole-net/prole-agent") if not scan_binary.exists(): self.err(f"[ERROR] Scan binary not found at {scan_binary}") return diff --git a/installer.spec b/installer.spec index 5fe1e70..7ff2853 100644 --- a/installer.spec +++ b/installer.spec @@ -22,7 +22,7 @@ datas = [ # Binaries to include (with execute permissions) binaries = [ - ('prole-net/prole-scan', 'prole-net'), + ('prole-net/prole-agent', 'prole-net'), ] # Hidden imports diff --git a/k8s/openbao/kerberos-configmap.yaml b/k8s/openbao/kerberos-configmap.yaml index 2037587..71b4258 100644 --- a/k8s/openbao/kerberos-configmap.yaml +++ b/k8s/openbao/kerberos-configmap.yaml @@ -12,8 +12,8 @@ data: [realms] PROLE.ORG = { - kdc = 10.0.0.3 - admin_server = 10.0.0.3 + kdc = kdc.prole.org + admin_server = kdc.prole.org } [domain_realm] diff --git a/k8s/prole/prole-db-recovery.yaml.tpl b/k8s/prole/prole-db-recovery.yaml.tpl index e4d9570..69e8553 100644 --- a/k8s/prole/prole-db-recovery.yaml.tpl +++ b/k8s/prole/prole-db-recovery.yaml.tpl @@ -5,7 +5,7 @@ metadata: name: prole-db spec: instances: 3 - imageName: prole-db:17.7-041 + imageName: prole-db:17.7-053 postgresUID: 100 postgresGID: 101 maxSyncReplicas: 1 diff --git a/k8s/prole/prole-db.yaml b/k8s/prole/prole-db.yaml index b0ed049..f604b56 100644 --- a/k8s/prole/prole-db.yaml +++ b/k8s/prole/prole-db.yaml @@ -4,7 +4,7 @@ metadata: name: prole-db spec: instances: 3 - imageName: prole-db:17.7-041 + imageName: prole-db:17.7-053 postgresUID: 100 postgresGID: 101 maxSyncReplicas: 1 diff --git a/prole-db/.version b/prole-db/.version index 87523dd..59343b0 100644 --- a/prole-db/.version +++ b/prole-db/.version @@ -1 +1 @@ -41 +53 diff --git a/prole-db/Dockerfile b/prole-db/Dockerfile index 5f79603..cc4b243 100644 --- a/prole-db/Dockerfile +++ b/prole-db/Dockerfile @@ -62,16 +62,6 @@ RUN set -eux; \ DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ cmake \ libssl-dev \ - libkrb5-dev \ - krb5-user \ - sssd \ - sssd-krb5 \ - sssd-krb5-common \ - sssd-tools \ - libpam-sss \ - libnss-sss \ - oddjob \ - oddjob-mkhomedir \ libgdal-dev \ libproj-dev \ libgeos-dev \ @@ -84,7 +74,7 @@ RUN set -eux; \ # Copy and install pg_prolelog from local .deb package ARG TARGETARCH -COPY pg_prolelog/percona-postgresql-17-prolelog_1.0-1_${TARGETARCH:-amd64}.deb /tmp/pg_prolelog.deb +COPY pg_prolelog/percona-postgresql-17-pg-prolelog_1.0-1_${TARGETARCH:-amd64}.deb /tmp/pg_prolelog.deb RUN set -eux; \ apt-get update; \ apt-get install -y --no-install-recommends /tmp/pg_prolelog.deb; \ @@ -99,6 +89,7 @@ RUN set -eux; \ percona-postgresql-17-pgvector \ percona-postgresql-17-postgis-3 \ percona-postgresql-contrib-17 \ + percona-postgresql-contrib \ freetds-dev \ ; \ rm -rf /var/lib/apt/lists/* @@ -123,7 +114,6 @@ RUN set -eux; \ echo " CREATE EXTENSION IF NOT EXISTS tds_fdw;" >> /docker-entrypoint-initdb.d/20_create_extensions.sh; \ echo " CREATE EXTENSION IF NOT EXISTS postgis;" >> /docker-entrypoint-initdb.d/20_create_extensions.sh; \ echo " CREATE EXTENSION IF NOT EXISTS postgis_topology;" >> /docker-entrypoint-initdb.d/20_create_extensions.sh; \ - echo " CREATE EXTENSION IF NOT EXISTS postgis_tiger_geocoder;" >> /docker-entrypoint-initdb.d/20_create_extensions.sh; \ echo " CREATE EXTENSION IF NOT EXISTS pg_prolelog;" >> /docker-entrypoint-initdb.d/20_create_extensions.sh; \ echo "EOSQL" >> /docker-entrypoint-initdb.d/20_create_extensions.sh; \ chmod +x /docker-entrypoint-initdb.d/20_create_extensions.sh diff --git a/prole-db/pg_prolelog/percona-postgresql-17-pg-prolelog_1.0-1_arm64.deb b/prole-db/pg_prolelog/percona-postgresql-17-pg-prolelog_1.0-1_arm64.deb new file mode 100644 index 0000000..b73cad9 Binary files /dev/null and b/prole-db/pg_prolelog/percona-postgresql-17-pg-prolelog_1.0-1_arm64.deb differ diff --git a/prole-db/pg_prolelog/percona-postgresql-17-prolelog_1.0-1_amd64.deb b/prole-db/pg_prolelog/percona-postgresql-17-prolelog_1.0-1_amd64.deb deleted file mode 100644 index e15aef0..0000000 Binary files a/prole-db/pg_prolelog/percona-postgresql-17-prolelog_1.0-1_amd64.deb and /dev/null differ diff --git a/prole-db/pg_prolelog/percona-postgresql-17-prolelog_1.0-1_arm64.deb b/prole-db/pg_prolelog/percona-postgresql-17-prolelog_1.0-1_arm64.deb deleted file mode 100644 index 3515dee..0000000 Binary files a/prole-db/pg_prolelog/percona-postgresql-17-prolelog_1.0-1_arm64.deb and /dev/null differ diff --git a/prole-net/prole-scan b/prole-net/prole-agent similarity index 94% rename from prole-net/prole-scan rename to prole-net/prole-agent index 11e1fb0..a425b17 100755 Binary files a/prole-net/prole-scan and b/prole-net/prole-agent differ diff --git a/prole_requirements.txt b/prole_requirements.txt index e1381c1..4c456e8 100644 --- a/prole_requirements.txt +++ b/prole_requirements.txt @@ -41,6 +41,7 @@ httpcore==1.0.9 httpx==0.28.1 idna==3.10 importlib_metadata==8.7.0 +langchain isodate==0.6.1 Jinja2==3.1.6 jiter==0.10.0 diff --git a/requirements.txt b/requirements.txt index ff77f02..3ea55dd 100644 --- a/requirements.txt +++ b/requirements.txt @@ -8,6 +8,7 @@ cryptography ghp-import griffe idna +langchain Jinja2 jsonpatch jsonpointer diff --git a/scripts/docker-root-prole-db.sh b/scripts/docker-root-prole-db.sh index ebb069e..a8ac278 100755 --- a/scripts/docker-root-prole-db.sh +++ b/scripts/docker-root-prole-db.sh @@ -7,7 +7,7 @@ export POSTGRES_PASSWORD=$(cat $PROLE_HOME/postgres-password.txt) PG_VERSION=$(cat "$PROLE_HOME/conf/postgresql/.version" 2>/dev/null | tr -d '[:space:]') RELEASE=$(cat "$PROLE_HOME/prole-db/.version" 2>/dev/null | tr -d '[:space:]') if [[ "$RELEASE" =~ ^[0-9]+$ ]]; then RELEASE=$(printf "%03d" "$RELEASE"); fi -IMAGE="prole-db:${PG_VERSION:-17.7}-${RELEASE:-041}" +IMAGE="prole-db:${PG_VERSION:-17.7}-${RELEASE:-043}" docker run --rm -it -u root \ --hostname=prole-db-00n \ diff --git a/scripts/docker-run-prole-db.sh b/scripts/docker-run-prole-db.sh index 3390be3..7492ba0 100755 --- a/scripts/docker-run-prole-db.sh +++ b/scripts/docker-run-prole-db.sh @@ -1,3 +1,4 @@ +set -eux # export POSTGRES_PASSWORD=$(openssl rand -base64 32) # echo "$POSTGRES_PASSWORD" > postgres-password.txt export POSTGRES_PASSWORD=$(cat postgres-password.txt) @@ -6,7 +7,7 @@ PROLE_HOME=~/dev/prole PG_VERSION=$(cat "$PROLE_HOME/conf/postgresql/.version" 2>/dev/null | tr -d '[:space:]') RELEASE=$(cat "$PROLE_HOME/prole-db/.version" 2>/dev/null | tr -d '[:space:]') if [[ "$RELEASE" =~ ^[0-9]+$ ]]; then RELEASE=$(printf "%03d" "$RELEASE"); fi -IMAGE="prole-db:${PG_VERSION:-17.7}-${RELEASE:-041}" +IMAGE="prole-db:${PG_VERSION:-17.7}-${RELEASE:-043}" docker run --rm -it --hostname=prole-db-00n \ -e POSTGRES_PASSWORD="$POSTGRES_PASSWORD" \ diff --git a/scripts/generate_spec.py b/scripts/generate_spec.py index b888500..6709d58 100755 --- a/scripts/generate_spec.py +++ b/scripts/generate_spec.py @@ -27,7 +27,7 @@ datas = [ # Binaries to include (with execute permissions) binaries = [ - ('prole-net/prole-scan', 'prole-net'), + ('prole-net/prole-agent', 'prole-net'), ] # Hidden imports diff --git a/tests/final_comprehensive_test.sh b/tests/final_comprehensive_test.sh index b63f4a9..e80c512 100755 --- a/tests/final_comprehensive_test.sh +++ b/tests/final_comprehensive_test.sh @@ -40,7 +40,7 @@ for img in ['img/proleIcon.png', 'img/proleLogo.png', 'img/proleLogoSepia.png']: sys.exit(1) " -run_test "Binary executable" test -x prole-net/prole-scan +run_test "Binary executable" test -x prole-net/prole-agent run_test "App bundle" test -d "prole-app/dist/Prole Tools.app" # Build system tests @@ -51,7 +51,7 @@ run_test "Icon conversion" make build/prole.icns echo -n "[$((TESTS_TOTAL + 1))] Spec includes all resources... " TESTS_TOTAL=$((TESTS_TOTAL + 1)) if grep -q "prole-app/dist/Prole Tools.app" installer.spec && \ - grep -q "prole-net/prole-scan" installer.spec && \ + grep -q "prole-net/prole-agent" installer.spec && \ grep -q "img.*img" installer.spec; then echo "✓" TESTS_PASSED=$((TESTS_PASSED + 1)) @@ -73,13 +73,13 @@ if [ $TESTS_PASSED -eq $TESTS_TOTAL ]; then echo " ✓ Ncurses terminal interface" echo " ✓ Automatic display detection" echo " ✓ Image resources embedded" - echo " ✓ Binary executables embedded (prole-scan)" + echo " ✓ Binary executables embedded (prole-agent)" echo " ✓ App bundles embedded (Prole Tools.app)" echo " ✓ Build system configured" echo "" echo "Embedded resources:" echo " • Images: ~11 MB" - echo " • prole-scan: 6.8 MB (universal)" + echo " • prole-agent: 6.8 MB (universal)" echo " • Prole Tools.app: ~12 MB" echo " • Expected final size: ~50-100 MB" echo "" diff --git a/tests/final_test.sh b/tests/final_test.sh index 0219804..5f8338a 100755 --- a/tests/final_test.sh +++ b/tests/final_test.sh @@ -47,8 +47,8 @@ test_item "proleLogoSepia.png" test -f img/proleLogoSepia.png # Binary Resources echo "" echo "BINARY RESOURCES" -test_item "prole-scan exists" test -f prole-net/prole-scan -test_item "prole-scan executable" test -x prole-net/prole-scan +test_item "prole-agent exists" test -f prole-net/prole-agent +test_item "prole-agent executable" test -x prole-net/prole-agent # App Bundle echo "" @@ -67,7 +67,7 @@ echo "SPEC FILE VERIFICATION" python3 scripts/generate_spec.py > /dev/null 2>&1 test_item "Spec includes img" grep -q "img" installer.spec test_item "Spec includes prole-db" grep -q "prole-db" installer.spec -test_item "Spec includes prole-scan" grep -q "prole-scan" installer.spec +test_item "Spec includes prole-agent" grep -q "prole-agent" installer.spec test_item "Spec includes Prole Tools" grep -q "Prole Tools" installer.spec # Resource Path Tests @@ -85,7 +85,7 @@ from pathlib import Path import sys sys.path.insert(0, str(Path.cwd())) from install import get_resource_path -assert get_resource_path('prole-net/prole-scan').exists() +assert get_resource_path('prole-net/prole-agent').exists() " test_item "Docker context works" python3 -c " from pathlib import Path @@ -113,14 +113,14 @@ if [ $TESTS_PASSED -eq $TESTS_TOTAL ]; then echo " ✓ Automatic display detection" echo " ✓ Static universal binary build system" echo " ✓ Image resources embedded" - echo " ✓ prole-scan binary embedded" + echo " ✓ prole-agent binary embedded" echo " ✓ Prole Tools.app embedded" echo " ✓ Docker build context embedded" echo " ✓ Docker build uses ~/.prole/build (writable)" echo "" echo "PACKAGE WILL INCLUDE:" echo " • Images (~11 MB)" - echo " • prole-scan (6.8 MB)" + echo " • prole-agent (6.8 MB)" echo " • Prole Tools.app (~12 MB)" echo " • prole-db build context (~1-5 MB)" echo " • Python runtime + deps (~30-50 MB)" diff --git a/tests/test_all_prole_home_fixes.sh b/tests/test_all_prole_home_fixes.sh index d88d0de..b422862 100755 --- a/tests/test_all_prole_home_fixes.sh +++ b/tests/test_all_prole_home_fixes.sh @@ -36,7 +36,7 @@ test_item "Build directory writable" python3 -c "from pathlib import Path; f = P echo "" echo "NETWORK SCAN FIX" -test_item "prole-scan binary exists" test -x prole-net/prole-scan +test_item "prole-agent binary exists" test -x prole-net/prole-agent test_item "Scan directory creation" python3 -c "from pathlib import Path; d = Path.home() / '.prole' / 'scan'; d.mkdir(parents=True, exist_ok=True); assert d.exists()" test_item "Scan directory writable" python3 -c "from pathlib import Path; f = Path.home() / '.prole' / 'scan' / 'test.txt'; f.write_text('test'); f.unlink()" @@ -51,7 +51,7 @@ echo "" echo "SPEC FILE VERIFICATION" python3 scripts/generate_spec.py > /dev/null 2>&1 test_item "Spec includes prole-db" grep -q "prole-db" installer.spec -test_item "Spec includes prole-scan" grep -q "prole-scan" installer.spec +test_item "Spec includes prole-agent" grep -q "prole-agent" installer.spec # Cleanup echo "" diff --git a/tests/test_embedded_resources.sh b/tests/test_embedded_resources.sh index 21757cc..6cac841 100755 --- a/tests/test_embedded_resources.sh +++ b/tests/test_embedded_resources.sh @@ -28,10 +28,10 @@ if [ $? -ne 0 ]; then exit 1; fi # Test 2: Binaries echo "2. Testing binary resources..." -if [ -f "prole-net/prole-scan" ] && [ -x "prole-net/prole-scan" ]; then - echo " ✓ prole-scan binary present and executable" +if [ -f "prole-net/prole-agent" ] && [ -x "prole-net/prole-agent" ]; then + echo " ✓ prole-agent binary present and executable" else - echo " ✗ prole-scan missing or not executable" + echo " ✗ prole-agent missing or not executable" exit 1 fi @@ -57,7 +57,7 @@ fi # Test 5: Verify spec includes everything echo "5. Verifying spec includes all resources..." grep -q "prole-app/dist/Prole Tools.app" installer.spec && \ -grep -q "prole-net/prole-scan" installer.spec && \ +grep -q "prole-net/prole-agent" installer.spec && \ grep -q "img" installer.spec if [ $? -eq 0 ]; then echo " ✓ All resources included in spec" @@ -68,8 +68,8 @@ fi # Test 6: Check binary size echo "6. Checking binary sizes..." -SCAN_SIZE=$(ls -lh prole-net/prole-scan | awk '{print $5}') -echo " prole-scan: $SCAN_SIZE (universal binary)" +SCAN_SIZE=$(ls -lh prole-net/prole-agent | awk '{print $5}') +echo " prole-agent: $SCAN_SIZE (universal binary)" APP_SIZE=$(du -sh "prole-app/dist/Prole Tools.app" | awk '{print $1}') echo " Prole Tools.app: $APP_SIZE" @@ -78,7 +78,7 @@ echo "=== All Embedded Resource Tests Passed ===" echo "" echo "Resources ready for packaging:" echo " ✓ Images (proleIcon.png, proleLogo.png, proleLogoSepia.png)" -echo " ✓ Binary (prole-net/prole-scan)" +echo " ✓ Binary (prole-net/prole-agent)" echo " ✓ App bundle (prole-app/dist/Prole Tools.app)" echo "" echo "Build with: make package" diff --git a/tests/test_network_scan_fix.sh b/tests/test_network_scan_fix.sh index f1a5333..9f21166 100755 --- a/tests/test_network_scan_fix.sh +++ b/tests/test_network_scan_fix.sh @@ -36,28 +36,28 @@ except Exception as e: PYTEST if [ $? -ne 0 ]; then exit 1; fi -# Test 3: prole-scan binary exists -echo "3. Testing prole-scan binary..." -if [ -x "prole-net/prole-scan" ]; then - echo " ✓ prole-scan binary exists and is executable" +# Test 3: prole-agent binary exists +echo "3. Testing prole-agent binary..." +if [ -x "prole-net/prole-agent" ]; then + echo " ✓ prole-agent binary exists and is executable" else - echo " ✗ prole-scan binary missing or not executable" + echo " ✗ prole-agent binary missing or not executable" exit 1 fi -# Test 4: prole-scan can run from scan directory -echo "4. Testing prole-scan runs from writable directory..." +# Test 4: prole-agent can run from scan directory +echo "4. Testing prole-agent runs from writable directory..." mkdir -p /tmp/scan-test cd /tmp/scan-test -/Users/chrisfu/dev/prole/prole-net/prole-scan 2>&1 & +/Users/chrisfu/dev/prole/prole-net/prole-agent 2>&1 & SCAN_PID=$! sleep 2 if ps -p $SCAN_PID > /dev/null 2>&1; then - echo " ✓ prole-scan runs successfully" + echo " ✓ prole-agent runs successfully" kill $SCAN_PID 2>/dev/null wait 2>/dev/null else - echo " ✗ prole-scan failed to start" + echo " ✗ prole-agent failed to start" exit 1 fi cd /Users/chrisfu/dev/prole @@ -85,5 +85,5 @@ echo " • From source: Uses writable cwd" echo " • From package: Uses ~/.prole/scan" echo "" echo "Scan directory: ~/.prole/scan" -echo "Purpose: Writable working directory for prole-scan" +echo "Purpose: Writable working directory for prole-agent" echo ""