mirror of
https://github.com/dredx/prole.git
synced 2026-09-23 12:03:59 +00:00
316 lines
12 KiB
Markdown
316 lines
12 KiB
Markdown
<div align="center">
|
|
<a href="https://svc.knoe.org"><pre>
|
|
# ###########################
|
|
# ╭──────────────────────╮ #
|
|
# │ _ │ #
|
|
# │ | _/ _ | _ |_ │ #
|
|
# │ |/ | \ | ( |_) │ #
|
|
# │ │ #
|
|
# ╰──────────────────────╯ #
|
|
# ###########################
|
|
</pre></a>
|
|
</div>
|
|
|
|
# Knoe
|
|
|
|
**Knoe makes it practical to run a Supabase-style platform across air-gapped, edge, and cloud environments.**
|
|
|
|
It packages the core building blocks needed for a modern internal developer platform around PostgreSQL, object storage, secrets, auth, observability, and Kubernetes-native operations — with a bias toward simple deployment, shard-based scale-out, and deterministic ingestion.
|
|
|
|
---
|
|
|
|
## Platform badges
|
|
|
|
<div align="center">
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</div>
|
|
|
|
---
|
|
|
|
## What Knoe is
|
|
|
|
Knoe is an infrastructure stack for running a Supabase-like data and application platform across constrained, sovereign, and cloud environments, including:
|
|
|
|
- air-gapped networks
|
|
- edge and small-cluster deployments (K3s, K3d)
|
|
- Google Cloud Platform (GCP) deployments
|
|
- sovereign or private data environments
|
|
- disconnected labs and field systems
|
|
- internal platforms where cloud dependencies are undesirable
|
|
|
|
The design goal is not to mimic Supabase branding or every managed feature exactly.
|
|
|
|
The design goal is to provide the **useful substrate** people actually want:
|
|
|
|
- PostgreSQL as the center of gravity
|
|
- object storage
|
|
- auth integration
|
|
- secret management
|
|
- ingress and service exposure
|
|
- observability
|
|
- reproducible Kubernetes deployment
|
|
- application and worker services around the database
|
|
- ingestion patterns that are safe, resumable, and idempotent
|
|
|
|
---
|
|
|
|
## Core idea
|
|
|
|
**Knoe can run Supabase-style workloads in an air gap, on the edge, or in the cloud.**
|
|
|
|
That means:
|
|
|
|
- PostgreSQL remains the system of record, managed as `knoe-db` within the `knoe-system` namespace
|
|
- services are deployed on Kubernetes (K3d locally, K3s on-prem, GKE on GCP)
|
|
- auth can come from local identity systems or cloud OIDC where available
|
|
- object storage stays inside the environment
|
|
- secrets stay inside the environment
|
|
- ingestion does not depend on live internet services
|
|
- the stack can be deployed in shards for locality, resilience, and operational simplicity
|
|
|
|
---
|
|
|
|
## Main components
|
|
|
|
### Application layer
|
|
- **Next.js app** for the user-facing platform
|
|
- **knoe-agent worker** for async and background execution
|
|
|
|
### Data layer
|
|
- **knoe-db** — CloudNativePG-managed PostgreSQL cluster in the `knoe-system` namespace
|
|
- **PostgreSQL** as the primary relational store
|
|
- **append-only fact tables** for durable ingestion and replay-friendly modeling
|
|
- **PostgreSQL COPY** for efficient bulk ingest
|
|
|
|
### Deployed add-ons
|
|
- **Supabase** — self-hosted Supabase stack for auth, realtime, storage, and API gateway
|
|
- **ArgoCD** — GitOps continuous delivery for the `knoe-system` cluster
|
|
- **Gitea** — lightweight self-hosted Git service
|
|
- **GitLab** — full-featured self-hosted DevOps platform
|
|
|
|
### Storage and secrets
|
|
- **Garage** for S3-compatible object storage
|
|
- **OpenBao** for secret storage and secret distribution
|
|
|
|
### Identity
|
|
- **Dev auth:** Samba AD + IdP
|
|
- **Cloud auth:** Google Workspace OIDC
|
|
|
|
### Observability
|
|
- **Vector** for log and event shipping
|
|
- **Prometheus / Grafana** for metrics and dashboards
|
|
|
|
### Platform operations
|
|
- **K3d** — local development clusters
|
|
- **K3s** — on-prem lightweight Kubernetes
|
|
- **GCP / GKE** — cloud-hosted cluster tier
|
|
- **Helm**
|
|
- **ArgoCD**
|
|
- **Ansible**
|
|
- **OpenTofu**
|
|
|
|
---
|
|
|
|
## Ingestion model
|
|
|
|
Knoe is designed around deterministic, replayable ingestion.
|
|
|
|
Key patterns:
|
|
|
|
- **SQLite delta cartridges** as portable input units
|
|
- **append-only fact tables** for auditability and recovery
|
|
- **PostgreSQL COPY** for high-throughput loading
|
|
- **queue-driven workers** using `SKIP LOCKED`
|
|
- **idempotent ingestion** so retries are safe
|
|
- **shard-based deployment** so data movement stays deliberate and bounded
|
|
|
|
This makes the platform well suited to environments where data may arrive in batches, be transferred physically, or need careful replay and provenance.
|
|
|
|
---
|
|
|
|
## Architecture at a glance
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
subgraph Clients
|
|
U["Users / Operators"]
|
|
end
|
|
|
|
subgraph knoe-system ["knoe-system (Kubernetes)"]
|
|
A["Next.js App"]
|
|
W["knoe-agent worker"]
|
|
SB["Supabase\n(auth · realtime · storage · API)"]
|
|
ARGO["ArgoCD\n(GitOps)"]
|
|
GITEA["Gitea\n(Git)"]
|
|
GITLAB["GitLab\n(DevOps)"]
|
|
PG["knoe-db\n(CloudNativePG / PostgreSQL)"]
|
|
OBJ["Garage\n(Object Store)"]
|
|
SEC["OpenBao\n(Secrets)"]
|
|
OBS["Vector / Prometheus / Grafana\n(Observability)"]
|
|
ING["Traefik / Kong\n(Ingress)"]
|
|
end
|
|
|
|
subgraph Infra ["Infrastructure"]
|
|
K3D["K3d\n(local dev)"]
|
|
K3S["K3s\n(on-prem)"]
|
|
GCP["GCP / GKE\n(cloud)"]
|
|
end
|
|
|
|
subgraph Identity
|
|
AUTH["Samba AD / Google Workspace OIDC"]
|
|
end
|
|
|
|
U --> ING
|
|
ING --> A
|
|
ING --> SB
|
|
ING --> ARGO
|
|
ING --> GITEA
|
|
ING --> GITLAB
|
|
|
|
A --> PG
|
|
A --> OBJ
|
|
A --> SEC
|
|
A --> AUTH
|
|
W --> PG
|
|
W --> OBJ
|
|
W --> SEC
|
|
SB --> PG
|
|
SB --> OBJ
|
|
SB --> AUTH
|
|
|
|
ARGO --> A
|
|
ARGO --> W
|
|
ARGO --> SB
|
|
ARGO --> PG
|
|
ARGO --> OBJ
|
|
ARGO --> SEC
|
|
|
|
GITEA --> ARGO
|
|
GITLAB --> ARGO
|
|
|
|
K3D & K3S & GCP --> |hosts| knoe-system
|
|
OBS -.-> PG
|
|
OBS -.-> A
|
|
OBS -.-> W
|
|
```
|
|
|
|
---
|
|
|
|
## Deployment model
|
|
|
|
Knoe favors simple, understandable deployment over excessive platform ceremony.
|
|
|
|
Typical characteristics:
|
|
|
|
- Kubernetes-native deployment across K3d (local), K3s (on-prem), and GCP (cloud)
|
|
- lightweight K3s/K3d-friendly footprint
|
|
- HA where it matters
|
|
- shard-oriented layout instead of one giant control surface
|
|
- GitOps-driven operations via ArgoCD, backed by Gitea or GitLab
|
|
- support for small clusters, including Pi-based or edge environments
|
|
|
|
---
|
|
|
|
## Use cases
|
|
|
|
Knoe is a good fit for teams that need:
|
|
|
|
- a PostgreSQL-centered application stack inside a disconnected or sovereign environment
|
|
- a self-hosted substrate for Supabase-style internal platforms
|
|
- ingestion from offline or intermittently connected sources
|
|
- reproducible deployment on small Kubernetes clusters (K3s/K3d) or GCP
|
|
- strong control over secrets, storage, and identity boundaries
|
|
- a full GitOps workflow with ArgoCD, Gitea, and GitLab
|
|
|
|
---
|
|
|
|
## Project principles
|
|
|
|
- **Air-gap first**
|
|
- **Postgres first**
|
|
- **Simple over ornate**
|
|
- **Deterministic ingestion**
|
|
- **Kubernetes-native**
|
|
- **Shard-friendly**
|
|
- **Operationally boring where possible**
|
|
|
|
---
|
|
|
|
## Repository scope
|
|
|
|
This repository contains the infrastructure and platform materials used to stand up Knoe components, including Kubernetes deployment, database operations (`knoe-db` in `knoe-system`), storage, auth, observability, and supporting services.
|
|
|
|
As the project evolves, this README should stay focused on:
|
|
- what Knoe is
|
|
- why it exists
|
|
- the major building blocks
|
|
- the deployment model
|
|
- the operational philosophy
|
|
|
|
Detailed setup docs, cluster procedures, and host-specific notes should live in dedicated documents under `docs/`, `ops/`, `k8s/`, or role-specific subdirectories rather than expanding this file indefinitely.
|
|
|
|
---
|
|
|
|
## Status
|
|
|
|
Knoe is an actively evolving platform stack aimed at practical self-hosted, edge, and cloud operation.
|
|
|
|
Expect the architecture to continue being refined toward:
|
|
- cleaner bootstrapping
|
|
- better shard isolation
|
|
- smoother rejoin/reset behavior for cluster nodes
|
|
- clearer service boundaries
|
|
- improved onboarding and operations documentation
|
|
|
|
---
|
|
|
|
## License
|
|
|
|
Add the project license here.
|