From 966b07758e3e633b8a429a685e5f4f1dc54b879f Mon Sep 17 00:00:00 2001
From: Story Crater Bot <19826264+Riotpiaole@users.noreply.github.com>
Date: Thu, 23 Jul 2026 10:15:25 -0700
Subject: [PATCH] docs(CLAUDE.md): document CNPG unified pattern and fix
 storage topology

ADDED:
  - CloudNativePG (CNPG) Database Pattern section
  - Explains shared 'app' user model (not per-app roles)
  - Documents bootstrap.sh credential distribution pattern
  - Working examples (Forgejo, Authentik)
  - Prescriptive DO/DON'T guidance for new apps

FIXED:
  - Storage topology: 3-node HA (not "sole Longhorn node")
  - Verified: all 17 PVCs have replicas across all 3 nodes
  - Updated last-modified date

This documents the architectural pattern established during CNPG refactor.
---
 CLAUDE.md | 84 ++++++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 77 insertions(+), 7 deletions(-)

diff --git a/CLAUDE.md b/CLAUDE.md
index f0a7fda..a2945b7 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -4,13 +4,14 @@
 
 | Node | IP | Zone | Scheduling | Storage |
 |------|----|----|-----------|---------|
-| `talos-cp-1` | .213 | az-a | schedulable (all workloads) | sole Longhorn node |
-| `talos-cp-2` | .163 | az-b | dedicated (`NoSchedule`) | none |
-| `talos-cp-3` | .166 | az-c | dedicated (`NoSchedule`) | none |
+| `talos-cp-1` | .213 | az-a | schedulable (all workloads) | Longhorn replica |
+| `talos-cp-2` | .163 | az-b | dedicated (`NoSchedule`) | Longhorn replica |
+| `talos-cp-3` | .166 | az-c | dedicated (`NoSchedule`) | Longhorn replica |
 
-3 voting etcd members peering on the LAN. Only `talos-cp-1` runs workloads and
-holds storage → stateful services are single-instance. Full detail + gotchas in
-`USAGE.md` and memory `reference_talos_etcd_and_ca_gotchas`.
+3 voting etcd members peering on the LAN. All 3 nodes run Longhorn with 3-replica
+HA storage (verified: all 17 PVCs have replicas across all nodes). Only `talos-cp-1`
+is schedulable for workloads (cp-2/cp-3 are control-plane only). Full detail + gotchas
+in `USAGE.md` and memory `reference_talos_etcd_and_ca_gotchas`.
 
 ## Deployment Model: ArgoCD GitOps (helmfile is retired)
 
@@ -70,6 +71,75 @@ Verify no changes (clean output = formatted correctly). If files change, review
 
 🔴 **Infrastructure changes should flow through GitOps when possible:** git commit → push → CI/CD runner (terraform apply) → ArgoCD sync. Local `terraform apply` is permitted (e.g. for local iteration, config regeneration, or when CI/CD isn't wired up for a given module) — still commit + push the resulting state/config afterward so git remains the record of truth. Manual `kubectl apply` remains disallowed for Terraform-managed resources.
 
+## CloudNativePG (CNPG) Database Pattern
+
+**Simple ownership model — all apps use shared 'app' user:**
+
+```yaml
+# CNPG Cluster (bootstrap, wave 0)
+bootstrap:
+  initdb:
+    database: app      # Bootstrap database
+    owner: app         # Bootstrap user (owns all databases)
+
+# Database CR (per-app, wave 6)
+spec:
+  name: authentik      # Database name
+  owner: app           # ← All apps use 'app' (not per-app roles)
+  cluster:
+    name: ddb-cluster
+```
+
+**Credential Distribution (bootstrap.sh pattern):**
+
+1. **Source of truth:** CNPG creates `ddb-cluster-app` secret in `ddb` namespace
+2. **Distribution:** `bootstrap.sh` copies secret to app namespaces:
+   ```bash
+   # For each app namespace (cicd, iam, etc):
+   kubectl get secret ddb-cluster-app -n ddb -o yaml \
+     | sed 's/namespace: ddb/namespace: <app-namespace>/' \
+     | kubectl apply -f -
+   ```
+3. **Apps reference local copy:**
+   ```yaml
+   env:
+     - name: DB_USER
+       valueFrom:
+         secretKeyRef:
+           name: ddb-cluster-app  # Local copy in app's namespace
+           key: username          # Always "app"
+     - name: DB_PASSWORD
+       valueFrom:
+         secretKeyRef:
+           name: ddb-cluster-app
+           key: password
+   ```
+
+**Key Points:**
+
+✅ **DO:**
+- All apps connect as `app` user
+- Database CRs specify `owner: app`
+- Isolation via separate database names (not roles)
+- Copy credentials to app namespace via `bootstrap.sh`
+- Reference local secret copy via `secretKeyRef`
+
+❌ **DON'T:**
+- Create per-app roles in `managed.roles` (CNPG doesn't transfer ownership properly)
+- Grant permissions via PostSync Jobs (owner already has full rights)
+- Use cross-namespace `secretKeyRef` (not supported)
+- Manually patch secrets (ArgoCD will revert)
+
+**Working Examples:**
+- Forgejo: `k8s/bootstrap-local/04-forgejo.yaml` (references `ddb-cluster-app` in cicd namespace)
+- Authentik: `k8s/security/iam/authentik-values.yaml` (references `ddb-cluster-app` in iam namespace)
+
+**For New Apps:**
+1. Add Database CR to `k8s/data/schemas/` with `owner: app`
+2. Add secret copy to `bootstrap.sh` (like cicd/iam examples)
+3. Reference `ddb-cluster-app` via `secretKeyRef` in app's namespace
+4. No permission grants needed — app user owns the database
+
 ## GitOps / ArgoCD Gotchas (hard-won, all confirmed live in this cluster)
 
 🟠 **`kustomization.yaml` with an explicit `resources:` allowlist silently
@@ -178,4 +248,4 @@ branch, which may not match your pinned version's schema at all.**
 
 ---
 
-**Last updated:** 2026-07-22
+**Last updated:** 2026-07-23