homelab/DDB-REVIEW.md

# DDB Cluster Configuration Review

## Current Configuration

**File:** `k8s/data/cluster/ddb-cluster.yaml` (bootstrap-only)

### ✅ Good Practices

1. **3-replica cluster** - HA across control-plane nodes (az-a, az-b, az-c)
2. **PostgreSQL 16.2** - Modern, stable version
3. **Managed roles** - Passwords from Secrets (CNPG reconciles automatically)
4. **Separate Database CRs** - Each app gets its own database (good separation)
5. **Extensions enabled** - vector, pgcrypto, pg_trgm (ready for Authentik/Temporal)
6. **Superuser disabled** - Security hardening
7. **Longhorn storage** - ⚠️ **STORAGE HA STATUS UNKNOWN!**
   - CLAUDE.md claims: Single-node storage (cp-1 only) = NO HA ❌
   - Longhorn manifests show: 3-node config (numberOfReplicas: 3) = HA ✅
   - **CRITICAL:** Verify actual state before assuming HA
   - See `STORAGE-ARCHITECTURE-CLARIFICATION.md` for verification commands

### ⚠️ Recommendations

#### 1. **Increase shared_buffers for multi-tenant workload**
   - Current: 256MB
   - Recommended: 512MB-1GB (with 3 replicas + multiple DBs)
   - Reason: Forgejo, Authentik, Temporal, Vault all share this cluster

#### 2. **Enable connection pooling (PgBouncer)**
   - Add pooler configuration for connection efficiency
   - Especially important for Temporal (high connection count)

#### 3. **Configure backups**
   - No backup configuration present
   - Add S3/MinIO backup schedule

#### 4. **Resource limits missing**
   - Add PostgreSQL pod resource requests/limits

#### 5. **Monitoring**
   - `enablePodMonitor: false` - should be `true` for Prometheus scraping
   - Add backup monitoring alerts

### 📝 Proposed Enhanced Configuration

```yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: ddb-cluster
  namespace: ddb
  labels:
    app: postgresql
    layer: data
    environment: production
spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:16.2

  # Resource limits (IMPORTANT for stability)
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
    limits:
      cpu: "2"
      memory: "2Gi"

  bootstrap:
    initdb:
      database: app
      owner: app
      encoding: UTF8
      localeCollate: C
      localeCType: C
      postInitApplicationSQL:
        - CREATE EXTENSION IF NOT EXISTS vector;
        - CREATE EXTENSION IF NOT EXISTS pgcrypto;
        - CREATE EXTENSION IF NOT EXISTS pg_trgm;
        - CREATE EXTENSION IF NOT EXISTS btree_gin;
        - CREATE EXTENSION IF NOT EXISTS btree_gist;

  managed:
    roles:
      - name: authentik
        ensure: present
        login: true
        passwordSecret:
          name: authentik-db-role
      - name: temporal
        ensure: present
        login: true
        passwordSecret:
          name: temporal-db-role

  enableSuperuserAccess: false

  postgresql:
    parameters:
      # Memory
      shared_buffers: "512MB"                   # Increased from 256MB
      effective_cache_size: "1536MB"            # ~75% of memory limit
      work_mem: "16MB"                          # Per-operation memory
      maintenance_work_mem: "128MB"             # For VACUUM/CREATE INDEX
      
      # Parallelism
      max_parallel_workers: "4"
      max_parallel_workers_per_gather: "2"
      max_worker_processes: "8"
      
      # Connection pooling (PgBouncer will handle this, but set reasonable limits)
      max_connections: "100"
      
      # WAL & Checkpoints
      archive_mode: "on"
      archive_timeout: "5min"
      wal_level: "replica"
      max_wal_size: "1GB"
      min_wal_size: "256MB"
      checkpoint_completion_target: "0.9"
      
      # Logging
      log_destination: "csvlog"
      log_directory: "/controller/log"
      log_filename: "postgres"
      log_rotation_age: "0"
      log_min_duration_statement: "1000"       # Log slow queries (>1s)
      log_line_prefix: "%t [%p]: [%l-1] user=%u,db=%d,app=%a,client=%h "
      
      # Performance
      random_page_cost: "1.1"                  # SSD-optimized
      effective_io_concurrency: "200"          # SSD concurrent I/O
      dynamic_shared_memory_type: "posix"

  storage:
    size: 20Gi                                  # Increased from 10Gi
    storageClass: longhorn

  # PgBouncer connection pooler
  pooler:
    enabled: true
    poolMode: transaction
    instances: 2
    parameters:
      max_client_conn: "1000"
      default_pool_size: "25"
      max_db_connections: "90"
    resources:
      requests:
        cpu: "100m"
        memory: "128Mi"
      limits:
        cpu: "500m"
        memory: "256Mi"

  # Monitoring (ENABLE for Prometheus)
  monitoring:
    enablePodMonitor: true                     # Changed from false
    disableDefaultQueries: false
    customQueriesConfigMap:
      - name: cnpg-default-monitoring
        key: queries

  # Backups to MinIO
  backup:
    barmanObjectStore:
      destinationPath: s3://ddb-backups/
      endpointURL: http://minio.storage.svc:9000
      s3Credentials:
        accessKeyId:
          name: ddb-backup-s3
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: ddb-backup-s3
          key: SECRET_ACCESS_KEY
      wal:
        compression: gzip
        maxParallel: 2
    retentionPolicy: "30d"

  affinity:
    podAntiAffinityType: preferred

  # Node affinity (prefer spreading across zones)
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          preference:
            matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                  - az-a
                  - az-b
                  - az-c
```

### 🔐 Required Secrets

Create these secrets before applying enhanced config:

```bash
# 1. Backup credentials (create MinIO bucket + user first)
kubectl create secret generic ddb-backup-s3 -n ddb \
  --from-literal=ACCESS_KEY_ID="<minio-access-key>" \
  --from-literal=SECRET_ACCESS_KEY="<minio-secret-key>"
```

### 📊 Connection Architecture

```
Application Pods
       ↓
PgBouncer Pooler (transaction mode, 2 replicas)
       ↓
ddb-cluster-rw.ddb.svc (read-write service)
       ↓
Primary PostgreSQL Pod
       ↓
Replica Pods (async replication)
```

### 🎯 Migration Path

1. **Current state:** Bootstrap-only (manual)
2. **Proposed:** Still bootstrap-only (circular dependency with Forgejo)
3. **Future consideration:** If Forgejo is decoupled from this cluster (e.g., external git host), DDB could move to full GitOps