homelab/DDB-REVIEW.md

DDB Cluster Configuration Review

Current Configuration

File: k8s/data/cluster/ddb-cluster.yaml (bootstrap-only)

✅ Good Practices

1. 3-replica cluster - HA across control-plane nodes (az-a, az-b, az-c)
2. PostgreSQL 16.2 - Modern, stable version
3. Managed roles - Passwords from Secrets (CNPG reconciles automatically)
4. Separate Database CRs - Each app gets its own database (good separation)
5. Extensions enabled - vector, pgcrypto, pg_trgm (ready for Authentik/Temporal)
6. Superuser disabled - Security hardening
7. Longhorn storage - ⚠️ STORAGE HA STATUS UNKNOWN!
   • CLAUDE.md claims: Single-node storage (cp-1 only) = NO HA ❌
   • Longhorn manifests show: 3-node config (numberOfReplicas: 3) = HA ✅
   • CRITICAL: Verify actual state before assuming HA
   • See STORAGE-ARCHITECTURE-CLARIFICATION.md for verification commands

⚠️ Recommendations

1. Increase shared_buffers for multi-tenant workload

• Current: 256MB
• Recommended: 512MB-1GB (with 3 replicas + multiple DBs)
• Reason: Forgejo, Authentik, Temporal, Vault all share this cluster

2. Enable connection pooling (PgBouncer)

• Add pooler configuration for connection efficiency
• Especially important for Temporal (high connection count)

3. Configure backups

• No backup configuration present
• Add S3/MinIO backup schedule

4. Resource limits missing

• Add PostgreSQL pod resource requests/limits

5. Monitoring

• enablePodMonitor: false - should be true for Prometheus scraping
• Add backup monitoring alerts

📝 Proposed Enhanced Configuration

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: ddb-cluster
  namespace: ddb
  labels:
    app: postgresql
    layer: data
    environment: production
spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:16.2

  # Resource limits (IMPORTANT for stability)
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
    limits:
      cpu: "2"
      memory: "2Gi"

  bootstrap:
    initdb:
      database: app
      owner: app
      encoding: UTF8
      localeCollate: C
      localeCType: C
      postInitApplicationSQL:
        - CREATE EXTENSION IF NOT EXISTS vector;
        - CREATE EXTENSION IF NOT EXISTS pgcrypto;
        - CREATE EXTENSION IF NOT EXISTS pg_trgm;
        - CREATE EXTENSION IF NOT EXISTS btree_gin;
        - CREATE EXTENSION IF NOT EXISTS btree_gist;

  managed:
    roles:
      - name: authentik
        ensure: present
        login: true
        passwordSecret:
          name: authentik-db-role
      - name: temporal
        ensure: present
        login: true
        passwordSecret:
          name: temporal-db-role

  enableSuperuserAccess: false

  postgresql:
    parameters:
      # Memory
      shared_buffers: "512MB"                   # Increased from 256MB
      effective_cache_size: "1536MB"            # ~75% of memory limit
      work_mem: "16MB"                          # Per-operation memory
      maintenance_work_mem: "128MB"             # For VACUUM/CREATE INDEX
      
      # Parallelism
      max_parallel_workers: "4"
      max_parallel_workers_per_gather: "2"
      max_worker_processes: "8"
      
      # Connection pooling (PgBouncer will handle this, but set reasonable limits)
      max_connections: "100"
      
      # WAL & Checkpoints
      archive_mode: "on"
      archive_timeout: "5min"
      wal_level: "replica"
      max_wal_size: "1GB"
      min_wal_size: "256MB"
      checkpoint_completion_target: "0.9"
      
      # Logging
      log_destination: "csvlog"
      log_directory: "/controller/log"
      log_filename: "postgres"
      log_rotation_age: "0"
      log_min_duration_statement: "1000"       # Log slow queries (>1s)
      log_line_prefix: "%t [%p]: [%l-1] user=%u,db=%d,app=%a,client=%h "
      
      # Performance
      random_page_cost: "1.1"                  # SSD-optimized
      effective_io_concurrency: "200"          # SSD concurrent I/O
      dynamic_shared_memory_type: "posix"

  storage:
    size: 20Gi                                  # Increased from 10Gi
    storageClass: longhorn

  # PgBouncer connection pooler
  pooler:
    enabled: true
    poolMode: transaction
    instances: 2
    parameters:
      max_client_conn: "1000"
      default_pool_size: "25"
      max_db_connections: "90"
    resources:
      requests:
        cpu: "100m"
        memory: "128Mi"
      limits:
        cpu: "500m"
        memory: "256Mi"

  # Monitoring (ENABLE for Prometheus)
  monitoring:
    enablePodMonitor: true                     # Changed from false
    disableDefaultQueries: false
    customQueriesConfigMap:
      - name: cnpg-default-monitoring
        key: queries

  # Backups to MinIO
  backup:
    barmanObjectStore:
      destinationPath: s3://ddb-backups/
      endpointURL: http://minio.storage.svc:9000
      s3Credentials:
        accessKeyId:
          name: ddb-backup-s3
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: ddb-backup-s3
          key: SECRET_ACCESS_KEY
      wal:
        compression: gzip
        maxParallel: 2
    retentionPolicy: "30d"

  affinity:
    podAntiAffinityType: preferred

  # Node affinity (prefer spreading across zones)
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          preference:
            matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                  - az-a
                  - az-b
                  - az-c

🔐 Required Secrets

Create these secrets before applying enhanced config:

# 1. Backup credentials (create MinIO bucket + user first)
kubectl create secret generic ddb-backup-s3 -n ddb \
  --from-literal=ACCESS_KEY_ID="<minio-access-key>" \
  --from-literal=SECRET_ACCESS_KEY="<minio-secret-key>"

📊 Connection Architecture

Application Pods
       ↓
PgBouncer Pooler (transaction mode, 2 replicas)
       ↓
ddb-cluster-rw.ddb.svc (read-write service)
       ↓
Primary PostgreSQL Pod
       ↓
Replica Pods (async replication)

🎯 Migration Path

1. Current state: Bootstrap-only (manual)
2. Proposed: Still bootstrap-only (circular dependency with Forgejo)
3. Future consideration: If Forgejo is decoupled from this cluster (e.g., external git host), DDB could move to full GitOps