homelab/project-usage/database-postgres.md

CloudNativePG PostgreSQL Database

Host: ddb-cluster-rw.ddb.svc.cluster.local (read-write)
Read replica: ddb-cluster-ro.ddb.svc.cluster.local (read-only)
Port: 5432
Namespace: ddb

When to Use

• Multi-replica HA — 3 replicas, automatic failover
• pgvector extension — Vector similarity search (LLM embeddings)
• Transactional data — Authentik, Story Crater backend, custom apps
• Declarative backups — Automated WAL archiving to MinIO

Quick Start

1. Connect from pod:

# Inside a pod (inject secret mount)
psql -h ddb-cluster-rw.ddb.svc.cluster.local \
  -U story_crater \
  -d story_crater \
  -W  # prompt for password (from Secret)

2. Create database & user (one-time):

# Already done by helmfile postsync hook
# But if needed manually:

psql -h ddb-cluster-rw.ddb.svc.cluster.local \
  -U postgres \
  -c "CREATE DATABASE myapp OWNER postgres;"

psql -h ddb-cluster-rw.ddb.svc.cluster.local \
  -U postgres \
  -d myapp \
  -c "CREATE USER myapp_user WITH PASSWORD 'secret';"

psql -h ddb-cluster-rw.ddb.svc.cluster.local \
  -U postgres \
  -d myapp \
  -c "GRANT ALL PRIVILEGES ON DATABASE myapp TO myapp_user;"

3. Enable pgvector:

psql -h ddb-cluster-rw.ddb.svc.cluster.local \
  -U postgres \
  -d myapp \
  -c "CREATE EXTENSION IF NOT EXISTS vector;"

4. Create table with embeddings:

CREATE TABLE documents (
  id BIGSERIAL PRIMARY KEY,
  content TEXT,
  embedding vector(1536),  -- OpenAI embeddings
  created_at TIMESTAMP DEFAULT NOW()
);

CREATE INDEX ON documents USING IVFFLAT (embedding vector_cosine_ops);

Configuration

Key	Value
Host (RW)	ddb-cluster-rw.ddb.svc.cluster.local
Host (RO)	ddb-cluster-ro.ddb.svc.cluster.local
Port	5432
Replicas	3 (automatic failover)
Extensions	pgvector (LLM embeddings), uuid-ossp
Backups	WAL archiving to MinIO (continuous)
Retention	30 days

Common Patterns

Connection pooling (from app):

import "github.com/jackc/pgx/v5/pgxpool"

config, _ := pgxpool.ParseConfig("postgres://user:[email protected]:5432/myapp")
config.MaxConns = 25
config.MinConns = 5
pool, _ := pgxpool.NewWithConfig(ctx, config)

// Use pool
row := pool.QueryRow(ctx, "SELECT COUNT(*) FROM users")

Read from replica (analytics):

// Offload SELECT queries to read replica
pool.QueryRow(ctx, "SELECT * FROM documents LIMIT 1") // auto-routes to RO if available

// Writes always go to RW
pool.Exec(ctx, "INSERT INTO documents ...")

Vector similarity search:

SELECT id, content, embedding <-> $1 AS distance
FROM documents
ORDER BY embedding <-> $1
LIMIT 10;
-- $1 = query embedding (e.g., from OpenAI API)

Backup & restore:

# Backups are automatic (WAL to MinIO)
# To restore from backup:
# 1. Check MinIO s3://postgresql-backups/
# 2. Use PostgreSQL PITR (point-in-time recovery)
# 3. Contact SRE for restore procedure

Monitoring

Grafana dashboard: svc-postgresql (auto-configured)

Key metrics:

• pg_stat_activity_connections — active connections
• pg_stat_database_blks_read — disk I/O
• pg_replication_lag_seconds — replica lag (goal: < 1s)

CLI health check:

# Check replication status
kubectl exec -n ddb pod/ddb-cluster-1 -- \
  psql -U postgres -c "SELECT slot_name, restart_lsn FROM pg_replication_slots;"

# Check replica lag
kubectl exec -n ddb pod/ddb-cluster-2 -- \
  psql -U postgres -c "SELECT now() - pg_last_xact_replay_timestamp() AS lag;"

Secrets & Credentials

All user passwords stored in Vault:

# Read password
core get cluster/STORY_CRATER_PG_PASSWORD --key STORY_CRATER_PG_PASSWORD

# Inject into pod (auto via Secret volume)
# Mount: /run/secrets/db-password

Connection string from env:

POSTGRES_CONNECTION="postgres://story_crater:${STORY_CRATER_PG_PASSWORD}@ddb-cluster-rw.ddb.svc.cluster.local:5432/story_crater"

Troubleshooting

Cannot connect (connection refused):

# Verify cluster is running
k get pods -n ddb

# Check Service DNS
k exec -it pod/debug-pod -- nslookup ddb-cluster-rw.ddb.svc.cluster.local

# Verify Secret has password
k get secret -n ddb ddb-cluster-superuser -o jsonpath='{.data.password}' | base64 -d

Replica lag is high (> 10s):

# Check replica pod CPU/memory
k top pod -n ddb

# Scale down other workloads if cluster is overloaded
# Or scale up database resources (helmfile.yaml.gotmpl)

pgvector queries slow:

-- Ensure index exists
SELECT * FROM pg_indexes WHERE tablename = 'documents' AND indexname LIKE '%embedding%';

-- Re-index if missing
CREATE INDEX ON documents USING IVFFLAT (embedding vector_cosine_ops);

See /TROUBLESHOOTING.md for full incident guide.