homelab/project-usage/database-postgres.md

# CloudNativePG PostgreSQL Database

**Host:** `ddb-cluster-rw.ddb.svc.cluster.local` (read-write)  
**Read replica:** `ddb-cluster-ro.ddb.svc.cluster.local` (read-only)  
**Port:** `5432`  
**Namespace:** `ddb`

## When to Use

- **Multi-replica HA** — 3 replicas, automatic failover
- **pgvector extension** — Vector similarity search (LLM embeddings)
- **Transactional data** — Authentik, Story Crater backend, custom apps
- **Declarative backups** — Automated WAL archiving to MinIO

## Quick Start

**1. Connect from pod:**
```bash
# Inside a pod (inject secret mount)
psql -h ddb-cluster-rw.ddb.svc.cluster.local \
  -U story_crater \
  -d story_crater \
  -W  # prompt for password (from Secret)
```

**2. Create database & user (one-time):**
```bash
# Already done by helmfile postsync hook
# But if needed manually:

psql -h ddb-cluster-rw.ddb.svc.cluster.local \
  -U postgres \
  -c "CREATE DATABASE myapp OWNER postgres;"

psql -h ddb-cluster-rw.ddb.svc.cluster.local \
  -U postgres \
  -d myapp \
  -c "CREATE USER myapp_user WITH PASSWORD 'secret';"

psql -h ddb-cluster-rw.ddb.svc.cluster.local \
  -U postgres \
  -d myapp \
  -c "GRANT ALL PRIVILEGES ON DATABASE myapp TO myapp_user;"
```

**3. Enable pgvector:**
```bash
psql -h ddb-cluster-rw.ddb.svc.cluster.local \
  -U postgres \
  -d myapp \
  -c "CREATE EXTENSION IF NOT EXISTS vector;"
```

**4. Create table with embeddings:**
```sql
CREATE TABLE documents (
  id BIGSERIAL PRIMARY KEY,
  content TEXT,
  embedding vector(1536),  -- OpenAI embeddings
  created_at TIMESTAMP DEFAULT NOW()
);

CREATE INDEX ON documents USING IVFFLAT (embedding vector_cosine_ops);
```

## Configuration

| Key | Value |
|-----|-------|
| Host (RW) | `ddb-cluster-rw.ddb.svc.cluster.local` |
| Host (RO) | `ddb-cluster-ro.ddb.svc.cluster.local` |
| Port | 5432 |
| Replicas | 3 (automatic failover) |
| Extensions | pgvector (LLM embeddings), uuid-ossp |
| Backups | WAL archiving to MinIO (continuous) |
| Retention | 30 days |

## Common Patterns

**Connection pooling (from app):**
```go
import "github.com/jackc/pgx/v5/pgxpool"

config, _ := pgxpool.ParseConfig("postgres://user:[email protected]:5432/myapp")
config.MaxConns = 25
config.MinConns = 5
pool, _ := pgxpool.NewWithConfig(ctx, config)

// Use pool
row := pool.QueryRow(ctx, "SELECT COUNT(*) FROM users")
```

**Read from replica (analytics):**
```go
// Offload SELECT queries to read replica
pool.QueryRow(ctx, "SELECT * FROM documents LIMIT 1") // auto-routes to RO if available

// Writes always go to RW
pool.Exec(ctx, "INSERT INTO documents ...")
```

**Vector similarity search:**
```sql
SELECT id, content, embedding <-> $1 AS distance
FROM documents
ORDER BY embedding <-> $1
LIMIT 10;
-- $1 = query embedding (e.g., from OpenAI API)
```

**Backup & restore:**
```bash
# Backups are automatic (WAL to MinIO)
# To restore from backup:
# 1. Check MinIO s3://postgresql-backups/
# 2. Use PostgreSQL PITR (point-in-time recovery)
# 3. Contact SRE for restore procedure
```

## Monitoring

**Grafana dashboard:** `svc-postgresql` (auto-configured)

**Key metrics:**
- `pg_stat_activity_connections` — active connections
- `pg_stat_database_blks_read` — disk I/O
- `pg_replication_lag_seconds` — replica lag (goal: < 1s)

**CLI health check:**
```bash
# Check replication status
kubectl exec -n ddb pod/ddb-cluster-1 -- \
  psql -U postgres -c "SELECT slot_name, restart_lsn FROM pg_replication_slots;"

# Check replica lag
kubectl exec -n ddb pod/ddb-cluster-2 -- \
  psql -U postgres -c "SELECT now() - pg_last_xact_replay_timestamp() AS lag;"
```

## Secrets & Credentials

**All user passwords stored in Vault:**
```bash
# Read password
talos get cluster/STORY_CRATER_PG_PASSWORD --key STORY_CRATER_PG_PASSWORD

# Inject into pod (auto via Secret volume)
# Mount: /run/secrets/db-password
```

**Connection string from env:**
```bash
POSTGRES_CONNECTION="postgres://story_crater:${STORY_CRATER_PG_PASSWORD}@ddb-cluster-rw.ddb.svc.cluster.local:5432/story_crater"
```

## Troubleshooting

**Cannot connect (connection refused):**
```bash
# Verify cluster is running
k get pods -n ddb

# Check Service DNS
k exec -it pod/debug-pod -- nslookup ddb-cluster-rw.ddb.svc.cluster.local

# Verify Secret has password
k get secret -n ddb ddb-cluster-superuser -o jsonpath='{.data.password}' | base64 -d
```

**Replica lag is high (> 10s):**
```bash
# Check replica pod CPU/memory
k top pod -n ddb

# Scale down other workloads if cluster is overloaded
# Or scale up database resources (helmfile.yaml.gotmpl)
```

**pgvector queries slow:**
```sql
-- Ensure index exists
SELECT * FROM pg_indexes WHERE tablename = 'documents' AND indexname LIKE '%embedding%';

-- Re-index if missing
CREATE INDEX ON documents USING IVFFLAT (embedding vector_cosine_ops);
```

See `/TROUBLESHOOTING.md` for full incident guide.