k8s/services: add ingress networking portainer llm and project guides

- Nginx ingress + TLS termination (homelab-ca)
- Portainer container UI
- CoreDNS internal DNS rewrites
- DuckDNS DDNS updater
- Ollama LLM inference
- 8 project-usage guides (team reference)

project-usage/database-postgres.md

@@ -0,0 +1,188 @@
+# CloudNativePG PostgreSQL Database
+
+**Host:** `ddb-cluster-rw.ddb.svc.cluster.local` (read-write)  
+**Read replica:** `ddb-cluster-ro.ddb.svc.cluster.local` (read-only)  
+**Port:** `5432`  
+**Namespace:** `ddb`
+
+## When to Use
+
+- **Multi-replica HA** — 3 replicas, automatic failover
+- **pgvector extension** — Vector similarity search (LLM embeddings)
+- **Transactional data** — Authentik, Story Crater backend, custom apps
+- **Declarative backups** — Automated WAL archiving to MinIO
+
+## Quick Start
+
+**1. Connect from pod:**
+```bash
+# Inside a pod (inject secret mount)
+psql -h ddb-cluster-rw.ddb.svc.cluster.local \
+  -U story_crater \
+  -d story_crater \
+  -W  # prompt for password (from Secret)
+```
+
+**2. Create database & user (one-time):**
+```bash
+# Already done by helmfile postsync hook
+# But if needed manually:
+
+psql -h ddb-cluster-rw.ddb.svc.cluster.local \
+  -U postgres \
+  -c "CREATE DATABASE myapp OWNER postgres;"
+
+psql -h ddb-cluster-rw.ddb.svc.cluster.local \
+  -U postgres \
+  -d myapp \
+  -c "CREATE USER myapp_user WITH PASSWORD 'secret';"
+
+psql -h ddb-cluster-rw.ddb.svc.cluster.local \
+  -U postgres \
+  -d myapp \
+  -c "GRANT ALL PRIVILEGES ON DATABASE myapp TO myapp_user;"
+```
+
+**3. Enable pgvector:**
+```bash
+psql -h ddb-cluster-rw.ddb.svc.cluster.local \
+  -U postgres \
+  -d myapp \
+  -c "CREATE EXTENSION IF NOT EXISTS vector;"
+```
+
+**4. Create table with embeddings:**
+```sql
+CREATE TABLE documents (
+  id BIGSERIAL PRIMARY KEY,
+  content TEXT,
+  embedding vector(1536),  -- OpenAI embeddings
+  created_at TIMESTAMP DEFAULT NOW()
+);
+
+CREATE INDEX ON documents USING IVFFLAT (embedding vector_cosine_ops);
+```
+
+## Configuration
+
+| Key | Value |
+|-----|-------|
+| Host (RW) | `ddb-cluster-rw.ddb.svc.cluster.local` |
+| Host (RO) | `ddb-cluster-ro.ddb.svc.cluster.local` |
+| Port | 5432 |
+| Replicas | 3 (automatic failover) |
+| Extensions | pgvector (LLM embeddings), uuid-ossp |
+| Backups | WAL archiving to MinIO (continuous) |
+| Retention | 30 days |
+
+## Common Patterns
+
+**Connection pooling (from app):**
+```go
+import "github.com/jackc/pgx/v5/pgxpool"
+
+config, _ := pgxpool.ParseConfig("postgres://user:[email protected]:5432/myapp")
+config.MaxConns = 25
+config.MinConns = 5
+pool, _ := pgxpool.NewWithConfig(ctx, config)
+
+// Use pool
+row := pool.QueryRow(ctx, "SELECT COUNT(*) FROM users")
+```
+
+**Read from replica (analytics):**
+```go
+// Offload SELECT queries to read replica
+pool.QueryRow(ctx, "SELECT * FROM documents LIMIT 1") // auto-routes to RO if available
+
+// Writes always go to RW
+pool.Exec(ctx, "INSERT INTO documents ...")
+```
+
+**Vector similarity search:**
+```sql
+SELECT id, content, embedding <-> $1 AS distance
+FROM documents
+ORDER BY embedding <-> $1
+LIMIT 10;
+-- $1 = query embedding (e.g., from OpenAI API)
+```
+
+**Backup & restore:**
+```bash
+# Backups are automatic (WAL to MinIO)
+# To restore from backup:
+# 1. Check MinIO s3://postgresql-backups/
+# 2. Use PostgreSQL PITR (point-in-time recovery)
+# 3. Contact SRE for restore procedure
+```
+
+## Monitoring
+
+**Grafana dashboard:** `svc-postgresql` (auto-configured)
+
+**Key metrics:**
+- `pg_stat_activity_connections` — active connections
+- `pg_stat_database_blks_read` — disk I/O
+- `pg_replication_lag_seconds` — replica lag (goal: < 1s)
+
+**CLI health check:**
+```bash
+# Check replication status
+kubectl exec -n ddb pod/ddb-cluster-1 -- \
+  psql -U postgres -c "SELECT slot_name, restart_lsn FROM pg_replication_slots;"
+
+# Check replica lag
+kubectl exec -n ddb pod/ddb-cluster-2 -- \
+  psql -U postgres -c "SELECT now() - pg_last_xact_replay_timestamp() AS lag;"
+```
+
+## Secrets & Credentials
+
+**All user passwords stored in Vault:**
+```bash
+# Read password
+talos get cluster/STORY_CRATER_PG_PASSWORD --key STORY_CRATER_PG_PASSWORD
+
+# Inject into pod (auto via Secret volume)
+# Mount: /run/secrets/db-password
+```
+
+**Connection string from env:**
+```bash
+POSTGRES_CONNECTION="postgres://story_crater:${STORY_CRATER_PG_PASSWORD}@ddb-cluster-rw.ddb.svc.cluster.local:5432/story_crater"
+```
+
+## Troubleshooting
+
+**Cannot connect (connection refused):**
+```bash
+# Verify cluster is running
+k get pods -n ddb
+
+# Check Service DNS
+k exec -it pod/debug-pod -- nslookup ddb-cluster-rw.ddb.svc.cluster.local
+
+# Verify Secret has password
+k get secret -n ddb ddb-cluster-superuser -o jsonpath='{.data.password}' | base64 -d
+```
+
+**Replica lag is high (> 10s):**
+```bash
+# Check replica pod CPU/memory
+k top pod -n ddb
+
+# Scale down other workloads if cluster is overloaded
+# Or scale up database resources (helmfile.yaml.gotmpl)
+```
+
+**pgvector queries slow:**
+```sql
+-- Ensure index exists
+SELECT * FROM pg_indexes WHERE tablename = 'documents' AND indexname LIKE '%embedding%';
+
+-- Re-index if missing
+CREATE INDEX ON documents USING IVFFLAT (embedding vector_cosine_ops);
+```
+
+See `/TROUBLESHOOTING.md` for full incident guide.