k8s/services: add ingress networking portainer llm and project guides

- Nginx ingress + TLS termination (homelab-ca) - Portainer container UI - CoreDNS internal DNS rewrites - DuckDNS DDNS updater - Ollama LLM inference - 8 project-usage guides (team reference)
2026-08-18 15:08:00 -07:00
parent 69ad5c371c
commit 61be24e27f
26 changed files with 3280 additions and 0 deletions
@@ -0,0 +1,67 @@
+# k8s/coredns/coredns-configmap.yaml
+# Patches the CoreDNS Corefile to rewrite homelab hostnames to internal services.
+#
+# Why this is needed:
+#   Grafana v10+ does OIDC auto-discovery by fetching
+#   /.well-known/openid-configuration from Authentik. When Grafana reaches
+#   Authentik via the external hostname (authentik.riotpiao.homelab.com), the
+#   HTTP Host header is preserved and Authentik returns external URLs in the
+#   discovery response. Without this rewrite, the hostname doesn't resolve
+#   inside the cluster and Grafana falls back to the internal service DNS,
+#   causing all OAuth redirects to go to authentik-server.iam.svc.cluster.local.
+#
+# Applied by helmfile presync hook on the ingress-nginx release.
+
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: coredns
+  namespace: kube-system
+data:
+  Corefile: |
+    .:53 {
+        errors
+        health {
+            lameduck 5s
+        }
+        ready
+        log . {
+            class error
+        }
+        prometheus :9153
+
+        # Forgejo: route to HTTP service (LoadBalancer handles both HTTPS:443 + SSH:2222 via same IP)
+        rewrite name forgejo.riotpiao.homelab.com forgejo-gitea-http.cicd.svc.cluster.local
+
+        # Rewrite homelab hostnames to the nginx ingress controller so in-cluster pods
+        # hit nginx TLS termination (cert-manager cert) and preserve the Host header.
+        # Routing through nginx — not directly to the backend service — is critical:
+        # direct rewrites to the backend bypass nginx TLS and expose each app's own
+        # self-signed cert, which nothing in the cluster trusts.
+        rewrite name authentik.riotpiao.homelab.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
+        rewrite name grafana.riotpiao.homelab.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
+        rewrite name minio.riotpiao.homelab.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
+        rewrite name minio-api.riotpiao.homelab.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
+        rewrite name argocd.riotpiao.homelab.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
+        rewrite name vault.riotpiao.homelab.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
+        rewrite name loki.riotpiao.homelab.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
+        rewrite name prometheus.riotpiao.homelab.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
+        rewrite name portainer.riotpiao.homelab.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
+        rewrite name longhorn.riotpiao.homelab.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
+
+        kubernetes cluster.local in-addr.arpa ip6.arpa {
+            pods insecure
+            fallthrough in-addr.arpa ip6.arpa
+            ttl 30
+        }
+        forward . 8.8.8.8 1.1.1.1 {
+           max_concurrent 1000
+        }
+        cache 30 {
+           disable success cluster.local
+           disable denial cluster.local
+        }
+        loop
+        reload
+        loadbalance
+    }
@@ -0,0 +1,40 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: duckdns-updater
+  namespace: kube-system
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: duckdns-updater
+  template:
+    metadata:
+      labels:
+        app: duckdns-updater
+    spec:
+      containers:
+        - name: updater
+          image: curlimages/curl:latest
+          command:
+            - sh
+            - -c
+            - |
+              while true; do
+                curl -fsS "https://www.duckdns.org/update?domains=riotpiao&token=${DUCKDNS_TOKEN}&ip="
+                sleep 300
+              done
+          env:
+            - name: DUCKDNS_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: duckdns-token
+                  key: token
+          resources:
+            requests:
+              cpu: 5m
+              memory: 16Mi
+            limits:
+              cpu: 50m
+              memory: 32Mi
+      restartPolicy: Always
@@ -0,0 +1,297 @@
+# k8s/ingress/ingress.yaml
+# Ingress rules for all homelab services.
+# TLS is handled centrally: nginx serves the wildcard-tls cert (*.riotpiao.homelab.com)
+# as its default-ssl-certificate. No per-rule tls: blocks or cert-manager annotations
+# are needed — cert-manager manages one cert, nginx uses it for all hosts.
+#
+# DNS: *.riotpiao.homelab.com must resolve to 10.6.0.1 (WireGuard) or 192.168.1.160 (LAN).
+
+# ── Grafana ───────────────────────────────────────────────────────────────────
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: grafana
+  namespace: logging
+  annotations:
+    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
+spec:
+  ingressClassName: nginx
+  rules:
+    - host: grafana.riotpiao.homelab.com
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: grafana
+                port:
+                  number: 80
+
+---
+# ── Loki (API access for external tools) ─────────────────────────────────────
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: loki
+  namespace: logging
+spec:
+  ingressClassName: nginx
+  rules:
+    - host: loki.riotpiao.homelab.com
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: loki
+                port:
+                  number: 3100
+
+---
+# ── Authentik ─────────────────────────────────────────────────────────────────
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: authentik
+  namespace: iam
+  annotations:
+    nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"
+    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
+spec:
+  ingressClassName: nginx
+  rules:
+    - host: authentik.riotpiao.homelab.com
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: authentik-server
+                port:
+                  number: 80
+
+---
+# ── Vault ─────────────────────────────────────────────────────────────────────
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: vault
+  namespace: storage
+  annotations:
+    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
+spec:
+  ingressClassName: nginx
+  rules:
+    - host: vault.riotpiao.homelab.com
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: vault
+                port:
+                  number: 8200
+
+---
+# ── MinIO console (storage namespace) ────────────────────────────────────────
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: minio
+  namespace: storage
+  annotations:
+    nginx.ingress.kubernetes.io/proxy-body-size: "0"
+    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
+    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
+    nginx.ingress.kubernetes.io/affinity: "cookie"
+    nginx.ingress.kubernetes.io/session-cookie-name: "minio-console-affinity"
+    nginx.ingress.kubernetes.io/session-cookie-max-age: "3600"
+spec:
+  ingressClassName: nginx
+  rules:
+    - host: minio.riotpiao.homelab.com
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: minio-console
+                port:
+                  number: 9001
+
+---
+# ── MinIO S3 API ──────────────────────────────────────────────────────────────
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: minio-api
+  namespace: storage
+  annotations:
+    nginx.ingress.kubernetes.io/proxy-body-size: "0"
+    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
+    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
+spec:
+  ingressClassName: nginx
+  rules:
+    - host: minio-api.riotpiao.homelab.com
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: minio
+                port:
+                  number: 9000
+
+---
+# ── Prometheus ────────────────────────────────────────────────────────────────
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: prometheus
+  namespace: monitoring
+spec:
+  ingressClassName: nginx
+  rules:
+    - host: prometheus.riotpiao.homelab.com
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: prometheus-kube-prometheus-prometheus
+                port:
+                  number: 9090
+
+---
+# ── Portainer ─────────────────────────────────────────────────────────────────
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: portainer
+  namespace: dashboard
+  annotations:
+    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
+    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
+spec:
+  ingressClassName: nginx
+  rules:
+    - host: portainer.riotpiao.homelab.com
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: portainer
+                port:
+                  number: 9000
+
+---
+# ── Forgejo (git forge + OCI registry UI) ────────────────────────────────────
+# nginx terminates TLS using the wildcard cert, then proxies plain HTTP to
+# Forgejo on port 3000. ROOT_URL stays https:// so Forgejo generates correct URLs.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: forgejo
+  namespace: cicd
+  annotations:
+    nginx.ingress.kubernetes.io/proxy-body-size: "0"
+    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
+    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
+spec:
+  ingressClassName: nginx
+  rules:
+    - host: forgejo.riotpiao.homelab.com
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: forgejo-gitea-http
+                port:
+                  number: 3000
+
+---
+# ── Argo CD (cicd namespace) ──────────────────────────────────────────────────
+# argocd-server runs HTTPS internally — nginx proxies via backend-protocol: HTTPS.
+# proxy-ssl-verify: off because argocd-server's pod cert is self-signed (not homelab-ca).
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: argocd
+  namespace: cicd
+  annotations:
+    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
+    nginx.ingress.kubernetes.io/proxy-ssl-verify: "off"
+    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
+    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
+spec:
+  ingressClassName: nginx
+  rules:
+    - host: argocd.riotpiao.homelab.com
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: argocd-server
+                port:
+                  number: 443
+
+---
+# ── Longhorn UI ───────────────────────────────────────────────────────────────
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: longhorn
+  namespace: longhorn-system
+spec:
+  ingressClassName: nginx
+  rules:
+    - host: longhorn.riotpiao.homelab.com
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: longhorn-frontend
+                port:
+                  number: 80
+
+---
+# ── Temporal Web UI ────────────────────────────────────────────────────────────
+# Temporal workflow orchestration Web UI with OIDC authentication
+# TLS: wildcard cert managed by cert-manager, served by nginx default-ssl-certificate
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: temporal
+  namespace: temporal
+  annotations:
+    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
+    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
+spec:
+  ingressClassName: nginx
+  rules:
+    - host: temporal.riotpiao.homelab.com
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: temporal-web
+                port:
+                  number: 8080
@@ -0,0 +1,78 @@
+# k8s/ingress/nginx-values.yaml
+# Nginx Ingress Controller — bare-metal homelab config.
+# LoadBalancer service with Cilium LB-IPAM assigns fixed IP (192.168.1.160).
+# Access services at https://grafana.riotpiao.homelab.com (80/443 via LoadBalancer).
+
+controller:
+  kind: DaemonSet
+
+  # Single wildcard cert served for every *.riotpiao.homelab.com host.
+  # Applied by the ingress-nginx presync hook (wildcard-cert.yaml) before nginx starts.
+  # nginx hot-reloads when cert-manager renews homelab-tls — no restart needed.
+  extraArgs:
+    default-ssl-certificate: "ingress-nginx/homelab-tls"
+
+  hostPort:
+    enabled: true
+    ports:
+      http: 80
+      https: 443
+
+  # TCP proxy: forward port 2222 on every node → Forgejo SSH service.
+  # This lets `git clone [email protected]:repo` work via the
+  # same hostname as HTTPS without a separate LoadBalancer IP for SSH.
+  tcp:
+    2222: "cicd/forgejo-gitea-ssh:2222"
+
+  # Service as LoadBalancer — Cilium LB-IPAM assigns fixed IP.
+  service:
+    type: LoadBalancer
+    annotations:
+      io.cilium/lb-ipam-ips: "192.168.1.160"
+
+  # Allow the controller to land on the control-plane node.
+  tolerations:
+    - key: node-role.kubernetes.io/control-plane
+      operator: Exists
+      effect: NoSchedule
+
+  # Use the ingress-nginx IngressClass by default.
+  ingressClassResource:
+    default: true
+
+  # Required when using hostPort so DNS resolves correctly inside the pod.
+  dnsPolicy: ClusterFirstWithHostNet
+
+  # Reduce noise in a single-admin homelab.
+  admissionWebhooks:
+    enabled: false
+
+  # ── Connection timeouts to upstreams ──────────────────────────────────────────
+  # Increased to tolerate 5+ second pod-to-pod network latency spikes.
+  # Default: 60s for all — acceptable but explicitly set for clarity.
+  config:
+    upstream-connect-timeout: "60"
+    upstream-send-timeout: "60"
+    upstream-read-timeout: "60"
+    keepalive-timeout: "65"
+    keepalive-requests: "100"
+
+  resources:
+    requests:
+      cpu: 50m
+      memory: 64Mi
+    limits:
+      cpu: 200m
+      memory: 128Mi
+
+  # RED metrics (rate/errors/duration) for every host fronted by this controller —
+  # every exposed service in the cluster goes through here, so this single block
+  # is the cluster-wide "latency and availability" signal. Prometheus auto-discovers
+  # the ServiceMonitor (serviceMonitorSelectorNilUsesHelmValues: false in prometheus-values.yaml).
+  metrics:
+    enabled: true
+    serviceMonitor:
+      enabled: true
+      namespace: ingress-nginx
+      interval: 30s
+      scrapeTimeout: 60s
@@ -0,0 +1,22 @@
+# k8s/ingress/wildcard-cert.yaml
+# Single wildcard TLS certificate for all *.riotpiao.homelab.com services.
+# Lives in the ingress-nginx namespace and is set as nginx's default-ssl-certificate,
+# so every ingress host gets it automatically — no per-service TLS blocks needed.
+#
+# Renewal: cert-manager auto-renews 30 days before expiry (renewBefore: 720h).
+# nginx detects the secret update via its K8s watch and hot-reloads — no pod restart.
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: homelab-tls
+  namespace: ingress-nginx
+spec:
+  secretName: homelab-tls
+  dnsNames:
+    - "*.riotpiao.homelab.com"
+    - "riotpiao.homelab.com"
+  issuerRef:
+    name: homelab-ca
+    kind: ClusterIssuer
+  duration: 8760h      # 1 year
+  renewBefore: 720h    # renew 30 days before expiry
@@ -0,0 +1,299 @@
+# Ollama LLM Inference Service
+
+CPU-only LLM inference server on talos-cp-1. Single model hot-loaded (DeepSeek-R1:70b), 42GB, 70Gi memory limit.
+
+## Quick Start
+
+### Access via port-forward
+```bash
+kubectl -n llm port-forward svc/ollama 11434:11434
+curl http://localhost:11434/api/tags
+```
+
+### Debug pod (in-cluster)
+```bash
+kubectl run debug --rm -it -n llm --image=curlimages/curl \
+  --labels="app.kubernetes.io/role=llm-debug" \
+  --serviceaccount=llm-worker -- sh
+
+# Inside pod
+TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
+curl -H "Authorization: Bearer $TOKEN" \
+  http://ollama.llm.svc.cluster.local:11434/api/tags
+```
+
+## Architecture
+
+| Component | Value |
+|-----------|-------|
+| Service | ClusterIP `ollama.llm.svc.cluster.local:11434` |
+| Namespace | `llm` |
+| Node | talos-cp-1 (pinned via nodeAffinity) |
+| Memory request | 50Gi |
+| Memory limit | 70Gi |
+| Storage | 115Gi PVC (Longhorn) |
+| Model | `deepseek-r1:70b` (~42GB) |
+| Max loaded | 1 model |
+| Parallelism | 1 request at a time |
+
+## API Endpoints
+
+### List models
+```bash
+curl http://ollama.llm.svc.cluster.local:11434/api/tags
+```
+
+Response:
+```json
+{
+  "models": [
+    {"name": "deepseek-r1:70b", "size": 42000000000, ...}
+  ]
+}
+```
+
+### Generate (non-streaming)
+```bash
+curl -X POST http://ollama.llm.svc.cluster.local:11434/api/generate \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "deepseek-r1:70b",
+    "prompt": "Why is the sky blue?",
+    "stream": false
+  }'
+```
+
+### Pull model
+```bash
+curl -X POST http://ollama.llm.svc.cluster.local:11434/api/pull \
+  -H "Content-Type: application/json" \
+  -d '{"name": "deepseek-r1:70b", "stream": false}'
+```
+
+## Operations
+
+### Check pod status
+```bash
+kubectl -n llm get pod -l app.kubernetes.io/name=ollama
+kubectl -n llm describe pod -l app.kubernetes.io/name=ollama
+```
+
+### View logs
+```bash
+kubectl -n llm logs deployment/ollama -f
+```
+
+### Monitor download progress (bootstrap)
+```bash
+kubectl -n llm logs -f job/bootstrap-models -c model-download
+```
+
+### Restart deployment
+```bash
+kubectl -n llm rollout restart deployment/ollama
+```
+
+## Storage
+
+- **PVC:** `ollama-models-cache`, 115Gi, Longhorn StorageClass
+- **Mount:** `/root/.ollama/models` (Ollama model cache)
+- **Lifecycle:** RWO (Read-Write-Once), tied to talos-cp-1
+
+### Resize PVC
+⚠️ PVCs can only expand, not shrink. Edit values.yaml and redeploy:
+
+```yaml
+pvc:
+  size: 120Gi  # increase only
+```
+
+```bash
+vsource .env && helmfile -f helmfile.yaml.gotmpl -l name=ollama apply
+```
+
+## Networking
+
+### NetworkPolicy
+- Default-deny ingress on Ollama pods
+- Allow from pods labeled `app.kubernetes.io/name: llm-worker` (port 11434)
+- Allow from pods labeled `app.kubernetes.io/role: llm-debug` (port 11434)
+
+View policy:
+```bash
+kubectl -n llm get networkpolicy ollama
+```
+
+Test access from external pod (should fail):
+```bash
+kubectl run test --rm -it --image=curlimages/curl -- \
+  curl http://ollama.llm.svc.cluster.local:11434/
+# Connection timeout (correct)
+```
+
+Test access from debug pod (should succeed):
+```bash
+kubectl -n llm logs job/bootstrap-models  # verify bootstrap completed
+# Then run debug pod as shown above
+```
+
+## Configuration
+
+### Helm values (`k8s/llm/charts/ollama/values.yaml`)
+
+```yaml
+resources:
+  requests:
+    cpu: 8
+    memory: 50Gi
+  limits:
+    cpu: 16
+    memory: 70Gi
+
+env:
+  OLLAMA_MAX_LOADED_MODELS: "1"
+  OLLAMA_NUM_PARALLEL: "1"
+  OLLAMA_MAX_QUEUE: "32"
+  OLLAMA_KEEP_ALIVE: "-1"
+  OLLAMA_HOST: "0.0.0.0:11434"
+
+preloadJob:
+  enabled: true
+  hotModels:
+    - deepseek-r1:70b
+```
+
+### Environment variables
+
+| Variable | Value | Purpose |
+|----------|-------|---------|
+| `OLLAMA_MODELS` | `/root/.ollama/models` | Model cache dir |
+| `OLLAMA_MAX_LOADED_MODELS` | `1` | Max concurrent models in RAM |
+| `OLLAMA_NUM_PARALLEL` | `1` | Parallel request threads |
+| `OLLAMA_MAX_QUEUE` | `32` | Request queue depth |
+| `OLLAMA_KEEP_ALIVE` | `-1` | Keep model resident (never unload) |
+| `OLLAMA_HOST` | `0.0.0.0:11434` | Bind address |
+
+Tune `OLLAMA_NUM_PARALLEL` based on CPU cores. Current: 1 (conservative, CPU bottleneck).
+
+## Model Management
+
+### Current model
+- **Name:** `deepseek-r1:70b`
+- **Size:** ~42GB
+- **Quantization:** Default Ollama quant
+- **Status:** Downloaded during pod init via bootstrap job
+
+### Change model
+
+1. Edit `values.yaml`:
+```yaml
+preloadJob:
+  hotModels:
+    - deepseek-r1:32b  # or any available model
+```
+
+2. Redeploy:
+```bash
+kubectl -n llm delete job bootstrap-models --ignore-not-found
+vsource .env && helmfile -f helmfile.yaml.gotmpl -l name=ollama apply
+```
+
+3. Monitor:
+```bash
+kubectl -n llm logs -f job/bootstrap-models -c model-download
+```
+
+### Available models
+Ollama registry: https://ollama.com/library
+
+Examples:
+- `deepseek-r1:70b` (reasoning, 42GB)
+- `deepseek-r1:32b` (faster, 20GB)
+- `llama3.1:70b` (general, 41GB)
+- `mistral:large` (26GB)
+
+## Troubleshooting
+
+### Pod stuck in `ContainerCreating`
+```bash
+kubectl -n llm describe pod -l app.kubernetes.io/name=ollama
+# Check Events section for PVC/image pull issues
+```
+
+### Bootstrap job failing
+```bash
+kubectl -n llm logs job/bootstrap-models -c model-download --tail=50
+# Common: model not found in registry, disk full, network timeout
+```
+
+### Model pull timeout
+```bash
+# Increase pod timeout (edit deployment directly)
+kubectl -n llm edit deployment ollama
+# Change readinessProbe.initialDelaySeconds, livenessProbe.periodSeconds
+```
+
+### Out of memory
+Model size exceeds limit. Reduce `memory.limits` or choose smaller model.
+
+```bash
+kubectl top pod -n llm  # check actual usage
+```
+
+### Cannot connect from other pods
+Verify NetworkPolicy:
+```bash
+kubectl -n llm get networkpolicy
+kubectl -n llm describe networkpolicy ollama
+# Add pod label: app.kubernetes.io/name: llm-worker or app.kubernetes.io/role: llm-debug
+```
+
+## Secrets
+
+Ollama pod receives MinIO credentials via Secret `ollama-minio` (created by helmfile presync):
+
+```bash
+kubectl -n llm get secret ollama-minio -o jsonpath='{.data}' | jq
+```
+
+Keys: `endpoint`, `bucket`, `access_key`, `secret_key`
+
+Used by bootstrap job to upload model blobs to MinIO (future: auto-backup).
+
+## Metrics & Observability
+
+### Prometheus scrape (if enabled)
+ServiceMonitor: Not yet configured (see `k8s/monitoring/dashboards/services/`)
+
+Metrics to add:
+- `ollama_requests_total` (counter)
+- `ollama_request_duration_seconds` (histogram)
+- `ollama_loaded_models` (gauge)
+
+### Logs
+Pod logs via kubectl. No log aggregation to Loki yet.
+
+```bash
+kubectl -n llm logs deployment/ollama -f --timestamps
+```
+
+## Cleanup
+
+### Delete Ollama completely
+```bash
+vsource .env && helmfile -f helmfile.yaml.gotmpl -l name=ollama destroy
+# Keeps PVC (data safety). To delete: kubectl -n llm delete pvc ollama-models-cache
+```
+
+### Delete just the model cache (keep deployment)
+```bash
+kubectl -n llm delete pvc ollama-models-cache
+# Recreate: kubectl -n llm patch deployment ollama -p '{"spec":{"template":{"metadata":{"annotations":{"restart":"now"}}}}}'
+```
+
+## See Also
+
+- Helmfile: `helmfile.yaml.gotmpl` (llm release block)
+- Chart: `k8s/llm/charts/ollama/`
+- Namespace: `llm`
+- Bootstrap: `k8s/llm/bootstrap-models-job.yaml` (manual preload fallback)
@@ -0,0 +1,6 @@
+apiVersion: v2
+name: ollama
+description: CPU-only Ollama LLM server with MinIO model registry
+type: application
+version: 0.1.0
+appVersion: "latest"
@@ -0,0 +1,126 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: ollama
+  namespace: llm
+  labels:
+    app.kubernetes.io/name: ollama
+    app.kubernetes.io/part-of: llm
+spec:
+  replicas: {{ .Values.replicaCount }}
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: ollama
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: ollama
+        app.kubernetes.io/part-of: llm
+    spec:
+      affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+              - matchExpressions:
+                  - key: topology.kubernetes.io/zone
+                    operator: In
+                    values:
+                      - {{ .Values.nodeAffinity.zone }}
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Equal
+          value: ""
+          effect: NoSchedule
+
+      initContainers:
+        - name: preload-model
+          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
+          imagePullPolicy: {{ .Values.image.pullPolicy }}
+          command:
+            - sh
+            - -c
+            - |
+              set -e
+              echo "Starting Ollama server for model preload..."
+              ollama serve &
+              OLLAMA_PID=$!
+              sleep 10
+              {{- range .Values.preloadJob.hotModels }}
+              echo "Preloading {{ . }}..."
+              if ollama ls | grep -q "{{ . }}"; then
+                echo "✓ {{ . }} already cached"
+              else
+                ollama pull {{ . }}
+              fi
+              {{- end }}
+              echo "Model preload complete"
+              kill $OLLAMA_PID || true
+              wait $OLLAMA_PID 2>/dev/null || true
+          volumeMounts:
+            - name: models-cache
+              mountPath: /root/.ollama/models
+          env:
+            - name: OLLAMA_HOST
+              value: "127.0.0.1:11434"
+
+      containers:
+        - name: ollama
+          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
+          imagePullPolicy: {{ .Values.image.pullPolicy }}
+          ports:
+            - containerPort: 11434
+              name: http
+          env:
+            {{- range $key, $value := .Values.env }}
+            - name: {{ $key }}
+              value: "{{ $value }}"
+            {{- end }}
+            - name: OLLAMA_MODELS_MINIO_ENDPOINT
+              valueFrom:
+                secretKeyRef:
+                  name: ollama-minio
+                  key: endpoint
+            - name: OLLAMA_MODELS_MINIO_BUCKET
+              valueFrom:
+                secretKeyRef:
+                  name: ollama-minio
+                  key: bucket
+            - name: OLLAMA_MODELS_MINIO_ACCESS_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: ollama-minio
+                  key: access_key
+            - name: OLLAMA_MODELS_MINIO_SECRET_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: ollama-minio
+                  key: secret_key
+          resources:
+            requests:
+              cpu: {{ .Values.resources.requests.cpu }}
+              memory: {{ .Values.resources.requests.memory }}
+            limits:
+              cpu: {{ .Values.resources.limits.cpu }}
+              memory: {{ .Values.resources.limits.memory }}
+          livenessProbe:
+            httpGet:
+              path: /
+              port: 11434
+            initialDelaySeconds: 30
+            periodSeconds: 10
+          readinessProbe:
+            httpGet:
+              path: /
+              port: 11434
+            initialDelaySeconds: 10
+            periodSeconds: 5
+          volumeMounts:
+            - name: models-cache
+              mountPath: /root/.ollama/models
+
+      volumes:
+        - name: models-cache
+          persistentVolumeClaim:
+            claimName: ollama-models-cache
@@ -0,0 +1,28 @@
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: ollama-access
+  namespace: llm
+  labels:
+    app.kubernetes.io/name: ollama
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/name: ollama
+  policyTypes:
+    - Ingress
+  ingress:
+    - from:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: llm-worker
+      ports:
+        - protocol: TCP
+          port: 11434
+    - from:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/role: llm-debug
+      ports:
+        - protocol: TCP
+          port: 11434
@@ -0,0 +1,92 @@
+{{- if .Values.preloadJob.enabled }}
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: ollama-preload
+  namespace: llm
+  labels:
+    app.kubernetes.io/name: ollama-preload
+spec:
+  backoffLimit: 3
+  template:
+    spec:
+      serviceAccountName: default
+      restartPolicy: Never
+      affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+              - matchExpressions:
+                  - key: topology.kubernetes.io/zone
+                    operator: In
+                    values:
+                      - az-a
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Equal
+          effect: NoSchedule
+      initContainers:
+        - name: model-cache-init
+          image: ollama/ollama:latest
+          imagePullPolicy: IfNotPresent
+          command:
+            - sh
+            - -c
+            - |
+              set -e
+              echo "Starting Ollama server to cache models..."
+              ollama serve &
+              OLLAMA_PID=$!
+              sleep 10
+              echo "Caching hot-tier models..."
+              {{- range .Values.preloadJob.hotModels }}
+              echo "Checking if {{ . }} is cached..."
+              if ollama ls | grep -q "{{ . }}"; then
+                echo "✓ {{ . }} already cached, skipping"
+              else
+                echo "Pulling {{ . }}..."
+                ollama pull {{ . }}
+              fi
+              {{- end }}
+              echo "Model cache initialization complete"
+              kill $OLLAMA_PID || true
+              wait $OLLAMA_PID 2>/dev/null || true
+          volumeMounts:
+            - name: models
+              mountPath: /root/.ollama
+          env:
+            - name: OLLAMA_HOST
+              value: "127.0.0.1:11434"
+
+      containers:
+        - name: cache-populate
+          image: curlimages/curl:latest
+          imagePullPolicy: IfNotPresent
+          command:
+            - sh
+            - -c
+            - |
+              set -e
+              echo "Waiting for Ollama pod to be ready..."
+              until curl -f http://ollama.llm.svc.cluster.local:11434/api/tags 2>/dev/null; do
+                echo "Ollama not ready, waiting..."
+                sleep 5
+              done
+              echo "Ollama is ready, populating local cache..."
+              {{- range .Values.preloadJob.hotModels }}
+              echo "Checking if {{ . }} is already cached..."
+              if curl -s http://ollama.llm.svc.cluster.local:11434/api/tags | grep -q "{{ . }}"; then
+                echo "✓ {{ . }} already cached, skipping"
+              else
+                echo "Caching {{ . }} locally..."
+                curl -X POST http://ollama.llm.svc.cluster.local:11434/api/pull \
+                  -H "Content-Type: application/json" \
+                  -d '{"name":"{{ . }}","stream":false}'
+              fi
+              {{- end }}
+              echo "Local cache population complete"
+
+      volumes:
+        - name: models
+          emptyDir: {}
+{{- end }}
@@ -0,0 +1,14 @@
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: ollama-models-cache
+  namespace: llm
+  labels:
+    app.kubernetes.io/name: ollama
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: {{ .Values.pvc.storageClassName }}
+  resources:
+    requests:
+      storage: {{ .Values.pvc.size }}
@@ -0,0 +1,16 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: ollama
+  namespace: llm
+  labels:
+    app.kubernetes.io/name: ollama
+spec:
+  type: {{ .Values.service.type }}
+  ports:
+    - port: {{ .Values.service.port }}
+      targetPort: http
+      protocol: TCP
+      name: http
+  selector:
+    app.kubernetes.io/name: ollama
@@ -0,0 +1,12 @@
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: longhorn-llm
+  labels:
+    app.kubernetes.io/name: ollama
+provisioner: driver.longhorn.io
+parameters:
+  numberOfReplicas: "1"
+  staleReplicaTimeout: "2880"
+reclaimPolicy: Retain
+allowVolumeExpansion: true
@@ -0,0 +1,40 @@
+replicaCount: 1
+
+image:
+  repository: ollama/ollama
+  pullPolicy: IfNotPresent
+  tag: "latest"
+
+service:
+  type: ClusterIP
+  port: 11434
+
+resources:
+  requests:
+    cpu: 8
+    memory: 60Gi
+  limits:
+    cpu: 16
+    memory: 100Gi
+
+pvc:
+  enabled: true
+  size: 115Gi
+  storageClassName: longhorn-llm
+
+nodeAffinity:
+  zone: az-a
+
+env:
+  OLLAMA_MODELS: /root/.ollama/models
+  OLLAMA_MAX_LOADED_MODELS: "2"
+  OLLAMA_NUM_PARALLEL: "2"
+  OLLAMA_MAX_QUEUE: "64"
+  OLLAMA_KEEP_ALIVE: "-1"
+  OLLAMA_HOST: "0.0.0.0:11434"
+
+preloadJob:
+  enabled: false
+  hotModels:
+    - ornith:35b
+    - deepseek-r1:70b
@@ -0,0 +1,71 @@
+#!/bin/bash
+
+# setup-minio-bucket.sh
+# Creates MinIO bucket and Kubernetes secrets for Ollama LLM server
+# Runs as helmfile presync hook; all commands are idempotent
+# Dependencies: kubectl, access to minio-az-a pod in storage namespace
+# Environment: MINIO_ROOT_USER, MINIO_ROOT_PASSWORD (from Vault), AUTHENTIK_OLLAMA_CLIENT_ID, AUTHENTIK_OLLAMA_CLIENT_SECRET
+
+set -e
+
+echo "=== Step 1: Create and label llm namespace ==="
+kubectl create namespace llm --dry-run=client -o yaml | kubectl apply -f -
+kubectl label namespace llm \
+  pod-security.kubernetes.io/enforce=baseline \
+  pod-security.kubernetes.io/enforce-version=latest \
+  --overwrite
+
+echo "✓ llm namespace created/labeled"
+
+echo ""
+echo "=== Step 2: Create MinIO bucket riotpiao-models ==="
+
+# Configure mc host inside MinIO pod
+kubectl -n storage exec deployment/minio-az-a -- \
+  mc config host add local http://localhost:9000 \
+  "${MINIO_ROOT_USER}" "${MINIO_ROOT_PASSWORD}"
+
+echo "✓ mc host configured"
+
+# Create bucket (idempotent)
+kubectl -n storage exec deployment/minio-az-a -- \
+  mc mb --ignore-existing local/riotpiao-models
+
+echo "✓ MinIO bucket riotpiao-models created (or already exists)"
+
+# Enable versioning for model rollback safety
+kubectl -n storage exec deployment/minio-az-a -- \
+  mc version enable local/riotpiao-models
+
+echo "✓ Versioning enabled on riotpiao-models bucket"
+
+echo ""
+echo "=== Step 3: Create ollama-minio Secret (MinIO credentials) ==="
+
+kubectl create secret generic ollama-minio -n llm \
+  --from-literal=endpoint="http://minio-az-a.storage:9000" \
+  --from-literal=bucket="riotpiao-models" \
+  --from-literal=access_key="${MINIO_ROOT_USER}" \
+  --from-literal=secret_key="${MINIO_ROOT_PASSWORD}" \
+  --dry-run=client -o yaml | kubectl apply -f -
+
+echo "✓ Secret ollama-minio created/updated"
+
+echo ""
+echo "=== Step 4: Create ollama-oidc Secret (Authentik credentials) ==="
+
+kubectl create secret generic ollama-oidc -n llm \
+  --from-literal=client_id="${AUTHENTIK_OLLAMA_CLIENT_ID}" \
+  --from-literal=client_secret="${AUTHENTIK_OLLAMA_CLIENT_SECRET}" \
+  --dry-run=client -o yaml | kubectl apply -f -
+
+echo "✓ Secret ollama-oidc created/updated"
+
+echo ""
+echo "=== Verification ==="
+echo ""
+echo "Run these commands to verify:"
+echo "  kubectl -n llm get secret ollama-minio ollama-oidc"
+echo "  kubectl -n storage exec deployment/minio-az-a -- mc ls local/riotpiao-models"
+echo ""
+echo "Setup complete!"
@@ -0,0 +1,54 @@
+#!/usr/bin/env bash
+# portainer/bootstrap.sh
+# Deploys Portainer CE into the dashboard namespace.
+# No credentials needed — Portainer prompts you to create an admin account
+# on first browser visit.
+#
+# Prerequisites:
+#   - kubectl configured (KUBECONFIG pointing to cluster-config/kubeconfig)
+#   - helm >= 3.x
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
+KUBECONFIG="${KUBECONFIG:-${REPO_ROOT}/cluster-config/kubeconfig}"
+export KUBECONFIG
+
+# ── Namespace ─────────────────────────────────────────────────────────────────
+echo "==> Creating dashboard namespace..."
+kubectl create namespace dashboard --dry-run=client -o yaml | kubectl apply -f -
+kubectl label namespace dashboard \
+  pod-security.kubernetes.io/enforce=privileged \
+  pod-security.kubernetes.io/enforce-version=latest \
+  --overwrite
+
+# ── Helm repo ─────────────────────────────────────────────────────────────────
+echo "==> Adding Portainer Helm repo..."
+helm repo add portainer https://portainer.github.io/k8s/
+helm repo update portainer
+
+# ── Portainer ─────────────────────────────────────────────────────────────────
+echo "==> Installing Portainer..."
+helm upgrade --install portainer portainer/portainer \
+  --namespace dashboard \
+  --values "${SCRIPT_DIR}/portainer-values.yaml" \
+  --wait \
+  --timeout 5m
+
+echo "==> Waiting for Portainer Deployment to be ready..."
+kubectl rollout status deployment/portainer -n dashboard --timeout=120s
+
+# ── Done ──────────────────────────────────────────────────────────────────────
+echo ""
+echo "==> Portainer is up."
+echo ""
+echo "Access Portainer UI:"
+echo "  make pf-portainer"
+echo "  http://localhost:9000"
+echo ""
+echo "First-time setup: Portainer will prompt you to create an admin account."
+echo "Choose 'Manage the local Kubernetes environment' when asked."
+echo ""
+echo "Node failure resilience tip:"
+echo "  For faster PVC failover on hard node failure, enable in Longhorn UI → Settings:"
+echo "    nodeDownPodDeletionPolicy = delete-deployment-pod"
@@ -0,0 +1,53 @@
+# k8s/portainer/portainer-values.yaml
+# Portainer — web UI for browsing cluster workloads, exec-ing into pods,
+# and viewing logs without kubectl. Operator-only access (ClusterIP + port-forward).
+#
+# Node failure behaviour:
+#   Portainer is a Deployment (not StatefulSet), so K8s auto-evicts and
+#   reschedules it ~5 min after a node becomes unreachable. Longhorn
+#   reattaches the PVC on the new node in ~1-2 min. Worst case: ~7-10 min.
+#
+#   To cut that down: in Longhorn UI → Settings set
+#     nodeDownPodDeletionPolicy = delete-deployment-pod
+#   Longhorn will force-delete the stuck pod immediately when the node is
+#   fenced rather than waiting for Kubernetes' eviction timeout.
+
+# ── Service ───────────────────────────────────────────────────────────────────
+# ClusterIP — no external exposure. Access via:
+#   kubectl -n dashboard port-forward svc/portainer 9000:9000
+# Portainer holds cluster-admin credentials; never expose as LoadBalancer.
+service:
+  type: ClusterIP
+
+# ── TLS ───────────────────────────────────────────────────────────────────────
+# Portainer by default redirects HTTP → HTTPS using a self-signed cert.
+# force: false disables the redirect so plain HTTP over port-forward works
+# without browser cert warnings. TLS is terminated at the ingress layer
+# if/when an ingress rule is added.
+tls:
+  force: false
+
+# ── Persistence ───────────────────────────────────────────────────────────────
+# Stores Portainer's own config: environment registrations, user accounts,
+# stack definitions, and access control settings. Longhorn provides the
+# RWO block volume. 10Gi is generous for config data but cheap on Longhorn.
+persistence:
+  enabled: true
+  storageClass: "longhorn"
+  size: 10Gi
+
+resources:
+  requests:
+    cpu: 100m
+    memory: 128Mi
+  limits:
+    cpu: 500m
+    memory: 512Mi
+
+# ── Scheduling ────────────────────────────────────────────────────────────────
+# Allow scheduling on talos-cp-1 (carries NoSchedule taint) so Portainer
+# keeps running even when the worker node is down.
+tolerations:
+  - key: node-role.kubernetes.io/control-plane
+    operator: Exists
+    effect: NoSchedule
@@ -0,0 +1,183 @@
+# Authentik Federated OIDC & SSO
+
+**Provider:** `https://authentik.riotpiao.homelab.com`  
+**OIDC Issuer:** `https://authentik.riotpiao.homelab.com/application/o/talos-federation/`  
+**Namespace:** `iam`
+
+## When to Use
+
+- **Federated login** — Single sign-on for Grafana, MinIO, Forgejo, Argo CD
+- **User groups** — RBAC via group membership (admins, devops, read-only)
+- **JWT tokens** — Authenticate CLI tools, API clients
+- **SSO for custom apps** — OAuth2/OIDC redirect flow
+
+## Quick Start
+
+**1. Login to Authentik console:**
+```bash
+# Browser: https://authentik.riotpiao.homelab.com
+# Default user: akadmin
+# Password: AUTHENTIK_BOOTSTRAP_PASSWORD (from .env)
+
+# Or via OIDC (after initial setup)
+# Click "Sign in with talos-federation"
+```
+
+**2. Create user:**
+```
+Authentik console → Users → Create
+- Username: alice
+- Email: alice@example.com
+- Group: homelab-devs (or homelab-admins)
+```
+
+**3. User logs into Grafana:**
+```
+https://grafana.riotpiao.homelab.com
+→ Sign in with Authentik (auto-redirects to OIDC provider)
+→ Approve access
+→ Logged in as alice (group determines role: Admin or Viewer)
+```
+
+## Configuration
+
+| Key | Value |
+|-----|-------|
+| OIDC provider | `talos-federation` (federated) |
+| OIDC issuer | `https://authentik.riotpiao.homelab.com/application/o/talos-federation/` |
+| JWKS endpoint | `https://authentik.riotpiao.homelab.com/application/o/talos-federation/.well-known/openid-configuration` |
+| Database | PostgreSQL (ddb namespace, authentik user) |
+| Backups | WAL archived to MinIO |
+
+## Common Patterns
+
+**Grafana OIDC login:**
+```yaml
+# k8s/logging/grafana-values.yaml
+grafana:
+  auth.generic_oauth:
+    enabled: true
+    name: Authentik
+    client_id: grafana
+    client_secret: $GRAFANA_OIDC_CLIENT_SECRET  # from Vault
+    auth_url: https://authentik.riotpiao.homelab.com/application/o/authorize/
+    token_url: https://authentik.riotpiao.homelab.com/application/o/token/
+    api_url: https://authentik.riotpiao.homelab.com/application/o/userinfo/
+    scopes: openid profile email groups
+    use_pkce: true
+```
+
+**MinIO OIDC login:**
+```yaml
+# k8s/storage/minio-values.yaml
+minio:
+  identity_oauth:
+    provider: authentik
+    client_id: minio
+    client_secret: $MINIO_OIDC_CLIENT_SECRET
+    redirect_uri: https://minio.riotpiao.homelab.com/oauth_callback
+    config_url: https://authentik.riotpiao.homelab.com/application/o/talos-federation/.well-known/openid-configuration
+    policy_mappings:
+      - group: homelab-admins → consoleAdmin
+      - group: homelab-devops → readwrite
+```
+
+**CLI device code flow (talos-cli):**
+```bash
+# Get JWT token (no kubeconfig needed)
+talos secrets login
+# → Opens browser, approve device code
+# → Token cached in ~/.talos/token
+
+# Use token to access Vault
+talos get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
+# → Vault validates JWT from Authentik
+# → Returns secret
+```
+
+**Custom app OIDC redirect:**
+```go
+import "github.com/coreos/go-oidc/v3/oidc"
+
+provider, _ := oidc.NewProvider(ctx, "https://authentik.riotpiao.homelab.com/application/o/talos-federation/")
+
+verifier := provider.Verifier(&oidc.Config{ClientID: "my-app"})
+
+// After OAuth2 redirect & token exchange:
+idToken, _ := verifier.Verify(ctx, rawIDToken)
+
+// Extract claims
+var claims struct {
+  Email string `json:"email"`
+  Groups []string `json:"groups"`
+}
+idToken.Claims(&claims)
+```
+
+## Group-Based RBAC
+
+**Default groups:**
+- `homelab-admins` — Full cluster access (Grafana Admin, MinIO admin, Argo CD admin, Vault admin)
+- `homelab-devops` — Deploy & monitor (Grafana Editor, MinIO readwrite, Argo CD user)
+- `homelab-viewers` — Read-only (Grafana Viewer, MinIO readonly)
+
+**Assign user to group:**
+```
+Authentik console → Users → alice → Edit
+→ Groups → Add "homelab-devops"
+→ Save
+```
+
+**Custom group-to-role mapping:**
+```yaml
+# Per-service (see cicd-workflow.md, monitoring-metrics.md for examples)
+# Grafana: auth.generic_oauth.role_attribute_path = contains(groups[*], 'homelab-admins') && 'Admin' || 'Viewer'
+# MinIO: policy_mappings (see above)
+```
+
+## Monitoring
+
+**Authentik dashboard:** https://authentik.riotpiao.homelab.com/api/v3/admin/dashboards
+
+**Key metrics:**
+- Login attempts (success/failure)
+- Active sessions
+- Token issuance rate
+- Provider sync status
+
+## Troubleshooting
+
+**Users can't login (redirect loop):**
+```bash
+# Check redirect URI matches
+# Authentik console → Applications → grafana → Edit
+# Verify Redirect URI = https://grafana.riotpiao.homelab.com/login/generic_oauth
+
+# Check OIDC provider is running
+k get pods -n iam -l app=authentik
+```
+
+**JWT token expired:**
+```bash
+# CLI tokens have 24h expiry
+# Re-authenticate
+talos secrets login
+```
+
+**Groups not syncing:**
+```bash
+# Check group attribute in OIDC config
+# Authentik console → Applications → <app> → OIDC Configuration
+# groups_attribute = "groups" (or custom claim name)
+```
+
+**Vault can't validate JWT:**
+```bash
+# Verify JWKS endpoint is accessible
+curl https://authentik.riotpiao.homelab.com/application/o/talos-federation/.well-known/openid-configuration
+
+# Restart Vault to refresh JWKS cache
+k rollout restart -n iam deployment/vault
+```
+
+See `/TROUBLESHOOTING.md` for full incident guide.
@@ -0,0 +1,222 @@
+# CI/CD Pipeline (Forgejo + Argo CD)
+
+**Git Forge:** `https://forgejo.riotpiao.homelab.com`  
+**Deployments:** `https://argocd.riotpiao.homelab.com` (or `kubectl port-forward`)  
+**Namespaces:** `cicd`, `forge`
+
+## When to Use
+
+- **Build & test** — Forgejo Actions CI (GitHub Actions syntax)
+- **Image push** — Build OCI images, push to Forgejo registry
+- **GitOps deployment** — Argo CD syncs deploy repo to cluster
+- **Secrets in CI** — ci-bot JWT tokens, never kubeconfig
+
+## Quick Start
+
+**1. Clone a repo from Forgejo:**
+```bash
+git clone https://forgejo.riotpiao.homelab.com/rock/source.git
+cd source
+```
+
+**2. Create workflow:**
+```bash
+mkdir -p .forgejo/workflows
+cat > .forgejo/workflows/ci.yml <<EOF
+name: CI
+on: [push]
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - run: npm test
+      - run: docker build -t myapp:latest .
+      - run: |
+          docker login forgejo.riotpiao.homelab.com \
+            -u ci-bot \
+            -p ${{ secrets.CI_BOT_TOKEN }}
+          docker push forgejo.riotpiao.homelab.com/rock/myapp:latest
+EOF
+
+git add .forgejo/workflows/ci.yml
+git commit -m "ci: add build workflow"
+git push
+```
+
+**3. Trigger deployment:**
+```bash
+# Update deployment repo (rock/deploy)
+git clone https://forgejo.riotpiao.homelab.com/rock/deploy.git
+cd deploy
+
+# Update image tag
+sed -i 's|forgejo.riotpiao.homelab.com/rock/myapp:.*|forgejo.riotpiao.homelab.com/rock/myapp:abc123|' k8s/deployment.yaml
+
+git add k8s/deployment.yaml
+git commit -m "deploy: bump myapp to abc123"
+git push
+```
+
+**4. Argo CD auto-syncs:**
+```bash
+# Watch deployment
+k rollout status -n <app-namespace> deployment/<app-name>
+
+# Or check Argo CD UI
+kubectl port-forward -n argocd svc/argocd-server 8443:443
+# https://localhost:8443 (login via Authentik)
+```
+
+## Workflow Syntax (GitHub Actions)
+
+**Basic structure:**
+```yaml
+name: CI
+on:
+  push:
+    branches: [main, develop]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - run: npm ci
+      - run: npm test
+      
+  build:
+    needs: test  # wait for test job
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - run: docker build -t myapp:${{ github.sha }} .
+      - run: docker push forgejo.riotpiao.homelab.com/rock/myapp:${{ github.sha }}
+```
+
+**Available variables:**
+```bash
+${{ github.sha }}              # commit hash
+${{ github.ref_name }}         # branch name
+${{ github.run_number }}       # build #
+${{ secrets.CI_BOT_TOKEN }}    # injected from Forgejo
+```
+
+## Secrets & Authentication
+
+**ci-bot JWT token (auto-injected):**
+```yaml
+- run: |
+    echo "${{ secrets.CI_BOT_TOKEN }}" | docker login \
+      forgejo.riotpiao.homelab.com \
+      -u ci-bot \
+      --password-stdin
+    docker push forgejo.riotpiao.homelab.com/rock/myapp:latest
+```
+
+**API token (for pushing commits):**
+```yaml
+- run: |
+    git config user.name "ci-bot"
+    git config user.email "ci-bot@homelab"
+    git commit --allow-empty -m "bump: version"
+    git push https://ci-bot:${{ secrets.CI_BOT_TOKEN }}@forgejo.riotpiao.homelab.com/rock/deploy.git main
+```
+
+**Vault secrets (via talos CLI):**
+```bash
+# Not available in CI runner — use Argo CD post-sync hooks instead
+# Or inject via init container before workflow runs
+```
+
+## Argo CD GitOps
+
+**Create app (one-time):**
+```bash
+argocd app create story-crater \
+  --repo https://forgejo.riotpiao.homelab.com/rock/deploy.git \
+  --path k8s/ \
+  --dest-server https://kubernetes.default.svc \
+  --dest-namespace story-crater-backend \
+  --sync-policy automated
+```
+
+**Monitor sync:**
+```bash
+# CLI
+argocd app get story-crater
+argocd app logs story-crater
+
+# UI: https://argocd.riotpiao.homelab.com
+# Login: Authentik SSO (homelab-admins group only)
+```
+
+**Manual sync:**
+```bash
+argocd app sync story-crater
+argocd app wait story-crater
+```
+
+## Security Rules
+
+✅ **DO:**
+- Store credentials in Forgejo Secrets (auto-injected)
+- Use ci-bot JWT for image push only
+- Commit to deploy repo (triggers Argo CD)
+- Enable branch protection (require CI pass)
+
+❌ **DON'T:**
+- Put kubeconfig in CI (Argo CD bridges gap)
+- Commit secrets to source repo
+- Use admin-bot in CI workflows (over-privileged)
+- Push images directly to cluster (use Argo CD)
+
+## Monitoring
+
+**Grafana dashboard:** `svc-forgejo`, `svc-argocd`
+
+**Key metrics:**
+- `forgejo_workflows_running` — active workflows
+- `argocd_app_sync_duration_seconds` — deployment time
+- `argocd_app_info{sync_status="OutOfSync"}` — drift detection
+
+## Troubleshooting
+
+**Workflow fails silently:**
+```bash
+# Check runner logs
+k logs -n cicd -f deploy/forgejo-runner
+
+# Check if runner pod is healthy
+k get pods -n cicd -l app=forgejo-runner
+```
+
+**Image push fails (401 Unauthorized):**
+```bash
+# Verify ci-bot token in Forgejo
+# Settings → Applications → ci-bot → check scopes (package:write)
+
+# Or re-create token
+talos put cluster/iam/agents/ci-bot-token TOKEN="$(openssl rand -hex 32)"
+```
+
+**Argo CD out-of-sync:**
+```bash
+# Check deploy repo changes
+argocd app diff story-crater
+
+# Manual sync
+argocd app sync story-crater --prune
+```
+
+**Webhook not triggering:**
+```bash
+# Verify Forgejo webhook config
+# Repo Settings → Webhooks → Check delivery logs
+
+# Or manually trigger
+argocd app sync story-crater
+```
+
+See `/TROUBLESHOOTING.md` for full incident guide.
@@ -0,0 +1,188 @@
+# CloudNativePG PostgreSQL Database
+
+**Host:** `ddb-cluster-rw.ddb.svc.cluster.local` (read-write)  
+**Read replica:** `ddb-cluster-ro.ddb.svc.cluster.local` (read-only)  
+**Port:** `5432`  
+**Namespace:** `ddb`
+
+## When to Use
+
+- **Multi-replica HA** — 3 replicas, automatic failover
+- **pgvector extension** — Vector similarity search (LLM embeddings)
+- **Transactional data** — Authentik, Story Crater backend, custom apps
+- **Declarative backups** — Automated WAL archiving to MinIO
+
+## Quick Start
+
+**1. Connect from pod:**
+```bash
+# Inside a pod (inject secret mount)
+psql -h ddb-cluster-rw.ddb.svc.cluster.local \
+  -U story_crater \
+  -d story_crater \
+  -W  # prompt for password (from Secret)
+```
+
+**2. Create database & user (one-time):**
+```bash
+# Already done by helmfile postsync hook
+# But if needed manually:
+
+psql -h ddb-cluster-rw.ddb.svc.cluster.local \
+  -U postgres \
+  -c "CREATE DATABASE myapp OWNER postgres;"
+
+psql -h ddb-cluster-rw.ddb.svc.cluster.local \
+  -U postgres \
+  -d myapp \
+  -c "CREATE USER myapp_user WITH PASSWORD 'secret';"
+
+psql -h ddb-cluster-rw.ddb.svc.cluster.local \
+  -U postgres \
+  -d myapp \
+  -c "GRANT ALL PRIVILEGES ON DATABASE myapp TO myapp_user;"
+```
+
+**3. Enable pgvector:**
+```bash
+psql -h ddb-cluster-rw.ddb.svc.cluster.local \
+  -U postgres \
+  -d myapp \
+  -c "CREATE EXTENSION IF NOT EXISTS vector;"
+```
+
+**4. Create table with embeddings:**
+```sql
+CREATE TABLE documents (
+  id BIGSERIAL PRIMARY KEY,
+  content TEXT,
+  embedding vector(1536),  -- OpenAI embeddings
+  created_at TIMESTAMP DEFAULT NOW()
+);
+
+CREATE INDEX ON documents USING IVFFLAT (embedding vector_cosine_ops);
+```
+
+## Configuration
+
+| Key | Value |
+|-----|-------|
+| Host (RW) | `ddb-cluster-rw.ddb.svc.cluster.local` |
+| Host (RO) | `ddb-cluster-ro.ddb.svc.cluster.local` |
+| Port | 5432 |
+| Replicas | 3 (automatic failover) |
+| Extensions | pgvector (LLM embeddings), uuid-ossp |
+| Backups | WAL archiving to MinIO (continuous) |
+| Retention | 30 days |
+
+## Common Patterns
+
+**Connection pooling (from app):**
+```go
+import "github.com/jackc/pgx/v5/pgxpool"
+
+config, _ := pgxpool.ParseConfig("postgres://user:pass@ddb-cluster-rw.ddb.svc.cluster.local:5432/myapp")
+config.MaxConns = 25
+config.MinConns = 5
+pool, _ := pgxpool.NewWithConfig(ctx, config)
+
+// Use pool
+row := pool.QueryRow(ctx, "SELECT COUNT(*) FROM users")
+```
+
+**Read from replica (analytics):**
+```go
+// Offload SELECT queries to read replica
+pool.QueryRow(ctx, "SELECT * FROM documents LIMIT 1") // auto-routes to RO if available
+
+// Writes always go to RW
+pool.Exec(ctx, "INSERT INTO documents ...")
+```
+
+**Vector similarity search:**
+```sql
+SELECT id, content, embedding <-> $1 AS distance
+FROM documents
+ORDER BY embedding <-> $1
+LIMIT 10;
+-- $1 = query embedding (e.g., from OpenAI API)
+```
+
+**Backup & restore:**
+```bash
+# Backups are automatic (WAL to MinIO)
+# To restore from backup:
+# 1. Check MinIO s3://postgresql-backups/
+# 2. Use PostgreSQL PITR (point-in-time recovery)
+# 3. Contact SRE for restore procedure
+```
+
+## Monitoring
+
+**Grafana dashboard:** `svc-postgresql` (auto-configured)
+
+**Key metrics:**
+- `pg_stat_activity_connections` — active connections
+- `pg_stat_database_blks_read` — disk I/O
+- `pg_replication_lag_seconds` — replica lag (goal: < 1s)
+
+**CLI health check:**
+```bash
+# Check replication status
+kubectl exec -n ddb pod/ddb-cluster-1 -- \
+  psql -U postgres -c "SELECT slot_name, restart_lsn FROM pg_replication_slots;"
+
+# Check replica lag
+kubectl exec -n ddb pod/ddb-cluster-2 -- \
+  psql -U postgres -c "SELECT now() - pg_last_xact_replay_timestamp() AS lag;"
+```
+
+## Secrets & Credentials
+
+**All user passwords stored in Vault:**
+```bash
+# Read password
+talos get cluster/STORY_CRATER_PG_PASSWORD --key STORY_CRATER_PG_PASSWORD
+
+# Inject into pod (auto via Secret volume)
+# Mount: /run/secrets/db-password
+```
+
+**Connection string from env:**
+```bash
+POSTGRES_CONNECTION="postgres://story_crater:${STORY_CRATER_PG_PASSWORD}@ddb-cluster-rw.ddb.svc.cluster.local:5432/story_crater"
+```
+
+## Troubleshooting
+
+**Cannot connect (connection refused):**
+```bash
+# Verify cluster is running
+k get pods -n ddb
+
+# Check Service DNS
+k exec -it pod/debug-pod -- nslookup ddb-cluster-rw.ddb.svc.cluster.local
+
+# Verify Secret has password
+k get secret -n ddb ddb-cluster-superuser -o jsonpath='{.data.password}' | base64 -d
+```
+
+**Replica lag is high (> 10s):**
+```bash
+# Check replica pod CPU/memory
+k top pod -n ddb
+
+# Scale down other workloads if cluster is overloaded
+# Or scale up database resources (helmfile.yaml.gotmpl)
+```
+
+**pgvector queries slow:**
+```sql
+-- Ensure index exists
+SELECT * FROM pg_indexes WHERE tablename = 'documents' AND indexname LIKE '%embedding%';
+
+-- Re-index if missing
+CREATE INDEX ON documents USING IVFFLAT (embedding vector_cosine_ops);
+```
+
+See `/TROUBLESHOOTING.md` for full incident guide.
@@ -0,0 +1,162 @@
+# MinIO S3-Compatible Object Storage
+
+**Endpoint:** `https://minio.riotpiao.homelab.com` (console)  
+**API:** `minio.storage.svc.cluster.local:9000` (cluster-internal)  
+**Namespace:** `storage`
+
+## When to Use
+
+- **File uploads** — Images, documents, backups
+- **Log backend** — Loki chunks storage
+- **Vault unsealing** — Store unseal keys
+- **CI/CD artifacts** — Build outputs, Docker layers cache
+
+## Quick Start
+
+**1. Access MinIO console:**
+```bash
+# Via browser: https://minio.riotpiao.homelab.com
+# Credentials: MINIO_ROOT_USER / MINIO_ROOT_PASSWORD (from .env)
+
+# Or port-forward
+make pf-minio  # localhost:9001
+```
+
+**2. Create bucket:**
+```bash
+# Via AWS CLI
+export AWS_ACCESS_KEY_ID=$MINIO_ROOT_USER
+export AWS_SECRET_ACCESS_KEY=$MINIO_ROOT_PASSWORD
+
+aws s3 mb s3://my-bucket \
+  --endpoint-url https://minio.riotpiao.homelab.com \
+  --region homelab
+
+# Or via console UI: Click "Create Bucket"
+```
+
+**3. Upload file:**
+```bash
+aws s3 cp /path/to/file.txt s3://my-bucket/ \
+  --endpoint-url https://minio.storage.svc.cluster.local:9000 \
+  --use-path-style
+```
+
+**4. List buckets:**
+```bash
+aws s3 ls --endpoint-url https://minio.storage.svc.cluster.local:9000
+```
+
+## Configuration
+
+| Key | Value |
+|-----|-------|
+| Access key | `MINIO_ROOT_USER` (from .env) |
+| Secret key | `MINIO_ROOT_PASSWORD` (from .env) |
+| Cluster API | `minio.storage.svc.cluster.local:9000` |
+| Console port | `9001` |
+| Replication | 3-node site-replication (az-a ↔ az-b ↔ az-c) |
+| Buckets (system) | `loki-chunks`, `loki-ruler`, `vault-backups` |
+
+## Common Patterns
+
+**Loki log backend (auto-configured):**
+```yaml
+# k8s/logging/loki-values.yaml
+loki:
+  storage:
+    s3:
+      endpoint: minio.storage.svc.cluster.local:9000
+      buckets: loki-chunks
+      secretAccessKey: $MINIO_ROOT_PASSWORD
+      accessKeyId: $MINIO_ROOT_USER
+```
+
+**Application usage (Go/Python/Node):**
+```go
+import "github.com/minio/minio-go/v7"
+
+client, _ := minio.New("minio.storage.svc.cluster.local:9000", &minio.Options{
+  Creds:  credentials.NewStaticV4(os.Getenv("MINIO_ROOT_USER"), os.Getenv("MINIO_ROOT_PASSWORD"), ""),
+  Secure: false,  // cluster-internal (no TLS)
+})
+
+// Upload
+client.FPutObject(ctx, "my-bucket", "file.txt", "/path/to/file.txt", minio.PutObjectOptions{})
+
+// Download
+client.FGetObject(ctx, "my-bucket", "file.txt", "/tmp/file.txt", minio.GetObjectOptions{})
+```
+
+**Vault backup bucket:**
+```bash
+# Vault stores unseal keys in s3://vault-backups
+# Auto-managed by helmfile; no manual action needed
+```
+
+## Monitoring
+
+**Grafana dashboard:** `svc-minio`
+
+**Key metrics:**
+- `minio_disk_drive_free_bytes` — available space
+- `minio_bucket_usage_object_count` — objects per bucket
+- `minio_bucket_usage_total_bytes` — total size per bucket
+
+**Site replication status:**
+```bash
+# Port-forward to MinIO pod
+k port-forward -n storage pod/minio-0 9000:9000 &
+
+# Check replication
+mc alias set local http://localhost:9000 $MINIO_ROOT_USER $MINIO_ROOT_PASSWORD
+mc admin replicate status local
+```
+
+## Authentication (Cluster-Internal)
+
+**From pods (cluster-internal):**
+```bash
+# Use credentials from Secret or env var
+export MINIO_ENDPOINT=minio.storage.svc.cluster.local:9000
+export MINIO_ACCESS_KEY=$MINIO_ROOT_USER
+export MINIO_SECRET_KEY=$MINIO_ROOT_PASSWORD
+aws s3 ls --endpoint-url http://$MINIO_ENDPOINT --use-path-style
+```
+
+**External access (HTTPS via Ingress):**
+```bash
+# Console: https://minio.riotpiao.homelab.com (port 9001)
+# API: Use AWS CLI with --endpoint-url https://minio.riotpiao.homelab.com:9000
+```
+
+## Troubleshooting
+
+**Bucket creation fails:**
+```bash
+# Check MinIO pod logs
+k logs -n storage pod/minio-0 | grep -i error
+
+# Verify storage space
+k get pvc -n storage
+```
+
+**Site replication lag:**
+```bash
+# Check if all 3 nodes are healthy
+k get pods -n storage -l app=minio
+
+# If one node is down, site replication queues changes (eventually consistent)
+```
+
+**Access denied:**
+```bash
+# Verify credentials in .env
+echo $MINIO_ROOT_USER $MINIO_ROOT_PASSWORD
+
+# If credentials rotated, update Secret
+k patch secret -n storage minio-root-credentials \
+  --type merge -p '{"stringData":{"MINIO_ROOT_PASSWORD":"newpass"}}'
+```
+
+See `/TROUBLESHOOTING.md` for full incident guide.
@@ -0,0 +1,265 @@
+# Monitoring: Prometheus, Grafana & Loki
+
+**Prometheus:** `prometheus-kube-prom-prometheus.monitoring.svc.cluster.local:9090`  
+**Grafana:** `https://grafana.riotpiao.homelab.com`  
+**Loki:** `loki.logging.svc.cluster.local:3100`  
+**Namespaces:** `monitoring`, `logging`
+
+## When to Use
+
+- **Metrics** — CPU, memory, latency, request rate, custom business metrics (Prometheus)
+- **Logs** — Pod logs, kernel logs, trace events (Loki)
+- **Dashboards** — Real-time visualization (Grafana)
+- **Alerts** — Page on high error rate, latency spike, or resource exhaustion
+
+## Quick Start
+
+**1. Access Grafana:**
+```bash
+# Browser: https://grafana.riotpiao.homelab.com
+# Login: admin / GRAFANA_ADMIN_PASSWORD (from .env)
+# Or via Authentik SSO
+
+# Port-forward (if no external access)
+make pf-grafana  # localhost:3000
+```
+
+**2. View dashboards:**
+```
+Grafana → Dashboards → Homelab folder
+- Service Availability (uptime probes + cert expiry)
+- Latency & Golden Signals (request rate/error %/duration)
+- Hardware Statistics (per-node CPU/mem/disk)
+- Kube-Controller Health (API server metrics)
+- Service Internals (per-service deep-dive)
+```
+
+**3. Add Prometheus data source:**
+```
+Grafana → Settings → Data Sources → Add
+- Type: Prometheus
+- URL: http://prometheus-kube-prom-prometheus.monitoring.svc.cluster.local:9090
+- Save & test
+```
+
+**4. Create dashboard:**
+```json
+{
+  "dashboard": {
+    "title": "My Service",
+    "panels": [
+      {
+        "targets": [
+          {
+            "expr": "rate(http_requests_total{service='myapp'}[5m])"
+          }
+        ],
+        "title": "Request Rate"
+      }
+    ]
+  }
+}
+```
+
+## Instrumentation Patterns
+
+**Go (Prometheus):**
+```go
+import "github.com/prometheus/client_golang/prometheus"
+
+// Counter (monotonic increase)
+requestsTotal := prometheus.NewCounterVec(
+  prometheus.CounterOpts{
+    Name: "http_requests_total",
+    Help: "Total requests",
+  },
+  []string{"method", "status"},
+)
+
+// Gauge (can go up or down)
+activeConnections := prometheus.NewGauge(prometheus.GaugeOpts{
+  Name: "active_connections",
+})
+
+// Histogram (latency buckets)
+requestDuration := prometheus.NewHistogramVec(
+  prometheus.HistogramOpts{
+    Name: "http_request_duration_seconds",
+    Buckets: []float64{.001, .01, .1, 1},
+  },
+  []string{"method", "path"},
+)
+
+// Record metric
+requestsTotal.WithLabelValues("POST", "200").Inc()
+requestDuration.WithLabelValues("POST", "/api/users").Observe(0.042)
+```
+
+**Node.js (prom-client):**
+```javascript
+const prometheus = require('prom-client');
+
+const httpRequestDuration = new prometheus.Histogram({
+  name: 'http_request_duration_seconds',
+  help: 'Duration of HTTP requests in seconds',
+  labelNames: ['method', 'path', 'status'],
+  buckets: [0.001, 0.01, 0.1, 1],
+});
+
+// Middleware
+app.use((req, res, next) => {
+  const start = Date.now();
+  res.on('finish', () => {
+    const duration = (Date.now() - start) / 1000;
+    httpRequestDuration.labels(req.method, req.path, res.statusCode).observe(duration);
+  });
+  next();
+});
+
+// Expose metrics
+app.get('/metrics', (req, res) => {
+  res.set('Content-Type', prometheus.register.contentType);
+  res.end(prometheus.register.metrics());
+});
+```
+
+## ServiceMonitor (Auto-Scrape)
+
+**Prometheus automatically discovers ServiceMonitor resources:**
+```yaml
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: myapp
+  namespace: myapp-ns
+spec:
+  selector:
+    matchLabels:
+      app: myapp
+  endpoints:
+  - port: metrics
+    interval: 30s
+    path: /metrics
+```
+
+**Deploy & verify:**
+```bash
+kubectl apply -f myapp-servicemonitor.yaml
+
+# Check Prometheus targets
+kubectl port-forward -n monitoring svc/prometheus-kube-prom-prometheus 9090:9090
+# http://localhost:9090/targets → should show myapp
+```
+
+## PrometheusRule (Alerting)
+
+**Define alert rules:**
+```yaml
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: myapp-alerts
+  namespace: myapp-ns
+spec:
+  groups:
+  - name: myapp
+    rules:
+    - alert: HighErrorRate
+      expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05  # > 5% error rate
+      for: 5m
+      annotations:
+        summary: "High error rate for {{ $labels.service }}"
+        
+    - alert: HighLatency
+      expr: histogram_quantile(0.99, http_request_duration_seconds_bucket) > 1  # p99 > 1s
+      for: 10m
+      annotations:
+        summary: "High latency for {{ $labels.path }}"
+```
+
+## Logging (Loki)
+
+**View pod logs in Grafana:**
+```
+Grafana → Explore → Loki
+Query: {namespace="story-crater-backend"} | json | level="error"
+
+Label filters:
+- namespace
+- pod_name
+- container
+- level (error, warn, info)
+```
+
+**Log retention:**
+- 10-day default (configurable)
+- Older logs deleted automatically
+- Chunks stored in MinIO (s3://loki-chunks)
+
+## Monitoring Checklist
+
+**For every service:**
+- [ ] Export `/metrics` (Prometheus format)
+- [ ] Add ServiceMonitor resource
+- [ ] Add PrometheusRule (errors, latency)
+- [ ] Create Grafana dashboard (6 rows: Availability, Resources, Domain, Logs, SLO, Related)
+- [ ] Set up Slack/PagerDuty integration (if on-call)
+
+## Key Queries
+
+**Request rate by status:**
+```promql
+sum(rate(http_requests_total[5m])) by (status)
+```
+
+**Error rate %:**
+```promql
+sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100
+```
+
+**P99 latency:**
+```promql
+histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))
+```
+
+**Memory usage %:**
+```promql
+(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
+```
+
+## Troubleshooting
+
+**Prometheus can't scrape ServiceMonitor:**
+```bash
+# Check ServiceMonitor exists
+k get servicemonitor -A | grep myapp
+
+# Verify Service has correct labels
+k get svc -A -o json | jq '.items[] | select(.metadata.name=="myapp") | .metadata.labels'
+
+# Check Prometheus targets
+kubectl port-forward -n monitoring svc/prometheus-kube-prom-prometheus 9090:9090
+# http://localhost:9090/targets → look for "Down" targets with error
+```
+
+**Metrics missing from Grafana:**
+```bash
+# Verify pod is exporting metrics
+k port-forward -n myapp-ns pod/myapp-0 8080:8080
+curl http://localhost:8080/metrics | grep my_metric_name
+
+# Check query syntax in Grafana
+# Hover over query → "Query Inspector" to see response
+```
+
+**Alert not firing:**
+```bash
+# Check PrometheusRule is loaded
+k get prometheusrule -n myapp-ns
+
+# Verify query in Prometheus
+kubectl port-forward -n monitoring svc/prometheus-kube-prom-prometheus 9090:9090
+# http://localhost:9090 → Graph tab → paste query
+```
+
+See `/TROUBLESHOOTING.md` for full incident guide.
@@ -0,0 +1,314 @@
+# Networking: Ingress, TLS & Service Discovery
+
+**Ingress Controller:** `nginx-ingress` (Nginx)  
+**Load Balancer:** Cilium LB-IPAM (eBPF-based)  
+**TLS CA:** homelab-ca (self-signed, 10-year)  
+**Namespace:** `ingress-nginx`
+
+## When to Use
+
+- **Public HTTPS endpoints** — External access via TLS
+- **Hostname-based routing** — Multiple services on same IP
+- **TLS termination** — Offload encryption/decryption
+- **Service discovery** — Internal DNS (CoreDNS)
+
+## Quick Start
+
+**1. Create Ingress rule:**
+```yaml
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: myapp
+  namespace: myapp-ns
+  annotations:
+    cert-manager.io/cluster-issuer: "letsencrypt-prod"  # or homelab-ca
+spec:
+  ingressClassName: nginx
+  tls:
+  - hosts:
+    - myapp.riotpiao.homelab.com
+    secretName: myapp-tls
+  rules:
+  - host: myapp.riotpiao.homelab.com
+    http:
+      paths:
+      - path: /
+        pathType: Prefix
+        backend:
+          service:
+            name: myapp-svc
+            port:
+              number: 8080
+```
+
+**2. Deploy:**
+```bash
+kubectl apply -f ingress.yaml
+
+# Wait for cert issuance
+kubectl get certificate -n myapp-ns
+# Should show "Ready" after ~30s
+```
+
+**3. Test from client:**
+```bash
+# Add to /etc/hosts (or use WireGuard)
+192.168.1.160 myapp.riotpiao.homelab.com
+
+# Access
+curl https://myapp.riotpiao.homelab.com
+```
+
+## Configuration
+
+| Key | Value |
+|-----|-------|
+| Ingress class | `nginx` |
+| Load balancer type | `LoadBalancer` (Cilium LB-IPAM) |
+| TLS issuer | `homelab-ca` (ClusterIssuer) |
+| TLS cert lifetime | 90 days (auto-renewed by cert-manager) |
+| DNS | CoreDNS (in-cluster), external via `/etc/hosts` or DuckDNS |
+
+## Common Patterns
+
+**Ingress with path-based routing:**
+```yaml
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: api
+  namespace: default
+spec:
+  ingressClassName: nginx
+  tls:
+  - hosts:
+    - api.riotpiao.homelab.com
+    secretName: api-tls
+  rules:
+  - host: api.riotpiao.homelab.com
+    http:
+      paths:
+      - path: /users
+        pathType: Prefix
+        backend:
+          service:
+            name: users-svc
+            port:
+              number: 3000
+      - path: /orders
+        pathType: Prefix
+        backend:
+          service:
+            name: orders-svc
+            port:
+              number: 3001
+```
+
+**Ingress with basic auth:**
+```bash
+# Generate htpasswd
+htpasswd -c auth admin
+# → prompted for password
+
+# Create Secret
+kubectl create secret generic basic-auth --from-file=auth -n default
+
+# Create Ingress
+```
+
+```yaml
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: protected
+  namespace: default
+  annotations:
+    nginx.ingress.kubernetes.io/auth-type: basic
+    nginx.ingress.kubernetes.io/auth-secret: basic-auth
+    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required'
+spec:
+  ingressClassName: nginx
+  rules:
+  - host: protected.riotpiao.homelab.com
+    http:
+      paths:
+      - path: /
+        pathType: Prefix
+        backend:
+          service:
+            name: app-svc
+            port:
+              number: 8080
+```
+
+**Internal DNS (CoreDNS rewrite):**
+```yaml
+# k8s/coredns/coredns-configmap.yaml
+# Rewrite:
+# - grafana.riotpiao.homelab.com → grafana.logging (cluster-internal)
+# - prometheus.riotpiao.homelab.com → prometheus-kube-prom-prometheus.monitoring
+#
+# Allows pods to use external URLs but resolve to internal Services
+```
+
+**Fixed LoadBalancer IP (Cilium LB-IPAM):**
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: ingress-nginx
+  namespace: ingress-nginx
+  annotations:
+    io.cilium/lb-ipam-ips: "192.168.1.160"  # fixed IP
+spec:
+  type: LoadBalancer
+  selector:
+    app: nginx-ingress
+  ports:
+  - port: 443
+    targetPort: 443
+    protocol: TCP
+```
+
+## TLS Certificate Management
+
+**Automatic renewal (cert-manager):**
+```yaml
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: myapp-cert
+  namespace: myapp-ns
+spec:
+  secretName: myapp-tls
+  duration: 2160h  # 90 days
+  renewBefore: 360h  # renew 15 days before expiry
+  commonName: myapp.riotpiao.homelab.com
+  dnsNames:
+  - myapp.riotpiao.homelab.com
+  issuerRef:
+    name: homelab-ca
+    kind: ClusterIssuer
+```
+
+**Check certificate status:**
+```bash
+# List certs
+k get certificate -A
+
+# View cert details
+k describe certificate -n myapp-ns myapp-cert
+
+# View TLS Secret
+k get secret -n myapp-ns myapp-tls -o json | jq '.data."tls.crt"' | base64 -d | openssl x509 -text
+
+# Check expiry date
+k get secret -n myapp-ns myapp-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate
+```
+
+## Service Discovery
+
+**Cluster-internal DNS:**
+```bash
+# From any pod, resolve via CoreDNS
+nslookup grafana.logging.svc.cluster.local  # full FQDN
+nslookup grafana.logging                     # short form (same namespace)
+nslookup grafana                             # if in logging namespace
+
+# Resolved to ClusterIP (internal only)
+```
+
+**External DNS (WireGuard VPN or port-forward):**
+```bash
+# Option 1: WireGuard tunnel
+# Client connects to 10.6.0.1 (WireGuard server on talos-cp-1)
+# All traffic tunneled to cluster
+
+# Option 2: Port-forward from jump box
+make pf-grafana  # localhost:3000 → grafana.logging:3000
+
+# Option 3: Add to /etc/hosts (on home network)
+192.168.1.160 grafana.riotpiao.homelab.com
+```
+
+## Monitoring
+
+**Grafana dashboard:** `svc-nginx-ingress`
+
+**Key metrics:**
+- `nginx_requests_total` — total requests
+- `nginx_request_duration_seconds` — latency histogram
+- `nginx_ingress_upstream_requests_total{status=~"5.."}` — backend errors
+- `nginx_ssl_expire_time_seconds` — cert expiry countdown
+
+**Alert on cert expiry:**
+```yaml
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: cert-expiry
+  namespace: ingress-nginx
+spec:
+  groups:
+  - name: cert-expiry
+    rules:
+    - alert: CertificateExpiringSoon
+      expr: nginx_ssl_expire_time_seconds < 86400 * 14  # < 14 days
+      annotations:
+        summary: "Certificate {{ $labels.host }} expires in {{ $value | humanizeDuration }}"
+```
+
+## Troubleshooting
+
+**Certificate stuck in "Pending":**
+```bash
+# Check cert-manager logs
+k logs -n cert-manager -f deploy/cert-manager
+
+# Verify ClusterIssuer exists
+k get clusterissuer
+
+# Check ACME order (if using LetsEncrypt)
+k describe certificate -n myapp-ns myapp-cert
+```
+
+**Ingress not exposing service (503 error):**
+```bash
+# Verify Service exists and has endpoints
+k get svc -n myapp-ns
+k get endpoints -n myapp-ns myapp-svc
+
+# Check if pods are ready
+k get pods -n myapp-ns
+
+# Test pod directly (port-forward)
+k port-forward -n myapp-ns pod/myapp-0 8080:8080
+curl http://localhost:8080
+```
+
+**DNS resolution fails from pod:**
+```bash
+# Test from pod
+k run -it --rm debug --image=busybox:1.28 --restart=Never -- \
+  nslookup grafana.logging.svc.cluster.local
+
+# If fails, CoreDNS may be unhealthy
+k get pods -n kube-system -l k8s-app=kube-dns
+k logs -n kube-system -l k8s-app=kube-dns
+```
+
+**TLS handshake error (cert not trusted):**
+```bash
+# Verify TLS cert Secret exists
+k get secret -n myapp-ns myapp-tls
+
+# Verify cert is correctly signed by homelab-ca
+k get secret -n myapp-ns myapp-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text | grep -A 5 "Issuer:"
+
+# If cert is self-signed (homelab-ca), add to client's trusted roots
+# Or bypass cert verification (dev only):
+curl -k https://myapp.riotpiao.homelab.com
+```
+
+See `/TROUBLESHOOTING.md` for full incident guide.
@@ -0,0 +1,193 @@
+# SQS-like Message Queue Service (kmsvc)
+
+**Endpoint:** `https://kmsvc.riotpiao.homelab.com` (REST + gRPC-Gateway)  
+**Internal:** `kmsvc-management-service.sqs.svc.cluster.local:8080`  
+**Namespace:** `sqs`
+
+## When to Use
+
+- **Decouple services** — Producer doesn't wait for consumer
+- **Async jobs** — Fire-and-forget processing (batch, email, webhooks)
+- **FIFO ordering** — Guarantee message order within `MessageGroupId`
+- **Durable delivery** — At-least-once (messages in Kafka, replicated 3×)
+
+## Quick Start
+
+**1. Create a queue:**
+```bash
+kubectl apply -f - <<EOF
+apiVersion: kmsvc.io/v1
+kind: Queue
+metadata:
+  name: orders
+spec:
+  fifoQueue: false                           # standard queue
+  visibilityTimeoutSeconds: 30               # re-deliver if not acked
+  messageRetentionPeriodSeconds: 345600      # 4 days
+  partitionsPerShard: 6
+  maxReceiveCount: 5                         # move to DLQ after 5 fails
+EOF
+```
+
+**2. Send message:**
+```bash
+curl -X POST https://kmsvc.riotpiao.homelab.com/v1/queues/orders/messages \
+  -H "Authorization: Bearer $JWT_TOKEN" \
+  -d '{
+    "body": "{\"order_id\":123,\"total\":99.99}",
+    "attributes": {"source":"web","priority":"high"}
+  }'
+```
+
+**3. Receive message:**
+```bash
+curl "https://kmsvc.riotpiao.homelab.com/v1/queues/orders/messages?max_number_of_messages=10&wait_time_seconds=20" \
+  -H "Authorization: Bearer $JWT_TOKEN"
+
+# Response:
+# {
+#   "messages": [
+#     {
+#       "message_id": "abc-123",
+#       "receipt_handle": "...",
+#       "body": "{...}",
+#       "attributes": {...},
+#       "receive_count": 1
+#     }
+#   ]
+# }
+```
+
+**4. Acknowledge (delete) message:**
+```bash
+curl -X DELETE "https://kmsvc.riotpiao.homelab.com/v1/queues/orders/messages/$receipt_handle" \
+  -H "Authorization: Bearer $JWT_TOKEN"
+```
+
+## Configuration
+
+| Key | Value |
+|-----|-------|
+| Kafka bootstrap | `kmsvc-kafka-bootstrap.sqs.svc.cluster.local:9092` |
+| Redis | `kmsvc-redis-master.sqs.svc.cluster.local:6379` |
+| Topic naming | `kmsvc.{queueName}.shard-{id}` |
+| Replication | 3 replicas, min.insync.replicas=2 |
+| Retention | 4 days (configurable per queue) |
+
+## Common Patterns
+
+**Batch processing:**
+```bash
+for i in {1..100}; do
+  curl -X POST https://kmsvc.riotpiao.homelab.com/v1/queues/jobs/messages \
+    -H "Authorization: Bearer $JWT_TOKEN" \
+    -d "{\"body\":\"task-$i\"}" &
+done
+wait
+```
+
+**FIFO queue (order guaranteed per group):**
+```yaml
+apiVersion: kmsvc.io/v1
+kind: Queue
+metadata:
+  name: checkout-fifo
+spec:
+  fifoQueue: true
+  visibilityTimeoutSeconds: 60
+  partitionsPerShard: 1
+```
+
+**Dead-letter queue (failed messages):**
+```yaml
+apiVersion: kmsvc.io/v1
+kind: Queue
+metadata:
+  name: orders-dlq
+spec:
+  fifoQueue: false
+
+---
+apiVersion: kmsvc.io/v1
+kind: Queue
+metadata:
+  name: orders
+spec:
+  fifoQueue: false
+  maxReceiveCount: 3
+  deadLetterTargetQueue: orders-dlq  # auto-route failures here
+```
+
+## Monitoring
+
+**Grafana dashboard:** `svc-kmsvc` (automatically loaded)
+
+**Key metrics:**
+- `kmsvc_messages_sent_total` — total sent
+- `kmsvc_messages_received_total` — total received
+- `kmsvc_queue_depth` — pending messages per queue
+- `kmsvc_message_visibility_timeout_seconds` — visibility window
+
+**Redis in-flight tracking:**
+```bash
+# Connect to Redis
+k port-forward -n sqs svc/redis 6379:6379 &
+redis-cli
+
+# Check pending messages
+KEYS "kmsvc:pending:orders:*"
+KEYS "kmsvc:inflight:*" | wc -l
+```
+
+## Authentication
+
+**Requires JWT from Authentik:**
+```bash
+# Get token (device code flow)
+talos secrets login
+
+# Use token
+export JWT_TOKEN=$(talos get cluster/kmsvc/jwt-token --key jwt-token)
+curl -H "Authorization: Bearer $JWT_TOKEN" https://kmsvc.riotpiao.homelab.com/v1/queues
+```
+
+## Integration Example
+
+**Story Crater backend consumer:**
+```go
+// Receive messages
+messages, err := kmsvc.ReceiveMessage(ctx, &kmsvc.ReceiveMessageRequest{
+  QueueName:          "story-crater",
+  MaxNumberOfMessages: 10,
+  WaitTimeSeconds:    20,
+})
+
+// Process
+for _, msg := range messages.Messages {
+  processMessage(msg.Body)
+  
+  // Acknowledge on success
+  kmsvc.DeleteMessage(ctx, &kmsvc.DeleteMessageRequest{
+    QueueName:     "story-crater",
+    ReceiptHandle: msg.ReceiptHandle,
+  })
+}
+```
+
+## Troubleshooting
+
+**Queue stuck / high lag:**
+```bash
+# Check Kafka broker status
+k exec -n sqs pod/kmsvc-kafka-0 -- kafka-broker-api-versions.sh --bootstrap-server localhost:9092
+
+# Inspect queue topics
+k exec -n sqs pod/kmsvc-kafka-0 -- kafka-topics.sh --bootstrap-server localhost:9092 --list | grep orders
+```
+
+**Messages not being consumed:**
+- Check `maxReceiveCount` (may be routing to DLQ)
+- Verify consumer has `ReceiveMessage` permission (JWT scope)
+- Check Redis: `KEYS "kmsvc:fifo_lock:orders:*"` (may be blocked by visibility timeout)
+
+See `/TROUBLESHOOTING.md` for full incident guide.
@@ -0,0 +1,227 @@
+# Temporal Workflow Orchestration
+
+**Server:** `temporal.temporal.svc.cluster.local:7233` (cluster-internal)  
+**Web UI:** `kubectl port-forward -n temporal svc/temporal-web 8088:8088`  
+**Namespace:** `temporal`
+
+## When to Use
+
+- **Long-running operations** — Tasks that take minutes/hours (send email, process batch, retry with backoff)
+- **State machines** — Multi-step workflows with decision logic
+- **Retries & timeouts** — Built-in exponential backoff, automatic retry
+- **Audit trail** — Full history of workflow executions (why it happened, when, by whom)
+
+## Quick Start
+
+**1. Access Temporal Web UI:**
+```bash
+k port-forward -n temporal svc/temporal-web 8088:8088
+# http://localhost:8088
+```
+
+**2. Define workflow (Go example):**
+```go
+package workflows
+
+import (
+  "time"
+  "go.temporal.io/sdk/workflow"
+  "go.temporal.io/sdk/activity"
+)
+
+type Inputs struct {
+  OrderID string
+  Amount  float64
+}
+
+// Workflow definition
+func OrderProcessing(ctx workflow.Context, input Inputs) (string, error) {
+  // Step 1: Charge payment
+  chargeResult := ""
+  err := workflow.ExecuteActivity(
+    ctx,
+    ChargePayment,
+    input.OrderID,
+    input.Amount,
+  ).Get(ctx, &chargeResult)
+  if err != nil {
+    return "", err
+  }
+
+  // Step 2: Send confirmation email (retry 3x on failure)
+  emailResult := ""
+  opts := workflow.ActivityOptions{
+    StartToCloseTimeout: time.Minute,
+    RetryPolicy: &temporal.RetryPolicy{
+      InitialInterval:    time.Second,
+      BackoffCoefficient: 2,
+      MaximumAttempts:    3,
+    },
+  }
+  ctx = workflow.WithActivityOptions(ctx, opts)
+  
+  err = workflow.ExecuteActivity(ctx, SendConfirmationEmail, input.OrderID).Get(ctx, &emailResult)
+  if err != nil {
+    return "", err
+  }
+
+  return "order_processed", nil
+}
+
+// Activity: payment processing
+func ChargePayment(ctx context.Context, orderID string, amount float64) (string, error) {
+  // Call payment gateway
+  return "payment_successful", nil
+}
+
+// Activity: email notification
+func SendConfirmationEmail(ctx context.Context, orderID string) (string, error) {
+  // Send email
+  return "email_sent", nil
+}
+```
+
+**3. Register & start workflow:**
+```go
+import "go.temporal.io/sdk/client"
+
+client, _ := client.Dial(client.Options{
+  HostPort: "temporal.temporal.svc.cluster.local:7233",
+})
+
+// Start workflow execution
+run, _ := client.ExecuteWorkflow(ctx, client.StartWorkflowOptions{
+  ID:        "order-123",
+  TaskQueue: "orders",
+}, OrderProcessing, Inputs{OrderID: "123", Amount: 99.99})
+
+// Wait for result
+var result string
+run.Get(ctx, &result)
+```
+
+## Configuration
+
+| Key | Value |
+|-----|-------|
+| Server | `temporal.temporal.svc.cluster.local:7233` |
+| Web UI | `localhost:8088` (via port-forward) |
+| Database | PostgreSQL (managed by helmfile) |
+| Task queue | `default`, `orders`, `emails` (custom per app) |
+| Retention | 30 days (configurable) |
+
+## Common Patterns
+
+**Retry with exponential backoff:**
+```go
+opts := workflow.ActivityOptions{
+  StartToCloseTimeout: 5 * time.Minute,
+  RetryPolicy: &temporal.RetryPolicy{
+    InitialInterval:    time.Second,
+    BackoffCoefficient: 2.0,           // double wait time each retry
+    MaximumInterval:    time.Minute,   // cap at 1 min between retries
+    MaximumAttempts:    5,             // give up after 5 tries
+  },
+}
+ctx = workflow.WithActivityOptions(ctx, opts)
+```
+
+**Wait for signal (user approval):**
+```go
+// Workflow waits for approval signal
+approval := ""
+workflow.GetSignalChannel(ctx, "approval").Receive(ctx, &approval)
+
+if approval == "approved" {
+  // Continue workflow
+} else {
+  return "", errors.New("request denied")
+}
+```
+
+**Parallel activities:**
+```go
+// Execute email & SMS in parallel
+emailFuture := workflow.ExecuteActivity(ctx, SendEmail, userID)
+smsFuture := workflow.ExecuteActivity(ctx, SendSMS, userID)
+
+// Wait for both to complete
+emailFuture.Get(ctx, nil)
+smsFuture.Get(ctx, nil)
+```
+
+**Scheduled workflow (cron):**
+```go
+run, _ := client.ExecuteWorkflow(ctx, client.StartWorkflowOptions{
+  ID:              "daily-report",
+  CronSchedule:    "0 9 * * MON-FRI",  // 9 AM weekdays
+  WorkflowTaskTimeout: time.Hour,
+}, GenerateDailyReport, nil)
+```
+
+## Monitoring
+
+**Web UI:**
+- List workflows: http://localhost:8088/namespaces/default/workflows
+- View execution history: Click workflow ID
+- See activity logs, errors, retry attempts
+
+**Grafana dashboard:** `svc-temporal` (auto-configured)
+
+**Key metrics:**
+- `temporal_workflow_execution_duration_seconds` — workflow time
+- `temporal_activity_execution_duration_seconds` — activity time
+- `temporal_activity_execution_failed_total` — failed activities
+
+## Integration with Story Crater
+
+**Example: Process message via Temporal:**
+```go
+// In message handler
+client, _ := temporal.Dial(/* ... */)
+
+run, _ := client.ExecuteWorkflow(ctx, client.StartWorkflowOptions{
+  ID:        fmt.Sprintf("msg-%s", messageID),
+  TaskQueue: "story-crater",
+}, ProcessMessageWorkflow, Message{
+  ID:     messageID,
+  Body:   body,
+  Source: "kafka-queue",
+})
+
+// Non-blocking: workflow runs independently
+// Check status later
+```
+
+## Troubleshooting
+
+**Workflow stuck:**
+```bash
+# Check Temporal server health
+k get pods -n temporal
+
+# View workflow execution history (via Web UI or CLI)
+tctl workflow show --workflow-id order-123
+
+# Terminate stuck workflow
+tctl workflow terminate --workflow-id order-123
+```
+
+**Activity retrying endlessly:**
+```go
+// Add max attempts or timeout
+RetryPolicy: &temporal.RetryPolicy{
+  MaximumAttempts: 5,  // must have this!
+}
+```
+
+**PostgreSQL connection fails:**
+```bash
+# Check temporal pod logs
+k logs -n temporal pod/temporal-0
+
+# Verify database is running
+k get pods -n ddb
+```
+
+See `/TROUBLESHOOTING.md` for full incident guide.
@@ -0,0 +1,211 @@
+# Vault: Secret Management & JWT Auth
+
+**Vault:** `https://vault.riotpiao.homelab.com`  
+**Internal:** `vault.iam.svc.cluster.local:8200`  
+**Namespace:** `iam`
+
+## When to Use
+
+- **Store secrets** — Database passwords, API keys, TLS certs
+- **Rotate credentials** — Auto-rotate, track rotation history
+- **JWT validation** — Verify tokens from Authentik, no external call needed
+- **Audit trail** — Who accessed what secret, when
+
+## Quick Start
+
+**1. Login to Vault:**
+```bash
+# Browser: https://vault.riotpiao.homelab.com
+# Auth method: OIDC → "Sign in with Authentik" (federated)
+# Or: Device code → talos secrets login (CLI)
+
+# Via CLI (device code flow)
+talos secrets login
+# → Opens browser, approve device code
+# → Token cached in ~/.talos/vault
+```
+
+**2. Store a secret:**
+```bash
+# Field name = variable name (SCREAMING_SNAKE_CASE)
+talos put cluster/ANTHROPIC_API_KEY ANTHROPIC_API_KEY="sk-..."
+talos put cluster/STORY_CRATER_DB_PASS STORY_CRATER_DB_PASS="dbpass123"
+```
+
+**3. Retrieve a secret:**
+```bash
+# Always use --key flag
+talos get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
+# → sk-...
+
+# Full secret as JSON
+talos get cluster/ANTHROPIC_API_KEY --json
+```
+
+**4. Load into shell (helmfile, scripts):**
+```bash
+vsource .env
+# → Expands empty vars from Vault (ANTHROPIC_API_KEY=)
+# → Passes hardcoded vars as-is (DEBUG=true)
+
+helmfile diff
+helmfile apply
+```
+
+## Vault Paths (KV v2)
+
+**Naming convention:** `cluster/<VARIABLE_NAME>`
+
+```
+cluster/
+├── ANTHROPIC_API_KEY            # third-party API
+├── STORY_CRATER_DB_PASS         # database password
+├── MINIO_ROOT_PASSWORD          # MinIO credentials
+├── AUTHENTIK_BOOTSTRAP_PASSWORD # initial admin pass
+├── iam/
+│   ├── federation               # Authentik OIDC app config
+│   ├── roles/admin              # RBAC role definitions
+│   ├── services/grafana         # OAuth2 client info
+│   ├── agents/ci-bot            # machine credentials
+│   └── bindings/alice           # user→role mappings
+└── kubernetes/
+    ├── ingress-tls              # TLS cert private keys
+    └── pull-secrets             # Docker registry creds
+```
+
+## Common Patterns
+
+**Store generated secret immediately (keeps it out of shell history):**
+```bash
+# Generate & store in one command
+talos put cluster/GRAFANA_OIDC_CLIENT_SECRET \
+  GRAFANA_OIDC_CLIENT_SECRET="$(openssl rand -hex 32)"
+```
+
+**Use in Kubernetes Secret:**
+```bash
+# Create Secret using Vault secret
+kubectl create secret generic grafana-oidc \
+  --from-literal=client-secret="$(talos get cluster/GRAFANA_OIDC_CLIENT_SECRET --key GRAFANA_OIDC_CLIENT_SECRET)" \
+  -n logging
+```
+
+**Rotate credentials safely:**
+```bash
+# 1. Generate new secret
+NEW_PASS=$(openssl rand -base64 24)
+
+# 2. Store in Vault
+talos put cluster/OLD_DB_PASS OLD_DB_PASS="$NEW_PASS"
+
+# 3. Update database user
+psql -h ddb-cluster-rw.ddb.svc.cluster.local -U postgres \
+  -c "ALTER USER story_crater PASSWORD '$NEW_PASS';"
+
+# 4. Update Kubernetes Secret
+kubectl patch secret db-credentials -n story-crater-backend \
+  --type merge -p "{\"stringData\":{\"password\":\"$NEW_PASS\"}}"
+
+# 5. Restart pods to pick up new Secret
+k rollout restart -n story-crater-backend deployment/app
+```
+
+**JWT token from CLI:**
+```bash
+# After device code login
+talos secrets login
+
+# Token is cached and auto-renewed
+# Use for API calls
+VAULT_TOKEN=$(cat ~/.talos/vault)
+curl -H "X-Vault-Token: $VAULT_TOKEN" \
+  https://vault.iam.svc.cluster.local:8200/v1/secret/data/cluster/ANTHROPIC_API_KEY
+```
+
+## Monitoring
+
+**Vault status:**
+```bash
+# Check if sealed
+talos status vault
+
+# If sealed (disaster recovery):
+# See /TROUBLESHOOTING.md § Vault Sealed
+```
+
+**Audit log (who accessed what):**
+```bash
+# Enable audit logging (already enabled by helmfile)
+# Logs stored in Loki under vault namespace
+
+# View recent access
+talos audit log --limit 50
+
+# Export for compliance
+talos audit export --format json > vault-audit.json
+```
+
+## Security Rules
+
+✅ **DO:**
+- Store ALL secrets in Vault (never .env in git)
+- Use field name = variable name
+- Always use `--key` flag when retrieving
+- Rotate credentials on schedule (quarterly)
+- Review audit log for anomalies
+
+❌ **DON'T:**
+- Commit `.env` with real secrets (only `.env.example`)
+- Use positional arguments (must use `--key`)
+- Share Vault root token
+- Store secrets in pod env (use Secret volumes)
+
+## Troubleshooting
+
+**Cannot login (Authentik OIDC fails):**
+```bash
+# Check Authentik is running
+k get pods -n iam -l app=authentik
+
+# Verify OIDC app in Authentik console
+# Applications → Vault OIDC → Check client ID/secret
+
+# Restart Vault to re-sync OIDC config
+k rollout restart -n iam statefulset/vault
+```
+
+**Vault is sealed:**
+```bash
+# Check status
+talos status vault
+
+# If sealed, use unseal keys (stored in MinIO backup)
+# See /TROUBLESHOOTING.md § Vault Sealed for recovery steps
+```
+
+**Secret not found:**
+```bash
+# Verify path exists
+talos list cluster
+
+# Check secret name (case-sensitive)
+talos get cluster/anthropic_api_key --key anthropic_api_key  # won't work
+talos get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY  # correct
+```
+
+**vsource not expanding secrets:**
+```bash
+# Verify .env has empty value
+grep ANTHROPIC_API_KEY .env
+# → Should be: ANTHROPIC_API_KEY=  (empty, not a value)
+
+# Verify Vault is accessible
+talos get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
+# → Should return secret
+
+# Run vsource explicitly
+vsource .env
+echo $ANTHROPIC_API_KEY  # should be populated
+```
+
+See `/TROUBLESHOOTING.md` for full incident guide.