fix: move gateway config from plaintext ConfigMap to SOPS-encrypted Secret

.sops.yaml

@@ -0,0 +1,3 @@
+creation_rules:
+  - path_regex: \.enc\.ya?ml$
+    age: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla

SLA.md

@@ -0,0 +1,131 @@
+# SLA: API Gateway & Platform Services
+
+## API Gateway (api.riotpiao.com)
+
+### Availability
+
+| Target | Measurement | Alert |
+|--------|------------|-------|
+| 99.9% uptime | `probe_success{instance=~".*api.riotpiao.com.*"}` | `APIGatewayProbeDown` fires after 2m down |
+| Monthly budget: 43.8 min downtime | 7-day SLO: `avg_over_time(probe_success[7d]) * 100` | |
+| Zero ready pods = critical | `sum(kube_pod_status_ready{namespace="api"}) == 0` | `APIGatewayDown` fires after 1m |
+
+### Latency
+
+Baselines measured from 200-request canary run against live cluster.
+SLA set at ~2x measured p99 for headroom.
+
+| Endpoint | Measured p50 | Measured p99 | SLA (p95) | SLA (p99) | Alert |
+|----------|-------------|-------------|-----------|-----------|-------|
+| LLM Chat (qwen) | 514ms | 609ms | <1s | <2s | `APIGatewayLatencyHigh` |
+| LLM Chat (reasoning) | 300ms | 328ms | <1s | <2s | `APIGatewayLatencyHigh` |
+| LLM Chat (ornith:35b) | 1.2s | 1.2s | <3s | <5s | `APIGatewayLatencyCritical` |
+| LLM Chat (streaming) | 569ms | 628ms | <1s | <2s | `APIGatewayLatencyHigh` |
+| Embeddings | 189ms | 287ms | <500ms | <1s | `APIGatewayLatencyHigh` |
+| Rerank | 106ms | 218ms | <500ms | <1s | `APIGatewayLatencyHigh` |
+| Models list | 68ms | 277ms | <300ms | <500ms | `APIGatewayLatencyHigh` |
+| Auth rejection | 69ms | 87ms | <200ms | <500ms | (no alert, expected fast) |
+
+### Error Rate
+
+| Target | Measurement | Alert |
+|--------|------------|-------|
+| 5xx < 1% | `nginx_ingress_controller_requests{status=~"5.."}` / total | `APIGateway5xxErrorRate` fires after 5m >1% |
+| Total errors < 10% | 4xx + 5xx / total | `APIGatewayHighErrorRate` fires after 10m >10% |
+
+---
+
+## LLM Serving (llm-serving namespace)
+
+| Target | Measurement | Alert |
+|--------|------------|-------|
+| All predictors running | replicas ready == desired per deployment | `LLMPredictorDown` fires after 5m |
+| Zero LLM pods = critical | `sum(ready{namespace="llm-serving"}) == 0` | `LLMServingDown` fires after 2m |
+| No restart storms | restart count in 15m | `LLMPredictorRestarted` on any restart |
+
+---
+
+## Cluster Infrastructure
+
+### Node Health
+
+| Target | Measurement | Alert |
+|--------|------------|-------|
+| All nodes Ready | `kube_node_status_condition` | `NodeNotReady` fires after 2m |
+| CPU < 90% sustained | `node_cpu_seconds_total` | `NodeHighCPU` fires after 15m |
+| Memory < 90% sustained | `node_memory_MemAvailable_bytes` | `NodeHighMemory` fires after 15m |
+| Disk < 85% | `node_filesystem_avail_bytes` | `NodeDiskFull` fires after 5m (critical) |
+
+### Pod Health
+
+| Target | Measurement | Alert |
+|--------|------------|-------|
+| No pods pending > 10m | `kube_pod_status_phase{phase="Pending"}` | `PodStuckPending` |
+| No CrashLoopBackOff > 5m | `kube_pod_container_status_waiting_reason` | `PodCrashLooping` (critical) |
+| OOMKilled < 3/hour | `kube_pod_container_status_last_terminated_reason` | `OOMKilledSpike` |
+| No restart storms | >5 restarts in 15m | `ContainerRestartStorm` |
+
+### Jobs
+
+| Target | Measurement | Alert |
+|--------|------------|-------|
+| No failed jobs | `kube_job_status_failed > 0` | `JobFailed` fires after 5m |
+| No stuck jobs > 2h | `kube_job_status_active` + age | `JobStuckRunning` |
+| CronJobs on schedule | last_schedule vs next_schedule | `CronJobMissedSchedule` fires after 10m |
+
+### Storage
+
+| Target | Measurement | Alert |
+|--------|------------|-------|
+| Longhorn drives healthy | `longhorn_disk_health` | `LonghornDriveOffline` fires after 5m (critical) |
+
+### DNS
+
+| Target | Measurement | Alert |
+|--------|------------|-------|
+| CoreDNS SERVFAIL < 0.5/s | `coredns_dns_responses_total{rcode="SERVFAIL"}` | `CoreDNSErrorSpike` fires after 5m |
+
+### Probes
+
+| Target | Measurement | Alert |
+|--------|------------|-------|
+| All service probes passing | `probe_success` | `ServiceProbeDown` fires after 3m (critical) |
+| Probe latency < 2s | `probe_duration_seconds` | `ServiceProbeSlow` fires after 5m |
+| Certs valid > 14 days | `certmanager_certificate_expiration_timestamp_seconds` | `CertificateExpiringSoon` |
+
+---
+
+## Alert Severity Levels
+
+| Severity | Meaning | Response Time |
+|----------|---------|--------------|
+| **critical** | Service down or data loss risk. Immediate impact on users. | Investigate within 15 min |
+| **warning** | Degraded performance or resource pressure. No immediate outage. | Investigate within 4 hours |
+
+### Critical Alerts (require immediate action)
+
+- `APIGatewayDown` — zero gateway pods
+- `LLMServingDown` — zero LLM pods
+- `NodeNotReady` — node lost
+- `PodCrashLooping` — service crashing repeatedly
+- `NodeDiskFull` — disk > 85%
+- `LonghornDriveOffline` — storage unhealthy
+- `ServiceProbeDown` — external service unreachable
+- `APIGateway5xxErrorRate` — 5xx > 1%
+- `APIGatewayLatencyCritical` — p99 > 5s
+
+---
+
+## Current Alert Status
+
+Alerts firing after deployment:
+
+| Alert | State | Root Cause |
+|-------|-------|-----------|
+| `APIGatewayProbeDown` | pending | Blackbox probe for api-gateway not yet active (pod restart needed) |
+| `PodStuckPending` | pending | `sms/macos-bluebubbles` pending 22d (scheduling constraint) |
+| `PodCrashLooping` | pending | `iam/authentik-provision` job in Error state |
+| `DeploymentReplicasUnavailable` | pending | Same root causes above |
+| `ServiceProbeDown` | pending | api-gateway probe target not in blackbox yet |
+
+None are false positives. All reflect real cluster state.

k8s/deployment.yaml

@@ -109,8 +109,8 @@ spec:
             - ALL
       volumes:
       - name: config
-        configMap:
+        secret:
-          name: api-gateway-config
+          secretName: api-gateway-config
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:

k8s/gateway-config-secret.enc.yaml

@@ -0,0 +1,129 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: api-gateway-config
+  namespace: api
+  labels:
+    app: api-gateway
+type: Opaque
+stringData:
+  config.yaml: |
+    auth:
+      enabled: true
+      issuer: "https://authentik.riotpiao.com/application/o/api-gw/"
+      audience: "api-gw"
+      jwksUrl: "http://authentik-server.iam.svc.cluster.local/application/o/api-gw/jwks/"
+      requiredCapability: "llm:inference"
+    routes: []
+    models:
+    - name: "reasoning"
+      address: "reasoning-predictor.llm-serving:80"
+      path: "/v1/chat/completions"
+    - name: "ornith:35b"
+      address: "ornith-predictor.llm-serving:80"
+      path: "/v1/chat/completions"
+    - name: "qwen2.5:3b-instruct"
+      address: "ornith-predictor.llm-serving:80"
+      path: "/v1/chat/completions"
+    - name: "nomic-ai/nomic-embed-text-v2-moe"
+      address: "embeddings-predictor.llm-serving:80"
+      path: "/v1/embeddings"
+    - name: "BAAI/bge-reranker-base"
+      address: "reranker-predictor.llm-serving:80"
+      path: "/v1/rerank"
+    adapters:
+    - serviceName: sqs
+      upstream:
+        url: http://management-service.sqs.svc.cluster.local:9090
+        timeoutSeconds: 30
+      auth:
+        required: true
+      resources:
+      - name: send-message
+        methods:
+        - verb: POST
+          upstreamPath: /sqs/send
+      - name: receive-message
+        methods:
+        - verb: POST
+          upstreamPath: /sqs/receive
+      - name: list-queues
+        methods:
+        - verb: GET
+          upstreamPath: /sqs/queues
+    - serviceName: workflow
+      upstream:
+        url: grpc://temporal-frontend.temporal.svc.cluster.local:7233
+        timeoutSeconds: 60
+      auth:
+        required: false
+      resources:
+      - name: execute
+        methods:
+        - verb: POST
+          upstreamPath: /temporal.api.workflowservice.v1.WorkflowService/ExecuteWorkflow
+      - name: describe
+        methods:
+        - verb: GET
+          upstreamPath: /temporal.api.workflowservice.v1.WorkflowService/DescribeWorkflowExecution
+      - name: list
+        methods:
+        - verb: GET
+          upstreamPath: /temporal.api.workflowservice.v1.WorkflowService/ListWorkflowExecutions
+    - serviceName: memory
+      upstream:
+        url: http://poimen-memory.poimen.svc.cluster.local:8080
+        timeoutSeconds: 30
+      auth:
+        required: false
+      resources:
+      - name: query
+        methods:
+        - verb: POST
+          upstreamPath: /memory/query
+      - name: ingest
+        methods:
+        - verb: POST
+          upstreamPath: /memory/ingest
+      - name: skills
+        methods:
+        - verb: GET
+          upstreamPath: /memory/skills
+    - serviceName: s3
+      upstream:
+        url: http://minio.storage.svc.cluster.local:9000
+        timeoutSeconds: 30
+      auth:
+        required: false
+      resources:
+      - name: list-objects
+        methods:
+        - verb: GET
+          upstreamPath: /
+      - name: get-object
+        methods:
+        - verb: GET
+          upstreamPath: /
+      - name: put-object
+        methods:
+        - verb: PUT
+          upstreamPath: /
+    - serviceName: iam
+      upstream:
+        url: http://authentik-server.iam.svc.cluster.local:80
+        timeoutSeconds: 30
+      auth:
+        required: false
+      resources:
+      - name: list-roles
+        methods:
+        - verb: GET
+          upstreamPath: /api/v3/roles
+      - name: list-users
+        methods:
+        - verb: GET
+          upstreamPath: /api/v3/users
+      - name: create-role
+        methods:
+        - verb: POST
+          upstreamPath: /api/v3/roles

k8s/kustomization.yaml

@@ -8,7 +8,7 @@ resources:
 - service.yaml
 - deployment.yaml
 - network-policy.yaml
-- configmap.yaml
+- gateway-config-secret.enc.yaml
 
 # The deployed image tag lives here and nowhere else. CI publishes
 # forgejo.riotpiao.com/rock/api-gateway:<commit-sha> and tags it as :latest on main.