(chore) init commit and add tasks

2026-08-18 18:33:49 -07:00
commit 6c6218ef36
61 changed files with 9851 additions and 0 deletions
@@ -0,0 +1,327 @@
+# Implementation Plan: `atlas` — Interactive Cluster Visualization
+
+**Status**: For review — no code written yet
+**Companion ADR**: [ADR-0001](adr/ADR-0001-atlas-cluster-visualization.md)
+**Supersedes**: `PLAN.md`, `IMPLEMENTATION.md` (Homarr + Terraform — abandoned)
+
+---
+
+## 1. Scope
+
+### In scope
+
+| ID | Surface | Description |
+|---|---|---|
+| B | Cluster topology | Live node → namespace → workload graph, health-colored, drill-down |
+| E | Delivery tree | Argo CD app-of-apps as a sync-wave-ordered DAG, live sync animation |
+| C | Terminal | Read-only, enum-dispatched cluster queries in the browser |
+| D | Chat | Streaming ChatGPT-style session against the `reasoning` model |
+
+### Out of scope (v1)
+
+- Any write operation against the cluster
+- LLM tool-calling / agentic loops against live infrastructure
+- Forgejo CI half of the pipeline view (`forgejo-gitea` is currently stuck `Init:0/3`)
+- Log streaming to the browser (Loki content is unredactable in practice)
+- Authenticated / operator-only views — Grafana already serves that need
+
+### Non-goals
+
+- Replacing Grafana
+- Multi-cluster support
+- Historical / time-travel views
+
+---
+
+## 2. Phase 0 — Unblock (blocking; nothing ships until done)
+
+Verified problems, in the order they must be fixed:
+
+| # | Problem | Evidence |
+|---|---|---|
+| 0.1 | `portfolio` namespace is empty; `portfolio` and `auth-infra` Applications do not exist in the cluster | `kubectl get pods -n portfolio` → no resources; neither name appears in `kubectl get app -n argocd` |
+| 0.2 | `infra/argocd-apps.yaml` `repoURL` = `forgejo.riotpiao.homelab.com` — NXDOMAIN | `dig` |
+| 0.3 | Deployment image `forgejo.riotpiao.homelab.com/rock/portfolio:latest` — dead host, and `:latest` + `imagePullPolicy: IfNotPresent` means a pushed image will never roll out | `infra/portfolio/base/deployment.yaml` |
+| 0.4 | `forgejo-gitea` stuck `Init:0/3` for 3h — no image builds possible | `kubectl get pods -n cicd` |
+| 0.5 | `riotpiao.com` returns HTTP 403 from the Cloudflare edge; no origin headers present | `curl -I https://riotpiao.com` |
+| 0.6 | No test framework installed — TDD is impossible as the repo stands | `package.json` has no test script or runner |
+| 0.7 | `sms` Application Degraded (`macos-bluebubbles` Pending 3h); `longhorn-config` OutOfSync | `kubectl get app -n argocd` |
+
+**Decisions required from you before 0.1–0.3 can be actioned** — see ADR open questions 1 and 2.
+
+**Actions**
+
+1. Resolve which GitOps repo owns the portfolio; delete or correct the losing manifest
+2. Point `repoURL` and the image reference at real hostnames
+3. Replace `:latest` with a commit-SHA tag; set `imagePullPolicy: IfNotPresent` (correct once tags are immutable)
+4. Diagnose `forgejo-gitea` init containers — read the actual init container logs before changing anything
+5. Diagnose the apex 403 — check the tunnel's Public Hostnames list and Cloudflare WAF events; the event log names the blocking rule
+6. Add Vitest + Testing Library + `msw`; add `test` and `test:watch` scripts
+7. Triage 0.7 separately — unrelated to this work, but the delivery tree will render both as red on day one
+
+**Verify**
+
+```bash
+kubectl get pods -n portfolio                     # 2/2 Running
+curl -sS -o /dev/null -w '%{http_code}\n' https://riotpiao.com   # 200
+pnpm test                                          # runner executes, 0 tests, exit 0
+```
+
+---
+
+## 3. Architecture
+
+```
+kube API (client-go informers)  ─┐
+Prometheus /api/v1/query         ├──> atlas (Go, ns: portfolio, RO ServiceAccount)
+Argo CD Application CRs          ┘         │
+                                           ├─ snapshot (in-memory, redacted at write)
+                                           ├─ Redis pub/sub (kmsvc-redis-master.sqs:6379)
+                                           └─ HTTP
+                                               GET  /api/topology
+                                               GET  /api/delivery
+                                               GET  /api/stream      (SSE)
+                                               POST /api/exec
+                                               POST /api/chat        (SSE)
+                                                      └──> reasoning-predictor.llm-serving:80
+```
+
+Event-driven, per project architectural preference: informers push to a reducer, the reducer publishes deltas to Redis, SSE handlers subscribe. No request-triggered upstream calls anywhere in the read path.
+
+Snapshot is redacted **at write time**, not at serialization time. A field that never enters the snapshot cannot leak from any surface.
+
+### Language
+
+Go, for `client-go` informers and because it matches the rest of the platform. Follow the repo's Go skill set (`go-naming`, `go-concurrency`, `go-error-handling`, `go-context`) — notably: every upstream call carries a context, no naked returns, no `_ =` on errors.
+
+---
+
+## 4. API contract
+
+Envelope for all non-stream responses:
+
+```json
+{ "data": { }, "meta": { "snapshotAge": 3.2, "generation": 88412 } }
+```
+
+Errors follow RFC 9457 (`application/problem+json`):
+
+```json
+{ "type": "https://riotpiao.com/errors/rate-limited",
+  "title": "Rate limit exceeded",
+  "status": 429, "detail": "12 of 12 messages used", "retryAfter": 3600 }
+```
+
+| Method | Path | Auth | Limit | Response |
+|---|---|---|---|---|
+| GET | `/api/topology` | none | 60/min/IP | Nodes, namespaces, workload summaries |
+| GET | `/api/delivery` | none | 60/min/IP | Argo apps, wave-grouped; resource children lazy |
+| GET | `/api/delivery/{app}/resources` | none | 60/min/IP | Virtualized child list for one app |
+| GET | `/api/stream` | session cookie | 2 concurrent/IP | SSE deltas, `topology` + `delivery` event types |
+| POST | `/api/exec` | session cookie | 20/min/session | Enum command result |
+| POST | `/api/chat` | session + Turnstile | 12/day/session, 6 global concurrent | SSE token stream |
+
+**Pagination**: `/api/delivery/{app}/resources` is cursor-paginated at 100 items. `prometheus` has 68 resources today, but `homelab-root`'s tree will grow.
+
+**Payload budget**: topology response capped at 256 KB, delivery at 256 KB. Exceeding the cap truncates and sets `meta.truncated: true` — never a silent drop.
+
+---
+
+## 5. Security
+
+Mapped against OWASP Top 10 (2021). Every item is a Phase gate, not a wish list.
+
+### A01 Broken Access Control
+
+- `atlas` ServiceAccount: one ClusterRole, verbs `get,list,watch` only, explicit resource list. **No `secrets`. No `*`. No wildcards on apiGroups.**
+- NetworkPolicy on `atlas`: egress restricted to kube API, `prometheus-operated.monitoring`, `reasoning-predictor.llm-serving`, `kmsvc-redis-master.sqs`. Ingress from `ingress-nginx` only.
+- Test: an integration test asserting the SA receives 403 on `get secrets` in every namespace.
+
+### A02 Cryptographic Failures
+
+- Session cookie: signed (HMAC), `HttpOnly`, `Secure`, `SameSite=Lax`, 24h expiry. No PII in the payload — a random session ID only.
+- Signing key from a Kubernetes Secret via SOPS (`sops-secrets` app already exists), never an env literal in a manifest.
+
+### A03 Injection
+
+The primary risk on surface C. Mitigation is structural, not filtering:
+
+- Input parses to a closed command enum. Anything unmatched is rejected before any lookup.
+- Namespace and resource-name arguments are validated by **set membership against the current snapshot**, not by regex or escaping.
+- No shell, no `exec`, no `kubectl` binary present in the container image.
+- Test: fuzz the parser; assert every input outside the allowlist returns a rejection and performs zero upstream calls.
+
+### A04 Insecure Design — information disclosure
+
+The core risk of the whole project. Redaction allowlist, enforced by DTO construction:
+
+**Emitted**: name, namespace, kind, phase, ready counts, restart count, age, node name, health status, sync status, sync wave, an explicit label subset.
+
+**Never emitted**: container env, container args, image digests, image tags, `spec.source.repoURL`, full `spec.source.path`, annotations, pod IPs, cluster IPs, Secret names, `status.conditions[].message`, node internal IPs.
+
+Specific known leaks in current data:
+- `reasoning` container args disclose the entire model and quantization strategy
+- `spec.source.repoURL` discloses a private GitHub repository
+- 21 `Secret` resources appear in Argo trees — render **kind and count only, never names**; `sops-secrets` included
+- `status.conditions[].message` echoes raw errors containing internal hostnames — emit condition **type** only
+
+Test: golden test asserting the serialized snapshot contains none of the denied field names, run against a fixture captured from the real cluster.
+
+### A05 Security Misconfiguration
+
+- Container: `runAsNonRoot`, read-only root filesystem, all capabilities dropped, `seccompProfile: RuntimeDefault`
+- Security headers on all responses: `Content-Security-Policy` (no `unsafe-inline`), `X-Content-Type-Options: nosniff`, `Referrer-Policy: no-referrer`, `Strict-Transport-Security`
+- CORS: same-origin only. No wildcard.
+
+### A07 Authentication Failures
+
+- Anonymous by design; Cloudflare Turnstile gates the first chat message
+- Session rotation on issue; no session fixation vector since there is no login
+
+### A08 Software and Data Integrity
+
+- Image tags are commit SHAs, never `:latest` (fixes Phase 0.3)
+- `pnpm audit` and `govulncheck` in CI, failing the build on high severity
+
+### A09 Logging and Monitoring
+
+- Structured logs: every rejected `/api/exec` input, every rate-limit trip, every chat queue rejection
+- Prometheus metrics from `atlas`: `atlas_chat_concurrent`, `atlas_chat_queue_depth`, `atlas_ratelimit_rejections_total`, `atlas_snapshot_age_seconds`
+- Alertmanager rule: chat queue saturated > 5 min, snapshot age > 60s
+
+### A10 SSRF
+
+- `atlas` calls a fixed, compile-time list of upstream URLs. No user input reaches any URL construction, on any path.
+
+### LLM-specific: prompt injection
+
+- System prompt is a compile-time constant, unreachable by user input
+- Cluster snapshot digest is injected in a delimited block explicitly labelled untrusted data
+- User message is always last
+- **No tool-calling.** The model reads a pre-built digest and cannot query anything. This removes the entire agentic attack surface for v1.
+- Output capped at `max_tokens: 1500` — DeepSeek-R1 will otherwise reason for minutes
+
+---
+
+## 6. Rate limiting
+
+Sized against the verified hard ceiling: **8 concurrent sequences** (`--max-num-seqs=4` × 2 replicas), single GPU node.
+
+**Tier 1 — Cloudflare edge.** WAF, Bot Fight Mode, per-IP rate rules, Turnstile before first chat message. Free, and stops scripted abuse before it reaches your hardware.
+
+**Tier 2 — Kong.** `rate-limiting` plugin, `policy: redis` against `kmsvc-redis-master.sqs:6379` so counters are cluster-wide rather than per-pod (Kong runs 2 replicas — a local policy would silently double every limit). Two profiles: generous for topology and delivery, tight for chat.
+
+**Tier 3 — `atlas`, the tier that actually protects the GPU.**
+
+| Control | Value | Rationale |
+|---|---|---|
+| Global chat semaphore | 6 | Leaves 2 of 8 sequence slots as operator headroom |
+| Queue depth | 20, then reject with `429` | A visible queue beats a wall; an unbounded queue beats nothing |
+| Per-session budget | 12 messages / 24h | Enough to explore, not enough to farm |
+| Per-request timeout | 120s hard, server-side | Independent of client behaviour |
+| Disconnect handling | cancel upstream immediately | **Critical** — a walked-away tab holding 1 of 8 slots is a real outage |
+
+Queue position is streamed to the client as SSE `{"type":"queue","position":N}` events, so waiting is legible rather than a hang.
+
+---
+
+## 7. Streaming
+
+SSE throughout — unidirectional server→client fits every surface, including token streaming. WebSockets are not justified; nothing flows client→server mid-stream.
+
+- Keepalive comment frame every 30s (idle SSE connections die at proxies)
+- `Last-Event-ID` supported on `/api/stream` for resumable topology/delivery deltas; chat is not resumable
+- `req.Context()` threaded to the upstream vLLM request so client abort cancels it — this is the mechanism that enforces the Tier 3 disconnect rule
+- Backpressure: bounded per-client channel; a slow consumer is dropped rather than allowed to grow memory
+- Chat events: `{"type":"reasoning"|"content"|"queue"|"done"|"error"}`. `--reasoning-parser=deepseek_r1` already separates `reasoning_content` from `content` — render thinking in a collapsible block. That block **is** the demo.
+
+**Context budget** (16384 total): system prompt ~300, snapshot digest capped at 2000, `max_tokens` 1500, leaving ~12500 for history. History is truncated oldest-first to fit. The digest is a compact rendering, never raw JSON.
+
+---
+
+## 8. Frontend
+
+- **Topology (B)**: React Flow, force layout. Node → namespace → workload.
+- **Delivery (E)**: React Flow, wave columns left→right from the existing `sync-wave` annotations (0→8). 31 app nodes — an ideal size for a readable DAG. Click an app → side panel with a `react-arborist` virtualized resource tree, children lazy-loaded. Live sync animation `OutOfSync → Syncing → Synced` driven by the Application watch. Push a commit during a demo and the wave cascades — that is the moment worth engineering for.
+- **Terminal (C)**: reuse [components/InteractiveTerminal.tsx](../components/InteractiveTerminal.tsx). Command set: `get nodes`, `get pods <ns>`, `get apps`, `top nodes`, `describe pod <ns> <name>`, `help`.
+- **Chat (D)**: new component. Collapsible reasoning block, queue position, streaming tokens.
+
+Rendering budget: ~550 resources total across the tree. Lazy expansion plus virtualization is required, not optional. Target: initial delivery view interactive in < 1.5s on a cold load.
+
+**Also in scope**: delete the fabricated statistics in [app/page.tsx](../app/page.tsx) — "40% CPU reduction", "99.2% uptime", "Mission-critical", "60% latency cut" — and either wire each card to a real number from `/api/topology` or remove the claim. Five of the six cards link to routes that do not exist (`/infrastructure`, `/systems`, `/llm`, `/kafka`, `/opensource`).
+
+---
+
+## 9. Phases, TDD-first
+
+Each phase is RED → GREEN → REFACTOR. Tests named before implementation exists.
+
+### Phase 1 — `atlas` core (~4 days)
+
+RED
+- `redact_test.go`: golden test — serialized snapshot contains no denied field, against a real-cluster fixture
+- `rbac_test.go`: SA receives 403 on `get secrets`
+- `snapshot_test.go`: informer event produces the expected delta
+
+GREEN: ClusterRole, informers, reducer, DTO construction, Redis publish.
+Verify: `kubectl auth can-i get secrets --as=system:serviceaccount:portfolio:atlas` → `no`.
+
+### Phase 2 — Surface B (~3 days)
+
+RED
+- `stream_test.go`: SSE emits a delta within 5s of a pod state change
+- `topology.test.tsx`: graph re-renders on delta without a full reload
+
+Verify: delete a pod, observe the graph update in < 5s without reloading.
+
+### Phase 3 — Surface E (~3 days)
+
+RED
+- `delivery_test.go`: apps group correctly by `sync-wave`; Secret names absent from output; `repoURL` absent from output
+- `delivery.test.tsx`: 550-node tree renders under the frame budget with virtualization on
+
+Verify: trigger an Argo sync, observe wave-ordered animation.
+
+### Phase 4 — Surface C (~2 days)
+
+RED
+- `exec_parse_test.go`: fuzz corpus — every non-allowlisted input rejects and performs zero upstream calls
+- `exec_test.go`: unknown namespace rejects on snapshot membership, not regex
+
+Verify: attempt injection payloads against `/api/exec`; all rejected, all logged.
+
+### Phase 5 — Surface D + rate limiter, one PR (~5 days)
+
+RED
+- `ratelimit_test.go`: 7th concurrent chat queues rather than reaching vLLM
+- `disconnect_test.go`: client abort cancels the upstream request
+- `budget_test.go`: 13th message in 24h returns 429 with `Retry-After`
+- `injection_test.go`: snapshot content cannot alter system-prompt behaviour
+- `context_test.go`: history truncation keeps total tokens under 16384
+
+Verify: load test at 20 concurrent clients — GPU sequence usage never exceeds 6, no upstream 5xx, queue drains.
+
+**Total: ~17 working days.** Rate limiter ships in the same PR as chat, never after.
+
+---
+
+## 10. Risks
+
+| Risk | Impact | Mitigation |
+|---|---|---|
+| Redaction miss leaks internal detail | High | Allowlist by DTO construction; golden test; manual review of every new field |
+| `worker-1` fails | Chat and all inference down | Out of scope to fix; degrade chat gracefully to "offline", never a hang |
+| 8-slot ceiling makes chat feel broken under traffic | Medium | Visible queue position; consider disabling chat and shipping B/E/C only |
+| Argo CD CRD schema changes | Low | Allowlist construction means new fields are ignored by default |
+| `atlas` compromised | High | Read-only SA, NetworkPolicy egress restriction, no write verbs anywhere |
+| Cost of GPU inference for anonymous visitors | Medium | Per-session daily budget; Turnstile; edge WAF |
+
+---
+
+## 11. Decisions needed before Phase 1
+
+1. **Which GitOps repo owns the portfolio** — GitHub (`riotpiao.homelab.com`) or Forgejo? Blocks 0.1–0.3.
+2. **Apex 403 cause** — tunnel route, WAF rule, or no origin? Blocks 0.5.
+3. **Is `homarr` still wanted?** Deployed and healthy, but from the abandoned plan.
+4. **Does chat stay in v1?** Given the 8-slot ceiling, shipping B + E + C first and treating D as a separate decision is defensible.
+5. **Where does `atlas` live** — this repo, or the homelab repo? Follows from decision 1.