riotpiao.com/docs/PLAN-atlas.md

# Implementation Plan: `atlas` — Interactive Cluster Visualization

**Status**: For review — no code written yet
**Companion ADR**: [ADR-0001](adr/ADR-0001-atlas-cluster-visualization.md)
**Supersedes**: `PLAN.md`, `IMPLEMENTATION.md` (Homarr + Terraform — abandoned)

---

## 1. Scope

### In scope

| ID | Surface | Description |
|---|---|---|
| B | Cluster topology | Live node → namespace → workload graph, health-colored, drill-down |
| E | Delivery tree | Argo CD app-of-apps as a sync-wave-ordered DAG, live sync animation |
| C | Terminal | Read-only, enum-dispatched cluster queries in the browser |
| D | Chat | Streaming ChatGPT-style session against the `reasoning` model |

### Out of scope (v1)

- Any write operation against the cluster
- LLM tool-calling / agentic loops against live infrastructure
- Forgejo CI half of the pipeline view (`forgejo-gitea` is currently stuck `Init:0/3`)
- Log streaming to the browser (Loki content is unredactable in practice)
- Authenticated / operator-only views — Grafana already serves that need

### Non-goals

- Replacing Grafana
- Multi-cluster support
- Historical / time-travel views

---

## 2. Phase 0 — Unblock (blocking; nothing ships until done)

Verified problems, in the order they must be fixed:

| # | Problem | Evidence |
|---|---|---|
| 0.1 | `portfolio` namespace is empty; `portfolio` and `auth-infra` Applications do not exist in the cluster | `kubectl get pods -n portfolio` → no resources; neither name appears in `kubectl get app -n argocd` |
| 0.2 | `infra/argocd-apps.yaml` `repoURL` = `forgejo.riotpiao.homelab.com` — NXDOMAIN | `dig` |
| 0.3 | Deployment image `forgejo.riotpiao.homelab.com/rock/portfolio:latest` — dead host, and `:latest` + `imagePullPolicy: IfNotPresent` means a pushed image will never roll out | `infra/portfolio/base/deployment.yaml` |
| 0.4 | `forgejo-gitea` stuck `Init:0/3` for 3h — no image builds possible | `kubectl get pods -n cicd` |
| 0.5 | `riotpiao.com` returns HTTP 403 from the Cloudflare edge; no origin headers present | `curl -I https://riotpiao.com` |
| 0.6 | No test framework installed — TDD is impossible as the repo stands | `package.json` has no test script or runner |
| 0.7 | `sms` Application Degraded (`macos-bluebubbles` Pending 3h); `longhorn-config` OutOfSync | `kubectl get app -n argocd` |

**Decisions required from you before 0.1–0.3 can be actioned** — see ADR open questions 1 and 2.

**Actions**

1. Resolve which GitOps repo owns the portfolio; delete or correct the losing manifest
2. Point `repoURL` and the image reference at real hostnames
3. Replace `:latest` with a commit-SHA tag; set `imagePullPolicy: IfNotPresent` (correct once tags are immutable)
4. Diagnose `forgejo-gitea` init containers — read the actual init container logs before changing anything
5. Diagnose the apex 403 — check the tunnel's Public Hostnames list and Cloudflare WAF events; the event log names the blocking rule
6. Add Vitest + Testing Library + `msw`; add `test` and `test:watch` scripts
7. Triage 0.7 separately — unrelated to this work, but the delivery tree will render both as red on day one

**Verify**

```bash
kubectl get pods -n portfolio                     # 2/2 Running
curl -sS -o /dev/null -w '%{http_code}\n' https://riotpiao.com   # 200
pnpm test                                          # runner executes, 0 tests, exit 0
```

---

## 3. Architecture

```
kube API (client-go informers)  ─┐
Prometheus /api/v1/query         ├──> atlas (Go, ns: portfolio, RO ServiceAccount)
Argo CD Application CRs          ┘         │
                                           ├─ snapshot (in-memory, redacted at write)
                                           ├─ Redis pub/sub (kmsvc-redis-master.sqs:6379)
                                           └─ HTTP
                                               GET  /api/topology
                                               GET  /api/delivery
                                               GET  /api/stream      (SSE)
                                               POST /api/exec
                                               POST /api/chat        (SSE)
                                                      └──> reasoning-predictor.llm-serving:80
```

Event-driven, per project architectural preference: informers push to a reducer, the reducer publishes deltas to Redis, SSE handlers subscribe. No request-triggered upstream calls anywhere in the read path.

Snapshot is redacted **at write time**, not at serialization time. A field that never enters the snapshot cannot leak from any surface.

### Language

Go, for `client-go` informers and because it matches the rest of the platform. Follow the repo's Go skill set (`go-naming`, `go-concurrency`, `go-error-handling`, `go-context`) — notably: every upstream call carries a context, no naked returns, no `_ =` on errors.

---

## 4. API contract

Envelope for all non-stream responses:

```json
{ "data": { }, "meta": { "snapshotAge": 3.2, "generation": 88412 } }
```

Errors follow RFC 9457 (`application/problem+json`):

```json
{ "type": "https://riotpiao.com/errors/rate-limited",
  "title": "Rate limit exceeded",
  "status": 429, "detail": "12 of 12 messages used", "retryAfter": 3600 }
```

| Method | Path | Auth | Limit | Response |
|---|---|---|---|---|
| GET | `/api/topology` | none | 60/min/IP | Nodes, namespaces, workload summaries |
| GET | `/api/delivery` | none | 60/min/IP | Argo apps, wave-grouped; resource children lazy |
| GET | `/api/delivery/{app}/resources` | none | 60/min/IP | Virtualized child list for one app |
| GET | `/api/stream` | session cookie | 2 concurrent/IP | SSE deltas, `topology` + `delivery` event types |
| POST | `/api/exec` | session cookie | 20/min/session | Enum command result |
| POST | `/api/chat` | session + Turnstile | 12/day/session, 6 global concurrent | SSE token stream |

**Pagination**: `/api/delivery/{app}/resources` is cursor-paginated at 100 items. `prometheus` has 68 resources today, but `homelab-root`'s tree will grow.

**Payload budget**: topology response capped at 256 KB, delivery at 256 KB. Exceeding the cap truncates and sets `meta.truncated: true` — never a silent drop.

---

## 5. Security

Mapped against OWASP Top 10 (2021). Every item is a Phase gate, not a wish list.

### A01 Broken Access Control

- `atlas` ServiceAccount: one ClusterRole, verbs `get,list,watch` only, explicit resource list. **No `secrets`. No `*`. No wildcards on apiGroups.**
- NetworkPolicy on `atlas`: egress restricted to kube API, `prometheus-operated.monitoring`, `reasoning-predictor.llm-serving`, `kmsvc-redis-master.sqs`. Ingress from `ingress-nginx` only.
- Test: an integration test asserting the SA receives 403 on `get secrets` in every namespace.

### A02 Cryptographic Failures

- Session cookie: signed (HMAC), `HttpOnly`, `Secure`, `SameSite=Lax`, 24h expiry. No PII in the payload — a random session ID only.
- Signing key from a Kubernetes Secret via SOPS (`sops-secrets` app already exists), never an env literal in a manifest.

### A03 Injection

The primary risk on surface C. Mitigation is structural, not filtering:

- Input parses to a closed command enum. Anything unmatched is rejected before any lookup.
- Namespace and resource-name arguments are validated by **set membership against the current snapshot**, not by regex or escaping.
- No shell, no `exec`, no `kubectl` binary present in the container image.
- Test: fuzz the parser; assert every input outside the allowlist returns a rejection and performs zero upstream calls.

### A04 Insecure Design — information disclosure

The core risk of the whole project. Redaction allowlist, enforced by DTO construction:

**Emitted**: name, namespace, kind, phase, ready counts, restart count, age, node name, health status, sync status, sync wave, an explicit label subset.

**Never emitted**: container env, container args, image digests, image tags, `spec.source.repoURL`, full `spec.source.path`, annotations, pod IPs, cluster IPs, Secret names, `status.conditions[].message`, node internal IPs.

Specific known leaks in current data:
- `reasoning` container args disclose the entire model and quantization strategy
- `spec.source.repoURL` discloses a private GitHub repository
- 21 `Secret` resources appear in Argo trees — render **kind and count only, never names**; `sops-secrets` included
- `status.conditions[].message` echoes raw errors containing internal hostnames — emit condition **type** only

Test: golden test asserting the serialized snapshot contains none of the denied field names, run against a fixture captured from the real cluster.

### A05 Security Misconfiguration

- Container: `runAsNonRoot`, read-only root filesystem, all capabilities dropped, `seccompProfile: RuntimeDefault`
- Security headers on all responses: `Content-Security-Policy` (no `unsafe-inline`), `X-Content-Type-Options: nosniff`, `Referrer-Policy: no-referrer`, `Strict-Transport-Security`
- CORS: same-origin only. No wildcard.

### A07 Authentication Failures

- Anonymous by design; Cloudflare Turnstile gates the first chat message
- Session rotation on issue; no session fixation vector since there is no login

### A08 Software and Data Integrity

- Image tags are commit SHAs, never `:latest` (fixes Phase 0.3)
- `pnpm audit` and `govulncheck` in CI, failing the build on high severity

### A09 Logging and Monitoring

- Structured logs: every rejected `/api/exec` input, every rate-limit trip, every chat queue rejection
- Prometheus metrics from `atlas`: `atlas_chat_concurrent`, `atlas_chat_queue_depth`, `atlas_ratelimit_rejections_total`, `atlas_snapshot_age_seconds`
- Alertmanager rule: chat queue saturated > 5 min, snapshot age > 60s

### A10 SSRF

- `atlas` calls a fixed, compile-time list of upstream URLs. No user input reaches any URL construction, on any path.

### LLM-specific: prompt injection

- System prompt is a compile-time constant, unreachable by user input
- Cluster snapshot digest is injected in a delimited block explicitly labelled untrusted data
- User message is always last
- **No tool-calling.** The model reads a pre-built digest and cannot query anything. This removes the entire agentic attack surface for v1.
- Output capped at `max_tokens: 1500` — DeepSeek-R1 will otherwise reason for minutes

---

## 6. Rate limiting

Sized against the verified hard ceiling: **8 concurrent sequences** (`--max-num-seqs=4` × 2 replicas), single GPU node.

**Tier 1 — Cloudflare edge.** WAF, Bot Fight Mode, per-IP rate rules, Turnstile before first chat message. Free, and stops scripted abuse before it reaches your hardware.

**Tier 2 — Kong.** `rate-limiting` plugin, `policy: redis` against `kmsvc-redis-master.sqs:6379` so counters are cluster-wide rather than per-pod (Kong runs 2 replicas — a local policy would silently double every limit). Two profiles: generous for topology and delivery, tight for chat.

**Tier 3 — `atlas`, the tier that actually protects the GPU.**

| Control | Value | Rationale |
|---|---|---|
| Global chat semaphore | 6 | Leaves 2 of 8 sequence slots as operator headroom |
| Queue depth | 20, then reject with `429` | A visible queue beats a wall; an unbounded queue beats nothing |
| Per-session budget | 12 messages / 24h | Enough to explore, not enough to farm |
| Per-request timeout | 120s hard, server-side | Independent of client behaviour |
| Disconnect handling | cancel upstream immediately | **Critical** — a walked-away tab holding 1 of 8 slots is a real outage |

Queue position is streamed to the client as SSE `{"type":"queue","position":N}` events, so waiting is legible rather than a hang.

---

## 7. Streaming

SSE throughout — unidirectional server→client fits every surface, including token streaming. WebSockets are not justified; nothing flows client→server mid-stream.

- Keepalive comment frame every 30s (idle SSE connections die at proxies)
- `Last-Event-ID` supported on `/api/stream` for resumable topology/delivery deltas; chat is not resumable
- `req.Context()` threaded to the upstream vLLM request so client abort cancels it — this is the mechanism that enforces the Tier 3 disconnect rule
- Backpressure: bounded per-client channel; a slow consumer is dropped rather than allowed to grow memory
- Chat events: `{"type":"reasoning"|"content"|"queue"|"done"|"error"}`. `--reasoning-parser=deepseek_r1` already separates `reasoning_content` from `content` — render thinking in a collapsible block. That block **is** the demo.

**Context budget** (16384 total): system prompt ~300, snapshot digest capped at 2000, `max_tokens` 1500, leaving ~12500 for history. History is truncated oldest-first to fit. The digest is a compact rendering, never raw JSON.

---

## 8. Frontend

- **Topology (B)**: React Flow, force layout. Node → namespace → workload.
- **Delivery (E)**: React Flow, wave columns left→right from the existing `sync-wave` annotations (0→8). 31 app nodes — an ideal size for a readable DAG. Click an app → side panel with a `react-arborist` virtualized resource tree, children lazy-loaded. Live sync animation `OutOfSync → Syncing → Synced` driven by the Application watch. Push a commit during a demo and the wave cascades — that is the moment worth engineering for.
- **Terminal (C)**: reuse [components/InteractiveTerminal.tsx](../components/InteractiveTerminal.tsx). Command set: `get nodes`, `get pods <ns>`, `get apps`, `top nodes`, `describe pod <ns> <name>`, `help`.
- **Chat (D)**: new component. Collapsible reasoning block, queue position, streaming tokens.

Rendering budget: ~550 resources total across the tree. Lazy expansion plus virtualization is required, not optional. Target: initial delivery view interactive in < 1.5s on a cold load.

**Also in scope**: delete the fabricated statistics in [app/page.tsx](../app/page.tsx) — "40% CPU reduction", "99.2% uptime", "Mission-critical", "60% latency cut" — and either wire each card to a real number from `/api/topology` or remove the claim. Five of the six cards link to routes that do not exist (`/infrastructure`, `/systems`, `/llm`, `/kafka`, `/opensource`).

---

## 9. Phases, TDD-first

Each phase is RED → GREEN → REFACTOR. Tests named before implementation exists.

### Phase 1 — `atlas` core (~4 days)

RED
- `redact_test.go`: golden test — serialized snapshot contains no denied field, against a real-cluster fixture
- `rbac_test.go`: SA receives 403 on `get secrets`
- `snapshot_test.go`: informer event produces the expected delta

GREEN: ClusterRole, informers, reducer, DTO construction, Redis publish.
Verify: `kubectl auth can-i get secrets --as=system:serviceaccount:portfolio:atlas` → `no`.

### Phase 2 — Surface B (~3 days)

RED
- `stream_test.go`: SSE emits a delta within 5s of a pod state change
- `topology.test.tsx`: graph re-renders on delta without a full reload

Verify: delete a pod, observe the graph update in < 5s without reloading.

### Phase 3 — Surface E (~3 days)

RED
- `delivery_test.go`: apps group correctly by `sync-wave`; Secret names absent from output; `repoURL` absent from output
- `delivery.test.tsx`: 550-node tree renders under the frame budget with virtualization on

Verify: trigger an Argo sync, observe wave-ordered animation.

### Phase 4 — Surface C (~2 days)

RED
- `exec_parse_test.go`: fuzz corpus — every non-allowlisted input rejects and performs zero upstream calls
- `exec_test.go`: unknown namespace rejects on snapshot membership, not regex

Verify: attempt injection payloads against `/api/exec`; all rejected, all logged.

### Phase 5 — Surface D + rate limiter, one PR (~5 days)

RED
- `ratelimit_test.go`: 7th concurrent chat queues rather than reaching vLLM
- `disconnect_test.go`: client abort cancels the upstream request
- `budget_test.go`: 13th message in 24h returns 429 with `Retry-After`
- `injection_test.go`: snapshot content cannot alter system-prompt behaviour
- `context_test.go`: history truncation keeps total tokens under 16384

Verify: load test at 20 concurrent clients — GPU sequence usage never exceeds 6, no upstream 5xx, queue drains.

**Total: ~17 working days.** Rate limiter ships in the same PR as chat, never after.

---

## 10. Risks

| Risk | Impact | Mitigation |
|---|---|---|
| Redaction miss leaks internal detail | High | Allowlist by DTO construction; golden test; manual review of every new field |
| `worker-1` fails | Chat and all inference down | Out of scope to fix; degrade chat gracefully to "offline", never a hang |
| 8-slot ceiling makes chat feel broken under traffic | Medium | Visible queue position; consider disabling chat and shipping B/E/C only |
| Argo CD CRD schema changes | Low | Allowlist construction means new fields are ignored by default |
| `atlas` compromised | High | Read-only SA, NetworkPolicy egress restriction, no write verbs anywhere |
| Cost of GPU inference for anonymous visitors | Medium | Per-session daily budget; Turnstile; edge WAF |

---

## 11. Decisions needed before Phase 1

1. **Which GitOps repo owns the portfolio** — GitHub (`riotpiao.homelab.com`) or Forgejo? Blocks 0.1–0.3.
2. **Apex 403 cause** — tunnel route, WAF rule, or no origin? Blocks 0.5.
3. **Is `homarr` still wanted?** Deployed and healthy, but from the abandoned plan.
4. **Does chat stay in v1?** Given the 8-slot ceiling, shipping B + E + C first and treating D as a separate decision is defensible.
5. **Where does `atlas` live** — this repo, or the homelab repo? Follows from decision 1.