328 lines
17 KiB
Markdown
328 lines
17 KiB
Markdown
# Implementation Plan: `atlas` — Interactive Cluster Visualization
|
||
|
||
**Status**: For review — no code written yet
|
||
**Companion ADR**: [ADR-0001](adr/ADR-0001-atlas-cluster-visualization.md)
|
||
**Supersedes**: `PLAN.md`, `IMPLEMENTATION.md` (Homarr + Terraform — abandoned)
|
||
|
||
---
|
||
|
||
## 1. Scope
|
||
|
||
### In scope
|
||
|
||
| ID | Surface | Description |
|
||
|---|---|---|
|
||
| B | Cluster topology | Live node → namespace → workload graph, health-colored, drill-down |
|
||
| E | Delivery tree | Argo CD app-of-apps as a sync-wave-ordered DAG, live sync animation |
|
||
| C | Terminal | Read-only, enum-dispatched cluster queries in the browser |
|
||
| D | Chat | Streaming ChatGPT-style session against the `reasoning` model |
|
||
|
||
### Out of scope (v1)
|
||
|
||
- Any write operation against the cluster
|
||
- LLM tool-calling / agentic loops against live infrastructure
|
||
- Forgejo CI half of the pipeline view (`forgejo-gitea` is currently stuck `Init:0/3`)
|
||
- Log streaming to the browser (Loki content is unredactable in practice)
|
||
- Authenticated / operator-only views — Grafana already serves that need
|
||
|
||
### Non-goals
|
||
|
||
- Replacing Grafana
|
||
- Multi-cluster support
|
||
- Historical / time-travel views
|
||
|
||
---
|
||
|
||
## 2. Phase 0 — Unblock (blocking; nothing ships until done)
|
||
|
||
Verified problems, in the order they must be fixed:
|
||
|
||
| # | Problem | Evidence |
|
||
|---|---|---|
|
||
| 0.1 | `portfolio` namespace is empty; `portfolio` and `auth-infra` Applications do not exist in the cluster | `kubectl get pods -n portfolio` → no resources; neither name appears in `kubectl get app -n argocd` |
|
||
| 0.2 | `infra/argocd-apps.yaml` `repoURL` = `forgejo.riotpiao.homelab.com` — NXDOMAIN | `dig` |
|
||
| 0.3 | Deployment image `forgejo.riotpiao.homelab.com/rock/portfolio:latest` — dead host, and `:latest` + `imagePullPolicy: IfNotPresent` means a pushed image will never roll out | `infra/portfolio/base/deployment.yaml` |
|
||
| 0.4 | `forgejo-gitea` stuck `Init:0/3` for 3h — no image builds possible | `kubectl get pods -n cicd` |
|
||
| 0.5 | `riotpiao.com` returns HTTP 403 from the Cloudflare edge; no origin headers present | `curl -I https://riotpiao.com` |
|
||
| 0.6 | No test framework installed — TDD is impossible as the repo stands | `package.json` has no test script or runner |
|
||
| 0.7 | `sms` Application Degraded (`macos-bluebubbles` Pending 3h); `longhorn-config` OutOfSync | `kubectl get app -n argocd` |
|
||
|
||
**Decisions required from you before 0.1–0.3 can be actioned** — see ADR open questions 1 and 2.
|
||
|
||
**Actions**
|
||
|
||
1. Resolve which GitOps repo owns the portfolio; delete or correct the losing manifest
|
||
2. Point `repoURL` and the image reference at real hostnames
|
||
3. Replace `:latest` with a commit-SHA tag; set `imagePullPolicy: IfNotPresent` (correct once tags are immutable)
|
||
4. Diagnose `forgejo-gitea` init containers — read the actual init container logs before changing anything
|
||
5. Diagnose the apex 403 — check the tunnel's Public Hostnames list and Cloudflare WAF events; the event log names the blocking rule
|
||
6. Add Vitest + Testing Library + `msw`; add `test` and `test:watch` scripts
|
||
7. Triage 0.7 separately — unrelated to this work, but the delivery tree will render both as red on day one
|
||
|
||
**Verify**
|
||
|
||
```bash
|
||
kubectl get pods -n portfolio # 2/2 Running
|
||
curl -sS -o /dev/null -w '%{http_code}\n' https://riotpiao.com # 200
|
||
pnpm test # runner executes, 0 tests, exit 0
|
||
```
|
||
|
||
---
|
||
|
||
## 3. Architecture
|
||
|
||
```
|
||
kube API (client-go informers) ─┐
|
||
Prometheus /api/v1/query ├──> atlas (Go, ns: portfolio, RO ServiceAccount)
|
||
Argo CD Application CRs ┘ │
|
||
├─ snapshot (in-memory, redacted at write)
|
||
├─ Redis pub/sub (kmsvc-redis-master.sqs:6379)
|
||
└─ HTTP
|
||
GET /api/topology
|
||
GET /api/delivery
|
||
GET /api/stream (SSE)
|
||
POST /api/exec
|
||
POST /api/chat (SSE)
|
||
└──> reasoning-predictor.llm-serving:80
|
||
```
|
||
|
||
Event-driven, per project architectural preference: informers push to a reducer, the reducer publishes deltas to Redis, SSE handlers subscribe. No request-triggered upstream calls anywhere in the read path.
|
||
|
||
Snapshot is redacted **at write time**, not at serialization time. A field that never enters the snapshot cannot leak from any surface.
|
||
|
||
### Language
|
||
|
||
Go, for `client-go` informers and because it matches the rest of the platform. Follow the repo's Go skill set (`go-naming`, `go-concurrency`, `go-error-handling`, `go-context`) — notably: every upstream call carries a context, no naked returns, no `_ =` on errors.
|
||
|
||
---
|
||
|
||
## 4. API contract
|
||
|
||
Envelope for all non-stream responses:
|
||
|
||
```json
|
||
{ "data": { }, "meta": { "snapshotAge": 3.2, "generation": 88412 } }
|
||
```
|
||
|
||
Errors follow RFC 9457 (`application/problem+json`):
|
||
|
||
```json
|
||
{ "type": "https://riotpiao.com/errors/rate-limited",
|
||
"title": "Rate limit exceeded",
|
||
"status": 429, "detail": "12 of 12 messages used", "retryAfter": 3600 }
|
||
```
|
||
|
||
| Method | Path | Auth | Limit | Response |
|
||
|---|---|---|---|---|
|
||
| GET | `/api/topology` | none | 60/min/IP | Nodes, namespaces, workload summaries |
|
||
| GET | `/api/delivery` | none | 60/min/IP | Argo apps, wave-grouped; resource children lazy |
|
||
| GET | `/api/delivery/{app}/resources` | none | 60/min/IP | Virtualized child list for one app |
|
||
| GET | `/api/stream` | session cookie | 2 concurrent/IP | SSE deltas, `topology` + `delivery` event types |
|
||
| POST | `/api/exec` | session cookie | 20/min/session | Enum command result |
|
||
| POST | `/api/chat` | session + Turnstile | 12/day/session, 6 global concurrent | SSE token stream |
|
||
|
||
**Pagination**: `/api/delivery/{app}/resources` is cursor-paginated at 100 items. `prometheus` has 68 resources today, but `homelab-root`'s tree will grow.
|
||
|
||
**Payload budget**: topology response capped at 256 KB, delivery at 256 KB. Exceeding the cap truncates and sets `meta.truncated: true` — never a silent drop.
|
||
|
||
---
|
||
|
||
## 5. Security
|
||
|
||
Mapped against OWASP Top 10 (2021). Every item is a Phase gate, not a wish list.
|
||
|
||
### A01 Broken Access Control
|
||
|
||
- `atlas` ServiceAccount: one ClusterRole, verbs `get,list,watch` only, explicit resource list. **No `secrets`. No `*`. No wildcards on apiGroups.**
|
||
- NetworkPolicy on `atlas`: egress restricted to kube API, `prometheus-operated.monitoring`, `reasoning-predictor.llm-serving`, `kmsvc-redis-master.sqs`. Ingress from `ingress-nginx` only.
|
||
- Test: an integration test asserting the SA receives 403 on `get secrets` in every namespace.
|
||
|
||
### A02 Cryptographic Failures
|
||
|
||
- Session cookie: signed (HMAC), `HttpOnly`, `Secure`, `SameSite=Lax`, 24h expiry. No PII in the payload — a random session ID only.
|
||
- Signing key from a Kubernetes Secret via SOPS (`sops-secrets` app already exists), never an env literal in a manifest.
|
||
|
||
### A03 Injection
|
||
|
||
The primary risk on surface C. Mitigation is structural, not filtering:
|
||
|
||
- Input parses to a closed command enum. Anything unmatched is rejected before any lookup.
|
||
- Namespace and resource-name arguments are validated by **set membership against the current snapshot**, not by regex or escaping.
|
||
- No shell, no `exec`, no `kubectl` binary present in the container image.
|
||
- Test: fuzz the parser; assert every input outside the allowlist returns a rejection and performs zero upstream calls.
|
||
|
||
### A04 Insecure Design — information disclosure
|
||
|
||
The core risk of the whole project. Redaction allowlist, enforced by DTO construction:
|
||
|
||
**Emitted**: name, namespace, kind, phase, ready counts, restart count, age, node name, health status, sync status, sync wave, an explicit label subset.
|
||
|
||
**Never emitted**: container env, container args, image digests, image tags, `spec.source.repoURL`, full `spec.source.path`, annotations, pod IPs, cluster IPs, Secret names, `status.conditions[].message`, node internal IPs.
|
||
|
||
Specific known leaks in current data:
|
||
- `reasoning` container args disclose the entire model and quantization strategy
|
||
- `spec.source.repoURL` discloses a private GitHub repository
|
||
- 21 `Secret` resources appear in Argo trees — render **kind and count only, never names**; `sops-secrets` included
|
||
- `status.conditions[].message` echoes raw errors containing internal hostnames — emit condition **type** only
|
||
|
||
Test: golden test asserting the serialized snapshot contains none of the denied field names, run against a fixture captured from the real cluster.
|
||
|
||
### A05 Security Misconfiguration
|
||
|
||
- Container: `runAsNonRoot`, read-only root filesystem, all capabilities dropped, `seccompProfile: RuntimeDefault`
|
||
- Security headers on all responses: `Content-Security-Policy` (no `unsafe-inline`), `X-Content-Type-Options: nosniff`, `Referrer-Policy: no-referrer`, `Strict-Transport-Security`
|
||
- CORS: same-origin only. No wildcard.
|
||
|
||
### A07 Authentication Failures
|
||
|
||
- Anonymous by design; Cloudflare Turnstile gates the first chat message
|
||
- Session rotation on issue; no session fixation vector since there is no login
|
||
|
||
### A08 Software and Data Integrity
|
||
|
||
- Image tags are commit SHAs, never `:latest` (fixes Phase 0.3)
|
||
- `pnpm audit` and `govulncheck` in CI, failing the build on high severity
|
||
|
||
### A09 Logging and Monitoring
|
||
|
||
- Structured logs: every rejected `/api/exec` input, every rate-limit trip, every chat queue rejection
|
||
- Prometheus metrics from `atlas`: `atlas_chat_concurrent`, `atlas_chat_queue_depth`, `atlas_ratelimit_rejections_total`, `atlas_snapshot_age_seconds`
|
||
- Alertmanager rule: chat queue saturated > 5 min, snapshot age > 60s
|
||
|
||
### A10 SSRF
|
||
|
||
- `atlas` calls a fixed, compile-time list of upstream URLs. No user input reaches any URL construction, on any path.
|
||
|
||
### LLM-specific: prompt injection
|
||
|
||
- System prompt is a compile-time constant, unreachable by user input
|
||
- Cluster snapshot digest is injected in a delimited block explicitly labelled untrusted data
|
||
- User message is always last
|
||
- **No tool-calling.** The model reads a pre-built digest and cannot query anything. This removes the entire agentic attack surface for v1.
|
||
- Output capped at `max_tokens: 1500` — DeepSeek-R1 will otherwise reason for minutes
|
||
|
||
---
|
||
|
||
## 6. Rate limiting
|
||
|
||
Sized against the verified hard ceiling: **8 concurrent sequences** (`--max-num-seqs=4` × 2 replicas), single GPU node.
|
||
|
||
**Tier 1 — Cloudflare edge.** WAF, Bot Fight Mode, per-IP rate rules, Turnstile before first chat message. Free, and stops scripted abuse before it reaches your hardware.
|
||
|
||
**Tier 2 — Kong.** `rate-limiting` plugin, `policy: redis` against `kmsvc-redis-master.sqs:6379` so counters are cluster-wide rather than per-pod (Kong runs 2 replicas — a local policy would silently double every limit). Two profiles: generous for topology and delivery, tight for chat.
|
||
|
||
**Tier 3 — `atlas`, the tier that actually protects the GPU.**
|
||
|
||
| Control | Value | Rationale |
|
||
|---|---|---|
|
||
| Global chat semaphore | 6 | Leaves 2 of 8 sequence slots as operator headroom |
|
||
| Queue depth | 20, then reject with `429` | A visible queue beats a wall; an unbounded queue beats nothing |
|
||
| Per-session budget | 12 messages / 24h | Enough to explore, not enough to farm |
|
||
| Per-request timeout | 120s hard, server-side | Independent of client behaviour |
|
||
| Disconnect handling | cancel upstream immediately | **Critical** — a walked-away tab holding 1 of 8 slots is a real outage |
|
||
|
||
Queue position is streamed to the client as SSE `{"type":"queue","position":N}` events, so waiting is legible rather than a hang.
|
||
|
||
---
|
||
|
||
## 7. Streaming
|
||
|
||
SSE throughout — unidirectional server→client fits every surface, including token streaming. WebSockets are not justified; nothing flows client→server mid-stream.
|
||
|
||
- Keepalive comment frame every 30s (idle SSE connections die at proxies)
|
||
- `Last-Event-ID` supported on `/api/stream` for resumable topology/delivery deltas; chat is not resumable
|
||
- `req.Context()` threaded to the upstream vLLM request so client abort cancels it — this is the mechanism that enforces the Tier 3 disconnect rule
|
||
- Backpressure: bounded per-client channel; a slow consumer is dropped rather than allowed to grow memory
|
||
- Chat events: `{"type":"reasoning"|"content"|"queue"|"done"|"error"}`. `--reasoning-parser=deepseek_r1` already separates `reasoning_content` from `content` — render thinking in a collapsible block. That block **is** the demo.
|
||
|
||
**Context budget** (16384 total): system prompt ~300, snapshot digest capped at 2000, `max_tokens` 1500, leaving ~12500 for history. History is truncated oldest-first to fit. The digest is a compact rendering, never raw JSON.
|
||
|
||
---
|
||
|
||
## 8. Frontend
|
||
|
||
- **Topology (B)**: React Flow, force layout. Node → namespace → workload.
|
||
- **Delivery (E)**: React Flow, wave columns left→right from the existing `sync-wave` annotations (0→8). 31 app nodes — an ideal size for a readable DAG. Click an app → side panel with a `react-arborist` virtualized resource tree, children lazy-loaded. Live sync animation `OutOfSync → Syncing → Synced` driven by the Application watch. Push a commit during a demo and the wave cascades — that is the moment worth engineering for.
|
||
- **Terminal (C)**: reuse [components/InteractiveTerminal.tsx](../components/InteractiveTerminal.tsx). Command set: `get nodes`, `get pods <ns>`, `get apps`, `top nodes`, `describe pod <ns> <name>`, `help`.
|
||
- **Chat (D)**: new component. Collapsible reasoning block, queue position, streaming tokens.
|
||
|
||
Rendering budget: ~550 resources total across the tree. Lazy expansion plus virtualization is required, not optional. Target: initial delivery view interactive in < 1.5s on a cold load.
|
||
|
||
**Also in scope**: delete the fabricated statistics in [app/page.tsx](../app/page.tsx) — "40% CPU reduction", "99.2% uptime", "Mission-critical", "60% latency cut" — and either wire each card to a real number from `/api/topology` or remove the claim. Five of the six cards link to routes that do not exist (`/infrastructure`, `/systems`, `/llm`, `/kafka`, `/opensource`).
|
||
|
||
---
|
||
|
||
## 9. Phases, TDD-first
|
||
|
||
Each phase is RED → GREEN → REFACTOR. Tests named before implementation exists.
|
||
|
||
### Phase 1 — `atlas` core (~4 days)
|
||
|
||
RED
|
||
- `redact_test.go`: golden test — serialized snapshot contains no denied field, against a real-cluster fixture
|
||
- `rbac_test.go`: SA receives 403 on `get secrets`
|
||
- `snapshot_test.go`: informer event produces the expected delta
|
||
|
||
GREEN: ClusterRole, informers, reducer, DTO construction, Redis publish.
|
||
Verify: `kubectl auth can-i get secrets --as=system:serviceaccount:portfolio:atlas` → `no`.
|
||
|
||
### Phase 2 — Surface B (~3 days)
|
||
|
||
RED
|
||
- `stream_test.go`: SSE emits a delta within 5s of a pod state change
|
||
- `topology.test.tsx`: graph re-renders on delta without a full reload
|
||
|
||
Verify: delete a pod, observe the graph update in < 5s without reloading.
|
||
|
||
### Phase 3 — Surface E (~3 days)
|
||
|
||
RED
|
||
- `delivery_test.go`: apps group correctly by `sync-wave`; Secret names absent from output; `repoURL` absent from output
|
||
- `delivery.test.tsx`: 550-node tree renders under the frame budget with virtualization on
|
||
|
||
Verify: trigger an Argo sync, observe wave-ordered animation.
|
||
|
||
### Phase 4 — Surface C (~2 days)
|
||
|
||
RED
|
||
- `exec_parse_test.go`: fuzz corpus — every non-allowlisted input rejects and performs zero upstream calls
|
||
- `exec_test.go`: unknown namespace rejects on snapshot membership, not regex
|
||
|
||
Verify: attempt injection payloads against `/api/exec`; all rejected, all logged.
|
||
|
||
### Phase 5 — Surface D + rate limiter, one PR (~5 days)
|
||
|
||
RED
|
||
- `ratelimit_test.go`: 7th concurrent chat queues rather than reaching vLLM
|
||
- `disconnect_test.go`: client abort cancels the upstream request
|
||
- `budget_test.go`: 13th message in 24h returns 429 with `Retry-After`
|
||
- `injection_test.go`: snapshot content cannot alter system-prompt behaviour
|
||
- `context_test.go`: history truncation keeps total tokens under 16384
|
||
|
||
Verify: load test at 20 concurrent clients — GPU sequence usage never exceeds 6, no upstream 5xx, queue drains.
|
||
|
||
**Total: ~17 working days.** Rate limiter ships in the same PR as chat, never after.
|
||
|
||
---
|
||
|
||
## 10. Risks
|
||
|
||
| Risk | Impact | Mitigation |
|
||
|---|---|---|
|
||
| Redaction miss leaks internal detail | High | Allowlist by DTO construction; golden test; manual review of every new field |
|
||
| `worker-1` fails | Chat and all inference down | Out of scope to fix; degrade chat gracefully to "offline", never a hang |
|
||
| 8-slot ceiling makes chat feel broken under traffic | Medium | Visible queue position; consider disabling chat and shipping B/E/C only |
|
||
| Argo CD CRD schema changes | Low | Allowlist construction means new fields are ignored by default |
|
||
| `atlas` compromised | High | Read-only SA, NetworkPolicy egress restriction, no write verbs anywhere |
|
||
| Cost of GPU inference for anonymous visitors | Medium | Per-session daily budget; Turnstile; edge WAF |
|
||
|
||
---
|
||
|
||
## 11. Decisions needed before Phase 1
|
||
|
||
1. **Which GitOps repo owns the portfolio** — GitHub (`riotpiao.homelab.com`) or Forgejo? Blocks 0.1–0.3.
|
||
2. **Apex 403 cause** — tunnel route, WAF rule, or no origin? Blocks 0.5.
|
||
3. **Is `homarr` still wanted?** Deployed and healthy, but from the abandoned plan.
|
||
4. **Does chat stay in v1?** Given the 8-slot ceiling, shipping B + E + C first and treating D as a separate decision is defensible.
|
||
5. **Where does `atlas` live** — this repo, or the homelab repo? Follows from decision 1.
|