Files

T

Story Crater Bot 6f496566df fix(ci): rename .forgejo to .gitea, use golang runner with docker container

2026-08-31 17:33:05 -07:00

17 KiB

Raw Blame History

Implementation Plan: `atlas` — Interactive Cluster Visualization

Status: For review — no code written yet Companion ADR: ADR-0001 Supersedes: PLAN.md, IMPLEMENTATION.md (Homarr + Terraform — abandoned)

1. Scope

In scope

ID	Surface	Description
B	Cluster topology	Live node → namespace → workload graph, health-colored, drill-down
E	Delivery tree	Argo CD app-of-apps as a sync-wave-ordered DAG, live sync animation
C	Terminal	Read-only, enum-dispatched cluster queries in the browser
D	Chat	Streaming ChatGPT-style session against the `reasoning` model

Out of scope (v1)

Any write operation against the cluster
LLM tool-calling / agentic loops against live infrastructure
Forgejo CI half of the pipeline view (forgejo-gitea is currently stuck Init:0/3)
Log streaming to the browser (Loki content is unredactable in practice)
Authenticated / operator-only views — Grafana already serves that need

Non-goals

Replacing Grafana
Multi-cluster support
Historical / time-travel views

2. Phase 0 — Unblock (blocking; nothing ships until done)

Verified problems, in the order they must be fixed:

#	Problem	Evidence
0.1	`portfolio` namespace is empty; `portfolio` and `auth-infra` Applications do not exist in the cluster	`kubectl get pods -n portfolio` → no resources; neither name appears in `kubectl get app -n argocd`
0.2	`infra/argocd-apps.yaml` `repoURL` = `forgejo.riotpiao.com` — NXDOMAIN	`dig`
0.3	Deployment image `forgejo.riotpiao.com/rock/portfolio:latest` — dead host, and `:latest` + `imagePullPolicy: IfNotPresent` means a pushed image will never roll out	`infra/portfolio/base/deployment.yaml`
0.4	`forgejo-gitea` stuck `Init:0/3` for 3h — no image builds possible	`kubectl get pods -n cicd`
0.5	`riotpiao.com` returns HTTP 403 from the Cloudflare edge; no origin headers present	`curl -I https://riotpiao.com`
0.6	No test framework installed — TDD is impossible as the repo stands	`package.json` has no test script or runner
0.7	`sms` Application Degraded (`macos-bluebubbles` Pending 3h); `longhorn-config` OutOfSync	`kubectl get app -n argocd`

Decisions required from you before 0.1–0.3 can be actioned — see ADR open questions 1 and 2.

Actions

Resolve which GitOps repo owns the portfolio; delete or correct the losing manifest
Point repoURL and the image reference at real hostnames
Replace :latest with a commit-SHA tag; set imagePullPolicy: IfNotPresent (correct once tags are immutable)
Diagnose forgejo-gitea init containers — read the actual init container logs before changing anything
Diagnose the apex 403 — check the tunnel's Public Hostnames list and Cloudflare WAF events; the event log names the blocking rule
Add Vitest + Testing Library + msw; add test and test:watch scripts
Triage 0.7 separately — unrelated to this work, but the delivery tree will render both as red on day one

Verify

kubectl get pods -n portfolio                     # 2/2 Running
curl -sS -o /dev/null -w '%{http_code}\n' https://riotpiao.com   # 200
pnpm test                                          # runner executes, 0 tests, exit 0

3. Architecture

kube API (client-go informers)  ─┐
Prometheus /api/v1/query         ├──> atlas (Go, ns: portfolio, RO ServiceAccount)
Argo CD Application CRs          ┘         │
                                           ├─ snapshot (in-memory, redacted at write)
                                           ├─ Redis pub/sub (kmsvc-redis-master.sqs:6379)
                                           └─ HTTP
                                               GET  /api/topology
                                               GET  /api/delivery
                                               GET  /api/stream      (SSE)
                                               POST /api/exec
                                               POST /api/chat        (SSE)
                                                      └──> reasoning-predictor.llm-serving:80

Event-driven, per project architectural preference: informers push to a reducer, the reducer publishes deltas to Redis, SSE handlers subscribe. No request-triggered upstream calls anywhere in the read path.

Snapshot is redacted at write time, not at serialization time. A field that never enters the snapshot cannot leak from any surface.

Language

Go, for client-go informers and because it matches the rest of the platform. Follow the repo's Go skill set (go-naming, go-concurrency, go-error-handling, go-context) — notably: every upstream call carries a context, no naked returns, no _ = on errors.

4. API contract

Envelope for all non-stream responses:

{ "data": { }, "meta": { "snapshotAge": 3.2, "generation": 88412 } }

Errors follow RFC 9457 (application/problem+json):

{ "type": "https://riotpiao.com/errors/rate-limited",
  "title": "Rate limit exceeded",
  "status": 429, "detail": "12 of 12 messages used", "retryAfter": 3600 }

Method	Path	Auth	Limit	Response
GET	`/api/topology`	none	60/min/IP	Nodes, namespaces, workload summaries
GET	`/api/delivery`	none	60/min/IP	Argo apps, wave-grouped; resource children lazy
GET	`/api/delivery/{app}/resources`	none	60/min/IP	Virtualized child list for one app
GET	`/api/stream`	session cookie	2 concurrent/IP	SSE deltas, `topology` + `delivery` event types
POST	`/api/exec`	session cookie	20/min/session	Enum command result
POST	`/api/chat`	session + Turnstile	12/day/session, 6 global concurrent	SSE token stream

Pagination: /api/delivery/{app}/resources is cursor-paginated at 100 items. prometheus has 68 resources today, but homelab-root's tree will grow.

Payload budget: topology response capped at 256 KB, delivery at 256 KB. Exceeding the cap truncates and sets meta.truncated: true — never a silent drop.

5. Security

Mapped against OWASP Top 10 (2021). Every item is a Phase gate, not a wish list.

A01 Broken Access Control

atlas ServiceAccount: one ClusterRole, verbs get,list,watch only, explicit resource list. No secrets. No *. No wildcards on apiGroups.
NetworkPolicy on atlas: egress restricted to kube API, prometheus-operated.monitoring, reasoning-predictor.llm-serving, kmsvc-redis-master.sqs. Ingress from ingress-nginx only.
Test: an integration test asserting the SA receives 403 on get secrets in every namespace.

A02 Cryptographic Failures

Session cookie: signed (HMAC), HttpOnly, Secure, SameSite=Lax, 24h expiry. No PII in the payload — a random session ID only.
Signing key from a Kubernetes Secret via SOPS (sops-secrets app already exists), never an env literal in a manifest.

A03 Injection

The primary risk on surface C. Mitigation is structural, not filtering:

Input parses to a closed command enum. Anything unmatched is rejected before any lookup.
Namespace and resource-name arguments are validated by set membership against the current snapshot, not by regex or escaping.
No shell, no exec, no kubectl binary present in the container image.
Test: fuzz the parser; assert every input outside the allowlist returns a rejection and performs zero upstream calls.

A04 Insecure Design — information disclosure

The core risk of the whole project. Redaction allowlist, enforced by DTO construction:

Emitted: name, namespace, kind, phase, ready counts, restart count, age, node name, health status, sync status, sync wave, an explicit label subset.

Never emitted: container env, container args, image digests, image tags, spec.source.repoURL, full spec.source.path, annotations, pod IPs, cluster IPs, Secret names, status.conditions[].message, node internal IPs.

Specific known leaks in current data:

reasoning container args disclose the entire model and quantization strategy
spec.source.repoURL discloses a private GitHub repository
21 Secret resources appear in Argo trees — render kind and count only, never names; sops-secrets included
status.conditions[].message echoes raw errors containing internal hostnames — emit condition type only

Test: golden test asserting the serialized snapshot contains none of the denied field names, run against a fixture captured from the real cluster.

A05 Security Misconfiguration

Container: runAsNonRoot, read-only root filesystem, all capabilities dropped, seccompProfile: RuntimeDefault
Security headers on all responses: Content-Security-Policy (no unsafe-inline), X-Content-Type-Options: nosniff, Referrer-Policy: no-referrer, Strict-Transport-Security
CORS: same-origin only. No wildcard.

A07 Authentication Failures

Anonymous by design; Cloudflare Turnstile gates the first chat message
Session rotation on issue; no session fixation vector since there is no login

A08 Software and Data Integrity

Image tags are commit SHAs, never :latest (fixes Phase 0.3)
pnpm audit and govulncheck in CI, failing the build on high severity

A09 Logging and Monitoring

Structured logs: every rejected /api/exec input, every rate-limit trip, every chat queue rejection
Prometheus metrics from atlas: atlas_chat_concurrent, atlas_chat_queue_depth, atlas_ratelimit_rejections_total, atlas_snapshot_age_seconds
Alertmanager rule: chat queue saturated > 5 min, snapshot age > 60s

A10 SSRF

atlas calls a fixed, compile-time list of upstream URLs. No user input reaches any URL construction, on any path.

LLM-specific: prompt injection

System prompt is a compile-time constant, unreachable by user input
Cluster snapshot digest is injected in a delimited block explicitly labelled untrusted data
User message is always last
No tool-calling. The model reads a pre-built digest and cannot query anything. This removes the entire agentic attack surface for v1.
Output capped at max_tokens: 1500 — DeepSeek-R1 will otherwise reason for minutes

6. Rate limiting

Sized against the verified hard ceiling: 8 concurrent sequences (--max-num-seqs=4 × 2 replicas), single GPU node.

Tier 1 — Cloudflare edge. WAF, Bot Fight Mode, per-IP rate rules, Turnstile before first chat message. Free, and stops scripted abuse before it reaches your hardware.

Tier 2 — Kong. rate-limiting plugin, policy: redis against kmsvc-redis-master.sqs:6379 so counters are cluster-wide rather than per-pod (Kong runs 2 replicas — a local policy would silently double every limit). Two profiles: generous for topology and delivery, tight for chat.

Tier 3 — atlas, the tier that actually protects the GPU.

Control	Value	Rationale
Global chat semaphore	6	Leaves 2 of 8 sequence slots as operator headroom
Queue depth	20, then reject with `429`	A visible queue beats a wall; an unbounded queue beats nothing
Per-session budget	12 messages / 24h	Enough to explore, not enough to farm
Per-request timeout	120s hard, server-side	Independent of client behaviour
Disconnect handling	cancel upstream immediately	Critical — a walked-away tab holding 1 of 8 slots is a real outage

Queue position is streamed to the client as SSE {"type":"queue","position":N} events, so waiting is legible rather than a hang.

7. Streaming

SSE throughout — unidirectional server→client fits every surface, including token streaming. WebSockets are not justified; nothing flows client→server mid-stream.

Keepalive comment frame every 30s (idle SSE connections die at proxies)
Last-Event-ID supported on /api/stream for resumable topology/delivery deltas; chat is not resumable
req.Context() threaded to the upstream vLLM request so client abort cancels it — this is the mechanism that enforces the Tier 3 disconnect rule
Backpressure: bounded per-client channel; a slow consumer is dropped rather than allowed to grow memory
Chat events: {"type":"reasoning"|"content"|"queue"|"done"|"error"}. --reasoning-parser=deepseek_r1 already separates reasoning_content from content — render thinking in a collapsible block. That block is the demo.

Context budget (16384 total): system prompt ~300, snapshot digest capped at 2000, max_tokens 1500, leaving ~12500 for history. History is truncated oldest-first to fit. The digest is a compact rendering, never raw JSON.

8. Frontend

Topology (B): React Flow, force layout. Node → namespace → workload.
Delivery (E): React Flow, wave columns left→right from the existing sync-wave annotations (0→8). 31 app nodes — an ideal size for a readable DAG. Click an app → side panel with a react-arborist virtualized resource tree, children lazy-loaded. Live sync animation OutOfSync → Syncing → Synced driven by the Application watch. Push a commit during a demo and the wave cascades — that is the moment worth engineering for.
Terminal (C): reuse components/InteractiveTerminal.tsx. Command set: get nodes, get pods <ns>, get apps, top nodes, describe pod <ns> <name>, help.
Chat (D): new component. Collapsible reasoning block, queue position, streaming tokens.

Rendering budget: ~550 resources total across the tree. Lazy expansion plus virtualization is required, not optional. Target: initial delivery view interactive in < 1.5s on a cold load.

Also in scope: delete the fabricated statistics in app/page.tsx — "40% CPU reduction", "99.2% uptime", "Mission-critical", "60% latency cut" — and either wire each card to a real number from /api/topology or remove the claim. Five of the six cards link to routes that do not exist (/infrastructure, /systems, /llm, /kafka, /opensource).

9. Phases, TDD-first

Each phase is RED → GREEN → REFACTOR. Tests named before implementation exists.

Phase 1 — `atlas` core (~4 days)

RED

redact_test.go: golden test — serialized snapshot contains no denied field, against a real-cluster fixture
rbac_test.go: SA receives 403 on get secrets
snapshot_test.go: informer event produces the expected delta

GREEN: ClusterRole, informers, reducer, DTO construction, Redis publish. Verify: kubectl auth can-i get secrets --as=system:serviceaccount:portfolio:atlas → no.

Phase 2 — Surface B (~3 days)

RED

stream_test.go: SSE emits a delta within 5s of a pod state change
topology.test.tsx: graph re-renders on delta without a full reload

Verify: delete a pod, observe the graph update in < 5s without reloading.

Phase 3 — Surface E (~3 days)

RED

delivery_test.go: apps group correctly by sync-wave; Secret names absent from output; repoURL absent from output
delivery.test.tsx: 550-node tree renders under the frame budget with virtualization on

Verify: trigger an Argo sync, observe wave-ordered animation.

Phase 4 — Surface C (~2 days)

RED

exec_parse_test.go: fuzz corpus — every non-allowlisted input rejects and performs zero upstream calls
exec_test.go: unknown namespace rejects on snapshot membership, not regex

Verify: attempt injection payloads against /api/exec; all rejected, all logged.

Phase 5 — Surface D + rate limiter, one PR (~5 days)

RED

ratelimit_test.go: 7th concurrent chat queues rather than reaching vLLM
disconnect_test.go: client abort cancels the upstream request
budget_test.go: 13th message in 24h returns 429 with Retry-After
injection_test.go: snapshot content cannot alter system-prompt behaviour
context_test.go: history truncation keeps total tokens under 16384

Verify: load test at 20 concurrent clients — GPU sequence usage never exceeds 6, no upstream 5xx, queue drains.

Total: ~17 working days. Rate limiter ships in the same PR as chat, never after.

10. Risks

Risk	Impact	Mitigation
Redaction miss leaks internal detail	High	Allowlist by DTO construction; golden test; manual review of every new field
`worker-1` fails	Chat and all inference down	Out of scope to fix; degrade chat gracefully to "offline", never a hang
8-slot ceiling makes chat feel broken under traffic	Medium	Visible queue position; consider disabling chat and shipping B/E/C only
Argo CD CRD schema changes	Low	Allowlist construction means new fields are ignored by default
`atlas` compromised	High	Read-only SA, NetworkPolicy egress restriction, no write verbs anywhere
Cost of GPU inference for anonymous visitors	Medium	Per-session daily budget; Turnstile; edge WAF

11. Decisions needed before Phase 1

Which GitOps repo owns the portfolio — GitHub (riotpiao.com) or Forgejo? Blocks 0.1–0.3.
Apex 403 cause — tunnel route, WAF rule, or no origin? Blocks 0.5.
Is homarr still wanted? Deployed and healthy, but from the abandoned plan.
Does chat stay in v1? Given the 8-slot ceiling, shipping B + E + C first and treating D as a separate decision is defensible.
Where does atlas live — this repo, or the homelab repo? Follows from decision 1.

17 KiB Raw Blame History Unescape Escape

Implementation Plan: atlas — Interactive Cluster Visualization