rock/riotpiao.com
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

(chore) init commit and add tasks

Browse Source
This commit is contained in:
Story Crater Bot
2026-08-18 18:33:49 -07:00
commit 6c6218ef36
61 changed files with 9851 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/PLAN-atlas.md
+327
View File
@@ -0,0 +1,327 @@
# Implementation Plan: `atlas` — Interactive Cluster Visualization
**Status**: For review — no code written yet
**Companion ADR**: [ADR-0001](adr/ADR-0001-atlas-cluster-visualization.md)
**Supersedes**: `PLAN.md`, `IMPLEMENTATION.md` (Homarr + Terraform — abandoned)
---
## 1. Scope
### In scope
| ID | Surface | Description |
|---|---|---|
| B | Cluster topology | Live node → namespace → workload graph, health-colored, drill-down |
| E | Delivery tree | Argo CD app-of-apps as a sync-wave-ordered DAG, live sync animation |
| C | Terminal | Read-only, enum-dispatched cluster queries in the browser |
| D | Chat | Streaming ChatGPT-style session against the `reasoning` model |
### Out of scope (v1)
- Any write operation against the cluster
- LLM tool-calling / agentic loops against live infrastructure
- Forgejo CI half of the pipeline view (`forgejo-gitea` is currently stuck `Init:0/3`)
- Log streaming to the browser (Loki content is unredactable in practice)
- Authenticated / operator-only views — Grafana already serves that need
### Non-goals
- Replacing Grafana
- Multi-cluster support
- Historical / time-travel views
---
## 2. Phase 0 — Unblock (blocking; nothing ships until done)
Verified problems, in the order they must be fixed:
| # | Problem | Evidence |
|---|---|---|
| 0.1 | `portfolio` namespace is empty; `portfolio` and `auth-infra` Applications do not exist in the cluster | `kubectl get pods -n portfolio` → no resources; neither name appears in `kubectl get app -n argocd` |
| 0.2 | `infra/argocd-apps.yaml` `repoURL` = `forgejo.riotpiao.homelab.com` — NXDOMAIN | `dig` |
| 0.3 | Deployment image `forgejo.riotpiao.homelab.com/rock/portfolio:latest` — dead host, and `:latest` + `imagePullPolicy: IfNotPresent` means a pushed image will never roll out | `infra/portfolio/base/deployment.yaml` |
| 0.4 | `forgejo-gitea` stuck `Init:0/3` for 3h — no image builds possible | `kubectl get pods -n cicd` |
| 0.5 | `riotpiao.com` returns HTTP 403 from the Cloudflare edge; no origin headers present | `curl -I https://riotpiao.com` |
| 0.6 | No test framework installed — TDD is impossible as the repo stands | `package.json` has no test script or runner |
| 0.7 | `sms` Application Degraded (`macos-bluebubbles` Pending 3h); `longhorn-config` OutOfSync | `kubectl get app -n argocd` |
**Decisions required from you before 0.1–0.3 can be actioned** — see ADR open questions 1 and 2.
**Actions**
1. Resolve which GitOps repo owns the portfolio; delete or correct the losing manifest
2. Point `repoURL` and the image reference at real hostnames
3. Replace `:latest` with a commit-SHA tag; set `imagePullPolicy: IfNotPresent` (correct once tags are immutable)
4. Diagnose `forgejo-gitea` init containers — read the actual init container logs before changing anything
5. Diagnose the apex 403 — check the tunnel's Public Hostnames list and Cloudflare WAF events; the event log names the blocking rule
6. Add Vitest + Testing Library + `msw`; add `test` and `test:watch` scripts
7. Triage 0.7 separately — unrelated to this work, but the delivery tree will render both as red on day one
**Verify**
```bash
kubectl get pods -n portfolio # 2/2 Running
curl -sS -o /dev/null -w '%{http_code}\n' https://riotpiao.com # 200
pnpm test # runner executes, 0 tests, exit 0
```
---
## 3. Architecture
```
kube API (client-go informers) ─┐
Prometheus /api/v1/query ├──> atlas (Go, ns: portfolio, RO ServiceAccount)
Argo CD Application CRs ┘ │
├─ snapshot (in-memory, redacted at write)
├─ Redis pub/sub (kmsvc-redis-master.sqs:6379)
└─ HTTP
GET /api/topology
GET /api/delivery
GET /api/stream (SSE)
POST /api/exec
POST /api/chat (SSE)
└──> reasoning-predictor.llm-serving:80
```
Event-driven, per project architectural preference: informers push to a reducer, the reducer publishes deltas to Redis, SSE handlers subscribe. No request-triggered upstream calls anywhere in the read path.
Snapshot is redacted **at write time**, not at serialization time. A field that never enters the snapshot cannot leak from any surface.
### Language
Go, for `client-go` informers and because it matches the rest of the platform. Follow the repo's Go skill set (`go-naming`, `go-concurrency`, `go-error-handling`, `go-context`) — notably: every upstream call carries a context, no naked returns, no `_ =` on errors.
---
## 4. API contract
Envelope for all non-stream responses:
```json
{ "data": { }, "meta": { "snapshotAge": 3.2, "generation": 88412 } }
```
Errors follow RFC 9457 (`application/problem+json`):
```json
{ "type": "https://riotpiao.com/errors/rate-limited",
"title": "Rate limit exceeded",
"status": 429, "detail": "12 of 12 messages used", "retryAfter": 3600 }
```
| Method | Path | Auth | Limit | Response |
|---|---|---|---|---|
| GET | `/api/topology` | none | 60/min/IP | Nodes, namespaces, workload summaries |
| GET | `/api/delivery` | none | 60/min/IP | Argo apps, wave-grouped; resource children lazy |
| GET | `/api/delivery/{app}/resources` | none | 60/min/IP | Virtualized child list for one app |
| GET | `/api/stream` | session cookie | 2 concurrent/IP | SSE deltas, `topology` + `delivery` event types |
| POST | `/api/exec` | session cookie | 20/min/session | Enum command result |
| POST | `/api/chat` | session + Turnstile | 12/day/session, 6 global concurrent | SSE token stream |
**Pagination**: `/api/delivery/{app}/resources` is cursor-paginated at 100 items. `prometheus` has 68 resources today, but `homelab-root`'s tree will grow.
**Payload budget**: topology response capped at 256 KB, delivery at 256 KB. Exceeding the cap truncates and sets `meta.truncated: true` — never a silent drop.
---
## 5. Security
Mapped against OWASP Top 10 (2021). Every item is a Phase gate, not a wish list.
### A01 Broken Access Control
- `atlas` ServiceAccount: one ClusterRole, verbs `get,list,watch` only, explicit resource list. **No `secrets`. No `*`. No wildcards on apiGroups.**
- NetworkPolicy on `atlas`: egress restricted to kube API, `prometheus-operated.monitoring`, `reasoning-predictor.llm-serving`, `kmsvc-redis-master.sqs`. Ingress from `ingress-nginx` only.
- Test: an integration test asserting the SA receives 403 on `get secrets` in every namespace.
### A02 Cryptographic Failures
- Session cookie: signed (HMAC), `HttpOnly`, `Secure`, `SameSite=Lax`, 24h expiry. No PII in the payload — a random session ID only.
- Signing key from a Kubernetes Secret via SOPS (`sops-secrets` app already exists), never an env literal in a manifest.
### A03 Injection
The primary risk on surface C. Mitigation is structural, not filtering:
- Input parses to a closed command enum. Anything unmatched is rejected before any lookup.
- Namespace and resource-name arguments are validated by **set membership against the current snapshot**, not by regex or escaping.
- No shell, no `exec`, no `kubectl` binary present in the container image.
- Test: fuzz the parser; assert every input outside the allowlist returns a rejection and performs zero upstream calls.
### A04 Insecure Design — information disclosure
The core risk of the whole project. Redaction allowlist, enforced by DTO construction:
**Emitted**: name, namespace, kind, phase, ready counts, restart count, age, node name, health status, sync status, sync wave, an explicit label subset.
**Never emitted**: container env, container args, image digests, image tags, `spec.source.repoURL`, full `spec.source.path`, annotations, pod IPs, cluster IPs, Secret names, `status.conditions[].message`, node internal IPs.
Specific known leaks in current data:
- `reasoning` container args disclose the entire model and quantization strategy
- `spec.source.repoURL` discloses a private GitHub repository
- 21 `Secret` resources appear in Argo trees — render **kind and count only, never names**; `sops-secrets` included
- `status.conditions[].message` echoes raw errors containing internal hostnames — emit condition **type** only
Test: golden test asserting the serialized snapshot contains none of the denied field names, run against a fixture captured from the real cluster.
### A05 Security Misconfiguration
- Container: `runAsNonRoot`, read-only root filesystem, all capabilities dropped, `seccompProfile: RuntimeDefault`
- Security headers on all responses: `Content-Security-Policy` (no `unsafe-inline`), `X-Content-Type-Options: nosniff`, `Referrer-Policy: no-referrer`, `Strict-Transport-Security`
- CORS: same-origin only. No wildcard.
### A07 Authentication Failures
- Anonymous by design; Cloudflare Turnstile gates the first chat message
- Session rotation on issue; no session fixation vector since there is no login
### A08 Software and Data Integrity
- Image tags are commit SHAs, never `:latest` (fixes Phase 0.3)
- `pnpm audit` and `govulncheck` in CI, failing the build on high severity
### A09 Logging and Monitoring
- Structured logs: every rejected `/api/exec` input, every rate-limit trip, every chat queue rejection
- Prometheus metrics from `atlas`: `atlas_chat_concurrent`, `atlas_chat_queue_depth`, `atlas_ratelimit_rejections_total`, `atlas_snapshot_age_seconds`
- Alertmanager rule: chat queue saturated > 5 min, snapshot age > 60s
### A10 SSRF
- `atlas` calls a fixed, compile-time list of upstream URLs. No user input reaches any URL construction, on any path.
### LLM-specific: prompt injection
- System prompt is a compile-time constant, unreachable by user input
- Cluster snapshot digest is injected in a delimited block explicitly labelled untrusted data
- User message is always last
- **No tool-calling.** The model reads a pre-built digest and cannot query anything. This removes the entire agentic attack surface for v1.
- Output capped at `max_tokens: 1500` — DeepSeek-R1 will otherwise reason for minutes
---
## 6. Rate limiting
Sized against the verified hard ceiling: **8 concurrent sequences** (`--max-num-seqs=4` × 2 replicas), single GPU node.
**Tier 1 — Cloudflare edge.** WAF, Bot Fight Mode, per-IP rate rules, Turnstile before first chat message. Free, and stops scripted abuse before it reaches your hardware.
**Tier 2 — Kong.** `rate-limiting` plugin, `policy: redis` against `kmsvc-redis-master.sqs:6379` so counters are cluster-wide rather than per-pod (Kong runs 2 replicas — a local policy would silently double every limit). Two profiles: generous for topology and delivery, tight for chat.
**Tier 3 — `atlas`, the tier that actually protects the GPU.**
| Control | Value | Rationale |
|---|---|---|
| Global chat semaphore | 6 | Leaves 2 of 8 sequence slots as operator headroom |
| Queue depth | 20, then reject with `429` | A visible queue beats a wall; an unbounded queue beats nothing |
| Per-session budget | 12 messages / 24h | Enough to explore, not enough to farm |
| Per-request timeout | 120s hard, server-side | Independent of client behaviour |
| Disconnect handling | cancel upstream immediately | **Critical** — a walked-away tab holding 1 of 8 slots is a real outage |
Queue position is streamed to the client as SSE `{"type":"queue","position":N}` events, so waiting is legible rather than a hang.
---
## 7. Streaming
SSE throughout — unidirectional server→client fits every surface, including token streaming. WebSockets are not justified; nothing flows client→server mid-stream.
- Keepalive comment frame every 30s (idle SSE connections die at proxies)
- `Last-Event-ID` supported on `/api/stream` for resumable topology/delivery deltas; chat is not resumable
- `req.Context()` threaded to the upstream vLLM request so client abort cancels it — this is the mechanism that enforces the Tier 3 disconnect rule
- Backpressure: bounded per-client channel; a slow consumer is dropped rather than allowed to grow memory
- Chat events: `{"type":"reasoning"|"content"|"queue"|"done"|"error"}`. `--reasoning-parser=deepseek_r1` already separates `reasoning_content` from `content` — render thinking in a collapsible block. That block **is** the demo.
**Context budget** (16384 total): system prompt ~300, snapshot digest capped at 2000, `max_tokens` 1500, leaving ~12500 for history. History is truncated oldest-first to fit. The digest is a compact rendering, never raw JSON.
---
## 8. Frontend
- **Topology (B)**: React Flow, force layout. Node → namespace → workload.
- **Delivery (E)**: React Flow, wave columns left→right from the existing `sync-wave` annotations (0→8). 31 app nodes — an ideal size for a readable DAG. Click an app → side panel with a `react-arborist` virtualized resource tree, children lazy-loaded. Live sync animation `OutOfSync → Syncing → Synced` driven by the Application watch. Push a commit during a demo and the wave cascades — that is the moment worth engineering for.
- **Terminal (C)**: reuse [components/InteractiveTerminal.tsx](../components/InteractiveTerminal.tsx). Command set: `get nodes`, `get pods <ns>`, `get apps`, `top nodes`, `describe pod <ns> <name>`, `help`.
- **Chat (D)**: new component. Collapsible reasoning block, queue position, streaming tokens.
Rendering budget: ~550 resources total across the tree. Lazy expansion plus virtualization is required, not optional. Target: initial delivery view interactive in < 1.5s on a cold load.
**Also in scope**: delete the fabricated statistics in [app/page.tsx](../app/page.tsx) — "40% CPU reduction", "99.2% uptime", "Mission-critical", "60% latency cut" — and either wire each card to a real number from `/api/topology` or remove the claim. Five of the six cards link to routes that do not exist (`/infrastructure`, `/systems`, `/llm`, `/kafka`, `/opensource`).
---
## 9. Phases, TDD-first
Each phase is RED → GREEN → REFACTOR. Tests named before implementation exists.
### Phase 1 — `atlas` core (~4 days)
RED
- `redact_test.go`: golden test — serialized snapshot contains no denied field, against a real-cluster fixture
- `rbac_test.go`: SA receives 403 on `get secrets`
- `snapshot_test.go`: informer event produces the expected delta
GREEN: ClusterRole, informers, reducer, DTO construction, Redis publish.
Verify: `kubectl auth can-i get secrets --as=system:serviceaccount:portfolio:atlas` → `no`.
### Phase 2 — Surface B (~3 days)
RED
- `stream_test.go`: SSE emits a delta within 5s of a pod state change
- `topology.test.tsx`: graph re-renders on delta without a full reload
Verify: delete a pod, observe the graph update in < 5s without reloading.
### Phase 3 — Surface E (~3 days)
RED
- `delivery_test.go`: apps group correctly by `sync-wave`; Secret names absent from output; `repoURL` absent from output
- `delivery.test.tsx`: 550-node tree renders under the frame budget with virtualization on
Verify: trigger an Argo sync, observe wave-ordered animation.
### Phase 4 — Surface C (~2 days)
RED
- `exec_parse_test.go`: fuzz corpus — every non-allowlisted input rejects and performs zero upstream calls
- `exec_test.go`: unknown namespace rejects on snapshot membership, not regex
Verify: attempt injection payloads against `/api/exec`; all rejected, all logged.
### Phase 5 — Surface D + rate limiter, one PR (~5 days)
RED
- `ratelimit_test.go`: 7th concurrent chat queues rather than reaching vLLM
- `disconnect_test.go`: client abort cancels the upstream request
- `budget_test.go`: 13th message in 24h returns 429 with `Retry-After`
- `injection_test.go`: snapshot content cannot alter system-prompt behaviour
- `context_test.go`: history truncation keeps total tokens under 16384
Verify: load test at 20 concurrent clients — GPU sequence usage never exceeds 6, no upstream 5xx, queue drains.
**Total: ~17 working days.** Rate limiter ships in the same PR as chat, never after.
---
## 10. Risks
| Risk | Impact | Mitigation |
|---|---|---|
| Redaction miss leaks internal detail | High | Allowlist by DTO construction; golden test; manual review of every new field |
| `worker-1` fails | Chat and all inference down | Out of scope to fix; degrade chat gracefully to "offline", never a hang |
| 8-slot ceiling makes chat feel broken under traffic | Medium | Visible queue position; consider disabling chat and shipping B/E/C only |
| Argo CD CRD schema changes | Low | Allowlist construction means new fields are ignored by default |
| `atlas` compromised | High | Read-only SA, NetworkPolicy egress restriction, no write verbs anywhere |
| Cost of GPU inference for anonymous visitors | Medium | Per-session daily budget; Turnstile; edge WAF |
---
## 11. Decisions needed before Phase 1
1. **Which GitOps repo owns the portfolio** — GitHub (`riotpiao.homelab.com`) or Forgejo? Blocks 0.1–0.3.
2. **Apex 403 cause** — tunnel route, WAF rule, or no origin? Blocks 0.5.
3. **Is `homarr` still wanted?** Deployed and healthy, but from the abandoned plan.
4. **Does chat stay in v1?** Given the 8-slot ceiling, shipping B + E + C first and treating D as a separate decision is defensible.
5. **Where does `atlas` live** — this repo, or the homelab repo? Follows from decision 1.
docs/adr/ADR-0001-atlas-cluster-visualization.md
+285
View File
@@ -0,0 +1,285 @@
# ADR-0001: `atlas` — Single Read-Only Aggregator for Public Cluster Visualization
**Status**: Proposed
**Date**: 2026-08-13
**Authors**: Rock Liang
**Supersedes**: `PLAN.md`, `IMPLEMENTATION.md` (Homarr + Terraform approach — abandoned; cluster is now GitOps/Kustomize + Argo CD)
---
## Context
### Problem Statement
The homelab cluster is fully operational and runs a non-trivial platform: Talos Kubernetes, GPU-backed LLM inference, event streaming, GitOps delivery, and a full observability stack. None of it is visible to anyone but the operator. The portfolio site meant to showcase it displays **hardcoded, fabricated statistics** and links to pages that do not exist.
Goal: replace fabricated claims with a live, interactive, public view of the real system.
### Current Situation (verified 2026-08-13)
**Cluster**
| Property | Value |
|---|---|
| Distro | Talos v1.13.3, Kubernetes v1.36.1 |
| Nodes | 3× control-plane (`talos-cp-1/2/3`) + 1× `worker-1` (role `gpu-node`) |
| `worker-1` allocatable | 95950m CPU, 65019644Ki memory, 1× `nvidia.com/gpu` |
| Ingress | nginx, LoadBalancer `192.168.1.160`; Kong `10.105.63.160` for `api.riotpiao.com` |
| Namespaces | 25 |
**GitOps**
- `homelab-root` is an app-of-apps: 31 child `Application` resources
- Source: `[email protected]:Riotpiaole/riotpiao.homelab.com.git`, path `k8s/argocd/apps`
- Sync waves **0 → 8** already annotated across apps
- ~550 managed resources total (largest: `prometheus` 68, `cert-manager` 47, `kong` 39)
**LLM serving** (`llm-serving`, KServe, all 5 `InferenceService` Ready)
| Model | Notes |
|---|---|
| `reasoning` | `unsloth/DeepSeek-R1-Distill-Qwen-32B-bnb-4bit`, vLLM `v0.11.0` |
| `ornith`, `embeddings`, `reranker`, `verifier` | 1 replica each |
`reasoning` runtime args, verbatim: `--max-num-seqs=4`, `--max-model-len=16384`, `--gpu-memory-utilization=0.90`, `--enable-prefix-caching`, `--reasoning-parser=deepseek_r1`. `minReplicas: 2`, `maxReplicas: 2`, pinned to `worker-1`.
**Existing building blocks**
- Prometheus (kube-prometheus-stack) + kube-state-metrics + node-exporter + blackbox + Alertmanager — `monitoring`
- Loki + promtail + Grafana — `logging`
- Kong 3.9 + Kubernetes Ingress Controller 3.5 — `api`, 2 replicas
- Redis (`kmsvc-redis-master.sqs.svc.cluster.local:6379`), 1 master + 3 replicas
- Next.js 15.5 / React 19.2 portfolio source in this repo (not deployed)
**Public exposure** (verified by DNS + HTTP probe)
```
api.riotpiao.com NXDOMAIN
argocd|grafana|vault|longhorn|prometheus|minio|temporal|forgejo|portainer.riotpiao.com
NXDOMAIN
riotpiao.com 172.67.196.33 / 104.21.60.115 (Cloudflare) → HTTP 403 at edge
```
Nothing in the cluster is currently reachable from the public internet. The Cloudflare tunnel has no public hostnames wired (cloudflared auto-creates a CNAME per public hostname; no CNAME exists).
### Requirements
1. Public, anonymous, interactive visualization of live cluster state
2. Live Argo CD delivery pipeline view, ordered by sync wave
3. Read-only browser terminal for cluster queries
4. Streaming chat against the `reasoning` model, ChatGPT-style
5. No internal service becomes publicly reachable as a side effect
6. No fabricated statistics anywhere on the site
### Constraints
- **Hard capacity ceiling: 8 concurrent LLM sequences** (`--max-num-seqs=4` × 2 replicas). Single GPU. Not horizontally scalable without more hardware.
- **Context ceiling: 16384 tokens** (`--max-model-len`)
- Single GPU node — `worker-1` is a single point of failure for all inference
- Solo operator, part-time
- Cluster is GitOps-managed; every change ships through git → Argo CD (per project hard rules)
- No test framework currently installed in the portfolio repo
### Forces
- **Impressiveness vs. attack surface** — the most impressive surfaces (terminal, chat) are the most dangerous
- **Live data vs. information disclosure** — real cluster state is the whole point, and real cluster state is exactly what an attacker wants for reconnaissance
- **Anonymous access vs. abuse** — requiring login kills the portfolio demo; not requiring it exposes 8 GPU slots to the open internet
- **Four surfaces vs. one operator** — four independently-built backends is four times the security review
---
## Decision
**We will build one read-only aggregator service, `atlas`, that is the sole public entry point to all cluster data, serving four presentation surfaces from one shared in-memory snapshot.**
### Architecture
```
kube API (informers) ─┐
Prometheus /api/v1 ├──> atlas (Go, ns: portfolio, read-only ServiceAccount)
Argo CD Application CRs┘ │
├─ snapshot: in-memory, redacted at write time
├─ Redis pub/sub (kmsvc-redis) — cross-replica fanout
│
└─ HTTP surfaces
GET /api/topology + /api/stream (B: cluster topology)
GET /api/delivery + /api/stream (E: Argo CD tree)
POST /api/exec (C: terminal)
POST /api/chat (D: chat) ──> reasoning-predictor.llm-serving
```
### Core invariants
These are the load-bearing decisions. Everything else is implementation detail.
**I1 — One public hostname, forever.**
`riotpiao.com` is the only name that ever gets a public DNS record. `argocd`, `grafana`, `vault`, `longhorn`, `prometheus`, `minio`, `temporal`, `forgejo`, `portainer` stay NXDOMAIN permanently. Every new public record is a new thing to defend, and `atlas` proxying makes all of them unnecessary.
**I2 — The browser never talks to an internal API.**
No kube API, no Prometheus, no Argo CD API, no vLLM endpoint is reachable from a browser. `atlas` is the only origin. One choke point for rate limiting, redaction, and audit.
**I3 — Redaction is an allowlist, never a denylist.**
Fields are serialized by explicit construction into DTO structs. A new field appearing in an upstream CRD cannot leak by default, because nothing copies it.
**I4 — No free-form string ever reaches an internal system.**
The terminal parses to a closed command enum. Resource names are validated by **membership in the current snapshot**, not by regex. The chat model reads a pre-built snapshot digest and has no tool-calling ability.
**I5 — Global GPU concurrency is capped below physical capacity.**
Hard semaphore at **6** concurrent chat streams, leaving 2 of 8 sequence slots as operator headroom. Client disconnect cancels the upstream vLLM request immediately.
### Technology
| Component | Choice | Rationale |
|---|---|---|
| Aggregator | Go, `client-go` informers | Watch-based, not poll-per-request; N visitors ≠ N API calls |
| Fanout | Redis pub/sub (existing `kmsvc-redis`) | Multi-replica `atlas` shares one snapshot stream; no new infrastructure |
| Transport | SSE | Unidirectional server→client fits every surface, including LLM token streaming |
| Topology + delivery graph | React Flow | Both are graphs; one library, one mental model |
| Resource drill-down | `react-arborist` | Virtualized; the 68-resource apps must not jank |
| Edge protection | Cloudflare WAF + Turnstile | Free; stops scripted abuse before it costs a packet |
| Gateway limits | Kong `rate-limiting`, `policy: redis` | Cluster-wide counters, not per-pod; Kong 3.9 OSS ships it |
### Implementation strategy — phased, in dependency order
| Phase | Deliverable | Why this position |
|---|---|---|
| 0 | Unblock deployment | Nothing is visible until this lands |
| 1 | `atlas` core: RBAC, informers, redaction | Every surface depends on it |
| 2 | Surface B — cluster topology | Proves the snapshot + SSE pipeline end to end |
| 3 | Surface E — Argo CD delivery tree | Zero new data sources, zero new attack surface, highest signal |
| 4 | Surface C — read-only terminal | First surface accepting user input |
| 5 | Surface D — chat + full rate limiter | Highest risk, highest cost; ships last, ships with its limiter |
Surface E precedes C and D deliberately: it reuses Phase 1 data wholesale and is the surface that reads as platform engineering rather than hobby.
**Timeline**: ~3 weeks part-time. **Responsibility**: solo.
---
## Consequences
### Positive
- Fabricated portfolio statistics replaced by live, verifiable data
- One service to secure, rate-limit, audit, and operate instead of four
- Sync-wave DAG makes a real dependency-ordering decision legible to a visitor in seconds
- Informer-based design means visitor traffic does not load the kube API
- Invariant I1 leaves the cluster's public footprint at exactly one hostname
- Existing `InteractiveTerminal.tsx` and `LiveIndicator.tsx` get real backing
### Negative
- New production service to build, secure, and maintain — currently zero
- `atlas` becomes a high-value target: it holds cluster-wide read access by design
- 8-slot GPU ceiling means chat will queue under real traffic; a "please wait" queue is a worse first impression than no chat at all
- Public chat on a single GPU node has a genuine cost/abuse tail even behind three tiers of limiting
- Redaction is permanent maintenance: every new field surfaced is a new disclosure review
- Test framework must be added to the repo before any of this can be built TDD-first
### Neutral
- Grafana remains for operator use; `atlas` is presentation-only and never replaces it
- Argo CD API is deliberately not used in v1 — `Application` CRs are read via the same informer, so no Argo CD token is ever minted
- `worker-1` remains a single point of failure; this ADR does not change that, only exposes it
---
## Alternatives Considered
### Alternative 1: Grafana dashboards + public iframe embeds
**Description**: Build dashboards on existing data, expose a read-only Grafana org, embed in the portfolio.
**Pros**
- Approximately one day of work
- Zero new services, zero new code, zero new attack surface of our own making
- Data already flows
**Cons**
- Grafana's design, not the portfolio's — reads as a screenshot, not a product
- Requires making Grafana publicly reachable, violating invariant I1
- Iframes leak internal metric names, job labels, and namespace structure with no redaction layer available
- No path to the terminal or chat surfaces
**Why not chosen**: The one-day cost is real, but it forces a second public hostname and surrenders all control over what is disclosed. Grafana has no allowlist redaction model. Kept as the fallback if Phase 1 proves too expensive.
### Alternative 2: Four independent backends, one per surface
**Description**: Separate services for topology, delivery, terminal, and chat.
**Pros**
- Blast radius isolation — a terminal compromise does not reach the chat service
- Independent scaling and deployment
- Aligns with the microservices instinct
**Cons**
- Four RBAC policies, four redaction layers, four rate limiters, four security reviews
- Four independent informer sets hammering the kube API for the same data
- Solo operator; four services will not receive equal maintenance attention
- Cross-surface consistency (terminal and topology disagreeing about pod state) becomes a real bug class
**Why not chosen**: Blast-radius isolation is genuine, but every surface needs the *same* read-only snapshot. Duplicating the highest-risk component — cluster-wide read access — four times increases total exposure rather than reducing it. Rejected on the specific ground that the shared component is the dangerous one.
### Alternative 3: Static snapshot generated at build time
**Description**: CI job dumps cluster state to JSON at build; site renders it statically. No runtime cluster access at all.
**Pros**
- Zero runtime attack surface — no live credentials anywhere near the public internet
- Trivially cacheable, effectively free to serve, cannot be DoS'd
- No rate limiting needed
**Cons**
- Not live; "interactive" degrades to "pre-rendered"
- Kills the terminal and chat surfaces entirely
- The sync-wave cascade animation — the single best demo moment — is impossible
- Data staleness makes the fabricated-statistics problem better but not solved
**Why not chosen**: Fails requirements 1, 3, and 4. Worth revisiting for the topology surface alone if runtime cost becomes a problem.
### Alternative 4: Authentik-gated access to all surfaces
**Description**: Put the existing Authentik SSO in front of the whole visualization.
**Pros**
- Abuse problem largely disappears; rate limiting becomes a formality
- Authentik is already deployed and working
- Redaction requirements relax substantially for authenticated viewers
**Cons**
- Nobody creates an account to look at a stranger's homelab — the demo dies
- Defeats the entire purpose of a public portfolio
- Still requires a public Authentik hostname, violating I1
**Why not chosen**: Directly contradicts requirement 1. Anonymous access plus Cloudflare Turnstile achieves most of the abuse resistance without the conversion cliff. Reconsider only if abuse proves unmanageable in production.
---
## References
- Cluster state verified 2026-08-13 via `kubectl` against context `admin@homelab-cluster`
- Public exposure verified 2026-08-13 via `dig` + `curl` against `riotpiao.com` and subdomains
- vLLM engine args: `kubectl get inferenceservice reasoning -n llm-serving -o jsonpath='{.spec}'`
- Superseded: [PLAN.md](../../PLAN.md), [IMPLEMENTATION.md](../../IMPLEMENTATION.md)
- Companion implementation plan: [PLAN-atlas.md](../PLAN-atlas.md)
---
## Review Notes (Before Acceptance)
**Open questions requiring an answer before Phase 1**
1. **Two GitOps roots exist.** `homelab-root` reads `[email protected]:Riotpiaole/riotpiao.homelab.com.git`. This repo's `infra/argocd-apps.yaml` points at `forgejo.riotpiao.homelab.com` — a hostname that does not resolve — and its `portfolio` and `auth-infra` Applications **do not exist in the cluster**. Which repository is authoritative for the portfolio?
2. **Apex returns 403 from the Cloudflare edge.** Is this an absent tunnel public-hostname route, a WAF rule, or a proxied record with no origin? The portfolio cannot ship until this is understood.
3. Is `homarr` still wanted? It is deployed and healthy, but the plan it came from is abandoned.
4. Does the chat surface stay in scope given the 8-slot ceiling, or ship topology + delivery + terminal first and treat chat as a separate decision?
**Approval**
- [ ] Architecture — invariants I1–I5 accepted
- [ ] Security — redaction allowlist and rate-limit tiers accepted
- [ ] Scope — four surfaces vs. three