riotpiao.com/docs/adr/ADR-0001-atlas-cluster-visualization.md

ADR-0001: atlas — Single Read-Only Aggregator for Public Cluster Visualization

Status: Proposed

Date: 2026-08-13

Authors: Rock Liang

Supersedes: PLAN.md, IMPLEMENTATION.md (Homarr + Terraform approach — abandoned; cluster is now GitOps/Kustomize + Argo CD)

---

Context

Problem Statement

The homelab cluster is fully operational and runs a non-trivial platform: Talos Kubernetes, GPU-backed LLM inference, event streaming, GitOps delivery, and a full observability stack. None of it is visible to anyone but the operator. The portfolio site meant to showcase it displays hardcoded, fabricated statistics and links to pages that do not exist.

Goal: replace fabricated claims with a live, interactive, public view of the real system.

Current Situation (verified 2026-08-13)

Cluster

Property	Value
Distro	Talos v1.13.3, Kubernetes v1.36.1
Nodes	3× control-plane (talos-cp-1/2/3) + 1× worker-1 (role gpu-node)
worker-1 allocatable	95950m CPU, 65019644Ki memory, 1× nvidia.com/gpu
Ingress	nginx, LoadBalancer 192.168.1.160; Kong 10.105.63.160 for api.riotpiao.com
Namespaces	25

GitOps

• homelab-root is an app-of-apps: 31 child Application resources
• Source: [email protected]:Riotpiaole/riotpiao.com.git, path k8s/argocd/apps
• Sync waves 0 → 8 already annotated across apps
• ~550 managed resources total (largest: prometheus 68, cert-manager 47, kong 39)

LLM serving (llm-serving, KServe, all 5 InferenceService Ready)

Model	Notes
reasoning	unsloth/DeepSeek-R1-Distill-Qwen-32B-bnb-4bit, vLLM v0.11.0
ornith, embeddings, reranker, verifier	1 replica each

reasoning runtime args, verbatim: --max-num-seqs=4, --max-model-len=16384, --gpu-memory-utilization=0.90, --enable-prefix-caching, --reasoning-parser=deepseek_r1. minReplicas: 2, maxReplicas: 2, pinned to worker-1.

Existing building blocks

• Prometheus (kube-prometheus-stack) + kube-state-metrics + node-exporter + blackbox + Alertmanager — monitoring
• Loki + promtail + Grafana — logging
• Kong 3.9 + Kubernetes Ingress Controller 3.5 — api, 2 replicas
• Redis (kmsvc-redis-master.sqs.svc.cluster.local:6379), 1 master + 3 replicas
• Next.js 15.5 / React 19.2 portfolio source in this repo (not deployed)

Public exposure (verified by DNS + HTTP probe)

api.riotpiao.com        NXDOMAIN
argocd|grafana|vault|longhorn|prometheus|minio|temporal|forgejo|portainer.riotpiao.com
                        NXDOMAIN
riotpiao.com            172.67.196.33 / 104.21.60.115 (Cloudflare) → HTTP 403 at edge

Nothing in the cluster is currently reachable from the public internet. The Cloudflare tunnel has no public hostnames wired (cloudflared auto-creates a CNAME per public hostname; no CNAME exists).

Requirements

1. Public, anonymous, interactive visualization of live cluster state
2. Live Argo CD delivery pipeline view, ordered by sync wave
3. Read-only browser terminal for cluster queries
4. Streaming chat against the reasoning model, ChatGPT-style
5. No internal service becomes publicly reachable as a side effect
6. No fabricated statistics anywhere on the site

Constraints

• Hard capacity ceiling: 8 concurrent LLM sequences (--max-num-seqs=4 × 2 replicas). Single GPU. Not horizontally scalable without more hardware.
• Context ceiling: 16384 tokens (--max-model-len)
• Single GPU node — worker-1 is a single point of failure for all inference
• Solo operator, part-time
• Cluster is GitOps-managed; every change ships through git → Argo CD (per project hard rules)
• No test framework currently installed in the portfolio repo

Forces

• Impressiveness vs. attack surface — the most impressive surfaces (terminal, chat) are the most dangerous
• Live data vs. information disclosure — real cluster state is the whole point, and real cluster state is exactly what an attacker wants for reconnaissance
• Anonymous access vs. abuse — requiring login kills the portfolio demo; not requiring it exposes 8 GPU slots to the open internet
• Four surfaces vs. one operator — four independently-built backends is four times the security review

---

Decision

We will build one read-only aggregator service, atlas, that is the sole public entry point to all cluster data, serving four presentation surfaces from one shared in-memory snapshot.

Architecture

kube API (informers)  ─┐
Prometheus /api/v1     ├──> atlas (Go, ns: portfolio, read-only ServiceAccount)
Argo CD Application CRs┘         │
                                 ├─ snapshot: in-memory, redacted at write time
                                 ├─ Redis pub/sub (kmsvc-redis) — cross-replica fanout
                                 │
                                 └─ HTTP surfaces
                                     GET  /api/topology   + /api/stream   (B: cluster topology)
                                     GET  /api/delivery   + /api/stream   (E: Argo CD tree)
                                     POST /api/exec                       (C: terminal)
                                     POST /api/chat                       (D: chat) ──> reasoning-predictor.llm-serving

Core invariants

These are the load-bearing decisions. Everything else is implementation detail.

I1 — One public hostname, forever. riotpiao.com is the only name that ever gets a public DNS record. argocd, grafana, vault, longhorn, prometheus, minio, temporal, forgejo, portainer stay NXDOMAIN permanently. Every new public record is a new thing to defend, and atlas proxying makes all of them unnecessary.

I2 — The browser never talks to an internal API. No kube API, no Prometheus, no Argo CD API, no vLLM endpoint is reachable from a browser. atlas is the only origin. One choke point for rate limiting, redaction, and audit.

I3 — Redaction is an allowlist, never a denylist. Fields are serialized by explicit construction into DTO structs. A new field appearing in an upstream CRD cannot leak by default, because nothing copies it.

I4 — No free-form string ever reaches an internal system. The terminal parses to a closed command enum. Resource names are validated by membership in the current snapshot, not by regex. The chat model reads a pre-built snapshot digest and has no tool-calling ability.

I5 — Global GPU concurrency is capped below physical capacity. Hard semaphore at 6 concurrent chat streams, leaving 2 of 8 sequence slots as operator headroom. Client disconnect cancels the upstream vLLM request immediately.

Technology

Component	Choice	Rationale
Aggregator	Go, client-go informers	Watch-based, not poll-per-request; N visitors ≠ N API calls
Fanout	Redis pub/sub (existing kmsvc-redis)	Multi-replica atlas shares one snapshot stream; no new infrastructure
Transport	SSE	Unidirectional server→client fits every surface, including LLM token streaming
Topology + delivery graph	React Flow	Both are graphs; one library, one mental model
Resource drill-down	react-arborist	Virtualized; the 68-resource apps must not jank
Edge protection	Cloudflare WAF + Turnstile	Free; stops scripted abuse before it costs a packet
Gateway limits	Kong rate-limiting, policy: redis	Cluster-wide counters, not per-pod; Kong 3.9 OSS ships it

Implementation strategy — phased, in dependency order

Phase	Deliverable	Why this position
0	Unblock deployment	Nothing is visible until this lands
1	atlas core: RBAC, informers, redaction	Every surface depends on it
2	Surface B — cluster topology	Proves the snapshot + SSE pipeline end to end
3	Surface E — Argo CD delivery tree	Zero new data sources, zero new attack surface, highest signal
4	Surface C — read-only terminal	First surface accepting user input
5	Surface D — chat + full rate limiter	Highest risk, highest cost; ships last, ships with its limiter

Surface E precedes C and D deliberately: it reuses Phase 1 data wholesale and is the surface that reads as platform engineering rather than hobby.

Timeline: ~3 weeks part-time. Responsibility: solo.

---

Consequences

Positive

• Fabricated portfolio statistics replaced by live, verifiable data
• One service to secure, rate-limit, audit, and operate instead of four
• Sync-wave DAG makes a real dependency-ordering decision legible to a visitor in seconds
• Informer-based design means visitor traffic does not load the kube API
• Invariant I1 leaves the cluster's public footprint at exactly one hostname
• Existing InteractiveTerminal.tsx and LiveIndicator.tsx get real backing

Negative

• New production service to build, secure, and maintain — currently zero
• atlas becomes a high-value target: it holds cluster-wide read access by design
• 8-slot GPU ceiling means chat will queue under real traffic; a "please wait" queue is a worse first impression than no chat at all
• Public chat on a single GPU node has a genuine cost/abuse tail even behind three tiers of limiting
• Redaction is permanent maintenance: every new field surfaced is a new disclosure review
• Test framework must be added to the repo before any of this can be built TDD-first

Neutral

• Grafana remains for operator use; atlas is presentation-only and never replaces it
• Argo CD API is deliberately not used in v1 — Application CRs are read via the same informer, so no Argo CD token is ever minted
• worker-1 remains a single point of failure; this ADR does not change that, only exposes it

---

Alternatives Considered

Alternative 1: Grafana dashboards + public iframe embeds

Description: Build dashboards on existing data, expose a read-only Grafana org, embed in the portfolio.

Pros

• Approximately one day of work
• Zero new services, zero new code, zero new attack surface of our own making
• Data already flows

Cons

• Grafana's design, not the portfolio's — reads as a screenshot, not a product
• Requires making Grafana publicly reachable, violating invariant I1
• Iframes leak internal metric names, job labels, and namespace structure with no redaction layer available
• No path to the terminal or chat surfaces

Why not chosen: The one-day cost is real, but it forces a second public hostname and surrenders all control over what is disclosed. Grafana has no allowlist redaction model. Kept as the fallback if Phase 1 proves too expensive.

Alternative 2: Four independent backends, one per surface

Description: Separate services for topology, delivery, terminal, and chat.

Pros

• Blast radius isolation — a terminal compromise does not reach the chat service
• Independent scaling and deployment
• Aligns with the microservices instinct

Cons

• Four RBAC policies, four redaction layers, four rate limiters, four security reviews
• Four independent informer sets hammering the kube API for the same data
• Solo operator; four services will not receive equal maintenance attention
• Cross-surface consistency (terminal and topology disagreeing about pod state) becomes a real bug class

Why not chosen: Blast-radius isolation is genuine, but every surface needs the same read-only snapshot. Duplicating the highest-risk component — cluster-wide read access — four times increases total exposure rather than reducing it. Rejected on the specific ground that the shared component is the dangerous one.

Alternative 3: Static snapshot generated at build time

Description: CI job dumps cluster state to JSON at build; site renders it statically. No runtime cluster access at all.

Pros

• Zero runtime attack surface — no live credentials anywhere near the public internet
• Trivially cacheable, effectively free to serve, cannot be DoS'd
• No rate limiting needed

Cons

• Not live; "interactive" degrades to "pre-rendered"
• Kills the terminal and chat surfaces entirely
• The sync-wave cascade animation — the single best demo moment — is impossible
• Data staleness makes the fabricated-statistics problem better but not solved

Why not chosen: Fails requirements 1, 3, and 4. Worth revisiting for the topology surface alone if runtime cost becomes a problem.

Alternative 4: Authentik-gated access to all surfaces

Description: Put the existing Authentik SSO in front of the whole visualization.

Pros

• Abuse problem largely disappears; rate limiting becomes a formality
• Authentik is already deployed and working
• Redaction requirements relax substantially for authenticated viewers

Cons

• Nobody creates an account to look at a stranger's homelab — the demo dies
• Defeats the entire purpose of a public portfolio
• Still requires a public Authentik hostname, violating I1

Why not chosen: Directly contradicts requirement 1. Anonymous access plus Cloudflare Turnstile achieves most of the abuse resistance without the conversion cliff. Reconsider only if abuse proves unmanageable in production.

---

References

• Cluster state verified 2026-08-13 via kubectl against context admin@homelab-cluster
• Public exposure verified 2026-08-13 via dig + curl against riotpiao.com and subdomains
• vLLM engine args: kubectl get inferenceservice reasoning -n llm-serving -o jsonpath='{.spec}'
• Superseded: PLAN.md, IMPLEMENTATION.md
• Companion implementation plan: PLAN-atlas.md

---

Review Notes (Before Acceptance)

Open questions requiring an answer before Phase 1

1. Two GitOps roots exist. homelab-root reads [email protected]:Riotpiaole/riotpiao.com.git. This repo's infra/argocd-apps.yaml points at forgejo.riotpiao.com — a hostname that does not resolve — and its portfolio and auth-infra Applications do not exist in the cluster. Which repository is authoritative for the portfolio?
2. Apex returns 403 from the Cloudflare edge. Is this an absent tunnel public-hostname route, a WAF rule, or a proxied record with no origin? The portfolio cannot ship until this is understood.
3. Is homarr still wanted? It is deployed and healthy, but the plan it came from is abandoned.
4. Does the chat surface stay in scope given the 8-slot ceiling, or ship topology + delivery + terminal first and treat chat as a separate decision?

Approval

• Architecture — invariants I1–I5 accepted
• Security — redaction allowlist and rate-limit tiers accepted
• Scope — four surfaces vs. three