# ADR-0001: `atlas` — Single Read-Only Aggregator for Public Cluster Visualization

**Status**: Proposed

**Date**: 2026-08-13

**Authors**: Rock Liang

**Supersedes**: `PLAN.md`, `IMPLEMENTATION.md` (Homarr + Terraform approach — abandoned; cluster is now GitOps/Kustomize + Argo CD)

---

## Context

### Problem Statement

The homelab cluster is fully operational and runs a non-trivial platform: Talos Kubernetes, GPU-backed LLM inference, event streaming, GitOps delivery, and a full observability stack. None of it is visible to anyone but the operator. The portfolio site meant to showcase it displays **hardcoded, fabricated statistics** and links to pages that do not exist.

Goal: replace fabricated claims with a live, interactive, public view of the real system.

### Current Situation (verified 2026-08-13)

**Cluster**

| Property | Value |
|---|---|
| Distro | Talos v1.13.3, Kubernetes v1.36.1 |
| Nodes | 3× control-plane (`talos-cp-1/2/3`) + 1× `worker-1` (role `gpu-node`) |
| `worker-1` allocatable | 95950m CPU, 65019644Ki memory, 1× `nvidia.com/gpu` |
| Ingress | nginx, LoadBalancer `192.168.1.160`; Kong `10.105.63.160` for `api.riotpiao.com` |
| Namespaces | 25 |

**GitOps**

- `homelab-root` is an app-of-apps: 31 child `Application` resources
- Source: `git@github.com:Riotpiaole/riotpiao.com.git`, path `k8s/argocd/apps`
- Sync waves **0 → 8** already annotated across apps
- ~550 managed resources total (largest: `prometheus` 68, `cert-manager` 47, `kong` 39)

**LLM serving** (`llm-serving`, KServe, all 5 `InferenceService` Ready)

| Model | Notes |
|---|---|
| `reasoning` | `unsloth/DeepSeek-R1-Distill-Qwen-32B-bnb-4bit`, vLLM `v0.11.0` |
| `ornith`, `embeddings`, `reranker`, `verifier` | 1 replica each |

`reasoning` runtime args, verbatim: `--max-num-seqs=4`, `--max-model-len=16384`, `--gpu-memory-utilization=0.90`, `--enable-prefix-caching`, `--reasoning-parser=deepseek_r1`. `minReplicas: 2`, `maxReplicas: 2`, pinned to `worker-1`.

**Existing building blocks**

- Prometheus (kube-prometheus-stack) + kube-state-metrics + node-exporter + blackbox + Alertmanager — `monitoring`
- Loki + promtail + Grafana — `logging`
- Kong 3.9 + Kubernetes Ingress Controller 3.5 — `api`, 2 replicas
- Redis (`kmsvc-redis-master.sqs.svc.cluster.local:6379`), 1 master + 3 replicas
- Next.js 15.5 / React 19.2 portfolio source in this repo (not deployed)

**Public exposure** (verified by DNS + HTTP probe)

```
api.riotpiao.com        NXDOMAIN
argocd|grafana|vault|longhorn|prometheus|minio|temporal|forgejo|portainer.riotpiao.com
                        NXDOMAIN
riotpiao.com            172.67.196.33 / 104.21.60.115 (Cloudflare) → HTTP 403 at edge
```

Nothing in the cluster is currently reachable from the public internet. The Cloudflare tunnel has no public hostnames wired (cloudflared auto-creates a CNAME per public hostname; no CNAME exists).

### Requirements

1. Public, anonymous, interactive visualization of live cluster state
2. Live Argo CD delivery pipeline view, ordered by sync wave
3. Read-only browser terminal for cluster queries
4. Streaming chat against the `reasoning` model, ChatGPT-style
5. No internal service becomes publicly reachable as a side effect
6. No fabricated statistics anywhere on the site

### Constraints

- **Hard capacity ceiling: 8 concurrent LLM sequences** (`--max-num-seqs=4` × 2 replicas). Single GPU. Not horizontally scalable without more hardware.
- **Context ceiling: 16384 tokens** (`--max-model-len`)
- Single GPU node — `worker-1` is a single point of failure for all inference
- Solo operator, part-time
- Cluster is GitOps-managed; every change ships through git → Argo CD (per project hard rules)
- No test framework currently installed in the portfolio repo

### Forces

- **Impressiveness vs. attack surface** — the most impressive surfaces (terminal, chat) are the most dangerous
- **Live data vs. information disclosure** — real cluster state is the whole point, and real cluster state is exactly what an attacker wants for reconnaissance
- **Anonymous access vs. abuse** — requiring login kills the portfolio demo; not requiring it exposes 8 GPU slots to the open internet
- **Four surfaces vs. one operator** — four independently-built backends is four times the security review

---

## Decision

**We will build one read-only aggregator service, `atlas`, that is the sole public entry point to all cluster data, serving four presentation surfaces from one shared in-memory snapshot.**

### Architecture

```
kube API (informers)  ─┐
Prometheus /api/v1     ├──> atlas (Go, ns: portfolio, read-only ServiceAccount)
Argo CD Application CRs┘         │
                                 ├─ snapshot: in-memory, redacted at write time
                                 ├─ Redis pub/sub (kmsvc-redis) — cross-replica fanout
                                 │
                                 └─ HTTP surfaces
                                     GET  /api/topology   + /api/stream   (B: cluster topology)
                                     GET  /api/delivery   + /api/stream   (E: Argo CD tree)
                                     POST /api/exec                       (C: terminal)
                                     POST /api/chat                       (D: chat) ──> reasoning-predictor.llm-serving
```

### Core invariants

These are the load-bearing decisions. Everything else is implementation detail.

**I1 — One public hostname, forever.**
`riotpiao.com` is the only name that ever gets a public DNS record. `argocd`, `grafana`, `vault`, `longhorn`, `prometheus`, `minio`, `temporal`, `forgejo`, `portainer` stay NXDOMAIN permanently. Every new public record is a new thing to defend, and `atlas` proxying makes all of them unnecessary.

**I2 — The browser never talks to an internal API.**
No kube API, no Prometheus, no Argo CD API, no vLLM endpoint is reachable from a browser. `atlas` is the only origin. One choke point for rate limiting, redaction, and audit.

**I3 — Redaction is an allowlist, never a denylist.**
Fields are serialized by explicit construction into DTO structs. A new field appearing in an upstream CRD cannot leak by default, because nothing copies it.

**I4 — No free-form string ever reaches an internal system.**
The terminal parses to a closed command enum. Resource names are validated by **membership in the current snapshot**, not by regex. The chat model reads a pre-built snapshot digest and has no tool-calling ability.

**I5 — Global GPU concurrency is capped below physical capacity.**
Hard semaphore at **6** concurrent chat streams, leaving 2 of 8 sequence slots as operator headroom. Client disconnect cancels the upstream vLLM request immediately.

### Technology

| Component | Choice | Rationale |
|---|---|---|
| Aggregator | Go, `client-go` informers | Watch-based, not poll-per-request; N visitors ≠ N API calls |
| Fanout | Redis pub/sub (existing `kmsvc-redis`) | Multi-replica `atlas` shares one snapshot stream; no new infrastructure |
| Transport | SSE | Unidirectional server→client fits every surface, including LLM token streaming |
| Topology + delivery graph | React Flow | Both are graphs; one library, one mental model |
| Resource drill-down | `react-arborist` | Virtualized; the 68-resource apps must not jank |
| Edge protection | Cloudflare WAF + Turnstile | Free; stops scripted abuse before it costs a packet |
| Gateway limits | Kong `rate-limiting`, `policy: redis` | Cluster-wide counters, not per-pod; Kong 3.9 OSS ships it |

### Implementation strategy — phased, in dependency order

| Phase | Deliverable | Why this position |
|---|---|---|
| 0 | Unblock deployment | Nothing is visible until this lands |
| 1 | `atlas` core: RBAC, informers, redaction | Every surface depends on it |
| 2 | Surface B — cluster topology | Proves the snapshot + SSE pipeline end to end |
| 3 | Surface E — Argo CD delivery tree | Zero new data sources, zero new attack surface, highest signal |
| 4 | Surface C — read-only terminal | First surface accepting user input |
| 5 | Surface D — chat + full rate limiter | Highest risk, highest cost; ships last, ships with its limiter |

Surface E precedes C and D deliberately: it reuses Phase 1 data wholesale and is the surface that reads as platform engineering rather than hobby.

**Timeline**: ~3 weeks part-time. **Responsibility**: solo.

---

## Consequences

### Positive

- Fabricated portfolio statistics replaced by live, verifiable data
- One service to secure, rate-limit, audit, and operate instead of four
- Sync-wave DAG makes a real dependency-ordering decision legible to a visitor in seconds
- Informer-based design means visitor traffic does not load the kube API
- Invariant I1 leaves the cluster's public footprint at exactly one hostname
- Existing `InteractiveTerminal.tsx` and `LiveIndicator.tsx` get real backing

### Negative

- New production service to build, secure, and maintain — currently zero
- `atlas` becomes a high-value target: it holds cluster-wide read access by design
- 8-slot GPU ceiling means chat will queue under real traffic; a "please wait" queue is a worse first impression than no chat at all
- Public chat on a single GPU node has a genuine cost/abuse tail even behind three tiers of limiting
- Redaction is permanent maintenance: every new field surfaced is a new disclosure review
- Test framework must be added to the repo before any of this can be built TDD-first

### Neutral

- Grafana remains for operator use; `atlas` is presentation-only and never replaces it
- Argo CD API is deliberately not used in v1 — `Application` CRs are read via the same informer, so no Argo CD token is ever minted
- `worker-1` remains a single point of failure; this ADR does not change that, only exposes it

---

## Alternatives Considered

### Alternative 1: Grafana dashboards + public iframe embeds

**Description**: Build dashboards on existing data, expose a read-only Grafana org, embed in the portfolio.

**Pros**
- Approximately one day of work
- Zero new services, zero new code, zero new attack surface of our own making
- Data already flows

**Cons**
- Grafana's design, not the portfolio's — reads as a screenshot, not a product
- Requires making Grafana publicly reachable, violating invariant I1
- Iframes leak internal metric names, job labels, and namespace structure with no redaction layer available
- No path to the terminal or chat surfaces

**Why not chosen**: The one-day cost is real, but it forces a second public hostname and surrenders all control over what is disclosed. Grafana has no allowlist redaction model. Kept as the fallback if Phase 1 proves too expensive.

### Alternative 2: Four independent backends, one per surface

**Description**: Separate services for topology, delivery, terminal, and chat.

**Pros**
- Blast radius isolation — a terminal compromise does not reach the chat service
- Independent scaling and deployment
- Aligns with the microservices instinct

**Cons**
- Four RBAC policies, four redaction layers, four rate limiters, four security reviews
- Four independent informer sets hammering the kube API for the same data
- Solo operator; four services will not receive equal maintenance attention
- Cross-surface consistency (terminal and topology disagreeing about pod state) becomes a real bug class

**Why not chosen**: Blast-radius isolation is genuine, but every surface needs the *same* read-only snapshot. Duplicating the highest-risk component — cluster-wide read access — four times increases total exposure rather than reducing it. Rejected on the specific ground that the shared component is the dangerous one.

### Alternative 3: Static snapshot generated at build time

**Description**: CI job dumps cluster state to JSON at build; site renders it statically. No runtime cluster access at all.

**Pros**
- Zero runtime attack surface — no live credentials anywhere near the public internet
- Trivially cacheable, effectively free to serve, cannot be DoS'd
- No rate limiting needed

**Cons**
- Not live; "interactive" degrades to "pre-rendered"
- Kills the terminal and chat surfaces entirely
- The sync-wave cascade animation — the single best demo moment — is impossible
- Data staleness makes the fabricated-statistics problem better but not solved

**Why not chosen**: Fails requirements 1, 3, and 4. Worth revisiting for the topology surface alone if runtime cost becomes a problem.

### Alternative 4: Authentik-gated access to all surfaces

**Description**: Put the existing Authentik SSO in front of the whole visualization.

**Pros**
- Abuse problem largely disappears; rate limiting becomes a formality
- Authentik is already deployed and working
- Redaction requirements relax substantially for authenticated viewers

**Cons**
- Nobody creates an account to look at a stranger's homelab — the demo dies
- Defeats the entire purpose of a public portfolio
- Still requires a public Authentik hostname, violating I1

**Why not chosen**: Directly contradicts requirement 1. Anonymous access plus Cloudflare Turnstile achieves most of the abuse resistance without the conversion cliff. Reconsider only if abuse proves unmanageable in production.

---

## References

- Cluster state verified 2026-08-13 via `kubectl` against context `admin@homelab-cluster`
- Public exposure verified 2026-08-13 via `dig` + `curl` against `riotpiao.com` and subdomains
- vLLM engine args: `kubectl get inferenceservice reasoning -n llm-serving -o jsonpath='{.spec}'`
- Superseded: [PLAN.md](../../PLAN.md), [IMPLEMENTATION.md](../../IMPLEMENTATION.md)
- Companion implementation plan: [PLAN-atlas.md](../PLAN-atlas.md)

---

## Review Notes (Before Acceptance)

**Open questions requiring an answer before Phase 1**

1. **Two GitOps roots exist.** `homelab-root` reads `git@github.com:Riotpiaole/riotpiao.com.git`. This repo's `infra/argocd-apps.yaml` points at `forgejo.riotpiao.com` — a hostname that does not resolve — and its `portfolio` and `auth-infra` Applications **do not exist in the cluster**. Which repository is authoritative for the portfolio?
2. **Apex returns 403 from the Cloudflare edge.** Is this an absent tunnel public-hostname route, a WAF rule, or a proxied record with no origin? The portfolio cannot ship until this is understood.
3. Is `homarr` still wanted? It is deployed and healthy, but the plan it came from is abandoned.
4. Does the chat surface stay in scope given the 8-slot ceiling, or ship topology + delivery + terminal first and treat chat as a separate decision?

**Approval**

- [ ] Architecture — invariants I1–I5 accepted
- [ ] Security — redaction allowlist and rate-limit tiers accepted
- [ ] Scope — four surfaces vs. three