286 lines
14 KiB
Markdown
286 lines
14 KiB
Markdown
# ADR-0001: `atlas` — Single Read-Only Aggregator for Public Cluster Visualization
|
||
|
||
**Status**: Proposed
|
||
|
||
**Date**: 2026-08-13
|
||
|
||
**Authors**: Rock Liang
|
||
|
||
**Supersedes**: `PLAN.md`, `IMPLEMENTATION.md` (Homarr + Terraform approach — abandoned; cluster is now GitOps/Kustomize + Argo CD)
|
||
|
||
---
|
||
|
||
## Context
|
||
|
||
### Problem Statement
|
||
|
||
The homelab cluster is fully operational and runs a non-trivial platform: Talos Kubernetes, GPU-backed LLM inference, event streaming, GitOps delivery, and a full observability stack. None of it is visible to anyone but the operator. The portfolio site meant to showcase it displays **hardcoded, fabricated statistics** and links to pages that do not exist.
|
||
|
||
Goal: replace fabricated claims with a live, interactive, public view of the real system.
|
||
|
||
### Current Situation (verified 2026-08-13)
|
||
|
||
**Cluster**
|
||
|
||
| Property | Value |
|
||
|---|---|
|
||
| Distro | Talos v1.13.3, Kubernetes v1.36.1 |
|
||
| Nodes | 3× control-plane (`talos-cp-1/2/3`) + 1× `worker-1` (role `gpu-node`) |
|
||
| `worker-1` allocatable | 95950m CPU, 65019644Ki memory, 1× `nvidia.com/gpu` |
|
||
| Ingress | nginx, LoadBalancer `192.168.1.160`; Kong `10.105.63.160` for `api.riotpiao.com` |
|
||
| Namespaces | 25 |
|
||
|
||
**GitOps**
|
||
|
||
- `homelab-root` is an app-of-apps: 31 child `Application` resources
|
||
- Source: `[email protected]:Riotpiaole/riotpiao.com.git`, path `k8s/argocd/apps`
|
||
- Sync waves **0 → 8** already annotated across apps
|
||
- ~550 managed resources total (largest: `prometheus` 68, `cert-manager` 47, `kong` 39)
|
||
|
||
**LLM serving** (`llm-serving`, KServe, all 5 `InferenceService` Ready)
|
||
|
||
| Model | Notes |
|
||
|---|---|
|
||
| `reasoning` | `unsloth/DeepSeek-R1-Distill-Qwen-32B-bnb-4bit`, vLLM `v0.11.0` |
|
||
| `ornith`, `embeddings`, `reranker`, `verifier` | 1 replica each |
|
||
|
||
`reasoning` runtime args, verbatim: `--max-num-seqs=4`, `--max-model-len=16384`, `--gpu-memory-utilization=0.90`, `--enable-prefix-caching`, `--reasoning-parser=deepseek_r1`. `minReplicas: 2`, `maxReplicas: 2`, pinned to `worker-1`.
|
||
|
||
**Existing building blocks**
|
||
|
||
- Prometheus (kube-prometheus-stack) + kube-state-metrics + node-exporter + blackbox + Alertmanager — `monitoring`
|
||
- Loki + promtail + Grafana — `logging`
|
||
- Kong 3.9 + Kubernetes Ingress Controller 3.5 — `api`, 2 replicas
|
||
- Redis (`kmsvc-redis-master.sqs.svc.cluster.local:6379`), 1 master + 3 replicas
|
||
- Next.js 15.5 / React 19.2 portfolio source in this repo (not deployed)
|
||
|
||
**Public exposure** (verified by DNS + HTTP probe)
|
||
|
||
```
|
||
api.riotpiao.com NXDOMAIN
|
||
argocd|grafana|vault|longhorn|prometheus|minio|temporal|forgejo|portainer.riotpiao.com
|
||
NXDOMAIN
|
||
riotpiao.com 172.67.196.33 / 104.21.60.115 (Cloudflare) → HTTP 403 at edge
|
||
```
|
||
|
||
Nothing in the cluster is currently reachable from the public internet. The Cloudflare tunnel has no public hostnames wired (cloudflared auto-creates a CNAME per public hostname; no CNAME exists).
|
||
|
||
### Requirements
|
||
|
||
1. Public, anonymous, interactive visualization of live cluster state
|
||
2. Live Argo CD delivery pipeline view, ordered by sync wave
|
||
3. Read-only browser terminal for cluster queries
|
||
4. Streaming chat against the `reasoning` model, ChatGPT-style
|
||
5. No internal service becomes publicly reachable as a side effect
|
||
6. No fabricated statistics anywhere on the site
|
||
|
||
### Constraints
|
||
|
||
- **Hard capacity ceiling: 8 concurrent LLM sequences** (`--max-num-seqs=4` × 2 replicas). Single GPU. Not horizontally scalable without more hardware.
|
||
- **Context ceiling: 16384 tokens** (`--max-model-len`)
|
||
- Single GPU node — `worker-1` is a single point of failure for all inference
|
||
- Solo operator, part-time
|
||
- Cluster is GitOps-managed; every change ships through git → Argo CD (per project hard rules)
|
||
- No test framework currently installed in the portfolio repo
|
||
|
||
### Forces
|
||
|
||
- **Impressiveness vs. attack surface** — the most impressive surfaces (terminal, chat) are the most dangerous
|
||
- **Live data vs. information disclosure** — real cluster state is the whole point, and real cluster state is exactly what an attacker wants for reconnaissance
|
||
- **Anonymous access vs. abuse** — requiring login kills the portfolio demo; not requiring it exposes 8 GPU slots to the open internet
|
||
- **Four surfaces vs. one operator** — four independently-built backends is four times the security review
|
||
|
||
---
|
||
|
||
## Decision
|
||
|
||
**We will build one read-only aggregator service, `atlas`, that is the sole public entry point to all cluster data, serving four presentation surfaces from one shared in-memory snapshot.**
|
||
|
||
### Architecture
|
||
|
||
```
|
||
kube API (informers) ─┐
|
||
Prometheus /api/v1 ├──> atlas (Go, ns: portfolio, read-only ServiceAccount)
|
||
Argo CD Application CRs┘ │
|
||
├─ snapshot: in-memory, redacted at write time
|
||
├─ Redis pub/sub (kmsvc-redis) — cross-replica fanout
|
||
│
|
||
└─ HTTP surfaces
|
||
GET /api/topology + /api/stream (B: cluster topology)
|
||
GET /api/delivery + /api/stream (E: Argo CD tree)
|
||
POST /api/exec (C: terminal)
|
||
POST /api/chat (D: chat) ──> reasoning-predictor.llm-serving
|
||
```
|
||
|
||
### Core invariants
|
||
|
||
These are the load-bearing decisions. Everything else is implementation detail.
|
||
|
||
**I1 — One public hostname, forever.**
|
||
`riotpiao.com` is the only name that ever gets a public DNS record. `argocd`, `grafana`, `vault`, `longhorn`, `prometheus`, `minio`, `temporal`, `forgejo`, `portainer` stay NXDOMAIN permanently. Every new public record is a new thing to defend, and `atlas` proxying makes all of them unnecessary.
|
||
|
||
**I2 — The browser never talks to an internal API.**
|
||
No kube API, no Prometheus, no Argo CD API, no vLLM endpoint is reachable from a browser. `atlas` is the only origin. One choke point for rate limiting, redaction, and audit.
|
||
|
||
**I3 — Redaction is an allowlist, never a denylist.**
|
||
Fields are serialized by explicit construction into DTO structs. A new field appearing in an upstream CRD cannot leak by default, because nothing copies it.
|
||
|
||
**I4 — No free-form string ever reaches an internal system.**
|
||
The terminal parses to a closed command enum. Resource names are validated by **membership in the current snapshot**, not by regex. The chat model reads a pre-built snapshot digest and has no tool-calling ability.
|
||
|
||
**I5 — Global GPU concurrency is capped below physical capacity.**
|
||
Hard semaphore at **6** concurrent chat streams, leaving 2 of 8 sequence slots as operator headroom. Client disconnect cancels the upstream vLLM request immediately.
|
||
|
||
### Technology
|
||
|
||
| Component | Choice | Rationale |
|
||
|---|---|---|
|
||
| Aggregator | Go, `client-go` informers | Watch-based, not poll-per-request; N visitors ≠ N API calls |
|
||
| Fanout | Redis pub/sub (existing `kmsvc-redis`) | Multi-replica `atlas` shares one snapshot stream; no new infrastructure |
|
||
| Transport | SSE | Unidirectional server→client fits every surface, including LLM token streaming |
|
||
| Topology + delivery graph | React Flow | Both are graphs; one library, one mental model |
|
||
| Resource drill-down | `react-arborist` | Virtualized; the 68-resource apps must not jank |
|
||
| Edge protection | Cloudflare WAF + Turnstile | Free; stops scripted abuse before it costs a packet |
|
||
| Gateway limits | Kong `rate-limiting`, `policy: redis` | Cluster-wide counters, not per-pod; Kong 3.9 OSS ships it |
|
||
|
||
### Implementation strategy — phased, in dependency order
|
||
|
||
| Phase | Deliverable | Why this position |
|
||
|---|---|---|
|
||
| 0 | Unblock deployment | Nothing is visible until this lands |
|
||
| 1 | `atlas` core: RBAC, informers, redaction | Every surface depends on it |
|
||
| 2 | Surface B — cluster topology | Proves the snapshot + SSE pipeline end to end |
|
||
| 3 | Surface E — Argo CD delivery tree | Zero new data sources, zero new attack surface, highest signal |
|
||
| 4 | Surface C — read-only terminal | First surface accepting user input |
|
||
| 5 | Surface D — chat + full rate limiter | Highest risk, highest cost; ships last, ships with its limiter |
|
||
|
||
Surface E precedes C and D deliberately: it reuses Phase 1 data wholesale and is the surface that reads as platform engineering rather than hobby.
|
||
|
||
**Timeline**: ~3 weeks part-time. **Responsibility**: solo.
|
||
|
||
---
|
||
|
||
## Consequences
|
||
|
||
### Positive
|
||
|
||
- Fabricated portfolio statistics replaced by live, verifiable data
|
||
- One service to secure, rate-limit, audit, and operate instead of four
|
||
- Sync-wave DAG makes a real dependency-ordering decision legible to a visitor in seconds
|
||
- Informer-based design means visitor traffic does not load the kube API
|
||
- Invariant I1 leaves the cluster's public footprint at exactly one hostname
|
||
- Existing `InteractiveTerminal.tsx` and `LiveIndicator.tsx` get real backing
|
||
|
||
### Negative
|
||
|
||
- New production service to build, secure, and maintain — currently zero
|
||
- `atlas` becomes a high-value target: it holds cluster-wide read access by design
|
||
- 8-slot GPU ceiling means chat will queue under real traffic; a "please wait" queue is a worse first impression than no chat at all
|
||
- Public chat on a single GPU node has a genuine cost/abuse tail even behind three tiers of limiting
|
||
- Redaction is permanent maintenance: every new field surfaced is a new disclosure review
|
||
- Test framework must be added to the repo before any of this can be built TDD-first
|
||
|
||
### Neutral
|
||
|
||
- Grafana remains for operator use; `atlas` is presentation-only and never replaces it
|
||
- Argo CD API is deliberately not used in v1 — `Application` CRs are read via the same informer, so no Argo CD token is ever minted
|
||
- `worker-1` remains a single point of failure; this ADR does not change that, only exposes it
|
||
|
||
---
|
||
|
||
## Alternatives Considered
|
||
|
||
### Alternative 1: Grafana dashboards + public iframe embeds
|
||
|
||
**Description**: Build dashboards on existing data, expose a read-only Grafana org, embed in the portfolio.
|
||
|
||
**Pros**
|
||
- Approximately one day of work
|
||
- Zero new services, zero new code, zero new attack surface of our own making
|
||
- Data already flows
|
||
|
||
**Cons**
|
||
- Grafana's design, not the portfolio's — reads as a screenshot, not a product
|
||
- Requires making Grafana publicly reachable, violating invariant I1
|
||
- Iframes leak internal metric names, job labels, and namespace structure with no redaction layer available
|
||
- No path to the terminal or chat surfaces
|
||
|
||
**Why not chosen**: The one-day cost is real, but it forces a second public hostname and surrenders all control over what is disclosed. Grafana has no allowlist redaction model. Kept as the fallback if Phase 1 proves too expensive.
|
||
|
||
### Alternative 2: Four independent backends, one per surface
|
||
|
||
**Description**: Separate services for topology, delivery, terminal, and chat.
|
||
|
||
**Pros**
|
||
- Blast radius isolation — a terminal compromise does not reach the chat service
|
||
- Independent scaling and deployment
|
||
- Aligns with the microservices instinct
|
||
|
||
**Cons**
|
||
- Four RBAC policies, four redaction layers, four rate limiters, four security reviews
|
||
- Four independent informer sets hammering the kube API for the same data
|
||
- Solo operator; four services will not receive equal maintenance attention
|
||
- Cross-surface consistency (terminal and topology disagreeing about pod state) becomes a real bug class
|
||
|
||
**Why not chosen**: Blast-radius isolation is genuine, but every surface needs the *same* read-only snapshot. Duplicating the highest-risk component — cluster-wide read access — four times increases total exposure rather than reducing it. Rejected on the specific ground that the shared component is the dangerous one.
|
||
|
||
### Alternative 3: Static snapshot generated at build time
|
||
|
||
**Description**: CI job dumps cluster state to JSON at build; site renders it statically. No runtime cluster access at all.
|
||
|
||
**Pros**
|
||
- Zero runtime attack surface — no live credentials anywhere near the public internet
|
||
- Trivially cacheable, effectively free to serve, cannot be DoS'd
|
||
- No rate limiting needed
|
||
|
||
**Cons**
|
||
- Not live; "interactive" degrades to "pre-rendered"
|
||
- Kills the terminal and chat surfaces entirely
|
||
- The sync-wave cascade animation — the single best demo moment — is impossible
|
||
- Data staleness makes the fabricated-statistics problem better but not solved
|
||
|
||
**Why not chosen**: Fails requirements 1, 3, and 4. Worth revisiting for the topology surface alone if runtime cost becomes a problem.
|
||
|
||
### Alternative 4: Authentik-gated access to all surfaces
|
||
|
||
**Description**: Put the existing Authentik SSO in front of the whole visualization.
|
||
|
||
**Pros**
|
||
- Abuse problem largely disappears; rate limiting becomes a formality
|
||
- Authentik is already deployed and working
|
||
- Redaction requirements relax substantially for authenticated viewers
|
||
|
||
**Cons**
|
||
- Nobody creates an account to look at a stranger's homelab — the demo dies
|
||
- Defeats the entire purpose of a public portfolio
|
||
- Still requires a public Authentik hostname, violating I1
|
||
|
||
**Why not chosen**: Directly contradicts requirement 1. Anonymous access plus Cloudflare Turnstile achieves most of the abuse resistance without the conversion cliff. Reconsider only if abuse proves unmanageable in production.
|
||
|
||
---
|
||
|
||
## References
|
||
|
||
- Cluster state verified 2026-08-13 via `kubectl` against context `admin@homelab-cluster`
|
||
- Public exposure verified 2026-08-13 via `dig` + `curl` against `riotpiao.com` and subdomains
|
||
- vLLM engine args: `kubectl get inferenceservice reasoning -n llm-serving -o jsonpath='{.spec}'`
|
||
- Superseded: [PLAN.md](../../PLAN.md), [IMPLEMENTATION.md](../../IMPLEMENTATION.md)
|
||
- Companion implementation plan: [PLAN-atlas.md](../PLAN-atlas.md)
|
||
|
||
---
|
||
|
||
## Review Notes (Before Acceptance)
|
||
|
||
**Open questions requiring an answer before Phase 1**
|
||
|
||
1. **Two GitOps roots exist.** `homelab-root` reads `[email protected]:Riotpiaole/riotpiao.com.git`. This repo's `infra/argocd-apps.yaml` points at `forgejo.riotpiao.com` — a hostname that does not resolve — and its `portfolio` and `auth-infra` Applications **do not exist in the cluster**. Which repository is authoritative for the portfolio?
|
||
2. **Apex returns 403 from the Cloudflare edge.** Is this an absent tunnel public-hostname route, a WAF rule, or a proxied record with no origin? The portfolio cannot ship until this is understood.
|
||
3. Is `homarr` still wanted? It is deployed and healthy, but the plan it came from is abandoned.
|
||
4. Does the chat surface stay in scope given the 8-slot ceiling, or ship topology + delivery + terminal first and treat chat as a separate decision?
|
||
|
||
**Approval**
|
||
|
||
- [ ] Architecture — invariants I1–I5 accepted
|
||
- [ ] Security — redaction allowlist and rate-limit tiers accepted
|
||
- [ ] Scope — four surfaces vs. three
|