rock/riotpiao.com
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

(chore) init commit and add tasks

Browse Source
This commit is contained in:
Story Crater Bot
2026-08-18 18:33:49 -07:00
commit 6c6218ef36
61 changed files with 9851 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/adr/ADR-0001-atlas-cluster-visualization.md
+285
View File
@@ -0,0 +1,285 @@
# ADR-0001: `atlas` — Single Read-Only Aggregator for Public Cluster Visualization
**Status**: Proposed
**Date**: 2026-08-13
**Authors**: Rock Liang
**Supersedes**: `PLAN.md`, `IMPLEMENTATION.md` (Homarr + Terraform approach — abandoned; cluster is now GitOps/Kustomize + Argo CD)
---
## Context
### Problem Statement
The homelab cluster is fully operational and runs a non-trivial platform: Talos Kubernetes, GPU-backed LLM inference, event streaming, GitOps delivery, and a full observability stack. None of it is visible to anyone but the operator. The portfolio site meant to showcase it displays **hardcoded, fabricated statistics** and links to pages that do not exist.
Goal: replace fabricated claims with a live, interactive, public view of the real system.
### Current Situation (verified 2026-08-13)
**Cluster**
| Property | Value |
|---|---|
| Distro | Talos v1.13.3, Kubernetes v1.36.1 |
| Nodes | 3× control-plane (`talos-cp-1/2/3`) + 1× `worker-1` (role `gpu-node`) |
| `worker-1` allocatable | 95950m CPU, 65019644Ki memory, 1× `nvidia.com/gpu` |
| Ingress | nginx, LoadBalancer `192.168.1.160`; Kong `10.105.63.160` for `api.riotpiao.com` |
| Namespaces | 25 |
**GitOps**
- `homelab-root` is an app-of-apps: 31 child `Application` resources
- Source: `[email protected]:Riotpiaole/riotpiao.homelab.com.git`, path `k8s/argocd/apps`
- Sync waves **0 → 8** already annotated across apps
- ~550 managed resources total (largest: `prometheus` 68, `cert-manager` 47, `kong` 39)
**LLM serving** (`llm-serving`, KServe, all 5 `InferenceService` Ready)
| Model | Notes |
|---|---|
| `reasoning` | `unsloth/DeepSeek-R1-Distill-Qwen-32B-bnb-4bit`, vLLM `v0.11.0` |
| `ornith`, `embeddings`, `reranker`, `verifier` | 1 replica each |
`reasoning` runtime args, verbatim: `--max-num-seqs=4`, `--max-model-len=16384`, `--gpu-memory-utilization=0.90`, `--enable-prefix-caching`, `--reasoning-parser=deepseek_r1`. `minReplicas: 2`, `maxReplicas: 2`, pinned to `worker-1`.
**Existing building blocks**
- Prometheus (kube-prometheus-stack) + kube-state-metrics + node-exporter + blackbox + Alertmanager — `monitoring`
- Loki + promtail + Grafana — `logging`
- Kong 3.9 + Kubernetes Ingress Controller 3.5 — `api`, 2 replicas
- Redis (`kmsvc-redis-master.sqs.svc.cluster.local:6379`), 1 master + 3 replicas
- Next.js 15.5 / React 19.2 portfolio source in this repo (not deployed)
**Public exposure** (verified by DNS + HTTP probe)
```
api.riotpiao.com NXDOMAIN
argocd|grafana|vault|longhorn|prometheus|minio|temporal|forgejo|portainer.riotpiao.com
NXDOMAIN
riotpiao.com 172.67.196.33 / 104.21.60.115 (Cloudflare) → HTTP 403 at edge
```
Nothing in the cluster is currently reachable from the public internet. The Cloudflare tunnel has no public hostnames wired (cloudflared auto-creates a CNAME per public hostname; no CNAME exists).
### Requirements
1. Public, anonymous, interactive visualization of live cluster state
2. Live Argo CD delivery pipeline view, ordered by sync wave
3. Read-only browser terminal for cluster queries
4. Streaming chat against the `reasoning` model, ChatGPT-style
5. No internal service becomes publicly reachable as a side effect
6. No fabricated statistics anywhere on the site
### Constraints
- **Hard capacity ceiling: 8 concurrent LLM sequences** (`--max-num-seqs=4` × 2 replicas). Single GPU. Not horizontally scalable without more hardware.
- **Context ceiling: 16384 tokens** (`--max-model-len`)
- Single GPU node — `worker-1` is a single point of failure for all inference
- Solo operator, part-time
- Cluster is GitOps-managed; every change ships through git → Argo CD (per project hard rules)
- No test framework currently installed in the portfolio repo
### Forces
- **Impressiveness vs. attack surface** — the most impressive surfaces (terminal, chat) are the most dangerous
- **Live data vs. information disclosure** — real cluster state is the whole point, and real cluster state is exactly what an attacker wants for reconnaissance
- **Anonymous access vs. abuse** — requiring login kills the portfolio demo; not requiring it exposes 8 GPU slots to the open internet
- **Four surfaces vs. one operator** — four independently-built backends is four times the security review
---
## Decision
**We will build one read-only aggregator service, `atlas`, that is the sole public entry point to all cluster data, serving four presentation surfaces from one shared in-memory snapshot.**
### Architecture
```
kube API (informers) ─┐
Prometheus /api/v1 ├──> atlas (Go, ns: portfolio, read-only ServiceAccount)
Argo CD Application CRs┘ │
├─ snapshot: in-memory, redacted at write time
├─ Redis pub/sub (kmsvc-redis) — cross-replica fanout
│
└─ HTTP surfaces
GET /api/topology + /api/stream (B: cluster topology)
GET /api/delivery + /api/stream (E: Argo CD tree)
POST /api/exec (C: terminal)
POST /api/chat (D: chat) ──> reasoning-predictor.llm-serving
```
### Core invariants
These are the load-bearing decisions. Everything else is implementation detail.
**I1 — One public hostname, forever.**
`riotpiao.com` is the only name that ever gets a public DNS record. `argocd`, `grafana`, `vault`, `longhorn`, `prometheus`, `minio`, `temporal`, `forgejo`, `portainer` stay NXDOMAIN permanently. Every new public record is a new thing to defend, and `atlas` proxying makes all of them unnecessary.
**I2 — The browser never talks to an internal API.**
No kube API, no Prometheus, no Argo CD API, no vLLM endpoint is reachable from a browser. `atlas` is the only origin. One choke point for rate limiting, redaction, and audit.
**I3 — Redaction is an allowlist, never a denylist.**
Fields are serialized by explicit construction into DTO structs. A new field appearing in an upstream CRD cannot leak by default, because nothing copies it.
**I4 — No free-form string ever reaches an internal system.**
The terminal parses to a closed command enum. Resource names are validated by **membership in the current snapshot**, not by regex. The chat model reads a pre-built snapshot digest and has no tool-calling ability.
**I5 — Global GPU concurrency is capped below physical capacity.**
Hard semaphore at **6** concurrent chat streams, leaving 2 of 8 sequence slots as operator headroom. Client disconnect cancels the upstream vLLM request immediately.
### Technology
| Component | Choice | Rationale |
|---|---|---|
| Aggregator | Go, `client-go` informers | Watch-based, not poll-per-request; N visitors ≠ N API calls |
| Fanout | Redis pub/sub (existing `kmsvc-redis`) | Multi-replica `atlas` shares one snapshot stream; no new infrastructure |
| Transport | SSE | Unidirectional server→client fits every surface, including LLM token streaming |
| Topology + delivery graph | React Flow | Both are graphs; one library, one mental model |
| Resource drill-down | `react-arborist` | Virtualized; the 68-resource apps must not jank |
| Edge protection | Cloudflare WAF + Turnstile | Free; stops scripted abuse before it costs a packet |
| Gateway limits | Kong `rate-limiting`, `policy: redis` | Cluster-wide counters, not per-pod; Kong 3.9 OSS ships it |
### Implementation strategy — phased, in dependency order
| Phase | Deliverable | Why this position |
|---|---|---|
| 0 | Unblock deployment | Nothing is visible until this lands |
| 1 | `atlas` core: RBAC, informers, redaction | Every surface depends on it |
| 2 | Surface B — cluster topology | Proves the snapshot + SSE pipeline end to end |
| 3 | Surface E — Argo CD delivery tree | Zero new data sources, zero new attack surface, highest signal |
| 4 | Surface C — read-only terminal | First surface accepting user input |
| 5 | Surface D — chat + full rate limiter | Highest risk, highest cost; ships last, ships with its limiter |
Surface E precedes C and D deliberately: it reuses Phase 1 data wholesale and is the surface that reads as platform engineering rather than hobby.
**Timeline**: ~3 weeks part-time. **Responsibility**: solo.
---
## Consequences
### Positive
- Fabricated portfolio statistics replaced by live, verifiable data
- One service to secure, rate-limit, audit, and operate instead of four
- Sync-wave DAG makes a real dependency-ordering decision legible to a visitor in seconds
- Informer-based design means visitor traffic does not load the kube API
- Invariant I1 leaves the cluster's public footprint at exactly one hostname
- Existing `InteractiveTerminal.tsx` and `LiveIndicator.tsx` get real backing
### Negative
- New production service to build, secure, and maintain — currently zero
- `atlas` becomes a high-value target: it holds cluster-wide read access by design
- 8-slot GPU ceiling means chat will queue under real traffic; a "please wait" queue is a worse first impression than no chat at all
- Public chat on a single GPU node has a genuine cost/abuse tail even behind three tiers of limiting
- Redaction is permanent maintenance: every new field surfaced is a new disclosure review
- Test framework must be added to the repo before any of this can be built TDD-first
### Neutral
- Grafana remains for operator use; `atlas` is presentation-only and never replaces it
- Argo CD API is deliberately not used in v1 — `Application` CRs are read via the same informer, so no Argo CD token is ever minted
- `worker-1` remains a single point of failure; this ADR does not change that, only exposes it
---
## Alternatives Considered
### Alternative 1: Grafana dashboards + public iframe embeds
**Description**: Build dashboards on existing data, expose a read-only Grafana org, embed in the portfolio.
**Pros**
- Approximately one day of work
- Zero new services, zero new code, zero new attack surface of our own making
- Data already flows
**Cons**
- Grafana's design, not the portfolio's — reads as a screenshot, not a product
- Requires making Grafana publicly reachable, violating invariant I1
- Iframes leak internal metric names, job labels, and namespace structure with no redaction layer available
- No path to the terminal or chat surfaces
**Why not chosen**: The one-day cost is real, but it forces a second public hostname and surrenders all control over what is disclosed. Grafana has no allowlist redaction model. Kept as the fallback if Phase 1 proves too expensive.
### Alternative 2: Four independent backends, one per surface
**Description**: Separate services for topology, delivery, terminal, and chat.
**Pros**
- Blast radius isolation — a terminal compromise does not reach the chat service
- Independent scaling and deployment
- Aligns with the microservices instinct
**Cons**
- Four RBAC policies, four redaction layers, four rate limiters, four security reviews
- Four independent informer sets hammering the kube API for the same data
- Solo operator; four services will not receive equal maintenance attention
- Cross-surface consistency (terminal and topology disagreeing about pod state) becomes a real bug class
**Why not chosen**: Blast-radius isolation is genuine, but every surface needs the *same* read-only snapshot. Duplicating the highest-risk component — cluster-wide read access — four times increases total exposure rather than reducing it. Rejected on the specific ground that the shared component is the dangerous one.
### Alternative 3: Static snapshot generated at build time
**Description**: CI job dumps cluster state to JSON at build; site renders it statically. No runtime cluster access at all.
**Pros**
- Zero runtime attack surface — no live credentials anywhere near the public internet
- Trivially cacheable, effectively free to serve, cannot be DoS'd
- No rate limiting needed
**Cons**
- Not live; "interactive" degrades to "pre-rendered"
- Kills the terminal and chat surfaces entirely
- The sync-wave cascade animation — the single best demo moment — is impossible
- Data staleness makes the fabricated-statistics problem better but not solved
**Why not chosen**: Fails requirements 1, 3, and 4. Worth revisiting for the topology surface alone if runtime cost becomes a problem.
### Alternative 4: Authentik-gated access to all surfaces
**Description**: Put the existing Authentik SSO in front of the whole visualization.
**Pros**
- Abuse problem largely disappears; rate limiting becomes a formality
- Authentik is already deployed and working
- Redaction requirements relax substantially for authenticated viewers
**Cons**
- Nobody creates an account to look at a stranger's homelab — the demo dies
- Defeats the entire purpose of a public portfolio
- Still requires a public Authentik hostname, violating I1
**Why not chosen**: Directly contradicts requirement 1. Anonymous access plus Cloudflare Turnstile achieves most of the abuse resistance without the conversion cliff. Reconsider only if abuse proves unmanageable in production.
---
## References
- Cluster state verified 2026-08-13 via `kubectl` against context `admin@homelab-cluster`
- Public exposure verified 2026-08-13 via `dig` + `curl` against `riotpiao.com` and subdomains
- vLLM engine args: `kubectl get inferenceservice reasoning -n llm-serving -o jsonpath='{.spec}'`
- Superseded: [PLAN.md](../../PLAN.md), [IMPLEMENTATION.md](../../IMPLEMENTATION.md)
- Companion implementation plan: [PLAN-atlas.md](../PLAN-atlas.md)
---
## Review Notes (Before Acceptance)
**Open questions requiring an answer before Phase 1**
1. **Two GitOps roots exist.** `homelab-root` reads `[email protected]:Riotpiaole/riotpiao.homelab.com.git`. This repo's `infra/argocd-apps.yaml` points at `forgejo.riotpiao.homelab.com` — a hostname that does not resolve — and its `portfolio` and `auth-infra` Applications **do not exist in the cluster**. Which repository is authoritative for the portfolio?
2. **Apex returns 403 from the Cloudflare edge.** Is this an absent tunnel public-hostname route, a WAF rule, or a proxied record with no origin? The portfolio cannot ship until this is understood.
3. Is `homarr` still wanted? It is deployed and healthy, but the plan it came from is abandoned.
4. Does the chat surface stay in scope given the 8-slot ceiling, or ship topology + delivery + terminal first and treat chat as a separate decision?
**Approval**
- [ ] Architecture — invariants I1–I5 accepted
- [ ] Security — redaction allowlist and rate-limit tiers accepted
- [ ] Scope — four surfaces vs. three