175 lines
12 KiB
Markdown
175 lines
12 KiB
Markdown
# Backend / Cluster Requirements — atlas
|
|
|
|
This repo (`riotpiao`) is now scoped to the Next.js portfolio frontend only. Everything below was cut from
|
|
`tasks/INDEX.md` because it requires `kubectl`, a real Kubernetes cluster, a Go backend service, or
|
|
infra-side config (Cloudflare, GitOps, RBAC) that doesn't belong in a frontend-only repo.
|
|
|
|
The frontend currently runs against mock API routes under `app/api/*` (see `tasks/INDEX.md` phases 2-5)
|
|
that serve static fixture data from `lib/clusterMock.ts`. This document is the contract those mocks need
|
|
to be replaced with — hand it to whoever builds the real `atlas` backend service.
|
|
|
|
Source of truth this was derived from: `docs/PLAN-atlas.md`, `docs/adr/ADR-0001-atlas-cluster-visualization.md`.
|
|
|
|
## Rules carried from the ADR (still apply to the real backend)
|
|
|
|
- I1: `riotpiao.com` is the only public hostname, ever
|
|
- I2: browser never talks to an internal API directly — atlas is the only origin
|
|
- I3: redaction is allowlist-only, enforced by DTO construction
|
|
- I4: no free-form string reaches an internal system (closed enum, snapshot-membership validation)
|
|
- I5: GPU chat concurrency capped at 6 of 8 sequence slots, disconnect cancels upstream immediately
|
|
|
|
---
|
|
|
|
## 1. Blocking decisions (infra/ops, not code)
|
|
|
|
These were tracked in the old `tasks/00-decisions.md`, now retired — folded in here.
|
|
|
|
### 1.1 Which GitOps repo owns the portfolio?
|
|
|
|
Two roots exist:
|
|
- `homelab-root` -> `[email protected]:Riotpiaole/riotpiao.homelab.com.git`, path `k8s/argocd/apps` (33 child Applications as of 2026-08-18, confirmed live via `kubectl get applications -n argocd`)
|
|
- This repo's `infra/argocd-apps.yaml` -> defines `portfolio`, `homarr`, `auth-infra` Applications, none of which exist in-cluster except `homarr` — and `homarr`'s live source is 100% `homelab-root`, not this repo
|
|
|
|
**Evidence as of 2026-08-18:** `homarr` — also defined in this repo's manifest — is deployed, but `kubectl get application homarr -n argocd -o yaml` shows `project: homelab`, sourced entirely from `homelab-root` (Helm chart + values from `github.com/Riotpiaole/riotpiao.homelab.com`). `portfolio` and `auth-infra` Applications from this repo's `infra/argocd-apps.yaml` do not exist in-cluster at all. Points toward `homelab-root` as authoritative — needs explicit sign-off, not inferred.
|
|
|
|
**Decision needed:** GitHub (`homelab-root`) or Forgejo (this repo's `infra/`).
|
|
|
|
### 1.2 Domain migration: `*.riotpiao.homelab.com` -> `*.riotpiao.com`
|
|
|
|
**Confirmed 2026-08-18:** `*.riotpiao.homelab.com` is fully retired — every subdomain NXDOMAINs (`argocd.`, `forgejo.`, etc.), not just the one this repo's manifest references. Live ingress hosts are all `*.riotpiao.com` (22 hosts: `argocd.`, `forgejo.` (200 OK), `grafana.`, `vault.`, `homarr.`, `portainer.`, `authentik.`, `loki.`, `longhorn.`, `prometheus.`, `kmsvc.`, `minio.`, `minio-api.`, `temporal.`, `api.`).
|
|
|
|
**Action:** `infra/argocd-apps.yaml`'s `repoURL: https://forgejo.riotpiao.homelab.com/rock/riotpiao.git` needs to become `https://forgejo.riotpiao.com/rock/riotpiao.git` on all three Application definitions (`portfolio`, `homarr`, `auth-infra`). This is a known, mechanical fix, not open diagnosis — blocked only by decision 1.1 (no point fixing a manifest that may get deleted).
|
|
|
|
**Also needed:** `infra/portfolio/base/deployment.yaml` still references `image: forgejo.riotpiao.homelab.com/rock/portfolio:latest` — same domain swap, plus see 1.4 below re: the `:latest` tag itself.
|
|
|
|
### 1.3 Apex 403
|
|
|
|
`riotpiao.com` (bare apex, no subdomain) returns HTTP 403 at the Cloudflare edge, no origin headers.
|
|
|
|
**Evidence as of 2026-08-18:** `kubectl get ingress -A` has zero ingress rules with host exactly `riotpiao.com` — every rule is a subdomain. No in-cluster origin exists for the apex today. Leans toward "no origin configured" as the cause, but the Cloudflare tunnel Public Hostnames list and WAF event log (not visible via `kubectl`) still need to be checked to confirm whether the tunnel even has an apex entry, or if a WAF rule is actively blocking one that exists.
|
|
|
|
**Decision needed:** tunnel route missing, WAF rule, or no origin configured at all.
|
|
|
|
### 1.4 Image tags
|
|
|
|
`infra/portfolio/base/deployment.yaml` still uses `image: ...:latest` with `imagePullPolicy: IfNotPresent`. Needs a commit-SHA tag per deploy so Argo's `selfHeal` actually rolls out new commits automatically instead of silently reusing a stale cached image.
|
|
|
|
### 1.5 Is homarr still wanted?
|
|
|
|
Deployed and healthy (confirmed 2026-08-18, `Synced`/`Healthy`, live via `homelab-root`), but originally shipped under an abandoned Homarr+Terraform plan. Product decision, not a technical one: keep, or decommission via GitOps (remove Application, let Argo prune).
|
|
|
|
### 1.6 Where does atlas (the real backend, section 2 below) live?
|
|
|
|
Follows from 1.1 — whichever repo is authoritative for the portfolio's GitOps should also own atlas's manifests. Given this repo is now frontend-only, atlas almost certainly does **not** belong here regardless of the 1.1 answer — it needs its own service repo or a path under `homelab-root`.
|
|
|
|
### 1.7 Known-red cluster state (informational, not blocking)
|
|
|
|
As of 2026-08-18: `sms` Application is `Synced`/`Degraded` (`macos-bluebubbles` pod Pending). `longhorn-config` was `OutOfSync`, now confirmed `Synced`/`Healthy` — already resolved, no action needed. `forgejo-gitea` was stuck `Init:0/3`, now confirmed `Running` 1/1 — already resolved, no action needed.
|
|
|
|
---
|
|
|
|
## 2. atlas backend service (Go) — does not exist yet
|
|
|
|
Everything in this section was Phase 1 of the old task board (`atlas core`). None of it is Next.js code; it's a
|
|
separate backend service that the mock `/api/*` routes in this repo currently stand in for.
|
|
|
|
### 2.1 RBAC
|
|
|
|
- ClusterRole: verbs `get,list,watch` only, explicit resource list — no `secrets`, no `*`, no wildcard apiGroups
|
|
- atlas ServiceAccount must get 403 on `get secrets` in every namespace (golden test: `rbac_test.go`)
|
|
|
|
### 2.2 Informers + snapshot
|
|
|
|
- client-go informers: kube API (nodes, namespaces, workloads), Argo CD `Application` CRs
|
|
- Reducer: informer events -> in-memory snapshot, **redacted at write time**
|
|
- Golden test (`snapshot_test.go`): an informer event produces the expected delta
|
|
|
|
### 2.3 Redaction — DTO construction, allowlist only
|
|
|
|
This is the load-bearing security property of the whole design (ADR I3). Golden test (`redact_test.go`): serialized snapshot contains none of the denied fields, run against a fixture captured from the real cluster.
|
|
|
|
**Emitted:** name, namespace, kind, phase, ready counts, restart count, age, node name, health status, sync status, sync wave, explicit label subset.
|
|
|
|
**Never emitted:** container env/args, image digests/tags, `spec.source.repoURL`, `spec.source.path`, annotations, pod IPs, cluster IPs, Secret names, `status.conditions[].message`, node internal IPs.
|
|
|
|
**Delivery-surface-specific redaction:** 21 `Secret` resources appear in Argo trees today — render kind+count only, never names (includes `sops-secrets`). `status.conditions[].message` echoes raw errors with internal hostnames — emit condition **type** only.
|
|
|
|
### 2.4 Redis publish
|
|
|
|
Snapshot deltas published to `kmsvc-redis-master.sqs:6379` (confirmed live in-cluster, `sqs` namespace).
|
|
|
|
### 2.5 NetworkPolicy + container hardening
|
|
|
|
- NetworkPolicy on atlas: egress restricted to kube API, `prometheus-operated.monitoring`, `reasoning-predictor.llm-serving`, `kmsvc-redis-master.sqs` (all three confirmed live in-cluster 2026-08-18); ingress from `ingress-nginx` only
|
|
- Container hardening: `runAsNonRoot`, read-only root filesystem, all capabilities dropped, `seccompProfile: RuntimeDefault`
|
|
- No shell, no `exec`, no `kubectl` binary in the container image (matters most for the exec surface, section 3.3)
|
|
|
|
### 2.6 Go code quality
|
|
|
|
No `_ =` on errors, no naked returns, every upstream call carries a `context.Context` (go-error-handling, go-context skills). Run `simplify` skill pass on reducer/DTO code before merge.
|
|
|
|
---
|
|
|
|
## 3. Real API endpoints the mocks need to be replaced with
|
|
|
|
Mock versions of all five live in this repo under `app/api/*` today, backed by `lib/clusterMock.ts`. Envelope
|
|
shape (`{"data": {}, "meta": {...}}`) is already matched; everything below is what's missing to make them real.
|
|
|
|
### 3.1 `GET /api/topology`
|
|
|
|
Nodes, namespaces, workload summaries. Envelope capped 256 KB, `meta.truncated: true` on overflow, never a silent drop. Depends on section 2.3 (DTO allowlist).
|
|
|
|
### 3.2 `GET /api/stream`
|
|
|
|
SSE, session cookie, 2 concurrent/IP, `topology` and `delivery` event types; keepalive comment frame every 30s (mock already does this); `Last-Event-ID` supported for resumable deltas (mock does not). Depends on section 2.4 (Redis publish) as the delta source.
|
|
|
|
### 3.3 `GET /api/delivery` + `GET /api/delivery/{app}/resources`
|
|
|
|
Argo apps, wave-grouped (0->8), resource children lazy, capped 256 KB. Resources endpoint cursor-paginated at 100 items (prometheus alone has 68 resources today, still under the cap, but the real cluster has apps with more). Live sync animation (`OutOfSync -> Syncing -> Synced`) driven by a real Argo `Application` watch, not a timer.
|
|
|
|
### 3.4 `POST /api/exec`
|
|
|
|
Session cookie, 20/min/session; input parses to a closed command enum, anything unmatched rejected before any lookup — same six commands the mock already implements (`get nodes`, `get pods <ns>`, `get apps`, `top nodes`, `describe pod <ns> <name>`, `help`). Namespace/resource-name arguments validated by set membership against the **live snapshot** (mock validates against the static fixture instead). Structured logging of every rejected input, shipped to a real log pipeline (mock just does `console.warn`).
|
|
|
|
### 3.5 `POST /api/chat`
|
|
|
|
Session + Cloudflare Turnstile, 12 msg/day/session, 6 global concurrent (2 of 8 GPU sequence slots kept as operator headroom — confirmed live: `reasoning-predictor` deployment runs `--max-num-seqs=4` x 2 replicas = 8 total slots), SSE token stream. Mock already matches the event shape (`queue`/`reasoning`/`content`/`done`) but replays a canned reply instead of calling vLLM.
|
|
|
|
**Rate limiting (Tier 3, atlas-side):** global chat semaphore = 6; queue depth 20 then reject `429`; per-session budget 12 msg/24h; per-request timeout 120s hard server-side; disconnect cancels upstream immediately (`req.Context()` threaded to the vLLM request).
|
|
|
|
**Prompt injection defenses:** system prompt is compile-time constant, unreachable by user input; cluster snapshot digest injected in a delimited block explicitly labelled untrusted data; user message always last; **no tool-calling** — model reads a pre-built digest, cannot query anything; `max_tokens: 1500` cap.
|
|
|
|
**Context budget (16384 total):** system prompt ~300, snapshot digest capped at 2000, `max_tokens` 1500, ~12500 for history, truncated oldest-first.
|
|
|
|
---
|
|
|
|
## 4. Rate limiting — Tiers 1 and 2 (infra, not atlas code)
|
|
|
|
- Tier 1: Cloudflare edge — WAF, Bot Fight Mode, per-IP rules
|
|
- Tier 2: Kong `rate-limiting` plugin, `policy: redis`, generous profile for the topology/delivery read surfaces
|
|
|
|
(Tier 3 is atlas-side, covered in section 3.5.)
|
|
|
|
---
|
|
|
|
## 5. Observability
|
|
|
|
- Prometheus metrics: `atlas_chat_concurrent`, `atlas_chat_queue_depth`, `atlas_ratelimit_rejections_total`, `atlas_snapshot_age_seconds`
|
|
- Alertmanager rule: chat queue saturated > 5 min, snapshot age > 60s
|
|
- `govulncheck` in CI for the Go service, fail build on high severity (this repo's CI only needs `pnpm audit` — no Go code here)
|
|
|
|
---
|
|
|
|
## 6. Security headers + CORS
|
|
|
|
CSP (no `unsafe-inline`), `X-Content-Type-Options: nosniff`, `Referrer-Policy: no-referrer`, HSTS, CORS same-origin only. Frontend-implementable via Next.js config/middleware (kept in `tasks/INDEX.md` phase 2), but the real atlas backend must set the same headers on its own responses if it's ever reachable directly.
|
|
|
|
---
|
|
|
|
## 7. Done criteria for a real atlas v1 (from the old success criteria list)
|
|
|
|
- [ ] Protected services redirect unauthenticated to Authentik (if still applicable post decision 1.5)
|
|
- [ ] `riotpiao.com` is the only public hostname (I1 held)
|
|
- [ ] Golden redaction test passes against a live-cluster fixture
|
|
- [ ] Load test: 20 concurrent clients, GPU usage <= 6, no 5xx, queue drains
|