etcd's backend file only grows: the apiserver compacts every 5m, but compaction
marks superseded revisions free inside the file, and only defrag returns that
space to the filesystem. Nothing ran defrag, so on 2026-10-07 the file reached
2,148,843,520 B, crossed the 2 GiB quota on talos-cp-1 and talos-cp-3, and etcd
raised NOSPACE. From then on etcd rejected every write cluster-wide: kubelets
could not post node status (3 nodes NotReady), apiserver writes returned 500 and
the tunnel API path returned 502. Live data was 15,943 keys, 69 MB after
compaction - the rest was dead pages. Compaction had also stalled while the
alarm was up, so dbSizeInUse read 100% of the file; defrag alone would have
reclaimed nothing. Compact must come first.
An earlier revision of this CronJob never ran, for two independent reasons, both
fixed here:
- its image quay.io/coreos/etcd ships no /bin/sh, so command: [/bin/sh, -c]
failed with exec: "/bin/sh": no such file or directory
- it existed only on a side branch while the ArgoCD application
cluster-maintenance tracks main, so the cluster never had it
Contents:
- CronJob etcd-defrag (kube-system): daily at 03:00, acts only when the UTC
epoch day is a multiple of 15. That is an exact 15-day interval; */15 in the
day-of-month field fires on the 1st, 16th and 31st, putting two runs one day
apart.
- Image bitnamilegacy/etcd:3.6.4 pinned by digest, ships bash and etcdctl 3.6.4
against the 3.6.11 servers.
- Compaction uses the MINIMUM revision across members, so a lagging member is
never compacted past its own revision. Measured 2026-10-07: talos-cp-2 sat
~1.2M entries behind the leader with a frozen raft term and needed a reboot
to rejoin.
- Defrag skips any member whose applied index lags the leader by 100k+, and
runs one member at a time since defrag pauses that member's apply loop.
- Secret etcd-defrag-creds (SOPS/ksops, kube-system) carries the etcd CA plus a
client certificate signed by the cluster's etcd CA, valid to 2036-10.
Exit code 143 proved the previous job killed itself: the script's own cmdline contains the
string 'kube-apiserver', so it matched its own process. Matching comm (== 'kube-apiserver')
and skipping $$ removes that failure mode. Staggered 2 minutes apart.
Static pods survive mirror-pod deletion, so the first bootstrap job had no effect.
These jobs share the host PID namespace and SIGTERM the kube-apiserver process itself,
staggered by 100s so only one control plane restarts at a time.
The first job could not pull its image; this one deletes the apiserver pods via the
in-cluster API with the mounted service-account token instead of a kubectl image.
Adds the pi coding-agent sandbox's cluster side:
- `k8s/infra/poimen/pi-agent-workspace-pv.yaml` — static hostPath PV + PVC for the sandbox /workspace (Longhorn cannot attach new volumes in this cluster; this gives real persistence and survived a pod delete/recreate test: marker file + 38 KB session.jsonl intact, the session answered from its own history afterwards).
- `k8s/argocd/secrets/pi-sandbox-secrets.enc.yaml` — the sandbox's API-key Secret, SOPS-encrypted (recipient from .sops.yaml, payload only: `encrypted_regex ^(data|stringData)$`), registered in the ksops generator. Round-trip verified against the live Secret.
- `terraform/files/coredns/Corefile` — `rewrite name pi.riotpiao.com ingress-nginx-controller...` so the hostname resolves in-cluster.
Note: the keys currently in that Secret are the ones leaked earlier and must be rotated; re-encrypt with `sops -e -i` after rotating.Reviewed-on: #54
Co-authored-by: rock <[email protected]>
## Why
`minio/minio:RELEASE.2025-07-23T15-54-02Z` no longer exists. The Docker Hub repository
`minio/minio` returns **404 for every tag** (checked: the pinned tag and `latest`), and
MinIO has moved its images behind registry authentication (`quay.io/minio/minio` → 401,
`registry.min.io/minio/minio` → auth required, `dl.min.io` binaries → 410 Gone).
The tenant therefore sat in `ImagePullBackOff` — 44,229 attempts over 6d23h — and the chain
from there is:
```
minio tenant pod not ready (1/2)
-> headless Service minio-cluster-hl has no endpoints
-> CoreDNS: lookup minio-cluster-hl.storage.svc.cluster.local -> no such host
-> Vault S3 storage init fails at startup
-> vault-0 CrashLoopBackOff, 1943 restarts, ~7 days
```
## Control checks (so this is not a network story)
| check | result |
| --- | --- |
| `alpine:3.20` pulled on talos-cp-1 | Succeeded — docker.io works |
| `quay.io/prometheus/node-exporter:v1.8.2` on the same node | Succeeded — public quay works |
| `minio/minio` (docker.io) | HTTP 404, repo removed |
| old tag `RELEASE.2024-06-13T20-48-48Z` with `IfNotPresent` | not cached on the node |
## Change
Pin `spec.image` to `coollabsio/minio:RELEASE.2025-10-15T17-29-55Z` — a public mirror, and a
newer upstream release than the previous pin. Verified by pulling it on talos-cp-1 and
running it: the image carries `/usr/bin/docker-entrypoint.sh` (the layout the operator
expects, which runs `minio server … --console-address :9090`) and reports
`minio version RELEASE.2025-10-15T17-29-55Z`.
## Caveat
This is a community mirror, not MinIO's official image, because the official ones now
require credentials. If MinIO registry credentials are available, add an imagePullSecret
and switch back to `quay.io/minio/minio`. Supply-chain decision, flagged deliberately.
## After merge (ArgoCD, not kubectl)
The storage app tracks `k8s/infra/minio`, so the sync rolls the tenant; then:
```
kubectl get pods -n storage # minio-cluster-az-a-0 -> 2/2
kubectl get pods -n iam | grep vault # vault-0 -> Running (storage init succeeds)
```
Residual risk, unrelated to this fix: the tenant's 4 x 25Gi volumes are Longhorn-backed, and
Longhorn currently fails to create engines for new volume attaches cluster-wide. If the
restarted pod cannot attach, that is a second, separate fault.Reviewed-on: #53
Co-authored-by: rock <[email protected]>
The reasoning engine starves every other request while one long prompt
prefills. Measured: 262 tok/s contended prefill, ~250s for a 65k prompt,
with Avg generation throughput 0.0-0.1 tok/s at Running: 2-4 reqs.
Root cause is NOT what the issue first diagnosed. The issue proposed
--max-num-partial-prefills / --max-long-partial-prefills, which are V0-only
knobs. Verified in the live container: vLLM 0.11.0 ships V1 only (vllm/core/
is absent, so there is no V0 to fall back to), and EngineArgs.
_is_v1_supported_oracle returns False whenever either flag differs from its
default, calling _raise_or_fallback("Concurrent Partial Prefill") ->
"Falling back to V0" -> AssertionError. With strategy Recreate that is an
outage, not a slow rollout. Both flags are therefore NOT set, and a comment
records why so they are not reintroduced.
V1 needs no such flag: it has no prefill/decode phase at all. Every step it
walks self.running and hands out tokens from one shared budget
(vllm/v1/core/sched/scheduler.py). The real defect is that the per-request
cap is disabled by default (long_prefill_token_threshold=0), so one long
prompt takes the WHOLE budget, token_budget hits 0, the `while ... and
token_budget > 0` loop exits, and every other request is skipped.
The cap is applied before the budget clamp, so threshold >= budget never
binds. Replaying the loop: 8192/8192 serves 1 request per step (still
starved); 8192/2048 serves 4, and two long prompts advance together, which
is the issue's actual acceptance criterion.
--max-num-batched-tokens=8192 (was unset -> default 2048)
--long-prefill-token-threshold=2048 (was unset -> 0, cap disabled)
--max-num-seqs=5, --gpu-memory-utilization=0.95
The ~166k KV figure is marked as a projection; the live reading is
num_gpu_blocks=9527 x block_size=16 = 152,432 tokens and is authoritative
post-boot. A preemption alert covers the accept-and-watch risk.
Observability for the companion admission gate (homelab-frontend#64, now
deployed at 64f2bfc1):
- four dashboard panels: parked, rejected by reason, engine state stale,
and gateway-view vs engine-truth. Each carries a description, because
an unexplained blank panel reads as a broken dashboard.
- alerts on observer_degraded and engine_state_stale_total, both with an
absent() guard: the gate fails OPEN, so a missing or renamed metric must
alert rather than look healthy. Verified live that the guards do not
false-fire (both series present, 3 each).
- LLMPreemptions and LLMPrefillStarvation: the acceptance metric and the
regression direction this change introduces.
- max(vllm_requests_running), not sum(): one engine-global gauge is read
by all three gateway replicas, so sum() read 3x and could never agree.
Also corrects comments that were factually wrong. An earlier revision of
this branch claimed llm_ttft_seconds/llm_itl_seconds/llm_tokens_total were
dead code; homelab-frontend#63 wired them, verified live as 3 series each.
request_duration_seconds is the genuinely dead one (0 samples, zero
non-test call sites) and is now marked as such instead of being listed as
working.
Closes#51
Previous comments described llm_ttft_seconds / llm_itl_seconds /
llm_tokens_total as "streaming-only". That was wrong, and optimistic.
They are dead code. internal/proxy/llm_metrics.go defines
LLMMetricsCapture, which is the only thing that calls RecordTTFT /
RecordITL / RecordTokens — but NewLLMMetricsCapture() has zero call sites
anywhere in the repo, verified by grep. The exporter skips keys with no
samples, so those three series render as HELP-only stubs and can never be
graphed, streaming or not.
No behaviour change here; only the comments and the dashboard generator's
header are corrected so the next person does not build on a false
assumption. The gateway latency panels are left in place so the gap stays
visible — they populate unchanged once the capture wrapper is wired into
the streaming proxy path.
Series that DO work from the gateway, confirmed live with 3 series (one
per replica): queue_full_requests_total, warmup_requests_delayed_total,
vllm_prefix_cache_hit_tokens_total.
Both new ServiceMonitors sat permanently OutOfSync. Cause: `action: replace`
is the API server default for relabeling rules, so the live object carried it
while the manifests omitted it.
Rather than write out the default to paper over drift, removed the rules that
were never needed:
- vllm.yaml: the namespace/pod/service relabelings were redundant —
Kubernetes service discovery already supplies all three. Only `app` has to
be derived (TEI panels aggregate by it and SD does not promote arbitrary
pod labels). Kept that one, with an explicit action.
- api-gateway.yaml: the metricRelabelings block was a no-op — it mapped
__address__ onto instance, which is already the default. Removed. The
series inventory it was documenting moved up into the file header, which
is where it belonged anyway.
New Grafana dashboard (llm-observability) with 27 panels across 5 rows:
TTFT, inter-token latency, batching/scheduling, KV cache, throughput.
Both metric sources are shown side by side because they answer different
questions and neither is complete alone:
vLLM 0.11.0 (reasoning) — engine truth: kv_cache_usage_perc,
num_requests_running/waiting, num_preemptions_total, prefill vs
decode time, prefix cache, per-request histogram quantiles.
TEI 1.8.2 (embeddings, reranker) — te_batch_next_size, te_queue_size,
te_embed_duration, te_embed_queue_duration.
api-gateway — the ONLY per-request latency source covering Ollama
models (ornith:35b, qwen2.5:3b-instruct), which expose no metrics.
It is derived from the SSE byte stream, so it is streaming-only and
counts SSE chunks rather than true tokens. vLLM's own histograms are
authoritative for reasoning.
All 44 panel queries were executed against live Prometheus before commit —
all return success. vLLM/TEI panels have live series; gateway panels show
no data until the netpol fix in homelab-frontend PR #62 merges.
Also fixes a dead alert. alerts/cag-kv-cache-rules.yaml derived
vllm:prefix_cache_hit_rate from vllm:cache_query_hit_total and
vllm:cache_query_miss_total. Those metric names do not exist in vLLM
0.11.0 — the real series are vllm:prefix_cache_hits_total and
vllm:prefix_cache_queries_total. The recording rule produced no data, so
VLLMPrefixCacheHitRateLow could never fire.
Not included: real GPU hardware metrics (SM utilisation, framebuffer,
power, temperature). Only nvidia-device-plugin is deployed, which exposes
none of them; DCGM exporter is required. vllm:kv_cache_usage_perc and
preemption rate are the only GPU-pressure proxies available today.
Both metric sources existed but neither was scraped:
- servicemonitors/vllm.yaml was orphaned (absent from kustomization
resources:) and would not have matched anyway — it used 'port: http'
but KServe-generated Services in llm-serving have UNNAMED ports.
Switched to targetPort: 8080 (prometheus-operator v0.94.1 supports it),
which avoids editing the InferenceServices and therefore avoids a
predictor restart / weights reload.
- api-gw exposes llm_ttft_seconds, llm_itl_seconds, llm_tokens_total,
queue_full_requests_total, warmup_requests_delayed_total and
vllm_prefix_cache_hit_tokens_total, but the Deployment's
prometheus.io/scrape annotation is ignored by kube-prometheus-stack
(no additionalScrapeConfigs). Added a ServiceMonitor for it. This is
the only per-request latency source covering the two Ollama models,
which expose no metrics endpoint at all.
Ollama services are excluded from the vllm ServiceMonitor selector to
avoid permanently-DOWN targets.
Note: alerts/cag-kv-cache-rules.yaml references vllm:prefix_cache_hit_rate
derived from vllm:cache_query_hit_total/miss, which were never scraped.
Those alerts were dead; verify the metric names after the first scrape.
Actual TSDB data is ~2Gi. 100Gi over-provisioned Longhorn storage
contributed to the cluster-wide scheduling pressure; 25Gi is ample
with room for retention growth.
GET /user/sign_up returned 200 — anyone reaching forgejo.riotpiao.com
could create an account. Add [service] DISABLE_REGISTRATION: true.
Admin-created accounts (UI/API) are unaffected.
32 degraded + 3 faulted volumes from replica rebuild failures.
cp-1/cp-2 over-committed; worker-1 had 602Gi free but scheduling
disabled. Enable it with 400Gi reserved for LLM weights.
- Add worker-1 to longhorn-nodes.yaml (allowScheduling: true)
- Add worker-1 disk config to add-disks-job (storageReserved: 400Gi,
evictionRequested: false)
GITEA__WEBHOOK__ALLOWED_HOST_LIST=external,api.riotpiao.com
api.riotpiao.com resolves to cluster ingress ClusterIP (private range)
which is blocked by Forgejo's default SSRF protection. Safe to allowlist
as it is our own api-gw.
Points to riotpiao-poimen/poimen-memory.git k8s/app/ (kustomize).
ArgoCD auto-syncs Redis deployment, NetworkPolicies, ConfigMap, Service.
Image updater: digest strategy on :latest tag for rolling deployments.
Wave 7 — same wave as poimen workflow app.
V100-PCIE-32GB has ~12GB VRAM headroom after model weights.
ornith:35b Q4_K_M KV cache at 64K ≈ 11.8GB — fits within budget.
Model natively supports 256K but hardware caps safe limit at 64K.
nginx-ingress v1.15.1 blocks configuration-snippet by default as a
security risk, silently dropping the entire Ingress resource. WebSocket
upgrade headers are handled natively with proxy-http-version 1.1.
- Increase PVC from 20Gi to 100Gi (talos-cp-1 disk full)
- Move from az-a (talos-cp-1) to worker-1 (648GB available)
- Increase retention from 15d to 30d (95GB limit)
- Fixes Prometheus pod restarts due to disk pressure
- Add MINIO_BROWSER_REDIRECT_URL pointing to public URL
- Add MINIO_BROWSER_LOGIN_ANIMATION=off
- Enables Console to show OIDC/Authentik SSO button on login page
- Change MINIO_IDENTITY_OPENID_CLAIM_NAME from 'policy' to 'groups'
- Add MINIO_IDENTITY_OPENID_CLAIM_PREFIX (empty) to use raw group names
- Update scopes to 'openid,profile,email,groups'
- Enables Authentik group-based access (homelab-admins, minio-admins, etc)
- Users in homelab-admins group will get admin access to MinIO
The correct env var is PAPERLESS_SOCIAL_AUTO_SIGNUP not PAPERLESS_SOCIALACCOUNT_AUTO_SIGNUP
This enables automatic linking of Authentik social accounts to existing Paperless users
- Add SOCIALACCOUNT_AUTO_SIGNUP for automatic social account creation
- Enable ACCOUNT_ALLOW_SIGNUPS for OIDC users
- Add SOCIALACCOUNT_EMAIL_AUTHENTICATION_AUTO_CONNECT to link existing users
- Fixes 'An account already exists with this email' error
- Allows Authentik login to auto-link to existing Paperless users
Re-enable skip_org_role_sync=false to sync Admin role from grafana-admins group.
User creation worked with skip_org_role_sync=true, now restore role sync.
Set skip_org_role_sync=true to allow user creation on first OAuth signin.
When false, Grafana tries to sync org roles before user exists, causing creation to fail.
Reverts to standard OpenID 'preferred_username' claim which is guaranteed to be present in all userinfo responses. The 'email' claim requires explicit scope mapping in Authentik that may not be consistently returned.
Authentik user 'rock' has preferred_username='rock' which matches the existing Grafana user login.
- Commit paperless-ai-config.yaml with production environment variables
- Configure Paperless API endpoint and token
- Enable AI processing with qwen2.5:3b model
- Set up auto-tagging, correspondent/document type extraction
- Configure 60s scan interval for document processing
- Add ConfigMap to kustomization.yaml for GitOps deployment
- Add ConfigMap with production paperless-ai config
- Mount .env config file for app startup
- Enable auto-tagging, correspondent extraction, document type detection
- Set LLM API endpoint and token file location
- Configure 60s scan interval for document processing
- Change login_attribute_path from preferred_username to email for stable user matching
- Enable allow_sign_up to permit OAuth user sync with existing local users
- Root cause: Authentik's 'rock' user matches existing Grafana 'rock' by email, not by preferred_username claim
- Remove hardcoded client secret from init container
- Reference secret from paperless-ai-config (SOPS-encrypted)
- Init container reads LLM_AUTH_CLIENT_SECRET env var
- Secret key must be added to k8s/argocd/secrets/paperless-ai-secrets.enc.yaml
by someone with SOPS/age key access
This removes the plaintext secret from commit history and future
deployments will source it securely from the encrypted Secret.
- Add paperless namespace to llm-serving NetworkPolicy ingress
- Point paperless-ai CUSTOM_BASE_URL to reasoning-predictor directly
- Gateway JWT auth not yet available for service accounts
- paperless-ai-agent has llm:inference role in Authentik
- stale-job-cleanup: run every 2m, delete completed/failed jobs+pods after 2m
- paperless-ai: add PAPERLESS_USERNAME=admin (required to find own user ID)
ROOT CAUSE:
- tektoncd/pipeline config/ dir uses ko:// image refs (build-time placeholders)
- ArgoCD synced the raw dev manifests → InvalidImageName on all pods
- tektoncd/operator requires its own CRDs and controller (too heavy)
- Tekton has no official Helm chart
FIX:
- Vendor the pre-built release.yaml (v0.68.0) into k8s/infra/tekton/
- Point ArgoCD Application at our own repo (forgejo)
- Release contains real container images (ghcr.io/tektoncd/pipeline/*)
- Remove external tektoncd repo from AppProject (not needed)
TO UPGRADE TEKTON:
1. Download new release from github.com/tektoncd/pipeline/releases
2. Replace k8s/infra/tekton/release.yaml
3. Commit — ArgoCD syncs automatically
Required by paperless-ai to find its own user ID for scanning.
Without this, scanning aborts with "Failed to get own user ID".Reviewed-on: #45
Co-authored-by: rock <[email protected]>
Root cause: k8s Service named 'comfyui' auto-injects COMFYUI_PORT=tcp://...
into pod env, clobbering ai-dock's own COMFYUI_PORT variable which expects
a port number. This broke caddy's proxy config, leaving port 8188 dead.
Fix: enableServiceLinks: false, revert to port 8188 (ai-dock default).
Also bump startup probe failureThreshold 60->120 (20min budget for model loading).
- paperless-ai deployment using clusterzx/paperless-ai
- LLM via local api-gateway (reasoning model, no auth - phase 3 pending)
- Paperless API token SOPS-encrypted
- Auto-tags new documents, 5min scan interval
- Adds 'ai-processed' tag to classified documents
Liveness probe was firing after 60s with failureThreshold:6, killing
the server container before it finished applying 200+ DB migrations.
The startup probe (20min timeout) never got a chance to complete.
Root cause: Authentik health checks fail during long DB bootstrap.
Both liveness and startup probes run in parallel. Liveness killed the
pod at 60s; migrations need 2-3min minimum.
Solution: Add initialDelaySeconds:300 to liveness/readiness probes so
they don't fire until 5min have passed (migrations definitely complete).
Worker gets same treatment since it depends on server's DB bootstrap.
- Added queue OAuth2 provider (pk=13, client_id=queue-sqs)
- Added client_credentials grant type to all OAuth2 providers
- Updated temporal-worker-agent service account:
- Added queue:send role
- Added sqs_queues=* claim
- Provisioning script now creates 6 OAuth2 providers (all complete)
- All groups, scopes, and service accounts ready for Phase 2
Updated 2026-09-12 in provision-rbac.py
uid/gid are FAT/NTFS/FUSE mount options, not ext4. Caused mount exit 32
on every newly provisioned CNPG PVC, blocking authentik-db initdb.
CNPG handles postgres ownership via its own init container.
Add ServiceMonitor for llm-serving namespace. Wire prometheus.io annotations to all vLLM pods (reasoning, ornith, embeddings, reranker). Scrape /metrics@8080 every 30s with proper relabeling.
ArgoCD couldn't decrypt secrets.yaml because it was listed as a plain
kustomize resource. Move the 3 secrets (gotify-admin, gotify-tokens,
gotify-smtp) to k8s/argocd/secrets/ as .enc.yaml files processed by
the ksops generator, matching the repo convention.
Fixes ComparisonError: 'Object Kind is missing' (SOPS ciphertext
parsed as raw YAML).
Repos transferred: homelab-frontend, kmsvc-manage, poimen, poimen-memory,
poimen-workflows, poimen-frontend. Old URLs return 301 which ArgoCD
doesn't follow.
NOT changed: container image registry paths (rock/ is correct for registry),
riotpiao.com (still under rock org).
Also adds poimen-frontend to AppProject sourceRepos allowlist.
rock