The disk behind /var/lib/longhorn-disk4 (sde1, 2TB PERC H710P virtual disk)
failed: 54 million SCSI I/O errors and climbing at ~7,000/minute, against 3 on
every other device on the node. Its XFS filesystem hung, so five Longhorn
replica processes and the longhorn-manager blocked in uninterruptible D state
in openat and unlinkat. D-state processes cannot be killed, so they held the
node's three replica rebuild slots permanently: no new replica could start on
talos-cp-2 and every new volume there failed to attach with DeadlineExceeded.
That is what stopped the azhi database from coming up.
Removed from all three declarations so it cannot come back:
- terraform.tfvars.local.bak, longhorn_disks for talos-cp-2
- the Longhorn add-disks Job, which would otherwise re-register it
- the Job's disk-count guard, 5 -> 4, or it would fire on every run once the
disk is gone
The running cluster already has the disk fenced
(allowScheduling=false, evictionRequested=true); the replicas are evicted and
the device retired at the controller after that.
The committed CronJob could never run, for two independent reasons:
- quay.io/coreos/etcd ships no /bin/sh, so command: [/bin/sh, -c] died with
exec: "/bin/sh": no such file or directory
- it existed only on branch fix/etcd-defrag-cron, while the ArgoCD application
cluster-maintenance tracks main (targetRevision: main), so the cluster never
had the CronJob or its Secret
This revision fixes both and records the 2026-10-07 incident:
- image bitnamilegacy/etcd:3.6.4 pinned by digest (ships bash and etcdctl 3.6.4,
matching the 3.6.11 servers)
- 15-day cadence by UTC epoch-day modulo; */15 in day-of-month fires on the 1st,
16th and 31st, which puts two runs one day apart
- compacts to the MINIMUM revision across members, so a lagging member is never
compacted past its own revision (measured: talos-cp-2 froze ~1.2M entries
behind the leader and needed a reboot to rejoin)
- skips defrag on a member whose applied index lags the leader by 100k+
- documents that compact must precede defrag: compaction had stalled while the
NOSPACE alarm was up, so dbSizeInUse read 100% of the file and defrag alone
would have reclaimed nothing
etcd's 2 GiB quota was reached on 2026-10-07 (backend 2.1 GiB, ~73 days of
churn). etcd raised NOSPACE and refused all writes, so kubelets could not post
node status: three nodes went NotReady and the tunnel origin pods on them
disappeared. Recovery was compact + defrag + alarm disarm per member.
The apiserver compacts every 5m, but compaction never shrinks the backend file -
only a defrag returns space, and nothing scheduled it. This adds:
- CronJob etcd-defrag (kube-system, Sundays 03:00): compact to the current
revision, defrag each member sequentially, disarm NOSPACE, print before/after.
- Secret etcd-defrag-creds (SOPS/ksops, k8s/argocd/secrets): etcd CA plus a
CN=etcd-defrag client cert signed by the cluster's etcd CA (expires 2036-10).
Applied by the existing apps: cluster-maintenance (directory) and 00-secrets
(ksops generator).
MinIO deleted the community edition images upstream. Docker Hub reports
{"message":"object not found"} for minio/minio, and an anonymous pull of
quay.io/minio/mc returns HTTP 401. talos-cp-1 dropped the cached image during
the DiskPressure event on 2026-09-22. The tenant pod was recreated on
2026-09-23 and has been in ImagePullBackOff since. MinIO served no traffic for
13 days, which put vault-0 and loki-0 into CrashLoopBackOff and left the
paperless backups stuck.
Pin the tenant image to the digest of the last community release, so a retag or
a deletion on the mirror cannot break the pull again. Repoint the two
quay.io/minio/mc references at the archived Bitnami client image, also by
digest. Alert on ImagePullBackOff and on a pod that stays not ready, because
neither case raised any alert.
Evidence: https://hub.docker.com/v2/repositories/minio/minio/tags -> {"message":"object not found"}
Adds the pi coding-agent sandbox's cluster side:
- `k8s/infra/poimen/pi-agent-workspace-pv.yaml` — static hostPath PV + PVC for the sandbox /workspace (Longhorn cannot attach new volumes in this cluster; this gives real persistence and survived a pod delete/recreate test: marker file + 38 KB session.jsonl intact, the session answered from its own history afterwards).
- `k8s/argocd/secrets/pi-sandbox-secrets.enc.yaml` — the sandbox's API-key Secret, SOPS-encrypted (recipient from .sops.yaml, payload only: `encrypted_regex ^(data|stringData)$`), registered in the ksops generator. Round-trip verified against the live Secret.
- `terraform/files/coredns/Corefile` — `rewrite name pi.riotpiao.com ingress-nginx-controller...` so the hostname resolves in-cluster.
Note: the keys currently in that Secret are the ones leaked earlier and must be rotated; re-encrypt with `sops -e -i` after rotating.Reviewed-on: #54
Co-authored-by: rock <[email protected]>
## Why
`minio/minio:RELEASE.2025-07-23T15-54-02Z` no longer exists. The Docker Hub repository
`minio/minio` returns **404 for every tag** (checked: the pinned tag and `latest`), and
MinIO has moved its images behind registry authentication (`quay.io/minio/minio` → 401,
`registry.min.io/minio/minio` → auth required, `dl.min.io` binaries → 410 Gone).
The tenant therefore sat in `ImagePullBackOff` — 44,229 attempts over 6d23h — and the chain
from there is:
```
minio tenant pod not ready (1/2)
-> headless Service minio-cluster-hl has no endpoints
-> CoreDNS: lookup minio-cluster-hl.storage.svc.cluster.local -> no such host
-> Vault S3 storage init fails at startup
-> vault-0 CrashLoopBackOff, 1943 restarts, ~7 days
```
## Control checks (so this is not a network story)
| check | result |
| --- | --- |
| `alpine:3.20` pulled on talos-cp-1 | Succeeded — docker.io works |
| `quay.io/prometheus/node-exporter:v1.8.2` on the same node | Succeeded — public quay works |
| `minio/minio` (docker.io) | HTTP 404, repo removed |
| old tag `RELEASE.2024-06-13T20-48-48Z` with `IfNotPresent` | not cached on the node |
## Change
Pin `spec.image` to `coollabsio/minio:RELEASE.2025-10-15T17-29-55Z` — a public mirror, and a
newer upstream release than the previous pin. Verified by pulling it on talos-cp-1 and
running it: the image carries `/usr/bin/docker-entrypoint.sh` (the layout the operator
expects, which runs `minio server … --console-address :9090`) and reports
`minio version RELEASE.2025-10-15T17-29-55Z`.
## Caveat
This is a community mirror, not MinIO's official image, because the official ones now
require credentials. If MinIO registry credentials are available, add an imagePullSecret
and switch back to `quay.io/minio/minio`. Supply-chain decision, flagged deliberately.
## After merge (ArgoCD, not kubectl)
The storage app tracks `k8s/infra/minio`, so the sync rolls the tenant; then:
```
kubectl get pods -n storage # minio-cluster-az-a-0 -> 2/2
kubectl get pods -n iam | grep vault # vault-0 -> Running (storage init succeeds)
```
Residual risk, unrelated to this fix: the tenant's 4 x 25Gi volumes are Longhorn-backed, and
Longhorn currently fails to create engines for new volume attaches cluster-wide. If the
restarted pod cannot attach, that is a second, separate fault.Reviewed-on: #53
Co-authored-by: rock <[email protected]>
The reasoning engine starves every other request while one long prompt
prefills. Measured: 262 tok/s contended prefill, ~250s for a 65k prompt,
with Avg generation throughput 0.0-0.1 tok/s at Running: 2-4 reqs.
Root cause is NOT what the issue first diagnosed. The issue proposed
--max-num-partial-prefills / --max-long-partial-prefills, which are V0-only
knobs. Verified in the live container: vLLM 0.11.0 ships V1 only (vllm/core/
is absent, so there is no V0 to fall back to), and EngineArgs.
_is_v1_supported_oracle returns False whenever either flag differs from its
default, calling _raise_or_fallback("Concurrent Partial Prefill") ->
"Falling back to V0" -> AssertionError. With strategy Recreate that is an
outage, not a slow rollout. Both flags are therefore NOT set, and a comment
records why so they are not reintroduced.
V1 needs no such flag: it has no prefill/decode phase at all. Every step it
walks self.running and hands out tokens from one shared budget
(vllm/v1/core/sched/scheduler.py). The real defect is that the per-request
cap is disabled by default (long_prefill_token_threshold=0), so one long
prompt takes the WHOLE budget, token_budget hits 0, the `while ... and
token_budget > 0` loop exits, and every other request is skipped.
The cap is applied before the budget clamp, so threshold >= budget never
binds. Replaying the loop: 8192/8192 serves 1 request per step (still
starved); 8192/2048 serves 4, and two long prompts advance together, which
is the issue's actual acceptance criterion.
--max-num-batched-tokens=8192 (was unset -> default 2048)
--long-prefill-token-threshold=2048 (was unset -> 0, cap disabled)
--max-num-seqs=5, --gpu-memory-utilization=0.95
The ~166k KV figure is marked as a projection; the live reading is
num_gpu_blocks=9527 x block_size=16 = 152,432 tokens and is authoritative
post-boot. A preemption alert covers the accept-and-watch risk.
Observability for the companion admission gate (homelab-frontend#64, now
deployed at 64f2bfc1):
- four dashboard panels: parked, rejected by reason, engine state stale,
and gateway-view vs engine-truth. Each carries a description, because
an unexplained blank panel reads as a broken dashboard.
- alerts on observer_degraded and engine_state_stale_total, both with an
absent() guard: the gate fails OPEN, so a missing or renamed metric must
alert rather than look healthy. Verified live that the guards do not
false-fire (both series present, 3 each).
- LLMPreemptions and LLMPrefillStarvation: the acceptance metric and the
regression direction this change introduces.
- max(vllm_requests_running), not sum(): one engine-global gauge is read
by all three gateway replicas, so sum() read 3x and could never agree.
Also corrects comments that were factually wrong. An earlier revision of
this branch claimed llm_ttft_seconds/llm_itl_seconds/llm_tokens_total were
dead code; homelab-frontend#63 wired them, verified live as 3 series each.
request_duration_seconds is the genuinely dead one (0 samples, zero
non-test call sites) and is now marked as such instead of being listed as
working.
Closes#51
Previous comments described llm_ttft_seconds / llm_itl_seconds /
llm_tokens_total as "streaming-only". That was wrong, and optimistic.
They are dead code. internal/proxy/llm_metrics.go defines
LLMMetricsCapture, which is the only thing that calls RecordTTFT /
RecordITL / RecordTokens — but NewLLMMetricsCapture() has zero call sites
anywhere in the repo, verified by grep. The exporter skips keys with no
samples, so those three series render as HELP-only stubs and can never be
graphed, streaming or not.
No behaviour change here; only the comments and the dashboard generator's
header are corrected so the next person does not build on a false
assumption. The gateway latency panels are left in place so the gap stays
visible — they populate unchanged once the capture wrapper is wired into
the streaming proxy path.
Series that DO work from the gateway, confirmed live with 3 series (one
per replica): queue_full_requests_total, warmup_requests_delayed_total,
vllm_prefix_cache_hit_tokens_total.
Both new ServiceMonitors sat permanently OutOfSync. Cause: `action: replace`
is the API server default for relabeling rules, so the live object carried it
while the manifests omitted it.
Rather than write out the default to paper over drift, removed the rules that
were never needed:
- vllm.yaml: the namespace/pod/service relabelings were redundant —
Kubernetes service discovery already supplies all three. Only `app` has to
be derived (TEI panels aggregate by it and SD does not promote arbitrary
pod labels). Kept that one, with an explicit action.
- api-gateway.yaml: the metricRelabelings block was a no-op — it mapped
__address__ onto instance, which is already the default. Removed. The
series inventory it was documenting moved up into the file header, which
is where it belonged anyway.
New Grafana dashboard (llm-observability) with 27 panels across 5 rows:
TTFT, inter-token latency, batching/scheduling, KV cache, throughput.
Both metric sources are shown side by side because they answer different
questions and neither is complete alone:
vLLM 0.11.0 (reasoning) — engine truth: kv_cache_usage_perc,
num_requests_running/waiting, num_preemptions_total, prefill vs
decode time, prefix cache, per-request histogram quantiles.
TEI 1.8.2 (embeddings, reranker) — te_batch_next_size, te_queue_size,
te_embed_duration, te_embed_queue_duration.
api-gateway — the ONLY per-request latency source covering Ollama
models (ornith:35b, qwen2.5:3b-instruct), which expose no metrics.
It is derived from the SSE byte stream, so it is streaming-only and
counts SSE chunks rather than true tokens. vLLM's own histograms are
authoritative for reasoning.
All 44 panel queries were executed against live Prometheus before commit —
all return success. vLLM/TEI panels have live series; gateway panels show
no data until the netpol fix in homelab-frontend PR #62 merges.
Also fixes a dead alert. alerts/cag-kv-cache-rules.yaml derived
vllm:prefix_cache_hit_rate from vllm:cache_query_hit_total and
vllm:cache_query_miss_total. Those metric names do not exist in vLLM
0.11.0 — the real series are vllm:prefix_cache_hits_total and
vllm:prefix_cache_queries_total. The recording rule produced no data, so
VLLMPrefixCacheHitRateLow could never fire.
Not included: real GPU hardware metrics (SM utilisation, framebuffer,
power, temperature). Only nvidia-device-plugin is deployed, which exposes
none of them; DCGM exporter is required. vllm:kv_cache_usage_perc and
preemption rate are the only GPU-pressure proxies available today.
Both metric sources existed but neither was scraped:
- servicemonitors/vllm.yaml was orphaned (absent from kustomization
resources:) and would not have matched anyway — it used 'port: http'
but KServe-generated Services in llm-serving have UNNAMED ports.
Switched to targetPort: 8080 (prometheus-operator v0.94.1 supports it),
which avoids editing the InferenceServices and therefore avoids a
predictor restart / weights reload.
- api-gw exposes llm_ttft_seconds, llm_itl_seconds, llm_tokens_total,
queue_full_requests_total, warmup_requests_delayed_total and
vllm_prefix_cache_hit_tokens_total, but the Deployment's
prometheus.io/scrape annotation is ignored by kube-prometheus-stack
(no additionalScrapeConfigs). Added a ServiceMonitor for it. This is
the only per-request latency source covering the two Ollama models,
which expose no metrics endpoint at all.
Ollama services are excluded from the vllm ServiceMonitor selector to
avoid permanently-DOWN targets.
Note: alerts/cag-kv-cache-rules.yaml references vllm:prefix_cache_hit_rate
derived from vllm:cache_query_hit_total/miss, which were never scraped.
Those alerts were dead; verify the metric names after the first scrape.
Actual TSDB data is ~2Gi. 100Gi over-provisioned Longhorn storage
contributed to the cluster-wide scheduling pressure; 25Gi is ample
with room for retention growth.
GET /user/sign_up returned 200 — anyone reaching forgejo.riotpiao.com
could create an account. Add [service] DISABLE_REGISTRATION: true.
Admin-created accounts (UI/API) are unaffected.
32 degraded + 3 faulted volumes from replica rebuild failures.
cp-1/cp-2 over-committed; worker-1 had 602Gi free but scheduling
disabled. Enable it with 400Gi reserved for LLM weights.
- Add worker-1 to longhorn-nodes.yaml (allowScheduling: true)
- Add worker-1 disk config to add-disks-job (storageReserved: 400Gi,
evictionRequested: false)
GITEA__WEBHOOK__ALLOWED_HOST_LIST=external,api.riotpiao.com
api.riotpiao.com resolves to cluster ingress ClusterIP (private range)
which is blocked by Forgejo's default SSRF protection. Safe to allowlist
as it is our own api-gw.
Points to riotpiao-poimen/poimen-memory.git k8s/app/ (kustomize).
ArgoCD auto-syncs Redis deployment, NetworkPolicies, ConfigMap, Service.
Image updater: digest strategy on :latest tag for rolling deployments.
Wave 7 — same wave as poimen workflow app.
V100-PCIE-32GB has ~12GB VRAM headroom after model weights.
ornith:35b Q4_K_M KV cache at 64K ≈ 11.8GB — fits within budget.
Model natively supports 256K but hardware caps safe limit at 64K.
nginx-ingress v1.15.1 blocks configuration-snippet by default as a
security risk, silently dropping the entire Ingress resource. WebSocket
upgrade headers are handled natively with proxy-http-version 1.1.
- Increase PVC from 20Gi to 100Gi (talos-cp-1 disk full)
- Move from az-a (talos-cp-1) to worker-1 (648GB available)
- Increase retention from 15d to 30d (95GB limit)
- Fixes Prometheus pod restarts due to disk pressure
- Add MINIO_BROWSER_REDIRECT_URL pointing to public URL
- Add MINIO_BROWSER_LOGIN_ANIMATION=off
- Enables Console to show OIDC/Authentik SSO button on login page
- Change MINIO_IDENTITY_OPENID_CLAIM_NAME from 'policy' to 'groups'
- Add MINIO_IDENTITY_OPENID_CLAIM_PREFIX (empty) to use raw group names
- Update scopes to 'openid,profile,email,groups'
- Enables Authentik group-based access (homelab-admins, minio-admins, etc)
- Users in homelab-admins group will get admin access to MinIO
The correct env var is PAPERLESS_SOCIAL_AUTO_SIGNUP not PAPERLESS_SOCIALACCOUNT_AUTO_SIGNUP
This enables automatic linking of Authentik social accounts to existing Paperless users
- Add SOCIALACCOUNT_AUTO_SIGNUP for automatic social account creation
- Enable ACCOUNT_ALLOW_SIGNUPS for OIDC users
- Add SOCIALACCOUNT_EMAIL_AUTHENTICATION_AUTO_CONNECT to link existing users
- Fixes 'An account already exists with this email' error
- Allows Authentik login to auto-link to existing Paperless users
Re-enable skip_org_role_sync=false to sync Admin role from grafana-admins group.
User creation worked with skip_org_role_sync=true, now restore role sync.
Set skip_org_role_sync=true to allow user creation on first OAuth signin.
When false, Grafana tries to sync org roles before user exists, causing creation to fail.
Reverts to standard OpenID 'preferred_username' claim which is guaranteed to be present in all userinfo responses. The 'email' claim requires explicit scope mapping in Authentik that may not be consistently returned.
Authentik user 'rock' has preferred_username='rock' which matches the existing Grafana user login.
- Commit paperless-ai-config.yaml with production environment variables
- Configure Paperless API endpoint and token
- Enable AI processing with qwen2.5:3b model
- Set up auto-tagging, correspondent/document type extraction
- Configure 60s scan interval for document processing
- Add ConfigMap to kustomization.yaml for GitOps deployment
- Add ConfigMap with production paperless-ai config
- Mount .env config file for app startup
- Enable auto-tagging, correspondent extraction, document type detection
- Set LLM API endpoint and token file location
- Configure 60s scan interval for document processing
- Change login_attribute_path from preferred_username to email for stable user matching
- Enable allow_sign_up to permit OAuth user sync with existing local users
- Root cause: Authentik's 'rock' user matches existing Grafana 'rock' by email, not by preferred_username claim
rock
rock