worker-1 holds 64Gi and the other predictors request about 55Gi, so the 16Gi request
left clef-flash Pending with '0/4 nodes are available: 1 Insufficient memory'. The
weights live on the GPU; host memory covers SGLang's own bookkeeping, so 8Gi is the
honest reservation. The 32Gi limit stays, so a burst still gets the room.
The installed InferenceService CRD declares annotations and labels directly under
spec.predictor, not under spec.predictor.podMetadata. Two manifests used podMetadata,
so every diff and apply failed with 'field not declared in schema' and the whole
Application could not reconcile.
Move the prometheus scrape annotations and the labels up one level. Nothing is lost:
the same keys, in the place the CRD accepts.
The resource list named ornith.yaml and reasoning.yaml, and neither file has ever been
committed to this directory. kustomize therefore failed to build, the Application sat
in ComparisonError, and nothing in this directory could reconcile — including the
clef-flash predictor added a commit earlier.
Removing the entries does not remove the models: the Application runs with prune=false,
so an object it no longer lists is left untouched. Re-export both from live state the
next time either is edited, so the directory describes what actually runs.
The gateway already dispatches /v1/systemone and its config already names
clef-flash-predictor, but this directory is an explicit allowlist and clef-flash.yaml
was not on it, so the model was silently dropped: no error, no drift, just no
predictor.
clef-flash is a decision model, not a chat model. It reads a state plus a schema of
typed questions and returns one probability per allowed option in a single prefill
pass, with no decode step. It needs SGLang rather than vLLM because it loads separate
joint_schema_head weights on top of the Qwen3.5-9B backbone and it implements the
/v1/systemone route.
worker-1 has four GPUs: reasoning holds two, ornith holds one, so one is free and this
fits without retiring ornith first.
The init container fetched a token once at pod start with a client_credentials grant
against a web app. A web app has no such grant, so the fetch could not work; and a
token fetched once is dead within the hour while the pod runs for weeks.
llm-token-vendor runs beside the app. It signs a machine-key assertion for the
paperless-llm service account, exchanges it for a token, and rewrites
/app/data/llm_token.txt atomically at 80% of the token lifetime. The vendor image
keeps using CUSTOM_API_KEY_FILE, so it needs no change and no OIDC support.
The key is re-read on every mint, so a rotation lands without a restart: publish the
new key, the next mint uses it, retire the old one.
paperless-llm is a separate service account from poimen-llm, so revoking one caller
leaves the other working.
Two defects that a real rotation exposed:
merge writeSecret replaced the whole Secret map, so every key a consumer
keeps beside the credential disappeared. paperless lost
SOCIALACCOUNT_PROVIDERS_JSON and immich lost config.json; both pods then
failed to start while the rotation reported success.
backoff a Failed attempt returned 0, so the controller retried in a tight loop.
It now waits two minutes and then retries.
All eight objects carry both identifiers ZITADEL needs: the app resource id for the
secret endpoint and the client id for introspection.
Result: seven rotations reach Succeeded with every consumer ready (grafana, comfyui,
argocd, forgejo, paperless, immich, minio). api-gw stays suspended because no workload
exists for it.
Three defects, each seen in a live rotation of grafana:
scope the access token needs urn:zitadel:iam:org:project:id:zitadel:aud,
otherwise the management API answers 401 Token.Invalid even though the
assertion was accepted.
app id the objects carry the client id, and the secret endpoint takes the app
resource id. They differ, so every call answered 404 Application
doesn't exist. The correct ids are recorded per object.
attempt time record ActiveCredentialID and LastRotation right after Prepare. Without
it a later failure leaves LastRotation empty, the due check says rotate
now, and each reconcile mints a new secret and restarts the consumer:
grafana was rolled continuously and never became ready.
Also raise the consumer-ready budget to 8 minutes, because grafana needs about four,
and retry a failed attempt after 2 minutes instead of waiting a whole interval.
Three faults stopped the controller from running, each found by running it:
image arch the first build was arm64 and every node is amd64, so the pod died
with 'exec format error'. imagePullPolicy IfNotPresent kept the
cached wrong-arch layer, so it is now Always.
leases leader election needs coordination.k8s.io/leases. Without it the
controller never acquired its lock and never reconciled.
secrets list the manager caches Secrets, so it needs list and watch, not only
get, create, update and patch.
go.mod and go.sum gained the procfs entry the build needs.
Dockerfile: static binary on distroless, no shell, user 65532.
Workflow runs on the golang runner for a push to main that touches the controller.
It builds two tags, the commit sha and 0.1.0, which is the tag the Deployment asks
for, and runs the image with --help so a broken binary fails the job instead of the
cluster.
The runners were dead before this: every task fetch logged 'lookup
forgejo-gitea-http.cicd.svc.cluster.local: no such host'. DNS resolved from a fresh
pod and the Service had a healthy endpoint, so the runner pods held stale network
state. A rollout restart cleared it and all three poll without error.
Eight objects cover grafana, argocd, comfyui, minio, paperless, immich, forgejo and
api-gw. Each names its ZITADEL app, the Secret it writes, the consumer to roll and
whether the consumer reloads hot or needs a restart.
All are suspended: the controller image does not exist yet, and an unsuspended
object would claim an intent that no process acts on.
The README records the retirement target for the 34 .enc.yaml files: six
root-of-trust secrets that cannot rotate themselves survive, everything else
becomes a KeyRotation and leaves git.
Gating /oauth2 with the app redirects /oauth2/start to itself and the browser
loops. ingress-nginx applies an ingress's annotations to the locations that
ingress defines, so the login path belongs on its own object without auth-url.
This split failed once before, and the split was not the reason: the app ingress
carried a configuration-snippet that the controller rejects, which dropped that
whole object and left only the login path rendered.
The controller logs "annotation group ConfigurationSnippet contains risky
annotation" and a rejected annotation group drops the entire ingress from
nginx.conf. comfyui.riotpiao.com therefore answered 404 for every request with no
line in the access log, which is why the host looked routed but served nothing.
ingress-nginx sets the Upgrade and Connection headers itself for a websocket
upgrade, so the snippet was never needed.
Two Ingresses on one host do not merge annotations. nginx applies the annotations
of a single ingress per host, so the separate /oauth2 object silently dropped the
auth_request on the app ingress: every request fell through to the backend and
ComfyUI answered 404 for the site root.
Move /oauth2 and / into the same ingress, so the auth annotations and both paths
belong to one object.
comfyui.riotpiao.com is served with no authentication at all: anyone who can
resolve the host reaches the UI and its queue. ComfyUI has no OIDC support, so the
front door is oauth2-proxy, and nginx asks it for a decision on every request.
Nothing inside the cluster calls this host, so the whole host is gated rather than
carved into open and closed paths. If a client ever needs direct access, add a skip
rule for its exact paths.
The ZITADEL app is created by the provisioning script, and the client secret,
cookie secret and client id live in a SOPS secret written by ksops.
Replace the dead secret-rotation scaffold (an Argo app that could not render, a
CRD that was never installed, and no controller code) with a working design.
One resource describes one credential:
spec.provider zitadel-machine-key | zitadel-oidc-client
spec.strategy overlap | atomic
spec.target Secret to write, consumers to roll, reload hot|restart
spec.verify introspection or token exchange
The state machine is shared by every provider:
Prepare -> Publish -> Verify -> Retire
Order is the safety property. Retire runs only for an overlapping provider and
only after the new credential obtained a token. A failure before that point leaves
the working credential in place, which is exactly what went wrong by hand today
when a regenerated Grafana secret was never re-pointed.
Machine keys overlap in ZITADEL, so that provider rotates with no gap. Client
secrets do not overlap, so that provider is atomic: it publishes, rolls the
consumer, verifies, and restores the previous value from a shadow key if the
introspection check fails.
The controller authenticates with its own machine key, the single bootstrap
credential, read from a mounted Secret. Everything else it creates itself, so no
credential value needs to live in git.
Verified: go build and go vet clean, 12 tests pass covering scheduling, the shadow
key rollback path, provider strategies and assertion signing.
Not yet wired to Argo CD: the image must be built and pushed first, so the app is
not added until a Deployment can actually pull.
The live cluster has no Authentik workload, no DNS record and no tunnel route,
but two declarations still pointed at it.
controlplane.tftpl still set kube-apiserver oidc-issuer-url to
https://authentik.riotpiao.com/application/o/kubernetes/ and pinned that hostname
in extraHostEntries. The live API server carries zero OIDC arguments, so the
template and reality had diverged: the next machine-config render would have
re-added an issuer that does not resolve.
management-service exported KMSVC_AUTHENTIK_ISSUER_URL and
KMSVC_AUTHENTIK_AUDIENCE, both empty. Rename them to KMSVC_ZITADEL_* so the names
describe the identity provider that actually exists.
The CoreDNS Corefile kept one rewrite for authentik.riotpiao.com. The Terraform
source file no longer contains it, so the live ConfigMap was stale; that line is
removed live and the next render converges without it.
Gotify rejects a Grafana payload with 400 "Field 'message' is required", so a
translation step is mandatory. This adds a small standard-library Python
service: it accepts Grafana's JSON, builds a Gotify message, and posts it with an
app token.
Mapping:
firing -> priority 8, title "[FIRING] <alertname>"
resolved -> priority 3, title "[RESOLVED] <alertname>"
body -> the summary annotation plus severity, namespace, pod, instance, job
A multi-alert notification names the alert count instead of dropping the rest.
The script ships as a ConfigMap so no image build is needed. The app token comes
from a SOPS secret and belongs to Gotify application `grafana`.
The bridge runs server-side, so it cannot run a browser flow and ZITADEL has no
part in it. Give it a Gotify client token instead: GOTIFY_URLS points at the
public host, GOTIFY_CLIENT_TOKENS comes from a SOPS secret.
The token belongs to user `rock`, is named igotify-assist, and has
expiresAfterInactivitySeconds 0 so it never idles out. Delete the client row in
Gotify to revoke it.
Verified with the token: GET /client returns 200 and GET /message returns the
latest message.
Gotify has no OIDC support, so it cannot validate a ZITADEL token itself. Put
oauth2-proxy in front of it and let nginx ask the proxy for every request.
The Gotify REST API keeps its own client-token check. /message, /application,
/client, /user, /version, /stream and /health are skipped by the proxy, so
Forgejo webhooks, the harness, poimen-sender and the Android client keep working
without a browser session. Only the web UI requires SSO.
The /oauth2 path needs its own ingress: the auth_request annotation would
otherwise also guard /oauth2/start, and the login entry point would demand a
login first.
The client id and secret come from project role provisioning:
`zitadel-provision.py apply` now creates the `gotify` OIDC app, and the secret is
SOPS-encrypted under k8s/argocd/secrets.
Gotify's own login remains behind the proxy. ZITADEL decides who reaches the
service, Gotify decides what that person may do inside it.
Sign-in failed with "Could not create session for user". The ZITADEL API
logged, on every session call:
authz: verify access token -> invalid token
token verifier repo: issuer does not match: Expected
https://zitadel.riotpiao.com, got: login-client
zitadel-login: [Password] Could not update session
The hosted Login UI authenticates as the machine user `login-client`. That
user did not exist in this instance, because the chart's setup job creates it
and setupJob is disabled after the first sync. Without the user the UI cannot
write a session, so no one can sign in, at ZITADEL or at Forgejo.
Disable the hosted Login UI so ZITADEL serves its built-in login, which needs
no machine user. Re-enable only after creating `login-client` and registering
the public key that matches zitadel-login-service-key.
Also register both callback spellings for the forgejo and paperless apps: the
callback path follows the login-source name, so a rename between authentik and
zitadel must not break sign-in.
Zitadel is running and has provisioned OIDC clients for all 12 consumers
(state.json: kubernetes, paperless, forgejo, argocd, grafana, immich, minio,
vault, poimen, api-gw, queue-sqs, homarr). Authentik's remaining job was to
serve until the last consumer moved.
Remove every Authentik artifact from GitOps:
- ArgoCD Application `authentik` and its Helm values
- ArgoCD Application `iam-jobs` (provisioner RBAC + migration job)
- ArgoCD Application `oidc-bootstrap` (one-shot apiserver restart)
- authentik-db CNPG cluster definition
- authentik-secrets SOPS file and secret-generator entry
- Ingress rule, cloudflared tunnel hostname, CoreDNS rewrite
- Prometheus alerts and ServiceMonitor
- admin OIDC ClusterRoleBinding provisioning binding
- scripts/iam/provision-rbac.py (the Authentik provisioner)
The chart repo https://charts.goauthentik.io is removed from the AppProject.
Kube-apiserver OIDC and consumer configs are switched to Zitadel in the
local cluster-config (gitignored) and applied separately via talosctl.
Commit a02bee5 had both the Ingress rule and the cloudflared tunnel
hostname for authentik.riotpiao.com. Both were lost, so the tunnel
answered 404 for the hostname and the ingress had no matching rule.
The CoreDNS rewrite and the authentik-server Service were still intact;
only the two routing entries were missing. Re-add the Ingress (namespace
iam, service authentik-server port 80, TLS terminated by the wildcard
default-ssl-certificate) and the tunnel hostname.
The tunnel was remotely managed from the Zero Trust dashboard: the
Deployment ran 'run --token' and ignored the ConfigMap that already
existed in the tree. Switch to a locally-managed tunnel so the ingress
routing table lives in git.
- deployment: mount cloudflared-config (config.yml) and the
cloudflared-credentials Secret; drop TUNNEL_TOKEN env
- configmap.yaml: per-hostname ingress rules to ingress-nginx via the
CoreDNS rewrite, plus a 404 catch-all
- secret-generator: register the SOPS-encrypted credentials file
- homelab-ca: add the cloudflared namespace copy so the connector trusts
the ingress certificate (force-ssl-redirect makes plaintext impossible)
Credentials were extracted from the live cloudflared-token Secret and
re-encrypted with SOPS; no plaintext secret is committed.
The macOS workload is scaled to 0 and its PVC macos-disk has been Pending
for 72 days against storage class longhorn-imessage-local. No node carries
the workload=imessage label it pins to. The namespace only holds that dead
PVC plus two unused Services. Remove the Application, the manifests and the
namespace so ArgoCD prunes them.
queue-operator could not list temporalworkers.kmsvc.io or deployments.apps
at cluster scope, so its informers failed on every retry and it reconciled
nothing. Add those rules to the ClusterRole.
macos-bluebubbles pins itself to a node labelled workload=imessage, and no
such node exists. Its PVC macos-disk uses StorageClass longhorn-imessage-local
and stayed unbound for 71 days. Scale the Deployment to 0 and record why.
The ACME DNS-01 self-check resolved through CoreDNS and never saw the TXT
record. The wildcard renewal stalled on 2026-09-23 and riotpiao-com-tls
expired on 2026-10-23. Every Cloudflare Tunnel origin then failed with
"x509: certificate has expired or is not yet valid".
Changes:
- cert-manager: pin the self-check to the public resolvers with
--dns01-recursive-nameservers and --dns01-recursive-nameservers-only.
Verified against Let's Encrypt staging: the challenge went valid and the
certificate issued.
- comfyui and forgejo: drop the per-host tls blocks and the
cert-manager.io/cluster-issuer annotations. The wildcard covers both
hosts, and the per-host certificates (comfyui-tls, forgejo-tls) expired
with the wildcard.
- riotpiao-com-cert: document the wildcard as nginx's default certificate
and the zero-downtime renewal path.
Renewal now runs 30 days before expiry, writes the same secret, and nginx
hot-reloads without a pod restart.
The reasoning engine starves every other request while one long prompt
prefills. Measured: 262 tok/s contended prefill, ~250s for a 65k prompt,
with Avg generation throughput 0.0-0.1 tok/s at Running: 2-4 reqs.
Root cause is NOT what the issue first diagnosed. The issue proposed
--max-num-partial-prefills / --max-long-partial-prefills, which are V0-only
knobs. Verified in the live container: vLLM 0.11.0 ships V1 only (vllm/core/
is absent, so there is no V0 to fall back to), and EngineArgs.
_is_v1_supported_oracle returns False whenever either flag differs from its
default, calling _raise_or_fallback("Concurrent Partial Prefill") ->
"Falling back to V0" -> AssertionError. With strategy Recreate that is an
outage, not a slow rollout. Both flags are therefore NOT set, and a comment
records why so they are not reintroduced.
V1 needs no such flag: it has no prefill/decode phase at all. Every step it
walks self.running and hands out tokens from one shared budget
(vllm/v1/core/sched/scheduler.py). The real defect is that the per-request
cap is disabled by default (long_prefill_token_threshold=0), so one long
prompt takes the WHOLE budget, token_budget hits 0, the `while ... and
token_budget > 0` loop exits, and every other request is skipped.
The cap is applied before the budget clamp, so threshold >= budget never
binds. Replaying the loop: 8192/8192 serves 1 request per step (still
starved); 8192/2048 serves 4, and two long prompts advance together, which
is the issue's actual acceptance criterion.
--max-num-batched-tokens=8192 (was unset -> default 2048)
--long-prefill-token-threshold=2048 (was unset -> 0, cap disabled)
--max-num-seqs=5, --gpu-memory-utilization=0.95
The ~166k KV figure is marked as a projection; the live reading is
num_gpu_blocks=9527 x block_size=16 = 152,432 tokens and is authoritative
post-boot. A preemption alert covers the accept-and-watch risk.
Observability for the companion admission gate (homelab-frontend#64, now
deployed at 64f2bfc1):
- four dashboard panels: parked, rejected by reason, engine state stale,
and gateway-view vs engine-truth. Each carries a description, because
an unexplained blank panel reads as a broken dashboard.
- alerts on observer_degraded and engine_state_stale_total, both with an
absent() guard: the gate fails OPEN, so a missing or renamed metric must
alert rather than look healthy. Verified live that the guards do not
false-fire (both series present, 3 each).
- LLMPreemptions and LLMPrefillStarvation: the acceptance metric and the
regression direction this change introduces.
- max(vllm_requests_running), not sum(): one engine-global gauge is read
by all three gateway replicas, so sum() read 3x and could never agree.
Also corrects comments that were factually wrong. An earlier revision of
this branch claimed llm_ttft_seconds/llm_itl_seconds/llm_tokens_total were
dead code; homelab-frontend#63 wired them, verified live as 3 series each.
request_duration_seconds is the genuinely dead one (0 samples, zero
non-test call sites) and is now marked as such instead of being listed as
working.
Closes#51
V100-PCIE-32GB has ~12GB VRAM headroom after model weights.
ornith:35b Q4_K_M KV cache at 64K ≈ 11.8GB — fits within budget.
Model natively supports 256K but hardware caps safe limit at 64K.
nginx-ingress v1.15.1 blocks configuration-snippet by default as a
security risk, silently dropping the entire Ingress resource. WebSocket
upgrade headers are handled natively with proxy-http-version 1.1.
The correct env var is PAPERLESS_SOCIAL_AUTO_SIGNUP not PAPERLESS_SOCIALACCOUNT_AUTO_SIGNUP
This enables automatic linking of Authentik social accounts to existing Paperless users
- Add SOCIALACCOUNT_AUTO_SIGNUP for automatic social account creation
- Enable ACCOUNT_ALLOW_SIGNUPS for OIDC users
- Add SOCIALACCOUNT_EMAIL_AUTHENTICATION_AUTO_CONNECT to link existing users
- Fixes 'An account already exists with this email' error
- Allows Authentik login to auto-link to existing Paperless users
rock
pi