poimen-memory will address `reasoning-long-predictor`, and the plan replaces
vLLM with candle-vllm underneath. Without an indirection, that handoff changes
a hostname the client is pinned to, and the ingest cutover has a window where
the name is wrong.
This Service is that indirection. It selects the vLLM reasoning pod today. The
handoff becomes a selector edit with no client change, so poimen-memory's
LLM_CHAT_BASE stays fixed across both engines.
Two ports, both verified through the Service from inside the poimen-memory pod:
80 -> 8080 HTTP 200 in 0.0057s matches every other predictor Service and
poimen-memory's LLM_CHAT_BASE, which has
no port in the URL
8080 -> 8080 HTTP 200 in 0.0182s the container port, the only port the two
NetworkPolicies permit, and the port the
handoff readiness check uses
Existing predictor Services publish port 80 only, so a curl to :8080 on one of
them fails to connect. Publishing both here means neither caller has to know
which convention the other follows.
Verified the selector resolves to the intended pod and nothing else: one
endpoint, 10.244.4.171, which is reasoning-predictor-59b67d5f5d-6mbbb.
Residual state, deliberately: /v1/models through this Service returns
['reasoning'], not ['reasoning-long'], because the second served name lands in
the previous commit and ArgoCD selfHeal reverts a live patch to a tracked pod
spec. Both names appear after this branch merges and syncs.
Not deleted and recreated at handoff. A delete drops the ClusterIP, and a
client that cached the name across the gap fails until it re-resolves.
The goal model map names this checkpoint `reasoning-long`: Qwen3-30B-A3B at
64K on one card, the model qwen-finetune wiring and the models.json overlay
target. It was deployed and healthy but only answerable as `reasoning`, so
that name did not exist.
It is a second served name on the existing pod, not a new
InferenceService. All four V100s are allocated (coder 2, this 1,
clef-serve 1), so a separate reasoning-long predictor would sit Pending
forever. It would also load the same 15.56 GiB of weights a second time to
serve a name, which buys nothing.
The two names are separate list entries. The comma form does NOT work here,
and fails silently in a way that looks like success: this image's parser
turns `--served-model-name=reasoning,reasoning-long` into
['reasoning,reasoning-long'], one model whose name contains a comma, and
api_server.py iterates args.served_model_name when it builds the model list.
Verified against vLLM 0.11.0 itself rather than assumed:
FlexibleArgumentParser().parse_args(["--model","m",
"--served-model-name=reasoning,reasoning-long"])
-> ['reasoning,reasoning-long']
FlexibleArgumentParser().parse_args(["--model","m",
"--served-model-name","reasoning","reasoning-long"])
-> ['reasoning', 'reasoning-long']
`reasoning` stays served from this pod on purpose. poimen-memory pins
LLM_CHAT_BASE=reasoning-predictor, so removing the name breaks ingest
outright. The duplicate with coder-predictor is known and is resolved by
moving the client host, not by deleting a name here.
Not verified end to end: a live apply was reverted by ArgoCD selfHeal before
the new pod finished loading, so the served-name list was never observed on
a running pod with both names. The parser result above is what this commit
rests on. Confirm with `GET /v1/models` after the merge syncs.
Coder-30B-A3B is natively 262,144 tokens with rope_scaling null. One V100
could not hold it: the measured KV pool was 132,672 tokens. TP=2 halves the
weights per card to 7.85 GiB each and frees room for the larger pool.
Measured after boot, both cards worker-1:
Available KV cache memory per GPU : 19.96 GiB
GPU KV cache size : 436,112 tokens
Max concurrency at 262,144 : 1.66x
max_model_len (both served names) : 262144
POST /v1/chat/completions : 200 in 2.83s
Weights, per worker : 7.8516 GiB
torch.compile : 78 s
Graph capture : 6 s
One fatal detail, measured twice. With vLLM's default custom all-reduce the
engine reaches "Graph capturing finished in 6 secs" and then hangs. The only
output afterwards was "No available shared memory broadcast block found in
60 seconds", repeated for 14 minutes, and the front-end never received the
ready message. The first attempt died on its startup probe at 20 min.
`--disable-custom-all-reduce` fixes it. Custom all-reduce registers its own
CUDA graph addresses and deadlocks on this Volta/PCIe pair. NCCL is the
fallback and costs nothing here, because TP=2 across two PCIe V100s gains
nothing from the custom kernel. Both workers now register cleanly and the
API binds at 17:18.
Two supporting changes, both required:
- memory request 12Gi. worker-1 exposes only 63,495 MiB allocatable, so the
earlier 24Gi request could never schedule.
- /dev/shm 8Gi. NCCL all-reduce moves through it, and the 64Mi default
shows up as an init hang.
`reasoning` remains a served name at the same 262,144 so the client needs no
model-name change.
The name now matches the model. `coder` serves
Qwen3-Coder-30B-A3B-Instruct, a 30.5B MoE with 3.3B active, quantized to
4-bit GPTQ. It is measured working on this node, not assumed.
Measured, on worker-1, driver 580.159.04, compute capability 7.0:
GET /v1/models -> id=coder, root=ModelCloud/Qwen3-Coder-30B-A3B-Instruct-GPTQMODEL-W4A16-A
POST /v1/chat/completions model=coder -> 200 in 3.18s,
"fn main() { println!(\"hi\"); }"
POST /v1/chat/completions model=reasoning -> 200 in 0.21s
Weights loaded in 15.61 GiB. torch.compile took 78s. The pod reached
1/1 Ready. `kubectl diff -f coder.yaml` returns empty, so the manifest
matches live state.
Two names are served on purpose: `coder` and `reasoning`. The rename
therefore needs no change to poimen-memory's LLM_MODEL, only its host.
That keeps the client working through the cutover.
Why not Qwen3-Coder-Next, the model in the original plan. It cannot run
here, and that is measured too. A v0.15.0 image on this free V100 gave:
TORCH 2.9.1+cu129
ARCH_LIST [] <- empty
RuntimeError: cudaGetDeviceCount() ... Error 803
fla: ModuleNotFoundError <- no GatedDeltaNet kernels
vLLM 0.15.0 requires compute capability 7.5 or higher and fails here
before it loads a model. Qwen3-Next's hybrid attention needs the `fla`
module, which the image does not ship. So this stays on the pinned
v0.11.0 image, which reports sm_70..sm_120 and works.
Qwen publishes this model as BF16 (61.1 GB) and FP8 only. V100 has no
FP8, and 61.1 GB does not fit 32 GB, so 4-bit GPTQ is the only form that
fits one card. The checkpoint is a community quant; the manifest says so
and names the reason.
reasoning.yaml stays listed until poimen-memory's LLM_CHAT_BASE moves
from reasoning-predictor to coder-predictor. Two GPUs are free, so coder
rolled out beside reasoning with no downtime.
poimen-memory could not reach any inference pod. The egress side was
already correct: NetworkPolicy poimen/poimen-memory-llm-egress permits
port 8080 to the pods labelled llm-reasoning, llm-embeddings and
llm-reranker. This policy had no matching ingress rule, so Cilium
allowed the packet out and dropped it at the destination.
The observable was a bare timeout. The request never reached vLLM, so
the predictor log stayed empty, and /health still answered because a
probe from inside the pod does not cross the policy. That made the
failure look like a hung inference engine.
Evidence, measured on 2026-10-31 against the running predictors:
1. A probe pod in llm-serving WITH label part-of=llm-serving:
/v1/chat/completions -> 200 in 3.1s, 8 completion tokens.
2. The same probe pod WITHOUT that label: 8s timeout, all services.
3. From inside poimen-memory-bfc8k: embeddings /health -> 12s
timeout; 10.96.0.1:443 -> 8s timeout. Same-namespace traffic
still worked, which limits the fault to this policy.
4. memory logged "readiness: embedder probe timed out
elapsed_ms=10000" every 60s.
All five services already serve inference correctly. They are
reasoning (3.1s), reasoning-short (2.5s), embeddings (0.5s, real
768-dim vector), reranker (0.09s, 0.982 vs 0.00004) and qwen-cpu
(4.1s, qwen2.5:3b-instruct).
This adds the missing ingress rule. It mirrors the existing paperless
rule, which bypasses the gateway for the same reason: poimen-memory
has no service-account JWT yet.
Refs: #TBD
The Clef runtime was running from hand-applied objects that existed only in the cluster.
A rebuild would have lost it.
Three things move into git:
- clef-serve.yaml: the Deployment and the Service the gateway routes to
- configMapGenerator for clef-serve/app.py, so an edit to the code rolls the pod
instead of leaving a stale copy mounted
- clef-flash.yaml is deleted: it described an SGLang predictor that cannot run on a
V100, and its slot is now clef-serve.yaml
Recorded in the manifest, because both cost hours: SGLang needs CUDA 13 for Clef and
CUDA 13 has no sm_70 kernels, and device_map over two cards breaks the forward pass with
'indices should be either on cpu or on the same device as the indexed tensor'.
Still open and written down in the file: the image cannot be pushed, because Cloudflare
answers 413 for layers above 100 MB. So each start pip-installs and needs PyPI.
Replaces the Qwen3-32B dense deployment, which needed two V100s for 131k context.
The MoE carries 96 KiB of KV per token against the dense model's 256 KiB, so the same
64k window fits on one card.
Measured on this cluster before the switch: 15.55 GiB of weights, 137k tokens of KV
cache, 2.09x concurrency at 65536 tokens per request, correct answer over a 5921-token
prompt, and clean reasoning_content separation through the gateway.
Three settings are load-bearing on sm_70 rather than tuning:
- VLLM_USE_FLASHINFER_SAMPLER=0 flashinfer's check_cuda_arch raises on CC 7.0
- VLLM_ATTENTION_BACKEND=TRITON_ATTN FlashInfer and FA2 both need sm75+
- deploymentStrategy Recreate a rolling update deadlocks once every card is allocated
The manifest is committed as well as applied: reasoning.yaml had never been in git.
Clef needs SGLang v0.5.22+, which upstream publishes only on CUDA 13. CUDA 13 dropped
Volta kernels and every GPU here is a Tesla V100 (sm_70):
torch.AcceleratorError: CUDA error: no kernel image is available for execution on
the device
SGLang also states sm75 as its floor: the README hardware table omits V100, sm75 support
was removed in sgl-project/sglang#9207, and FlashInfer requires sm75+. No build has both
Clef and Volta kernels, so the model is not runnable here at any dtype or flag.
Remove it from the kustomization so nothing crash-loops, and record the reason in both
files. The manifest stays in git for the day an Ampere-or-newer GPU arrives.
forgejo-db-app is owned by the Cluster CR and was generated by CNPG: there is no
bootstrap secret, so nothing outside cloudnative-pg ever knew that password.
CNPG reads spec.bootstrap.initdb.secret only at creation. Recreate the Cluster CR and
CNPG mints a new password while the data directory keeps the old one — Forgejo starts,
stays Ready, and fails every query. That is a credential fault wearing a network fault's
clothes, and it is exactly the shape of tonight's outage.
This file is the record. Point spec.bootstrap.initdb.secret at it when the cluster is
rebuilt. The value matches the live Secret today; verified by decrypting and comparing.
The CLI path was wrong and is now correct, so the pod reaches the model loader — and
dies there:
Found GPU0 Tesla V100-PCIE-32GB which is of cuda capability 7.0.
Your installed torch==2.14.1+cu130 does not include kernels for this GPU.
Every GPU on worker-1 is a V100 (CC 7.0) and the pinned dev-clef image ships kernels for
CC 8.0 and above. Scale to zero so the pod stops crash-looping, and record the reason in
the manifest. Serving a 9.4B backbone on CPU is not a substitute.
The container args began with a bare 'serve'. This image installs its CLI at
/opt/sglang/bin/sglang and puts nothing called serve on PATH, so every start ended
with 'exec: serve: not found' — 19 restarts, and the InferenceService never became
ready. Name the binary, then the subcommand.
The gitea chart reads pod annotations from gitea.podAnnotations (values.yaml line 583).
A top-level key is silently ignored, so the previous commit changed nothing — the
Deployment came back without the annotation and Reloader still had no trigger.
GITEA__DATABASE_PASSWD comes from forgejo-db-app, which the CNPG Cluster CR owns and
generates. There is no bootstrap secret, so nothing but cloudnative-pg knows that
password. The Deployment reads it into the environment at start.
So a regenerated password leaves Forgejo authenticating with a value the database no
longer accepts: the pod stays Ready, every query fails, and the symptom looks like a
network problem rather than a credential problem. The cnpg.io/reload label on the
Secret is only a hint; this annotation is what makes Reloader act on it.
The gateway compares the token audience with its allow-list of client ids. The scope
carried urn:zitadel:iam:org:project🆔<project>:aud, so every token said
aud = [396757796898471937] and the gateway answered
token validation failed: audience not allowed: [396757796898471937]
With plain openid the audience is the machine user, which is the entry that exists on
the list.
A Secret volume is owned by root and the sidecar runs as 65532, so defaultMode 0400
left it unable to open /var/run/llm-key/key: the container restarted with
'permission denied' on every start and never minted a token.
Set mode 0440 and give the pod fsGroup 65532, so the group that owns the volume can
read the key while nothing else on the host can.
worker-1 holds 64Gi and the other predictors request about 55Gi, so the 16Gi request
left clef-flash Pending with '0/4 nodes are available: 1 Insufficient memory'. The
weights live on the GPU; host memory covers SGLang's own bookkeeping, so 8Gi is the
honest reservation. The 32Gi limit stays, so a burst still gets the room.
The installed InferenceService CRD declares annotations and labels directly under
spec.predictor, not under spec.predictor.podMetadata. Two manifests used podMetadata,
so every diff and apply failed with 'field not declared in schema' and the whole
Application could not reconcile.
Move the prometheus scrape annotations and the labels up one level. Nothing is lost:
the same keys, in the place the CRD accepts.
The resource list named ornith.yaml and reasoning.yaml, and neither file has ever been
committed to this directory. kustomize therefore failed to build, the Application sat
in ComparisonError, and nothing in this directory could reconcile — including the
clef-flash predictor added a commit earlier.
Removing the entries does not remove the models: the Application runs with prune=false,
so an object it no longer lists is left untouched. Re-export both from live state the
next time either is edited, so the directory describes what actually runs.
The gateway already dispatches /v1/systemone and its config already names
clef-flash-predictor, but this directory is an explicit allowlist and clef-flash.yaml
was not on it, so the model was silently dropped: no error, no drift, just no
predictor.
clef-flash is a decision model, not a chat model. It reads a state plus a schema of
typed questions and returns one probability per allowed option in a single prefill
pass, with no decode step. It needs SGLang rather than vLLM because it loads separate
joint_schema_head weights on top of the Qwen3.5-9B backbone and it implements the
/v1/systemone route.
worker-1 has four GPUs: reasoning holds two, ornith holds one, so one is free and this
fits without retiring ornith first.
The init container fetched a token once at pod start with a client_credentials grant
against a web app. A web app has no such grant, so the fetch could not work; and a
token fetched once is dead within the hour while the pod runs for weeks.
llm-token-vendor runs beside the app. It signs a machine-key assertion for the
paperless-llm service account, exchanges it for a token, and rewrites
/app/data/llm_token.txt atomically at 80% of the token lifetime. The vendor image
keeps using CUSTOM_API_KEY_FILE, so it needs no change and no OIDC support.
The key is re-read on every mint, so a rotation lands without a restart: publish the
new key, the next mint uses it, retire the old one.
paperless-llm is a separate service account from poimen-llm, so revoking one caller
leaves the other working.
Adds svc-caller, a capability that names a machine-to-machine caller, separate from
the human roles so a service account can be identified by its grant alone.
Adds MACHINE_CALLERS: one ZITADEL machine user per caller, created once with its own
key and granted its roles. poimen-llm is the first, granted llm:inference and
svc-caller. Its key is separate from every other caller's, so revoking one service
leaves the rest working and ZITADEL can overlap keys during a rotation.
The role is granted even though ZITADEL does not assert roles into a machine-user
access token on this instance. The grant is the record of what the account may do, and
it takes effect the moment that behaviour changes. Until then the gateway authorises
the caller from its allow-list entry.
storage/minio-oidc is owned by the KeyRotation object iam/minio now, so its value
leaves git. minio-creds and minio-user-ollama stay: they are the root console
credential and an application key, and neither rotates through ZITADEL yet.
The file keeps a header that says so, because a prune of the sops-secrets application
would delete that Secret and the controller recreates it on the next rotation.
Two defects that a real rotation exposed:
merge writeSecret replaced the whole Secret map, so every key a consumer
keeps beside the credential disappeared. paperless lost
SOCIALACCOUNT_PROVIDERS_JSON and immich lost config.json; both pods then
failed to start while the rotation reported success.
backoff a Failed attempt returned 0, so the controller retried in a tight loop.
It now waits two minutes and then retries.
All eight objects carry both identifiers ZITADEL needs: the app resource id for the
secret endpoint and the client id for introspection.
Result: seven rotations reach Succeeded with every consumer ready (grafana, comfyui,
argocd, forgejo, paperless, immich, minio). api-gw stays suspended because no workload
exists for it.
Three defects, each seen in a live rotation of grafana:
scope the access token needs urn:zitadel:iam:org:project:id:zitadel:aud,
otherwise the management API answers 401 Token.Invalid even though the
assertion was accepted.
app id the objects carry the client id, and the secret endpoint takes the app
resource id. They differ, so every call answered 404 Application
doesn't exist. The correct ids are recorded per object.
attempt time record ActiveCredentialID and LastRotation right after Prepare. Without
it a later failure leaves LastRotation empty, the due check says rotate
now, and each reconcile mints a new secret and restarts the consumer:
grafana was rolled continuously and never became ready.
Also raise the consumer-ready budget to 8 minutes, because grafana needs about four,
and retry a failed attempt after 2 minutes instead of waiting a whole interval.
Three faults stopped the controller from running, each found by running it:
image arch the first build was arm64 and every node is amd64, so the pod died
with 'exec format error'. imagePullPolicy IfNotPresent kept the
cached wrong-arch layer, so it is now Always.
leases leader election needs coordination.k8s.io/leases. Without it the
controller never acquired its lock and never reconciled.
secrets list the manager caches Secrets, so it needs list and watch, not only
get, create, update and patch.
go.mod and go.sum gained the procfs entry the build needs.
Forgejo reports zero workflows for this repository even though
.forgejo/workflows/key-rotation-controller.yaml is on main. The poimen-frontend
workflows that do run live under .gitea/workflows/, so this instance reads that path
only. Move the file there.
The frontend's callback path is /auth/callback. The old /callback spelling stays in
the list so an image built before this change still completes a login.
Dockerfile: static binary on distroless, no shell, user 65532.
Workflow runs on the golang runner for a push to main that touches the controller.
It builds two tags, the commit sha and 0.1.0, which is the tag the Deployment asks
for, and runs the image with --help so a broken binary fails the job instead of the
cluster.
The runners were dead before this: every task fetch logged 'lookup
forgejo-gitea-http.cicd.svc.cluster.local: no such host'. DNS resolved from a fresh
pod and the Service had a healthy endpoint, so the runner pods held stale network
state. A rollout restart cleared it and all three poll without error.
grafana-oidc and comfyui-oauth2-proxy held nothing but the rotated credential, so
both files are gone. oauth2-credentials survives with the two secrets that were not
rotated (api-gw, poimen) and drops the three that were (grafana, minio, paperless).
Leaving them would be worse than untidy: sops-secrets ran with selfHeal, so a stale
file silently rewrites a rotated secret back to a value ZITADEL no longer accepts.
32 files remain, down from 34.
Generates a new secret in ZITADEL, writes it into the Kubernetes Secret, rolls the
consumer and checks it comes back. This is the manual execution of a KeyRotation,
and the reference the controller must match.
Two traps it records: the app resource id differs from the client id, and the
secret endpoint is oidc_config/_generate_client_secret.
Eight objects cover grafana, argocd, comfyui, minio, paperless, immich, forgejo and
api-gw. Each names its ZITADEL app, the Secret it writes, the consumer to roll and
whether the consumer reloads hot or needs a restart.
All are suspended: the controller image does not exist yet, and an unsuspended
object would claim an intent that no process acts on.
The README records the retirement target for the 34 .enc.yaml files: six
root-of-trust secrets that cannot rotate themselves survive, everything else
becomes a KeyRotation and leaves git.
Gating /oauth2 with the app redirects /oauth2/start to itself and the browser
loops. ingress-nginx applies an ingress's annotations to the locations that
ingress defines, so the login path belongs on its own object without auth-url.
This split failed once before, and the split was not the reason: the app ingress
carried a configuration-snippet that the controller rejects, which dropped that
whole object and left only the login path rendered.
The controller logs "annotation group ConfigurationSnippet contains risky
annotation" and a rejected annotation group drops the entire ingress from
nginx.conf. comfyui.riotpiao.com therefore answered 404 for every request with no
line in the access log, which is why the host looked routed but served nothing.
ingress-nginx sets the Upgrade and Connection headers itself for a websocket
upgrade, so the snippet was never needed.
Two Ingresses on one host do not merge annotations. nginx applies the annotations
of a single ingress per host, so the separate /oauth2 object silently dropped the
auth_request on the app ingress: every request fell through to the backend and
ComfyUI answered 404 for the site root.
Move /oauth2 and / into the same ingress, so the auth annotations and both paths
belong to one object.
comfyui.riotpiao.com is served with no authentication at all: anyone who can
resolve the host reaches the UI and its queue. ComfyUI has no OIDC support, so the
front door is oauth2-proxy, and nginx asks it for a decision on every request.
Nothing inside the cluster calls this host, so the whole host is gated rather than
carved into open and closed paths. If a client ever needs direct access, add a skip
rule for its exact paths.
The ZITADEL app is created by the provisioning script, and the client secret,
cookie secret and client id live in a SOPS secret written by ksops.
Replace the dead secret-rotation scaffold (an Argo app that could not render, a
CRD that was never installed, and no controller code) with a working design.
One resource describes one credential:
spec.provider zitadel-machine-key | zitadel-oidc-client
spec.strategy overlap | atomic
spec.target Secret to write, consumers to roll, reload hot|restart
spec.verify introspection or token exchange
The state machine is shared by every provider:
Prepare -> Publish -> Verify -> Retire
Order is the safety property. Retire runs only for an overlapping provider and
only after the new credential obtained a token. A failure before that point leaves
the working credential in place, which is exactly what went wrong by hand today
when a regenerated Grafana secret was never re-pointed.
Machine keys overlap in ZITADEL, so that provider rotates with no gap. Client
secrets do not overlap, so that provider is atomic: it publishes, rolls the
consumer, verifies, and restores the previous value from a shadow key if the
introspection check fails.
The controller authenticates with its own machine key, the single bootstrap
credential, read from a mounted Secret. Everything else it creates itself, so no
credential value needs to live in git.
Verified: go build and go vet clean, 12 tests pass covering scheduling, the shadow
key rollback path, provider strategies and assertion signing.
Not yet wired to Argo CD: the image must be built and pushed first, so the app is
not added until a Deployment can actually pull.
The live cluster has no Authentik workload, no DNS record and no tunnel route,
but two declarations still pointed at it.
controlplane.tftpl still set kube-apiserver oidc-issuer-url to
https://authentik.riotpiao.com/application/o/kubernetes/ and pinned that hostname
in extraHostEntries. The live API server carries zero OIDC arguments, so the
template and reality had diverged: the next machine-config render would have
re-added an issuer that does not resolve.
management-service exported KMSVC_AUTHENTIK_ISSUER_URL and
KMSVC_AUTHENTIK_AUDIENCE, both empty. Rename them to KMSVC_ZITADEL_* so the names
describe the identity provider that actually exists.
The CoreDNS Corefile kept one rewrite for authentik.riotpiao.com. The Terraform
source file no longer contains it, so the live ConfigMap was stale; that line is
removed live and the next render converges without it.
Grafana sign-in failed with "Failed to get token from provider". The deployed
client secret no longer matched, because _ensure_oidc_apps called
_generate_client_secret for every app on every run, and ZITADEL stores exactly one
client secret per app: each call replaced the previous value and invalidated every
deployed copy.
Two changes:
1. Generate a secret only when the app is created. For an app that already exists,
reuse the value from state.json, because the API cannot read a secret back.
2. Re-encrypt grafana-oidc-secrets.enc.yaml with the live secret, so Grafana's
token exchange succeeds again.
Verified with the introspection endpoint:
old deployed secret -> {"error":"unauthorized_client"}
new secret -> {"active":false} (credentials accepted)
Grafana gains one contact point, `gotify`, of type webhook. It posts to the
bridge in the notifications namespace because Gotify rejects a Grafana payload
with 400 "Field 'message' is required".
maxAlerts 0 keeps every alert in one request, and the bridge names the extra
count in the message body instead of dropping alerts.
Gotify rejects a Grafana payload with 400 "Field 'message' is required", so a
translation step is mandatory. This adds a small standard-library Python
service: it accepts Grafana's JSON, builds a Gotify message, and posts it with an
app token.
Mapping:
firing -> priority 8, title "[FIRING] <alertname>"
resolved -> priority 3, title "[RESOLVED] <alertname>"
body -> the summary annotation plus severity, namespace, pod, instance, job
A multi-alert notification names the alert count instead of dropping the rest.
The script ships as a ConfigMap so no image build is needed. The app token comes
from a SOPS secret and belongs to Gotify application `grafana`.
The bridge runs server-side, so it cannot run a browser flow and ZITADEL has no
part in it. Give it a Gotify client token instead: GOTIFY_URLS points at the
public host, GOTIFY_CLIENT_TOKENS comes from a SOPS secret.
The token belongs to user `rock`, is named igotify-assist, and has
expiresAfterInactivitySeconds 0 so it never idles out. Delete the client row in
Gotify to revoke it.
Verified with the token: GET /client returns 200 and GET /message returns the
latest message.
The first value was a base32 string, so oauth2-proxy rejected it at startup:
cookie_secret must be 16, 24, or 32 bytes to create an AES cipher, but is 56
bytes.
Gotify has no OIDC support, so it cannot validate a ZITADEL token itself. Put
oauth2-proxy in front of it and let nginx ask the proxy for every request.
The Gotify REST API keeps its own client-token check. /message, /application,
/client, /user, /version, /stream and /health are skipped by the proxy, so
Forgejo webhooks, the harness, poimen-sender and the Android client keep working
without a browser session. Only the web UI requires SSO.
The /oauth2 path needs its own ingress: the auth_request annotation would
otherwise also guard /oauth2/start, and the login entry point would demand a
login first.
The client id and secret come from project role provisioning:
`zitadel-provision.py apply` now creates the `gotify` OIDC app, and the secret is
SOPS-encrypted under k8s/argocd/secrets.
Gotify's own login remains behind the proxy. ZITADEL decides who reaches the
service, Gotify decides what that person may do inside it.
forceMFA never took effect because of the JSON key. ZITADEL's v1 endpoint speaks
protojson, which accepts "forceMfa" (lowerCamel of force_mfa). The spelling
"forceMFA" is treated as an unknown field and silently dropped, so every earlier
run returned 200 with a new sequence while force_mfa stayed false in
projections.login_policies5.
Send forceMfa and forceMfaLocalOnly. Verified in the database:
force_mfa = t | force_mfa_local_only = t | mfa_init_skip_lifetime = 0
and in the login flow: after the password step the browser lands on
/ui/login/mfa/prompt with the "2-Factor Setup" page, so a second factor is now
required instead of optional.
rock