Merge origin/main into fix/secrets-placeholders (rebuild ksops file list, fix #54 indent)

fix(secrets): populate the empty sops placeholders and drop the unused one
(cherry picked from commit f6207ff2b0)
2026-10-06 10:05:45 +08:00 · 2026-10-06 10:03:04 +08:00 · 2026-10-06 01:49:12 +00:00 · 2026-10-06 01:48:57 +00:00 · 2026-09-30 11:49:52 +00:00 · 2026-09-30 19:49:05 +08:00
15 changed files with 410 additions and 64 deletions
@@ -28,9 +28,14 @@ spec:
      # KV cache math: 256KB/token total (64 layers, 8 KV heads, 128
      # head_dim, fp16), PP=2 splits both weights and KV load ~evenly, so
      # each GPU carries ~9.67GB weights + ~128KB/token KV. At
-      # gpu-memory-utilization=0.90 (28.8GB/GPU usable), that leaves
-      # ~19.1GB/GPU for KV cache -> ~156k tokens/GPU capacity, comfortably
-      # above the 131072 target with room to spare -- the old
+      # gpu-memory-utilization=0.95 (30.4GB/GPU usable), that leaves
+      # ~20.7GB/GPU for KV cache. NOTE this is a projection, not a
+      # measurement: the live reading at 0.90 is num_gpu_blocks=9527 x
+      # block_size=16 = 152,432 tokens (vllm:cache_config_info). Re-read that
+      # metric after the rollout -- max_num_batched_tokens=8192 also raises
+      # activation memory out of the same budget, so the post-boot
+      # num_gpu_blocks is the authoritative number, and --max-num-seqs=5 is
+      # only valid if it still covers 2x65k + 3x8k. The old
      # OffloadingConnector CPU-DRAM spillover (tuned for the previous
      # model's much smaller 16384 context) is no longer needed and is
      # dropped. Staying on PP=2 and vLLM 0.11.0 (no version bump needed,
@@ -54,10 +59,37 @@ spec:
      - --tensor-parallel-size=1
      - --pipeline-parallel-size=2
      - --max-model-len=131072
-      - --gpu-memory-utilization=0.90
-      - --max-num-seqs=4
+      - --gpu-memory-utilization=0.95
+      - --max-num-seqs=5
+      # Prefill scheduling -- see the measured starvation this fixes.
+      # V1's scheduler has no prefill/decode phase: it walks self.running and
+      # hands out tokens from one shared budget (vllm/v1/core/sched/scheduler.py:
+      # "There's no decoding phase nor prefill phase in the scheduler"). The cap
+      # per request is applied BEFORE the budget clamp:
+      #     if 0 < long_prefill_token_threshold < num_new_tokens: take = threshold
+      #     take = min(take, token_budget)
+      # The loop is `while ... and token_budget > 0`. With the threshold at its
+      # default of 0 the cap is disabled, so one long prompt takes the ENTIRE
+      # budget, token_budget hits 0, the loop exits and every other request is
+      # skipped for that step -- measured as 262 tok/s contended prefill and
+      # 0.0-0.1 tok/s generation for everything else, Running: 2-4 reqs.
+      # Hence threshold must be strictly BELOW the budget or it never binds:
+      # at 8192/8192 a single request is served per step (still starved), while
+      # 8192/2048 serves four. Verified by replaying the loop.
+      - --max-num-batched-tokens=8192
+      - --long-prefill-token-threshold=2048
      - --enable-chunked-prefill
      - --enable-prefix-caching
+      #
+      # NOT set, deliberately: --max-num-partial-prefills and
+      # --max-long-partial-prefills. Those are V0-only knobs. vLLM 0.11.0 ships
+      # V1 only (vllm/core/ absent), and EngineArgs._is_v1_supported_oracle
+      # rejects any value other than the default for either one, calling
+      # _raise_or_fallback("Concurrent Partial Prefill") -> "Falling back to V0"
+      # -> AssertionError, because there is no V0 engine to fall back to.
+      # The Deployment uses strategy Recreate, so setting them is an outage.
+      # V1 does concurrent chunked prefill natively; it needs no flag, only the
+      # budget/threshold ratio above.
      # qwen3 is vLLM's dedicated reasoning parser for this family's <think>
      # blocks.
      - --reasoning-parser=qwen3
@@ -0,0 +1,24 @@
+apiVersion: v1
+kind: Secret
+type: Opaque
+metadata:
+    name: forgejo-registry-token
+    namespace: cicd
+data:
+    username: ENC[AES256_GCM,data:oRNecs0a7rs=,iv:/5EFdNqpSjoxe7XDmAEDMd6lXd0SMsRiMmBM9SIwDHY=,tag:Wi+ggoP+9/vgz9rHrxB9nA==,type:str]
+    password: ENC[AES256_GCM,data:HbmwnWk3ST5SDWXwIhu+M5f/FOfFbJHaG0Q/IWrO63HRXmJTmLMpbjtWIXC9IMFoHeDZ1MNAV8A=,iv:ztJZgpaS//yYCiyr6/ubs/UiXkbGMnokvy4Pe45jQMk=,tag:WMpawc/SVAV4mxsCd6XqAA==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjQmhxN2E3ZnNTdkR6U2c1
+            ZStsQktkeWUwMEdmdFIyMkRYRVk1ZGppWlZ3CmUzR2JubWw2VjdWUGt6S2lpNmQ3
+            cDdSWjJiYzRIWS9UVERpVDFJenFnaDAKLS0tIGhDSmNMSjZwcDVPR3U0dUcwNk9Q
+            WmdGS1lyS3dkSXIvOFhyRklGRll2UlUK8NGts6a6Fv10c6gxIIPqIIqfgzQ3EEyk
+            SFrMThwbdaOqLY4AMeDrg+k0Y2t1Xj/XOvdNbRCRictDwZUsptvO7A==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-06T02:01:24Z"
+    mac: ENC[AES256_GCM,data:imebczVzG9YWd42ER4a+w+Bi3dxbN0HFm/1igxzJt6Z0HG/q6i0s59swpGVruk9dQJWrlO8KKqX739ATp2KGC5szrEuRFvPemF594XFCg6ykW+ZeP5VtTvpJg2nHv7/Pu+71ljKKi4gqOJ4SJDn8owAbSN+su6PiT323wax930A=,iv:iG/G2+qQkvbBU95w/GG2PrbAHDOaamuvjqMDBxSW6zI=,tag:wy9Vn72HBDHqCyzTcV15/Q==,type:str]
+    version: 3.13.2
@@ -0,0 +1,26 @@
+apiVersion: v1
+kind: Secret
+type: Opaque
+metadata:
+    name: memory-agent-oidc
+    namespace: poimen
+data:
+    CLIENT_ID: ENC[AES256_GCM,data:c/RvtV6anofX8h1BmlKu9w==,iv:3lQ1BR8kVvF1RHVfKstmb0Y7u/uX5OrvmGiCi/xTpjw=,tag:9JC2WkfQFWaiZroAEmPenw==,type:str]
+    CLIENT_SECRET: ENC[AES256_GCM,data:I258/6O43PccyllYN/wfoKvTgNgZONTivM/KFIr8TGqSl+2XQkwBXvK2VQc5Xgc5cpwlrgnjk8pjQbqr,iv:MZyA+V5z9Po2VoxqvOSlWh1fZp1lrkcbYMx7VcSIaJs=,tag:gXH0BpA8KOZiVd+4RnhRzg==,type:str]
+    ISSUER: ENC[AES256_GCM,data:GX/uQPDoeJWXXQZlzD6oOIvxo7H4LRAv4MrPAiCgmv6WxW59+KKkymuDT/T9MiDm4bNhxVPgB7nHbDcMHfPvU2zCNM2TWYUsNOgGwxNAqM8=,iv:5OEsdrUZvMLO2C8Ea2i5alsyFGtmEk3othTemk8fOk4=,tag:8CMlRY2JCe4z4fv4WuWjRQ==,type:str]
+    TOKEN_URL: ENC[AES256_GCM,data:KeU1pQzvQ2/1lQhOqNtcR7t5TGVx1J7NY8ogxCXS6IuHhVe0zWYIbC4BYFc2H9PAjsMJwCCg75N1FrIiCYBvpfK8ln0=,iv:NbPiJC/B7W1Zm+JwnEGDikE2cb3L4l6TBeQTKEb2boM=,tag:FRmllFLZzQW03lsCu9mAUA==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBmK0dPSmhKT01FSitDdUNW
+            Um1XQU9FVnBUeDdoWXpCcXRzVXh0VXkvNGxNClZaQ3R5U2NkYTJjaFdHRWhUVmFS
+            K0tHZVRFRDZGREVRZzZqcWJ6dnRpUUEKLS0tIE15UUNtMStKbWwzb3dEVGEwN1VQ
+            cktlVTM2SUhpOGJZMXJOOG84RnhCQzAKOdM0pThEaN37SJtRrKIivncMu8Z91Btq
+            es6NOlJdBUhzIhnEJWfDzZ9aA6I/fYzugKbC+AfUwdGgThAgHx9crg==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-06T02:01:24Z"
+    mac: ENC[AES256_GCM,data:/BTQ984D+hsk0LmnjD3hxopB65NAR54LhoAPNcATWA7MCTJXfdnYRu3gNxUzyiyCBd+rDaPp77TBnak4jVjMnOiYJsYLj1R7hrrhTZopz8pT+YNMupperwsirKZwK8it4Mu+fnpCyox2mtGED0hk+Nh/b6IteTRt5lS7noumo2M=,iv:RWIrVkNYj+8mqMSPiLueToexabzAtVbIR525m6bUrlg=,tag:5+wcNoNYsjGKpFlML+N70Q==,type:str]
+    version: 3.13.2
@@ -0,0 +1,27 @@
+apiVersion: v1
+kind: Secret
+type: Opaque
+metadata:
+    name: oauth2-credentials
+    namespace: iam
+data:
+    api-gw-client-secret: ENC[AES256_GCM,data:YiUCk61cCsTJE8gTLZWhqVPsCSu3b4OnG+Oh456A+mnxip0FjLTTqV+kYBdfZivIkn9ua4RcVW+rYILu,iv:K08m1EaB7/SIVNWZ3FfeRP4tbCAo6KeiNC0NpE9VO1o=,tag:ZXV9YRWUOPJ/vdoqMtvQeA==,type:str]
+    minio-client-secret: ENC[AES256_GCM,data:vcwLo1cJqbZr2MwkAKO09xPjb09ss7+s6HIvEfaLxt8ka5PUAlXQ37pTkynuktlKq/t0DvSk+x1+5G7QqVD1JJlwyDGM6yuOhBQ3MwucLNl+T4RDKtgGuA==,iv:sCdd+UNX2TfTAM6Zslrh8KrvBSzqpe67F3YMGrl4YDU=,tag:9Tpzm9NEXNuXyCd7l7GeHw==,type:str]
+    poimen-client-secret: ENC[AES256_GCM,data:dYtCiMGZaOFokihfGD2SXJDfXrhfwEQcTOPNu7JNHDUnx0hqwAdYAPAJTEN21p0OpHYDINMs5lwUpMwY,iv:ll8USuzs0881ryHImVjHGO0iFxUzqg5XtI6Ekd+EFXE=,tag:I/4ZfB9Ss4zEP57sRtRTXQ==,type:str]
+    paperless-client-secret: ENC[AES256_GCM,data:XMj9rHkI20lI37vChq9wS5lUq7l95SisRX0LChuj09O5w76Zt9NdG6OEUYQFf1z8vCfMJxBInIEEIq0e,iv:UpYy2ZEZ4AgBACEyrlxEcr13V+8Ksu4p8JHYApmyYHo=,tag:o9Z+/b6T9POgowIsks56kw==,type:str]
+    grafana-client-secret: ENC[AES256_GCM,data:V/j0+P3HoDLqYjRalWK4v1hvJTuVmzxVra47HsF1/BZvWZjIyEIH3Pt8BekT3O1dUFynxjcaxdZMBVYg,iv:9pMp+vIsp/sA7GP9wqNDf5DdZVLDow46OURW0mMRXIc=,tag:i7VJFA5o0hToZEJ5JPWIvw==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB2amZPendIOU9PeStOcmJG
+            c1dzOFVvd3RoT3dXQlJUNkdCSytEZUVGVEJJCklMUkNjUytBMXh0OVUrNUlpMWw3
+            aHRFT1hBZTZHNHVtaGJCTEFyT2ZWNWsKLS0tIENxR3FKMEplV0V5Y0dsZzBkODVG
+            U0ZlT2R0VFdEQmpsakVvZ1FJRkRQYTAKESQcSlMAuCG0/tLWWZaeoPayf1EKxSaf
+            e4o8TmjmSgPEhepQ/LmNXl6obyuqUxztkh4Mm6pocqDp+gfzt7X/1g==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-06T02:01:24Z"
+    mac: ENC[AES256_GCM,data:AI/znHiUa/Rwg0YRz65qUZ+b6V73oi3ZfAG/V5uZZvrGFrA0EISzD5LaP5Kl4GUU1svo4G5EI4aftPDgpdPptz21CCa8iYkXGe9pTDmtaE5ZICQAUbRcVhGMTfuWFhrgEhMfCJdOpfHaIQVEsQJQJgrnkXQVuUe9aT75LQz7pnY=,iv:KbIaYT0VeBT2WeSU5V5ZB4e+04JICYaY+egGN1Pv5bM=,tag:pPbhDxeLUx6V1XC/LPwxTg==,type:str]
+    version: 3.13.2
@@ -0,0 +1,24 @@
+apiVersion: v1
+kind: Secret
+type: Opaque
+metadata:
+    name: pi-code-agent-api-keys
+    namespace: poimen-sandbox
+stringData:
+    ANTHROPIC_API_KEY: ENC[AES256_GCM,data:TBcno/dzj3qykHjeX96UKZekuA/WMQT2E7a1ziPyYo5J6lWLPydgli84KeJZhH62/8CnnFtFOhVIe4FWCGPGjvqKH6M8WbiQHaP9y7FN0qQUpFHKNVAQ2R+s0wInhw0eUmANvd7bRx8aIZ2X,iv:oex19yVtWUevoJ0A3mkMLCfWu85o1gPSNp43x1iV0LE=,tag:BW3zQnwvagPceeNird9CZg==,type:str]
+    DEEPSEEK_API_KEY: ENC[AES256_GCM,data:vq/5gOsMVjlVMd0oXiBtHFHFEr+IF2YR2iP2OBFWx3ZFNoQ=,iv:QhLU/BRIJb/7QKDCRyw+HWPphZifVEhIOLywYDgOb20=,tag:BZdtXw+WsuK+P4E0caTa0Q==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBqNm1LNE5nWktnd3c3STg3
+            UlVWbWJxWElxUHMwa0JVMk81Z0c4d0JaQm1vCnpLQWJWV3hxOU00NzJIaG51Mmlr
+            ZWpNTFJWcmdtV0w4YndQeHpvSUx2MTAKLS0tIDVnNkNuVWhPUFZ5V2hFOEt1eitG
+            blVqRy8yNnYzSU1FNzZGNkxrRElvWVEKL3J77INje75+CvKXaJHC3CvONgHv7d0a
+            a10MITvfreFr6M4AN3Z/mno49SDMp15Q4oQRagHHxSTS1K54mWHo8g==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-10-06T01:19:49Z"
+    mac: ENC[AES256_GCM,data:GpYl8IRefms36GvO3RW41c0US7SOgoOhd4rumxsCdEf5ME9kJ0eOxPx8klMgnlYrkBnwHmad8jj7C+HcA5Eim7mdUaYOZZWPFohvC6N4GLHX85aH78P09CelqG5tdm/Gm+1QfxYu0vANYGCAVjLK7YbBbZjbZmbvVfQvAbs3mOY=,iv:y3lglGu8FhTHQpO1poSIKgGYJFu5J9yjIaOjulda6fY=,tag:5Sgh4TbtNFMw7xOJjYRn0g==,type:str]
+    version: 3.13.2
@@ -7,29 +7,33 @@ metadata:
      exec:
        path: ksops
 files:
-  - agent-pod-models.enc.yaml
-  - agent-pod-ssh-key.enc.yaml
-  - authentik-secrets.enc.yaml
-  - cloudflare-secrets.enc.yaml
-  - forgejo-registry-pat.enc.yaml
-  - forgejo-registry-pat-api.enc.yaml
-  - forgejo-registry-pull.enc.yaml
-  - forgejo-runner-token.enc.yaml
-  - forgejo-secrets.enc.yaml
-  - grafana-oidc-secrets.enc.yaml
-  - grafana-secrets.enc.yaml
-  - homarr-auth-oidc.enc.yaml
-  - homarr-db-encryption.enc.yaml
-  - homarr-secrets.enc.yaml
-  - homelab-ca-secrets.enc.yaml
-  - loki-secrets.enc.yaml
-  - minio-secrets.enc.yaml
-  - paperless-secrets.enc.yaml
-  - vault-secrets.enc.yaml
-  - vault-unseal-keys.enc.yaml
-  - portfolio-secrets.enc.yaml
-  - gotify-admin-secrets.enc.yaml
-  - gotify-tokens-secrets.enc.yaml
-  - gotify-smtp-secrets.enc.yaml
-  - forgejo-smtp-secrets.enc.yaml
-  - paperless-ai-secrets.enc.yaml
+    - agent-pod-models.enc.yaml
+    - agent-pod-ssh-key.enc.yaml
+    - authentik-secrets.enc.yaml
+    - cloudflare-secrets.enc.yaml
+    - forgejo-registry-pat.enc.yaml
+    - forgejo-registry-pat-api.enc.yaml
+    - forgejo-registry-pull.enc.yaml
+    - forgejo-runner-token.enc.yaml
+    - forgejo-secrets.enc.yaml
+    - grafana-oidc-secrets.enc.yaml
+    - grafana-secrets.enc.yaml
+    - homarr-auth-oidc.enc.yaml
+    - homarr-db-encryption.enc.yaml
+    - homarr-secrets.enc.yaml
+    - homelab-ca-secrets.enc.yaml
+    - loki-secrets.enc.yaml
+    - minio-secrets.enc.yaml
+    - paperless-secrets.enc.yaml
+    - vault-secrets.enc.yaml
+    - vault-unseal-keys.enc.yaml
+    - portfolio-secrets.enc.yaml
+    - gotify-admin-secrets.enc.yaml
+    - gotify-tokens-secrets.enc.yaml
+    - gotify-smtp-secrets.enc.yaml
+    - forgejo-smtp-secrets.enc.yaml
+    - paperless-ai-secrets.enc.yaml
+    - memory-agent-oidc.enc.yaml
+    - oauth2-credentials.enc.yaml
+    - forgejo-registry-token.enc.yaml
+    - pi-sandbox-secrets.enc.yaml
@@ -11,7 +11,12 @@ metadata:
    prometheus.io/port: "9000"
    prometheus.io/scrape: "true"
 spec:
-  image: minio/minio:RELEASE.2025-07-23T15-54-02Z
+  # Was minio/minio:RELEASE.2025-07-23T15-54-02Z — that Docker Hub repository no longer
+  # exists (404 for every tag), so the tenant sat in ImagePullBackOff and Vault's S3
+  # backend could not resolve minio-cluster-hl. Official images now need registry auth
+  # (quay.io/minio/minio returns 401), so this pins a public mirror of the same upstream
+  # release; swap it for the official image once MinIO registry credentials exist.
+  image: coollabsio/minio:RELEASE.2025-10-15T17-29-55Z

  # Disable operator auto-TLS: MinIO serves plain HTTP internally on 9000.
  # External TLS is terminated at nginx ingress (wildcard riotpiao-com-tls cert).
@@ -158,3 +158,66 @@ spec:
            severity: warning
          annotations:
            summary: "Gateway pod {{ $labels.pod }} memory at {{ $value | humanizePercentage }} of limit"
+
+    - name: api-gateway.admission
+      # The admission gate fails OPEN when its view of the engine gauges goes
+      # stale, so a broken observer degrades silently to the pre-#64 behaviour
+      # of admitting everything. These two series are the only detection —
+      # the gate has no automated tests.
+      rules:
+        # The gate fails OPEN when it cannot read the engine, so "no data" must
+        # alert: otherwise a missing emitter, a renamed metric, or a broken
+        # scrape is indistinguishable from a healthy observer. Both expressions
+        # therefore OR in an absent() guard on the metric itself.
+        - alert: APIGatewayObserverDegraded
+          expr: observer_degraded == 1 or absent(observer_degraded)
+          for: 2m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Gateway admission observer degraded or not reporting — gate is failing open"
+            description: "observer_degraded is either 1 or absent. Absent means the emitter is missing, renamed, or unscraped, which is the fail-open case."
+
+        # Engine gauges went stale, or are not being reported at all
+        - alert: GatewayEngineStateStale
+          expr: increase(engine_state_stale_total[15m]) > 0 or absent(engine_state_stale_total)
+          for: 2m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Gateway engine state stale or not reporting — admission may be based on stale gauges"
+            description: "Non-zero means the observer lost the engine; absent means the metric is not being emitted at all."
+
+        # The acceptance metric for reasoning.yaml's prefill change. Preemption
+        # of a 65k prompt costs a full re-prefill (~250s at the measured
+        # 262 tok/s), so any sustained rate is a regression, not noise.
+        - alert: LLMPreemptions
+          # Guarded absent() is deliberately NOT used bare here: the engine
+          # restarts (strategy Recreate) and the series legitimately vanishes
+          # for the length of a boot, so a bare absent() would false-fire on
+          # every rollout. It only fires when the metric is missing WHILE the
+          # predictor is up, which is the real "renamed or unscraped" case.
+          expr: |
+            increase(vllm:num_preemptions_total[15m]) > 0
+            or (absent(vllm:num_preemptions_total) and on() up{job="reasoning-predictor"} == 1)
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "vLLM preempting requests ({{ $value }} in 15m) — KV cache over-committed, re-prefill cost is severe"
+            description: "Any sustained preemption rate is a regression for the reasoning prefill change: evicting a 65k-token prompt costs a full re-prefill (~250s at the measured 262 tok/s). Orchestrator verified `up{job=\"reasoning-predictor\"}` is the live job label — if that label is wrong the guard is inert but harmless."
+
+        # This change trades single-request latency for fairness: capping a long
+        # prompt at 2048 tokens/step means it takes more steps to finish, so a
+        # lone large prompt can now be slower even while preemptions stay 0.
+        # Without this alert the change has no way to show a regression.
+        - alert: LLMPrefillStarvation
+          expr: |
+            (sum(vllm:num_requests_waiting) > 0 and max(vllm:kv_cache_usage_perc) < 0.5)
+            or histogram_quantile(0.95, sum(rate(vllm:time_to_first_token_seconds_bucket[10m])) by (le)) > 60
+          for: 10m
+          labels:
+            severity: warning
+          annotations:
+            summary: "reasoning queueing or TTFT p95 above 60s while KV is under 50% — scheduling, not capacity"
+            description: "Queue depth with spare KV means requests are being starved rather than blocked on memory; a high TTFT p95 with headroom points at the same. Both series verified live (vllm:num_requests_waiting and vllm:time_to_first_token_seconds exist)."
@@ -26,20 +26,23 @@ def stat_panel(id, title, expr, x, y, w=4, h=4, unit="short", mappings=None, thr
    return p


-def ts_panel(id, title, exprs, x, y, w=8, h=8, unit="short"):
+def ts_panel(id, title, exprs, x, y, w=8, h=8, unit="short", description=None):
    targets = []
    for e in exprs:
        if isinstance(e, tuple):
            targets.append({"expr": e[0], "legendFormat": e[1]})
        else:
            targets.append({"expr": e, "legendFormat": "{{pod}}"})
-    return {
+    p = {
        "id": id, "title": title, "type": "timeseries",
        "gridPos": {"h": h, "w": w, "x": x, "y": y},
        "datasource": DS_PROM,
        "fieldConfig": {"defaults": {"unit": unit}},
        "targets": targets,
    }
+    if description:
+        p["description"] = description
+    return p


 def table_panel(id, title, expr, x, y, w=12, h=8):
@@ -274,23 +277,29 @@ def build_api_gateway():
 #   vLLM + TEI (llm-serving)  — engine-truth, but only for the three engines
 #                               that expose /metrics. Ollama (ornith:35b,
 #                               qwen2.5:3b-instruct) exposes nothing at all.
-#   api-gateway (/metrics)    — intended to cover EVERY model including the
-#                               Ollama ones, but see the dead-metrics note
-#                               below: its latency series are not recorded.
+#   api-gateway (/metrics)    — covers EVERY model including the two Ollama
+#                               ones, derived from the SSE byte stream: TTFT is
+#                               request-start to first byte, ITL is the gap
+#                               between SSE writes carrying "data: ", so it
+#                               counts SSE chunks rather than true tokens.
+#                               vLLM's own histograms remain authoritative for
+#                               `reasoning`.
 #
-# NOTE — llm_ttft_seconds, llm_itl_seconds and llm_tokens_total are DEAD as of
-# 2026-09-29. The only recorder is LLMMetricsCapture
-# (internal/proxy/llm_metrics.go), and NewLLMMetricsCapture() has ZERO call
-# sites in the whole repo, so the exporter emits HELP-only stubs and any panel
-# querying them stays empty forever. Verified by grep, not assumed.
+# These three series ARE live as of 2026-09-29 — 3 series each, one per replica,
+# verified against the running gateway and Prometheus:
+#   llm_ttft_seconds, llm_itl_seconds, llm_tokens_total
 #
-# The gateway latency panels are kept deliberately so the gap is visible — they
-# light up unchanged once the capture wrapper is wired into the streaming proxy
-# path. Until then vLLM's own histograms are the latency source for `reasoning`,
-# and the two Ollama models have no latency visibility from anywhere.
+# History, recorded so nobody re-derives it the hard way: on 2026-09-29 these
+# recorded nothing at all. internal/proxy/llm_metrics.go declared a capture
+# wrapper that was never constructed, so the exporter emitted HELP-only stubs.
+# homelab-frontend#63 fixed exactly that — the wrapper is now
+# ResponseWriterWrapper, constructed by NewResponseWriterWrapper
+# (internal/proxy/llm_metrics.go:38) and installed on the streaming proxy path.
+# The "dead code" note that used to live in this header is superseded.
 #
 # Panels backed by queue_full_requests_total, warmup_requests_delayed_total and
-# vllm_prefix_cache_hit_tokens_total DO work today (3 series, one per replica).
+# vllm_prefix_cache_hit_tokens_total also work (3 series, one per replica).
+#
 #
 # All metric names below were verified against a live scrape, not assumed:
 #   vLLM 0.11.0 exposes prefix_cache_hits_total / prefix_cache_queries_total,
@@ -436,6 +445,48 @@ def build_llm_observability():
                 16, 15, w=8, h=8, unit="s"),
    ]))

+    # ── Admission control ───────────────────────────────────────────────────
+    # These four panels read the gateway's admission gate, which is the
+    # companion change in homelab-frontend#64. That emitter is NOT deployed
+    # yet (verified against the live /metrics endpoint: 0 series matching
+    # ^(admission_|observer_|engine_state_)), so they render "No data" until
+    # #64 ships. That is expected, and each panel carries a description
+    # saying so — an unexplained blank panel would read as a broken
+    # dashboard, which is exactly the failure mode this row exists to catch.
+    panels.append(row(160, "Admission Control", 16, [
+        ts_panel(161, "Admission parked (5m)",
+                 [("increase(admission_parked_total[5m])", "parked")],
+                 0, 17, w=8, h=8, unit="short",
+                 description="Requests the gateway held back rather than admitting. "
+                             "Emitted by the admission gate (homelab-frontend#64); "
+                             "empty until that change is deployed."),
+        ts_panel(162, "Admission rejected by reason",
+                 [("sum by (reason) (increase(admission_rejected_total[5m]))", "{{reason}}")],
+                 8, 17, w=8, h=8, unit="short",
+                 description="Admission rejections split by the gate's own reason label. "
+                             "Emitted by homelab-frontend#64; empty until deployed."),
+        ts_panel(163, "Engine state stale",
+                 [("increase(engine_state_stale_total[5m])", "stale")],
+                 16, 17, w=8, h=8, unit="short",
+                 description="Times the gateway's view of the engine gauges went stale. "
+                             "The gate fails OPEN when this happens, so any non-zero rate "
+                             "means admission is silently degraded to admit-everything. "
+                             "Emitted by homelab-frontend#64; empty until deployed."),
+        ts_panel(164, "Gateway view vs engine truth",
+                 [("max(vllm_requests_running)", "gateway: running"),
+                  ("sum(vllm:num_requests_running)", "engine: running"),
+                  ("max(vllm_kv_usage_perc)", "gateway: kv (fraction)  [same as engine if healthy]"),
+                  ("max(vllm:kv_cache_usage_perc)", "engine: kv (fraction)")],
+                 0, 18, w=24, h=8, unit="short",
+                 description="The gateway's own reading of the engine against the engine's "
+                             "self-reported value. A disagreement between a pair is how a "
+                             "stale observer or a mistuned admission target shows itself, "
+                             "and nothing else in this dashboard reveals it. The two "
+                             "gateway series (vllm_requests_running, vllm_kv_usage_perc) "
+                             "ship with homelab-frontend#64 and are absent until it is "
+                             "deployed — until then this panel shows the engine's pair only."),
+    ]))
+
    return {
        "title": "LLM Observability",
        "uid": "llm-observability",
@@ -64,7 +64,27 @@ data:
    / rerank latency p95","type":"timeseries","gridPos":{"h":8,"w":8,"x":16,"y":15},"datasource":{"type":"prometheus","uid":"prometheus"},"fieldConfig":{"defaults":{"unit":"s"}},"targets":[{"expr":"histogram_quantile(0.95,
    sum(rate(te_embed_duration_bucket[5m])) by (le, app))","legendFormat":"{{app}}
    embed"},{"expr":"histogram_quantile(0.95, sum(rate(te_embed_queue_duration_bucket[5m]))
-    by (le, app))","legendFormat":"{{app}} queue"}]}]}]}'
+    by (le, app))","legendFormat":"{{app}} queue"}]}]},{"id":160,"title":"Admission
+    Control","type":"row","collapsed":true,"gridPos":{"h":1,"w":24,"x":0,"y":16},"panels":[{"id":161,"title":"Admission
+    parked (5m)","type":"timeseries","gridPos":{"h":8,"w":8,"x":0,"y":17},"datasource":{"type":"prometheus","uid":"prometheus"},"fieldConfig":{"defaults":{"unit":"short"}},"targets":[{"expr":"increase(admission_parked_total[5m])","legendFormat":"parked"}],"description":"Requests
+    the gateway held back rather than admitting. Emitted by the admission gate (homelab-frontend#64);
+    empty until that change is deployed."},{"id":162,"title":"Admission rejected by
+    reason","type":"timeseries","gridPos":{"h":8,"w":8,"x":8,"y":17},"datasource":{"type":"prometheus","uid":"prometheus"},"fieldConfig":{"defaults":{"unit":"short"}},"targets":[{"expr":"sum
+    by (reason) (increase(admission_rejected_total[5m]))","legendFormat":"{{reason}}"}],"description":"Admission
+    rejections split by the gate''s own reason label. Emitted by homelab-frontend#64;
+    empty until deployed."},{"id":163,"title":"Engine state stale","type":"timeseries","gridPos":{"h":8,"w":8,"x":16,"y":17},"datasource":{"type":"prometheus","uid":"prometheus"},"fieldConfig":{"defaults":{"unit":"short"}},"targets":[{"expr":"increase(engine_state_stale_total[5m])","legendFormat":"stale"}],"description":"Times
+    the gateway''s view of the engine gauges went stale. The gate fails OPEN when
+    this happens, so any non-zero rate means admission is silently degraded to admit-everything.
+    Emitted by homelab-frontend#64; empty until deployed."},{"id":164,"title":"Gateway
+    view vs engine truth","type":"timeseries","gridPos":{"h":8,"w":24,"x":0,"y":18},"datasource":{"type":"prometheus","uid":"prometheus"},"fieldConfig":{"defaults":{"unit":"short"}},"targets":[{"expr":"max(vllm_requests_running)","legendFormat":"gateway:
+    running"},{"expr":"sum(vllm:num_requests_running)","legendFormat":"engine: running"},{"expr":"max(vllm_kv_usage_perc)","legendFormat":"gateway:
+    kv (fraction)  [same as engine if healthy]"},{"expr":"max(vllm:kv_cache_usage_perc)","legendFormat":"engine:
+    kv (fraction)"}],"description":"The gateway''s own reading of the engine against
+    the engine''s self-reported value. A disagreement between a pair is how a stale
+    observer or a mistuned admission target shows itself, and nothing else in this
+    dashboard reveals it. The two gateway series (vllm_requests_running, vllm_kv_usage_perc)
+    ship with homelab-frontend#64 and are absent until it is deployed \u2014 until
+    then this panel shows the engine''s pair only."}]}]}'
 kind: ConfigMap
 metadata:
  annotations:
@@ -1,25 +1,36 @@
-# WARNING — three of the series documented below are DEAD CODE as of
-# 2026-09-29 and will never produce data:
+# History, recorded so nobody re-derives it the hard way: on 2026-09-29 the
+# three latency series below recorded nothing. internal/proxy/llm_metrics.go
+# declared a capture wrapper (LLMMetricsCapture) that was never constructed
+# anywhere, so the exporter emitted HELP-only stubs and the dashboard panels
+# stayed empty.
 #
-#   llm_ttft_seconds, llm_itl_seconds, llm_tokens_total
+# homelab-frontend#63 fixed exactly that. The wrapper is now
+# ResponseWriterWrapper, constructed by NewResponseWriterWrapper
+# (internal/proxy/llm_metrics.go:38) and installed on the streaming proxy
+# request path. Verified live on 2026-09-29:
+#   llm_ttft_seconds  count=7  sum=598.212s
+#   llm_itl_seconds   count=16116
+#   llm_tokens_total  3 series
 #
-# They are declared and exported in internal/observability/prometheus.go, but
-# the only thing that would ever record samples is LLMMetricsCapture
-# (internal/proxy/llm_metrics.go) — and NewLLMMetricsCapture() has ZERO call
-# sites. Verified: `grep -rn NewLLMMetricsCapture` returns only the definition,
-# and RecordTTFT/RecordITL/RecordTokens are called from nowhere else. The
-# exporter skips keys with no samples, so these render as HELP-only stubs.
+# These derived-from-SSE series cover EVERY model including the two Ollama ones
+# (ornith:35b, qwen2.5:3b-instruct), which expose no metrics endpoint at all.
+# They are proxy-side: TTFT is request-start to first byte and ITL is the gap
+# between SSE writes, so both are granular to SSE write boundaries rather than
+# to the engine's own token cadence. llm_tokens_total counts `data:` events,
+# and the documented assumption in internal/proxy/llm_metrics.go is that these
+# upstreams emit one such event per token — so it is a token count only as far
+# as that assumption holds. vLLM's own histograms remain authoritative.
 #
-# The gateway latency panels on the llm-observability dashboard depend on that
-# wiring. Until it lands, use vLLM's own histograms as the latency source for
-# `reasoning`; the two Ollama models (ornith:35b, qwen2.5:3b-instruct) have no
-# latency visibility at all, because Ollama exposes no metrics endpoint either.
-#
-# These three ARE recorded and do work (3 series, one per replica):
+# Also recorded and working (3 series, one per replica):
 #   queue_full_requests_total           counter    backpressure rejects
 #   warmup_requests_delayed_total       counter    vLLM slot contention
 #   vllm_prefix_cache_hit_tokens_total  counter    prefix-cache hits
-#   request_duration_seconds            histogram  full request
+#
+# NOT working, do not build on it: request_duration_seconds is declared and
+# exported but has 0 samples, and its only writer Metrics.RecordRequest has
+# zero call sites outside _test.go — verified by ast-grep, not assumed. It is
+# the same dead-code shape this branch already corrected for the latency
+# series, so it is listed here rather than left to be rediscovered.
 #
 # Also note the Deployment already carried prometheus.io/scrape annotations,
 # but kube-prometheus-stack ignores pod annotations (no additionalScrapeConfigs,
@@ -9,3 +9,4 @@ resources:
  - poimen-sandbox-namespace.yaml
  - poimen-sandbox-quota.yaml
  - pi-session-netpol.yaml
+  - pi-agent-workspace-pv.yaml
@@ -0,0 +1,57 @@
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  finalizers:
+  - kubernetes.io/pv-protection
+  labels:
+    app: pi-coding-agent-sandbox
+    part-of: poimen-toolkit
+  name: pi-agent-workspace-pv
+  annotations:
+    argocd.argoproj.io/sync-wave: '-1'
+spec:
+  accessModes:
+  - ReadWriteOnce
+  capacity:
+    storage: 10Gi
+  claimRef:
+    apiVersion: v1
+    kind: PersistentVolumeClaim
+    name: pi-agent-workspace
+    namespace: poimen-sandbox
+  hostPath:
+    path: /var/lib/pi-agent-sandbox
+    type: DirectoryOrCreate
+  nodeAffinity:
+    required:
+      nodeSelectorTerms:
+      - matchExpressions:
+        - key: kubernetes.io/hostname
+          operator: In
+          values:
+          - talos-cp-2
+  persistentVolumeReclaimPolicy: Retain
+  volumeMode: Filesystem
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  annotations:
+    pv.kubernetes.io/bind-completed: 'yes'
+    argocd.argoproj.io/sync-wave: '-1'
+  finalizers:
+  - kubernetes.io/pvc-protection
+  labels:
+    app: pi-coding-agent-sandbox
+    part-of: poimen-toolkit
+  name: pi-agent-workspace
+  namespace: poimen-sandbox
+spec:
+  accessModes:
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 10Gi
+  storageClassName: ''
+  volumeMode: Filesystem
+  volumeName: pi-agent-workspace-pv
@@ -39,6 +39,7 @@
    rewrite name comfy.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    rewrite name comfyui.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    rewrite name gotify.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
+    rewrite name pi.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    rewrite name igotify.riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
    rewrite name riotpiao.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
Author	SHA1	Message	Date
rock	f780e31f9c	Merge origin/main into fix/secrets-placeholders (rebuild ksops file list, fix #54 indent)	2026-10-06 10:05:45 +08:00
rock	c93a24f786	fix(secrets): populate the empty sops placeholders and drop the unused one (cherry picked from commit `f6207ff2b0`)	2026-10-06 10:03:04 +08:00
rock	b6e317b34c	feat(poimen): pi sandbox workspace volume, encrypted api keys, cluster dns rewrite (#54 ) Adds the pi coding-agent sandbox's cluster side: - `k8s/infra/poimen/pi-agent-workspace-pv.yaml` — static hostPath PV + PVC for the sandbox /workspace (Longhorn cannot attach new volumes in this cluster; this gives real persistence and survived a pod delete/recreate test: marker file + 38 KB session.jsonl intact, the session answered from its own history afterwards). - `k8s/argocd/secrets/pi-sandbox-secrets.enc.yaml` — the sandbox's API-key Secret, SOPS-encrypted (recipient from .sops.yaml, payload only: `encrypted_regex ^(data\|stringData)$`), registered in the ksops generator. Round-trip verified against the live Secret. - `terraform/files/coredns/Corefile` — `rewrite name pi.riotpiao.com ingress-nginx-controller...` so the hostname resolves in-cluster. Note: the keys currently in that Secret are the ones leaked earlier and must be rotated; re-encrypt with `sops -e -i` after rotating.Reviewed-on: #54 Co-authored-by: rock <[email protected]>	2026-10-06 01:49:12 +00:00
rock	f3015522e3	fix(storage): pin minio tenant to a published image release (#53 ) ## Why `minio/minio:RELEASE.2025-07-23T15-54-02Z` no longer exists. The Docker Hub repository `minio/minio` returns 404 for every tag (checked: the pinned tag and `latest`), and MinIO has moved its images behind registry authentication (`quay.io/minio/minio` → 401, `registry.min.io/minio/minio` → auth required, `dl.min.io` binaries → 410 Gone). The tenant therefore sat in `ImagePullBackOff` — 44,229 attempts over 6d23h — and the chain from there is: ``` minio tenant pod not ready (1/2) -> headless Service minio-cluster-hl has no endpoints -> CoreDNS: lookup minio-cluster-hl.storage.svc.cluster.local -> no such host -> Vault S3 storage init fails at startup -> vault-0 CrashLoopBackOff, 1943 restarts, ~7 days ``` ## Control checks (so this is not a network story) \| check \| result \| \| --- \| --- \| \| `alpine:3.20` pulled on talos-cp-1 \| Succeeded — docker.io works \| \| `quay.io/prometheus/node-exporter:v1.8.2` on the same node \| Succeeded — public quay works \| \| `minio/minio` (docker.io) \| HTTP 404, repo removed \| \| old tag `RELEASE.2024-06-13T20-48-48Z` with `IfNotPresent` \| not cached on the node \| ## Change Pin `spec.image` to `coollabsio/minio:RELEASE.2025-10-15T17-29-55Z` — a public mirror, and a newer upstream release than the previous pin. Verified by pulling it on talos-cp-1 and running it: the image carries `/usr/bin/docker-entrypoint.sh` (the layout the operator expects, which runs `minio server … --console-address :9090`) and reports `minio version RELEASE.2025-10-15T17-29-55Z`. ## Caveat This is a community mirror, not MinIO's official image, because the official ones now require credentials. If MinIO registry credentials are available, add an imagePullSecret and switch back to `quay.io/minio/minio`. Supply-chain decision, flagged deliberately. ## After merge (ArgoCD, not kubectl) The storage app tracks `k8s/infra/minio`, so the sync rolls the tenant; then: ``` kubectl get pods -n storage # minio-cluster-az-a-0 -> 2/2 kubectl get pods -n iam \| grep vault # vault-0 -> Running (storage init succeeds) ``` Residual risk, unrelated to this fix: the tenant's 4 x 25Gi volumes are Longhorn-backed, and Longhorn currently fails to create engines for new volume attaches cluster-wide. If the restarted pod cannot attach, that is a second, separate fault.Reviewed-on: #53 Co-authored-by: rock <[email protected]>	2026-10-06 01:48:57 +00:00
rock	d60b520968	Merge pull request 'fix(llm-serving): V1 prefill scheduling + admission observability (#51 )' (#52 ) from harness/51-vllm-prefill-scheduling into main	2026-09-30 11:49:52 +00:00
rock	c7d52b5971	fix(llm-serving): V1 prefill scheduling + admission-control observability (#51 ) The reasoning engine starves every other request while one long prompt prefills. Measured: 262 tok/s contended prefill, ~250s for a 65k prompt, with Avg generation throughput 0.0-0.1 tok/s at Running: 2-4 reqs. Root cause is NOT what the issue first diagnosed. The issue proposed --max-num-partial-prefills / --max-long-partial-prefills, which are V0-only knobs. Verified in the live container: vLLM 0.11.0 ships V1 only (vllm/core/ is absent, so there is no V0 to fall back to), and EngineArgs. _is_v1_supported_oracle returns False whenever either flag differs from its default, calling _raise_or_fallback("Concurrent Partial Prefill") -> "Falling back to V0" -> AssertionError. With strategy Recreate that is an outage, not a slow rollout. Both flags are therefore NOT set, and a comment records why so they are not reintroduced. V1 needs no such flag: it has no prefill/decode phase at all. Every step it walks self.running and hands out tokens from one shared budget (vllm/v1/core/sched/scheduler.py). The real defect is that the per-request cap is disabled by default (long_prefill_token_threshold=0), so one long prompt takes the WHOLE budget, token_budget hits 0, the `while ... and token_budget > 0` loop exits, and every other request is skipped. The cap is applied before the budget clamp, so threshold >= budget never binds. Replaying the loop: 8192/8192 serves 1 request per step (still starved); 8192/2048 serves 4, and two long prompts advance together, which is the issue's actual acceptance criterion. --max-num-batched-tokens=8192 (was unset -> default 2048) --long-prefill-token-threshold=2048 (was unset -> 0, cap disabled) --max-num-seqs=5, --gpu-memory-utilization=0.95 The ~166k KV figure is marked as a projection; the live reading is num_gpu_blocks=9527 x block_size=16 = 152,432 tokens and is authoritative post-boot. A preemption alert covers the accept-and-watch risk. Observability for the companion admission gate (homelab-frontend#64, now deployed at 64f2bfc1): - four dashboard panels: parked, rejected by reason, engine state stale, and gateway-view vs engine-truth. Each carries a description, because an unexplained blank panel reads as a broken dashboard. - alerts on observer_degraded and engine_state_stale_total, both with an absent() guard: the gate fails OPEN, so a missing or renamed metric must alert rather than look healthy. Verified live that the guards do not false-fire (both series present, 3 each). - LLMPreemptions and LLMPrefillStarvation: the acceptance metric and the regression direction this change introduces. - max(vllm_requests_running), not sum(): one engine-global gauge is read by all three gateway replicas, so sum() read 3x and could never agree. Also corrects comments that were factually wrong. An earlier revision of this branch claimed llm_ttft_seconds/llm_itl_seconds/llm_tokens_total were dead code; homelab-frontend#63 wired them, verified live as 3 series each. request_duration_seconds is the genuinely dead one (0 samples, zero non-test call sites) and is now marked as such instead of being listed as working. Closes #51	2026-09-30 19:49:05 +08:00