From a2344515487061480ebf90eefe5bec7a324d826c Mon Sep 17 00:00:00 2001 From: rock Date: Sat, 31 Oct 2026 14:58:39 -0700 Subject: [PATCH] fix(llm-serving): allow poimen namespace ingress so memory ingest works poimen-memory could not reach any inference pod. The egress side was already correct: NetworkPolicy poimen/poimen-memory-llm-egress permits port 8080 to the pods labelled llm-reasoning, llm-embeddings and llm-reranker. This policy had no matching ingress rule, so Cilium allowed the packet out and dropped it at the destination. The observable was a bare timeout. The request never reached vLLM, so the predictor log stayed empty, and /health still answered because a probe from inside the pod does not cross the policy. That made the failure look like a hung inference engine. Evidence, measured on 2026-10-31 against the running predictors: 1. A probe pod in llm-serving WITH label part-of=llm-serving: /v1/chat/completions -> 200 in 3.1s, 8 completion tokens. 2. The same probe pod WITHOUT that label: 8s timeout, all services. 3. From inside poimen-memory-bfc8k: embeddings /health -> 12s timeout; 10.96.0.1:443 -> 8s timeout. Same-namespace traffic still worked, which limits the fault to this policy. 4. memory logged "readiness: embedder probe timed out elapsed_ms=10000" every 60s. All five services already serve inference correctly. They are reasoning (3.1s), reasoning-short (2.5s), embeddings (0.5s, real 768-dim vector), reranker (0.09s, 0.982 vs 0.00004) and qwen-cpu (4.1s, qwen2.5:3b-instruct). This adds the missing ingress rule. It mirrors the existing paperless rule, which bypasses the gateway for the same reason: poimen-memory has no service-account JWT yet. Refs: #TBD --- k8s/apps/llm-serving/networkpolicy.yaml | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/k8s/apps/llm-serving/networkpolicy.yaml b/k8s/apps/llm-serving/networkpolicy.yaml index d403e60..17a1ecd 100644 --- a/k8s/apps/llm-serving/networkpolicy.yaml +++ b/k8s/apps/llm-serving/networkpolicy.yaml @@ -66,6 +66,21 @@ spec: port: 80 - protocol: TCP port: 8000 + # Allow from poimen namespace (poimen-memory ingest + query). + # The client-side egress rule is the other half and must stay in step: + # NetworkPolicy poimen/poimen-memory-llm-egress permits port 8080 to pods + # labelled part-of=llm-serving, which is the same selector used here. This + # policy had no matching ingress rule at all, so the packet was allowed out + # and dropped here. The client then saw a bare timeout and the predictor log + # stayed empty, which is why this failure looked like a hung inference engine + # for 34 days. + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: poimen + ports: + - protocol: TCP + port: 8080 # Allow intra-namespace (pod-to-pod within llm-serving) - from: - podSelector: