rock
a234451548
poimen-memory could not reach any inference pod. The egress side was
already correct: NetworkPolicy poimen/poimen-memory-llm-egress permits
port 8080 to the pods labelled llm-reasoning, llm-embeddings and
llm-reranker. This policy had no matching ingress rule, so Cilium
allowed the packet out and dropped it at the destination.
The observable was a bare timeout. The request never reached vLLM, so
the predictor log stayed empty, and /health still answered because a
probe from inside the pod does not cross the policy. That made the
failure look like a hung inference engine.
Evidence, measured on 2026-10-31 against the running predictors:
1. A probe pod in llm-serving WITH label part-of=llm-serving:
/v1/chat/completions -> 200 in 3.1s, 8 completion tokens.
2. The same probe pod WITHOUT that label: 8s timeout, all services.
3. From inside poimen-memory-bfc8k: embeddings /health -> 12s
timeout; 10.96.0.1:443 -> 8s timeout. Same-namespace traffic
still worked, which limits the fault to this policy.
4. memory logged "readiness: embedder probe timed out
elapsed_ms=10000" every 60s.
All five services already serve inference correctly. They are
reasoning (3.1s), reasoning-short (2.5s), embeddings (0.5s, real
768-dim vector), reranker (0.09s, 0.982 vs 0.00004) and qwen-cpu
(4.1s, qwen2.5:3b-instruct).
This adds the missing ingress rule. It mirrors the existing paperless
rule, which bypasses the gateway for the same reason: poimen-memory
has no service-account JWT yet.
Refs: #TBD
92 lines
3.3 KiB
YAML
92 lines
3.3 KiB
YAML
# NetworkPolicy for LLM inference engines (llm-serving namespace).
|
|
#
|
|
# These pods have NO auth — vLLM, Ollama, and TEI accept any request.
|
|
# All access MUST go through the api-gateway, which validates JWTs and
|
|
# injects identity headers (X-Forwarded-User, X-Auth-Verified).
|
|
#
|
|
# Replaces the hand-applied llm-serving-default-deny policy that used
|
|
# `llm-client: "true"` pod label as a selector — any pod in any namespace
|
|
# could self-grant access by adding that label, which defeats the purpose.
|
|
#
|
|
# This policy restricts ingress to:
|
|
# 1. api namespace (gateway) — the sole entry point for inference
|
|
# 2. monitoring namespace — Prometheus scraping vLLM/TEI /metrics
|
|
# 3. intra-namespace — pod-to-pod (future: multi-replica comms)
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: NetworkPolicy
|
|
metadata:
|
|
name: llm-serving-ingress
|
|
namespace: llm-serving
|
|
labels:
|
|
app.kubernetes.io/part-of: llm-serving
|
|
spec:
|
|
podSelector:
|
|
matchLabels:
|
|
app.kubernetes.io/part-of: llm-serving
|
|
policyTypes:
|
|
- Ingress
|
|
ingress:
|
|
# Allow from api-gateway (namespace: api)
|
|
# Gateway proxies /v1/chat/completions, /v1/embeddings, /v1/rerank
|
|
- from:
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: api
|
|
ports:
|
|
- protocol: TCP
|
|
port: 8080 # vLLM, Ollama HTTP
|
|
- protocol: TCP
|
|
port: 80 # KServe predictor services
|
|
- protocol: TCP
|
|
port: 8000 # vLLM direct (some configs)
|
|
- protocol: TCP
|
|
port: 11434 # Ollama native port
|
|
# Allow Prometheus scraping from monitoring namespace
|
|
# vLLM: :8080/metrics, TEI: :9000/metrics
|
|
- from:
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: monitoring
|
|
ports:
|
|
- protocol: TCP
|
|
port: 8080
|
|
- protocol: TCP
|
|
port: 9000
|
|
# Allow from paperless namespace (paperless-ai auto-tagging)
|
|
# Bypasses gateway until service-account JWT token exchange is implemented.
|
|
# paperless-ai-agent has llm:inference role in Authentik.
|
|
- from:
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: paperless
|
|
ports:
|
|
- protocol: TCP
|
|
port: 8080
|
|
- protocol: TCP
|
|
port: 80
|
|
- protocol: TCP
|
|
port: 8000
|
|
# Allow from poimen namespace (poimen-memory ingest + query).
|
|
# The client-side egress rule is the other half and must stay in step:
|
|
# NetworkPolicy poimen/poimen-memory-llm-egress permits port 8080 to pods
|
|
# labelled part-of=llm-serving, which is the same selector used here. This
|
|
# policy had no matching ingress rule at all, so the packet was allowed out
|
|
# and dropped here. The client then saw a bare timeout and the predictor log
|
|
# stayed empty, which is why this failure looked like a hung inference engine
|
|
# for 34 days.
|
|
- from:
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: poimen
|
|
ports:
|
|
- protocol: TCP
|
|
port: 8080
|
|
# Allow intra-namespace (pod-to-pod within llm-serving)
|
|
- from:
|
|
- podSelector:
|
|
matchLabels:
|
|
app.kubernetes.io/part-of: llm-serving
|
|
ports:
|
|
- protocol: TCP
|
|
port: 8080
|