Files
homelab-frontend/k8s/network-policy.yaml
T
rockandAdmin Bot 28753d11ed
CI / CI (push) Successful in 32m48s
fix(netpol): allow Prometheus scrape of gateway /metrics (#62)
## Problem

The new `api-gateway` ServiceMonitor discovers all 3 replicas, but every scrape
fails with `context deadline exceeded`:

```
down   api-gateway   api-gateway-6986b7c889-pmt4b   err=Get "http://10.244.0.180:8080/metrics": context deadline
down   api-gateway   api-gateway-6986b7c889-c28dt   err=Get "http://10.244.1.1:8080/metrics": context deadline
down   api-gateway   api-gateway-6986b7c889-552gl   err=Get "http://10.244.2.61:8080/metrics": context deadline
```

## Cause

`network-policy.yaml` allowed Prometheus ingress with:

```yaml
namespaceSelector:
  matchLabels:
    name: monitoring
```

Only `ingress-nginx` carries a plain `name:` label in this cluster. Every other
namespace has just `kubernetes.io/metadata.name`, so this rule matched **nothing**
and silently denied the scrape. (The rule directly above it — `name: ingress-nginx`
— works only because that namespace is the documented exception.)

## Fix

Select on `kubernetes.io/metadata.name: monitoring`.

## Impact

Unblocks gateway metrics: `llm_ttft_seconds`, `llm_itl_seconds`,
`llm_tokens_total`, `queue_full_requests_total`,
`warmup_requests_delayed_total`, `vllm_prefix_cache_hit_tokens_total`.

These are the only per-request LLM latency series that cover the two Ollama
models (`ornith:35b`, `qwen2.5:3b-instruct`), which expose no metrics endpoint
of their own.

---------

Co-authored-by: Admin Bot <[email protected]>
Reviewed-on: #62
2026-09-29 14:40:31 +00:00

186 lines
4.9 KiB
YAML

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: api-gateway
namespace: api
labels:
app: api-gateway
spec:
podSelector:
matchLabels:
app: api-gateway
policyTypes:
- Ingress
- Egress
ingress:
# Allow from ingress-nginx controller (from ingress-nginx namespace)
- from:
- namespaceSelector:
matchLabels:
name: ingress-nginx
ports:
- protocol: TCP
port: 8080
# Allow from Prometheus scraping (monitoring namespace)
# MUST select on kubernetes.io/metadata.name, not a plain `name:` label.
# Only ingress-nginx carries a plain `name` label in this cluster; every
# other namespace has just kubernetes.io/metadata.name, so `name: monitoring`
# matched nothing and silently denied every Prometheus scrape of :8080
# (surfaced as context-deadline-exceeded on all 3 gateway replicas while the
# ServiceMonitor itself was discovered and healthy).
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: monitoring
ports:
- protocol: TCP
port: 8080
# Allow from poimen namespace (orchestrator & worker pods)
# Enable Poimen workflows to call the LLM API gateway
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: poimen
ports:
- protocol: TCP
port: 8080
# Allow from portfolio namespace (riotpiao.com chat terminal)
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: portfolio
ports:
- protocol: TCP
port: 8080
# Allow from paperless namespace (paperless-ai document auto-tagging)
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: paperless
ports:
- protocol: TCP
port: 8080
egress:
# Allow DNS
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# Allow to upstreams (LLM services in llm-serving namespace)
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: llm-serving
ports:
- protocol: TCP
port: 80
- protocol: TCP
port: 8000
- protocol: TCP
port: 8001
# Allow to other upstreams if needed (embeddings, reranker, etc.)
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: llm-serving
ports:
- protocol: TCP
port: 8080
# Allow to atlas (riotpiao-backend) for /cluster/* routes
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: atlas
ports:
- protocol: TCP
port: 8080
# Allow to IAM (Authentik) for JWKS fetch + API
# Authentik pod listens on 9000 (http) and 9443 (https)
# Service translates 80→9000, 443→9443
# NetworkPolicy matches destination pod port, not service port
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: iam
ports:
- protocol: TCP
port: 9000
- protocol: TCP
port: 9443
# Allow to SQS (queue management service)
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: sqs
ports:
- protocol: TCP
port: 9090
# Allow to Temporal (workflow engine gRPC)
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: temporal
ports:
- protocol: TCP
port: 7233
# Allow to Poimen (memory:8080 + graphql:8090 + redis:6379 + fleet-controller:8788)
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: poimen
ports:
- protocol: TCP
port: 8080
- protocol: TCP
port: 8090
- protocol: TCP
port: 6379
- protocol: TCP
port: 8788
# Allow to Gotify (push notifications)
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: notifications
ports:
- protocol: TCP
port: 80
# Allow to MinIO (S3-compatible storage)
# Service `minio` listens on port 80 (targetPort 9000).
# Headless `minio-cluster-hl` is 9000. Allow both.
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: storage
ports:
- protocol: TCP
port: 80
- protocol: TCP
port: 9000
# Allow to Forgejo registry v2 API (image digest lookup for promotion)
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: cicd
ports:
- protocol: TCP
port: 3000
# Allow to kubernetes API server (ArgoCD Application patching via in-cluster SA).
# kube-apiserver runs on host network (192.168.1.162/166:6443) — must use ipBlock.
# ClusterIP 10.96.0.1:443 also routes there, covered by the 10.96.0.0/12 CIDR.
- to:
- ipBlock:
cidr: 10.96.0.0/12
- ipBlock:
cidr: 192.168.1.0/24
ports:
- protocol: TCP
port: 443
- protocol: TCP
port: 6443