P3.8: Restrict llm-serving ingress to api-gateway only (#15 )

## Summary Replaces hand-applied `llm-serving-default-deny` NetworkPolicy with a git-managed, namespace-scoped policy that only allows traffic from the api-gateway. **Issue:** #13 Co-authored-by: rock <[email protected]>
feat: switch image-updater to digest-based :latest tracking
2026-09-08 16:56:36 +00:00 · 2026-09-07 18:19:46 -07:00 · 2026-09-07 17:49:58 -07:00 · 2026-09-07 17:47:03 -07:00 · 2026-09-07 13:22:09 -07:00 · 2026-09-07 13:20:20 -07:00
15 changed files with 192 additions and 129 deletions
@@ -1,49 +0,0 @@
-name: Build and push runner images
-
-on:
-  push:
-    paths:
-      - 'k8s/infra/forgejo-runner/Dockerfile.golang'
-      - 'k8s/infra/forgejo-runner/Dockerfile.rust'
-      - 'k8s/infra/forgejo-runner/Dockerfile.node'
-    branches:
-      - main
-
-jobs:
-  build-runners:
-    runs-on: golang
-    env:
-      REGISTRY: forgejo.riotpiao.com
-      IMAGE_BASE: forgejo.riotpiao.com/rock
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@v4
-
-      - name: Get short SHA
-        id: sha
-        run: |
-          SHORT_SHA=$(git rev-parse --short HEAD)
-          echo "short_sha=${SHORT_SHA}" >> $GITHUB_OUTPUT
-
-      - name: Registry login
-        run: |
-          echo "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" \
-            --username "${REGISTRY_USER}" --password-stdin
-        env:
-          REGISTRY_USER: ${{ secrets.FORGEJO_REGISTRY_USER }}
-          REGISTRY_TOKEN: ${{ secrets.FORGEJO_REGISTRY_TOKEN }}
-
-      - name: Build and push all runner images
-        run: |
-          set -e
-          for RUNNER in golang rust node; do
-            echo "📦 Building ${RUNNER}-runner..."
-            docker build -f "k8s/infra/forgejo-runner/Dockerfile.${RUNNER}" \
-              -t "${IMAGE_BASE}/forgejo-runner-${RUNNER}:${{ steps.sha.outputs.short_sha }}" \
-              -t "${IMAGE_BASE}/forgejo-runner-${RUNNER}:latest" \
-              .
-            docker push "${IMAGE_BASE}/forgejo-runner-${RUNNER}:${{ steps.sha.outputs.short_sha }}"
-            docker push "${IMAGE_BASE}/forgejo-runner-${RUNNER}:latest"
-            echo "✅ Pushed ${RUNNER}-runner"
-          done
-          echo "\n✅ All runner images built and pushed"
@@ -208,3 +208,95 @@ versions without warning in your own values file.
 Grouping by layer (rather than by day or by "misc fixes") makes it much
 easier to `git log --oneline -- <path>` your way back to *why* a given
 piece of config looks the way it does, months later.
+
+## Unified Forgejo CI Workflow Pattern (Enforced 2026-09-07+)
+
+All repositories MUST follow this exact structure. No variations.
+
+```yaml
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+env:
+  REGISTRY: <your-registry-hostname>
+  IMAGE: <registry>/<org>/<service-name>
+
+jobs:
+  test:
+    name: Test
+    runs-on: [golang|node|rust]
+    steps:
+      - name: Install Node.js for actions runtime
+        run: apt-get update && apt-get install -y nodejs
+
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      # Language-specific tests here (no docker, no registry)
+      # - name: Run tests
+      #   run: npm test -- --run || true
+
+  build-push:
+    name: Build & Push Image
+    needs: test
+    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+    runs-on: [golang|node|rust]
+    steps:
+      - name: Install Node.js and Docker
+        run: |
+          apt-get update
+          apt-get install -y nodejs docker.io
+
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Get short SHA
+        id: sha
+        run: |
+          SHORT_SHA=$(git rev-parse --short HEAD)
+          echo "short_sha=${SHORT_SHA}" >> $GITHUB_OUTPUT
+
+      - name: Registry login
+        run: |
+          echo "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" \
+            --username "${REGISTRY_USER}" --password-stdin
+        env:
+          REGISTRY_USER: ${{ secrets.FORGEJO_REGISTRY_USER }}
+          REGISTRY_TOKEN: ${{ secrets.FORGEJO_REGISTRY_TOKEN }}
+
+      - name: Build Docker image
+        run: |
+          docker build --no-cache \
+            -t "${IMAGE}:${{ steps.sha.outputs.short_sha }}" \
+            -t "${IMAGE}:latest" \
+            .
+
+      - name: Push Docker image
+        run: |
+          docker push "${IMAGE}:${{ steps.sha.outputs.short_sha }}"
+          docker push "${IMAGE}:latest"
+
+      - name: Prune unused images
+        run: docker image prune -a --force 2>&1 | tail -3 || true
+```
+
+### Anti-Patterns (DO NOT USE)
+
+- ❌ `container: image: golang:1.26` overrides — breaks docker socket sharing
+- ❌ Conditional `if:` on individual steps — use separate jobs instead
+- ❌ Installing docker.io in test job — only needed in build-push
+- ❌ Monolithic job doing test + build + push — hard to debug
+- ❌ Using `{{ github.sha }}` for image tag — use short commit SHA for readability
+
+### How It Works
+
+1. **PR to feature branch** → test job runs, build-push skipped, nothing pushed
+2. **Push to main** → test runs, build-push runs after test passes, image pushed
+3. Docker socket shared between dind sidecar and runner via emptyDir mount at `/run`
+4. `docker_host: automount` in runner config injects socket into workflow containers
+5. Secrets (FORGEJO_REGISTRY_USER, TOKEN) set in Forgejo repo settings, NOT in git
@@ -14,5 +14,6 @@ resources:
  - ornith.yaml
  - reasoning.yaml
  - reranker.yaml
+  - networkpolicy.yaml
 # No namespace transformer: every file sets its own, and the transformer would
 # rewrite metadata.namespace on anything cross-namespace added later.
@@ -0,0 +1,62 @@
+# NetworkPolicy for LLM inference engines (llm-serving namespace).
+#
+# These pods have NO auth — vLLM, Ollama, and TEI accept any request.
+# All access MUST go through the api-gateway, which validates JWTs and
+# injects identity headers (X-Forwarded-User, X-Auth-Verified).
+#
+# Replaces the hand-applied llm-serving-default-deny policy that used
+# `llm-client: "true"` pod label as a selector — any pod in any namespace
+# could self-grant access by adding that label, which defeats the purpose.
+#
+# This policy restricts ingress to:
+#   1. api namespace (gateway) — the sole entry point for inference
+#   2. monitoring namespace — Prometheus scraping vLLM/TEI /metrics
+#   3. intra-namespace — pod-to-pod (future: multi-replica comms)
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: llm-serving-ingress
+  namespace: llm-serving
+  labels:
+    app.kubernetes.io/part-of: llm-serving
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/part-of: llm-serving
+  policyTypes:
+    - Ingress
+  ingress:
+    # Allow from api-gateway (namespace: api)
+    # Gateway proxies /v1/chat/completions, /v1/embeddings, /v1/rerank
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: api
+      ports:
+        - protocol: TCP
+          port: 8080   # vLLM, Ollama HTTP
+        - protocol: TCP
+          port: 80      # KServe predictor services
+        - protocol: TCP
+          port: 8000    # vLLM direct (some configs)
+        - protocol: TCP
+          port: 11434   # Ollama native port
+    # Allow Prometheus scraping from monitoring namespace
+    # vLLM: :8080/metrics, TEI: :9000/metrics
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: monitoring
+      ports:
+        - protocol: TCP
+          port: 8080
+        - protocol: TCP
+          port: 9000
+    # Allow intra-namespace (pod-to-pod within llm-serving)
+    - from:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/part-of: llm-serving
+      ports:
+        - protocol: TCP
+          port: 8080
@@ -38,8 +38,8 @@ metadata:
    argocd.argoproj.io/sync-wave: "7"
    # ArgoCD Image Updater - auto-update on new image push
    argocd-image-updater.argoproj.io/image-list: gw=forgejo.riotpiao.com/rock/api-gateway
-    argocd-image-updater.argoproj.io/gw.update-strategy: newest-build
-    argocd-image-updater.argoproj.io/gw.allow-tags: regexp:^[0-9a-f]{7}$
+    argocd-image-updater.argoproj.io/gw.update-strategy: digest
+    argocd-image-updater.argoproj.io/gw.allow-tags: regexp:^latest$
    argocd-image-updater.argoproj.io/write-back-method: argocd
 spec:
  project: homelab
@@ -248,8 +248,8 @@ metadata:
    argocd.argoproj.io/sync-wave: "8"
    # ArgoCD Image Updater - auto-update on new image push
    argocd-image-updater.argoproj.io/image-list: app=forgejo.riotpiao.com/rock/portfolio
-    argocd-image-updater.argoproj.io/app.update-strategy: newest-build
-    argocd-image-updater.argoproj.io/app.allow-tags: regexp:^[0-9a-f]{7}$
+    argocd-image-updater.argoproj.io/app.update-strategy: digest
+    argocd-image-updater.argoproj.io/app.allow-tags: regexp:^latest$
    argocd-image-updater.argoproj.io/write-back-method: argocd
 spec:
  project: homelab
@@ -10,14 +10,13 @@ metadata:
      memory=forgejo.riotpiao.com/rock/poimen-memory
      workflows=forgejo.riotpiao.com/rock/poimen-workflows
      frontend=forgejo.riotpiao.com/rock/poimen-frontend
-    argocd-image-updater.argoproj.io/memory.update-strategy: newest-build
-    argocd-image-updater.argoproj.io/memory.allow-tags: regexp:^[0-9a-f]{7}$
-    argocd-image-updater.argoproj.io/workflows.update-strategy: newest-build
-    argocd-image-updater.argoproj.io/workflows.allow-tags: regexp:^[0-9a-f]{7}$
-    argocd-image-updater.argoproj.io/frontend.update-strategy: newest-build
-    argocd-image-updater.argoproj.io/frontend.allow-tags: regexp:^[0-9a-f]{7}$
-    argocd-image-updater.argoproj.io/write-back-method: git
-    argocd-image-updater.argoproj.io/git-branch: main
+    argocd-image-updater.argoproj.io/memory.update-strategy: digest
+    argocd-image-updater.argoproj.io/memory.allow-tags: regexp:^latest$
+    argocd-image-updater.argoproj.io/workflows.update-strategy: digest
+    argocd-image-updater.argoproj.io/workflows.allow-tags: regexp:^latest$
+    argocd-image-updater.argoproj.io/frontend.update-strategy: digest
+    argocd-image-updater.argoproj.io/frontend.allow-tags: regexp:^latest$
+    argocd-image-updater.argoproj.io/write-back-method: argocd
 spec:
  project: homelab
  sources:
@@ -1,16 +0,0 @@
-FROM code.forgejo.org/forgejo/runner:6
-
-# Switch to root to install packages (Alpine)
-USER root
-
-# Alpine uses apk, not apt-get
-RUN apk update && apk add --no-cache \
-      nodejs \
-      npm \
-      docker-cli
-
-# Verify installations
-RUN docker --version && node --version && git --version
-
-# Switch back to runner user
-USER 1000:1000
@@ -1,16 +0,0 @@
-FROM code.forgejo.org/forgejo/runner:6
-
-# Switch to root to install packages (Alpine)
-USER root
-
-# Alpine uses apk, not apt-get
-RUN apk update && apk add --no-cache \
-      docker-cli \
-      nodejs \
-      npm
-
-# Verify installations
-RUN node --version && docker --version && git --version
-
-# Switch back to runner user
-USER 1000:1000
@@ -1,20 +0,0 @@
-FROM code.forgejo.org/forgejo/runner:6
-
-# Switch to root to install packages (Alpine)
-USER root
-
-# Alpine uses apk, not apt-get
-RUN apk update && apk add --no-cache \
-      nodejs \
-      npm \
-      curl \
-      docker-cli
-
-# Install Rust (as root, skip verification for now)
-RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable || true
-
-# Verify core installations
-RUN docker --version && node --version && git --version
-
-# Switch back to runner user
-USER 1000:1000
@@ -36,3 +36,4 @@ data:
      valid_volumes:
        - /docker-certs/client
      network: host
+      docker_host: automount
@@ -34,7 +34,11 @@ spec:
          command: ["sh", "-c"]
          args:
            - |
-              test -f /data/.runner || forgejo-runner register --no-interactive \
+              # Always re-register to keep labels in sync with values.yaml.
+              # Without this, changing a runner label requires manually deleting
+              # the PVC or .runner file — not GitOps-friendly.
+              rm -f /data/.runner
+              forgejo-runner register --no-interactive \
                --instance {{ .Values.runner.forgejoUrl }} \
                --token $(RUNNER_TOKEN) \
                --name {{ .Values.runner.name }} \
@@ -56,20 +60,18 @@ spec:
      containers:
        - name: runner
          image: {{ .Values.runner.image.repository }}:{{ .Values.runner.image.tag }}
-          command: ["sh", "-c", "forgejo-runner daemon --config /etc/forgejo-runner/config.yaml"]
+          command: ["sh", "-c", "while ! wget -q -O- http://localhost:2375/_ping >/dev/null 2>&1; do echo 'waiting for dind...'; sleep 2; done; echo 'dind ready'; forgejo-runner daemon --config /etc/forgejo-runner/config.yaml"]
          workingDir: /data
          env:
            - name: DOCKER_HOST
-              value: tcp://localhost:2376
-            - name: DOCKER_TLS_VERIFY
-              value: "1"
-            - name: DOCKER_CERT_PATH
-              value: /docker-certs/client
+              value: tcp://localhost:2375
          volumeMounts:
            - name: runner-data
              mountPath: /data
            - name: docker-certs
              mountPath: /docker-certs
+            - name: docker-sock
+              mountPath: /run
            - name: homelab-ca
              mountPath: /etc/ssl/certs/homelab-ca.pem
              subPath: ca.crt
@@ -85,10 +87,12 @@ spec:
            privileged: true    # required for DinD; cicd namespace is labelled privileged
          env:
            - name: DOCKER_TLS_CERTDIR
-              value: /docker-certs
+              value: ""
          volumeMounts:
            - name: docker-certs
              mountPath: /docker-certs
+            - name: docker-sock
+              mountPath: /run
            - name: dind-storage
              mountPath: /var/lib/docker
            - name: homelab-ca
@@ -113,6 +117,8 @@ spec:
            claimName: {{ .Release.Name }}-dind
        - name: docker-certs
          emptyDir: {}    # DinD regenerates mTLS certs on each start
+        - name: docker-sock
+          emptyDir: {}    # Shared docker socket between dind and runner
        - name: homelab-ca
          # homelab-ca is a ConfigMap (public CA trust bundle), not a Secret.
          # The volumeMounts use subPath: ca.crt to project the single cert file.
@@ -2,15 +2,14 @@
 # runner instance. Only runner.name and runner.labels differ -- everything
 # else (image, dind, persistence, tolerations, nodeSelector) is shared.
 #
-# node:22-bookworm ships Node natively. Docker client installed via workflow step if needed.
-# (homelab has no CI; custom runner images built manually if desired)
-# Bootstrap with runner image (already has Node.js), CI builds custom
+# Label image: node:22-bookworm — Debian, root, apt-get, Node.js, npm, git.
+# Install docker in workflow steps as needed.
 runner:
  image:
    repository: code.forgejo.org/forgejo/runner
    tag: "6"
  name: node-runner
-  labels: "node:docker://code.forgejo.org/forgejo/runner:6"
+  labels: "node:docker://node:22-bookworm"

 # GC CronJob renders only from the default (golang) values to avoid duplicates
 gc:
@@ -2,13 +2,14 @@
 # runner instance. Only runner.name and runner.labels differ -- everything
 # else (image, dind, persistence, tolerations, nodeSelector) is shared.
 #
-# Bootstrap with runner image, CI builds custom with Node.js+Rust
+# Label image: rust:1-bookworm — Debian, root, apt-get, Rust, cargo, git.
+# Install Node.js/docker in workflow steps as needed.
 runner:
  image:
    repository: code.forgejo.org/forgejo/runner
    tag: "6"
  name: rust-runner
-  labels: "rust:docker://code.forgejo.org/forgejo/runner:6"
+  labels: "rust:docker://rust:1-bookworm"



@@ -1,9 +1,12 @@
 runner:
  image:
    repository: code.forgejo.org/forgejo/runner
-    tag: "6"  # Bootstrap with runner image, CI builds custom with Node.js
+    tag: "6"
  name: golang-runner
-  labels: "golang:docker://code.forgejo.org/forgejo/runner:6"
+  # Label image is what workflow steps run in (NOT the runner daemon image).
+  # golang:1.26-bookworm: Debian, root, apt-get, Go, git.
+  # TODO: Switch to custom image once build-runner-images.yml pushes images
+  labels: "golang:docker://golang:1.26-bookworm"
  forgejoUrl: http://forgejo-gitea-http.cicd.svc.cluster.local:3000
  tokenSecret: runner-token
  resources:
Author	SHA1	Message	Date
rock	9fe867e84f	P3.8: Restrict llm-serving ingress to api-gateway only (#15 ) ## Summary Replaces hand-applied `llm-serving-default-deny` NetworkPolicy with a git-managed, namespace-scoped policy that only allows traffic from the api-gateway. Issue: #13 Co-authored-by: rock <[email protected]>	2026-09-08 16:56:36 +00:00
rock	8983720d3a	feat: switch image-updater to digest-based :latest tracking All image-updater annotations now use update-strategy: digest with allow-tags: ^latest$ and write-back-method: argocd. No SHA tags committed to git — digest overrides stored in ArgoCD state only. - poimen: git write-back → argocd, removed git-branch - portfolio: newest-build SHA → digest latest - api-gateway: newest-build SHA → digest latest	2026-09-07 18:19:46 -07:00
rock	f13ee1ba21	revert: remove unsupported buildOptions (ArgoCD v3.4.5 doesn't support it) - buildOptions field not available in ArgoCD v3.4.5 - SOPS decryption already handled by repo-server ksops plugin - Revert to simple kustomize config - Portfolio Application can now sync properly	2026-09-07 17:49:58 -07:00
rock	99bd163f0b	fix: enable ksops plugin for portfolio Application - Add kustomize config with --enable-alpha-plugins to support ksops - Allows ArgoCD to properly decrypt SOPS-encrypted files - Fixes Image Updater compatibility with sops field in kustomization.yaml	2026-09-07 17:47:03 -07:00
rock	e28e3cae81	fix: wait for dind to be ready before starting runner daemon	2026-09-07 13:22:09 -07:00
rock	febfb055ae	fix: use tcp://localhost:2375 for dind (no TLS, no socket permission issues)	2026-09-07 13:20:20 -07:00
rock	9d61c42b99	fix: run runner as root to access dind socket	2026-09-07 13:18:54 -07:00
rock	594e752d47	fix: runner uses unix socket instead of TLS TCP for dind Job containers spawned by the runner run inside dind. With TCP+TLS (tcp://localhost:2376), localhost inside those containers doesn't reach the dind sidecar. Unix socket at /run/docker.sock works because both runner and dind share the /run emptyDir. Also disables DOCKER_TLS_CERTDIR so dind creates the socket instead of only listening on TLS TCP.	2026-09-07 13:16:49 -07:00
rock	1a8e87131b	ci: fix runner labels + CoreDNS rewrite + cleanup - Runners use public images (code.forgejo.org/forgejo/runner:6) - Labels pull from Docker Hub: golang:1.26, node:22, rust:1-bookworm - Add CoreDNS api.riotpiao.com rewrite - Fix runner re-registration to keep labels in sync - Add unified CI pattern docs to CLAUDE.example.md - Remove dead .forgejo/ workflow dir (Forgejo uses .gitea/)	2026-09-07 13:01:56 -07:00