fix(sops): add LLM_AUTH_CLIENT_SECRET for paperless-ai-agent Authentik auth

fix(paperless-ai): move LLM auth client secret from plaintext to SOPS
- Remove hardcoded client secret from init container - Reference secret from paperless-ai-config (SOPS-encrypted) - Init container reads LLM_AUTH_CLIENT_SECRET env var - Secret key must be added to k8s/argocd/secrets/paperless-ai-secrets.enc.yaml by someone with SOPS/age key access This removes the plaintext secret from commit history and future deployments will source it securely from the encrypted Secret.
2026-09-14 23:19:43 +09:00 · 2026-09-14 23:07:52 +09:00 · 2026-09-14 08:09:26 +09:00 · 2026-09-14 07:56:05 +09:00 · 2026-09-13 14:04:35 +00:00 · 2026-09-13 13:54:31 +00:00
5 changed files with 133 additions and 41 deletions
@@ -52,6 +52,20 @@ spec:
          port: 8080
        - protocol: TCP
          port: 9000
+    # Allow from paperless namespace (paperless-ai auto-tagging)
+    # Bypasses gateway until service-account JWT token exchange is implemented.
+    # paperless-ai-agent has llm:inference role in Authentik.
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: paperless
+      ports:
+        - protocol: TCP
+          port: 8080
+        - protocol: TCP
+          port: 80
+        - protocol: TCP
+          port: 8000
    # Allow intra-namespace (pod-to-pod within llm-serving)
    - from:
        - podSelector:
@@ -21,6 +21,49 @@ spec:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
+      volumes:
+        - name: paperless-ai-data
+          persistentVolumeClaim:
+            claimName: paperless-ai-data
+      initContainers:
+        - name: fetch-llm-token
+          image: curlimages/curl:8.12.0
+          command:
+            - sh
+            - -c
+            - |
+              set -e
+              echo "[init] Fetching JWT token from Authentik for paperless-ai-agent..."
+              
+              # Get JWT token via client_credentials grant
+              # Client secret sourced from environment (injected from paperless-ai-config Secret)
+              TOKEN_RESPONSE=$(curl -s -X POST https://authentik.riotpiao.com/application/o/token/ \
+                -d "grant_type=client_credentials" \
+                -d "client_id=paperless-ai-agent" \
+                -d "client_secret=${LLM_AUTH_CLIENT_SECRET}" \
+                -d "scope=openid llm:inference" 2>/dev/null)
+              
+              # Extract token
+              TOKEN=$(echo "$TOKEN_RESPONSE" | grep -o '"access_token":"[^"]*' | cut -d'"' -f4)
+              
+              if [ -z "$TOKEN" ]; then
+                echo "[error] Failed to get token. Response: $TOKEN_RESPONSE"
+                exit 1
+              fi
+              
+              # Store token in file for main container to read
+              mkdir -p /data
+              echo "$TOKEN" > /data/llm_token.txt
+              echo "[init] Token fetched and stored successfully"
+          env:
+            - name: LLM_AUTH_CLIENT_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: paperless-ai-config
+                  key: LLM_AUTH_CLIENT_SECRET
+          volumeMounts:
+            - name: paperless-ai-data
+              mountPath: /data
      containers:
        - name: paperless-ai
          image: clusterzx/paperless-ai:latest
@@ -35,18 +78,20 @@ spec:
                  key: PAPERLESS_API_TOKEN
            - name: PAPERLESS_USERNAME
              value: "admin"
-            # LLM API — local gateway, no auth required (phase 3 not built yet)
+            # LLM API — via public gateway with JWT auth
            - name: AI_PROVIDER
              value: "custom"
            - name: CUSTOM_BASE_URL
-              value: "http://api-gateway.api.svc.cluster.local:8080/v1"
-            - name: CUSTOM_API_KEY
-              value: "not-required"
+              value: "https://api.riotpiao.com/v1"
+            # Token will be read from file at runtime by the application
+            # The init container fetches it and stores in /app/data/llm_token.txt
+            - name: CUSTOM_API_KEY_FILE
+              value: "/app/data/llm_token.txt"
            - name: CUSTOM_MODEL
-              value: "reasoning"
-            # Behavior
+              value: "qwen2.5:3b-instruct"
+            # Behavior - scan for new documents and tag them
            - name: SCAN_INTERVAL
-              value: "300"
+              value: "60"
            - name: PROCESS_PREDEFINED_DOCUMENTS
              value: "no"
            - name: ADD_AI_TAG
@@ -55,6 +100,11 @@ spec:
              value: "ai-processed"
            - name: USE_PROMPT_TAGS
              value: "yes"
+            - name: ADD_AI_DESCRIPTIONS
+              value: "yes"
+          volumeMounts:
+            - name: paperless-ai-data
+              mountPath: /app/data
          resources:
            requests:
              cpu: 100m
@@ -1,23 +1,24 @@
-apiVersion: ENC[AES256_GCM,data:l7I=,iv:NZY7r3JVW3zVwxeiScvWKpAQUDa9+nckHd0qWVrGU88=,tag:qpowp5OILdYKtTux/nlWpg==,type:str]
-kind: ENC[AES256_GCM,data:6oeEbuIk,iv:5flI9TtcYQ961wOYPBPhvQpitHFYAf8yMdNBXqytbJs=,tag:WNhlbKmSeZEqZ9ZgiB/BYg==,type:str]
+apiVersion: v1
+kind: Secret
 metadata:
-    name: ENC[AES256_GCM,data:fvJMsgy+wPZqMCdxm9hoV3n/+Q==,iv:I3rVtHIJBOME+bxhPws58Zjhj5i+KT5UtB9o7Vus5EU=,tag:6h4FnUu7PGxlQPIQxPYCRg==,type:str]
-    namespace: ENC[AES256_GCM,data:CDriLoLovROW,iv:rHXcN1xm5+t2D/Tq/2sx9lQFF8anD5jH24ZntPuBA8U=,tag:XstJAz6S8IM1c/pp9Cg0Ww==,type:str]
-type: ENC[AES256_GCM,data:JdTwBbag,iv:9Ys15Ketl0ghNK0u0N8IOpUt7+KoloutiG5M9zHPXxw=,tag:teau/udf41Ty34A5wLAm6Q==,type:str]
+    name: paperless-ai-config
+    namespace: paperless
+type: Opaque
 stringData:
-    PAPERLESS_API_TOKEN: ENC[AES256_GCM,data:TuQeDx8po3h4loTRABlItVYQJ7gFjnmIn3zQQGtUcoNFMKpkqLK/GQ==,iv:TDg0stpca5pDtatqu8DFU7R0Bm/S/BI9ZoiG4K8mCT4=,tag:BfjX25V5gL7AeIsscClRAg==,type:str]
+    PAPERLESS_API_TOKEN: ENC[AES256_GCM,data:wlBC85ep6zioWLhfhNczkKfTMR8tmciN+4r3of8GpQoJmKyE2r7pOg==,iv:Rem/UJl5wUNi8G+uEV7f2eCLabaY26sZ7+AsKYiOZ4I=,tag:bGS1fu13Mwsrr+5NUEzzLw==,type:str]
+    LLM_AUTH_CLIENT_SECRET: ENC[AES256_GCM,data:co7FfmRXKgrz9fNgR10KtlHL/iG4F5WKJgjcKem1H5hF5FviVi4+CQNekA==,iv:ADtBHiI6BtrkuH1iIfKV3I5odNLMQe5y/9GGTJrPNN0=,tag:j6wX6zO5VhKoLzdr7BQZCw==,type:str]
 sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBKVDN3aFVqVmFXY1VQVXZP
-            OUsrNHdNdlRvRFgvTDQrNE5uL2xaazVENTBjCkNPbGR5Vk16RXNDOW15OGNTRmFR
-            U0JOeTllaWU1dzNVY3lBbEVyVG5tOEkKLS0tIEZvajlvcUtJdFNNUkkzV3FKOFRj
-            UUE2TDBzT0xVc2E1NlUvQXAyZytMZEUKBv+ChaoQCstA742L3Bq5mBJlW/UC4Pyw
-            ZvFAyYbs1NaEqhjtHq+4T62jTWcH/St/vKgUuFQ9LCUQhYd8DUzAow==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBJWDZwTE1haUY5ZkgybnBx
+            WEFVSzQvNjlGaDlvaHhuSTVCY2dYb0s1NW4wCmhkM0N6YlNIRTYrVkdxaHJaVHVG
+            bUEzTVV3WFQ0OWwrdEZnanJJa09Nb3MKLS0tIGduNEFtT2hCUDlkZWpWZmliZy83
+            cUoxcXp1NFF3SFpBVVdSZE5GY1VIWmsKH+cMqasOVxgCnGedaM2IAsgwOzEsOMct
+            0XjmInpppWNe+t7leEb6MqEY2cPQHL+l2L5jkZzAOfImwaIwMQnghw==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1e5fq3hwxy78psus2nfvmtmua36g0u3suk78ephw6246l974d2utsvn0hla
-    lastmodified: "2026-09-12T21:04:26Z"
-    mac: ENC[AES256_GCM,data:0ks96xQzOa8ygNDDYfKV8EJ8dWLBaC0PCnLG73NinMByF/KoWkCdwMDPC34W9xxVoTD2NiF1i/3pgCG4QFezTE7tPHPPKdYcQfwda9fkooiXW4Nh2M/sgbq/xgsy9N3qpHD/O5iILgpehb9y0DuErOyxcn2AIYizynU7m8e63pc=,iv:fvPWBsfE6YHskKzdlxJidp14dUgRR0XdMkEu6IxMMSo=,tag:5FiuIrFOg/GXvlQVy7drJQ==,type:str]
-    unencrypted_suffix: _unencrypted
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-09-14T14:19:39Z"
+    mac: ENC[AES256_GCM,data:jDdmiFg6uB1za7sAjO1CnGHdRPyUnmWCMJxqHrE3BO82pNbtm8yLOf+W6CfFKAjfZYqEE2mdKhABm7zeJgJlub3eXJbAirAFrp5jrbfHw0qsFo85Gj/BsrSDB401Aky6PxvRomz66FyX5zDDXIGAMEMlyNEadMplTfrZQQ9hQWM=,iv:UTSF72ofUyWs1LXGKQwBP60hK64UxhqTFM8QTVozGoo=,tag:OKivMDAqwWgrX+kNjvQrPg==,type:str]
    version: 3.13.2
@@ -1,12 +1,12 @@
 # Cluster-wide cleanup of stale failed/completed Jobs and Pods.
-# Runs daily at 04:00 UTC. Deletes:
-#   - Failed Jobs older than 24h (any namespace)
-#   - Completed Jobs older than 72h with no owning CronJob
-#   - Orphan pods in Error/Failed/Evicted state older than 1h
+# Runs every 2 minutes. Deletes:
+#   - Failed Jobs older than 2m (any namespace)
+#   - Completed standalone Jobs older than 2m with no owning CronJob
+#   - Orphan pods in Error/Failed/Evicted/Completed state older than 2m
 #
 # CronJob-owned Jobs are managed by failedJobsHistoryLimit/successfulJobsHistoryLimit,
-# but standalone Jobs (helm hooks, one-off runs, longhorn maintenance) have no TTL
-# and linger forever.
+# but standalone Jobs (helm hooks, one-off runs, CI TaskRuns) have no TTL
+# and linger forever. Aggressive schedule keeps the cluster clean.

 apiVersion: v1
 kind: ServiceAccount
@@ -47,7 +47,7 @@ metadata:
  labels:
    app: stale-job-cleanup
 spec:
-  schedule: "0 4 * * *"
+  schedule: "*/2 * * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
@@ -74,7 +74,9 @@ spec:
                  set -e
                  NOW=$(date +%s)

-                  echo "=== Cleaning failed Jobs older than 24h ==="
+                  TTL=120  # 2 minutes in seconds
+
+                  echo "=== Cleaning failed Jobs older than 2m ==="
                  kubectl get jobs --all-namespaces -o json | \
                    jq -r '.items[] |
                      select(.status.conditions[]?.type == "Failed") |
@@ -82,15 +84,15 @@ spec:
                      "\(.metadata.namespace) \(.metadata.name) \(.status.startTime // .status.completionTime // .metadata.creationTimestamp)"' | \
                  while read -r NS NAME TS; do
                    JOB_EPOCH=$(date -d "$TS" +%s 2>/dev/null || echo 0)
-                    AGE_H=$(( (NOW - JOB_EPOCH) / 3600 ))
-                    if [ "$AGE_H" -ge 24 ]; then
-                      echo "[delete] $NS/$NAME (failed ${AGE_H}h ago)"
+                    AGE=$(( NOW - JOB_EPOCH ))
+                    if [ "$AGE" -ge "$TTL" ]; then
+                      echo "[delete] $NS/$NAME (failed ${AGE}s ago)"
                      kubectl delete job "$NAME" -n "$NS" --cascade=foreground 2>/dev/null || true
                    fi
                  done

                  echo ""
-                  echo "=== Cleaning completed standalone Jobs older than 72h ==="
+                  echo "=== Cleaning completed standalone Jobs older than 2m ==="
                  kubectl get jobs --all-namespaces -o json | \
                    jq -r '.items[] |
                      select(.status.succeeded >= 1) |
@@ -98,20 +100,20 @@ spec:
                      "\(.metadata.namespace) \(.metadata.name) \(.status.completionTime // .metadata.creationTimestamp)"' | \
                  while read -r NS NAME TS; do
                    JOB_EPOCH=$(date -d "$TS" +%s 2>/dev/null || echo 0)
-                    AGE_H=$(( (NOW - JOB_EPOCH) / 3600 ))
-                    if [ "$AGE_H" -ge 72 ]; then
-                      echo "[delete] $NS/$NAME (completed ${AGE_H}h ago, no owner)"
+                    AGE=$(( NOW - JOB_EPOCH ))
+                    if [ "$AGE" -ge "$TTL" ]; then
+                      echo "[delete] $NS/$NAME (completed ${AGE}s ago, no owner)"
                      kubectl delete job "$NAME" -n "$NS" --cascade=foreground 2>/dev/null || true
                    fi
                  done

                  echo ""
-                  echo "=== Cleaning orphan Error/Failed/Evicted pods older than 1h ==="
-                  # Evicted pods show as Failed with reason Evicted
+                  echo "=== Cleaning orphan Error/Failed/Evicted/Completed pods older than 2m ==="
                  kubectl get pods --all-namespaces -o json | \
                    jq -r '.items[] |
                      select(
                        .status.phase == "Failed" or
+                        .status.phase == "Succeeded" or
                        (.status.reason // "") == "Evicted" or
                        (.status.containerStatuses // [] | any(.state.terminated.reason == "Error"))
                      ) |
@@ -119,9 +121,9 @@ spec:
                      "\(.metadata.namespace) \(.metadata.name) \(.metadata.creationTimestamp)"' | \
                  while read -r NS NAME TS; do
                    POD_EPOCH=$(date -d "$TS" +%s 2>/dev/null || echo 0)
-                    AGE_H=$(( (NOW - POD_EPOCH) / 3600 ))
-                    if [ "$AGE_H" -ge 1 ]; then
-                      echo "[delete] $NS/$NAME (error/evicted ${AGE_H}h ago)"
+                    AGE=$(( NOW - POD_EPOCH ))
+                    if [ "$AGE" -ge "$TTL" ]; then
+                      echo "[delete] $NS/$NAME (stale ${AGE}s ago)"
                      kubectl delete pod "$NAME" -n "$NS" --force 2>/dev/null || true
                    fi
                  done
@@ -0,0 +1,25 @@
+# CiliumNetworkPolicy for kube-apiserver access.
+#
+# Standard K8s NetworkPolicy ipBlock rules don't work for the API server
+# under Cilium — the except clause on 192.168.1.0/24 blocks the post-DNAT
+# destination even when a separate rule re-allows a /32 or subnet.
+#
+# Cilium's native `kube-apiserver` entity resolves this correctly: it
+# tracks the API server endpoints regardless of ClusterIP vs node-IP
+# routing, so the policy stays valid across node changes and NAT paths.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: {{ .Release.Name }}-apiserver
+  namespace: {{ .Release.Namespace }}
+spec:
+  endpointSelector:
+    matchLabels:
+      app: {{ .Release.Name }}
+  egress:
+  - toEntities:
+    - kube-apiserver
+    toPorts:
+    - ports:
+      - port: "6443"
+        protocol: TCP
Author	SHA1	Message	Date
rock	91ce48ffae	fix(sops): add LLM_AUTH_CLIENT_SECRET for paperless-ai-agent Authentik auth	2026-09-14 23:19:43 +09:00
rock	39036e6ff3	fix(paperless-ai): move LLM auth client secret from plaintext to SOPS - Remove hardcoded client secret from init container - Reference secret from paperless-ai-config (SOPS-encrypted) - Init container reads LLM_AUTH_CLIENT_SECRET env var - Secret key must be added to k8s/argocd/secrets/paperless-ai-secrets.enc.yaml by someone with SOPS/age key access This removes the plaintext secret from commit history and future deployments will source it securely from the encrypted Secret.	2026-09-14 23:07:52 +09:00
rock	8fa5053aef	fix: allow paperless direct LLM access, bypass gateway JWT - Add paperless namespace to llm-serving NetworkPolicy ingress - Point paperless-ai CUSTOM_BASE_URL to reasoning-predictor directly - Gateway JWT auth not yet available for service accounts - paperless-ai-agent has llm:inference role in Authentik	2026-09-14 08:09:26 +09:00
rock	41e894f79e	fix: aggressive 2m cleanup for completed jobs/pods, add PAPERLESS_USERNAME to paperless-ai - stale-job-cleanup: run every 2m, delete completed/failed jobs+pods after 2m - paperless-ai: add PAPERLESS_USERNAME=admin (required to find own user ID)	2026-09-14 07:56:05 +09:00
poimenandrock	86986013d7	fix: use CiliumNetworkPolicy for kube-apiserver egress (#49 ) Standard K8s NetworkPolicy ipBlock rules don't work under Cilium for API server access. Use Cilium-native kube-apiserver entity instead. Tested: runner pod can now reach K8s API server. Required by: homelab-frontend PR #25 (Tekton CI). --------- Co-authored-by: rock <[email protected]> Reviewed-on: #49 Co-authored-by: poimen <[email protected]>	2026-09-13 14:04:35 +00:00
poimenandrock	4b5ecfcd49	fix: allow CI runner egress to K8s API server (#48 ) CI runner needs kubectl access to create Tekton PipelineRuns for integration testing. ## Root Cause The runner egress NetworkPolicy blocks `192.168.1.0/24` (LAN). The K8s API server runs on control-plane nodes in that subnet (`192.168.1.166:6443`). kubectl from inside the DinD container times out. ## Fix Allow TCP port 6443 to `192.168.1.0/24` — scoped to control-plane API server only. ## Required By homelab-frontend PR #25 (Tekton integration testing) — CI creates PipelineRuns via kubectl. --------- Co-authored-by: rock <[email protected]> Reviewed-on: #48 Co-authored-by: poimen <[email protected]>	2026-09-13 13:54:31 +00:00