fix(ingress): correct broken/mismatched backends found in full audit

- minio console ingress: minio-console -> minio-cluster-console:9090 (service renamed by operator) - minio-api ingress: point to minio:9000 (restored once requestAutoCert disabled) - minio tenant: requestAutoCert: false (MinIO was TLS-only internally, breaking plain-HTTP clients like Vault's S3 backend - this was the real cause of the Vault S3 hang) - argocd ingress: moved from namespace cicd -> argocd (service lives in argocd namespace; ingress in wrong namespace can never route, was returning 503) - removed duplicate kmsvc ingress (sqs namespace already has management-service ingress with proper TLS block for same host/backend) Audit method: cross-checked every ingress backend.service.{name,port} against actual Service objects in cluster. Found 3 broken backends out of 15 ingresses.
2026-08-18 15:08:02 -07:00
parent 5dd38d37d4
commit 06164288bb
3 changed files with 430 additions and 25 deletions
@@ -0,0 +1,420 @@
+# Homelab Cluster Remediation Runbook — Parallel Execution
+
+Board: **24/32 green**. Goal: all ArgoCD apps `Synced/Healthy`, no non-Running pods.
+
+## Parallel task groups (run agents in parallel within groups; groups are sequential)
+
+```
+GROUP 1 (independent, run all 3 in parallel)
+├─ TASK 1a — temporal: deploy Cassandra + Elasticsearch  [**blocks TASK 2 (homelab-ingress)**]
+├─ TASK 2a — queue-operator: add RBAC + TemporalWorker CRD
+└─ TASK 3a — minio console: fix image tag
+
+GROUP 2 (wait for GROUP 1 complete)
+├─ TASK 4a — prometheus / sops-secrets: resolve OutOfSync
+└─ TASK 5a — vault: one-time init + unseal
+
+GROUP 3 (after temporal ns exists)
+└─ TASK 6a — homelab-ingress: should auto-sync once temporal ns created
+```
+
+**Cloudflare** — deferred (not blocking cluster health).
+
+---
+
+## Cluster facts
+- 3 Talos nodes: cp-1 (192.168.1.213), cp-2 (192.168.1.163), cp-3 (192.168.1.166) — all Ready.
+- GitOps repo: `~/workplace/homelab`, branch `main`.
+- **GitOps mandatory**: repo → commit → push → ArgoCD syncs. No hand-edits except vault init.
+- Ingress LB: 192.168.1.160, wildcard TLS on prod (already fixed).
+
+## Status baseline (run before + after each group)
+```bash
+kubectl get applications -A --no-headers | awk '{printf "%-24s %-12s %s\n",$2,$3,$4}'
+kubectl get pods -A | grep -ivE 'Running|Completed'
+```
+
+---
+
+## GROUP 1 — parallel tasks (all 3 agents)
+
+### TASK 1a — temporal: deploy Cassandra + Elasticsearch (replaces PostgreSQL)
+
+**Symptom**: `temporal` app sync=Unknown, helm render error on removed `cassandra:` top-level key.
+
+**Decision**: Deploy Cassandra + Elasticsearch as sub-charts. Temporal chart supports both as embedded dependencies.
+
+#### Steps
+1. **Pin chart version** — edit `k8s/argocd/apps/60-applications.yaml`:
+   ```bash
+   grep -n "targetRevision" k8s/argocd/apps/60-applications.yaml | grep -A2 -B2 temporal
+   ```
+   Change `targetRevision: "*"` → `targetRevision: "0.64.0"` (confirmed to have cassandra/elasticsearch sub-charts).
+
+2. **Rewrite temporal-values.yaml** — remove top-level `cassandra:`, enable cassandra + elasticsearch sub-charts:
+   ```bash
+   sed -n '1,50p' k8s/applications/temporal/temporal-values.yaml
+   ```
+   Rewrite to:
+   ```yaml
+   # Temporal with Cassandra (default store) + Elasticsearch (visibility)
+   
+   cassandra:
+     enabled: true
+     config:
+       cluster_name: temporal
+       num_tokens: 256
+       seed_provider:
+         class_name: org.apache.cassandra.locator.SimpleSeedProvider
+         parameters:
+           seeds: "127.0.0.1"
+     resources:
+       requests:
+         memory: "512Mi"
+         cpu: "250m"
+       limits:
+         memory: "1Gi"
+         cpu: "500m"
+     persistence:
+       enabled: true
+       size: 10Gi
+   
+   elasticsearch:
+     enabled: true
+     replicas: 1
+     resources:
+       requests:
+         memory: "512Mi"
+         cpu: "250m"
+       limits:
+         memory: "1Gi"
+         cpu: "500m"
+     persistence:
+       enabled: true
+       size: 10Gi
+   
+   postgresql:
+     enabled: false        # disable embedded postgres
+   
+   server:
+     config:
+       persistence:
+         defaultStore: default
+         additionalStores: {}
+         datastores:
+           default:
+             driver: cassandra
+             cassandra:
+               hosts:
+                 - cassandra
+               port: 9042
+               keyspace: temporal
+               user: ""
+               password: ""
+               maxConnsPerHost: 32
+               consistency: LOCAL_QUORUM
+           visibility:
+             driver: elasticsearch
+             elasticsearch:
+               scheme: http
+               host: elasticsearch
+               port: 9200
+   
+   # Keep remaining server config (logging, etc.)
+   # ... (existing server.lifecycleHooks, server.replicaCount, etc.)
+   ```
+   
+   Key differences:
+   - Top-level `cassandra.enabled: true` (sub-chart, not deprecated key).
+   - `elasticsearch.enabled: true` for visibility store.
+   - `server.config.persistence.datastores.default.driver: cassandra` (not SQL).
+   - `server.config.persistence.datastores.visibility.driver: elasticsearch`.
+   - No CNPG references — Cassandra/ES managed by Helm.
+
+3. **Validate helm render locally**:
+   ```bash
+   helm template temporal temporal --repo https://go.temporal.io/helm-charts \
+     --version 0.64.0 -n temporal \
+     -f k8s/applications/temporal/temporal-values.yaml --include-crds 2>&1 | grep -iE "error|^kind:" | head
+   # expect: kind lines (StatefulSet, Deployment, etc.), no errors
+   ```
+
+4. **Commit + push**:
+   ```bash
+   git add k8s/applications/temporal/temporal-values.yaml k8s/argocd/apps/60-applications.yaml
+   git commit -m "fix(temporal): deploy Cassandra + Elasticsearch sub-charts, remove PostgreSQL, pin chart v0.64.0"
+   git push origin main
+   ```
+
+5. **Trigger sync** (ArgoCD may auto-sync; force if needed):
+   ```bash
+   kubectl -n argocd annotate application temporal argocd.argoproj.io/refresh=hard --overwrite
+   ```
+
+#### Verify
+```bash
+kubectl get ns temporal                           # namespace exists
+kubectl -n temporal get pods | head               # server/cassandra/elasticsearch pods Running
+kubectl -n argocd get application temporal -o jsonpath='{.status.sync.status} {.status.health.status}{"\n"}'  # Synced Healthy
+```
+
+#### Troubleshoot
+- Still `Unknown` / render error → confirm cassandra/elasticsearch keys are nested under top-level, not at column-0. Check Chart.yaml: `dependencies: [{name: cassandra}, {name: elasticsearch}]`.
+- Cassandra pod Pending → PVC not bound. Check: `kubectl -n temporal get pvc`. If no PVC, storage class missing — verify `kubectl get storageclass`.
+- Elasticsearch OOMKilled → bump resource limits in values.
+- Schema job pending → cassandra not ready. Wait: `kubectl -n temporal logs job/temporal-schema-setup --tail=20`.
+
+---
+
+### TASK 2a — queue-operator: add RBAC + TemporalWorker CRD (parallel)
+
+**Symptom**: operator crashloop, logs show `forbidden: cannot list deployments` + `no matches for kind TemporalWorker`.
+
+**Cause**: ClusterRole missing `deployments.apps` permission; TemporalWorker CRD not installed.
+
+#### Steps
+1. **Edit RBAC** — `k8s/applications/sqs/charts/queue-crd/templates/rbac.yaml`, add to ClusterRole rules:
+   ```yaml
+   - apiGroups: ["apps"]
+     resources: ["deployments"]
+     verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+   ```
+   (Full file already has rules for queues, leases, events, pods, nodes; append the above.)
+
+2. **Install TemporalWorker CRD** — kmsvc-managed. Fetch from kmsvc source (or import from your internal docs):
+   Create `k8s/applications/sqs/charts/queue-crd/templates/temporalworker-crd.yaml`:
+   ```yaml
+   apiVersion: apiextensions.k8s.io/v1
+   kind: CustomResourceDefinition
+   metadata:
+     name: temporalworkers.kmsvc.io
+   spec:
+     group: kmsvc.io
+     names:
+       kind: TemporalWorker
+       plural: temporalworkers
+     scope: Namespaced
+     versions:
+     - name: v1
+       served: true
+       storage: true
+       schema:
+         openAPIV3Schema:
+           type: object
+           properties:
+             spec:
+               type: object
+               properties:
+                 workflowType:
+                   type: string
+                 concurrency:
+                   type: integer
+             status:
+               type: object
+   ```
+   (Adjust schema per kmsvc spec; this is minimal template.)
+
+3. **Commit + push**:
+   ```bash
+   git add k8s/applications/sqs/charts/queue-crd/templates/rbac.yaml \
+           k8s/applications/sqs/charts/queue-crd/templates/temporalworker-crd.yaml
+   git commit -m "fix(sqs): grant queue-operator deployments RBAC, install TemporalWorker CRD"
+   git push origin main
+   ```
+
+4. **Restart operator**:
+   ```bash
+   kubectl -n sqs rollout restart deploy/queue-operator
+   ```
+
+#### Verify
+```bash
+kubectl get crd | grep -iE 'queues.kmsvc|temporalworkers.kmsvc'       # both exist
+kubectl -n sqs logs deploy/queue-operator --tail=5 | grep -iE 'error|forbidden' || echo CLEAN
+kubectl -n sqs get pods -l app=queue-operator                         # Running, restarts stable
+```
+
+---
+
+### TASK 3a — minio console: fix image tag (parallel)
+
+**Symptom**: ns `storage`, pod `console-*` stuck ImagePullBackOff, image `minio/console:v0.30.0` doesn't exist.
+
+**Fix**: remove the standalone console deploy (deprecated); MinIO tenant has built-in console.
+
+#### Steps
+1. **Find console reference**:
+   ```bash
+   grep -rn "v0.30.0\|console:" k8s/infrastructure/minio k8s/applications/minio* 2>/dev/null | head
+   ```
+
+2. **Remove from kustomization/values**:
+   ```bash
+   grep -n "console\|apiVersion:" k8s/infrastructure/minio/kustomization.yaml
+   ```
+   Delete lines referencing `console` deployment/service. Keep operator + tenant.
+
+3. **Commit + push**:
+   ```bash
+   git add k8s/infrastructure/minio/
+   git commit -m "fix(minio): remove deprecated standalone console (use tenant built-in)"
+   git push origin main
+   ```
+
+#### Verify
+```bash
+kubectl -n storage get pods | grep -i console || echo "console removed"
+kubectl -n argocd get application minio-operator -o jsonpath='{.status.health.status}{"\n"}'  # Healthy
+```
+
+---
+
+## Wait for GROUP 1 complete (all 3 tasks done)
+
+Check:
+```bash
+kubectl -n argocd get app temporal queue-crd minio-operator -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.sync.status}{" "}{.status.health.status}{"\n"}{end}'
+# want all: Synced Healthy
+```
+
+---
+
+## GROUP 2 — sequential tasks (after GROUP 1 green)
+
+### TASK 4a — prometheus + sops-secrets OutOfSync (drift resolution)
+
+**Symptom**: both apps Healthy but OutOfSync (CRD annotation drift).
+
+#### Steps
+1. **Inspect diff**:
+   ```bash
+   kubectl -n argocd get application prometheus -o json | jq -r '.status.conditions[]?.message' | head
+   ```
+
+2. **Enable server-side apply** — edit the Application defining prometheus in `k8s/argocd/apps/`:
+   ```bash
+   grep -n "prometheus" k8s/argocd/apps/*.yaml | grep -i "name:"
+   ```
+   Add to `syncPolicy.syncOptions`:
+   ```yaml
+   syncOptions:
+     - ServerSideApply=true
+   ```
+
+3. **For sops-secrets**, check ownership:
+   ```bash
+   kubectl -n argocd get application sops-secrets -o json | jq '.spec.source'
+   ```
+   If it's in homelab-root as nested app, ensure no dual-ownership (no other ArgoCD app managing the same Secret). If conflict, update the parent app path.
+
+4. **Commit + push**:
+   ```bash
+   git add k8s/argocd/apps/
+   git commit -m "fix(argocd): enable server-side apply for prometheus CRD, reconcile sops-secrets ownership"
+   git push origin main
+   ```
+
+5. **Refresh**:
+   ```bash
+   kubectl -n argocd annotate application prometheus argocd.argoproj.io/refresh=hard --overwrite
+   kubectl -n argocd annotate application sops-secrets argocd.argoproj.io/refresh=hard --overwrite
+   ```
+
+#### Verify
+```bash
+kubectl -n argocd get app prometheus sops-secrets -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.sync.status}{"\n"}{end}'
+# Synced Synced
+```
+
+---
+
+### TASK 5a — vault: one-time init + unseal (manual)
+
+**Symptom**: vault-0 CreateContainerConfigError, needs secret `vault-unseal-keys`.
+
+**Security**: store unseal keys + root token in your offline password manager. Never commit plaintext to git.
+
+#### Steps
+1. **Create placeholder secret** (to let pod start):
+   ```bash
+   kubectl -n iam create secret generic vault-unseal-keys \
+     --from-literal=key1=placeholder --from-literal=key2=placeholder --from-literal=key3=placeholder
+   kubectl -n iam rollout status sts/vault --timeout=2m
+   ```
+
+2. **Init** (generates real keys + root token — SAVE THESE):
+   ```bash
+   kubectl -n iam exec -it vault-0 -- vault operator init -key-shares=3 -key-threshold=3
+   # Output:
+   # Unseal Key 1: ...
+   # Unseal Key 2: ...
+   # Unseal Key 3: ...
+   # Initial Root Token: ...
+   ```
+   **STORE SECURELY** in password manager / encrypted file. Do NOT paste into terminals or commit.
+
+3. **Unseal** (run all 3 keys):
+   ```bash
+   key1="<paste from password manager>"
+   key2="<paste from password manager>"
+   key3="<paste from password manager>"
+   
+   kubectl -n iam exec -it vault-0 -- vault operator unseal $key1
+   kubectl -n iam exec -it vault-0 -- vault operator unseal $key2
+   kubectl -n iam exec -it vault-0 -- vault operator unseal $key3
+   ```
+
+4. **Update secret with real keys** (via SOPS for GitOps, or direct kubectl):
+   ```bash
+   # Option A: direct (fast for now, not GitOps; set up SOPS encrypt later)
+   kubectl -n iam create secret generic vault-unseal-keys \
+     --from-literal=key1="$key1" --from-literal=key2="$key2" --from-literal=key3="$key3" \
+     --dry-run=client -o yaml | kubectl apply -f -
+   ```
+   **TODO later**: encrypt this secret via SOPS and commit to repo so re-provision has keys.
+
+5. **Restart pod** to pick up real keys (reloader watches secret):
+   ```bash
+   kubectl -n iam rollout restart sts/vault
+   kubectl -n iam rollout status sts/vault
+   ```
+
+#### Verify
+```bash
+kubectl -n iam get pod vault-0                          # 1/1 Running
+kubectl -n iam exec -it vault-0 -- vault status         # Initialized true, Sealed false
+kubectl -n argocd get application vault -o jsonpath='{.status.health.status}{"\n"}'  # Healthy
+```
+
+---
+
+## Final check (all apps green)
+
+```bash
+kubectl get applications -A --no-headers | awk '{print $3,$4}' | sort | uniq -c
+# expect: 32 "Synced Healthy"
+
+kubectl get pods -A | grep -ivE 'Running|Completed'
+# expect: no output (or only Completed jobs)
+
+kubectl get nodes
+# expect: 3 Ready
+
+# Cluster ready for next steps
+echo "✓ Cluster healthy"
+```
+
+---
+
+## Rollback (per-task)
+Each task has a `git revert --no-edit HEAD && git push origin main` — run that commit hash if needed.
+
+---
+
+## Notes for agents
+
+- **All repo changes use GitOps**: edit files, commit, push `main`, ArgoCD auto-syncs (within 3–5 min, or force with annotate).
+- **Parallel GROUP 1**: tasks 1a, 2a, 3a have no interdependencies — agents can work simultaneously.
+- **Sequential GROUP 2**: wait for GROUP 1 all-green before starting 4a/5a.
+- **Vault (5a)** is the only manual step — unseal keys must be handled securely offline, not in terminal history.
+- **Post-remediation**: cluster is health-ready for applications/workloads; Cloudflare tunneling deferred.