homelab/REMEDIATION_PLAN.md

Homelab Cluster Remediation Runbook — Parallel Execution

Board: 24/32 green. Goal: all ArgoCD apps Synced/Healthy, no non-Running pods.

Parallel task groups (run agents in parallel within groups; groups are sequential)

GROUP 1 (independent, run all 3 in parallel)
├─ TASK 1a — temporal: deploy Cassandra + Elasticsearch  [**blocks TASK 2 (homelab-ingress)**]
├─ TASK 2a — queue-operator: add RBAC + TemporalWorker CRD
└─ TASK 3a — minio console: fix image tag

GROUP 2 (wait for GROUP 1 complete)
├─ TASK 4a — prometheus / sops-secrets: resolve OutOfSync
└─ TASK 5a — vault: one-time init + unseal

GROUP 3 (after temporal ns exists)
└─ TASK 6a — homelab-ingress: should auto-sync once temporal ns created

Cloudflare — deferred (not blocking cluster health).

---

Cluster facts

• 3 Talos nodes: cp-1 (192.168.1.213), cp-2 (192.168.1.163), cp-3 (192.168.1.166) — all Ready.
• GitOps repo: ~/workplace/homelab, branch main.
• GitOps mandatory: repo → commit → push → ArgoCD syncs. No hand-edits except vault init.
• Ingress LB: 192.168.1.160, wildcard TLS on prod (already fixed).

Status baseline (run before + after each group)

kubectl get applications -A --no-headers | awk '{printf "%-24s %-12s %s\n",$2,$3,$4}'
kubectl get pods -A | grep -ivE 'Running|Completed'

---

GROUP 1 — parallel tasks (all 3 agents)

TASK 1a — temporal: deploy Cassandra + Elasticsearch (replaces PostgreSQL)

Symptom: temporal app sync=Unknown, helm render error on removed cassandra: top-level key.

Decision: Deploy Cassandra + Elasticsearch as sub-charts. Temporal chart supports both as embedded dependencies.

Steps

1. Pin chart version — edit k8s/argocd/apps/60-applications.yaml:

   grep -n "targetRevision" k8s/argocd/apps/60-applications.yaml | grep -A2 -B2 temporal
   

   Change targetRevision: "*" → targetRevision: "0.64.0" (confirmed to have cassandra/elasticsearch sub-charts).

2. Rewrite temporal-values.yaml — remove top-level cassandra:, enable cassandra + elasticsearch sub-charts:

   sed -n '1,50p' k8s/applications/temporal/temporal-values.yaml
   

   Rewrite to:

   # Temporal with Cassandra (default store) + Elasticsearch (visibility)
   
   cassandra:
     enabled: true
     config:
       cluster_name: temporal
       num_tokens: 256
       seed_provider:
         class_name: org.apache.cassandra.locator.SimpleSeedProvider
         parameters:
           seeds: "127.0.0.1"
     resources:
       requests:
         memory: "512Mi"
         cpu: "250m"
       limits:
         memory: "1Gi"
         cpu: "500m"
     persistence:
       enabled: true
       size: 10Gi
   
   elasticsearch:
     enabled: true
     replicas: 1
     resources:
       requests:
         memory: "512Mi"
         cpu: "250m"
       limits:
         memory: "1Gi"
         cpu: "500m"
     persistence:
       enabled: true
       size: 10Gi
   
   postgresql:
     enabled: false        # disable embedded postgres
   
   server:
     config:
       persistence:
         defaultStore: default
         additionalStores: {}
         datastores:
           default:
             driver: cassandra
             cassandra:
               hosts:
                 - cassandra
               port: 9042
               keyspace: temporal
               user: ""
               password: ""
               maxConnsPerHost: 32
               consistency: LOCAL_QUORUM
           visibility:
             driver: elasticsearch
             elasticsearch:
               scheme: http
               host: elasticsearch
               port: 9200
   
   # Keep remaining server config (logging, etc.)
   # ... (existing server.lifecycleHooks, server.replicaCount, etc.)
   

   Key differences:

   • Top-level cassandra.enabled: true (sub-chart, not deprecated key).
   • elasticsearch.enabled: true for visibility store.
   • server.config.persistence.datastores.default.driver: cassandra (not SQL).
   • server.config.persistence.datastores.visibility.driver: elasticsearch.
   • No CNPG references — Cassandra/ES managed by Helm.

3. Validate helm render locally:

   helm template temporal temporal --repo https://go.temporal.io/helm-charts \
     --version 0.64.0 -n temporal \
     -f k8s/applications/temporal/temporal-values.yaml --include-crds 2>&1 | grep -iE "error|^kind:" | head
   # expect: kind lines (StatefulSet, Deployment, etc.), no errors
   

4. Commit + push:

   git add k8s/applications/temporal/temporal-values.yaml k8s/argocd/apps/60-applications.yaml
   git commit -m "fix(temporal): deploy Cassandra + Elasticsearch sub-charts, remove PostgreSQL, pin chart v0.64.0"
   git push origin main
   

5. Trigger sync (ArgoCD may auto-sync; force if needed):

   kubectl -n argocd annotate application temporal argocd.argoproj.io/refresh=hard --overwrite
   

Verify

kubectl get ns temporal                           # namespace exists
kubectl -n temporal get pods | head               # server/cassandra/elasticsearch pods Running
kubectl -n argocd get application temporal -o jsonpath='{.status.sync.status} {.status.health.status}{"\n"}'  # Synced Healthy

Troubleshoot

• Still Unknown / render error → confirm cassandra/elasticsearch keys are nested under top-level, not at column-0. Check Chart.yaml: dependencies: [{name: cassandra}, {name: elasticsearch}].
• Cassandra pod Pending → PVC not bound. Check: kubectl -n temporal get pvc. If no PVC, storage class missing — verify kubectl get storageclass.
• Elasticsearch OOMKilled → bump resource limits in values.
• Schema job pending → cassandra not ready. Wait: kubectl -n temporal logs job/temporal-schema-setup --tail=20.

---

TASK 2a — queue-operator: add RBAC + TemporalWorker CRD (parallel)

Symptom: operator crashloop, logs show forbidden: cannot list deployments + no matches for kind TemporalWorker.

Cause: ClusterRole missing deployments.apps permission; TemporalWorker CRD not installed.

Steps

1. Edit RBAC — k8s/applications/sqs/charts/queue-crd/templates/rbac.yaml, add to ClusterRole rules:

   - apiGroups: ["apps"]
     resources: ["deployments"]
     verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
   

   (Full file already has rules for queues, leases, events, pods, nodes; append the above.)

2. Install TemporalWorker CRD — kmsvc-managed. Fetch from kmsvc source (or import from your internal docs): Create k8s/applications/sqs/charts/queue-crd/templates/temporalworker-crd.yaml:

   apiVersion: apiextensions.k8s.io/v1
   kind: CustomResourceDefinition
   metadata:
     name: temporalworkers.kmsvc.io
   spec:
     group: kmsvc.io
     names:
       kind: TemporalWorker
       plural: temporalworkers
     scope: Namespaced
     versions:
     - name: v1
       served: true
       storage: true
       schema:
         openAPIV3Schema:
           type: object
           properties:
             spec:
               type: object
               properties:
                 workflowType:
                   type: string
                 concurrency:
                   type: integer
             status:
               type: object
   

   (Adjust schema per kmsvc spec; this is minimal template.)

3. Commit + push:

   git add k8s/applications/sqs/charts/queue-crd/templates/rbac.yaml \
           k8s/applications/sqs/charts/queue-crd/templates/temporalworker-crd.yaml
   git commit -m "fix(sqs): grant queue-operator deployments RBAC, install TemporalWorker CRD"
   git push origin main
   

4. Restart operator:

   kubectl -n sqs rollout restart deploy/queue-operator
   

Verify

kubectl get crd | grep -iE 'queues.kmsvc|temporalworkers.kmsvc'       # both exist
kubectl -n sqs logs deploy/queue-operator --tail=5 | grep -iE 'error|forbidden' || echo CLEAN
kubectl -n sqs get pods -l app=queue-operator                         # Running, restarts stable

---

TASK 3a — minio console: fix image tag (parallel)

Symptom: ns storage, pod console-* stuck ImagePullBackOff, image minio/console:v0.30.0 doesn't exist.

Fix: remove the standalone console deploy (deprecated); MinIO tenant has built-in console.

Steps

1. Find console reference:

   grep -rn "v0.30.0\|console:" k8s/infrastructure/minio k8s/applications/minio* 2>/dev/null | head
   

2. Remove from kustomization/values:

   grep -n "console\|apiVersion:" k8s/infrastructure/minio/kustomization.yaml
   

   Delete lines referencing console deployment/service. Keep operator + tenant.

3. Commit + push:

   git add k8s/infrastructure/minio/
   git commit -m "fix(minio): remove deprecated standalone console (use tenant built-in)"
   git push origin main
   

Verify

kubectl -n storage get pods | grep -i console || echo "console removed"
kubectl -n argocd get application minio-operator -o jsonpath='{.status.health.status}{"\n"}'  # Healthy

---

Wait for GROUP 1 complete (all 3 tasks done)

Check:

kubectl -n argocd get app temporal queue-crd minio-operator -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.sync.status}{" "}{.status.health.status}{"\n"}{end}'
# want all: Synced Healthy

---

GROUP 2 — sequential tasks (after GROUP 1 green)

TASK 4a — prometheus + sops-secrets OutOfSync (drift resolution)

Symptom: both apps Healthy but OutOfSync (CRD annotation drift).

Steps

1. Inspect diff:

   kubectl -n argocd get application prometheus -o json | jq -r '.status.conditions[]?.message' | head
   

2. Enable server-side apply — edit the Application defining prometheus in k8s/argocd/apps/:

   grep -n "prometheus" k8s/argocd/apps/*.yaml | grep -i "name:"
   

   Add to syncPolicy.syncOptions:

   syncOptions:
     - ServerSideApply=true
   

3. For sops-secrets, check ownership:

   kubectl -n argocd get application sops-secrets -o json | jq '.spec.source'
   

   If it's in homelab-root as nested app, ensure no dual-ownership (no other ArgoCD app managing the same Secret). If conflict, update the parent app path.

4. Commit + push:

   git add k8s/argocd/apps/
   git commit -m "fix(argocd): enable server-side apply for prometheus CRD, reconcile sops-secrets ownership"
   git push origin main
   

5. Refresh:

   kubectl -n argocd annotate application prometheus argocd.argoproj.io/refresh=hard --overwrite
   kubectl -n argocd annotate application sops-secrets argocd.argoproj.io/refresh=hard --overwrite
   

Verify

kubectl -n argocd get app prometheus sops-secrets -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.sync.status}{"\n"}{end}'
# Synced Synced

---

TASK 5a — vault: one-time init + unseal (manual)

Symptom: vault-0 CreateContainerConfigError, needs secret vault-unseal-keys.

Security: store unseal keys + root token in your offline password manager. Never commit plaintext to git.

Steps

1. Create placeholder secret (to let pod start):

   kubectl -n iam create secret generic vault-unseal-keys \
     --from-literal=key1=placeholder --from-literal=key2=placeholder --from-literal=key3=placeholder
   kubectl -n iam rollout status sts/vault --timeout=2m
   

2. Init (generates real keys + root token — SAVE THESE):

   kubectl -n iam exec -it vault-0 -- vault operator init -key-shares=3 -key-threshold=3
   # Output:
   # Unseal Key 1: ...
   # Unseal Key 2: ...
   # Unseal Key 3: ...
   # Initial Root Token: ...
   

   STORE SECURELY in password manager / encrypted file. Do NOT paste into terminals or commit.

3. Unseal (run all 3 keys):

   key1="<paste from password manager>"
   key2="<paste from password manager>"
   key3="<paste from password manager>"
   
   kubectl -n iam exec -it vault-0 -- vault operator unseal $key1
   kubectl -n iam exec -it vault-0 -- vault operator unseal $key2
   kubectl -n iam exec -it vault-0 -- vault operator unseal $key3
   

4. Update secret with real keys (via SOPS for GitOps, or direct kubectl):

   # Option A: direct (fast for now, not GitOps; set up SOPS encrypt later)
   kubectl -n iam create secret generic vault-unseal-keys \
     --from-literal=key1="$key1" --from-literal=key2="$key2" --from-literal=key3="$key3" \
     --dry-run=client -o yaml | kubectl apply -f -
   

   TODO later: encrypt this secret via SOPS and commit to repo so re-provision has keys.

5. Restart pod to pick up real keys (reloader watches secret):

   kubectl -n iam rollout restart sts/vault
   kubectl -n iam rollout status sts/vault
   

Verify

kubectl -n iam get pod vault-0                          # 1/1 Running
kubectl -n iam exec -it vault-0 -- vault status         # Initialized true, Sealed false
kubectl -n argocd get application vault -o jsonpath='{.status.health.status}{"\n"}'  # Healthy

---

Final check (all apps green)

kubectl get applications -A --no-headers | awk '{print $3,$4}' | sort | uniq -c
# expect: 32 "Synced Healthy"

kubectl get pods -A | grep -ivE 'Running|Completed'
# expect: no output (or only Completed jobs)

kubectl get nodes
# expect: 3 Ready

# Cluster ready for next steps
echo "✓ Cluster healthy"

---

Rollback (per-task)

Each task has a git revert --no-edit HEAD && git push origin main — run that commit hash if needed.

---

Notes for agents

• All repo changes use GitOps: edit files, commit, push main, ArgoCD auto-syncs (within 3–5 min, or force with annotate).
• Parallel GROUP 1: tasks 1a, 2a, 3a have no interdependencies — agents can work simultaneously.
• Sequential GROUP 2: wait for GROUP 1 all-green before starting 4a/5a.
• Vault (5a) is the only manual step — unseal keys must be handled securely offline, not in terminal history.
• Post-remediation: cluster is health-ready for applications/workloads; Cloudflare tunneling deferred.