homelab/REMEDIATION_PLAN.md

# Homelab Cluster Remediation Runbook — Parallel Execution

Board: **24/32 green**. Goal: all ArgoCD apps `Synced/Healthy`, no non-Running pods.

## Parallel task groups (run agents in parallel within groups; groups are sequential)

```
GROUP 1 (independent, run all 3 in parallel)
├─ TASK 1a — temporal: deploy Cassandra + Elasticsearch  [**blocks TASK 2 (homelab-ingress)**]
├─ TASK 2a — queue-operator: add RBAC + TemporalWorker CRD
└─ TASK 3a — minio console: fix image tag

GROUP 2 (wait for GROUP 1 complete)
├─ TASK 4a — prometheus / sops-secrets: resolve OutOfSync
└─ TASK 5a — vault: one-time init + unseal

GROUP 3 (after temporal ns exists)
└─ TASK 6a — homelab-ingress: should auto-sync once temporal ns created
```

**Cloudflare** — deferred (not blocking cluster health).

---

## Cluster facts
- 3 Talos nodes: cp-1 (192.168.1.213), cp-2 (192.168.1.163), cp-3 (192.168.1.166) — all Ready.
- GitOps repo: `~/workplace/homelab`, branch `main`.
- **GitOps mandatory**: repo → commit → push → ArgoCD syncs. No hand-edits except vault init.
- Ingress LB: 192.168.1.160, wildcard TLS on prod (already fixed).

## Status baseline (run before + after each group)
```bash
kubectl get applications -A --no-headers | awk '{printf "%-24s %-12s %s\n",$2,$3,$4}'
kubectl get pods -A | grep -ivE 'Running|Completed'
```

---

## GROUP 1 — parallel tasks (all 3 agents)

### TASK 1a — temporal: deploy Cassandra + Elasticsearch (replaces PostgreSQL)

**Symptom**: `temporal` app sync=Unknown, helm render error on removed `cassandra:` top-level key.

**Decision**: Deploy Cassandra + Elasticsearch as sub-charts. Temporal chart supports both as embedded dependencies.

#### Steps
1. **Pin chart version** — edit `k8s/argocd/apps/60-applications.yaml`:
   ```bash
   grep -n "targetRevision" k8s/argocd/apps/60-applications.yaml | grep -A2 -B2 temporal
   ```
   Change `targetRevision: "*"` → `targetRevision: "0.64.0"` (confirmed to have cassandra/elasticsearch sub-charts).

2. **Rewrite temporal-values.yaml** — remove top-level `cassandra:`, enable cassandra + elasticsearch sub-charts:
   ```bash
   sed -n '1,50p' k8s/applications/temporal/temporal-values.yaml
   ```
   Rewrite to:
   ```yaml
   # Temporal with Cassandra (default store) + Elasticsearch (visibility)
   
   cassandra:
     enabled: true
     config:
       cluster_name: temporal
       num_tokens: 256
       seed_provider:
         class_name: org.apache.cassandra.locator.SimpleSeedProvider
         parameters:
           seeds: "127.0.0.1"
     resources:
       requests:
         memory: "512Mi"
         cpu: "250m"
       limits:
         memory: "1Gi"
         cpu: "500m"
     persistence:
       enabled: true
       size: 10Gi
   
   elasticsearch:
     enabled: true
     replicas: 1
     resources:
       requests:
         memory: "512Mi"
         cpu: "250m"
       limits:
         memory: "1Gi"
         cpu: "500m"
     persistence:
       enabled: true
       size: 10Gi
   
   postgresql:
     enabled: false        # disable embedded postgres
   
   server:
     config:
       persistence:
         defaultStore: default
         additionalStores: {}
         datastores:
           default:
             driver: cassandra
             cassandra:
               hosts:
                 - cassandra
               port: 9042
               keyspace: temporal
               user: ""
               password: ""
               maxConnsPerHost: 32
               consistency: LOCAL_QUORUM
           visibility:
             driver: elasticsearch
             elasticsearch:
               scheme: http
               host: elasticsearch
               port: 9200
   
   # Keep remaining server config (logging, etc.)
   # ... (existing server.lifecycleHooks, server.replicaCount, etc.)
   ```
   
   Key differences:
   - Top-level `cassandra.enabled: true` (sub-chart, not deprecated key).
   - `elasticsearch.enabled: true` for visibility store.
   - `server.config.persistence.datastores.default.driver: cassandra` (not SQL).
   - `server.config.persistence.datastores.visibility.driver: elasticsearch`.
   - No CNPG references — Cassandra/ES managed by Helm.

3. **Validate helm render locally**:
   ```bash
   helm template temporal temporal --repo https://go.temporal.io/helm-charts \
     --version 0.64.0 -n temporal \
     -f k8s/applications/temporal/temporal-values.yaml --include-crds 2>&1 | grep -iE "error|^kind:" | head
   # expect: kind lines (StatefulSet, Deployment, etc.), no errors
   ```

4. **Commit + push**:
   ```bash
   git add k8s/applications/temporal/temporal-values.yaml k8s/argocd/apps/60-applications.yaml
   git commit -m "fix(temporal): deploy Cassandra + Elasticsearch sub-charts, remove PostgreSQL, pin chart v0.64.0"
   git push origin main
   ```

5. **Trigger sync** (ArgoCD may auto-sync; force if needed):
   ```bash
   kubectl -n argocd annotate application temporal argocd.argoproj.io/refresh=hard --overwrite
   ```

#### Verify
```bash
kubectl get ns temporal                           # namespace exists
kubectl -n temporal get pods | head               # server/cassandra/elasticsearch pods Running
kubectl -n argocd get application temporal -o jsonpath='{.status.sync.status} {.status.health.status}{"\n"}'  # Synced Healthy
```

#### Troubleshoot
- Still `Unknown` / render error → confirm cassandra/elasticsearch keys are nested under top-level, not at column-0. Check Chart.yaml: `dependencies: [{name: cassandra}, {name: elasticsearch}]`.
- Cassandra pod Pending → PVC not bound. Check: `kubectl -n temporal get pvc`. If no PVC, storage class missing — verify `kubectl get storageclass`.
- Elasticsearch OOMKilled → bump resource limits in values.
- Schema job pending → cassandra not ready. Wait: `kubectl -n temporal logs job/temporal-schema-setup --tail=20`.

---

### TASK 2a — queue-operator: add RBAC + TemporalWorker CRD (parallel)

**Symptom**: operator crashloop, logs show `forbidden: cannot list deployments` + `no matches for kind TemporalWorker`.

**Cause**: ClusterRole missing `deployments.apps` permission; TemporalWorker CRD not installed.

#### Steps
1. **Edit RBAC** — `k8s/applications/sqs/charts/queue-crd/templates/rbac.yaml`, add to ClusterRole rules:
   ```yaml
   - apiGroups: ["apps"]
     resources: ["deployments"]
     verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
   ```
   (Full file already has rules for queues, leases, events, pods, nodes; append the above.)

2. **Install TemporalWorker CRD** — kmsvc-managed. Fetch from kmsvc source (or import from your internal docs):
   Create `k8s/applications/sqs/charts/queue-crd/templates/temporalworker-crd.yaml`:
   ```yaml
   apiVersion: apiextensions.k8s.io/v1
   kind: CustomResourceDefinition
   metadata:
     name: temporalworkers.kmsvc.io
   spec:
     group: kmsvc.io
     names:
       kind: TemporalWorker
       plural: temporalworkers
     scope: Namespaced
     versions:
     - name: v1
       served: true
       storage: true
       schema:
         openAPIV3Schema:
           type: object
           properties:
             spec:
               type: object
               properties:
                 workflowType:
                   type: string
                 concurrency:
                   type: integer
             status:
               type: object
   ```
   (Adjust schema per kmsvc spec; this is minimal template.)

3. **Commit + push**:
   ```bash
   git add k8s/applications/sqs/charts/queue-crd/templates/rbac.yaml \
           k8s/applications/sqs/charts/queue-crd/templates/temporalworker-crd.yaml
   git commit -m "fix(sqs): grant queue-operator deployments RBAC, install TemporalWorker CRD"
   git push origin main
   ```

4. **Restart operator**:
   ```bash
   kubectl -n sqs rollout restart deploy/queue-operator
   ```

#### Verify
```bash
kubectl get crd | grep -iE 'queues.kmsvc|temporalworkers.kmsvc'       # both exist
kubectl -n sqs logs deploy/queue-operator --tail=5 | grep -iE 'error|forbidden' || echo CLEAN
kubectl -n sqs get pods -l app=queue-operator                         # Running, restarts stable
```

---

### TASK 3a — minio console: fix image tag (parallel)

**Symptom**: ns `storage`, pod `console-*` stuck ImagePullBackOff, image `minio/console:v0.30.0` doesn't exist.

**Fix**: remove the standalone console deploy (deprecated); MinIO tenant has built-in console.

#### Steps
1. **Find console reference**:
   ```bash
   grep -rn "v0.30.0\|console:" k8s/infrastructure/minio k8s/applications/minio* 2>/dev/null | head
   ```

2. **Remove from kustomization/values**:
   ```bash
   grep -n "console\|apiVersion:" k8s/infrastructure/minio/kustomization.yaml
   ```
   Delete lines referencing `console` deployment/service. Keep operator + tenant.

3. **Commit + push**:
   ```bash
   git add k8s/infrastructure/minio/
   git commit -m "fix(minio): remove deprecated standalone console (use tenant built-in)"
   git push origin main
   ```

#### Verify
```bash
kubectl -n storage get pods | grep -i console || echo "console removed"
kubectl -n argocd get application minio-operator -o jsonpath='{.status.health.status}{"\n"}'  # Healthy
```

---

## Wait for GROUP 1 complete (all 3 tasks done)

Check:
```bash
kubectl -n argocd get app temporal queue-crd minio-operator -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.sync.status}{" "}{.status.health.status}{"\n"}{end}'
# want all: Synced Healthy
```

---

## GROUP 2 — sequential tasks (after GROUP 1 green)

### TASK 4a — prometheus + sops-secrets OutOfSync (drift resolution)

**Symptom**: both apps Healthy but OutOfSync (CRD annotation drift).

#### Steps
1. **Inspect diff**:
   ```bash
   kubectl -n argocd get application prometheus -o json | jq -r '.status.conditions[]?.message' | head
   ```

2. **Enable server-side apply** — edit the Application defining prometheus in `k8s/argocd/apps/`:
   ```bash
   grep -n "prometheus" k8s/argocd/apps/*.yaml | grep -i "name:"
   ```
   Add to `syncPolicy.syncOptions`:
   ```yaml
   syncOptions:
     - ServerSideApply=true
   ```

3. **For sops-secrets**, check ownership:
   ```bash
   kubectl -n argocd get application sops-secrets -o json | jq '.spec.source'
   ```
   If it's in homelab-root as nested app, ensure no dual-ownership (no other ArgoCD app managing the same Secret). If conflict, update the parent app path.

4. **Commit + push**:
   ```bash
   git add k8s/argocd/apps/
   git commit -m "fix(argocd): enable server-side apply for prometheus CRD, reconcile sops-secrets ownership"
   git push origin main
   ```

5. **Refresh**:
   ```bash
   kubectl -n argocd annotate application prometheus argocd.argoproj.io/refresh=hard --overwrite
   kubectl -n argocd annotate application sops-secrets argocd.argoproj.io/refresh=hard --overwrite
   ```

#### Verify
```bash
kubectl -n argocd get app prometheus sops-secrets -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.sync.status}{"\n"}{end}'
# Synced Synced
```

---

### TASK 5a — vault: one-time init + unseal (manual)

**Symptom**: vault-0 CreateContainerConfigError, needs secret `vault-unseal-keys`.

**Security**: store unseal keys + root token in your offline password manager. Never commit plaintext to git.

#### Steps
1. **Create placeholder secret** (to let pod start):
   ```bash
   kubectl -n iam create secret generic vault-unseal-keys \
     --from-literal=key1=placeholder --from-literal=key2=placeholder --from-literal=key3=placeholder
   kubectl -n iam rollout status sts/vault --timeout=2m
   ```

2. **Init** (generates real keys + root token — SAVE THESE):
   ```bash
   kubectl -n iam exec -it vault-0 -- vault operator init -key-shares=3 -key-threshold=3
   # Output:
   # Unseal Key 1: ...
   # Unseal Key 2: ...
   # Unseal Key 3: ...
   # Initial Root Token: ...
   ```
   **STORE SECURELY** in password manager / encrypted file. Do NOT paste into terminals or commit.

3. **Unseal** (run all 3 keys):
   ```bash
   key1="<paste from password manager>"
   key2="<paste from password manager>"
   key3="<paste from password manager>"
   
   kubectl -n iam exec -it vault-0 -- vault operator unseal $key1
   kubectl -n iam exec -it vault-0 -- vault operator unseal $key2
   kubectl -n iam exec -it vault-0 -- vault operator unseal $key3
   ```

4. **Update secret with real keys** (via SOPS for GitOps, or direct kubectl):
   ```bash
   # Option A: direct (fast for now, not GitOps; set up SOPS encrypt later)
   kubectl -n iam create secret generic vault-unseal-keys \
     --from-literal=key1="$key1" --from-literal=key2="$key2" --from-literal=key3="$key3" \
     --dry-run=client -o yaml | kubectl apply -f -
   ```
   **TODO later**: encrypt this secret via SOPS and commit to repo so re-provision has keys.

5. **Restart pod** to pick up real keys (reloader watches secret):
   ```bash
   kubectl -n iam rollout restart sts/vault
   kubectl -n iam rollout status sts/vault
   ```

#### Verify
```bash
kubectl -n iam get pod vault-0                          # 1/1 Running
kubectl -n iam exec -it vault-0 -- vault status         # Initialized true, Sealed false
kubectl -n argocd get application vault -o jsonpath='{.status.health.status}{"\n"}'  # Healthy
```

---

## Final check (all apps green)

```bash
kubectl get applications -A --no-headers | awk '{print $3,$4}' | sort | uniq -c
# expect: 32 "Synced Healthy"

kubectl get pods -A | grep -ivE 'Running|Completed'
# expect: no output (or only Completed jobs)

kubectl get nodes
# expect: 3 Ready

# Cluster ready for next steps
echo "✓ Cluster healthy"
```

---

## Rollback (per-task)
Each task has a `git revert --no-edit HEAD && git push origin main` — run that commit hash if needed.

---

## Notes for agents

- **All repo changes use GitOps**: edit files, commit, push `main`, ArgoCD auto-syncs (within 3–5 min, or force with annotate).
- **Parallel GROUP 1**: tasks 1a, 2a, 3a have no interdependencies — agents can work simultaneously.
- **Sequential GROUP 2**: wait for GROUP 1 all-green before starting 4a/5a.
- **Vault (5a)** is the only manual step — unseal keys must be handled securely offline, not in terminal history.
- **Post-remediation**: cluster is health-ready for applications/workloads; Cloudflare tunneling deferred.