diff --git a/REMEDIATION_PLAN.md b/REMEDIATION_PLAN.md new file mode 100644 index 0000000..3337d01 --- /dev/null +++ b/REMEDIATION_PLAN.md @@ -0,0 +1,420 @@ +# Homelab Cluster Remediation Runbook — Parallel Execution + +Board: **24/32 green**. Goal: all ArgoCD apps `Synced/Healthy`, no non-Running pods. + +## Parallel task groups (run agents in parallel within groups; groups are sequential) + +``` +GROUP 1 (independent, run all 3 in parallel) +├─ TASK 1a — temporal: deploy Cassandra + Elasticsearch [**blocks TASK 2 (homelab-ingress)**] +├─ TASK 2a — queue-operator: add RBAC + TemporalWorker CRD +└─ TASK 3a — minio console: fix image tag + +GROUP 2 (wait for GROUP 1 complete) +├─ TASK 4a — prometheus / sops-secrets: resolve OutOfSync +└─ TASK 5a — vault: one-time init + unseal + +GROUP 3 (after temporal ns exists) +└─ TASK 6a — homelab-ingress: should auto-sync once temporal ns created +``` + +**Cloudflare** — deferred (not blocking cluster health). + +--- + +## Cluster facts +- 3 Talos nodes: cp-1 (192.168.1.213), cp-2 (192.168.1.163), cp-3 (192.168.1.166) — all Ready. +- GitOps repo: `~/workplace/homelab`, branch `main`. +- **GitOps mandatory**: repo → commit → push → ArgoCD syncs. No hand-edits except vault init. +- Ingress LB: 192.168.1.160, wildcard TLS on prod (already fixed). + +## Status baseline (run before + after each group) +```bash +kubectl get applications -A --no-headers | awk '{printf "%-24s %-12s %s\n",$2,$3,$4}' +kubectl get pods -A | grep -ivE 'Running|Completed' +``` + +--- + +## GROUP 1 — parallel tasks (all 3 agents) + +### TASK 1a — temporal: deploy Cassandra + Elasticsearch (replaces PostgreSQL) + +**Symptom**: `temporal` app sync=Unknown, helm render error on removed `cassandra:` top-level key. + +**Decision**: Deploy Cassandra + Elasticsearch as sub-charts. Temporal chart supports both as embedded dependencies. + +#### Steps +1. **Pin chart version** — edit `k8s/argocd/apps/60-applications.yaml`: + ```bash + grep -n "targetRevision" k8s/argocd/apps/60-applications.yaml | grep -A2 -B2 temporal + ``` + Change `targetRevision: "*"` → `targetRevision: "0.64.0"` (confirmed to have cassandra/elasticsearch sub-charts). + +2. **Rewrite temporal-values.yaml** — remove top-level `cassandra:`, enable cassandra + elasticsearch sub-charts: + ```bash + sed -n '1,50p' k8s/applications/temporal/temporal-values.yaml + ``` + Rewrite to: + ```yaml + # Temporal with Cassandra (default store) + Elasticsearch (visibility) + + cassandra: + enabled: true + config: + cluster_name: temporal + num_tokens: 256 + seed_provider: + class_name: org.apache.cassandra.locator.SimpleSeedProvider + parameters: + seeds: "127.0.0.1" + resources: + requests: + memory: "512Mi" + cpu: "250m" + limits: + memory: "1Gi" + cpu: "500m" + persistence: + enabled: true + size: 10Gi + + elasticsearch: + enabled: true + replicas: 1 + resources: + requests: + memory: "512Mi" + cpu: "250m" + limits: + memory: "1Gi" + cpu: "500m" + persistence: + enabled: true + size: 10Gi + + postgresql: + enabled: false # disable embedded postgres + + server: + config: + persistence: + defaultStore: default + additionalStores: {} + datastores: + default: + driver: cassandra + cassandra: + hosts: + - cassandra + port: 9042 + keyspace: temporal + user: "" + password: "" + maxConnsPerHost: 32 + consistency: LOCAL_QUORUM + visibility: + driver: elasticsearch + elasticsearch: + scheme: http + host: elasticsearch + port: 9200 + + # Keep remaining server config (logging, etc.) + # ... (existing server.lifecycleHooks, server.replicaCount, etc.) + ``` + + Key differences: + - Top-level `cassandra.enabled: true` (sub-chart, not deprecated key). + - `elasticsearch.enabled: true` for visibility store. + - `server.config.persistence.datastores.default.driver: cassandra` (not SQL). + - `server.config.persistence.datastores.visibility.driver: elasticsearch`. + - No CNPG references — Cassandra/ES managed by Helm. + +3. **Validate helm render locally**: + ```bash + helm template temporal temporal --repo https://go.temporal.io/helm-charts \ + --version 0.64.0 -n temporal \ + -f k8s/applications/temporal/temporal-values.yaml --include-crds 2>&1 | grep -iE "error|^kind:" | head + # expect: kind lines (StatefulSet, Deployment, etc.), no errors + ``` + +4. **Commit + push**: + ```bash + git add k8s/applications/temporal/temporal-values.yaml k8s/argocd/apps/60-applications.yaml + git commit -m "fix(temporal): deploy Cassandra + Elasticsearch sub-charts, remove PostgreSQL, pin chart v0.64.0" + git push origin main + ``` + +5. **Trigger sync** (ArgoCD may auto-sync; force if needed): + ```bash + kubectl -n argocd annotate application temporal argocd.argoproj.io/refresh=hard --overwrite + ``` + +#### Verify +```bash +kubectl get ns temporal # namespace exists +kubectl -n temporal get pods | head # server/cassandra/elasticsearch pods Running +kubectl -n argocd get application temporal -o jsonpath='{.status.sync.status} {.status.health.status}{"\n"}' # Synced Healthy +``` + +#### Troubleshoot +- Still `Unknown` / render error → confirm cassandra/elasticsearch keys are nested under top-level, not at column-0. Check Chart.yaml: `dependencies: [{name: cassandra}, {name: elasticsearch}]`. +- Cassandra pod Pending → PVC not bound. Check: `kubectl -n temporal get pvc`. If no PVC, storage class missing — verify `kubectl get storageclass`. +- Elasticsearch OOMKilled → bump resource limits in values. +- Schema job pending → cassandra not ready. Wait: `kubectl -n temporal logs job/temporal-schema-setup --tail=20`. + +--- + +### TASK 2a — queue-operator: add RBAC + TemporalWorker CRD (parallel) + +**Symptom**: operator crashloop, logs show `forbidden: cannot list deployments` + `no matches for kind TemporalWorker`. + +**Cause**: ClusterRole missing `deployments.apps` permission; TemporalWorker CRD not installed. + +#### Steps +1. **Edit RBAC** — `k8s/applications/sqs/charts/queue-crd/templates/rbac.yaml`, add to ClusterRole rules: + ```yaml + - apiGroups: ["apps"] + resources: ["deployments"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] + ``` + (Full file already has rules for queues, leases, events, pods, nodes; append the above.) + +2. **Install TemporalWorker CRD** — kmsvc-managed. Fetch from kmsvc source (or import from your internal docs): + Create `k8s/applications/sqs/charts/queue-crd/templates/temporalworker-crd.yaml`: + ```yaml + apiVersion: apiextensions.k8s.io/v1 + kind: CustomResourceDefinition + metadata: + name: temporalworkers.kmsvc.io + spec: + group: kmsvc.io + names: + kind: TemporalWorker + plural: temporalworkers + scope: Namespaced + versions: + - name: v1 + served: true + storage: true + schema: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + workflowType: + type: string + concurrency: + type: integer + status: + type: object + ``` + (Adjust schema per kmsvc spec; this is minimal template.) + +3. **Commit + push**: + ```bash + git add k8s/applications/sqs/charts/queue-crd/templates/rbac.yaml \ + k8s/applications/sqs/charts/queue-crd/templates/temporalworker-crd.yaml + git commit -m "fix(sqs): grant queue-operator deployments RBAC, install TemporalWorker CRD" + git push origin main + ``` + +4. **Restart operator**: + ```bash + kubectl -n sqs rollout restart deploy/queue-operator + ``` + +#### Verify +```bash +kubectl get crd | grep -iE 'queues.kmsvc|temporalworkers.kmsvc' # both exist +kubectl -n sqs logs deploy/queue-operator --tail=5 | grep -iE 'error|forbidden' || echo CLEAN +kubectl -n sqs get pods -l app=queue-operator # Running, restarts stable +``` + +--- + +### TASK 3a — minio console: fix image tag (parallel) + +**Symptom**: ns `storage`, pod `console-*` stuck ImagePullBackOff, image `minio/console:v0.30.0` doesn't exist. + +**Fix**: remove the standalone console deploy (deprecated); MinIO tenant has built-in console. + +#### Steps +1. **Find console reference**: + ```bash + grep -rn "v0.30.0\|console:" k8s/infrastructure/minio k8s/applications/minio* 2>/dev/null | head + ``` + +2. **Remove from kustomization/values**: + ```bash + grep -n "console\|apiVersion:" k8s/infrastructure/minio/kustomization.yaml + ``` + Delete lines referencing `console` deployment/service. Keep operator + tenant. + +3. **Commit + push**: + ```bash + git add k8s/infrastructure/minio/ + git commit -m "fix(minio): remove deprecated standalone console (use tenant built-in)" + git push origin main + ``` + +#### Verify +```bash +kubectl -n storage get pods | grep -i console || echo "console removed" +kubectl -n argocd get application minio-operator -o jsonpath='{.status.health.status}{"\n"}' # Healthy +``` + +--- + +## Wait for GROUP 1 complete (all 3 tasks done) + +Check: +```bash +kubectl -n argocd get app temporal queue-crd minio-operator -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.sync.status}{" "}{.status.health.status}{"\n"}{end}' +# want all: Synced Healthy +``` + +--- + +## GROUP 2 — sequential tasks (after GROUP 1 green) + +### TASK 4a — prometheus + sops-secrets OutOfSync (drift resolution) + +**Symptom**: both apps Healthy but OutOfSync (CRD annotation drift). + +#### Steps +1. **Inspect diff**: + ```bash + kubectl -n argocd get application prometheus -o json | jq -r '.status.conditions[]?.message' | head + ``` + +2. **Enable server-side apply** — edit the Application defining prometheus in `k8s/argocd/apps/`: + ```bash + grep -n "prometheus" k8s/argocd/apps/*.yaml | grep -i "name:" + ``` + Add to `syncPolicy.syncOptions`: + ```yaml + syncOptions: + - ServerSideApply=true + ``` + +3. **For sops-secrets**, check ownership: + ```bash + kubectl -n argocd get application sops-secrets -o json | jq '.spec.source' + ``` + If it's in homelab-root as nested app, ensure no dual-ownership (no other ArgoCD app managing the same Secret). If conflict, update the parent app path. + +4. **Commit + push**: + ```bash + git add k8s/argocd/apps/ + git commit -m "fix(argocd): enable server-side apply for prometheus CRD, reconcile sops-secrets ownership" + git push origin main + ``` + +5. **Refresh**: + ```bash + kubectl -n argocd annotate application prometheus argocd.argoproj.io/refresh=hard --overwrite + kubectl -n argocd annotate application sops-secrets argocd.argoproj.io/refresh=hard --overwrite + ``` + +#### Verify +```bash +kubectl -n argocd get app prometheus sops-secrets -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.sync.status}{"\n"}{end}' +# Synced Synced +``` + +--- + +### TASK 5a — vault: one-time init + unseal (manual) + +**Symptom**: vault-0 CreateContainerConfigError, needs secret `vault-unseal-keys`. + +**Security**: store unseal keys + root token in your offline password manager. Never commit plaintext to git. + +#### Steps +1. **Create placeholder secret** (to let pod start): + ```bash + kubectl -n iam create secret generic vault-unseal-keys \ + --from-literal=key1=placeholder --from-literal=key2=placeholder --from-literal=key3=placeholder + kubectl -n iam rollout status sts/vault --timeout=2m + ``` + +2. **Init** (generates real keys + root token — SAVE THESE): + ```bash + kubectl -n iam exec -it vault-0 -- vault operator init -key-shares=3 -key-threshold=3 + # Output: + # Unseal Key 1: ... + # Unseal Key 2: ... + # Unseal Key 3: ... + # Initial Root Token: ... + ``` + **STORE SECURELY** in password manager / encrypted file. Do NOT paste into terminals or commit. + +3. **Unseal** (run all 3 keys): + ```bash + key1="" + key2="" + key3="" + + kubectl -n iam exec -it vault-0 -- vault operator unseal $key1 + kubectl -n iam exec -it vault-0 -- vault operator unseal $key2 + kubectl -n iam exec -it vault-0 -- vault operator unseal $key3 + ``` + +4. **Update secret with real keys** (via SOPS for GitOps, or direct kubectl): + ```bash + # Option A: direct (fast for now, not GitOps; set up SOPS encrypt later) + kubectl -n iam create secret generic vault-unseal-keys \ + --from-literal=key1="$key1" --from-literal=key2="$key2" --from-literal=key3="$key3" \ + --dry-run=client -o yaml | kubectl apply -f - + ``` + **TODO later**: encrypt this secret via SOPS and commit to repo so re-provision has keys. + +5. **Restart pod** to pick up real keys (reloader watches secret): + ```bash + kubectl -n iam rollout restart sts/vault + kubectl -n iam rollout status sts/vault + ``` + +#### Verify +```bash +kubectl -n iam get pod vault-0 # 1/1 Running +kubectl -n iam exec -it vault-0 -- vault status # Initialized true, Sealed false +kubectl -n argocd get application vault -o jsonpath='{.status.health.status}{"\n"}' # Healthy +``` + +--- + +## Final check (all apps green) + +```bash +kubectl get applications -A --no-headers | awk '{print $3,$4}' | sort | uniq -c +# expect: 32 "Synced Healthy" + +kubectl get pods -A | grep -ivE 'Running|Completed' +# expect: no output (or only Completed jobs) + +kubectl get nodes +# expect: 3 Ready + +# Cluster ready for next steps +echo "✓ Cluster healthy" +``` + +--- + +## Rollback (per-task) +Each task has a `git revert --no-edit HEAD && git push origin main` — run that commit hash if needed. + +--- + +## Notes for agents + +- **All repo changes use GitOps**: edit files, commit, push `main`, ArgoCD auto-syncs (within 3–5 min, or force with annotate). +- **Parallel GROUP 1**: tasks 1a, 2a, 3a have no interdependencies — agents can work simultaneously. +- **Sequential GROUP 2**: wait for GROUP 1 all-green before starting 4a/5a. +- **Vault (5a)** is the only manual step — unseal keys must be handled securely offline, not in terminal history. +- **Post-remediation**: cluster is health-ready for applications/workloads; Cloudflare tunneling deferred. diff --git a/k8s/bootstrap/ingress/ingress.yaml b/k8s/bootstrap/ingress/ingress.yaml index fd6a285..d6b5af3 100644 --- a/k8s/bootstrap/ingress/ingress.yaml +++ b/k8s/bootstrap/ingress/ingress.yaml @@ -120,9 +120,9 @@ spec: pathType: Prefix backend: service: - name: minio-console + name: minio-cluster-console port: - number: 9001 + number: 9090 --- # ── MinIO S3 API ────────────────────────────────────────────────────────────── @@ -229,7 +229,7 @@ apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: argocd - namespace: cicd + namespace: argocd annotations: nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" nginx.ingress.kubernetes.io/proxy-ssl-verify: "off" @@ -295,25 +295,3 @@ spec: name: temporal-web port: number: 8080 - ---- -# ── kmsvc (SQS/Kafka REST API) ──────────────────────────────────────────── -# SQS-compatible messaging API via kmsvc management service -apiVersion: networking.k8s.io/v1 -kind: Ingress -metadata: - name: kmsvc - namespace: sqs -spec: - ingressClassName: nginx - rules: - - host: kmsvc.riotpiao.com - http: - paths: - - path: / - pathType: Prefix - backend: - service: - name: management-service - port: - number: 8080 diff --git a/k8s/infrastructure/minio/minio-tenant.yaml b/k8s/infrastructure/minio/minio-tenant.yaml index 876d248..927dbc8 100644 --- a/k8s/infrastructure/minio/minio-tenant.yaml +++ b/k8s/infrastructure/minio/minio-tenant.yaml @@ -13,6 +13,13 @@ metadata: spec: image: minio/minio:RELEASE.2025-07-23T15-54-02Z + # Disable operator auto-TLS: MinIO serves plain HTTP internally on 9000. + # External TLS is terminated at nginx ingress (wildcard riotpiao-com-tls cert). + # Without this, MinIO auto-generates self-signed certs and serves HTTPS-only + # on 9000, which breaks plain-HTTP internal clients like Vault's S3 backend + # (they hang waiting for a TLS handshake that never completes on an HTTP request). + requestAutoCert: false + # Root credentials. v5 pods read `configuration` — a Secret whose `config.env` # key holds shell `export MINIO_ROOT_USER=...` lines. Created out-of-band # (SOPS), see minio-secrets.enc.yaml. NOTE: the operator health-monitor logs a