Infrastructure Practice Playbook

Standardized procedures for troubleshooting, developing, deploying, and operating the homelab platform. Each procedure explicitly calls out where the core CLI fits vs kubectl/helmfile/direct cluster access. See core-cli-tools.md for the auth/secrets domain split; see infra-troubleshooting.md for quick patterns and gotchas.

Procedure A: Troubleshoot a Service or Cluster Issue

Identify the domain.
- Vault/secrets: core get/core put failing, Vault unreachable.
- Node/Talos: node crashes, disk full, kubelet unreachable, network issues.
- Plain Kubernetes: pod CrashLoop, service 503, deployment stuck.
If Vault-adjacent (secrets, authentication failing):
- Run core secrets status first (NOT core auth status — common mistake).
- If Vault UNREACHABLE, check DNS/networking:
  - No wildcard DNS exists; verify manual /etc/hosts entries (10.6.0.1 for WireGuard, 192.168.1.160 for LAN).
  - Ping the Vault service: kubectl get svc -n vault | grep vault.
- If Vault token not cached, run core secrets login, approve device code in browser.
- Verify: core secrets status shows ✓ Authenticated.
If node/Talos-adjacent (kubelet logs, node state, services failing):
- Run core auth status first.
- If token expired, run core auth login-oob, approve device code in browser.
- Then run core nodes to list cluster nodes.
- For a specific node, run core status <ip> (Talos state).
- Inspect Talos services: core services <ip> (kubelet, etcd, controller, etc.).
- Check service logs: core logs <ip> (main Talos logs) or core log-svc <ip> kubelet (specific service).
- Consult infra-troubleshooting.md for Pod stuck in CrashLoopBackOff and kubelet restart patterns.
If plain Kubernetes (pod/deployment/service issues):
- Consult root TROUBLESHOOTING.md for the layer-before-tool SRE methodology (procedures 1–10).
- Use infra-troubleshooting.md § Quick Patterns for common diagnoses:
  - CrashLoopBackOff: kubectl logs -n <ns> <pod> --tail=50 + kubectl describe pod -n <ns> <pod> | grep -A 10 Events.
  - Service 503: kubectl get endpoints -n <ns> <svc> (endpoints missing?) + kubectl get pods -n <ns> -o wide (pods not Ready?).
  - Helm release stuck: helmfile status | grep -E "FAILED|UNKNOWN|PENDING" + helm status <release> -n <ns> --show-resources.
- If still unclear, escalate to kubectl get all -n <ns> and review resource events.
For dashboards and live metrics:
- Use core pf grafana (port-forward to localhost:3000) rather than raw kubectl port-forward — keeps forwarded ports consistent.
- If Prometheus unavailable, check: kubectl get pods -n monitoring | grep prometheus.
- If ServiceMonitor not scraping, verify: kubectl get servicemonitor -A | grep <name> and inspect .spec.selector matches the target pod's app label.

Procedure B: Launch/Develop a New Service or POC

Plan the service.
- Determine namespace (e.g., sqs, temporal, databases, monitoring).
- Decide if metrics exported (most should) and if OIDC-gated.
- Sketch a Helm values.yaml structure (secrets, replicas, resource requests, affinity).
Authenticate to Vault.
- Run core secrets login and approve device code in browser.
- Verify: core secrets status shows ✓ Authenticated.
- You'll need Vault access to store service secrets in step 5.
Create service Helm chart directory.
- Create k8s/<service>/ with at minimum:
  - values.yaml (Helm values for deployment, service, replicas, resource limits).
  - charts/ subdirectory for any custom local Helm charts (optional).
- Follow naming conventions from coding-standards.md.

Add Helm release to helmfile.

Open helmfile.yaml.gotmpl.

Add release block under releases: section, following this structure:

- name: <service>
  namespace: <namespace>
  chart: <chart-repo>/<chart-name>
  version: ~1.0  # pin major.minor, allow patch updates
  needs:
    - <dependency-namespace>/<dependency-release>  # if applicable
  values:
    - k8s/<service>/values.yaml
    - secretsInline:
        DB_PASSWORD: "{{ env \"<SERVICE>_DB_PASSWORD\" }}"

Consult coding-standards.md for needs: ordering (example: sqs section shows strimzi-operator → kafka-cluster → queue-crd → management-service).
Reference real example: root helmfile's sqs section.

If the service needs secrets (DB password, API key, OAuth secret):
- Generate value (e.g., openssl rand -hex 32 for passwords).
- Store in Vault: core put cluster/<SERVICE>_<KEY> <SERVICE>_<KEY>="value".
- Critical gotcha: field name MUST equal variable name (e.g., FORGEJO_ADMIN_PASSWORD= not value=) per coding-standards.md § Vault field=variable convention.
- Reference in values.yaml via {{ env "VARIABLE_NAME" }} (Helmfile Go template syntax, NOT shell ${VAR}).
- Do NOT hardcode secrets in values.yaml or ConfigMaps.
Verify Helm syntax before deploy.
- Run helmfile lint (catches template errors, duplicate releases).
- Run helmfile diff -l name=<service> (show what will be deployed).
- Review diff for correctness (verify env var substitutions, resource limits, affinity rules).
Deploy the service.
- Run helmfile apply -l name=<service>.
- Monitor: kubectl get pods -n <namespace> -w (watch until Running).
- If pods stuck: kubectl describe pod -n <namespace> <pod-name> (check Events for SchedulingFailed, ImagePullBackOff, etc.).
If the service exports /metrics (Prometheus format):
- Create ServiceMonitor: k8s/monitoring/servicemonitors/svc-<name>.yaml.
  - .spec.selector.matchLabels must match the service's pod labels (usually app: <service>).
  - .spec.endpoints[0].port must match the service port name or number exporting metrics.
- Create PrometheusRule: k8s/monitoring/alerts/svc-<name>-rules.yaml.
  - Include error rate, latency, and SLO alert rules.
  - Use prometheus as the rule group.
- Create Grafana dashboard: k8s/monitoring/dashboards/svc-<name>.yaml.
  - Use 6-row template: Availability, Resources, Domain metrics, Logs, SLO, Related.
  - See README.md § Example Applications for a full walkthrough.
- Verify scrape: kubectl get servicemonitor -A | grep <name> and check Prometheus Targets UI for green status.
If OIDC/IAM-gated (admin UI, restricted API):
- Create app in Authentik: core iam create-app "my-service" --slug my-service --redirect-uri "https://my-service.riotpiao.com/callback".
- Bind app to group: core iam bind-app my-service <group> (e.g., grafana-admins for admin-only UI).
- Retrieve credentials: core iam describe-app my-service (client ID, client secret).
- Deploy secret: kubectl create secret generic <service>-oidc --from-literal=client-id=<ID> --from-literal=client-secret=<SECRET> -n <namespace>.
- Reference secret in values.yaml: mount via .spec.template.spec.containers[].env or volumeMounts.
- See core-cli-tools.md § Access Control Tiers for Tier A (OIDC + RBAC) vs Tier B (network perimeter only).
Verify service is live.
- Pods: kubectl get pods -n <namespace> -o wide (all Running, 1/1 Ready).
- Metrics (if applicable): kubectl get servicemonitor -A | grep <name> and visit Prometheus Targets or Grafana dashboard.
- Endpoint: If publicly routed via Ingress, verify /etc/hosts entry (10.6.0.1 for WireGuard, 192.168.1.160 for LAN) and curl https://my-service.riotpiao.com/health (or equivalent health endpoint).
- Logs: kubectl logs -n <namespace> <pod> (no errors).

Definition of Done (Per Service)

Helm chart version pinned (~1.0 format in helmfile)
All secrets in Vault (none in values.yaml or ConfigMap)
/metrics endpoint exported (if applicable)
ServiceMonitor resource created (if metrics exported)
PrometheusRule with error/latency/SLO alerts (if metrics exported)
Grafana dashboard (if metrics exported; 6-row template: Availability, Resources, Domain, Logs, SLO, Related)
Ingress rule (if external access needed)
OIDC integration via core iam (if UI component)
Verified: helmfile diff clean, pods Running, dashboard live or /metrics returning 200

Procedure C: Operate the Cluster (Node Health, Context, Cleanup)

Daily health check.
- Check auth: core auth status (if OK, node ops will work).
- List nodes: core nodes.
- For each node, check Talos state: core status <ip>.
- Check K8s nodes: kubectl get nodes -o wide (all Ready, no NotReady).
- Check pod pressure: kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, memory: .status.allocatable.memory, pods: .status.allocatable.pods}'.
Troubleshoot a specific node.
- Get node IP: core nodes and note the IP.
- Check Talos services: core services <ip> (kubelet, etcd, controller should be running).
- Check service logs: core logs <ip> (main Talos daemon logs).
- Filter to specific service: core log-svc <ip> kubelet (kubelet logs only).
- Restart a service if needed: core restart <ip> kubelet (graceful kubelet restart).
Pod cleanup (Failed, Evicted, Terminating pods).
- Run core pods clean (scans all namespaces, removes stale pods).
- Verify: kubectl get pods -A | grep -E "Failed|Evicted" (should be empty).
Switch kubectl context (when off-LAN, on WireGuard).
- List available contexts: core config kube-list.
- Switch to WireGuard path (10.6.0.1:6443): core config kube-use admin@homelab-cluster-1.
- Known limitation: core config use <talos-context> doesn't map to WireGuard; use kube-use directly.
- Verify: kubectl cluster-info shows 10.6.0.1 (not 192.168.1.213).
MinIO bucket operations (if managing data/backups).
- List buckets: core bucket list.
- Upload file: core bucket upload <bucket> <local-file>.
- Download file: core bucket download <bucket> <remote-file> -o <local-file>.
- Delete file: core bucket delete <bucket> <remote-file>.
Bootstrap or hardware runbooks (infrequent).
- Fresh cluster setup: See README.md § Bootstrap Order (14 steps).
- Adding a new Talos node: See README.md § Adding Hardware.
- Do not re-explain those long procedures here; consult README.md directly.

Notes

Queue subsystem (Kafka/kmsvc/Temporal namespace auto-registration): Already deployed and stable. If re-deploying:

Primary deploy method: helmfile apply -l namespace=sqs (live from root helmfile).
Alternate isolated iterate path: k8s/sqs/helmfile.yaml.gotmpl (not recommended for production).
Planned future: GitOps via k8s/sqs/argocd/ (companion repo, not yet active).
Critical rule: Temporal namespace registration is automatic via queue-operator; never manually temporal operator namespace create for any namespace referenced by a Queue's temporal.io/namespace label. See ~/workplace/kmsvc-manage/CLAUDE.md ("Temporal Namespace Registration") for the full rule and why.

Shared/reusable service repositories: If a service's Helm chart and container image live in a separate repository, they must be:

Published as a public GitHub repository under the Riotpiaole organization.
Images pushed to GHCR (ghcr.io/riotpiaole/...) for public pullability.
Consult coding-standards.md § Shared/Reusable Repos for the full publishing rule.

Cross-References

core-cli-tools.md: Auth/secrets domain split, command inventory, when to use core vs kubectl.
coding-standards.md: Helm naming conventions, needs: ordering rules, helmfile template syntax ({{ env "VAR" }} not ${VAR}), Vault field=variable convention, shared-repo publishing rule.
infra-troubleshooting.md: Quick patterns (CrashLoopBackOff, 503, helm stuck), gotchas, hard rules.
USAGE.md: Exhaustive core command reference.
README.md: Bootstrap order, hardware addition, example app walkthrough, 6-row Grafana dashboard template.
root TROUBLESHOOTING.md: Generic Kubernetes SRE layer-before-tool methodology (10 diagnostic procedures).

12 KiB Raw Permalink Blame History Unescape Escape