Step 1 complete: Bootstrap layer with ArgoCD, cert-manager, namespaces imported to TF

- ArgoCD migrated to argocd namespace - Cert-manager issuers/certs created - 20 namespaces imported with pod-security labels - S3 backend temporarily offline (MinIO), using local backup - Pending: Remove metadata drift from helm releases, re-apply
2026-07-14 13:14:46 -07:00
parent 9a4d486b86
commit dd608d3231
34 changed files with 2015 additions and 87 deletions
@@ -0,0 +1,11 @@
 #!/bin/bash
 # Terraform environment setup example
 # Copy to .env.terraform.sh and fill in real values from .env
 # Then: source .env.terraform.sh && cd terraform && terraform plan
 # MinIO S3 backend credentials (from .env MINIO_ROOT_USER / MINIO_ROOT_PASSWORD)
 export AWS_ACCESS_KEY_ID="minioadmin"
 export AWS_SECRET_ACCESS_KEY="<MINIO_ROOT_PASSWORD from .env>"
 # Kubernetes config (points to cluster-config/kubeconfig)
 export KUBECONFIG="${PWD}/cluster-config/kubeconfig"
@@ -1,5 +1,6 @@
 # Environment files — real values must never be committed
 .env
 .env.terraform.sh
 # Private CA key and generated TLS certs — ca.key must never enter the cluster or git.
 # Only ca.crt is safe to share, but we exclude the whole dir to avoid accidents.
@@ -285,14 +285,27 @@ Add to `/etc/hosts` on every client machine (Mac/Linux):
 ```
 # WireGuard access (remote — via talos-cp-1)
-10.6.0.1  grafana.riotpiao.homelab.com authentik.riotpiao.homelab.com vault.riotpiao.homelab.com minio.riotpiao.homelab.com prometheus.riotpiao.homelab.com portainer.riotpiao.homelab.com longhorn.riotpiao.homelab.com loki.riotpiao.homelab.com forgejo.riotpiao.homelab.com
+10.6.0.1  grafana.riotpiao.homelab.com authentik.riotpiao.homelab.com vault.riotpiao.homelab.com minio.riotpiao.homelab.com prometheus.riotpiao.homelab.com portainer.riotpiao.homelab.com longhorn.riotpiao.homelab.com loki.riotpiao.homelab.com forgejo.riotpiao.homelab.com temporal.riotpiao.homelab.com temporal-grpc.riotpiao.homelab.com kmsvc.riotpiao.homelab.com
 # LAN access (on the home network — use actual LoadBalancer IP from above)
-192.168.1.160  grafana.riotpiao.homelab.com authentik.riotpiao.homelab.com vault.riotpiao.homelab.com minio.riotpiao.homelab.com prometheus.riotpiao.homelab.com portainer.riotpiao.homelab.com longhorn.riotpiao.homelab.com loki.riotpiao.homelab.com forgejo.riotpiao.homelab.com
+192.168.1.160  grafana.riotpiao.homelab.com authentik.riotpiao.homelab.com vault.riotpiao.homelab.com minio.riotpiao.homelab.com prometheus.riotpiao.homelab.com portainer.riotpiao.homelab.com longhorn.riotpiao.homelab.com loki.riotpiao.homelab.com forgejo.riotpiao.homelab.com temporal.riotpiao.homelab.com temporal-grpc.riotpiao.homelab.com kmsvc.riotpiao.homelab.com
 ```
 **Note:** `192.168.1.160` is an example Cilium LB-IPAM assignment. Verify with `kubectl get svc -n ingress-nginx ingress-nginx`.
 **There is no real DNS wildcard for `*.riotpiao.homelab.com`** — every hostname must be added to `/etc/hosts` explicitly (as above) before it resolves. Adding a new Ingress host doesn't make it reachable by itself; add the line too.
 ### kubectl Context
 Two contexts exist in `cluster-config/kubeconfig`, pointed at the same cluster over different paths:
 | Context | Server | Use when |
 |---|---|---|
 | `admin@homelab-cluster` | `192.168.1.213:6443` (LAN) | On the home network |
 | `admin@homelab-cluster-1` | `10.6.0.1:6443` (WireGuard) | Remote / off-LAN |
 If `kubectl` commands hang or refuse the connection, switch: `kubectl config use-context admin@homelab-cluster-1`.
 Then access services at:
 | Service | URL | Credentials |
@@ -380,7 +393,7 @@ Authentik is the central OIDC identity provider. Vault stores secrets and delega
  └── secret/    — KV v2: mcp/*, cluster/*, cloud/*
 ```
-**talos-cli device code login:**
+**core CLI device code login:**
 ```bash
 core secrets login        # prints URL + code → approve in browser → Vault token cached
 core put cluster/DUCKDNS_TOKEN DUCKDNS_TOKEN="abc"            # field name = variable name, never `value`
@@ -118,6 +118,23 @@ core log-svc <ip> <svc>       # logs for specific service (etcd, kubelet, etc.)
 core pods clean               # delete Failed/Evicted/Terminating pods
 ```
 #### kubectl Context (LAN vs. WireGuard)
 `cluster-config/kubeconfig` has two contexts pointed at the same cluster:
 `admin@homelab-cluster` (LAN, `192.168.1.213:6443`) and `admin@homelab-cluster-1`
 (WireGuard, `10.6.0.1:6443`). If `kubectl`/`core nodes` hangs or refuses the
 connection, you're likely off-LAN — switch contexts:
 ```bash
 core config kube-list             # list contexts
 core config kube-use admin@homelab-cluster-1   # switch to WireGuard path
 ```
 **Known gap:** `core config use <talos-context>` (the combined talosctl+kubectl
 switch) only maps to `admin@homelab-cluster` today — its WireGuard mapping
 (`home-cluster-wire-guard`) is stale, that kubectl context doesn't exist. Use
 `core config kube-use admin@homelab-cluster-1` directly until that's fixed.
 ---
 ### Secret Management (Vault)
@@ -311,7 +328,7 @@ core get cluster/iam/roles/admin --key roles
 ```bash
 # 1. Get ci-bot JWT token (runner has this injected via ServiceAccount)
-export REGISTRY_TOKEN=$(talos get cluster/iam/agents/ci-bot --key token)
+export REGISTRY_TOKEN=$(core get cluster/iam/agents/ci-bot --key token)
 # 2. Authenticate docker/podman to Forgejo registry
 docker login forgejo.riotpiao.homelab.com \
@@ -0,0 +1,10 @@
 # Trust the homelab-ca CA for pulling from the Forgejo container registry.
 # Without this, containerd fails: x509 certificate signed by unknown authority
 # (nginx terminates forgejo.riotpiao.homelab.com TLS with a homelab-ca cert).
 # Apply: talosctl -n <node> patch mc --patch @cluster-config/patches/forgejo-registry-ca.yaml
 machine:
  registries:
    config:
      forgejo.riotpiao.homelab.com:
        tls:
          ca: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJiVENDQVJTZ0F3SUJBZ0lVYTBVaGs3Rm81d3BiWjZsRzVEWWJUVkFic1k4d0NnWUlLb1pJemowRUF3SXcKRlRFVE1CRUdBMVVFQXhNS2FHOXRaV3hoWWkxallUQWVGdzB5TmpBM01UQXdORFF3TXpaYUZ3MHpOakEzTURjdwpORFF3TXpaYU1CVXhFekFSQmdOVkJBTVRDbWh2YldWc1lXSXRZMkV3V1RBVEJnY3Foa2pPUFFJQkJnZ3Foa2pPClBRTUJCd05DQUFTc1pNU2piUWI0YzNiUk00MjMxVEVrRXVLTnFLUUhaaW5uYnUzbWZGbStRc0wweTF3cjg1Uk0KUWJ6ZEZ2N01JZmN4REpMbHFqQTY1bEJ6TE9pdHRZZHRvMEl3UURBT0JnTlZIUThCQWY4RUJBTUNBcVF3RHdZRApWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVUVmcGJQL3FnYWsxaXUvQzdaQi9uZk5zc0hpd3dDZ1lJCktvWkl6ajBFQXdJRFJ3QXdSQUlnR2ltdnJiWU1xZjhGYThCeTBBM0M1ak1VL0d3dGU0NHgzOU4rRDRyaTJ1a0MKSUNkOEtIQXhhV0s2ZkVJcEFYZGdUQ1FxQmFiZjVZUDdhQzNDVzkzYkNsTjIKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
@@ -30,8 +30,13 @@ data:
        }
        prometheus :9153
-        # Forgejo: route to HTTP service (LoadBalancer handles both HTTPS:443 + SSH:2222 via same IP)
+        # Forgejo: route through nginx ingress like every other host below. nginx
-        rewrite name forgejo.riotpiao.homelab.com forgejo-gitea-http.cicd.svc.cluster.local
+        # terminates TLS (wildcard-tls) on :443 and routes both /v2/ (container
        # registry) and web/git to forgejo-gitea-http:3000.
        # Do NOT point this at forgejo-gitea-http directly: that service only serves
        # port 3000, so containerd image pulls (which use https/:443) get
        # `dial tcp <clusterIP>:443: i/o timeout`. SSH stays on its own LB service.
        rewrite name forgejo.riotpiao.homelab.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
        # Rewrite homelab hostnames to the nginx ingress controller so in-cluster pods
        # hit nginx TLS termination (cert-manager cert) and preserve the Host header.
@@ -79,7 +79,7 @@ apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
  name: vault
-  namespace: storage
+  namespace: iam
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
 spec:
@@ -82,15 +82,15 @@ minio:
      - group: homelab-devops → readwrite
 ```
-**CLI device code flow (talos-cli):**
+**CLI device code flow (core CLI):**
 ```bash
 # Get JWT token (no kubeconfig needed)
-talos secrets login
+core secrets login
 # → Opens browser, approve device code
-# → Token cached in ~/.talos/token
+# → Token cached in ~/.core/token
 # Use token to access Vault
-talos get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
+core get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
 # → Vault validates JWT from Authentik
 # → Returns secret
 ```
@@ -161,7 +161,7 @@ k get pods -n iam -l app=authentik
 ```bash
 # CLI tokens have 24h expiry
 # Re-authenticate
-talos secrets login
+core secrets login
 ```
 **Groups not syncing:**
@@ -198,7 +198,7 @@ k get pods -n cicd -l app=forgejo-runner
 # Settings → Applications → ci-bot → check scopes (package:write)
 # Or re-create token
-talos put cluster/iam/agents/ci-bot-token TOKEN="$(openssl rand -hex 32)"
+core put cluster/iam/agents/ci-bot-token TOKEN="$(openssl rand -hex 32)"
 ```
 **Argo CD out-of-sync:**
@@ -0,0 +1,335 @@
 # Coding Standards — Helm, Helmfile, Kubernetes, Shell
 Conventions for contributing to homelab's Helm charts, helmfiles, Kubernetes manifests, and shell scripts. These are repo-local patterns — not Go/Rust/general standards from companion repos (core CLI, kmsvc).
 ---
 ## Helm & Helmfile Conventions
 ### Chart Naming & Layout
 - **Local charts:** `k8s/<service>/charts/<component>/` directory structure.
 - **Helm chart versions:** Always use semver ranges in helmfile releases (e.g., `~1.0`, `~10`, not floating `latest`).
  - Rationale: Predictable upgrades, avoids surprise breaking changes.
 ### helmfile.yaml.gotmpl Pattern
 The helmfile is a **Go template**, not a shell script. Use Go template syntax for environment variable interpolation, not shell syntax.
 **Correct:**
 ```yaml
 set:
  - name: adminPassword
    value: {{ env "GRAFANA_ADMIN_PASSWORD" }}
 ```
 **Incorrect:**
 ```yaml
 set:
  - name: adminPassword
    value: ${GRAFANA_ADMIN_PASSWORD}  # Shell syntax — not expanded by helmfile
 ```
 **Organization:**
 - Use logical comment headers to separate sections: `# ── cert-manager ──`.
 - Group related releases together.
 - Use `needs:` for dependency ordering (release A waits for release B before deploying).
  - Example (from helmfile.yaml.gotmpl, lines 439–449):
    ```yaml
    - name: loki
      namespace: logging
      needs:
        - storage/minio
    - name: promtail
      namespace: logging
      needs:
        - logging/loki
    ```
 - **Namespace declaration:** Specify namespace at the release block level, not helmfile-level default.
  ```yaml
  - name: prometheus
    namespace: monitoring
    createNamespace: true
  ```
 ### Values Files Pattern
 - **Never hardcode secrets** in values.yaml or ConfigMap keys.
  - Store secrets in Vault via `core put cluster/KEY KEY="value"`.
  - Reference at deploy time using `{{ env "VAR" }}` in helmfile.
 - **External values files:** Always use `values:` block pointing to `.yaml` files, not inline YAML.
  ```yaml
  - name: minio
    values:
      - k8s/storage/minio-values.yaml
  ```
 - **Environment-specific overrides:** For complex releases (e.g., SQS), use `environments/` subdirectory with `helmfile.yaml.gotmpl` at the service level.
  - Example: `k8s/sqs/environments/homelab.yaml` (lines 1–29 show namespace, kafkaCluster, redis, managementService blocks).
  - Helmfile loads environment-specific values dynamically: `{{ .Values.kafkaCluster.nodePool.replicas }}`.
 ---
 ## YAML & ConfigMap/Secret Patterns
 ### Secret Field Naming
 **Rule: Field name in Secret = environment variable name in Vault.**
 When storing a secret via `core put cluster/KEY KEY="value"`, the field name and Vault variable name must match. This ensures helmfile's `{{ env "VAR" }}` expansion works correctly.
 Example (from CLAUDE.md gotcha "Field name = variable name"):
 ```bash
 # Correct
 core put cluster/MINIO_ROOT_PASSWORD MINIO_ROOT_PASSWORD="value"
 core put cluster/GRAFANA_ADMIN_PASSWORD GRAFANA_ADMIN_PASSWORD="value"
 # Incorrect (won't expand in helmfile)
 core put cluster/MINIO_SECRET value="value"  # Field name ≠ variable name
 ```
 Reference in helmfile (helmfile.yaml.gotmpl, lines 400–401):
 ```yaml
 set:
  - name: rootPassword
    value: {{ env "MINIO_ROOT_PASSWORD" }}
 ```
 ### Never Use `--env` Flags in Manifests
 Avoid kubectl flags like `--env KEY=value` in manifests or deployment specs. This exposes secrets in `kubectl describe` output.
 **Correct:** Use Secret volumes (k8s/storage/minio-values.yaml, lines 40–42):
 ```yaml
 envFrom:
  - secretRef:
      name: minio-oidc  # Reference a Secret, don't expose in manifest
 ```
 **Incorrect:**
 ```yaml
 env:
  - name: MINIO_OIDC_SECRET
    value: "sensitive-value"  # Visible in kubectl describe
 ```
 ### Namespace-First Organization
 Organize manifests by namespace: `k8s/<namespace>/`
 Structure per namespace:
 ```
 k8s/storage/
  ├── minio-values.yaml
  ├── minio-bucket-init.sh
  └── (local charts if any)
 k8s/monitoring/
  ├── prometheus-values.yaml
  ├── dashboards/         # ConfigMap files auto-loaded via helmfile postsync hook
  ├── servicemonitors/    # ServiceMonitor CRD instances
  └── alerts/             # PrometheusRule CRD instances
 ```
 ---
 ## Hook Scripts & Integration Workflows
 ### Pre/Post-Sync Hooks
 Helmfile hooks (presync/postsync) drive setup workflows. Use **relative paths from repo root**, never absolute paths.
 **Pattern (helmfile.yaml.gotmpl, lines 402–416):**
 ```yaml
 - name: minio
  hooks:
    - events: ["presync"]
      command: bash
      args:
        - -c
        - |
          bash k8s/base/namespace-setup.sh storage
          bash k8s/storage/minio-bucket-init.sh storage loki-chunks loki-ruler
 ```
 **Common presync tasks:**
 - Create namespace (via `k8s/base/namespace-setup.sh`).
 - Pre-create ConfigMaps/Secrets for the release.
 - Initialize infrastructure (buckets, databases, etc.).
 **Common postsync tasks:**
 - Wait for operator/webhook readiness.
 - Apply CRD instances (ServiceMonitor, PrometheusRule, Certificate).
 - Perform post-deployment setup (cluster initialization, user creation).
 Example (helmfile.yaml.gotmpl, lines 64–72):
 ```yaml
 hooks:
  - events: ["postsync"]
    command: bash
    args:
      - -c
      - |
        kubectl rollout status deploy/cert-manager -n cert-manager --timeout=120s
        kubectl apply -f - <<'EOF'
        # ClusterIssuer and Certificate CRD instances...
        EOF
 ```
 ### ServiceMonitor Pattern
 One file per service in `k8s/monitoring/servicemonitors/svc-<name>.yaml`.
 **Key elements:**
 - `namespaceSelector`: Match the namespace where the app runs.
 - `selector.matchLabels`: Match the app label from the Deployment (e.g., `app.kubernetes.io/name: argocd-metrics`).
 - `endpoints.port`: Name of the metrics port in the Service.
 - `interval`: Scrape frequency (e.g., `30s`).
 Example (`k8s/monitoring/servicemonitors/argocd.yaml`, lines 1–37):
 ```yaml
 apiVersion: monitoring.coreos.com/v1
 kind: ServiceMonitor
 metadata:
  name: argocd
  namespace: monitoring
  labels:
    release: kube-prometheus-stack  # Required: links to Prometheus release
 spec:
  namespaceSelector:
    matchNames:
      - cicd  # App namespace
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-metrics  # Matches Deployment pod label
  endpoints:
    - port: metrics  # Service port name (not port number)
      interval: 30s
      scrapeTimeout: 10s
 ```
 ### PrometheusRule Pattern
 One file per service in `k8s/monitoring/alerts/svc-<name>-rules.yaml`.
 **Structure:**
 - `groups[].name`: Logical grouping (e.g., `minio.rules`).
 - `groups[].rules[].alert`: Alert name.
 - `expr`: PromQL expression (5-minute windows for rate alerts).
 - `for`: Duration threshold (e.g., `10m`).
 - `labels.severity`: `critical`, `warning`.
 - `annotations`: summary + description (use `{{ $value }}` for metric value).
 Example (`k8s/monitoring/alerts/svc-minio-rules.yaml`, lines 1–47):
 ```yaml
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
 metadata:
  name: minio-rules
  namespace: storage
 spec:
  groups:
    - name: minio.rules
      interval: 15s
      rules:
        - alert: MinIOHighErrorRate
          expr: |
            (
              sum(rate(minio_s3_requests_total{error="true"}[5m]))
              /
              sum(rate(minio_s3_requests_total[5m]))
            ) > 0.05
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "High error rate on MinIO"
            description: "{{ $value | humanizePercentage }}"
 ```
 ### Grafana Dashboard Pattern
 Dashboards are stored as JSON ConfigMaps in `k8s/monitoring/dashboards/` and auto-loaded via helmfile postsync hook (see helmfile.yaml.gotmpl, lines 471–477).
 **6-row template (recommended layout):**
 1. **Availability:** Uptime, error rate, latency (SLO band).
 2. **Resources:** CPU, memory, disk usage, network I/O.
 3. **Domain metrics:** Service-specific KPIs (throughput, queue depth, cache hit rate).
 4. **Logs:** Recent error logs from Loki.
 5. **SLO:** SLI tracking (burn rate, error budget).
 6. **Related dashboards:** Links to dependent services.
 Auto-load hook (helmfile.yaml.gotmpl, lines 471–477):
 ```yaml
 hooks:
  - events: ["postsync"]
    command: kubectl
    args:
      - apply
      - -f
      - k8s/monitoring/dashboards/
 ```
 ---
 ## Integration Checklist
 When adding a new service to the homelab, verify the following:
 - [ ] **Chart pinning:** Helm chart version pinned (`~1.0` format in helmfile, not floating `latest`).
 - [ ] **Secrets management:** All secrets stored in Vault (none in values.yaml, ConfigMap, or CLI flags).
 - [ ] **Metrics endpoint:** Service exports `/metrics` endpoint (Prometheus format).
 - [ ] **ServiceMonitor:** Created and auto-scraped by Prometheus operator.
 - [ ] **PrometheusRule:** Alert rules defined for errors, latency, SLO violations.
 - [ ] **Grafana dashboard:** 6-row template auto-loaded via ConfigMap.
 - [ ] **Ingress:** Rule added if external access needed (see `k8s/ingress/ingress.yaml`).
 - [ ] **OIDC integration:** If UI component, integrated with Authentik (see `core iam bootstrap`).
 ---
 ## Shared/Reusable Repos & Image Publication
 ### When to Publish as Public GHCR
 **Rule:** If a service's chart + image source lives in a separate repo (not in homelab), it **must** be published as a **public GitHub repo** under the `Riotpiaole` org.
 **Rationale:**
 - Local in-cluster charts are fine for infra-owned services.
 - Shared/reusable service charts should be version-pinned and publicly available for:
  - Reuse across different clusters (other labs, staging, prod).
  - Consumption by Argo CD apps (CI/CD pipeline).
  - Independent evolution without tight coupling to homelab repo.
 ### Workflow
 **Develop locally:**
 1. Create companion repo (e.g., `kafaka-management-service`, `queue-operator`) in a private or public repo.
 2. Include Dockerfile and Helm chart.
 **Build & publish:**
 1. Set up CI/CD in the companion repo (GitHub Actions).
 2. Build image and push to GHCR: `ghcr.io/Riotpiaole/<repo>:<tag>`.
 3. Tag release and publish chart (npm registry, GitHub releases, or OCI registry).
 **Reference in homelab:**
 1. Pin image tag + chart version in helmfile.yaml.gotmpl.
 2. Chart `repositories` block references the public Helm repo (or OCI registry).
 **Example (commit cd1c569 — SQS charts):**
 ```yaml
 - name: management-service
  namespace: sqs
  chart: k8s/sqs/charts/management-service
  values:
    - image:
        repository: ghcr.io/Riotpiaole/kafaka-management-service
        tag: v1.0.0  # Pinned tag from public build
 ```
 ---
 ## Cross-References
 - **CLAUDE.md gotchas:** Field name = variable name, MinIO `--env` flag exposure, helmfile template syntax (`{{ env "VAR" }}` not `${VAR}`), CNPG password templating.
 - **README.md:** kubectl context setup, port-forward aliases, cluster topology.
 - **USAGE.md:** IAM bootstrap, secret management, deployment recipes.
@@ -0,0 +1,342 @@
 # Core CLI Tools: Decision Guide
 The `core` CLI is your primary tool for cluster auth, node operations, Vault secrets management, IAM administration, and object storage. This doc covers **when** to reach for `core` vs `kubectl`/`helmfile`, the two separate auth domains that power different commands, and the complete command inventory.
 For exhaustive flag-level detail on each subcommand, see [~/workplace/core/USAGE.md](../../core/USAGE.md).
 ---
 ## The Two Auth Domains
 The `core` CLI maintains **two independent authentication systems** that gate different command families. Confusing them causes authentication failures.
 ### Domain 1: Node/Talos Operations (`core auth`)
 **Gates:** Node-level commands, Talos service management, cluster status.
 **Login mechanism:** `core auth login-oob`  
 - Out-of-band device-code flow via Authentik
 - Token cached in `~/.core/token` (24h expiry)
 - Required before: `core nodes`, `core status <ip>`, `core services <ip>`, `core logs <ip>`, `core log-svc <ip> <svc>`, `core dmesg <ip>`, `core config`, `core upgrade`, `core shutdown`, `core reboot`, `core node`, `core pods clean`
 **Check auth state:**
 ```bash
 core auth status          # Show token expiry
 core auth clear           # Force re-auth on next command
 ```
 **When to use:**
 - Troubleshooting node-level issues (crashes, disk space, network on a specific Talos node)
 - Checking Talos service health (etcd, kubelet, scheduler, etc.)
 - Upgrading cluster OS or managing node lifecycle
 - Reading kernel logs (dmesg) or kubelet logs on specific nodes
 ### Domain 2: Vault Secrets (`core secrets`)
 **Gates:** Secrets management (read/write), secret listing, secret export.
 **Login mechanism:** `core secrets login`  
 - Interactive login to Vault (opens browser, approves auth)
 - Reads/writes to `~/.config/talos/secrets.toml`
 - Required before: `core get <path> [--key KEY]`, `core put <path> key=value`, `core secrets list`, `core secrets export`, `core secrets exec`
 **Check auth state:**
 ```bash
 core secrets status       # Show Vault token / scopes
 core secrets clear        # Force re-auth on next command
 ```
 **When to use:**
 - Reading or writing cluster secrets (API keys, database passwords, OAuth client secrets)
 - Managing application credentials in Vault (centralized secret store)
 - Rotating secrets for services (e.g., OAuth2 client secrets)
 - Listing all secrets under a path
 - Bootstrapping `.env` files for helmfile deployment (via `vsource`)
 ---
 ## Auth Bug Fixes
 **Bug #1 (Fixed):** `core status <NODE>` is a **node-ops command**, not an auth verifier. It does NOT verify that you're authenticated to Vault. To verify auth state, use the correct domain-specific command:
 - For node access: `core auth status`
 - For Vault access: `core secrets status`
 **Bug #2 (Fixed):** `core secrets login` and `core auth login-oob` are **NOT interchangeable**. They gate completely different systems:
 - `core auth login-oob` gates node/cluster operations
 - `core secrets login` gates Vault secret read/write
 - You may be authenticated to one domain and not the other. Check state separately.
 ---
 ## When to Reach for `core`
 | Scenario | Tool | Why |
 |----------|------|-----|
 | Pod/deployment issue | `kubectl` | Pods, replicas, rollouts, events |
 | Package release management | `helmfile` | Install/upgrade Helm charts |
 | Node crashes, disk, kernel panic | `core auth` + node commands | Direct access to Talos node state |
 | Service unreachable on Talos node | `core status <ip>`, `core services <ip>` | Talos service health |
 | Database password rotation | `core secrets` + `core put` | Vault secret write |
 | Need API key for app deployment | `core secrets` + `core get` | Fetch from Vault, inject via `vsource` |
 | Create OAuth2 app in Authentik | `core iam create-app` | Manage federated identity apps |
 | Add user to service (MinIO, Grafana, etc.) | `core iam add-member` | Group membership binding |
 | S3 object upload/download | `core bucket` | MinIO operations |
 ---
 ## Command Inventory
 Complete list of `core` subcommands, organized by domain. See [~/workplace/core/USAGE.md](../../core/USAGE.md) for usage flags and examples.
 ### Authentication
 ```bash
 # Node/Talos operations auth
 core auth login-oob       # Authenticate with Authentik (device code flow)
 core auth status          # Check token expiry / auth state
 core auth clear           # Clear cached auth token
 # Vault secrets auth
 core secrets login        # Authenticate to Vault (interactive)
 core secrets status       # Check Vault token / scopes / expiry
 core secrets clear        # Clear Vault auth token
 ```
 ### Cluster & Node Operations
 ```bash
 # Node discovery and status
 core nodes                # List all cluster nodes (IP, hostname, status)
 core status <ip>          # Talos node overview (resources, uptime)
 core services <ip>        # List Talos services (etcd, kubelet, etc.)
 # Logs
 core logs <ip>            # Stream kubelet logs on node
 core log-svc <ip> <svc>   # Logs for specific Talos service (etcd, scheduler, etc.)
 core dmesg <ip>           # Kernel logs (ring buffer) from node
 # Node lifecycle
 core upgrade <ip>         # Upgrade Talos OS on node
 core shutdown <ip>        # Graceful shutdown
 core reboot <ip>          # Reboot node
 # Kubernetes context
 core config kube-list     # List available kubeconfig contexts
 core config kube-use <ctx>  # Switch to kubeconfig context (LAN vs WireGuard)
 # Pod cleanup
 core pods clean           # Delete Failed/Evicted/Terminating pods
 core node <node-name>     # Get node details (includes pod stats)
 ```
 ### Secrets Management (Vault)
 ```bash
 # Read secrets
 core get <path>                      # Fetch all fields under path
 core get <path> --key KEY_NAME       # Fetch specific field
 # Write secrets
 core put <path> key=value [key2=value2 ...]  # Store secrets in Vault
 # List and export
 core secrets list                    # List all secret paths
 core secrets export <path>           # Export secrets as shell-sourceable format
 # Execute with secrets in environment
 core secrets exec -- <command>       # Run command with secrets loaded in env
 ```
 **Convention:** Field name = variable name (SCREAMING_SNAKE_CASE). Never use `value=`.
 Example:
 ```bash
 # Write
 core put cluster/ANTHROPIC_API_KEY ANTHROPIC_API_KEY="sk-ant-..."
 # Read
 core get cluster/ANTHROPIC_API_KEY
 ```
 ### IAM Management (Authentik)
 ```bash
 # Groups
 core iam list-groups              # List all groups
 core iam create-group <name>      # Create new group
 # OAuth2 Applications
 core iam list-apps                # List all OAuth2 apps
 core iam create-app <name> --slug <slug> --redirect-uri <uri>  # Create app
 core iam describe-app <app>       # Show client ID, secret, URIs, scope claims
 core iam rotate-secret <app>      # Rotate OAuth2 client secret
 # User management
 core iam add-member <group> <username>  # Add user to group
 core iam bind-app <app> <group>   # Grant group access to application
 # Cleanup
 core iam delete-app <app>         # Delete OAuth2 application
 ```
 ### Object Storage (MinIO)
 ```bash
 # List and manage buckets
 core bucket list                  # List all buckets
 core bucket upload <bucket> <file> [<remote-path>]  # Upload file to S3
 core bucket download <bucket> <remote-path> <file>  # Download file from S3
 core bucket delete <bucket> <remote-path>  # Delete object from S3
 ```
 ### Utilities
 ```bash
 core help                         # Show command help
 core pf grafana                   # Port-forward to Grafana (localhost:3000)
 core pf prometheus                # Port-forward to Prometheus (localhost:9090)
 core pf minio                     # Port-forward to MinIO console (localhost:9001)
 core pf iam                       # Port-forward to Authentik (localhost:7000)
 ```
 ---
 ## Access Control Tiers
 The cluster uses a tiered access model based on authentication mechanism and network perimeter.
 ### Tier A: OIDC + RBAC (Authentik-enforced)
 Services with federated OIDC login and group-based role assignment.
 | Service | Auth Method | Group Claim | Role Binding |
 |---------|-------------|------------|--------------|
 | **Grafana** | Authentik OIDC | `groups` | Mapped to Admin / Viewer / Viewer |
 | **Argo CD** | Authentik OIDC | `groups` | RBAC role binding (policy.csv) |
 | **MinIO** | Authentik OIDC | `groups` / custom `policy` | Policy-based access (readwrite / readonly) |
 | **Forgejo** | Authentik OIDC | `email` (OAuth login only) | No group enforcement (open git repo) |
 | **kmsvc** | Vault JWT | `sub` / `aud` | Audience validation + service scope |
 **Provisioning:**
 ```bash
 # After core auth login-oob, bootstrap IAM apps:
 bash k8s/talos-iam/bootstrap-iam.sh
 ```
 ### Tier B: Network-Perimeter Only (No Authentik Enforcement)
 Services with no OIDC support (product limitation). Access is restricted to LAN/WireGuard perimeter only.
 | Service | Network Access | Use Case |
 |---------|----------------|----------|
 | **Portainer** | LAN + WireGuard only | Container UI, workload browsing |
 | **Longhorn** | LAN + WireGuard only | Storage volume management |
 | **Temporal** | LAN + WireGuard only | Workflow execution (auth TBD) |
 All three are reachable **only** via WireGuard/LAN-only Ingress rules. Zero remote access risk, but also zero federated identity. If remote Temporal access is needed, upgrade the chart's auth configuration or replace with an OIDC-compatible workflow platform.
 **Accessing Tier B services:**
 ```bash
 # From off-LAN, use WireGuard context
 core config kube-use admin@homelab-cluster-1
 core pf minio  # connects via 10.6.0.1:9001
 ```
 ---
 ## Workflow: Rotate an OAuth2 Application Secret
 ```bash
 # 1. Rotate secret in Authentik
 SECRET=$(core iam rotate-secret grafana | jq -r '.client_secret')
 # 2. Update Helm values
 vi k8s/logging/grafana-values.yaml
 # Set: GRAFANA_OIDC_CLIENT_SECRET="$SECRET"
 # 3. Redeploy the app
 helmfile apply -l app=grafana
 # 4. Verify new secret is in use
 core iam describe-app grafana | grep client_secret
 ```
 ---
 ## Workflow: Add User to Service
 ```bash
 # 1. Verify group exists (or create it)
 core iam list-groups | grep minio-admins
 # If not found:
 core iam create-group minio-admins
 # 2. Add user to group
 core iam add-member minio-admins alice
 # 3. Bind group to MinIO application
 core iam bind-app minio minio-admins
 # 4. User has access on next login
 # (OIDC login to MinIO → Authentik → group check → MinIO policy applied)
 ```
 ---
 ## Workflow: Rotate a Database Password (Vault)
 ```bash
 # 1. Generate new password
 NEW_PASS=$(openssl rand -hex 32)
 # 2. Store in Vault
 core put cluster/POSTGRES_ADMIN_PASSWORD POSTGRES_ADMIN_PASSWORD="$NEW_PASS"
 # 3. Update database user
 kubectl exec -n ddb pod/ddb-cluster-0 -- psql -U postgres -c \
  "ALTER USER postgres WITH PASSWORD '$NEW_PASS';"
 # 4. Update Helm values with new password reference
 # (Or if using helmfile hook: helmfile will re-run postInitApplicationSQL with new password)
 # 5. Restart pods to pick up new secret
 kubectl rollout restart -n <ns> deployment/<app>
 ```
 ---
 ## Common Questions
 **Q: I'm getting "not authenticated" on `core nodes`. What do I do?**
 A: Run `core auth login-oob`. Node commands use a different auth domain than secrets. After login, `core nodes` should work.
 **Q: I have a valid Vault token but `core get` fails. Why?**
 A: Check that both auth domains are active:
 ```bash
 core auth status        # Verify node auth is valid
 core secrets status     # Verify Vault auth is valid
 ```
 Both must succeed. If one is expired, re-auth that domain.
 **Q: Should I use `core bucket` or S3 CLI tools (aws-cli, s3cmd)?**
 A: Use `core bucket` for simplicity (no AWS credentials). Use s3cmd/aws-cli if you need advanced sync or bandwidth control. All three talk to the same MinIO backend.
 **Q: Can I add a user without creating a group first?**
 A: Groups are the unit of access control. Always create the group, then add users to it, then bind it to applications. Single-user bindings are not supported (by design).
 **Q: I rotated an OAuth2 secret but the app still fails to authenticate. What's next?**
 A: 
 1. Verify the new secret is stored: `core iam describe-app grafana`
 2. Check the deployment has the new secret: `kubectl get secret -n logging grafana-oidc -o yaml | grep client_secret`
 3. Restart the pod: `kubectl rollout restart -n logging deployment/grafana`
 4. Check logs: `kubectl logs -n logging deployment/grafana | grep -i oauth`
 ---
 ## See Also
 - [~/workplace/core/USAGE.md](../../core/USAGE.md) — Exhaustive command reference with flags and examples
 - [CLAUDE.md](../CLAUDE.md) § Sign In (Device Code Flow) — Quick reference for `core auth login-oob`
 - [CLAUDE.md](../CLAUDE.md) § Quick Shortcuts — One-liners for common tasks
 - [CLAUDE.md](../CLAUDE.md) § Integration Checklist — New service onboarding
@@ -142,7 +142,7 @@ kubectl exec -n ddb pod/ddb-cluster-2 -- \
 **All user passwords stored in Vault:**
 ```bash
 # Read password
-talos get cluster/STORY_CRATER_PG_PASSWORD --key STORY_CRATER_PG_PASSWORD
+core get cluster/STORY_CRATER_PG_PASSWORD --key STORY_CRATER_PG_PASSWORD
 # Inject into pod (auto via Secret volume)
 # Mount: /run/secrets/db-password
@@ -0,0 +1,198 @@
 # Infrastructure Practice Playbook
 Standardized procedures for troubleshooting, developing, deploying, and operating the homelab platform. Each procedure explicitly calls out where the `core` CLI fits vs `kubectl`/`helmfile`/direct cluster access. See `core-cli-tools.md` for the auth/secrets domain split; see `infra-troubleshooting.md` for quick patterns and gotchas.
 ## Procedure A: Troubleshoot a Service or Cluster Issue
 1. **Identify the domain.**
   - Vault/secrets: `core get`/`core put` failing, Vault unreachable.
   - Node/Talos: node crashes, disk full, kubelet unreachable, network issues.
   - Plain Kubernetes: pod CrashLoop, service 503, deployment stuck.
 2. **If Vault-adjacent (secrets, authentication failing):**
   - Run `core secrets status` first (NOT `core auth status` — common mistake).
   - If `Vault UNREACHABLE`, check DNS/networking:
     - No wildcard DNS exists; verify manual `/etc/hosts` entries (10.6.0.1 for WireGuard, 192.168.1.160 for LAN).
     - Ping the Vault service: `kubectl get svc -n vault | grep vault`.
   - If `Vault token not cached`, run `core secrets login`, approve device code in browser.
   - Verify: `core secrets status` shows `✓ Authenticated`.
 3. **If node/Talos-adjacent (kubelet logs, node state, services failing):**
   - Run `core auth status` first.
   - If token expired, run `core auth login-oob`, approve device code in browser.
   - Then run `core nodes` to list cluster nodes.
   - For a specific node, run `core status <ip>` (Talos state).
   - Inspect Talos services: `core services <ip>` (kubelet, etcd, controller, etc.).
   - Check service logs: `core logs <ip>` (main Talos logs) or `core log-svc <ip> kubelet` (specific service).
   - Consult `infra-troubleshooting.md` for Pod stuck in CrashLoopBackOff and kubelet restart patterns.
 4. **If plain Kubernetes (pod/deployment/service issues):**
   - Consult root `TROUBLESHOOTING.md` for the layer-before-tool SRE methodology (procedures 1–10).
   - Use `infra-troubleshooting.md` § Quick Patterns for common diagnoses:
     - CrashLoopBackOff: `kubectl logs -n <ns> <pod> --tail=50` + `kubectl describe pod -n <ns> <pod> | grep -A 10 Events`.
     - Service 503: `kubectl get endpoints -n <ns> <svc>` (endpoints missing?) + `kubectl get pods -n <ns> -o wide` (pods not Ready?).
     - Helm release stuck: `helmfile status | grep -E "FAILED|UNKNOWN|PENDING"` + `helm status <release> -n <ns> --show-resources`.
   - If still unclear, escalate to `kubectl get all -n <ns>` and review resource events.
 5. **For dashboards and live metrics:**
   - Use `core pf grafana` (port-forward to localhost:3000) rather than raw `kubectl port-forward` — keeps forwarded ports consistent.
   - If Prometheus unavailable, check: `kubectl get pods -n monitoring | grep prometheus`.
   - If ServiceMonitor not scraping, verify: `kubectl get servicemonitor -A | grep <name>` and inspect `.spec.selector` matches the target pod's app label.
 ---
 ## Procedure B: Launch/Develop a New Service or POC
 1. **Plan the service.**
   - Determine namespace (e.g., `sqs`, `temporal`, `databases`, `monitoring`).
   - Decide if metrics exported (most should) and if OIDC-gated.
   - Sketch a Helm values.yaml structure (secrets, replicas, resource requests, affinity).
 2. **Authenticate to Vault.**
   - Run `core secrets login` and approve device code in browser.
   - Verify: `core secrets status` shows `✓ Authenticated`.
   - You'll need Vault access to store service secrets in step 5.
 3. **Create service Helm chart directory.**
   - Create `k8s/<service>/` with at minimum:
     - `values.yaml` (Helm values for deployment, service, replicas, resource limits).
     - `charts/` subdirectory for any custom local Helm charts (optional).
   - Follow naming conventions from `coding-standards.md`.
 4. **Add Helm release to helmfile.**
   - Open `helmfile.yaml.gotmpl`.
   - Add release block under `releases:` section, following this structure:
     ```yaml
     - name: <service>
       namespace: <namespace>
       chart: <chart-repo>/<chart-name>
       version: ~1.0  # pin major.minor, allow patch updates
       needs:
         - <dependency-namespace>/<dependency-release>  # if applicable
       values:
         - k8s/<service>/values.yaml
         - secretsInline:
             DB_PASSWORD: "{{ env \"<SERVICE>_DB_PASSWORD\" }}"
     ```
   - Consult `coding-standards.md` for `needs:` ordering (example: sqs section shows strimzi-operator → kafka-cluster → queue-crd → management-service).
   - Reference real example: root helmfile's `sqs` section.
 5. **If the service needs secrets (DB password, API key, OAuth secret):**
   - Generate value (e.g., `openssl rand -hex 32` for passwords).
   - Store in Vault: `core put cluster/<SERVICE>_<KEY> <SERVICE>_<KEY>="value"`.
   - **Critical gotcha:** field name MUST equal variable name (e.g., `FORGEJO_ADMIN_PASSWORD=` not `value=`) per `coding-standards.md` § Vault field=variable convention.
   - Reference in values.yaml via `{{ env "VARIABLE_NAME" }}` (Helmfile Go template syntax, NOT shell `${VAR}`).
   - Do NOT hardcode secrets in values.yaml or ConfigMaps.
 6. **Verify Helm syntax before deploy.**
   - Run `helmfile lint` (catches template errors, duplicate releases).
   - Run `helmfile diff -l name=<service>` (show what will be deployed).
   - Review diff for correctness (verify env var substitutions, resource limits, affinity rules).
 7. **Deploy the service.**
   - Run `helmfile apply -l name=<service>`.
   - Monitor: `kubectl get pods -n <namespace> -w` (watch until Running).
   - If pods stuck: `kubectl describe pod -n <namespace> <pod-name>` (check Events for SchedulingFailed, ImagePullBackOff, etc.).
 8. **If the service exports `/metrics` (Prometheus format):**
   - Create ServiceMonitor: `k8s/monitoring/servicemonitors/svc-<name>.yaml`.
     - `.spec.selector.matchLabels` must match the service's pod labels (usually `app: <service>`).
     - `.spec.endpoints[0].port` must match the service port name or number exporting metrics.
   - Create PrometheusRule: `k8s/monitoring/alerts/svc-<name>-rules.yaml`.
     - Include error rate, latency, and SLO alert rules.
     - Use `prometheus` as the rule group.
   - Create Grafana dashboard: `k8s/monitoring/dashboards/svc-<name>.yaml`.
     - Use 6-row template: Availability, Resources, Domain metrics, Logs, SLO, Related.
     - See README.md § Example Applications for a full walkthrough.
   - Verify scrape: `kubectl get servicemonitor -A | grep <name>` and check Prometheus Targets UI for green status.
 9. **If OIDC/IAM-gated (admin UI, restricted API):**
   - Create app in Authentik: `core iam create-app "my-service" --slug my-service --redirect-uri "https://my-service.riotpiao.homelab.com/callback"`.
   - Bind app to group: `core iam bind-app my-service <group>` (e.g., `grafana-admins` for admin-only UI).
   - Retrieve credentials: `core iam describe-app my-service` (client ID, client secret).
   - Deploy secret: `kubectl create secret generic <service>-oidc --from-literal=client-id=<ID> --from-literal=client-secret=<SECRET> -n <namespace>`.
   - Reference secret in values.yaml: mount via `.spec.template.spec.containers[].env` or volumeMounts.
   - See `core-cli-tools.md` § Access Control Tiers for Tier A (OIDC + RBAC) vs Tier B (network perimeter only).
 10. **Verify service is live.**
    - Pods: `kubectl get pods -n <namespace> -o wide` (all Running, 1/1 Ready).
    - Metrics (if applicable): `kubectl get servicemonitor -A | grep <name>` and visit Prometheus Targets or Grafana dashboard.
    - Endpoint: If publicly routed via Ingress, verify `/etc/hosts` entry (10.6.0.1 for WireGuard, 192.168.1.160 for LAN) and `curl https://my-service.riotpiao.homelab.com/health` (or equivalent health endpoint).
    - Logs: `kubectl logs -n <namespace> <pod>` (no errors).
 ### Definition of Done (Per Service)
 - [ ] Helm chart version pinned (~1.0 format in helmfile)
 - [ ] All secrets in Vault (none in values.yaml or ConfigMap)
 - [ ] `/metrics` endpoint exported (if applicable)
 - [ ] ServiceMonitor resource created (if metrics exported)
 - [ ] PrometheusRule with error/latency/SLO alerts (if metrics exported)
 - [ ] Grafana dashboard (if metrics exported; 6-row template: Availability, Resources, Domain, Logs, SLO, Related)
 - [ ] Ingress rule (if external access needed)
 - [ ] OIDC integration via `core iam` (if UI component)
 - [ ] Verified: `helmfile diff` clean, pods Running, dashboard live or `/metrics` returning 200
 ---
 ## Procedure C: Operate the Cluster (Node Health, Context, Cleanup)
 1. **Daily health check.**
   - Check auth: `core auth status` (if OK, node ops will work).
   - List nodes: `core nodes`.
   - For each node, check Talos state: `core status <ip>`.
   - Check K8s nodes: `kubectl get nodes -o wide` (all Ready, no NotReady).
   - Check pod pressure: `kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, memory: .status.allocatable.memory, pods: .status.allocatable.pods}'`.
 2. **Troubleshoot a specific node.**
   - Get node IP: `core nodes` and note the IP.
   - Check Talos services: `core services <ip>` (kubelet, etcd, controller should be running).
   - Check service logs: `core logs <ip>` (main Talos daemon logs).
   - Filter to specific service: `core log-svc <ip> kubelet` (kubelet logs only).
   - Restart a service if needed: `core restart <ip> kubelet` (graceful kubelet restart).
 3. **Pod cleanup (Failed, Evicted, Terminating pods).**
   - Run `core pods clean` (scans all namespaces, removes stale pods).
   - Verify: `kubectl get pods -A | grep -E "Failed|Evicted"` (should be empty).
 4. **Switch kubectl context (when off-LAN, on WireGuard).**
   - List available contexts: `core config kube-list`.
   - Switch to WireGuard path (10.6.0.1:6443): `core config kube-use admin@homelab-cluster-1`.
   - **Known limitation:** `core config use <talos-context>` doesn't map to WireGuard; use `kube-use` directly.
   - Verify: `kubectl cluster-info` shows 10.6.0.1 (not 192.168.1.213).
 5. **MinIO bucket operations (if managing data/backups).**
   - List buckets: `core bucket list`.
   - Upload file: `core bucket upload <bucket> <local-file>`.
   - Download file: `core bucket download <bucket> <remote-file> -o <local-file>`.
   - Delete file: `core bucket delete <bucket> <remote-file>`.
 6. **Bootstrap or hardware runbooks (infrequent).**
   - **Fresh cluster setup:** See README.md § Bootstrap Order (14 steps).
   - **Adding a new Talos node:** See README.md § Adding Hardware.
   - Do not re-explain those long procedures here; consult README.md directly.
 ---
 ## Notes
 **Queue subsystem (Kafka/kmsvc/Temporal namespace auto-registration):** Already deployed and stable. If re-deploying:
 - Primary deploy method: `helmfile apply -l namespace=sqs` (live from root helmfile).
 - Alternate isolated iterate path: `k8s/sqs/helmfile.yaml.gotmpl` (not recommended for production).
 - Planned future: GitOps via `k8s/sqs/argocd/` (companion repo, not yet active).
 - **Critical rule:** Temporal namespace registration is automatic via `queue-operator`; never manually `temporal operator namespace create` for any namespace referenced by a Queue's `temporal.io/namespace` label. See `~/workplace/kmsvc-manage/CLAUDE.md` ("Temporal Namespace Registration") for the full rule and why.
 **Shared/reusable service repositories:** If a service's Helm chart and container image live in a separate repository, they must be:
 - Published as a public GitHub repository under the `Riotpiaole` organization.
 - Images pushed to GHCR (`ghcr.io/riotpiaole/...`) for public pullability.
 - Consult `coding-standards.md` § Shared/Reusable Repos for the full publishing rule.
 ---
 ## Cross-References
 - **core-cli-tools.md:** Auth/secrets domain split, command inventory, when to use `core` vs `kubectl`.
 - **coding-standards.md:** Helm naming conventions, `needs:` ordering rules, helmfile template syntax (`{{ env "VAR" }}` not `${VAR}`), Vault field=variable convention, shared-repo publishing rule.
 - **infra-troubleshooting.md:** Quick patterns (CrashLoopBackOff, 503, helm stuck), gotchas, hard rules.
 - **USAGE.md:** Exhaustive `core` command reference.
 - **README.md:** Bootstrap order, hardware addition, example app walkthrough, 6-row Grafana dashboard template.
 - **root TROUBLESHOOTING.md:** Generic Kubernetes SRE layer-before-tool methodology (10 diagnostic procedures).
@@ -0,0 +1,66 @@
 # Troubleshooting: Homelab Cluster Operations
 Troubleshooting procedures and operational gotchas for homelab cluster. Cross-reference root `TROUBLESHOOTING.md` for full SRE diagnostic methodology (layer-before-tool, control-plane/kubelet/networking); this doc is the quick cheatsheet + gotchas specific to this repo.
 ---
 ## Quick Patterns
 ### Pod stuck in CrashLoopBackOff
 ```bash
 k logs -n <ns> <pod> --tail=50
 k describe pod -n <ns> <pod> | grep -A 10 Events
 ```
 ### Service unreachable (503)
 ```bash
 # Check endpoints exist
 k get endpoints -n <ns> <svc>
 k get pods -n <ns> -o wide
 # Test connectivity
 k exec -it <pod> -- curl http://<svc>.<ns>.svc.cluster.local:8080
 ```
 ### Helm release stuck
 ```bash
 helmfile status | grep -E "FAILED|UNKNOWN|PENDING"
 helm status <release> -n <ns> --show-resources
 k logs -n <ns> deploy/<app> | head -100
 ```
 ---
 ## Hard Rules (Never Violate)
 🔴 **NEVER delete a PVC unless there are replicas or backups.** A PVC deletion = permanent data loss. Verify replication status first.
 ```bash
 # Before ANY PVC delete:
 k get pvc -n <ns> <pvc>
 k get pv <pvc-backing-pv> -o json | jq '.spec'  # check replication config
 # For Longhorn: verify replicas >= 2
 k get longhorn-volume -n longhorn-system <vol> -o json | jq '.status.replicaStatus'
 # For PostgreSQL: verify standby replicas are healthy
 k exec -n ddb pod/ddb-cluster-0 -- psql -U postgres -c "SELECT * FROM pg_stat_replication;"
 ```
 ---
 ## Project Gotchas
 - **Field name = variable name:** In Vault, use `core put cluster/KEY KEY="value"` (never `value=`)
 - **vsource expansion:** `.env` values must be empty (`KEY=`) to fetch from Vault; hardcoded values pass through
 - **Helmfile template syntax:** Use `{{ env "VAR" }}` not `${VAR}` (shell syntax, not Go template)
 - **CNPG initialization:** Wait for cluster to be ready before creating databases (use `postInitApplicationSQL`, not helmfile hooks)
 - **MinIO credentials:** Use Secret volumes, never `--env` flag (exposes in `kubectl describe`)
 - **Temporal namespace registration:** never manually `temporal operator namespace create` for a namespace a Queue's `temporal.io/namespace` label references — `queue-operator` registers it automatically. See `~/workplace/kmsvc-manage/CLAUDE.md` ("Temporal Namespace Registration") for the full rule and why.
 - **kubectl hangs / connection refused:** you're probably off-LAN. `kubectl config use-context admin@homelab-cluster-1` (WireGuard path, `10.6.0.1:6443`) — the default context (`admin@homelab-cluster`, `192.168.1.213:6443`) only works on-LAN. See README.md "kubectl Context".
 - **No wildcard DNS:** `*.riotpiao.homelab.com` isn't a real DNS zone — every hostname is a manual `/etc/hosts` line (`10.6.0.1` for WireGuard, the Cilium LB-IPAM IP for LAN). Adding an Ingress `host:` rule doesn't make it resolvable; you must also add the `/etc/hosts` line, on every client machine that needs it.
 - **gRPC through nginx ingress:** add `nginx.ingress.kubernetes.io/backend-protocol: "GRPC"` to proxy h2c to a plaintext-gRPC backend (see `temporal-grpc` Ingress in `k8s/ingress/ingress.yaml`). TLS still terminates at nginx via the wildcard cert — clients need `--tls`, not client certs.
 - **Soft podAntiAffinity doesn't rebalance existing pods.** `preferredDuringSchedulingIgnoredDuringExecution` only applies at scheduling time — pods that landed on the same node before the constraint existed (or before other nodes were `Ready`) stay there forever. Fixing the affinity config in values.yaml/CRD only affects *future* pod creation; existing skew needs `kubectl delete pod <name>` (one at a time, verify healthy before the next) to force a reschedule under the now-correct constraint. Hit this with Cassandra, the Kafka `KafkaNodePool`, and CNPG's `ddb-cluster` all stacking onto `talos-cp-1`.
 - **Cassandra/StatefulSet pod deleted+recreated with no PVC (ephemeral storage) can crash-loop on rejoin:** `Other bootstrapping/leaving/moving nodes detected` or `A node required to move the data consistently is down (/<old-ip>)` — the ring still holds a stale gossip entry for the deleted pod's old IP. Fix: `kubectl exec <a live cassandra pod> -- nodetool assassinate <stale-ip>` from a healthy node, then let the crash-looping pod's next restart retry.
 - **`helm upgrade` failing with `conflict ... using v1: .data.<field>` after a manual `kubectl apply` patch:** you (or an agent) hand-patched a resource Helm manages, and `kubectl apply`'s default client-side-apply field manager now owns that field. Reclaim it once: `kubectl get <resource> -o yaml | kubectl apply -f - --server-side --field-manager=helm --force-conflicts`, then retry the plain `helm upgrade` (no `--force` needed).
 - **`~/.config/talos/secrets.toml` still uses the legacy `talos` name in its path:** Even though the CLI binary is `core`, the config file it reads is `~/.config/talos/secrets.toml` (not `~/.config/core/...`). This is real CLI behavior, not a doc bug — don't try to rename the path, just be aware if you're troubleshooting Vault access (`core get`/`core put` failing) and checking whether `~/.config/talos/secrets.toml` exists or is readable.
@@ -144,10 +144,10 @@ KEYS "kmsvc:inflight:*" | wc -l
 **Requires JWT from Authentik:**
 ```bash
 # Get token (device code flow)
-talos secrets login
+core secrets login
 # Use token
-export JWT_TOKEN=$(talos get cluster/kmsvc/jwt-token --key jwt-token)
+export JWT_TOKEN=$(core get cluster/kmsvc/jwt-token --key jwt-token)
 curl -H "Authorization: Bearer $JWT_TOKEN" https://kmsvc.riotpiao.homelab.com/v1/queues
 ```
@@ -17,10 +17,10 @@
 ```bash
 # Browser: https://vault.riotpiao.homelab.com
 # Auth method: OIDC → "Sign in with Authentik" (federated)
-# Or: Device code → talos secrets login (CLI)
+# Or: Device code → core secrets login (CLI)
 # Via CLI (device code flow)
-talos secrets login
+core secrets login
 # → Opens browser, approve device code
 # → Token cached in ~/.talos/vault
 ```
@@ -28,18 +28,18 @@ talos secrets login
 **2. Store a secret:**
 ```bash
 # Field name = variable name (SCREAMING_SNAKE_CASE)
-talos put cluster/ANTHROPIC_API_KEY ANTHROPIC_API_KEY="sk-..."
+core put cluster/ANTHROPIC_API_KEY ANTHROPIC_API_KEY="sk-..."
-talos put cluster/STORY_CRATER_DB_PASS STORY_CRATER_DB_PASS="dbpass123"
+core put cluster/STORY_CRATER_DB_PASS STORY_CRATER_DB_PASS="dbpass123"
 ```
 **3. Retrieve a secret:**
 ```bash
 # Always use --key flag
-talos get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
+core get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
 # → sk-...
 # Full secret as JSON
-talos get cluster/ANTHROPIC_API_KEY --json
+core get cluster/ANTHROPIC_API_KEY --json
 ```
 **4. Load into shell (helmfile, scripts):**
@@ -78,7 +78,7 @@ cluster/
 **Store generated secret immediately (keeps it out of shell history):**
 ```bash
 # Generate & store in one command
-talos put cluster/GRAFANA_OIDC_CLIENT_SECRET \
+core put cluster/GRAFANA_OIDC_CLIENT_SECRET \
  GRAFANA_OIDC_CLIENT_SECRET="$(openssl rand -hex 32)"
 ```
@@ -86,7 +86,7 @@ talos put cluster/GRAFANA_OIDC_CLIENT_SECRET \
 ```bash
 # Create Secret using Vault secret
 kubectl create secret generic grafana-oidc \
-  --from-literal=client-secret="$(talos get cluster/GRAFANA_OIDC_CLIENT_SECRET --key GRAFANA_OIDC_CLIENT_SECRET)" \
+  --from-literal=client-secret="$(core get cluster/GRAFANA_OIDC_CLIENT_SECRET --key GRAFANA_OIDC_CLIENT_SECRET)" \
  -n logging
 ```
@@ -96,7 +96,7 @@ kubectl create secret generic grafana-oidc \
 NEW_PASS=$(openssl rand -base64 24)
 # 2. Store in Vault
-talos put cluster/OLD_DB_PASS OLD_DB_PASS="$NEW_PASS"
+core put cluster/OLD_DB_PASS OLD_DB_PASS="$NEW_PASS"
 # 3. Update database user
 psql -h ddb-cluster-rw.ddb.svc.cluster.local -U postgres \
@@ -113,7 +113,7 @@ k rollout restart -n story-crater-backend deployment/app
 **JWT token from CLI:**
 ```bash
 # After device code login
-talos secrets login
+core secrets login
 # Token is cached and auto-renewed
 # Use for API calls
@@ -127,7 +127,7 @@ curl -H "X-Vault-Token: $VAULT_TOKEN" \
 **Vault status:**
 ```bash
 # Check if sealed
-talos status vault
+core status
 # If sealed (disaster recovery):
 # See /TROUBLESHOOTING.md § Vault Sealed
@@ -139,10 +139,10 @@ talos status vault
 # Logs stored in Loki under vault namespace
 # View recent access
-talos audit log --limit 50
+core audit log --limit 50
 # Export for compliance
-talos audit export --format json > vault-audit.json
+core audit export --format json > vault-audit.json
 ```
 ## Security Rules
@@ -177,7 +177,7 @@ k rollout restart -n iam statefulset/vault
 **Vault is sealed:**
 ```bash
 # Check status
-talos status vault
+core status
 # If sealed, use unseal keys (stored in MinIO backup)
 # See /TROUBLESHOOTING.md § Vault Sealed for recovery steps
@@ -186,11 +186,11 @@ talos status vault
 **Secret not found:**
 ```bash
 # Verify path exists
-talos list cluster
+core list cluster
 # Check secret name (case-sensitive)
-talos get cluster/anthropic_api_key --key anthropic_api_key  # won't work
+core get cluster/anthropic_api_key --key anthropic_api_key  # won't work
-talos get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY  # correct
+core get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY  # correct
 ```
 **vsource not expanding secrets:**
@@ -200,7 +200,7 @@ grep ANTHROPIC_API_KEY .env
 # → Should be: ANTHROPIC_API_KEY=  (empty, not a value)
 # Verify Vault is accessible
-talos get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
+core get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
 # → Should return secret
 # Run vsource explicitly
@@ -0,0 +1,11 @@
 {
  "version": 1,
  "skills": {
    "caveman": {
      "source": "JuliusBrussee/skills",
      "sourceType": "github",
      "skillPath": "skills/caveman/SKILL.md",
      "computedHash": "1902fa0b569912d0c05736d8d98a72097d9b82719aac88c0c1d03bb546f9176d"
    }
  }
 }
@@ -61,3 +61,23 @@ provider "registry.terraform.io/hashicorp/null" {
    "zh:f0ce55d8d9ffdb33dab612b1246f9bab060a9d54fc32ce2b4a038646155660af",
  ]
 }
 provider "registry.terraform.io/hashicorp/vault" {
  version     = "4.8.0"
  constraints = "~> 4.0"
  hashes = [
    "h1:GPfhH6dr1LY0foPBDYv9bEGifx7eSwYqFcEAOWOUxLk=",
    "zh:269ab13433f67684012ae7e15876532b0312f5d0d2002a9cf9febb1279ce5ea6",
    "zh:4babc95bf0c40eb85005db1dc2ca403c46be4a71dd3e409db3711a56f7a5ca0e",
    "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3",
    "zh:86e27c1c625ecc24446a11eeffc3ac319b36c2b4e51251db8579256a0dbcf136",
    "zh:a32f31da94824009e26b077374440b52098aecb93c92ff55dc3d31dd37c4ea25",
    "zh:be0a18c6c0425518bab4fbffd82078b82036a88503b5d76064de551c9f646cbf",
    "zh:be5a77fdfd36863ebeec79cd12b1d13322ffad6821d157a0b279789fa06b5937",
    "zh:be8317d142a3caad74c7d936039ae27076a1b2b8312ef5208e2871a5f525977c",
    "zh:c94a84895a3d9954b80e983eed4603330a5cdbbd8eef5b3c99278c2d1402ef3c",
    "zh:de1fb712784dd8415f011ca5346a34f87fab6046c730557615247e511dbc7d98",
    "zh:e3eafae7da550f86cae395d6660b2a0e93ec8d2b0e0e5ef982ec762e961fc952",
    "zh:ff35fb1ab6add288f0f368981e56f780b50405accd1937131cba1137999c8d83",
  ]
 }
@@ -0,0 +1,375 @@
 Copyright (c) 2017 HashiCorp, Inc.
 Mozilla Public License Version 2.0
 ==================================
 1. Definitions
 --------------
 1.1. "Contributor"
    means each individual or legal entity that creates, contributes to
    the creation of, or owns Covered Software.
 1.2. "Contributor Version"
    means the combination of the Contributions of others (if any) used
    by a Contributor and that particular Contributor's Contribution.
 1.3. "Contribution"
    means Covered Software of a particular Contributor.
 1.4. "Covered Software"
    means Source Code Form to which the initial Contributor has attached
    the notice in Exhibit A, the Executable Form of such Source Code
    Form, and Modifications of such Source Code Form, in each case
    including portions thereof.
 1.5. "Incompatible With Secondary Licenses"
    means
    (a) that the initial Contributor has attached the notice described
        in Exhibit B to the Covered Software; or
    (b) that the Covered Software was made available under the terms of
        version 1.1 or earlier of the License, but not also under the
        terms of a Secondary License.
 1.6. "Executable Form"
    means any form of the work other than Source Code Form.
 1.7. "Larger Work"
    means a work that combines Covered Software with other material, in
    a separate file or files, that is not Covered Software.
 1.8. "License"
    means this document.
 1.9. "Licensable"
    means having the right to grant, to the maximum extent possible,
    whether at the time of the initial grant or subsequently, any and
    all of the rights conveyed by this License.
 1.10. "Modifications"
    means any of the following:
    (a) any file in Source Code Form that results from an addition to,
        deletion from, or modification of the contents of Covered
        Software; or
    (b) any new file in Source Code Form that contains any Covered
        Software.
 1.11. "Patent Claims" of a Contributor
    means any patent claim(s), including without limitation, method,
    process, and apparatus claims, in any patent Licensable by such
    Contributor that would be infringed, but for the grant of the
    License, by the making, using, selling, offering for sale, having
    made, import, or transfer of either its Contributions or its
    Contributor Version.
 1.12. "Secondary License"
    means either the GNU General Public License, Version 2.0, the GNU
    Lesser General Public License, Version 2.1, the GNU Affero General
    Public License, Version 3.0, or any later versions of those
    licenses.
 1.13. "Source Code Form"
    means the form of the work preferred for making modifications.
 1.14. "You" (or "Your")
    means an individual or a legal entity exercising rights under this
    License. For legal entities, "You" includes any entity that
    controls, is controlled by, or is under common control with You. For
    purposes of this definition, "control" means (a) the power, direct
    or indirect, to cause the direction or management of such entity,
    whether by contract or otherwise, or (b) ownership of more than
    fifty percent (50%) of the outstanding shares or beneficial
    ownership of such entity.
 2. License Grants and Conditions
 --------------------------------
 2.1. Grants
 Each Contributor hereby grants You a world-wide, royalty-free,
 non-exclusive license:
 (a) under intellectual property rights (other than patent or trademark)
    Licensable by such Contributor to use, reproduce, make available,
    modify, display, perform, distribute, and otherwise exploit its
    Contributions, either on an unmodified basis, with Modifications, or
    as part of a Larger Work; and
 (b) under Patent Claims of such Contributor to make, use, sell, offer
    for sale, have made, import, and otherwise transfer either its
    Contributions or its Contributor Version.
 2.2. Effective Date
 The licenses granted in Section 2.1 with respect to any Contribution
 become effective for each Contribution on the date the Contributor first
 distributes such Contribution.
 2.3. Limitations on Grant Scope
 The licenses granted in this Section 2 are the only rights granted under
 this License. No additional rights or licenses will be implied from the
 distribution or licensing of Covered Software under this License.
 Notwithstanding Section 2.1(b) above, no patent license is granted by a
 Contributor:
 (a) for any code that a Contributor has removed from Covered Software;
    or
 (b) for infringements caused by: (i) Your and any other third party's
    modifications of Covered Software, or (ii) the combination of its
    Contributions with other software (except as part of its Contributor
    Version); or
 (c) under Patent Claims infringed by Covered Software in the absence of
    its Contributions.
 This License does not grant any rights in the trademarks, service marks,
 or logos of any Contributor (except as may be necessary to comply with
 the notice requirements in Section 3.4).
 2.4. Subsequent Licenses
 No Contributor makes additional grants as a result of Your choice to
 distribute the Covered Software under a subsequent version of this
 License (see Section 10.2) or under the terms of a Secondary License (if
 permitted under the terms of Section 3.3).
 2.5. Representation
 Each Contributor represents that the Contributor believes its
 Contributions are its original creation(s) or it has sufficient rights
 to grant the rights to its Contributions conveyed by this License.
 2.6. Fair Use
 This License is not intended to limit any rights You have under
 applicable copyright doctrines of fair use, fair dealing, or other
 equivalents.
 2.7. Conditions
 Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted
 in Section 2.1.
 3. Responsibilities
 -------------------
 3.1. Distribution of Source Form
 All distribution of Covered Software in Source Code Form, including any
 Modifications that You create or to which You contribute, must be under
 the terms of this License. You must inform recipients that the Source
 Code Form of the Covered Software is governed by the terms of this
 License, and how they can obtain a copy of this License. You may not
 attempt to alter or restrict the recipients' rights in the Source Code
 Form.
 3.2. Distribution of Executable Form
 If You distribute Covered Software in Executable Form then:
 (a) such Covered Software must also be made available in Source Code
    Form, as described in Section 3.1, and You must inform recipients of
    the Executable Form how they can obtain a copy of such Source Code
    Form by reasonable means in a timely manner, at a charge no more
    than the cost of distribution to the recipient; and
 (b) You may distribute such Executable Form under the terms of this
    License, or sublicense it under different terms, provided that the
    license for the Executable Form does not attempt to limit or alter
    the recipients' rights in the Source Code Form under this License.
 3.3. Distribution of a Larger Work
 You may create and distribute a Larger Work under terms of Your choice,
 provided that You also comply with the requirements of this License for
 the Covered Software. If the Larger Work is a combination of Covered
 Software with a work governed by one or more Secondary Licenses, and the
 Covered Software is not Incompatible With Secondary Licenses, this
 License permits You to additionally distribute such Covered Software
 under the terms of such Secondary License(s), so that the recipient of
 the Larger Work may, at their option, further distribute the Covered
 Software under the terms of either this License or such Secondary
 License(s).
 3.4. Notices
 You may not remove or alter the substance of any license notices
 (including copyright notices, patent notices, disclaimers of warranty,
 or limitations of liability) contained within the Source Code Form of
 the Covered Software, except that You may alter any license notices to
 the extent required to remedy known factual inaccuracies.
 3.5. Application of Additional Terms
 You may choose to offer, and to charge a fee for, warranty, support,
 indemnity or liability obligations to one or more recipients of Covered
 Software. However, You may do so only on Your own behalf, and not on
 behalf of any Contributor. You must make it absolutely clear that any
 such warranty, support, indemnity, or liability obligation is offered by
 You alone, and You hereby agree to indemnify every Contributor for any
 liability incurred by such Contributor as a result of warranty, support,
 indemnity or liability terms You offer. You may include additional
 disclaimers of warranty and limitations of liability specific to any
 jurisdiction.
 4. Inability to Comply Due to Statute or Regulation
 ---------------------------------------------------
 If it is impossible for You to comply with any of the terms of this
 License with respect to some or all of the Covered Software due to
 statute, judicial order, or regulation then You must: (a) comply with
 the terms of this License to the maximum extent possible; and (b)
 describe the limitations and the code they affect. Such description must
 be placed in a text file included with all distributions of the Covered
 Software under this License. Except to the extent prohibited by statute
 or regulation, such description must be sufficiently detailed for a
 recipient of ordinary skill to be able to understand it.
 5. Termination
 --------------
 5.1. The rights granted under this License will terminate automatically
 if You fail to comply with any of its terms. However, if You become
 compliant, then the rights granted under this License from a particular
 Contributor are reinstated (a) provisionally, unless and until such
 Contributor explicitly and finally terminates Your grants, and (b) on an
 ongoing basis, if such Contributor fails to notify You of the
 non-compliance by some reasonable means prior to 60 days after You have
 come back into compliance. Moreover, Your grants from a particular
 Contributor are reinstated on an ongoing basis if such Contributor
 notifies You of the non-compliance by some reasonable means, this is the
 first time You have received notice of non-compliance with this License
 from such Contributor, and You become compliant prior to 30 days after
 Your receipt of the notice.
 5.2. If You initiate litigation against any entity by asserting a patent
 infringement claim (excluding declaratory judgment actions,
 counter-claims, and cross-claims) alleging that a Contributor Version
 directly or indirectly infringes any patent, then the rights granted to
 You by any and all Contributors for the Covered Software under Section
 2.1 of this License shall terminate.
 5.3. In the event of termination under Sections 5.1 or 5.2 above, all
 end user license agreements (excluding distributors and resellers) which
 have been validly granted by You or Your distributors under this License
 prior to termination shall survive termination.
 ************************************************************************
 *                                                                      *
 *  6. Disclaimer of Warranty                                           *
 *  -------------------------                                           *
 *                                                                      *
 *  Covered Software is provided under this License on an "as is"       *
 *  basis, without warranty of any kind, either expressed, implied, or  *
 *  statutory, including, without limitation, warranties that the       *
 *  Covered Software is free of defects, merchantable, fit for a        *
 *  particular purpose or non-infringing. The entire risk as to the     *
 *  quality and performance of the Covered Software is with You.        *
 *  Should any Covered Software prove defective in any respect, You     *
 *  (not any Contributor) assume the cost of any necessary servicing,   *
 *  repair, or correction. This disclaimer of warranty constitutes an   *
 *  essential part of this License. No use of any Covered Software is   *
 *  authorized under this License except under this disclaimer.         *
 *                                                                      *
 ************************************************************************
 ************************************************************************
 *                                                                      *
 *  7. Limitation of Liability                                          *
 *  --------------------------                                          *
 *                                                                      *
 *  Under no circumstances and under no legal theory, whether tort      *
 *  (including negligence), contract, or otherwise, shall any           *
 *  Contributor, or anyone who distributes Covered Software as          *
 *  permitted above, be liable to You for any direct, indirect,         *
 *  special, incidental, or consequential damages of any character      *
 *  including, without limitation, damages for lost profits, loss of    *
 *  goodwill, work stoppage, computer failure or malfunction, or any    *
 *  and all other commercial damages or losses, even if such party      *
 *  shall have been informed of the possibility of such damages. This   *
 *  limitation of liability shall not apply to liability for death or   *
 *  personal injury resulting from such party's negligence to the       *
 *  extent applicable law prohibits such limitation. Some               *
 *  jurisdictions do not allow the exclusion or limitation of           *
 *  incidental or consequential damages, so this exclusion and          *
 *  limitation may not apply to You.                                    *
 *                                                                      *
 ************************************************************************
 8. Litigation
 -------------
 Any litigation relating to this License may be brought only in the
 courts of a jurisdiction where the defendant maintains its principal
 place of business and such litigation shall be governed by laws of that
 jurisdiction, without reference to its conflict-of-law provisions.
 Nothing in this Section shall prevent a party's ability to bring
 cross-claims or counter-claims.
 9. Miscellaneous
 ----------------
 This License represents the complete agreement concerning the subject
 matter hereof. If any provision of this License is held to be
 unenforceable, such provision shall be reformed only to the extent
 necessary to make it enforceable. Any law or regulation which provides
 that the language of a contract shall be construed against the drafter
 shall not be used to construe this License against a Contributor.
 10. Versions of the License
 ---------------------------
 10.1. New Versions
 Mozilla Foundation is the license steward. Except as provided in Section
 10.3, no one other than the license steward has the right to modify or
 publish new versions of this License. Each version will be given a
 distinguishing version number.
 10.2. Effect of New Versions
 You may distribute the Covered Software under the terms of the version
 of the License under which You originally received the Covered Software,
 or under the terms of any subsequent version published by the license
 steward.
 10.3. Modified Versions
 If you create software not governed by this License, and you want to
 create a new license for such software, you may create and use a
 modified version of this License if you rename the license and remove
 any references to the name of the license steward (except to note that
 such modified license differs from this License).
 10.4. Distributing Source Code Form that is Incompatible With Secondary
 Licenses
 If You choose to distribute Source Code Form that is Incompatible With
 Secondary Licenses under the terms of this version of the License, the
 notice described in Exhibit B of this License must be attached.
 Exhibit A - Source Code Form License Notice
 -------------------------------------------
  This Source Code Form is subject to the terms of the Mozilla Public
  License, v. 2.0. If a copy of the MPL was not distributed with this
  file, You can obtain one at http://mozilla.org/MPL/2.0/.
 If it is not possible or desirable to put the notice in a particular
 file, then You may include the notice in a location (such as a LICENSE
 file in a relevant directory) where a recipient would be likely to look
 for such a notice.
 You may add additional accurate notices of copyright ownership.
 Exhibit B - "Incompatible With Secondary Licenses" Notice
 ---------------------------------------------------------
  This Source Code Form is "Incompatible With Secondary Licenses", as
  defined by the Mozilla Public License, v. 2.0.
@@ -0,0 +1,143 @@
 resource "kubernetes_namespace" "argocd" {
  metadata {
    name = "argocd"
    labels = {
      "pod-security.kubernetes.io/enforce"         = "baseline"
      "pod-security.kubernetes.io/enforce-version" = "latest"
    }
  }
 }
 resource "helm_release" "argocd" {
  name       = "argocd"
  repository = local.helm_repos["argo"]
  chart      = "argo-cd"
  version    = "7.9.1"
  namespace  = kubernetes_namespace.argocd.metadata[0].name
  values = [
    yamlencode({
      global = {
        domain = "argocd.${var.cluster_domain}"
      }
      configs = {
        params = {
          "application.instanceLabelKey" = "argocd.argoproj.io/instance"
        }
        rbac = {
          "policy.default" = "role:readonly"
        }
      }
      server = {
        extraArgs = [
          "--insecure"
        ]
        ingress = {
          enabled = true
          hosts = [
            "argocd.${var.cluster_domain}"
          ]
        }
      }
      repoServer = {
        autoscaling = {
          enabled     = true
          minReplicas = 2
        }
      }
      controller = {
        replicas = 2
      }
    })
  ]
  depends_on = [
    kubernetes_namespace.argocd,
    kubernetes_manifest.wildcard_cert
  ]
  lifecycle {
    ignore_changes = [
      values
    ]
  }
 }
 resource "kubernetes_manifest" "argocd_project" {
  manifest = {
    apiVersion = "argoproj.io/v1alpha1"
    kind       = "AppProject"
    metadata = {
      name      = "homelab"
      namespace = kubernetes_namespace.argocd.metadata[0].name
    }
    spec = {
      sourceRepos = [
        "https://forgejo.forge.riotpiao.homelab.com/rock/*"
      ]
      destinations = [
        {
          server    = "https://kubernetes.default.svc"
          namespace = "*"
        }
      ]
      clusterResourceWhitelist = [
        {
          group = "*"
          kind  = "*"
        }
      ]
    }
  }
  field_manager {
    name = "terraform"
  }
  depends_on = [
    helm_release.argocd
  ]
 }
 resource "kubernetes_manifest" "argocd_root_app" {
  manifest = {
    apiVersion = "argoproj.io/v1alpha1"
    kind       = "Application"
    metadata = {
      name      = "homelab-root"
      namespace = kubernetes_namespace.argocd.metadata[0].name
    }
    spec = {
      project = kubernetes_manifest.argocd_project.manifest.metadata.name
      source = {
        repoURL        = "https://forgejo.riotpiao.homelab.com/riotpiao.com/homelab.git"
        targetRevision = "main"
        path           = "k8s/argocd/apps"
        directory = {
          recurse = true
        }
      }
      destination = {
        server    = "https://kubernetes.default.svc"
        namespace = kubernetes_namespace.argocd.metadata[0].name
      }
      syncPolicy = {
        automated = {
          prune    = true
          selfHeal = true
        }
        syncOptions = [
          "CreateNamespace=true"
        ]
      }
    }
  }
  field_manager {
    name = "terraform"
  }
  depends_on = [
    kubernetes_manifest.argocd_project
  ]
 }
@@ -0,0 +1,80 @@
 ---
 # ClusterIssuer: selfsigned-bootstrap (initial issuer for homelab-ca cert)
 apiVersion: cert-manager.io/v1
 kind: ClusterIssuer
 metadata:
  name: selfsigned-bootstrap
 spec:
  selfSigned: {}
 ---
 # ClusterIssuer: homelab-ca (uses generated homelab-ca cert)
 apiVersion: cert-manager.io/v1
 kind: ClusterIssuer
 metadata:
  name: homelab-ca
 spec:
  ca:
    secretName: homelab-ca-secret
 ---
 # Self-signed CA certificate (10 year lifetime)
 apiVersion: cert-manager.io/v1
 kind: Certificate
 metadata:
  name: homelab-ca
  namespace: cert-manager
 spec:
  commonName: homelab-ca
  duration: 87600h
  isCA: true
  issuerRef:
    kind: ClusterIssuer
    name: selfsigned-bootstrap
  privateKey:
    algorithm: ECDSA
    size: 256
  renewBefore: 720h
  secretName: homelab-ca-secret
 ---
 # Wildcard TLS certificate for *.riotpiao.homelab.com
 apiVersion: cert-manager.io/v1
 kind: Certificate
 metadata:
  name: homelab-tls
  namespace: ingress-nginx
 spec:
  commonName: "*.riotpiao.homelab.com"
  dnsNames:
    - "*.riotpiao.homelab.com"
  duration: 2160h
  issuerRef:
    kind: ClusterIssuer
    name: homelab-ca
  renewBefore: 720h
  secretName: homelab-tls
 ---
 # ArgoCD namespace
 apiVersion: v1
 kind: Namespace
 metadata:
  name: argocd
  labels:
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/enforce-version: latest
 ---
 # dev-tools namespace
 apiVersion: v1
 kind: Namespace
 metadata:
  name: dev-tools
  labels:
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: latest
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest
@@ -15,6 +15,8 @@ locals {
    "story-crater-backend"  = {}
    "llm"                   = {}
    "dev-tools"             = {}
    "longhorn-system"       = {}
    "cilium-secrets"        = {}
  }
 }
@@ -0,0 +1,62 @@
 locals {
  bootstrap_releases = {
    cert-manager = {
      chart_version = "v1.21.0"
      namespace     = "cert-manager"
      repo          = "jetstack"
    }
    reloader = {
      chart_version = "1.3.0"
      namespace     = "reloader"
      repo          = "stakater"
    }
    ingress-nginx = {
      chart_version = "4.15.1"
      namespace     = "ingress-nginx"
      repo          = "ingress_nginx"
    }
  }
 }
 resource "helm_release" "bootstrap" {
  for_each = local.bootstrap_releases
  name       = each.key
  repository = local.helm_repos[each.value.repo]
  chart      = each.key
  version    = each.value.chart_version
  namespace  = each.value.namespace
  dynamic "set" {
    for_each = each.key == "cert-manager" ? {
      "crds.enabled"                       = "true"
      "prometheus.enabled"                 = "true"
      "prometheus.servicemonitor.enabled"  = "true"
      "prometheus.servicemonitor.interval" = "60s"
      } : (each.key == "ingress-nginx" ? {
        "controller.admissionWebhooks.enabled"         = "false"
        "controller.dnsPolicy"                         = "ClusterFirstWithHostNet"
        "controller.extraArgs.default-ssl-certificate" = "ingress-nginx/homelab-tls"
        "controller.hostPort.enabled"                  = "true"
        "controller.ingressClassResource.default"      = "true"
        "controller.kind"                              = "DaemonSet"
        "controller.metrics.enabled"                   = "true"
        "controller.metrics.serviceMonitor.enabled"    = "true"
        "controller.metrics.serviceMonitor.interval"   = "30s"
    } : {})
    content {
      name  = set.key
      value = set.value
    }
  }
  depends_on = [
    kubernetes_namespace.namespaces
  ]
  lifecycle {
    ignore_changes = [
      values
    ]
  }
 }
@@ -1,40 +0,0 @@
 resource "kubernetes_storage_class" "longhorn" {
  metadata {
    name = "longhorn"
    annotations = {
      "storageclass.kubernetes.io/is-default-class" = "true"
    }
  }
  provisioner            = "driver.longhorn.io"
  reclaim_policy         = "Delete"
  allow_volume_expansion = true
  parameters = {
    numberOfReplicas    = "2"
    staleReplicaTimeout = "30"
    fromBackup          = ""
    fstype              = "ext4"
  }
  lifecycle {
    prevent_destroy = true
  }
 }
 resource "kubernetes_storage_class" "longhorn_kafka" {
  metadata {
    name = "longhorn-kafka"
  }
  provisioner            = "driver.longhorn.io"
  reclaim_policy         = "Delete"
  allow_volume_expansion = true
  parameters = {
    numberOfReplicas    = "3"
    staleReplicaTimeout = "30"
  }
  lifecycle {
    prevent_destroy = true
  }
 }
@@ -9,6 +9,10 @@ terraform {
      source  = "hashicorp/helm"
      version = "~> 2.14"
    }
    vault = {
      source  = "hashicorp/vault"
      version = "~> 4.0"
    }
    null = {
      source  = "hashicorp/null"
      version = "~> 3.2"
@@ -25,3 +29,8 @@ provider "helm" {
    config_path = var.kubeconfig_path
  }
 }
 provider "vault" {
  address         = "https://vault.riotpiao.homelab.com"
  skip_tls_verify = true
 }
@@ -0,0 +1,81 @@
 #!/bin/bash
 set -e
 echo "=== Step 1: Apply bootstrap manifests (namespaces, cert-manager issuers/certs) ==="
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 source "$SCRIPT_DIR/../../.env.terraform.sh"
 kubectl apply -f "$SCRIPT_DIR/../bootstrap-manifests.yaml" --validate=false
 echo "✓ Bootstrap manifests applied"
 echo ""
 echo "=== Step 2: Wait for cert-manager to be ready ==="
 kubectl -n cert-manager rollout status deployment/cert-manager --timeout=5m
 echo "✓ cert-manager ready"
 echo ""
 echo "=== Step 3: Wait for homelab-ca certificate to be issued ==="
 for i in {1..60}; do
  if kubectl -n cert-manager get secret homelab-ca-secret &>/dev/null; then
    echo "✓ homelab-ca-secret created"
    break
  fi
  echo "Waiting for homelab-ca-secret... ($i/60)"
  sleep 2
 done
 echo ""
 echo "=== Step 4a: Delete old ArgoCD release from cicd namespace ==="
 if helm list -n cicd | grep -q "^argocd"; then
  helm delete argocd -n cicd
  echo "✓ Old ArgoCD removed from cicd"
  sleep 5
 else
  echo "✓ No ArgoCD in cicd to delete"
 fi
 echo ""
 echo "=== Step 4a2: Delete orphaned CRDs left behind by helm resource policy ==="
 kubectl delete crd applications.argoproj.io applicationsets.argoproj.io appprojects.argoproj.io --ignore-not-found
 echo "✓ CRDs deleted"
 sleep 2
 echo ""
 echo "=== Step 4b: Install ArgoCD helm release in argocd namespace ==="
 helm repo add argocd https://argoproj.github.io/argo-helm
 helm repo update
 ARGOCD_VALUES="$SCRIPT_DIR/../../k8s/talos-ci-cd/argocd-values.yaml"
 helm upgrade --install argocd argocd/argo-cd \
  --namespace argocd \
  --version 7.3.3 \
  --values "$ARGOCD_VALUES" \
  --wait
 echo "✓ ArgoCD installed in argocd namespace"
 echo ""
 echo "=== Step 5: Import kubernetes resources into TF state ==="
 cd "$SCRIPT_DIR/../.."
 source .env.terraform.sh
 cd terraform
 # Import namespaces
 terraform import kubernetes_namespace.argocd argocd
 terraform import 'kubernetes_namespace.namespaces["dev-tools"]' dev-tools
 # Import cert-manager objects (kubernetes_manifest import syntax uses colons: apiVersion:kind:name or apiVersion:kind:namespace:name)
 terraform import kubernetes_manifest.selfsigned_bootstrap 'cert-manager.io:v1:ClusterIssuer:selfsigned-bootstrap'
 terraform import kubernetes_manifest.homelab_ca_issuer 'cert-manager.io:v1:ClusterIssuer:homelab-ca'
 terraform import kubernetes_manifest.homelab_ca_cert 'cert-manager.io:v1:Certificate:cert-manager:homelab-ca'
 terraform import kubernetes_manifest.wildcard_cert 'cert-manager.io:v1:Certificate:ingress-nginx:homelab-tls'
 echo "✓ Kubernetes resources imported"
 echo ""
 echo "=== Step 6: Import ArgoCD helm release ==="
 terraform import helm_release.argocd argocd/argocd
 echo "✓ ArgoCD helm release imported"
 echo ""
 echo "=== Step 7: Verify clean plan ==="
 terraform plan
 echo ""
 echo "✓ All bootstrap resources imported. Plan should show zero changes."
@@ -0,0 +1,69 @@
 #!/bin/bash
 set -e
 cd "$(dirname "$0")/.."
 echo "=== Importing Namespaces ==="
 for ns in cert-manager reloader ingress-nginx ddb iam storage logging monitoring cicd dashboard sqs temporal story-crater-backend llm dev-tools cilium-secrets kube-node-lease kube-public; do
  if kubectl get ns "$ns" &>/dev/null; then
    echo "Importing namespace: $ns"
    terraform import "kubernetes_namespace.namespaces[\"$ns\"]" "$ns" || echo "  (already imported or skipped)"
  fi
 done
 echo -e "\n=== Importing Storage Classes ==="
 for sc in longhorn longhorn-kafka longhorn-llm longhorn-static; do
  if kubectl get sc "$sc" &>/dev/null; then
    echo "Importing storage class: $sc"
    terraform import "kubernetes_storage_class.$sc" "$sc" || echo "  (already imported or skipped)"
  fi
 done
 echo -e "\n=== Importing Bootstrap Helm Releases (TF-managed) ==="
 declare -a bootstrap_releases=(
  "cert-manager/cert-manager"
  "reloader/reloader"
  "ingress-nginx/ingress-nginx"
 )
 for rel_ns in "${bootstrap_releases[@]}"; do
  rel=$(echo "$rel_ns" | cut -d/ -f1)
  ns=$(echo "$rel_ns" | cut -d/ -f2)
  if helm list -n "$ns" --output json 2>/dev/null | grep -q "\"name\":\"$rel\""; then
    echo "Importing helm release: $rel_ns"
    terraform import "helm_release.bootstrap[\"$rel\"]" "$rel_ns" || echo "  (already imported or skipped)"
  fi
 done
 echo -e "\n=== Skipping Workload Helm Releases ==="
 echo "The following releases will be managed by ArgoCD (not imported to TF):"
 declare -a workload_releases=(
  "argocd/cicd"
  "authentik/iam"
  "cilium/kube-system"
  "cloudnative-pg/ddb"
  "duckdns/kube-system"
  "forgejo/cicd"
  "forgejo-runner/cicd"
  "grafana/logging"
  "kafka-cluster/sqs"
  "kmsvc-redis/sqs"
  "loki/logging"
  "management-service/sqs"
  "minio/storage"
  "ollama/llm"
  "portainer/dashboard"
  "prometheus/monitoring"
  "promtail/logging"
  "queue-crd/sqs"
  "strimzi-operator/sqs"
  "temporal/temporal"
  "vault/iam"
 )
 for rel in "${workload_releases[@]}"; do
  echo "  - $rel"
 done
 echo -e "\n=== Import Complete ==="
 echo "Next: review terraform plan and apply"
 terraform plan
@@ -0,0 +1,60 @@
 #!/bin/bash
 set -e
 cd "$(dirname "$0")/.."
 echo "=== Terraform Bootstrap Setup ==="
 if [ ! -f "../.env" ]; then
  echo "ERROR: ../.env not found. Run from terraform/ directory."
  exit 1
 fi
 # Extract MinIO credentials from .env
 export AWS_ACCESS_KEY_ID=$(grep "^MINIO_ROOT_USER=" ../.env | cut -d= -f2)
 export AWS_SECRET_ACCESS_KEY=$(grep "^MINIO_ROOT_PASSWORD=" ../.env | cut -d= -f2)
 export KUBECONFIG=$(pwd)/../cluster-config/kubeconfig
 if [ -z "$AWS_ACCESS_KEY_ID" ] || [ -z "$AWS_SECRET_ACCESS_KEY" ]; then
  echo "ERROR: MINIO_ROOT_USER or MINIO_ROOT_PASSWORD not found in .env"
  exit 1
 fi
 echo "✓ MinIO S3 credentials loaded"
 # Check Vault token
 if [ -z "$VAULT_TOKEN" ]; then
  echo "WARNING: VAULT_TOKEN not set. Set it before terraform init:"
  echo "  export VAULT_TOKEN=\$(vault login -method=oidc ...)"
 fi
 # Generate terraform.tfvars from .env
 echo "Generating terraform.tfvars from .env..."
 cat > terraform.tfvars.local << 'EOF'
 # Auto-generated from .env
 EOF
 grep -E '^(AUTHENTIK_SECRET_KEY|AUTHENTIK_BOOTSTRAP_PASSWORD|AUTHENTIK_BOOTSTRAP_TOKEN|AUTHENTIK_PG_PASSWORD|POSTGRES_PASSWORD|MINIO_ROOT_|GRAFANA_|FORGEJO_|ARGOCD_)' ../.env | sed 's/_ROOT_USER=/=/' | while read line; do
  key=$(echo "$line" | cut -d= -f1 | sed 's/_/ /g; s/.*/\L&/; s/ /_/g')
  val=$(echo "$line" | cut -d= -f2-)
  if [[ "$key" == *"secret"* ]] || [[ "$key" == *"password"* ]]; then
    echo "$key = \"$val\"" >> terraform.tfvars.local
  else
    echo "$key = \"$val\"" >> terraform.tfvars.local
  fi
 done
 echo "✓ terraform.tfvars.local generated"
 # Initialize terraform
 echo "Initializing terraform with S3 backend..."
 terraform init \
  -backend-config="access_key=$AWS_ACCESS_KEY_ID" \
  -backend-config="secret_key=$AWS_SECRET_ACCESS_KEY"
 echo "✓ Terraform initialized"
 echo ""
 echo "Next steps:"
 echo "  1. Review: terraform plan"
 echo "  2. Import existing resources: ./scripts/import-existing.sh"
 echo "  3. Apply: terraform apply"
@@ -1,17 +1,20 @@
 # Temporarily disabled S3 (MinIO unreachable)
 # terraform {
 #   backend "s3" {
 #     bucket                      = "terraform-state"
 #     key                         = "homelab/terraform.tfstate"
 #     region                      = "us-east-1"
 #     endpoint                    = "https://minio-api.riotpiao.homelab.com"
 #     skip_credentials_validation = true
 #     skip_requesting_account_id  = true
 #     skip_region_validation      = true
 #     use_path_style              = true
 #   }
 # }
 # Local backend fallback (active during MinIO outage)
 terraform {
  backend "local" {
    path = "terraform.tfstate"
  }
 }
 # TODO: Configure remote state (S3, GCS, or Terraform Cloud)
 # Uncomment when ready:
 # terraform {
 #   backend "s3" {
 #     bucket         = "homelab-terraform-state"
 #     key            = "prod/terraform.tfstate"
 #     region         = "us-east-1"
 #     encrypt        = true
 #     dynamodb_table = "terraform-lock"
 #   }
 # }
@@ -0,0 +1,21 @@
 kubeconfig_path = "/Users/rockliang/workplace/homelab/cluster-config/kubeconfig"
 cluster_domain  = "riotpiao.homelab.com"
 authentik_secret_key = "v9TTdMxpP9XtrwH2HFUjHC8MKwfeW+fYOpa5RTyP6EniqFclFSDXWbRp6crhrkLJkFeCfIcAMN/JODyy"
 authentik_bootstrap_password = "8+7MFMCtAHiOxSaiBJwDiR85nnrhLyX2"
 authentik_bootstrap_token = "5a534cb785aecddac23647e11ff4d824b50b03f70c173faee7ba46f12db55dc3"
 authentik_pg_password = "vueM/7N6bUR/j/hUUPsWTM2pRm8lCewq"
 postgres_password = "a071b1f7b721a216b57d396b8d906fa10f6db1597d68a0f2e9f95e2872e7cb0f"
 minio_root_user = "minioadmin"
 minio_root_password = "nhKRAxwIjDBCzwFDvsAa7dNLouCXXh13LvMoxMVkUtY="
 minio_oidc_client_secret = "9d2867fe08c3bf7fedd7e32bbaf4456fce3b0aaf788966d7559e1955947b0219"
 grafana_admin_password = "your-secure-password"
 grafana_oidc_client_secret = "966bad4fa43812100e7775b3c73fed2ce1d07217fa5a23fbb0f190e46d2f0fa4"
 forgejo_admin_password = "stRe4cawnQH/agM0QsPaWPdKNaQ0rp4p"
 authentik_forgejo_client_secret = "e417c1a3b3ee79b44c9d8d3490a735aae7b2c19a81afe1ba6a262775d5ae9edc"
 authentik_argocd_client_id = "argocd"
 authentik_argocd_client_secret = "acc51c1ab043530b84b17dc5e8128b2a656b17558ea3f93e723618ea5c6d9774"
 authentik_temporal_client_id = "temporal"
 authentik_temporal_client_secret = "6UAuC651l21fajBZQwjV+bbkD1k6uCoV6PnQxaFlPaQ="
 argocd_admin_password = "placeholder"
 argocd_oidc_client_secret = "placeholder"
@@ -1,7 +1,20 @@
 kubeconfig_path = "cluster-config/kubeconfig"
 cluster_domain  = "riotpiao.homelab.com"
-# Load these from .env or Vault in production
+# Backend S3 credentials (MinIO)
 # Set via environment variables instead:
 #   export AWS_ACCESS_KEY_ID=$(grep MINIO_ROOT_USER .env | cut -d= -f2)
 #   export AWS_SECRET_ACCESS_KEY=$(grep MINIO_ROOT_PASSWORD .env | cut -d= -f2)
 s3_access_key = ""
 s3_secret_key = ""
 # Vault token for terraform vault provider
 # Set via environment variable:
 #   export VAULT_TOKEN=$(vault login -method=oidc ...)
 vault_token = ""
 # Cluster secrets loaded from .env (extract via 'vsource' alias)
 # Or populate these manually from Vault/secrets manager
 authentik_secret_key              = "changeme-min-32-characters-long-value"
 authentik_bootstrap_password      = "changeme"
 authentik_bootstrap_token         = "changeme"
@@ -10,6 +10,27 @@ variable "cluster_domain" {
  default     = "riotpiao.homelab.com"
 }
 variable "vault_token" {
  description = "Vault token (set via VAULT_TOKEN env var or tfvars)"
  type        = string
  sensitive   = true
  default     = ""
 }
 variable "s3_access_key" {
  description = "MinIO S3 access key for terraform state backend"
  type        = string
  sensitive   = true
  default     = ""
 }
 variable "s3_secret_key" {
  description = "MinIO S3 secret key for terraform state backend"
  type        = string
  sensitive   = true
  default     = ""
 }
 # Cluster secrets (load from .env.tfvars or vault)
 variable "authentik_secret_key" {
  description = "Authentik secret key"