feat:Fix the bootstrap to be deploy key application

2026-07-23 19:07:39 -07:00
parent eba9f2144c
commit 1c7395d9e1
37 changed files with 1216 additions and 1202 deletions
@@ -50,4 +50,6 @@ terraform/*.tfstate.*
 terraform.tfvars.local
 skills-lock.json
 secrets-plaintext.yaml
-skills-lock.json
+skills-lock.json
 CLAUDE.md
@@ -1,251 +0,0 @@
 # CLAUDE.md — Homelab Project Reference
 ## Cluster Topology (3 control-plane HA)
 | Node | IP | Zone | Scheduling | Storage |
 |------|----|----|-----------|---------|
 | `talos-cp-1` | .213 | az-a | schedulable (all workloads) | Longhorn replica |
 | `talos-cp-2` | .163 | az-b | dedicated (`NoSchedule`) | Longhorn replica |
 | `talos-cp-3` | .166 | az-c | dedicated (`NoSchedule`) | Longhorn replica |
 3 voting etcd members peering on the LAN. All 3 nodes run Longhorn with 3-replica
 HA storage (verified: all 17 PVCs have replicas across all nodes). Only `talos-cp-1`
 is schedulable for workloads (cp-2/cp-3 are control-plane only). Full detail + gotchas
 in `USAGE.md` and memory `reference_talos_etcd_and_ca_gotchas`.
 ## Deployment Model: ArgoCD GitOps (helmfile is retired)
 **helmfile.yaml.gotmpl and the `core` CLI's helmfile-era workflows described in
 `USAGE.md`/`project-usage/*.md` are STALE.** Actual practice, 100% of the time:
 ```
 git commit → git push (Forgejo) → ArgoCD auto-sync → cluster
 ```
 App-of-apps structure: `k8s/argocd/root/homelab-root.yaml` (root Application)
 → `k8s/argocd/apps/*.yaml` (one file per "wave" of Applications, each
 `argocd.argoproj.io/sync-wave` annotated) → each Application points at either
 a remote Helm chart (+ a second `ref: values` git source for the values file)
 or a plain git directory of raw manifests.
 **Never `kubectl apply`/`patch`/`delete` a resource ArgoCD manages** except:
 - Pure cleanup of stuck/dead state (delete a failed hook Job so the *next*
  legitimate sync creates a fresh one — not a config change, just clearing
  wreckage). Confirm with the user first if in doubt.
 - One-time bootstrap actions with a genuine circular dependency (Vault
  `operator init`/unseal — see `k8s/security/iam/VAULT-BOOTSTRAP-README.md`
  equivalent scripts).
 ## Hard Rules (Never Violate)
 🔴 **NEVER rename or wipe `talos-cp-1` (.213).** It is the sole Longhorn storage
 node — all replicas are pinned to that node name. Renaming orphans its Longhorn
 node CR and faults every volume (permanent data loss). Rename/reprovision only
 the dedicated CPs (.163/.166), never the data node.
 🔴 **Control-plane etcd must advertise on the LAN.** Keep
 `cluster.etcd.advertisedSubnets: ["192.168.1.0/24"]` in the controlplane
 template — without it Talos advertises on the WireGuard IP and new members hang
 as non-promoting etcd learners.
 🔴 **ALWAYS run `terraform fmt` after any terraform code changes.** Before commit:
 ```bash
 terraform fmt -recursive terraform/
 ```
 Verify no changes (clean output = formatted correctly). If files change, review diffs, commit fmt changes separately. Workflow's "Terraform Format Check" step will fail otherwise.
 🔴 **NEVER manually kubectl delete/patch resources managed by Terraform.** Terraform is the source of truth for IaC-managed resources (deployments, PVCs, services in Terraform-controlled namespaces). Manual edits create state drift. If a resource is stuck:
 1. Update Terraform config (variables.tf, *.tf files)
 2. Run `terraform apply` (locally or via CI/CD)
 3. Never bypass with manual kubectl operations
 🔴 **NEVER delete a PVC unless there are replicas or backups.** A PVC deletion = permanent data loss. Verify replication status first.
 🔴 **NO co-authored commit messages.** All commits are solo work. Never append `Co-Authored-By:` footer.
 🔴 **Commit message format:** First line must capture what is added, what is fixed, what is removed, and why in one sentence. Example: `fix(kubernetes): use direct in-cluster auth in providers — removes kubeconfig file dependency in CI runner`. No multi-paragraph messages.
 🔴 **Git workflow: rebase only, no pull/merge.** Always rebase when pulling. Use `git pull --rebase` or `git rebase main` before pushing. Keep history linear.
 🔴 **Long-running commands (>10s) must run async.** Use `run_in_background: true` for Bash or spawn Agent. Don't actively wait. Prevents blocking on terraform plan, kubectl apply, downloads.
 🔴 **Infrastructure changes should flow through GitOps when possible:** git commit → push → CI/CD runner (terraform apply) → ArgoCD sync. Local `terraform apply` is permitted (e.g. for local iteration, config regeneration, or when CI/CD isn't wired up for a given module) — still commit + push the resulting state/config afterward so git remains the record of truth. Manual `kubectl apply` remains disallowed for Terraform-managed resources.
 ## CloudNativePG (CNPG) Database Pattern
 **Simple ownership model — all apps use shared 'app' user:**
 ```yaml
 # CNPG Cluster (bootstrap, wave 0)
 bootstrap:
  initdb:
    database: app      # Bootstrap database
    owner: app         # Bootstrap user (owns all databases)
 # Database CR (per-app, wave 6)
 spec:
  name: authentik      # Database name
  owner: app           # ← All apps use 'app' (not per-app roles)
  cluster:
    name: ddb-cluster
 ```
 **Credential Distribution (bootstrap.sh pattern):**
 1. **Source of truth:** CNPG creates `ddb-cluster-app` secret in `ddb` namespace
 2. **Distribution:** `bootstrap.sh` copies secret to app namespaces:
   ```bash
   # For each app namespace (cicd, iam, etc):
   kubectl get secret ddb-cluster-app -n ddb -o yaml \
     | sed 's/namespace: ddb/namespace: <app-namespace>/' \
     | kubectl apply -f -
   ```
 3. **Apps reference local copy:**
   ```yaml
   env:
     - name: DB_USER
       valueFrom:
         secretKeyRef:
           name: ddb-cluster-app  # Local copy in app's namespace
           key: username          # Always "app"
     - name: DB_PASSWORD
       valueFrom:
         secretKeyRef:
           name: ddb-cluster-app
           key: password
   ```
 **Key Points:**
 ✅ **DO:**
 - All apps connect as `app` user
 - Database CRs specify `owner: app`
 - Isolation via separate database names (not roles)
 - Copy credentials to app namespace via `bootstrap.sh`
 - Reference local secret copy via `secretKeyRef`
 ❌ **DON'T:**
 - Create per-app roles in `managed.roles` (CNPG doesn't transfer ownership properly)
 - Grant permissions via PostSync Jobs (owner already has full rights)
 - Use cross-namespace `secretKeyRef` (not supported)
 - Manually patch secrets (ArgoCD will revert)
 **Working Examples:**
 - Forgejo: `k8s/bootstrap-local/04-forgejo.yaml` (references `ddb-cluster-app` in cicd namespace)
 - Authentik: `k8s/security/iam/authentik-values.yaml` (references `ddb-cluster-app` in iam namespace)
 **For New Apps:**
 1. Add Database CR to `k8s/data/schemas/` with `owner: app`
 2. Add secret copy to `bootstrap.sh` (like cicd/iam examples)
 3. Reference `ddb-cluster-app` via `secretKeyRef` in app's namespace
 4. No permission grants needed — app user owns the database
 ## GitOps / ArgoCD Gotchas (hard-won, all confirmed live in this cluster)
 🟠 **`kustomization.yaml` with an explicit `resources:` allowlist silently
 drops anything not listed — no error, no drift shown.** ArgoCD reports
 `Synced/Healthy` against a manifest set that never included the missing
 file at all. Symptom: you commit+push a new manifest, ArgoCD says
 "Synced", but the resource never appears in-cluster. Fix: check every
 `kustomization.yaml` along the app's source path actually lists your new
 file. `kubectl kustomize <dir>/` locally reproduces exactly what ArgoCD
 will apply — always verify with it before pushing.
 🟠 **A `kustomization.yaml` top-level `namespace:` transformer rewrites
 `metadata.namespace` on *every* resource it builds — including RBAC
 RoleBindings deliberately targeting a *different* namespace.** If any
 manifest in that directory needs cross-namespace resources (e.g. a
 RoleBinding granting access to Secrets in another namespace), either drop
 the transformer (safe if every resource already sets its own explicit
 namespace) or move that manifest to its own directory/Application.
 🟠 **PreSync hooks run *before* an Application's own normal (non-hook)
 resources are synced.** A PreSync-hooked Job that depends on a
 ServiceAccount/RBAC defined as plain resources in the *same* Application
 deadlocks: the Job tries to start before its own ServiceAccount exists.
 Confirmed live — Job sat "Running" for 14+ minutes producing zero pods,
 `job-controller` event log showed `serviceaccount ... not found` on every
 retry. Fix: use PostSync instead (runs after that app's own resources are
 applied), or split the hook into its own earlier-sync-wave Application.
 🟠 **ArgoCD hooks (PreSync/PostSync) are NOT continuously reconciled by
 `selfHeal` the way normal resources are.** Once a hook Job completes
 (success or exhausts `backoffLimit` into Failed), it only gets
 deleted+recreated (per `hook-delete-policy: BeforeHookCreation`) during an
 **actual new Sync operation** — not from passive drift detection, even
 with `automated.selfHeal: true`. If you fix a hook Job's spec (image,
 command, RBAC) and push, `status.sync.revision` may show "caught up" while
 the *live* hook resource is still running the old, broken spec — because
 no new operation actually re-ran it. To force a real resync: delete the
 stuck Job (clear the `argocd.argoproj.io/hook-finalizer` if it's stuck
 `Terminating`), and if that alone doesn't trigger a fresh full sync, delete
 + `kubectl apply -f` the Application object itself (re-reads current git
 HEAD, starts a genuinely new operation, no cascade-delete of underlying
 resources since Applications don't carry a cascade finalizer by default —
 confirm with `kubectl get app <name> -o jsonpath='{.metadata.finalizers}'`
 first).
 🟠 **ArgoCD's repo-server caches rendered manifests (~120s TTL by default,
 `argocd-cm` → `timeout.reconciliation`).** If you change a values file and
 `Application` fields both, sometimes the old rendering wins on the next
 sync. Restart `argocd-repo-server` after big multi-source/values changes if
 sync behavior looks stale.
 🟠 **ArgoCD `repoURL` pointing at a hostname that CoreDNS rewrites to the
 nginx ingress controller (for TLS termination) breaks if the URL includes
 a non-standard port.** nginx only listens on 80/443 — `http://host:3000/...`
 silently times out (`context deadline exceeded`) if `host` resolves to the
 ingress controller, not the actual backend service. This blocked **every
 single Application's sync** cluster-wide simultaneously (all showed
 `Unknown` sync status) because the repo-server couldn't fetch git refs at
 all. Use `https://host/...` (no port) so nginx's default TLS cert + normal
 443 routing handles it.
 🟠 **Bitnami Docker Hub images no longer publish versioned tags (2025
 policy change) — only `latest` and sha256-pinned digests remain for their
 free tier.** A pinned tag like `bitnami/kubectl:1.30` will 404/ImagePullBackOff
 forever. Verify tags exist first: `curl -s "https://hub.docker.com/v2/repositories/<org>/<image>/tags?page_size=25" | jq -r '.results[].name'`.
 Prefer avoiding third-party utility images entirely where possible — e.g.
 `python:3.12-alpine` + stdlib `urllib.request` to fetch a static binary
 (kubectl) avoids depending on any registry's tagging policy at all.
 🟠 **Non-root containers (`runAsNonRoot: true`, non-zero UID) can't `apk
 add` in Alpine-based images** — apk's working directories and most of
 `/usr/local/bin` are root-owned. Symptom: `ERROR: Unable to open log:
 Permission denied`. Use `/tmp` (world-writable) for any binary you need to
 download/install at runtime, and extend `PATH` rather than writing to
 `/usr/local/bin`.
 🟠 **Helm does not validate unknown `values.yaml` keys — a typo'd or
 wrong-schema key is silently a no-op, not an error.** Confirmed root cause
 of a multi-week "Temporal doesn't support PostgreSQL" belief: the actual
 chart version pinned (`temporalio/helm-charts@0.74.0`) uses a flat
 `server.config.persistence.<store>.driver/.sql` schema, but the values file
 used the *newer* chart's `datastores:`-wrapped schema (introduced in a
 later major version) — silently ignored, so persistence stayed on the
 chart's Cassandra default the entire time. **Before assuming "this chart
 doesn't support X," clone the chart at the exact pinned tag/version and run
 `helm template` with your real values — diff the rendered output, don't
 trust values.yaml comments/examples from the chart's current `main`
 branch, which may not match your pinned version's schema at all.**
 ## Workflow
 - **Modify** → **Format** (`terraform fmt`) → **Apply** (locally or let CI/CD do it) → **Commit** → **Push**
 - If fmt check fails in CI, fix locally, commit fmt changes, push again
 - For k8s/ManifestsChanges: **Modify** → **validate** (`kubectl apply --dry-run=client -f`, or `kubectl kustomize <dir>/` if a `kustomization.yaml` is involved, or `helm template` against the exact pinned chart version for Helm-sourced Applications) → **Commit** → **Push** → confirm ArgoCD picked it up (check `status.sync.revision` matches your commit, not just `status.sync.status`)
 ## Documentation Map
 - `CLAUDE.md` (this file) — private, cluster-specific, always current
 - `CLAUDE.example.md` — sanitized, hardware-generic template for reuse on other 3-node Talos clusters; update alongside this file when a lesson is genuinely hardware/topology-generic (not homelab-specific secrets/IPs)
 - `USAGE.md` / `project-usage/*.md` — **STALE, helmfile-era.** Written for a
  deprecated `helmfile apply` + `core iam`/`core secrets` CLI workflow that
  does not reflect actual current practice (100% ArgoCD GitOps as of this
  writing). Treat as historical reference only until rewritten; do not
  follow their deployment procedures literally.
 - `TROUBLESHOOTING.md` — generic Kubernetes SRE layer-before-tool methodology, still broadly applicable regardless of deployment mechanism
 ---
 **Last updated:** 2026-07-23
@@ -1,142 +0,0 @@
 #!/usr/bin/env bash
 #
 # Homelab Cluster Bootstrap (Phase 0)
 # Bootstrap a fresh Talos cluster to GitOps-ready state (ArgoCD + Forgejo).
 # Run once from local checkout, then all future changes via git push.
 #
 # Prerequisites:
 #   - Talos cluster up (terraform apply completed)
 #   - kubectl configured (KUBECONFIG points at cluster)
 #   - SOPS age key at ~/.sops/homelab-age.key
 #   - ArgoCD CLI installed (for final sync)
 #
 set -euo pipefail
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 K8S_DIR="$SCRIPT_DIR/k8s"
 BOOTSTRAP_DIR="$K8S_DIR/bootstrap-local"
 SOPS_KEY="${SOPS_KEY:-$HOME/.sops/homelab-age.key}"
 # Colors
 RED='\033[0;31m'
 GREEN='\033[0;32m'
 YELLOW='\033[1;33m'
 NC='\033[0m' # No Color
 log() { echo -e "${GREEN}[$(date +'%H:%M:%S')]${NC} $*"; }
 warn() { echo -e "${YELLOW}[$(date +'%H:%M:%S')]${NC} $*"; }
 error() { echo -e "${RED}[$(date +'%H:%M:%S')]${NC} $*"; exit 1; }
 # Preflight checks
 log "Running preflight checks..."
 kubectl cluster-info > /dev/null || error "kubectl not configured or cluster unreachable"
 [[ -f "$SOPS_KEY" ]] || error "SOPS age key not found at $SOPS_KEY"
 command -v argocd > /dev/null || warn "ArgoCD CLI not found - manual sync required at end"
 # 1. Install ArgoCD itself (if not already present)
 if ! kubectl get namespace argocd &>/dev/null; then
    log "Installing ArgoCD..."
    kubectl create namespace argocd
    kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
    log "Waiting for ArgoCD to be ready..."
    kubectl wait --for=condition=available --timeout=300s deployment/argocd-server -n argocd
 else
    log "ArgoCD already installed, skipping..."
 fi
 # 2. Create SOPS age secret (NEVER commit this to git)
 log "Creating SOPS age secret..."
 kubectl create namespace argocd --dry-run=client -o yaml | kubectl apply -f -
 kubectl create secret generic sops-age \
  -n argocd \
  --from-file=keys.txt="$SOPS_KEY" \
  --dry-run=client -o yaml | kubectl apply -f -
 # 3. Apply bootstrap bundle (namespaces, CNPG, DDB, Forgejo)
 log "Applying bootstrap bundle..."
 kubectl apply -k "$BOOTSTRAP_DIR" --server-side
 # 4. Wait for CNPG operator
 log "Waiting for CNPG operator..."
 kubectl wait --for=condition=available --timeout=300s \
  deployment/cnpg-controller-manager -n ddb 2>/dev/null || {
    warn "CNPG operator not found - checking if it exists as different deployment name..."
    kubectl get deployments -n ddb
 }
 # 5. Wait for DDB cluster
 log "Waiting for PostgreSQL cluster (ddb-cluster) to be ready..."
 for i in {1..60}; do
    STATUS=$(kubectl get cluster ddb-cluster -n ddb -o jsonpath='{.status.phase}' 2>/dev/null || echo "NotFound")
    if [[ "$STATUS" == "Cluster in healthy state" ]]; then
        log "DDB cluster is ready!"
        break
    fi
    [[ $i -eq 60 ]] && error "Timeout waiting for ddb-cluster"
    sleep 5
 done
 # 6. Copy DB secret from ddb to cicd namespace
 log "Copying ddb-cluster-app secret to cicd namespace..."
 kubectl get secret ddb-cluster-app -n ddb -o yaml \
  | sed 's/namespace: ddb/namespace: cicd/' \
  | kubectl apply -f -
 # Copy DB secret to iam namespace (for authentik)
 log "Copying ddb-cluster-app secret to iam namespace..."
 kubectl get secret ddb-cluster-app -n ddb -o yaml \
  | sed 's/namespace: ddb/namespace: iam/' \
  | kubectl apply -f -
 # Copy DB secret to temporal namespace (for temporal)
 log "Copying ddb-cluster-app secret to temporal namespace..."
 kubectl get secret ddb-cluster-app -n ddb -o yaml \
  | sed 's/namespace: ddb/namespace: temporal/' \
  | kubectl apply -f -
 # 7. Wait for Forgejo
 log "Waiting for Forgejo to be ready..."
 kubectl wait --for=condition=available --timeout=600s \
  deployment/forgejo -n cicd 2>/dev/null || {
    warn "Forgejo not found as deployment - checking StatefulSet..."
    kubectl wait --for=condition=available --timeout=600s \
      statefulset/forgejo -n cicd || warn "Could not find Forgejo - check manually"
 }
 # 8. Apply Longhorn 3-node configuration (if it exists)
 if [[ -d "$K8S_DIR/infrastructure/longhorn" ]]; then
    log "Applying Longhorn 3-node HA configuration..."
    kubectl apply -k "$K8S_DIR/infrastructure/longhorn/" || warn "Longhorn config failed - may need manual intervention"
 else
    warn "Longhorn config not found at k8s/infrastructure/longhorn/ - storage may be single-node only!"
 fi
 # 9. Get Forgejo LoadBalancer IP
 FORGEJO_IP=$(kubectl get svc forgejo-http -n cicd -o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>/dev/null || echo "unknown")
 log "Forgejo available at: http://$FORGEJO_IP:3000 (or https://forgejo.riotpiao.com)"
 echo ""
 log "${GREEN}========================================${NC}"
 log "${GREEN}✅ Bootstrap Complete!${NC}"
 log "${GREEN}========================================${NC}"
 echo ""
 echo "Next steps:"
 echo ""
 echo "  1. Push this repo to Forgejo:"
 echo "     git remote add forgejo https://forgejo.riotpiao.com/riotpiao.com/homelab.git"
 echo "     git push forgejo main"
 echo ""
 echo "  2. Apply app-of-apps root:"
 echo "     kubectl apply -k k8s/argocd/root"
 echo ""
 echo "  3. VERIFY STORAGE REPLICATION (CRITICAL!):"
 echo "     See STORAGE-ARCHITECTURE-CLARIFICATION.md"
 echo "     kubectl get nodes.longhorn.io -n longhorn-system"
 echo "     kubectl get volumes.longhorn.io -n longhorn-system -o wide"
 echo ""
 echo "  4. Sync all applications:"
 echo "     argocd app sync homelab-root"
 echo "     # Or via UI: https://argocd.riotpiao.com"
 echo ""
 echo "  5. All future changes: git commit → git push (ArgoCD auto-syncs)"
 echo ""
@@ -1,140 +0,0 @@
 # k8s/temporal/temporal-values.yaml
 # Temporal — workflow engine
 # Uses external CNPG PostgreSQL for persistence (ddb-cluster)
 # Visibility via same PostgreSQL instance, separate database.
 #
 # IMPORTANT — chart schema note (root-caused after Postgres never actually
 # taking effect despite looking configured):
 # We're pinned to temporalio/helm-charts @ 0.74.0 (see targetRevision in
 # k8s/argocd/apps/60-applications.yaml), which uses the OLD flat persistence
 # schema:
 #   server.config.persistence.<default|visibility>.driver: "sql"|"cassandra"
 #   server.config.persistence.<default|visibility>.sql: {...}
 # NOT the newer `datastores:`-wrapped schema
 # (server.config.persistence.datastores.<store>.sql) shown in the current
 # chart's values/values.postgresql.yaml example - that key was introduced in
 # a later major version and doesn't exist in 0.74.0. Helm doesn't validate
 # unknown keys, so a `datastores:` block here is silently a no-op: Temporal
 # would keep defaulting to Cassandra (with empty hosts: []) regardless of
 # anything nested inside it. Verified via `helm template` against the actual
 # 0.74.0 chart before writing this file - see chat history for the
 # side-by-side proof (rendered manifest showed CASSANDRA_HOST env vars and
 # temporal-cassandra-tool commands using the old datastores:-based values).
 #
 # Likewise `schema.setup.enabled` / `schema.update.enabled` /
 # `schema.createDatabase.enabled` are the real toggles for the schema-setup
 # Job (all default true) - there is no `jobs.autoSetup` key in this chart.
 # ── Disable every bundled/optional sub-chart ─────────────────────────────────
 # postgresql/mysql: never enable - we never want the chart to deploy its own
 # DB, only to know how to talk to our external CNPG instance (which happens
 # via server.config.persistence.*.sql below, independent of these flags).
 postgresql:
  enabled: false
 mysql:
  enabled: false
 cassandra:
  enabled: false
 elasticsearch:
  enabled: false
 prometheus:
  enabled: false
 grafana:
  enabled: false
 # ── Schema setup/update Jobs ──────────────────────────────────────────────────
 # The `temporal` and `temporal_visibility` databases are provisioned
 # declaratively by CNPG Database CRs (k8s/data/temporal-database.yaml,
 # temporal-visibility-database.yaml), so createDatabase stays disabled (the
 # `temporal` role also lacks CREATEDB). setup/update run temporal-sql-tool as
 # the `temporal` owner against those existing DBs to install and migrate the
 # Temporal server schema — without them both DBs have zero tables and the
 # server dies on "no usable database connection found" (no schema_version row).
 schema:
  createDatabase:
    enabled: false
  setup:
    enabled: true
  update:
    enabled: true
 # ── Temporal server config (PostgreSQL persistence) ──────────────────────────
 server:
  replicaCount: 1
  # temporalio/server:1.30.0+ dropped the `dockerize` binary and switched to
  # built-in sprig config templating. The chart still defaults to the legacy
  # configMapsToMount: "dockerize" + setConfigFilePath: false, which produces a
  # config the 1.30 server never loads — it then falls back to its embedded
  # env-only template (Cassandra default) and dies with
  # "Persistence.DataStores[default](value).Cassandra.Hosts: zero value".
  # Switch to the sprig ConfigMap and point the server at it (chart's own
  # recommendation for 1.30.0+ images; sprig mode requires setConfigFilePath).
  configMapsToMount: "sprig"
  setConfigFilePath: true
  jobService:
    enabled: false
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchLabels:
                app.kubernetes.io/instance: temporal
            topologyKey: kubernetes.io/hostname
  config:
    logLevel: "info"
    persistence:
      defaultStore: default
      visibilityStore: visibility
      numHistoryShards: 512
      default:
        driver: "sql"
        sql:
          driver: "postgres12"
          host: "ddb-cluster-rw.ddb.svc.cluster.local"
          port: 5432
          database: "temporal"
          user: "temporal"
          # existingSecret + secretKey: point directly at the CNPG-generated
          # Secret (kubernetes.io/basic-auth, keys: username/password/...)
          # rather than duplicating the password in git as plaintext. When
          # existingSecret is set the chart's own server-secret.yaml Secret
          # template is skipped entirely (see templates/server-secret.yaml:
          # `not $driverConfig.existingSecret` guards its creation).
          existingSecret: "temporal-db-role"
          secretKey: "password"
          maxConns: 20
          maxIdleConns: 10
          maxConnLifetime: "1h"
          # NOTE: no `connectAttributes: { tx_isolation: ... }` here — tx_isolation
          # is a MySQL-only connection parameter. The Postgres `pq` driver rejects
          # it ("unrecognized configuration parameter"), which killed every DB
          # connection (schema-setup job AND server) with the misleading
          # "no usable database connection found". Postgres defaults to READ
          # COMMITTED isolation anyway, so nothing is lost by omitting it.
      visibility:
        driver: "sql"
        sql:
          driver: "postgres12"
          host: "ddb-cluster-rw.ddb.svc.cluster.local"
          port: 5432
          database: "temporal_visibility"
          user: "temporal"
          existingSecret: "temporal-db-role"
          secretKey: "password"
          maxConns: 20
          maxIdleConns: 10
          maxConnLifetime: "1h"
  service:
    type: ClusterIP
 # ── Temporal Web UI ────────────────────────────────────────────────────────
 web:
  replicaCount: 1
  service:
    type: ClusterIP
 # ── Ingress ────────────────────────────────────────────────────────
 ingress:
  enabled: false
@@ -1,119 +0,0 @@
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: argocd
  labels:
    name: argocd
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: ddb
  labels:
    name: ddb
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: cicd
  labels:
    name: cicd
    # REQUIRED: Forgejo runner needs privileged (DinD, hostPath, securityContext.privileged)
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/warn: privileged
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: cert-manager
  labels:
    name: cert-manager
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: ingress-nginx
  labels:
    name: ingress-nginx
    # REQUIRED: nginx controller needs hostPort 80/443
    pod-security.kubernetes.io/enforce: privileged
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: reloader
  labels:
    name: reloader
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: storage
  labels:
    name: storage
    # REQUIRED: minio operator needs privileged securityContext
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/warn: privileged
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: monitoring
  labels:
    name: monitoring
    # REQUIRED: node-exporter needs hostNetwork/hostPID/hostPath/hostPort
    pod-security.kubernetes.io/enforce: privileged
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: logging
  labels:
    name: logging
    # REQUIRED: promtail needs hostPath, DAC_READ_SEARCH, privileged:true
    pod-security.kubernetes.io/enforce: privileged
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: iam
  labels:
    name: iam
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: sqs
  labels:
    name: sqs
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: temporal
  labels:
    name: temporal
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: dashboard
  labels:
    name: dashboard
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: cloudflared
  labels:
    name: cloudflared
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: duckdns
  labels:
    name: duckdns
@@ -1,47 +0,0 @@
 # ArgoCD installation - NOT managed by ArgoCD itself (bootstrap only).
 # Install via: kubectl apply -k k8s/bootstrap-local/
 # Or manually: kubectl create namespace argocd
 #              kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
 ---
 apiVersion: v1
 kind: ConfigMap
 metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
 data:
  # Point at Forgejo (will be available after 04-forgejo.yaml completes)
  repositories: |
    - url: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
      name: homelab
      type: git
  # Reconciliation timeout (default 180s)
  timeout.reconciliation: "300"
  # Resource exclusions (prevent ArgoCD from managing certain resources)
  resource.exclusions: |
    - apiGroups:
      - cilium.io
      kinds:
      - CiliumIdentity
      clusters:
      - "*"
 ---
 apiVersion: v1
 kind: ConfigMap
 metadata:
  name: argocd-rbac-cm
  namespace: argocd
 data:
  # Admin policy (adjust as needed)
  policy.default: role:readonly
  policy.csv: |
    g, admin, role:admin
 ---
 # NOTE: ArgoCD installation itself not included here - apply it separately:
 # kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
 # Or use Helm chart (recommended for production):
 # helm install argocd argo/argo-cd -n argocd --version 7.x.x
@@ -1,30 +0,0 @@
 # CloudNativePG operator - deployed as Helm chart via kubectl/ArgoCD.
 # This file is a placeholder - actual install via Helm:
 #   helm repo add cnpg https://cloudnative-pg.github.io/charts
 #   helm install cnpg cnpg/cloudnative-pg -n ddb --create-namespace --version ~0.20
 #
 # Or create an ArgoCD Application (recommended):
 ---
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: cnpg-operator
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-1"  # Bootstrap wave (before everything)
 spec:
  project: homelab
  source:
    repoURL: https://cloudnative-pg.github.io/charts
    chart: cloudnative-pg
    targetRevision: "~0.20"
  destination:
    server: https://kubernetes.default.svc
    namespace: ddb
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
@@ -1,152 +0,0 @@
 # PostgreSQL cluster + Forgejo dependencies (bootstrap only, not GitOps-managed).
 # These resources MUST exist before Forgejo can start, and Forgejo MUST exist
 # before ArgoCD can sync from the git repo it hosts → circular dependency.
 # Apply once via bootstrap.sh, never touched by ArgoCD afterward.
 ---
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:
  name: ddb-cluster
  namespace: ddb
  labels:
    app: postgresql
    layer: data
    bootstrap-phase: "0"
 spec:
  instances: 3  # HA across 3 control-plane nodes
  imageName: ghcr.io/cloudnative-pg/postgresql:16.2
  bootstrap:
    initdb:
      database: app
      owner: app
      encoding: UTF8
      localeCollate: C
      localeCType: C
      postInitApplicationSQL:
        - CREATE EXTENSION IF NOT EXISTS vector;
        - CREATE EXTENSION IF NOT EXISTS pgcrypto;
        - CREATE EXTENSION IF NOT EXISTS pg_trgm;
  # Role management: passwords from secrets, databases from separate Database CRs
  managed:
    roles:
      - name: authentik
        ensure: present
        login: true
        passwordSecret:
          name: authentik-db-role
      - name: temporal
        ensure: present
        login: true
        passwordSecret:
          name: temporal-db-role
  enableSuperuserAccess: false
  postgresql:
    parameters:
      shared_buffers: "256MB"
      max_parallel_workers: "4"
      max_parallel_workers_per_gather: "4"
      archive_mode: "on"
      archive_timeout: "5min"
      log_destination: "csvlog"
      log_directory: "/controller/log"
      log_filename: "postgres"
      log_rotation_age: "0"
      dynamic_shared_memory_type: "posix"
  storage:
    size: 10Gi
    storageClass: longhorn
  monitoring:
    enablePodMonitor: false
    disableDefaultQueries: false
    customQueriesConfigMap:
      - name: cnpg-default-monitoring
        key: queries
  affinity:
    podAntiAffinityType: preferred
 ---
 # Forgejo database (depends on ddb-cluster being ready)
 apiVersion: postgresql.cnpg.io/v1
 kind: Database
 metadata:
  name: forgejo
  namespace: ddb
  labels:
    bootstrap-phase: "0"
 spec:
  name: forgejo
  owner: app
  cluster:
    name: ddb-cluster
 ---
 # Forgejo Redis (cache, session, queue)
 apiVersion: v1
 kind: Service
 metadata:
  name: forgejo-redis
  namespace: cicd
  labels:
    app: forgejo-redis
    bootstrap-phase: "0"
 spec:
  ports:
    - port: 6379
      targetPort: 6379
      protocol: TCP
      name: redis
  selector:
    app: forgejo-redis
  type: ClusterIP
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: forgejo-redis
  namespace: cicd
  labels:
    app: forgejo-redis
    bootstrap-phase: "0"
 spec:
  replicas: 1
  selector:
    matchLabels:
      app: forgejo-redis
  template:
    metadata:
      labels:
        app: forgejo-redis
    spec:
      containers:
        - name: redis
          image: redis:7-alpine
          ports:
            - containerPort: 6379
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              cpu: 200m
              memory: 256Mi
          livenessProbe:
            tcpSocket:
              port: 6379
            initialDelaySeconds: 30
            periodSeconds: 10
          readinessProbe:
            exec:
              command:
                - redis-cli
                - ping
            initialDelaySeconds: 5
            periodSeconds: 5
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
@@ -1,151 +0,0 @@
 # Forgejo - Git server hosting the GitOps repo (bootstrap only, manual sync).
 # ArgoCD cannot auto-sync Forgejo because Forgejo hosts the repo ArgoCD syncs
 # from → circular dependency. Apply once via bootstrap, manual sync only afterward.
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: forgejo
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"  # Bootstrap wave
    bootstrap-phase: "0"
    description: "Bootstrap-only: Forgejo hosts the GitOps repo"
 spec:
  project: homelab
  source:
    repoURL: https://dl.gitea.com/charts/
    chart: gitea
    targetRevision: "~10"
    helm:
      # Inline values (git-independent) - keep in sync with k8s/security/ci-cd/forgejo-values.yaml
      valuesObject:
        image:
          repository: codeberg.org/forgejo/forgejo
          tag: "13"
          pullPolicy: IfNotPresent
        gitea:
          admin:
            username: rock
            email: [email protected]
          config:
            server:
              PROTOCOL: http
              DOMAIN: forgejo.riotpiao.com
              ROOT_URL: https://forgejo.riotpiao.com/
              HTTP_PORT: 3000
              START_SSH_SERVER: true
              SSH_DOMAIN: forgejo.riotpiao.com
              SSH_PORT: 2222
              SSH_LISTEN_PORT: 2222
            database:
              DB_TYPE: postgres
              HOST: ddb-cluster-rw.ddb.svc:5432
              NAME: forgejo
              USER: app
            repository:
              ROOT: /data/git
            actions:
              ENABLED: true
            packages:
              ENABLED: true
            metrics:
              ENABLED: true
            service:
              DISABLE_REGISTRATION: true
            oauth2:
              ENABLED: true
              PROVIDER: openidconnect
              OPENID_CONNECT_DISCOVERY_URL: https://authentik.riotpiao.com/application/o/forgejo/.well-known/openid-configuration
              CLIENT_ID: forgejo
              AUTO_DISCOVER_URL: https://authentik.riotpiao.com/application/o/forgejo/.well-known/openid-configuration
            cache:
              ADAPTER: redis
              HOST: "redis://forgejo-redis.cicd.svc:6379/0"
            session:
              PROVIDER: redis
              PROVIDER_CONFIG: "redis://forgejo-redis.cicd.svc:6379/1"
            queue:
              TYPE: redis
              CONN_STR: "redis://forgejo-redis.cicd.svc:6379/2"
          metrics:
            enabled: true
            serviceMonitor:
              enabled: false
        persistence:
          enabled: true
          storageClass: longhorn
          size: 20Gi
          accessModes:
            - ReadWriteMany
        replicaCount: 2
        deployment:
          strategy:
            type: RollingUpdate
          env:
            - name: SSL_CERT_DIR
              value: /homelab-ca
            - name: GITEA__database__PASSWD
              valueFrom:
                secretKeyRef:
                  name: ddb-cluster-app
                  key: password
            - name: GITEA__oauth2__CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  name: forgejo-oidc
                  key: CLIENT_SECRET
        podAnnotations:
          configmap.reloader.stakater.com/reload: "homelab-ca"
        service:
          http:
            type: LoadBalancer
            port: 3000
            targetPort: 3000
            annotations:
              io.cilium/lb-ipam-ips: "192.168.1.165"
              io.cilium/lb-ipam-sharing-key: "forgejo"
          ssh:
            type: LoadBalancer
            port: 2222
            targetPort: 2222
            annotations:
              io.cilium/lb-ipam-ips: "192.168.1.165"
              io.cilium/lb-ipam-sharing-key: "forgejo"
        resources:
          requests:
            cpu: 250m
            memory: 512Mi
          limits:
            cpu: "1"
            memory: 1Gi
        tolerations:
          - key: node-role.kubernetes.io/control-plane
            operator: Exists
            effect: NoSchedule
        extraVolumes:
          - name: homelab-ca
            configMap:
              name: homelab-ca
        extraVolumeMounts:
          - name: homelab-ca
            mountPath: /homelab-ca
            readOnly: true
        ingress:
          enabled: false
        postgresql:
          enabled: false
        postgresql-ha:
          enabled: false
        mysql:
          enabled: false
        redis-cluster:
          enabled: false
        act_runner:
          enabled: false
  destination:
    server: https://kubernetes.default.svc
    namespace: cicd
  syncPolicy:
    # NO automated sync - Forgejo hosts the repo; auto-sync would let a bad
    # CI commit break the system CI depends on. Manual sync only.
    syncOptions: []
@@ -1,89 +0,0 @@
 # Wait-for-databases Job - ensures Database CRs are reconciled before apps start
 # This solves the race condition where Forgejo starts before CNPG creates the database
 apiVersion: batch/v1
 kind: Job
 metadata:
  name: wait-for-databases
  namespace: ddb
  annotations:
    description: "Waits for CNPG to reconcile Database CRs and create databases in PostgreSQL"
 spec:
  backoffLimit: 5
  template:
    metadata:
      name: wait-for-databases
    spec:
      restartPolicy: Never
      serviceAccountName: wait-for-databases
      containers:
      - name: wait
        image: bitnami/kubectl:latest
        command:
        - /bin/bash
        - -c
        - |
          set -euo pipefail
          echo "==> Waiting for CNPG Database CRs to be reconciled..."
          DATABASES="forgejo authentik temporal temporal-visibility"
          for db in $DATABASES; do
            echo "Checking database: $db"
            for i in {1..60}; do
              # Check if Database CR exists and is ready
              READY=$(kubectl get database $db -n ddb -o jsonpath='{.status.ready}' 2>/dev/null || echo "false")
              if [ "$READY" = "true" ]; then
                echo "  ✓ $db is ready"
                break
              fi
              echo "  Waiting for $db to be ready... ($i/60)"
              sleep 5
              if [ $i -eq 60 ]; then
                echo "  ✗ Timeout waiting for $db"
                exit 1
              fi
            done
          done
          echo ""
          echo "==> All databases are ready!"
          echo "CNPG has created the following databases:"
          kubectl get databases -n ddb
          echo ""
          echo "✅ Safe to deploy applications now"
 ---
 apiVersion: v1
 kind: ServiceAccount
 metadata:
  name: wait-for-databases
  namespace: ddb
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
  name: wait-for-databases
  namespace: ddb
 rules:
 - apiGroups: ["postgresql.cnpg.io"]
  resources: ["databases"]
  verbs: ["get", "list", "watch"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
  name: wait-for-databases
  namespace: ddb
 roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: wait-for-databases
 subjects:
 - kind: ServiceAccount
  name: wait-for-databases
  namespace: ddb
@@ -1,56 +0,0 @@
 # ingress-nginx - required for Forgejo domain access before ArgoCD can sync
 # This breaks the circular dependency: ArgoCD needs https://forgejo.riotpiao.com
 # but that domain requires ingress-nginx to route traffic.
 apiVersion: v1
 kind: Namespace
 metadata:
  name: ingress-nginx
  labels:
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/warn: privileged
 ---
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: ingress-nginx-bootstrap
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
    description: "Bootstrap ingress-nginx to enable Forgejo domain access"
 spec:
  project: homelab
  source:
    repoURL: https://kubernetes.github.io/ingress-nginx
    chart: ingress-nginx
    targetRevision: "4.15.1"
    helm:
      values: |
        controller:
          kind: DaemonSet
          service:
            type: LoadBalancer
            annotations:
              io.cilium/lb-ipam-ips: "192.168.1.160"
          hostPort:
            enabled: false
          tolerations:
            - key: node-role.kubernetes.io/control-plane
              operator: Exists
              effect: NoSchedule
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 512Mi
  destination:
    server: https://kubernetes.default.svc
    namespace: ingress-nginx
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
@@ -1,24 +0,0 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 metadata:
  name: bootstrap-local
  annotations:
    description: |
      Phase 0 bootstrap bundle - apply once from local checkout on a fresh cluster.
      Contains only resources that have circular git dependencies (Forgejo hosts
      the repo ArgoCD syncs from). Everything else is GitOps-managed via ArgoCD.
 # Resources in strict dependency order
 resources:
  - 00-namespaces.yaml           # Pre-create with PodSecurity labels
  - 01-argocd.yaml               # ArgoCD + SOPS plugin ConfigMap
  - 02-cnpg-operator.yaml        # CloudNativePG operator + CRDs
  - 03-ddb-bootstrap.yaml        # PostgreSQL cluster + Forgejo DB + Redis
  - 05-wait-for-databases.yaml   # Wait for CNPG to create databases
  - 04-forgejo.yaml              # Forgejo Helm chart (inline values)
  - 06-ingress-nginx.yaml        # Ingress for Forgejo domain access
 # Notes:
 # - SOPS age secret created via bootstrap.sh (not in git)
 # - After bootstrap: git push → kubectl apply -k k8s/argocd/root → done
 # - ALL future changes via git push (ArgoCD auto-syncs)
@@ -0,0 +1,128 @@
 # Homelab Bootstrap — Single-Cluster, GitOps-Ready
 **Run once manually, GitOps forever after.**
 This bootstrap breaks the ArgoCD ↔ Forgejo circular dependency by:
 1. Installing infrastructure in correct dependency order
 2. Pointing ArgoCD at a GitHub mirror initially
 3. Cutting over to Forgejo once healthy
 4. Using Helm for reproducible installs
 5. Ensuring ArgoCD adopts (not duplicates) bootstrap resources
 ## Prerequisites
 - Talos cluster running (terraform applied)
 - kubectl configured (`KUBECONFIG` points at cluster)
 - Helm 3 installed
 - SOPS age key at `~/.sops/homelab-age.key`
 - GitHub mirror of this repo (for initial ArgoCD source)
 ## Directory Structure
 ```
 bootstrap/
 ├── phase1-storage/         # Longhorn via Helm
 ├── phase2-cnpg/           # CNPG operator via Helm
 ├── phase3-forgejo/        # Forgejo DB + Forgejo via Helm
 ├── phase4-argocd/         # ArgoCD via Helm → GitHub initially
 └── phase5-cutover/        # Switch ArgoCD source to Forgejo
 ```
 ## Usage
 ```bash
 # From repo root:
 ./bootstrap.sh
 # Or step-by-step:
 ./bootstrap.sh phase1  # Storage
 ./bootstrap.sh phase2  # CNPG
 ./bootstrap.sh phase3  # Forgejo
 ./bootstrap.sh phase4  # ArgoCD (GitHub mirror)
 ./bootstrap.sh phase5  # Cut over to Forgejo
 ```
 ## Design Principles
 1. **DRY**: Helm values used by both bootstrap and ArgoCD
 2. **Single Source of Truth**: Manifests match what ArgoCD will manage
 3. **Idempotent**: Can re-run phases safely
 4. **Adoption Ready**: Resources have `argocd.argoproj.io/sync-options: Prune=false`
 5. **Dependency Ordered**: Each phase waits for previous to be Ready
 ## Phase Details
 ### Phase 1: Storage (Longhorn)
 Installs Longhorn with:
 - 3-node HA configuration
 - Unified `longhorn` StorageClass (default)
 - Special `longhorn-cnpg` StorageClass with postgres UID/GID mount options
 - CSI plugin tolerations for control-plane nodes
 **Source of Truth**: `phase1-storage/longhorn-values.yaml`
 ### Phase 2: CNPG Operator
 Installs CloudNativePG operator with:
 - CRD registration (blocks until CRD available)
 - Webhook configuration
 - Monitoring enabled
 **Source of Truth**: `phase2-cnpg/cnpg-values.yaml`
 ### Phase 3: Forgejo Database + Forgejo
 1. Creates `forgejo-db` CNPG Cluster
 2. Waits for cluster Ready (PostgreSQL accepting connections)
 3. Installs Forgejo via Helm pointing at `forgejo-db-rw` service
 4. Waits for Forgejo healthy
 **Source of Truth**: 
 - `phase3-forgejo/forgejo-db.yaml` (CNPG Cluster CR)
 - `phase3-forgejo/forgejo-values.yaml` (Helm values)
 ### Phase 4: ArgoCD (GitHub Mirror)
 Installs ArgoCD via Helm, then applies root app-of-apps pointing at **GitHub mirror**.
 This is the circle-breaker: ArgoCD syncs from GitHub (not Forgejo) initially.
 **Source of Truth**: 
 - `phase4-argocd/argocd-values.yaml`
 - `phase4-argocd/root-app-github.yaml` (repoURL = GitHub)
 ArgoCD **adopts** Phases 1-3 resources (no duplication) because manifests match.
 ### Phase 5: Cut Over to Forgejo
 1. Push repo to Forgejo
 2. Update root app `repoURL` from GitHub → Forgejo
 3. ArgoCD re-syncs from Forgejo
 **The circle is broken. GitHub mirror is now disaster recovery only.**
 ## Post-Bootstrap
 All changes via Git:
 ```bash
 git commit -m "feat(app): add new service"
 git push forgejo main
 # ArgoCD auto-syncs
 ```
 ## Troubleshooting
 - **Phase stuck?** Check `kubectl get events -n <namespace> --sort-by='.lastTimestamp'`
 - **ArgoCD duplicating?** Verify manifests match exactly (Helm values ↔ ArgoCD Application)
 - **Forgejo won't start?** Check CNPG cluster Ready: `kubectl get cluster forgejo-db -n forgejo`
 - **Can't push to Forgejo?** Verify ingress-nginx healthy, DNS resolves `forgejo.riotpiao.com`
 ## Migration from Old Bootstrap
 If you have existing `k8s/bootstrap-local/`:
 1. **DO NOT delete** existing resources (Longhorn data!)
 2. Run refined bootstrap in "adoption mode" (no delete, just apply)
 3. Verify ArgoCD shows "Synced" for all apps
 4. Archive old bootstrap: `git mv k8s/bootstrap-local k8s/archive/bootstrap-local-v1`
@@ -0,0 +1,56 @@
 # Longhorn Helm Values — Single Source of Truth
 # Used by both bootstrap.sh (Helm install) and ArgoCD (adoption)
 # Chart: https://github.com/longhorn/charts
 defaultSettings:
  # 3-node HA configuration
  replicaReplenishmentWaitInterval: 600  # 10min before auto-repair
  replicaSoftAntiAffinity: false  # REQUIRED for true HA
  replicaAutoBalance: best-effort
  storageMinimalAvailablePercentage: 10
  # Performance tuning
  defaultDataPath: /var/lib/longhorn
  defaultDataLocality: best-effort
  backupTarget: ""  # TODO: Add MinIO backup target later
  # Monitoring
  guaranteedEngineManagerCPU: 12  # mCPU
  guaranteedReplicaManagerCPU: 12
 persistence:
  defaultClass: true  # Make 'longhorn' the default StorageClass
  defaultClassReplicaCount: 3
  defaultFsType: ext4
  reclaimPolicy: Delete
 # CSI plugin must tolerate control-plane taints
 csi:
  kubeletRootDir: /var/lib/kubelet
  attacherReplicaCount: 3
  provisionerReplicaCount: 3
  resizerReplicaCount: 3
  snapshotterReplicaCount: 3
 # Longhorn manager on all nodes
 longhornManager:
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
 # Driver deployer tolerations
 longhornDriver:
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
 # UI for debugging
 longhornUI:
  replicas: 1
 # Monitoring (Prometheus ServiceMonitor)
 metrics:
  serviceMonitor:
    enabled: true
@@ -0,0 +1,27 @@
 # StorageClasses — Applied after Longhorn installation
 # The default 'longhorn' SC is created by Helm chart
 # These are additional specialized classes
 ---
 # CNPG-specific StorageClass with postgres UID/GID mount options
 # Fixes "read-only filesystem" error when PostgreSQL (UID 26) tries to write
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
 metadata:
  name: longhorn-cnpg
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
    argocd.argoproj.io/sync-options: Prune=false  # Allow ArgoCD adoption
 provisioner: driver.longhorn.io
 allowVolumeExpansion: true
 parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "30"
  fromBackup: ""
  dataLocality: "best-effort"
  fsType: "ext4"
 mountOptions:
  - "noatime"
  - "uid=26"   # postgres user
  - "gid=26"   # postgres group
 reclaimPolicy: Delete
 volumeBindingMode: Immediate
@@ -0,0 +1,55 @@
 # CloudNativePG Operator Helm Values — Single Source of Truth
 # Chart: https://github.com/cloudnative-pg/charts
 # Basic operator configuration
 replicaCount: 1
 # CRDs must be installed (Helm default behavior)
 crds:
  create: true
 # Webhook configuration
 webhook:
  port: 9443
  mutating:
    create: true
    failurePolicy: Fail
  validating:
    create: true
    failurePolicy: Fail
 # Monitoring
 monitoring:
  podMonitorEnabled: true
  grafanaDashboard:
    create: false  # We'll manage dashboards via ArgoCD later
 # Resource limits for operator
 resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 512Mi
 # Security context
 securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 1000
  capabilities:
    drop:
      - ALL
 # Tolerate control-plane taints
 tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
    effect: NoSchedule
 # Additional labels for ArgoCD adoption
 commonLabels:
  app.kubernetes.io/managed-by: Helm
  argocd.argoproj.io/instance: cnpg-operator
@@ -0,0 +1,122 @@
 # Forgejo PostgreSQL Database — CNPG Cluster CR
 # This is the source of truth for Forgejo's database
 # ArgoCD will adopt this (not recreate it)
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: forgejo
  annotations:
    argocd.argoproj.io/sync-options: Prune=false
 ---
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:
  name: forgejo-db
  namespace: forgejo
  annotations:
    argocd.argoproj.io/sync-options: Prune=false  # Let ArgoCD adopt, don't delete
  labels:
    app: forgejo-db
    layer: data
 spec:
  instances: 3  # HA configuration
  imageName: ghcr.io/cloudnative-pg/postgresql:16.2
  bootstrap:
    initdb:
      database: forgejo
      owner: forgejo
      encoding: UTF8
      localeCollate: C
      localeCType: C
  enableSuperuserAccess: false
  # Resource limits per best practices
  resources:
    requests:
      memory: "4Gi"
      cpu: "1"
    limits:
      memory: "8Gi"
      cpu: "2"
  postgresql:
    parameters:
      # Tuned for 4-8GB RAM
      shared_buffers: "1GB"
      effective_cache_size: "3GB"
      maintenance_work_mem: "512MB"
      work_mem: "64MB"
      max_connections: "100"
      max_parallel_workers: "2"
      max_parallel_workers_per_gather: "1"
      # WAL
      wal_buffers: "16MB"
      checkpoint_completion_target: "0.9"
      min_wal_size: "512MB"
      max_wal_size: "2GB"
      # Logging
      log_destination: "csvlog"
      log_directory: "/controller/log"
      log_filename: "postgres"
  storage:
    size: 50Gi
    storageClass: longhorn-cnpg  # Uses postgres UID/GID mount options
  monitoring:
    enablePodMonitor: true
  affinity:
    podAntiAffinityType: required
    topologyKey: kubernetes.io/hostname
 ---
 # Forgejo Redis (cache, session, queue)
 apiVersion: v1
 kind: Service
 metadata:
  name: forgejo-redis
  namespace: forgejo
  annotations:
    argocd.argoproj.io/sync-options: Prune=false
 spec:
  ports:
    - port: 6379
      targetPort: 6379
      protocol: TCP
  selector:
    app: forgejo-redis
  type: ClusterIP
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: forgejo-redis
  namespace: forgejo
  annotations:
    argocd.argoproj.io/sync-options: Prune=false
 spec:
  replicas: 1
  selector:
    matchLabels:
      app: forgejo-redis
  template:
    metadata:
      labels:
        app: forgejo-redis
    spec:
      containers:
        - name: redis
          image: redis:7-alpine
          ports:
            - containerPort: 6379
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              cpu: 200m
              memory: 256Mi
@@ -0,0 +1,86 @@
 # Forgejo Helm Values — Single Source of Truth
 # Chart: https://codeberg.org/forgejo-contrib/forgejo-helm
 gitea:
  admin:
    username: "admin"
    email: "[email protected]"
    # Password set via secret (not in values)
  config:
    server:
      DOMAIN: forgejo.riotpiao.com
      ROOT_URL: https://forgejo.riotpiao.com
      SSH_DOMAIN: forgejo.riotpiao.com
      SSH_PORT: 22
    database:
      DB_TYPE: postgres
      HOST: forgejo-db-rw.forgejo.svc.cluster.local:5432
      NAME: forgejo
      # User/password from CNPG-generated secret
      USER:
        valueFrom:
          secretKeyRef:
            name: forgejo-db-app
            key: username
      PASSWD:
        valueFrom:
          secretKeyRef:
            name: forgejo-db-app
            key: password
    cache:
      ADAPTER: redis
      HOST: redis://forgejo-redis.forgejo.svc.cluster.local:6379/0
    session:
      PROVIDER: redis
      PROVIDER_CONFIG: redis://forgejo-redis.forgejo.svc.cluster.local:6379/1
    queue:
      TYPE: redis
      CONN_STR: redis://forgejo-redis.forgejo.svc.cluster.local:6379/2
 # Persistence (shared storage for repos)
 persistence:
  enabled: true
  storageClass: longhorn
  size: 20Gi
  accessModes:
    - ReadWriteOnce
 # Ingress
 ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  hosts:
    - host: forgejo.riotpiao.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: forgejo-tls
      hosts:
        - forgejo.riotpiao.com
 # Resources
 resources:
  requests:
    cpu: 200m
    memory: 512Mi
  limits:
    cpu: 1000m
    memory: 2Gi
 # Tolerations for control-plane
 tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
    effect: NoSchedule
 # ArgoCD adoption labels
 labels:
  argocd.argoproj.io/instance: forgejo
@@ -0,0 +1,124 @@
 # ArgoCD Helm Values — Single Source of Truth
 # Chart: https://github.com/argoproj/argo-helm
 global:
  domain: argocd.riotpiao.com
 # Server configuration
 server:
  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
      nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    hosts:
      - argocd.riotpiao.com
    tls:
      - secretName: argocd-server-tls
        hosts:
          - argocd.riotpiao.com
  # Allow insecure mode (terminate TLS at ingress)
  extraArgs:
    - --insecure
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      cpu: 500m
      memory: 1Gi
 # Repo server configuration
 repoServer:
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      cpu: 500m
      memory: 1Gi
  # SOPS plugin for encrypted secrets
  volumes:
    - name: sops-age
      secret:
        secretName: sops-age
        optional: true
  volumeMounts:
    - name: sops-age
      mountPath: /home/argocd/.config/sops/age
      readOnly: true
  # Environment for SOPS
  env:
    - name: SOPS_AGE_KEY_FILE
      value: /home/argocd/.config/sops/age/keys.txt
 # Controller configuration
 controller:
  resources:
    requests:
      cpu: 200m
      memory: 512Mi
    limits:
      cpu: 1000m
      memory: 2Gi
 # Application controller configuration
 applicationSet:
  enabled: true
 # Notifications (optional, for Slack/Discord alerts)
 notifications:
  enabled: false
 # Redis for caching
 redis:
  enabled: true
  resources:
    requests:
      cpu: 50m
      memory: 64Mi
    limits:
      cpu: 200m
      memory: 256Mi
 # Tolerations for control-plane
 server:
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
 repoServer:
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
 controller:
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
 # ArgoCD projects
 configs:
  # Default project allows all repos
  cm:
    admin.enabled: "true"
    application.instanceLabelKey: argocd.argoproj.io/instance
  params:
    server.insecure: true
 # RBAC (allow admin full access)
 configs:
  rbac:
    policy.default: role:readonly
    policy.csv: |
      g, admin, role:admin
@@ -0,0 +1,66 @@
 # ArgoCD Root App-of-Apps — GitHub Mirror Source
 # This is the initial configuration that breaks the circular dependency
 # Points at GitHub mirror, not Forgejo (Forgejo isn't ready yet)
 # 
 # After Forgejo is healthy and repo is pushed, use phase5-cutover/root-app-forgejo.yaml
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: argocd
 ---
 apiVersion: argoproj.io/v1alpha1
 kind: AppProject
 metadata:
  name: homelab
  namespace: argocd
 spec:
  description: Homelab infrastructure and applications
  sourceRepos:
    - 'https://github.com/YOUR-ORG/homelab.git'  # ← REPLACE with your GitHub mirror
    - 'https://forgejo.riotpiao.com/YOUR-ORG/homelab.git'
    - 'https://*.github.io/*'  # Helm charts from GitHub Pages
    - 'https://charts.*'       # Public Helm repos
  destinations:
    - namespace: '*'
      server: 'https://kubernetes.default.svc'
  clusterResourceWhitelist:
    - group: '*'
      kind: '*'
  namespaceResourceWhitelist:
    - group: '*'
      kind: '*'
 ---
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: homelab-root
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
 spec:
  project: homelab
  source:
    repoURL: https://github.com/YOUR-ORG/homelab.git  # ← REPLACE with your GitHub mirror
    targetRevision: main
    path: k8s/argocd/apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
@@ -0,0 +1,39 @@
 # ArgoCD Root App-of-Apps — Forgejo Source (Final State)
 # This replaces the GitHub mirror with Forgejo as the source of truth
 # Apply this AFTER Forgejo is healthy and you've pushed the repo
 #
 # The circle is broken: ArgoCD → Forgejo works because both already exist
 ---
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
  name: homelab-root
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
 spec:
  project: homelab
  source:
    repoURL: https://forgejo.riotpiao.com/YOUR-ORG/homelab.git  # ← REPLACE with your Forgejo URL
    targetRevision: main
    path: k8s/argocd/apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
@@ -3,6 +3,7 @@ kind: Kustomization
 namespace: longhorn-system
 resources:
  - longhorn-storageclass.yaml
  - longhorn-cnpg-storageclass.yaml  # CNPG-specific with postgres UID/GID
  - longhorn-servicemonitor.yaml
  - longhorn-taint-toleration.yaml
  - longhorn-nodes.yaml
@@ -0,0 +1,26 @@
 # StorageClass specifically for CNPG (CloudNativePG) PostgreSQL clusters
 # Fixes the "read-only filesystem" issue by mounting with postgres UID/GID
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
 metadata:
  name: longhorn-cnpg
  namespace: longhorn-system
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
 provisioner: driver.longhorn.io
 allowVolumeExpansion: true
 parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "30"
  fromBackup: ""
  dataLocality: "best-effort"
  fsType: "ext4"
  # Mount options to ensure PostgreSQL can write
  mkfsParams: "-O ^64bit,^metadata_csum"
 mountOptions:
  - "noatime"
  # Critical: mount with postgres UID/GID (26:26) to avoid permission issues
  - "uid=26"
  - "gid=26"
 reclaimPolicy: Delete
 volumeBindingMode: Immediate
@@ -0,0 +1,55 @@
 # Continuous smoke — runs the full WebKit suite every 15 min. A failed run means
 # something users touch broke; alert on it (Job failure → kube-state-metrics
 # kube_job_status_failed → Alertmanager). Same image/env as the deploy-gate Job.
 apiVersion: batch/v1
 kind: CronJob
 metadata:
  name: e2e-smoke
  namespace: platform
  annotations:
    argocd.argoproj.io/sync-options: Prune=false
 spec:
  schedule: "*/15 * * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      backoffLimit: 1
      ttlSecondsAfterFinished: 86400
      template:
        spec:
          restartPolicy: Never
          tolerations:
            - key: node-role.kubernetes.io/control-plane
              operator: Exists
              effect: NoSchedule
          containers:
            - name: e2e
              image: forgejo-gitea-http.cicd.svc.cluster.local:3000/riotpiao.com/homelab-e2e:latest
              imagePullPolicy: Always
              env:
                - name: BASE_DOMAIN
                  value: riotpiao.com
                - name: E2E_IGNORE_TLS
                  value: "0"
                - name: CI
                  value: "1"
                - name: AK_ADMIN_USER
                  value: akadmin
                - name: AK_ADMIN_PASSWORD
                  valueFrom:
                    secretKeyRef: { name: e2e-credentials, key: authentik-admin-password }
                - name: MINIO_ENDPOINT
                  value: http://minio.storage.svc.cluster.local:9000
                - name: MINIO_BUCKET
                  value: e2e-artifacts
                - name: MINIO_ACCESS_KEY
                  valueFrom:
                    secretKeyRef: { name: e2e-credentials, key: minio-access-key }
                - name: MINIO_SECRET_KEY
                  valueFrom:
                    secretKeyRef: { name: e2e-credentials, key: minio-secret-key }
              resources:
                requests: { cpu: 200m, memory: 512Mi }
                limits: { cpu: "1", memory: 2Gi }
@@ -0,0 +1,59 @@
 # One-shot E2E smoke — run as a deploy verification gate.
 # Wire as an ArgoCD PostSync hook (annotation below) OR call from an Argo Rollouts
 # AnalysisTemplate. Job success == the app is actually viewable in Safari's engine;
 # failure fails the sync/rollout.
 #
 # NOTE: not yet added to any kustomization / app-of-apps. Wiring = migration phase 7.
 apiVersion: batch/v1
 kind: Job
 metadata:
  name: e2e-smoke
  namespace: platform
  annotations:
    # Uncomment to make this a deploy gate on the owning Application:
    # argocd.argoproj.io/hook: PostSync
    # argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
    argocd.argoproj.io/sync-options: Prune=false
 spec:
  backoffLimit: 1
  ttlSecondsAfterFinished: 86400
  template:
    spec:
      restartPolicy: Never
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      containers:
        - name: e2e
          # Built + pushed by CI from tests/e2e/Dockerfile. Pin a digest in prod.
          image: forgejo-gitea-http.cicd.svc.cluster.local:3000/riotpiao.com/homelab-e2e:latest
          imagePullPolicy: Always
          env:
            - name: BASE_DOMAIN
              value: riotpiao.com
            # strict TLS by default; set "1" only during staging-cert bootstrap
            - name: E2E_IGNORE_TLS
              value: "0"
            - name: CI
              value: "1"
            - name: AK_ADMIN_USER
              value: akadmin
            - name: AK_ADMIN_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: e2e-credentials
                  key: authentik-admin-password
            - name: MINIO_ENDPOINT
              value: http://minio.storage.svc.cluster.local:9000
            - name: MINIO_BUCKET
              value: e2e-artifacts
            - name: MINIO_ACCESS_KEY
              valueFrom:
                secretKeyRef: { name: e2e-credentials, key: minio-access-key }
            - name: MINIO_SECRET_KEY
              valueFrom:
                secretKeyRef: { name: e2e-credentials, key: minio-secret-key }
          resources:
            requests: { cpu: 200m, memory: 512Mi }
            limits: { cpu: "1", memory: 2Gi }
@@ -0,0 +1,4 @@
 node_modules/
 test-results/
 playwright-report/
 .env
@@ -0,0 +1,18 @@
 # Playwright base ships WebKit + all OS deps. Pin must match @playwright/test in
 # package.json. Verify tag exists: https://mcr.microsoft.com/en-us/product/playwright/tags
 FROM mcr.microsoft.com/playwright:v1.50.0-jammy
 WORKDIR /e2e
 # minio client for artifact upload (video/trace → MinIO bucket)
 RUN curl -sSLo /usr/local/bin/mc https://dl.min.io/client/mc/release/linux-amd64/mc \
    && chmod +x /usr/local/bin/mc
 COPY package.json ./
 RUN npm install --no-audit --no-fund
 COPY . .
 # run-and-upload.sh runs the suite, uploads artifacts regardless of result,
 # then exits with the suite's real exit code so the Job/gate reflects pass/fail.
 ENTRYPOINT ["/e2e/run-and-upload.sh"]
@@ -0,0 +1,44 @@
 # Homelab E2E smoke (Playwright · WebKit / Safari engine)
 Proves apps are **viewable**, not just that a pod is `Running`. Verifies real
 ingress + TLS + rendered UI through WebKit — Safari's engine (hard constraint).
 ## Layout
 - `playwright.config.ts` — WebKit-only project, video + trace + screenshot always on.
 - `targets.ts` — the app list (`NAME-APP.<BASE_DOMAIN>`) + per-app "ready" selector. **Edit to match your ingress hosts.**
 - `tests/reachability.spec.ts` — every non-auth app: 2xx/3xx + own-UI rendered (not nginx default backend).
 - `tests/portainer.spec.ts` — login page renders.
 - `tests/authentik.spec.ts` — full admin sign-in (needs `AK_ADMIN_PASSWORD`).
 - `tests/oauth.spec.ts` — OIDC/OAuth **federation** flow per app: app → Authentik → back, authenticated. Driven by `OAUTH_TARGETS` in `targets.ts`; uses `helpers/authentik.ts`.
 - `Dockerfile` / `run-and-upload.sh` — container image; runs suite, uploads artifacts to MinIO, exits with the suite's real code.
 ## Run locally
 ```bash
 cd tests/e2e
 npm install
 npx playwright install webkit
 BASE_DOMAIN=riotpiao.com npm test          # add E2E_IGNORE_TLS=1 for staging certs
 AK_ADMIN_PASSWORD=… npm test               # to exercise the authentik login
 npm run report                              # open HTML report (video/trace)
 ```
 ## Env
 | Var | Default | Purpose |
 |---|---|---|
 | `BASE_DOMAIN` | `riotpiao.com` | domain suffix for `NAME.<domain>` |
 | `E2E_IGNORE_TLS` | `0` | `1` = accept staging/self-signed certs (bootstrap only) |
 | `AK_ADMIN_USER` / `AK_ADMIN_PASSWORD` | `akadmin` / — | authentik login (password from secret store) |
 | `MINIO_ENDPOINT` / `MINIO_BUCKET` / `MINIO_ACCESS_KEY` / `MINIO_SECRET_KEY` | — | artifact upload target |
 ## In-cluster
 `k8s/platform/e2e/` has a **Job** (deploy gate — attach as ArgoCD PostSync hook or
 Rollouts AnalysisTemplate) and a **CronJob** (continuous smoke, alert on failure).
 Both need a `e2e-credentials` Secret (authentik password + MinIO keys) — create it
 SOPS-encrypted. **Not yet wired into any kustomization / app-of-apps** — that is
 ADR 0001 migration phase 7.
 ## Build image
 ```bash
 docker build -t <registry>/riotpiao.com/homelab-e2e:$(git rev-parse --short HEAD) tests/e2e
 docker push  <registry>/riotpiao.com/homelab-e2e:…
 ```
@@ -0,0 +1,27 @@
 import { Page, expect } from '@playwright/test';
 // Complete Authentik's identification → password flow on whatever page is
 // currently showing it (used both for direct login and mid-OAuth redirect).
 // Idempotent: if Authentik already has a session and skipped straight through,
 // the fields won't appear and this returns without error.
 export async function completeAuthentikLogin(page: Page, user: string, pass: string): Promise<void> {
  const uid = page.locator('input[name="uidField"], input[type="email"], input[type="text"]').first();
  if (await uid.isVisible({ timeout: 8_000 }).catch(() => false)) {
    await uid.fill(user);
    await page.keyboard.press('Enter');
  }
  const pw = page.locator('input[type="password"]').first();
  if (await pw.isVisible({ timeout: 8_000 }).catch(() => false)) {
    await pw.fill(pass);
    await page.keyboard.press('Enter');
  }
  // Authentik may show a consent/authorize step on first SSO to an app.
  const authorize = page.getByRole('button', { name: /authorize|continue|allow/i }).first();
  if (await authorize.isVisible({ timeout: 5_000 }).catch(() => false)) {
    await authorize.click();
  }
  await expect(page.getByText(/invalid|incorrect|failed to/i)).toHaveCount(0);
 }
@@ -0,0 +1,12 @@
 {
  "name": "homelab-e2e",
  "private": true,
  "description": "Playwright WebKit (Safari-engine) synthetic E2E smoke suite for homelab ingress apps",
  "scripts": {
    "test": "playwright test --project=webkit",
    "report": "playwright show-report"
  },
  "devDependencies": {
    "@playwright/test": "1.50.0"
  }
 }
@@ -0,0 +1,40 @@
 import { defineConfig, devices } from '@playwright/test';
 // WebKit only — hard constraint: verify apps in Safari's rendering engine.
 // Video + trace recorded for every test so a failed deploy has a replay.
 const BASE_DOMAIN = process.env.BASE_DOMAIN ?? 'riotpiao.com';
 // Set E2E_IGNORE_TLS=1 only during early bootstrap (staging certs). Default is
 // strict so a broken TLS chain FAILS the suite — that is a real deploy failure.
 const ignoreHTTPSErrors = process.env.E2E_IGNORE_TLS === '1';
 export default defineConfig({
  testDir: './tests',
  outputDir: './test-results',
  // Serial + retries: this is a smoke gate, not a load test.
  fullyParallel: false,
  workers: 1,
  retries: process.env.CI ? 1 : 0,
  timeout: 60_000,
  expect: { timeout: 15_000 },
  reporter: [
    ['list'],
    ['html', { outputFolder: 'playwright-report', open: 'never' }],
    ['json', { outputFile: 'test-results/results.json' }],
  ],
  use: {
    baseURL: `https://${BASE_DOMAIN}`,
    ignoreHTTPSErrors,
    video: 'on',
    trace: 'on',
    screenshot: 'on',
    navigationTimeout: 30_000,
    actionTimeout: 15_000,
  },
  projects: [
    { name: 'webkit', use: { ...devices['Desktop Safari'] } },
  ],
 });
@@ -0,0 +1,26 @@
 #!/usr/bin/env bash
 # Run the WebKit smoke suite, upload artifacts (video/trace/screenshots/report)
 # to MinIO, then exit with the suite's real result so a CD gate can act on it.
 set -uo pipefail
 STAMP="${RUN_ID:-$(date +%Y%m%d-%H%M%S)}"
 DEST="e2e/${STAMP}"
 echo "▶ Playwright WebKit smoke — BASE_DOMAIN=${BASE_DOMAIN:-riotpiao.com}"
 npm test
 RESULT=$?
 echo "▶ suite exit=${RESULT}"
 # Upload artifacts even on failure — a failed deploy is exactly when you want the video.
 if [[ -n "${MINIO_ENDPOINT:-}" && -n "${MINIO_ACCESS_KEY:-}" ]]; then
  echo "▶ uploading artifacts → ${MINIO_ENDPOINT}/${MINIO_BUCKET:-e2e-artifacts}/${DEST}"
  mc alias set store "${MINIO_ENDPOINT}" "${MINIO_ACCESS_KEY}" "${MINIO_SECRET_KEY}" >/dev/null 2>&1
  mc mb --ignore-existing "store/${MINIO_BUCKET:-e2e-artifacts}" >/dev/null 2>&1
  mc cp --recursive test-results/       "store/${MINIO_BUCKET:-e2e-artifacts}/${DEST}/test-results/"       >/dev/null 2>&1 || true
  mc cp --recursive playwright-report/  "store/${MINIO_BUCKET:-e2e-artifacts}/${DEST}/playwright-report/"  >/dev/null 2>&1 || true
  echo "▶ artifacts uploaded under ${DEST}/"
 else
  echo "▶ MINIO_ENDPOINT unset — skipping upload (artifacts in ./test-results)"
 fi
 exit ${RESULT}
@@ -0,0 +1,72 @@
 // Ingress apps to smoke-test. Each entry = NAME-APP.<BASE_DOMAIN>.
 // `ready` is a selector or text that only appears once the app actually rendered
 // (not just a 200 from nginx's default backend).
 //
 // EDIT THIS LIST to match what you actually expose. Confirmed hosts are marked;
 // others are best-guess — fix the subdomain if yours differs.
 export type Target = {
  name: string;
  subdomain: string;
  path?: string;
  // A locator that proves the app's own UI rendered.
  ready: { role?: string; name?: RegExp; text?: RegExp; selector?: string };
  // Skip in the default reachability run (has a dedicated spec, e.g. auth flow).
  dedicated?: boolean;
 };
 export const BASE_DOMAIN = process.env.BASE_DOMAIN ?? 'riotpiao.com';
 export function url(t: Pick<Target, 'subdomain' | 'path'>): string {
  return `https://${t.subdomain}.${BASE_DOMAIN}${t.path ?? '/'}`;
 }
 // Apps that federate login through Authentik (OIDC/OAuth2). The OAuth spec drives
 // the full chain: open the app, click its "sign in via SSO" control, complete the
 // Authentik login on redirect, and assert we land back in the app authenticated.
 // EDIT selectors to match your apps — the SSO button text/label varies per app.
 export type OAuthTarget = {
  name: string;
  subdomain: string;
  path?: string;
  // The app's "log in with Authentik/SSO" control.
  ssoButton: { role?: string; name?: RegExp; selector?: string };
  // Proof we are authenticated back inside the app after the round-trip.
  success: { urlRe?: RegExp; text?: RegExp; selector?: string };
 };
 export const OAUTH_TARGETS: OAuthTarget[] = [
  {
    name: 'argocd',
    subdomain: 'argocd',
    ssoButton: { role: 'link', name: /log ?in via|authentik|sso|oidc/i },
    success: { urlRe: /\/applications/i, text: /Applications/i },
  },
  {
    name: 'forgejo',
    subdomain: 'forgejo',
    path: '/user/login',
    ssoButton: { role: 'link', name: /authentik|oauth|sign in with|openid/i },
    success: { selector: 'a[href="/notifications"], .avatar, nav .user' },
  },
  {
    name: 'grafana',
    subdomain: 'grafana',
    path: '/login',
    ssoButton: { role: 'link', name: /sign in with|authentik|oauth/i },
    success: { urlRe: /\/(\?|$)|\/d\//, text: /Welcome to Grafana|Dashboards|Home/i },
  },
 ];
 export const TARGETS: Target[] = [
  // confirmed hosts (from bootstrap values)
  { name: 'argocd', subdomain: 'argocd', ready: { text: /Argo\s*CD|Let's get started|Login/i } },
  { name: 'forgejo', subdomain: 'forgejo', ready: { text: /Forgejo|Sign In|Explore/i } },
  // dashboards / apps — adjust subdomain to your actual ingress host
  { name: 'portainer', subdomain: 'portainer', dedicated: true, ready: { selector: 'input[type="password"]' } },
  { name: 'authentik', subdomain: 'authentik', dedicated: true, ready: { selector: 'input[name="uidField"], input[type="password"]' } },
  { name: 'homarr', subdomain: 'homarr', ready: { text: /Homarr|Dashboard/i } },
  { name: 'grafana', subdomain: 'grafana', ready: { text: /Grafana|Welcome/i } },
  { name: 'minio', subdomain: 'minio', ready: { selector: 'input, button' } },
 ];
@@ -0,0 +1,32 @@
 import { test, expect } from '@playwright/test';
 import { TARGETS, url } from '../targets';
 const t = TARGETS.find((x) => x.name === 'authentik')!;
 // Authentik: the real signal. A healthy pod does NOT prove OIDC/redirect/session
 // work — only a completed sign-in does. Credentials come from env (injected from
 // the secret store in-cluster), never hardcoded.
 const USER = process.env.AK_ADMIN_USER ?? 'akadmin';
 const PASS = process.env.AK_ADMIN_PASSWORD;
 test('authentik admin can sign in', async ({ page }) => {
  test.skip(!PASS, 'AK_ADMIN_PASSWORD not set — provide via secret to run the login flow');
  await page.goto(url(t), { waitUntil: 'domcontentloaded' });
  // Authentik identification stage: username, Enter/continue, then password.
  const uid = page.locator('input[name="uidField"], input[type="text"], input[type="email"]').first();
  await expect(uid).toBeVisible();
  await uid.fill(USER);
  await page.keyboard.press('Enter');
  const pw = page.locator('input[type="password"]').first();
  await expect(pw).toBeVisible();
  await pw.fill(PASS!);
  await page.keyboard.press('Enter');
  // Landed on the user dashboard — no auth error banner.
  await expect(page).toHaveURL(/\/if\/user|\/if\/admin|\/library/i, { timeout: 20_000 });
  await expect(page.getByText(/invalid|incorrect|failed/i)).toHaveCount(0);
  await page.screenshot({ path: 'test-results/authentik-dashboard.png', fullPage: true });
 });
@@ -0,0 +1,47 @@
 import { test, expect } from '@playwright/test';
 import { OAUTH_TARGETS, url, BASE_DOMAIN } from '../targets';
 import { completeAuthentikLogin } from '../helpers/authentik';
 // OIDC/OAuth federation flow — the real SSO integration signal. A healthy app pod
 // does NOT prove SSO works: client mis-registration, redirect-URI mismatch, issuer
 // cert, or a broken Authentik provider all fail HERE, not at the pod. Each test:
 //   1. open the app,
 //   2. click its "sign in via Authentik" control,
 //   3. complete the Authentik login on the redirect,
 //   4. assert we return to the app authenticated.
 const USER = process.env.AK_ADMIN_USER ?? 'akadmin';
 const PASS = process.env.AK_ADMIN_PASSWORD;
 for (const t of OAUTH_TARGETS) {
  test(`${t.name} SSO login via Authentik`, async ({ page }) => {
    test.skip(!PASS, 'AK_ADMIN_PASSWORD not set — provide via secret to run OAuth flows');
    await page.goto(url(t), { waitUntil: 'domcontentloaded' });
    // Click the SSO control.
    const b = t.ssoButton;
    const btn = b.selector
      ? page.locator(b.selector).first()
      : page.getByRole((b.role as any) ?? 'link', { name: b.name! }).first();
    await expect(btn, `${t.name}: SSO login control not found`).toBeVisible();
    await btn.click();
    // We should be redirected to the Authentik domain (or already have a session).
    await page.waitForLoadState('domcontentloaded');
    if (page.url().includes(`.${BASE_DOMAIN}`) && /authentik|\/if\/flow/i.test(page.url())) {
      await completeAuthentikLogin(page, USER, PASS!);
    } else {
      // Not obviously on Authentik — still attempt, in case creds render inline.
      await completeAuthentikLogin(page, USER, PASS!).catch(() => {});
    }
    // Back in the app, authenticated.
    const s = t.success;
    if (s.urlRe) await expect(page).toHaveURL(s.urlRe, { timeout: 25_000 });
    if (s.selector) await expect(page.locator(s.selector).first()).toBeVisible({ timeout: 25_000 });
    if (s.text) await expect(page.getByText(s.text).first()).toBeVisible({ timeout: 25_000 });
    await expect(page.getByText(/invalid|unauthorized|access denied|redirect_uri/i)).toHaveCount(0);
    await page.screenshot({ path: `test-results/oauth-${t.name}.png`, fullPage: true });
  });
 }
@@ -0,0 +1,14 @@
 import { test, expect } from '@playwright/test';
 import { TARGETS, url } from '../targets';
 const t = TARGETS.find((x) => x.name === 'portainer')!;
 // Portainer: prove the login UI renders over real ingress+TLS in Safari's engine.
 // (Full authenticated flow needs an initial-admin password; add once seeded.)
 test('portainer login page renders', async ({ page }) => {
  const resp = await page.goto(url(t), { waitUntil: 'domcontentloaded' });
  expect(resp?.status(), 'ingress did not serve portainer').toBeLessThan(400);
  await expect(page.locator('input[type="password"]').first()).toBeVisible();
  await page.screenshot({ path: 'test-results/portainer-login.png', fullPage: true });
 });
@@ -0,0 +1,33 @@
 import { test, expect } from '@playwright/test';
 import { TARGETS, url } from '../targets';
 // Core assertion the user asked for: once ingress-nginx is up and the app is
 // ready, NAME-APP.<domain> must be VIEWABLE in Safari's engine — not just return
 // a status code. We navigate, assert a real 2xx/3xx (not nginx 404/503 default
 // backend), and assert the app's own UI actually rendered.
 // Only apps without a dedicated flow spec (auth logins live in their own file).
 const smokeTargets = TARGETS.filter((t) => !t.dedicated);
 for (const t of smokeTargets) {
  test(`${t.name} is viewable at ${url(t)}`, async ({ page }) => {
    const resp = await page.goto(url(t), { waitUntil: 'domcontentloaded' });
    expect(resp, 'no response from ingress').toBeTruthy();
    const status = resp!.status();
    expect(status, `unexpected HTTP status ${status}`).toBeLessThan(400);
    // Not the nginx default backend / error page.
    await expect(page.locator('body')).not.toContainText(/default backend - 404|503 Service Temporarily/i);
    // App's own UI rendered.
    const r = t.ready;
    if (r.selector) {
      await expect(page.locator(r.selector).first()).toBeVisible();
    } else if (r.role && r.name) {
      await expect(page.getByRole(r.role as any, { name: r.name }).first()).toBeVisible();
    } else if (r.text) {
      await expect(page.getByText(r.text).first()).toBeVisible();
    }
  });
 }