diff --git a/.forgejo/CI-CD.md b/.forgejo/CI-CD.md deleted file mode 100644 index fdc434a..0000000 --- a/.forgejo/CI-CD.md +++ /dev/null @@ -1,460 +0,0 @@ -# CI/CD Pipeline: GitOps Validation & Deployment - -## Overview - -Pure GitOps CI/CD pipeline using Forgejo Actions (self-hosted runner). - -**Principle:** Validate in CI, deploy via ArgoCD (no manual steps). - -``` -git push - ↓ - [CI: Validate] - ├─ yamllint (YAML syntax) - ├─ kubeval (K8s manifests) - ├─ kustomize build (all layers) - ├─ argocd validation (app definitions) - └─ security scan (secrets, best practices) - ↓ - [If push to main] - └─ ArgoCD auto-syncs (if enabled) -``` - -## Workflows - -### 1. validate-k8s.yaml (Mandatory) - -**Trigger:** Any push/PR with k8s/ changes - -**What it does:** -1. Lints all YAML files (`yamllint`) -2. Validates K8s manifests (`kubeval`) -3. Builds all kustomization layers -4. Validates ArgoCD applications -5. Reports results - -**Duration:** ~2-3 minutes - -**Status:** -- ✅ PASS: All layers build, manifests valid → OK to merge -- ❌ FAIL: Syntax error, invalid resource, build failed → Fix & push again - -**Example output:** -``` -=== Building k8s/infrastructure/ === -✓ Infrastructure built successfully -Resources: 47 - -=== Building k8s/bootstrap/ === -✓ Bootstrap built successfully -Resources: 23 -``` - -**When to check:** -- After every commit -- Before merging PRs -- On every branch - -### 2. argocd-sync.yaml (Recommended) - -**Trigger:** Push to main only (k8s/ changed) - -**What it does:** -1. Authenticates with ArgoCD -2. Syncs `homelab-root` application -3. Waits for sync to complete (5 min timeout) -4. Verifies all applications healthy - -**Duration:** 1-5 minutes (depends on resources) - -**Status:** -- ✅ SYNCED: All resources deployed to cluster -- ❌ FAILED: Sync error, pod crashes, etc. → Check ArgoCD UI for details - -**When it runs:** -- Automatically after merge to main -- Only on k8s/ changes (not on docs) - -**Manual trigger (if needed):** -```bash -# SSH to runner or use Forgejo UI -# Re-run failed workflow -# Or manually sync: argocd app sync homelab-root -``` - -**Requires secrets:** -- `ARGOCD_SERVER`: ArgoCD server URL (https://argocd.riotpiao.com) -- `ARGOCD_AUTH_TOKEN`: ArgoCD API token (generate via ArgoCD UI) - -### 3. security-scan.yaml (Optional) - -**Trigger:** Any push/PR with k8s/ changes - -**What it does:** -1. Scans Dockerfiles for vulnerabilities (`trivy`) -2. Scans Helm charts for security issues -3. Audits K8s manifests (`polaris`) -4. Checks for hardcoded secrets -5. Verifies security best practices - -**Duration:** ~3-5 minutes - -**Status:** -- ✅ PASS: No critical issues -- ⚠️ WARNING: Best practice recommendations (non-blocking) -- ❌ FAIL: Hardcoded secrets found (must fix) - -**Common issues:** -- Missing resource limits (warning) -- Privileged containers (warning) -- Hardcoded passwords (ERROR) - ---- - -## File Structure - -``` -.forgejo/ -├── workflows/ # CI/CD workflows -│ ├── validate-k8s.yaml # Validate manifests (required) -│ ├── argocd-sync.yaml # Sync to cluster (auto on main) -│ └── security-scan.yaml # Security checks (optional) -└── CI-CD.md # This file -``` - ---- - -## Setup Instructions - -### 1. Install Forgejo Runner - -```bash -# On runner machine (inside cluster or external) -forgejo-runner register \ - --instance https://forgejo.riotpiao.com \ - --token \ - --name homelab-runner \ - --labels docker - -forgejo-runner daemon -``` - -### 2. Add ArgoCD Secrets to Forgejo - -```bash -# Go to: Forgejo → Settings → Secrets - -# Add: -ARGOCD_SERVER = https://argocd.riotpiao.com -ARGOCD_AUTH_TOKEN = # Generate: argocd account generate-token -``` - -### 3. Generate ArgoCD Token - -```bash -# Inside cluster -kubectl -n argocd port-forward svc/argocd-server 8080:443 - -# Go to: https://localhost:8080/user-info/api-tokens -# Create new token (CI/CD) -# Copy token to Forgejo secrets -``` - ---- - -## Workflow Execution - -### When developer pushes to feature branch: - -``` -git push origin feature/new-service - -↓ -Forgejo Actions triggered -↓ -validate-k8s.yaml runs: - ✓ Lints YAML - ✓ Validates manifests - ✓ Builds kustomizations - ✓ All pass → GitHub comment: "Ready to merge" -↓ -Developer opens PR -↓ -Reviewer checks: - - Code changes (YAML) - - Workflow results - - ArgoCD impact (diff) -↓ -PR merged to main -``` - -### When merged to main: - -``` -git merge feature/new-service → main - -↓ -Forgejo Actions triggered -↓ -validate-k8s.yaml runs: - ✓ Same validation as above -↓ -argocd-sync.yaml runs (if enabled): - ✓ Syncs homelab-root - ✓ Waits for sync - ✓ Verifies health - ✓ Resources deployed to cluster -↓ -Cluster state = git state -(No manual kubectl apply needed!) -``` - ---- - -## Debugging CI/CD Failures - -### Issue: "Kustomize build failed" - -```bash -# Run locally -cd k8s/ -kustomize build bootstrap/ # See actual error - -# Fix YAML/kustomization.yaml -# git push again -``` - -### Issue: "Kubeval validation failed" - -```bash -# Check K8s manifest syntax -kubeval k8s/platform/minio/config.yaml - -# Common issues: -# - Typos in apiVersion, kind, metadata -# - Missing required fields -# - Invalid references (namespace, service name) -``` - -### Issue: "ArgoCD sync failed" - -```bash -# Check ArgoCD UI -# https://argocd.riotpiao.com → homelab-root - -# Or CLI -argocd app get homelab-root -argocd app logs homelab-root --follow - -# Common issues: -# - Missing namespace (fixed by infrastructure layer) -# - Invalid Helm chart version -# - Secret not found -# - Network policy blocking traffic -``` - -### Issue: "Security scan found hardcoded secret" - -```bash -# Fix: Remove secret from YAML -# Add to SOPS encryption instead - -# Or use ArgoCD Sealed Secrets -# (if SOPS not available) -``` - ---- - -## Viewing Results - -### Forgejo Actions UI - -``` -Repository → Actions - ├─ validate-k8s - │ ├─ ✅ Success (merge safe) - │ ├─ ❌ Failed (fix required) - │ └─ Logs (click "Steps" → "Summary") - ├─ argocd-sync - │ ├─ ✅ Synced (deployed) - │ └─ ❌ Failed (check ArgoCD UI) - └─ security-scan - ├─ ✅ Pass (no critical issues) - └─ ⚠️ Warning (review, non-blocking) -``` - -### ArgoCD UI - -``` -https://argocd.riotpiao.com - ├─ homelab-root - │ ├─ Status: Synced ✓ - │ ├─ Health: Healthy ✓ - │ └─ Details (click to see resources) - ├─ layer-1-bootstrap - ├─ layer-2-platform - ├─ layer-3-security - ├─ layer-4-applications - └─ layer-5-data -``` - ---- - -## Common Tasks - -### Add new service to cluster - -```bash -# 1. Create directory and kustomization.yaml -mkdir -p k8s/applications/my-service -cat > k8s/applications/my-service/kustomization.yaml << EOF -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization -namespace: my-namespace -helmCharts: -- name: my-chart - repo: https://charts.example.com - version: 1.0.0 - releaseName: my-service - valuesFile: values.yaml -EOF - -# 2. Add values.yaml -cp /template/values.yaml k8s/applications/my-service/ - -# 3. Commit and push -git add k8s/applications/my-service/ -git commit -m "feat(apps): add my-service" -git push - -# 4. CI validates -# 5. Merge to main -# 6. ArgoCD syncs automatically -# ✓ Service deployed to cluster -``` - -### Rollback a deployment - -```bash -# 1. Find broken commit -git log --oneline k8s/ # Identify bad commit - -# 2. Revert -git revert -git push - -# 3. CI validates (should pass) -# 4. Merge to main -# 5. ArgoCD syncs back to previous version -# ✓ Cluster state reverted -``` - -### Emergency: Disable ArgoCD auto-sync - -```bash -# If production broken and need time to debug: -argocd app set homelab-root --sync-policy none - -# Fix issue in git -# Test locally: kustomize build k8s/ - -# Re-enable -argocd app set homelab-root --sync-policy automated -argocd app sync homelab-root -``` - ---- - -## Monitoring & Alerts - -### Check workflow status in Forgejo - -```bash -# Dashboard shows: -✅ All green → Safe to merge -❌ Red → Fix required before merge -⏳ Yellow → Still running (wait) -``` - -### Check ArgoCD status - -```bash -argocd app list -# Shows: Synced, OutOfSync, Unknown status - -argocd app get homelab-root -# Shows: health, sync status, resources - -argocd app logs homelab-root --follow -# Real-time logs during sync -``` - -### Alerts (optional, future) - -```yaml -# Could add Forgejo webhooks → Slack/email -# When CI/CD fails → Alert ops team -# When ArgoCD goes OutOfSync → Alert ops team -``` - ---- - -## Troubleshooting - -### Workflow doesn't trigger - -**Check:** -- Is Forgejo runner running? `forgejo-runner daemon` -- Did you push to correct branch? (validate runs on all, argocd-sync only on main) -- Did path match filter? (must change k8s/ or .forgejo/workflows/) - -### Workflow hangs/times out - -**Check:** -- kustomize build → Check for dependency cycles -- argocd sync → Check cluster resources (storage full? network down?) -- security scan → Large image scan → Takes time - -**Fix:** -- Increase timeout in workflow -- Optimize kustomization (remove unused resources) -- Add resource limits to pods - -### ArgoCD token invalid - -**Fix:** -```bash -# Regenerate token -argocd account generate-token - -# Update Forgejo secret -# Settings → Secrets → ARGOCD_AUTH_TOKEN = -``` - ---- - -## Best Practices - -✅ **DO:** -- Commit all K8s changes to git (no manual kubectl apply) -- Run validate-k8s locally before push -- Write descriptive commit messages (why this change?) -- Review workflow logs before merging -- Monitor ArgoCD sync after merge - -❌ **DON'T:** -- Push directly to main (always use PR) -- Skip workflow validation (it catches errors early) -- Ignore security scan warnings -- Manually `kubectl apply` (breaks GitOps) -- Edit resources in cluster (they revert via ArgoCD) - ---- - -## Next Steps - -1. **Setup Forgejo runner** (if not already running) -2. **Add ArgoCD secrets** to Forgejo -3. **Test workflows** on feature branch -4. **Merge to main** → Watch ArgoCD sync -5. **Celebrate:** Full GitOps pipeline working! 🎉 diff --git a/k8s_authentik-migration.md b/k8s_authentik-migration.md deleted file mode 100644 index 5702e65..0000000 --- a/k8s_authentik-migration.md +++ /dev/null @@ -1,270 +0,0 @@ -# Kubernetes RBAC → Authentik Migration Plan - -**Status:** Draft — awaiting decisions in [Open Decisions](#open-decisions) -**Date:** 2026-08-19 -**Scope:** Move human identity + authorization for the cluster and its apps onto Authentik groups. - ---- - -## 1. Finding: there is nothing to literally import - -The original ask was "import the existing RBAC from the cluster into Authentik". Inventory of the live -cluster says that set is empty. - -### 1.1 Cluster RBAC audit (live, 2026-08-19) - -Every `ClusterRoleBinding` / `RoleBinding` carrying a `User` or `Group` subject: - -| Binding | Role | Subjects | -|---|---|---| -| `cluster-admin` | `ClusterRole/cluster-admin` | `Group:system:masters` | -| `system-bootstrap-approve-node-client-csr` | `ClusterRole/system:certificates.k8s.io:certificatesigningrequests:nodeclient` | `Group:system:bootstrappers:nodes` | -| `system-bootstrap-node-bootstrapper` | `ClusterRole/system:node-bootstrapper` | `Group:system:bootstrappers:nodes`, `Group:system:nodes` | -| `system-bootstrap-node-renewal` | `ClusterRole/system:certificates.k8s.io:certificatesigningrequests:selfnodeclient` | `Group:system:nodes` | -| `kube-system/system::extension-apiserver-authentication-reader` | `Role/extension-apiserver-authentication-reader` | `User:system:kube-controller-manager`, `User:system:kube-scheduler` | -| `kube-system/system::leader-locking-kube-controller-manager` | `Role/…` | `User:system:kube-controller-manager` | -| `kube-system/system::leader-locking-kube-scheduler` | `Role/…` | `User:system:kube-scheduler` | - -All of these are Talos/control-plane machinery. Every other binding in the cluster targets a -`ServiceAccount` (controllers: Argo CD, cert-manager, Cilium, Longhorn, MinIO operator, Prometheus, …). - -**There are zero human users in Kubernetes RBAC.** The only human access path today is the Talos-issued -admin certificate in `cluster-config/kubeconfig`, which lands in `system:masters` — a single shared, -unattributable, non-revocable god credential. - -The kube-apiserver has no OIDC configured. `apiServer:` in -[`terraform/templates/controlplane.tftpl:110`](terraform/templates/controlplane.tftpl#L110) contains only -`certSANs`, `image`, `admissionControl` and `auditPolicy`. - -### 1.2 What human RBAC *does* exist (app level, scattered) - -| Location | Rule | -|---|---| -| [`k8s/bootstrap/phase4-argocd/argocd-values.yaml:163-169`](k8s/bootstrap/phase4-argocd/argocd-values.yaml#L163-L169) | `policy.default: role:readonly`; `g, admin/rock/cicd, role:admin`; `g, homelab-admins, role:admin` | -| [`k8s/infra/logging/grafana-values.yaml:86`](k8s/infra/logging/grafana-values.yaml#L86) | `role_attribute_path: contains(groups[*], 'grafana-admins') && 'Admin' \|\| 'Viewer'` | -| [`k8s/infra/iam/scripts/authentik-provision.py`](k8s/infra/iam/scripts/authentik-provision.py) (`minio` scope mapping) | `homelab-admins → consoleAdmin`, everyone else → `readonly` | -| Authentik, provisioned | Groups `homelab-admins` (superuser), `grafana-admins`. Single user: `rock`, member of both. | - -### 1.3 Known drift - -[`project-usage/authentik-oidc.md`](project-usage/authentik-oidc.md) documents three groups — -`homelab-admins`, `homelab-devops`, `homelab-viewers`. Only `homelab-admins` and `grafana-admins` are -actually created by the provisioner. The documentation is wrong today; Phase 1 fixes it. - -### 1.4 Restated goal - -Not an import. This is: **define the group taxonomy in Authentik, make Authentik the sole issuer of human -identity, then bind those groups to Kubernetes roles and app roles declaratively through GitOps.** - ---- - -## 2. Constraints - -- **GitOps only.** No local `terraform apply`, no manual `kubectl apply`, no Authentik console clicking for - anything that must survive a rebuild. Changes land in git; Forgejo Actions and Argo CD do the applying. -- **Authentik config stays Argo CD-managed** via the `authentik-provision.py` PostSync hook. No Terraform - Authentik provider. A prior attempt at one is abandoned at - `.claude/worktrees/debug-auth-issue/terraform/authentik-config.tf` — do not revive it. -- **Terraform is limited to Talos machine config** (Phase 3 only), applied through the pipeline. -- **The Talos admin certificate stays as break-glass.** It is unaffected by OIDC and remains the recovery - path when Authentik is unavailable. - ---- - -## 3. Phases - -Each phase is independently shippable and independently revertible. - -### Phase 1 — Group taxonomy - -Extend the provisioner's group step (`[2/5]` in `authentik-provision.py`) from two groups to four: - -| Group | `is_superuser` | Intent | -|---|---|---| -| `homelab-admins` | `true` | Full cluster + full app admin | -| `homelab-devops` | `false` | Deploy and operate workloads; no IAM, no cluster config | -| `homelab-viewers` | `false` | Read-only everywhere | -| `grafana-admins` | `false` | Grafana `Admin` role specifically | - -Files: `k8s/infra/iam/scripts/authentik-provision.py`, `project-usage/authentik-oidc.md` (fix §1.3 drift). - -**Verify** -```bash -kubectl -n iam exec deploy/authentik-server -- \ - curl -sH "Authorization: Bearer $TOKEN" localhost:9000/api/v3/core/groups/ \ - | jq -r '.results[].name' -# expect: homelab-admins, homelab-devops, homelab-viewers, grafana-admins -# and: rock still a member of homelab-admins + grafana-admins -``` - -**Rollback:** revert the commit. Extra groups with no bindings grant nothing. - ---- - -### Phase 2 — Authentik OAuth2 provider for Kubernetes - -Add a `kubernetes` entry to the `SERVICES` dict in `authentik-provision.py`, with two departures from the -existing web-app entries: - -- `client_type: public` — kubectl is a public client and cannot hold a secret. No k8s Secret is created for - it, so the `client_secret_source` machinery is skipped for this service. -- `grant_types: ["authorization_code", "refresh_token", "urn:ietf:params:oauth:grant-type:device_code"]` - plus PKCE, to support both browser and headless login. -- `redirect_uris`: `http://localhost:8000`, `http://localhost:18000` (kubelogin's loopback listeners). - -Reuses the existing `groups` property mapping created in step `[1/5]`. No new claim work. - -**Verify** -```bash -curl -s https://authentik.riotpiao.com/application/o/kubernetes/.well-known/openid-configuration | jq . - -# device-code login, then decode: -kubectl oidc-login get-token --oidc-issuer-url=... --oidc-client-id=kubernetes --oidc-extra-scope=groups -# id_token payload must contain: preferred_username, groups[] -``` - -**Rollback:** delete provider + application in Authentik; nothing else consumes them yet. - ---- - -### Phase 3 — Enable OIDC on kube-apiserver (Terraform, pipeline-applied) - -Add to `apiServer.extraArgs` in -[`terraform/templates/controlplane.tftpl`](terraform/templates/controlplane.tftpl#L110): - -```yaml - extraArgs: - oidc-issuer-url: https://authentik.riotpiao.com/application/o/kubernetes/ - oidc-client-id: kubernetes - oidc-username-claim: preferred_username - oidc-username-prefix: "authentik:" - oidc-groups-claim: groups - oidc-groups-prefix: "authentik:" -``` - -> **Security — the prefixes are mandatory, not cosmetic.** -> Without `oidc-groups-prefix`, anyone able to create or rename an Authentik group to `system:masters` -> obtains `cluster-admin` on the cluster. Without `oidc-username-prefix`, an Authentik username can -> impersonate a built-in identity such as `system:kube-controller-manager`. Both prefixes must be present -> in the same change that enables OIDC. Do not ship this phase partially. - -Operational notes: - -- Apply is a rolling control-plane machine-config change: one node at a time, wait for `Ready` and for - etcd quorum before proceeding to the next. -- The Talos admin certificate authenticates via client cert, not OIDC, and is unaffected. It is the - recovery path if the issuer URL is wrong or Authentik is down. -- The issuer must be reachable from the control-plane nodes and its TLS chain must be trusted by them. - Confirm this before applying, because a bad issuer URL means the apiserver logs OIDC discovery failures - on every start. - -**Verify** (per node, after each rolls) -```bash -kubectl auth whoami --token="$ID_TOKEN" -# expect Username: authentik:rock -# Groups: authentik:homelab-admins, authentik:grafana-admins, system:authenticated - -kubectl --kubeconfig cluster-config/kubeconfig get nodes # cert path still works -``` - -**Rollback:** revert the template change, re-run the pipeline, roll the control plane back. Cert-based -access is never interrupted, so this rollback is safe at any point. - ---- - -### Phase 4 — Bind Authentik groups to Kubernetes roles - -New manifest `k8s/infra/iam/rbac-oidc-bindings.yaml`, referenced from -[`k8s/infra/iam/kustomization.yaml`](k8s/infra/iam/kustomization.yaml): - -| Subject | Role | Kind | -|---|---|---| -| `Group:authentik:homelab-admins` | `ClusterRole/cluster-admin` | ClusterRoleBinding | -| `Group:authentik:homelab-devops` | `ClusterRole/edit` | see [Open Decisions](#open-decisions) #2 | -| `Group:authentik:homelab-viewers` | `ClusterRole/view` | ClusterRoleBinding | - -`admin`, `edit` and `view` are the built-in aggregated ClusterRoles and already exist in the cluster — no -custom roles needed. - -**Verify** -```bash -kubectl auth can-i '*' '*' --all-namespaces --as=probe --as-group=authentik:homelab-admins # yes -kubectl auth can-i delete pods -A --as=probe --as-group=authentik:homelab-viewers # no -kubectl auth can-i get pods -n apps --as=probe --as-group=authentik:homelab-viewers # yes -kubectl auth can-i get secrets -n iam --as=probe --as-group=authentik:homelab-devops # no -``` - -**Rollback:** revert the commit; Argo CD prunes the bindings. - ---- - -### Phase 5 — Client wiring and documentation - -- Ship a kubeconfig template using the `kubelogin` (`kubectl oidc-login`) exec credential plugin — no - embedded certs, no long-lived token on disk. -- Document login, token cache location, and the break-glass cert path in - [`project-usage/authentik-oidc.md`](project-usage/authentik-oidc.md). -- Same edit corrects the stale group list from §1.3. - -**Verify:** on a clean machine with no certificates, `kubectl get nodes` triggers a browser login and -succeeds; `kubectl auth whoami` reports the `authentik:`-prefixed identity. - ---- - -### Phase 6 — Converge app RBAC on the same four groups - -| Target | Change | -|---|---| -| Argo CD `policy.csv` | add `g, homelab-devops, role:admin` (or a scoped custom role) and `g, homelab-viewers, role:readonly` | -| Grafana `role_attribute_path` | three-tier: `grafana-admins`/`homelab-admins` → `Admin`, `homelab-devops` → `Editor`, else `Viewer` | -| MinIO `policy` claim expression | `homelab-admins` → `consoleAdmin`, `homelab-devops` → `readwrite`, else `readonly` | -| Argo CD local accounts | retire `accounts.rock` once SSO admin is proven. **Keep `accounts.cicd`** — the pipeline needs a non-SSO apiKey. | - -**Verify:** end-to-end browser login per app as a test user who is *only* in `homelab-devops`, confirming -the expected role in each of Argo CD, Grafana and MinIO. The e2e suite under -[`tests/e2e/tests/authentik.spec.ts`](tests/e2e/tests/authentik.spec.ts) is the place to encode this. - -**Rollback:** per-app revert; each app's mapping is independent. - ---- - -## 4. Risks - -| Risk | Severity | Mitigation | -|---|---|---| -| Missing `oidc-groups-prefix` → group-name privilege escalation to `cluster-admin` | Critical | Prefixes ship in the same commit as the OIDC flags; Phase 4 verification asserts the prefixed form | -| Bad issuer URL / untrusted TLS bricks apiserver OIDC | Medium | Cert-based admin access is unaffected; roll one control-plane node at a time and check apiserver logs before continuing | -| Authentik outage blocks all human cluster access | Medium | Talos admin cert is the documented break-glass path and never depends on Authentik | -| Argo CD `cicd` account removed by accident → pipeline loses cluster access | Medium | Phase 6 explicitly retires only `accounts.rock` | -| Group renamed in Authentik silently revokes cluster access | Low | Group names are asserted by the provisioner on every Argo CD sync, so drift self-heals | - ---- - -## 5. Open Decisions - -1. **Is kubectl-via-OIDC actually wanted?** - Phases 3–5 only pay for themselves if humans besides the shared admin cert need API access. If the - answer is no, the plan collapses to Phases 1 and 6: no Terraform change, no control-plane roll, app-level - SSO consolidation only. - -2. **`homelab-devops` blast radius** — cluster-wide `ClusterRoleBinding` to `edit`, or namespaced - `RoleBinding`s limited to application namespaces (excluding `iam`, `kube-system`, `argocd`)? - Namespaced is the tighter default; cluster-wide is less to maintain. - -3. **Username claim** — `preferred_username` (human-readable in audit logs, mutable) versus `sub` - (stable, opaque). Recommendation: `preferred_username`, given a single-operator homelab where audit - readability beats rename-safety. - ---- - -## 6. Reference - -- Kubernetes OIDC authentication: https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens -- Kubernetes RBAC: https://kubernetes.io/docs/reference/access-authn-authz/rbac/ -- Talos apiserver `extraArgs`: https://www.talos.dev/latest/reference/configuration/v1alpha1/config/#Config.cluster.apiServer -- Authentik OAuth2 provider: https://docs.goauthentik.io/docs/add-secure-apps/providers/oauth2/ -- Authentik property mappings (scope/claims): https://docs.goauthentik.io/docs/add-secure-apps/providers/property-mappings/ -- kubelogin (`kubectl oidc-login`): https://github.com/int128/kubelogin -- Argo CD RBAC: https://argo-cd.readthedocs.io/en/stable/operator-manual/rbac/ -- Grafana generic OAuth: https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-authentication/generic-oauth/ -- MinIO OpenID identity management: https://min.io/docs/minio/linux/administration/identity-access-management/oidc-access-management.html diff --git a/project-usage/authentik-oidc.md b/project-usage/authentik-oidc.md deleted file mode 100644 index 98a5578..0000000 --- a/project-usage/authentik-oidc.md +++ /dev/null @@ -1,183 +0,0 @@ -# Authentik Federated OIDC & SSO - -**Provider:** `https://authentik.riotpiao.com` -**OIDC Issuer:** `https://authentik.riotpiao.com/application/o/talos-federation/` -**Namespace:** `iam` - -## When to Use - -- **Federated login** — Single sign-on for Grafana, MinIO, Forgejo, Argo CD -- **User groups** — RBAC via group membership (admins, devops, read-only) -- **JWT tokens** — Authenticate CLI tools, API clients -- **SSO for custom apps** — OAuth2/OIDC redirect flow - -## Quick Start - -**1. Login to Authentik console:** -```bash -# Browser: https://authentik.riotpiao.com -# Default user: akadmin -# Password: AUTHENTIK_BOOTSTRAP_PASSWORD (from .env) - -# Or via OIDC (after initial setup) -# Click "Sign in with talos-federation" -``` - -**2. Create user:** -``` -Authentik console → Users → Create -- Username: alice -- Email: alice@example.com -- Group: homelab-devs (or homelab-admins) -``` - -**3. User logs into Grafana:** -``` -https://grafana.riotpiao.com -→ Sign in with Authentik (auto-redirects to OIDC provider) -→ Approve access -→ Logged in as alice (group determines role: Admin or Viewer) -``` - -## Configuration - -| Key | Value | -|-----|-------| -| OIDC provider | `talos-federation` (federated) | -| OIDC issuer | `https://authentik.riotpiao.com/application/o/talos-federation/` | -| JWKS endpoint | `https://authentik.riotpiao.com/application/o/talos-federation/.well-known/openid-configuration` | -| Database | PostgreSQL (ddb namespace, authentik user) | -| Backups | WAL archived to MinIO | - -## Common Patterns - -**Grafana OIDC login:** -```yaml -# k8s/logging/grafana-values.yaml -grafana: - auth.generic_oauth: - enabled: true - name: Authentik - client_id: grafana - client_secret: $GRAFANA_OIDC_CLIENT_SECRET # from Vault - auth_url: https://authentik.riotpiao.com/application/o/authorize/ - token_url: https://authentik.riotpiao.com/application/o/token/ - api_url: https://authentik.riotpiao.com/application/o/userinfo/ - scopes: openid profile email groups - use_pkce: true -``` - -**MinIO OIDC login:** -```yaml -# k8s/storage/minio-values.yaml -minio: - identity_oauth: - provider: authentik - client_id: minio - client_secret: $MINIO_OIDC_CLIENT_SECRET - redirect_uri: https://minio.riotpiao.com/oauth_callback - config_url: https://authentik.riotpiao.com/application/o/talos-federation/.well-known/openid-configuration - policy_mappings: - - group: homelab-admins → consoleAdmin - - group: homelab-devops → readwrite -``` - -**CLI device code flow (core CLI):** -```bash -# Get JWT token (no kubeconfig needed) -core secrets login -# → Opens browser, approve device code -# → Token cached in ~/.core/token - -# Use token to access Vault -core get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY -# → Vault validates JWT from Authentik -# → Returns secret -``` - -**Custom app OIDC redirect:** -```go -import "github.com/coreos/go-oidc/v3/oidc" - -provider, _ := oidc.NewProvider(ctx, "https://authentik.riotpiao.com/application/o/talos-federation/") - -verifier := provider.Verifier(&oidc.Config{ClientID: "my-app"}) - -// After OAuth2 redirect & token exchange: -idToken, _ := verifier.Verify(ctx, rawIDToken) - -// Extract claims -var claims struct { - Email string `json:"email"` - Groups []string `json:"groups"` -} -idToken.Claims(&claims) -``` - -## Group-Based RBAC - -**Default groups:** -- `homelab-admins` — Full cluster access (Grafana Admin, MinIO admin, Argo CD admin, Vault admin) -- `homelab-devops` — Deploy & monitor (Grafana Editor, MinIO readwrite, Argo CD user) -- `homelab-viewers` — Read-only (Grafana Viewer, MinIO readonly) - -**Assign user to group:** -``` -Authentik console → Users → alice → Edit -→ Groups → Add "homelab-devops" -→ Save -``` - -**Custom group-to-role mapping:** -```yaml -# Per-service (see cicd-workflow.md, monitoring-metrics.md for examples) -# Grafana: auth.generic_oauth.role_attribute_path = contains(groups[*], 'homelab-admins') && 'Admin' || 'Viewer' -# MinIO: policy_mappings (see above) -``` - -## Monitoring - -**Authentik dashboard:** https://authentik.riotpiao.com/api/v3/admin/dashboards - -**Key metrics:** -- Login attempts (success/failure) -- Active sessions -- Token issuance rate -- Provider sync status - -## Troubleshooting - -**Users can't login (redirect loop):** -```bash -# Check redirect URI matches -# Authentik console → Applications → grafana → Edit -# Verify Redirect URI = https://grafana.riotpiao.com/login/generic_oauth - -# Check OIDC provider is running -k get pods -n iam -l app=authentik -``` - -**JWT token expired:** -```bash -# CLI tokens have 24h expiry -# Re-authenticate -core secrets login -``` - -**Groups not syncing:** -```bash -# Check group attribute in OIDC config -# Authentik console → Applications → → OIDC Configuration -# groups_attribute = "groups" (or custom claim name) -``` - -**Vault can't validate JWT:** -```bash -# Verify JWKS endpoint is accessible -curl https://authentik.riotpiao.com/application/o/talos-federation/.well-known/openid-configuration - -# Restart Vault to refresh JWKS cache -k rollout restart -n iam deployment/vault -``` - -See `/TROUBLESHOOTING.md` for full incident guide. diff --git a/project-usage/cicd-workflow.md b/project-usage/cicd-workflow.md deleted file mode 100644 index 3b13781..0000000 --- a/project-usage/cicd-workflow.md +++ /dev/null @@ -1,222 +0,0 @@ -# CI/CD Pipeline (Forgejo + Argo CD) - -**Git Forge:** `https://forgejo.riotpiao.com` -**Deployments:** `https://argocd.riotpiao.com` (or `kubectl port-forward`) -**Namespaces:** `cicd`, `forge` - -## When to Use - -- **Build & test** — Forgejo Actions CI (GitHub Actions syntax) -- **Image push** — Build OCI images, push to Forgejo registry -- **GitOps deployment** — Argo CD syncs deploy repo to cluster -- **Secrets in CI** — ci-bot JWT tokens, never kubeconfig - -## Quick Start - -**1. Clone a repo from Forgejo:** -```bash -git clone https://forgejo.riotpiao.com/rock/source.git -cd source -``` - -**2. Create workflow:** -```bash -mkdir -p .forgejo/workflows -cat > .forgejo/workflows/ci.yml < deployment/ - -# Or check Argo CD UI -kubectl port-forward -n argocd svc/argocd-server 8443:443 -# https://localhost:8443 (login via Authentik) -``` - -## Workflow Syntax (GitHub Actions) - -**Basic structure:** -```yaml -name: CI -on: - push: - branches: [main, develop] - pull_request: - -jobs: - test: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v3 - - run: npm ci - - run: npm test - - build: - needs: test # wait for test job - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v3 - - run: docker build -t myapp:${{ github.sha }} . - - run: docker push forgejo.riotpiao.com/rock/myapp:${{ github.sha }} -``` - -**Available variables:** -```bash -${{ github.sha }} # commit hash -${{ github.ref_name }} # branch name -${{ github.run_number }} # build # -${{ secrets.CI_BOT_TOKEN }} # injected from Forgejo -``` - -## Secrets & Authentication - -**ci-bot JWT token (auto-injected):** -```yaml -- run: | - echo "${{ secrets.CI_BOT_TOKEN }}" | docker login \ - forgejo.riotpiao.com \ - -u ci-bot \ - --password-stdin - docker push forgejo.riotpiao.com/rock/myapp:latest -``` - -**API token (for pushing commits):** -```yaml -- run: | - git config user.name "ci-bot" - git config user.email "ci-bot@homelab" - git commit --allow-empty -m "bump: version" - git push https://ci-bot:${{ secrets.CI_BOT_TOKEN }}@forgejo.riotpiao.com/rock/deploy.git main -``` - -**Vault secrets (via talos CLI):** -```bash -# Not available in CI runner — use Argo CD post-sync hooks instead -# Or inject via init container before workflow runs -``` - -## Argo CD GitOps - -**Create app (one-time):** -```bash -argocd app create story-crater \ - --repo https://forgejo.riotpiao.com/rock/deploy.git \ - --path k8s/ \ - --dest-server https://kubernetes.default.svc \ - --dest-namespace story-crater-backend \ - --sync-policy automated -``` - -**Monitor sync:** -```bash -# CLI -argocd app get story-crater -argocd app logs story-crater - -# UI: https://argocd.riotpiao.com -# Login: Authentik SSO (homelab-admins group only) -``` - -**Manual sync:** -```bash -argocd app sync story-crater -argocd app wait story-crater -``` - -## Security Rules - -✅ **DO:** -- Store credentials in Forgejo Secrets (auto-injected) -- Use ci-bot JWT for image push only -- Commit to deploy repo (triggers Argo CD) -- Enable branch protection (require CI pass) - -❌ **DON'T:** -- Put kubeconfig in CI (Argo CD bridges gap) -- Commit secrets to source repo -- Use admin-bot in CI workflows (over-privileged) -- Push images directly to cluster (use Argo CD) - -## Monitoring - -**Grafana dashboard:** `svc-forgejo`, `svc-argocd` - -**Key metrics:** -- `forgejo_workflows_running` — active workflows -- `argocd_app_sync_duration_seconds` — deployment time -- `argocd_app_info{sync_status="OutOfSync"}` — drift detection - -## Troubleshooting - -**Workflow fails silently:** -```bash -# Check runner logs -k logs -n cicd -f deploy/forgejo-runner - -# Check if runner pod is healthy -k get pods -n cicd -l app=forgejo-runner -``` - -**Image push fails (401 Unauthorized):** -```bash -# Verify ci-bot token in Forgejo -# Settings → Applications → ci-bot → check scopes (package:write) - -# Or re-create token -core put cluster/iam/agents/ci-bot-token TOKEN="$(openssl rand -hex 32)" -``` - -**Argo CD out-of-sync:** -```bash -# Check deploy repo changes -argocd app diff story-crater - -# Manual sync -argocd app sync story-crater --prune -``` - -**Webhook not triggering:** -```bash -# Verify Forgejo webhook config -# Repo Settings → Webhooks → Check delivery logs - -# Or manually trigger -argocd app sync story-crater -``` - -See `/TROUBLESHOOTING.md` for full incident guide. diff --git a/project-usage/coding-standards.md b/project-usage/coding-standards.md deleted file mode 100644 index d26ced2..0000000 --- a/project-usage/coding-standards.md +++ /dev/null @@ -1,388 +0,0 @@ -# Coding Standards — Helm, Helmfile, Kubernetes, Shell - -Conventions for contributing to homelab's Helm charts, helmfiles, Kubernetes manifests, and shell scripts. These are repo-local patterns — not Go/Rust/general standards from companion repos (core CLI, kmsvc). - ---- - -## Helm & Helmfile Conventions - -### Chart Naming & Layout - -- **Local charts:** `k8s//charts//` directory structure. -- **Helm chart versions:** Always use semver ranges in helmfile releases (e.g., `~1.0`, `~10`, not floating `latest`). - - Rationale: Predictable upgrades, avoids surprise breaking changes. - -### helmfile.yaml.gotmpl Pattern - -The helmfile is a **Go template**, not a shell script. Use Go template syntax for environment variable interpolation, not shell syntax. - -**Correct:** -```yaml -set: - - name: adminPassword - value: {{ env "GRAFANA_ADMIN_PASSWORD" }} -``` - -**Incorrect:** -```yaml -set: - - name: adminPassword - value: ${GRAFANA_ADMIN_PASSWORD} # Shell syntax — not expanded by helmfile -``` - -**Organization:** -- Use logical comment headers to separate sections: `# ── cert-manager ──`. -- Group related releases together. -- Use `needs:` for dependency ordering (release A waits for release B before deploying). - - Example (from helmfile.yaml.gotmpl, lines 439–449): - ```yaml - - name: loki - namespace: logging - needs: - - storage/minio - - - name: promtail - namespace: logging - needs: - - logging/loki - ``` -- **Namespace declaration:** Specify namespace at the release block level, not helmfile-level default. - ```yaml - - name: prometheus - namespace: monitoring - createNamespace: true - ``` - -### Values Files Pattern - -- **Never hardcode secrets** in values.yaml or ConfigMap keys. - - Store secrets in Vault via `core put cluster/KEY KEY="value"`. - - Reference at deploy time using `{{ env "VAR" }}` in helmfile. -- **External values files:** Always use `values:` block pointing to `.yaml` files, not inline YAML. - ```yaml - - name: minio - values: - - k8s/storage/minio-values.yaml - ``` -- **Environment-specific overrides:** For complex releases (e.g., SQS), use `environments/` subdirectory with `helmfile.yaml.gotmpl` at the service level. - - Example: `k8s/sqs/environments/homelab.yaml` (lines 1–29 show namespace, kafkaCluster, redis, managementService blocks). - - Helmfile loads environment-specific values dynamically: `{{ .Values.kafkaCluster.nodePool.replicas }}`. - ---- - -## Infrastructure as Code (IaC) — Single Source of Truth - -**Core principle:** All infrastructure state must be declaratively managed via Terraform or Helm (via helmfile + Terraform). No ad-hoc scripts, manual kubectl, or side-by-side resource definitions. - -### Terraform + Helm Division of Labor - -- **Terraform manages:** - - Helm releases (chart + version + values) - - Namespaces - - StorageClasses - - Static Kubernetes resources (RBAC, NetworkPolicies, IngressClasses) - - Cloud infrastructure (Vault, S3 backends, secrets) - - State persistence (S3 backend in MinIO) - -- **Helmfile manages:** - - Chart release ordering via `needs:` - - Environment-specific value interpolation (Go templating, not shell) - - Hook workflows (pre/post-sync orchestration) - - **Never use helmfile for one-off bucket creation, job runs, or manual setup** — those belong in Terraform or a documented bootstrap process - -- **Kubernetes manifests (`k8s/`) manage:** - - ArgoCD applications (single source of truth for GitOps) - - Service definitions that ArgoCD syncs - - **Never manage app objects (Deployments, StatefulSets) directly** — let helm + ArgoCD own them - -### Anti-Pattern: Ad-Hoc Resource Creation - -❌ **Bad:** Separate `minio-buckets.tf` using `aws_s3_bucket` resources + post-deploy scripts -- Split responsibility: some buckets in Terraform, others in helmfile, others manual -- State drift: unclear what's managed where -- Credential duplication: secrets in multiple places - -✅ **Good:** Single source in `minio.tf` helm release: -```hcl -buckets = [ - { name = "terraform-state", policy = "none", purge = false }, - { name = "vault", policy = "none", purge = false }, - ... -] -``` -- One place to define, one place to audit -- Credentials in variables + Vault, not scattered -- TF state tracks all changes - -### When to Break the Rule - -Only when **explicitly documented**: -- Bootstrap scripts (one-time cluster init) — commit to `scripts/` with clear "run once" warning -- Temporary debugging (never leave in git) — stash or delete before committing -- Manual steps for constraint (e.g., "create namespace before ArgoCD bootstraps") — document in `TROUBLESHOOTING.md` with rationale - ---- - -## YAML & ConfigMap/Secret Patterns - -### Secret Field Naming - -**Rule: Field name in Secret = environment variable name in Vault.** - -When storing a secret via `core put cluster/KEY KEY="value"`, the field name and Vault variable name must match. This ensures helmfile's `{{ env "VAR" }}` expansion works correctly. - -Example (from CLAUDE.md gotcha "Field name = variable name"): -```bash -# Correct -core put cluster/MINIO_ROOT_PASSWORD MINIO_ROOT_PASSWORD="value" -core put cluster/GRAFANA_ADMIN_PASSWORD GRAFANA_ADMIN_PASSWORD="value" - -# Incorrect (won't expand in helmfile) -core put cluster/MINIO_SECRET value="value" # Field name ≠ variable name -``` - -Reference in helmfile (helmfile.yaml.gotmpl, lines 400–401): -```yaml -set: - - name: rootPassword - value: {{ env "MINIO_ROOT_PASSWORD" }} -``` - -### Never Use `--env` Flags in Manifests - -Avoid kubectl flags like `--env KEY=value` in manifests or deployment specs. This exposes secrets in `kubectl describe` output. - -**Correct:** Use Secret volumes (k8s/storage/minio-values.yaml, lines 40–42): -```yaml -envFrom: - - secretRef: - name: minio-oidc # Reference a Secret, don't expose in manifest -``` - -**Incorrect:** -```yaml -env: - - name: MINIO_OIDC_SECRET - value: "sensitive-value" # Visible in kubectl describe -``` - -### Namespace-First Organization - -Organize manifests by namespace: `k8s//` - -Structure per namespace: -``` -k8s/storage/ - ├── minio-values.yaml - ├── minio-bucket-init.sh - └── (local charts if any) - -k8s/monitoring/ - ├── prometheus-values.yaml - ├── dashboards/ # ConfigMap files auto-loaded via helmfile postsync hook - ├── servicemonitors/ # ServiceMonitor CRD instances - └── alerts/ # PrometheusRule CRD instances -``` - ---- - -## Hook Scripts & Integration Workflows - -### Pre/Post-Sync Hooks - -Helmfile hooks (presync/postsync) drive setup workflows. Use **relative paths from repo root**, never absolute paths. - -**Pattern (helmfile.yaml.gotmpl, lines 402–416):** -```yaml -- name: minio - hooks: - - events: ["presync"] - command: bash - args: - - -c - - | - bash k8s/base/namespace-setup.sh storage - bash k8s/storage/minio-bucket-init.sh storage loki-chunks loki-ruler -``` - -**Common presync tasks:** -- Create namespace (via `k8s/base/namespace-setup.sh`). -- Pre-create ConfigMaps/Secrets for the release. -- Initialize infrastructure (buckets, databases, etc.). - -**Common postsync tasks:** -- Wait for operator/webhook readiness. -- Apply CRD instances (ServiceMonitor, PrometheusRule, Certificate). -- Perform post-deployment setup (cluster initialization, user creation). - -Example (helmfile.yaml.gotmpl, lines 64–72): -```yaml -hooks: - - events: ["postsync"] - command: bash - args: - - -c - - | - kubectl rollout status deploy/cert-manager -n cert-manager --timeout=120s - kubectl apply -f - <<'EOF' - # ClusterIssuer and Certificate CRD instances... - EOF -``` - -### ServiceMonitor Pattern - -One file per service in `k8s/monitoring/servicemonitors/svc-.yaml`. - -**Key elements:** -- `namespaceSelector`: Match the namespace where the app runs. -- `selector.matchLabels`: Match the app label from the Deployment (e.g., `app.kubernetes.io/name: argocd-metrics`). -- `endpoints.port`: Name of the metrics port in the Service. -- `interval`: Scrape frequency (e.g., `30s`). - -Example (`k8s/monitoring/servicemonitors/argocd.yaml`, lines 1–37): -```yaml -apiVersion: monitoring.coreos.com/v1 -kind: ServiceMonitor -metadata: - name: argocd - namespace: monitoring - labels: - release: kube-prometheus-stack # Required: links to Prometheus release -spec: - namespaceSelector: - matchNames: - - cicd # App namespace - selector: - matchLabels: - app.kubernetes.io/name: argocd-metrics # Matches Deployment pod label - endpoints: - - port: metrics # Service port name (not port number) - interval: 30s - scrapeTimeout: 10s -``` - -### PrometheusRule Pattern - -One file per service in `k8s/monitoring/alerts/svc--rules.yaml`. - -**Structure:** -- `groups[].name`: Logical grouping (e.g., `minio.rules`). -- `groups[].rules[].alert`: Alert name. -- `expr`: PromQL expression (5-minute windows for rate alerts). -- `for`: Duration threshold (e.g., `10m`). -- `labels.severity`: `critical`, `warning`. -- `annotations`: summary + description (use `{{ $value }}` for metric value). - -Example (`k8s/monitoring/alerts/svc-minio-rules.yaml`, lines 1–47): -```yaml -apiVersion: monitoring.coreos.com/v1 -kind: PrometheusRule -metadata: - name: minio-rules - namespace: storage -spec: - groups: - - name: minio.rules - interval: 15s - rules: - - alert: MinIOHighErrorRate - expr: | - ( - sum(rate(minio_s3_requests_total{error="true"}[5m])) - / - sum(rate(minio_s3_requests_total[5m])) - ) > 0.05 - for: 10m - labels: - severity: warning - annotations: - summary: "High error rate on MinIO" - description: "{{ $value | humanizePercentage }}" -``` - -### Grafana Dashboard Pattern - -Dashboards are stored as JSON ConfigMaps in `k8s/monitoring/dashboards/` and auto-loaded via helmfile postsync hook (see helmfile.yaml.gotmpl, lines 471–477). - -**6-row template (recommended layout):** -1. **Availability:** Uptime, error rate, latency (SLO band). -2. **Resources:** CPU, memory, disk usage, network I/O. -3. **Domain metrics:** Service-specific KPIs (throughput, queue depth, cache hit rate). -4. **Logs:** Recent error logs from Loki. -5. **SLO:** SLI tracking (burn rate, error budget). -6. **Related dashboards:** Links to dependent services. - -Auto-load hook (helmfile.yaml.gotmpl, lines 471–477): -```yaml -hooks: - - events: ["postsync"] - command: kubectl - args: - - apply - - -f - - k8s/monitoring/dashboards/ -``` - ---- - -## Integration Checklist - -When adding a new service to the homelab, verify the following: - -- [ ] **Chart pinning:** Helm chart version pinned (`~1.0` format in helmfile, not floating `latest`). -- [ ] **Secrets management:** All secrets stored in Vault (none in values.yaml, ConfigMap, or CLI flags). -- [ ] **Metrics endpoint:** Service exports `/metrics` endpoint (Prometheus format). -- [ ] **ServiceMonitor:** Created and auto-scraped by Prometheus operator. -- [ ] **PrometheusRule:** Alert rules defined for errors, latency, SLO violations. -- [ ] **Grafana dashboard:** 6-row template auto-loaded via ConfigMap. -- [ ] **Ingress:** Rule added if external access needed (see `k8s/ingress/ingress.yaml`). -- [ ] **OIDC integration:** If UI component, integrated with Authentik (see `core iam bootstrap`). - ---- - -## Shared/Reusable Repos & Image Publication - -### When to Publish as Public GHCR - -**Rule:** If a service's chart + image source lives in a separate repo (not in homelab), it **must** be published as a **public GitHub repo** under the `Riotpiaole` org. - -**Rationale:** -- Local in-cluster charts are fine for infra-owned services. -- Shared/reusable service charts should be version-pinned and publicly available for: - - Reuse across different clusters (other labs, staging, prod). - - Consumption by Argo CD apps (CI/CD pipeline). - - Independent evolution without tight coupling to homelab repo. - -### Workflow - -**Develop locally:** -1. Create companion repo (e.g., `kafaka-management-service`, `queue-operator`) in a private or public repo. -2. Include Dockerfile and Helm chart. - -**Build & publish:** -1. Set up CI/CD in the companion repo (GitHub Actions). -2. Build image and push to GHCR: `ghcr.io/Riotpiaole/:`. -3. Tag release and publish chart (npm registry, GitHub releases, or OCI registry). - -**Reference in homelab:** -1. Pin image tag + chart version in helmfile.yaml.gotmpl. -2. Chart `repositories` block references the public Helm repo (or OCI registry). - -**Example (commit cd1c569 — SQS charts):** -```yaml -- name: management-service - namespace: sqs - chart: k8s/sqs/charts/management-service - values: - - image: - repository: ghcr.io/Riotpiaole/kafaka-management-service - tag: v1.0.0 # Pinned tag from public build -``` - ---- - -## Cross-References - -- **CLAUDE.md gotchas:** Field name = variable name, MinIO `--env` flag exposure, helmfile template syntax (`{{ env "VAR" }}` not `${VAR}`), CNPG password templating. -- **README.md:** kubectl context setup, port-forward aliases, cluster topology. -- **USAGE.md:** IAM bootstrap, secret management, deployment recipes. diff --git a/project-usage/core-cli-tools.md b/project-usage/core-cli-tools.md deleted file mode 100644 index 915a84d..0000000 --- a/project-usage/core-cli-tools.md +++ /dev/null @@ -1,342 +0,0 @@ -# Core CLI Tools: Decision Guide - -The `core` CLI is your primary tool for cluster auth, node operations, Vault secrets management, IAM administration, and object storage. This doc covers **when** to reach for `core` vs `kubectl`/`helmfile`, the two separate auth domains that power different commands, and the complete command inventory. - -For exhaustive flag-level detail on each subcommand, see [~/workplace/core/USAGE.md](../../core/USAGE.md). - ---- - -## The Two Auth Domains - -The `core` CLI maintains **two independent authentication systems** that gate different command families. Confusing them causes authentication failures. - -### Domain 1: Node/Talos Operations (`core auth`) - -**Gates:** Node-level commands, Talos service management, cluster status. - -**Login mechanism:** `core auth login-oob` -- Out-of-band device-code flow via Authentik -- Token cached in `~/.core/token` (24h expiry) -- Required before: `core nodes`, `core status `, `core services `, `core logs `, `core log-svc `, `core dmesg `, `core config`, `core upgrade`, `core shutdown`, `core reboot`, `core node`, `core pods clean` - -**Check auth state:** -```bash -core auth status # Show token expiry -core auth clear # Force re-auth on next command -``` - -**When to use:** -- Troubleshooting node-level issues (crashes, disk space, network on a specific Talos node) -- Checking Talos service health (etcd, kubelet, scheduler, etc.) -- Upgrading cluster OS or managing node lifecycle -- Reading kernel logs (dmesg) or kubelet logs on specific nodes - -### Domain 2: Vault Secrets (`core secrets`) - -**Gates:** Secrets management (read/write), secret listing, secret export. - -**Login mechanism:** `core secrets login` -- Interactive login to Vault (opens browser, approves auth) -- Reads/writes to `~/.config/talos/secrets.toml` -- Required before: `core get [--key KEY]`, `core put key=value`, `core secrets list`, `core secrets export`, `core secrets exec` - -**Check auth state:** -```bash -core secrets status # Show Vault token / scopes -core secrets clear # Force re-auth on next command -``` - -**When to use:** -- Reading or writing cluster secrets (API keys, database passwords, OAuth client secrets) -- Managing application credentials in Vault (centralized secret store) -- Rotating secrets for services (e.g., OAuth2 client secrets) -- Listing all secrets under a path -- Bootstrapping `.env` files for helmfile deployment (via `vsource`) - ---- - -## Auth Bug Fixes - -**Bug #1 (Fixed):** `core status ` is a **node-ops command**, not an auth verifier. It does NOT verify that you're authenticated to Vault. To verify auth state, use the correct domain-specific command: -- For node access: `core auth status` -- For Vault access: `core secrets status` - -**Bug #2 (Fixed):** `core secrets login` and `core auth login-oob` are **NOT interchangeable**. They gate completely different systems: -- `core auth login-oob` gates node/cluster operations -- `core secrets login` gates Vault secret read/write -- You may be authenticated to one domain and not the other. Check state separately. - ---- - -## When to Reach for `core` - -| Scenario | Tool | Why | -|----------|------|-----| -| Pod/deployment issue | `kubectl` | Pods, replicas, rollouts, events | -| Package release management | `helmfile` | Install/upgrade Helm charts | -| Node crashes, disk, kernel panic | `core auth` + node commands | Direct access to Talos node state | -| Service unreachable on Talos node | `core status `, `core services ` | Talos service health | -| Database password rotation | `core secrets` + `core put` | Vault secret write | -| Need API key for app deployment | `core secrets` + `core get` | Fetch from Vault, inject via `vsource` | -| Create OAuth2 app in Authentik | `core iam create-app` | Manage federated identity apps | -| Add user to service (MinIO, Grafana, etc.) | `core iam add-member` | Group membership binding | -| S3 object upload/download | `core bucket` | MinIO operations | - ---- - -## Command Inventory - -Complete list of `core` subcommands, organized by domain. See [~/workplace/core/USAGE.md](../../core/USAGE.md) for usage flags and examples. - -### Authentication - -```bash -# Node/Talos operations auth -core auth login-oob # Authenticate with Authentik (device code flow) -core auth status # Check token expiry / auth state -core auth clear # Clear cached auth token - -# Vault secrets auth -core secrets login # Authenticate to Vault (interactive) -core secrets status # Check Vault token / scopes / expiry -core secrets clear # Clear Vault auth token -``` - -### Cluster & Node Operations - -```bash -# Node discovery and status -core nodes # List all cluster nodes (IP, hostname, status) -core status # Talos node overview (resources, uptime) -core services # List Talos services (etcd, kubelet, etc.) - -# Logs -core logs # Stream kubelet logs on node -core log-svc # Logs for specific Talos service (etcd, scheduler, etc.) -core dmesg # Kernel logs (ring buffer) from node - -# Node lifecycle -core upgrade # Upgrade Talos OS on node -core shutdown # Graceful shutdown -core reboot # Reboot node - -# Kubernetes context -core config kube-list # List available kubeconfig contexts -core config kube-use # Switch to kubeconfig context (LAN vs WireGuard) - -# Pod cleanup -core pods clean # Delete Failed/Evicted/Terminating pods -core node # Get node details (includes pod stats) -``` - -### Secrets Management (Vault) - -```bash -# Read secrets -core get # Fetch all fields under path -core get --key KEY_NAME # Fetch specific field - -# Write secrets -core put key=value [key2=value2 ...] # Store secrets in Vault - -# List and export -core secrets list # List all secret paths -core secrets export # Export secrets as shell-sourceable format - -# Execute with secrets in environment -core secrets exec -- # Run command with secrets loaded in env -``` - -**Convention:** Field name = variable name (SCREAMING_SNAKE_CASE). Never use `value=`. - -Example: -```bash -# Write -core put cluster/ANTHROPIC_API_KEY ANTHROPIC_API_KEY="sk-ant-..." - -# Read -core get cluster/ANTHROPIC_API_KEY -``` - -### IAM Management (Authentik) - -```bash -# Groups -core iam list-groups # List all groups -core iam create-group # Create new group - -# OAuth2 Applications -core iam list-apps # List all OAuth2 apps -core iam create-app --slug --redirect-uri # Create app -core iam describe-app # Show client ID, secret, URIs, scope claims -core iam rotate-secret # Rotate OAuth2 client secret - -# User management -core iam add-member # Add user to group -core iam bind-app # Grant group access to application - -# Cleanup -core iam delete-app # Delete OAuth2 application -``` - -### Object Storage (MinIO) - -```bash -# List and manage buckets -core bucket list # List all buckets -core bucket upload [] # Upload file to S3 -core bucket download # Download file from S3 -core bucket delete # Delete object from S3 -``` - -### Utilities - -```bash -core help # Show command help -core pf grafana # Port-forward to Grafana (localhost:3000) -core pf prometheus # Port-forward to Prometheus (localhost:9090) -core pf minio # Port-forward to MinIO console (localhost:9001) -core pf iam # Port-forward to Authentik (localhost:7000) -``` - ---- - -## Access Control Tiers - -The cluster uses a tiered access model based on authentication mechanism and network perimeter. - -### Tier A: OIDC + RBAC (Authentik-enforced) - -Services with federated OIDC login and group-based role assignment. - -| Service | Auth Method | Group Claim | Role Binding | -|---------|-------------|------------|--------------| -| **Grafana** | Authentik OIDC | `groups` | Mapped to Admin / Viewer / Viewer | -| **Argo CD** | Authentik OIDC | `groups` | RBAC role binding (policy.csv) | -| **MinIO** | Authentik OIDC | `groups` / custom `policy` | Policy-based access (readwrite / readonly) | -| **Forgejo** | Authentik OIDC | `email` (OAuth login only) | No group enforcement (open git repo) | -| **kmsvc** | Vault JWT | `sub` / `aud` | Audience validation + service scope | - -**Provisioning:** -```bash -# After core auth login-oob, bootstrap IAM apps: -bash k8s/talos-iam/bootstrap-iam.sh -``` - -### Tier B: Network-Perimeter Only (No Authentik Enforcement) - -Services with no OIDC support (product limitation). Access is restricted to LAN/WireGuard perimeter only. - -| Service | Network Access | Use Case | -|---------|----------------|----------| -| **Portainer** | LAN + WireGuard only | Container UI, workload browsing | -| **Longhorn** | LAN + WireGuard only | Storage volume management | -| **Temporal** | LAN + WireGuard only | Workflow execution (auth TBD) | - -All three are reachable **only** via WireGuard/LAN-only Ingress rules. Zero remote access risk, but also zero federated identity. If remote Temporal access is needed, upgrade the chart's auth configuration or replace with an OIDC-compatible workflow platform. - -**Accessing Tier B services:** -```bash -# From off-LAN, use WireGuard context -core config kube-use admin@homelab-cluster-1 -core pf minio # connects via 10.6.0.1:9001 -``` - ---- - -## Workflow: Rotate an OAuth2 Application Secret - -```bash -# 1. Rotate secret in Authentik -SECRET=$(core iam rotate-secret grafana | jq -r '.client_secret') - -# 2. Update Helm values -vi k8s/logging/grafana-values.yaml -# Set: GRAFANA_OIDC_CLIENT_SECRET="$SECRET" - -# 3. Redeploy the app -helmfile apply -l app=grafana - -# 4. Verify new secret is in use -core iam describe-app grafana | grep client_secret -``` - ---- - -## Workflow: Add User to Service - -```bash -# 1. Verify group exists (or create it) -core iam list-groups | grep minio-admins - -# If not found: -core iam create-group minio-admins - -# 2. Add user to group -core iam add-member minio-admins alice - -# 3. Bind group to MinIO application -core iam bind-app minio minio-admins - -# 4. User has access on next login -# (OIDC login to MinIO → Authentik → group check → MinIO policy applied) -``` - ---- - -## Workflow: Rotate a Database Password (Vault) - -```bash -# 1. Generate new password -NEW_PASS=$(openssl rand -hex 32) - -# 2. Store in Vault -core put cluster/POSTGRES_ADMIN_PASSWORD POSTGRES_ADMIN_PASSWORD="$NEW_PASS" - -# 3. Update database user -kubectl exec -n ddb pod/ddb-cluster-0 -- psql -U postgres -c \ - "ALTER USER postgres WITH PASSWORD '$NEW_PASS';" - -# 4. Update Helm values with new password reference -# (Or if using helmfile hook: helmfile will re-run postInitApplicationSQL with new password) - -# 5. Restart pods to pick up new secret -kubectl rollout restart -n deployment/ -``` - ---- - -## Common Questions - -**Q: I'm getting "not authenticated" on `core nodes`. What do I do?** -A: Run `core auth login-oob`. Node commands use a different auth domain than secrets. After login, `core nodes` should work. - -**Q: I have a valid Vault token but `core get` fails. Why?** -A: Check that both auth domains are active: -```bash -core auth status # Verify node auth is valid -core secrets status # Verify Vault auth is valid -``` -Both must succeed. If one is expired, re-auth that domain. - -**Q: Should I use `core bucket` or S3 CLI tools (aws-cli, s3cmd)?** -A: Use `core bucket` for simplicity (no AWS credentials). Use s3cmd/aws-cli if you need advanced sync or bandwidth control. All three talk to the same MinIO backend. - -**Q: Can I add a user without creating a group first?** -A: Groups are the unit of access control. Always create the group, then add users to it, then bind it to applications. Single-user bindings are not supported (by design). - -**Q: I rotated an OAuth2 secret but the app still fails to authenticate. What's next?** -A: -1. Verify the new secret is stored: `core iam describe-app grafana` -2. Check the deployment has the new secret: `kubectl get secret -n logging grafana-oidc -o yaml | grep client_secret` -3. Restart the pod: `kubectl rollout restart -n logging deployment/grafana` -4. Check logs: `kubectl logs -n logging deployment/grafana | grep -i oauth` - ---- - -## See Also - -- [~/workplace/core/USAGE.md](../../core/USAGE.md) — Exhaustive command reference with flags and examples -- [CLAUDE.md](../CLAUDE.md) § Sign In (Device Code Flow) — Quick reference for `core auth login-oob` -- [CLAUDE.md](../CLAUDE.md) § Quick Shortcuts — One-liners for common tasks -- [CLAUDE.md](../CLAUDE.md) § Integration Checklist — New service onboarding diff --git a/project-usage/database-postgres.md b/project-usage/database-postgres.md deleted file mode 100644 index 207a611..0000000 --- a/project-usage/database-postgres.md +++ /dev/null @@ -1,188 +0,0 @@ -# CloudNativePG PostgreSQL Database - -**Host:** `ddb-cluster-rw.ddb.svc.cluster.local` (read-write) -**Read replica:** `ddb-cluster-ro.ddb.svc.cluster.local` (read-only) -**Port:** `5432` -**Namespace:** `ddb` - -## When to Use - -- **Multi-replica HA** — 3 replicas, automatic failover -- **pgvector extension** — Vector similarity search (LLM embeddings) -- **Transactional data** — Authentik, Story Crater backend, custom apps -- **Declarative backups** — Automated WAL archiving to MinIO - -## Quick Start - -**1. Connect from pod:** -```bash -# Inside a pod (inject secret mount) -psql -h ddb-cluster-rw.ddb.svc.cluster.local \ - -U story_crater \ - -d story_crater \ - -W # prompt for password (from Secret) -``` - -**2. Create database & user (one-time):** -```bash -# Already done by helmfile postsync hook -# But if needed manually: - -psql -h ddb-cluster-rw.ddb.svc.cluster.local \ - -U postgres \ - -c "CREATE DATABASE myapp OWNER postgres;" - -psql -h ddb-cluster-rw.ddb.svc.cluster.local \ - -U postgres \ - -d myapp \ - -c "CREATE USER myapp_user WITH PASSWORD 'secret';" - -psql -h ddb-cluster-rw.ddb.svc.cluster.local \ - -U postgres \ - -d myapp \ - -c "GRANT ALL PRIVILEGES ON DATABASE myapp TO myapp_user;" -``` - -**3. Enable pgvector:** -```bash -psql -h ddb-cluster-rw.ddb.svc.cluster.local \ - -U postgres \ - -d myapp \ - -c "CREATE EXTENSION IF NOT EXISTS vector;" -``` - -**4. Create table with embeddings:** -```sql -CREATE TABLE documents ( - id BIGSERIAL PRIMARY KEY, - content TEXT, - embedding vector(1536), -- OpenAI embeddings - created_at TIMESTAMP DEFAULT NOW() -); - -CREATE INDEX ON documents USING IVFFLAT (embedding vector_cosine_ops); -``` - -## Configuration - -| Key | Value | -|-----|-------| -| Host (RW) | `ddb-cluster-rw.ddb.svc.cluster.local` | -| Host (RO) | `ddb-cluster-ro.ddb.svc.cluster.local` | -| Port | 5432 | -| Replicas | 3 (automatic failover) | -| Extensions | pgvector (LLM embeddings), uuid-ossp | -| Backups | WAL archiving to MinIO (continuous) | -| Retention | 30 days | - -## Common Patterns - -**Connection pooling (from app):** -```go -import "github.com/jackc/pgx/v5/pgxpool" - -config, _ := pgxpool.ParseConfig("postgres://user:pass@ddb-cluster-rw.ddb.svc.cluster.local:5432/myapp") -config.MaxConns = 25 -config.MinConns = 5 -pool, _ := pgxpool.NewWithConfig(ctx, config) - -// Use pool -row := pool.QueryRow(ctx, "SELECT COUNT(*) FROM users") -``` - -**Read from replica (analytics):** -```go -// Offload SELECT queries to read replica -pool.QueryRow(ctx, "SELECT * FROM documents LIMIT 1") // auto-routes to RO if available - -// Writes always go to RW -pool.Exec(ctx, "INSERT INTO documents ...") -``` - -**Vector similarity search:** -```sql -SELECT id, content, embedding <-> $1 AS distance -FROM documents -ORDER BY embedding <-> $1 -LIMIT 10; --- $1 = query embedding (e.g., from OpenAI API) -``` - -**Backup & restore:** -```bash -# Backups are automatic (WAL to MinIO) -# To restore from backup: -# 1. Check MinIO s3://postgresql-backups/ -# 2. Use PostgreSQL PITR (point-in-time recovery) -# 3. Contact SRE for restore procedure -``` - -## Monitoring - -**Grafana dashboard:** `svc-postgresql` (auto-configured) - -**Key metrics:** -- `pg_stat_activity_connections` — active connections -- `pg_stat_database_blks_read` — disk I/O -- `pg_replication_lag_seconds` — replica lag (goal: < 1s) - -**CLI health check:** -```bash -# Check replication status -kubectl exec -n ddb pod/ddb-cluster-1 -- \ - psql -U postgres -c "SELECT slot_name, restart_lsn FROM pg_replication_slots;" - -# Check replica lag -kubectl exec -n ddb pod/ddb-cluster-2 -- \ - psql -U postgres -c "SELECT now() - pg_last_xact_replay_timestamp() AS lag;" -``` - -## Secrets & Credentials - -**All user passwords stored in Vault:** -```bash -# Read password -core get cluster/STORY_CRATER_PG_PASSWORD --key STORY_CRATER_PG_PASSWORD - -# Inject into pod (auto via Secret volume) -# Mount: /run/secrets/db-password -``` - -**Connection string from env:** -```bash -POSTGRES_CONNECTION="postgres://story_crater:${STORY_CRATER_PG_PASSWORD}@ddb-cluster-rw.ddb.svc.cluster.local:5432/story_crater" -``` - -## Troubleshooting - -**Cannot connect (connection refused):** -```bash -# Verify cluster is running -k get pods -n ddb - -# Check Service DNS -k exec -it pod/debug-pod -- nslookup ddb-cluster-rw.ddb.svc.cluster.local - -# Verify Secret has password -k get secret -n ddb ddb-cluster-superuser -o jsonpath='{.data.password}' | base64 -d -``` - -**Replica lag is high (> 10s):** -```bash -# Check replica pod CPU/memory -k top pod -n ddb - -# Scale down other workloads if cluster is overloaded -# Or scale up database resources (helmfile.yaml.gotmpl) -``` - -**pgvector queries slow:** -```sql --- Ensure index exists -SELECT * FROM pg_indexes WHERE tablename = 'documents' AND indexname LIKE '%embedding%'; - --- Re-index if missing -CREATE INDEX ON documents USING IVFFLAT (embedding vector_cosine_ops); -``` - -See `/TROUBLESHOOTING.md` for full incident guide. diff --git a/project-usage/forgejo-registry-cleanup.md b/project-usage/forgejo-registry-cleanup.md deleted file mode 100644 index 224fe70..0000000 --- a/project-usage/forgejo-registry-cleanup.md +++ /dev/null @@ -1,179 +0,0 @@ -# Forgejo OCI Registry Cleanup - -Automatic garbage collection for the Forgejo container registry. Deletes old image tags when newer versions are pushed, keeping only the latest N versions per repository. - -## Why - -The Forgejo OCI registry stores all pushed images indefinitely. Without cleanup: -- Old/retired image versions accumulate -- Storage fills up (`longhorn` PVC) -- Old versions clutter the UI - -## What it does - -**CronJob** (`forgejo-registry-cleanup`): -- Runs daily at 2 AM UTC (configurable) -- Lists all images in the registry -- For each image, keeps only the **latest 3 versions** (configurable) -- Deletes tags for older versions -- Skips images with ≤ 3 tags (nothing to delete) - -## How to enable - -The manifest is in `k8s/bootstrap/phase3-forgejo/registry-cleanup-cronjob.yaml`. It's **currently disabled** (suspended) because: - -1. **Forgejo registry auth** needs to be configured - - `forgejo-registry-token` secret must exist in `cicd` namespace - - Should contain `username` and `password` keys - - User needs permission to delete images in the registry - -2. **Registry must expose `/v2/_catalog`** endpoint - - Standard for OCI registries - - Forgejo includes this, but may be behind auth - -### Step 1: Create registry token - -If `forgejo-registry-token` doesn't exist or is empty: - -```bash -# As a Forgejo admin, create an API token with full scope -# https://forgejo.riotpiao.com/user/settings/tokens -# Copy the token - -kubectl create secret generic forgejo-registry-token \ - -n cicd \ - --from-literal=username= \ - --from-literal=password= \ - --dry-run=client -o yaml | sops -e -i - -``` - -Or edit via `k8s/argocd/secrets/forgejo-registry-token.enc.yaml`: - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: forgejo-registry-token - namespace: cicd -type: Opaque -stringData: - username: ci-bot # or any user with admin rights - password: -``` - -### Step 2: Test in dry-run mode - -Before enabling for real, verify it works: - -```bash -# Edit the CronJob to set DRY_RUN=true -kubectl set env cronjob/forgejo-registry-cleanup -n cicd DRY_RUN=true - -# Trigger a test run -kubectl create job --from=cronjob/forgejo-registry-cleanup \ - -n cicd forgejo-registry-cleanup-test - -# Check logs -kubectl logs -n cicd -l job-name=forgejo-registry-cleanup-test -f -``` - -Dry-run output shows which images **would** be deleted without deleting them. - -### Step 3: Enable for real - -```bash -# Set DRY_RUN=false and unsuspend -kubectl patch cronjob forgejo-registry-cleanup -n cicd \ - -p '{"spec":{"suspend":false}}' - -kubectl set env cronjob/forgejo-registry-cleanup -n cicd DRY_RUN=false -``` - -## Configuration - -Edit `registry-cleanup-cronjob.yaml` or patch the CronJob: - -| Env var | Default | Purpose | -|---|---|---| -| `REGISTRY_HOST` | `forgejo.riotpiao.com` | Registry hostname | -| `KEEP_VERSIONS` | `3` | How many recent versions to keep per image | -| `DRY_RUN` | `false` | If `true`, log what would be deleted without deleting | - -**Schedule:** Edit `.spec.schedule` (cron format). Current: `0 2 * * *` (2 AM UTC daily). - -Examples: -- `0 2 * * 0` → Weekly on Sunday at 2 AM -- `0 0 1 * *` → Monthly on the 1st at midnight -- `0 */6 * * *` → Every 6 hours - -## Monitoring - -### Check if running - -```bash -# See all runs -kubectl get jobs -n cicd -l app=forgejo-registry-cleanup - -# Latest run logs -kubectl logs -n cicd -l app=forgejo-registry-cleanup --tail=100 -f -``` - -### Failed runs - -If a job fails: -1. Check logs: `kubectl logs -n cicd ` -2. Common issues: - - **403 Unauthorized**: Registry token invalid or expired - - **404 _catalog**: Registry doesn't expose catalog endpoint - - **Connection refused**: Registry unreachable (DNS, network policy) - -### Metrics - -The job doesn't currently emit Prometheus metrics, but you can: -- Check pod exit codes in K8s events -- Parse logs for "Total images deleted: N" -- Set up log aggregation to alert on failures - -## Limitations - -1. **No version sorting**: Tags are deleted in the order returned by the registry - - Assumption: registries return newest first (not always true) - - **Fix**: Parse semantic versions explicitly if needed - -2. **No protection for `latest` tag**: If `latest` is old, it will be kept but others deleted - - Desired behavior: prioritize newest build + never delete `latest` - - Could add logic to always keep `latest` + latest N-1 tagged versions - -3. **No size-aware deletion**: Deletes by tag count, not storage size - - Desired: keep until storage threshold is reached - - Would need registry V2 API extensions (`HEAD /v2//blobs/` for size) - -## Customizing the script - -Edit the `cleanup.sh` script in the ConfigMap to: -- Change sorting/selection logic -- Integrate with external systems (Slack alerts, Prometheus metrics) -- Add per-image exceptions (e.g., never delete `production-*` tags) -- Use `--delete-by-digest` to reclaim actual disk space (not just catalog entries) - -Example: Keep all tags matching `v*.*.*.` plus latest 2: - -```bash -# In cleanup.sh, replace the tag filtering logic: -SEMVER_TAGS=$(echo "$TAGS" | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | sort -rV) -KEEP_TAGS="$SEMVER_TAGS $(echo "$TAGS" | head -2 | tr '\n' ' ')" -TAGS_TO_DELETE=$(echo "$TAGS" | grep -v -F "$KEEP_TAGS") -``` - -## Future improvements - -- [ ] Semantic version sorting (v1.0.0 > v0.9.9) -- [ ] Storage size-aware retention (keep until >80% full) -- [ ] Slack/email notifications on deletion -- [ ] Prometheus metrics export -- [ ] Per-image exception rules (YAML config) -- [ ] Integration with CI/CD pipeline (delete old PR images automatically) - ---- - -**Related:** `k8s/bootstrap/phase3-forgejo/` — Forgejo deployment manifests diff --git a/project-usage/infra-practice.md b/project-usage/infra-practice.md deleted file mode 100644 index 169f09f..0000000 --- a/project-usage/infra-practice.md +++ /dev/null @@ -1,198 +0,0 @@ -# Infrastructure Practice Playbook - -Standardized procedures for troubleshooting, developing, deploying, and operating the homelab platform. Each procedure explicitly calls out where the `core` CLI fits vs `kubectl`/`helmfile`/direct cluster access. See `core-cli-tools.md` for the auth/secrets domain split; see `infra-troubleshooting.md` for quick patterns and gotchas. - -## Procedure A: Troubleshoot a Service or Cluster Issue - -1. **Identify the domain.** - - Vault/secrets: `core get`/`core put` failing, Vault unreachable. - - Node/Talos: node crashes, disk full, kubelet unreachable, network issues. - - Plain Kubernetes: pod CrashLoop, service 503, deployment stuck. - -2. **If Vault-adjacent (secrets, authentication failing):** - - Run `core secrets status` first (NOT `core auth status` — common mistake). - - If `Vault UNREACHABLE`, check DNS/networking: - - No wildcard DNS exists; verify manual `/etc/hosts` entries (10.6.0.1 for WireGuard, 192.168.1.160 for LAN). - - Ping the Vault service: `kubectl get svc -n vault | grep vault`. - - If `Vault token not cached`, run `core secrets login`, approve device code in browser. - - Verify: `core secrets status` shows `✓ Authenticated`. - -3. **If node/Talos-adjacent (kubelet logs, node state, services failing):** - - Run `core auth status` first. - - If token expired, run `core auth login-oob`, approve device code in browser. - - Then run `core nodes` to list cluster nodes. - - For a specific node, run `core status ` (Talos state). - - Inspect Talos services: `core services ` (kubelet, etcd, controller, etc.). - - Check service logs: `core logs ` (main Talos logs) or `core log-svc kubelet` (specific service). - - Consult `infra-troubleshooting.md` for Pod stuck in CrashLoopBackOff and kubelet restart patterns. - -4. **If plain Kubernetes (pod/deployment/service issues):** - - Consult root `TROUBLESHOOTING.md` for the layer-before-tool SRE methodology (procedures 1–10). - - Use `infra-troubleshooting.md` § Quick Patterns for common diagnoses: - - CrashLoopBackOff: `kubectl logs -n --tail=50` + `kubectl describe pod -n | grep -A 10 Events`. - - Service 503: `kubectl get endpoints -n ` (endpoints missing?) + `kubectl get pods -n -o wide` (pods not Ready?). - - Helm release stuck: `helmfile status | grep -E "FAILED|UNKNOWN|PENDING"` + `helm status -n --show-resources`. - - If still unclear, escalate to `kubectl get all -n ` and review resource events. - -5. **For dashboards and live metrics:** - - Use `core pf grafana` (port-forward to localhost:3000) rather than raw `kubectl port-forward` — keeps forwarded ports consistent. - - If Prometheus unavailable, check: `kubectl get pods -n monitoring | grep prometheus`. - - If ServiceMonitor not scraping, verify: `kubectl get servicemonitor -A | grep ` and inspect `.spec.selector` matches the target pod's app label. - ---- - -## Procedure B: Launch/Develop a New Service or POC - -1. **Plan the service.** - - Determine namespace (e.g., `sqs`, `temporal`, `databases`, `monitoring`). - - Decide if metrics exported (most should) and if OIDC-gated. - - Sketch a Helm values.yaml structure (secrets, replicas, resource requests, affinity). - -2. **Authenticate to Vault.** - - Run `core secrets login` and approve device code in browser. - - Verify: `core secrets status` shows `✓ Authenticated`. - - You'll need Vault access to store service secrets in step 5. - -3. **Create service Helm chart directory.** - - Create `k8s//` with at minimum: - - `values.yaml` (Helm values for deployment, service, replicas, resource limits). - - `charts/` subdirectory for any custom local Helm charts (optional). - - Follow naming conventions from `coding-standards.md`. - -4. **Add Helm release to helmfile.** - - Open `helmfile.yaml.gotmpl`. - - Add release block under `releases:` section, following this structure: - ```yaml - - name: - namespace: - chart: / - version: ~1.0 # pin major.minor, allow patch updates - needs: - - / # if applicable - values: - - k8s//values.yaml - - secretsInline: - DB_PASSWORD: "{{ env \"_DB_PASSWORD\" }}" - ``` - - Consult `coding-standards.md` for `needs:` ordering (example: sqs section shows strimzi-operator → kafka-cluster → queue-crd → management-service). - - Reference real example: root helmfile's `sqs` section. - -5. **If the service needs secrets (DB password, API key, OAuth secret):** - - Generate value (e.g., `openssl rand -hex 32` for passwords). - - Store in Vault: `core put cluster/_ _="value"`. - - **Critical gotcha:** field name MUST equal variable name (e.g., `FORGEJO_ADMIN_PASSWORD=` not `value=`) per `coding-standards.md` § Vault field=variable convention. - - Reference in values.yaml via `{{ env "VARIABLE_NAME" }}` (Helmfile Go template syntax, NOT shell `${VAR}`). - - Do NOT hardcode secrets in values.yaml or ConfigMaps. - -6. **Verify Helm syntax before deploy.** - - Run `helmfile lint` (catches template errors, duplicate releases). - - Run `helmfile diff -l name=` (show what will be deployed). - - Review diff for correctness (verify env var substitutions, resource limits, affinity rules). - -7. **Deploy the service.** - - Run `helmfile apply -l name=`. - - Monitor: `kubectl get pods -n -w` (watch until Running). - - If pods stuck: `kubectl describe pod -n ` (check Events for SchedulingFailed, ImagePullBackOff, etc.). - -8. **If the service exports `/metrics` (Prometheus format):** - - Create ServiceMonitor: `k8s/monitoring/servicemonitors/svc-.yaml`. - - `.spec.selector.matchLabels` must match the service's pod labels (usually `app: `). - - `.spec.endpoints[0].port` must match the service port name or number exporting metrics. - - Create PrometheusRule: `k8s/monitoring/alerts/svc--rules.yaml`. - - Include error rate, latency, and SLO alert rules. - - Use `prometheus` as the rule group. - - Create Grafana dashboard: `k8s/monitoring/dashboards/svc-.yaml`. - - Use 6-row template: Availability, Resources, Domain metrics, Logs, SLO, Related. - - See README.md § Example Applications for a full walkthrough. - - Verify scrape: `kubectl get servicemonitor -A | grep ` and check Prometheus Targets UI for green status. - -9. **If OIDC/IAM-gated (admin UI, restricted API):** - - Create app in Authentik: `core iam create-app "my-service" --slug my-service --redirect-uri "https://my-service.riotpiao.com/callback"`. - - Bind app to group: `core iam bind-app my-service ` (e.g., `grafana-admins` for admin-only UI). - - Retrieve credentials: `core iam describe-app my-service` (client ID, client secret). - - Deploy secret: `kubectl create secret generic -oidc --from-literal=client-id= --from-literal=client-secret= -n `. - - Reference secret in values.yaml: mount via `.spec.template.spec.containers[].env` or volumeMounts. - - See `core-cli-tools.md` § Access Control Tiers for Tier A (OIDC + RBAC) vs Tier B (network perimeter only). - -10. **Verify service is live.** - - Pods: `kubectl get pods -n -o wide` (all Running, 1/1 Ready). - - Metrics (if applicable): `kubectl get servicemonitor -A | grep ` and visit Prometheus Targets or Grafana dashboard. - - Endpoint: If publicly routed via Ingress, verify `/etc/hosts` entry (10.6.0.1 for WireGuard, 192.168.1.160 for LAN) and `curl https://my-service.riotpiao.com/health` (or equivalent health endpoint). - - Logs: `kubectl logs -n ` (no errors). - -### Definition of Done (Per Service) - -- [ ] Helm chart version pinned (~1.0 format in helmfile) -- [ ] All secrets in Vault (none in values.yaml or ConfigMap) -- [ ] `/metrics` endpoint exported (if applicable) -- [ ] ServiceMonitor resource created (if metrics exported) -- [ ] PrometheusRule with error/latency/SLO alerts (if metrics exported) -- [ ] Grafana dashboard (if metrics exported; 6-row template: Availability, Resources, Domain, Logs, SLO, Related) -- [ ] Ingress rule (if external access needed) -- [ ] OIDC integration via `core iam` (if UI component) -- [ ] Verified: `helmfile diff` clean, pods Running, dashboard live or `/metrics` returning 200 - ---- - -## Procedure C: Operate the Cluster (Node Health, Context, Cleanup) - -1. **Daily health check.** - - Check auth: `core auth status` (if OK, node ops will work). - - List nodes: `core nodes`. - - For each node, check Talos state: `core status `. - - Check K8s nodes: `kubectl get nodes -o wide` (all Ready, no NotReady). - - Check pod pressure: `kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, memory: .status.allocatable.memory, pods: .status.allocatable.pods}'`. - -2. **Troubleshoot a specific node.** - - Get node IP: `core nodes` and note the IP. - - Check Talos services: `core services ` (kubelet, etcd, controller should be running). - - Check service logs: `core logs ` (main Talos daemon logs). - - Filter to specific service: `core log-svc kubelet` (kubelet logs only). - - Restart a service if needed: `core restart kubelet` (graceful kubelet restart). - -3. **Pod cleanup (Failed, Evicted, Terminating pods).** - - Run `core pods clean` (scans all namespaces, removes stale pods). - - Verify: `kubectl get pods -A | grep -E "Failed|Evicted"` (should be empty). - -4. **Switch kubectl context (when off-LAN, on WireGuard).** - - List available contexts: `core config kube-list`. - - Switch to WireGuard path (10.6.0.1:6443): `core config kube-use admin@homelab-cluster-1`. - - **Known limitation:** `core config use ` doesn't map to WireGuard; use `kube-use` directly. - - Verify: `kubectl cluster-info` shows 10.6.0.1 (not 192.168.1.213). - -5. **MinIO bucket operations (if managing data/backups).** - - List buckets: `core bucket list`. - - Upload file: `core bucket upload `. - - Download file: `core bucket download -o `. - - Delete file: `core bucket delete `. - -6. **Bootstrap or hardware runbooks (infrequent).** - - **Fresh cluster setup:** See README.md § Bootstrap Order (14 steps). - - **Adding a new Talos node:** See README.md § Adding Hardware. - - Do not re-explain those long procedures here; consult README.md directly. - ---- - -## Notes - -**Queue subsystem (Kafka/kmsvc/Temporal namespace auto-registration):** Already deployed and stable. If re-deploying: -- Primary deploy method: `helmfile apply -l namespace=sqs` (live from root helmfile). -- Alternate isolated iterate path: `k8s/sqs/helmfile.yaml.gotmpl` (not recommended for production). -- Planned future: GitOps via `k8s/sqs/argocd/` (companion repo, not yet active). -- **Critical rule:** Temporal namespace registration is automatic via `queue-operator`; never manually `temporal operator namespace create` for any namespace referenced by a Queue's `temporal.io/namespace` label. See `~/workplace/kmsvc-manage/CLAUDE.md` ("Temporal Namespace Registration") for the full rule and why. - -**Shared/reusable service repositories:** If a service's Helm chart and container image live in a separate repository, they must be: -- Published as a public GitHub repository under the `Riotpiaole` organization. -- Images pushed to GHCR (`ghcr.io/riotpiaole/...`) for public pullability. -- Consult `coding-standards.md` § Shared/Reusable Repos for the full publishing rule. - ---- - -## Cross-References - -- **core-cli-tools.md:** Auth/secrets domain split, command inventory, when to use `core` vs `kubectl`. -- **coding-standards.md:** Helm naming conventions, `needs:` ordering rules, helmfile template syntax (`{{ env "VAR" }}` not `${VAR}`), Vault field=variable convention, shared-repo publishing rule. -- **infra-troubleshooting.md:** Quick patterns (CrashLoopBackOff, 503, helm stuck), gotchas, hard rules. -- **USAGE.md:** Exhaustive `core` command reference. -- **README.md:** Bootstrap order, hardware addition, example app walkthrough, 6-row Grafana dashboard template. -- **root TROUBLESHOOTING.md:** Generic Kubernetes SRE layer-before-tool methodology (10 diagnostic procedures). diff --git a/project-usage/minio-s3.md b/project-usage/minio-s3.md deleted file mode 100644 index 9ae989e..0000000 --- a/project-usage/minio-s3.md +++ /dev/null @@ -1,162 +0,0 @@ -# MinIO S3-Compatible Object Storage - -**Endpoint:** `https://minio.riotpiao.com` (console) -**API:** `minio.storage.svc.cluster.local:9000` (cluster-internal) -**Namespace:** `storage` - -## When to Use - -- **File uploads** — Images, documents, backups -- **Log backend** — Loki chunks storage -- **Vault unsealing** — Store unseal keys -- **CI/CD artifacts** — Build outputs, Docker layers cache - -## Quick Start - -**1. Access MinIO console:** -```bash -# Via browser: https://minio.riotpiao.com -# Credentials: MINIO_ROOT_USER / MINIO_ROOT_PASSWORD (from .env) - -# Or port-forward -make pf-minio # localhost:9001 -``` - -**2. Create bucket:** -```bash -# Via AWS CLI -export AWS_ACCESS_KEY_ID=$MINIO_ROOT_USER -export AWS_SECRET_ACCESS_KEY=$MINIO_ROOT_PASSWORD - -aws s3 mb s3://my-bucket \ - --endpoint-url https://minio.riotpiao.com \ - --region homelab - -# Or via console UI: Click "Create Bucket" -``` - -**3. Upload file:** -```bash -aws s3 cp /path/to/file.txt s3://my-bucket/ \ - --endpoint-url https://minio.storage.svc.cluster.local:9000 \ - --use-path-style -``` - -**4. List buckets:** -```bash -aws s3 ls --endpoint-url https://minio.storage.svc.cluster.local:9000 -``` - -## Configuration - -| Key | Value | -|-----|-------| -| Access key | `MINIO_ROOT_USER` (from .env) | -| Secret key | `MINIO_ROOT_PASSWORD` (from .env) | -| Cluster API | `minio.storage.svc.cluster.local:9000` | -| Console port | `9001` | -| Replication | 3-node site-replication (az-a ↔ az-b ↔ az-c) | -| Buckets (system) | `loki-chunks`, `loki-ruler`, `vault-backups` | - -## Common Patterns - -**Loki log backend (auto-configured):** -```yaml -# k8s/logging/loki-values.yaml -loki: - storage: - s3: - endpoint: minio.storage.svc.cluster.local:9000 - buckets: loki-chunks - secretAccessKey: $MINIO_ROOT_PASSWORD - accessKeyId: $MINIO_ROOT_USER -``` - -**Application usage (Go/Python/Node):** -```go -import "github.com/minio/minio-go/v7" - -client, _ := minio.New("minio.storage.svc.cluster.local:9000", &minio.Options{ - Creds: credentials.NewStaticV4(os.Getenv("MINIO_ROOT_USER"), os.Getenv("MINIO_ROOT_PASSWORD"), ""), - Secure: false, // cluster-internal (no TLS) -}) - -// Upload -client.FPutObject(ctx, "my-bucket", "file.txt", "/path/to/file.txt", minio.PutObjectOptions{}) - -// Download -client.FGetObject(ctx, "my-bucket", "file.txt", "/tmp/file.txt", minio.GetObjectOptions{}) -``` - -**Vault backup bucket:** -```bash -# Vault stores unseal keys in s3://vault-backups -# Auto-managed by helmfile; no manual action needed -``` - -## Monitoring - -**Grafana dashboard:** `svc-minio` - -**Key metrics:** -- `minio_disk_drive_free_bytes` — available space -- `minio_bucket_usage_object_count` — objects per bucket -- `minio_bucket_usage_total_bytes` — total size per bucket - -**Site replication status:** -```bash -# Port-forward to MinIO pod -k port-forward -n storage pod/minio-0 9000:9000 & - -# Check replication -mc alias set local http://localhost:9000 $MINIO_ROOT_USER $MINIO_ROOT_PASSWORD -mc admin replicate status local -``` - -## Authentication (Cluster-Internal) - -**From pods (cluster-internal):** -```bash -# Use credentials from Secret or env var -export MINIO_ENDPOINT=minio.storage.svc.cluster.local:9000 -export MINIO_ACCESS_KEY=$MINIO_ROOT_USER -export MINIO_SECRET_KEY=$MINIO_ROOT_PASSWORD -aws s3 ls --endpoint-url http://$MINIO_ENDPOINT --use-path-style -``` - -**External access (HTTPS via Ingress):** -```bash -# Console: https://minio.riotpiao.com (port 9001) -# API: Use AWS CLI with --endpoint-url https://minio.riotpiao.com:9000 -``` - -## Troubleshooting - -**Bucket creation fails:** -```bash -# Check MinIO pod logs -k logs -n storage pod/minio-0 | grep -i error - -# Verify storage space -k get pvc -n storage -``` - -**Site replication lag:** -```bash -# Check if all 3 nodes are healthy -k get pods -n storage -l app=minio - -# If one node is down, site replication queues changes (eventually consistent) -``` - -**Access denied:** -```bash -# Verify credentials in .env -echo $MINIO_ROOT_USER $MINIO_ROOT_PASSWORD - -# If credentials rotated, update Secret -k patch secret -n storage minio-root-credentials \ - --type merge -p '{"stringData":{"MINIO_ROOT_PASSWORD":"newpass"}}' -``` - -See `/TROUBLESHOOTING.md` for full incident guide. diff --git a/project-usage/sqs-messaging.md b/project-usage/sqs-messaging.md deleted file mode 100644 index 007c7ea..0000000 --- a/project-usage/sqs-messaging.md +++ /dev/null @@ -1,193 +0,0 @@ -# SQS-like Message Queue Service (kmsvc) - -**Endpoint:** `https://kmsvc.riotpiao.com` (REST + gRPC-Gateway) -**Internal:** `kmsvc-management-service.sqs.svc.cluster.local:8080` -**Namespace:** `sqs` - -## When to Use - -- **Decouple services** — Producer doesn't wait for consumer -- **Async jobs** — Fire-and-forget processing (batch, email, webhooks) -- **FIFO ordering** — Guarantee message order within `MessageGroupId` -- **Durable delivery** — At-least-once (messages in Kafka, replicated 3×) - -## Quick Start - -**1. Create a queue:** -```bash -kubectl apply -f - <` - -``` -cluster/ -├── ANTHROPIC_API_KEY # third-party API -├── STORY_CRATER_DB_PASS # database password -├── MINIO_ROOT_PASSWORD # MinIO credentials -├── AUTHENTIK_BOOTSTRAP_PASSWORD # initial admin pass -├── iam/ -│ ├── federation # Authentik OIDC app config -│ ├── roles/admin # RBAC role definitions -│ ├── services/grafana # OAuth2 client info -│ ├── agents/ci-bot # machine credentials -│ └── bindings/alice # user→role mappings -└── kubernetes/ - ├── ingress-tls # TLS cert private keys - └── pull-secrets # Docker registry creds -``` - -## Common Patterns - -**Store generated secret immediately (keeps it out of shell history):** -```bash -# Generate & store in one command -core put cluster/GRAFANA_OIDC_CLIENT_SECRET \ - GRAFANA_OIDC_CLIENT_SECRET="$(openssl rand -hex 32)" -``` - -**Use in Kubernetes Secret:** -```bash -# Create Secret using Vault secret -kubectl create secret generic grafana-oidc \ - --from-literal=client-secret="$(core get cluster/GRAFANA_OIDC_CLIENT_SECRET --key GRAFANA_OIDC_CLIENT_SECRET)" \ - -n logging -``` - -**Rotate credentials safely:** -```bash -# 1. Generate new secret -NEW_PASS=$(openssl rand -base64 24) - -# 2. Store in Vault -core put cluster/OLD_DB_PASS OLD_DB_PASS="$NEW_PASS" - -# 3. Update database user -psql -h ddb-cluster-rw.ddb.svc.cluster.local -U postgres \ - -c "ALTER USER story_crater PASSWORD '$NEW_PASS';" - -# 4. Update Kubernetes Secret -kubectl patch secret db-credentials -n story-crater-backend \ - --type merge -p "{\"stringData\":{\"password\":\"$NEW_PASS\"}}" - -# 5. Restart pods to pick up new Secret -k rollout restart -n story-crater-backend deployment/app -``` - -**JWT token from CLI:** -```bash -# After device code login -core secrets login - -# Token is cached and auto-renewed -# Use for API calls -VAULT_TOKEN=$(cat ~/.talos/vault) -curl -H "X-Vault-Token: $VAULT_TOKEN" \ - https://vault.iam.svc.cluster.local:8200/v1/secret/data/cluster/ANTHROPIC_API_KEY -``` - -## Monitoring - -**Vault status:** -```bash -# Check if sealed -core status - -# If sealed (disaster recovery): -# See /TROUBLESHOOTING.md § Vault Sealed -``` - -**Audit log (who accessed what):** -```bash -# Enable audit logging (already enabled by helmfile) -# Logs stored in Loki under vault namespace - -# View recent access -core audit log --limit 50 - -# Export for compliance -core audit export --format json > vault-audit.json -``` - -## Security Rules - -✅ **DO:** -- Store ALL secrets in Vault (never .env in git) -- Use field name = variable name -- Always use `--key` flag when retrieving -- Rotate credentials on schedule (quarterly) -- Review audit log for anomalies - -❌ **DON'T:** -- Commit `.env` with real secrets (only `.env.example`) -- Use positional arguments (must use `--key`) -- Share Vault root token -- Store secrets in pod env (use Secret volumes) - -## Troubleshooting - -**Cannot login (Authentik OIDC fails):** -```bash -# Check Authentik is running -k get pods -n iam -l app=authentik - -# Verify OIDC app in Authentik console -# Applications → Vault OIDC → Check client ID/secret - -# Restart Vault to re-sync OIDC config -k rollout restart -n iam statefulset/vault -``` - -**Vault is sealed:** -```bash -# Check status -core status - -# If sealed, use unseal keys (stored in MinIO backup) -# See /TROUBLESHOOTING.md § Vault Sealed for recovery steps -``` - -**Secret not found:** -```bash -# Verify path exists -core list cluster - -# Check secret name (case-sensitive) -core get cluster/anthropic_api_key --key anthropic_api_key # won't work -core get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY # correct -``` - -**vsource not expanding secrets:** -```bash -# Verify .env has empty value -grep ANTHROPIC_API_KEY .env -# → Should be: ANTHROPIC_API_KEY= (empty, not a value) - -# Verify Vault is accessible -core get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY -# → Should return secret - -# Run vsource explicitly -vsource .env -echo $ANTHROPIC_API_KEY # should be populated -``` - -See `/TROUBLESHOOTING.md` for full incident guide. diff --git a/scripts/pi-stage1-prompt.md b/scripts/pi-stage1-prompt.md deleted file mode 100644 index e83dd3d..0000000 --- a/scripts/pi-stage1-prompt.md +++ /dev/null @@ -1,72 +0,0 @@ -Implement **Stage 1 only** of the approved plan at -`/Users/rockliang/.claude/plans/fluttering-cooking-thunder.md`. Read that file first — it is -the spec. Do not implement Stage 2, 3 or 4. - -## Already done, do not redo - -Stage 0 passed. The Forgejo registry returns distinct, correctly-ordered image -creation timestamps, so `newest-build` is viable: - -``` -rock/api-gateway v0.0.0 2026-08-20T05:19:48.655Z -rock/api-gateway v0.1.0 2026-08-20T06:57:11.943Z -rock/api-gateway v0.1.1 2026-08-20T07:10:13.093Z -``` - -Note the images are multi-arch OCI indexes: reading `created` means descending -index -> amd64 manifest -> config blob. - -## Scope: Stage 1 = A1, A2, A4, A6, B, C1 - -- **A1** — in `~/workplace/homelab`, replace the per-repo Forgejo entry in - `k8s/argocd/projects/homelab-project.yaml` `sourceRepos` with a wildcard - `https://forgejo.riotpiao.com/rock/*`. -- **A2** — add an Argo `Application` at sync-wave `-1` that syncs - `k8s/argocd/projects/`. Nothing owns that directory today, which is why the - AppProject only ever reaches the cluster by hand. -- **A4** — Forgejo webhook to `https://argocd.riotpiao.com/api/webhook` with a - shared secret stored in `argocd-secret` (SOPS/ksops). Register it on - `rock/homelab` and `rock/homelab-frontend`. -- **A6** — add the `forgejo-registry` dockerconfigjson pull secret for any - namespace that needs it, as a new `*.enc.yaml` listed in - `k8s/argocd/secrets/secret-generator.yaml`. It currently exists only in `api`. -- **B + C1** — in `~/workplace/homelab-frontend`: delete the dead - `.github/workflows/ci.yml`, add `.forgejo/workflows/ci.yaml` and - `.forgejo/workflows/build.yaml`, and add a multi-stage distroless `Dockerfile` - (none exists today). - -Stage 1 stops before Argo CD Image Updater. Do **not** install it and do not add -image-updater annotations — that is Stage 2. - -## Hard constraints - -- **`runs-on: docker`.** That is the runner's only registered label. The existing - `.github/workflows/ci.yml` uses `ubuntu-latest`, which is exactly why it has - never executed once. -- **No git tags, ever.** Image tag is the commit short SHA: `$(git rev-parse --short HEAD)`. - Do not use `git describe`, do not create or push tags, do not use `latest`. -- **Build workflow only builds and pushes.** No manifest write-back, no git push - from CI, no `[skip ci]` guard needed. -- **GitOps only.** No `kubectl apply`, no `helm upgrade`, no local `terraform apply`. - `kubectl --dry-run=server` and read-only `kubectl get`/`logs` are fine. -- **Never `git reset --hard`.** -- Push directly to `main`, no PRs, no branches. The cluster repo has three - remotes — `origin` is Forgejo, `github` is GitHub. Push to both; they are - currently in sync at `43483da`. -- Match surrounding file style. This repo comments the *why* on non-obvious - config, and `kustomization.yaml` uses explicit `resources:` allowlists — a file - you add and forget to list is silently dropped. - -## Verify before claiming done - -- `kubectl kustomize` each directory you touch. -- `kubectl apply --dry-run=server -f` every manifest you add or change. -- Confirm the AppProject wildcard is live: - `kubectl -n argocd get appproject homelab -o jsonpath='{.spec.sourceRepos}'` -- Push an empty commit to `homelab-frontend` and confirm the build job actually - runs on the Forgejo runner and pushes `forgejo.riotpiao.com/rock/api-gateway:`. - A workflow that does not trigger is the single most likely failure here. -- Report what you verified with real command output, not assertions. If a step - fails, say so and stop rather than working around it. - -Kubeconfig: `/Users/rockliang/workplace/homelab/cluster-config/kubeconfig`.