docs: remove outdated docs describing the retired core/talos CLI workflow
This commit is contained in:
Story Crater Bot
@@ -1,460 +0,0 @@
|
||||
# CI/CD Pipeline: GitOps Validation & Deployment
|
||||
|
||||
## Overview
|
||||
|
||||
Pure GitOps CI/CD pipeline using Forgejo Actions (self-hosted runner).
|
||||
|
||||
**Principle:** Validate in CI, deploy via ArgoCD (no manual steps).
|
||||
|
||||
```
|
||||
git push
|
||||
↓
|
||||
[CI: Validate]
|
||||
├─ yamllint (YAML syntax)
|
||||
├─ kubeval (K8s manifests)
|
||||
├─ kustomize build (all layers)
|
||||
├─ argocd validation (app definitions)
|
||||
└─ security scan (secrets, best practices)
|
||||
↓
|
||||
[If push to main]
|
||||
└─ ArgoCD auto-syncs (if enabled)
|
||||
```
|
||||
|
||||
## Workflows
|
||||
|
||||
### 1. validate-k8s.yaml (Mandatory)
|
||||
|
||||
**Trigger:** Any push/PR with k8s/ changes
|
||||
|
||||
**What it does:**
|
||||
1. Lints all YAML files (`yamllint`)
|
||||
2. Validates K8s manifests (`kubeval`)
|
||||
3. Builds all kustomization layers
|
||||
4. Validates ArgoCD applications
|
||||
5. Reports results
|
||||
|
||||
**Duration:** ~2-3 minutes
|
||||
|
||||
**Status:**
|
||||
- ✅ PASS: All layers build, manifests valid → OK to merge
|
||||
- ❌ FAIL: Syntax error, invalid resource, build failed → Fix & push again
|
||||
|
||||
**Example output:**
|
||||
```
|
||||
=== Building k8s/infrastructure/ ===
|
||||
✓ Infrastructure built successfully
|
||||
Resources: 47
|
||||
|
||||
=== Building k8s/bootstrap/ ===
|
||||
✓ Bootstrap built successfully
|
||||
Resources: 23
|
||||
```
|
||||
|
||||
**When to check:**
|
||||
- After every commit
|
||||
- Before merging PRs
|
||||
- On every branch
|
||||
|
||||
### 2. argocd-sync.yaml (Recommended)
|
||||
|
||||
**Trigger:** Push to main only (k8s/ changed)
|
||||
|
||||
**What it does:**
|
||||
1. Authenticates with ArgoCD
|
||||
2. Syncs `homelab-root` application
|
||||
3. Waits for sync to complete (5 min timeout)
|
||||
4. Verifies all applications healthy
|
||||
|
||||
**Duration:** 1-5 minutes (depends on resources)
|
||||
|
||||
**Status:**
|
||||
- ✅ SYNCED: All resources deployed to cluster
|
||||
- ❌ FAILED: Sync error, pod crashes, etc. → Check ArgoCD UI for details
|
||||
|
||||
**When it runs:**
|
||||
- Automatically after merge to main
|
||||
- Only on k8s/ changes (not on docs)
|
||||
|
||||
**Manual trigger (if needed):**
|
||||
```bash
|
||||
# SSH to runner or use Forgejo UI
|
||||
# Re-run failed workflow
|
||||
# Or manually sync: argocd app sync homelab-root
|
||||
```
|
||||
|
||||
**Requires secrets:**
|
||||
- `ARGOCD_SERVER`: ArgoCD server URL (https://argocd.riotpiao.com)
|
||||
- `ARGOCD_AUTH_TOKEN`: ArgoCD API token (generate via ArgoCD UI)
|
||||
|
||||
### 3. security-scan.yaml (Optional)
|
||||
|
||||
**Trigger:** Any push/PR with k8s/ changes
|
||||
|
||||
**What it does:**
|
||||
1. Scans Dockerfiles for vulnerabilities (`trivy`)
|
||||
2. Scans Helm charts for security issues
|
||||
3. Audits K8s manifests (`polaris`)
|
||||
4. Checks for hardcoded secrets
|
||||
5. Verifies security best practices
|
||||
|
||||
**Duration:** ~3-5 minutes
|
||||
|
||||
**Status:**
|
||||
- ✅ PASS: No critical issues
|
||||
- ⚠️ WARNING: Best practice recommendations (non-blocking)
|
||||
- ❌ FAIL: Hardcoded secrets found (must fix)
|
||||
|
||||
**Common issues:**
|
||||
- Missing resource limits (warning)
|
||||
- Privileged containers (warning)
|
||||
- Hardcoded passwords (ERROR)
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
.forgejo/
|
||||
├── workflows/ # CI/CD workflows
|
||||
│ ├── validate-k8s.yaml # Validate manifests (required)
|
||||
│ ├── argocd-sync.yaml # Sync to cluster (auto on main)
|
||||
│ └── security-scan.yaml # Security checks (optional)
|
||||
└── CI-CD.md # This file
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Setup Instructions
|
||||
|
||||
### 1. Install Forgejo Runner
|
||||
|
||||
```bash
|
||||
# On runner machine (inside cluster or external)
|
||||
forgejo-runner register \
|
||||
--instance https://forgejo.riotpiao.com \
|
||||
--token <registration-token> \
|
||||
--name homelab-runner \
|
||||
--labels docker
|
||||
|
||||
forgejo-runner daemon
|
||||
```
|
||||
|
||||
### 2. Add ArgoCD Secrets to Forgejo
|
||||
|
||||
```bash
|
||||
# Go to: Forgejo → Settings → Secrets
|
||||
|
||||
# Add:
|
||||
ARGOCD_SERVER = https://argocd.riotpiao.com
|
||||
ARGOCD_AUTH_TOKEN = <token> # Generate: argocd account generate-token
|
||||
```
|
||||
|
||||
### 3. Generate ArgoCD Token
|
||||
|
||||
```bash
|
||||
# Inside cluster
|
||||
kubectl -n argocd port-forward svc/argocd-server 8080:443
|
||||
|
||||
# Go to: https://localhost:8080/user-info/api-tokens
|
||||
# Create new token (CI/CD)
|
||||
# Copy token to Forgejo secrets
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Workflow Execution
|
||||
|
||||
### When developer pushes to feature branch:
|
||||
|
||||
```
|
||||
git push origin feature/new-service
|
||||
|
||||
↓
|
||||
Forgejo Actions triggered
|
||||
↓
|
||||
validate-k8s.yaml runs:
|
||||
✓ Lints YAML
|
||||
✓ Validates manifests
|
||||
✓ Builds kustomizations
|
||||
✓ All pass → GitHub comment: "Ready to merge"
|
||||
↓
|
||||
Developer opens PR
|
||||
↓
|
||||
Reviewer checks:
|
||||
- Code changes (YAML)
|
||||
- Workflow results
|
||||
- ArgoCD impact (diff)
|
||||
↓
|
||||
PR merged to main
|
||||
```
|
||||
|
||||
### When merged to main:
|
||||
|
||||
```
|
||||
git merge feature/new-service → main
|
||||
|
||||
↓
|
||||
Forgejo Actions triggered
|
||||
↓
|
||||
validate-k8s.yaml runs:
|
||||
✓ Same validation as above
|
||||
↓
|
||||
argocd-sync.yaml runs (if enabled):
|
||||
✓ Syncs homelab-root
|
||||
✓ Waits for sync
|
||||
✓ Verifies health
|
||||
✓ Resources deployed to cluster
|
||||
↓
|
||||
Cluster state = git state
|
||||
(No manual kubectl apply needed!)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Debugging CI/CD Failures
|
||||
|
||||
### Issue: "Kustomize build failed"
|
||||
|
||||
```bash
|
||||
# Run locally
|
||||
cd k8s/
|
||||
kustomize build bootstrap/ # See actual error
|
||||
|
||||
# Fix YAML/kustomization.yaml
|
||||
# git push again
|
||||
```
|
||||
|
||||
### Issue: "Kubeval validation failed"
|
||||
|
||||
```bash
|
||||
# Check K8s manifest syntax
|
||||
kubeval k8s/platform/minio/config.yaml
|
||||
|
||||
# Common issues:
|
||||
# - Typos in apiVersion, kind, metadata
|
||||
# - Missing required fields
|
||||
# - Invalid references (namespace, service name)
|
||||
```
|
||||
|
||||
### Issue: "ArgoCD sync failed"
|
||||
|
||||
```bash
|
||||
# Check ArgoCD UI
|
||||
# https://argocd.riotpiao.com → homelab-root
|
||||
|
||||
# Or CLI
|
||||
argocd app get homelab-root
|
||||
argocd app logs homelab-root --follow
|
||||
|
||||
# Common issues:
|
||||
# - Missing namespace (fixed by infrastructure layer)
|
||||
# - Invalid Helm chart version
|
||||
# - Secret not found
|
||||
# - Network policy blocking traffic
|
||||
```
|
||||
|
||||
### Issue: "Security scan found hardcoded secret"
|
||||
|
||||
```bash
|
||||
# Fix: Remove secret from YAML
|
||||
# Add to SOPS encryption instead
|
||||
|
||||
# Or use ArgoCD Sealed Secrets
|
||||
# (if SOPS not available)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Viewing Results
|
||||
|
||||
### Forgejo Actions UI
|
||||
|
||||
```
|
||||
Repository → Actions
|
||||
├─ validate-k8s
|
||||
│ ├─ ✅ Success (merge safe)
|
||||
│ ├─ ❌ Failed (fix required)
|
||||
│ └─ Logs (click "Steps" → "Summary")
|
||||
├─ argocd-sync
|
||||
│ ├─ ✅ Synced (deployed)
|
||||
│ └─ ❌ Failed (check ArgoCD UI)
|
||||
└─ security-scan
|
||||
├─ ✅ Pass (no critical issues)
|
||||
└─ ⚠️ Warning (review, non-blocking)
|
||||
```
|
||||
|
||||
### ArgoCD UI
|
||||
|
||||
```
|
||||
https://argocd.riotpiao.com
|
||||
├─ homelab-root
|
||||
│ ├─ Status: Synced ✓
|
||||
│ ├─ Health: Healthy ✓
|
||||
│ └─ Details (click to see resources)
|
||||
├─ layer-1-bootstrap
|
||||
├─ layer-2-platform
|
||||
├─ layer-3-security
|
||||
├─ layer-4-applications
|
||||
└─ layer-5-data
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Tasks
|
||||
|
||||
### Add new service to cluster
|
||||
|
||||
```bash
|
||||
# 1. Create directory and kustomization.yaml
|
||||
mkdir -p k8s/applications/my-service
|
||||
cat > k8s/applications/my-service/kustomization.yaml << EOF
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
namespace: my-namespace
|
||||
helmCharts:
|
||||
- name: my-chart
|
||||
repo: https://charts.example.com
|
||||
version: 1.0.0
|
||||
releaseName: my-service
|
||||
valuesFile: values.yaml
|
||||
EOF
|
||||
|
||||
# 2. Add values.yaml
|
||||
cp /template/values.yaml k8s/applications/my-service/
|
||||
|
||||
# 3. Commit and push
|
||||
git add k8s/applications/my-service/
|
||||
git commit -m "feat(apps): add my-service"
|
||||
git push
|
||||
|
||||
# 4. CI validates
|
||||
# 5. Merge to main
|
||||
# 6. ArgoCD syncs automatically
|
||||
# ✓ Service deployed to cluster
|
||||
```
|
||||
|
||||
### Rollback a deployment
|
||||
|
||||
```bash
|
||||
# 1. Find broken commit
|
||||
git log --oneline k8s/ # Identify bad commit
|
||||
|
||||
# 2. Revert
|
||||
git revert <commit-hash>
|
||||
git push
|
||||
|
||||
# 3. CI validates (should pass)
|
||||
# 4. Merge to main
|
||||
# 5. ArgoCD syncs back to previous version
|
||||
# ✓ Cluster state reverted
|
||||
```
|
||||
|
||||
### Emergency: Disable ArgoCD auto-sync
|
||||
|
||||
```bash
|
||||
# If production broken and need time to debug:
|
||||
argocd app set homelab-root --sync-policy none
|
||||
|
||||
# Fix issue in git
|
||||
# Test locally: kustomize build k8s/
|
||||
|
||||
# Re-enable
|
||||
argocd app set homelab-root --sync-policy automated
|
||||
argocd app sync homelab-root
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring & Alerts
|
||||
|
||||
### Check workflow status in Forgejo
|
||||
|
||||
```bash
|
||||
# Dashboard shows:
|
||||
✅ All green → Safe to merge
|
||||
❌ Red → Fix required before merge
|
||||
⏳ Yellow → Still running (wait)
|
||||
```
|
||||
|
||||
### Check ArgoCD status
|
||||
|
||||
```bash
|
||||
argocd app list
|
||||
# Shows: Synced, OutOfSync, Unknown status
|
||||
|
||||
argocd app get homelab-root
|
||||
# Shows: health, sync status, resources
|
||||
|
||||
argocd app logs homelab-root --follow
|
||||
# Real-time logs during sync
|
||||
```
|
||||
|
||||
### Alerts (optional, future)
|
||||
|
||||
```yaml
|
||||
# Could add Forgejo webhooks → Slack/email
|
||||
# When CI/CD fails → Alert ops team
|
||||
# When ArgoCD goes OutOfSync → Alert ops team
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Workflow doesn't trigger
|
||||
|
||||
**Check:**
|
||||
- Is Forgejo runner running? `forgejo-runner daemon`
|
||||
- Did you push to correct branch? (validate runs on all, argocd-sync only on main)
|
||||
- Did path match filter? (must change k8s/ or .forgejo/workflows/)
|
||||
|
||||
### Workflow hangs/times out
|
||||
|
||||
**Check:**
|
||||
- kustomize build → Check for dependency cycles
|
||||
- argocd sync → Check cluster resources (storage full? network down?)
|
||||
- security scan → Large image scan → Takes time
|
||||
|
||||
**Fix:**
|
||||
- Increase timeout in workflow
|
||||
- Optimize kustomization (remove unused resources)
|
||||
- Add resource limits to pods
|
||||
|
||||
### ArgoCD token invalid
|
||||
|
||||
**Fix:**
|
||||
```bash
|
||||
# Regenerate token
|
||||
argocd account generate-token
|
||||
|
||||
# Update Forgejo secret
|
||||
# Settings → Secrets → ARGOCD_AUTH_TOKEN = <new-token>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
✅ **DO:**
|
||||
- Commit all K8s changes to git (no manual kubectl apply)
|
||||
- Run validate-k8s locally before push
|
||||
- Write descriptive commit messages (why this change?)
|
||||
- Review workflow logs before merging
|
||||
- Monitor ArgoCD sync after merge
|
||||
|
||||
❌ **DON'T:**
|
||||
- Push directly to main (always use PR)
|
||||
- Skip workflow validation (it catches errors early)
|
||||
- Ignore security scan warnings
|
||||
- Manually `kubectl apply` (breaks GitOps)
|
||||
- Edit resources in cluster (they revert via ArgoCD)
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Setup Forgejo runner** (if not already running)
|
||||
2. **Add ArgoCD secrets** to Forgejo
|
||||
3. **Test workflows** on feature branch
|
||||
4. **Merge to main** → Watch ArgoCD sync
|
||||
5. **Celebrate:** Full GitOps pipeline working! 🎉
|
||||
@@ -1,270 +0,0 @@
|
||||
# Kubernetes RBAC → Authentik Migration Plan
|
||||
|
||||
**Status:** Draft — awaiting decisions in [Open Decisions](#open-decisions)
|
||||
**Date:** 2026-08-19
|
||||
**Scope:** Move human identity + authorization for the cluster and its apps onto Authentik groups.
|
||||
|
||||
---
|
||||
|
||||
## 1. Finding: there is nothing to literally import
|
||||
|
||||
The original ask was "import the existing RBAC from the cluster into Authentik". Inventory of the live
|
||||
cluster says that set is empty.
|
||||
|
||||
### 1.1 Cluster RBAC audit (live, 2026-08-19)
|
||||
|
||||
Every `ClusterRoleBinding` / `RoleBinding` carrying a `User` or `Group` subject:
|
||||
|
||||
| Binding | Role | Subjects |
|
||||
|---|---|---|
|
||||
| `cluster-admin` | `ClusterRole/cluster-admin` | `Group:system:masters` |
|
||||
| `system-bootstrap-approve-node-client-csr` | `ClusterRole/system:certificates.k8s.io:certificatesigningrequests:nodeclient` | `Group:system:bootstrappers:nodes` |
|
||||
| `system-bootstrap-node-bootstrapper` | `ClusterRole/system:node-bootstrapper` | `Group:system:bootstrappers:nodes`, `Group:system:nodes` |
|
||||
| `system-bootstrap-node-renewal` | `ClusterRole/system:certificates.k8s.io:certificatesigningrequests:selfnodeclient` | `Group:system:nodes` |
|
||||
| `kube-system/system::extension-apiserver-authentication-reader` | `Role/extension-apiserver-authentication-reader` | `User:system:kube-controller-manager`, `User:system:kube-scheduler` |
|
||||
| `kube-system/system::leader-locking-kube-controller-manager` | `Role/…` | `User:system:kube-controller-manager` |
|
||||
| `kube-system/system::leader-locking-kube-scheduler` | `Role/…` | `User:system:kube-scheduler` |
|
||||
|
||||
All of these are Talos/control-plane machinery. Every other binding in the cluster targets a
|
||||
`ServiceAccount` (controllers: Argo CD, cert-manager, Cilium, Longhorn, MinIO operator, Prometheus, …).
|
||||
|
||||
**There are zero human users in Kubernetes RBAC.** The only human access path today is the Talos-issued
|
||||
admin certificate in `cluster-config/kubeconfig`, which lands in `system:masters` — a single shared,
|
||||
unattributable, non-revocable god credential.
|
||||
|
||||
The kube-apiserver has no OIDC configured. `apiServer:` in
|
||||
[`terraform/templates/controlplane.tftpl:110`](terraform/templates/controlplane.tftpl#L110) contains only
|
||||
`certSANs`, `image`, `admissionControl` and `auditPolicy`.
|
||||
|
||||
### 1.2 What human RBAC *does* exist (app level, scattered)
|
||||
|
||||
| Location | Rule |
|
||||
|---|---|
|
||||
| [`k8s/bootstrap/phase4-argocd/argocd-values.yaml:163-169`](k8s/bootstrap/phase4-argocd/argocd-values.yaml#L163-L169) | `policy.default: role:readonly`; `g, admin/rock/cicd, role:admin`; `g, homelab-admins, role:admin` |
|
||||
| [`k8s/infra/logging/grafana-values.yaml:86`](k8s/infra/logging/grafana-values.yaml#L86) | `role_attribute_path: contains(groups[*], 'grafana-admins') && 'Admin' \|\| 'Viewer'` |
|
||||
| [`k8s/infra/iam/scripts/authentik-provision.py`](k8s/infra/iam/scripts/authentik-provision.py) (`minio` scope mapping) | `homelab-admins → consoleAdmin`, everyone else → `readonly` |
|
||||
| Authentik, provisioned | Groups `homelab-admins` (superuser), `grafana-admins`. Single user: `rock`, member of both. |
|
||||
|
||||
### 1.3 Known drift
|
||||
|
||||
[`project-usage/authentik-oidc.md`](project-usage/authentik-oidc.md) documents three groups —
|
||||
`homelab-admins`, `homelab-devops`, `homelab-viewers`. Only `homelab-admins` and `grafana-admins` are
|
||||
actually created by the provisioner. The documentation is wrong today; Phase 1 fixes it.
|
||||
|
||||
### 1.4 Restated goal
|
||||
|
||||
Not an import. This is: **define the group taxonomy in Authentik, make Authentik the sole issuer of human
|
||||
identity, then bind those groups to Kubernetes roles and app roles declaratively through GitOps.**
|
||||
|
||||
---
|
||||
|
||||
## 2. Constraints
|
||||
|
||||
- **GitOps only.** No local `terraform apply`, no manual `kubectl apply`, no Authentik console clicking for
|
||||
anything that must survive a rebuild. Changes land in git; Forgejo Actions and Argo CD do the applying.
|
||||
- **Authentik config stays Argo CD-managed** via the `authentik-provision.py` PostSync hook. No Terraform
|
||||
Authentik provider. A prior attempt at one is abandoned at
|
||||
`.claude/worktrees/debug-auth-issue/terraform/authentik-config.tf` — do not revive it.
|
||||
- **Terraform is limited to Talos machine config** (Phase 3 only), applied through the pipeline.
|
||||
- **The Talos admin certificate stays as break-glass.** It is unaffected by OIDC and remains the recovery
|
||||
path when Authentik is unavailable.
|
||||
|
||||
---
|
||||
|
||||
## 3. Phases
|
||||
|
||||
Each phase is independently shippable and independently revertible.
|
||||
|
||||
### Phase 1 — Group taxonomy
|
||||
|
||||
Extend the provisioner's group step (`[2/5]` in `authentik-provision.py`) from two groups to four:
|
||||
|
||||
| Group | `is_superuser` | Intent |
|
||||
|---|---|---|
|
||||
| `homelab-admins` | `true` | Full cluster + full app admin |
|
||||
| `homelab-devops` | `false` | Deploy and operate workloads; no IAM, no cluster config |
|
||||
| `homelab-viewers` | `false` | Read-only everywhere |
|
||||
| `grafana-admins` | `false` | Grafana `Admin` role specifically |
|
||||
|
||||
Files: `k8s/infra/iam/scripts/authentik-provision.py`, `project-usage/authentik-oidc.md` (fix §1.3 drift).
|
||||
|
||||
**Verify**
|
||||
```bash
|
||||
kubectl -n iam exec deploy/authentik-server -- \
|
||||
curl -sH "Authorization: Bearer $TOKEN" localhost:9000/api/v3/core/groups/ \
|
||||
| jq -r '.results[].name'
|
||||
# expect: homelab-admins, homelab-devops, homelab-viewers, grafana-admins
|
||||
# and: rock still a member of homelab-admins + grafana-admins
|
||||
```
|
||||
|
||||
**Rollback:** revert the commit. Extra groups with no bindings grant nothing.
|
||||
|
||||
---
|
||||
|
||||
### Phase 2 — Authentik OAuth2 provider for Kubernetes
|
||||
|
||||
Add a `kubernetes` entry to the `SERVICES` dict in `authentik-provision.py`, with two departures from the
|
||||
existing web-app entries:
|
||||
|
||||
- `client_type: public` — kubectl is a public client and cannot hold a secret. No k8s Secret is created for
|
||||
it, so the `client_secret_source` machinery is skipped for this service.
|
||||
- `grant_types: ["authorization_code", "refresh_token", "urn:ietf:params:oauth:grant-type:device_code"]`
|
||||
plus PKCE, to support both browser and headless login.
|
||||
- `redirect_uris`: `http://localhost:8000`, `http://localhost:18000` (kubelogin's loopback listeners).
|
||||
|
||||
Reuses the existing `groups` property mapping created in step `[1/5]`. No new claim work.
|
||||
|
||||
**Verify**
|
||||
```bash
|
||||
curl -s https://authentik.riotpiao.com/application/o/kubernetes/.well-known/openid-configuration | jq .
|
||||
|
||||
# device-code login, then decode:
|
||||
kubectl oidc-login get-token --oidc-issuer-url=... --oidc-client-id=kubernetes --oidc-extra-scope=groups
|
||||
# id_token payload must contain: preferred_username, groups[]
|
||||
```
|
||||
|
||||
**Rollback:** delete provider + application in Authentik; nothing else consumes them yet.
|
||||
|
||||
---
|
||||
|
||||
### Phase 3 — Enable OIDC on kube-apiserver (Terraform, pipeline-applied)
|
||||
|
||||
Add to `apiServer.extraArgs` in
|
||||
[`terraform/templates/controlplane.tftpl`](terraform/templates/controlplane.tftpl#L110):
|
||||
|
||||
```yaml
|
||||
extraArgs:
|
||||
oidc-issuer-url: https://authentik.riotpiao.com/application/o/kubernetes/
|
||||
oidc-client-id: kubernetes
|
||||
oidc-username-claim: preferred_username
|
||||
oidc-username-prefix: "authentik:"
|
||||
oidc-groups-claim: groups
|
||||
oidc-groups-prefix: "authentik:"
|
||||
```
|
||||
|
||||
> **Security — the prefixes are mandatory, not cosmetic.**
|
||||
> Without `oidc-groups-prefix`, anyone able to create or rename an Authentik group to `system:masters`
|
||||
> obtains `cluster-admin` on the cluster. Without `oidc-username-prefix`, an Authentik username can
|
||||
> impersonate a built-in identity such as `system:kube-controller-manager`. Both prefixes must be present
|
||||
> in the same change that enables OIDC. Do not ship this phase partially.
|
||||
|
||||
Operational notes:
|
||||
|
||||
- Apply is a rolling control-plane machine-config change: one node at a time, wait for `Ready` and for
|
||||
etcd quorum before proceeding to the next.
|
||||
- The Talos admin certificate authenticates via client cert, not OIDC, and is unaffected. It is the
|
||||
recovery path if the issuer URL is wrong or Authentik is down.
|
||||
- The issuer must be reachable from the control-plane nodes and its TLS chain must be trusted by them.
|
||||
Confirm this before applying, because a bad issuer URL means the apiserver logs OIDC discovery failures
|
||||
on every start.
|
||||
|
||||
**Verify** (per node, after each rolls)
|
||||
```bash
|
||||
kubectl auth whoami --token="$ID_TOKEN"
|
||||
# expect Username: authentik:rock
|
||||
# Groups: authentik:homelab-admins, authentik:grafana-admins, system:authenticated
|
||||
|
||||
kubectl --kubeconfig cluster-config/kubeconfig get nodes # cert path still works
|
||||
```
|
||||
|
||||
**Rollback:** revert the template change, re-run the pipeline, roll the control plane back. Cert-based
|
||||
access is never interrupted, so this rollback is safe at any point.
|
||||
|
||||
---
|
||||
|
||||
### Phase 4 — Bind Authentik groups to Kubernetes roles
|
||||
|
||||
New manifest `k8s/infra/iam/rbac-oidc-bindings.yaml`, referenced from
|
||||
[`k8s/infra/iam/kustomization.yaml`](k8s/infra/iam/kustomization.yaml):
|
||||
|
||||
| Subject | Role | Kind |
|
||||
|---|---|---|
|
||||
| `Group:authentik:homelab-admins` | `ClusterRole/cluster-admin` | ClusterRoleBinding |
|
||||
| `Group:authentik:homelab-devops` | `ClusterRole/edit` | see [Open Decisions](#open-decisions) #2 |
|
||||
| `Group:authentik:homelab-viewers` | `ClusterRole/view` | ClusterRoleBinding |
|
||||
|
||||
`admin`, `edit` and `view` are the built-in aggregated ClusterRoles and already exist in the cluster — no
|
||||
custom roles needed.
|
||||
|
||||
**Verify**
|
||||
```bash
|
||||
kubectl auth can-i '*' '*' --all-namespaces --as=probe --as-group=authentik:homelab-admins # yes
|
||||
kubectl auth can-i delete pods -A --as=probe --as-group=authentik:homelab-viewers # no
|
||||
kubectl auth can-i get pods -n apps --as=probe --as-group=authentik:homelab-viewers # yes
|
||||
kubectl auth can-i get secrets -n iam --as=probe --as-group=authentik:homelab-devops # no
|
||||
```
|
||||
|
||||
**Rollback:** revert the commit; Argo CD prunes the bindings.
|
||||
|
||||
---
|
||||
|
||||
### Phase 5 — Client wiring and documentation
|
||||
|
||||
- Ship a kubeconfig template using the `kubelogin` (`kubectl oidc-login`) exec credential plugin — no
|
||||
embedded certs, no long-lived token on disk.
|
||||
- Document login, token cache location, and the break-glass cert path in
|
||||
[`project-usage/authentik-oidc.md`](project-usage/authentik-oidc.md).
|
||||
- Same edit corrects the stale group list from §1.3.
|
||||
|
||||
**Verify:** on a clean machine with no certificates, `kubectl get nodes` triggers a browser login and
|
||||
succeeds; `kubectl auth whoami` reports the `authentik:`-prefixed identity.
|
||||
|
||||
---
|
||||
|
||||
### Phase 6 — Converge app RBAC on the same four groups
|
||||
|
||||
| Target | Change |
|
||||
|---|---|
|
||||
| Argo CD `policy.csv` | add `g, homelab-devops, role:admin` (or a scoped custom role) and `g, homelab-viewers, role:readonly` |
|
||||
| Grafana `role_attribute_path` | three-tier: `grafana-admins`/`homelab-admins` → `Admin`, `homelab-devops` → `Editor`, else `Viewer` |
|
||||
| MinIO `policy` claim expression | `homelab-admins` → `consoleAdmin`, `homelab-devops` → `readwrite`, else `readonly` |
|
||||
| Argo CD local accounts | retire `accounts.rock` once SSO admin is proven. **Keep `accounts.cicd`** — the pipeline needs a non-SSO apiKey. |
|
||||
|
||||
**Verify:** end-to-end browser login per app as a test user who is *only* in `homelab-devops`, confirming
|
||||
the expected role in each of Argo CD, Grafana and MinIO. The e2e suite under
|
||||
[`tests/e2e/tests/authentik.spec.ts`](tests/e2e/tests/authentik.spec.ts) is the place to encode this.
|
||||
|
||||
**Rollback:** per-app revert; each app's mapping is independent.
|
||||
|
||||
---
|
||||
|
||||
## 4. Risks
|
||||
|
||||
| Risk | Severity | Mitigation |
|
||||
|---|---|---|
|
||||
| Missing `oidc-groups-prefix` → group-name privilege escalation to `cluster-admin` | Critical | Prefixes ship in the same commit as the OIDC flags; Phase 4 verification asserts the prefixed form |
|
||||
| Bad issuer URL / untrusted TLS bricks apiserver OIDC | Medium | Cert-based admin access is unaffected; roll one control-plane node at a time and check apiserver logs before continuing |
|
||||
| Authentik outage blocks all human cluster access | Medium | Talos admin cert is the documented break-glass path and never depends on Authentik |
|
||||
| Argo CD `cicd` account removed by accident → pipeline loses cluster access | Medium | Phase 6 explicitly retires only `accounts.rock` |
|
||||
| Group renamed in Authentik silently revokes cluster access | Low | Group names are asserted by the provisioner on every Argo CD sync, so drift self-heals |
|
||||
|
||||
---
|
||||
|
||||
## 5. Open Decisions
|
||||
|
||||
1. **Is kubectl-via-OIDC actually wanted?**
|
||||
Phases 3–5 only pay for themselves if humans besides the shared admin cert need API access. If the
|
||||
answer is no, the plan collapses to Phases 1 and 6: no Terraform change, no control-plane roll, app-level
|
||||
SSO consolidation only.
|
||||
|
||||
2. **`homelab-devops` blast radius** — cluster-wide `ClusterRoleBinding` to `edit`, or namespaced
|
||||
`RoleBinding`s limited to application namespaces (excluding `iam`, `kube-system`, `argocd`)?
|
||||
Namespaced is the tighter default; cluster-wide is less to maintain.
|
||||
|
||||
3. **Username claim** — `preferred_username` (human-readable in audit logs, mutable) versus `sub`
|
||||
(stable, opaque). Recommendation: `preferred_username`, given a single-operator homelab where audit
|
||||
readability beats rename-safety.
|
||||
|
||||
---
|
||||
|
||||
## 6. Reference
|
||||
|
||||
- Kubernetes OIDC authentication: https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens
|
||||
- Kubernetes RBAC: https://kubernetes.io/docs/reference/access-authn-authz/rbac/
|
||||
- Talos apiserver `extraArgs`: https://www.talos.dev/latest/reference/configuration/v1alpha1/config/#Config.cluster.apiServer
|
||||
- Authentik OAuth2 provider: https://docs.goauthentik.io/docs/add-secure-apps/providers/oauth2/
|
||||
- Authentik property mappings (scope/claims): https://docs.goauthentik.io/docs/add-secure-apps/providers/property-mappings/
|
||||
- kubelogin (`kubectl oidc-login`): https://github.com/int128/kubelogin
|
||||
- Argo CD RBAC: https://argo-cd.readthedocs.io/en/stable/operator-manual/rbac/
|
||||
- Grafana generic OAuth: https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-authentication/generic-oauth/
|
||||
- MinIO OpenID identity management: https://min.io/docs/minio/linux/administration/identity-access-management/oidc-access-management.html
|
||||
@@ -1,183 +0,0 @@
|
||||
# Authentik Federated OIDC & SSO
|
||||
|
||||
**Provider:** `https://authentik.riotpiao.com`
|
||||
**OIDC Issuer:** `https://authentik.riotpiao.com/application/o/talos-federation/`
|
||||
**Namespace:** `iam`
|
||||
|
||||
## When to Use
|
||||
|
||||
- **Federated login** — Single sign-on for Grafana, MinIO, Forgejo, Argo CD
|
||||
- **User groups** — RBAC via group membership (admins, devops, read-only)
|
||||
- **JWT tokens** — Authenticate CLI tools, API clients
|
||||
- **SSO for custom apps** — OAuth2/OIDC redirect flow
|
||||
|
||||
## Quick Start
|
||||
|
||||
**1. Login to Authentik console:**
|
||||
```bash
|
||||
# Browser: https://authentik.riotpiao.com
|
||||
# Default user: akadmin
|
||||
# Password: AUTHENTIK_BOOTSTRAP_PASSWORD (from .env)
|
||||
|
||||
# Or via OIDC (after initial setup)
|
||||
# Click "Sign in with talos-federation"
|
||||
```
|
||||
|
||||
**2. Create user:**
|
||||
```
|
||||
Authentik console → Users → Create
|
||||
- Username: alice
|
||||
- Email: [email protected]
|
||||
- Group: homelab-devs (or homelab-admins)
|
||||
```
|
||||
|
||||
**3. User logs into Grafana:**
|
||||
```
|
||||
https://grafana.riotpiao.com
|
||||
→ Sign in with Authentik (auto-redirects to OIDC provider)
|
||||
→ Approve access
|
||||
→ Logged in as alice (group determines role: Admin or Viewer)
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
| Key | Value |
|
||||
|-----|-------|
|
||||
| OIDC provider | `talos-federation` (federated) |
|
||||
| OIDC issuer | `https://authentik.riotpiao.com/application/o/talos-federation/` |
|
||||
| JWKS endpoint | `https://authentik.riotpiao.com/application/o/talos-federation/.well-known/openid-configuration` |
|
||||
| Database | PostgreSQL (ddb namespace, authentik user) |
|
||||
| Backups | WAL archived to MinIO |
|
||||
|
||||
## Common Patterns
|
||||
|
||||
**Grafana OIDC login:**
|
||||
```yaml
|
||||
# k8s/logging/grafana-values.yaml
|
||||
grafana:
|
||||
auth.generic_oauth:
|
||||
enabled: true
|
||||
name: Authentik
|
||||
client_id: grafana
|
||||
client_secret: $GRAFANA_OIDC_CLIENT_SECRET # from Vault
|
||||
auth_url: https://authentik.riotpiao.com/application/o/authorize/
|
||||
token_url: https://authentik.riotpiao.com/application/o/token/
|
||||
api_url: https://authentik.riotpiao.com/application/o/userinfo/
|
||||
scopes: openid profile email groups
|
||||
use_pkce: true
|
||||
```
|
||||
|
||||
**MinIO OIDC login:**
|
||||
```yaml
|
||||
# k8s/storage/minio-values.yaml
|
||||
minio:
|
||||
identity_oauth:
|
||||
provider: authentik
|
||||
client_id: minio
|
||||
client_secret: $MINIO_OIDC_CLIENT_SECRET
|
||||
redirect_uri: https://minio.riotpiao.com/oauth_callback
|
||||
config_url: https://authentik.riotpiao.com/application/o/talos-federation/.well-known/openid-configuration
|
||||
policy_mappings:
|
||||
- group: homelab-admins → consoleAdmin
|
||||
- group: homelab-devops → readwrite
|
||||
```
|
||||
|
||||
**CLI device code flow (core CLI):**
|
||||
```bash
|
||||
# Get JWT token (no kubeconfig needed)
|
||||
core secrets login
|
||||
# → Opens browser, approve device code
|
||||
# → Token cached in ~/.core/token
|
||||
|
||||
# Use token to access Vault
|
||||
core get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
|
||||
# → Vault validates JWT from Authentik
|
||||
# → Returns secret
|
||||
```
|
||||
|
||||
**Custom app OIDC redirect:**
|
||||
```go
|
||||
import "github.com/coreos/go-oidc/v3/oidc"
|
||||
|
||||
provider, _ := oidc.NewProvider(ctx, "https://authentik.riotpiao.com/application/o/talos-federation/")
|
||||
|
||||
verifier := provider.Verifier(&oidc.Config{ClientID: "my-app"})
|
||||
|
||||
// After OAuth2 redirect & token exchange:
|
||||
idToken, _ := verifier.Verify(ctx, rawIDToken)
|
||||
|
||||
// Extract claims
|
||||
var claims struct {
|
||||
Email string `json:"email"`
|
||||
Groups []string `json:"groups"`
|
||||
}
|
||||
idToken.Claims(&claims)
|
||||
```
|
||||
|
||||
## Group-Based RBAC
|
||||
|
||||
**Default groups:**
|
||||
- `homelab-admins` — Full cluster access (Grafana Admin, MinIO admin, Argo CD admin, Vault admin)
|
||||
- `homelab-devops` — Deploy & monitor (Grafana Editor, MinIO readwrite, Argo CD user)
|
||||
- `homelab-viewers` — Read-only (Grafana Viewer, MinIO readonly)
|
||||
|
||||
**Assign user to group:**
|
||||
```
|
||||
Authentik console → Users → alice → Edit
|
||||
→ Groups → Add "homelab-devops"
|
||||
→ Save
|
||||
```
|
||||
|
||||
**Custom group-to-role mapping:**
|
||||
```yaml
|
||||
# Per-service (see cicd-workflow.md, monitoring-metrics.md for examples)
|
||||
# Grafana: auth.generic_oauth.role_attribute_path = contains(groups[*], 'homelab-admins') && 'Admin' || 'Viewer'
|
||||
# MinIO: policy_mappings (see above)
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
**Authentik dashboard:** https://authentik.riotpiao.com/api/v3/admin/dashboards
|
||||
|
||||
**Key metrics:**
|
||||
- Login attempts (success/failure)
|
||||
- Active sessions
|
||||
- Token issuance rate
|
||||
- Provider sync status
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Users can't login (redirect loop):**
|
||||
```bash
|
||||
# Check redirect URI matches
|
||||
# Authentik console → Applications → grafana → Edit
|
||||
# Verify Redirect URI = https://grafana.riotpiao.com/login/generic_oauth
|
||||
|
||||
# Check OIDC provider is running
|
||||
k get pods -n iam -l app=authentik
|
||||
```
|
||||
|
||||
**JWT token expired:**
|
||||
```bash
|
||||
# CLI tokens have 24h expiry
|
||||
# Re-authenticate
|
||||
core secrets login
|
||||
```
|
||||
|
||||
**Groups not syncing:**
|
||||
```bash
|
||||
# Check group attribute in OIDC config
|
||||
# Authentik console → Applications → <app> → OIDC Configuration
|
||||
# groups_attribute = "groups" (or custom claim name)
|
||||
```
|
||||
|
||||
**Vault can't validate JWT:**
|
||||
```bash
|
||||
# Verify JWKS endpoint is accessible
|
||||
curl https://authentik.riotpiao.com/application/o/talos-federation/.well-known/openid-configuration
|
||||
|
||||
# Restart Vault to refresh JWKS cache
|
||||
k rollout restart -n iam deployment/vault
|
||||
```
|
||||
|
||||
See `/TROUBLESHOOTING.md` for full incident guide.
|
||||
@@ -1,222 +0,0 @@
|
||||
# CI/CD Pipeline (Forgejo + Argo CD)
|
||||
|
||||
**Git Forge:** `https://forgejo.riotpiao.com`
|
||||
**Deployments:** `https://argocd.riotpiao.com` (or `kubectl port-forward`)
|
||||
**Namespaces:** `cicd`, `forge`
|
||||
|
||||
## When to Use
|
||||
|
||||
- **Build & test** — Forgejo Actions CI (GitHub Actions syntax)
|
||||
- **Image push** — Build OCI images, push to Forgejo registry
|
||||
- **GitOps deployment** — Argo CD syncs deploy repo to cluster
|
||||
- **Secrets in CI** — ci-bot JWT tokens, never kubeconfig
|
||||
|
||||
## Quick Start
|
||||
|
||||
**1. Clone a repo from Forgejo:**
|
||||
```bash
|
||||
git clone https://forgejo.riotpiao.com/rock/source.git
|
||||
cd source
|
||||
```
|
||||
|
||||
**2. Create workflow:**
|
||||
```bash
|
||||
mkdir -p .forgejo/workflows
|
||||
cat > .forgejo/workflows/ci.yml <<EOF
|
||||
name: CI
|
||||
on: [push]
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- run: npm test
|
||||
- run: docker build -t myapp:latest .
|
||||
- run: |
|
||||
docker login forgejo.riotpiao.com \
|
||||
-u ci-bot \
|
||||
-p ${{ secrets.CI_BOT_TOKEN }}
|
||||
docker push forgejo.riotpiao.com/rock/myapp:latest
|
||||
EOF
|
||||
|
||||
git add .forgejo/workflows/ci.yml
|
||||
git commit -m "ci: add build workflow"
|
||||
git push
|
||||
```
|
||||
|
||||
**3. Trigger deployment:**
|
||||
```bash
|
||||
# Update deployment repo (rock/deploy)
|
||||
git clone https://forgejo.riotpiao.com/rock/deploy.git
|
||||
cd deploy
|
||||
|
||||
# Update image tag
|
||||
sed -i 's|forgejo.riotpiao.com/rock/myapp:.*|forgejo.riotpiao.com/rock/myapp:abc123|' k8s/deployment.yaml
|
||||
|
||||
git add k8s/deployment.yaml
|
||||
git commit -m "deploy: bump myapp to abc123"
|
||||
git push
|
||||
```
|
||||
|
||||
**4. Argo CD auto-syncs:**
|
||||
```bash
|
||||
# Watch deployment
|
||||
k rollout status -n <app-namespace> deployment/<app-name>
|
||||
|
||||
# Or check Argo CD UI
|
||||
kubectl port-forward -n argocd svc/argocd-server 8443:443
|
||||
# https://localhost:8443 (login via Authentik)
|
||||
```
|
||||
|
||||
## Workflow Syntax (GitHub Actions)
|
||||
|
||||
**Basic structure:**
|
||||
```yaml
|
||||
name: CI
|
||||
on:
|
||||
push:
|
||||
branches: [main, develop]
|
||||
pull_request:
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- run: npm ci
|
||||
- run: npm test
|
||||
|
||||
build:
|
||||
needs: test # wait for test job
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- run: docker build -t myapp:${{ github.sha }} .
|
||||
- run: docker push forgejo.riotpiao.com/rock/myapp:${{ github.sha }}
|
||||
```
|
||||
|
||||
**Available variables:**
|
||||
```bash
|
||||
${{ github.sha }} # commit hash
|
||||
${{ github.ref_name }} # branch name
|
||||
${{ github.run_number }} # build #
|
||||
${{ secrets.CI_BOT_TOKEN }} # injected from Forgejo
|
||||
```
|
||||
|
||||
## Secrets & Authentication
|
||||
|
||||
**ci-bot JWT token (auto-injected):**
|
||||
```yaml
|
||||
- run: |
|
||||
echo "${{ secrets.CI_BOT_TOKEN }}" | docker login \
|
||||
forgejo.riotpiao.com \
|
||||
-u ci-bot \
|
||||
--password-stdin
|
||||
docker push forgejo.riotpiao.com/rock/myapp:latest
|
||||
```
|
||||
|
||||
**API token (for pushing commits):**
|
||||
```yaml
|
||||
- run: |
|
||||
git config user.name "ci-bot"
|
||||
git config user.email "ci-bot@homelab"
|
||||
git commit --allow-empty -m "bump: version"
|
||||
git push https://ci-bot:${{ secrets.CI_BOT_TOKEN }}@forgejo.riotpiao.com/rock/deploy.git main
|
||||
```
|
||||
|
||||
**Vault secrets (via talos CLI):**
|
||||
```bash
|
||||
# Not available in CI runner — use Argo CD post-sync hooks instead
|
||||
# Or inject via init container before workflow runs
|
||||
```
|
||||
|
||||
## Argo CD GitOps
|
||||
|
||||
**Create app (one-time):**
|
||||
```bash
|
||||
argocd app create story-crater \
|
||||
--repo https://forgejo.riotpiao.com/rock/deploy.git \
|
||||
--path k8s/ \
|
||||
--dest-server https://kubernetes.default.svc \
|
||||
--dest-namespace story-crater-backend \
|
||||
--sync-policy automated
|
||||
```
|
||||
|
||||
**Monitor sync:**
|
||||
```bash
|
||||
# CLI
|
||||
argocd app get story-crater
|
||||
argocd app logs story-crater
|
||||
|
||||
# UI: https://argocd.riotpiao.com
|
||||
# Login: Authentik SSO (homelab-admins group only)
|
||||
```
|
||||
|
||||
**Manual sync:**
|
||||
```bash
|
||||
argocd app sync story-crater
|
||||
argocd app wait story-crater
|
||||
```
|
||||
|
||||
## Security Rules
|
||||
|
||||
✅ **DO:**
|
||||
- Store credentials in Forgejo Secrets (auto-injected)
|
||||
- Use ci-bot JWT for image push only
|
||||
- Commit to deploy repo (triggers Argo CD)
|
||||
- Enable branch protection (require CI pass)
|
||||
|
||||
❌ **DON'T:**
|
||||
- Put kubeconfig in CI (Argo CD bridges gap)
|
||||
- Commit secrets to source repo
|
||||
- Use admin-bot in CI workflows (over-privileged)
|
||||
- Push images directly to cluster (use Argo CD)
|
||||
|
||||
## Monitoring
|
||||
|
||||
**Grafana dashboard:** `svc-forgejo`, `svc-argocd`
|
||||
|
||||
**Key metrics:**
|
||||
- `forgejo_workflows_running` — active workflows
|
||||
- `argocd_app_sync_duration_seconds` — deployment time
|
||||
- `argocd_app_info{sync_status="OutOfSync"}` — drift detection
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Workflow fails silently:**
|
||||
```bash
|
||||
# Check runner logs
|
||||
k logs -n cicd -f deploy/forgejo-runner
|
||||
|
||||
# Check if runner pod is healthy
|
||||
k get pods -n cicd -l app=forgejo-runner
|
||||
```
|
||||
|
||||
**Image push fails (401 Unauthorized):**
|
||||
```bash
|
||||
# Verify ci-bot token in Forgejo
|
||||
# Settings → Applications → ci-bot → check scopes (package:write)
|
||||
|
||||
# Or re-create token
|
||||
core put cluster/iam/agents/ci-bot-token TOKEN="$(openssl rand -hex 32)"
|
||||
```
|
||||
|
||||
**Argo CD out-of-sync:**
|
||||
```bash
|
||||
# Check deploy repo changes
|
||||
argocd app diff story-crater
|
||||
|
||||
# Manual sync
|
||||
argocd app sync story-crater --prune
|
||||
```
|
||||
|
||||
**Webhook not triggering:**
|
||||
```bash
|
||||
# Verify Forgejo webhook config
|
||||
# Repo Settings → Webhooks → Check delivery logs
|
||||
|
||||
# Or manually trigger
|
||||
argocd app sync story-crater
|
||||
```
|
||||
|
||||
See `/TROUBLESHOOTING.md` for full incident guide.
|
||||
@@ -1,388 +0,0 @@
|
||||
# Coding Standards — Helm, Helmfile, Kubernetes, Shell
|
||||
|
||||
Conventions for contributing to homelab's Helm charts, helmfiles, Kubernetes manifests, and shell scripts. These are repo-local patterns — not Go/Rust/general standards from companion repos (core CLI, kmsvc).
|
||||
|
||||
---
|
||||
|
||||
## Helm & Helmfile Conventions
|
||||
|
||||
### Chart Naming & Layout
|
||||
|
||||
- **Local charts:** `k8s/<service>/charts/<component>/` directory structure.
|
||||
- **Helm chart versions:** Always use semver ranges in helmfile releases (e.g., `~1.0`, `~10`, not floating `latest`).
|
||||
- Rationale: Predictable upgrades, avoids surprise breaking changes.
|
||||
|
||||
### helmfile.yaml.gotmpl Pattern
|
||||
|
||||
The helmfile is a **Go template**, not a shell script. Use Go template syntax for environment variable interpolation, not shell syntax.
|
||||
|
||||
**Correct:**
|
||||
```yaml
|
||||
set:
|
||||
- name: adminPassword
|
||||
value: {{ env "GRAFANA_ADMIN_PASSWORD" }}
|
||||
```
|
||||
|
||||
**Incorrect:**
|
||||
```yaml
|
||||
set:
|
||||
- name: adminPassword
|
||||
value: ${GRAFANA_ADMIN_PASSWORD} # Shell syntax — not expanded by helmfile
|
||||
```
|
||||
|
||||
**Organization:**
|
||||
- Use logical comment headers to separate sections: `# ── cert-manager ──`.
|
||||
- Group related releases together.
|
||||
- Use `needs:` for dependency ordering (release A waits for release B before deploying).
|
||||
- Example (from helmfile.yaml.gotmpl, lines 439–449):
|
||||
```yaml
|
||||
- name: loki
|
||||
namespace: logging
|
||||
needs:
|
||||
- storage/minio
|
||||
|
||||
- name: promtail
|
||||
namespace: logging
|
||||
needs:
|
||||
- logging/loki
|
||||
```
|
||||
- **Namespace declaration:** Specify namespace at the release block level, not helmfile-level default.
|
||||
```yaml
|
||||
- name: prometheus
|
||||
namespace: monitoring
|
||||
createNamespace: true
|
||||
```
|
||||
|
||||
### Values Files Pattern
|
||||
|
||||
- **Never hardcode secrets** in values.yaml or ConfigMap keys.
|
||||
- Store secrets in Vault via `core put cluster/KEY KEY="value"`.
|
||||
- Reference at deploy time using `{{ env "VAR" }}` in helmfile.
|
||||
- **External values files:** Always use `values:` block pointing to `.yaml` files, not inline YAML.
|
||||
```yaml
|
||||
- name: minio
|
||||
values:
|
||||
- k8s/storage/minio-values.yaml
|
||||
```
|
||||
- **Environment-specific overrides:** For complex releases (e.g., SQS), use `environments/` subdirectory with `helmfile.yaml.gotmpl` at the service level.
|
||||
- Example: `k8s/sqs/environments/homelab.yaml` (lines 1–29 show namespace, kafkaCluster, redis, managementService blocks).
|
||||
- Helmfile loads environment-specific values dynamically: `{{ .Values.kafkaCluster.nodePool.replicas }}`.
|
||||
|
||||
---
|
||||
|
||||
## Infrastructure as Code (IaC) — Single Source of Truth
|
||||
|
||||
**Core principle:** All infrastructure state must be declaratively managed via Terraform or Helm (via helmfile + Terraform). No ad-hoc scripts, manual kubectl, or side-by-side resource definitions.
|
||||
|
||||
### Terraform + Helm Division of Labor
|
||||
|
||||
- **Terraform manages:**
|
||||
- Helm releases (chart + version + values)
|
||||
- Namespaces
|
||||
- StorageClasses
|
||||
- Static Kubernetes resources (RBAC, NetworkPolicies, IngressClasses)
|
||||
- Cloud infrastructure (Vault, S3 backends, secrets)
|
||||
- State persistence (S3 backend in MinIO)
|
||||
|
||||
- **Helmfile manages:**
|
||||
- Chart release ordering via `needs:`
|
||||
- Environment-specific value interpolation (Go templating, not shell)
|
||||
- Hook workflows (pre/post-sync orchestration)
|
||||
- **Never use helmfile for one-off bucket creation, job runs, or manual setup** — those belong in Terraform or a documented bootstrap process
|
||||
|
||||
- **Kubernetes manifests (`k8s/`) manage:**
|
||||
- ArgoCD applications (single source of truth for GitOps)
|
||||
- Service definitions that ArgoCD syncs
|
||||
- **Never manage app objects (Deployments, StatefulSets) directly** — let helm + ArgoCD own them
|
||||
|
||||
### Anti-Pattern: Ad-Hoc Resource Creation
|
||||
|
||||
❌ **Bad:** Separate `minio-buckets.tf` using `aws_s3_bucket` resources + post-deploy scripts
|
||||
- Split responsibility: some buckets in Terraform, others in helmfile, others manual
|
||||
- State drift: unclear what's managed where
|
||||
- Credential duplication: secrets in multiple places
|
||||
|
||||
✅ **Good:** Single source in `minio.tf` helm release:
|
||||
```hcl
|
||||
buckets = [
|
||||
{ name = "terraform-state", policy = "none", purge = false },
|
||||
{ name = "vault", policy = "none", purge = false },
|
||||
...
|
||||
]
|
||||
```
|
||||
- One place to define, one place to audit
|
||||
- Credentials in variables + Vault, not scattered
|
||||
- TF state tracks all changes
|
||||
|
||||
### When to Break the Rule
|
||||
|
||||
Only when **explicitly documented**:
|
||||
- Bootstrap scripts (one-time cluster init) — commit to `scripts/` with clear "run once" warning
|
||||
- Temporary debugging (never leave in git) — stash or delete before committing
|
||||
- Manual steps for constraint (e.g., "create namespace before ArgoCD bootstraps") — document in `TROUBLESHOOTING.md` with rationale
|
||||
|
||||
---
|
||||
|
||||
## YAML & ConfigMap/Secret Patterns
|
||||
|
||||
### Secret Field Naming
|
||||
|
||||
**Rule: Field name in Secret = environment variable name in Vault.**
|
||||
|
||||
When storing a secret via `core put cluster/KEY KEY="value"`, the field name and Vault variable name must match. This ensures helmfile's `{{ env "VAR" }}` expansion works correctly.
|
||||
|
||||
Example (from CLAUDE.md gotcha "Field name = variable name"):
|
||||
```bash
|
||||
# Correct
|
||||
core put cluster/MINIO_ROOT_PASSWORD MINIO_ROOT_PASSWORD="value"
|
||||
core put cluster/GRAFANA_ADMIN_PASSWORD GRAFANA_ADMIN_PASSWORD="value"
|
||||
|
||||
# Incorrect (won't expand in helmfile)
|
||||
core put cluster/MINIO_SECRET value="value" # Field name ≠ variable name
|
||||
```
|
||||
|
||||
Reference in helmfile (helmfile.yaml.gotmpl, lines 400–401):
|
||||
```yaml
|
||||
set:
|
||||
- name: rootPassword
|
||||
value: {{ env "MINIO_ROOT_PASSWORD" }}
|
||||
```
|
||||
|
||||
### Never Use `--env` Flags in Manifests
|
||||
|
||||
Avoid kubectl flags like `--env KEY=value` in manifests or deployment specs. This exposes secrets in `kubectl describe` output.
|
||||
|
||||
**Correct:** Use Secret volumes (k8s/storage/minio-values.yaml, lines 40–42):
|
||||
```yaml
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: minio-oidc # Reference a Secret, don't expose in manifest
|
||||
```
|
||||
|
||||
**Incorrect:**
|
||||
```yaml
|
||||
env:
|
||||
- name: MINIO_OIDC_SECRET
|
||||
value: "sensitive-value" # Visible in kubectl describe
|
||||
```
|
||||
|
||||
### Namespace-First Organization
|
||||
|
||||
Organize manifests by namespace: `k8s/<namespace>/`
|
||||
|
||||
Structure per namespace:
|
||||
```
|
||||
k8s/storage/
|
||||
├── minio-values.yaml
|
||||
├── minio-bucket-init.sh
|
||||
└── (local charts if any)
|
||||
|
||||
k8s/monitoring/
|
||||
├── prometheus-values.yaml
|
||||
├── dashboards/ # ConfigMap files auto-loaded via helmfile postsync hook
|
||||
├── servicemonitors/ # ServiceMonitor CRD instances
|
||||
└── alerts/ # PrometheusRule CRD instances
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Hook Scripts & Integration Workflows
|
||||
|
||||
### Pre/Post-Sync Hooks
|
||||
|
||||
Helmfile hooks (presync/postsync) drive setup workflows. Use **relative paths from repo root**, never absolute paths.
|
||||
|
||||
**Pattern (helmfile.yaml.gotmpl, lines 402–416):**
|
||||
```yaml
|
||||
- name: minio
|
||||
hooks:
|
||||
- events: ["presync"]
|
||||
command: bash
|
||||
args:
|
||||
- -c
|
||||
- |
|
||||
bash k8s/base/namespace-setup.sh storage
|
||||
bash k8s/storage/minio-bucket-init.sh storage loki-chunks loki-ruler
|
||||
```
|
||||
|
||||
**Common presync tasks:**
|
||||
- Create namespace (via `k8s/base/namespace-setup.sh`).
|
||||
- Pre-create ConfigMaps/Secrets for the release.
|
||||
- Initialize infrastructure (buckets, databases, etc.).
|
||||
|
||||
**Common postsync tasks:**
|
||||
- Wait for operator/webhook readiness.
|
||||
- Apply CRD instances (ServiceMonitor, PrometheusRule, Certificate).
|
||||
- Perform post-deployment setup (cluster initialization, user creation).
|
||||
|
||||
Example (helmfile.yaml.gotmpl, lines 64–72):
|
||||
```yaml
|
||||
hooks:
|
||||
- events: ["postsync"]
|
||||
command: bash
|
||||
args:
|
||||
- -c
|
||||
- |
|
||||
kubectl rollout status deploy/cert-manager -n cert-manager --timeout=120s
|
||||
kubectl apply -f - <<'EOF'
|
||||
# ClusterIssuer and Certificate CRD instances...
|
||||
EOF
|
||||
```
|
||||
|
||||
### ServiceMonitor Pattern
|
||||
|
||||
One file per service in `k8s/monitoring/servicemonitors/svc-<name>.yaml`.
|
||||
|
||||
**Key elements:**
|
||||
- `namespaceSelector`: Match the namespace where the app runs.
|
||||
- `selector.matchLabels`: Match the app label from the Deployment (e.g., `app.kubernetes.io/name: argocd-metrics`).
|
||||
- `endpoints.port`: Name of the metrics port in the Service.
|
||||
- `interval`: Scrape frequency (e.g., `30s`).
|
||||
|
||||
Example (`k8s/monitoring/servicemonitors/argocd.yaml`, lines 1–37):
|
||||
```yaml
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: argocd
|
||||
namespace: monitoring
|
||||
labels:
|
||||
release: kube-prometheus-stack # Required: links to Prometheus release
|
||||
spec:
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- cicd # App namespace
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: argocd-metrics # Matches Deployment pod label
|
||||
endpoints:
|
||||
- port: metrics # Service port name (not port number)
|
||||
interval: 30s
|
||||
scrapeTimeout: 10s
|
||||
```
|
||||
|
||||
### PrometheusRule Pattern
|
||||
|
||||
One file per service in `k8s/monitoring/alerts/svc-<name>-rules.yaml`.
|
||||
|
||||
**Structure:**
|
||||
- `groups[].name`: Logical grouping (e.g., `minio.rules`).
|
||||
- `groups[].rules[].alert`: Alert name.
|
||||
- `expr`: PromQL expression (5-minute windows for rate alerts).
|
||||
- `for`: Duration threshold (e.g., `10m`).
|
||||
- `labels.severity`: `critical`, `warning`.
|
||||
- `annotations`: summary + description (use `{{ $value }}` for metric value).
|
||||
|
||||
Example (`k8s/monitoring/alerts/svc-minio-rules.yaml`, lines 1–47):
|
||||
```yaml
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: minio-rules
|
||||
namespace: storage
|
||||
spec:
|
||||
groups:
|
||||
- name: minio.rules
|
||||
interval: 15s
|
||||
rules:
|
||||
- alert: MinIOHighErrorRate
|
||||
expr: |
|
||||
(
|
||||
sum(rate(minio_s3_requests_total{error="true"}[5m]))
|
||||
/
|
||||
sum(rate(minio_s3_requests_total[5m]))
|
||||
) > 0.05
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "High error rate on MinIO"
|
||||
description: "{{ $value | humanizePercentage }}"
|
||||
```
|
||||
|
||||
### Grafana Dashboard Pattern
|
||||
|
||||
Dashboards are stored as JSON ConfigMaps in `k8s/monitoring/dashboards/` and auto-loaded via helmfile postsync hook (see helmfile.yaml.gotmpl, lines 471–477).
|
||||
|
||||
**6-row template (recommended layout):**
|
||||
1. **Availability:** Uptime, error rate, latency (SLO band).
|
||||
2. **Resources:** CPU, memory, disk usage, network I/O.
|
||||
3. **Domain metrics:** Service-specific KPIs (throughput, queue depth, cache hit rate).
|
||||
4. **Logs:** Recent error logs from Loki.
|
||||
5. **SLO:** SLI tracking (burn rate, error budget).
|
||||
6. **Related dashboards:** Links to dependent services.
|
||||
|
||||
Auto-load hook (helmfile.yaml.gotmpl, lines 471–477):
|
||||
```yaml
|
||||
hooks:
|
||||
- events: ["postsync"]
|
||||
command: kubectl
|
||||
args:
|
||||
- apply
|
||||
- -f
|
||||
- k8s/monitoring/dashboards/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Integration Checklist
|
||||
|
||||
When adding a new service to the homelab, verify the following:
|
||||
|
||||
- [ ] **Chart pinning:** Helm chart version pinned (`~1.0` format in helmfile, not floating `latest`).
|
||||
- [ ] **Secrets management:** All secrets stored in Vault (none in values.yaml, ConfigMap, or CLI flags).
|
||||
- [ ] **Metrics endpoint:** Service exports `/metrics` endpoint (Prometheus format).
|
||||
- [ ] **ServiceMonitor:** Created and auto-scraped by Prometheus operator.
|
||||
- [ ] **PrometheusRule:** Alert rules defined for errors, latency, SLO violations.
|
||||
- [ ] **Grafana dashboard:** 6-row template auto-loaded via ConfigMap.
|
||||
- [ ] **Ingress:** Rule added if external access needed (see `k8s/ingress/ingress.yaml`).
|
||||
- [ ] **OIDC integration:** If UI component, integrated with Authentik (see `core iam bootstrap`).
|
||||
|
||||
---
|
||||
|
||||
## Shared/Reusable Repos & Image Publication
|
||||
|
||||
### When to Publish as Public GHCR
|
||||
|
||||
**Rule:** If a service's chart + image source lives in a separate repo (not in homelab), it **must** be published as a **public GitHub repo** under the `Riotpiaole` org.
|
||||
|
||||
**Rationale:**
|
||||
- Local in-cluster charts are fine for infra-owned services.
|
||||
- Shared/reusable service charts should be version-pinned and publicly available for:
|
||||
- Reuse across different clusters (other labs, staging, prod).
|
||||
- Consumption by Argo CD apps (CI/CD pipeline).
|
||||
- Independent evolution without tight coupling to homelab repo.
|
||||
|
||||
### Workflow
|
||||
|
||||
**Develop locally:**
|
||||
1. Create companion repo (e.g., `kafaka-management-service`, `queue-operator`) in a private or public repo.
|
||||
2. Include Dockerfile and Helm chart.
|
||||
|
||||
**Build & publish:**
|
||||
1. Set up CI/CD in the companion repo (GitHub Actions).
|
||||
2. Build image and push to GHCR: `ghcr.io/Riotpiaole/<repo>:<tag>`.
|
||||
3. Tag release and publish chart (npm registry, GitHub releases, or OCI registry).
|
||||
|
||||
**Reference in homelab:**
|
||||
1. Pin image tag + chart version in helmfile.yaml.gotmpl.
|
||||
2. Chart `repositories` block references the public Helm repo (or OCI registry).
|
||||
|
||||
**Example (commit cd1c569 — SQS charts):**
|
||||
```yaml
|
||||
- name: management-service
|
||||
namespace: sqs
|
||||
chart: k8s/sqs/charts/management-service
|
||||
values:
|
||||
- image:
|
||||
repository: ghcr.io/Riotpiaole/kafaka-management-service
|
||||
tag: v1.0.0 # Pinned tag from public build
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cross-References
|
||||
|
||||
- **CLAUDE.md gotchas:** Field name = variable name, MinIO `--env` flag exposure, helmfile template syntax (`{{ env "VAR" }}` not `${VAR}`), CNPG password templating.
|
||||
- **README.md:** kubectl context setup, port-forward aliases, cluster topology.
|
||||
- **USAGE.md:** IAM bootstrap, secret management, deployment recipes.
|
||||
@@ -1,342 +0,0 @@
|
||||
# Core CLI Tools: Decision Guide
|
||||
|
||||
The `core` CLI is your primary tool for cluster auth, node operations, Vault secrets management, IAM administration, and object storage. This doc covers **when** to reach for `core` vs `kubectl`/`helmfile`, the two separate auth domains that power different commands, and the complete command inventory.
|
||||
|
||||
For exhaustive flag-level detail on each subcommand, see [~/workplace/core/USAGE.md](../../core/USAGE.md).
|
||||
|
||||
---
|
||||
|
||||
## The Two Auth Domains
|
||||
|
||||
The `core` CLI maintains **two independent authentication systems** that gate different command families. Confusing them causes authentication failures.
|
||||
|
||||
### Domain 1: Node/Talos Operations (`core auth`)
|
||||
|
||||
**Gates:** Node-level commands, Talos service management, cluster status.
|
||||
|
||||
**Login mechanism:** `core auth login-oob`
|
||||
- Out-of-band device-code flow via Authentik
|
||||
- Token cached in `~/.core/token` (24h expiry)
|
||||
- Required before: `core nodes`, `core status <ip>`, `core services <ip>`, `core logs <ip>`, `core log-svc <ip> <svc>`, `core dmesg <ip>`, `core config`, `core upgrade`, `core shutdown`, `core reboot`, `core node`, `core pods clean`
|
||||
|
||||
**Check auth state:**
|
||||
```bash
|
||||
core auth status # Show token expiry
|
||||
core auth clear # Force re-auth on next command
|
||||
```
|
||||
|
||||
**When to use:**
|
||||
- Troubleshooting node-level issues (crashes, disk space, network on a specific Talos node)
|
||||
- Checking Talos service health (etcd, kubelet, scheduler, etc.)
|
||||
- Upgrading cluster OS or managing node lifecycle
|
||||
- Reading kernel logs (dmesg) or kubelet logs on specific nodes
|
||||
|
||||
### Domain 2: Vault Secrets (`core secrets`)
|
||||
|
||||
**Gates:** Secrets management (read/write), secret listing, secret export.
|
||||
|
||||
**Login mechanism:** `core secrets login`
|
||||
- Interactive login to Vault (opens browser, approves auth)
|
||||
- Reads/writes to `~/.config/talos/secrets.toml`
|
||||
- Required before: `core get <path> [--key KEY]`, `core put <path> key=value`, `core secrets list`, `core secrets export`, `core secrets exec`
|
||||
|
||||
**Check auth state:**
|
||||
```bash
|
||||
core secrets status # Show Vault token / scopes
|
||||
core secrets clear # Force re-auth on next command
|
||||
```
|
||||
|
||||
**When to use:**
|
||||
- Reading or writing cluster secrets (API keys, database passwords, OAuth client secrets)
|
||||
- Managing application credentials in Vault (centralized secret store)
|
||||
- Rotating secrets for services (e.g., OAuth2 client secrets)
|
||||
- Listing all secrets under a path
|
||||
- Bootstrapping `.env` files for helmfile deployment (via `vsource`)
|
||||
|
||||
---
|
||||
|
||||
## Auth Bug Fixes
|
||||
|
||||
**Bug #1 (Fixed):** `core status <NODE>` is a **node-ops command**, not an auth verifier. It does NOT verify that you're authenticated to Vault. To verify auth state, use the correct domain-specific command:
|
||||
- For node access: `core auth status`
|
||||
- For Vault access: `core secrets status`
|
||||
|
||||
**Bug #2 (Fixed):** `core secrets login` and `core auth login-oob` are **NOT interchangeable**. They gate completely different systems:
|
||||
- `core auth login-oob` gates node/cluster operations
|
||||
- `core secrets login` gates Vault secret read/write
|
||||
- You may be authenticated to one domain and not the other. Check state separately.
|
||||
|
||||
---
|
||||
|
||||
## When to Reach for `core`
|
||||
|
||||
| Scenario | Tool | Why |
|
||||
|----------|------|-----|
|
||||
| Pod/deployment issue | `kubectl` | Pods, replicas, rollouts, events |
|
||||
| Package release management | `helmfile` | Install/upgrade Helm charts |
|
||||
| Node crashes, disk, kernel panic | `core auth` + node commands | Direct access to Talos node state |
|
||||
| Service unreachable on Talos node | `core status <ip>`, `core services <ip>` | Talos service health |
|
||||
| Database password rotation | `core secrets` + `core put` | Vault secret write |
|
||||
| Need API key for app deployment | `core secrets` + `core get` | Fetch from Vault, inject via `vsource` |
|
||||
| Create OAuth2 app in Authentik | `core iam create-app` | Manage federated identity apps |
|
||||
| Add user to service (MinIO, Grafana, etc.) | `core iam add-member` | Group membership binding |
|
||||
| S3 object upload/download | `core bucket` | MinIO operations |
|
||||
|
||||
---
|
||||
|
||||
## Command Inventory
|
||||
|
||||
Complete list of `core` subcommands, organized by domain. See [~/workplace/core/USAGE.md](../../core/USAGE.md) for usage flags and examples.
|
||||
|
||||
### Authentication
|
||||
|
||||
```bash
|
||||
# Node/Talos operations auth
|
||||
core auth login-oob # Authenticate with Authentik (device code flow)
|
||||
core auth status # Check token expiry / auth state
|
||||
core auth clear # Clear cached auth token
|
||||
|
||||
# Vault secrets auth
|
||||
core secrets login # Authenticate to Vault (interactive)
|
||||
core secrets status # Check Vault token / scopes / expiry
|
||||
core secrets clear # Clear Vault auth token
|
||||
```
|
||||
|
||||
### Cluster & Node Operations
|
||||
|
||||
```bash
|
||||
# Node discovery and status
|
||||
core nodes # List all cluster nodes (IP, hostname, status)
|
||||
core status <ip> # Talos node overview (resources, uptime)
|
||||
core services <ip> # List Talos services (etcd, kubelet, etc.)
|
||||
|
||||
# Logs
|
||||
core logs <ip> # Stream kubelet logs on node
|
||||
core log-svc <ip> <svc> # Logs for specific Talos service (etcd, scheduler, etc.)
|
||||
core dmesg <ip> # Kernel logs (ring buffer) from node
|
||||
|
||||
# Node lifecycle
|
||||
core upgrade <ip> # Upgrade Talos OS on node
|
||||
core shutdown <ip> # Graceful shutdown
|
||||
core reboot <ip> # Reboot node
|
||||
|
||||
# Kubernetes context
|
||||
core config kube-list # List available kubeconfig contexts
|
||||
core config kube-use <ctx> # Switch to kubeconfig context (LAN vs WireGuard)
|
||||
|
||||
# Pod cleanup
|
||||
core pods clean # Delete Failed/Evicted/Terminating pods
|
||||
core node <node-name> # Get node details (includes pod stats)
|
||||
```
|
||||
|
||||
### Secrets Management (Vault)
|
||||
|
||||
```bash
|
||||
# Read secrets
|
||||
core get <path> # Fetch all fields under path
|
||||
core get <path> --key KEY_NAME # Fetch specific field
|
||||
|
||||
# Write secrets
|
||||
core put <path> key=value [key2=value2 ...] # Store secrets in Vault
|
||||
|
||||
# List and export
|
||||
core secrets list # List all secret paths
|
||||
core secrets export <path> # Export secrets as shell-sourceable format
|
||||
|
||||
# Execute with secrets in environment
|
||||
core secrets exec -- <command> # Run command with secrets loaded in env
|
||||
```
|
||||
|
||||
**Convention:** Field name = variable name (SCREAMING_SNAKE_CASE). Never use `value=`.
|
||||
|
||||
Example:
|
||||
```bash
|
||||
# Write
|
||||
core put cluster/ANTHROPIC_API_KEY ANTHROPIC_API_KEY="sk-ant-..."
|
||||
|
||||
# Read
|
||||
core get cluster/ANTHROPIC_API_KEY
|
||||
```
|
||||
|
||||
### IAM Management (Authentik)
|
||||
|
||||
```bash
|
||||
# Groups
|
||||
core iam list-groups # List all groups
|
||||
core iam create-group <name> # Create new group
|
||||
|
||||
# OAuth2 Applications
|
||||
core iam list-apps # List all OAuth2 apps
|
||||
core iam create-app <name> --slug <slug> --redirect-uri <uri> # Create app
|
||||
core iam describe-app <app> # Show client ID, secret, URIs, scope claims
|
||||
core iam rotate-secret <app> # Rotate OAuth2 client secret
|
||||
|
||||
# User management
|
||||
core iam add-member <group> <username> # Add user to group
|
||||
core iam bind-app <app> <group> # Grant group access to application
|
||||
|
||||
# Cleanup
|
||||
core iam delete-app <app> # Delete OAuth2 application
|
||||
```
|
||||
|
||||
### Object Storage (MinIO)
|
||||
|
||||
```bash
|
||||
# List and manage buckets
|
||||
core bucket list # List all buckets
|
||||
core bucket upload <bucket> <file> [<remote-path>] # Upload file to S3
|
||||
core bucket download <bucket> <remote-path> <file> # Download file from S3
|
||||
core bucket delete <bucket> <remote-path> # Delete object from S3
|
||||
```
|
||||
|
||||
### Utilities
|
||||
|
||||
```bash
|
||||
core help # Show command help
|
||||
core pf grafana # Port-forward to Grafana (localhost:3000)
|
||||
core pf prometheus # Port-forward to Prometheus (localhost:9090)
|
||||
core pf minio # Port-forward to MinIO console (localhost:9001)
|
||||
core pf iam # Port-forward to Authentik (localhost:7000)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Access Control Tiers
|
||||
|
||||
The cluster uses a tiered access model based on authentication mechanism and network perimeter.
|
||||
|
||||
### Tier A: OIDC + RBAC (Authentik-enforced)
|
||||
|
||||
Services with federated OIDC login and group-based role assignment.
|
||||
|
||||
| Service | Auth Method | Group Claim | Role Binding |
|
||||
|---------|-------------|------------|--------------|
|
||||
| **Grafana** | Authentik OIDC | `groups` | Mapped to Admin / Viewer / Viewer |
|
||||
| **Argo CD** | Authentik OIDC | `groups` | RBAC role binding (policy.csv) |
|
||||
| **MinIO** | Authentik OIDC | `groups` / custom `policy` | Policy-based access (readwrite / readonly) |
|
||||
| **Forgejo** | Authentik OIDC | `email` (OAuth login only) | No group enforcement (open git repo) |
|
||||
| **kmsvc** | Vault JWT | `sub` / `aud` | Audience validation + service scope |
|
||||
|
||||
**Provisioning:**
|
||||
```bash
|
||||
# After core auth login-oob, bootstrap IAM apps:
|
||||
bash k8s/talos-iam/bootstrap-iam.sh
|
||||
```
|
||||
|
||||
### Tier B: Network-Perimeter Only (No Authentik Enforcement)
|
||||
|
||||
Services with no OIDC support (product limitation). Access is restricted to LAN/WireGuard perimeter only.
|
||||
|
||||
| Service | Network Access | Use Case |
|
||||
|---------|----------------|----------|
|
||||
| **Portainer** | LAN + WireGuard only | Container UI, workload browsing |
|
||||
| **Longhorn** | LAN + WireGuard only | Storage volume management |
|
||||
| **Temporal** | LAN + WireGuard only | Workflow execution (auth TBD) |
|
||||
|
||||
All three are reachable **only** via WireGuard/LAN-only Ingress rules. Zero remote access risk, but also zero federated identity. If remote Temporal access is needed, upgrade the chart's auth configuration or replace with an OIDC-compatible workflow platform.
|
||||
|
||||
**Accessing Tier B services:**
|
||||
```bash
|
||||
# From off-LAN, use WireGuard context
|
||||
core config kube-use admin@homelab-cluster-1
|
||||
core pf minio # connects via 10.6.0.1:9001
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Workflow: Rotate an OAuth2 Application Secret
|
||||
|
||||
```bash
|
||||
# 1. Rotate secret in Authentik
|
||||
SECRET=$(core iam rotate-secret grafana | jq -r '.client_secret')
|
||||
|
||||
# 2. Update Helm values
|
||||
vi k8s/logging/grafana-values.yaml
|
||||
# Set: GRAFANA_OIDC_CLIENT_SECRET="$SECRET"
|
||||
|
||||
# 3. Redeploy the app
|
||||
helmfile apply -l app=grafana
|
||||
|
||||
# 4. Verify new secret is in use
|
||||
core iam describe-app grafana | grep client_secret
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Workflow: Add User to Service
|
||||
|
||||
```bash
|
||||
# 1. Verify group exists (or create it)
|
||||
core iam list-groups | grep minio-admins
|
||||
|
||||
# If not found:
|
||||
core iam create-group minio-admins
|
||||
|
||||
# 2. Add user to group
|
||||
core iam add-member minio-admins alice
|
||||
|
||||
# 3. Bind group to MinIO application
|
||||
core iam bind-app minio minio-admins
|
||||
|
||||
# 4. User has access on next login
|
||||
# (OIDC login to MinIO → Authentik → group check → MinIO policy applied)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Workflow: Rotate a Database Password (Vault)
|
||||
|
||||
```bash
|
||||
# 1. Generate new password
|
||||
NEW_PASS=$(openssl rand -hex 32)
|
||||
|
||||
# 2. Store in Vault
|
||||
core put cluster/POSTGRES_ADMIN_PASSWORD POSTGRES_ADMIN_PASSWORD="$NEW_PASS"
|
||||
|
||||
# 3. Update database user
|
||||
kubectl exec -n ddb pod/ddb-cluster-0 -- psql -U postgres -c \
|
||||
"ALTER USER postgres WITH PASSWORD '$NEW_PASS';"
|
||||
|
||||
# 4. Update Helm values with new password reference
|
||||
# (Or if using helmfile hook: helmfile will re-run postInitApplicationSQL with new password)
|
||||
|
||||
# 5. Restart pods to pick up new secret
|
||||
kubectl rollout restart -n <ns> deployment/<app>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Questions
|
||||
|
||||
**Q: I'm getting "not authenticated" on `core nodes`. What do I do?**
|
||||
A: Run `core auth login-oob`. Node commands use a different auth domain than secrets. After login, `core nodes` should work.
|
||||
|
||||
**Q: I have a valid Vault token but `core get` fails. Why?**
|
||||
A: Check that both auth domains are active:
|
||||
```bash
|
||||
core auth status # Verify node auth is valid
|
||||
core secrets status # Verify Vault auth is valid
|
||||
```
|
||||
Both must succeed. If one is expired, re-auth that domain.
|
||||
|
||||
**Q: Should I use `core bucket` or S3 CLI tools (aws-cli, s3cmd)?**
|
||||
A: Use `core bucket` for simplicity (no AWS credentials). Use s3cmd/aws-cli if you need advanced sync or bandwidth control. All three talk to the same MinIO backend.
|
||||
|
||||
**Q: Can I add a user without creating a group first?**
|
||||
A: Groups are the unit of access control. Always create the group, then add users to it, then bind it to applications. Single-user bindings are not supported (by design).
|
||||
|
||||
**Q: I rotated an OAuth2 secret but the app still fails to authenticate. What's next?**
|
||||
A:
|
||||
1. Verify the new secret is stored: `core iam describe-app grafana`
|
||||
2. Check the deployment has the new secret: `kubectl get secret -n logging grafana-oidc -o yaml | grep client_secret`
|
||||
3. Restart the pod: `kubectl rollout restart -n logging deployment/grafana`
|
||||
4. Check logs: `kubectl logs -n logging deployment/grafana | grep -i oauth`
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [~/workplace/core/USAGE.md](../../core/USAGE.md) — Exhaustive command reference with flags and examples
|
||||
- [CLAUDE.md](../CLAUDE.md) § Sign In (Device Code Flow) — Quick reference for `core auth login-oob`
|
||||
- [CLAUDE.md](../CLAUDE.md) § Quick Shortcuts — One-liners for common tasks
|
||||
- [CLAUDE.md](../CLAUDE.md) § Integration Checklist — New service onboarding
|
||||
@@ -1,188 +0,0 @@
|
||||
# CloudNativePG PostgreSQL Database
|
||||
|
||||
**Host:** `ddb-cluster-rw.ddb.svc.cluster.local` (read-write)
|
||||
**Read replica:** `ddb-cluster-ro.ddb.svc.cluster.local` (read-only)
|
||||
**Port:** `5432`
|
||||
**Namespace:** `ddb`
|
||||
|
||||
## When to Use
|
||||
|
||||
- **Multi-replica HA** — 3 replicas, automatic failover
|
||||
- **pgvector extension** — Vector similarity search (LLM embeddings)
|
||||
- **Transactional data** — Authentik, Story Crater backend, custom apps
|
||||
- **Declarative backups** — Automated WAL archiving to MinIO
|
||||
|
||||
## Quick Start
|
||||
|
||||
**1. Connect from pod:**
|
||||
```bash
|
||||
# Inside a pod (inject secret mount)
|
||||
psql -h ddb-cluster-rw.ddb.svc.cluster.local \
|
||||
-U story_crater \
|
||||
-d story_crater \
|
||||
-W # prompt for password (from Secret)
|
||||
```
|
||||
|
||||
**2. Create database & user (one-time):**
|
||||
```bash
|
||||
# Already done by helmfile postsync hook
|
||||
# But if needed manually:
|
||||
|
||||
psql -h ddb-cluster-rw.ddb.svc.cluster.local \
|
||||
-U postgres \
|
||||
-c "CREATE DATABASE myapp OWNER postgres;"
|
||||
|
||||
psql -h ddb-cluster-rw.ddb.svc.cluster.local \
|
||||
-U postgres \
|
||||
-d myapp \
|
||||
-c "CREATE USER myapp_user WITH PASSWORD 'secret';"
|
||||
|
||||
psql -h ddb-cluster-rw.ddb.svc.cluster.local \
|
||||
-U postgres \
|
||||
-d myapp \
|
||||
-c "GRANT ALL PRIVILEGES ON DATABASE myapp TO myapp_user;"
|
||||
```
|
||||
|
||||
**3. Enable pgvector:**
|
||||
```bash
|
||||
psql -h ddb-cluster-rw.ddb.svc.cluster.local \
|
||||
-U postgres \
|
||||
-d myapp \
|
||||
-c "CREATE EXTENSION IF NOT EXISTS vector;"
|
||||
```
|
||||
|
||||
**4. Create table with embeddings:**
|
||||
```sql
|
||||
CREATE TABLE documents (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
content TEXT,
|
||||
embedding vector(1536), -- OpenAI embeddings
|
||||
created_at TIMESTAMP DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX ON documents USING IVFFLAT (embedding vector_cosine_ops);
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
| Key | Value |
|
||||
|-----|-------|
|
||||
| Host (RW) | `ddb-cluster-rw.ddb.svc.cluster.local` |
|
||||
| Host (RO) | `ddb-cluster-ro.ddb.svc.cluster.local` |
|
||||
| Port | 5432 |
|
||||
| Replicas | 3 (automatic failover) |
|
||||
| Extensions | pgvector (LLM embeddings), uuid-ossp |
|
||||
| Backups | WAL archiving to MinIO (continuous) |
|
||||
| Retention | 30 days |
|
||||
|
||||
## Common Patterns
|
||||
|
||||
**Connection pooling (from app):**
|
||||
```go
|
||||
import "github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
config, _ := pgxpool.ParseConfig("postgres://user:[email protected]:5432/myapp")
|
||||
config.MaxConns = 25
|
||||
config.MinConns = 5
|
||||
pool, _ := pgxpool.NewWithConfig(ctx, config)
|
||||
|
||||
// Use pool
|
||||
row := pool.QueryRow(ctx, "SELECT COUNT(*) FROM users")
|
||||
```
|
||||
|
||||
**Read from replica (analytics):**
|
||||
```go
|
||||
// Offload SELECT queries to read replica
|
||||
pool.QueryRow(ctx, "SELECT * FROM documents LIMIT 1") // auto-routes to RO if available
|
||||
|
||||
// Writes always go to RW
|
||||
pool.Exec(ctx, "INSERT INTO documents ...")
|
||||
```
|
||||
|
||||
**Vector similarity search:**
|
||||
```sql
|
||||
SELECT id, content, embedding <-> $1 AS distance
|
||||
FROM documents
|
||||
ORDER BY embedding <-> $1
|
||||
LIMIT 10;
|
||||
-- $1 = query embedding (e.g., from OpenAI API)
|
||||
```
|
||||
|
||||
**Backup & restore:**
|
||||
```bash
|
||||
# Backups are automatic (WAL to MinIO)
|
||||
# To restore from backup:
|
||||
# 1. Check MinIO s3://postgresql-backups/
|
||||
# 2. Use PostgreSQL PITR (point-in-time recovery)
|
||||
# 3. Contact SRE for restore procedure
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
**Grafana dashboard:** `svc-postgresql` (auto-configured)
|
||||
|
||||
**Key metrics:**
|
||||
- `pg_stat_activity_connections` — active connections
|
||||
- `pg_stat_database_blks_read` — disk I/O
|
||||
- `pg_replication_lag_seconds` — replica lag (goal: < 1s)
|
||||
|
||||
**CLI health check:**
|
||||
```bash
|
||||
# Check replication status
|
||||
kubectl exec -n ddb pod/ddb-cluster-1 -- \
|
||||
psql -U postgres -c "SELECT slot_name, restart_lsn FROM pg_replication_slots;"
|
||||
|
||||
# Check replica lag
|
||||
kubectl exec -n ddb pod/ddb-cluster-2 -- \
|
||||
psql -U postgres -c "SELECT now() - pg_last_xact_replay_timestamp() AS lag;"
|
||||
```
|
||||
|
||||
## Secrets & Credentials
|
||||
|
||||
**All user passwords stored in Vault:**
|
||||
```bash
|
||||
# Read password
|
||||
core get cluster/STORY_CRATER_PG_PASSWORD --key STORY_CRATER_PG_PASSWORD
|
||||
|
||||
# Inject into pod (auto via Secret volume)
|
||||
# Mount: /run/secrets/db-password
|
||||
```
|
||||
|
||||
**Connection string from env:**
|
||||
```bash
|
||||
POSTGRES_CONNECTION="postgres://story_crater:${STORY_CRATER_PG_PASSWORD}@ddb-cluster-rw.ddb.svc.cluster.local:5432/story_crater"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Cannot connect (connection refused):**
|
||||
```bash
|
||||
# Verify cluster is running
|
||||
k get pods -n ddb
|
||||
|
||||
# Check Service DNS
|
||||
k exec -it pod/debug-pod -- nslookup ddb-cluster-rw.ddb.svc.cluster.local
|
||||
|
||||
# Verify Secret has password
|
||||
k get secret -n ddb ddb-cluster-superuser -o jsonpath='{.data.password}' | base64 -d
|
||||
```
|
||||
|
||||
**Replica lag is high (> 10s):**
|
||||
```bash
|
||||
# Check replica pod CPU/memory
|
||||
k top pod -n ddb
|
||||
|
||||
# Scale down other workloads if cluster is overloaded
|
||||
# Or scale up database resources (helmfile.yaml.gotmpl)
|
||||
```
|
||||
|
||||
**pgvector queries slow:**
|
||||
```sql
|
||||
-- Ensure index exists
|
||||
SELECT * FROM pg_indexes WHERE tablename = 'documents' AND indexname LIKE '%embedding%';
|
||||
|
||||
-- Re-index if missing
|
||||
CREATE INDEX ON documents USING IVFFLAT (embedding vector_cosine_ops);
|
||||
```
|
||||
|
||||
See `/TROUBLESHOOTING.md` for full incident guide.
|
||||
@@ -1,179 +0,0 @@
|
||||
# Forgejo OCI Registry Cleanup
|
||||
|
||||
Automatic garbage collection for the Forgejo container registry. Deletes old image tags when newer versions are pushed, keeping only the latest N versions per repository.
|
||||
|
||||
## Why
|
||||
|
||||
The Forgejo OCI registry stores all pushed images indefinitely. Without cleanup:
|
||||
- Old/retired image versions accumulate
|
||||
- Storage fills up (`longhorn` PVC)
|
||||
- Old versions clutter the UI
|
||||
|
||||
## What it does
|
||||
|
||||
**CronJob** (`forgejo-registry-cleanup`):
|
||||
- Runs daily at 2 AM UTC (configurable)
|
||||
- Lists all images in the registry
|
||||
- For each image, keeps only the **latest 3 versions** (configurable)
|
||||
- Deletes tags for older versions
|
||||
- Skips images with ≤ 3 tags (nothing to delete)
|
||||
|
||||
## How to enable
|
||||
|
||||
The manifest is in `k8s/bootstrap/phase3-forgejo/registry-cleanup-cronjob.yaml`. It's **currently disabled** (suspended) because:
|
||||
|
||||
1. **Forgejo registry auth** needs to be configured
|
||||
- `forgejo-registry-token` secret must exist in `cicd` namespace
|
||||
- Should contain `username` and `password` keys
|
||||
- User needs permission to delete images in the registry
|
||||
|
||||
2. **Registry must expose `/v2/_catalog`** endpoint
|
||||
- Standard for OCI registries
|
||||
- Forgejo includes this, but may be behind auth
|
||||
|
||||
### Step 1: Create registry token
|
||||
|
||||
If `forgejo-registry-token` doesn't exist or is empty:
|
||||
|
||||
```bash
|
||||
# As a Forgejo admin, create an API token with full scope
|
||||
# https://forgejo.riotpiao.com/user/settings/tokens
|
||||
# Copy the token
|
||||
|
||||
kubectl create secret generic forgejo-registry-token \
|
||||
-n cicd \
|
||||
--from-literal=username=<your-username> \
|
||||
--from-literal=password=<the-api-token> \
|
||||
--dry-run=client -o yaml | sops -e -i -
|
||||
```
|
||||
|
||||
Or edit via `k8s/argocd/secrets/forgejo-registry-token.enc.yaml`:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: forgejo-registry-token
|
||||
namespace: cicd
|
||||
type: Opaque
|
||||
stringData:
|
||||
username: ci-bot # or any user with admin rights
|
||||
password: <api-token>
|
||||
```
|
||||
|
||||
### Step 2: Test in dry-run mode
|
||||
|
||||
Before enabling for real, verify it works:
|
||||
|
||||
```bash
|
||||
# Edit the CronJob to set DRY_RUN=true
|
||||
kubectl set env cronjob/forgejo-registry-cleanup -n cicd DRY_RUN=true
|
||||
|
||||
# Trigger a test run
|
||||
kubectl create job --from=cronjob/forgejo-registry-cleanup \
|
||||
-n cicd forgejo-registry-cleanup-test
|
||||
|
||||
# Check logs
|
||||
kubectl logs -n cicd -l job-name=forgejo-registry-cleanup-test -f
|
||||
```
|
||||
|
||||
Dry-run output shows which images **would** be deleted without deleting them.
|
||||
|
||||
### Step 3: Enable for real
|
||||
|
||||
```bash
|
||||
# Set DRY_RUN=false and unsuspend
|
||||
kubectl patch cronjob forgejo-registry-cleanup -n cicd \
|
||||
-p '{"spec":{"suspend":false}}'
|
||||
|
||||
kubectl set env cronjob/forgejo-registry-cleanup -n cicd DRY_RUN=false
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Edit `registry-cleanup-cronjob.yaml` or patch the CronJob:
|
||||
|
||||
| Env var | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `REGISTRY_HOST` | `forgejo.riotpiao.com` | Registry hostname |
|
||||
| `KEEP_VERSIONS` | `3` | How many recent versions to keep per image |
|
||||
| `DRY_RUN` | `false` | If `true`, log what would be deleted without deleting |
|
||||
|
||||
**Schedule:** Edit `.spec.schedule` (cron format). Current: `0 2 * * *` (2 AM UTC daily).
|
||||
|
||||
Examples:
|
||||
- `0 2 * * 0` → Weekly on Sunday at 2 AM
|
||||
- `0 0 1 * *` → Monthly on the 1st at midnight
|
||||
- `0 */6 * * *` → Every 6 hours
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Check if running
|
||||
|
||||
```bash
|
||||
# See all runs
|
||||
kubectl get jobs -n cicd -l app=forgejo-registry-cleanup
|
||||
|
||||
# Latest run logs
|
||||
kubectl logs -n cicd -l app=forgejo-registry-cleanup --tail=100 -f
|
||||
```
|
||||
|
||||
### Failed runs
|
||||
|
||||
If a job fails:
|
||||
1. Check logs: `kubectl logs -n cicd <job-pod>`
|
||||
2. Common issues:
|
||||
- **403 Unauthorized**: Registry token invalid or expired
|
||||
- **404 _catalog**: Registry doesn't expose catalog endpoint
|
||||
- **Connection refused**: Registry unreachable (DNS, network policy)
|
||||
|
||||
### Metrics
|
||||
|
||||
The job doesn't currently emit Prometheus metrics, but you can:
|
||||
- Check pod exit codes in K8s events
|
||||
- Parse logs for "Total images deleted: N"
|
||||
- Set up log aggregation to alert on failures
|
||||
|
||||
## Limitations
|
||||
|
||||
1. **No version sorting**: Tags are deleted in the order returned by the registry
|
||||
- Assumption: registries return newest first (not always true)
|
||||
- **Fix**: Parse semantic versions explicitly if needed
|
||||
|
||||
2. **No protection for `latest` tag**: If `latest` is old, it will be kept but others deleted
|
||||
- Desired behavior: prioritize newest build + never delete `latest`
|
||||
- Could add logic to always keep `latest` + latest N-1 tagged versions
|
||||
|
||||
3. **No size-aware deletion**: Deletes by tag count, not storage size
|
||||
- Desired: keep until storage threshold is reached
|
||||
- Would need registry V2 API extensions (`HEAD /v2/<image>/blobs/<digest>` for size)
|
||||
|
||||
## Customizing the script
|
||||
|
||||
Edit the `cleanup.sh` script in the ConfigMap to:
|
||||
- Change sorting/selection logic
|
||||
- Integrate with external systems (Slack alerts, Prometheus metrics)
|
||||
- Add per-image exceptions (e.g., never delete `production-*` tags)
|
||||
- Use `--delete-by-digest` to reclaim actual disk space (not just catalog entries)
|
||||
|
||||
Example: Keep all tags matching `v*.*.*.` plus latest 2:
|
||||
|
||||
```bash
|
||||
# In cleanup.sh, replace the tag filtering logic:
|
||||
SEMVER_TAGS=$(echo "$TAGS" | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | sort -rV)
|
||||
KEEP_TAGS="$SEMVER_TAGS $(echo "$TAGS" | head -2 | tr '\n' ' ')"
|
||||
TAGS_TO_DELETE=$(echo "$TAGS" | grep -v -F "$KEEP_TAGS")
|
||||
```
|
||||
|
||||
## Future improvements
|
||||
|
||||
- [ ] Semantic version sorting (v1.0.0 > v0.9.9)
|
||||
- [ ] Storage size-aware retention (keep until >80% full)
|
||||
- [ ] Slack/email notifications on deletion
|
||||
- [ ] Prometheus metrics export
|
||||
- [ ] Per-image exception rules (YAML config)
|
||||
- [ ] Integration with CI/CD pipeline (delete old PR images automatically)
|
||||
|
||||
---
|
||||
|
||||
**Related:** `k8s/bootstrap/phase3-forgejo/` — Forgejo deployment manifests
|
||||
@@ -1,198 +0,0 @@
|
||||
# Infrastructure Practice Playbook
|
||||
|
||||
Standardized procedures for troubleshooting, developing, deploying, and operating the homelab platform. Each procedure explicitly calls out where the `core` CLI fits vs `kubectl`/`helmfile`/direct cluster access. See `core-cli-tools.md` for the auth/secrets domain split; see `infra-troubleshooting.md` for quick patterns and gotchas.
|
||||
|
||||
## Procedure A: Troubleshoot a Service or Cluster Issue
|
||||
|
||||
1. **Identify the domain.**
|
||||
- Vault/secrets: `core get`/`core put` failing, Vault unreachable.
|
||||
- Node/Talos: node crashes, disk full, kubelet unreachable, network issues.
|
||||
- Plain Kubernetes: pod CrashLoop, service 503, deployment stuck.
|
||||
|
||||
2. **If Vault-adjacent (secrets, authentication failing):**
|
||||
- Run `core secrets status` first (NOT `core auth status` — common mistake).
|
||||
- If `Vault UNREACHABLE`, check DNS/networking:
|
||||
- No wildcard DNS exists; verify manual `/etc/hosts` entries (10.6.0.1 for WireGuard, 192.168.1.160 for LAN).
|
||||
- Ping the Vault service: `kubectl get svc -n vault | grep vault`.
|
||||
- If `Vault token not cached`, run `core secrets login`, approve device code in browser.
|
||||
- Verify: `core secrets status` shows `✓ Authenticated`.
|
||||
|
||||
3. **If node/Talos-adjacent (kubelet logs, node state, services failing):**
|
||||
- Run `core auth status` first.
|
||||
- If token expired, run `core auth login-oob`, approve device code in browser.
|
||||
- Then run `core nodes` to list cluster nodes.
|
||||
- For a specific node, run `core status <ip>` (Talos state).
|
||||
- Inspect Talos services: `core services <ip>` (kubelet, etcd, controller, etc.).
|
||||
- Check service logs: `core logs <ip>` (main Talos logs) or `core log-svc <ip> kubelet` (specific service).
|
||||
- Consult `infra-troubleshooting.md` for Pod stuck in CrashLoopBackOff and kubelet restart patterns.
|
||||
|
||||
4. **If plain Kubernetes (pod/deployment/service issues):**
|
||||
- Consult root `TROUBLESHOOTING.md` for the layer-before-tool SRE methodology (procedures 1–10).
|
||||
- Use `infra-troubleshooting.md` § Quick Patterns for common diagnoses:
|
||||
- CrashLoopBackOff: `kubectl logs -n <ns> <pod> --tail=50` + `kubectl describe pod -n <ns> <pod> | grep -A 10 Events`.
|
||||
- Service 503: `kubectl get endpoints -n <ns> <svc>` (endpoints missing?) + `kubectl get pods -n <ns> -o wide` (pods not Ready?).
|
||||
- Helm release stuck: `helmfile status | grep -E "FAILED|UNKNOWN|PENDING"` + `helm status <release> -n <ns> --show-resources`.
|
||||
- If still unclear, escalate to `kubectl get all -n <ns>` and review resource events.
|
||||
|
||||
5. **For dashboards and live metrics:**
|
||||
- Use `core pf grafana` (port-forward to localhost:3000) rather than raw `kubectl port-forward` — keeps forwarded ports consistent.
|
||||
- If Prometheus unavailable, check: `kubectl get pods -n monitoring | grep prometheus`.
|
||||
- If ServiceMonitor not scraping, verify: `kubectl get servicemonitor -A | grep <name>` and inspect `.spec.selector` matches the target pod's app label.
|
||||
|
||||
---
|
||||
|
||||
## Procedure B: Launch/Develop a New Service or POC
|
||||
|
||||
1. **Plan the service.**
|
||||
- Determine namespace (e.g., `sqs`, `temporal`, `databases`, `monitoring`).
|
||||
- Decide if metrics exported (most should) and if OIDC-gated.
|
||||
- Sketch a Helm values.yaml structure (secrets, replicas, resource requests, affinity).
|
||||
|
||||
2. **Authenticate to Vault.**
|
||||
- Run `core secrets login` and approve device code in browser.
|
||||
- Verify: `core secrets status` shows `✓ Authenticated`.
|
||||
- You'll need Vault access to store service secrets in step 5.
|
||||
|
||||
3. **Create service Helm chart directory.**
|
||||
- Create `k8s/<service>/` with at minimum:
|
||||
- `values.yaml` (Helm values for deployment, service, replicas, resource limits).
|
||||
- `charts/` subdirectory for any custom local Helm charts (optional).
|
||||
- Follow naming conventions from `coding-standards.md`.
|
||||
|
||||
4. **Add Helm release to helmfile.**
|
||||
- Open `helmfile.yaml.gotmpl`.
|
||||
- Add release block under `releases:` section, following this structure:
|
||||
```yaml
|
||||
- name: <service>
|
||||
namespace: <namespace>
|
||||
chart: <chart-repo>/<chart-name>
|
||||
version: ~1.0 # pin major.minor, allow patch updates
|
||||
needs:
|
||||
- <dependency-namespace>/<dependency-release> # if applicable
|
||||
values:
|
||||
- k8s/<service>/values.yaml
|
||||
- secretsInline:
|
||||
DB_PASSWORD: "{{ env \"<SERVICE>_DB_PASSWORD\" }}"
|
||||
```
|
||||
- Consult `coding-standards.md` for `needs:` ordering (example: sqs section shows strimzi-operator → kafka-cluster → queue-crd → management-service).
|
||||
- Reference real example: root helmfile's `sqs` section.
|
||||
|
||||
5. **If the service needs secrets (DB password, API key, OAuth secret):**
|
||||
- Generate value (e.g., `openssl rand -hex 32` for passwords).
|
||||
- Store in Vault: `core put cluster/<SERVICE>_<KEY> <SERVICE>_<KEY>="value"`.
|
||||
- **Critical gotcha:** field name MUST equal variable name (e.g., `FORGEJO_ADMIN_PASSWORD=` not `value=`) per `coding-standards.md` § Vault field=variable convention.
|
||||
- Reference in values.yaml via `{{ env "VARIABLE_NAME" }}` (Helmfile Go template syntax, NOT shell `${VAR}`).
|
||||
- Do NOT hardcode secrets in values.yaml or ConfigMaps.
|
||||
|
||||
6. **Verify Helm syntax before deploy.**
|
||||
- Run `helmfile lint` (catches template errors, duplicate releases).
|
||||
- Run `helmfile diff -l name=<service>` (show what will be deployed).
|
||||
- Review diff for correctness (verify env var substitutions, resource limits, affinity rules).
|
||||
|
||||
7. **Deploy the service.**
|
||||
- Run `helmfile apply -l name=<service>`.
|
||||
- Monitor: `kubectl get pods -n <namespace> -w` (watch until Running).
|
||||
- If pods stuck: `kubectl describe pod -n <namespace> <pod-name>` (check Events for SchedulingFailed, ImagePullBackOff, etc.).
|
||||
|
||||
8. **If the service exports `/metrics` (Prometheus format):**
|
||||
- Create ServiceMonitor: `k8s/monitoring/servicemonitors/svc-<name>.yaml`.
|
||||
- `.spec.selector.matchLabels` must match the service's pod labels (usually `app: <service>`).
|
||||
- `.spec.endpoints[0].port` must match the service port name or number exporting metrics.
|
||||
- Create PrometheusRule: `k8s/monitoring/alerts/svc-<name>-rules.yaml`.
|
||||
- Include error rate, latency, and SLO alert rules.
|
||||
- Use `prometheus` as the rule group.
|
||||
- Create Grafana dashboard: `k8s/monitoring/dashboards/svc-<name>.yaml`.
|
||||
- Use 6-row template: Availability, Resources, Domain metrics, Logs, SLO, Related.
|
||||
- See README.md § Example Applications for a full walkthrough.
|
||||
- Verify scrape: `kubectl get servicemonitor -A | grep <name>` and check Prometheus Targets UI for green status.
|
||||
|
||||
9. **If OIDC/IAM-gated (admin UI, restricted API):**
|
||||
- Create app in Authentik: `core iam create-app "my-service" --slug my-service --redirect-uri "https://my-service.riotpiao.com/callback"`.
|
||||
- Bind app to group: `core iam bind-app my-service <group>` (e.g., `grafana-admins` for admin-only UI).
|
||||
- Retrieve credentials: `core iam describe-app my-service` (client ID, client secret).
|
||||
- Deploy secret: `kubectl create secret generic <service>-oidc --from-literal=client-id=<ID> --from-literal=client-secret=<SECRET> -n <namespace>`.
|
||||
- Reference secret in values.yaml: mount via `.spec.template.spec.containers[].env` or volumeMounts.
|
||||
- See `core-cli-tools.md` § Access Control Tiers for Tier A (OIDC + RBAC) vs Tier B (network perimeter only).
|
||||
|
||||
10. **Verify service is live.**
|
||||
- Pods: `kubectl get pods -n <namespace> -o wide` (all Running, 1/1 Ready).
|
||||
- Metrics (if applicable): `kubectl get servicemonitor -A | grep <name>` and visit Prometheus Targets or Grafana dashboard.
|
||||
- Endpoint: If publicly routed via Ingress, verify `/etc/hosts` entry (10.6.0.1 for WireGuard, 192.168.1.160 for LAN) and `curl https://my-service.riotpiao.com/health` (or equivalent health endpoint).
|
||||
- Logs: `kubectl logs -n <namespace> <pod>` (no errors).
|
||||
|
||||
### Definition of Done (Per Service)
|
||||
|
||||
- [ ] Helm chart version pinned (~1.0 format in helmfile)
|
||||
- [ ] All secrets in Vault (none in values.yaml or ConfigMap)
|
||||
- [ ] `/metrics` endpoint exported (if applicable)
|
||||
- [ ] ServiceMonitor resource created (if metrics exported)
|
||||
- [ ] PrometheusRule with error/latency/SLO alerts (if metrics exported)
|
||||
- [ ] Grafana dashboard (if metrics exported; 6-row template: Availability, Resources, Domain, Logs, SLO, Related)
|
||||
- [ ] Ingress rule (if external access needed)
|
||||
- [ ] OIDC integration via `core iam` (if UI component)
|
||||
- [ ] Verified: `helmfile diff` clean, pods Running, dashboard live or `/metrics` returning 200
|
||||
|
||||
---
|
||||
|
||||
## Procedure C: Operate the Cluster (Node Health, Context, Cleanup)
|
||||
|
||||
1. **Daily health check.**
|
||||
- Check auth: `core auth status` (if OK, node ops will work).
|
||||
- List nodes: `core nodes`.
|
||||
- For each node, check Talos state: `core status <ip>`.
|
||||
- Check K8s nodes: `kubectl get nodes -o wide` (all Ready, no NotReady).
|
||||
- Check pod pressure: `kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, memory: .status.allocatable.memory, pods: .status.allocatable.pods}'`.
|
||||
|
||||
2. **Troubleshoot a specific node.**
|
||||
- Get node IP: `core nodes` and note the IP.
|
||||
- Check Talos services: `core services <ip>` (kubelet, etcd, controller should be running).
|
||||
- Check service logs: `core logs <ip>` (main Talos daemon logs).
|
||||
- Filter to specific service: `core log-svc <ip> kubelet` (kubelet logs only).
|
||||
- Restart a service if needed: `core restart <ip> kubelet` (graceful kubelet restart).
|
||||
|
||||
3. **Pod cleanup (Failed, Evicted, Terminating pods).**
|
||||
- Run `core pods clean` (scans all namespaces, removes stale pods).
|
||||
- Verify: `kubectl get pods -A | grep -E "Failed|Evicted"` (should be empty).
|
||||
|
||||
4. **Switch kubectl context (when off-LAN, on WireGuard).**
|
||||
- List available contexts: `core config kube-list`.
|
||||
- Switch to WireGuard path (10.6.0.1:6443): `core config kube-use admin@homelab-cluster-1`.
|
||||
- **Known limitation:** `core config use <talos-context>` doesn't map to WireGuard; use `kube-use` directly.
|
||||
- Verify: `kubectl cluster-info` shows 10.6.0.1 (not 192.168.1.213).
|
||||
|
||||
5. **MinIO bucket operations (if managing data/backups).**
|
||||
- List buckets: `core bucket list`.
|
||||
- Upload file: `core bucket upload <bucket> <local-file>`.
|
||||
- Download file: `core bucket download <bucket> <remote-file> -o <local-file>`.
|
||||
- Delete file: `core bucket delete <bucket> <remote-file>`.
|
||||
|
||||
6. **Bootstrap or hardware runbooks (infrequent).**
|
||||
- **Fresh cluster setup:** See README.md § Bootstrap Order (14 steps).
|
||||
- **Adding a new Talos node:** See README.md § Adding Hardware.
|
||||
- Do not re-explain those long procedures here; consult README.md directly.
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
**Queue subsystem (Kafka/kmsvc/Temporal namespace auto-registration):** Already deployed and stable. If re-deploying:
|
||||
- Primary deploy method: `helmfile apply -l namespace=sqs` (live from root helmfile).
|
||||
- Alternate isolated iterate path: `k8s/sqs/helmfile.yaml.gotmpl` (not recommended for production).
|
||||
- Planned future: GitOps via `k8s/sqs/argocd/` (companion repo, not yet active).
|
||||
- **Critical rule:** Temporal namespace registration is automatic via `queue-operator`; never manually `temporal operator namespace create` for any namespace referenced by a Queue's `temporal.io/namespace` label. See `~/workplace/kmsvc-manage/CLAUDE.md` ("Temporal Namespace Registration") for the full rule and why.
|
||||
|
||||
**Shared/reusable service repositories:** If a service's Helm chart and container image live in a separate repository, they must be:
|
||||
- Published as a public GitHub repository under the `Riotpiaole` organization.
|
||||
- Images pushed to GHCR (`ghcr.io/riotpiaole/...`) for public pullability.
|
||||
- Consult `coding-standards.md` § Shared/Reusable Repos for the full publishing rule.
|
||||
|
||||
---
|
||||
|
||||
## Cross-References
|
||||
|
||||
- **core-cli-tools.md:** Auth/secrets domain split, command inventory, when to use `core` vs `kubectl`.
|
||||
- **coding-standards.md:** Helm naming conventions, `needs:` ordering rules, helmfile template syntax (`{{ env "VAR" }}` not `${VAR}`), Vault field=variable convention, shared-repo publishing rule.
|
||||
- **infra-troubleshooting.md:** Quick patterns (CrashLoopBackOff, 503, helm stuck), gotchas, hard rules.
|
||||
- **USAGE.md:** Exhaustive `core` command reference.
|
||||
- **README.md:** Bootstrap order, hardware addition, example app walkthrough, 6-row Grafana dashboard template.
|
||||
- **root TROUBLESHOOTING.md:** Generic Kubernetes SRE layer-before-tool methodology (10 diagnostic procedures).
|
||||
@@ -1,162 +0,0 @@
|
||||
# MinIO S3-Compatible Object Storage
|
||||
|
||||
**Endpoint:** `https://minio.riotpiao.com` (console)
|
||||
**API:** `minio.storage.svc.cluster.local:9000` (cluster-internal)
|
||||
**Namespace:** `storage`
|
||||
|
||||
## When to Use
|
||||
|
||||
- **File uploads** — Images, documents, backups
|
||||
- **Log backend** — Loki chunks storage
|
||||
- **Vault unsealing** — Store unseal keys
|
||||
- **CI/CD artifacts** — Build outputs, Docker layers cache
|
||||
|
||||
## Quick Start
|
||||
|
||||
**1. Access MinIO console:**
|
||||
```bash
|
||||
# Via browser: https://minio.riotpiao.com
|
||||
# Credentials: MINIO_ROOT_USER / MINIO_ROOT_PASSWORD (from .env)
|
||||
|
||||
# Or port-forward
|
||||
make pf-minio # localhost:9001
|
||||
```
|
||||
|
||||
**2. Create bucket:**
|
||||
```bash
|
||||
# Via AWS CLI
|
||||
export AWS_ACCESS_KEY_ID=$MINIO_ROOT_USER
|
||||
export AWS_SECRET_ACCESS_KEY=$MINIO_ROOT_PASSWORD
|
||||
|
||||
aws s3 mb s3://my-bucket \
|
||||
--endpoint-url https://minio.riotpiao.com \
|
||||
--region homelab
|
||||
|
||||
# Or via console UI: Click "Create Bucket"
|
||||
```
|
||||
|
||||
**3. Upload file:**
|
||||
```bash
|
||||
aws s3 cp /path/to/file.txt s3://my-bucket/ \
|
||||
--endpoint-url https://minio.storage.svc.cluster.local:9000 \
|
||||
--use-path-style
|
||||
```
|
||||
|
||||
**4. List buckets:**
|
||||
```bash
|
||||
aws s3 ls --endpoint-url https://minio.storage.svc.cluster.local:9000
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
| Key | Value |
|
||||
|-----|-------|
|
||||
| Access key | `MINIO_ROOT_USER` (from .env) |
|
||||
| Secret key | `MINIO_ROOT_PASSWORD` (from .env) |
|
||||
| Cluster API | `minio.storage.svc.cluster.local:9000` |
|
||||
| Console port | `9001` |
|
||||
| Replication | 3-node site-replication (az-a ↔ az-b ↔ az-c) |
|
||||
| Buckets (system) | `loki-chunks`, `loki-ruler`, `vault-backups` |
|
||||
|
||||
## Common Patterns
|
||||
|
||||
**Loki log backend (auto-configured):**
|
||||
```yaml
|
||||
# k8s/logging/loki-values.yaml
|
||||
loki:
|
||||
storage:
|
||||
s3:
|
||||
endpoint: minio.storage.svc.cluster.local:9000
|
||||
buckets: loki-chunks
|
||||
secretAccessKey: $MINIO_ROOT_PASSWORD
|
||||
accessKeyId: $MINIO_ROOT_USER
|
||||
```
|
||||
|
||||
**Application usage (Go/Python/Node):**
|
||||
```go
|
||||
import "github.com/minio/minio-go/v7"
|
||||
|
||||
client, _ := minio.New("minio.storage.svc.cluster.local:9000", &minio.Options{
|
||||
Creds: credentials.NewStaticV4(os.Getenv("MINIO_ROOT_USER"), os.Getenv("MINIO_ROOT_PASSWORD"), ""),
|
||||
Secure: false, // cluster-internal (no TLS)
|
||||
})
|
||||
|
||||
// Upload
|
||||
client.FPutObject(ctx, "my-bucket", "file.txt", "/path/to/file.txt", minio.PutObjectOptions{})
|
||||
|
||||
// Download
|
||||
client.FGetObject(ctx, "my-bucket", "file.txt", "/tmp/file.txt", minio.GetObjectOptions{})
|
||||
```
|
||||
|
||||
**Vault backup bucket:**
|
||||
```bash
|
||||
# Vault stores unseal keys in s3://vault-backups
|
||||
# Auto-managed by helmfile; no manual action needed
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
**Grafana dashboard:** `svc-minio`
|
||||
|
||||
**Key metrics:**
|
||||
- `minio_disk_drive_free_bytes` — available space
|
||||
- `minio_bucket_usage_object_count` — objects per bucket
|
||||
- `minio_bucket_usage_total_bytes` — total size per bucket
|
||||
|
||||
**Site replication status:**
|
||||
```bash
|
||||
# Port-forward to MinIO pod
|
||||
k port-forward -n storage pod/minio-0 9000:9000 &
|
||||
|
||||
# Check replication
|
||||
mc alias set local http://localhost:9000 $MINIO_ROOT_USER $MINIO_ROOT_PASSWORD
|
||||
mc admin replicate status local
|
||||
```
|
||||
|
||||
## Authentication (Cluster-Internal)
|
||||
|
||||
**From pods (cluster-internal):**
|
||||
```bash
|
||||
# Use credentials from Secret or env var
|
||||
export MINIO_ENDPOINT=minio.storage.svc.cluster.local:9000
|
||||
export MINIO_ACCESS_KEY=$MINIO_ROOT_USER
|
||||
export MINIO_SECRET_KEY=$MINIO_ROOT_PASSWORD
|
||||
aws s3 ls --endpoint-url http://$MINIO_ENDPOINT --use-path-style
|
||||
```
|
||||
|
||||
**External access (HTTPS via Ingress):**
|
||||
```bash
|
||||
# Console: https://minio.riotpiao.com (port 9001)
|
||||
# API: Use AWS CLI with --endpoint-url https://minio.riotpiao.com:9000
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Bucket creation fails:**
|
||||
```bash
|
||||
# Check MinIO pod logs
|
||||
k logs -n storage pod/minio-0 | grep -i error
|
||||
|
||||
# Verify storage space
|
||||
k get pvc -n storage
|
||||
```
|
||||
|
||||
**Site replication lag:**
|
||||
```bash
|
||||
# Check if all 3 nodes are healthy
|
||||
k get pods -n storage -l app=minio
|
||||
|
||||
# If one node is down, site replication queues changes (eventually consistent)
|
||||
```
|
||||
|
||||
**Access denied:**
|
||||
```bash
|
||||
# Verify credentials in .env
|
||||
echo $MINIO_ROOT_USER $MINIO_ROOT_PASSWORD
|
||||
|
||||
# If credentials rotated, update Secret
|
||||
k patch secret -n storage minio-root-credentials \
|
||||
--type merge -p '{"stringData":{"MINIO_ROOT_PASSWORD":"newpass"}}'
|
||||
```
|
||||
|
||||
See `/TROUBLESHOOTING.md` for full incident guide.
|
||||
@@ -1,193 +0,0 @@
|
||||
# SQS-like Message Queue Service (kmsvc)
|
||||
|
||||
**Endpoint:** `https://kmsvc.riotpiao.com` (REST + gRPC-Gateway)
|
||||
**Internal:** `kmsvc-management-service.sqs.svc.cluster.local:8080`
|
||||
**Namespace:** `sqs`
|
||||
|
||||
## When to Use
|
||||
|
||||
- **Decouple services** — Producer doesn't wait for consumer
|
||||
- **Async jobs** — Fire-and-forget processing (batch, email, webhooks)
|
||||
- **FIFO ordering** — Guarantee message order within `MessageGroupId`
|
||||
- **Durable delivery** — At-least-once (messages in Kafka, replicated 3×)
|
||||
|
||||
## Quick Start
|
||||
|
||||
**1. Create a queue:**
|
||||
```bash
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: kmsvc.io/v1
|
||||
kind: Queue
|
||||
metadata:
|
||||
name: orders
|
||||
spec:
|
||||
fifoQueue: false # standard queue
|
||||
visibilityTimeoutSeconds: 30 # re-deliver if not acked
|
||||
messageRetentionPeriodSeconds: 345600 # 4 days
|
||||
partitionsPerShard: 6
|
||||
maxReceiveCount: 5 # move to DLQ after 5 fails
|
||||
EOF
|
||||
```
|
||||
|
||||
**2. Send message:**
|
||||
```bash
|
||||
curl -X POST https://kmsvc.riotpiao.com/v1/queues/orders/messages \
|
||||
-H "Authorization: Bearer $JWT_TOKEN" \
|
||||
-d '{
|
||||
"body": "{\"order_id\":123,\"total\":99.99}",
|
||||
"attributes": {"source":"web","priority":"high"}
|
||||
}'
|
||||
```
|
||||
|
||||
**3. Receive message:**
|
||||
```bash
|
||||
curl "https://kmsvc.riotpiao.com/v1/queues/orders/messages?max_number_of_messages=10&wait_time_seconds=20" \
|
||||
-H "Authorization: Bearer $JWT_TOKEN"
|
||||
|
||||
# Response:
|
||||
# {
|
||||
# "messages": [
|
||||
# {
|
||||
# "message_id": "abc-123",
|
||||
# "receipt_handle": "...",
|
||||
# "body": "{...}",
|
||||
# "attributes": {...},
|
||||
# "receive_count": 1
|
||||
# }
|
||||
# ]
|
||||
# }
|
||||
```
|
||||
|
||||
**4. Acknowledge (delete) message:**
|
||||
```bash
|
||||
curl -X DELETE "https://kmsvc.riotpiao.com/v1/queues/orders/messages/$receipt_handle" \
|
||||
-H "Authorization: Bearer $JWT_TOKEN"
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
| Key | Value |
|
||||
|-----|-------|
|
||||
| Kafka bootstrap | `kmsvc-kafka-bootstrap.sqs.svc.cluster.local:9092` |
|
||||
| Redis | `kmsvc-redis-master.sqs.svc.cluster.local:6379` |
|
||||
| Topic naming | `kmsvc.{queueName}.shard-{id}` |
|
||||
| Replication | 3 replicas, min.insync.replicas=2 |
|
||||
| Retention | 4 days (configurable per queue) |
|
||||
|
||||
## Common Patterns
|
||||
|
||||
**Batch processing:**
|
||||
```bash
|
||||
for i in {1..100}; do
|
||||
curl -X POST https://kmsvc.riotpiao.com/v1/queues/jobs/messages \
|
||||
-H "Authorization: Bearer $JWT_TOKEN" \
|
||||
-d "{\"body\":\"task-$i\"}" &
|
||||
done
|
||||
wait
|
||||
```
|
||||
|
||||
**FIFO queue (order guaranteed per group):**
|
||||
```yaml
|
||||
apiVersion: kmsvc.io/v1
|
||||
kind: Queue
|
||||
metadata:
|
||||
name: checkout-fifo
|
||||
spec:
|
||||
fifoQueue: true
|
||||
visibilityTimeoutSeconds: 60
|
||||
partitionsPerShard: 1
|
||||
```
|
||||
|
||||
**Dead-letter queue (failed messages):**
|
||||
```yaml
|
||||
apiVersion: kmsvc.io/v1
|
||||
kind: Queue
|
||||
metadata:
|
||||
name: orders-dlq
|
||||
spec:
|
||||
fifoQueue: false
|
||||
|
||||
---
|
||||
apiVersion: kmsvc.io/v1
|
||||
kind: Queue
|
||||
metadata:
|
||||
name: orders
|
||||
spec:
|
||||
fifoQueue: false
|
||||
maxReceiveCount: 3
|
||||
deadLetterTargetQueue: orders-dlq # auto-route failures here
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
**Grafana dashboard:** `svc-kmsvc` (automatically loaded)
|
||||
|
||||
**Key metrics:**
|
||||
- `kmsvc_messages_sent_total` — total sent
|
||||
- `kmsvc_messages_received_total` — total received
|
||||
- `kmsvc_queue_depth` — pending messages per queue
|
||||
- `kmsvc_message_visibility_timeout_seconds` — visibility window
|
||||
|
||||
**Redis in-flight tracking:**
|
||||
```bash
|
||||
# Connect to Redis
|
||||
k port-forward -n sqs svc/redis 6379:6379 &
|
||||
redis-cli
|
||||
|
||||
# Check pending messages
|
||||
KEYS "kmsvc:pending:orders:*"
|
||||
KEYS "kmsvc:inflight:*" | wc -l
|
||||
```
|
||||
|
||||
## Authentication
|
||||
|
||||
**Requires JWT from Authentik:**
|
||||
```bash
|
||||
# Get token (device code flow)
|
||||
core secrets login
|
||||
|
||||
# Use token
|
||||
export JWT_TOKEN=$(core get cluster/kmsvc/jwt-token --key jwt-token)
|
||||
curl -H "Authorization: Bearer $JWT_TOKEN" https://kmsvc.riotpiao.com/v1/queues
|
||||
```
|
||||
|
||||
## Integration Example
|
||||
|
||||
**Story Crater backend consumer:**
|
||||
```go
|
||||
// Receive messages
|
||||
messages, err := kmsvc.ReceiveMessage(ctx, &kmsvc.ReceiveMessageRequest{
|
||||
QueueName: "story-crater",
|
||||
MaxNumberOfMessages: 10,
|
||||
WaitTimeSeconds: 20,
|
||||
})
|
||||
|
||||
// Process
|
||||
for _, msg := range messages.Messages {
|
||||
processMessage(msg.Body)
|
||||
|
||||
// Acknowledge on success
|
||||
kmsvc.DeleteMessage(ctx, &kmsvc.DeleteMessageRequest{
|
||||
QueueName: "story-crater",
|
||||
ReceiptHandle: msg.ReceiptHandle,
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Queue stuck / high lag:**
|
||||
```bash
|
||||
# Check Kafka broker status
|
||||
k exec -n sqs pod/kmsvc-kafka-0 -- kafka-broker-api-versions.sh --bootstrap-server localhost:9092
|
||||
|
||||
# Inspect queue topics
|
||||
k exec -n sqs pod/kmsvc-kafka-0 -- kafka-topics.sh --bootstrap-server localhost:9092 --list | grep orders
|
||||
```
|
||||
|
||||
**Messages not being consumed:**
|
||||
- Check `maxReceiveCount` (may be routing to DLQ)
|
||||
- Verify consumer has `ReceiveMessage` permission (JWT scope)
|
||||
- Check Redis: `KEYS "kmsvc:fifo_lock:orders:*"` (may be blocked by visibility timeout)
|
||||
|
||||
See `/TROUBLESHOOTING.md` for full incident guide.
|
||||
@@ -1,211 +0,0 @@
|
||||
# Vault: Secret Management & JWT Auth
|
||||
|
||||
**Vault:** `https://vault.riotpiao.com`
|
||||
**Internal:** `vault.iam.svc.cluster.local:8200`
|
||||
**Namespace:** `iam`
|
||||
|
||||
## When to Use
|
||||
|
||||
- **Store secrets** — Database passwords, API keys, TLS certs
|
||||
- **Rotate credentials** — Auto-rotate, track rotation history
|
||||
- **JWT validation** — Verify tokens from Authentik, no external call needed
|
||||
- **Audit trail** — Who accessed what secret, when
|
||||
|
||||
## Quick Start
|
||||
|
||||
**1. Login to Vault:**
|
||||
```bash
|
||||
# Browser: https://vault.riotpiao.com
|
||||
# Auth method: OIDC → "Sign in with Authentik" (federated)
|
||||
# Or: Device code → core secrets login (CLI)
|
||||
|
||||
# Via CLI (device code flow)
|
||||
core secrets login
|
||||
# → Opens browser, approve device code
|
||||
# → Token cached in ~/.talos/vault
|
||||
```
|
||||
|
||||
**2. Store a secret:**
|
||||
```bash
|
||||
# Field name = variable name (SCREAMING_SNAKE_CASE)
|
||||
core put cluster/ANTHROPIC_API_KEY ANTHROPIC_API_KEY="sk-..."
|
||||
core put cluster/STORY_CRATER_DB_PASS STORY_CRATER_DB_PASS="dbpass123"
|
||||
```
|
||||
|
||||
**3. Retrieve a secret:**
|
||||
```bash
|
||||
# Always use --key flag
|
||||
core get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
|
||||
# → sk-...
|
||||
|
||||
# Full secret as JSON
|
||||
core get cluster/ANTHROPIC_API_KEY --json
|
||||
```
|
||||
|
||||
**4. Load into shell (helmfile, scripts):**
|
||||
```bash
|
||||
vsource .env
|
||||
# → Expands empty vars from Vault (ANTHROPIC_API_KEY=)
|
||||
# → Passes hardcoded vars as-is (DEBUG=true)
|
||||
|
||||
helmfile diff
|
||||
helmfile apply
|
||||
```
|
||||
|
||||
## Vault Paths (KV v2)
|
||||
|
||||
**Naming convention:** `cluster/<VARIABLE_NAME>`
|
||||
|
||||
```
|
||||
cluster/
|
||||
├── ANTHROPIC_API_KEY # third-party API
|
||||
├── STORY_CRATER_DB_PASS # database password
|
||||
├── MINIO_ROOT_PASSWORD # MinIO credentials
|
||||
├── AUTHENTIK_BOOTSTRAP_PASSWORD # initial admin pass
|
||||
├── iam/
|
||||
│ ├── federation # Authentik OIDC app config
|
||||
│ ├── roles/admin # RBAC role definitions
|
||||
│ ├── services/grafana # OAuth2 client info
|
||||
│ ├── agents/ci-bot # machine credentials
|
||||
│ └── bindings/alice # user→role mappings
|
||||
└── kubernetes/
|
||||
├── ingress-tls # TLS cert private keys
|
||||
└── pull-secrets # Docker registry creds
|
||||
```
|
||||
|
||||
## Common Patterns
|
||||
|
||||
**Store generated secret immediately (keeps it out of shell history):**
|
||||
```bash
|
||||
# Generate & store in one command
|
||||
core put cluster/GRAFANA_OIDC_CLIENT_SECRET \
|
||||
GRAFANA_OIDC_CLIENT_SECRET="$(openssl rand -hex 32)"
|
||||
```
|
||||
|
||||
**Use in Kubernetes Secret:**
|
||||
```bash
|
||||
# Create Secret using Vault secret
|
||||
kubectl create secret generic grafana-oidc \
|
||||
--from-literal=client-secret="$(core get cluster/GRAFANA_OIDC_CLIENT_SECRET --key GRAFANA_OIDC_CLIENT_SECRET)" \
|
||||
-n logging
|
||||
```
|
||||
|
||||
**Rotate credentials safely:**
|
||||
```bash
|
||||
# 1. Generate new secret
|
||||
NEW_PASS=$(openssl rand -base64 24)
|
||||
|
||||
# 2. Store in Vault
|
||||
core put cluster/OLD_DB_PASS OLD_DB_PASS="$NEW_PASS"
|
||||
|
||||
# 3. Update database user
|
||||
psql -h ddb-cluster-rw.ddb.svc.cluster.local -U postgres \
|
||||
-c "ALTER USER story_crater PASSWORD '$NEW_PASS';"
|
||||
|
||||
# 4. Update Kubernetes Secret
|
||||
kubectl patch secret db-credentials -n story-crater-backend \
|
||||
--type merge -p "{\"stringData\":{\"password\":\"$NEW_PASS\"}}"
|
||||
|
||||
# 5. Restart pods to pick up new Secret
|
||||
k rollout restart -n story-crater-backend deployment/app
|
||||
```
|
||||
|
||||
**JWT token from CLI:**
|
||||
```bash
|
||||
# After device code login
|
||||
core secrets login
|
||||
|
||||
# Token is cached and auto-renewed
|
||||
# Use for API calls
|
||||
VAULT_TOKEN=$(cat ~/.talos/vault)
|
||||
curl -H "X-Vault-Token: $VAULT_TOKEN" \
|
||||
https://vault.iam.svc.cluster.local:8200/v1/secret/data/cluster/ANTHROPIC_API_KEY
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
**Vault status:**
|
||||
```bash
|
||||
# Check if sealed
|
||||
core status
|
||||
|
||||
# If sealed (disaster recovery):
|
||||
# See /TROUBLESHOOTING.md § Vault Sealed
|
||||
```
|
||||
|
||||
**Audit log (who accessed what):**
|
||||
```bash
|
||||
# Enable audit logging (already enabled by helmfile)
|
||||
# Logs stored in Loki under vault namespace
|
||||
|
||||
# View recent access
|
||||
core audit log --limit 50
|
||||
|
||||
# Export for compliance
|
||||
core audit export --format json > vault-audit.json
|
||||
```
|
||||
|
||||
## Security Rules
|
||||
|
||||
✅ **DO:**
|
||||
- Store ALL secrets in Vault (never .env in git)
|
||||
- Use field name = variable name
|
||||
- Always use `--key` flag when retrieving
|
||||
- Rotate credentials on schedule (quarterly)
|
||||
- Review audit log for anomalies
|
||||
|
||||
❌ **DON'T:**
|
||||
- Commit `.env` with real secrets (only `.env.example`)
|
||||
- Use positional arguments (must use `--key`)
|
||||
- Share Vault root token
|
||||
- Store secrets in pod env (use Secret volumes)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Cannot login (Authentik OIDC fails):**
|
||||
```bash
|
||||
# Check Authentik is running
|
||||
k get pods -n iam -l app=authentik
|
||||
|
||||
# Verify OIDC app in Authentik console
|
||||
# Applications → Vault OIDC → Check client ID/secret
|
||||
|
||||
# Restart Vault to re-sync OIDC config
|
||||
k rollout restart -n iam statefulset/vault
|
||||
```
|
||||
|
||||
**Vault is sealed:**
|
||||
```bash
|
||||
# Check status
|
||||
core status
|
||||
|
||||
# If sealed, use unseal keys (stored in MinIO backup)
|
||||
# See /TROUBLESHOOTING.md § Vault Sealed for recovery steps
|
||||
```
|
||||
|
||||
**Secret not found:**
|
||||
```bash
|
||||
# Verify path exists
|
||||
core list cluster
|
||||
|
||||
# Check secret name (case-sensitive)
|
||||
core get cluster/anthropic_api_key --key anthropic_api_key # won't work
|
||||
core get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY # correct
|
||||
```
|
||||
|
||||
**vsource not expanding secrets:**
|
||||
```bash
|
||||
# Verify .env has empty value
|
||||
grep ANTHROPIC_API_KEY .env
|
||||
# → Should be: ANTHROPIC_API_KEY= (empty, not a value)
|
||||
|
||||
# Verify Vault is accessible
|
||||
core get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
|
||||
# → Should return secret
|
||||
|
||||
# Run vsource explicitly
|
||||
vsource .env
|
||||
echo $ANTHROPIC_API_KEY # should be populated
|
||||
```
|
||||
|
||||
See `/TROUBLESHOOTING.md` for full incident guide.
|
||||
@@ -1,72 +0,0 @@
|
||||
Implement **Stage 1 only** of the approved plan at
|
||||
`/Users/rockliang/.claude/plans/fluttering-cooking-thunder.md`. Read that file first — it is
|
||||
the spec. Do not implement Stage 2, 3 or 4.
|
||||
|
||||
## Already done, do not redo
|
||||
|
||||
Stage 0 passed. The Forgejo registry returns distinct, correctly-ordered image
|
||||
creation timestamps, so `newest-build` is viable:
|
||||
|
||||
```
|
||||
rock/api-gateway v0.0.0 2026-08-20T05:19:48.655Z
|
||||
rock/api-gateway v0.1.0 2026-08-20T06:57:11.943Z
|
||||
rock/api-gateway v0.1.1 2026-08-20T07:10:13.093Z
|
||||
```
|
||||
|
||||
Note the images are multi-arch OCI indexes: reading `created` means descending
|
||||
index -> amd64 manifest -> config blob.
|
||||
|
||||
## Scope: Stage 1 = A1, A2, A4, A6, B, C1
|
||||
|
||||
- **A1** — in `~/workplace/homelab`, replace the per-repo Forgejo entry in
|
||||
`k8s/argocd/projects/homelab-project.yaml` `sourceRepos` with a wildcard
|
||||
`https://forgejo.riotpiao.com/rock/*`.
|
||||
- **A2** — add an Argo `Application` at sync-wave `-1` that syncs
|
||||
`k8s/argocd/projects/`. Nothing owns that directory today, which is why the
|
||||
AppProject only ever reaches the cluster by hand.
|
||||
- **A4** — Forgejo webhook to `https://argocd.riotpiao.com/api/webhook` with a
|
||||
shared secret stored in `argocd-secret` (SOPS/ksops). Register it on
|
||||
`rock/homelab` and `rock/homelab-frontend`.
|
||||
- **A6** — add the `forgejo-registry` dockerconfigjson pull secret for any
|
||||
namespace that needs it, as a new `*.enc.yaml` listed in
|
||||
`k8s/argocd/secrets/secret-generator.yaml`. It currently exists only in `api`.
|
||||
- **B + C1** — in `~/workplace/homelab-frontend`: delete the dead
|
||||
`.github/workflows/ci.yml`, add `.forgejo/workflows/ci.yaml` and
|
||||
`.forgejo/workflows/build.yaml`, and add a multi-stage distroless `Dockerfile`
|
||||
(none exists today).
|
||||
|
||||
Stage 1 stops before Argo CD Image Updater. Do **not** install it and do not add
|
||||
image-updater annotations — that is Stage 2.
|
||||
|
||||
## Hard constraints
|
||||
|
||||
- **`runs-on: docker`.** That is the runner's only registered label. The existing
|
||||
`.github/workflows/ci.yml` uses `ubuntu-latest`, which is exactly why it has
|
||||
never executed once.
|
||||
- **No git tags, ever.** Image tag is the commit short SHA: `$(git rev-parse --short HEAD)`.
|
||||
Do not use `git describe`, do not create or push tags, do not use `latest`.
|
||||
- **Build workflow only builds and pushes.** No manifest write-back, no git push
|
||||
from CI, no `[skip ci]` guard needed.
|
||||
- **GitOps only.** No `kubectl apply`, no `helm upgrade`, no local `terraform apply`.
|
||||
`kubectl --dry-run=server` and read-only `kubectl get`/`logs` are fine.
|
||||
- **Never `git reset --hard`.**
|
||||
- Push directly to `main`, no PRs, no branches. The cluster repo has three
|
||||
remotes — `origin` is Forgejo, `github` is GitHub. Push to both; they are
|
||||
currently in sync at `43483da`.
|
||||
- Match surrounding file style. This repo comments the *why* on non-obvious
|
||||
config, and `kustomization.yaml` uses explicit `resources:` allowlists — a file
|
||||
you add and forget to list is silently dropped.
|
||||
|
||||
## Verify before claiming done
|
||||
|
||||
- `kubectl kustomize` each directory you touch.
|
||||
- `kubectl apply --dry-run=server -f` every manifest you add or change.
|
||||
- Confirm the AppProject wildcard is live:
|
||||
`kubectl -n argocd get appproject homelab -o jsonpath='{.spec.sourceRepos}'`
|
||||
- Push an empty commit to `homelab-frontend` and confirm the build job actually
|
||||
runs on the Forgejo runner and pushes `forgejo.riotpiao.com/rock/api-gateway:<short-sha>`.
|
||||
A workflow that does not trigger is the single most likely failure here.
|
||||
- Report what you verified with real command output, not assertions. If a step
|
||||
fails, say so and stop rather than working around it.
|
||||
|
||||
Kubeconfig: `/Users/rockliang/workplace/homelab/cluster-config/kubeconfig`.
|
||||
Reference in New Issue
Block a user