feat(gitops): migrate domain to riotpiao.com, add CNPG + Forgejo HA on Redis/Postgres, wire ArgoCD apps — enables cluster rebuild after etcd wipe and unblocks the git-source chicken-egg via standalone Helm-source Applications

This commit is contained in:
Story Crater Bot
2026-07-19 09:29:17 -07:00
parent c6493f14ae
commit 578a707867
86 changed files with 889 additions and 446 deletions
+15 -15
View File
@@ -1,7 +1,7 @@
# Authentik Federated OIDC & SSO
**Provider:** `https://authentik.riotpiao.homelab.com`
**OIDC Issuer:** `https://authentik.riotpiao.homelab.com/application/o/talos-federation/`
**Provider:** `https://authentik.riotpiao.com`
**OIDC Issuer:** `https://authentik.riotpiao.com/application/o/talos-federation/`
**Namespace:** `iam`
## When to Use
@@ -15,7 +15,7 @@
**1. Login to Authentik console:**
```bash
# Browser: https://authentik.riotpiao.homelab.com
# Browser: https://authentik.riotpiao.com
# Default user: akadmin
# Password: AUTHENTIK_BOOTSTRAP_PASSWORD (from .env)
@@ -33,7 +33,7 @@ Authentik console → Users → Create
**3. User logs into Grafana:**
```
https://grafana.riotpiao.homelab.com
https://grafana.riotpiao.com
→ Sign in with Authentik (auto-redirects to OIDC provider)
→ Approve access
→ Logged in as alice (group determines role: Admin or Viewer)
@@ -44,8 +44,8 @@ https://grafana.riotpiao.homelab.com
| Key | Value |
|-----|-------|
| OIDC provider | `talos-federation` (federated) |
| OIDC issuer | `https://authentik.riotpiao.homelab.com/application/o/talos-federation/` |
| JWKS endpoint | `https://authentik.riotpiao.homelab.com/application/o/talos-federation/.well-known/openid-configuration` |
| OIDC issuer | `https://authentik.riotpiao.com/application/o/talos-federation/` |
| JWKS endpoint | `https://authentik.riotpiao.com/application/o/talos-federation/.well-known/openid-configuration` |
| Database | PostgreSQL (ddb namespace, authentik user) |
| Backups | WAL archived to MinIO |
@@ -60,9 +60,9 @@ grafana:
name: Authentik
client_id: grafana
client_secret: $GRAFANA_OIDC_CLIENT_SECRET # from Vault
auth_url: https://authentik.riotpiao.homelab.com/application/o/authorize/
token_url: https://authentik.riotpiao.homelab.com/application/o/token/
api_url: https://authentik.riotpiao.homelab.com/application/o/userinfo/
auth_url: https://authentik.riotpiao.com/application/o/authorize/
token_url: https://authentik.riotpiao.com/application/o/token/
api_url: https://authentik.riotpiao.com/application/o/userinfo/
scopes: openid profile email groups
use_pkce: true
```
@@ -75,8 +75,8 @@ minio:
provider: authentik
client_id: minio
client_secret: $MINIO_OIDC_CLIENT_SECRET
redirect_uri: https://minio.riotpiao.homelab.com/oauth_callback
config_url: https://authentik.riotpiao.homelab.com/application/o/talos-federation/.well-known/openid-configuration
redirect_uri: https://minio.riotpiao.com/oauth_callback
config_url: https://authentik.riotpiao.com/application/o/talos-federation/.well-known/openid-configuration
policy_mappings:
- group: homelab-admins → consoleAdmin
- group: homelab-devops → readwrite
@@ -99,7 +99,7 @@ core get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
```go
import "github.com/coreos/go-oidc/v3/oidc"
provider, _ := oidc.NewProvider(ctx, "https://authentik.riotpiao.homelab.com/application/o/talos-federation/")
provider, _ := oidc.NewProvider(ctx, "https://authentik.riotpiao.com/application/o/talos-federation/")
verifier := provider.Verifier(&oidc.Config{ClientID: "my-app"})
@@ -137,7 +137,7 @@ Authentik console → Users → alice → Edit
## Monitoring
**Authentik dashboard:** https://authentik.riotpiao.homelab.com/api/v3/admin/dashboards
**Authentik dashboard:** https://authentik.riotpiao.com/api/v3/admin/dashboards
**Key metrics:**
- Login attempts (success/failure)
@@ -151,7 +151,7 @@ Authentik console → Users → alice → Edit
```bash
# Check redirect URI matches
# Authentik console → Applications → grafana → Edit
# Verify Redirect URI = https://grafana.riotpiao.homelab.com/login/generic_oauth
# Verify Redirect URI = https://grafana.riotpiao.com/login/generic_oauth
# Check OIDC provider is running
k get pods -n iam -l app=authentik
@@ -174,7 +174,7 @@ core secrets login
**Vault can't validate JWT:**
```bash
# Verify JWKS endpoint is accessible
curl https://authentik.riotpiao.homelab.com/application/o/talos-federation/.well-known/openid-configuration
curl https://authentik.riotpiao.com/application/o/talos-federation/.well-known/openid-configuration
# Restart Vault to refresh JWKS cache
k rollout restart -n iam deployment/vault
+13 -13
View File
@@ -1,7 +1,7 @@
# CI/CD Pipeline (Forgejo + Argo CD)
**Git Forge:** `https://forgejo.riotpiao.homelab.com`
**Deployments:** `https://argocd.riotpiao.homelab.com` (or `kubectl port-forward`)
**Git Forge:** `https://forgejo.riotpiao.com`
**Deployments:** `https://argocd.riotpiao.com` (or `kubectl port-forward`)
**Namespaces:** `cicd`, `forge`
## When to Use
@@ -15,7 +15,7 @@
**1. Clone a repo from Forgejo:**
```bash
git clone https://forgejo.riotpiao.homelab.com/rock/source.git
git clone https://forgejo.riotpiao.com/rock/source.git
cd source
```
@@ -33,10 +33,10 @@ jobs:
- run: npm test
- run: docker build -t myapp:latest .
- run: |
docker login forgejo.riotpiao.homelab.com \
docker login forgejo.riotpiao.com \
-u ci-bot \
-p ${{ secrets.CI_BOT_TOKEN }}
docker push forgejo.riotpiao.homelab.com/rock/myapp:latest
docker push forgejo.riotpiao.com/rock/myapp:latest
EOF
git add .forgejo/workflows/ci.yml
@@ -47,11 +47,11 @@ git push
**3. Trigger deployment:**
```bash
# Update deployment repo (rock/deploy)
git clone https://forgejo.riotpiao.homelab.com/rock/deploy.git
git clone https://forgejo.riotpiao.com/rock/deploy.git
cd deploy
# Update image tag
sed -i 's|forgejo.riotpiao.homelab.com/rock/myapp:.*|forgejo.riotpiao.homelab.com/rock/myapp:abc123|' k8s/deployment.yaml
sed -i 's|forgejo.riotpiao.com/rock/myapp:.*|forgejo.riotpiao.com/rock/myapp:abc123|' k8s/deployment.yaml
git add k8s/deployment.yaml
git commit -m "deploy: bump myapp to abc123"
@@ -92,7 +92,7 @@ jobs:
steps:
- uses: actions/checkout@v3
- run: docker build -t myapp:${{ github.sha }} .
- run: docker push forgejo.riotpiao.homelab.com/rock/myapp:${{ github.sha }}
- run: docker push forgejo.riotpiao.com/rock/myapp:${{ github.sha }}
```
**Available variables:**
@@ -109,10 +109,10 @@ ${{ secrets.CI_BOT_TOKEN }} # injected from Forgejo
```yaml
- run: |
echo "${{ secrets.CI_BOT_TOKEN }}" | docker login \
forgejo.riotpiao.homelab.com \
forgejo.riotpiao.com \
-u ci-bot \
--password-stdin
docker push forgejo.riotpiao.homelab.com/rock/myapp:latest
docker push forgejo.riotpiao.com/rock/myapp:latest
```
**API token (for pushing commits):**
@@ -121,7 +121,7 @@ ${{ secrets.CI_BOT_TOKEN }} # injected from Forgejo
git config user.name "ci-bot"
git config user.email "ci-bot@homelab"
git commit --allow-empty -m "bump: version"
git push https://ci-bot:${{ secrets.CI_BOT_TOKEN }}@forgejo.riotpiao.homelab.com/rock/deploy.git main
git push https://ci-bot:${{ secrets.CI_BOT_TOKEN }}@forgejo.riotpiao.com/rock/deploy.git main
```
**Vault secrets (via talos CLI):**
@@ -135,7 +135,7 @@ ${{ secrets.CI_BOT_TOKEN }} # injected from Forgejo
**Create app (one-time):**
```bash
argocd app create story-crater \
--repo https://forgejo.riotpiao.homelab.com/rock/deploy.git \
--repo https://forgejo.riotpiao.com/rock/deploy.git \
--path k8s/ \
--dest-server https://kubernetes.default.svc \
--dest-namespace story-crater-backend \
@@ -148,7 +148,7 @@ argocd app create story-crater \
argocd app get story-crater
argocd app logs story-crater
# UI: https://argocd.riotpiao.homelab.com
# UI: https://argocd.riotpiao.com
# Login: Authentik SSO (homelab-admins group only)
```
+2 -2
View File
@@ -107,7 +107,7 @@ Standardized procedures for troubleshooting, developing, deploying, and operatin
- Verify scrape: `kubectl get servicemonitor -A | grep <name>` and check Prometheus Targets UI for green status.
9. **If OIDC/IAM-gated (admin UI, restricted API):**
- Create app in Authentik: `core iam create-app "my-service" --slug my-service --redirect-uri "https://my-service.riotpiao.homelab.com/callback"`.
- Create app in Authentik: `core iam create-app "my-service" --slug my-service --redirect-uri "https://my-service.riotpiao.com/callback"`.
- Bind app to group: `core iam bind-app my-service <group>` (e.g., `grafana-admins` for admin-only UI).
- Retrieve credentials: `core iam describe-app my-service` (client ID, client secret).
- Deploy secret: `kubectl create secret generic <service>-oidc --from-literal=client-id=<ID> --from-literal=client-secret=<SECRET> -n <namespace>`.
@@ -117,7 +117,7 @@ Standardized procedures for troubleshooting, developing, deploying, and operatin
10. **Verify service is live.**
- Pods: `kubectl get pods -n <namespace> -o wide` (all Running, 1/1 Ready).
- Metrics (if applicable): `kubectl get servicemonitor -A | grep <name>` and visit Prometheus Targets or Grafana dashboard.
- Endpoint: If publicly routed via Ingress, verify `/etc/hosts` entry (10.6.0.1 for WireGuard, 192.168.1.160 for LAN) and `curl https://my-service.riotpiao.homelab.com/health` (or equivalent health endpoint).
- Endpoint: If publicly routed via Ingress, verify `/etc/hosts` entry (10.6.0.1 for WireGuard, 192.168.1.160 for LAN) and `curl https://my-service.riotpiao.com/health` (or equivalent health endpoint).
- Logs: `kubectl logs -n <namespace> <pod>` (no errors).
### Definition of Done (Per Service)
+1 -1
View File
@@ -58,7 +58,7 @@ k exec -n ddb pod/ddb-cluster-0 -- psql -U postgres -c "SELECT * FROM pg_stat_re
- **MinIO credentials:** Use Secret volumes, never `--env` flag (exposes in `kubectl describe`)
- **Temporal namespace registration:** never manually `temporal operator namespace create` for a namespace a Queue's `temporal.io/namespace` label references — `queue-operator` registers it automatically. See `~/workplace/kmsvc-manage/CLAUDE.md` ("Temporal Namespace Registration") for the full rule and why.
- **kubectl hangs / connection refused:** you're probably off-LAN. `kubectl config use-context admin@homelab-cluster-1` (WireGuard path, `10.6.0.1:6443`) — the default context (`admin@homelab-cluster`, `192.168.1.213:6443`) only works on-LAN. See README.md "kubectl Context".
- **No wildcard DNS:** `*.riotpiao.homelab.com` isn't a real DNS zone — every hostname is a manual `/etc/hosts` line (`10.6.0.1` for WireGuard, the Cilium LB-IPAM IP for LAN). Adding an Ingress `host:` rule doesn't make it resolvable; you must also add the `/etc/hosts` line, on every client machine that needs it.
- **No wildcard DNS:** `*.riotpiao.com` isn't a real DNS zone — every hostname is a manual `/etc/hosts` line (`10.6.0.1` for WireGuard, the Cilium LB-IPAM IP for LAN). Adding an Ingress `host:` rule doesn't make it resolvable; you must also add the `/etc/hosts` line, on every client machine that needs it.
- **gRPC through nginx ingress:** add `nginx.ingress.kubernetes.io/backend-protocol: "GRPC"` to proxy h2c to a plaintext-gRPC backend (see `temporal-grpc` Ingress in `k8s/ingress/ingress.yaml`). TLS still terminates at nginx via the wildcard cert — clients need `--tls`, not client certs.
- **Soft podAntiAffinity doesn't rebalance existing pods.** `preferredDuringSchedulingIgnoredDuringExecution` only applies at scheduling time — pods that landed on the same node before the constraint existed (or before other nodes were `Ready`) stay there forever. Fixing the affinity config in values.yaml/CRD only affects *future* pod creation; existing skew needs `kubectl delete pod <name>` (one at a time, verify healthy before the next) to force a reschedule under the now-correct constraint. Hit this with Cassandra, the Kafka `KafkaNodePool`, and CNPG's `ddb-cluster` all stacking onto `talos-cp-1`.
- **Cassandra/StatefulSet pod deleted+recreated with no PVC (ephemeral storage) can crash-loop on rejoin:** `Other bootstrapping/leaving/moving nodes detected` or `A node required to move the data consistently is down (/<old-ip>)` — the ring still holds a stale gossip entry for the deleted pod's old IP. Fix: `kubectl exec <a live cassandra pod> -- nodetool assassinate <stale-ip>` from a healthy node, then let the crash-looping pod's next restart retry.
+5 -5
View File
@@ -1,6 +1,6 @@
# MinIO S3-Compatible Object Storage
**Endpoint:** `https://minio.riotpiao.homelab.com` (console)
**Endpoint:** `https://minio.riotpiao.com` (console)
**API:** `minio.storage.svc.cluster.local:9000` (cluster-internal)
**Namespace:** `storage`
@@ -15,7 +15,7 @@
**1. Access MinIO console:**
```bash
# Via browser: https://minio.riotpiao.homelab.com
# Via browser: https://minio.riotpiao.com
# Credentials: MINIO_ROOT_USER / MINIO_ROOT_PASSWORD (from .env)
# Or port-forward
@@ -29,7 +29,7 @@ export AWS_ACCESS_KEY_ID=$MINIO_ROOT_USER
export AWS_SECRET_ACCESS_KEY=$MINIO_ROOT_PASSWORD
aws s3 mb s3://my-bucket \
--endpoint-url https://minio.riotpiao.homelab.com \
--endpoint-url https://minio.riotpiao.com \
--region homelab
# Or via console UI: Click "Create Bucket"
@@ -126,8 +126,8 @@ aws s3 ls --endpoint-url http://$MINIO_ENDPOINT --use-path-style
**External access (HTTPS via Ingress):**
```bash
# Console: https://minio.riotpiao.homelab.com (port 9001)
# API: Use AWS CLI with --endpoint-url https://minio.riotpiao.homelab.com:9000
# Console: https://minio.riotpiao.com (port 9001)
# API: Use AWS CLI with --endpoint-url https://minio.riotpiao.com:9000
```
## Troubleshooting
+2 -2
View File
@@ -1,7 +1,7 @@
# Monitoring: Prometheus, Grafana & Loki
**Prometheus:** `prometheus-kube-prom-prometheus.monitoring.svc.cluster.local:9090`
**Grafana:** `https://grafana.riotpiao.homelab.com`
**Grafana:** `https://grafana.riotpiao.com`
**Loki:** `loki.logging.svc.cluster.local:3100`
**Namespaces:** `monitoring`, `logging`
@@ -16,7 +16,7 @@
**1. Access Grafana:**
```bash
# Browser: https://grafana.riotpiao.homelab.com
# Browser: https://grafana.riotpiao.com
# Login: admin / GRAFANA_ADMIN_PASSWORD (from .env)
# Or via Authentik SSO
+13 -13
View File
@@ -27,10 +27,10 @@ spec:
ingressClassName: nginx
tls:
- hosts:
- myapp.riotpiao.homelab.com
- myapp.riotpiao.com
secretName: myapp-tls
rules:
- host: myapp.riotpiao.homelab.com
- host: myapp.riotpiao.com
http:
paths:
- path: /
@@ -54,10 +54,10 @@ kubectl get certificate -n myapp-ns
**3. Test from client:**
```bash
# Add to /etc/hosts (or use WireGuard)
192.168.1.160 myapp.riotpiao.homelab.com
192.168.1.160 myapp.riotpiao.com
# Access
curl https://myapp.riotpiao.homelab.com
curl https://myapp.riotpiao.com
```
## Configuration
@@ -83,10 +83,10 @@ spec:
ingressClassName: nginx
tls:
- hosts:
- api.riotpiao.homelab.com
- api.riotpiao.com
secretName: api-tls
rules:
- host: api.riotpiao.homelab.com
- host: api.riotpiao.com
http:
paths:
- path: /users
@@ -130,7 +130,7 @@ metadata:
spec:
ingressClassName: nginx
rules:
- host: protected.riotpiao.homelab.com
- host: protected.riotpiao.com
http:
paths:
- path: /
@@ -146,8 +146,8 @@ spec:
```yaml
# k8s/coredns/coredns-configmap.yaml
# Rewrite:
# - grafana.riotpiao.homelab.com → grafana.logging (cluster-internal)
# - prometheus.riotpiao.homelab.com → prometheus-kube-prom-prometheus.monitoring
# - grafana.riotpiao.com → grafana.logging (cluster-internal)
# - prometheus.riotpiao.com → prometheus-kube-prom-prometheus.monitoring
#
# Allows pods to use external URLs but resolve to internal Services
```
@@ -184,9 +184,9 @@ spec:
secretName: myapp-tls
duration: 2160h # 90 days
renewBefore: 360h # renew 15 days before expiry
commonName: myapp.riotpiao.homelab.com
commonName: myapp.riotpiao.com
dnsNames:
- myapp.riotpiao.homelab.com
- myapp.riotpiao.com
issuerRef:
name: homelab-ca
kind: ClusterIssuer
@@ -229,7 +229,7 @@ nslookup grafana # if in logging namespace
make pf-grafana # localhost:3000 → grafana.logging:3000
# Option 3: Add to /etc/hosts (on home network)
192.168.1.160 grafana.riotpiao.homelab.com
192.168.1.160 grafana.riotpiao.com
```
## Monitoring
@@ -308,7 +308,7 @@ k get secret -n myapp-ns myapp-tls -o jsonpath='{.data.tls\.crt}' | base64 -d |
# If cert is self-signed (homelab-ca), add to client's trusted roots
# Or bypass cert verification (dev only):
curl -k https://myapp.riotpiao.homelab.com
curl -k https://myapp.riotpiao.com
```
See `/TROUBLESHOOTING.md` for full incident guide.
+6 -6
View File
@@ -1,6 +1,6 @@
# SQS-like Message Queue Service (kmsvc)
**Endpoint:** `https://kmsvc.riotpiao.homelab.com` (REST + gRPC-Gateway)
**Endpoint:** `https://kmsvc.riotpiao.com` (REST + gRPC-Gateway)
**Internal:** `kmsvc-management-service.sqs.svc.cluster.local:8080`
**Namespace:** `sqs`
@@ -31,7 +31,7 @@ EOF
**2. Send message:**
```bash
curl -X POST https://kmsvc.riotpiao.homelab.com/v1/queues/orders/messages \
curl -X POST https://kmsvc.riotpiao.com/v1/queues/orders/messages \
-H "Authorization: Bearer $JWT_TOKEN" \
-d '{
"body": "{\"order_id\":123,\"total\":99.99}",
@@ -41,7 +41,7 @@ curl -X POST https://kmsvc.riotpiao.homelab.com/v1/queues/orders/messages \
**3. Receive message:**
```bash
curl "https://kmsvc.riotpiao.homelab.com/v1/queues/orders/messages?max_number_of_messages=10&wait_time_seconds=20" \
curl "https://kmsvc.riotpiao.com/v1/queues/orders/messages?max_number_of_messages=10&wait_time_seconds=20" \
-H "Authorization: Bearer $JWT_TOKEN"
# Response:
@@ -60,7 +60,7 @@ curl "https://kmsvc.riotpiao.homelab.com/v1/queues/orders/messages?max_number_of
**4. Acknowledge (delete) message:**
```bash
curl -X DELETE "https://kmsvc.riotpiao.homelab.com/v1/queues/orders/messages/$receipt_handle" \
curl -X DELETE "https://kmsvc.riotpiao.com/v1/queues/orders/messages/$receipt_handle" \
-H "Authorization: Bearer $JWT_TOKEN"
```
@@ -79,7 +79,7 @@ curl -X DELETE "https://kmsvc.riotpiao.homelab.com/v1/queues/orders/messages/$re
**Batch processing:**
```bash
for i in {1..100}; do
curl -X POST https://kmsvc.riotpiao.homelab.com/v1/queues/jobs/messages \
curl -X POST https://kmsvc.riotpiao.com/v1/queues/jobs/messages \
-H "Authorization: Bearer $JWT_TOKEN" \
-d "{\"body\":\"task-$i\"}" &
done
@@ -148,7 +148,7 @@ core secrets login
# Use token
export JWT_TOKEN=$(core get cluster/kmsvc/jwt-token --key jwt-token)
curl -H "Authorization: Bearer $JWT_TOKEN" https://kmsvc.riotpiao.homelab.com/v1/queues
curl -H "Authorization: Bearer $JWT_TOKEN" https://kmsvc.riotpiao.com/v1/queues
```
## Integration Example
+2 -2
View File
@@ -1,6 +1,6 @@
# Vault: Secret Management & JWT Auth
**Vault:** `https://vault.riotpiao.homelab.com`
**Vault:** `https://vault.riotpiao.com`
**Internal:** `vault.iam.svc.cluster.local:8200`
**Namespace:** `iam`
@@ -15,7 +15,7 @@
**1. Login to Vault:**
```bash
# Browser: https://vault.riotpiao.homelab.com
# Browser: https://vault.riotpiao.com
# Auth method: OIDC → "Sign in with Authentik" (federated)
# Or: Device code → core secrets login (CLI)