rock/homelab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

k8s/services: add ingress networking portainer llm and project guides

Browse Source 
- Nginx ingress + TLS termination (homelab-ca)
- Portainer container UI
- CoreDNS internal DNS rewrites
- DuckDNS DDNS updater
- Ollama LLM inference
- 8 project-usage guides (team reference)
This commit is contained in:
Story Crater Bot
2026-07-11 19:17:54 -07:00
parent 1c02e2b831
commit 6d5a0ba205
26 changed files with 3280 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
k8s/coredns/coredns-configmap.yaml
+67
View File
@@ -0,0 +1,67 @@
# k8s/coredns/coredns-configmap.yaml
# Patches the CoreDNS Corefile to rewrite homelab hostnames to internal services.
#
# Why this is needed:
# Grafana v10+ does OIDC auto-discovery by fetching
# /.well-known/openid-configuration from Authentik. When Grafana reaches
# Authentik via the external hostname (authentik.riotpiao.homelab.com), the
# HTTP Host header is preserved and Authentik returns external URLs in the
# discovery response. Without this rewrite, the hostname doesn't resolve
# inside the cluster and Grafana falls back to the internal service DNS,
# causing all OAuth redirects to go to authentik-server.iam.svc.cluster.local.
#
# Applied by helmfile presync hook on the ingress-nginx release.
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
log . {
class error
}
prometheus :9153
# Forgejo: route to HTTP service (LoadBalancer handles both HTTPS:443 + SSH:2222 via same IP)
rewrite name forgejo.riotpiao.homelab.com forgejo-gitea-http.cicd.svc.cluster.local
# Rewrite homelab hostnames to the nginx ingress controller so in-cluster pods
# hit nginx TLS termination (cert-manager cert) and preserve the Host header.
# Routing through nginx — not directly to the backend service — is critical:
# direct rewrites to the backend bypass nginx TLS and expose each app's own
# self-signed cert, which nothing in the cluster trusts.
rewrite name authentik.riotpiao.homelab.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name grafana.riotpiao.homelab.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name minio.riotpiao.homelab.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name minio-api.riotpiao.homelab.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name argocd.riotpiao.homelab.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name vault.riotpiao.homelab.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name loki.riotpiao.homelab.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name prometheus.riotpiao.homelab.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name portainer.riotpiao.homelab.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name longhorn.riotpiao.homelab.com ingress-nginx-controller.ingress-nginx.svc.cluster.local
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
forward . 8.8.8.8 1.1.1.1 {
max_concurrent 1000
}
cache 30 {
disable success cluster.local
disable denial cluster.local
}
loop
reload
loadbalance
}
k8s/duckdns/duckdns-corn.yaml
+40
View File
@@ -0,0 +1,40 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: duckdns-updater
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: duckdns-updater
template:
metadata:
labels:
app: duckdns-updater
spec:
containers:
- name: updater
image: curlimages/curl:latest
command:
- sh
- -c
- |
while true; do
curl -fsS "https://www.duckdns.org/update?domains=riotpiao&token=${DUCKDNS_TOKEN}&ip="
sleep 300
done
env:
- name: DUCKDNS_TOKEN
valueFrom:
secretKeyRef:
name: duckdns-token
key: token
resources:
requests:
cpu: 5m
memory: 16Mi
limits:
cpu: 50m
memory: 32Mi
restartPolicy: Always
k8s/ingress/ingress.yaml
+297
View File
@@ -0,0 +1,297 @@
# k8s/ingress/ingress.yaml
# Ingress rules for all homelab services.
# TLS is handled centrally: nginx serves the wildcard-tls cert (*.riotpiao.homelab.com)
# as its default-ssl-certificate. No per-rule tls: blocks or cert-manager annotations
# are needed — cert-manager manages one cert, nginx uses it for all hosts.
#
# DNS: *.riotpiao.homelab.com must resolve to 10.6.0.1 (WireGuard) or 192.168.1.160 (LAN).
# ── Grafana ───────────────────────────────────────────────────────────────────
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: grafana
namespace: logging
annotations:
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
spec:
ingressClassName: nginx
rules:
- host: grafana.riotpiao.homelab.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: grafana
port:
number: 80
---
# ── Loki (API access for external tools) ─────────────────────────────────────
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: loki
namespace: logging
spec:
ingressClassName: nginx
rules:
- host: loki.riotpiao.homelab.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: loki
port:
number: 3100
---
# ── Authentik ─────────────────────────────────────────────────────────────────
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: authentik
namespace: iam
annotations:
nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
spec:
ingressClassName: nginx
rules:
- host: authentik.riotpiao.homelab.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: authentik-server
port:
number: 80
---
# ── Vault ─────────────────────────────────────────────────────────────────────
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: vault
namespace: storage
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
ingressClassName: nginx
rules:
- host: vault.riotpiao.homelab.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: vault
port:
number: 8200
---
# ── MinIO console (storage namespace) ────────────────────────────────────────
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: minio
namespace: storage
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: "minio-console-affinity"
nginx.ingress.kubernetes.io/session-cookie-max-age: "3600"
spec:
ingressClassName: nginx
rules:
- host: minio.riotpiao.homelab.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: minio-console
port:
number: 9001
---
# ── MinIO S3 API ──────────────────────────────────────────────────────────────
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: minio-api
namespace: storage
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
spec:
ingressClassName: nginx
rules:
- host: minio-api.riotpiao.homelab.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: minio
port:
number: 9000
---
# ── Prometheus ────────────────────────────────────────────────────────────────
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prometheus
namespace: monitoring
spec:
ingressClassName: nginx
rules:
- host: prometheus.riotpiao.homelab.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: prometheus-kube-prometheus-prometheus
port:
number: 9090
---
# ── Portainer ─────────────────────────────────────────────────────────────────
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: portainer
namespace: dashboard
annotations:
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
spec:
ingressClassName: nginx
rules:
- host: portainer.riotpiao.homelab.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: portainer
port:
number: 9000
---
# ── Forgejo (git forge + OCI registry UI) ────────────────────────────────────
# nginx terminates TLS using the wildcard cert, then proxies plain HTTP to
# Forgejo on port 3000. ROOT_URL stays https:// so Forgejo generates correct URLs.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: forgejo
namespace: cicd
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
spec:
ingressClassName: nginx
rules:
- host: forgejo.riotpiao.homelab.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: forgejo-gitea-http
port:
number: 3000
---
# ── Argo CD (cicd namespace) ──────────────────────────────────────────────────
# argocd-server runs HTTPS internally — nginx proxies via backend-protocol: HTTPS.
# proxy-ssl-verify: off because argocd-server's pod cert is self-signed (not homelab-ca).
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: argocd
namespace: cicd
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/proxy-ssl-verify: "off"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
spec:
ingressClassName: nginx
rules:
- host: argocd.riotpiao.homelab.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: argocd-server
port:
number: 443
---
# ── Longhorn UI ───────────────────────────────────────────────────────────────
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: longhorn
namespace: longhorn-system
spec:
ingressClassName: nginx
rules:
- host: longhorn.riotpiao.homelab.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: longhorn-frontend
port:
number: 80
---
# ── Temporal Web UI ────────────────────────────────────────────────────────────
# Temporal workflow orchestration Web UI with OIDC authentication
# TLS: wildcard cert managed by cert-manager, served by nginx default-ssl-certificate
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: temporal
namespace: temporal
annotations:
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
spec:
ingressClassName: nginx
rules:
- host: temporal.riotpiao.homelab.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: temporal-web
port:
number: 8080
k8s/ingress/nginx-values.yaml
+78
View File
@@ -0,0 +1,78 @@
# k8s/ingress/nginx-values.yaml
# Nginx Ingress Controller — bare-metal homelab config.
# LoadBalancer service with Cilium LB-IPAM assigns fixed IP (192.168.1.160).
# Access services at https://grafana.riotpiao.homelab.com (80/443 via LoadBalancer).
controller:
kind: DaemonSet
# Single wildcard cert served for every *.riotpiao.homelab.com host.
# Applied by the ingress-nginx presync hook (wildcard-cert.yaml) before nginx starts.
# nginx hot-reloads when cert-manager renews homelab-tls — no restart needed.
extraArgs:
default-ssl-certificate: "ingress-nginx/homelab-tls"
hostPort:
enabled: true
ports:
http: 80
https: 443
# TCP proxy: forward port 2222 on every node → Forgejo SSH service.
# This lets `git clone [email protected]:repo` work via the
# same hostname as HTTPS without a separate LoadBalancer IP for SSH.
tcp:
2222: "cicd/forgejo-gitea-ssh:2222"
# Service as LoadBalancer — Cilium LB-IPAM assigns fixed IP.
service:
type: LoadBalancer
annotations:
io.cilium/lb-ipam-ips: "192.168.1.160"
# Allow the controller to land on the control-plane node.
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
# Use the ingress-nginx IngressClass by default.
ingressClassResource:
default: true
# Required when using hostPort so DNS resolves correctly inside the pod.
dnsPolicy: ClusterFirstWithHostNet
# Reduce noise in a single-admin homelab.
admissionWebhooks:
enabled: false
# ── Connection timeouts to upstreams ──────────────────────────────────────────
# Increased to tolerate 5+ second pod-to-pod network latency spikes.
# Default: 60s for all — acceptable but explicitly set for clarity.
config:
upstream-connect-timeout: "60"
upstream-send-timeout: "60"
upstream-read-timeout: "60"
keepalive-timeout: "65"
keepalive-requests: "100"
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 200m
memory: 128Mi
# RED metrics (rate/errors/duration) for every host fronted by this controller —
# every exposed service in the cluster goes through here, so this single block
# is the cluster-wide "latency and availability" signal. Prometheus auto-discovers
# the ServiceMonitor (serviceMonitorSelectorNilUsesHelmValues: false in prometheus-values.yaml).
metrics:
enabled: true
serviceMonitor:
enabled: true
namespace: ingress-nginx
interval: 30s
scrapeTimeout: 60s
k8s/ingress/wildcard-cert.yaml
+22
View File
@@ -0,0 +1,22 @@
# k8s/ingress/wildcard-cert.yaml
# Single wildcard TLS certificate for all *.riotpiao.homelab.com services.
# Lives in the ingress-nginx namespace and is set as nginx's default-ssl-certificate,
# so every ingress host gets it automatically — no per-service TLS blocks needed.
#
# Renewal: cert-manager auto-renews 30 days before expiry (renewBefore: 720h).
# nginx detects the secret update via its K8s watch and hot-reloads — no pod restart.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: homelab-tls
namespace: ingress-nginx
spec:
secretName: homelab-tls
dnsNames:
- "*.riotpiao.homelab.com"
- "riotpiao.homelab.com"
issuerRef:
name: homelab-ca
kind: ClusterIssuer
duration: 8760h # 1 year
renewBefore: 720h # renew 30 days before expiry
k8s/llm/README.md
+299
View File
@@ -0,0 +1,299 @@
# Ollama LLM Inference Service
CPU-only LLM inference server on talos-cp-1. Single model hot-loaded (DeepSeek-R1:70b), 42GB, 70Gi memory limit.
## Quick Start
### Access via port-forward
```bash
kubectl -n llm port-forward svc/ollama 11434:11434
curl http://localhost:11434/api/tags
```
### Debug pod (in-cluster)
```bash
kubectl run debug --rm -it -n llm --image=curlimages/curl \
--labels="app.kubernetes.io/role=llm-debug" \
--serviceaccount=llm-worker -- sh
# Inside pod
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl -H "Authorization: Bearer $TOKEN" \
http://ollama.llm.svc.cluster.local:11434/api/tags
```
## Architecture
| Component | Value |
|-----------|-------|
| Service | ClusterIP `ollama.llm.svc.cluster.local:11434` |
| Namespace | `llm` |
| Node | talos-cp-1 (pinned via nodeAffinity) |
| Memory request | 50Gi |
| Memory limit | 70Gi |
| Storage | 115Gi PVC (Longhorn) |
| Model | `deepseek-r1:70b` (~42GB) |
| Max loaded | 1 model |
| Parallelism | 1 request at a time |
## API Endpoints
### List models
```bash
curl http://ollama.llm.svc.cluster.local:11434/api/tags
```
Response:
```json
{
"models": [
{"name": "deepseek-r1:70b", "size": 42000000000, ...}
]
}
```
### Generate (non-streaming)
```bash
curl -X POST http://ollama.llm.svc.cluster.local:11434/api/generate \
-H "Content-Type: application/json" \
-d '{
"model": "deepseek-r1:70b",
"prompt": "Why is the sky blue?",
"stream": false
}'
```
### Pull model
```bash
curl -X POST http://ollama.llm.svc.cluster.local:11434/api/pull \
-H "Content-Type: application/json" \
-d '{"name": "deepseek-r1:70b", "stream": false}'
```
## Operations
### Check pod status
```bash
kubectl -n llm get pod -l app.kubernetes.io/name=ollama
kubectl -n llm describe pod -l app.kubernetes.io/name=ollama
```
### View logs
```bash
kubectl -n llm logs deployment/ollama -f
```
### Monitor download progress (bootstrap)
```bash
kubectl -n llm logs -f job/bootstrap-models -c model-download
```
### Restart deployment
```bash
kubectl -n llm rollout restart deployment/ollama
```
## Storage
- **PVC:** `ollama-models-cache`, 115Gi, Longhorn StorageClass
- **Mount:** `/root/.ollama/models` (Ollama model cache)
- **Lifecycle:** RWO (Read-Write-Once), tied to talos-cp-1
### Resize PVC
⚠️ PVCs can only expand, not shrink. Edit values.yaml and redeploy:
```yaml
pvc:
size: 120Gi # increase only
```
```bash
vsource .env && helmfile -f helmfile.yaml.gotmpl -l name=ollama apply
```
## Networking
### NetworkPolicy
- Default-deny ingress on Ollama pods
- Allow from pods labeled `app.kubernetes.io/name: llm-worker` (port 11434)
- Allow from pods labeled `app.kubernetes.io/role: llm-debug` (port 11434)
View policy:
```bash
kubectl -n llm get networkpolicy ollama
```
Test access from external pod (should fail):
```bash
kubectl run test --rm -it --image=curlimages/curl -- \
curl http://ollama.llm.svc.cluster.local:11434/
# Connection timeout (correct)
```
Test access from debug pod (should succeed):
```bash
kubectl -n llm logs job/bootstrap-models # verify bootstrap completed
# Then run debug pod as shown above
```
## Configuration
### Helm values (`k8s/llm/charts/ollama/values.yaml`)
```yaml
resources:
requests:
cpu: 8
memory: 50Gi
limits:
cpu: 16
memory: 70Gi
env:
OLLAMA_MAX_LOADED_MODELS: "1"
OLLAMA_NUM_PARALLEL: "1"
OLLAMA_MAX_QUEUE: "32"
OLLAMA_KEEP_ALIVE: "-1"
OLLAMA_HOST: "0.0.0.0:11434"
preloadJob:
enabled: true
hotModels:
- deepseek-r1:70b
```
### Environment variables
| Variable | Value | Purpose |
|----------|-------|---------|
| `OLLAMA_MODELS` | `/root/.ollama/models` | Model cache dir |
| `OLLAMA_MAX_LOADED_MODELS` | `1` | Max concurrent models in RAM |
| `OLLAMA_NUM_PARALLEL` | `1` | Parallel request threads |
| `OLLAMA_MAX_QUEUE` | `32` | Request queue depth |
| `OLLAMA_KEEP_ALIVE` | `-1` | Keep model resident (never unload) |
| `OLLAMA_HOST` | `0.0.0.0:11434` | Bind address |
Tune `OLLAMA_NUM_PARALLEL` based on CPU cores. Current: 1 (conservative, CPU bottleneck).
## Model Management
### Current model
- **Name:** `deepseek-r1:70b`
- **Size:** ~42GB
- **Quantization:** Default Ollama quant
- **Status:** Downloaded during pod init via bootstrap job
### Change model
1. Edit `values.yaml`:
```yaml
preloadJob:
hotModels:
- deepseek-r1:32b # or any available model
```
2. Redeploy:
```bash
kubectl -n llm delete job bootstrap-models --ignore-not-found
vsource .env && helmfile -f helmfile.yaml.gotmpl -l name=ollama apply
```
3. Monitor:
```bash
kubectl -n llm logs -f job/bootstrap-models -c model-download
```
### Available models
Ollama registry: https://ollama.com/library
Examples:
- `deepseek-r1:70b` (reasoning, 42GB)
- `deepseek-r1:32b` (faster, 20GB)
- `llama3.1:70b` (general, 41GB)
- `mistral:large` (26GB)
## Troubleshooting
### Pod stuck in `ContainerCreating`
```bash
kubectl -n llm describe pod -l app.kubernetes.io/name=ollama
# Check Events section for PVC/image pull issues
```
### Bootstrap job failing
```bash
kubectl -n llm logs job/bootstrap-models -c model-download --tail=50
# Common: model not found in registry, disk full, network timeout
```
### Model pull timeout
```bash
# Increase pod timeout (edit deployment directly)
kubectl -n llm edit deployment ollama
# Change readinessProbe.initialDelaySeconds, livenessProbe.periodSeconds
```
### Out of memory
Model size exceeds limit. Reduce `memory.limits` or choose smaller model.
```bash
kubectl top pod -n llm # check actual usage
```
### Cannot connect from other pods
Verify NetworkPolicy:
```bash
kubectl -n llm get networkpolicy
kubectl -n llm describe networkpolicy ollama
# Add pod label: app.kubernetes.io/name: llm-worker or app.kubernetes.io/role: llm-debug
```
## Secrets
Ollama pod receives MinIO credentials via Secret `ollama-minio` (created by helmfile presync):
```bash
kubectl -n llm get secret ollama-minio -o jsonpath='{.data}' | jq
```
Keys: `endpoint`, `bucket`, `access_key`, `secret_key`
Used by bootstrap job to upload model blobs to MinIO (future: auto-backup).
## Metrics & Observability
### Prometheus scrape (if enabled)
ServiceMonitor: Not yet configured (see `k8s/monitoring/dashboards/services/`)
Metrics to add:
- `ollama_requests_total` (counter)
- `ollama_request_duration_seconds` (histogram)
- `ollama_loaded_models` (gauge)
### Logs
Pod logs via kubectl. No log aggregation to Loki yet.
```bash
kubectl -n llm logs deployment/ollama -f --timestamps
```
## Cleanup
### Delete Ollama completely
```bash
vsource .env && helmfile -f helmfile.yaml.gotmpl -l name=ollama destroy
# Keeps PVC (data safety). To delete: kubectl -n llm delete pvc ollama-models-cache
```
### Delete just the model cache (keep deployment)
```bash
kubectl -n llm delete pvc ollama-models-cache
# Recreate: kubectl -n llm patch deployment ollama -p '{"spec":{"template":{"metadata":{"annotations":{"restart":"now"}}}}}'
```
## See Also
- Helmfile: `helmfile.yaml.gotmpl` (llm release block)
- Chart: `k8s/llm/charts/ollama/`
- Namespace: `llm`
- Bootstrap: `k8s/llm/bootstrap-models-job.yaml` (manual preload fallback)
k8s/llm/charts/ollama/Chart.yaml
+6
View File
@@ -0,0 +1,6 @@
apiVersion: v2
name: ollama
description: CPU-only Ollama LLM server with MinIO model registry
type: application
version: 0.1.0
appVersion: "latest"
k8s/llm/charts/ollama/templates/deployment.yaml
+126
View File
@@ -0,0 +1,126 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: ollama
namespace: llm
labels:
app.kubernetes.io/name: ollama
app.kubernetes.io/part-of: llm
spec:
replicas: {{ .Values.replicaCount }}
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: ollama
template:
metadata:
labels:
app.kubernetes.io/name: ollama
app.kubernetes.io/part-of: llm
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- {{ .Values.nodeAffinity.zone }}
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Equal
value: ""
effect: NoSchedule
initContainers:
- name: preload-model
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
command:
- sh
- -c
- |
set -e
echo "Starting Ollama server for model preload..."
ollama serve &
OLLAMA_PID=$!
sleep 10
{{- range .Values.preloadJob.hotModels }}
echo "Preloading {{ . }}..."
if ollama ls | grep -q "{{ . }}"; then
echo "✓ {{ . }} already cached"
else
ollama pull {{ . }}
fi
{{- end }}
echo "Model preload complete"
kill $OLLAMA_PID || true
wait $OLLAMA_PID 2>/dev/null || true
volumeMounts:
- name: models-cache
mountPath: /root/.ollama/models
env:
- name: OLLAMA_HOST
value: "127.0.0.1:11434"
containers:
- name: ollama
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- containerPort: 11434
name: http
env:
{{- range $key, $value := .Values.env }}
- name: {{ $key }}
value: "{{ $value }}"
{{- end }}
- name: OLLAMA_MODELS_MINIO_ENDPOINT
valueFrom:
secretKeyRef:
name: ollama-minio
key: endpoint
- name: OLLAMA_MODELS_MINIO_BUCKET
valueFrom:
secretKeyRef:
name: ollama-minio
key: bucket
- name: OLLAMA_MODELS_MINIO_ACCESS_KEY
valueFrom:
secretKeyRef:
name: ollama-minio
key: access_key
- name: OLLAMA_MODELS_MINIO_SECRET_KEY
valueFrom:
secretKeyRef:
name: ollama-minio
key: secret_key
resources:
requests:
cpu: {{ .Values.resources.requests.cpu }}
memory: {{ .Values.resources.requests.memory }}
limits:
cpu: {{ .Values.resources.limits.cpu }}
memory: {{ .Values.resources.limits.memory }}
livenessProbe:
httpGet:
path: /
port: 11434
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /
port: 11434
initialDelaySeconds: 10
periodSeconds: 5
volumeMounts:
- name: models-cache
mountPath: /root/.ollama/models
volumes:
- name: models-cache
persistentVolumeClaim:
claimName: ollama-models-cache
k8s/llm/charts/ollama/templates/networkpolicy.yaml
+28
View File
@@ -0,0 +1,28 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: ollama-access
namespace: llm
labels:
app.kubernetes.io/name: ollama
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: ollama
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: llm-worker
ports:
- protocol: TCP
port: 11434
- from:
- podSelector:
matchLabels:
app.kubernetes.io/role: llm-debug
ports:
- protocol: TCP
port: 11434
k8s/llm/charts/ollama/templates/preload-job.yaml
+92
View File
@@ -0,0 +1,92 @@
{{- if .Values.preloadJob.enabled }}
apiVersion: batch/v1
kind: Job
metadata:
name: ollama-preload
namespace: llm
labels:
app.kubernetes.io/name: ollama-preload
spec:
backoffLimit: 3
template:
spec:
serviceAccountName: default
restartPolicy: Never
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- az-a
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Equal
effect: NoSchedule
initContainers:
- name: model-cache-init
image: ollama/ollama:latest
imagePullPolicy: IfNotPresent
command:
- sh
- -c
- |
set -e
echo "Starting Ollama server to cache models..."
ollama serve &
OLLAMA_PID=$!
sleep 10
echo "Caching hot-tier models..."
{{- range .Values.preloadJob.hotModels }}
echo "Checking if {{ . }} is cached..."
if ollama ls | grep -q "{{ . }}"; then
echo "✓ {{ . }} already cached, skipping"
else
echo "Pulling {{ . }}..."
ollama pull {{ . }}
fi
{{- end }}
echo "Model cache initialization complete"
kill $OLLAMA_PID || true
wait $OLLAMA_PID 2>/dev/null || true
volumeMounts:
- name: models
mountPath: /root/.ollama
env:
- name: OLLAMA_HOST
value: "127.0.0.1:11434"
containers:
- name: cache-populate
image: curlimages/curl:latest
imagePullPolicy: IfNotPresent
command:
- sh
- -c
- |
set -e
echo "Waiting for Ollama pod to be ready..."
until curl -f http://ollama.llm.svc.cluster.local:11434/api/tags 2>/dev/null; do
echo "Ollama not ready, waiting..."
sleep 5
done
echo "Ollama is ready, populating local cache..."
{{- range .Values.preloadJob.hotModels }}
echo "Checking if {{ . }} is already cached..."
if curl -s http://ollama.llm.svc.cluster.local:11434/api/tags | grep -q "{{ . }}"; then
echo "✓ {{ . }} already cached, skipping"
else
echo "Caching {{ . }} locally..."
curl -X POST http://ollama.llm.svc.cluster.local:11434/api/pull \
-H "Content-Type: application/json" \
-d '{"name":"{{ . }}","stream":false}'
fi
{{- end }}
echo "Local cache population complete"
volumes:
- name: models
emptyDir: {}
{{- end }}
k8s/llm/charts/ollama/templates/pvc.yaml
+14
View File
@@ -0,0 +1,14 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ollama-models-cache
namespace: llm
labels:
app.kubernetes.io/name: ollama
spec:
accessModes:
- ReadWriteOnce
storageClassName: {{ .Values.pvc.storageClassName }}
resources:
requests:
storage: {{ .Values.pvc.size }}
k8s/llm/charts/ollama/templates/service.yaml
+16
View File
@@ -0,0 +1,16 @@
apiVersion: v1
kind: Service
metadata:
name: ollama
namespace: llm
labels:
app.kubernetes.io/name: ollama
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.port }}
targetPort: http
protocol: TCP
name: http
selector:
app.kubernetes.io/name: ollama
k8s/llm/charts/ollama/templates/storageclass.yaml
+12
View File
@@ -0,0 +1,12 @@
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: longhorn-llm
labels:
app.kubernetes.io/name: ollama
provisioner: driver.longhorn.io
parameters:
numberOfReplicas: "1"
staleReplicaTimeout: "2880"
reclaimPolicy: Retain
allowVolumeExpansion: true
k8s/llm/charts/ollama/values.yaml
+40
View File
@@ -0,0 +1,40 @@
replicaCount: 1
image:
repository: ollama/ollama
pullPolicy: IfNotPresent
tag: "latest"
service:
type: ClusterIP
port: 11434
resources:
requests:
cpu: 8
memory: 60Gi
limits:
cpu: 16
memory: 100Gi
pvc:
enabled: true
size: 115Gi
storageClassName: longhorn-llm
nodeAffinity:
zone: az-a
env:
OLLAMA_MODELS: /root/.ollama/models
OLLAMA_MAX_LOADED_MODELS: "2"
OLLAMA_NUM_PARALLEL: "2"
OLLAMA_MAX_QUEUE: "64"
OLLAMA_KEEP_ALIVE: "-1"
OLLAMA_HOST: "0.0.0.0:11434"
preloadJob:
enabled: false
hotModels:
- ornith:35b
- deepseek-r1:70b
k8s/llm/scripts/setup-minio-bucket.sh
+71
View File
@@ -0,0 +1,71 @@
#!/bin/bash
# setup-minio-bucket.sh
# Creates MinIO bucket and Kubernetes secrets for Ollama LLM server
# Runs as helmfile presync hook; all commands are idempotent
# Dependencies: kubectl, access to minio-az-a pod in storage namespace
# Environment: MINIO_ROOT_USER, MINIO_ROOT_PASSWORD (from Vault), AUTHENTIK_OLLAMA_CLIENT_ID, AUTHENTIK_OLLAMA_CLIENT_SECRET
set -e
echo "=== Step 1: Create and label llm namespace ==="
kubectl create namespace llm --dry-run=client -o yaml | kubectl apply -f -
kubectl label namespace llm \
pod-security.kubernetes.io/enforce=baseline \
pod-security.kubernetes.io/enforce-version=latest \
--overwrite
echo "✓ llm namespace created/labeled"
echo ""
echo "=== Step 2: Create MinIO bucket riotpiao-models ==="
# Configure mc host inside MinIO pod
kubectl -n storage exec deployment/minio-az-a -- \
mc config host add local http://localhost:9000 \
"${MINIO_ROOT_USER}" "${MINIO_ROOT_PASSWORD}"
echo "✓ mc host configured"
# Create bucket (idempotent)
kubectl -n storage exec deployment/minio-az-a -- \
mc mb --ignore-existing local/riotpiao-models
echo "✓ MinIO bucket riotpiao-models created (or already exists)"
# Enable versioning for model rollback safety
kubectl -n storage exec deployment/minio-az-a -- \
mc version enable local/riotpiao-models
echo "✓ Versioning enabled on riotpiao-models bucket"
echo ""
echo "=== Step 3: Create ollama-minio Secret (MinIO credentials) ==="
kubectl create secret generic ollama-minio -n llm \
--from-literal=endpoint="http://minio-az-a.storage:9000" \
--from-literal=bucket="riotpiao-models" \
--from-literal=access_key="${MINIO_ROOT_USER}" \
--from-literal=secret_key="${MINIO_ROOT_PASSWORD}" \
--dry-run=client -o yaml | kubectl apply -f -
echo "✓ Secret ollama-minio created/updated"
echo ""
echo "=== Step 4: Create ollama-oidc Secret (Authentik credentials) ==="
kubectl create secret generic ollama-oidc -n llm \
--from-literal=client_id="${AUTHENTIK_OLLAMA_CLIENT_ID}" \
--from-literal=client_secret="${AUTHENTIK_OLLAMA_CLIENT_SECRET}" \
--dry-run=client -o yaml | kubectl apply -f -
echo "✓ Secret ollama-oidc created/updated"
echo ""
echo "=== Verification ==="
echo ""
echo "Run these commands to verify:"
echo " kubectl -n llm get secret ollama-minio ollama-oidc"
echo " kubectl -n storage exec deployment/minio-az-a -- mc ls local/riotpiao-models"
echo ""
echo "Setup complete!"
k8s/portainer/bootstrap.sh
Executable
+54
View File
@@ -0,0 +1,54 @@
#!/usr/bin/env bash
# portainer/bootstrap.sh
# Deploys Portainer CE into the dashboard namespace.
# No credentials needed — Portainer prompts you to create an admin account
# on first browser visit.
#
# Prerequisites:
# - kubectl configured (KUBECONFIG pointing to cluster-config/kubeconfig)
# - helm >= 3.x
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
KUBECONFIG="${KUBECONFIG:-${REPO_ROOT}/cluster-config/kubeconfig}"
export KUBECONFIG
# ── Namespace ─────────────────────────────────────────────────────────────────
echo "==> Creating dashboard namespace..."
kubectl create namespace dashboard --dry-run=client -o yaml | kubectl apply -f -
kubectl label namespace dashboard \
pod-security.kubernetes.io/enforce=privileged \
pod-security.kubernetes.io/enforce-version=latest \
--overwrite
# ── Helm repo ─────────────────────────────────────────────────────────────────
echo "==> Adding Portainer Helm repo..."
helm repo add portainer https://portainer.github.io/k8s/
helm repo update portainer
# ── Portainer ─────────────────────────────────────────────────────────────────
echo "==> Installing Portainer..."
helm upgrade --install portainer portainer/portainer \
--namespace dashboard \
--values "${SCRIPT_DIR}/portainer-values.yaml" \
--wait \
--timeout 5m
echo "==> Waiting for Portainer Deployment to be ready..."
kubectl rollout status deployment/portainer -n dashboard --timeout=120s
# ── Done ──────────────────────────────────────────────────────────────────────
echo ""
echo "==> Portainer is up."
echo ""
echo "Access Portainer UI:"
echo " make pf-portainer"
echo " http://localhost:9000"
echo ""
echo "First-time setup: Portainer will prompt you to create an admin account."
echo "Choose 'Manage the local Kubernetes environment' when asked."
echo ""
echo "Node failure resilience tip:"
echo " For faster PVC failover on hard node failure, enable in Longhorn UI → Settings:"
echo " nodeDownPodDeletionPolicy = delete-deployment-pod"
k8s/portainer/portainer-values.yaml
+53
View File
@@ -0,0 +1,53 @@
# k8s/portainer/portainer-values.yaml
# Portainer — web UI for browsing cluster workloads, exec-ing into pods,
# and viewing logs without kubectl. Operator-only access (ClusterIP + port-forward).
#
# Node failure behaviour:
# Portainer is a Deployment (not StatefulSet), so K8s auto-evicts and
# reschedules it ~5 min after a node becomes unreachable. Longhorn
# reattaches the PVC on the new node in ~1-2 min. Worst case: ~7-10 min.
#
# To cut that down: in Longhorn UI → Settings set
# nodeDownPodDeletionPolicy = delete-deployment-pod
# Longhorn will force-delete the stuck pod immediately when the node is
# fenced rather than waiting for Kubernetes' eviction timeout.
# ── Service ───────────────────────────────────────────────────────────────────
# ClusterIP — no external exposure. Access via:
# kubectl -n dashboard port-forward svc/portainer 9000:9000
# Portainer holds cluster-admin credentials; never expose as LoadBalancer.
service:
type: ClusterIP
# ── TLS ───────────────────────────────────────────────────────────────────────
# Portainer by default redirects HTTP → HTTPS using a self-signed cert.
# force: false disables the redirect so plain HTTP over port-forward works
# without browser cert warnings. TLS is terminated at the ingress layer
# if/when an ingress rule is added.
tls:
force: false
# ── Persistence ───────────────────────────────────────────────────────────────
# Stores Portainer's own config: environment registrations, user accounts,
# stack definitions, and access control settings. Longhorn provides the
# RWO block volume. 10Gi is generous for config data but cheap on Longhorn.
persistence:
enabled: true
storageClass: "longhorn"
size: 10Gi
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
# ── Scheduling ────────────────────────────────────────────────────────────────
# Allow scheduling on talos-cp-1 (carries NoSchedule taint) so Portainer
# keeps running even when the worker node is down.
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
project-usage/authentik-oidc.md
+183
View File
@@ -0,0 +1,183 @@
# Authentik Federated OIDC & SSO
**Provider:** `https://authentik.riotpiao.homelab.com`
**OIDC Issuer:** `https://authentik.riotpiao.homelab.com/application/o/talos-federation/`
**Namespace:** `iam`
## When to Use
- **Federated login** — Single sign-on for Grafana, MinIO, Forgejo, Argo CD
- **User groups** — RBAC via group membership (admins, devops, read-only)
- **JWT tokens** — Authenticate CLI tools, API clients
- **SSO for custom apps** — OAuth2/OIDC redirect flow
## Quick Start
**1. Login to Authentik console:**
```bash
# Browser: https://authentik.riotpiao.homelab.com
# Default user: akadmin
# Password: AUTHENTIK_BOOTSTRAP_PASSWORD (from .env)
# Or via OIDC (after initial setup)
# Click "Sign in with talos-federation"
```
**2. Create user:**
```
Authentik console → Users → Create
- Username: alice
- Email: [email protected]
- Group: homelab-devs (or homelab-admins)
```
**3. User logs into Grafana:**
```
https://grafana.riotpiao.homelab.com
→ Sign in with Authentik (auto-redirects to OIDC provider)
→ Approve access
→ Logged in as alice (group determines role: Admin or Viewer)
```
## Configuration
| Key | Value |
|-----|-------|
| OIDC provider | `talos-federation` (federated) |
| OIDC issuer | `https://authentik.riotpiao.homelab.com/application/o/talos-federation/` |
| JWKS endpoint | `https://authentik.riotpiao.homelab.com/application/o/talos-federation/.well-known/openid-configuration` |
| Database | PostgreSQL (ddb namespace, authentik user) |
| Backups | WAL archived to MinIO |
## Common Patterns
**Grafana OIDC login:**
```yaml
# k8s/logging/grafana-values.yaml
grafana:
auth.generic_oauth:
enabled: true
name: Authentik
client_id: grafana
client_secret: $GRAFANA_OIDC_CLIENT_SECRET # from Vault
auth_url: https://authentik.riotpiao.homelab.com/application/o/authorize/
token_url: https://authentik.riotpiao.homelab.com/application/o/token/
api_url: https://authentik.riotpiao.homelab.com/application/o/userinfo/
scopes: openid profile email groups
use_pkce: true
```
**MinIO OIDC login:**
```yaml
# k8s/storage/minio-values.yaml
minio:
identity_oauth:
provider: authentik
client_id: minio
client_secret: $MINIO_OIDC_CLIENT_SECRET
redirect_uri: https://minio.riotpiao.homelab.com/oauth_callback
config_url: https://authentik.riotpiao.homelab.com/application/o/talos-federation/.well-known/openid-configuration
policy_mappings:
- group: homelab-admins → consoleAdmin
- group: homelab-devops → readwrite
```
**CLI device code flow (talos-cli):**
```bash
# Get JWT token (no kubeconfig needed)
talos secrets login
# → Opens browser, approve device code
# → Token cached in ~/.talos/token
# Use token to access Vault
talos get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
# → Vault validates JWT from Authentik
# → Returns secret
```
**Custom app OIDC redirect:**
```go
import "github.com/coreos/go-oidc/v3/oidc"
provider, _ := oidc.NewProvider(ctx, "https://authentik.riotpiao.homelab.com/application/o/talos-federation/")
verifier := provider.Verifier(&oidc.Config{ClientID: "my-app"})
// After OAuth2 redirect & token exchange:
idToken, _ := verifier.Verify(ctx, rawIDToken)
// Extract claims
var claims struct {
Email string `json:"email"`
Groups []string `json:"groups"`
}
idToken.Claims(&claims)
```
## Group-Based RBAC
**Default groups:**
- `homelab-admins` — Full cluster access (Grafana Admin, MinIO admin, Argo CD admin, Vault admin)
- `homelab-devops` — Deploy & monitor (Grafana Editor, MinIO readwrite, Argo CD user)
- `homelab-viewers` — Read-only (Grafana Viewer, MinIO readonly)
**Assign user to group:**
```
Authentik console → Users → alice → Edit
→ Groups → Add "homelab-devops"
→ Save
```
**Custom group-to-role mapping:**
```yaml
# Per-service (see cicd-workflow.md, monitoring-metrics.md for examples)
# Grafana: auth.generic_oauth.role_attribute_path = contains(groups[*], 'homelab-admins') && 'Admin' || 'Viewer'
# MinIO: policy_mappings (see above)
```
## Monitoring
**Authentik dashboard:** https://authentik.riotpiao.homelab.com/api/v3/admin/dashboards
**Key metrics:**
- Login attempts (success/failure)
- Active sessions
- Token issuance rate
- Provider sync status
## Troubleshooting
**Users can't login (redirect loop):**
```bash
# Check redirect URI matches
# Authentik console → Applications → grafana → Edit
# Verify Redirect URI = https://grafana.riotpiao.homelab.com/login/generic_oauth
# Check OIDC provider is running
k get pods -n iam -l app=authentik
```
**JWT token expired:**
```bash
# CLI tokens have 24h expiry
# Re-authenticate
talos secrets login
```
**Groups not syncing:**
```bash
# Check group attribute in OIDC config
# Authentik console → Applications → <app> → OIDC Configuration
# groups_attribute = "groups" (or custom claim name)
```
**Vault can't validate JWT:**
```bash
# Verify JWKS endpoint is accessible
curl https://authentik.riotpiao.homelab.com/application/o/talos-federation/.well-known/openid-configuration
# Restart Vault to refresh JWKS cache
k rollout restart -n iam deployment/vault
```
See `/TROUBLESHOOTING.md` for full incident guide.
project-usage/cicd-workflow.md
+222
View File
@@ -0,0 +1,222 @@
# CI/CD Pipeline (Forgejo + Argo CD)
**Git Forge:** `https://forgejo.riotpiao.homelab.com`
**Deployments:** `https://argocd.riotpiao.homelab.com` (or `kubectl port-forward`)
**Namespaces:** `cicd`, `forge`
## When to Use
- **Build & test** — Forgejo Actions CI (GitHub Actions syntax)
- **Image push** — Build OCI images, push to Forgejo registry
- **GitOps deployment** — Argo CD syncs deploy repo to cluster
- **Secrets in CI** — ci-bot JWT tokens, never kubeconfig
## Quick Start
**1. Clone a repo from Forgejo:**
```bash
git clone https://forgejo.riotpiao.homelab.com/rock/source.git
cd source
```
**2. Create workflow:**
```bash
mkdir -p .forgejo/workflows
cat > .forgejo/workflows/ci.yml <<EOF
name: CI
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- run: npm test
- run: docker build -t myapp:latest .
- run: |
docker login forgejo.riotpiao.homelab.com \
-u ci-bot \
-p ${{ secrets.CI_BOT_TOKEN }}
docker push forgejo.riotpiao.homelab.com/rock/myapp:latest
EOF
git add .forgejo/workflows/ci.yml
git commit -m "ci: add build workflow"
git push
```
**3. Trigger deployment:**
```bash
# Update deployment repo (rock/deploy)
git clone https://forgejo.riotpiao.homelab.com/rock/deploy.git
cd deploy
# Update image tag
sed -i 's|forgejo.riotpiao.homelab.com/rock/myapp:.*|forgejo.riotpiao.homelab.com/rock/myapp:abc123|' k8s/deployment.yaml
git add k8s/deployment.yaml
git commit -m "deploy: bump myapp to abc123"
git push
```
**4. Argo CD auto-syncs:**
```bash
# Watch deployment
k rollout status -n <app-namespace> deployment/<app-name>
# Or check Argo CD UI
kubectl port-forward -n argocd svc/argocd-server 8443:443
# https://localhost:8443 (login via Authentik)
```
## Workflow Syntax (GitHub Actions)
**Basic structure:**
```yaml
name: CI
on:
push:
branches: [main, develop]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- run: npm ci
- run: npm test
build:
needs: test # wait for test job
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- run: docker build -t myapp:${{ github.sha }} .
- run: docker push forgejo.riotpiao.homelab.com/rock/myapp:${{ github.sha }}
```
**Available variables:**
```bash
${{ github.sha }} # commit hash
${{ github.ref_name }} # branch name
${{ github.run_number }} # build #
${{ secrets.CI_BOT_TOKEN }} # injected from Forgejo
```
## Secrets & Authentication
**ci-bot JWT token (auto-injected):**
```yaml
- run: |
echo "${{ secrets.CI_BOT_TOKEN }}" | docker login \
forgejo.riotpiao.homelab.com \
-u ci-bot \
--password-stdin
docker push forgejo.riotpiao.homelab.com/rock/myapp:latest
```
**API token (for pushing commits):**
```yaml
- run: |
git config user.name "ci-bot"
git config user.email "ci-bot@homelab"
git commit --allow-empty -m "bump: version"
git push https://ci-bot:${{ secrets.CI_BOT_TOKEN }}@forgejo.riotpiao.homelab.com/rock/deploy.git main
```
**Vault secrets (via talos CLI):**
```bash
# Not available in CI runner — use Argo CD post-sync hooks instead
# Or inject via init container before workflow runs
```
## Argo CD GitOps
**Create app (one-time):**
```bash
argocd app create story-crater \
--repo https://forgejo.riotpiao.homelab.com/rock/deploy.git \
--path k8s/ \
--dest-server https://kubernetes.default.svc \
--dest-namespace story-crater-backend \
--sync-policy automated
```
**Monitor sync:**
```bash
# CLI
argocd app get story-crater
argocd app logs story-crater
# UI: https://argocd.riotpiao.homelab.com
# Login: Authentik SSO (homelab-admins group only)
```
**Manual sync:**
```bash
argocd app sync story-crater
argocd app wait story-crater
```
## Security Rules
✅ **DO:**
- Store credentials in Forgejo Secrets (auto-injected)
- Use ci-bot JWT for image push only
- Commit to deploy repo (triggers Argo CD)
- Enable branch protection (require CI pass)
❌ **DON'T:**
- Put kubeconfig in CI (Argo CD bridges gap)
- Commit secrets to source repo
- Use admin-bot in CI workflows (over-privileged)
- Push images directly to cluster (use Argo CD)
## Monitoring
**Grafana dashboard:** `svc-forgejo`, `svc-argocd`
**Key metrics:**
- `forgejo_workflows_running` — active workflows
- `argocd_app_sync_duration_seconds` — deployment time
- `argocd_app_info{sync_status="OutOfSync"}` — drift detection
## Troubleshooting
**Workflow fails silently:**
```bash
# Check runner logs
k logs -n cicd -f deploy/forgejo-runner
# Check if runner pod is healthy
k get pods -n cicd -l app=forgejo-runner
```
**Image push fails (401 Unauthorized):**
```bash
# Verify ci-bot token in Forgejo
# Settings → Applications → ci-bot → check scopes (package:write)
# Or re-create token
talos put cluster/iam/agents/ci-bot-token TOKEN="$(openssl rand -hex 32)"
```
**Argo CD out-of-sync:**
```bash
# Check deploy repo changes
argocd app diff story-crater
# Manual sync
argocd app sync story-crater --prune
```
**Webhook not triggering:**
```bash
# Verify Forgejo webhook config
# Repo Settings → Webhooks → Check delivery logs
# Or manually trigger
argocd app sync story-crater
```
See `/TROUBLESHOOTING.md` for full incident guide.
project-usage/database-postgres.md
+188
View File
@@ -0,0 +1,188 @@
# CloudNativePG PostgreSQL Database
**Host:** `ddb-cluster-rw.ddb.svc.cluster.local` (read-write)
**Read replica:** `ddb-cluster-ro.ddb.svc.cluster.local` (read-only)
**Port:** `5432`
**Namespace:** `ddb`
## When to Use
- **Multi-replica HA** — 3 replicas, automatic failover
- **pgvector extension** — Vector similarity search (LLM embeddings)
- **Transactional data** — Authentik, Story Crater backend, custom apps
- **Declarative backups** — Automated WAL archiving to MinIO
## Quick Start
**1. Connect from pod:**
```bash
# Inside a pod (inject secret mount)
psql -h ddb-cluster-rw.ddb.svc.cluster.local \
-U story_crater \
-d story_crater \
-W # prompt for password (from Secret)
```
**2. Create database & user (one-time):**
```bash
# Already done by helmfile postsync hook
# But if needed manually:
psql -h ddb-cluster-rw.ddb.svc.cluster.local \
-U postgres \
-c "CREATE DATABASE myapp OWNER postgres;"
psql -h ddb-cluster-rw.ddb.svc.cluster.local \
-U postgres \
-d myapp \
-c "CREATE USER myapp_user WITH PASSWORD 'secret';"
psql -h ddb-cluster-rw.ddb.svc.cluster.local \
-U postgres \
-d myapp \
-c "GRANT ALL PRIVILEGES ON DATABASE myapp TO myapp_user;"
```
**3. Enable pgvector:**
```bash
psql -h ddb-cluster-rw.ddb.svc.cluster.local \
-U postgres \
-d myapp \
-c "CREATE EXTENSION IF NOT EXISTS vector;"
```
**4. Create table with embeddings:**
```sql
CREATE TABLE documents (
id BIGSERIAL PRIMARY KEY,
content TEXT,
embedding vector(1536), -- OpenAI embeddings
created_at TIMESTAMP DEFAULT NOW()
);
CREATE INDEX ON documents USING IVFFLAT (embedding vector_cosine_ops);
```
## Configuration
| Key | Value |
|-----|-------|
| Host (RW) | `ddb-cluster-rw.ddb.svc.cluster.local` |
| Host (RO) | `ddb-cluster-ro.ddb.svc.cluster.local` |
| Port | 5432 |
| Replicas | 3 (automatic failover) |
| Extensions | pgvector (LLM embeddings), uuid-ossp |
| Backups | WAL archiving to MinIO (continuous) |
| Retention | 30 days |
## Common Patterns
**Connection pooling (from app):**
```go
import "github.com/jackc/pgx/v5/pgxpool"
config, _ := pgxpool.ParseConfig("postgres://user:[email protected]:5432/myapp")
config.MaxConns = 25
config.MinConns = 5
pool, _ := pgxpool.NewWithConfig(ctx, config)
// Use pool
row := pool.QueryRow(ctx, "SELECT COUNT(*) FROM users")
```
**Read from replica (analytics):**
```go
// Offload SELECT queries to read replica
pool.QueryRow(ctx, "SELECT * FROM documents LIMIT 1") // auto-routes to RO if available
// Writes always go to RW
pool.Exec(ctx, "INSERT INTO documents ...")
```
**Vector similarity search:**
```sql
SELECT id, content, embedding <-> $1 AS distance
FROM documents
ORDER BY embedding <-> $1
LIMIT 10;
-- $1 = query embedding (e.g., from OpenAI API)
```
**Backup & restore:**
```bash
# Backups are automatic (WAL to MinIO)
# To restore from backup:
# 1. Check MinIO s3://postgresql-backups/
# 2. Use PostgreSQL PITR (point-in-time recovery)
# 3. Contact SRE for restore procedure
```
## Monitoring
**Grafana dashboard:** `svc-postgresql` (auto-configured)
**Key metrics:**
- `pg_stat_activity_connections` — active connections
- `pg_stat_database_blks_read` — disk I/O
- `pg_replication_lag_seconds` — replica lag (goal: < 1s)
**CLI health check:**
```bash
# Check replication status
kubectl exec -n ddb pod/ddb-cluster-1 -- \
psql -U postgres -c "SELECT slot_name, restart_lsn FROM pg_replication_slots;"
# Check replica lag
kubectl exec -n ddb pod/ddb-cluster-2 -- \
psql -U postgres -c "SELECT now() - pg_last_xact_replay_timestamp() AS lag;"
```
## Secrets & Credentials
**All user passwords stored in Vault:**
```bash
# Read password
talos get cluster/STORY_CRATER_PG_PASSWORD --key STORY_CRATER_PG_PASSWORD
# Inject into pod (auto via Secret volume)
# Mount: /run/secrets/db-password
```
**Connection string from env:**
```bash
POSTGRES_CONNECTION="postgres://story_crater:${STORY_CRATER_PG_PASSWORD}@ddb-cluster-rw.ddb.svc.cluster.local:5432/story_crater"
```
## Troubleshooting
**Cannot connect (connection refused):**
```bash
# Verify cluster is running
k get pods -n ddb
# Check Service DNS
k exec -it pod/debug-pod -- nslookup ddb-cluster-rw.ddb.svc.cluster.local
# Verify Secret has password
k get secret -n ddb ddb-cluster-superuser -o jsonpath='{.data.password}' | base64 -d
```
**Replica lag is high (> 10s):**
```bash
# Check replica pod CPU/memory
k top pod -n ddb
# Scale down other workloads if cluster is overloaded
# Or scale up database resources (helmfile.yaml.gotmpl)
```
**pgvector queries slow:**
```sql
-- Ensure index exists
SELECT * FROM pg_indexes WHERE tablename = 'documents' AND indexname LIKE '%embedding%';
-- Re-index if missing
CREATE INDEX ON documents USING IVFFLAT (embedding vector_cosine_ops);
```
See `/TROUBLESHOOTING.md` for full incident guide.
project-usage/minio-s3.md
+162
View File
@@ -0,0 +1,162 @@
# MinIO S3-Compatible Object Storage
**Endpoint:** `https://minio.riotpiao.homelab.com` (console)
**API:** `minio.storage.svc.cluster.local:9000` (cluster-internal)
**Namespace:** `storage`
## When to Use
- **File uploads** — Images, documents, backups
- **Log backend** — Loki chunks storage
- **Vault unsealing** — Store unseal keys
- **CI/CD artifacts** — Build outputs, Docker layers cache
## Quick Start
**1. Access MinIO console:**
```bash
# Via browser: https://minio.riotpiao.homelab.com
# Credentials: MINIO_ROOT_USER / MINIO_ROOT_PASSWORD (from .env)
# Or port-forward
make pf-minio # localhost:9001
```
**2. Create bucket:**
```bash
# Via AWS CLI
export AWS_ACCESS_KEY_ID=$MINIO_ROOT_USER
export AWS_SECRET_ACCESS_KEY=$MINIO_ROOT_PASSWORD
aws s3 mb s3://my-bucket \
--endpoint-url https://minio.riotpiao.homelab.com \
--region homelab
# Or via console UI: Click "Create Bucket"
```
**3. Upload file:**
```bash
aws s3 cp /path/to/file.txt s3://my-bucket/ \
--endpoint-url https://minio.storage.svc.cluster.local:9000 \
--use-path-style
```
**4. List buckets:**
```bash
aws s3 ls --endpoint-url https://minio.storage.svc.cluster.local:9000
```
## Configuration
| Key | Value |
|-----|-------|
| Access key | `MINIO_ROOT_USER` (from .env) |
| Secret key | `MINIO_ROOT_PASSWORD` (from .env) |
| Cluster API | `minio.storage.svc.cluster.local:9000` |
| Console port | `9001` |
| Replication | 3-node site-replication (az-a ↔ az-b ↔ az-c) |
| Buckets (system) | `loki-chunks`, `loki-ruler`, `vault-backups` |
## Common Patterns
**Loki log backend (auto-configured):**
```yaml
# k8s/logging/loki-values.yaml
loki:
storage:
s3:
endpoint: minio.storage.svc.cluster.local:9000
buckets: loki-chunks
secretAccessKey: $MINIO_ROOT_PASSWORD
accessKeyId: $MINIO_ROOT_USER
```
**Application usage (Go/Python/Node):**
```go
import "github.com/minio/minio-go/v7"
client, _ := minio.New("minio.storage.svc.cluster.local:9000", &minio.Options{
Creds: credentials.NewStaticV4(os.Getenv("MINIO_ROOT_USER"), os.Getenv("MINIO_ROOT_PASSWORD"), ""),
Secure: false, // cluster-internal (no TLS)
})
// Upload
client.FPutObject(ctx, "my-bucket", "file.txt", "/path/to/file.txt", minio.PutObjectOptions{})
// Download
client.FGetObject(ctx, "my-bucket", "file.txt", "/tmp/file.txt", minio.GetObjectOptions{})
```
**Vault backup bucket:**
```bash
# Vault stores unseal keys in s3://vault-backups
# Auto-managed by helmfile; no manual action needed
```
## Monitoring
**Grafana dashboard:** `svc-minio`
**Key metrics:**
- `minio_disk_drive_free_bytes` — available space
- `minio_bucket_usage_object_count` — objects per bucket
- `minio_bucket_usage_total_bytes` — total size per bucket
**Site replication status:**
```bash
# Port-forward to MinIO pod
k port-forward -n storage pod/minio-0 9000:9000 &
# Check replication
mc alias set local http://localhost:9000 $MINIO_ROOT_USER $MINIO_ROOT_PASSWORD
mc admin replicate status local
```
## Authentication (Cluster-Internal)
**From pods (cluster-internal):**
```bash
# Use credentials from Secret or env var
export MINIO_ENDPOINT=minio.storage.svc.cluster.local:9000
export MINIO_ACCESS_KEY=$MINIO_ROOT_USER
export MINIO_SECRET_KEY=$MINIO_ROOT_PASSWORD
aws s3 ls --endpoint-url http://$MINIO_ENDPOINT --use-path-style
```
**External access (HTTPS via Ingress):**
```bash
# Console: https://minio.riotpiao.homelab.com (port 9001)
# API: Use AWS CLI with --endpoint-url https://minio.riotpiao.homelab.com:9000
```
## Troubleshooting
**Bucket creation fails:**
```bash
# Check MinIO pod logs
k logs -n storage pod/minio-0 | grep -i error
# Verify storage space
k get pvc -n storage
```
**Site replication lag:**
```bash
# Check if all 3 nodes are healthy
k get pods -n storage -l app=minio
# If one node is down, site replication queues changes (eventually consistent)
```
**Access denied:**
```bash
# Verify credentials in .env
echo $MINIO_ROOT_USER $MINIO_ROOT_PASSWORD
# If credentials rotated, update Secret
k patch secret -n storage minio-root-credentials \
--type merge -p '{"stringData":{"MINIO_ROOT_PASSWORD":"newpass"}}'
```
See `/TROUBLESHOOTING.md` for full incident guide.
project-usage/monitoring-metrics.md
+265
View File
@@ -0,0 +1,265 @@
# Monitoring: Prometheus, Grafana & Loki
**Prometheus:** `prometheus-kube-prom-prometheus.monitoring.svc.cluster.local:9090`
**Grafana:** `https://grafana.riotpiao.homelab.com`
**Loki:** `loki.logging.svc.cluster.local:3100`
**Namespaces:** `monitoring`, `logging`
## When to Use
- **Metrics** — CPU, memory, latency, request rate, custom business metrics (Prometheus)
- **Logs** — Pod logs, kernel logs, trace events (Loki)
- **Dashboards** — Real-time visualization (Grafana)
- **Alerts** — Page on high error rate, latency spike, or resource exhaustion
## Quick Start
**1. Access Grafana:**
```bash
# Browser: https://grafana.riotpiao.homelab.com
# Login: admin / GRAFANA_ADMIN_PASSWORD (from .env)
# Or via Authentik SSO
# Port-forward (if no external access)
make pf-grafana # localhost:3000
```
**2. View dashboards:**
```
Grafana → Dashboards → Homelab folder
- Service Availability (uptime probes + cert expiry)
- Latency & Golden Signals (request rate/error %/duration)
- Hardware Statistics (per-node CPU/mem/disk)
- Kube-Controller Health (API server metrics)
- Service Internals (per-service deep-dive)
```
**3. Add Prometheus data source:**
```
Grafana → Settings → Data Sources → Add
- Type: Prometheus
- URL: http://prometheus-kube-prom-prometheus.monitoring.svc.cluster.local:9090
- Save & test
```
**4. Create dashboard:**
```json
{
"dashboard": {
"title": "My Service",
"panels": [
{
"targets": [
{
"expr": "rate(http_requests_total{service='myapp'}[5m])"
}
],
"title": "Request Rate"
}
]
}
}
```
## Instrumentation Patterns
**Go (Prometheus):**
```go
import "github.com/prometheus/client_golang/prometheus"
// Counter (monotonic increase)
requestsTotal := prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "http_requests_total",
Help: "Total requests",
},
[]string{"method", "status"},
)
// Gauge (can go up or down)
activeConnections := prometheus.NewGauge(prometheus.GaugeOpts{
Name: "active_connections",
})
// Histogram (latency buckets)
requestDuration := prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name: "http_request_duration_seconds",
Buckets: []float64{.001, .01, .1, 1},
},
[]string{"method", "path"},
)
// Record metric
requestsTotal.WithLabelValues("POST", "200").Inc()
requestDuration.WithLabelValues("POST", "/api/users").Observe(0.042)
```
**Node.js (prom-client):**
```javascript
const prometheus = require('prom-client');
const httpRequestDuration = new prometheus.Histogram({
name: 'http_request_duration_seconds',
help: 'Duration of HTTP requests in seconds',
labelNames: ['method', 'path', 'status'],
buckets: [0.001, 0.01, 0.1, 1],
});
// Middleware
app.use((req, res, next) => {
const start = Date.now();
res.on('finish', () => {
const duration = (Date.now() - start) / 1000;
httpRequestDuration.labels(req.method, req.path, res.statusCode).observe(duration);
});
next();
});
// Expose metrics
app.get('/metrics', (req, res) => {
res.set('Content-Type', prometheus.register.contentType);
res.end(prometheus.register.metrics());
});
```
## ServiceMonitor (Auto-Scrape)
**Prometheus automatically discovers ServiceMonitor resources:**
```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: myapp
namespace: myapp-ns
spec:
selector:
matchLabels:
app: myapp
endpoints:
- port: metrics
interval: 30s
path: /metrics
```
**Deploy & verify:**
```bash
kubectl apply -f myapp-servicemonitor.yaml
# Check Prometheus targets
kubectl port-forward -n monitoring svc/prometheus-kube-prom-prometheus 9090:9090
# http://localhost:9090/targets → should show myapp
```
## PrometheusRule (Alerting)
**Define alert rules:**
```yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: myapp-alerts
namespace: myapp-ns
spec:
groups:
- name: myapp
rules:
- alert: HighErrorRate
expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05 # > 5% error rate
for: 5m
annotations:
summary: "High error rate for {{ $labels.service }}"
- alert: HighLatency
expr: histogram_quantile(0.99, http_request_duration_seconds_bucket) > 1 # p99 > 1s
for: 10m
annotations:
summary: "High latency for {{ $labels.path }}"
```
## Logging (Loki)
**View pod logs in Grafana:**
```
Grafana → Explore → Loki
Query: {namespace="story-crater-backend"} | json | level="error"
Label filters:
- namespace
- pod_name
- container
- level (error, warn, info)
```
**Log retention:**
- 10-day default (configurable)
- Older logs deleted automatically
- Chunks stored in MinIO (s3://loki-chunks)
## Monitoring Checklist
**For every service:**
- [ ] Export `/metrics` (Prometheus format)
- [ ] Add ServiceMonitor resource
- [ ] Add PrometheusRule (errors, latency)
- [ ] Create Grafana dashboard (6 rows: Availability, Resources, Domain, Logs, SLO, Related)
- [ ] Set up Slack/PagerDuty integration (if on-call)
## Key Queries
**Request rate by status:**
```promql
sum(rate(http_requests_total[5m])) by (status)
```
**Error rate %:**
```promql
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100
```
**P99 latency:**
```promql
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))
```
**Memory usage %:**
```promql
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
```
## Troubleshooting
**Prometheus can't scrape ServiceMonitor:**
```bash
# Check ServiceMonitor exists
k get servicemonitor -A | grep myapp
# Verify Service has correct labels
k get svc -A -o json | jq '.items[] | select(.metadata.name=="myapp") | .metadata.labels'
# Check Prometheus targets
kubectl port-forward -n monitoring svc/prometheus-kube-prom-prometheus 9090:9090
# http://localhost:9090/targets → look for "Down" targets with error
```
**Metrics missing from Grafana:**
```bash
# Verify pod is exporting metrics
k port-forward -n myapp-ns pod/myapp-0 8080:8080
curl http://localhost:8080/metrics | grep my_metric_name
# Check query syntax in Grafana
# Hover over query → "Query Inspector" to see response
```
**Alert not firing:**
```bash
# Check PrometheusRule is loaded
k get prometheusrule -n myapp-ns
# Verify query in Prometheus
kubectl port-forward -n monitoring svc/prometheus-kube-prom-prometheus 9090:9090
# http://localhost:9090 → Graph tab → paste query
```
See `/TROUBLESHOOTING.md` for full incident guide.
project-usage/networking-ingress.md
+314
View File
@@ -0,0 +1,314 @@
# Networking: Ingress, TLS & Service Discovery
**Ingress Controller:** `nginx-ingress` (Nginx)
**Load Balancer:** Cilium LB-IPAM (eBPF-based)
**TLS CA:** homelab-ca (self-signed, 10-year)
**Namespace:** `ingress-nginx`
## When to Use
- **Public HTTPS endpoints** — External access via TLS
- **Hostname-based routing** — Multiple services on same IP
- **TLS termination** — Offload encryption/decryption
- **Service discovery** — Internal DNS (CoreDNS)
## Quick Start
**1. Create Ingress rule:**
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myapp
namespace: myapp-ns
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-prod" # or homelab-ca
spec:
ingressClassName: nginx
tls:
- hosts:
- myapp.riotpiao.homelab.com
secretName: myapp-tls
rules:
- host: myapp.riotpiao.homelab.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: myapp-svc
port:
number: 8080
```
**2. Deploy:**
```bash
kubectl apply -f ingress.yaml
# Wait for cert issuance
kubectl get certificate -n myapp-ns
# Should show "Ready" after ~30s
```
**3. Test from client:**
```bash
# Add to /etc/hosts (or use WireGuard)
192.168.1.160 myapp.riotpiao.homelab.com
# Access
curl https://myapp.riotpiao.homelab.com
```
## Configuration
| Key | Value |
|-----|-------|
| Ingress class | `nginx` |
| Load balancer type | `LoadBalancer` (Cilium LB-IPAM) |
| TLS issuer | `homelab-ca` (ClusterIssuer) |
| TLS cert lifetime | 90 days (auto-renewed by cert-manager) |
| DNS | CoreDNS (in-cluster), external via `/etc/hosts` or DuckDNS |
## Common Patterns
**Ingress with path-based routing:**
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: api
namespace: default
spec:
ingressClassName: nginx
tls:
- hosts:
- api.riotpiao.homelab.com
secretName: api-tls
rules:
- host: api.riotpiao.homelab.com
http:
paths:
- path: /users
pathType: Prefix
backend:
service:
name: users-svc
port:
number: 3000
- path: /orders
pathType: Prefix
backend:
service:
name: orders-svc
port:
number: 3001
```
**Ingress with basic auth:**
```bash
# Generate htpasswd
htpasswd -c auth admin
# → prompted for password
# Create Secret
kubectl create secret generic basic-auth --from-file=auth -n default
# Create Ingress
```
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: protected
namespace: default
annotations:
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: basic-auth
nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required'
spec:
ingressClassName: nginx
rules:
- host: protected.riotpiao.homelab.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: app-svc
port:
number: 8080
```
**Internal DNS (CoreDNS rewrite):**
```yaml
# k8s/coredns/coredns-configmap.yaml
# Rewrite:
# - grafana.riotpiao.homelab.com → grafana.logging (cluster-internal)
# - prometheus.riotpiao.homelab.com → prometheus-kube-prom-prometheus.monitoring
#
# Allows pods to use external URLs but resolve to internal Services
```
**Fixed LoadBalancer IP (Cilium LB-IPAM):**
```yaml
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: ingress-nginx
annotations:
io.cilium/lb-ipam-ips: "192.168.1.160" # fixed IP
spec:
type: LoadBalancer
selector:
app: nginx-ingress
ports:
- port: 443
targetPort: 443
protocol: TCP
```
## TLS Certificate Management
**Automatic renewal (cert-manager):**
```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: myapp-cert
namespace: myapp-ns
spec:
secretName: myapp-tls
duration: 2160h # 90 days
renewBefore: 360h # renew 15 days before expiry
commonName: myapp.riotpiao.homelab.com
dnsNames:
- myapp.riotpiao.homelab.com
issuerRef:
name: homelab-ca
kind: ClusterIssuer
```
**Check certificate status:**
```bash
# List certs
k get certificate -A
# View cert details
k describe certificate -n myapp-ns myapp-cert
# View TLS Secret
k get secret -n myapp-ns myapp-tls -o json | jq '.data."tls.crt"' | base64 -d | openssl x509 -text
# Check expiry date
k get secret -n myapp-ns myapp-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate
```
## Service Discovery
**Cluster-internal DNS:**
```bash
# From any pod, resolve via CoreDNS
nslookup grafana.logging.svc.cluster.local # full FQDN
nslookup grafana.logging # short form (same namespace)
nslookup grafana # if in logging namespace
# Resolved to ClusterIP (internal only)
```
**External DNS (WireGuard VPN or port-forward):**
```bash
# Option 1: WireGuard tunnel
# Client connects to 10.6.0.1 (WireGuard server on talos-cp-1)
# All traffic tunneled to cluster
# Option 2: Port-forward from jump box
make pf-grafana # localhost:3000 → grafana.logging:3000
# Option 3: Add to /etc/hosts (on home network)
192.168.1.160 grafana.riotpiao.homelab.com
```
## Monitoring
**Grafana dashboard:** `svc-nginx-ingress`
**Key metrics:**
- `nginx_requests_total` — total requests
- `nginx_request_duration_seconds` — latency histogram
- `nginx_ingress_upstream_requests_total{status=~"5.."}` — backend errors
- `nginx_ssl_expire_time_seconds` — cert expiry countdown
**Alert on cert expiry:**
```yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: cert-expiry
namespace: ingress-nginx
spec:
groups:
- name: cert-expiry
rules:
- alert: CertificateExpiringSoon
expr: nginx_ssl_expire_time_seconds < 86400 * 14 # < 14 days
annotations:
summary: "Certificate {{ $labels.host }} expires in {{ $value | humanizeDuration }}"
```
## Troubleshooting
**Certificate stuck in "Pending":**
```bash
# Check cert-manager logs
k logs -n cert-manager -f deploy/cert-manager
# Verify ClusterIssuer exists
k get clusterissuer
# Check ACME order (if using LetsEncrypt)
k describe certificate -n myapp-ns myapp-cert
```
**Ingress not exposing service (503 error):**
```bash
# Verify Service exists and has endpoints
k get svc -n myapp-ns
k get endpoints -n myapp-ns myapp-svc
# Check if pods are ready
k get pods -n myapp-ns
# Test pod directly (port-forward)
k port-forward -n myapp-ns pod/myapp-0 8080:8080
curl http://localhost:8080
```
**DNS resolution fails from pod:**
```bash
# Test from pod
k run -it --rm debug --image=busybox:1.28 --restart=Never -- \
nslookup grafana.logging.svc.cluster.local
# If fails, CoreDNS may be unhealthy
k get pods -n kube-system -l k8s-app=kube-dns
k logs -n kube-system -l k8s-app=kube-dns
```
**TLS handshake error (cert not trusted):**
```bash
# Verify TLS cert Secret exists
k get secret -n myapp-ns myapp-tls
# Verify cert is correctly signed by homelab-ca
k get secret -n myapp-ns myapp-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text | grep -A 5 "Issuer:"
# If cert is self-signed (homelab-ca), add to client's trusted roots
# Or bypass cert verification (dev only):
curl -k https://myapp.riotpiao.homelab.com
```
See `/TROUBLESHOOTING.md` for full incident guide.
project-usage/sqs-messaging.md
+193
View File
@@ -0,0 +1,193 @@
# SQS-like Message Queue Service (kmsvc)
**Endpoint:** `https://kmsvc.riotpiao.homelab.com` (REST + gRPC-Gateway)
**Internal:** `kmsvc-management-service.sqs.svc.cluster.local:8080`
**Namespace:** `sqs`
## When to Use
- **Decouple services** — Producer doesn't wait for consumer
- **Async jobs** — Fire-and-forget processing (batch, email, webhooks)
- **FIFO ordering** — Guarantee message order within `MessageGroupId`
- **Durable delivery** — At-least-once (messages in Kafka, replicated 3×)
## Quick Start
**1. Create a queue:**
```bash
kubectl apply -f - <<EOF
apiVersion: kmsvc.io/v1
kind: Queue
metadata:
name: orders
spec:
fifoQueue: false # standard queue
visibilityTimeoutSeconds: 30 # re-deliver if not acked
messageRetentionPeriodSeconds: 345600 # 4 days
partitionsPerShard: 6
maxReceiveCount: 5 # move to DLQ after 5 fails
EOF
```
**2. Send message:**
```bash
curl -X POST https://kmsvc.riotpiao.homelab.com/v1/queues/orders/messages \
-H "Authorization: Bearer $JWT_TOKEN" \
-d '{
"body": "{\"order_id\":123,\"total\":99.99}",
"attributes": {"source":"web","priority":"high"}
}'
```
**3. Receive message:**
```bash
curl "https://kmsvc.riotpiao.homelab.com/v1/queues/orders/messages?max_number_of_messages=10&wait_time_seconds=20" \
-H "Authorization: Bearer $JWT_TOKEN"
# Response:
# {
# "messages": [
# {
# "message_id": "abc-123",
# "receipt_handle": "...",
# "body": "{...}",
# "attributes": {...},
# "receive_count": 1
# }
# ]
# }
```
**4. Acknowledge (delete) message:**
```bash
curl -X DELETE "https://kmsvc.riotpiao.homelab.com/v1/queues/orders/messages/$receipt_handle" \
-H "Authorization: Bearer $JWT_TOKEN"
```
## Configuration
| Key | Value |
|-----|-------|
| Kafka bootstrap | `kmsvc-kafka-bootstrap.sqs.svc.cluster.local:9092` |
| Redis | `kmsvc-redis-master.sqs.svc.cluster.local:6379` |
| Topic naming | `kmsvc.{queueName}.shard-{id}` |
| Replication | 3 replicas, min.insync.replicas=2 |
| Retention | 4 days (configurable per queue) |
## Common Patterns
**Batch processing:**
```bash
for i in {1..100}; do
curl -X POST https://kmsvc.riotpiao.homelab.com/v1/queues/jobs/messages \
-H "Authorization: Bearer $JWT_TOKEN" \
-d "{\"body\":\"task-$i\"}" &
done
wait
```
**FIFO queue (order guaranteed per group):**
```yaml
apiVersion: kmsvc.io/v1
kind: Queue
metadata:
name: checkout-fifo
spec:
fifoQueue: true
visibilityTimeoutSeconds: 60
partitionsPerShard: 1
```
**Dead-letter queue (failed messages):**
```yaml
apiVersion: kmsvc.io/v1
kind: Queue
metadata:
name: orders-dlq
spec:
fifoQueue: false
---
apiVersion: kmsvc.io/v1
kind: Queue
metadata:
name: orders
spec:
fifoQueue: false
maxReceiveCount: 3
deadLetterTargetQueue: orders-dlq # auto-route failures here
```
## Monitoring
**Grafana dashboard:** `svc-kmsvc` (automatically loaded)
**Key metrics:**
- `kmsvc_messages_sent_total` — total sent
- `kmsvc_messages_received_total` — total received
- `kmsvc_queue_depth` — pending messages per queue
- `kmsvc_message_visibility_timeout_seconds` — visibility window
**Redis in-flight tracking:**
```bash
# Connect to Redis
k port-forward -n sqs svc/redis 6379:6379 &
redis-cli
# Check pending messages
KEYS "kmsvc:pending:orders:*"
KEYS "kmsvc:inflight:*" | wc -l
```
## Authentication
**Requires JWT from Authentik:**
```bash
# Get token (device code flow)
talos secrets login
# Use token
export JWT_TOKEN=$(talos get cluster/kmsvc/jwt-token --key jwt-token)
curl -H "Authorization: Bearer $JWT_TOKEN" https://kmsvc.riotpiao.homelab.com/v1/queues
```
## Integration Example
**Story Crater backend consumer:**
```go
// Receive messages
messages, err := kmsvc.ReceiveMessage(ctx, &kmsvc.ReceiveMessageRequest{
QueueName: "story-crater",
MaxNumberOfMessages: 10,
WaitTimeSeconds: 20,
})
// Process
for _, msg := range messages.Messages {
processMessage(msg.Body)
// Acknowledge on success
kmsvc.DeleteMessage(ctx, &kmsvc.DeleteMessageRequest{
QueueName: "story-crater",
ReceiptHandle: msg.ReceiptHandle,
})
}
```
## Troubleshooting
**Queue stuck / high lag:**
```bash
# Check Kafka broker status
k exec -n sqs pod/kmsvc-kafka-0 -- kafka-broker-api-versions.sh --bootstrap-server localhost:9092
# Inspect queue topics
k exec -n sqs pod/kmsvc-kafka-0 -- kafka-topics.sh --bootstrap-server localhost:9092 --list | grep orders
```
**Messages not being consumed:**
- Check `maxReceiveCount` (may be routing to DLQ)
- Verify consumer has `ReceiveMessage` permission (JWT scope)
- Check Redis: `KEYS "kmsvc:fifo_lock:orders:*"` (may be blocked by visibility timeout)
See `/TROUBLESHOOTING.md` for full incident guide.
project-usage/temporal-workflows.md
+227
View File
@@ -0,0 +1,227 @@
# Temporal Workflow Orchestration
**Server:** `temporal.temporal.svc.cluster.local:7233` (cluster-internal)
**Web UI:** `kubectl port-forward -n temporal svc/temporal-web 8088:8088`
**Namespace:** `temporal`
## When to Use
- **Long-running operations** — Tasks that take minutes/hours (send email, process batch, retry with backoff)
- **State machines** — Multi-step workflows with decision logic
- **Retries & timeouts** — Built-in exponential backoff, automatic retry
- **Audit trail** — Full history of workflow executions (why it happened, when, by whom)
## Quick Start
**1. Access Temporal Web UI:**
```bash
k port-forward -n temporal svc/temporal-web 8088:8088
# http://localhost:8088
```
**2. Define workflow (Go example):**
```go
package workflows
import (
"time"
"go.temporal.io/sdk/workflow"
"go.temporal.io/sdk/activity"
)
type Inputs struct {
OrderID string
Amount float64
}
// Workflow definition
func OrderProcessing(ctx workflow.Context, input Inputs) (string, error) {
// Step 1: Charge payment
chargeResult := ""
err := workflow.ExecuteActivity(
ctx,
ChargePayment,
input.OrderID,
input.Amount,
).Get(ctx, &chargeResult)
if err != nil {
return "", err
}
// Step 2: Send confirmation email (retry 3x on failure)
emailResult := ""
opts := workflow.ActivityOptions{
StartToCloseTimeout: time.Minute,
RetryPolicy: &temporal.RetryPolicy{
InitialInterval: time.Second,
BackoffCoefficient: 2,
MaximumAttempts: 3,
},
}
ctx = workflow.WithActivityOptions(ctx, opts)
err = workflow.ExecuteActivity(ctx, SendConfirmationEmail, input.OrderID).Get(ctx, &emailResult)
if err != nil {
return "", err
}
return "order_processed", nil
}
// Activity: payment processing
func ChargePayment(ctx context.Context, orderID string, amount float64) (string, error) {
// Call payment gateway
return "payment_successful", nil
}
// Activity: email notification
func SendConfirmationEmail(ctx context.Context, orderID string) (string, error) {
// Send email
return "email_sent", nil
}
```
**3. Register & start workflow:**
```go
import "go.temporal.io/sdk/client"
client, _ := client.Dial(client.Options{
HostPort: "temporal.temporal.svc.cluster.local:7233",
})
// Start workflow execution
run, _ := client.ExecuteWorkflow(ctx, client.StartWorkflowOptions{
ID: "order-123",
TaskQueue: "orders",
}, OrderProcessing, Inputs{OrderID: "123", Amount: 99.99})
// Wait for result
var result string
run.Get(ctx, &result)
```
## Configuration
| Key | Value |
|-----|-------|
| Server | `temporal.temporal.svc.cluster.local:7233` |
| Web UI | `localhost:8088` (via port-forward) |
| Database | PostgreSQL (managed by helmfile) |
| Task queue | `default`, `orders`, `emails` (custom per app) |
| Retention | 30 days (configurable) |
## Common Patterns
**Retry with exponential backoff:**
```go
opts := workflow.ActivityOptions{
StartToCloseTimeout: 5 * time.Minute,
RetryPolicy: &temporal.RetryPolicy{
InitialInterval: time.Second,
BackoffCoefficient: 2.0, // double wait time each retry
MaximumInterval: time.Minute, // cap at 1 min between retries
MaximumAttempts: 5, // give up after 5 tries
},
}
ctx = workflow.WithActivityOptions(ctx, opts)
```
**Wait for signal (user approval):**
```go
// Workflow waits for approval signal
approval := ""
workflow.GetSignalChannel(ctx, "approval").Receive(ctx, &approval)
if approval == "approved" {
// Continue workflow
} else {
return "", errors.New("request denied")
}
```
**Parallel activities:**
```go
// Execute email & SMS in parallel
emailFuture := workflow.ExecuteActivity(ctx, SendEmail, userID)
smsFuture := workflow.ExecuteActivity(ctx, SendSMS, userID)
// Wait for both to complete
emailFuture.Get(ctx, nil)
smsFuture.Get(ctx, nil)
```
**Scheduled workflow (cron):**
```go
run, _ := client.ExecuteWorkflow(ctx, client.StartWorkflowOptions{
ID: "daily-report",
CronSchedule: "0 9 * * MON-FRI", // 9 AM weekdays
WorkflowTaskTimeout: time.Hour,
}, GenerateDailyReport, nil)
```
## Monitoring
**Web UI:**
- List workflows: http://localhost:8088/namespaces/default/workflows
- View execution history: Click workflow ID
- See activity logs, errors, retry attempts
**Grafana dashboard:** `svc-temporal` (auto-configured)
**Key metrics:**
- `temporal_workflow_execution_duration_seconds` — workflow time
- `temporal_activity_execution_duration_seconds` — activity time
- `temporal_activity_execution_failed_total` — failed activities
## Integration with Story Crater
**Example: Process message via Temporal:**
```go
// In message handler
client, _ := temporal.Dial(/* ... */)
run, _ := client.ExecuteWorkflow(ctx, client.StartWorkflowOptions{
ID: fmt.Sprintf("msg-%s", messageID),
TaskQueue: "story-crater",
}, ProcessMessageWorkflow, Message{
ID: messageID,
Body: body,
Source: "kafka-queue",
})
// Non-blocking: workflow runs independently
// Check status later
```
## Troubleshooting
**Workflow stuck:**
```bash
# Check Temporal server health
k get pods -n temporal
# View workflow execution history (via Web UI or CLI)
tctl workflow show --workflow-id order-123
# Terminate stuck workflow
tctl workflow terminate --workflow-id order-123
```
**Activity retrying endlessly:**
```go
// Add max attempts or timeout
RetryPolicy: &temporal.RetryPolicy{
MaximumAttempts: 5, // must have this!
}
```
**PostgreSQL connection fails:**
```bash
# Check temporal pod logs
k logs -n temporal pod/temporal-0
# Verify database is running
k get pods -n ddb
```
See `/TROUBLESHOOTING.md` for full incident guide.
project-usage/vault-secrets.md
+211
View File
@@ -0,0 +1,211 @@
# Vault: Secret Management & JWT Auth
**Vault:** `https://vault.riotpiao.homelab.com`
**Internal:** `vault.iam.svc.cluster.local:8200`
**Namespace:** `iam`
## When to Use
- **Store secrets** — Database passwords, API keys, TLS certs
- **Rotate credentials** — Auto-rotate, track rotation history
- **JWT validation** — Verify tokens from Authentik, no external call needed
- **Audit trail** — Who accessed what secret, when
## Quick Start
**1. Login to Vault:**
```bash
# Browser: https://vault.riotpiao.homelab.com
# Auth method: OIDC → "Sign in with Authentik" (federated)
# Or: Device code → talos secrets login (CLI)
# Via CLI (device code flow)
talos secrets login
# → Opens browser, approve device code
# → Token cached in ~/.talos/vault
```
**2. Store a secret:**
```bash
# Field name = variable name (SCREAMING_SNAKE_CASE)
talos put cluster/ANTHROPIC_API_KEY ANTHROPIC_API_KEY="sk-..."
talos put cluster/STORY_CRATER_DB_PASS STORY_CRATER_DB_PASS="dbpass123"
```
**3. Retrieve a secret:**
```bash
# Always use --key flag
talos get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
# → sk-...
# Full secret as JSON
talos get cluster/ANTHROPIC_API_KEY --json
```
**4. Load into shell (helmfile, scripts):**
```bash
vsource .env
# → Expands empty vars from Vault (ANTHROPIC_API_KEY=)
# → Passes hardcoded vars as-is (DEBUG=true)
helmfile diff
helmfile apply
```
## Vault Paths (KV v2)
**Naming convention:** `cluster/<VARIABLE_NAME>`
```
cluster/
├── ANTHROPIC_API_KEY # third-party API
├── STORY_CRATER_DB_PASS # database password
├── MINIO_ROOT_PASSWORD # MinIO credentials
├── AUTHENTIK_BOOTSTRAP_PASSWORD # initial admin pass
├── iam/
│ ├── federation # Authentik OIDC app config
│ ├── roles/admin # RBAC role definitions
│ ├── services/grafana # OAuth2 client info
│ ├── agents/ci-bot # machine credentials
│ └── bindings/alice # user→role mappings
└── kubernetes/
├── ingress-tls # TLS cert private keys
└── pull-secrets # Docker registry creds
```
## Common Patterns
**Store generated secret immediately (keeps it out of shell history):**
```bash
# Generate & store in one command
talos put cluster/GRAFANA_OIDC_CLIENT_SECRET \
GRAFANA_OIDC_CLIENT_SECRET="$(openssl rand -hex 32)"
```
**Use in Kubernetes Secret:**
```bash
# Create Secret using Vault secret
kubectl create secret generic grafana-oidc \
--from-literal=client-secret="$(talos get cluster/GRAFANA_OIDC_CLIENT_SECRET --key GRAFANA_OIDC_CLIENT_SECRET)" \
-n logging
```
**Rotate credentials safely:**
```bash
# 1. Generate new secret
NEW_PASS=$(openssl rand -base64 24)
# 2. Store in Vault
talos put cluster/OLD_DB_PASS OLD_DB_PASS="$NEW_PASS"
# 3. Update database user
psql -h ddb-cluster-rw.ddb.svc.cluster.local -U postgres \
-c "ALTER USER story_crater PASSWORD '$NEW_PASS';"
# 4. Update Kubernetes Secret
kubectl patch secret db-credentials -n story-crater-backend \
--type merge -p "{\"stringData\":{\"password\":\"$NEW_PASS\"}}"
# 5. Restart pods to pick up new Secret
k rollout restart -n story-crater-backend deployment/app
```
**JWT token from CLI:**
```bash
# After device code login
talos secrets login
# Token is cached and auto-renewed
# Use for API calls
VAULT_TOKEN=$(cat ~/.talos/vault)
curl -H "X-Vault-Token: $VAULT_TOKEN" \
https://vault.iam.svc.cluster.local:8200/v1/secret/data/cluster/ANTHROPIC_API_KEY
```
## Monitoring
**Vault status:**
```bash
# Check if sealed
talos status vault
# If sealed (disaster recovery):
# See /TROUBLESHOOTING.md § Vault Sealed
```
**Audit log (who accessed what):**
```bash
# Enable audit logging (already enabled by helmfile)
# Logs stored in Loki under vault namespace
# View recent access
talos audit log --limit 50
# Export for compliance
talos audit export --format json > vault-audit.json
```
## Security Rules
✅ **DO:**
- Store ALL secrets in Vault (never .env in git)
- Use field name = variable name
- Always use `--key` flag when retrieving
- Rotate credentials on schedule (quarterly)
- Review audit log for anomalies
❌ **DON'T:**
- Commit `.env` with real secrets (only `.env.example`)
- Use positional arguments (must use `--key`)
- Share Vault root token
- Store secrets in pod env (use Secret volumes)
## Troubleshooting
**Cannot login (Authentik OIDC fails):**
```bash
# Check Authentik is running
k get pods -n iam -l app=authentik
# Verify OIDC app in Authentik console
# Applications → Vault OIDC → Check client ID/secret
# Restart Vault to re-sync OIDC config
k rollout restart -n iam statefulset/vault
```
**Vault is sealed:**
```bash
# Check status
talos status vault
# If sealed, use unseal keys (stored in MinIO backup)
# See /TROUBLESHOOTING.md § Vault Sealed for recovery steps
```
**Secret not found:**
```bash
# Verify path exists
talos list cluster
# Check secret name (case-sensitive)
talos get cluster/anthropic_api_key --key anthropic_api_key # won't work
talos get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY # correct
```
**vsource not expanding secrets:**
```bash
# Verify .env has empty value
grep ANTHROPIC_API_KEY .env
# → Should be: ANTHROPIC_API_KEY= (empty, not a value)
# Verify Vault is accessible
talos get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
# → Should return secret
# Run vsource explicitly
vsource .env
echo $ANTHROPIC_API_KEY # should be populated
```
See `/TROUBLESHOOTING.md` for full incident guide.