k8s/services: add ingress networking portainer llm and project guides
- Nginx ingress + TLS termination (homelab-ca) - Portainer container UI - CoreDNS internal DNS rewrites - DuckDNS DDNS updater - Ollama LLM inference - 8 project-usage guides (team reference)
This commit is contained in:
Story Crater Bot
@@ -0,0 +1,183 @@
|
||||
# Authentik Federated OIDC & SSO
|
||||
|
||||
**Provider:** `https://authentik.riotpiao.homelab.com`
|
||||
**OIDC Issuer:** `https://authentik.riotpiao.homelab.com/application/o/talos-federation/`
|
||||
**Namespace:** `iam`
|
||||
|
||||
## When to Use
|
||||
|
||||
- **Federated login** — Single sign-on for Grafana, MinIO, Forgejo, Argo CD
|
||||
- **User groups** — RBAC via group membership (admins, devops, read-only)
|
||||
- **JWT tokens** — Authenticate CLI tools, API clients
|
||||
- **SSO for custom apps** — OAuth2/OIDC redirect flow
|
||||
|
||||
## Quick Start
|
||||
|
||||
**1. Login to Authentik console:**
|
||||
```bash
|
||||
# Browser: https://authentik.riotpiao.homelab.com
|
||||
# Default user: akadmin
|
||||
# Password: AUTHENTIK_BOOTSTRAP_PASSWORD (from .env)
|
||||
|
||||
# Or via OIDC (after initial setup)
|
||||
# Click "Sign in with talos-federation"
|
||||
```
|
||||
|
||||
**2. Create user:**
|
||||
```
|
||||
Authentik console → Users → Create
|
||||
- Username: alice
|
||||
- Email: [email protected]
|
||||
- Group: homelab-devs (or homelab-admins)
|
||||
```
|
||||
|
||||
**3. User logs into Grafana:**
|
||||
```
|
||||
https://grafana.riotpiao.homelab.com
|
||||
→ Sign in with Authentik (auto-redirects to OIDC provider)
|
||||
→ Approve access
|
||||
→ Logged in as alice (group determines role: Admin or Viewer)
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
| Key | Value |
|
||||
|-----|-------|
|
||||
| OIDC provider | `talos-federation` (federated) |
|
||||
| OIDC issuer | `https://authentik.riotpiao.homelab.com/application/o/talos-federation/` |
|
||||
| JWKS endpoint | `https://authentik.riotpiao.homelab.com/application/o/talos-federation/.well-known/openid-configuration` |
|
||||
| Database | PostgreSQL (ddb namespace, authentik user) |
|
||||
| Backups | WAL archived to MinIO |
|
||||
|
||||
## Common Patterns
|
||||
|
||||
**Grafana OIDC login:**
|
||||
```yaml
|
||||
# k8s/logging/grafana-values.yaml
|
||||
grafana:
|
||||
auth.generic_oauth:
|
||||
enabled: true
|
||||
name: Authentik
|
||||
client_id: grafana
|
||||
client_secret: $GRAFANA_OIDC_CLIENT_SECRET # from Vault
|
||||
auth_url: https://authentik.riotpiao.homelab.com/application/o/authorize/
|
||||
token_url: https://authentik.riotpiao.homelab.com/application/o/token/
|
||||
api_url: https://authentik.riotpiao.homelab.com/application/o/userinfo/
|
||||
scopes: openid profile email groups
|
||||
use_pkce: true
|
||||
```
|
||||
|
||||
**MinIO OIDC login:**
|
||||
```yaml
|
||||
# k8s/storage/minio-values.yaml
|
||||
minio:
|
||||
identity_oauth:
|
||||
provider: authentik
|
||||
client_id: minio
|
||||
client_secret: $MINIO_OIDC_CLIENT_SECRET
|
||||
redirect_uri: https://minio.riotpiao.homelab.com/oauth_callback
|
||||
config_url: https://authentik.riotpiao.homelab.com/application/o/talos-federation/.well-known/openid-configuration
|
||||
policy_mappings:
|
||||
- group: homelab-admins → consoleAdmin
|
||||
- group: homelab-devops → readwrite
|
||||
```
|
||||
|
||||
**CLI device code flow (talos-cli):**
|
||||
```bash
|
||||
# Get JWT token (no kubeconfig needed)
|
||||
talos secrets login
|
||||
# → Opens browser, approve device code
|
||||
# → Token cached in ~/.talos/token
|
||||
|
||||
# Use token to access Vault
|
||||
talos get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
|
||||
# → Vault validates JWT from Authentik
|
||||
# → Returns secret
|
||||
```
|
||||
|
||||
**Custom app OIDC redirect:**
|
||||
```go
|
||||
import "github.com/coreos/go-oidc/v3/oidc"
|
||||
|
||||
provider, _ := oidc.NewProvider(ctx, "https://authentik.riotpiao.homelab.com/application/o/talos-federation/")
|
||||
|
||||
verifier := provider.Verifier(&oidc.Config{ClientID: "my-app"})
|
||||
|
||||
// After OAuth2 redirect & token exchange:
|
||||
idToken, _ := verifier.Verify(ctx, rawIDToken)
|
||||
|
||||
// Extract claims
|
||||
var claims struct {
|
||||
Email string `json:"email"`
|
||||
Groups []string `json:"groups"`
|
||||
}
|
||||
idToken.Claims(&claims)
|
||||
```
|
||||
|
||||
## Group-Based RBAC
|
||||
|
||||
**Default groups:**
|
||||
- `homelab-admins` — Full cluster access (Grafana Admin, MinIO admin, Argo CD admin, Vault admin)
|
||||
- `homelab-devops` — Deploy & monitor (Grafana Editor, MinIO readwrite, Argo CD user)
|
||||
- `homelab-viewers` — Read-only (Grafana Viewer, MinIO readonly)
|
||||
|
||||
**Assign user to group:**
|
||||
```
|
||||
Authentik console → Users → alice → Edit
|
||||
→ Groups → Add "homelab-devops"
|
||||
→ Save
|
||||
```
|
||||
|
||||
**Custom group-to-role mapping:**
|
||||
```yaml
|
||||
# Per-service (see cicd-workflow.md, monitoring-metrics.md for examples)
|
||||
# Grafana: auth.generic_oauth.role_attribute_path = contains(groups[*], 'homelab-admins') && 'Admin' || 'Viewer'
|
||||
# MinIO: policy_mappings (see above)
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
**Authentik dashboard:** https://authentik.riotpiao.homelab.com/api/v3/admin/dashboards
|
||||
|
||||
**Key metrics:**
|
||||
- Login attempts (success/failure)
|
||||
- Active sessions
|
||||
- Token issuance rate
|
||||
- Provider sync status
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Users can't login (redirect loop):**
|
||||
```bash
|
||||
# Check redirect URI matches
|
||||
# Authentik console → Applications → grafana → Edit
|
||||
# Verify Redirect URI = https://grafana.riotpiao.homelab.com/login/generic_oauth
|
||||
|
||||
# Check OIDC provider is running
|
||||
k get pods -n iam -l app=authentik
|
||||
```
|
||||
|
||||
**JWT token expired:**
|
||||
```bash
|
||||
# CLI tokens have 24h expiry
|
||||
# Re-authenticate
|
||||
talos secrets login
|
||||
```
|
||||
|
||||
**Groups not syncing:**
|
||||
```bash
|
||||
# Check group attribute in OIDC config
|
||||
# Authentik console → Applications → <app> → OIDC Configuration
|
||||
# groups_attribute = "groups" (or custom claim name)
|
||||
```
|
||||
|
||||
**Vault can't validate JWT:**
|
||||
```bash
|
||||
# Verify JWKS endpoint is accessible
|
||||
curl https://authentik.riotpiao.homelab.com/application/o/talos-federation/.well-known/openid-configuration
|
||||
|
||||
# Restart Vault to refresh JWKS cache
|
||||
k rollout restart -n iam deployment/vault
|
||||
```
|
||||
|
||||
See `/TROUBLESHOOTING.md` for full incident guide.
|
||||
@@ -0,0 +1,222 @@
|
||||
# CI/CD Pipeline (Forgejo + Argo CD)
|
||||
|
||||
**Git Forge:** `https://forgejo.riotpiao.homelab.com`
|
||||
**Deployments:** `https://argocd.riotpiao.homelab.com` (or `kubectl port-forward`)
|
||||
**Namespaces:** `cicd`, `forge`
|
||||
|
||||
## When to Use
|
||||
|
||||
- **Build & test** — Forgejo Actions CI (GitHub Actions syntax)
|
||||
- **Image push** — Build OCI images, push to Forgejo registry
|
||||
- **GitOps deployment** — Argo CD syncs deploy repo to cluster
|
||||
- **Secrets in CI** — ci-bot JWT tokens, never kubeconfig
|
||||
|
||||
## Quick Start
|
||||
|
||||
**1. Clone a repo from Forgejo:**
|
||||
```bash
|
||||
git clone https://forgejo.riotpiao.homelab.com/rock/source.git
|
||||
cd source
|
||||
```
|
||||
|
||||
**2. Create workflow:**
|
||||
```bash
|
||||
mkdir -p .forgejo/workflows
|
||||
cat > .forgejo/workflows/ci.yml <<EOF
|
||||
name: CI
|
||||
on: [push]
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- run: npm test
|
||||
- run: docker build -t myapp:latest .
|
||||
- run: |
|
||||
docker login forgejo.riotpiao.homelab.com \
|
||||
-u ci-bot \
|
||||
-p ${{ secrets.CI_BOT_TOKEN }}
|
||||
docker push forgejo.riotpiao.homelab.com/rock/myapp:latest
|
||||
EOF
|
||||
|
||||
git add .forgejo/workflows/ci.yml
|
||||
git commit -m "ci: add build workflow"
|
||||
git push
|
||||
```
|
||||
|
||||
**3. Trigger deployment:**
|
||||
```bash
|
||||
# Update deployment repo (rock/deploy)
|
||||
git clone https://forgejo.riotpiao.homelab.com/rock/deploy.git
|
||||
cd deploy
|
||||
|
||||
# Update image tag
|
||||
sed -i 's|forgejo.riotpiao.homelab.com/rock/myapp:.*|forgejo.riotpiao.homelab.com/rock/myapp:abc123|' k8s/deployment.yaml
|
||||
|
||||
git add k8s/deployment.yaml
|
||||
git commit -m "deploy: bump myapp to abc123"
|
||||
git push
|
||||
```
|
||||
|
||||
**4. Argo CD auto-syncs:**
|
||||
```bash
|
||||
# Watch deployment
|
||||
k rollout status -n <app-namespace> deployment/<app-name>
|
||||
|
||||
# Or check Argo CD UI
|
||||
kubectl port-forward -n argocd svc/argocd-server 8443:443
|
||||
# https://localhost:8443 (login via Authentik)
|
||||
```
|
||||
|
||||
## Workflow Syntax (GitHub Actions)
|
||||
|
||||
**Basic structure:**
|
||||
```yaml
|
||||
name: CI
|
||||
on:
|
||||
push:
|
||||
branches: [main, develop]
|
||||
pull_request:
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- run: npm ci
|
||||
- run: npm test
|
||||
|
||||
build:
|
||||
needs: test # wait for test job
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- run: docker build -t myapp:${{ github.sha }} .
|
||||
- run: docker push forgejo.riotpiao.homelab.com/rock/myapp:${{ github.sha }}
|
||||
```
|
||||
|
||||
**Available variables:**
|
||||
```bash
|
||||
${{ github.sha }} # commit hash
|
||||
${{ github.ref_name }} # branch name
|
||||
${{ github.run_number }} # build #
|
||||
${{ secrets.CI_BOT_TOKEN }} # injected from Forgejo
|
||||
```
|
||||
|
||||
## Secrets & Authentication
|
||||
|
||||
**ci-bot JWT token (auto-injected):**
|
||||
```yaml
|
||||
- run: |
|
||||
echo "${{ secrets.CI_BOT_TOKEN }}" | docker login \
|
||||
forgejo.riotpiao.homelab.com \
|
||||
-u ci-bot \
|
||||
--password-stdin
|
||||
docker push forgejo.riotpiao.homelab.com/rock/myapp:latest
|
||||
```
|
||||
|
||||
**API token (for pushing commits):**
|
||||
```yaml
|
||||
- run: |
|
||||
git config user.name "ci-bot"
|
||||
git config user.email "ci-bot@homelab"
|
||||
git commit --allow-empty -m "bump: version"
|
||||
git push https://ci-bot:${{ secrets.CI_BOT_TOKEN }}@forgejo.riotpiao.homelab.com/rock/deploy.git main
|
||||
```
|
||||
|
||||
**Vault secrets (via talos CLI):**
|
||||
```bash
|
||||
# Not available in CI runner — use Argo CD post-sync hooks instead
|
||||
# Or inject via init container before workflow runs
|
||||
```
|
||||
|
||||
## Argo CD GitOps
|
||||
|
||||
**Create app (one-time):**
|
||||
```bash
|
||||
argocd app create story-crater \
|
||||
--repo https://forgejo.riotpiao.homelab.com/rock/deploy.git \
|
||||
--path k8s/ \
|
||||
--dest-server https://kubernetes.default.svc \
|
||||
--dest-namespace story-crater-backend \
|
||||
--sync-policy automated
|
||||
```
|
||||
|
||||
**Monitor sync:**
|
||||
```bash
|
||||
# CLI
|
||||
argocd app get story-crater
|
||||
argocd app logs story-crater
|
||||
|
||||
# UI: https://argocd.riotpiao.homelab.com
|
||||
# Login: Authentik SSO (homelab-admins group only)
|
||||
```
|
||||
|
||||
**Manual sync:**
|
||||
```bash
|
||||
argocd app sync story-crater
|
||||
argocd app wait story-crater
|
||||
```
|
||||
|
||||
## Security Rules
|
||||
|
||||
✅ **DO:**
|
||||
- Store credentials in Forgejo Secrets (auto-injected)
|
||||
- Use ci-bot JWT for image push only
|
||||
- Commit to deploy repo (triggers Argo CD)
|
||||
- Enable branch protection (require CI pass)
|
||||
|
||||
❌ **DON'T:**
|
||||
- Put kubeconfig in CI (Argo CD bridges gap)
|
||||
- Commit secrets to source repo
|
||||
- Use admin-bot in CI workflows (over-privileged)
|
||||
- Push images directly to cluster (use Argo CD)
|
||||
|
||||
## Monitoring
|
||||
|
||||
**Grafana dashboard:** `svc-forgejo`, `svc-argocd`
|
||||
|
||||
**Key metrics:**
|
||||
- `forgejo_workflows_running` — active workflows
|
||||
- `argocd_app_sync_duration_seconds` — deployment time
|
||||
- `argocd_app_info{sync_status="OutOfSync"}` — drift detection
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Workflow fails silently:**
|
||||
```bash
|
||||
# Check runner logs
|
||||
k logs -n cicd -f deploy/forgejo-runner
|
||||
|
||||
# Check if runner pod is healthy
|
||||
k get pods -n cicd -l app=forgejo-runner
|
||||
```
|
||||
|
||||
**Image push fails (401 Unauthorized):**
|
||||
```bash
|
||||
# Verify ci-bot token in Forgejo
|
||||
# Settings → Applications → ci-bot → check scopes (package:write)
|
||||
|
||||
# Or re-create token
|
||||
talos put cluster/iam/agents/ci-bot-token TOKEN="$(openssl rand -hex 32)"
|
||||
```
|
||||
|
||||
**Argo CD out-of-sync:**
|
||||
```bash
|
||||
# Check deploy repo changes
|
||||
argocd app diff story-crater
|
||||
|
||||
# Manual sync
|
||||
argocd app sync story-crater --prune
|
||||
```
|
||||
|
||||
**Webhook not triggering:**
|
||||
```bash
|
||||
# Verify Forgejo webhook config
|
||||
# Repo Settings → Webhooks → Check delivery logs
|
||||
|
||||
# Or manually trigger
|
||||
argocd app sync story-crater
|
||||
```
|
||||
|
||||
See `/TROUBLESHOOTING.md` for full incident guide.
|
||||
@@ -0,0 +1,188 @@
|
||||
# CloudNativePG PostgreSQL Database
|
||||
|
||||
**Host:** `ddb-cluster-rw.ddb.svc.cluster.local` (read-write)
|
||||
**Read replica:** `ddb-cluster-ro.ddb.svc.cluster.local` (read-only)
|
||||
**Port:** `5432`
|
||||
**Namespace:** `ddb`
|
||||
|
||||
## When to Use
|
||||
|
||||
- **Multi-replica HA** — 3 replicas, automatic failover
|
||||
- **pgvector extension** — Vector similarity search (LLM embeddings)
|
||||
- **Transactional data** — Authentik, Story Crater backend, custom apps
|
||||
- **Declarative backups** — Automated WAL archiving to MinIO
|
||||
|
||||
## Quick Start
|
||||
|
||||
**1. Connect from pod:**
|
||||
```bash
|
||||
# Inside a pod (inject secret mount)
|
||||
psql -h ddb-cluster-rw.ddb.svc.cluster.local \
|
||||
-U story_crater \
|
||||
-d story_crater \
|
||||
-W # prompt for password (from Secret)
|
||||
```
|
||||
|
||||
**2. Create database & user (one-time):**
|
||||
```bash
|
||||
# Already done by helmfile postsync hook
|
||||
# But if needed manually:
|
||||
|
||||
psql -h ddb-cluster-rw.ddb.svc.cluster.local \
|
||||
-U postgres \
|
||||
-c "CREATE DATABASE myapp OWNER postgres;"
|
||||
|
||||
psql -h ddb-cluster-rw.ddb.svc.cluster.local \
|
||||
-U postgres \
|
||||
-d myapp \
|
||||
-c "CREATE USER myapp_user WITH PASSWORD 'secret';"
|
||||
|
||||
psql -h ddb-cluster-rw.ddb.svc.cluster.local \
|
||||
-U postgres \
|
||||
-d myapp \
|
||||
-c "GRANT ALL PRIVILEGES ON DATABASE myapp TO myapp_user;"
|
||||
```
|
||||
|
||||
**3. Enable pgvector:**
|
||||
```bash
|
||||
psql -h ddb-cluster-rw.ddb.svc.cluster.local \
|
||||
-U postgres \
|
||||
-d myapp \
|
||||
-c "CREATE EXTENSION IF NOT EXISTS vector;"
|
||||
```
|
||||
|
||||
**4. Create table with embeddings:**
|
||||
```sql
|
||||
CREATE TABLE documents (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
content TEXT,
|
||||
embedding vector(1536), -- OpenAI embeddings
|
||||
created_at TIMESTAMP DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX ON documents USING IVFFLAT (embedding vector_cosine_ops);
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
| Key | Value |
|
||||
|-----|-------|
|
||||
| Host (RW) | `ddb-cluster-rw.ddb.svc.cluster.local` |
|
||||
| Host (RO) | `ddb-cluster-ro.ddb.svc.cluster.local` |
|
||||
| Port | 5432 |
|
||||
| Replicas | 3 (automatic failover) |
|
||||
| Extensions | pgvector (LLM embeddings), uuid-ossp |
|
||||
| Backups | WAL archiving to MinIO (continuous) |
|
||||
| Retention | 30 days |
|
||||
|
||||
## Common Patterns
|
||||
|
||||
**Connection pooling (from app):**
|
||||
```go
|
||||
import "github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
config, _ := pgxpool.ParseConfig("postgres://user:[email protected]:5432/myapp")
|
||||
config.MaxConns = 25
|
||||
config.MinConns = 5
|
||||
pool, _ := pgxpool.NewWithConfig(ctx, config)
|
||||
|
||||
// Use pool
|
||||
row := pool.QueryRow(ctx, "SELECT COUNT(*) FROM users")
|
||||
```
|
||||
|
||||
**Read from replica (analytics):**
|
||||
```go
|
||||
// Offload SELECT queries to read replica
|
||||
pool.QueryRow(ctx, "SELECT * FROM documents LIMIT 1") // auto-routes to RO if available
|
||||
|
||||
// Writes always go to RW
|
||||
pool.Exec(ctx, "INSERT INTO documents ...")
|
||||
```
|
||||
|
||||
**Vector similarity search:**
|
||||
```sql
|
||||
SELECT id, content, embedding <-> $1 AS distance
|
||||
FROM documents
|
||||
ORDER BY embedding <-> $1
|
||||
LIMIT 10;
|
||||
-- $1 = query embedding (e.g., from OpenAI API)
|
||||
```
|
||||
|
||||
**Backup & restore:**
|
||||
```bash
|
||||
# Backups are automatic (WAL to MinIO)
|
||||
# To restore from backup:
|
||||
# 1. Check MinIO s3://postgresql-backups/
|
||||
# 2. Use PostgreSQL PITR (point-in-time recovery)
|
||||
# 3. Contact SRE for restore procedure
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
**Grafana dashboard:** `svc-postgresql` (auto-configured)
|
||||
|
||||
**Key metrics:**
|
||||
- `pg_stat_activity_connections` — active connections
|
||||
- `pg_stat_database_blks_read` — disk I/O
|
||||
- `pg_replication_lag_seconds` — replica lag (goal: < 1s)
|
||||
|
||||
**CLI health check:**
|
||||
```bash
|
||||
# Check replication status
|
||||
kubectl exec -n ddb pod/ddb-cluster-1 -- \
|
||||
psql -U postgres -c "SELECT slot_name, restart_lsn FROM pg_replication_slots;"
|
||||
|
||||
# Check replica lag
|
||||
kubectl exec -n ddb pod/ddb-cluster-2 -- \
|
||||
psql -U postgres -c "SELECT now() - pg_last_xact_replay_timestamp() AS lag;"
|
||||
```
|
||||
|
||||
## Secrets & Credentials
|
||||
|
||||
**All user passwords stored in Vault:**
|
||||
```bash
|
||||
# Read password
|
||||
talos get cluster/STORY_CRATER_PG_PASSWORD --key STORY_CRATER_PG_PASSWORD
|
||||
|
||||
# Inject into pod (auto via Secret volume)
|
||||
# Mount: /run/secrets/db-password
|
||||
```
|
||||
|
||||
**Connection string from env:**
|
||||
```bash
|
||||
POSTGRES_CONNECTION="postgres://story_crater:${STORY_CRATER_PG_PASSWORD}@ddb-cluster-rw.ddb.svc.cluster.local:5432/story_crater"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Cannot connect (connection refused):**
|
||||
```bash
|
||||
# Verify cluster is running
|
||||
k get pods -n ddb
|
||||
|
||||
# Check Service DNS
|
||||
k exec -it pod/debug-pod -- nslookup ddb-cluster-rw.ddb.svc.cluster.local
|
||||
|
||||
# Verify Secret has password
|
||||
k get secret -n ddb ddb-cluster-superuser -o jsonpath='{.data.password}' | base64 -d
|
||||
```
|
||||
|
||||
**Replica lag is high (> 10s):**
|
||||
```bash
|
||||
# Check replica pod CPU/memory
|
||||
k top pod -n ddb
|
||||
|
||||
# Scale down other workloads if cluster is overloaded
|
||||
# Or scale up database resources (helmfile.yaml.gotmpl)
|
||||
```
|
||||
|
||||
**pgvector queries slow:**
|
||||
```sql
|
||||
-- Ensure index exists
|
||||
SELECT * FROM pg_indexes WHERE tablename = 'documents' AND indexname LIKE '%embedding%';
|
||||
|
||||
-- Re-index if missing
|
||||
CREATE INDEX ON documents USING IVFFLAT (embedding vector_cosine_ops);
|
||||
```
|
||||
|
||||
See `/TROUBLESHOOTING.md` for full incident guide.
|
||||
@@ -0,0 +1,162 @@
|
||||
# MinIO S3-Compatible Object Storage
|
||||
|
||||
**Endpoint:** `https://minio.riotpiao.homelab.com` (console)
|
||||
**API:** `minio.storage.svc.cluster.local:9000` (cluster-internal)
|
||||
**Namespace:** `storage`
|
||||
|
||||
## When to Use
|
||||
|
||||
- **File uploads** — Images, documents, backups
|
||||
- **Log backend** — Loki chunks storage
|
||||
- **Vault unsealing** — Store unseal keys
|
||||
- **CI/CD artifacts** — Build outputs, Docker layers cache
|
||||
|
||||
## Quick Start
|
||||
|
||||
**1. Access MinIO console:**
|
||||
```bash
|
||||
# Via browser: https://minio.riotpiao.homelab.com
|
||||
# Credentials: MINIO_ROOT_USER / MINIO_ROOT_PASSWORD (from .env)
|
||||
|
||||
# Or port-forward
|
||||
make pf-minio # localhost:9001
|
||||
```
|
||||
|
||||
**2. Create bucket:**
|
||||
```bash
|
||||
# Via AWS CLI
|
||||
export AWS_ACCESS_KEY_ID=$MINIO_ROOT_USER
|
||||
export AWS_SECRET_ACCESS_KEY=$MINIO_ROOT_PASSWORD
|
||||
|
||||
aws s3 mb s3://my-bucket \
|
||||
--endpoint-url https://minio.riotpiao.homelab.com \
|
||||
--region homelab
|
||||
|
||||
# Or via console UI: Click "Create Bucket"
|
||||
```
|
||||
|
||||
**3. Upload file:**
|
||||
```bash
|
||||
aws s3 cp /path/to/file.txt s3://my-bucket/ \
|
||||
--endpoint-url https://minio.storage.svc.cluster.local:9000 \
|
||||
--use-path-style
|
||||
```
|
||||
|
||||
**4. List buckets:**
|
||||
```bash
|
||||
aws s3 ls --endpoint-url https://minio.storage.svc.cluster.local:9000
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
| Key | Value |
|
||||
|-----|-------|
|
||||
| Access key | `MINIO_ROOT_USER` (from .env) |
|
||||
| Secret key | `MINIO_ROOT_PASSWORD` (from .env) |
|
||||
| Cluster API | `minio.storage.svc.cluster.local:9000` |
|
||||
| Console port | `9001` |
|
||||
| Replication | 3-node site-replication (az-a ↔ az-b ↔ az-c) |
|
||||
| Buckets (system) | `loki-chunks`, `loki-ruler`, `vault-backups` |
|
||||
|
||||
## Common Patterns
|
||||
|
||||
**Loki log backend (auto-configured):**
|
||||
```yaml
|
||||
# k8s/logging/loki-values.yaml
|
||||
loki:
|
||||
storage:
|
||||
s3:
|
||||
endpoint: minio.storage.svc.cluster.local:9000
|
||||
buckets: loki-chunks
|
||||
secretAccessKey: $MINIO_ROOT_PASSWORD
|
||||
accessKeyId: $MINIO_ROOT_USER
|
||||
```
|
||||
|
||||
**Application usage (Go/Python/Node):**
|
||||
```go
|
||||
import "github.com/minio/minio-go/v7"
|
||||
|
||||
client, _ := minio.New("minio.storage.svc.cluster.local:9000", &minio.Options{
|
||||
Creds: credentials.NewStaticV4(os.Getenv("MINIO_ROOT_USER"), os.Getenv("MINIO_ROOT_PASSWORD"), ""),
|
||||
Secure: false, // cluster-internal (no TLS)
|
||||
})
|
||||
|
||||
// Upload
|
||||
client.FPutObject(ctx, "my-bucket", "file.txt", "/path/to/file.txt", minio.PutObjectOptions{})
|
||||
|
||||
// Download
|
||||
client.FGetObject(ctx, "my-bucket", "file.txt", "/tmp/file.txt", minio.GetObjectOptions{})
|
||||
```
|
||||
|
||||
**Vault backup bucket:**
|
||||
```bash
|
||||
# Vault stores unseal keys in s3://vault-backups
|
||||
# Auto-managed by helmfile; no manual action needed
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
**Grafana dashboard:** `svc-minio`
|
||||
|
||||
**Key metrics:**
|
||||
- `minio_disk_drive_free_bytes` — available space
|
||||
- `minio_bucket_usage_object_count` — objects per bucket
|
||||
- `minio_bucket_usage_total_bytes` — total size per bucket
|
||||
|
||||
**Site replication status:**
|
||||
```bash
|
||||
# Port-forward to MinIO pod
|
||||
k port-forward -n storage pod/minio-0 9000:9000 &
|
||||
|
||||
# Check replication
|
||||
mc alias set local http://localhost:9000 $MINIO_ROOT_USER $MINIO_ROOT_PASSWORD
|
||||
mc admin replicate status local
|
||||
```
|
||||
|
||||
## Authentication (Cluster-Internal)
|
||||
|
||||
**From pods (cluster-internal):**
|
||||
```bash
|
||||
# Use credentials from Secret or env var
|
||||
export MINIO_ENDPOINT=minio.storage.svc.cluster.local:9000
|
||||
export MINIO_ACCESS_KEY=$MINIO_ROOT_USER
|
||||
export MINIO_SECRET_KEY=$MINIO_ROOT_PASSWORD
|
||||
aws s3 ls --endpoint-url http://$MINIO_ENDPOINT --use-path-style
|
||||
```
|
||||
|
||||
**External access (HTTPS via Ingress):**
|
||||
```bash
|
||||
# Console: https://minio.riotpiao.homelab.com (port 9001)
|
||||
# API: Use AWS CLI with --endpoint-url https://minio.riotpiao.homelab.com:9000
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Bucket creation fails:**
|
||||
```bash
|
||||
# Check MinIO pod logs
|
||||
k logs -n storage pod/minio-0 | grep -i error
|
||||
|
||||
# Verify storage space
|
||||
k get pvc -n storage
|
||||
```
|
||||
|
||||
**Site replication lag:**
|
||||
```bash
|
||||
# Check if all 3 nodes are healthy
|
||||
k get pods -n storage -l app=minio
|
||||
|
||||
# If one node is down, site replication queues changes (eventually consistent)
|
||||
```
|
||||
|
||||
**Access denied:**
|
||||
```bash
|
||||
# Verify credentials in .env
|
||||
echo $MINIO_ROOT_USER $MINIO_ROOT_PASSWORD
|
||||
|
||||
# If credentials rotated, update Secret
|
||||
k patch secret -n storage minio-root-credentials \
|
||||
--type merge -p '{"stringData":{"MINIO_ROOT_PASSWORD":"newpass"}}'
|
||||
```
|
||||
|
||||
See `/TROUBLESHOOTING.md` for full incident guide.
|
||||
@@ -0,0 +1,265 @@
|
||||
# Monitoring: Prometheus, Grafana & Loki
|
||||
|
||||
**Prometheus:** `prometheus-kube-prom-prometheus.monitoring.svc.cluster.local:9090`
|
||||
**Grafana:** `https://grafana.riotpiao.homelab.com`
|
||||
**Loki:** `loki.logging.svc.cluster.local:3100`
|
||||
**Namespaces:** `monitoring`, `logging`
|
||||
|
||||
## When to Use
|
||||
|
||||
- **Metrics** — CPU, memory, latency, request rate, custom business metrics (Prometheus)
|
||||
- **Logs** — Pod logs, kernel logs, trace events (Loki)
|
||||
- **Dashboards** — Real-time visualization (Grafana)
|
||||
- **Alerts** — Page on high error rate, latency spike, or resource exhaustion
|
||||
|
||||
## Quick Start
|
||||
|
||||
**1. Access Grafana:**
|
||||
```bash
|
||||
# Browser: https://grafana.riotpiao.homelab.com
|
||||
# Login: admin / GRAFANA_ADMIN_PASSWORD (from .env)
|
||||
# Or via Authentik SSO
|
||||
|
||||
# Port-forward (if no external access)
|
||||
make pf-grafana # localhost:3000
|
||||
```
|
||||
|
||||
**2. View dashboards:**
|
||||
```
|
||||
Grafana → Dashboards → Homelab folder
|
||||
- Service Availability (uptime probes + cert expiry)
|
||||
- Latency & Golden Signals (request rate/error %/duration)
|
||||
- Hardware Statistics (per-node CPU/mem/disk)
|
||||
- Kube-Controller Health (API server metrics)
|
||||
- Service Internals (per-service deep-dive)
|
||||
```
|
||||
|
||||
**3. Add Prometheus data source:**
|
||||
```
|
||||
Grafana → Settings → Data Sources → Add
|
||||
- Type: Prometheus
|
||||
- URL: http://prometheus-kube-prom-prometheus.monitoring.svc.cluster.local:9090
|
||||
- Save & test
|
||||
```
|
||||
|
||||
**4. Create dashboard:**
|
||||
```json
|
||||
{
|
||||
"dashboard": {
|
||||
"title": "My Service",
|
||||
"panels": [
|
||||
{
|
||||
"targets": [
|
||||
{
|
||||
"expr": "rate(http_requests_total{service='myapp'}[5m])"
|
||||
}
|
||||
],
|
||||
"title": "Request Rate"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Instrumentation Patterns
|
||||
|
||||
**Go (Prometheus):**
|
||||
```go
|
||||
import "github.com/prometheus/client_golang/prometheus"
|
||||
|
||||
// Counter (monotonic increase)
|
||||
requestsTotal := prometheus.NewCounterVec(
|
||||
prometheus.CounterOpts{
|
||||
Name: "http_requests_total",
|
||||
Help: "Total requests",
|
||||
},
|
||||
[]string{"method", "status"},
|
||||
)
|
||||
|
||||
// Gauge (can go up or down)
|
||||
activeConnections := prometheus.NewGauge(prometheus.GaugeOpts{
|
||||
Name: "active_connections",
|
||||
})
|
||||
|
||||
// Histogram (latency buckets)
|
||||
requestDuration := prometheus.NewHistogramVec(
|
||||
prometheus.HistogramOpts{
|
||||
Name: "http_request_duration_seconds",
|
||||
Buckets: []float64{.001, .01, .1, 1},
|
||||
},
|
||||
[]string{"method", "path"},
|
||||
)
|
||||
|
||||
// Record metric
|
||||
requestsTotal.WithLabelValues("POST", "200").Inc()
|
||||
requestDuration.WithLabelValues("POST", "/api/users").Observe(0.042)
|
||||
```
|
||||
|
||||
**Node.js (prom-client):**
|
||||
```javascript
|
||||
const prometheus = require('prom-client');
|
||||
|
||||
const httpRequestDuration = new prometheus.Histogram({
|
||||
name: 'http_request_duration_seconds',
|
||||
help: 'Duration of HTTP requests in seconds',
|
||||
labelNames: ['method', 'path', 'status'],
|
||||
buckets: [0.001, 0.01, 0.1, 1],
|
||||
});
|
||||
|
||||
// Middleware
|
||||
app.use((req, res, next) => {
|
||||
const start = Date.now();
|
||||
res.on('finish', () => {
|
||||
const duration = (Date.now() - start) / 1000;
|
||||
httpRequestDuration.labels(req.method, req.path, res.statusCode).observe(duration);
|
||||
});
|
||||
next();
|
||||
});
|
||||
|
||||
// Expose metrics
|
||||
app.get('/metrics', (req, res) => {
|
||||
res.set('Content-Type', prometheus.register.contentType);
|
||||
res.end(prometheus.register.metrics());
|
||||
});
|
||||
```
|
||||
|
||||
## ServiceMonitor (Auto-Scrape)
|
||||
|
||||
**Prometheus automatically discovers ServiceMonitor resources:**
|
||||
```yaml
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: myapp
|
||||
namespace: myapp-ns
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: myapp
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
path: /metrics
|
||||
```
|
||||
|
||||
**Deploy & verify:**
|
||||
```bash
|
||||
kubectl apply -f myapp-servicemonitor.yaml
|
||||
|
||||
# Check Prometheus targets
|
||||
kubectl port-forward -n monitoring svc/prometheus-kube-prom-prometheus 9090:9090
|
||||
# http://localhost:9090/targets → should show myapp
|
||||
```
|
||||
|
||||
## PrometheusRule (Alerting)
|
||||
|
||||
**Define alert rules:**
|
||||
```yaml
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: myapp-alerts
|
||||
namespace: myapp-ns
|
||||
spec:
|
||||
groups:
|
||||
- name: myapp
|
||||
rules:
|
||||
- alert: HighErrorRate
|
||||
expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05 # > 5% error rate
|
||||
for: 5m
|
||||
annotations:
|
||||
summary: "High error rate for {{ $labels.service }}"
|
||||
|
||||
- alert: HighLatency
|
||||
expr: histogram_quantile(0.99, http_request_duration_seconds_bucket) > 1 # p99 > 1s
|
||||
for: 10m
|
||||
annotations:
|
||||
summary: "High latency for {{ $labels.path }}"
|
||||
```
|
||||
|
||||
## Logging (Loki)
|
||||
|
||||
**View pod logs in Grafana:**
|
||||
```
|
||||
Grafana → Explore → Loki
|
||||
Query: {namespace="story-crater-backend"} | json | level="error"
|
||||
|
||||
Label filters:
|
||||
- namespace
|
||||
- pod_name
|
||||
- container
|
||||
- level (error, warn, info)
|
||||
```
|
||||
|
||||
**Log retention:**
|
||||
- 10-day default (configurable)
|
||||
- Older logs deleted automatically
|
||||
- Chunks stored in MinIO (s3://loki-chunks)
|
||||
|
||||
## Monitoring Checklist
|
||||
|
||||
**For every service:**
|
||||
- [ ] Export `/metrics` (Prometheus format)
|
||||
- [ ] Add ServiceMonitor resource
|
||||
- [ ] Add PrometheusRule (errors, latency)
|
||||
- [ ] Create Grafana dashboard (6 rows: Availability, Resources, Domain, Logs, SLO, Related)
|
||||
- [ ] Set up Slack/PagerDuty integration (if on-call)
|
||||
|
||||
## Key Queries
|
||||
|
||||
**Request rate by status:**
|
||||
```promql
|
||||
sum(rate(http_requests_total[5m])) by (status)
|
||||
```
|
||||
|
||||
**Error rate %:**
|
||||
```promql
|
||||
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100
|
||||
```
|
||||
|
||||
**P99 latency:**
|
||||
```promql
|
||||
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))
|
||||
```
|
||||
|
||||
**Memory usage %:**
|
||||
```promql
|
||||
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Prometheus can't scrape ServiceMonitor:**
|
||||
```bash
|
||||
# Check ServiceMonitor exists
|
||||
k get servicemonitor -A | grep myapp
|
||||
|
||||
# Verify Service has correct labels
|
||||
k get svc -A -o json | jq '.items[] | select(.metadata.name=="myapp") | .metadata.labels'
|
||||
|
||||
# Check Prometheus targets
|
||||
kubectl port-forward -n monitoring svc/prometheus-kube-prom-prometheus 9090:9090
|
||||
# http://localhost:9090/targets → look for "Down" targets with error
|
||||
```
|
||||
|
||||
**Metrics missing from Grafana:**
|
||||
```bash
|
||||
# Verify pod is exporting metrics
|
||||
k port-forward -n myapp-ns pod/myapp-0 8080:8080
|
||||
curl http://localhost:8080/metrics | grep my_metric_name
|
||||
|
||||
# Check query syntax in Grafana
|
||||
# Hover over query → "Query Inspector" to see response
|
||||
```
|
||||
|
||||
**Alert not firing:**
|
||||
```bash
|
||||
# Check PrometheusRule is loaded
|
||||
k get prometheusrule -n myapp-ns
|
||||
|
||||
# Verify query in Prometheus
|
||||
kubectl port-forward -n monitoring svc/prometheus-kube-prom-prometheus 9090:9090
|
||||
# http://localhost:9090 → Graph tab → paste query
|
||||
```
|
||||
|
||||
See `/TROUBLESHOOTING.md` for full incident guide.
|
||||
@@ -0,0 +1,314 @@
|
||||
# Networking: Ingress, TLS & Service Discovery
|
||||
|
||||
**Ingress Controller:** `nginx-ingress` (Nginx)
|
||||
**Load Balancer:** Cilium LB-IPAM (eBPF-based)
|
||||
**TLS CA:** homelab-ca (self-signed, 10-year)
|
||||
**Namespace:** `ingress-nginx`
|
||||
|
||||
## When to Use
|
||||
|
||||
- **Public HTTPS endpoints** — External access via TLS
|
||||
- **Hostname-based routing** — Multiple services on same IP
|
||||
- **TLS termination** — Offload encryption/decryption
|
||||
- **Service discovery** — Internal DNS (CoreDNS)
|
||||
|
||||
## Quick Start
|
||||
|
||||
**1. Create Ingress rule:**
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: myapp
|
||||
namespace: myapp-ns
|
||||
annotations:
|
||||
cert-manager.io/cluster-issuer: "letsencrypt-prod" # or homelab-ca
|
||||
spec:
|
||||
ingressClassName: nginx
|
||||
tls:
|
||||
- hosts:
|
||||
- myapp.riotpiao.homelab.com
|
||||
secretName: myapp-tls
|
||||
rules:
|
||||
- host: myapp.riotpiao.homelab.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: myapp-svc
|
||||
port:
|
||||
number: 8080
|
||||
```
|
||||
|
||||
**2. Deploy:**
|
||||
```bash
|
||||
kubectl apply -f ingress.yaml
|
||||
|
||||
# Wait for cert issuance
|
||||
kubectl get certificate -n myapp-ns
|
||||
# Should show "Ready" after ~30s
|
||||
```
|
||||
|
||||
**3. Test from client:**
|
||||
```bash
|
||||
# Add to /etc/hosts (or use WireGuard)
|
||||
192.168.1.160 myapp.riotpiao.homelab.com
|
||||
|
||||
# Access
|
||||
curl https://myapp.riotpiao.homelab.com
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
| Key | Value |
|
||||
|-----|-------|
|
||||
| Ingress class | `nginx` |
|
||||
| Load balancer type | `LoadBalancer` (Cilium LB-IPAM) |
|
||||
| TLS issuer | `homelab-ca` (ClusterIssuer) |
|
||||
| TLS cert lifetime | 90 days (auto-renewed by cert-manager) |
|
||||
| DNS | CoreDNS (in-cluster), external via `/etc/hosts` or DuckDNS |
|
||||
|
||||
## Common Patterns
|
||||
|
||||
**Ingress with path-based routing:**
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: api
|
||||
namespace: default
|
||||
spec:
|
||||
ingressClassName: nginx
|
||||
tls:
|
||||
- hosts:
|
||||
- api.riotpiao.homelab.com
|
||||
secretName: api-tls
|
||||
rules:
|
||||
- host: api.riotpiao.homelab.com
|
||||
http:
|
||||
paths:
|
||||
- path: /users
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: users-svc
|
||||
port:
|
||||
number: 3000
|
||||
- path: /orders
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: orders-svc
|
||||
port:
|
||||
number: 3001
|
||||
```
|
||||
|
||||
**Ingress with basic auth:**
|
||||
```bash
|
||||
# Generate htpasswd
|
||||
htpasswd -c auth admin
|
||||
# → prompted for password
|
||||
|
||||
# Create Secret
|
||||
kubectl create secret generic basic-auth --from-file=auth -n default
|
||||
|
||||
# Create Ingress
|
||||
```
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: protected
|
||||
namespace: default
|
||||
annotations:
|
||||
nginx.ingress.kubernetes.io/auth-type: basic
|
||||
nginx.ingress.kubernetes.io/auth-secret: basic-auth
|
||||
nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required'
|
||||
spec:
|
||||
ingressClassName: nginx
|
||||
rules:
|
||||
- host: protected.riotpiao.homelab.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: app-svc
|
||||
port:
|
||||
number: 8080
|
||||
```
|
||||
|
||||
**Internal DNS (CoreDNS rewrite):**
|
||||
```yaml
|
||||
# k8s/coredns/coredns-configmap.yaml
|
||||
# Rewrite:
|
||||
# - grafana.riotpiao.homelab.com → grafana.logging (cluster-internal)
|
||||
# - prometheus.riotpiao.homelab.com → prometheus-kube-prom-prometheus.monitoring
|
||||
#
|
||||
# Allows pods to use external URLs but resolve to internal Services
|
||||
```
|
||||
|
||||
**Fixed LoadBalancer IP (Cilium LB-IPAM):**
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: ingress-nginx
|
||||
namespace: ingress-nginx
|
||||
annotations:
|
||||
io.cilium/lb-ipam-ips: "192.168.1.160" # fixed IP
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
selector:
|
||||
app: nginx-ingress
|
||||
ports:
|
||||
- port: 443
|
||||
targetPort: 443
|
||||
protocol: TCP
|
||||
```
|
||||
|
||||
## TLS Certificate Management
|
||||
|
||||
**Automatic renewal (cert-manager):**
|
||||
```yaml
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: myapp-cert
|
||||
namespace: myapp-ns
|
||||
spec:
|
||||
secretName: myapp-tls
|
||||
duration: 2160h # 90 days
|
||||
renewBefore: 360h # renew 15 days before expiry
|
||||
commonName: myapp.riotpiao.homelab.com
|
||||
dnsNames:
|
||||
- myapp.riotpiao.homelab.com
|
||||
issuerRef:
|
||||
name: homelab-ca
|
||||
kind: ClusterIssuer
|
||||
```
|
||||
|
||||
**Check certificate status:**
|
||||
```bash
|
||||
# List certs
|
||||
k get certificate -A
|
||||
|
||||
# View cert details
|
||||
k describe certificate -n myapp-ns myapp-cert
|
||||
|
||||
# View TLS Secret
|
||||
k get secret -n myapp-ns myapp-tls -o json | jq '.data."tls.crt"' | base64 -d | openssl x509 -text
|
||||
|
||||
# Check expiry date
|
||||
k get secret -n myapp-ns myapp-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate
|
||||
```
|
||||
|
||||
## Service Discovery
|
||||
|
||||
**Cluster-internal DNS:**
|
||||
```bash
|
||||
# From any pod, resolve via CoreDNS
|
||||
nslookup grafana.logging.svc.cluster.local # full FQDN
|
||||
nslookup grafana.logging # short form (same namespace)
|
||||
nslookup grafana # if in logging namespace
|
||||
|
||||
# Resolved to ClusterIP (internal only)
|
||||
```
|
||||
|
||||
**External DNS (WireGuard VPN or port-forward):**
|
||||
```bash
|
||||
# Option 1: WireGuard tunnel
|
||||
# Client connects to 10.6.0.1 (WireGuard server on talos-cp-1)
|
||||
# All traffic tunneled to cluster
|
||||
|
||||
# Option 2: Port-forward from jump box
|
||||
make pf-grafana # localhost:3000 → grafana.logging:3000
|
||||
|
||||
# Option 3: Add to /etc/hosts (on home network)
|
||||
192.168.1.160 grafana.riotpiao.homelab.com
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
**Grafana dashboard:** `svc-nginx-ingress`
|
||||
|
||||
**Key metrics:**
|
||||
- `nginx_requests_total` — total requests
|
||||
- `nginx_request_duration_seconds` — latency histogram
|
||||
- `nginx_ingress_upstream_requests_total{status=~"5.."}` — backend errors
|
||||
- `nginx_ssl_expire_time_seconds` — cert expiry countdown
|
||||
|
||||
**Alert on cert expiry:**
|
||||
```yaml
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: cert-expiry
|
||||
namespace: ingress-nginx
|
||||
spec:
|
||||
groups:
|
||||
- name: cert-expiry
|
||||
rules:
|
||||
- alert: CertificateExpiringSoon
|
||||
expr: nginx_ssl_expire_time_seconds < 86400 * 14 # < 14 days
|
||||
annotations:
|
||||
summary: "Certificate {{ $labels.host }} expires in {{ $value | humanizeDuration }}"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Certificate stuck in "Pending":**
|
||||
```bash
|
||||
# Check cert-manager logs
|
||||
k logs -n cert-manager -f deploy/cert-manager
|
||||
|
||||
# Verify ClusterIssuer exists
|
||||
k get clusterissuer
|
||||
|
||||
# Check ACME order (if using LetsEncrypt)
|
||||
k describe certificate -n myapp-ns myapp-cert
|
||||
```
|
||||
|
||||
**Ingress not exposing service (503 error):**
|
||||
```bash
|
||||
# Verify Service exists and has endpoints
|
||||
k get svc -n myapp-ns
|
||||
k get endpoints -n myapp-ns myapp-svc
|
||||
|
||||
# Check if pods are ready
|
||||
k get pods -n myapp-ns
|
||||
|
||||
# Test pod directly (port-forward)
|
||||
k port-forward -n myapp-ns pod/myapp-0 8080:8080
|
||||
curl http://localhost:8080
|
||||
```
|
||||
|
||||
**DNS resolution fails from pod:**
|
||||
```bash
|
||||
# Test from pod
|
||||
k run -it --rm debug --image=busybox:1.28 --restart=Never -- \
|
||||
nslookup grafana.logging.svc.cluster.local
|
||||
|
||||
# If fails, CoreDNS may be unhealthy
|
||||
k get pods -n kube-system -l k8s-app=kube-dns
|
||||
k logs -n kube-system -l k8s-app=kube-dns
|
||||
```
|
||||
|
||||
**TLS handshake error (cert not trusted):**
|
||||
```bash
|
||||
# Verify TLS cert Secret exists
|
||||
k get secret -n myapp-ns myapp-tls
|
||||
|
||||
# Verify cert is correctly signed by homelab-ca
|
||||
k get secret -n myapp-ns myapp-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text | grep -A 5 "Issuer:"
|
||||
|
||||
# If cert is self-signed (homelab-ca), add to client's trusted roots
|
||||
# Or bypass cert verification (dev only):
|
||||
curl -k https://myapp.riotpiao.homelab.com
|
||||
```
|
||||
|
||||
See `/TROUBLESHOOTING.md` for full incident guide.
|
||||
@@ -0,0 +1,193 @@
|
||||
# SQS-like Message Queue Service (kmsvc)
|
||||
|
||||
**Endpoint:** `https://kmsvc.riotpiao.homelab.com` (REST + gRPC-Gateway)
|
||||
**Internal:** `kmsvc-management-service.sqs.svc.cluster.local:8080`
|
||||
**Namespace:** `sqs`
|
||||
|
||||
## When to Use
|
||||
|
||||
- **Decouple services** — Producer doesn't wait for consumer
|
||||
- **Async jobs** — Fire-and-forget processing (batch, email, webhooks)
|
||||
- **FIFO ordering** — Guarantee message order within `MessageGroupId`
|
||||
- **Durable delivery** — At-least-once (messages in Kafka, replicated 3×)
|
||||
|
||||
## Quick Start
|
||||
|
||||
**1. Create a queue:**
|
||||
```bash
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: kmsvc.io/v1
|
||||
kind: Queue
|
||||
metadata:
|
||||
name: orders
|
||||
spec:
|
||||
fifoQueue: false # standard queue
|
||||
visibilityTimeoutSeconds: 30 # re-deliver if not acked
|
||||
messageRetentionPeriodSeconds: 345600 # 4 days
|
||||
partitionsPerShard: 6
|
||||
maxReceiveCount: 5 # move to DLQ after 5 fails
|
||||
EOF
|
||||
```
|
||||
|
||||
**2. Send message:**
|
||||
```bash
|
||||
curl -X POST https://kmsvc.riotpiao.homelab.com/v1/queues/orders/messages \
|
||||
-H "Authorization: Bearer $JWT_TOKEN" \
|
||||
-d '{
|
||||
"body": "{\"order_id\":123,\"total\":99.99}",
|
||||
"attributes": {"source":"web","priority":"high"}
|
||||
}'
|
||||
```
|
||||
|
||||
**3. Receive message:**
|
||||
```bash
|
||||
curl "https://kmsvc.riotpiao.homelab.com/v1/queues/orders/messages?max_number_of_messages=10&wait_time_seconds=20" \
|
||||
-H "Authorization: Bearer $JWT_TOKEN"
|
||||
|
||||
# Response:
|
||||
# {
|
||||
# "messages": [
|
||||
# {
|
||||
# "message_id": "abc-123",
|
||||
# "receipt_handle": "...",
|
||||
# "body": "{...}",
|
||||
# "attributes": {...},
|
||||
# "receive_count": 1
|
||||
# }
|
||||
# ]
|
||||
# }
|
||||
```
|
||||
|
||||
**4. Acknowledge (delete) message:**
|
||||
```bash
|
||||
curl -X DELETE "https://kmsvc.riotpiao.homelab.com/v1/queues/orders/messages/$receipt_handle" \
|
||||
-H "Authorization: Bearer $JWT_TOKEN"
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
| Key | Value |
|
||||
|-----|-------|
|
||||
| Kafka bootstrap | `kmsvc-kafka-bootstrap.sqs.svc.cluster.local:9092` |
|
||||
| Redis | `kmsvc-redis-master.sqs.svc.cluster.local:6379` |
|
||||
| Topic naming | `kmsvc.{queueName}.shard-{id}` |
|
||||
| Replication | 3 replicas, min.insync.replicas=2 |
|
||||
| Retention | 4 days (configurable per queue) |
|
||||
|
||||
## Common Patterns
|
||||
|
||||
**Batch processing:**
|
||||
```bash
|
||||
for i in {1..100}; do
|
||||
curl -X POST https://kmsvc.riotpiao.homelab.com/v1/queues/jobs/messages \
|
||||
-H "Authorization: Bearer $JWT_TOKEN" \
|
||||
-d "{\"body\":\"task-$i\"}" &
|
||||
done
|
||||
wait
|
||||
```
|
||||
|
||||
**FIFO queue (order guaranteed per group):**
|
||||
```yaml
|
||||
apiVersion: kmsvc.io/v1
|
||||
kind: Queue
|
||||
metadata:
|
||||
name: checkout-fifo
|
||||
spec:
|
||||
fifoQueue: true
|
||||
visibilityTimeoutSeconds: 60
|
||||
partitionsPerShard: 1
|
||||
```
|
||||
|
||||
**Dead-letter queue (failed messages):**
|
||||
```yaml
|
||||
apiVersion: kmsvc.io/v1
|
||||
kind: Queue
|
||||
metadata:
|
||||
name: orders-dlq
|
||||
spec:
|
||||
fifoQueue: false
|
||||
|
||||
---
|
||||
apiVersion: kmsvc.io/v1
|
||||
kind: Queue
|
||||
metadata:
|
||||
name: orders
|
||||
spec:
|
||||
fifoQueue: false
|
||||
maxReceiveCount: 3
|
||||
deadLetterTargetQueue: orders-dlq # auto-route failures here
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
**Grafana dashboard:** `svc-kmsvc` (automatically loaded)
|
||||
|
||||
**Key metrics:**
|
||||
- `kmsvc_messages_sent_total` — total sent
|
||||
- `kmsvc_messages_received_total` — total received
|
||||
- `kmsvc_queue_depth` — pending messages per queue
|
||||
- `kmsvc_message_visibility_timeout_seconds` — visibility window
|
||||
|
||||
**Redis in-flight tracking:**
|
||||
```bash
|
||||
# Connect to Redis
|
||||
k port-forward -n sqs svc/redis 6379:6379 &
|
||||
redis-cli
|
||||
|
||||
# Check pending messages
|
||||
KEYS "kmsvc:pending:orders:*"
|
||||
KEYS "kmsvc:inflight:*" | wc -l
|
||||
```
|
||||
|
||||
## Authentication
|
||||
|
||||
**Requires JWT from Authentik:**
|
||||
```bash
|
||||
# Get token (device code flow)
|
||||
talos secrets login
|
||||
|
||||
# Use token
|
||||
export JWT_TOKEN=$(talos get cluster/kmsvc/jwt-token --key jwt-token)
|
||||
curl -H "Authorization: Bearer $JWT_TOKEN" https://kmsvc.riotpiao.homelab.com/v1/queues
|
||||
```
|
||||
|
||||
## Integration Example
|
||||
|
||||
**Story Crater backend consumer:**
|
||||
```go
|
||||
// Receive messages
|
||||
messages, err := kmsvc.ReceiveMessage(ctx, &kmsvc.ReceiveMessageRequest{
|
||||
QueueName: "story-crater",
|
||||
MaxNumberOfMessages: 10,
|
||||
WaitTimeSeconds: 20,
|
||||
})
|
||||
|
||||
// Process
|
||||
for _, msg := range messages.Messages {
|
||||
processMessage(msg.Body)
|
||||
|
||||
// Acknowledge on success
|
||||
kmsvc.DeleteMessage(ctx, &kmsvc.DeleteMessageRequest{
|
||||
QueueName: "story-crater",
|
||||
ReceiptHandle: msg.ReceiptHandle,
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Queue stuck / high lag:**
|
||||
```bash
|
||||
# Check Kafka broker status
|
||||
k exec -n sqs pod/kmsvc-kafka-0 -- kafka-broker-api-versions.sh --bootstrap-server localhost:9092
|
||||
|
||||
# Inspect queue topics
|
||||
k exec -n sqs pod/kmsvc-kafka-0 -- kafka-topics.sh --bootstrap-server localhost:9092 --list | grep orders
|
||||
```
|
||||
|
||||
**Messages not being consumed:**
|
||||
- Check `maxReceiveCount` (may be routing to DLQ)
|
||||
- Verify consumer has `ReceiveMessage` permission (JWT scope)
|
||||
- Check Redis: `KEYS "kmsvc:fifo_lock:orders:*"` (may be blocked by visibility timeout)
|
||||
|
||||
See `/TROUBLESHOOTING.md` for full incident guide.
|
||||
@@ -0,0 +1,227 @@
|
||||
# Temporal Workflow Orchestration
|
||||
|
||||
**Server:** `temporal.temporal.svc.cluster.local:7233` (cluster-internal)
|
||||
**Web UI:** `kubectl port-forward -n temporal svc/temporal-web 8088:8088`
|
||||
**Namespace:** `temporal`
|
||||
|
||||
## When to Use
|
||||
|
||||
- **Long-running operations** — Tasks that take minutes/hours (send email, process batch, retry with backoff)
|
||||
- **State machines** — Multi-step workflows with decision logic
|
||||
- **Retries & timeouts** — Built-in exponential backoff, automatic retry
|
||||
- **Audit trail** — Full history of workflow executions (why it happened, when, by whom)
|
||||
|
||||
## Quick Start
|
||||
|
||||
**1. Access Temporal Web UI:**
|
||||
```bash
|
||||
k port-forward -n temporal svc/temporal-web 8088:8088
|
||||
# http://localhost:8088
|
||||
```
|
||||
|
||||
**2. Define workflow (Go example):**
|
||||
```go
|
||||
package workflows
|
||||
|
||||
import (
|
||||
"time"
|
||||
"go.temporal.io/sdk/workflow"
|
||||
"go.temporal.io/sdk/activity"
|
||||
)
|
||||
|
||||
type Inputs struct {
|
||||
OrderID string
|
||||
Amount float64
|
||||
}
|
||||
|
||||
// Workflow definition
|
||||
func OrderProcessing(ctx workflow.Context, input Inputs) (string, error) {
|
||||
// Step 1: Charge payment
|
||||
chargeResult := ""
|
||||
err := workflow.ExecuteActivity(
|
||||
ctx,
|
||||
ChargePayment,
|
||||
input.OrderID,
|
||||
input.Amount,
|
||||
).Get(ctx, &chargeResult)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
// Step 2: Send confirmation email (retry 3x on failure)
|
||||
emailResult := ""
|
||||
opts := workflow.ActivityOptions{
|
||||
StartToCloseTimeout: time.Minute,
|
||||
RetryPolicy: &temporal.RetryPolicy{
|
||||
InitialInterval: time.Second,
|
||||
BackoffCoefficient: 2,
|
||||
MaximumAttempts: 3,
|
||||
},
|
||||
}
|
||||
ctx = workflow.WithActivityOptions(ctx, opts)
|
||||
|
||||
err = workflow.ExecuteActivity(ctx, SendConfirmationEmail, input.OrderID).Get(ctx, &emailResult)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
return "order_processed", nil
|
||||
}
|
||||
|
||||
// Activity: payment processing
|
||||
func ChargePayment(ctx context.Context, orderID string, amount float64) (string, error) {
|
||||
// Call payment gateway
|
||||
return "payment_successful", nil
|
||||
}
|
||||
|
||||
// Activity: email notification
|
||||
func SendConfirmationEmail(ctx context.Context, orderID string) (string, error) {
|
||||
// Send email
|
||||
return "email_sent", nil
|
||||
}
|
||||
```
|
||||
|
||||
**3. Register & start workflow:**
|
||||
```go
|
||||
import "go.temporal.io/sdk/client"
|
||||
|
||||
client, _ := client.Dial(client.Options{
|
||||
HostPort: "temporal.temporal.svc.cluster.local:7233",
|
||||
})
|
||||
|
||||
// Start workflow execution
|
||||
run, _ := client.ExecuteWorkflow(ctx, client.StartWorkflowOptions{
|
||||
ID: "order-123",
|
||||
TaskQueue: "orders",
|
||||
}, OrderProcessing, Inputs{OrderID: "123", Amount: 99.99})
|
||||
|
||||
// Wait for result
|
||||
var result string
|
||||
run.Get(ctx, &result)
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
| Key | Value |
|
||||
|-----|-------|
|
||||
| Server | `temporal.temporal.svc.cluster.local:7233` |
|
||||
| Web UI | `localhost:8088` (via port-forward) |
|
||||
| Database | PostgreSQL (managed by helmfile) |
|
||||
| Task queue | `default`, `orders`, `emails` (custom per app) |
|
||||
| Retention | 30 days (configurable) |
|
||||
|
||||
## Common Patterns
|
||||
|
||||
**Retry with exponential backoff:**
|
||||
```go
|
||||
opts := workflow.ActivityOptions{
|
||||
StartToCloseTimeout: 5 * time.Minute,
|
||||
RetryPolicy: &temporal.RetryPolicy{
|
||||
InitialInterval: time.Second,
|
||||
BackoffCoefficient: 2.0, // double wait time each retry
|
||||
MaximumInterval: time.Minute, // cap at 1 min between retries
|
||||
MaximumAttempts: 5, // give up after 5 tries
|
||||
},
|
||||
}
|
||||
ctx = workflow.WithActivityOptions(ctx, opts)
|
||||
```
|
||||
|
||||
**Wait for signal (user approval):**
|
||||
```go
|
||||
// Workflow waits for approval signal
|
||||
approval := ""
|
||||
workflow.GetSignalChannel(ctx, "approval").Receive(ctx, &approval)
|
||||
|
||||
if approval == "approved" {
|
||||
// Continue workflow
|
||||
} else {
|
||||
return "", errors.New("request denied")
|
||||
}
|
||||
```
|
||||
|
||||
**Parallel activities:**
|
||||
```go
|
||||
// Execute email & SMS in parallel
|
||||
emailFuture := workflow.ExecuteActivity(ctx, SendEmail, userID)
|
||||
smsFuture := workflow.ExecuteActivity(ctx, SendSMS, userID)
|
||||
|
||||
// Wait for both to complete
|
||||
emailFuture.Get(ctx, nil)
|
||||
smsFuture.Get(ctx, nil)
|
||||
```
|
||||
|
||||
**Scheduled workflow (cron):**
|
||||
```go
|
||||
run, _ := client.ExecuteWorkflow(ctx, client.StartWorkflowOptions{
|
||||
ID: "daily-report",
|
||||
CronSchedule: "0 9 * * MON-FRI", // 9 AM weekdays
|
||||
WorkflowTaskTimeout: time.Hour,
|
||||
}, GenerateDailyReport, nil)
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
**Web UI:**
|
||||
- List workflows: http://localhost:8088/namespaces/default/workflows
|
||||
- View execution history: Click workflow ID
|
||||
- See activity logs, errors, retry attempts
|
||||
|
||||
**Grafana dashboard:** `svc-temporal` (auto-configured)
|
||||
|
||||
**Key metrics:**
|
||||
- `temporal_workflow_execution_duration_seconds` — workflow time
|
||||
- `temporal_activity_execution_duration_seconds` — activity time
|
||||
- `temporal_activity_execution_failed_total` — failed activities
|
||||
|
||||
## Integration with Story Crater
|
||||
|
||||
**Example: Process message via Temporal:**
|
||||
```go
|
||||
// In message handler
|
||||
client, _ := temporal.Dial(/* ... */)
|
||||
|
||||
run, _ := client.ExecuteWorkflow(ctx, client.StartWorkflowOptions{
|
||||
ID: fmt.Sprintf("msg-%s", messageID),
|
||||
TaskQueue: "story-crater",
|
||||
}, ProcessMessageWorkflow, Message{
|
||||
ID: messageID,
|
||||
Body: body,
|
||||
Source: "kafka-queue",
|
||||
})
|
||||
|
||||
// Non-blocking: workflow runs independently
|
||||
// Check status later
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Workflow stuck:**
|
||||
```bash
|
||||
# Check Temporal server health
|
||||
k get pods -n temporal
|
||||
|
||||
# View workflow execution history (via Web UI or CLI)
|
||||
tctl workflow show --workflow-id order-123
|
||||
|
||||
# Terminate stuck workflow
|
||||
tctl workflow terminate --workflow-id order-123
|
||||
```
|
||||
|
||||
**Activity retrying endlessly:**
|
||||
```go
|
||||
// Add max attempts or timeout
|
||||
RetryPolicy: &temporal.RetryPolicy{
|
||||
MaximumAttempts: 5, // must have this!
|
||||
}
|
||||
```
|
||||
|
||||
**PostgreSQL connection fails:**
|
||||
```bash
|
||||
# Check temporal pod logs
|
||||
k logs -n temporal pod/temporal-0
|
||||
|
||||
# Verify database is running
|
||||
k get pods -n ddb
|
||||
```
|
||||
|
||||
See `/TROUBLESHOOTING.md` for full incident guide.
|
||||
@@ -0,0 +1,211 @@
|
||||
# Vault: Secret Management & JWT Auth
|
||||
|
||||
**Vault:** `https://vault.riotpiao.homelab.com`
|
||||
**Internal:** `vault.iam.svc.cluster.local:8200`
|
||||
**Namespace:** `iam`
|
||||
|
||||
## When to Use
|
||||
|
||||
- **Store secrets** — Database passwords, API keys, TLS certs
|
||||
- **Rotate credentials** — Auto-rotate, track rotation history
|
||||
- **JWT validation** — Verify tokens from Authentik, no external call needed
|
||||
- **Audit trail** — Who accessed what secret, when
|
||||
|
||||
## Quick Start
|
||||
|
||||
**1. Login to Vault:**
|
||||
```bash
|
||||
# Browser: https://vault.riotpiao.homelab.com
|
||||
# Auth method: OIDC → "Sign in with Authentik" (federated)
|
||||
# Or: Device code → talos secrets login (CLI)
|
||||
|
||||
# Via CLI (device code flow)
|
||||
talos secrets login
|
||||
# → Opens browser, approve device code
|
||||
# → Token cached in ~/.talos/vault
|
||||
```
|
||||
|
||||
**2. Store a secret:**
|
||||
```bash
|
||||
# Field name = variable name (SCREAMING_SNAKE_CASE)
|
||||
talos put cluster/ANTHROPIC_API_KEY ANTHROPIC_API_KEY="sk-..."
|
||||
talos put cluster/STORY_CRATER_DB_PASS STORY_CRATER_DB_PASS="dbpass123"
|
||||
```
|
||||
|
||||
**3. Retrieve a secret:**
|
||||
```bash
|
||||
# Always use --key flag
|
||||
talos get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
|
||||
# → sk-...
|
||||
|
||||
# Full secret as JSON
|
||||
talos get cluster/ANTHROPIC_API_KEY --json
|
||||
```
|
||||
|
||||
**4. Load into shell (helmfile, scripts):**
|
||||
```bash
|
||||
vsource .env
|
||||
# → Expands empty vars from Vault (ANTHROPIC_API_KEY=)
|
||||
# → Passes hardcoded vars as-is (DEBUG=true)
|
||||
|
||||
helmfile diff
|
||||
helmfile apply
|
||||
```
|
||||
|
||||
## Vault Paths (KV v2)
|
||||
|
||||
**Naming convention:** `cluster/<VARIABLE_NAME>`
|
||||
|
||||
```
|
||||
cluster/
|
||||
├── ANTHROPIC_API_KEY # third-party API
|
||||
├── STORY_CRATER_DB_PASS # database password
|
||||
├── MINIO_ROOT_PASSWORD # MinIO credentials
|
||||
├── AUTHENTIK_BOOTSTRAP_PASSWORD # initial admin pass
|
||||
├── iam/
|
||||
│ ├── federation # Authentik OIDC app config
|
||||
│ ├── roles/admin # RBAC role definitions
|
||||
│ ├── services/grafana # OAuth2 client info
|
||||
│ ├── agents/ci-bot # machine credentials
|
||||
│ └── bindings/alice # user→role mappings
|
||||
└── kubernetes/
|
||||
├── ingress-tls # TLS cert private keys
|
||||
└── pull-secrets # Docker registry creds
|
||||
```
|
||||
|
||||
## Common Patterns
|
||||
|
||||
**Store generated secret immediately (keeps it out of shell history):**
|
||||
```bash
|
||||
# Generate & store in one command
|
||||
talos put cluster/GRAFANA_OIDC_CLIENT_SECRET \
|
||||
GRAFANA_OIDC_CLIENT_SECRET="$(openssl rand -hex 32)"
|
||||
```
|
||||
|
||||
**Use in Kubernetes Secret:**
|
||||
```bash
|
||||
# Create Secret using Vault secret
|
||||
kubectl create secret generic grafana-oidc \
|
||||
--from-literal=client-secret="$(talos get cluster/GRAFANA_OIDC_CLIENT_SECRET --key GRAFANA_OIDC_CLIENT_SECRET)" \
|
||||
-n logging
|
||||
```
|
||||
|
||||
**Rotate credentials safely:**
|
||||
```bash
|
||||
# 1. Generate new secret
|
||||
NEW_PASS=$(openssl rand -base64 24)
|
||||
|
||||
# 2. Store in Vault
|
||||
talos put cluster/OLD_DB_PASS OLD_DB_PASS="$NEW_PASS"
|
||||
|
||||
# 3. Update database user
|
||||
psql -h ddb-cluster-rw.ddb.svc.cluster.local -U postgres \
|
||||
-c "ALTER USER story_crater PASSWORD '$NEW_PASS';"
|
||||
|
||||
# 4. Update Kubernetes Secret
|
||||
kubectl patch secret db-credentials -n story-crater-backend \
|
||||
--type merge -p "{\"stringData\":{\"password\":\"$NEW_PASS\"}}"
|
||||
|
||||
# 5. Restart pods to pick up new Secret
|
||||
k rollout restart -n story-crater-backend deployment/app
|
||||
```
|
||||
|
||||
**JWT token from CLI:**
|
||||
```bash
|
||||
# After device code login
|
||||
talos secrets login
|
||||
|
||||
# Token is cached and auto-renewed
|
||||
# Use for API calls
|
||||
VAULT_TOKEN=$(cat ~/.talos/vault)
|
||||
curl -H "X-Vault-Token: $VAULT_TOKEN" \
|
||||
https://vault.iam.svc.cluster.local:8200/v1/secret/data/cluster/ANTHROPIC_API_KEY
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
**Vault status:**
|
||||
```bash
|
||||
# Check if sealed
|
||||
talos status vault
|
||||
|
||||
# If sealed (disaster recovery):
|
||||
# See /TROUBLESHOOTING.md § Vault Sealed
|
||||
```
|
||||
|
||||
**Audit log (who accessed what):**
|
||||
```bash
|
||||
# Enable audit logging (already enabled by helmfile)
|
||||
# Logs stored in Loki under vault namespace
|
||||
|
||||
# View recent access
|
||||
talos audit log --limit 50
|
||||
|
||||
# Export for compliance
|
||||
talos audit export --format json > vault-audit.json
|
||||
```
|
||||
|
||||
## Security Rules
|
||||
|
||||
✅ **DO:**
|
||||
- Store ALL secrets in Vault (never .env in git)
|
||||
- Use field name = variable name
|
||||
- Always use `--key` flag when retrieving
|
||||
- Rotate credentials on schedule (quarterly)
|
||||
- Review audit log for anomalies
|
||||
|
||||
❌ **DON'T:**
|
||||
- Commit `.env` with real secrets (only `.env.example`)
|
||||
- Use positional arguments (must use `--key`)
|
||||
- Share Vault root token
|
||||
- Store secrets in pod env (use Secret volumes)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Cannot login (Authentik OIDC fails):**
|
||||
```bash
|
||||
# Check Authentik is running
|
||||
k get pods -n iam -l app=authentik
|
||||
|
||||
# Verify OIDC app in Authentik console
|
||||
# Applications → Vault OIDC → Check client ID/secret
|
||||
|
||||
# Restart Vault to re-sync OIDC config
|
||||
k rollout restart -n iam statefulset/vault
|
||||
```
|
||||
|
||||
**Vault is sealed:**
|
||||
```bash
|
||||
# Check status
|
||||
talos status vault
|
||||
|
||||
# If sealed, use unseal keys (stored in MinIO backup)
|
||||
# See /TROUBLESHOOTING.md § Vault Sealed for recovery steps
|
||||
```
|
||||
|
||||
**Secret not found:**
|
||||
```bash
|
||||
# Verify path exists
|
||||
talos list cluster
|
||||
|
||||
# Check secret name (case-sensitive)
|
||||
talos get cluster/anthropic_api_key --key anthropic_api_key # won't work
|
||||
talos get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY # correct
|
||||
```
|
||||
|
||||
**vsource not expanding secrets:**
|
||||
```bash
|
||||
# Verify .env has empty value
|
||||
grep ANTHROPIC_API_KEY .env
|
||||
# → Should be: ANTHROPIC_API_KEY= (empty, not a value)
|
||||
|
||||
# Verify Vault is accessible
|
||||
talos get cluster/ANTHROPIC_API_KEY --key ANTHROPIC_API_KEY
|
||||
# → Should return secret
|
||||
|
||||
# Run vsource explicitly
|
||||
vsource .env
|
||||
echo $ANTHROPIC_API_KEY # should be populated
|
||||
```
|
||||
|
||||
See `/TROUBLESHOOTING.md` for full incident guide.
|
||||
Reference in New Issue
Block a user