k8s/services: add ingress networking portainer llm and project guides

- Nginx ingress + TLS termination (homelab-ca) - Portainer container UI - CoreDNS internal DNS rewrites - DuckDNS DDNS updater - Ollama LLM inference - 8 project-usage guides (team reference)
2026-08-18 15:08:00 -07:00
parent 69ad5c371c
commit 61be24e27f
26 changed files with 3280 additions and 0 deletions
@@ -0,0 +1,162 @@
+# MinIO S3-Compatible Object Storage
+
+**Endpoint:** `https://minio.riotpiao.homelab.com` (console)  
+**API:** `minio.storage.svc.cluster.local:9000` (cluster-internal)  
+**Namespace:** `storage`
+
+## When to Use
+
+- **File uploads** — Images, documents, backups
+- **Log backend** — Loki chunks storage
+- **Vault unsealing** — Store unseal keys
+- **CI/CD artifacts** — Build outputs, Docker layers cache
+
+## Quick Start
+
+**1. Access MinIO console:**
+```bash
+# Via browser: https://minio.riotpiao.homelab.com
+# Credentials: MINIO_ROOT_USER / MINIO_ROOT_PASSWORD (from .env)
+
+# Or port-forward
+make pf-minio  # localhost:9001
+```
+
+**2. Create bucket:**
+```bash
+# Via AWS CLI
+export AWS_ACCESS_KEY_ID=$MINIO_ROOT_USER
+export AWS_SECRET_ACCESS_KEY=$MINIO_ROOT_PASSWORD
+
+aws s3 mb s3://my-bucket \
+  --endpoint-url https://minio.riotpiao.homelab.com \
+  --region homelab
+
+# Or via console UI: Click "Create Bucket"
+```
+
+**3. Upload file:**
+```bash
+aws s3 cp /path/to/file.txt s3://my-bucket/ \
+  --endpoint-url https://minio.storage.svc.cluster.local:9000 \
+  --use-path-style
+```
+
+**4. List buckets:**
+```bash
+aws s3 ls --endpoint-url https://minio.storage.svc.cluster.local:9000
+```
+
+## Configuration
+
+| Key | Value |
+|-----|-------|
+| Access key | `MINIO_ROOT_USER` (from .env) |
+| Secret key | `MINIO_ROOT_PASSWORD` (from .env) |
+| Cluster API | `minio.storage.svc.cluster.local:9000` |
+| Console port | `9001` |
+| Replication | 3-node site-replication (az-a ↔ az-b ↔ az-c) |
+| Buckets (system) | `loki-chunks`, `loki-ruler`, `vault-backups` |
+
+## Common Patterns
+
+**Loki log backend (auto-configured):**
+```yaml
+# k8s/logging/loki-values.yaml
+loki:
+  storage:
+    s3:
+      endpoint: minio.storage.svc.cluster.local:9000
+      buckets: loki-chunks
+      secretAccessKey: $MINIO_ROOT_PASSWORD
+      accessKeyId: $MINIO_ROOT_USER
+```
+
+**Application usage (Go/Python/Node):**
+```go
+import "github.com/minio/minio-go/v7"
+
+client, _ := minio.New("minio.storage.svc.cluster.local:9000", &minio.Options{
+  Creds:  credentials.NewStaticV4(os.Getenv("MINIO_ROOT_USER"), os.Getenv("MINIO_ROOT_PASSWORD"), ""),
+  Secure: false,  // cluster-internal (no TLS)
+})
+
+// Upload
+client.FPutObject(ctx, "my-bucket", "file.txt", "/path/to/file.txt", minio.PutObjectOptions{})
+
+// Download
+client.FGetObject(ctx, "my-bucket", "file.txt", "/tmp/file.txt", minio.GetObjectOptions{})
+```
+
+**Vault backup bucket:**
+```bash
+# Vault stores unseal keys in s3://vault-backups
+# Auto-managed by helmfile; no manual action needed
+```
+
+## Monitoring
+
+**Grafana dashboard:** `svc-minio`
+
+**Key metrics:**
+- `minio_disk_drive_free_bytes` — available space
+- `minio_bucket_usage_object_count` — objects per bucket
+- `minio_bucket_usage_total_bytes` — total size per bucket
+
+**Site replication status:**
+```bash
+# Port-forward to MinIO pod
+k port-forward -n storage pod/minio-0 9000:9000 &
+
+# Check replication
+mc alias set local http://localhost:9000 $MINIO_ROOT_USER $MINIO_ROOT_PASSWORD
+mc admin replicate status local
+```
+
+## Authentication (Cluster-Internal)
+
+**From pods (cluster-internal):**
+```bash
+# Use credentials from Secret or env var
+export MINIO_ENDPOINT=minio.storage.svc.cluster.local:9000
+export MINIO_ACCESS_KEY=$MINIO_ROOT_USER
+export MINIO_SECRET_KEY=$MINIO_ROOT_PASSWORD
+aws s3 ls --endpoint-url http://$MINIO_ENDPOINT --use-path-style
+```
+
+**External access (HTTPS via Ingress):**
+```bash
+# Console: https://minio.riotpiao.homelab.com (port 9001)
+# API: Use AWS CLI with --endpoint-url https://minio.riotpiao.homelab.com:9000
+```
+
+## Troubleshooting
+
+**Bucket creation fails:**
+```bash
+# Check MinIO pod logs
+k logs -n storage pod/minio-0 | grep -i error
+
+# Verify storage space
+k get pvc -n storage
+```
+
+**Site replication lag:**
+```bash
+# Check if all 3 nodes are healthy
+k get pods -n storage -l app=minio
+
+# If one node is down, site replication queues changes (eventually consistent)
+```
+
+**Access denied:**
+```bash
+# Verify credentials in .env
+echo $MINIO_ROOT_USER $MINIO_ROOT_PASSWORD
+
+# If credentials rotated, update Secret
+k patch secret -n storage minio-root-credentials \
+  --type merge -p '{"stringData":{"MINIO_ROOT_PASSWORD":"newpass"}}'
+```
+
+See `/TROUBLESHOOTING.md` for full incident guide.