k8s/storage: add minio s3 with 3-way replication and oidc

- MinIO 3-node site replication (az-a/b/c) - S3 backend for Loki chunks (10-day retention) - OIDC integration with Authentik - envFrom for secret injection
2026-07-11 19:16:56 -07:00
parent 11c26f3f29
commit 36aea89e47
15 changed files with 1014 additions and 0 deletions
@@ -0,0 +1,137 @@
+# MinIO CRUD Example (Go)
+
+A minimal Go program that exercises the cluster's object storage through the
+**universal storage frontend** — `minio.storage.svc.cluster.local:9000` — the
+single DNS name that load-balances across both node-pinned MinIO instances
+(`minio-az-a` on talos-cp-1, `minio-az-b` on talos-worker-1).
+
+It runs a full CRUD cycle with a random text file:
+
+| Step | S3 call | What it proves |
+|------|---------|----------------|
+| Ensure bucket | `BucketExists` / `MakeBucket` | bucket `crud-test` exists (idempotent) |
+| **C**reate | `PutObject` | write path through the frontend |
+| **R**ead | `GetObject` + byte compare | content round-trips intact |
+| **U**pdate | `PutObject` (overwrite) | S3 update semantics (objects are replaced, not edited) |
+| List | `ListObjects` prefix `demo/` | enumeration |
+| **D**elete | `RemoveObject` + `StatObject` | object gone (`NoSuchKey` confirmed) |
+
+Every operation emits `[SERVICE_METRIC] s3.<op>.latency_ms=<n> ms`; any failure
+emits `[APP_METRIC] ERROR s3.<op> failed ... | trace=...` and exits non-zero.
+
+## Run it
+
+```bash
+# 1. expose the frontend locally (leave running in another terminal)
+kubectl port-forward svc/minio -n storage 9000:9000
+
+# 2. credentials — same root creds used by both MinIO sites
+source logging/.env        # exports MINIO_ROOT_USER / MINIO_ROOT_PASSWORD
+
+# 3. run
+cd storage/test
+go mod tidy && go run .
+```
+
+In-cluster (e.g. from a Job), skip the port-forward and set
+`MINIO_ENDPOINT=minio.storage.svc.cluster.local:9000`.
+
+Expected output:
+
+```
+[SERVICE_METRIC] s3.ensure_bucket.latency_ms=145 ms
+[SERVICE_METRIC] s3.put.latency_ms=19 ms
+created crud-test/demo/<unix-ts>.txt (256 bytes of random text)
+[SERVICE_METRIC] s3.get.latency_ms=9 ms
+read back and verified content
+[SERVICE_METRIC] s3.update.latency_ms=70 ms
+updated (overwrote) object
+  demo/<unix-ts>.txt  140 bytes  <timestamp>
+[SERVICE_METRIC] s3.list.latency_ms=11 ms
+[SERVICE_METRIC] s3.delete.latency_ms=68 ms
+deleted and verified gone — CRUD cycle complete
+```
+
+## Validating each state with kubectl
+
+The program verifies itself in-process (read-back compare, post-delete stat),
+but every state is also independently observable from outside with `kubectl`.
+The helper below drops you into a throwaway `mc` shell wired to both sites —
+all subsequent checks use it:
+
+```bash
+source logging/.env
+kubectl run mc-shell --rm -it --restart=Never --image=minio/mc -n storage \
+  --env="U=$MINIO_ROOT_USER" --env="P=$MINIO_ROOT_PASSWORD" \
+  --command -- /bin/sh -c '
+mc alias set front http://minio.storage.svc.cluster.local:9000 "$U" "$P"
+mc alias set az-a  http://minio-az-a.storage.svc.cluster.local:9000 "$U" "$P"
+mc alias set az-b  http://minio-az-b.storage.svc.cluster.local:9000 "$U" "$P"
+exec /bin/sh'
+```
+
+> The demo deletes its object at the end, so to inspect the CREATE/UPDATE
+> states at your own pace, comment out the `// DELETE` block in `main.go`
+> and re-run (the delete is idempotent to re-apply later).
+
+**0. Frontend is healthy (before running anything)**
+
+```bash
+kubectl get endpoints minio -n storage          # expect TWO pod IPs on :9000
+kubectl get pods -n storage -o wide             # az-a on talos-cp-1, az-b on talos-worker-1
+```
+
+**1. Bucket created** — and replicated to BOTH sites
+
+```bash
+# inside mc-shell — the bucket must appear on each site individually
+mc ls az-a | grep crud-test
+mc ls az-b | grep crud-test     # proves site replication propagated it
+```
+
+**2. Object created (CREATE)** — 256 bytes, present on both nodes
+
+```bash
+mc ls az-a/crud-test/demo/      # <ts>.txt, 256 B
+mc ls az-b/crud-test/demo/      # same object, replicated (allow ~seconds of lag)
+mc cat front/crud-test/demo/<ts>.txt   # the random text itself
+```
+
+**3. Object updated (UPDATE)** — size changed 256 → 140 bytes, content starts with `UPDATED ---`
+
+```bash
+mc stat az-a/crud-test/demo/<ts>.txt   # Size: 140 B, fresh LastModified
+mc cat  az-b/crud-test/demo/<ts>.txt | head -1    # "UPDATED ---" (replicated overwrite)
+```
+
+**4. Object deleted (DELETE)** — gone from both sites
+
+```bash
+mc ls az-a/crud-test/demo/      # empty
+mc ls az-b/crud-test/demo/      # empty — deletes replicate too
+mc stat front/crud-test/demo/<ts>.txt   # error: Object does not exist
+```
+
+**5. Replication layer itself**
+
+```bash
+# inside mc-shell
+mc admin replicate status az-a   # buckets/policies/users "in sync"
+```
+
+**6. Storage layer under it**
+
+```bash
+kubectl get volumes.longhorn.io -n longhorn-system   # both volumes attached / healthy
+```
+
+## Troubleshooting
+
+- `connection refused` on localhost:9000 → the port-forward isn't running.
+- `[APP_METRIC] ERROR config missing` → `source logging/.env` first.
+- Object visible on az-a but not az-b → check `mc admin replicate status az-a`;
+  replication is near-synchronous, not instant. Persistent divergence:
+  `mc admin replicate resync start az-a az-b`.
+- Frontend has one endpoint instead of two → a MinIO pod is unready;
+  `kubectl describe pod -n storage <pod>`. Traffic still flows via the
+  surviving pod (that's the failover design — see `minio_migration.html`).