k8s/storage: add minio s3 with 3-way replication and oidc

- MinIO 3-node site replication (az-a/b/c) - S3 backend for Loki chunks (10-day retention) - OIDC integration with Authentik - envFrom for secret injection
2026-08-18 15:08:00 -07:00
parent df777fb829
commit 7da222e243
15 changed files with 1014 additions and 0 deletions
@@ -0,0 +1,137 @@
+# MinIO CRUD Example (Go)
+
+A minimal Go program that exercises the cluster's object storage through the
+**universal storage frontend** — `minio.storage.svc.cluster.local:9000` — the
+single DNS name that load-balances across both node-pinned MinIO instances
+(`minio-az-a` on talos-cp-1, `minio-az-b` on talos-worker-1).
+
+It runs a full CRUD cycle with a random text file:
+
+| Step | S3 call | What it proves |
+|------|---------|----------------|
+| Ensure bucket | `BucketExists` / `MakeBucket` | bucket `crud-test` exists (idempotent) |
+| **C**reate | `PutObject` | write path through the frontend |
+| **R**ead | `GetObject` + byte compare | content round-trips intact |
+| **U**pdate | `PutObject` (overwrite) | S3 update semantics (objects are replaced, not edited) |
+| List | `ListObjects` prefix `demo/` | enumeration |
+| **D**elete | `RemoveObject` + `StatObject` | object gone (`NoSuchKey` confirmed) |
+
+Every operation emits `[SERVICE_METRIC] s3.<op>.latency_ms=<n> ms`; any failure
+emits `[APP_METRIC] ERROR s3.<op> failed ... | trace=...` and exits non-zero.
+
+## Run it
+
+```bash
+# 1. expose the frontend locally (leave running in another terminal)
+kubectl port-forward svc/minio -n storage 9000:9000
+
+# 2. credentials — same root creds used by both MinIO sites
+source logging/.env        # exports MINIO_ROOT_USER / MINIO_ROOT_PASSWORD
+
+# 3. run
+cd storage/test
+go mod tidy && go run .
+```
+
+In-cluster (e.g. from a Job), skip the port-forward and set
+`MINIO_ENDPOINT=minio.storage.svc.cluster.local:9000`.
+
+Expected output:
+
+```
+[SERVICE_METRIC] s3.ensure_bucket.latency_ms=145 ms
+[SERVICE_METRIC] s3.put.latency_ms=19 ms
+created crud-test/demo/<unix-ts>.txt (256 bytes of random text)
+[SERVICE_METRIC] s3.get.latency_ms=9 ms
+read back and verified content
+[SERVICE_METRIC] s3.update.latency_ms=70 ms
+updated (overwrote) object
+  demo/<unix-ts>.txt  140 bytes  <timestamp>
+[SERVICE_METRIC] s3.list.latency_ms=11 ms
+[SERVICE_METRIC] s3.delete.latency_ms=68 ms
+deleted and verified gone — CRUD cycle complete
+```
+
+## Validating each state with kubectl
+
+The program verifies itself in-process (read-back compare, post-delete stat),
+but every state is also independently observable from outside with `kubectl`.
+The helper below drops you into a throwaway `mc` shell wired to both sites —
+all subsequent checks use it:
+
+```bash
+source logging/.env
+kubectl run mc-shell --rm -it --restart=Never --image=minio/mc -n storage \
+  --env="U=$MINIO_ROOT_USER" --env="P=$MINIO_ROOT_PASSWORD" \
+  --command -- /bin/sh -c '
+mc alias set front http://minio.storage.svc.cluster.local:9000 "$U" "$P"
+mc alias set az-a  http://minio-az-a.storage.svc.cluster.local:9000 "$U" "$P"
+mc alias set az-b  http://minio-az-b.storage.svc.cluster.local:9000 "$U" "$P"
+exec /bin/sh'
+```
+
+> The demo deletes its object at the end, so to inspect the CREATE/UPDATE
+> states at your own pace, comment out the `// DELETE` block in `main.go`
+> and re-run (the delete is idempotent to re-apply later).
+
+**0. Frontend is healthy (before running anything)**
+
+```bash
+kubectl get endpoints minio -n storage          # expect TWO pod IPs on :9000
+kubectl get pods -n storage -o wide             # az-a on talos-cp-1, az-b on talos-worker-1
+```
+
+**1. Bucket created** — and replicated to BOTH sites
+
+```bash
+# inside mc-shell — the bucket must appear on each site individually
+mc ls az-a | grep crud-test
+mc ls az-b | grep crud-test     # proves site replication propagated it
+```
+
+**2. Object created (CREATE)** — 256 bytes, present on both nodes
+
+```bash
+mc ls az-a/crud-test/demo/      # <ts>.txt, 256 B
+mc ls az-b/crud-test/demo/      # same object, replicated (allow ~seconds of lag)
+mc cat front/crud-test/demo/<ts>.txt   # the random text itself
+```
+
+**3. Object updated (UPDATE)** — size changed 256 → 140 bytes, content starts with `UPDATED ---`
+
+```bash
+mc stat az-a/crud-test/demo/<ts>.txt   # Size: 140 B, fresh LastModified
+mc cat  az-b/crud-test/demo/<ts>.txt | head -1    # "UPDATED ---" (replicated overwrite)
+```
+
+**4. Object deleted (DELETE)** — gone from both sites
+
+```bash
+mc ls az-a/crud-test/demo/      # empty
+mc ls az-b/crud-test/demo/      # empty — deletes replicate too
+mc stat front/crud-test/demo/<ts>.txt   # error: Object does not exist
+```
+
+**5. Replication layer itself**
+
+```bash
+# inside mc-shell
+mc admin replicate status az-a   # buckets/policies/users "in sync"
+```
+
+**6. Storage layer under it**
+
+```bash
+kubectl get volumes.longhorn.io -n longhorn-system   # both volumes attached / healthy
+```
+
+## Troubleshooting
+
+- `connection refused` on localhost:9000 → the port-forward isn't running.
+- `[APP_METRIC] ERROR config missing` → `source logging/.env` first.
+- Object visible on az-a but not az-b → check `mc admin replicate status az-a`;
+  replication is near-synchronous, not instant. Persistent divergence:
+  `mc admin replicate resync start az-a az-b`.
+- Frontend has one endpoint instead of two → a MinIO pod is unready;
+  `kubectl describe pod -n storage <pod>`. Traffic still flows via the
+  surviving pod (that's the failover design — see `minio_migration.html`).
@@ -0,0 +1,23 @@
+module homelab/storage/test
+
+go 1.22
+
+require github.com/minio/minio-go/v7 v7.0.80
+
+require (
+	github.com/davecgh/go-spew v1.1.1 // indirect
+	github.com/dustin/go-humanize v1.0.1 // indirect
+	github.com/go-ini/ini v1.67.0 // indirect
+	github.com/goccy/go-json v0.10.3 // indirect
+	github.com/google/uuid v1.6.0 // indirect
+	github.com/klauspost/compress v1.17.11 // indirect
+	github.com/klauspost/cpuid/v2 v2.2.8 // indirect
+	github.com/minio/md5-simd v1.1.2 // indirect
+	github.com/pmezard/go-difflib v1.0.0 // indirect
+	github.com/rs/xid v1.6.0 // indirect
+	golang.org/x/crypto v0.28.0 // indirect
+	golang.org/x/net v0.30.0 // indirect
+	golang.org/x/sys v0.26.0 // indirect
+	golang.org/x/text v0.19.0 // indirect
+	gopkg.in/yaml.v3 v3.0.1 // indirect
+)
@@ -0,0 +1,37 @@
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
+github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
+github.com/go-ini/ini v1.67.0 h1:z6ZrTEZqSWOTyH2FlglNbNgARyHG8oLW9gMELqKr06A=
+github.com/go-ini/ini v1.67.0/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8=
+github.com/goccy/go-json v0.10.3 h1:KZ5WoDbxAIgm2HNbYckL0se1fHD6rz5j4ywS6ebzDqA=
+github.com/goccy/go-json v0.10.3/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
+github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
+github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/klauspost/compress v1.17.11 h1:In6xLpyWOi1+C7tXUUWv2ot1QvBjxevKAaI6IXrJmUc=
+github.com/klauspost/compress v1.17.11/go.mod h1:pMDklpSncoRMuLFrf1W9Ss9KT+0rH90U12bZKk7uwG0=
+github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
+github.com/klauspost/cpuid/v2 v2.2.8 h1:+StwCXwm9PdpiEkPyzBXIy+M9KUb4ODm0Zarf1kS5BM=
+github.com/klauspost/cpuid/v2 v2.2.8/go.mod h1:Lcz8mBdAVJIBVzewtcLocK12l3Y+JytZYpaMropDUws=
+github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
+github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
+github.com/minio/minio-go/v7 v7.0.80 h1:2mdUHXEykRdY/BigLt3Iuu1otL0JTogT0Nmltg0wujk=
+github.com/minio/minio-go/v7 v7.0.80/go.mod h1:84gmIilaX4zcvAWWzJ5Z1WI5axN+hAbM5w25xf8xvC0=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU=
+github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
+github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
+github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
+golang.org/x/crypto v0.28.0 h1:GBDwsMXVQi34v5CCYUm2jkJvu4cbtru2U4TN2PSyQnw=
+golang.org/x/crypto v0.28.0/go.mod h1:rmgy+3RHxRZMyY0jjAJShp2zgEdOqj2AO7U0pYmeQ7U=
+golang.org/x/net v0.30.0 h1:AcW1SDZMkb8IpzCdQUaIq2sP4sZ4zw+55h6ynffypl4=
+golang.org/x/net v0.30.0/go.mod h1:2wGyMJ5iFasEhkwi13ChkO/t1ECNC4X4eBKkVFyYFlU=
+golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.26.0 h1:KHjCJyddX0LoSTb3J+vWpupP9p0oznkqVk/IfjymZbo=
+golang.org/x/sys v0.26.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
+golang.org/x/text v0.19.0 h1:kTxAhCbGbxhK0IwgSKiMO5awPoDQ0RpfiVYBfK860YM=
+golang.org/x/text v0.19.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
@@ -0,0 +1,152 @@
+// CRUD demo against the universal MinIO storage frontend
+// (minio.storage.svc.cluster.local:9000).
+//
+// Run from outside the cluster via a port-forward:
+//
+//	kubectl port-forward svc/minio -n storage 9000:9000 &
+//	source logging/.env
+//	cd storage/test && go mod tidy && go run .
+//
+// In-cluster, set MINIO_ENDPOINT=minio.storage.svc.cluster.local:9000.
+//
+// Every S3 call emits [SERVICE_METRIC] op latency; every failure emits
+// [APP_METRIC] ERROR with context and aborts (no silent catches).
+package main
+
+import (
+	"bytes"
+	"context"
+	"fmt"
+	"io"
+	"math/rand"
+	"os"
+	"time"
+
+	"github.com/minio/minio-go/v7"
+	"github.com/minio/minio-go/v7/pkg/credentials"
+)
+
+const bucket = "crud-test"
+
+func getenv(key, fallback string) string {
+	if v := os.Getenv(key); v != "" {
+		return v
+	}
+	return fallback
+}
+
+// timed wraps an S3 operation: emits a serviceMetric on success,
+// an applicationMetric and exit(1) on failure.
+func timed(op string, fn func() error) {
+	start := time.Now()
+	if err := fn(); err != nil {
+		fmt.Printf("[APP_METRIC] ERROR s3.%s failed bucket=%s | trace=%v\n", op, bucket, err)
+		os.Exit(1)
+	}
+	fmt.Printf("[SERVICE_METRIC] s3.%s.latency_ms=%d ms\n", op, time.Since(start).Milliseconds())
+}
+
+func randomText(n int) []byte {
+	const letters = "abcdefghijklmnopqrstuvwxyz \n"
+	b := make([]byte, n)
+	for i := range b {
+		b[i] = letters[rand.Intn(len(letters))]
+	}
+	return b
+}
+
+func main() {
+	endpoint := getenv("MINIO_ENDPOINT", "localhost:9000")
+	user := os.Getenv("MINIO_ROOT_USER")
+	pass := os.Getenv("MINIO_ROOT_PASSWORD")
+	if user == "" || pass == "" {
+		fmt.Println("[APP_METRIC] ERROR config missing | trace=MINIO_ROOT_USER / MINIO_ROOT_PASSWORD not set (source storage/.env)")
+		os.Exit(1)
+	}
+
+	ctx := context.Background()
+	client, err := minio.New(endpoint, &minio.Options{
+		Creds:  credentials.NewStaticV4(user, pass, ""),
+		Secure: false, // in-cluster traffic, no TLS
+	})
+	if err != nil {
+		fmt.Printf("[APP_METRIC] ERROR s3.connect failed endpoint=%s | trace=%v\n", endpoint, err)
+		os.Exit(1)
+	}
+
+	key := fmt.Sprintf("demo/%d.txt", time.Now().Unix())
+	original := randomText(256)
+	updated := append([]byte("UPDATED ---\n"), randomText(128)...)
+
+	// Ensure bucket (idempotent). Site replication propagates it to az-b.
+	timed("ensure_bucket", func() error {
+		exists, err := client.BucketExists(ctx, bucket)
+		if err != nil || exists {
+			return err
+		}
+		return client.MakeBucket(ctx, bucket, minio.MakeBucketOptions{})
+	})
+
+	// CREATE
+	timed("put", func() error {
+		_, err := client.PutObject(ctx, bucket, key,
+			bytes.NewReader(original), int64(len(original)),
+			minio.PutObjectOptions{ContentType: "text/plain"})
+		return err
+	})
+	fmt.Printf("created %s/%s (%d bytes of random text)\n", bucket, key, len(original))
+
+	// READ — and verify content round-trips
+	timed("get", func() error {
+		obj, err := client.GetObject(ctx, bucket, key, minio.GetObjectOptions{})
+		if err != nil {
+			return err
+		}
+		defer obj.Close()
+		got, err := io.ReadAll(obj)
+		if err != nil {
+			return err
+		}
+		if !bytes.Equal(got, original) {
+			return fmt.Errorf("read-back mismatch: want %d bytes, got %d", len(original), len(got))
+		}
+		return nil
+	})
+	fmt.Println("read back and verified content")
+
+	// UPDATE — S3 semantics: overwrite the object in place
+	timed("update", func() error {
+		_, err := client.PutObject(ctx, bucket, key,
+			bytes.NewReader(updated), int64(len(updated)),
+			minio.PutObjectOptions{ContentType: "text/plain"})
+		return err
+	})
+	fmt.Println("updated (overwrote) object")
+
+	// LIST the demo/ prefix
+	timed("list", func() error {
+		for obj := range client.ListObjects(ctx, bucket, minio.ListObjectsOptions{Prefix: "demo/", Recursive: true}) {
+			if obj.Err != nil {
+				return obj.Err
+			}
+			fmt.Printf("  %s  %d bytes  %s\n", obj.Key, obj.Size, obj.LastModified.Format(time.RFC3339))
+		}
+		return nil
+	})
+
+	// DELETE — and verify it is gone
+	timed("delete", func() error {
+		if err := client.RemoveObject(ctx, bucket, key, minio.RemoveObjectOptions{}); err != nil {
+			return err
+		}
+		_, err := client.StatObject(ctx, bucket, key, minio.StatObjectOptions{})
+		if err == nil {
+			return fmt.Errorf("object %s still exists after delete", key)
+		}
+		if minio.ToErrorResponse(err).Code != "NoSuchKey" {
+			return err
+		}
+		return nil
+	})
+	fmt.Println("deleted and verified gone — CRUD cycle complete")
+}