Story Crater Bot
36aea89e47
- MinIO 3-node site replication (az-a/b/c) - S3 backend for Loki chunks (10-day retention) - OIDC integration with Authentik - envFrom for secret injection
138 lines
5.0 KiB
Markdown
138 lines
5.0 KiB
Markdown
# MinIO CRUD Example (Go)
|
|
|
|
A minimal Go program that exercises the cluster's object storage through the
|
|
**universal storage frontend** — `minio.storage.svc.cluster.local:9000` — the
|
|
single DNS name that load-balances across both node-pinned MinIO instances
|
|
(`minio-az-a` on talos-cp-1, `minio-az-b` on talos-worker-1).
|
|
|
|
It runs a full CRUD cycle with a random text file:
|
|
|
|
| Step | S3 call | What it proves |
|
|
|------|---------|----------------|
|
|
| Ensure bucket | `BucketExists` / `MakeBucket` | bucket `crud-test` exists (idempotent) |
|
|
| **C**reate | `PutObject` | write path through the frontend |
|
|
| **R**ead | `GetObject` + byte compare | content round-trips intact |
|
|
| **U**pdate | `PutObject` (overwrite) | S3 update semantics (objects are replaced, not edited) |
|
|
| List | `ListObjects` prefix `demo/` | enumeration |
|
|
| **D**elete | `RemoveObject` + `StatObject` | object gone (`NoSuchKey` confirmed) |
|
|
|
|
Every operation emits `[SERVICE_METRIC] s3.<op>.latency_ms=<n> ms`; any failure
|
|
emits `[APP_METRIC] ERROR s3.<op> failed ... | trace=...` and exits non-zero.
|
|
|
|
## Run it
|
|
|
|
```bash
|
|
# 1. expose the frontend locally (leave running in another terminal)
|
|
kubectl port-forward svc/minio -n storage 9000:9000
|
|
|
|
# 2. credentials — same root creds used by both MinIO sites
|
|
source logging/.env # exports MINIO_ROOT_USER / MINIO_ROOT_PASSWORD
|
|
|
|
# 3. run
|
|
cd storage/test
|
|
go mod tidy && go run .
|
|
```
|
|
|
|
In-cluster (e.g. from a Job), skip the port-forward and set
|
|
`MINIO_ENDPOINT=minio.storage.svc.cluster.local:9000`.
|
|
|
|
Expected output:
|
|
|
|
```
|
|
[SERVICE_METRIC] s3.ensure_bucket.latency_ms=145 ms
|
|
[SERVICE_METRIC] s3.put.latency_ms=19 ms
|
|
created crud-test/demo/<unix-ts>.txt (256 bytes of random text)
|
|
[SERVICE_METRIC] s3.get.latency_ms=9 ms
|
|
read back and verified content
|
|
[SERVICE_METRIC] s3.update.latency_ms=70 ms
|
|
updated (overwrote) object
|
|
demo/<unix-ts>.txt 140 bytes <timestamp>
|
|
[SERVICE_METRIC] s3.list.latency_ms=11 ms
|
|
[SERVICE_METRIC] s3.delete.latency_ms=68 ms
|
|
deleted and verified gone — CRUD cycle complete
|
|
```
|
|
|
|
## Validating each state with kubectl
|
|
|
|
The program verifies itself in-process (read-back compare, post-delete stat),
|
|
but every state is also independently observable from outside with `kubectl`.
|
|
The helper below drops you into a throwaway `mc` shell wired to both sites —
|
|
all subsequent checks use it:
|
|
|
|
```bash
|
|
source logging/.env
|
|
kubectl run mc-shell --rm -it --restart=Never --image=minio/mc -n storage \
|
|
--env="U=$MINIO_ROOT_USER" --env="P=$MINIO_ROOT_PASSWORD" \
|
|
--command -- /bin/sh -c '
|
|
mc alias set front http://minio.storage.svc.cluster.local:9000 "$U" "$P"
|
|
mc alias set az-a http://minio-az-a.storage.svc.cluster.local:9000 "$U" "$P"
|
|
mc alias set az-b http://minio-az-b.storage.svc.cluster.local:9000 "$U" "$P"
|
|
exec /bin/sh'
|
|
```
|
|
|
|
> The demo deletes its object at the end, so to inspect the CREATE/UPDATE
|
|
> states at your own pace, comment out the `// DELETE` block in `main.go`
|
|
> and re-run (the delete is idempotent to re-apply later).
|
|
|
|
**0. Frontend is healthy (before running anything)**
|
|
|
|
```bash
|
|
kubectl get endpoints minio -n storage # expect TWO pod IPs on :9000
|
|
kubectl get pods -n storage -o wide # az-a on talos-cp-1, az-b on talos-worker-1
|
|
```
|
|
|
|
**1. Bucket created** — and replicated to BOTH sites
|
|
|
|
```bash
|
|
# inside mc-shell — the bucket must appear on each site individually
|
|
mc ls az-a | grep crud-test
|
|
mc ls az-b | grep crud-test # proves site replication propagated it
|
|
```
|
|
|
|
**2. Object created (CREATE)** — 256 bytes, present on both nodes
|
|
|
|
```bash
|
|
mc ls az-a/crud-test/demo/ # <ts>.txt, 256 B
|
|
mc ls az-b/crud-test/demo/ # same object, replicated (allow ~seconds of lag)
|
|
mc cat front/crud-test/demo/<ts>.txt # the random text itself
|
|
```
|
|
|
|
**3. Object updated (UPDATE)** — size changed 256 → 140 bytes, content starts with `UPDATED ---`
|
|
|
|
```bash
|
|
mc stat az-a/crud-test/demo/<ts>.txt # Size: 140 B, fresh LastModified
|
|
mc cat az-b/crud-test/demo/<ts>.txt | head -1 # "UPDATED ---" (replicated overwrite)
|
|
```
|
|
|
|
**4. Object deleted (DELETE)** — gone from both sites
|
|
|
|
```bash
|
|
mc ls az-a/crud-test/demo/ # empty
|
|
mc ls az-b/crud-test/demo/ # empty — deletes replicate too
|
|
mc stat front/crud-test/demo/<ts>.txt # error: Object does not exist
|
|
```
|
|
|
|
**5. Replication layer itself**
|
|
|
|
```bash
|
|
# inside mc-shell
|
|
mc admin replicate status az-a # buckets/policies/users "in sync"
|
|
```
|
|
|
|
**6. Storage layer under it**
|
|
|
|
```bash
|
|
kubectl get volumes.longhorn.io -n longhorn-system # both volumes attached / healthy
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
- `connection refused` on localhost:9000 → the port-forward isn't running.
|
|
- `[APP_METRIC] ERROR config missing` → `source logging/.env` first.
|
|
- Object visible on az-a but not az-b → check `mc admin replicate status az-a`;
|
|
replication is near-synchronous, not instant. Persistent divergence:
|
|
`mc admin replicate resync start az-a az-b`.
|
|
- Frontend has one endpoint instead of two → a MinIO pod is unready;
|
|
`kubectl describe pod -n storage <pod>`. Traffic still flows via the
|
|
surviving pod (that's the failover design — see `minio_migration.html`).
|