The StatefulSet database could not attach its Longhorn volume, so the branch
runs the database as a CNPG Cluster instead. This pins the Application to that
branch for the test; it returns to the default branch once the branch merges.
The homelab AppProject allowlist had no azhi repository, so an Application
pointing at ToyoGlobal/azhi-infra fails admission with 'repository not
permitted'. Adding the URL is the whole gate.
The Application itself targets the repository default branch
(harness/staging-slice), not main, and path k8s/overlays/azi-riotpiao. That
overlay renders postgres and the API into namespace azi-riotpiao.
The two Secrets the namespace needs stay out of Git: acr-pull carries the ACR
pull credential and azhi-riotpiao-app-env carries the database credentials,
the JWT secrets and the AES key.
etcd's backend file only grows: the apiserver compacts every 5m, but compaction
marks superseded revisions free inside the file, and only defrag returns that
space to the filesystem. Nothing ran defrag, so on 2026-10-07 the file reached
2,148,843,520 B, crossed the 2 GiB quota on talos-cp-1 and talos-cp-3, and etcd
raised NOSPACE. From then on etcd rejected every write cluster-wide: kubelets
could not post node status (3 nodes NotReady), apiserver writes returned 500 and
the tunnel API path returned 502. Live data was 15,943 keys, 69 MB after
compaction - the rest was dead pages. Compaction had also stalled while the
alarm was up, so dbSizeInUse read 100% of the file; defrag alone would have
reclaimed nothing. Compact must come first.
An earlier revision of this CronJob never ran, for two independent reasons, both
fixed here:
- its image quay.io/coreos/etcd ships no /bin/sh, so command: [/bin/sh, -c]
failed with exec: "/bin/sh": no such file or directory
- it existed only on a side branch while the ArgoCD application
cluster-maintenance tracks main, so the cluster never had it
Contents:
- CronJob etcd-defrag (kube-system): daily at 03:00, acts only when the UTC
epoch day is a multiple of 15. That is an exact 15-day interval; */15 in the
day-of-month field fires on the 1st, 16th and 31st, putting two runs one day
apart.
- Image bitnamilegacy/etcd:3.6.4 pinned by digest, ships bash and etcdctl 3.6.4
against the 3.6.11 servers.
- Compaction uses the MINIMUM revision across members, so a lagging member is
never compacted past its own revision. Measured 2026-10-07: talos-cp-2 sat
~1.2M entries behind the leader with a frozen raft term and needed a reboot
to rejoin.
- Defrag skips any member whose applied index lags the leader by 100k+, and
runs one member at a time since defrag pauses that member's apply loop.
- Secret etcd-defrag-creds (SOPS/ksops, kube-system) carries the etcd CA plus a
client certificate signed by the cluster's etcd CA, valid to 2036-10.
etcd's 2 GiB quota was reached on 2026-10-07 (backend 2.1 GiB, ~73 days of
churn). etcd raised NOSPACE and refused all writes, so kubelets could not post
node status: three nodes went NotReady and the tunnel origin pods on them
disappeared. Recovery was compact + defrag + alarm disarm per member.
The apiserver compacts every 5m, but compaction never shrinks the backend file -
only a defrag returns space, and nothing scheduled it. This adds:
- CronJob etcd-defrag (kube-system, Sundays 03:00): compact to the current
revision, defrag each member sequentially, disarm NOSPACE, print before/after.
- Secret etcd-defrag-creds (SOPS/ksops, k8s/argocd/secrets): etcd CA plus a
CN=etcd-defrag client cert signed by the cluster's etcd CA (expires 2036-10).
Applied by the existing apps: cluster-maintenance (directory) and 00-secrets
(ksops generator).
Adds the pi coding-agent sandbox's cluster side:
- `k8s/infra/poimen/pi-agent-workspace-pv.yaml` — static hostPath PV + PVC for the sandbox /workspace (Longhorn cannot attach new volumes in this cluster; this gives real persistence and survived a pod delete/recreate test: marker file + 38 KB session.jsonl intact, the session answered from its own history afterwards).
- `k8s/argocd/secrets/pi-sandbox-secrets.enc.yaml` — the sandbox's API-key Secret, SOPS-encrypted (recipient from .sops.yaml, payload only: `encrypted_regex ^(data|stringData)$`), registered in the ksops generator. Round-trip verified against the live Secret.
- `terraform/files/coredns/Corefile` — `rewrite name pi.riotpiao.com ingress-nginx-controller...` so the hostname resolves in-cluster.
Note: the keys currently in that Secret are the ones leaked earlier and must be rotated; re-encrypt with `sops -e -i` after rotating.Reviewed-on: #54
Co-authored-by: rock <[email protected]>
Points to riotpiao-poimen/poimen-memory.git k8s/app/ (kustomize).
ArgoCD auto-syncs Redis deployment, NetworkPolicies, ConfigMap, Service.
Image updater: digest strategy on :latest tag for rolling deployments.
Wave 7 — same wave as poimen workflow app.
ROOT CAUSE:
- tektoncd/pipeline config/ dir uses ko:// image refs (build-time placeholders)
- ArgoCD synced the raw dev manifests → InvalidImageName on all pods
- tektoncd/operator requires its own CRDs and controller (too heavy)
- Tekton has no official Helm chart
FIX:
- Vendor the pre-built release.yaml (v0.68.0) into k8s/infra/tekton/
- Point ArgoCD Application at our own repo (forgejo)
- Release contains real container images (ghcr.io/tektoncd/pipeline/*)
- Remove external tektoncd repo from AppProject (not needed)
TO UPGRADE TEKTON:
1. Download new release from github.com/tektoncd/pipeline/releases
2. Replace k8s/infra/tekton/release.yaml
3. Commit — ArgoCD syncs automatically
- paperless-ai deployment using clusterzx/paperless-ai
- LLM via local api-gateway (reasoning model, no auth - phase 3 pending)
- Paperless API token SOPS-encrypted
- Auto-tags new documents, 5min scan interval
- Adds 'ai-processed' tag to classified documents
ArgoCD couldn't decrypt secrets.yaml because it was listed as a plain
kustomize resource. Move the 3 secrets (gotify-admin, gotify-tokens,
gotify-smtp) to k8s/argocd/secrets/ as .enc.yaml files processed by
the ksops generator, matching the repo convention.
Fixes ComparisonError: 'Object Kind is missing' (SOPS ciphertext
parsed as raw YAML).
Repos transferred: homelab-frontend, kmsvc-manage, poimen, poimen-memory,
poimen-workflows, poimen-frontend. Old URLs return 301 which ArgoCD
doesn't follow.
NOT changed: container image registry paths (rock/ is correct for registry),
riotpiao.com (still under rock org).
Also adds poimen-frontend to AppProject sourceRepos allowlist.
- buildOptions field not available in ArgoCD v3.4.5
- SOPS decryption already handled by repo-server ksops plugin
- Revert to simple kustomize config
- Portfolio Application can now sync properly
- Add kustomize config with --enable-alpha-plugins to support ksops
- Allows ArgoCD to properly decrypt SOPS-encrypted files
- Fixes Image Updater compatibility with sops field in kustomization.yaml
- RBAC kustomization contains cluster-scoped (ClusterRoleBinding) and
namespace-scoped (Role/RoleBinding) resources
- Each resource has explicit metadata.namespace, so Application shouldn't
force a default namespace
- Fixes: ClusterRoleBinding gets namespace=default, causing sync failure
with 'unsupported role reference kind: ""'
- Removed dependency on external poimen.git repo
- Poimen manifests should be managed locally or via separate workflow
- Simplifies homelab GitOps to only manage homelab-owned services
- Applications now managed by kmsvc-root from kmsvc-manage.git
- Added ServerSideApply to homelab-root for proper annotation sync
- Avoids duplicate Application conflicts with Image Updater
rock
rock
poimenandrock