A1: Replace per-repo Forgejo entries with https://forgejo.riotpiao.com/rock/*
wildcard so onboarding never requires touching AppProject.
A2: Add wave -1 Application for k8s/argocd/projects/ so it syncs before
any Application references the AppProject.
Also add kustomization.yaml to k8s/argocd/projects/ to make it renderable.
Enabled by Stage 1 (A1, A2).
Kong was retired 2026-08-19, replaced by the rock/homelab-frontend Go
gateway (single /v1/chat/completions endpoint, model routed via the
request body's "model" field per API.md). Old per-model baseUrls
(/v1/ornith, /v1/reasoning, /v1/qwen) all 404 against the new gateway.
Also flipping reasoning's supportsTools to true -- confirmed working via
live test now that reasoning runs Qwen3-32B instead of DeepSeek-R1.
Ingress api/api now backs onto api-gateway:8080; the kong Application, its
Helm values, plugins and llm-routes are removed. Gateway image v0.0.0 is in
the Forgejo registry and the pull secret is in the api namespace.
Values changes were inert as a bootstrap release, so the proxy-body-size fix
never reached the live Ingress. First sync is manual — the chart owns the
Forgejo PVC.
- Runner egress: allow 192.168.1.160/32:443. forgejo.riotpiao.com resolves to
the ingress LB, inside the 192.168.1.0/24 block the NetworkPolicy denies, so
docker push hung until timeout.
- dind CA: also mount homelab-ca at /etc/docker/certs.d/forgejo.riotpiao.com/,
the path dockerd actually reads for per-registry trust.
- Pull secret: dockerconfigjson for the api namespace; /v2/ answers 401.
- AppProject: allow the Forgejo repo as a source for api-gw.
- namespace: PodSecurity privileged, needed for /dev/kvm + privileged QEMU
- storageclass: 1 replica, strict-local, WaitForFirstConsumer
- deployment: nodeSelector workload=imessage + matching NoSchedule toleration,
Recreate strategy (two QEMU procs on one qcow2 corrupts it), no readiness
probe (guest install is interactive and takes many minutes)
- services: ClusterIP only; VNC is an unauthenticated console, reach it with
port-forward, never an Ingress
- networkpolicy: default-deny, opt-in via sms-client=true on port 1234
ArgoCD directory.include uses Go filepath.Match glob syntax, not shell
brace expansion - {a,b,c} silently matched nothing, only the original 2
files stayed tracked.
homelab-ca was referenced by 6 manifests (authentik, forgejo-runner,
blackbox-exporter, management-service) as a CA trust ConfigMap but never
existed anywhere - not in git, not live in cluster. Generated a new
10-year self-signed root CA, wired it as a ClusterIssuer (cert-manager
namespace) and distributed the public cert as a ConfigMap to every
consuming namespace (iam, cicd, monitoring, sqs). Private key lives only
in the encrypted Secret. Widened cert-manager-issuers' directory include
glob rather than creating a new Application - destination.namespace is
just a fallback default on a plain directory source, not a transformer,
so it doesn't fight with each ConfigMap's own explicit namespace.
Also adds grafana-oidc secret (GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET),
same pre-existing gap as grafana-admin - was meant to come from a deleted
manual script, value already available in .env.
- Replace all forgejo.riotpiao.com repo URLs with [email protected] SSH URLs
- Enables immediate GitOps sync without waiting for Forgejo mirror setup
- Includes ingress-nginx now fully ArgoCD-managed (wave 0)
- SOPS secrets can now sync and decrypt TLS certificates
With Longhorn now running on all 3 control-plane nodes (commit be7881d),
Forgejo pods no longer need to be pinned to talos-cp-1. The gitea-shared-storage
PVC can attach on any node, and the scheduler will properly co-locate pod + volume
via WaitForFirstConsumer + 3-replica Longhorn volumes.
Removes the kubernetes.io/hostname: talos-cp-1 nodeSelector added in commit
dde4b60 (which was a workaround for single-node storage).
Adds missing CLIENT_SECRET env injection + nodeSelector constraint:
- k8s/argocd/bootstrap/forgejo.yaml: inject GITEA__oauth2__CLIENT_SECRET
from forgejo-oidc Secret (created by authentik-provision Job), and pin
pods to talos-cp-1 via nodeSelector (only node with Longhorn storage —
gitea-shared-storage PVC can't attach on cp-2/cp-3)
Root cause chain for 'Forgejo SSO not working':
1. Authentik 2026.5.5 requires explicit grant_types on OAuth2 providers
2. Old provision script never set it → all providers had grant_types=[]
3. /authorize returned 'Invalid grant_type for provider' → all SSO broken
4. Fixed in k8s/security/iam/scripts/authentik-provision.py (commit be2a56c)
+ successfully re-ran via iam-jobs Application sync
5. But Forgejo deployment still missing CLIENT_SECRET env var → no creds
6. Forgejo bootstrap App used inline valuesObject (chicken-egg with git
repo self-hosting), but missing the extraEnv block that was only in
k8s/security/ci-cd/forgejo-values.yaml → CLIENT_SECRET never injected
All 4 OAuth2 providers now have correct grant_types=['authorization_code',
'refresh_token'], Forgejo pods now have CLIENT_SECRET env, and pods are
constrained to the storage node. SSO login flow should now work end-to-end.
Root cause: pinned to temporalio/helm-charts @ 0.74.0, which uses the OLD
flat persistence schema (server.config.persistence.<store>.driver/.sql),
NOT the datastores:-wrapped schema shown in the CURRENT chart's
values/values.postgresql.yaml example (that key was introduced in a later
major version). Our old values.yaml used the datastores: key, which doesn't
exist in 0.74.0 - Helm doesn't validate unknown keys, so it was silently a
no-op. persistence.default.driver / persistence.visibility.driver stayed at
their chart default ("cassandra", with empty hosts: []) the entire time,
regardless of anything nested under datastores:.
Verified before writing this fix: cloned temporalio/helm-charts, checked out
tag temporal-0.74.0 (exact pin), ran +
against our actual values.yaml - confirmed the rendered
schema-setup Job used CASSANDRA_HOST/temporal-cassandra-tool the whole time.
Re-rendered with the corrected flat schema - zero Cassandra references,
correct postgres12 pluginName/connectAddr wired to ddb-cluster-rw.
Also fixed two compounding no-ops found the same way:
- -> real keys are schema.setup.enabled /
schema.update.enabled / schema.createDatabase.enabled (jobs.autoSetup
doesn't exist anywhere in this chart's templates or values.yaml).
- cassandra.enabled was never actually set to false (stayed at chart
default true) - now explicitly false, along with mysql/elasticsearch/
prometheus/grafana (none of which we want).
Password wiring: existingSecret: temporal-db-role + secretKey: password,
pointing at the CNPG-generated Secret - avoids storing the DB password as
plaintext in this values file. Added a new temporal-db-secret-sync
Application (sync-wave 7, one before temporal's wave 8) with a PreSync hook
Job that copies that Secret from the ddb namespace into temporal (Secrets
are namespace-scoped; CNPG creates it in ddb, but Temporal's pods run in
temporal). Deliberately a standalone directory/Application rather than
folded into temporal/'s own kustomization.yaml, which has a The Temporal CLI manages, monitors, and debugs Temporal apps. It lets you run
a local Temporal Service, start Workflow Executions, pass messages to running
Workflows, inspect state, and more.
* Start a local development service:
`temporal server start-dev`
* View help: pass `--help` to any command:
`temporal activity complete --help`
Usage:
temporal [command]
Available Commands:
activity Operate on Activity Executions
batch Manage running batch jobs
completion Generate the autocompletion script for the specified shell
config Manage config files (EXPERIMENTAL)
env Manage environments
help Help about any command
operator Manage Temporal deployments
schedule Perform operations on Schedules
server Run Temporal Server
task-queue Manage Task Queues
worker Read or update Worker state
workflow Start, list, and operate on Workflows
Flags:
--client-connect-timeout duration
The client connection timeout. 0s means no timeout.
(default 0s)
--color string
Output coloring. Accepted values: always, never, auto.
(default "auto")
--command-timeout duration
The command execution timeout. 0s means no timeout.
(default 0s)
--config-file $CONFIG_PATH/temporalio/temporal.toml
File path to read TOML config from, defaults to
$CONFIG_PATH/temporalio/temporal.toml where
`$CONFIG_PATH` is defined as `$HOME/.config` on Unix,
`$HOME/Library/Application Support` on macOS, and
`%AppData%` on Windows.
--disable-config-env
If set, disables loading environment config from
environment variables.
--disable-config-file
If set, disables loading environment config from config file.
--env ENV
Active environment name (ENV). (default "default")
--env-file $HOME/.config/temporalio/temporal.yaml
Path to environment settings file. Defaults to
$HOME/.config/temporalio/temporal.yaml.
-h, --help
help for temporal
--log-format string
Log format. Accepted values: text, json. (default "text")
--log-level string
Log level. Default is "never" for most commands and
"warn" for "server start-dev". Accepted values: debug,
info, warn, error, never. (default "never")
--no-json-shorthand-payloads
Raw payload output, even if the JSON option was used.
-o, --output string
Non-logging data output format. Accepted values: text,
json, jsonl, none. (default "text")
--profile string
Profile to use for config file.
--time-format string
Time format. Accepted values: relative, iso, raw.
(default "relative")
-v, --version
version for temporal
Use "temporal [command] --help" for more information about a command. transformer that would silently rewrite the copy-job's ddb-scoped
RoleBinding back to temporal (same class of bug just fixed in
k8s/security/iam/kustomization.yaml).
1. prometheus CRD sync failure (OutOfSync, permanently failing):
- helm.skipCrds: true on the prometheus Application - stop ArgoCD from
managing these CRDs through client-side apply (kube-prometheus-stack's
CRDs are large enough that the kubectl.kubernetes.io/last-applied-
configuration annotation exceeds etcd's 262144-byte limit on every sync).
- New prometheus-crds Application: plain git-sourced YAML (extracted via
helm show crds, committed under k8s/platform/monitoring/crds/), synced
with ServerSideApply=true. Chosen over a Helm-sourced 'CRDs only' app
because there's no clean way to ask ArgoCD's Helm source for 'render only
the crds/ directory' - a committed plain-YAML source is unambiguous.
- ServerSideApply=true can't go on the main prometheus Application: it
conflicts with managedNamespaceMetadata's forced namespace apply
('--force cannot be used with --server-side'), hence the split.
2. minio-tenant stuck OutOfSync (blocked 97+ minutes):
- minio-policy-setup PostSync hook Job was NAME:
mc alias set - set a new alias to configuration file
USAGE:
mc alias set ALIAS URL ACCESSKEY SECRETKEY
FLAGS:
--path value bucket path lookup supported by the server. Valid options are '[auto, on, off]' (default: "auto")
--api value API signature. Valid options are '[S3v4, S3v2]'
--config-dir value, -C value path to configuration folder (default: "/Users/rockliang/.mc") [$MC_CONFIG_DIR]
--quiet, -q disable progress bar display [$MC_QUIET]
--disable-pager, --dp disable mc internal pager and print to raw stdout [$MC_DISABLE_PAGER]
--no-color disable color theme [$MC_NO_COLOR]
--json enable JSON lines formatted output [$MC_JSON]
--debug enable debug output [$MC_DEBUG]
--resolve value resolves HOST[:PORT] to an IP address. Example: minio.local:9000=10.10.75.1 [$MC_RESOLVE]
--insecure disable SSL certificate verification [$MC_INSECURE]
--limit-upload value limits uploads to a maximum rate in KiB/s, MiB/s, GiB/s. (default: unlimited) [$MC_LIMIT_UPLOAD]
--limit-download value limits downloads to a maximum rate in KiB/s, MiB/s, GiB/s. (default: unlimited) [$MC_LIMIT_DOWNLOAD]
--custom-header value, -H value add custom HTTP header to the request. 'key:value' format.
--help, -h show help
EXAMPLES:
1. Add MinIO service under "myminio" alias. For security reasons turn off bash history momentarily.
$ set +o history
$ mc alias set myminio http://localhost:9000 minio minio123
$ set -o history
2. Add MinIO service under "myminio" alias, to use dns style bucket lookup. For security reasons
turn off bash history momentarily.
$ set +o history
$ mc alias set myminio http://localhost:9000 minio minio123 --api "s3v4" --path "off"
$ set -o history
3. Add Amazon S3 storage service under "mys3" alias. For security reasons turn off bash history momentarily.
$ set +o history
$ mc alias set mys3 https://s3.amazonaws.com \
BKIKJAA5BMMU2RHO6IBB V8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12
$ set -o history
4. Add Amazon S3 storage service under "mys3" alias, prompting for keys.
$ mc alias set mys3 https://s3.amazonaws.com --api "s3v4" --path "off"
Enter Access Key: BKIKJAA5BMMU2RHO6IBB
Enter Secret Key: V8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12
5. Add Amazon S3 storage service under "mys3" alias using piped keys.
$ set +o history
$ echo -e "BKIKJAA5BMMU2RHO6IBB\nV8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12" | \
mc alias set mys3 https://s3.amazonaws.com --api "s3v4" --path "off"
$ set -o history against
http://minio.storage.svc.cluster.local:9000 - stale port. The minio
Service's port now tracks requestAutoCert on the Tenant (443 when
auto-TLS is on, 80 when off - we set it to false earlier), so 9000
doesn't exist on that Service anymore and the job hung in its 'waiting
for minio...' retry loop indefinitely, blocking ArgoCD's sync operation
(PostSync hooks block the sync from completing until they succeed).
- Fixed to use minio-cluster-hl.storage.svc.cluster.local:9000 - the
headless per-pod Service, which always listens on 9000 regardless of
the Tenant's TLS mode, so this can't silently break again the same way.
Root cause of widespread 'Unknown' sync status / Skipping auto-sync across
almost every Application: CoreDNS rewrites forgejo.riotpiao.com to the nginx
ingress controller service (rewrite name forgejo.riotpiao.com -> ingress-nginx-
controller...), which only listens on 80/443, not 3000. Every git fetch from
argocd-repo-server to the :3000 repoURL was timing out (context deadline
exceeded), so ArgoCD couldn't compare desired vs live state for any app.
Fix: use https://forgejo.riotpiao.com (no port, TLS via nginx + wildcard cert)
consistent with the 'all external endpoints HTTPS' requirement. Verified git
smart-http response 200 on the new URL before committing.
- Create SOPS-encrypted cloudflared-secrets.enc.yaml with tunnel token
- Add Cloudflare vars to .env.example (CLOUDFLARE_CONNECTOR_TOKEN, ACCOUNT_ID, TUNNEL_ID, API_TOKEN)
- Document Phase 0 cloudflared-token Secret creation in BOOTSTRAP.md (manual step until CMP plugin wires it)
- Note: Cloudflare-side TCP routing (cp1.homelab -> 192.168.1.213:6443, etc.) must be configured manually in Zero Trust dashboard
Tunnel already deployed as ArgoCD Application in k8s/argocd/apps/60-applications.yaml (wave 8); this closes the missing Secret gap and documents the bootstrap path.
Story Crater Bot