homelab/project-usage/gitops-workflow.md

GitOps Workflow (ArgoCD) — Current Practice

This is the actual, current deployment workflow. Every other doc in this directory (infra-practice.md, core-cli-tools.md, etc.) describes an older helmfile apply + core iam/core secrets CLI-driven workflow that has been retired. This file exists to be the accurate replacement for "how do I actually deploy/change something" until those older docs get a full rewrite.

The Loop

1. Edit files under k8s/
2. Validate locally (see "Validation" below — do not skip this)
3. git add -A && git commit -m "..." && git push
4. ArgoCD (automated sync + selfHeal on almost every Application) picks it
   up on its next reconciliation cycle (default ~3 min, or immediately if
   you trigger a refresh)
5. Confirm: kubectl -n argocd get app <name> -o jsonpath='{.status.sync.revision}'
   matches your new commit hash - not just that status.sync.status says
   "Synced" (a stale hook resource can hide behind an otherwise-current app,
   see "ArgoCD Hooks" below)

There is no helmfile apply, no core iam create-app, no manual helm install in the current workflow. If you find yourself reaching for any of those, stop — figure out the ArgoCD-native equivalent instead.

App-of-Apps Structure

k8s/argocd/root/homelab-root.yaml       # the one Application ArgoCD bootstraps by hand
  → source: k8s/argocd/apps/            # directory of child Application manifests
      00-substrate.yaml                 # sync-wave 0: cert-manager, ingress-nginx, CRDs
      00-secrets.yaml                   # sync-wave 0: SOPS secrets plugin
      05-networking.yaml                # sync-wave... etc, ascending
      10-storage-observability.yaml
      20-logging.yaml
      30-security.yaml                  # iam (Authentik, Vault)
      40-data.yaml                      # CNPG postgres
      50-messaging.yaml                 # Kafka/sqs
      60-applications.yaml              # end-user workloads (Temporal, Portainer, etc.)

Each child Application either:

• Points at a remote Helm chart with a second git source (ref: values) supplying just the values file — lets you pin the chart version independently while your values live under normal review/history, e.g.:

  spec:
    sources:
      - repoURL: https://example.com/helm-charts
        chart: some-chart
        targetRevision: "1.2.3"
        helm:
          valueFiles:
            - $values/k8s/applications/foo/foo-values.yaml
      - repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
        targetRevision: main
        ref: values
  

• Or points at a plain git directory of raw manifests, optionally with its own kustomization.yaml.

argocd.argoproj.io/sync-wave: "N" on the Application's own metadata controls ordering — lower N syncs first. Within a single Application, resource-level sync-wave annotations further order that app's own resources.

Validation Before Every Push

Pick whichever applies:

# Plain manifests, no kustomization.yaml in the directory
kubectl apply --dry-run=client -f path/to/file.yaml

# Directory has a kustomization.yaml - THIS is what ArgoCD actually builds,
# a plain `apply --dry-run` on individual files will NOT catch a resource
# missing from the resources: allowlist
kubectl kustomize path/to/dir/

# Helm-sourced Application - render against the EXACT pinned chart version,
# not whatever `helm repo` happens to have cached, and not the chart's
# current main-branch example values (schemas can differ between versions)
git clone --depth 1 <chart-repo-url> /tmp/chart-check
cd /tmp/chart-check && git fetch --tags --depth 1 && git checkout <exact-pinned-tag>
helm dependency build  # if the chart has subchart dependencies
helm template . --values /path/to/your/values.yaml --namespace <ns>

ArgoCD Hooks (PreSync / PostSync / Sync)

Used for one-shot Jobs that need to run at a specific point in an Application's sync (schema migrations, secret-copy jobs, provisioning scripts). Two properties that bite people:

1. Ordering is PreSync → Sync (normal resources, by sync-wave) → PostSync. A PreSync-hooked Job that depends on a ServiceAccount/RBAC defined as plain (non-hook) resources in the same Application will deadlock — those get created during the Sync phase, which is after PreSync. Symptom: the Job sits "Running" indefinitely, kubectl describe job shows FailedCreate ... serviceaccount ... not found repeating forever. If your hook needs resources from its own Application, make it PostSync, not PreSync.

2. Hooks are not continuously reconciled by selfHeal. A completed or failed hook Job only gets deleted+recreated (per hook-delete-policy: BeforeHookCreation) during an actual new Sync operation — not from ambient drift detection. If you fix a broken hook's spec (image, RBAC, command) and push, the Application's status.sync.revision can update to show "caught up" (because the normal, non-hook resources genuinely match) while the live hook Job is still running the old broken spec, because no fresh sync operation actually re-executed it.

   To force it, in order of preference:

   # 1. Just delete the stuck/failed job - if a legitimate future sync
   #    happens for any other reason, this clears the way for it
   kubectl -n <ns> delete job <hook-job-name>
   
   # 2. If it's stuck "Terminating" (ArgoCD's hook-finalizer blocking
   #    deletion because the operation tracking it is itself stuck):
   kubectl -n <ns> get job <hook-job-name> -o jsonpath='{.metadata.finalizers}'
   kubectl -n <ns> patch job <hook-job-name> --type json \
     -p '[{"op":"remove","path":"/metadata/finalizers"}]'
   
   # 3. If deleting the job alone doesn't trigger a genuinely fresh sync
   #    (check: does a new job appear using your LATEST commit's spec?
   #    compare kubectl -n <ns> get job <name> -o jsonpath='{.spec.template.spec.containers[0].image}'
   #    against what your latest commit says), the Application's own
   #    operation state is stuck. Confirm it has no cascade finalizer first
   #    (Applications don't carry one by default - only delete if this is
   #    empty):
   kubectl -n argocd get app <app-name> -o jsonpath='{.metadata.finalizers}'
   # then:
   kubectl -n argocd delete application <app-name>
   kubectl apply -f k8s/argocd/apps/<the-file-defining-it>.yaml
   

   Step 3 re-reads current git HEAD from scratch and starts a genuinely new operation — this is the reliable way to actually pick up a hook fix when steps 1-2 don't unstick it.

kustomization.yaml Pitfalls

• resources: is an explicit allowlist, not a directory scan. Add a new manifest file to a directory that has a kustomization.yaml, forget to list it under resources:, and ArgoCD will silently never apply it — no error, Synced/Healthy shown regardless. Always `kubectl kustomize /` locally before pushing to see exactly what will be built.
• A top-level namespace: transformer rewrites metadata.namespace on every resource in the build, including RBAC bindings that deliberately target a different namespace (e.g. a RoleBinding granting cross- namespace Secret access for a sync job's ServiceAccount). If any manifest in the directory needs to live in a different namespace than the transformer specifies, either remove the transformer (safe if every manifest already sets its own explicit namespace:) or move that manifest to its own directory/Application entirely.

Cross-Namespace Secrets

Kubernetes Secrets are strictly namespace-scoped — a Deployment in namespace A cannot reference a Secret living in namespace B via secretKeyRef, full stop. If a Secret is generated in one namespace (e.g. CNPG auto-generates DB credentials in its own operator namespace) but a consumer lives in a different namespace, you need an explicit copy mechanism. Pattern used in this repo: a small PostSync-hooked Job (see k8s/applications/temporal/db-secret-sync/copy-job.yaml for a worked example) with a dedicated ServiceAccount + ClusterRole + two RoleBindings (one per namespace involved) that reads the source Secret and re-creates it in the target namespace. Give this its own Application/sync-wave (earlier than whatever consumes the copied Secret) rather than folding it into an existing Application that has a namespace-transforming kustomization.yaml (see above).

Common Failure: Everything Shows "Unknown" Sync Status At Once

If every Application (not just one) suddenly shows Unknown sync status simultaneously, check the Application controller's logs for the actual repoURL fetch error before assuming something is wrong with any individual app's manifests:

kubectl -n argocd logs argocd-application-controller-0 --tail=50 | grep -i "failed to list refs\|context deadline"

Common cause in a homelab with CoreDNS rewriting your git host to the ingress controller for internal traffic: if repoURL uses a non-standard port (http://forgejo.example.com:3000/...), and CoreDNS rewrites that hostname to the ingress controller service (which only listens on 80/443), every fetch attempt times out. Fix: use the default-port HTTPS URL (https://forgejo.example.com/...) so it actually reaches the ingress controller correctly.

See Also

• CLAUDE.md § GitOps / ArgoCD Gotchas — condensed version of the above, cross-referenced from the main project reference
• Root TROUBLESHOOTING.md — generic Kubernetes SRE methodology, still applicable regardless of deployment mechanism