# GitOps Workflow (ArgoCD) — Current Practice

**This is the actual, current deployment workflow.** Every other doc in this
directory (`infra-practice.md`, `core-cli-tools.md`, etc.) describes an older
`helmfile apply` + `core iam`/`core secrets` CLI-driven workflow that has been
retired. This file exists to be the accurate replacement for "how do I
actually deploy/change something" until those older docs get a full rewrite.

## The Loop

```
1. Edit files under k8s/
2. Validate locally (see "Validation" below — do not skip this)
3. git add -A && git commit -m "..." && git push
4. ArgoCD (automated sync + selfHeal on almost every Application) picks it
   up on its next reconciliation cycle (default ~3 min, or immediately if
   you trigger a refresh)
5. Confirm: kubectl -n argocd get app <name> -o jsonpath='{.status.sync.revision}'
   matches your new commit hash - not just that status.sync.status says
   "Synced" (a stale hook resource can hide behind an otherwise-current app,
   see "ArgoCD Hooks" below)
```

There is no `helmfile apply`, no `core iam create-app`, no manual `helm
install` in the current workflow. If you find yourself reaching for any of
those, stop — figure out the ArgoCD-native equivalent instead.

## App-of-Apps Structure

```
k8s/argocd/root/homelab-root.yaml       # the one Application ArgoCD bootstraps by hand
  → source: k8s/argocd/apps/            # directory of child Application manifests
      00-substrate.yaml                 # sync-wave 0: cert-manager, ingress-nginx, CRDs
      00-secrets.yaml                   # sync-wave 0: SOPS secrets plugin
      05-networking.yaml                # sync-wave... etc, ascending
      10-storage-observability.yaml
      20-logging.yaml
      30-security.yaml                  # iam (Authentik, Vault)
      40-data.yaml                      # CNPG postgres
      50-messaging.yaml                 # Kafka/sqs
      60-applications.yaml              # end-user workloads (Temporal, Portainer, etc.)
```

Each child `Application` either:
- Points at a **remote Helm chart** with a **second git source** (`ref:
  values`) supplying just the values file — lets you pin the chart version
  independently while your values live under normal review/history, e.g.:
  ```yaml
  spec:
    sources:
      - repoURL: https://example.com/helm-charts
        chart: some-chart
        targetRevision: "1.2.3"
        helm:
          valueFiles:
            - $values/k8s/applications/foo/foo-values.yaml
      - repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
        targetRevision: main
        ref: values
  ```
- Or points at a **plain git directory** of raw manifests, optionally with
  its own `kustomization.yaml`.

`argocd.argoproj.io/sync-wave: "N"` on the Application's own metadata
controls ordering — lower N syncs first. Within a single Application,
resource-level sync-wave annotations further order that app's own
resources.

## Validation Before Every Push

Pick whichever applies:

```bash
# Plain manifests, no kustomization.yaml in the directory
kubectl apply --dry-run=client -f path/to/file.yaml

# Directory has a kustomization.yaml - THIS is what ArgoCD actually builds,
# a plain `apply --dry-run` on individual files will NOT catch a resource
# missing from the resources: allowlist
kubectl kustomize path/to/dir/

# Helm-sourced Application - render against the EXACT pinned chart version,
# not whatever `helm repo` happens to have cached, and not the chart's
# current main-branch example values (schemas can differ between versions)
git clone --depth 1 <chart-repo-url> /tmp/chart-check
cd /tmp/chart-check && git fetch --tags --depth 1 && git checkout <exact-pinned-tag>
helm dependency build  # if the chart has subchart dependencies
helm template . --values /path/to/your/values.yaml --namespace <ns>
```

## ArgoCD Hooks (PreSync / PostSync / Sync)

Used for one-shot Jobs that need to run at a specific point in an
Application's sync (schema migrations, secret-copy jobs, provisioning
scripts). Two properties that bite people:

1. **Ordering is PreSync → Sync (normal resources, by sync-wave) →
   PostSync.** A PreSync-hooked Job that depends on a ServiceAccount/RBAC
   defined as *plain* (non-hook) resources in the same Application will
   deadlock — those get created during the Sync phase, which is *after*
   PreSync. Symptom: the Job sits "Running" indefinitely, `kubectl
   describe job` shows `FailedCreate ... serviceaccount ... not found`
   repeating forever. **If your hook needs resources from its own
   Application, make it PostSync, not PreSync.**

2. **Hooks are not continuously reconciled by `selfHeal`.** A completed or
   failed hook Job only gets deleted+recreated (per
   `hook-delete-policy: BeforeHookCreation`) during an *actual new Sync
   operation* — not from ambient drift detection. If you fix a broken
   hook's spec (image, RBAC, command) and push, the Application's
   `status.sync.revision` can update to show "caught up" (because the
   *normal*, non-hook resources genuinely match) while the *live* hook Job
   is still running the old broken spec, because no fresh sync operation
   actually re-executed it.

   **To force it, in order of preference:**
   ```bash
   # 1. Just delete the stuck/failed job - if a legitimate future sync
   #    happens for any other reason, this clears the way for it
   kubectl -n <ns> delete job <hook-job-name>

   # 2. If it's stuck "Terminating" (ArgoCD's hook-finalizer blocking
   #    deletion because the operation tracking it is itself stuck):
   kubectl -n <ns> get job <hook-job-name> -o jsonpath='{.metadata.finalizers}'
   kubectl -n <ns> patch job <hook-job-name> --type json \
     -p '[{"op":"remove","path":"/metadata/finalizers"}]'

   # 3. If deleting the job alone doesn't trigger a genuinely fresh sync
   #    (check: does a new job appear using your LATEST commit's spec?
   #    compare kubectl -n <ns> get job <name> -o jsonpath='{.spec.template.spec.containers[0].image}'
   #    against what your latest commit says), the Application's own
   #    operation state is stuck. Confirm it has no cascade finalizer first
   #    (Applications don't carry one by default - only delete if this is
   #    empty):
   kubectl -n argocd get app <app-name> -o jsonpath='{.metadata.finalizers}'
   # then:
   kubectl -n argocd delete application <app-name>
   kubectl apply -f k8s/argocd/apps/<the-file-defining-it>.yaml
   ```
   Step 3 re-reads current git HEAD from scratch and starts a genuinely new
   operation — this is the reliable way to actually pick up a hook fix when
   steps 1-2 don't unstick it.

## kustomization.yaml Pitfalls

- **`resources:` is an explicit allowlist, not a directory scan.** Add a new
  manifest file to a directory that has a `kustomization.yaml`, forget to
  list it under `resources:`, and ArgoCD will silently never apply it — no
  error, `Synced/Healthy` shown regardless. Always `kubectl kustomize
  <dir>/` locally before pushing to see exactly what will be built.
- **A top-level `namespace:` transformer rewrites `metadata.namespace` on
  every resource in the build**, including RBAC bindings that deliberately
  target a *different* namespace (e.g. a RoleBinding granting cross-
  namespace Secret access for a sync job's ServiceAccount). If any manifest
  in the directory needs to live in a different namespace than the
  transformer specifies, either remove the transformer (safe if every
  manifest already sets its own explicit `namespace:`) or move that
  manifest to its own directory/Application entirely.

## Cross-Namespace Secrets

Kubernetes Secrets are strictly namespace-scoped — a Deployment in
namespace A cannot reference a Secret living in namespace B via
`secretKeyRef`, full stop. If a Secret is generated in one namespace (e.g.
CNPG auto-generates DB credentials in its own operator namespace) but a
consumer lives in a different namespace, you need an explicit copy
mechanism. Pattern used in this repo: a small PostSync-hooked Job (see
`k8s/applications/temporal/db-secret-sync/copy-job.yaml` for a worked
example) with a dedicated ServiceAccount + ClusterRole + two RoleBindings
(one per namespace involved) that reads the source Secret and re-creates it
in the target namespace. Give this its own Application/sync-wave (earlier
than whatever consumes the copied Secret) rather than folding it into an
existing Application that has a namespace-transforming `kustomization.yaml`
(see above).

## Common Failure: Everything Shows "Unknown" Sync Status At Once

If *every* Application (not just one) suddenly shows `Unknown` sync status
simultaneously, check the Application controller's logs for the actual
`repoURL` fetch error before assuming something is wrong with any
individual app's manifests:

```bash
kubectl -n argocd logs argocd-application-controller-0 --tail=50 | grep -i "failed to list refs\|context deadline"
```

Common cause in a homelab with CoreDNS rewriting your git host to the
ingress controller for internal traffic: if `repoURL` uses a non-standard
port (`http://forgejo.example.com:3000/...`), and CoreDNS rewrites that
hostname to the ingress controller service (which only listens on 80/443),
every fetch attempt times out. Fix: use the default-port HTTPS URL
(`https://forgejo.example.com/...`) so it actually reaches the ingress
controller correctly.

## See Also

- `CLAUDE.md` § GitOps / ArgoCD Gotchas — condensed version of the above,
  cross-referenced from the main project reference
- Root `TROUBLESHOOTING.md` — generic Kubernetes SRE methodology, still
  applicable regardless of deployment mechanism