201 lines
9.4 KiB
Markdown
201 lines
9.4 KiB
Markdown
# GitOps Workflow (ArgoCD) — Current Practice
|
|
|
|
**This is the actual, current deployment workflow.** Every other doc in this
|
|
directory (`infra-practice.md`, `core-cli-tools.md`, etc.) describes an older
|
|
`helmfile apply` + `core iam`/`core secrets` CLI-driven workflow that has been
|
|
retired. This file exists to be the accurate replacement for "how do I
|
|
actually deploy/change something" until those older docs get a full rewrite.
|
|
|
|
## The Loop
|
|
|
|
```
|
|
1. Edit files under k8s/
|
|
2. Validate locally (see "Validation" below — do not skip this)
|
|
3. git add -A && git commit -m "..." && git push
|
|
4. ArgoCD (automated sync + selfHeal on almost every Application) picks it
|
|
up on its next reconciliation cycle (default ~3 min, or immediately if
|
|
you trigger a refresh)
|
|
5. Confirm: kubectl -n argocd get app <name> -o jsonpath='{.status.sync.revision}'
|
|
matches your new commit hash - not just that status.sync.status says
|
|
"Synced" (a stale hook resource can hide behind an otherwise-current app,
|
|
see "ArgoCD Hooks" below)
|
|
```
|
|
|
|
There is no `helmfile apply`, no `core iam create-app`, no manual `helm
|
|
install` in the current workflow. If you find yourself reaching for any of
|
|
those, stop — figure out the ArgoCD-native equivalent instead.
|
|
|
|
## App-of-Apps Structure
|
|
|
|
```
|
|
k8s/argocd/root/homelab-root.yaml # the one Application ArgoCD bootstraps by hand
|
|
→ source: k8s/argocd/apps/ # directory of child Application manifests
|
|
00-substrate.yaml # sync-wave 0: cert-manager, ingress-nginx, CRDs
|
|
00-secrets.yaml # sync-wave 0: SOPS secrets plugin
|
|
05-networking.yaml # sync-wave... etc, ascending
|
|
10-storage-observability.yaml
|
|
20-logging.yaml
|
|
30-security.yaml # iam (Authentik, Vault)
|
|
40-data.yaml # CNPG postgres
|
|
50-messaging.yaml # Kafka/sqs
|
|
60-applications.yaml # end-user workloads (Temporal, Portainer, etc.)
|
|
```
|
|
|
|
Each child `Application` either:
|
|
- Points at a **remote Helm chart** with a **second git source** (`ref:
|
|
values`) supplying just the values file — lets you pin the chart version
|
|
independently while your values live under normal review/history, e.g.:
|
|
```yaml
|
|
spec:
|
|
sources:
|
|
- repoURL: https://example.com/helm-charts
|
|
chart: some-chart
|
|
targetRevision: "1.2.3"
|
|
helm:
|
|
valueFiles:
|
|
- $values/k8s/applications/foo/foo-values.yaml
|
|
- repoURL: https://forgejo.riotpiao.com/riotpiao.com/homelab.git
|
|
targetRevision: main
|
|
ref: values
|
|
```
|
|
- Or points at a **plain git directory** of raw manifests, optionally with
|
|
its own `kustomization.yaml`.
|
|
|
|
`argocd.argoproj.io/sync-wave: "N"` on the Application's own metadata
|
|
controls ordering — lower N syncs first. Within a single Application,
|
|
resource-level sync-wave annotations further order that app's own
|
|
resources.
|
|
|
|
## Validation Before Every Push
|
|
|
|
Pick whichever applies:
|
|
|
|
```bash
|
|
# Plain manifests, no kustomization.yaml in the directory
|
|
kubectl apply --dry-run=client -f path/to/file.yaml
|
|
|
|
# Directory has a kustomization.yaml - THIS is what ArgoCD actually builds,
|
|
# a plain `apply --dry-run` on individual files will NOT catch a resource
|
|
# missing from the resources: allowlist
|
|
kubectl kustomize path/to/dir/
|
|
|
|
# Helm-sourced Application - render against the EXACT pinned chart version,
|
|
# not whatever `helm repo` happens to have cached, and not the chart's
|
|
# current main-branch example values (schemas can differ between versions)
|
|
git clone --depth 1 <chart-repo-url> /tmp/chart-check
|
|
cd /tmp/chart-check && git fetch --tags --depth 1 && git checkout <exact-pinned-tag>
|
|
helm dependency build # if the chart has subchart dependencies
|
|
helm template . --values /path/to/your/values.yaml --namespace <ns>
|
|
```
|
|
|
|
## ArgoCD Hooks (PreSync / PostSync / Sync)
|
|
|
|
Used for one-shot Jobs that need to run at a specific point in an
|
|
Application's sync (schema migrations, secret-copy jobs, provisioning
|
|
scripts). Two properties that bite people:
|
|
|
|
1. **Ordering is PreSync → Sync (normal resources, by sync-wave) →
|
|
PostSync.** A PreSync-hooked Job that depends on a ServiceAccount/RBAC
|
|
defined as *plain* (non-hook) resources in the same Application will
|
|
deadlock — those get created during the Sync phase, which is *after*
|
|
PreSync. Symptom: the Job sits "Running" indefinitely, `kubectl
|
|
describe job` shows `FailedCreate ... serviceaccount ... not found`
|
|
repeating forever. **If your hook needs resources from its own
|
|
Application, make it PostSync, not PreSync.**
|
|
|
|
2. **Hooks are not continuously reconciled by `selfHeal`.** A completed or
|
|
failed hook Job only gets deleted+recreated (per
|
|
`hook-delete-policy: BeforeHookCreation`) during an *actual new Sync
|
|
operation* — not from ambient drift detection. If you fix a broken
|
|
hook's spec (image, RBAC, command) and push, the Application's
|
|
`status.sync.revision` can update to show "caught up" (because the
|
|
*normal*, non-hook resources genuinely match) while the *live* hook Job
|
|
is still running the old broken spec, because no fresh sync operation
|
|
actually re-executed it.
|
|
|
|
**To force it, in order of preference:**
|
|
```bash
|
|
# 1. Just delete the stuck/failed job - if a legitimate future sync
|
|
# happens for any other reason, this clears the way for it
|
|
kubectl -n <ns> delete job <hook-job-name>
|
|
|
|
# 2. If it's stuck "Terminating" (ArgoCD's hook-finalizer blocking
|
|
# deletion because the operation tracking it is itself stuck):
|
|
kubectl -n <ns> get job <hook-job-name> -o jsonpath='{.metadata.finalizers}'
|
|
kubectl -n <ns> patch job <hook-job-name> --type json \
|
|
-p '[{"op":"remove","path":"/metadata/finalizers"}]'
|
|
|
|
# 3. If deleting the job alone doesn't trigger a genuinely fresh sync
|
|
# (check: does a new job appear using your LATEST commit's spec?
|
|
# compare kubectl -n <ns> get job <name> -o jsonpath='{.spec.template.spec.containers[0].image}'
|
|
# against what your latest commit says), the Application's own
|
|
# operation state is stuck. Confirm it has no cascade finalizer first
|
|
# (Applications don't carry one by default - only delete if this is
|
|
# empty):
|
|
kubectl -n argocd get app <app-name> -o jsonpath='{.metadata.finalizers}'
|
|
# then:
|
|
kubectl -n argocd delete application <app-name>
|
|
kubectl apply -f k8s/argocd/apps/<the-file-defining-it>.yaml
|
|
```
|
|
Step 3 re-reads current git HEAD from scratch and starts a genuinely new
|
|
operation — this is the reliable way to actually pick up a hook fix when
|
|
steps 1-2 don't unstick it.
|
|
|
|
## kustomization.yaml Pitfalls
|
|
|
|
- **`resources:` is an explicit allowlist, not a directory scan.** Add a new
|
|
manifest file to a directory that has a `kustomization.yaml`, forget to
|
|
list it under `resources:`, and ArgoCD will silently never apply it — no
|
|
error, `Synced/Healthy` shown regardless. Always `kubectl kustomize
|
|
<dir>/` locally before pushing to see exactly what will be built.
|
|
- **A top-level `namespace:` transformer rewrites `metadata.namespace` on
|
|
every resource in the build**, including RBAC bindings that deliberately
|
|
target a *different* namespace (e.g. a RoleBinding granting cross-
|
|
namespace Secret access for a sync job's ServiceAccount). If any manifest
|
|
in the directory needs to live in a different namespace than the
|
|
transformer specifies, either remove the transformer (safe if every
|
|
manifest already sets its own explicit `namespace:`) or move that
|
|
manifest to its own directory/Application entirely.
|
|
|
|
## Cross-Namespace Secrets
|
|
|
|
Kubernetes Secrets are strictly namespace-scoped — a Deployment in
|
|
namespace A cannot reference a Secret living in namespace B via
|
|
`secretKeyRef`, full stop. If a Secret is generated in one namespace (e.g.
|
|
CNPG auto-generates DB credentials in its own operator namespace) but a
|
|
consumer lives in a different namespace, you need an explicit copy
|
|
mechanism. Pattern used in this repo: a small PostSync-hooked Job (see
|
|
`k8s/applications/temporal/db-secret-sync/copy-job.yaml` for a worked
|
|
example) with a dedicated ServiceAccount + ClusterRole + two RoleBindings
|
|
(one per namespace involved) that reads the source Secret and re-creates it
|
|
in the target namespace. Give this its own Application/sync-wave (earlier
|
|
than whatever consumes the copied Secret) rather than folding it into an
|
|
existing Application that has a namespace-transforming `kustomization.yaml`
|
|
(see above).
|
|
|
|
## Common Failure: Everything Shows "Unknown" Sync Status At Once
|
|
|
|
If *every* Application (not just one) suddenly shows `Unknown` sync status
|
|
simultaneously, check the Application controller's logs for the actual
|
|
`repoURL` fetch error before assuming something is wrong with any
|
|
individual app's manifests:
|
|
|
|
```bash
|
|
kubectl -n argocd logs argocd-application-controller-0 --tail=50 | grep -i "failed to list refs\|context deadline"
|
|
```
|
|
|
|
Common cause in a homelab with CoreDNS rewriting your git host to the
|
|
ingress controller for internal traffic: if `repoURL` uses a non-standard
|
|
port (`http://forgejo.example.com:3000/...`), and CoreDNS rewrites that
|
|
hostname to the ingress controller service (which only listens on 80/443),
|
|
every fetch attempt times out. Fix: use the default-port HTTPS URL
|
|
(`https://forgejo.example.com/...`) so it actually reaches the ingress
|
|
controller correctly.
|
|
|
|
## See Also
|
|
|
|
- `CLAUDE.md` § GitOps / ArgoCD Gotchas — condensed version of the above,
|
|
cross-referenced from the main project reference
|
|
- Root `TROUBLESHOOTING.md` — generic Kubernetes SRE methodology, still
|
|
applicable regardless of deployment mechanism
|