homelab/project-usage/coding-standards.md

# Coding Standards — Helm, Helmfile, Kubernetes, Shell

Conventions for contributing to homelab's Helm charts, helmfiles, Kubernetes manifests, and shell scripts. These are repo-local patterns — not Go/Rust/general standards from companion repos (core CLI, kmsvc).

---

## Helm & Helmfile Conventions

### Chart Naming & Layout

- **Local charts:** `k8s/<service>/charts/<component>/` directory structure.
- **Helm chart versions:** Always use semver ranges in helmfile releases (e.g., `~1.0`, `~10`, not floating `latest`).
  - Rationale: Predictable upgrades, avoids surprise breaking changes.

### helmfile.yaml.gotmpl Pattern

The helmfile is a **Go template**, not a shell script. Use Go template syntax for environment variable interpolation, not shell syntax.

**Correct:**
```yaml
set:
  - name: adminPassword
    value: {{ env "GRAFANA_ADMIN_PASSWORD" }}
```

**Incorrect:**
```yaml
set:
  - name: adminPassword
    value: ${GRAFANA_ADMIN_PASSWORD}  # Shell syntax — not expanded by helmfile
```

**Organization:**
- Use logical comment headers to separate sections: `# ── cert-manager ──`.
- Group related releases together.
- Use `needs:` for dependency ordering (release A waits for release B before deploying).
  - Example (from helmfile.yaml.gotmpl, lines 439–449):
    ```yaml
    - name: loki
      namespace: logging
      needs:
        - storage/minio
    
    - name: promtail
      namespace: logging
      needs:
        - logging/loki
    ```
- **Namespace declaration:** Specify namespace at the release block level, not helmfile-level default.
  ```yaml
  - name: prometheus
    namespace: monitoring
    createNamespace: true
  ```

### Values Files Pattern

- **Never hardcode secrets** in values.yaml or ConfigMap keys.
  - Store secrets in Vault via `core put cluster/KEY KEY="value"`.
  - Reference at deploy time using `{{ env "VAR" }}` in helmfile.
- **External values files:** Always use `values:` block pointing to `.yaml` files, not inline YAML.
  ```yaml
  - name: minio
    values:
      - k8s/storage/minio-values.yaml
  ```
- **Environment-specific overrides:** For complex releases (e.g., SQS), use `environments/` subdirectory with `helmfile.yaml.gotmpl` at the service level.
  - Example: `k8s/sqs/environments/homelab.yaml` (lines 1–29 show namespace, kafkaCluster, redis, managementService blocks).
  - Helmfile loads environment-specific values dynamically: `{{ .Values.kafkaCluster.nodePool.replicas }}`.

---

## Infrastructure as Code (IaC) — Single Source of Truth

**Core principle:** All infrastructure state must be declaratively managed via Terraform or Helm (via helmfile + Terraform). No ad-hoc scripts, manual kubectl, or side-by-side resource definitions.

### Terraform + Helm Division of Labor

- **Terraform manages:** 
  - Helm releases (chart + version + values)
  - Namespaces
  - StorageClasses
  - Static Kubernetes resources (RBAC, NetworkPolicies, IngressClasses)
  - Cloud infrastructure (Vault, S3 backends, secrets)
  - State persistence (S3 backend in MinIO)

- **Helmfile manages:** 
  - Chart release ordering via `needs:`
  - Environment-specific value interpolation (Go templating, not shell)
  - Hook workflows (pre/post-sync orchestration)
  - **Never use helmfile for one-off bucket creation, job runs, or manual setup** — those belong in Terraform or a documented bootstrap process

- **Kubernetes manifests (`k8s/`) manage:**
  - ArgoCD applications (single source of truth for GitOps)
  - Service definitions that ArgoCD syncs
  - **Never manage app objects (Deployments, StatefulSets) directly** — let helm + ArgoCD own them

### Anti-Pattern: Ad-Hoc Resource Creation

❌ **Bad:** Separate `minio-buckets.tf` using `aws_s3_bucket` resources + post-deploy scripts
- Split responsibility: some buckets in Terraform, others in helmfile, others manual
- State drift: unclear what's managed where
- Credential duplication: secrets in multiple places

✅ **Good:** Single source in `minio.tf` helm release:
```hcl
buckets = [
  { name = "terraform-state", policy = "none", purge = false },
  { name = "vault", policy = "none", purge = false },
  ...
]
```
- One place to define, one place to audit
- Credentials in variables + Vault, not scattered
- TF state tracks all changes

### When to Break the Rule

Only when **explicitly documented**:
- Bootstrap scripts (one-time cluster init) — commit to `scripts/` with clear "run once" warning
- Temporary debugging (never leave in git) — stash or delete before committing
- Manual steps for constraint (e.g., "create namespace before ArgoCD bootstraps") — document in `TROUBLESHOOTING.md` with rationale

---

## YAML & ConfigMap/Secret Patterns

### Secret Field Naming

**Rule: Field name in Secret = environment variable name in Vault.**

When storing a secret via `core put cluster/KEY KEY="value"`, the field name and Vault variable name must match. This ensures helmfile's `{{ env "VAR" }}` expansion works correctly.

Example (from CLAUDE.md gotcha "Field name = variable name"):
```bash
# Correct
core put cluster/MINIO_ROOT_PASSWORD MINIO_ROOT_PASSWORD="value"
core put cluster/GRAFANA_ADMIN_PASSWORD GRAFANA_ADMIN_PASSWORD="value"

# Incorrect (won't expand in helmfile)
core put cluster/MINIO_SECRET value="value"  # Field name ≠ variable name
```

Reference in helmfile (helmfile.yaml.gotmpl, lines 400–401):
```yaml
set:
  - name: rootPassword
    value: {{ env "MINIO_ROOT_PASSWORD" }}
```

### Never Use `--env` Flags in Manifests

Avoid kubectl flags like `--env KEY=value` in manifests or deployment specs. This exposes secrets in `kubectl describe` output.

**Correct:** Use Secret volumes (k8s/storage/minio-values.yaml, lines 40–42):
```yaml
envFrom:
  - secretRef:
      name: minio-oidc  # Reference a Secret, don't expose in manifest
```

**Incorrect:**
```yaml
env:
  - name: MINIO_OIDC_SECRET
    value: "sensitive-value"  # Visible in kubectl describe
```

### Namespace-First Organization

Organize manifests by namespace: `k8s/<namespace>/`

Structure per namespace:
```
k8s/storage/
  ├── minio-values.yaml
  ├── minio-bucket-init.sh
  └── (local charts if any)

k8s/monitoring/
  ├── prometheus-values.yaml
  ├── dashboards/         # ConfigMap files auto-loaded via helmfile postsync hook
  ├── servicemonitors/    # ServiceMonitor CRD instances
  └── alerts/             # PrometheusRule CRD instances
```

---

## Hook Scripts & Integration Workflows

### Pre/Post-Sync Hooks

Helmfile hooks (presync/postsync) drive setup workflows. Use **relative paths from repo root**, never absolute paths.

**Pattern (helmfile.yaml.gotmpl, lines 402–416):**
```yaml
- name: minio
  hooks:
    - events: ["presync"]
      command: bash
      args:
        - -c
        - |
          bash k8s/base/namespace-setup.sh storage
          bash k8s/storage/minio-bucket-init.sh storage loki-chunks loki-ruler
```

**Common presync tasks:**
- Create namespace (via `k8s/base/namespace-setup.sh`).
- Pre-create ConfigMaps/Secrets for the release.
- Initialize infrastructure (buckets, databases, etc.).

**Common postsync tasks:**
- Wait for operator/webhook readiness.
- Apply CRD instances (ServiceMonitor, PrometheusRule, Certificate).
- Perform post-deployment setup (cluster initialization, user creation).

Example (helmfile.yaml.gotmpl, lines 64–72):
```yaml
hooks:
  - events: ["postsync"]
    command: bash
    args:
      - -c
      - |
        kubectl rollout status deploy/cert-manager -n cert-manager --timeout=120s
        kubectl apply -f - <<'EOF'
        # ClusterIssuer and Certificate CRD instances...
        EOF
```

### ServiceMonitor Pattern

One file per service in `k8s/monitoring/servicemonitors/svc-<name>.yaml`.

**Key elements:**
- `namespaceSelector`: Match the namespace where the app runs.
- `selector.matchLabels`: Match the app label from the Deployment (e.g., `app.kubernetes.io/name: argocd-metrics`).
- `endpoints.port`: Name of the metrics port in the Service.
- `interval`: Scrape frequency (e.g., `30s`).

Example (`k8s/monitoring/servicemonitors/argocd.yaml`, lines 1–37):
```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd
  namespace: monitoring
  labels:
    release: kube-prometheus-stack  # Required: links to Prometheus release
spec:
  namespaceSelector:
    matchNames:
      - cicd  # App namespace
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-metrics  # Matches Deployment pod label
  endpoints:
    - port: metrics  # Service port name (not port number)
      interval: 30s
      scrapeTimeout: 10s
```

### PrometheusRule Pattern

One file per service in `k8s/monitoring/alerts/svc-<name>-rules.yaml`.

**Structure:**
- `groups[].name`: Logical grouping (e.g., `minio.rules`).
- `groups[].rules[].alert`: Alert name.
- `expr`: PromQL expression (5-minute windows for rate alerts).
- `for`: Duration threshold (e.g., `10m`).
- `labels.severity`: `critical`, `warning`.
- `annotations`: summary + description (use `{{ $value }}` for metric value).

Example (`k8s/monitoring/alerts/svc-minio-rules.yaml`, lines 1–47):
```yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: minio-rules
  namespace: storage
spec:
  groups:
    - name: minio.rules
      interval: 15s
      rules:
        - alert: MinIOHighErrorRate
          expr: |
            (
              sum(rate(minio_s3_requests_total{error="true"}[5m]))
              /
              sum(rate(minio_s3_requests_total[5m]))
            ) > 0.05
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "High error rate on MinIO"
            description: "{{ $value | humanizePercentage }}"
```

### Grafana Dashboard Pattern

Dashboards are stored as JSON ConfigMaps in `k8s/monitoring/dashboards/` and auto-loaded via helmfile postsync hook (see helmfile.yaml.gotmpl, lines 471–477).

**6-row template (recommended layout):**
1. **Availability:** Uptime, error rate, latency (SLO band).
2. **Resources:** CPU, memory, disk usage, network I/O.
3. **Domain metrics:** Service-specific KPIs (throughput, queue depth, cache hit rate).
4. **Logs:** Recent error logs from Loki.
5. **SLO:** SLI tracking (burn rate, error budget).
6. **Related dashboards:** Links to dependent services.

Auto-load hook (helmfile.yaml.gotmpl, lines 471–477):
```yaml
hooks:
  - events: ["postsync"]
    command: kubectl
    args:
      - apply
      - -f
      - k8s/monitoring/dashboards/
```

---

## Integration Checklist

When adding a new service to the homelab, verify the following:

- [ ] **Chart pinning:** Helm chart version pinned (`~1.0` format in helmfile, not floating `latest`).
- [ ] **Secrets management:** All secrets stored in Vault (none in values.yaml, ConfigMap, or CLI flags).
- [ ] **Metrics endpoint:** Service exports `/metrics` endpoint (Prometheus format).
- [ ] **ServiceMonitor:** Created and auto-scraped by Prometheus operator.
- [ ] **PrometheusRule:** Alert rules defined for errors, latency, SLO violations.
- [ ] **Grafana dashboard:** 6-row template auto-loaded via ConfigMap.
- [ ] **Ingress:** Rule added if external access needed (see `k8s/ingress/ingress.yaml`).
- [ ] **OIDC integration:** If UI component, integrated with Authentik (see `core iam bootstrap`).

---

## Shared/Reusable Repos & Image Publication

### When to Publish as Public GHCR

**Rule:** If a service's chart + image source lives in a separate repo (not in homelab), it **must** be published as a **public GitHub repo** under the `Riotpiaole` org.

**Rationale:**
- Local in-cluster charts are fine for infra-owned services.
- Shared/reusable service charts should be version-pinned and publicly available for:
  - Reuse across different clusters (other labs, staging, prod).
  - Consumption by Argo CD apps (CI/CD pipeline).
  - Independent evolution without tight coupling to homelab repo.

### Workflow

**Develop locally:**
1. Create companion repo (e.g., `kafaka-management-service`, `queue-operator`) in a private or public repo.
2. Include Dockerfile and Helm chart.

**Build & publish:**
1. Set up CI/CD in the companion repo (GitHub Actions).
2. Build image and push to GHCR: `ghcr.io/Riotpiaole/<repo>:<tag>`.
3. Tag release and publish chart (npm registry, GitHub releases, or OCI registry).

**Reference in homelab:**
1. Pin image tag + chart version in helmfile.yaml.gotmpl.
2. Chart `repositories` block references the public Helm repo (or OCI registry).

**Example (commit cd1c569 — SQS charts):**
```yaml
- name: management-service
  namespace: sqs
  chart: k8s/sqs/charts/management-service
  values:
    - image:
        repository: ghcr.io/Riotpiaole/kafaka-management-service
        tag: v1.0.0  # Pinned tag from public build
```

---

## Cross-References

- **CLAUDE.md gotchas:** Field name = variable name, MinIO `--env` flag exposure, helmfile template syntax (`{{ env "VAR" }}` not `${VAR}`), CNPG password templating.
- **README.md:** kubectl context setup, port-forward aliases, cluster topology.
- **USAGE.md:** IAM bootstrap, secret management, deployment recipes.