homelab/project-usage/coding-standards.md

Coding Standards — Helm, Helmfile, Kubernetes, Shell

Conventions for contributing to homelab's Helm charts, helmfiles, Kubernetes manifests, and shell scripts. These are repo-local patterns — not Go/Rust/general standards from companion repos (core CLI, kmsvc).

---

Helm & Helmfile Conventions

Chart Naming & Layout

• Local charts: k8s/<service>/charts/<component>/ directory structure.
• Helm chart versions: Always use semver ranges in helmfile releases (e.g., ~1.0, ~10, not floating latest).
  • Rationale: Predictable upgrades, avoids surprise breaking changes.

helmfile.yaml.gotmpl Pattern

The helmfile is a Go template, not a shell script. Use Go template syntax for environment variable interpolation, not shell syntax.

Correct:

set:
  - name: adminPassword
    value: {{ env "GRAFANA_ADMIN_PASSWORD" }}

Incorrect:

set:
  - name: adminPassword
    value: ${GRAFANA_ADMIN_PASSWORD}  # Shell syntax — not expanded by helmfile

Organization:

• Use logical comment headers to separate sections: # ── cert-manager ──.
• Group related releases together.
• Use needs: for dependency ordering (release A waits for release B before deploying).
  • Example (from helmfile.yaml.gotmpl, lines 439–449):

    - name: loki
      namespace: logging
      needs:
        - storage/minio
    
    - name: promtail
      namespace: logging
      needs:
        - logging/loki
    

• Namespace declaration: Specify namespace at the release block level, not helmfile-level default.

  - name: prometheus
    namespace: monitoring
    createNamespace: true
  

Values Files Pattern

• Never hardcode secrets in values.yaml or ConfigMap keys.
  • Store secrets in Vault via core put cluster/KEY KEY="value".
  • Reference at deploy time using {{ env "VAR" }} in helmfile.
• External values files: Always use values: block pointing to .yaml files, not inline YAML.

  - name: minio
    values:
      - k8s/storage/minio-values.yaml
  

• Environment-specific overrides: For complex releases (e.g., SQS), use environments/ subdirectory with helmfile.yaml.gotmpl at the service level.
  • Example: k8s/sqs/environments/homelab.yaml (lines 1–29 show namespace, kafkaCluster, redis, managementService blocks).
  • Helmfile loads environment-specific values dynamically: {{ .Values.kafkaCluster.nodePool.replicas }}.

---

Infrastructure as Code (IaC) — Single Source of Truth

Core principle: All infrastructure state must be declaratively managed via Terraform or Helm (via helmfile + Terraform). No ad-hoc scripts, manual kubectl, or side-by-side resource definitions.

Terraform + Helm Division of Labor

• Terraform manages:

  • Helm releases (chart + version + values)
  • Namespaces
  • StorageClasses
  • Static Kubernetes resources (RBAC, NetworkPolicies, IngressClasses)
  • Cloud infrastructure (Vault, S3 backends, secrets)
  • State persistence (S3 backend in MinIO)

• Helmfile manages:

  • Chart release ordering via needs:
  • Environment-specific value interpolation (Go templating, not shell)
  • Hook workflows (pre/post-sync orchestration)
  • Never use helmfile for one-off bucket creation, job runs, or manual setup — those belong in Terraform or a documented bootstrap process

• Kubernetes manifests (k8s/) manage:

  • ArgoCD applications (single source of truth for GitOps)
  • Service definitions that ArgoCD syncs
  • Never manage app objects (Deployments, StatefulSets) directly — let helm + ArgoCD own them

Anti-Pattern: Ad-Hoc Resource Creation

❌ Bad: Separate minio-buckets.tf using aws_s3_bucket resources + post-deploy scripts

• Split responsibility: some buckets in Terraform, others in helmfile, others manual
• State drift: unclear what's managed where
• Credential duplication: secrets in multiple places

✅ Good: Single source in minio.tf helm release:

buckets = [
  { name = "terraform-state", policy = "none", purge = false },
  { name = "vault", policy = "none", purge = false },
  ...
]

• One place to define, one place to audit
• Credentials in variables + Vault, not scattered
• TF state tracks all changes

When to Break the Rule

Only when explicitly documented:

• Bootstrap scripts (one-time cluster init) — commit to scripts/ with clear "run once" warning
• Temporary debugging (never leave in git) — stash or delete before committing
• Manual steps for constraint (e.g., "create namespace before ArgoCD bootstraps") — document in TROUBLESHOOTING.md with rationale

---

YAML & ConfigMap/Secret Patterns

Secret Field Naming

Rule: Field name in Secret = environment variable name in Vault.

When storing a secret via core put cluster/KEY KEY="value", the field name and Vault variable name must match. This ensures helmfile's {{ env "VAR" }} expansion works correctly.

Example (from CLAUDE.md gotcha "Field name = variable name"):

# Correct
core put cluster/MINIO_ROOT_PASSWORD MINIO_ROOT_PASSWORD="value"
core put cluster/GRAFANA_ADMIN_PASSWORD GRAFANA_ADMIN_PASSWORD="value"

# Incorrect (won't expand in helmfile)
core put cluster/MINIO_SECRET value="value"  # Field name ≠ variable name

Reference in helmfile (helmfile.yaml.gotmpl, lines 400–401):

set:
  - name: rootPassword
    value: {{ env "MINIO_ROOT_PASSWORD" }}

Never Use --env Flags in Manifests

Avoid kubectl flags like --env KEY=value in manifests or deployment specs. This exposes secrets in kubectl describe output.

Correct: Use Secret volumes (k8s/storage/minio-values.yaml, lines 40–42):

envFrom:
  - secretRef:
      name: minio-oidc  # Reference a Secret, don't expose in manifest

Incorrect:

env:
  - name: MINIO_OIDC_SECRET
    value: "sensitive-value"  # Visible in kubectl describe

Namespace-First Organization

Organize manifests by namespace: k8s/<namespace>/

Structure per namespace:

k8s/storage/
  ├── minio-values.yaml
  ├── minio-bucket-init.sh
  └── (local charts if any)

k8s/monitoring/
  ├── prometheus-values.yaml
  ├── dashboards/         # ConfigMap files auto-loaded via helmfile postsync hook
  ├── servicemonitors/    # ServiceMonitor CRD instances
  └── alerts/             # PrometheusRule CRD instances

---

Hook Scripts & Integration Workflows

Pre/Post-Sync Hooks

Helmfile hooks (presync/postsync) drive setup workflows. Use relative paths from repo root, never absolute paths.

Pattern (helmfile.yaml.gotmpl, lines 402–416):

- name: minio
  hooks:
    - events: ["presync"]
      command: bash
      args:
        - -c
        - |
          bash k8s/base/namespace-setup.sh storage
          bash k8s/storage/minio-bucket-init.sh storage loki-chunks loki-ruler

Common presync tasks:

• Create namespace (via k8s/base/namespace-setup.sh).
• Pre-create ConfigMaps/Secrets for the release.
• Initialize infrastructure (buckets, databases, etc.).

Common postsync tasks:

• Wait for operator/webhook readiness.
• Apply CRD instances (ServiceMonitor, PrometheusRule, Certificate).
• Perform post-deployment setup (cluster initialization, user creation).

Example (helmfile.yaml.gotmpl, lines 64–72):

hooks:
  - events: ["postsync"]
    command: bash
    args:
      - -c
      - |
        kubectl rollout status deploy/cert-manager -n cert-manager --timeout=120s
        kubectl apply -f - <<'EOF'
        # ClusterIssuer and Certificate CRD instances...
        EOF

ServiceMonitor Pattern

One file per service in k8s/monitoring/servicemonitors/svc-<name>.yaml.

Key elements:

• namespaceSelector: Match the namespace where the app runs.
• selector.matchLabels: Match the app label from the Deployment (e.g., app.kubernetes.io/name: argocd-metrics).
• endpoints.port: Name of the metrics port in the Service.
• interval: Scrape frequency (e.g., 30s).

Example (k8s/monitoring/servicemonitors/argocd.yaml, lines 1–37):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd
  namespace: monitoring
  labels:
    release: kube-prometheus-stack  # Required: links to Prometheus release
spec:
  namespaceSelector:
    matchNames:
      - cicd  # App namespace
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-metrics  # Matches Deployment pod label
  endpoints:
    - port: metrics  # Service port name (not port number)
      interval: 30s
      scrapeTimeout: 10s

PrometheusRule Pattern

One file per service in k8s/monitoring/alerts/svc-<name>-rules.yaml.

Structure:

• groups[].name: Logical grouping (e.g., minio.rules).
• groups[].rules[].alert: Alert name.
• expr: PromQL expression (5-minute windows for rate alerts).
• for: Duration threshold (e.g., 10m).
• labels.severity: critical, warning.
• annotations: summary + description (use {{ $value }} for metric value).

Example (k8s/monitoring/alerts/svc-minio-rules.yaml, lines 1–47):

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: minio-rules
  namespace: storage
spec:
  groups:
    - name: minio.rules
      interval: 15s
      rules:
        - alert: MinIOHighErrorRate
          expr: |
            (
              sum(rate(minio_s3_requests_total{error="true"}[5m]))
              /
              sum(rate(minio_s3_requests_total[5m]))
            ) > 0.05
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "High error rate on MinIO"
            description: "{{ $value | humanizePercentage }}"

Grafana Dashboard Pattern

Dashboards are stored as JSON ConfigMaps in k8s/monitoring/dashboards/ and auto-loaded via helmfile postsync hook (see helmfile.yaml.gotmpl, lines 471–477).

6-row template (recommended layout):

1. Availability: Uptime, error rate, latency (SLO band).
2. Resources: CPU, memory, disk usage, network I/O.
3. Domain metrics: Service-specific KPIs (throughput, queue depth, cache hit rate).
4. Logs: Recent error logs from Loki.
5. SLO: SLI tracking (burn rate, error budget).
6. Related dashboards: Links to dependent services.

Auto-load hook (helmfile.yaml.gotmpl, lines 471–477):

hooks:
  - events: ["postsync"]
    command: kubectl
    args:
      - apply
      - -f
      - k8s/monitoring/dashboards/

---

Integration Checklist

When adding a new service to the homelab, verify the following:

• Chart pinning: Helm chart version pinned (~1.0 format in helmfile, not floating latest).
• Secrets management: All secrets stored in Vault (none in values.yaml, ConfigMap, or CLI flags).
• Metrics endpoint: Service exports /metrics endpoint (Prometheus format).
• ServiceMonitor: Created and auto-scraped by Prometheus operator.
• PrometheusRule: Alert rules defined for errors, latency, SLO violations.
• Grafana dashboard: 6-row template auto-loaded via ConfigMap.
• Ingress: Rule added if external access needed (see k8s/ingress/ingress.yaml).
• OIDC integration: If UI component, integrated with Authentik (see core iam bootstrap).

---

Shared/Reusable Repos & Image Publication

When to Publish as Public GHCR

Rule: If a service's chart + image source lives in a separate repo (not in homelab), it must be published as a public GitHub repo under the Riotpiaole org.

Rationale:

• Local in-cluster charts are fine for infra-owned services.
• Shared/reusable service charts should be version-pinned and publicly available for:
  • Reuse across different clusters (other labs, staging, prod).
  • Consumption by Argo CD apps (CI/CD pipeline).
  • Independent evolution without tight coupling to homelab repo.

Workflow

Develop locally:

1. Create companion repo (e.g., kafaka-management-service, queue-operator) in a private or public repo.
2. Include Dockerfile and Helm chart.

Build & publish:

1. Set up CI/CD in the companion repo (GitHub Actions).
2. Build image and push to GHCR: ghcr.io/Riotpiaole/<repo>:<tag>.
3. Tag release and publish chart (npm registry, GitHub releases, or OCI registry).

Reference in homelab:

1. Pin image tag + chart version in helmfile.yaml.gotmpl.
2. Chart repositories block references the public Helm repo (or OCI registry).

Example (commit cd1c569 — SQS charts):

- name: management-service
  namespace: sqs
  chart: k8s/sqs/charts/management-service
  values:
    - image:
        repository: ghcr.io/Riotpiaole/kafaka-management-service
        tag: v1.0.0  # Pinned tag from public build

---

Cross-References

• CLAUDE.md gotchas: Field name = variable name, MinIO --env flag exposure, helmfile template syntax ({{ env "VAR" }} not ${VAR}), CNPG password templating.
• README.md: kubectl context setup, port-forward aliases, cluster topology.
• USAGE.md: IAM bootstrap, secret management, deployment recipes.