feat: CI/CD integration testing with network access #23

2026-09-13T04:24:56Z

poimen commented

2026-09-13 04:24:56 +00:00

Problem

Current CI/CD pipeline builds and deploys images immediately without testing. Broken code can reach production without verification.

Solution

Implement gated CI/CD workflow that builds with SHA tag, deploys test pod, runs integration tests inside pod, promotes to 'latest' only if tests pass, and auto-cleanups.

What to Check

CI Workflow

Image built with SHA tag only (not latest)
Test pod deployed with labels: app=api-gateway, managed-by=argocd, role=test, test-run=
Tests run inside pod via kubectl exec
Image promotion only if tests pass
Test pod cleanup happens

Network Access

Test pod labels match network policy selector
Can reach: kube-system:53, poimen:8080, temporal:7233, storage:80/9000, sqs:9090, iam:9000/9443

Integration Tests

6 test functions cover all adapters
Tests verify X-Service & X-Resource headers
Tests check correct status codes

Security

Non-root security context
Network policy enforced
Auto-cleanup of test resources
Failed builds don't promote

After Merge

Add KUBECONFIG_B64 secret to Gitea
Next push triggers integration testing

## Problem Current CI/CD pipeline builds and deploys images immediately without testing. Broken code can reach production without verification. ## Solution Implement gated CI/CD workflow that builds with SHA tag, deploys test pod, runs integration tests inside pod, promotes to 'latest' only if tests pass, and auto-cleanups. ## What to Check ### CI Workflow - [ ] Image built with SHA tag only (not latest) - [ ] Test pod deployed with labels: app=api-gateway, managed-by=argocd, role=test, test-run=<sha> - [ ] Tests run inside pod via kubectl exec - [ ] Image promotion only if tests pass - [ ] Test pod cleanup happens ### Network Access - [ ] Test pod labels match network policy selector - [ ] Can reach: kube-system:53, poimen:8080, temporal:7233, storage:80/9000, sqs:9090, iam:9000/9443 ### Integration Tests - [ ] 6 test functions cover all adapters - [ ] Tests verify X-Service & X-Resource headers - [ ] Tests check correct status codes ### Security - [ ] Non-root security context - [ ] Network policy enforced - [ ] Auto-cleanup of test resources - [ ] Failed builds don't promote ## After Merge 1. Add KUBECONFIG_B64 secret to Gitea 2. Next push triggers integration testing

poimen added 2 commits 2026-09-13 04:24:57 +00:00

feat: proper CI/CD workflow with integration testing 52c36e587b

BREAKING CHANGE: CI now requires kubeconfig to run integration tests

Changes:
- Build image with commit SHA tag (NOT latest yet)
- Deploy dedicated test pod from new image
- Run full integration test suite against test pod
- Only promote to latest tag AFTER tests pass
- Cleanup test pod after run

CI/CD Flow:
  1. go vet + go test (unit tests)
  2. Build image: api-gateway:<sha>
  3. Push to registry
  4. Deploy test pod with <sha> image
  5. Run integration tests (memory, S3, SQS, workflow, IAM, health)
  6. If tests pass: tag as latest and push
  7. If tests fail: keep <sha> tag, don't promote to latest
  8. Cleanup test pod

This ensures:
- New code is tested in cluster before production deployment
- ArgoCD only pulls latest after tests pass
- Failed builds don't get promoted to production
- Full test coverage of all adapters

Requires: KUBECONFIG_B64 secret in Gitea for cluster access

fix: ensure test pod can reach all downstream services

CI / CI (pull_request) Failing after 2m58s

Details

e0622449cc

Add labels to test pod to match network policy selectors:
- app=api-gateway (matches network policy pod selector)
- managed-by=argocd (matches network policy pod selector)
- role=test (identify as test pod)
- test-run=<sha> (track which test run spawned it)

Network policy 'api-gateway' in api namespace already allows egress to:
✅ kube-system (DNS resolution)
✅ poimen (port 8080 - Memory service)
✅ temporal (port 7233 - Workflow service)
✅ storage (ports 80, 9000 - S3/MinIO)
✅ sqs (port 9090 - SQS service)
✅ iam (ports 9000, 9443 - Authentik/IAM)

Test pod inherits same network access as production pods via labels.
No additional network policies needed.

rock added 1 commit 2026-09-13 04:44:31 +00:00

fix: remove kubectl installation, assume available in runner

CI / CI (pull_request) Failing after 3m6s

Details

a700ac065b

OPTIMIZATIONS:
- Remove curl-based kubectl installation (inefficient)
- Assume kubectl is available in Gitea runner environment
- Replace port-forward with kubectl exec for test execution
- Tests now run directly inside test pod (not from runner)
- Simpler, faster, more reliable

CI Flow:
  1. go vet + go test (unit tests)
  2. Build image: api-gateway:<sha>
  3. Push: <sha> tag only
  4. Deploy test pod with proper labels
  5. kubectl exec into pod to run tests
  6. Tests run inside pod, can reach services via network policy
  7. Promote to latest only if tests pass
  8. Cleanup test pod

rock added 1 commit 2026-09-13 04:47:18 +00:00

refactor: use Kubernetes Job for integration testing instead of manual pod management

CI / CI (pull_request) Failing after 2m56s

Details

fa9938df8e

RATIONALE:
The Kubernetes way to run integration tests is via Jobs, not manual pod management.
Jobs are simpler, more idiomatic, and handle all the complexity for us.

CHANGES:
- Remove manual: kubectl run, kubectl wait, kubectl exec
- Use Kubernetes Job (already defined in k8s/integration-test-job.yaml)
- Job handles: pod creation, retry, cleanup, status reporting
- CI only does: apply job, set image, wait, check status

SIMPLIFIED CI FLOW:
  1. go vet + go test (unit tests)
  2. Build image: api-gateway:<sha>
  3. Push: <sha> tag only
  4. Apply Job from k8s/integration-test-job.yaml
  5. Set job image to new build
  6. Wait for job completion
  7. Get logs
  8. Check job status
  9. Promote to latest (if job succeeded)
  10. Cleanup job

BENEFITS:
✅ More idiomatic (Kubernetes Job is the standard way)
✅ Simpler CI workflow (fewer manual steps)
✅ Job handles retries, backoff, cleanup automatically
✅ Better status reporting
✅ Declarative (job spec in git, not imperative in CI)
✅ Easier to test locally (just kubectl apply -f k8s/integration-test-job.yaml)

WHAT KUBERNETES JOB HANDLES:
✓ Pod creation and lifecycle
✓ Restart policy and retries
✓ Cleanup on completion
✓ Status tracking
✓ Log aggregation
✓ Resource limits

rock added 1 commit 2026-09-13 04:52:12 +00:00

refactor: use in-cluster authentication instead of kubeconfig secret

CI / CI (pull_request) Failing after 2m56s

Details

1dd71de97f

RATIONALE:
Gitea CI runner is running IN-CLUSTER, so we should use Kubernetes' built-in
in-cluster authentication mechanism instead of storing kubeconfig secrets.

IN-CLUSTER AUTHENTICATION:
- Kubernetes automatically mounts service account token
- Location: /var/run/secrets/kubernetes.io/serviceaccount/token
- Location: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
- kubectl automatically detects and uses these
- No need to pass credentials via secrets

CHANGES:
1. Remove KUBECONFIG_B64 secret requirement
2. Add in-cluster auth detection step
3. Update Job to use actual built image (not golang base)
4. Job uses imagePullSecrets for registry auth (can be encrypted with SOPS)
5. Add regcred image pull secret reference

CI FLOW:
  1. Detect in-cluster authentication is available
  2. kubectl commands automatically use mounted service account
  3. No secrets needed in CI env vars
  4. Job applies with RBAC service account
  5. Registry credentials via imagePullSecrets (encrypted with SOPS)

SECURITY:
✓ In-cluster auth is more secure (bound to service account)
✓ No kubeconfig stored in secrets
✓ Sensitive data encrypted with SOPS
✓ Principle of least privilege (service account RBAC)

rock closed this pull request

2026-09-13 05:41:41 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/homelab-frontend#23