homelab/project-usage/cicd-workflow.md

# CI/CD Pipeline (Forgejo + Argo CD)

**Git Forge:** `https://forgejo.riotpiao.homelab.com`  
**Deployments:** `https://argocd.riotpiao.homelab.com` (or `kubectl port-forward`)  
**Namespaces:** `cicd`, `forge`

## When to Use

- **Build & test** — Forgejo Actions CI (GitHub Actions syntax)
- **Image push** — Build OCI images, push to Forgejo registry
- **GitOps deployment** — Argo CD syncs deploy repo to cluster
- **Secrets in CI** — ci-bot JWT tokens, never kubeconfig

## Quick Start

**1. Clone a repo from Forgejo:**
```bash
git clone https://forgejo.riotpiao.homelab.com/rock/source.git
cd source
```

**2. Create workflow:**
```bash
mkdir -p .forgejo/workflows
cat > .forgejo/workflows/ci.yml <<EOF
name: CI
on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: npm test
      - run: docker build -t myapp:latest .
      - run: |
          docker login forgejo.riotpiao.homelab.com \
            -u ci-bot \
            -p ${{ secrets.CI_BOT_TOKEN }}
          docker push forgejo.riotpiao.homelab.com/rock/myapp:latest
EOF

git add .forgejo/workflows/ci.yml
git commit -m "ci: add build workflow"
git push
```

**3. Trigger deployment:**
```bash
# Update deployment repo (rock/deploy)
git clone https://forgejo.riotpiao.homelab.com/rock/deploy.git
cd deploy

# Update image tag
sed -i 's|forgejo.riotpiao.homelab.com/rock/myapp:.*|forgejo.riotpiao.homelab.com/rock/myapp:abc123|' k8s/deployment.yaml

git add k8s/deployment.yaml
git commit -m "deploy: bump myapp to abc123"
git push
```

**4. Argo CD auto-syncs:**
```bash
# Watch deployment
k rollout status -n <app-namespace> deployment/<app-name>

# Or check Argo CD UI
kubectl port-forward -n argocd svc/argocd-server 8443:443
# https://localhost:8443 (login via Authentik)
```

## Workflow Syntax (GitHub Actions)

**Basic structure:**
```yaml
name: CI
on:
  push:
    branches: [main, develop]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: npm ci
      - run: npm test
      
  build:
    needs: test  # wait for test job
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: docker build -t myapp:${{ github.sha }} .
      - run: docker push forgejo.riotpiao.homelab.com/rock/myapp:${{ github.sha }}
```

**Available variables:**
```bash
${{ github.sha }}              # commit hash
${{ github.ref_name }}         # branch name
${{ github.run_number }}       # build #
${{ secrets.CI_BOT_TOKEN }}    # injected from Forgejo
```

## Secrets & Authentication

**ci-bot JWT token (auto-injected):**
```yaml
- run: |
    echo "${{ secrets.CI_BOT_TOKEN }}" | docker login \
      forgejo.riotpiao.homelab.com \
      -u ci-bot \
      --password-stdin
    docker push forgejo.riotpiao.homelab.com/rock/myapp:latest
```

**API token (for pushing commits):**
```yaml
- run: |
    git config user.name "ci-bot"
    git config user.email "ci-bot@homelab"
    git commit --allow-empty -m "bump: version"
    git push https://ci-bot:${{ secrets.CI_BOT_TOKEN }}@forgejo.riotpiao.homelab.com/rock/deploy.git main
```

**Vault secrets (via talos CLI):**
```bash
# Not available in CI runner — use Argo CD post-sync hooks instead
# Or inject via init container before workflow runs
```

## Argo CD GitOps

**Create app (one-time):**
```bash
argocd app create story-crater \
  --repo https://forgejo.riotpiao.homelab.com/rock/deploy.git \
  --path k8s/ \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace story-crater-backend \
  --sync-policy automated
```

**Monitor sync:**
```bash
# CLI
argocd app get story-crater
argocd app logs story-crater

# UI: https://argocd.riotpiao.homelab.com
# Login: Authentik SSO (homelab-admins group only)
```

**Manual sync:**
```bash
argocd app sync story-crater
argocd app wait story-crater
```

## Security Rules

✅ **DO:**
- Store credentials in Forgejo Secrets (auto-injected)
- Use ci-bot JWT for image push only
- Commit to deploy repo (triggers Argo CD)
- Enable branch protection (require CI pass)

❌ **DON'T:**
- Put kubeconfig in CI (Argo CD bridges gap)
- Commit secrets to source repo
- Use admin-bot in CI workflows (over-privileged)
- Push images directly to cluster (use Argo CD)

## Monitoring

**Grafana dashboard:** `svc-forgejo`, `svc-argocd`

**Key metrics:**
- `forgejo_workflows_running` — active workflows
- `argocd_app_sync_duration_seconds` — deployment time
- `argocd_app_info{sync_status="OutOfSync"}` — drift detection

## Troubleshooting

**Workflow fails silently:**
```bash
# Check runner logs
k logs -n cicd -f deploy/forgejo-runner

# Check if runner pod is healthy
k get pods -n cicd -l app=forgejo-runner
```

**Image push fails (401 Unauthorized):**
```bash
# Verify ci-bot token in Forgejo
# Settings → Applications → ci-bot → check scopes (package:write)

# Or re-create token
talos put cluster/iam/agents/ci-bot-token TOKEN="$(openssl rand -hex 32)"
```

**Argo CD out-of-sync:**
```bash
# Check deploy repo changes
argocd app diff story-crater

# Manual sync
argocd app sync story-crater --prune
```

**Webhook not triggering:**
```bash
# Verify Forgejo webhook config
# Repo Settings → Webhooks → Check delivery logs

# Or manually trigger
argocd app sync story-crater
```

See `/TROUBLESHOOTING.md` for full incident guide.