homelab/project-usage/forgejo-registry-cleanup.md

# Forgejo OCI Registry Cleanup

Automatic garbage collection for the Forgejo container registry. Deletes old image tags when newer versions are pushed, keeping only the latest N versions per repository.

## Why

The Forgejo OCI registry stores all pushed images indefinitely. Without cleanup:
- Old/retired image versions accumulate
- Storage fills up (`longhorn` PVC)
- Old versions clutter the UI

## What it does

**CronJob** (`forgejo-registry-cleanup`):
- Runs daily at 2 AM UTC (configurable)
- Lists all images in the registry
- For each image, keeps only the **latest 3 versions** (configurable)
- Deletes tags for older versions
- Skips images with ≤ 3 tags (nothing to delete)

## How to enable

The manifest is in `k8s/bootstrap/phase3-forgejo/registry-cleanup-cronjob.yaml`. It's **currently disabled** (suspended) because:

1. **Forgejo registry auth** needs to be configured
   - `forgejo-registry-token` secret must exist in `cicd` namespace
   - Should contain `username` and `password` keys
   - User needs permission to delete images in the registry

2. **Registry must expose `/v2/_catalog`** endpoint
   - Standard for OCI registries
   - Forgejo includes this, but may be behind auth

### Step 1: Create registry token

If `forgejo-registry-token` doesn't exist or is empty:

```bash
# As a Forgejo admin, create an API token with full scope
# https://forgejo.riotpiao.com/user/settings/tokens
# Copy the token

kubectl create secret generic forgejo-registry-token \
  -n cicd \
  --from-literal=username=<your-username> \
  --from-literal=password=<the-api-token> \
  --dry-run=client -o yaml | sops -e -i -
```

Or edit via `k8s/argocd/secrets/forgejo-registry-token.enc.yaml`:

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: forgejo-registry-token
  namespace: cicd
type: Opaque
stringData:
  username: ci-bot         # or any user with admin rights
  password: <api-token>
```

### Step 2: Test in dry-run mode

Before enabling for real, verify it works:

```bash
# Edit the CronJob to set DRY_RUN=true
kubectl set env cronjob/forgejo-registry-cleanup -n cicd DRY_RUN=true

# Trigger a test run
kubectl create job --from=cronjob/forgejo-registry-cleanup \
  -n cicd forgejo-registry-cleanup-test

# Check logs
kubectl logs -n cicd -l job-name=forgejo-registry-cleanup-test -f
```

Dry-run output shows which images **would** be deleted without deleting them.

### Step 3: Enable for real

```bash
# Set DRY_RUN=false and unsuspend
kubectl patch cronjob forgejo-registry-cleanup -n cicd \
  -p '{"spec":{"suspend":false}}'

kubectl set env cronjob/forgejo-registry-cleanup -n cicd DRY_RUN=false
```

## Configuration

Edit `registry-cleanup-cronjob.yaml` or patch the CronJob:

| Env var | Default | Purpose |
|---|---|---|
| `REGISTRY_HOST` | `forgejo.riotpiao.com` | Registry hostname |
| `KEEP_VERSIONS` | `3` | How many recent versions to keep per image |
| `DRY_RUN` | `false` | If `true`, log what would be deleted without deleting |

**Schedule:** Edit `.spec.schedule` (cron format). Current: `0 2 * * *` (2 AM UTC daily).

Examples:
- `0 2 * * 0` → Weekly on Sunday at 2 AM
- `0 0 1 * *` → Monthly on the 1st at midnight
- `0 */6 * * *` → Every 6 hours

## Monitoring

### Check if running

```bash
# See all runs
kubectl get jobs -n cicd -l app=forgejo-registry-cleanup

# Latest run logs
kubectl logs -n cicd -l app=forgejo-registry-cleanup --tail=100 -f
```

### Failed runs

If a job fails:
1. Check logs: `kubectl logs -n cicd <job-pod>`
2. Common issues:
   - **403 Unauthorized**: Registry token invalid or expired
   - **404 _catalog**: Registry doesn't expose catalog endpoint
   - **Connection refused**: Registry unreachable (DNS, network policy)

### Metrics

The job doesn't currently emit Prometheus metrics, but you can:
- Check pod exit codes in K8s events
- Parse logs for "Total images deleted: N"
- Set up log aggregation to alert on failures

## Limitations

1. **No version sorting**: Tags are deleted in the order returned by the registry
   - Assumption: registries return newest first (not always true)
   - **Fix**: Parse semantic versions explicitly if needed

2. **No protection for `latest` tag**: If `latest` is old, it will be kept but others deleted
   - Desired behavior: prioritize newest build + never delete `latest`
   - Could add logic to always keep `latest` + latest N-1 tagged versions

3. **No size-aware deletion**: Deletes by tag count, not storage size
   - Desired: keep until storage threshold is reached
   - Would need registry V2 API extensions (`HEAD /v2/<image>/blobs/<digest>` for size)

## Customizing the script

Edit the `cleanup.sh` script in the ConfigMap to:
- Change sorting/selection logic
- Integrate with external systems (Slack alerts, Prometheus metrics)
- Add per-image exceptions (e.g., never delete `production-*` tags)
- Use `--delete-by-digest` to reclaim actual disk space (not just catalog entries)

Example: Keep all tags matching `v*.*.*.` plus latest 2:

```bash
# In cleanup.sh, replace the tag filtering logic:
SEMVER_TAGS=$(echo "$TAGS" | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | sort -rV)
KEEP_TAGS="$SEMVER_TAGS $(echo "$TAGS" | head -2 | tr '\n' ' ')"
TAGS_TO_DELETE=$(echo "$TAGS" | grep -v -F "$KEEP_TAGS")
```

## Future improvements

- [ ] Semantic version sorting (v1.0.0 > v0.9.9)
- [ ] Storage size-aware retention (keep until >80% full)
- [ ] Slack/email notifications on deletion
- [ ] Prometheus metrics export
- [ ] Per-image exception rules (YAML config)
- [ ] Integration with CI/CD pipeline (delete old PR images automatically)

---

**Related:** `k8s/bootstrap/phase3-forgejo/` — Forgejo deployment manifests