poimen-memory/docs/api/T7_REBUILD_API.md

# T7.5: Deterministic Rebuild API Reference

## Overview

Verify rebuild determinism with SHA-256 checksums. Daily CI integration with zero drift tolerance.

---

## Endpoints

### POST /memory/rebuild
Trigger rebuild with optional verification

**Request**:
```bash
curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "project": "poimen",
    "verify": true,
    "dry_run": false
  }' \
  http://localhost:8080/memory/rebuild
```

**Response** (200 OK - Success):
```json
{
  "status": "success",
  "rebuild_id": "rebuild-2025-01-30-100000",
  "records_processed": 45230,
  "duration_ms": 142500,
  "checksum": {
    "before": "a3f8c9d2e1b4...",
    "after": "a3f8c9d2e1b4...",
    "match": true
  },
  "incremental": false,
  "from_checkpoint": null,
  "diff_summary": null
}
```

**Response** (409 Conflict - Checksum Mismatch):
```json
{
  "status": "failed",
  "rebuild_id": "rebuild-2025-01-30-100001",
  "records_processed": 45230,
  "duration_ms": 145000,
  "checksum": {
    "before": "a3f8c9d2e1b4...",
    "after": "7f2e1a9c8b3d...",
    "match": false
  },
  "incremental": false,
  "from_checkpoint": null,
  "diff_summary": {
    "entities_added": 3,
    "entities_removed": 0,
    "entities_modified": 12,
    "edges_added": 5,
    "edges_modified": 8
  }
}
```

---

### GET /memory/rebuild/status
Get last rebuild result and checkpoints

**Request**:
```bash
curl -H "Authorization: Bearer $TOKEN" \
  http://localhost:8080/memory/rebuild/status
```

**Response** (200 OK):
```json
{
  "last_rebuild": {
    "rebuild_id": "rebuild-2025-01-30-100000",
    "started_at": "2025-01-30T10:00:00Z",
    "completed_at": "2025-01-30T10:02:22Z",
    "status": "success",
    "checksum": "a3f8c9d2e1b4...",
    "records_processed": 45230
  },
  "checkpoints": [
    {
      "id": "cp-2025-01-29",
      "created_at": "2025-01-29T00:00:00Z",
      "event_count": 44000,
      "checksum": "b4c9d8e7f6a5..."
    },
    {
      "id": "cp-2025-01-28",
      "created_at": "2025-01-28T00:00:00Z",
      "event_count": 43000,
      "checksum": "c5d8e7f6a4b3..."
    }
  ],
  "health": {
    "event_log_size": 45230,
    "last_event_at": "2025-01-30T09:55:00Z",
    "estimated_rebuild_time_ms": 145000
  }
}
```

---

## Request Options

### verify (Boolean, default: true)
Compute checksums before/after rebuild. Fails if mismatch.

```bash
# With verification (recommended)
curl -X POST http://localhost:8080/memory/rebuild \
  -d '{"project": "poimen", "verify": true}'

# Without verification (fast path, less safe)
curl -X POST http://localhost:8080/memory/rebuild \
  -d '{"project": "poimen", "verify": false}'
```

### dry_run (Boolean, default: false)
Preview rebuild without applying. Returns what would happen.

```bash
curl -X POST http://localhost:8080/memory/rebuild \
  -d '{"project": "poimen", "verify": true, "dry_run": true}'

# Response:
{
  "status": "dry_run",
  "message": "Rebuild would succeed",
  "result": {...}
}
```

### from_checkpoint (String, optional)
Resume incremental rebuild from checkpoint.

```bash
# Full rebuild
curl -X POST http://localhost:8080/memory/rebuild \
  -d '{"project": "poimen", "from_checkpoint": null}'

# Incremental from checkpoint
curl -X POST http://localhost:8080/memory/rebuild \
  -d '{"project": "poimen", "from_checkpoint": "cp-2025-01-29"}'
```

---

## Use Cases

### Daily Verification (CI)
```bash
#!/bin/bash
# Run every day at 2 AM UTC

TOKEN=$(get_jwt_token)
RESULT=$(curl -s -X POST http://localhost:8080/memory/rebuild \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "project": "poimen",
    "verify": true,
    "dry_run": false
  }')

STATUS=$(echo $RESULT | jq -r '.status')
MATCH=$(echo $RESULT | jq -r '.checksum.match')

if [ "$MATCH" != "true" ]; then
  echo "CRITICAL: Rebuild parity failed!"
  echo $RESULT | jq '.diff_summary'
  alert_team
  exit 1
fi

echo "OK: Rebuild is deterministic"
```

### Dry-Run Before Production
```bash
# Test the rebuild without committing
curl -X POST http://localhost:8080/memory/rebuild \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "project": "prod",
    "verify": true,
    "dry_run": true
  }'

# Response shows what would happen
# If satisfied, run again without dry_run
```

### Incremental Rebuild
```bash
# For large datasets, resume from checkpoint
curl -X POST http://localhost:8080/memory/rebuild \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "project": "poimen",
    "verify": true,
    "from_checkpoint": "cp-2025-01-29"
  }'
```

### Post-Incident Recovery
```bash
# After fixing corruption, verify system recovers
curl -X POST http://localhost:8080/memory/rebuild \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "project": "poimen",
    "verify": true
  }'

# Check status
curl -H "Authorization: Bearer $TOKEN" \
  http://localhost:8080/memory/rebuild/status
```

---

## Checksum Details

### Computation
```
checksum = SHA256(
  sorted_entity_ids ||
  sorted_edge_ids ||
  event_metadata
)
```

**Deterministic because**:
1. Entities sorted by ID (no random order)
2. Edges sorted by ID
3. Event replay sequential (no async variation)
4. Embedding model pinned (same version)
5. JSON serialization canonical (sorted keys)

### Interpretation

**Checksums match** (`match: true`):
- ✅ Rebuild is byte-identical
- ✅ No hidden non-determinism
- ✅ Data is consistent

**Checksums differ** (`match: false`):
- ❌ Non-determinism detected
- ❌ Investigate: embedding model change? Event log corruption?
- ❌ BLOCK further rebuilds until root cause fixed

---

## Error Codes

| Code | Meaning |
|------|---------|
| 200 | Rebuild succeeded (or dry_run verified) |
| 201 | Checkpoint created |
| 409 | Checksum mismatch (parity failed) |
| 401 | Unauthorized (missing/invalid JWT) |
| 403 | Forbidden (insufficient permissions) |
| 408 | Request timeout (rebuild took > 60s) |
| 503 | Service unavailable (DB connection failed) |

---

## Rate Limits

| Endpoint | Limit |
|----------|-------|
| /rebuild | 10/day (prevent spam) |
| /rebuild/status | 100/hr |

---

## Monitoring

### Prometheus Metrics

```promql
# Rebuild success rate (daily)
sum(rate(memory_rebuild_total{status="success"}[1d]))
/
sum(rate(memory_rebuild_total[1d]))

# Average rebuild time
avg(memory_rebuild_duration_seconds)

# Checksum matches (should be 100%)
memory_rebuild_parity_check{status="success"} > 0
```

### Alerts

```yaml
- alert: RebuildParityFailed
  expr: memory_rebuild_parity_check{status="failed"} > 0
  for: 0m
  annotations:
    summary: "Deterministic rebuild checksum mismatch"
    severity: critical
```

---

## CI Integration

### GitHub Actions Workflow

```yaml
# .github/workflows/rebuild-verify.yml
name: Verify Deterministic Rebuild
on:
  schedule:
    - cron: '0 2 * * *'  # 2 AM UTC daily
jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - name: Rebuild with verification
        env:
          API_URL: ${{ secrets.MEMORY_API_URL }}
          TOKEN: ${{ secrets.MEMORY_API_TOKEN }}
        run: |
          RESULT=$(curl -s -X POST $API_URL/memory/rebuild \
            -H "Authorization: Bearer $TOKEN" \
            -d '{"project":"poimen","verify":true}')
          
          MATCH=$(echo $RESULT | jq -r '.checksum.match')
          [ "$MATCH" = "true" ] || exit 1
      
      - name: Notify on failure
        if: failure()
        uses: slackapi/slack-github-action@v1
        with:
          payload: |
            {"text": "🚨 Rebuild parity check FAILED!"}
```