poimen-memory/tasks/M7.8-source-http-endpoints.md

M7.8 — GET /memory/sources + POST /memory/sources/sync HTTP endpoints

Field	Value
Phase	M7 — Source connectors
Size	M — 1–3 days
Status	⬜ Not started
Flags	—
Spec	inlined below
Blocks	M7.10
Depends	M7.7, M3.5.1

Goal

Expose source connector management through the HTTP API so agents and UIs can trigger syncs, check connector health, and view status without CLI access.

Facts (inlined — no spec read needed)

Endpoints:

GET  /memory/sources                    # list all connectors + health
GET  /memory/sources/{name}             # one connector detail + drift
GET  /memory/sources/{name}/health      # health check only
POST /memory/sources/sync               # trigger sync (async)
POST /memory/sources/{name}/sync        # trigger sync for one connector
GET  /memory/sources/sync/{job_id}      # sync job status

Response shapes:

// GET /memory/sources
[
  {
    "name": "homelab-vault",
    "kind": "obsidian",
    "document_count": 42,
    "last_sync": "2026-08-26T10:00:00Z",
    "health": { "reachable": true, "document_count": 42 }
  }
]

// POST /memory/sources/sync
// Request: { "names": ["homelab-vault"] }  (or omit for all)
// Response: 202 Accepted
{ "job_id": "sync-abc123", "status_url": "/memory/sources/sync/sync-abc123" }

// GET /memory/sources/sync/{job_id}
{
  "job_id": "sync-abc123",
  "status": "completed",
  "connectors": {
    "homelab-vault": { "new": 0, "changed": 2, "unchanged": 40, "removed": 0 }
  }
}

Sync is async. POST returns 202 immediately; client polls status endpoint. Same pattern as /memory/ingest.

Steps

1. Add routes to http_server.rs for all six endpoints.
2. sources_list_handler — load registry, health check each, return JSON array.
3. source_detail_handler — one connector, include drift report.
4. source_health_handler — health check only.
5. source_sync_handler — validate connector names, spawn async sync job, return job_id + status URL.
6. source_sync_status_handler — look up job by id, return progress.
7. Auth: all endpoints require apikey header (existing pattern).
8. Rate limiting: sync endpoint limited to 10 req/hour (it's expensive).

Acceptance

• All endpoints return correct status codes and JSON shapes.
• Sync is async — POST returns immediately, job runs in background.
• Unknown connector name returns 404.
• Auth required on all endpoints.
• Rate limiting on sync endpoint.

Verify

Integration test — tests/it_source_http.rs:

1. a1_list_returns_connectors — register connectors, GET /memory/sources; assert JSON array with correct fields.
2. a2_detail_includes_drift — GET /memory/sources/{name}; assert drift fields.
3. a3_health_check_via_http — GET /memory/sources/{name}/health; assert reachable field.
4. a4_sync_returns_202 — POST /memory/sources/sync; assert 202 + job_id.
5. a5_sync_status_tracks_progress — poll status endpoint; assert eventually "completed".
6. a6_unknown_connector_404 — GET /memory/sources/nonexistent; assert 404.
7. a7_auth_required — request without apikey; assert 401.
8. a8_sync_rate_limited — 11 sync requests; assert 429 on the 11th.

Command: cargo test --test it_source_http

Traps

• Making sync synchronous. A paperless connector with 500 docs takes minutes; blocking the HTTP response is a client timeout.
• Not capping concurrent sync jobs. Two simultaneous syncs to the same connector can corrupt the manifest.

---

Background: DESIGN.md — source connectors, HTTP endpoints