108 lines
3.5 KiB
Markdown
108 lines
3.5 KiB
Markdown
|
# M7.8 — `GET /memory/sources` + `POST /memory/sources/sync` HTTP endpoints
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | M7 — Source connectors |
|
|||
|
|
| Size | M — 1–3 days |
|
|||
|
|
| Status | ⬜ Not started |
|
|||
|
|
| Flags | — |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | M7.10 |
|
|||
|
|
| Depends | M7.7, M3.5.1 |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Expose source connector management through the HTTP API so agents and UIs can
|
|||
|
|
trigger syncs, check connector health, and view status without CLI access.
|
|||
|
|
|
|||
|
|
## Facts (inlined — no spec read needed)
|
|||
|
|
|
|||
|
|
**Endpoints:**
|
|||
|
|
```
|
|||
|
|
GET /memory/sources # list all connectors + health
|
|||
|
|
GET /memory/sources/{name} # one connector detail + drift
|
|||
|
|
GET /memory/sources/{name}/health # health check only
|
|||
|
|
POST /memory/sources/sync # trigger sync (async)
|
|||
|
|
POST /memory/sources/{name}/sync # trigger sync for one connector
|
|||
|
|
GET /memory/sources/sync/{job_id} # sync job status
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**Response shapes:**
|
|||
|
|
```json
|
|||
|
|
// GET /memory/sources
|
|||
|
|
[
|
|||
|
|
{
|
|||
|
|
"name": "homelab-vault",
|
|||
|
|
"kind": "obsidian",
|
|||
|
|
"document_count": 42,
|
|||
|
|
"last_sync": "2026-08-26T10:00:00Z",
|
|||
|
|
"health": { "reachable": true, "document_count": 42 }
|
|||
|
|
}
|
|||
|
|
]
|
|||
|
|
|
|||
|
|
// POST /memory/sources/sync
|
|||
|
|
// Request: { "names": ["homelab-vault"] } (or omit for all)
|
|||
|
|
// Response: 202 Accepted
|
|||
|
|
{ "job_id": "sync-abc123", "status_url": "/memory/sources/sync/sync-abc123" }
|
|||
|
|
|
|||
|
|
// GET /memory/sources/sync/{job_id}
|
|||
|
|
{
|
|||
|
|
"job_id": "sync-abc123",
|
|||
|
|
"status": "completed",
|
|||
|
|
"connectors": {
|
|||
|
|
"homelab-vault": { "new": 0, "changed": 2, "unchanged": 40, "removed": 0 }
|
|||
|
|
}
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**Sync is async.** POST returns 202 immediately; client polls status endpoint.
|
|||
|
|
Same pattern as `/memory/ingest`.
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. Add routes to `http_server.rs` for all six endpoints.
|
|||
|
|
2. `sources_list_handler` — load registry, health check each, return JSON array.
|
|||
|
|
3. `source_detail_handler` — one connector, include drift report.
|
|||
|
|
4. `source_health_handler` — health check only.
|
|||
|
|
5. `source_sync_handler` — validate connector names, spawn async sync job,
|
|||
|
|
return job_id + status URL.
|
|||
|
|
6. `source_sync_status_handler` — look up job by id, return progress.
|
|||
|
|
7. Auth: all endpoints require apikey header (existing pattern).
|
|||
|
|
8. Rate limiting: sync endpoint limited to 10 req/hour (it's expensive).
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
- All endpoints return correct status codes and JSON shapes.
|
|||
|
|
- Sync is async — POST returns immediately, job runs in background.
|
|||
|
|
- Unknown connector name returns 404.
|
|||
|
|
- Auth required on all endpoints.
|
|||
|
|
- Rate limiting on sync endpoint.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Integration test** — `tests/it_source_http.rs`:
|
|||
|
|
1. `a1_list_returns_connectors` — register connectors, GET /memory/sources;
|
|||
|
|
assert JSON array with correct fields.
|
|||
|
|
2. `a2_detail_includes_drift` — GET /memory/sources/{name}; assert drift fields.
|
|||
|
|
3. `a3_health_check_via_http` — GET /memory/sources/{name}/health; assert
|
|||
|
|
reachable field.
|
|||
|
|
4. `a4_sync_returns_202` — POST /memory/sources/sync; assert 202 + job_id.
|
|||
|
|
5. `a5_sync_status_tracks_progress` — poll status endpoint; assert eventually
|
|||
|
|
"completed".
|
|||
|
|
6. `a6_unknown_connector_404` — GET /memory/sources/nonexistent; assert 404.
|
|||
|
|
7. `a7_auth_required` — request without apikey; assert 401.
|
|||
|
|
8. `a8_sync_rate_limited` — 11 sync requests; assert 429 on the 11th.
|
|||
|
|
|
|||
|
|
**Command:** `cargo test --test it_source_http`
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- Making sync synchronous. A paperless connector with 500 docs takes minutes;
|
|||
|
|
blocking the HTTP response is a client timeout.
|
|||
|
|
- Not capping concurrent sync jobs. Two simultaneous syncs to the same connector
|
|||
|
|
can corrupt the manifest.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background: [DESIGN.md](../DESIGN.md) — source connectors, HTTP endpoints
|