Story Crater Bot
71ecf482e7
- M7.1-M7.10: Extensible SourceConnector trait, Obsidian/paperless/git/S3 connectors, sync framework, CLI, HTTP endpoints, health monitoring, gate - M3.5.10: Auth integration with Authentik OIDC → Vault token validation - DESIGN.md: Add source connectors architecture, update auth to Authentik/Vault (Kong removed from cluster) - INDEX.md: 75 tasks, 11 gates - Fix all Kong references in M3.5.1 task
108 lines
3.5 KiB
Markdown
108 lines
3.5 KiB
Markdown
# M7.8 — `GET /memory/sources` + `POST /memory/sources/sync` HTTP endpoints
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M7 — Source connectors |
|
||
| Size | M — 1–3 days |
|
||
| Status | ⬜ Not started |
|
||
| Flags | — |
|
||
| Spec | inlined below |
|
||
| Blocks | M7.10 |
|
||
| Depends | M7.7, M3.5.1 |
|
||
|
||
## Goal
|
||
|
||
Expose source connector management through the HTTP API so agents and UIs can
|
||
trigger syncs, check connector health, and view status without CLI access.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
**Endpoints:**
|
||
```
|
||
GET /memory/sources # list all connectors + health
|
||
GET /memory/sources/{name} # one connector detail + drift
|
||
GET /memory/sources/{name}/health # health check only
|
||
POST /memory/sources/sync # trigger sync (async)
|
||
POST /memory/sources/{name}/sync # trigger sync for one connector
|
||
GET /memory/sources/sync/{job_id} # sync job status
|
||
```
|
||
|
||
**Response shapes:**
|
||
```json
|
||
// GET /memory/sources
|
||
[
|
||
{
|
||
"name": "homelab-vault",
|
||
"kind": "obsidian",
|
||
"document_count": 42,
|
||
"last_sync": "2026-08-26T10:00:00Z",
|
||
"health": { "reachable": true, "document_count": 42 }
|
||
}
|
||
]
|
||
|
||
// POST /memory/sources/sync
|
||
// Request: { "names": ["homelab-vault"] } (or omit for all)
|
||
// Response: 202 Accepted
|
||
{ "job_id": "sync-abc123", "status_url": "/memory/sources/sync/sync-abc123" }
|
||
|
||
// GET /memory/sources/sync/{job_id}
|
||
{
|
||
"job_id": "sync-abc123",
|
||
"status": "completed",
|
||
"connectors": {
|
||
"homelab-vault": { "new": 0, "changed": 2, "unchanged": 40, "removed": 0 }
|
||
}
|
||
}
|
||
```
|
||
|
||
**Sync is async.** POST returns 202 immediately; client polls status endpoint.
|
||
Same pattern as `/memory/ingest`.
|
||
|
||
## Steps
|
||
|
||
1. Add routes to `http_server.rs` for all six endpoints.
|
||
2. `sources_list_handler` — load registry, health check each, return JSON array.
|
||
3. `source_detail_handler` — one connector, include drift report.
|
||
4. `source_health_handler` — health check only.
|
||
5. `source_sync_handler` — validate connector names, spawn async sync job,
|
||
return job_id + status URL.
|
||
6. `source_sync_status_handler` — look up job by id, return progress.
|
||
7. Auth: all endpoints require apikey header (existing pattern).
|
||
8. Rate limiting: sync endpoint limited to 10 req/hour (it's expensive).
|
||
|
||
## Acceptance
|
||
|
||
- All endpoints return correct status codes and JSON shapes.
|
||
- Sync is async — POST returns immediately, job runs in background.
|
||
- Unknown connector name returns 404.
|
||
- Auth required on all endpoints.
|
||
- Rate limiting on sync endpoint.
|
||
|
||
## Verify
|
||
|
||
**Integration test** — `tests/it_source_http.rs`:
|
||
1. `a1_list_returns_connectors` — register connectors, GET /memory/sources;
|
||
assert JSON array with correct fields.
|
||
2. `a2_detail_includes_drift` — GET /memory/sources/{name}; assert drift fields.
|
||
3. `a3_health_check_via_http` — GET /memory/sources/{name}/health; assert
|
||
reachable field.
|
||
4. `a4_sync_returns_202` — POST /memory/sources/sync; assert 202 + job_id.
|
||
5. `a5_sync_status_tracks_progress` — poll status endpoint; assert eventually
|
||
"completed".
|
||
6. `a6_unknown_connector_404` — GET /memory/sources/nonexistent; assert 404.
|
||
7. `a7_auth_required` — request without apikey; assert 401.
|
||
8. `a8_sync_rate_limited` — 11 sync requests; assert 429 on the 11th.
|
||
|
||
**Command:** `cargo test --test it_source_http`
|
||
|
||
## Traps
|
||
|
||
- Making sync synchronous. A paperless connector with 500 docs takes minutes;
|
||
blocking the HTTP response is a client timeout.
|
||
- Not capping concurrent sync jobs. Two simultaneous syncs to the same connector
|
||
can corrupt the manifest.
|
||
|
||
---
|
||
|
||
Background: [DESIGN.md](../DESIGN.md) — source connectors, HTTP endpoints
|