docs: Add M7 source connectors (10 tasks), M3.5.10 auth integration, remove Kong refs
- M7.1-M7.10: Extensible SourceConnector trait, Obsidian/paperless/git/S3 connectors, sync framework, CLI, HTTP endpoints, health monitoring, gate - M3.5.10: Auth integration with Authentik OIDC → Vault token validation - DESIGN.md: Add source connectors architecture, update auth to Authentik/Vault (Kong removed from cluster) - INDEX.md: 75 tasks, 11 gates - Fix all Kong references in M3.5.1 task
This commit is contained in:
@@ -0,0 +1,107 @@
|
||||
# M7.8 — `GET /memory/sources` + `POST /memory/sources/sync` HTTP endpoints
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M7 — Source connectors |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M7.10 |
|
||||
| Depends | M7.7, M3.5.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Expose source connector management through the HTTP API so agents and UIs can
|
||||
trigger syncs, check connector health, and view status without CLI access.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
**Endpoints:**
|
||||
```
|
||||
GET /memory/sources # list all connectors + health
|
||||
GET /memory/sources/{name} # one connector detail + drift
|
||||
GET /memory/sources/{name}/health # health check only
|
||||
POST /memory/sources/sync # trigger sync (async)
|
||||
POST /memory/sources/{name}/sync # trigger sync for one connector
|
||||
GET /memory/sources/sync/{job_id} # sync job status
|
||||
```
|
||||
|
||||
**Response shapes:**
|
||||
```json
|
||||
// GET /memory/sources
|
||||
[
|
||||
{
|
||||
"name": "homelab-vault",
|
||||
"kind": "obsidian",
|
||||
"document_count": 42,
|
||||
"last_sync": "2026-08-26T10:00:00Z",
|
||||
"health": { "reachable": true, "document_count": 42 }
|
||||
}
|
||||
]
|
||||
|
||||
// POST /memory/sources/sync
|
||||
// Request: { "names": ["homelab-vault"] } (or omit for all)
|
||||
// Response: 202 Accepted
|
||||
{ "job_id": "sync-abc123", "status_url": "/memory/sources/sync/sync-abc123" }
|
||||
|
||||
// GET /memory/sources/sync/{job_id}
|
||||
{
|
||||
"job_id": "sync-abc123",
|
||||
"status": "completed",
|
||||
"connectors": {
|
||||
"homelab-vault": { "new": 0, "changed": 2, "unchanged": 40, "removed": 0 }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Sync is async.** POST returns 202 immediately; client polls status endpoint.
|
||||
Same pattern as `/memory/ingest`.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Add routes to `http_server.rs` for all six endpoints.
|
||||
2. `sources_list_handler` — load registry, health check each, return JSON array.
|
||||
3. `source_detail_handler` — one connector, include drift report.
|
||||
4. `source_health_handler` — health check only.
|
||||
5. `source_sync_handler` — validate connector names, spawn async sync job,
|
||||
return job_id + status URL.
|
||||
6. `source_sync_status_handler` — look up job by id, return progress.
|
||||
7. Auth: all endpoints require apikey header (existing pattern).
|
||||
8. Rate limiting: sync endpoint limited to 10 req/hour (it's expensive).
|
||||
|
||||
## Acceptance
|
||||
|
||||
- All endpoints return correct status codes and JSON shapes.
|
||||
- Sync is async — POST returns immediately, job runs in background.
|
||||
- Unknown connector name returns 404.
|
||||
- Auth required on all endpoints.
|
||||
- Rate limiting on sync endpoint.
|
||||
|
||||
## Verify
|
||||
|
||||
**Integration test** — `tests/it_source_http.rs`:
|
||||
1. `a1_list_returns_connectors` — register connectors, GET /memory/sources;
|
||||
assert JSON array with correct fields.
|
||||
2. `a2_detail_includes_drift` — GET /memory/sources/{name}; assert drift fields.
|
||||
3. `a3_health_check_via_http` — GET /memory/sources/{name}/health; assert
|
||||
reachable field.
|
||||
4. `a4_sync_returns_202` — POST /memory/sources/sync; assert 202 + job_id.
|
||||
5. `a5_sync_status_tracks_progress` — poll status endpoint; assert eventually
|
||||
"completed".
|
||||
6. `a6_unknown_connector_404` — GET /memory/sources/nonexistent; assert 404.
|
||||
7. `a7_auth_required` — request without apikey; assert 401.
|
||||
8. `a8_sync_rate_limited` — 11 sync requests; assert 429 on the 11th.
|
||||
|
||||
**Command:** `cargo test --test it_source_http`
|
||||
|
||||
## Traps
|
||||
|
||||
- Making sync synchronous. A paperless connector with 500 docs takes minutes;
|
||||
blocking the HTTP response is a client timeout.
|
||||
- Not capping concurrent sync jobs. Two simultaneous syncs to the same connector
|
||||
can corrupt the manifest.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — source connectors, HTTP endpoints
|
||||
Reference in New Issue
Block a user