docs: Add M7 source connectors (10 tasks), M3.5.10 auth integration, remove Kong refs
- M7.1-M7.10: Extensible SourceConnector trait, Obsidian/paperless/git/S3 connectors, sync framework, CLI, HTTP endpoints, health monitoring, gate - M3.5.10: Auth integration with Authentik OIDC → Vault token validation - DESIGN.md: Add source connectors architecture, update auth to Authentik/Vault (Kong removed from cluster) - INDEX.md: 75 tasks, 11 gates - Fix all Kong references in M3.5.1 task
This commit is contained in:
Story Crater Bot
@@ -0,0 +1,146 @@
|
||||
# M7.6 — Sync framework
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M7 — Source connectors |
|
||||
| Size | L — 3–5 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M7.7, M7.10 |
|
||||
| Depends | M7.1, M3.6.3 |
|
||||
|
||||
## Goal
|
||||
|
||||
Build the shared sync engine that handles change detection, tombstoning, drift
|
||||
reporting, and resumable sync for **all** connectors — so individual connector
|
||||
implementations only fetch documents and everything else is handled once.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
Every connector faces the same sync problems:
|
||||
1. **What changed?** Compare content hashes from `list_documents()` against a
|
||||
manifest of last-known hashes. Only fetch and process changed documents.
|
||||
2. **What disappeared?** Documents in the manifest but not in `list_documents()`
|
||||
need tombstone records in the log. Append-only — never delete.
|
||||
3. **What if sync crashes midway?** Track progress per-document. Restart
|
||||
processes only unseen documents.
|
||||
4. **How much will this cost?** Drift report shows counts without mutating
|
||||
anything.
|
||||
|
||||
This is the generalization of M3.6.3's `mem ref sync/list/rm` logic, applied to
|
||||
any `SourceConnector` instead of just `DocCorpusSource`.
|
||||
|
||||
**Manifest storage.** Per-connector manifest in `log/connectors/<name>.manifest.jsonl`:
|
||||
```jsonl
|
||||
{"doc_id":"abc","source_uri":"file:///vault/k8s.md","content_hash":"sha256:...","chunk_count":12,"synced_at":"..."}
|
||||
{"doc_id":"def","source_uri":"paperless://doc/42","content_hash":"sha256:...","chunk_count":3,"synced_at":"..."}
|
||||
```
|
||||
|
||||
**Sync algorithm:**
|
||||
```
|
||||
current = connector.list_documents()
|
||||
previous = load_manifest(connector.name)
|
||||
|
||||
for doc in current:
|
||||
if doc.content_hash == previous[doc.doc_id].content_hash:
|
||||
skip (unchanged)
|
||||
else if doc.doc_id in previous:
|
||||
tombstone previous chunks, fetch + chunk + embed new (changed)
|
||||
else:
|
||||
fetch + chunk + embed (new)
|
||||
|
||||
for doc_id in previous not in current:
|
||||
tombstone all chunks (removed)
|
||||
|
||||
save_manifest(connector.name, current)
|
||||
```
|
||||
|
||||
**Chunking delegation.** The sync framework owns the chunking step. It routes
|
||||
fetched `DocumentContent` through the appropriate `ChunkPolicy`:
|
||||
- Markdown → heading-boundary chunking (reuse `DocCorpusSource` logic)
|
||||
- Plain text → paragraph-boundary chunking
|
||||
- Configurable per connector kind in `connectors.yaml`
|
||||
|
||||
**Level routing.** Session connectors → RecordSource → gated loop (L0/L1/L2).
|
||||
Document connectors → Reference records (Level R). The `source_type()` method
|
||||
on the connector determines the pipeline.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Define `SyncEngine` struct in `mem-ingest/src/sync.rs`.
|
||||
2. Implement manifest loading/saving (JSONL per connector).
|
||||
3. Implement diff algorithm: `(new, changed, unchanged, removed)` from
|
||||
`list_documents()` vs manifest.
|
||||
4. Implement sync loop: for each `new`/`changed` doc, fetch → chunk → emit
|
||||
records. For each `removed` doc, emit tombstones.
|
||||
5. Implement drift reporting: same diff algorithm, print counts, no mutations.
|
||||
6. Implement resume: track synced doc_ids in a progress file. On restart,
|
||||
skip already-synced docs.
|
||||
7. Implement rate limiting: configurable max concurrent fetches per connector.
|
||||
8. Integrate with `mem-store` for writing Reference records to the log.
|
||||
9. Integrate with `mem-llm` for embedding new chunks.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Sync of unchanged connector produces zero embedding calls.
|
||||
- Changed document: old chunks tombstoned, new chunks embedded and stored.
|
||||
- Removed document: chunks tombstoned, manifest updated.
|
||||
- Drift report matches actual changes without mutating anything.
|
||||
- Crash mid-sync → restart processes only remaining documents.
|
||||
- Rate limiting: never exceeds configured concurrent fetch limit.
|
||||
- Log remains append-only (tombstones are records, not deletions).
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** `VecConnector` with mutable document list, counting embedder.
|
||||
|
||||
**Integration test** — `tests/it_sync_framework.rs`:
|
||||
1. `a1_initial_sync_all_new` — 3 docs, no manifest; assert all 3 fetched and
|
||||
embedded, manifest written with 3 entries.
|
||||
2. `a2_unchanged_skipped` — sync again with same content; assert zero fetch
|
||||
calls, zero embed calls.
|
||||
3. `a3_changed_doc_replaced` — modify one doc's content; sync; assert old chunks
|
||||
tombstoned, new chunks embedded, embed count equals changed doc's chunk count.
|
||||
4. `a4_removed_doc_tombstoned` — remove a doc from connector; sync; assert
|
||||
tombstone records emitted, manifest entry removed.
|
||||
5. `a5_new_doc_added` — add a doc to connector; sync; assert only new doc
|
||||
fetched and embedded.
|
||||
6. `a6_drift_report_read_only` — modify docs, run drift report; assert correct
|
||||
counts (1 new, 1 changed, 1 unchanged, 1 removed); assert no mutations to
|
||||
manifest or log.
|
||||
7. `a7_resume_after_crash` — sync 5 docs, simulate crash after 3; restart;
|
||||
assert only 2 remaining docs processed.
|
||||
8. `a8_tombstone_is_append` — count log lines before and after remove; assert
|
||||
count only grew.
|
||||
9. `a9_manifest_roundtrip` — save manifest, load it; assert field-for-field
|
||||
equality.
|
||||
10. `a10_rebuild_parity` — after full sync, `mem rebuild --from-log` produces
|
||||
identical state.
|
||||
|
||||
**Command:** `cargo test --test it_sync_framework`
|
||||
|
||||
**False pass:**
|
||||
- Asserting "no duplicate rows" instead of counting embed calls. A sync that
|
||||
re-embeds everything and upserts by sha produces correct rows and wasted
|
||||
compute.
|
||||
- Testing drift report without actually changing documents first.
|
||||
|
||||
## Traps
|
||||
|
||||
- Comparing document-level hashes instead of chunk-level. A doc that changed
|
||||
one paragraph should re-embed only the affected chunks, not all of them.
|
||||
However, heading-boundary chunking means changing one heading can shift all
|
||||
subsequent chunks. Accept document-level granularity for now; chunk-level
|
||||
optimization is a future refinement.
|
||||
- Making the manifest a database table instead of JSONL. The manifest must
|
||||
survive `mem rebuild --from-log` — it is metadata about the sync process,
|
||||
not a projection of the log.
|
||||
- Running fetch + embed serially. A connector with 500 docs at 200ms per embed
|
||||
takes 100s serially. Concurrent fetch + sequential embed is the right shape.
|
||||
- Ignoring the derived filter (M4.2). Document connectors produce Level R content
|
||||
that must register in the artifact manifest so it cannot re-enter as evidence.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — source connectors, sync framework
|
||||
Reference in New Issue
Block a user