Story Crater Bot
71ecf482e7
- M7.1-M7.10: Extensible SourceConnector trait, Obsidian/paperless/git/S3 connectors, sync framework, CLI, HTTP endpoints, health monitoring, gate - M3.5.10: Auth integration with Authentik OIDC → Vault token validation - DESIGN.md: Add source connectors architecture, update auth to Authentik/Vault (Kong removed from cluster) - INDEX.md: 75 tasks, 11 gates - Fix all Kong references in M3.5.1 task
147 lines
6.2 KiB
Markdown
147 lines
6.2 KiB
Markdown
# M7.6 — Sync framework
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M7 — Source connectors |
|
||
| Size | L — 3–5 days |
|
||
| Status | ⬜ Not started |
|
||
| Flags | — |
|
||
| Spec | inlined below |
|
||
| Blocks | M7.7, M7.10 |
|
||
| Depends | M7.1, M3.6.3 |
|
||
|
||
## Goal
|
||
|
||
Build the shared sync engine that handles change detection, tombstoning, drift
|
||
reporting, and resumable sync for **all** connectors — so individual connector
|
||
implementations only fetch documents and everything else is handled once.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
Every connector faces the same sync problems:
|
||
1. **What changed?** Compare content hashes from `list_documents()` against a
|
||
manifest of last-known hashes. Only fetch and process changed documents.
|
||
2. **What disappeared?** Documents in the manifest but not in `list_documents()`
|
||
need tombstone records in the log. Append-only — never delete.
|
||
3. **What if sync crashes midway?** Track progress per-document. Restart
|
||
processes only unseen documents.
|
||
4. **How much will this cost?** Drift report shows counts without mutating
|
||
anything.
|
||
|
||
This is the generalization of M3.6.3's `mem ref sync/list/rm` logic, applied to
|
||
any `SourceConnector` instead of just `DocCorpusSource`.
|
||
|
||
**Manifest storage.** Per-connector manifest in `log/connectors/<name>.manifest.jsonl`:
|
||
```jsonl
|
||
{"doc_id":"abc","source_uri":"file:///vault/k8s.md","content_hash":"sha256:...","chunk_count":12,"synced_at":"..."}
|
||
{"doc_id":"def","source_uri":"paperless://doc/42","content_hash":"sha256:...","chunk_count":3,"synced_at":"..."}
|
||
```
|
||
|
||
**Sync algorithm:**
|
||
```
|
||
current = connector.list_documents()
|
||
previous = load_manifest(connector.name)
|
||
|
||
for doc in current:
|
||
if doc.content_hash == previous[doc.doc_id].content_hash:
|
||
skip (unchanged)
|
||
else if doc.doc_id in previous:
|
||
tombstone previous chunks, fetch + chunk + embed new (changed)
|
||
else:
|
||
fetch + chunk + embed (new)
|
||
|
||
for doc_id in previous not in current:
|
||
tombstone all chunks (removed)
|
||
|
||
save_manifest(connector.name, current)
|
||
```
|
||
|
||
**Chunking delegation.** The sync framework owns the chunking step. It routes
|
||
fetched `DocumentContent` through the appropriate `ChunkPolicy`:
|
||
- Markdown → heading-boundary chunking (reuse `DocCorpusSource` logic)
|
||
- Plain text → paragraph-boundary chunking
|
||
- Configurable per connector kind in `connectors.yaml`
|
||
|
||
**Level routing.** Session connectors → RecordSource → gated loop (L0/L1/L2).
|
||
Document connectors → Reference records (Level R). The `source_type()` method
|
||
on the connector determines the pipeline.
|
||
|
||
## Steps
|
||
|
||
1. Define `SyncEngine` struct in `mem-ingest/src/sync.rs`.
|
||
2. Implement manifest loading/saving (JSONL per connector).
|
||
3. Implement diff algorithm: `(new, changed, unchanged, removed)` from
|
||
`list_documents()` vs manifest.
|
||
4. Implement sync loop: for each `new`/`changed` doc, fetch → chunk → emit
|
||
records. For each `removed` doc, emit tombstones.
|
||
5. Implement drift reporting: same diff algorithm, print counts, no mutations.
|
||
6. Implement resume: track synced doc_ids in a progress file. On restart,
|
||
skip already-synced docs.
|
||
7. Implement rate limiting: configurable max concurrent fetches per connector.
|
||
8. Integrate with `mem-store` for writing Reference records to the log.
|
||
9. Integrate with `mem-llm` for embedding new chunks.
|
||
|
||
## Acceptance
|
||
|
||
- Sync of unchanged connector produces zero embedding calls.
|
||
- Changed document: old chunks tombstoned, new chunks embedded and stored.
|
||
- Removed document: chunks tombstoned, manifest updated.
|
||
- Drift report matches actual changes without mutating anything.
|
||
- Crash mid-sync → restart processes only remaining documents.
|
||
- Rate limiting: never exceeds configured concurrent fetch limit.
|
||
- Log remains append-only (tombstones are records, not deletions).
|
||
|
||
## Verify
|
||
|
||
**Harness:** `VecConnector` with mutable document list, counting embedder.
|
||
|
||
**Integration test** — `tests/it_sync_framework.rs`:
|
||
1. `a1_initial_sync_all_new` — 3 docs, no manifest; assert all 3 fetched and
|
||
embedded, manifest written with 3 entries.
|
||
2. `a2_unchanged_skipped` — sync again with same content; assert zero fetch
|
||
calls, zero embed calls.
|
||
3. `a3_changed_doc_replaced` — modify one doc's content; sync; assert old chunks
|
||
tombstoned, new chunks embedded, embed count equals changed doc's chunk count.
|
||
4. `a4_removed_doc_tombstoned` — remove a doc from connector; sync; assert
|
||
tombstone records emitted, manifest entry removed.
|
||
5. `a5_new_doc_added` — add a doc to connector; sync; assert only new doc
|
||
fetched and embedded.
|
||
6. `a6_drift_report_read_only` — modify docs, run drift report; assert correct
|
||
counts (1 new, 1 changed, 1 unchanged, 1 removed); assert no mutations to
|
||
manifest or log.
|
||
7. `a7_resume_after_crash` — sync 5 docs, simulate crash after 3; restart;
|
||
assert only 2 remaining docs processed.
|
||
8. `a8_tombstone_is_append` — count log lines before and after remove; assert
|
||
count only grew.
|
||
9. `a9_manifest_roundtrip` — save manifest, load it; assert field-for-field
|
||
equality.
|
||
10. `a10_rebuild_parity` — after full sync, `mem rebuild --from-log` produces
|
||
identical state.
|
||
|
||
**Command:** `cargo test --test it_sync_framework`
|
||
|
||
**False pass:**
|
||
- Asserting "no duplicate rows" instead of counting embed calls. A sync that
|
||
re-embeds everything and upserts by sha produces correct rows and wasted
|
||
compute.
|
||
- Testing drift report without actually changing documents first.
|
||
|
||
## Traps
|
||
|
||
- Comparing document-level hashes instead of chunk-level. A doc that changed
|
||
one paragraph should re-embed only the affected chunks, not all of them.
|
||
However, heading-boundary chunking means changing one heading can shift all
|
||
subsequent chunks. Accept document-level granularity for now; chunk-level
|
||
optimization is a future refinement.
|
||
- Making the manifest a database table instead of JSONL. The manifest must
|
||
survive `mem rebuild --from-log` — it is metadata about the sync process,
|
||
not a projection of the log.
|
||
- Running fetch + embed serially. A connector with 500 docs at 200ms per embed
|
||
takes 100s serially. Concurrent fetch + sequential embed is the right shape.
|
||
- Ignoring the derived filter (M4.2). Document connectors produce Level R content
|
||
that must register in the artifact manifest so it cannot re-enter as evidence.
|
||
|
||
---
|
||
|
||
Background: [DESIGN.md](../DESIGN.md) — source connectors, sync framework
|