poimen-memory/tasks/M7.6-sync-framework.md

M7.6 — Sync framework

Field	Value
Phase	M7 — Source connectors
Size	L — 3–5 days
Status	⬜ Not started
Flags	—
Spec	inlined below
Blocks	M7.7, M7.10
Depends	M7.1, M3.6.3

Goal

Build the shared sync engine that handles change detection, tombstoning, drift reporting, and resumable sync for all connectors — so individual connector implementations only fetch documents and everything else is handled once.

Facts (inlined — no spec read needed)

Every connector faces the same sync problems:

1. What changed? Compare content hashes from list_documents() against a manifest of last-known hashes. Only fetch and process changed documents.
2. What disappeared? Documents in the manifest but not in list_documents() need tombstone records in the log. Append-only — never delete.
3. What if sync crashes midway? Track progress per-document. Restart processes only unseen documents.
4. How much will this cost? Drift report shows counts without mutating anything.

This is the generalization of M3.6.3's mem ref sync/list/rm logic, applied to any SourceConnector instead of just DocCorpusSource.

Manifest storage. Per-connector manifest in log/connectors/<name>.manifest.jsonl:

{"doc_id":"abc","source_uri":"file:///vault/k8s.md","content_hash":"sha256:...","chunk_count":12,"synced_at":"..."}
{"doc_id":"def","source_uri":"paperless://doc/42","content_hash":"sha256:...","chunk_count":3,"synced_at":"..."}

Sync algorithm:

current = connector.list_documents()
previous = load_manifest(connector.name)

for doc in current:
  if doc.content_hash == previous[doc.doc_id].content_hash:
    skip (unchanged)
  else if doc.doc_id in previous:
    tombstone previous chunks, fetch + chunk + embed new (changed)
  else:
    fetch + chunk + embed (new)

for doc_id in previous not in current:
  tombstone all chunks (removed)

save_manifest(connector.name, current)

Chunking delegation. The sync framework owns the chunking step. It routes fetched DocumentContent through the appropriate ChunkPolicy:

• Markdown → heading-boundary chunking (reuse DocCorpusSource logic)
• Plain text → paragraph-boundary chunking
• Configurable per connector kind in connectors.yaml

Level routing. Session connectors → RecordSource → gated loop (L0/L1/L2). Document connectors → Reference records (Level R). The source_type() method on the connector determines the pipeline.

Steps

1. Define SyncEngine struct in mem-ingest/src/sync.rs.
2. Implement manifest loading/saving (JSONL per connector).
3. Implement diff algorithm: (new, changed, unchanged, removed) from list_documents() vs manifest.
4. Implement sync loop: for each new/changed doc, fetch → chunk → emit records. For each removed doc, emit tombstones.
5. Implement drift reporting: same diff algorithm, print counts, no mutations.
6. Implement resume: track synced doc_ids in a progress file. On restart, skip already-synced docs.
7. Implement rate limiting: configurable max concurrent fetches per connector.
8. Integrate with mem-store for writing Reference records to the log.
9. Integrate with mem-llm for embedding new chunks.

Acceptance

• Sync of unchanged connector produces zero embedding calls.
• Changed document: old chunks tombstoned, new chunks embedded and stored.
• Removed document: chunks tombstoned, manifest updated.
• Drift report matches actual changes without mutating anything.
• Crash mid-sync → restart processes only remaining documents.
• Rate limiting: never exceeds configured concurrent fetch limit.
• Log remains append-only (tombstones are records, not deletions).

Verify

Harness: VecConnector with mutable document list, counting embedder.

Integration test — tests/it_sync_framework.rs:

1. a1_initial_sync_all_new — 3 docs, no manifest; assert all 3 fetched and embedded, manifest written with 3 entries.
2. a2_unchanged_skipped — sync again with same content; assert zero fetch calls, zero embed calls.
3. a3_changed_doc_replaced — modify one doc's content; sync; assert old chunks tombstoned, new chunks embedded, embed count equals changed doc's chunk count.
4. a4_removed_doc_tombstoned — remove a doc from connector; sync; assert tombstone records emitted, manifest entry removed.
5. a5_new_doc_added — add a doc to connector; sync; assert only new doc fetched and embedded.
6. a6_drift_report_read_only — modify docs, run drift report; assert correct counts (1 new, 1 changed, 1 unchanged, 1 removed); assert no mutations to manifest or log.
7. a7_resume_after_crash — sync 5 docs, simulate crash after 3; restart; assert only 2 remaining docs processed.
8. a8_tombstone_is_append — count log lines before and after remove; assert count only grew.
9. a9_manifest_roundtrip — save manifest, load it; assert field-for-field equality.
10. a10_rebuild_parity — after full sync, mem rebuild --from-log produces identical state.

Command: cargo test --test it_sync_framework

False pass:

• Asserting "no duplicate rows" instead of counting embed calls. A sync that re-embeds everything and upserts by sha produces correct rows and wasted compute.
• Testing drift report without actually changing documents first.

Traps

• Comparing document-level hashes instead of chunk-level. A doc that changed one paragraph should re-embed only the affected chunks, not all of them. However, heading-boundary chunking means changing one heading can shift all subsequent chunks. Accept document-level granularity for now; chunk-level optimization is a future refinement.
• Making the manifest a database table instead of JSONL. The manifest must survive mem rebuild --from-log — it is metadata about the sync process, not a projection of the log.
• Running fetch + embed serially. A connector with 500 docs at 200ms per embed takes 100s serially. Concurrent fetch + sequential embed is the right shape.
• Ignoring the derived filter (M4.2). Document connectors produce Level R content that must register in the artifact manifest so it cannot re-enter as evidence.

---

Background: DESIGN.md — source connectors, sync framework