tasks/M7.6-sync-framework.md

# M7.6 — Sync framework

| Field | Value |
|---|---|
| Phase | M7 — Source connectors |
| Size | L — 3–5 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M7.7, M7.10 |
| Depends | M7.1, M3.6.3 |

## Goal

Build the shared sync engine that handles change detection, tombstoning, drift
reporting, and resumable sync for **all** connectors — so individual connector
implementations only fetch documents and everything else is handled once.

## Facts (inlined — no spec read needed)

Every connector faces the same sync problems:
1. **What changed?** Compare content hashes from `list_documents()` against a
   manifest of last-known hashes. Only fetch and process changed documents.
2. **What disappeared?** Documents in the manifest but not in `list_documents()`
   need tombstone records in the log. Append-only — never delete.
3. **What if sync crashes midway?** Track progress per-document. Restart
   processes only unseen documents.
4. **How much will this cost?** Drift report shows counts without mutating
   anything.

This is the generalization of M3.6.3's `mem ref sync/list/rm` logic, applied to
any `SourceConnector` instead of just `DocCorpusSource`.

**Manifest storage.** Per-connector manifest in `log/connectors/<name>.manifest.jsonl`:
```jsonl
{"doc_id":"abc","source_uri":"file:///vault/k8s.md","content_hash":"sha256:...","chunk_count":12,"synced_at":"..."}
{"doc_id":"def","source_uri":"paperless://doc/42","content_hash":"sha256:...","chunk_count":3,"synced_at":"..."}
```

**Sync algorithm:**
```
current = connector.list_documents()
previous = load_manifest(connector.name)

for doc in current:
  if doc.content_hash == previous[doc.doc_id].content_hash:
    skip (unchanged)
  else if doc.doc_id in previous:
    tombstone previous chunks, fetch + chunk + embed new (changed)
  else:
    fetch + chunk + embed (new)

for doc_id in previous not in current:
  tombstone all chunks (removed)

save_manifest(connector.name, current)
```

**Chunking delegation.** The sync framework owns the chunking step. It routes
fetched `DocumentContent` through the appropriate `ChunkPolicy`:
- Markdown → heading-boundary chunking (reuse `DocCorpusSource` logic)
- Plain text → paragraph-boundary chunking
- Configurable per connector kind in `connectors.yaml`

**Level routing.** Session connectors → RecordSource → gated loop (L0/L1/L2).
Document connectors → Reference records (Level R). The `source_type()` method
on the connector determines the pipeline.

## Steps

1. Define `SyncEngine` struct in `mem-ingest/src/sync.rs`.
2. Implement manifest loading/saving (JSONL per connector).
3. Implement diff algorithm: `(new, changed, unchanged, removed)` from
   `list_documents()` vs manifest.
4. Implement sync loop: for each `new`/`changed` doc, fetch → chunk → emit
   records. For each `removed` doc, emit tombstones.
5. Implement drift reporting: same diff algorithm, print counts, no mutations.
6. Implement resume: track synced doc_ids in a progress file. On restart,
   skip already-synced docs.
7. Implement rate limiting: configurable max concurrent fetches per connector.
8. Integrate with `mem-store` for writing Reference records to the log.
9. Integrate with `mem-llm` for embedding new chunks.

## Acceptance

- Sync of unchanged connector produces zero embedding calls.
- Changed document: old chunks tombstoned, new chunks embedded and stored.
- Removed document: chunks tombstoned, manifest updated.
- Drift report matches actual changes without mutating anything.
- Crash mid-sync → restart processes only remaining documents.
- Rate limiting: never exceeds configured concurrent fetch limit.
- Log remains append-only (tombstones are records, not deletions).

## Verify

**Harness:** `VecConnector` with mutable document list, counting embedder.

**Integration test** — `tests/it_sync_framework.rs`:
1. `a1_initial_sync_all_new` — 3 docs, no manifest; assert all 3 fetched and
   embedded, manifest written with 3 entries.
2. `a2_unchanged_skipped` — sync again with same content; assert zero fetch
   calls, zero embed calls.
3. `a3_changed_doc_replaced` — modify one doc's content; sync; assert old chunks
   tombstoned, new chunks embedded, embed count equals changed doc's chunk count.
4. `a4_removed_doc_tombstoned` — remove a doc from connector; sync; assert
   tombstone records emitted, manifest entry removed.
5. `a5_new_doc_added` — add a doc to connector; sync; assert only new doc
   fetched and embedded.
6. `a6_drift_report_read_only` — modify docs, run drift report; assert correct
   counts (1 new, 1 changed, 1 unchanged, 1 removed); assert no mutations to
   manifest or log.
7. `a7_resume_after_crash` — sync 5 docs, simulate crash after 3; restart;
   assert only 2 remaining docs processed.
8. `a8_tombstone_is_append` — count log lines before and after remove; assert
   count only grew.
9. `a9_manifest_roundtrip` — save manifest, load it; assert field-for-field
   equality.
10. `a10_rebuild_parity` — after full sync, `mem rebuild --from-log` produces
    identical state.

**Command:** `cargo test --test it_sync_framework`

**False pass:**
- Asserting "no duplicate rows" instead of counting embed calls. A sync that
  re-embeds everything and upserts by sha produces correct rows and wasted
  compute.
- Testing drift report without actually changing documents first.

## Traps

- Comparing document-level hashes instead of chunk-level. A doc that changed
  one paragraph should re-embed only the affected chunks, not all of them.
  However, heading-boundary chunking means changing one heading can shift all
  subsequent chunks. Accept document-level granularity for now; chunk-level
  optimization is a future refinement.
- Making the manifest a database table instead of JSONL. The manifest must
  survive `mem rebuild --from-log` — it is metadata about the sync process,
  not a projection of the log.
- Running fetch + embed serially. A connector with 500 docs at 200ms per embed
  takes 100s serially. Concurrent fetch + sequential embed is the right shape.
- Ignoring the derived filter (M4.2). Document connectors produce Level R content
  that must register in the artifact manifest so it cannot re-enter as evidence.

---

Background: [DESIGN.md](../DESIGN.md) — source connectors, sync framework