93 lines
3.7 KiB
Markdown
93 lines
3.7 KiB
Markdown
|
# M7.10 — M7 composition gate — source connectors
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | M7 — Source connectors |
|
|||
|
|
| Size | M — 1–3 days |
|
|||
|
|
| Status | ⬜ Not started |
|
|||
|
|
| Flags | gate |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | — |
|
|||
|
|
| Depends | M7.1, M7.2, M7.3, M7.6, M7.7, M7.8, M7.9 |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Prove the connector framework composes: two different connector kinds sync
|
|||
|
|
through the same framework, change detection skips unchanged documents, tombstoning
|
|||
|
|
works, drift reports are accurate, rebuild parity holds, and the gated loop's
|
|||
|
|
update-rate is untouched.
|
|||
|
|
|
|||
|
|
## What the gate proves
|
|||
|
|
|
|||
|
|
1. **Extensibility works.** Two different connector kinds (at minimum: obsidian +
|
|||
|
|
one remote connector) register, sync, and produce queryable Level R content
|
|||
|
|
through the same `SyncEngine`. No connector-specific code exists outside the
|
|||
|
|
connector itself.
|
|||
|
|
|
|||
|
|
2. **Change detection is efficient.** Re-syncing an unchanged connector produces
|
|||
|
|
zero embedding calls. This is the cost guard — without it, every sync is a
|
|||
|
|
full re-embed.
|
|||
|
|
|
|||
|
|
3. **Tombstoning is correct.** Removing a document from a source results in
|
|||
|
|
tombstone records in the log, removal from the index, and correct behavior
|
|||
|
|
on rebuild.
|
|||
|
|
|
|||
|
|
4. **Drift report is read-only.** Running `mem source status` mutates nothing.
|
|||
|
|
|
|||
|
|
5. **Rebuild parity holds for connectors.** `mem rebuild --from-log` with
|
|||
|
|
connector-sourced data produces byte-identical results.
|
|||
|
|
|
|||
|
|
6. **Update-rate is untouched.** Adding connector-sourced reference documents
|
|||
|
|
does not change the gated loop's update-rate (same property M3.6.6 asserts,
|
|||
|
|
now for any connector).
|
|||
|
|
|
|||
|
|
7. **Health monitoring detects failures.** An unreachable connector is reported,
|
|||
|
|
not silently ignored.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Integration test** — `tests/it_m7_gate.rs`:
|
|||
|
|
|
|||
|
|
1. `a1_two_kinds_sync` — register an obsidian + vec connector; sync both; assert
|
|||
|
|
Level R nodes exist for both sources.
|
|||
|
|
2. `a2_unchanged_zero_embeds` — re-sync both; assert zero embedding calls.
|
|||
|
|
3. `a3_change_detected_and_replaced` — modify a doc in one connector; sync;
|
|||
|
|
assert old chunks tombstoned, new chunks present.
|
|||
|
|
4. `a4_removal_tombstoned` — remove a doc; sync; assert tombstone records and
|
|||
|
|
doc absent from query results.
|
|||
|
|
5. `a5_drift_report_is_read_only` — snapshot log + manifest; run status; assert
|
|||
|
|
unchanged.
|
|||
|
|
6. `a6_rebuild_parity` — after full sync, rebuild from log; assert byte-identical
|
|||
|
|
state.
|
|||
|
|
7. `a7_update_rate_untouched` — record update-rate before adding connectors;
|
|||
|
|
add connectors + sync; re-run gated loop; assert update-rate unchanged.
|
|||
|
|
8. `a8_health_failure_reported` — configure unreachable connector; assert
|
|||
|
|
health check reports failure.
|
|||
|
|
9. `a9_no_connector_specific_code_in_sync` — assert `SyncEngine` has no
|
|||
|
|
`match kind` or `if kind ==` statements (the trait is the dispatch, not
|
|||
|
|
the framework).
|
|||
|
|
10. `a10_query_returns_connector_content` — sync a connector; query its content;
|
|||
|
|
assert results include connector-sourced Level R nodes with correct source_uri.
|
|||
|
|
|
|||
|
|
**Command:** `cargo test --test it_m7_gate`
|
|||
|
|
|
|||
|
|
**False pass:**
|
|||
|
|
- Testing with only one connector kind. The gate's value is proving two
|
|||
|
|
*different* kinds work through the same framework.
|
|||
|
|
- Testing rebuild parity without connector data in the log. An empty log
|
|||
|
|
trivially rebuilds.
|
|||
|
|
- Asserting update-rate "is still below 30%" instead of "is unchanged". Adding
|
|||
|
|
reference docs should not move the number at all.
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- Running the gate before M7.6 (sync framework) is solid. The gate tests
|
|||
|
|
composition; if the sync framework has bugs, every gate assertion fails
|
|||
|
|
for the wrong reason.
|
|||
|
|
- Not testing with a connector that produces multiple chunks per document.
|
|||
|
|
Single-chunk documents hide change-detection bugs at the chunk level.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background: [DESIGN.md](../DESIGN.md) — source connectors, composition gates
|