Story Crater Bot
71ecf482e7
- M7.1-M7.10: Extensible SourceConnector trait, Obsidian/paperless/git/S3 connectors, sync framework, CLI, HTTP endpoints, health monitoring, gate - M3.5.10: Auth integration with Authentik OIDC → Vault token validation - DESIGN.md: Add source connectors architecture, update auth to Authentik/Vault (Kong removed from cluster) - INDEX.md: 75 tasks, 11 gates - Fix all Kong references in M3.5.1 task
93 lines
3.7 KiB
Markdown
93 lines
3.7 KiB
Markdown
# M7.10 — M7 composition gate — source connectors
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M7 — Source connectors |
|
||
| Size | M — 1–3 days |
|
||
| Status | ⬜ Not started |
|
||
| Flags | gate |
|
||
| Spec | inlined below |
|
||
| Blocks | — |
|
||
| Depends | M7.1, M7.2, M7.3, M7.6, M7.7, M7.8, M7.9 |
|
||
|
||
## Goal
|
||
|
||
Prove the connector framework composes: two different connector kinds sync
|
||
through the same framework, change detection skips unchanged documents, tombstoning
|
||
works, drift reports are accurate, rebuild parity holds, and the gated loop's
|
||
update-rate is untouched.
|
||
|
||
## What the gate proves
|
||
|
||
1. **Extensibility works.** Two different connector kinds (at minimum: obsidian +
|
||
one remote connector) register, sync, and produce queryable Level R content
|
||
through the same `SyncEngine`. No connector-specific code exists outside the
|
||
connector itself.
|
||
|
||
2. **Change detection is efficient.** Re-syncing an unchanged connector produces
|
||
zero embedding calls. This is the cost guard — without it, every sync is a
|
||
full re-embed.
|
||
|
||
3. **Tombstoning is correct.** Removing a document from a source results in
|
||
tombstone records in the log, removal from the index, and correct behavior
|
||
on rebuild.
|
||
|
||
4. **Drift report is read-only.** Running `mem source status` mutates nothing.
|
||
|
||
5. **Rebuild parity holds for connectors.** `mem rebuild --from-log` with
|
||
connector-sourced data produces byte-identical results.
|
||
|
||
6. **Update-rate is untouched.** Adding connector-sourced reference documents
|
||
does not change the gated loop's update-rate (same property M3.6.6 asserts,
|
||
now for any connector).
|
||
|
||
7. **Health monitoring detects failures.** An unreachable connector is reported,
|
||
not silently ignored.
|
||
|
||
## Verify
|
||
|
||
**Integration test** — `tests/it_m7_gate.rs`:
|
||
|
||
1. `a1_two_kinds_sync` — register an obsidian + vec connector; sync both; assert
|
||
Level R nodes exist for both sources.
|
||
2. `a2_unchanged_zero_embeds` — re-sync both; assert zero embedding calls.
|
||
3. `a3_change_detected_and_replaced` — modify a doc in one connector; sync;
|
||
assert old chunks tombstoned, new chunks present.
|
||
4. `a4_removal_tombstoned` — remove a doc; sync; assert tombstone records and
|
||
doc absent from query results.
|
||
5. `a5_drift_report_is_read_only` — snapshot log + manifest; run status; assert
|
||
unchanged.
|
||
6. `a6_rebuild_parity` — after full sync, rebuild from log; assert byte-identical
|
||
state.
|
||
7. `a7_update_rate_untouched` — record update-rate before adding connectors;
|
||
add connectors + sync; re-run gated loop; assert update-rate unchanged.
|
||
8. `a8_health_failure_reported` — configure unreachable connector; assert
|
||
health check reports failure.
|
||
9. `a9_no_connector_specific_code_in_sync` — assert `SyncEngine` has no
|
||
`match kind` or `if kind ==` statements (the trait is the dispatch, not
|
||
the framework).
|
||
10. `a10_query_returns_connector_content` — sync a connector; query its content;
|
||
assert results include connector-sourced Level R nodes with correct source_uri.
|
||
|
||
**Command:** `cargo test --test it_m7_gate`
|
||
|
||
**False pass:**
|
||
- Testing with only one connector kind. The gate's value is proving two
|
||
*different* kinds work through the same framework.
|
||
- Testing rebuild parity without connector data in the log. An empty log
|
||
trivially rebuilds.
|
||
- Asserting update-rate "is still below 30%" instead of "is unchanged". Adding
|
||
reference docs should not move the number at all.
|
||
|
||
## Traps
|
||
|
||
- Running the gate before M7.6 (sync framework) is solid. The gate tests
|
||
composition; if the sync framework has bugs, every gate assertion fails
|
||
for the wrong reason.
|
||
- Not testing with a connector that produces multiple chunks per document.
|
||
Single-chunk documents hide change-detection bugs at the chunk level.
|
||
|
||
---
|
||
|
||
Background: [DESIGN.md](../DESIGN.md) — source connectors, composition gates
|