docs: Add M7 source connectors (10 tasks), M3.5.10 auth integration, remove Kong refs
- M7.1-M7.10: Extensible SourceConnector trait, Obsidian/paperless/git/S3 connectors, sync framework, CLI, HTTP endpoints, health monitoring, gate - M3.5.10: Auth integration with Authentik OIDC → Vault token validation - DESIGN.md: Add source connectors architecture, update auth to Authentik/Vault (Kong removed from cluster) - INDEX.md: 75 tasks, 11 gates - Fix all Kong references in M3.5.1 task
This commit is contained in:
@@ -0,0 +1,92 @@
|
||||
# M7.10 — M7 composition gate — source connectors
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M7 — Source connectors |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | gate |
|
||||
| Spec | inlined below |
|
||||
| Blocks | — |
|
||||
| Depends | M7.1, M7.2, M7.3, M7.6, M7.7, M7.8, M7.9 |
|
||||
|
||||
## Goal
|
||||
|
||||
Prove the connector framework composes: two different connector kinds sync
|
||||
through the same framework, change detection skips unchanged documents, tombstoning
|
||||
works, drift reports are accurate, rebuild parity holds, and the gated loop's
|
||||
update-rate is untouched.
|
||||
|
||||
## What the gate proves
|
||||
|
||||
1. **Extensibility works.** Two different connector kinds (at minimum: obsidian +
|
||||
one remote connector) register, sync, and produce queryable Level R content
|
||||
through the same `SyncEngine`. No connector-specific code exists outside the
|
||||
connector itself.
|
||||
|
||||
2. **Change detection is efficient.** Re-syncing an unchanged connector produces
|
||||
zero embedding calls. This is the cost guard — without it, every sync is a
|
||||
full re-embed.
|
||||
|
||||
3. **Tombstoning is correct.** Removing a document from a source results in
|
||||
tombstone records in the log, removal from the index, and correct behavior
|
||||
on rebuild.
|
||||
|
||||
4. **Drift report is read-only.** Running `mem source status` mutates nothing.
|
||||
|
||||
5. **Rebuild parity holds for connectors.** `mem rebuild --from-log` with
|
||||
connector-sourced data produces byte-identical results.
|
||||
|
||||
6. **Update-rate is untouched.** Adding connector-sourced reference documents
|
||||
does not change the gated loop's update-rate (same property M3.6.6 asserts,
|
||||
now for any connector).
|
||||
|
||||
7. **Health monitoring detects failures.** An unreachable connector is reported,
|
||||
not silently ignored.
|
||||
|
||||
## Verify
|
||||
|
||||
**Integration test** — `tests/it_m7_gate.rs`:
|
||||
|
||||
1. `a1_two_kinds_sync` — register an obsidian + vec connector; sync both; assert
|
||||
Level R nodes exist for both sources.
|
||||
2. `a2_unchanged_zero_embeds` — re-sync both; assert zero embedding calls.
|
||||
3. `a3_change_detected_and_replaced` — modify a doc in one connector; sync;
|
||||
assert old chunks tombstoned, new chunks present.
|
||||
4. `a4_removal_tombstoned` — remove a doc; sync; assert tombstone records and
|
||||
doc absent from query results.
|
||||
5. `a5_drift_report_is_read_only` — snapshot log + manifest; run status; assert
|
||||
unchanged.
|
||||
6. `a6_rebuild_parity` — after full sync, rebuild from log; assert byte-identical
|
||||
state.
|
||||
7. `a7_update_rate_untouched` — record update-rate before adding connectors;
|
||||
add connectors + sync; re-run gated loop; assert update-rate unchanged.
|
||||
8. `a8_health_failure_reported` — configure unreachable connector; assert
|
||||
health check reports failure.
|
||||
9. `a9_no_connector_specific_code_in_sync` — assert `SyncEngine` has no
|
||||
`match kind` or `if kind ==` statements (the trait is the dispatch, not
|
||||
the framework).
|
||||
10. `a10_query_returns_connector_content` — sync a connector; query its content;
|
||||
assert results include connector-sourced Level R nodes with correct source_uri.
|
||||
|
||||
**Command:** `cargo test --test it_m7_gate`
|
||||
|
||||
**False pass:**
|
||||
- Testing with only one connector kind. The gate's value is proving two
|
||||
*different* kinds work through the same framework.
|
||||
- Testing rebuild parity without connector data in the log. An empty log
|
||||
trivially rebuilds.
|
||||
- Asserting update-rate "is still below 30%" instead of "is unchanged". Adding
|
||||
reference docs should not move the number at all.
|
||||
|
||||
## Traps
|
||||
|
||||
- Running the gate before M7.6 (sync framework) is solid. The gate tests
|
||||
composition; if the sync framework has bugs, every gate assertion fails
|
||||
for the wrong reason.
|
||||
- Not testing with a connector that produces multiple chunks per document.
|
||||
Single-chunk documents hide change-detection bugs at the chunk level.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — source connectors, composition gates
|
||||
Reference in New Issue
Block a user