docs: Add M7 source connectors (10 tasks), M3.5.10 auth integration, remove Kong refs
Build and Push / Test (push) Successful in 3m36s
Build and Push / Build and push image (push) Successful in 20s

- M7.1-M7.10: Extensible SourceConnector trait, Obsidian/paperless/git/S3
  connectors, sync framework, CLI, HTTP endpoints, health monitoring, gate
- M3.5.10: Auth integration with Authentik OIDC → Vault token validation
- DESIGN.md: Add source connectors architecture, update auth to
  Authentik/Vault (Kong removed from cluster)
- INDEX.md: 75 tasks, 11 gates
- Fix all Kong references in M3.5.1 task
This commit is contained in:
Story Crater Bot
2026-08-26 16:56:39 -07:00
parent c8d754b0ba
commit 71ecf482e7
14 changed files with 1560 additions and 13 deletions
+92
View File
@@ -0,0 +1,92 @@
# M7.10 — M7 composition gate — source connectors
| Field | Value |
|---|---|
| Phase | M7 — Source connectors |
| Size | M — 13 days |
| Status | ⬜ Not started |
| Flags | gate |
| Spec | inlined below |
| Blocks | — |
| Depends | M7.1, M7.2, M7.3, M7.6, M7.7, M7.8, M7.9 |
## Goal
Prove the connector framework composes: two different connector kinds sync
through the same framework, change detection skips unchanged documents, tombstoning
works, drift reports are accurate, rebuild parity holds, and the gated loop's
update-rate is untouched.
## What the gate proves
1. **Extensibility works.** Two different connector kinds (at minimum: obsidian +
one remote connector) register, sync, and produce queryable Level R content
through the same `SyncEngine`. No connector-specific code exists outside the
connector itself.
2. **Change detection is efficient.** Re-syncing an unchanged connector produces
zero embedding calls. This is the cost guard — without it, every sync is a
full re-embed.
3. **Tombstoning is correct.** Removing a document from a source results in
tombstone records in the log, removal from the index, and correct behavior
on rebuild.
4. **Drift report is read-only.** Running `mem source status` mutates nothing.
5. **Rebuild parity holds for connectors.** `mem rebuild --from-log` with
connector-sourced data produces byte-identical results.
6. **Update-rate is untouched.** Adding connector-sourced reference documents
does not change the gated loop's update-rate (same property M3.6.6 asserts,
now for any connector).
7. **Health monitoring detects failures.** An unreachable connector is reported,
not silently ignored.
## Verify
**Integration test**`tests/it_m7_gate.rs`:
1. `a1_two_kinds_sync` — register an obsidian + vec connector; sync both; assert
Level R nodes exist for both sources.
2. `a2_unchanged_zero_embeds` — re-sync both; assert zero embedding calls.
3. `a3_change_detected_and_replaced` — modify a doc in one connector; sync;
assert old chunks tombstoned, new chunks present.
4. `a4_removal_tombstoned` — remove a doc; sync; assert tombstone records and
doc absent from query results.
5. `a5_drift_report_is_read_only` — snapshot log + manifest; run status; assert
unchanged.
6. `a6_rebuild_parity` — after full sync, rebuild from log; assert byte-identical
state.
7. `a7_update_rate_untouched` — record update-rate before adding connectors;
add connectors + sync; re-run gated loop; assert update-rate unchanged.
8. `a8_health_failure_reported` — configure unreachable connector; assert
health check reports failure.
9. `a9_no_connector_specific_code_in_sync` — assert `SyncEngine` has no
`match kind` or `if kind ==` statements (the trait is the dispatch, not
the framework).
10. `a10_query_returns_connector_content` — sync a connector; query its content;
assert results include connector-sourced Level R nodes with correct source_uri.
**Command:** `cargo test --test it_m7_gate`
**False pass:**
- Testing with only one connector kind. The gate's value is proving two
*different* kinds work through the same framework.
- Testing rebuild parity without connector data in the log. An empty log
trivially rebuilds.
- Asserting update-rate "is still below 30%" instead of "is unchanged". Adding
reference docs should not move the number at all.
## Traps
- Running the gate before M7.6 (sync framework) is solid. The gate tests
composition; if the sync framework has bugs, every gate assertion fails
for the wrong reason.
- Not testing with a connector that produces multiple chunks per document.
Single-chunk documents hide change-detection bugs at the chunk level.
---
Background: [DESIGN.md](../DESIGN.md) — source connectors, composition gates