docs: Add M7 source connectors (10 tasks), M3.5.10 auth integration, remove Kong refs
- M7.1-M7.10: Extensible SourceConnector trait, Obsidian/paperless/git/S3 connectors, sync framework, CLI, HTTP endpoints, health monitoring, gate - M3.5.10: Auth integration with Authentik OIDC → Vault token validation - DESIGN.md: Add source connectors architecture, update auth to Authentik/Vault (Kong removed from cluster) - INDEX.md: 75 tasks, 11 gates - Fix all Kong references in M3.5.1 task
This commit is contained in:
@@ -0,0 +1,122 @@
|
||||
# M7.1 — `SourceConnector` trait + registry
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M7 — Source connectors |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M7.2, M7.3, M7.4, M7.5, M7.6, M7.10 |
|
||||
| Depends | M0.3, M3.6.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Define the extensible connector interface so that adding a new document source
|
||||
(paperless-ngx, S3, git repo, etc.) requires implementing one trait and adding
|
||||
one YAML config block — no changes to the ingest pipeline, chunking, embedding,
|
||||
storage, or query layers.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
Memory service is a **cluster-wide RAG** serving multiple agents. Knowledge lives
|
||||
in many places: an Obsidian vault, paperless-ngx, git repositories, S3 buckets.
|
||||
The connector abstraction makes all of them look the same to the ingest pipeline.
|
||||
|
||||
**The trait has three methods.** `list_documents()` enumerates what's available
|
||||
without fetching content. `fetch_document()` retrieves one document's text.
|
||||
`health_check()` reports reachability. The sync framework (M7.6) handles
|
||||
everything else — change detection, tombstoning, chunking, embedding.
|
||||
|
||||
**Configuration is YAML-driven.** Each connector instance is a block in
|
||||
`connectors.yaml` with `kind`, `name`, and source-specific `config`. The registry
|
||||
maps `kind` to a factory function that constructs the connector from its config.
|
||||
|
||||
**Two connector families exist.** Session connectors (pi, claude) produce evidence
|
||||
for the gated loop (L0/L1/L2). Document connectors (obsidian, paperless, git, s3)
|
||||
produce reference material at Level R, bypassing the gate. The connector's
|
||||
`source_type()` method declares which family it belongs to.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Define `SourceConnector` trait in `mem-ingest/src/connector.rs`:
|
||||
```rust
|
||||
#[async_trait]
|
||||
pub trait SourceConnector: Send + Sync {
|
||||
fn kind(&self) -> &str;
|
||||
fn name(&self) -> &str;
|
||||
fn source_type(&self) -> SourceType; // Evidence or Reference
|
||||
async fn list_documents(&self) -> Result<Vec<SourceDocument>>;
|
||||
async fn fetch_document(&self, doc_id: &str) -> Result<DocumentContent>;
|
||||
async fn health_check(&self) -> Result<SourceHealth>;
|
||||
}
|
||||
```
|
||||
2. Define supporting types: `SourceDocument`, `DocumentContent`, `SourceHealth`,
|
||||
`SourceType` enum (`Evidence`, `Reference`).
|
||||
3. Define `ConnectorConfig` serde struct for YAML deserialization:
|
||||
```yaml
|
||||
connectors:
|
||||
- kind: obsidian
|
||||
name: homelab-vault
|
||||
config: { root: /data/vault, extensions: [md, txt] }
|
||||
```
|
||||
4. Implement `ConnectorRegistry` — maps `kind` string to factory function,
|
||||
constructs connectors from config at startup.
|
||||
5. Add `connectors.yaml` loading in `mem-cli` startup path.
|
||||
6. Provide a `VecConnector` test helper (in-memory documents) for testing
|
||||
downstream consumers without real I/O.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- The trait compiles and is object-safe (`Box<dyn SourceConnector>`).
|
||||
- `ConnectorRegistry` can register and construct connectors by kind string.
|
||||
- `VecConnector` implements the trait and passes basic list/fetch assertions.
|
||||
- YAML config deserialization works for known and unknown kinds (unknown = skip
|
||||
with warning, not crash).
|
||||
- `source_type()` is enforced at the type level — no runtime flag confusion.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** in-memory `VecConnector`, YAML config fixtures.
|
||||
|
||||
**Integration test** — `tests/it_source_connector.rs`:
|
||||
1. `a1_trait_is_object_safe` — construct a `Box<dyn SourceConnector>` from
|
||||
`VecConnector`; call all three methods.
|
||||
2. `a2_registry_constructs_by_kind` — register "vec" kind, construct from config,
|
||||
assert `kind()` and `name()` match.
|
||||
3. `a3_list_documents_returns_all` — `VecConnector` with 3 docs, assert
|
||||
`list_documents()` returns 3.
|
||||
4. `a4_fetch_document_by_id` — assert content matches what was registered.
|
||||
5. `a5_fetch_unknown_id_errors` — assert `fetch_document("nonexistent")` returns
|
||||
an error, not a panic.
|
||||
6. `a6_health_check_reports_count` — assert `health_check()` returns
|
||||
`document_count = Some(3)`.
|
||||
7. `a7_yaml_config_loads` — parse a fixture `connectors.yaml` with two connector
|
||||
blocks; assert both are constructed.
|
||||
8. `a8_unknown_kind_skipped` — config with `kind: "nonexistent"`; assert registry
|
||||
logs a warning and continues without the connector.
|
||||
9. `a9_source_type_evidence_vs_reference` — assert session connectors return
|
||||
`Evidence`, document connectors return `Reference`.
|
||||
10. `a10_empty_config_is_valid` — no `connectors.yaml` or empty file; registry
|
||||
starts with zero connectors, no crash.
|
||||
|
||||
**Command:** `cargo test --test it_source_connector`
|
||||
|
||||
**False pass:**
|
||||
- Testing only `VecConnector` and claiming the trait works. The trait is
|
||||
validated by M7.2–M7.5 implementing it against real sources.
|
||||
- Parsing YAML without verifying the constructed connector's methods work.
|
||||
|
||||
## Traps
|
||||
|
||||
- Making the trait not object-safe (generic methods, `Self` in return types).
|
||||
Every consumer stores `Box<dyn SourceConnector>`, so object safety is load-bearing.
|
||||
- Putting chunking logic inside the connector. Connectors fetch documents; the
|
||||
sync framework (M7.6) chunks them. Mixing concerns means every connector
|
||||
reimplements chunking.
|
||||
- Hard-coding the list of known kinds. The registry must be extensible — a
|
||||
`register(kind, factory_fn)` call, not a match statement.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — source connectors section
|
||||
Reference in New Issue
Block a user