poimen-memory/tasks/M7.1-source-connector-trait.md

# M7.1 — `SourceConnector` trait + registry

| Field | Value |
|---|---|
| Phase | M7 — Source connectors |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M7.2, M7.3, M7.4, M7.5, M7.6, M7.10 |
| Depends | M0.3, M3.6.1 |

## Goal

Define the extensible connector interface so that adding a new document source
(paperless-ngx, S3, git repo, etc.) requires implementing one trait and adding
one YAML config block — no changes to the ingest pipeline, chunking, embedding,
storage, or query layers.

## Facts (inlined — no spec read needed)

Memory service is a **cluster-wide RAG** serving multiple agents. Knowledge lives
in many places: an Obsidian vault, paperless-ngx, git repositories, S3 buckets.
The connector abstraction makes all of them look the same to the ingest pipeline.

**The trait has three methods.** `list_documents()` enumerates what's available
without fetching content. `fetch_document()` retrieves one document's text.
`health_check()` reports reachability. The sync framework (M7.6) handles
everything else — change detection, tombstoning, chunking, embedding.

**Configuration is YAML-driven.** Each connector instance is a block in
`connectors.yaml` with `kind`, `name`, and source-specific `config`. The registry
maps `kind` to a factory function that constructs the connector from its config.

**Two connector families exist.** Session connectors (pi, claude) produce evidence
for the gated loop (L0/L1/L2). Document connectors (obsidian, paperless, git, s3)
produce reference material at Level R, bypassing the gate. The connector's
`source_type()` method declares which family it belongs to.

## Steps

1. Define `SourceConnector` trait in `mem-ingest/src/connector.rs`:
   ```rust
   #[async_trait]
   pub trait SourceConnector: Send + Sync {
       fn kind(&self) -> &str;
       fn name(&self) -> &str;
       fn source_type(&self) -> SourceType;  // Evidence or Reference
       async fn list_documents(&self) -> Result<Vec<SourceDocument>>;
       async fn fetch_document(&self, doc_id: &str) -> Result<DocumentContent>;
       async fn health_check(&self) -> Result<SourceHealth>;
   }
   ```
2. Define supporting types: `SourceDocument`, `DocumentContent`, `SourceHealth`,
   `SourceType` enum (`Evidence`, `Reference`).
3. Define `ConnectorConfig` serde struct for YAML deserialization:
   ```yaml
   connectors:
     - kind: obsidian
       name: homelab-vault
       config: { root: /data/vault, extensions: [md, txt] }
   ```
4. Implement `ConnectorRegistry` — maps `kind` string to factory function,
   constructs connectors from config at startup.
5. Add `connectors.yaml` loading in `mem-cli` startup path.
6. Provide a `VecConnector` test helper (in-memory documents) for testing
   downstream consumers without real I/O.

## Acceptance

- The trait compiles and is object-safe (`Box<dyn SourceConnector>`).
- `ConnectorRegistry` can register and construct connectors by kind string.
- `VecConnector` implements the trait and passes basic list/fetch assertions.
- YAML config deserialization works for known and unknown kinds (unknown = skip
  with warning, not crash).
- `source_type()` is enforced at the type level — no runtime flag confusion.

## Verify

**Harness:** in-memory `VecConnector`, YAML config fixtures.

**Integration test** — `tests/it_source_connector.rs`:
1. `a1_trait_is_object_safe` — construct a `Box<dyn SourceConnector>` from
   `VecConnector`; call all three methods.
2. `a2_registry_constructs_by_kind` — register "vec" kind, construct from config,
   assert `kind()` and `name()` match.
3. `a3_list_documents_returns_all` — `VecConnector` with 3 docs, assert
   `list_documents()` returns 3.
4. `a4_fetch_document_by_id` — assert content matches what was registered.
5. `a5_fetch_unknown_id_errors` — assert `fetch_document("nonexistent")` returns
   an error, not a panic.
6. `a6_health_check_reports_count` — assert `health_check()` returns
   `document_count = Some(3)`.
7. `a7_yaml_config_loads` — parse a fixture `connectors.yaml` with two connector
   blocks; assert both are constructed.
8. `a8_unknown_kind_skipped` — config with `kind: "nonexistent"`; assert registry
   logs a warning and continues without the connector.
9. `a9_source_type_evidence_vs_reference` — assert session connectors return
   `Evidence`, document connectors return `Reference`.
10. `a10_empty_config_is_valid` — no `connectors.yaml` or empty file; registry
    starts with zero connectors, no crash.

**Command:** `cargo test --test it_source_connector`

**False pass:**
- Testing only `VecConnector` and claiming the trait works. The trait is
  validated by M7.2–M7.5 implementing it against real sources.
- Parsing YAML without verifying the constructed connector's methods work.

## Traps

- Making the trait not object-safe (generic methods, `Self` in return types).
  Every consumer stores `Box<dyn SourceConnector>`, so object safety is load-bearing.
- Putting chunking logic inside the connector. Connectors fetch documents; the
  sync framework (M7.6) chunks them. Mixing concerns means every connector
  reimplements chunking.
- Hard-coding the list of known kinds. The registry must be extensible — a
  `register(kind, factory_fn)` call, not a match statement.

---

Background: [DESIGN.md](../DESIGN.md) — source connectors section