poimen-memory/tasks/M7.1-source-connector-trait.md

M7.1 — SourceConnector trait + registry

Field	Value
Phase	M7 — Source connectors
Size	M — 1–3 days
Status	⬜ Not started
Flags	—
Spec	inlined below
Blocks	M7.2, M7.3, M7.4, M7.5, M7.6, M7.10
Depends	M0.3, M3.6.1

Goal

Define the extensible connector interface so that adding a new document source (paperless-ngx, S3, git repo, etc.) requires implementing one trait and adding one YAML config block — no changes to the ingest pipeline, chunking, embedding, storage, or query layers.

Facts (inlined — no spec read needed)

Memory service is a cluster-wide RAG serving multiple agents. Knowledge lives in many places: an Obsidian vault, paperless-ngx, git repositories, S3 buckets. The connector abstraction makes all of them look the same to the ingest pipeline.

The trait has three methods. list_documents() enumerates what's available without fetching content. fetch_document() retrieves one document's text. health_check() reports reachability. The sync framework (M7.6) handles everything else — change detection, tombstoning, chunking, embedding.

Configuration is YAML-driven. Each connector instance is a block in connectors.yaml with kind, name, and source-specific config. The registry maps kind to a factory function that constructs the connector from its config.

Two connector families exist. Session connectors (pi, claude) produce evidence for the gated loop (L0/L1/L2). Document connectors (obsidian, paperless, git, s3) produce reference material at Level R, bypassing the gate. The connector's source_type() method declares which family it belongs to.

Steps

1. Define SourceConnector trait in mem-ingest/src/connector.rs:

   #[async_trait]
   pub trait SourceConnector: Send + Sync {
       fn kind(&self) -> &str;
       fn name(&self) -> &str;
       fn source_type(&self) -> SourceType;  // Evidence or Reference
       async fn list_documents(&self) -> Result<Vec<SourceDocument>>;
       async fn fetch_document(&self, doc_id: &str) -> Result<DocumentContent>;
       async fn health_check(&self) -> Result<SourceHealth>;
   }
   

2. Define supporting types: SourceDocument, DocumentContent, SourceHealth, SourceType enum (Evidence, Reference).
3. Define ConnectorConfig serde struct for YAML deserialization:

   connectors:
     - kind: obsidian
       name: homelab-vault
       config: { root: /data/vault, extensions: [md, txt] }
   

4. Implement ConnectorRegistry — maps kind string to factory function, constructs connectors from config at startup.
5. Add connectors.yaml loading in mem-cli startup path.
6. Provide a VecConnector test helper (in-memory documents) for testing downstream consumers without real I/O.

Acceptance

• The trait compiles and is object-safe (Box<dyn SourceConnector>).
• ConnectorRegistry can register and construct connectors by kind string.
• VecConnector implements the trait and passes basic list/fetch assertions.
• YAML config deserialization works for known and unknown kinds (unknown = skip with warning, not crash).
• source_type() is enforced at the type level — no runtime flag confusion.

Verify

Harness: in-memory VecConnector, YAML config fixtures.

Integration test — tests/it_source_connector.rs:

1. a1_trait_is_object_safe — construct a Box<dyn SourceConnector> from VecConnector; call all three methods.
2. a2_registry_constructs_by_kind — register "vec" kind, construct from config, assert kind() and name() match.
3. a3_list_documents_returns_all — VecConnector with 3 docs, assert list_documents() returns 3.
4. a4_fetch_document_by_id — assert content matches what was registered.
5. a5_fetch_unknown_id_errors — assert fetch_document("nonexistent") returns an error, not a panic.
6. a6_health_check_reports_count — assert health_check() returns document_count = Some(3).
7. a7_yaml_config_loads — parse a fixture connectors.yaml with two connector blocks; assert both are constructed.
8. a8_unknown_kind_skipped — config with kind: "nonexistent"; assert registry logs a warning and continues without the connector.
9. a9_source_type_evidence_vs_reference — assert session connectors return Evidence, document connectors return Reference.
10. a10_empty_config_is_valid — no connectors.yaml or empty file; registry starts with zero connectors, no crash.

Command: cargo test --test it_source_connector

False pass:

• Testing only VecConnector and claiming the trait works. The trait is validated by M7.2–M7.5 implementing it against real sources.
• Parsing YAML without verifying the constructed connector's methods work.

Traps

• Making the trait not object-safe (generic methods, Self in return types). Every consumer stores Box<dyn SourceConnector>, so object safety is load-bearing.
• Putting chunking logic inside the connector. Connectors fetch documents; the sync framework (M7.6) chunks them. Mixing concerns means every connector reimplements chunking.
• Hard-coding the list of known kinds. The registry must be extensible — a register(kind, factory_fn) call, not a match statement.

---

Background: DESIGN.md — source connectors section