docs: Add M7 source connectors (10 tasks), M3.5.10 auth integration, remove Kong refs
- M7.1-M7.10: Extensible SourceConnector trait, Obsidian/paperless/git/S3 connectors, sync framework, CLI, HTTP endpoints, health monitoring, gate - M3.5.10: Auth integration with Authentik OIDC → Vault token validation - DESIGN.md: Add source connectors architecture, update auth to Authentik/Vault (Kong removed from cluster) - INDEX.md: 75 tasks, 11 gates - Fix all Kong references in M3.5.1 task
This commit is contained in:
@@ -0,0 +1,97 @@
|
||||
# M7.2 — Obsidian vault connector
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M7 — Source connectors |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M7.10 |
|
||||
| Depends | M7.1, M3.6.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Wrap the existing `DocCorpusSource` (M3.6.1) in the `SourceConnector` interface
|
||||
so the Obsidian vault is managed through the unified connector framework — with
|
||||
change detection, config-driven setup, and registry integration.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
The Obsidian vault is the **primary reference source** for the cluster. It is
|
||||
human-maintained, optionally git-backed, and contains runbooks, procedures, and
|
||||
domain knowledge that agents query through the memory service.
|
||||
|
||||
`DocCorpusSource` already handles markdown parsing, heading-boundary chunking,
|
||||
breadcrumb paths, and file filtering. This task wraps it, not rewrites it.
|
||||
|
||||
**Deployment models:**
|
||||
1. **Git-sync sidecar** — a sidecar container clones the vault repo into a shared
|
||||
PVC. Memory service reads from the PVC via this connector.
|
||||
2. **Local mount** — for development, mount the vault directory directly.
|
||||
3. **PVC direct** — vault files managed via kubectl cp or a web uploader.
|
||||
|
||||
**Configuration:**
|
||||
```yaml
|
||||
connectors:
|
||||
- kind: obsidian
|
||||
name: homelab-vault
|
||||
config:
|
||||
root: /data/vault
|
||||
extensions: [md, markdown, txt]
|
||||
exclude_dirs: [.obsidian, .trash, .git]
|
||||
max_file_size: 10485760 # 10MB
|
||||
```
|
||||
|
||||
## Steps
|
||||
|
||||
1. Implement `ObsidianConnector` in `mem-ingest/src/connectors/obsidian.rs`.
|
||||
2. `list_documents()` — walk `root` directory, filter by extension, compute
|
||||
sha256 per file, return `SourceDocument` per file.
|
||||
3. `fetch_document()` — read file content, return as `DocumentContent` with
|
||||
metadata (file path, last modified, size).
|
||||
4. `health_check()` — verify `root` exists, is readable, count files.
|
||||
5. Register `"obsidian"` kind in the connector registry factory.
|
||||
6. `source_type()` returns `Reference` (vault docs bypass the gated loop).
|
||||
7. Reuse `DocCorpusSource` internals for heading-based chunking when the sync
|
||||
framework (M7.6) processes this connector's documents.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- `ObsidianConnector` implements `SourceConnector` fully.
|
||||
- `list_documents()` respects `extensions`, `exclude_dirs`, `max_file_size`.
|
||||
- `fetch_document()` returns content matching file on disk.
|
||||
- `health_check()` distinguishes readable vs. missing root directory.
|
||||
- Config-driven: changing `root` path in YAML changes what gets scanned.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** fixture directory with markdown files, config YAML.
|
||||
|
||||
**Integration test** — `tests/it_obsidian_connector.rs`:
|
||||
1. `a1_list_filters_extensions` — fixture with .md, .txt, .json; assert only
|
||||
.md and .txt are listed.
|
||||
2. `a2_list_excludes_dirs` — fixture with `.obsidian/` subdir; assert its files
|
||||
are excluded.
|
||||
3. `a3_fetch_returns_content` — fetch a known doc; assert text matches file.
|
||||
4. `a4_fetch_unknown_errors` — fetch non-existent doc_id; assert error.
|
||||
5. `a5_health_check_reachable` — valid root; assert `reachable: true` with count.
|
||||
6. `a6_health_check_missing_root` — non-existent root; assert `reachable: false`.
|
||||
7. `a7_content_hash_stable` — fetch same file twice; assert same hash.
|
||||
8. `a8_config_from_yaml` — parse connector from YAML; assert fields match.
|
||||
|
||||
**Command:** `cargo test --test it_obsidian_connector`
|
||||
|
||||
**False pass:**
|
||||
- Testing with an empty directory. Assertions 1–3 need real files.
|
||||
- Not testing `exclude_dirs` with nested paths (`.obsidian/plugins/x.md`).
|
||||
|
||||
## Traps
|
||||
|
||||
- Re-implementing markdown parsing instead of delegating to `DocCorpusSource`.
|
||||
- Making `doc_id` platform-dependent (use relative path from root, unix separators).
|
||||
- Ignoring symlinks — Obsidian uses them for multi-vault setups.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — source connectors, Obsidian vault as connector
|
||||
Reference in New Issue
Block a user