98 lines
3.8 KiB
Markdown
98 lines
3.8 KiB
Markdown
|
# M7.2 — Obsidian vault connector
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | M7 — Source connectors |
|
|||
|
|
| Size | M — 1–3 days |
|
|||
|
|
| Status | ⬜ Not started |
|
|||
|
|
| Flags | — |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | M7.10 |
|
|||
|
|
| Depends | M7.1, M3.6.1 |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Wrap the existing `DocCorpusSource` (M3.6.1) in the `SourceConnector` interface
|
|||
|
|
so the Obsidian vault is managed through the unified connector framework — with
|
|||
|
|
change detection, config-driven setup, and registry integration.
|
|||
|
|
|
|||
|
|
## Facts (inlined — no spec read needed)
|
|||
|
|
|
|||
|
|
The Obsidian vault is the **primary reference source** for the cluster. It is
|
|||
|
|
human-maintained, optionally git-backed, and contains runbooks, procedures, and
|
|||
|
|
domain knowledge that agents query through the memory service.
|
|||
|
|
|
|||
|
|
`DocCorpusSource` already handles markdown parsing, heading-boundary chunking,
|
|||
|
|
breadcrumb paths, and file filtering. This task wraps it, not rewrites it.
|
|||
|
|
|
|||
|
|
**Deployment models:**
|
|||
|
|
1. **Git-sync sidecar** — a sidecar container clones the vault repo into a shared
|
|||
|
|
PVC. Memory service reads from the PVC via this connector.
|
|||
|
|
2. **Local mount** — for development, mount the vault directory directly.
|
|||
|
|
3. **PVC direct** — vault files managed via kubectl cp or a web uploader.
|
|||
|
|
|
|||
|
|
**Configuration:**
|
|||
|
|
```yaml
|
|||
|
|
connectors:
|
|||
|
|
- kind: obsidian
|
|||
|
|
name: homelab-vault
|
|||
|
|
config:
|
|||
|
|
root: /data/vault
|
|||
|
|
extensions: [md, markdown, txt]
|
|||
|
|
exclude_dirs: [.obsidian, .trash, .git]
|
|||
|
|
max_file_size: 10485760 # 10MB
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. Implement `ObsidianConnector` in `mem-ingest/src/connectors/obsidian.rs`.
|
|||
|
|
2. `list_documents()` — walk `root` directory, filter by extension, compute
|
|||
|
|
sha256 per file, return `SourceDocument` per file.
|
|||
|
|
3. `fetch_document()` — read file content, return as `DocumentContent` with
|
|||
|
|
metadata (file path, last modified, size).
|
|||
|
|
4. `health_check()` — verify `root` exists, is readable, count files.
|
|||
|
|
5. Register `"obsidian"` kind in the connector registry factory.
|
|||
|
|
6. `source_type()` returns `Reference` (vault docs bypass the gated loop).
|
|||
|
|
7. Reuse `DocCorpusSource` internals for heading-based chunking when the sync
|
|||
|
|
framework (M7.6) processes this connector's documents.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
- `ObsidianConnector` implements `SourceConnector` fully.
|
|||
|
|
- `list_documents()` respects `extensions`, `exclude_dirs`, `max_file_size`.
|
|||
|
|
- `fetch_document()` returns content matching file on disk.
|
|||
|
|
- `health_check()` distinguishes readable vs. missing root directory.
|
|||
|
|
- Config-driven: changing `root` path in YAML changes what gets scanned.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Harness:** fixture directory with markdown files, config YAML.
|
|||
|
|
|
|||
|
|
**Integration test** — `tests/it_obsidian_connector.rs`:
|
|||
|
|
1. `a1_list_filters_extensions` — fixture with .md, .txt, .json; assert only
|
|||
|
|
.md and .txt are listed.
|
|||
|
|
2. `a2_list_excludes_dirs` — fixture with `.obsidian/` subdir; assert its files
|
|||
|
|
are excluded.
|
|||
|
|
3. `a3_fetch_returns_content` — fetch a known doc; assert text matches file.
|
|||
|
|
4. `a4_fetch_unknown_errors` — fetch non-existent doc_id; assert error.
|
|||
|
|
5. `a5_health_check_reachable` — valid root; assert `reachable: true` with count.
|
|||
|
|
6. `a6_health_check_missing_root` — non-existent root; assert `reachable: false`.
|
|||
|
|
7. `a7_content_hash_stable` — fetch same file twice; assert same hash.
|
|||
|
|
8. `a8_config_from_yaml` — parse connector from YAML; assert fields match.
|
|||
|
|
|
|||
|
|
**Command:** `cargo test --test it_obsidian_connector`
|
|||
|
|
|
|||
|
|
**False pass:**
|
|||
|
|
- Testing with an empty directory. Assertions 1–3 need real files.
|
|||
|
|
- Not testing `exclude_dirs` with nested paths (`.obsidian/plugins/x.md`).
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- Re-implementing markdown parsing instead of delegating to `DocCorpusSource`.
|
|||
|
|
- Making `doc_id` platform-dependent (use relative path from root, unix separators).
|
|||
|
|
- Ignoring symlinks — Obsidian uses them for multi-vault setups.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background: [DESIGN.md](../DESIGN.md) — source connectors, Obsidian vault as connector
|