Story Crater Bot
71ecf482e7
- M7.1-M7.10: Extensible SourceConnector trait, Obsidian/paperless/git/S3 connectors, sync framework, CLI, HTTP endpoints, health monitoring, gate - M3.5.10: Auth integration with Authentik OIDC → Vault token validation - DESIGN.md: Add source connectors architecture, update auth to Authentik/Vault (Kong removed from cluster) - INDEX.md: 75 tasks, 11 gates - Fix all Kong references in M3.5.1 task
98 lines
3.8 KiB
Markdown
98 lines
3.8 KiB
Markdown
# M7.2 — Obsidian vault connector
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M7 — Source connectors |
|
||
| Size | M — 1–3 days |
|
||
| Status | ⬜ Not started |
|
||
| Flags | — |
|
||
| Spec | inlined below |
|
||
| Blocks | M7.10 |
|
||
| Depends | M7.1, M3.6.1 |
|
||
|
||
## Goal
|
||
|
||
Wrap the existing `DocCorpusSource` (M3.6.1) in the `SourceConnector` interface
|
||
so the Obsidian vault is managed through the unified connector framework — with
|
||
change detection, config-driven setup, and registry integration.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
The Obsidian vault is the **primary reference source** for the cluster. It is
|
||
human-maintained, optionally git-backed, and contains runbooks, procedures, and
|
||
domain knowledge that agents query through the memory service.
|
||
|
||
`DocCorpusSource` already handles markdown parsing, heading-boundary chunking,
|
||
breadcrumb paths, and file filtering. This task wraps it, not rewrites it.
|
||
|
||
**Deployment models:**
|
||
1. **Git-sync sidecar** — a sidecar container clones the vault repo into a shared
|
||
PVC. Memory service reads from the PVC via this connector.
|
||
2. **Local mount** — for development, mount the vault directory directly.
|
||
3. **PVC direct** — vault files managed via kubectl cp or a web uploader.
|
||
|
||
**Configuration:**
|
||
```yaml
|
||
connectors:
|
||
- kind: obsidian
|
||
name: homelab-vault
|
||
config:
|
||
root: /data/vault
|
||
extensions: [md, markdown, txt]
|
||
exclude_dirs: [.obsidian, .trash, .git]
|
||
max_file_size: 10485760 # 10MB
|
||
```
|
||
|
||
## Steps
|
||
|
||
1. Implement `ObsidianConnector` in `mem-ingest/src/connectors/obsidian.rs`.
|
||
2. `list_documents()` — walk `root` directory, filter by extension, compute
|
||
sha256 per file, return `SourceDocument` per file.
|
||
3. `fetch_document()` — read file content, return as `DocumentContent` with
|
||
metadata (file path, last modified, size).
|
||
4. `health_check()` — verify `root` exists, is readable, count files.
|
||
5. Register `"obsidian"` kind in the connector registry factory.
|
||
6. `source_type()` returns `Reference` (vault docs bypass the gated loop).
|
||
7. Reuse `DocCorpusSource` internals for heading-based chunking when the sync
|
||
framework (M7.6) processes this connector's documents.
|
||
|
||
## Acceptance
|
||
|
||
- `ObsidianConnector` implements `SourceConnector` fully.
|
||
- `list_documents()` respects `extensions`, `exclude_dirs`, `max_file_size`.
|
||
- `fetch_document()` returns content matching file on disk.
|
||
- `health_check()` distinguishes readable vs. missing root directory.
|
||
- Config-driven: changing `root` path in YAML changes what gets scanned.
|
||
|
||
## Verify
|
||
|
||
**Harness:** fixture directory with markdown files, config YAML.
|
||
|
||
**Integration test** — `tests/it_obsidian_connector.rs`:
|
||
1. `a1_list_filters_extensions` — fixture with .md, .txt, .json; assert only
|
||
.md and .txt are listed.
|
||
2. `a2_list_excludes_dirs` — fixture with `.obsidian/` subdir; assert its files
|
||
are excluded.
|
||
3. `a3_fetch_returns_content` — fetch a known doc; assert text matches file.
|
||
4. `a4_fetch_unknown_errors` — fetch non-existent doc_id; assert error.
|
||
5. `a5_health_check_reachable` — valid root; assert `reachable: true` with count.
|
||
6. `a6_health_check_missing_root` — non-existent root; assert `reachable: false`.
|
||
7. `a7_content_hash_stable` — fetch same file twice; assert same hash.
|
||
8. `a8_config_from_yaml` — parse connector from YAML; assert fields match.
|
||
|
||
**Command:** `cargo test --test it_obsidian_connector`
|
||
|
||
**False pass:**
|
||
- Testing with an empty directory. Assertions 1–3 need real files.
|
||
- Not testing `exclude_dirs` with nested paths (`.obsidian/plugins/x.md`).
|
||
|
||
## Traps
|
||
|
||
- Re-implementing markdown parsing instead of delegating to `DocCorpusSource`.
|
||
- Making `doc_id` platform-dependent (use relative path from root, unix separators).
|
||
- Ignoring symlinks — Obsidian uses them for multi-vault setups.
|
||
|
||
---
|
||
|
||
Background: [DESIGN.md](../DESIGN.md) — source connectors, Obsidian vault as connector
|