Story Crater Bot
71ecf482e7
- M7.1-M7.10: Extensible SourceConnector trait, Obsidian/paperless/git/S3 connectors, sync framework, CLI, HTTP endpoints, health monitoring, gate - M3.5.10: Auth integration with Authentik OIDC → Vault token validation - DESIGN.md: Add source connectors architecture, update auth to Authentik/Vault (Kong removed from cluster) - INDEX.md: 75 tasks, 11 gates - Fix all Kong references in M3.5.1 task
108 lines
4.2 KiB
Markdown
108 lines
4.2 KiB
Markdown
# M7.7 — `mem source` CLI
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M7 — Source connectors |
|
||
| Size | M — 1–3 days |
|
||
| Status | ⬜ Not started |
|
||
| Flags | — |
|
||
| Spec | inlined below |
|
||
| Blocks | M7.8, M7.10 |
|
||
| Depends | M7.6 |
|
||
|
||
## Goal
|
||
|
||
Provide CLI commands to manage source connectors: sync documents, check status,
|
||
list connectors, add/remove connectors.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
```
|
||
mem source list # all registered connectors + health
|
||
mem source status # drift report per connector, no mutations
|
||
mem source sync --all # sync all connectors
|
||
mem source sync --name homelab-vault # sync one connector
|
||
mem source sync --name homelab-vault --dry-run # show plan, no mutations
|
||
mem source add --kind paperless --name docs --config '{"base_url":"...","token":"..."}'
|
||
mem source rm --name old-source # deregister + tombstone all chunks
|
||
mem source health # connectivity check per connector
|
||
```
|
||
|
||
**Output format.** CLI outputs human-readable tables by default, `--json` for
|
||
machine consumption. Example:
|
||
|
||
```
|
||
$ mem source list
|
||
NAME KIND DOCS LAST SYNC HEALTH
|
||
homelab-vault obsidian 42 2026-08-26T10:00:00Z ✅ reachable
|
||
homelab-paperless paperless 127 2026-08-26T09:00:00Z ✅ reachable
|
||
infra-docs git_repo 18 2026-08-25T20:00:00Z ⚠️ pull failed
|
||
|
||
$ mem source status
|
||
NAME NEW CHANGED UNCHANGED REMOVED
|
||
homelab-vault 0 2 40 0
|
||
homelab-paperless 3 0 124 0
|
||
infra-docs 1 1 16 0
|
||
```
|
||
|
||
## Steps
|
||
|
||
1. Add `Source` subcommand group to clap CLI in `mem-cli/src/main.rs`.
|
||
2. Implement `cmd_source_list()` — load registry, health check each, tabulate.
|
||
3. Implement `cmd_source_status()` — load registry, run drift report per
|
||
connector, tabulate.
|
||
4. Implement `cmd_source_sync()` — load registry, run sync engine for selected
|
||
connector(s), report results.
|
||
5. Implement `cmd_source_add()` — validate kind, parse config, register in
|
||
`connectors.yaml`, run initial health check.
|
||
6. Implement `cmd_source_rm()` — tombstone all chunks from connector, remove
|
||
from `connectors.yaml`.
|
||
7. Implement `cmd_source_health()` — connectivity check per connector.
|
||
8. Add `--dry-run` to sync (show plan only).
|
||
9. Add `--json` flag for machine-readable output.
|
||
|
||
## Acceptance
|
||
|
||
- All subcommands execute without panic.
|
||
- `sync --dry-run` shows plan without mutations.
|
||
- `add` validates kind exists in registry before writing config.
|
||
- `rm` tombstones chunks and removes config entry.
|
||
- `status` shows drift without mutations.
|
||
- Exit codes: 0 on success, non-zero on failure.
|
||
|
||
## Verify
|
||
|
||
**Harness:** `VecConnector` registered in registry, temp connectors.yaml.
|
||
|
||
**Integration test** — `tests/it_source_cli.rs`:
|
||
1. `a1_list_shows_connectors` — register two connectors; assert list output
|
||
contains both names and kinds.
|
||
2. `a2_status_shows_drift` — modify connector docs; assert status shows correct
|
||
new/changed/unchanged/removed counts.
|
||
3. `a3_sync_processes_changes` — sync with changes; assert documents processed.
|
||
4. `a4_sync_dry_run_no_mutations` — sync --dry-run; assert no log entries written.
|
||
5. `a5_add_registers_connector` — add a new connector; assert it appears in list.
|
||
6. `a6_add_bad_kind_fails` — add with unknown kind; assert non-zero exit.
|
||
7. `a7_rm_tombstones_and_deregisters` — rm a connector; assert tombstone records
|
||
written and connector removed from list.
|
||
8. `a8_health_reports_status` — assert health output includes reachable/unreachable.
|
||
9. `a9_json_output` — assert --json flag produces valid JSON.
|
||
|
||
**Command:** `cargo test --test it_source_cli`
|
||
|
||
**False pass:**
|
||
- Testing `list` without checking that connectors are actually registered (not
|
||
just config parsed).
|
||
|
||
## Traps
|
||
|
||
- `sync --all` with a broken connector should not halt all syncs. Sync each
|
||
independently, report failures per connector at the end.
|
||
- `rm` without `--yes` should prompt for confirmation (destructive operation).
|
||
- Don't write `connectors.yaml` atomically — a crash mid-write corrupts config.
|
||
Write to temp file, then rename.
|
||
|
||
---
|
||
|
||
Background: [DESIGN.md](../DESIGN.md) — source connectors, CLI commands
|