docs: Add M7 source connectors (10 tasks), M3.5.10 auth integration, remove Kong refs

- M7.1-M7.10: Extensible SourceConnector trait, Obsidian/paperless/git/S3 connectors, sync framework, CLI, HTTP endpoints, health monitoring, gate - M3.5.10: Auth integration with Authentik OIDC → Vault token validation - DESIGN.md: Add source connectors architecture, update auth to Authentik/Vault (Kong removed from cluster) - INDEX.md: 75 tasks, 11 gates - Fix all Kong references in M3.5.1 task
2026-08-26 16:56:39 -07:00
parent c8d754b0ba
commit 71ecf482e7
14 changed files with 1560 additions and 13 deletions
@@ -0,0 +1,107 @@
+# M7.7 — `mem source` CLI
+
+| Field | Value |
+|---|---|
+| Phase | M7 — Source connectors |
+| Size | M — 1–3 days |
+| Status | ⬜ Not started |
+| Flags | — |
+| Spec | inlined below |
+| Blocks | M7.8, M7.10 |
+| Depends | M7.6 |
+
+## Goal
+
+Provide CLI commands to manage source connectors: sync documents, check status,
+list connectors, add/remove connectors.
+
+## Facts (inlined — no spec read needed)
+
+```
+mem source list                          # all registered connectors + health
+mem source status                        # drift report per connector, no mutations
+mem source sync --all                    # sync all connectors
+mem source sync --name homelab-vault     # sync one connector
+mem source sync --name homelab-vault --dry-run  # show plan, no mutations
+mem source add --kind paperless --name docs --config '{"base_url":"...","token":"..."}'
+mem source rm --name old-source          # deregister + tombstone all chunks
+mem source health                        # connectivity check per connector
+```
+
+**Output format.** CLI outputs human-readable tables by default, `--json` for
+machine consumption. Example:
+
+```
+$ mem source list
+NAME              KIND       DOCS  LAST SYNC            HEALTH
+homelab-vault     obsidian    42   2026-08-26T10:00:00Z  ✅ reachable
+homelab-paperless paperless  127   2026-08-26T09:00:00Z  ✅ reachable
+infra-docs        git_repo    18   2026-08-25T20:00:00Z  ⚠️ pull failed
+
+$ mem source status
+NAME              NEW  CHANGED  UNCHANGED  REMOVED
+homelab-vault       0        2         40        0
+homelab-paperless   3        0        124        0
+infra-docs          1        1         16        0
+```
+
+## Steps
+
+1. Add `Source` subcommand group to clap CLI in `mem-cli/src/main.rs`.
+2. Implement `cmd_source_list()` — load registry, health check each, tabulate.
+3. Implement `cmd_source_status()` — load registry, run drift report per
+   connector, tabulate.
+4. Implement `cmd_source_sync()` — load registry, run sync engine for selected
+   connector(s), report results.
+5. Implement `cmd_source_add()` — validate kind, parse config, register in
+   `connectors.yaml`, run initial health check.
+6. Implement `cmd_source_rm()` — tombstone all chunks from connector, remove
+   from `connectors.yaml`.
+7. Implement `cmd_source_health()` — connectivity check per connector.
+8. Add `--dry-run` to sync (show plan only).
+9. Add `--json` flag for machine-readable output.
+
+## Acceptance
+
+- All subcommands execute without panic.
+- `sync --dry-run` shows plan without mutations.
+- `add` validates kind exists in registry before writing config.
+- `rm` tombstones chunks and removes config entry.
+- `status` shows drift without mutations.
+- Exit codes: 0 on success, non-zero on failure.
+
+## Verify
+
+**Harness:** `VecConnector` registered in registry, temp connectors.yaml.
+
+**Integration test** — `tests/it_source_cli.rs`:
+1. `a1_list_shows_connectors` — register two connectors; assert list output
+   contains both names and kinds.
+2. `a2_status_shows_drift` — modify connector docs; assert status shows correct
+   new/changed/unchanged/removed counts.
+3. `a3_sync_processes_changes` — sync with changes; assert documents processed.
+4. `a4_sync_dry_run_no_mutations` — sync --dry-run; assert no log entries written.
+5. `a5_add_registers_connector` — add a new connector; assert it appears in list.
+6. `a6_add_bad_kind_fails` — add with unknown kind; assert non-zero exit.
+7. `a7_rm_tombstones_and_deregisters` — rm a connector; assert tombstone records
+   written and connector removed from list.
+8. `a8_health_reports_status` — assert health output includes reachable/unreachable.
+9. `a9_json_output` — assert --json flag produces valid JSON.
+
+**Command:** `cargo test --test it_source_cli`
+
+**False pass:**
+- Testing `list` without checking that connectors are actually registered (not
+  just config parsed).
+
+## Traps
+
+- `sync --all` with a broken connector should not halt all syncs. Sync each
+  independently, report failures per connector at the end.
+- `rm` without `--yes` should prompt for confirmation (destructive operation).
+- Don't write `connectors.yaml` atomically — a crash mid-write corrupts config.
+  Write to temp file, then rename.
+
+---
+
+Background: [DESIGN.md](../DESIGN.md) — source connectors, CLI commands