poimen-memory/tasks/M3.6.3-mem-ref-cli.md

M3.6.3 — RETIRED: mem ref CLI (replaced by Obsidian UI)

Field	Value
Phase	M3.6 — Reference corpora
Size	M — 1–3 days
Status	❌ RETIRED
Flags	—
Spec	inlined below (historical)
Blocks	—
Depends	—

Retirement Rationale

Obsidian UI replaces CLI corpus management.

With Obsidian vault deployed (M2.5 complete) as the source of truth for reference documents, the CLI reference management commands are now redundant:

• Document modification: Users edit files in Obsidian UI (web/desktop)
• Corpus management: Obsidian vault (filesystem) is the authoritative store
• Synchronization: M3.6.2 (ObsidianRefSource) automatically fetches from Obsidian REST API on every rebuild
• Change detection: Rebuild compares file SHA256 hashes and re-chunks/re-embeds only changed documents

What M3.6.3 did (historical):

Historical Functionality (M3.6.3 commands, now retired)

# These commands are no longer needed:
mem ref add --project homelab --corpus kubectl ~/path/to/docs
mem ref list --project homelab
mem ref sync --corpus kubectl
mem ref rm --corpus kubectl

Why they're no longer needed:

1. Users don't add corpora via CLI. They use Obsidian to create/edit .md files in the vault. The vault is the source of truth, fetched via REST API.

2. Sync happens automatically on rebuild. mem rebuild --from-log calls ObsidianRefSource which:

   • Fetches file list from Obsidian REST API
   • Compares current file SHA256 vs. previous (already computed in log)
   • Re-chunks changed files only (deterministic embedding)
   • Re-indexes in Postgres + OpenSearch
   • Tombstones deleted files (via log records)

3. List/drift detection can be queried. mem query --level=R shows which reference chunks are indexed. Drift is detected on rebuild by comparing shas.

4. No external file paths to manage. Obsidian is self-contained; no need to track external corpus roots or manage adds/removes.

Workflow Instead (Current Architecture)

User modifies reference documents:

1. User opens Obsidian UI (http://obsidian.poimen.local)
2. User creates/edits `.md` files in vault
3. Files saved to persistent storage (K8s PVC)

Memory system syncs automatically:

1. Admin runs: mem rebuild --from-log
2. ObsidianRefSource (M3.6.2) fetches file list from Obsidian REST API
3. For each file: compare SHA256 vs. previous (from log)
   - Changed: re-chunk via M3.6.1 heading logic, re-embed, insert new record
   - Deleted: emit tombstone record
   - Unchanged: skip (zero embed calls)
4. Postgres + OpenSearch indexes updated deterministically
5. mem query results include latest reference documents

Corpus management is implicit:

• Add document: Create .md file in Obsidian
• Remove document: Delete .md file in Obsidian
• Update document: Edit .md file in Obsidian
• Sync: mem rebuild --from-log

Acceptance (Shifted to M3.6.2)

The properties M3.6.3 enforced are now M3.6.2's responsibility:

• ✅ Changed file: re-embeds only that file's chunks (M3.6.2 chunk-level diff)
• ✅ Deleted file: emits tombstone record (M3.6.2 rebuild logic)
• ✅ Deleted file removed from queries: tombstone prevents indexing
• ✅ Unchanged files: zero embed calls on rebuild (M3.6.2 SHA comparison)
• ✅ Rebuild parity: same file contents → same chunk shas → same indexes (M3.6.2 assertion a14_rebuild_byte_identical)

Verify

Verification moved to M3.6.2 (ObsidianRefSource) which now covers:

• a7_rebuild_byte_identical — zero embeds on second rebuild of unchanged corpus
• a8_changed_doc_reembeds_only_itself — edit one file, rebuild re-embeds only that file
• a9_deleted_doc_emits_tombstone — delete file, rebuild emits tombstone, query excludes it

No separate M3.6.3 tests needed. All functionality tested as part of M3.6.2 integration harness. embed count equals that file's chunk count, not the corpus total. 3. a3_replace_tombstones_predecessor — after a change, assert the old chunk sha has a tombstone record and no live row. 4. a4_removed_doc_tombstoned — delete a file, sync; assert its chunks are gone from memory_node and present in the log. 5. a5_rm_preserves_log — count log lines before and after rm; assert the count only grew. 6. a6_rebuild_after_churn — after add/change/sync/rm, mem rebuild --from-log; assert the reconstructed state matches the live state exactly. 7. a7_list_is_read_only — snapshot database and log, run list, assert both unchanged and that reported drift matches the mutations made. 8. a8_unreadable_root_exits_nonzero — point add at a missing path; assert non-zero exit and no partial corpus registration.

Command: cargo test -p mem-cli mem_ref

False pass:

• Asserting "no duplicate rows" instead of counting embedder calls. A sync that re-embeds everything and then upserts by sha produces a correct table and a bill; assertion 1 and 2 are the only ones that see it.
• Verifying tombstones by querying memory_node. The row being absent is the projector working; assertion 3 has to read the log to prove the tombstone was actually written and the row was not just deleted.
• Running the churn test without a final rebuild. Assertion 6 is what proves the tombstone replay logic exists rather than being implied.

Historical Traps (No Longer Applicable)

These were issues with the CLI approach; no longer relevant since Obsidian is the source of truth:

Reusing run_id semantics... → Now: ref-obsidian-<timestamp> in M3.6.2 Tracking external corpus roots... → Now: Obsidian REST API is the root Managing adds/removes via state files... → Now: Obsidian vault is the state

• Registering the corpus before the walk succeeds. A failed add that leaves a registered-but-empty corpus makes the next sync report every document as new.
• Comparing document mtime instead of sha. Checkouts and rsync rewrite mtimes; a corpus that re-embeds on every clone costs real money on the TEI endpoint.
• Making rm delete log records "because they are noise". That converts the log from authoritative to advisory, and nothing downstream can tell.

---

Background: DESIGN.md — reference corpora, JSONL event log