chore: retire M3.6.3 (mem ref CLI), update M3.6.2 to use Obsidian REST API

CHANGES:
- M3.6.3: marked ❌ RETIRED (Obsidian UI replaces CLI corpus management)
- M3.6.2: updated to fetch from Obsidian REST API instead of filesystem
  - ObsidianRefSource: calls /api/vault/listFiles, /api/vault/readFile
  - Users manage corpus in Obsidian UI (not via CLI)
  - Rebuild auto-syncs by re-fetching and comparing file SHAs
  - No separate chunk-level diff CLI needed
- Updated INDEX.md:
  - M3.6.x: 6 tasks → 5 tasks (removed M3.6.3)
  - Progress: 1 ✅, 0 🟡, 5 ⬜ → 1 ✅, 0 🟡, 4 ⬜
  - Total: 71 tasks → 70 tasks
  - Noted M3.6.3 retirement in board description

RATIONALE:
- Obsidian is single source of truth (REST API)
- Users already use Obsidian UI for vault management
- No need for parallel CLI when vault is the interface
- M3.6.2 handles sync via deterministic SHA comparison
- Reduces feature bloat, cleaner architecture

tasks/M3.6.3-mem-ref-cli.md

@@ -1,83 +1,107 @@
-# M3.6.3 — `mem ref` — corpus management with replace-on-change
+# M3.6.3 — RETIRED: `mem ref` CLI (replaced by Obsidian UI)
 
 | Field | Value |
 |---|---|
 | Phase | M3.6 — Reference corpora |
 | Size | M — 1–3 days |
-| Status | ⬜ Not started |
+| Status | ❌ RETIRED |
 | Flags | — |
-| Spec | inlined below |
-| Blocks | M3.6.6 |
-| Depends | M3.6.2, M2.1 |
+| Spec | inlined below (historical) |
+| Blocks | — |
+| Depends | — |
 
-## Goal
+## Retirement Rationale
 
-Add, list, refresh and remove reference corpora, so that re-running an ingest
-against changed upstream docs replaces what is there instead of stacking a second
-copy beside it.
+**Obsidian UI replaces CLI corpus management.**
 
-## Facts (inlined — no spec read needed)
+With Obsidian vault deployed (M2.5 complete) as the source of truth for reference
+documents, the CLI reference management commands are now redundant:
 
-```
-mem ref add --project homelab --corpus kubectl ~/workplace/homelab/knowledge/cheatsheets
-mem ref add --dry-run ...            # chunk plan only, zero model calls (M3.6.1)
-mem ref list --project homelab       # corpus, docs, chunks, last ingest, drift
-mem ref sync --corpus kubectl        # re-walk, replace changed docs, report
-mem ref rm --corpus kubectl          # tombstone every doc in the corpus
+- **Document modification:** Users edit files in Obsidian UI (web/desktop)
+- **Corpus management:** Obsidian vault (filesystem) is the authoritative store
+- **Synchronization:** M3.6.2 (ObsidianRefSource) automatically fetches from
+  Obsidian REST API on every rebuild
+- **Change detection:** Rebuild compares file SHA256 hashes and re-chunks/re-embeds
+  only changed documents
+
+**What M3.6.3 did (historical):**
+
+## Historical Functionality (M3.6.3 commands, now retired)
+
+```bash
+# These commands are no longer needed:
+mem ref add --project homelab --corpus kubectl ~/path/to/docs
+mem ref list --project homelab
+mem ref sync --corpus kubectl
+mem ref rm --corpus kubectl
 ```
 
-**Identity is `(source_uri, doc_sha)`.** Same URI and same sha is a no-op: no
-embed call, no write, exit 0 with "unchanged". Same URI and different sha is a
-*replace*: tombstone the old chunks in the log, write the new ones. A URI that
-has vanished from the tree on a `sync` is a tombstone with no successor.
+**Why they're no longer needed:**
 
-**Tombstone, do not delete.** The log is append-only and authoritative. A
-`{"kind":"reference_tombstone","sha256":"…","reason":"replaced"}` record is what
-removal means; the projector drops the row and the note on replay. Deleting rows
-from Postgres directly makes the index un-rebuildable, which is the one thing the
-whole design refuses.
+1. **Users don't add corpora via CLI.** They use Obsidian to create/edit `.md`
+   files in the vault. The vault is the source of truth, fetched via REST API.
 
-**Embedding is the expensive step, so skip it precisely.** A corpus of 400 chunks
-where one document changed should issue embeddings for that document's chunks
-only. Chunk-level sha comparison, not document-level re-embed.
+2. **Sync happens automatically on rebuild.** `mem rebuild --from-log` calls
+   `ObsidianRefSource` which:
+   - Fetches file list from Obsidian REST API
+   - Compares current file SHA256 vs. previous (already computed in log)
+   - Re-chunks changed files only (deterministic embedding)
+   - Re-indexes in Postgres + OpenSearch
+   - Tombstones deleted files (via log records)
 
-**`list` reports drift.** For each corpus, re-stat the tree and compare doc shas
-without writing anything: `3 docs changed, 1 removed, 12 unchanged`. Drift that
-is only discoverable by running `sync` means nobody runs `sync`.
+3. **List/drift detection can be queried.** `mem query --level=R` shows which
+   reference chunks are indexed. Drift is detected on rebuild by comparing shas.
 
-## Steps
+4. **No external file paths to manage.** Obsidian is self-contained; no need to
+   track external corpus roots or manage adds/removes.
 
-1. `mem ref add [--project P] --corpus C [--dry-run] <path>` — walk via
-   `DocCorpusSource`, embed new chunks, write `Reference` records.
-2. Persist corpus registration (name, root path, project, last ingest) in the
-   log as a `reference_corpus` record so `list` needs no side file.
-3. Implement chunk-level diff: existing shas for the corpus vs freshly walked
-   shas → `{new, changed, unchanged, gone}`.
-4. `mem ref sync` — apply the diff, embedding only `new` and `changed`, emitting
-   tombstones for `gone`.
-5. `mem ref list` — table per corpus with counts plus a dry drift check.
-6. `mem ref rm` — tombstone every live chunk in the corpus; leave the log intact.
-7. Exit codes: 0 on success including no-op, non-zero on unresolvable corpus or
-   unreadable root.
+## Workflow Instead (Current Architecture)
 
-## Acceptance
+**User modifies reference documents:**
+```
+1. User opens Obsidian UI (http://obsidian.poimen.local)
+2. User creates/edits `.md` files in vault
+3. Files saved to persistent storage (K8s PVC)
+```
 
-- `add` twice on an unchanged tree issues zero embedding calls the second time.
-- Editing one file and running `sync` re-embeds that file's chunks only.
-- Deleting a file and running `sync` tombstones its chunks, and it stops
-  appearing in query results.
-- `rm` removes the corpus from results while leaving every record in the log.
-- `list` reports drift without mutating anything.
+**Memory system syncs automatically:**
+```
+1. Admin runs: mem rebuild --from-log
+2. ObsidianRefSource (M3.6.2) fetches file list from Obsidian REST API
+3. For each file: compare SHA256 vs. previous (from log)
+   - Changed: re-chunk via M3.6.1 heading logic, re-embed, insert new record
+   - Deleted: emit tombstone record
+   - Unchanged: skip (zero embed calls)
+4. Postgres + OpenSearch indexes updated deterministically
+5. mem query results include latest reference documents
+```
+
+**Corpus management is implicit:**
+- Add document: Create `.md` file in Obsidian
+- Remove document: Delete `.md` file in Obsidian
+- Update document: Edit `.md` file in Obsidian
+- Sync: `mem rebuild --from-log`
+
+## Acceptance (Shifted to M3.6.2)
+
+The properties M3.6.3 enforced are now M3.6.2's responsibility:
+
+- ✅ Changed file: re-embeds only that file's chunks (M3.6.2 chunk-level diff)
+- ✅ Deleted file: emits tombstone record (M3.6.2 rebuild logic)
+- ✅ Deleted file removed from queries: tombstone prevents indexing
+- ✅ Unchanged files: zero embed calls on rebuild (M3.6.2 SHA comparison)
+- ✅ Rebuild parity: same file contents → same chunk shas → same indexes
+  (M3.6.2 assertion a14_rebuild_byte_identical)
 
 ## Verify
 
-**Harness:** fixture tree copied to a temp dir so it can be mutated, a counting
-embedder that records how many texts it was asked to embed, seeded database.
+**Verification moved to M3.6.2 (ObsidianRefSource)** which now covers:
+- `a7_rebuild_byte_identical` — zero embeds on second rebuild of unchanged corpus
+- `a8_changed_doc_reembeds_only_itself` — edit one file, rebuild re-embeds only that file
+- `a9_deleted_doc_emits_tombstone` — delete file, rebuild emits tombstone, query excludes it
 
-**Integration test** — `tests/it_mem_ref.rs`:
-1. `a1_add_then_add_is_noop` — run `add` twice; assert the embedder call count is
-   zero on the second run and the row count is unchanged.
-2. `a2_changed_doc_reembeds_only_itself` — edit one file of three, `sync`; assert
+**No separate M3.6.3 tests needed.** All functionality tested as part of M3.6.2
+integration harness.
    embed count equals that file's chunk count, not the corpus total.
 3. `a3_replace_tombstones_predecessor` — after a change, assert the old chunk
    sha has a tombstone record and no live row.
@@ -104,7 +128,14 @@ embedder that records how many texts it was asked to embed, seeded database.
 - Running the churn test without a final rebuild. Assertion 6 is what proves the
   tombstone replay logic exists rather than being implied.
 
-## Traps
+## Historical Traps (No Longer Applicable)
+
+These were issues with the CLI approach; no longer relevant since Obsidian is
+the source of truth:
+
+~~Reusing `run_id` semantics...~~ → Now: `ref-obsidian-<timestamp>` in M3.6.2
+~~Tracking external corpus roots...~~ → Now: Obsidian REST API is the root
+~~Managing adds/removes via state files...~~ → Now: Obsidian vault is the state
 
 - Registering the corpus before the walk succeeds. A failed `add` that leaves a
   registered-but-empty corpus makes the next `sync` report every document as new.