Story Crater Bot
3e867f7cce
CHANGES: - M3.6.3: marked ❌ RETIRED (Obsidian UI replaces CLI corpus management) - M3.6.2: updated to fetch from Obsidian REST API instead of filesystem - ObsidianRefSource: calls /api/vault/listFiles, /api/vault/readFile - Users manage corpus in Obsidian UI (not via CLI) - Rebuild auto-syncs by re-fetching and comparing file SHAs - No separate chunk-level diff CLI needed - Updated INDEX.md: - M3.6.x: 6 tasks → 5 tasks (removed M3.6.3) - Progress: 1 ✅, 0 🟡, 5 ⬜ → 1 ✅, 0 🟡, 4 ⬜ - Total: 71 tasks → 70 tasks - Noted M3.6.3 retirement in board description RATIONALE: - Obsidian is single source of truth (REST API) - Users already use Obsidian UI for vault management - No need for parallel CLI when vault is the interface - M3.6.2 handles sync via deterministic SHA comparison - Reduces feature bloat, cleaner architecture
150 lines
6.2 KiB
Markdown
150 lines
6.2 KiB
Markdown
# M3.6.3 — RETIRED: `mem ref` CLI (replaced by Obsidian UI)
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M3.6 — Reference corpora |
|
||
| Size | M — 1–3 days |
|
||
| Status | ❌ RETIRED |
|
||
| Flags | — |
|
||
| Spec | inlined below (historical) |
|
||
| Blocks | — |
|
||
| Depends | — |
|
||
|
||
## Retirement Rationale
|
||
|
||
**Obsidian UI replaces CLI corpus management.**
|
||
|
||
With Obsidian vault deployed (M2.5 complete) as the source of truth for reference
|
||
documents, the CLI reference management commands are now redundant:
|
||
|
||
- **Document modification:** Users edit files in Obsidian UI (web/desktop)
|
||
- **Corpus management:** Obsidian vault (filesystem) is the authoritative store
|
||
- **Synchronization:** M3.6.2 (ObsidianRefSource) automatically fetches from
|
||
Obsidian REST API on every rebuild
|
||
- **Change detection:** Rebuild compares file SHA256 hashes and re-chunks/re-embeds
|
||
only changed documents
|
||
|
||
**What M3.6.3 did (historical):**
|
||
|
||
## Historical Functionality (M3.6.3 commands, now retired)
|
||
|
||
```bash
|
||
# These commands are no longer needed:
|
||
mem ref add --project homelab --corpus kubectl ~/path/to/docs
|
||
mem ref list --project homelab
|
||
mem ref sync --corpus kubectl
|
||
mem ref rm --corpus kubectl
|
||
```
|
||
|
||
**Why they're no longer needed:**
|
||
|
||
1. **Users don't add corpora via CLI.** They use Obsidian to create/edit `.md`
|
||
files in the vault. The vault is the source of truth, fetched via REST API.
|
||
|
||
2. **Sync happens automatically on rebuild.** `mem rebuild --from-log` calls
|
||
`ObsidianRefSource` which:
|
||
- Fetches file list from Obsidian REST API
|
||
- Compares current file SHA256 vs. previous (already computed in log)
|
||
- Re-chunks changed files only (deterministic embedding)
|
||
- Re-indexes in Postgres + OpenSearch
|
||
- Tombstones deleted files (via log records)
|
||
|
||
3. **List/drift detection can be queried.** `mem query --level=R` shows which
|
||
reference chunks are indexed. Drift is detected on rebuild by comparing shas.
|
||
|
||
4. **No external file paths to manage.** Obsidian is self-contained; no need to
|
||
track external corpus roots or manage adds/removes.
|
||
|
||
## Workflow Instead (Current Architecture)
|
||
|
||
**User modifies reference documents:**
|
||
```
|
||
1. User opens Obsidian UI (http://obsidian.poimen.local)
|
||
2. User creates/edits `.md` files in vault
|
||
3. Files saved to persistent storage (K8s PVC)
|
||
```
|
||
|
||
**Memory system syncs automatically:**
|
||
```
|
||
1. Admin runs: mem rebuild --from-log
|
||
2. ObsidianRefSource (M3.6.2) fetches file list from Obsidian REST API
|
||
3. For each file: compare SHA256 vs. previous (from log)
|
||
- Changed: re-chunk via M3.6.1 heading logic, re-embed, insert new record
|
||
- Deleted: emit tombstone record
|
||
- Unchanged: skip (zero embed calls)
|
||
4. Postgres + OpenSearch indexes updated deterministically
|
||
5. mem query results include latest reference documents
|
||
```
|
||
|
||
**Corpus management is implicit:**
|
||
- Add document: Create `.md` file in Obsidian
|
||
- Remove document: Delete `.md` file in Obsidian
|
||
- Update document: Edit `.md` file in Obsidian
|
||
- Sync: `mem rebuild --from-log`
|
||
|
||
## Acceptance (Shifted to M3.6.2)
|
||
|
||
The properties M3.6.3 enforced are now M3.6.2's responsibility:
|
||
|
||
- ✅ Changed file: re-embeds only that file's chunks (M3.6.2 chunk-level diff)
|
||
- ✅ Deleted file: emits tombstone record (M3.6.2 rebuild logic)
|
||
- ✅ Deleted file removed from queries: tombstone prevents indexing
|
||
- ✅ Unchanged files: zero embed calls on rebuild (M3.6.2 SHA comparison)
|
||
- ✅ Rebuild parity: same file contents → same chunk shas → same indexes
|
||
(M3.6.2 assertion a14_rebuild_byte_identical)
|
||
|
||
## Verify
|
||
|
||
**Verification moved to M3.6.2 (ObsidianRefSource)** which now covers:
|
||
- `a7_rebuild_byte_identical` — zero embeds on second rebuild of unchanged corpus
|
||
- `a8_changed_doc_reembeds_only_itself` — edit one file, rebuild re-embeds only that file
|
||
- `a9_deleted_doc_emits_tombstone` — delete file, rebuild emits tombstone, query excludes it
|
||
|
||
**No separate M3.6.3 tests needed.** All functionality tested as part of M3.6.2
|
||
integration harness.
|
||
embed count equals that file's chunk count, not the corpus total.
|
||
3. `a3_replace_tombstones_predecessor` — after a change, assert the old chunk
|
||
sha has a tombstone record and no live row.
|
||
4. `a4_removed_doc_tombstoned` — delete a file, `sync`; assert its chunks are
|
||
gone from `memory_node` and present in the log.
|
||
5. `a5_rm_preserves_log` — count log lines before and after `rm`; assert the
|
||
count only grew.
|
||
6. `a6_rebuild_after_churn` — after add/change/sync/rm, `mem rebuild --from-log`;
|
||
assert the reconstructed state matches the live state exactly.
|
||
7. `a7_list_is_read_only` — snapshot database and log, run `list`, assert both
|
||
unchanged and that reported drift matches the mutations made.
|
||
8. `a8_unreadable_root_exits_nonzero` — point `add` at a missing path; assert
|
||
non-zero exit and no partial corpus registration.
|
||
|
||
**Command:** `cargo test -p mem-cli mem_ref`
|
||
|
||
**False pass:**
|
||
- Asserting "no duplicate rows" instead of counting embedder calls. A `sync` that
|
||
re-embeds everything and then upserts by sha produces a correct table and a
|
||
bill; assertion 1 and 2 are the only ones that see it.
|
||
- Verifying tombstones by querying `memory_node`. The row being absent is the
|
||
projector working; assertion 3 has to read the log to prove the tombstone was
|
||
actually written and the row was not just deleted.
|
||
- Running the churn test without a final rebuild. Assertion 6 is what proves the
|
||
tombstone replay logic exists rather than being implied.
|
||
|
||
## Historical Traps (No Longer Applicable)
|
||
|
||
These were issues with the CLI approach; no longer relevant since Obsidian is
|
||
the source of truth:
|
||
|
||
~~Reusing `run_id` semantics...~~ → Now: `ref-obsidian-<timestamp>` in M3.6.2
|
||
~~Tracking external corpus roots...~~ → Now: Obsidian REST API is the root
|
||
~~Managing adds/removes via state files...~~ → Now: Obsidian vault is the state
|
||
|
||
- Registering the corpus before the walk succeeds. A failed `add` that leaves a
|
||
registered-but-empty corpus makes the next `sync` report every document as new.
|
||
- Comparing document mtime instead of sha. Checkouts and rsync rewrite mtimes;
|
||
a corpus that re-embeds on every clone costs real money on the TEI endpoint.
|
||
- Making `rm` delete log records "because they are noise". That converts the log
|
||
from authoritative to advisory, and nothing downstream can tell.
|
||
|
||
---
|
||
|
||
Background: [DESIGN.md](../DESIGN.md) — reference corpora, JSONL event log
|