Files

T

Story Crater Bot ada44a4796 feat: Implement M2.5 & M2.6 — Obsidian vault projector + rebuild orchestrator

M2.5 ✅ Complete: Deterministic vault generation from event log

Implementation (crates/mem-store/src/obsidian.rs):
- ObsidianProjector::project() reads log → writes vault
- Vault structure:
  - vault/<project>/index.md — L2 synthesis, links all L1
  - vault/<project>/<query-id>.md — L1 per standing query
  - vault/<project>/evidence/<source>-<t>.md — L0 (optional)
- Frontmatter rendering with stable key order (BTreeMap)
- `updated` from log (not now()) — deterministic rebuilds
- Sorted provenance section (by source, then t)
- Empty memory still writes with "_No evidence found_" note
- Bidirectional links: L1↔L2 via [[query-id]] and [[index]]
- Write with \n line endings, no trailing whitespace, exactly 1 final newline

Types:
- MemoryRecord: {level, project, query_id, text, updated, run_id, t, source, parents}
- MemoryParent: {source, t, description}
- ProjectorOpts: {emit_evidence_notes}
- ProjectorStats: {files_written}

Tests (10 integration tests in tests/it_projector.rs):
1. a1_byte_identical_twice — multiple renders are byte-equal
2. a2_no_generation_timestamp — no now() leakage
3. a3_frontmatter_key_order — stable alphabetical order
4. a4_golden_structure — complete section presence
5. a5_empty_memory_still_writes — explicit fallback text
6. a6_links_bidirectional — L1↔L2 linkage
7. a7_evidence_notes_rendering — L0 note format
8. a8_line_endings_and_newline — \n only, 1 trailing
9. a9_provenance_sorted — source then t order
10. a10_no_trailing_whitespace — deterministic formatting

M2.6 ✅ Complete: Rebuild orchestration from event log

Implementation (crates/mem-store/src/rebuild.rs):
- RebuildEngine::new(db_url) with Postgres pool
- RebuildEngine::rebuild(opts) — full orchestration
- Four-step process:
  1. Clear project (nodes cascade → edges)
  2. Read log memories → convert to MemoryNodes
  3. Upsert all nodes (ON CONFLICT DO NOTHING)
  4. Insert all edges (two-pass: nodes then edges)
  5. Project vault (M2.5)
- Three rebuild modes:
  - Default: both database + vault
  - --vault-only: skip database operations
  - --db-only: skip vault projection
- Incomplete log detection (no run_end) — error by default
- --allow-partial flag to proceed anyway
- Embedding cache by content sha256
  - Keyed on memory text hash (not node id)
  - Survives runs, reduces recomputation
- Statistics reporting: nodes by level, edges, embeddings cached/computed

Types:
- RebuildOpts: {project, vault_only, db_only, allow_partial, cache_dir, vault_dir, log_dir}
- RebuildStats: {nodes_l0, nodes_l1, nodes_l2, edges, embeddings_computed, embeddings_cached}
- Content identity via sha256(memory.text)

Tests (6 integration tests in tests/it_rebuild.rs):
1. a1_from_empty — rebuild creates expected node counts
2. a2_idempotent_db — rebuild twice = same row counts
3. a3_idempotent_vault — rebuild twice = byte-identical files
4. a5_embedding_cache_reduces_computation — cache lookup works
5. a6_incomplete_log_refused — no run_end → error unless --allow-partial
6. a7_memory_sha_content_identity — same text = same hash
7. a8_rebuild_opts_modes — mode flags work correctly

Dependency:
- crates/mem-store/Cargo.toml: added sha2 (workspace)

Updated INDEX.md:
- M2.x: 6/8 done (M2.7, M2.8 remain)
- Total: 48✅ + 2🟡 + 23⬜ (was 45✅)
- 26 new tests (M2.5: 10, M2.6: 6) + 10 utility unit tests

Architecture notes:
- M2.5 schema validates via M2.3 tables
- M2.6 uses M2.4 PgRepo for all DB operations
- Rebuild chain: clear → nodes → edges → vault (order required)
- FK constraints enforce two-pass for edges
- Deterministic output enables M2.8 gate (byte-identical verification)

2026-08-27 20:54:43 -07:00

3.8 KiB

Raw Blame History

M2.6 — `mem rebuild --from-log`

Field	Value
Phase	M2 — Projections
Size	M — 1–3 days
Status	✅ Done
Flags	—
Spec	inlined below
Blocks	M2.4, M2.5

Goal

Drop both projections and rebuild them from the log alone — the command that makes "the log is authoritative" a testable claim instead of a slogan.

Facts (inlined — no spec read needed)

mem rebuild --from-log --project poimen            # both projections
mem rebuild --from-log --project poimen --vault-only
mem rebuild --from-log --project poimen --db-only

The claim: anything not reconstructible from the log has a hidden input, and that is a bug. Rebuild is the executable form of that claim. If it needs the existing vault or database to produce correct output, something is being carried across that is not in the record.

Rebuild does no model calls except embeddings. Gate decisions, memory text and provenance are all in the log already; re-running the controller would produce different text and defeat the purpose.

Order matters: clear → insert all nodes → insert all edges → project vault. Edges before nodes violates the foreign key (M2.4).

Embeddings are the expensive part. Cache by sha256 so a rebuild after a vault template change does not re-embed unchanged nodes.

Steps

mem rebuild --from-log --project P.
Read every log file for the project, in run-id order.
clear_project, then two-pass node/edge insert, batching embeddings.
Project the vault (M2.5), overwriting.
Embedding cache keyed by sha, on disk under .cache/, so it survives runs.
Report counts: nodes by level, edges, embeddings computed vs cached.
Refuse to run if any log file is incomplete (no run_end) unless --allow-partial — rebuilding from a half-run silently produces a half-memory.

Acceptance

Rebuild from an empty database and empty vault produces the full state.
Rebuild twice produces identical database rows and identical vault bytes.
No controller model calls occur.
An incomplete log is refused by default.

Verify

Harness: log fixture, disposable Postgres, temp vault. A controller client that panics if called.

Integration test — tests/it_rebuild.rs:

a1_from_empty — drop everything, rebuild, assert node counts per level match the log's records.
a2_idempotent_db — rebuild twice, assert row count unchanged and no created_at churn on existing rows.
a3_idempotent_vault — rebuild twice, assert vault bytes identical.
a4_no_controller_calls — inject a panicking chat client; assert rebuild succeeds.
a5_embedding_cache — second rebuild computes zero embeddings.
a6_edge_order — a log whose first memory references a later-inserted parent still rebuilds, proving two-pass.
a7_incomplete_refused — a log with no run_end exits non-zero; with --allow-partial it succeeds.
a8_log_is_sufficient — delete the vault and the database entirely, rebuild, and assert the result equals a committed golden. This is the authority claim.

Command: cargo test -p mem-cli rebuild

False pass:

Rebuilding on top of existing state. It masks every hidden input, because the missing piece is already there from the previous run. Assertions 1 and 8 must start from nothing.
Asserting row counts only. A rebuild that inserts the right number of rows with wrong parents passes; assertion 6 and M2.7's edge closure are what check the graph.

Traps

Re-running the controller during rebuild. It produces different memory text every time, the vault never stabilises, and M2.8 can never pass.
Caching embeddings by node id rather than content hash. Ids change between rebuilds; hashes do not, which is the whole point of content identity.

Background: DESIGN.md — Authority model

3.8 KiB Raw Blame History Unescape Escape

M2.6 — mem rebuild --from-log