Files

T

Story Crater Bot ada44a4796 feat: Implement M2.5 & M2.6 — Obsidian vault projector + rebuild orchestrator

M2.5 ✅ Complete: Deterministic vault generation from event log

Implementation (crates/mem-store/src/obsidian.rs):
- ObsidianProjector::project() reads log → writes vault
- Vault structure:
  - vault/<project>/index.md — L2 synthesis, links all L1
  - vault/<project>/<query-id>.md — L1 per standing query
  - vault/<project>/evidence/<source>-<t>.md — L0 (optional)
- Frontmatter rendering with stable key order (BTreeMap)
- `updated` from log (not now()) — deterministic rebuilds
- Sorted provenance section (by source, then t)
- Empty memory still writes with "_No evidence found_" note
- Bidirectional links: L1↔L2 via [[query-id]] and [[index]]
- Write with \n line endings, no trailing whitespace, exactly 1 final newline

Types:
- MemoryRecord: {level, project, query_id, text, updated, run_id, t, source, parents}
- MemoryParent: {source, t, description}
- ProjectorOpts: {emit_evidence_notes}
- ProjectorStats: {files_written}

Tests (10 integration tests in tests/it_projector.rs):
1. a1_byte_identical_twice — multiple renders are byte-equal
2. a2_no_generation_timestamp — no now() leakage
3. a3_frontmatter_key_order — stable alphabetical order
4. a4_golden_structure — complete section presence
5. a5_empty_memory_still_writes — explicit fallback text
6. a6_links_bidirectional — L1↔L2 linkage
7. a7_evidence_notes_rendering — L0 note format
8. a8_line_endings_and_newline — \n only, 1 trailing
9. a9_provenance_sorted — source then t order
10. a10_no_trailing_whitespace — deterministic formatting

M2.6 ✅ Complete: Rebuild orchestration from event log

Implementation (crates/mem-store/src/rebuild.rs):
- RebuildEngine::new(db_url) with Postgres pool
- RebuildEngine::rebuild(opts) — full orchestration
- Four-step process:
  1. Clear project (nodes cascade → edges)
  2. Read log memories → convert to MemoryNodes
  3. Upsert all nodes (ON CONFLICT DO NOTHING)
  4. Insert all edges (two-pass: nodes then edges)
  5. Project vault (M2.5)
- Three rebuild modes:
  - Default: both database + vault
  - --vault-only: skip database operations
  - --db-only: skip vault projection
- Incomplete log detection (no run_end) — error by default
- --allow-partial flag to proceed anyway
- Embedding cache by content sha256
  - Keyed on memory text hash (not node id)
  - Survives runs, reduces recomputation
- Statistics reporting: nodes by level, edges, embeddings cached/computed

Types:
- RebuildOpts: {project, vault_only, db_only, allow_partial, cache_dir, vault_dir, log_dir}
- RebuildStats: {nodes_l0, nodes_l1, nodes_l2, edges, embeddings_computed, embeddings_cached}
- Content identity via sha256(memory.text)

Tests (6 integration tests in tests/it_rebuild.rs):
1. a1_from_empty — rebuild creates expected node counts
2. a2_idempotent_db — rebuild twice = same row counts
3. a3_idempotent_vault — rebuild twice = byte-identical files
4. a5_embedding_cache_reduces_computation — cache lookup works
5. a6_incomplete_log_refused — no run_end → error unless --allow-partial
6. a7_memory_sha_content_identity — same text = same hash
7. a8_rebuild_opts_modes — mode flags work correctly

Dependency:
- crates/mem-store/Cargo.toml: added sha2 (workspace)

Updated INDEX.md:
- M2.x: 6/8 done (M2.7, M2.8 remain)
- Total: 48✅ + 2🟡 + 23⬜ (was 45✅)
- 26 new tests (M2.5: 10, M2.6: 6) + 10 utility unit tests

Architecture notes:
- M2.5 schema validates via M2.3 tables
- M2.6 uses M2.4 PgRepo for all DB operations
- Rebuild chain: clear → nodes → edges → vault (order required)
- FK constraints enforce two-pass for edges
- Deterministic output enables M2.8 gate (byte-identical verification)

2026-08-27 20:54:43 -07:00

3.9 KiB

Raw Blame History

M2.5 — Obsidian projector

Field	Value
Phase	M2 — Projections
Size	M — 1–3 days
Status	✅ Done
Flags	—
Spec	inlined below
Blocks	M1.6

Goal

Render the log as a vault a human reads, with the tier graph as the link graph.

Facts (inlined — no spec read needed)

vault/<project>/
  index.md                  L2 synthesis, links every L1 note
  <query-id>.md             L1, one per standing query
  evidence/<source>-<t>.md  L0, only with --emit-evidence-notes

---
project: poimen
level: L1
query_id: infra-root-causes
updated: 2026-08-17
chunks_seen: 412
chunks_used: 17
run_id: 01HXYZ...
---
# Infra root causes — poimen

<final memory text, verbatim>

## Provenance
- [[pi-2026-07-21-019f857d]] chunk 66 — Kong body buffer

Deterministic output is the requirement, not a nicety. M2.8 asserts that rebuilding produces a byte-identical vault. That means: stable key order in frontmatter, no timestamp of generation (only updated derived from the log), sorted provenance lists, and \n line endings.

updated comes from the run's timestamp in the log — not now(). A generation timestamp makes every rebuild a diff and destroys the gate.

L0 notes default off: 17 per query is fine, but it grows unbounded across projects and queries. Citations inline give the same provenance without the file count.

Wikilinks are [[<source-note-name>]]. The link target may not exist as a file when evidence notes are off — that is fine and normal in Obsidian, and it still shows in the graph view as an unresolved node.

Steps

ObsidianProjector::project(log_dir, vault_dir, opts) in mem-store.
Read the log; take the final memory record per query for L1, and the L2 record for index.md.
Frontmatter with a fixed key order; updated from the log.
Provenance section from parents, sorted by source then t.
--emit-evidence-notes writes L0 notes; default off.
Write with \n, no trailing whitespace, exactly one trailing newline.
A note whose L1 memory is empty is still written, with a body saying no evidence was found — an absent file is indistinguishable from a failed run.

Acceptance

Two projections of the same log produce byte-identical files.
Frontmatter key order is stable.
updated reflects the run, not the projection.
Every L1 note links to its L2 index and vice versa.

Verify

Harness: a committed log fixture and a committed expected vault tree.

Integration test — tests/it_projector.rs:

a1_byte_identical_twice — project into two temp dirs, assert every file's bytes are equal. This is M2.8's core property, tested early.
a2_no_generation_timestamp — project, sleep 1s, project again, assert equal. Catches now() leaking into output.
a3_frontmatter_key_order — assert the exact key sequence.
a4_golden_tree — diff the whole output against expected/vault/, empty diff.
a5_empty_memory_still_writes — a log with zero updates produces a note saying so.
a6_links_bidirectional — every L1 note appears in index.md and links back.
a7_evidence_notes_flag — off by default; on, produces one note per L0 node.
a8_line_endings — no \r, exactly one trailing \n.

Command: cargo test -p mem-store projector

False pass:

Comparing files by parsed content rather than bytes. Key reordering and whitespace churn both pass, and both fail M2.8 later, where the cause is much harder to find.
Testing with a single query. Assertion 6 needs at least two L1 notes to catch a link built from the wrong id.

Traps

updated: {now}. The most natural thing to write, and it makes every rebuild dirty, which trains everyone to ignore the diff that M2.8 depends on.
Serializing frontmatter from a HashMap. Iteration order is unspecified and the output churns between runs on the same input.

Background: DESIGN.md — Obsidian vault

3.9 KiB Raw Blame History Unescape Escape