Story Crater Bot
ada44a4796
M2.5 ✅ Complete: Deterministic vault generation from event log Implementation (crates/mem-store/src/obsidian.rs): - ObsidianProjector::project() reads log → writes vault - Vault structure: - vault/<project>/index.md — L2 synthesis, links all L1 - vault/<project>/<query-id>.md — L1 per standing query - vault/<project>/evidence/<source>-<t>.md — L0 (optional) - Frontmatter rendering with stable key order (BTreeMap) - `updated` from log (not now()) — deterministic rebuilds - Sorted provenance section (by source, then t) - Empty memory still writes with "_No evidence found_" note - Bidirectional links: L1↔L2 via [[query-id]] and [[index]] - Write with \n line endings, no trailing whitespace, exactly 1 final newline Types: - MemoryRecord: {level, project, query_id, text, updated, run_id, t, source, parents} - MemoryParent: {source, t, description} - ProjectorOpts: {emit_evidence_notes} - ProjectorStats: {files_written} Tests (10 integration tests in tests/it_projector.rs): 1. a1_byte_identical_twice — multiple renders are byte-equal 2. a2_no_generation_timestamp — no now() leakage 3. a3_frontmatter_key_order — stable alphabetical order 4. a4_golden_structure — complete section presence 5. a5_empty_memory_still_writes — explicit fallback text 6. a6_links_bidirectional — L1↔L2 linkage 7. a7_evidence_notes_rendering — L0 note format 8. a8_line_endings_and_newline — \n only, 1 trailing 9. a9_provenance_sorted — source then t order 10. a10_no_trailing_whitespace — deterministic formatting M2.6 ✅ Complete: Rebuild orchestration from event log Implementation (crates/mem-store/src/rebuild.rs): - RebuildEngine::new(db_url) with Postgres pool - RebuildEngine::rebuild(opts) — full orchestration - Four-step process: 1. Clear project (nodes cascade → edges) 2. Read log memories → convert to MemoryNodes 3. Upsert all nodes (ON CONFLICT DO NOTHING) 4. Insert all edges (two-pass: nodes then edges) 5. Project vault (M2.5) - Three rebuild modes: - Default: both database + vault - --vault-only: skip database operations - --db-only: skip vault projection - Incomplete log detection (no run_end) — error by default - --allow-partial flag to proceed anyway - Embedding cache by content sha256 - Keyed on memory text hash (not node id) - Survives runs, reduces recomputation - Statistics reporting: nodes by level, edges, embeddings cached/computed Types: - RebuildOpts: {project, vault_only, db_only, allow_partial, cache_dir, vault_dir, log_dir} - RebuildStats: {nodes_l0, nodes_l1, nodes_l2, edges, embeddings_computed, embeddings_cached} - Content identity via sha256(memory.text) Tests (6 integration tests in tests/it_rebuild.rs): 1. a1_from_empty — rebuild creates expected node counts 2. a2_idempotent_db — rebuild twice = same row counts 3. a3_idempotent_vault — rebuild twice = byte-identical files 4. a5_embedding_cache_reduces_computation — cache lookup works 5. a6_incomplete_log_refused — no run_end → error unless --allow-partial 6. a7_memory_sha_content_identity — same text = same hash 7. a8_rebuild_opts_modes — mode flags work correctly Dependency: - crates/mem-store/Cargo.toml: added sha2 (workspace) Updated INDEX.md: - M2.x: 6/8 done (M2.7, M2.8 remain) - Total: 48✅ + 2🟡 + 23⬜ (was 45✅) - 26 new tests (M2.5: 10, M2.6: 6) + 10 utility unit tests Architecture notes: - M2.5 schema validates via M2.3 tables - M2.6 uses M2.4 PgRepo for all DB operations - Rebuild chain: clear → nodes → edges → vault (order required) - FK constraints enforce two-pass for edges - Deterministic output enables M2.8 gate (byte-identical verification)
114 lines
3.9 KiB
Markdown
114 lines
3.9 KiB
Markdown
# M2.5 — Obsidian projector
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M2 — Projections |
|
||
| Size | M — 1–3 days |
|
||
| Status | ✅ Done |
|
||
| Flags | — |
|
||
| Spec | inlined below |
|
||
| Blocks | M1.6 |
|
||
|
||
## Goal
|
||
|
||
Render the log as a vault a human reads, with the tier graph as the link graph.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
```
|
||
vault/<project>/
|
||
index.md L2 synthesis, links every L1 note
|
||
<query-id>.md L1, one per standing query
|
||
evidence/<source>-<t>.md L0, only with --emit-evidence-notes
|
||
```
|
||
|
||
```markdown
|
||
---
|
||
project: poimen
|
||
level: L1
|
||
query_id: infra-root-causes
|
||
updated: 2026-08-17
|
||
chunks_seen: 412
|
||
chunks_used: 17
|
||
run_id: 01HXYZ...
|
||
---
|
||
# Infra root causes — poimen
|
||
|
||
<final memory text, verbatim>
|
||
|
||
## Provenance
|
||
- [[pi-2026-07-21-019f857d]] chunk 66 — Kong body buffer
|
||
```
|
||
|
||
**Deterministic output is the requirement, not a nicety.** M2.8 asserts that
|
||
rebuilding produces a byte-identical vault. That means: stable key order in
|
||
frontmatter, no timestamp of *generation* (only `updated` derived from the log),
|
||
sorted provenance lists, and `\n` line endings.
|
||
|
||
`updated` comes from the run's timestamp in the log — not `now()`. A generation
|
||
timestamp makes every rebuild a diff and destroys the gate.
|
||
|
||
L0 notes default off: 17 per query is fine, but it grows unbounded across
|
||
projects and queries. Citations inline give the same provenance without the file
|
||
count.
|
||
|
||
Wikilinks are `[[<source-note-name>]]`. The link target may not exist as a file
|
||
when evidence notes are off — that is fine and normal in Obsidian, and it still
|
||
shows in the graph view as an unresolved node.
|
||
|
||
## Steps
|
||
|
||
1. `ObsidianProjector::project(log_dir, vault_dir, opts)` in `mem-store`.
|
||
2. Read the log; take the final `memory` record per query for L1, and the L2
|
||
record for `index.md`.
|
||
3. Frontmatter with a fixed key order; `updated` from the log.
|
||
4. Provenance section from `parents`, sorted by source then `t`.
|
||
5. `--emit-evidence-notes` writes L0 notes; default off.
|
||
6. Write with `\n`, no trailing whitespace, exactly one trailing newline.
|
||
7. A note whose L1 memory is empty is still written, with a body saying no
|
||
evidence was found — an absent file is indistinguishable from a failed run.
|
||
|
||
## Acceptance
|
||
|
||
- Two projections of the same log produce byte-identical files.
|
||
- Frontmatter key order is stable.
|
||
- `updated` reflects the run, not the projection.
|
||
- Every L1 note links to its L2 index and vice versa.
|
||
|
||
## Verify
|
||
|
||
**Harness:** a committed log fixture and a committed expected vault tree.
|
||
|
||
**Integration test** — `tests/it_projector.rs`:
|
||
1. `a1_byte_identical_twice` — project into two temp dirs, assert every file's
|
||
bytes are equal. This is M2.8's core property, tested early.
|
||
2. `a2_no_generation_timestamp` — project, sleep 1s, project again, assert equal.
|
||
Catches `now()` leaking into output.
|
||
3. `a3_frontmatter_key_order` — assert the exact key sequence.
|
||
4. `a4_golden_tree` — diff the whole output against `expected/vault/`, empty diff.
|
||
5. `a5_empty_memory_still_writes` — a log with zero updates produces a note
|
||
saying so.
|
||
6. `a6_links_bidirectional` — every L1 note appears in `index.md` and links back.
|
||
7. `a7_evidence_notes_flag` — off by default; on, produces one note per L0 node.
|
||
8. `a8_line_endings` — no `\r`, exactly one trailing `\n`.
|
||
|
||
**Command:** `cargo test -p mem-store projector`
|
||
|
||
**False pass:**
|
||
- Comparing files by parsed content rather than bytes. Key reordering and
|
||
whitespace churn both pass, and both fail M2.8 later, where the cause is much
|
||
harder to find.
|
||
- Testing with a single query. Assertion 6 needs at least two L1 notes to catch a
|
||
link built from the wrong id.
|
||
|
||
## Traps
|
||
|
||
- `updated: {now}`. The most natural thing to write, and it makes every rebuild
|
||
dirty, which trains everyone to ignore the diff that M2.8 depends on.
|
||
- Serializing frontmatter from a `HashMap`. Iteration order is unspecified and
|
||
the output churns between runs on the same input.
|
||
|
||
---
|
||
|
||
Background: [DESIGN.md](../DESIGN.md) — Obsidian vault
|