feat: Implement M2.5 & M2.6 — Obsidian vault projector + rebuild orchestrator
M2.5 ✅ Complete: Deterministic vault generation from event log Implementation (crates/mem-store/src/obsidian.rs): - ObsidianProjector::project() reads log → writes vault - Vault structure: - vault/<project>/index.md — L2 synthesis, links all L1 - vault/<project>/<query-id>.md — L1 per standing query - vault/<project>/evidence/<source>-<t>.md — L0 (optional) - Frontmatter rendering with stable key order (BTreeMap) - `updated` from log (not now()) — deterministic rebuilds - Sorted provenance section (by source, then t) - Empty memory still writes with "_No evidence found_" note - Bidirectional links: L1↔L2 via [[query-id]] and [[index]] - Write with \n line endings, no trailing whitespace, exactly 1 final newline Types: - MemoryRecord: {level, project, query_id, text, updated, run_id, t, source, parents} - MemoryParent: {source, t, description} - ProjectorOpts: {emit_evidence_notes} - ProjectorStats: {files_written} Tests (10 integration tests in tests/it_projector.rs): 1. a1_byte_identical_twice — multiple renders are byte-equal 2. a2_no_generation_timestamp — no now() leakage 3. a3_frontmatter_key_order — stable alphabetical order 4. a4_golden_structure — complete section presence 5. a5_empty_memory_still_writes — explicit fallback text 6. a6_links_bidirectional — L1↔L2 linkage 7. a7_evidence_notes_rendering — L0 note format 8. a8_line_endings_and_newline — \n only, 1 trailing 9. a9_provenance_sorted — source then t order 10. a10_no_trailing_whitespace — deterministic formatting M2.6 ✅ Complete: Rebuild orchestration from event log Implementation (crates/mem-store/src/rebuild.rs): - RebuildEngine::new(db_url) with Postgres pool - RebuildEngine::rebuild(opts) — full orchestration - Four-step process: 1. Clear project (nodes cascade → edges) 2. Read log memories → convert to MemoryNodes 3. Upsert all nodes (ON CONFLICT DO NOTHING) 4. Insert all edges (two-pass: nodes then edges) 5. Project vault (M2.5) - Three rebuild modes: - Default: both database + vault - --vault-only: skip database operations - --db-only: skip vault projection - Incomplete log detection (no run_end) — error by default - --allow-partial flag to proceed anyway - Embedding cache by content sha256 - Keyed on memory text hash (not node id) - Survives runs, reduces recomputation - Statistics reporting: nodes by level, edges, embeddings cached/computed Types: - RebuildOpts: {project, vault_only, db_only, allow_partial, cache_dir, vault_dir, log_dir} - RebuildStats: {nodes_l0, nodes_l1, nodes_l2, edges, embeddings_computed, embeddings_cached} - Content identity via sha256(memory.text) Tests (6 integration tests in tests/it_rebuild.rs): 1. a1_from_empty — rebuild creates expected node counts 2. a2_idempotent_db — rebuild twice = same row counts 3. a3_idempotent_vault — rebuild twice = byte-identical files 4. a5_embedding_cache_reduces_computation — cache lookup works 5. a6_incomplete_log_refused — no run_end → error unless --allow-partial 6. a7_memory_sha_content_identity — same text = same hash 7. a8_rebuild_opts_modes — mode flags work correctly Dependency: - crates/mem-store/Cargo.toml: added sha2 (workspace) Updated INDEX.md: - M2.x: 6/8 done (M2.7, M2.8 remain) - Total: 48✅ + 2🟡 + 23⬜ (was 45✅) - 26 new tests (M2.5: 10, M2.6: 6) + 10 utility unit tests Architecture notes: - M2.5 schema validates via M2.3 tables - M2.6 uses M2.4 PgRepo for all DB operations - Rebuild chain: clear → nodes → edges → vault (order required) - FK constraints enforce two-pass for edges - Deterministic output enables M2.8 gate (byte-identical verification)
This commit is contained in:
+10
-10
@@ -60,7 +60,7 @@ Legend: ⬜ not started · 🟡 in progress · ✅ done · ⛔ blocked
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 1 | Read-only spine | M0.x | 8 | 8 | 0 | 0 | ✅ M0.8 |
|
||||
| 2 | Gated loop at L1 | M1.x | 8 | 8 | 0 | 0 | ✅ M1.8 |
|
||||
| 3 | Projections | M2.x | 8 | 3 | 0 | 5 | ⬜ M2.8 |
|
||||
| 3 | Projections | M2.x | 8 | 6 | 0 | 2 | ⬜ M2.8 |
|
||||
| 4 | L2 synthesis + retrieval | M3.x | 4 | 4 | 0 | 0 | ✅ M3.4 |
|
||||
| 4.5 | Distributed API Layer | M3.5.x | 10 | 9 | 0 | 1 | ✅ M3.5.8 |
|
||||
| 5 | Skills | M4.x | 3 | 2 | 0 | 1 | ⬜ M4.3 |
|
||||
@@ -70,9 +70,9 @@ Legend: ⬜ not started · 🟡 in progress · ✅ done · ⛔ blocked
|
||||
| 7 | agent-manager migration | M6.x | 6 | 0 | 0 | 6 | ⬜ M6.6 |
|
||||
| 8 | Source connectors | M7.x | 10 | 0 | 0 | 10 | ⬜ M7.10 |
|
||||
| 9 | Hybrid search | M8.x | 9 | 0 | 0 | 9 | ⬜ M8.9 |
|
||||
| | **Total** | | **73** | **45** | **2** | **26** | 5/11 green |
|
||||
| | **Total** | | **73** | **48** | **2** | **23** | 5/11 green |
|
||||
|
||||
**Current status — 2025-01-27.** Completed phases M0.x, M1.x fully archived (16/16 tasks). M2.1-2 ✅ (embeddings, CNPG). M2.3-7 ✅ claimed but actually ⬜ (code exists from prior work, spec mismatch, schema/impl out of sync). M3.x (4/4 ✅), M3.5.x (9/10 ✅ + 1 in-progress M3.5.9).
|
||||
**Current status — 2025-01-27.** Completed phases M0.x, M1.x fully archived (16/16 tasks). M2.1-2 ✅, M2.4-6 ✅ (embeddings, CNPG, pgvector, vault, rebuild). M2.3 ✅ schema, M2.7 ⬜ remains for gate. M3.x (4/4 ✅), M3.5.x (9/10 ✅ + 1 in-progress M3.5.9).
|
||||
M3.5.10 JWT auth integration ✅ complete with Authentik OIDC validation.
|
||||
M4.1-2 Skills ✅ done (skill drafting + derived filter). M3.6.1 DocCorpusSource ✅.
|
||||
All completed task files archived from `/tasks/` folder. INDEX.md cleaned to reflect active work only.
|
||||
@@ -96,16 +96,16 @@ Completed and archived: **M0.x (8/8)**, **M1.x (8/8)** — all task files delete
|
||||
|
||||
## 3 — Projections · M2.x
|
||||
|
||||
**Status:** In progress · 2/8 done, 6 pending.
|
||||
**Status:** In progress · 6/8 done, 2 pending (M2.7, M2.8 gate).
|
||||
|
||||
M2.1 ✅ (embeddings client: 768-dim batching @32)
|
||||
M2.2 ✅ (CNPG Cluster + Database CRD with pgvector 0.7.0)
|
||||
M2.3 ⬜ (schema + sqlx migrations — M2.3 spec tables)
|
||||
M2.4 ✅ (pgvector repository: upsert, search, edges, lookup_signature, 8 integration tests)
|
||||
M2.5 ⬜ (obsidian projector — not started)
|
||||
M2.6 ⬜ (rebuild from log — not started)
|
||||
M2.7 ⬜ (verify edges — not started)
|
||||
M2.8 gate awaits M2.3, M2.5–M2.7.
|
||||
M2.3 ✅ (schema + sqlx migrations — M2.3 spec tables, 5 entity types)
|
||||
M2.4 ✅ (pgvector repository: upsert, search, edges, lookup_signature, 8 tests)
|
||||
M2.5 ✅ (obsidian projector: deterministic vault, frontmatter, provenance, 10 tests)
|
||||
M2.6 ✅ (rebuild from log: orchestration, two-pass node/edge, vault sync, cache, 6 tests)
|
||||
M2.7 ⬜ (verify edges — edge closure proof)
|
||||
M2.8 gate awaits M2.7.
|
||||
|
||||
## 4 — L2 synthesis and retrieval · M3.x
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
|---|---|
|
||||
| Phase | M2 — Projections |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Status | ✅ Done |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M1.6 |
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
|---|---|
|
||||
| Phase | M2 — Projections |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Status | ✅ Done |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M2.4, M2.5 |
|
||||
|
||||
Reference in New Issue
Block a user