poimen-memory/VAULT-GITOPS-ARCHITECTURE.md

Vault GitOps Architecture

Design principle

Obsidian vault is derived, never edited directly. All memory changes flow through the authoritative JSONL log.

JSONL log (source of truth)
    ↓ [mem rebuild --from-log]
Obsidian vault (derived, GitOps)
    ↓ [git push]
Remote vault repo

Repository structure

Parent repo: poimen-memory

• Location: /Users/rockliang/workplace/poimen-memory
• Remote: ssh://[email protected]:2222/rock/poimen-memory.git
• Contains: Rust crates (M0-M2 complete), DESIGN.md, tasks/, log/, queries/
• Tracks: Source code, task definitions, JSONL event log (authoritative)

Vault repo: poimen-obesdient-memory

• Location: /Users/rockliang/workplace/poimen-memory/vault (separate .git)
• Remote: ssh://[email protected]:2222/rock/poimen-obesdient-memory.git
• Contains: Generated memories (poimen/), skill drafts (skills/_drafts/), human docs (notes/)
• Tracks: Markdown files (generated and human-written), but NOT source code

What gets tracked

Parent repo tracks:

✅ Cargo.toml, crates/*              source code
✅ DESIGN.md, README.md              documentation
✅ tasks/M0.1-M6.6.md               38 tasks with acceptance criteria
✅ queries/poimen.yaml               standing question definitions
✅ log/poimen/**/*.jsonl             authoritative JSONL event log
✅ k8s/, migrations/                 infrastructure
✅ .gitignore                        editor temp files, build artifacts

❌ /target/, *.swp, .DS_Store       ignored
❌ vault/                            separate repo, own .git

Vault repo tracks:

✅ README.md                         workflow guide
✅ notes/*.md                        human annotations (writable)
✅ poimen/*.md                       generated L0/L1/L2 memories (read-only)
✅ poimen/evidence/*.md             generated L0 chunks (optional, read-only)
✅ skills/_drafts/*.md              generated skill drafts (read-only)
✅ .gitignore                        Obsidian metadata, private files

❌ .obsidian/                        local Obsidian settings
❌ *.readonly                        local markers

Data flow

Ingest → Memory → Vault

1. Ingest run completes
   cargo run -p mem-cli -- ingest --project poimen --query infra-root-causes

2. Creates JSONL event log
   log/poimen/infra-root-causes/<run-id>.jsonl
   
3. Commit to parent repo
   git add log/poimen/infra-root-causes/<run-id>.jsonl
   git commit -m "ingest: infra-root-causes (update-rate=0.22)"
   git push origin main

4. CI/CD trigger: log changed
   
5. Rebuild vault from log
   mem rebuild --from-log --project poimen

6. Vault markdown updated
   vault/poimen/infra-root-causes.md (generated)
   vault/poimen/index.md (L2 synthesis, generated)

7. Commit to vault repo
   cd vault
   git add poimen/*.md
   git commit -m "rebuild: infra-root-causes (2026-08-17T...)"
   git push origin main

Verification: byte-identical rebuild

M2.8 gate verifies that rebuild is deterministic:

# Delete vault and index
rm -rf vault/poimen vault/skills/_drafts
psql -c "delete from memory_node where project='poimen'"

# Rebuild from JSONL alone
mem rebuild --from-log --project poimen

# Vault must be byte-identical
git -C vault diff --exit-code
# Exit 0 = pass. Any diff = gate failed.

This guarantees: JSONL is truly the source of truth. No hidden inputs in vault projections.

Human workflow in Obsidian

1. Clone vault: git clone ssh://[email protected]:2222/rock/poimen-obesdient-memory.git
2. Open in Obsidian: File → Open vault as folder
3. Start at: notes/INDEX.md (entry point)
4. Read generated notes (links auto-generated from memory_edge provenance)
5. Add annotations in notes/my-research.md (writable, won't be overwritten)
6. Link to generated notes: [[poimen/infra-root-causes]]
7. Commit annotations normally: git commit -m "notes: my-research findings"

Generated notes in poimen/ are read-only (rebuilt on log changes). Human notes in notes/ are writable (never auto-overwritten).

Skill promotion flow

Drafts are auto-generated, read-only. Promotion is manual:

1. mem skill draft --from poimen/infra-root-causes
   Creates: vault/skills/_drafts/tool-effectiveness-v1/SKILL.md
   (auto-generated, read-only)

2. User reviews in Obsidian
   vault/skills/_drafts/tool-effectiveness-v1/SKILL.md

3. Decision: promote or wait for v2
   
4. If promote (manual git action):
   git mv skills/_drafts/tool-effectiveness-v1 skills/tool-effectiveness-v1
   git commit -m "promote: tool-effectiveness-v1"
   git push

5. Skill now loadable:
   pi --skill vault/skills/
   pi --list-skills
   # tool-effectiveness-v1 appears in list

6. Use the skill:
   # Skill carries generated_from: <L2-sha> metadata
   # Links back to L2 memory that produced it
   # No feedback loop: if used in next session, ingest marks chunk as derived:true

Authority guarantees

1. JSONL is immutable — only append new runs, never modify existing records
2. Rebuild is deterministic — same log, byte-identical vault
3. Vault is ephemeral — can be dropped and rebuilt from log
4. No vault edits corrupt the record — generated content won't persist edits
5. Promotion is auditable — git history of skill promotions

Tools used

• mem CLI: cargo run -p mem-cli -- rebuild|ingest|query|skill|verify
• pgvector: 768-dim embeddings, HNSW index for retrieval
• Ollama: Qwen2.5-3B controller (prompted, no LoRA yet)
• Obsidian: Vault app for reading, optional annotations
• Git: Both repos track; vault has separate remote

Current status

Phase	Status	Details
M0	✅ COMPLETE	8/8 tasks, 35 tests
M1	✅ COMPLETE	8/8 tasks, 30+ tests
M2	✅ COMPLETE (core)	5/8 tasks, 26 tests
M3	⏳ READY	L2 synthesis, retrieval (blocked on M2 completion)
M4	⏳ READY	Skill drafting (blocked on M3)
M5	⏳ DEFERRED	Post-training, LoRA adapter (separate Python)
Vault	✅ DEPLOYED	Separate repo, documentation migrated, GitOps ready

Next steps

1. M3 implementation: L2 pass, HNSW retrieval, reranking
2. First live run: Full ingest → rebuild → vault sync
3. Skill drafting: M4 implementation
4. Post-training: M5 (Python, separate from main Rust)

References

• Design: DESIGN.md — full system design (460 lines)
• Vault docs: vault/notes/INDEX.md — entry point for Obsidian users
• Task board: tasks/INDEX.md — 64 tasks, 6 phases
• GitOps: This file — authority model and data flow