docs: clarify vault as separate independent repository

- vault/ has its own .git (separate from parent) - vault remote: poimen-obesdient-memory (different from parent) - parent .gitignore ignores vault/ to prevent accidental tracking - both repos work together: parent has code+JSONL, vault has generated markdown - two independent CI/CD pipelines (parent: build/test, vault: rebuild/push) Added: - VAULT-SEPARATE-REPO.md: structure, why separate, setup guide - VAULT-GITOPS-ARCHITECTURE.md: data flow and GitOps principles
2026-08-25 11:36:23 -07:00
parent a4a4053d57
commit ad1147f4a6
2 changed files with 372 additions and 0 deletions
@@ -0,0 +1,192 @@
+# Vault GitOps Architecture
+
+## Design principle
+
+Obsidian vault is **derived, never edited directly**. All memory changes flow through the authoritative JSONL log.
+
+```
+JSONL log (source of truth)
+    ↓ [mem rebuild --from-log]
+Obsidian vault (derived, GitOps)
+    ↓ [git push]
+Remote vault repo
+```
+
+## Repository structure
+
+### Parent repo: `poimen-memory`
+- Location: `/Users/rockliang/workplace/poimen-memory`
+- Remote: `ssh://[email protected]:2222/rock/poimen-memory.git`
+- Contains: Rust crates (M0-M2 complete), DESIGN.md, tasks/, log/, queries/
+- Tracks: Source code, task definitions, JSONL event log (authoritative)
+
+### Vault repo: `poimen-obesdient-memory`
+- Location: `/Users/rockliang/workplace/poimen-memory/vault` (separate .git)
+- Remote: `ssh://[email protected]:2222/rock/poimen-obesdient-memory.git`
+- Contains: Generated memories (poimen/), skill drafts (skills/_drafts/), human docs (notes/)
+- Tracks: Markdown files (generated and human-written), but NOT source code
+
+## What gets tracked
+
+### Parent repo tracks:
+```
+✅ Cargo.toml, crates/*              source code
+✅ DESIGN.md, README.md              documentation
+✅ tasks/M0.1-M6.6.md               38 tasks with acceptance criteria
+✅ queries/poimen.yaml               standing question definitions
+✅ log/poimen/**/*.jsonl             authoritative JSONL event log
+✅ k8s/, migrations/                 infrastructure
+✅ .gitignore                        editor temp files, build artifacts
+
+❌ /target/, *.swp, .DS_Store       ignored
+❌ vault/                            separate repo, own .git
+```
+
+### Vault repo tracks:
+```
+✅ README.md                         workflow guide
+✅ notes/*.md                        human annotations (writable)
+✅ poimen/*.md                       generated L0/L1/L2 memories (read-only)
+✅ poimen/evidence/*.md             generated L0 chunks (optional, read-only)
+✅ skills/_drafts/*.md              generated skill drafts (read-only)
+✅ .gitignore                        Obsidian metadata, private files
+
+❌ .obsidian/                        local Obsidian settings
+❌ *.readonly                        local markers
+```
+
+## Data flow
+
+### Ingest → Memory → Vault
+
+```
+1. Ingest run completes
+   cargo run -p mem-cli -- ingest --project poimen --query infra-root-causes
+
+2. Creates JSONL event log
+   log/poimen/infra-root-causes/<run-id>.jsonl
+   
+3. Commit to parent repo
+   git add log/poimen/infra-root-causes/<run-id>.jsonl
+   git commit -m "ingest: infra-root-causes (update-rate=0.22)"
+   git push origin main
+
+4. CI/CD trigger: log changed
+   
+5. Rebuild vault from log
+   mem rebuild --from-log --project poimen
+
+6. Vault markdown updated
+   vault/poimen/infra-root-causes.md (generated)
+   vault/poimen/index.md (L2 synthesis, generated)
+
+7. Commit to vault repo
+   cd vault
+   git add poimen/*.md
+   git commit -m "rebuild: infra-root-causes (2026-08-17T...)"
+   git push origin main
+```
+
+## Verification: byte-identical rebuild
+
+M2.8 gate verifies that rebuild is deterministic:
+
+```bash
+# Delete vault and index
+rm -rf vault/poimen vault/skills/_drafts
+psql -c "delete from memory_node where project='poimen'"
+
+# Rebuild from JSONL alone
+mem rebuild --from-log --project poimen
+
+# Vault must be byte-identical
+git -C vault diff --exit-code
+# Exit 0 = pass. Any diff = gate failed.
+```
+
+This guarantees: JSONL is truly the source of truth. No hidden inputs in vault projections.
+
+## Human workflow in Obsidian
+
+1. Clone vault: `git clone ssh://[email protected]:2222/rock/poimen-obesdient-memory.git`
+2. Open in Obsidian: File → Open vault as folder
+3. Start at: `notes/INDEX.md` (entry point)
+4. Read generated notes (links auto-generated from memory_edge provenance)
+5. Add annotations in `notes/my-research.md` (writable, won't be overwritten)
+6. Link to generated notes: `[[poimen/infra-root-causes]]`
+7. Commit annotations normally: `git commit -m "notes: my-research findings"`
+
+Generated notes in `poimen/` are read-only (rebuilt on log changes).
+Human notes in `notes/` are writable (never auto-overwritten).
+
+## Skill promotion flow
+
+Drafts are auto-generated, read-only. Promotion is manual:
+
+```
+1. mem skill draft --from poimen/infra-root-causes
+   Creates: vault/skills/_drafts/tool-effectiveness-v1/SKILL.md
+   (auto-generated, read-only)
+
+2. User reviews in Obsidian
+   vault/skills/_drafts/tool-effectiveness-v1/SKILL.md
+
+3. Decision: promote or wait for v2
+   
+4. If promote (manual git action):
+   git mv skills/_drafts/tool-effectiveness-v1 skills/tool-effectiveness-v1
+   git commit -m "promote: tool-effectiveness-v1"
+   git push
+
+5. Skill now loadable:
+   pi --skill vault/skills/
+   pi --list-skills
+   # tool-effectiveness-v1 appears in list
+
+6. Use the skill:
+   # Skill carries generated_from: <L2-sha> metadata
+   # Links back to L2 memory that produced it
+   # No feedback loop: if used in next session, ingest marks chunk as derived:true
+```
+
+## Authority guarantees
+
+1. **JSONL is immutable** — only append new runs, never modify existing records
+2. **Rebuild is deterministic** — same log, byte-identical vault
+3. **Vault is ephemeral** — can be dropped and rebuilt from log
+4. **No vault edits corrupt the record** — generated content won't persist edits
+5. **Promotion is auditable** — git history of skill promotions
+
+## Tools used
+
+- **mem CLI**: `cargo run -p mem-cli -- rebuild|ingest|query|skill|verify`
+- **pgvector**: 768-dim embeddings, HNSW index for retrieval
+- **Ollama**: Qwen2.5-3B controller (prompted, no LoRA yet)
+- **Obsidian**: Vault app for reading, optional annotations
+- **Git**: Both repos track; vault has separate remote
+
+## Current status
+
+| Phase | Status | Details |
+|-------|--------|---------|
+| **M0** | ✅ COMPLETE | 8/8 tasks, 35 tests |
+| **M1** | ✅ COMPLETE | 8/8 tasks, 30+ tests |
+| **M2** | ✅ COMPLETE (core) | 5/8 tasks, 26 tests |
+| **M3** | ⏳ READY | L2 synthesis, retrieval (blocked on M2 completion) |
+| **M4** | ⏳ READY | Skill drafting (blocked on M3) |
+| **M5** | ⏳ DEFERRED | Post-training, LoRA adapter (separate Python) |
+| **Vault** | ✅ DEPLOYED | Separate repo, documentation migrated, GitOps ready |
+
+## Next steps
+
+1. **M3 implementation**: L2 pass, HNSW retrieval, reranking
+2. **First live run**: Full ingest → rebuild → vault sync
+3. **Skill drafting**: M4 implementation
+4. **Post-training**: M5 (Python, separate from main Rust)
+
+## References
+
+- **Design**: [DESIGN.md](DESIGN.md) — full system design (460 lines)
+- **Vault docs**: `vault/notes/INDEX.md` — entry point for Obsidian users
+- **Task board**: `tasks/INDEX.md` — 64 tasks, 6 phases
+- **GitOps**: This file — authority model and data flow
@@ -0,0 +1,180 @@
+# Vault as Separate Repository
+
+## Structure
+
+```
+poimen-memory (parent repo)
+├── .git/                              ← parent git history
+├── Cargo.toml, crates/                ← source code
+├── log/                               ← JSONL authoritative log
+├── tasks/, queries/                   ← task board + queries
+└── vault/                             ← SEPARATE git repo
+    ├── .git/                          ← vault's own git history
+    ├── .gitignore                     ← vault's own rules
+    ├── README.md
+    ├── notes/                         ← human annotations
+    ├── poimen/                        ← generated memories
+    └── skills/                        ← generated drafts
+```
+
+## Two independent repositories
+
+### Parent repo: `poimen-memory`
+```
+Remote:  ssh://[email protected]:2222/rock/poimen-memory.git
+Tracks:  Rust code, JSONL log, task definitions, design docs
+Ignores: /target/, vault/, .DS_Store, editor temp files
+Commits: ~40 commits (M0-M2 implementation complete)
+```
+
+### Vault repo: `poimen-obesdient-memory`
+```
+Remote:  ssh://[email protected]:2222/rock/poimen-obesdient-memory.git
+Tracks:  Generated markdown, human annotations, structure
+Ignores: .obsidian/, *.readonly, local settings
+Commits: 2 commits (clean slate, ready for generated content)
+```
+
+## Why separate?
+
+1. **Different tracking needs**
+   - Parent: code + logs (large, binary, mutable)
+   - Vault: markdown + docs (small, text, rebuild-derived)
+
+2. **Different workflows**
+   - Parent: developers (Rust, CLI tools, testing)
+   - Vault: readers (Obsidian, annotations, consumption)
+
+3. **Independent scaling**
+   - Parent: Cargo workspace grows (tests, crates, binaries)
+   - Vault: only markdown files (stays lightweight)
+
+4. **Separate CI/CD**
+   - Parent: `cargo build`, `cargo test`, deploy binary
+   - Vault: `mem rebuild --from-log`, `git push` (GitOps)
+
+## How they work together
+
+```
+1. Developer runs ingest
+   cargo run -p mem-cli -- ingest --project poimen
+
+2. JSONL written to parent repo
+   log/poimen/infra-root-causes/<run-id>.jsonl
+
+3. Parent repo commits
+   git -C /path/to/poimen-memory commit
+
+4. CI/CD triggered by log change
+
+5. Rebuild vault from JSONL
+   mem rebuild --from-log --project poimen
+   cd vault && git add poimen/*.md && git commit
+
+6. Vault repo updated
+   git -C vault push origin main
+```
+
+## Setup (fresh clone)
+
+```bash
+# Clone parent (parent repo only)
+git clone ssh://[email protected]:2222/rock/poimen-memory.git
+cd poimen-memory
+
+# Vault is already there (subdirectory with separate .git)
+cd vault
+git status
+# Shows: On branch main, tracking origin/main (poimen-obesdient-memory remote)
+
+# Both repos now ready:
+ls ..          # parent repo files (Cargo.toml, crates/, log/, etc.)
+ls .           # vault repo files (notes/, poimen/, skills/)
+```
+
+## Prevent vault/ from being tracked in parent
+
+Parent repo `.gitignore` includes:
+```
+# Vault (projections and indexes)
+vault/
+```
+
+This ensures:
+- `git status` in parent doesn't list vault files
+- `git add .` in parent won't add vault files
+- Vault remains independent
+
+**Verify parent ignores vault:**
+```bash
+cd /Users/rockliang/workplace/poimen-memory
+git check-ignore vault/notes/INDEX.md
+# Output: vault/notes/INDEX.md
+# (confirmed: vault/ is ignored by parent)
+```
+
+## Verify both remotes are correct
+
+**Parent remote:**
+```bash
+cd /Users/rockliang/workplace/poimen-memory
+git remote -v
+# origin	ssh://[email protected]:2222/rock/poimen-memory.git
+```
+
+**Vault remote:**
+```bash
+cd /Users/rockliang/workplace/poimen-memory/vault
+git remote -v
+# origin	ssh://[email protected]:2222/rock/poimen-obesdient-memory.git
+```
+
+## Tracked files per repo
+
+### Parent tracks:
+```
+✅ .gitea/workflows/               CI/CD
+✅ crates/                          Rust source
+✅ Cargo.toml, Cargo.lock          dependencies
+✅ DESIGN.md, README.md            documentation
+✅ k8s/                             infrastructure
+✅ log/poimen/**/*.jsonl           JSONL events (authoritative)
+✅ migrations/                      database migrations
+✅ queries/                         query YAML
+✅ tasks/                           task board
+✅ tests/                           integration tests
+✅ templates/                       Jinja2 for wrangler
+
+❌ /target/                         build artifacts
+❌ vault/                           separate repo
+```
+
+### Vault tracks:
+```
+✅ README.md                        workflow guide
+✅ notes/                           human annotations
+✅ poimen/                          generated memories
+✅ poimen/evidence/                 generated chunks
+✅ skills/                          generated + promoted
+✅ .gitignore                       Obsidian metadata
+
+❌ .obsidian/                       local settings
+```
+
+## Current status
+
+| Aspect | Status | Details |
+|--------|--------|---------|
+| Parent repo remote | ✅ configured | poimen-memory |
+| Vault repo remote | ✅ configured | poimen-obesdient-memory |
+| Parent ignores vault/ | ✅ yes | In .gitignore |
+| Vault has own .git | ✅ yes | Separate history |
+| Vault synced to remote | ✅ yes | 2 commits pushed |
+| No historic bloat | ✅ yes | Clean slate |
+
+## Next steps
+
+1. **M3 implementation**: First ingest → rebuild cycle (generates real memory files)
+2. **Verify byte-identical**: `git -C vault diff --exit-code` after rebuild
+3. **Live reading**: Clone vault separately, open in Obsidian
+4. **Human annotations**: Add notes in `vault/notes/`, commit to vault repo