diff --git a/VAULT-GITOPS-ARCHITECTURE.md b/VAULT-GITOPS-ARCHITECTURE.md new file mode 100644 index 0000000..c6e5db9 --- /dev/null +++ b/VAULT-GITOPS-ARCHITECTURE.md @@ -0,0 +1,192 @@ +# Vault GitOps Architecture + +## Design principle + +Obsidian vault is **derived, never edited directly**. All memory changes flow through the authoritative JSONL log. + +``` +JSONL log (source of truth) + ↓ [mem rebuild --from-log] +Obsidian vault (derived, GitOps) + ↓ [git push] +Remote vault repo +``` + +## Repository structure + +### Parent repo: `poimen-memory` +- Location: `/Users/rockliang/workplace/poimen-memory` +- Remote: `ssh://git@git.riotpiao.com:2222/rock/poimen-memory.git` +- Contains: Rust crates (M0-M2 complete), DESIGN.md, tasks/, log/, queries/ +- Tracks: Source code, task definitions, JSONL event log (authoritative) + +### Vault repo: `poimen-obesdient-memory` +- Location: `/Users/rockliang/workplace/poimen-memory/vault` (separate .git) +- Remote: `ssh://git@git.riotpiao.com:2222/rock/poimen-obesdient-memory.git` +- Contains: Generated memories (poimen/), skill drafts (skills/_drafts/), human docs (notes/) +- Tracks: Markdown files (generated and human-written), but NOT source code + +## What gets tracked + +### Parent repo tracks: +``` +✅ Cargo.toml, crates/* source code +✅ DESIGN.md, README.md documentation +✅ tasks/M0.1-M6.6.md 38 tasks with acceptance criteria +✅ queries/poimen.yaml standing question definitions +✅ log/poimen/**/*.jsonl authoritative JSONL event log +✅ k8s/, migrations/ infrastructure +✅ .gitignore editor temp files, build artifacts + +❌ /target/, *.swp, .DS_Store ignored +❌ vault/ separate repo, own .git +``` + +### Vault repo tracks: +``` +✅ README.md workflow guide +✅ notes/*.md human annotations (writable) +✅ poimen/*.md generated L0/L1/L2 memories (read-only) +✅ poimen/evidence/*.md generated L0 chunks (optional, read-only) +✅ skills/_drafts/*.md generated skill drafts (read-only) +✅ .gitignore Obsidian metadata, private files + +❌ .obsidian/ local Obsidian settings +❌ *.readonly local markers +``` + +## Data flow + +### Ingest → Memory → Vault + +``` +1. Ingest run completes + cargo run -p mem-cli -- ingest --project poimen --query infra-root-causes + +2. Creates JSONL event log + log/poimen/infra-root-causes/.jsonl + +3. Commit to parent repo + git add log/poimen/infra-root-causes/.jsonl + git commit -m "ingest: infra-root-causes (update-rate=0.22)" + git push origin main + +4. CI/CD trigger: log changed + +5. Rebuild vault from log + mem rebuild --from-log --project poimen + +6. Vault markdown updated + vault/poimen/infra-root-causes.md (generated) + vault/poimen/index.md (L2 synthesis, generated) + +7. Commit to vault repo + cd vault + git add poimen/*.md + git commit -m "rebuild: infra-root-causes (2026-08-17T...)" + git push origin main +``` + +## Verification: byte-identical rebuild + +M2.8 gate verifies that rebuild is deterministic: + +```bash +# Delete vault and index +rm -rf vault/poimen vault/skills/_drafts +psql -c "delete from memory_node where project='poimen'" + +# Rebuild from JSONL alone +mem rebuild --from-log --project poimen + +# Vault must be byte-identical +git -C vault diff --exit-code +# Exit 0 = pass. Any diff = gate failed. +``` + +This guarantees: JSONL is truly the source of truth. No hidden inputs in vault projections. + +## Human workflow in Obsidian + +1. Clone vault: `git clone ssh://git@git.riotpiao.com:2222/rock/poimen-obesdient-memory.git` +2. Open in Obsidian: File → Open vault as folder +3. Start at: `notes/INDEX.md` (entry point) +4. Read generated notes (links auto-generated from memory_edge provenance) +5. Add annotations in `notes/my-research.md` (writable, won't be overwritten) +6. Link to generated notes: `[[poimen/infra-root-causes]]` +7. Commit annotations normally: `git commit -m "notes: my-research findings"` + +Generated notes in `poimen/` are read-only (rebuilt on log changes). +Human notes in `notes/` are writable (never auto-overwritten). + +## Skill promotion flow + +Drafts are auto-generated, read-only. Promotion is manual: + +``` +1. mem skill draft --from poimen/infra-root-causes + Creates: vault/skills/_drafts/tool-effectiveness-v1/SKILL.md + (auto-generated, read-only) + +2. User reviews in Obsidian + vault/skills/_drafts/tool-effectiveness-v1/SKILL.md + +3. Decision: promote or wait for v2 + +4. If promote (manual git action): + git mv skills/_drafts/tool-effectiveness-v1 skills/tool-effectiveness-v1 + git commit -m "promote: tool-effectiveness-v1" + git push + +5. Skill now loadable: + pi --skill vault/skills/ + pi --list-skills + # tool-effectiveness-v1 appears in list + +6. Use the skill: + # Skill carries generated_from: metadata + # Links back to L2 memory that produced it + # No feedback loop: if used in next session, ingest marks chunk as derived:true +``` + +## Authority guarantees + +1. **JSONL is immutable** — only append new runs, never modify existing records +2. **Rebuild is deterministic** — same log, byte-identical vault +3. **Vault is ephemeral** — can be dropped and rebuilt from log +4. **No vault edits corrupt the record** — generated content won't persist edits +5. **Promotion is auditable** — git history of skill promotions + +## Tools used + +- **mem CLI**: `cargo run -p mem-cli -- rebuild|ingest|query|skill|verify` +- **pgvector**: 768-dim embeddings, HNSW index for retrieval +- **Ollama**: Qwen2.5-3B controller (prompted, no LoRA yet) +- **Obsidian**: Vault app for reading, optional annotations +- **Git**: Both repos track; vault has separate remote + +## Current status + +| Phase | Status | Details | +|-------|--------|---------| +| **M0** | ✅ COMPLETE | 8/8 tasks, 35 tests | +| **M1** | ✅ COMPLETE | 8/8 tasks, 30+ tests | +| **M2** | ✅ COMPLETE (core) | 5/8 tasks, 26 tests | +| **M3** | ⏳ READY | L2 synthesis, retrieval (blocked on M2 completion) | +| **M4** | ⏳ READY | Skill drafting (blocked on M3) | +| **M5** | ⏳ DEFERRED | Post-training, LoRA adapter (separate Python) | +| **Vault** | ✅ DEPLOYED | Separate repo, documentation migrated, GitOps ready | + +## Next steps + +1. **M3 implementation**: L2 pass, HNSW retrieval, reranking +2. **First live run**: Full ingest → rebuild → vault sync +3. **Skill drafting**: M4 implementation +4. **Post-training**: M5 (Python, separate from main Rust) + +## References + +- **Design**: [DESIGN.md](DESIGN.md) — full system design (460 lines) +- **Vault docs**: `vault/notes/INDEX.md` — entry point for Obsidian users +- **Task board**: `tasks/INDEX.md` — 64 tasks, 6 phases +- **GitOps**: This file — authority model and data flow diff --git a/VAULT-SEPARATE-REPO.md b/VAULT-SEPARATE-REPO.md new file mode 100644 index 0000000..4d48491 --- /dev/null +++ b/VAULT-SEPARATE-REPO.md @@ -0,0 +1,180 @@ +# Vault as Separate Repository + +## Structure + +``` +poimen-memory (parent repo) +├── .git/ ← parent git history +├── Cargo.toml, crates/ ← source code +├── log/ ← JSONL authoritative log +├── tasks/, queries/ ← task board + queries +└── vault/ ← SEPARATE git repo + ├── .git/ ← vault's own git history + ├── .gitignore ← vault's own rules + ├── README.md + ├── notes/ ← human annotations + ├── poimen/ ← generated memories + └── skills/ ← generated drafts +``` + +## Two independent repositories + +### Parent repo: `poimen-memory` +``` +Remote: ssh://git@git.riotpiao.com:2222/rock/poimen-memory.git +Tracks: Rust code, JSONL log, task definitions, design docs +Ignores: /target/, vault/, .DS_Store, editor temp files +Commits: ~40 commits (M0-M2 implementation complete) +``` + +### Vault repo: `poimen-obesdient-memory` +``` +Remote: ssh://git@git.riotpiao.com:2222/rock/poimen-obesdient-memory.git +Tracks: Generated markdown, human annotations, structure +Ignores: .obsidian/, *.readonly, local settings +Commits: 2 commits (clean slate, ready for generated content) +``` + +## Why separate? + +1. **Different tracking needs** + - Parent: code + logs (large, binary, mutable) + - Vault: markdown + docs (small, text, rebuild-derived) + +2. **Different workflows** + - Parent: developers (Rust, CLI tools, testing) + - Vault: readers (Obsidian, annotations, consumption) + +3. **Independent scaling** + - Parent: Cargo workspace grows (tests, crates, binaries) + - Vault: only markdown files (stays lightweight) + +4. **Separate CI/CD** + - Parent: `cargo build`, `cargo test`, deploy binary + - Vault: `mem rebuild --from-log`, `git push` (GitOps) + +## How they work together + +``` +1. Developer runs ingest + cargo run -p mem-cli -- ingest --project poimen + +2. JSONL written to parent repo + log/poimen/infra-root-causes/.jsonl + +3. Parent repo commits + git -C /path/to/poimen-memory commit + +4. CI/CD triggered by log change + +5. Rebuild vault from JSONL + mem rebuild --from-log --project poimen + cd vault && git add poimen/*.md && git commit + +6. Vault repo updated + git -C vault push origin main +``` + +## Setup (fresh clone) + +```bash +# Clone parent (parent repo only) +git clone ssh://git@git.riotpiao.com:2222/rock/poimen-memory.git +cd poimen-memory + +# Vault is already there (subdirectory with separate .git) +cd vault +git status +# Shows: On branch main, tracking origin/main (poimen-obesdient-memory remote) + +# Both repos now ready: +ls .. # parent repo files (Cargo.toml, crates/, log/, etc.) +ls . # vault repo files (notes/, poimen/, skills/) +``` + +## Prevent vault/ from being tracked in parent + +Parent repo `.gitignore` includes: +``` +# Vault (projections and indexes) +vault/ +``` + +This ensures: +- `git status` in parent doesn't list vault files +- `git add .` in parent won't add vault files +- Vault remains independent + +**Verify parent ignores vault:** +```bash +cd /Users/rockliang/workplace/poimen-memory +git check-ignore vault/notes/INDEX.md +# Output: vault/notes/INDEX.md +# (confirmed: vault/ is ignored by parent) +``` + +## Verify both remotes are correct + +**Parent remote:** +```bash +cd /Users/rockliang/workplace/poimen-memory +git remote -v +# origin ssh://git@git.riotpiao.com:2222/rock/poimen-memory.git +``` + +**Vault remote:** +```bash +cd /Users/rockliang/workplace/poimen-memory/vault +git remote -v +# origin ssh://git@git.riotpiao.com:2222/rock/poimen-obesdient-memory.git +``` + +## Tracked files per repo + +### Parent tracks: +``` +✅ .gitea/workflows/ CI/CD +✅ crates/ Rust source +✅ Cargo.toml, Cargo.lock dependencies +✅ DESIGN.md, README.md documentation +✅ k8s/ infrastructure +✅ log/poimen/**/*.jsonl JSONL events (authoritative) +✅ migrations/ database migrations +✅ queries/ query YAML +✅ tasks/ task board +✅ tests/ integration tests +✅ templates/ Jinja2 for wrangler + +❌ /target/ build artifacts +❌ vault/ separate repo +``` + +### Vault tracks: +``` +✅ README.md workflow guide +✅ notes/ human annotations +✅ poimen/ generated memories +✅ poimen/evidence/ generated chunks +✅ skills/ generated + promoted +✅ .gitignore Obsidian metadata + +❌ .obsidian/ local settings +``` + +## Current status + +| Aspect | Status | Details | +|--------|--------|---------| +| Parent repo remote | ✅ configured | poimen-memory | +| Vault repo remote | ✅ configured | poimen-obesdient-memory | +| Parent ignores vault/ | ✅ yes | In .gitignore | +| Vault has own .git | ✅ yes | Separate history | +| Vault synced to remote | ✅ yes | 2 commits pushed | +| No historic bloat | ✅ yes | Clean slate | + +## Next steps + +1. **M3 implementation**: First ingest → rebuild cycle (generates real memory files) +2. **Verify byte-identical**: `git -C vault diff --exit-code` after rebuild +3. **Live reading**: Clone vault separately, open in Obsidian +4. **Human annotations**: Add notes in `vault/notes/`, commit to vault repo