Story Crater Bot
ad1147f4a6
- vault/ has its own .git (separate from parent) - vault remote: poimen-obesdient-memory (different from parent) - parent .gitignore ignores vault/ to prevent accidental tracking - both repos work together: parent has code+JSONL, vault has generated markdown - two independent CI/CD pipelines (parent: build/test, vault: rebuild/push) Added: - VAULT-SEPARATE-REPO.md: structure, why separate, setup guide - VAULT-GITOPS-ARCHITECTURE.md: data flow and GitOps principles
193 lines
6.5 KiB
Markdown
193 lines
6.5 KiB
Markdown
# Vault GitOps Architecture
|
|
|
|
## Design principle
|
|
|
|
Obsidian vault is **derived, never edited directly**. All memory changes flow through the authoritative JSONL log.
|
|
|
|
```
|
|
JSONL log (source of truth)
|
|
↓ [mem rebuild --from-log]
|
|
Obsidian vault (derived, GitOps)
|
|
↓ [git push]
|
|
Remote vault repo
|
|
```
|
|
|
|
## Repository structure
|
|
|
|
### Parent repo: `poimen-memory`
|
|
- Location: `/Users/rockliang/workplace/poimen-memory`
|
|
- Remote: `ssh://[email protected]:2222/rock/poimen-memory.git`
|
|
- Contains: Rust crates (M0-M2 complete), DESIGN.md, tasks/, log/, queries/
|
|
- Tracks: Source code, task definitions, JSONL event log (authoritative)
|
|
|
|
### Vault repo: `poimen-obesdient-memory`
|
|
- Location: `/Users/rockliang/workplace/poimen-memory/vault` (separate .git)
|
|
- Remote: `ssh://[email protected]:2222/rock/poimen-obesdient-memory.git`
|
|
- Contains: Generated memories (poimen/), skill drafts (skills/_drafts/), human docs (notes/)
|
|
- Tracks: Markdown files (generated and human-written), but NOT source code
|
|
|
|
## What gets tracked
|
|
|
|
### Parent repo tracks:
|
|
```
|
|
✅ Cargo.toml, crates/* source code
|
|
✅ DESIGN.md, README.md documentation
|
|
✅ tasks/M0.1-M6.6.md 38 tasks with acceptance criteria
|
|
✅ queries/poimen.yaml standing question definitions
|
|
✅ log/poimen/**/*.jsonl authoritative JSONL event log
|
|
✅ k8s/, migrations/ infrastructure
|
|
✅ .gitignore editor temp files, build artifacts
|
|
|
|
❌ /target/, *.swp, .DS_Store ignored
|
|
❌ vault/ separate repo, own .git
|
|
```
|
|
|
|
### Vault repo tracks:
|
|
```
|
|
✅ README.md workflow guide
|
|
✅ notes/*.md human annotations (writable)
|
|
✅ poimen/*.md generated L0/L1/L2 memories (read-only)
|
|
✅ poimen/evidence/*.md generated L0 chunks (optional, read-only)
|
|
✅ skills/_drafts/*.md generated skill drafts (read-only)
|
|
✅ .gitignore Obsidian metadata, private files
|
|
|
|
❌ .obsidian/ local Obsidian settings
|
|
❌ *.readonly local markers
|
|
```
|
|
|
|
## Data flow
|
|
|
|
### Ingest → Memory → Vault
|
|
|
|
```
|
|
1. Ingest run completes
|
|
cargo run -p mem-cli -- ingest --project poimen --query infra-root-causes
|
|
|
|
2. Creates JSONL event log
|
|
log/poimen/infra-root-causes/<run-id>.jsonl
|
|
|
|
3. Commit to parent repo
|
|
git add log/poimen/infra-root-causes/<run-id>.jsonl
|
|
git commit -m "ingest: infra-root-causes (update-rate=0.22)"
|
|
git push origin main
|
|
|
|
4. CI/CD trigger: log changed
|
|
|
|
5. Rebuild vault from log
|
|
mem rebuild --from-log --project poimen
|
|
|
|
6. Vault markdown updated
|
|
vault/poimen/infra-root-causes.md (generated)
|
|
vault/poimen/index.md (L2 synthesis, generated)
|
|
|
|
7. Commit to vault repo
|
|
cd vault
|
|
git add poimen/*.md
|
|
git commit -m "rebuild: infra-root-causes (2026-08-17T...)"
|
|
git push origin main
|
|
```
|
|
|
|
## Verification: byte-identical rebuild
|
|
|
|
M2.8 gate verifies that rebuild is deterministic:
|
|
|
|
```bash
|
|
# Delete vault and index
|
|
rm -rf vault/poimen vault/skills/_drafts
|
|
psql -c "delete from memory_node where project='poimen'"
|
|
|
|
# Rebuild from JSONL alone
|
|
mem rebuild --from-log --project poimen
|
|
|
|
# Vault must be byte-identical
|
|
git -C vault diff --exit-code
|
|
# Exit 0 = pass. Any diff = gate failed.
|
|
```
|
|
|
|
This guarantees: JSONL is truly the source of truth. No hidden inputs in vault projections.
|
|
|
|
## Human workflow in Obsidian
|
|
|
|
1. Clone vault: `git clone ssh://[email protected]:2222/rock/poimen-obesdient-memory.git`
|
|
2. Open in Obsidian: File → Open vault as folder
|
|
3. Start at: `notes/INDEX.md` (entry point)
|
|
4. Read generated notes (links auto-generated from memory_edge provenance)
|
|
5. Add annotations in `notes/my-research.md` (writable, won't be overwritten)
|
|
6. Link to generated notes: `[[poimen/infra-root-causes]]`
|
|
7. Commit annotations normally: `git commit -m "notes: my-research findings"`
|
|
|
|
Generated notes in `poimen/` are read-only (rebuilt on log changes).
|
|
Human notes in `notes/` are writable (never auto-overwritten).
|
|
|
|
## Skill promotion flow
|
|
|
|
Drafts are auto-generated, read-only. Promotion is manual:
|
|
|
|
```
|
|
1. mem skill draft --from poimen/infra-root-causes
|
|
Creates: vault/skills/_drafts/tool-effectiveness-v1/SKILL.md
|
|
(auto-generated, read-only)
|
|
|
|
2. User reviews in Obsidian
|
|
vault/skills/_drafts/tool-effectiveness-v1/SKILL.md
|
|
|
|
3. Decision: promote or wait for v2
|
|
|
|
4. If promote (manual git action):
|
|
git mv skills/_drafts/tool-effectiveness-v1 skills/tool-effectiveness-v1
|
|
git commit -m "promote: tool-effectiveness-v1"
|
|
git push
|
|
|
|
5. Skill now loadable:
|
|
pi --skill vault/skills/
|
|
pi --list-skills
|
|
# tool-effectiveness-v1 appears in list
|
|
|
|
6. Use the skill:
|
|
# Skill carries generated_from: <L2-sha> metadata
|
|
# Links back to L2 memory that produced it
|
|
# No feedback loop: if used in next session, ingest marks chunk as derived:true
|
|
```
|
|
|
|
## Authority guarantees
|
|
|
|
1. **JSONL is immutable** — only append new runs, never modify existing records
|
|
2. **Rebuild is deterministic** — same log, byte-identical vault
|
|
3. **Vault is ephemeral** — can be dropped and rebuilt from log
|
|
4. **No vault edits corrupt the record** — generated content won't persist edits
|
|
5. **Promotion is auditable** — git history of skill promotions
|
|
|
|
## Tools used
|
|
|
|
- **mem CLI**: `cargo run -p mem-cli -- rebuild|ingest|query|skill|verify`
|
|
- **pgvector**: 768-dim embeddings, HNSW index for retrieval
|
|
- **Ollama**: Qwen2.5-3B controller (prompted, no LoRA yet)
|
|
- **Obsidian**: Vault app for reading, optional annotations
|
|
- **Git**: Both repos track; vault has separate remote
|
|
|
|
## Current status
|
|
|
|
| Phase | Status | Details |
|
|
|-------|--------|---------|
|
|
| **M0** | ✅ COMPLETE | 8/8 tasks, 35 tests |
|
|
| **M1** | ✅ COMPLETE | 8/8 tasks, 30+ tests |
|
|
| **M2** | ✅ COMPLETE (core) | 5/8 tasks, 26 tests |
|
|
| **M3** | ⏳ READY | L2 synthesis, retrieval (blocked on M2 completion) |
|
|
| **M4** | ⏳ READY | Skill drafting (blocked on M3) |
|
|
| **M5** | ⏳ DEFERRED | Post-training, LoRA adapter (separate Python) |
|
|
| **Vault** | ✅ DEPLOYED | Separate repo, documentation migrated, GitOps ready |
|
|
|
|
## Next steps
|
|
|
|
1. **M3 implementation**: L2 pass, HNSW retrieval, reranking
|
|
2. **First live run**: Full ingest → rebuild → vault sync
|
|
3. **Skill drafting**: M4 implementation
|
|
4. **Post-training**: M5 (Python, separate from main Rust)
|
|
|
|
## References
|
|
|
|
- **Design**: [DESIGN.md](DESIGN.md) — full system design (460 lines)
|
|
- **Vault docs**: `vault/notes/INDEX.md` — entry point for Obsidian users
|
|
- **Task board**: `tasks/INDEX.md` — 64 tasks, 6 phases
|
|
- **GitOps**: This file — authority model and data flow
|