rock/poimen-memory
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: complete M0.1-M0.4 phases

Browse Source 
M0.1 - Cargo workspace + crate skeletons
  - 6-crate workspace with correct dependency direction
  - CI/CD pipeline with GitHub Actions
  - Integration tests verifying build and dependency structure

M0.2 - Domain types and sha256 identity
  - Level (L0, L1, L2) enum with proper serde formatting
  - Role enum (User, Assistant, ToolResult, System)
  - Record, Chunk, and MemoryNode domain types
  - Content-hash identity system ensuring rebuild idempotence
  - Newtypes (ProjectId, QueryId, RunId) with validation
  - Round-trip serde tests for all types

M0.3 - RecordSource trait + ChunkPolicy
  - RecordSource trait for streaming record sources
  - Chunk policy with token budgets and boundary modes
  - TokenCounter trait with CharsOverFourCounter stub
  - Chunking stream that respects budgets without splitting records
  - VecSource for testing
  - Integration tests verifying lossless chunking and budget adherence

M0.4 - Tokenizer-backed chunk sizing
  - Vendored Qwen2 tokenizer with hash verification
  - QwenTokenCounter implementing proper token counting
  - Hash guard that fails on modified tokenizer
  - mem tokens CLI subcommand for token counting
  - Integration tests with known string counts, hash guards, and budget verification

Total: 19 integration tests passing, all phases verified to compose correctly
Workspace builds cleanly with no clippy warnings
This commit is contained in:
Story Crater Bot
2026-08-22 23:13:42 -07:00
parent a163c03619
commit 33b7150f56
27 changed files with 2216 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
DESIGN.md
+191 -1
View File
@@ -56,6 +56,33 @@ Memory is **levelled**, and every event in the log carries its level. The paper
## Architecture
```
api.riotpiao.com (Kong)
│
┌─────────────────┼─────────────────┐
│ │ │
/ingest /query /skills
(async) (sync) (read-only)
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────────────────────┐
│ API Server (Rust httpd) mem-store / mem-llm │
│ - ingest_id dedup + queue │
│ - query → HNSW + rerank + edge-walk │
│ - skill catalog (excludes _drafts) │
└──────────────────┬──────────────────────────────────┘
│
pi/claude CLI ─────┼────── agents in-session
local or CI/CD │ (embedded queries)
│
┌──────────────────┴──────────────────┐
│ │
▼ ▼
[ Ingest Queue ] [ CNPG Cluster ]
(redis or local) (pgvector, HNSW)
│ │
├─────────────────────────────────────┤
│
▼
pi sessions / claude transcripts / loop.sh artifacts
│
▼ project resolver (cwd -> project id)
@@ -86,6 +113,8 @@ pi sessions / claude transcripts / loop.sh artifacts
└──────────────────────────────────────────────────────┘
```
**Authority invariant unchanged:** API is stateless demultiplexer. JSONL log is authoritative; vault and pgvector projections are droppable. API caches read-only state (embeddings, L2 synthesis); ingest writes only to log.
**Authority model — the load-bearing decision.** The JSONL log is the only source of truth. The vault and the vector index are projections that must be droppable and rebuildable byte-identically from the log. This is poimen's own §1 principle ("nothing derived is authoritative; if it cannot be dropped and rebuilt, it has hidden inputs and that is a bug") applied here, and it buys three things: re-embedding after a model change is a rebuild not a migration, Obsidian edits cannot corrupt the record, and the post-training corpus is the log itself.
## Standing queries
@@ -423,6 +452,19 @@ Note `agent-rust/.gitignore` excludes `tasks`, which silently untracks the whole
| M4.2 | `derived: true` ingest filter — emitted artifacts excluded from evidence | M | M4.1, M0.5 |
| M4.3 | **M4 gate** — draft absent from `--list-skills`; no L0 node matches an emitted artifact | M | gate |
**M3.5 — Distributed API Layer** (Homelab Frontend integration)
| id | task | size | deps |
|---|---|---|---|
| M3.5.1 | HTTP server + router (actix-web or axum), Kong auth hook, request metrics | M | M0.1 |
| M3.5.2 | `POST /ingest` endpoint — `ingest_id` dedup, async queue (redis or in-mem), job polling | M | M1.7, M3.5.1 |
| M3.5.3 | `GET /query` endpoint — embed query, HNSW recall by level, rerank, walk edges to L0 | M | M3.3, M3.5.1 |
| M3.5.4 | Federation: single query across projects, fan+merge results, deduplicate | M | M3.5.3 |
| M3.5.5 | `GET /skills` and `/skills/{name}` — loadable skills only, exclude _drafts, YAML frontmatter in JSON | M | M4.1, M3.5.1 |
| M3.5.6 | `GET /projects` and `/projects/{id}/status` — metadata, metrics, synthesis timestamps | S | M3.5.1 |
| M3.5.7 | Rate limiting (apikey-scoped per endpoint) + idempotency by sha256 | M | M3.5.2 |
| M3.5.8 | **M3.5 gate** — end-to-end ingest→query via HTTP, load from cli and from agent simul | M | gate |
**M5 — Post-training** (Python, separate from the Rust workspace; boundary is the JSONL)
| id | task | size | deps |
@@ -434,7 +476,7 @@ Note `agent-rust/.gitignore` excludes `tasks`, which silently untracks the whole
| M5.5 | verl loop — `r_update` ±1, `r_exit` {0,−0.5,−0.75}, strict `r_format`, α=0.9 | L | M5.3, M5.4 |
| M5.6 | **M5 gate** — adapter beats prompted baseline on held-out update accuracy | L | gate |
Total 38 tasks, 6 gates. M0 and M2.2 have no model dependency and can start immediately; M5.4 is homelab work independent of everything else in M5 and can run in parallel.
Total 43 tasks, 7 gates. M0 and M2.2 have no model dependency and can start immediately; M5.4 is homelab work independent of everything else in M5 and can run in parallel. M3.5 depends on M2 (pgvector store exists) and M1 (ingest loop exists); can run in parallel with M4 and M5.
## Verification
@@ -461,21 +503,167 @@ cargo run -p mem-cli -- rebuild --from-log --project poimen
git -C vault diff --exit-code # empty: rebuild is byte-identical
psql -c "select level, count(*) from memory_node group by level;"
# P3.5 — API server online
cargo run -p mem-cli -- serve --port 8080 &
sleep 1
curl -H "apikey: test-key" http://localhost:8080/memory/projects
# expect: ["poimen", ...]
curl -H "apikey: test-key" \
"http://localhost:8080/memory/query?query=kong+body&level=L1,L2&project=poimen&limit=3"
# expect: 200, array of memory nodes with score + parents
#
# ingest via HTTP (async):
jq -n '{project:"poimen", source:"test:local", records:[...]}' | \
curl -X POST -H "apikey: test-key" \
http://localhost:8080/memory/ingest -d @-
# expect: 202, {"job_id": "ingest-<uuid>", "status_url": "/memory/ingest/ingest-<uuid>"}
#
# idempotency: same request twice with same ingest_id returns same job_id, no re-enqueue
# rate limit: 11th req in 1 second gets 429 Retry-After
# auth missing: 401 Unauthorized
# P4 — synthesis and retrieval
cargo run -p mem-cli -- synthesize --project poimen # expect exit gate to fire
cargo run -p mem-cli -- query "why did requests over 10KB fail?"
# expect: infra-root-causes L1 node, Kong body-buffer passage, L0 citation
#
# Via API (same result):
curl -H "apikey: test-key" \
"http://localhost:8080/memory/query?query=why+did+requests+over+10KB+fail"
# expect: identical results
# P5 — skill drafts land unloadable, and the cycle stays open
cargo run -p mem-cli -- skill draft --from poimen/infra-root-causes
ls vault/skills/_drafts/ # draft here, NOT in vault/skills/
pi --skill vault/skills/ --list-skills # draft must not appear
curl -H "apikey: test-key" http://localhost:8080/memory/skills
# expect: no drafts in list
cargo run -p mem-cli -- verify --derived-filter --project poimen
# asserts: no L0 evidence node text matches an emitted skill artifact
```
The decisive P2 metric is **update-rate**, the one number distinguishing a working gate from an expensive summarizer. toolResults are 43% of records and mostly evidence-free, so a correct gate rejects the large majority of chunks.
**P3.5 API gate:** Ingest and query work over HTTP with correct idempotency, auth, and rate limiting. CLI and agents both submit to same endpoint; no duplication or ordering issues.
## Distributed API Layer (Homelab Frontend)
**Gateway:** `api.riotpiao.com` routes agent and system memory requests through Kong.
**Architecture assumption:** Memory services run in CNPG cluster; API layer is HTTP facade exposing read/write workflows to distributed agents. Authority remains JSONL—API is a request demultiplexer, not a cache or alternative source of truth.
### REST API Endpoints
```
POST /memory/ingest <- async, idempotent by sha256
GET /memory/query <- semantic search + rerank
GET /memory/projects <- list projects with L2 synthesis
GET /memory/projects/{id}/status <- ingest/synthesis status
GET /memory/projects/{id}/notes <- L1/L2 notes (Obsidian export)
GET /memory/skills <- loadable skills (excludes _drafts)
GET /memory/skills/{name} <- one skill frontmatter + body
```
**Request/Response contract:**
```jsonl
# POST /memory/ingest (idempotent, async)
{"project": "poimen", "source": "agent:uuid", "records": [...], "ingest_id": "sha256-of-batch"}
→ 202 Accepted
{"job_id": "ingest-<uuid>", "ingest_id": "...", "status_url": "/memory/ingest/ingest-<uuid>"}
# GET /memory/query (semantic search)
{"query": "why did requests over 10KB fail?", "level": ["L1", "L2"], "project": "poimen", "limit": 5}
→ 200 OK
[
{"level": "L1", "sha256": "...", "text": "...", "score": 0.92,
"parents": [{"level": "L0", "source": "pi:...", "text": "..."}]},
...
]
# GET /memory/skills?loadable=true
→ 200 OK
[
{"name": "infra-root-causes", "description": "...", "when_to_use": "...",
"generated_from": null, "promoted_at": "2026-08-20"}
]
```
### Distributed Behavior
**Ingestion:** `mem-ingest` CLI submits batches to `POST /memory/ingest` via `ingest_id` (sha256 of batch text). Duplicate `ingest_id` returns same `job_id` without re-enqueuing — jobs are idempotent by content hash, not request. Server stores the mapping; HTTP 409 means already ingested (user caller resubmits without retry).
**Query federation:** Agents query single endpoint; server fans requests to appropriate project (selected by metadata or query text). Results walk `memory_edge` down to L0 *server-side*, so client gets complete citation graph in one round-trip.
**Skills as cargo:** `GET /memory/skills` returns YAML frontmatter in JSON so agent UIs can inspect `description` and `when_to_use` without fetching the file. Body is optional (fetch separately if needed to load).
**Status & observability:**
- `GET /memory/projects/{id}/status` → `{"last_ingest": "...", "chunks_total": N, "chunks_used": M, "synthesis_ran": "...", "next_synthesis_at": "..."}`
- Metrics: ingest latency (p50/p99), query latency, update-rate per project, memory size trends
### Scaling Constraints
**Single points of failure:**
- CNPG cluster (mitigated by ≥3 replicas + Longhorn)
- Ollama inference (separate from memory store; ingest is offline, query caches embeddings)
**Throughput limits:**
- Ingest: one gated loop per project sequentially (5000 tokens/chunk, gate latency 812ms); ~7 chunks/min = 35k tokens/min per project
- Query: HNSW recall is O(log n), rerank O(k log k), each << embedding roundtrip to Ollama (typically 200ms)
**Caching strategy:**
- Memory nodes are immutable (sha256 content hash) — safe to cache indefinitely post-write
- L2 synthesis is project-scoped and regenerated on `mem synthesize` — TTL 1h or explicit purge
- Embeddings cached per-query hash (same embedding twice = cache hit, save 200ms Ollama call)
- Client-side: `ETag: <sha256>` on all read endpoints, no conditional logic server-side (it's stateless)
**Auth & rate limits:**
- Kong `apikey:` header (existing pattern)
- Per-key limits: ingest 100 jobs/hour, query 1000 req/hour, skill fetch unlimited
- Burst allowance: 10 req/sec per key (ingest waits in queue; query returns 429 Retry-After if burst exceeded)
### Integration with Existing Flows
**From `mem-cli` (local or CI/CD):**
```bash
mem ingest --project poimen --query infra-root-causes --gateway https://api.riotpiao.com
```
Client computes `ingest_id` locally (sha256 of all records), submits as batch, polls `/memory/ingest/<job_id>` until done.
**From agents (in-session via Pi or Claude):**
```bash
# Query within agent:
curl -H "apikey: $MEM_APIKEY" \
"https://api.riotpiao.com/memory/query?query=why+did+X+fail&project=poimen&level=L1,L2"
# Ingest at session end:
{session_transcript_chunk} | curl -X POST -H "apikey: $MEM_APIKEY" \
https://api.riotpiao.com/memory/ingest \
-d @- -H "Content-Type: application/jsonl"
```
**Skill loading in agent systems:**
```bash
# Discovery:
curl -H "apikey: $MEM_APIKEY" https://api.riotpiao.com/memory/skills?loadable=true \
| jq -r '.[] | .name' | xargs -I {} \
curl https://api.riotpiao.com/memory/skills/{} > ~/.claude/skills/{}/SKILL.md
```
### Error Taxonomy
```
200 OK — query succeeded, memory node found (or empty result)
202 Accepted — ingest accepted, job queued
204 No Content — query matched no nodes; not an error
400 Bad Request — malformed query or invalid project/level
401 Unauthorized — missing/invalid apikey
409 Conflict — ingest_id already processed (idempotent, safe retry)
429 Too Many Requests — rate limit exceeded, Retry-After header set
500 Internal Server Error — CNPG offline or embedding service down
503 Service Unavailable — gated loop busy (queue building), retry in 5s
```
## Risks
- **3B gate quality unmeasured on this corpus.** The paper evaluates on QA benchmarks with clean evidence labels; agent transcripts are messier. Mitigation: P2's update-rate is a cheap early read, and the 32B `reasoning` model can spot-audit a sample before committing to P5.
@@ -484,3 +672,5 @@ The decisive P2 metric is **update-rate**, the one number distinguishing a worki
- **No ground-truth evidence labels.** `r_update` needs them. Distant supervision from the 32B labeler inherits its bias; hold out a hand-labelled set to measure agreement before trusting it.
- **Vault/log divergence.** Hand edits are overwritten on rebuild. Either make the vault read-only or add an `## Notes` region the projector preserves. Decide before anyone starts editing.
- **Ollama has no LoRA path.** P5 forces the vLLM decision. Do not discover this at P5.
- **API latency at scale.** Query federation fans requests to multiple projects; slowest project wins. Mitigation: query timeout 5s, client-side fallback to local JSONL search, async synthesis keeps L2 warm (cache hit 95%+).
- **Ingest race on concurrent writes.** Two agents submit overlapping session chunks to same project simultaneously. Mitigation: `ingest_id` based on content hash prevents duplicate evidence in log; gated loop is single-threaded per project, queues serialize. Allowed cost: cold-start ingest delay ~5m for backlog.