feat: POST /memory/learn endpoint + refactor mem learn CLI

Learning flow now goes through the service, not local JSONL:
- POST /memory/learn: accepts markdown, chunks it, runs gated loop
  (LLM evaluates + compacts), stores in pgvector. OpenAI-style API.
- mem learn CLI: reads files, calls POST /memory/learn per file
- Removed cmd_compact (gated loop IS the compaction)
- Updated README with new commands and API docs

Memory never grows unbounded — every update is a rewrite, not append.
The gated loop LLM acts as evaluator + compactor in one pass.
This commit is contained in:
2026-08-30 13:21:07 -07:00
parent 92458e643c
commit ae1a2ef9a2
3 changed files with 328 additions and 287 deletions
+63 -8
View File
@@ -4,8 +4,8 @@ Gated recurrent memory over agent context. Reads session history chunk-by-chunk,
keeps only what answers standing questions, projects result into an Obsidian
vault and a pgvector index.
**Status: design complete, no code yet.** 37 tasks in [memory-tasks/](memory-tasks/INDEX.md),
0 done. Start at [M0.1](memory-tasks/M0.1-cargo-workspace.md).
**Status: 78/78 tasks complete, all 13 phases done.** Production-deployed on Kubernetes
via ArgoCD. See [CLAUDE.md](CLAUDE.md) for full API reference.
## Problem
@@ -171,16 +171,71 @@ with an EOF; telemetry or a live tail will not have one. `RecordSource` returns
## Commands
```sh
mem ingest --project poimen --dry-run # chunk plan, zero model calls
# Ingest knowledge via gated loop (LLM evaluates + compacts automatically)
mem learn knowledge/rust.md # single file
mem learn knowledge/ --project myproject # directory
mem learn knowledge/ --dry-run # preview chunks
mem learn knowledge/ --memory-budget 8192 # larger memory window
mem learn knowledge/ --model ornith:35b # use stronger model
# Traditional ingest (from session transcripts)
mem ingest --project poimen --dry-run
mem ingest --project poimen --query infra-root-causes
mem synthesize --project poimen # L2 pass, exit gate on
mem rebuild --from-log --project poimen # drop and rebuild projections
mem verify --project poimen # provenance graph closure
mem query "why did requests over 10KB fail?"
# Failure capture + lesson derivation
mem capture --cmd "cargo build" --exit 1 --output-file error.log
mem sig --tool cargo --file error.log # extract failure signature
mem resolve --json # pair failure with fix
mem lookup --tool cargo --file error.log # search known fixes
# Skills + projections
mem skill draft --from poimen/infra-root-causes
mem label --project poimen # evidence labels for training
mem materialize # generate SKILL.md files
mem verify --project poimen # provenance graph closure
# Server
mem serve --port 8080
```
## HTTP API
```sh
# Health
curl http://localhost:8080/health
# Learn — gated loop ingest (LLM evaluates + compacts)
curl -X POST http://localhost:8080/memory/learn \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"project": "knowledge", "text": "## Rust\n- ownership...", "query": "key patterns?"}'
# Returns: {chunks_seen, chunks_used, memory: "compacted...", stored: true}
# Ingest — queue-based async ingest
curl -X POST http://localhost:8080/memory/ingest ...
# Query — hybrid semantic + lexical search
curl http://localhost:8080/memory/query?project=poimen&q=port+conflict
# Context — three-tier retrieval (signature > vector > reference)
curl -X POST http://localhost:8080/memory/context \
-d '{"project": "poimen", "tool": "cargo", "task": "build", "budget": 4096}'
```
### Learning Flow
```
Agent/CLI → POST /memory/learn → chunk markdown → gated loop:
For each chunk:
LLM evaluates: does this add new knowledge? (update gate)
If yes → LLM rewrites memory incorporating new fact (compaction)
If no → chunk rejected, memory unchanged
→ Store compacted memory in pgvector (embedded, searchable)
→ Return {chunks_seen, chunks_used, memory, stored}
```
Memory never grows unbounded — every update is a rewrite, not an append.
The LLM acts as both evaluator and compactor in one pass.
## M3.8 Pluggable Query Optimization
**Purpose**: Compress and optimize search results before passing them to the LLM context window, improving token efficiency and response quality.