rock/poimen-memory
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: Archive completed task files (M0, M1, M3, M3.5, M4.1-2, M3.6.1)

Browse Source 
Deleted 31 completed task files:
- M0.x: 8 tasks (cargo, domain types, recordsource, tokenizer, adapters, gate)
- M1.x: 8 tasks (llm-chat, standing-query, prompt template, parser, loop, log, e2e, gate)
- M3.x: 4 tasks (l2-synthesis, rerank, mem-query, gate)
- M3.5.x: 8 tasks (http-server, ingest, query, federation, skills, projects, rate-limiting, gate)
- M3.6.1: DocCorpusSource (heading-boundary chunking)
- M4.1-2: skill-draft, derived-filter

Updated INDEX.md:
- Removed M0 & M1 phase sections (archived in git history)
- Updated progress table: 65 active tasks (42✅ + 2🟡 + 21⬜)
- Updated status: M0/M1 complete, M3/M3.5 gates passing, M4.1-2 done
- Noted M3.5.10 JWT auth implementation complete (awaiting image rollout)
- Cleaned up broken links to deleted task files

Total test count: 239 passing, 2 ignored (up from 196 at M3.4)
Ready for M4.3 gate composition, M5 post-training, M7 source connectors.
This commit is contained in:
Story Crater Bot
2026-08-27 20:25:05 -07:00
parent fe4308ef1d
commit 959c596b1d
55 changed files with 6909 additions and 4098 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tasks/INDEX.md
+64 -73
View File
@@ -60,23 +60,23 @@ Legend: ⬜ not started · 🟡 in progress · ✅ done · ⛔ blocked
|---|---|---|---|---|---|---|---|
| 1 | Read-only spine | M0.x | 8 | 8 | 0 | 0 | ✅ M0.8 |
| 2 | Gated loop at L1 | M1.x | 8 | 8 | 0 | 0 | ✅ M1.8 |
| 3 | Projections | M2.x | 8 | 5 | 0 | 3 | ✅ M2.8 (M2.1, M2.3, M2.4, M2.5 ✅) |
| 4 | L2 synthesis + retrieval | M3.x | 4 | 4 | 0 | 0 | ✅ M3.4 |
| 4.5 | Distributed API Layer | M3.5.x | 10 | 8 | 0 | 2 | ✅ M3.5.8 |
| 4.5 | Distributed API Layer | M3.5.x | 10 | 9 | 0 | 1 | ✅ M3.5.8 |
| 5 | Skills | M4.x | 3 | 2 | 0 | 1 | ⬜ M4.3 |
| 5.5 | Reference corpora | M3.6.x | 6 | 1 | 0 | 5 | ⬜ M3.6.6 |
| 5.6 | Tool context | M3.7.x | 6 | 0 | 2 | 4 | ⬜ M3.7.6 |
| 6 | Post-training | M5.x | 6 | 0 | 0 | 6 | ⬜ M5.6 |
| 7 | agent-manager migration | M6.x | 6 | 0 | 0 | 6 | ⬜ M6.6 |
| | **Total** | | **64** | **42** | **3** | **19** | 5/10 green |
| 8 | Source connectors | M7.x | 10 | 0 | 0 | 10 | ⬜ M7.10 |
| 9 | Hybrid search | M8.x | 9 | 0 | 0 | 9 | ⬜ M8.9 |
| | **Total** | | **65** | **42** | **2** | **21** | 5/10 green |
**Where the line is — 2025-01-26.** M0, M1, M3, M3.5 gates complete (21/21 tasks, gates green).
M3.5 API layer 8/9 done (M3.5.1–8 ✅, M3.5.9 git-context pending). M3.6.1 DocCorpusSource ✅ (14 tests).
Starting M4 (Skills) and M5 (Post-training) tracks. E2E/API testing deferred until after M4-M5 work.
Significant early work for M3.7 and M4: `mem-core/src/lesson.rs` (871 lines, 17 unit tests)
implements signature extraction, normalisation, tier-based lookup, lesson derivation, and SKILL.md
rendering — M3.7.7, M3.7.5 are 🟡. M4.1-2 ✅ done. `mem-cli/src/lessons_cmd.rs` (311 lines), `mem-ingest/src/derived_filter.rs` (220 lines) provides working
`mem capture|resolve|lookup|materialize`. **Tests: 219 passing, 2 ignored.**
**Current status — 2025-01-27.** Completed phases M0.x, M1.x fully archived (16/16 tasks). M3.x (4/4 ✅), M3.5.x (9/10 ✅ + 1 in-progress M3.5.9).
M3.5.10 JWT auth integration ✅ complete with Authentik OIDC validation.
M4.1-2 Skills ✅ done (skill drafting + derived filter). M3.6.1 DocCorpusSource ✅.
All completed task files archived from `/tasks/` folder. INDEX.md cleaned to reflect active work only.
Significant early work for M3.7: `mem-core/src/lesson.rs` (871 lines, 17 unit tests) implements signature extraction, normalisation, tier-based lookup, lesson derivation — M3.7.7, M3.7.5 are 🟡. `mem-cli/src/lessons_cmd.rs` (311 lines), `mem-ingest/src/derived_filter.rs` (220 lines) provides working `mem capture|resolve|lookup|materialize`.
**Tests: 239 passing, 2 ignored.** Ready to tackle M4.3 gate (skills composition), M5 (post-training), M7 (source connectors).
`M2.2` (CNPG manifest), `M5.4` (vLLM+LoRA), `M3.5.9` (git refs), and
`M6.x` (agent-manager migration) are homelab/infra work independent of prior phases, can start parallel.
@@ -89,81 +89,31 @@ happening alongside M2.2/M5.4, and because the two projects' Postgres schemas
landing in the same cluster around the same time need to look like siblings,
not strangers — see M6.6's convention-consistency check.
## 1 — Read-only spine · M0.x
## ✅ Archived Phases
No model calls anywhere in this phase. The point is to prove the corpus parses
and chunks sanely before spending inference on it.
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M0.1](M0.1-cargo-workspace.md) | Cargo workspace + crate skeletons | S | — | ✅ |
| [M0.2](M0.2-domain-types.md) | Domain types and sha256 identity | S | — | ✅ |
| [M0.3](M0.3-recordsource-and-chunkpolicy.md) | `RecordSource` trait + `ChunkPolicy` | M | — | ✅ |
| [M0.4](M0.4-tokenizer-sizing.md) | Tokenizer-backed chunk sizing | M | — | ✅ |
| [M0.5](M0.5-pi-session-adapter.md) | pi session adapter | M | — | ✅ |
| [M0.6](M0.6-claude-transcript-adapter.md) | Claude transcript adapter | S | — | ✅ |
| [M0.7](M0.7-ingest-dry-run.md) | `mem ingest --dry-run` | S | — | ✅ |
| [M0.8](M0.8-m0-gate.md) | **M0 composition gate** | M | gate | ✅ |
## 2 — Gated loop at L1 · M1.x
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M1.1](M1.1-llm-chat-client.md) | `mem-llm` chat client | M | — | ✅ |
| [M1.2](M1.2-standing-query-loader.md) | Standing-query YAML loader | M | — | ✅ |
| [M1.3](M1.3-prompt-template.md) | GRU-Mem prompt template | M | — | ✅ |
| [M1.4](M1.4-gate-response-parser.md) | Gate-response parser | M | — | ✅ |
| [M1.5](M1.5-gated-loop.md) | The gated loop | L | — | ✅ |
| [M1.6](M1.6-jsonl-event-log.md) | JSONL event log writer | M | — | ✅ |
| [M1.7](M1.7-ingest-end-to-end.md) | `mem ingest` end to end | M | — | ✅ |
| [M1.8](M1.8-m1-gate.md) | **M1 composition gate** | M | gate | ✅ |
Completed and archived: **M0.x (8/8)**, **M1.x (8/8)** — all task files deleted from `/tasks/` after verification. See git log for historical record and `CLAUDE.md` for session context.
## 3 — Projections · M2.x
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M2.1](M2.1-embeddings-client.md) | Embeddings client | S | — | ⬜ |
| [M2.2](M2.2-memory-db-manifest.md) | CNPG `memory-db` + pgvector | M | homelab | ⬜ |
| [M2.3](M2.3-schema-and-migrations.md) | Schema + sqlx migrations | M | — | ⬜ |
| [M2.4](M2.4-pgvector-repo.md) | pgvector repository | M | — | ⬜ |
| [M2.5](M2.5-obsidian-projector.md) | Obsidian projector | M | — | ⬜ |
| [M2.6](M2.6-rebuild-from-log.md) | `mem rebuild --from-log` | M | — | ⬜ |
| [M2.7](M2.7-verify-edges.md) | `mem verify` — edge closure | S | — | ⬜ |
| [M2.8](M2.8-m2-gate.md) | **M2 composition gate** | M | gate | ⬜ |
**Status:** In progress · 5/8 done, 3 pending.
M2.1, M2.3, M2.4, M2.5 ✅ complete. M2.2 (CNPG manifest), M2.6, M2.7 remain. M2.8 gate awaits dependency clearance.
## 4 — L2 synthesis and retrieval · M3.x
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M3.1](M3.1-l2-synthesis.md) | L2 synthesis pass | M | — | ✅ |
| [M3.2](M3.2-rerank-client.md) | Rerank client | S | — | ✅ |
| [M3.3](M3.3-mem-query.md) | `mem query` with provenance | M | — | ✅ |
| [M3.4](M3.4-m3-gate.md) | **M3 composition gate** | M | gate | ✅ |
**Status:** ✅ Complete · 4/4 tasks done. Gate M3.4 passing. Task files archived.
## 4.5 — Distributed API Layer · M3.5.x
Homelab frontend integration: HTTP facade via `api.riotpiao.com`. Runs in parallel with M4 and M5 after M3.4 green.
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M3.5.1](M3.5.1-http-server.md) | HTTP server + router, auth hook, metrics | M | — | ✅ |
| [M3.5.2](M3.5.2-ingest-endpoint.md) | POST /ingest async queue | M | — | ✅ |
| [M3.5.3](M3.5.3-query-endpoint.md) | GET /query HNSW+rerank | M | — | ✅ |
| [M3.5.4](M3.5.4-query-federation.md) | Query federation | M | — | ✅ |
| [M3.5.5](M3.5.5-skills-endpoint.md) | GET /skills endpoint | M | — | ✅ |
| [M3.5.6](M3.5.6-projects-endpoint.md) | GET /projects endpoint | S | — | ✅ |
| [M3.5.7](M3.5.7-rate-limiting.md) | Rate limiting | M | — | ✅ |
| [M3.5.8](M3.5.8-m3.5-gate.md) | **M3.5 composition gate** | M | gate | ✅ |
| [M3.5.9](M3.5.9-git-aware-references.md) | Git-aware references: lookup by code location | M | — | ⬜ |
| [M3.5.10](M3.5.10-auth-integration.md) | Auth: Authentik/Vault OIDC token validation | M | — | ⬜ |
**Status:** 9/10 done · M3.5.8 gate ✅ passing. M3.5.1–8 archived (task files deleted). M3.5.9 (git-aware refs) and M3.5.10 (JWT/OIDC auth) remain. M3.5.10 implementation ✅ complete: Authentik OIDC provider, RS256 validation, capability-based access control. Awaiting new Docker image rollout to pods.
## 5 — Skills · M4.x
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M4.1](M4.1-skill-draft.md) | `mem skill draft` | M | — | ✅ |
| [M4.2](M4.2-derived-filter.md) | `derived: true` ingest filter | M | — | ✅ |
| [M4.3](M4.3-m4-gate.md) | **M4 composition gate** | M | gate | ⬜ |
**Status:** 2/3 done · M4.1 (skill draft) ✅ and M4.2 (derived filter) ✅ archived. Pending M4.3 gate composition.
`mem skill draft --project <proj> --from <query-id>` writes to `vault/skills/_drafts/`. Dry-run mode supported. Shingle matcher (M4.2) provides derived LessonSource filtering. M4.1-2 task files deleted after archival.
## 5.5 — Reference corpora · M3.6.x
@@ -183,7 +133,7 @@ clear while the memory got worse.
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M3.6.1](M3.6.1-doc-corpus-source.md) | `DocCorpusSource` + heading chunking | M | — | ✅ |
| M3.6.1 | `DocCorpusSource` + heading chunking | M | — | ✅ |
| [M3.6.2](M3.6.2-level-r-storage.md) | Level R: log, index, vault, rebuild parity | M | — | ⬜ |
| [M3.6.3](M3.6.3-mem-ref-cli.md) | `mem ref` — replace-on-change corpus management | M | — | ⬜ |
| [M3.6.4](M3.6.4-reference-cycle-guard.md) | Reference text cannot re-enter as evidence | M | — | ⬜ |
@@ -220,9 +170,9 @@ rather than reused.
|---|---|---|---|---|
| [M3.7.3](M3.7.3-skill-matching.md) | `GET /memory/skills?task=` — match a subset | M | — | ⬜ |
| [M3.7.4](M3.7.4-context-endpoint.md) | `/memory/context` — three-tier lookup | M | — | ⬜ |
| [M3.7.5](M3.7.5-tool-failure-learning.md) | `tool-failures` standing query | M | — | 🟡 `derive_lessons()` + `tool_of_cmd()` in `lesson.rs`, `mem resolve` in CLI |
| [M3.7.5](M3.7.5-tool-failure-learning.md) | `tool-failures` standing query | M | — | 🟡 |
| [M3.7.6](M3.7.6-m3.7-gate.md) | **M3.7 composition gate** | M | gate | ⬜ |
| [M3.7.7](M3.7.7-signature-extraction.md) | Failure signature extraction + normalisation | M | — | 🟡 `extract()` + `normalise()` in `lesson.rs` (10 unit tests passing) |
| [M3.7.7](M3.7.7-signature-extraction.md) | Failure signature extraction + normalisation | M | — | 🟡 |
| [M3.7.8](M3.7.8-symptom-projection.md) | Symptom projection at ingest | M | — | ⬜ |
## 6 — Post-training · M5.x
@@ -284,4 +234,45 @@ reporting, and resumable sync for all connectors.
---
## 9 — Hybrid search · M8.x
Parallel retrieval from pgvector (semantic) and OpenSearch (lexical), fused with
Reciprocal Rank Fusion. Adds a `QueryOptimizer` that classifies queries and
routes to the best strategy before any database call.
**The load-bearing property is accuracy.** Hybrid must produce measurably better
NDCG@10 than either engine alone. If it doesn't, the gate fails — not because
the code is broken, but because the system isn't earning its complexity budget.
M8.9 requires benchmark numbers, not just green tests.
**Approach A: Parallel RRF.** Both engines run simultaneously via `tokio::try_join!`.
Results are merged by rank position, not score magnitude, because pgvector cosine
(`[0,1]`) and BM25 (`[0,50+]`) are incomparable distributions. RRF needs no
parameter tuning (`k=60` is the academic standard). The alternative — weighted
linear combination — requires labelled data for weight selection that we don't
have yet.
**Dual-write indexing.** Every chunk gets the same UUID in both pgvector and
OpenSearch. If OpenSearch is unreachable during ingest, the chunk is marked
`opensearch_pending` and retried by a background task. The gate (M8.9) checks
for zero orphans.
**Fallback.** If OpenSearch is down at query time, the worker degrades to
semantic-only. If the embedding model is down, it degrades to lexical-only.
The response `search_strategy` field always reports which mode was actually used.
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M8.1](M8.1-opensearch-deployment.md) | OpenSearch cluster + JWT realm | M | homelab | ⬜ |
| [M8.2](M8.2-dual-write-indexer.md) | Dual-write indexing pipeline | M | — | ⬜ |
| [M8.3](M8.3-query-optimizer.md) | Query optimizer: context + routing | M | — | ⬜ |
| [M8.4](M8.4-rrf-fusion.md) | Reciprocal Rank Fusion engine | S | — | ⬜ |
| [M8.5](M8.5-hybrid-query-worker.md) | Hybrid query worker: parallel retrieval | L | — | ⬜ |
| [M8.6](M8.6-query-endpoint-upgrade.md) | Upgrade GET /query to hybrid + fallback | M | — | ⬜ |
| [M8.7](M8.7-index-optimization.md) | Index tuning: HNSW + OpenSearch analyzers | M | — | ⬜ |
| [M8.8](M8.8-accuracy-benchmarks.md) | Accuracy benchmarks: NDCG, MRR, P@K | M | — | ⬜ |
| [M8.9](M8.9-m8-gate.md) | **M8 composition gate** | M | gate | ⬜ |
---
Background: [DESIGN.md](../DESIGN.md) · GRU-Mem, arXiv 2602.10560 · `internal/store/store.go` (agent-manager, `add-headless-spawn` branch)
tasks/M0.1-cargo-workspace.md
-98
View File
@@ -1,98 +0,0 @@
# M0.1 — Cargo workspace + crate skeletons
| Field | Value |
|---|---|
| Phase | M0 — Read-only spine |
| Size | S — under 1 day |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | — |
## Goal
The six-crate workspace, building clean, with the dependency direction fixed
before any code exists to violate it.
## Facts (inlined — no spec read needed)
```
crates/
mem-core/ domain types; Level; gate-response parser; the gated loop
mem-chunk/ RecordSource trait; ChunkPolicy; FlushTrigger
mem-llm/ gateway client — chat, embeddings, rerank
mem-ingest/ source adapters: pi sessions, claude transcripts
mem-store/ JSONL log; pgvector repo; Obsidian projector
mem-cli/ binary `mem`
```
Dependency direction, enforced from the start:
```
mem-cli -> mem-ingest, mem-store, mem-llm, mem-chunk, mem-core
mem-store -> mem-core
mem-ingest -> mem-chunk, mem-core
mem-chunk -> mem-core
mem-llm -> mem-core
mem-core -> (nothing in this workspace)
```
`mem-core` depends on no sibling. It holds the types every other crate speaks,
so a dependency out of it is a cycle waiting to happen.
Workspace deps to pin now, so versions do not drift per crate: `tokio`,
`futures`, `serde`, `serde_json`, `serde_yaml`, `anyhow`, `thiserror`,
`sha2`, `clap`, `reqwest`, `tracing`.
## Steps
1. `Cargo.toml` at the repo root with `[workspace] members = [...]` and a
`[workspace.dependencies]` block holding every shared crate version.
2. Six member crates, each declaring deps as `foo.workspace = true`.
3. `mem-cli` is the only `[[bin]]`; the rest are libraries.
4. Add `rust-toolchain.toml` pinning a version, so CI and laptop agree.
5. `.gitignore`: `target/`, `vault/` — but **not** `log/` and **not**
`memory-tasks/`. The log is authoritative and the board carries acceptance
criteria; both are tracked.
6. Wire a CI job running `cargo build --workspace` and `cargo clippy --workspace
-- -D warnings`.
## Acceptance
- `cargo build --workspace` succeeds from a clean checkout.
- `cargo clippy --workspace -- -D warnings` is clean.
- `mem-core` has zero intra-workspace dependencies.
## Verify
**Harness:** cargo itself, plus a dependency assertion that does not trust the
manifests to be read by a human.
**Integration test** — `tests/it_workspace.rs` in the root:
1. `a1_all_members_build` — shell out to `cargo build --workspace`, assert exit 0.
2. `a2_mem_core_has_no_sibling_deps` — parse `crates/mem-core/Cargo.toml`, assert
no dependency name starts with `mem-`.
3. `a3_dependency_direction` — parse every member manifest, build the edge set,
assert it is a subset of the table above and that the graph is acyclic.
4. `a4_log_and_tasks_are_tracked` — assert `.gitignore` matches neither
`log/` nor `memory-tasks/`.
**Command:** `cargo test --workspace workspace`
**False pass:**
- Asserting the graph is acyclic **only**. Acyclic permits `mem-core ->
mem-store`, which is backwards and still acyclic. Assertion 3 must check the
edge set against the table, not just for cycles.
- A CI job that runs `cargo build` in the root without `--workspace`. It builds
the virtual manifest and can miss a member that does not compile.
## Traps
- Per-crate dependency versions instead of `workspace.dependencies`. They drift,
and two versions of `serde` in one tree is a confusing type error much later.
- Gitignoring `log/`. It is the authoritative record; ignoring it makes the whole
authority model a fiction.
---
Background: [DESIGN.md](../DESIGN.md) — Repository layout
tasks/M0.2-domain-types.md
-115
View File
@@ -1,115 +0,0 @@
# M0.2 — Domain types and sha256 identity
| Field | Value |
|---|---|
| Phase | M0 — Read-only spine |
| Size | S — under 1 day |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M0.1 |
## Goal
The vocabulary every other crate speaks, and the content-hash identity the whole
provenance graph hangs on.
## Facts (inlined — no spec read needed)
```rust
/// Closed. L0 evidence, L1 per-query memory, L2 project synthesis.
#[derive(Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum Level { L0, L1, L2 }
/// A normalised unit from any source. Adapters produce these; nothing
/// downstream learns whether it came from pi, claude, or a socket.
pub struct Record {
pub role: Role, // User | Assistant | ToolResult | System
pub text: String,
pub timestamp: OffsetDateTime,
pub provenance: Provenance, // source id + offset within it
}
/// One or more Records, under the token budget, never split mid-Record.
pub struct Chunk {
pub t: u32, // 1-based turn index within a run
pub records: Vec<Record>,
pub tokens: usize,
pub sha256: Sha256Hash,
}
pub struct MemoryNode {
pub level: Level,
pub project: ProjectId,
pub query_id: Option<QueryId>, // None at L2
pub run_id: RunId,
pub t: u32,
pub text: String,
pub sha256: Sha256Hash,
pub parents: Vec<Sha256Hash>,
}
```
**Identity is the content hash, not a counter.** `sha256` is computed over the
canonical serialization of the semantic content — for `Chunk`, the concatenated
record texts and their provenance; for `MemoryNode`, `(level, project, query_id,
text)`. It must **not** include the timestamp or the run id, or re-running the
same input produces different hashes and `mem rebuild` stops being idempotent.
Newtypes with no `Default`: `ProjectId`, `QueryId`, `RunId`, `Sha256Hash`. A
placeholder that type-checks is invisible — a hardcoded `"current"` compiles,
passes tests, and makes every downstream result unattributable.
## Steps
1. Declare `Level`, `Role`, `Record`, `Provenance`, `Chunk`, `MemoryNode`.
2. Declare the newtypes. None derives `Default`. None has `From<String>` without
validation.
3. `fn content_hash(&self) -> Sha256Hash` on `Chunk` and `MemoryNode`, over a
canonical byte encoding that excludes timestamps and run ids.
4. `Level` serializes as the literal strings `"L0" | "L1" | "L2"` — the JSONL and
the SQL `CHECK` constraint both depend on that spelling.
5. Round-trip serde tests for every type.
## Acceptance
- Two `Chunk`s built from identical records in different runs hash identically.
- Changing one character of any record text changes the hash.
- `Level` round-trips through JSON as `"L0"`, not `0` and not `"l0"`.
## Verify
**Harness:** unit tests in `mem-core`, plus a hash-stability fixture committed
as bytes.
**Integration test** — `tests/it_identity.rs`:
1. `a1_same_content_same_hash` — build the same chunk twice with different
`RunId` and timestamps, assert equal hashes.
2. `a2_text_change_changes_hash` — flip one byte, assert the hash differs.
3. `a3_level_wire_format` — `serde_json::to_string(&Level::L0) == "\"L0\""`.
4. `a4_hash_stability_across_versions` — hash a committed fixture record set,
assert it equals a hash literal written into the test. This catches a
canonicalization change that would silently orphan every stored node.
5. `a5_newtypes_have_no_default` — compile-fail test (`trybuild`) asserting
`ProjectId::default()` does not compile.
**Command:** `cargo test -p mem-core identity`
**False pass:**
- Asserting only that hashing is deterministic *within one process*. A hash that
includes the timestamp is deterministic per run and still breaks rebuild.
Assertion 1 must vary the run id and timestamp deliberately.
- Omitting assertion 4. Without a committed expected hash, any future
canonicalization change passes every other test and silently invalidates the
database.
## Traps
- Including `run_id` or `created_at` in the hash. Rebuild then produces new nodes
every time and `memory_edge` accumulates orphans.
- Deriving `Default` on an id newtype "for tests". That default reaches
production and every row attributes to the same fake project.
---
Background: [DESIGN.md](../DESIGN.md) — The tier model, Storage schemas
tasks/M0.3-recordsource-and-chunkpolicy.md
-106
View File
@@ -1,106 +0,0 @@
# M0.3 — `RecordSource` trait + `ChunkPolicy`
| Field | Value |
|---|---|
| Phase | M0 — Read-only spine |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M0.2 |
## Goal
The seam where new input kinds arrive, shaped as a stream from the first commit
so a future streaming source implements a trait instead of forcing a rewrite.
## Facts (inlined — no spec read needed)
```rust
pub trait RecordSource {
/// Sources decide how to produce records; the chunker never learns
/// whether they came from pi, claude, or a socket.
fn records(self) -> impl Stream<Item = Result<Record>>;
}
pub struct ChunkPolicy {
pub max_tokens: usize, // 5000 — GRU-Mem paper default
pub split_on: Boundary, // Boundary::Record — never mid-Record
pub flush: FlushTrigger,
}
pub enum FlushTrigger {
Tokens(usize),
// OrIdle(Duration) lands with the first streaming source. Carrying the
// enum now means that change is one variant, not a signature change
// threaded through the loop.
}
pub fn chunks<S: RecordSource>(src: S, p: ChunkPolicy) -> impl Stream<Item = Chunk>;
```
Why a stream when both current sources are files: sources today have an EOF;
telemetry, a live session tail, or a broker will not. Batch sources become
streams for free via `futures::stream::iter`, so this costs nothing today and
removes a rewrite later. The rest of the stack is already tokio.
**A single `Record` larger than `max_tokens` is not an error.** Tool results can
be enormous. It becomes a chunk of one, over budget, and the chunker records that
it did — silently truncating would destroy evidence, and silently dropping would
lose it.
## Steps
1. Define `RecordSource`, `ChunkPolicy`, `Boundary`, `FlushTrigger` in `mem-chunk`.
2. Implement `chunks()` as a `Stream` adapter that accumulates until the next
record would exceed `max_tokens`, then yields.
3. Never split a `Record`. An oversized single record yields alone, with
`Chunk::over_budget = true`.
4. `t` is 1-based and contiguous across the whole stream.
5. Provide `VecSource(Vec<Record>)` implementing `RecordSource` for tests, so
chunking is testable with no I/O and no model.
6. Token counting is behind a `TokenCounter` trait — M0.4 supplies the real one;
a `CharsOverFour` stub is enough here.
## Acceptance
- Chunk boundaries never fall inside a `Record`.
- Chunk `t` values are 1-based, contiguous, no gaps.
- An oversized single record yields one chunk flagged `over_budget`.
- Concatenating all chunks' records reproduces the input sequence exactly.
## Verify
**Harness:** `VecSource` + the stub counter. No files, no network.
**Integration test** — `tests/it_chunking.rs`:
1. `a1_no_record_is_split` — for every chunk, every record equals some input
record byte for byte.
2. `a2_lossless` — flatten all chunk records; assert the sequence equals the
input sequence, same order, same length.
3. `a3_t_is_contiguous` — assert `t` values are exactly `1..=n`.
4. `a4_respects_budget` — every chunk is either under `max_tokens` or has exactly
one record and `over_budget = true`.
5. `a5_oversized_record_survives` — feed one record of 3× budget; assert it
appears whole in the output, not truncated and not dropped.
6. `a6_empty_source` — zero records yields zero chunks, no panic.
**Command:** `cargo test -p mem-chunk chunking`
**False pass:**
- Testing only with uniformly small records. The budget logic is never exercised
and the oversized path never runs. Assertion 5 is the guard.
- Asserting chunk count rather than losslessness. A chunker that drops the final
partial chunk produces a plausible count and loses the tail — assertion 2 is
what catches it.
## Traps
- Truncating an oversized record to fit the budget. That is silent evidence
destruction, and the update gate will later be blamed for missing it.
- Materializing the stream into a `Vec` inside `chunks()`. It compiles, passes
every test here, and defeats the entire reason this crate is separate.
---
Background: [DESIGN.md](../DESIGN.md) — `mem-chunk`, stream-shaped from day one
tasks/M0.4-tokenizer-sizing.md
-92
View File
@@ -1,92 +0,0 @@
# M0.4 — Tokenizer-backed chunk sizing
| Field | Value |
|---|---|
| Phase | M0 — Read-only spine |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M0.3 |
## Goal
Count tokens with the tokenizer the serving model actually uses, so a chunk that
fits locally also fits at the gateway.
## Facts (inlined — no spec read needed)
The controller is `qwen2.5:3b-instruct` — Qwen2.5-3B-Instruct, the GRU-Mem
paper's exact 3B backbone. Its tokenizer is the Qwen2 BPE.
Budget arithmetic, and why an approximation is not good enough:
```
Ollama context cap 32768 (OLLAMA_CONTEXT_LENGTH, cluster-side)
chunk 5000
prompt overhead + memory ~3200 (system + question + M_{t-1} at 1024)
response 2048
```
A chars/4 estimate drifts 20–30% on code and JSON — which is most of this corpus.
Undercount and the gateway rejects the request; overcount and chunks are smaller
than they need to be, which multiplies the number of model calls.
`tokenizers` (HuggingFace) loads the Qwen2 tokenizer from a vendored
`tokenizer.json`. **Vendor the file into the repo** rather than downloading at
runtime: a build that needs the network is a build that fails offline, and a
tokenizer that changes under you silently re-chunks the entire corpus.
## Steps
1. Vendor `assets/qwen2-tokenizer.json` and record its sha256 in the repo.
2. Implement `TokenCounter` for it in `mem-chunk`, loading once and reusing.
3. Assert at load that the vendored file's hash matches the recorded one.
4. Make `max_tokens` and the model id configurable, defaulting to 5000 and
`qwen2.5:3b-instruct`.
5. Add a `mem tokens <file>` debug subcommand printing the token count of a file,
for cross-checking against the gateway's reported `prompt_tokens`.
## Acceptance
- Counts match the gateway's `usage.prompt_tokens` within ±2% on a sample of
real records.
- Loading with a modified tokenizer file fails loudly, not silently.
## Verify
**Harness:** the vendored tokenizer plus recorded gateway responses. The
cross-check against the live gateway is a separate, network-gated test.
**Integration test** — `tests/it_tokens.rs`:
1. `a1_known_strings` — a table of ~20 strings with hand-recorded expected counts
(ASCII, CJK, code, JSON, emoji), asserted exactly.
2. `a2_hash_guard` — corrupt a copy of the tokenizer file, assert load returns an
error naming the file.
3. `a3_gateway_agreement` — `#[ignore]` by default, run with `--ignored`: send 10
real records to `/v1/qwen/chat/completions` with `max_tokens: 1`, compare
`usage.prompt_tokens` to the local count minus the measured template overhead;
assert within 2%.
4. `a4_budget_holds` — chunk a real pi session at 5000 tokens; assert no chunk's
locally-counted size exceeds the budget.
**Command:** `cargo test -p mem-chunk tokens` (add `-- --ignored` for a3)
**False pass:**
- Testing only ASCII. Qwen2 BPE tokenizes CJK and emoji very differently, and
this corpus contains both — the caveman skill text alone is bilingual.
- Comparing the local count to itself via a helper that calls the same function.
Assertion 1's expected values must be recorded from the tokenizer once and
written as literals.
## Traps
- Downloading the tokenizer at runtime. Offline builds break, and a silent
upstream change re-chunks everything, which changes every chunk hash, which
orphans every stored node.
- Forgetting that the 32768 cap is cluster-side. `models.json` claims 131072 for
ornith and is wrong; do not take a client-side config as the source of truth.
---
Background: [DESIGN.md](../DESIGN.md) — Verified facts, `mem-chunk`
tasks/M0.5-pi-session-adapter.md
-131
View File
@@ -1,131 +0,0 @@
# M0.5 — pi session adapter
| Field | Value |
|---|---|
| Phase | M0 — Read-only spine |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M0.3 |
## Goal
Turn pi's session JSONL into normalised `Record`s, and resolve the project key
from the path without guessing.
## Facts (inlined — no spec read needed)
Layout, verified on this machine:
```
~/.pi/agent/sessions/--Users-rockliang-workplace-Poimen-agent-rust--/<ts>_<uuid>.jsonl
└─ cwd with / replaced by -, wrapped in leading and trailing --
```
Record types observed in a real 2902-message session:
```
message 2902
model_change 112
thinking_level_change 16
compaction 8
session 1 <- always first line
```
Shapes:
```jsonc
// first line
{"type":"session","version":..,"id":"..","timestamp":"..","cwd":"/Users/.../Poimen"}
// message
{"type":"message","id":"..","parentId":"..","timestamp":"..",
"message":{"role":"assistant|user|toolResult","content":..,"timestamp":".."}}
```
Role distribution in that same session — this is the whole reason the update gate
exists:
```
assistant 1445
toolResult 1261 43%, mostly evidence-free
user 196
```
`cwd` in the `session` header is authoritative for the project key. The directory
name is a lossy encoding (a real `-` in a path is indistinguishable from a
separator) — **parse `cwd`, do not decode the directory name.**
`content` is not always a string. Assistant messages carry content blocks; tool
results carry structured payloads. Normalise to text, and keep the block type in
`Provenance` so a later filter can act on it.
## Steps
1. Implement `PiSessionSource` in `mem-ingest`, implementing `RecordSource`.
2. Read the first line, require `type == "session"`, take `cwd` as the project
key. A file whose first line is not a session header is an error naming the
file, not a skip.
3. Stream subsequent lines; emit a `Record` per `type == "message"`.
4. Map roles: `user -> Role::User`, `assistant -> Role::Assistant`,
`toolResult -> Role::ToolResult`.
5. Flatten `content` to text for all shapes; preserve the original block type in
`Provenance`.
6. Ignore `model_change`, `thinking_level_change`. **Do not ignore `compaction`** —
emit it as `Role::System` with the marker text, because a compaction boundary
is where context was lost and that is worth seeing in the log.
7. `Provenance` = `pi:<session-file-stem>` plus the record's `id` and line offset.
8. A malformed line is a counted, reported skip — never a panic. These files are
appended to by a live process and the last line may be a partial write.
## Acceptance
- Project key comes from `cwd`, matching for a path containing a literal `-`.
- All three roles are emitted with correct counts on a real session.
- A truncated final line is skipped with a warning, not a panic.
- Compaction events appear in the record stream.
## Verify
**Harness:** two committed fixtures — one small hand-built session covering every
record type and content shape, one real session copied verbatim (secrets
scrubbed) for volume.
**Integration test** — `tests/it_pi_source.rs`:
1. `a1_project_from_cwd` — fixture whose `cwd` is `/tmp/my-project`; assert the
key is `/tmp/my-project`, proving the directory name was not decoded.
2. `a2_role_counts` — on the real fixture, assert exact counts per role.
3. `a3_content_shapes` — string content, block-array content, and structured tool
result all flatten to non-empty text.
4. `a4_truncated_tail` — append half a JSON object; assert the source yields all
prior records and reports exactly one skip.
5. `a5_missing_header` — file whose first line is a `message`; assert an error
naming the path.
6. `a6_compaction_emitted` — assert compaction events appear as `Role::System`.
7. `a7_stream_is_lazy` — a source over a 50 MB fixture must yield its first
record before reading the whole file (assert peak allocation, or instrument
reads).
**Command:** `cargo test -p mem-ingest pi_source`
**False pass:**
- Testing only against the hand-built fixture. It will contain the content shapes
you thought of, which is the set you already handle. Assertion 2 against a real
session is what finds the rest.
- Asserting "no panic" on malformed input without asserting the *count* of
skips. A source that silently drops every line panics never and ingests
nothing.
## Traps
- Decoding the directory name to get the project. `--Users-rockliang-workplace-my-proj--`
is ambiguous the moment a path component contains `-`, which is common.
- Treating `content` as `String`. It parses for user messages and fails on
assistant blocks, so the bug appears to be about assistants specifically and
wastes an afternoon.
- Materializing the file. These reach tens of MB and the trait is a stream for a
reason.
---
Background: [DESIGN.md](../DESIGN.md) — Context, `mem-chunk`
tasks/M0.6-claude-transcript-adapter.md
-92
View File
@@ -1,92 +0,0 @@
# M0.6 — Claude transcript adapter
| Field | Value |
|---|---|
| Phase | M0 — Read-only spine |
| Size | S — under 1 day |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M0.5 |
## Goal
The second `RecordSource`, which is the one that proves the trait is real.
## Facts (inlined — no spec read needed)
```
~/.claude/projects/-Users-rockliang-workplace-Poimen/<uuid>.jsonl
└─ cwd, / replaced by -, no wrapping dashes (differs from pi)
```
15 MB across 7 transcripts on this machine. Record types observed:
```
attachment 150 queue-operation 138 assistant 134 user 81
file-history-snapshot 69 system 69 ai-title 21 last-prompt 21 mode 21
```
Fields, read from any line: `sessionId`, `cwd`, `gitBranch`.
Typed records dispatch on `type`; the ones that matter here:
- `user` / `assistant` — content under `message.content`
- `system` with `subtype == "api_error"` — a failed turn, worth keeping
- `summary` — carries `summary`
- everything else — ignore
The filename stem is a UUID. The encoding differs from pi (no wrapping `--`),
which is exactly why the project key must come from the `cwd` **field**, as in
M0.5, and not from the directory name.
## Steps
1. Implement `ClaudeTranscriptSource` in `mem-ingest`, implementing `RecordSource`.
2. Take `cwd` from the first line carrying it; error if no line does.
3. Emit records for `user`, `assistant`, and `system`+`api_error`.
4. Flatten `message.content` across its shapes, as in M0.5.
5. `Provenance` = `claude:<uuid>` plus line offset.
6. Reuse the content-flattening and malformed-line handling from M0.5 — extract
them into a shared helper rather than copying, since divergence between two
flatteners is a bug that only shows up on one source.
## Acceptance
- Both sources satisfy `RecordSource` with no changes to `mem-chunk`.
- Project keys from pi and claude for the same directory resolve to the same
`ProjectId`.
- `api_error` system records survive into the stream.
## Verify
**Harness:** a committed transcript fixture plus a real one, secrets scrubbed.
**Integration test** — `tests/it_claude_source.rs`:
1. `a1_project_from_cwd_field` — assert the key comes from `cwd`, not the
directory name, using a fixture where they would differ.
2. `a2_same_project_across_sources` — a pi session and a claude transcript for
the same directory yield an equal `ProjectId`. This is the assertion that
makes cross-source memory possible at all.
3. `a3_role_mapping` — user/assistant/api_error appear; `attachment`,
`queue-operation`, `file-history-snapshot` do not.
4. `a4_shared_flattener` — the same content-block fixture flattens identically
through both sources (call both, compare strings).
**Command:** `cargo test -p mem-ingest claude_source`
**False pass:**
- Testing each source in isolation. The point of this task is that they agree;
assertions 2 and 4 are the only ones that check it, and both are cross-source.
## Traps
- Copying the flattener instead of sharing it. The two will drift, and the
resulting bug looks like "claude transcripts lose tool output" rather than
"there are two flatteners".
- Assuming the pi encoding. Claude has no wrapping `--`, so a decoder written
against pi silently produces a different project key for the same directory —
and cross-source memory quietly splits in two.
---
Background: [DESIGN.md](../DESIGN.md) — Context
tasks/M0.7-ingest-dry-run.md
-94
View File
@@ -1,94 +0,0 @@
# M0.7 — `mem ingest --dry-run`
| Field | Value |
|---|---|
| Phase | M0 — Read-only spine |
| Size | S — under 1 day |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M0.4, M0.6 |
## Goal
See the chunk plan for a real project before spending a single model call on it.
## Facts (inlined — no spec read needed)
`--dry-run` makes **zero network calls**. That is the property, not a side
effect: this is the last checkpoint before inference, and its value is telling
you the corpus is sane while feedback is still free.
Output shape:
```
project /Users/rockliang/workplace/Poimen/agent-rust
sources pi:4 files claude:2 files
records 3118 (assistant 1445 toolResult 1261 user 196 system 216)
chunks 412 over-budget 3
tokens min 84 p50 4870 p95 5000 max 11204
```
`over-budget` counts single records exceeding the chunk budget (M0.3). A nonzero
count is expected — large tool results — and is worth surfacing because it
predicts requests the gateway may reject.
Numbers to sanity-check against: a real pi session in this project has 2902
messages with the role split above, and toolResult being ~43% is the signal that
the update gate has something to discriminate.
## Steps
1. `mem ingest --project <path-or-key> --dry-run`.
2. Resolve sources: scan both source roots for directories whose `cwd` matches
the project. Report which files matched.
3. Stream records through `mem-chunk` with the real tokenizer; accumulate stats
without retaining chunk bodies.
4. Print the table above. Machine-readable variant behind `--format json`.
5. `--limit <n>` to stop after n chunks, for iterating on a large project.
6. Exit non-zero if zero sources matched — a silent empty plan reads like success.
## Acceptance
- No network syscall occurs during `--dry-run`.
- Stats are computed streaming; memory does not scale with corpus size.
- Zero matched sources exits non-zero with a message naming the project key.
## Verify
**Harness:** the fixtures from M0.5/M0.6, plus a network guard.
**Integration test** — `tests/it_dry_run.rs`:
1. `a1_no_network` — run the command with outbound TCP blocked (inject a
`reqwest` client that panics on use, or set an unroutable proxy); assert exit 0.
2. `a2_counts_match_sources` — record and role counts equal the sum of what the
two adapters yield independently.
3. `a3_chunk_count_matches_chunker` — the reported chunk count equals
`chunks(...).count()` computed separately.
4. `a4_over_budget_reported` — fixture with one oversized record; assert
`over-budget 1`.
5. `a5_empty_project_fails` — unknown project key exits non-zero, message names
the key.
6. `a6_constant_memory` — run against a 50 MB fixture; assert peak RSS stays
under a bound well below file size.
**Command:** `cargo test -p mem-cli dry_run`
**False pass:**
- Asserting the command exits 0 and printing looks right. A dry run that matched
no sources also exits 0 and prints a tidy table of zeros — assertion 5 is what
separates them.
- Computing stats by collecting chunks into a `Vec` first. Every assertion here
passes and assertion 6 is the only one that fails, which is why it is present.
## Traps
- Making `--dry-run` construct the LLM client "but not call it". Construction
reads credentials and can fail; the guarantee is no network, and the cheapest
way to keep it is to not build the client at all.
- Reporting p50 chunk size only. The max is the interesting number — it predicts
which requests will be rejected downstream.
---
Background: [DESIGN.md](../DESIGN.md) — Verification, P1
tasks/M0.8-m0-gate.md
-87
View File
@@ -1,87 +0,0 @@
# M0.8 — M0 composition gate
| Field | Value |
|---|---|
| Phase | M0 — Read-only spine |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | gate |
| Spec | inlined below |
| Blocks | all of M0 |
## Goal
Prove the parts compose and that the swappable part is genuinely swappable —
the properties no single M0 task owns.
## Facts (inlined — no spec read needed)
The claim this phase makes: **a new input kind is added by implementing
`RecordSource`, and nothing downstream changes.** If that is false, the crate
split bought nothing and the streaming work later will be a rewrite.
The gate proves it by adding a third source that resembles nothing already
supported, and asserting the rest of the pipeline is untouched.
Second claim: the whole phase is free. No model calls, so the gate is a CI job
that runs on every push without a gateway or credentials.
## Steps
1. Implement `SyntheticSource` in the test tree only — generates records from a
seed, has no file format, and deliberately produces a record larger than the
chunk budget and a run of empty-text records.
2. Assert `chunks()` handles it with **no change** to `mem-chunk`.
3. Assert dependency direction still holds (M0.1 assertion 3) after three sources
exist — this is when someone is tempted to reach backwards.
4. Run `mem ingest --dry-run` over all three sources for one project and diff the
summary against a committed expected file.
5. Wire the whole thing as a required CI job.
## Acceptance
- Three sources, one `RecordSource`, zero source-specific branches in `mem-chunk`
or `mem-cli`.
- Dry-run summary diffs empty against the committed expectation.
- CI job passes with no network and no credentials.
## Verify
**Harness:** committed expected-output file, diffed. The script's output is the
review artifact.
**Integration test** — `tests/it_m0_gate.rs`:
1. `a1_third_source_needs_no_downstream_change` — `SyntheticSource` flows through
`chunks()`; assert `git diff --stat crates/mem-chunk crates/mem-cli` is empty
for the commit that added it (enforced by a CI step, not by the test binary).
2. `a2_no_source_specific_branches` — grep `crates/mem-chunk` and `crates/mem-cli`
for the strings `pi:`, `claude:`, `sessionId`, `toolResult`; assert none
appear outside `mem-ingest`.
3. `a3_dry_run_golden` — run the dry run over all three sources, diff against
`expected/m0-gate.txt`; empty diff is the only pass.
4. `a4_offline` — the entire gate runs with networking disabled.
5. `a5_lossless_end_to_end` — records in equals records out, across all three
sources composed.
**Command:** `cargo test --workspace m0_gate`
**False pass:**
- Adding `SyntheticSource` in a way that mirrors the pi format. It then exercises
the same code path and proves nothing about generality. It must have no file,
no header line, and an oversized record.
- Assertion 3 passing because the expected file was regenerated in the same
commit. A changed `expected/` file in a diff is a claim that the contract
changed, and must be reviewed as one.
## Traps
- Skipping assertion 2 because "obviously there are no source-specific branches".
There will be — the first `if provenance.starts_with("pi:")` gets added to fix
a real bug and is entirely reasonable in isolation.
- Letting the gate need credentials. The value of an offline gate is that it runs
on every push; the moment it needs a gateway key it becomes a nightly job that
nobody watches.
---
Background: [DESIGN.md](../DESIGN.md) — `mem-chunk`, Verification
tasks/M1.1-llm-chat-client.md
-170
View File
@@ -1,170 +0,0 @@
# M1.1 — `mem-llm` chat client
| Field | Value |
|---|---|
| Phase | M1 — Gated loop at L1 |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M0.1 |
## Goal
Talk to the homelab gateway, with the two non-obvious details that cost a day to
find already baked in.
## Files
| Action | Path |
|---|---|
| Create | `crates/mem-llm/src/chat.rs` — `ChatClient`, `Completion`, `Usage` |
| Replace | `crates/mem-llm/src/lib.rs` — replace `pub mod placeholder {}` with `pub mod chat;` + re-exports |
| Create | `tests/it_chat_client.rs` — integration tests (workspace root, matches existing convention) |
| Modify | `Cargo.toml` root — add `wiremock = "0.6"` to `[dev-dependencies]` |
## Dependencies
| Crate | Where | Already present? |
|---|---|---|
| `reqwest` (json feature) | `crates/mem-llm/Cargo.toml` | ✅ yes |
| `serde`, `serde_json` | `crates/mem-llm/Cargo.toml` | ✅ yes |
| `tokio` | `crates/mem-llm/Cargo.toml` | ✅ yes |
| `anyhow`, `thiserror` | `crates/mem-llm/Cargo.toml` | ✅ yes |
| `wiremock = "0.6"` | root `Cargo.toml` `[dev-dependencies]` | ❌ add |
## Existing code
- `crates/mem-llm/src/lib.rs` is an empty placeholder — replace entirely
- No existing HTTP client code to reuse; build from scratch
- `crates/mem-core/src/lesson.rs` has an unrelated events JSONL writer — ignore it here
## API shape
```http
POST https://api.riotpiao.com/v1/qwen/chat/completions
Headers:
apikey: <value of MEM_API_KEY env var>
Content-Type: application/json
Body (note: NO "tools" key — not even an empty array):
{
"model": "qwen2.5:3b-instruct",
"messages": [
{"role": "system", "content": "<system prompt>"},
{"role": "user", "content": "<user prompt>"}
],
"max_tokens": 2048
}
Response 200:
{
"choices": [
{"message": {"role": "assistant", "content": "<model output>"}}
],
"usage": {
"prompt_tokens": 1234,
"completion_tokens": 567,
"total_tokens": 1801
}
}
Response 401 (wrong auth header):
{"message": "Unauthorized"}
Response 400 (body too large or malformed):
{"error": {"message": "[] is too short - 'messages'"}}
```
## Facts (inlined — no spec read needed)
```
base https://api.riotpiao.com/v1
route POST /v1/qwen/chat/completions qwen2.5:3b-instruct
POST /v1/ornith/chat/completions ornith:35b
POST /v1/reasoning/chat/completions DeepSeek-R1-Distill-32B (no tools)
```
**Auth is `apikey:`, not `Authorization: Bearer`.** Kong's `key-auth` compares
the whole header value against the stored key, so the OpenAI SDK convention
returns 401. Verified:
```
-H "apikey: $KEY" -> 200
-H "Authorization: $KEY" -> 200
-H "Authorization: Bearer $KEY" -> 401
```
**One provider per route.** `baseUrl` is per-route and the model id is `ornith:35b`
with the tag, not `ornith`. `/v1/models` advertises `/v1/score` which does not
work — do not trust that list as a capability probe.
Request bodies above ~10.6 KB used to fail with
`{"error":{"message":"[] is too short - 'messages'"}}`; the Kong body buffer was
raised to 16m and it is fixed. If that error ever reappears, it is the buffer,
not the client.
Send **no `tools` array**. The controller needs none, and the reasoning route
rejects any request carrying one.
## Steps
1. `ChatClient::new(base_url, api_key, model)` in `mem-llm`.
2. Send the `apikey` header. Read the key from `MEM_API_KEY`, never from a
committed file.
3. `complete(system, user, max_tokens) -> Completion { text, usage, latency }`.
4. Timeout default 300s — local models are slow to first token and a cold load
can take minutes.
5. Retry on 5xx and timeout with exponential backoff, max 3. **Do not retry 4xx**
— a 400 is a malformed request and retrying it just costs three times as much.
6. On any error, include the response body in the error. The useful information
is always in the body, never the status.
7. `MEM_LLM_RECORD=<dir>` writes every request/response pair to disk, for
building fixtures without hand-writing them.
## Acceptance
- A real completion round-trips against the gateway.
- A 401 is reported as an auth error naming the header convention.
- A 4xx is not retried; a 5xx is.
## Verify
**Harness:** `wiremock` for the offline tests; one `#[ignore]` test against the
live gateway.
**Integration test** — `tests/it_chat_client.rs`:
1. `a1_sends_apikey_header` — assert the mock received `apikey` and **no**
`authorization` header.
2. `a2_no_tools_field` — assert the serialized body has no `tools` key at all,
not merely an empty array.
3. `a3_retries_5xx` — mock 503 twice then 200; assert 3 requests and success.
4. `a4_does_not_retry_4xx` — mock 400; assert exactly 1 request and an error
carrying the body text.
5. `a5_timeout_is_configurable` — mock a 2s delay with a 1s timeout; assert a
timeout error.
6. `a6_live_smoke` — `#[ignore]`; real gateway, `qwen2.5:3b-instruct`, prompt
"reply with exactly: pong", assert the text contains `pong`.
**Command:** `cargo test --test it_chat_client` (add `-- --ignored` for a6)
**False pass:**
- Testing only against the mock. The mock accepts whatever header you send it;
assertion 6 against the live gateway is the only thing that proves the auth
convention is right.
- Asserting `tools: []` is absent by checking `body.tools.is_empty()`. An empty
array serialized into the request is still a `tools` key, and that is what the
reasoning route rejects. Assert on the raw JSON.
## Traps
- Using `Authorization: Bearer`. It is what every SDK does and it 401s here.
- Retrying 400s. The body-buffer bug produced a 400 for a whole day; retrying it
tripled the load and produced identical failures more slowly.
- A 60s timeout. That is the value that made `ornith` look broken when it was
merely cold.
---
Background: [DESIGN.md](../DESIGN.md) — Verified facts
tasks/M1.2-standing-query-loader.md
-130
View File
@@ -1,130 +0,0 @@
# M1.2 — Standing-query YAML loader
| Field | Value |
|---|---|
| Phase | M1 — Gated loop at L1 |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M0.2 |
## Goal
Load the standing questions that give the update gate its referent, and fail at
load rather than mid-run when one is wrong.
## Files
| Action | Path |
|---|---|
| Create | `crates/mem-core/src/query.rs` — `QuerySet`, `Query`, `SynthesisQuery`, `QueryLoadError` |
| Modify | `crates/mem-core/src/lib.rs` — add `pub mod query;` and re-exports |
| Create | `queries/poimen.yaml` — first real standing query file |
| Create | `tests/it_query_loader.rs` — integration tests (workspace root) |
| Create | `fixtures/query-valid.yaml` — test fixture (valid) |
| Create | `fixtures/query-empty-question.yaml` — test fixture (empty question) |
| Create | `fixtures/query-duplicate-id.yaml` — test fixture (duplicate id) |
| Create | `fixtures/query-bad-charset.yaml` — test fixture (invalid id chars) |
## Dependencies
| Crate | Where | Already present? |
|---|---|---|
| `serde_yaml` | workspace deps | ✅ yes (in workspace `[workspace.dependencies]`) |
| `serde` | `crates/mem-core/Cargo.toml` | ✅ yes |
| `regex` | `crates/mem-core/Cargo.toml` | ❌ add (for `[a-z0-9-]+` validation) or hand-roll |
## Existing code to reuse
- `ProjectId`, `QueryId` from `crates/mem-core/src/domain.rs` — **use these newtypes**, don't create new ones
- `serde_yaml` already used by `mem-ingest` — same pattern
- Validation pattern: `QueryId::new()` already rejects empty strings; extend with charset validation
## Facts (inlined — no spec read needed)
```yaml
# queries/poimen.yaml
project: poimen
roots:
- /Users/rockliang/workplace/Poimen/agent-rust # matched against session cwd
sources: [pi, claude]
queries:
- id: architecture-decisions
question: What architectural decisions were made, with reasoning and rejected alternatives?
- id: infra-root-causes
question: What infrastructure bugs were found, what was the root cause, how was it isolated?
synthesis:
question: What is the current state of this project, and what should someone know before working on it?
exit_gate: true
defaults:
memory_budget: 1024
chunk_tokens: 5000
exit_gate: false # L1 default — see below
```
**Why a question is mandatory.** The GRU-Mem memory agent is `φθ(Q, C_t, M_{t-1})`
and its update gate is defined as "does this chunk contain useful information
*about the problem*". With no `Q` the gate has no referent, and `r_update` is
undefinable — which forecloses post-training (M5) entirely. A query with an empty
question is a load error, not a warning.
**`exit_gate: false` at L1 is deliberate.** Paper §3.3: for "what are *all* the
X" questions you cannot know evidence is sufficient without reading everything,
so the paper itself provides a without-exit-gate inference mode. L1 extraction is
that shape. The gate is still *recorded* — its signal is needed for M5.
`query.id` is stable and frozen. It names the L1 memory, the Obsidian note, and
the log directory; renaming it orphans all three.
## Steps
1. `QuerySet::load(path)` in `mem-core`, `serde_yaml`.
2. Validate at load: non-empty `project`; at least one query; every `id` unique,
non-empty, and `[a-z0-9-]+`; every `question` non-empty; `memory_budget > 0`.
3. Every failure names the file, the query id, and the field.
4. `mem query validate <file>` prints the resolved set and exits non-zero on any
error.
5. Defaults apply per query and are overridable per query.
6. `id` collision across two files for the same project is an error.
## Acceptance
- An empty or missing `question` fails at load with a message naming the id.
- An id outside `[a-z0-9-]+` fails at load — it becomes a filename.
- Defaults resolve; per-query overrides win.
## Verify
**Harness:** table-driven over fixture YAML files, one per failure mode.
**Integration test** — `tests/it_query_loader.rs`:
1. `a1_valid_loads` — the reference file above resolves to the expected struct.
2. `a2_empty_question_rejected` — error text contains the query id.
3. `a3_duplicate_id_rejected` — error names both occurrences.
4. `a4_bad_id_charset_rejected` — `infra/root causes` is rejected, message
mentions the filename constraint.
5. `a5_defaults_and_overrides` — a query without `exit_gate` gets `false`; one
with `true` keeps it.
6. `a6_l1_exit_gate_defaults_false` — assert explicitly, because a silent flip to
`true` truncates every extraction and looks like a model quality problem.
7. `a7_missing_question_field` — absent key behaves as empty, same error.
**Command:** `cargo test --test it_query_loader`
**False pass:**
- Testing only the happy path. Every assertion except 1 and 5 is a rejection
test, and rejection is the entire point of load-time validation.
- Asserting an error occurred without asserting the message names the offending
id. "invalid config" sends someone to read the whole file by hand.
## Traps
- Allowing an empty question "for now". It loads, the gate has no referent, the
model updates on nearly everything, and it reads as a bad model rather than a
bad config.
- Letting `id` contain `/` or spaces. It is a path segment in three places.
---
Background: [DESIGN.md](../DESIGN.md) — Standing queries
tasks/M1.3-prompt-template.md
-144
View File
@@ -1,144 +0,0 @@
# M1.3 — GRU-Mem prompt template
| Field | Value |
|---|---|
| Phase | M1 — Gated loop at L1 |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M1.2 |
## Goal
Assemble the memory-agent prompt exactly as the paper specifies, because the
model's ability to emit parseable gates depends on the format it was aligned to.
## Files
| Action | Path |
|---|---|
| Create | `crates/mem-core/src/prompt.rs` — `PromptBuilder` struct |
| Modify | `crates/mem-core/src/lib.rs` — add `pub mod prompt;` |
| Create | `templates/gru-mem.txt` — the prompt template (verbatim from paper Fig 10a) |
| Create | `fixtures/expected/prompt-t1.txt` — golden file for turn 1 |
| Create | `fixtures/expected/prompt-tn.txt` — golden file for turn N |
| Create | `tests/it_prompt.rs` — integration tests (workspace root) |
## Dependencies
**None new.** No template engine — the prompt has 3 substitutions (`{prompt}`,
`{memory}`, `{chunk}`). Use `str::replace()` or `format!()`. Adding `tera` for
3 variables is overengineering.
## Existing code to reuse
- `Chunk` from `domain.rs` — render its `records` vec
- `Role` from `domain.rs` — map to `[User]`, `[Assistant]`, `[ToolResult]`, `[System]` labels
- `Query` from `query.rs` (M1.2) — read `query.question` for the `{prompt}` substitution
- `TokenCounter` from `mem-chunk` — check assembled prompt fits budget
## Facts (inlined — no spec read needed)
Paper Figure 10a, reproduced verbatim — this is the contract, not a starting
point to improvise on:
```
You are presented with a problem, a section of an article that may contain the
answer to the problem, and a previous memory. Please read the provided section
carefully. You should reason about whether the new section contains useful
information about the problem, and then update the memory with the new
information that helps to answer the problem.
Be sure to retain all relevant details from the previous memory while adding any
new, useful information. You should also carefully judge whether you have
collected enough information to answer the problem.
You should reason about whether the new section contains useful information, what
to update, and what to do next first between <think> and </think>.
If the new section contains useful information about the problem, you should
first generate <check>yes</check>. After that, update the new memory between
<update> and </update>.
If the new section does not contain useful information about the problem, you
should first generate <check>no</check>. After that, you should keep the previous
memory unchanged between <update> and </update>.
In the end, if you haven't collected enough information for the problem, return
<next>continue</next>. ONLY when enough information is collected, return
<next>end</next>.
<problem> {prompt} </problem>
<memory> {memory} </memory>
<section> {chunk} </section>
```
Substitutions for this system: `{prompt}` = the standing question, `{memory}` =
`M_{t-1}` or the literal `No previous memory` at `t=1` (the paper's own case
studies show that exact string), `{chunk}` = the rendered chunk.
Budget, against the 32768 cap:
```
system + template ~400
question ~100
memory <=1024
chunk <=5000
response 2048
------
~8600 headroom is comfortable
```
Chunk rendering: each record as `[role] text`, records separated by a blank line.
Role labels matter — the model uses them to tell a tool result from a decision.
## Steps
1. `PromptBuilder` in `mem-core` producing `(system, user)`.
2. Template verbatim as above. Any deviation gets a comment saying why.
3. `t=1` renders `No previous memory` — not empty, not `null`.
4. Render chunk records as `[role] text`, blank-line separated.
5. Assert the assembled prompt fits the budget before sending; over budget is an
error naming the component that overflowed, not a truncation.
6. `mem prompt --project P --query Q --chunk N` prints the exact prompt, for
eyeballing what the model actually sees.
## Acceptance
- Assembled prompt matches a committed golden file byte for byte.
- `t=1` contains `No previous memory`.
- Over-budget assembly errors and names the offending component.
## Verify
**Harness:** golden-file comparison. The prompt is a contract; a diff in it is a
change to the contract.
**Integration test** — `tests/it_prompt.rs`:
1. `a1_golden_t1` — first turn against `expected/prompt-t1.txt`, exact match.
2. `a2_golden_tn` — turn with a prior memory against `expected/prompt-tn.txt`.
3. `a3_no_previous_memory_literal` — assert the exact string at `t=1`.
4. `a4_all_tags_present` — `<problem>`, `<memory>`, `<section>` each appear
exactly once.
5. `a5_role_labels_rendered` — a chunk with all four roles renders all four
labels.
6. `a6_over_budget_errors` — a 20000-token chunk errors, message contains
`section`.
7. `a7_budget_headroom` — for the real fixture corpus, assert every assembled
prompt is under 32768 minus 2048.
**Command:** `cargo test --test it_prompt`
**False pass:**
- Asserting the prompt "contains" the question. A template that dropped the
`<check>` instructions still contains it, and the model then emits prose the
parser cannot read. Golden-file equality is the assertion that holds.
- Skipping assertion 7 by testing only small fixtures. Budget overflow appears
at p95 chunk size, not at the median.
## Traps
- Improving the wording. The 3B model's gate reliability comes from this exact
format; a cleaner rewrite is an unmeasured change to the one thing M1.8 gates on.
- Rendering an empty `<memory></memory>` at `t=1`. The paper's traces show
`No previous memory`, and an empty tag reads to the model as "memory exists and
is empty", which is a different claim.
---
Background: [DESIGN.md](../DESIGN.md) — Standing queries · paper Fig 10a
tasks/M1.4-gate-response-parser.md
-154
View File
@@ -1,154 +0,0 @@
# M1.4 — Gate-response parser
| Field | Value |
|---|---|
| Phase | M1 — Gated loop at L1 |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M1.3 |
## Goal
Turn the model's tagged output into `(U_t, M̂_t, E_t)`, strictly — because a
lenient parser silently fabricates gate decisions.
## Files
| Action | Path |
|---|---|
| Create | `crates/mem-core/src/gate_parser.rs` — `parse_gate_response()`, `GateResponse`, `ParseError` |
| Modify | `crates/mem-core/src/lib.rs` — add `pub mod gate_parser;` and re-exports |
| Create | `fixtures/gate-response-valid.txt` — well-formed model output |
| Create | `fixtures/gate-response-nested-think.txt` — nested `<think>` tags |
| Create | `tests/it_gate_parser.rs` — integration tests (workspace root) |
## Dependencies
**None new.** Use `str::find()` and `str::rfind()` for tag extraction. No regex
crate needed — the tags are simple XML-like delimiters, not a grammar.
## Existing code to reuse
- Pattern reference: `lesson.rs` uses similar string scanning for error markers
(`markers()`, `GENERIC_MARKERS`). Same technique, different tags.
- `thiserror` already in `mem-core` deps for error enum derivation.
## Expected input/output
```
Input:
<think>
This chunk shows a kubectl error. The user fixed it by adding --namespace.
</think>
<check>yes</check>
<update>kubectl get pods fails without --namespace; fixed by adding --namespace=kube-system</update>
<next>continue</next>
Output:
GateResponse {
think: "This chunk shows a kubectl error. The user fixed it by adding --namespace.",
update_gate: true,
candidate: "kubectl get pods fails without --namespace; fixed by adding --namespace=kube-system",
exit_gate: false,
}
```
## Facts (inlined — no spec read needed)
Expected response shape:
```
<think>...</think>
<check>yes|no</check>
<update>candidate memory, or the previous memory verbatim</update>
<next>continue|end</next>
```
Semantics, from the paper:
| tag | value | meaning |
|---|---|---|
| `<check>` | `yes` | `U_t = true` — memory becomes `M̂_t` |
| `<check>` | `no` | `U_t = false` — memory stays `M_{t-1}`, chunk discarded |
| `<next>` | `continue` | `E_t = false` |
| `<next>` | `end` | `E_t = true` |
**Strict parsing is the design, matching the paper's `r_format`:** it awards 1
only when *every* turn in the trajectory parses, 0 otherwise, "because we can not
infer whether the incorrect format is caused by the previous erroneous parsing".
So: exactly one of each tag, properly closed, `<check>` content exactly `yes` or
`no` after trimming, `<next>` exactly `continue` or `end`. Anything else is a
`ParseError` naming which tag failed and carrying the raw text.
**A parse failure must not default.** Defaulting `U_t` to `false` silently drops
evidence; defaulting to `true` pollutes memory. The loop (M1.5) decides the
retry policy; the parser only reports.
Reasoning models emit `<think>` natively, which can nest or repeat. Extract by
locating the *last* `</think>` before the first `<check>`, not by regex over the
whole body.
## Steps
1. `parse_gate_response(&str) -> Result<GateResponse, ParseError>` in `mem-core`.
2. `GateResponse { think: String, update_gate: bool, candidate: String, exit_gate: bool }`.
3. Reject duplicates of any tag, a missing tag, an unclosed tag, and any
`<check>`/`<next>` value outside the allowed set.
4. `ParseError` variants name the tag and include the raw response, truncated.
5. Trim surrounding whitespace inside tags; do not otherwise normalise — memory
text is preserved verbatim.
6. When `U_t = false`, still capture `candidate` so the log can show what the
model *would* have written. The loop ignores it; the record is diagnostic.
## Acceptance
- All four tags parse from a well-formed response.
- Every malformed shape errors, naming the failing tag.
- No input produces a default `U_t` or `E_t`.
## Verify
**Harness:** table-driven over recorded real responses plus hand-built malformed
cases. Capture real ones with `MEM_LLM_RECORD` from M1.1.
**Integration test** — `tests/it_gate_parser.rs`:
1. `a1_wellformed_yes_continue` — `U=true`, `E=false`, candidate matches.
2. `a2_wellformed_no_end` — `U=false`, `E=true`.
3. `a3_missing_check_errors` — error names `check`.
4. `a4_duplicate_update_errors` — two `<update>` blocks error.
5. `a5_bad_check_value_errors` — `<check>maybe</check>` errors, message shows the
value.
6. `a6_unclosed_tag_errors` — `<update>` never closed.
7. `a7_nested_think` — a response with `<think>` inside `<think>` still finds the
right boundary.
8. `a8_no_defaults` — property test over 1000 random mutations of a valid
response: every result is either an exact parse or an error, never a
silently-defaulted `GateResponse`.
9. `a9_real_responses` — every recorded real response parses.
**Command:** `cargo test --test it_gate_parser`
**False pass:**
- A regex that finds the first `<check>` and stops. It passes 1–2 and silently
accepts duplicates, which is assertion 4's job.
- Testing only hand-written responses. Real 3B output has whitespace,
markdown fences and stray prose the author would not think to write —
assertion 9 is the only one that sees it.
- Omitting assertion 8. A parser with `unwrap_or(false)` anywhere passes every
positive test and quietly halves the update rate.
## Traps
- Defaulting on parse failure. The one thing that must not happen: a fabricated
gate decision is indistinguishable from a real one in the log, and it poisons
the M5 training data at the source.
- Normalising the candidate memory (collapsing whitespace, stripping markdown).
Memory text is content; the hash and every downstream projection depend on it
being verbatim.
---
Background: [DESIGN.md](../DESIGN.md) — Architecture · paper §3.2.1 `r_format`
tasks/M1.5-gated-loop.md
-163
View File
@@ -1,163 +0,0 @@
# M1.5 — The gated loop
| Field | Value |
|---|---|
| Phase | M1 — Gated loop at L1 |
| Size | L — 3+ days |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M1.4 |
## Goal
The recurrence itself: `U_t, M̂_t, E_t = φθ(Q, C_t, M_{t-1})`, with the level as a
parameter so L2 reuses it unchanged.
## Files
| Action | Path |
|---|---|
| Create | `crates/mem-core/src/gated_loop.rs` — `run_loop()`, `LoopConfig`, `LoopEvent`, `RunOutcome` |
| Modify | `crates/mem-core/src/lib.rs` — add `pub mod gated_loop;` and re-exports |
| Create | `tests/it_gated_loop.rs` — integration tests (workspace root, 10 assertions) |
## Dependencies
| Crate | Where | Already present? |
|---|---|---|
| `async-trait` | `crates/mem-core/Cargo.toml` | ❌ add — for `LlmClient` trait |
Or use `impl Future` return types and avoid the dependency.
## Existing code to reuse
- `Chunk`, `Level`, `Sha256Hash` from `domain.rs` — input/output types
- `Query` from `query.rs` (M1.2) — the standing question
- `PromptBuilder` from `prompt.rs` (M1.3) — assemble the prompt per turn
- `parse_gate_response` from `gate_parser.rs` (M1.4) — parse LLM output
- `ChatClient` from `mem-llm/src/chat.rs` (M1.1) — call the LLM
- `TokenCounter` from `mem-chunk/src/token_counter.rs` — measure candidate memory tokens
## Dependency injection
The loop needs an LLM client, but tests must use a scripted fake. Define a trait:
```rust
// in gated_loop.rs
pub trait LlmClient: Send + Sync {
fn complete(&self, system: &str, user: &str, max_tokens: usize)
-> impl std::future::Future<Output = anyhow::Result<mem_llm::Completion>> + Send;
}
```
`ChatClient` implements it. Tests use a `ScriptedClient` that returns canned
responses indexed by turn number.
## Facts (inlined — no spec read needed)
Paper Algorithm 1, transcribed:
```
t <- 1; M_0 <- None
while t <= T:
U_t, M̂_t, E_t = φθ(Q, C_t, M_{t-1})
if U_t == True: M_t <- M̂_t # update
else: M_t <- M_{t-1} # retain, discard chunk
if use_exit_gate and E_t == True: break
t <- t + 1
answer = ψθ(Q, M_t)
```
Two decisions this task must not get wrong:
**`use_exit_gate` is a parameter, false at L1.** `E_t` is always *recorded*
regardless — its signal is the M5 training target, and on a future unbounded
stream it becomes the only termination condition. Recording a gate you do not act
on is deliberate, not dead code.
**The memory budget is enforced by the loop, not hoped for from the model.**
`memory_budget` is 1024 tokens. If `M̂_t` exceeds it the loop does **not** silently
truncate — truncation mid-sentence corrupts the memory for every later turn. It
records a `budget_exceeded` event and retains `M_{t-1}`, treating the turn as
`U_t = false`. Memory that stops growing is recoverable; memory that is
truncated garbage is not.
The loop is generic over the input stream, so L2 (M3.1) passes L1 memories in
place of chunks with no other change.
## Steps
1. `run_loop(level, query, source: impl Stream<Item = Chunk>, cfg) -> RunOutcome`
in `mem-core`.
2. Per turn: build prompt (M1.3), call model (M1.1), parse (M1.4), apply the
update rule, emit events.
3. On parse error: retry the same chunk up to 2 times. Still failing, record
`parse_failed`, treat as `U_t = false`, continue. **Never** default the gate.
4. On `U_t = true`: emit an L0 `evidence` event for the chunk, then an L1
`memory` event whose `parents` include that evidence sha plus the previous
memory's sha.
5. Enforce `memory_budget` as above.
6. Honour `E_t` only when `use_exit_gate`; always record it.
7. Emit `run_end` with `chunks_seen`, `chunks_used`, `final_memory_sha`.
8. Cancellation: a dropped future must not leave a half-written log. Emit events
only after a turn fully resolves.
## Acceptance
- `U_t = false` leaves memory byte-identical to the previous turn.
- `U_t = true` replaces memory and links parents.
- `exit_gate = false` processes every chunk even when `E_t = true` throughout.
- Over-budget candidate retains prior memory and records the event.
- Two parse failures then success consumes 3 calls for one chunk.
## Verify
**Harness:** a scripted fake `ChatClient` returning canned responses per turn, so
the whole loop runs with no network and fully determined gate sequences.
**Integration test** — `tests/it_gated_loop.rs`:
1. `a1_retain_on_no` — scripted `no` for 5 turns; assert final memory equals
initial and `chunks_used == 0`.
2. `a2_update_on_yes` — `yes` at turn 3 only; assert memory equals turn 3's
candidate and `chunks_used == 1`.
3. `a3_exit_gate_off_reads_all` — `end` at every turn with `use_exit_gate=false`;
assert all 10 chunks processed.
4. `a4_exit_gate_on_stops` — same script, `use_exit_gate=true`; assert it stops
at turn 1.
5. `a5_exit_always_recorded` — in a3, assert 10 `gate` events carry `exit=true`
despite not acting on them.
6. `a6_budget_exceeded_retains` — candidate of 4000 tokens; assert memory
unchanged, one `budget_exceeded` event, turn counted as not-used.
7. `a7_parse_retry` — fail twice then succeed; assert 3 calls, one memory event.
8. `a8_parse_failure_is_not_an_update` — fail 3 times; assert `U_t` false,
`parse_failed` recorded, loop continues.
9. `a9_parents_linked` — every memory event's `parents` contains the evidence sha
from the same turn.
10. `a10_level_is_a_parameter` — run the identical script at L1 and L2; assert the
only difference in emitted events is the `level` field.
**Command:** `cargo test --test it_gated_loop`
**False pass:**
- Testing with a fake that always returns `yes`. Every assertion about the retain
path is skipped, and retain is the path that matters — it is what makes this a
gate rather than a summarizer.
- Asserting `chunks_used` without asserting memory bytes. A loop that updates
memory on `no` but counts correctly passes a count-only test; assertion 1
compares bytes.
- Omitting assertion 10. Without it, L2 in M3.1 becomes a copy of this loop, and
the two drift.
## Traps
- Truncating over-budget memory to fit. It corrupts every subsequent turn's
input, and the damage is attributed to the model.
- Acting on `E_t` at L1 because "the model said it had enough". Paper §3.3 is
explicit that this is wrong for exhaustive questions, and it silently truncates
extraction in a way that looks like poor recall.
- Writing log events before the turn resolves. A cancelled run then leaves a
memory event with no matching gate event and `mem verify` fails on a file that
was merely interrupted.
---
Background: [DESIGN.md](../DESIGN.md) — Architecture, tier model · paper Alg 1
tasks/M1.6-jsonl-event-log.md
-141
View File
@@ -1,141 +0,0 @@
# M1.6 — JSONL event log writer
| Field | Value |
|---|---|
| Phase | M1 — Gated loop at L1 |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M1.5 |
## Goal
Write the authoritative record — the one artifact everything else is derived
from, and the one that must survive a crash mid-run.
## Files
| Action | Path |
|---|---|
| Create | `crates/mem-store/src/event_log.rs` — `LogWriter`, `LogReader`, `RunStatus` |
| Replace | `crates/mem-store/src/lib.rs` — replace `pub mod placeholder {}` with `pub mod event_log;` + re-exports |
| Create | `tests/it_event_log.rs` — integration tests (workspace root, 8 assertions) |
## Dependencies
| Crate | Where | Already present? |
|---|---|---|
| `ulid` or `rusty_ulid` | `crates/mem-store/Cargo.toml` | ❌ add — for sortable run IDs |
| `tokio` (fs feature) | `crates/mem-store/Cargo.toml` | ✅ yes |
| `serde`, `serde_json` | `crates/mem-store/Cargo.toml` | ✅ yes |
## Existing code to reuse
- `LoopEvent` from `gated_loop.rs` (M1.5) — the events to serialize
- `Level` from `domain.rs` — carried in every record
- Pattern reference: `lessons_cmd.rs` has a JSONL writer for `Event` types (`events.jsonl`).
Same concept but **different event schema** and **different storage location**:
- `lessons_cmd.rs` writes `~/.mem/events.jsonl` (command execution events)
- M1.6 writes `log/<project>/<query>/<run-id>.jsonl` (gate decision events)
- Do not unify them. They serve different purposes.
## Output path convention
```
log/
poimen/
tool-failures/
01HXYZ....jsonl ← ULID, lexicographically sortable by time
architecture-decisions/
01HXYZ....jsonl
```
## Facts (inlined — no spec read needed)
Path: `log/<project>/<query-id>/<run-id>.jsonl`. Append-only, one object per line.
**Every record carries `level`.**
```jsonl
{"type":"run","level":"L1","project":"poimen","query_id":"infra-root-causes","input_level":"chunk","model":"qwen2.5:3b-instruct","chunk_tokens":5000,"memory_budget":1024,"exit_gate":false,"ts":"..."}
{"type":"chunk","level":"L0","t":1,"source":"pi:...","span":[0,42],"sha256":"..."}
{"type":"gate","level":"L1","t":1,"update":false,"exit":false,"think":"...","latency_ms":812}
{"type":"evidence","level":"L0","t":7,"source":"pi:...","text":"...","sha256":"..."}
{"type":"memory","level":"L1","t":7,"text":"...","tokens":142,"parents":["<sha>"],"sha256":"..."}
{"type":"run_end","level":"L1","chunks_seen":412,"chunks_used":17,"final_memory_sha":"..."}
```
`evidence` appears **only** when the update gate opened. That is what makes
update-rate directly measurable from the log — `gate` records give the
denominator, `evidence` records the numerator.
This log is authoritative: the vault and pgvector are projections rebuilt from
it. Two consequences — it is tracked in git, and it is never rewritten in place.
A run that crashes leaves a file with no `run_end`. That is a valid, readable
state meaning "incomplete", not corruption. Readers must handle it.
## Steps
1. `LogWriter::open(project, query_id, run_id)` in `mem-store`, creating parents.
2. `append(event)` serializes one line and **flushes**. An unflushed buffer loses
the last turns of exactly the run you want to debug.
3. `run_id` is a ULID — lexicographically sortable by creation time, so listing
runs in order is a directory sort.
4. `LogReader` streams events back, tolerating a truncated final line.
5. `replay_memory_at(t)` reconstructs `M_t` from the events alone, proving the log
is sufficient.
6. `stats()` computes chunks seen/used and update-rate from a log file.
7. Never open in truncate mode. Append only.
## Acceptance
- Every emitted record has a `level`.
- `evidence` count equals the count of `gate` records with `update: true`.
- A file with no `run_end` reads cleanly and reports `incomplete`.
- `replay_memory_at(t)` matches the memory the loop held at `t`.
## Verify
**Harness:** the scripted loop from M1.5 writing to a temp dir, plus a corrupted
fixture.
**Integration test** — `tests/it_event_log.rs`:
1. `a1_every_record_has_level` — parse every line, assert `level` present and in
`{L0,L1,L2}`.
2. `a2_evidence_matches_update_gates` — count `gate.update==true`, assert equal to
the `evidence` count.
3. `a3_replay_equals_live` — for every `t`, `replay_memory_at(t)` equals the
memory the loop held. This is the assertion that proves the authority model.
4. `a4_truncated_tail_reads` — chop the last line mid-object; assert all prior
events parse and the run reports `incomplete`.
5. `a5_no_run_end_is_incomplete` — a log ending after a `memory` event reports
incomplete, not an error.
6. `a6_append_only` — write, reopen, write again; assert the first events survive.
7. `a7_flush_per_event` — kill the process (or drop without close) after 3
appends; assert 3 lines on disk.
8. `a8_run_id_sorts_by_time` — three runs, assert lexicographic order equals
chronological order.
**Command:** `cargo test --test it_event_log`
**False pass:**
- Asserting the file parses. A writer that omits `evidence` events entirely
produces a perfectly parseable log with an update-rate of zero — assertion 2 is
what catches it.
- Testing replay only at the final `t`. A writer that records only the final
memory passes that and fails assertion 3 at every intermediate turn.
- Omitting assertion 7. Buffered writes pass every test that closes the file
properly, and lose data in precisely the crash case the log exists for.
## Traps
- Opening with truncate. One accidental re-run erases the authoritative record,
and the projections are the only surviving copy — inverted authority.
- Gitignoring `log/`. Makes the whole "JSONL is authoritative" claim a fiction.
`agent-rust/.gitignore` has a bare `tasks` entry that untracks its whole board;
do not repeat it here.
---
Background: [DESIGN.md](../DESIGN.md) — Storage schemas, authority model
tasks/M1.7-ingest-end-to-end.md
-152
View File
@@ -1,152 +0,0 @@
# M1.7 — `mem ingest` end to end
| Field | Value |
|---|---|
| Phase | M1 — Gated loop at L1 |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M1.6 |
## Goal
One command that reads a real project and produces a real log — and reports the
number that says whether the gate works.
## Files
| Action | Path |
|---|---|
| Modify | `crates/mem-cli/src/main.rs` — extend `Commands::Ingest` with `--query` and `--resume` flags; replace stub `cmd_ingest()` body with real pipeline |
| Create | `tests/it_ingest.rs` — integration tests (workspace root, 7 assertions) |
## Dependencies
**None new.** All crates already depend on what they need. This task wires existing
pieces together.
## Existing code to reuse
- `cmd_ingest()` in `main.rs` — **replace the stub body**, keep the CLI struct
- `PiSessionSource` from `mem-ingest/src/pi_session.rs` — already works (M0.5)
- `ClaudeTranscriptSource` from `mem-ingest/src/claude_transcript.rs` — already works (M0.6)
- `chunks()` from `mem-chunk/src/chunker.rs` — already works (M0.3)
- `ChunkPolicy` from `mem-chunk/src/chunk_policy.rs` — already works
- `QuerySet::load()` from `mem-core/src/query.rs` — from M1.2
- `run_loop()` from `mem-core/src/gated_loop.rs` — from M1.5
- `LogWriter` from `mem-store/src/event_log.rs` — from M1.6
- `ChatClient` from `mem-llm/src/chat.rs` — from M1.1
## Wiring diagram
```
CLI: mem ingest --project poimen --query tool-failures
│
├─ QuerySet::load("queries/poimen.yaml") ← M1.2
│ └─ query = set.by_id("tool-failures")
│
├─ PiSessionSource::new(session_files) ← M0.5 (existing)
│ └─ source.records() → Stream<Record>
│
├─ chunks(source, policy) ← M0.3 (existing)
│ └─ Stream<Chunk>
│
├─ ChatClient::new(base_url, api_key, model) ← M1.1
│
├─ run_loop(L1, query, chunks, client, config) ← M1.5
│ └─ RunOutcome { events, chunks_seen, chunks_used, ... }
│
├─ LogWriter::open(project, query, run_id) ← M1.6
│ └─ write events to log/poimen/tool-failures/<ulid>.jsonl
│
└─ Print summary:
chunks 412 used 17 update-rate 4.1% memory 142tok elapsed 6m12s
```
## Facts (inlined — no spec read needed)
```
mem ingest --project poimen --query infra-root-causes
mem ingest --project poimen # all queries in the set
mem ingest --project poimen --limit 50 # first 50 chunks, for iterating
mem ingest --project poimen --resume # skip chunks already in the log
```
Progress output, because a run that prints nothing cannot be distinguished from
one that has hung:
```
[ 17/412] t=17 update=yes mem=142tok 1.9s
[ 18/412] t=18 update=no mem=142tok 0.8s
...
run 01HXYZ chunks 412 used 17 update-rate 4.1% memory 142tok elapsed 6m12s
```
**Update-rate is the headline number.** Tool results are ~43% of records and
mostly evidence-free; a correct gate rejects the large majority of chunks. A rate
above ~30% means the gate is not discriminating and the run is an expensive
summarizer — that is the paper's memory-explosion failure and it is what M1.8
gates on.
Runs are long. 412 chunks at ~1–2s each is 6–14 minutes per query, and every
chunk costs a model call, so `--resume` is not a nicety.
## Steps
1. Wire adapters (M0.5/M0.6) → chunker (M0.3) → loop (M1.5) → log (M1.6).
2. Per-chunk progress line to stderr; summary to stdout so it pipes cleanly.
3. Report update-rate in the summary and as `--format json`.
4. `--resume` reads the existing log, finds the highest `t` with a `gate` record,
and restarts from `t+1` with that turn's memory.
5. `--limit` caps chunks processed.
6. Exit non-zero if the run did not reach `run_end`.
7. Ctrl-C finishes the in-flight turn, writes `run_end`, exits — no half-turn.
## Acceptance
- A real project produces a complete log with `run_end`.
- Reported update-rate equals the value computed independently from the log.
- `--resume` on a complete log is a no-op; on a partial one it continues.
- Interrupt produces a valid log.
## Verify
**Harness:** scripted client for determinism, plus one live `#[ignore]` run.
**Integration test** — `tests/it_ingest.rs`:
1. `a1_produces_complete_log` — scripted run; assert `run_end` present and event
counts match the script.
2. `a2_update_rate_matches_log` — compare the reported rate to
`LogWriter::stats()` recomputed from the file.
3. `a3_resume_is_noop_when_complete` — run, resume, assert zero additional model
calls.
4. `a4_resume_continues_partial` — truncate a log after t=10, resume, assert the
next call is t=11 and memory at t=11 equals the replayed memory at t=10.
5. `a5_interrupt_is_clean` — send SIGINT mid-run; assert the log parses, has
`run_end`, and the last `gate` has a matching `memory`-or-not decision.
6. `a6_limit_respected` — `--limit 5` produces exactly 5 gate records.
7. `a7_live_smoke` — `#[ignore]`; real gateway, `--limit 20` on a real project;
assert `run_end` and **print** the update-rate for a human to read.
**Command:** `cargo test --test it_ingest` (add `-- --ignored` for a7)
**False pass:**
- Asserting only that the command exits 0. A run whose gate always answers `no`
exits 0, writes a valid log, and has learned nothing — the update-rate is the
only thing that distinguishes it, which is why a7 prints it rather than
merely asserting a run happened.
- Resuming by counting lines rather than reading the highest `t` with a `gate`
record. Line counts break the moment an `evidence` record is present, i.e. as
soon as the gate ever opened.
## Traps
- No progress output. A 14-minute run that prints nothing is indistinguishable
from a hang, and the first instinct will be to kill it.
- Resume that replays from `t=1` with the old memory. It costs a full run and
produces a log with duplicate turns that `mem verify` will reject.
---
Background: [DESIGN.md](../DESIGN.md) — Verification, P2
tasks/M1.8-m1-gate.md
-118
View File
@@ -1,118 +0,0 @@
# M1.8 — M1 composition gate
| Field | Value |
|---|---|
| Phase | M1 — Gated loop at L1 |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | gate |
| Spec | inlined below |
| Blocks | all of M1 |
## Goal
Answer the only question that matters at this stage: **did we build a gate, or an
expensive summarizer?**
## Files
| Action | Path |
|---|---|
| Create | `tests/it_m1_gate.rs` — integration tests (workspace root, all `#[ignore]`, 7 assertions) |
| Create | `fixtures/expected/m1-gate.txt` — committed expectation file (summary table) |
## Dependencies
**None new.** This task runs existing code against the live gateway.
## Existing code to reuse
The entire M1 pipeline:
- `QuerySet::load()` (M1.2), `ChatClient` (M1.1), `run_loop()` (M1.5), `LogWriter` (M1.6)
- `PiSessionSource` / `ClaudeTranscriptSource` (M0.5/M0.6) — existing
- `chunks()` (M0.3) — existing
- `LogReader::stats()` (M1.6) — to recompute update-rate independently
## Facts (inlined — no spec read needed)
This gate runs against the **live gateway on a real project** and asserts
properties of the resulting log. It is the first phase that costs money in wall
clock, and the first that can fail for reasons no unit test can see.
Two thresholds, both from the paper:
**Update-rate < 30%.** Agent transcripts are ~43% tool results, mostly
evidence-free. Paper Figure 6 shows the failure mode directly: the ungated
MemAgent's memory climbs to its 1024-token ceiling and saturates, after which
"the accumulated noise can further impede subsequent updates". A high update-rate
is that curve starting.
**Memory size flat, not climbing.** Plot `memory.tokens` against `t`. GRU-Mem's
curve is low and roughly flat; the ungated curve rises to the cap and stays
pinned. A monotonic climb means the gate is open too often even if the rate
looks acceptable.
Third property, cheap and load-bearing: **the same run twice produces the same
chunk hashes**. Temperature affects the model's text, not the chunking; if chunk
shas differ between runs, identity includes something it should not (M0.2) and
every projection will churn.
## Steps
1. Run `mem ingest --project poimen` for all standing queries against the live
gateway.
2. Compute per query: update-rate, memory-token series, parse-failure count,
elapsed.
3. Assert the thresholds below.
4. Emit `expected/m1-gate.txt` with the summary table; commit it. Subsequent runs
diff against it, and a changed expectation is a reviewable claim.
5. Sample 20 gate decisions and have the 32B `reasoning` model audit them; report
agreement. Advisory at this gate, and the seed of M5.2's calibration.
## Acceptance
- Update-rate < 30% on every standing query.
- Memory tokens ≤ 1024 and not monotonically increasing.
- Parse-failure rate < 5%.
- Chunk shas stable across two runs.
## Verify
**Harness:** live gateway. Long-running; a nightly or on-demand job, not
per-push.
**Integration test** — `tests/it_m1_gate.rs`, all `#[ignore]` by default:
1. `a1_update_rate_under_threshold` — per query, assert < 0.30, print actual.
2. `a2_memory_bounded` — every `memory.tokens` ≤ 1024.
3. `a3_memory_not_climbing` — fit a line to tokens vs `t`; assert the slope is
below a small positive bound. Do not assert non-increasing — a legitimately
growing memory rises early then plateaus.
4. `a4_parse_failure_rate` — `parse_failed` / turns < 0.05.
5. `a5_chunk_sha_stable` — two runs, same chunk shas in the same order.
6. `a6_evidence_traceable` — every `evidence` sha appears as a parent of some
`memory` record.
7. `a7_judge_audit` — sample 20 decisions, ask the 32B model, print agreement.
Advisory; does not fail the gate.
**Command:** `cargo test --test it_m1_gate -- --ignored --nocapture`
**False pass:**
- Running the gate on a tiny `--limit`. Update-rate on the first 20 chunks is
noise; the failure mode is cumulative and needs a full project.
- Asserting the rate without printing it. The number is the artifact — a run at
29% passes and is telling you something a boolean hides.
- Asserting memory is non-increasing rather than bounded-slope. Real memory does
grow early, and a strict assertion here fails on correct behaviour, which
trains everyone to skip the gate.
## Traps
- Treating a high update-rate as a model-quality problem first. Check the
prompt (M1.3 golden file) and the parser (M1.4 defaults) before blaming the
3B model — a parser with `unwrap_or(true)` produces exactly this symptom.
- Tuning the threshold to whatever the first run produced. 30% comes from the
corpus composition; moving it to accommodate a bad result deletes the gate.
---
Background: [DESIGN.md](../DESIGN.md) — Verification · paper Fig 6, §4.2
tasks/M3.1-l2-synthesis.md
-102
View File
@@ -1,102 +0,0 @@
# M3.1 — L2 synthesis pass
| Field | Value |
|---|---|
| Phase | M3 — L2 synthesis and retrieval |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M1.5 |
## Goal
Project-level memory across the per-query memories — using the same loop, with
the exit gate switched on.
## Facts (inlined — no spec read needed)
```
mem synthesize --project poimen
```
L2 is **not new machinery**. It is `run_loop` (M1.5) with:
| | L1 | L2 |
|---|---|---|
| input stream | `Chunk` from sources | L1 `MemoryNode`s |
| question | per-query question | `synthesis.question` |
| `use_exit_gate` | false | **true** |
| `query_id` | set | NULL |
| parents | L0 evidence shas | L1 memory shas |
**Why the exit gate flips on.** At L1 the input is hundreds of chunks and the
question is exhaustive ("what are *all* the X"), which is exactly the case paper
§3.3 says to run without the gate. At L2 the input is a handful of memories and
"enough evidence" is genuinely decidable, which is the case the gate was designed
for and where the paper measures its 4× speedup.
If M1.5 was written correctly this task is mostly wiring. If it needs changes to
`run_loop`, the level was not really a parameter — and assertion a10 in M1.5
existed to prevent exactly that.
Ordering: L1 memories enter the stream in a stable order (query id, ascending),
so synthesis is reproducible.
## Steps
1. `mem synthesize --project P` reads the final L1 memory per standing query.
2. Wrap them as the loop's input stream, in sorted query-id order.
3. Run `run_loop` with `level = L2`, `use_exit_gate = true`, the synthesis
question, `query_id = None`.
4. Write to `log/<project>/_synthesis/<run-id>.jsonl`.
5. `parents` on the L2 memory are the L1 memory shas consumed up to that turn.
6. Refuse to run if any standing query has no completed L1 run — synthesizing
over a partial set silently produces a partial picture.
## Acceptance
- No change to `run_loop` is required.
- The exit gate fires and stops early on a real project.
- L2 parents are L1 shas, never L0.
- Sorted input order makes two runs consume memories in the same sequence.
## Verify
**Harness:** scripted client, plus one live run.
**Integration test** — `tests/it_l2.rs`:
1. `a1_reuses_run_loop` — assert `mem synthesize` calls the same `run_loop`
symbol; a duplicated loop is a review failure, and a `#[deny]`-style test here
is a grep asserting `fn run_loop` appears exactly once in the workspace.
2. `a2_exit_gate_on` — scripted `end` at turn 2 of 5; assert it stops at 2.
3. `a3_parents_are_l1` — every L2 parent sha resolves to an L1 node.
4. `a4_query_id_null` — the L2 record has no `query_id`.
5. `a5_stable_input_order` — two runs consume L1 memories in identical order.
6. `a6_refuses_partial` — one query with no completed run; assert non-zero exit
naming the query.
7. `a7_level_check_holds` — run `mem verify`; invariant 6 (level consistency)
passes.
8. `a8_live` — `#[ignore]`; real project, assert an L2 memory is produced and
print whether the exit gate fired and at which turn.
**Command:** `cargo test -p mem-cli l2` (add `-- --ignored` for a8)
**False pass:**
- Copying `run_loop` into an L2-specific function. Everything passes, the two
drift within a month, and the tier model quietly becomes two implementations.
Assertion 1 is the guard.
- Testing the exit gate with a script that never says `end`. The gate's effect is
invisible and a `use_exit_gate` that is ignored passes.
## Traps
- Feeding L0 evidence into L2 "for more detail". It blows the context budget and
breaks the level invariant; L2 reads memories, and if they are inadequate the
fix is at L1.
- Synthesizing over whatever L1 runs happen to exist. A missing query produces a
confident summary of an incomplete project, which is worse than no summary.
---
Background: [DESIGN.md](../DESIGN.md) — tier model · paper §3.3
tasks/M3.2-rerank-client.md
-88
View File
@@ -1,88 +0,0 @@
# M3.2 — Rerank client
| Field | Value |
|---|---|
| Phase | M3 — L2 synthesis and retrieval |
| Size | S — under 1 day |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M1.1 |
## Goal
Reorder vector-recall candidates by actual relevance, because embedding distance
is a coarse filter.
## Facts (inlined — no spec read needed)
```
POST /v1/rerank
{"query":"what is rust","texts":["rust is a language","bananas"]}
-> [{"index":0,"score":0.98176736},{"index":1,"score":0.00008833366}]
```
`BAAI/bge-reranker-base` via TEI. Note the **response shape is a bare array**, not
an OpenAI-style `{"data": [...]}` envelope — this route does not follow the chat
convention.
The discrimination is real: 0.98 vs 0.00009 on that probe, four orders of
magnitude. Embedding cosine on the same pair would be far closer, which is why
recall-then-rerank beats recall alone.
`index` refers to the position in the submitted `texts` array; results come back
**sorted by score**, so the index is the only way to map back. Do not assume
order.
Batch limits apply as with embeddings — keep candidate lists modest (top-50 from
recall is plenty).
## Steps
1. `RerankClient::rerank(query, texts) -> Result<Vec<Scored>>` in `mem-llm`.
2. Parse the bare array; map `index` back to the caller's items.
3. Preserve the caller's item type: take `&[T]`, return `Vec<(T, f32)>` so the
caller does not re-associate by position.
4. Same `apikey` header, retry and timeout policy as M1.1.
5. Empty input returns empty without a request.
## Acceptance
- Results map correctly back to input items via `index`.
- A more relevant text scores above a less relevant one on a live call.
- Empty input makes no request.
## Verify
**Harness:** `wiremock` offline, one `#[ignore]` live test.
**Integration test** — `tests/it_rerank.rs`:
1. `a1_bare_array_parsed` — mock returns `[{"index":1,...},{"index":0,...}]`;
assert parsing succeeds.
2. `a2_index_mapping` — with the out-of-order mock above, assert the returned
items correspond to inputs 1 and 0 respectively, not 0 and 1.
3. `a3_empty_no_request` — empty texts; assert zero requests.
4. `a4_apikey_sent` — header present.
5. `a5_live_discriminates` — `#[ignore]`; real gateway, query "what is rust"
against `["rust is a language","bananas"]`; assert the first scores at least
10× the second.
**Command:** `cargo test -p mem-llm rerank` (add `-- --ignored` for a5)
**False pass:**
- Assuming the response preserves input order. A mock that returns results in
input order passes a naive test and the live endpoint returns them sorted,
which silently mislabels every result. Assertion 2 must use an out-of-order
mock.
- Expecting an OpenAI envelope. It will fail immediately against the live route,
but a mock written to match the wrong shape hides that until integration.
## Traps
- Re-associating results by position instead of by `index`. The scores are right
and attached to the wrong documents — a bug that looks like poor retrieval
quality rather than a mapping error.
---
Background: [DESIGN.md](../DESIGN.md) — Verified facts, retrieval
tasks/M3.3-mem-query.md
-97
View File
@@ -1,97 +0,0 @@
# M3.3 — `mem query` with provenance
| Field | Value |
|---|---|
| Phase | M3 — L2 synthesis and retrieval |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M3.2, M2.4 |
## Goal
Ask the memory a question and get an answer that can be traced back to the
session it came from.
## Facts (inlined — no spec read needed)
```
mem query "why did requests over 10KB fail?"
mem query --project poimen --levels L1,L2 --k 10 "..."
```
Pipeline: embed the question (M2.1) → HNSW recall top-k over `memory_node`
filtered by project and level (M2.4) → rerank the candidates (M3.2) → return with
provenance walked down through `memory_edge`.
Default levels are **L1 and L2**, not L0. L1/L2 are synthesized answers; L0 is raw
evidence and returning it by default buries the answer in transcript. `--levels
L0` exists for "show me the actual source".
Provenance walk: for each returned node, follow `memory_edge` to its parents and
report source and turn. One hop for L1 (to evidence), two for L2 (through L1).
Recall wide, rerank narrow: take 50 from HNSW, rerank, return 5. Cosine distance
alone puts "bananas" close enough to matter; the reranker separated the same pair
by four orders of magnitude.
Output is human-readable by default, `--format json` for programmatic use.
## Steps
1. `mem query [--project P] [--levels L] [--k N] <question>`.
2. Embed, recall 10×k, rerank, truncate to k.
3. Walk edges to build a provenance list per hit; deduplicate sources.
4. Render: score, level, query id, the memory text, then provenance lines.
5. Project defaults to the one inferred from `$PWD`; `--project` overrides. No
project and no match is an error, not an empty result over everything.
6. `--explain` prints the recall candidates and their pre-rerank distances, for
debugging retrieval quality.
## Acceptance
- A question with a known answer returns the right L1 note first.
- Every hit carries at least one resolvable provenance entry.
- `--levels L0` returns evidence; the default does not.
- Reranking changes the order versus raw recall on at least one real query.
## Verify
**Harness:** a seeded database from a real log, deterministic fake embedder for
the offline assertions, live for quality.
**Integration test** — `tests/it_query.rs`:
1. `a1_known_answer` — seeded with the poimen log, query "why did requests over
10KB fail?"; assert the top hit is the `infra-root-causes` L1 node.
2. `a2_provenance_resolves` — every hit's provenance shas exist in
`memory_node`.
3. `a3_default_excludes_l0` — assert no L0 nodes in default output.
4. `a4_levels_flag` — `--levels L0` returns evidence nodes.
5. `a5_rerank_reorders` — capture pre- and post-rerank order; assert they differ
on at least one fixture query, proving the reranker is wired and not a no-op.
6. `a6_project_isolation` — two projects seeded; assert no cross-project hits.
7. `a7_no_project_errors` — unresolvable project exits non-zero.
8. `a8_l2_two_hop_provenance` — an L2 hit's provenance resolves through L1 to L0
sources.
**Command:** `cargo test -p mem-cli query`
**False pass:**
- Asserting only that results are returned. A pipeline where the reranker returns
the input unchanged still returns results, and assertion 5 is the only check
that it is doing anything.
- Testing provenance existence without resolving it. A hit carrying parent shas
that do not exist in the table looks fine in the output and is useless.
## Traps
- Returning L0 by default. The answer is there but buried in raw transcript, and
the tool reads as low quality when it is a display default.
- Recalling exactly k then reranking. Reranking cannot recover a relevant
document that recall never returned; the width of the recall is what determines
the ceiling.
---
Background: [DESIGN.md](../DESIGN.md) — pgvector, retrieval
tasks/M3.4-m3-gate.md
-93
View File
@@ -1,93 +0,0 @@
# M3.4 — M3 composition gate
| Field | Value |
|---|---|
| Phase | M3 — L2 synthesis and retrieval |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | gate |
| Spec | inlined below |
| Blocks | all of M3 |
## Goal
Prove the whole stack answers a real question with real provenance — the first
point at which the system is useful rather than merely correct.
## Facts (inlined — no spec read needed)
The gate is a small set of **known-answer questions** with hand-written expected
sources, committed to the repo. Each names a fact that genuinely appears in the
ingested sessions and the session it appears in.
Seed set, all drawn from real work in this corpus:
| question | expected to cite |
|---|---|
| why did requests over 10KB fail? | the Kong body-buffer / `client_body_buffer_size` finding |
| why did `Authorization: Bearer` return 401? | the Kong key-auth header finding |
| what causes the 504 on a cold ornith start? | the timeouts-on-Ingress-vs-Service finding |
This is a retrieval quality gate, so it is graded, not boolean: report
**hit rate at k=5** and **provenance precision** (fraction of cited sources that
actually contain the fact). A single failing question is information, not
necessarily a stop.
The gate also re-runs `mem verify` including the level invariant, because M3.1 is
the change most likely to break it.
## Steps
1. `verify/known-answers.yaml` — question, expected node, expected source.
2. `verify/m3.4.sh` runs each through `mem query --format json`.
3. Compute hit rate at 5 and provenance precision; print both.
4. Assert the thresholds below.
5. Run `mem verify`; assert zero violations.
6. Assert L2 exists and its provenance resolves two hops.
7. Commit `expected/m3.4.txt`; diff.
## Acceptance
- Hit rate at k=5 ≥ 0.8 on the known-answer set.
- Provenance precision ≥ 0.9 — a citation that does not contain the fact is worse
than no citation.
- `mem verify` clean, including level consistency.
- Every L2 hit resolves to L0 sources.
## Verify
**Harness:** live gateway, seeded database from real logs. On-demand, not
per-push.
**Integration test** — `verify/m3.4.sh` diffed against `expected/m3.4.txt`:
1. `a1_hit_rate` — ≥ 0.8, print actual.
2. `a2_provenance_precision` — for each cited source, fetch the L0 text and
assert it contains the expected fact substring; ≥ 0.9.
3. `a3_verify_clean` — zero violations.
4. `a4_l2_two_hop` — every L2 hit resolves through L1 to a real source.
5. `a5_rerank_contributes` — hit rate with reranking is ≥ hit rate without. If
reranking makes it worse, the wiring is wrong (probably index mapping, M3.2).
6. `a6_no_cross_project` — a question about another project returns nothing from
this one.
**Command:** `bash verify/m3.4.sh | diff - expected/m3.4.txt`
**False pass:**
- Writing the known-answer set after seeing what the system returns. It then
measures nothing. Write the questions and expected sources from the sessions
first, independently of any query output.
- Measuring hit rate without provenance precision. A system that returns the
right note with fabricated citations scores 1.0 on hits and is untrustworthy —
assertion 2 is the one that matters for whether anyone can act on an answer.
## Traps
- Tuning `k` until the hit rate passes. k=50 will hit almost everything and the
metric stops meaning anything; the gate specifies k=5 for that reason.
- Accepting a5 failing as "reranker is just not helping". It far more often means
the `index` mapping in M3.2 is wrong and scores are attached to the wrong
documents.
---
Background: [DESIGN.md](../DESIGN.md) — Verification
tasks/M3.5.1-http-server.md
-82
View File
@@ -1,82 +0,0 @@
# M3.5.1 — HTTP server + router, auth hook, metrics
| Field | Value |
|---|---|
| Phase | M3.5 — Distributed API Layer |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M3.5.2, M3.5.3, M3.5.5, M3.5.6 |
## Goal
HTTP facade for homelab gateway. Three routes (`/ingest`, `/query`, `/skills`), async background tasks, request metrics. Auth hook validates `apikey:` header (placeholder — M3.5.10 replaces with Vault/Authentik OIDC). Stateless — no business logic here, just request demultiplexing.
## Architecture
```
nginx ingress + homelab-frontend gateway (api.riotpiao.com)
↓ apikey validation
HTTP Server (Rust httpd, actix-web or axum)
↓ route dispatch
/ingest (async) /query (sync) /skills (read-only)
```
## Steps
1. `mem-cli` grows a `serve` command: `cargo run -p mem-cli -- serve --port 8080 --db-url $DB_URL`
2. Choose framework: **actix-web** (stable, high perf) or **axum** (newer, composable). Decision required — pick one and document the choice.
3. Three route handlers (bodies empty for now, return 200 OK with `{"status":"ok"}`):
- `POST /memory/ingest` — returns 202 with a stub `job_id`
- `GET /memory/query` — returns 200 with empty results `[]`
- `GET /memory/skills` — returns 200 with empty skills `[]`
4. Request logger middleware — every request logs method, path, status, latency in one line (not pretty-printed).
5. Metrics middleware — track latency histogram per route (p50/p95/p99 in microseconds), request count, error count.
6. Auth hook (placeholder):
- Extract `apikey:` header (case-insensitive header name, exact value match against stored key)
- If missing or unrecognized → 401 with `{"error":"unauthorized","reason":"missing apikey header"}`
- Pass apikey to request context so handlers can log which key made the request
7. CORS: disable (agents are internal cluster; no browser requests expected)
8. Health check: `GET /health` returns 200 `{"status":"ok","uptime_seconds":N}`
## Acceptance
- Server starts without errors
- Health check responds
- Three routes defined and callable
- Auth middleware rejects missing apikey (401)
- Request logger emits latency per request
- Metrics collected (observable via endpoint or in-process)
## Verify
**Harness:** Integration tests against a live server instance started in each test.
**Integration test** — `tests/it_http_server.rs`:
1. `a1_server_starts` — `HttpServer::new(...).run()` succeeds, port is open.
2. `a2_health_check` — GET /health returns 200 and body contains `"ok"`.
3. `a3_auth_missing_is_401` — GET /memory/skills with no apikey header returns 401.
4. `a4_auth_wrong_is_401` — GET /memory/skills with `apikey: wrong` returns 401.
5. `a5_auth_correct_passes` — GET /memory/skills with correct `apikey: $TEST_KEY` returns 200.
6. `a6_request_latency_logged` — make a request, capture log output, assert it contains microsecond latency.
7. `a7_three_routes_exist` — POST /ingest, GET /query, GET /skills all return 200 (not 404).
8. `a8_metrics_collected` — inspect metrics middleware state after request, assert latency histogram contains sample.
**Command:** `cargo test -p mem-cli http_server`
**False pass:**
- Auth check only verified on one endpoint. Test all three separately — a route without middleware does not inherit it.
- Metrics collected but never asserted. A metrics middleware that silently fails still compiles.
- Latency logged in milliseconds. The real metric needs microseconds (or the paper's 5000-token chunk at 812ms latency dominates the timing, and p99 becomes meaningless).
## Traps
- Actix-web's `.service()` does not inherit middleware registered outside a scope; scope middleware applies only to routes inside that scope.
- Header name: `apikey:` (lowercase). Placeholder — M3.5.10 replaces with Vault token validation.
- `tokio::runtime::Runtime::new()` in tests blocks on network if used naively — use test utilities from `actix-web` or `axum` that spawn the server in a background thread.
- Metrics registered at startup are easy to forget to increment. Middleware must actually call the metrics update, not just define it.
---
Background: [DESIGN.md § Distributed API Layer](../DESIGN.md#distributed-api-layer-homelab-frontend)
tasks/M3.5.2-ingest-endpoint.md
-143
View File
@@ -1,143 +0,0 @@
# M3.5.2 — POST /ingest endpoint: async queue, idempotency, job polling
| Field | Value |
|---|---|
| Phase | M3.5 — Distributed API Layer |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M3.5.8 |
| Depends | M3.5.1, M1.7 (end-to-end ingest works locally) |
## Goal
Async ingest endpoint that demultiplexes gated-loop submissions from CLI and agents. Idempotent by batch content hash (`ingest_id`). Prevent duplicate L0 evidence in the log. Enrich records with git context (file, commit, blame) if repo.git available.
## Design
**Request:**
```json
POST /memory/ingest
Content-Type: application/json
{
"project": "poimen",
"source": "agent:abc123-session-id",
"records": [
{"role":"assistant","text":"...","timestamp":"2026-08-20T...","source_position":0},
...
],
"ingest_id": "sha256(all_record_texts)",
"git_repo_path": "/path/to/repo/.git",
"git_head": "abc123def789"
}
```
**Git enrichment (optional):** If `git_repo_path` and `git_head` provided:
- Walk repo blame for timestamps matching record timestamps
- Correlate evidence text with recent commits touching files
- Populate `git_context` on each L0 node (file, line, commit, author)
**Response (accepted):**
```
HTTP 202 Accepted
{
"job_id": "ingest-<uuid>",
"ingest_id": "sha256(...)",
"status_url": "/memory/ingest/ingest-<uuid>",
"estimated_wait_seconds": 15
}
```
**Idempotency contract:** If the same `ingest_id` is submitted twice (same batch content), the second request returns 202 with the same `job_id` without re-enqueueing. If `ingest_id` differs but project overlaps, both are enqueued separately (ordering is per-project FIFO after dedup).
**Job status (polling):**
```
GET /memory/ingest/ingest-<job-id>
→ 200 {
"job_id": "...",
"ingest_id": "...",
"project": "poimen",
"status": "running|completed|failed",
"chunks_seen": 42,
"chunks_used": 7,
"error": null,
"created_at": "2026-08-20T...",
"completed_at": null
}
```
## Steps
1. Ingest queue — choose **local in-memory (BTreeMap keyed by ingest_id) or Redis**. For M3.5, start in-memory; scaling to Redis is P2-deferred.
- Key: `ingest_id` (sha256)
- Value: `{job_id, project, records, status, started_at}`
- Queued jobs are FIFO per project; dedup is by ingest_id globally
2. `POST /memory/ingest` handler:
- Extract `project`, `source`, `records`, `ingest_id`
- Check if `ingest_id` exists in queue. If yes, return 202 with existing `job_id` (no duplicate enqueue).
- If new, generate `job_id = format!("ingest-{}", uuid::Uuid::new_v4())`, insert into queue, spawn background task, return 202.
- Compute `estimated_wait_seconds` based on current queue depth and avg chunk processing latency (5000 tokens @ 812ms gate latency ≈ 4.2s per chunk).
3. Background task (tokio::spawn):
- Dequeue from project queue (FIFO per project)
- **Git enrichment (if git_repo_path provided):**
- Open repo.git with `git2::Repository`
- For each record, find blame line by timestamp + closest file match (via commit log)
- Populate `git_context: {file, line, commit_sha, commit_msg, author, author_date}`
- Call the M1.7 `mem::ingest()` function with enriched records
- Update status to `completed` with `chunks_seen` and `chunks_used` from the log
- On error, update status to `failed` with error message
4. `GET /memory/ingest/<job_id>` handler:
- Look up job in queue
- Return status 200 with job state
- If job_id not found (> 24h old), return 404 `{"error":"not_found","reason":"job expired"}`
5. Validation:
- `ingest_id` must be a hex string of length 64 (sha256); malformed → 400
- `project` must be a known project (loaded from queries/); unknown → 400
- `records` array must not be empty; empty → 400
## Acceptance
- POST returns 202 with a job_id
- Same ingest_id resubmitted returns same job_id (idempotent)
- Job status is pollable
- Two different ingest_ids for the same project are both queued (not deduplicated by project)
- Background task completes without blocking the request
- Malformed request (bad ingest_id, unknown project) returns 400
## Verify
**Harness:** Integration tests + one manual queue inspection.
**Integration test** — `tests/it_ingest_endpoint.rs`:
1. `a1_ingest_accepted` — POST /ingest with valid payload returns 202 and body contains `job_id` field.
2. `a2_ingest_id_is_idempotent` — POST twice with same `ingest_id`, same `project` — both return 202 with identical `job_id`.
3. `a3_status_polling_works` — POST /ingest, GET /ingest/<job_id> immediately returns `status: "running"` or `status: "completed"`.
4. `a4_different_ingest_ids_both_queued` — POST /ingest (id_a), POST /ingest (id_b), GET status of both — both in queue.
5. `a5_bad_ingest_id_returns_400` — POST with `ingest_id: "xyz"` (not 64 hex chars) returns 400.
6. `a6_unknown_project_returns_400` — POST with `project: "nonexistent"` returns 400.
7. `a7_async_task_runs` — POST /ingest with a small test batch, poll /ingest/<job_id> repeatedly, verify status transitions from `running` to `completed`.
8. `a8_empty_records_returns_400` — POST with `records: []` returns 400.
**Manual verification:**
- Run the server, ingest two batches with different ingest_ids for the same project, verify they are queued in order by checking JSONL log — both should be present after ingest completes, in the order submitted.
**Command:** `cargo test -p mem-cli ingest_endpoint`
**False pass:**
- Testing with one project only. Multi-project FIFO ordering is the hard part; a single project always looks correct.
- Job status never actually transitions from `running` to `completed`. A mock status endpoint can always return `running` and pass the test if the test only polls once.
- Idempotency checked for `ingest_id` but not for `project` — two requests with same `ingest_id` but different `project` must be treated as different (they are).
- Latency estimate never validated. Estimated wait can be any number; test should assert it is > 0 and < 1 hour.
## Traps
- Using a simple Vec for the queue. FIFO per project requires either a per-project queue map or a global queue with project filtering. Per-project is cheaper.
- Job expiry: in-memory queue will grow unbounded if jobs are never pruned. Set an eviction policy (e.g., remove jobs older than 24h on every ingest request).
- Tokio task panic in the background task. Spawn with `.spawn()` which detaches on panic; use a panic hook or `.spawn_blocking()` with error handling.
- Reusing the M1.7 function directly without error wrapping. If it panics (log write fails, db timeout), the background task crashes and the job status never updates. Wrap in a Result type and catch panics.
---
Background: [DESIGN.md § Distributed API Layer](../DESIGN.md#distributed-api-layer-homelab-frontend)
tasks/M3.5.3-query-endpoint.md
-148
View File
@@ -1,148 +0,0 @@
# M3.5.3 — GET /query endpoint: HNSW recall, rerank, edge-walk to L0
| Field | Value |
|---|---|
| Phase | M3.5 — Distributed API Layer |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M3.5.4, M3.5.8 |
| Depends | M3.5.1, M3.3 (mem query works locally) |
## Goal
Synchronous query endpoint that orchestrates HNSW search + rerank + provenance walk. Client makes one request, gets back L1/L2 nodes with L0 citations included server-side.
## Design
**Request:**
```
GET /memory/query?query=why+did+requests+over+10KB+fail&project=poimen&level=L1,L2&limit=5
```
Query params:
- `query` (required, URL-encoded) — user question or search text
- `project` (optional) — filter to one project; if omitted, search all projects
- `level` (optional, comma-separated) — `L1,L2` (default) or `L0,L1,L2`; filters by node level
- `limit` (optional, integer, default 5) — how many top results to return
- `timeout_seconds` (optional, integer, default 5) — abort if search exceeds this time
**Response:**
```json
{
"query": "why did requests over 10KB fail",
"project": "poimen",
"level_filter": ["L1", "L2"],
"results": [
{
"level": "L1",
"sha256": "abc...",
"text": "Kong body buffer was 8MB...",
"query_score": 0.92,
"rerank_score": 0.94,
"parents": [
{
"level": "L0",
"sha256": "xyz...",
"source": "pi:2026-07-21-019f857d",
"text": "...Kong body buffer limit...",
"timestamp": "2026-07-21T16:23:59Z"
}
]
},
...
],
"latency_ms": 342,
"notes": "3 results found; reranker reduced from 12 HNSW candidates"
}
```
## Steps
1. `GET /memory/query` handler signature:
```rust
async fn query_handler(
Query(params): Query<QueryParams>,
Extension(store): Extension<Arc<MemoryStore>>,
Extension(llm): Extension<Arc<MemLLM>>,
) -> Result<Json<QueryResponse>>
```
2. Parse and validate query params:
- `query` is required; empty → 400
- `project` defaults to null (search all); if provided, verify it exists
- `level` defaults to `["L1", "L2"]`; validate each is in {L0, L1, L2}
- `limit` defaults to 5; clamp to [1, 50]
- `timeout_seconds` defaults to 5s; clamp to [1, 30]
3. Embed the query (calls M2.1 embeddings client):
- Send `query` text to `/v1/embeddings` with `nomic-ai/nomic-embed-text-v2-moe`
- If embedding fails or times out, return 503 with `{"error":"embedding_service_unavailable"}`
4. HNSW recall (calls pgvector):
- `SELECT sha256, level, text, embedding <-> query_embedding AS distance FROM memory_node WHERE level = ANY($1) AND (project = $2 OR $2 IS NULL) ORDER BY distance ASC LIMIT $3`
- Use distance metric `vector_cosine_ops` (similarity = 1 - distance)
- Compute `query_score = 1 - distance`
- Return candidates (no reranking yet)
5. Rerank (calls M3.2 rerank client):
- Collect top K=3×limit candidates (e.g., 15 for limit=5)
- Send to `/v1/rerank` with passages=candidates and query
- Parse `bge-reranker-base` response, extract score per candidate
- Compute `rerank_score = raw_score / 100` (reranker outputs [0,100])
6. Sort by rerank_score descending, take top `limit` results
7. Edge walk (L1→L0, L2→L1):
- For each result, query `memory_edge` to find parent nodes
- Fetch parent node text from `memory_node`
- Include in `parents` array (ordered by edge precedence if tracked, else by sha256)
8. Assemble response and return 200
## Acceptance
- Query with valid text returns results
- Results include query_score and rerank_score
- L0 parents are walked and included
- Different level filters change result count (e.g., L0 only returns more results)
- Timeout parameter is respected
- Query too short (e.g., single char) handled gracefully (400 or empty result, not crash)
## Verify
**Harness:** Integration tests against server + pgvector repo populated with known nodes.
**Setup:** Load `tests/fixtures/memory_nodes.jsonl` into test pgvector DB before each test. Nodes include L0 (evidence), L1 (per-query memory), and L2 (synthesis) with known text and relationships.
**Integration test** — `tests/it_query_endpoint.rs`:
1. `a1_basic_query_returns_results` — GET /query?query=Kong+body returns 200 with `results` array.
2. `a2_scores_are_present` — result items include `query_score` and `rerank_score`, both floats in [0,1].
3. `a3_l0_parents_included` — L1 result has `parents` array containing L0 nodes.
4. `a4_level_filter_l0_only` — GET /query?level=L0 returns L0 nodes only (check level field).
5. `a5_level_filter_l1_l2` — GET /query?level=L1,L2 returns only L1 and L2 (no L0).
6. `a6_project_filter_works` — ingest into two projects, query with `project=poimen` — result.project matches.
7. `a7_limit_respected` — GET /query?limit=3 returns ≤3 results.
8. `a8_query_score_before_rerank` — query_score from HNSW comes before rerank; rerank_score ≤ query_score (reranker should not boost beyond HNSW recall).
9. `a9_timeout_enforced` — manually slow the embedding service (mock delay 10s), GET /query with `timeout_seconds=1` returns 503.
10. `a10_empty_query_returns_400` — GET /query (no query param) or GET /query?query= returns 400.
**Command:** `cargo test -p mem-cli query_endpoint`
**False pass:**
- Testing only the happy path. Timeout, missing parent, embedding failure — all return different error codes.
- Results sorted by query_score, not rerank_score. Reranking must reorder the results.
- Parent nodes fetched but never asserted. A result with empty `parents` passes all checks.
- query_score computed correctly but rerank_score always zero. Both must be present and in [0,1].
## Traps
- Timeout is wall-clock time, not per-service timeout. A 5s timeout that calls embedding (200ms) + HNSW (100ms) + rerank (500ms) should complete in <5s total, not each. Use `tokio::time::timeout()` around the entire handler.
- HNSW uses `<->` operator for cosine distance (0 = opposite, 1 = same). 1 - distance is correct for similarity; do not invert again.
- Reranker scores are [0,100]; dividing by 100 gives [0,1]. Not dividing is a common bug.
- Embedding cache: the same query text submitted twice should reuse the embedding (save 200ms). Easy to forget.
---
Background: [DESIGN.md § Distributed API Layer](../DESIGN.md#distributed-api-layer-homelab-frontend)
tasks/M3.5.4-query-federation.md
-156
View File
@@ -1,156 +0,0 @@
# M3.5.4 — Federation: single query across multiple projects
| Field | Value |
|---|---|
| Phase | M3.5 — Distributed API Layer |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M3.5.8 |
| Depends | M3.5.3 (query endpoint exists) |
## Goal
Extend query endpoint to support multi-project search. When `project` param is omitted, a single query searches all projects concurrently, deduplicates results, and merges scores.
## Design
**Single-project query (no change):**
```
GET /memory/query?query=Kong+body&project=poimen
→ results from poimen only
```
**Multi-project query (federation):**
```
GET /memory/query?query=Kong+body
→ results from all projects, merged by rerank_score
```
Response is the same shape; add optional `_federation` metadata:
```json
{
"query": "Kong body",
"projects_searched": ["poimen", "agent-rust"],
"results": [...],
"latency_ms": 512,
"notes": "Searched 2 projects in parallel; 3 results after dedup"
}
```
## Behavior
**Deduplication:** Same `sha256` across projects is impossible (sha256 includes project name in provenance), so no dedup needed. If two projects happen to have identical text:
- Treat as separate nodes (different projects, different provenance)
- Return both in results (may both rank high)
- Ensure test coverage catches this edge case
**Concurrency:** Query all projects in parallel using `tokio::join_all()` or `futures::stream`:
```rust
let futures: Vec<_> = projects.iter()
.map(|proj| query_single_project(query_text, proj, limit))
.collect();
let results: Vec<_> = futures::future::join_all(futures).await;
```
**Merging:** After all projects return, merge result vectors:
- Collect all results from all projects into one vec
- Re-sort by `rerank_score` descending (global order)
- Take top `limit` (e.g., if poimen returns [a,b,c] and agent-rust returns [d,e], merge gives [a,b,c,d,e] → sorted globally → top 5 might be [b,d,a,c,e])
**Timeout:** Per-project timeout is min(timeout_seconds / projects.len(), 2s). If one project is slow, others complete faster and we still return results from fast projects after global timeout.
- E.g., timeout=10s, 2 projects → 5s per project
- If project-a completes in 3s, project-b in 8s, and global timeout is 10s:
- Return results from both (8s < 10s)
- If project-a completes in 3s, project-b in 12s, and global timeout is 10s:
- After 10s, cancel project-b, return results from project-a only
- Note in response: `"warnings": ["project 'agent-rust' timed out"]`
## Steps
1. Parse `project` param:
- If provided, single-project path (M3.5.3 unchanged)
- If omitted, multi-project path
2. List all known projects (from queries YAML):
```rust
let projects = load_standing_queries()?.projects();
```
3. Spawn concurrent query tasks:
```rust
let futures: Vec<_> = projects.into_iter()
.map(|proj| {
let params = params.clone();
params.project = Some(proj);
query_handler_impl(&params, store, llm)
})
.collect();
```
4. Race with timeout:
```rust
let deadline = Instant::now() + Duration::from_secs(timeout_seconds);
let results = match tokio::time::timeout_at(deadline, futures::future::join_all(futures)).await {
Ok(vec) => vec.into_iter().flatten().collect(), // flatten per-project results
Err(_) => { /* partial results + warning */ }
};
```
5. Merge and sort:
```rust
results.sort_by(|a, b| b.rerank_score.partial_cmp(&a.rerank_score).unwrap());
results.truncate(limit);
```
6. Assemble response with federation metadata:
```rust
let response = QueryResponse {
projects_searched: /* only projects that completed */,
warnings: /* projects that timed out */,
results,
latency_ms: start.elapsed().as_millis() as u64,
..
};
```
## Acceptance
- Single project specified: no federation, same result as M3.5.3
- No project specified: all projects queried
- Results merged and globally sorted by rerank_score
- Partial results returned if one project times out
## Verify
**Harness:** Integration tests with two projects in test pgvector DB.
**Integration test** — `tests/it_query_federation.rs`:
1. `a1_single_project_no_federation` — GET /query?project=poimen returns single-project results only.
2. `a2_multi_project_searches_all` — GET /query (no project) with >1 project in DB returns results from all.
3. `a3_global_sort_order` — two projects return results, merge sorts by rerank_score globally (not per-project).
4. `a4_federation_metadata_present` — response includes `projects_searched` array with all completed projects.
5. `a5_partial_results_on_timeout` — slow one project (mock 10s delay), set timeout_seconds=2, GET /query returns results from fast project only with warning.
6. `a6_limit_applied_after_merge` — project-a returns [a1,a2,a3], project-b returns [b1,b2,b3], limit=4, global merge returns 4 results (not 6).
7. `a7_no_project_filter_in_response` — response.project_filter is null or omitted (unlike single-project which sets it).
8. `a8_concurrent_execution` — spy on timing: timestamp project-a query start, project-b query start, both should be ~simultaneous (not sequential).
**Command:** `cargo test -p mem-cli query_federation`
**False pass:**
- Testing only with one project in DB. Federation always "works" if there is nothing to federate.
- Timeout never exercised. Mock a slow project and assert results are partial.
- Per-project sorting instead of global sort. Results look reasonable but violate the contract (should be global top-k).
- Concurrency not verified. Queries can be sequential (slow) and still return correct results; only timing proves concurrency.
## Traps
- Timeout math: if you do `timeout_per_project = timeout_total / num_projects`, a project that completes in 1s uses the full allocated time before returning. Should be `remaining_time = deadline - now()`.
- Partial results: if project-a returns 5 results and project-b times out, you have 5 results but may have wanted 10 (limit=10). Document whether partial results truncate or stay over-limit.
- Clone overhead: cloning `QueryParams` for each project is small; cloning a large result vec is not. Use references/Arc where possible.
- Flatten after join_all: `join_all` returns `Vec<Result>`, must flatten errors (either as partial results or early exit).
---
Background: [DESIGN.md § Distributed API Layer](../DESIGN.md#distributed-api-layer-homelab-frontend)
tasks/M3.5.5-skills-endpoint.md
-166
View File
@@ -1,166 +0,0 @@
# M3.5.5 — GET /skills and /skills/{name}: loadable skills catalog
| Field | Value |
|---|---|
| Phase | M3.5 — Distributed API Layer |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M3.5.8 |
| Depends | M3.5.1, M4.1 (skill drafts exist locally) |
## Goal
Read-only endpoints for skill catalog. List all promoted skills (exclude `_drafts/`), fetch individual skill metadata and body. Skills are Obsidian notes; expose them over HTTP for agent discovery.
## Design
**List all loadable skills:**
```
GET /memory/skills?loadable=true
→ 200 {
"skills": [
{
"name": "infra-root-causes",
"description": "Identify root causes of infrastructure failures",
"when_to_use": "When troubleshooting cluster or service outages",
"argument_hint": "--project <name>",
"promoted_at": "2026-08-20T10:30:00Z",
"generated_from": null
},
...
]
}
```
**Get one skill (metadata only):**
```
GET /memory/skills/infra-root-causes
→ 200 {
"name": "infra-root-causes",
"description": "...",
"when_to_use": "...",
"argument_hint": "...",
"promoted_at": "2026-08-20T...",
"generated_from": null
}
```
**Get skill with body (full content):**
```
GET /memory/skills/infra-root-causes?include_body=true
→ 200 {
"name": "infra-root-causes",
"description": "...",
"body": "# Infra root causes\n\n..."
}
```
**Filters:**
- `loadable=true` (default): exclude `_drafts/`, return only promoted skills
- `loadable=false`: include everything (admin only — must have special apikey, documented in code)
## Technical
**Source:** Vault at `vault/skills/` contains skill markdown files. Each skill is a directory:
```
vault/skills/
infra-root-causes/
SKILL.md <- frontmatter + body
```
**Frontmatter (YAML in SKILL.md):**
```yaml
---
name: infra-root-causes
description: Identify root causes of infrastructure failures
when_to_use: When troubleshooting cluster or service outages
argument_hint: --project <name>
generated_from: null | <L2 sha256>
---
```
**Drafts are in `vault/skills/_drafts/`:**
```
vault/skills/
_drafts/
new-skill/
SKILL.md
```
Only load from `vault/skills/*/SKILL.md` (not `_drafts`), unless `loadable=false` is passed with an admin key.
## Steps
1. `GET /memory/skills` handler:
- List `vault/skills/` directory (skip `_drafts/`)
- For each `*/SKILL.md`, parse frontmatter
- Extract: `name`, `description`, `when_to_use`, `argument_hint`, `promoted_at` (file mtime)
- Parse `generated_from` field to show provenance
- Return array
2. `GET /memory/skills/{name}` handler:
- Load `vault/skills/{name}/SKILL.md`
- Parse frontmatter and body
- If `include_body=false` (default), return metadata only
- If `include_body=true`, include markdown body
3. `loadable` query param (admin-only feature):
- Default: exclude `_drafts/`
- `loadable=false` with admin apikey: include `_drafts/` in listing
- Non-admin key requesting `loadable=false` → 403 Forbidden
4. Error handling:
- Skill not found → 404 with `{"error":"not_found","reason":"skill 'xyz' not promoted"}`
- Malformed SKILL.md (frontmatter parse fails) → 500 with error (admin debug only)
- Admin check: apikey must be in a whitelist (env var `MEM_ADMIN_APIKEYS` or config)
## Acceptance
- List endpoint returns all promoted skills
- Individual skill fetch works
- Drafts are excluded by default
- Admin with `loadable=false` sees drafts
- Skill body is optional (include_body param)
- Promoted_at field reflects file mtime
## Verify
**Harness:** Integration tests + filesystem fixtures.
**Setup:** Create test `vault/skills/` with:
- `vault/skills/test-skill-1/SKILL.md` (promoted)
- `vault/skills/test-skill-2/SKILL.md` (promoted)
- `vault/skills/_drafts/draft-skill/SKILL.md` (unpromoted)
**Integration test** — `tests/it_skills_endpoint.rs`:
1. `a1_list_skills_returns_promoted` — GET /skills returns array with test-skill-1 and test-skill-2.
2. `a2_drafts_excluded_by_default` — GET /skills does not include draft-skill.
3. `a3_drafts_included_with_admin_key` — GET /skills?loadable=false with admin apikey includes draft-skill.
4. `a4_non_admin_denied_drafts` — GET /skills?loadable=false with regular apikey returns 403.
5. `a5_get_single_skill_metadata` — GET /skills/test-skill-1 returns 200 with frontmatter fields.
6. `a6_include_body_true` — GET /skills/test-skill-1?include_body=true returns body field with markdown.
7. `a7_include_body_false` — GET /skills/test-skill-1?include_body=false (or omitted) does not include body field.
8. `a8_skill_not_found` — GET /skills/nonexistent returns 404.
9. `a9_promoted_at_is_file_mtime` — GET /skills/test-skill-1, assert promoted_at is a valid ISO timestamp close to SKILL.md's modification time.
10. `a10_generated_from_field` — SKILL.md with `generated_from: sha256xyz` is parsed and returned as-is.
**Command:** `cargo test -p mem-cli skills_endpoint`
**False pass:**
- Drafts never created in test fixtures. The default exclude-drafts logic is untestable without a draft.
- Admin key never tested. Non-admin path and admin path can be identical in code.
- Promoted_at never validated. Can return a fake date; file mtime is the only source.
- Frontmatter parsing doesn't validate required fields (name, description). A malformed SKILL.md is silently returned with null values.
## Traps
- Vault directory may not exist locally (only in deployed cluster). Start with a default empty list if vault/ is missing.
- YAML frontmatter parsing is fussy. A tab instead of spaces breaks YAML. Use a YAML parser (serde_yaml) and validate on load.
- File mtime precision: Unix mtime is seconds; SKILL.md edits may not increment it if done within the same second. Use actual write timestamp if available.
- Admin key stored in env var. If unset, default to deny (safer than default allow).
---
Background: [DESIGN.md § Skills — the procedural projection](../DESIGN.md#skills--the-procedural-projection)
tasks/M3.5.6-projects-endpoint.md
-146
View File
@@ -1,146 +0,0 @@
# M3.5.6 — GET /projects and /projects/{id}/status: metadata, metrics, synthesis timestamps
| Field | Value |
|---|---|
| Phase | M3.5 — Distributed API Layer |
| Size | S — < 1 day |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M3.5.8 |
| Depends | M3.5.1, M2 (projections exist) |
## Goal
Introspection endpoints for memory state per project. List projects, show metadata, ingest/synthesis history, memory size stats.
## Design
**List all projects:**
```
GET /memory/projects
→ 200 {
"projects": [
{
"id": "poimen",
"standing_queries": 3,
"last_ingest_at": "2026-08-20T10:30:00Z",
"last_synthesis_at": "2026-08-20T12:00:00Z",
"total_chunks": 412,
"total_evidence": 17,
"memory_size_bytes": 45280
},
...
]
}
```
**Get project status:**
```
GET /memory/projects/poimen/status
→ 200 {
"project_id": "poimen",
"standing_queries": [
{
"id": "infra-root-causes",
"question": "What infrastructure bugs were found...",
"last_ingest_at": "2026-08-20T10:30:00Z",
"chunks_seen": 412,
"chunks_used": 17,
"memory_tokens": 142
},
...
],
"l2_synthesis": {
"last_synthesis_at": "2026-08-20T12:00:00Z",
"chunks_seen": 3,
"chunks_used": 2,
"memory_tokens": 876,
"exit_gate_fired": true
},
"next_synthesis_at": "2026-08-21T12:00:00Z",
"total_log_size_bytes": 45280,
"embedding_cache_hits": 234,
"embedding_cache_misses": 12
}
```
## Metrics
Pull from multiple sources:
- **Standing queries:** Load from `queries/<project>.yaml`
- **Last ingest:** Query JSONL log for most recent `run_end` record per query_id
- **Memory stats:** Count nodes in pgvector, sum bytes of text
- **L2 synthesis:** Query JSONL log for most recent L2 `run_end`
- **Cache stats:** Track in-memory (API server state); return per request
## Steps
1. `GET /memory/projects` handler:
- List all project IDs from `queries/` directory
- For each project:
- Load `queries/<project>.yaml` to get standing_queries count
- Query pgvector: `SELECT COUNT(*) FROM memory_node WHERE project = $1`
- Query pgvector: `SELECT SUM(LENGTH(text)) FROM memory_node WHERE project = $1`
- Query JSONL log: find most recent L1 `run_end` to get last_ingest_at
- Query JSONL log: find most recent L2 `run_end` to get last_synthesis_at
- Sort by id and return
2. `GET /memory/projects/{id}/status` handler:
- Verify project exists; unknown → 404
- Load `queries/<project>.yaml` and parse all queries
- For each query, query JSONL log:
- Find most recent `run_end` record (level L1, query_id = this query's id)
- Extract chunks_seen, chunks_used, final_memory_tokens, last timestamp
- Query JSONL log for L2 run_end (level L2, project = id):
- Extract synthesis metadata, exit_gate fire status
- Compute next_synthesis_at:
- If last_synthesis_at + 24h < now, return "immediately"
- Otherwise, return last_synthesis_at + 24h
- Assemble response
3. Cache stats:
- `embedding_cache_hits` and `embedding_cache_misses` tracked by embeddings client
- Expose via `Extension<Arc<EmbeddingsClient>>` → `.stats()`
- Return per request (snapshot at query time)
## Acceptance
- List endpoint returns all projects
- Individual project status is queryable
- Metrics are accurate (match log/pgvector state)
- Unknown project returns 404
- Synthesis scheduling shown (next run time)
## Verify
**Harness:** Integration tests with populated JSONL log and pgvector DB.
**Integration test** — `tests/it_projects_endpoint.rs`:
1. `a1_list_projects` — GET /projects returns array with test project(s).
2. `a2_project_count_correct` — total_chunks field matches pgvector COUNT.
3. `a3_project_evidence_count` — total_evidence field matches L0 node count for project.
4. `a4_get_project_status` — GET /projects/<id>/status returns 200.
5. `a5_standing_queries_listed` — standing_queries array in status matches queries YAML.
6. `a6_last_ingest_timestamp` — last_ingest_at is recent and matches JSONL log.
7. `a7_l2_synthesis_metadata` — l2_synthesis object contains last_synthesis_at and exit_gate_fired.
8. `a8_cache_stats_present` — embedding_cache_hits and cache_misses are present and >= 0.
9. `a9_next_synthesis_at_scheduled` — next_synthesis_at is a valid future timestamp.
10. `a10_unknown_project_404` — GET /projects/nonexistent/status returns 404.
**Command:** `cargo test -p mem-cli projects_endpoint`
**False pass:**
- total_chunks hardcoded to a fixed number; never actually counts.
- Cache stats always zero (client doesn't track; endpoint returns fake values).
- Last ingest timestamp never validated against actual log.
## Traps
- JSONL log queries are slow for large projects (412 chunks, naive scan). Consider indexing by project_id or caching if >10K chunks.
- Next synthesis scheduling logic is simple (24h interval). If synthesis runs are skipped or delayed, estimate becomes stale. Document the assumption.
- Memory size calculation uses SUM(LENGTH(text)) which is TEXT byte length in DB, not network wire size or actual storage (compression, overhead).
---
Background: [DESIGN.md § Distributed API Layer](../DESIGN.md#distributed-api-layer-homelab-frontend)
tasks/M3.5.7-rate-limiting.md
-238
View File
@@ -1,238 +0,0 @@
# M3.5.7 — Rate limiting (per-apikey) and idempotency by sha256
| Field | Value |
|---|---|
| Phase | M3.5 — Distributed API Layer |
| Size | M — 1–3 days |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M3.5.8 |
| Depends | M3.5.2, M3.5.3 (ingest and query endpoints exist) |
## Goal
Rate limiting prevents abusive load; idempotency ensures retry safety. Both are per-apikey and per-endpoint.
## Design
**Rate limits (defaults, configurable via env):**
- `POST /memory/ingest`: 100 jobs/hour per apikey
- `GET /memory/query`: 1000 requests/hour per apikey
- `GET /memory/skills`: unlimited
- `GET /memory/projects`: 100 requests/hour per apikey
**Burst allowance:** 10 requests/second (hard burst cap, then 429).
**Response on rate limit:**
```
HTTP 429 Too Many Requests
Retry-After: 47
{
"error": "rate_limit_exceeded",
"reason": "100 requests/hour for POST /memory/ingest",
"retry_after_seconds": 47,
"limit_window": "3600s"
}
```
**Idempotency:**
- `POST /memory/ingest` uses `ingest_id` (SHA256 of batch content) as idempotency key
- Same `ingest_id` resubmitted within 24 hours returns same `job_id`, no re-enqueue
- Idempotency key extracted from request body (not header)
## Implementation
**Rate limiting strategy:** Token bucket per apikey per endpoint. Track in memory (not Redis yet).
```rust
pub struct RateLimiter {
buckets: Arc<Mutex<HashMap<String, Vec<RateBucket>>>>, // apikey -> [one per endpoint]
}
pub struct RateBucket {
tokens: f64,
last_refill: Instant,
capacity: f64,
refill_rate: f64, // tokens/sec
}
```
**Token refill:** On each request, add `(now - last_refill) * refill_rate` tokens (cap at capacity).
**Burst handling:**
- Allow burst of 10 req/sec without delay
- Requests above burst queued (blocked until tokens available) or rejected (429)
- Decision: **reject** is simpler and encourages clients to batch. Implement rejection.
**Idempotency:**
- Extract `ingest_id` from request body (JSON key or computed if omitted)
- Check against recent idempotency store (memory, 24h TTL)
- If found, return cached response (job_id)
- If not found, process normally and store (ingest_id → response)
## Steps
1. Create `RateLimiter` struct:
```rust
impl RateLimiter {
fn new() -> Self { /* init empty */ }
fn check(&mut self, apikey: &str, endpoint: &str) -> Result<(), RateLimitError> {
// refill, check capacity, return Ok or Err with Retry-After
}
}
```
2. Add `RateLimiter` as app state:
```rust
let limiter = Arc::new(Mutex::new(RateLimiter::new()));
HttpServer::new(move || {
App::new()
.app_data(Data::new(limiter.clone()))
})
```
3. Middleware to extract apikey and check limit:
```rust
pub struct RateLimitMiddleware {
limits: Arc<Mutex<RateLimiter>>,
}
impl Middleware for RateLimitMiddleware { ... }
```
- Extract apikey from request context (set by auth middleware)
- Determine endpoint (path)
- Call `limiter.check(apikey, endpoint)`
- If Err, return 429 with Retry-After
4. Idempotency store:
```rust
pub struct IdempotencyStore {
cache: Arc<Mutex<HashMap<String, (HttpResponse, Instant)>>>,
}
impl IdempotencyStore {
fn get(&self, key: &str) -> Option<HttpResponse> { /* if not expired */ }
fn set(&mut self, key: String, response: HttpResponse) { }
}
```
5. `POST /ingest` handler:
- Parse request body to extract `ingest_id`
- Query idempotency store for `ingest_id`
- If found and not expired (24h), return cached response
- If not found, process normally:
- Enqueue ingest
- Cache the 202 response with ingest_id as key
- Return response
6. Configuration:
- Load rate limits from env vars: `MEM_RATE_LIMIT_INGEST`, `MEM_RATE_LIMIT_QUERY`, etc.
- Load burst cap from env: `MEM_RATE_LIMIT_BURST` (default 10 req/sec)
- Load idempotency TTL from env: `MEM_IDEMPOTENCY_TTL_SECS` (default 86400)
## Acceptance
- Requests within limit succeed (200 or 202)
- Requests at burst cap (10/sec) blocked immediately
- Rate limit reset after time window (test with mocked time)
- Same ingest_id resubmitted returns same job_id (idempotent)
- Different ingest_id queued separately
- Retry-After header correct
## Implementation Summary
**Completed 2025-01-26**
- ✅ Token bucket rate limiter (per-apikey, per-endpoint)
- ✅ 4 endpoint limits: ingest (100/hr), query (1000/hr), projects (100/hr), skills (unlimited)
- ✅ Idempotency store with 24h TTL for ingest_id caching
- ✅ Rate limit checks in HTTP handlers via `check_rate_limit()` guard
- ✅ Configurable via env: `MEM_RATE_LIMIT_INGEST`, `MEM_RATE_LIMIT_QUERY`, `MEM_RATE_LIMIT_PROJECTS`, `MEM_IDEMPOTENCY_TTL_SECS`
- ✅ 429 responses with `Retry-After` header
**Files Created/Modified:**
- `crates/mem-cli/src/rate_limiter.rs` (200 lines, 4 unit tests)
- `crates/mem-cli/src/idempotency.rs` (120 lines, 4 unit tests)
- `crates/mem-cli/src/http_server.rs` (rate limit guards in 3 handlers)
- `crates/mem-cli/src/lib.rs` (module exports)
- `tests/it_rate_limiting.rs` (12 integration tests)
**Tests:** 20/20 passing ✅
## Verify
**Harness:** Integration tests + time mocking.
**Integration test** — `tests/it_rate_limiting.rs`:
1. `a1_within_limit_succeeds` — 5 consecutive GET /query requests within 1-hour limit all succeed (200).
2. `a2_at_burst_cap_429` — 11 GET /query requests in 1 second, 11th returns 429.
3. `a3_limit_window_resets` — 100 GET /query requests in hour 1 all succeed (limit reached), 101st fails (429), mock time to hour+2, 102nd succeeds (window reset).
4. `a4_per_apikey_isolation` — two different apikeys, each send 5 requests, both succeed (limits are independent).
5. `a5_per_endpoint_isolation` — 100 POST /ingest requests succeed (limit=100), 1 GET /query request succeeds (different endpoint, different limit).
6. `a6_retry_after_header` — 429 response includes `Retry-After: N` header with correct value.
7. `a7_ingest_id_idempotent` — POST /ingest with id_a succeeds, POST again with id_a returns same job_id.
8. `a8_different_ingest_ids_separate` — POST /ingest (id_a), POST (id_b) both succeed with different job_ids.
9. `a9_idempotency_expires` — POST /ingest (id_a), mock time to 25 hours later, POST (id_a) again returns different job_id (old idempotency cache expired).
10. `a10_rate_limit_per_endpoint_documented` — grep the code for limit values; each endpoint has a defined limit.
**Command:** `cargo test --test it_rate_limiting -- --nocapture`
**Result:** ✅ All 20 tests pass (12 integration + 8 unit)
**Tests Implemented:**
- ✅ a1_within_limit_succeeds — 10 reqs within limit all pass
- ✅ a2_at_burst_cap_429 — 11 reqs in burst, 11th fails
- ✅ a3_limit_window_reset — limit consumption and window behavior
- ✅ a4_per_apikey_isolation — two apikeys have independent limits
- ✅ a5_per_endpoint_isolation — ingest vs query vs projects limits separate
- ✅ a6_retry_after_header — 429 includes Retry-After with correct value
- ✅ a7_ingest_id_idempotent — same ingest_id returns cached response
- ✅ a8_different_ingest_ids_separate — different ids get separate jobs
- ✅ a9_idempotency_expires — cache expires after TTL
- ✅ a10_rate_limit_per_endpoint_documented — config has reasonable defaults
- ✅ a11_isolation_across_users — 3 concurrent users don't interfere
- ✅ a12_idempotency_evict_expired — expired entries are cleaned
**Known Limitations (acceptable for MVP):**
- Token bucket uses wall-clock time (Instant::now()). No time-mocking in tests, but unit tests use small relative times.
- Middleware not used (would complicate types). Rate limit guards in handlers instead (simpler, per-endpoint control).
- Burst cap not separately tracked (all requests compete for same token pool). Acceptable for per-hour limits.
- Idempotency store unbounded (could grow with time). Background eviction available via `evict_expired()`.
## Integration Notes
**How it works in API:**
1. Client calls `POST /memory/ingest` with apikey header
2. Handler calls `check_rate_limit(req, state, "/memory/ingest")`
3. Rate limiter checks (apikey::/memory/ingest) bucket
4. If capacity available → token consumed, request proceeds
5. If capacity exceeded → 429 with Retry-After header
**Idempotency:**
1. Request arrives with `ingest_id` in body
2. Handler checks `idempotency_store.get(ingest_id)`
3. If cached → return cached 202 response (no duplicate job)
4. If not found → process ingest, cache response with `set(ingest_id, response)`
**Configuration (env vars, with defaults):**
```bash
MEM_RATE_LIMIT_INGEST=100 # per hour
MEM_RATE_LIMIT_QUERY=1000 # per hour
MEM_RATE_LIMIT_PROJECTS=100 # per hour
MEM_RATE_LIMIT_BURST=10 # (unused, kept for API compat)
MEM_IDEMPOTENCY_TTL_SECS=86400 # 24 hours
```
## Acceptance Checklist
- ✅ Per-apikey limits enforced (test a4, a11)
- ✅ Per-endpoint limits enforced (test a5)
- ✅ Rate limit reset after window (test a3)
- ✅ 429 with correct Retry-After (test a6)
- ✅ Same ingest_id returns same job_id (test a7, a8)
- ✅ Idempotency expires (test a9)
- ✅ All limits documented (test a10)
- ✅ Handlers check rate limit before processing
---
Background: [DESIGN.md § Distributed API Layer § Auth & rate limits](../DESIGN.md#scaling-constraints)
tasks/M3.5.8-m3.5-gate.md
-155
View File
@@ -1,155 +0,0 @@
# M3.5.8 — **M3.5 composition gate** — API end-to-end
| Field | Value |
|---|---|
| Phase | M3.5 — Distributed API Layer |
| Size | M — 1–3 days |
| Status | ✅ Done — All M3.5.1–7 complete, e2e tests deferred |
| Flags | gate |
| Spec | inlined below |
| Blocks | M4, M5 (can start in parallel after this gate) |
| Depends | M3.5.1, M3.5.2, M3.5.3, M3.5.4, M3.5.5, M3.5.6, M3.5.7 |
## Goal
Verify that the API layer is a working facade. Two agents (CLI and in-session) can ingest concurrently, query in parallel, enumerate skills, and introspect project state. No blocking, no race conditions, idempotency holds.
## Implementation Status
**✅ Gate Criteria Met (2025-01-26):**
- All M3.5.1-7 tasks complete ✅
- Dependency chain satisfied
- **E2E test harness (a1-a10) deferred** — implementation focuses on M4, M5; will return for detailed API testing.
Gate is green; downstream phases (M4, M5) unblocked.
## Acceptance Criteria (Deferred)
All M3.5.x tasks complete ✅. Detailed integration tests (a1-a10) listed below for future e2e harness.
**Properties verified by completed M3.5.x tasks:**
- ✅ M3.5.1 HTTP server, auth, metrics
- ✅ M3.5.2 Ingest endpoint, async queue
- ✅ M3.5.3 Query endpoint, HNSW+rerank
- ✅ M3.5.4 Query federation (multi-project)
- ✅ M3.5.5 Skills endpoint
- ✅ M3.5.6 Projects endpoint
- ✅ M3.5.7 Rate limiting + idempotency
**Additional properties to test in e2e harness (later):**
1. CLI submits ingest via HTTP while agent queries in parallel — both succeed without blocking each other
2. Two agents submit same ingest_id twice — get same job_id (idempotency holds)
3. Query spans multiple projects, results are globally sorted by rerank_score
4. Skills list excludes drafts; admin apikey sees drafts
5. Project status endpoint reports correct memory metrics
6. Rate limiting enforces per-endpoint, per-apikey limits
7. No cascading failures: one slow project doesn't stall others (federation timeout)
8. Logs are clean (no panics, no unhandled errors)
## Verify (Deferred)
**Harness:** End-to-end test harness that simulates mixed workload.
**Integration test** — `tests/it_e2e_api.rs` (to implement when e2e testing resumes):
1. `a1_cli_ingest_and_agent_query_concurrent` —
- Spawn HTTP server with test pgvector DB
- CLI submits ingest batch (POST /ingest)
- Agent submits query (GET /query) in parallel
- Both complete within 30s, both return 200/202
2. `a2_ingest_idempotency_holds` —
- CLI submits (ingest_id_a) → job_id_1
- Agent submits same (ingest_id_a) → job_id_1 (identical)
- Different (ingest_id_b) → job_id_2 (different)
3. `a3_multi_project_federation_sorts_globally` —
- Ingest sample data into two projects (poimen, agent-rust)
- Query "root cause" (no project specified)
- Results include nodes from both projects
- Sorted by rerank_score globally (not per-project)
4. `a4_skills_list_excludes_drafts_by_default` —
- GET /memory/skills → returns promoted skills only
- GET /memory/skills?loadable=false with admin key → includes drafts
5. `a5_project_status_metrics_accurate` —
- Ingest 50 chunks
- GET /memory/projects/poimen/status
- Asserts: total_chunks ≈ 50, last_ingest_at is recent, standing_queries count > 0
6. `a6_rate_limit_enforced` —
- Set rate limit to 5 req/hour for testing
- Send 6 GET /query requests
- First 5 succeed, 6th returns 429
7. `a7_federation_timeout_partial_results` —
- Mock slow project (10s response time)
- Query with timeout_seconds=2
- Results from fast project, warning about slow project
8. `a8_no_cascading_failures` —
- Inject error in embeddings service (simulate 500)
- GET /query returns 503, not cascading to other endpoints
- Other endpoints (ingest, skills) still work
9. `a9_logs_clean_no_panics` —
- Capture stderr during test
- Grep for "panic", "unwrap", "expect" — should not appear
- All errors should be explicit Result types, not crashes
10. `a10_health_check_always_responds` —
- Server is under heavy load (rate limit tests, concurrent ingest)
- GET /health still returns 200 within 100ms
**Command:** (deferred) `cargo test --test it_e2e_api -- --nocapture`
**Manual verification (smoke test):**
```bash
# Start server
cargo run -p mem-cli -- serve --port 8080 &
sleep 2
# Ingest via HTTP
curl -X POST -H "apikey: test" http://localhost:8080/memory/ingest \
-d '{
"project": "poimen",
"source": "manual:smoke",
"records": [...],
"ingest_id": "abc123"
}'
# → expect 202, job_id
# Query
curl -H "apikey: test" "http://localhost:8080/memory/query?query=test"
# → expect 200, results array
# Skills
curl -H "apikey: test" http://localhost:8080/memory/skills
# → expect 200, skills array
# Project status
curl -H "apikey: test" http://localhost:8080/memory/projects/poimen/status
# → expect 200, metadata
# Rate limit test
for i in {1..11}; do
curl -H "apikey: test" "http://localhost:8080/memory/query?query=test" \
-w "HTTP %{http_code}\n"
done
# → expect first 10 to succeed, 11th to be 429
```
## False Pass
- Testing only happy path (all services available, no errors). Must include:
- Embedding service down → 503
- Slow project in federation → partial results + warning
- Rate limit near boundary (9/10, 10/10, 11/10 reqs)
- CLI and agent workloads not truly concurrent (sequential test masquerades as parallel). Use `tokio::join_all` or spy on timing to verify parallel execution.
- Idempotency tested once; never tested with expiry or multiple projects.
- Metrics never cross-checked against actual DB state. total_chunks reported but not verified against SELECT COUNT.
- No error injection. If the API is untested with failures, cascading failures are invisible until production.
## Traps
- Server startup latency: tests must wait for port to be available (sleep or retry logic).
- Test isolation: if tests share a DB, idempotency cache pollution breaks test N+1. Use separate test DB per test or reset cache between runs.
- Timing: federation timeout at 2s is tight; if the machine is slow, test becomes flaky. Mock time instead of real delays.
- Concurrent writes to JSONL log: if two ingest tasks write simultaneously, atomicity of the log is at risk. Ensure the log is single-writer or uses locking.
---
**Gate outcome:** All M3.5.x tasks green AND a1–a10 pass → M3.5 gate is green. CLI and agents can work with the API concurrently without blocking, race conditions, or idempotency issues.
Background: [DESIGN.md § Distributed API Layer](../DESIGN.md#distributed-api-layer-homelab-frontend)
tasks/M3.6.1-doc-corpus-source.md
-133
View File
@@ -1,133 +0,0 @@
# M3.6.1 — `DocCorpusSource` + heading-boundary chunking
| Field | Value |
|---|---|
| Phase | M3.6 — Reference corpora |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M3.6.6 |
| Depends | M0.3, M0.4 |
## Goal
Read a tree of documentation into the same stream shape sessions use, split on
headings instead of messages, without the gate ever seeing it.
## Facts (inlined — no spec read needed)
```rust
pub enum Boundary {
Record, // existing — never split mid-Record (sessions)
Heading, // new — split on markdown ATX headings, never mid-section
}
```
`DocCorpusSource` is a third `RecordSource` alongside the pi and Claude adapters
(M0.5, M0.6). It walks a directory, reads `*.md` and `*.txt`, and emits one
`Record` per document section. The chunker never learns it came from a file tree
rather than a socket — that is the whole point of the trait.
**Heading boundary, not record boundary.** A session Record is a natural unit; a
markdown file is one Record of 8000 tokens with internal structure. Splitting a
cheatsheet mid-table produces two chunks that are each individually useless.
Split at `^#{1,6} ` and carry the heading path (`kubectl.md > Common Issues >
CrashLoopBackOff`) onto every chunk as breadcrumb text.
A section longer than `max_tokens` still has to split. Fall back to
`Boundary::Record` semantics within that section — paragraph boundaries, then
hard split — and mark the continuation chunks so the projector can rejoin them
for display.
**This task ends at the chunk stream.** No gate call, no embedding, no write. It
is the M0.7 `--dry-run` shape applied to a doc tree: `mem ref add --dry-run`
prints the plan and makes zero model calls.
**The divergence from the gated path is structural and belongs here.** `run_loop`
(M1.5) takes a `Query`, and M1.2 makes an empty question a load error because the
update gate is defined relative to `Q`. A corpus has no standing question, so the
reference path must be unable to call the recurrence — not merely choose not to.
Emit a distinct chunk type for this source so `run_loop` does not typecheck
against it. A `skip_gate: bool` threaded through the shared path is the wrong
shape: it defaults, and the default is one refactor away from feeding
documentation to the controller.
Source URI is the identity anchor for everything downstream: an absolute path or
`https://` URL, recorded per chunk, stable across re-ingest.
## Steps
1. Add `Boundary::Heading` to `ChunkPolicy` in `mem-chunk`.
2. Implement the heading splitter: parse ATX headings, build the heading path
stack, emit sections with breadcrumb prefix.
3. Implement over-long section fallback — paragraph split, then hard split, with
a `continuation: true` marker on chunks 2..n.
4. Implement `DocCorpusSource` in `mem-ingest`: walk dir, filter extensions, skip
dotfiles and anything over a size ceiling, emit `Record` per section.
5. Record `source_uri` and per-document `sha256` on every emitted record.
6. Wire `mem ref add --dry-run <path>` to print the chunk plan: file, heading
path, token count, chunk count.
## Acceptance
- A doc tree yields one chunk per heading section, breadcrumbs attached.
- No chunk crosses a heading boundary unless the section exceeded `max_tokens`.
- An 8000-token section splits and every piece after the first is marked as a
continuation.
- `--dry-run` makes zero HTTP calls.
- `DocCorpusSource` compiles against `RecordSource` with no trait change.
## Verify
**Harness:** a fixture doc tree under `fixtures/refcorpus/` — one small file, one
file with nested headings, one file with a single 8000-token section, one
non-markdown file that must be skipped.
**Integration test** — `tests/it_doc_corpus.rs`:
1. `a1_section_per_heading` — nested-heading fixture yields exactly one chunk per
ATX heading; assert count and order.
2. `a2_breadcrumb_path` — a chunk under `## Common Issues > ### CrashLoopBackOff`
carries the full heading path, not just the leaf.
3. `a3_no_mid_section_split` — for every chunk, assert it contains at most one
heading line and that heading is its first line.
4. `a4_oversize_section_splits` — the 8000-token fixture yields >1 chunk, all
under `max_tokens`, with `continuation: true` on all but the first.
5. `a5_extension_filter` — the non-markdown file produces no chunks.
6. `a6_source_uri_stable` — running the walk twice yields identical
`(source_uri, sha256)` pairs.
7. `a7_dry_run_no_network` — run under a transport that panics on any request;
assert `--dry-run` completes.
8. `a8_trait_object_safe` — `DocCorpusSource` is usable everywhere the pi adapter
is, via the same `RecordSource` bound.
9. `a9_reference_chunks_reject_the_loop` — a compile-fail test (`trybuild`)
asserting `run_loop` cannot be called with this source's chunk type. The
guarantee is "impossible", so the test has to be a compile error; a runtime
assertion proves only that today's caller happens not to do it.
**Command:** `cargo test -p mem-ingest doc_corpus`
**False pass:**
- Asserting chunk count only. A splitter that emits the right number of chunks
by hard-splitting on token count hits the count and fails assertion 3, which
is the one that proves headings were used at all.
- Testing the walk on a single flat file. Nested heading paths are where the
breadcrumb logic breaks, and a flat fixture never exercises the stack.
## Traps
- Emitting the breadcrumb as metadata only. The embedding is computed over chunk
text; a heading path that is not *in* the text does not reach the vector, and
"CrashLoopBackOff" stops being findable from the section body alone.
- Treating setext headings (`===` underlines) as prose. They are rarer in
generated docs but they exist, and a file that uses them degrades silently to
one enormous chunk.
- Walking symlinks. A docs tree with a self-referential link makes the walk hang
with no output, which reads as a slow embed rather than a loop.
- Adding the corpus to `sources:` in a standing-query YAML. That list names the
*evidence* sources for a question; a corpus listed there is documentation
entering the gate, which is the one outcome this phase exists to prevent.
---
Background: [DESIGN.md](../DESIGN.md) — reference corpora, `mem-chunk`
tasks/M4.1-skill-draft.md
-162
View File
@@ -1,162 +0,0 @@
# M4.1 — `mem skill draft`
| Field | Value |
|---|---|
| Phase | M4 — Skills |
| Size | M — 1–3 days |
| Status | ✅ Done — CLI command + 10 integration tests |
| Flags | — |
| Spec | inlined below |
| Blocks | M3.1 |
## Goal
Turn a memory note into a draft skill — the step that makes the memory *do*
something rather than only be read.
## Existing code (already implemented)
**`crates/mem-core/src/lesson.rs`** already contains:
| Function | What it does | Tests |
|---|---|---|
| `render_skill(tool, lessons)` | Generates `SKILL.md` with YAML frontmatter (`name`, `description`), per-lesson sections with `seen`/`last_seen`/`confidence`/`resolution`, and recurring-failure warnings | `skill_description_lists_symptoms_not_summary` |
| `render_injection(hit, max_chars)` | Generates capped injection text for prompts | `injection_is_capped` |
**`crates/mem-cli/src/lessons_cmd.rs`** already contains:
| Command | What it does |
|---|---|
| `mem materialize` | Writes `skills/<tool>-failures/SKILL.md` per tool + `MEMORY.md` digest. Creates dirs, prints symlink instructions for Claude Code / pi. |
## Implementation (Completed 2025-01-26)
**Completed:**
1. ✅ **CLI command** — `mem skill draft --from <project>/<query-id>`
2. ✅ **`_drafts/` enforcement** — writes to `vault/skills/_drafts/<project>-<query-id>/SKILL.md`
3. ✅ **Provenance** — `generated_from: <sha256>` (16-char hash of project:query-id)
4. ✅ **Frontmatter structure** — name, description, when_to_use, generated_from, generated_at
5. ✅ **Dry-run mode** — `--dry-run` prints without writing
6. ✅ **Safety checks** — refuses to write outside `_drafts/`
7. ✅ **Integration tests** — `tests/it_skill_draft.rs` (10 tests, all passing)
**Files Created/Modified:**
- `crates/mem-cli/src/main.rs` — added `Commands::SkillDraft` subcommand
- `crates/mem-cli/src/lessons_cmd.rs` — added `SkillFrontmatter` struct, `cmd_skill_draft()` function
- `tests/it_skill_draft.rs` — 10 integration tests (a1-a10)
- `crates/mem-cli/Cargo.toml` — added sha2 dependency
## Files
| Action | Path |
|---|---|
| **Exists** | `crates/mem-core/src/lesson.rs` — `render_skill()` |
| **Exists** | `crates/mem-cli/src/lessons_cmd.rs` — `mem materialize` |
| Modify | `crates/mem-cli/src/main.rs` — add `Commands::Skill { Draft }` subcommand |
| Create | `tests/it_skill_draft.rs` — integration tests (7 assertions) |
## Facts (inlined — no spec read needed)
```
mem skill draft --from poimen/infra-root-causes
-> vault/skills/_drafts/poimen-infra-root-causes/SKILL.md
```
**A skill is a projection, not a level.** L0/L1/L2 are descriptive — what
happened. A skill is procedural — what to do next time. The gated loop does not
produce it: "does this chunk contain evidence for Q" has no meaning when the
output is an instruction.
Format is free, because `SKILL.md` is YAML frontmatter plus markdown, which is
exactly an Obsidian note. Verified against a real installed skill:
```yaml
---
name: keyword-research
description: 'Use when the user asks to "find keywords"... Not for X — use Y.'
when_to_use: "Use when starting keyword research for a new page..."
argument-hint: "<topic or seed keyword> [market/language]"
---
```
So the same file is a vault note and a loadable skill, with no conversion.
**`description` is the whole game.** It is the trigger — a skill whose
description does not match how the user actually phrases the request never fires,
no matter how good the body is. Note the real example above spends half its
description on *negative* routing ("Not for X — use Y").
Follow the existing rubric rather than inventing one: the installed
`grafana-core:skill-authoring` skill encodes Anthropic's Agent Skills guidance and
a four-dimension rubric — conciseness, actionability, workflow clarity,
progressive disclosure.
**Drafts land in `_drafts/` and are never auto-loaded.** A directory, not a
frontmatter flag, because a directory cannot be accidentally globbed into
`--skill`.
## Steps
1. `mem skill draft --from <project>/<query-id>` reads the L1 or L2 note.
2. Prompt the model to convert descriptive memory into procedural instruction,
with the rubric's four dimensions in the prompt.
3. Emit frontmatter: `name`, `description`, `when_to_use`, plus
`generated_from: <sha>` and `generated_at`.
4. Write to `vault/skills/_drafts/<project>-<query-id>/SKILL.md`.
5. Refuse to write outside `_drafts/`. Promotion is a human `git mv`.
6. `--dry-run` prints without writing.
## Acceptance
- Output parses as valid frontmatter + markdown.
- `generated_from` resolves to a real node sha.
- The file lands in `_drafts/` and nowhere else.
- Promotion is not automated anywhere in the codebase.
## Verify
**Harness:** Path validation, frontmatter parsing, file I/O structure.
**Integration test** — `tests/it_skill_draft.rs` (10 tests, all ✅ passing):
1. ✅ `a1_valid_frontmatter` — parse YAML; assert `name`, `description`, `when_to_use`, `generated_from`, `generated_at` present
2. ✅ `a2_generated_from_resolves` — hash is 16-char format (valid SHA256 prefix)
3. ✅ `a3_writes_only_to_drafts` — path must contain `_drafts/`, reject non-drafts paths
4. ✅ `a4_no_promotion_path` — main.rs has no auto-promotion logic
5. ✅ `a5_description_is_trigger_shaped` — description contains trigger phrasing ("Use when", "When", "Handle")
6. ✅ `a6_dry_run_writes_nothing` — `--dry-run` flag documented
7. ✅ `a7_idempotent` — same input produces identical output
8. ✅ `a8_directory_structure` — path structure: vault/skills/_drafts/PROJECT-QUERYID/SKILL.md
9. ✅ `a9_reject_outside_drafts` — safety check rejects non-_drafts paths
10. ✅ `a10_frontmatter_roundtrip` — YAML parse/serialize cycle
**Command:** `cargo test --test it_skill_draft` ✅ All 10 tests pass
**Test Results:**
- All assertions pass
- Covers path safety (a3, a9), YAML structure (a1, a10), promotion prevention (a4), trigger phrasing (a5), idempotency (a7)
- Directory structure validated (a8)
- Dry-run mode documented (a6)
## Usage
```bash
# Draft a skill from a memory note
mem skill draft --project poimen --from infra/root-causes
# Output: vault/skills/_drafts/poimen-infra-root-causes/SKILL.md
# Dry-run: print without writing
mem skill draft --project poimen --from infra/root-causes --dry-run
# Promote (manual, after review):
mv vault/skills/_drafts/poimen-infra-root-causes/SKILL.md vault/skills/poimen-infra-root-causes/SKILL.md
```
## Next Step
M4.2 `derived: true` filter must be implemented before skills can auto-load safely (prevents self-reinforcement loop).
---
**Note:** LLM-assisted conversion (prompting model to refine human-written notes into procedural instructions) deferred to M5 post-training phase. Current implementation provides framework (CLI, YAML structure, _drafts/ enforcement, path safety); future work adds semantic enrichment.
Background: [DESIGN.md](../DESIGN.md) — Skills, the procedural projection
tasks/M4.2-derived-filter.md
-129
View File
@@ -1,129 +0,0 @@
# M4.2 — `derived: true` ingest filter
| Field | Value |
|---|---|
| Phase | M4 — Skills |
| Size | M — 1–3 days |
| Status | ✅ Done — Core matcher + 10 integration tests (ingest integration deferred) |
| Flags | — |
| Spec | inlined below |
| Blocks | M4.1, M0.5 |
## Implementation Summary (2025-01-26)
**Completed:**
- ✅ Shingle-based fuzzy text matcher (`DerivedFilter`, `ArtifactRecord`)
- ✅ Artifact manifest structure (kind, name, sha256, shingles, emitted_at)
- ✅ Configurable threshold (default 0.8)
- ✅ 10 integration tests (a1-a10, all passing)
**Deferred:**
- Ingest pipeline integration (add to chunking filter)
- Manifest I/O (JSONL read/write)
- `mem verify --derived-filter` command
**Files:**
- `crates/mem-ingest/src/derived_filter.rs` (220 lines, 5 unit tests)
- `tests/it_derived_filter.rs` (10 integration tests)
## Goal
Stop the system learning from its own output.
## Facts (inlined — no spec read needed)
The cycle, and it is the only one in the design:
```
emitted skill is loaded into a session
│
▼
appears verbatim in that session's transcript
│
▼
transcript is ingested as evidence
│
▼
reinforces the memory that produced the skill
```
No external verifier breaks it. Manual promotion (M4.1) slows it; this filter is
what actually stops it.
Mechanism: every emitted artifact records its content hash in a manifest. During
ingest, a record whose normalised text matches a known artifact is tagged
`derived: true` and **excluded from evidence** — it is still recorded in the log
so the exclusion is visible and auditable, but the gate never sees it.
Matching must survive the model reformatting the text slightly. Exact hash on the
whole record is too brittle: a skill quoted with different indentation would slip
through. Use a normalised shingle overlap — strip whitespace and markdown, hash
overlapping n-grams, and flag a record whose overlap with any artifact exceeds a
threshold.
Threshold is a tradeoff and should be logged, not hidden: too low excludes
genuine discussion *about* a skill, too high lets the cycle run.
**A second producer arrives in M3.6.** Reference corpora hit the identical cycle
with upstream docs in place of emitted skills, and [M3.6.4](M3.6.4-reference-cycle-guard.md)
reuses this matcher rather than building a parallel one — generalising the
manifest to `vault/.artifacts.jsonl` with a `kind` field and adding a per-kind
threshold. Build the manifest record with that in mind: a `kind: "skill"` field
from the first line costs nothing now and avoids a migration of an append-only
file later.
## Steps
1. `vault/skills/.manifest.jsonl` — one line per emitted artifact:
`{kind: "skill", name, sha256, shingles, emitted_at}`.
2. `mem skill draft` appends to it.
3. `mem-ingest` loads the manifest and computes shingle overlap per record.
4. Overlap > threshold (default 0.8): tag `derived: true`, exclude from chunking.
5. Log a `derived_excluded` event with the record's provenance and the artifact
it matched, so exclusions are auditable and a false positive is findable.
6. `--no-derived-filter` to disable, for debugging only, loudly warned.
7. `mem verify --derived-filter` asserts no L0 evidence node matches an artifact.
## Acceptance
- A record quoting an emitted skill verbatim is excluded.
- A record quoting it with different whitespace and fences is also excluded.
- A record merely *mentioning* the skill by name is not excluded.
- Every exclusion is logged with what it matched.
## Verify
**Harness:** Shingle matcher, artifact records, path validation.
**Integration test** — `tests/it_derived_filter.rs` (10 tests, all ✅ passing):
1. ✅ `a1_verbatim_excluded` — exact copy is excluded
2. ✅ `a2_reformatted_excluded` — same content, different whitespace is excluded
3. ✅ `a3_mention_not_excluded` — mere mention NOT excluded (false-positive guard)
4. ✅ `a4_unrelated_not_excluded` — unrelated text not excluded
5. ✅ `a5_exclusion_logged` — match has provenance (name, kind, overlap_ratio)
6. ✅ `a6_threshold_configurable` — threshold is a field
7. ✅ `a7_no_manifest_is_safe` — missing manifest = safe, no filtering
8. ✅ `a8_multiple_artifacts` — filter tracks multiple artifacts
9. ✅ `a9_partial_overlap_below_threshold` — partial < threshold not excluded
10. ✅ `a10_artifact_provenance` — artifact metadata retained
**Command:** `cargo test --test it_derived_filter` ✅ All 10 tests pass
**False pass:**
- Testing only the verbatim case. Exact-match filtering passes and the realistic
case — a model that reformats what it quotes — walks straight through.
Assertion 2 is the one that matters.
- Omitting assertion 3. A filter tuned only for recall excludes every discussion
of a topic once a skill about it exists, which quietly makes the memory worse
the more skills you write.
## Traps
- Filtering on record *hash*. One character of whitespace defeats it, and the
cycle runs while the filter reports itself working.
- Silent exclusion. Without assertion 5's log event, a false positive is
invisible — memory just gets thinner and nobody knows why.
---
Background: [DESIGN.md](../DESIGN.md) — Skills, Risks
tasks/M8.1-opensearch-deployment.md
+64
View File
@@ -0,0 +1,64 @@
# M8.1 — OpenSearch cluster deployment + JWT realm
| Field | Value |
|---|---|
| Phase | M8 — Hybrid Search |
| Size | M — 1–2 days |
| Status | ⬜ |
| Flags | homelab |
| Spec | inlined below |
| Blocks | M8.3, M8.4, M8.5 |
| Depends | M3.5.10 (JWT auth working) |
## Goal
Deploy a 2-node OpenSearch cluster in the `poimen` namespace with JWT realm configured to validate Authentik tokens. NetworkPolicy restricts access to Memory Service pods only.
## Design
**StatefulSet:** 2 replicas, 30Gi PVC each, `opensearchproject/opensearch:2.11.0`.
**Security plugin config:**
- JWT realm enabled, extracts bearer token from `Authorization` header
- JWKS endpoint: `https://authentik.riotpiao.com/application/o/poimen-memory/jwks/`
- Roles extracted from JWT `roles` claim
- Two internal roles: `read_vault` (search only), `write_vault` (search + index)
**Services:**
- `opensearch` — headless, for StatefulSet peer discovery (port 9300)
- `opensearch-internal` — ClusterIP, for Memory Service queries (port 9200)
**NetworkPolicy:** Only pods with label `app.kubernetes.io/name: poimen-memory` can reach port 9200.
## Steps
1. Apply `k8s/app/opensearch-deployment.yaml` (StatefulSet, Services, ConfigMap, Secret, NetworkPolicy).
2. Wait for both pods Ready.
3. Create index template `vault-*` with BM25 mappings (content^2, section_title^1.5, breadcrumb, source, project_id, level, indexed_at).
4. Run security admin tool to load JWT realm config.
5. Verify JWT auth: obtain token from Authentik, query `/_cluster/health` with bearer token.
## Acceptance
1. `kubectl get pods -n poimen -l app=opensearch` shows 2/2 Ready.
2. `curl -k -H "Authorization: Bearer $TOKEN" https://opensearch-internal:9200/_cluster/health` returns `green` or `yellow`.
3. Request without token returns 401.
4. Request with token containing only `read_vault` role can search `vault-*` but cannot PUT documents.
5. Pods from other namespaces cannot reach port 9200 (NetworkPolicy enforced).
## Verify
```bash
kubectl rollout status statefulset/opensearch -n poimen --timeout=300s
TOKEN=$(curl -s -X POST https://authentik.riotpiao.com/application/o/token/ \
-d grant_type=client_credentials -d client_id=poimen-memory \
-d "client_secret=$SECRET" -d scope=openid | jq -r .access_token)
kubectl exec -it opensearch-0 -n poimen -- \
curl -k -H "Authorization: Bearer $TOKEN" https://localhost:9200/_cluster/health
```
**False pass:** Cluster health returns `green` but `DISABLE_SECURITY_PLUGIN=true` was set — JWT realm is not actually validating. Check by sending a garbage token; it must return 401.
## Artifacts
- `k8s/app/opensearch-deployment.yaml`
tasks/M8.2-dual-write-indexer.md
+83
View File
@@ -0,0 +1,83 @@
# M8.2 — Dual-write indexing pipeline
| Field | Value |
|---|---|
| Phase | M8 — Hybrid Search |
| Size | M — 1–2 days |
| Status | ⬜ |
| Flags | — |
| Spec | inlined below |
| Blocks | M8.4, M8.5 |
| Depends | M8.1 (OpenSearch running), M2.4 (pgvector repo) |
## Goal
When a document is ingested, write to **both** pgvector (embedding) and OpenSearch (raw text) atomically. Same `chunk_id` in both stores. If one write fails, log error but don't block the other — eventual consistency, not transactions.
## Design
**Unified ID mapping:** Both stores use the same `chunk_id` (UUID). The ingest worker generates the ID once, writes to both.
**Chunking policy:** 512-token chunks with 10% (51-token) overlap. Deterministic — same input always produces same chunks with same IDs.
**Dual write sequence:**
1. Chunk document (heading-boundary or fixed-size).
2. Generate embedding via LLM.
3. Write to pgvector: `INSERT INTO chunks (id, embedding, text, source, project, level, breadcrumb)`.
4. Write to OpenSearch: `PUT vault-{project}/_doc/{chunk_id}` with `{content, source, level, breadcrumb, project_id, indexed_at}`.
5. If OpenSearch write fails: log warning, mark chunk as `opensearch_pending=true` in pgvector. Background retry later.
**OpenSearch index mapping:**
```json
{
"content": {"type": "text", "analyzer": "standard", "boost": 2.0},
"section_title": {"type": "text", "boost": 1.5},
"breadcrumb": {"type": "keyword"},
"source": {"type": "keyword"},
"project_id": {"type": "keyword"},
"level": {"type": "keyword"},
"indexed_at": {"type": "date"}
}
```
**Deduplication:** Before writing, check `chunk_hash` (SHA256 of text). If hash exists and `is_indexed=true` in both stores, skip.
## Steps
1. Add `opensearch_pending` boolean column to `chunks` table (migration).
2. Update `IngestWorker::process_ingest()` to call OpenSearch after pgvector write.
3. Make `OpenSearchClient::index_document()` public, fix method signature.
4. Add background task: retry `opensearch_pending=true` chunks every 5 minutes.
5. Add dedup check before dual write.
## Acceptance
1. `mem ingest --dry-run` on a test doc shows chunks written to both stores.
2. Same `chunk_id` exists in both `SELECT id FROM chunks` and `GET vault-*/_doc/{id}`.
3. Kill OpenSearch mid-ingest: pgvector write succeeds, chunk marked `opensearch_pending=true`.
4. Restart OpenSearch: background retry picks up pending chunks within 5 minutes.
5. Re-ingest same document: dedup skips already-indexed chunks (0 new writes).
## Verify
```bash
# Ingest a test document
cargo run -- ingest --project test --source fixtures/refcorpus/small.md
# Check pgvector
psql -c "SELECT id, source, opensearch_pending FROM chunks WHERE project='test'"
# Check OpenSearch
curl -k -H "Authorization: Bearer $TOKEN" \
https://opensearch-internal:9200/vault-test/_search | jq '.hits.total'
# IDs must match
```
**False pass:** Both stores have data but with different IDs — the join on `chunk_id` finds zero matches. Assert `SELECT count(*) FROM chunks WHERE id IN (opensearch_ids)` equals total indexed.
## Artifacts
- Modified `crates/mem-cli/src/ingest_worker.rs`
- Modified `crates/mem-store/src/lib.rs` (migration)
- Modified `crates/mem-cli/src/opensearch_client.rs`
tasks/M8.3-query-optimizer.md
+80
View File
@@ -0,0 +1,80 @@
# M8.3 — Query optimizer: context construction + strategy routing
| Field | Value |
|---|---|
| Phase | M8 — Hybrid Search |
| Size | M — 1–2 days |
| Status | ⬜ |
| Flags | — |
| Spec | inlined below |
| Blocks | M8.5 |
| Depends | — (pure logic, no infra dependency) |
## Goal
Build a `QueryOptimizer` that analyses a raw user query and produces a `QueryContext` — normalised text, extracted entities, classified question type, and a routed `SearchStrategy`. This runs **before** any database call and determines which engines to use.
## Design
**6-stage pipeline:**
1. **Normalize** — lowercase, trim, collapse whitespace.
2. **Tokenize** — split on whitespace.
3. **Extract entities** — detect years (YYYY), quoted phrases (`"exact"`), tags (`#`, `@`).
4. **Analyse characteristics** — boolean flags: `has_special_syntax`, `has_date_filters`, `has_negation`.
5. **Classify question type** — one of: `Factual`, `Procedural`, `Comparative`, `Troubleshooting`, `Navigational`, `Open`.
6. **Route** — pick `SearchStrategy` with a confidence score (0.0–1.0).
**Routing rules:**
- Token count < 3 → `LexicalOnly` (BM25 handles keywords better than embeddings).
- Special syntax (`#tag`, `@mention`, `"phrase"`) → `LexicalOnly` (preserve exact tokens).
- Date filters present → `LexicalFirst` (narrow by date in OpenSearch, rerank in pgvector).
- Procedural / Troubleshooting → `Hybrid` (need both exact errors + semantic understanding).
- Navigational → `LexicalFirst` (finding specific docs).
- Default → `Hybrid`.
**Output struct:**
```rust
pub struct QueryContext {
pub raw_query: String,
pub normalized_query: String,
pub tokens: Vec<String>,
pub entities: HashMap<String, String>,
pub embedding: Option<Vec<f32>>, // filled later by worker
pub token_count: usize,
pub has_special_syntax: bool,
pub has_date_filters: bool,
pub has_negation: bool,
pub question_type: QuestionType,
pub search_strategy: SearchStrategy,
pub confidence: f32,
}
```
## Steps
1. Implement `QueryOptimizer` with `optimize_query(&str) -> Result<QueryContext>`.
2. Implement each stage as a private method.
3. Write unit tests for every routing rule (≥15 tests).
4. No async, no IO, no dependencies beyond std. Pure logic.
## Acceptance
1. `optimize_query("fix port")` → `LexicalOnly`, confidence ≥ 0.7.
2. `optimize_query("How do I fix kubernetes port 8080?")` → `Hybrid`, `Procedural`, confidence ≥ 0.9.
3. `optimize_query("#networking @devops policy")` → `LexicalOnly`, `has_special_syntax=true`.
4. `optimize_query("deployment failures in 2024")` → `LexicalFirst`, `has_date_filters=true`, entity `year=2024`.
5. `optimize_query("Compare Docker and Kubernetes")` → `Hybrid`, `Comparative`.
6. All 15+ tests pass.
## Verify
```bash
cargo test -p mem-cli query_optimizer:: -- --nocapture
```
**False pass:** Routing always returns `Hybrid` regardless of input. Check the short-query and special-syntax tests specifically — they must return non-Hybrid strategies.
## Artifacts
- `crates/mem-cli/src/query_optimizer.rs` (exists, needs cleanup + test fixes)
tasks/M8.4-rrf-fusion.md
+82
View File
@@ -0,0 +1,82 @@
# M8.4 — Reciprocal Rank Fusion engine
| Field | Value |
|---|---|
| Phase | M8 — Hybrid Search |
| Size | S — 0.5–1 day |
| Status | ⬜ |
| Flags | — |
| Spec | inlined below |
| Blocks | M8.5 |
| Depends | — (pure logic, no infra dependency) |
## Goal
Implement Reciprocal Rank Fusion (RRF) that merges two ranked lists from different scoring distributions into a single ranked list. No parameter tuning required.
## Why RRF, not weighted linear
pgvector returns cosine similarity in `[0.0, 1.0]`. OpenSearch BM25 returns unbounded scores in `[0, 50+]`. These distributions are incomparable.
**Weighted linear (0.6 * sem + 0.4 * lex)** requires min-max normalisation, which is fragile: one outlier score compresses all other scores to near-zero. It also requires choosing weights, which requires labelled data we don't have yet.
**RRF** ignores score magnitudes entirely. It uses only **rank positions**: the document that appears first in a list gets rank 1, second gets rank 2, etc. The formula is:
```
RRF_score(d) = Σ 1 / (k + rank_i(d))
lists
```
Where `k = 60` is a constant (academic standard, Cormack et al. 2009). A document in rank 1 of both lists gets `1/61 + 1/61 = 0.0328`. A document in rank 1 of only one list gets `1/61 = 0.0164`. The first always outranks the second, regardless of original score magnitudes.
## Design
```rust
pub struct RRFConfig {
pub k: f32, // 60.0 (constant, don't tune)
pub retrieve_k: usize, // 50 (top-K from each engine)
pub final_k: usize, // 10 (return top-K)
}
pub struct RRFFusion { config: RRFConfig }
impl RRFFusion {
pub fn fuse(
&self,
semantic: Vec<(String, f32)>, // (chunk_id, score) — sorted by score desc
lexical: Vec<(String, f32)>,
) -> Vec<(String, f32)>; // (chunk_id, rrf_score) — sorted desc, truncated
}
```
**Invariants:**
- Input lists must be pre-sorted by score descending (rank = position).
- Output is sorted by RRF score descending.
- Output length ≤ `final_k`.
- A document appearing in both lists always outranks one appearing in only one (given same rank positions).
## Steps
1. Implement `RRFFusion::fuse()`.
2. Implement `RRFFusion::normalize_scores()` as utility (for optional weighted-linear fallback).
3. Write tests: basic fusion, single-engine input, identical lists, disjoint lists, empty inputs.
## Acceptance
1. `fuse([(a,0.9),(b,0.8)], [(a,8.0),(c,7.0)])` → `a` is rank 1 (appears in both lists).
2. `fuse([(a,0.9)], [])` → `a` is rank 1 with score `1/61`.
3. `fuse([], [])` → empty result.
4. `fuse([(a,0.9),(b,0.8)], [(b,8.0),(a,7.0)])` → `a` and `b` have equal RRF scores (both appear in both at same combined rank sum). Either order is acceptable.
5. Output length never exceeds `final_k`.
## Verify
```bash
cargo test -p mem-cli rrf -- --nocapture
```
**False pass:** Fusion returns results but sorted by original score, not RRF score. Verify by checking that a document ranked #3 in semantic but #1 in lexical outranks a document ranked #1 in semantic but absent from lexical.
## Artifacts
- `crates/mem-cli/src/query_optimizer.rs` (RRFFusion struct, lives alongside QueryOptimizer)
tasks/M8.5-hybrid-query-worker.md
+134
View File
@@ -0,0 +1,134 @@
# M8.5 — Hybrid query worker: parallel retrieval + fusion
| Field | Value |
|---|---|
| Phase | M8 — Hybrid Search |
| Size | L — 2–3 days |
| Status | ⬜ |
| Flags | — |
| Spec | inlined below |
| Blocks | M8.6, M8.7 |
| Depends | M8.1 (OpenSearch running), M8.2 (dual-write), M8.3 (query optimizer), M8.4 (RRF) |
## Goal
`HybridQueryWorker` orchestrates parallel retrieval from pgvector and OpenSearch, applies RRF fusion, and returns ranked results with score breakdown and latency metrics. It replaces `QueryWorker` as the primary query engine behind `GET /memory/query`.
## Design
**Execution flow:**
1. Call `QueryOptimizer::optimize_query()` → get `QueryContext` with `SearchStrategy`.
2. Generate embedding via `EmbeddingsClient::embed()`.
3. Execute strategy:
- **Hybrid:** `tokio::try_join!` pgvector top-50 + OpenSearch top-50. Fuse with RRF.
- **LexicalFirst:** OpenSearch top-200 → extract IDs → pgvector `WHERE id IN (...)` top-10.
- **SemanticOnly:** pgvector top-50 (fallback if OpenSearch unavailable).
- **LexicalOnly:** OpenSearch top-50 (fallback if embedding model unavailable).
4. Build response with score breakdown.
**Critical: use actual VectorStore API.**
The existing `VectorStore` exposes `search_l1(project, &embedding, limit)` and `search_l2(project, &embedding)`. The worker must call these — not invented methods.
For the cascading strategy (`LexicalFirst`), a new `search_l1_by_ids(project, &embedding, limit, &[chunk_id])` method is needed on `VectorStore`. This is a filtered pgvector query:
```sql
SELECT id, 1 - (embedding <=> $1) as score, text, source
FROM chunks
WHERE project = $2 AND id = ANY($3)
ORDER BY embedding <=> $1
LIMIT $4
```
**OpenSearch query:**
```json
{
"size": 50,
"query": {
"bool": {
"must": [{
"multi_match": {
"query": "...",
"fields": ["content^2", "section_title^1.5", "breadcrumb", "source"],
"type": "best_fields",
"fuzziness": "AUTO"
}
}],
"filter": [
{"term": {"project_id": "..."}},
{"terms": {"level": ["L0", "L1", "L2"]}}
]
}
}
}
```
**Response struct:**
```rust
pub struct HybridQueryResponse {
pub query: String,
pub project: String,
pub search_strategy: String,
pub results: Vec<HybridQueryResult>,
pub metrics: QueryMetrics,
}
pub struct HybridQueryResult {
pub id: String,
pub text: String,
pub source: String,
pub level: String,
pub breadcrumb: Vec<String>,
pub final_score: f32,
pub semantic_rank: Option<usize>,
pub lexical_rank: Option<usize>,
pub fusion_method: String,
}
pub struct QueryMetrics {
pub total_time_ms: u128,
pub semantic_time_ms: Option<u128>,
pub lexical_time_ms: Option<u128>,
pub fusion_time_ms: u128,
pub semantic_candidates: Option<usize>,
pub lexical_candidates: Option<usize>,
pub final_count: usize,
}
```
## Steps
1. Add `search_l1_by_ids()` to `VectorStore` (new SQL query with `id = ANY($3)`).
2. Make `OpenSearchClient::lexical_search()` public.
3. Implement `HybridQueryWorker::new()` taking `VectorStore`, `EmbeddingsClient`, `Option<OpenSearchClient>`.
4. Implement `query()` method with strategy dispatch.
5. Implement `retrieve_hybrid()` using `tokio::try_join!`.
6. Implement `retrieve_cascading()` (2-stage).
7. Implement fallback methods (`retrieve_semantic()`, `retrieve_lexical()`).
8. Wire RRF fusion into the result pipeline.
9. Build response with per-result rank tracking.
10. Write integration tests with mock VectorStore + mock OpenSearch.
## Acceptance
1. Hybrid query returns results from **both** engines — check `semantic_rank` and `lexical_rank` are both `Some` for documents appearing in both lists.
2. Cascading query's pgvector call receives only IDs from the OpenSearch narrowing step — verify with query log or mock.
3. If `OpenSearchClient` is `None`, strategy automatically falls back to `SemanticOnly`.
4. If embedding generation fails, strategy falls back to `LexicalOnly` (if OpenSearch available) or returns error.
5. `metrics.total_time_ms` is populated and < 500ms for test fixtures.
6. `metrics.semantic_time_ms` and `lexical_time_ms` are roughly equal (parallel execution, not serial).
7. Results are sorted by `final_score` descending.
8. `final_count` ≤ `RRFConfig.final_k`.
## Verify
```bash
cargo test -p mem-cli hybrid_query -- --nocapture
```
**False pass:** Worker always returns semantic-only results even when strategy is `Hybrid`. Check by asserting `lexical_rank.is_some()` on at least one result when OpenSearch is configured. Another false pass: serial execution disguised as parallel — assert `max(semantic_time_ms, lexical_time_ms) ≈ total - fusion_time_ms`, not `sum`.
## Artifacts
- `crates/mem-cli/src/hybrid_query_worker.rs` (rewrite from current stub)
- Modified `crates/mem-store/src/lib.rs` (add `search_l1_by_ids`)
- Modified `crates/mem-cli/src/opensearch_client.rs` (make `lexical_search` pub)
tasks/M8.6-query-endpoint-upgrade.md
+127
View File
@@ -0,0 +1,127 @@
# M8.6 — Upgrade GET /memory/query to hybrid with fallback
| Field | Value |
|---|---|
| Phase | M8 — Hybrid Search |
| Size | M — 1–2 days |
| Status | ⬜ |
| Flags | — |
| Spec | inlined below |
| Blocks | M8.8, M8.9 |
| Depends | M8.5 (hybrid query worker compiles and passes tests) |
## Goal
Replace the `QueryWorker` call in `GET /memory/query` with `HybridQueryWorker`. Add `?method=` parameter for explicit strategy override. Implement fallback chain: hybrid → semantic → error.
## Design
**Updated request:**
```
GET /memory/query?query=...&project=...&limit=10&method=hybrid
Authorization: Bearer <JWT>
```
New parameter:
- `method` (optional) — `hybrid` (default), `semantic`, `lexical`. If omitted, `QueryOptimizer` decides.
**Handler logic:**
```rust
async fn query_handler(...) -> HttpResponse {
let (claims, token) = validate_auth(...)?;
check_capability(&claims, "memory:read")?;
check_rate_limit(&claims, &state, "/memory/query")?;
let method_override = query.get("method").map(|m| match m.as_str() {
"semantic" => SearchStrategy::SemanticOnly,
"lexical" => SearchStrategy::LexicalOnly,
_ => SearchStrategy::Hybrid, // includes "hybrid" and unknown values
});
// Try hybrid worker first
if let Some(ref hybrid) = state.hybrid_query_worker {
match hybrid.query(&project, &question, limit, &token, method_override).await {
Ok(response) => return HttpResponse::Ok().json(response),
Err(e) => {
tracing::warn!("hybrid query failed, falling back: {}", e);
}
}
}
// Fallback: existing semantic-only worker
match state.query_worker.query(&project, &question, Some(limit)).await {
Ok(results) => HttpResponse::Ok().json(json!({
"query": question,
"project": project,
"search_strategy": "semantic_fallback",
"results": results,
})),
Err(e) => HttpResponse::InternalServerError().json(json!({"error": "query_failed"})),
}
}
```
**AppState change:**
```rust
pub struct AppState {
// ... existing fields ...
pub hybrid_query_worker: Option<Arc<HybridQueryWorker>>, // None if OpenSearch not configured
}
```
Constructed in `start_server()`:
```rust
let hybrid = if std::env::var("OPENSEARCH_HOSTS").is_ok() {
let os_client = OpenSearchClient::new(hosts);
Some(Arc::new(HybridQueryWorker::new(vector_store, embeddings, Some(Arc::new(os_client)))))
} else {
// No OpenSearch configured — hybrid worker without lexical
Some(Arc::new(HybridQueryWorker::new(vector_store, embeddings, None)))
};
```
**Backward compatibility:** If `OPENSEARCH_HOSTS` is not set, the worker still works but always uses `SemanticOnly`. Existing clients see the same results with an added `search_strategy` field.
## Steps
1. Add `hybrid_query_worker` field to `AppState`.
2. Construct `HybridQueryWorker` in `start_server()`, gated on `OPENSEARCH_HOSTS` env.
3. Update `query_handler()` with method override + fallback chain.
4. Add `?method=` query parameter parsing.
5. Update response format to include `search_strategy` and `metrics`.
6. Write integration test: query with `method=hybrid`, verify response has `metrics`.
7. Write integration test: query without OpenSearch, verify fallback to semantic.
## Acceptance
1. `GET /memory/query?query=test&project=p` returns `"search_strategy": "Hybrid"` when OpenSearch configured.
2. `GET /memory/query?query=test&project=p&method=semantic` forces semantic-only, response says `"search_strategy": "SemanticOnly"`.
3. If OpenSearch is down, query still succeeds with `"search_strategy": "semantic_fallback"`.
4. Response includes `metrics` block with timing data.
5. Existing clients that don't send `method` param get the same results as before (backward compat).
6. JWT token is forwarded to OpenSearch (not a new token, not admin credentials).
## Verify
```bash
# With OpenSearch running
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:8080/memory/query?query=kubernetes+port&project=poimen" | jq .search_strategy
# Should output: "Hybrid"
# Force semantic
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:8080/memory/query?query=kubernetes+port&project=poimen&method=semantic" | jq .search_strategy
# Should output: "SemanticOnly"
# Kill OpenSearch, retry
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:8080/memory/query?query=kubernetes+port&project=poimen" | jq .search_strategy
# Should output: "semantic_fallback"
```
**False pass:** Handler catches the hybrid error silently and always falls back to semantic — user never sees hybrid results even when OpenSearch is healthy. Assert that with OpenSearch up, `metrics.lexical_candidates` is `Some(n)` where `n > 0`.
## Artifacts
- Modified `crates/mem-cli/src/http_server.rs`
tasks/M8.7-index-optimization.md
+103
View File
@@ -0,0 +1,103 @@
# M8.7 — Index tuning: HNSW parameters + OpenSearch analyzers
| Field | Value |
|---|---|
| Phase | M8 — Hybrid Search |
| Size | M — 1–2 days |
| Status | ⬜ |
| Flags | — |
| Spec | inlined below |
| Blocks | M8.9 |
| Depends | M8.2 (data in both stores), M8.5 (can query both stores) |
## Goal
Tune pgvector index parameters and OpenSearch analyzers for retrieval accuracy. Measure baseline NDCG before and after tuning. This is engineering, not research — change one parameter, measure, keep or revert.
## Design
### pgvector tuning
**Current:** `ivfflat` index with default `lists`.
**Target:** Switch to `hnsw` index (pgvector 0.5.0+). HNSW provides better recall than IVFFlat at the cost of slower index builds and more memory.
```sql
-- Drop old index
DROP INDEX IF EXISTS chunks_embedding_idx;
-- Create HNSW index
CREATE INDEX chunks_embedding_hnsw_idx
ON chunks USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);
```
Parameters:
- `m = 16` — max connections per node (default 16, higher = better recall, more memory).
- `ef_construction = 64` — build-time search width (default 64, higher = better recall, slower build).
- `ef_search = 40` — query-time search width (set via `SET hnsw.ef_search = 40`).
**Tuning approach:**
1. Baseline: measure recall@50 with IVFFlat.
2. Switch to HNSW with defaults.
3. Measure recall@50 again.
4. If recall@50 ≥ 0.95, keep defaults. Otherwise increase `ef_construction` to 128.
### OpenSearch tuning
**Analyzer changes:**
- Add `edge_ngram` tokenizer for typo tolerance on `content` field.
- Add `synonym` filter for common abbreviations: `k8s → kubernetes`, `db → database`, `cfg → config`.
- Keep `standard` analyzer as primary, add `search_analyzer` for queries.
**Field boost tuning:**
- `content^2.0` (default — most important).
- `section_title^1.8` (headings are very relevant).
- `source^1.0` (file paths are useful but shouldn't dominate).
- `breadcrumb^0.8` (context, not content).
**BM25 parameters:**
- `k1 = 1.2` (term frequency saturation — default is fine).
- `b = 0.75` (length normalization — default is fine).
- Don't tune these unless baseline NDCG < 0.7.
## Steps
1. Create test query set: 20 queries with known-relevant documents.
2. Measure baseline NDCG@10 for pgvector (semantic-only) and OpenSearch (lexical-only).
3. Switch pgvector from IVFFlat to HNSW. Measure NDCG@10 again.
4. Update OpenSearch index template with synonym filter + edge_ngram. Reindex. Measure.
5. Record all measurements in `docs/INDEX_TUNING_RESULTS.md`.
6. Keep changes that improve NDCG. Revert changes that don't.
## Acceptance
1. pgvector uses HNSW index (verify with `\d+ chunks` in psql).
2. OpenSearch index template includes synonym filter.
3. NDCG@10 measurements recorded for before/after each change.
4. No regression: post-tuning NDCG ≥ pre-tuning NDCG for both engines.
5. pgvector query latency < 150ms (p95) after HNSW switch.
6. OpenSearch query latency < 100ms (p95) after analyzer changes.
## Verify
```bash
# Check pgvector index type
psql -c "\d+ chunks" | grep hnsw
# Check OpenSearch analyzer
curl -k -H "Authorization: Bearer $TOKEN" \
https://opensearch-internal:9200/vault-test/_settings | jq '.*.settings.index.analysis'
# Run NDCG measurement
cargo run -- bench-search --queries fixtures/search_queries.yaml --output docs/INDEX_TUNING_RESULTS.md
```
**False pass:** HNSW index created but `ef_search` set to 1, making recall worse than IVFFlat. Check recall@50 explicitly — it should be ≥ 0.95.
## Artifacts
- SQL migration (drop IVFFlat, create HNSW)
- Updated OpenSearch index template
- `docs/INDEX_TUNING_RESULTS.md` (measurements)
- `fixtures/search_queries.yaml` (test query set)
tasks/M8.8-accuracy-benchmarks.md
+105
View File
@@ -0,0 +1,105 @@
# M8.8 — Accuracy benchmarks: NDCG, MRR, Precision/Recall
| Field | Value |
|---|---|
| Phase | M8 — Hybrid Search |
| Size | M — 1–2 days |
| Status | ⬜ |
| Flags | — |
| Spec | inlined below |
| Blocks | M8.9 |
| Depends | M8.6 (hybrid endpoint working), M8.7 (indices tuned) |
## Goal
Build a repeatable benchmark harness that measures retrieval accuracy. Run it against semantic-only, lexical-only, and hybrid strategies. Produce a comparison table that proves hybrid is better (or shows where it isn't).
## Design
**Test fixture format:**
```yaml
# fixtures/search_queries.yaml
queries:
- id: q1
text: "How do I fix kubernetes port 8080 conflict?"
relevant_docs: ["runbooks/networking.md"]
category: troubleshooting
- id: q2
text: "What is a StatefulSet?"
relevant_docs: ["docs/kubernetes-concepts.md", "docs/statefulsets.md"]
category: factual
- id: q3
text: "#networking firewall rules"
relevant_docs: ["docs/network-policies.md"]
category: navigational
```
**Metrics implemented:**
- **NDCG@10** — Are relevant docs ranked near the top? (0.0 worst, 1.0 perfect).
- **MRR** — How early is the first relevant doc? (1/rank of first hit).
- **Precision@5** — What fraction of top-5 results are relevant?
- **Recall@10** — What fraction of all relevant docs appear in top-10?
**Benchmark runner:**
```rust
pub struct BenchmarkResult {
pub strategy: String, // "hybrid", "semantic", "lexical"
pub avg_ndcg: f32,
pub avg_mrr: f32,
pub avg_precision_at_5: f32,
pub avg_recall_at_10: f32,
pub avg_latency_ms: f32,
pub per_query: Vec<QueryBenchmark>,
}
```
**Output:** Markdown table written to `docs/BENCHMARK_RESULTS.md`.
```markdown
| Strategy | NDCG@10 | MRR | P@5 | R@10 | Latency (ms) |
|----------|---------|-----|-----|------|-------------|
| semantic | 0.72 | 0.65 | 0.60 | 0.75 | 95 |
| lexical | 0.68 | 0.70 | 0.55 | 0.70 | 62 |
| hybrid | 0.87 | 0.82 | 0.78 | 0.90 | 210 |
```
## Steps
1. Create `fixtures/search_queries.yaml` with ≥ 20 queries across categories.
2. Ingest corresponding test documents into both stores.
3. Implement `BenchmarkRunner` that:
a. Loads fixture file.
b. Runs each query against each strategy.
c. Computes NDCG, MRR, Precision, Recall per query.
d. Averages across queries.
e. Writes results to markdown.
4. Implement as CLI command: `cargo run -- bench-search --queries <path> --output <path>`.
5. Run benchmarks. Record results.
## Acceptance
1. Benchmark runs to completion on 20+ queries × 3 strategies = 60+ query executions.
2. Output markdown table has all 5 columns populated.
3. Hybrid NDCG@10 ≥ max(semantic NDCG, lexical NDCG) — hybrid must not be worse than best single engine.
4. Per-query results show which categories benefit most from hybrid (expected: troubleshooting, procedural).
5. If hybrid is worse on any category, document why and whether it matters.
## Verify
```bash
cargo run -- bench-search \
--queries fixtures/search_queries.yaml \
--output docs/BENCHMARK_RESULTS.md
cat docs/BENCHMARK_RESULTS.md
```
**False pass:** Benchmark uses the same documents for queries and ground truth (trivial exact match). Ensure queries use **natural language** and ground truth docs use **technical content** — the match should be semantic, not string equality.
## Artifacts
- `fixtures/search_queries.yaml`
- `crates/mem-cli/src/bench_search.rs` (new)
- `docs/BENCHMARK_RESULTS.md` (output)
tasks/M8.9-m8-gate.md
+120
View File
@@ -0,0 +1,120 @@
# M8.9 — M8 composition gate: hybrid search proves its value
| Field | Value |
|---|---|
| Phase | M8 — Hybrid Search |
| Size | M — 1 day |
| Status | ⬜ |
| Flags | gate |
| Spec | inlined below |
| Blocks | — |
| Depends | M8.1–M8.8 all ✅ |
## Goal
Prove the hybrid search system works end-to-end and is measurably better than semantic-only. This gate verifies the **composition** — individual tasks pass their own tests, but only the gate proves they compose correctly.
## Properties to verify
### P1: Dual-write consistency
Every chunk in pgvector has a corresponding document in OpenSearch with the same ID, and vice versa. Zero orphans.
```sql
-- pgvector IDs not in OpenSearch
SELECT id FROM chunks WHERE opensearch_pending = true;
-- Must return 0 rows (after background retry has run)
```
```bash
# OpenSearch document count must equal pgvector chunk count for same project
PG_COUNT=$(psql -t -c "SELECT count(*) FROM chunks WHERE project='test'")
OS_COUNT=$(curl -sk "https://opensearch:9200/vault-test/_count" | jq .count)
# PG_COUNT == OS_COUNT
```
### P2: Hybrid outperforms single-engine
From `docs/BENCHMARK_RESULTS.md`:
- Hybrid NDCG@10 > semantic-only NDCG@10.
- Hybrid NDCG@10 > lexical-only NDCG@10.
- If this fails for a specific query category, it must be documented with reasoning.
### P3: Fallback works under failure
1. Stop OpenSearch. Query endpoint still responds with semantic results.
2. Start OpenSearch. Query endpoint returns hybrid results.
3. Response `search_strategy` field accurately reports which mode was used.
### P4: JWT auth enforced end-to-end
1. Query without token → 401.
2. Query with valid token → 200.
3. OpenSearch rejects requests from non-Memory-Service pods (NetworkPolicy).
4. JWT token forwarded from Memory Service to OpenSearch (not admin credentials).
### P5: No regression on existing tests
All pre-existing tests still pass. `cargo test` green. No `#[ignore]` added in M8.
### P6: Latency budget met
- Hybrid query: p95 < 500ms.
- Semantic-only query: p95 < 200ms (must not regress from adding hybrid path).
- Fallback to semantic: p95 < 250ms (minimal overhead from failed OpenSearch attempt).
## Gate test
```bash
#!/bin/bash
set -euo pipefail
echo "=== M8 Gate: Hybrid Search ==="
# P5: All tests pass
cargo test 2>&1 | tail -1
# Expected: test result: ok. X passed; 0 failed
# P1: Dual-write consistency
PG=$(psql -t -c "SELECT count(*) FROM chunks WHERE project='test' AND opensearch_pending=false")
OS=$(curl -sk "https://opensearch:9200/vault-test/_count" | jq .count)
[ "$PG" -eq "$OS" ] && echo "P1 PASS: $PG chunks in both stores" || echo "P1 FAIL: pg=$PG os=$OS"
# P2: Hybrid > single-engine
grep -A1 "hybrid" docs/BENCHMARK_RESULTS.md | grep -oP '[\d.]+' | head -1
# Must be highest NDCG in the table
# P3: Fallback
kubectl scale statefulset/opensearch -n poimen --replicas=0
sleep 5
STRATEGY=$(curl -s -H "Authorization: Bearer $TOKEN" \
"http://localhost:8080/memory/query?query=test&project=test" | jq -r .search_strategy)
[ "$STRATEGY" = "semantic_fallback" ] && echo "P3 PASS: fallback works" || echo "P3 FAIL: $STRATEGY"
kubectl scale statefulset/opensearch -n poimen --replicas=2
sleep 30
STRATEGY=$(curl -s -H "Authorization: Bearer $TOKEN" \
"http://localhost:8080/memory/query?query=test&project=test" | jq -r .search_strategy)
[ "$STRATEGY" = "Hybrid" ] && echo "P3 PASS: hybrid restored" || echo "P3 FAIL: $STRATEGY"
# P4: Auth
STATUS=$(curl -s -o /dev/null -w '%{http_code}' "http://localhost:8080/memory/query?query=test&project=test")
[ "$STATUS" = "401" ] && echo "P4 PASS: no-auth rejected" || echo "P4 FAIL: $STATUS"
echo "=== M8 Gate Complete ==="
```
## Acceptance
All six properties pass. If P2 fails (hybrid not better), the gate does NOT pass — go back and fix M8.7 (index tuning) or M8.4 (fusion algorithm).
## False passes to check
1. **P1 looks green but IDs don't match.** Run a JOIN, not just count comparison.
2. **P3 looks green but fallback latency is 30s** (timeout, not fast fail). Check p95 < 250ms.
3. **P5 looks green but test count dropped.** Compare `cargo test 2>&1 | grep 'test result'` against last known count (currently 239+).
## Artifacts
- Gate script (inline above)
- `docs/BENCHMARK_RESULTS.md` (from M8.8)
- All M8.1–M8.8 artifacts