Story Crater Bot
959c596b1d
Deleted 31 completed task files: - M0.x: 8 tasks (cargo, domain types, recordsource, tokenizer, adapters, gate) - M1.x: 8 tasks (llm-chat, standing-query, prompt template, parser, loop, log, e2e, gate) - M3.x: 4 tasks (l2-synthesis, rerank, mem-query, gate) - M3.5.x: 8 tasks (http-server, ingest, query, federation, skills, projects, rate-limiting, gate) - M3.6.1: DocCorpusSource (heading-boundary chunking) - M4.1-2: skill-draft, derived-filter Updated INDEX.md: - Removed M0 & M1 phase sections (archived in git history) - Updated progress table: 65 active tasks (42✅ + 2🟡 + 21⬜) - Updated status: M0/M1 complete, M3/M3.5 gates passing, M4.1-2 done - Noted M3.5.10 JWT auth implementation complete (awaiting image rollout) - Cleaned up broken links to deleted task files Total test count: 239 passing, 2 ignored (up from 196 at M3.4) Ready for M4.3 gate composition, M5 post-training, M7 source connectors.
279 lines
15 KiB
Markdown
279 lines
15 KiB
Markdown
# poimen-memory — task board
|
||
|
||
64 tasks — 54 build tasks plus **10 composition gates**, one per phase. One file
|
||
per task, **self-contained**: inlined design facts, executable steps, acceptance
|
||
criteria, a `Verify` section written for someone who did not build the thing, and
|
||
the traps worth naming. Reading `DESIGN.md` is not required to do a task — it is
|
||
linked as background only.
|
||
|
||
Each `Verify` section names the harness, the integration test with numbered
|
||
assertions, the command to run, and the **false pass** — the shape of test that
|
||
goes green while the feature is broken. Treat the false-pass list as part of the
|
||
acceptance criteria, not commentary.
|
||
|
||
## Ordering — declared, never derived
|
||
|
||
**Phase order is the list below. Task ids are opaque and frozen.**
|
||
|
||
`M0.3` is `M0.3` forever, in whatever phase it currently sits, because its
|
||
artifacts and cross-references key on that id. New tasks take new ids rather than
|
||
renumbering neighbours. This is the `StepId` rule applied to the board itself —
|
||
a board that renumbers to reorder has the bug it warns its own users about.
|
||
|
||
No phase starts until its predecessor's gate is green. The `gate` task at the end
|
||
of each phase **is** that gate: it proves the phase's parts compose and that its
|
||
swappable parts are genuinely swappable. Every build task is verified alone; the
|
||
gate verifies the properties no single task owns.
|
||
|
||
## Two rules this board exists to protect
|
||
|
||
**1. The JSONL log is authoritative; the vault and the vector index are
|
||
projections.** Anything that cannot be dropped and rebuilt byte-identically from
|
||
the log has hidden inputs, and that is a bug. `M2.8` is the gate that enforces it.
|
||
|
||
**2. The update gate must discriminate, not summarize.** Agent transcripts are
|
||
~43% tool results and mostly evidence-free. A gate that accepts most chunks is an
|
||
expensive summarizer that will reproduce the memory-explosion failure the whole
|
||
design exists to avoid. `M1.8` is the gate that enforces it, and **update-rate is
|
||
the single number to watch.**
|
||
|
||
## Verification practice — script first, source second
|
||
|
||
A task is verified by running its command and reading the output, then opening
|
||
the source. Reviewing the diff first is how an assertion that was quietly dropped
|
||
still gets called done: the code looks right, and nothing proves the test ran.
|
||
|
||
Numbered assertion N in a `Verify` section is test fn `aN_<slug>`. The numbering
|
||
is the contract — a test fn that does not exist should report missing rather than
|
||
be silently absent from a green summary.
|
||
|
||
`cargo test` reporting `ok` with zero tests run is not a pass.
|
||
|
||
## Progress
|
||
|
||
**Source of truth is the `Status` field in each task file.** The tables below
|
||
mirror it; a status changed here and not there is a lie.
|
||
|
||
Legend: ⬜ not started · 🟡 in progress · ✅ done · ⛔ blocked
|
||
|
||
| # | Phase | Ids | Tasks | ✅ | 🟡 | ⬜ | Gate |
|
||
|---|---|---|---|---|---|---|---|
|
||
| 1 | Read-only spine | M0.x | 8 | 8 | 0 | 0 | ✅ M0.8 |
|
||
| 2 | Gated loop at L1 | M1.x | 8 | 8 | 0 | 0 | ✅ M1.8 |
|
||
| 4 | L2 synthesis + retrieval | M3.x | 4 | 4 | 0 | 0 | ✅ M3.4 |
|
||
| 4.5 | Distributed API Layer | M3.5.x | 10 | 9 | 0 | 1 | ✅ M3.5.8 |
|
||
| 5 | Skills | M4.x | 3 | 2 | 0 | 1 | ⬜ M4.3 |
|
||
| 5.5 | Reference corpora | M3.6.x | 6 | 1 | 0 | 5 | ⬜ M3.6.6 |
|
||
| 5.6 | Tool context | M3.7.x | 6 | 0 | 2 | 4 | ⬜ M3.7.6 |
|
||
| 6 | Post-training | M5.x | 6 | 0 | 0 | 6 | ⬜ M5.6 |
|
||
| 7 | agent-manager migration | M6.x | 6 | 0 | 0 | 6 | ⬜ M6.6 |
|
||
| 8 | Source connectors | M7.x | 10 | 0 | 0 | 10 | ⬜ M7.10 |
|
||
| 9 | Hybrid search | M8.x | 9 | 0 | 0 | 9 | ⬜ M8.9 |
|
||
| | **Total** | | **65** | **42** | **2** | **21** | 5/10 green |
|
||
|
||
**Current status — 2025-01-27.** Completed phases M0.x, M1.x fully archived (16/16 tasks). M3.x (4/4 ✅), M3.5.x (9/10 ✅ + 1 in-progress M3.5.9).
|
||
M3.5.10 JWT auth integration ✅ complete with Authentik OIDC validation.
|
||
M4.1-2 Skills ✅ done (skill drafting + derived filter). M3.6.1 DocCorpusSource ✅.
|
||
All completed task files archived from `/tasks/` folder. INDEX.md cleaned to reflect active work only.
|
||
Significant early work for M3.7: `mem-core/src/lesson.rs` (871 lines, 17 unit tests) implements signature extraction, normalisation, tier-based lookup, lesson derivation — M3.7.7, M3.7.5 are 🟡. `mem-cli/src/lessons_cmd.rs` (311 lines), `mem-ingest/src/derived_filter.rs` (220 lines) provides working `mem capture|resolve|lookup|materialize`.
|
||
**Tests: 239 passing, 2 ignored.** Ready to tackle M4.3 gate (skills composition), M5 (post-training), M7 (source connectors).
|
||
|
||
`M2.2` (CNPG manifest), `M5.4` (vLLM+LoRA), `M3.5.9` (git refs), and
|
||
`M6.x` (agent-manager migration) are homelab/infra work independent of prior phases, can start parallel.
|
||
|
||
**M6 is a different repo, not a dependency of M0-M5.** It migrates
|
||
`github.com/Riotpiaole/agent-manager`'s session store (a separate Go CLI tool,
|
||
unrelated to this project's own memory system) from local sqlite to its own
|
||
dedicated CNPG cluster. It rides in this board because it's homelab work
|
||
happening alongside M2.2/M5.4, and because the two projects' Postgres schemas
|
||
landing in the same cluster around the same time need to look like siblings,
|
||
not strangers — see M6.6's convention-consistency check.
|
||
|
||
## ✅ Archived Phases
|
||
|
||
Completed and archived: **M0.x (8/8)**, **M1.x (8/8)** — all task files deleted from `/tasks/` after verification. See git log for historical record and `CLAUDE.md` for session context.
|
||
|
||
## 3 — Projections · M2.x
|
||
|
||
**Status:** In progress · 5/8 done, 3 pending.
|
||
|
||
M2.1, M2.3, M2.4, M2.5 ✅ complete. M2.2 (CNPG manifest), M2.6, M2.7 remain. M2.8 gate awaits dependency clearance.
|
||
|
||
## 4 — L2 synthesis and retrieval · M3.x
|
||
|
||
**Status:** ✅ Complete · 4/4 tasks done. Gate M3.4 passing. Task files archived.
|
||
|
||
## 4.5 — Distributed API Layer · M3.5.x
|
||
|
||
Homelab frontend integration: HTTP facade via `api.riotpiao.com`. Runs in parallel with M4 and M5 after M3.4 green.
|
||
|
||
**Status:** 9/10 done · M3.5.8 gate ✅ passing. M3.5.1–8 archived (task files deleted). M3.5.9 (git-aware refs) and M3.5.10 (JWT/OIDC auth) remain. M3.5.10 implementation ✅ complete: Authentik OIDC provider, RS256 validation, capability-based access control. Awaiting new Docker image rollout to pods.
|
||
|
||
## 5 — Skills · M4.x
|
||
|
||
**Status:** 2/3 done · M4.1 (skill draft) ✅ and M4.2 (derived filter) ✅ archived. Pending M4.3 gate composition.
|
||
|
||
`mem skill draft --project <proj> --from <query-id>` writes to `vault/skills/_drafts/`. Dry-run mode supported. Shingle matcher (M4.2) provides derived LessonSource filtering. M4.1-2 task files deleted after archival.
|
||
|
||
## 5.5 — Reference corpora · M3.6.x
|
||
|
||
Documentation the local models are weak at — `kubectl`, `tea` — made retrievable
|
||
as level **R**: embedded and indexed, never evidence. Ids are `M3.6.x` and stay
|
||
`M3.6.x`; the phase sits here rather than at 4.6 because [M3.6.4](M3.6.4-reference-cycle-guard.md)
|
||
extends M4.2's matcher instead of duplicating it, and because skills are the
|
||
better answer to the same problem and should exist first.
|
||
|
||
**The load-bearing property is a negative one.** Adding a corpus must not change
|
||
update-rate, must not change default query output, and must not put an R node in
|
||
any provenance chain. R bypasses the recurrence structurally — `run_loop` needs a
|
||
`Query` and a corpus has none — not by a flag. [M3.6.6](M3.6.6-m3.6-gate.md)
|
||
asserts M1.8's numbers are *unchanged*, not merely still-passing, because
|
||
documentation fed to the gate would lower update-rate and make M1.8 easier to
|
||
clear while the memory got worse.
|
||
|
||
| Task | Title | Size | Flags | Status |
|
||
|---|---|---|---|---|
|
||
| M3.6.1 | `DocCorpusSource` + heading chunking | M | — | ✅ |
|
||
| [M3.6.2](M3.6.2-level-r-storage.md) | Level R: log, index, vault, rebuild parity | M | — | ⬜ |
|
||
| [M3.6.3](M3.6.3-mem-ref-cli.md) | `mem ref` — replace-on-change corpus management | M | — | ⬜ |
|
||
| [M3.6.4](M3.6.4-reference-cycle-guard.md) | Reference text cannot re-enter as evidence | M | — | ⬜ |
|
||
| [M3.6.5](M3.6.5-query-levels-and-floor.md) | Query: filter-then-recall, R opt-in, floor | M | — | ⬜ |
|
||
| [M3.6.6](M3.6.6-m3.6-gate.md) | **M3.6 composition gate** | M | gate | ⬜ |
|
||
|
||
## 5.6 — Tool context · M3.7.x
|
||
|
||
Answers *"what do we already know about this failure, tool or task"* over HTTP.
|
||
Consumers are `pi`, curl, or an MCP call — nothing here executes a tool, and
|
||
nothing here serves a tool catalog, because every caller already holds its own
|
||
MCP schemas.
|
||
|
||
**Three tiers, cheapest first.** An exact hash hit on a normalised failure
|
||
signature means *this happened here before*; a symptom-vector match means
|
||
*something similar did*; the R corpus means *nobody here has hit this, read the
|
||
docs*. The tier is a field in the response, because those three answers must not
|
||
arrive in the same register.
|
||
|
||
**The two tasks that make it work are the least obvious ones.**
|
||
[M3.7.7](M3.7.7-signature-extraction.md) decides whether tier 1 ever fires — if
|
||
normalisation leaves a timestamp in, the same failure never hashes twice and the
|
||
system silently degrades to vector search.
|
||
[M3.7.8](M3.7.8-symptom-projection.md) closes the gap between memories written as
|
||
answers and queries that arrive as stack traces. Both fail invisibly, which is why
|
||
[M3.7.6](M3.7.6-m3.7-gate.md) ablates them rather than trusting an end-to-end
|
||
green.
|
||
|
||
Ids are `M3.7.x` and frozen. `M3.7.1` and `M3.7.2` were a tool-catalog surface,
|
||
deleted before implementation once the consumer was settled; their ids are retired
|
||
rather than reused.
|
||
|
||
| Task | Title | Size | Flags | Status |
|
||
|---|---|---|---|---|
|
||
| [M3.7.3](M3.7.3-skill-matching.md) | `GET /memory/skills?task=` — match a subset | M | — | ⬜ |
|
||
| [M3.7.4](M3.7.4-context-endpoint.md) | `/memory/context` — three-tier lookup | M | — | ⬜ |
|
||
| [M3.7.5](M3.7.5-tool-failure-learning.md) | `tool-failures` standing query | M | — | 🟡 |
|
||
| [M3.7.6](M3.7.6-m3.7-gate.md) | **M3.7 composition gate** | M | gate | ⬜ |
|
||
| [M3.7.7](M3.7.7-signature-extraction.md) | Failure signature extraction + normalisation | M | — | 🟡 |
|
||
| [M3.7.8](M3.7.8-symptom-projection.md) | Symptom projection at ingest | M | — | ⬜ |
|
||
|
||
## 6 — Post-training · M5.x
|
||
|
||
Python, separate from the Rust workspace. The boundary is the JSONL log.
|
||
|
||
| Task | Title | Size | Flags | Status |
|
||
|---|---|---|---|---|
|
||
| [M5.1](M5.1-evidence-labeler.md) | `mem label` — evidence labeler | M | — | ⬜ |
|
||
| [M5.2](M5.2-labeler-calibration.md) | Labeler calibration | M | — | ⬜ |
|
||
| [M5.3](M5.3-training-corpus-export.md) | Training corpus export | M | — | ⬜ |
|
||
| [M5.4](M5.4-vllm-lora-serving.md) | vLLM + `--enable-lora` | L | homelab | ⬜ |
|
||
| [M5.5](M5.5-verl-training-loop.md) | verl training loop | L | — | ⬜ |
|
||
| [M5.6](M5.6-m5-gate.md) | **M5 composition gate** | L | gate | ⬜ |
|
||
|
||
## 7 — agent-manager migration · M6.x
|
||
|
||
Separate repo (`github.com/Riotpiaole/agent-manager`, fork branch
|
||
`add-headless-spawn`), separate cluster resource, no Rust/GRU-Mem
|
||
dependency. Moves its session store off local sqlite onto a dedicated CNPG
|
||
Postgres, reachable from the Mac client through a dedicated nginx route —
|
||
durability-of-location, not a multi-host requirement.
|
||
|
||
| Task | Title | Size | Flags | Status |
|
||
|---|---|---|---|---|
|
||
| [M6.1](M6.1-agent-manager-db-manifest.md) | CNPG `agent-manager-db` manifest | M | homelab | ⬜ |
|
||
| [M6.2](M6.2-schema-port.md) | Postgres schema for agent-manager sessions | M | — | ⬜ |
|
||
| [M6.3](M6.3-store-query-port.md) | store.go query port to Postgres | L | — | ⬜ |
|
||
| [M6.4](M6.4-nginx-stream-routing.md) | nginx TCP routing to `agent-manager-db` | S | homelab | ⬜ |
|
||
| [M6.5](M6.5-credentials-secret.md) | Postgres credentials for the Mac client | S | homelab | ⬜ |
|
||
| [M6.6](M6.6-m6-gate.md) | **M6 composition gate** | M | gate | ⬜ |
|
||
|
||
## 8 — Source connectors · M7.x
|
||
|
||
Extensible multi-source ingestion. `SourceConnector` trait + YAML-driven registry.
|
||
Adding a new document source (paperless-ngx, S3, git repo) requires implementing
|
||
one trait and adding one config block — no changes to the ingest pipeline, chunking,
|
||
embedding, storage, or query layers.
|
||
|
||
**Document connectors** produce Level R content (reference material, bypasses gated
|
||
loop). **Session connectors** (pi, claude) produce evidence for L0/L1/L2. The
|
||
connector's `source_type()` declares the pipeline.
|
||
|
||
Sync framework handles change detection (sha-based skip), tombstoning, drift
|
||
reporting, and resumable sync for all connectors.
|
||
|
||
| Task | Title | Size | Flags | Status |
|
||
|---|---|---|---|---|
|
||
| [M7.1](M7.1-source-connector-trait.md) | `SourceConnector` trait + registry | M | — | ⬜ |
|
||
| [M7.2](M7.2-obsidian-connector.md) | Obsidian vault connector | M | — | ⬜ |
|
||
| [M7.3](M7.3-paperless-connector.md) | paperless-ngx connector | M | — | ⬜ |
|
||
| [M7.4](M7.4-git-repo-connector.md) | Git repo connector | M | — | ⬜ |
|
||
| [M7.5](M7.5-s3-connector.md) | S3-compatible storage connector | M | — | ⬜ |
|
||
| [M7.6](M7.6-sync-framework.md) | Sync framework | L | — | ⬜ |
|
||
| [M7.7](M7.7-source-cli.md) | `mem source` CLI | M | — | ⬜ |
|
||
| [M7.8](M7.8-source-http-endpoints.md) | Source HTTP endpoints | M | — | ⬜ |
|
||
| [M7.9](M7.9-connector-health-monitoring.md) | Connector health + observability | S | — | ⬜ |
|
||
| [M7.10](M7.10-m7-gate.md) | **M7 composition gate** | M | gate | ⬜ |
|
||
|
||
---
|
||
|
||
## 9 — Hybrid search · M8.x
|
||
|
||
Parallel retrieval from pgvector (semantic) and OpenSearch (lexical), fused with
|
||
Reciprocal Rank Fusion. Adds a `QueryOptimizer` that classifies queries and
|
||
routes to the best strategy before any database call.
|
||
|
||
**The load-bearing property is accuracy.** Hybrid must produce measurably better
|
||
NDCG@10 than either engine alone. If it doesn't, the gate fails — not because
|
||
the code is broken, but because the system isn't earning its complexity budget.
|
||
M8.9 requires benchmark numbers, not just green tests.
|
||
|
||
**Approach A: Parallel RRF.** Both engines run simultaneously via `tokio::try_join!`.
|
||
Results are merged by rank position, not score magnitude, because pgvector cosine
|
||
(`[0,1]`) and BM25 (`[0,50+]`) are incomparable distributions. RRF needs no
|
||
parameter tuning (`k=60` is the academic standard). The alternative — weighted
|
||
linear combination — requires labelled data for weight selection that we don't
|
||
have yet.
|
||
|
||
**Dual-write indexing.** Every chunk gets the same UUID in both pgvector and
|
||
OpenSearch. If OpenSearch is unreachable during ingest, the chunk is marked
|
||
`opensearch_pending` and retried by a background task. The gate (M8.9) checks
|
||
for zero orphans.
|
||
|
||
**Fallback.** If OpenSearch is down at query time, the worker degrades to
|
||
semantic-only. If the embedding model is down, it degrades to lexical-only.
|
||
The response `search_strategy` field always reports which mode was actually used.
|
||
|
||
| Task | Title | Size | Flags | Status |
|
||
|---|---|---|---|---|
|
||
| [M8.1](M8.1-opensearch-deployment.md) | OpenSearch cluster + JWT realm | M | homelab | ⬜ |
|
||
| [M8.2](M8.2-dual-write-indexer.md) | Dual-write indexing pipeline | M | — | ⬜ |
|
||
| [M8.3](M8.3-query-optimizer.md) | Query optimizer: context + routing | M | — | ⬜ |
|
||
| [M8.4](M8.4-rrf-fusion.md) | Reciprocal Rank Fusion engine | S | — | ⬜ |
|
||
| [M8.5](M8.5-hybrid-query-worker.md) | Hybrid query worker: parallel retrieval | L | — | ⬜ |
|
||
| [M8.6](M8.6-query-endpoint-upgrade.md) | Upgrade GET /query to hybrid + fallback | M | — | ⬜ |
|
||
| [M8.7](M8.7-index-optimization.md) | Index tuning: HNSW + OpenSearch analyzers | M | — | ⬜ |
|
||
| [M8.8](M8.8-accuracy-benchmarks.md) | Accuracy benchmarks: NDCG, MRR, P@K | M | — | ⬜ |
|
||
| [M8.9](M8.9-m8-gate.md) | **M8 composition gate** | M | gate | ⬜ |
|
||
|
||
---
|
||
|
||
Background: [DESIGN.md](../DESIGN.md) · GRU-Mem, arXiv 2602.10560 · `internal/store/store.go` (agent-manager, `add-headless-spawn` branch)
|