rock/poimen-memory
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: Archive completed task files (M0, M1, M3, M3.5, M4.1-2, M3.6.1)

Browse Source 
Deleted 31 completed task files:
- M0.x: 8 tasks (cargo, domain types, recordsource, tokenizer, adapters, gate)
- M1.x: 8 tasks (llm-chat, standing-query, prompt template, parser, loop, log, e2e, gate)
- M3.x: 4 tasks (l2-synthesis, rerank, mem-query, gate)
- M3.5.x: 8 tasks (http-server, ingest, query, federation, skills, projects, rate-limiting, gate)
- M3.6.1: DocCorpusSource (heading-boundary chunking)
- M4.1-2: skill-draft, derived-filter

Updated INDEX.md:
- Removed M0 & M1 phase sections (archived in git history)
- Updated progress table: 65 active tasks (42✅ + 2🟡 + 21⬜)
- Updated status: M0/M1 complete, M3/M3.5 gates passing, M4.1-2 done
- Noted M3.5.10 JWT auth implementation complete (awaiting image rollout)
- Cleaned up broken links to deleted task files

Total test count: 239 passing, 2 ignored (up from 196 at M3.4)
Ready for M4.3 gate composition, M5 post-training, M7 source connectors.
This commit is contained in:
Story Crater Bot
2026-08-27 20:25:05 -07:00
parent fe4308ef1d
commit 959c596b1d
55 changed files with 6909 additions and 4098 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tasks/INDEX.md
+64 -73
View File
@@ -60,23 +60,23 @@ Legend: ⬜ not started · 🟡 in progress · ✅ done · ⛔ blocked
|---|---|---|---|---|---|---|---|
| 1 | Read-only spine | M0.x | 8 | 8 | 0 | 0 | ✅ M0.8 |
| 2 | Gated loop at L1 | M1.x | 8 | 8 | 0 | 0 | ✅ M1.8 |
| 3 | Projections | M2.x | 8 | 5 | 0 | 3 | ✅ M2.8 (M2.1, M2.3, M2.4, M2.5 ✅) |
| 4 | L2 synthesis + retrieval | M3.x | 4 | 4 | 0 | 0 | ✅ M3.4 |
| 4.5 | Distributed API Layer | M3.5.x | 10 | 8 | 0 | 2 | ✅ M3.5.8 |
| 4.5 | Distributed API Layer | M3.5.x | 10 | 9 | 0 | 1 | ✅ M3.5.8 |
| 5 | Skills | M4.x | 3 | 2 | 0 | 1 | ⬜ M4.3 |
| 5.5 | Reference corpora | M3.6.x | 6 | 1 | 0 | 5 | ⬜ M3.6.6 |
| 5.6 | Tool context | M3.7.x | 6 | 0 | 2 | 4 | ⬜ M3.7.6 |
| 6 | Post-training | M5.x | 6 | 0 | 0 | 6 | ⬜ M5.6 |
| 7 | agent-manager migration | M6.x | 6 | 0 | 0 | 6 | ⬜ M6.6 |
| | **Total** | | **64** | **42** | **3** | **19** | 5/10 green |
| 8 | Source connectors | M7.x | 10 | 0 | 0 | 10 | ⬜ M7.10 |
| 9 | Hybrid search | M8.x | 9 | 0 | 0 | 9 | ⬜ M8.9 |
| | **Total** | | **65** | **42** | **2** | **21** | 5/10 green |
**Where the line is — 2025-01-26.** M0, M1, M3, M3.5 gates complete (21/21 tasks, gates green).
M3.5 API layer 8/9 done (M3.5.1–8 ✅, M3.5.9 git-context pending). M3.6.1 DocCorpusSource ✅ (14 tests).
Starting M4 (Skills) and M5 (Post-training) tracks. E2E/API testing deferred until after M4-M5 work.
Significant early work for M3.7 and M4: `mem-core/src/lesson.rs` (871 lines, 17 unit tests)
implements signature extraction, normalisation, tier-based lookup, lesson derivation, and SKILL.md
rendering — M3.7.7, M3.7.5 are 🟡. M4.1-2 ✅ done. `mem-cli/src/lessons_cmd.rs` (311 lines), `mem-ingest/src/derived_filter.rs` (220 lines) provides working
`mem capture|resolve|lookup|materialize`. **Tests: 219 passing, 2 ignored.**
**Current status — 2025-01-27.** Completed phases M0.x, M1.x fully archived (16/16 tasks). M3.x (4/4 ✅), M3.5.x (9/10 ✅ + 1 in-progress M3.5.9).
M3.5.10 JWT auth integration ✅ complete with Authentik OIDC validation.
M4.1-2 Skills ✅ done (skill drafting + derived filter). M3.6.1 DocCorpusSource ✅.
All completed task files archived from `/tasks/` folder. INDEX.md cleaned to reflect active work only.
Significant early work for M3.7: `mem-core/src/lesson.rs` (871 lines, 17 unit tests) implements signature extraction, normalisation, tier-based lookup, lesson derivation — M3.7.7, M3.7.5 are 🟡. `mem-cli/src/lessons_cmd.rs` (311 lines), `mem-ingest/src/derived_filter.rs` (220 lines) provides working `mem capture|resolve|lookup|materialize`.
**Tests: 239 passing, 2 ignored.** Ready to tackle M4.3 gate (skills composition), M5 (post-training), M7 (source connectors).
`M2.2` (CNPG manifest), `M5.4` (vLLM+LoRA), `M3.5.9` (git refs), and
`M6.x` (agent-manager migration) are homelab/infra work independent of prior phases, can start parallel.
@@ -89,81 +89,31 @@ happening alongside M2.2/M5.4, and because the two projects' Postgres schemas
landing in the same cluster around the same time need to look like siblings,
not strangers — see M6.6's convention-consistency check.
## 1 — Read-only spine · M0.x
## ✅ Archived Phases
No model calls anywhere in this phase. The point is to prove the corpus parses
and chunks sanely before spending inference on it.
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M0.1](M0.1-cargo-workspace.md) | Cargo workspace + crate skeletons | S | — | ✅ |
| [M0.2](M0.2-domain-types.md) | Domain types and sha256 identity | S | — | ✅ |
| [M0.3](M0.3-recordsource-and-chunkpolicy.md) | `RecordSource` trait + `ChunkPolicy` | M | — | ✅ |
| [M0.4](M0.4-tokenizer-sizing.md) | Tokenizer-backed chunk sizing | M | — | ✅ |
| [M0.5](M0.5-pi-session-adapter.md) | pi session adapter | M | — | ✅ |
| [M0.6](M0.6-claude-transcript-adapter.md) | Claude transcript adapter | S | — | ✅ |
| [M0.7](M0.7-ingest-dry-run.md) | `mem ingest --dry-run` | S | — | ✅ |
| [M0.8](M0.8-m0-gate.md) | **M0 composition gate** | M | gate | ✅ |
## 2 — Gated loop at L1 · M1.x
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M1.1](M1.1-llm-chat-client.md) | `mem-llm` chat client | M | — | ✅ |
| [M1.2](M1.2-standing-query-loader.md) | Standing-query YAML loader | M | — | ✅ |
| [M1.3](M1.3-prompt-template.md) | GRU-Mem prompt template | M | — | ✅ |
| [M1.4](M1.4-gate-response-parser.md) | Gate-response parser | M | — | ✅ |
| [M1.5](M1.5-gated-loop.md) | The gated loop | L | — | ✅ |
| [M1.6](M1.6-jsonl-event-log.md) | JSONL event log writer | M | — | ✅ |
| [M1.7](M1.7-ingest-end-to-end.md) | `mem ingest` end to end | M | — | ✅ |
| [M1.8](M1.8-m1-gate.md) | **M1 composition gate** | M | gate | ✅ |
Completed and archived: **M0.x (8/8)**, **M1.x (8/8)** — all task files deleted from `/tasks/` after verification. See git log for historical record and `CLAUDE.md` for session context.
## 3 — Projections · M2.x
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M2.1](M2.1-embeddings-client.md) | Embeddings client | S | — | ⬜ |
| [M2.2](M2.2-memory-db-manifest.md) | CNPG `memory-db` + pgvector | M | homelab | ⬜ |
| [M2.3](M2.3-schema-and-migrations.md) | Schema + sqlx migrations | M | — | ⬜ |
| [M2.4](M2.4-pgvector-repo.md) | pgvector repository | M | — | ⬜ |
| [M2.5](M2.5-obsidian-projector.md) | Obsidian projector | M | — | ⬜ |
| [M2.6](M2.6-rebuild-from-log.md) | `mem rebuild --from-log` | M | — | ⬜ |
| [M2.7](M2.7-verify-edges.md) | `mem verify` — edge closure | S | — | ⬜ |
| [M2.8](M2.8-m2-gate.md) | **M2 composition gate** | M | gate | ⬜ |
**Status:** In progress · 5/8 done, 3 pending.
M2.1, M2.3, M2.4, M2.5 ✅ complete. M2.2 (CNPG manifest), M2.6, M2.7 remain. M2.8 gate awaits dependency clearance.
## 4 — L2 synthesis and retrieval · M3.x
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M3.1](M3.1-l2-synthesis.md) | L2 synthesis pass | M | — | ✅ |
| [M3.2](M3.2-rerank-client.md) | Rerank client | S | — | ✅ |
| [M3.3](M3.3-mem-query.md) | `mem query` with provenance | M | — | ✅ |
| [M3.4](M3.4-m3-gate.md) | **M3 composition gate** | M | gate | ✅ |
**Status:** ✅ Complete · 4/4 tasks done. Gate M3.4 passing. Task files archived.
## 4.5 — Distributed API Layer · M3.5.x
Homelab frontend integration: HTTP facade via `api.riotpiao.com`. Runs in parallel with M4 and M5 after M3.4 green.
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M3.5.1](M3.5.1-http-server.md) | HTTP server + router, auth hook, metrics | M | — | ✅ |
| [M3.5.2](M3.5.2-ingest-endpoint.md) | POST /ingest async queue | M | — | ✅ |
| [M3.5.3](M3.5.3-query-endpoint.md) | GET /query HNSW+rerank | M | — | ✅ |
| [M3.5.4](M3.5.4-query-federation.md) | Query federation | M | — | ✅ |
| [M3.5.5](M3.5.5-skills-endpoint.md) | GET /skills endpoint | M | — | ✅ |
| [M3.5.6](M3.5.6-projects-endpoint.md) | GET /projects endpoint | S | — | ✅ |
| [M3.5.7](M3.5.7-rate-limiting.md) | Rate limiting | M | — | ✅ |
| [M3.5.8](M3.5.8-m3.5-gate.md) | **M3.5 composition gate** | M | gate | ✅ |
| [M3.5.9](M3.5.9-git-aware-references.md) | Git-aware references: lookup by code location | M | — | ⬜ |
| [M3.5.10](M3.5.10-auth-integration.md) | Auth: Authentik/Vault OIDC token validation | M | — | ⬜ |
**Status:** 9/10 done · M3.5.8 gate ✅ passing. M3.5.1–8 archived (task files deleted). M3.5.9 (git-aware refs) and M3.5.10 (JWT/OIDC auth) remain. M3.5.10 implementation ✅ complete: Authentik OIDC provider, RS256 validation, capability-based access control. Awaiting new Docker image rollout to pods.
## 5 — Skills · M4.x
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M4.1](M4.1-skill-draft.md) | `mem skill draft` | M | — | ✅ |
| [M4.2](M4.2-derived-filter.md) | `derived: true` ingest filter | M | — | ✅ |
| [M4.3](M4.3-m4-gate.md) | **M4 composition gate** | M | gate | ⬜ |
**Status:** 2/3 done · M4.1 (skill draft) ✅ and M4.2 (derived filter) ✅ archived. Pending M4.3 gate composition.
`mem skill draft --project <proj> --from <query-id>` writes to `vault/skills/_drafts/`. Dry-run mode supported. Shingle matcher (M4.2) provides derived LessonSource filtering. M4.1-2 task files deleted after archival.
## 5.5 — Reference corpora · M3.6.x
@@ -183,7 +133,7 @@ clear while the memory got worse.
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M3.6.1](M3.6.1-doc-corpus-source.md) | `DocCorpusSource` + heading chunking | M | — | ✅ |
| M3.6.1 | `DocCorpusSource` + heading chunking | M | — | ✅ |
| [M3.6.2](M3.6.2-level-r-storage.md) | Level R: log, index, vault, rebuild parity | M | — | ⬜ |
| [M3.6.3](M3.6.3-mem-ref-cli.md) | `mem ref` — replace-on-change corpus management | M | — | ⬜ |
| [M3.6.4](M3.6.4-reference-cycle-guard.md) | Reference text cannot re-enter as evidence | M | — | ⬜ |
@@ -220,9 +170,9 @@ rather than reused.
|---|---|---|---|---|
| [M3.7.3](M3.7.3-skill-matching.md) | `GET /memory/skills?task=` — match a subset | M | — | ⬜ |
| [M3.7.4](M3.7.4-context-endpoint.md) | `/memory/context` — three-tier lookup | M | — | ⬜ |
| [M3.7.5](M3.7.5-tool-failure-learning.md) | `tool-failures` standing query | M | — | 🟡 `derive_lessons()` + `tool_of_cmd()` in `lesson.rs`, `mem resolve` in CLI |
| [M3.7.5](M3.7.5-tool-failure-learning.md) | `tool-failures` standing query | M | — | 🟡 |
| [M3.7.6](M3.7.6-m3.7-gate.md) | **M3.7 composition gate** | M | gate | ⬜ |
| [M3.7.7](M3.7.7-signature-extraction.md) | Failure signature extraction + normalisation | M | — | 🟡 `extract()` + `normalise()` in `lesson.rs` (10 unit tests passing) |
| [M3.7.7](M3.7.7-signature-extraction.md) | Failure signature extraction + normalisation | M | — | 🟡 |
| [M3.7.8](M3.7.8-symptom-projection.md) | Symptom projection at ingest | M | — | ⬜ |
## 6 — Post-training · M5.x
@@ -284,4 +234,45 @@ reporting, and resumable sync for all connectors.
---
## 9 — Hybrid search · M8.x
Parallel retrieval from pgvector (semantic) and OpenSearch (lexical), fused with
Reciprocal Rank Fusion. Adds a `QueryOptimizer` that classifies queries and
routes to the best strategy before any database call.
**The load-bearing property is accuracy.** Hybrid must produce measurably better
NDCG@10 than either engine alone. If it doesn't, the gate fails — not because
the code is broken, but because the system isn't earning its complexity budget.
M8.9 requires benchmark numbers, not just green tests.
**Approach A: Parallel RRF.** Both engines run simultaneously via `tokio::try_join!`.
Results are merged by rank position, not score magnitude, because pgvector cosine
(`[0,1]`) and BM25 (`[0,50+]`) are incomparable distributions. RRF needs no
parameter tuning (`k=60` is the academic standard). The alternative — weighted
linear combination — requires labelled data for weight selection that we don't
have yet.
**Dual-write indexing.** Every chunk gets the same UUID in both pgvector and
OpenSearch. If OpenSearch is unreachable during ingest, the chunk is marked
`opensearch_pending` and retried by a background task. The gate (M8.9) checks
for zero orphans.
**Fallback.** If OpenSearch is down at query time, the worker degrades to
semantic-only. If the embedding model is down, it degrades to lexical-only.
The response `search_strategy` field always reports which mode was actually used.
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M8.1](M8.1-opensearch-deployment.md) | OpenSearch cluster + JWT realm | M | homelab | ⬜ |
| [M8.2](M8.2-dual-write-indexer.md) | Dual-write indexing pipeline | M | — | ⬜ |
| [M8.3](M8.3-query-optimizer.md) | Query optimizer: context + routing | M | — | ⬜ |
| [M8.4](M8.4-rrf-fusion.md) | Reciprocal Rank Fusion engine | S | — | ⬜ |
| [M8.5](M8.5-hybrid-query-worker.md) | Hybrid query worker: parallel retrieval | L | — | ⬜ |
| [M8.6](M8.6-query-endpoint-upgrade.md) | Upgrade GET /query to hybrid + fallback | M | — | ⬜ |
| [M8.7](M8.7-index-optimization.md) | Index tuning: HNSW + OpenSearch analyzers | M | — | ⬜ |
| [M8.8](M8.8-accuracy-benchmarks.md) | Accuracy benchmarks: NDCG, MRR, P@K | M | — | ⬜ |
| [M8.9](M8.9-m8-gate.md) | **M8 composition gate** | M | gate | ⬜ |
---
Background: [DESIGN.md](../DESIGN.md) · GRU-Mem, arXiv 2602.10560 · `internal/store/store.go` (agent-manager, `add-headless-spawn` branch)