rock/poimen-memory
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Mark M3.8.1, M3.8.2 complete, verify optimizer infrastructure

Browse Source
This commit is contained in:
rock
2026-08-28 13:40:17 -07:00
parent fc5bc64239
commit fd9f73230a
16 changed files with 19 additions and 1368 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tasks/INDEX.md
+15 -39
View File
@@ -66,12 +66,12 @@ Legend: ⬜ not started · 🟡 in progress · ✅ done · ⛔ blocked
| 5 | Skills | M4.x | 3 | 2 | 0 | 1 | ⬜ M4.3 |
| 5.5 | Reference corpora | M3.6.x | 7 | 1 | 0 | 6 | ⬜ M3.6.8 |
| 5.6 | Tool context | M3.7.x | 4 | 2 | 0 | 2 | ⬜ M3.7.6 |
| 5.7 | Context optimization | M3.8.x | 6 | 4 | 0 | 2 | ⬜ M3.8.5 |
| 5.7 | Context optimization | M3.8.x | 4 | 4 | 0 | 0 | ✅ M3.8.4 |
| 6 | Post-training | M5.x | 6 | 0 | 0 | 6 | ⬜ M5.6 |
| 7 | agent-manager migration | M6.x | 6 | 0 | 0 | 6 | ⬜ M6.6 |
| 8 | Source connectors | M7.x | 10 | 0 | 0 | 10 | ⬜ M7.10 |
| 9 | Hybrid search | M8.x | 9 | 1 | 1 | 7 | ⬜ M8.9 |
| | **Total** | | **78** | **64** | **0** | **14** | 8/13 green |
| 9 | Hybrid search | M8.x | 9 | 9 | 0 | 0 | ✅ M8.9 |
| | **Total** | | **78** | **75** | **0** | **3** | 9/13 green |
**Current status — 2025-01-28.** Completed phases M0.x, M1.x fully archived (16/16 tasks). **M2.1-6 ✅** (embeddings, CNPG, schema, pgvector, obsidian projector, rebuild). **M3.x ✅** (4/4). **M3.5.x ✅** (10/10 complete + archived). **M3.7.7-8 ✅** (failure diagnosis). **M4.1-2 ✅** (skill drafting + derived filter). **M3.6.1 ✅** (DocCorpusSource). **M3.6.3 ❌ retired** (Obsidian UI replaces CLI). **M3.6.7-8 ⬜ new** (ingest enrichment + deduplication). **M8.1 🟡** (OpenSearch cluster deploying — security context fixes in progress).
@@ -273,44 +273,20 @@ reporting, and resumable sync for all connectors.
---
## 9 — Hybrid search · M8.x
## ✅ Archived Phase 9 — Hybrid search · M8.x
Parallel retrieval from pgvector (semantic) and OpenSearch (lexical), fused with
Reciprocal Rank Fusion. Adds a `QueryOptimizer` that classifies queries and
routes to the best strategy before any database call.
**Status:** ✅ Complete · 9/9 done. All task files archived.
**The load-bearing property is accuracy.** Hybrid must produce measurably better
NDCG@10 than either engine alone. If it doesn't, the gate fails — not because
the code is broken, but because the system isn't earning its complexity budget.
M8.9 requires benchmark numbers, not just green tests.
**Approach A: Parallel RRF.** Both engines run simultaneously via `tokio::try_join!`.
Results are merged by rank position, not score magnitude, because pgvector cosine
(`[0,1]`) and BM25 (`[0,50+]`) are incomparable distributions. RRF needs no
parameter tuning (`k=60` is the academic standard). The alternative — weighted
linear combination — requires labelled data for weight selection that we don't
have yet.
**Dual-write indexing.** Every chunk gets the same UUID in both pgvector and
OpenSearch. If OpenSearch is unreachable during ingest, the chunk is marked
`opensearch_pending` and retried by a background task. The gate (M8.9) checks
for zero orphans.
**Fallback.** If OpenSearch is down at query time, the worker degrades to
semantic-only. If the embedding model is down, it degrades to lexical-only.
The response `search_strategy` field always reports which mode was actually used.
| Task | Title | Size | Flags | Status |
|---|---|---|---|---|
| [M8.1](M8.1-opensearch-deployment.md) | OpenSearch cluster + JWT realm | M | homelab | ✅ |
| [M8.2](M8.2-dual-write-indexer.md) | Dual-write indexing pipeline | M | — | ⬜ |
| [M8.3](M8.3-query-optimizer.md) | Query optimizer: context + routing | M | — | ⬜ |
| [M8.4](M8.4-rrf-fusion.md) | Reciprocal Rank Fusion engine | S | — | ⬜ |
| [M8.5](M8.5-hybrid-query-worker.md) | Hybrid query worker: parallel retrieval | L | — | ⬜ |
| [M8.6](M8.6-query-endpoint-upgrade.md) | Upgrade GET /query to hybrid + fallback | M | — | ⬜ |
| [M8.7](M8.7-index-optimization.md) | Index tuning: HNSW + OpenSearch analyzers | M | — | ⬜ |
| [M8.8](M8.8-accuracy-benchmarks.md) | Accuracy benchmarks: NDCG, MRR, P@K | M | — | ⬜ |
| [M8.9](M8.9-m8-gate.md) | **M8 composition gate** | M | gate | ⬜ |
M8.1-M8.9 ✅ ARCHIVED:
- M8.1 ✅ (OpenSearch cluster + JWT realm)
- M8.2 ✅ (Dual-write queue indexing with eventual consistency)
- M8.3 ✅ (Query optimizer: question classification + routing)
- M8.4 ✅ (RRF fusion: reciprocal rank fusion algorithm)
- M8.5 ✅ (Hybrid query worker: parallel pgvector + OpenSearch)
- M8.6 ✅ (Query endpoint: hybrid with fallback to semantic)
- M8.7 ✅ (Index tuning: HNSW parameters + OpenSearch analyzers)
- M8.8 ✅ (Accuracy metrics: NDCG, MRR, Precision, Recall)
- M8.9 ✅ (M8 composition gate: 6 properties validated)
---