chore: Archive completed task files (M0, M1, M3, M3.5, M4.1-2, M3.6.1)

Deleted 31 completed task files:
- M0.x: 8 tasks (cargo, domain types, recordsource, tokenizer, adapters, gate)
- M1.x: 8 tasks (llm-chat, standing-query, prompt template, parser, loop, log, e2e, gate)
- M3.x: 4 tasks (l2-synthesis, rerank, mem-query, gate)
- M3.5.x: 8 tasks (http-server, ingest, query, federation, skills, projects, rate-limiting, gate)
- M3.6.1: DocCorpusSource (heading-boundary chunking)
- M4.1-2: skill-draft, derived-filter

Updated INDEX.md:
- Removed M0 & M1 phase sections (archived in git history)
- Updated progress table: 65 active tasks (42 + 2🟡 + 21)
- Updated status: M0/M1 complete, M3/M3.5 gates passing, M4.1-2 done
- Noted M3.5.10 JWT auth implementation complete (awaiting image rollout)
- Cleaned up broken links to deleted task files

Total test count: 239 passing, 2 ignored (up from 196 at M3.4)
Ready for M4.3 gate composition, M5 post-training, M7 source connectors.
This commit is contained in:
Story Crater Bot
2026-08-27 20:25:05 -07:00
parent fe4308ef1d
commit 959c596b1d
55 changed files with 6909 additions and 4098 deletions
+80
View File
@@ -0,0 +1,80 @@
# M8.3 — Query optimizer: context construction + strategy routing
| Field | Value |
|---|---|
| Phase | M8 — Hybrid Search |
| Size | M — 12 days |
| Status | ⬜ |
| Flags | — |
| Spec | inlined below |
| Blocks | M8.5 |
| Depends | — (pure logic, no infra dependency) |
## Goal
Build a `QueryOptimizer` that analyses a raw user query and produces a `QueryContext` — normalised text, extracted entities, classified question type, and a routed `SearchStrategy`. This runs **before** any database call and determines which engines to use.
## Design
**6-stage pipeline:**
1. **Normalize** — lowercase, trim, collapse whitespace.
2. **Tokenize** — split on whitespace.
3. **Extract entities** — detect years (YYYY), quoted phrases (`"exact"`), tags (`#`, `@`).
4. **Analyse characteristics** — boolean flags: `has_special_syntax`, `has_date_filters`, `has_negation`.
5. **Classify question type** — one of: `Factual`, `Procedural`, `Comparative`, `Troubleshooting`, `Navigational`, `Open`.
6. **Route** — pick `SearchStrategy` with a confidence score (0.01.0).
**Routing rules:**
- Token count < 3 → `LexicalOnly` (BM25 handles keywords better than embeddings).
- Special syntax (`#tag`, `@mention`, `"phrase"`) → `LexicalOnly` (preserve exact tokens).
- Date filters present → `LexicalFirst` (narrow by date in OpenSearch, rerank in pgvector).
- Procedural / Troubleshooting → `Hybrid` (need both exact errors + semantic understanding).
- Navigational → `LexicalFirst` (finding specific docs).
- Default → `Hybrid`.
**Output struct:**
```rust
pub struct QueryContext {
pub raw_query: String,
pub normalized_query: String,
pub tokens: Vec<String>,
pub entities: HashMap<String, String>,
pub embedding: Option<Vec<f32>>, // filled later by worker
pub token_count: usize,
pub has_special_syntax: bool,
pub has_date_filters: bool,
pub has_negation: bool,
pub question_type: QuestionType,
pub search_strategy: SearchStrategy,
pub confidence: f32,
}
```
## Steps
1. Implement `QueryOptimizer` with `optimize_query(&str) -> Result<QueryContext>`.
2. Implement each stage as a private method.
3. Write unit tests for every routing rule (≥15 tests).
4. No async, no IO, no dependencies beyond std. Pure logic.
## Acceptance
1. `optimize_query("fix port")``LexicalOnly`, confidence ≥ 0.7.
2. `optimize_query("How do I fix kubernetes port 8080?")``Hybrid`, `Procedural`, confidence ≥ 0.9.
3. `optimize_query("#networking @devops policy")``LexicalOnly`, `has_special_syntax=true`.
4. `optimize_query("deployment failures in 2024")``LexicalFirst`, `has_date_filters=true`, entity `year=2024`.
5. `optimize_query("Compare Docker and Kubernetes")``Hybrid`, `Comparative`.
6. All 15+ tests pass.
## Verify
```bash
cargo test -p mem-cli query_optimizer:: -- --nocapture
```
**False pass:** Routing always returns `Hybrid` regardless of input. Check the short-query and special-syntax tests specifically — they must return non-Hybrid strategies.
## Artifacts
- `crates/mem-cli/src/query_optimizer.rs` (exists, needs cleanup + test fixes)