Files

T

Story Crater Bot 959c596b1d chore: Archive completed task files (M0, M1, M3, M3.5, M4.1-2, M3.6.1)

Deleted 31 completed task files:
- M0.x: 8 tasks (cargo, domain types, recordsource, tokenizer, adapters, gate)
- M1.x: 8 tasks (llm-chat, standing-query, prompt template, parser, loop, log, e2e, gate)
- M3.x: 4 tasks (l2-synthesis, rerank, mem-query, gate)
- M3.5.x: 8 tasks (http-server, ingest, query, federation, skills, projects, rate-limiting, gate)
- M3.6.1: DocCorpusSource (heading-boundary chunking)
- M4.1-2: skill-draft, derived-filter

Updated INDEX.md:
- Removed M0 & M1 phase sections (archived in git history)
- Updated progress table: 65 active tasks (42✅ + 2🟡 + 21⬜)
- Updated status: M0/M1 complete, M3/M3.5 gates passing, M4.1-2 done
- Noted M3.5.10 JWT auth implementation complete (awaiting image rollout)
- Cleaned up broken links to deleted task files

Total test count: 239 passing, 2 ignored (up from 196 at M3.4)
Ready for M4.3 gate composition, M5 post-training, M7 source connectors.

2026-08-27 20:25:05 -07:00

3.1 KiB

Raw Blame History

M8.3 — Query optimizer: context construction + strategy routing

Field	Value
Phase	M8 — Hybrid Search
Size	M — 1–2 days
Status	⬜
Flags	—
Spec	inlined below
Blocks	M8.5
Depends	— (pure logic, no infra dependency)

Goal

Build a QueryOptimizer that analyses a raw user query and produces a QueryContext — normalised text, extracted entities, classified question type, and a routed SearchStrategy. This runs before any database call and determines which engines to use.

Design

6-stage pipeline:

Normalize — lowercase, trim, collapse whitespace.
Tokenize — split on whitespace.
Extract entities — detect years (YYYY), quoted phrases ("exact"), tags (#, @).
Analyse characteristics — boolean flags: has_special_syntax, has_date_filters, has_negation.
Classify question type — one of: Factual, Procedural, Comparative, Troubleshooting, Navigational, Open.
Route — pick SearchStrategy with a confidence score (0.0–1.0).

Routing rules:

Token count < 3 → LexicalOnly (BM25 handles keywords better than embeddings).
Special syntax (#tag, @mention, "phrase") → LexicalOnly (preserve exact tokens).
Date filters present → LexicalFirst (narrow by date in OpenSearch, rerank in pgvector).
Procedural / Troubleshooting → Hybrid (need both exact errors + semantic understanding).
Navigational → LexicalFirst (finding specific docs).
Default → Hybrid.

Output struct:

pub struct QueryContext {
    pub raw_query: String,
    pub normalized_query: String,
    pub tokens: Vec<String>,
    pub entities: HashMap<String, String>,
    pub embedding: Option<Vec<f32>>,  // filled later by worker
    pub token_count: usize,
    pub has_special_syntax: bool,
    pub has_date_filters: bool,
    pub has_negation: bool,
    pub question_type: QuestionType,
    pub search_strategy: SearchStrategy,
    pub confidence: f32,
}

Steps

Implement QueryOptimizer with optimize_query(&str) -> Result<QueryContext>.
Implement each stage as a private method.
Write unit tests for every routing rule (≥15 tests).
No async, no IO, no dependencies beyond std. Pure logic.

Acceptance

optimize_query("fix port") → LexicalOnly, confidence ≥ 0.7.
optimize_query("How do I fix kubernetes port 8080?") → Hybrid, Procedural, confidence ≥ 0.9.
optimize_query("#networking @devops policy") → LexicalOnly, has_special_syntax=true.
optimize_query("deployment failures in 2024") → LexicalFirst, has_date_filters=true, entity year=2024.
optimize_query("Compare Docker and Kubernetes") → Hybrid, Comparative.
All 15+ tests pass.

Verify

cargo test -p mem-cli query_optimizer:: -- --nocapture

False pass: Routing always returns Hybrid regardless of input. Check the short-query and special-syntax tests specifically — they must return non-Hybrid strategies.

Artifacts

crates/mem-cli/src/query_optimizer.rs (exists, needs cleanup + test fixes)

3.1 KiB Raw Blame History Unescape Escape