81 lines
3.1 KiB
Markdown
81 lines
3.1 KiB
Markdown
|
# M8.3 — Query optimizer: context construction + strategy routing
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | M8 — Hybrid Search |
|
|||
|
|
| Size | M — 1–2 days |
|
|||
|
|
| Status | ⬜ |
|
|||
|
|
| Flags | — |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | M8.5 |
|
|||
|
|
| Depends | — (pure logic, no infra dependency) |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Build a `QueryOptimizer` that analyses a raw user query and produces a `QueryContext` — normalised text, extracted entities, classified question type, and a routed `SearchStrategy`. This runs **before** any database call and determines which engines to use.
|
|||
|
|
|
|||
|
|
## Design
|
|||
|
|
|
|||
|
|
**6-stage pipeline:**
|
|||
|
|
|
|||
|
|
1. **Normalize** — lowercase, trim, collapse whitespace.
|
|||
|
|
2. **Tokenize** — split on whitespace.
|
|||
|
|
3. **Extract entities** — detect years (YYYY), quoted phrases (`"exact"`), tags (`#`, `@`).
|
|||
|
|
4. **Analyse characteristics** — boolean flags: `has_special_syntax`, `has_date_filters`, `has_negation`.
|
|||
|
|
5. **Classify question type** — one of: `Factual`, `Procedural`, `Comparative`, `Troubleshooting`, `Navigational`, `Open`.
|
|||
|
|
6. **Route** — pick `SearchStrategy` with a confidence score (0.0–1.0).
|
|||
|
|
|
|||
|
|
**Routing rules:**
|
|||
|
|
- Token count < 3 → `LexicalOnly` (BM25 handles keywords better than embeddings).
|
|||
|
|
- Special syntax (`#tag`, `@mention`, `"phrase"`) → `LexicalOnly` (preserve exact tokens).
|
|||
|
|
- Date filters present → `LexicalFirst` (narrow by date in OpenSearch, rerank in pgvector).
|
|||
|
|
- Procedural / Troubleshooting → `Hybrid` (need both exact errors + semantic understanding).
|
|||
|
|
- Navigational → `LexicalFirst` (finding specific docs).
|
|||
|
|
- Default → `Hybrid`.
|
|||
|
|
|
|||
|
|
**Output struct:**
|
|||
|
|
```rust
|
|||
|
|
pub struct QueryContext {
|
|||
|
|
pub raw_query: String,
|
|||
|
|
pub normalized_query: String,
|
|||
|
|
pub tokens: Vec<String>,
|
|||
|
|
pub entities: HashMap<String, String>,
|
|||
|
|
pub embedding: Option<Vec<f32>>, // filled later by worker
|
|||
|
|
pub token_count: usize,
|
|||
|
|
pub has_special_syntax: bool,
|
|||
|
|
pub has_date_filters: bool,
|
|||
|
|
pub has_negation: bool,
|
|||
|
|
pub question_type: QuestionType,
|
|||
|
|
pub search_strategy: SearchStrategy,
|
|||
|
|
pub confidence: f32,
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. Implement `QueryOptimizer` with `optimize_query(&str) -> Result<QueryContext>`.
|
|||
|
|
2. Implement each stage as a private method.
|
|||
|
|
3. Write unit tests for every routing rule (≥15 tests).
|
|||
|
|
4. No async, no IO, no dependencies beyond std. Pure logic.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
1. `optimize_query("fix port")` → `LexicalOnly`, confidence ≥ 0.7.
|
|||
|
|
2. `optimize_query("How do I fix kubernetes port 8080?")` → `Hybrid`, `Procedural`, confidence ≥ 0.9.
|
|||
|
|
3. `optimize_query("#networking @devops policy")` → `LexicalOnly`, `has_special_syntax=true`.
|
|||
|
|
4. `optimize_query("deployment failures in 2024")` → `LexicalFirst`, `has_date_filters=true`, entity `year=2024`.
|
|||
|
|
5. `optimize_query("Compare Docker and Kubernetes")` → `Hybrid`, `Comparative`.
|
|||
|
|
6. All 15+ tests pass.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
cargo test -p mem-cli query_optimizer:: -- --nocapture
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**False pass:** Routing always returns `Hybrid` regardless of input. Check the short-query and special-syntax tests specifically — they must return non-Hybrid strategies.
|
|||
|
|
|
|||
|
|
## Artifacts
|
|||
|
|
|
|||
|
|
- `crates/mem-cli/src/query_optimizer.rs` (exists, needs cleanup + test fixes)
|