104 lines
3.7 KiB
Markdown
104 lines
3.7 KiB
Markdown
|
# M8.7 — Index tuning: HNSW parameters + OpenSearch analyzers
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | M8 — Hybrid Search |
|
|||
|
|
| Size | M — 1–2 days |
|
|||
|
|
| Status | ⬜ |
|
|||
|
|
| Flags | — |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | M8.9 |
|
|||
|
|
| Depends | M8.2 (data in both stores), M8.5 (can query both stores) |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Tune pgvector index parameters and OpenSearch analyzers for retrieval accuracy. Measure baseline NDCG before and after tuning. This is engineering, not research — change one parameter, measure, keep or revert.
|
|||
|
|
|
|||
|
|
## Design
|
|||
|
|
|
|||
|
|
### pgvector tuning
|
|||
|
|
|
|||
|
|
**Current:** `ivfflat` index with default `lists`.
|
|||
|
|
|
|||
|
|
**Target:** Switch to `hnsw` index (pgvector 0.5.0+). HNSW provides better recall than IVFFlat at the cost of slower index builds and more memory.
|
|||
|
|
|
|||
|
|
```sql
|
|||
|
|
-- Drop old index
|
|||
|
|
DROP INDEX IF EXISTS chunks_embedding_idx;
|
|||
|
|
|
|||
|
|
-- Create HNSW index
|
|||
|
|
CREATE INDEX chunks_embedding_hnsw_idx
|
|||
|
|
ON chunks USING hnsw (embedding vector_cosine_ops)
|
|||
|
|
WITH (m = 16, ef_construction = 64);
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Parameters:
|
|||
|
|
- `m = 16` — max connections per node (default 16, higher = better recall, more memory).
|
|||
|
|
- `ef_construction = 64` — build-time search width (default 64, higher = better recall, slower build).
|
|||
|
|
- `ef_search = 40` — query-time search width (set via `SET hnsw.ef_search = 40`).
|
|||
|
|
|
|||
|
|
**Tuning approach:**
|
|||
|
|
1. Baseline: measure recall@50 with IVFFlat.
|
|||
|
|
2. Switch to HNSW with defaults.
|
|||
|
|
3. Measure recall@50 again.
|
|||
|
|
4. If recall@50 ≥ 0.95, keep defaults. Otherwise increase `ef_construction` to 128.
|
|||
|
|
|
|||
|
|
### OpenSearch tuning
|
|||
|
|
|
|||
|
|
**Analyzer changes:**
|
|||
|
|
- Add `edge_ngram` tokenizer for typo tolerance on `content` field.
|
|||
|
|
- Add `synonym` filter for common abbreviations: `k8s → kubernetes`, `db → database`, `cfg → config`.
|
|||
|
|
- Keep `standard` analyzer as primary, add `search_analyzer` for queries.
|
|||
|
|
|
|||
|
|
**Field boost tuning:**
|
|||
|
|
- `content^2.0` (default — most important).
|
|||
|
|
- `section_title^1.8` (headings are very relevant).
|
|||
|
|
- `source^1.0` (file paths are useful but shouldn't dominate).
|
|||
|
|
- `breadcrumb^0.8` (context, not content).
|
|||
|
|
|
|||
|
|
**BM25 parameters:**
|
|||
|
|
- `k1 = 1.2` (term frequency saturation — default is fine).
|
|||
|
|
- `b = 0.75` (length normalization — default is fine).
|
|||
|
|
- Don't tune these unless baseline NDCG < 0.7.
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. Create test query set: 20 queries with known-relevant documents.
|
|||
|
|
2. Measure baseline NDCG@10 for pgvector (semantic-only) and OpenSearch (lexical-only).
|
|||
|
|
3. Switch pgvector from IVFFlat to HNSW. Measure NDCG@10 again.
|
|||
|
|
4. Update OpenSearch index template with synonym filter + edge_ngram. Reindex. Measure.
|
|||
|
|
5. Record all measurements in `docs/INDEX_TUNING_RESULTS.md`.
|
|||
|
|
6. Keep changes that improve NDCG. Revert changes that don't.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
1. pgvector uses HNSW index (verify with `\d+ chunks` in psql).
|
|||
|
|
2. OpenSearch index template includes synonym filter.
|
|||
|
|
3. NDCG@10 measurements recorded for before/after each change.
|
|||
|
|
4. No regression: post-tuning NDCG ≥ pre-tuning NDCG for both engines.
|
|||
|
|
5. pgvector query latency < 150ms (p95) after HNSW switch.
|
|||
|
|
6. OpenSearch query latency < 100ms (p95) after analyzer changes.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
# Check pgvector index type
|
|||
|
|
psql -c "\d+ chunks" | grep hnsw
|
|||
|
|
|
|||
|
|
# Check OpenSearch analyzer
|
|||
|
|
curl -k -H "Authorization: Bearer $TOKEN" \
|
|||
|
|
https://opensearch-internal:9200/vault-test/_settings | jq '.*.settings.index.analysis'
|
|||
|
|
|
|||
|
|
# Run NDCG measurement
|
|||
|
|
cargo run -- bench-search --queries fixtures/search_queries.yaml --output docs/INDEX_TUNING_RESULTS.md
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**False pass:** HNSW index created but `ef_search` set to 1, making recall worse than IVFFlat. Check recall@50 explicitly — it should be ≥ 0.95.
|
|||
|
|
|
|||
|
|
## Artifacts
|
|||
|
|
|
|||
|
|
- SQL migration (drop IVFFlat, create HNSW)
|
|||
|
|
- Updated OpenSearch index template
|
|||
|
|
- `docs/INDEX_TUNING_RESULTS.md` (measurements)
|
|||
|
|
- `fixtures/search_queries.yaml` (test query set)
|