poimen-memory/tasks/M8.7-index-optimization.md

M8.7 — Index tuning: HNSW parameters + OpenSearch analyzers

Field	Value
Phase	M8 — Hybrid Search
Size	M — 1–2 days
Status	✅ COMPLETE
Flags	—
Spec	inlined below
Blocks	M8.9
Depends	M8.2 (data in both stores), M8.5 (can query both stores)

Goal

Tune pgvector index parameters and OpenSearch analyzers for retrieval accuracy. Measure baseline NDCG before and after tuning. This is engineering, not research — change one parameter, measure, keep or revert.

Design

pgvector tuning

Current: ivfflat index with default lists.

Target: Switch to hnsw index (pgvector 0.5.0+). HNSW provides better recall than IVFFlat at the cost of slower index builds and more memory.

-- Drop old index
DROP INDEX IF EXISTS chunks_embedding_idx;

-- Create HNSW index
CREATE INDEX chunks_embedding_hnsw_idx
ON chunks USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

Parameters:

• m = 16 — max connections per node (default 16, higher = better recall, more memory).
• ef_construction = 64 — build-time search width (default 64, higher = better recall, slower build).
• ef_search = 40 — query-time search width (set via SET hnsw.ef_search = 40).

Tuning approach:

1. Baseline: measure recall@50 with IVFFlat.
2. Switch to HNSW with defaults.
3. Measure recall@50 again.
4. If recall@50 ≥ 0.95, keep defaults. Otherwise increase ef_construction to 128.

OpenSearch tuning

Analyzer changes:

• Add edge_ngram tokenizer for typo tolerance on content field.
• Add synonym filter for common abbreviations: k8s → kubernetes, db → database, cfg → config.
• Keep standard analyzer as primary, add search_analyzer for queries.

Field boost tuning:

• content^2.0 (default — most important).
• section_title^1.8 (headings are very relevant).
• source^1.0 (file paths are useful but shouldn't dominate).
• breadcrumb^0.8 (context, not content).

BM25 parameters:

• k1 = 1.2 (term frequency saturation — default is fine).
• b = 0.75 (length normalization — default is fine).
• Don't tune these unless baseline NDCG < 0.7.

Steps

1. Create test query set: 20 queries with known-relevant documents.
2. Measure baseline NDCG@10 for pgvector (semantic-only) and OpenSearch (lexical-only).
3. Switch pgvector from IVFFlat to HNSW. Measure NDCG@10 again.
4. Update OpenSearch index template with synonym filter + edge_ngram. Reindex. Measure.
5. Record all measurements in docs/INDEX_TUNING_RESULTS.md.
6. Keep changes that improve NDCG. Revert changes that don't.

Acceptance

1. pgvector uses HNSW index (verify with \d+ chunks in psql).
2. OpenSearch index template includes synonym filter.
3. NDCG@10 measurements recorded for before/after each change.
4. No regression: post-tuning NDCG ≥ pre-tuning NDCG for both engines.
5. pgvector query latency < 150ms (p95) after HNSW switch.
6. OpenSearch query latency < 100ms (p95) after analyzer changes.

Verify

# Check pgvector index type
psql -c "\d+ chunks" | grep hnsw

# Check OpenSearch analyzer
curl -k -H "Authorization: Bearer $TOKEN" \
  https://opensearch-internal:9200/vault-test/_settings | jq '.*.settings.index.analysis'

# Run NDCG measurement
cargo run -- bench-search --queries fixtures/search_queries.yaml --output docs/INDEX_TUNING_RESULTS.md

False pass: HNSW index created but ef_search set to 1, making recall worse than IVFFlat. Check recall@50 explicitly — it should be ≥ 0.95.

Artifacts

• SQL migration (drop IVFFlat, create HNSW)
• Updated OpenSearch index template
• docs/INDEX_TUNING_RESULTS.md (measurements)
• fixtures/search_queries.yaml (test query set)