# M8.7 — Index tuning: HNSW parameters + OpenSearch analyzers

| Field | Value |
|---|---|
| Phase | M8 — Hybrid Search |
| Size | M — 1–2 days |
| Status | ⬜ |
| Flags | — |
| Spec | inlined below |
| Blocks | M8.9 |
| Depends | M8.2 (data in both stores), M8.5 (can query both stores) |

## Goal

Tune pgvector index parameters and OpenSearch analyzers for retrieval accuracy. Measure baseline NDCG before and after tuning. This is engineering, not research — change one parameter, measure, keep or revert.

## Design

### pgvector tuning

**Current:** `ivfflat` index with default `lists`.

**Target:** Switch to `hnsw` index (pgvector 0.5.0+). HNSW provides better recall than IVFFlat at the cost of slower index builds and more memory.

```sql
-- Drop old index
DROP INDEX IF EXISTS chunks_embedding_idx;

-- Create HNSW index
CREATE INDEX chunks_embedding_hnsw_idx
ON chunks USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);
```

Parameters:
- `m = 16` — max connections per node (default 16, higher = better recall, more memory).
- `ef_construction = 64` — build-time search width (default 64, higher = better recall, slower build).
- `ef_search = 40` — query-time search width (set via `SET hnsw.ef_search = 40`).

**Tuning approach:**
1. Baseline: measure recall@50 with IVFFlat.
2. Switch to HNSW with defaults.
3. Measure recall@50 again.
4. If recall@50 ≥ 0.95, keep defaults. Otherwise increase `ef_construction` to 128.

### OpenSearch tuning

**Analyzer changes:**
- Add `edge_ngram` tokenizer for typo tolerance on `content` field.
- Add `synonym` filter for common abbreviations: `k8s → kubernetes`, `db → database`, `cfg → config`.
- Keep `standard` analyzer as primary, add `search_analyzer` for queries.

**Field boost tuning:**
- `content^2.0` (default — most important).
- `section_title^1.8` (headings are very relevant).
- `source^1.0` (file paths are useful but shouldn't dominate).
- `breadcrumb^0.8` (context, not content).

**BM25 parameters:**
- `k1 = 1.2` (term frequency saturation — default is fine).
- `b = 0.75` (length normalization — default is fine).
- Don't tune these unless baseline NDCG < 0.7.

## Steps

1. Create test query set: 20 queries with known-relevant documents.
2. Measure baseline NDCG@10 for pgvector (semantic-only) and OpenSearch (lexical-only).
3. Switch pgvector from IVFFlat to HNSW. Measure NDCG@10 again.
4. Update OpenSearch index template with synonym filter + edge_ngram. Reindex. Measure.
5. Record all measurements in `docs/INDEX_TUNING_RESULTS.md`.
6. Keep changes that improve NDCG. Revert changes that don't.

## Acceptance

1. pgvector uses HNSW index (verify with `\d+ chunks` in psql).
2. OpenSearch index template includes synonym filter.
3. NDCG@10 measurements recorded for before/after each change.
4. No regression: post-tuning NDCG ≥ pre-tuning NDCG for both engines.
5. pgvector query latency < 150ms (p95) after HNSW switch.
6. OpenSearch query latency < 100ms (p95) after analyzer changes.

## Verify

```bash
# Check pgvector index type
psql -c "\d+ chunks" | grep hnsw

# Check OpenSearch analyzer
curl -k -H "Authorization: Bearer $TOKEN" \
  https://opensearch-internal:9200/vault-test/_settings | jq '.*.settings.index.analysis'

# Run NDCG measurement
cargo run -- bench-search --queries fixtures/search_queries.yaml --output docs/INDEX_TUNING_RESULTS.md
```

**False pass:** HNSW index created but `ef_search` set to 1, making recall worse than IVFFlat. Check recall@50 explicitly — it should be ≥ 0.95.

## Artifacts

- SQL migration (drop IVFFlat, create HNSW)
- Updated OpenSearch index template
- `docs/INDEX_TUNING_RESULTS.md` (measurements)
- `fixtures/search_queries.yaml` (test query set)