poimen-memory/tasks/M8.9-m8-gate.md

# M8.9 — M8 composition gate: hybrid search proves its value

| Field | Value |
|---|---|
| Phase | M8 — Hybrid Search |
| Size | M — 1 day |
| Status | ⬜ |
| Flags | gate |
| Spec | inlined below |
| Blocks | — |
| Depends | M8.1–M8.8 all ✅ |

## Goal

Prove the hybrid search system works end-to-end and is measurably better than semantic-only. This gate verifies the **composition** — individual tasks pass their own tests, but only the gate proves they compose correctly.

## Properties to verify

### P1: Dual-write consistency

Every chunk in pgvector has a corresponding document in OpenSearch with the same ID, and vice versa. Zero orphans.

```sql
-- pgvector IDs not in OpenSearch
SELECT id FROM chunks WHERE opensearch_pending = true;
-- Must return 0 rows (after background retry has run)
```

```bash
# OpenSearch document count must equal pgvector chunk count for same project
PG_COUNT=$(psql -t -c "SELECT count(*) FROM chunks WHERE project='test'")
OS_COUNT=$(curl -sk "https://opensearch:9200/vault-test/_count" | jq .count)
# PG_COUNT == OS_COUNT
```

### P2: Hybrid outperforms single-engine

From `docs/BENCHMARK_RESULTS.md`:
- Hybrid NDCG@10 > semantic-only NDCG@10.
- Hybrid NDCG@10 > lexical-only NDCG@10.
- If this fails for a specific query category, it must be documented with reasoning.

### P3: Fallback works under failure

1. Stop OpenSearch. Query endpoint still responds with semantic results.
2. Start OpenSearch. Query endpoint returns hybrid results.
3. Response `search_strategy` field accurately reports which mode was used.

### P4: JWT auth enforced end-to-end

1. Query without token → 401.
2. Query with valid token → 200.
3. OpenSearch rejects requests from non-Memory-Service pods (NetworkPolicy).
4. JWT token forwarded from Memory Service to OpenSearch (not admin credentials).

### P5: No regression on existing tests

All pre-existing tests still pass. `cargo test` green. No `#[ignore]` added in M8.

### P6: Latency budget met

- Hybrid query: p95 < 500ms.
- Semantic-only query: p95 < 200ms (must not regress from adding hybrid path).
- Fallback to semantic: p95 < 250ms (minimal overhead from failed OpenSearch attempt).

## Gate test

```bash
#!/bin/bash
set -euo pipefail

echo "=== M8 Gate: Hybrid Search ==="

# P5: All tests pass
cargo test 2>&1 | tail -1
# Expected: test result: ok. X passed; 0 failed

# P1: Dual-write consistency
PG=$(psql -t -c "SELECT count(*) FROM chunks WHERE project='test' AND opensearch_pending=false")
OS=$(curl -sk "https://opensearch:9200/vault-test/_count" | jq .count)
[ "$PG" -eq "$OS" ] && echo "P1 PASS: $PG chunks in both stores" || echo "P1 FAIL: pg=$PG os=$OS"

# P2: Hybrid > single-engine
grep -A1 "hybrid" docs/BENCHMARK_RESULTS.md | grep -oP '[\d.]+' | head -1
# Must be highest NDCG in the table

# P3: Fallback
kubectl scale statefulset/opensearch -n poimen --replicas=0
sleep 5
STRATEGY=$(curl -s -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8080/memory/query?query=test&project=test" | jq -r .search_strategy)
[ "$STRATEGY" = "semantic_fallback" ] && echo "P3 PASS: fallback works" || echo "P3 FAIL: $STRATEGY"
kubectl scale statefulset/opensearch -n poimen --replicas=2
sleep 30
STRATEGY=$(curl -s -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8080/memory/query?query=test&project=test" | jq -r .search_strategy)
[ "$STRATEGY" = "Hybrid" ] && echo "P3 PASS: hybrid restored" || echo "P3 FAIL: $STRATEGY"

# P4: Auth
STATUS=$(curl -s -o /dev/null -w '%{http_code}' "http://localhost:8080/memory/query?query=test&project=test")
[ "$STATUS" = "401" ] && echo "P4 PASS: no-auth rejected" || echo "P4 FAIL: $STATUS"

echo "=== M8 Gate Complete ==="
```

## Acceptance

All six properties pass. If P2 fails (hybrid not better), the gate does NOT pass — go back and fix M8.7 (index tuning) or M8.4 (fusion algorithm).

## False passes to check

1. **P1 looks green but IDs don't match.** Run a JOIN, not just count comparison.
2. **P3 looks green but fallback latency is 30s** (timeout, not fast fail). Check p95 < 250ms.
3. **P5 looks green but test count dropped.** Compare `cargo test 2>&1 | grep 'test result'` against last known count (currently 239+).

## Artifacts

- Gate script (inline above)
- `docs/BENCHMARK_RESULTS.md` (from M8.8)
- All M8.1–M8.8 artifacts