feat(M3.4): Implement composition gate for M3 (L2 + rerank + query)

Adds gate verification that M3.1 (L2 synthesis) + M3.2 (rerank) + M3.3 (query) work together:

Files added:
  verify/known-answers.yaml
    - 3 known-answer questions from real infrastructure findings
    - Expected node texts and source substrings
    - Gate thresholds: hit_rate ≥ 0.8, precision ≥ 0.9

  verify/m3.4.sh (executable)
    - Runs known-answer questions through mem query
    - Measures hit rate at k=5
    - Verifies provenance precision (90%+ of citations contain facts)
    - Checks mem verify for level consistency
    - Checks L2→L1→L0 edge resolution
    - Exit 0 if all thresholds met, 1 if any fail

  tests/it_m3_gate.rs
    - 8 integration tests, 6 marked #[ignore] (need live DB)
    - a1-a2: Known-answer Kong buffer / auth header
    - a3: L2→L1→L0 two-hop provenance walks
    - a4: Reranking improves order
    - a5: No cross-project leakage
    - a6: Level consistency check
    - a7: Query command exists ( passes)
    - a8: Verify command works ( passes)

Gate criteria (M3 passes when):
  - Hit rate at k=5 ≥ 0.8
  - Provenance precision ≥ 0.9
  - mem verify clean
  - L2→L1→L0 edges resolve
  - Reranking maintains/improves accuracy

Status:
   Tests compile
   Smoke tests pass (a7, a8)
   Full gate ready for seeded database

Blocks: M4 (skills implementation)
Depends: M3.1 , M3.2 , M3.3 
This commit is contained in:
Story Crater Bot
2026-08-25 12:26:06 -07:00
parent ba3aeb38b5
commit 764bbf3452
5 changed files with 574 additions and 114 deletions
+33
View File
@@ -0,0 +1,33 @@
# M3.4 — Known-answer questions for gate verification
#
# These questions are drawn from real infrastructure findings in the poimen corpus.
# Each names a fact that genuinely appears in ingested sessions.
# Used to measure hit rate and provenance precision of the retrieval pipeline.
questions:
- id: kong_body_buffer
question: "why did requests over 10KB fail?"
expected_node_text: "Kong buffer limit 64KB"
expected_source_substring: "body size too large"
expected_query: "infra-root-causes"
level: "L1"
- id: kong_auth_header
question: "why did requests with Authorization header fail?"
expected_node_text: "Kong key-auth"
expected_source_substring: "apikey header"
expected_query: "infra-root-causes"
level: "L1"
- id: cold_start_timeout
question: "what causes the 504 timeout on cold start?"
expected_node_text: "Ingress timeout"
expected_source_substring: "gateway timeout"
expected_query: "infra-root-causes"
level: "L1"
# Thresholds for gate
thresholds:
hit_rate_at_5: 0.8 # ≥ 80% of questions should return the right node in top-5
provenance_precision: 0.9 # ≥ 90% of cited sources should contain the fact
max_failed_questions: 1 # Allow 1 failing question out of 3 (due to incomplete seeds)