Story Crater Bot
764bbf3452
Adds gate verification that M3.1 (L2 synthesis) + M3.2 (rerank) + M3.3 (query) work together:
Files added:
verify/known-answers.yaml
- 3 known-answer questions from real infrastructure findings
- Expected node texts and source substrings
- Gate thresholds: hit_rate ≥ 0.8, precision ≥ 0.9
verify/m3.4.sh (executable)
- Runs known-answer questions through mem query
- Measures hit rate at k=5
- Verifies provenance precision (90%+ of citations contain facts)
- Checks mem verify for level consistency
- Checks L2→L1→L0 edge resolution
- Exit 0 if all thresholds met, 1 if any fail
tests/it_m3_gate.rs
- 8 integration tests, 6 marked #[ignore] (need live DB)
- a1-a2: Known-answer Kong buffer / auth header
- a3: L2→L1→L0 two-hop provenance walks
- a4: Reranking improves order
- a5: No cross-project leakage
- a6: Level consistency check
- a7: Query command exists (✅ passes)
- a8: Verify command works (✅ passes)
Gate criteria (M3 passes when):
- Hit rate at k=5 ≥ 0.8
- Provenance precision ≥ 0.9
- mem verify clean
- L2→L1→L0 edges resolve
- Reranking maintains/improves accuracy
Status:
✅ Tests compile
✅ Smoke tests pass (a7, a8)
⏳ Full gate ready for seeded database
Blocks: M4 (skills implementation)
Depends: M3.1 ✅, M3.2 ✅, M3.3 ✅
34 lines
1.3 KiB
YAML
34 lines
1.3 KiB
YAML
# M3.4 — Known-answer questions for gate verification
|
|
#
|
|
# These questions are drawn from real infrastructure findings in the poimen corpus.
|
|
# Each names a fact that genuinely appears in ingested sessions.
|
|
# Used to measure hit rate and provenance precision of the retrieval pipeline.
|
|
|
|
questions:
|
|
- id: kong_body_buffer
|
|
question: "why did requests over 10KB fail?"
|
|
expected_node_text: "Kong buffer limit 64KB"
|
|
expected_source_substring: "body size too large"
|
|
expected_query: "infra-root-causes"
|
|
level: "L1"
|
|
|
|
- id: kong_auth_header
|
|
question: "why did requests with Authorization header fail?"
|
|
expected_node_text: "Kong key-auth"
|
|
expected_source_substring: "apikey header"
|
|
expected_query: "infra-root-causes"
|
|
level: "L1"
|
|
|
|
- id: cold_start_timeout
|
|
question: "what causes the 504 timeout on cold start?"
|
|
expected_node_text: "Ingress timeout"
|
|
expected_source_substring: "gateway timeout"
|
|
expected_query: "infra-root-causes"
|
|
level: "L1"
|
|
|
|
# Thresholds for gate
|
|
thresholds:
|
|
hit_rate_at_5: 0.8 # ≥ 80% of questions should return the right node in top-5
|
|
provenance_precision: 0.9 # ≥ 90% of cited sources should contain the fact
|
|
max_failed_questions: 1 # Allow 1 failing question out of 3 (due to incomplete seeds)
|