94 lines
3.6 KiB
Markdown
94 lines
3.6 KiB
Markdown
|
# M3.4 — M3 composition gate
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | M3 — L2 synthesis and retrieval |
|
|||
|
|
| Size | M — 1–3 days |
|
|||
|
|
| Status | ⬜ Not started |
|
|||
|
|
| Flags | gate |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | all of M3 |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Prove the whole stack answers a real question with real provenance — the first
|
|||
|
|
point at which the system is useful rather than merely correct.
|
|||
|
|
|
|||
|
|
## Facts (inlined — no spec read needed)
|
|||
|
|
|
|||
|
|
The gate is a small set of **known-answer questions** with hand-written expected
|
|||
|
|
sources, committed to the repo. Each names a fact that genuinely appears in the
|
|||
|
|
ingested sessions and the session it appears in.
|
|||
|
|
|
|||
|
|
Seed set, all drawn from real work in this corpus:
|
|||
|
|
|
|||
|
|
| question | expected to cite |
|
|||
|
|
|---|---|
|
|||
|
|
| why did requests over 10KB fail? | the Kong body-buffer / `client_body_buffer_size` finding |
|
|||
|
|
| why did `Authorization: Bearer` return 401? | the Kong key-auth header finding |
|
|||
|
|
| what causes the 504 on a cold ornith start? | the timeouts-on-Ingress-vs-Service finding |
|
|||
|
|
|
|||
|
|
This is a retrieval quality gate, so it is graded, not boolean: report
|
|||
|
|
**hit rate at k=5** and **provenance precision** (fraction of cited sources that
|
|||
|
|
actually contain the fact). A single failing question is information, not
|
|||
|
|
necessarily a stop.
|
|||
|
|
|
|||
|
|
The gate also re-runs `mem verify` including the level invariant, because M3.1 is
|
|||
|
|
the change most likely to break it.
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. `verify/known-answers.yaml` — question, expected node, expected source.
|
|||
|
|
2. `verify/m3.4.sh` runs each through `mem query --format json`.
|
|||
|
|
3. Compute hit rate at 5 and provenance precision; print both.
|
|||
|
|
4. Assert the thresholds below.
|
|||
|
|
5. Run `mem verify`; assert zero violations.
|
|||
|
|
6. Assert L2 exists and its provenance resolves two hops.
|
|||
|
|
7. Commit `expected/m3.4.txt`; diff.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
- Hit rate at k=5 ≥ 0.8 on the known-answer set.
|
|||
|
|
- Provenance precision ≥ 0.9 — a citation that does not contain the fact is worse
|
|||
|
|
than no citation.
|
|||
|
|
- `mem verify` clean, including level consistency.
|
|||
|
|
- Every L2 hit resolves to L0 sources.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Harness:** live gateway, seeded database from real logs. On-demand, not
|
|||
|
|
per-push.
|
|||
|
|
|
|||
|
|
**Integration test** — `verify/m3.4.sh` diffed against `expected/m3.4.txt`:
|
|||
|
|
1. `a1_hit_rate` — ≥ 0.8, print actual.
|
|||
|
|
2. `a2_provenance_precision` — for each cited source, fetch the L0 text and
|
|||
|
|
assert it contains the expected fact substring; ≥ 0.9.
|
|||
|
|
3. `a3_verify_clean` — zero violations.
|
|||
|
|
4. `a4_l2_two_hop` — every L2 hit resolves through L1 to a real source.
|
|||
|
|
5. `a5_rerank_contributes` — hit rate with reranking is ≥ hit rate without. If
|
|||
|
|
reranking makes it worse, the wiring is wrong (probably index mapping, M3.2).
|
|||
|
|
6. `a6_no_cross_project` — a question about another project returns nothing from
|
|||
|
|
this one.
|
|||
|
|
|
|||
|
|
**Command:** `bash verify/m3.4.sh | diff - expected/m3.4.txt`
|
|||
|
|
|
|||
|
|
**False pass:**
|
|||
|
|
- Writing the known-answer set after seeing what the system returns. It then
|
|||
|
|
measures nothing. Write the questions and expected sources from the sessions
|
|||
|
|
first, independently of any query output.
|
|||
|
|
- Measuring hit rate without provenance precision. A system that returns the
|
|||
|
|
right note with fabricated citations scores 1.0 on hits and is untrustworthy —
|
|||
|
|
assertion 2 is the one that matters for whether anyone can act on an answer.
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- Tuning `k` until the hit rate passes. k=50 will hit almost everything and the
|
|||
|
|
metric stops meaning anything; the gate specifies k=5 for that reason.
|
|||
|
|
- Accepting a5 failing as "reranker is just not helping". It far more often means
|
|||
|
|
the `index` mapping in M3.2 is wrong and scores are attached to the wrong
|
|||
|
|
documents.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background: [DESIGN.md](../DESIGN.md) — Verification
|