143 lines
6.4 KiB
Markdown
143 lines
6.4 KiB
Markdown
# M3.6.4 — Reference text cannot re-enter as evidence
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M3.6 — Reference corpora |
|
||
| Size | M — 1–3 days |
|
||
| Status | ⬜ Not started |
|
||
| Flags | — |
|
||
| Spec | inlined below |
|
||
| Blocks | M3.6.6 |
|
||
| Depends | M3.6.3, M4.2 |
|
||
|
||
## Goal
|
||
|
||
Stop a retrieved manual page from coming back through the front door as a project
|
||
finding.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
The cycle is M4.2's, with documentation substituted for emitted skills:
|
||
|
||
```
|
||
agent queries memory, gets an R section
|
||
│
|
||
▼
|
||
section is pasted into the agent's context
|
||
│
|
||
▼
|
||
appears verbatim in that session's transcript
|
||
│
|
||
▼
|
||
transcript ingested; the gate sees upstream doc text
|
||
│
|
||
▼
|
||
"kubectl describe shows events" becomes an L1 project memory
|
||
```
|
||
|
||
The gate is *right* to accept it — the chunk genuinely contains information about
|
||
the question. That is what makes this dangerous rather than merely noisy: no
|
||
threshold tuning catches it, because the text really is relevant. Only knowing
|
||
that the system emitted the text itself distinguishes the two cases.
|
||
|
||
**Reuse M4.2, do not rebuild it.** M4.2 already computes normalised shingle
|
||
overlap against an artifact manifest and tags matching records `derived: true`,
|
||
excluding them from evidence while keeping them in the log so the exclusion is
|
||
auditable. R chunks are a second artifact kind in that same manifest. A parallel
|
||
matcher would drift from it and double the tuning surface.
|
||
|
||
```jsonl
|
||
{"kind":"reference","name":"kubectl/common-issues","sha256":"cd34…","shingles":[…],"emitted_at":"…"}
|
||
{"kind":"skill","name":"infra-root-causes","sha256":"ab12…","shingles":[…],"emitted_at":"…"}
|
||
```
|
||
|
||
**Threshold pressure differs by kind and this is the real work.** A skill is
|
||
emitted once and quoted rarely. Documentation is quoted constantly and partially
|
||
— one command from a fifty-line cheatsheet. Shingle overlap against a whole R
|
||
chunk will sit far below M4.2's 0.8 default for exactly the case that matters, so
|
||
matching must be at section granularity with its own threshold, tuned and logged
|
||
separately. One shared matcher, two configured thresholds.
|
||
|
||
**A false positive here is costly and must stay visible.** Excluding a genuine
|
||
discussion *about* `kubectl` because it quotes two lines of the cheatsheet
|
||
silently drops real evidence. Every exclusion emits `derived_excluded` naming the
|
||
matched artifact, and `mem verify` can list them for audit.
|
||
|
||
## Steps
|
||
|
||
1. Generalise M4.2's manifest to `vault/.artifacts.jsonl` with a `kind` field;
|
||
keep skills writing to it unchanged.
|
||
2. `mem ref add`/`sync` append `kind: "reference"` entries per R chunk;
|
||
tombstones remove them.
|
||
3. Add per-kind thresholds to the matcher config; default reference threshold
|
||
lower than the skill threshold, and record the value in the exclusion event.
|
||
4. Extend `mem verify --derived-filter` to assert no L0 evidence node matches a
|
||
live R artifact.
|
||
5. `mem verify --exclusions` lists recent `derived_excluded` events with the
|
||
matched artifact and overlap score, for false-positive review.
|
||
|
||
## Acceptance
|
||
|
||
- A session transcript containing a verbatim R section is excluded from evidence.
|
||
- The same transcript still appears in the log, tagged, with the match named.
|
||
- A session that merely *mentions* the tool without quoting it is not excluded.
|
||
- Skill exclusion behaviour from M4.2 is unchanged.
|
||
- Removing a corpus removes its manifest entries; previously excluded text is not
|
||
retroactively rewritten in the log.
|
||
|
||
## Verify
|
||
|
||
**Harness:** fixture corpus ingested as R, plus three synthetic transcripts — one
|
||
quoting a section verbatim, one paraphrasing it heavily, one discussing the tool
|
||
without quoting. Deterministic embedder.
|
||
|
||
**Integration test** — `tests/it_reference_cycle.rs`:
|
||
1. `a1_verbatim_quote_excluded` — the quoting transcript produces zero L0
|
||
evidence nodes; assert a `derived_excluded` event naming the R artifact.
|
||
2. `a2_discussion_not_excluded` — the non-quoting transcript produces evidence
|
||
normally. This is the false-positive guard and it is the assertion that fails
|
||
when the threshold is set too low.
|
||
3. `a3_partial_quote_caught` — the transcript quoting ~10 lines of a 50-line
|
||
section is excluded, proving section-granularity matching rather than
|
||
whole-chunk overlap.
|
||
4. `a4_skill_path_unchanged` — run M4.2's own test fixtures; assert identical
|
||
results before and after the manifest generalisation.
|
||
5. `a5_exclusion_is_auditable` — every exclusion event carries artifact name,
|
||
overlap score and the threshold in force.
|
||
6. `a6_tombstone_removes_manifest_entry` — `mem ref rm`, then assert the R
|
||
entries are gone from the manifest and the same transcript now ingests
|
||
normally.
|
||
7. `a7_verify_catches_leak` — hand-insert an L0 node whose text matches an R
|
||
artifact; assert `mem verify --derived-filter` exits non-zero and names it.
|
||
8. `a8_no_retroactive_log_edit` — after `rm`, assert prior `derived_excluded`
|
||
records are still present and unmodified.
|
||
|
||
**Command:** `cargo test -p mem-ingest reference_cycle && cargo test -p mem-cli verify`
|
||
|
||
**False pass:**
|
||
- Testing only the verbatim case. Verbatim is easy and a whole-chunk hash catches
|
||
it; assertion 3 is the one that distinguishes a working matcher, and assertion
|
||
2 is the one that proves it is not simply excluding everything that mentions
|
||
the tool.
|
||
- Asserting exclusion by checking evidence count is zero. A filter that is
|
||
accidentally excluding *all* records also yields zero; assertion 2 has to run
|
||
in the same test binary.
|
||
- Reusing M4.2's threshold unchanged and declaring it done. The default is tuned
|
||
for whole-artifact quoting; assertion 3 fails against it, which is the point.
|
||
|
||
## Traps
|
||
|
||
- Registering R chunks in the manifest before they are committed to the log. A
|
||
failed ingest then leaves manifest entries that exclude evidence for a corpus
|
||
that does not exist, and the symptom is missing memories with no obvious cause.
|
||
- Normalising differently in the two paths. If the shingler treats markdown
|
||
tables differently at emit-time and at ingest-time, overlap collapses and the
|
||
filter silently stops firing — same failure M4.2 already warns about, now with
|
||
two producers to keep in step.
|
||
- Letting the exclusion event omit the threshold. A tuning change makes every
|
||
historical exclusion uninterpretable, and this filter will be tuned.
|
||
|
||
---
|
||
|
||
Background: [DESIGN.md](../DESIGN.md) — reference corpora, skills · [M4.2](M4.2-derived-filter.md)
|