110 lines
4.1 KiB
Markdown
110 lines
4.1 KiB
Markdown
|
# M4.2 — `derived: true` ingest filter
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | M4 — Skills |
|
|||
|
|
| Size | M — 1–3 days |
|
|||
|
|
| Status | ⬜ Not started |
|
|||
|
|
| Flags | — |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | M4.1, M0.5 |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Stop the system learning from its own output.
|
|||
|
|
|
|||
|
|
## Facts (inlined — no spec read needed)
|
|||
|
|
|
|||
|
|
The cycle, and it is the only one in the design:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
emitted skill is loaded into a session
|
|||
|
|
│
|
|||
|
|
▼
|
|||
|
|
appears verbatim in that session's transcript
|
|||
|
|
│
|
|||
|
|
▼
|
|||
|
|
transcript is ingested as evidence
|
|||
|
|
│
|
|||
|
|
▼
|
|||
|
|
reinforces the memory that produced the skill
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
No external verifier breaks it. Manual promotion (M4.1) slows it; this filter is
|
|||
|
|
what actually stops it.
|
|||
|
|
|
|||
|
|
Mechanism: every emitted artifact records its content hash in a manifest. During
|
|||
|
|
ingest, a record whose normalised text matches a known artifact is tagged
|
|||
|
|
`derived: true` and **excluded from evidence** — it is still recorded in the log
|
|||
|
|
so the exclusion is visible and auditable, but the gate never sees it.
|
|||
|
|
|
|||
|
|
Matching must survive the model reformatting the text slightly. Exact hash on the
|
|||
|
|
whole record is too brittle: a skill quoted with different indentation would slip
|
|||
|
|
through. Use a normalised shingle overlap — strip whitespace and markdown, hash
|
|||
|
|
overlapping n-grams, and flag a record whose overlap with any artifact exceeds a
|
|||
|
|
threshold.
|
|||
|
|
|
|||
|
|
Threshold is a tradeoff and should be logged, not hidden: too low excludes
|
|||
|
|
genuine discussion *about* a skill, too high lets the cycle run.
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. `vault/skills/.manifest.jsonl` — one line per emitted artifact:
|
|||
|
|
`{name, sha256, shingles, emitted_at}`.
|
|||
|
|
2. `mem skill draft` appends to it.
|
|||
|
|
3. `mem-ingest` loads the manifest and computes shingle overlap per record.
|
|||
|
|
4. Overlap > threshold (default 0.8): tag `derived: true`, exclude from chunking.
|
|||
|
|
5. Log a `derived_excluded` event with the record's provenance and the artifact
|
|||
|
|
it matched, so exclusions are auditable and a false positive is findable.
|
|||
|
|
6. `--no-derived-filter` to disable, for debugging only, loudly warned.
|
|||
|
|
7. `mem verify --derived-filter` asserts no L0 evidence node matches an artifact.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
- A record quoting an emitted skill verbatim is excluded.
|
|||
|
|
- A record quoting it with different whitespace and fences is also excluded.
|
|||
|
|
- A record merely *mentioning* the skill by name is not excluded.
|
|||
|
|
- Every exclusion is logged with what it matched.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Harness:** a fixture manifest with one artifact, plus records in several
|
|||
|
|
paraphrase grades.
|
|||
|
|
|
|||
|
|
**Integration test** — `tests/it_derived_filter.rs`:
|
|||
|
|
1. `a1_verbatim_excluded` — exact copy of an artifact is excluded.
|
|||
|
|
2. `a2_reformatted_excluded` — same content, different indentation and code
|
|||
|
|
fences; excluded.
|
|||
|
|
3. `a3_mention_not_excluded` — "I used the infra-root-causes skill" is kept. This
|
|||
|
|
is the false-positive guard, and over-filtering silently starves the memory.
|
|||
|
|
4. `a4_unrelated_not_excluded` — random session text is kept.
|
|||
|
|
5. `a5_exclusion_logged` — assert a `derived_excluded` event naming the matched
|
|||
|
|
artifact.
|
|||
|
|
6. `a6_verify_catches_leak` — insert an L0 node matching an artifact directly into
|
|||
|
|
the database; assert `mem verify --derived-filter` fails.
|
|||
|
|
7. `a7_threshold_configurable` — assert the threshold is read from config and
|
|||
|
|
appears in the run record.
|
|||
|
|
8. `a8_no_manifest_is_safe` — with no manifest file, ingest proceeds and filters
|
|||
|
|
nothing, rather than failing.
|
|||
|
|
|
|||
|
|
**Command:** `cargo test -p mem-ingest derived_filter`
|
|||
|
|
|
|||
|
|
**False pass:**
|
|||
|
|
- Testing only the verbatim case. Exact-match filtering passes and the realistic
|
|||
|
|
case — a model that reformats what it quotes — walks straight through.
|
|||
|
|
Assertion 2 is the one that matters.
|
|||
|
|
- Omitting assertion 3. A filter tuned only for recall excludes every discussion
|
|||
|
|
of a topic once a skill about it exists, which quietly makes the memory worse
|
|||
|
|
the more skills you write.
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- Filtering on record *hash*. One character of whitespace defeats it, and the
|
|||
|
|
cycle runs while the filter reports itself working.
|
|||
|
|
- Silent exclusion. Without assertion 5's log event, a false positive is
|
|||
|
|
invisible — memory just gets thinner and nobody knows why.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background: [DESIGN.md](../DESIGN.md) — Skills, Risks
|