# M4.2 — `derived: true` ingest filter

| Field | Value |
|---|---|
| Phase | M4 — Skills |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M4.1, M0.5 |

## Goal

Stop the system learning from its own output.

## Facts (inlined — no spec read needed)

The cycle, and it is the only one in the design:

```
emitted skill is loaded into a session
        │
        ▼
appears verbatim in that session's transcript
        │
        ▼
transcript is ingested as evidence
        │
        ▼
reinforces the memory that produced the skill
```

No external verifier breaks it. Manual promotion (M4.1) slows it; this filter is
what actually stops it.

Mechanism: every emitted artifact records its content hash in a manifest. During
ingest, a record whose normalised text matches a known artifact is tagged
`derived: true` and **excluded from evidence** — it is still recorded in the log
so the exclusion is visible and auditable, but the gate never sees it.

Matching must survive the model reformatting the text slightly. Exact hash on the
whole record is too brittle: a skill quoted with different indentation would slip
through. Use a normalised shingle overlap — strip whitespace and markdown, hash
overlapping n-grams, and flag a record whose overlap with any artifact exceeds a
threshold.

Threshold is a tradeoff and should be logged, not hidden: too low excludes
genuine discussion *about* a skill, too high lets the cycle run.

## Steps

1. `vault/skills/.manifest.jsonl` — one line per emitted artifact:
   `{name, sha256, shingles, emitted_at}`.
2. `mem skill draft` appends to it.
3. `mem-ingest` loads the manifest and computes shingle overlap per record.
4. Overlap > threshold (default 0.8): tag `derived: true`, exclude from chunking.
5. Log a `derived_excluded` event with the record's provenance and the artifact
   it matched, so exclusions are auditable and a false positive is findable.
6. `--no-derived-filter` to disable, for debugging only, loudly warned.
7. `mem verify --derived-filter` asserts no L0 evidence node matches an artifact.

## Acceptance

- A record quoting an emitted skill verbatim is excluded.
- A record quoting it with different whitespace and fences is also excluded.
- A record merely *mentioning* the skill by name is not excluded.
- Every exclusion is logged with what it matched.

## Verify

**Harness:** a fixture manifest with one artifact, plus records in several
paraphrase grades.

**Integration test** — `tests/it_derived_filter.rs`:
1. `a1_verbatim_excluded` — exact copy of an artifact is excluded.
2. `a2_reformatted_excluded` — same content, different indentation and code
   fences; excluded.
3. `a3_mention_not_excluded` — "I used the infra-root-causes skill" is kept. This
   is the false-positive guard, and over-filtering silently starves the memory.
4. `a4_unrelated_not_excluded` — random session text is kept.
5. `a5_exclusion_logged` — assert a `derived_excluded` event naming the matched
   artifact.
6. `a6_verify_catches_leak` — insert an L0 node matching an artifact directly into
   the database; assert `mem verify --derived-filter` fails.
7. `a7_threshold_configurable` — assert the threshold is read from config and
   appears in the run record.
8. `a8_no_manifest_is_safe` — with no manifest file, ingest proceeds and filters
   nothing, rather than failing.

**Command:** `cargo test -p mem-ingest derived_filter`

**False pass:**
- Testing only the verbatim case. Exact-match filtering passes and the realistic
  case — a model that reformats what it quotes — walks straight through.
  Assertion 2 is the one that matters.
- Omitting assertion 3. A filter tuned only for recall excludes every discussion
  of a topic once a skill about it exists, which quietly makes the memory worse
  the more skills you write.

## Traps

- Filtering on record *hash*. One character of whitespace defeats it, and the
  cycle runs while the filter reports itself working.
- Silent exclusion. Without assertion 5's log event, a false positive is
  invisible — memory just gets thinner and nobody knows why.

---

Background: [DESIGN.md](../DESIGN.md) — Skills, Risks