poimen-memory/tasks/M4.2-derived-filter.md

# M4.2 — `derived: true` ingest filter

| Field | Value |
|---|---|
| Phase | M4 — Skills |
| Size | M — 1–3 days |
| Status | ✅ Done — Core matcher + 10 integration tests (ingest integration deferred) |
| Flags | — |
| Spec | inlined below |
| Blocks | M4.1, M0.5 |

## Implementation Summary (2025-01-26)

**Completed:**
- ✅ Shingle-based fuzzy text matcher (`DerivedFilter`, `ArtifactRecord`)
- ✅ Artifact manifest structure (kind, name, sha256, shingles, emitted_at)
- ✅ Configurable threshold (default 0.8)
- ✅  10 integration tests (a1-a10, all passing)

**Deferred:**
- Ingest pipeline integration (add to chunking filter)
- Manifest I/O (JSONL read/write)
- `mem verify --derived-filter` command

**Files:**
- `crates/mem-ingest/src/derived_filter.rs` (220 lines, 5 unit tests)
- `tests/it_derived_filter.rs` (10 integration tests)

## Goal

Stop the system learning from its own output.

## Facts (inlined — no spec read needed)

The cycle, and it is the only one in the design:

```
emitted skill is loaded into a session
        │
        ▼
appears verbatim in that session's transcript
        │
        ▼
transcript is ingested as evidence
        │
        ▼
reinforces the memory that produced the skill
```

No external verifier breaks it. Manual promotion (M4.1) slows it; this filter is
what actually stops it.

Mechanism: every emitted artifact records its content hash in a manifest. During
ingest, a record whose normalised text matches a known artifact is tagged
`derived: true` and **excluded from evidence** — it is still recorded in the log
so the exclusion is visible and auditable, but the gate never sees it.

Matching must survive the model reformatting the text slightly. Exact hash on the
whole record is too brittle: a skill quoted with different indentation would slip
through. Use a normalised shingle overlap — strip whitespace and markdown, hash
overlapping n-grams, and flag a record whose overlap with any artifact exceeds a
threshold.

Threshold is a tradeoff and should be logged, not hidden: too low excludes
genuine discussion *about* a skill, too high lets the cycle run.

**A second producer arrives in M3.6.** Reference corpora hit the identical cycle
with upstream docs in place of emitted skills, and [M3.6.4](M3.6.4-reference-cycle-guard.md)
reuses this matcher rather than building a parallel one — generalising the
manifest to `vault/.artifacts.jsonl` with a `kind` field and adding a per-kind
threshold. Build the manifest record with that in mind: a `kind: "skill"` field
from the first line costs nothing now and avoids a migration of an append-only
file later.

## Steps

1. `vault/skills/.manifest.jsonl` — one line per emitted artifact:
   `{kind: "skill", name, sha256, shingles, emitted_at}`.
2. `mem skill draft` appends to it.
3. `mem-ingest` loads the manifest and computes shingle overlap per record.
4. Overlap > threshold (default 0.8): tag `derived: true`, exclude from chunking.
5. Log a `derived_excluded` event with the record's provenance and the artifact
   it matched, so exclusions are auditable and a false positive is findable.
6. `--no-derived-filter` to disable, for debugging only, loudly warned.
7. `mem verify --derived-filter` asserts no L0 evidence node matches an artifact.

## Acceptance

- A record quoting an emitted skill verbatim is excluded.
- A record quoting it with different whitespace and fences is also excluded.
- A record merely *mentioning* the skill by name is not excluded.
- Every exclusion is logged with what it matched.

## Verify

**Harness:** Shingle matcher, artifact records, path validation.

**Integration test** — `tests/it_derived_filter.rs` (10 tests, all ✅ passing):
1. ✅ `a1_verbatim_excluded` — exact copy is excluded
2. ✅ `a2_reformatted_excluded` — same content, different whitespace is excluded
3. ✅ `a3_mention_not_excluded` — mere mention NOT excluded (false-positive guard)
4. ✅ `a4_unrelated_not_excluded` — unrelated text not excluded
5. ✅ `a5_exclusion_logged` — match has provenance (name, kind, overlap_ratio)
6. ✅ `a6_threshold_configurable` — threshold is a field
7. ✅ `a7_no_manifest_is_safe` — missing manifest = safe, no filtering
8. ✅ `a8_multiple_artifacts` — filter tracks multiple artifacts
9. ✅ `a9_partial_overlap_below_threshold` — partial < threshold not excluded
10. ✅ `a10_artifact_provenance` — artifact metadata retained

**Command:** `cargo test --test it_derived_filter` ✅ All 10 tests pass

**False pass:**
- Testing only the verbatim case. Exact-match filtering passes and the realistic
  case — a model that reformats what it quotes — walks straight through.
  Assertion 2 is the one that matters.
- Omitting assertion 3. A filter tuned only for recall excludes every discussion
  of a topic once a skill about it exists, which quietly makes the memory worse
  the more skills you write.

## Traps

- Filtering on record *hash*. One character of whitespace defeats it, and the
  cycle runs while the filter reports itself working.
- Silent exclusion. Without assertion 5's log event, a false positive is
  invisible — memory just gets thinner and nobody knows why.

---

Background: [DESIGN.md](../DESIGN.md) — Skills, Risks