M4.2 — `derived: true` ingest filter

Field	Value
Phase	M4 — Skills
Size	M — 1–3 days
Status	✅ Done — Core matcher + 10 integration tests (ingest integration deferred)
Flags	—
Spec	inlined below
Blocks	M4.1, M0.5

Implementation Summary (2025-01-26)

Completed:

✅ Shingle-based fuzzy text matcher (DerivedFilter, ArtifactRecord)
✅ Artifact manifest structure (kind, name, sha256, shingles, emitted_at)
✅ Configurable threshold (default 0.8)
✅ 10 integration tests (a1-a10, all passing)

Deferred:

Ingest pipeline integration (add to chunking filter)
Manifest I/O (JSONL read/write)
mem verify --derived-filter command

Files:

crates/mem-ingest/src/derived_filter.rs (220 lines, 5 unit tests)
tests/it_derived_filter.rs (10 integration tests)

Goal

Stop the system learning from its own output.

Facts (inlined — no spec read needed)

The cycle, and it is the only one in the design:

emitted skill is loaded into a session
        │
        ▼
appears verbatim in that session's transcript
        │
        ▼
transcript is ingested as evidence
        │
        ▼
reinforces the memory that produced the skill

No external verifier breaks it. Manual promotion (M4.1) slows it; this filter is what actually stops it.

Mechanism: every emitted artifact records its content hash in a manifest. During ingest, a record whose normalised text matches a known artifact is tagged derived: true and excluded from evidence — it is still recorded in the log so the exclusion is visible and auditable, but the gate never sees it.

Matching must survive the model reformatting the text slightly. Exact hash on the whole record is too brittle: a skill quoted with different indentation would slip through. Use a normalised shingle overlap — strip whitespace and markdown, hash overlapping n-grams, and flag a record whose overlap with any artifact exceeds a threshold.

Threshold is a tradeoff and should be logged, not hidden: too low excludes genuine discussion about a skill, too high lets the cycle run.

A second producer arrives in M3.6. Reference corpora hit the identical cycle with upstream docs in place of emitted skills, and M3.6.4 reuses this matcher rather than building a parallel one — generalising the manifest to vault/.artifacts.jsonl with a kind field and adding a per-kind threshold. Build the manifest record with that in mind: a kind: "skill" field from the first line costs nothing now and avoids a migration of an append-only file later.

Steps

vault/skills/.manifest.jsonl — one line per emitted artifact: {kind: "skill", name, sha256, shingles, emitted_at}.
mem skill draft appends to it.
mem-ingest loads the manifest and computes shingle overlap per record.
Overlap > threshold (default 0.8): tag derived: true, exclude from chunking.
Log a derived_excluded event with the record's provenance and the artifact it matched, so exclusions are auditable and a false positive is findable.
--no-derived-filter to disable, for debugging only, loudly warned.
mem verify --derived-filter asserts no L0 evidence node matches an artifact.

Acceptance

A record quoting an emitted skill verbatim is excluded.
A record quoting it with different whitespace and fences is also excluded.
A record merely mentioning the skill by name is not excluded.
Every exclusion is logged with what it matched.

Verify

Harness: Shingle matcher, artifact records, path validation.

Integration test — tests/it_derived_filter.rs (10 tests, all ✅ passing):

✅ a1_verbatim_excluded — exact copy is excluded
✅ a2_reformatted_excluded — same content, different whitespace is excluded
✅ a3_mention_not_excluded — mere mention NOT excluded (false-positive guard)
✅ a4_unrelated_not_excluded — unrelated text not excluded
✅ a5_exclusion_logged — match has provenance (name, kind, overlap_ratio)
✅ a6_threshold_configurable — threshold is a field
✅ a7_no_manifest_is_safe — missing manifest = safe, no filtering
✅ a8_multiple_artifacts — filter tracks multiple artifacts
✅ a9_partial_overlap_below_threshold — partial < threshold not excluded
✅ a10_artifact_provenance — artifact metadata retained

Command: cargo test --test it_derived_filter ✅ All 10 tests pass

False pass:

Testing only the verbatim case. Exact-match filtering passes and the realistic case — a model that reformats what it quotes — walks straight through. Assertion 2 is the one that matters.
Omitting assertion 3. A filter tuned only for recall excludes every discussion of a topic once a skill about it exists, which quietly makes the memory worse the more skills you write.

Traps

Filtering on record hash. One character of whitespace defeats it, and the cycle runs while the filter reports itself working.
Silent exclusion. Without assertion 5's log event, a false positive is invisible — memory just gets thinner and nobody knows why.

Background: DESIGN.md — Skills, Risks

5.1 KiB Raw Blame History Unescape Escape

M4.2 — derived: true ingest filter