poimen-memory/tasks/M4.2-derived-filter.md

M4.2 — derived: true ingest filter

Field	Value
Phase	M4 — Skills
Size	M — 1–3 days
Status	⬜ Not started
Flags	—
Spec	inlined below
Blocks	M4.1, M0.5

Goal

Stop the system learning from its own output.

Facts (inlined — no spec read needed)

The cycle, and it is the only one in the design:

emitted skill is loaded into a session
        │
        ▼
appears verbatim in that session's transcript
        │
        ▼
transcript is ingested as evidence
        │
        ▼
reinforces the memory that produced the skill

No external verifier breaks it. Manual promotion (M4.1) slows it; this filter is what actually stops it.

Mechanism: every emitted artifact records its content hash in a manifest. During ingest, a record whose normalised text matches a known artifact is tagged derived: true and excluded from evidence — it is still recorded in the log so the exclusion is visible and auditable, but the gate never sees it.

Matching must survive the model reformatting the text slightly. Exact hash on the whole record is too brittle: a skill quoted with different indentation would slip through. Use a normalised shingle overlap — strip whitespace and markdown, hash overlapping n-grams, and flag a record whose overlap with any artifact exceeds a threshold.

Threshold is a tradeoff and should be logged, not hidden: too low excludes genuine discussion about a skill, too high lets the cycle run.

Steps

1. vault/skills/.manifest.jsonl — one line per emitted artifact: {name, sha256, shingles, emitted_at}.
2. mem skill draft appends to it.
3. mem-ingest loads the manifest and computes shingle overlap per record.
4. Overlap > threshold (default 0.8): tag derived: true, exclude from chunking.
5. Log a derived_excluded event with the record's provenance and the artifact it matched, so exclusions are auditable and a false positive is findable.
6. --no-derived-filter to disable, for debugging only, loudly warned.
7. mem verify --derived-filter asserts no L0 evidence node matches an artifact.

Acceptance

• A record quoting an emitted skill verbatim is excluded.
• A record quoting it with different whitespace and fences is also excluded.
• A record merely mentioning the skill by name is not excluded.
• Every exclusion is logged with what it matched.

Verify

Harness: a fixture manifest with one artifact, plus records in several paraphrase grades.

Integration test — tests/it_derived_filter.rs:

1. a1_verbatim_excluded — exact copy of an artifact is excluded.
2. a2_reformatted_excluded — same content, different indentation and code fences; excluded.
3. a3_mention_not_excluded — "I used the infra-root-causes skill" is kept. This is the false-positive guard, and over-filtering silently starves the memory.
4. a4_unrelated_not_excluded — random session text is kept.
5. a5_exclusion_logged — assert a derived_excluded event naming the matched artifact.
6. a6_verify_catches_leak — insert an L0 node matching an artifact directly into the database; assert mem verify --derived-filter fails.
7. a7_threshold_configurable — assert the threshold is read from config and appears in the run record.
8. a8_no_manifest_is_safe — with no manifest file, ingest proceeds and filters nothing, rather than failing.

Command: cargo test -p mem-ingest derived_filter

False pass:

• Testing only the verbatim case. Exact-match filtering passes and the realistic case — a model that reformats what it quotes — walks straight through. Assertion 2 is the one that matters.
• Omitting assertion 3. A filter tuned only for recall excludes every discussion of a topic once a skill about it exists, which quietly makes the memory worse the more skills you write.

Traps

• Filtering on record hash. One character of whitespace defeats it, and the cycle runs while the filter reports itself working.
• Silent exclusion. Without assertion 5's log event, a false positive is invisible — memory just gets thinner and nobody knows why.

---

Background: DESIGN.md — Skills, Risks