feat(M4.2): Implement shingle-based cycle guard (derived filter)
Adds normalized shingle matching to prevent feedback loops where emitted skills
are re-ingested as evidence:
Files created:
crates/mem-core/src/shingle.rs (250 LOC)
- Shingle: normalized n-gram wrapper
- ShingleConfig: configurable threshold (default 0.80) and size (default 4)
- normalize(): removes markdown, code fences, collapses whitespace
- get_shingles(): overlapping token n-grams
- jaccard_similarity(): Jaccard index for text comparison
- matches_artifact(): detect if record matches any artifact above threshold
tests/it_derived_filter.rs (11 tests, all passing)
- a1: Verbatim artifact copies detected
- a2: Reformatted copies (whitespace/markdown) detected
- a3: Mere mentions of skill names NOT excluded (false positive guard)
- a4: Unrelated text NOT excluded
- a5: Multiple artifacts handled correctly
- a6: Threshold configurable
- a7: Similarity score returned
- a8: No artifacts is safe (empty list)
- a9: Empty text is safe
- a10: Case-insensitive matching
- a11: Partial coverage detection
Files modified:
crates/mem-core/src/lib.rs
- Add shingle module
- Export ShingleConfig, jaccard_similarity, matches_artifact
Architecture:
During ingest: compare record against vault/.artifacts.jsonl
If overlap >= threshold: tag derived=true, exclude from evidence
Log exclusion event for auditability
Threshold tuning:
- 0.80: strict, catches verbatim + reformatted
- 0.70: moderate, catches variants
- 0.60: permissive, catches substantial overlap
Default 0.80 prevents false positives (mentioning skill != using skill text)
Tests:
✓ 11/11 passing
✓ Unit tests in shingle module: 11/11 passing
✓ Integration tests: 11/11 passing
Blocks: M4.3 gate (needs ingest integration)
Depends: M4.1 ✓ (skill draft)
Story Crater Bot
383d5ae0d1