Deploy Poimen Memory K8s cluster with ArgoCD tracking (M2.2, M3.5-M3.7)
ci / markdown (push) Waiting to run
ci / markdown (push) Waiting to run
This commit is contained in:
Story Crater Bot
@@ -0,0 +1,142 @@
|
||||
# M3.6.4 — Reference text cannot re-enter as evidence
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M3.6 — Reference corpora |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M3.6.6 |
|
||||
| Depends | M3.6.3, M4.2 |
|
||||
|
||||
## Goal
|
||||
|
||||
Stop a retrieved manual page from coming back through the front door as a project
|
||||
finding.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
The cycle is M4.2's, with documentation substituted for emitted skills:
|
||||
|
||||
```
|
||||
agent queries memory, gets an R section
|
||||
│
|
||||
▼
|
||||
section is pasted into the agent's context
|
||||
│
|
||||
▼
|
||||
appears verbatim in that session's transcript
|
||||
│
|
||||
▼
|
||||
transcript ingested; the gate sees upstream doc text
|
||||
│
|
||||
▼
|
||||
"kubectl describe shows events" becomes an L1 project memory
|
||||
```
|
||||
|
||||
The gate is *right* to accept it — the chunk genuinely contains information about
|
||||
the question. That is what makes this dangerous rather than merely noisy: no
|
||||
threshold tuning catches it, because the text really is relevant. Only knowing
|
||||
that the system emitted the text itself distinguishes the two cases.
|
||||
|
||||
**Reuse M4.2, do not rebuild it.** M4.2 already computes normalised shingle
|
||||
overlap against an artifact manifest and tags matching records `derived: true`,
|
||||
excluding them from evidence while keeping them in the log so the exclusion is
|
||||
auditable. R chunks are a second artifact kind in that same manifest. A parallel
|
||||
matcher would drift from it and double the tuning surface.
|
||||
|
||||
```jsonl
|
||||
{"kind":"reference","name":"kubectl/common-issues","sha256":"cd34…","shingles":[…],"emitted_at":"…"}
|
||||
{"kind":"skill","name":"infra-root-causes","sha256":"ab12…","shingles":[…],"emitted_at":"…"}
|
||||
```
|
||||
|
||||
**Threshold pressure differs by kind and this is the real work.** A skill is
|
||||
emitted once and quoted rarely. Documentation is quoted constantly and partially
|
||||
— one command from a fifty-line cheatsheet. Shingle overlap against a whole R
|
||||
chunk will sit far below M4.2's 0.8 default for exactly the case that matters, so
|
||||
matching must be at section granularity with its own threshold, tuned and logged
|
||||
separately. One shared matcher, two configured thresholds.
|
||||
|
||||
**A false positive here is costly and must stay visible.** Excluding a genuine
|
||||
discussion *about* `kubectl` because it quotes two lines of the cheatsheet
|
||||
silently drops real evidence. Every exclusion emits `derived_excluded` naming the
|
||||
matched artifact, and `mem verify` can list them for audit.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Generalise M4.2's manifest to `vault/.artifacts.jsonl` with a `kind` field;
|
||||
keep skills writing to it unchanged.
|
||||
2. `mem ref add`/`sync` append `kind: "reference"` entries per R chunk;
|
||||
tombstones remove them.
|
||||
3. Add per-kind thresholds to the matcher config; default reference threshold
|
||||
lower than the skill threshold, and record the value in the exclusion event.
|
||||
4. Extend `mem verify --derived-filter` to assert no L0 evidence node matches a
|
||||
live R artifact.
|
||||
5. `mem verify --exclusions` lists recent `derived_excluded` events with the
|
||||
matched artifact and overlap score, for false-positive review.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- A session transcript containing a verbatim R section is excluded from evidence.
|
||||
- The same transcript still appears in the log, tagged, with the match named.
|
||||
- A session that merely *mentions* the tool without quoting it is not excluded.
|
||||
- Skill exclusion behaviour from M4.2 is unchanged.
|
||||
- Removing a corpus removes its manifest entries; previously excluded text is not
|
||||
retroactively rewritten in the log.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** fixture corpus ingested as R, plus three synthetic transcripts — one
|
||||
quoting a section verbatim, one paraphrasing it heavily, one discussing the tool
|
||||
without quoting. Deterministic embedder.
|
||||
|
||||
**Integration test** — `tests/it_reference_cycle.rs`:
|
||||
1. `a1_verbatim_quote_excluded` — the quoting transcript produces zero L0
|
||||
evidence nodes; assert a `derived_excluded` event naming the R artifact.
|
||||
2. `a2_discussion_not_excluded` — the non-quoting transcript produces evidence
|
||||
normally. This is the false-positive guard and it is the assertion that fails
|
||||
when the threshold is set too low.
|
||||
3. `a3_partial_quote_caught` — the transcript quoting ~10 lines of a 50-line
|
||||
section is excluded, proving section-granularity matching rather than
|
||||
whole-chunk overlap.
|
||||
4. `a4_skill_path_unchanged` — run M4.2's own test fixtures; assert identical
|
||||
results before and after the manifest generalisation.
|
||||
5. `a5_exclusion_is_auditable` — every exclusion event carries artifact name,
|
||||
overlap score and the threshold in force.
|
||||
6. `a6_tombstone_removes_manifest_entry` — `mem ref rm`, then assert the R
|
||||
entries are gone from the manifest and the same transcript now ingests
|
||||
normally.
|
||||
7. `a7_verify_catches_leak` — hand-insert an L0 node whose text matches an R
|
||||
artifact; assert `mem verify --derived-filter` exits non-zero and names it.
|
||||
8. `a8_no_retroactive_log_edit` — after `rm`, assert prior `derived_excluded`
|
||||
records are still present and unmodified.
|
||||
|
||||
**Command:** `cargo test -p mem-ingest reference_cycle && cargo test -p mem-cli verify`
|
||||
|
||||
**False pass:**
|
||||
- Testing only the verbatim case. Verbatim is easy and a whole-chunk hash catches
|
||||
it; assertion 3 is the one that distinguishes a working matcher, and assertion
|
||||
2 is the one that proves it is not simply excluding everything that mentions
|
||||
the tool.
|
||||
- Asserting exclusion by checking evidence count is zero. A filter that is
|
||||
accidentally excluding *all* records also yields zero; assertion 2 has to run
|
||||
in the same test binary.
|
||||
- Reusing M4.2's threshold unchanged and declaring it done. The default is tuned
|
||||
for whole-artifact quoting; assertion 3 fails against it, which is the point.
|
||||
|
||||
## Traps
|
||||
|
||||
- Registering R chunks in the manifest before they are committed to the log. A
|
||||
failed ingest then leaves manifest entries that exclude evidence for a corpus
|
||||
that does not exist, and the symptom is missing memories with no obvious cause.
|
||||
- Normalising differently in the two paths. If the shingler treats markdown
|
||||
tables differently at emit-time and at ingest-time, overlap collapses and the
|
||||
filter silently stops firing — same failure M4.2 already warns about, now with
|
||||
two producers to keep in step.
|
||||
- Letting the exclusion event omit the threshold. A tuning change makes every
|
||||
historical exclusion uninterpretable, and this filter will be tuned.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — reference corpora, skills · [M4.2](M4.2-derived-filter.md)
|
||||
Reference in New Issue
Block a user