142 lines
6.3 KiB
Markdown
142 lines
6.3 KiB
Markdown
|
# M3.7.8 — Symptom projection: make an answer findable from an error
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | M3.7 — Tool context |
|
|||
|
|
| Size | M — 1–3 days |
|
|||
|
|
| Status | ⬜ Not started |
|
|||
|
|
| Flags | — |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | M3.7.6 |
|
|||
|
|
| Depends | M3.7.7, M2.4, M1.5 |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Give every memory a second vector describing the failures it would explain, so a
|
|||
|
|
stack trace can find an answer written in prose.
|
|||
|
|
|
|||
|
|
## Facts (inlined — no spec read needed)
|
|||
|
|
|
|||
|
|
The asymmetry this exists to fix:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
L1 memory (how it is written):
|
|||
|
|
"Requests over 10KB failed because Kong buffered the whole body before
|
|||
|
|
proxying; resolved with proxy-body-size: 0 on the ingress."
|
|||
|
|
|
|||
|
|
Query (how it arrives):
|
|||
|
|
"413 Request Entity Too Large" + a curl trace
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Same incident. Embedded with the same model and compared by cosine, they are
|
|||
|
|
mediocre neighbours — one is an explanation, the other is a symptom. This is the
|
|||
|
|
main reason retrieval that looks correct in a unit test disappoints in use.
|
|||
|
|
|
|||
|
|
**Fix at write time, not read time.** When the gated loop accepts a memory,
|
|||
|
|
generate a short *symptom projection* — the errors, messages and observable
|
|||
|
|
behaviour this memory would explain — and embed that as a second vector:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
symptom projection for the memory above:
|
|||
|
|
"413 Request Entity Too Large; large POST bodies rejected at the ingress;
|
|||
|
|
uploads over 10KB fail while small ones succeed; nginx/Kong body buffer limit"
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The alternative, HyDE, generates a hypothetical answer per *query* and puts an
|
|||
|
|
LLM call on every lookup. Writes are rare here — the gate keeps acceptance under
|
|||
|
|
30% by design — and lookups should be fast, so paying once at write is the right
|
|||
|
|
side of that trade.
|
|||
|
|
|
|||
|
|
**It is a projection, so it obeys the projection rules.** Regenerated by
|
|||
|
|
`mem rebuild --from-log`, never authoritative, and byte-identical on replay —
|
|||
|
|
which means the generation call must be deterministic: temperature 0, pinned
|
|||
|
|
prompt, and the model id recorded in the log record so a model change is visible
|
|||
|
|
as a rebuild difference rather than silent drift.
|
|||
|
|
|
|||
|
|
**Only L1 and L2 get one.** L0 is raw evidence already phrased as symptoms; R is
|
|||
|
|
documentation and its headings already read like problems. Generating projections
|
|||
|
|
for those doubles the index for no gain.
|
|||
|
|
|
|||
|
|
**Empty is allowed.** Not every memory explains a failure — an architectural
|
|||
|
|
decision has no symptoms. The controller returns nothing, no vector is written,
|
|||
|
|
and the memory remains findable by its text vector alone. A model that invents
|
|||
|
|
symptoms for a memory that has none pollutes the symptom index with plausible
|
|||
|
|
nonsense, which is worse than a smaller index.
|
|||
|
|
|
|||
|
|
**Signature linking.** Where the L0 evidence behind an accepted memory contains a
|
|||
|
|
parseable failure (M3.7.7), write a `failure_signature` row pointing at the L1.
|
|||
|
|
That is what turns the next occurrence into an exact-match tier-1 hit instead of
|
|||
|
|
a vector search.
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. Prompt template `prompts/symptom.tmpl` — memory text in, symptom lines out,
|
|||
|
|
explicit "return nothing if this describes no failure".
|
|||
|
|
2. Hook into the gated loop after a memory is accepted; L1 and L2 only.
|
|||
|
|
3. Temperature 0; record model id and prompt sha on the log record.
|
|||
|
|
4. Embed and write `memory_vector(kind='symptom')`.
|
|||
|
|
5. Extract signatures from the memory's L0 parents; write `failure_signature`
|
|||
|
|
rows keyed to the memory.
|
|||
|
|
6. `mem rebuild --from-log` regenerates projections; assert stability.
|
|||
|
|
7. `mem symptoms show <sha>` prints the projection for inspection.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
- An accepted L1 that explains a failure gains a symptom vector.
|
|||
|
|
- A memory describing a decision gains none.
|
|||
|
|
- Symptom text retrieves its memory from a raw error string that shares no
|
|||
|
|
vocabulary with the memory text.
|
|||
|
|
- Rebuild regenerates projections byte-identically.
|
|||
|
|
- Signature rows link to the right memory.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Harness:** fixture memories — five that explain failures, three that do not —
|
|||
|
|
plus raw error strings for the five, deliberately worded with no vocabulary
|
|||
|
|
overlap with the memory text.
|
|||
|
|
|
|||
|
|
**Integration test** — `tests/it_symptom_projection.rs`:
|
|||
|
|
1. `a1_projection_generated` — the five failure memories each gain a
|
|||
|
|
`kind='symptom'` vector.
|
|||
|
|
2. `a2_no_projection_for_non_failure` — the three others gain none.
|
|||
|
|
3. `a3_retrieval_by_symptom` — searching `kind='symptom'` with each raw error
|
|||
|
|
string returns its memory first. This is the assertion the task exists for.
|
|||
|
|
4. `a4_beats_text_vector` — the same query against `kind='text'` ranks the
|
|||
|
|
correct memory *lower*; assert the symptom search strictly improves rank.
|
|||
|
|
Without this the projection could be doing nothing.
|
|||
|
|
5. `a5_deterministic` — generate twice; assert byte-identical projections.
|
|||
|
|
6. `a6_rebuild_stable` — drop and rebuild; assert projections and their
|
|||
|
|
embeddings match the originals.
|
|||
|
|
7. `a7_signature_linked` — a memory whose evidence contains a parseable error has
|
|||
|
|
a `failure_signature` row pointing at it, with the tool set.
|
|||
|
|
8. `a8_l0_and_r_skipped` — assert no L0 or R node has a symptom vector.
|
|||
|
|
9. `a9_model_id_recorded` — the log record names the model and prompt sha.
|
|||
|
|
|
|||
|
|
**Command:** `cargo test -p mem-core symptom && cargo test -p mem-cli rebuild`
|
|||
|
|
|
|||
|
|
**False pass:**
|
|||
|
|
- Test queries that reuse the memory's own wording. The text vector already finds
|
|||
|
|
those, assertion 3 passes, and the projection is never exercised. The error
|
|||
|
|
strings must share no meaningful vocabulary — that constraint is the test.
|
|||
|
|
- Asserting 3 without 4. If the text vector already ranked it first, assertion 3
|
|||
|
|
is satisfied by a projection that is empty or useless.
|
|||
|
|
- Skipping determinism because output "looks stable". A default temperature makes
|
|||
|
|
it stable for ten runs and different on the eleventh, and the symptom is a
|
|||
|
|
rebuild diff nobody can explain.
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- Generating projections for rejected chunks. The gate rejected them; embedding
|
|||
|
|
their symptoms puts evidence-free content in the index through a side door.
|
|||
|
|
- Letting the projection restate the memory. If the model paraphrases the answer
|
|||
|
|
instead of naming the symptoms, the second vector duplicates the first and
|
|||
|
|
assertion 4 fails — which is the correct outcome, but the cause is the prompt,
|
|||
|
|
not the plumbing.
|
|||
|
|
- Treating an empty projection as an error and retrying. It is the right answer
|
|||
|
|
for most non-incident memories, and a retry loop turns it into invented
|
|||
|
|
symptoms.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background: [DESIGN.md](../DESIGN.md) — tool context, write path
|