poimen-memory/tasks/M3.7.5-tool-failure-learning.md

# M3.7.5 — `tool-failures` standing query — the loop that makes it improve

| Field | Value |
|---|---|
| Phase | M3.7 — Tool context |
| Size | M — 1–3 days |
| Status | 🟡 In progress — lesson derivation implemented in `lesson.rs` |
| Flags | — |
| Spec | inlined below |
| Blocks | M3.7.6 |
| Depends | M3.7.4, M1.2, M1.5 |

## Goal

Turn invocations that failed into memory that prevents them, and prove the
prevention actually reaches the next task’s prompt.

## Existing code (already implemented)

**`crates/mem-core/src/lesson.rs`** already contains:

| Function | What it does | Tests |
|---|---|---|
| `derive_lessons(events, tool_of)` | Pairs fail→success from command events, filters opaque edits, captures resolution | `derives_lesson_from_fail_then_success`, `opaque_edits_do_not_become_a_resolution`, `failed_attempts_are_not_the_resolution`, `bare_retry_is_not_a_lesson` |
| `tool_of_cmd(cmd)` | Infers tool name from command (npm, cargo, kubectl, etc.) | used by `derive_lessons` |

**`crates/mem-cli/src/lessons_cmd.rs`** already contains:

| Command | What it does |
|---|---|
| `mem capture --cmd ... --exit ...` | Records command execution events to `~/.mem/events.jsonl` |
| `mem resolve` | Derives lessons from events, preserves human confirmations |

## What remains to complete this task

The existing code operates on **command execution events** (individual tool invocations with exit codes). This task requires integration with the **GRU-Mem gated loop** (M1.5) which operates on full session transcripts:

1. **Standing query YAML** — add `tool-failures` query to `queries/<project>.yaml` with the verbatim-invocation requirement
2. **Gate-based extraction** — the gated loop (M1.5) decides which transcript chunks contain tool-failure evidence, not `derive_lessons()` from command events
3. **End-to-end test** — ingest a failure session, then verify `/memory/context` returns the failure ranked above docs
4. **Orchestrator lesson ingestion** — ingest `Poimen/workflows` lesson artifacts as a source
5. **Derived filter interaction** — verify lessons are NOT caught by M4.2’s derived filter

The existing `derive_lessons()` remains useful as a **complementary path** for command-level failures, while the standing query handles full-transcript extraction.

## Files

| Action | Path |
|---|---|
| **Exists** | `crates/mem-core/src/lesson.rs` — `derive_lessons()`, `tool_of_cmd()` |
| **Exists** | `crates/mem-cli/src/lessons_cmd.rs` — `mem capture`, `mem resolve` |
| Modify | `queries/<project>.yaml` — add `tool-failures` standing query |
| Create | `tests/it_tool_failure_learning.rs` — integration tests (8 assertions) |

## Facts (inlined — no spec read needed)

```yaml
# queries/<project>.yaml
  - id: tool-failures
    question: >
      Which tool or command invocations failed, what was the exact error,
      and what was the working alternative? Record the invocation verbatim.
```

This is the only leg of the tool-context bundle that **goes through the update
gate**, and it should. A failed `kubectl` invocation is genuine evidence about
what happened in this project — unlike reference text (M3.6), which has no
evidence to gate on. No bypass, no special casing, no new machinery: one standing
question whose answers happen to be operationally useful at task time.

**The gate's discrimination is the feature here.** Sessions are full of commands
that failed for uninteresting reasons — a typo the model immediately fixed, a
transient 503. The question asks for the *working alternative*, which is what
separates a durable lesson from noise, and the gate is what enforces it. If
update-rate on this query runs high, the question is too permissive, not the gate.

**Verbatim invocation matters.** "Use the right namespace flag" is unusable. The
memory has to carry `kubectl get pods --all` → `error: unknown flag: --all` →
`kubectl get pods --all-namespaces`, because the next model needs the exact
string to pattern-match against what it was about to emit.

**This is where the orchestrator's lessons should end up.** `Poimen/workflows`
already generates lessons on judge rejection (`action/lessons.go`) and discards
them at task end. Ingesting those artifacts gives this query a dense, pre-filtered
source — failures already judged consequential by a second model.

**Success is measured end to end, not at L1.** An L1 memory nobody retrieves is
worthless. The acceptance criterion is that a task mentioning the tool gets the
failure in its `/memory/context` bundle, ranked above the cheatsheet.

## Steps

1. Add `tool-failures` to the shipped query templates, with the verbatim
   requirement in the question text.
2. Ingest orchestrator lesson artifacts as a source alongside session
   transcripts.
3. Confirm no interaction with M3.6.4's manifest: lessons are project output, not
   emitted artifacts, and must not be excluded as derived.
4. Measure update-rate for this query separately; it should sit well under the
   30% M1.8 threshold.
5. End-to-end check: ingest a failure session, then request `/memory/context` for
   a related task and assert the failure is present and ranked above R.

## Acceptance

- A session containing a failure-then-fix yields an L1 memory with both forms
  verbatim.
- A session with only transient errors yields none.
- The memory appears in `/memory/context` for a related task, above the docs.
- Lesson artifacts are not caught by the derived filter.
- Update-rate for this query stays under the M1.8 threshold.

## Verify

**Harness:** three fixture sessions — one clean failure-then-fix, one transient
503 with no lesson, one where the model tried three wrong forms before succeeding.
Live gateway for the gate decisions.

**Integration test** — `tests/it_tool_failure_learning.rs`:
1. `a1_failure_becomes_memory` — fixture 1 yields an L1 under `tool-failures`
   containing both the failing and the working invocation, verbatim.
2. `a2_transient_rejected` — fixture 2 produces no L1. This is the assertion that
   proves the gate is discriminating rather than recording every non-zero exit.
3. `a3_multi_attempt_keeps_final` — fixture 3's memory names the working form,
   not merely the last error.
4. `a4_reaches_the_bundle` — after ingest, `GET /memory/context?task=…kubectl…`
   contains the memory.
5. `a5_outranks_documentation` — in that same bundle, assert it sorts above the
   R cheatsheet section covering the same command.
6. `a6_lessons_not_derived` — ingest a lesson artifact; assert no
   `derived_excluded` event fires for it.
7. `a7_update_rate_bounded` — update-rate for this query is under 0.30, reported
   alongside the other standing queries.
8. `a8_provenance_resolves` — the memory's parents resolve to the L0 span
   containing the actual error text.

**Command:** `cargo test --workspace tool_failure -- --ignored --nocapture`

**False pass:**
- Asserting only `a1`. A gate that accepts every chunk also produces the right
  memory for fixture 1; `a2` is the one that distinguishes a filter from a
  recorder, and it must run in the same binary.
- Stopping at L1. Assertions 4 and 5 are the task — an L1 that never reaches a
  prompt has changed nothing about how the implementer behaves.
- Fixtures written by the same model that will be judged on them. Use real
  session transcripts; synthetic failures are unnaturally clean and the gate
  accepts them at a rate real sessions will not reproduce.

## Traps

- Writing the question to ask for "errors". Every tool result containing the word
  error becomes evidence, update-rate climbs, and M1.8 goes red for reasons that
  look unrelated to this task. The working-alternative clause is what bounds it.
- Ingesting lessons without a project key. They arrive from the orchestrator, not
  from a session with a `cwd`, so project resolution has to be explicit or they
  land in the wrong memory.
- Treating a high update-rate here as success. It means the question is loose;
  the paper's failure mode is a memory that accepts everything.

---

Background: [DESIGN.md](../DESIGN.md) — tool context, standing queries · [M1.8](M1.8-m1-gate.md)