poimen-memory/tasks/M3.7.5-tool-failure-learning.md

M3.7.5 — tool-failures standing query — the loop that makes it improve

Field	Value
Phase	M3.7 — Tool context
Size	M — 1–3 days
Status	🟡 In progress — lesson derivation implemented in lesson.rs
Flags	—
Spec	inlined below
Blocks	M3.7.6
Depends	M3.7.4, M1.2, M1.5

Goal

Turn invocations that failed into memory that prevents them, and prove the prevention actually reaches the next task’s prompt.

Existing code (already implemented)

crates/mem-core/src/lesson.rs already contains:

Function	What it does	Tests
derive_lessons(events, tool_of)	Pairs fail→success from command events, filters opaque edits, captures resolution	derives_lesson_from_fail_then_success, opaque_edits_do_not_become_a_resolution, failed_attempts_are_not_the_resolution, bare_retry_is_not_a_lesson
tool_of_cmd(cmd)	Infers tool name from command (npm, cargo, kubectl, etc.)	used by derive_lessons

crates/mem-cli/src/lessons_cmd.rs already contains:

Command	What it does
mem capture --cmd ... --exit ...	Records command execution events to ~/.mem/events.jsonl
mem resolve	Derives lessons from events, preserves human confirmations

What remains to complete this task

The existing code operates on command execution events (individual tool invocations with exit codes). This task requires integration with the GRU-Mem gated loop (M1.5) which operates on full session transcripts:

1. Standing query YAML — add tool-failures query to queries/<project>.yaml with the verbatim-invocation requirement
2. Gate-based extraction — the gated loop (M1.5) decides which transcript chunks contain tool-failure evidence, not derive_lessons() from command events
3. End-to-end test — ingest a failure session, then verify /memory/context returns the failure ranked above docs
4. Orchestrator lesson ingestion — ingest Poimen/workflows lesson artifacts as a source
5. Derived filter interaction — verify lessons are NOT caught by M4.2’s derived filter

The existing derive_lessons() remains useful as a complementary path for command-level failures, while the standing query handles full-transcript extraction.

Files

Action	Path
Exists	crates/mem-core/src/lesson.rs — derive_lessons(), tool_of_cmd()
Exists	crates/mem-cli/src/lessons_cmd.rs — mem capture, mem resolve
Modify	queries/<project>.yaml — add tool-failures standing query
Create	tests/it_tool_failure_learning.rs — integration tests (8 assertions)

Facts (inlined — no spec read needed)

# queries/<project>.yaml
  - id: tool-failures
    question: >
      Which tool or command invocations failed, what was the exact error,
      and what was the working alternative? Record the invocation verbatim.

This is the only leg of the tool-context bundle that goes through the update gate, and it should. A failed kubectl invocation is genuine evidence about what happened in this project — unlike reference text (M3.6), which has no evidence to gate on. No bypass, no special casing, no new machinery: one standing question whose answers happen to be operationally useful at task time.

The gate's discrimination is the feature here. Sessions are full of commands that failed for uninteresting reasons — a typo the model immediately fixed, a transient 503. The question asks for the working alternative, which is what separates a durable lesson from noise, and the gate is what enforces it. If update-rate on this query runs high, the question is too permissive, not the gate.

Verbatim invocation matters. "Use the right namespace flag" is unusable. The memory has to carry kubectl get pods --all → error: unknown flag: --all → kubectl get pods --all-namespaces, because the next model needs the exact string to pattern-match against what it was about to emit.

This is where the orchestrator's lessons should end up. Poimen/workflows already generates lessons on judge rejection (action/lessons.go) and discards them at task end. Ingesting those artifacts gives this query a dense, pre-filtered source — failures already judged consequential by a second model.

Success is measured end to end, not at L1. An L1 memory nobody retrieves is worthless. The acceptance criterion is that a task mentioning the tool gets the failure in its /memory/context bundle, ranked above the cheatsheet.

Steps

1. Add tool-failures to the shipped query templates, with the verbatim requirement in the question text.
2. Ingest orchestrator lesson artifacts as a source alongside session transcripts.
3. Confirm no interaction with M3.6.4's manifest: lessons are project output, not emitted artifacts, and must not be excluded as derived.
4. Measure update-rate for this query separately; it should sit well under the 30% M1.8 threshold.
5. End-to-end check: ingest a failure session, then request /memory/context for a related task and assert the failure is present and ranked above R.

Acceptance

• A session containing a failure-then-fix yields an L1 memory with both forms verbatim.
• A session with only transient errors yields none.
• The memory appears in /memory/context for a related task, above the docs.
• Lesson artifacts are not caught by the derived filter.
• Update-rate for this query stays under the M1.8 threshold.

Verify

Harness: three fixture sessions — one clean failure-then-fix, one transient 503 with no lesson, one where the model tried three wrong forms before succeeding. Live gateway for the gate decisions.

Integration test — tests/it_tool_failure_learning.rs:

1. a1_failure_becomes_memory — fixture 1 yields an L1 under tool-failures containing both the failing and the working invocation, verbatim.
2. a2_transient_rejected — fixture 2 produces no L1. This is the assertion that proves the gate is discriminating rather than recording every non-zero exit.
3. a3_multi_attempt_keeps_final — fixture 3's memory names the working form, not merely the last error.
4. a4_reaches_the_bundle — after ingest, GET /memory/context?task=…kubectl… contains the memory.
5. a5_outranks_documentation — in that same bundle, assert it sorts above the R cheatsheet section covering the same command.
6. a6_lessons_not_derived — ingest a lesson artifact; assert no derived_excluded event fires for it.
7. a7_update_rate_bounded — update-rate for this query is under 0.30, reported alongside the other standing queries.
8. a8_provenance_resolves — the memory's parents resolve to the L0 span containing the actual error text.

Command: cargo test --workspace tool_failure -- --ignored --nocapture

False pass:

• Asserting only a1. A gate that accepts every chunk also produces the right memory for fixture 1; a2 is the one that distinguishes a filter from a recorder, and it must run in the same binary.
• Stopping at L1. Assertions 4 and 5 are the task — an L1 that never reaches a prompt has changed nothing about how the implementer behaves.
• Fixtures written by the same model that will be judged on them. Use real session transcripts; synthetic failures are unnaturally clean and the gate accepts them at a rate real sessions will not reproduce.

Traps

• Writing the question to ask for "errors". Every tool result containing the word error becomes evidence, update-rate climbs, and M1.8 goes red for reasons that look unrelated to this task. The working-alternative clause is what bounds it.
• Ingesting lessons without a project key. They arrive from the orchestrator, not from a session with a cwd, so project resolution has to be explicit or they land in the wrong memory.
• Treating a high update-rate here as success. It means the question is loose; the paper's failure mode is a memory that accepts everything.

---

Background: DESIGN.md — tool context, standing queries · M1.8