From 5c99cf68d1b75f18250c666afd4c908b3053c162 Mon Sep 17 00:00:00 2001 From: poimen Date: Fri, 28 Aug 2026 13:41:47 -0700 Subject: [PATCH] refactor: Remove retired M3.7.3, M3.7.5 - hybrid search covers --- tasks/INDEX.md | 4 +- tasks/M3.7.3-skill-matching.md | 115 ------------------ tasks/M3.7.5-tool-failure-learning.md | 162 -------------------------- 3 files changed, 2 insertions(+), 279 deletions(-) delete mode 100644 tasks/M3.7.3-skill-matching.md delete mode 100644 tasks/M3.7.5-tool-failure-learning.md diff --git a/tasks/INDEX.md b/tasks/INDEX.md index 21605e3..6f663c8 100644 --- a/tasks/INDEX.md +++ b/tasks/INDEX.md @@ -65,13 +65,13 @@ Legend: ⬜ not started Β· 🟑 in progress Β· βœ… done Β· β›” blocked | 4.5 | Distributed API Layer | M3.5.x | 10 | 10 | 0 | 0 | βœ… M3.5.8 | | 5 | Skills | M4.x | 3 | 2 | 0 | 1 | ⬜ M4.3 | | 5.5 | Reference corpora | M3.6.x | 7 | 1 | 0 | 6 | ⬜ M3.6.8 | -| 5.6 | Tool context | M3.7.x | 4 | 2 | 0 | 2 | ⬜ M3.7.6 | +| 5.6 | Tool context | M3.7.x | 2 | 0 | 0 | 2 | ⬜ M3.7.6 | | 5.7 | Context optimization | M3.8.x | 6 | 6 | 0 | 0 | βœ… M3.8.6 | | 6 | Post-training | M5.x | 6 | 0 | 0 | 6 | ⬜ M5.6 | | 7 | agent-manager migration | M6.x | 6 | 0 | 0 | 6 | ⬜ M6.6 | | 8 | Source connectors | M7.x | 10 | 0 | 0 | 10 | ⬜ M7.10 | | 9 | Hybrid search | M8.x | 9 | 9 | 0 | 0 | βœ… M8.9 | -| | **Total** | | **65** | **60** | **0** | **5** | 10/13 green | +| | **Total** | | **63** | **60** | **0** | **3** | 10/13 green | **Current status β€” 2025-01-28.** Completed phases M0.x, M1.x fully archived (16/16 tasks). **M2.1-6 βœ…** (embeddings, CNPG, schema, pgvector, obsidian projector, rebuild). **M3.x βœ…** (4/4). **M3.5.x βœ…** (10/10 complete + archived). **M3.7.7-8 βœ…** (failure diagnosis). **M4.1-2 βœ…** (skill drafting + derived filter). **M3.6.1 βœ…** (DocCorpusSource). **M3.6.3 ❌ retired** (Obsidian UI replaces CLI). **M3.6.7-8 ⬜ new** (ingest enrichment + deduplication). **M8.1 🟑** (OpenSearch cluster deploying β€” security context fixes in progress). diff --git a/tasks/M3.7.3-skill-matching.md b/tasks/M3.7.3-skill-matching.md deleted file mode 100644 index 2466d10..0000000 --- a/tasks/M3.7.3-skill-matching.md +++ /dev/null @@ -1,115 +0,0 @@ -# M3.7.3 β€” `GET /memory/skills?task=` β€” match a subset to the work - -| Field | Value | -|---|---| -| Phase | M3.7 β€” Tool context | -| Size | M β€” 1–3 days | -| Status | ⬜ Not started | -| Flags | β€” | -| Spec | inlined below | -| Blocks | M3.7.6 | -| Depends | M3.5.5, M3.2, M2.1 | - -## Goal - -Given a task, return the few skills that apply, so the orchestrator stops cloning -the same static list for every piece of work. - -## Facts (inlined β€” no spec read needed) - -``` -GET /memory/skills?task=fix+the+kubectl+parsing+in+the+pod+debugger&project=homelab -β†’ 200 [{"name":"infra-root-causes","score":0.81, - "matched_on":"when_to_use","when_to_use":"When troubleshooting cluster…"}] -``` - -**Match on `description` and `when_to_use`, never on the body.** Anthropic's own -skill guidance, encoded in the installed `grafana-core:skill-authoring` rubric, -makes `description` the field that decides whether a skill fires. Bodies are long, -full of example output, and match everything β€” a skill whose body mentions -`kubectl` in passing would be selected for every Kubernetes task. Matching the -field the author wrote *for this purpose* also gives authors a lever they can -reason about. - -**Embed the metadata, rerank the shortlist.** Same two-stage shape as `mem query` -(M3.3): embed `description + when_to_use`, cosine-recall a shortlist, then rerank -against the task text with `bge-reranker-base` (M3.2). The corpus is small enough -that recall could be exhaustive, but the reranker is what separates "mentions -Kubernetes" from "is about diagnosing a failing pod". - -**Empty is a valid answer and must stay cheap.** Most tasks match no skill. The -endpoint returns `[]`, not the closest thing it found, and the caller proceeds -with tools and knowledge alone. A floor applies here for the same reason it does -in M3.6.5: a plausible-but-wrong skill actively steers the implementer. - -**`_drafts/` stays excluded.** M3.5.5's rule is unchanged and load-bearing β€” -matching must not become a side channel that loads an unpromoted skill. - -**Deterministic ties.** Two skills at the same score sort by name, so an -orchestrator that caches on the response is not invalidated by rank flapping -between identical requests. - -## Steps - -1. Extend the M3.5.5 handler with `?task=` and `?limit=` (default 3). -2. Build the match index over `description + when_to_use` for promoted skills. -3. Recall then rerank against the task text; apply the score floor. -4. Return `score` and `matched_on` so a bad match is diagnosable without a rerun. -5. Rebuild the index on skill promotion; no restart required. -6. `?task=` absent keeps the existing full-catalog behaviour exactly. - -## Acceptance - -- A Kubernetes debugging task matches the infra skill; an unrelated task does not. -- Draft skills never appear. -- No match returns `[]` with 200. -- Omitting `task` returns the full catalog, byte-identical to today. -- Equal scores order deterministically. - -## Verify - -**Harness:** vault fixture with 6 promoted skills across distinct domains plus 2 -drafts. Live reranker for scoring; deterministic embedder elsewhere. - -**Integration test** β€” `tests/it_skill_matching.rs`: -1. `a1_relevant_match` β€” a pod-debugging task returns the infra skill first. -2. `a2_irrelevant_no_match` β€” "update the README changelog" returns `[]`. -3. `a3_drafts_excluded` β€” a task whose text matches a draft's description - verbatim returns `[]`. -4. `a4_body_not_matched` β€” a skill whose *body* mentions `kubectl` but whose - description is about something else is not returned for a `kubectl` task. - This is the assertion that proves the field restriction. -5. `a5_no_task_unchanged` β€” omit `task`; assert byte-identical to M3.5.5's - existing fixture output. -6. `a6_floor_applies` β€” a weakly-related task returns `[]` rather than the - best-of-bad. -7. `a7_deterministic_ties` β€” two identically-described skills; assert stable - name-ordered output across 10 calls. -8. `a8_reranker_reorders` β€” capture pre- and post-rerank order; assert they - differ on at least one fixture task, proving the reranker is wired. -9. `a9_promotion_visible` β€” promote a draft, re-query without restart; assert it - is now matchable. - -**Command:** `cargo test -p mem-api skill_matching` - -**False pass:** -- Fixtures whose descriptions share no vocabulary. Any embedder separates - unrelated topics; assertion 4 needs a deliberate body/description conflict, and - assertion 6 needs a genuinely borderline task, or both pass with a keyword - `LIKE`. -- Asserting only that the right skill is *present*. Returning all 6 sorted also - contains the right one; assert the length and the floor. - -## Traps - -- Indexing skill bodies "for better recall". It inverts the design: bodies are - where every skill looks alike, and the author's `description` stops being the - control surface it was written to be. -- Tuning the floor against the same fixtures used to assert matching. It converges - on a threshold that fits six skills and fails on sixty; hold out tasks. -- Rebuilding the index per request. It is small, but this endpoint sits in the - path of every task the orchestrator runs. - ---- - -Background: [DESIGN.md](../DESIGN.md) β€” tool context, skills Β· [M3.5.5](M3.5.5-skills-endpoint.md) diff --git a/tasks/M3.7.5-tool-failure-learning.md b/tasks/M3.7.5-tool-failure-learning.md deleted file mode 100644 index e2b5b52..0000000 --- a/tasks/M3.7.5-tool-failure-learning.md +++ /dev/null @@ -1,162 +0,0 @@ -# M3.7.5 β€” `tool-failures` standing query β€” the loop that makes it improve - -| Field | Value | -|---|---| -| Phase | M3.7 β€” Tool context | -| Size | M β€” 1–3 days | -| Status | 🟑 In progress β€” lesson derivation implemented in `lesson.rs` | -| Flags | β€” | -| Spec | inlined below | -| Blocks | M3.7.6 | -| Depends | M3.7.4, M1.2, M1.5 | - -## Goal - -Turn invocations that failed into memory that prevents them, and prove the -prevention actually reaches the next task’s prompt. - -## Existing code (already implemented) - -**`crates/mem-core/src/lesson.rs`** already contains: - -| Function | What it does | Tests | -|---|---|---| -| `derive_lessons(events, tool_of)` | Pairs failβ†’success from command events, filters opaque edits, captures resolution | `derives_lesson_from_fail_then_success`, `opaque_edits_do_not_become_a_resolution`, `failed_attempts_are_not_the_resolution`, `bare_retry_is_not_a_lesson` | -| `tool_of_cmd(cmd)` | Infers tool name from command (npm, cargo, kubectl, etc.) | used by `derive_lessons` | - -**`crates/mem-cli/src/lessons_cmd.rs`** already contains: - -| Command | What it does | -|---|---| -| `mem capture --cmd ... --exit ...` | Records command execution events to `~/.mem/events.jsonl` | -| `mem resolve` | Derives lessons from events, preserves human confirmations | - -## What remains to complete this task - -The existing code operates on **command execution events** (individual tool invocations with exit codes). This task requires integration with the **GRU-Mem gated loop** (M1.5) which operates on full session transcripts: - -1. **Standing query YAML** β€” add `tool-failures` query to `queries/.yaml` with the verbatim-invocation requirement -2. **Gate-based extraction** β€” the gated loop (M1.5) decides which transcript chunks contain tool-failure evidence, not `derive_lessons()` from command events -3. **End-to-end test** β€” ingest a failure session, then verify `/memory/context` returns the failure ranked above docs -4. **Orchestrator lesson ingestion** β€” ingest `Poimen/workflows` lesson artifacts as a source -5. **Derived filter interaction** β€” verify lessons are NOT caught by M4.2’s derived filter - -The existing `derive_lessons()` remains useful as a **complementary path** for command-level failures, while the standing query handles full-transcript extraction. - -## Files - -| Action | Path | -|---|---| -| **Exists** | `crates/mem-core/src/lesson.rs` β€” `derive_lessons()`, `tool_of_cmd()` | -| **Exists** | `crates/mem-cli/src/lessons_cmd.rs` β€” `mem capture`, `mem resolve` | -| Modify | `queries/.yaml` β€” add `tool-failures` standing query | -| Create | `tests/it_tool_failure_learning.rs` β€” integration tests (8 assertions) | - -## Facts (inlined β€” no spec read needed) - -```yaml -# queries/.yaml - - id: tool-failures - question: > - Which tool or command invocations failed, what was the exact error, - and what was the working alternative? Record the invocation verbatim. -``` - -This is the only leg of the tool-context bundle that **goes through the update -gate**, and it should. A failed `kubectl` invocation is genuine evidence about -what happened in this project β€” unlike reference text (M3.6), which has no -evidence to gate on. No bypass, no special casing, no new machinery: one standing -question whose answers happen to be operationally useful at task time. - -**The gate's discrimination is the feature here.** Sessions are full of commands -that failed for uninteresting reasons β€” a typo the model immediately fixed, a -transient 503. The question asks for the *working alternative*, which is what -separates a durable lesson from noise, and the gate is what enforces it. If -update-rate on this query runs high, the question is too permissive, not the gate. - -**Verbatim invocation matters.** "Use the right namespace flag" is unusable. The -memory has to carry `kubectl get pods --all` β†’ `error: unknown flag: --all` β†’ -`kubectl get pods --all-namespaces`, because the next model needs the exact -string to pattern-match against what it was about to emit. - -**This is where the orchestrator's lessons should end up.** `Poimen/workflows` -already generates lessons on judge rejection (`action/lessons.go`) and discards -them at task end. Ingesting those artifacts gives this query a dense, pre-filtered -source β€” failures already judged consequential by a second model. - -**Success is measured end to end, not at L1.** An L1 memory nobody retrieves is -worthless. The acceptance criterion is that a task mentioning the tool gets the -failure in its `/memory/context` bundle, ranked above the cheatsheet. - -## Steps - -1. Add `tool-failures` to the shipped query templates, with the verbatim - requirement in the question text. -2. Ingest orchestrator lesson artifacts as a source alongside session - transcripts. -3. Confirm no interaction with M3.6.4's manifest: lessons are project output, not - emitted artifacts, and must not be excluded as derived. -4. Measure update-rate for this query separately; it should sit well under the - 30% M1.8 threshold. -5. End-to-end check: ingest a failure session, then request `/memory/context` for - a related task and assert the failure is present and ranked above R. - -## Acceptance - -- A session containing a failure-then-fix yields an L1 memory with both forms - verbatim. -- A session with only transient errors yields none. -- The memory appears in `/memory/context` for a related task, above the docs. -- Lesson artifacts are not caught by the derived filter. -- Update-rate for this query stays under the M1.8 threshold. - -## Verify - -**Harness:** three fixture sessions β€” one clean failure-then-fix, one transient -503 with no lesson, one where the model tried three wrong forms before succeeding. -Live gateway for the gate decisions. - -**Integration test** β€” `tests/it_tool_failure_learning.rs`: -1. `a1_failure_becomes_memory` β€” fixture 1 yields an L1 under `tool-failures` - containing both the failing and the working invocation, verbatim. -2. `a2_transient_rejected` β€” fixture 2 produces no L1. This is the assertion that - proves the gate is discriminating rather than recording every non-zero exit. -3. `a3_multi_attempt_keeps_final` β€” fixture 3's memory names the working form, - not merely the last error. -4. `a4_reaches_the_bundle` β€” after ingest, `GET /memory/context?task=…kubectl…` - contains the memory. -5. `a5_outranks_documentation` β€” in that same bundle, assert it sorts above the - R cheatsheet section covering the same command. -6. `a6_lessons_not_derived` β€” ingest a lesson artifact; assert no - `derived_excluded` event fires for it. -7. `a7_update_rate_bounded` β€” update-rate for this query is under 0.30, reported - alongside the other standing queries. -8. `a8_provenance_resolves` β€” the memory's parents resolve to the L0 span - containing the actual error text. - -**Command:** `cargo test --workspace tool_failure -- --ignored --nocapture` - -**False pass:** -- Asserting only `a1`. A gate that accepts every chunk also produces the right - memory for fixture 1; `a2` is the one that distinguishes a filter from a - recorder, and it must run in the same binary. -- Stopping at L1. Assertions 4 and 5 are the task β€” an L1 that never reaches a - prompt has changed nothing about how the implementer behaves. -- Fixtures written by the same model that will be judged on them. Use real - session transcripts; synthetic failures are unnaturally clean and the gate - accepts them at a rate real sessions will not reproduce. - -## Traps - -- Writing the question to ask for "errors". Every tool result containing the word - error becomes evidence, update-rate climbs, and M1.8 goes red for reasons that - look unrelated to this task. The working-alternative clause is what bounds it. -- Ingesting lessons without a project key. They arrive from the orchestrator, not - from a session with a `cwd`, so project resolution has to be explicit or they - land in the wrong memory. -- Treating a high update-rate here as success. It means the question is loose; - the paper's failure mode is a memory that accepts everything. - ---- - -Background: [DESIGN.md](../DESIGN.md) β€” tool context, standing queries Β· [M1.8](M1.8-m1-gate.md)