refactor: Remove retired M3.7.3, M3.7.5 - hybrid search covers

2026-08-28 13:41:47 -07:00
parent 68d544e31e
commit 5c99cf68d1
3 changed files with 2 additions and 279 deletions
@@ -65,13 +65,13 @@ Legend: ⬜ not started · 🟡 in progress · ✅ done · ⛔ blocked
 | 4.5 | Distributed API Layer | M3.5.x | 10 | 10 | 0 | 0 | ✅ M3.5.8 |
 | 5 | Skills | M4.x | 3 | 2 | 0 | 1 | ⬜ M4.3 |
 | 5.5 | Reference corpora | M3.6.x | 7 | 1 | 0 | 6 | ⬜ M3.6.8 |
-| 5.6 | Tool context | M3.7.x | 4 | 2 | 0 | 2 | ⬜ M3.7.6 |
+| 5.6 | Tool context | M3.7.x | 2 | 0 | 0 | 2 | ⬜ M3.7.6 |
 | 5.7 | Context optimization | M3.8.x | 6 | 6 | 0 | 0 | ✅ M3.8.6 |
 | 6 | Post-training | M5.x | 6 | 0 | 0 | 6 | ⬜ M5.6 |
 | 7 | agent-manager migration | M6.x | 6 | 0 | 0 | 6 | ⬜ M6.6 |
 | 8 | Source connectors | M7.x | 10 | 0 | 0 | 10 | ⬜ M7.10 |
 | 9 | Hybrid search | M8.x | 9 | 9 | 0 | 0 | ✅ M8.9 |
-| | **Total** | | **65** | **60** | **0** | **5** | 10/13 green |
+| | **Total** | | **63** | **60** | **0** | **3** | 10/13 green |
 **Current status — 2025-01-28.** Completed phases M0.x, M1.x fully archived (16/16 tasks). **M2.1-6 ✅** (embeddings, CNPG, schema, pgvector, obsidian projector, rebuild). **M3.x ✅** (4/4). **M3.5.x ✅** (10/10 complete + archived). **M3.7.7-8 ✅** (failure diagnosis). **M4.1-2 ✅** (skill drafting + derived filter). **M3.6.1 ✅** (DocCorpusSource). **M3.6.3 ❌ retired** (Obsidian UI replaces CLI). **M3.6.7-8 ⬜ new** (ingest enrichment + deduplication). **M8.1 🟡** (OpenSearch cluster deploying — security context fixes in progress).
@@ -1,115 +0,0 @@
 # M3.7.3 — `GET /memory/skills?task=` — match a subset to the work
 | Field | Value |
 |---|---|
 | Phase | M3.7 — Tool context |
 | Size | M — 1–3 days |
 | Status | ⬜ Not started |
 | Flags | — |
 | Spec | inlined below |
 | Blocks | M3.7.6 |
 | Depends | M3.5.5, M3.2, M2.1 |
 ## Goal
 Given a task, return the few skills that apply, so the orchestrator stops cloning
 the same static list for every piece of work.
 ## Facts (inlined — no spec read needed)
 ```
 GET /memory/skills?task=fix+the+kubectl+parsing+in+the+pod+debugger&project=homelab
 → 200 [{"name":"infra-root-causes","score":0.81,
        "matched_on":"when_to_use","when_to_use":"When troubleshooting cluster…"}]
 ```
 **Match on `description` and `when_to_use`, never on the body.** Anthropic's own
 skill guidance, encoded in the installed `grafana-core:skill-authoring` rubric,
 makes `description` the field that decides whether a skill fires. Bodies are long,
 full of example output, and match everything — a skill whose body mentions
 `kubectl` in passing would be selected for every Kubernetes task. Matching the
 field the author wrote *for this purpose* also gives authors a lever they can
 reason about.
 **Embed the metadata, rerank the shortlist.** Same two-stage shape as `mem query`
 (M3.3): embed `description + when_to_use`, cosine-recall a shortlist, then rerank
 against the task text with `bge-reranker-base` (M3.2). The corpus is small enough
 that recall could be exhaustive, but the reranker is what separates "mentions
 Kubernetes" from "is about diagnosing a failing pod".
 **Empty is a valid answer and must stay cheap.** Most tasks match no skill. The
 endpoint returns `[]`, not the closest thing it found, and the caller proceeds
 with tools and knowledge alone. A floor applies here for the same reason it does
 in M3.6.5: a plausible-but-wrong skill actively steers the implementer.
 **`_drafts/` stays excluded.** M3.5.5's rule is unchanged and load-bearing —
 matching must not become a side channel that loads an unpromoted skill.
 **Deterministic ties.** Two skills at the same score sort by name, so an
 orchestrator that caches on the response is not invalidated by rank flapping
 between identical requests.
 ## Steps
 1. Extend the M3.5.5 handler with `?task=` and `?limit=` (default 3).
 2. Build the match index over `description + when_to_use` for promoted skills.
 3. Recall then rerank against the task text; apply the score floor.
 4. Return `score` and `matched_on` so a bad match is diagnosable without a rerun.
 5. Rebuild the index on skill promotion; no restart required.
 6. `?task=` absent keeps the existing full-catalog behaviour exactly.
 ## Acceptance
 - A Kubernetes debugging task matches the infra skill; an unrelated task does not.
 - Draft skills never appear.
 - No match returns `[]` with 200.
 - Omitting `task` returns the full catalog, byte-identical to today.
 - Equal scores order deterministically.
 ## Verify
 **Harness:** vault fixture with 6 promoted skills across distinct domains plus 2
 drafts. Live reranker for scoring; deterministic embedder elsewhere.
 **Integration test** — `tests/it_skill_matching.rs`:
 1. `a1_relevant_match` — a pod-debugging task returns the infra skill first.
 2. `a2_irrelevant_no_match` — "update the README changelog" returns `[]`.
 3. `a3_drafts_excluded` — a task whose text matches a draft's description
   verbatim returns `[]`.
 4. `a4_body_not_matched` — a skill whose *body* mentions `kubectl` but whose
   description is about something else is not returned for a `kubectl` task.
   This is the assertion that proves the field restriction.
 5. `a5_no_task_unchanged` — omit `task`; assert byte-identical to M3.5.5's
   existing fixture output.
 6. `a6_floor_applies` — a weakly-related task returns `[]` rather than the
   best-of-bad.
 7. `a7_deterministic_ties` — two identically-described skills; assert stable
   name-ordered output across 10 calls.
 8. `a8_reranker_reorders` — capture pre- and post-rerank order; assert they
   differ on at least one fixture task, proving the reranker is wired.
 9. `a9_promotion_visible` — promote a draft, re-query without restart; assert it
   is now matchable.
 **Command:** `cargo test -p mem-api skill_matching`
 **False pass:**
 - Fixtures whose descriptions share no vocabulary. Any embedder separates
  unrelated topics; assertion 4 needs a deliberate body/description conflict, and
  assertion 6 needs a genuinely borderline task, or both pass with a keyword
  `LIKE`.
 - Asserting only that the right skill is *present*. Returning all 6 sorted also
  contains the right one; assert the length and the floor.
 ## Traps
 - Indexing skill bodies "for better recall". It inverts the design: bodies are
  where every skill looks alike, and the author's `description` stops being the
  control surface it was written to be.
 - Tuning the floor against the same fixtures used to assert matching. It converges
  on a threshold that fits six skills and fails on sixty; hold out tasks.
 - Rebuilding the index per request. It is small, but this endpoint sits in the
  path of every task the orchestrator runs.
 ---
 Background: [DESIGN.md](../DESIGN.md) — tool context, skills · [M3.5.5](M3.5.5-skills-endpoint.md)
@@ -1,162 +0,0 @@
 # M3.7.5 — `tool-failures` standing query — the loop that makes it improve
 | Field | Value |
 |---|---|
 | Phase | M3.7 — Tool context |
 | Size | M — 1–3 days |
 | Status | 🟡 In progress — lesson derivation implemented in `lesson.rs` |
 | Flags | — |
 | Spec | inlined below |
 | Blocks | M3.7.6 |
 | Depends | M3.7.4, M1.2, M1.5 |
 ## Goal
 Turn invocations that failed into memory that prevents them, and prove the
 prevention actually reaches the next task’s prompt.
 ## Existing code (already implemented)
 **`crates/mem-core/src/lesson.rs`** already contains:
 | Function | What it does | Tests |
 |---|---|---|
 | `derive_lessons(events, tool_of)` | Pairs fail→success from command events, filters opaque edits, captures resolution | `derives_lesson_from_fail_then_success`, `opaque_edits_do_not_become_a_resolution`, `failed_attempts_are_not_the_resolution`, `bare_retry_is_not_a_lesson` |
 | `tool_of_cmd(cmd)` | Infers tool name from command (npm, cargo, kubectl, etc.) | used by `derive_lessons` |
 **`crates/mem-cli/src/lessons_cmd.rs`** already contains:
 | Command | What it does |
 |---|---|
 | `mem capture --cmd ... --exit ...` | Records command execution events to `~/.mem/events.jsonl` |
 | `mem resolve` | Derives lessons from events, preserves human confirmations |
 ## What remains to complete this task
 The existing code operates on **command execution events** (individual tool invocations with exit codes). This task requires integration with the **GRU-Mem gated loop** (M1.5) which operates on full session transcripts:
 1. **Standing query YAML** — add `tool-failures` query to `queries/<project>.yaml` with the verbatim-invocation requirement
 2. **Gate-based extraction** — the gated loop (M1.5) decides which transcript chunks contain tool-failure evidence, not `derive_lessons()` from command events
 3. **End-to-end test** — ingest a failure session, then verify `/memory/context` returns the failure ranked above docs
 4. **Orchestrator lesson ingestion** — ingest `Poimen/workflows` lesson artifacts as a source
 5. **Derived filter interaction** — verify lessons are NOT caught by M4.2’s derived filter
 The existing `derive_lessons()` remains useful as a **complementary path** for command-level failures, while the standing query handles full-transcript extraction.
 ## Files
 | Action | Path |
 |---|---|
 | **Exists** | `crates/mem-core/src/lesson.rs` — `derive_lessons()`, `tool_of_cmd()` |
 | **Exists** | `crates/mem-cli/src/lessons_cmd.rs` — `mem capture`, `mem resolve` |
 | Modify | `queries/<project>.yaml` — add `tool-failures` standing query |
 | Create | `tests/it_tool_failure_learning.rs` — integration tests (8 assertions) |
 ## Facts (inlined — no spec read needed)
 ```yaml
 # queries/<project>.yaml
  - id: tool-failures
    question: >
      Which tool or command invocations failed, what was the exact error,
      and what was the working alternative? Record the invocation verbatim.
 ```
 This is the only leg of the tool-context bundle that **goes through the update
 gate**, and it should. A failed `kubectl` invocation is genuine evidence about
 what happened in this project — unlike reference text (M3.6), which has no
 evidence to gate on. No bypass, no special casing, no new machinery: one standing
 question whose answers happen to be operationally useful at task time.
 **The gate's discrimination is the feature here.** Sessions are full of commands
 that failed for uninteresting reasons — a typo the model immediately fixed, a
 transient 503. The question asks for the *working alternative*, which is what
 separates a durable lesson from noise, and the gate is what enforces it. If
 update-rate on this query runs high, the question is too permissive, not the gate.
 **Verbatim invocation matters.** "Use the right namespace flag" is unusable. The
 memory has to carry `kubectl get pods --all` → `error: unknown flag: --all` →
 `kubectl get pods --all-namespaces`, because the next model needs the exact
 string to pattern-match against what it was about to emit.
 **This is where the orchestrator's lessons should end up.** `Poimen/workflows`
 already generates lessons on judge rejection (`action/lessons.go`) and discards
 them at task end. Ingesting those artifacts gives this query a dense, pre-filtered
 source — failures already judged consequential by a second model.
 **Success is measured end to end, not at L1.** An L1 memory nobody retrieves is
 worthless. The acceptance criterion is that a task mentioning the tool gets the
 failure in its `/memory/context` bundle, ranked above the cheatsheet.
 ## Steps
 1. Add `tool-failures` to the shipped query templates, with the verbatim
   requirement in the question text.
 2. Ingest orchestrator lesson artifacts as a source alongside session
   transcripts.
 3. Confirm no interaction with M3.6.4's manifest: lessons are project output, not
   emitted artifacts, and must not be excluded as derived.
 4. Measure update-rate for this query separately; it should sit well under the
   30% M1.8 threshold.
 5. End-to-end check: ingest a failure session, then request `/memory/context` for
   a related task and assert the failure is present and ranked above R.
 ## Acceptance
 - A session containing a failure-then-fix yields an L1 memory with both forms
  verbatim.
 - A session with only transient errors yields none.
 - The memory appears in `/memory/context` for a related task, above the docs.
 - Lesson artifacts are not caught by the derived filter.
 - Update-rate for this query stays under the M1.8 threshold.
 ## Verify
 **Harness:** three fixture sessions — one clean failure-then-fix, one transient
 503 with no lesson, one where the model tried three wrong forms before succeeding.
 Live gateway for the gate decisions.
 **Integration test** — `tests/it_tool_failure_learning.rs`:
 1. `a1_failure_becomes_memory` — fixture 1 yields an L1 under `tool-failures`
   containing both the failing and the working invocation, verbatim.
 2. `a2_transient_rejected` — fixture 2 produces no L1. This is the assertion that
   proves the gate is discriminating rather than recording every non-zero exit.
 3. `a3_multi_attempt_keeps_final` — fixture 3's memory names the working form,
   not merely the last error.
 4. `a4_reaches_the_bundle` — after ingest, `GET /memory/context?task=…kubectl…`
   contains the memory.
 5. `a5_outranks_documentation` — in that same bundle, assert it sorts above the
   R cheatsheet section covering the same command.
 6. `a6_lessons_not_derived` — ingest a lesson artifact; assert no
   `derived_excluded` event fires for it.
 7. `a7_update_rate_bounded` — update-rate for this query is under 0.30, reported
   alongside the other standing queries.
 8. `a8_provenance_resolves` — the memory's parents resolve to the L0 span
   containing the actual error text.
 **Command:** `cargo test --workspace tool_failure -- --ignored --nocapture`
 **False pass:**
 - Asserting only `a1`. A gate that accepts every chunk also produces the right
  memory for fixture 1; `a2` is the one that distinguishes a filter from a
  recorder, and it must run in the same binary.
 - Stopping at L1. Assertions 4 and 5 are the task — an L1 that never reaches a
  prompt has changed nothing about how the implementer behaves.
 - Fixtures written by the same model that will be judged on them. Use real
  session transcripts; synthetic failures are unnaturally clean and the gate
  accepts them at a rate real sessions will not reproduce.
 ## Traps
 - Writing the question to ask for "errors". Every tool result containing the word
  error becomes evidence, update-rate climbs, and M1.8 goes red for reasons that
  look unrelated to this task. The working-alternative clause is what bounds it.
 - Ingesting lessons without a project key. They arrive from the orchestrator, not
  from a session with a `cwd`, so project resolution has to be explicit or they
  land in the wrong memory.
 - Treating a high update-rate here as success. It means the question is loose;
  the paper's failure mode is a memory that accepts everything.
 ---
 Background: [DESIGN.md](../DESIGN.md) — tool context, standing queries · [M1.8](M1.8-m1-gate.md)