refactor: Remove retired M3.7.3, M3.7.5 - hybrid search covers
This commit is contained in:
+2
-2
@@ -65,13 +65,13 @@ Legend: ⬜ not started · 🟡 in progress · ✅ done · ⛔ blocked
|
||||
| 4.5 | Distributed API Layer | M3.5.x | 10 | 10 | 0 | 0 | ✅ M3.5.8 |
|
||||
| 5 | Skills | M4.x | 3 | 2 | 0 | 1 | ⬜ M4.3 |
|
||||
| 5.5 | Reference corpora | M3.6.x | 7 | 1 | 0 | 6 | ⬜ M3.6.8 |
|
||||
| 5.6 | Tool context | M3.7.x | 4 | 2 | 0 | 2 | ⬜ M3.7.6 |
|
||||
| 5.6 | Tool context | M3.7.x | 2 | 0 | 0 | 2 | ⬜ M3.7.6 |
|
||||
| 5.7 | Context optimization | M3.8.x | 6 | 6 | 0 | 0 | ✅ M3.8.6 |
|
||||
| 6 | Post-training | M5.x | 6 | 0 | 0 | 6 | ⬜ M5.6 |
|
||||
| 7 | agent-manager migration | M6.x | 6 | 0 | 0 | 6 | ⬜ M6.6 |
|
||||
| 8 | Source connectors | M7.x | 10 | 0 | 0 | 10 | ⬜ M7.10 |
|
||||
| 9 | Hybrid search | M8.x | 9 | 9 | 0 | 0 | ✅ M8.9 |
|
||||
| | **Total** | | **65** | **60** | **0** | **5** | 10/13 green |
|
||||
| | **Total** | | **63** | **60** | **0** | **3** | 10/13 green |
|
||||
|
||||
**Current status — 2025-01-28.** Completed phases M0.x, M1.x fully archived (16/16 tasks). **M2.1-6 ✅** (embeddings, CNPG, schema, pgvector, obsidian projector, rebuild). **M3.x ✅** (4/4). **M3.5.x ✅** (10/10 complete + archived). **M3.7.7-8 ✅** (failure diagnosis). **M4.1-2 ✅** (skill drafting + derived filter). **M3.6.1 ✅** (DocCorpusSource). **M3.6.3 ❌ retired** (Obsidian UI replaces CLI). **M3.6.7-8 ⬜ new** (ingest enrichment + deduplication). **M8.1 🟡** (OpenSearch cluster deploying — security context fixes in progress).
|
||||
|
||||
|
||||
@@ -1,115 +0,0 @@
|
||||
# M3.7.3 — `GET /memory/skills?task=` — match a subset to the work
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M3.7 — Tool context |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M3.7.6 |
|
||||
| Depends | M3.5.5, M3.2, M2.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Given a task, return the few skills that apply, so the orchestrator stops cloning
|
||||
the same static list for every piece of work.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
```
|
||||
GET /memory/skills?task=fix+the+kubectl+parsing+in+the+pod+debugger&project=homelab
|
||||
→ 200 [{"name":"infra-root-causes","score":0.81,
|
||||
"matched_on":"when_to_use","when_to_use":"When troubleshooting cluster…"}]
|
||||
```
|
||||
|
||||
**Match on `description` and `when_to_use`, never on the body.** Anthropic's own
|
||||
skill guidance, encoded in the installed `grafana-core:skill-authoring` rubric,
|
||||
makes `description` the field that decides whether a skill fires. Bodies are long,
|
||||
full of example output, and match everything — a skill whose body mentions
|
||||
`kubectl` in passing would be selected for every Kubernetes task. Matching the
|
||||
field the author wrote *for this purpose* also gives authors a lever they can
|
||||
reason about.
|
||||
|
||||
**Embed the metadata, rerank the shortlist.** Same two-stage shape as `mem query`
|
||||
(M3.3): embed `description + when_to_use`, cosine-recall a shortlist, then rerank
|
||||
against the task text with `bge-reranker-base` (M3.2). The corpus is small enough
|
||||
that recall could be exhaustive, but the reranker is what separates "mentions
|
||||
Kubernetes" from "is about diagnosing a failing pod".
|
||||
|
||||
**Empty is a valid answer and must stay cheap.** Most tasks match no skill. The
|
||||
endpoint returns `[]`, not the closest thing it found, and the caller proceeds
|
||||
with tools and knowledge alone. A floor applies here for the same reason it does
|
||||
in M3.6.5: a plausible-but-wrong skill actively steers the implementer.
|
||||
|
||||
**`_drafts/` stays excluded.** M3.5.5's rule is unchanged and load-bearing —
|
||||
matching must not become a side channel that loads an unpromoted skill.
|
||||
|
||||
**Deterministic ties.** Two skills at the same score sort by name, so an
|
||||
orchestrator that caches on the response is not invalidated by rank flapping
|
||||
between identical requests.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Extend the M3.5.5 handler with `?task=` and `?limit=` (default 3).
|
||||
2. Build the match index over `description + when_to_use` for promoted skills.
|
||||
3. Recall then rerank against the task text; apply the score floor.
|
||||
4. Return `score` and `matched_on` so a bad match is diagnosable without a rerun.
|
||||
5. Rebuild the index on skill promotion; no restart required.
|
||||
6. `?task=` absent keeps the existing full-catalog behaviour exactly.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- A Kubernetes debugging task matches the infra skill; an unrelated task does not.
|
||||
- Draft skills never appear.
|
||||
- No match returns `[]` with 200.
|
||||
- Omitting `task` returns the full catalog, byte-identical to today.
|
||||
- Equal scores order deterministically.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** vault fixture with 6 promoted skills across distinct domains plus 2
|
||||
drafts. Live reranker for scoring; deterministic embedder elsewhere.
|
||||
|
||||
**Integration test** — `tests/it_skill_matching.rs`:
|
||||
1. `a1_relevant_match` — a pod-debugging task returns the infra skill first.
|
||||
2. `a2_irrelevant_no_match` — "update the README changelog" returns `[]`.
|
||||
3. `a3_drafts_excluded` — a task whose text matches a draft's description
|
||||
verbatim returns `[]`.
|
||||
4. `a4_body_not_matched` — a skill whose *body* mentions `kubectl` but whose
|
||||
description is about something else is not returned for a `kubectl` task.
|
||||
This is the assertion that proves the field restriction.
|
||||
5. `a5_no_task_unchanged` — omit `task`; assert byte-identical to M3.5.5's
|
||||
existing fixture output.
|
||||
6. `a6_floor_applies` — a weakly-related task returns `[]` rather than the
|
||||
best-of-bad.
|
||||
7. `a7_deterministic_ties` — two identically-described skills; assert stable
|
||||
name-ordered output across 10 calls.
|
||||
8. `a8_reranker_reorders` — capture pre- and post-rerank order; assert they
|
||||
differ on at least one fixture task, proving the reranker is wired.
|
||||
9. `a9_promotion_visible` — promote a draft, re-query without restart; assert it
|
||||
is now matchable.
|
||||
|
||||
**Command:** `cargo test -p mem-api skill_matching`
|
||||
|
||||
**False pass:**
|
||||
- Fixtures whose descriptions share no vocabulary. Any embedder separates
|
||||
unrelated topics; assertion 4 needs a deliberate body/description conflict, and
|
||||
assertion 6 needs a genuinely borderline task, or both pass with a keyword
|
||||
`LIKE`.
|
||||
- Asserting only that the right skill is *present*. Returning all 6 sorted also
|
||||
contains the right one; assert the length and the floor.
|
||||
|
||||
## Traps
|
||||
|
||||
- Indexing skill bodies "for better recall". It inverts the design: bodies are
|
||||
where every skill looks alike, and the author's `description` stops being the
|
||||
control surface it was written to be.
|
||||
- Tuning the floor against the same fixtures used to assert matching. It converges
|
||||
on a threshold that fits six skills and fails on sixty; hold out tasks.
|
||||
- Rebuilding the index per request. It is small, but this endpoint sits in the
|
||||
path of every task the orchestrator runs.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — tool context, skills · [M3.5.5](M3.5.5-skills-endpoint.md)
|
||||
@@ -1,162 +0,0 @@
|
||||
# M3.7.5 — `tool-failures` standing query — the loop that makes it improve
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M3.7 — Tool context |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | 🟡 In progress — lesson derivation implemented in `lesson.rs` |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M3.7.6 |
|
||||
| Depends | M3.7.4, M1.2, M1.5 |
|
||||
|
||||
## Goal
|
||||
|
||||
Turn invocations that failed into memory that prevents them, and prove the
|
||||
prevention actually reaches the next task’s prompt.
|
||||
|
||||
## Existing code (already implemented)
|
||||
|
||||
**`crates/mem-core/src/lesson.rs`** already contains:
|
||||
|
||||
| Function | What it does | Tests |
|
||||
|---|---|---|
|
||||
| `derive_lessons(events, tool_of)` | Pairs fail→success from command events, filters opaque edits, captures resolution | `derives_lesson_from_fail_then_success`, `opaque_edits_do_not_become_a_resolution`, `failed_attempts_are_not_the_resolution`, `bare_retry_is_not_a_lesson` |
|
||||
| `tool_of_cmd(cmd)` | Infers tool name from command (npm, cargo, kubectl, etc.) | used by `derive_lessons` |
|
||||
|
||||
**`crates/mem-cli/src/lessons_cmd.rs`** already contains:
|
||||
|
||||
| Command | What it does |
|
||||
|---|---|
|
||||
| `mem capture --cmd ... --exit ...` | Records command execution events to `~/.mem/events.jsonl` |
|
||||
| `mem resolve` | Derives lessons from events, preserves human confirmations |
|
||||
|
||||
## What remains to complete this task
|
||||
|
||||
The existing code operates on **command execution events** (individual tool invocations with exit codes). This task requires integration with the **GRU-Mem gated loop** (M1.5) which operates on full session transcripts:
|
||||
|
||||
1. **Standing query YAML** — add `tool-failures` query to `queries/<project>.yaml` with the verbatim-invocation requirement
|
||||
2. **Gate-based extraction** — the gated loop (M1.5) decides which transcript chunks contain tool-failure evidence, not `derive_lessons()` from command events
|
||||
3. **End-to-end test** — ingest a failure session, then verify `/memory/context` returns the failure ranked above docs
|
||||
4. **Orchestrator lesson ingestion** — ingest `Poimen/workflows` lesson artifacts as a source
|
||||
5. **Derived filter interaction** — verify lessons are NOT caught by M4.2’s derived filter
|
||||
|
||||
The existing `derive_lessons()` remains useful as a **complementary path** for command-level failures, while the standing query handles full-transcript extraction.
|
||||
|
||||
## Files
|
||||
|
||||
| Action | Path |
|
||||
|---|---|
|
||||
| **Exists** | `crates/mem-core/src/lesson.rs` — `derive_lessons()`, `tool_of_cmd()` |
|
||||
| **Exists** | `crates/mem-cli/src/lessons_cmd.rs` — `mem capture`, `mem resolve` |
|
||||
| Modify | `queries/<project>.yaml` — add `tool-failures` standing query |
|
||||
| Create | `tests/it_tool_failure_learning.rs` — integration tests (8 assertions) |
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
```yaml
|
||||
# queries/<project>.yaml
|
||||
- id: tool-failures
|
||||
question: >
|
||||
Which tool or command invocations failed, what was the exact error,
|
||||
and what was the working alternative? Record the invocation verbatim.
|
||||
```
|
||||
|
||||
This is the only leg of the tool-context bundle that **goes through the update
|
||||
gate**, and it should. A failed `kubectl` invocation is genuine evidence about
|
||||
what happened in this project — unlike reference text (M3.6), which has no
|
||||
evidence to gate on. No bypass, no special casing, no new machinery: one standing
|
||||
question whose answers happen to be operationally useful at task time.
|
||||
|
||||
**The gate's discrimination is the feature here.** Sessions are full of commands
|
||||
that failed for uninteresting reasons — a typo the model immediately fixed, a
|
||||
transient 503. The question asks for the *working alternative*, which is what
|
||||
separates a durable lesson from noise, and the gate is what enforces it. If
|
||||
update-rate on this query runs high, the question is too permissive, not the gate.
|
||||
|
||||
**Verbatim invocation matters.** "Use the right namespace flag" is unusable. The
|
||||
memory has to carry `kubectl get pods --all` → `error: unknown flag: --all` →
|
||||
`kubectl get pods --all-namespaces`, because the next model needs the exact
|
||||
string to pattern-match against what it was about to emit.
|
||||
|
||||
**This is where the orchestrator's lessons should end up.** `Poimen/workflows`
|
||||
already generates lessons on judge rejection (`action/lessons.go`) and discards
|
||||
them at task end. Ingesting those artifacts gives this query a dense, pre-filtered
|
||||
source — failures already judged consequential by a second model.
|
||||
|
||||
**Success is measured end to end, not at L1.** An L1 memory nobody retrieves is
|
||||
worthless. The acceptance criterion is that a task mentioning the tool gets the
|
||||
failure in its `/memory/context` bundle, ranked above the cheatsheet.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Add `tool-failures` to the shipped query templates, with the verbatim
|
||||
requirement in the question text.
|
||||
2. Ingest orchestrator lesson artifacts as a source alongside session
|
||||
transcripts.
|
||||
3. Confirm no interaction with M3.6.4's manifest: lessons are project output, not
|
||||
emitted artifacts, and must not be excluded as derived.
|
||||
4. Measure update-rate for this query separately; it should sit well under the
|
||||
30% M1.8 threshold.
|
||||
5. End-to-end check: ingest a failure session, then request `/memory/context` for
|
||||
a related task and assert the failure is present and ranked above R.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- A session containing a failure-then-fix yields an L1 memory with both forms
|
||||
verbatim.
|
||||
- A session with only transient errors yields none.
|
||||
- The memory appears in `/memory/context` for a related task, above the docs.
|
||||
- Lesson artifacts are not caught by the derived filter.
|
||||
- Update-rate for this query stays under the M1.8 threshold.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** three fixture sessions — one clean failure-then-fix, one transient
|
||||
503 with no lesson, one where the model tried three wrong forms before succeeding.
|
||||
Live gateway for the gate decisions.
|
||||
|
||||
**Integration test** — `tests/it_tool_failure_learning.rs`:
|
||||
1. `a1_failure_becomes_memory` — fixture 1 yields an L1 under `tool-failures`
|
||||
containing both the failing and the working invocation, verbatim.
|
||||
2. `a2_transient_rejected` — fixture 2 produces no L1. This is the assertion that
|
||||
proves the gate is discriminating rather than recording every non-zero exit.
|
||||
3. `a3_multi_attempt_keeps_final` — fixture 3's memory names the working form,
|
||||
not merely the last error.
|
||||
4. `a4_reaches_the_bundle` — after ingest, `GET /memory/context?task=…kubectl…`
|
||||
contains the memory.
|
||||
5. `a5_outranks_documentation` — in that same bundle, assert it sorts above the
|
||||
R cheatsheet section covering the same command.
|
||||
6. `a6_lessons_not_derived` — ingest a lesson artifact; assert no
|
||||
`derived_excluded` event fires for it.
|
||||
7. `a7_update_rate_bounded` — update-rate for this query is under 0.30, reported
|
||||
alongside the other standing queries.
|
||||
8. `a8_provenance_resolves` — the memory's parents resolve to the L0 span
|
||||
containing the actual error text.
|
||||
|
||||
**Command:** `cargo test --workspace tool_failure -- --ignored --nocapture`
|
||||
|
||||
**False pass:**
|
||||
- Asserting only `a1`. A gate that accepts every chunk also produces the right
|
||||
memory for fixture 1; `a2` is the one that distinguishes a filter from a
|
||||
recorder, and it must run in the same binary.
|
||||
- Stopping at L1. Assertions 4 and 5 are the task — an L1 that never reaches a
|
||||
prompt has changed nothing about how the implementer behaves.
|
||||
- Fixtures written by the same model that will be judged on them. Use real
|
||||
session transcripts; synthetic failures are unnaturally clean and the gate
|
||||
accepts them at a rate real sessions will not reproduce.
|
||||
|
||||
## Traps
|
||||
|
||||
- Writing the question to ask for "errors". Every tool result containing the word
|
||||
error becomes evidence, update-rate climbs, and M1.8 goes red for reasons that
|
||||
look unrelated to this task. The working-alternative clause is what bounds it.
|
||||
- Ingesting lessons without a project key. They arrive from the orchestrator, not
|
||||
from a session with a `cwd`, so project resolution has to be explicit or they
|
||||
land in the wrong memory.
|
||||
- Treating a high update-rate here as success. It means the question is loose;
|
||||
the paper's failure mode is a memory that accepts everything.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — tool context, standing queries · [M1.8](M1.8-m1-gate.md)
|
||||
Reference in New Issue
Block a user