116 lines
5.1 KiB
Markdown
116 lines
5.1 KiB
Markdown
# M3.7.3 — `GET /memory/skills?task=` — match a subset to the work
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M3.7 — Tool context |
|
||
| Size | M — 1–3 days |
|
||
| Status | ⬜ Not started |
|
||
| Flags | — |
|
||
| Spec | inlined below |
|
||
| Blocks | M3.7.6 |
|
||
| Depends | M3.5.5, M3.2, M2.1 |
|
||
|
||
## Goal
|
||
|
||
Given a task, return the few skills that apply, so the orchestrator stops cloning
|
||
the same static list for every piece of work.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
```
|
||
GET /memory/skills?task=fix+the+kubectl+parsing+in+the+pod+debugger&project=homelab
|
||
→ 200 [{"name":"infra-root-causes","score":0.81,
|
||
"matched_on":"when_to_use","when_to_use":"When troubleshooting cluster…"}]
|
||
```
|
||
|
||
**Match on `description` and `when_to_use`, never on the body.** Anthropic's own
|
||
skill guidance, encoded in the installed `grafana-core:skill-authoring` rubric,
|
||
makes `description` the field that decides whether a skill fires. Bodies are long,
|
||
full of example output, and match everything — a skill whose body mentions
|
||
`kubectl` in passing would be selected for every Kubernetes task. Matching the
|
||
field the author wrote *for this purpose* also gives authors a lever they can
|
||
reason about.
|
||
|
||
**Embed the metadata, rerank the shortlist.** Same two-stage shape as `mem query`
|
||
(M3.3): embed `description + when_to_use`, cosine-recall a shortlist, then rerank
|
||
against the task text with `bge-reranker-base` (M3.2). The corpus is small enough
|
||
that recall could be exhaustive, but the reranker is what separates "mentions
|
||
Kubernetes" from "is about diagnosing a failing pod".
|
||
|
||
**Empty is a valid answer and must stay cheap.** Most tasks match no skill. The
|
||
endpoint returns `[]`, not the closest thing it found, and the caller proceeds
|
||
with tools and knowledge alone. A floor applies here for the same reason it does
|
||
in M3.6.5: a plausible-but-wrong skill actively steers the implementer.
|
||
|
||
**`_drafts/` stays excluded.** M3.5.5's rule is unchanged and load-bearing —
|
||
matching must not become a side channel that loads an unpromoted skill.
|
||
|
||
**Deterministic ties.** Two skills at the same score sort by name, so an
|
||
orchestrator that caches on the response is not invalidated by rank flapping
|
||
between identical requests.
|
||
|
||
## Steps
|
||
|
||
1. Extend the M3.5.5 handler with `?task=` and `?limit=` (default 3).
|
||
2. Build the match index over `description + when_to_use` for promoted skills.
|
||
3. Recall then rerank against the task text; apply the score floor.
|
||
4. Return `score` and `matched_on` so a bad match is diagnosable without a rerun.
|
||
5. Rebuild the index on skill promotion; no restart required.
|
||
6. `?task=` absent keeps the existing full-catalog behaviour exactly.
|
||
|
||
## Acceptance
|
||
|
||
- A Kubernetes debugging task matches the infra skill; an unrelated task does not.
|
||
- Draft skills never appear.
|
||
- No match returns `[]` with 200.
|
||
- Omitting `task` returns the full catalog, byte-identical to today.
|
||
- Equal scores order deterministically.
|
||
|
||
## Verify
|
||
|
||
**Harness:** vault fixture with 6 promoted skills across distinct domains plus 2
|
||
drafts. Live reranker for scoring; deterministic embedder elsewhere.
|
||
|
||
**Integration test** — `tests/it_skill_matching.rs`:
|
||
1. `a1_relevant_match` — a pod-debugging task returns the infra skill first.
|
||
2. `a2_irrelevant_no_match` — "update the README changelog" returns `[]`.
|
||
3. `a3_drafts_excluded` — a task whose text matches a draft's description
|
||
verbatim returns `[]`.
|
||
4. `a4_body_not_matched` — a skill whose *body* mentions `kubectl` but whose
|
||
description is about something else is not returned for a `kubectl` task.
|
||
This is the assertion that proves the field restriction.
|
||
5. `a5_no_task_unchanged` — omit `task`; assert byte-identical to M3.5.5's
|
||
existing fixture output.
|
||
6. `a6_floor_applies` — a weakly-related task returns `[]` rather than the
|
||
best-of-bad.
|
||
7. `a7_deterministic_ties` — two identically-described skills; assert stable
|
||
name-ordered output across 10 calls.
|
||
8. `a8_reranker_reorders` — capture pre- and post-rerank order; assert they
|
||
differ on at least one fixture task, proving the reranker is wired.
|
||
9. `a9_promotion_visible` — promote a draft, re-query without restart; assert it
|
||
is now matchable.
|
||
|
||
**Command:** `cargo test -p mem-api skill_matching`
|
||
|
||
**False pass:**
|
||
- Fixtures whose descriptions share no vocabulary. Any embedder separates
|
||
unrelated topics; assertion 4 needs a deliberate body/description conflict, and
|
||
assertion 6 needs a genuinely borderline task, or both pass with a keyword
|
||
`LIKE`.
|
||
- Asserting only that the right skill is *present*. Returning all 6 sorted also
|
||
contains the right one; assert the length and the floor.
|
||
|
||
## Traps
|
||
|
||
- Indexing skill bodies "for better recall". It inverts the design: bodies are
|
||
where every skill looks alike, and the author's `description` stops being the
|
||
control surface it was written to be.
|
||
- Tuning the floor against the same fixtures used to assert matching. It converges
|
||
on a threshold that fits six skills and fails on sixty; hold out tasks.
|
||
- Rebuilding the index per request. It is small, but this endpoint sits in the
|
||
path of every task the orchestrator runs.
|
||
|
||
---
|
||
|
||
Background: [DESIGN.md](../DESIGN.md) — tool context, skills · [M3.5.5](M3.5.5-skills-endpoint.md)
|