feat: complete M0.1-M0.4 phases

M0.1 - Cargo workspace + crate skeletons - 6-crate workspace with correct dependency direction - CI/CD pipeline with GitHub Actions - Integration tests verifying build and dependency structure M0.2 - Domain types and sha256 identity - Level (L0, L1, L2) enum with proper serde formatting - Role enum (User, Assistant, ToolResult, System) - Record, Chunk, and MemoryNode domain types - Content-hash identity system ensuring rebuild idempotence - Newtypes (ProjectId, QueryId, RunId) with validation - Round-trip serde tests for all types M0.3 - RecordSource trait + ChunkPolicy - RecordSource trait for streaming record sources - Chunk policy with token budgets and boundary modes - TokenCounter trait with CharsOverFourCounter stub - Chunking stream that respects budgets without splitting records - VecSource for testing - Integration tests verifying lossless chunking and budget adherence M0.4 - Tokenizer-backed chunk sizing - Vendored Qwen2 tokenizer with hash verification - QwenTokenCounter implementing proper token counting - Hash guard that fails on modified tokenizer - mem tokens CLI subcommand for token counting - Integration tests with known string counts, hash guards, and budget verification Total: 19 integration tests passing, all phases verified to compose correctly Workspace builds cleanly with no clippy warnings
2026-08-22 23:13:42 -07:00
parent 144fa33574
commit 631cbfa3e9
36 changed files with 3379 additions and 5 deletions
@@ -0,0 +1,166 @@
+# M3.5.5 — GET /skills and /skills/{name}: loadable skills catalog
+
+| Field | Value |
+|---|---|
+| Phase | M3.5 — Distributed API Layer |
+| Size | M — 1–3 days |
+| Status | ⬜ Not started |
+| Flags | — |
+| Spec | inlined below |
+| Blocks | M3.5.8 |
+| Depends | M3.5.1, M4.1 (skill drafts exist locally) |
+
+## Goal
+
+Read-only endpoints for skill catalog. List all promoted skills (exclude `_drafts/`), fetch individual skill metadata and body. Skills are Obsidian notes; expose them over HTTP for agent discovery.
+
+## Design
+
+**List all loadable skills:**
+```
+GET /memory/skills?loadable=true
+→ 200 {
+  "skills": [
+    {
+      "name": "infra-root-causes",
+      "description": "Identify root causes of infrastructure failures",
+      "when_to_use": "When troubleshooting cluster or service outages",
+      "argument_hint": "--project <name>",
+      "promoted_at": "2026-08-20T10:30:00Z",
+      "generated_from": null
+    },
+    ...
+  ]
+}
+```
+
+**Get one skill (metadata only):**
+```
+GET /memory/skills/infra-root-causes
+→ 200 {
+  "name": "infra-root-causes",
+  "description": "...",
+  "when_to_use": "...",
+  "argument_hint": "...",
+  "promoted_at": "2026-08-20T...",
+  "generated_from": null
+}
+```
+
+**Get skill with body (full content):**
+```
+GET /memory/skills/infra-root-causes?include_body=true
+→ 200 {
+  "name": "infra-root-causes",
+  "description": "...",
+  "body": "# Infra root causes\n\n..."
+}
+```
+
+**Filters:**
+- `loadable=true` (default): exclude `_drafts/`, return only promoted skills
+- `loadable=false`: include everything (admin only — must have special apikey, documented in code)
+
+## Technical
+
+**Source:** Vault at `vault/skills/` contains skill markdown files. Each skill is a directory:
+```
+vault/skills/
+  infra-root-causes/
+    SKILL.md         <- frontmatter + body
+```
+
+**Frontmatter (YAML in SKILL.md):**
+```yaml
+---
+name: infra-root-causes
+description: Identify root causes of infrastructure failures
+when_to_use: When troubleshooting cluster or service outages
+argument_hint: --project <name>
+generated_from: null | <L2 sha256>
+---
+```
+
+**Drafts are in `vault/skills/_drafts/`:**
+```
+vault/skills/
+  _drafts/
+    new-skill/
+      SKILL.md
+```
+
+Only load from `vault/skills/*/SKILL.md` (not `_drafts`), unless `loadable=false` is passed with an admin key.
+
+## Steps
+
+1. `GET /memory/skills` handler:
+   - List `vault/skills/` directory (skip `_drafts/`)
+   - For each `*/SKILL.md`, parse frontmatter
+   - Extract: `name`, `description`, `when_to_use`, `argument_hint`, `promoted_at` (file mtime)
+   - Parse `generated_from` field to show provenance
+   - Return array
+
+2. `GET /memory/skills/{name}` handler:
+   - Load `vault/skills/{name}/SKILL.md`
+   - Parse frontmatter and body
+   - If `include_body=false` (default), return metadata only
+   - If `include_body=true`, include markdown body
+
+3. `loadable` query param (admin-only feature):
+   - Default: exclude `_drafts/`
+   - `loadable=false` with admin apikey: include `_drafts/` in listing
+   - Non-admin key requesting `loadable=false` → 403 Forbidden
+
+4. Error handling:
+   - Skill not found → 404 with `{"error":"not_found","reason":"skill 'xyz' not promoted"}`
+   - Malformed SKILL.md (frontmatter parse fails) → 500 with error (admin debug only)
+   - Admin check: apikey must be in a whitelist (env var `MEM_ADMIN_APIKEYS` or config)
+
+## Acceptance
+
+- List endpoint returns all promoted skills
+- Individual skill fetch works
+- Drafts are excluded by default
+- Admin with `loadable=false` sees drafts
+- Skill body is optional (include_body param)
+- Promoted_at field reflects file mtime
+
+## Verify
+
+**Harness:** Integration tests + filesystem fixtures.
+
+**Setup:** Create test `vault/skills/` with:
+- `vault/skills/test-skill-1/SKILL.md` (promoted)
+- `vault/skills/test-skill-2/SKILL.md` (promoted)
+- `vault/skills/_drafts/draft-skill/SKILL.md` (unpromoted)
+
+**Integration test** — `tests/it_skills_endpoint.rs`:
+1. `a1_list_skills_returns_promoted` — GET /skills returns array with test-skill-1 and test-skill-2.
+2. `a2_drafts_excluded_by_default` — GET /skills does not include draft-skill.
+3. `a3_drafts_included_with_admin_key` — GET /skills?loadable=false with admin apikey includes draft-skill.
+4. `a4_non_admin_denied_drafts` — GET /skills?loadable=false with regular apikey returns 403.
+5. `a5_get_single_skill_metadata` — GET /skills/test-skill-1 returns 200 with frontmatter fields.
+6. `a6_include_body_true` — GET /skills/test-skill-1?include_body=true returns body field with markdown.
+7. `a7_include_body_false` — GET /skills/test-skill-1?include_body=false (or omitted) does not include body field.
+8. `a8_skill_not_found` — GET /skills/nonexistent returns 404.
+9. `a9_promoted_at_is_file_mtime` — GET /skills/test-skill-1, assert promoted_at is a valid ISO timestamp close to SKILL.md's modification time.
+10. `a10_generated_from_field` — SKILL.md with `generated_from: sha256xyz` is parsed and returned as-is.
+
+**Command:** `cargo test -p mem-cli skills_endpoint`
+
+**False pass:**
+- Drafts never created in test fixtures. The default exclude-drafts logic is untestable without a draft.
+- Admin key never tested. Non-admin path and admin path can be identical in code.
+- Promoted_at never validated. Can return a fake date; file mtime is the only source.
+- Frontmatter parsing doesn't validate required fields (name, description). A malformed SKILL.md is silently returned with null values.
+
+## Traps
+
+- Vault directory may not exist locally (only in deployed cluster). Start with a default empty list if vault/ is missing.
+- YAML frontmatter parsing is fussy. A tab instead of spaces breaks YAML. Use a YAML parser (serde_yaml) and validate on load.
+- File mtime precision: Unix mtime is seconds; SKILL.md edits may not increment it if done within the same second. Use actual write timestamp if available.
+- Admin key stored in env var. If unset, default to deny (safer than default allow).
+
+---
+
+Background: [DESIGN.md § Skills — the procedural projection](../DESIGN.md#skills--the-procedural-projection)