365 lines
13 KiB
Markdown
365 lines
13 KiB
Markdown
# M3.7 Failure Diagnosis Pipeline — Complete Design
|
||
|
||
**Phase:** M3.7 — Tool context
|
||
**Status:** M3.7.7 ✅ Complete | M3.7.8 🔄 In design | M3.7.4 ⬜ Pending M8.2
|
||
**Last updated:** 2026-08-28
|
||
|
||
---
|
||
|
||
## Overview
|
||
|
||
M3.7 answers: **"What do we already know about this failure or tool?"** over HTTP.
|
||
|
||
Provides three-tier context lookups with increasing cost and falling confidence:
|
||
|
||
1. **Tier 1 (Exact)** — Exact signature match (past solution)
|
||
2. **Tier 2 (Semantic)** — Hybrid search (similar cases)
|
||
3. **Tier 3 (Reference)** — Documentation (general guidance)
|
||
|
||
---
|
||
|
||
## Pipeline Architecture
|
||
|
||
```
|
||
User Query
|
||
│ (e.g., "npm ERESOLVE error resolving typescript")
|
||
│
|
||
├─────────────────────────────────────────────────────────────────┐
|
||
│ │
|
||
│ LAYER 1: SYMPTOM PROJECTION (M3.7.8) │
|
||
│ ├─ Normalize query to symptom vector │
|
||
│ ├─ Extract keywords, expand abbreviations (ERESOLVE → error) │
|
||
│ ├─ Generate deterministic hash (sym_sha) │
|
||
│ └─ Output: SymptomVector { tool, normalised, sym_sha } │
|
||
│ │
|
||
└──────────────────────┬──────────────────────────────────────────┘
|
||
│
|
||
↓
|
||
┌─────────────────────────────────────────────────────────────────┐
|
||
│ │
|
||
│ LAYER 2: SIGNATURE MATCHING (M3.7.7 + M3.7.8) │
|
||
│ ├─ Query DB: lessons WHERE sym_sha = ? │
|
||
│ ├─ If hit → TIER 1: Return cached solution │
|
||
│ └─ If miss → Continue to LAYER 3 │
|
||
│ │
|
||
└──────────────────────┬──────────────────────────────────────────┘
|
||
│ (Tier 1 miss)
|
||
↓
|
||
┌─────────────────────────────────────────────────────────────────┐
|
||
│ │
|
||
│ LAYER 3: HYBRID SEARCH (M8) │
|
||
│ ├─ Embed query (semantic, 60% weight) │
|
||
│ ├─ Search pgvector (cosine similarity) │
|
||
│ ├─ Search OpenSearch (BM25, 40% weight) │
|
||
│ ├─ Fuse results (weighted linear: 0.6*sem + 0.4*lex) │
|
||
│ └─ TIER 2: Return top-10 ranked results │
|
||
│ │
|
||
└──────────────────────┬──────────────────────────────────────────┘
|
||
│ (Tier 2 miss)
|
||
↓
|
||
┌─────────────────────────────────────────────────────────────────┐
|
||
│ │
|
||
│ LAYER 4: REFERENCE CORPUS (M3.6) │
|
||
│ ├─ Query reference docs (kubectl, npm, etc.) │
|
||
│ └─ TIER 3: Return general guidance (lowest confidence) │
|
||
│ │
|
||
└─────────────────────────────────────────────────────────────────┘
|
||
|
||
Response (M3.7.4 context endpoint):
|
||
{
|
||
"tier": 1,
|
||
"confidence": "high",
|
||
"query_normalised": "npm error resolve dependency typescript",
|
||
"results": [
|
||
{
|
||
"source": "lesson",
|
||
"title": "Fix npm ERESOLVE errors",
|
||
"solution": "npm ci (deterministic); npm install (latest versions)"
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## Task Breakdown
|
||
|
||
### M3.7.7 ✅ COMPLETE — Failure Signature Extraction
|
||
|
||
**Commit:** `463958b`
|
||
**Status:** 18/18 unit tests passing
|
||
**Implementation:** `crates/mem-core/src/lesson.rs` (871 LOC)
|
||
|
||
#### What it does:
|
||
- Extracts root error from 50KB failure logs
|
||
- Normalizes timestamps, paths, SHAs, addresses
|
||
- Generates deterministic SHA256 hash (sig_sha)
|
||
- Handles cascading failures (picks root cause, not consequence)
|
||
|
||
#### Example:
|
||
```
|
||
Raw log (50KB):
|
||
2026-08-21T10:02:11.482Z
|
||
/home/runner/work/Poimen/memory/k8s/app/opensearch.yaml
|
||
error: error validating data: [ValidationError(...)]
|
||
The server is rejecting the request. (422)
|
||
[... 100 lines of structured output ...]
|
||
|
||
↓ [M3.7.7 extract()]
|
||
|
||
Signature:
|
||
tool: "kubectl"
|
||
raw: "error: error validating data: [ValidationError(...)]"
|
||
normalised: "error validating data kubernetes"
|
||
sig_sha: "abc123def456789..." ← deterministic
|
||
rule: "kubectl_error_line"
|
||
```
|
||
|
||
#### Tests (18 passing):
|
||
- ✅ Same failure (2 runs) → identical hash
|
||
- ✅ Different failures → different hashes
|
||
- ✅ Removes ANSI, timestamps, paths, SHAs
|
||
- ✅ Picks first error (cascade suppression)
|
||
- ✅ Unknown tools fallback gracefully
|
||
- ✅ Tool name part of identity
|
||
- ✅ Similar wording still matches
|
||
- ✅ + 11 more (see `cargo test -p mem-core lesson`)
|
||
|
||
---
|
||
|
||
### M3.7.8 🔄 IN DESIGN — Symptom Projection
|
||
|
||
**Status:** Full design complete (see `docs/M3.7.8-SYMPTOM_PROJECTION.md`)
|
||
**Estimated:** 1–2 days
|
||
**Implementation Plan:** 250 LOC + 6 test assertions
|
||
|
||
#### What it does:
|
||
Transforms **user queries** into normalized **symptom vectors** that can match extracted signatures.
|
||
|
||
**Three-stage pipeline:**
|
||
|
||
**Stage 1: Extract Keywords**
|
||
```
|
||
Query: "npm can't find tslib module error"
|
||
↓
|
||
Keywords: ["npm", "find", "tslib", "module", "error"]
|
||
```
|
||
|
||
**Stage 2: Normalize**
|
||
```
|
||
Keywords: ["npm", "find", "tslib", "module", "error"]
|
||
↓ [Remove stop words: can, find (weak), t]
|
||
Normalized: "error module npm tslib"
|
||
↓ [Sort alphabetically for idempotence]
|
||
Canonical: "error module npm tslib"
|
||
```
|
||
|
||
**Stage 3: Generate Hash**
|
||
```
|
||
Canonical: "error module npm tslib"
|
||
↓ [SHA256(tool + "\n" + normalized)]
|
||
sym_sha: "xyz789abc..." ← matches M3.7.7 signature if similar
|
||
```
|
||
|
||
#### Example Match:
|
||
```
|
||
Extracted signature (M3.7.7):
|
||
normalised: "error module tslib"
|
||
sig_sha: "abc123..."
|
||
|
||
User query (M3.7.8):
|
||
query: "npm cannot resolve tslib"
|
||
normalised: "error module npm tslib" (after expand "cannot")
|
||
sym_sha: "abc123..." ← MATCH!
|
||
|
||
→ Tier 1 hit: Return past solution
|
||
```
|
||
|
||
#### Tests (6 assertions):
|
||
1. Same symptom variant → same hash
|
||
2. Abbreviation expansion (ERESOLVE, ERR, OOM, EACCES)
|
||
3. Stop word removal (can, the, able, to)
|
||
4. Tool consistency (npm != cargo for same error)
|
||
5. Case insensitive (NPM = npm)
|
||
6. Keyword order irrelevant (sorted before hash)
|
||
|
||
#### Files to create:
|
||
- `crates/mem-core/src/symptom_projection.rs` (250 LOC)
|
||
- `tests/it_symptom_projection.rs` (400 LOC, 6 assertions)
|
||
- `fixtures/symptoms/` (test query examples)
|
||
|
||
---
|
||
|
||
### M3.7.4 ⬜ PENDING — Context Endpoint Integration
|
||
|
||
**Status:** Waiting for M3.7.8 + M8.2 (hybrid search live)
|
||
**Purpose:** HTTP endpoint gluing tiers 1–3 together
|
||
**Endpoint:** `GET /memory/context?query=...&tool=...`
|
||
|
||
#### Three-tier logic:
|
||
```rust
|
||
pub async fn get_context(query: &str, tool: Option<&str>) -> ContextResult {
|
||
// TIER 1: Exact signature match
|
||
let symptom = project_symptom(tool.unwrap_or("unknown"), query); // M3.7.8
|
||
if let Some(lesson) = find_lesson_by_sym_sha(&symptom.sym_sha) {
|
||
return high_confidence_result(lesson);
|
||
}
|
||
|
||
// TIER 2: Hybrid search (requires M8.2)
|
||
let hybrid = hybrid_search(query, tool).await?;
|
||
if !hybrid.is_empty() {
|
||
return medium_confidence_result(hybrid);
|
||
}
|
||
|
||
// TIER 3: Reference corpus (requires M3.6)
|
||
let reference = reference_corpus_search(query, tool).await?;
|
||
return low_confidence_result(reference);
|
||
}
|
||
```
|
||
|
||
#### Response format:
|
||
```json
|
||
{
|
||
"tier": 1,
|
||
"confidence": "high",
|
||
"query_normalised": "error module npm tslib",
|
||
"results": [
|
||
{
|
||
"source": "lesson",
|
||
"title": "Fix npm module not found",
|
||
"keywords": ["npm", "error", "module"],
|
||
"solution": "Check package.json, npm install",
|
||
"applies_to": ["npm", "yarn"],
|
||
"severity": "medium"
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
### M3.7.6 ⬜ PENDING — Composition Gate
|
||
|
||
**Status:** Depends on M3.7.4 + M3.7.8
|
||
**Purpose:** Verify tiers work end-to-end (accuracy, latency, coverage)
|
||
|
||
#### Gate assertions:
|
||
1. Tier 1 queries resolve in < 50ms (cached lookup)
|
||
2. Tier 2 queries resolve in < 500ms (hybrid search)
|
||
3. Tier 3 queries resolve in < 1000ms (reference corpus)
|
||
4. Exactly one tier returns results (no duplicates across tiers)
|
||
5. Confidence scores monotonically decrease (tier 1 > 2 > 3)
|
||
6. Coverage: 95% of real failures find a tier 1 or 2 match
|
||
|
||
---
|
||
|
||
## Retired Tasks
|
||
|
||
✅ **M3.7.3** — `GET /memory/skills?task=...` (skill matching)
|
||
→ Redundant with M8 hybrid search
|
||
→ Removed: 1 task
|
||
|
||
✅ **M3.7.5** — `tool-failures` standing query
|
||
→ Redundant with M8 hybrid search
|
||
→ Removed: 1 task
|
||
|
||
**Outcome:** M3.7 reduced from 6 tasks → 4 tasks (gain: 2 simplified, no loss)
|
||
|
||
---
|
||
|
||
## Integration Points
|
||
|
||
| Component | Depends On | Used By |
|
||
|-----------|-----------|---------|
|
||
| M3.7.7 (Signature) | M3.6.1 (DocCorpus) | M3.7.8 (Symptom) |
|
||
| M3.7.8 (Symptom) | M3.7.7 (Signature) | M3.7.4 (Context endpoint) |
|
||
| M3.7.4 (Context) | M3.7.8 + M8.2 + M3.6 | M3.7.6 (Gate) |
|
||
| M3.7.6 (Gate) | M3.7.4 | — |
|
||
|
||
**Critical path:**
|
||
```
|
||
M3.7.7 ✅ → M3.7.8 🔄 → M3.7.4 ⬜
|
||
↓ [waits for M8.2]
|
||
M3.7.6 ⬜
|
||
```
|
||
|
||
---
|
||
|
||
## Performance Targets
|
||
|
||
| Operation | Latency | Notes |
|
||
|-----------|---------|-------|
|
||
| M3.7.7: Extract signature | < 50ms | 50KB log → hash (rule-based) |
|
||
| M3.7.8: Project symptom | < 10ms | Query → normalized vector |
|
||
| Tier 1 lookup (M3.7.4) | < 50ms | DB hash lookup |
|
||
| Tier 2 lookup (M3.7.4) | < 500ms | Hybrid search (parallel engines) |
|
||
| Tier 3 lookup (M3.7.4) | < 1000ms | Reference corpus search |
|
||
|
||
---
|
||
|
||
## Success Criteria
|
||
|
||
### M3.7.7 ✅
|
||
- [x] 18 unit tests passing (signature extraction)
|
||
- [x] Deterministic hashing (same failure → same hash)
|
||
- [x] CLI command: `mem sig explain`
|
||
- [x] Fixtures: real captured logs (npm, cargo, kubectl, etc.)
|
||
|
||
### M3.7.8 🔄
|
||
- [ ] 6 integration tests passing (symptom projection)
|
||
- [ ] Deterministic hashing (same query variant → same hash)
|
||
- [ ] Abbreviation expansion per tool (ERESOLVE, ERR, EACCES, etc.)
|
||
- [ ] Integrated with M3.7.4 context endpoint
|
||
- [ ] Tier 1 lookups work end-to-end
|
||
|
||
### M3.7.4 ⬜
|
||
- [ ] HTTP endpoint operational
|
||
- [ ] Three-tier logic working
|
||
- [ ] Response includes `tier`, `confidence` fields
|
||
- [ ] Integrated with M8.2 (hybrid search)
|
||
|
||
### M3.7.6 ⬜
|
||
- [ ] Gate assertions passing (latency, coverage, accuracy)
|
||
- [ ] End-to-end failure diagnosis working
|
||
|
||
---
|
||
|
||
## Documents
|
||
|
||
- **Core Implementation:** `crates/mem-core/src/lesson.rs` (M3.7.7, 871 LOC)
|
||
- **M3.7.8 Design:** `docs/M3.7.8-SYMPTOM_PROJECTION.md` (this guide, 14KB)
|
||
- **Pipeline Flow:** `memory-flow.md` § "M3.7.7 → M3.7.8: Failure Diagnosis Pipeline" (176 lines)
|
||
- **Tests:** `tests/it_signature.rs` (M3.7.7, 9 assertions)
|
||
|
||
---
|
||
|
||
## Timeline
|
||
|
||
**Completed:** M3.7.7 (18 unit tests passing)
|
||
**Next (1–2 days):** M3.7.8 symptom projection
|
||
**Then (1 day):** M3.7.4 context endpoint
|
||
**Finally (1 day):** M3.7.6 composition gate
|
||
|
||
**Total M3.7:** ~5 days (1 done, 4 remaining)
|
||
|
||
---
|
||
|
||
## Team Notes
|
||
|
||
- **Determinism is critical:** Both M3.7.7 and M3.7.8 must hash identically across runs
|
||
- Use sorted keywords (not insertion order)
|
||
- Include tool name in identity
|
||
- Strip all volatile data (timestamps, addresses, etc.)
|
||
|
||
- **Tier 1 only fires if M3.7.8 sym_sha matches M3.7.7 sig_sha**
|
||
- If query is ambiguous or doesn't match any signature → skip to tier 2
|
||
- Fall back to hybrid search (tier 2) gracefully
|
||
|
||
- **M8.2 is a blocker for M3.7.4**
|
||
- Hybrid search must be live before context endpoint can work
|
||
- (Tier 2 fallback requires functional search)
|
||
|
||
---
|
||
|
||
**End of M3.7 summary.**
|