poimen-memory/docs/M3.7-FAILURE_DIAGNOSIS_SUMMARY.md

M3.7 Failure Diagnosis Pipeline — Complete Design

Phase: M3.7 — Tool context
Status: M3.7.7 ✅ Complete | M3.7.8 🔄 In design | M3.7.4 ⬜ Pending M8.2
Last updated: 2026-08-28

---

Overview

M3.7 answers: "What do we already know about this failure or tool?" over HTTP.

Provides three-tier context lookups with increasing cost and falling confidence:

1. Tier 1 (Exact) — Exact signature match (past solution)
2. Tier 2 (Semantic) — Hybrid search (similar cases)
3. Tier 3 (Reference) — Documentation (general guidance)

---

Pipeline Architecture

User Query
  │ (e.g., "npm ERESOLVE error resolving typescript")
  │
  ├─────────────────────────────────────────────────────────────────┐
  │                                                                  │
  │ LAYER 1: SYMPTOM PROJECTION (M3.7.8)                            │
  │ ├─ Normalize query to symptom vector                            │
  │ ├─ Extract keywords, expand abbreviations (ERESOLVE → error)    │
  │ ├─ Generate deterministic hash (sym_sha)                        │
  │ └─ Output: SymptomVector { tool, normalised, sym_sha }          │
  │                                                                  │
  └──────────────────────┬──────────────────────────────────────────┘
                         │
                         ↓
  ┌─────────────────────────────────────────────────────────────────┐
  │                                                                  │
  │ LAYER 2: SIGNATURE MATCHING (M3.7.7 + M3.7.8)                   │
  │ ├─ Query DB: lessons WHERE sym_sha = ?                          │
  │ ├─ If hit → TIER 1: Return cached solution                      │
  │ └─ If miss → Continue to LAYER 3                                │
  │                                                                  │
  └──────────────────────┬──────────────────────────────────────────┘
                         │ (Tier 1 miss)
                         ↓
  ┌─────────────────────────────────────────────────────────────────┐
  │                                                                  │
  │ LAYER 3: HYBRID SEARCH (M8)                                     │
  │ ├─ Embed query (semantic, 60% weight)                           │
  │ ├─ Search pgvector (cosine similarity)                          │
  │ ├─ Search OpenSearch (BM25, 40% weight)                         │
  │ ├─ Fuse results (weighted linear: 0.6*sem + 0.4*lex)            │
  │ └─ TIER 2: Return top-10 ranked results                         │
  │                                                                  │
  └──────────────────────┬──────────────────────────────────────────┘
                         │ (Tier 2 miss)
                         ↓
  ┌─────────────────────────────────────────────────────────────────┐
  │                                                                  │
  │ LAYER 4: REFERENCE CORPUS (M3.6)                                │
  │ ├─ Query reference docs (kubectl, npm, etc.)                    │
  │ └─ TIER 3: Return general guidance (lowest confidence)          │
  │                                                                  │
  └─────────────────────────────────────────────────────────────────┘

Response (M3.7.4 context endpoint):
{
  "tier": 1,
  "confidence": "high",
  "query_normalised": "npm error resolve dependency typescript",
  "results": [
    {
      "source": "lesson",
      "title": "Fix npm ERESOLVE errors",
      "solution": "npm ci (deterministic); npm install (latest versions)"
    }
  ]
}

---

Task Breakdown

M3.7.7 ✅ COMPLETE — Failure Signature Extraction

Commit: 463958b
Status: 18/18 unit tests passing
Implementation: crates/mem-core/src/lesson.rs (871 LOC)

What it does:

• Extracts root error from 50KB failure logs
• Normalizes timestamps, paths, SHAs, addresses
• Generates deterministic SHA256 hash (sig_sha)
• Handles cascading failures (picks root cause, not consequence)

Example:

Raw log (50KB):
  2026-08-21T10:02:11.482Z
  /home/runner/work/Poimen/memory/k8s/app/opensearch.yaml
  error: error validating data: [ValidationError(...)]
  The server is rejecting the request. (422)
  [... 100 lines of structured output ...]

↓ [M3.7.7 extract()]

Signature:
  tool: "kubectl"
  raw: "error: error validating data: [ValidationError(...)]"
  normalised: "error validating data kubernetes"
  sig_sha: "abc123def456789..." ← deterministic
  rule: "kubectl_error_line"

Tests (18 passing):

• ✅ Same failure (2 runs) → identical hash
• ✅ Different failures → different hashes
• ✅ Removes ANSI, timestamps, paths, SHAs
• ✅ Picks first error (cascade suppression)
• ✅ Unknown tools fallback gracefully
• ✅ Tool name part of identity
• ✅ Similar wording still matches
• ✅ + 11 more (see cargo test -p mem-core lesson)

---

M3.7.8 🔄 IN DESIGN — Symptom Projection

Status: Full design complete (see docs/M3.7.8-SYMPTOM_PROJECTION.md)
Estimated: 1–2 days
Implementation Plan: 250 LOC + 6 test assertions

What it does:

Transforms user queries into normalized symptom vectors that can match extracted signatures.

Three-stage pipeline:

Stage 1: Extract Keywords

Query: "npm can't find tslib module error"
↓
Keywords: ["npm", "find", "tslib", "module", "error"]

Stage 2: Normalize

Keywords: ["npm", "find", "tslib", "module", "error"]
↓ [Remove stop words: can, find (weak), t]
Normalized: "error module npm tslib"
↓ [Sort alphabetically for idempotence]
Canonical: "error module npm tslib"

Stage 3: Generate Hash

Canonical: "error module npm tslib"
↓ [SHA256(tool + "\n" + normalized)]
sym_sha: "xyz789abc..." ← matches M3.7.7 signature if similar

Example Match:

Extracted signature (M3.7.7):
  normalised: "error module tslib"
  sig_sha: "abc123..."

User query (M3.7.8):
  query: "npm cannot resolve tslib"
  normalised: "error module npm tslib" (after expand "cannot")
  sym_sha: "abc123..." ← MATCH!
  
→ Tier 1 hit: Return past solution

Tests (6 assertions):

1. Same symptom variant → same hash
2. Abbreviation expansion (ERESOLVE, ERR, OOM, EACCES)
3. Stop word removal (can, the, able, to)
4. Tool consistency (npm != cargo for same error)
5. Case insensitive (NPM = npm)
6. Keyword order irrelevant (sorted before hash)

Files to create:

• crates/mem-core/src/symptom_projection.rs (250 LOC)
• tests/it_symptom_projection.rs (400 LOC, 6 assertions)
• fixtures/symptoms/ (test query examples)

---

M3.7.4 ⬜ PENDING — Context Endpoint Integration

Status: Waiting for M3.7.8 + M8.2 (hybrid search live)
Purpose: HTTP endpoint gluing tiers 1–3 together
Endpoint: GET /memory/context?query=...&tool=...

Three-tier logic:

pub async fn get_context(query: &str, tool: Option<&str>) -> ContextResult {
    // TIER 1: Exact signature match
    let symptom = project_symptom(tool.unwrap_or("unknown"), query); // M3.7.8
    if let Some(lesson) = find_lesson_by_sym_sha(&symptom.sym_sha) {
        return high_confidence_result(lesson);
    }
    
    // TIER 2: Hybrid search (requires M8.2)
    let hybrid = hybrid_search(query, tool).await?;
    if !hybrid.is_empty() {
        return medium_confidence_result(hybrid);
    }
    
    // TIER 3: Reference corpus (requires M3.6)
    let reference = reference_corpus_search(query, tool).await?;
    return low_confidence_result(reference);
}

Response format:

{
  "tier": 1,
  "confidence": "high",
  "query_normalised": "error module npm tslib",
  "results": [
    {
      "source": "lesson",
      "title": "Fix npm module not found",
      "keywords": ["npm", "error", "module"],
      "solution": "Check package.json, npm install",
      "applies_to": ["npm", "yarn"],
      "severity": "medium"
    }
  ]
}

---

M3.7.6 ⬜ PENDING — Composition Gate

Status: Depends on M3.7.4 + M3.7.8
Purpose: Verify tiers work end-to-end (accuracy, latency, coverage)

Gate assertions:

1. Tier 1 queries resolve in < 50ms (cached lookup)
2. Tier 2 queries resolve in < 500ms (hybrid search)
3. Tier 3 queries resolve in < 1000ms (reference corpus)
4. Exactly one tier returns results (no duplicates across tiers)
5. Confidence scores monotonically decrease (tier 1 > 2 > 3)
6. Coverage: 95% of real failures find a tier 1 or 2 match

---

Retired Tasks

✅ M3.7.3 — GET /memory/skills?task=... (skill matching)
→ Redundant with M8 hybrid search
→ Removed: 1 task

✅ M3.7.5 — tool-failures standing query
→ Redundant with M8 hybrid search
→ Removed: 1 task

Outcome: M3.7 reduced from 6 tasks → 4 tasks (gain: 2 simplified, no loss)

---

Integration Points

Component	Depends On	Used By
M3.7.7 (Signature)	M3.6.1 (DocCorpus)	M3.7.8 (Symptom)
M3.7.8 (Symptom)	M3.7.7 (Signature)	M3.7.4 (Context endpoint)
M3.7.4 (Context)	M3.7.8 + M8.2 + M3.6	M3.7.6 (Gate)
M3.7.6 (Gate)	M3.7.4	—

Critical path:

M3.7.7 ✅ → M3.7.8 🔄 → M3.7.4 ⬜
                           ↓ [waits for M8.2]
                        M3.7.6 ⬜

---

Performance Targets

Operation	Latency	Notes
M3.7.7: Extract signature	< 50ms	50KB log → hash (rule-based)
M3.7.8: Project symptom	< 10ms	Query → normalized vector
Tier 1 lookup (M3.7.4)	< 50ms	DB hash lookup
Tier 2 lookup (M3.7.4)	< 500ms	Hybrid search (parallel engines)
Tier 3 lookup (M3.7.4)	< 1000ms	Reference corpus search

---

Success Criteria

M3.7.7 ✅

• 18 unit tests passing (signature extraction)
• Deterministic hashing (same failure → same hash)
• CLI command: mem sig explain
• Fixtures: real captured logs (npm, cargo, kubectl, etc.)

M3.7.8 🔄

• 6 integration tests passing (symptom projection)
• Deterministic hashing (same query variant → same hash)
• Abbreviation expansion per tool (ERESOLVE, ERR, EACCES, etc.)
• Integrated with M3.7.4 context endpoint
• Tier 1 lookups work end-to-end

M3.7.4 ⬜

• HTTP endpoint operational
• Three-tier logic working
• Response includes tier, confidence fields
• Integrated with M8.2 (hybrid search)

M3.7.6 ⬜

• Gate assertions passing (latency, coverage, accuracy)
• End-to-end failure diagnosis working

---

Documents

• Core Implementation: crates/mem-core/src/lesson.rs (M3.7.7, 871 LOC)
• M3.7.8 Design: docs/M3.7.8-SYMPTOM_PROJECTION.md (this guide, 14KB)
• Pipeline Flow: memory-flow.md § "M3.7.7 → M3.7.8: Failure Diagnosis Pipeline" (176 lines)
• Tests: tests/it_signature.rs (M3.7.7, 9 assertions)

---

Timeline

Completed: M3.7.7 (18 unit tests passing)
Next (1–2 days): M3.7.8 symptom projection
Then (1 day): M3.7.4 context endpoint
Finally (1 day): M3.7.6 composition gate

Total M3.7: ~5 days (1 done, 4 remaining)

---

Team Notes

• Determinism is critical: Both M3.7.7 and M3.7.8 must hash identically across runs

  • Use sorted keywords (not insertion order)
  • Include tool name in identity
  • Strip all volatile data (timestamps, addresses, etc.)

• Tier 1 only fires if M3.7.8 sym_sha matches M3.7.7 sig_sha

  • If query is ambiguous or doesn't match any signature → skip to tier 2
  • Fall back to hybrid search (tier 2) gracefully

• M8.2 is a blocker for M3.7.4

  • Hybrid search must be live before context endpoint can work
  • (Tier 2 fallback requires functional search)

---

End of M3.7 summary.