poimen-memory/docs/M3.7.8-SYMPTOM_PROJECTION.md

# M3.7.8 — Symptom Projection at Ingest

**Status**: Design · Ready to implement  
**Size**: M (1–2 days)  
**Depends**: M3.7.7 (signature extraction)  
**Blocks**: M3.7.4 (context endpoint), M3.7.6 (gate)

---

## Goal

Transform incoming user queries and agent failure reports into normalized symptom vectors that can be matched against extracted failure signatures from M3.7.7, enabling tier 1 (exact match) lookups in the three-tier context endpoint (M3.7.4).

---

## The Problem

**M3.7.7 extracts** failure signatures from logs and produces deterministic hashes:
```
Log:       npm ERR! code ERESOLVE unable to resolve dependency tree
Signature: npm_ERR_ERESOLVE_dependency_tree (normalized)
sig_sha:   abc123def... (deterministic)
```

**M3.7.8 must handle** user queries that don't match the log format:
```
User query: "npm can't find module tslib"
Expected:   "npm error module not found"
Problem:    These don't normalize to the same string
```

**Solution:** Symptom projection normalizes BOTH:
- Extracted signature ← M3.7.7 (log normalization)
- User query ← M3.7.8 (query normalization)
- If both normalize to the same sym_sha → tier 1 hit ✅

---

## Design

### Three-Stage Normalization Pipeline

#### Stage 1: Extract Symptom Keywords

**Goal**: Pull actionable error signals from free text.

```rust
Input query: "npm error: unable to resolve typescript dependency tree"

Step 1a: Detect tool
  → "npm" (from query context or user param)

Step 1b: Identify error patterns
  • Keywords: ["unable", "resolve", "typescript", "dependency", "tree"]
  • Error phrases: ["unable to resolve", "dependency tree"]

Step 1c: Expand abbreviations
  • "ERESOLVE" → ["error", "resolve"]
  • "ERR" → ["error"]
  • "cannot" → "can not" (splits)
  • "can't" → "cannot" (normalizes)

Step 1d: Extract entity types
  • Module/package names (typescript)
  • Error codes (E400, EACCES)
  • Version specifiers (^1.0, 2.1.3)
  
Output: SymptomTokens {
  tool: "npm",
  keywords: ["unable", "resolve", "typescript", "dependency", "tree"],
  error_codes: [],
  modules: ["typescript"]
}
```

#### Stage 2: Normalize to Canonical Form

**Goal**: Create identical strings from variant phrasings.

```rust
Input: SymptomTokens {
  tool: "npm",
  keywords: ["unable", "resolve", "typescript", "dependency", "tree"],
  error_codes: [],
  modules: ["typescript"]
}

Step 2a: Remove stop words
  Remove: [a, an, the, is, are, be, can, may, to, in, of, ...]
  Remaining: ["unable", "resolve", "typescript", "dependency", "tree"]

Step 2b: Normalize case
  → ["unable", "resolve", "typescript", "dependency", "tree"]

Step 2c: Apply stemming (if needed)
  resolve ← resolved, resolving, resolution
  depend ← dependency, dependent

Step 2d: Expand tool-specific shorthand
  For npm:
    • ERESOLVE → "error resolve"
    • ENOENT → "not found"
    • EACCES → "permission denied"
  For cargo:
    • "error[E0599]" → "error 0599"
  For kubectl:
    • "connection refused" → "connection refused"

Step 2e: Dedup and sort (for idempotence)
  Before: ["dependency", "error", "npm", "resolve", "tree"]
  After:  ["dependency", "error", "npm", "resolve", "tree"]

Output: "dependency error npm resolve tree"
```

#### Stage 3: Generate Deterministic Hash

**Goal**: Create sym_sha that matches M3.7.7 signatures.

```rust
Input: "dependency error npm resolve tree" (sorted, deduplicated)

Step 3a: Construct hashable string
  → "npm\ndependency error npm resolve tree"
             ↑
       tool is part of identity
       (npm ERR vs cargo error are different)

Step 3b: Hash with SHA256
  sym_sha = SHA256("npm\ndependency error npm resolve tree")
          = "xyz789abc123..."

Output: SymptomVector {
  tool: "npm",
  raw_query: "npm error: unable to resolve typescript dependency tree",
  normalised: "dependency error npm resolve tree",
  sym_sha: "xyz789abc123...",
  keywords: ["dependency", "error", "npm", "resolve", "tree"],
  confidence: 0.85  // based on keyword match strength
}
```

---

## Implementation

### Core Functions

#### `project_symptom(tool: &str, query: &str) -> SymptomVector`

**Purpose**: Transform user query into normalized symptom vector.

```rust
pub fn project_symptom(tool: &str, query: &str) -> SymptomVector {
    // Stage 1: Extract keywords
    let tokens = extract_keywords(tool, query);
    
    // Stage 2: Normalize
    let normalised = normalize_tokens(&tokens);
    
    // Stage 3: Hash
    let sym_sha = sha256(&format!("{}\n{}", tool, normalised));
    
    SymptomVector {
        tool: tool.to_string(),
        raw_query: query.to_string(),
        normalised,
        sym_sha,
        keywords: tokens.keywords,
        confidence: tokens.confidence,
    }
}
```

#### `extract_keywords(tool: &str, query: &str) -> SymptomTokens`

**Goal**: Pull error signals from query text.

```rust
fn extract_keywords(tool: &str, query: &str) -> SymptomTokens {
    let mut keywords = Vec::new();
    let mut error_codes = Vec::new();
    let mut modules = Vec::new();
    
    // Tokenize by whitespace + punctuation
    let tokens = tokenize(query);
    
    for token in tokens {
        // Skip stop words
        if STOP_WORDS.contains(&token.to_lowercase()) {
            continue;
        }
        
        // Expand abbreviations
        let expanded = expand_abbrev(tool, &token);
        for word in expanded.split_whitespace() {
            if !word.is_empty() {
                keywords.push(word.to_lowercase());
            }
        }
        
        // Extract structured signals
        if let Some(code) = extract_error_code(&token) {
            error_codes.push(code);
        }
        if let Some(module) = extract_module_name(tool, &token) {
            modules.push(module);
        }
    }
    
    // Dedup + sort for idempotence
    keywords.sort();
    keywords.dedup();
    
    SymptomTokens {
        keywords,
        error_codes,
        modules,
        confidence: keyword_strength(&keywords),
    }
}
```

#### `normalize_tokens(tokens: &SymptomTokens) -> String`

```rust
fn normalize_tokens(tokens: &SymptomTokens) -> String {
    let mut normalized = tokens.keywords.clone();
    
    // Remove duplicates (again, for safety)
    normalized.sort();
    normalized.dedup();
    
    // Join with spaces
    normalized.join(" ")
}
```

#### `expand_abbrev(tool: &str, word: &str) -> String`

**Per-tool abbreviation mappings:**

```rust
fn expand_abbrev(tool: &str, word: &str) -> String {
    let lower = word.to_lowercase();
    
    // Universal abbreviations
    match lower.as_str() {
        "err" | "error" => return "error".to_string(),
        "rc" | "return" => return "return code".to_string(),
        "oom" => return "out of memory".to_string(),
        "enoent" => return "not found".to_string(),
        "eacces" => return "permission denied".to_string(),
        "eperm" => return "operation not permitted".to_string(),
        "econnrefused" => return "connection refused".to_string(),
        "econnreset" => return "connection reset".to_string(),
        _ => {}
    }
    
    // Tool-specific abbreviations
    match tool {
        "npm" => match lower.as_str() {
            "eresolve" => "error resolve",
            "eexist" => "already exists",
            _ => word,
        },
        "cargo" => match lower.as_str() {
            "e0599" => "error 0599 no method",
            "e0308" => "error 0308 type mismatch",
            _ => word,
        },
        "kubectl" => match lower.as_str() {
            "crd" => "custom resource definition",
            "etcd" => "etcd",
            _ => word,
        },
        _ => word,
    }
    .to_string()
}
```

#### `sha256(s: &str) -> String`

```rust
use sha2::{Digest, Sha256};

fn sha256(s: &str) -> String {
    let mut hasher = Sha256::new();
    hasher.update(s.as_bytes());
    hex::encode(hasher.finalize())
}
```

---

## Data Structures

### SymptomVector

```rust
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SymptomVector {
    /// Tool name (npm, cargo, kubectl, etc.)
    pub tool: String,
    
    /// Original user query (for display)
    pub raw_query: String,
    
    /// Normalized, sorted keywords
    pub normalised: String,
    
    /// Deterministic hash (for lookup)
    pub sym_sha: String,
    
    /// Extracted keywords
    pub keywords: Vec<String>,
    
    /// Confidence score (0.0-1.0) based on keyword coverage
    pub confidence: f32,
}

impl SymptomVector {
    /// Check if this symptom matches a signature
    pub fn matches_signature(&self, sig: &Signature) -> bool {
        self.sym_sha == sig.sig_sha && self.tool == sig.tool
    }
}
```

### SymptomTokens (internal)

```rust
struct SymptomTokens {
    keywords: Vec<String>,
    error_codes: Vec<String>,
    modules: Vec<String>,
    confidence: f32,
}
```

---

## Stop Words

```rust
const STOP_WORDS: &[&str] = &[
    // Articles
    "a", "an", "the",
    
    // Common verbs
    "is", "are", "be", "been", "being",
    "have", "has", "had",
    "do", "does", "did",
    "can", "could", "may", "might", "must", "shall", "should", "will", "would",
    
    // Prepositions
    "in", "on", "at", "to", "from", "of", "for", "by", "with", "about",
    
    // Conjunctions
    "and", "or", "but", "nor", "yet", "so",
    
    // Pronouns
    "i", "you", "he", "she", "it", "we", "they",
    
    // Common words
    "not", "no", "yes", "what", "which", "who", "when", "where", "why", "how",
    
    // Extra
    "this", "that", "these", "those", "there", "here",
];
```

---

## Tests (6 assertions)

### `tests/it_symptom_projection.rs`

```rust
#[test]
fn a1_same_symptom_same_hash() {
    // Multiple query formulations normalize to same sym_sha
    let q1 = "npm ERR! ERESOLVE unable to resolve typescript";
    let q2 = "npm error: eresolve dependency tree";
    let q3 = "cannot resolve typescript (npm)";
    
    let sym1 = project_symptom("npm", q1);
    let sym2 = project_symptom("npm", q2);
    let sym3 = project_symptom("npm", q3);
    
    assert_eq!(sym1.sym_sha, sym2.sym_sha);
    assert_eq!(sym2.sym_sha, sym3.sym_sha);
}

#[test]
fn a2_abbrev_expansion() {
    // ERESOLVE, ERR, OOM, EACCES expand correctly
    let cases = vec![
        ("npm", "ERESOLVE", "error resolve"),
        ("npm", "ERR", "error"),
        ("cargo", "E0599", "error 0599"),
        ("kubectl", "CRD", "custom resource definition"),
    ];
    
    for (tool, abbrev, expected) in cases {
        let expanded = expand_abbrev(tool, abbrev);
        assert!(expanded.contains(&expected.replace(" ", "")));
    }
}

#[test]
fn a3_stop_word_removal() {
    // "unable to resolve the dependency tree" → "resolve dependency tree"
    let query = "npm is unable to resolve the typescript dependency tree";
    let sym = project_symptom("npm", query);
    
    // Should NOT contain stop words
    for stop in STOP_WORDS {
        assert!(!sym.normalised.contains(stop), 
                 "Stop word '{}' should be removed", stop);
    }
    
    // Should contain key terms
    assert!(sym.normalised.contains("resolve"));
    assert!(sym.normalised.contains("dependency"));
    assert!(sym.normalised.contains("typescript"));
}

#[test]
fn a4_tool_consistency() {
    // Same error text under different tools = different hashes
    let query = "error 1234 module not found";
    
    let sym_npm = project_symptom("npm", query);
    let sym_cargo = project_symptom("cargo", query);
    
    assert_ne!(sym_npm.sym_sha, sym_cargo.sym_sha,
               "Tool must be part of signature identity");
}

#[test]
fn a5_case_insensitive() {
    // "NPM ERROR" = "npm error"
    let q1 = project_symptom("npm", "NPM ERROR ERESOLVE");
    let q2 = project_symptom("npm", "npm error eresolve");
    
    assert_eq!(q1.sym_sha, q2.sym_sha);
}

#[test]
fn a6_keyword_order_irrelevant() {
    // "error npm resolve" = "npm resolve error" (after sorting)
    let q1 = project_symptom("npm", "error npm resolve typescript");
    let q2 = project_symptom("npm", "npm typescript resolve error");
    
    assert_eq!(q1.sym_sha, q2.sym_sha);
}
```

---

## Integration with M3.7.4 Context Endpoint

### Tier 1 (Exact Match) Lookup

```rust
// In M3.7.4: context_endpoint()
pub async fn handle_context_query(
    query: &str,
    tool: Option<&str>,
) -> ContextResult {
    // Step 1: Project symptom
    let symptom = project_symptom(tool.unwrap_or("unknown"), query);
    
    // Step 2: Check tier 1 (exact signature match)
    if let Some(lesson) = find_lesson_by_sig_sha(&symptom.sym_sha) {
        return ContextResult {
            tier: 1,
            confidence: "high",
            source: "lesson",
            result: lesson,
        };
    }
    
    // Step 3: Fall back to tier 2 (hybrid search)
    let hybrid_results = hybrid_search(query, tool).await?;
    
    return ContextResult {
        tier: 2,
        confidence: "medium",
        source: "hybrid_search",
        results: hybrid_results,
    };
}
```

---

## Verification Criteria

✅ Same query variant → same sym_sha  
✅ Different queries → different sym_sha  
✅ Abbreviations expand correctly per tool  
✅ Stop words removed consistently  
✅ Tool name included in hash identity  
✅ Case insensitive normalization  
✅ Keyword order irrelevant (sorted before hash)  

---

## Files to Create

| File | Lines | Purpose |
|------|-------|---------|
| `crates/mem-core/src/symptom_projection.rs` | 250 | Core normalization logic |
| `tests/it_symptom_projection.rs` | 400 | 6 integration test assertions |
| `fixtures/symptoms/` | 3 per tool | Test query examples + expected output |

---

## Dependencies

- ✅ M3.7.7 (Signature extraction) — provides `Signature` struct and `sig_sha`
- ⏳ M3.7.4 (Context endpoint) — will call `project_symptom()`
- ⏳ M8.2 (Dual-write) — needed for tier 2 fallback (hybrid search)

---

## Success Criteria

1. **18 unit tests passing** (mem-core lesson extraction still green)
2. **6 symptom projection integration tests passing**
3. **Symptom vectors deterministic** (run twice = same sym_sha)
4. **Integrated with M3.7.4 context endpoint**
5. **Tier 1 lookups work** (exact signature match via symptom projection)

---

## Timeline

**Estimated: 1–2 days**

- Day 1: Implement `symptom_projection.rs` + test fixtures
- Day 1.5: Integration tests + M3.7.4 endpoint integration
- Day 2: Verify tier 1 (exact match) lookups work end-to-end

---

## References

- M3.7.7 (Signature extraction): `crates/mem-core/src/lesson.rs` (871 LOC)
- M3.7.4 (Context endpoint): `crates/mem-cli/src/context_endpoint.rs` (TBD)
- M8 (Hybrid search): `memory-flow.md` § "Search Flow"