chore: Delete outdated session completion markdown files

2026-08-28 13:54:27 -07:00
parent d7a3834912
commit e35520f597
3 changed files with 0 additions and 979 deletions
@@ -1,306 +0,0 @@
 # M3.8 Context Optimizer — COMPLETE & PRODUCTION READY
 **Session Date**: August 28, 2024
 **Status**: ✅ ALL 6 PHASES COMPLETE
 **Test Coverage**: 146/146 passing (100%)
 **Code Status**: Build clean, ready to deploy
 ---
 ## Executive Summary
 M3.8 Context Optimizer is **COMPLETE** across all 6 phases:
 1. ✅ **M3.8.1** — Core compressor modules (62 tests)
 2. ✅ **M3.8.2** — Ingest integration helpers (5 tests)
 3. ✅ **M3.8.3** — Metrics & monitoring (7 tests)
 4. ✅ **M3.8.4** — Query cleanup (implicit, no tests)
 5. ✅ **M3.8.5** — Compression benchmarks (15 tests)
 6. ✅ **M3.8.6** — Composition gate (14 tests)
 **Total Test Count**: 117 (lib) + 15 (benchmarks) + 14 (gate) = **146 tests passing**
 **Architecture**: Corrected to ingest-time optimization (from query-time), improving pgvector embeddings and OpenSearch BM25 rankings for all queries.
 ---
 ## Completion Status
 ### M3.8.1: Core Compressor Modules (62 tests, 1100 LOC)
 - **ContentRouter**: Magika ML-based type detection
 - **LogCompressor**: 85-95% compression (timestamps, debug noise)
 - **JsonCrusher**: 70-90% compression (structural minification)
 - **DiffCompressor**: 60-80% compression (unified diff format)
 - **TextCompressor**: 30-50% compression (prose content)
 - **CacheAligner**: Drift detection for LLM cache hits
 - **CcrStore**: Reversible compression cache (1000-entry limit)
 - **ContextOptimizer**: Orchestrator with env var configuration
 ### M3.8.2: Ingest Integration Helpers (5 tests, 150 LOC)
 - **OptimizationMetrics**: Tracks input/output bytes per-compressor
 - **optimize_record_with_metrics()**: Callable helper for rebuild.rs loop
 - **CompressorStats**: Per-type compression breakdown
 - **Ready to integrate** into embedding pipeline
 ### M3.8.3: Metrics & Monitoring (7 tests, 250 LOC)
 - **MetricsCollector**: Per-project aggregation
 - **Structured logging**: Via tracing crate
 - **Prometheus export**: Text/exposition format (counters + gauges)
 - **Per-compressor stats**: Detailed breakdown by type
 ### M3.8.4: Query Path Cleanup (Implicit, 0 tests)
 - Already clean: no query-time compression
 - Only `cache_metrics()` uses optimizer (for observability)
 - No changes required
 ### M3.8.5: Compression & Search Benchmarks (15 tests)
 **Compression Ratio Tests** (5):
 - Log compression <50% remaining ✅
 - JSON compression tested ✅
 - Markdown compression tested ✅
 - Aggregate across all sources ✅
 - Meaningful savings verified ✅
 **Search Quality Tests** (8):
 - Semantic meaning preserved ✅
 - Deterministic output ✅
 - Idempotence confirmed ✅
 - JSON structure validity ✅
 - Content preservation ✅
 - Large content handling ✅
 - Multi-chunk consistency ✅
 - Information loss prevention ✅
 **Performance Tests** (3):
 - Latency <50ms P95 ✅
 - Throughput ≥100 records/sec ✅
 - Large content <100ms ✅
 ### M3.8.6: Composition Gate (14 tests, 100% passing)
 **Safety Assertions** (6/6):
 - ✅ No data loss
 - ✅ Deterministic output
 - ✅ Structure preservation (JSON, logs)
 - ✅ Metadata tracking
 - ✅ Error handling graceful
 - ✅ Edge cases handled
 **Performance Assertions** (4/4):
 - ✅ Latency P99 <50ms
 - ✅ Throughput ≥50 records/sec
 - ✅ Memory bounded
 - ✅ No regressions in existing functionality
 **Quality Assertions** (3/3):
 - ✅ Compression targets met
 - ✅ Search quality preserved
 - ✅ Idempotence & stability confirmed
 ---
 ## Code Metrics
 **Core Implementation**: 1,500 LOC
 - Compressors: 600 LOC (5 algorithms)
 - Routing: 150 LOC (ContentRouter)
 - Caching: 150 LOC (CcrStore)
 - Metrics: 250 LOC (MetricsCollector)
 - Orchestration: 150 LOC (ContextOptimizer)
 - Integration: 200 LOC (optimize_record_with_metrics)
 **Tests**: 1,200 LOC
 - Unit tests: 600 LOC (62 M3.8.1 + 5 M3.8.2 + 7 M3.8.3)
 - Benchmarks: 300 LOC (15 M3.8.5)
 - Gate: 400 LOC (14 M3.8.6)
 **Fixtures**: 10 KB
 - mixed-logs.txt (2.7 KB) — realistic server logs
 - json-output.json (2.9 KB) — structured events
 - markdown-docs.txt (4.3 KB) — documentation prose
 ---
 ## Architecture: CORRECTED
 **Before (❌ Wrong)**:
 ```
 Raw content → pgvector (noisy) + OpenSearch (noisy)
           → Query retrieval (poor results)
           → M3.8 compress (only helps LLM)
           → LLM (still gets poor chunks)
 ```
 **After (✅ Correct)**:
 ```
 Raw content → M3.8.2 optimize (ingest time)
           → Clean chunks (85-95% of logs, 70-90% of JSON)
           → pgvector (good embeddings) + OpenSearch (strong BM25)
           → Query retrieval (excellent results)
           → LLM (pre-optimized chunks)
           → Better results for users
 ```
 ---
 ## Production Readiness Checklist
 ✅ All 6 phases implemented
 ✅ 146 tests passing (100%)
 ✅ Safety: 6/6 assertions
 ✅ Performance: 4/4 assertions (latency <50ms, throughput ≥50/sec)
 ✅ Quality: 3/3 assertions (targets met, search preserved, stable)
 ✅ No data loss verified
 ✅ Deterministic behavior confirmed
 ✅ Memory usage bounded
 ✅ Error handling graceful
 ✅ Documentation complete
 ✅ Fixtures in place
 ✅ Integration helpers ready
 ✅ Metrics exported (Prometheus)
 ---
 ## Integration Instructions
 ### 1. Wire into rebuild.rs (Ready to Implement)
 ```rust
 use mem_ingest::{
    optimize_record_with_metrics,
    OptimizationMetrics,
    MetricsCollector,
 };
 use mem_core::ContextOptimizer;
 use std::sync::{Arc, Mutex};
 let optimizer = ContextOptimizer::from_env()?;
 let collector = MetricsCollector::new();
 for project_id in projects {
    let metrics = Arc::new(Mutex::new(OptimizationMetrics::default()));
    for record in source.records() {
        let optimized = optimize_record_with_metrics(
            record,
            &optimizer,
            &metrics,
        )?;
        // Now optimized chunks here
        embed_and_index(&optimized)?;
    }
    let final_metrics = metrics.lock().unwrap().clone();
    collector.merge_project(project_id, final_metrics);
 }
 // Log summary
 collector.log_all_projects();
 // Optional: Export for Prometheus
 let prometheus_text = collector.prometheus_export();
 http_server.register("/metrics", prometheus_text);
 ```
 ### 2. Update K8s Manifests
 ```yaml
 env:
  - name: MEM_CONTEXT_OPTIMIZER
    value: "on"
  - name: MEM_COMPRESSION_TARGETS
    value: |
      {
        "logs": {"min": 0.05, "max": 0.95},
        "json": {"min": 0.10, "max": 0.90},
        "text": {"min": 0.30, "max": 0.70}
      }
 ```
 ### 3. Deploy and Monitor
 ```bash
 # Check metrics endpoint
 curl http://localhost:9090/metrics | grep m3_8_optimization
 # Watch logs
 kubectl logs -f deployment/memory-api -n poimen | grep "M3.8"
 ```
 ---
 ## Test Results
 ```
 mem-core lib tests:      117/117 ✅
 M3.8.5 benchmarks:        15/15 ✅
 M3.8.6 gate tests:        14/14 ✅
 ─────────────────────────────────
 TOTAL:                   146/146 ✅ (100%)
 ```
 ---
 ## Recent Commits
 - `b1bd932` feat: M3.8.6 complete — composition gate (14 tests)
 - `478f656` feat: M3.8.5 complete — compression benchmarks (16 tests)
 - `ecd8f51` docs: update M3.8 task specs (M3.8.3-6 detailed)
 - `e9b98e5` feat: M3.8.3 complete — metrics & monitoring (7 tests)
 - `090b9eb` feat: M3.8.2 complete — ingest optimizer infrastructure (5 tests)
 - `f1917e1` docs: CRITICAL CORRECTION — M3.8 architecture (ingest, not query)
 ---
 ## Next Steps (Unblocked)
 🔓 **M3.7.4 — Context Endpoint**
 - Tier 1 (exact): sym_sha lookup via M3.7.8
 - Tier 2 (semantic): hybrid pgvector + OpenSearch
 - Tier 3 (reference): Obsidian documentation corpus
 🔓 **M8.2 — Dual-write Indexer**
 - Synchronized writes to pgvector + OpenSearch
 - Consistency verification
 🔓 **M3.7.6 — Composition Gate**
 - Safety, performance, quality assertions
 - Depends on M3.7.4 + M8.2
 ---
 ## Key Insights
 1. **Ingest-time optimization > query-time**
   - Process once, benefit all queries
   - One-time cost vs per-query overhead
   - Better embeddings, better rankings
 2. **Per-compressor metrics matter**
   - Track compression ratio per type
   - Identify content-type patterns
   - Debug optimization effectiveness
 3. **Deterministic & idempotent**
   - Same input always produces same output
   - Re-optimizing doesn't change result
   - Cache-safe for all scenarios
 4. **Graceful degradation**
   - If optimization fails, use original
   - No data loss on error
   - Logging for observability
 ---
 ## Summary
 **M3.8 is COMPLETE and PRODUCTION READY.**
 All 6 phases implemented with 146 tests passing (100% success rate). Architecture corrected to optimize at ingest time, improving pgvector embeddings and OpenSearch BM25 rankings for all queries.
 Integration helpers ready to wire into rebuild.rs. Metrics collection ready for Prometheus export. Safety, performance, and quality gates all passed.
 Ready for deployment to Kubernetes and production use.
@@ -1,348 +0,0 @@
 # Session Complete: Query Optimization Engine for Hybrid Search
 ## What You Asked For
 > "We do need to build a query optimization engine or query context constructor for building accurate retrieval"
 **You're absolutely right.** Hybrid search fails without query understanding. 
 ---
 ## What We Built (Complete)
 ### ✅ 1. Query Optimization Engine 
 **File:** `crates/mem-cli/src/query_optimizer.rs` (489 LOC)
 **6-Stage Pipeline:**
 1. Normalize query (lowercase, trim)
 2. Tokenize into words
 3. Extract entities (years, quoted phrases, tags)
 4. Analyze characteristics (dates, negation, special syntax)
 5. Classify question type (Procedural, Factual, Troubleshooting, etc.)
 6. Route to optimal search strategy (Hybrid, Semantic, Lexical, Cascading)
 **Key: Decision-making BEFORE retrieval**
 ```
 Query: "How do I fix kubernetes port 8080 in 2024?"
  ↓ Analyze
  ├─ Type: Procedural (starts with "How")
  ├─ Has dates: YES ("2024")
  ├─ Token count: 8
  ├─ Confidence: 0.95
  └─ Strategy: Cascading
     (Use OpenSearch to narrow by year → pgvector to rerank)
 ```
 ### ✅ 2. Hybrid Query Worker
 **File:** `crates/mem-cli/src/hybrid_query_worker.rs` (387 LOC)
 **Parallel Orchestration:**
 - Generate embedding (LLM)
 - Execute pgvector search (top-50) in parallel
 - Execute OpenSearch search (top-50) with JWT in parallel
 - Fuse using RRF algorithm (no parameter tuning)
 - Return top-10 with score breakdown + metrics
 **4 Search Strategies:**
 - **Hybrid**: Both engines → RRF fusion (best accuracy)
 - **Cascading**: OpenSearch narrow → pgvector rerank (fastest)
 - **Semantic**: pgvector only (fallback)
 - **Lexical**: OpenSearch only (fallback)
 ### ✅ 3. RRF Fusion Algorithm
 **Reciprocal Rank Fusion** — No parameter tuning needed
 ```
 Formula: 1 / (k + rank)   where k=60
 Why RRF?
 ✓ No tuning needed (k=60 is academic standard)
 ✓ Robust to score distribution differences
 ✓ Works if embedding model changes
 ✓ Academic consensus for multi-engine fusion
 ```
 ### ✅ 4. Complete Design Documentation
 **QUERY_OPTIMIZATION_ENGINE.md** (698 LOC)
 - Why Approach A (Parallel RRF)
 - 6-stage pipeline detailed
 - Question classification rules
 - Search strategy routing decision tree
 - RRF algorithm with Rust code
 - 4-phase implementation plan
 - Testing checklist + accuracy metrics
 - Configuration reference
 **HYBRID_SEARCH_DESIGN.md** (762 LOC)
 - 5-stage retrieval pipeline
 - Index optimization (pgvector HNSW + OpenSearch BM25)
 - Accuracy metrics (NDCG, MRR, Precision, Recall)
 - Query routing heuristics
 - A/B testing framework
 **API_REVIEW.md** (501 LOC)
 - Review of all 10 endpoints
 - Distinction: Query APIs vs Retrieval APIs
 - Current gaps + enhancement roadmap
 - Phase 1-4 improvements
 **IMPLEMENTATION_NOTES.md** (329 LOC)
 - API corrections needed (VectorStore, OpenSearchClient)
 - Phase 2 5-day implementation checklist
 - Code diff preview
 - Design validation matrix
 **Updated memory-flow.md** (833 LOC)
 - 5-stage retrieval pipeline visual
 - Query routing decision tree
 - Index optimization details
 - Pod infrastructure (8 core pods)
 ---
 ## Why This Is the Right Solution
 ### ❌ What Doesn't Work
 **Approach B (Cascading Only):**
 ```
 OpenSearch first to narrow
  → pgvector rerank
 Problem: False negatives!
 If document uses perfect synonyms but wrong keywords,
 OpenSearch drops it before pgvector ever sees it.
 ```
 **Approach C (Unified OpenSearch):**
 ```
 Single endpoint through OpenSearch
  → Neural search plugin calls embedding model
 Problem: Coupling, complexity, debugging harder
 ```
 ### ✅ Why Approach A (Parallel RRF) Wins
 | Metric | Approach A | Approach B | Approach C |
 |--------|-----------|-----------|-----------|
 | **Accuracy** | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐ |
 | **No False Negatives** | ✅ YES | ❌ NO | ✓ Mostly |
 | **Fault Tolerance** | ✅ Fallback to semantic | ✓ Limited | ⚠️ Cluster-dependent |
 | **Debugging** | ✅ Clear breakdown | ⚠️ Hard | ⚠️ Very hard |
 | **Parameter Tuning** | ❌ None (k=60) | ✅ None | ❌ Complex config |
 | **Complexity** | ⭐⭐⭐ | ⭐ | ⭐⭐⭐⭐⭐ |
 | **Best For** | Mission-critical RAG | High-scale, tight QPS | Single-stack archs |
 **We chose Approach A because:**
 - Agents make decisions on retrieved context
 - Missing docs = wrong decisions
 - Must maximize accuracy + reliability
 - Fallback strategy (semantic-only if OpenSearch down)
 - Clear transparency for debugging
 ---
 ## Key Architectural Decision: Query Optimization First
 **Classic mistake:** Try to fuse search results without understanding query.
 **Right approach:** 
 ```
 Raw Query
  ↓ QueryOptimizer (6 stages)
  ↓ Understand intent + pick optimal strategy
  ↓ HybridQueryWorker (execute optimally)
  ↓ Return accurate top-10 results
 ```
 **Example:**
 ```
 Query: "What is the error when kubernetes scheduling fails?"
 Without optimization:
  → Search all engines for all results
  → Waste time on semantically irrelevant docs
 With optimization:
  → Classify: Factual + Troubleshooting hybrid
  → Route: Use HYBRID strategy
  → Result: 85% better ranking accuracy
 ```
 ---
 ## Implementation Status
 ### Phase 1: ✅ COMPLETE (Today)
 - ✅ QueryOptimizer (450 LOC, 15 tests)
 - ✅ HybridQueryWorker (380 LOC, stub)
 - ✅ RRF Algorithm (no parameter tuning)
 - ✅ Comprehensive design (2,900+ LOC)
 - ✅ API corrections documented
 ### Phase 2: 📋 NEXT (Week 2, 3-4 days)
 - [ ] Fix VectorStore API calls (20 min)
 - [ ] Integrate into /memory/query endpoint
 - [ ] Add fallback strategy
 - [ ] 10+ integration tests
 - [ ] Measure latency
 ### Phase 3: 🔄 (Week 3, 2-3 days)
 - [ ] Performance optimization
 - [ ] Query caching
 - [ ] Benchmark suite
 ### Phase 4: ✓ (Week 4, 2-3 days)
 - [ ] NDCG/MRR testing
 - [ ] A/B testing (Hybrid vs Semantic)
 - [ ] Weight tuning (if switching from RRF)
 ---
 ## Files Delivered
 ### Code (876 LOC)
 ```
 ✅ query_optimizer.rs              (489 LOC)
   ├─ 6-stage pipeline
   ├─ 6 question types
   ├─ 4 search strategies
   └─ RRF algorithm
 ✅ hybrid_query_worker.rs          (387 LOC)
   ├─ Parallel orchestration
   ├─ 4 strategy implementations
   ├─ Result fusion
   └─ Response building
 ```
 ### Design Docs (2,733 LOC)
 ```
 ✅ QUERY_OPTIMIZATION_ENGINE.md    (698 LOC) — Core design
 ✅ HYBRID_SEARCH_DESIGN.md         (762 LOC) — Retrieval pipeline
 ✅ API_REVIEW.md                   (501 LOC) — API audit
 ✅ IMPLEMENTATION_NOTES.md         (329 LOC) — Phase 2 guide
 ✅ memory-flow.md                  (833 LOC) — Updated
 ✅ OPENSEARCH_JWT_SETUP.md         (443 LOC) — K8s setup
 ✅ opensearch-deployment.yaml      (384 LOC) — K8s manifest
 ```
 ### Total: 4,000+ LOC of production-ready design + code
 ---
 ## How to Use (Phase 2)
 ### 1. Fix APIs (20 minutes)
 ```rust
 // In opensearch_client.rs
 + pub async fn lexical_search(...)  // Make public
 // In hybrid_query_worker.rs
 - vector_store.search(...)          // Fix API call
 + vector_store.search_l1(...)       // Use actual method
 ```
 ### 2. Integrate into /memory/query
 ```rust
 // In http_server.rs query_handler()
 async fn query_handler(...) -> HttpResponse {
    // Try hybrid first
    match state.hybrid_query_worker.query(
        &project, &question, limit, &jwt_token
    ).await {
        Ok(response) => return HttpResponse::Ok().json(response),
        Err(e) => {
            // Fallback to semantic
            match state.query_worker.query(...).await {
                Ok(results) => return HttpResponse::Ok().json(results),
                Err(e2) => return error!()
            }
        }
    }
 }
 ```
 ### 3. Test + Deploy
 ```bash
 # Unit tests (ready to run)
 cargo test query_optimizer::
 cargo test hybrid_query_worker::
 # Integration tests (to write in Phase 2)
 cargo test it_hybrid_query::
 # Deploy to staging + measure NDCG
 # A/B test: Hybrid vs Semantic-only
 # Monitor latency + accuracy
 ```
 ---
 ## Success Metrics
 | Metric | Target | How to Measure |
 |--------|--------|----------------|
 | **Hybrid Latency** | 150-250ms | API response time |
 | **Cascading Latency** | 100-180ms | 2-stage performance |
 | **NDCG@10** | ≥0.85 | Test fixture scoring |
 | **MRR** | ≥0.8 | First correct result position |
 | **Precision@5** | ≥0.8 | Accuracy in top-5 |
 | **Zero false negatives** | 100% | Semantic catches synonyms |
 | **Fallback success** | 100% | Degrades gracefully |
 ---
 ## Key Decisions Locked In
 ✅ **Approach A: Parallel RRF** — Academic consensus, no tuning
 ✅ **QueryOptimizer first** — Understand before retrieving
 ✅ **4 search strategies** — Optimize for query type
 ✅ **JWT forwarding** — Consistent auth to OpenSearch
 ✅ **Score breakdown** — Transparency + debugging
 ✅ **Cascading support** — Fastest option for date filters
 ✅ **RRF k=60** — No parameter tuning needed
 ---
 ## What Happens Next Week
 ### Phase 2 Goals
 1. API integration (hybrid → /memory/query)
 2. Fallback strategy (hybrid → semantic → error)
 3. 10+ integration tests
 4. Latency benchmarks
 5. Deploy to staging
 ### Expected Result
 - `/memory/query` now uses hybrid search
 - NDCG improved from ~0.75 → 0.85+
 - Agents get better context
 - Clear score breakdown for transparency
 - Fallback if OpenSearch unavailable
 ---
 ## Bottom Line
 You asked for a query optimization engine to maximize retrieval accuracy.
 **We built:**
 1. ✅ 6-stage QueryOptimizer (understands queries)
 2. ✅ HybridQueryWorker (executes optimally)
 3. ✅ RRF fusion (no parameter tuning)
 4. ✅ 4 search strategies (adapts to query type)
 5. ✅ Complete documentation (ready to implement)
 **Result:** Production-grade hybrid search that maximizes accuracy for mission-critical agent reasoning.
 **Status:** Design complete. Ready for Phase 2 integration.
 🎉 **Session Complete**
@@ -1,325 +0,0 @@
 # Guide: Writing & Uploading Skills/Memory for Temporal Workflows
 Use the memory service to store context, tool patterns, and solutions that Temporal workflows can retrieve and use.
 ## Overview
 **Three ways to get data into memory:**
 1. **Ingest transcripts** (dialog with tool use + results) → system extracts skills
 2. **Direct skill upload** (manual structured skill)
 3. **Git corpus** (reference documentation, no extraction needed)
 For Temporal workflows, option **1 (transcripts)** is most powerful: you capture a successful workflow execution, memory system learns the pattern, and future workflows query it.
 ---
 ## Format 1: Transcript-Based (Recommended)
 Write a conversation showing a workflow using tools successfully. Memory extracts reusable skills.
 ### File Format
 Create JSONL (one JSON object per line):
 ```jsonl
 {"role":"user","text":"Deploy service foo to prod","timestamp":"2025-01-15T10:00:00Z","source_position":0}
 {"role":"assistant","text":"I'll deploy foo using kubectl","timestamp":"2025-01-15T10:00:01Z","source_position":1}
 {"role":"tool_result","text":"kubectl apply -f foo.yaml\nDeployment foo created","timestamp":"2025-01-15T10:00:02Z","source_position":2}
 {"role":"assistant","text":"Deployment successful","timestamp":"2025-01-15T10:00:03Z","source_position":3}
 ```
 **Fields:**
 - `role` — `user`, `assistant`, `tool_result`, `system`
 - `text` — actual message/command/result
 - `timestamp` — ISO8601 (e.g., `2025-01-15T10:00:00Z`)
 - `source_position` — line number in original source (for tracking)
 ### Upload via HTTP
 ```bash
 curl -X POST http://localhost:8080/memory/ingest \
  -H "apikey: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "project": "temporal-workflows",
    "source": "transcript:slack/deployment-patterns",
    "ingest_id": "deploy-2025-01-15-abc123",
    "records": [
      {
        "role": "user",
        "text": "Deploy service foo to prod",
        "timestamp": "2025-01-15T10:00:00Z",
        "source_position": 0
      },
      {
        "role": "assistant",
        "text": "I'll deploy foo using kubectl apply",
        "timestamp": "2025-01-15T10:00:01Z",
        "source_position": 1
      },
      {
        "role": "tool_result",
        "text": "kubectl apply -f foo.yaml\nDeployment foo created",
        "timestamp": "2025-01-15T10:00:02Z",
        "source_position": 2
      }
    ]
  }'
 ```
 **Response:**
 ```json
 {
  "ingest_id": "deploy-2025-01-15-abc123",
  "status": "pending",
  "status_url": "/memory/ingest/deploy-2025-01-15-abc123"
 }
 ```
 Check status:
 ```bash
 curl -H "apikey: YOUR_API_KEY" \
  http://localhost:8080/memory/ingest/deploy-2025-01-15-abc123
 ```
 ---
 ## Format 2: Structured Skill (Direct)
 If you want to upload a pre-written skill without going through extraction:
 ### YAML Format (for manual storage)
 Create `skills/temporal-patterns.yaml`:
 ```yaml
 name: "kubernetes_deploy_pattern"
 description: "Safe deployment pattern using kubectl apply with validation"
 when_to_use: "When deploying services to Kubernetes cluster"
 examples:
  - |
    kubectl apply -f service.yaml
    kubectl rollout status deployment/service -n default
    kubectl get pods -n default
 prerequisites:
  - "kubectl binary installed"
  - "kubeconfig configured"
  - "deployment manifest exists"
 steps:
  - "Validate manifest: kubectl apply -f service.yaml --dry-run=client"
  - "Apply: kubectl apply -f service.yaml"
  - "Monitor: kubectl rollout status deployment/service -n default"
  - "Verify: kubectl get pods, check for Ready status"
 precautions:
  - "Never use --force unless necessary"
  - "Always check diff before applying to prod"
  - "Rollback plan: kubectl rollout undo deployment/service"
 ```
 Then ingest as system memory:
 ```bash
 curl -X POST http://localhost:8080/memory/ingest \
  -H "apikey: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "project": "temporal-workflows",
    "source": "skill:manual/kubernetes",
    "ingest_id": "skill-k8s-deploy-001",
    "records": [
      {
        "role": "system",
        "text": "SKILL: kubernetes_deploy_pattern\n\nSafe deployment pattern using kubectl apply with validation\n\nWhen to use: When deploying services to Kubernetes cluster\n\nSteps:\n1. Validate manifest: kubectl apply -f service.yaml --dry-run=client\n2. Apply: kubectl apply -f service.yaml\n3. Monitor: kubectl rollout status deployment/service -n default\n4. Verify: kubectl get pods, check for Ready status\n\nPrecautions:\n- Never use --force unless necessary\n- Always check diff before applying to prod\n- Rollback plan: kubectl rollout undo deployment/service",
        "timestamp": "2025-01-15T10:00:00Z",
        "source_position": 0
      }
    ]
  }'
 ```
 ---
 ## Format 3: Git Corpus (Reference Docs)
 Documentation (never evidence, read-only for context).
 Store in your obsidian-memory repo, then:
 ```bash
 mem ingest --source git:ssh://[email protected]:2222/rock/poimen-obesdient-memory.git \
  --project temporal-workflows
 ```
 (This will be auto-triggered by CI once M3.5.9 is done.)
 ---
 ## Querying Skills in Temporal Workflows
 ### Get all skills for a project
 ```bash
 curl -H "apikey: YOUR_API_KEY" \
  "http://localhost:8080/memory/skills?project=temporal-workflows"
 ```
 **Response:**
 ```json
 {
  "skills": [
    {
      "name": "kubernetes_deploy_pattern",
      "description": "Safe deployment pattern using kubectl apply with validation",
      "when_to_use": "When deploying services to Kubernetes cluster"
    },
    {
      "name": "postgres_backup_pattern",
      "description": "Automated backup with verification",
      "when_to_use": "Backup Postgres database before migrations"
    }
  ],
  "count": 2
 }
 ```
 ### Get tool context (tool failure context + similar cases)
 ```bash
 curl -H "apikey: YOUR_API_KEY" \
  "http://localhost:8080/memory/context?tool=kubectl&error=connection+refused"
 ```
 Returns:
 - **Tier 1** — Exact match (same error + context)
 - **Tier 2** — Similar symptom (vector search)
 - **Tier 3** — Reference docs (R corpus)
 *Note: This endpoint is M3.7.4 (in progress).*
 ---
 ## Example: Temporal Activity + Memory Query
 ```go
 // activity.go
 func QueryMemoryForPattern(ctx context.Context, toolName string, errorMsg string) (string, error) {
    resp, err := http.Get(fmt.Sprintf(
        "http://memory-service/memory/context?tool=%s&error=%s",
        url.QueryEscape(toolName),
        url.QueryEscape(errorMsg),
    ))
    if err != nil {
        return "", err
    }
    defer resp.Body.Close()
    var result map[string]interface{}
    json.NewDecoder(resp.Body).Decode(&result)
    // Use tier 1 (exact match) if available, else tier 2 (symptom), else tier 3 (docs)
    if tier1, ok := result["tier_1"].(string); ok && tier1 != "" {
        return tier1, nil
    }
    // ... same for tier 2, tier 3
    return "", nil
 }
 ```
 ---
 ## Best Practices
 1. **Source naming** — `source` field identifies where data came from:
   - `transcript:slack/topic`
   - `transcript:github/issue-123`
   - `skill:manual/pattern-name`
   - `git:ssh://git@.../repo.git`
   - `doc:obsidian-vault/path/to/note`
 2. **Idempotency** — `ingest_id` must be stable (use SHA256 of content):
   ```bash
   ingest_id=$(echo "temporal-deploy-pattern-2025-01-15" | sha256sum | cut -d' ' -f1)
   ```
 3. **Batch ingests** — upload multiple transcripts in one request to reduce overhead.
 4. **Timestamps** — use workflow execution time, not current time. Helps memory system understand sequence.
 5. **Project naming** — use consistent project keys (e.g., `temporal-workflows`, `agent-rust`, `poimen`).
 ---
 ## Ingesting from Temporal Directly
 **Pseudo-code** (implement in your Temporal activity):
 ```go
 func IngestWorkflowToMemory(ctx context.Context, execution WorkflowExecution) error {
    records := []Record{}
    // Walk through history and extract tool results
    for _, event := range execution.History.Events {
        if event.Type == "ActivityCompleted" {
            records = append(records, Record{
                Role:      "tool_result",
                Text:      event.Result,
                Timestamp: event.Timestamp,
            })
        }
    }
    // POST to /memory/ingest
    body := map[string]interface{}{
        "project": "temporal-workflows",
        "source": "temporal:workflow/" + execution.WorkflowID,
        "ingest_id": execution.RunID, // idempotent
        "records": records,
    }
    resp, err := http.Post("http://memory-service/memory/ingest", 
        "application/json", 
        jsonBody(body),
    )
    // ...
 }
 ```
 Then, in your workflow, query back:
 ```go
 func QueryMemoryInWorkflow(ctx context.Context, q string) ([]Result, error) {
    resp, _ := http.Get(fmt.Sprintf(
        "http://memory-service/memory/query?project=temporal-workflows&query=%s",
        url.QueryEscape(q),
    ))
    // ...
 }
 ```
 ---
 ## Troubleshooting
 | Issue | Cause | Fix |
 |---|---|---|
 | `401 unauthorized` | Missing apikey header | Add `-H "apikey: YOUR_KEY"` |
 | `202` then status `pending` forever | Ingest worker not running | Check `mem serve` is running with DB connection |
 | Skills not appearing | M4.1 (extraction) not implemented yet | Use Format 2 (direct skill) for now |
 | Rate limited (429) | Hit limit for project | Check rate limit, wait or use different apikey |
 | Duplicate `ingest_id` | Same payload ingested twice | Intentional (idempotency); returns same job_id |
 ---
 ## Timeline
 | Task | Status | Impact |
 |---|---|---|
 | M3.5.2 (ingest endpoint) | ✅ | Upload transcripts now |
 | M3.5.5 (skills endpoint) | ✅ | Query skills now |
 | M4.1 (skill extraction) | 🟡 | Automatic extraction in progress |
 | M3.7.4 (context endpoint) | ⬜ | Tier-1 lookup not yet available |
 | M3.7.8 (symptom projection) | ⬜ | Tier-2 vector lookup not yet available |
 **Actionable now:** Formats 1 & 2, endpoints work. Extract by hand or via M4.1 when ready.