From e35520f59751d7eea1ecf04fadf4259a78722a80 Mon Sep 17 00:00:00 2001 From: poimen Date: Fri, 28 Aug 2026 13:54:27 -0700 Subject: [PATCH] chore: Delete outdated session completion markdown files --- CLAUDE_M3.8_COMPLETE.md | 306 ---------------------------------- SESSION_COMPLETE.md | 348 --------------------------------------- TEMPORAL_SKILLS_GUIDE.md | 325 ------------------------------------ 3 files changed, 979 deletions(-) delete mode 100644 CLAUDE_M3.8_COMPLETE.md delete mode 100644 SESSION_COMPLETE.md delete mode 100644 TEMPORAL_SKILLS_GUIDE.md diff --git a/CLAUDE_M3.8_COMPLETE.md b/CLAUDE_M3.8_COMPLETE.md deleted file mode 100644 index 9951f4c..0000000 --- a/CLAUDE_M3.8_COMPLETE.md +++ /dev/null @@ -1,306 +0,0 @@ -# M3.8 Context Optimizer — COMPLETE & PRODUCTION READY - -**Session Date**: August 28, 2024 -**Status**: ✅ ALL 6 PHASES COMPLETE -**Test Coverage**: 146/146 passing (100%) -**Code Status**: Build clean, ready to deploy - ---- - -## Executive Summary - -M3.8 Context Optimizer is **COMPLETE** across all 6 phases: - -1. ✅ **M3.8.1** — Core compressor modules (62 tests) -2. ✅ **M3.8.2** — Ingest integration helpers (5 tests) -3. ✅ **M3.8.3** — Metrics & monitoring (7 tests) -4. ✅ **M3.8.4** — Query cleanup (implicit, no tests) -5. ✅ **M3.8.5** — Compression benchmarks (15 tests) -6. ✅ **M3.8.6** — Composition gate (14 tests) - -**Total Test Count**: 117 (lib) + 15 (benchmarks) + 14 (gate) = **146 tests passing** - -**Architecture**: Corrected to ingest-time optimization (from query-time), improving pgvector embeddings and OpenSearch BM25 rankings for all queries. - ---- - -## Completion Status - -### M3.8.1: Core Compressor Modules (62 tests, 1100 LOC) -- **ContentRouter**: Magika ML-based type detection -- **LogCompressor**: 85-95% compression (timestamps, debug noise) -- **JsonCrusher**: 70-90% compression (structural minification) -- **DiffCompressor**: 60-80% compression (unified diff format) -- **TextCompressor**: 30-50% compression (prose content) -- **CacheAligner**: Drift detection for LLM cache hits -- **CcrStore**: Reversible compression cache (1000-entry limit) -- **ContextOptimizer**: Orchestrator with env var configuration - -### M3.8.2: Ingest Integration Helpers (5 tests, 150 LOC) -- **OptimizationMetrics**: Tracks input/output bytes per-compressor -- **optimize_record_with_metrics()**: Callable helper for rebuild.rs loop -- **CompressorStats**: Per-type compression breakdown -- **Ready to integrate** into embedding pipeline - -### M3.8.3: Metrics & Monitoring (7 tests, 250 LOC) -- **MetricsCollector**: Per-project aggregation -- **Structured logging**: Via tracing crate -- **Prometheus export**: Text/exposition format (counters + gauges) -- **Per-compressor stats**: Detailed breakdown by type - -### M3.8.4: Query Path Cleanup (Implicit, 0 tests) -- Already clean: no query-time compression -- Only `cache_metrics()` uses optimizer (for observability) -- No changes required - -### M3.8.5: Compression & Search Benchmarks (15 tests) - -**Compression Ratio Tests** (5): -- Log compression <50% remaining ✅ -- JSON compression tested ✅ -- Markdown compression tested ✅ -- Aggregate across all sources ✅ -- Meaningful savings verified ✅ - -**Search Quality Tests** (8): -- Semantic meaning preserved ✅ -- Deterministic output ✅ -- Idempotence confirmed ✅ -- JSON structure validity ✅ -- Content preservation ✅ -- Large content handling ✅ -- Multi-chunk consistency ✅ -- Information loss prevention ✅ - -**Performance Tests** (3): -- Latency <50ms P95 ✅ -- Throughput ≥100 records/sec ✅ -- Large content <100ms ✅ - -### M3.8.6: Composition Gate (14 tests, 100% passing) - -**Safety Assertions** (6/6): -- ✅ No data loss -- ✅ Deterministic output -- ✅ Structure preservation (JSON, logs) -- ✅ Metadata tracking -- ✅ Error handling graceful -- ✅ Edge cases handled - -**Performance Assertions** (4/4): -- ✅ Latency P99 <50ms -- ✅ Throughput ≥50 records/sec -- ✅ Memory bounded -- ✅ No regressions in existing functionality - -**Quality Assertions** (3/3): -- ✅ Compression targets met -- ✅ Search quality preserved -- ✅ Idempotence & stability confirmed - ---- - -## Code Metrics - -**Core Implementation**: 1,500 LOC -- Compressors: 600 LOC (5 algorithms) -- Routing: 150 LOC (ContentRouter) -- Caching: 150 LOC (CcrStore) -- Metrics: 250 LOC (MetricsCollector) -- Orchestration: 150 LOC (ContextOptimizer) -- Integration: 200 LOC (optimize_record_with_metrics) - -**Tests**: 1,200 LOC -- Unit tests: 600 LOC (62 M3.8.1 + 5 M3.8.2 + 7 M3.8.3) -- Benchmarks: 300 LOC (15 M3.8.5) -- Gate: 400 LOC (14 M3.8.6) - -**Fixtures**: 10 KB -- mixed-logs.txt (2.7 KB) — realistic server logs -- json-output.json (2.9 KB) — structured events -- markdown-docs.txt (4.3 KB) — documentation prose - ---- - -## Architecture: CORRECTED - -**Before (❌ Wrong)**: -``` -Raw content → pgvector (noisy) + OpenSearch (noisy) - → Query retrieval (poor results) - → M3.8 compress (only helps LLM) - → LLM (still gets poor chunks) -``` - -**After (✅ Correct)**: -``` -Raw content → M3.8.2 optimize (ingest time) - → Clean chunks (85-95% of logs, 70-90% of JSON) - → pgvector (good embeddings) + OpenSearch (strong BM25) - → Query retrieval (excellent results) - → LLM (pre-optimized chunks) - → Better results for users -``` - ---- - -## Production Readiness Checklist - -✅ All 6 phases implemented -✅ 146 tests passing (100%) -✅ Safety: 6/6 assertions -✅ Performance: 4/4 assertions (latency <50ms, throughput ≥50/sec) -✅ Quality: 3/3 assertions (targets met, search preserved, stable) -✅ No data loss verified -✅ Deterministic behavior confirmed -✅ Memory usage bounded -✅ Error handling graceful -✅ Documentation complete -✅ Fixtures in place -✅ Integration helpers ready -✅ Metrics exported (Prometheus) - ---- - -## Integration Instructions - -### 1. Wire into rebuild.rs (Ready to Implement) - -```rust -use mem_ingest::{ - optimize_record_with_metrics, - OptimizationMetrics, - MetricsCollector, -}; -use mem_core::ContextOptimizer; -use std::sync::{Arc, Mutex}; - -let optimizer = ContextOptimizer::from_env()?; -let collector = MetricsCollector::new(); - -for project_id in projects { - let metrics = Arc::new(Mutex::new(OptimizationMetrics::default())); - - for record in source.records() { - let optimized = optimize_record_with_metrics( - record, - &optimizer, - &metrics, - )?; - - // Now optimized chunks here - embed_and_index(&optimized)?; - } - - let final_metrics = metrics.lock().unwrap().clone(); - collector.merge_project(project_id, final_metrics); -} - -// Log summary -collector.log_all_projects(); - -// Optional: Export for Prometheus -let prometheus_text = collector.prometheus_export(); -http_server.register("/metrics", prometheus_text); -``` - -### 2. Update K8s Manifests - -```yaml -env: - - name: MEM_CONTEXT_OPTIMIZER - value: "on" - - name: MEM_COMPRESSION_TARGETS - value: | - { - "logs": {"min": 0.05, "max": 0.95}, - "json": {"min": 0.10, "max": 0.90}, - "text": {"min": 0.30, "max": 0.70} - } -``` - -### 3. Deploy and Monitor - -```bash -# Check metrics endpoint -curl http://localhost:9090/metrics | grep m3_8_optimization - -# Watch logs -kubectl logs -f deployment/memory-api -n poimen | grep "M3.8" -``` - ---- - -## Test Results - -``` -mem-core lib tests: 117/117 ✅ -M3.8.5 benchmarks: 15/15 ✅ -M3.8.6 gate tests: 14/14 ✅ -───────────────────────────────── -TOTAL: 146/146 ✅ (100%) -``` - ---- - -## Recent Commits - -- `b1bd932` feat: M3.8.6 complete — composition gate (14 tests) -- `478f656` feat: M3.8.5 complete — compression benchmarks (16 tests) -- `ecd8f51` docs: update M3.8 task specs (M3.8.3-6 detailed) -- `e9b98e5` feat: M3.8.3 complete — metrics & monitoring (7 tests) -- `090b9eb` feat: M3.8.2 complete — ingest optimizer infrastructure (5 tests) -- `f1917e1` docs: CRITICAL CORRECTION — M3.8 architecture (ingest, not query) - ---- - -## Next Steps (Unblocked) - -🔓 **M3.7.4 — Context Endpoint** -- Tier 1 (exact): sym_sha lookup via M3.7.8 -- Tier 2 (semantic): hybrid pgvector + OpenSearch -- Tier 3 (reference): Obsidian documentation corpus - -🔓 **M8.2 — Dual-write Indexer** -- Synchronized writes to pgvector + OpenSearch -- Consistency verification - -🔓 **M3.7.6 — Composition Gate** -- Safety, performance, quality assertions -- Depends on M3.7.4 + M8.2 - ---- - -## Key Insights - -1. **Ingest-time optimization > query-time** - - Process once, benefit all queries - - One-time cost vs per-query overhead - - Better embeddings, better rankings - -2. **Per-compressor metrics matter** - - Track compression ratio per type - - Identify content-type patterns - - Debug optimization effectiveness - -3. **Deterministic & idempotent** - - Same input always produces same output - - Re-optimizing doesn't change result - - Cache-safe for all scenarios - -4. **Graceful degradation** - - If optimization fails, use original - - No data loss on error - - Logging for observability - ---- - -## Summary - -**M3.8 is COMPLETE and PRODUCTION READY.** - -All 6 phases implemented with 146 tests passing (100% success rate). Architecture corrected to optimize at ingest time, improving pgvector embeddings and OpenSearch BM25 rankings for all queries. - -Integration helpers ready to wire into rebuild.rs. Metrics collection ready for Prometheus export. Safety, performance, and quality gates all passed. - -Ready for deployment to Kubernetes and production use. diff --git a/SESSION_COMPLETE.md b/SESSION_COMPLETE.md deleted file mode 100644 index cceb337..0000000 --- a/SESSION_COMPLETE.md +++ /dev/null @@ -1,348 +0,0 @@ -# Session Complete: Query Optimization Engine for Hybrid Search - -## What You Asked For - -> "We do need to build a query optimization engine or query context constructor for building accurate retrieval" - -**You're absolutely right.** Hybrid search fails without query understanding. - ---- - -## What We Built (Complete) - -### ✅ 1. Query Optimization Engine -**File:** `crates/mem-cli/src/query_optimizer.rs` (489 LOC) - -**6-Stage Pipeline:** -1. Normalize query (lowercase, trim) -2. Tokenize into words -3. Extract entities (years, quoted phrases, tags) -4. Analyze characteristics (dates, negation, special syntax) -5. Classify question type (Procedural, Factual, Troubleshooting, etc.) -6. Route to optimal search strategy (Hybrid, Semantic, Lexical, Cascading) - -**Key: Decision-making BEFORE retrieval** - -``` -Query: "How do I fix kubernetes port 8080 in 2024?" - ↓ Analyze - ├─ Type: Procedural (starts with "How") - ├─ Has dates: YES ("2024") - ├─ Token count: 8 - ├─ Confidence: 0.95 - └─ Strategy: Cascading - (Use OpenSearch to narrow by year → pgvector to rerank) -``` - -### ✅ 2. Hybrid Query Worker -**File:** `crates/mem-cli/src/hybrid_query_worker.rs` (387 LOC) - -**Parallel Orchestration:** -- Generate embedding (LLM) -- Execute pgvector search (top-50) in parallel -- Execute OpenSearch search (top-50) with JWT in parallel -- Fuse using RRF algorithm (no parameter tuning) -- Return top-10 with score breakdown + metrics - -**4 Search Strategies:** -- **Hybrid**: Both engines → RRF fusion (best accuracy) -- **Cascading**: OpenSearch narrow → pgvector rerank (fastest) -- **Semantic**: pgvector only (fallback) -- **Lexical**: OpenSearch only (fallback) - -### ✅ 3. RRF Fusion Algorithm -**Reciprocal Rank Fusion** — No parameter tuning needed - -``` -Formula: 1 / (k + rank) where k=60 - -Why RRF? -✓ No tuning needed (k=60 is academic standard) -✓ Robust to score distribution differences -✓ Works if embedding model changes -✓ Academic consensus for multi-engine fusion -``` - -### ✅ 4. Complete Design Documentation - -**QUERY_OPTIMIZATION_ENGINE.md** (698 LOC) -- Why Approach A (Parallel RRF) -- 6-stage pipeline detailed -- Question classification rules -- Search strategy routing decision tree -- RRF algorithm with Rust code -- 4-phase implementation plan -- Testing checklist + accuracy metrics -- Configuration reference - -**HYBRID_SEARCH_DESIGN.md** (762 LOC) -- 5-stage retrieval pipeline -- Index optimization (pgvector HNSW + OpenSearch BM25) -- Accuracy metrics (NDCG, MRR, Precision, Recall) -- Query routing heuristics -- A/B testing framework - -**API_REVIEW.md** (501 LOC) -- Review of all 10 endpoints -- Distinction: Query APIs vs Retrieval APIs -- Current gaps + enhancement roadmap -- Phase 1-4 improvements - -**IMPLEMENTATION_NOTES.md** (329 LOC) -- API corrections needed (VectorStore, OpenSearchClient) -- Phase 2 5-day implementation checklist -- Code diff preview -- Design validation matrix - -**Updated memory-flow.md** (833 LOC) -- 5-stage retrieval pipeline visual -- Query routing decision tree -- Index optimization details -- Pod infrastructure (8 core pods) - ---- - -## Why This Is the Right Solution - -### ❌ What Doesn't Work - -**Approach B (Cascading Only):** -``` -OpenSearch first to narrow - → pgvector rerank - -Problem: False negatives! -If document uses perfect synonyms but wrong keywords, -OpenSearch drops it before pgvector ever sees it. -``` - -**Approach C (Unified OpenSearch):** -``` -Single endpoint through OpenSearch - → Neural search plugin calls embedding model - -Problem: Coupling, complexity, debugging harder -``` - -### ✅ Why Approach A (Parallel RRF) Wins - -| Metric | Approach A | Approach B | Approach C | -|--------|-----------|-----------|-----------| -| **Accuracy** | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐ | -| **No False Negatives** | ✅ YES | ❌ NO | ✓ Mostly | -| **Fault Tolerance** | ✅ Fallback to semantic | ✓ Limited | ⚠️ Cluster-dependent | -| **Debugging** | ✅ Clear breakdown | ⚠️ Hard | ⚠️ Very hard | -| **Parameter Tuning** | ❌ None (k=60) | ✅ None | ❌ Complex config | -| **Complexity** | ⭐⭐⭐ | ⭐ | ⭐⭐⭐⭐⭐ | -| **Best For** | Mission-critical RAG | High-scale, tight QPS | Single-stack archs | - -**We chose Approach A because:** -- Agents make decisions on retrieved context -- Missing docs = wrong decisions -- Must maximize accuracy + reliability -- Fallback strategy (semantic-only if OpenSearch down) -- Clear transparency for debugging - ---- - -## Key Architectural Decision: Query Optimization First - -**Classic mistake:** Try to fuse search results without understanding query. - -**Right approach:** -``` -Raw Query - ↓ QueryOptimizer (6 stages) - ↓ Understand intent + pick optimal strategy - ↓ HybridQueryWorker (execute optimally) - ↓ Return accurate top-10 results -``` - -**Example:** -``` -Query: "What is the error when kubernetes scheduling fails?" - -Without optimization: - → Search all engines for all results - → Waste time on semantically irrelevant docs - -With optimization: - → Classify: Factual + Troubleshooting hybrid - → Route: Use HYBRID strategy - → Result: 85% better ranking accuracy -``` - ---- - -## Implementation Status - -### Phase 1: ✅ COMPLETE (Today) -- ✅ QueryOptimizer (450 LOC, 15 tests) -- ✅ HybridQueryWorker (380 LOC, stub) -- ✅ RRF Algorithm (no parameter tuning) -- ✅ Comprehensive design (2,900+ LOC) -- ✅ API corrections documented - -### Phase 2: 📋 NEXT (Week 2, 3-4 days) -- [ ] Fix VectorStore API calls (20 min) -- [ ] Integrate into /memory/query endpoint -- [ ] Add fallback strategy -- [ ] 10+ integration tests -- [ ] Measure latency - -### Phase 3: 🔄 (Week 3, 2-3 days) -- [ ] Performance optimization -- [ ] Query caching -- [ ] Benchmark suite - -### Phase 4: ✓ (Week 4, 2-3 days) -- [ ] NDCG/MRR testing -- [ ] A/B testing (Hybrid vs Semantic) -- [ ] Weight tuning (if switching from RRF) - ---- - -## Files Delivered - -### Code (876 LOC) -``` -✅ query_optimizer.rs (489 LOC) - ├─ 6-stage pipeline - ├─ 6 question types - ├─ 4 search strategies - └─ RRF algorithm - -✅ hybrid_query_worker.rs (387 LOC) - ├─ Parallel orchestration - ├─ 4 strategy implementations - ├─ Result fusion - └─ Response building -``` - -### Design Docs (2,733 LOC) -``` -✅ QUERY_OPTIMIZATION_ENGINE.md (698 LOC) — Core design -✅ HYBRID_SEARCH_DESIGN.md (762 LOC) — Retrieval pipeline -✅ API_REVIEW.md (501 LOC) — API audit -✅ IMPLEMENTATION_NOTES.md (329 LOC) — Phase 2 guide -✅ memory-flow.md (833 LOC) — Updated -✅ OPENSEARCH_JWT_SETUP.md (443 LOC) — K8s setup -✅ opensearch-deployment.yaml (384 LOC) — K8s manifest -``` - -### Total: 4,000+ LOC of production-ready design + code - ---- - -## How to Use (Phase 2) - -### 1. Fix APIs (20 minutes) - -```rust -// In opensearch_client.rs -+ pub async fn lexical_search(...) // Make public - -// In hybrid_query_worker.rs -- vector_store.search(...) // Fix API call -+ vector_store.search_l1(...) // Use actual method -``` - -### 2. Integrate into /memory/query - -```rust -// In http_server.rs query_handler() -async fn query_handler(...) -> HttpResponse { - // Try hybrid first - match state.hybrid_query_worker.query( - &project, &question, limit, &jwt_token - ).await { - Ok(response) => return HttpResponse::Ok().json(response), - Err(e) => { - // Fallback to semantic - match state.query_worker.query(...).await { - Ok(results) => return HttpResponse::Ok().json(results), - Err(e2) => return error!() - } - } - } -} -``` - -### 3. Test + Deploy - -```bash -# Unit tests (ready to run) -cargo test query_optimizer:: -cargo test hybrid_query_worker:: - -# Integration tests (to write in Phase 2) -cargo test it_hybrid_query:: - -# Deploy to staging + measure NDCG -# A/B test: Hybrid vs Semantic-only -# Monitor latency + accuracy -``` - ---- - -## Success Metrics - -| Metric | Target | How to Measure | -|--------|--------|----------------| -| **Hybrid Latency** | 150-250ms | API response time | -| **Cascading Latency** | 100-180ms | 2-stage performance | -| **NDCG@10** | ≥0.85 | Test fixture scoring | -| **MRR** | ≥0.8 | First correct result position | -| **Precision@5** | ≥0.8 | Accuracy in top-5 | -| **Zero false negatives** | 100% | Semantic catches synonyms | -| **Fallback success** | 100% | Degrades gracefully | - ---- - -## Key Decisions Locked In - -✅ **Approach A: Parallel RRF** — Academic consensus, no tuning -✅ **QueryOptimizer first** — Understand before retrieving -✅ **4 search strategies** — Optimize for query type -✅ **JWT forwarding** — Consistent auth to OpenSearch -✅ **Score breakdown** — Transparency + debugging -✅ **Cascading support** — Fastest option for date filters -✅ **RRF k=60** — No parameter tuning needed - ---- - -## What Happens Next Week - -### Phase 2 Goals -1. API integration (hybrid → /memory/query) -2. Fallback strategy (hybrid → semantic → error) -3. 10+ integration tests -4. Latency benchmarks -5. Deploy to staging - -### Expected Result -- `/memory/query` now uses hybrid search -- NDCG improved from ~0.75 → 0.85+ -- Agents get better context -- Clear score breakdown for transparency -- Fallback if OpenSearch unavailable - ---- - -## Bottom Line - -You asked for a query optimization engine to maximize retrieval accuracy. - -**We built:** -1. ✅ 6-stage QueryOptimizer (understands queries) -2. ✅ HybridQueryWorker (executes optimally) -3. ✅ RRF fusion (no parameter tuning) -4. ✅ 4 search strategies (adapts to query type) -5. ✅ Complete documentation (ready to implement) - -**Result:** Production-grade hybrid search that maximizes accuracy for mission-critical agent reasoning. - -**Status:** Design complete. Ready for Phase 2 integration. - -🎉 **Session Complete** - diff --git a/TEMPORAL_SKILLS_GUIDE.md b/TEMPORAL_SKILLS_GUIDE.md deleted file mode 100644 index f1ef2ce..0000000 --- a/TEMPORAL_SKILLS_GUIDE.md +++ /dev/null @@ -1,325 +0,0 @@ -# Guide: Writing & Uploading Skills/Memory for Temporal Workflows - -Use the memory service to store context, tool patterns, and solutions that Temporal workflows can retrieve and use. - -## Overview - -**Three ways to get data into memory:** - -1. **Ingest transcripts** (dialog with tool use + results) → system extracts skills -2. **Direct skill upload** (manual structured skill) -3. **Git corpus** (reference documentation, no extraction needed) - -For Temporal workflows, option **1 (transcripts)** is most powerful: you capture a successful workflow execution, memory system learns the pattern, and future workflows query it. - ---- - -## Format 1: Transcript-Based (Recommended) - -Write a conversation showing a workflow using tools successfully. Memory extracts reusable skills. - -### File Format - -Create JSONL (one JSON object per line): - -```jsonl -{"role":"user","text":"Deploy service foo to prod","timestamp":"2025-01-15T10:00:00Z","source_position":0} -{"role":"assistant","text":"I'll deploy foo using kubectl","timestamp":"2025-01-15T10:00:01Z","source_position":1} -{"role":"tool_result","text":"kubectl apply -f foo.yaml\nDeployment foo created","timestamp":"2025-01-15T10:00:02Z","source_position":2} -{"role":"assistant","text":"Deployment successful","timestamp":"2025-01-15T10:00:03Z","source_position":3} -``` - -**Fields:** -- `role` — `user`, `assistant`, `tool_result`, `system` -- `text` — actual message/command/result -- `timestamp` — ISO8601 (e.g., `2025-01-15T10:00:00Z`) -- `source_position` — line number in original source (for tracking) - -### Upload via HTTP - -```bash -curl -X POST http://localhost:8080/memory/ingest \ - -H "apikey: YOUR_API_KEY" \ - -H "Content-Type: application/json" \ - -d '{ - "project": "temporal-workflows", - "source": "transcript:slack/deployment-patterns", - "ingest_id": "deploy-2025-01-15-abc123", - "records": [ - { - "role": "user", - "text": "Deploy service foo to prod", - "timestamp": "2025-01-15T10:00:00Z", - "source_position": 0 - }, - { - "role": "assistant", - "text": "I'll deploy foo using kubectl apply", - "timestamp": "2025-01-15T10:00:01Z", - "source_position": 1 - }, - { - "role": "tool_result", - "text": "kubectl apply -f foo.yaml\nDeployment foo created", - "timestamp": "2025-01-15T10:00:02Z", - "source_position": 2 - } - ] - }' -``` - -**Response:** -```json -{ - "ingest_id": "deploy-2025-01-15-abc123", - "status": "pending", - "status_url": "/memory/ingest/deploy-2025-01-15-abc123" -} -``` - -Check status: -```bash -curl -H "apikey: YOUR_API_KEY" \ - http://localhost:8080/memory/ingest/deploy-2025-01-15-abc123 -``` - ---- - -## Format 2: Structured Skill (Direct) - -If you want to upload a pre-written skill without going through extraction: - -### YAML Format (for manual storage) - -Create `skills/temporal-patterns.yaml`: - -```yaml -name: "kubernetes_deploy_pattern" -description: "Safe deployment pattern using kubectl apply with validation" -when_to_use: "When deploying services to Kubernetes cluster" -examples: - - | - kubectl apply -f service.yaml - kubectl rollout status deployment/service -n default - kubectl get pods -n default -prerequisites: - - "kubectl binary installed" - - "kubeconfig configured" - - "deployment manifest exists" -steps: - - "Validate manifest: kubectl apply -f service.yaml --dry-run=client" - - "Apply: kubectl apply -f service.yaml" - - "Monitor: kubectl rollout status deployment/service -n default" - - "Verify: kubectl get pods, check for Ready status" -precautions: - - "Never use --force unless necessary" - - "Always check diff before applying to prod" - - "Rollback plan: kubectl rollout undo deployment/service" -``` - -Then ingest as system memory: - -```bash -curl -X POST http://localhost:8080/memory/ingest \ - -H "apikey: YOUR_API_KEY" \ - -H "Content-Type: application/json" \ - -d '{ - "project": "temporal-workflows", - "source": "skill:manual/kubernetes", - "ingest_id": "skill-k8s-deploy-001", - "records": [ - { - "role": "system", - "text": "SKILL: kubernetes_deploy_pattern\n\nSafe deployment pattern using kubectl apply with validation\n\nWhen to use: When deploying services to Kubernetes cluster\n\nSteps:\n1. Validate manifest: kubectl apply -f service.yaml --dry-run=client\n2. Apply: kubectl apply -f service.yaml\n3. Monitor: kubectl rollout status deployment/service -n default\n4. Verify: kubectl get pods, check for Ready status\n\nPrecautions:\n- Never use --force unless necessary\n- Always check diff before applying to prod\n- Rollback plan: kubectl rollout undo deployment/service", - "timestamp": "2025-01-15T10:00:00Z", - "source_position": 0 - } - ] - }' -``` - ---- - -## Format 3: Git Corpus (Reference Docs) - -Documentation (never evidence, read-only for context). - -Store in your obsidian-memory repo, then: - -```bash -mem ingest --source git:ssh://git@git.riotpiao.com:2222/rock/poimen-obesdient-memory.git \ - --project temporal-workflows -``` - -(This will be auto-triggered by CI once M3.5.9 is done.) - ---- - -## Querying Skills in Temporal Workflows - -### Get all skills for a project - -```bash -curl -H "apikey: YOUR_API_KEY" \ - "http://localhost:8080/memory/skills?project=temporal-workflows" -``` - -**Response:** -```json -{ - "skills": [ - { - "name": "kubernetes_deploy_pattern", - "description": "Safe deployment pattern using kubectl apply with validation", - "when_to_use": "When deploying services to Kubernetes cluster" - }, - { - "name": "postgres_backup_pattern", - "description": "Automated backup with verification", - "when_to_use": "Backup Postgres database before migrations" - } - ], - "count": 2 -} -``` - -### Get tool context (tool failure context + similar cases) - -```bash -curl -H "apikey: YOUR_API_KEY" \ - "http://localhost:8080/memory/context?tool=kubectl&error=connection+refused" -``` - -Returns: -- **Tier 1** — Exact match (same error + context) -- **Tier 2** — Similar symptom (vector search) -- **Tier 3** — Reference docs (R corpus) - -*Note: This endpoint is M3.7.4 (in progress).* - ---- - -## Example: Temporal Activity + Memory Query - -```go -// activity.go -func QueryMemoryForPattern(ctx context.Context, toolName string, errorMsg string) (string, error) { - resp, err := http.Get(fmt.Sprintf( - "http://memory-service/memory/context?tool=%s&error=%s", - url.QueryEscape(toolName), - url.QueryEscape(errorMsg), - )) - if err != nil { - return "", err - } - defer resp.Body.Close() - - var result map[string]interface{} - json.NewDecoder(resp.Body).Decode(&result) - - // Use tier 1 (exact match) if available, else tier 2 (symptom), else tier 3 (docs) - if tier1, ok := result["tier_1"].(string); ok && tier1 != "" { - return tier1, nil - } - // ... same for tier 2, tier 3 - - return "", nil -} -``` - ---- - -## Best Practices - -1. **Source naming** — `source` field identifies where data came from: - - `transcript:slack/topic` - - `transcript:github/issue-123` - - `skill:manual/pattern-name` - - `git:ssh://git@.../repo.git` - - `doc:obsidian-vault/path/to/note` - -2. **Idempotency** — `ingest_id` must be stable (use SHA256 of content): - ```bash - ingest_id=$(echo "temporal-deploy-pattern-2025-01-15" | sha256sum | cut -d' ' -f1) - ``` - -3. **Batch ingests** — upload multiple transcripts in one request to reduce overhead. - -4. **Timestamps** — use workflow execution time, not current time. Helps memory system understand sequence. - -5. **Project naming** — use consistent project keys (e.g., `temporal-workflows`, `agent-rust`, `poimen`). - ---- - -## Ingesting from Temporal Directly - -**Pseudo-code** (implement in your Temporal activity): - -```go -func IngestWorkflowToMemory(ctx context.Context, execution WorkflowExecution) error { - records := []Record{} - - // Walk through history and extract tool results - for _, event := range execution.History.Events { - if event.Type == "ActivityCompleted" { - records = append(records, Record{ - Role: "tool_result", - Text: event.Result, - Timestamp: event.Timestamp, - }) - } - } - - // POST to /memory/ingest - body := map[string]interface{}{ - "project": "temporal-workflows", - "source": "temporal:workflow/" + execution.WorkflowID, - "ingest_id": execution.RunID, // idempotent - "records": records, - } - - resp, err := http.Post("http://memory-service/memory/ingest", - "application/json", - jsonBody(body), - ) - // ... -} -``` - -Then, in your workflow, query back: - -```go -func QueryMemoryInWorkflow(ctx context.Context, q string) ([]Result, error) { - resp, _ := http.Get(fmt.Sprintf( - "http://memory-service/memory/query?project=temporal-workflows&query=%s", - url.QueryEscape(q), - )) - // ... -} -``` - ---- - -## Troubleshooting - -| Issue | Cause | Fix | -|---|---|---| -| `401 unauthorized` | Missing apikey header | Add `-H "apikey: YOUR_KEY"` | -| `202` then status `pending` forever | Ingest worker not running | Check `mem serve` is running with DB connection | -| Skills not appearing | M4.1 (extraction) not implemented yet | Use Format 2 (direct skill) for now | -| Rate limited (429) | Hit limit for project | Check rate limit, wait or use different apikey | -| Duplicate `ingest_id` | Same payload ingested twice | Intentional (idempotency); returns same job_id | - ---- - -## Timeline - -| Task | Status | Impact | -|---|---|---| -| M3.5.2 (ingest endpoint) | ✅ | Upload transcripts now | -| M3.5.5 (skills endpoint) | ✅ | Query skills now | -| M4.1 (skill extraction) | 🟡 | Automatic extraction in progress | -| M3.7.4 (context endpoint) | ⬜ | Tier-1 lookup not yet available | -| M3.7.8 (symptom projection) | ⬜ | Tier-2 vector lookup not yet available | - -**Actionable now:** Formats 1 & 2, endpoints work. Extract by hand or via M4.1 when ready.