# Poimen Memory System Architecture

## Overview

The Poimen Memory system is a distributed, multi-tier memory management platform designed for AI applications. It provides persistent storage, semantic search, and intelligent caching for conversations, logs, and structured data.

## Core Components

### 1. PostgreSQL with pgvector

PostgreSQL serves as our primary data store with pgvector extension for semantic search. The system uses 768-dimensional embeddings generated by the nomic-embed-text-v2-moe model.

Features:
- HNSW indexes for fast approximate nearest neighbor search
- Full ACID compliance with 2-node HA cluster
- Automatic failover with 10-minute RTO
- 10GB persistent volumes with daily backups

### 2. OpenSearch Cluster

OpenSearch provides full-text search and BM25 ranking. Documents are indexed with both raw text and preprocessed fields.

Configuration:
- 2-node cluster (1 master, 1 data)
- 8GB heap per node
- 20GB storage per node
- Refresh interval: 10s
- Index shards: 3, replicas: 1

### 3. Memory Ingest Pipeline

Records flow through a 4-stage pipeline:
1. Source extraction (Pi sessions, Claude transcripts, doc corpus)
2. Content routing (Magika ML classification)
3. Type-specific compression (log, json, diff, text)
4. Embedding generation and indexing

### 4. Query Path (Hybrid Search)

Queries use dual retrieval:
- 60% pgvector semantic search (top-k nearest neighbors)
- 40% OpenSearch BM25 ranking
- Fusion via Reciprocal Rank Weighting (RRW)

Results are re-ranked and deduplicated before LLM context window.

## M3.8 Context Optimization

The context optimizer runs at ingest time, improving data quality before embedding:

### Compression Targets
- Logs: 85-95% (remove timestamps, debug lines)
- JSON: 70-90% (minify, remove verbose keys)
- Text: 30-50% (remove markdown artifacts)
- Diffs: 60-80% (remove context lines)

### Benefits
- Better pgvector embeddings (clean input = better semantic quality)
- Better BM25 ranking (signal-rich text = stronger matches)
- Reduced storage (lower bandwidth, faster queries)
- All queries benefit (optimization happens once)

## Performance Targets

- Ingest latency: <1ms per record
- Query latency: <100ms P95 (hybrid search)
- Embedding generation: <500ms for 50-record batch
- Indexing throughput: 1000+ records/sec
- Search throughput: 100+ queries/sec

## Monitoring & Observability

### Metrics Exported

Via Prometheus `/metrics` endpoint:
- `m3_8_optimization_records_total` - records processed
- `m3_8_optimization_compression_ratio` - overall compression %
- `m3_8_optimization_compressor_ratio` - per-type compression
- Query latency distribution (P50, P95, P99)
- Embedding cache hit ratio

### Logging

Structured logs via tracing:
- INFO: ingest completion, query execution, errors
- DEBUG: compression stats, cache hits, routing decisions
- TRACE: individual record processing

## Deployment

### Kubernetes

Resources deployed in `poimen` namespace:
- Deployment: memory-api (2 replicas)
- StatefulSet: memory-db (PostgreSQL)
- Deployment: opensearch (2 replicas)
- ConfigMap: optimization settings
- Secret: database credentials, API keys

### Environment Variables

- `MEM_CONTEXT_OPTIMIZER` - optimizer mode (on|off)
- `MEM_COMPRESSION_TARGETS` - JSON targets per type
- `MEM_CACHE_SIZE_MB` - compression cache size
- `MEM_PROMETHEUS_ENABLED` - metrics export

## Testing Strategy

### Unit Tests (62 tests)
- Individual compressor algorithms
- Content routing accuracy
- Cache behavior

### Integration Tests (37 tests)
- End-to-end ingest pipeline
- Search quality on compressed content
- Metrics collection accuracy

### Benchmark Tests (16 tests)
- Compression ratio validation
- Query performance with/without optimization
- Throughput and latency targets

### Gate Tests (13 tests)
- Safety assertions (no data loss)
- Performance assertions (latency <3ms)
- Quality assertions (compression targets met)

## Roadmap

### Current (M3.8)
✅ Core optimizer (62 tests)
✅ Ingest integration (5 tests)
✅ Metrics & monitoring (7 tests)
⏳ Benchmarks (16 tests)
⏳ Gate verification (13 tests)

### Next (M3.7.4-6)
- Context endpoint (semantic + reference tiers)
- Dual-write indexer (pgvector + OpenSearch)
- Composition gate

### Future (M4-M7)
- Skill management
- Source connectors (Obsidian, git)
- Frontend React app