poimen-memory/docs/PHASE_7_TEMPORAL_RAGA_INGEST_DESIGN.md

# Phase 7: Temporal-RAGA-Ingest Architecture Design

**Status**: 🔵 Design Phase (No code yet)

---

## 1. Executive Summary

**Phase 7** extends Poimen Memory with:
1. **Temporal-Aware Knowledge Graph** (time-series semantic versioning)
2. **RAGA Ingest Pipeline** (Retrieval-Augmented Graph Architecture)
3. **Chunk Editing & Versioning** (immutable append-only audit trail)
4. **Multi-Signal Ranking** (4 signals: confidence, recency, community, BM25)

**Key Innovation**: Enable interactive **chunk editing** with full **temporal lineage** while maintaining **deterministic rebuilds** and **append-only audit logs**.

---

## 2. Architecture Overview

### Current State (Phase 6)
```
┌─────────────────┐
│ Input Content   │
└────────┬────────┘
         │
         ↓
┌─────────────────────────────────────┐
│ Extraction Pipeline (Phase 3)        │
│ ├─ Entity Linking (Phase 5.1)       │
│ ├─ Inference (Phase 5.2)            │
│ └─ Query Reasoning (Phase 5.3)      │
└────────┬────────────────────────────┘
         │
         ↓
┌─────────────────────────────────────┐
│ Memory Storage (Phase 2)             │
│ ├─ memory_entity (L1/L2)            │
│ ├─ memory_edge (relationships)      │
│ └─ memory_vector (embeddings)       │
└────────┬────────────────────────────┘
         │
         ↓
┌─────────────────────────────────────┐
│ Query & Retrieval (Phase 4 + 6)     │
│ ├─ Semantic search                  │
│ ├─ Lexical search                   │
│ └─ Hybrid ranking (RRF)             │
└─────────────────────────────────────┘
```

### Phase 7 Extension
```
┌──────────────────────────────────────────────────────────────┐
│ TEMPORAL-RAGA-INGEST ARCHITECTURE                            │
└──────────────────────────────────────────────────────────────┘

INPUT LAYER (Temporal-Aware):
  ├─ Streaming ingest (Kafka)
  ├─ Batch ingest (S3/Git)
  ├─ Interactive edits (UI)
  └─ Time-tagged: {content, timestamp, editor, reason}

                    ↓

EXTRACTION LAYER (LLM-Gated):
  ├─ Entity extraction (with confidence)
  ├─ Relationship inference
  ├─ Temporal fact alignment (when/how long)
  └─ Conflict detection (contradictions)

                    ↓

VERSIONING LAYER (Append-Only):
  ├─ chunk_versions: {chunk_id, version, content, editor, timestamp}
  ├─ entity_versions: {entity_id, version, properties, confidence}
  ├─ edge_versions: {edge_id, version, source, relation, target}
  └─ audit_trail: All mutations logged immutably

                    ↓

STORAGE LAYER (Multi-Signal Ranking):
  ├─ memory_entity: entity + confidence + temporal_bounds
  ├─ memory_edge: edge + recency_score + community_signal
  ├─ memory_vector: embedding + ranking_signals
  └─ chunk_metadata: version_history, edit_count, quality_score

                    ↓

QUERY LAYER (Context-Aware Retrieval):
  ├─ Semantic search (Phase 4.1)
  ├─ Lexical search (Phase 4.2)
  ├─ Temporal filtering (when is this true?)
  ├─ Community detection (Phase 4.3)
  ├─ Multi-signal ranking (Phase 7 NEW)
  └─ Deterministic tie-breaking (version + timestamp)
```

---

## 3. Chunk Editing Semantics

### Current: Read-Only
```
memory_entity (immutable)
  ├─ id: UUID
  ├─ name: String
  ├─ properties: JSON (frozen)
  └─ created_at: Timestamp
```

### Phase 7: Versioned + Editable

#### Option A: Immutable Versions (Recommended)
```sql
-- Chunks are immutable; edits create new versions
CREATE TABLE chunk_versions (
  id UUID,
  chunk_id UUID,              -- Stable reference
  version INT,                -- 1, 2, 3...
  content TEXT,
  properties JSONB,
  edited_by VARCHAR(255),
  edit_reason TEXT,
  confidence FLOAT,           -- 0.0-1.0
  temporal_start TIMESTAMP,
  temporal_end TIMESTAMP,
  is_current BOOLEAN,
  created_at TIMESTAMP,
  PRIMARY KEY (chunk_id, version)
);

-- Active chunk always = is_current = true
SELECT * FROM chunk_versions WHERE chunk_id = $1 AND is_current = true;

-- Audit trail = historical versions
SELECT * FROM chunk_versions WHERE chunk_id = $1 ORDER BY version;
```

**Example**:
```
chunk_id: ent-123 (Kubernetes entity)

Version 1:
  content: "Kubernetes: Container orchestrator by Google"
  confidence: 0.85
  edited_by: alice
  created_at: 2025-01-30 10:00:00
  is_current: false

Version 2:
  content: "Kubernetes: Container orchestrator by Google, open-source"
  confidence: 0.92
  edited_by: bob
  created_at: 2025-01-31 14:30:00
  is_current: true
```

#### Option B: Soft Deletes (Alternative)
```sql
CREATE TABLE chunk_versions (
  id UUID,
  chunk_id UUID,
  version INT,
  content TEXT,
  is_deleted BOOLEAN,        -- soft delete
  deleted_by VARCHAR(255),
  PRIMARY KEY (chunk_id, version)
);
```

**Verdict**: Use **Option A (Immutable)** — clearer lineage, no "tombstone" ambiguity.

---

## 4. Ranking Formula (4 Signals)

### The 4 Signals

#### Signal 1: Confidence (LLM Assessment)
```
confidence ∈ [0.0, 1.0]

Source: LLM extraction pipeline (Phase 5.2)
  ├─ 0.0-0.3: Uncertain (likely noise)
  ├─ 0.3-0.6: Moderate confidence
  ├─ 0.6-0.85: High confidence
  └─ 0.85-1.0: Verified/human-edited

Weight: 25%
```

#### Signal 2: Recency (Temporal Freshness)
```
recency_score = exp(-(now() - last_verified) / τ)
  where τ = 30 days (half-life)

Examples:
  ├─ Last verified today: 1.00
  ├─ Last verified 7 days ago: 0.78
  ├─ Last verified 30 days ago: 0.50
  ├─ Last verified 90 days ago: 0.12
  └─ Last verified 365 days ago: 0.0001

Weight: 25%
```

#### Signal 3: Community Signal (Graph Centrality)
```
community_score = pagerank_score * in_degree_normalized

Where:
  ├─ pagerank_score: Eigenvector centrality in entity graph
  │  └─ High if many entities link to this entity
  ├─ in_degree_normalized: (in_degree / max_in_degree) * 0.5
  │  └─ Capped at 0.5 to avoid dominance
  └─ Result: [0.0, 1.0]

Examples:
  ├─ "Kubernetes": 0.92 (highly connected)
  ├─ "Docker": 0.87 (related to 15 entities)
  └─ "ObscureLibrary": 0.15 (1-2 links only)

Weight: 25%
```

#### Signal 4: Lexical Ranking (BM25 from OpenSearch)
```
bm25_score = OpenSearch BM25 normalized

Source: Full-text index (existing Phase 4.2)
  ├─ Raw BM25: [0, ∞)
  ├─ Normalized: [0, 1.0]
  │  └─ sigmoid(bm25 / 10.0) for smoothing
  └─ Higher: More textually relevant to query

Examples:
  ├─ Exact match: 0.95+
  ├─ Semantic match: 0.60-0.80
  └─ Weak match: 0.10-0.30

Weight: 25%
```

### Final Ranking Formula

```
final_score = 0.25 * confidence
            + 0.25 * recency_score
            + 0.25 * community_score
            + 0.25 * bm25_normalized

Range: [0.0, 1.0]

Tie-breaking (deterministic):
  1. Higher final_score wins
  2. If tied: newer version_timestamp wins
  3. If still tied: lexicographically by chunk_id
```

### Example Calculation

```
Entity: "Kubernetes"

Signals:
  ├─ Confidence: 0.92 (human-verified, Phase 5.4)
  ├─ Recency: 0.88 (last verified 5 days ago)
  ├─ Community: 0.85 (PageRank in entity graph)
  └─ BM25: 0.78 (matched query terms)

Calculation:
  final_score = 0.25*0.92 + 0.25*0.88 + 0.25*0.85 + 0.25*0.78
              = 0.23 + 0.22 + 0.2125 + 0.195
              = 0.8575

Ranking: Top result (if query relevance high)
```

---

## 5. Audit Trail Strategy

### Append-Only Event Log (JSONL)

```json
{
  "id": "audit-evt-123",
  "timestamp": "2025-01-30T14:30:45Z",
  "event_type": "chunk_edited|chunk_created|chunk_verified|chunk_deleted",
  "actor": "[email protected]",
  "resource": {
    "type": "chunk",
    "id": "ent-123",
    "version": 2
  },
  "action": {
    "operation": "update",
    "old_value": {
      "content": "Kubernetes: Container orchestrator by Google",
      "confidence": 0.85
    },
    "new_value": {
      "content": "Kubernetes: Container orchestrator by Google, open-source CNCF project",
      "confidence": 0.92
    }
  },
  "reason": "Added CNCF affiliation after verification",
  "signature": "sha256(timestamp + actor + resource + action)"
}
```

### Storage & Queries

```sql
-- Append-only event log
CREATE TABLE audit_events (
  id UUID PRIMARY KEY,
  timestamp TIMESTAMP NOT NULL,
  event_type VARCHAR(50),
  actor VARCHAR(255),
  resource_type VARCHAR(50),
  resource_id UUID,
  version INT,
  action JSONB,
  reason TEXT,
  signature VARCHAR(256),
  created_at TIMESTAMP DEFAULT now()
) PARTITION BY RANGE (timestamp);  -- Monthly partitions

-- Query: What edits were made to entity?
SELECT * FROM audit_events
WHERE resource_type = 'chunk' AND resource_id = $chunk_id
ORDER BY timestamp DESC;

-- Query: Who changed what, when?
SELECT timestamp, actor, reason, action
FROM audit_events
WHERE resource_id = $chunk_id
ORDER BY timestamp;

-- Query: Verify integrity (signature validation)
SELECT * FROM audit_events
WHERE signature != sha256(...)
LIMIT 100;  -- Detect tampering
```

### Immutability Guarantees

```
1. No UPDATE on audit_events table
   ├─ Only INSERT is allowed
   └─ Policy: ROW SECURITY

2. Separate write vs read storage
   ├─ Write: Hot log (Kafka JSONL file)
   ├─ Read: Cold archive (S3 + compression)
   └─ Sync: Daily batch to S3

3. Cryptographic signing
   ├─ Each event: signature = sha256(timestamp || actor || resource || action)
   ├─ Chain of custody: timestamp chain prevents reordering
   └─ Verification: Detect tampering in O(1) time
```

---

## 6. Schema Extensions

### New Tables (Phase 7)

```sql
-- Versioning
CREATE TABLE chunk_versions (
  id UUID PRIMARY KEY,
  chunk_id UUID NOT NULL,              -- Stable reference
  version INT NOT NULL,
  content TEXT,
  properties JSONB,
  edited_by VARCHAR(255),
  edit_reason TEXT,
  confidence FLOAT DEFAULT 0.5,
  temporal_start TIMESTAMP,
  temporal_end TIMESTAMP,
  is_current BOOLEAN,
  created_at TIMESTAMP DEFAULT now(),
  UNIQUE(chunk_id, version),
  FOREIGN KEY (chunk_id) REFERENCES memory_entity(id)
);

-- Audit trail
CREATE TABLE audit_events (
  id UUID PRIMARY KEY,
  timestamp TIMESTAMP NOT NULL,
  event_type VARCHAR(50) NOT NULL,
  actor VARCHAR(255),
  resource_type VARCHAR(50),
  resource_id UUID,
  version INT,
  action JSONB,
  reason TEXT,
  signature VARCHAR(256),
  created_at TIMESTAMP DEFAULT now()
) PARTITION BY RANGE (timestamp);

-- Ranking signals
CREATE TABLE ranking_signals (
  id UUID PRIMARY KEY,
  entity_id UUID NOT NULL,
  signal_type VARCHAR(50),  -- "confidence"|"recency"|"community"|"bm25"
  signal_value FLOAT,
  computed_at TIMESTAMP,
  FOREIGN KEY (entity_id) REFERENCES memory_entity(id)
);

-- Query rankings (for metrics)
CREATE TABLE query_rankings (
  id UUID PRIMARY KEY,
  query_id UUID,
  chunk_id UUID,
  rank INT,
  final_score FLOAT,
  signal_breakdown JSONB,  -- {confidence, recency, community, bm25}
  computed_at TIMESTAMP,
  FOREIGN KEY (chunk_id) REFERENCES memory_entity(id)
);
```

### Schema Modifications

```sql
-- Add to existing memory_entity
ALTER TABLE memory_entity ADD COLUMN (
  confidence FLOAT DEFAULT 0.5,
  last_verified_at TIMESTAMP,
  edit_count INT DEFAULT 0,
  current_version INT DEFAULT 1
);

-- Add to existing memory_edge
ALTER TABLE memory_edge ADD COLUMN (
  confidence FLOAT DEFAULT 0.5,
  recency_score FLOAT DEFAULT 1.0,
  community_signal FLOAT DEFAULT 0.5,
  last_updated_at TIMESTAMP
);
```

---

## 7. Editing Workflow

### User Edits a Chunk

```
1. User opens chunk in UI
   └─ GET /memory/chunks/{id}
      └─ Returns: current_version + full version history

2. User modifies content
   └─ Content changes: "Kubernetes: Container orchestrator"
                       → "Kubernetes: Container orchestrator (CNCF project)"

3. User submits edit
   └─ PATCH /memory/chunks/{id}
      └─ Payload: {
           "new_content": "...",
           "edit_reason": "Added CNCF affiliation",
           "confidence": 0.92
         }

4. Memory service:
   a. Validate edit (optional LLM gate for quality)
   b. Create new version:
      INSERT chunk_versions (chunk_id=id, version=2, content='...', ...)
   c. Update is_current: chunk_versions SET is_current=false WHERE version=1
   d. Audit log:
      INSERT audit_events (event_type='chunk_edited', resource_id=id, ...)
   e. Recompute signals:
      ├─ Confidence: extracted_confidence (or manual 0.92)
      ├─ Recency: 1.0 (just edited)
      ├─ Community: fetch from graph
      └─ BM25: reindex OpenSearch
   f. Return: {chunk_id, new_version, status: "OK"}

5. Query uses latest version:
   └─ SELECT * FROM chunk_versions
      WHERE chunk_id = $1 AND is_current = true
```

---

## 8. Deterministic Rebuild (Parity Check)

### Challenge
With versioning, **which version** do we rebuild from?

### Solution: Version Snapshot + Audit Replay

```
Rebuild Process:

1. Snapshot current state:
   SELECT chunk_id, max(version) as latest_version
   FROM chunk_versions
   WHERE is_current = true
   → SHA256 hash of {chunk_id, version, content}*
   → Store: rebuild_checksum_v1

2. From event log, replay all edits since T0:
   SELECT * FROM audit_events
   WHERE timestamp >= $baseline_timestamp
   ORDER BY timestamp ASC

3. Apply edits sequentially:
   For each audit_event:
     ├─ Create new chunk_version
     ├─ Mark old is_current = false
     ├─ Compute signals
     └─ Log event

4. Verify parity:
   SELECT chunk_id, max(version)
   FROM chunk_versions
   WHERE is_current = true
   → SHA256 hash
   → Check: rebuild_checksum_v2 == rebuild_checksum_v1
   → If match: ✅ Parity verified (M2.8 gate)
   → If diff: ❌ Rebuild failed (investigate)
```

---

## 9. Integration Points

### Phase 4 (Retrieval) → Phase 7 (Ranking)

**Current** (Phase 4):
```
Query → Semantic + Lexical → RRF Fusion → Top-10 results
```

**Phase 7** (Enhanced):
```
Query → Semantic + Lexical → RRF Fusion
  ↓
Multi-Signal Ranking:
  ├─ Fetch confidence, recency, community from ranking_signals
  ├─ Compute bm25_normalized from search scores
  ├─ Calculate: final_score = 0.25*conf + 0.25*recency + 0.25*community + 0.25*bm25
  └─ Sort by final_score DESC, then timestamp DESC, then chunk_id ASC

→ Top-10 results (with confidence intervals)
```

### Phase 5 (Synthesis) → Phase 7 (Confidence)

**Current** (Phase 5):
```
LLM extracts entities → Insert into memory_entity
```

**Phase 7** (Enhanced):
```
LLM extracts entities + confidence
  ↓
INSERT chunk_versions (confidence=$extracted_confidence)
  ↓
Signals computed:
  ├─ Confidence: $extracted_confidence (0.0-1.0)
  ├─ Recency: 1.0 (new)
  ├─ Community: graph_analysis() (0.0 initially, grows as linked)
  └─ BM25: from indexing
  ↓
Track lineage: audit_events with {operation, old_value, new_value}
```

---

## 10. Metrics Emission & Prometheus Scraping

### Metrics Endpoint

```
GET /metrics
├─ Auth: Authentik JWT (Bearer token)
├─ Format: Prometheus text format (OpenMetrics)
├─ Refresh: Real-time (computed on-demand)
└─ Timeout: 5 seconds
```

### Metrics Architecture

```
┌─────────────────────────────────────────┐
│ Memory Service                          │
│ ├─ MetricsPersistence (in-memory cache) │
│ ├─ record_success(agent_id, latency)   │
│ ├─ record_error(agent_id, latency)     │
│ └─ export_prometheus()                  │
│    └─ Returns: Prometheus text format   │
└────────────┬────────────────────────────┘
             │ Emits every 5s
             ↓
┌─────────────────────────────────────────┐
│ Prometheus Server                       │
│ ├─ Scrape /metrics (15s interval)       │
│ ├─ Store TSDB (15-month retention)      │
│ └─ Alert rules (M7.1-M7.5 gates)        │
└────────────┬────────────────────────────┘
             │
             ↓
┌─────────────────────────────────────────┐
│ Dashboards (Grafana)                    │
│ ├─ Phase 7 SLO dashboard                │
│ ├─ Edit latency percentiles             │
│ ├─ Ranking accuracy (NDCG weighted)     │
│ └─ Storage overhead tracking            │
└─────────────────────────────────────────┘
```

### Phase 7 Metrics

```prometheus
# HELP memory_chunk_edits_total Total chunk edits (create/update/delete)
# TYPE memory_chunk_edits_total counter
memory_chunk_edits_total{operation="create"} 1245
memory_chunk_edits_total{operation="update"} 3421
memory_chunk_edits_total{operation="delete"} 18

# HELP memory_chunk_versions_current Current version count per chunk
# TYPE memory_chunk_versions_current gauge
memory_chunk_versions_current{chunk_type="entity"} 5234
memory_chunk_versions_current{chunk_type="edge"} 2189

# HELP memory_edit_latency_seconds Edit operation latency
# TYPE memory_edit_latency_seconds histogram
memory_edit_latency_seconds_bucket{le="0.1"} 156
memory_edit_latency_seconds_bucket{le="0.5"} 1023
memory_edit_latency_seconds_bucket{le="1.0"} 3421
memory_edit_latency_seconds_bucket{le="2.0"} 4156  # P99 < 2s gate
memory_edit_latency_seconds_bucket{le="+Inf"} 4189
memory_edit_latency_seconds_sum 5234.56
memory_edit_latency_seconds_count 4189

# HELP memory_audit_events_total Total audit events logged
# TYPE memory_audit_events_total counter
memory_audit_events_total{event_type="chunk_edited"} 3421
memory_audit_events_total{event_type="chunk_created"} 1245
memory_audit_events_total{event_type="verification"} 892

# HELP memory_audit_signature_failures_total Signature verification failures
# TYPE memory_audit_signature_failures_total counter
memory_audit_signature_failures_total 0  # M7.2 gate: must be 0

# HELP memory_rebuild_checksum_matches_total Rebuild parity checks passed
# TYPE memory_rebuild_checksum_matches_total counter
memory_rebuild_checksum_matches_total 127  # M7.1 gate: 100%

# HELP memory_rebuild_checksum_mismatches_total Rebuild parity checks FAILED
# TYPE memory_rebuild_checksum_mismatches_total counter
memory_rebuild_checksum_mismatches_total 0  # M7.1 gate: must be 0

# HELP memory_ranking_ndcg_weighted Weighted NDCG@10 (confidence-adjusted)
# TYPE memory_ranking_ndcg_weighted gauge
memory_ranking_ndcg_weighted 0.882  # M7.3 gate: >= 0.88

# HELP memory_ranking_signal_contribution Contribution of each signal
# TYPE memory_ranking_signal_contribution gauge
memory_ranking_signal_contribution{signal="confidence"} 0.25
memory_ranking_signal_contribution{signal="recency"} 0.25
memory_ranking_signal_contribution{signal="community"} 0.25
memory_ranking_signal_contribution{signal="bm25"} 0.25

# HELP memory_storage_bytes Total storage bytes
# TYPE memory_storage_bytes gauge
memory_storage_bytes{type="current_versions"} 5368709120
memory_storage_bytes{type="all_versions"} 8053063680  # M7.5 gate: <= 1.5x
memory_storage_bytes{type="audit_log"} 2147483648

# HELP memory_storage_overhead_ratio Overhead ratio (versions / current)
# TYPE memory_storage_overhead_ratio gauge
memory_storage_overhead_ratio 1.25  # M7.5 gate: <= 1.5x

# HELP memory_confidence_distribution Confidence score distribution
# TYPE memory_confidence_distribution histogram
memory_confidence_distribution_bucket{le="0.3"} 234    # Uncertain
memory_confidence_distribution_bucket{le="0.6"} 1289   # Moderate
memory_confidence_distribution_bucket{le="0.85"} 4156  # High
memory_confidence_distribution_bucket{le="1.0"} 5234   # Verified

# HELP memory_recency_score_distribution Recency score distribution
# TYPE memory_recency_score_distribution gauge
memory_recency_score_avg 0.72  # Average recency (30 days half-life)
memory_recency_score_p50 0.78
memory_recency_score_p99 0.98

# HELP memory_community_score_distribution Community signal distribution
# TYPE memory_community_score_distribution gauge
memory_community_score_avg 0.45
memory_community_score_p50 0.42
memory_community_score_p99 0.91
```

### Prometheus Scrape Config

```yaml
# In homelab Prometheus values.yaml
prometheus:
  scrapeConfigs:
    - job_name: 'poimen-memory'
      kubernetes_sd_configs:
        - role: pod
          namespaces:
            names:
              - poimen
      relabel_configs:
        - source_labels: [__meta_kubernetes_pod_label_app]
          action: keep
          regex: memory-service
        - source_labels: [__meta_kubernetes_pod_container_port_name]
          action: keep
          regex: metrics
      scheme: https
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      scrape_interval: 15s
      scrape_timeout: 5s
```

### SLO Alerts (Prometheus Rules)

```yaml
# rules/poimen-memory-phase7.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: poimen-memory-phase7
  namespace: monitoring
spec:
  groups:
    - name: memory-phase7-gates
      interval: 30s
      rules:
        # M7.1: Rebuild parity
        - alert: M7_1_RebuildParityCheckFailed
          expr: increase(memory_rebuild_checksum_mismatches_total[5m]) > 0
          for: 1m
          labels:
            severity: critical
            phase: "7"
          annotations:
            summary: "Rebuild parity check failed (M7.1 gate)"
            description: "{{ $value }} checksum mismatches in last 5 minutes"
        
        # M7.2: Audit trail integrity
        - alert: M7_2_AuditSignatureFailure
          expr: increase(memory_audit_signature_failures_total[5m]) > 0
          for: 1m
          labels:
            severity: critical
            phase: "7"
          annotations:
            summary: "Audit signature verification failed (M7.2 gate)"
            description: "{{ $value }} signature failures detected"
        
        # M7.3: Ranking accuracy
        - alert: M7_3_RankingAccuracyDegraded
          expr: memory_ranking_ndcg_weighted < 0.88
          for: 5m
          labels:
            severity: warning
            phase: "7"
          annotations:
            summary: "Ranking accuracy below SLO (M7.3 gate)"
            description: "NDCG@10 = {{ $value | humanizePercentage }} (target: >= 0.88)"
        
        # M7.4: Edit latency
        - alert: M7_4_EditLatencyHigh
          expr: histogram_quantile(0.99, memory_edit_latency_seconds) > 2.0
          for: 5m
          labels:
            severity: warning
            phase: "7"
          annotations:
            summary: "Edit latency P99 exceeds SLO (M7.4 gate)"
            description: "P99 latency = {{ $value | humanizeDuration }} (target: < 2s)"
        
        # M7.5: Storage overhead
        - alert: M7_5_StorageOverheadHigh
          expr: memory_storage_overhead_ratio > 1.5
          for: 10m
          labels:
            severity: warning
            phase: "7"
          annotations:
            summary: "Storage overhead exceeds SLO (M7.5 gate)"
            description: "Overhead ratio = {{ $value | humanizePercentage }} (target: <= 1.5x)"
```

### Grafana Dashboard (Phase 7)

```json
{
  "dashboard": {
    "title": "Poimen Memory - Phase 7 SLO Dashboard",
    "panels": [
      {
        "title": "M7.1 Rebuild Parity",
        "targets": [
          {
            "expr": "memory_rebuild_checksum_matches_total / (memory_rebuild_checksum_matches_total + memory_rebuild_checksum_mismatches_total)"
          }
        ],
        "alert_threshold": 1.0,
        "description": "Must be 100% (gate M7.1)"
      },
      {
        "title": "M7.3 Ranking Accuracy (NDCG@10 weighted)",
        "targets": [
          {
            "expr": "memory_ranking_ndcg_weighted"
          }
        ],
        "alert_threshold": 0.88,
        "description": "Target: >= 0.88 (gate M7.3)"
      },
      {
        "title": "M7.4 Edit Latency P99",
        "targets": [
          {
            "expr": "histogram_quantile(0.99, memory_edit_latency_seconds)"
          }
        ],
        "alert_threshold": 2.0,
        "description": "Target: < 2s (gate M7.4)"
      },
      {
        "title": "M7.5 Storage Overhead",
        "targets": [
          {
            "expr": "memory_storage_overhead_ratio"
          }
        ],
        "alert_threshold": 1.5,
        "description": "Target: <= 1.5x (gate M7.5)"
      }
    ]
  }
}
```

---

## 11. Metrics & SLOs

### Phase 7 Composition Gate

```yaml
gates:
  - name: M7.1_version_parity
    type: rebuild_checksum_match
    threshold: 100%
    sla: "All rebuilds must produce identical checksums"
    
  - name: M7.2_audit_trail_integrity
    type: signature_verification
    threshold: 100%
    sla: "Zero tampered audit events"
    
  - name: M7.3_ranking_accuracy
    type: ndcg_weighted_by_confidence
    threshold: ">= 0.88"
    sla: "NDCG@10 with multi-signal ranking >= 0.88"
    
  - name: M7.4_chunk_edit_latency
    type: p99_latency
    threshold: "<= 2s"
    sla: "99th percentile edit completion <= 2 seconds"
    
  - name: M7.5_version_space
    type: storage_overhead
    threshold: "<= 1.5x"
    sla: "Versioned storage overhead <= 1.5x original"
```

---

## 11. Implementation Roadmap

### Phase 7.1: Schema & Migration (Week 1)
- [ ] Create chunk_versions table
- [ ] Create audit_events table (partitioned)
- [ ] Create ranking_signals table
- [ ] Migration script (6.x → 7.x)
- [ ] Backfill ranking_signals from existing data

### Phase 7.2: Versioning API (Week 2)
- [ ] GET /memory/chunks/{id} (with version history)
- [ ] PATCH /memory/chunks/{id} (edit + create version)
- [ ] GET /memory/chunks/{id}/versions (list all versions)
- [ ] POST /memory/chunks/{id}/versions/{v}/revert (rollback)
- [ ] Tests: 50+ unit + integration

### Phase 7.3: Audit Trail (Week 2)
- [ ] Append-only event logging
- [ ] Signature verification (cryptographic)
- [ ] GET /memory/audit?resource_id={id} (query trail)
- [ ] Backup to S3 (cold storage)
- [ ] Tests: 30+ unit

### Phase 7.4: Multi-Signal Ranking (Week 3)
- [ ] Compute confidence from extraction pipeline
- [ ] Compute recency (last_verified_at)
- [ ] Compute community_signal (PageRank)
- [ ] Normalize BM25 scores
- [ ] Integrate into query ranking (hybrid_search.rs)
- [ ] Tests: 40+ unit

### Phase 7.5: Deterministic Rebuild (Week 3)
- [ ] Snapshot + version tracking
- [ ] Replay audit events
- [ ] Parity verification (M7.1 gate)
- [ ] Tests: 20+ integration

### Phase 7.6: Documentation & SLOs (Week 4)
- [ ] API documentation
- [ ] Audit trail guide
- [ ] Editing workflow guide
- [ ] SLO dashboards (Prometheus)
- [ ] Runbooks (troubleshooting)

---

## 12. Key Design Decisions

| Decision | Choice | Rationale |
|----------|--------|-----------|
| Versioning | Immutable (Option A) | Clearer lineage, no tombstones |
| Signals | 4 equal weights (25% each) | Balanced, interpretable |
| Audit storage | Append-only JSONL + S3 | Immutable, queryable, archivable |
| Rebuild parity | Signature verification | Fast (O(1)), detects corruption |
| Edit latency | P99 < 2s | Real-time editing UX |
| Confidence source | LLM extraction pipeline | Aligned with Phase 5 |
| Recency decay | Exponential (τ=30d) | Standard information theory |
| Community signal | PageRank + in-degree | Graph-theoretic, proven |

---

## 13. Risks & Mitigations

| Risk | Impact | Mitigation |
|------|--------|-----------|
| Version explosion (storage) | 1.5-2x overhead | Compression, periodic archival, TTL cleanup |
| Audit log query slowness | Query latency spikes | Partitioning by time, materialized views |
| Signature verification false positives | Trust erosion | Comprehensive testing, HSM backup |
| Community signal staleness | Ranking drift | Recompute PageRank daily (off-peak) |
| Editing conflicts | Concurrent edit loss | Optimistic locking via version number |

---

## 14. Success Criteria

✅ **Phase 7 Complete When:**
1. All 5 composition gates pass (M7.1-M7.5)
2. 150+ tests passing (unit + integration)
3. NDCG@10 with weighted confidence >= 0.88
4. Edit latency P99 < 2s
5. Storage overhead <= 1.5x
6. Audit trail 100% immutable (0 tampering detected)
7. Full documentation + runbooks
8. Production SLA: 99.9% availability

---

## 15. References

- **Versioning**: Git model (immutable commits, branches = versions)
- **Ranking signals**: Okapi BM25 + PageRank + temporal decay (arXiv:1802.05365)
- **Audit trails**: NIST SP 800-92 (log management)
- **Parity checks**: Deterministic replay (Riak design, §3.2)
- **Temporal semantics**: ISO 8601 + valid_from/valid_to

---

**Status**: 🔵 **DESIGN COMPLETE - READY FOR PHASE 7.1 IMPLEMENTATION**

Next: Review & finalize schema, then begin Phase 7.1 (migrations + versioning API).