poimen-memory/docs/PHASE_7_TEMPORAL_RAGA_INGEST_DESIGN.md

Phase 7: Temporal-RAGA-Ingest Architecture Design

Status: 🔵 Design Phase (No code yet)

---

1. Executive Summary

Phase 7 extends Poimen Memory with:

1. Temporal-Aware Knowledge Graph (time-series semantic versioning)
2. RAGA Ingest Pipeline (Retrieval-Augmented Graph Architecture)
3. Chunk Editing & Versioning (immutable append-only audit trail)
4. Multi-Signal Ranking (4 signals: confidence, recency, community, BM25)

Key Innovation: Enable interactive chunk editing with full temporal lineage while maintaining deterministic rebuilds and append-only audit logs.

---

2. Architecture Overview

Current State (Phase 6)

┌─────────────────┐
│ Input Content   │
└────────┬────────┘
         │
         ↓
┌─────────────────────────────────────┐
│ Extraction Pipeline (Phase 3)        │
│ ├─ Entity Linking (Phase 5.1)       │
│ ├─ Inference (Phase 5.2)            │
│ └─ Query Reasoning (Phase 5.3)      │
└────────┬────────────────────────────┘
         │
         ↓
┌─────────────────────────────────────┐
│ Memory Storage (Phase 2)             │
│ ├─ memory_entity (L1/L2)            │
│ ├─ memory_edge (relationships)      │
│ └─ memory_vector (embeddings)       │
└────────┬────────────────────────────┘
         │
         ↓
┌─────────────────────────────────────┐
│ Query & Retrieval (Phase 4 + 6)     │
│ ├─ Semantic search                  │
│ ├─ Lexical search                   │
│ └─ Hybrid ranking (RRF)             │
└─────────────────────────────────────┘

Phase 7 Extension

┌──────────────────────────────────────────────────────────────┐
│ TEMPORAL-RAGA-INGEST ARCHITECTURE                            │
└──────────────────────────────────────────────────────────────┘

INPUT LAYER (Temporal-Aware):
  ├─ Streaming ingest (Kafka)
  ├─ Batch ingest (S3/Git)
  ├─ Interactive edits (UI)
  └─ Time-tagged: {content, timestamp, editor, reason}

                    ↓

EXTRACTION LAYER (LLM-Gated):
  ├─ Entity extraction (with confidence)
  ├─ Relationship inference
  ├─ Temporal fact alignment (when/how long)
  └─ Conflict detection (contradictions)

                    ↓

VERSIONING LAYER (Append-Only):
  ├─ chunk_versions: {chunk_id, version, content, editor, timestamp}
  ├─ entity_versions: {entity_id, version, properties, confidence}
  ├─ edge_versions: {edge_id, version, source, relation, target}
  └─ audit_trail: All mutations logged immutably

                    ↓

STORAGE LAYER (Multi-Signal Ranking):
  ├─ memory_entity: entity + confidence + temporal_bounds
  ├─ memory_edge: edge + recency_score + community_signal
  ├─ memory_vector: embedding + ranking_signals
  └─ chunk_metadata: version_history, edit_count, quality_score

                    ↓

QUERY LAYER (Context-Aware Retrieval):
  ├─ Semantic search (Phase 4.1)
  ├─ Lexical search (Phase 4.2)
  ├─ Temporal filtering (when is this true?)
  ├─ Community detection (Phase 4.3)
  ├─ Multi-signal ranking (Phase 7 NEW)
  └─ Deterministic tie-breaking (version + timestamp)

---

3. Chunk Editing Semantics

Current: Read-Only

memory_entity (immutable)
  ├─ id: UUID
  ├─ name: String
  ├─ properties: JSON (frozen)
  └─ created_at: Timestamp

Phase 7: Versioned + Editable

Option A: Immutable Versions (Recommended)

-- Chunks are immutable; edits create new versions
CREATE TABLE chunk_versions (
  id UUID,
  chunk_id UUID,              -- Stable reference
  version INT,                -- 1, 2, 3...
  content TEXT,
  properties JSONB,
  edited_by VARCHAR(255),
  edit_reason TEXT,
  confidence FLOAT,           -- 0.0-1.0
  temporal_start TIMESTAMP,
  temporal_end TIMESTAMP,
  is_current BOOLEAN,
  created_at TIMESTAMP,
  PRIMARY KEY (chunk_id, version)
);

-- Active chunk always = is_current = true
SELECT * FROM chunk_versions WHERE chunk_id = $1 AND is_current = true;

-- Audit trail = historical versions
SELECT * FROM chunk_versions WHERE chunk_id = $1 ORDER BY version;

Example:

chunk_id: ent-123 (Kubernetes entity)

Version 1:
  content: "Kubernetes: Container orchestrator by Google"
  confidence: 0.85
  edited_by: alice
  created_at: 2025-01-30 10:00:00
  is_current: false

Version 2:
  content: "Kubernetes: Container orchestrator by Google, open-source"
  confidence: 0.92
  edited_by: bob
  created_at: 2025-01-31 14:30:00
  is_current: true

Option B: Soft Deletes (Alternative)

CREATE TABLE chunk_versions (
  id UUID,
  chunk_id UUID,
  version INT,
  content TEXT,
  is_deleted BOOLEAN,        -- soft delete
  deleted_by VARCHAR(255),
  PRIMARY KEY (chunk_id, version)
);

Verdict: Use Option A (Immutable) — clearer lineage, no "tombstone" ambiguity.

---

4. Ranking Formula (4 Signals)

The 4 Signals

Signal 1: Confidence (LLM Assessment)

confidence ∈ [0.0, 1.0]

Source: LLM extraction pipeline (Phase 5.2)
  ├─ 0.0-0.3: Uncertain (likely noise)
  ├─ 0.3-0.6: Moderate confidence
  ├─ 0.6-0.85: High confidence
  └─ 0.85-1.0: Verified/human-edited

Weight: 25%

Signal 2: Recency (Temporal Freshness)

recency_score = exp(-(now() - last_verified) / τ)
  where τ = 30 days (half-life)

Examples:
  ├─ Last verified today: 1.00
  ├─ Last verified 7 days ago: 0.78
  ├─ Last verified 30 days ago: 0.50
  ├─ Last verified 90 days ago: 0.12
  └─ Last verified 365 days ago: 0.0001

Weight: 25%

Signal 3: Community Signal (Graph Centrality)

community_score = pagerank_score * in_degree_normalized

Where:
  ├─ pagerank_score: Eigenvector centrality in entity graph
  │  └─ High if many entities link to this entity
  ├─ in_degree_normalized: (in_degree / max_in_degree) * 0.5
  │  └─ Capped at 0.5 to avoid dominance
  └─ Result: [0.0, 1.0]

Examples:
  ├─ "Kubernetes": 0.92 (highly connected)
  ├─ "Docker": 0.87 (related to 15 entities)
  └─ "ObscureLibrary": 0.15 (1-2 links only)

Weight: 25%

Signal 4: Lexical Ranking (BM25 from OpenSearch)

bm25_score = OpenSearch BM25 normalized

Source: Full-text index (existing Phase 4.2)
  ├─ Raw BM25: [0, ∞)
  ├─ Normalized: [0, 1.0]
  │  └─ sigmoid(bm25 / 10.0) for smoothing
  └─ Higher: More textually relevant to query

Examples:
  ├─ Exact match: 0.95+
  ├─ Semantic match: 0.60-0.80
  └─ Weak match: 0.10-0.30

Weight: 25%

Final Ranking Formula

final_score = 0.25 * confidence
            + 0.25 * recency_score
            + 0.25 * community_score
            + 0.25 * bm25_normalized

Range: [0.0, 1.0]

Tie-breaking (deterministic):
  1. Higher final_score wins
  2. If tied: newer version_timestamp wins
  3. If still tied: lexicographically by chunk_id

Example Calculation

Entity: "Kubernetes"

Signals:
  ├─ Confidence: 0.92 (human-verified, Phase 5.4)
  ├─ Recency: 0.88 (last verified 5 days ago)
  ├─ Community: 0.85 (PageRank in entity graph)
  └─ BM25: 0.78 (matched query terms)

Calculation:
  final_score = 0.25*0.92 + 0.25*0.88 + 0.25*0.85 + 0.25*0.78
              = 0.23 + 0.22 + 0.2125 + 0.195
              = 0.8575

Ranking: Top result (if query relevance high)

---

5. Audit Trail Strategy

Append-Only Event Log (JSONL)

{
  "id": "audit-evt-123",
  "timestamp": "2025-01-30T14:30:45Z",
  "event_type": "chunk_edited|chunk_created|chunk_verified|chunk_deleted",
  "actor": "[email protected]",
  "resource": {
    "type": "chunk",
    "id": "ent-123",
    "version": 2
  },
  "action": {
    "operation": "update",
    "old_value": {
      "content": "Kubernetes: Container orchestrator by Google",
      "confidence": 0.85
    },
    "new_value": {
      "content": "Kubernetes: Container orchestrator by Google, open-source CNCF project",
      "confidence": 0.92
    }
  },
  "reason": "Added CNCF affiliation after verification",
  "signature": "sha256(timestamp + actor + resource + action)"
}

Storage & Queries

-- Append-only event log
CREATE TABLE audit_events (
  id UUID PRIMARY KEY,
  timestamp TIMESTAMP NOT NULL,
  event_type VARCHAR(50),
  actor VARCHAR(255),
  resource_type VARCHAR(50),
  resource_id UUID,
  version INT,
  action JSONB,
  reason TEXT,
  signature VARCHAR(256),
  created_at TIMESTAMP DEFAULT now()
) PARTITION BY RANGE (timestamp);  -- Monthly partitions

-- Query: What edits were made to entity?
SELECT * FROM audit_events
WHERE resource_type = 'chunk' AND resource_id = $chunk_id
ORDER BY timestamp DESC;

-- Query: Who changed what, when?
SELECT timestamp, actor, reason, action
FROM audit_events
WHERE resource_id = $chunk_id
ORDER BY timestamp;

-- Query: Verify integrity (signature validation)
SELECT * FROM audit_events
WHERE signature != sha256(...)
LIMIT 100;  -- Detect tampering

Immutability Guarantees

1. No UPDATE on audit_events table
   ├─ Only INSERT is allowed
   └─ Policy: ROW SECURITY

2. Separate write vs read storage
   ├─ Write: Hot log (Kafka JSONL file)
   ├─ Read: Cold archive (S3 + compression)
   └─ Sync: Daily batch to S3

3. Cryptographic signing
   ├─ Each event: signature = sha256(timestamp || actor || resource || action)
   ├─ Chain of custody: timestamp chain prevents reordering
   └─ Verification: Detect tampering in O(1) time

---

6. Schema Extensions

New Tables (Phase 7)

-- Versioning
CREATE TABLE chunk_versions (
  id UUID PRIMARY KEY,
  chunk_id UUID NOT NULL,              -- Stable reference
  version INT NOT NULL,
  content TEXT,
  properties JSONB,
  edited_by VARCHAR(255),
  edit_reason TEXT,
  confidence FLOAT DEFAULT 0.5,
  temporal_start TIMESTAMP,
  temporal_end TIMESTAMP,
  is_current BOOLEAN,
  created_at TIMESTAMP DEFAULT now(),
  UNIQUE(chunk_id, version),
  FOREIGN KEY (chunk_id) REFERENCES memory_entity(id)
);

-- Audit trail
CREATE TABLE audit_events (
  id UUID PRIMARY KEY,
  timestamp TIMESTAMP NOT NULL,
  event_type VARCHAR(50) NOT NULL,
  actor VARCHAR(255),
  resource_type VARCHAR(50),
  resource_id UUID,
  version INT,
  action JSONB,
  reason TEXT,
  signature VARCHAR(256),
  created_at TIMESTAMP DEFAULT now()
) PARTITION BY RANGE (timestamp);

-- Ranking signals
CREATE TABLE ranking_signals (
  id UUID PRIMARY KEY,
  entity_id UUID NOT NULL,
  signal_type VARCHAR(50),  -- "confidence"|"recency"|"community"|"bm25"
  signal_value FLOAT,
  computed_at TIMESTAMP,
  FOREIGN KEY (entity_id) REFERENCES memory_entity(id)
);

-- Query rankings (for metrics)
CREATE TABLE query_rankings (
  id UUID PRIMARY KEY,
  query_id UUID,
  chunk_id UUID,
  rank INT,
  final_score FLOAT,
  signal_breakdown JSONB,  -- {confidence, recency, community, bm25}
  computed_at TIMESTAMP,
  FOREIGN KEY (chunk_id) REFERENCES memory_entity(id)
);

Schema Modifications

-- Add to existing memory_entity
ALTER TABLE memory_entity ADD COLUMN (
  confidence FLOAT DEFAULT 0.5,
  last_verified_at TIMESTAMP,
  edit_count INT DEFAULT 0,
  current_version INT DEFAULT 1
);

-- Add to existing memory_edge
ALTER TABLE memory_edge ADD COLUMN (
  confidence FLOAT DEFAULT 0.5,
  recency_score FLOAT DEFAULT 1.0,
  community_signal FLOAT DEFAULT 0.5,
  last_updated_at TIMESTAMP
);

---

7. Editing Workflow

User Edits a Chunk

1. User opens chunk in UI
   └─ GET /memory/chunks/{id}
      └─ Returns: current_version + full version history

2. User modifies content
   └─ Content changes: "Kubernetes: Container orchestrator"
                       → "Kubernetes: Container orchestrator (CNCF project)"

3. User submits edit
   └─ PATCH /memory/chunks/{id}
      └─ Payload: {
           "new_content": "...",
           "edit_reason": "Added CNCF affiliation",
           "confidence": 0.92
         }

4. Memory service:
   a. Validate edit (optional LLM gate for quality)
   b. Create new version:
      INSERT chunk_versions (chunk_id=id, version=2, content='...', ...)
   c. Update is_current: chunk_versions SET is_current=false WHERE version=1
   d. Audit log:
      INSERT audit_events (event_type='chunk_edited', resource_id=id, ...)
   e. Recompute signals:
      ├─ Confidence: extracted_confidence (or manual 0.92)
      ├─ Recency: 1.0 (just edited)
      ├─ Community: fetch from graph
      └─ BM25: reindex OpenSearch
   f. Return: {chunk_id, new_version, status: "OK"}

5. Query uses latest version:
   └─ SELECT * FROM chunk_versions
      WHERE chunk_id = $1 AND is_current = true

---

8. Deterministic Rebuild (Parity Check)

Challenge

With versioning, which version do we rebuild from?

Solution: Version Snapshot + Audit Replay

Rebuild Process:

1. Snapshot current state:
   SELECT chunk_id, max(version) as latest_version
   FROM chunk_versions
   WHERE is_current = true
   → SHA256 hash of {chunk_id, version, content}*
   → Store: rebuild_checksum_v1

2. From event log, replay all edits since T0:
   SELECT * FROM audit_events
   WHERE timestamp >= $baseline_timestamp
   ORDER BY timestamp ASC

3. Apply edits sequentially:
   For each audit_event:
     ├─ Create new chunk_version
     ├─ Mark old is_current = false
     ├─ Compute signals
     └─ Log event

4. Verify parity:
   SELECT chunk_id, max(version)
   FROM chunk_versions
   WHERE is_current = true
   → SHA256 hash
   → Check: rebuild_checksum_v2 == rebuild_checksum_v1
   → If match: ✅ Parity verified (M2.8 gate)
   → If diff: ❌ Rebuild failed (investigate)

---

9. Integration Points

Phase 4 (Retrieval) → Phase 7 (Ranking)

Current (Phase 4):

Query → Semantic + Lexical → RRF Fusion → Top-10 results

Phase 7 (Enhanced):

Query → Semantic + Lexical → RRF Fusion
  ↓
Multi-Signal Ranking:
  ├─ Fetch confidence, recency, community from ranking_signals
  ├─ Compute bm25_normalized from search scores
  ├─ Calculate: final_score = 0.25*conf + 0.25*recency + 0.25*community + 0.25*bm25
  └─ Sort by final_score DESC, then timestamp DESC, then chunk_id ASC

→ Top-10 results (with confidence intervals)

Phase 5 (Synthesis) → Phase 7 (Confidence)

Current (Phase 5):

LLM extracts entities → Insert into memory_entity

Phase 7 (Enhanced):

LLM extracts entities + confidence
  ↓
INSERT chunk_versions (confidence=$extracted_confidence)
  ↓
Signals computed:
  ├─ Confidence: $extracted_confidence (0.0-1.0)
  ├─ Recency: 1.0 (new)
  ├─ Community: graph_analysis() (0.0 initially, grows as linked)
  └─ BM25: from indexing
  ↓
Track lineage: audit_events with {operation, old_value, new_value}

---

10. Metrics Emission & Prometheus Scraping

Metrics Endpoint

GET /metrics
├─ Auth: Authentik JWT (Bearer token)
├─ Format: Prometheus text format (OpenMetrics)
├─ Refresh: Real-time (computed on-demand)
└─ Timeout: 5 seconds

Metrics Architecture

┌─────────────────────────────────────────┐
│ Memory Service                          │
│ ├─ MetricsPersistence (in-memory cache) │
│ ├─ record_success(agent_id, latency)   │
│ ├─ record_error(agent_id, latency)     │
│ └─ export_prometheus()                  │
│    └─ Returns: Prometheus text format   │
└────────────┬────────────────────────────┘
             │ Emits every 5s
             ↓
┌─────────────────────────────────────────┐
│ Prometheus Server                       │
│ ├─ Scrape /metrics (15s interval)       │
│ ├─ Store TSDB (15-month retention)      │
│ └─ Alert rules (M7.1-M7.5 gates)        │
└────────────┬────────────────────────────┘
             │
             ↓
┌─────────────────────────────────────────┐
│ Dashboards (Grafana)                    │
│ ├─ Phase 7 SLO dashboard                │
│ ├─ Edit latency percentiles             │
│ ├─ Ranking accuracy (NDCG weighted)     │
│ └─ Storage overhead tracking            │
└─────────────────────────────────────────┘

Phase 7 Metrics

# HELP memory_chunk_edits_total Total chunk edits (create/update/delete)
# TYPE memory_chunk_edits_total counter
memory_chunk_edits_total{operation="create"} 1245
memory_chunk_edits_total{operation="update"} 3421
memory_chunk_edits_total{operation="delete"} 18

# HELP memory_chunk_versions_current Current version count per chunk
# TYPE memory_chunk_versions_current gauge
memory_chunk_versions_current{chunk_type="entity"} 5234
memory_chunk_versions_current{chunk_type="edge"} 2189

# HELP memory_edit_latency_seconds Edit operation latency
# TYPE memory_edit_latency_seconds histogram
memory_edit_latency_seconds_bucket{le="0.1"} 156
memory_edit_latency_seconds_bucket{le="0.5"} 1023
memory_edit_latency_seconds_bucket{le="1.0"} 3421
memory_edit_latency_seconds_bucket{le="2.0"} 4156  # P99 < 2s gate
memory_edit_latency_seconds_bucket{le="+Inf"} 4189
memory_edit_latency_seconds_sum 5234.56
memory_edit_latency_seconds_count 4189

# HELP memory_audit_events_total Total audit events logged
# TYPE memory_audit_events_total counter
memory_audit_events_total{event_type="chunk_edited"} 3421
memory_audit_events_total{event_type="chunk_created"} 1245
memory_audit_events_total{event_type="verification"} 892

# HELP memory_audit_signature_failures_total Signature verification failures
# TYPE memory_audit_signature_failures_total counter
memory_audit_signature_failures_total 0  # M7.2 gate: must be 0

# HELP memory_rebuild_checksum_matches_total Rebuild parity checks passed
# TYPE memory_rebuild_checksum_matches_total counter
memory_rebuild_checksum_matches_total 127  # M7.1 gate: 100%

# HELP memory_rebuild_checksum_mismatches_total Rebuild parity checks FAILED
# TYPE memory_rebuild_checksum_mismatches_total counter
memory_rebuild_checksum_mismatches_total 0  # M7.1 gate: must be 0

# HELP memory_ranking_ndcg_weighted Weighted NDCG@10 (confidence-adjusted)
# TYPE memory_ranking_ndcg_weighted gauge
memory_ranking_ndcg_weighted 0.882  # M7.3 gate: >= 0.88

# HELP memory_ranking_signal_contribution Contribution of each signal
# TYPE memory_ranking_signal_contribution gauge
memory_ranking_signal_contribution{signal="confidence"} 0.25
memory_ranking_signal_contribution{signal="recency"} 0.25
memory_ranking_signal_contribution{signal="community"} 0.25
memory_ranking_signal_contribution{signal="bm25"} 0.25

# HELP memory_storage_bytes Total storage bytes
# TYPE memory_storage_bytes gauge
memory_storage_bytes{type="current_versions"} 5368709120
memory_storage_bytes{type="all_versions"} 8053063680  # M7.5 gate: <= 1.5x
memory_storage_bytes{type="audit_log"} 2147483648

# HELP memory_storage_overhead_ratio Overhead ratio (versions / current)
# TYPE memory_storage_overhead_ratio gauge
memory_storage_overhead_ratio 1.25  # M7.5 gate: <= 1.5x

# HELP memory_confidence_distribution Confidence score distribution
# TYPE memory_confidence_distribution histogram
memory_confidence_distribution_bucket{le="0.3"} 234    # Uncertain
memory_confidence_distribution_bucket{le="0.6"} 1289   # Moderate
memory_confidence_distribution_bucket{le="0.85"} 4156  # High
memory_confidence_distribution_bucket{le="1.0"} 5234   # Verified

# HELP memory_recency_score_distribution Recency score distribution
# TYPE memory_recency_score_distribution gauge
memory_recency_score_avg 0.72  # Average recency (30 days half-life)
memory_recency_score_p50 0.78
memory_recency_score_p99 0.98

# HELP memory_community_score_distribution Community signal distribution
# TYPE memory_community_score_distribution gauge
memory_community_score_avg 0.45
memory_community_score_p50 0.42
memory_community_score_p99 0.91

Prometheus Scrape Config

# In homelab Prometheus values.yaml
prometheus:
  scrapeConfigs:
    - job_name: 'poimen-memory'
      kubernetes_sd_configs:
        - role: pod
          namespaces:
            names:
              - poimen
      relabel_configs:
        - source_labels: [__meta_kubernetes_pod_label_app]
          action: keep
          regex: memory-service
        - source_labels: [__meta_kubernetes_pod_container_port_name]
          action: keep
          regex: metrics
      scheme: https
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      scrape_interval: 15s
      scrape_timeout: 5s

SLO Alerts (Prometheus Rules)

# rules/poimen-memory-phase7.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: poimen-memory-phase7
  namespace: monitoring
spec:
  groups:
    - name: memory-phase7-gates
      interval: 30s
      rules:
        # M7.1: Rebuild parity
        - alert: M7_1_RebuildParityCheckFailed
          expr: increase(memory_rebuild_checksum_mismatches_total[5m]) > 0
          for: 1m
          labels:
            severity: critical
            phase: "7"
          annotations:
            summary: "Rebuild parity check failed (M7.1 gate)"
            description: "{{ $value }} checksum mismatches in last 5 minutes"
        
        # M7.2: Audit trail integrity
        - alert: M7_2_AuditSignatureFailure
          expr: increase(memory_audit_signature_failures_total[5m]) > 0
          for: 1m
          labels:
            severity: critical
            phase: "7"
          annotations:
            summary: "Audit signature verification failed (M7.2 gate)"
            description: "{{ $value }} signature failures detected"
        
        # M7.3: Ranking accuracy
        - alert: M7_3_RankingAccuracyDegraded
          expr: memory_ranking_ndcg_weighted < 0.88
          for: 5m
          labels:
            severity: warning
            phase: "7"
          annotations:
            summary: "Ranking accuracy below SLO (M7.3 gate)"
            description: "NDCG@10 = {{ $value | humanizePercentage }} (target: >= 0.88)"
        
        # M7.4: Edit latency
        - alert: M7_4_EditLatencyHigh
          expr: histogram_quantile(0.99, memory_edit_latency_seconds) > 2.0
          for: 5m
          labels:
            severity: warning
            phase: "7"
          annotations:
            summary: "Edit latency P99 exceeds SLO (M7.4 gate)"
            description: "P99 latency = {{ $value | humanizeDuration }} (target: < 2s)"
        
        # M7.5: Storage overhead
        - alert: M7_5_StorageOverheadHigh
          expr: memory_storage_overhead_ratio > 1.5
          for: 10m
          labels:
            severity: warning
            phase: "7"
          annotations:
            summary: "Storage overhead exceeds SLO (M7.5 gate)"
            description: "Overhead ratio = {{ $value | humanizePercentage }} (target: <= 1.5x)"

Grafana Dashboard (Phase 7)

{
  "dashboard": {
    "title": "Poimen Memory - Phase 7 SLO Dashboard",
    "panels": [
      {
        "title": "M7.1 Rebuild Parity",
        "targets": [
          {
            "expr": "memory_rebuild_checksum_matches_total / (memory_rebuild_checksum_matches_total + memory_rebuild_checksum_mismatches_total)"
          }
        ],
        "alert_threshold": 1.0,
        "description": "Must be 100% (gate M7.1)"
      },
      {
        "title": "M7.3 Ranking Accuracy (NDCG@10 weighted)",
        "targets": [
          {
            "expr": "memory_ranking_ndcg_weighted"
          }
        ],
        "alert_threshold": 0.88,
        "description": "Target: >= 0.88 (gate M7.3)"
      },
      {
        "title": "M7.4 Edit Latency P99",
        "targets": [
          {
            "expr": "histogram_quantile(0.99, memory_edit_latency_seconds)"
          }
        ],
        "alert_threshold": 2.0,
        "description": "Target: < 2s (gate M7.4)"
      },
      {
        "title": "M7.5 Storage Overhead",
        "targets": [
          {
            "expr": "memory_storage_overhead_ratio"
          }
        ],
        "alert_threshold": 1.5,
        "description": "Target: <= 1.5x (gate M7.5)"
      }
    ]
  }
}

---

11. Metrics & SLOs

Phase 7 Composition Gate

gates:
  - name: M7.1_version_parity
    type: rebuild_checksum_match
    threshold: 100%
    sla: "All rebuilds must produce identical checksums"
    
  - name: M7.2_audit_trail_integrity
    type: signature_verification
    threshold: 100%
    sla: "Zero tampered audit events"
    
  - name: M7.3_ranking_accuracy
    type: ndcg_weighted_by_confidence
    threshold: ">= 0.88"
    sla: "NDCG@10 with multi-signal ranking >= 0.88"
    
  - name: M7.4_chunk_edit_latency
    type: p99_latency
    threshold: "<= 2s"
    sla: "99th percentile edit completion <= 2 seconds"
    
  - name: M7.5_version_space
    type: storage_overhead
    threshold: "<= 1.5x"
    sla: "Versioned storage overhead <= 1.5x original"

---

11. Implementation Roadmap

Phase 7.1: Schema & Migration (Week 1)

• Create chunk_versions table
• Create audit_events table (partitioned)
• Create ranking_signals table
• Migration script (6.x → 7.x)
• Backfill ranking_signals from existing data

Phase 7.2: Versioning API (Week 2)

• GET /memory/chunks/{id} (with version history)
• PATCH /memory/chunks/{id} (edit + create version)
• GET /memory/chunks/{id}/versions (list all versions)
• POST /memory/chunks/{id}/versions/{v}/revert (rollback)
• Tests: 50+ unit + integration

Phase 7.3: Audit Trail (Week 2)

• Append-only event logging
• Signature verification (cryptographic)
• GET /memory/audit?resource_id={id} (query trail)
• Backup to S3 (cold storage)
• Tests: 30+ unit

Phase 7.4: Multi-Signal Ranking (Week 3)

• Compute confidence from extraction pipeline
• Compute recency (last_verified_at)
• Compute community_signal (PageRank)
• Normalize BM25 scores
• Integrate into query ranking (hybrid_search.rs)
• Tests: 40+ unit

Phase 7.5: Deterministic Rebuild (Week 3)

• Snapshot + version tracking
• Replay audit events
• Parity verification (M7.1 gate)
• Tests: 20+ integration

Phase 7.6: Documentation & SLOs (Week 4)

• API documentation
• Audit trail guide
• Editing workflow guide
• SLO dashboards (Prometheus)
• Runbooks (troubleshooting)

---

12. Key Design Decisions

Decision	Choice	Rationale
Versioning	Immutable (Option A)	Clearer lineage, no tombstones
Signals	4 equal weights (25% each)	Balanced, interpretable
Audit storage	Append-only JSONL + S3	Immutable, queryable, archivable
Rebuild parity	Signature verification	Fast (O(1)), detects corruption
Edit latency	P99 < 2s	Real-time editing UX
Confidence source	LLM extraction pipeline	Aligned with Phase 5
Recency decay	Exponential (τ=30d)	Standard information theory
Community signal	PageRank + in-degree	Graph-theoretic, proven

---

13. Risks & Mitigations

Risk	Impact	Mitigation
Version explosion (storage)	1.5-2x overhead	Compression, periodic archival, TTL cleanup
Audit log query slowness	Query latency spikes	Partitioning by time, materialized views
Signature verification false positives	Trust erosion	Comprehensive testing, HSM backup
Community signal staleness	Ranking drift	Recompute PageRank daily (off-peak)
Editing conflicts	Concurrent edit loss	Optimistic locking via version number

---

14. Success Criteria

✅ Phase 7 Complete When:

1. All 5 composition gates pass (M7.1-M7.5)
2. 150+ tests passing (unit + integration)
3. NDCG@10 with weighted confidence >= 0.88
4. Edit latency P99 < 2s
5. Storage overhead <= 1.5x
6. Audit trail 100% immutable (0 tampering detected)
7. Full documentation + runbooks
8. Production SLA: 99.9% availability

---

15. References

• Versioning: Git model (immutable commits, branches = versions)
• Ranking signals: Okapi BM25 + PageRank + temporal decay (arXiv:1802.05365)
• Audit trails: NIST SP 800-92 (log management)
• Parity checks: Deterministic replay (Riak design, §3.2)
• Temporal semantics: ISO 8601 + valid_from/valid_to

---

Status: 🔵 DESIGN COMPLETE - READY FOR PHASE 7.1 IMPLEMENTATION

Next: Review & finalize schema, then begin Phase 7.1 (migrations + versioning API).