Phase 7: Temporal-RAGA-Ingest Architecture Design (Complete)
📋 DESIGN DOCUMENT (18.7 KB) Architecture: ├─ Temporal-aware knowledge graph (versioning) ├─ RAGA ingest pipeline (Retrieval-Augmented Graph Architecture) ├─ Chunk editing with immutable audit trail └─ Multi-signal ranking (4 signals, 25% each) Key Sections: 1. Chunk Editing Semantics (Immutable versions) ├─ chunk_versions table (version 1, 2, 3...) ├─ is_current flag (which version is active) ├─ edited_by, edit_reason, confidence tracking └─ Example: Kubernetes entity v1 → v2 (added CNCF affiliation) 2. Ranking Formula (4 Equal Signals) ├─ Signal 1: Confidence (LLM extraction, 0.0-1.0) ├─ Signal 2: Recency (exponential decay, τ=30d) ├─ Signal 3: Community (PageRank + in-degree) ├─ Signal 4: BM25 (lexical relevance, normalized) └─ final_score = 0.25*conf + 0.25*recency + 0.25*community + 0.25*bm25 3. Audit Trail (Append-only immutable log) ├─ audit_events table (partitioned by timestamp) ├─ Every mutation logged: chunk_edited, created, verified, deleted ├─ Cryptographic signing (SHA256 for tamper detection) ├─ Queryable: Who changed what, when, why └─ Archive: Daily batch to S3 cold storage 4. Schema Extensions ├─ chunk_versions: id, chunk_id, version, content, confidence, is_current ├─ audit_events: id, timestamp, event_type, actor, resource_id, action, reason ├─ ranking_signals: id, entity_id, signal_type, signal_value └─ query_rankings: query_id, chunk_id, rank, final_score, signal_breakdown 5. Deterministic Rebuild (Parity Check - M2.8 extended) ├─ Snapshot current state ├─ Replay audit events in order ├─ Recompute all signals ├─ Verify: checksum_before == checksum_after └─ Detects corruption in O(1) time 6. Metrics Emission & Prometheus Scraping ├─ GET /metrics endpoint (Authentik protected) ├─ Real-time Prometheus format (OpenMetrics) ├─ Prometheus scrapes every 15s ├─ Grafana dashboard tracks Phase 7 SLOs └─ Alerts for M7.1-M7.5 gates Phase 7 Metrics (Prometheus): ├─ memory_chunk_edits_total (counter: create/update/delete) ├─ memory_edit_latency_seconds (histogram: P50/P99) ├─ memory_audit_events_total (counter: by event_type) ├─ memory_audit_signature_failures_total (counter: must be 0) ├─ memory_rebuild_checksum_matches_total (counter: parity checks) ├─ memory_ranking_ndcg_weighted (gauge: weighted accuracy) ├─ memory_storage_overhead_ratio (gauge: 1.5x max) ├─ memory_confidence_distribution (histogram: score buckets) ├─ memory_recency_score_* (gauge: avg/p50/p99) └─ memory_community_score_* (gauge: avg/p50/p99) SLO Alerts (Prometheus Rules): ├─ M7.1_RebuildParityCheckFailed (critical) ├─ M7_2_AuditSignatureFailure (critical) ├─ M7_3_RankingAccuracyDegraded (warning: NDCG < 0.88) ├─ M7_4_EditLatencyHigh (warning: P99 > 2s) └─ M7_5_StorageOverheadHigh (warning: ratio > 1.5x) Implementation Roadmap: ├─ Phase 7.1: Schema & Migrations (Week 1, ~200 LOC) ├─ Phase 7.2: Versioning API (Week 2, ~400 LOC, 50+ tests) ├─ Phase 7.3: Audit Trail (Week 2, ~300 LOC, 30+ tests) ├─ Phase 7.4: Multi-Signal Ranking (Week 3, ~350 LOC, 40+ tests) ├─ Phase 7.5: Deterministic Rebuild (Week 3, ~200 LOC, 20+ tests) └─ Phase 7.6: Documentation & SLOs (Week 4, ~500 LOC docs) Success Criteria: ✅ All 5 composition gates pass (M7.1-M7.5) ✅ 150+ tests (unit + integration) ✅ NDCG@10 weighted >= 0.88 (M7.3) ✅ Edit latency P99 < 2s (M7.4) ✅ Storage overhead <= 1.5x (M7.5) ✅ Audit trail 100% immutable (M7.2) ✅ Rebuild parity 100% (M7.1) ✅ Full documentation + runbooks Key Design Decisions: ├─ Versioning: Immutable (Option A, not Option B soft deletes) ├─ Signals: 4 equal weights (25% each, not weighted differently) ├─ Audit: Append-only JSONL + S3 (not mutable log) ├─ Rebuild: Signature verification (O(1), not full replay) ├─ Confidence: From LLM pipeline (Phase 5) ├─ Recency: Exponential decay τ=30d (standard info theory) ├─ Community: PageRank + in-degree (graph-theoretic) └─ Edit latency: P99 < 2s (real-time UX) Risks & Mitigations: ├─ Version explosion: Compression + archival + TTL cleanup ├─ Audit log query slowness: Partitioning + materialized views ├─ Signature false positives: Comprehensive testing + HSM backup ├─ Community signal staleness: Recompute PageRank daily └─ Concurrent edits: Optimistic locking via version number Integration Points: ├─ Phase 4 (Retrieval) → Multi-signal ranking ├─ Phase 5 (Synthesis) → Confidence extraction ├─ Phase 6 (Agents) → Metrics emission └─ Phase 7 (Versioning) → Deterministic rebuild References: ├─ Git model (immutable commits) ├─ Okapi BM25 + PageRank (arXiv:1802.05365) ├─ NIST SP 800-92 (audit logs) ├─ Riak parity checks (deterministic replay) └─ ISO 8601 (temporal semantics) Next: Architecture review, then Phase 7.1 (migrations)
This commit is contained in:
@@ -0,0 +1,989 @@
|
||||
# Phase 7: Temporal-RAGA-Ingest Architecture Design
|
||||
|
||||
**Status**: 🔵 Design Phase (No code yet)
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Summary
|
||||
|
||||
**Phase 7** extends Poimen Memory with:
|
||||
1. **Temporal-Aware Knowledge Graph** (time-series semantic versioning)
|
||||
2. **RAGA Ingest Pipeline** (Retrieval-Augmented Graph Architecture)
|
||||
3. **Chunk Editing & Versioning** (immutable append-only audit trail)
|
||||
4. **Multi-Signal Ranking** (4 signals: confidence, recency, community, BM25)
|
||||
|
||||
**Key Innovation**: Enable interactive **chunk editing** with full **temporal lineage** while maintaining **deterministic rebuilds** and **append-only audit logs**.
|
||||
|
||||
---
|
||||
|
||||
## 2. Architecture Overview
|
||||
|
||||
### Current State (Phase 6)
|
||||
```
|
||||
┌─────────────────┐
|
||||
│ Input Content │
|
||||
└────────┬────────┘
|
||||
│
|
||||
↓
|
||||
┌─────────────────────────────────────┐
|
||||
│ Extraction Pipeline (Phase 3) │
|
||||
│ ├─ Entity Linking (Phase 5.1) │
|
||||
│ ├─ Inference (Phase 5.2) │
|
||||
│ └─ Query Reasoning (Phase 5.3) │
|
||||
└────────┬────────────────────────────┘
|
||||
│
|
||||
↓
|
||||
┌─────────────────────────────────────┐
|
||||
│ Memory Storage (Phase 2) │
|
||||
│ ├─ memory_entity (L1/L2) │
|
||||
│ ├─ memory_edge (relationships) │
|
||||
│ └─ memory_vector (embeddings) │
|
||||
└────────┬────────────────────────────┘
|
||||
│
|
||||
↓
|
||||
┌─────────────────────────────────────┐
|
||||
│ Query & Retrieval (Phase 4 + 6) │
|
||||
│ ├─ Semantic search │
|
||||
│ ├─ Lexical search │
|
||||
│ └─ Hybrid ranking (RRF) │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Phase 7 Extension
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ TEMPORAL-RAGA-INGEST ARCHITECTURE │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
|
||||
INPUT LAYER (Temporal-Aware):
|
||||
├─ Streaming ingest (Kafka)
|
||||
├─ Batch ingest (S3/Git)
|
||||
├─ Interactive edits (UI)
|
||||
└─ Time-tagged: {content, timestamp, editor, reason}
|
||||
|
||||
↓
|
||||
|
||||
EXTRACTION LAYER (LLM-Gated):
|
||||
├─ Entity extraction (with confidence)
|
||||
├─ Relationship inference
|
||||
├─ Temporal fact alignment (when/how long)
|
||||
└─ Conflict detection (contradictions)
|
||||
|
||||
↓
|
||||
|
||||
VERSIONING LAYER (Append-Only):
|
||||
├─ chunk_versions: {chunk_id, version, content, editor, timestamp}
|
||||
├─ entity_versions: {entity_id, version, properties, confidence}
|
||||
├─ edge_versions: {edge_id, version, source, relation, target}
|
||||
└─ audit_trail: All mutations logged immutably
|
||||
|
||||
↓
|
||||
|
||||
STORAGE LAYER (Multi-Signal Ranking):
|
||||
├─ memory_entity: entity + confidence + temporal_bounds
|
||||
├─ memory_edge: edge + recency_score + community_signal
|
||||
├─ memory_vector: embedding + ranking_signals
|
||||
└─ chunk_metadata: version_history, edit_count, quality_score
|
||||
|
||||
↓
|
||||
|
||||
QUERY LAYER (Context-Aware Retrieval):
|
||||
├─ Semantic search (Phase 4.1)
|
||||
├─ Lexical search (Phase 4.2)
|
||||
├─ Temporal filtering (when is this true?)
|
||||
├─ Community detection (Phase 4.3)
|
||||
├─ Multi-signal ranking (Phase 7 NEW)
|
||||
└─ Deterministic tie-breaking (version + timestamp)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Chunk Editing Semantics
|
||||
|
||||
### Current: Read-Only
|
||||
```
|
||||
memory_entity (immutable)
|
||||
├─ id: UUID
|
||||
├─ name: String
|
||||
├─ properties: JSON (frozen)
|
||||
└─ created_at: Timestamp
|
||||
```
|
||||
|
||||
### Phase 7: Versioned + Editable
|
||||
|
||||
#### Option A: Immutable Versions (Recommended)
|
||||
```sql
|
||||
-- Chunks are immutable; edits create new versions
|
||||
CREATE TABLE chunk_versions (
|
||||
id UUID,
|
||||
chunk_id UUID, -- Stable reference
|
||||
version INT, -- 1, 2, 3...
|
||||
content TEXT,
|
||||
properties JSONB,
|
||||
edited_by VARCHAR(255),
|
||||
edit_reason TEXT,
|
||||
confidence FLOAT, -- 0.0-1.0
|
||||
temporal_start TIMESTAMP,
|
||||
temporal_end TIMESTAMP,
|
||||
is_current BOOLEAN,
|
||||
created_at TIMESTAMP,
|
||||
PRIMARY KEY (chunk_id, version)
|
||||
);
|
||||
|
||||
-- Active chunk always = is_current = true
|
||||
SELECT * FROM chunk_versions WHERE chunk_id = $1 AND is_current = true;
|
||||
|
||||
-- Audit trail = historical versions
|
||||
SELECT * FROM chunk_versions WHERE chunk_id = $1 ORDER BY version;
|
||||
```
|
||||
|
||||
**Example**:
|
||||
```
|
||||
chunk_id: ent-123 (Kubernetes entity)
|
||||
|
||||
Version 1:
|
||||
content: "Kubernetes: Container orchestrator by Google"
|
||||
confidence: 0.85
|
||||
edited_by: alice
|
||||
created_at: 2025-01-30 10:00:00
|
||||
is_current: false
|
||||
|
||||
Version 2:
|
||||
content: "Kubernetes: Container orchestrator by Google, open-source"
|
||||
confidence: 0.92
|
||||
edited_by: bob
|
||||
created_at: 2025-01-31 14:30:00
|
||||
is_current: true
|
||||
```
|
||||
|
||||
#### Option B: Soft Deletes (Alternative)
|
||||
```sql
|
||||
CREATE TABLE chunk_versions (
|
||||
id UUID,
|
||||
chunk_id UUID,
|
||||
version INT,
|
||||
content TEXT,
|
||||
is_deleted BOOLEAN, -- soft delete
|
||||
deleted_by VARCHAR(255),
|
||||
PRIMARY KEY (chunk_id, version)
|
||||
);
|
||||
```
|
||||
|
||||
**Verdict**: Use **Option A (Immutable)** — clearer lineage, no "tombstone" ambiguity.
|
||||
|
||||
---
|
||||
|
||||
## 4. Ranking Formula (4 Signals)
|
||||
|
||||
### The 4 Signals
|
||||
|
||||
#### Signal 1: Confidence (LLM Assessment)
|
||||
```
|
||||
confidence ∈ [0.0, 1.0]
|
||||
|
||||
Source: LLM extraction pipeline (Phase 5.2)
|
||||
├─ 0.0-0.3: Uncertain (likely noise)
|
||||
├─ 0.3-0.6: Moderate confidence
|
||||
├─ 0.6-0.85: High confidence
|
||||
└─ 0.85-1.0: Verified/human-edited
|
||||
|
||||
Weight: 25%
|
||||
```
|
||||
|
||||
#### Signal 2: Recency (Temporal Freshness)
|
||||
```
|
||||
recency_score = exp(-(now() - last_verified) / τ)
|
||||
where τ = 30 days (half-life)
|
||||
|
||||
Examples:
|
||||
├─ Last verified today: 1.00
|
||||
├─ Last verified 7 days ago: 0.78
|
||||
├─ Last verified 30 days ago: 0.50
|
||||
├─ Last verified 90 days ago: 0.12
|
||||
└─ Last verified 365 days ago: 0.0001
|
||||
|
||||
Weight: 25%
|
||||
```
|
||||
|
||||
#### Signal 3: Community Signal (Graph Centrality)
|
||||
```
|
||||
community_score = pagerank_score * in_degree_normalized
|
||||
|
||||
Where:
|
||||
├─ pagerank_score: Eigenvector centrality in entity graph
|
||||
│ └─ High if many entities link to this entity
|
||||
├─ in_degree_normalized: (in_degree / max_in_degree) * 0.5
|
||||
│ └─ Capped at 0.5 to avoid dominance
|
||||
└─ Result: [0.0, 1.0]
|
||||
|
||||
Examples:
|
||||
├─ "Kubernetes": 0.92 (highly connected)
|
||||
├─ "Docker": 0.87 (related to 15 entities)
|
||||
└─ "ObscureLibrary": 0.15 (1-2 links only)
|
||||
|
||||
Weight: 25%
|
||||
```
|
||||
|
||||
#### Signal 4: Lexical Ranking (BM25 from OpenSearch)
|
||||
```
|
||||
bm25_score = OpenSearch BM25 normalized
|
||||
|
||||
Source: Full-text index (existing Phase 4.2)
|
||||
├─ Raw BM25: [0, ∞)
|
||||
├─ Normalized: [0, 1.0]
|
||||
│ └─ sigmoid(bm25 / 10.0) for smoothing
|
||||
└─ Higher: More textually relevant to query
|
||||
|
||||
Examples:
|
||||
├─ Exact match: 0.95+
|
||||
├─ Semantic match: 0.60-0.80
|
||||
└─ Weak match: 0.10-0.30
|
||||
|
||||
Weight: 25%
|
||||
```
|
||||
|
||||
### Final Ranking Formula
|
||||
|
||||
```
|
||||
final_score = 0.25 * confidence
|
||||
+ 0.25 * recency_score
|
||||
+ 0.25 * community_score
|
||||
+ 0.25 * bm25_normalized
|
||||
|
||||
Range: [0.0, 1.0]
|
||||
|
||||
Tie-breaking (deterministic):
|
||||
1. Higher final_score wins
|
||||
2. If tied: newer version_timestamp wins
|
||||
3. If still tied: lexicographically by chunk_id
|
||||
```
|
||||
|
||||
### Example Calculation
|
||||
|
||||
```
|
||||
Entity: "Kubernetes"
|
||||
|
||||
Signals:
|
||||
├─ Confidence: 0.92 (human-verified, Phase 5.4)
|
||||
├─ Recency: 0.88 (last verified 5 days ago)
|
||||
├─ Community: 0.85 (PageRank in entity graph)
|
||||
└─ BM25: 0.78 (matched query terms)
|
||||
|
||||
Calculation:
|
||||
final_score = 0.25*0.92 + 0.25*0.88 + 0.25*0.85 + 0.25*0.78
|
||||
= 0.23 + 0.22 + 0.2125 + 0.195
|
||||
= 0.8575
|
||||
|
||||
Ranking: Top result (if query relevance high)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Audit Trail Strategy
|
||||
|
||||
### Append-Only Event Log (JSONL)
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "audit-evt-123",
|
||||
"timestamp": "2025-01-30T14:30:45Z",
|
||||
"event_type": "chunk_edited|chunk_created|chunk_verified|chunk_deleted",
|
||||
"actor": "[email protected]",
|
||||
"resource": {
|
||||
"type": "chunk",
|
||||
"id": "ent-123",
|
||||
"version": 2
|
||||
},
|
||||
"action": {
|
||||
"operation": "update",
|
||||
"old_value": {
|
||||
"content": "Kubernetes: Container orchestrator by Google",
|
||||
"confidence": 0.85
|
||||
},
|
||||
"new_value": {
|
||||
"content": "Kubernetes: Container orchestrator by Google, open-source CNCF project",
|
||||
"confidence": 0.92
|
||||
}
|
||||
},
|
||||
"reason": "Added CNCF affiliation after verification",
|
||||
"signature": "sha256(timestamp + actor + resource + action)"
|
||||
}
|
||||
```
|
||||
|
||||
### Storage & Queries
|
||||
|
||||
```sql
|
||||
-- Append-only event log
|
||||
CREATE TABLE audit_events (
|
||||
id UUID PRIMARY KEY,
|
||||
timestamp TIMESTAMP NOT NULL,
|
||||
event_type VARCHAR(50),
|
||||
actor VARCHAR(255),
|
||||
resource_type VARCHAR(50),
|
||||
resource_id UUID,
|
||||
version INT,
|
||||
action JSONB,
|
||||
reason TEXT,
|
||||
signature VARCHAR(256),
|
||||
created_at TIMESTAMP DEFAULT now()
|
||||
) PARTITION BY RANGE (timestamp); -- Monthly partitions
|
||||
|
||||
-- Query: What edits were made to entity?
|
||||
SELECT * FROM audit_events
|
||||
WHERE resource_type = 'chunk' AND resource_id = $chunk_id
|
||||
ORDER BY timestamp DESC;
|
||||
|
||||
-- Query: Who changed what, when?
|
||||
SELECT timestamp, actor, reason, action
|
||||
FROM audit_events
|
||||
WHERE resource_id = $chunk_id
|
||||
ORDER BY timestamp;
|
||||
|
||||
-- Query: Verify integrity (signature validation)
|
||||
SELECT * FROM audit_events
|
||||
WHERE signature != sha256(...)
|
||||
LIMIT 100; -- Detect tampering
|
||||
```
|
||||
|
||||
### Immutability Guarantees
|
||||
|
||||
```
|
||||
1. No UPDATE on audit_events table
|
||||
├─ Only INSERT is allowed
|
||||
└─ Policy: ROW SECURITY
|
||||
|
||||
2. Separate write vs read storage
|
||||
├─ Write: Hot log (Kafka JSONL file)
|
||||
├─ Read: Cold archive (S3 + compression)
|
||||
└─ Sync: Daily batch to S3
|
||||
|
||||
3. Cryptographic signing
|
||||
├─ Each event: signature = sha256(timestamp || actor || resource || action)
|
||||
├─ Chain of custody: timestamp chain prevents reordering
|
||||
└─ Verification: Detect tampering in O(1) time
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Schema Extensions
|
||||
|
||||
### New Tables (Phase 7)
|
||||
|
||||
```sql
|
||||
-- Versioning
|
||||
CREATE TABLE chunk_versions (
|
||||
id UUID PRIMARY KEY,
|
||||
chunk_id UUID NOT NULL, -- Stable reference
|
||||
version INT NOT NULL,
|
||||
content TEXT,
|
||||
properties JSONB,
|
||||
edited_by VARCHAR(255),
|
||||
edit_reason TEXT,
|
||||
confidence FLOAT DEFAULT 0.5,
|
||||
temporal_start TIMESTAMP,
|
||||
temporal_end TIMESTAMP,
|
||||
is_current BOOLEAN,
|
||||
created_at TIMESTAMP DEFAULT now(),
|
||||
UNIQUE(chunk_id, version),
|
||||
FOREIGN KEY (chunk_id) REFERENCES memory_entity(id)
|
||||
);
|
||||
|
||||
-- Audit trail
|
||||
CREATE TABLE audit_events (
|
||||
id UUID PRIMARY KEY,
|
||||
timestamp TIMESTAMP NOT NULL,
|
||||
event_type VARCHAR(50) NOT NULL,
|
||||
actor VARCHAR(255),
|
||||
resource_type VARCHAR(50),
|
||||
resource_id UUID,
|
||||
version INT,
|
||||
action JSONB,
|
||||
reason TEXT,
|
||||
signature VARCHAR(256),
|
||||
created_at TIMESTAMP DEFAULT now()
|
||||
) PARTITION BY RANGE (timestamp);
|
||||
|
||||
-- Ranking signals
|
||||
CREATE TABLE ranking_signals (
|
||||
id UUID PRIMARY KEY,
|
||||
entity_id UUID NOT NULL,
|
||||
signal_type VARCHAR(50), -- "confidence"|"recency"|"community"|"bm25"
|
||||
signal_value FLOAT,
|
||||
computed_at TIMESTAMP,
|
||||
FOREIGN KEY (entity_id) REFERENCES memory_entity(id)
|
||||
);
|
||||
|
||||
-- Query rankings (for metrics)
|
||||
CREATE TABLE query_rankings (
|
||||
id UUID PRIMARY KEY,
|
||||
query_id UUID,
|
||||
chunk_id UUID,
|
||||
rank INT,
|
||||
final_score FLOAT,
|
||||
signal_breakdown JSONB, -- {confidence, recency, community, bm25}
|
||||
computed_at TIMESTAMP,
|
||||
FOREIGN KEY (chunk_id) REFERENCES memory_entity(id)
|
||||
);
|
||||
```
|
||||
|
||||
### Schema Modifications
|
||||
|
||||
```sql
|
||||
-- Add to existing memory_entity
|
||||
ALTER TABLE memory_entity ADD COLUMN (
|
||||
confidence FLOAT DEFAULT 0.5,
|
||||
last_verified_at TIMESTAMP,
|
||||
edit_count INT DEFAULT 0,
|
||||
current_version INT DEFAULT 1
|
||||
);
|
||||
|
||||
-- Add to existing memory_edge
|
||||
ALTER TABLE memory_edge ADD COLUMN (
|
||||
confidence FLOAT DEFAULT 0.5,
|
||||
recency_score FLOAT DEFAULT 1.0,
|
||||
community_signal FLOAT DEFAULT 0.5,
|
||||
last_updated_at TIMESTAMP
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Editing Workflow
|
||||
|
||||
### User Edits a Chunk
|
||||
|
||||
```
|
||||
1. User opens chunk in UI
|
||||
└─ GET /memory/chunks/{id}
|
||||
└─ Returns: current_version + full version history
|
||||
|
||||
2. User modifies content
|
||||
└─ Content changes: "Kubernetes: Container orchestrator"
|
||||
→ "Kubernetes: Container orchestrator (CNCF project)"
|
||||
|
||||
3. User submits edit
|
||||
└─ PATCH /memory/chunks/{id}
|
||||
└─ Payload: {
|
||||
"new_content": "...",
|
||||
"edit_reason": "Added CNCF affiliation",
|
||||
"confidence": 0.92
|
||||
}
|
||||
|
||||
4. Memory service:
|
||||
a. Validate edit (optional LLM gate for quality)
|
||||
b. Create new version:
|
||||
INSERT chunk_versions (chunk_id=id, version=2, content='...', ...)
|
||||
c. Update is_current: chunk_versions SET is_current=false WHERE version=1
|
||||
d. Audit log:
|
||||
INSERT audit_events (event_type='chunk_edited', resource_id=id, ...)
|
||||
e. Recompute signals:
|
||||
├─ Confidence: extracted_confidence (or manual 0.92)
|
||||
├─ Recency: 1.0 (just edited)
|
||||
├─ Community: fetch from graph
|
||||
└─ BM25: reindex OpenSearch
|
||||
f. Return: {chunk_id, new_version, status: "OK"}
|
||||
|
||||
5. Query uses latest version:
|
||||
└─ SELECT * FROM chunk_versions
|
||||
WHERE chunk_id = $1 AND is_current = true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Deterministic Rebuild (Parity Check)
|
||||
|
||||
### Challenge
|
||||
With versioning, **which version** do we rebuild from?
|
||||
|
||||
### Solution: Version Snapshot + Audit Replay
|
||||
|
||||
```
|
||||
Rebuild Process:
|
||||
|
||||
1. Snapshot current state:
|
||||
SELECT chunk_id, max(version) as latest_version
|
||||
FROM chunk_versions
|
||||
WHERE is_current = true
|
||||
→ SHA256 hash of {chunk_id, version, content}*
|
||||
→ Store: rebuild_checksum_v1
|
||||
|
||||
2. From event log, replay all edits since T0:
|
||||
SELECT * FROM audit_events
|
||||
WHERE timestamp >= $baseline_timestamp
|
||||
ORDER BY timestamp ASC
|
||||
|
||||
3. Apply edits sequentially:
|
||||
For each audit_event:
|
||||
├─ Create new chunk_version
|
||||
├─ Mark old is_current = false
|
||||
├─ Compute signals
|
||||
└─ Log event
|
||||
|
||||
4. Verify parity:
|
||||
SELECT chunk_id, max(version)
|
||||
FROM chunk_versions
|
||||
WHERE is_current = true
|
||||
→ SHA256 hash
|
||||
→ Check: rebuild_checksum_v2 == rebuild_checksum_v1
|
||||
→ If match: ✅ Parity verified (M2.8 gate)
|
||||
→ If diff: ❌ Rebuild failed (investigate)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. Integration Points
|
||||
|
||||
### Phase 4 (Retrieval) → Phase 7 (Ranking)
|
||||
|
||||
**Current** (Phase 4):
|
||||
```
|
||||
Query → Semantic + Lexical → RRF Fusion → Top-10 results
|
||||
```
|
||||
|
||||
**Phase 7** (Enhanced):
|
||||
```
|
||||
Query → Semantic + Lexical → RRF Fusion
|
||||
↓
|
||||
Multi-Signal Ranking:
|
||||
├─ Fetch confidence, recency, community from ranking_signals
|
||||
├─ Compute bm25_normalized from search scores
|
||||
├─ Calculate: final_score = 0.25*conf + 0.25*recency + 0.25*community + 0.25*bm25
|
||||
└─ Sort by final_score DESC, then timestamp DESC, then chunk_id ASC
|
||||
|
||||
→ Top-10 results (with confidence intervals)
|
||||
```
|
||||
|
||||
### Phase 5 (Synthesis) → Phase 7 (Confidence)
|
||||
|
||||
**Current** (Phase 5):
|
||||
```
|
||||
LLM extracts entities → Insert into memory_entity
|
||||
```
|
||||
|
||||
**Phase 7** (Enhanced):
|
||||
```
|
||||
LLM extracts entities + confidence
|
||||
↓
|
||||
INSERT chunk_versions (confidence=$extracted_confidence)
|
||||
↓
|
||||
Signals computed:
|
||||
├─ Confidence: $extracted_confidence (0.0-1.0)
|
||||
├─ Recency: 1.0 (new)
|
||||
├─ Community: graph_analysis() (0.0 initially, grows as linked)
|
||||
└─ BM25: from indexing
|
||||
↓
|
||||
Track lineage: audit_events with {operation, old_value, new_value}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. Metrics Emission & Prometheus Scraping
|
||||
|
||||
### Metrics Endpoint
|
||||
|
||||
```
|
||||
GET /metrics
|
||||
├─ Auth: Authentik JWT (Bearer token)
|
||||
├─ Format: Prometheus text format (OpenMetrics)
|
||||
├─ Refresh: Real-time (computed on-demand)
|
||||
└─ Timeout: 5 seconds
|
||||
```
|
||||
|
||||
### Metrics Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Memory Service │
|
||||
│ ├─ MetricsPersistence (in-memory cache) │
|
||||
│ ├─ record_success(agent_id, latency) │
|
||||
│ ├─ record_error(agent_id, latency) │
|
||||
│ └─ export_prometheus() │
|
||||
│ └─ Returns: Prometheus text format │
|
||||
└────────────┬────────────────────────────┘
|
||||
│ Emits every 5s
|
||||
↓
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Prometheus Server │
|
||||
│ ├─ Scrape /metrics (15s interval) │
|
||||
│ ├─ Store TSDB (15-month retention) │
|
||||
│ └─ Alert rules (M7.1-M7.5 gates) │
|
||||
└────────────┬────────────────────────────┘
|
||||
│
|
||||
↓
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Dashboards (Grafana) │
|
||||
│ ├─ Phase 7 SLO dashboard │
|
||||
│ ├─ Edit latency percentiles │
|
||||
│ ├─ Ranking accuracy (NDCG weighted) │
|
||||
│ └─ Storage overhead tracking │
|
||||
└─────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Phase 7 Metrics
|
||||
|
||||
```prometheus
|
||||
# HELP memory_chunk_edits_total Total chunk edits (create/update/delete)
|
||||
# TYPE memory_chunk_edits_total counter
|
||||
memory_chunk_edits_total{operation="create"} 1245
|
||||
memory_chunk_edits_total{operation="update"} 3421
|
||||
memory_chunk_edits_total{operation="delete"} 18
|
||||
|
||||
# HELP memory_chunk_versions_current Current version count per chunk
|
||||
# TYPE memory_chunk_versions_current gauge
|
||||
memory_chunk_versions_current{chunk_type="entity"} 5234
|
||||
memory_chunk_versions_current{chunk_type="edge"} 2189
|
||||
|
||||
# HELP memory_edit_latency_seconds Edit operation latency
|
||||
# TYPE memory_edit_latency_seconds histogram
|
||||
memory_edit_latency_seconds_bucket{le="0.1"} 156
|
||||
memory_edit_latency_seconds_bucket{le="0.5"} 1023
|
||||
memory_edit_latency_seconds_bucket{le="1.0"} 3421
|
||||
memory_edit_latency_seconds_bucket{le="2.0"} 4156 # P99 < 2s gate
|
||||
memory_edit_latency_seconds_bucket{le="+Inf"} 4189
|
||||
memory_edit_latency_seconds_sum 5234.56
|
||||
memory_edit_latency_seconds_count 4189
|
||||
|
||||
# HELP memory_audit_events_total Total audit events logged
|
||||
# TYPE memory_audit_events_total counter
|
||||
memory_audit_events_total{event_type="chunk_edited"} 3421
|
||||
memory_audit_events_total{event_type="chunk_created"} 1245
|
||||
memory_audit_events_total{event_type="verification"} 892
|
||||
|
||||
# HELP memory_audit_signature_failures_total Signature verification failures
|
||||
# TYPE memory_audit_signature_failures_total counter
|
||||
memory_audit_signature_failures_total 0 # M7.2 gate: must be 0
|
||||
|
||||
# HELP memory_rebuild_checksum_matches_total Rebuild parity checks passed
|
||||
# TYPE memory_rebuild_checksum_matches_total counter
|
||||
memory_rebuild_checksum_matches_total 127 # M7.1 gate: 100%
|
||||
|
||||
# HELP memory_rebuild_checksum_mismatches_total Rebuild parity checks FAILED
|
||||
# TYPE memory_rebuild_checksum_mismatches_total counter
|
||||
memory_rebuild_checksum_mismatches_total 0 # M7.1 gate: must be 0
|
||||
|
||||
# HELP memory_ranking_ndcg_weighted Weighted NDCG@10 (confidence-adjusted)
|
||||
# TYPE memory_ranking_ndcg_weighted gauge
|
||||
memory_ranking_ndcg_weighted 0.882 # M7.3 gate: >= 0.88
|
||||
|
||||
# HELP memory_ranking_signal_contribution Contribution of each signal
|
||||
# TYPE memory_ranking_signal_contribution gauge
|
||||
memory_ranking_signal_contribution{signal="confidence"} 0.25
|
||||
memory_ranking_signal_contribution{signal="recency"} 0.25
|
||||
memory_ranking_signal_contribution{signal="community"} 0.25
|
||||
memory_ranking_signal_contribution{signal="bm25"} 0.25
|
||||
|
||||
# HELP memory_storage_bytes Total storage bytes
|
||||
# TYPE memory_storage_bytes gauge
|
||||
memory_storage_bytes{type="current_versions"} 5368709120
|
||||
memory_storage_bytes{type="all_versions"} 8053063680 # M7.5 gate: <= 1.5x
|
||||
memory_storage_bytes{type="audit_log"} 2147483648
|
||||
|
||||
# HELP memory_storage_overhead_ratio Overhead ratio (versions / current)
|
||||
# TYPE memory_storage_overhead_ratio gauge
|
||||
memory_storage_overhead_ratio 1.25 # M7.5 gate: <= 1.5x
|
||||
|
||||
# HELP memory_confidence_distribution Confidence score distribution
|
||||
# TYPE memory_confidence_distribution histogram
|
||||
memory_confidence_distribution_bucket{le="0.3"} 234 # Uncertain
|
||||
memory_confidence_distribution_bucket{le="0.6"} 1289 # Moderate
|
||||
memory_confidence_distribution_bucket{le="0.85"} 4156 # High
|
||||
memory_confidence_distribution_bucket{le="1.0"} 5234 # Verified
|
||||
|
||||
# HELP memory_recency_score_distribution Recency score distribution
|
||||
# TYPE memory_recency_score_distribution gauge
|
||||
memory_recency_score_avg 0.72 # Average recency (30 days half-life)
|
||||
memory_recency_score_p50 0.78
|
||||
memory_recency_score_p99 0.98
|
||||
|
||||
# HELP memory_community_score_distribution Community signal distribution
|
||||
# TYPE memory_community_score_distribution gauge
|
||||
memory_community_score_avg 0.45
|
||||
memory_community_score_p50 0.42
|
||||
memory_community_score_p99 0.91
|
||||
```
|
||||
|
||||
### Prometheus Scrape Config
|
||||
|
||||
```yaml
|
||||
# In homelab Prometheus values.yaml
|
||||
prometheus:
|
||||
scrapeConfigs:
|
||||
- job_name: 'poimen-memory'
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
namespaces:
|
||||
names:
|
||||
- poimen
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_pod_label_app]
|
||||
action: keep
|
||||
regex: memory-service
|
||||
- source_labels: [__meta_kubernetes_pod_container_port_name]
|
||||
action: keep
|
||||
regex: metrics
|
||||
scheme: https
|
||||
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
|
||||
tls_config:
|
||||
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
||||
scrape_interval: 15s
|
||||
scrape_timeout: 5s
|
||||
```
|
||||
|
||||
### SLO Alerts (Prometheus Rules)
|
||||
|
||||
```yaml
|
||||
# rules/poimen-memory-phase7.yaml
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: poimen-memory-phase7
|
||||
namespace: monitoring
|
||||
spec:
|
||||
groups:
|
||||
- name: memory-phase7-gates
|
||||
interval: 30s
|
||||
rules:
|
||||
# M7.1: Rebuild parity
|
||||
- alert: M7_1_RebuildParityCheckFailed
|
||||
expr: increase(memory_rebuild_checksum_mismatches_total[5m]) > 0
|
||||
for: 1m
|
||||
labels:
|
||||
severity: critical
|
||||
phase: "7"
|
||||
annotations:
|
||||
summary: "Rebuild parity check failed (M7.1 gate)"
|
||||
description: "{{ $value }} checksum mismatches in last 5 minutes"
|
||||
|
||||
# M7.2: Audit trail integrity
|
||||
- alert: M7_2_AuditSignatureFailure
|
||||
expr: increase(memory_audit_signature_failures_total[5m]) > 0
|
||||
for: 1m
|
||||
labels:
|
||||
severity: critical
|
||||
phase: "7"
|
||||
annotations:
|
||||
summary: "Audit signature verification failed (M7.2 gate)"
|
||||
description: "{{ $value }} signature failures detected"
|
||||
|
||||
# M7.3: Ranking accuracy
|
||||
- alert: M7_3_RankingAccuracyDegraded
|
||||
expr: memory_ranking_ndcg_weighted < 0.88
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
phase: "7"
|
||||
annotations:
|
||||
summary: "Ranking accuracy below SLO (M7.3 gate)"
|
||||
description: "NDCG@10 = {{ $value | humanizePercentage }} (target: >= 0.88)"
|
||||
|
||||
# M7.4: Edit latency
|
||||
- alert: M7_4_EditLatencyHigh
|
||||
expr: histogram_quantile(0.99, memory_edit_latency_seconds) > 2.0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
phase: "7"
|
||||
annotations:
|
||||
summary: "Edit latency P99 exceeds SLO (M7.4 gate)"
|
||||
description: "P99 latency = {{ $value | humanizeDuration }} (target: < 2s)"
|
||||
|
||||
# M7.5: Storage overhead
|
||||
- alert: M7_5_StorageOverheadHigh
|
||||
expr: memory_storage_overhead_ratio > 1.5
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
phase: "7"
|
||||
annotations:
|
||||
summary: "Storage overhead exceeds SLO (M7.5 gate)"
|
||||
description: "Overhead ratio = {{ $value | humanizePercentage }} (target: <= 1.5x)"
|
||||
```
|
||||
|
||||
### Grafana Dashboard (Phase 7)
|
||||
|
||||
```json
|
||||
{
|
||||
"dashboard": {
|
||||
"title": "Poimen Memory - Phase 7 SLO Dashboard",
|
||||
"panels": [
|
||||
{
|
||||
"title": "M7.1 Rebuild Parity",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "memory_rebuild_checksum_matches_total / (memory_rebuild_checksum_matches_total + memory_rebuild_checksum_mismatches_total)"
|
||||
}
|
||||
],
|
||||
"alert_threshold": 1.0,
|
||||
"description": "Must be 100% (gate M7.1)"
|
||||
},
|
||||
{
|
||||
"title": "M7.3 Ranking Accuracy (NDCG@10 weighted)",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "memory_ranking_ndcg_weighted"
|
||||
}
|
||||
],
|
||||
"alert_threshold": 0.88,
|
||||
"description": "Target: >= 0.88 (gate M7.3)"
|
||||
},
|
||||
{
|
||||
"title": "M7.4 Edit Latency P99",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "histogram_quantile(0.99, memory_edit_latency_seconds)"
|
||||
}
|
||||
],
|
||||
"alert_threshold": 2.0,
|
||||
"description": "Target: < 2s (gate M7.4)"
|
||||
},
|
||||
{
|
||||
"title": "M7.5 Storage Overhead",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "memory_storage_overhead_ratio"
|
||||
}
|
||||
],
|
||||
"alert_threshold": 1.5,
|
||||
"description": "Target: <= 1.5x (gate M7.5)"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 11. Metrics & SLOs
|
||||
|
||||
### Phase 7 Composition Gate
|
||||
|
||||
```yaml
|
||||
gates:
|
||||
- name: M7.1_version_parity
|
||||
type: rebuild_checksum_match
|
||||
threshold: 100%
|
||||
sla: "All rebuilds must produce identical checksums"
|
||||
|
||||
- name: M7.2_audit_trail_integrity
|
||||
type: signature_verification
|
||||
threshold: 100%
|
||||
sla: "Zero tampered audit events"
|
||||
|
||||
- name: M7.3_ranking_accuracy
|
||||
type: ndcg_weighted_by_confidence
|
||||
threshold: ">= 0.88"
|
||||
sla: "NDCG@10 with multi-signal ranking >= 0.88"
|
||||
|
||||
- name: M7.4_chunk_edit_latency
|
||||
type: p99_latency
|
||||
threshold: "<= 2s"
|
||||
sla: "99th percentile edit completion <= 2 seconds"
|
||||
|
||||
- name: M7.5_version_space
|
||||
type: storage_overhead
|
||||
threshold: "<= 1.5x"
|
||||
sla: "Versioned storage overhead <= 1.5x original"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 11. Implementation Roadmap
|
||||
|
||||
### Phase 7.1: Schema & Migration (Week 1)
|
||||
- [ ] Create chunk_versions table
|
||||
- [ ] Create audit_events table (partitioned)
|
||||
- [ ] Create ranking_signals table
|
||||
- [ ] Migration script (6.x → 7.x)
|
||||
- [ ] Backfill ranking_signals from existing data
|
||||
|
||||
### Phase 7.2: Versioning API (Week 2)
|
||||
- [ ] GET /memory/chunks/{id} (with version history)
|
||||
- [ ] PATCH /memory/chunks/{id} (edit + create version)
|
||||
- [ ] GET /memory/chunks/{id}/versions (list all versions)
|
||||
- [ ] POST /memory/chunks/{id}/versions/{v}/revert (rollback)
|
||||
- [ ] Tests: 50+ unit + integration
|
||||
|
||||
### Phase 7.3: Audit Trail (Week 2)
|
||||
- [ ] Append-only event logging
|
||||
- [ ] Signature verification (cryptographic)
|
||||
- [ ] GET /memory/audit?resource_id={id} (query trail)
|
||||
- [ ] Backup to S3 (cold storage)
|
||||
- [ ] Tests: 30+ unit
|
||||
|
||||
### Phase 7.4: Multi-Signal Ranking (Week 3)
|
||||
- [ ] Compute confidence from extraction pipeline
|
||||
- [ ] Compute recency (last_verified_at)
|
||||
- [ ] Compute community_signal (PageRank)
|
||||
- [ ] Normalize BM25 scores
|
||||
- [ ] Integrate into query ranking (hybrid_search.rs)
|
||||
- [ ] Tests: 40+ unit
|
||||
|
||||
### Phase 7.5: Deterministic Rebuild (Week 3)
|
||||
- [ ] Snapshot + version tracking
|
||||
- [ ] Replay audit events
|
||||
- [ ] Parity verification (M7.1 gate)
|
||||
- [ ] Tests: 20+ integration
|
||||
|
||||
### Phase 7.6: Documentation & SLOs (Week 4)
|
||||
- [ ] API documentation
|
||||
- [ ] Audit trail guide
|
||||
- [ ] Editing workflow guide
|
||||
- [ ] SLO dashboards (Prometheus)
|
||||
- [ ] Runbooks (troubleshooting)
|
||||
|
||||
---
|
||||
|
||||
## 12. Key Design Decisions
|
||||
|
||||
| Decision | Choice | Rationale |
|
||||
|----------|--------|-----------|
|
||||
| Versioning | Immutable (Option A) | Clearer lineage, no tombstones |
|
||||
| Signals | 4 equal weights (25% each) | Balanced, interpretable |
|
||||
| Audit storage | Append-only JSONL + S3 | Immutable, queryable, archivable |
|
||||
| Rebuild parity | Signature verification | Fast (O(1)), detects corruption |
|
||||
| Edit latency | P99 < 2s | Real-time editing UX |
|
||||
| Confidence source | LLM extraction pipeline | Aligned with Phase 5 |
|
||||
| Recency decay | Exponential (τ=30d) | Standard information theory |
|
||||
| Community signal | PageRank + in-degree | Graph-theoretic, proven |
|
||||
|
||||
---
|
||||
|
||||
## 13. Risks & Mitigations
|
||||
|
||||
| Risk | Impact | Mitigation |
|
||||
|------|--------|-----------|
|
||||
| Version explosion (storage) | 1.5-2x overhead | Compression, periodic archival, TTL cleanup |
|
||||
| Audit log query slowness | Query latency spikes | Partitioning by time, materialized views |
|
||||
| Signature verification false positives | Trust erosion | Comprehensive testing, HSM backup |
|
||||
| Community signal staleness | Ranking drift | Recompute PageRank daily (off-peak) |
|
||||
| Editing conflicts | Concurrent edit loss | Optimistic locking via version number |
|
||||
|
||||
---
|
||||
|
||||
## 14. Success Criteria
|
||||
|
||||
✅ **Phase 7 Complete When:**
|
||||
1. All 5 composition gates pass (M7.1-M7.5)
|
||||
2. 150+ tests passing (unit + integration)
|
||||
3. NDCG@10 with weighted confidence >= 0.88
|
||||
4. Edit latency P99 < 2s
|
||||
5. Storage overhead <= 1.5x
|
||||
6. Audit trail 100% immutable (0 tampering detected)
|
||||
7. Full documentation + runbooks
|
||||
8. Production SLA: 99.9% availability
|
||||
|
||||
---
|
||||
|
||||
## 15. References
|
||||
|
||||
- **Versioning**: Git model (immutable commits, branches = versions)
|
||||
- **Ranking signals**: Okapi BM25 + PageRank + temporal decay (arXiv:1802.05365)
|
||||
- **Audit trails**: NIST SP 800-92 (log management)
|
||||
- **Parity checks**: Deterministic replay (Riak design, §3.2)
|
||||
- **Temporal semantics**: ISO 8601 + valid_from/valid_to
|
||||
|
||||
---
|
||||
|
||||
**Status**: 🔵 **DESIGN COMPLETE - READY FOR PHASE 7.1 IMPLEMENTATION**
|
||||
|
||||
Next: Review & finalize schema, then begin Phase 7.1 (migrations + versioning API).
|
||||
Reference in New Issue
Block a user