poimen-memory

rock/poimen-memory

Fork 0

Commit Graph

Author	SHA1	Message	Date
rock	c6bfe0e032	Phase 7: Temporal-RAGA-Ingest Architecture Design (Complete) 📋 DESIGN DOCUMENT (18.7 KB) Architecture: ├─ Temporal-aware knowledge graph (versioning) ├─ RAGA ingest pipeline (Retrieval-Augmented Graph Architecture) ├─ Chunk editing with immutable audit trail └─ Multi-signal ranking (4 signals, 25% each) Key Sections: 1. Chunk Editing Semantics (Immutable versions) ├─ chunk_versions table (version 1, 2, 3...) ├─ is_current flag (which version is active) ├─ edited_by, edit_reason, confidence tracking └─ Example: Kubernetes entity v1 → v2 (added CNCF affiliation) 2. Ranking Formula (4 Equal Signals) ├─ Signal 1: Confidence (LLM extraction, 0.0-1.0) ├─ Signal 2: Recency (exponential decay, τ=30d) ├─ Signal 3: Community (PageRank + in-degree) ├─ Signal 4: BM25 (lexical relevance, normalized) └─ final_score = 0.25conf + 0.25recency + 0.25community + 0.25bm25 3. Audit Trail (Append-only immutable log) ├─ audit_events table (partitioned by timestamp) ├─ Every mutation logged: chunk_edited, created, verified, deleted ├─ Cryptographic signing (SHA256 for tamper detection) ├─ Queryable: Who changed what, when, why └─ Archive: Daily batch to S3 cold storage 4. Schema Extensions ├─ chunk_versions: id, chunk_id, version, content, confidence, is_current ├─ audit_events: id, timestamp, event_type, actor, resource_id, action, reason ├─ ranking_signals: id, entity_id, signal_type, signal_value └─ query_rankings: query_id, chunk_id, rank, final_score, signal_breakdown 5. Deterministic Rebuild (Parity Check - M2.8 extended) ├─ Snapshot current state ├─ Replay audit events in order ├─ Recompute all signals ├─ Verify: checksum_before == checksum_after └─ Detects corruption in O(1) time 6. Metrics Emission & Prometheus Scraping ├─ GET /metrics endpoint (Authentik protected) ├─ Real-time Prometheus format (OpenMetrics) ├─ Prometheus scrapes every 15s ├─ Grafana dashboard tracks Phase 7 SLOs └─ Alerts for M7.1-M7.5 gates Phase 7 Metrics (Prometheus): ├─ memory_chunk_edits_total (counter: create/update/delete) ├─ memory_edit_latency_seconds (histogram: P50/P99) ├─ memory_audit_events_total (counter: by event_type) ├─ memory_audit_signature_failures_total (counter: must be 0) ├─ memory_rebuild_checksum_matches_total (counter: parity checks) ├─ memory_ranking_ndcg_weighted (gauge: weighted accuracy) ├─ memory_storage_overhead_ratio (gauge: 1.5x max) ├─ memory_confidence_distribution (histogram: score buckets) ├─ memory_recency_score_* (gauge: avg/p50/p99) └─ memory_community_score_* (gauge: avg/p50/p99) SLO Alerts (Prometheus Rules): ├─ M7.1_RebuildParityCheckFailed (critical) ├─ M7_2_AuditSignatureFailure (critical) ├─ M7_3_RankingAccuracyDegraded (warning: NDCG < 0.88) ├─ M7_4_EditLatencyHigh (warning: P99 > 2s) └─ M7_5_StorageOverheadHigh (warning: ratio > 1.5x) Implementation Roadmap: ├─ Phase 7.1: Schema & Migrations (Week 1, ~200 LOC) ├─ Phase 7.2: Versioning API (Week 2, ~400 LOC, 50+ tests) ├─ Phase 7.3: Audit Trail (Week 2, ~300 LOC, 30+ tests) ├─ Phase 7.4: Multi-Signal Ranking (Week 3, ~350 LOC, 40+ tests) ├─ Phase 7.5: Deterministic Rebuild (Week 3, ~200 LOC, 20+ tests) └─ Phase 7.6: Documentation & SLOs (Week 4, ~500 LOC docs) Success Criteria: ✅ All 5 composition gates pass (M7.1-M7.5) ✅ 150+ tests (unit + integration) ✅ NDCG@10 weighted >= 0.88 (M7.3) ✅ Edit latency P99 < 2s (M7.4) ✅ Storage overhead <= 1.5x (M7.5) ✅ Audit trail 100% immutable (M7.2) ✅ Rebuild parity 100% (M7.1) ✅ Full documentation + runbooks Key Design Decisions: ├─ Versioning: Immutable (Option A, not Option B soft deletes) ├─ Signals: 4 equal weights (25% each, not weighted differently) ├─ Audit: Append-only JSONL + S3 (not mutable log) ├─ Rebuild: Signature verification (O(1), not full replay) ├─ Confidence: From LLM pipeline (Phase 5) ├─ Recency: Exponential decay τ=30d (standard info theory) ├─ Community: PageRank + in-degree (graph-theoretic) └─ Edit latency: P99 < 2s (real-time UX) Risks & Mitigations: ├─ Version explosion: Compression + archival + TTL cleanup ├─ Audit log query slowness: Partitioning + materialized views ├─ Signature false positives: Comprehensive testing + HSM backup ├─ Community signal staleness: Recompute PageRank daily └─ Concurrent edits: Optimistic locking via version number Integration Points: ├─ Phase 4 (Retrieval) → Multi-signal ranking ├─ Phase 5 (Synthesis) → Confidence extraction ├─ Phase 6 (Agents) → Metrics emission └─ Phase 7 (Versioning) → Deterministic rebuild References: ├─ Git model (immutable commits) ├─ Okapi BM25 + PageRank (arXiv:1802.05365) ├─ NIST SP 800-92 (audit logs) ├─ Riak parity checks (deterministic replay) └─ ISO 8601 (temporal semantics) Next: Architecture review, then Phase 7.1 (migrations)	2026-09-05 01:14:32 -07:00

Author

SHA1

Message

Date

rock

c6bfe0e032

Phase 7: Temporal-RAGA-Ingest Architecture Design (Complete)

📋 DESIGN DOCUMENT (18.7 KB)

Architecture:
  ├─ Temporal-aware knowledge graph (versioning)
  ├─ RAGA ingest pipeline (Retrieval-Augmented Graph Architecture)
  ├─ Chunk editing with immutable audit trail
  └─ Multi-signal ranking (4 signals, 25% each)

Key Sections:

1. Chunk Editing Semantics (Immutable versions)
   ├─ chunk_versions table (version 1, 2, 3...)
   ├─ is_current flag (which version is active)
   ├─ edited_by, edit_reason, confidence tracking
   └─ Example: Kubernetes entity v1 → v2 (added CNCF affiliation)

2. Ranking Formula (4 Equal Signals)
   ├─ Signal 1: Confidence (LLM extraction, 0.0-1.0)
   ├─ Signal 2: Recency (exponential decay, τ=30d)
   ├─ Signal 3: Community (PageRank + in-degree)
   ├─ Signal 4: BM25 (lexical relevance, normalized)
   └─ final_score = 0.25*conf + 0.25*recency + 0.25*community + 0.25*bm25

3. Audit Trail (Append-only immutable log)
   ├─ audit_events table (partitioned by timestamp)
   ├─ Every mutation logged: chunk_edited, created, verified, deleted
   ├─ Cryptographic signing (SHA256 for tamper detection)
   ├─ Queryable: Who changed what, when, why
   └─ Archive: Daily batch to S3 cold storage

4. Schema Extensions
   ├─ chunk_versions: id, chunk_id, version, content, confidence, is_current
   ├─ audit_events: id, timestamp, event_type, actor, resource_id, action, reason
   ├─ ranking_signals: id, entity_id, signal_type, signal_value
   └─ query_rankings: query_id, chunk_id, rank, final_score, signal_breakdown

5. Deterministic Rebuild (Parity Check - M2.8 extended)
   ├─ Snapshot current state
   ├─ Replay audit events in order
   ├─ Recompute all signals
   ├─ Verify: checksum_before == checksum_after
   └─ Detects corruption in O(1) time

6. Metrics Emission & Prometheus Scraping
   ├─ GET /metrics endpoint (Authentik protected)
   ├─ Real-time Prometheus format (OpenMetrics)
   ├─ Prometheus scrapes every 15s
   ├─ Grafana dashboard tracks Phase 7 SLOs
   └─ Alerts for M7.1-M7.5 gates

Phase 7 Metrics (Prometheus):
  ├─ memory_chunk_edits_total (counter: create/update/delete)
  ├─ memory_edit_latency_seconds (histogram: P50/P99)
  ├─ memory_audit_events_total (counter: by event_type)
  ├─ memory_audit_signature_failures_total (counter: must be 0)
  ├─ memory_rebuild_checksum_matches_total (counter: parity checks)
  ├─ memory_ranking_ndcg_weighted (gauge: weighted accuracy)
  ├─ memory_storage_overhead_ratio (gauge: 1.5x max)
  ├─ memory_confidence_distribution (histogram: score buckets)
  ├─ memory_recency_score_* (gauge: avg/p50/p99)
  └─ memory_community_score_* (gauge: avg/p50/p99)

SLO Alerts (Prometheus Rules):
  ├─ M7.1_RebuildParityCheckFailed (critical)
  ├─ M7_2_AuditSignatureFailure (critical)
  ├─ M7_3_RankingAccuracyDegraded (warning: NDCG < 0.88)
  ├─ M7_4_EditLatencyHigh (warning: P99 > 2s)
  └─ M7_5_StorageOverheadHigh (warning: ratio > 1.5x)

Implementation Roadmap:
  ├─ Phase 7.1: Schema & Migrations (Week 1, ~200 LOC)
  ├─ Phase 7.2: Versioning API (Week 2, ~400 LOC, 50+ tests)
  ├─ Phase 7.3: Audit Trail (Week 2, ~300 LOC, 30+ tests)
  ├─ Phase 7.4: Multi-Signal Ranking (Week 3, ~350 LOC, 40+ tests)
  ├─ Phase 7.5: Deterministic Rebuild (Week 3, ~200 LOC, 20+ tests)
  └─ Phase 7.6: Documentation & SLOs (Week 4, ~500 LOC docs)

Success Criteria:
  ✅ All 5 composition gates pass (M7.1-M7.5)
  ✅ 150+ tests (unit + integration)
  ✅ NDCG@10 weighted >= 0.88 (M7.3)
  ✅ Edit latency P99 < 2s (M7.4)
  ✅ Storage overhead <= 1.5x (M7.5)
  ✅ Audit trail 100% immutable (M7.2)
  ✅ Rebuild parity 100% (M7.1)
  ✅ Full documentation + runbooks

Key Design Decisions:
  ├─ Versioning: Immutable (Option A, not Option B soft deletes)
  ├─ Signals: 4 equal weights (25% each, not weighted differently)
  ├─ Audit: Append-only JSONL + S3 (not mutable log)
  ├─ Rebuild: Signature verification (O(1), not full replay)
  ├─ Confidence: From LLM pipeline (Phase 5)
  ├─ Recency: Exponential decay τ=30d (standard info theory)
  ├─ Community: PageRank + in-degree (graph-theoretic)
  └─ Edit latency: P99 < 2s (real-time UX)

Risks & Mitigations:
  ├─ Version explosion: Compression + archival + TTL cleanup
  ├─ Audit log query slowness: Partitioning + materialized views
  ├─ Signature false positives: Comprehensive testing + HSM backup
  ├─ Community signal staleness: Recompute PageRank daily
  └─ Concurrent edits: Optimistic locking via version number

Integration Points:
  ├─ Phase 4 (Retrieval) → Multi-signal ranking
  ├─ Phase 5 (Synthesis) → Confidence extraction
  ├─ Phase 6 (Agents) → Metrics emission
  └─ Phase 7 (Versioning) → Deterministic rebuild

References:
  ├─ Git model (immutable commits)
  ├─ Okapi BM25 + PageRank (arXiv:1802.05365)
  ├─ NIST SP 800-92 (audit logs)
  ├─ Riak parity checks (deterministic replay)
  └─ ISO 8601 (temporal semantics)

Next: Architecture review, then Phase 7.1 (migrations)

2026-09-05 01:14:32 -07:00