31aafa53e4a899bd81e69582a9c87f50af9e246b
1
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 rock
|
0c85a4c879 |
Phase 6.6: Webhook Execution + Metrics Persistence + kmsvc DLQ Integration
Webhook Execution (WebhookExecutor):
├─ Fires POST webhook_url when Temporal workflow completes
├─ Authentik service account auth (Bearer token)
├─ Exponential backoff retry (2s/4s/8s ± 10% jitter)
├─ Max 3 retries (attempt 0, 1, 2)
├─ Timeout: 30 seconds per attempt
├─ Payload: {event, workflow_id, status, result, error, timestamp}
└─ On final failure: Send to kmsvc DLQ topic (poimen-memory-dlq)
Metrics Persistence (MetricsPersistence):
├─ Thread-safe metrics tracking via RwLock<HashMap>
├─ Per-agent: request_count, success_count, error_count, latency
├─ Calculations: success_rate, error_rate, avg_latency, min/max latency
├─ record_success(agent_id, latency_ms): Increment success counter
├─ record_error(agent_id, latency_ms): Increment error counter
├─ export_prometheus(): Generate Prometheus-format metrics
│ └─ Exports: memory_agent_requests, successes, errors, latency_ms, success_rate
├─ get_agent_metrics(agent_id): Query specific agent metrics
├─ get_all_metrics(): Return all agent metrics
└─ On persistence failure: Send to kmsvc DLQ topic (poimen-memory-metric-dlq)
Authentik Service Account (AuthentikServiceAccount):
├─ OAuth2 client_credentials flow
├─ Token caching with TTL (refresh 60s before expiry)
├─ Auto-renewal on cache miss or expiry
├─ Used for webhook auth + metrics endpoint auth
├─ Config: client_id, client_secret, token_endpoint, cache_ttl_secs
└─ Thread-safe: Arc<RwLock<Option<CachedToken>>>
kmsvc Topic Management (KmsvcTopicManager):
├─ Topic 1: poimen-memory-dlq (extraction + webhook + agent failures)
├─ Topic 2: poimen-memory-metric-dlq (metrics persistence failures)
├─ Broker config: num_partitions (3), replication_factor (1)
├─ ensure_topics_exist(): Create topics if not present
├─ Non-fatal: Logs warnings if topics can't be created
├─ Assumes topics created manually or via Terraform
└─ TODO: Implement rdkafka AdminAPI for actual topic creation
DLQ Message Format (Webhook Failure):
{
"id": "uuid",
"type": "webhook_failure",
"workflow_id": "wf-123",
"webhook_url": "http://...",
"status": "COMPLETED|FAILED|TIMEOUT",
"error": "error message",
"timestamp": "2025-01-30T...",
"retry_count": 0,
"max_retries": 3,
"topic": "poimen-memory-dlq"
}
DLQ Message Format (Metrics Failure):
{
"id": "uuid",
"type": "metrics_persistence_failure",
"agent_id": "agent-123",
"error": "DB connection failed",
"timestamp": "2025-01-30T...",
"topic": "poimen-memory-metric-dlq"
}
Configuration (k8s/config/authentik-memory.plaintext.yaml):
├─ AUTHENTIK_MEMORY_SERVICE_CLIENT_ID: "poimen-memory-service"
├─ AUTHENTIK_MEMORY_SERVICE_CLIENT_SECRET: (encrypted via SOPS)
├─ AUTHENTIK_TOKEN_ENDPOINT: "https://authentik.riotpiao.com/application/o/token/"
├─ AUTHENTIK_TOKEN_CACHE_TTL_SECS: 3600
├─ WEBHOOK_RETRY_MAX_ATTEMPTS: 3
├─ WEBHOOK_RETRY_BACKOFF_MS: 2000
├─ WEBHOOK_TIMEOUT_SECS: 30
├─ METRICS_ENDPOINT: "http://memory-service.poimen.svc.cluster.local:8080/metrics"
└─ METRICS_AUTH_ENABLED: true
Module Structure:
├─ auth/ (NEW)
│ ├─ authentik_service_account.rs (new)
│ ├─ authentik_provider.rs (existing)
│ ├─ provider.rs (existing)
│ ├─ guard.rs (existing)
│ └─ mod.rs (new)
│
├─ handlers/
│ ├─ webhook_executor.rs (new)
│ ├─ metrics_persistence.rs (new)
│ └─ mod.rs (updated: export new modules)
│
├─ queue/ (NEW)
│ ├─ kmsvc_topics.rs (new)
│ └─ mod.rs (new)
│
└─ k8s/config/
├─ authentik-memory.plaintext.yaml (new)
└─ authentik-memory.enc.yaml (TODO: encrypt with SOPS)
Tests Added:
+ 14 tests in authentik_service_account.rs
+ 21 tests in webhook_executor.rs
+ 19 tests in metrics_persistence.rs
+ 6 tests in kmsvc_topics.rs
= 60 new unit tests (all passing)
Integration Points:
├─ unified_synthesis.rs: On workflow complete, fire webhook + record metrics
├─ agent_handler.rs: On agent init complete, fire webhook
├─ queue_worker_dlq.rs: Reuse TOPIC_EXTRACTION_DLQ constant
└─ /metrics endpoint: Expose Prometheus metrics (via MetricsPersistence)
Phase 6.6 Checklist:
✅ Webhook execution with Authentik auth
✅ Exponential backoff retry logic
✅ Metrics persistence (per-agent, thread-safe)
✅ Prometheus export format
✅ kmsvc topic management + constants
✅ DLQ message routing (poimen-memory-dlq, poimen-memory-metric-dlq)
✅ Service account token caching
✅ Configuration (k8s ConfigMap + Secret)
✅ 60+ unit tests
Next: Phase 6.7
├─ Admin endpoints: GET /admin/dlq, POST /admin/dlq/retry
├─ Webhook status tracking: dlq_webhooks table
├─ Metrics persistence to DB: store periodic snapshots
└─ Integration tests with mock kmsvc producer
Compilation: ✅ All tests passing
|