Story Crater Bot
71ecf482e7
- M7.1-M7.10: Extensible SourceConnector trait, Obsidian/paperless/git/S3 connectors, sync framework, CLI, HTTP endpoints, health monitoring, gate - M3.5.10: Auth integration with Authentik OIDC → Vault token validation - DESIGN.md: Add source connectors architecture, update auth to Authentik/Vault (Kong removed from cluster) - INDEX.md: 75 tasks, 11 gates - Fix all Kong references in M3.5.1 task
2.8 KiB
2.8 KiB
M7.9 — Connector health monitoring + observability
| Field | Value |
|---|---|
| Phase | M7 — Source connectors |
| Size | S — < 1 day |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M7.10 |
| Depends | M7.8 |
Goal
Add periodic health checks, sync metrics, and alerting hooks for source connectors so connector failures are detected before knowledge goes stale.
Facts (inlined — no spec read needed)
A connector that silently fails means the knowledge base is stale with no indication. The monitoring layer detects this.
Metrics (Prometheus-compatible):
mem_source_health{name="homelab-vault",kind="obsidian"} 1 # 1=healthy, 0=unhealthy
mem_source_last_sync_seconds{name="homelab-vault"} 1724680000 # unix timestamp
mem_source_documents_total{name="homelab-vault"} 42
mem_source_sync_duration_seconds{name="homelab-vault"} 12.3
mem_source_sync_errors_total{name="homelab-vault"} 0
mem_source_drift_new{name="homelab-vault"} 2 # docs pending sync
mem_source_drift_changed{name="homelab-vault"} 1
Periodic health check. Configurable interval (default 5 minutes). Log warnings for unreachable connectors. Update Prometheus gauges.
Staleness alert. If last_sync exceeds a configurable threshold (default 24h),
log a warning and set a metric. This is the "knowledge is going stale" signal.
Steps
- Add health check background task (tokio interval, configurable period).
- Export Prometheus metrics via
/metricsendpoint (existing pattern or new). - Track per-connector: health, last sync, doc count, sync duration, errors.
- Staleness detection: compare
last_syncto now, warn if exceeds threshold. - Log structured health events for observability.
Acceptance
- Periodic health checks run at configured interval.
- Metrics endpoint returns valid Prometheus format.
- Staleness warning fires when last_sync exceeds threshold.
- Unhealthy connector logged with error details.
Verify
Integration test — tests/it_source_monitoring.rs:
a1_health_metric_updated— register connector, wait for health check; assert metric value matches connector health.a2_staleness_detected— set last_sync to 25h ago; assert staleness warning.a3_metrics_format_valid— GET /metrics; assert valid Prometheus text format.a4_error_count_incremented— force connector health check failure; assert error counter incremented.
Command: cargo test --test it_source_monitoring
Traps
- Running health checks synchronously. A connector that times out blocks all other connectors' health checks. Use tokio::select with timeout.
- Not distinguishing "unreachable" from "empty". A connector that returns 0 docs is healthy; a connector that can't connect is not.
Background: DESIGN.md — source connectors, observability