77 lines
2.8 KiB
Markdown
77 lines
2.8 KiB
Markdown
|
# M7.9 — Connector health monitoring + observability
|
||
|
|
|
||
|
|
| Field | Value |
|
||
|
|
|---|---|
|
||
|
|
| Phase | M7 — Source connectors |
|
||
|
|
| Size | S — < 1 day |
|
||
|
|
| Status | ⬜ Not started |
|
||
|
|
| Flags | — |
|
||
|
|
| Spec | inlined below |
|
||
|
|
| Blocks | M7.10 |
|
||
|
|
| Depends | M7.8 |
|
||
|
|
|
||
|
|
## Goal
|
||
|
|
|
||
|
|
Add periodic health checks, sync metrics, and alerting hooks for source
|
||
|
|
connectors so connector failures are detected before knowledge goes stale.
|
||
|
|
|
||
|
|
## Facts (inlined — no spec read needed)
|
||
|
|
|
||
|
|
A connector that silently fails means the knowledge base is stale with no
|
||
|
|
indication. The monitoring layer detects this.
|
||
|
|
|
||
|
|
**Metrics (Prometheus-compatible):**
|
||
|
|
```
|
||
|
|
mem_source_health{name="homelab-vault",kind="obsidian"} 1 # 1=healthy, 0=unhealthy
|
||
|
|
mem_source_last_sync_seconds{name="homelab-vault"} 1724680000 # unix timestamp
|
||
|
|
mem_source_documents_total{name="homelab-vault"} 42
|
||
|
|
mem_source_sync_duration_seconds{name="homelab-vault"} 12.3
|
||
|
|
mem_source_sync_errors_total{name="homelab-vault"} 0
|
||
|
|
mem_source_drift_new{name="homelab-vault"} 2 # docs pending sync
|
||
|
|
mem_source_drift_changed{name="homelab-vault"} 1
|
||
|
|
```
|
||
|
|
|
||
|
|
**Periodic health check.** Configurable interval (default 5 minutes). Log
|
||
|
|
warnings for unreachable connectors. Update Prometheus gauges.
|
||
|
|
|
||
|
|
**Staleness alert.** If `last_sync` exceeds a configurable threshold (default 24h),
|
||
|
|
log a warning and set a metric. This is the "knowledge is going stale" signal.
|
||
|
|
|
||
|
|
## Steps
|
||
|
|
|
||
|
|
1. Add health check background task (tokio interval, configurable period).
|
||
|
|
2. Export Prometheus metrics via `/metrics` endpoint (existing pattern or new).
|
||
|
|
3. Track per-connector: health, last sync, doc count, sync duration, errors.
|
||
|
|
4. Staleness detection: compare `last_sync` to now, warn if exceeds threshold.
|
||
|
|
5. Log structured health events for observability.
|
||
|
|
|
||
|
|
## Acceptance
|
||
|
|
|
||
|
|
- Periodic health checks run at configured interval.
|
||
|
|
- Metrics endpoint returns valid Prometheus format.
|
||
|
|
- Staleness warning fires when last_sync exceeds threshold.
|
||
|
|
- Unhealthy connector logged with error details.
|
||
|
|
|
||
|
|
## Verify
|
||
|
|
|
||
|
|
**Integration test** — `tests/it_source_monitoring.rs`:
|
||
|
|
1. `a1_health_metric_updated` — register connector, wait for health check;
|
||
|
|
assert metric value matches connector health.
|
||
|
|
2. `a2_staleness_detected` — set last_sync to 25h ago; assert staleness warning.
|
||
|
|
3. `a3_metrics_format_valid` — GET /metrics; assert valid Prometheus text format.
|
||
|
|
4. `a4_error_count_incremented` — force connector health check failure; assert
|
||
|
|
error counter incremented.
|
||
|
|
|
||
|
|
**Command:** `cargo test --test it_source_monitoring`
|
||
|
|
|
||
|
|
## Traps
|
||
|
|
|
||
|
|
- Running health checks synchronously. A connector that times out blocks all
|
||
|
|
other connectors' health checks. Use tokio::select with timeout.
|
||
|
|
- Not distinguishing "unreachable" from "empty". A connector that returns 0 docs
|
||
|
|
is healthy; a connector that can't connect is not.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
Background: [DESIGN.md](../DESIGN.md) — source connectors, observability
|