# M7.9 — Connector health monitoring + observability

| Field | Value |
|---|---|
| Phase | M7 — Source connectors |
| Size | S — < 1 day |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M7.10 |
| Depends | M7.8 |

## Goal

Add periodic health checks, sync metrics, and alerting hooks for source
connectors so connector failures are detected before knowledge goes stale.

## Facts (inlined — no spec read needed)

A connector that silently fails means the knowledge base is stale with no
indication. The monitoring layer detects this.

**Metrics (Prometheus-compatible):**
```
mem_source_health{name="homelab-vault",kind="obsidian"} 1      # 1=healthy, 0=unhealthy
mem_source_last_sync_seconds{name="homelab-vault"} 1724680000  # unix timestamp
mem_source_documents_total{name="homelab-vault"} 42
mem_source_sync_duration_seconds{name="homelab-vault"} 12.3
mem_source_sync_errors_total{name="homelab-vault"} 0
mem_source_drift_new{name="homelab-vault"} 2                   # docs pending sync
mem_source_drift_changed{name="homelab-vault"} 1
```

**Periodic health check.** Configurable interval (default 5 minutes). Log
warnings for unreachable connectors. Update Prometheus gauges.

**Staleness alert.** If `last_sync` exceeds a configurable threshold (default 24h),
log a warning and set a metric. This is the "knowledge is going stale" signal.

## Steps

1. Add health check background task (tokio interval, configurable period).
2. Export Prometheus metrics via `/metrics` endpoint (existing pattern or new).
3. Track per-connector: health, last sync, doc count, sync duration, errors.
4. Staleness detection: compare `last_sync` to now, warn if exceeds threshold.
5. Log structured health events for observability.

## Acceptance

- Periodic health checks run at configured interval.
- Metrics endpoint returns valid Prometheus format.
- Staleness warning fires when last_sync exceeds threshold.
- Unhealthy connector logged with error details.

## Verify

**Integration test** — `tests/it_source_monitoring.rs`:
1. `a1_health_metric_updated` — register connector, wait for health check;
   assert metric value matches connector health.
2. `a2_staleness_detected` — set last_sync to 25h ago; assert staleness warning.
3. `a3_metrics_format_valid` — GET /metrics; assert valid Prometheus text format.
4. `a4_error_count_incremented` — force connector health check failure; assert
   error counter incremented.

**Command:** `cargo test --test it_source_monitoring`

## Traps

- Running health checks synchronously. A connector that times out blocks all
  other connectors' health checks. Use tokio::select with timeout.
- Not distinguishing "unreachable" from "empty". A connector that returns 0 docs
  is healthy; a connector that can't connect is not.

---

Background: [DESIGN.md](../DESIGN.md) — source connectors, observability