docs: Add M7 source connectors (10 tasks), M3.5.10 auth integration, remove Kong refs
- M7.1-M7.10: Extensible SourceConnector trait, Obsidian/paperless/git/S3 connectors, sync framework, CLI, HTTP endpoints, health monitoring, gate - M3.5.10: Auth integration with Authentik OIDC → Vault token validation - DESIGN.md: Add source connectors architecture, update auth to Authentik/Vault (Kong removed from cluster) - INDEX.md: 75 tasks, 11 gates - Fix all Kong references in M3.5.1 task
This commit is contained in:
@@ -0,0 +1,76 @@
|
||||
# M7.9 — Connector health monitoring + observability
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M7 — Source connectors |
|
||||
| Size | S — < 1 day |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M7.10 |
|
||||
| Depends | M7.8 |
|
||||
|
||||
## Goal
|
||||
|
||||
Add periodic health checks, sync metrics, and alerting hooks for source
|
||||
connectors so connector failures are detected before knowledge goes stale.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
A connector that silently fails means the knowledge base is stale with no
|
||||
indication. The monitoring layer detects this.
|
||||
|
||||
**Metrics (Prometheus-compatible):**
|
||||
```
|
||||
mem_source_health{name="homelab-vault",kind="obsidian"} 1 # 1=healthy, 0=unhealthy
|
||||
mem_source_last_sync_seconds{name="homelab-vault"} 1724680000 # unix timestamp
|
||||
mem_source_documents_total{name="homelab-vault"} 42
|
||||
mem_source_sync_duration_seconds{name="homelab-vault"} 12.3
|
||||
mem_source_sync_errors_total{name="homelab-vault"} 0
|
||||
mem_source_drift_new{name="homelab-vault"} 2 # docs pending sync
|
||||
mem_source_drift_changed{name="homelab-vault"} 1
|
||||
```
|
||||
|
||||
**Periodic health check.** Configurable interval (default 5 minutes). Log
|
||||
warnings for unreachable connectors. Update Prometheus gauges.
|
||||
|
||||
**Staleness alert.** If `last_sync` exceeds a configurable threshold (default 24h),
|
||||
log a warning and set a metric. This is the "knowledge is going stale" signal.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Add health check background task (tokio interval, configurable period).
|
||||
2. Export Prometheus metrics via `/metrics` endpoint (existing pattern or new).
|
||||
3. Track per-connector: health, last sync, doc count, sync duration, errors.
|
||||
4. Staleness detection: compare `last_sync` to now, warn if exceeds threshold.
|
||||
5. Log structured health events for observability.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Periodic health checks run at configured interval.
|
||||
- Metrics endpoint returns valid Prometheus format.
|
||||
- Staleness warning fires when last_sync exceeds threshold.
|
||||
- Unhealthy connector logged with error details.
|
||||
|
||||
## Verify
|
||||
|
||||
**Integration test** — `tests/it_source_monitoring.rs`:
|
||||
1. `a1_health_metric_updated` — register connector, wait for health check;
|
||||
assert metric value matches connector health.
|
||||
2. `a2_staleness_detected` — set last_sync to 25h ago; assert staleness warning.
|
||||
3. `a3_metrics_format_valid` — GET /metrics; assert valid Prometheus text format.
|
||||
4. `a4_error_count_incremented` — force connector health check failure; assert
|
||||
error counter incremented.
|
||||
|
||||
**Command:** `cargo test --test it_source_monitoring`
|
||||
|
||||
## Traps
|
||||
|
||||
- Running health checks synchronously. A connector that times out blocks all
|
||||
other connectors' health checks. Use tokio::select with timeout.
|
||||
- Not distinguishing "unreachable" from "empty". A connector that returns 0 docs
|
||||
is healthy; a connector that can't connect is not.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — source connectors, observability
|
||||
Reference in New Issue
Block a user