diff --git a/DESIGN.md b/DESIGN.md index 816938c..0c9015b 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -32,7 +32,9 @@ The mechanism is GRU-Mem (arXiv 2602.10560). Its two gates map directly onto the | CNPG operator | **1.30.0**, supports declarative `Database.spec.extensions` | `kubectl explain database.spec.extensions` | | Ollama context cap | **32768** (`OLLAMA_CONTEXT_LENGTH`) | `k8s/apps/llm-serving/ornith.yaml` | | Controller model | `qwen2.5:3b-instruct` | is Qwen2.5-3B-Instruct — **the paper's exact 3B backbone** | -| Gateway auth | `apikey:` header, **not** `Authorization: Bearer` | Kong key-auth compares whole header value | +| Gateway | nginx ingress + homelab-frontend (Go, ServiceAdapter CRDs) | Kong is **removed** from cluster | +| IAM | Authentik (OIDC) → Vault (OIDC auth method) | Authentik issues JWT, Vault validates via OIDC, `homelab-admin` role bound on `permissions: "*"` claim | +| Auth (placeholder) | `apikey:` header in mem-cli | **To be replaced** — does not integrate with Authentik/Vault chain | The 32K cap is the binding constraint and it fits: paper uses 5000-token chunks, 8192 max prompt, 2048 max response. @@ -57,7 +59,7 @@ Memory is **levelled**, and every event in the log carries its level. The paper ## Architecture ``` - api.riotpiao.com (Kong) + api.riotpiao.com (nginx ingress + homelab-frontend gateway) │ ┌─────────────────┼─────────────────┐ │ │ │ @@ -400,6 +402,211 @@ Two mechanical guards: 1. **Promotion is a human move** out of `_drafts/`, reviewable as a diff. 2. **Provenance marks derived text.** Every emitted skill carries `generated_from: ` in frontmatter, and `mem-ingest` tags chunks matching a known emitted artifact as `derived: true` and excludes them from evidence. Without this the corpus slowly becomes its own training data. +## Source connectors — extensible multi-source ingestion + +Memory service is a **cluster-wide RAG** serving multiple agents and workflows. Knowledge does not live in one place — it is spread across an Obsidian vault, a paperless-ngx instance, agent session logs, and whatever document stores appear next. The connector architecture makes adding a new source a single-trait implementation rather than a pipeline rewrite. + +### The problem with hardcoded sources + +Today `mem-ingest` has three concrete sources: `PiSessionSource`, `ClaudeTranscriptSource`, `DocCorpusSource`. Each knows its own file format and emits `Record`s through the `RecordSource` trait. This works, but: + +1. **Every new source requires Rust code.** paperless-ngx has a REST API; Confluence has another; S3 is a third protocol. Each lands as a new `.rs` file compiled into the binary. +2. **No runtime discovery.** The CLI must know about every source at compile time. A homelab that adds Bookstack next month needs a code change, a rebuild, and a redeploy. +3. **No shared sync logic.** Change detection (sha-based skip), tombstoning, drift reporting — each source will reimplement these unless the framework provides them. + +### Design: `SourceConnector` trait + registry + +A connector is anything that can enumerate documents and yield their content. The framework handles chunking, embedding, change detection, and storage. + +```rust +/// A connector to an external document source. +#[async_trait] +pub trait SourceConnector: Send + Sync { + /// Unique connector kind identifier (e.g., "obsidian", "paperless", "s3") + fn kind(&self) -> &str; + + /// Human-readable name for this connector instance + fn name(&self) -> &str; + + /// Enumerate all documents available from this source. + /// Returns (doc_id, doc_metadata) pairs. + async fn list_documents(&self) -> Result>; + + /// Fetch content for a single document by its source-specific ID. + async fn fetch_document(&self, doc_id: &str) -> Result; + + /// Health check — can we reach this source? + async fn health_check(&self) -> Result; +} + +/// Metadata about a source document (before fetching content) +pub struct SourceDocument { + pub doc_id: String, // source-specific unique ID + pub title: String, // human-readable title + pub source_uri: String, // canonical URI (file://, https://, paperless://) + pub content_hash: Option, // if source provides hash, skip fetch on match + pub mime_type: String, // text/markdown, application/pdf, etc. + pub updated_at: Option, +} + +/// Fetched document content ready for chunking +pub struct DocumentContent { + pub doc_id: String, + pub text: String, // extracted text (markdown preferred) + pub source_uri: String, + pub content_hash: String, // sha256 of text content + pub metadata: HashMap, // source-specific metadata +} + +/// Source health status +pub struct SourceHealth { + pub reachable: bool, + pub document_count: Option, + pub last_error: Option, +} +``` + +### Connector registry + +Connectors are registered at startup from a YAML config file. Adding a new source is: implement the trait, register the kind, add a config block. + +```yaml +# connectors.yaml — source connector configuration +connectors: + - kind: obsidian + name: homelab-vault + config: + root: /data/vault # local path (PVC mount or git-sync) + extensions: [md, markdown, txt] + exclude_dirs: [.obsidian, .trash] + + - kind: paperless + name: homelab-paperless + config: + base_url: http://paperless-ngx.paperless.svc.cluster.local:8000 + token_secret: paperless-api-token # k8s secret name + tags: [reference, manual, runbook] # only sync docs with these tags + format: markdown # request markdown export + + - kind: git_repo + name: infra-docs + config: + repo_url: https://forgejo.riotpiao.com/rock/homelab-docs.git + branch: main + sync_interval: 1h + paths: [docs/, runbooks/] + + - kind: s3 + name: backup-docs + config: + endpoint: https://minio.riotpiao.com + bucket: knowledge-base + prefix: docs/ + access_key_secret: minio-credentials +``` + +### Sync framework + +The connector trait provides enumeration and fetch. The **sync framework** handles everything else: + +1. **Change detection:** Compare `content_hash` from `list_documents()` against last-known hash in the manifest. Skip unchanged docs (zero embed calls). +2. **Chunking:** Route through existing `ChunkPolicy` — heading-based for markdown, paragraph-based for plain text, configurable per connector. +3. **Tombstoning:** Documents that vanish from `list_documents()` get tombstone records in the log. Append-only, never delete. +4. **Drift reporting:** `mem source status` shows per-connector: docs total, changed, new, removed — without mutating anything. +5. **Rate limiting:** Configurable fetch concurrency per connector (don't DDoS paperless with 500 parallel fetches). +6. **Resume:** Sync is resumable. Crash mid-sync, restart, only unseen docs are processed. + +``` +mem source sync --all # sync all registered connectors +mem source sync --name homelab-vault # sync one connector +mem source status # drift report, no mutations +mem source list # registered connectors + health +mem source add --kind paperless ... # register new connector +mem source rm --name old-source # deregister + tombstone +``` + +### Built-in connectors (shipped with binary) + +| Kind | Source | Protocol | Status | +|------|--------|----------|--------| +| `obsidian` | Local Obsidian vault | filesystem (walkdir) | M7.2 | +| `paperless` | paperless-ngx | REST API (`/api/documents/`) | M7.3 | +| `git_repo` | Git repository | git clone/pull | M7.4 | +| `s3` | S3-compatible storage | S3 API (minio, AWS) | M7.5 | +| `pi_session` | Pi agent sessions | filesystem (existing) | M7.1 (wrap existing) | +| `claude` | Claude transcripts | filesystem (existing) | M7.1 (wrap existing) | + +### How connectors interact with levels + +**Session connectors** (pi, claude) produce evidence that flows through the gated loop → L0/L1/L2. These are the existing `RecordSource` implementations, wrapped in `SourceConnector` for unified management. + +**Document connectors** (obsidian, paperless, git, s3) produce reference material → Level R. These bypass the gated loop entirely (no standing question, no gate decision). This is the existing M3.6 design, now generalized. + +The connector kind determines the pipeline: +``` +Session connector → RecordSource → ChunkPolicy → GatedLoop → L0/L1/L2 +Document connector → SourceConnector → ChunkPolicy → Level R (no gate) +``` + +### paperless-ngx integration (concrete example) + +paperless-ngx is already running in the cluster. It has: +- REST API at `http://paperless-ngx.paperless.svc.cluster.local:8000/api/` +- Documents with tags, correspondents, document types +- Full-text content available via API +- Thumbnail and original file access + +```rust +pub struct PaperlessConnector { + base_url: String, + token: String, + tag_filter: Vec, +} + +#[async_trait] +impl SourceConnector for PaperlessConnector { + fn kind(&self) -> &str { "paperless" } + fn name(&self) -> &str { &self.name } + + async fn list_documents(&self) -> Result> { + // GET /api/documents/?tags__name__in=reference,manual + // Paginate through results + // Return doc_id, title, checksum (paperless provides this) + } + + async fn fetch_document(&self, doc_id: &str) -> Result { + // GET /api/documents/{id}/ + // Extract content field (full text) + // Or GET /api/documents/{id}/download/ for original + } + + async fn health_check(&self) -> Result { + // GET /api/ — check 200 + } +} +``` + +### Obsidian vault as a connector + +The Obsidian vault is the **primary reference source**. It is human-maintained, git-backed, and the canonical location for runbooks, procedures, and domain knowledge that agents need. + +Deployment options: +1. **Git-sync sidecar:** A sidecar container clones the vault repo into a shared PVC. Memory service reads from the PVC. +2. **Local mount:** For development, mount the vault directory directly. +3. **API upload:** Push vault changes to memory service via HTTP. + +The Obsidian connector wraps the existing `DocCorpusSource` (M3.6.1) with the `SourceConnector` interface, adding change detection and registry management. + +### Future extensibility + +Adding a new source requires: +1. Implement `SourceConnector` trait (~100-200 lines) +2. Register the `kind` in the connector factory +3. Add config block to `connectors.yaml` +4. Run `mem source sync --name new-source` + +No changes to the ingest pipeline, chunking, embedding, storage, or query layers. The connector is the only new code. + ## Reference corpora — the non-evidential tier `RecordSource` takes session transcripts, and the update gate asks "does this chunk contain evidence for Q". A `kubectl` or `tea` cheatsheet answers neither question: it has no session, no turn, and no evidence. Left alone the system faithfully retains *what happened when a model used a tool badly* and never learns the tool. Level **R** closes that gap, and the shape of the fix matters more than the fact of it. @@ -509,6 +716,8 @@ A response that cannot fit its tier-1 hits inside the budget is an error, not a **P6 — Post-training (separate, Python).** Boundary is the JSONL. `mem label` uses the 32B `reasoning` model as an offline evidence labeler to produce per-chunk `U_t` ground truth (the paper had synthetic NIAH labels; we do not, and this is the honest cheapest substitute). Then verl trains a LoRA with the paper's rewards: `r_update` ±1, `r_exit` {0, −0.5 late, −0.75 early}, strict `r_format`, `α=0.9` mixing trajectory- and turn-level advantage. Requires the vLLM decision above. +**P7 — Source connectors (board `M7`).** Generalizes the ingestion layer from hardcoded file sources to an extensible `SourceConnector` trait with a YAML-driven registry. Existing sources (pi sessions, Claude transcripts, `DocCorpusSource`) are wrapped in the new interface; new sources (paperless-ngx, git repos, S3) implement the trait directly. The sync framework handles change detection, tombstoning, drift reporting, and resumable sync for all connectors. The connector is the only new code when a source is added — no changes to chunking, embedding, storage, or query layers. Ordered after M3.6 because it generalizes `mem ref` rather than duplicating it, and after M4.2 because document connectors must register in the derived-content manifest to prevent cycle contamination. + ## Task breakdown Board lives in **`memory-tasks/`** at the repo root. Format follows `agent-rust/tasks/`: one file per task, self-contained, each with `Acceptance` / `Verify` (harness, numbered assertions, command) / `False pass` / `Traps`, a `Status` field as source of truth, and `memory-tasks/INDEX.md` mirroring it. Ids are `M.` and frozen once written — phase order is declared in `INDEX.md`, never derived from the id. @@ -584,7 +793,7 @@ Note `agent-rust/.gitignore` excludes `tasks`, which silently untracks the whole | id | task | size | deps | |---|---|---|---| -| M3.5.1 | HTTP server + router (actix-web or axum), Kong auth hook, request metrics | M | M0.1 | +| M3.5.1 | HTTP server + router (actix-web or axum), auth hook, request metrics | M | M0.1 | | M3.5.2 | `POST /ingest` endpoint — `ingest_id` dedup, async queue, git context enrichment | M | M1.7, M3.5.1 | | M3.5.3 | `GET /query` endpoint — embed query, HNSW recall by level, rerank, walk edges to L0 | M | M3.3, M3.5.1 | | M3.5.4 | Federation: single query across projects, fan+merge results, deduplicate | M | M3.5.3 | @@ -605,7 +814,22 @@ Note `agent-rust/.gitignore` excludes `tasks`, which silently untracks the whole | M5.5 | verl loop — `r_update` ±1, `r_exit` {0,−0.5,−0.75}, strict `r_format`, α=0.9 | L | M5.3, M5.4 | | M5.6 | **M5 gate** — adapter beats prompted baseline on held-out update accuracy | L | gate | -Total 52 tasks, 8 gates. M0 and M2.2 have no model dependency and can start immediately; M5.4 is homelab work independent of everything else in M5 and can run in parallel. M3.5 depends on M2 (pgvector store exists) and M1 (ingest loop exists); can run in parallel with M4 and M5. M3.5.9 (git-aware references) is optional, depends on M3.5.2. +**M7 — Source connectors** (extensible multi-source ingestion) + +| id | task | size | deps | +|---|---|---|---| +| M7.1 | `SourceConnector` trait + `SourceDocument` / `DocumentContent` types + connector registry | M | M0.3, M3.6.1 | +| M7.2 | Obsidian vault connector — wraps `DocCorpusSource`, adds change detection, config-driven | M | M7.1, M3.6.1 | +| M7.3 | paperless-ngx connector — REST API client, tag filtering, markdown export | M | M7.1 | +| M7.4 | Git repo connector — clone/pull, path filtering, branch tracking | M | M7.1 | +| M7.5 | S3 connector — S3-compatible API, prefix filtering, content-type routing | M | M7.1 | +| M7.6 | Sync framework — change detection, tombstoning, drift report, resume | L | M7.1, M3.6.3 | +| M7.7 | `mem source` CLI — sync/status/list/add/rm subcommands | M | M7.6 | +| M7.8 | `GET /memory/sources` + `POST /memory/sources/sync` HTTP endpoints | M | M7.7, M3.5.1 | +| M7.9 | Connector health monitoring + observability | S | M7.8 | +| M7.10 | **M7 composition gate** — two connectors sync, change detection skips unchanged, tombstone works, rebuild parity | M | gate | + +Total 74 tasks, 11 gates. M0 and M2.2 have no model dependency and can start immediately; M5.4 is homelab work independent of everything else in M5 and can run in parallel. M3.5 depends on M2 (pgvector store exists) and M1 (ingest loop exists); can run in parallel with M4 and M5. M3.5.9 (git-aware references) is optional, depends on M3.5.2. M7 depends on M3.6 (reference corpora) and M4.2 (derived filter) for cycle prevention. ## Verification @@ -677,7 +901,9 @@ The decisive P2 metric is **update-rate**, the one number distinguishing a worki ## Distributed API Layer (Homelab Frontend) -**Gateway:** `api.riotpiao.com` routes agent and system memory requests through Kong. +**Gateway:** `api.riotpiao.com` routes requests through **nginx ingress** to **homelab-frontend** gateway (Go, ServiceAdapter CRDs). Kong is **removed** from the cluster. + +**Auth:** Authentik (OIDC provider) → Vault (OIDC auth method, `auth/oidc/role/homelab-admin`, bound on `permissions: "*"` claim) → Vault token → service validates token. Memory service currently uses placeholder `apikey` header — M3.5.10 replaces this with Vault token validation. **Architecture assumption:** Memory services run in CNPG cluster; API layer is HTTP facade exposing read/write workflows to distributed agents. Authority remains JSONL—API is a request demultiplexer, not a cache or alternative source of truth. @@ -748,7 +974,29 @@ POST /memory/context <- 3-tier lookup: signature, symptom, doc - Client-side: `ETag: ` on all read endpoints, no conditional logic server-side (it's stateless) **Auth & rate limits:** -- Kong `apikey:` header (existing pattern) + +*Auth chain (production):* +``` +User/Agent → Authentik (OIDC login) + → JWT with claims (including `permissions`) + → Vault validates via OIDC auth method + → Vault issues token based on role match + → Services validate Vault token or trust gateway-forwarded identity + +Authentik OIDC: https://authentik.riotpiao.com/application/o/vault/ +Vault OIDC role: auth/oidc/role/homelab-admin + bound_claims: { "permissions": "*" } + policy: homelab-admin (path "*" full access) +``` + +*Current (placeholder, to be replaced):* +- `apikey:` header, raw string match — **does not integrate with Authentik/Vault** +- Must be replaced with one of: + 1. Vault token validation (call Vault's `auth/token/lookup-self`) + 2. Authentik JWT validation via JWKS + 3. Trust gateway-forwarded headers (`X-User-Id`, `X-Capabilities`) + +*Rate limits (unchanged):* - Per-key limits: ingest 100 jobs/hour, query 1000 req/hour, skill fetch unlimited - Burst allowance: 10 req/sec per key (ingest waits in queue; query returns 429 Retry-After if burst exceeded) diff --git a/tasks/INDEX.md b/tasks/INDEX.md index 2b4e762..3c198ec 100644 --- a/tasks/INDEX.md +++ b/tasks/INDEX.md @@ -62,7 +62,7 @@ Legend: ⬜ not started · 🟡 in progress · ✅ done · ⛔ blocked | 2 | Gated loop at L1 | M1.x | 8 | 8 | 0 | 0 | ✅ M1.8 | | 3 | Projections | M2.x | 8 | 5 | 0 | 3 | ✅ M2.8 (M2.1, M2.3, M2.4, M2.5 ✅) | | 4 | L2 synthesis + retrieval | M3.x | 4 | 4 | 0 | 0 | ✅ M3.4 | -| 4.5 | Distributed API Layer | M3.5.x | 9 | 8 | 0 | 1 | ✅ M3.5.8 | +| 4.5 | Distributed API Layer | M3.5.x | 10 | 8 | 0 | 2 | ✅ M3.5.8 | | 5 | Skills | M4.x | 3 | 2 | 0 | 1 | ⬜ M4.3 | | 5.5 | Reference corpora | M3.6.x | 6 | 1 | 0 | 5 | ⬜ M3.6.6 | | 5.6 | Tool context | M3.7.x | 6 | 0 | 2 | 4 | ⬜ M3.7.6 | @@ -146,7 +146,7 @@ Homelab frontend integration: HTTP facade via `api.riotpiao.com`. Runs in parall | Task | Title | Size | Flags | Status | |---|---|---|---|---| -| [M3.5.1](M3.5.1-http-server.md) | HTTP server + router, Kong auth, metrics | M | — | ✅ | +| [M3.5.1](M3.5.1-http-server.md) | HTTP server + router, auth hook, metrics | M | — | ✅ | | [M3.5.2](M3.5.2-ingest-endpoint.md) | POST /ingest async queue | M | — | ✅ | | [M3.5.3](M3.5.3-query-endpoint.md) | GET /query HNSW+rerank | M | — | ✅ | | [M3.5.4](M3.5.4-query-federation.md) | Query federation | M | — | ✅ | @@ -155,6 +155,7 @@ Homelab frontend integration: HTTP facade via `api.riotpiao.com`. Runs in parall | [M3.5.7](M3.5.7-rate-limiting.md) | Rate limiting | M | — | ✅ | | [M3.5.8](M3.5.8-m3.5-gate.md) | **M3.5 composition gate** | M | gate | ✅ | | [M3.5.9](M3.5.9-git-aware-references.md) | Git-aware references: lookup by code location | M | — | ⬜ | +| [M3.5.10](M3.5.10-auth-integration.md) | Auth: Authentik/Vault OIDC token validation | M | — | ⬜ | ## 5 — Skills · M4.x @@ -254,6 +255,33 @@ durability-of-location, not a multi-host requirement. | [M6.5](M6.5-credentials-secret.md) | Postgres credentials for the Mac client | S | homelab | ⬜ | | [M6.6](M6.6-m6-gate.md) | **M6 composition gate** | M | gate | ⬜ | +## 8 — Source connectors · M7.x + +Extensible multi-source ingestion. `SourceConnector` trait + YAML-driven registry. +Adding a new document source (paperless-ngx, S3, git repo) requires implementing +one trait and adding one config block — no changes to the ingest pipeline, chunking, +embedding, storage, or query layers. + +**Document connectors** produce Level R content (reference material, bypasses gated +loop). **Session connectors** (pi, claude) produce evidence for L0/L1/L2. The +connector's `source_type()` declares the pipeline. + +Sync framework handles change detection (sha-based skip), tombstoning, drift +reporting, and resumable sync for all connectors. + +| Task | Title | Size | Flags | Status | +|---|---|---|---|---| +| [M7.1](M7.1-source-connector-trait.md) | `SourceConnector` trait + registry | M | — | ⬜ | +| [M7.2](M7.2-obsidian-connector.md) | Obsidian vault connector | M | — | ⬜ | +| [M7.3](M7.3-paperless-connector.md) | paperless-ngx connector | M | — | ⬜ | +| [M7.4](M7.4-git-repo-connector.md) | Git repo connector | M | — | ⬜ | +| [M7.5](M7.5-s3-connector.md) | S3-compatible storage connector | M | — | ⬜ | +| [M7.6](M7.6-sync-framework.md) | Sync framework | L | — | ⬜ | +| [M7.7](M7.7-source-cli.md) | `mem source` CLI | M | — | ⬜ | +| [M7.8](M7.8-source-http-endpoints.md) | Source HTTP endpoints | M | — | ⬜ | +| [M7.9](M7.9-connector-health-monitoring.md) | Connector health + observability | S | — | ⬜ | +| [M7.10](M7.10-m7-gate.md) | **M7 composition gate** | M | gate | ⬜ | + --- Background: [DESIGN.md](../DESIGN.md) · GRU-Mem, arXiv 2602.10560 · `internal/store/store.go` (agent-manager, `add-headless-spawn` branch) diff --git a/tasks/M3.5.1-http-server.md b/tasks/M3.5.1-http-server.md index 19db533..2b2c6ba 100644 --- a/tasks/M3.5.1-http-server.md +++ b/tasks/M3.5.1-http-server.md @@ -1,4 +1,4 @@ -# M3.5.1 — HTTP server + router, Kong auth hook, metrics +# M3.5.1 — HTTP server + router, auth hook, metrics | Field | Value | |---|---| @@ -11,12 +11,12 @@ ## Goal -HTTP facade for homelab gateway. Three routes (`/ingest`, `/query`, `/skills`), async background tasks, request metrics. Auth hook validates Kong `apikey:` header. Stateless — no business logic here, just request demultiplexing. +HTTP facade for homelab gateway. Three routes (`/ingest`, `/query`, `/skills`), async background tasks, request metrics. Auth hook validates `apikey:` header (placeholder — M3.5.10 replaces with Vault/Authentik OIDC). Stateless — no business logic here, just request demultiplexing. ## Architecture ``` -Kong (api.riotpiao.com) +nginx ingress + homelab-frontend gateway (api.riotpiao.com) ↓ apikey validation HTTP Server (Rust httpd, actix-web or axum) ↓ route dispatch @@ -33,7 +33,7 @@ HTTP Server (Rust httpd, actix-web or axum) - `GET /memory/skills` — returns 200 with empty skills `[]` 4. Request logger middleware — every request logs method, path, status, latency in one line (not pretty-printed). 5. Metrics middleware — track latency histogram per route (p50/p95/p99 in microseconds), request count, error count. -6. Kong auth hook: +6. Auth hook (placeholder): - Extract `apikey:` header (case-insensitive header name, exact value match against stored key) - If missing or unrecognized → 401 with `{"error":"unauthorized","reason":"missing apikey header"}` - Pass apikey to request context so handlers can log which key made the request @@ -73,7 +73,7 @@ HTTP Server (Rust httpd, actix-web or axum) ## Traps - Actix-web's `.service()` does not inherit middleware registered outside a scope; scope middleware applies only to routes inside that scope. -- Header name case matters for Kong's key-auth; `apikey:` is lowercase. +- Header name: `apikey:` (lowercase). Placeholder — M3.5.10 replaces with Vault token validation. - `tokio::runtime::Runtime::new()` in tests blocks on network if used naively — use test utilities from `actix-web` or `axum` that spawn the server in a background thread. - Metrics registered at startup are easy to forget to increment. Middleware must actually call the metrics update, not just define it. diff --git a/tasks/M3.5.10-auth-integration.md b/tasks/M3.5.10-auth-integration.md new file mode 100644 index 0000000..1672706 --- /dev/null +++ b/tasks/M3.5.10-auth-integration.md @@ -0,0 +1,178 @@ +# M3.5.10 — Auth integration with Authentik/Vault + +| Field | Value | +|---|---| +| Phase | M3.5 — Distributed API Layer | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | — | +| Depends | M3.5.1 | + +## Goal + +Replace the placeholder `apikey` header check with proper authentication via the +cluster's IAM stack: **Authentik** (OIDC provider) → **HashiCorp Vault** (token +issuer) → **memory service** (token validator). + +## Facts (inlined — no spec read needed) + +**Current (wrong):** +```rust +fn check_auth(req: &HttpRequest, state: &AppState) -> Result<(), HttpResponse> { + let api_key = req.headers().get("apikey").and_then(|h| h.to_str().ok()); + if api_key != Some(&state.api_key) { + return Err(HttpResponse::Unauthorized().json(...)); + } + Ok(()) +} +``` +This is a raw string match against `MEM_API_KEY` env var. No JWT, no Vault, no +user identity. It does not integrate with the cluster's IAM stack. + +**Cluster IAM stack:** +- **Authentik** (`iam` namespace) — OIDC provider at + `https://authentik.riotpiao.com/application/o/vault/` +- **HashiCorp Vault** (`iam` namespace) — OIDC auth method enabled, validates + Authentik JWTs, issues Vault tokens based on role/policy. +- **Vault OIDC role:** `auth/oidc/role/homelab-admin` + - `bound_claims: { "permissions": "*" }` + - Policy: `homelab-admin` (path `"*"` full access) +- **Vault unseal:** Shamir 3/3, keys in `vault-unseal-keys` secret, S3 backend + via MinIO. + +**Auth flow (production):** +``` +User/Agent authenticates with Authentik (OIDC) + → Receives JWT with claims { sub, permissions, groups, ... } + → Presents JWT to Vault OIDC auth method + → Vault validates JWT against Authentik JWKS + → Vault issues Vault token with matched policy + → Client sends Vault token to memory service + → Memory service validates token via Vault API +``` + +**Three integration options (pick one):** + +### Option A: Vault token validation (recommended) +Memory service receives `X-Vault-Token` header, calls Vault's +`POST /v1/auth/token/lookup-self` to validate. Extracts policy and metadata. +- Pro: Vault is the single source of truth for authorization. +- Pro: Token revocation is immediate (Vault controls lifecycle). +- Con: Extra network call per request (cache with TTL to mitigate). + +### Option B: Direct JWKS validation +Memory service fetches Authentik's JWKS endpoint, validates JWT `Authorization: +Bearer ` directly. No Vault in the request path. +- Pro: No Vault dependency at request time. +- Pro: Standard OAuth2/OIDC pattern. +- Con: Token revocation is delayed (until JWT expires). +- Con: Memory service must know about Authentik's OIDC config. + +### Option C: Trust gateway +Memory service trusts homelab-frontend gateway (cluster-internal traffic). +Gateway validates auth, forwards `X-User-Id` and `X-Capabilities` headers. +Memory service checks capabilities against ServiceAdapter CRD requirements. +- Pro: Auth logic centralized in gateway. +- Pro: Memory service stays simple. +- Con: Gateway auth is currently a stub (`hasCapability()` returns true for any + `Authorization` header). +- Con: Requires gateway auth to be completed first (homelab-frontend task 8.3). + +**ServiceAdapter CRD for memory (`memory-adapter`):** +```yaml +auth: + capability: memory:read # default + required: true +resources: + - name: ingest + methods: + - verb: POST + auth: { capability: memory:write, required: true } + - name: query + methods: + - verb: POST + - name: skills + methods: + - verb: GET +``` + +**Capabilities needed:** +- `memory:read` — query, skills, vault browse, projects, sources +- `memory:write` — ingest, source sync, skill draft + +## Steps + +### Option A (Vault token — recommended) + +1. Add `vault_addr` to `AppState` (default: `http://vault.iam.svc.cluster.local:8200`). +2. Replace `check_auth()` with `validate_vault_token()`: + ```rust + async fn validate_vault_token(req: &HttpRequest, state: &AppState) -> Result { + let token = req.headers().get("X-Vault-Token") + .or_else(|| req.headers().get("Authorization")) // Bearer + .and_then(|h| h.to_str().ok()); + // POST vault_addr/v1/auth/token/lookup-self with X-Vault-Token header + // Parse response: policies, metadata, ttl + // Cache token -> identity for TTL duration + } + ``` +3. Add token cache (HashMap) with configurable TTL. +4. Extract `VaultIdentity` (policies, metadata) from lookup response. +5. Map policies to capabilities: `homelab-admin` → `memory:read` + `memory:write`. +6. Update each handler to check required capability. +7. Keep `apikey` as fallback for dev/test (controlled by env var `MEM_AUTH_MODE=vault|apikey`). + +### For all options + +8. Add env vars: `VAULT_ADDR`, `MEM_AUTH_MODE` (vault/jwks/gateway/apikey). +9. Update K8s deployment to inject `VAULT_ADDR`. +10. Update ServiceAdapter CRD if needed. +11. Document auth flow in README. + +## Acceptance + +- Requests with valid Vault token are accepted. +- Requests with expired/revoked Vault token are rejected (401). +- Requests without any auth are rejected (401). +- `memory:write` capability required for ingest/sync endpoints. +- `memory:read` capability sufficient for query/skills/vault endpoints. +- Token cache reduces Vault API calls on repeated requests. +- Fallback to `apikey` mode for dev/test environments. + +## Verify + +**Integration test** — `tests/it_auth_integration.rs`: +1. `a1_vault_token_accepted` — mock Vault lookup-self returning valid response; + assert request proceeds. +2. `a2_expired_token_rejected` — mock Vault returning 403; assert 401 response. +3. `a3_no_auth_rejected` — request with no auth headers; assert 401. +4. `a4_write_requires_capability` — token with `memory:read` only; POST /ingest; + assert 403. +5. `a5_read_with_read_capability` — token with `memory:read`; GET /query; + assert proceeds. +6. `a6_token_cache_hit` — same token twice; assert Vault called once. +7. `a7_apikey_fallback` — `MEM_AUTH_MODE=apikey`; assert old behavior works. +8. `a8_auth_mode_configurable` — assert `MEM_AUTH_MODE` switches validation logic. + +**Command:** `cargo test --test it_auth_integration` + +**False pass:** +- Testing only with apikey fallback. The Vault integration is the whole point. +- Mocking Vault without testing cache expiry. A cache that never expires accepts + revoked tokens forever. + +## Traps + +- Calling Vault on every request without caching. Vault API calls add 5-10ms + per request. Cache with TTL matching token TTL (or shorter). +- Not handling Vault being temporarily unreachable. Return 503 (not 401) if + Vault is down — "cannot verify" is not "unauthorized". +- Hardcoding Vault addr. Use env var + service discovery. +- Not supporting `Authorization: Bearer ` format alongside `X-Vault-Token`. + Different clients use different conventions. + +--- + +Background: [DESIGN.md](../DESIGN.md) — auth section, Authentik/Vault IAM stack diff --git a/tasks/M7.1-source-connector-trait.md b/tasks/M7.1-source-connector-trait.md new file mode 100644 index 0000000..0d1ea64 --- /dev/null +++ b/tasks/M7.1-source-connector-trait.md @@ -0,0 +1,122 @@ +# M7.1 — `SourceConnector` trait + registry + +| Field | Value | +|---|---| +| Phase | M7 — Source connectors | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M7.2, M7.3, M7.4, M7.5, M7.6, M7.10 | +| Depends | M0.3, M3.6.1 | + +## Goal + +Define the extensible connector interface so that adding a new document source +(paperless-ngx, S3, git repo, etc.) requires implementing one trait and adding +one YAML config block — no changes to the ingest pipeline, chunking, embedding, +storage, or query layers. + +## Facts (inlined — no spec read needed) + +Memory service is a **cluster-wide RAG** serving multiple agents. Knowledge lives +in many places: an Obsidian vault, paperless-ngx, git repositories, S3 buckets. +The connector abstraction makes all of them look the same to the ingest pipeline. + +**The trait has three methods.** `list_documents()` enumerates what's available +without fetching content. `fetch_document()` retrieves one document's text. +`health_check()` reports reachability. The sync framework (M7.6) handles +everything else — change detection, tombstoning, chunking, embedding. + +**Configuration is YAML-driven.** Each connector instance is a block in +`connectors.yaml` with `kind`, `name`, and source-specific `config`. The registry +maps `kind` to a factory function that constructs the connector from its config. + +**Two connector families exist.** Session connectors (pi, claude) produce evidence +for the gated loop (L0/L1/L2). Document connectors (obsidian, paperless, git, s3) +produce reference material at Level R, bypassing the gate. The connector's +`source_type()` method declares which family it belongs to. + +## Steps + +1. Define `SourceConnector` trait in `mem-ingest/src/connector.rs`: + ```rust + #[async_trait] + pub trait SourceConnector: Send + Sync { + fn kind(&self) -> &str; + fn name(&self) -> &str; + fn source_type(&self) -> SourceType; // Evidence or Reference + async fn list_documents(&self) -> Result>; + async fn fetch_document(&self, doc_id: &str) -> Result; + async fn health_check(&self) -> Result; + } + ``` +2. Define supporting types: `SourceDocument`, `DocumentContent`, `SourceHealth`, + `SourceType` enum (`Evidence`, `Reference`). +3. Define `ConnectorConfig` serde struct for YAML deserialization: + ```yaml + connectors: + - kind: obsidian + name: homelab-vault + config: { root: /data/vault, extensions: [md, txt] } + ``` +4. Implement `ConnectorRegistry` — maps `kind` string to factory function, + constructs connectors from config at startup. +5. Add `connectors.yaml` loading in `mem-cli` startup path. +6. Provide a `VecConnector` test helper (in-memory documents) for testing + downstream consumers without real I/O. + +## Acceptance + +- The trait compiles and is object-safe (`Box`). +- `ConnectorRegistry` can register and construct connectors by kind string. +- `VecConnector` implements the trait and passes basic list/fetch assertions. +- YAML config deserialization works for known and unknown kinds (unknown = skip + with warning, not crash). +- `source_type()` is enforced at the type level — no runtime flag confusion. + +## Verify + +**Harness:** in-memory `VecConnector`, YAML config fixtures. + +**Integration test** — `tests/it_source_connector.rs`: +1. `a1_trait_is_object_safe` — construct a `Box` from + `VecConnector`; call all three methods. +2. `a2_registry_constructs_by_kind` — register "vec" kind, construct from config, + assert `kind()` and `name()` match. +3. `a3_list_documents_returns_all` — `VecConnector` with 3 docs, assert + `list_documents()` returns 3. +4. `a4_fetch_document_by_id` — assert content matches what was registered. +5. `a5_fetch_unknown_id_errors` — assert `fetch_document("nonexistent")` returns + an error, not a panic. +6. `a6_health_check_reports_count` — assert `health_check()` returns + `document_count = Some(3)`. +7. `a7_yaml_config_loads` — parse a fixture `connectors.yaml` with two connector + blocks; assert both are constructed. +8. `a8_unknown_kind_skipped` — config with `kind: "nonexistent"`; assert registry + logs a warning and continues without the connector. +9. `a9_source_type_evidence_vs_reference` — assert session connectors return + `Evidence`, document connectors return `Reference`. +10. `a10_empty_config_is_valid` — no `connectors.yaml` or empty file; registry + starts with zero connectors, no crash. + +**Command:** `cargo test --test it_source_connector` + +**False pass:** +- Testing only `VecConnector` and claiming the trait works. The trait is + validated by M7.2–M7.5 implementing it against real sources. +- Parsing YAML without verifying the constructed connector's methods work. + +## Traps + +- Making the trait not object-safe (generic methods, `Self` in return types). + Every consumer stores `Box`, so object safety is load-bearing. +- Putting chunking logic inside the connector. Connectors fetch documents; the + sync framework (M7.6) chunks them. Mixing concerns means every connector + reimplements chunking. +- Hard-coding the list of known kinds. The registry must be extensible — a + `register(kind, factory_fn)` call, not a match statement. + +--- + +Background: [DESIGN.md](../DESIGN.md) — source connectors section diff --git a/tasks/M7.10-m7-gate.md b/tasks/M7.10-m7-gate.md new file mode 100644 index 0000000..e1762dd --- /dev/null +++ b/tasks/M7.10-m7-gate.md @@ -0,0 +1,92 @@ +# M7.10 — M7 composition gate — source connectors + +| Field | Value | +|---|---| +| Phase | M7 — Source connectors | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | gate | +| Spec | inlined below | +| Blocks | — | +| Depends | M7.1, M7.2, M7.3, M7.6, M7.7, M7.8, M7.9 | + +## Goal + +Prove the connector framework composes: two different connector kinds sync +through the same framework, change detection skips unchanged documents, tombstoning +works, drift reports are accurate, rebuild parity holds, and the gated loop's +update-rate is untouched. + +## What the gate proves + +1. **Extensibility works.** Two different connector kinds (at minimum: obsidian + + one remote connector) register, sync, and produce queryable Level R content + through the same `SyncEngine`. No connector-specific code exists outside the + connector itself. + +2. **Change detection is efficient.** Re-syncing an unchanged connector produces + zero embedding calls. This is the cost guard — without it, every sync is a + full re-embed. + +3. **Tombstoning is correct.** Removing a document from a source results in + tombstone records in the log, removal from the index, and correct behavior + on rebuild. + +4. **Drift report is read-only.** Running `mem source status` mutates nothing. + +5. **Rebuild parity holds for connectors.** `mem rebuild --from-log` with + connector-sourced data produces byte-identical results. + +6. **Update-rate is untouched.** Adding connector-sourced reference documents + does not change the gated loop's update-rate (same property M3.6.6 asserts, + now for any connector). + +7. **Health monitoring detects failures.** An unreachable connector is reported, + not silently ignored. + +## Verify + +**Integration test** — `tests/it_m7_gate.rs`: + +1. `a1_two_kinds_sync` — register an obsidian + vec connector; sync both; assert + Level R nodes exist for both sources. +2. `a2_unchanged_zero_embeds` — re-sync both; assert zero embedding calls. +3. `a3_change_detected_and_replaced` — modify a doc in one connector; sync; + assert old chunks tombstoned, new chunks present. +4. `a4_removal_tombstoned` — remove a doc; sync; assert tombstone records and + doc absent from query results. +5. `a5_drift_report_is_read_only` — snapshot log + manifest; run status; assert + unchanged. +6. `a6_rebuild_parity` — after full sync, rebuild from log; assert byte-identical + state. +7. `a7_update_rate_untouched` — record update-rate before adding connectors; + add connectors + sync; re-run gated loop; assert update-rate unchanged. +8. `a8_health_failure_reported` — configure unreachable connector; assert + health check reports failure. +9. `a9_no_connector_specific_code_in_sync` — assert `SyncEngine` has no + `match kind` or `if kind ==` statements (the trait is the dispatch, not + the framework). +10. `a10_query_returns_connector_content` — sync a connector; query its content; + assert results include connector-sourced Level R nodes with correct source_uri. + +**Command:** `cargo test --test it_m7_gate` + +**False pass:** +- Testing with only one connector kind. The gate's value is proving two + *different* kinds work through the same framework. +- Testing rebuild parity without connector data in the log. An empty log + trivially rebuilds. +- Asserting update-rate "is still below 30%" instead of "is unchanged". Adding + reference docs should not move the number at all. + +## Traps + +- Running the gate before M7.6 (sync framework) is solid. The gate tests + composition; if the sync framework has bugs, every gate assertion fails + for the wrong reason. +- Not testing with a connector that produces multiple chunks per document. + Single-chunk documents hide change-detection bugs at the chunk level. + +--- + +Background: [DESIGN.md](../DESIGN.md) — source connectors, composition gates diff --git a/tasks/M7.2-obsidian-connector.md b/tasks/M7.2-obsidian-connector.md new file mode 100644 index 0000000..b2cdbac --- /dev/null +++ b/tasks/M7.2-obsidian-connector.md @@ -0,0 +1,97 @@ +# M7.2 — Obsidian vault connector + +| Field | Value | +|---|---| +| Phase | M7 — Source connectors | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M7.10 | +| Depends | M7.1, M3.6.1 | + +## Goal + +Wrap the existing `DocCorpusSource` (M3.6.1) in the `SourceConnector` interface +so the Obsidian vault is managed through the unified connector framework — with +change detection, config-driven setup, and registry integration. + +## Facts (inlined — no spec read needed) + +The Obsidian vault is the **primary reference source** for the cluster. It is +human-maintained, optionally git-backed, and contains runbooks, procedures, and +domain knowledge that agents query through the memory service. + +`DocCorpusSource` already handles markdown parsing, heading-boundary chunking, +breadcrumb paths, and file filtering. This task wraps it, not rewrites it. + +**Deployment models:** +1. **Git-sync sidecar** — a sidecar container clones the vault repo into a shared + PVC. Memory service reads from the PVC via this connector. +2. **Local mount** — for development, mount the vault directory directly. +3. **PVC direct** — vault files managed via kubectl cp or a web uploader. + +**Configuration:** +```yaml +connectors: + - kind: obsidian + name: homelab-vault + config: + root: /data/vault + extensions: [md, markdown, txt] + exclude_dirs: [.obsidian, .trash, .git] + max_file_size: 10485760 # 10MB +``` + +## Steps + +1. Implement `ObsidianConnector` in `mem-ingest/src/connectors/obsidian.rs`. +2. `list_documents()` — walk `root` directory, filter by extension, compute + sha256 per file, return `SourceDocument` per file. +3. `fetch_document()` — read file content, return as `DocumentContent` with + metadata (file path, last modified, size). +4. `health_check()` — verify `root` exists, is readable, count files. +5. Register `"obsidian"` kind in the connector registry factory. +6. `source_type()` returns `Reference` (vault docs bypass the gated loop). +7. Reuse `DocCorpusSource` internals for heading-based chunking when the sync + framework (M7.6) processes this connector's documents. + +## Acceptance + +- `ObsidianConnector` implements `SourceConnector` fully. +- `list_documents()` respects `extensions`, `exclude_dirs`, `max_file_size`. +- `fetch_document()` returns content matching file on disk. +- `health_check()` distinguishes readable vs. missing root directory. +- Config-driven: changing `root` path in YAML changes what gets scanned. + +## Verify + +**Harness:** fixture directory with markdown files, config YAML. + +**Integration test** — `tests/it_obsidian_connector.rs`: +1. `a1_list_filters_extensions` — fixture with .md, .txt, .json; assert only + .md and .txt are listed. +2. `a2_list_excludes_dirs` — fixture with `.obsidian/` subdir; assert its files + are excluded. +3. `a3_fetch_returns_content` — fetch a known doc; assert text matches file. +4. `a4_fetch_unknown_errors` — fetch non-existent doc_id; assert error. +5. `a5_health_check_reachable` — valid root; assert `reachable: true` with count. +6. `a6_health_check_missing_root` — non-existent root; assert `reachable: false`. +7. `a7_content_hash_stable` — fetch same file twice; assert same hash. +8. `a8_config_from_yaml` — parse connector from YAML; assert fields match. + +**Command:** `cargo test --test it_obsidian_connector` + +**False pass:** +- Testing with an empty directory. Assertions 1–3 need real files. +- Not testing `exclude_dirs` with nested paths (`.obsidian/plugins/x.md`). + +## Traps + +- Re-implementing markdown parsing instead of delegating to `DocCorpusSource`. +- Making `doc_id` platform-dependent (use relative path from root, unix separators). +- Ignoring symlinks — Obsidian uses them for multi-vault setups. + +--- + +Background: [DESIGN.md](../DESIGN.md) — source connectors, Obsidian vault as connector diff --git a/tasks/M7.3-paperless-connector.md b/tasks/M7.3-paperless-connector.md new file mode 100644 index 0000000..0129e2e --- /dev/null +++ b/tasks/M7.3-paperless-connector.md @@ -0,0 +1,119 @@ +# M7.3 — paperless-ngx connector + +| Field | Value | +|---|---| +| Phase | M7 — Source connectors | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M7.10 | +| Depends | M7.1 | + +## Goal + +Implement a `SourceConnector` for paperless-ngx so OCR'd documents, manuals, and +reference PDFs already stored in the cluster's paperless instance become +searchable through the memory service without manual export. + +## Facts (inlined — no spec read needed) + +paperless-ngx is already running in the cluster. Its REST API provides: + +- `GET /api/documents/` — paginated list with filtering by tags, document type, + correspondent, dates. +- `GET /api/documents/{id}/` — full metadata including `content` (extracted text). +- `GET /api/documents/{id}/download/` — original file. +- `GET /api/documents/{id}/preview/` — thumbnail. +- Authentication via `Authorization: Token ` header. +- Documents have `checksum` field (sha256 of original file). + +**Tag filtering is the selection mechanism.** Not every scanned receipt belongs +in the knowledge base. Config specifies which tags to include: +```yaml +connectors: + - kind: paperless + name: homelab-paperless + config: + base_url: http://paperless-ngx.paperless.svc.cluster.local:8000 + token_secret: paperless-api-token # k8s secret ref + tags: [reference, manual, runbook] # only sync these + format: text # use extracted text content + page_size: 100 # API pagination size +``` + +**Content comes as extracted text.** paperless-ngx OCRs documents on import and +stores the text in the `content` field. Use this directly — no PDF parsing needed +in the connector. The text quality depends on paperless's OCR config. + +**Checksum enables cheap change detection.** paperless provides `checksum` per +document. The sync framework (M7.6) compares this against the last-known hash +to skip unchanged documents. + +## Steps + +1. Implement `PaperlessConnector` in `mem-ingest/src/connectors/paperless.rs`. +2. `list_documents()` — paginate `GET /api/documents/?tags__name__in=...`, + extract `id`, `title`, `checksum`, `modified` for each. +3. `fetch_document()` — `GET /api/documents/{id}/`, extract `content` field, + return with metadata (title, tags, correspondent, date_created). +4. `health_check()` — `GET /api/` and verify 200; report document count from + `GET /api/documents/?tags__name__in=...&page=1&page_size=1` (read `count`). +5. Handle pagination (paperless returns `next` URL for subsequent pages). +6. Token auth from k8s secret (resolve `token_secret` to actual token value). +7. Register `"paperless"` kind in connector registry factory. +8. `source_type()` returns `Reference`. +9. Rate limit API calls (configurable, default 10 req/s) to avoid overloading + the paperless instance. + +## Acceptance + +- `PaperlessConnector` implements `SourceConnector` fully. +- Tag filtering limits which documents are listed. +- Pagination handles > 100 documents correctly. +- `content_hash` uses paperless's `checksum` field for change detection. +- Auth token resolved from k8s secret (not hardcoded). +- Health check reports document count matching tag filter. + +## Verify + +**Harness:** mock HTTP server (wiremock or similar) returning paperless API +responses; fixture JSON responses for list/detail endpoints. + +**Integration test** — `tests/it_paperless_connector.rs`: +1. `a1_list_filters_by_tags` — mock returns 5 docs, 3 with matching tags; assert + `list_documents()` returns 3. +2. `a2_pagination_fetches_all` — mock returns 2 pages of 50; assert 100 docs. +3. `a3_fetch_returns_content` — mock detail endpoint; assert text matches fixture. +4. `a4_fetch_includes_metadata` — assert returned metadata includes title, tags, + correspondent, date fields. +5. `a5_health_check_reachable` — mock 200; assert `reachable: true` with count. +6. `a6_health_check_unreachable` — mock connection refused; assert + `reachable: false` with error message. +7. `a7_checksum_as_content_hash` — assert `SourceDocument.content_hash` is + populated from paperless `checksum` field. +8. `a8_auth_header_sent` — assert mock received `Authorization: Token `. +9. `a9_config_from_yaml` — parse connector from YAML fixture; assert fields match. +10. `a10_live` — `#[ignore]`; real paperless instance; list + fetch one doc; print + title and content length for human sanity check. + +**Command:** `cargo test --test it_paperless_connector` (add `-- --ignored` for a10) + +**False pass:** +- Mocking without verifying auth header. A connector that works in tests but + sends no auth fails silently against real paperless. +- Testing single page only. Pagination bugs are invisible with < `page_size` docs. + +## Traps + +- Assuming `content` is always populated. paperless may have documents without + OCR text (e.g., empty scans). Return empty content with a warning, don't panic. +- Hardcoding the base URL without trailing-slash normalization. `/api/documents/` + vs `/api/documents` behaves differently. +- Not handling paperless API rate limits (429 responses). Add retry-after logic. +- Resolving k8s secrets at config parse time. Defer to runtime — secret may not + exist in dev/test environments. Use env var fallback. + +--- + +Background: [DESIGN.md](../DESIGN.md) — source connectors, paperless-ngx integration diff --git a/tasks/M7.4-git-repo-connector.md b/tasks/M7.4-git-repo-connector.md new file mode 100644 index 0000000..f73a730 --- /dev/null +++ b/tasks/M7.4-git-repo-connector.md @@ -0,0 +1,118 @@ +# M7.4 — Git repository connector + +| Field | Value | +|---|---| +| Phase | M7 — Source connectors | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M7.10 | +| Depends | M7.1 | + +## Goal + +Implement a `SourceConnector` that clones or pulls a git repository and exposes +its documents for ingestion — so documentation repos, wikis, and runbook +collections tracked in git become searchable through the memory service. + +## Facts (inlined — no spec read needed) + +Many knowledge sources are already in git: internal wikis, Forgejo repos, +infrastructure documentation, README collections. This connector pulls them +without requiring manual export. + +**Configuration:** +```yaml +connectors: + - kind: git_repo + name: infra-docs + config: + repo_url: https://forgejo.riotpiao.com/rock/homelab-docs.git + branch: main + clone_dir: /tmp/mem-connectors/infra-docs # local checkout + paths: [docs/, runbooks/] # only scan these dirs + extensions: [md, txt, rst] + auth_secret: forgejo-token # k8s secret for private repos + sync_depth: 1 # shallow clone +``` + +**Clone-then-walk model.** The connector clones (or pulls) the repo to a local +directory, then walks the filesystem like the Obsidian connector. This reuses +`DocCorpusSource` internals and avoids git-specific content access APIs. + +**Change detection uses git.** `git diff --name-only HEAD@{1}..HEAD` after a pull +tells the sync framework exactly which files changed — more efficient than +re-hashing every file. + +**Branch tracking.** Connector watches one branch. Branch changes (main → prod) +require config update. No multi-branch support — each branch is a separate +connector instance. + +## Steps + +1. Implement `GitRepoConnector` in `mem-ingest/src/connectors/git_repo.rs`. +2. On first sync: `git clone --depth N --branch B `. +3. On subsequent syncs: `git -C pull --ff-only`. +4. `list_documents()` — walk `clone_dir` filtered by `paths` and `extensions`, + compute sha256 per file. +5. `fetch_document()` — read file from `clone_dir`, return content with git + metadata (last commit sha, author, date for that file via `git log -1`). +6. `health_check()` — verify `clone_dir` is a valid git repo, check remote + connectivity via `git ls-remote`. +7. Handle auth for private repos (token from k8s secret → git credential helper + or URL embedding). +8. Register `"git_repo"` kind in connector registry factory. +9. `source_type()` returns `Reference`. + +## Acceptance + +- Clone + walk produces correct document list for fixture repo. +- Pull detects changed files without re-processing unchanged ones. +- Path filtering limits scan to configured subdirectories. +- Private repo auth works (token in URL or credential helper). +- Health check distinguishes: valid repo, invalid remote, auth failure. + +## Verify + +**Harness:** fixture git repo created in temp dir with known content. + +**Integration test** — `tests/it_git_repo_connector.rs`: +1. `a1_clone_and_list` — init fixture repo, connector clones it; assert + `list_documents()` returns expected files. +2. `a2_path_filtering` — fixture with `docs/` and `src/`; config says `paths: [docs/]`; + assert only `docs/` files returned. +3. `a3_extension_filtering` — fixture with .md, .rs, .txt; assert only .md/.txt. +4. `a4_fetch_returns_content` — fetch a doc; assert content matches fixture file. +5. `a5_pull_detects_changes` — add a commit to fixture repo; pull; assert changed + file appears in list with new hash. +6. `a6_health_check_valid_repo` — valid clone_dir; assert `reachable: true`. +7. `a7_health_check_no_clone` — no clone_dir; assert `reachable: false` with + message indicating clone needed. +8. `a8_shallow_clone` — assert clone depth matches config (`git rev-list --count HEAD`). +9. `a9_config_from_yaml` — parse connector from YAML; assert fields match. + +**Command:** `cargo test --test it_git_repo_connector` + +**False pass:** +- Testing with a local repo path instead of a clone. The clone/pull machinery + is the whole point — a connector that reads a pre-existing checkout is just + the Obsidian connector. +- Not testing pull-after-change. First sync always works; second sync is where + change detection matters. + +## Traps + +- Running `git clone` on every sync. Check if `clone_dir` already has a valid + checkout first; clone only on first run. +- Not cleaning up failed clones. A partial clone leaves a directory that is + neither valid nor absent — subsequent runs fail on both clone (dir exists) + and pull (not a repo). +- Force-push upstream breaks `--ff-only`. Detect non-fast-forward, delete + `clone_dir`, re-clone. Log a warning — this means all docs are re-processed. +- Symlinks across the repo boundary. `walkdir` follows symlinks by default; + a symlink to `/etc/passwd` is a real concern in a cluster-wide service. + +--- + +Background: [DESIGN.md](../DESIGN.md) — source connectors section diff --git a/tasks/M7.5-s3-connector.md b/tasks/M7.5-s3-connector.md new file mode 100644 index 0000000..e4ac264 --- /dev/null +++ b/tasks/M7.5-s3-connector.md @@ -0,0 +1,109 @@ +# M7.5 — S3-compatible storage connector + +| Field | Value | +|---|---| +| Phase | M7 — Source connectors | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M7.10 | +| Depends | M7.1 | + +## Goal + +Implement a `SourceConnector` for S3-compatible object storage (MinIO, AWS S3, +R2, etc.) so documents stored in buckets become searchable through the memory +service. + +## Facts (inlined — no spec read needed) + +S3 is the universal storage protocol. MinIO runs in many homelabs, and cloud +providers expose the same API. This connector makes any S3 bucket a knowledge +source. + +**Configuration:** +```yaml +connectors: + - kind: s3 + name: knowledge-bucket + config: + endpoint: https://minio.riotpiao.com + bucket: knowledge-base + prefix: docs/ # only this prefix + extensions: [md, txt, pdf] # filter by key suffix + access_key_secret: minio-creds # k8s secret with access/secret keys + region: us-east-1 # for AWS; ignored by MinIO +``` + +**ETag for change detection.** S3 objects have ETags (usually MD5 of content). +Use this as `content_hash` in `SourceDocument` — the sync framework skips objects +whose ETag hasn't changed. + +**Text extraction.** S3 stores raw files. Markdown and text files are read +directly. PDF/DOCX support is out of scope for M7.5 — those MIME types are +skipped with a warning. Future: add a text extraction layer or require pre-processed +text. + +**Pagination via continuation tokens.** S3 ListObjectsV2 returns max 1000 keys +per request. Use `ContinuationToken` for subsequent pages. + +## Steps + +1. Implement `S3Connector` in `mem-ingest/src/connectors/s3.rs`. +2. `list_documents()` — `ListObjectsV2` with `Prefix`, paginate, filter by + extension, return `SourceDocument` per object. +3. `fetch_document()` — `GetObject`, read body as text (UTF-8), return with + metadata (key, size, last_modified, ETag). +4. `health_check()` — `HeadBucket` to verify access. +5. Auth via access key + secret key from k8s secret. +6. Use `aws-sdk-s3` or `rust-s3` crate for S3 API. +7. Register `"s3"` kind in connector registry factory. +8. `source_type()` returns `Reference`. + +## Acceptance + +- `S3Connector` implements `SourceConnector` fully. +- Prefix filtering limits to configured path. +- Extension filtering skips non-text objects. +- ETag is used as `content_hash` for change detection. +- Pagination handles > 1000 objects. +- Auth works with MinIO and AWS-style credentials. + +## Verify + +**Harness:** mock S3 server (localstack or in-process mock) with fixture objects. + +**Integration test** — `tests/it_s3_connector.rs`: +1. `a1_list_with_prefix` — mock bucket with objects under `docs/` and `images/`; + assert only `docs/` objects listed. +2. `a2_extension_filtering` — mock with .md, .png, .txt; assert .png excluded. +3. `a3_fetch_returns_content` — fetch a .md object; assert content matches. +4. `a4_etag_as_content_hash` — assert `SourceDocument.content_hash` equals + the object's ETag. +5. `a5_pagination` — mock 1500 objects; assert all listed via continuation tokens. +6. `a6_health_check_valid_bucket` — mock HeadBucket 200; assert reachable. +7. `a7_health_check_no_access` — mock HeadBucket 403; assert not reachable with + error message. +8. `a8_non_utf8_skipped` — mock object with binary content; assert skipped with + warning, not crash. +9. `a9_config_from_yaml` — parse connector from YAML; assert fields match. + +**Command:** `cargo test --test it_s3_connector` + +**False pass:** +- Testing with a local filesystem mock instead of S3 API mock. The pagination + and ETag handling are S3-specific. + +## Traps + +- Assuming ETags are always MD5. Multipart uploads produce composite ETags + (`hash-N`). These are still unique per version — use as-is for change detection. +- Not handling `NoSuchBucket` vs `AccessDenied`. Both are errors but mean + different things for health reporting. +- Reading binary files as UTF-8. A JPEG read as text produces garbage. Check + content-type header and skip non-text MIME types. + +--- + +Background: [DESIGN.md](../DESIGN.md) — source connectors section diff --git a/tasks/M7.6-sync-framework.md b/tasks/M7.6-sync-framework.md new file mode 100644 index 0000000..3a8f3d3 --- /dev/null +++ b/tasks/M7.6-sync-framework.md @@ -0,0 +1,146 @@ +# M7.6 — Sync framework + +| Field | Value | +|---|---| +| Phase | M7 — Source connectors | +| Size | L — 3–5 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M7.7, M7.10 | +| Depends | M7.1, M3.6.3 | + +## Goal + +Build the shared sync engine that handles change detection, tombstoning, drift +reporting, and resumable sync for **all** connectors — so individual connector +implementations only fetch documents and everything else is handled once. + +## Facts (inlined — no spec read needed) + +Every connector faces the same sync problems: +1. **What changed?** Compare content hashes from `list_documents()` against a + manifest of last-known hashes. Only fetch and process changed documents. +2. **What disappeared?** Documents in the manifest but not in `list_documents()` + need tombstone records in the log. Append-only — never delete. +3. **What if sync crashes midway?** Track progress per-document. Restart + processes only unseen documents. +4. **How much will this cost?** Drift report shows counts without mutating + anything. + +This is the generalization of M3.6.3's `mem ref sync/list/rm` logic, applied to +any `SourceConnector` instead of just `DocCorpusSource`. + +**Manifest storage.** Per-connector manifest in `log/connectors/.manifest.jsonl`: +```jsonl +{"doc_id":"abc","source_uri":"file:///vault/k8s.md","content_hash":"sha256:...","chunk_count":12,"synced_at":"..."} +{"doc_id":"def","source_uri":"paperless://doc/42","content_hash":"sha256:...","chunk_count":3,"synced_at":"..."} +``` + +**Sync algorithm:** +``` +current = connector.list_documents() +previous = load_manifest(connector.name) + +for doc in current: + if doc.content_hash == previous[doc.doc_id].content_hash: + skip (unchanged) + else if doc.doc_id in previous: + tombstone previous chunks, fetch + chunk + embed new (changed) + else: + fetch + chunk + embed (new) + +for doc_id in previous not in current: + tombstone all chunks (removed) + +save_manifest(connector.name, current) +``` + +**Chunking delegation.** The sync framework owns the chunking step. It routes +fetched `DocumentContent` through the appropriate `ChunkPolicy`: +- Markdown → heading-boundary chunking (reuse `DocCorpusSource` logic) +- Plain text → paragraph-boundary chunking +- Configurable per connector kind in `connectors.yaml` + +**Level routing.** Session connectors → RecordSource → gated loop (L0/L1/L2). +Document connectors → Reference records (Level R). The `source_type()` method +on the connector determines the pipeline. + +## Steps + +1. Define `SyncEngine` struct in `mem-ingest/src/sync.rs`. +2. Implement manifest loading/saving (JSONL per connector). +3. Implement diff algorithm: `(new, changed, unchanged, removed)` from + `list_documents()` vs manifest. +4. Implement sync loop: for each `new`/`changed` doc, fetch → chunk → emit + records. For each `removed` doc, emit tombstones. +5. Implement drift reporting: same diff algorithm, print counts, no mutations. +6. Implement resume: track synced doc_ids in a progress file. On restart, + skip already-synced docs. +7. Implement rate limiting: configurable max concurrent fetches per connector. +8. Integrate with `mem-store` for writing Reference records to the log. +9. Integrate with `mem-llm` for embedding new chunks. + +## Acceptance + +- Sync of unchanged connector produces zero embedding calls. +- Changed document: old chunks tombstoned, new chunks embedded and stored. +- Removed document: chunks tombstoned, manifest updated. +- Drift report matches actual changes without mutating anything. +- Crash mid-sync → restart processes only remaining documents. +- Rate limiting: never exceeds configured concurrent fetch limit. +- Log remains append-only (tombstones are records, not deletions). + +## Verify + +**Harness:** `VecConnector` with mutable document list, counting embedder. + +**Integration test** — `tests/it_sync_framework.rs`: +1. `a1_initial_sync_all_new` — 3 docs, no manifest; assert all 3 fetched and + embedded, manifest written with 3 entries. +2. `a2_unchanged_skipped` — sync again with same content; assert zero fetch + calls, zero embed calls. +3. `a3_changed_doc_replaced` — modify one doc's content; sync; assert old chunks + tombstoned, new chunks embedded, embed count equals changed doc's chunk count. +4. `a4_removed_doc_tombstoned` — remove a doc from connector; sync; assert + tombstone records emitted, manifest entry removed. +5. `a5_new_doc_added` — add a doc to connector; sync; assert only new doc + fetched and embedded. +6. `a6_drift_report_read_only` — modify docs, run drift report; assert correct + counts (1 new, 1 changed, 1 unchanged, 1 removed); assert no mutations to + manifest or log. +7. `a7_resume_after_crash` — sync 5 docs, simulate crash after 3; restart; + assert only 2 remaining docs processed. +8. `a8_tombstone_is_append` — count log lines before and after remove; assert + count only grew. +9. `a9_manifest_roundtrip` — save manifest, load it; assert field-for-field + equality. +10. `a10_rebuild_parity` — after full sync, `mem rebuild --from-log` produces + identical state. + +**Command:** `cargo test --test it_sync_framework` + +**False pass:** +- Asserting "no duplicate rows" instead of counting embed calls. A sync that + re-embeds everything and upserts by sha produces correct rows and wasted + compute. +- Testing drift report without actually changing documents first. + +## Traps + +- Comparing document-level hashes instead of chunk-level. A doc that changed + one paragraph should re-embed only the affected chunks, not all of them. + However, heading-boundary chunking means changing one heading can shift all + subsequent chunks. Accept document-level granularity for now; chunk-level + optimization is a future refinement. +- Making the manifest a database table instead of JSONL. The manifest must + survive `mem rebuild --from-log` — it is metadata about the sync process, + not a projection of the log. +- Running fetch + embed serially. A connector with 500 docs at 200ms per embed + takes 100s serially. Concurrent fetch + sequential embed is the right shape. +- Ignoring the derived filter (M4.2). Document connectors produce Level R content + that must register in the artifact manifest so it cannot re-enter as evidence. + +--- + +Background: [DESIGN.md](../DESIGN.md) — source connectors, sync framework diff --git a/tasks/M7.7-source-cli.md b/tasks/M7.7-source-cli.md new file mode 100644 index 0000000..2c7ae2b --- /dev/null +++ b/tasks/M7.7-source-cli.md @@ -0,0 +1,107 @@ +# M7.7 — `mem source` CLI + +| Field | Value | +|---|---| +| Phase | M7 — Source connectors | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M7.8, M7.10 | +| Depends | M7.6 | + +## Goal + +Provide CLI commands to manage source connectors: sync documents, check status, +list connectors, add/remove connectors. + +## Facts (inlined — no spec read needed) + +``` +mem source list # all registered connectors + health +mem source status # drift report per connector, no mutations +mem source sync --all # sync all connectors +mem source sync --name homelab-vault # sync one connector +mem source sync --name homelab-vault --dry-run # show plan, no mutations +mem source add --kind paperless --name docs --config '{"base_url":"...","token":"..."}' +mem source rm --name old-source # deregister + tombstone all chunks +mem source health # connectivity check per connector +``` + +**Output format.** CLI outputs human-readable tables by default, `--json` for +machine consumption. Example: + +``` +$ mem source list +NAME KIND DOCS LAST SYNC HEALTH +homelab-vault obsidian 42 2026-08-26T10:00:00Z ✅ reachable +homelab-paperless paperless 127 2026-08-26T09:00:00Z ✅ reachable +infra-docs git_repo 18 2026-08-25T20:00:00Z ⚠️ pull failed + +$ mem source status +NAME NEW CHANGED UNCHANGED REMOVED +homelab-vault 0 2 40 0 +homelab-paperless 3 0 124 0 +infra-docs 1 1 16 0 +``` + +## Steps + +1. Add `Source` subcommand group to clap CLI in `mem-cli/src/main.rs`. +2. Implement `cmd_source_list()` — load registry, health check each, tabulate. +3. Implement `cmd_source_status()` — load registry, run drift report per + connector, tabulate. +4. Implement `cmd_source_sync()` — load registry, run sync engine for selected + connector(s), report results. +5. Implement `cmd_source_add()` — validate kind, parse config, register in + `connectors.yaml`, run initial health check. +6. Implement `cmd_source_rm()` — tombstone all chunks from connector, remove + from `connectors.yaml`. +7. Implement `cmd_source_health()` — connectivity check per connector. +8. Add `--dry-run` to sync (show plan only). +9. Add `--json` flag for machine-readable output. + +## Acceptance + +- All subcommands execute without panic. +- `sync --dry-run` shows plan without mutations. +- `add` validates kind exists in registry before writing config. +- `rm` tombstones chunks and removes config entry. +- `status` shows drift without mutations. +- Exit codes: 0 on success, non-zero on failure. + +## Verify + +**Harness:** `VecConnector` registered in registry, temp connectors.yaml. + +**Integration test** — `tests/it_source_cli.rs`: +1. `a1_list_shows_connectors` — register two connectors; assert list output + contains both names and kinds. +2. `a2_status_shows_drift` — modify connector docs; assert status shows correct + new/changed/unchanged/removed counts. +3. `a3_sync_processes_changes` — sync with changes; assert documents processed. +4. `a4_sync_dry_run_no_mutations` — sync --dry-run; assert no log entries written. +5. `a5_add_registers_connector` — add a new connector; assert it appears in list. +6. `a6_add_bad_kind_fails` — add with unknown kind; assert non-zero exit. +7. `a7_rm_tombstones_and_deregisters` — rm a connector; assert tombstone records + written and connector removed from list. +8. `a8_health_reports_status` — assert health output includes reachable/unreachable. +9. `a9_json_output` — assert --json flag produces valid JSON. + +**Command:** `cargo test --test it_source_cli` + +**False pass:** +- Testing `list` without checking that connectors are actually registered (not + just config parsed). + +## Traps + +- `sync --all` with a broken connector should not halt all syncs. Sync each + independently, report failures per connector at the end. +- `rm` without `--yes` should prompt for confirmation (destructive operation). +- Don't write `connectors.yaml` atomically — a crash mid-write corrupts config. + Write to temp file, then rename. + +--- + +Background: [DESIGN.md](../DESIGN.md) — source connectors, CLI commands diff --git a/tasks/M7.8-source-http-endpoints.md b/tasks/M7.8-source-http-endpoints.md new file mode 100644 index 0000000..b8663a8 --- /dev/null +++ b/tasks/M7.8-source-http-endpoints.md @@ -0,0 +1,107 @@ +# M7.8 — `GET /memory/sources` + `POST /memory/sources/sync` HTTP endpoints + +| Field | Value | +|---|---| +| Phase | M7 — Source connectors | +| Size | M — 1–3 days | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M7.10 | +| Depends | M7.7, M3.5.1 | + +## Goal + +Expose source connector management through the HTTP API so agents and UIs can +trigger syncs, check connector health, and view status without CLI access. + +## Facts (inlined — no spec read needed) + +**Endpoints:** +``` +GET /memory/sources # list all connectors + health +GET /memory/sources/{name} # one connector detail + drift +GET /memory/sources/{name}/health # health check only +POST /memory/sources/sync # trigger sync (async) +POST /memory/sources/{name}/sync # trigger sync for one connector +GET /memory/sources/sync/{job_id} # sync job status +``` + +**Response shapes:** +```json +// GET /memory/sources +[ + { + "name": "homelab-vault", + "kind": "obsidian", + "document_count": 42, + "last_sync": "2026-08-26T10:00:00Z", + "health": { "reachable": true, "document_count": 42 } + } +] + +// POST /memory/sources/sync +// Request: { "names": ["homelab-vault"] } (or omit for all) +// Response: 202 Accepted +{ "job_id": "sync-abc123", "status_url": "/memory/sources/sync/sync-abc123" } + +// GET /memory/sources/sync/{job_id} +{ + "job_id": "sync-abc123", + "status": "completed", + "connectors": { + "homelab-vault": { "new": 0, "changed": 2, "unchanged": 40, "removed": 0 } + } +} +``` + +**Sync is async.** POST returns 202 immediately; client polls status endpoint. +Same pattern as `/memory/ingest`. + +## Steps + +1. Add routes to `http_server.rs` for all six endpoints. +2. `sources_list_handler` — load registry, health check each, return JSON array. +3. `source_detail_handler` — one connector, include drift report. +4. `source_health_handler` — health check only. +5. `source_sync_handler` — validate connector names, spawn async sync job, + return job_id + status URL. +6. `source_sync_status_handler` — look up job by id, return progress. +7. Auth: all endpoints require apikey header (existing pattern). +8. Rate limiting: sync endpoint limited to 10 req/hour (it's expensive). + +## Acceptance + +- All endpoints return correct status codes and JSON shapes. +- Sync is async — POST returns immediately, job runs in background. +- Unknown connector name returns 404. +- Auth required on all endpoints. +- Rate limiting on sync endpoint. + +## Verify + +**Integration test** — `tests/it_source_http.rs`: +1. `a1_list_returns_connectors` — register connectors, GET /memory/sources; + assert JSON array with correct fields. +2. `a2_detail_includes_drift` — GET /memory/sources/{name}; assert drift fields. +3. `a3_health_check_via_http` — GET /memory/sources/{name}/health; assert + reachable field. +4. `a4_sync_returns_202` — POST /memory/sources/sync; assert 202 + job_id. +5. `a5_sync_status_tracks_progress` — poll status endpoint; assert eventually + "completed". +6. `a6_unknown_connector_404` — GET /memory/sources/nonexistent; assert 404. +7. `a7_auth_required` — request without apikey; assert 401. +8. `a8_sync_rate_limited` — 11 sync requests; assert 429 on the 11th. + +**Command:** `cargo test --test it_source_http` + +## Traps + +- Making sync synchronous. A paperless connector with 500 docs takes minutes; + blocking the HTTP response is a client timeout. +- Not capping concurrent sync jobs. Two simultaneous syncs to the same connector + can corrupt the manifest. + +--- + +Background: [DESIGN.md](../DESIGN.md) — source connectors, HTTP endpoints diff --git a/tasks/M7.9-connector-health-monitoring.md b/tasks/M7.9-connector-health-monitoring.md new file mode 100644 index 0000000..3123ae7 --- /dev/null +++ b/tasks/M7.9-connector-health-monitoring.md @@ -0,0 +1,76 @@ +# M7.9 — Connector health monitoring + observability + +| Field | Value | +|---|---| +| Phase | M7 — Source connectors | +| Size | S — < 1 day | +| Status | ⬜ Not started | +| Flags | — | +| Spec | inlined below | +| Blocks | M7.10 | +| Depends | M7.8 | + +## Goal + +Add periodic health checks, sync metrics, and alerting hooks for source +connectors so connector failures are detected before knowledge goes stale. + +## Facts (inlined — no spec read needed) + +A connector that silently fails means the knowledge base is stale with no +indication. The monitoring layer detects this. + +**Metrics (Prometheus-compatible):** +``` +mem_source_health{name="homelab-vault",kind="obsidian"} 1 # 1=healthy, 0=unhealthy +mem_source_last_sync_seconds{name="homelab-vault"} 1724680000 # unix timestamp +mem_source_documents_total{name="homelab-vault"} 42 +mem_source_sync_duration_seconds{name="homelab-vault"} 12.3 +mem_source_sync_errors_total{name="homelab-vault"} 0 +mem_source_drift_new{name="homelab-vault"} 2 # docs pending sync +mem_source_drift_changed{name="homelab-vault"} 1 +``` + +**Periodic health check.** Configurable interval (default 5 minutes). Log +warnings for unreachable connectors. Update Prometheus gauges. + +**Staleness alert.** If `last_sync` exceeds a configurable threshold (default 24h), +log a warning and set a metric. This is the "knowledge is going stale" signal. + +## Steps + +1. Add health check background task (tokio interval, configurable period). +2. Export Prometheus metrics via `/metrics` endpoint (existing pattern or new). +3. Track per-connector: health, last sync, doc count, sync duration, errors. +4. Staleness detection: compare `last_sync` to now, warn if exceeds threshold. +5. Log structured health events for observability. + +## Acceptance + +- Periodic health checks run at configured interval. +- Metrics endpoint returns valid Prometheus format. +- Staleness warning fires when last_sync exceeds threshold. +- Unhealthy connector logged with error details. + +## Verify + +**Integration test** — `tests/it_source_monitoring.rs`: +1. `a1_health_metric_updated` — register connector, wait for health check; + assert metric value matches connector health. +2. `a2_staleness_detected` — set last_sync to 25h ago; assert staleness warning. +3. `a3_metrics_format_valid` — GET /metrics; assert valid Prometheus text format. +4. `a4_error_count_incremented` — force connector health check failure; assert + error counter incremented. + +**Command:** `cargo test --test it_source_monitoring` + +## Traps + +- Running health checks synchronously. A connector that times out blocks all + other connectors' health checks. Use tokio::select with timeout. +- Not distinguishing "unreachable" from "empty". A connector that returns 0 docs + is healthy; a connector that can't connect is not. + +--- + +Background: [DESIGN.md](../DESIGN.md) — source connectors, observability