From 82cc2c83102cf952cc927e30a4e3281794ab4e84 Mon Sep 17 00:00:00 2001 From: Story Crater Bot <19826264+Riotpiaole@users.noreply.github.com> Date: Wed, 26 Aug 2026 16:56:39 -0700 Subject: [PATCH] docs: Add M7 source connectors (10 tasks), M3.5.10 auth integration, remove Kong refs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - M7.1-M7.10: Extensible SourceConnector trait, Obsidian/paperless/git/S3 connectors, sync framework, CLI, HTTP endpoints, health monitoring, gate - M3.5.10: Auth integration with Authentik OIDC → Vault token validation - DESIGN.md: Add source connectors architecture, update auth to Authentik/Vault (Kong removed from cluster) - INDEX.md: 75 tasks, 11 gates - Fix all Kong references in M3.5.1 task --- DESIGN.md | 260 ++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 254 insertions(+), 6 deletions(-) diff --git a/DESIGN.md b/DESIGN.md index 816938c..0c9015b 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -32,7 +32,9 @@ The mechanism is GRU-Mem (arXiv 2602.10560). Its two gates map directly onto the | CNPG operator | **1.30.0**, supports declarative `Database.spec.extensions` | `kubectl explain database.spec.extensions` | | Ollama context cap | **32768** (`OLLAMA_CONTEXT_LENGTH`) | `k8s/apps/llm-serving/ornith.yaml` | | Controller model | `qwen2.5:3b-instruct` | is Qwen2.5-3B-Instruct — **the paper's exact 3B backbone** | -| Gateway auth | `apikey:` header, **not** `Authorization: Bearer` | Kong key-auth compares whole header value | +| Gateway | nginx ingress + homelab-frontend (Go, ServiceAdapter CRDs) | Kong is **removed** from cluster | +| IAM | Authentik (OIDC) → Vault (OIDC auth method) | Authentik issues JWT, Vault validates via OIDC, `homelab-admin` role bound on `permissions: "*"` claim | +| Auth (placeholder) | `apikey:` header in mem-cli | **To be replaced** — does not integrate with Authentik/Vault chain | The 32K cap is the binding constraint and it fits: paper uses 5000-token chunks, 8192 max prompt, 2048 max response. @@ -57,7 +59,7 @@ Memory is **levelled**, and every event in the log carries its level. The paper ## Architecture ``` - api.riotpiao.com (Kong) + api.riotpiao.com (nginx ingress + homelab-frontend gateway) │ ┌─────────────────┼─────────────────┐ │ │ │ @@ -400,6 +402,211 @@ Two mechanical guards: 1. **Promotion is a human move** out of `_drafts/`, reviewable as a diff. 2. **Provenance marks derived text.** Every emitted skill carries `generated_from: ` in frontmatter, and `mem-ingest` tags chunks matching a known emitted artifact as `derived: true` and excludes them from evidence. Without this the corpus slowly becomes its own training data. +## Source connectors — extensible multi-source ingestion + +Memory service is a **cluster-wide RAG** serving multiple agents and workflows. Knowledge does not live in one place — it is spread across an Obsidian vault, a paperless-ngx instance, agent session logs, and whatever document stores appear next. The connector architecture makes adding a new source a single-trait implementation rather than a pipeline rewrite. + +### The problem with hardcoded sources + +Today `mem-ingest` has three concrete sources: `PiSessionSource`, `ClaudeTranscriptSource`, `DocCorpusSource`. Each knows its own file format and emits `Record`s through the `RecordSource` trait. This works, but: + +1. **Every new source requires Rust code.** paperless-ngx has a REST API; Confluence has another; S3 is a third protocol. Each lands as a new `.rs` file compiled into the binary. +2. **No runtime discovery.** The CLI must know about every source at compile time. A homelab that adds Bookstack next month needs a code change, a rebuild, and a redeploy. +3. **No shared sync logic.** Change detection (sha-based skip), tombstoning, drift reporting — each source will reimplement these unless the framework provides them. + +### Design: `SourceConnector` trait + registry + +A connector is anything that can enumerate documents and yield their content. The framework handles chunking, embedding, change detection, and storage. + +```rust +/// A connector to an external document source. +#[async_trait] +pub trait SourceConnector: Send + Sync { + /// Unique connector kind identifier (e.g., "obsidian", "paperless", "s3") + fn kind(&self) -> &str; + + /// Human-readable name for this connector instance + fn name(&self) -> &str; + + /// Enumerate all documents available from this source. + /// Returns (doc_id, doc_metadata) pairs. + async fn list_documents(&self) -> Result>; + + /// Fetch content for a single document by its source-specific ID. + async fn fetch_document(&self, doc_id: &str) -> Result; + + /// Health check — can we reach this source? + async fn health_check(&self) -> Result; +} + +/// Metadata about a source document (before fetching content) +pub struct SourceDocument { + pub doc_id: String, // source-specific unique ID + pub title: String, // human-readable title + pub source_uri: String, // canonical URI (file://, https://, paperless://) + pub content_hash: Option, // if source provides hash, skip fetch on match + pub mime_type: String, // text/markdown, application/pdf, etc. + pub updated_at: Option, +} + +/// Fetched document content ready for chunking +pub struct DocumentContent { + pub doc_id: String, + pub text: String, // extracted text (markdown preferred) + pub source_uri: String, + pub content_hash: String, // sha256 of text content + pub metadata: HashMap, // source-specific metadata +} + +/// Source health status +pub struct SourceHealth { + pub reachable: bool, + pub document_count: Option, + pub last_error: Option, +} +``` + +### Connector registry + +Connectors are registered at startup from a YAML config file. Adding a new source is: implement the trait, register the kind, add a config block. + +```yaml +# connectors.yaml — source connector configuration +connectors: + - kind: obsidian + name: homelab-vault + config: + root: /data/vault # local path (PVC mount or git-sync) + extensions: [md, markdown, txt] + exclude_dirs: [.obsidian, .trash] + + - kind: paperless + name: homelab-paperless + config: + base_url: http://paperless-ngx.paperless.svc.cluster.local:8000 + token_secret: paperless-api-token # k8s secret name + tags: [reference, manual, runbook] # only sync docs with these tags + format: markdown # request markdown export + + - kind: git_repo + name: infra-docs + config: + repo_url: https://forgejo.riotpiao.com/rock/homelab-docs.git + branch: main + sync_interval: 1h + paths: [docs/, runbooks/] + + - kind: s3 + name: backup-docs + config: + endpoint: https://minio.riotpiao.com + bucket: knowledge-base + prefix: docs/ + access_key_secret: minio-credentials +``` + +### Sync framework + +The connector trait provides enumeration and fetch. The **sync framework** handles everything else: + +1. **Change detection:** Compare `content_hash` from `list_documents()` against last-known hash in the manifest. Skip unchanged docs (zero embed calls). +2. **Chunking:** Route through existing `ChunkPolicy` — heading-based for markdown, paragraph-based for plain text, configurable per connector. +3. **Tombstoning:** Documents that vanish from `list_documents()` get tombstone records in the log. Append-only, never delete. +4. **Drift reporting:** `mem source status` shows per-connector: docs total, changed, new, removed — without mutating anything. +5. **Rate limiting:** Configurable fetch concurrency per connector (don't DDoS paperless with 500 parallel fetches). +6. **Resume:** Sync is resumable. Crash mid-sync, restart, only unseen docs are processed. + +``` +mem source sync --all # sync all registered connectors +mem source sync --name homelab-vault # sync one connector +mem source status # drift report, no mutations +mem source list # registered connectors + health +mem source add --kind paperless ... # register new connector +mem source rm --name old-source # deregister + tombstone +``` + +### Built-in connectors (shipped with binary) + +| Kind | Source | Protocol | Status | +|------|--------|----------|--------| +| `obsidian` | Local Obsidian vault | filesystem (walkdir) | M7.2 | +| `paperless` | paperless-ngx | REST API (`/api/documents/`) | M7.3 | +| `git_repo` | Git repository | git clone/pull | M7.4 | +| `s3` | S3-compatible storage | S3 API (minio, AWS) | M7.5 | +| `pi_session` | Pi agent sessions | filesystem (existing) | M7.1 (wrap existing) | +| `claude` | Claude transcripts | filesystem (existing) | M7.1 (wrap existing) | + +### How connectors interact with levels + +**Session connectors** (pi, claude) produce evidence that flows through the gated loop → L0/L1/L2. These are the existing `RecordSource` implementations, wrapped in `SourceConnector` for unified management. + +**Document connectors** (obsidian, paperless, git, s3) produce reference material → Level R. These bypass the gated loop entirely (no standing question, no gate decision). This is the existing M3.6 design, now generalized. + +The connector kind determines the pipeline: +``` +Session connector → RecordSource → ChunkPolicy → GatedLoop → L0/L1/L2 +Document connector → SourceConnector → ChunkPolicy → Level R (no gate) +``` + +### paperless-ngx integration (concrete example) + +paperless-ngx is already running in the cluster. It has: +- REST API at `http://paperless-ngx.paperless.svc.cluster.local:8000/api/` +- Documents with tags, correspondents, document types +- Full-text content available via API +- Thumbnail and original file access + +```rust +pub struct PaperlessConnector { + base_url: String, + token: String, + tag_filter: Vec, +} + +#[async_trait] +impl SourceConnector for PaperlessConnector { + fn kind(&self) -> &str { "paperless" } + fn name(&self) -> &str { &self.name } + + async fn list_documents(&self) -> Result> { + // GET /api/documents/?tags__name__in=reference,manual + // Paginate through results + // Return doc_id, title, checksum (paperless provides this) + } + + async fn fetch_document(&self, doc_id: &str) -> Result { + // GET /api/documents/{id}/ + // Extract content field (full text) + // Or GET /api/documents/{id}/download/ for original + } + + async fn health_check(&self) -> Result { + // GET /api/ — check 200 + } +} +``` + +### Obsidian vault as a connector + +The Obsidian vault is the **primary reference source**. It is human-maintained, git-backed, and the canonical location for runbooks, procedures, and domain knowledge that agents need. + +Deployment options: +1. **Git-sync sidecar:** A sidecar container clones the vault repo into a shared PVC. Memory service reads from the PVC. +2. **Local mount:** For development, mount the vault directory directly. +3. **API upload:** Push vault changes to memory service via HTTP. + +The Obsidian connector wraps the existing `DocCorpusSource` (M3.6.1) with the `SourceConnector` interface, adding change detection and registry management. + +### Future extensibility + +Adding a new source requires: +1. Implement `SourceConnector` trait (~100-200 lines) +2. Register the `kind` in the connector factory +3. Add config block to `connectors.yaml` +4. Run `mem source sync --name new-source` + +No changes to the ingest pipeline, chunking, embedding, storage, or query layers. The connector is the only new code. + ## Reference corpora — the non-evidential tier `RecordSource` takes session transcripts, and the update gate asks "does this chunk contain evidence for Q". A `kubectl` or `tea` cheatsheet answers neither question: it has no session, no turn, and no evidence. Left alone the system faithfully retains *what happened when a model used a tool badly* and never learns the tool. Level **R** closes that gap, and the shape of the fix matters more than the fact of it. @@ -509,6 +716,8 @@ A response that cannot fit its tier-1 hits inside the budget is an error, not a **P6 — Post-training (separate, Python).** Boundary is the JSONL. `mem label` uses the 32B `reasoning` model as an offline evidence labeler to produce per-chunk `U_t` ground truth (the paper had synthetic NIAH labels; we do not, and this is the honest cheapest substitute). Then verl trains a LoRA with the paper's rewards: `r_update` ±1, `r_exit` {0, −0.5 late, −0.75 early}, strict `r_format`, `α=0.9` mixing trajectory- and turn-level advantage. Requires the vLLM decision above. +**P7 — Source connectors (board `M7`).** Generalizes the ingestion layer from hardcoded file sources to an extensible `SourceConnector` trait with a YAML-driven registry. Existing sources (pi sessions, Claude transcripts, `DocCorpusSource`) are wrapped in the new interface; new sources (paperless-ngx, git repos, S3) implement the trait directly. The sync framework handles change detection, tombstoning, drift reporting, and resumable sync for all connectors. The connector is the only new code when a source is added — no changes to chunking, embedding, storage, or query layers. Ordered after M3.6 because it generalizes `mem ref` rather than duplicating it, and after M4.2 because document connectors must register in the derived-content manifest to prevent cycle contamination. + ## Task breakdown Board lives in **`memory-tasks/`** at the repo root. Format follows `agent-rust/tasks/`: one file per task, self-contained, each with `Acceptance` / `Verify` (harness, numbered assertions, command) / `False pass` / `Traps`, a `Status` field as source of truth, and `memory-tasks/INDEX.md` mirroring it. Ids are `M.` and frozen once written — phase order is declared in `INDEX.md`, never derived from the id. @@ -584,7 +793,7 @@ Note `agent-rust/.gitignore` excludes `tasks`, which silently untracks the whole | id | task | size | deps | |---|---|---|---| -| M3.5.1 | HTTP server + router (actix-web or axum), Kong auth hook, request metrics | M | M0.1 | +| M3.5.1 | HTTP server + router (actix-web or axum), auth hook, request metrics | M | M0.1 | | M3.5.2 | `POST /ingest` endpoint — `ingest_id` dedup, async queue, git context enrichment | M | M1.7, M3.5.1 | | M3.5.3 | `GET /query` endpoint — embed query, HNSW recall by level, rerank, walk edges to L0 | M | M3.3, M3.5.1 | | M3.5.4 | Federation: single query across projects, fan+merge results, deduplicate | M | M3.5.3 | @@ -605,7 +814,22 @@ Note `agent-rust/.gitignore` excludes `tasks`, which silently untracks the whole | M5.5 | verl loop — `r_update` ±1, `r_exit` {0,−0.5,−0.75}, strict `r_format`, α=0.9 | L | M5.3, M5.4 | | M5.6 | **M5 gate** — adapter beats prompted baseline on held-out update accuracy | L | gate | -Total 52 tasks, 8 gates. M0 and M2.2 have no model dependency and can start immediately; M5.4 is homelab work independent of everything else in M5 and can run in parallel. M3.5 depends on M2 (pgvector store exists) and M1 (ingest loop exists); can run in parallel with M4 and M5. M3.5.9 (git-aware references) is optional, depends on M3.5.2. +**M7 — Source connectors** (extensible multi-source ingestion) + +| id | task | size | deps | +|---|---|---|---| +| M7.1 | `SourceConnector` trait + `SourceDocument` / `DocumentContent` types + connector registry | M | M0.3, M3.6.1 | +| M7.2 | Obsidian vault connector — wraps `DocCorpusSource`, adds change detection, config-driven | M | M7.1, M3.6.1 | +| M7.3 | paperless-ngx connector — REST API client, tag filtering, markdown export | M | M7.1 | +| M7.4 | Git repo connector — clone/pull, path filtering, branch tracking | M | M7.1 | +| M7.5 | S3 connector — S3-compatible API, prefix filtering, content-type routing | M | M7.1 | +| M7.6 | Sync framework — change detection, tombstoning, drift report, resume | L | M7.1, M3.6.3 | +| M7.7 | `mem source` CLI — sync/status/list/add/rm subcommands | M | M7.6 | +| M7.8 | `GET /memory/sources` + `POST /memory/sources/sync` HTTP endpoints | M | M7.7, M3.5.1 | +| M7.9 | Connector health monitoring + observability | S | M7.8 | +| M7.10 | **M7 composition gate** — two connectors sync, change detection skips unchanged, tombstone works, rebuild parity | M | gate | + +Total 74 tasks, 11 gates. M0 and M2.2 have no model dependency and can start immediately; M5.4 is homelab work independent of everything else in M5 and can run in parallel. M3.5 depends on M2 (pgvector store exists) and M1 (ingest loop exists); can run in parallel with M4 and M5. M3.5.9 (git-aware references) is optional, depends on M3.5.2. M7 depends on M3.6 (reference corpora) and M4.2 (derived filter) for cycle prevention. ## Verification @@ -677,7 +901,9 @@ The decisive P2 metric is **update-rate**, the one number distinguishing a worki ## Distributed API Layer (Homelab Frontend) -**Gateway:** `api.riotpiao.com` routes agent and system memory requests through Kong. +**Gateway:** `api.riotpiao.com` routes requests through **nginx ingress** to **homelab-frontend** gateway (Go, ServiceAdapter CRDs). Kong is **removed** from the cluster. + +**Auth:** Authentik (OIDC provider) → Vault (OIDC auth method, `auth/oidc/role/homelab-admin`, bound on `permissions: "*"` claim) → Vault token → service validates token. Memory service currently uses placeholder `apikey` header — M3.5.10 replaces this with Vault token validation. **Architecture assumption:** Memory services run in CNPG cluster; API layer is HTTP facade exposing read/write workflows to distributed agents. Authority remains JSONL—API is a request demultiplexer, not a cache or alternative source of truth. @@ -748,7 +974,29 @@ POST /memory/context <- 3-tier lookup: signature, symptom, doc - Client-side: `ETag: ` on all read endpoints, no conditional logic server-side (it's stateless) **Auth & rate limits:** -- Kong `apikey:` header (existing pattern) + +*Auth chain (production):* +``` +User/Agent → Authentik (OIDC login) + → JWT with claims (including `permissions`) + → Vault validates via OIDC auth method + → Vault issues token based on role match + → Services validate Vault token or trust gateway-forwarded identity + +Authentik OIDC: https://authentik.riotpiao.com/application/o/vault/ +Vault OIDC role: auth/oidc/role/homelab-admin + bound_claims: { "permissions": "*" } + policy: homelab-admin (path "*" full access) +``` + +*Current (placeholder, to be replaced):* +- `apikey:` header, raw string match — **does not integrate with Authentik/Vault** +- Must be replaced with one of: + 1. Vault token validation (call Vault's `auth/token/lookup-self`) + 2. Authentik JWT validation via JWKS + 3. Trust gateway-forwarded headers (`X-User-Id`, `X-Capabilities`) + +*Rate limits (unchanged):* - Per-key limits: ingest 100 jobs/hour, query 1000 req/hour, skill fetch unlimited - Burst allowance: 10 req/sec per key (ingest waits in queue; query returns 429 Retry-After if burst exceeded)