docs: Add M7 source connectors (10 tasks), M3.5.10 auth integration, remove Kong refs

- M7.1-M7.10: Extensible SourceConnector trait, Obsidian/paperless/git/S3
  connectors, sync framework, CLI, HTTP endpoints, health monitoring, gate
- M3.5.10: Auth integration with Authentik OIDC → Vault token validation
- DESIGN.md: Add source connectors architecture, update auth to
  Authentik/Vault (Kong removed from cluster)
- INDEX.md: 75 tasks, 11 gates
- Fix all Kong references in M3.5.1 task

DESIGN.md

@@ -32,7 +32,9 @@ The mechanism is GRU-Mem (arXiv 2602.10560). Its two gates map directly onto the
 | CNPG operator | **1.30.0**, supports declarative `Database.spec.extensions` | `kubectl explain database.spec.extensions` |
 | Ollama context cap | **32768** (`OLLAMA_CONTEXT_LENGTH`) | `k8s/apps/llm-serving/ornith.yaml` |
 | Controller model | `qwen2.5:3b-instruct` | is Qwen2.5-3B-Instruct — **the paper's exact 3B backbone** |
-| Gateway auth | `apikey:` header, **not** `Authorization: Bearer` | Kong key-auth compares whole header value |
+| Gateway | nginx ingress + homelab-frontend (Go, ServiceAdapter CRDs) | Kong is **removed** from cluster |
+| IAM | Authentik (OIDC) → Vault (OIDC auth method) | Authentik issues JWT, Vault validates via OIDC, `homelab-admin` role bound on `permissions: "*"` claim |
+| Auth (placeholder) | `apikey:` header in mem-cli | **To be replaced** — does not integrate with Authentik/Vault chain |
 
 The 32K cap is the binding constraint and it fits: paper uses 5000-token chunks, 8192 max prompt, 2048 max response.
 
@@ -57,7 +59,7 @@ Memory is **levelled**, and every event in the log carries its level. The paper
 ## Architecture
 
 ```
-                            api.riotpiao.com (Kong)
+                            api.riotpiao.com (nginx ingress + homelab-frontend gateway)
                                     │
                   ┌─────────────────┼─────────────────┐
                   │                 │                 │
@@ -400,6 +402,211 @@ Two mechanical guards:
 1. **Promotion is a human move** out of `_drafts/`, reviewable as a diff.
 2. **Provenance marks derived text.** Every emitted skill carries `generated_from: <L2 sha>` in frontmatter, and `mem-ingest` tags chunks matching a known emitted artifact as `derived: true` and excludes them from evidence. Without this the corpus slowly becomes its own training data.
 
+## Source connectors — extensible multi-source ingestion
+
+Memory service is a **cluster-wide RAG** serving multiple agents and workflows. Knowledge does not live in one place — it is spread across an Obsidian vault, a paperless-ngx instance, agent session logs, and whatever document stores appear next. The connector architecture makes adding a new source a single-trait implementation rather than a pipeline rewrite.
+
+### The problem with hardcoded sources
+
+Today `mem-ingest` has three concrete sources: `PiSessionSource`, `ClaudeTranscriptSource`, `DocCorpusSource`. Each knows its own file format and emits `Record`s through the `RecordSource` trait. This works, but:
+
+1. **Every new source requires Rust code.** paperless-ngx has a REST API; Confluence has another; S3 is a third protocol. Each lands as a new `.rs` file compiled into the binary.
+2. **No runtime discovery.** The CLI must know about every source at compile time. A homelab that adds Bookstack next month needs a code change, a rebuild, and a redeploy.
+3. **No shared sync logic.** Change detection (sha-based skip), tombstoning, drift reporting — each source will reimplement these unless the framework provides them.
+
+### Design: `SourceConnector` trait + registry
+
+A connector is anything that can enumerate documents and yield their content. The framework handles chunking, embedding, change detection, and storage.
+
+```rust
+/// A connector to an external document source.
+#[async_trait]
+pub trait SourceConnector: Send + Sync {
+    /// Unique connector kind identifier (e.g., "obsidian", "paperless", "s3")
+    fn kind(&self) -> &str;
+
+    /// Human-readable name for this connector instance
+    fn name(&self) -> &str;
+
+    /// Enumerate all documents available from this source.
+    /// Returns (doc_id, doc_metadata) pairs.
+    async fn list_documents(&self) -> Result<Vec<SourceDocument>>;
+
+    /// Fetch content for a single document by its source-specific ID.
+    async fn fetch_document(&self, doc_id: &str) -> Result<DocumentContent>;
+
+    /// Health check — can we reach this source?
+    async fn health_check(&self) -> Result<SourceHealth>;
+}
+
+/// Metadata about a source document (before fetching content)
+pub struct SourceDocument {
+    pub doc_id: String,           // source-specific unique ID
+    pub title: String,            // human-readable title
+    pub source_uri: String,       // canonical URI (file://, https://, paperless://)
+    pub content_hash: Option<String>,  // if source provides hash, skip fetch on match
+    pub mime_type: String,        // text/markdown, application/pdf, etc.
+    pub updated_at: Option<OffsetDateTime>,
+}
+
+/// Fetched document content ready for chunking
+pub struct DocumentContent {
+    pub doc_id: String,
+    pub text: String,             // extracted text (markdown preferred)
+    pub source_uri: String,
+    pub content_hash: String,     // sha256 of text content
+    pub metadata: HashMap<String, String>,  // source-specific metadata
+}
+
+/// Source health status
+pub struct SourceHealth {
+    pub reachable: bool,
+    pub document_count: Option<u64>,
+    pub last_error: Option<String>,
+}
+```
+
+### Connector registry
+
+Connectors are registered at startup from a YAML config file. Adding a new source is: implement the trait, register the kind, add a config block.
+
+```yaml
+# connectors.yaml — source connector configuration
+connectors:
+  - kind: obsidian
+    name: homelab-vault
+    config:
+      root: /data/vault                # local path (PVC mount or git-sync)
+      extensions: [md, markdown, txt]
+      exclude_dirs: [.obsidian, .trash]
+
+  - kind: paperless
+    name: homelab-paperless
+    config:
+      base_url: http://paperless-ngx.paperless.svc.cluster.local:8000
+      token_secret: paperless-api-token  # k8s secret name
+      tags: [reference, manual, runbook] # only sync docs with these tags
+      format: markdown                   # request markdown export
+
+  - kind: git_repo
+    name: infra-docs
+    config:
+      repo_url: https://forgejo.riotpiao.com/rock/homelab-docs.git
+      branch: main
+      sync_interval: 1h
+      paths: [docs/, runbooks/]
+
+  - kind: s3
+    name: backup-docs
+    config:
+      endpoint: https://minio.riotpiao.com
+      bucket: knowledge-base
+      prefix: docs/
+      access_key_secret: minio-credentials
+```
+
+### Sync framework
+
+The connector trait provides enumeration and fetch. The **sync framework** handles everything else:
+
+1. **Change detection:** Compare `content_hash` from `list_documents()` against last-known hash in the manifest. Skip unchanged docs (zero embed calls).
+2. **Chunking:** Route through existing `ChunkPolicy` — heading-based for markdown, paragraph-based for plain text, configurable per connector.
+3. **Tombstoning:** Documents that vanish from `list_documents()` get tombstone records in the log. Append-only, never delete.
+4. **Drift reporting:** `mem source status` shows per-connector: docs total, changed, new, removed — without mutating anything.
+5. **Rate limiting:** Configurable fetch concurrency per connector (don't DDoS paperless with 500 parallel fetches).
+6. **Resume:** Sync is resumable. Crash mid-sync, restart, only unseen docs are processed.
+
+```
+mem source sync --all                    # sync all registered connectors
+mem source sync --name homelab-vault     # sync one connector
+mem source status                        # drift report, no mutations
+mem source list                          # registered connectors + health
+mem source add --kind paperless ...      # register new connector
+mem source rm --name old-source          # deregister + tombstone
+```
+
+### Built-in connectors (shipped with binary)
+
+| Kind | Source | Protocol | Status |
+|------|--------|----------|--------|
+| `obsidian` | Local Obsidian vault | filesystem (walkdir) | M7.2 |
+| `paperless` | paperless-ngx | REST API (`/api/documents/`) | M7.3 |
+| `git_repo` | Git repository | git clone/pull | M7.4 |
+| `s3` | S3-compatible storage | S3 API (minio, AWS) | M7.5 |
+| `pi_session` | Pi agent sessions | filesystem (existing) | M7.1 (wrap existing) |
+| `claude` | Claude transcripts | filesystem (existing) | M7.1 (wrap existing) |
+
+### How connectors interact with levels
+
+**Session connectors** (pi, claude) produce evidence that flows through the gated loop → L0/L1/L2. These are the existing `RecordSource` implementations, wrapped in `SourceConnector` for unified management.
+
+**Document connectors** (obsidian, paperless, git, s3) produce reference material → Level R. These bypass the gated loop entirely (no standing question, no gate decision). This is the existing M3.6 design, now generalized.
+
+The connector kind determines the pipeline:
+```
+Session connector → RecordSource → ChunkPolicy → GatedLoop → L0/L1/L2
+Document connector → SourceConnector → ChunkPolicy → Level R (no gate)
+```
+
+### paperless-ngx integration (concrete example)
+
+paperless-ngx is already running in the cluster. It has:
+- REST API at `http://paperless-ngx.paperless.svc.cluster.local:8000/api/`
+- Documents with tags, correspondents, document types
+- Full-text content available via API
+- Thumbnail and original file access
+
+```rust
+pub struct PaperlessConnector {
+    base_url: String,
+    token: String,
+    tag_filter: Vec<String>,
+}
+
+#[async_trait]
+impl SourceConnector for PaperlessConnector {
+    fn kind(&self) -> &str { "paperless" }
+    fn name(&self) -> &str { &self.name }
+
+    async fn list_documents(&self) -> Result<Vec<SourceDocument>> {
+        // GET /api/documents/?tags__name__in=reference,manual
+        // Paginate through results
+        // Return doc_id, title, checksum (paperless provides this)
+    }
+
+    async fn fetch_document(&self, doc_id: &str) -> Result<DocumentContent> {
+        // GET /api/documents/{id}/
+        // Extract content field (full text)
+        // Or GET /api/documents/{id}/download/ for original
+    }
+
+    async fn health_check(&self) -> Result<SourceHealth> {
+        // GET /api/ — check 200
+    }
+}
+```
+
+### Obsidian vault as a connector
+
+The Obsidian vault is the **primary reference source**. It is human-maintained, git-backed, and the canonical location for runbooks, procedures, and domain knowledge that agents need.
+
+Deployment options:
+1. **Git-sync sidecar:** A sidecar container clones the vault repo into a shared PVC. Memory service reads from the PVC.
+2. **Local mount:** For development, mount the vault directory directly.
+3. **API upload:** Push vault changes to memory service via HTTP.
+
+The Obsidian connector wraps the existing `DocCorpusSource` (M3.6.1) with the `SourceConnector` interface, adding change detection and registry management.
+
+### Future extensibility
+
+Adding a new source requires:
+1. Implement `SourceConnector` trait (~100-200 lines)
+2. Register the `kind` in the connector factory
+3. Add config block to `connectors.yaml`
+4. Run `mem source sync --name new-source`
+
+No changes to the ingest pipeline, chunking, embedding, storage, or query layers. The connector is the only new code.
+
 ## Reference corpora — the non-evidential tier
 
 `RecordSource` takes session transcripts, and the update gate asks "does this chunk contain evidence for Q". A `kubectl` or `tea` cheatsheet answers neither question: it has no session, no turn, and no evidence. Left alone the system faithfully retains *what happened when a model used a tool badly* and never learns the tool. Level **R** closes that gap, and the shape of the fix matters more than the fact of it.
@@ -509,6 +716,8 @@ A response that cannot fit its tier-1 hits inside the budget is an error, not a
 
 **P6 — Post-training (separate, Python).** Boundary is the JSONL. `mem label` uses the 32B `reasoning` model as an offline evidence labeler to produce per-chunk `U_t` ground truth (the paper had synthetic NIAH labels; we do not, and this is the honest cheapest substitute). Then verl trains a LoRA with the paper's rewards: `r_update` ±1, `r_exit` {0, −0.5 late, −0.75 early}, strict `r_format`, `α=0.9` mixing trajectory- and turn-level advantage. Requires the vLLM decision above.
 
+**P7 — Source connectors (board `M7`).** Generalizes the ingestion layer from hardcoded file sources to an extensible `SourceConnector` trait with a YAML-driven registry. Existing sources (pi sessions, Claude transcripts, `DocCorpusSource`) are wrapped in the new interface; new sources (paperless-ngx, git repos, S3) implement the trait directly. The sync framework handles change detection, tombstoning, drift reporting, and resumable sync for all connectors. The connector is the only new code when a source is added — no changes to chunking, embedding, storage, or query layers. Ordered after M3.6 because it generalizes `mem ref` rather than duplicating it, and after M4.2 because document connectors must register in the derived-content manifest to prevent cycle contamination.
+
 ## Task breakdown
 
 Board lives in **`memory-tasks/`** at the repo root. Format follows `agent-rust/tasks/`: one file per task, self-contained, each with `Acceptance` / `Verify` (harness, numbered assertions, command) / `False pass` / `Traps`, a `Status` field as source of truth, and `memory-tasks/INDEX.md` mirroring it. Ids are `M<phase>.<n>` and frozen once written — phase order is declared in `INDEX.md`, never derived from the id.
@@ -584,7 +793,7 @@ Note `agent-rust/.gitignore` excludes `tasks`, which silently untracks the whole
 
 | id | task | size | deps |
 |---|---|---|---|
-| M3.5.1 | HTTP server + router (actix-web or axum), Kong auth hook, request metrics | M | M0.1 |
+| M3.5.1 | HTTP server + router (actix-web or axum), auth hook, request metrics | M | M0.1 |
 | M3.5.2 | `POST /ingest` endpoint — `ingest_id` dedup, async queue, git context enrichment | M | M1.7, M3.5.1 |
 | M3.5.3 | `GET /query` endpoint — embed query, HNSW recall by level, rerank, walk edges to L0 | M | M3.3, M3.5.1 |
 | M3.5.4 | Federation: single query across projects, fan+merge results, deduplicate | M | M3.5.3 |
@@ -605,7 +814,22 @@ Note `agent-rust/.gitignore` excludes `tasks`, which silently untracks the whole
 | M5.5 | verl loop — `r_update` ±1, `r_exit` {0,−0.5,−0.75}, strict `r_format`, α=0.9 | L | M5.3, M5.4 |
 | M5.6 | **M5 gate** — adapter beats prompted baseline on held-out update accuracy | L | gate |
 
-Total 52 tasks, 8 gates. M0 and M2.2 have no model dependency and can start immediately; M5.4 is homelab work independent of everything else in M5 and can run in parallel. M3.5 depends on M2 (pgvector store exists) and M1 (ingest loop exists); can run in parallel with M4 and M5. M3.5.9 (git-aware references) is optional, depends on M3.5.2.
+**M7 — Source connectors** (extensible multi-source ingestion)
+
+| id | task | size | deps |
+|---|---|---|---|
+| M7.1 | `SourceConnector` trait + `SourceDocument` / `DocumentContent` types + connector registry | M | M0.3, M3.6.1 |
+| M7.2 | Obsidian vault connector — wraps `DocCorpusSource`, adds change detection, config-driven | M | M7.1, M3.6.1 |
+| M7.3 | paperless-ngx connector — REST API client, tag filtering, markdown export | M | M7.1 |
+| M7.4 | Git repo connector — clone/pull, path filtering, branch tracking | M | M7.1 |
+| M7.5 | S3 connector — S3-compatible API, prefix filtering, content-type routing | M | M7.1 |
+| M7.6 | Sync framework — change detection, tombstoning, drift report, resume | L | M7.1, M3.6.3 |
+| M7.7 | `mem source` CLI — sync/status/list/add/rm subcommands | M | M7.6 |
+| M7.8 | `GET /memory/sources` + `POST /memory/sources/sync` HTTP endpoints | M | M7.7, M3.5.1 |
+| M7.9 | Connector health monitoring + observability | S | M7.8 |
+| M7.10 | **M7 composition gate** — two connectors sync, change detection skips unchanged, tombstone works, rebuild parity | M | gate |
+
+Total 74 tasks, 11 gates. M0 and M2.2 have no model dependency and can start immediately; M5.4 is homelab work independent of everything else in M5 and can run in parallel. M3.5 depends on M2 (pgvector store exists) and M1 (ingest loop exists); can run in parallel with M4 and M5. M3.5.9 (git-aware references) is optional, depends on M3.5.2. M7 depends on M3.6 (reference corpora) and M4.2 (derived filter) for cycle prevention.
 
 ## Verification
 
@@ -677,7 +901,9 @@ The decisive P2 metric is **update-rate**, the one number distinguishing a worki
 
 ## Distributed API Layer (Homelab Frontend)
 
-**Gateway:** `api.riotpiao.com` routes agent and system memory requests through Kong.
+**Gateway:** `api.riotpiao.com` routes requests through **nginx ingress** to **homelab-frontend** gateway (Go, ServiceAdapter CRDs). Kong is **removed** from the cluster.
+
+**Auth:** Authentik (OIDC provider) → Vault (OIDC auth method, `auth/oidc/role/homelab-admin`, bound on `permissions: "*"` claim) → Vault token → service validates token. Memory service currently uses placeholder `apikey` header — M3.5.10 replaces this with Vault token validation.
 
 **Architecture assumption:** Memory services run in CNPG cluster; API layer is HTTP facade exposing read/write workflows to distributed agents. Authority remains JSONL—API is a request demultiplexer, not a cache or alternative source of truth.
 
@@ -748,7 +974,29 @@ POST   /memory/context                 <- 3-tier lookup: signature, symptom, doc
 - Client-side: `ETag: <sha256>` on all read endpoints, no conditional logic server-side (it's stateless)
 
 **Auth & rate limits:**
-- Kong `apikey:` header (existing pattern)
+
+*Auth chain (production):*
+```
+User/Agent → Authentik (OIDC login)
+  → JWT with claims (including `permissions`)
+  → Vault validates via OIDC auth method
+  → Vault issues token based on role match
+  → Services validate Vault token or trust gateway-forwarded identity
+
+Authentik OIDC: https://authentik.riotpiao.com/application/o/vault/
+Vault OIDC role: auth/oidc/role/homelab-admin
+  bound_claims: { "permissions": "*" }
+  policy: homelab-admin (path "*" full access)
+```
+
+*Current (placeholder, to be replaced):*
+- `apikey:` header, raw string match — **does not integrate with Authentik/Vault**
+- Must be replaced with one of:
+  1. Vault token validation (call Vault's `auth/token/lookup-self`)
+  2. Authentik JWT validation via JWKS
+  3. Trust gateway-forwarded headers (`X-User-Id`, `X-Capabilities`)
+
+*Rate limits (unchanged):*
 - Per-key limits: ingest 100 jobs/hour, query 1000 req/hour, skill fetch unlimited
 - Burst allowance: 10 req/sec per key (ingest waits in queue; query returns 429 Retry-After if burst exceeded)