docs: Add M7 source connectors (10 tasks), M3.5.10 auth integration, remove Kong refs
- M7.1-M7.10: Extensible SourceConnector trait, Obsidian/paperless/git/S3 connectors, sync framework, CLI, HTTP endpoints, health monitoring, gate - M3.5.10: Auth integration with Authentik OIDC → Vault token validation - DESIGN.md: Add source connectors architecture, update auth to Authentik/Vault (Kong removed from cluster) - INDEX.md: 75 tasks, 11 gates - Fix all Kong references in M3.5.1 task
This commit is contained in:
Story Crater Bot
@@ -32,7 +32,9 @@ The mechanism is GRU-Mem (arXiv 2602.10560). Its two gates map directly onto the
|
||||
| CNPG operator | **1.30.0**, supports declarative `Database.spec.extensions` | `kubectl explain database.spec.extensions` |
|
||||
| Ollama context cap | **32768** (`OLLAMA_CONTEXT_LENGTH`) | `k8s/apps/llm-serving/ornith.yaml` |
|
||||
| Controller model | `qwen2.5:3b-instruct` | is Qwen2.5-3B-Instruct — **the paper's exact 3B backbone** |
|
||||
| Gateway auth | `apikey:` header, **not** `Authorization: Bearer` | Kong key-auth compares whole header value |
|
||||
| Gateway | nginx ingress + homelab-frontend (Go, ServiceAdapter CRDs) | Kong is **removed** from cluster |
|
||||
| IAM | Authentik (OIDC) → Vault (OIDC auth method) | Authentik issues JWT, Vault validates via OIDC, `homelab-admin` role bound on `permissions: "*"` claim |
|
||||
| Auth (placeholder) | `apikey:` header in mem-cli | **To be replaced** — does not integrate with Authentik/Vault chain |
|
||||
|
||||
The 32K cap is the binding constraint and it fits: paper uses 5000-token chunks, 8192 max prompt, 2048 max response.
|
||||
|
||||
@@ -57,7 +59,7 @@ Memory is **levelled**, and every event in the log carries its level. The paper
|
||||
## Architecture
|
||||
|
||||
```
|
||||
api.riotpiao.com (Kong)
|
||||
api.riotpiao.com (nginx ingress + homelab-frontend gateway)
|
||||
│
|
||||
┌─────────────────┼─────────────────┐
|
||||
│ │ │
|
||||
@@ -400,6 +402,211 @@ Two mechanical guards:
|
||||
1. **Promotion is a human move** out of `_drafts/`, reviewable as a diff.
|
||||
2. **Provenance marks derived text.** Every emitted skill carries `generated_from: <L2 sha>` in frontmatter, and `mem-ingest` tags chunks matching a known emitted artifact as `derived: true` and excludes them from evidence. Without this the corpus slowly becomes its own training data.
|
||||
|
||||
## Source connectors — extensible multi-source ingestion
|
||||
|
||||
Memory service is a **cluster-wide RAG** serving multiple agents and workflows. Knowledge does not live in one place — it is spread across an Obsidian vault, a paperless-ngx instance, agent session logs, and whatever document stores appear next. The connector architecture makes adding a new source a single-trait implementation rather than a pipeline rewrite.
|
||||
|
||||
### The problem with hardcoded sources
|
||||
|
||||
Today `mem-ingest` has three concrete sources: `PiSessionSource`, `ClaudeTranscriptSource`, `DocCorpusSource`. Each knows its own file format and emits `Record`s through the `RecordSource` trait. This works, but:
|
||||
|
||||
1. **Every new source requires Rust code.** paperless-ngx has a REST API; Confluence has another; S3 is a third protocol. Each lands as a new `.rs` file compiled into the binary.
|
||||
2. **No runtime discovery.** The CLI must know about every source at compile time. A homelab that adds Bookstack next month needs a code change, a rebuild, and a redeploy.
|
||||
3. **No shared sync logic.** Change detection (sha-based skip), tombstoning, drift reporting — each source will reimplement these unless the framework provides them.
|
||||
|
||||
### Design: `SourceConnector` trait + registry
|
||||
|
||||
A connector is anything that can enumerate documents and yield their content. The framework handles chunking, embedding, change detection, and storage.
|
||||
|
||||
```rust
|
||||
/// A connector to an external document source.
|
||||
#[async_trait]
|
||||
pub trait SourceConnector: Send + Sync {
|
||||
/// Unique connector kind identifier (e.g., "obsidian", "paperless", "s3")
|
||||
fn kind(&self) -> &str;
|
||||
|
||||
/// Human-readable name for this connector instance
|
||||
fn name(&self) -> &str;
|
||||
|
||||
/// Enumerate all documents available from this source.
|
||||
/// Returns (doc_id, doc_metadata) pairs.
|
||||
async fn list_documents(&self) -> Result<Vec<SourceDocument>>;
|
||||
|
||||
/// Fetch content for a single document by its source-specific ID.
|
||||
async fn fetch_document(&self, doc_id: &str) -> Result<DocumentContent>;
|
||||
|
||||
/// Health check — can we reach this source?
|
||||
async fn health_check(&self) -> Result<SourceHealth>;
|
||||
}
|
||||
|
||||
/// Metadata about a source document (before fetching content)
|
||||
pub struct SourceDocument {
|
||||
pub doc_id: String, // source-specific unique ID
|
||||
pub title: String, // human-readable title
|
||||
pub source_uri: String, // canonical URI (file://, https://, paperless://)
|
||||
pub content_hash: Option<String>, // if source provides hash, skip fetch on match
|
||||
pub mime_type: String, // text/markdown, application/pdf, etc.
|
||||
pub updated_at: Option<OffsetDateTime>,
|
||||
}
|
||||
|
||||
/// Fetched document content ready for chunking
|
||||
pub struct DocumentContent {
|
||||
pub doc_id: String,
|
||||
pub text: String, // extracted text (markdown preferred)
|
||||
pub source_uri: String,
|
||||
pub content_hash: String, // sha256 of text content
|
||||
pub metadata: HashMap<String, String>, // source-specific metadata
|
||||
}
|
||||
|
||||
/// Source health status
|
||||
pub struct SourceHealth {
|
||||
pub reachable: bool,
|
||||
pub document_count: Option<u64>,
|
||||
pub last_error: Option<String>,
|
||||
}
|
||||
```
|
||||
|
||||
### Connector registry
|
||||
|
||||
Connectors are registered at startup from a YAML config file. Adding a new source is: implement the trait, register the kind, add a config block.
|
||||
|
||||
```yaml
|
||||
# connectors.yaml — source connector configuration
|
||||
connectors:
|
||||
- kind: obsidian
|
||||
name: homelab-vault
|
||||
config:
|
||||
root: /data/vault # local path (PVC mount or git-sync)
|
||||
extensions: [md, markdown, txt]
|
||||
exclude_dirs: [.obsidian, .trash]
|
||||
|
||||
- kind: paperless
|
||||
name: homelab-paperless
|
||||
config:
|
||||
base_url: http://paperless-ngx.paperless.svc.cluster.local:8000
|
||||
token_secret: paperless-api-token # k8s secret name
|
||||
tags: [reference, manual, runbook] # only sync docs with these tags
|
||||
format: markdown # request markdown export
|
||||
|
||||
- kind: git_repo
|
||||
name: infra-docs
|
||||
config:
|
||||
repo_url: https://forgejo.riotpiao.com/rock/homelab-docs.git
|
||||
branch: main
|
||||
sync_interval: 1h
|
||||
paths: [docs/, runbooks/]
|
||||
|
||||
- kind: s3
|
||||
name: backup-docs
|
||||
config:
|
||||
endpoint: https://minio.riotpiao.com
|
||||
bucket: knowledge-base
|
||||
prefix: docs/
|
||||
access_key_secret: minio-credentials
|
||||
```
|
||||
|
||||
### Sync framework
|
||||
|
||||
The connector trait provides enumeration and fetch. The **sync framework** handles everything else:
|
||||
|
||||
1. **Change detection:** Compare `content_hash` from `list_documents()` against last-known hash in the manifest. Skip unchanged docs (zero embed calls).
|
||||
2. **Chunking:** Route through existing `ChunkPolicy` — heading-based for markdown, paragraph-based for plain text, configurable per connector.
|
||||
3. **Tombstoning:** Documents that vanish from `list_documents()` get tombstone records in the log. Append-only, never delete.
|
||||
4. **Drift reporting:** `mem source status` shows per-connector: docs total, changed, new, removed — without mutating anything.
|
||||
5. **Rate limiting:** Configurable fetch concurrency per connector (don't DDoS paperless with 500 parallel fetches).
|
||||
6. **Resume:** Sync is resumable. Crash mid-sync, restart, only unseen docs are processed.
|
||||
|
||||
```
|
||||
mem source sync --all # sync all registered connectors
|
||||
mem source sync --name homelab-vault # sync one connector
|
||||
mem source status # drift report, no mutations
|
||||
mem source list # registered connectors + health
|
||||
mem source add --kind paperless ... # register new connector
|
||||
mem source rm --name old-source # deregister + tombstone
|
||||
```
|
||||
|
||||
### Built-in connectors (shipped with binary)
|
||||
|
||||
| Kind | Source | Protocol | Status |
|
||||
|------|--------|----------|--------|
|
||||
| `obsidian` | Local Obsidian vault | filesystem (walkdir) | M7.2 |
|
||||
| `paperless` | paperless-ngx | REST API (`/api/documents/`) | M7.3 |
|
||||
| `git_repo` | Git repository | git clone/pull | M7.4 |
|
||||
| `s3` | S3-compatible storage | S3 API (minio, AWS) | M7.5 |
|
||||
| `pi_session` | Pi agent sessions | filesystem (existing) | M7.1 (wrap existing) |
|
||||
| `claude` | Claude transcripts | filesystem (existing) | M7.1 (wrap existing) |
|
||||
|
||||
### How connectors interact with levels
|
||||
|
||||
**Session connectors** (pi, claude) produce evidence that flows through the gated loop → L0/L1/L2. These are the existing `RecordSource` implementations, wrapped in `SourceConnector` for unified management.
|
||||
|
||||
**Document connectors** (obsidian, paperless, git, s3) produce reference material → Level R. These bypass the gated loop entirely (no standing question, no gate decision). This is the existing M3.6 design, now generalized.
|
||||
|
||||
The connector kind determines the pipeline:
|
||||
```
|
||||
Session connector → RecordSource → ChunkPolicy → GatedLoop → L0/L1/L2
|
||||
Document connector → SourceConnector → ChunkPolicy → Level R (no gate)
|
||||
```
|
||||
|
||||
### paperless-ngx integration (concrete example)
|
||||
|
||||
paperless-ngx is already running in the cluster. It has:
|
||||
- REST API at `http://paperless-ngx.paperless.svc.cluster.local:8000/api/`
|
||||
- Documents with tags, correspondents, document types
|
||||
- Full-text content available via API
|
||||
- Thumbnail and original file access
|
||||
|
||||
```rust
|
||||
pub struct PaperlessConnector {
|
||||
base_url: String,
|
||||
token: String,
|
||||
tag_filter: Vec<String>,
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl SourceConnector for PaperlessConnector {
|
||||
fn kind(&self) -> &str { "paperless" }
|
||||
fn name(&self) -> &str { &self.name }
|
||||
|
||||
async fn list_documents(&self) -> Result<Vec<SourceDocument>> {
|
||||
// GET /api/documents/?tags__name__in=reference,manual
|
||||
// Paginate through results
|
||||
// Return doc_id, title, checksum (paperless provides this)
|
||||
}
|
||||
|
||||
async fn fetch_document(&self, doc_id: &str) -> Result<DocumentContent> {
|
||||
// GET /api/documents/{id}/
|
||||
// Extract content field (full text)
|
||||
// Or GET /api/documents/{id}/download/ for original
|
||||
}
|
||||
|
||||
async fn health_check(&self) -> Result<SourceHealth> {
|
||||
// GET /api/ — check 200
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Obsidian vault as a connector
|
||||
|
||||
The Obsidian vault is the **primary reference source**. It is human-maintained, git-backed, and the canonical location for runbooks, procedures, and domain knowledge that agents need.
|
||||
|
||||
Deployment options:
|
||||
1. **Git-sync sidecar:** A sidecar container clones the vault repo into a shared PVC. Memory service reads from the PVC.
|
||||
2. **Local mount:** For development, mount the vault directory directly.
|
||||
3. **API upload:** Push vault changes to memory service via HTTP.
|
||||
|
||||
The Obsidian connector wraps the existing `DocCorpusSource` (M3.6.1) with the `SourceConnector` interface, adding change detection and registry management.
|
||||
|
||||
### Future extensibility
|
||||
|
||||
Adding a new source requires:
|
||||
1. Implement `SourceConnector` trait (~100-200 lines)
|
||||
2. Register the `kind` in the connector factory
|
||||
3. Add config block to `connectors.yaml`
|
||||
4. Run `mem source sync --name new-source`
|
||||
|
||||
No changes to the ingest pipeline, chunking, embedding, storage, or query layers. The connector is the only new code.
|
||||
|
||||
## Reference corpora — the non-evidential tier
|
||||
|
||||
`RecordSource` takes session transcripts, and the update gate asks "does this chunk contain evidence for Q". A `kubectl` or `tea` cheatsheet answers neither question: it has no session, no turn, and no evidence. Left alone the system faithfully retains *what happened when a model used a tool badly* and never learns the tool. Level **R** closes that gap, and the shape of the fix matters more than the fact of it.
|
||||
@@ -509,6 +716,8 @@ A response that cannot fit its tier-1 hits inside the budget is an error, not a
|
||||
|
||||
**P6 — Post-training (separate, Python).** Boundary is the JSONL. `mem label` uses the 32B `reasoning` model as an offline evidence labeler to produce per-chunk `U_t` ground truth (the paper had synthetic NIAH labels; we do not, and this is the honest cheapest substitute). Then verl trains a LoRA with the paper's rewards: `r_update` ±1, `r_exit` {0, −0.5 late, −0.75 early}, strict `r_format`, `α=0.9` mixing trajectory- and turn-level advantage. Requires the vLLM decision above.
|
||||
|
||||
**P7 — Source connectors (board `M7`).** Generalizes the ingestion layer from hardcoded file sources to an extensible `SourceConnector` trait with a YAML-driven registry. Existing sources (pi sessions, Claude transcripts, `DocCorpusSource`) are wrapped in the new interface; new sources (paperless-ngx, git repos, S3) implement the trait directly. The sync framework handles change detection, tombstoning, drift reporting, and resumable sync for all connectors. The connector is the only new code when a source is added — no changes to chunking, embedding, storage, or query layers. Ordered after M3.6 because it generalizes `mem ref` rather than duplicating it, and after M4.2 because document connectors must register in the derived-content manifest to prevent cycle contamination.
|
||||
|
||||
## Task breakdown
|
||||
|
||||
Board lives in **`memory-tasks/`** at the repo root. Format follows `agent-rust/tasks/`: one file per task, self-contained, each with `Acceptance` / `Verify` (harness, numbered assertions, command) / `False pass` / `Traps`, a `Status` field as source of truth, and `memory-tasks/INDEX.md` mirroring it. Ids are `M<phase>.<n>` and frozen once written — phase order is declared in `INDEX.md`, never derived from the id.
|
||||
@@ -584,7 +793,7 @@ Note `agent-rust/.gitignore` excludes `tasks`, which silently untracks the whole
|
||||
|
||||
| id | task | size | deps |
|
||||
|---|---|---|---|
|
||||
| M3.5.1 | HTTP server + router (actix-web or axum), Kong auth hook, request metrics | M | M0.1 |
|
||||
| M3.5.1 | HTTP server + router (actix-web or axum), auth hook, request metrics | M | M0.1 |
|
||||
| M3.5.2 | `POST /ingest` endpoint — `ingest_id` dedup, async queue, git context enrichment | M | M1.7, M3.5.1 |
|
||||
| M3.5.3 | `GET /query` endpoint — embed query, HNSW recall by level, rerank, walk edges to L0 | M | M3.3, M3.5.1 |
|
||||
| M3.5.4 | Federation: single query across projects, fan+merge results, deduplicate | M | M3.5.3 |
|
||||
@@ -605,7 +814,22 @@ Note `agent-rust/.gitignore` excludes `tasks`, which silently untracks the whole
|
||||
| M5.5 | verl loop — `r_update` ±1, `r_exit` {0,−0.5,−0.75}, strict `r_format`, α=0.9 | L | M5.3, M5.4 |
|
||||
| M5.6 | **M5 gate** — adapter beats prompted baseline on held-out update accuracy | L | gate |
|
||||
|
||||
Total 52 tasks, 8 gates. M0 and M2.2 have no model dependency and can start immediately; M5.4 is homelab work independent of everything else in M5 and can run in parallel. M3.5 depends on M2 (pgvector store exists) and M1 (ingest loop exists); can run in parallel with M4 and M5. M3.5.9 (git-aware references) is optional, depends on M3.5.2.
|
||||
**M7 — Source connectors** (extensible multi-source ingestion)
|
||||
|
||||
| id | task | size | deps |
|
||||
|---|---|---|---|
|
||||
| M7.1 | `SourceConnector` trait + `SourceDocument` / `DocumentContent` types + connector registry | M | M0.3, M3.6.1 |
|
||||
| M7.2 | Obsidian vault connector — wraps `DocCorpusSource`, adds change detection, config-driven | M | M7.1, M3.6.1 |
|
||||
| M7.3 | paperless-ngx connector — REST API client, tag filtering, markdown export | M | M7.1 |
|
||||
| M7.4 | Git repo connector — clone/pull, path filtering, branch tracking | M | M7.1 |
|
||||
| M7.5 | S3 connector — S3-compatible API, prefix filtering, content-type routing | M | M7.1 |
|
||||
| M7.6 | Sync framework — change detection, tombstoning, drift report, resume | L | M7.1, M3.6.3 |
|
||||
| M7.7 | `mem source` CLI — sync/status/list/add/rm subcommands | M | M7.6 |
|
||||
| M7.8 | `GET /memory/sources` + `POST /memory/sources/sync` HTTP endpoints | M | M7.7, M3.5.1 |
|
||||
| M7.9 | Connector health monitoring + observability | S | M7.8 |
|
||||
| M7.10 | **M7 composition gate** — two connectors sync, change detection skips unchanged, tombstone works, rebuild parity | M | gate |
|
||||
|
||||
Total 74 tasks, 11 gates. M0 and M2.2 have no model dependency and can start immediately; M5.4 is homelab work independent of everything else in M5 and can run in parallel. M3.5 depends on M2 (pgvector store exists) and M1 (ingest loop exists); can run in parallel with M4 and M5. M3.5.9 (git-aware references) is optional, depends on M3.5.2. M7 depends on M3.6 (reference corpora) and M4.2 (derived filter) for cycle prevention.
|
||||
|
||||
## Verification
|
||||
|
||||
@@ -677,7 +901,9 @@ The decisive P2 metric is **update-rate**, the one number distinguishing a worki
|
||||
|
||||
## Distributed API Layer (Homelab Frontend)
|
||||
|
||||
**Gateway:** `api.riotpiao.com` routes agent and system memory requests through Kong.
|
||||
**Gateway:** `api.riotpiao.com` routes requests through **nginx ingress** to **homelab-frontend** gateway (Go, ServiceAdapter CRDs). Kong is **removed** from the cluster.
|
||||
|
||||
**Auth:** Authentik (OIDC provider) → Vault (OIDC auth method, `auth/oidc/role/homelab-admin`, bound on `permissions: "*"` claim) → Vault token → service validates token. Memory service currently uses placeholder `apikey` header — M3.5.10 replaces this with Vault token validation.
|
||||
|
||||
**Architecture assumption:** Memory services run in CNPG cluster; API layer is HTTP facade exposing read/write workflows to distributed agents. Authority remains JSONL—API is a request demultiplexer, not a cache or alternative source of truth.
|
||||
|
||||
@@ -748,7 +974,29 @@ POST /memory/context <- 3-tier lookup: signature, symptom, doc
|
||||
- Client-side: `ETag: <sha256>` on all read endpoints, no conditional logic server-side (it's stateless)
|
||||
|
||||
**Auth & rate limits:**
|
||||
- Kong `apikey:` header (existing pattern)
|
||||
|
||||
*Auth chain (production):*
|
||||
```
|
||||
User/Agent → Authentik (OIDC login)
|
||||
→ JWT with claims (including `permissions`)
|
||||
→ Vault validates via OIDC auth method
|
||||
→ Vault issues token based on role match
|
||||
→ Services validate Vault token or trust gateway-forwarded identity
|
||||
|
||||
Authentik OIDC: https://authentik.riotpiao.com/application/o/vault/
|
||||
Vault OIDC role: auth/oidc/role/homelab-admin
|
||||
bound_claims: { "permissions": "*" }
|
||||
policy: homelab-admin (path "*" full access)
|
||||
```
|
||||
|
||||
*Current (placeholder, to be replaced):*
|
||||
- `apikey:` header, raw string match — **does not integrate with Authentik/Vault**
|
||||
- Must be replaced with one of:
|
||||
1. Vault token validation (call Vault's `auth/token/lookup-self`)
|
||||
2. Authentik JWT validation via JWKS
|
||||
3. Trust gateway-forwarded headers (`X-User-Id`, `X-Capabilities`)
|
||||
|
||||
*Rate limits (unchanged):*
|
||||
- Per-key limits: ingest 100 jobs/hour, query 1000 req/hour, skill fetch unlimited
|
||||
- Burst allowance: 10 req/sec per key (ingest waits in queue; query returns 429 Retry-After if burst exceeded)
|
||||
|
||||
|
||||
+30
-2
@@ -62,7 +62,7 @@ Legend: ⬜ not started · 🟡 in progress · ✅ done · ⛔ blocked
|
||||
| 2 | Gated loop at L1 | M1.x | 8 | 8 | 0 | 0 | ✅ M1.8 |
|
||||
| 3 | Projections | M2.x | 8 | 5 | 0 | 3 | ✅ M2.8 (M2.1, M2.3, M2.4, M2.5 ✅) |
|
||||
| 4 | L2 synthesis + retrieval | M3.x | 4 | 4 | 0 | 0 | ✅ M3.4 |
|
||||
| 4.5 | Distributed API Layer | M3.5.x | 9 | 8 | 0 | 1 | ✅ M3.5.8 |
|
||||
| 4.5 | Distributed API Layer | M3.5.x | 10 | 8 | 0 | 2 | ✅ M3.5.8 |
|
||||
| 5 | Skills | M4.x | 3 | 2 | 0 | 1 | ⬜ M4.3 |
|
||||
| 5.5 | Reference corpora | M3.6.x | 6 | 1 | 0 | 5 | ⬜ M3.6.6 |
|
||||
| 5.6 | Tool context | M3.7.x | 6 | 0 | 2 | 4 | ⬜ M3.7.6 |
|
||||
@@ -146,7 +146,7 @@ Homelab frontend integration: HTTP facade via `api.riotpiao.com`. Runs in parall
|
||||
|
||||
| Task | Title | Size | Flags | Status |
|
||||
|---|---|---|---|---|
|
||||
| [M3.5.1](M3.5.1-http-server.md) | HTTP server + router, Kong auth, metrics | M | — | ✅ |
|
||||
| [M3.5.1](M3.5.1-http-server.md) | HTTP server + router, auth hook, metrics | M | — | ✅ |
|
||||
| [M3.5.2](M3.5.2-ingest-endpoint.md) | POST /ingest async queue | M | — | ✅ |
|
||||
| [M3.5.3](M3.5.3-query-endpoint.md) | GET /query HNSW+rerank | M | — | ✅ |
|
||||
| [M3.5.4](M3.5.4-query-federation.md) | Query federation | M | — | ✅ |
|
||||
@@ -155,6 +155,7 @@ Homelab frontend integration: HTTP facade via `api.riotpiao.com`. Runs in parall
|
||||
| [M3.5.7](M3.5.7-rate-limiting.md) | Rate limiting | M | — | ✅ |
|
||||
| [M3.5.8](M3.5.8-m3.5-gate.md) | **M3.5 composition gate** | M | gate | ✅ |
|
||||
| [M3.5.9](M3.5.9-git-aware-references.md) | Git-aware references: lookup by code location | M | — | ⬜ |
|
||||
| [M3.5.10](M3.5.10-auth-integration.md) | Auth: Authentik/Vault OIDC token validation | M | — | ⬜ |
|
||||
|
||||
## 5 — Skills · M4.x
|
||||
|
||||
@@ -254,6 +255,33 @@ durability-of-location, not a multi-host requirement.
|
||||
| [M6.5](M6.5-credentials-secret.md) | Postgres credentials for the Mac client | S | homelab | ⬜ |
|
||||
| [M6.6](M6.6-m6-gate.md) | **M6 composition gate** | M | gate | ⬜ |
|
||||
|
||||
## 8 — Source connectors · M7.x
|
||||
|
||||
Extensible multi-source ingestion. `SourceConnector` trait + YAML-driven registry.
|
||||
Adding a new document source (paperless-ngx, S3, git repo) requires implementing
|
||||
one trait and adding one config block — no changes to the ingest pipeline, chunking,
|
||||
embedding, storage, or query layers.
|
||||
|
||||
**Document connectors** produce Level R content (reference material, bypasses gated
|
||||
loop). **Session connectors** (pi, claude) produce evidence for L0/L1/L2. The
|
||||
connector's `source_type()` declares the pipeline.
|
||||
|
||||
Sync framework handles change detection (sha-based skip), tombstoning, drift
|
||||
reporting, and resumable sync for all connectors.
|
||||
|
||||
| Task | Title | Size | Flags | Status |
|
||||
|---|---|---|---|---|
|
||||
| [M7.1](M7.1-source-connector-trait.md) | `SourceConnector` trait + registry | M | — | ⬜ |
|
||||
| [M7.2](M7.2-obsidian-connector.md) | Obsidian vault connector | M | — | ⬜ |
|
||||
| [M7.3](M7.3-paperless-connector.md) | paperless-ngx connector | M | — | ⬜ |
|
||||
| [M7.4](M7.4-git-repo-connector.md) | Git repo connector | M | — | ⬜ |
|
||||
| [M7.5](M7.5-s3-connector.md) | S3-compatible storage connector | M | — | ⬜ |
|
||||
| [M7.6](M7.6-sync-framework.md) | Sync framework | L | — | ⬜ |
|
||||
| [M7.7](M7.7-source-cli.md) | `mem source` CLI | M | — | ⬜ |
|
||||
| [M7.8](M7.8-source-http-endpoints.md) | Source HTTP endpoints | M | — | ⬜ |
|
||||
| [M7.9](M7.9-connector-health-monitoring.md) | Connector health + observability | S | — | ⬜ |
|
||||
| [M7.10](M7.10-m7-gate.md) | **M7 composition gate** | M | gate | ⬜ |
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) · GRU-Mem, arXiv 2602.10560 · `internal/store/store.go` (agent-manager, `add-headless-spawn` branch)
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# M3.5.1 — HTTP server + router, Kong auth hook, metrics
|
||||
# M3.5.1 — HTTP server + router, auth hook, metrics
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
@@ -11,12 +11,12 @@
|
||||
|
||||
## Goal
|
||||
|
||||
HTTP facade for homelab gateway. Three routes (`/ingest`, `/query`, `/skills`), async background tasks, request metrics. Auth hook validates Kong `apikey:` header. Stateless — no business logic here, just request demultiplexing.
|
||||
HTTP facade for homelab gateway. Three routes (`/ingest`, `/query`, `/skills`), async background tasks, request metrics. Auth hook validates `apikey:` header (placeholder — M3.5.10 replaces with Vault/Authentik OIDC). Stateless — no business logic here, just request demultiplexing.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Kong (api.riotpiao.com)
|
||||
nginx ingress + homelab-frontend gateway (api.riotpiao.com)
|
||||
↓ apikey validation
|
||||
HTTP Server (Rust httpd, actix-web or axum)
|
||||
↓ route dispatch
|
||||
@@ -33,7 +33,7 @@ HTTP Server (Rust httpd, actix-web or axum)
|
||||
- `GET /memory/skills` — returns 200 with empty skills `[]`
|
||||
4. Request logger middleware — every request logs method, path, status, latency in one line (not pretty-printed).
|
||||
5. Metrics middleware — track latency histogram per route (p50/p95/p99 in microseconds), request count, error count.
|
||||
6. Kong auth hook:
|
||||
6. Auth hook (placeholder):
|
||||
- Extract `apikey:` header (case-insensitive header name, exact value match against stored key)
|
||||
- If missing or unrecognized → 401 with `{"error":"unauthorized","reason":"missing apikey header"}`
|
||||
- Pass apikey to request context so handlers can log which key made the request
|
||||
@@ -73,7 +73,7 @@ HTTP Server (Rust httpd, actix-web or axum)
|
||||
## Traps
|
||||
|
||||
- Actix-web's `.service()` does not inherit middleware registered outside a scope; scope middleware applies only to routes inside that scope.
|
||||
- Header name case matters for Kong's key-auth; `apikey:` is lowercase.
|
||||
- Header name: `apikey:` (lowercase). Placeholder — M3.5.10 replaces with Vault token validation.
|
||||
- `tokio::runtime::Runtime::new()` in tests blocks on network if used naively — use test utilities from `actix-web` or `axum` that spawn the server in a background thread.
|
||||
- Metrics registered at startup are easy to forget to increment. Middleware must actually call the metrics update, not just define it.
|
||||
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
# M3.5.10 — Auth integration with Authentik/Vault
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M3.5 — Distributed API Layer |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | — |
|
||||
| Depends | M3.5.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Replace the placeholder `apikey` header check with proper authentication via the
|
||||
cluster's IAM stack: **Authentik** (OIDC provider) → **HashiCorp Vault** (token
|
||||
issuer) → **memory service** (token validator).
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
**Current (wrong):**
|
||||
```rust
|
||||
fn check_auth(req: &HttpRequest, state: &AppState) -> Result<(), HttpResponse> {
|
||||
let api_key = req.headers().get("apikey").and_then(|h| h.to_str().ok());
|
||||
if api_key != Some(&state.api_key) {
|
||||
return Err(HttpResponse::Unauthorized().json(...));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
This is a raw string match against `MEM_API_KEY` env var. No JWT, no Vault, no
|
||||
user identity. It does not integrate with the cluster's IAM stack.
|
||||
|
||||
**Cluster IAM stack:**
|
||||
- **Authentik** (`iam` namespace) — OIDC provider at
|
||||
`https://authentik.riotpiao.com/application/o/vault/`
|
||||
- **HashiCorp Vault** (`iam` namespace) — OIDC auth method enabled, validates
|
||||
Authentik JWTs, issues Vault tokens based on role/policy.
|
||||
- **Vault OIDC role:** `auth/oidc/role/homelab-admin`
|
||||
- `bound_claims: { "permissions": "*" }`
|
||||
- Policy: `homelab-admin` (path `"*"` full access)
|
||||
- **Vault unseal:** Shamir 3/3, keys in `vault-unseal-keys` secret, S3 backend
|
||||
via MinIO.
|
||||
|
||||
**Auth flow (production):**
|
||||
```
|
||||
User/Agent authenticates with Authentik (OIDC)
|
||||
→ Receives JWT with claims { sub, permissions, groups, ... }
|
||||
→ Presents JWT to Vault OIDC auth method
|
||||
→ Vault validates JWT against Authentik JWKS
|
||||
→ Vault issues Vault token with matched policy
|
||||
→ Client sends Vault token to memory service
|
||||
→ Memory service validates token via Vault API
|
||||
```
|
||||
|
||||
**Three integration options (pick one):**
|
||||
|
||||
### Option A: Vault token validation (recommended)
|
||||
Memory service receives `X-Vault-Token` header, calls Vault's
|
||||
`POST /v1/auth/token/lookup-self` to validate. Extracts policy and metadata.
|
||||
- Pro: Vault is the single source of truth for authorization.
|
||||
- Pro: Token revocation is immediate (Vault controls lifecycle).
|
||||
- Con: Extra network call per request (cache with TTL to mitigate).
|
||||
|
||||
### Option B: Direct JWKS validation
|
||||
Memory service fetches Authentik's JWKS endpoint, validates JWT `Authorization:
|
||||
Bearer <token>` directly. No Vault in the request path.
|
||||
- Pro: No Vault dependency at request time.
|
||||
- Pro: Standard OAuth2/OIDC pattern.
|
||||
- Con: Token revocation is delayed (until JWT expires).
|
||||
- Con: Memory service must know about Authentik's OIDC config.
|
||||
|
||||
### Option C: Trust gateway
|
||||
Memory service trusts homelab-frontend gateway (cluster-internal traffic).
|
||||
Gateway validates auth, forwards `X-User-Id` and `X-Capabilities` headers.
|
||||
Memory service checks capabilities against ServiceAdapter CRD requirements.
|
||||
- Pro: Auth logic centralized in gateway.
|
||||
- Pro: Memory service stays simple.
|
||||
- Con: Gateway auth is currently a stub (`hasCapability()` returns true for any
|
||||
`Authorization` header).
|
||||
- Con: Requires gateway auth to be completed first (homelab-frontend task 8.3).
|
||||
|
||||
**ServiceAdapter CRD for memory (`memory-adapter`):**
|
||||
```yaml
|
||||
auth:
|
||||
capability: memory:read # default
|
||||
required: true
|
||||
resources:
|
||||
- name: ingest
|
||||
methods:
|
||||
- verb: POST
|
||||
auth: { capability: memory:write, required: true }
|
||||
- name: query
|
||||
methods:
|
||||
- verb: POST
|
||||
- name: skills
|
||||
methods:
|
||||
- verb: GET
|
||||
```
|
||||
|
||||
**Capabilities needed:**
|
||||
- `memory:read` — query, skills, vault browse, projects, sources
|
||||
- `memory:write` — ingest, source sync, skill draft
|
||||
|
||||
## Steps
|
||||
|
||||
### Option A (Vault token — recommended)
|
||||
|
||||
1. Add `vault_addr` to `AppState` (default: `http://vault.iam.svc.cluster.local:8200`).
|
||||
2. Replace `check_auth()` with `validate_vault_token()`:
|
||||
```rust
|
||||
async fn validate_vault_token(req: &HttpRequest, state: &AppState) -> Result<VaultIdentity, HttpResponse> {
|
||||
let token = req.headers().get("X-Vault-Token")
|
||||
.or_else(|| req.headers().get("Authorization")) // Bearer <token>
|
||||
.and_then(|h| h.to_str().ok());
|
||||
// POST vault_addr/v1/auth/token/lookup-self with X-Vault-Token header
|
||||
// Parse response: policies, metadata, ttl
|
||||
// Cache token -> identity for TTL duration
|
||||
}
|
||||
```
|
||||
3. Add token cache (HashMap<token_hash, (VaultIdentity, Instant)>) with configurable TTL.
|
||||
4. Extract `VaultIdentity` (policies, metadata) from lookup response.
|
||||
5. Map policies to capabilities: `homelab-admin` → `memory:read` + `memory:write`.
|
||||
6. Update each handler to check required capability.
|
||||
7. Keep `apikey` as fallback for dev/test (controlled by env var `MEM_AUTH_MODE=vault|apikey`).
|
||||
|
||||
### For all options
|
||||
|
||||
8. Add env vars: `VAULT_ADDR`, `MEM_AUTH_MODE` (vault/jwks/gateway/apikey).
|
||||
9. Update K8s deployment to inject `VAULT_ADDR`.
|
||||
10. Update ServiceAdapter CRD if needed.
|
||||
11. Document auth flow in README.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Requests with valid Vault token are accepted.
|
||||
- Requests with expired/revoked Vault token are rejected (401).
|
||||
- Requests without any auth are rejected (401).
|
||||
- `memory:write` capability required for ingest/sync endpoints.
|
||||
- `memory:read` capability sufficient for query/skills/vault endpoints.
|
||||
- Token cache reduces Vault API calls on repeated requests.
|
||||
- Fallback to `apikey` mode for dev/test environments.
|
||||
|
||||
## Verify
|
||||
|
||||
**Integration test** — `tests/it_auth_integration.rs`:
|
||||
1. `a1_vault_token_accepted` — mock Vault lookup-self returning valid response;
|
||||
assert request proceeds.
|
||||
2. `a2_expired_token_rejected` — mock Vault returning 403; assert 401 response.
|
||||
3. `a3_no_auth_rejected` — request with no auth headers; assert 401.
|
||||
4. `a4_write_requires_capability` — token with `memory:read` only; POST /ingest;
|
||||
assert 403.
|
||||
5. `a5_read_with_read_capability` — token with `memory:read`; GET /query;
|
||||
assert proceeds.
|
||||
6. `a6_token_cache_hit` — same token twice; assert Vault called once.
|
||||
7. `a7_apikey_fallback` — `MEM_AUTH_MODE=apikey`; assert old behavior works.
|
||||
8. `a8_auth_mode_configurable` — assert `MEM_AUTH_MODE` switches validation logic.
|
||||
|
||||
**Command:** `cargo test --test it_auth_integration`
|
||||
|
||||
**False pass:**
|
||||
- Testing only with apikey fallback. The Vault integration is the whole point.
|
||||
- Mocking Vault without testing cache expiry. A cache that never expires accepts
|
||||
revoked tokens forever.
|
||||
|
||||
## Traps
|
||||
|
||||
- Calling Vault on every request without caching. Vault API calls add 5-10ms
|
||||
per request. Cache with TTL matching token TTL (or shorter).
|
||||
- Not handling Vault being temporarily unreachable. Return 503 (not 401) if
|
||||
Vault is down — "cannot verify" is not "unauthorized".
|
||||
- Hardcoding Vault addr. Use env var + service discovery.
|
||||
- Not supporting `Authorization: Bearer <token>` format alongside `X-Vault-Token`.
|
||||
Different clients use different conventions.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — auth section, Authentik/Vault IAM stack
|
||||
@@ -0,0 +1,122 @@
|
||||
# M7.1 — `SourceConnector` trait + registry
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M7 — Source connectors |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M7.2, M7.3, M7.4, M7.5, M7.6, M7.10 |
|
||||
| Depends | M0.3, M3.6.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Define the extensible connector interface so that adding a new document source
|
||||
(paperless-ngx, S3, git repo, etc.) requires implementing one trait and adding
|
||||
one YAML config block — no changes to the ingest pipeline, chunking, embedding,
|
||||
storage, or query layers.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
Memory service is a **cluster-wide RAG** serving multiple agents. Knowledge lives
|
||||
in many places: an Obsidian vault, paperless-ngx, git repositories, S3 buckets.
|
||||
The connector abstraction makes all of them look the same to the ingest pipeline.
|
||||
|
||||
**The trait has three methods.** `list_documents()` enumerates what's available
|
||||
without fetching content. `fetch_document()` retrieves one document's text.
|
||||
`health_check()` reports reachability. The sync framework (M7.6) handles
|
||||
everything else — change detection, tombstoning, chunking, embedding.
|
||||
|
||||
**Configuration is YAML-driven.** Each connector instance is a block in
|
||||
`connectors.yaml` with `kind`, `name`, and source-specific `config`. The registry
|
||||
maps `kind` to a factory function that constructs the connector from its config.
|
||||
|
||||
**Two connector families exist.** Session connectors (pi, claude) produce evidence
|
||||
for the gated loop (L0/L1/L2). Document connectors (obsidian, paperless, git, s3)
|
||||
produce reference material at Level R, bypassing the gate. The connector's
|
||||
`source_type()` method declares which family it belongs to.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Define `SourceConnector` trait in `mem-ingest/src/connector.rs`:
|
||||
```rust
|
||||
#[async_trait]
|
||||
pub trait SourceConnector: Send + Sync {
|
||||
fn kind(&self) -> &str;
|
||||
fn name(&self) -> &str;
|
||||
fn source_type(&self) -> SourceType; // Evidence or Reference
|
||||
async fn list_documents(&self) -> Result<Vec<SourceDocument>>;
|
||||
async fn fetch_document(&self, doc_id: &str) -> Result<DocumentContent>;
|
||||
async fn health_check(&self) -> Result<SourceHealth>;
|
||||
}
|
||||
```
|
||||
2. Define supporting types: `SourceDocument`, `DocumentContent`, `SourceHealth`,
|
||||
`SourceType` enum (`Evidence`, `Reference`).
|
||||
3. Define `ConnectorConfig` serde struct for YAML deserialization:
|
||||
```yaml
|
||||
connectors:
|
||||
- kind: obsidian
|
||||
name: homelab-vault
|
||||
config: { root: /data/vault, extensions: [md, txt] }
|
||||
```
|
||||
4. Implement `ConnectorRegistry` — maps `kind` string to factory function,
|
||||
constructs connectors from config at startup.
|
||||
5. Add `connectors.yaml` loading in `mem-cli` startup path.
|
||||
6. Provide a `VecConnector` test helper (in-memory documents) for testing
|
||||
downstream consumers without real I/O.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- The trait compiles and is object-safe (`Box<dyn SourceConnector>`).
|
||||
- `ConnectorRegistry` can register and construct connectors by kind string.
|
||||
- `VecConnector` implements the trait and passes basic list/fetch assertions.
|
||||
- YAML config deserialization works for known and unknown kinds (unknown = skip
|
||||
with warning, not crash).
|
||||
- `source_type()` is enforced at the type level — no runtime flag confusion.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** in-memory `VecConnector`, YAML config fixtures.
|
||||
|
||||
**Integration test** — `tests/it_source_connector.rs`:
|
||||
1. `a1_trait_is_object_safe` — construct a `Box<dyn SourceConnector>` from
|
||||
`VecConnector`; call all three methods.
|
||||
2. `a2_registry_constructs_by_kind` — register "vec" kind, construct from config,
|
||||
assert `kind()` and `name()` match.
|
||||
3. `a3_list_documents_returns_all` — `VecConnector` with 3 docs, assert
|
||||
`list_documents()` returns 3.
|
||||
4. `a4_fetch_document_by_id` — assert content matches what was registered.
|
||||
5. `a5_fetch_unknown_id_errors` — assert `fetch_document("nonexistent")` returns
|
||||
an error, not a panic.
|
||||
6. `a6_health_check_reports_count` — assert `health_check()` returns
|
||||
`document_count = Some(3)`.
|
||||
7. `a7_yaml_config_loads` — parse a fixture `connectors.yaml` with two connector
|
||||
blocks; assert both are constructed.
|
||||
8. `a8_unknown_kind_skipped` — config with `kind: "nonexistent"`; assert registry
|
||||
logs a warning and continues without the connector.
|
||||
9. `a9_source_type_evidence_vs_reference` — assert session connectors return
|
||||
`Evidence`, document connectors return `Reference`.
|
||||
10. `a10_empty_config_is_valid` — no `connectors.yaml` or empty file; registry
|
||||
starts with zero connectors, no crash.
|
||||
|
||||
**Command:** `cargo test --test it_source_connector`
|
||||
|
||||
**False pass:**
|
||||
- Testing only `VecConnector` and claiming the trait works. The trait is
|
||||
validated by M7.2–M7.5 implementing it against real sources.
|
||||
- Parsing YAML without verifying the constructed connector's methods work.
|
||||
|
||||
## Traps
|
||||
|
||||
- Making the trait not object-safe (generic methods, `Self` in return types).
|
||||
Every consumer stores `Box<dyn SourceConnector>`, so object safety is load-bearing.
|
||||
- Putting chunking logic inside the connector. Connectors fetch documents; the
|
||||
sync framework (M7.6) chunks them. Mixing concerns means every connector
|
||||
reimplements chunking.
|
||||
- Hard-coding the list of known kinds. The registry must be extensible — a
|
||||
`register(kind, factory_fn)` call, not a match statement.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — source connectors section
|
||||
@@ -0,0 +1,92 @@
|
||||
# M7.10 — M7 composition gate — source connectors
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M7 — Source connectors |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | gate |
|
||||
| Spec | inlined below |
|
||||
| Blocks | — |
|
||||
| Depends | M7.1, M7.2, M7.3, M7.6, M7.7, M7.8, M7.9 |
|
||||
|
||||
## Goal
|
||||
|
||||
Prove the connector framework composes: two different connector kinds sync
|
||||
through the same framework, change detection skips unchanged documents, tombstoning
|
||||
works, drift reports are accurate, rebuild parity holds, and the gated loop's
|
||||
update-rate is untouched.
|
||||
|
||||
## What the gate proves
|
||||
|
||||
1. **Extensibility works.** Two different connector kinds (at minimum: obsidian +
|
||||
one remote connector) register, sync, and produce queryable Level R content
|
||||
through the same `SyncEngine`. No connector-specific code exists outside the
|
||||
connector itself.
|
||||
|
||||
2. **Change detection is efficient.** Re-syncing an unchanged connector produces
|
||||
zero embedding calls. This is the cost guard — without it, every sync is a
|
||||
full re-embed.
|
||||
|
||||
3. **Tombstoning is correct.** Removing a document from a source results in
|
||||
tombstone records in the log, removal from the index, and correct behavior
|
||||
on rebuild.
|
||||
|
||||
4. **Drift report is read-only.** Running `mem source status` mutates nothing.
|
||||
|
||||
5. **Rebuild parity holds for connectors.** `mem rebuild --from-log` with
|
||||
connector-sourced data produces byte-identical results.
|
||||
|
||||
6. **Update-rate is untouched.** Adding connector-sourced reference documents
|
||||
does not change the gated loop's update-rate (same property M3.6.6 asserts,
|
||||
now for any connector).
|
||||
|
||||
7. **Health monitoring detects failures.** An unreachable connector is reported,
|
||||
not silently ignored.
|
||||
|
||||
## Verify
|
||||
|
||||
**Integration test** — `tests/it_m7_gate.rs`:
|
||||
|
||||
1. `a1_two_kinds_sync` — register an obsidian + vec connector; sync both; assert
|
||||
Level R nodes exist for both sources.
|
||||
2. `a2_unchanged_zero_embeds` — re-sync both; assert zero embedding calls.
|
||||
3. `a3_change_detected_and_replaced` — modify a doc in one connector; sync;
|
||||
assert old chunks tombstoned, new chunks present.
|
||||
4. `a4_removal_tombstoned` — remove a doc; sync; assert tombstone records and
|
||||
doc absent from query results.
|
||||
5. `a5_drift_report_is_read_only` — snapshot log + manifest; run status; assert
|
||||
unchanged.
|
||||
6. `a6_rebuild_parity` — after full sync, rebuild from log; assert byte-identical
|
||||
state.
|
||||
7. `a7_update_rate_untouched` — record update-rate before adding connectors;
|
||||
add connectors + sync; re-run gated loop; assert update-rate unchanged.
|
||||
8. `a8_health_failure_reported` — configure unreachable connector; assert
|
||||
health check reports failure.
|
||||
9. `a9_no_connector_specific_code_in_sync` — assert `SyncEngine` has no
|
||||
`match kind` or `if kind ==` statements (the trait is the dispatch, not
|
||||
the framework).
|
||||
10. `a10_query_returns_connector_content` — sync a connector; query its content;
|
||||
assert results include connector-sourced Level R nodes with correct source_uri.
|
||||
|
||||
**Command:** `cargo test --test it_m7_gate`
|
||||
|
||||
**False pass:**
|
||||
- Testing with only one connector kind. The gate's value is proving two
|
||||
*different* kinds work through the same framework.
|
||||
- Testing rebuild parity without connector data in the log. An empty log
|
||||
trivially rebuilds.
|
||||
- Asserting update-rate "is still below 30%" instead of "is unchanged". Adding
|
||||
reference docs should not move the number at all.
|
||||
|
||||
## Traps
|
||||
|
||||
- Running the gate before M7.6 (sync framework) is solid. The gate tests
|
||||
composition; if the sync framework has bugs, every gate assertion fails
|
||||
for the wrong reason.
|
||||
- Not testing with a connector that produces multiple chunks per document.
|
||||
Single-chunk documents hide change-detection bugs at the chunk level.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — source connectors, composition gates
|
||||
@@ -0,0 +1,97 @@
|
||||
# M7.2 — Obsidian vault connector
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M7 — Source connectors |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M7.10 |
|
||||
| Depends | M7.1, M3.6.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Wrap the existing `DocCorpusSource` (M3.6.1) in the `SourceConnector` interface
|
||||
so the Obsidian vault is managed through the unified connector framework — with
|
||||
change detection, config-driven setup, and registry integration.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
The Obsidian vault is the **primary reference source** for the cluster. It is
|
||||
human-maintained, optionally git-backed, and contains runbooks, procedures, and
|
||||
domain knowledge that agents query through the memory service.
|
||||
|
||||
`DocCorpusSource` already handles markdown parsing, heading-boundary chunking,
|
||||
breadcrumb paths, and file filtering. This task wraps it, not rewrites it.
|
||||
|
||||
**Deployment models:**
|
||||
1. **Git-sync sidecar** — a sidecar container clones the vault repo into a shared
|
||||
PVC. Memory service reads from the PVC via this connector.
|
||||
2. **Local mount** — for development, mount the vault directory directly.
|
||||
3. **PVC direct** — vault files managed via kubectl cp or a web uploader.
|
||||
|
||||
**Configuration:**
|
||||
```yaml
|
||||
connectors:
|
||||
- kind: obsidian
|
||||
name: homelab-vault
|
||||
config:
|
||||
root: /data/vault
|
||||
extensions: [md, markdown, txt]
|
||||
exclude_dirs: [.obsidian, .trash, .git]
|
||||
max_file_size: 10485760 # 10MB
|
||||
```
|
||||
|
||||
## Steps
|
||||
|
||||
1. Implement `ObsidianConnector` in `mem-ingest/src/connectors/obsidian.rs`.
|
||||
2. `list_documents()` — walk `root` directory, filter by extension, compute
|
||||
sha256 per file, return `SourceDocument` per file.
|
||||
3. `fetch_document()` — read file content, return as `DocumentContent` with
|
||||
metadata (file path, last modified, size).
|
||||
4. `health_check()` — verify `root` exists, is readable, count files.
|
||||
5. Register `"obsidian"` kind in the connector registry factory.
|
||||
6. `source_type()` returns `Reference` (vault docs bypass the gated loop).
|
||||
7. Reuse `DocCorpusSource` internals for heading-based chunking when the sync
|
||||
framework (M7.6) processes this connector's documents.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- `ObsidianConnector` implements `SourceConnector` fully.
|
||||
- `list_documents()` respects `extensions`, `exclude_dirs`, `max_file_size`.
|
||||
- `fetch_document()` returns content matching file on disk.
|
||||
- `health_check()` distinguishes readable vs. missing root directory.
|
||||
- Config-driven: changing `root` path in YAML changes what gets scanned.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** fixture directory with markdown files, config YAML.
|
||||
|
||||
**Integration test** — `tests/it_obsidian_connector.rs`:
|
||||
1. `a1_list_filters_extensions` — fixture with .md, .txt, .json; assert only
|
||||
.md and .txt are listed.
|
||||
2. `a2_list_excludes_dirs` — fixture with `.obsidian/` subdir; assert its files
|
||||
are excluded.
|
||||
3. `a3_fetch_returns_content` — fetch a known doc; assert text matches file.
|
||||
4. `a4_fetch_unknown_errors` — fetch non-existent doc_id; assert error.
|
||||
5. `a5_health_check_reachable` — valid root; assert `reachable: true` with count.
|
||||
6. `a6_health_check_missing_root` — non-existent root; assert `reachable: false`.
|
||||
7. `a7_content_hash_stable` — fetch same file twice; assert same hash.
|
||||
8. `a8_config_from_yaml` — parse connector from YAML; assert fields match.
|
||||
|
||||
**Command:** `cargo test --test it_obsidian_connector`
|
||||
|
||||
**False pass:**
|
||||
- Testing with an empty directory. Assertions 1–3 need real files.
|
||||
- Not testing `exclude_dirs` with nested paths (`.obsidian/plugins/x.md`).
|
||||
|
||||
## Traps
|
||||
|
||||
- Re-implementing markdown parsing instead of delegating to `DocCorpusSource`.
|
||||
- Making `doc_id` platform-dependent (use relative path from root, unix separators).
|
||||
- Ignoring symlinks — Obsidian uses them for multi-vault setups.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — source connectors, Obsidian vault as connector
|
||||
@@ -0,0 +1,119 @@
|
||||
# M7.3 — paperless-ngx connector
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M7 — Source connectors |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M7.10 |
|
||||
| Depends | M7.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Implement a `SourceConnector` for paperless-ngx so OCR'd documents, manuals, and
|
||||
reference PDFs already stored in the cluster's paperless instance become
|
||||
searchable through the memory service without manual export.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
paperless-ngx is already running in the cluster. Its REST API provides:
|
||||
|
||||
- `GET /api/documents/` — paginated list with filtering by tags, document type,
|
||||
correspondent, dates.
|
||||
- `GET /api/documents/{id}/` — full metadata including `content` (extracted text).
|
||||
- `GET /api/documents/{id}/download/` — original file.
|
||||
- `GET /api/documents/{id}/preview/` — thumbnail.
|
||||
- Authentication via `Authorization: Token <token>` header.
|
||||
- Documents have `checksum` field (sha256 of original file).
|
||||
|
||||
**Tag filtering is the selection mechanism.** Not every scanned receipt belongs
|
||||
in the knowledge base. Config specifies which tags to include:
|
||||
```yaml
|
||||
connectors:
|
||||
- kind: paperless
|
||||
name: homelab-paperless
|
||||
config:
|
||||
base_url: http://paperless-ngx.paperless.svc.cluster.local:8000
|
||||
token_secret: paperless-api-token # k8s secret ref
|
||||
tags: [reference, manual, runbook] # only sync these
|
||||
format: text # use extracted text content
|
||||
page_size: 100 # API pagination size
|
||||
```
|
||||
|
||||
**Content comes as extracted text.** paperless-ngx OCRs documents on import and
|
||||
stores the text in the `content` field. Use this directly — no PDF parsing needed
|
||||
in the connector. The text quality depends on paperless's OCR config.
|
||||
|
||||
**Checksum enables cheap change detection.** paperless provides `checksum` per
|
||||
document. The sync framework (M7.6) compares this against the last-known hash
|
||||
to skip unchanged documents.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Implement `PaperlessConnector` in `mem-ingest/src/connectors/paperless.rs`.
|
||||
2. `list_documents()` — paginate `GET /api/documents/?tags__name__in=...`,
|
||||
extract `id`, `title`, `checksum`, `modified` for each.
|
||||
3. `fetch_document()` — `GET /api/documents/{id}/`, extract `content` field,
|
||||
return with metadata (title, tags, correspondent, date_created).
|
||||
4. `health_check()` — `GET /api/` and verify 200; report document count from
|
||||
`GET /api/documents/?tags__name__in=...&page=1&page_size=1` (read `count`).
|
||||
5. Handle pagination (paperless returns `next` URL for subsequent pages).
|
||||
6. Token auth from k8s secret (resolve `token_secret` to actual token value).
|
||||
7. Register `"paperless"` kind in connector registry factory.
|
||||
8. `source_type()` returns `Reference`.
|
||||
9. Rate limit API calls (configurable, default 10 req/s) to avoid overloading
|
||||
the paperless instance.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- `PaperlessConnector` implements `SourceConnector` fully.
|
||||
- Tag filtering limits which documents are listed.
|
||||
- Pagination handles > 100 documents correctly.
|
||||
- `content_hash` uses paperless's `checksum` field for change detection.
|
||||
- Auth token resolved from k8s secret (not hardcoded).
|
||||
- Health check reports document count matching tag filter.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** mock HTTP server (wiremock or similar) returning paperless API
|
||||
responses; fixture JSON responses for list/detail endpoints.
|
||||
|
||||
**Integration test** — `tests/it_paperless_connector.rs`:
|
||||
1. `a1_list_filters_by_tags` — mock returns 5 docs, 3 with matching tags; assert
|
||||
`list_documents()` returns 3.
|
||||
2. `a2_pagination_fetches_all` — mock returns 2 pages of 50; assert 100 docs.
|
||||
3. `a3_fetch_returns_content` — mock detail endpoint; assert text matches fixture.
|
||||
4. `a4_fetch_includes_metadata` — assert returned metadata includes title, tags,
|
||||
correspondent, date fields.
|
||||
5. `a5_health_check_reachable` — mock 200; assert `reachable: true` with count.
|
||||
6. `a6_health_check_unreachable` — mock connection refused; assert
|
||||
`reachable: false` with error message.
|
||||
7. `a7_checksum_as_content_hash` — assert `SourceDocument.content_hash` is
|
||||
populated from paperless `checksum` field.
|
||||
8. `a8_auth_header_sent` — assert mock received `Authorization: Token <value>`.
|
||||
9. `a9_config_from_yaml` — parse connector from YAML fixture; assert fields match.
|
||||
10. `a10_live` — `#[ignore]`; real paperless instance; list + fetch one doc; print
|
||||
title and content length for human sanity check.
|
||||
|
||||
**Command:** `cargo test --test it_paperless_connector` (add `-- --ignored` for a10)
|
||||
|
||||
**False pass:**
|
||||
- Mocking without verifying auth header. A connector that works in tests but
|
||||
sends no auth fails silently against real paperless.
|
||||
- Testing single page only. Pagination bugs are invisible with < `page_size` docs.
|
||||
|
||||
## Traps
|
||||
|
||||
- Assuming `content` is always populated. paperless may have documents without
|
||||
OCR text (e.g., empty scans). Return empty content with a warning, don't panic.
|
||||
- Hardcoding the base URL without trailing-slash normalization. `/api/documents/`
|
||||
vs `/api/documents` behaves differently.
|
||||
- Not handling paperless API rate limits (429 responses). Add retry-after logic.
|
||||
- Resolving k8s secrets at config parse time. Defer to runtime — secret may not
|
||||
exist in dev/test environments. Use env var fallback.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — source connectors, paperless-ngx integration
|
||||
@@ -0,0 +1,118 @@
|
||||
# M7.4 — Git repository connector
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M7 — Source connectors |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M7.10 |
|
||||
| Depends | M7.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Implement a `SourceConnector` that clones or pulls a git repository and exposes
|
||||
its documents for ingestion — so documentation repos, wikis, and runbook
|
||||
collections tracked in git become searchable through the memory service.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
Many knowledge sources are already in git: internal wikis, Forgejo repos,
|
||||
infrastructure documentation, README collections. This connector pulls them
|
||||
without requiring manual export.
|
||||
|
||||
**Configuration:**
|
||||
```yaml
|
||||
connectors:
|
||||
- kind: git_repo
|
||||
name: infra-docs
|
||||
config:
|
||||
repo_url: https://forgejo.riotpiao.com/rock/homelab-docs.git
|
||||
branch: main
|
||||
clone_dir: /tmp/mem-connectors/infra-docs # local checkout
|
||||
paths: [docs/, runbooks/] # only scan these dirs
|
||||
extensions: [md, txt, rst]
|
||||
auth_secret: forgejo-token # k8s secret for private repos
|
||||
sync_depth: 1 # shallow clone
|
||||
```
|
||||
|
||||
**Clone-then-walk model.** The connector clones (or pulls) the repo to a local
|
||||
directory, then walks the filesystem like the Obsidian connector. This reuses
|
||||
`DocCorpusSource` internals and avoids git-specific content access APIs.
|
||||
|
||||
**Change detection uses git.** `git diff --name-only HEAD@{1}..HEAD` after a pull
|
||||
tells the sync framework exactly which files changed — more efficient than
|
||||
re-hashing every file.
|
||||
|
||||
**Branch tracking.** Connector watches one branch. Branch changes (main → prod)
|
||||
require config update. No multi-branch support — each branch is a separate
|
||||
connector instance.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Implement `GitRepoConnector` in `mem-ingest/src/connectors/git_repo.rs`.
|
||||
2. On first sync: `git clone --depth N --branch B <url> <clone_dir>`.
|
||||
3. On subsequent syncs: `git -C <clone_dir> pull --ff-only`.
|
||||
4. `list_documents()` — walk `clone_dir` filtered by `paths` and `extensions`,
|
||||
compute sha256 per file.
|
||||
5. `fetch_document()` — read file from `clone_dir`, return content with git
|
||||
metadata (last commit sha, author, date for that file via `git log -1`).
|
||||
6. `health_check()` — verify `clone_dir` is a valid git repo, check remote
|
||||
connectivity via `git ls-remote`.
|
||||
7. Handle auth for private repos (token from k8s secret → git credential helper
|
||||
or URL embedding).
|
||||
8. Register `"git_repo"` kind in connector registry factory.
|
||||
9. `source_type()` returns `Reference`.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Clone + walk produces correct document list for fixture repo.
|
||||
- Pull detects changed files without re-processing unchanged ones.
|
||||
- Path filtering limits scan to configured subdirectories.
|
||||
- Private repo auth works (token in URL or credential helper).
|
||||
- Health check distinguishes: valid repo, invalid remote, auth failure.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** fixture git repo created in temp dir with known content.
|
||||
|
||||
**Integration test** — `tests/it_git_repo_connector.rs`:
|
||||
1. `a1_clone_and_list` — init fixture repo, connector clones it; assert
|
||||
`list_documents()` returns expected files.
|
||||
2. `a2_path_filtering` — fixture with `docs/` and `src/`; config says `paths: [docs/]`;
|
||||
assert only `docs/` files returned.
|
||||
3. `a3_extension_filtering` — fixture with .md, .rs, .txt; assert only .md/.txt.
|
||||
4. `a4_fetch_returns_content` — fetch a doc; assert content matches fixture file.
|
||||
5. `a5_pull_detects_changes` — add a commit to fixture repo; pull; assert changed
|
||||
file appears in list with new hash.
|
||||
6. `a6_health_check_valid_repo` — valid clone_dir; assert `reachable: true`.
|
||||
7. `a7_health_check_no_clone` — no clone_dir; assert `reachable: false` with
|
||||
message indicating clone needed.
|
||||
8. `a8_shallow_clone` — assert clone depth matches config (`git rev-list --count HEAD`).
|
||||
9. `a9_config_from_yaml` — parse connector from YAML; assert fields match.
|
||||
|
||||
**Command:** `cargo test --test it_git_repo_connector`
|
||||
|
||||
**False pass:**
|
||||
- Testing with a local repo path instead of a clone. The clone/pull machinery
|
||||
is the whole point — a connector that reads a pre-existing checkout is just
|
||||
the Obsidian connector.
|
||||
- Not testing pull-after-change. First sync always works; second sync is where
|
||||
change detection matters.
|
||||
|
||||
## Traps
|
||||
|
||||
- Running `git clone` on every sync. Check if `clone_dir` already has a valid
|
||||
checkout first; clone only on first run.
|
||||
- Not cleaning up failed clones. A partial clone leaves a directory that is
|
||||
neither valid nor absent — subsequent runs fail on both clone (dir exists)
|
||||
and pull (not a repo).
|
||||
- Force-push upstream breaks `--ff-only`. Detect non-fast-forward, delete
|
||||
`clone_dir`, re-clone. Log a warning — this means all docs are re-processed.
|
||||
- Symlinks across the repo boundary. `walkdir` follows symlinks by default;
|
||||
a symlink to `/etc/passwd` is a real concern in a cluster-wide service.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — source connectors section
|
||||
@@ -0,0 +1,109 @@
|
||||
# M7.5 — S3-compatible storage connector
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M7 — Source connectors |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M7.10 |
|
||||
| Depends | M7.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Implement a `SourceConnector` for S3-compatible object storage (MinIO, AWS S3,
|
||||
R2, etc.) so documents stored in buckets become searchable through the memory
|
||||
service.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
S3 is the universal storage protocol. MinIO runs in many homelabs, and cloud
|
||||
providers expose the same API. This connector makes any S3 bucket a knowledge
|
||||
source.
|
||||
|
||||
**Configuration:**
|
||||
```yaml
|
||||
connectors:
|
||||
- kind: s3
|
||||
name: knowledge-bucket
|
||||
config:
|
||||
endpoint: https://minio.riotpiao.com
|
||||
bucket: knowledge-base
|
||||
prefix: docs/ # only this prefix
|
||||
extensions: [md, txt, pdf] # filter by key suffix
|
||||
access_key_secret: minio-creds # k8s secret with access/secret keys
|
||||
region: us-east-1 # for AWS; ignored by MinIO
|
||||
```
|
||||
|
||||
**ETag for change detection.** S3 objects have ETags (usually MD5 of content).
|
||||
Use this as `content_hash` in `SourceDocument` — the sync framework skips objects
|
||||
whose ETag hasn't changed.
|
||||
|
||||
**Text extraction.** S3 stores raw files. Markdown and text files are read
|
||||
directly. PDF/DOCX support is out of scope for M7.5 — those MIME types are
|
||||
skipped with a warning. Future: add a text extraction layer or require pre-processed
|
||||
text.
|
||||
|
||||
**Pagination via continuation tokens.** S3 ListObjectsV2 returns max 1000 keys
|
||||
per request. Use `ContinuationToken` for subsequent pages.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Implement `S3Connector` in `mem-ingest/src/connectors/s3.rs`.
|
||||
2. `list_documents()` — `ListObjectsV2` with `Prefix`, paginate, filter by
|
||||
extension, return `SourceDocument` per object.
|
||||
3. `fetch_document()` — `GetObject`, read body as text (UTF-8), return with
|
||||
metadata (key, size, last_modified, ETag).
|
||||
4. `health_check()` — `HeadBucket` to verify access.
|
||||
5. Auth via access key + secret key from k8s secret.
|
||||
6. Use `aws-sdk-s3` or `rust-s3` crate for S3 API.
|
||||
7. Register `"s3"` kind in connector registry factory.
|
||||
8. `source_type()` returns `Reference`.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- `S3Connector` implements `SourceConnector` fully.
|
||||
- Prefix filtering limits to configured path.
|
||||
- Extension filtering skips non-text objects.
|
||||
- ETag is used as `content_hash` for change detection.
|
||||
- Pagination handles > 1000 objects.
|
||||
- Auth works with MinIO and AWS-style credentials.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** mock S3 server (localstack or in-process mock) with fixture objects.
|
||||
|
||||
**Integration test** — `tests/it_s3_connector.rs`:
|
||||
1. `a1_list_with_prefix` — mock bucket with objects under `docs/` and `images/`;
|
||||
assert only `docs/` objects listed.
|
||||
2. `a2_extension_filtering` — mock with .md, .png, .txt; assert .png excluded.
|
||||
3. `a3_fetch_returns_content` — fetch a .md object; assert content matches.
|
||||
4. `a4_etag_as_content_hash` — assert `SourceDocument.content_hash` equals
|
||||
the object's ETag.
|
||||
5. `a5_pagination` — mock 1500 objects; assert all listed via continuation tokens.
|
||||
6. `a6_health_check_valid_bucket` — mock HeadBucket 200; assert reachable.
|
||||
7. `a7_health_check_no_access` — mock HeadBucket 403; assert not reachable with
|
||||
error message.
|
||||
8. `a8_non_utf8_skipped` — mock object with binary content; assert skipped with
|
||||
warning, not crash.
|
||||
9. `a9_config_from_yaml` — parse connector from YAML; assert fields match.
|
||||
|
||||
**Command:** `cargo test --test it_s3_connector`
|
||||
|
||||
**False pass:**
|
||||
- Testing with a local filesystem mock instead of S3 API mock. The pagination
|
||||
and ETag handling are S3-specific.
|
||||
|
||||
## Traps
|
||||
|
||||
- Assuming ETags are always MD5. Multipart uploads produce composite ETags
|
||||
(`hash-N`). These are still unique per version — use as-is for change detection.
|
||||
- Not handling `NoSuchBucket` vs `AccessDenied`. Both are errors but mean
|
||||
different things for health reporting.
|
||||
- Reading binary files as UTF-8. A JPEG read as text produces garbage. Check
|
||||
content-type header and skip non-text MIME types.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — source connectors section
|
||||
@@ -0,0 +1,146 @@
|
||||
# M7.6 — Sync framework
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M7 — Source connectors |
|
||||
| Size | L — 3–5 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M7.7, M7.10 |
|
||||
| Depends | M7.1, M3.6.3 |
|
||||
|
||||
## Goal
|
||||
|
||||
Build the shared sync engine that handles change detection, tombstoning, drift
|
||||
reporting, and resumable sync for **all** connectors — so individual connector
|
||||
implementations only fetch documents and everything else is handled once.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
Every connector faces the same sync problems:
|
||||
1. **What changed?** Compare content hashes from `list_documents()` against a
|
||||
manifest of last-known hashes. Only fetch and process changed documents.
|
||||
2. **What disappeared?** Documents in the manifest but not in `list_documents()`
|
||||
need tombstone records in the log. Append-only — never delete.
|
||||
3. **What if sync crashes midway?** Track progress per-document. Restart
|
||||
processes only unseen documents.
|
||||
4. **How much will this cost?** Drift report shows counts without mutating
|
||||
anything.
|
||||
|
||||
This is the generalization of M3.6.3's `mem ref sync/list/rm` logic, applied to
|
||||
any `SourceConnector` instead of just `DocCorpusSource`.
|
||||
|
||||
**Manifest storage.** Per-connector manifest in `log/connectors/<name>.manifest.jsonl`:
|
||||
```jsonl
|
||||
{"doc_id":"abc","source_uri":"file:///vault/k8s.md","content_hash":"sha256:...","chunk_count":12,"synced_at":"..."}
|
||||
{"doc_id":"def","source_uri":"paperless://doc/42","content_hash":"sha256:...","chunk_count":3,"synced_at":"..."}
|
||||
```
|
||||
|
||||
**Sync algorithm:**
|
||||
```
|
||||
current = connector.list_documents()
|
||||
previous = load_manifest(connector.name)
|
||||
|
||||
for doc in current:
|
||||
if doc.content_hash == previous[doc.doc_id].content_hash:
|
||||
skip (unchanged)
|
||||
else if doc.doc_id in previous:
|
||||
tombstone previous chunks, fetch + chunk + embed new (changed)
|
||||
else:
|
||||
fetch + chunk + embed (new)
|
||||
|
||||
for doc_id in previous not in current:
|
||||
tombstone all chunks (removed)
|
||||
|
||||
save_manifest(connector.name, current)
|
||||
```
|
||||
|
||||
**Chunking delegation.** The sync framework owns the chunking step. It routes
|
||||
fetched `DocumentContent` through the appropriate `ChunkPolicy`:
|
||||
- Markdown → heading-boundary chunking (reuse `DocCorpusSource` logic)
|
||||
- Plain text → paragraph-boundary chunking
|
||||
- Configurable per connector kind in `connectors.yaml`
|
||||
|
||||
**Level routing.** Session connectors → RecordSource → gated loop (L0/L1/L2).
|
||||
Document connectors → Reference records (Level R). The `source_type()` method
|
||||
on the connector determines the pipeline.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Define `SyncEngine` struct in `mem-ingest/src/sync.rs`.
|
||||
2. Implement manifest loading/saving (JSONL per connector).
|
||||
3. Implement diff algorithm: `(new, changed, unchanged, removed)` from
|
||||
`list_documents()` vs manifest.
|
||||
4. Implement sync loop: for each `new`/`changed` doc, fetch → chunk → emit
|
||||
records. For each `removed` doc, emit tombstones.
|
||||
5. Implement drift reporting: same diff algorithm, print counts, no mutations.
|
||||
6. Implement resume: track synced doc_ids in a progress file. On restart,
|
||||
skip already-synced docs.
|
||||
7. Implement rate limiting: configurable max concurrent fetches per connector.
|
||||
8. Integrate with `mem-store` for writing Reference records to the log.
|
||||
9. Integrate with `mem-llm` for embedding new chunks.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Sync of unchanged connector produces zero embedding calls.
|
||||
- Changed document: old chunks tombstoned, new chunks embedded and stored.
|
||||
- Removed document: chunks tombstoned, manifest updated.
|
||||
- Drift report matches actual changes without mutating anything.
|
||||
- Crash mid-sync → restart processes only remaining documents.
|
||||
- Rate limiting: never exceeds configured concurrent fetch limit.
|
||||
- Log remains append-only (tombstones are records, not deletions).
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** `VecConnector` with mutable document list, counting embedder.
|
||||
|
||||
**Integration test** — `tests/it_sync_framework.rs`:
|
||||
1. `a1_initial_sync_all_new` — 3 docs, no manifest; assert all 3 fetched and
|
||||
embedded, manifest written with 3 entries.
|
||||
2. `a2_unchanged_skipped` — sync again with same content; assert zero fetch
|
||||
calls, zero embed calls.
|
||||
3. `a3_changed_doc_replaced` — modify one doc's content; sync; assert old chunks
|
||||
tombstoned, new chunks embedded, embed count equals changed doc's chunk count.
|
||||
4. `a4_removed_doc_tombstoned` — remove a doc from connector; sync; assert
|
||||
tombstone records emitted, manifest entry removed.
|
||||
5. `a5_new_doc_added` — add a doc to connector; sync; assert only new doc
|
||||
fetched and embedded.
|
||||
6. `a6_drift_report_read_only` — modify docs, run drift report; assert correct
|
||||
counts (1 new, 1 changed, 1 unchanged, 1 removed); assert no mutations to
|
||||
manifest or log.
|
||||
7. `a7_resume_after_crash` — sync 5 docs, simulate crash after 3; restart;
|
||||
assert only 2 remaining docs processed.
|
||||
8. `a8_tombstone_is_append` — count log lines before and after remove; assert
|
||||
count only grew.
|
||||
9. `a9_manifest_roundtrip` — save manifest, load it; assert field-for-field
|
||||
equality.
|
||||
10. `a10_rebuild_parity` — after full sync, `mem rebuild --from-log` produces
|
||||
identical state.
|
||||
|
||||
**Command:** `cargo test --test it_sync_framework`
|
||||
|
||||
**False pass:**
|
||||
- Asserting "no duplicate rows" instead of counting embed calls. A sync that
|
||||
re-embeds everything and upserts by sha produces correct rows and wasted
|
||||
compute.
|
||||
- Testing drift report without actually changing documents first.
|
||||
|
||||
## Traps
|
||||
|
||||
- Comparing document-level hashes instead of chunk-level. A doc that changed
|
||||
one paragraph should re-embed only the affected chunks, not all of them.
|
||||
However, heading-boundary chunking means changing one heading can shift all
|
||||
subsequent chunks. Accept document-level granularity for now; chunk-level
|
||||
optimization is a future refinement.
|
||||
- Making the manifest a database table instead of JSONL. The manifest must
|
||||
survive `mem rebuild --from-log` — it is metadata about the sync process,
|
||||
not a projection of the log.
|
||||
- Running fetch + embed serially. A connector with 500 docs at 200ms per embed
|
||||
takes 100s serially. Concurrent fetch + sequential embed is the right shape.
|
||||
- Ignoring the derived filter (M4.2). Document connectors produce Level R content
|
||||
that must register in the artifact manifest so it cannot re-enter as evidence.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — source connectors, sync framework
|
||||
@@ -0,0 +1,107 @@
|
||||
# M7.7 — `mem source` CLI
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M7 — Source connectors |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M7.8, M7.10 |
|
||||
| Depends | M7.6 |
|
||||
|
||||
## Goal
|
||||
|
||||
Provide CLI commands to manage source connectors: sync documents, check status,
|
||||
list connectors, add/remove connectors.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
```
|
||||
mem source list # all registered connectors + health
|
||||
mem source status # drift report per connector, no mutations
|
||||
mem source sync --all # sync all connectors
|
||||
mem source sync --name homelab-vault # sync one connector
|
||||
mem source sync --name homelab-vault --dry-run # show plan, no mutations
|
||||
mem source add --kind paperless --name docs --config '{"base_url":"...","token":"..."}'
|
||||
mem source rm --name old-source # deregister + tombstone all chunks
|
||||
mem source health # connectivity check per connector
|
||||
```
|
||||
|
||||
**Output format.** CLI outputs human-readable tables by default, `--json` for
|
||||
machine consumption. Example:
|
||||
|
||||
```
|
||||
$ mem source list
|
||||
NAME KIND DOCS LAST SYNC HEALTH
|
||||
homelab-vault obsidian 42 2026-08-26T10:00:00Z ✅ reachable
|
||||
homelab-paperless paperless 127 2026-08-26T09:00:00Z ✅ reachable
|
||||
infra-docs git_repo 18 2026-08-25T20:00:00Z ⚠️ pull failed
|
||||
|
||||
$ mem source status
|
||||
NAME NEW CHANGED UNCHANGED REMOVED
|
||||
homelab-vault 0 2 40 0
|
||||
homelab-paperless 3 0 124 0
|
||||
infra-docs 1 1 16 0
|
||||
```
|
||||
|
||||
## Steps
|
||||
|
||||
1. Add `Source` subcommand group to clap CLI in `mem-cli/src/main.rs`.
|
||||
2. Implement `cmd_source_list()` — load registry, health check each, tabulate.
|
||||
3. Implement `cmd_source_status()` — load registry, run drift report per
|
||||
connector, tabulate.
|
||||
4. Implement `cmd_source_sync()` — load registry, run sync engine for selected
|
||||
connector(s), report results.
|
||||
5. Implement `cmd_source_add()` — validate kind, parse config, register in
|
||||
`connectors.yaml`, run initial health check.
|
||||
6. Implement `cmd_source_rm()` — tombstone all chunks from connector, remove
|
||||
from `connectors.yaml`.
|
||||
7. Implement `cmd_source_health()` — connectivity check per connector.
|
||||
8. Add `--dry-run` to sync (show plan only).
|
||||
9. Add `--json` flag for machine-readable output.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- All subcommands execute without panic.
|
||||
- `sync --dry-run` shows plan without mutations.
|
||||
- `add` validates kind exists in registry before writing config.
|
||||
- `rm` tombstones chunks and removes config entry.
|
||||
- `status` shows drift without mutations.
|
||||
- Exit codes: 0 on success, non-zero on failure.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** `VecConnector` registered in registry, temp connectors.yaml.
|
||||
|
||||
**Integration test** — `tests/it_source_cli.rs`:
|
||||
1. `a1_list_shows_connectors` — register two connectors; assert list output
|
||||
contains both names and kinds.
|
||||
2. `a2_status_shows_drift` — modify connector docs; assert status shows correct
|
||||
new/changed/unchanged/removed counts.
|
||||
3. `a3_sync_processes_changes` — sync with changes; assert documents processed.
|
||||
4. `a4_sync_dry_run_no_mutations` — sync --dry-run; assert no log entries written.
|
||||
5. `a5_add_registers_connector` — add a new connector; assert it appears in list.
|
||||
6. `a6_add_bad_kind_fails` — add with unknown kind; assert non-zero exit.
|
||||
7. `a7_rm_tombstones_and_deregisters` — rm a connector; assert tombstone records
|
||||
written and connector removed from list.
|
||||
8. `a8_health_reports_status` — assert health output includes reachable/unreachable.
|
||||
9. `a9_json_output` — assert --json flag produces valid JSON.
|
||||
|
||||
**Command:** `cargo test --test it_source_cli`
|
||||
|
||||
**False pass:**
|
||||
- Testing `list` without checking that connectors are actually registered (not
|
||||
just config parsed).
|
||||
|
||||
## Traps
|
||||
|
||||
- `sync --all` with a broken connector should not halt all syncs. Sync each
|
||||
independently, report failures per connector at the end.
|
||||
- `rm` without `--yes` should prompt for confirmation (destructive operation).
|
||||
- Don't write `connectors.yaml` atomically — a crash mid-write corrupts config.
|
||||
Write to temp file, then rename.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — source connectors, CLI commands
|
||||
@@ -0,0 +1,107 @@
|
||||
# M7.8 — `GET /memory/sources` + `POST /memory/sources/sync` HTTP endpoints
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M7 — Source connectors |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M7.10 |
|
||||
| Depends | M7.7, M3.5.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Expose source connector management through the HTTP API so agents and UIs can
|
||||
trigger syncs, check connector health, and view status without CLI access.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
**Endpoints:**
|
||||
```
|
||||
GET /memory/sources # list all connectors + health
|
||||
GET /memory/sources/{name} # one connector detail + drift
|
||||
GET /memory/sources/{name}/health # health check only
|
||||
POST /memory/sources/sync # trigger sync (async)
|
||||
POST /memory/sources/{name}/sync # trigger sync for one connector
|
||||
GET /memory/sources/sync/{job_id} # sync job status
|
||||
```
|
||||
|
||||
**Response shapes:**
|
||||
```json
|
||||
// GET /memory/sources
|
||||
[
|
||||
{
|
||||
"name": "homelab-vault",
|
||||
"kind": "obsidian",
|
||||
"document_count": 42,
|
||||
"last_sync": "2026-08-26T10:00:00Z",
|
||||
"health": { "reachable": true, "document_count": 42 }
|
||||
}
|
||||
]
|
||||
|
||||
// POST /memory/sources/sync
|
||||
// Request: { "names": ["homelab-vault"] } (or omit for all)
|
||||
// Response: 202 Accepted
|
||||
{ "job_id": "sync-abc123", "status_url": "/memory/sources/sync/sync-abc123" }
|
||||
|
||||
// GET /memory/sources/sync/{job_id}
|
||||
{
|
||||
"job_id": "sync-abc123",
|
||||
"status": "completed",
|
||||
"connectors": {
|
||||
"homelab-vault": { "new": 0, "changed": 2, "unchanged": 40, "removed": 0 }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Sync is async.** POST returns 202 immediately; client polls status endpoint.
|
||||
Same pattern as `/memory/ingest`.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Add routes to `http_server.rs` for all six endpoints.
|
||||
2. `sources_list_handler` — load registry, health check each, return JSON array.
|
||||
3. `source_detail_handler` — one connector, include drift report.
|
||||
4. `source_health_handler` — health check only.
|
||||
5. `source_sync_handler` — validate connector names, spawn async sync job,
|
||||
return job_id + status URL.
|
||||
6. `source_sync_status_handler` — look up job by id, return progress.
|
||||
7. Auth: all endpoints require apikey header (existing pattern).
|
||||
8. Rate limiting: sync endpoint limited to 10 req/hour (it's expensive).
|
||||
|
||||
## Acceptance
|
||||
|
||||
- All endpoints return correct status codes and JSON shapes.
|
||||
- Sync is async — POST returns immediately, job runs in background.
|
||||
- Unknown connector name returns 404.
|
||||
- Auth required on all endpoints.
|
||||
- Rate limiting on sync endpoint.
|
||||
|
||||
## Verify
|
||||
|
||||
**Integration test** — `tests/it_source_http.rs`:
|
||||
1. `a1_list_returns_connectors` — register connectors, GET /memory/sources;
|
||||
assert JSON array with correct fields.
|
||||
2. `a2_detail_includes_drift` — GET /memory/sources/{name}; assert drift fields.
|
||||
3. `a3_health_check_via_http` — GET /memory/sources/{name}/health; assert
|
||||
reachable field.
|
||||
4. `a4_sync_returns_202` — POST /memory/sources/sync; assert 202 + job_id.
|
||||
5. `a5_sync_status_tracks_progress` — poll status endpoint; assert eventually
|
||||
"completed".
|
||||
6. `a6_unknown_connector_404` — GET /memory/sources/nonexistent; assert 404.
|
||||
7. `a7_auth_required` — request without apikey; assert 401.
|
||||
8. `a8_sync_rate_limited` — 11 sync requests; assert 429 on the 11th.
|
||||
|
||||
**Command:** `cargo test --test it_source_http`
|
||||
|
||||
## Traps
|
||||
|
||||
- Making sync synchronous. A paperless connector with 500 docs takes minutes;
|
||||
blocking the HTTP response is a client timeout.
|
||||
- Not capping concurrent sync jobs. Two simultaneous syncs to the same connector
|
||||
can corrupt the manifest.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — source connectors, HTTP endpoints
|
||||
@@ -0,0 +1,76 @@
|
||||
# M7.9 — Connector health monitoring + observability
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M7 — Source connectors |
|
||||
| Size | S — < 1 day |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M7.10 |
|
||||
| Depends | M7.8 |
|
||||
|
||||
## Goal
|
||||
|
||||
Add periodic health checks, sync metrics, and alerting hooks for source
|
||||
connectors so connector failures are detected before knowledge goes stale.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
A connector that silently fails means the knowledge base is stale with no
|
||||
indication. The monitoring layer detects this.
|
||||
|
||||
**Metrics (Prometheus-compatible):**
|
||||
```
|
||||
mem_source_health{name="homelab-vault",kind="obsidian"} 1 # 1=healthy, 0=unhealthy
|
||||
mem_source_last_sync_seconds{name="homelab-vault"} 1724680000 # unix timestamp
|
||||
mem_source_documents_total{name="homelab-vault"} 42
|
||||
mem_source_sync_duration_seconds{name="homelab-vault"} 12.3
|
||||
mem_source_sync_errors_total{name="homelab-vault"} 0
|
||||
mem_source_drift_new{name="homelab-vault"} 2 # docs pending sync
|
||||
mem_source_drift_changed{name="homelab-vault"} 1
|
||||
```
|
||||
|
||||
**Periodic health check.** Configurable interval (default 5 minutes). Log
|
||||
warnings for unreachable connectors. Update Prometheus gauges.
|
||||
|
||||
**Staleness alert.** If `last_sync` exceeds a configurable threshold (default 24h),
|
||||
log a warning and set a metric. This is the "knowledge is going stale" signal.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Add health check background task (tokio interval, configurable period).
|
||||
2. Export Prometheus metrics via `/metrics` endpoint (existing pattern or new).
|
||||
3. Track per-connector: health, last sync, doc count, sync duration, errors.
|
||||
4. Staleness detection: compare `last_sync` to now, warn if exceeds threshold.
|
||||
5. Log structured health events for observability.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Periodic health checks run at configured interval.
|
||||
- Metrics endpoint returns valid Prometheus format.
|
||||
- Staleness warning fires when last_sync exceeds threshold.
|
||||
- Unhealthy connector logged with error details.
|
||||
|
||||
## Verify
|
||||
|
||||
**Integration test** — `tests/it_source_monitoring.rs`:
|
||||
1. `a1_health_metric_updated` — register connector, wait for health check;
|
||||
assert metric value matches connector health.
|
||||
2. `a2_staleness_detected` — set last_sync to 25h ago; assert staleness warning.
|
||||
3. `a3_metrics_format_valid` — GET /metrics; assert valid Prometheus text format.
|
||||
4. `a4_error_count_incremented` — force connector health check failure; assert
|
||||
error counter incremented.
|
||||
|
||||
**Command:** `cargo test --test it_source_monitoring`
|
||||
|
||||
## Traps
|
||||
|
||||
- Running health checks synchronously. A connector that times out blocks all
|
||||
other connectors' health checks. Use tokio::select with timeout.
|
||||
- Not distinguishing "unreachable" from "empty". A connector that returns 0 docs
|
||||
is healthy; a connector that can't connect is not.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — source connectors, observability
|
||||
Reference in New Issue
Block a user