docs: Add M7 source connectors (10 tasks), M3.5.10 auth integration, remove Kong refs
- M7.1-M7.10: Extensible SourceConnector trait, Obsidian/paperless/git/S3 connectors, sync framework, CLI, HTTP endpoints, health monitoring, gate - M3.5.10: Auth integration with Authentik OIDC → Vault token validation - DESIGN.md: Add source connectors architecture, update auth to Authentik/Vault (Kong removed from cluster) - INDEX.md: 75 tasks, 11 gates - Fix all Kong references in M3.5.1 task
This commit is contained in:
@@ -0,0 +1,119 @@
|
||||
# M7.3 — paperless-ngx connector
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M7 — Source connectors |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M7.10 |
|
||||
| Depends | M7.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Implement a `SourceConnector` for paperless-ngx so OCR'd documents, manuals, and
|
||||
reference PDFs already stored in the cluster's paperless instance become
|
||||
searchable through the memory service without manual export.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
paperless-ngx is already running in the cluster. Its REST API provides:
|
||||
|
||||
- `GET /api/documents/` — paginated list with filtering by tags, document type,
|
||||
correspondent, dates.
|
||||
- `GET /api/documents/{id}/` — full metadata including `content` (extracted text).
|
||||
- `GET /api/documents/{id}/download/` — original file.
|
||||
- `GET /api/documents/{id}/preview/` — thumbnail.
|
||||
- Authentication via `Authorization: Token <token>` header.
|
||||
- Documents have `checksum` field (sha256 of original file).
|
||||
|
||||
**Tag filtering is the selection mechanism.** Not every scanned receipt belongs
|
||||
in the knowledge base. Config specifies which tags to include:
|
||||
```yaml
|
||||
connectors:
|
||||
- kind: paperless
|
||||
name: homelab-paperless
|
||||
config:
|
||||
base_url: http://paperless-ngx.paperless.svc.cluster.local:8000
|
||||
token_secret: paperless-api-token # k8s secret ref
|
||||
tags: [reference, manual, runbook] # only sync these
|
||||
format: text # use extracted text content
|
||||
page_size: 100 # API pagination size
|
||||
```
|
||||
|
||||
**Content comes as extracted text.** paperless-ngx OCRs documents on import and
|
||||
stores the text in the `content` field. Use this directly — no PDF parsing needed
|
||||
in the connector. The text quality depends on paperless's OCR config.
|
||||
|
||||
**Checksum enables cheap change detection.** paperless provides `checksum` per
|
||||
document. The sync framework (M7.6) compares this against the last-known hash
|
||||
to skip unchanged documents.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Implement `PaperlessConnector` in `mem-ingest/src/connectors/paperless.rs`.
|
||||
2. `list_documents()` — paginate `GET /api/documents/?tags__name__in=...`,
|
||||
extract `id`, `title`, `checksum`, `modified` for each.
|
||||
3. `fetch_document()` — `GET /api/documents/{id}/`, extract `content` field,
|
||||
return with metadata (title, tags, correspondent, date_created).
|
||||
4. `health_check()` — `GET /api/` and verify 200; report document count from
|
||||
`GET /api/documents/?tags__name__in=...&page=1&page_size=1` (read `count`).
|
||||
5. Handle pagination (paperless returns `next` URL for subsequent pages).
|
||||
6. Token auth from k8s secret (resolve `token_secret` to actual token value).
|
||||
7. Register `"paperless"` kind in connector registry factory.
|
||||
8. `source_type()` returns `Reference`.
|
||||
9. Rate limit API calls (configurable, default 10 req/s) to avoid overloading
|
||||
the paperless instance.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- `PaperlessConnector` implements `SourceConnector` fully.
|
||||
- Tag filtering limits which documents are listed.
|
||||
- Pagination handles > 100 documents correctly.
|
||||
- `content_hash` uses paperless's `checksum` field for change detection.
|
||||
- Auth token resolved from k8s secret (not hardcoded).
|
||||
- Health check reports document count matching tag filter.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** mock HTTP server (wiremock or similar) returning paperless API
|
||||
responses; fixture JSON responses for list/detail endpoints.
|
||||
|
||||
**Integration test** — `tests/it_paperless_connector.rs`:
|
||||
1. `a1_list_filters_by_tags` — mock returns 5 docs, 3 with matching tags; assert
|
||||
`list_documents()` returns 3.
|
||||
2. `a2_pagination_fetches_all` — mock returns 2 pages of 50; assert 100 docs.
|
||||
3. `a3_fetch_returns_content` — mock detail endpoint; assert text matches fixture.
|
||||
4. `a4_fetch_includes_metadata` — assert returned metadata includes title, tags,
|
||||
correspondent, date fields.
|
||||
5. `a5_health_check_reachable` — mock 200; assert `reachable: true` with count.
|
||||
6. `a6_health_check_unreachable` — mock connection refused; assert
|
||||
`reachable: false` with error message.
|
||||
7. `a7_checksum_as_content_hash` — assert `SourceDocument.content_hash` is
|
||||
populated from paperless `checksum` field.
|
||||
8. `a8_auth_header_sent` — assert mock received `Authorization: Token <value>`.
|
||||
9. `a9_config_from_yaml` — parse connector from YAML fixture; assert fields match.
|
||||
10. `a10_live` — `#[ignore]`; real paperless instance; list + fetch one doc; print
|
||||
title and content length for human sanity check.
|
||||
|
||||
**Command:** `cargo test --test it_paperless_connector` (add `-- --ignored` for a10)
|
||||
|
||||
**False pass:**
|
||||
- Mocking without verifying auth header. A connector that works in tests but
|
||||
sends no auth fails silently against real paperless.
|
||||
- Testing single page only. Pagination bugs are invisible with < `page_size` docs.
|
||||
|
||||
## Traps
|
||||
|
||||
- Assuming `content` is always populated. paperless may have documents without
|
||||
OCR text (e.g., empty scans). Return empty content with a warning, don't panic.
|
||||
- Hardcoding the base URL without trailing-slash normalization. `/api/documents/`
|
||||
vs `/api/documents` behaves differently.
|
||||
- Not handling paperless API rate limits (429 responses). Add retry-after logic.
|
||||
- Resolving k8s secrets at config parse time. Defer to runtime — secret may not
|
||||
exist in dev/test environments. Use env var fallback.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — source connectors, paperless-ngx integration
|
||||
Reference in New Issue
Block a user