poimen-memory/tasks/M7.3-paperless-connector.md

M7.3 — paperless-ngx connector

Field	Value
Phase	M7 — Source connectors
Size	M — 1–3 days
Status	⬜ Not started
Flags	—
Spec	inlined below
Blocks	M7.10
Depends	M7.1

Goal

Implement a SourceConnector for paperless-ngx so OCR'd documents, manuals, and reference PDFs already stored in the cluster's paperless instance become searchable through the memory service without manual export.

Facts (inlined — no spec read needed)

paperless-ngx is already running in the cluster. Its REST API provides:

• GET /api/documents/ — paginated list with filtering by tags, document type, correspondent, dates.
• GET /api/documents/{id}/ — full metadata including content (extracted text).
• GET /api/documents/{id}/download/ — original file.
• GET /api/documents/{id}/preview/ — thumbnail.
• Authentication via Authorization: Token <token> header.
• Documents have checksum field (sha256 of original file).

Tag filtering is the selection mechanism. Not every scanned receipt belongs in the knowledge base. Config specifies which tags to include:

connectors:
  - kind: paperless
    name: homelab-paperless
    config:
      base_url: http://paperless-ngx.paperless.svc.cluster.local:8000
      token_secret: paperless-api-token   # k8s secret ref
      tags: [reference, manual, runbook]  # only sync these
      format: text                        # use extracted text content
      page_size: 100                      # API pagination size

Content comes as extracted text. paperless-ngx OCRs documents on import and stores the text in the content field. Use this directly — no PDF parsing needed in the connector. The text quality depends on paperless's OCR config.

Checksum enables cheap change detection. paperless provides checksum per document. The sync framework (M7.6) compares this against the last-known hash to skip unchanged documents.

Steps

1. Implement PaperlessConnector in mem-ingest/src/connectors/paperless.rs.
2. list_documents() — paginate GET /api/documents/?tags__name__in=..., extract id, title, checksum, modified for each.
3. fetch_document() — GET /api/documents/{id}/, extract content field, return with metadata (title, tags, correspondent, date_created).
4. health_check() — GET /api/ and verify 200; report document count from GET /api/documents/?tags__name__in=...&page=1&page_size=1 (read count).
5. Handle pagination (paperless returns next URL for subsequent pages).
6. Token auth from k8s secret (resolve token_secret to actual token value).
7. Register "paperless" kind in connector registry factory.
8. source_type() returns Reference.
9. Rate limit API calls (configurable, default 10 req/s) to avoid overloading the paperless instance.

Acceptance

• PaperlessConnector implements SourceConnector fully.
• Tag filtering limits which documents are listed.
• Pagination handles > 100 documents correctly.
• content_hash uses paperless's checksum field for change detection.
• Auth token resolved from k8s secret (not hardcoded).
• Health check reports document count matching tag filter.

Verify

Harness: mock HTTP server (wiremock or similar) returning paperless API responses; fixture JSON responses for list/detail endpoints.

Integration test — tests/it_paperless_connector.rs:

1. a1_list_filters_by_tags — mock returns 5 docs, 3 with matching tags; assert list_documents() returns 3.
2. a2_pagination_fetches_all — mock returns 2 pages of 50; assert 100 docs.
3. a3_fetch_returns_content — mock detail endpoint; assert text matches fixture.
4. a4_fetch_includes_metadata — assert returned metadata includes title, tags, correspondent, date fields.
5. a5_health_check_reachable — mock 200; assert reachable: true with count.
6. a6_health_check_unreachable — mock connection refused; assert reachable: false with error message.
7. a7_checksum_as_content_hash — assert SourceDocument.content_hash is populated from paperless checksum field.
8. a8_auth_header_sent — assert mock received Authorization: Token <value>.
9. a9_config_from_yaml — parse connector from YAML fixture; assert fields match.
10. a10_live — #[ignore]; real paperless instance; list + fetch one doc; print title and content length for human sanity check.

Command: cargo test --test it_paperless_connector (add -- --ignored for a10)

False pass:

• Mocking without verifying auth header. A connector that works in tests but sends no auth fails silently against real paperless.
• Testing single page only. Pagination bugs are invisible with < page_size docs.

Traps

• Assuming content is always populated. paperless may have documents without OCR text (e.g., empty scans). Return empty content with a warning, don't panic.
• Hardcoding the base URL without trailing-slash normalization. /api/documents/ vs /api/documents behaves differently.
• Not handling paperless API rate limits (429 responses). Add retry-after logic.
• Resolving k8s secrets at config parse time. Defer to runtime — secret may not exist in dev/test environments. Use env var fallback.

---

Background: DESIGN.md — source connectors, paperless-ngx integration