poimen-memory/tasks/M7.3-paperless-connector.md

# M7.3 — paperless-ngx connector

| Field | Value |
|---|---|
| Phase | M7 — Source connectors |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M7.10 |
| Depends | M7.1 |

## Goal

Implement a `SourceConnector` for paperless-ngx so OCR'd documents, manuals, and
reference PDFs already stored in the cluster's paperless instance become
searchable through the memory service without manual export.

## Facts (inlined — no spec read needed)

paperless-ngx is already running in the cluster. Its REST API provides:

- `GET /api/documents/` — paginated list with filtering by tags, document type,
  correspondent, dates.
- `GET /api/documents/{id}/` — full metadata including `content` (extracted text).
- `GET /api/documents/{id}/download/` — original file.
- `GET /api/documents/{id}/preview/` — thumbnail.
- Authentication via `Authorization: Token <token>` header.
- Documents have `checksum` field (sha256 of original file).

**Tag filtering is the selection mechanism.** Not every scanned receipt belongs
in the knowledge base. Config specifies which tags to include:
```yaml
connectors:
  - kind: paperless
    name: homelab-paperless
    config:
      base_url: http://paperless-ngx.paperless.svc.cluster.local:8000
      token_secret: paperless-api-token   # k8s secret ref
      tags: [reference, manual, runbook]  # only sync these
      format: text                        # use extracted text content
      page_size: 100                      # API pagination size
```

**Content comes as extracted text.** paperless-ngx OCRs documents on import and
stores the text in the `content` field. Use this directly — no PDF parsing needed
in the connector. The text quality depends on paperless's OCR config.

**Checksum enables cheap change detection.** paperless provides `checksum` per
document. The sync framework (M7.6) compares this against the last-known hash
to skip unchanged documents.

## Steps

1. Implement `PaperlessConnector` in `mem-ingest/src/connectors/paperless.rs`.
2. `list_documents()` — paginate `GET /api/documents/?tags__name__in=...`,
   extract `id`, `title`, `checksum`, `modified` for each.
3. `fetch_document()` — `GET /api/documents/{id}/`, extract `content` field,
   return with metadata (title, tags, correspondent, date_created).
4. `health_check()` — `GET /api/` and verify 200; report document count from
   `GET /api/documents/?tags__name__in=...&page=1&page_size=1` (read `count`).
5. Handle pagination (paperless returns `next` URL for subsequent pages).
6. Token auth from k8s secret (resolve `token_secret` to actual token value).
7. Register `"paperless"` kind in connector registry factory.
8. `source_type()` returns `Reference`.
9. Rate limit API calls (configurable, default 10 req/s) to avoid overloading
   the paperless instance.

## Acceptance

- `PaperlessConnector` implements `SourceConnector` fully.
- Tag filtering limits which documents are listed.
- Pagination handles > 100 documents correctly.
- `content_hash` uses paperless's `checksum` field for change detection.
- Auth token resolved from k8s secret (not hardcoded).
- Health check reports document count matching tag filter.

## Verify

**Harness:** mock HTTP server (wiremock or similar) returning paperless API
responses; fixture JSON responses for list/detail endpoints.

**Integration test** — `tests/it_paperless_connector.rs`:
1. `a1_list_filters_by_tags` — mock returns 5 docs, 3 with matching tags; assert
   `list_documents()` returns 3.
2. `a2_pagination_fetches_all` — mock returns 2 pages of 50; assert 100 docs.
3. `a3_fetch_returns_content` — mock detail endpoint; assert text matches fixture.
4. `a4_fetch_includes_metadata` — assert returned metadata includes title, tags,
   correspondent, date fields.
5. `a5_health_check_reachable` — mock 200; assert `reachable: true` with count.
6. `a6_health_check_unreachable` — mock connection refused; assert
   `reachable: false` with error message.
7. `a7_checksum_as_content_hash` — assert `SourceDocument.content_hash` is
   populated from paperless `checksum` field.
8. `a8_auth_header_sent` — assert mock received `Authorization: Token <value>`.
9. `a9_config_from_yaml` — parse connector from YAML fixture; assert fields match.
10. `a10_live` — `#[ignore]`; real paperless instance; list + fetch one doc; print
    title and content length for human sanity check.

**Command:** `cargo test --test it_paperless_connector` (add `-- --ignored` for a10)

**False pass:**
- Mocking without verifying auth header. A connector that works in tests but
  sends no auth fails silently against real paperless.
- Testing single page only. Pagination bugs are invisible with < `page_size` docs.

## Traps

- Assuming `content` is always populated. paperless may have documents without
  OCR text (e.g., empty scans). Return empty content with a warning, don't panic.
- Hardcoding the base URL without trailing-slash normalization. `/api/documents/`
  vs `/api/documents` behaves differently.
- Not handling paperless API rate limits (429 responses). Add retry-after logic.
- Resolving k8s secrets at config parse time. Defer to runtime — secret may not
  exist in dev/test environments. Use env var fallback.

---

Background: [DESIGN.md](../DESIGN.md) — source connectors, paperless-ngx integration