poimen-memory/tasks/M7.5-s3-connector.md

M7.5 — S3-compatible storage connector

Field	Value
Phase	M7 — Source connectors
Size	M — 1–3 days
Status	⬜ Not started
Flags	—
Spec	inlined below
Blocks	M7.10
Depends	M7.1

Goal

Implement a SourceConnector for S3-compatible object storage (MinIO, AWS S3, R2, etc.) so documents stored in buckets become searchable through the memory service.

Facts (inlined — no spec read needed)

S3 is the universal storage protocol. MinIO runs in many homelabs, and cloud providers expose the same API. This connector makes any S3 bucket a knowledge source.

Configuration:

connectors:
  - kind: s3
    name: knowledge-bucket
    config:
      endpoint: https://minio.riotpiao.com
      bucket: knowledge-base
      prefix: docs/                    # only this prefix
      extensions: [md, txt, pdf]       # filter by key suffix
      access_key_secret: minio-creds   # k8s secret with access/secret keys
      region: us-east-1                # for AWS; ignored by MinIO

ETag for change detection. S3 objects have ETags (usually MD5 of content). Use this as content_hash in SourceDocument — the sync framework skips objects whose ETag hasn't changed.

Text extraction. S3 stores raw files. Markdown and text files are read directly. PDF/DOCX support is out of scope for M7.5 — those MIME types are skipped with a warning. Future: add a text extraction layer or require pre-processed text.

Pagination via continuation tokens. S3 ListObjectsV2 returns max 1000 keys per request. Use ContinuationToken for subsequent pages.

Steps

1. Implement S3Connector in mem-ingest/src/connectors/s3.rs.
2. list_documents() — ListObjectsV2 with Prefix, paginate, filter by extension, return SourceDocument per object.
3. fetch_document() — GetObject, read body as text (UTF-8), return with metadata (key, size, last_modified, ETag).
4. health_check() — HeadBucket to verify access.
5. Auth via access key + secret key from k8s secret.
6. Use aws-sdk-s3 or rust-s3 crate for S3 API.
7. Register "s3" kind in connector registry factory.
8. source_type() returns Reference.

Acceptance

• S3Connector implements SourceConnector fully.
• Prefix filtering limits to configured path.
• Extension filtering skips non-text objects.
• ETag is used as content_hash for change detection.
• Pagination handles > 1000 objects.
• Auth works with MinIO and AWS-style credentials.

Verify

Harness: mock S3 server (localstack or in-process mock) with fixture objects.

Integration test — tests/it_s3_connector.rs:

1. a1_list_with_prefix — mock bucket with objects under docs/ and images/; assert only docs/ objects listed.
2. a2_extension_filtering — mock with .md, .png, .txt; assert .png excluded.
3. a3_fetch_returns_content — fetch a .md object; assert content matches.
4. a4_etag_as_content_hash — assert SourceDocument.content_hash equals the object's ETag.
5. a5_pagination — mock 1500 objects; assert all listed via continuation tokens.
6. a6_health_check_valid_bucket — mock HeadBucket 200; assert reachable.
7. a7_health_check_no_access — mock HeadBucket 403; assert not reachable with error message.
8. a8_non_utf8_skipped — mock object with binary content; assert skipped with warning, not crash.
9. a9_config_from_yaml — parse connector from YAML; assert fields match.

Command: cargo test --test it_s3_connector

False pass:

• Testing with a local filesystem mock instead of S3 API mock. The pagination and ETag handling are S3-specific.

Traps

• Assuming ETags are always MD5. Multipart uploads produce composite ETags (hash-N). These are still unique per version — use as-is for change detection.
• Not handling NoSuchBucket vs AccessDenied. Both are errors but mean different things for health reporting.
• Reading binary files as UTF-8. A JPEG read as text produces garbage. Check content-type header and skip non-text MIME types.

---

Background: DESIGN.md — source connectors section