# M7.5 — S3-compatible storage connector

| Field | Value |
|---|---|
| Phase | M7 — Source connectors |
| Size | M — 1–3 days |
| Status | ⬜ Not started |
| Flags | — |
| Spec | inlined below |
| Blocks | M7.10 |
| Depends | M7.1 |

## Goal

Implement a `SourceConnector` for S3-compatible object storage (MinIO, AWS S3,
R2, etc.) so documents stored in buckets become searchable through the memory
service.

## Facts (inlined — no spec read needed)

S3 is the universal storage protocol. MinIO runs in many homelabs, and cloud
providers expose the same API. This connector makes any S3 bucket a knowledge
source.

**Configuration:**
```yaml
connectors:
  - kind: s3
    name: knowledge-bucket
    config:
      endpoint: https://minio.riotpiao.com
      bucket: knowledge-base
      prefix: docs/                    # only this prefix
      extensions: [md, txt, pdf]       # filter by key suffix
      access_key_secret: minio-creds   # k8s secret with access/secret keys
      region: us-east-1                # for AWS; ignored by MinIO
```

**ETag for change detection.** S3 objects have ETags (usually MD5 of content).
Use this as `content_hash` in `SourceDocument` — the sync framework skips objects
whose ETag hasn't changed.

**Text extraction.** S3 stores raw files. Markdown and text files are read
directly. PDF/DOCX support is out of scope for M7.5 — those MIME types are
skipped with a warning. Future: add a text extraction layer or require pre-processed
text.

**Pagination via continuation tokens.** S3 ListObjectsV2 returns max 1000 keys
per request. Use `ContinuationToken` for subsequent pages.

## Steps

1. Implement `S3Connector` in `mem-ingest/src/connectors/s3.rs`.
2. `list_documents()` — `ListObjectsV2` with `Prefix`, paginate, filter by
   extension, return `SourceDocument` per object.
3. `fetch_document()` — `GetObject`, read body as text (UTF-8), return with
   metadata (key, size, last_modified, ETag).
4. `health_check()` — `HeadBucket` to verify access.
5. Auth via access key + secret key from k8s secret.
6. Use `aws-sdk-s3` or `rust-s3` crate for S3 API.
7. Register `"s3"` kind in connector registry factory.
8. `source_type()` returns `Reference`.

## Acceptance

- `S3Connector` implements `SourceConnector` fully.
- Prefix filtering limits to configured path.
- Extension filtering skips non-text objects.
- ETag is used as `content_hash` for change detection.
- Pagination handles > 1000 objects.
- Auth works with MinIO and AWS-style credentials.

## Verify

**Harness:** mock S3 server (localstack or in-process mock) with fixture objects.

**Integration test** — `tests/it_s3_connector.rs`:
1. `a1_list_with_prefix` — mock bucket with objects under `docs/` and `images/`;
   assert only `docs/` objects listed.
2. `a2_extension_filtering` — mock with .md, .png, .txt; assert .png excluded.
3. `a3_fetch_returns_content` — fetch a .md object; assert content matches.
4. `a4_etag_as_content_hash` — assert `SourceDocument.content_hash` equals
   the object's ETag.
5. `a5_pagination` — mock 1500 objects; assert all listed via continuation tokens.
6. `a6_health_check_valid_bucket` — mock HeadBucket 200; assert reachable.
7. `a7_health_check_no_access` — mock HeadBucket 403; assert not reachable with
   error message.
8. `a8_non_utf8_skipped` — mock object with binary content; assert skipped with
   warning, not crash.
9. `a9_config_from_yaml` — parse connector from YAML; assert fields match.

**Command:** `cargo test --test it_s3_connector`

**False pass:**
- Testing with a local filesystem mock instead of S3 API mock. The pagination
  and ETag handling are S3-specific.

## Traps

- Assuming ETags are always MD5. Multipart uploads produce composite ETags
  (`hash-N`). These are still unique per version — use as-is for change detection.
- Not handling `NoSuchBucket` vs `AccessDenied`. Both are errors but mean
  different things for health reporting.
- Reading binary files as UTF-8. A JPEG read as text produces garbage. Check
  content-type header and skip non-text MIME types.

---

Background: [DESIGN.md](../DESIGN.md) — source connectors section