110 lines
4.1 KiB
Markdown
110 lines
4.1 KiB
Markdown
|
# M7.5 — S3-compatible storage connector
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | M7 — Source connectors |
|
|||
|
|
| Size | M — 1–3 days |
|
|||
|
|
| Status | ⬜ Not started |
|
|||
|
|
| Flags | — |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | M7.10 |
|
|||
|
|
| Depends | M7.1 |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Implement a `SourceConnector` for S3-compatible object storage (MinIO, AWS S3,
|
|||
|
|
R2, etc.) so documents stored in buckets become searchable through the memory
|
|||
|
|
service.
|
|||
|
|
|
|||
|
|
## Facts (inlined — no spec read needed)
|
|||
|
|
|
|||
|
|
S3 is the universal storage protocol. MinIO runs in many homelabs, and cloud
|
|||
|
|
providers expose the same API. This connector makes any S3 bucket a knowledge
|
|||
|
|
source.
|
|||
|
|
|
|||
|
|
**Configuration:**
|
|||
|
|
```yaml
|
|||
|
|
connectors:
|
|||
|
|
- kind: s3
|
|||
|
|
name: knowledge-bucket
|
|||
|
|
config:
|
|||
|
|
endpoint: https://minio.riotpiao.com
|
|||
|
|
bucket: knowledge-base
|
|||
|
|
prefix: docs/ # only this prefix
|
|||
|
|
extensions: [md, txt, pdf] # filter by key suffix
|
|||
|
|
access_key_secret: minio-creds # k8s secret with access/secret keys
|
|||
|
|
region: us-east-1 # for AWS; ignored by MinIO
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**ETag for change detection.** S3 objects have ETags (usually MD5 of content).
|
|||
|
|
Use this as `content_hash` in `SourceDocument` — the sync framework skips objects
|
|||
|
|
whose ETag hasn't changed.
|
|||
|
|
|
|||
|
|
**Text extraction.** S3 stores raw files. Markdown and text files are read
|
|||
|
|
directly. PDF/DOCX support is out of scope for M7.5 — those MIME types are
|
|||
|
|
skipped with a warning. Future: add a text extraction layer or require pre-processed
|
|||
|
|
text.
|
|||
|
|
|
|||
|
|
**Pagination via continuation tokens.** S3 ListObjectsV2 returns max 1000 keys
|
|||
|
|
per request. Use `ContinuationToken` for subsequent pages.
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. Implement `S3Connector` in `mem-ingest/src/connectors/s3.rs`.
|
|||
|
|
2. `list_documents()` — `ListObjectsV2` with `Prefix`, paginate, filter by
|
|||
|
|
extension, return `SourceDocument` per object.
|
|||
|
|
3. `fetch_document()` — `GetObject`, read body as text (UTF-8), return with
|
|||
|
|
metadata (key, size, last_modified, ETag).
|
|||
|
|
4. `health_check()` — `HeadBucket` to verify access.
|
|||
|
|
5. Auth via access key + secret key from k8s secret.
|
|||
|
|
6. Use `aws-sdk-s3` or `rust-s3` crate for S3 API.
|
|||
|
|
7. Register `"s3"` kind in connector registry factory.
|
|||
|
|
8. `source_type()` returns `Reference`.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
- `S3Connector` implements `SourceConnector` fully.
|
|||
|
|
- Prefix filtering limits to configured path.
|
|||
|
|
- Extension filtering skips non-text objects.
|
|||
|
|
- ETag is used as `content_hash` for change detection.
|
|||
|
|
- Pagination handles > 1000 objects.
|
|||
|
|
- Auth works with MinIO and AWS-style credentials.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Harness:** mock S3 server (localstack or in-process mock) with fixture objects.
|
|||
|
|
|
|||
|
|
**Integration test** — `tests/it_s3_connector.rs`:
|
|||
|
|
1. `a1_list_with_prefix` — mock bucket with objects under `docs/` and `images/`;
|
|||
|
|
assert only `docs/` objects listed.
|
|||
|
|
2. `a2_extension_filtering` — mock with .md, .png, .txt; assert .png excluded.
|
|||
|
|
3. `a3_fetch_returns_content` — fetch a .md object; assert content matches.
|
|||
|
|
4. `a4_etag_as_content_hash` — assert `SourceDocument.content_hash` equals
|
|||
|
|
the object's ETag.
|
|||
|
|
5. `a5_pagination` — mock 1500 objects; assert all listed via continuation tokens.
|
|||
|
|
6. `a6_health_check_valid_bucket` — mock HeadBucket 200; assert reachable.
|
|||
|
|
7. `a7_health_check_no_access` — mock HeadBucket 403; assert not reachable with
|
|||
|
|
error message.
|
|||
|
|
8. `a8_non_utf8_skipped` — mock object with binary content; assert skipped with
|
|||
|
|
warning, not crash.
|
|||
|
|
9. `a9_config_from_yaml` — parse connector from YAML; assert fields match.
|
|||
|
|
|
|||
|
|
**Command:** `cargo test --test it_s3_connector`
|
|||
|
|
|
|||
|
|
**False pass:**
|
|||
|
|
- Testing with a local filesystem mock instead of S3 API mock. The pagination
|
|||
|
|
and ETag handling are S3-specific.
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- Assuming ETags are always MD5. Multipart uploads produce composite ETags
|
|||
|
|
(`hash-N`). These are still unique per version — use as-is for change detection.
|
|||
|
|
- Not handling `NoSuchBucket` vs `AccessDenied`. Both are errors but mean
|
|||
|
|
different things for health reporting.
|
|||
|
|
- Reading binary files as UTF-8. A JPEG read as text produces garbage. Check
|
|||
|
|
content-type header and skip non-text MIME types.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background: [DESIGN.md](../DESIGN.md) — source connectors section
|