Story Crater Bot
71ecf482e7
- M7.1-M7.10: Extensible SourceConnector trait, Obsidian/paperless/git/S3 connectors, sync framework, CLI, HTTP endpoints, health monitoring, gate - M3.5.10: Auth integration with Authentik OIDC → Vault token validation - DESIGN.md: Add source connectors architecture, update auth to Authentik/Vault (Kong removed from cluster) - INDEX.md: 75 tasks, 11 gates - Fix all Kong references in M3.5.1 task
110 lines
4.1 KiB
Markdown
110 lines
4.1 KiB
Markdown
# M7.5 — S3-compatible storage connector
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | M7 — Source connectors |
|
||
| Size | M — 1–3 days |
|
||
| Status | ⬜ Not started |
|
||
| Flags | — |
|
||
| Spec | inlined below |
|
||
| Blocks | M7.10 |
|
||
| Depends | M7.1 |
|
||
|
||
## Goal
|
||
|
||
Implement a `SourceConnector` for S3-compatible object storage (MinIO, AWS S3,
|
||
R2, etc.) so documents stored in buckets become searchable through the memory
|
||
service.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
S3 is the universal storage protocol. MinIO runs in many homelabs, and cloud
|
||
providers expose the same API. This connector makes any S3 bucket a knowledge
|
||
source.
|
||
|
||
**Configuration:**
|
||
```yaml
|
||
connectors:
|
||
- kind: s3
|
||
name: knowledge-bucket
|
||
config:
|
||
endpoint: https://minio.riotpiao.com
|
||
bucket: knowledge-base
|
||
prefix: docs/ # only this prefix
|
||
extensions: [md, txt, pdf] # filter by key suffix
|
||
access_key_secret: minio-creds # k8s secret with access/secret keys
|
||
region: us-east-1 # for AWS; ignored by MinIO
|
||
```
|
||
|
||
**ETag for change detection.** S3 objects have ETags (usually MD5 of content).
|
||
Use this as `content_hash` in `SourceDocument` — the sync framework skips objects
|
||
whose ETag hasn't changed.
|
||
|
||
**Text extraction.** S3 stores raw files. Markdown and text files are read
|
||
directly. PDF/DOCX support is out of scope for M7.5 — those MIME types are
|
||
skipped with a warning. Future: add a text extraction layer or require pre-processed
|
||
text.
|
||
|
||
**Pagination via continuation tokens.** S3 ListObjectsV2 returns max 1000 keys
|
||
per request. Use `ContinuationToken` for subsequent pages.
|
||
|
||
## Steps
|
||
|
||
1. Implement `S3Connector` in `mem-ingest/src/connectors/s3.rs`.
|
||
2. `list_documents()` — `ListObjectsV2` with `Prefix`, paginate, filter by
|
||
extension, return `SourceDocument` per object.
|
||
3. `fetch_document()` — `GetObject`, read body as text (UTF-8), return with
|
||
metadata (key, size, last_modified, ETag).
|
||
4. `health_check()` — `HeadBucket` to verify access.
|
||
5. Auth via access key + secret key from k8s secret.
|
||
6. Use `aws-sdk-s3` or `rust-s3` crate for S3 API.
|
||
7. Register `"s3"` kind in connector registry factory.
|
||
8. `source_type()` returns `Reference`.
|
||
|
||
## Acceptance
|
||
|
||
- `S3Connector` implements `SourceConnector` fully.
|
||
- Prefix filtering limits to configured path.
|
||
- Extension filtering skips non-text objects.
|
||
- ETag is used as `content_hash` for change detection.
|
||
- Pagination handles > 1000 objects.
|
||
- Auth works with MinIO and AWS-style credentials.
|
||
|
||
## Verify
|
||
|
||
**Harness:** mock S3 server (localstack or in-process mock) with fixture objects.
|
||
|
||
**Integration test** — `tests/it_s3_connector.rs`:
|
||
1. `a1_list_with_prefix` — mock bucket with objects under `docs/` and `images/`;
|
||
assert only `docs/` objects listed.
|
||
2. `a2_extension_filtering` — mock with .md, .png, .txt; assert .png excluded.
|
||
3. `a3_fetch_returns_content` — fetch a .md object; assert content matches.
|
||
4. `a4_etag_as_content_hash` — assert `SourceDocument.content_hash` equals
|
||
the object's ETag.
|
||
5. `a5_pagination` — mock 1500 objects; assert all listed via continuation tokens.
|
||
6. `a6_health_check_valid_bucket` — mock HeadBucket 200; assert reachable.
|
||
7. `a7_health_check_no_access` — mock HeadBucket 403; assert not reachable with
|
||
error message.
|
||
8. `a8_non_utf8_skipped` — mock object with binary content; assert skipped with
|
||
warning, not crash.
|
||
9. `a9_config_from_yaml` — parse connector from YAML; assert fields match.
|
||
|
||
**Command:** `cargo test --test it_s3_connector`
|
||
|
||
**False pass:**
|
||
- Testing with a local filesystem mock instead of S3 API mock. The pagination
|
||
and ETag handling are S3-specific.
|
||
|
||
## Traps
|
||
|
||
- Assuming ETags are always MD5. Multipart uploads produce composite ETags
|
||
(`hash-N`). These are still unique per version — use as-is for change detection.
|
||
- Not handling `NoSuchBucket` vs `AccessDenied`. Both are errors but mean
|
||
different things for health reporting.
|
||
- Reading binary files as UTF-8. A JPEG read as text produces garbage. Check
|
||
content-type header and skip non-text MIME types.
|
||
|
||
---
|
||
|
||
Background: [DESIGN.md](../DESIGN.md) — source connectors section
|