73 lines
2.6 KiB
Markdown
73 lines
2.6 KiB
Markdown
|
# Session M3.6.1 — DocCorpusSource Implementation
|
||
|
|
|
||
|
|
## Completed Tasks
|
||
|
|
|
||
|
|
### 1. **M3.6.1: DocCorpusSource + Heading-Boundary Chunking** ✅
|
||
|
|
- **Status**: COMPLETE with 14 new tests
|
||
|
|
- **Implementation**:
|
||
|
|
- Added `Boundary::Heading` variant to `ChunkPolicy`
|
||
|
|
- Implemented `DocCorpusSource` in `mem-ingest`
|
||
|
|
- Heading-based document chunking with breadcrumb paths
|
||
|
|
- Automatic handling of over-long sections with continuation markers
|
||
|
|
- File filtering (MD/TXT only) and size limits
|
||
|
|
- SHA256 stability checks
|
||
|
|
|
||
|
|
### 2. **Test Coverage**
|
||
|
|
- 5 unit tests in `doc_corpus.rs` — all passing
|
||
|
|
- 9 integration tests in `it_doc_corpus.rs` — all passing
|
||
|
|
- **Total tests**: 196 (up from 182)
|
||
|
|
|
||
|
|
### 3. **Test Fixtures** ✅
|
||
|
|
- `fixtures/refcorpus/small.md` — simple 2-section file
|
||
|
|
- `fixtures/refcorpus/nested.md` — nested headings (up to 4 levels)
|
||
|
|
- `fixtures/refcorpus/large_section.md` — 206KB test file for splitting
|
||
|
|
- `fixtures/refcorpus/skip_me.json` — non-markdown (skipped)
|
||
|
|
|
||
|
|
### 4. **Key Features**
|
||
|
|
- Headings marked with breadcrumbs prepended to chunk content
|
||
|
|
- Sections without body content are skipped (only-heading sections)
|
||
|
|
- Continuation marker for chunks split from large sections
|
||
|
|
- Dry-run mode: `source.dry_run()` prints plan without network calls
|
||
|
|
- Stable `source_uri` and file hashing for identity
|
||
|
|
|
||
|
|
## In Progress
|
||
|
|
|
||
|
|
### Option A: App Deployment
|
||
|
|
- Manifests created (deployment, service, kustomization)
|
||
|
|
- Secrets created
|
||
|
|
- **Blocked**: Awaiting Docker image from Forgejo CI build
|
||
|
|
|
||
|
|
### Deployment Commands (once image is ready)
|
||
|
|
```bash
|
||
|
|
kubectl apply -k k8s/app/
|
||
|
|
kubectl port-forward -n poimen svc/poimen-memory 8080:80
|
||
|
|
curl http://localhost:8080/health
|
||
|
|
```
|
||
|
|
|
||
|
|
## Recent Commits
|
||
|
|
- `43239d2` — Implement M3.6.1: DocCorpusSource (14 tests)
|
||
|
|
- `ae606a0` — Fix LLM gateway path, update M1.8 test
|
||
|
|
- `a0ebc11` — Add K8s app deployment, Dockerfile, CI workflow
|
||
|
|
|
||
|
|
## Test Status
|
||
|
|
✅ **196 tests passing, 0 failing**
|
||
|
|
- 182 tests (previous)
|
||
|
|
- +14 new tests (M3.6.1)
|
||
|
|
|
||
|
|
## Next Steps
|
||
|
|
1. Wait for Forgejo CI to build and push Docker image
|
||
|
|
2. Apply `kubectl apply -k k8s/app/` when image ready
|
||
|
|
3. Run manual tests against deployed app:
|
||
|
|
```bash
|
||
|
|
curl http://localhost:8080/health
|
||
|
|
curl http://localhost:8080/query -X POST -H "Content-Type: application/yaml" -d @queries/poimen.yaml
|
||
|
|
```
|
||
|
|
4. Continue with M3.6.2 — Level-R storage
|
||
|
|
|
||
|
|
## Architecture Notes
|
||
|
|
- **Reference sources** (DocCorpusSource) cannot pass to gated loop
|
||
|
|
- Breadcrumb path attached to every chunk for display/tracking
|
||
|
|
- Continuation chunks split at paragraph, then hard split at char boundaries
|
||
|
|
- All sections emitted as single Record per section (RecordSource interface)
|
||
|
|
|