Session summary: M3.6.1 complete (196 tests, heading-boundary chunking)

2026-08-23 09:43:37 -07:00
parent 43239d24ce
commit 074f87312e
1 changed files with 72 additions and 0 deletions
@@ -0,0 +1,72 @@
+# Session M3.6.1 — DocCorpusSource Implementation
+
+## Completed Tasks
+
+### 1. **M3.6.1: DocCorpusSource + Heading-Boundary Chunking** ✅
+- **Status**: COMPLETE with 14 new tests
+- **Implementation**:
+  - Added `Boundary::Heading` variant to `ChunkPolicy`
+  - Implemented `DocCorpusSource` in `mem-ingest`
+  - Heading-based document chunking with breadcrumb paths
+  - Automatic handling of over-long sections with continuation markers
+  - File filtering (MD/TXT only) and size limits
+  - SHA256 stability checks
+
+### 2. **Test Coverage**
+- 5 unit tests in `doc_corpus.rs` — all passing
+- 9 integration tests in `it_doc_corpus.rs` — all passing
+- **Total tests**: 196 (up from 182)
+
+### 3. **Test Fixtures** ✅
+- `fixtures/refcorpus/small.md` — simple 2-section file
+- `fixtures/refcorpus/nested.md` — nested headings (up to 4 levels)
+- `fixtures/refcorpus/large_section.md` — 206KB test file for splitting
+- `fixtures/refcorpus/skip_me.json` — non-markdown (skipped)
+
+### 4. **Key Features**
+- Headings marked with breadcrumbs prepended to chunk content
+- Sections without body content are skipped (only-heading sections)
+- Continuation marker for chunks split from large sections
+- Dry-run mode: `source.dry_run()` prints plan without network calls
+- Stable `source_uri` and file hashing for identity
+
+## In Progress
+
+### Option A: App Deployment
+- Manifests created (deployment, service, kustomization)
+- Secrets created
+- **Blocked**: Awaiting Docker image from Forgejo CI build
+
+### Deployment Commands (once image is ready)
+```bash
+kubectl apply -k k8s/app/
+kubectl port-forward -n poimen svc/poimen-memory 8080:80
+curl http://localhost:8080/health
+```
+
+## Recent Commits
+- `43239d2` — Implement M3.6.1: DocCorpusSource (14 tests)
+- `ae606a0` — Fix LLM gateway path, update M1.8 test
+- `a0ebc11` — Add K8s app deployment, Dockerfile, CI workflow
+
+## Test Status
+✅ **196 tests passing, 0 failing**
+- 182 tests (previous)
+- +14 new tests (M3.6.1)
+
+## Next Steps
+1. Wait for Forgejo CI to build and push Docker image
+2. Apply `kubectl apply -k k8s/app/` when image ready
+3. Run manual tests against deployed app:
+   ```bash
+   curl http://localhost:8080/health
+   curl http://localhost:8080/query -X POST -H "Content-Type: application/yaml" -d @queries/poimen.yaml
+   ```
+4. Continue with M3.6.2 — Level-R storage
+
+## Architecture Notes
+- **Reference sources** (DocCorpusSource) cannot pass to gated loop
+- Breadcrumb path attached to every chunk for display/tracking
+- Continuation chunks split at paragraph, then hard split at char boundaries
+- All sections emitted as single Record per section (RecordSource interface)
+