poimen-memory/CLAUDE.md

Session M3.6.1 — DocCorpusSource Implementation

Completed Tasks

1. M3.6.1: DocCorpusSource + Heading-Boundary Chunking ✅

• Status: COMPLETE with 14 new tests
• Implementation:
  • Added Boundary::Heading variant to ChunkPolicy
  • Implemented DocCorpusSource in mem-ingest
  • Heading-based document chunking with breadcrumb paths
  • Automatic handling of over-long sections with continuation markers
  • File filtering (MD/TXT only) and size limits
  • SHA256 stability checks

2. Test Coverage

• 5 unit tests in doc_corpus.rs — all passing
• 9 integration tests in it_doc_corpus.rs — all passing
• Total tests: 196 (up from 182)

3. Test Fixtures ✅

• fixtures/refcorpus/small.md — simple 2-section file
• fixtures/refcorpus/nested.md — nested headings (up to 4 levels)
• fixtures/refcorpus/large_section.md — 206KB test file for splitting
• fixtures/refcorpus/skip_me.json — non-markdown (skipped)

4. Key Features

• Headings marked with breadcrumbs prepended to chunk content
• Sections without body content are skipped (only-heading sections)
• Continuation marker for chunks split from large sections
• Dry-run mode: source.dry_run() prints plan without network calls
• Stable source_uri and file hashing for identity

In Progress

Option A: App Deployment

• Manifests created (deployment, service, kustomization)
• Secrets created
• Blocked: Awaiting Docker image from Forgejo CI build

Deployment Commands (once image is ready)

kubectl apply -k k8s/app/
kubectl port-forward -n poimen svc/poimen-memory 8080:80
curl http://localhost:8080/health

Recent Commits

• 43239d2 — Implement M3.6.1: DocCorpusSource (14 tests)
• ae606a0 — Fix LLM gateway path, update M1.8 test
• a0ebc11 — Add K8s app deployment, Dockerfile, CI workflow

Test Status

✅ 196 tests passing, 0 failing

• 182 tests (previous)
• +14 new tests (M3.6.1)

Next Steps

1. Wait for Forgejo CI to build and push Docker image
2. Apply kubectl apply -k k8s/app/ when image ready
3. Run manual tests against deployed app:

   curl http://localhost:8080/health
   curl http://localhost:8080/query -X POST -H "Content-Type: application/yaml" -d @queries/poimen.yaml
   

4. Continue with M3.6.2 — Level-R storage

Architecture Notes

• Reference sources (DocCorpusSource) cannot pass to gated loop
• Breadcrumb path attached to every chunk for display/tracking
• Continuation chunks split at paragraph, then hard split at char boundaries
• All sections emitted as single Record per section (RecordSource interface)