poimen-memory/CLAUDE.md

# Session M3.5.7 + M3.6.1 — Rate Limiting & DocCorpusSource

## Completed Tasks

### 1. **M3.5.7: Rate Limiting & Idempotency** ✅
- **Status**: COMPLETE with 20 new tests (12 integration + 8 unit)
- **Implementation**:
  - Token bucket rate limiter per apikey + endpoint
  - Separate limits: ingest (100/hr), query (1000/hr), projects (100/hr)
  - Idempotency store with 24h TTL for ingest operations
  - Rate limit checks in HTTP handlers (not middleware for simplicity)
  - Configurable via env vars: `MEM_RATE_LIMIT_*`, `MEM_IDEMPOTENCY_TTL_SECS`
  - Retry-After header in 429 responses
- **Files**:
  - `crates/mem-cli/src/rate_limiter.rs` (200 lines)
  - `crates/mem-cli/src/idempotency.rs` (120 lines)
  - `tests/it_rate_limiting.rs` (350 lines, 20 tests)

### 2. **M3.6.1: DocCorpusSource + Heading-Boundary Chunking** ✅
- **Status**: COMPLETE with 14 new tests
- **Implementation**:
  - Added `Boundary::Heading` variant to `ChunkPolicy`
  - Implemented `DocCorpusSource` in `mem-ingest`
  - Heading-based document chunking with breadcrumb paths
  - Automatic handling of over-long sections with continuation markers
  - File filtering (MD/TXT only) and size limits
  - SHA256 stability checks

### 3. **Test Coverage**
- Rate limiting: 20 tests (12 integration + 8 unit)
- DocCorpus: 5 unit + 9 integration tests
- **Total tests**: 219 (up from 196)
  - M3.5.7: +23 tests
  - Previous: 196

### 4. **Test Fixtures** ✅
- `fixtures/refcorpus/small.md` — simple 2-section file
- `fixtures/refcorpus/nested.md` — nested headings (up to 4 levels)
- `fixtures/refcorpus/large_section.md` — 206KB test file for splitting
- `fixtures/refcorpus/skip_me.json` — non-markdown (skipped)

### 5. **Key Features**
- Headings marked with breadcrumbs prepended to chunk content
- Sections without body content are skipped (only-heading sections)
- Continuation marker for chunks split from large sections
- Dry-run mode: `source.dry_run()` prints plan without network calls
- Stable `source_uri` and file hashing for identity

## In Progress

### Option A: App Deployment
- Manifests created (deployment, service, kustomization)
- Secrets created
- **Blocked**: Awaiting Docker image from Forgejo CI build

### Deployment Commands (once image is ready)
```bash
kubectl apply -k k8s/app/
kubectl port-forward -n poimen svc/poimen-memory 8080:80
curl http://localhost:8080/health
```

## Recent Commits
- `43239d2` — Implement M3.6.1: DocCorpusSource (14 tests)
- `ae606a0` — Fix LLM gateway path, update M1.8 test
- `a0ebc11` — Add K8s app deployment, Dockerfile, CI workflow

## Test Status
✅ **219 tests passing, 0 failing, 2 ignored**
- M3.5.7: +20 rate limiting tests
- M3.6.1: +3 from before (14 doc corpus tests already counted)
- Previous: 196 tests

## Next Steps
1. ✅ M3.5.7 complete — rate limiting implemented
2. ⏳ M3.5.8 — API end-to-end gate (depends on M3.5.7 ✅)
3. ⏳ M3.5.9 — git-aware references
4. M3.6.2 — Level-R storage
5. M4.x — Skills extraction & filtering

## Architecture Notes
- **Reference sources** (DocCorpusSource) cannot pass to gated loop
- Breadcrumb path attached to every chunk for display/tracking
- Continuation chunks split at paragraph, then hard split at char boundaries
- All sections emitted as single Record per section (RecordSource interface)


## ✅ ArgoCD Deployment Setup

**Application created**: `poimen-memory-app` in ArgoCD
- **Status**: Synced (awaiting image)
- **Watches**: https://forgejo.riotpiao.com/rock/poimen-memory.git (main)
- **Deploys**: k8s/app/ → poimen namespace
- **Auto-sync**: Enabled (prune + selfHeal)
- **Revision**: 074f873 (latest commit)

### Deployment Timeline
1. ✅ ArgoCD Application created
2. ⏳ Waiting for Forgejo CI to build Docker image
3. ⏳ Once image available → pods will become Ready
4. ✅ Then: Manual testing via port-forward

### Manual Deployment Check
```bash
# Monitor app status
kubectl get application -n argocd poimen-memory-app -w

# Watch pod rollout
kubectl get pods -n poimen -l app.kubernetes.io/name=poimen-memory -w

# When Ready, test
kubectl port-forward -n poimen svc/poimen-memory 8080:80
curl http://localhost:8080/health
```