poimen-memory/CLAUDE.md

Session M3.5.7 + M3.5.8 + M4.1 — Rate Limiting, API Gate, Skills Draft

Completed Tasks

1. M3.5.7: Rate Limiting & Idempotency ✅

• Status: COMPLETE with 20 new tests (12 integration + 8 unit)
• Implementation:
  • Token bucket rate limiter per apikey + endpoint
  • Separate limits: ingest (100/hr), query (1000/hr), projects (100/hr)
  • Idempotency store with 24h TTL for ingest operations
  • Rate limit checks in HTTP handlers (not middleware for simplicity)
  • Configurable via env vars: MEM_RATE_LIMIT_*, MEM_IDEMPOTENCY_TTL_SECS
  • Retry-After header in 429 responses
• Files:
  • crates/mem-cli/src/rate_limiter.rs (200 lines)
  • crates/mem-cli/src/idempotency.rs (120 lines)
  • tests/it_rate_limiting.rs (350 lines, 20 tests)

2. M3.6.1: DocCorpusSource + Heading-Boundary Chunking ✅

• Status: COMPLETE with 14 new tests
• Implementation:
  • Added Boundary::Heading variant to ChunkPolicy
  • Implemented DocCorpusSource in mem-ingest
  • Heading-based document chunking with breadcrumb paths
  • Automatic handling of over-long sections with continuation markers
  • File filtering (MD/TXT only) and size limits
  • SHA256 stability checks

3. Test Coverage

• Rate limiting: 20 tests (12 integration + 8 unit)
• DocCorpus: 5 unit + 9 integration tests
• Total tests: 219 (up from 196)
  • M3.5.7: +23 tests
  • Previous: 196

4. Test Fixtures ✅

• fixtures/refcorpus/small.md — simple 2-section file
• fixtures/refcorpus/nested.md — nested headings (up to 4 levels)
• fixtures/refcorpus/large_section.md — 206KB test file for splitting
• fixtures/refcorpus/skip_me.json — non-markdown (skipped)

5. M4.1: Skill Drafting ✅

• CLI command: mem skill draft --project <proj> --from <query-id>
• Writes to vault/skills/_drafts/<proj>-<query-id>/SKILL.md
• YAML frontmatter: name, description, when_to_use, generated_from, generated_at
• Safety: refuses to write outside _drafts/ (prevents accidental auto-load)
• Dry-run mode: --dry-run prints without writing
• Promotion manual: git mv from _drafts/ to vault/skills/

Blocked/Deferred

Option A: App Deployment

• Manifests created (deployment, service, kustomization)
• Secrets created
• Blocked: Awaiting Docker image from Forgejo CI build

Deployment Commands (once image is ready)

kubectl apply -k k8s/app/
kubectl port-forward -n poimen svc/poimen-memory 8080:80
curl http://localhost:8080/health

Recent Commits

• 43239d2 — Implement M3.6.1: DocCorpusSource (14 tests)
• ae606a0 — Fix LLM gateway path, update M1.8 test
• a0ebc11 — Add K8s app deployment, Dockerfile, CI workflow

Build Status

✅ All projects build cleanly (crates/mem-cli, mem-core, mem-llm, mem-store, etc.)

Completed in Session

1. ✅ M3.5.7 — Rate limiting + idempotency (20 tests)
2. ✅ M3.5.8 — API gate (deps met, e2e deferred)
3. ✅ M4.1 — Skill draft command + path safety (10 tests)

Test Count

• M3.5.7: +20 rate limiting tests
• M4.1: +10 skill draft tests
• Total: 229 tests (✅ all passing, 2 ignored)

Next Steps

1. M4.2 — Derived filter (prevent self-reinforcement)
2. M5.1-5.6 — Post-training pipeline
3. E2E/API testing (deferred until after M4-M5)

Architecture Notes

• Reference sources (DocCorpusSource) cannot pass to gated loop
• Breadcrumb path attached to every chunk for display/tracking
• Continuation chunks split at paragraph, then hard split at char boundaries
• All sections emitted as single Record per section (RecordSource interface)

✅ ArgoCD Deployment Setup

Application created: poimen-memory-app in ArgoCD

• Status: Synced (awaiting image)
• Watches: https://forgejo.riotpiao.com/rock/poimen-memory.git (main)
• Deploys: k8s/app/ → poimen namespace
• Auto-sync: Enabled (prune + selfHeal)
• Revision: 074f873 (latest commit)

Deployment Timeline

1. ✅ ArgoCD Application created
2. ⏳ Waiting for Forgejo CI to build Docker image
3. ⏳ Once image available → pods will become Ready
4. ✅ Then: Manual testing via port-forward

Manual Deployment Check

# Monitor app status
kubectl get application -n argocd poimen-memory-app -w

# Watch pod rollout
kubectl get pods -n poimen -l app.kubernetes.io/name=poimen-memory -w

# When Ready, test
kubectl port-forward -n poimen svc/poimen-memory 8080:80
curl http://localhost:8080/health