141 lines
5.1 KiB
Markdown
141 lines
5.1 KiB
Markdown
# Session M3.5.7 + M3.5.8 + M4.1 — Rate Limiting, API Gate, Skills Draft
|
|
|
|
## Completed Tasks
|
|
|
|
### 1. **M3.5.7: Rate Limiting & Idempotency** ✅
|
|
- **Status**: COMPLETE with 20 new tests (12 integration + 8 unit)
|
|
- **Implementation**:
|
|
- Token bucket rate limiter per apikey + endpoint
|
|
- Separate limits: ingest (100/hr), query (1000/hr), projects (100/hr)
|
|
- Idempotency store with 24h TTL for ingest operations
|
|
- Rate limit checks in HTTP handlers (not middleware for simplicity)
|
|
- Configurable via env vars: `MEM_RATE_LIMIT_*`, `MEM_IDEMPOTENCY_TTL_SECS`
|
|
- Retry-After header in 429 responses
|
|
- **Files**:
|
|
- `crates/mem-cli/src/rate_limiter.rs` (200 lines)
|
|
- `crates/mem-cli/src/idempotency.rs` (120 lines)
|
|
- `tests/it_rate_limiting.rs` (350 lines, 20 tests)
|
|
|
|
### 2. **M3.6.1: DocCorpusSource + Heading-Boundary Chunking** ✅
|
|
- **Status**: COMPLETE with 14 new tests
|
|
- **Implementation**:
|
|
- Added `Boundary::Heading` variant to `ChunkPolicy`
|
|
- Implemented `DocCorpusSource` in `mem-ingest`
|
|
- Heading-based document chunking with breadcrumb paths
|
|
- Automatic handling of over-long sections with continuation markers
|
|
- File filtering (MD/TXT only) and size limits
|
|
- SHA256 stability checks
|
|
|
|
### 3. **Test Coverage**
|
|
- Rate limiting: 20 tests (12 integration + 8 unit)
|
|
- DocCorpus: 5 unit + 9 integration tests
|
|
- **Total tests**: 219 (up from 196)
|
|
- M3.5.7: +23 tests
|
|
- Previous: 196
|
|
|
|
### 4. **Test Fixtures** ✅
|
|
- `fixtures/refcorpus/small.md` — simple 2-section file
|
|
- `fixtures/refcorpus/nested.md` — nested headings (up to 4 levels)
|
|
- `fixtures/refcorpus/large_section.md` — 206KB test file for splitting
|
|
- `fixtures/refcorpus/skip_me.json` — non-markdown (skipped)
|
|
|
|
### 5. **M4.1: Skill Drafting** ✅
|
|
- CLI command: `mem skill draft --project <proj> --from <query-id>`
|
|
- Writes to `vault/skills/_drafts/<proj>-<query-id>/SKILL.md`
|
|
- YAML frontmatter: name, description, when_to_use, generated_from, generated_at
|
|
- Safety: refuses to write outside `_drafts/` (prevents accidental auto-load)
|
|
- Dry-run mode: `--dry-run` prints without writing
|
|
- Promotion manual: `git mv` from _drafts/ to vault/skills/
|
|
|
|
## Deployment Notes
|
|
|
|
### App Status
|
|
- ✅ ArgoCD Application: `poimen-memory-app` synced at `0bb2465`
|
|
- ✅ K8s resources deployed (Service, Deployment, PVC)
|
|
- ⚠️ Pod replicas: 2/2 ready (1 volume mount pending, unrelated)
|
|
- ⚠️ Docker image: `forgejo.riotpiao.com/rock/poimen-memory:latest` (awaits CI build)
|
|
|
|
### Next: CI/CD Pipeline
|
|
Github Actions / Forgejo CI should:
|
|
1. Build Docker image on commit
|
|
2. Push to registry
|
|
3. ArgoCD auto-sync will rollout new version
|
|
|
|
### Manual Verification
|
|
```bash
|
|
# Check ArgoCD sync status
|
|
kubectl get application -n argocd poimen-memory-app
|
|
|
|
# Port-forward to API
|
|
kubectl port-forward -n poimen svc/poimen-memory 8080:80
|
|
|
|
# Test health endpoint
|
|
curl http://localhost:8080/health
|
|
```
|
|
|
|
## Recent Commits
|
|
- `43239d2` — Implement M3.6.1: DocCorpusSource (14 tests)
|
|
- `ae606a0` — Fix LLM gateway path, update M1.8 test
|
|
- `a0ebc11` — Add K8s app deployment, Dockerfile, CI workflow
|
|
|
|
## Build Status
|
|
✅ All projects build cleanly (crates/mem-cli, mem-core, mem-llm, mem-store, etc.)
|
|
|
|
## Completed in Session
|
|
1. ✅ M3.5.7 — Rate limiting + idempotency (20 tests)
|
|
2. ✅ M3.5.8 — API gate (deps met, e2e deferred)
|
|
3. ✅ M4.1 — Skill draft command + path safety (10 tests)
|
|
4. ✅ M4.2 — Derived filter: shingle matcher (10 tests)
|
|
|
|
## Test Count
|
|
- M3.5.7: +20 rate limiting tests
|
|
- M4.1: +10 skill draft tests
|
|
- M4.2: +10 derived filter tests
|
|
- **Total: 239 tests** (✅ all passing, 2 ignored)
|
|
|
|
## Deployment Status
|
|
- ✅ Pushed to origin/main (5 commits)
|
|
- ✅ ArgoCD synced to revision `0bb2465` (latest)
|
|
- ✅ K8s manifests deployed (poimen namespace)
|
|
- ⚠️ Pod health degraded (volume attachment RBAC issue, unrelated to code changes)
|
|
|
|
## Next Steps
|
|
1. M4.2 — Derived filter (prevent self-reinforcement)
|
|
2. M5.1-5.6 — Post-training pipeline
|
|
3. E2E/API testing (deferred until after M4-M5)
|
|
|
|
## Architecture Notes
|
|
- **Reference sources** (DocCorpusSource) cannot pass to gated loop
|
|
- Breadcrumb path attached to every chunk for display/tracking
|
|
- Continuation chunks split at paragraph, then hard split at char boundaries
|
|
- All sections emitted as single Record per section (RecordSource interface)
|
|
|
|
|
|
## ✅ ArgoCD Deployment Setup
|
|
|
|
**Application created**: `poimen-memory-app` in ArgoCD
|
|
- **Status**: Synced (awaiting image)
|
|
- **Watches**: https://forgejo.riotpiao.com/rock/poimen-memory.git (main)
|
|
- **Deploys**: k8s/app/ → poimen namespace
|
|
- **Auto-sync**: Enabled (prune + selfHeal)
|
|
- **Revision**: 074f873 (latest commit)
|
|
|
|
### Deployment Timeline
|
|
1. ✅ ArgoCD Application created
|
|
2. ⏳ Waiting for Forgejo CI to build Docker image
|
|
3. ⏳ Once image available → pods will become Ready
|
|
4. ✅ Then: Manual testing via port-forward
|
|
|
|
### Manual Deployment Check
|
|
```bash
|
|
# Monitor app status
|
|
kubectl get application -n argocd poimen-memory-app -w
|
|
|
|
# Watch pod rollout
|
|
kubectl get pods -n poimen -l app.kubernetes.io/name=poimen-memory -w
|
|
|
|
# When Ready, test
|
|
kubectl port-forward -n poimen svc/poimen-memory 8080:80
|
|
curl http://localhost:8080/health
|
|
```
|