riotpiao-poimen/poimen-memory
2
0
Fork 0
Code Issues 11 Pull Requests Actions Packages Projects Releases Wiki Activity
204 Commits 15 Branches 0 Tags
4169effd8a6b84e503fa3e3841c1840c46d9bd27
Commit Graph
11 Commits
Author SHA1 Message Date
poimenandrock 4169effd8a feat: complete observability stack (O1-O13) (#52)
CI / CI (push) Successful in 12m36s
Details
Deploy / Tag & Push Latest (push) Successful in 1m56s
Details 
## Complete Observability Stack (O1-O13)

Implements all 13 observability issues in a single PR. 119 metrics total.

### Commits (one per issue)

| Issue | Title | Metrics |
|-------|-------|---------|
| **O10** | Prometheus metrics module + /metrics endpoint | Foundation |
| **O1** | Instrument ingest handler | I1-I12 (12) |
| **O2** | Instrument query handler | Q1-Q12 (12) |
| **O3** | Instrument context endpoint | C1-C8 (8) |
| **O4** | Relevance judge | R1-R9 (9) |
| **O5** | Write volume + storage metrics | W1-W12 (12) |
| **O6** | Pod resource observability | P1-P13 |
| **O7** | Availability + dependency health | A1-A10 (10) |
| **O8** | Ingest rate pattern tracking | IR1-IR10 (10) |
| **O9** | Postgres internal observability | PG1-PG33 |
| **O11** | Grafana dashboard | 12 panels |
| **O12** | Prometheus alerting rules | 11 alerts |
| **O13** | Relevance evaluation CronJob | K8s manifest |

### Key Changes

- **metrics.rs**: Zero-dependency Prometheus metrics (Counter, Gauge, Histogram, Timer)
- **GET /metrics**: Prometheus text exposition format endpoint
- **Ingest/Query/Context handlers**: Instrumented with latency, errors, auth failures
- **Health check**: DB dependency check with latency tracking
- **Background task**: Periodic DB stats collection (entity/edge counts, pool stats)
- **Relevance judge**: Threshold-based eval with precision/recall/F1 tracking
- **Grafana dashboard**: 12 panels covering all metric groups
- **Alert rules**: 11 PrometheusRule alerts (availability, latency, errors, quality)
- **CronJob**: Periodic relevance evaluation with sample queries

### Testing

- 506 tests passing (0 failures)
- All metrics modules have unit tests
- Relevance judge: 4 tests

### Deploy

```bash
# Grafana dashboard
kubectl apply -f k8s/infra/grafana-dashboard.json

# Prometheus alerts
kubectl apply -f k8s/infra/prometheus-alerts.yaml

# Relevance eval CronJob
kubectl apply -f k8s/infra/relevance-eval-cronjob.yaml
```

Closes #27 #28 #29 #30 #31 #32 #33 #34 #35 #36 #37 #38 #39

---------

Co-authored-by: rock <[email protected]>
Reviewed-on: #52
Co-authored-by: poimen <[email protected]>
 2026-09-13 13:53:50 +00:00
rock d8c3b06cb0 fix: resolve 75 mem-cli compilation errors
CI / CI (push) Successful in 15m14s
Details 
All errors were API mismatches — handler code calling wrong method
   names, wrong argument types, or missing imports/derives. No logic
   changes. Build now passes with SQLX_OFFLINE=true.

   Key fixes:
   - embed_text -> embed_one, Vector -> Vec<f32> conversion
   - extract_token: extract auth header from HttpRequest first
   - AuthError variants aligned to actual enum definition
   - recursive async fns boxed (dfs_paths in inference + path_finder)
   - missing derives (Default, Serialize), imports (sqlx::Row, Timelike)
   - borrow-after-move: compute .len() before struct field move
   - streaming_body -> streaming with Result<Bytes> for SSE
   - CI: add SQLX_OFFLINE=true for offline builds without DB

   25 files changed, 99 insertions(+), 81 deletions(-)

Co-authored-by: rock <[email protected]>
 2026-09-08 01:11:14 +00:00
rock 985f65d1f4 feat: implement core architecture modules 
Phase 1: Wiki-Link Graph Indexing
- WikiLinkParser: extract [[links]] from markdown
- WikiLinkGraph: BFS traversal, reachable docs, backlinks
- Support relative path resolution (../../../)

Phase 2: ScoringPipeline trait (SOLID design)
- DocumentScorer trait: single interface for all scorers
- GlobalTfIdfScorer, ProjectTfIdfScorer, SemanticScorer
- MetadataBoostingScorer (decorator pattern)
- ScoringPipeline: orchestrate multiple scorers with RRF fusion
- Benefits: add new scorers without modifying existing code

Phase 7: RBAC + PolicyProvider trait
- PolicyProvider trait: pluggable backends (Vault, Postgres, Redis)
- VaultPolicyProvider: load YAML from vault/projects/* and vault/shared/skills/*
- MockPolicyProvider: for testing (no I/O)
- AccessChecker trait: single-purpose RBAC checks
- AccessLevelChecker, RoleChecker, PermissionChecker
- AccessDecisionEngine: orchestrate checkers with short-circuit eval
- AuditLogger trait: pluggable audit backends

Test Fixtures (DRY principle)
- OidcClaimsBuilder: fluent API for test data
- AccessPolicyBuilder: fluent API for policies
- MockPolicyProvider, MockAuditLogger: testing mocks

All modules compile and unit tests pass.
 2026-08-30 20:40:43 -07:00
rock a5ff20c9f7 feat: add 'mem learn' CLI for markdown knowledge ingestion 
6 knowledge files: rust, SOLID/DRY, ast-grep, karpathy, golang, caveman
65 chunks ingested to log/knowledge/learn/latest.jsonl
Chunks on ## headings, SHA256 dedup, configurable chunk size
 2026-08-29 22:04:14 -07:00
rock cd3d00048a feat: M8.2 Gateway Queue Adapter for SQS via api.riotpiao.com 
- Unified QueueAdapter trait for concurrent dual-write operations
- GatewayQueueAdapter routes messages via api.riotpiao.com with X-Service: sqs header
- TokenProvider abstraction: StaticTokenProvider + AuthentikTokenProvider
- JWT bearer token support (from Authentik OAuth2)
- InMemoryQueueAdapter for testing
- Base64 encoding/decoding for SQS message bodies
- HTTP/REST integration (no direct gRPC complexity)
- 8 unit tests + comprehensive documentation
- Supports long-polling (ReceiveMessage), visibility timeout, DLQ

Uses standard SQS API patterns:
- SendMessage: Queue chunk for dual-write processing
- ReceiveMessage: Long-poll up to 10 messages, 20s wait
- DeleteMessage: Acknowledge on success
- ChangeMessageVisibility: Retry on failure
- SendToDLQ: After max retries

Files:
- crates/mem-cli/src/queue_adapter.rs (310 LOC)
- crates/mem-cli/src/gateway_queue_adapter.rs (530 LOC)
- tests/it_gateway_queue_adapter.rs (110 LOC)
- docs/M8.2-GATEWAY_QUEUE_ADAPTER.md (400 LOC)
 2026-08-28 13:11:56 -07:00
Story Crater Bot 47e55afae3 feat: JWT auth validation with Authentik OIDC 
- Add jwt_validator module with JWKS caching (TTL + refresh-on-miss)
- Implement RS256 algorithm pinning + claim validation
- Replace apikey with Bearer token validation in http_server
- Add capability-based access control (memory:read/write/*)
- Backward compatible: MEM_AUTH_MODE=jwt|apikey (default: apikey)
- 16 tests passing (7 unit + 9 integration)
- Docs: JWT_AUTH.md with deployment guide

Config via env vars:
- MEM_AUTH_MODE=jwt
- AUTHENTIK_ISSUER=https://authentik.riotpiao.com/application/o/poimen-memory/
- AUTHENTIK_AUDIENCE=poimen-memory
- JWT_CACHE_TTL_SECS=3600 (optional)

Gw passes Authorization: Bearer <token> header
Memory validates + checks permissions claim
 2026-08-27 12:29:23 -07:00
Story Crater Bot c4fdf36e5f Implement M4.1: Skill draft command + 10 tests (229 total) 2026-08-26 13:50:22 -07:00
rock af6f22217d feat(core): implement full memory pipeline (#11) 2026-08-24 01:37:16 +00:00
Story Crater Bot 457ec85680 Implement M3.5.2: POST /ingest endpoint with idempotent async queue (204 tests) 2026-08-23 16:33:34 -07:00
Story Crater Bot 695e115212 Deploy Poimen Memory K8s cluster with ArgoCD tracking (M2.2, M3.5-M3.7) 2026-08-22 23:13:42 -07:00
Story Crater Bot 33b7150f56 feat: complete M0.1-M0.4 phases 
M0.1 - Cargo workspace + crate skeletons
  - 6-crate workspace with correct dependency direction
  - CI/CD pipeline with GitHub Actions
  - Integration tests verifying build and dependency structure

M0.2 - Domain types and sha256 identity
  - Level (L0, L1, L2) enum with proper serde formatting
  - Role enum (User, Assistant, ToolResult, System)
  - Record, Chunk, and MemoryNode domain types
  - Content-hash identity system ensuring rebuild idempotence
  - Newtypes (ProjectId, QueryId, RunId) with validation
  - Round-trip serde tests for all types

M0.3 - RecordSource trait + ChunkPolicy
  - RecordSource trait for streaming record sources
  - Chunk policy with token budgets and boundary modes
  - TokenCounter trait with CharsOverFourCounter stub
  - Chunking stream that respects budgets without splitting records
  - VecSource for testing
  - Integration tests verifying lossless chunking and budget adherence

M0.4 - Tokenizer-backed chunk sizing
  - Vendored Qwen2 tokenizer with hash verification
  - QwenTokenCounter implementing proper token counting
  - Hash guard that fails on modified tokenizer
  - mem tokens CLI subcommand for token counting
  - Integration tests with known string counts, hash guards, and budget verification

Total: 19 integration tests passing, all phases verified to compose correctly
Workspace builds cleanly with no clippy warnings
 2026-08-22 23:13:42 -07:00