riotpiao-poimen/poimen-memory
2
0
Fork 0
Code Issues 13 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

O4: Implement Qwen-7B relevance judge (R1-R9) #30

New Issue
Open
opened 2026-09-08 05:00:26 +00:00 by rock · 0 comments
rock commented 2026-09-08 05:00:26 +00:00
Owner
Copy Link
Copy Source

Daily batch evaluation of retrieval quality using Qwen-7B as automated relevance judge.

Tasks:

  1. Add mem evaluate-relevance CLI command
  2. Sample 500 queries from 24h query log
  3. Replay each query, get top-10 results
  4. For each (query, result) pair, call Qwen-7B with relevance prompt (score 0/1/2)
  5. Compute NDCG@10, MRR, Precision@10, Recall@10
  6. Store daily results in Postgres (time-series table)
  7. Expose as Prometheus gauges

Metrics:

  • R1: relevance_sample_size (Gauge)
  • R3: relevance_judge_duration_seconds (Histogram)
  • R3a: relevance_judge_cost_usd (Counter)
  • R4: relevance_score_distribution (Histogram)
  • R5: relevance_ndcg_10 (Gauge) — PRIMARY QUALITY METRIC, SLO >= 0.85
  • R6: relevance_mrr (Gauge)
  • R7: relevance_precision_10 (Gauge)
  • R8: relevance_recall_10 (Gauge)
  • R9: relevance_judge_agreement (Gauge, weekly human validation)

Deploy as K8s CronJob (03:00 UTC daily), Qwen endpoint: ollama.poimen.svc:11434

Reference: docs/OBSERVABILITY.md Section 4

Daily batch evaluation of retrieval quality using Qwen-7B as automated relevance judge. Tasks: 1. Add `mem evaluate-relevance` CLI command 2. Sample 500 queries from 24h query log 3. Replay each query, get top-10 results 4. For each (query, result) pair, call Qwen-7B with relevance prompt (score 0/1/2) 5. Compute NDCG@10, MRR, Precision@10, Recall@10 6. Store daily results in Postgres (time-series table) 7. Expose as Prometheus gauges Metrics: - R1: relevance_sample_size (Gauge) - R3: relevance_judge_duration_seconds (Histogram) - R3a: relevance_judge_cost_usd (Counter) - R4: relevance_score_distribution (Histogram) - R5: relevance_ndcg_10 (Gauge) — PRIMARY QUALITY METRIC, SLO >= 0.85 - R6: relevance_mrr (Gauge) - R7: relevance_precision_10 (Gauge) - R8: relevance_recall_10 (Gauge) - R9: relevance_judge_agreement (Gauge, weekly human validation) Deploy as K8s CronJob (03:00 UTC daily), Qwen endpoint: ollama.poimen.svc:11434 Reference: docs/OBSERVABILITY.md Section 4
rock added the memory-observability label 2026-09-08 05:00:26 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
memory-observability
Observability implementation tasks
No labels memory-observability
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
poimen rock (Rock)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: riotpiao-poimen/poimen-memory#30