Files
poimen-memory/crates/mem-core/src/query_levels.rs
T
poimenandrock a88ea918bf
CI / CI (push) Successful in 25m7s
Deploy / Tag & Push Latest (push) Successful in 3m49s
test: production ingest E2E test suite with enhanced logging (#55)
## Summary
Production testing of ingest + embedding pipeline with api-gw integration.

## Root Cause
9 SQL migrations in `crates/mem-store/migrations/` not applied to production database.

Missing tables:
- `memory_entity`
- `memory_edge`
- `memory_edge_temporal`
- Vector embeddings tables
- And 15+ more schema objects

Evidence from logs:
```
WARN: Failed to save entity Docker:
      error returned from database: relation "memory_entity" does not exist
```

## Deliverables
- `test_prod_ingest_real.sh` - Full E2E test against K8s + api-gw
- `apply_migrations.sh` - Manual schema migration (backup)
- `collect_prod_logs.sh` - Pod log collection before/after
- `run_production_test.sh` - Test orchestrator
- `tests/integration_ingest_with_gw.rs` - Integration test
- `tests/unit_ingest_logging.rs` - Unit tests for extraction
- Enhanced logging in `ingest_worker.rs` - Per-record event tracking

## Next Steps
1. Trigger "DB Migration" workflow in Forgejo Actions
2. This applies all 9 migrations from `crates/mem-store/migrations/`
3. Pod restart (automatic)
4. Re-run E2E test - should pass completely

**ETA:** ~15 minutes (3-5 min migrations + 2 min restart + verification)

## How to Test Locally
```bash
./test_prod_ingest_real.sh --verbose
```

Requires:
- kubectl access to poimen namespace
- Port-forwarding to memory-service

---------

Co-authored-by: rock <[email protected]>
Reviewed-on: #55
Co-authored-by: poimen <[email protected]>
2026-09-16 00:10:58 +00:00

161 lines
4.3 KiB
Rust

//! M3.6.5 — Query Levels and Floor: Multi-tier query filtering
//!
//! Allows queries to filter results by level (L0/L1/L2 vs R),
//! set floor thresholds, and control which tiers contribute to responses.
use serde::{Deserialize, Serialize};
/// Query filtering parameters
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct QueryLevels {
/// Include L0/L1/L2 evidence? (default: true)
pub include_evidence: bool,
/// Include R (reference) docs? (default: true)
pub include_reference: bool,
/// Minimum relevance score (0.0-1.0) to include a result
pub floor: f32,
/// Specific levels to include (empty = all)
pub level_filter: Vec<String>, // ["L1", "L2"] or empty for all
}
impl Default for QueryLevels {
fn default() -> Self {
Self {
include_evidence: true,
include_reference: false, // R opt-in only
floor: 0.0,
level_filter: vec![],
}
}
}
impl QueryLevels {
/// Create evidence-only filter (R excluded by default)
pub fn evidence_only() -> Self {
Self {
include_evidence: true,
include_reference: false,
floor: 0.0,
level_filter: vec![],
}
}
/// Create reference-only filter
pub fn reference_only() -> Self {
Self {
include_evidence: false,
include_reference: true,
floor: 0.0,
level_filter: vec!["R".to_string()],
}
}
/// Create hybrid (both evidence and reference)
pub fn hybrid(floor: f32) -> Self {
Self {
include_evidence: true,
include_reference: true,
floor,
level_filter: vec![],
}
}
/// Check if a result should be included
pub fn should_include(&self, level: &str, score: f32) -> bool {
// Check floor threshold first
if score < self.floor {
return false;
}
// Check level filter
if !self.level_filter.is_empty()
&& !self.level_filter.contains(&level.to_string()) {
return false;
}
// Check evidence/reference flags
if level == "R" {
return self.include_reference;
} else if level.starts_with("L") {
return self.include_evidence;
}
false
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_default_levels() {
let levels = QueryLevels::default();
assert!(levels.include_evidence);
assert!(!levels.include_reference);
}
#[test]
fn test_evidence_only_filter() {
let levels = QueryLevels::evidence_only();
assert!(levels.should_include("L1", 0.9));
assert!(levels.should_include("L2", 0.7));
assert!(!levels.should_include("R", 0.99));
}
#[test]
fn test_reference_only_filter() {
let levels = QueryLevels::reference_only();
assert!(!levels.should_include("L1", 0.9));
assert!(levels.should_include("R", 0.5));
}
#[test]
fn test_hybrid_filter() {
let levels = QueryLevels::hybrid(0.6);
assert!(levels.should_include("L1", 0.8));
assert!(levels.should_include("R", 0.7));
assert!(!levels.should_include("L0", 0.5)); // Below floor
}
#[test]
fn test_floor_threshold() {
let levels = QueryLevels {
include_evidence: true,
include_reference: true,
floor: 0.75,
level_filter: vec![],
};
assert!(levels.should_include("L1", 0.80));
assert!(!levels.should_include("L1", 0.70));
}
#[test]
fn test_level_filter_specific() {
let levels = QueryLevels {
include_evidence: true,
include_reference: true,
floor: 0.0,
level_filter: vec!["L1".to_string(), "L2".to_string()],
};
assert!(levels.should_include("L1", 0.5));
assert!(levels.should_include("L2", 0.5));
assert!(!levels.should_include("L0", 0.9)); // Filtered out
}
#[test]
fn test_r_opt_in() {
// R should require explicit opt-in
let default_levels = QueryLevels::default();
assert!(!default_levels.should_include("R", 1.0));
let with_r = QueryLevels::hybrid(0.0);
assert!(with_r.should_include("R", 1.0));
}
}