# Query-Aware Metrics Tracking — M3.8 Output Examples

Track optimization progress and metrics per `query_id`, allowing clients to monitor compression ratios, latency, and progress in real-time.

## Quick Start: API Usage

### 1. Create Query Metrics

```rust
use mem_ingest::QueryMetricsRepository;

let repo = QueryMetricsRepository::new();

// Start tracking a query's optimization
let query_id = repo.create_query("query-20250127-abc123", "myproject");
println!("Created metrics for: {}", query_id);
```

### 2. Record Progress During Optimization

```rust
// Simulate optimization happening
repo.update_metrics(&query_id, |metrics| {
    metrics.total_records = 1500; // Expected total
    metrics.status = OptimizationStatus::InProgress;
}).unwrap();

// As records are optimized, record them
repo.update_metrics(&query_id, |metrics| {
    metrics.record_record_optimized("log_compressor", "text/plain", 1024, 256);
}).unwrap();

repo.update_metrics(&query_id, |metrics| {
    metrics.record_record_optimized("text_compressor", "text/plain", 512, 300);
}).unwrap();

// ... more records ...

repo.update_metrics(&query_id, |metrics| {
    metrics.status = OptimizationStatus::Completed;
}).unwrap();
```

### 3. Query Progress (Real-Time)

```rust
// Get current progress
let progress = repo.get_progress(&query_id).unwrap();
println!("{:.1}% complete", progress.percent_complete);
println!("Records: {}/{}", progress.records_completed, progress.total_records);
println!("Compression: {:.1}%", progress.compression_ratio);
```

### 4. Get Final Summary

```rust
let metrics = repo.get_metrics(&query_id).unwrap();
let summary = metrics.to_summary();
println!("{}", serde_json::to_string_pretty(&summary).unwrap());
```

---

## Sample Output Examples

### Progress Snapshot (Real-Time Monitoring)

**25% Complete:**

```json
{
  "query_id": "query-20250127-abc123",
  "project": "myproject",
  "status": "InProgress",
  "percent_complete": 25.0,
  "records_completed": 375,
  "total_records": 1500,
  "compression_ratio": 28.4,
  "input_bytes": 10485760,
  "output_bytes": 2973696,
  "eta_secs": 180
}
```

**50% Complete:**

```json
{
  "query_id": "query-20250127-abc123",
  "project": "myproject",
  "status": "InProgress",
  "percent_complete": 50.0,
  "records_completed": 750,
  "total_records": 1500,
  "compression_ratio": 29.7,
  "input_bytes": 20971520,
  "output_bytes": 6229197,
  "eta_secs": 90
}
```

**100% Complete:**

```json
{
  "query_id": "query-20250127-abc123",
  "project": "myproject",
  "status": "Completed",
  "percent_complete": 100.0,
  "records_completed": 1500,
  "total_records": 1500,
  "compression_ratio": 30.1,
  "input_bytes": 41943040,
  "output_bytes": 12633697,
  "eta_secs": null
}
```

### Final Metrics Summary (Complete)

```json
{
  "query_id": "query-20250127-abc123",
  "project": "myproject",
  "started_at": "2025-01-27T14:35:42.123456Z",
  "total_records": 1500,
  "input_bytes_total": 41943040,
  "output_bytes_total": 12633697,
  "compression_ratio": 30.1,
  "per_compressor": {
    "log_compressor": {
      "count": 750,
      "input_bytes": 20971520,
      "output_bytes": 2097152,
      "compression_ratio": 10.0
    },
    "text_compressor": {
      "count": 600,
      "input_bytes": 15728640,
      "output_bytes": 8388608,
      "compression_ratio": 53.3
    },
    "json_compressor": {
      "count": 150,
      "input_bytes": 5242880,
      "output_bytes": 2147937,
      "compression_ratio": 40.9
    }
  },
  "per_content_type": {
    "text/plain": {
      "count": 900,
      "input_bytes": 26214400,
      "output_bytes": 7864320,
      "compression_ratio": 30.0
    },
    "application/json": {
      "count": 450,
      "input_bytes": 10485760,
      "output_bytes": 4287360,
      "compression_ratio": 40.8
    },
    "application/xml": {
      "count": 150,
      "input_bytes": 5242880,
      "output_bytes": 1481017,
      "compression_ratio": 28.2
    }
  },
  "status": "Completed",
  "error": null
}
```

---

## HTTP API Integration Examples

### GET /memory/query/metrics/{query_id}

Get current progress for a specific query:

```bash
curl http://localhost:8080/memory/query/metrics/query-20250127-abc123
```

**Response (In Progress):**

```json
{
  "data": {
    "query_id": "query-20250127-abc123",
    "project": "myproject",
    "status": "InProgress",
    "percent_complete": 45.2,
    "records_completed": 678,
    "total_records": 1500,
    "compression_ratio": 29.5,
    "input_bytes": 35651584,
    "output_bytes": 10517267,
    "eta_secs": 95
  },
  "timestamp": "2025-01-27T14:36:15Z"
}
```

**Response (Completed):**

```json
{
  "data": {
    "query_id": "query-20250127-abc123",
    "project": "myproject",
    "status": "Completed",
    "percent_complete": 100.0,
    "records_completed": 1500,
    "total_records": 1500,
    "compression_ratio": 30.1,
    "input_bytes": 41943040,
    "output_bytes": 12633697,
    "eta_secs": null
  },
  "timestamp": "2025-01-27T14:37:45Z"
}
```

### GET /memory/query/metrics/{query_id}/summary

Get final summary after completion:

```bash
curl http://localhost:8080/memory/query/metrics/query-20250127-abc123/summary
```

**Response:**

```json
{
  "data": {
    "query_id": "query-20250127-abc123",
    "project": "myproject",
    "started_at": "2025-01-27T14:35:42.123456Z",
    "total_records": 1500,
    "input_bytes_total": 41943040,
    "output_bytes_total": 12633697,
    "compression_ratio": 30.1,
    "per_compressor": {
      "log_compressor": {
        "count": 750,
        "input_bytes": 20971520,
        "output_bytes": 2097152,
        "compression_ratio": 10.0
      },
      "text_compressor": {
        "count": 600,
        "input_bytes": 15728640,
        "output_bytes": 8388608,
        "compression_ratio": 53.3
      },
      "json_compressor": {
        "count": 150,
        "input_bytes": 5242880,
        "output_bytes": 2147937,
        "compression_ratio": 40.9
      }
    },
    "per_content_type": {
      "text/plain": {
        "count": 900,
        "input_bytes": 26214400,
        "output_bytes": 7864320,
        "compression_ratio": 30.0
      },
      "application/json": {
        "count": 450,
        "input_bytes": 10485760,
        "output_bytes": 4287360,
        "compression_ratio": 40.8
      },
      "application/xml": {
        "count": 150,
        "input_bytes": 5242880,
        "output_bytes": 1481017,
        "compression_ratio": 28.2
      }
    },
    "status": "Completed",
    "error": null
  },
  "duration_secs": 123,
  "timestamp": "2025-01-27T14:37:45Z"
}
```

### GET /memory/query/metrics/project/{project}

Get all queries for a project:

```bash
curl http://localhost:8080/memory/query/metrics/project/myproject
```

**Response:**

```json
{
  "data": [
    {
      "query_id": "query-20250127-abc123",
      "project": "myproject",
      "status": "Completed",
      "percent_complete": 100.0,
      "records_completed": 1500,
      "total_records": 1500,
      "compression_ratio": 30.1
    },
    {
      "query_id": "query-20250127-def456",
      "project": "myproject",
      "status": "InProgress",
      "percent_complete": 62.3,
      "records_completed": 934,
      "total_records": 1500,
      "compression_ratio": 31.5
    },
    {
      "query_id": "query-20250127-ghi789",
      "project": "myproject",
      "status": "Pending",
      "percent_complete": 0.0,
      "records_completed": 0,
      "total_records": 1500,
      "compression_ratio": 0.0
    }
  ],
  "count": 3,
  "timestamp": "2025-01-27T14:37:45Z"
}
```

---

## Structured Logging Output

### Progress Logging (During Optimization)

```
2025-01-27T14:35:42Z INFO mem_ingest::query_metrics
  query_id=query-20250127-abc123
  project=myproject
  status=InProgress
  percent_complete=5.0
  records_completed=75
  total_records=1500
  compression_ratio=28.2
  input_bytes=4194304
  output_bytes=1182989
  message="Query optimization progress"
```

### Per-Compressor Progress

```
2025-01-27T14:35:43Z DEBUG mem_ingest::query_metrics
  query_id=query-20250127-abc123
  compressor=log_compressor
  count=37
  input_bytes=2097152
  output_bytes=209715
  compression_ratio=10.0
  message="Compressor progress update"
```

### Per-Content-Type Progress

```
2025-01-27T14:35:44Z DEBUG mem_ingest::query_metrics
  query_id=query-20250127-abc123
  content_type=text/plain
  count=45
  input_bytes=2621440
  output_bytes=786432
  compression_ratio=30.0
  message="Content type progress update"
```

### Completion Logging

```
2025-01-27T14:37:45Z INFO mem_ingest::query_metrics
  query_id=query-20250127-abc123
  project=myproject
  status=Completed
  total_records=1500
  input_bytes_total=41943040
  output_bytes_total=12633697
  compression_ratio=30.1
  duration_secs=123
  message="Query optimization complete"
```

### Per-Compressor Summary

```
2025-01-27T14:37:45Z INFO mem_ingest::query_metrics
  query_id=query-20250127-abc123
  compressor=log_compressor
  count=750
  input_bytes=20971520
  output_bytes=2097152
  compression_ratio=10.0
  message="Compressor summary"

2025-01-27T14:37:45Z INFO mem_ingest::query_metrics
  query_id=query-20250127-abc123
  compressor=text_compressor
  count=600
  input_bytes=15728640
  output_bytes=8388608
  compression_ratio=53.3
  message="Compressor summary"

2025-01-27T14:37:45Z INFO mem_ingest::query_metrics
  query_id=query-20250127-abc123
  compressor=json_compressor
  count=150
  input_bytes=5242880
  output_bytes=2147937
  compression_ratio=40.9
  message="Compressor summary"
```

---

## Prometheus Metrics (Exported)

```
# HELP mem_query_optimization_records_total Total records optimized
# TYPE mem_query_optimization_records_total counter
mem_query_optimization_records_total{query_id="query-20250127-abc123",project="myproject"} 1500.0

# HELP mem_query_optimization_bytes_input Total input bytes
# TYPE mem_query_optimization_bytes_input gauge
mem_query_optimization_bytes_input{query_id="query-20250127-abc123",project="myproject"} 41943040.0

# HELP mem_query_optimization_bytes_output Total output bytes
# TYPE mem_query_optimization_bytes_output gauge
mem_query_optimization_bytes_output{query_id="query-20250127-abc123",project="myproject"} 12633697.0

# HELP mem_query_optimization_compression_ratio Compression ratio (%)
# TYPE mem_query_optimization_compression_ratio gauge
mem_query_optimization_compression_ratio{query_id="query-20250127-abc123",project="myproject"} 30.1

# HELP mem_query_optimization_duration_secs Duration in seconds
# TYPE mem_query_optimization_duration_secs histogram
mem_query_optimization_duration_secs_bucket{query_id="query-20250127-abc123",project="myproject",le="10"} 0.0
mem_query_optimization_duration_secs_bucket{query_id="query-20250127-abc123",project="myproject",le="50"} 0.0
mem_query_optimization_duration_secs_bucket{query_id="query-20250127-abc123",project="myproject",le="100"} 0.0
mem_query_optimization_duration_secs_bucket{query_id="query-20250127-abc123",project="myproject",le="500"} 1.0
mem_query_optimization_duration_secs_bucket{query_id="query-20250127-abc123",project="myproject",le="+Inf"} 1.0
mem_query_optimization_duration_secs_sum{query_id="query-20250127-abc123",project="myproject"} 123.45
mem_query_optimization_duration_secs_count{query_id="query-20250127-abc123",project="myproject"} 1.0

# HELP mem_query_optimization_compressor_ratio Compression ratio by compressor (%)
# TYPE mem_query_optimization_compressor_ratio gauge
mem_query_optimization_compressor_ratio{query_id="query-20250127-abc123",project="myproject",compressor="log_compressor"} 10.0
mem_query_optimization_compressor_ratio{query_id="query-20250127-abc123",project="myproject",compressor="text_compressor"} 53.3
mem_query_optimization_compressor_ratio{query_id="query-20250127-abc123",project="myproject",compressor="json_compressor"} 40.9
```

---

## CLI Usage Example

### Monitor Query Progress

```bash
#!/bin/bash
# Watch query optimization progress in real-time

QUERY_ID="query-20250127-abc123"
PROJECT="myproject"

while true; do
  PROGRESS=$(curl -s "http://localhost:8080/memory/query/metrics/$QUERY_ID")
  
  STATUS=$(echo $PROGRESS | jq -r '.data.status')
  PERCENT=$(echo $PROGRESS | jq -r '.data.percent_complete')
  RATIO=$(echo $PROGRESS | jq -r '.data.compression_ratio')
  ETA=$(echo $PROGRESS | jq -r '.data.eta_secs')
  
  clear
  echo "Query: $QUERY_ID"
  echo "Project: $PROJECT"
  echo "Status: $STATUS"
  echo "Progress: ${PERCENT}%"
  echo "Compression: ${RATIO}%"
  echo "ETA: ${ETA}s"
  
  if [ "$STATUS" = "Completed" ]; then
    break
  fi
  
  sleep 2
done

# Get final summary
echo ""
echo "Final Summary:"
curl -s "http://localhost:8080/memory/query/metrics/$QUERY_ID/summary" | jq .
```

**Output:**

```
Query: query-20250127-abc123
Project: myproject
Status: InProgress
Progress: 45.2%
Compression: 29.5%
ETA: 95s

--- (after completion) ---

Query: query-20250127-abc123
Project: myproject
Status: Completed
Progress: 100.0%
Compression: 30.1%
ETA: null

Final Summary:
{
  "data": {
    "query_id": "query-20250127-abc123",
    "project": "myproject",
    ...
  },
  "duration_secs": 123,
  "timestamp": "2025-01-27T14:37:45Z"
}
```

---

## Key Takeaways

✅ **Per-Query Tracking**: Metrics indexed by `query_id`  
✅ **Real-Time Progress**: `percent_complete`, `eta_secs` for monitoring  
✅ **Detailed Breakdown**: Per-compressor and per-content-type statistics  
✅ **Multiple Output Formats**: JSON APIs, structured logs, Prometheus metrics  
✅ **Production Ready**: Thread-safe repository, idempotent updates  
✅ **Easy Integration**: Drop-in to rebuild.rs and query handlers