poimen-memory/tasks/M3.5.6-projects-endpoint.md

# M3.5.6 — GET /projects and /projects/{id}/status: metadata, metrics, synthesis timestamps

| Field | Value |
|---|---|
| Phase | M3.5 — Distributed API Layer |
| Size | S — < 1 day |
| Status | ✅ Done |
| Flags | — |
| Spec | inlined below |
| Blocks | M3.5.8 |
| Depends | M3.5.1, M2 (projections exist) |

## Goal

Introspection endpoints for memory state per project. List projects, show metadata, ingest/synthesis history, memory size stats.

## Design

**List all projects:**
```
GET /memory/projects
→ 200 {
  "projects": [
    {
      "id": "poimen",
      "standing_queries": 3,
      "last_ingest_at": "2026-08-20T10:30:00Z",
      "last_synthesis_at": "2026-08-20T12:00:00Z",
      "total_chunks": 412,
      "total_evidence": 17,
      "memory_size_bytes": 45280
    },
    ...
  ]
}
```

**Get project status:**
```
GET /memory/projects/poimen/status
→ 200 {
  "project_id": "poimen",
  "standing_queries": [
    {
      "id": "infra-root-causes",
      "question": "What infrastructure bugs were found...",
      "last_ingest_at": "2026-08-20T10:30:00Z",
      "chunks_seen": 412,
      "chunks_used": 17,
      "memory_tokens": 142
    },
    ...
  ],
  "l2_synthesis": {
    "last_synthesis_at": "2026-08-20T12:00:00Z",
    "chunks_seen": 3,
    "chunks_used": 2,
    "memory_tokens": 876,
    "exit_gate_fired": true
  },
  "next_synthesis_at": "2026-08-21T12:00:00Z",
  "total_log_size_bytes": 45280,
  "embedding_cache_hits": 234,
  "embedding_cache_misses": 12
}
```

## Metrics

Pull from multiple sources:
- **Standing queries:** Load from `queries/<project>.yaml`
- **Last ingest:** Query JSONL log for most recent `run_end` record per query_id
- **Memory stats:** Count nodes in pgvector, sum bytes of text
- **L2 synthesis:** Query JSONL log for most recent L2 `run_end`
- **Cache stats:** Track in-memory (API server state); return per request

## Steps

1. `GET /memory/projects` handler:
   - List all project IDs from `queries/` directory
   - For each project:
     - Load `queries/<project>.yaml` to get standing_queries count
     - Query pgvector: `SELECT COUNT(*) FROM memory_node WHERE project = $1`
     - Query pgvector: `SELECT SUM(LENGTH(text)) FROM memory_node WHERE project = $1`
     - Query JSONL log: find most recent L1 `run_end` to get last_ingest_at
     - Query JSONL log: find most recent L2 `run_end` to get last_synthesis_at
   - Sort by id and return

2. `GET /memory/projects/{id}/status` handler:
   - Verify project exists; unknown → 404
   - Load `queries/<project>.yaml` and parse all queries
   - For each query, query JSONL log:
     - Find most recent `run_end` record (level L1, query_id = this query's id)
     - Extract chunks_seen, chunks_used, final_memory_tokens, last timestamp
   - Query JSONL log for L2 run_end (level L2, project = id):
     - Extract synthesis metadata, exit_gate fire status
   - Compute next_synthesis_at:
     - If last_synthesis_at + 24h < now, return "immediately"
     - Otherwise, return last_synthesis_at + 24h
   - Assemble response

3. Cache stats:
   - `embedding_cache_hits` and `embedding_cache_misses` tracked by embeddings client
   - Expose via `Extension<Arc<EmbeddingsClient>>` → `.stats()`
   - Return per request (snapshot at query time)

## Acceptance

- List endpoint returns all projects
- Individual project status is queryable
- Metrics are accurate (match log/pgvector state)
- Unknown project returns 404
- Synthesis scheduling shown (next run time)

## Verify

**Harness:** Integration tests with populated JSONL log and pgvector DB.

**Integration test** — `tests/it_projects_endpoint.rs`:
1. `a1_list_projects` — GET /projects returns array with test project(s).
2. `a2_project_count_correct` — total_chunks field matches pgvector COUNT.
3. `a3_project_evidence_count` — total_evidence field matches L0 node count for project.
4. `a4_get_project_status` — GET /projects/<id>/status returns 200.
5. `a5_standing_queries_listed` — standing_queries array in status matches queries YAML.
6. `a6_last_ingest_timestamp` — last_ingest_at is recent and matches JSONL log.
7. `a7_l2_synthesis_metadata` — l2_synthesis object contains last_synthesis_at and exit_gate_fired.
8. `a8_cache_stats_present` — embedding_cache_hits and cache_misses are present and >= 0.
9. `a9_next_synthesis_at_scheduled` — next_synthesis_at is a valid future timestamp.
10. `a10_unknown_project_404` — GET /projects/nonexistent/status returns 404.

**Command:** `cargo test -p mem-cli projects_endpoint`

**False pass:**
- total_chunks hardcoded to a fixed number; never actually counts.
- Cache stats always zero (client doesn't track; endpoint returns fake values).
- Last ingest timestamp never validated against actual log.

## Traps

- JSONL log queries are slow for large projects (412 chunks, naive scan). Consider indexing by project_id or caching if >10K chunks.
- Next synthesis scheduling logic is simple (24h interval). If synthesis runs are skipped or delayed, estimate becomes stale. Document the assumption.
- Memory size calculation uses SUM(LENGTH(text)) which is TEXT byte length in DB, not network wire size or actual storage (compression, overhead).

---

Background: [DESIGN.md § Distributed API Layer](../DESIGN.md#distributed-api-layer-homelab-frontend)