poimen-memory/tasks/M3.5.6-projects-endpoint.md

M3.5.6 — GET /projects and /projects/{id}/status: metadata, metrics, synthesis timestamps

Field	Value
Phase	M3.5 — Distributed API Layer
Size	S — < 1 day
Status	⬜ Not started
Flags	—
Spec	inlined below
Blocks	M3.5.8
Depends	M3.5.1, M2 (projections exist)

Goal

Introspection endpoints for memory state per project. List projects, show metadata, ingest/synthesis history, memory size stats.

Design

List all projects:

GET /memory/projects
→ 200 {
  "projects": [
    {
      "id": "poimen",
      "standing_queries": 3,
      "last_ingest_at": "2026-08-20T10:30:00Z",
      "last_synthesis_at": "2026-08-20T12:00:00Z",
      "total_chunks": 412,
      "total_evidence": 17,
      "memory_size_bytes": 45280
    },
    ...
  ]
}

Get project status:

GET /memory/projects/poimen/status
→ 200 {
  "project_id": "poimen",
  "standing_queries": [
    {
      "id": "infra-root-causes",
      "question": "What infrastructure bugs were found...",
      "last_ingest_at": "2026-08-20T10:30:00Z",
      "chunks_seen": 412,
      "chunks_used": 17,
      "memory_tokens": 142
    },
    ...
  ],
  "l2_synthesis": {
    "last_synthesis_at": "2026-08-20T12:00:00Z",
    "chunks_seen": 3,
    "chunks_used": 2,
    "memory_tokens": 876,
    "exit_gate_fired": true
  },
  "next_synthesis_at": "2026-08-21T12:00:00Z",
  "total_log_size_bytes": 45280,
  "embedding_cache_hits": 234,
  "embedding_cache_misses": 12
}

Metrics

Pull from multiple sources:

• Standing queries: Load from queries/<project>.yaml
• Last ingest: Query JSONL log for most recent run_end record per query_id
• Memory stats: Count nodes in pgvector, sum bytes of text
• L2 synthesis: Query JSONL log for most recent L2 run_end
• Cache stats: Track in-memory (API server state); return per request

Steps

1. GET /memory/projects handler:

   • List all project IDs from queries/ directory
   • For each project:
     • Load queries/<project>.yaml to get standing_queries count
     • Query pgvector: SELECT COUNT(*) FROM memory_node WHERE project = $1
     • Query pgvector: SELECT SUM(LENGTH(text)) FROM memory_node WHERE project = $1
     • Query JSONL log: find most recent L1 run_end to get last_ingest_at
     • Query JSONL log: find most recent L2 run_end to get last_synthesis_at
   • Sort by id and return

2. GET /memory/projects/{id}/status handler:

   • Verify project exists; unknown → 404
   • Load queries/<project>.yaml and parse all queries
   • For each query, query JSONL log:
     • Find most recent run_end record (level L1, query_id = this query's id)
     • Extract chunks_seen, chunks_used, final_memory_tokens, last timestamp
   • Query JSONL log for L2 run_end (level L2, project = id):
     • Extract synthesis metadata, exit_gate fire status
   • Compute next_synthesis_at:
     • If last_synthesis_at + 24h < now, return "immediately"
     • Otherwise, return last_synthesis_at + 24h
   • Assemble response

3. Cache stats:

   • embedding_cache_hits and embedding_cache_misses tracked by embeddings client
   • Expose via Extension<Arc<EmbeddingsClient>> → .stats()
   • Return per request (snapshot at query time)

Acceptance

• List endpoint returns all projects
• Individual project status is queryable
• Metrics are accurate (match log/pgvector state)
• Unknown project returns 404
• Synthesis scheduling shown (next run time)

Verify

Harness: Integration tests with populated JSONL log and pgvector DB.

Integration test — tests/it_projects_endpoint.rs:

1. a1_list_projects — GET /projects returns array with test project(s).
2. a2_project_count_correct — total_chunks field matches pgvector COUNT.
3. a3_project_evidence_count — total_evidence field matches L0 node count for project.
4. a4_get_project_status — GET /projects//status returns 200.
5. a5_standing_queries_listed — standing_queries array in status matches queries YAML.
6. a6_last_ingest_timestamp — last_ingest_at is recent and matches JSONL log.
7. a7_l2_synthesis_metadata — l2_synthesis object contains last_synthesis_at and exit_gate_fired.
8. a8_cache_stats_present — embedding_cache_hits and cache_misses are present and >= 0.
9. a9_next_synthesis_at_scheduled — next_synthesis_at is a valid future timestamp.
10. a10_unknown_project_404 — GET /projects/nonexistent/status returns 404.

Command: cargo test -p mem-cli projects_endpoint

False pass:

• total_chunks hardcoded to a fixed number; never actually counts.
• Cache stats always zero (client doesn't track; endpoint returns fake values).
• Last ingest timestamp never validated against actual log.

Traps

• JSONL log queries are slow for large projects (412 chunks, naive scan). Consider indexing by project_id or caching if >10K chunks.
• Next synthesis scheduling logic is simple (24h interval). If synthesis runs are skipped or delayed, estimate becomes stale. Document the assumption.
• Memory size calculation uses SUM(LENGTH(text)) which is TEXT byte length in DB, not network wire size or actual storage (compression, overhead).

---

Background: DESIGN.md § Distributed API Layer