M3.5.6 — GET /projects and /projects/{id}/status: metadata, metrics, synthesis timestamps

Field	Value
Phase	M3.5 — Distributed API Layer
Size	S — < 1 day
Status	✅ Done
Flags	—
Spec	inlined below
Blocks	M3.5.8
Depends	M3.5.1, M2 (projections exist)

Goal

Introspection endpoints for memory state per project. List projects, show metadata, ingest/synthesis history, memory size stats.

Design

List all projects:

GET /memory/projects
→ 200 {
  "projects": [
    {
      "id": "poimen",
      "standing_queries": 3,
      "last_ingest_at": "2026-08-20T10:30:00Z",
      "last_synthesis_at": "2026-08-20T12:00:00Z",
      "total_chunks": 412,
      "total_evidence": 17,
      "memory_size_bytes": 45280
    },
    ...
  ]
}

Get project status:

GET /memory/projects/poimen/status
→ 200 {
  "project_id": "poimen",
  "standing_queries": [
    {
      "id": "infra-root-causes",
      "question": "What infrastructure bugs were found...",
      "last_ingest_at": "2026-08-20T10:30:00Z",
      "chunks_seen": 412,
      "chunks_used": 17,
      "memory_tokens": 142
    },
    ...
  ],
  "l2_synthesis": {
    "last_synthesis_at": "2026-08-20T12:00:00Z",
    "chunks_seen": 3,
    "chunks_used": 2,
    "memory_tokens": 876,
    "exit_gate_fired": true
  },
  "next_synthesis_at": "2026-08-21T12:00:00Z",
  "total_log_size_bytes": 45280,
  "embedding_cache_hits": 234,
  "embedding_cache_misses": 12
}

Metrics

Pull from multiple sources:

Standing queries: Load from queries/<project>.yaml
Last ingest: Query JSONL log for most recent run_end record per query_id
Memory stats: Count nodes in pgvector, sum bytes of text
L2 synthesis: Query JSONL log for most recent L2 run_end
Cache stats: Track in-memory (API server state); return per request

Steps

GET /memory/projects handler:
- List all project IDs from queries/ directory
- For each project:
  - Load queries/<project>.yaml to get standing_queries count
  - Query pgvector: SELECT COUNT(*) FROM memory_node WHERE project = $1
  - Query pgvector: SELECT SUM(LENGTH(text)) FROM memory_node WHERE project = $1
  - Query JSONL log: find most recent L1 run_end to get last_ingest_at
  - Query JSONL log: find most recent L2 run_end to get last_synthesis_at
- Sort by id and return
GET /memory/projects/{id}/status handler:
- Verify project exists; unknown → 404
- Load queries/<project>.yaml and parse all queries
- For each query, query JSONL log:
  - Find most recent run_end record (level L1, query_id = this query's id)
  - Extract chunks_seen, chunks_used, final_memory_tokens, last timestamp
- Query JSONL log for L2 run_end (level L2, project = id):
  - Extract synthesis metadata, exit_gate fire status
- Compute next_synthesis_at:
  - If last_synthesis_at + 24h < now, return "immediately"
  - Otherwise, return last_synthesis_at + 24h
- Assemble response
Cache stats:
- embedding_cache_hits and embedding_cache_misses tracked by embeddings client
- Expose via Extension<Arc<EmbeddingsClient>> → .stats()
- Return per request (snapshot at query time)

Acceptance

List endpoint returns all projects
Individual project status is queryable
Metrics are accurate (match log/pgvector state)
Unknown project returns 404
Synthesis scheduling shown (next run time)

Verify

Harness: Integration tests with populated JSONL log and pgvector DB.

Integration test — tests/it_projects_endpoint.rs:

a1_list_projects — GET /projects returns array with test project(s).
a2_project_count_correct — total_chunks field matches pgvector COUNT.
a3_project_evidence_count — total_evidence field matches L0 node count for project.
a4_get_project_status — GET /projects//status returns 200.
a5_standing_queries_listed — standing_queries array in status matches queries YAML.
a6_last_ingest_timestamp — last_ingest_at is recent and matches JSONL log.
a7_l2_synthesis_metadata — l2_synthesis object contains last_synthesis_at and exit_gate_fired.
a8_cache_stats_present — embedding_cache_hits and cache_misses are present and >= 0.
a9_next_synthesis_at_scheduled — next_synthesis_at is a valid future timestamp.
a10_unknown_project_404 — GET /projects/nonexistent/status returns 404.

Command: cargo test -p mem-cli projects_endpoint

False pass:

total_chunks hardcoded to a fixed number; never actually counts.
Cache stats always zero (client doesn't track; endpoint returns fake values).
Last ingest timestamp never validated against actual log.

Traps

JSONL log queries are slow for large projects (412 chunks, naive scan). Consider indexing by project_id or caching if >10K chunks.
Next synthesis scheduling logic is simple (24h interval). If synthesis runs are skipped or delayed, estimate becomes stale. Document the assumption.
Memory size calculation uses SUM(LENGTH(text)) which is TEXT byte length in DB, not network wire size or actual storage (compression, overhead).

Background: DESIGN.md § Distributed API Layer

5.2 KiB Raw Blame History