feat: complete M0.1-M0.4 phases
M0.1 - Cargo workspace + crate skeletons - 6-crate workspace with correct dependency direction - CI/CD pipeline with GitHub Actions - Integration tests verifying build and dependency structure M0.2 - Domain types and sha256 identity - Level (L0, L1, L2) enum with proper serde formatting - Role enum (User, Assistant, ToolResult, System) - Record, Chunk, and MemoryNode domain types - Content-hash identity system ensuring rebuild idempotence - Newtypes (ProjectId, QueryId, RunId) with validation - Round-trip serde tests for all types M0.3 - RecordSource trait + ChunkPolicy - RecordSource trait for streaming record sources - Chunk policy with token budgets and boundary modes - TokenCounter trait with CharsOverFourCounter stub - Chunking stream that respects budgets without splitting records - VecSource for testing - Integration tests verifying lossless chunking and budget adherence M0.4 - Tokenizer-backed chunk sizing - Vendored Qwen2 tokenizer with hash verification - QwenTokenCounter implementing proper token counting - Hash guard that fails on modified tokenizer - mem tokens CLI subcommand for token counting - Integration tests with known string counts, hash guards, and budget verification Total: 19 integration tests passing, all phases verified to compose correctly Workspace builds cleanly with no clippy warnings
This commit is contained in:
Story Crater Bot
@@ -0,0 +1,146 @@
|
||||
# M3.5.6 — GET /projects and /projects/{id}/status: metadata, metrics, synthesis timestamps
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M3.5 — Distributed API Layer |
|
||||
| Size | S — < 1 day |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M3.5.8 |
|
||||
| Depends | M3.5.1, M2 (projections exist) |
|
||||
|
||||
## Goal
|
||||
|
||||
Introspection endpoints for memory state per project. List projects, show metadata, ingest/synthesis history, memory size stats.
|
||||
|
||||
## Design
|
||||
|
||||
**List all projects:**
|
||||
```
|
||||
GET /memory/projects
|
||||
→ 200 {
|
||||
"projects": [
|
||||
{
|
||||
"id": "poimen",
|
||||
"standing_queries": 3,
|
||||
"last_ingest_at": "2026-08-20T10:30:00Z",
|
||||
"last_synthesis_at": "2026-08-20T12:00:00Z",
|
||||
"total_chunks": 412,
|
||||
"total_evidence": 17,
|
||||
"memory_size_bytes": 45280
|
||||
},
|
||||
...
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Get project status:**
|
||||
```
|
||||
GET /memory/projects/poimen/status
|
||||
→ 200 {
|
||||
"project_id": "poimen",
|
||||
"standing_queries": [
|
||||
{
|
||||
"id": "infra-root-causes",
|
||||
"question": "What infrastructure bugs were found...",
|
||||
"last_ingest_at": "2026-08-20T10:30:00Z",
|
||||
"chunks_seen": 412,
|
||||
"chunks_used": 17,
|
||||
"memory_tokens": 142
|
||||
},
|
||||
...
|
||||
],
|
||||
"l2_synthesis": {
|
||||
"last_synthesis_at": "2026-08-20T12:00:00Z",
|
||||
"chunks_seen": 3,
|
||||
"chunks_used": 2,
|
||||
"memory_tokens": 876,
|
||||
"exit_gate_fired": true
|
||||
},
|
||||
"next_synthesis_at": "2026-08-21T12:00:00Z",
|
||||
"total_log_size_bytes": 45280,
|
||||
"embedding_cache_hits": 234,
|
||||
"embedding_cache_misses": 12
|
||||
}
|
||||
```
|
||||
|
||||
## Metrics
|
||||
|
||||
Pull from multiple sources:
|
||||
- **Standing queries:** Load from `queries/<project>.yaml`
|
||||
- **Last ingest:** Query JSONL log for most recent `run_end` record per query_id
|
||||
- **Memory stats:** Count nodes in pgvector, sum bytes of text
|
||||
- **L2 synthesis:** Query JSONL log for most recent L2 `run_end`
|
||||
- **Cache stats:** Track in-memory (API server state); return per request
|
||||
|
||||
## Steps
|
||||
|
||||
1. `GET /memory/projects` handler:
|
||||
- List all project IDs from `queries/` directory
|
||||
- For each project:
|
||||
- Load `queries/<project>.yaml` to get standing_queries count
|
||||
- Query pgvector: `SELECT COUNT(*) FROM memory_node WHERE project = $1`
|
||||
- Query pgvector: `SELECT SUM(LENGTH(text)) FROM memory_node WHERE project = $1`
|
||||
- Query JSONL log: find most recent L1 `run_end` to get last_ingest_at
|
||||
- Query JSONL log: find most recent L2 `run_end` to get last_synthesis_at
|
||||
- Sort by id and return
|
||||
|
||||
2. `GET /memory/projects/{id}/status` handler:
|
||||
- Verify project exists; unknown → 404
|
||||
- Load `queries/<project>.yaml` and parse all queries
|
||||
- For each query, query JSONL log:
|
||||
- Find most recent `run_end` record (level L1, query_id = this query's id)
|
||||
- Extract chunks_seen, chunks_used, final_memory_tokens, last timestamp
|
||||
- Query JSONL log for L2 run_end (level L2, project = id):
|
||||
- Extract synthesis metadata, exit_gate fire status
|
||||
- Compute next_synthesis_at:
|
||||
- If last_synthesis_at + 24h < now, return "immediately"
|
||||
- Otherwise, return last_synthesis_at + 24h
|
||||
- Assemble response
|
||||
|
||||
3. Cache stats:
|
||||
- `embedding_cache_hits` and `embedding_cache_misses` tracked by embeddings client
|
||||
- Expose via `Extension<Arc<EmbeddingsClient>>` → `.stats()`
|
||||
- Return per request (snapshot at query time)
|
||||
|
||||
## Acceptance
|
||||
|
||||
- List endpoint returns all projects
|
||||
- Individual project status is queryable
|
||||
- Metrics are accurate (match log/pgvector state)
|
||||
- Unknown project returns 404
|
||||
- Synthesis scheduling shown (next run time)
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** Integration tests with populated JSONL log and pgvector DB.
|
||||
|
||||
**Integration test** — `tests/it_projects_endpoint.rs`:
|
||||
1. `a1_list_projects` — GET /projects returns array with test project(s).
|
||||
2. `a2_project_count_correct` — total_chunks field matches pgvector COUNT.
|
||||
3. `a3_project_evidence_count` — total_evidence field matches L0 node count for project.
|
||||
4. `a4_get_project_status` — GET /projects/<id>/status returns 200.
|
||||
5. `a5_standing_queries_listed` — standing_queries array in status matches queries YAML.
|
||||
6. `a6_last_ingest_timestamp` — last_ingest_at is recent and matches JSONL log.
|
||||
7. `a7_l2_synthesis_metadata` — l2_synthesis object contains last_synthesis_at and exit_gate_fired.
|
||||
8. `a8_cache_stats_present` — embedding_cache_hits and cache_misses are present and >= 0.
|
||||
9. `a9_next_synthesis_at_scheduled` — next_synthesis_at is a valid future timestamp.
|
||||
10. `a10_unknown_project_404` — GET /projects/nonexistent/status returns 404.
|
||||
|
||||
**Command:** `cargo test -p mem-cli projects_endpoint`
|
||||
|
||||
**False pass:**
|
||||
- total_chunks hardcoded to a fixed number; never actually counts.
|
||||
- Cache stats always zero (client doesn't track; endpoint returns fake values).
|
||||
- Last ingest timestamp never validated against actual log.
|
||||
|
||||
## Traps
|
||||
|
||||
- JSONL log queries are slow for large projects (412 chunks, naive scan). Consider indexing by project_id or caching if >10K chunks.
|
||||
- Next synthesis scheduling logic is simple (24h interval). If synthesis runs are skipped or delayed, estimate becomes stale. Document the assumption.
|
||||
- Memory size calculation uses SUM(LENGTH(text)) which is TEXT byte length in DB, not network wire size or actual storage (compression, overhead).
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md § Distributed API Layer](../DESIGN.md#distributed-api-layer-homelab-frontend)
|
||||
Reference in New Issue
Block a user