feat: complete M0.1-M0.4 phases

M0.1 - Cargo workspace + crate skeletons
  - 6-crate workspace with correct dependency direction
  - CI/CD pipeline with GitHub Actions
  - Integration tests verifying build and dependency structure

M0.2 - Domain types and sha256 identity
  - Level (L0, L1, L2) enum with proper serde formatting
  - Role enum (User, Assistant, ToolResult, System)
  - Record, Chunk, and MemoryNode domain types
  - Content-hash identity system ensuring rebuild idempotence
  - Newtypes (ProjectId, QueryId, RunId) with validation
  - Round-trip serde tests for all types

M0.3 - RecordSource trait + ChunkPolicy
  - RecordSource trait for streaming record sources
  - Chunk policy with token budgets and boundary modes
  - TokenCounter trait with CharsOverFourCounter stub
  - Chunking stream that respects budgets without splitting records
  - VecSource for testing
  - Integration tests verifying lossless chunking and budget adherence

M0.4 - Tokenizer-backed chunk sizing
  - Vendored Qwen2 tokenizer with hash verification
  - QwenTokenCounter implementing proper token counting
  - Hash guard that fails on modified tokenizer
  - mem tokens CLI subcommand for token counting
  - Integration tests with known string counts, hash guards, and budget verification

Total: 19 integration tests passing, all phases verified to compose correctly
Workspace builds cleanly with no clippy warnings

tasks/M3.5.2-ingest-endpoint.md

@@ -0,0 +1,132 @@
+# M3.5.2 — POST /ingest endpoint: async queue, idempotency, job polling
+
+| Field | Value |
+|---|---|
+| Phase | M3.5 — Distributed API Layer |
+| Size | M — 1–3 days |
+| Status | ⬜ Not started |
+| Flags | — |
+| Spec | inlined below |
+| Blocks | M3.5.8 |
+| Depends | M3.5.1, M1.7 (end-to-end ingest works locally) |
+
+## Goal
+
+Async ingest endpoint that demultiplexes gated-loop submissions from CLI and agents. Idempotent by batch content hash (`ingest_id`). Prevent duplicate L0 evidence in the log.
+
+## Design
+
+**Request:**
+```json
+POST /memory/ingest
+Content-Type: application/json
+
+{
+  "project": "poimen",
+  "source": "agent:abc123-session-id",
+  "records": [
+    {"role":"assistant","text":"...","timestamp":"2026-08-20T...","source_position":0},
+    ...
+  ],
+  "ingest_id": "sha256(all_record_texts)"
+}
+```
+
+**Response (accepted):**
+```
+HTTP 202 Accepted
+{
+  "job_id": "ingest-<uuid>",
+  "ingest_id": "sha256(...)",
+  "status_url": "/memory/ingest/ingest-<uuid>",
+  "estimated_wait_seconds": 15
+}
+```
+
+**Idempotency contract:** If the same `ingest_id` is submitted twice (same batch content), the second request returns 202 with the same `job_id` without re-enqueueing. If `ingest_id` differs but project overlaps, both are enqueued separately (ordering is per-project FIFO after dedup).
+
+**Job status (polling):**
+```
+GET /memory/ingest/ingest-<job-id>
+→ 200 {
+  "job_id": "...",
+  "ingest_id": "...",
+  "project": "poimen",
+  "status": "running|completed|failed",
+  "chunks_seen": 42,
+  "chunks_used": 7,
+  "error": null,
+  "created_at": "2026-08-20T...",
+  "completed_at": null
+}
+```
+
+## Steps
+
+1. Ingest queue — choose **local in-memory (BTreeMap keyed by ingest_id) or Redis**. For M3.5, start in-memory; scaling to Redis is P2-deferred.
+   - Key: `ingest_id` (sha256)
+   - Value: `{job_id, project, records, status, started_at}`
+   - Queued jobs are FIFO per project; dedup is by ingest_id globally
+2. `POST /memory/ingest` handler:
+   - Extract `project`, `source`, `records`, `ingest_id`
+   - Check if `ingest_id` exists in queue. If yes, return 202 with existing `job_id` (no duplicate enqueue).
+   - If new, generate `job_id = format!("ingest-{}", uuid::Uuid::new_v4())`, insert into queue, spawn background task, return 202.
+   - Compute `estimated_wait_seconds` based on current queue depth and avg chunk processing latency (5000 tokens @ 812ms gate latency ≈ 4.2s per chunk).
+3. Background task (tokio::spawn):
+   - Dequeue from project queue (FIFO per project)
+   - Call the M1.7 `mem::ingest()` function with records
+   - Update status to `completed` with `chunks_seen` and `chunks_used` from the log
+   - On error, update status to `failed` with error message
+4. `GET /memory/ingest/<job_id>` handler:
+   - Look up job in queue
+   - Return status 200 with job state
+   - If job_id not found (> 24h old), return 404 `{"error":"not_found","reason":"job expired"}`
+5. Validation:
+   - `ingest_id` must be a hex string of length 64 (sha256); malformed → 400
+   - `project` must be a known project (loaded from queries/); unknown → 400
+   - `records` array must not be empty; empty → 400
+
+## Acceptance
+
+- POST returns 202 with a job_id
+- Same ingest_id resubmitted returns same job_id (idempotent)
+- Job status is pollable
+- Two different ingest_ids for the same project are both queued (not deduplicated by project)
+- Background task completes without blocking the request
+- Malformed request (bad ingest_id, unknown project) returns 400
+
+## Verify
+
+**Harness:** Integration tests + one manual queue inspection.
+
+**Integration test** — `tests/it_ingest_endpoint.rs`:
+1. `a1_ingest_accepted` — POST /ingest with valid payload returns 202 and body contains `job_id` field.
+2. `a2_ingest_id_is_idempotent` — POST twice with same `ingest_id`, same `project` — both return 202 with identical `job_id`.
+3. `a3_status_polling_works` — POST /ingest, GET /ingest/<job_id> immediately returns `status: "running"` or `status: "completed"`.
+4. `a4_different_ingest_ids_both_queued` — POST /ingest (id_a), POST /ingest (id_b), GET status of both — both in queue.
+5. `a5_bad_ingest_id_returns_400` — POST with `ingest_id: "xyz"` (not 64 hex chars) returns 400.
+6. `a6_unknown_project_returns_400` — POST with `project: "nonexistent"` returns 400.
+7. `a7_async_task_runs` — POST /ingest with a small test batch, poll /ingest/<job_id> repeatedly, verify status transitions from `running` to `completed`.
+8. `a8_empty_records_returns_400` — POST with `records: []` returns 400.
+
+**Manual verification:**
+- Run the server, ingest two batches with different ingest_ids for the same project, verify they are queued in order by checking JSONL log — both should be present after ingest completes, in the order submitted.
+
+**Command:** `cargo test -p mem-cli ingest_endpoint`
+
+**False pass:**
+- Testing with one project only. Multi-project FIFO ordering is the hard part; a single project always looks correct.
+- Job status never actually transitions from `running` to `completed`. A mock status endpoint can always return `running` and pass the test if the test only polls once.
+- Idempotency checked for `ingest_id` but not for `project` — two requests with same `ingest_id` but different `project` must be treated as different (they are).
+- Latency estimate never validated. Estimated wait can be any number; test should assert it is > 0 and < 1 hour.
+
+## Traps
+
+- Using a simple Vec for the queue. FIFO per project requires either a per-project queue map or a global queue with project filtering. Per-project is cheaper.
+- Job expiry: in-memory queue will grow unbounded if jobs are never pruned. Set an eviction policy (e.g., remove jobs older than 24h on every ingest request).
+- Tokio task panic in the background task. Spawn with `.spawn()` which detaches on panic; use a panic hook or `.spawn_blocking()` with error handling.
+- Reusing the M1.7 function directly without error wrapping. If it panics (log write fails, db timeout), the background task crashes and the job status never updates. Wrap in a Result type and catch panics.
+
+---
+
+Background: [DESIGN.md § Distributed API Layer](../DESIGN.md#distributed-api-layer-homelab-frontend)