rock c5a46dd82e feat: M8.2 Queue Worker integration with DualWriteIndexer ...

Complete async dual-write pipeline:
- QueueWorker: Background task receiving from queue, processing concurrently
- DualWriteIndexer: Coordinated writes to pgvector + OpenSearch
- Full decoupling: IngestWorker queues quickly, workers process asynchronously
- Gateway integration: Uses GatewayQueueAdapter for api.riotpiao.com routing
- Fallback: InMemoryQueueAdapter for local development
- Long-polling: Efficient message consumption (up to 20s wait)
- Retry logic: Visibility timeout extends on failure, max retries → DLQ
- Metrics: Per-worker tracking (received, processed, failed, dlq)
- Configuration: Env vars for batch size, timeout, retry count

Architecture:
- IngestWorker → queue.send_chunk() → returns 202 immediately
- QueueWorker → receive_chunks(10, 30s) in background loop
  - For each message: embed → write_pgvector → write_opensearch
  - Success: delete_chunk()
  - pgvector failure: change_visibility() for retry
  - OpenSearch failure: mark pending, delete (eventual consistency)
  - Max retries: send_to_dlq()

Files:
- crates/mem-cli/src/queue_worker.rs (430 LOC)
- crates/mem-cli/src/http_server.rs (+100 LOC queue worker init)
- tests/it_queue_worker_integration.rs (260 LOC, 11 tests)
- docs/M8.2-QUEUE_WORKER_INTEGRATION.md (350 LOC)

Benefits:
- 10-100x faster ingest API response
- True concurrent processing (multiple workers)
- Fault tolerance (retries, DLQ)
- Observability (metrics, logs)
- Horizontal scalability (replicas)

2026-08-28 13:14:39 -07:00

..

it_chat_client.rs

Fix LLM gateway path, update M1.8 gate test to load real chunks (Option B)

2026-08-23 00:32:27 -07:00

it_chunking.rs

feat: complete M0.1-M0.4 phases

2026-08-22 23:13:42 -07:00

it_claude_source.rs

feat: complete M0 phase - read-only spine (8/51 tasks)

2026-08-22 23:13:42 -07:00

it_context_optimizer.rs

test: M3.8.1 phase 1 integration tests (10 scenarios)

2026-08-28 09:30:48 -07:00

it_derived_filter.rs

Implement M4.2: Derived filter (shingle matcher + 10 tests, 239 total)

2026-08-26 13:55:37 -07:00

it_doc_corpus.rs

Implement M3.6.1: DocCorpusSource with heading-boundary chunking (196 tests)

2026-08-23 09:42:09 -07:00

it_dry_run.rs

fix: Add jwt_validator module declaration to main.rs

2026-08-27 12:54:50 -07:00

it_e2e_pipeline.rs

Deploy Poimen Memory K8s cluster with ArgoCD tracking (M2.2, M3.5-M3.7)

2026-08-22 23:13:42 -07:00

it_embeddings.rs

feat: Implement M2.1 Embeddings client (768-dim batching @32)

2026-08-27 20:36:57 -07:00

it_endpoints.rs

Deploy Poimen Memory K8s cluster with ArgoCD tracking (M2.2, M3.5-M3.7)

2026-08-22 23:13:42 -07:00

it_event_log.rs

Deploy Poimen Memory K8s cluster with ArgoCD tracking (M2.2, M3.5-M3.7)

2026-08-22 23:13:42 -07:00

it_gate_parser.rs

Deploy Poimen Memory K8s cluster with ArgoCD tracking (M2.2, M3.5-M3.7)

2026-08-22 23:13:42 -07:00

it_gated_loop.rs

Deploy Poimen Memory K8s cluster with ArgoCD tracking (M2.2, M3.5-M3.7)

2026-08-22 23:13:42 -07:00

it_gateway_queue_adapter.rs

feat: M8.2 Gateway Queue Adapter for SQS via api.riotpiao.com

2026-08-28 13:11:56 -07:00

it_http_server.rs

Deploy Poimen Memory K8s cluster with ArgoCD tracking (M2.2, M3.5-M3.7)

2026-08-22 23:13:42 -07:00

it_identity.rs

feat: complete M0.1-M0.4 phases

2026-08-22 23:13:42 -07:00

it_ingest_endpoint.rs

Implement M3.5.2: POST /ingest endpoint with idempotent async queue (204 tests)

2026-08-23 16:33:34 -07:00

it_jwt_auth.rs

feat: JWT auth validation with Authentik OIDC

2026-08-27 12:29:23 -07:00

it_jwt_integration.rs

feat: JWT auth validation with Authentik OIDC

2026-08-27 12:29:23 -07:00

it_l2.rs

Deploy Poimen Memory K8s cluster with ArgoCD tracking (M2.2, M3.5-M3.7)

2026-08-22 23:13:42 -07:00

it_m0_gate.rs

feat: complete M0 phase - read-only spine (8/51 tasks)

2026-08-22 23:13:42 -07:00

it_m1_gate.rs

Fix LLM gateway path, update M1.8 gate test to load real chunks (Option B)

2026-08-23 00:32:27 -07:00

it_m2_gate.rs

Deploy Poimen Memory K8s cluster with ArgoCD tracking (M2.2, M3.5-M3.7)

2026-08-22 23:13:42 -07:00

it_m3_5_gate.rs

feat: M3.5.8 complete - all endpoints, rate limiting, and deployment (253 tests)

2026-08-23 17:19:42 -07:00

it_m3_8_query_optimization.rs

feat: M3.8 query path optimization wired into http_server query handler

2026-08-28 12:49:34 -07:00

it_m3_gate.rs

Deploy Poimen Memory K8s cluster with ArgoCD tracking (M2.2, M3.5-M3.7)

2026-08-22 23:13:42 -07:00

it_m8_2_dual_write.rs

feat: M8.2 Gateway Queue Adapter for SQS via api.riotpiao.com

2026-08-28 13:11:56 -07:00

it_pg_repo.rs

feat: Implement M2.4 pgvector repository with real Postgres

2026-08-27 20:48:37 -07:00

it_pgvector.rs

feat(core): implement full memory pipeline (#11)

2026-08-24 01:37:16 +00:00

it_pi_source.rs

feat: complete M0 phase - read-only spine (8/51 tasks)

2026-08-22 23:13:42 -07:00

it_projects_endpoint.rs

feat: M3.5.8 complete - all endpoints, rate limiting, and deployment (253 tests)

2026-08-23 17:19:42 -07:00

it_prompt.rs

feat: add cache-aligned prompt builder for LLM API cost savings

2026-08-28 08:24:38 -07:00

it_query_endpoint.rs

feat: M3.5.8 complete - all endpoints, rate limiting, and deployment (253 tests)

2026-08-23 17:19:42 -07:00

it_query_executor.rs

Deploy Poimen Memory K8s cluster with ArgoCD tracking (M2.2, M3.5-M3.7)

2026-08-22 23:13:42 -07:00

it_query_federation.rs

feat: M3.5.8 complete - all endpoints, rate limiting, and deployment (253 tests)

2026-08-23 17:19:42 -07:00

it_query_loader.rs

Deploy Poimen Memory K8s cluster with ArgoCD tracking (M2.2, M3.5-M3.7)

2026-08-22 23:13:42 -07:00

it_queue_worker_integration.rs

feat: M8.2 Queue Worker integration with DualWriteIndexer

2026-08-28 13:14:39 -07:00

it_rate_limiting.rs

Implement M3.5.7: Rate limiting + idempotency (20 tests)

2026-08-26 13:35:50 -07:00

it_rebuild.rs

feat: Implement M2.5 & M2.6 — Obsidian vault projector + rebuild orchestrator

2026-08-27 20:54:43 -07:00

it_rerank.rs

fix: resolve module imports and rerank test format (#12)

2026-08-24 01:45:47 +00:00

it_signature.rs

feat: M3.7.7 signature extraction CLI + integration tests (unit tests pass, integration tests pending mem-cli fix)

2026-08-28 07:46:36 -07:00

it_skill_draft.rs

Implement M4.1: Skill draft command + 10 tests (229 total)

2026-08-26 13:50:22 -07:00

it_skills_endpoint.rs

feat: M3.5.8 complete - all endpoints, rate limiting, and deployment (253 tests)

2026-08-23 17:19:42 -07:00

it_symptom_projection.rs

feat: implement M3.7.8 symptom projection (250 LOC) + 22 tests (10 unit + 12 integration)

2026-08-28 08:08:55 -07:00

it_tokens.rs

feat: complete M0.1-M0.4 phases

2026-08-22 23:13:42 -07:00

it_verify.rs

refactor: replace Obsidian projector with standalone service (ppatlabs/obsidian)

2026-08-27 21:35:07 -07:00

it_workspace.rs

feat: complete M0.1-M0.4 phases

2026-08-22 23:13:42 -07:00

quick_queue_test.rs

feat: M8.2 Gateway Queue Adapter for SQS via api.riotpiao.com

2026-08-28 13:11:56 -07:00