feat: M8.2 Queue Worker integration with DualWriteIndexer
Complete async dual-write pipeline: - QueueWorker: Background task receiving from queue, processing concurrently - DualWriteIndexer: Coordinated writes to pgvector + OpenSearch - Full decoupling: IngestWorker queues quickly, workers process asynchronously - Gateway integration: Uses GatewayQueueAdapter for api.riotpiao.com routing - Fallback: InMemoryQueueAdapter for local development - Long-polling: Efficient message consumption (up to 20s wait) - Retry logic: Visibility timeout extends on failure, max retries → DLQ - Metrics: Per-worker tracking (received, processed, failed, dlq) - Configuration: Env vars for batch size, timeout, retry count Architecture: - IngestWorker → queue.send_chunk() → returns 202 immediately - QueueWorker → receive_chunks(10, 30s) in background loop - For each message: embed → write_pgvector → write_opensearch - Success: delete_chunk() - pgvector failure: change_visibility() for retry - OpenSearch failure: mark pending, delete (eventual consistency) - Max retries: send_to_dlq() Files: - crates/mem-cli/src/queue_worker.rs (430 LOC) - crates/mem-cli/src/http_server.rs (+100 LOC queue worker init) - tests/it_queue_worker_integration.rs (260 LOC, 11 tests) - docs/M8.2-QUEUE_WORKER_INTEGRATION.md (350 LOC) Benefits: - 10-100x faster ingest API response - True concurrent processing (multiple workers) - Fault tolerance (retries, DLQ) - Observability (metrics, logs) - Horizontal scalability (replicas)
This commit is contained in:
@@ -9,6 +9,7 @@ pub mod opensearch_client;
|
||||
pub mod dual_write_indexer;
|
||||
pub mod queue_adapter;
|
||||
pub mod gateway_queue_adapter;
|
||||
pub mod queue_worker;
|
||||
pub mod query_optimizer;
|
||||
pub mod hybrid_query_worker;
|
||||
pub mod verify;
|
||||
|
||||
Reference in New Issue
Block a user