rock
c5a46dd82e
Complete async dual-write pipeline: - QueueWorker: Background task receiving from queue, processing concurrently - DualWriteIndexer: Coordinated writes to pgvector + OpenSearch - Full decoupling: IngestWorker queues quickly, workers process asynchronously - Gateway integration: Uses GatewayQueueAdapter for api.riotpiao.com routing - Fallback: InMemoryQueueAdapter for local development - Long-polling: Efficient message consumption (up to 20s wait) - Retry logic: Visibility timeout extends on failure, max retries → DLQ - Metrics: Per-worker tracking (received, processed, failed, dlq) - Configuration: Env vars for batch size, timeout, retry count Architecture: - IngestWorker → queue.send_chunk() → returns 202 immediately - QueueWorker → receive_chunks(10, 30s) in background loop - For each message: embed → write_pgvector → write_opensearch - Success: delete_chunk() - pgvector failure: change_visibility() for retry - OpenSearch failure: mark pending, delete (eventual consistency) - Max retries: send_to_dlq() Files: - crates/mem-cli/src/queue_worker.rs (430 LOC) - crates/mem-cli/src/http_server.rs (+100 LOC queue worker init) - tests/it_queue_worker_integration.rs (260 LOC, 11 tests) - docs/M8.2-QUEUE_WORKER_INTEGRATION.md (350 LOC) Benefits: - 10-100x faster ingest API response - True concurrent processing (multiple workers) - Fault tolerance (retries, DLQ) - Observability (metrics, logs) - Horizontal scalability (replicas)
20 lines
485 B
Rust
20 lines
485 B
Rust
pub mod endpoints;
|
|
pub mod http_server;
|
|
pub mod ingest_worker;
|
|
pub mod query_worker;
|
|
pub mod rate_limiter;
|
|
pub mod idempotency;
|
|
pub mod jwt_validator;
|
|
pub mod opensearch_client;
|
|
pub mod dual_write_indexer;
|
|
pub mod queue_adapter;
|
|
pub mod gateway_queue_adapter;
|
|
pub mod queue_worker;
|
|
pub mod query_optimizer;
|
|
pub mod hybrid_query_worker;
|
|
pub mod verify;
|
|
|
|
pub use endpoints::{IngestQueue, IngestRequest, JobStatus};
|
|
pub use ingest_worker::IngestWorker;
|
|
pub use query_worker::QueryWorker;
|