feat: M8.2 Gateway Queue Adapter for SQS via api.riotpiao.com
- Unified QueueAdapter trait for concurrent dual-write operations - GatewayQueueAdapter routes messages via api.riotpiao.com with X-Service: sqs header - TokenProvider abstraction: StaticTokenProvider + AuthentikTokenProvider - JWT bearer token support (from Authentik OAuth2) - InMemoryQueueAdapter for testing - Base64 encoding/decoding for SQS message bodies - HTTP/REST integration (no direct gRPC complexity) - 8 unit tests + comprehensive documentation - Supports long-polling (ReceiveMessage), visibility timeout, DLQ Uses standard SQS API patterns: - SendMessage: Queue chunk for dual-write processing - ReceiveMessage: Long-poll up to 10 messages, 20s wait - DeleteMessage: Acknowledge on success - ChangeMessageVisibility: Retry on failure - SendToDLQ: After max retries Files: - crates/mem-cli/src/queue_adapter.rs (310 LOC) - crates/mem-cli/src/gateway_queue_adapter.rs (530 LOC) - tests/it_gateway_queue_adapter.rs (110 LOC) - docs/M8.2-GATEWAY_QUEUE_ADAPTER.md (400 LOC)
This commit is contained in:
@@ -0,0 +1,503 @@
|
||||
# M8.2 — Gateway Queue Adapter for SQS/kmsvc
|
||||
|
||||
**Status**: Implementation complete
|
||||
**Version**: 1.0
|
||||
**Architecture**: Unified queue API via api.riotpiao.com gateway
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
The Gateway Queue Adapter provides a unified interface for enqueueing chunk dual-write operations via the `api.riotpiao.com` gateway. Rather than connecting directly to kmsvc gRPC, this adapter uses standard HTTP/REST with JWT bearer tokens.
|
||||
|
||||
### Design Rationale
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Traditional Direct gRPC Approach (NOT used) │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ mem-cli kmsvc (gRPC) │
|
||||
│ │ │ │
|
||||
│ │──── gRPC stub ───────>│ (complex connection mgmt) │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ NEW: Gateway-based approach (THIS IMPLEMENTATION) │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ mem-cli api.riotpiao.com kmsvc │
|
||||
│ │ │ │ │
|
||||
│ │─ HTTP + JWT ──────>│──RoutedBy──────>│ │
|
||||
│ │ (Bearer token) │ X-Service: sqs │ │
|
||||
│ │ │ │
|
||||
│ │ (gateway validates JWT before routing)│
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Benefits**:
|
||||
- ✅ JWT tokens handled by Authentik (same as HTTP API)
|
||||
- ✅ Standard HTTP/REST interface (easier debugging via curl)
|
||||
- ✅ Leverage existing API gateway infrastructure
|
||||
- ✅ No direct gRPC connection management
|
||||
- ✅ Unified authentication across all services
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
### Trait: `QueueAdapter`
|
||||
|
||||
```rust
|
||||
#[async_trait]
|
||||
pub trait QueueAdapter: Send + Sync {
|
||||
async fn send_chunk(
|
||||
&self,
|
||||
chunk_id: Uuid,
|
||||
body: String,
|
||||
project: String,
|
||||
attributes: HashMap<String, String>,
|
||||
) -> Result<String>;
|
||||
|
||||
async fn receive_chunks(
|
||||
&self,
|
||||
max_messages: i32,
|
||||
visibility_timeout_secs: i32,
|
||||
project: Option<&str>,
|
||||
) -> Result<Vec<QueueMessage>>;
|
||||
|
||||
async fn delete_chunk(
|
||||
&self,
|
||||
message_id: &str,
|
||||
receipt_handle: &str,
|
||||
) -> Result<()>;
|
||||
|
||||
async fn change_visibility(
|
||||
&self,
|
||||
message_id: &str,
|
||||
receipt_handle: &str,
|
||||
visibility_timeout_secs: i32,
|
||||
) -> Result<()>;
|
||||
|
||||
async fn send_to_dlq(
|
||||
&self,
|
||||
message_id: &str,
|
||||
receipt_handle: &str,
|
||||
reason: &str,
|
||||
) -> Result<()>;
|
||||
|
||||
async fn get_stats(&self, project: Option<&str>) -> Result<QueueStats>;
|
||||
async fn purge(&self, project: Option<&str>) -> Result<usize>;
|
||||
async fn health_check(&self) -> Result<()>;
|
||||
}
|
||||
```
|
||||
|
||||
### Queue Message Format
|
||||
|
||||
```rust
|
||||
pub struct QueueMessage {
|
||||
pub message_id: String, // From SQS
|
||||
pub chunk_id: Uuid, // Original chunk ID
|
||||
pub body: String, // Serialized chunk data
|
||||
pub receive_count: i32, // Number of receives
|
||||
pub receipt_handle: String, // For delete/visibility ops
|
||||
pub project: String, // Project context
|
||||
pub attributes: HashMap<String, String>, // Metadata
|
||||
}
|
||||
```
|
||||
|
||||
### Token Provider Trait
|
||||
|
||||
```rust
|
||||
#[async_trait]
|
||||
pub trait TokenProvider: Send + Sync {
|
||||
async fn token(&self) -> Result<String>;
|
||||
}
|
||||
```
|
||||
|
||||
Implementations:
|
||||
- `StaticTokenProvider` — Fixed token (testing)
|
||||
- `AuthentikTokenProvider` — OAuth2 client credentials flow (production)
|
||||
|
||||
---
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Setup: Static Token (Testing)
|
||||
|
||||
```rust
|
||||
use mem_cli::gateway_queue_adapter::GatewayQueueAdapter;
|
||||
use mem_cli::queue_adapter::QueueAdapter;
|
||||
use uuid::Uuid;
|
||||
use std::collections::HashMap;
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> anyhow::Result<()> {
|
||||
// Create adapter with static token
|
||||
let adapter = GatewayQueueAdapter::with_static_token(
|
||||
"https://api.riotpiao.com".to_string(),
|
||||
"eyJ...my-jwt-token".to_string(),
|
||||
);
|
||||
|
||||
// Queue a chunk
|
||||
let msg_id = adapter.send_chunk(
|
||||
Uuid::new_v4(),
|
||||
r#"{"content": "hello world"}"#.to_string(),
|
||||
"myproject".to_string(),
|
||||
HashMap::new(),
|
||||
).await?;
|
||||
|
||||
println!("Queued: {}", msg_id);
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
### Setup: Authentik Token (Production)
|
||||
|
||||
```rust
|
||||
let adapter = GatewayQueueAdapter::with_authentik(
|
||||
"https://api.riotpiao.com".to_string(),
|
||||
"https://authentik.riotpiao.com/application/o/poimen-memory/".to_string(),
|
||||
"poimen-memory".to_string(), // client_id
|
||||
"your-client-secret".to_string(),
|
||||
);
|
||||
|
||||
// Token is automatically refreshed when expired
|
||||
```
|
||||
|
||||
### Queue a Chunk
|
||||
|
||||
```rust
|
||||
let mut attrs = std::collections::HashMap::new();
|
||||
attrs.insert("source".to_string(), "obsidian".to_string());
|
||||
attrs.insert("level".to_string(), "L0".to_string());
|
||||
attrs.insert("breadcrumb".to_string(),
|
||||
serde_json::to_string(&vec!["root", "section"])?,
|
||||
);
|
||||
|
||||
let message_id = adapter.send_chunk(
|
||||
Uuid::new_v4(),
|
||||
serde_json::json!({
|
||||
"content": "chunk text",
|
||||
"metadata": "...",
|
||||
}).to_string(),
|
||||
"myproject".to_string(),
|
||||
attrs,
|
||||
).await?;
|
||||
|
||||
tracing::info!("Chunk queued: {}", message_id);
|
||||
```
|
||||
|
||||
### Receive Messages (Long-poll)
|
||||
|
||||
```rust
|
||||
// Receive up to 10 messages, wait up to 20 seconds for availability
|
||||
let messages = adapter.receive_chunks(
|
||||
10, // max_messages (1-10)
|
||||
30, // visibility_timeout_secs
|
||||
Some("myproject"), // optional project filter
|
||||
).await?;
|
||||
|
||||
for msg in messages {
|
||||
println!("Message ID: {}", msg.message_id);
|
||||
println!("Receive count: {}", msg.receive_count);
|
||||
println!("Receipt handle: {}", msg.receipt_handle);
|
||||
|
||||
// Process the message...
|
||||
match process_chunk(&msg).await {
|
||||
Ok(_) => {
|
||||
// Delete on success
|
||||
adapter.delete_chunk(&msg.message_id, &msg.receipt_handle).await?;
|
||||
}
|
||||
Err(e) if msg.receive_count < 3 => {
|
||||
// Retry: extend visibility for 5 minutes
|
||||
adapter.change_visibility(
|
||||
&msg.message_id,
|
||||
&msg.receipt_handle,
|
||||
300,
|
||||
).await?;
|
||||
}
|
||||
Err(e) => {
|
||||
// Max retries: send to DLQ
|
||||
adapter.send_to_dlq(
|
||||
&msg.message_id,
|
||||
&msg.receipt_handle,
|
||||
&e.to_string(),
|
||||
).await?;
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Health Check
|
||||
|
||||
```rust
|
||||
if let Err(e) = adapter.health_check().await {
|
||||
eprintln!("Gateway unavailable: {}", e);
|
||||
}
|
||||
```
|
||||
|
||||
### Monitor Queue
|
||||
|
||||
```rust
|
||||
let stats = adapter.get_stats(Some("myproject")).await?;
|
||||
|
||||
println!("Available: {}", stats.available_messages);
|
||||
println!("In-flight: {}", stats.in_flight_messages);
|
||||
println!("DLQ: {}", stats.dead_letter_messages);
|
||||
println!("Processed: {}", stats.total_processed);
|
||||
println!("Avg delay: {}s", stats.average_delay_secs);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## HTTP Message Flow
|
||||
|
||||
### 1. Send Chunk (POST)
|
||||
|
||||
**Request**:
|
||||
```bash
|
||||
POST https://api.riotpiao.com/
|
||||
X-Service: sqs
|
||||
Authorization: Bearer eyJ...
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"messageBody": "aGVsbG8gd29ybGQ=", # Base64-encoded chunk data
|
||||
"messageAttributes": {
|
||||
"values": {
|
||||
"chunk_id": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"project": "myproject",
|
||||
"source": "obsidian",
|
||||
"level": "L0",
|
||||
"breadcrumb": "[\"root\", \"section\"]"
|
||||
}
|
||||
},
|
||||
"delaySeconds": 0
|
||||
}
|
||||
```
|
||||
|
||||
**Response** (200 OK):
|
||||
```json
|
||||
{
|
||||
"messageId": "d9f94e63-b2c1-4e9f-8c5f-8d5e3c1b7a0f"
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Receive Messages (GET)
|
||||
|
||||
**Request**:
|
||||
```bash
|
||||
GET https://api.riotpiao.com/?X-Service=sqs&queue=poimen-chunks-myproject&maxNumberOfMessages=10&waitTimeSeconds=20&visibilityTimeoutSeconds=30
|
||||
Authorization: Bearer eyJ...
|
||||
```
|
||||
|
||||
**Response** (200 OK):
|
||||
```json
|
||||
{
|
||||
"messages": [
|
||||
{
|
||||
"messageId": "d9f94e63-b2c1-4e9f-8c5f-8d5e3c1b7a0f",
|
||||
"receiptHandle": "AQEBxxxx...",
|
||||
"body": "aGVsbG8gd29ybGQ=", # Base64-encoded
|
||||
"attributes": {
|
||||
"values": {
|
||||
"chunk_id": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"project": "myproject",
|
||||
"source": "obsidian"
|
||||
}
|
||||
},
|
||||
"receiveCount": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Delete Message (DELETE)
|
||||
|
||||
**Request**:
|
||||
```bash
|
||||
DELETE https://api.riotpiao.com/
|
||||
X-Service: sqs
|
||||
Authorization: Bearer eyJ...
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"receiptHandle": "AQEBxxxx..."
|
||||
}
|
||||
```
|
||||
|
||||
**Response** (204 No Content)
|
||||
|
||||
---
|
||||
|
||||
## Integration with DualWriteIndexer
|
||||
|
||||
The `DualWriteIndexer` uses the queue adapter for concurrent dual-write processing:
|
||||
|
||||
```rust
|
||||
use mem_cli::dual_write_indexer::DualWriteIndexer;
|
||||
use mem_cli::gateway_queue_adapter::GatewayQueueAdapter;
|
||||
use std::sync::Arc;
|
||||
|
||||
// Create queue adapter
|
||||
let queue = Arc::new(
|
||||
GatewayQueueAdapter::with_authentik(
|
||||
"https://api.riotpiao.com".to_string(),
|
||||
issuer,
|
||||
client_id,
|
||||
client_secret,
|
||||
)
|
||||
);
|
||||
|
||||
// Create dual-write indexer with queue
|
||||
let indexer = DualWriteIndexer::new(
|
||||
pg_pool,
|
||||
opensearch_client,
|
||||
queue,
|
||||
);
|
||||
|
||||
// Queue chunk for processing
|
||||
let message_id = indexer.queue_chunk(&chunk_input, &embedding).await?;
|
||||
|
||||
// Concurrent workers receive and process
|
||||
let messages = queue.receive_chunks(10, 30, None).await?;
|
||||
for msg in messages {
|
||||
match indexer.process_queued_chunk(&msg, &embedding).await {
|
||||
Ok(result) => {
|
||||
queue.delete_chunk(&msg.message_id, &msg.receipt_handle).await?;
|
||||
}
|
||||
Err(e) => {
|
||||
queue.change_visibility(&msg.message_id, &msg.receipt_handle, 300).await?;
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Common Errors
|
||||
|
||||
| Status | Meaning | Recovery |
|
||||
|--------|---------|----------|
|
||||
| `401 Unauthorized` | Missing/expired token | Refresh token via TokenProvider |
|
||||
| `403 Forbidden` | Token valid but no permission | Check JWT claims in Authentik |
|
||||
| `404 Not Found` | Queue doesn't exist | Create queue via Queue CRD |
|
||||
| `429 Too Many Requests` | Rate limited | Implement backoff |
|
||||
| `502 Bad Gateway` | kmsvc unreachable | Retry with exponential backoff |
|
||||
| `503 Service Unavailable` | Gateway overloaded | Circuit breaker pattern |
|
||||
|
||||
### Retry Strategy
|
||||
|
||||
```rust
|
||||
use std::time::Duration;
|
||||
|
||||
let mut retries = 0;
|
||||
const MAX_RETRIES: usize = 3;
|
||||
|
||||
loop {
|
||||
match adapter.send_chunk(...).await {
|
||||
Ok(msg_id) => {
|
||||
tracing::info!("Sent: {}", msg_id);
|
||||
break;
|
||||
}
|
||||
Err(e) if retries < MAX_RETRIES => {
|
||||
retries += 1;
|
||||
let backoff = Duration::from_millis(100 * 2_u64.pow(retries as u32));
|
||||
tracing::warn!("Retry {} in {:?}: {}", retries, backoff, e);
|
||||
tokio::time::sleep(backoff).await;
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::error!("Max retries exceeded: {}", e);
|
||||
return Err(e);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration (Environment Variables)
|
||||
|
||||
```bash
|
||||
# Gateway endpoint
|
||||
export GATEWAY_URL=https://api.riotpiao.com
|
||||
|
||||
# Authentik (for JWT)
|
||||
export AUTHENTIK_ISSUER=https://authentik.riotpiao.com/application/o/poimen-memory/
|
||||
export AUTHENTIK_CLIENT_ID=poimen-memory
|
||||
export AUTHENTIK_CLIENT_SECRET=your-secret
|
||||
|
||||
# Optional: Static token (for testing)
|
||||
export STATIC_JWT_TOKEN=eyJ...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit Tests
|
||||
|
||||
```bash
|
||||
cargo test --lib gateway_queue_adapter
|
||||
```
|
||||
|
||||
### Integration Tests
|
||||
|
||||
```bash
|
||||
# Requires running api.riotpiao.com
|
||||
cargo test --test it_gateway_queue_adapter -- --ignored
|
||||
```
|
||||
|
||||
### Manual Testing with curl
|
||||
|
||||
```bash
|
||||
# Get token
|
||||
TOKEN=$(curl -s -X POST https://authentik.riotpiao.com/application/o/token/ \
|
||||
-d "grant_type=client_credentials&client_id=poimen-memory&client_secret=secret" \
|
||||
| jq -r '.access_token')
|
||||
|
||||
# Send message
|
||||
curl -X POST https://api.riotpiao.com/ \
|
||||
-H "X-Service: sqs" \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"messageBody": "aGVsbG8gd29ybGQ=", "messageAttributes": {"values": {}}}'
|
||||
|
||||
# Receive messages
|
||||
curl -X GET "https://api.riotpiao.com/?X-Service=sqs&queue=poimen-chunks-test&maxNumberOfMessages=10&waitTimeSeconds=20" \
|
||||
-H "Authorization: Bearer $TOKEN" | jq .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security Notes
|
||||
|
||||
✅ **JWT Validation**: Gateway validates token signature and claims before routing
|
||||
✅ **Bearer Token Format**: Strictly requires `Authorization: Bearer <token>`
|
||||
✅ **Token Expiry**: Automatic refresh via TokenProvider
|
||||
✅ **HTTPS Only**: All calls to api.riotpiao.com are encrypted
|
||||
✅ **Header Validation**: X-Service header validated by gateway
|
||||
|
||||
---
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- [ ] ChangeMessageVisibility support in gateway
|
||||
- [ ] GetQueueAttributes for monitoring
|
||||
- [ ] Batch operations (SendMessageBatch, DeleteMessageBatch)
|
||||
- [ ] Circuit breaker pattern for fault tolerance
|
||||
- [ ] Metrics export (Prometheus)
|
||||
- [ ] Tracing integration (OpenTelemetry)
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [SERVICE-USAGE.md](../homelab-frontend/docs/SERVICE-USAGE.md) — Gateway usage guide
|
||||
- [kmsvc-SDK README](../kmsvc-SDK/README.md) — Underlying SQS implementation
|
||||
- [Authentik Docs](https://goauthentik.io/) — JWT token provider
|
||||
Reference in New Issue
Block a user