fix: default auth to Bearer token (riotpiao gateway uses JWT now)
This commit is contained in:
@@ -0,0 +1,643 @@
|
||||
# Document Lifecycle — Control-Plane Flow
|
||||
|
||||
This document traces the complete lifecycle of a single piece of knowledge from ingestion to retrieval, showing every component it touches and every decision made along the way.
|
||||
|
||||
---
|
||||
|
||||
## Example: "Rust Ownership & Borrowing" knowledge file
|
||||
|
||||
```
|
||||
rust-ownership.md (source) → mem learn CLI → POST /memory/learn → chunk →
|
||||
gated loop (evaluate + compact) → embed → pgvector → searchable via /memory/query
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Submission
|
||||
|
||||
**Actor:** Agent or human operator
|
||||
**Entry point:** `mem learn` CLI or `POST /memory/learn`
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ rust-ownership.md │
|
||||
│ │
|
||||
│ # Rust Ownership │
|
||||
│ ## Borrowing Rules │
|
||||
│ - &T immutable borrow, &mut T mutable borrow │
|
||||
│ - Cannot have &mut while & exists │
|
||||
│ ## Lifetimes │
|
||||
│ - 'a annotations tell compiler how long refs live │
|
||||
│ ## Smart Pointers │
|
||||
│ - Box<T> heap allocation, single ownership │
|
||||
│ - Arc<T> atomic reference counting, thread-safe │
|
||||
└─────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ mem learn CLI │
|
||||
│ │
|
||||
│ Reads file │
|
||||
│ Validates .md │
|
||||
│ Sends to API │
|
||||
└────────┬─────────┘
|
||||
│
|
||||
HTTP POST /memory/learn
|
||||
{
|
||||
"project": "knowledge",
|
||||
"text": "<full file content>",
|
||||
"query": "What are key facts in rust-ownership?",
|
||||
"model": "qwen2.5:3b-instruct",
|
||||
"memory_budget": 4096
|
||||
}
|
||||
│
|
||||
▼
|
||||
```
|
||||
|
||||
## Phase 2: Authentication & Rate Limiting
|
||||
|
||||
**Component:** `http_server.rs` → `jwt_validator.rs` → `rate_limiter.rs`
|
||||
|
||||
```
|
||||
┌──────────────────────┐
|
||||
│ Auth Gate │
|
||||
│ │
|
||||
│ JWT token valid? │──── no ──→ 401 Unauthorized
|
||||
│ Has memory:write? │──── no ──→ 403 Forbidden
|
||||
│ Rate limit ok? │──── no ──→ 429 Too Many Requests
|
||||
│ (100 ingest/hr) │
|
||||
└──────────┬───────────┘
|
||||
│ yes
|
||||
▼
|
||||
```
|
||||
|
||||
## Phase 3: Chunking
|
||||
|
||||
**Component:** `chunk_markdown_text()` in `http_server.rs`
|
||||
|
||||
The raw markdown is split into chunks on `## ` heading boundaries. Each chunk stays under `chunk_size` (default 2000 chars).
|
||||
|
||||
```
|
||||
┌──────────────────────────────────┐
|
||||
│ Chunker │
|
||||
│ │
|
||||
│ Split on ## headings │
|
||||
│ Hard-split if > chunk_size │
|
||||
│ │
|
||||
│ Input: 1 file (3200 chars) │
|
||||
│ Output: 4 chunks │
|
||||
└──────────┬───────────────────────┘
|
||||
│
|
||||
┌────────────┼────────────┬──────────────┐
|
||||
▼ ▼ ▼ ▼
|
||||
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
|
||||
│ Chunk 1 │ │ Chunk 2 │ │ Chunk 3 │ │ Chunk 4 │
|
||||
│ # Title │ │ ## Borrow│ │ ## Life │ │ ## Smart │
|
||||
│ + intro │ │ Rules │ │ times │ │ Pointers │
|
||||
│ 400 ch │ │ 600 ch │ │ 500 ch │ │ 700 ch │
|
||||
└──────────┘ └──────────┘ └──────────┘ └──────────┘
|
||||
│ │ │ │
|
||||
└────────────┴────────────┴──────────────┘
|
||||
│
|
||||
▼
|
||||
```
|
||||
|
||||
## Phase 4: Gated Loop (The Core)
|
||||
|
||||
**Component:** `gated_loop.rs` → LLM (`qwen2.5:3b-instruct`)
|
||||
|
||||
This is where evaluation and compaction happen. The LLM sees each chunk alongside the current memory and decides:
|
||||
- **Update gate** (`<check>yes|no</check>`): Does this chunk contain new knowledge?
|
||||
- **Candidate memory** (`<update>...</update>`): Rewritten memory incorporating the chunk
|
||||
- **Exit gate** (`<next>continue|end</next>`): Should we stop scanning?
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ GATED LOOP │
|
||||
│ │
|
||||
│ Memory = "" (empty at start) │
|
||||
│ Query = "What are key facts in rust-ownership?" │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────┐ │
|
||||
│ │ Turn 1: Chunk 1 (# Title + intro) │ │
|
||||
│ │ │ │
|
||||
│ │ LLM receives: │ │
|
||||
│ │ System: "You are evaluating evidence..." │ │
|
||||
│ │ User: "Memory: (empty) │ │
|
||||
│ │ Chunk: # Rust Ownership..." │ │
|
||||
│ │ │ │
|
||||
│ │ LLM responds: │ │
|
||||
│ │ <think>Title only, no substance</think> │ │
|
||||
│ │ <check>no</check> │ │
|
||||
│ │ <update></update> │ │
|
||||
│ │ <next>continue</next> │ │
|
||||
│ │ │ │
|
||||
│ │ Decision: REJECTED (no new knowledge) │ │
|
||||
│ │ Memory: "" (unchanged) │ │
|
||||
│ └─────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────┐ │
|
||||
│ │ Turn 2: Chunk 2 (## Borrowing Rules) │ │
|
||||
│ │ │ │
|
||||
│ │ LLM responds: │ │
|
||||
│ │ <think>Core ownership rules, new</think> │ │
|
||||
│ │ <check>yes</check> │ │
|
||||
│ │ <update> │ │
|
||||
│ │ Rust borrowing: &T immutable, &mut T │ │
|
||||
│ │ mutable. Cannot hold &mut while & alive │ │
|
||||
│ │ </update> │ │
|
||||
│ │ <next>continue</next> │ │
|
||||
│ │ │ │
|
||||
│ │ Decision: ACCEPTED │ │
|
||||
│ │ Memory: "Rust borrowing: &T immutable..." │ │
|
||||
│ └─────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────┐ │
|
||||
│ │ Turn 3: Chunk 3 (## Lifetimes) │ │
|
||||
│ │ │ │
|
||||
│ │ LLM receives current memory + new chunk │ │
|
||||
│ │ │ │
|
||||
│ │ LLM responds: │ │
|
||||
│ │ <think>New info, adds to borrowing</think>│ │
|
||||
│ │ <check>yes</check> │ │
|
||||
│ │ <update> │ │
|
||||
│ │ Rust ownership: &T immutable borrow, │ │
|
||||
│ │ &mut T mutable. Cannot coexist. │ │
|
||||
│ │ Lifetimes: 'a annotations for ref │ │
|
||||
│ │ duration. Elision: single input → output│ │
|
||||
│ │ </update> │ │
|
||||
│ │ <next>continue</next> │ │
|
||||
│ │ │ │
|
||||
│ │ Decision: ACCEPTED + COMPACTED │ │
|
||||
│ │ Memory: merged borrowing + lifetimes │ │
|
||||
│ │ (Note: LLM rewrote, not appended) │ │
|
||||
│ └─────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────┐ │
|
||||
│ │ Turn 4: Chunk 4 (## Smart Pointers) │ │
|
||||
│ │ │ │
|
||||
│ │ Decision: ACCEPTED + COMPACTED │ │
|
||||
│ │ Memory: ownership + lifetimes + pointers │ │
|
||||
│ │ (Still within memory_budget: 4096 tokens) │ │
|
||||
│ └─────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ Result: │
|
||||
│ chunks_seen: 4 │
|
||||
│ chunks_used: 3 (chunk 1 rejected) │
|
||||
│ final_memory: "Rust ownership: &T immutable │
|
||||
│ borrow, &mut T mutable. Cannot coexist. │
|
||||
│ Lifetimes: 'a annotations... Box<T> heap, │
|
||||
│ Arc<T> thread-safe shared ownership..." │
|
||||
│ │
|
||||
└──────────────────────┬──────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
```
|
||||
|
||||
### Pluggable LLM Architecture
|
||||
|
||||
The gated loop accepts any `LlmClient` trait implementation. The current design
|
||||
uses a single model for both gate and rewrite. The target design splits them:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ GATED LOOP (pluggable LLMs) │
|
||||
│ │
|
||||
│ LoopConfig { │
|
||||
│ gate_model: "qwen2.5:3b-instruct" ← fast, cheap │
|
||||
│ rewrite_model: "ornith:35b" ← accurate, expensive │
|
||||
│ embed_model: "nomic-embed-text-v2" ← fixed 768-dim │
|
||||
│ } │
|
||||
│ │
|
||||
│ For each chunk: │
|
||||
│ │
|
||||
│ ┌──────────┐ ┌───────────────────────────────────────────┐ │
|
||||
│ │ Chunk │────▶│ GATE LLM (3B, fast) │ │
|
||||
│ │ + M_t-1 │ │ │ │
|
||||
│ └──────────┘ │ Prompt: "Does this chunk add new │ │
|
||||
│ │ knowledge not already in memory?" │ │
|
||||
│ │ │ │
|
||||
│ │ Output: <check>yes|no</check> │ │
|
||||
│ │ <next>continue|end</next> │ │
|
||||
│ │ │ │
|
||||
│ │ Cost: ~200 tokens in, ~20 tokens out │ │
|
||||
│ │ Latency: ~100ms │ │
|
||||
│ └──────────┬──────────┬─────────────────────┘ │
|
||||
│ │ │ │
|
||||
│ yes │ │ no │
|
||||
│ ▼ ▼ │
|
||||
│ ┌─────────────┐ ┌──────────┐ │
|
||||
│ │ REWRITE LLM │ │ SKIP │ │
|
||||
│ │ (35B, smart)│ │ │ │
|
||||
│ │ │ │ M_t = M │ │
|
||||
│ │ Prompt: │ │ (no │ │
|
||||
│ │ "Rewrite │ │ change) │ │
|
||||
│ │ memory │ └──────────┘ │
|
||||
│ │ merging │ │
|
||||
│ │ old + new. │ │
|
||||
│ │ Keep all │ │
|
||||
│ │ facts. │ │
|
||||
│ │ Stay under │ │
|
||||
│ │ budget." │ │
|
||||
│ │ │ │
|
||||
│ │ Cost: ~2K │ │
|
||||
│ │ tokens in, │ │
|
||||
│ │ ~1K out │ │
|
||||
│ │ Latency: │ │
|
||||
│ │ ~2-5s │ │
|
||||
│ └──────┬──────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────┐ │
|
||||
│ │ Budget │ │
|
||||
│ │ Check │ │
|
||||
│ │ │ │
|
||||
│ │ len(M_new) │ │
|
||||
│ │ > budget? │ │
|
||||
│ └──┬──────┬───┘ │
|
||||
│ no │ │ yes │
|
||||
│ ▼ ▼ │
|
||||
│ ┌─────────┐ ┌──────────┐ │
|
||||
│ │ Accept │ │ Reject │ │
|
||||
│ │ M_t = │ │ Budget │ │
|
||||
│ │ M_new │ │ exceeded │ │
|
||||
│ └─────────┘ └──────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
#### Cost Model (per file, ~5 chunks)
|
||||
|
||||
| Step | Model | Calls | Tokens/call | Total tokens |
|
||||
|---|---|---|---|---|
|
||||
| Gate (all chunks) | 3B | 5 | ~220 | ~1,100 |
|
||||
| Rewrite (accepted only, ~3) | 35B | 3 | ~3,000 | ~9,000 |
|
||||
| Embed (final memory) | nomic-embed | 1 | ~500 | ~500 |
|
||||
|
||||
Rejected chunks only cost a gate call (~100ms). The expensive rewrite
|
||||
only runs when the gate accepts. For a typical file where 60% of chunks
|
||||
are accepted, this saves ~40% of 35B inference cost vs single-model.
|
||||
|
||||
#### Trait Interface
|
||||
|
||||
```rust
|
||||
/// Pluggable LLM backend for the gated loop.
|
||||
pub trait LlmClient: Send + Sync {
|
||||
fn complete_blocking(
|
||||
&self, system: &str, user: &str, max_tokens: usize
|
||||
) -> Result<String>;
|
||||
}
|
||||
|
||||
/// Split-model client: fast gate + accurate rewrite.
|
||||
pub struct SplitModelClient {
|
||||
gate: Box<dyn LlmClient>, // qwen2.5:3b — decides yes/no
|
||||
rewrite: Box<dyn LlmClient>, // ornith:35b — produces compacted memory
|
||||
}
|
||||
|
||||
/// Single-model client (current default).
|
||||
pub struct SingleModelClient {
|
||||
llm: ChatClient, // same model for gate + rewrite
|
||||
}
|
||||
```
|
||||
|
||||
#### API Request (pluggable models)
|
||||
|
||||
```json
|
||||
POST /memory/learn
|
||||
{
|
||||
"project": "knowledge",
|
||||
"text": "## Rust Ownership\n...",
|
||||
"gate_model": "qwen2.5:3b-instruct",
|
||||
"rewrite_model": "ornith:35b",
|
||||
"memory_budget": 4096
|
||||
}
|
||||
```
|
||||
|
||||
If `rewrite_model` is omitted, falls back to `gate_model` (single-model mode,
|
||||
current behavior). This keeps the API backward-compatible.
|
||||
|
||||
### Supported Providers
|
||||
|
||||
The `ChatClient` uses OpenAI-compatible `/v1/chat/completions` format.
|
||||
Auth mode is auto-detected from the base URL:
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ Provider Support │
|
||||
│ │
|
||||
│ ┌─────────────────┐ ┌──────────────────┐ │
|
||||
│ │ riotpiao gateway│ │ OpenRouter │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ base: api.riot │ │ base: openrouter │ │
|
||||
│ │ piao.com/v1 │ │ .ai/api/v1 │ │
|
||||
│ │ auth: apikey: │ │ auth: Bearer │ │
|
||||
│ │ <key> header │ │ <key> header │ │
|
||||
│ │ models: │ │ models: │ │
|
||||
│ │ qwen2.5:3b │ │ openai/gpt-4o │ │
|
||||
│ │ ornith:35b │ │ anthropic/ │ │
|
||||
│ │ reasoning │ │ claude-sonnet │ │
|
||||
│ └────────┬────────┘ │ google/gemini │ │
|
||||
│ │ └────────┬─────────┘ │
|
||||
│ │ │ │
|
||||
│ ▼ ▼ │
|
||||
│ ┌──────────────────────────────────────────┐ │
|
||||
│ │ ChatClient (OpenAI-compatible) │ │
|
||||
│ │ │ │
|
||||
│ │ POST {base_url}/chat/completions │ │
|
||||
│ │ { │ │
|
||||
│ │ "model": "...", │ │
|
||||
│ │ "messages": [{role, content}, ...], │ │
|
||||
│ │ "max_tokens": N │ │
|
||||
│ │ } │ │
|
||||
│ │ │ │
|
||||
│ │ Auth auto-detected from URL: │ │
|
||||
│ │ *openrouter.ai → Bearer │ │
|
||||
│ │ *api.openai.com → Bearer │ │
|
||||
│ │ *api.riotpiao.com → apikey header │ │
|
||||
│ │ *localhost → None │ │
|
||||
│ └──────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌─────────────────┐ ┌──────────────────┐ │
|
||||
│ │ OpenAI │ │ Ollama (local) │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ base: api.open │ │ base: localhost │ │
|
||||
│ │ ai.com/v1 │ │ :11434/v1 │ │
|
||||
│ │ auth: Bearer │ │ auth: None │ │
|
||||
│ │ models: │ │ models: │ │
|
||||
│ │ gpt-4o │ │ llama3.1 │ │
|
||||
│ │ gpt-4o-mini │ │ qwen2.5:3b │ │
|
||||
│ └─────────────────┘ └──────────────────┘ │
|
||||
│ │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
#### Environment Configuration
|
||||
|
||||
```bash
|
||||
# riotpiao gateway (default)
|
||||
export LLM_API_BASE=https://api.riotpiao.com/v1
|
||||
export LLM_API_KEY=your-gateway-key
|
||||
|
||||
# OpenRouter (access to GPT-4o, Claude, Gemini, etc.)
|
||||
export LLM_API_BASE=https://openrouter.ai/api/v1
|
||||
export LLM_API_KEY=sk-or-v1-...
|
||||
|
||||
# OpenAI direct
|
||||
export LLM_API_BASE=https://api.openai.com/v1
|
||||
export LLM_API_KEY=sk-...
|
||||
|
||||
# Ollama local (no auth)
|
||||
export LLM_API_BASE=http://localhost:11434/v1
|
||||
export LLM_API_KEY=
|
||||
```
|
||||
|
||||
#### Mixed Provider Example (gate local, rewrite cloud)
|
||||
|
||||
```json
|
||||
POST /memory/learn
|
||||
{
|
||||
"project": "knowledge",
|
||||
"text": "...",
|
||||
"gate_model": "qwen2.5:3b-instruct",
|
||||
"gate_base_url": "https://api.riotpiao.com/v1",
|
||||
"rewrite_model": "anthropic/claude-sonnet-4",
|
||||
"rewrite_base_url": "https://openrouter.ai/api/v1"
|
||||
}
|
||||
```
|
||||
|
||||
Gate runs on local 3B (free, fast), rewrite uses Claude via OpenRouter
|
||||
(accurate, paid per token). Only accepted chunks hit the cloud.
|
||||
|
||||
### Why This Prevents Unbounded Growth
|
||||
|
||||
The key insight: **the LLM rewrites memory on every accepted chunk, not appends.** Turn 3 doesn't add lifetimes after borrowing — it produces a new memory that integrates both. If the same file is ingested again, most chunks will be rejected (`<check>no</check>`) because the knowledge already exists in memory.
|
||||
|
||||
The `memory_budget` parameter (default 4096 tokens) enforces a hard cap. If the LLM's candidate exceeds it, the chunk triggers `BudgetExceeded` and is skipped.
|
||||
|
||||
## Phase 5: Embedding
|
||||
|
||||
**Component:** `EmbeddingsClient` → `nomic-ai/nomic-embed-text-v2-moe`
|
||||
|
||||
The compacted final memory is embedded into a 768-dimensional vector.
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────┐
|
||||
│ Embedding │
|
||||
│ │
|
||||
│ Input: "Rust ownership: &T immutable │
|
||||
│ borrow, &mut T mutable. Cannot..." │
|
||||
│ │
|
||||
│ Model: nomic-embed-text-v2-moe │
|
||||
│ Output: [0.035, 0.022, -0.018, ...] │
|
||||
│ 768 dimensions │
|
||||
│ │
|
||||
└──────────────────┬───────────────────────┘
|
||||
│
|
||||
▼
|
||||
```
|
||||
|
||||
## Phase 6: Storage
|
||||
|
||||
**Component:** `pgvector` (CNPG cluster, 2 replicas)
|
||||
|
||||
The compacted memory + embedding is stored. SHA256 dedup ensures re-ingesting the same content updates rather than duplicates.
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────┐
|
||||
│ pgvector (memory_chunks) │
|
||||
│ │
|
||||
│ INSERT INTO memory_chunks │
|
||||
│ (id, project, level, text, │
|
||||
│ embedding, sha256, source) │
|
||||
│ ON CONFLICT (sha256) │
|
||||
│ DO UPDATE SET text, embedding │
|
||||
│ │
|
||||
│ ┌────────────────────────────────────┐ │
|
||||
│ │ id: uuid-abc-123 │ │
|
||||
│ │ project: knowledge │ │
|
||||
│ │ level: L1 │ │
|
||||
│ │ text: "Rust ownership: ..." │ │
|
||||
│ │ embedding: [0.035, 0.022, ...] │ │
|
||||
│ │ sha256: de12cd34ef56... │ │
|
||||
│ │ source: learn://knowledge │ │
|
||||
│ └────────────────────────────────────┘ │
|
||||
│ │
|
||||
└──────────────────┬───────────────────────┘
|
||||
│
|
||||
▼
|
||||
```
|
||||
|
||||
## Phase 7: Response
|
||||
|
||||
**Component:** `http_server.rs`
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────┐
|
||||
│ HTTP Response │
|
||||
│ │
|
||||
│ 200 OK │
|
||||
│ { │
|
||||
│ "project": "knowledge", │
|
||||
│ "status": "completed", │
|
||||
│ "chunks_seen": 4, │
|
||||
│ "chunks_used": 3, │
|
||||
│ "memory": "Rust ownership: &T...", │
|
||||
│ "memory_tokens": 156, │
|
||||
│ "stored": true, │
|
||||
│ "model": "qwen2.5:3b-instruct" │
|
||||
│ } │
|
||||
└──────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 8: Retrieval (Later)
|
||||
|
||||
**Actor:** Agent querying memory during a task
|
||||
**Entry point:** `POST /memory/query` or `POST /memory/context`
|
||||
|
||||
```
|
||||
Agent: "How does Rust handle shared ownership across threads?"
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Retrieval Pipeline │
|
||||
│ │
|
||||
│ 1. Embed query → [0.041, -0.012, ...] (768-dim) │
|
||||
│ │
|
||||
│ 2. Parallel search: │
|
||||
│ ├─ Semantic: pgvector cosine similarity │
|
||||
│ │ SELECT * FROM memory_chunks │
|
||||
│ │ ORDER BY embedding <=> query_vec │
|
||||
│ │ → score: 0.89 │
|
||||
│ │ │
|
||||
│ └─ Lexical: OpenSearch BM25 │
|
||||
│ multi_match: "shared ownership threads" │
|
||||
│ → score: 0.76 │
|
||||
│ │
|
||||
│ 3. RRF Fusion: 0.6 × semantic + 0.4 × lexical │
|
||||
│ → fused_score: 0.84 │
|
||||
│ │
|
||||
│ 4. Return: │
|
||||
│ { │
|
||||
│ "score": 0.84, │
|
||||
│ "text": "Rust ownership: ... Arc<T> atomic │
|
||||
│ reference counting, thread-safe shared ownership" │
|
||||
│ } │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Re-Ingestion (Idempotent)
|
||||
|
||||
When the same file is ingested again:
|
||||
|
||||
```
|
||||
rust-ownership.md (unchanged) → POST /memory/learn → chunk → gated loop:
|
||||
|
||||
Turn 1: "# Rust Ownership" → <check>no</check> (title, no substance)
|
||||
Turn 2: "## Borrowing" → <check>no</check> (already in memory)
|
||||
Turn 3: "## Lifetimes" → <check>no</check> (already in memory)
|
||||
Turn 4: "## Smart Pointers"→ <check>no</check> (already in memory)
|
||||
|
||||
Result: chunks_seen=4, chunks_used=0
|
||||
Memory: unchanged, nothing new stored
|
||||
```
|
||||
|
||||
When the file is updated with new content:
|
||||
|
||||
```
|
||||
rust-ownership.md (added ## Async section) → POST /memory/learn → gated loop:
|
||||
|
||||
Turn 1-4: existing sections → <check>no</check> (known)
|
||||
Turn 5: "## Async/Await" → <check>yes</check> (new!)
|
||||
Memory rewritten: ownership + lifetimes + pointers + async
|
||||
|
||||
Result: chunks_seen=5, chunks_used=1
|
||||
pgvector: ON CONFLICT (sha256) DO UPDATE → old record replaced
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Complete Data Flow Diagram
|
||||
|
||||
```
|
||||
┌──────────┐
|
||||
│ Source │
|
||||
│ (.md) │
|
||||
└────┬─────┘
|
||||
│
|
||||
┌────▼─────┐
|
||||
│ mem learn│ CLI
|
||||
│ (or any │ agent)
|
||||
└────┬─────┘
|
||||
│ POST /memory/learn
|
||||
┌────▼─────────────────────┐
|
||||
│ Memory Service (k8s) │
|
||||
│ │
|
||||
│ ┌─────────┐ │
|
||||
│ │ Auth │ JWT/rate │
|
||||
│ └────┬────┘ │
|
||||
│ ▼ │
|
||||
│ ┌─────────┐ │
|
||||
│ │ Chunker │ split on ## │
|
||||
│ └────┬────┘ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────┐ │
|
||||
│ │ Gated Loop │ │
|
||||
│ │ │ │
|
||||
│ │ chunk ──→ LLM ──→ │
|
||||
│ │ evaluate + compact │
|
||||
│ │ (update gate) │
|
||||
│ │ │ │
|
||||
│ │ Memory rewritten │
|
||||
│ │ each accepted turn │
|
||||
│ └──────┬───────┘ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────┐ │
|
||||
│ │ Embed │ │
|
||||
│ │ 768-dim vec │ │
|
||||
│ └──────┬───────┘ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────┐ │
|
||||
│ │ pgvector │ │
|
||||
│ │ UPSERT │ │
|
||||
│ │ (sha256 │ │
|
||||
│ │ dedup) │ │
|
||||
│ └──────────────┘ │
|
||||
│ │
|
||||
└──────────────────────────┘
|
||||
│
|
||||
┌─────────▼───────────┐
|
||||
│ Query Path │
|
||||
│ │
|
||||
│ /memory/query │
|
||||
│ /memory/context │
|
||||
│ │
|
||||
│ embed query → │
|
||||
│ pgvector search + │
|
||||
│ OpenSearch BM25 → │
|
||||
│ RRF fusion → │
|
||||
│ ranked results │
|
||||
└─────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key Properties
|
||||
|
||||
| Property | Mechanism |
|
||||
|---|---|
|
||||
| **No unbounded growth** | Gated loop rewrites memory, doesn't append. `memory_budget` caps size. |
|
||||
| **Deduplication** | SHA256 on compacted memory. `ON CONFLICT DO UPDATE`. |
|
||||
| **Idempotent re-ingest** | Same content → LLM rejects all chunks → nothing changes. |
|
||||
| **Quality gate** | LLM decides what's worth keeping. Title-only chunks rejected. |
|
||||
| **Compaction built-in** | Every accepted chunk triggers a rewrite that merges + trims. |
|
||||
| **Searchable** | Embedded in pgvector + indexed in OpenSearch. Hybrid retrieval. |
|
||||
| **Auditable** | JSONL event log records all gated loop decisions (immutable). |
|
||||
| **Model-agnostic** | `model` parameter selects evaluator LLM per request. |
|
||||
Reference in New Issue
Block a user