feat: add cache-aligned prompt builder for LLM API cost savings
PROBLEM: - PromptBuilder.build() puts everything in a single user message - System + query + memory + chunk all change together - LLM prompt caching gets 0% hits (entire message differs per call) - For a 50-chunk ingestion run, we pay full input price 50 times SOLUTION: PromptBuilder.build_cache_aligned() - Splits prompt into 3 separate messages: 1. SYSTEM: instructions (stable across ALL calls) → CACHED 2. USER[0]: query/problem (stable per run) → CACHED 3. USER[1]: memory + chunk (varies per call) → not cached - Cache prefix (system + query) reused across all chunks in a run - Estimated 30-70% cache hit ratio depending on chunk sizes - ~50% input token cost savings for multi-chunk ingestion TEMPLATES: - templates/gru-mem-system.txt (instructions only, 840B) - templates/gru-mem-query.txt (problem wrapper, 29B) - templates/gru-mem-turn.txt (memory + section, 57B) - templates/gru-mem.txt (legacy, unchanged) API: - PromptBuilder::build() — legacy, backward compatible - PromptBuilder::build_cache_aligned() → PromptMessages - PromptMessages.cache_prefix_tokens() — cacheable token count - PromptMessages.total_tokens() — total estimated tokens - PromptMessages.headroom() — tokens available for response TESTS: 11 unit + 3 integration = 14 new tests - test_cache_aligned_produces_two_user_messages - test_cache_prefix_is_stable_across_chunks - test_cache_prefix_is_stable_across_memory_changes - test_cache_prefix_tokens_positive - test_headroom_positive_under_budget - test_legacy_build_still_works - test_cache_aligned_contains_query - test_cache_aligned_memory/chunk_budget_exceeded - a8_cache_prefix_stable_across_50_chunks - a9_cache_aligned_headroom - a10_cache_savings_estimate TOTAL: 64 mem-core tests passing (52 unit + 12 integration)
This commit is contained in:
@@ -0,0 +1,3 @@
|
||||
<problem>
|
||||
{prompt}
|
||||
</problem>
|
||||
@@ -0,0 +1,10 @@
|
||||
You are a memory gate for a recurrent update system. You evaluate whether new evidence sections contain information relevant to a standing problem, and update the running memory accordingly.
|
||||
|
||||
Rules:
|
||||
1. Retain all relevant details from the previous memory while adding new useful information.
|
||||
2. Judge whether you have collected enough information to answer the problem.
|
||||
3. Reason about the new section between <think> and </think>.
|
||||
4. If the section contains useful information: output <check>yes</check>, then update memory between <update> and </update>.
|
||||
5. If the section does NOT contain useful information: output <check>no</check>, then keep previous memory unchanged between <update> and </update>.
|
||||
6. If more information is needed: return <next>continue</next>.
|
||||
7. ONLY when enough information is collected: return <next>end</next>.
|
||||
@@ -0,0 +1,7 @@
|
||||
<memory>
|
||||
{memory}
|
||||
</memory>
|
||||
|
||||
<section>
|
||||
{chunk}
|
||||
</section>
|
||||
Reference in New Issue
Block a user