Story Crater Bot
1991291bc9
PROBLEM: - PromptBuilder.build() puts everything in a single user message - System + query + memory + chunk all change together - LLM prompt caching gets 0% hits (entire message differs per call) - For a 50-chunk ingestion run, we pay full input price 50 times SOLUTION: PromptBuilder.build_cache_aligned() - Splits prompt into 3 separate messages: 1. SYSTEM: instructions (stable across ALL calls) → CACHED 2. USER[0]: query/problem (stable per run) → CACHED 3. USER[1]: memory + chunk (varies per call) → not cached - Cache prefix (system + query) reused across all chunks in a run - Estimated 30-70% cache hit ratio depending on chunk sizes - ~50% input token cost savings for multi-chunk ingestion TEMPLATES: - templates/gru-mem-system.txt (instructions only, 840B) - templates/gru-mem-query.txt (problem wrapper, 29B) - templates/gru-mem-turn.txt (memory + section, 57B) - templates/gru-mem.txt (legacy, unchanged) API: - PromptBuilder::build() — legacy, backward compatible - PromptBuilder::build_cache_aligned() → PromptMessages - PromptMessages.cache_prefix_tokens() — cacheable token count - PromptMessages.total_tokens() — total estimated tokens - PromptMessages.headroom() — tokens available for response TESTS: 11 unit + 3 integration = 14 new tests - test_cache_aligned_produces_two_user_messages - test_cache_prefix_is_stable_across_chunks - test_cache_prefix_is_stable_across_memory_changes - test_cache_prefix_tokens_positive - test_headroom_positive_under_budget - test_legacy_build_still_works - test_cache_aligned_contains_query - test_cache_aligned_memory/chunk_budget_exceeded - a8_cache_prefix_stable_across_50_chunks - a9_cache_aligned_headroom - a10_cache_savings_estimate TOTAL: 64 mem-core tests passing (52 unit + 12 integration)
3 lines
29 B
Plaintext
3 lines
29 B
Plaintext
<problem>
|
|
{prompt}
|
|
</problem> |