diff --git a/API.md b/API.md index 0fa26c9..3d513b5 100644 --- a/API.md +++ b/API.md @@ -783,3 +783,147 @@ curl -X POST https://api.riotpiao.com/workflow \ - **Readiness:** `curl https://api.riotpiao.com/readyz` - **Models:** `curl https://api.riotpiao.com/v1/models` - **Logs:** `kubectl -n api logs deployment/homelab-frontend` + +--- + +## LLM Inference in Workflows + +The Poimen workflows system includes built-in LLM inference activities that call `/v1/chat/completions` via the gateway. + +### LLMInferenceActivity + +Single-prompt LLM inference within a workflow. + +**Workflow Definition (Canvas Node):** +```json +{ + "id": "llm-node-1", + "type": "llm-inference", + "label": "Analyze Code with LLM", + "data": { + "model": "reasoning", + "system_prompt": "You are a code analysis expert. Provide detailed feedback.", + "user_prompt": "Analyze this code for security issues: {{ previous_output.code }}", + "temperature": 0.7, + "max_tokens": 2048 + } +} +``` + +**Backend Implementation:** +The LLMInferenceActivity in the workflows backend automatically: +1. Substitutes template variables (e.g., `{{ previous_output.code }}`) +2. Calls `/v1/chat/completions` with the resolved prompt +3. Returns the LLM response as activity output +4. Retries on transient failures (up to 3 attempts) +5. Timeouts after 120 seconds + +**Output:** +```json +{ + "response": "The code has several security vulnerabilities...", + "model": "reasoning", + "stop_reason": "stop_sequence", + "tokens_used": 450 +} +``` + +**Supported Models:** +- `reasoning` — DeepSeek-R1-Distill (best for complex analysis) +- `ornith:35b` — Ollama 35B +- `ornith:13b` — Ollama 13B +- `qwen2.5:3b` — Qwen 2.5 3B + +--- + +### LLMBatchInferenceActivity + +Multiple-prompt LLM inference (sequential processing). + +**Workflow Definition:** +```json +{ + "id": "llm-batch-1", + "type": "llm-batch-inference", + "label": "Batch Code Review", + "data": { + "model": "reasoning", + "system_prompt": "Review each code snippet and provide feedback.", + "prompts": [ + "Review snippet 1: {{ files[0].content }}", + "Review snippet 2: {{ files[1].content }}", + "Review snippet 3: {{ files[2].content }}" + ] + } +} +``` + +**Output:** +```json +{ + "responses": [ + "Snippet 1 review...", + "Snippet 2 review...", + "Snippet 3 review..." + ], + "model": "reasoning", + "errors": [] +} +``` + +**Typical Use Cases:** +- Batch code review across multiple files +- Parallel document summarization +- Comparative analysis of alternatives +- Policy compliance checking + +--- + +### Workflow Integration Examples + +**1. Code Analysis Workflow** +``` +Clone Repo → Analyze Code → LLM Security Review → Generate Report → Notify +``` + +**2. Document Processing** +``` +Retrieve Documents → Embed + Index → LLM Summarize (batch) → Archive +``` + +**3. Multi-Stage Review** +``` +Retrieve Memory → LLM Context Extraction → Route to Activity A/B/C → Notify +``` + +--- + +### Error Handling + +If LLM inference fails: +- First activity retry (2-second backoff) +- Second activity retry (4-second backoff) +- Third activity retry (8-second backoff) +- If all retries fail, workflow records error and proceeds to next activity (or fails if terminal) + +**Common Failure Scenarios:** +- Network timeout: `connection refused` (retry automatically) +- Model not found: `unknown model: xyz` (terminal error) +- Rate limited: HTTP 429 (retry with exponential backoff) +- Prompt too long: `context length exceeded` (terminal error) + +--- + +### Performance & Cost + +- Single prompt inference: ~100-500ms (model-dependent) +- Batch processing: Serial (not parallel), ~100-500ms per prompt +- Model inference costs: Free (on-premise Ollama/Reasoning models) +- Token counting: Provided in response for quota tracking + +**Optimization Tips:** +- Use `ornith:13b` or `qwen2.5:3b` for faster inference +- Use `reasoning` only for complex analysis that needs reasoning +- Cache frequently-used prompts at workflow level +- Use batch activity for multiple similar prompts (better throughput) +