homelab-frontend/LLM_TOOL_CALLS.md

# LLM Tool Calls Testing Guide

This guide shows how to test the gateway with LLM tool calling (function calling) across different APIs.

## What's Tested

The gateway fully supports tool calling for:
- **OpenAI API** (`/v1/chat/completions`) - OpenAI, DeepSeek, etc.
- **Anthropic API** (`/llm/v1/messages`) - Claude models
- **Custom APIs** - Any LLM that supports tool definitions and responses

### Test Coverage

```
✅ OpenAI-style tool calling
✅ Streaming tool calls (SSE with tool_use blocks)
✅ Multi-turn conversations with tool results
✅ Parallel tool calls (multiple tools at once)
✅ Anthropic tool_use format
✅ Complex nested tool arguments
```

---

## Quick Start: Run Tests Locally

```bash
cd /Users/rockliang/workplace/homelab-frontend

# Run all tool call tests
go test ./internal/proxy/... -run "Tool" -v

# Or run with race detector (recommended)
go test -race ./internal/proxy/... -run "Tool" -v

# Expected output: 6 tests, all passing
```

## Test Scenarios

### 1. OpenAI-Style Tool Calling

**What it tests:**
- Request with tool definitions reaches upstream unmodified
- Upstream can return tool_calls in response
- Response with tool_calls passes through to client

**Test code:**
```go
// Request
{
  "model": "reasoning",
  "messages": [{"role": "user", "content": "What's the weather?"}],
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_weather",
      "description": "Get weather for a location",
      "parameters": {...}
    }
  }]
}

// Response (from upstream)
{
  "choices": [{
    "message": {
      "tool_calls": [{
        "id": "call_abc123",
        "type": "function",
        "function": {
          "name": "get_weather",
          "arguments": "{\"location\":\"San Francisco\"}"
        }
      }]
    }
  }]
}
```

**Run:**
```bash
go test ./internal/proxy/... -run TestToolCallOpenAIStyle -v
```

---

### 2. Streaming Tool Calls

**What it tests:**
- Tool calls can be streamed (SSE format)
- Multiple chunks arrive with tool_call deltas
- Stream completes with `[DONE]` sentinel

**Test code:**
```
Chunk 1: {"delta": {"role": "assistant"}, ...}
Chunk 2: {"delta": {"tool_calls": [{"id": "call_123", "function": {...}}]}, ...}
Chunk 3: {"delta": {}, "finish_reason": "tool_calls"}
Chunk 4: [DONE]
```

**Run:**
```bash
go test ./internal/proxy/... -run TestToolCallStreaming -v
```

---

### 3. Multi-Turn Conversation with Tool Results

**What it tests:**
- Client can send previous assistant's tool_calls back
- Tool result can be sent as a "tool" role message
- Assistant responds with final answer using tool result

**Flow:**
```
Turn 1: User asks → LLM decides to call tool
Turn 2: Client sends tool result → LLM generates final answer
```

**Test code:**
```go
// Turn 1 Request
{
  "model": "reasoning",
  "messages": [{"role": "user", "content": "What's the weather?"}],
  "tools": [...]
}

// Turn 1 Response (tool_calls)
{
  "message": {
    "tool_calls": [{
      "id": "call_abc",
      "function": {"name": "get_weather", "arguments": "..."}
    }]
  }
}

// Turn 2 Request (with tool result)
{
  "model": "reasoning",
  "messages": [
    {"role": "user", "content": "What's the weather?"},
    {"role": "assistant", "tool_calls": [...]},
    {"role": "tool", "content": "{\"temperature\": 22, \"condition\": \"cloudy\"}"}
  ],
  "tools": [...]
}

// Turn 2 Response (final answer)
{
  "message": {
    "content": "The weather in San Francisco is 22°C and cloudy."
  }
}
```

**Run:**
```bash
go test ./internal/proxy/... -run TestToolCallMultiTurn -v
```

---

### 4. Parallel Tool Calls

**What it tests:**
- LLM can request multiple tools in one response
- Gateway preserves all tool_calls
- Client can execute them in parallel

**Test code:**
```go
// Single response with 3 tool_calls
{
  "message": {
    "tool_calls": [
      {"id": "call_1", "function": {"name": "get_weather", "arguments": "{\"location\":\"New York\"}"}},
      {"id": "call_2", "function": {"name": "get_weather", "arguments": "{\"location\":\"London\"}"}},
      {"id": "call_3", "function": {"name": "get_weather", "arguments": "{\"location\":\"Tokyo\"}"}}
    ]
  }
}
```

**Run:**
```bash
go test ./internal/proxy/... -run TestParallelToolCalls -v
```

---

### 5. Anthropic Tool Use Format

**What it tests:**
- Different tool format: Anthropic uses `tool_use` blocks instead of `tool_calls`
- Gateway handles both formats transparently
- Tools are sent with `tools` parameter

**OpenAI format:**
```json
{"tool_calls": [{"type": "function", "function": {...}}]}
```

**Anthropic format:**
```json
{"content": [
  {"type": "text", "text": "..."},
  {"type": "tool_use", "id": "...", "name": "...", "input": {...}}
]}
```

**Test code:**
```go
// Request
{
  "model": "claude",
  "messages": [{"role": "user", "content": "What's the weather?"}],
  "tools": [{
    "name": "get_weather",
    "description": "Get weather",
    "input_schema": {...}
  }]
}

// Response (Anthropic format)
{
  "content": [
    {"type": "text", "text": "I'll check the weather..."},
    {"type": "tool_use", "id": "toolu_123", "name": "get_weather", "input": {...}}
  ],
  "stop_reason": "tool_use"
}
```

**Run:**
```bash
go test ./internal/proxy/... -run TestAnthropicToolUse -v
```

---

### 6. Complex Nested Tool Arguments

**What it tests:**
- Tool arguments can be complex JSON structures
- Nested objects, arrays, and deeply nested data preserved
- No argument modification or parsing

**Test code:**
```json
{
  "function": {
    "name": "create_event",
    "arguments": {
      "title": "Team Meeting",
      "time": "2025-08-20T14:00:00Z",
      "attendees": [
        {"name": "Alice", "email": "[email protected]"},
        {"name": "Bob", "email": "[email protected]"}
      ],
      "location": {
        "address": "123 Main St",
        "city": "San Francisco",
        "country": "USA"
      },
      "tags": ["important", "recurring"]
    }
  }
}
```

**Run:**
```bash
go test ./internal/proxy/... -run TestComplexToolArguments -v
```

---

## Running Against Real LLMs

### With Local Stubs (Current)

Tests use mock HTTP servers, so they run instantly:

```bash
go test ./internal/proxy/... -run "Tool" -v
# All 6 tests complete in ~220ms
```

### With Real Upstreams (Future)

Once you have real LLM services running, update the config:

```yaml
# k8s/configmap.yaml
models:
  - name: "reasoning"
    address: "reasoning-predictor.llm-serving:80"  # Real upstream
  - name: "claude"
    address: "claude-api.anthropic.com:443"         # Real Anthropic
```

Then use the gateway normally:

```bash
# Terminal 1: Start gateway
export CONFIG_PATH=config.yaml
go run ./cmd/gateway

# Terminal 2: Test with real LLM
curl -X POST http://localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "reasoning",
    "messages": [{"role": "user", "content": "What color is the sky?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "search",
        "description": "Search the internet",
        "parameters": {"type": "object", "properties": {}}
      }
    }]
  }'
```

---

## Gateway Behavior with Tool Calls

### Request Path

```
Client Request
    ↓
Body-based dispatch (find model)
    ↓
Look up upstream address
    ↓
Forward request unmodified (including tools)
    ↓
Upstream LLM processes tools
```

### Response Path

```
Upstream Response (with tool_calls or tool_use)
    ↓
Stream unbuffered if streaming
    ↓
Return to client exactly as received
```

### Key Properties

1. **No Rewriting**: Tool definitions and responses pass through unmodified
2. **Format Agnostic**: Both OpenAI `tool_calls` and Anthropic `tool_use` work
3. **Streaming Safe**: Tool calls stream incrementally without buffering
4. **Nested Structures**: Complex JSON arguments fully preserved

---

## Common Tool Call Patterns

### Pattern 1: Sequential Tool Use
```
Client → LLM (please use search tool)
       ← LLM (tool_calls: [search(...)])
Client → (execute search, send results)
Client → LLM (here are search results)
       ← LLM (final answer)
```

### Pattern 2: Parallel Tool Calls
```
Client → LLM (check weather in 3 cities)
       ← LLM (tool_calls: [get_weather(NY), get_weather(LA), get_weather(SF)])
Client → (execute all 3 in parallel)
Client → LLM (here are all results)
       ← LLM (summary)
```

### Pattern 3: Tool Result Formatting
```
Client receives tool_calls with:
- id: unique identifier
- function.name: tool name
- function.arguments: JSON string (always a string, not parsed object)

Client sends back:
- role: "tool"
- content: result JSON string
- tool_call_id: matches the original call id
```

---

## Verification Checklist

- [x] OpenAI-style tool definitions forward to upstream
- [x] Tool calls in response reach client unmodified
- [x] Streaming tool calls arrive incrementally
- [x] Multi-turn conversations preserve tool context
- [x] Parallel tool calls all included in response
- [x] Anthropic tool_use format works
- [x] Complex nested arguments preserved

Run all:
```bash
go test ./internal/proxy/... -run "Tool" -v --race
```

Expected: 6/6 passing, race detector clean

---

## Integration with Other Phases

### Phase 2.9: Anthropic Dialect
Currently, Anthropic tool calls work through the generic route handler. Phase 2.9 will add a dedicated `/llm/v1/messages` endpoint with full Anthropic-specific handling.

### Phase 2.13: Error Handling
Tool call errors (unknown tool, parsing errors) will have proper error responses in both OpenAI and Anthropic formats.

### Phase 3: Authentication
Tool calls work with all authentication methods (bearer tokens, API keys) - no special handling needed since tools are just part of the message payload.

### Phase 4: Rate Limiting
Tool calling counts the same as regular chat requests. Rate limits apply per conversation, not per tool call.

---

## Debugging Tool Calls

### Check if tool definitions reach upstream:

```bash
# Enable request logging
go test ./internal/proxy/... -run TestToolCallOpenAIStyle -v 2>&1 | grep -A5 "tool"
```

### Verify tool response format:

```bash
# Extract and pretty-print response
curl -X POST http://localhost:8080/v1/chat/completions ... | jq '.choices[0].message.tool_calls'
```

### Test streaming tool calls:

```bash
curl -N http://localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{..., "stream": true, "tools": [...]}'
# Should see incremental chunks with tool_use deltas
```

---

## FAQ

**Q: Do I need to modify the gateway code to support tool calls?**
A: No. Tool calls are just JSON in the request/response body. The gateway forwards them unchanged.

**Q: What if the LLM doesn't support tools?**
A: The tool definitions are simply ignored. The gateway doesn't validate or enforce tool support.

**Q: Can I mix OpenAI and Anthropic tool formats?**
A: Not in the same request. OpenAI clients expect `tool_calls`, Anthropic clients expect `tool_use` blocks. The upstream API determines the format.

**Q: How are tool arguments limited?**
A: By the per-route `maxBodySize` config. Complex nested arguments count toward that limit.

**Q: Can tool calls be streamed?**
A: Yes! SSE streaming fully supports tool calls. They arrive in delta chunks like text tokens.

---

## Next Steps

1. **Run tests locally:**
   ```bash
   go test ./internal/proxy/... -run "Tool" -v
   ```

2. **Deploy to cluster:**
   See [CLUSTER_REPO_SETUP.md](./CLUSTER_REPO_SETUP.md)

3. **Test against real LLMs:**
   Update config with real upstream addresses, restart gateway

4. **Phase 2.9:** Implement Anthropic dialect handler for `/llm/v1/messages`

5. **Phase 4:** Add tool call budgeting and rate limits