homelab/k8s/applications/llm/README.md

# Ollama LLM Inference Service

CPU-only LLM inference server on talos-cp-1. Single model hot-loaded (DeepSeek-R1:70b), 42GB, 70Gi memory limit.

## Quick Start

### Access via port-forward
```bash
kubectl -n llm port-forward svc/ollama 11434:11434
curl http://localhost:11434/api/tags
```

### Debug pod (in-cluster)
```bash
kubectl run debug --rm -it -n llm --image=curlimages/curl \
  --labels="app.kubernetes.io/role=llm-debug" \
  --serviceaccount=llm-worker -- sh

# Inside pod
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl -H "Authorization: Bearer $TOKEN" \
  http://ollama.llm.svc.cluster.local:11434/api/tags
```

## Architecture

| Component | Value |
|-----------|-------|
| Service | ClusterIP `ollama.llm.svc.cluster.local:11434` |
| Namespace | `llm` |
| Node | talos-cp-1 (pinned via nodeAffinity) |
| Memory request | 50Gi |
| Memory limit | 70Gi |
| Storage | 115Gi PVC (Longhorn) |
| Model | `deepseek-r1:70b` (~42GB) |
| Max loaded | 1 model |
| Parallelism | 1 request at a time |

## API Endpoints

### List models
```bash
curl http://ollama.llm.svc.cluster.local:11434/api/tags
```

Response:
```json
{
  "models": [
    {"name": "deepseek-r1:70b", "size": 42000000000, ...}
  ]
}
```

### Generate (non-streaming)
```bash
curl -X POST http://ollama.llm.svc.cluster.local:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-r1:70b",
    "prompt": "Why is the sky blue?",
    "stream": false
  }'
```

### Pull model
```bash
curl -X POST http://ollama.llm.svc.cluster.local:11434/api/pull \
  -H "Content-Type: application/json" \
  -d '{"name": "deepseek-r1:70b", "stream": false}'
```

## Operations

### Check pod status
```bash
kubectl -n llm get pod -l app.kubernetes.io/name=ollama
kubectl -n llm describe pod -l app.kubernetes.io/name=ollama
```

### View logs
```bash
kubectl -n llm logs deployment/ollama -f
```

### Monitor download progress (bootstrap)
```bash
kubectl -n llm logs -f job/bootstrap-models -c model-download
```

### Restart deployment
```bash
kubectl -n llm rollout restart deployment/ollama
```

## Storage

- **PVC:** `ollama-models-cache`, 115Gi, Longhorn StorageClass
- **Mount:** `/root/.ollama/models` (Ollama model cache)
- **Lifecycle:** RWO (Read-Write-Once), tied to talos-cp-1

### Resize PVC
⚠️ PVCs can only expand, not shrink. Edit values.yaml and redeploy:

```yaml
pvc:
  size: 120Gi  # increase only
```

```bash
vsource .env && helmfile -f helmfile.yaml.gotmpl -l name=ollama apply
```

## Networking

### NetworkPolicy
- Default-deny ingress on Ollama pods
- Allow from pods labeled `app.kubernetes.io/name: llm-worker` (port 11434)
- Allow from pods labeled `app.kubernetes.io/role: llm-debug` (port 11434)

View policy:
```bash
kubectl -n llm get networkpolicy ollama
```

Test access from external pod (should fail):
```bash
kubectl run test --rm -it --image=curlimages/curl -- \
  curl http://ollama.llm.svc.cluster.local:11434/
# Connection timeout (correct)
```

Test access from debug pod (should succeed):
```bash
kubectl -n llm logs job/bootstrap-models  # verify bootstrap completed
# Then run debug pod as shown above
```

## Configuration

### Helm values (`k8s/llm/charts/ollama/values.yaml`)

```yaml
resources:
  requests:
    cpu: 8
    memory: 50Gi
  limits:
    cpu: 16
    memory: 70Gi

env:
  OLLAMA_MAX_LOADED_MODELS: "1"
  OLLAMA_NUM_PARALLEL: "1"
  OLLAMA_MAX_QUEUE: "32"
  OLLAMA_KEEP_ALIVE: "-1"
  OLLAMA_HOST: "0.0.0.0:11434"

preloadJob:
  enabled: true
  hotModels:
    - deepseek-r1:70b
```

### Environment variables

| Variable | Value | Purpose |
|----------|-------|---------|
| `OLLAMA_MODELS` | `/root/.ollama/models` | Model cache dir |
| `OLLAMA_MAX_LOADED_MODELS` | `1` | Max concurrent models in RAM |
| `OLLAMA_NUM_PARALLEL` | `1` | Parallel request threads |
| `OLLAMA_MAX_QUEUE` | `32` | Request queue depth |
| `OLLAMA_KEEP_ALIVE` | `-1` | Keep model resident (never unload) |
| `OLLAMA_HOST` | `0.0.0.0:11434` | Bind address |

Tune `OLLAMA_NUM_PARALLEL` based on CPU cores. Current: 1 (conservative, CPU bottleneck).

## Model Management

### Current model
- **Name:** `deepseek-r1:70b`
- **Size:** ~42GB
- **Quantization:** Default Ollama quant
- **Status:** Downloaded during pod init via bootstrap job

### Change model

1. Edit `values.yaml`:
```yaml
preloadJob:
  hotModels:
    - deepseek-r1:32b  # or any available model
```

2. Redeploy:
```bash
kubectl -n llm delete job bootstrap-models --ignore-not-found
vsource .env && helmfile -f helmfile.yaml.gotmpl -l name=ollama apply
```

3. Monitor:
```bash
kubectl -n llm logs -f job/bootstrap-models -c model-download
```

### Available models
Ollama registry: https://ollama.com/library

Examples:
- `deepseek-r1:70b` (reasoning, 42GB)
- `deepseek-r1:32b` (faster, 20GB)
- `llama3.1:70b` (general, 41GB)
- `mistral:large` (26GB)

## Troubleshooting

### Pod stuck in `ContainerCreating`
```bash
kubectl -n llm describe pod -l app.kubernetes.io/name=ollama
# Check Events section for PVC/image pull issues
```

### Bootstrap job failing
```bash
kubectl -n llm logs job/bootstrap-models -c model-download --tail=50
# Common: model not found in registry, disk full, network timeout
```

### Model pull timeout
```bash
# Increase pod timeout (edit deployment directly)
kubectl -n llm edit deployment ollama
# Change readinessProbe.initialDelaySeconds, livenessProbe.periodSeconds
```

### Out of memory
Model size exceeds limit. Reduce `memory.limits` or choose smaller model.

```bash
kubectl top pod -n llm  # check actual usage
```

### Cannot connect from other pods
Verify NetworkPolicy:
```bash
kubectl -n llm get networkpolicy
kubectl -n llm describe networkpolicy ollama
# Add pod label: app.kubernetes.io/name: llm-worker or app.kubernetes.io/role: llm-debug
```

## Secrets

Ollama pod receives MinIO credentials via Secret `ollama-minio` (created by helmfile presync):

```bash
kubectl -n llm get secret ollama-minio -o jsonpath='{.data}' | jq
```

Keys: `endpoint`, `bucket`, `access_key`, `secret_key`

Used by bootstrap job to upload model blobs to MinIO (future: auto-backup).

## Metrics & Observability

### Prometheus scrape (if enabled)
ServiceMonitor: Not yet configured (see `k8s/monitoring/dashboards/services/`)

Metrics to add:
- `ollama_requests_total` (counter)
- `ollama_request_duration_seconds` (histogram)
- `ollama_loaded_models` (gauge)

### Logs
Pod logs via kubectl. No log aggregation to Loki yet.

```bash
kubectl -n llm logs deployment/ollama -f --timestamps
```

## Cleanup

### Delete Ollama completely
```bash
vsource .env && helmfile -f helmfile.yaml.gotmpl -l name=ollama destroy
# Keeps PVC (data safety). To delete: kubectl -n llm delete pvc ollama-models-cache
```

### Delete just the model cache (keep deployment)
```bash
kubectl -n llm delete pvc ollama-models-cache
# Recreate: kubectl -n llm patch deployment ollama -p '{"spec":{"template":{"metadata":{"annotations":{"restart":"now"}}}}}'
```

## See Also

- Helmfile: `helmfile.yaml.gotmpl` (llm release block)
- Chart: `k8s/llm/charts/ollama/`
- Namespace: `llm`
- Bootstrap: `k8s/llm/bootstrap-models-job.yaml` (manual preload fallback)