homelab/project-usage/temporal-workflows.md

Temporal Workflow Orchestration

Server: temporal.temporal.svc.cluster.local:7233 (cluster-internal)
Web UI: kubectl port-forward -n temporal svc/temporal-web 8088:8088
Namespace: temporal

When to Use

• Long-running operations — Tasks that take minutes/hours (send email, process batch, retry with backoff)
• State machines — Multi-step workflows with decision logic
• Retries & timeouts — Built-in exponential backoff, automatic retry
• Audit trail — Full history of workflow executions (why it happened, when, by whom)

Quick Start

1. Access Temporal Web UI:

k port-forward -n temporal svc/temporal-web 8088:8088
# http://localhost:8088

2. Define workflow (Go example):

package workflows

import (
  "time"
  "go.temporal.io/sdk/workflow"
  "go.temporal.io/sdk/activity"
)

type Inputs struct {
  OrderID string
  Amount  float64
}

// Workflow definition
func OrderProcessing(ctx workflow.Context, input Inputs) (string, error) {
  // Step 1: Charge payment
  chargeResult := ""
  err := workflow.ExecuteActivity(
    ctx,
    ChargePayment,
    input.OrderID,
    input.Amount,
  ).Get(ctx, &chargeResult)
  if err != nil {
    return "", err
  }

  // Step 2: Send confirmation email (retry 3x on failure)
  emailResult := ""
  opts := workflow.ActivityOptions{
    StartToCloseTimeout: time.Minute,
    RetryPolicy: &temporal.RetryPolicy{
      InitialInterval:    time.Second,
      BackoffCoefficient: 2,
      MaximumAttempts:    3,
    },
  }
  ctx = workflow.WithActivityOptions(ctx, opts)
  
  err = workflow.ExecuteActivity(ctx, SendConfirmationEmail, input.OrderID).Get(ctx, &emailResult)
  if err != nil {
    return "", err
  }

  return "order_processed", nil
}

// Activity: payment processing
func ChargePayment(ctx context.Context, orderID string, amount float64) (string, error) {
  // Call payment gateway
  return "payment_successful", nil
}

// Activity: email notification
func SendConfirmationEmail(ctx context.Context, orderID string) (string, error) {
  // Send email
  return "email_sent", nil
}

3. Register & start workflow:

import "go.temporal.io/sdk/client"

client, _ := client.Dial(client.Options{
  HostPort: "temporal.temporal.svc.cluster.local:7233",
})

// Start workflow execution
run, _ := client.ExecuteWorkflow(ctx, client.StartWorkflowOptions{
  ID:        "order-123",
  TaskQueue: "orders",
}, OrderProcessing, Inputs{OrderID: "123", Amount: 99.99})

// Wait for result
var result string
run.Get(ctx, &result)

Configuration

Key	Value
Server	temporal.temporal.svc.cluster.local:7233
Web UI	localhost:8088 (via port-forward)
Database	PostgreSQL (managed by helmfile)
Task queue	default, orders, emails (custom per app)
Retention	30 days (configurable)

Common Patterns

Retry with exponential backoff:

opts := workflow.ActivityOptions{
  StartToCloseTimeout: 5 * time.Minute,
  RetryPolicy: &temporal.RetryPolicy{
    InitialInterval:    time.Second,
    BackoffCoefficient: 2.0,           // double wait time each retry
    MaximumInterval:    time.Minute,   // cap at 1 min between retries
    MaximumAttempts:    5,             // give up after 5 tries
  },
}
ctx = workflow.WithActivityOptions(ctx, opts)

Wait for signal (user approval):

// Workflow waits for approval signal
approval := ""
workflow.GetSignalChannel(ctx, "approval").Receive(ctx, &approval)

if approval == "approved" {
  // Continue workflow
} else {
  return "", errors.New("request denied")
}

Parallel activities:

// Execute email & SMS in parallel
emailFuture := workflow.ExecuteActivity(ctx, SendEmail, userID)
smsFuture := workflow.ExecuteActivity(ctx, SendSMS, userID)

// Wait for both to complete
emailFuture.Get(ctx, nil)
smsFuture.Get(ctx, nil)

Scheduled workflow (cron):

run, _ := client.ExecuteWorkflow(ctx, client.StartWorkflowOptions{
  ID:              "daily-report",
  CronSchedule:    "0 9 * * MON-FRI",  // 9 AM weekdays
  WorkflowTaskTimeout: time.Hour,
}, GenerateDailyReport, nil)

Monitoring

Web UI:

• List workflows: http://localhost:8088/namespaces/default/workflows
• View execution history: Click workflow ID
• See activity logs, errors, retry attempts

Grafana dashboard: svc-temporal (auto-configured)

Key metrics:

• temporal_workflow_execution_duration_seconds — workflow time
• temporal_activity_execution_duration_seconds — activity time
• temporal_activity_execution_failed_total — failed activities

Integration with Story Crater

Example: Process message via Temporal:

// In message handler
client, _ := temporal.Dial(/* ... */)

run, _ := client.ExecuteWorkflow(ctx, client.StartWorkflowOptions{
  ID:        fmt.Sprintf("msg-%s", messageID),
  TaskQueue: "story-crater",
}, ProcessMessageWorkflow, Message{
  ID:     messageID,
  Body:   body,
  Source: "kafka-queue",
})

// Non-blocking: workflow runs independently
// Check status later

Troubleshooting

Workflow stuck:

# Check Temporal server health
k get pods -n temporal

# View workflow execution history (via Web UI or CLI)
tctl workflow show --workflow-id order-123

# Terminate stuck workflow
tctl workflow terminate --workflow-id order-123

Activity retrying endlessly:

// Add max attempts or timeout
RetryPolicy: &temporal.RetryPolicy{
  MaximumAttempts: 5,  // must have this!
}

PostgreSQL connection fails:

# Check temporal pod logs
k logs -n temporal pod/temporal-0

# Verify database is running
k get pods -n ddb

See /TROUBLESHOOTING.md for full incident guide.