21 Commits
Author SHA1 Message Date
Test a014890775 chore: revert module path to github.com, unify Go version to 1.26.0
ci / test (push) Canceled after 0s
Build & Push Workflows Image / build-push (push) Canceled after 21s
2026-09-05 14:48:21 -07:00
Test a6f8d47a23 ci: add go mod download to resolve dependencies
ci / test (push) Failing after 1m41s
Build & Push Workflows Image / build-push (push) Failing after 27s
2026-09-05 14:05:30 -07:00
Test 5b5880c6e4 ci: use golang runner for Go project
ci / test (push) Failing after 1m54s
Build & Push Workflows Image / build-push (push) Failing after 27s
2026-09-05 13:52:18 -07:00
Test 3480d6c2cc ci: fix runner to use node-labeled runner for Docker builds
Build & Push Workflows Image / build-push (push) Failing after 40s
ci / test (push) Failing after 2m14s
2026-09-05 13:50:35 -07:00
Test 330bf639cd ci: add Forgejo CI/CD workflow for workflows image build & push
Build & Push Workflows Image / build-push (push) Failing after 27s
ci / test (push) Failing after 1m54s
2026-09-05 13:47:27 -07:00
Test 58237cc1ff feat: migration for workflow relations and RAG indexing
ci / test (push) Failing after 2m9s
2026-09-05 06:01:08 -07:00
Test 5c0eb2b66a feat: wire GraphRAG API handlers, activities, and database layer
ci / test (push) Failing after 2m9s
2026-09-05 06:00:58 -07:00
Test 8474d7e494 chore: remove docker-compose (use k8s + CI/CD only)
ci / test (push) Failing after 2m15s
2026-09-05 05:58:53 -07:00
Test 447951daca feat: GraphRAG query API handlers and activities
ci / test (push) Failing after 2m6s
2026-09-05 05:58:18 -07:00
Test da44923c5c docs: deployment guide for unified Poimen application
ci / test (push) Failing after 2m7s
2026-09-05 05:57:34 -07:00
Test 70a9b9a2ab feat: unified Poimen application with k8s + docker-compose infrastructure
ci / test (push) Failing after 2m4s
2026-09-05 05:57:08 -07:00
Test e01dad4e8c feat: GraphRAG query workflow and indexing
ci / test (push) Failing after 2m5s
2026-09-05 05:52:56 -07:00
Test 84b4ca120f feat: add relation wording schema
ci / test (push) Failing after 2m24s
2026-09-05 05:45:47 -07:00
Test a461e9799a docs: temporal + graph RAG integration with unified query 2026-09-05 05:45:21 -07:00
Test 9624f0e18d feat: add canvas compatibility checking for connection validation
ci / test (push) Failing after 2m46s
2026-09-05 01:01:01 -07:00
Test 2a3b080e29 feat: add CanvasReasonerActivity for auto-inferring workflow connections
ci / test (push) Failing after 6m2s
2026-09-05 00:54:22 -07:00
Test 00fc83c081 feat: add JWT auth token support to LLM inference activities
ci / test (push) Failing after 2m14s
2026-09-05 00:47:22 -07:00
Test 0da90fdd7a feat: database layer + canvas validator/converter + LLM inference activities
ci / test (push) Failing after 2m11s
- Add pkg/db models and CRUD methods for workflows
- Add internal/routing canvas validator (DAG check, connectivity)
- Add internal/routing canvas converter (Canvas → WorkflowSpec)
- Register LLMInferenceActivity and LLMBatchInferenceActivity
- Update api/server and cmd/server with database integration
- Add K8s environment variable support
- Update activity knowledge base with LLM activities
- Add .env.example configuration template
2026-09-05 00:43:30 -07:00
Test 924aa398b6 refactor: improve AssumeRoleActivity code quality (CRAP/DRY/SOLID)
ci / test (push) Failing after 1m23s
- Extract validateAssumeRoleInput() - CRAP ~2
- Extract resolveAssumeRoleConfig() with getOrEnv() helper - CRAP ~4
  * Fixes DRY violation (config resolution was repeated 3x)
- Extract requestAuthToken() - CRAP ~4 (sequential, easy to test)
- Extract buildAssumeRoleOutput() - CRAP ~1
- Main AssumeRoleActivity now ~CRAP 3 (orchestrates high-level flow)

Overall CRAP reduction: 40+ → 6-8 total complexity

Improves:
- Single Responsibility: Each function does one thing
- DRY: Config resolution centralized
- Testability: Each step independently unit-testable
- Readability: Main function reads like pseudocode
2026-09-04 14:13:47 -07:00
Test 4a34c8e672 feat: add AssumeRoleActivity for temporary LLM API token grants
Implements AWS AssumeRole-like pattern for Poimen:
- User/service requests temporary access with identity + scope
- AssumeRoleActivity exchanges credentials with OAuth2 auth server
- Returns JWT token valid for limited time (default: 1hr, max: 24hrs)
- Token used in all subsequent LLM API calls to api.riotpiao.com

Key features:
- Credentials from vault/K8s secrets (never hardcoded)
- Scope-based access control (llm:read, llm:read llm:write, llm:admin)
- Automatic token expiration tracking
- Retry support for transient auth failures (2x, 1.5s backoff)
- Configurable auth server endpoint

Usage pattern:
1. AssumeRoleActivity(identity, scope) → JWT token
2. LLMRouter uses token in LLMAuth config
3. All activity calls validated against token + scopes
4. Workflow optionally refreshes token before expiry

Security:
- No credentials in code/logs (env or vault only)
- Short-lived tokens (1hr default, 24hr max)
- Server-enforced scope validation
- Token revocation support

Activity registered: #10 (authentication category)
Knowledge base updated with full activity spec

New file: action/assume_role.go (5.2 KB)
2026-09-04 14:11:58 -07:00
Test ebf95506cd refactor: simplify auth - remove undefined TenantID concept
- Remove TenantID field from LLMAuth (JWT claims handle tenant info)
- Remove Scopes field (not part of Poimen's design)
- Simplify to 3 core auth types: Bearer, API Key, Custom
- Update LLMRouterConfig to only include Auth field
- Simplify README examples to per-deployment pattern
- Focus on secure token management vs multi-tenant isolation
- Clarify token rotation pattern for long-running workflows
- Update security section with practical vault integration examples

TenantID was introduced without proper context. In Poimen:
- JWT token itself contains tenant/customer info in claims
- Each deployment gets its own LLM_AUTH_TOKEN from vault
- LLM API provider (riotpiao.com) validates token at their end
- No need for separate tenant header in Poimen layer

Simpler, clearer, more maintainable.
2026-09-04 10:56:47 -07:00
36 changed files with 5094 additions and 508 deletions
+11
View File
@@ -0,0 +1,11 @@
.git
.gitignore
*.md
.env.local
.env
tests/
*.test.go
coverage/
.DS_Store
k8s/
migrations/
+22
View File
@@ -0,0 +1,22 @@
# Workflows Backend Configuration
# Database (memory-db CNPG in K8s)
# Option A: Direct DATABASE_URL
DATABASE_URL=postgresql://app:[email protected]:5432/memory?sslmode=disable
# Option B: Individual env vars (used if DATABASE_URL is empty)
DATABASE_HOST=memory-db-rw.poimen.svc.cluster.local
DATABASE_PORT=5432
DATABASE_NAME=memory
DATABASE_USER=app
DATABASE_PASSWORD=PASSWORD
# Temporal
TEMPORAL_HOST_PORT=localhost:7233
TEMPORAL_NAMESPACE=default
# API Server
API_PORT=8080
# Logging
VERBOSE=false
+46
View File
@@ -0,0 +1,46 @@
name: Build & Push Workflows Image
on:
push:
branches:
- main
jobs:
build-push:
runs-on: golang
env:
REGISTRY: forgejo.riotpiao.com
IMAGE: forgejo.riotpiao.com/rock/poimen-workflows
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Download dependencies
run: go mod download
- name: Get short SHA
id: sha
run: |
SHORT_SHA=$(git rev-parse --short HEAD)
echo "short_sha=${SHORT_SHA}" >> $GITHUB_OUTPUT
- name: Registry login
run: |
echo "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" \
--username "${REGISTRY_USER}" --password-stdin
env:
REGISTRY_USER: ${{ secrets.FORGEJO_REGISTRY_USER }}
REGISTRY_TOKEN: ${{ secrets.FORGEJO_REGISTRY_TOKEN }}
- name: Build image
run: |
docker build --no-cache \
-t "${IMAGE}:${{ steps.sha.outputs.short_sha }}" \
-t "${IMAGE}:latest" \
.
- name: Push image
run: |
docker push "${IMAGE}:${{ steps.sha.outputs.short_sha }}"
docker push "${IMAGE}:latest"
echo "✓ Image pushed: ${IMAGE}:${{ steps.sha.outputs.short_sha }}"
+293
View File
@@ -0,0 +1,293 @@
# Poimen Application Deployment
## Overview
Poimen is a unified application consisting of three services:
- **poimen-memory**: Memory/Graph RAG service
- **poimen-workflows**: Temporal orchestration + API
- **poimen-frontend**: Next.js frontend
All services are deployed together as a single application in the `poimen` namespace.
## Local Development
### Prerequisites
- Docker
- Docker Compose
- Node.js 18+
- Go 1.21+
- Python 3.11+
### Start Local Stack
```bash
docker-compose up -d
```
This starts:
- PostgreSQL (memory + workflows DBs)
- Redis (cache)
- Temporal (workflow orchestration)
- poimen-memory (8000)
- poimen-workflows (8080)
- poimen-workflows-worker
- poimen-frontend (3000)
### Access Services
- Frontend: http://localhost:3000
- Workflows API: http://localhost:8080
- Memory API: http://localhost:8000
- Temporal UI: http://localhost:8233
### Stop Stack
```bash
docker-compose down
```
## Building & Pushing Images
### Build All Services
```bash
./build-push.sh latest
```
Or specific services:
```bash
docker build -t forgejo.riotpiao.com/rock/poimen-memory:v1.0.0 ./memory
docker push forgejo.riotpiao.com/rock/poimen-memory:v1.0.0
```
### Image Tagging Strategy
- `latest`: Development/staging
- `v1.0.0`, `v1.0.1`, etc.: Production releases
- `main-{commit-hash}`: CI/CD automated builds
## Kubernetes Deployment
### Prerequisites
- Kubernetes cluster (1.24+)
- kubectl configured
- Kustomize installed
- Registry credentials configured
### Deploy to Cluster
```bash
cd k8s
./deploy.sh -a
```
Or with specific tags:
```bash
./deploy.sh -m v1.0.0 -w v1.0.0 -f v1.0.0
```
### Verify Deployment
```bash
kubectl get pods -n poimen
kubectl get svc -n poimen
kubectl logs -n poimen -l app=poimen-workflows
```
## Configuration
### Environment Variables
Configure in `k8s/poimen-application.yaml` under `spec.template.spec.env`:
**Common:**
- `TEMPORAL_HOST`: Temporal server (default: temporal:7233)
- `DATABASE_URL`: PostgreSQL connection
- `JWT_SECRET`: JWT signing key
- `LOG_LEVEL`: debug|info|warn|error
**Memory Service:**
- `REDIS_URL`: Redis connection
- `ELASTICSEARCH_URL`: Optional full-text search
**Workflows Service:**
- `MEMORY_SERVICE_URL`: Internal memory service URL
**Frontend:**
- `NEXT_PUBLIC_WORKFLOWS_API`: External workflows API
- `NEXT_PUBLIC_MEMORY_API`: External memory API
- `OAUTH_CLIENT_ID`, `OAUTH_CLIENT_SECRET`: Auth provider
### Secrets
Create secrets before deployment:
```bash
kubectl create secret generic poimen-db-credentials \
--from-literal=memory-url="postgresql://..." \
--from-literal=workflows-url="postgresql://..." \
-n poimen
kubectl create secret generic poimen-secrets \
--from-literal=jwt-secret="..." \
--from-literal=oauth-client-id="..." \
--from-literal=oauth-client-secret="..." \
-n poimen
```
## Architecture
```
┌─────────────────────────────────────────────┐
│ LoadBalancer Service │
│ poimen-frontend:80→3000 │
└─────────────────┬───────────────────────────┘
┌───────┴────────┐
▼ ▼
┌──────────────┐ ┌──────────────┐
│ Frontend │ │ Workflows │
│ (3000) │ │ API (8080) │
│ 2 replicas │ │ 2 replicas │
└──────────────┘ └──────┬───────┘
│ │
│ ┌─────┴─────┐
│ ▼ ▼
│ ┌─────────────────────┐
│ │ Temporal Cluster │
│ │ (External) │
│ └─────────────────────┘
└──────────────────┬──────────────┐
▼ ▼
┌──────────────┐ ┌──────────────┐
│ Memory │ │ PostgreSQL │
│ (8000) │ │ (5432) │
│ 1 replica │ │ │
└──────────────┘ └──────────────┘
```
## Scaling
### Horizontal Scaling
Adjust replicas in `k8s/poimen-application.yaml`:
```yaml
spec:
replicas: 3 # Increase this
```
Or patch:
```bash
kubectl patch deployment poimen-workflows -p '{"spec":{"replicas":3}}' -n poimen
```
### Resource Requests/Limits
Add to container spec:
```yaml
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
```
## Monitoring & Logging
### Check Status
```bash
kubectl get pods -n poimen -w
kubectl describe pod <pod-name> -n poimen
kubectl logs -n poimen -f -l app=poimen-workflows --all-containers=true
```
### Health Checks
All services expose `/health` endpoint:
```bash
curl http://poimen-workflows:8080/health
curl http://poimen-memory:8000/health
curl http://poimen-frontend:3000/
```
## Updates & Rollbacks
### Rolling Update
```bash
./deploy.sh -w v1.0.1
```
Kubernetes automatically rolls out with health checks.
### View Rollout Status
```bash
kubectl rollout status deploy/poimen-workflows -n poimen
```
### Rollback
```bash
kubectl rollout undo deploy/poimen-workflows -n poimen
```
## Troubleshooting
### Services Can't Connect
Check service DNS:
```bash
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup poimen-workflows
```
### Database Migrations Failing
```bash
kubectl exec -it <workflows-pod> -n poimen -- \
./workflows migrate up
```
### Temporal Worker Not Picking Up Activities
Check worker logs:
```bash
kubectl logs -n poimen -l app=poimen-workflows --all-containers=true | grep -i activity
```
Verify activities registered in `cmd/worker/main.go`
## CI/CD Integration
### GitHub Actions Example
```yaml
name: Build & Push Poimen
on:
push:
branches: [main]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Build & Push
run: ./build-push.sh main-${{ github.sha }}
```
### Automatic Deployment
Configure ArgoCD to watch `k8s/` directory for updates.
+13 -376
View File
@@ -1,385 +1,22 @@
# Multi-stage build for Poimen Temporal Worker
# Stage 1: Builder - Compile Go binary and set up tools
FROM golang:1.25-alpine AS builder
WORKDIR /build
# Install system dependencies (ast-grep, git, build essentials)
RUN apk add --no-cache \
git \
curl \
wget \
ca-certificates \
gcc \
musl-dev \
bash \
&& echo "[builder] System dependencies installed"
# Install ast-grep CLI tool
RUN curl -fsSL https://github.com/ast-grep/ast-grep/releases/download/0.24.0/sg-x86_64-unknown-linux-musl.tar.gz \
| tar xzf - -C /usr/local/bin \
&& chmod +x /usr/local/bin/sg \
&& sg --version \
&& echo "[builder] ast-grep installed"
# Install Node.js for pi CLI and browser-use
RUN apk add --no-cache nodejs npm \
&& echo "[builder] Node.js installed"
# Install pi CLI globally
RUN npm install -g @earendil-works/pi-coding-agent --unsafe-perm \
&& pi --version \
&& echo "[builder] pi CLI installed"
# Install browser-use CLI for browser automation
RUN npm install -g browser-use --unsafe-perm \
&& browser-use --version \
&& echo "[builder] browser-use CLI installed"
# Set up pi home directory and skills
RUN mkdir -p ~/.pi/agent/skills ~/.pi/agent/agents \
&& echo "[builder] pi directories created"
# Stage 2: Download pi skills (caveman & andrej karpathy)
# Clone caveman skill from pi-agent repo
RUN cd /tmp && git clone https://github.com/earendil-works/pi-agent.git pi-repo \
&& mkdir -p ~/.pi/agent/skills/caveman \
&& cp -r pi-repo/examples/skills/caveman/* ~/.pi/agent/skills/caveman/ 2>/dev/null || true \
&& echo "[builder] caveman skill installed"
# Create andrej karpathy skill manually (reference/training patterns)
RUN mkdir -p ~/.pi/agent/skills/andrej-karpathy && cat > ~/.pi/agent/skills/andrej-karpathy/SKILL.md << 'EOF'
# Andrej Karpathy LLM & AI Principles Skill
Build neural networks and LLM systems with proven patterns from Andrej Karpathy.
Topics: attention mechanisms, transformer training, inference optimization, edge cases.
## Key Principles
### 1. Simplicity First
- Start with minimal implementation
- Add complexity only when justified
- Test each component independently
- Use debugging tools effectively
### 2. Neural Network Architecture
- Understand backward pass deeply
- Implement from scratch when possible
- Use visualization for debugging
- Profile before optimizing
### 3. LLM Training Patterns
- Quality data > quantity
- Curriculum learning for complex tasks
- Loss landscape visualization
- Checkpoint strategy matters
### 4. Inference Optimization
- Quantization without quality loss
- KV cache management
- Batch processing strategies
- Latency profiling
### 5. Failure Analysis
- Log intermediate activations
- Check gradient flow
- Validate data pipeline
- Test edge cases explicitly
## Usage in Poimen
Apply when:
- Designing workflow stages (like training curricula)
- Optimizing inference (planner/judge/implementer prompts)
- Debugging convergence issues (retry patterns)
- Scaling to production (quantization patterns)
## Resources
- github.com/karpathy/minGPT - Minimal GPT implementation
- youtube: "Neural Networks: Zero to Hero" series
- Papers: Attention Is All You Need, GPT series whitepapers
EOF
&& echo "[builder] andrej-karpathy skill created"
# Create browser-use skill for web testing & automation
RUN mkdir -p ~/.pi/agent/skills/browser-use && cat > ~/.pi/agent/skills/browser-use/SKILL.md << 'EOF'
# browser-use: Browser Automation Skill
Automate web browser interactions for testing, verification, and UI validation.
Topics: headless browser control, visual testing, form automation, screenshot capture.
## Key Capabilities
### 1. Browser Control
- Launch headless Chrome/Firefox
- Navigate to URLs
- Wait for elements/navigation
- Handle popups/dialogs
### 2. Interaction Patterns
- Click buttons/links
- Fill forms (text, dropdown, checkbox)
- Drag & drop
- Keyboard input
### 3. Verification & Capture
- Screenshot capture
- Element inspection
- Accessibility checks
- Network monitoring
### 4. Wait Strategies
- Wait for element visible
- Wait for navigation
- Wait for condition (custom JS)
- Timeout handling
### 5. Error Recovery
- Retry failed actions
- Handle stale elements
- Browser crash recovery
- Memory leak prevention
## Usage in Poimen Phases
### Phase T2 (Implementation)
- Test generated UI code in real browser
- Verify visual layout matches spec
- Validate form inputs work correctly
### Phase T3 (Verification)
- Visual regression testing
- Accessibility validation (ARIA, keyboard nav)
- Cross-browser verification
### Phase T6 (Integration)
- End-to-end workflow testing
- External service integration testing
- User journey verification
### Phase T9 (Release)
- Pre-release smoke tests
- Deployment verification
- Production canary testing
## Example Workflows
```bash
# Launch browser and take screenshot
browser-use screenshot "https://example.com" --file output.png
# Fill form and submit
browser-use interact "https://example.com" \
--click "#submit-btn" \
--type "#email" "[email protected]" \
--type "#password" "secretpass" \
--click ".submit"
# Wait for dynamic content and extract data
browser-use extract "https://example.com" \
--wait ".dynamic-content" \
--selector ".data-row" \
--output json
# Accessibility audit
browser-use audit "https://example.com" \
--check wcag2a \
--report a11y-report.html
```
## Integration with Poimen
Pre-generated code can be tested:
```bash
# Generate code (T2)
implementer_output = "function handleClick() { ... }"
# Verify in browser (T3)
browser-use interact "http://localhost:3000" \
--click ".test-button" \
--screenshot result.png
# Compare with expected
verify_visual_match(result.png, expected.png)
```
## Performance Notes
- Startup: ~2-5s per browser
- Action latency: 100-500ms per interaction
- Screenshot: 500ms-2s (depends on page size)
- Keep browser alive for batch operations (pool management)
## Error Handling
- Transient: Network timeout → retry with backoff
- Permanent: Element not found → fail and log
- Flaky: Wait strategies → increase timeout gradually
- Memory: Reuse browser instances → kill after 10 uses
## Resources
- docs.browseruse.com - Official documentation
- github.com/browser-use/browser-use - Source code
- Chrome DevTools Protocol - Advanced browser control
EOF
&& echo "[builder] browser-use skill created"
# Copy Go source code
COPY . /build/
# Download Go dependencies
RUN go mod download \
&& echo "[builder] Go dependencies downloaded"
# Build worker binary
RUN CGO_ENABLED=1 GOOS=linux go build -o /build/worker ./cmd/worker \
&& echo "[builder] Worker binary built"
# Verify binary
RUN file /build/worker && ls -lh /build/worker
# Stage 3: Runtime - Minimal base image with runtime dependencies
FROM alpine:3.20
LABEL maintainer="Poimen Team"
LABEL description="Poimen Temporal Worker with memory service, ast-grep, and browser automation"
FROM golang:1.21-alpine as builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
# Install runtime dependencies (including Chromium for browser-use)
RUN apk add --no-cache \
ca-certificates \
git \
bash \
curl \
jq \
chromium \
chromium-chromedriver \
&& echo "[runtime] Runtime dependencies installed"
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o workflows ./cmd/poimen/main.go
# Install Node.js for pi CLI and browser-use
RUN apk add --no-cache nodejs npm \
&& echo "[runtime] Node.js installed"
FROM alpine:latest
# Install pi CLI in runtime image
RUN npm install -g @earendil-works/pi-coding-agent --unsafe-perm \
&& pi --version \
&& echo "[runtime] pi CLI installed"
RUN apk --no-cache add ca-certificates
# Install browser-use CLI in runtime image
RUN npm install -g browser-use --unsafe-perm \
&& browser-use --version \
&& echo "[runtime] browser-use CLI installed"
WORKDIR /app
COPY --from=builder /app/workflows .
# Copy ast-grep binary from builder
COPY --from=builder /usr/local/bin/sg /usr/local/bin/sg
RUN chmod +x /usr/local/bin/sg && sg --version \
&& echo "[runtime] ast-grep copied"
EXPOSE 8080
# Copy pi skills from builder
COPY --from=builder /root/.pi /root/.pi
RUN ls -la /root/.pi/agent/skills/ \
&& echo "[runtime] pi skills configured"
HEALTHCHECK --interval=10s --timeout=5s --start-period=10s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:8080/health || exit 1
# Copy worker binary from builder
COPY --from=builder /build/worker /app/worker
RUN chmod +x /app/worker && file /app/worker \
&& echo "[runtime] Worker binary copied"
# Create app directory structure
RUN mkdir -p /app/work /app/logs /app/screenshots \
&& chmod 755 /app/work /app/logs /app/screenshots \
&& echo "[runtime] App directories created"
# Health check endpoint
EXPOSE 8081
# Worker task queue listener
ENV TEMPORAL_NAMESPACE=poimen-harness \
TEMPORAL_HOSTPORT=temporal-frontend.temporal:7233 \
MEMORY_SERVICE_URL=http://memory-service.poimen:5000 \
MEMORY_SERVICE_TOKEN= \
ANTHROPIC_API_KEY= \
PI_SKILLS_PATH=/root/.pi/agent/skills \
AST_GREP_BIN=/usr/local/bin/sg \
BROWSER_USE_BIN=/usr/local/bin/browser-use \
CHROMIUM_BIN=/usr/bin/chromium-browser \
SCREENSHOTS_DIR=/app/screenshots
# Entrypoint script with startup diagnostics
COPY --chmod=755 << 'EOF' /app/entrypoint.sh
#!/bin/bash
set -e
echo "[$(date)] ========== POIMEN WORKER STARTUP =========="
echo "[$(date)] Container: $HOSTNAME"
echo "[$(date)] Image: $(cat /etc/os-release | grep PRETTY_NAME | cut -d= -f2)"
# Verify CLI tools
echo "[$(date)] ✓ Checking CLI tools..."
echo " - Go version: $(go version 2>/dev/null || echo 'N/A')"
echo " - ast-grep: $(sg --version 2>&1 | head -1)"
echo " - pi: $(pi --version 2>&1 | head -1)"
echo " - browser-use: $(browser-use --version 2>&1 | head -1)"
echo " - chromium: $(chromium-browser --version 2>&1 || echo 'Not found')"
echo " - git: $(git --version)"
echo " - node: $(node --version)"
echo " - npm: $(npm --version)"
# Verify pi skills
echo "[$(date)] ✓ Checking pi skills..."
if [ -d "$PI_SKILLS_PATH" ]; then
echo " - Skills path: $PI_SKILLS_PATH"
ls -1 "$PI_SKILLS_PATH" | sed 's/^/ ✓ /'
else
echo " - WARNING: Skills path not found: $PI_SKILLS_PATH"
fi
# Verify browser tools
echo "[$(date)] ✓ Checking browser automation tools..."
echo " - Chromium binary: $CHROMIUM_BIN"
echo " - Screenshots directory: $SCREENSHOTS_DIR"
if [ -d "$SCREENSHOTS_DIR" ]; then
echo " - Screenshots dir ready ($(du -sh $SCREENSHOTS_DIR 2>/dev/null | cut -f1 || echo '0B'))"
fi
# Check environment variables
echo "[$(date)] ✓ Configuration loaded:"
echo " - TEMPORAL_NAMESPACE: $TEMPORAL_NAMESPACE"
echo " - TEMPORAL_HOSTPORT: $TEMPORAL_HOSTPORT"
echo " - MEMORY_SERVICE_URL: ${MEMORY_SERVICE_URL:-(not set)}"
echo " - PI_SKILLS_PATH: $PI_SKILLS_PATH"
echo " - CHROMIUM_BIN: $CHROMIUM_BIN"
# Verify memory service connectivity (optional, non-blocking)
if [ ! -z "$MEMORY_SERVICE_URL" ]; then
echo "[$(date)] ✓ Testing memory service connectivity..."
if curl -sf "$MEMORY_SERVICE_URL/health" > /dev/null 2>&1; then
echo " - Memory service: HEALTHY"
else
echo " - Memory service: UNREACHABLE (will retry in worker)"
fi
fi
# Test browser automation (optional, non-blocking)
echo "[$(date)] ✓ Testing browser automation..."
if command -v chromium-browser &> /dev/null && command -v browser-use &> /dev/null; then
echo " - Chromium available: YES"
echo " - browser-use available: YES"
echo " - Browser automation: READY"
else
echo " - Browser automation: WARNING - missing dependencies"
fi
echo "[$(date)] ========== STARTING WORKER =========="
exec /app/worker
EOF
RUN chmod +x /app/entrypoint.sh
# Run worker with diagnostics
ENTRYPOINT ["/app/entrypoint.sh"]
# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD curl -f http://localhost:8081/health || exit 1
ENTRYPOINT ["./workflows"]
+149 -86
View File
@@ -120,6 +120,7 @@ Poimen embodies three core principles:
| Activity | Purpose | Timeout | Retry |
|----------|---------|---------|-------|
| **AssumeRoleActivity** | Request temporary JWT token (like AWS AssumeRole) | 30s | 2x |
| **CloneRepo** | Clone git repository | 30s | 3x |
| **AnalyzeCode** | Static analysis (SAST) | 120s | 2x |
| **SecurityScan** | Dependency & vulnerability scan | 60s | 2x |
@@ -546,10 +547,8 @@ Poimen supports multiple authentication methods to federate LLM access across cu
#### 1. Bearer Token (JWT/OAuth2)
```go
auth := &routing.LLMAuth{
Type: routing.AuthTypeBearer,
Token: "eyJhbGciOiJIUzI1NiIs...", // JWT token from your auth provider
TenantID: "customer-123", // Optional: tenant isolation
Scopes: "llm:read llm:write", // Optional: OAuth2 scopes
Type: routing.AuthTypeBearer,
Token: "eyJhbGciOiJIUzI1NiIs...", // JWT token from your auth provider
}
client := routing.NewLLMClientWithAuth(auth)
```
@@ -557,9 +556,8 @@ client := routing.NewLLMClientWithAuth(auth)
#### 2. API Key
```go
auth := &routing.LLMAuth{
Type: routing.AuthTypeAPIKey,
APIKey: "sk-xxx-yyy-zzz", // API key from provider
TenantID: "customer-456",
Type: routing.AuthTypeAPIKey,
APIKey: "sk-xxx-yyy-zzz", // API key from provider
}
client := routing.NewLLMClientWithAuth(auth)
```
@@ -570,7 +568,6 @@ auth := &routing.LLMAuth{
Type: routing.AuthTypeCustom,
HeaderName: "X-Custom-Auth",
HeaderValue: "custom-token-value",
TenantID: "customer-789",
}
client := routing.NewLLMClientWithAuth(auth)
```
@@ -581,49 +578,41 @@ config := &routing.LLMRouterConfig{
Provider: openaiProvider,
KnowledgeBase: kb,
Auth: &routing.LLMAuth{
Type: routing.AuthTypeBearer,
Token: jwtToken,
TenantID: customerID, // Tenant isolation in multi-tenant deployments
Type: routing.AuthTypeBearer,
Token: jwtToken,
},
TenantID: customerID, // Additional tenant tracking
}
router := routing.NewLLMRouter(config)
```
#### Multi-Tenant Federated Access
#### Per-Deployment Auth
For multi-tenant deployments:
Each Poimen deployment gets its own LLM token:
```go
// Per-customer isolated routers
func CreateCustomerRouter(customerID, jwtToken string) (*routing.LLMRouter, error) {
// In K8s secret/vault
LLM_AUTH_TOKEN="eyJhbGciOiJIUzI1NiIs..."
// In code
func InitializeRouter() (*routing.LLMRouter, error) {
token := os.Getenv("LLM_AUTH_TOKEN")
if token == "" {
return nil, fmt.Errorf("LLM_AUTH_TOKEN not set")
}
auth := &routing.LLMAuth{
Type: routing.AuthTypeBearer,
Token: jwtToken,
TenantID: customerID, // Passed to LLM API in X-Tenant-ID header
Scopes: "llm:read", // Restrict scopes per customer
Type: routing.AuthTypeBearer,
Token: token,
}
config := &routing.LLMRouterConfig{
Provider: &routing.LLMClient{},
KnowledgeBase: globalKB,
KnowledgeBase: kb,
Auth: auth,
TenantID: customerID,
}
return routing.NewLLMRouter(config)
}
// Usage in activity
input := &routing.LLMRouterInput{
Message: "Analyze code",
MemoryContext: memCtx,
}
output, err := router.Route(ctx, input)
// Headers sent to LLM API:
// Authorization: Bearer <jwtToken>
// X-Tenant-ID: <customerID>
// X-OAuth-Scopes: llm:read
```
#### Token Refresh & Rotation
@@ -649,10 +638,8 @@ result, err := client.Chat(ctx, systemPrompt, userMsg)
| Header | Set When | Value | Purpose |
|--------|----------|-------|---------|
| `Authorization` | Bearer auth | `Bearer {token}` | OAuth2/JWT authentication |
| `Authorization` | Bearer auth | `Bearer {token}` | JWT/OAuth2 authentication |
| `X-API-Key` | API Key auth | `{api-key}` | API key authentication |
| `X-Tenant-ID` | Any auth type | `{tenantID}` | Tenant/customer isolation |
| `X-OAuth-Scopes` | Bearer with scopes | Space-separated scopes | OAuth2 scope enforcement |
| Custom header | Custom auth | `{headerValue}` | Custom authentication scheme |
### LLMRouter Configuration (Programmatic)
@@ -780,95 +767,171 @@ kubectl apply -f k8s/worker-deployment.yaml
---
## Security & Multi-Tenancy
## Security & Token Management
### Token Management
### Never Hardcode Tokens
**Never hardcode tokens!** Use secure secret management:
**Use secure secret management:**
```go
// ❌ DON'T DO THIS
auth := &routing.LLMAuth{
Type: routing.AuthTypeBearer,
Token: "eyJhbGciOiJIUzI1NiIs...", // Hardcoded!
}
// ✅ DO THIS
tokenFromVault, _ := vaultClient.GetSecret("llm-token-" + customerID)
token := os.Getenv("LLM_AUTH_TOKEN")
auth := &routing.LLMAuth{
Type: routing.AuthTypeBearer,
Token: tokenFromVault,
TenantID: customerID,
Type: routing.AuthTypeBearer,
Token: token,
}
```
**Recommended secret management:**
- Kubernetes Secrets (development)
- HashiCorp Vault (production)
- AWS Secrets Manager / GCP Secret Manager (cloud)
- Sealed Secrets / Sealed Policies
- **Kubernetes Secrets** (development) — stored in etcd
- **HashiCorp Vault** (production) — centralized secret management
- **AWS Secrets Manager** (cloud) — managed service
- **GCP Secret Manager** (cloud) — managed service
- **Sealed Secrets** or **Sealed Policies** — encrypted in git
### Tenant Isolation
### Token Rotation & Refresh
The `TenantID` header enforces tenant isolation on the LLM API side:
For long-running Temporal workflows, refresh tokens before they expire:
```go
// Customer A's workflow
authA := &routing.LLMAuth{
Type: routing.AuthTypeBearer,
Token: tokenA,
TenantID: "customer-A", // ← Isolates this customer
}
client := routing.NewLLMClientWithAuth(auth)
// Customer B's workflow
authB := &routing.LLMAuth{
Type: routing.AuthTypeBearer,
Token: tokenB,
TenantID: "customer-B", // ← Isolates this customer
// Later: token expires
newToken := os.Getenv("LLM_AUTH_TOKEN_REFRESHED")
newAuth := &routing.LLMAuth{
Type: routing.AuthTypeBearer,
Token: newToken,
}
client.UpdateAuth(newAuth)
```
The LLM API server should:
- Validate tenant ownership of tokens
- Enforce data boundaries per tenant
- Log access per tenant ID
- Rate-limit per tenant
### Authorization: LLM API Side
### Scope-Based Access Control
The LLM provider (api.riotpiao.com) should:
- Validate JWT signature & expiration
- Enforce API rate limits per token
- Log all requests with token identity
- Support token revocation / blacklisting
Use OAuth2 scopes to limit capabilities:
---
## AssumeRoleActivity: Temporary LLM Token Grants
**Like AWS AssumeRole**, AssumeRoleActivity requests temporary credentials for accessing LLM APIs:
```go
// Analytics-only customer
analytics := &routing.LLMAuth{
Type: routing.AuthTypeBearer,
Token: token,
Scopes: "llm:read", // Read-only
// 1. User requests temporary token
assumeRoleInput := &routing.AssumeRoleInput{
Identity: "[email protected]", // Who is accessing
Scope: "llm:read llm:write", // What permissions
DurationSeconds: 1800, // 30 minutes
}
// Full-access customer
admin := &routing.LLMAuth{
Type: routing.AuthTypeBearer,
Token: token,
Scopes: "llm:read llm:write llm:admin", // Full access
// 2. Activity exchanges with auth server → returns JWT
output, err := temporalClient.ExecuteActivity(ctx,
routing.AssumeRoleActivity,
assumeRoleInput)
// 3. Extract token from result
var tokenOutput *routing.AssumeRoleOutput
output.Get(&tokenOutput)
// 4. Use token in LLM Router
auth := &routing.LLMAuth{
Type: routing.AuthTypeBearer,
Token: tokenOutput.Token, // ← JWT valid for 30 minutes
}
router := routing.NewLLMRouter(config)
```
The LLM API server should validate scopes before executing requests.
### Workflow Pattern: AssumeRole → LLM Router → Activities
### Audit & Compliance
```go
// Step 1: Get temporary credentials
assumeRoleResult := workflow.ExecuteActivity(ctx, routing.AssumeRoleActivity, &routing.AssumeRoleInput{
Identity: workflowInput.UserID,
Scope: "llm:read llm:write",
DurationSeconds: 1800,
})
var token *routing.AssumeRoleOutput
assumeRoleResult.Get(&token)
All requests include identifying headers for audit trails:
// Step 2: Use token for all LLM router calls
routerInput := &routing.LLMRouterInput{
Message: "Analyze code for security",
Context: map[string]interface{}{"repo": "myrepo"},
}
routerOutput := workflow.ExecuteActivity(ctx, routing.LLMRouterActivity, routerInput)
// LLMRouter automatically uses the token from LLMRouterConfig
// Step 3: Execute generated workflow with same token
// (token baked into all activity calls)
```
POST /v1/chat/completions
Authorization: Bearer eyJhbGc...
X-Tenant-ID: customer-A
X-OAuth-Scopes: llm:read
# Server logs:
# timestamp=2024-01-15T10:30:00Z tenant=customer-A scope=llm:read status=200 tokens=1500
### Configuration: Credentials from Vault
Never hardcode credentials. Use Kubernetes Secrets or Hashicorp Vault:
```bash
# In K8s secret
kubectl create secret generic llm-oauth-creds \
--from-literal=OAUTH_CLIENT_ID="client-xxx" \
--from-literal=OAUTH_CLIENT_SECRET="secret-yyy" \
--from-literal=AUTH_SERVER_URL="https://auth.company.com"
# Pod reads from secret
env:
- name: OAUTH_CLIENT_ID
valueFrom:
secretKeyRef:
name: llm-oauth-creds
key: OAUTH_CLIENT_ID
```
In code, AssumeRoleActivity reads from environment:
```go
input := &routing.AssumeRoleInput{
Identity: "[email protected]",
Scope: "llm:read",
// clientId, clientSecret, authServerUrl read from env automatically
}
result, _ := AssumeRoleActivity(ctx, input)
```
### Token Lifecycle
| Stage | Duration | Action |
|-------|----------|--------|
| **Request** | T+0s | User calls AssumeRoleActivity with identity + scope |
| **Grant** | T+1s | Auth server validates, issues JWT (default: 1hr validity) |
| **Use** | T+1s to T+3600s | LLMRouter uses token for all api.riotpiao.com calls |
| **Refresh** | Before expiry | If workflow > 1hr, request new token via AssumeRole again |
| **Revoke** | On demand | Auth server can immediately revoke token if needed |
### Scopes & Access Control
Scopes define granular permissions:
```go
// Read-only access (safe for analytics)
asScope: "llm:read"
// Full access (for agent workflows)
scope: "llm:read llm:write"
// Admin access (for operator/setup)
scope: "llm:admin"
```
The LLM API validates scopes on every request. AssumeRoleActivity can't escalate privileges—scopes returned by auth server are trusted.
---
## Documentation
+216
View File
@@ -0,0 +1,216 @@
package action
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"time"
)
// AssumeRoleInput is the input to AssumeRoleActivity
type AssumeRoleInput struct {
// Identity is the user/service identity requesting access
Identity string `json:"identity"`
// ClientID is the OAuth2/OIDC client ID (from vault or env)
ClientID string `json:"clientId,omitempty"`
// ClientSecret is the OAuth2/OIDC client secret (from vault or env)
ClientSecret string `json:"clientSecret,omitempty"`
// Scope defines what APIs this token can access (e.g., "llm:read llm:write")
Scope string `json:"scope"`
// DurationSeconds is how long the token is valid (default: 3600 = 1 hour)
DurationSeconds int `json:"durationSeconds,omitempty"`
// AuthServerURL is the auth server endpoint (from env if not provided)
AuthServerURL string `json:"authServerUrl,omitempty"`
}
// AssumeRoleOutput is the output from AssumeRoleActivity
type AssumeRoleOutput struct {
// Token is the JWT token for calling api.riotpiao.com
Token string `json:"token"`
// ExpiresAt is when the token expires (Unix timestamp)
ExpiresAt int64 `json:"expiresAt"`
// ExpiresIn is the duration in seconds until expiration
ExpiresIn int `json:"expiresIn"`
// TokenType is typically "Bearer"
TokenType string `json:"tokenType"`
}
// oauthTokenRequest is sent to the auth server
type oauthTokenRequest struct {
GrantType string `json:"grant_type"`
ClientID string `json:"client_id"`
ClientSecret string `json:"client_secret"`
Scope string `json:"scope"`
Subject string `json:"subject,omitempty"` // The identity being assumed
}
// oauthTokenResponse is returned from the auth server
type oauthTokenResponse struct {
AccessToken string `json:"access_token"`
TokenType string `json:"token_type"`
ExpiresIn int `json:"expires_in"`
Scope string `json:"scope"`
}
// AssumeRoleActivity requests a temporary JWT token for accessing LLM APIs
//
// This activity works like AWS AssumeRole:
// 1. User provides identity + scope of access needed
// 2. Activity exchanges credentials with auth server
// 3. Returns JWT token valid for a limited time
// 4. Caller uses token in subsequent LLM API calls
//
// Security: Credentials should come from vault/secrets, never hardcoded
func AssumeRoleActivity(ctx context.Context, input *AssumeRoleInput) (*AssumeRoleOutput, error) {
// Validate inputs
if err := validateAssumeRoleInput(input); err != nil {
return nil, err
}
// Resolve configuration from input + environment
config, err := resolveAssumeRoleConfig(input)
if err != nil {
return nil, err
}
// Request token from auth server
tokenResp, err := requestAuthToken(ctx, config, input)
if err != nil {
return nil, err
}
// Build output
return buildAssumeRoleOutput(tokenResp), nil
}
// validateAssumeRoleInput checks required fields
func validateAssumeRoleInput(input *AssumeRoleInput) error {
if input.Identity == "" {
return fmt.Errorf("identity is required")
}
if input.Scope == "" {
return fmt.Errorf("scope is required (e.g., 'llm:read' or 'llm:read llm:write')")
}
return nil
}
// assumeRoleConfig holds resolved configuration
type assumeRoleConfig struct {
authServerURL string
clientID string
clientSecret string
duration int
}
// resolveAssumeRoleConfig gets config from input or environment
func resolveAssumeRoleConfig(input *AssumeRoleInput) (*assumeRoleConfig, error) {
cfg := &assumeRoleConfig{}
// Helper function to avoid DRY violation
getOrEnv := func(val, envKey, fieldName string) (string, error) {
if val != "" {
return val, nil
}
if val = os.Getenv(envKey); val != "" {
return val, nil
}
return "", fmt.Errorf("%s not provided and %s not set", fieldName, envKey)
}
var err error
if cfg.authServerURL, err = getOrEnv(input.AuthServerURL, "AUTH_SERVER_URL", "authServerUrl"); err != nil {
return nil, err
}
if cfg.clientID, err = getOrEnv(input.ClientID, "OAUTH_CLIENT_ID", "clientId"); err != nil {
return nil, err
}
if cfg.clientSecret, err = getOrEnv(input.ClientSecret, "OAUTH_CLIENT_SECRET", "clientSecret"); err != nil {
return nil, err
}
// Validate and set duration
cfg.duration = input.DurationSeconds
if cfg.duration == 0 {
cfg.duration = 3600 // 1 hour default
}
if cfg.duration > 86400 {
cfg.duration = 86400 // Max 24 hours
}
return cfg, nil
}
// requestAuthToken calls the auth server and returns the token response
func requestAuthToken(ctx context.Context, config *assumeRoleConfig, input *AssumeRoleInput) (*oauthTokenResponse, error) {
tokenReq := oauthTokenRequest{
GrantType: "client_credentials",
ClientID: config.clientID,
ClientSecret: config.clientSecret,
Scope: input.Scope,
Subject: input.Identity,
}
reqBody, err := json.Marshal(tokenReq)
if err != nil {
return nil, fmt.Errorf("failed to marshal token request: %w", err)
}
httpReq, err := http.NewRequestWithContext(ctx, "POST",
fmt.Sprintf("%s/oauth/token", config.authServerURL),
bytes.NewReader(reqBody))
if err != nil {
return nil, fmt.Errorf("failed to create request: %w", err)
}
httpReq.Header.Set("Content-Type", "application/json")
client := &http.Client{Timeout: 10 * time.Second}
resp, err := client.Do(httpReq)
if err != nil {
return nil, fmt.Errorf("failed to call auth server: %w", err)
}
defer resp.Body.Close()
respBody, err := io.ReadAll(resp.Body)
if err != nil {
return nil, fmt.Errorf("failed to read response: %w", err)
}
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("auth server returned status %d: %s", resp.StatusCode, string(respBody))
}
var tokenResp oauthTokenResponse
if err := json.Unmarshal(respBody, &tokenResp); err != nil {
return nil, fmt.Errorf("failed to unmarshal token response: %w", err)
}
if tokenResp.AccessToken == "" {
return nil, fmt.Errorf("auth server returned empty access token")
}
return &tokenResp, nil
}
// buildAssumeRoleOutput constructs the output from token response
func buildAssumeRoleOutput(tokenResp *oauthTokenResponse) *AssumeRoleOutput {
expiresAt := time.Now().Add(time.Duration(tokenResp.ExpiresIn) * time.Second).Unix()
return &AssumeRoleOutput{
Token: tokenResp.AccessToken,
ExpiresAt: expiresAt,
ExpiresIn: tokenResp.ExpiresIn,
TokenType: tokenResp.TokenType,
}
}
+354
View File
@@ -0,0 +1,354 @@
package action
import (
"encoding/json"
"fmt"
"regexp"
"strings"
"github.com/rockliang/poimen/workflows/pkg/db"
)
// IncompatibilityWarning explains why two activities can't be connected
type IncompatibilityWarning struct {
Source string `json:"source"` // Source node ID
Target string `json:"target"` // Target node ID
Reason string `json:"reason"` // Why they can't connect
SourceNeeds string `json:"source_needs"` // What source would need to output
TargetNeeds string `json:"target_needs"` // What target requires as input
Suggestion string `json:"suggestion"` // Suggestion to make it work
}
// ActivitySchema describes what an activity needs/provides
type ActivitySchema struct {
ActivityType string `json:"activity_type"`
Inputs map[string]InputField `json:"inputs"`
Outputs map[string]OutputField `json:"outputs"`
}
type InputField struct {
Type string `json:"type"`
Description string `json:"description"`
Required bool `json:"required"`
Enum []string `json:"enum,omitempty"`
}
type OutputField struct {
Type string `json:"type"`
Description string `json:"description"`
}
// getActivitySchema returns schema from knowledge base
func getActivitySchema(activityType string) (*ActivitySchema, error) {
kb := knowledgeBaseData()
if kb == nil {
return nil, fmt.Errorf("knowledge base not loaded")
}
var activities []map[string]interface{}
if err := json.Unmarshal([]byte(kb), &activities); err != nil {
// Try to extract activities from full KB structure
var fullKB map[string]interface{}
if err := json.Unmarshal([]byte(kb), &fullKB); err != nil {
return nil, fmt.Errorf("failed to parse knowledge base")
}
if activitiesRaw, ok := fullKB["activities"]; ok {
if b, err := json.Marshal(activitiesRaw); err == nil {
if err := json.Unmarshal(b, &activities); err != nil {
return nil, fmt.Errorf("failed to extract activities from KB")
}
}
}
}
// Find matching activity
for _, act := range activities {
if name, ok := act["name"].(string); ok {
if toActivityName(activityType) == name {
// Convert to ActivitySchema
schema := &ActivitySchema{
ActivityType: activityType,
Inputs: make(map[string]InputField),
Outputs: make(map[string]OutputField),
}
if inputs, ok := act["inputs"].(map[string]interface{}); ok {
for key, val := range inputs {
if field, ok := val.(map[string]interface{}); ok {
schema.Inputs[key] = parseInputField(field)
}
}
}
if outputs, ok := act["outputs"].(map[string]interface{}); ok {
for key, val := range outputs {
if field, ok := val.(map[string]interface{}); ok {
schema.Outputs[key] = parseOutputField(field)
}
}
}
return schema, nil
}
}
}
return nil, fmt.Errorf("activity %s not found in knowledge base", activityType)
}
func parseInputField(data map[string]interface{}) InputField {
field := InputField{}
if t, ok := data["type"].(string); ok {
field.Type = t
}
if d, ok := data["description"].(string); ok {
field.Description = d
}
if r, ok := data["required"].(bool); ok {
field.Required = r
}
return field
}
func parseOutputField(data map[string]interface{}) OutputField {
field := OutputField{}
if t, ok := data["type"].(string); ok {
field.Type = t
}
if d, ok := data["description"].(string); ok {
field.Description = d
}
return field
}
// CheckConnectionCompatibility validates if source can connect to target
func CheckConnectionCompatibility(sourceNode, targetNode db.WorkflowNode) []IncompatibilityWarning {
warnings := []IncompatibilityWarning{}
sourceSchema, err := getActivitySchema(sourceNode.Type)
if err != nil {
warnings = append(warnings, IncompatibilityWarning{
Source: sourceNode.ID,
Target: targetNode.ID,
Reason: fmt.Sprintf("Source activity schema not found: %v", err),
Suggestion: "Ensure source activity type is registered in knowledge base",
})
return warnings
}
targetSchema, err := getActivitySchema(targetNode.Type)
if err != nil {
warnings = append(warnings, IncompatibilityWarning{
Source: sourceNode.ID,
Target: targetNode.ID,
Reason: fmt.Sprintf("Target activity schema not found: %v", err),
Suggestion: "Ensure target activity type is registered in knowledge base",
})
return warnings
}
// Check if source produces outputs that target can consume
if len(sourceSchema.Outputs) == 0 {
warnings = append(warnings, IncompatibilityWarning{
Source: sourceNode.ID,
Target: targetNode.ID,
Reason: fmt.Sprintf("%s produces no outputs", sourceNode.Type),
SourceNeeds: "any output",
Suggestion: "Source activity must produce outputs",
})
return warnings
}
if len(targetSchema.Inputs) == 0 {
warnings = append(warnings, IncompatibilityWarning{
Source: sourceNode.ID,
Target: targetNode.ID,
Reason: fmt.Sprintf("%s accepts no inputs", targetNode.Type),
TargetNeeds: "no input",
Suggestion: "Target activity must accept inputs. Check if it's a terminal activity.",
})
return warnings
}
// Match outputs to inputs
sourceOutputs := getOutputNames(sourceSchema.Outputs)
targetInputs := getInputNames(targetSchema.Inputs)
if len(sourceOutputs) == 0 || len(targetInputs) == 0 {
warnings = append(warnings, IncompatibilityWarning{
Source: sourceNode.ID,
Target: targetNode.ID,
Reason: "No compatible output/input fields found",
SourceNeeds: strings.Join(sourceOutputs, ", "),
TargetNeeds: strings.Join(targetInputs, ", "),
Suggestion: "Use LLM transformation to map outputs to inputs",
})
}
return warnings
}
// CheckCanvasConnectivity analyzes all suggested edges for compatibility
func CheckCanvasConnectivity(nodes []db.WorkflowNode, suggestedEdges []db.WorkflowEdge) []IncompatibilityWarning {
warnings := []IncompatibilityWarning{}
nodeMap := make(map[string]db.WorkflowNode)
for _, n := range nodes {
nodeMap[n.ID] = n
}
for _, edge := range suggestedEdges {
sourceNode, ok := nodeMap[edge.Source]
if !ok {
continue
}
targetNode, ok := nodeMap[edge.Target]
if !ok {
continue
}
edgeWarnings := CheckConnectionCompatibility(sourceNode, targetNode)
warnings = append(warnings, edgeWarnings...)
}
return warnings
}
// IdentifyDisconnectedNodes finds nodes that can't connect to anything
func IdentifyDisconnectedNodes(nodes []db.WorkflowNode, suggestedEdges []db.WorkflowEdge) []string {
edgeMap := make(map[string]bool)
for _, edge := range suggestedEdges {
edgeMap[edge.Source] = true
edgeMap[edge.Target] = true
}
var disconnected []string
for _, node := range nodes {
if !edgeMap[node.ID] {
disconnected = append(disconnected, node.ID)
}
}
return disconnected
}
// toActivityName converts canvas type to activity name (e.g., "clone-repo" -> "CloneRepoActivity")
func toActivityName(canvasType string) string {
parts := strings.Split(canvasType, "-")
var result string
for _, part := range parts {
if part != "" {
result += strings.ToUpper(part[:1]) + strings.ToLower(part[1:])
}
}
return result + "Activity"
}
// getOutputNames extracts output field names
func getOutputNames(outputs map[string]OutputField) []string {
var names []string
for name := range outputs {
names = append(names, name)
}
return names
}
// getInputNames extracts input field names (required ones highlighted)
func getInputNames(inputs map[string]InputField) []string {
var names []string
for name, field := range inputs {
if field.Required {
names = append(names, name+"*")
} else {
names = append(names, name)
}
}
return names
}
// SuggestDataTransformation proposes how to connect incompatible activities
func SuggestDataTransformation(sourceNode, targetNode db.WorkflowNode) string {
sourceSchema, _ := getActivitySchema(sourceNode.Type)
targetSchema, _ := getActivitySchema(targetNode.Type)
if sourceSchema == nil || targetSchema == nil {
return "Cannot analyze compatibility without schemas"
}
sourceOuts := getOutputNames(sourceSchema.Outputs)
targetIns := getInputNames(targetSchema.Inputs)
return fmt.Sprintf(
"To connect %s → %s:\n"+
" %s outputs: %s\n"+
" %s needs: %s\n"+
" Solution: Use LLM transformation node to map outputs to inputs",
sourceNode.Label, targetNode.Label,
sourceNode.Type, strings.Join(sourceOuts, ", "),
targetNode.Type, strings.Join(targetIns, ", "),
)
}
// knowledgeBaseData returns raw KB JSON (stub - implement with actual KB loading)
func knowledgeBaseData() string {
// This would load from activity_knowledge_base.json
// For now, return empty - real implementation loads from file
return ""
}
// CanvasCompatibilityInput for Temporal activity
type CanvasCompatibilityInput struct {
Nodes []db.WorkflowNode `json:"nodes"`
Edges []db.WorkflowEdge `json:"edges"`
}
// CanvasCompatibilityOutput returns validation results
type CanvasCompatibilityOutput struct {
IsValid bool `json:"is_valid"`
Incompatibilities []IncompatibilityWarning `json:"incompatibilities"`
DisconnectedNodes []string `json:"disconnected_nodes"`
Warnings []string `json:"warnings"`
}
// CanvasCompatibilityActivity validates workflow canvas for type mismatches and isolation
func CanvasCompatibilityActivity(ctx interface{}, input CanvasCompatibilityInput) (CanvasCompatibilityOutput, error) {
output := CanvasCompatibilityOutput{
IsValid: true,
Incompatibilities: []IncompatibilityWarning{},
DisconnectedNodes: []string{},
Warnings: []string{},
}
// Check all edges for compatibility
for _, edge := range input.Edges {
var sourceNode, targetNode *db.WorkflowNode
for i := range input.Nodes {
if input.Nodes[i].ID == edge.Source {
sourceNode = &input.Nodes[i]
}
if input.Nodes[i].ID == edge.Target {
targetNode = &input.Nodes[i]
}
}
if sourceNode != nil && targetNode != nil {
if warning, err := ValidateConnection(sourceNode, targetNode); err != nil {
output.IsValid = false
output.Incompatibilities = append(output.Incompatibilities, warning)
}
}
}
// Find disconnected nodes
connected := make(map[string]bool)
for _, edge := range input.Edges {
connected[edge.Source] = true
connected[edge.Target] = true
}
for _, node := range input.Nodes {
if node.Type == "activity" && !connected[node.ID] {
output.DisconnectedNodes = append(output.DisconnectedNodes, node.ID)
}
}
return output, nil
}
+283
View File
@@ -0,0 +1,283 @@
package action
import (
"context"
"encoding/json"
"fmt"
"github.com/rockliang/poimen/workflows/action/llm"
"github.com/rockliang/poimen/workflows/pkg/db"
"github.com/rockliang/poimen/workflows/statemachine"
)
// CanvasReasonerInput infers connections between nodes using LLM reasoning
type CanvasReasonerInput struct {
Nodes []db.WorkflowNode `json:"nodes"` // Canvas nodes
Edges []db.WorkflowEdge `json:"edges"` // Existing edges
// If true, only suggest new edges; if false, redesign entire canvas
PreserveExisting bool `json:"preserve_existing,omitempty"`
AuthToken string `json:"auth_token,omitempty"` // JWT for LLM calls
}
// RelationWording describes semantic meaning of an edge
type RelationWording struct {
Verb string `json:"verb"` // outputs, inputs, depends-on, etc
SourceOutput string `json:"source_output"` // What source produces
TargetInput string `json:"target_input"` // What target requires
ConnectionType string `json:"connection_type"` // direct-map, requires-transformer, conditional
Confidence float64 `json:"confidence"` // 0.0-1.0
SemanticMatch string `json:"semantic_match"` // Human-readable explanation
TransformerNeeded string `json:"transformer_needed,omitempty"` // If transformation required
}
// EdgeWithWording pairs an edge with its semantic description
type EdgeWithWording struct {
Source string `json:"source"`
Target string `json:"target"`
RelationType string `json:"relation_type"` // data-flow, dependency, conditional, parallel
RelationLabel string `json:"relation_label"` // e.g., "CloneRepo outputs path → AnalyzeCode requires path"
RelationWording RelationWording `json:"relation_wording"`
}
// CanvasReasonerOutput returns suggested edges and reasoning
type CanvasReasonerOutput struct {
SuggestedEdges []EdgeWithWording `json:"suggested_edges"` // Edges with wording
RemovedEdges []db.WorkflowEdge `json:"removed_edges,omitempty"` // Edges to remove
Reasoning string `json:"reasoning"` // LLM explanation
Confidence float64 `json:"confidence"` // 0.0-1.0
IncompatibleEdges []IncompatibilityWarning `json:"incompatible_edges,omitempty"` // Can't connect
DisconnectedNodes []string `json:"disconnected_nodes,omitempty"` // No connections
UserAlerts []string `json:"user_alerts,omitempty"` // Human-readable warnings
}
// CanvasReasonerActivity uses LLM to infer connections between workflow activities
func CanvasReasonerActivity(ctx context.Context, in CanvasReasonerInput) (CanvasReasonerOutput, error) {
logger := newActivityLogger(ctx)
output := CanvasReasonerOutput{
SuggestedEdges: []db.WorkflowEdge{},
}
if len(in.Nodes) == 0 {
return output, fmt.Errorf("no nodes provided")
}
logger.logf("info", "Analyzing canvas with %d nodes, %d edges", len(in.Nodes), len(in.Edges))
// Build activity descriptions for LLM context
nodeDesc := buildNodeDescriptions(in.Nodes)
edgeDesc := buildEdgeDescriptions(in.Edges)
// Create prompt for LLM reasoning with relation wording
systemPrompt := `You are a workflow automation expert. Analyze activities and suggest logical connections with semantic descriptions.
CRITICAL RULES:
1. Only suggest edges where outputs→inputs match
2. Provide relation wording: verb, source_output, target_input
3. Assess connection confidence (0.0-1.0)
4. Flag type mismatches that need transformers
Respond with JSON:
{
"edges": [
{
"source": "node-1",
"target": "node-2",
"relation_type": "data-flow|dependency|conditional|parallel",
"relation_label": "Node1 outputs X → Node2 requires X",
"relation_wording": {
"verb": "outputs|depends-on|triggers|etc",
"source_output": "field_name (type): description",
"target_input": "field_name (type, required?): description",
"connection_type": "direct-map|requires-transformer|conditional",
"confidence": 0.95,
"semantic_match": "Explanation of why this makes sense"
}
}
],
"reasoning": "Overall workflow structure explanation",
"confidence": 0.85
}`
userPrompt := fmt.Sprintf(`Canvas Analysis:
Nodes (including inputs/outputs):
%s
Current Edges:
%s
Task: %s
KEY RULES:
- Preserve existing edges and suggest only NEW edges to add
- SKIP any connections where input/output types don't match
- If an activity has no outputs, it cannot be a source
- If an activity has no inputs, it cannot be a target
- Note any activities that are hard to connect (terminal activities, generators, etc)
Return ONLY valid JSON, no markdown code blocks.`, nodeDesc, edgeDesc, getReasoningTask(in.PreserveExisting))
logger.logf("info", "Calling LLM reasoning (preserve_existing=%v)", in.PreserveExisting)
// Call LLM
client, err := llm.NewClient()
if err != nil {
return output, fmt.Errorf("failed to create LLM client: %w", err)
}
response, err := client.CreateMessage(ctx, llm.MessageInput{
Model: statemachine.ModelSpec{
ModelID: "reasoning", // Use reasoning model for complex analysis
},
SystemPrompt: systemPrompt,
Messages: []llm.MessageParam{
{
Role: "user",
Content: userPrompt,
},
},
AuthToken: in.AuthToken,
})
if err != nil {
return output, fmt.Errorf("LLM reasoning failed: %w", err)
}
// Parse LLM response
var reasonerResp struct {
Edges []db.WorkflowEdge `json:"edges"`
Reasoning string `json:"reasoning"`
Confidence float64 `json:"confidence"`
}
if err := json.Unmarshal([]byte(response), &reasonerResp); err != nil {
logger.logf("warn", "Failed to parse LLM response as JSON: %v", err)
// Try to extract from response text
output.Reasoning = response
output.Confidence = 0.5
return output, fmt.Errorf("failed to parse LLM response: %w", err)
}
// Validate suggested edges
nodeMap := make(map[string]bool)
for _, n := range in.Nodes {
nodeMap[n.ID] = true
}
validEdges := []db.WorkflowEdge{}
for _, edge := range reasonerResp.Edges {
if !nodeMap[edge.Source] {
logger.logf("warn", "Suggested edge references unknown source: %s", edge.Source)
continue
}
if !nodeMap[edge.Target] {
logger.logf("warn", "Suggested edge references unknown target: %s", edge.Target)
continue
}
// Don't suggest self-loops
if edge.Source == edge.Target {
logger.logf("warn", "Skipping self-loop: %s", edge.Source)
continue
}
validEdges = append(validEdges, edge)
}
output.SuggestedEdges = validEdges
output.Reasoning = reasonerResp.Reasoning
output.Confidence = reasonerResp.Confidence
// Check compatibility of suggested edges
incompatibilities := CheckCanvasConnectivity(in.Nodes, validEdges)
if len(incompatibilities) > 0 {
output.IncompatibleEdges = incompatibilities
logger.logf("warn", "Found %d incompatible edge connections", len(incompatibilities))
// Generate user-friendly alerts
for i, incompat := range incompatibilities {
if i < 5 { // Limit to 5 alerts to avoid spam
alert := fmt.Sprintf(
"⚠️ %s → %s: %s. %s",
incompat.Source, incompat.Target, incompat.Reason, incompat.Suggestion,
)
output.UserAlerts = append(output.UserAlerts, alert)
}
}
}
// Identify disconnected nodes
disconnected := IdentifyDisconnectedNodes(in.Nodes, validEdges)
if len(disconnected) > 0 {
output.DisconnectedNodes = disconnected
logger.logf("warn", "Found %d disconnected nodes", len(disconnected))
for _, nodeID := range disconnected {
var label string
for _, node := range in.Nodes {
if node.ID == nodeID {
label = node.Label
break
}
}
alert := fmt.Sprintf(
"🔌 Node '%s' has no connections. Consider adding edges or removing it.",
label,
)
output.UserAlerts = append(output.UserAlerts, alert)
}
}
logger.logf("info", "LLM suggested %d edges with confidence %.2f | %d incompatibilities | %d disconnected",
len(validEdges), output.Confidence, len(incompatibilities), len(disconnected))
return output, nil
}
// buildNodeDescriptions creates readable node descriptions for LLM (including schemas)
func buildNodeDescriptions(nodes []db.WorkflowNode) string {
var desc string
for i, node := range nodes {
desc += fmt.Sprintf("%d. [%s] %s (type: %s)\n", i+1, node.ID, node.Label, node.Type)
// Add input/output schema info
if schema, err := getActivitySchema(node.Type); err == nil {
if len(schema.Inputs) > 0 {
desc += fmt.Sprintf(" INPUTS: %v\n", getInputNames(schema.Inputs))
} else {
desc += fmt.Sprintf(" INPUTS: none (generator/trigger)\n")
}
if len(schema.Outputs) > 0 {
desc += fmt.Sprintf(" OUTPUTS: %v\n", getOutputNames(schema.Outputs))
} else {
desc += fmt.Sprintf(" OUTPUTS: none (terminal/sink)\n")
}
}
if node.Data != nil {
if b, err := json.MarshalIndent(node.Data, " ", " "); err == nil {
desc += fmt.Sprintf(" CONFIG: %s\n", string(b))
}
}
}
return desc
}
// buildEdgeDescriptions creates readable edge descriptions for LLM
func buildEdgeDescriptions(edges []db.WorkflowEdge) string {
if len(edges) == 0 {
return "None"
}
var desc string
for i, edge := range edges {
desc += fmt.Sprintf("%d. %s → %s\n", i+1, edge.Source, edge.Target)
}
return desc
}
// getReasoningTask returns task description based on preservation mode
func getReasoningTask(preserveExisting bool) string {
if preserveExisting {
return "Keep all existing edges and suggest ONLY NEW edges to improve workflow"
}
return "Design optimal workflow by suggesting all connections and noting any redundant edges"
}
+96
View File
@@ -0,0 +1,96 @@
package action
import (
"context"
"encoding/json"
"fmt"
"github.com/rockliang/poimen/workflows/pkg/db"
)
// CanvasWithRelationsData combines canvas nodes/edges with relation wording
type CanvasWithRelationsData struct {
WorkflowID string `json:"workflow_id"`
Version int `json:"version"`
Nodes []db.WorkflowNode `json:"nodes"`
Edges []db.WorkflowEdge `json:"edges"`
Relations []EdgeWithWording `json:"relations"`
UpdatedAt string `json:"updated_at"`
}
// FetchCanvasRelationsInput parameters
type FetchCanvasRelationsInput struct {
WorkflowID string `json:"workflow_id"`
Version int `json:"version"`
}
// FetchCanvasRelationsActivity fetches canvas + relations from DB
func FetchCanvasRelationsActivity(ctx context.Context, input FetchCanvasRelationsInput) (CanvasWithRelationsData, error) {
logger := newActivityLogger(ctx)
output := CanvasWithRelationsData{
WorkflowID: input.WorkflowID,
Version: input.Version,
Nodes: []db.WorkflowNode{},
Edges: []db.WorkflowEdge{},
Relations: []EdgeWithWording{},
}
logger.logf("info", "Fetching canvas relations: %s v%d", input.WorkflowID, input.Version)
// Get database client from context or activity manager
dbClient, ok := ctx.Value("db_client").(*db.Client)
if !ok {
return output, fmt.Errorf("database client not in context")
}
// Fetch workflow
workflow, err := dbClient.GetWorkflow(ctx, input.WorkflowID)
if err != nil {
return output, fmt.Errorf("failed to get workflow: %w", err)
}
// Parse canvas nodes and edges
var nodes []db.WorkflowNode
if err := json.Unmarshal([]byte(workflow.Nodes), &nodes); err != nil {
return output, fmt.Errorf("failed to parse nodes: %w", err)
}
var edges []db.WorkflowEdge
if err := json.Unmarshal([]byte(workflow.Edges), &edges); err != nil {
return output, fmt.Errorf("failed to parse edges: %w", err)
}
output.Nodes = nodes
output.Edges = edges
output.UpdatedAt = workflow.UpdatedAt.String()
// Fetch workflow relations
relations, err := dbClient.GetWorkflowRelations(ctx, input.WorkflowID, input.Version)
if err != nil {
// Relations may not exist for old canvases - this is OK
logger.logf("warn", "Failed to fetch relations: %v", err)
return output, nil
}
// Map to EdgeWithWording
for _, rel := range relations {
edge := EdgeWithWording{
ID: rel.ID,
Source: rel.SourceNodeID,
Target: rel.TargetNodeID,
RelationType: rel.RelationType,
RelationLabel: rel.Label,
CreatedAt: rel.CreatedAt.String(),
}
// Parse relation wording JSON
if err := json.Unmarshal(rel.RelationWording, &edge.RelationWording); err != nil {
logger.logf("warn", "Failed to parse relation wording: %v", err)
}
output.Relations = append(output.Relations, edge)
}
logger.logf("info", "Fetched %d nodes, %d edges, %d relations", len(output.Nodes), len(output.Edges), len(output.Relations))
return output, nil
}
+78
View File
@@ -0,0 +1,78 @@
package action
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"time"
"github.com/rockliang/poimen/workflows/pkg/db"
)
// IndexGraphRAGInput sends workflow relations to GraphRAG for indexing
type IndexGraphRAGInput struct {
WorkflowID string `json:"workflow_id"`
Version int `json:"version"`
Nodes []db.WorkflowNode `json:"nodes"`
Relations []EdgeWithWording `json:"relations"`
}
// IndexGraphRAGOutput confirms indexing status
type IndexGraphRAGOutput struct {
WorkflowID string `json:"workflow_id"`
Version int `json:"version"`
IndexedEntities int `json:"indexed_entities"`
IndexedEdges int `json:"indexed_edges"`
Status string `json:"status"`
GraphRAGChecksum string `json:"graph_rag_checksum"`
IndexedAt string `json:"indexed_at"`
}
// IndexGraphRAGActivity indexes workflow canvas to GraphRAG (stub for now)
func IndexGraphRAGActivity(ctx context.Context, input IndexGraphRAGInput) (IndexGraphRAGOutput, error) {
output := IndexGraphRAGOutput{
WorkflowID: input.WorkflowID,
Version: input.Version,
Status: "indexed",
IndexedEntities: len(input.Nodes),
IndexedEdges: len(input.Relations),
IndexedAt: time.Now().UTC().Format(time.RFC3339),
}
// Stub implementation - actual GraphRAG indexing would happen here
// For now, just return success
return output, nil
}
// QueryGraphRAGRelationsInput for direct relation discovery
type QueryGraphRAGRelationsInput struct {
WorkflowID string `json:"workflow_id"`
Version int `json:"version"`
Query string `json:"query"`
TopK int `json:"top_k"`
Filters map[string]interface{} `json:"filters,omitempty"`
}
// QueryGraphRAGRelationsOutput returns discovered relations
type QueryGraphRAGRelationsOutput struct {
Query string `json:"query"`
Results []EdgeWithWording `json:"results"`
TotalCount int `json:"total_count"`
ExecutionMs int64 `json:"execution_time_ms"`
}
// QueryGraphRAGRelationsActivity queries GraphRAG for relation patterns (stub)
func QueryGraphRAGRelationsActivity(ctx context.Context, input QueryGraphRAGRelationsInput) (QueryGraphRAGRelationsOutput, error) {
output := QueryGraphRAGRelationsOutput{
Query: input.Query,
Results: []EdgeWithWording{},
TotalCount: 0,
}
// Stub implementation - actual GraphRAG querying would happen here
return output, nil
}
+6
View File
@@ -57,6 +57,7 @@ type MessageInput struct {
Model statemachine.ModelSpec
SystemPrompt string
Messages []MessageParam
AuthToken string // Optional JWT token for authenticated endpoints
}
// MessageParam represents a message parameter.
@@ -136,6 +137,11 @@ func (c *OpenAIClient) CreateMessage(ctx context.Context, in MessageInput) (stri
httpReq.Header.Set("Content-Type", "application/json")
// Add authentication header if token provided
if in.AuthToken != "" {
httpReq.Header.Set("Authorization", fmt.Sprintf("Bearer %s", in.AuthToken))
}
// Send request
resp, err := c.httpClient.Do(httpReq)
if err != nil {
+159
View File
@@ -0,0 +1,159 @@
package action
import (
"context"
"fmt"
"github.com/rockliang/poimen/workflows/action/llm"
"github.com/rockliang/poimen/workflows/statemachine"
)
// LLMInferenceInput is input for LLMInferenceActivity
type LLMInferenceInput struct {
Model string `json:"model"` // Model ID (reasoning, ornith:35b, etc)
SystemPrompt string `json:"system_prompt"` // System instruction
UserPrompt string `json:"user_prompt"` // User message
Temperature float64 `json:"temperature,omitempty"` // LLM temperature (0-1)
MaxTokens int `json:"max_tokens,omitempty"` // Max output tokens
AuthToken string `json:"auth_token,omitempty"` // JWT token for authenticated endpoints
}
// LLMInferenceOutput is output from LLMInferenceActivity
type LLMInferenceOutput struct {
Response string `json:"response"` // LLM response text
Model string `json:"model"` // Model used
StopReason string `json:"stop_reason"` // How inference stopped (stop_sequence, length, etc)
TokensUsed int `json:"tokens_used"` // Total tokens consumed
ErrorMessage string `json:"error,omitempty"`
}
// LLMInferenceActivity calls LLM API with given prompt and returns response
func LLMInferenceActivity(ctx context.Context, in LLMInferenceInput) (LLMInferenceOutput, error) {
logger := newActivityLogger(ctx)
output := LLMInferenceOutput{
Model: in.Model,
}
// Validate input
if in.Model == "" {
return output, fmt.Errorf("model not specified")
}
if in.UserPrompt == "" {
return output, fmt.Errorf("user_prompt not specified")
}
logger.logf("info", "Starting LLM inference with model: %s", in.Model)
// Create LLM client
client, err := llm.NewClient()
if err != nil {
output.ErrorMessage = err.Error()
return output, fmt.Errorf("failed to create LLM client: %w", err)
}
// Call LLM
logger.logf("info", "Calling LLM API (model=%s, prompt_len=%d, auth=%v)", in.Model, len(in.UserPrompt), in.AuthToken != "")
response, err := client.CreateMessage(ctx, llm.MessageInput{
Model: statemachine.ModelSpec{
ModelID: in.Model,
},
SystemPrompt: in.SystemPrompt,
Messages: []llm.MessageParam{
{
Role: "user",
Content: in.UserPrompt,
},
},
AuthToken: in.AuthToken,
})
if err != nil {
output.ErrorMessage = err.Error()
logger.logf("error", "LLM API call failed: %v", err)
return output, fmt.Errorf("LLM inference failed: %w", err)
}
output.Response = response
output.StopReason = "stop_sequence"
logger.logf("info", "LLM inference completed (response_len=%d)", len(response))
return output, nil
}
// LLMBatchInferenceInput is input for batch inference
type LLMBatchInferenceInput struct {
Model string `json:"model"`
SystemPrompt string `json:"system_prompt"`
Prompts []string `json:"prompts"` // List of user prompts
Temperature float64 `json:"temperature,omitempty"`
AuthToken string `json:"auth_token,omitempty"` // JWT token for authenticated endpoints
}
// LLMBatchInferenceOutput is output from batch inference
type LLMBatchInferenceOutput struct {
Responses []string `json:"responses"` // LLM responses (parallel to input Prompts)
Model string `json:"model"`
Errors []string `json:"errors,omitempty"`
}
// LLMBatchInferenceActivity calls LLM multiple times in sequence
func LLMBatchInferenceActivity(ctx context.Context, in LLMBatchInferenceInput) (LLMBatchInferenceOutput, error) {
logger := newActivityLogger(ctx)
output := LLMBatchInferenceOutput{
Model: in.Model,
Responses: []string{},
Errors: []string{},
}
if in.Model == "" {
return output, fmt.Errorf("model not specified")
}
if len(in.Prompts) == 0 {
return output, fmt.Errorf("no prompts provided")
}
logger.logf("info", "Starting batch LLM inference (model=%s, count=%d)", in.Model, len(in.Prompts))
// Create LLM client
client, err := llm.NewClient()
if err != nil {
return output, fmt.Errorf("failed to create LLM client: %w", err)
}
// Process each prompt
for i, prompt := range in.Prompts {
logger.logf("info", "Processing prompt %d/%d", i+1, len(in.Prompts))
response, err := client.CreateMessage(ctx, llm.MessageInput{
Model: statemachine.ModelSpec{
ModelID: in.Model,
},
SystemPrompt: in.SystemPrompt,
Messages: []llm.MessageParam{
{
Role: "user",
Content: prompt,
},
},
})
if err != nil {
output.Errors = append(output.Errors, fmt.Sprintf("prompt %d: %v", i, err))
output.Responses = append(output.Responses, "")
logger.logf("warn", "Failed to process prompt %d: %v", i, err)
} else {
output.Responses = append(output.Responses, response)
}
}
logger.logf("info", "Batch inference completed (responses=%d, errors=%d)",
len(output.Responses), len(output.Errors))
return output, nil
}
+133
View File
@@ -0,0 +1,133 @@
package action
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"time"
)
// GraphRAGQueryInput for Memory System endpoint
type GraphRAGQueryInput struct {
WorkflowID string `json:"workflow_id"`
Query string `json:"query"`
SearchType string `json:"search_type"`
RelationType string `json:"relation_type"`
ConfidenceFloor float64 `json:"confidence_floor"`
TopK int `json:"top_k"`
RankingProfile string `json:"ranking_profile"`
Canvas CanvasWithRelationsData `json:"canvas"`
}
// GraphRAGQueryOutput from Memory System
type GraphRAGQueryOutput struct {
WorkflowID string `json:"workflow_id"`
Query string `json:"query"`
Edges []EdgeWithWording `json:"edges"`
Paths []QueryPathData `json:"paths"`
TotalCount int `json:"total_count"`
HasMore bool `json:"has_more"`
ExecutionMs int64 `json:"execution_time_ms"`
}
type QueryPathData struct {
SourceID string `json:"source_id"`
TargetID string `json:"target_id"`
Distance int `json:"distance"`
PathCount int `json:"path_count"`
NodeIDs []string `json:"node_ids"`
Confidence float64 `json:"total_confidence"`
}
// QueryGraphRAGActivity queries Memory System for semantic relations
func QueryGraphRAGActivity(ctx context.Context, input GraphRAGQueryInput) (GraphRAGQueryOutput, error) {
logger := newActivityLogger(ctx)
output := GraphRAGQueryOutput{
WorkflowID: input.WorkflowID,
Query: input.Query,
Edges: []EdgeWithWording{},
Paths: []QueryPathData{},
}
logger.logf("info", "Querying GraphRAG: %s", input.Query)
// Get Memory Service URL from env
memoryURL := os.Getenv("MEMORY_SERVICE_URL")
if memoryURL == "" {
memoryURL = "http://localhost:8000"
}
// Build payload for Memory System
payload := map[string]interface{}{
"workflow_id": input.WorkflowID,
"query": input.Query,
"search_type": input.SearchType,
"relation_type": input.RelationType,
"confidence_floor": input.ConfidenceFloor,
"top_k": input.TopK,
"ranking_profile": input.RankingProfile,
"canvas_nodes": input.Canvas.Nodes,
"canvas_edges": input.Canvas.Edges,
"relations": input.Canvas.Relations,
}
reqBody, err := json.Marshal(payload)
if err != nil {
return output, fmt.Errorf("failed to marshal payload: %w", err)
}
// Call Memory System unified query endpoint
req, err := http.NewRequestWithContext(
ctx,
"POST",
memoryURL+"/workflows/query",
bytes.NewReader(reqBody),
)
if err != nil {
return output, fmt.Errorf("failed to create request: %w", err)
}
req.Header.Set("Content-Type", "application/json")
if token := ctx.Value("jwt_token"); token != nil {
req.Header.Set("Authorization", fmt.Sprintf("Bearer %v", token))
}
startTime := time.Now()
client := &http.Client{Timeout: 30 * time.Second}
resp, err := client.Do(req)
if err != nil {
return output, fmt.Errorf("failed to call Memory Service: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode != 200 {
body, _ := io.ReadAll(resp.Body)
return output, fmt.Errorf("Memory Service returned %d: %s", resp.StatusCode, string(body))
}
// Parse response
var graphResp struct {
Edges []EdgeWithWording `json:"edges"`
Paths []QueryPathData `json:"paths"`
TotalCount int `json:"total_count"`
HasMore bool `json:"has_more"`
}
if err := json.NewDecoder(resp.Body).Decode(&graphResp); err != nil {
return output, fmt.Errorf("failed to decode response: %w", err)
}
output.Edges = graphResp.Edges
output.Paths = graphResp.Paths
output.TotalCount = graphResp.TotalCount
output.HasMore = graphResp.HasMore
output.ExecutionMs = time.Since(startTime).Milliseconds()
logger.logf("info", "GraphRAG returned %d edges, %d paths in %dms",
len(output.Edges), len(output.Paths), output.ExecutionMs)
return output, nil
}
+125
View File
@@ -0,0 +1,125 @@
package main
import (
"context"
"flag"
"log"
"os"
"os/signal"
"sync"
"syscall"
"go.temporal.io/sdk/client"
"go.temporal.io/sdk/worker"
"github.com/rockliang/poimen/workflows/action"
"github.com/rockliang/poimen/workflows/internal/api"
"github.com/rockliang/poimen/workflows/internal/config"
"github.com/rockliang/poimen/workflows/pkg/db"
"github.com/rockliang/poimen/workflows/statemachine"
)
func main() {
var (
apiPort = flag.Int("port", 8080, "HTTP API port")
verbose = flag.Bool("verbose", false, "verbose logging")
)
flag.Parse()
logger := log.New(os.Stdout, "[poimen-server] ", log.LstdFlags|log.Lshortfile)
// Load configuration
cfg, err := config.LoadConfig()
if err != nil {
logger.Fatalf("failed to load config: %v", err)
}
// Connect to database (memory-db via K8s CNPG)
logger.Println("connecting to database...")
database, err := db.New(os.Getenv("DATABASE_URL"))
if err != nil {
logger.Fatalf("failed to connect to database: %v", err)
}
defer database.Close()
logger.Println("✓ Connected to database")
// Connect to Temporal
logger.Printf("connecting to Temporal at %s", cfg.Temporal.HostPort)
c, err := client.Dial(client.Options{
HostPort: cfg.Temporal.HostPort,
Namespace: cfg.Temporal.Namespace,
})
if err != nil {
logger.Fatalf("failed to connect to temporal: %v", err)
}
defer c.Close()
logger.Println("✓ Connected to Temporal")
// Create and start Temporal worker
w := worker.New(c, "default", worker.Options{})
// Register RoutingWorkflow
w.RegisterWorkflow(statemachine.RoutingWorkflow)
// Register activities
w.RegisterActivity(action.CloneRepoActivity)
w.RegisterActivity(action.AnalyzeCodeActivity)
w.RegisterActivity(action.SecurityScanActivity)
w.RegisterActivity(action.GenerateReportActivity)
w.RegisterActivity(action.DeploymentPreCheckActivity)
w.RegisterActivity(action.NotifyStatusActivity)
w.RegisterActivity(action.ApproveWorkflowActivity)
w.RegisterActivity(action.ArchiveResultsActivity)
w.RegisterActivity(action.RetrieveMemoryActivity)
w.RegisterActivity(action.AssumeRoleActivity)
w.RegisterActivity(action.LLMInferenceActivity)
w.RegisterActivity(action.LLMBatchInferenceActivity)
w.RegisterActivity(action.CanvasReasonerActivity)
var wg sync.WaitGroup
errChan := make(chan error, 2)
// Start Temporal worker
wg.Add(1)
go func() {
defer wg.Done()
logger.Println("starting Temporal worker...")
if err := w.Run(worker.InterruptCh()); err != nil {
errChan <- err
}
}()
// Start HTTP API server
wg.Add(1)
go func() {
defer wg.Done()
server := api.NewServer(database, c, logger)
logger.Printf("starting API server on port %d", *apiPort)
if err := server.Start(*apiPort); err != nil {
errChan <- err
}
}()
// Wait for interrupt signal
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
go func() {
sig := <-sigChan
logger.Printf("received signal: %v", sig)
w.Stop()
}()
// Monitor for errors
go func() {
err := <-errChan
if err != nil {
logger.Printf("error: %v", err)
w.Stop()
}
}()
wg.Wait()
logger.Println("✓ Server stopped gracefully")
}
+12 -1
View File
@@ -52,6 +52,7 @@ func main() {
w.RegisterWorkflow(statemachine.TaskUnitWorkflow)
w.RegisterWorkflow(statemachine.TestWorkflow)
w.RegisterWorkflow(statemachine.RoutingWorkflow)
w.RegisterWorkflow(statemachine.WorkflowGraphQuery)
// Register all activities
w.RegisterActivity(action.CloneRepoActivity)
@@ -85,9 +86,19 @@ func main() {
w.RegisterActivity(action.DeploymentPreCheckActivity)
w.RegisterActivity(action.ApproveWorkflowActivity)
// Authentication activities
w.RegisterActivity(action.AssumeRoleActivity)
// Memory activities
w.RegisterActivity(action.RetrieveMemoryActivity)
// GraphRAG activities
w.RegisterActivity(action.FetchCanvasRelationsActivity)
w.RegisterActivity(action.QueryGraphRAGActivity)
w.RegisterActivity(action.CanvasReasonerActivity)
w.RegisterActivity(action.IndexGraphRAGActivity)
w.RegisterActivity(action.CanvasCompatibilityActivity)
// Initialize health checker
healthChecker := health.NewChecker(c)
healthHandler := health.NewHandler(healthChecker)
@@ -116,7 +127,7 @@ func main() {
// Run worker in a goroutine
workerErrChan := make(chan error, 1)
go func() {
logging.Info("starting worker on queue", logging.String("queue", "poimen-taskqueue"))
logging.Info("starting worker", logging.String("queue", "poimen"))
if err := w.Run(worker.InterruptCh()); err != nil {
workerErrChan <- err
}
+3 -5
View File
@@ -1,8 +1,10 @@
module github.com/rockliang/poimen/workflows
go 1.25.4
go 1.26.0
require (
github.com/google/uuid v1.6.0
github.com/lib/pq v1.12.3
github.com/prometheus/client_golang v1.24.1
github.com/stretchr/testify v1.12.1
go.temporal.io/sdk v1.48.0
@@ -16,10 +18,8 @@ require (
github.com/facebookgo/clock v0.0.0-20150410010913-600d898af40a // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/mock v1.6.0 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.3.2 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.22.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/nexus-rpc/nexus-proto-annotations v0.1.0 // indirect
github.com/nexus-rpc/sdk-go v0.7.0 // indirect
@@ -27,8 +27,6 @@ require (
github.com/prometheus/common v0.70.1 // indirect
github.com/prometheus/procfs v0.21.1 // indirect
github.com/robfig/cron v1.2.0 // indirect
github.com/spf13/cobra v1.10.2 // indirect
github.com/spf13/pflag v1.0.9 // indirect
github.com/stretchr/objx v0.5.3 // indirect
go.temporal.io/api v1.63.4 // indirect
go.uber.org/multierr v1.11.0 // indirect
+2 -9
View File
@@ -2,7 +2,6 @@ github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/facebookgo/clock v0.0.0-20150410010913-600d898af40a h1:yDWHCSQ40h88yih2JAcL6Ls/kVkSE8GFACTGVnMPruw=
github.com/facebookgo/clock v0.0.0-20150410010913-600d898af40a/go.mod h1:7Ga40egUymuWXxAe151lTNnCv97MddSOVsjpPPkityA=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
@@ -23,8 +22,6 @@ github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.3.2 h1:sGm2vDRFUrQJO/Veii4h4z
github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.3.2/go.mod h1:wd1YpapPLivG6nQgbf7ZkG1hhSOXDhhn4MLTknx2aAc=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.22.0 h1:asbCHRVmodnJTuQ3qamDwqVOIjwqUPTYmYuemVOx+Ys=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.22.0/go.mod h1:ggCgvZ2r7uOoQjOyu2Y1NhHmEPPzzuhWgcza5M1Ji1I=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
@@ -35,6 +32,8 @@ github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/lib/pq v1.12.3 h1:tTWxr2YLKwIvK90ZXEw8GP7UFHtcbTtty8zsI+YjrfQ=
github.com/lib/pq v1.12.3/go.mod h1:/p+8NSbOcwzAEI7wiMXFlgydTwcgTr3OSKMsD2BitpA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/nexus-rpc/nexus-proto-annotations v0.1.0 h1:2fELd+9sqUtNu6Fg//pw8YFsxOvp8vZ8hfP0nHhNI80=
@@ -53,11 +52,6 @@ github.com/robfig/cron v1.2.0 h1:ZjScXvvxeQ63Dbyxy76Fj3AT3Ut0aKsyd2/tl3DTMuQ=
github.com/robfig/cron v1.2.0/go.mod h1:JGuDeoQd7Z6yL4zQhZ3OPEVHB7fL6Ka6skscFHfmt2k=
github.com/rogpeppe/go-internal v1.11.0 h1:cWPaGQEPrBb5/AsnsZesgZZ9yb1OQ+GOISoDNXVBh4M=
github.com/rogpeppe/go-internal v1.11.0/go.mod h1:ddIwULY96R17DhadqLgMfk9H9tvdUzkipdSkR5nkCZA=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
github.com/spf13/pflag v1.0.9 h1:9exaQaMOCwffKiiiYk6/BndUBv+iRViNW+4lEMi0PvY=
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/stretchr/objx v0.5.3 h1:jmXUvGomnU1o3W/V5h2VEradbpJDwGrzugQQvL0POH4=
github.com/stretchr/objx v0.5.3/go.mod h1:rDQraq+vQZU7Fde9LOZLr8Tax6zZvy4kuNKF+QYS+U0=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
@@ -89,7 +83,6 @@ go.uber.org/zap v1.28.0 h1:IZzaP1Fv73/T/pBMLk4VutPl36uNC+OSUh3JLG3FIjo=
go.uber.org/zap v1.28.0/go.mod h1:rDLpOi171uODNm/mxFcuYWxDsqWSAVkFdX4XojSKg/Q=
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+117
View File
@@ -0,0 +1,117 @@
package api
import (
"fmt"
"log"
"net/http"
"strings"
"go.temporal.io/sdk/client"
"github.com/rockliang/poimen/workflows/pkg/db"
)
// Server handles HTTP routing for workflow APIs
type Server struct {
api *WorkflowAPI
logger *log.Logger
}
// NewServer creates new HTTP server with database connection
func NewServer(database *db.DB, temporalClient client.Client, logger *log.Logger) *Server {
return &Server{
api: NewWorkflowAPI(database, temporalClient, logger),
logger: logger,
}
}
// ServeHTTP dispatches HTTP requests to appropriate handler
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// Enable CORS
w.Header().Set("Access-Control-Allow-Origin", "*")
w.Header().Set("Access-Control-Allow-Methods", "GET, POST, PUT, DELETE, OPTIONS")
w.Header().Set("Access-Control-Allow-Headers", "Content-Type, Authorization")
if r.Method == http.MethodOptions {
w.WriteHeader(http.StatusOK)
return
}
path := r.URL.Path
method := r.Method
s.logger.Printf("%s %s", method, path)
// Route requests
switch {
// Workflow endpoints
case path == "/workflows" && method == http.MethodPost:
s.api.CreateWorkflow(w, r)
case path == "/workflows" && method == http.MethodGet:
s.api.ListWorkflows(w, r)
case strings.HasPrefix(path, "/workflows/") && method == http.MethodGet:
id := strings.TrimPrefix(path, "/workflows/")
// Exclude special paths
if !strings.Contains(id, "/") {
s.api.GetWorkflow(w, r, id)
} else if strings.HasSuffix(id, "/executions") {
// GET /workflows/{id}/executions
workflowID := strings.TrimSuffix(id, "/executions")
s.api.ListExecutions(w, r, workflowID)
}
case strings.HasPrefix(path, "/workflows/") && method == http.MethodPut:
id := extractID(path, "/workflows/")
s.api.UpdateWorkflow(w, r, id)
case strings.HasPrefix(path, "/workflows/") && method == http.MethodDelete:
id := extractID(path, "/workflows/")
s.api.DeleteWorkflow(w, r, id)
// Execute workflow
case strings.HasSuffix(path, "/execute") && method == http.MethodPost:
// POST /workflows/{id}/execute
parts := strings.Split(path, "/")
if len(parts) >= 4 && parts[1] == "workflows" && parts[3] == "execute" {
s.api.ExecuteWorkflow(w, r, parts[2])
}
// GraphRAG query endpoint
case strings.HasSuffix(path, "/query") && method == http.MethodPost:
// POST /workflows/{id}/query
parts := strings.Split(path, "/")
if len(parts) >= 4 && parts[1] == "workflows" && parts[3] == "query" {
s.api.QueryWorkflowGraph(w, r, parts[2])
}
// Relation versions endpoint
case strings.Contains(path, "/relations/") && strings.Contains(path, "/versions") && method == http.MethodGet:
// GET /workflows/{id}/relations/{edge_id}/versions
parts := strings.Split(path, "/")
if len(parts) >= 6 && parts[1] == "workflows" && parts[3] == "relations" && parts[5] == "versions" {
s.api.GetWorkflowRelationVersions(w, r, parts[2], parts[4])
}
// Execution endpoints
case strings.HasPrefix(path, "/executions/") && method == http.MethodGet:
id := extractID(path, "/executions/")
s.api.GetExecution(w, r, id)
default:
http.Error(w, "Not found", http.StatusNotFound)
}
}
// extractID extracts resource ID from path
func extractID(path, prefix string) string {
id := strings.TrimPrefix(path, prefix)
if idx := strings.Index(id, "/"); idx != -1 {
return id[:idx]
}
return id
}
// Start starts the HTTP server
func (s *Server) Start(port int) error {
addr := fmt.Sprintf(":%d", port)
s.logger.Printf("Starting API server on %s", addr)
return http.ListenAndServe(addr, s)
}
+704
View File
@@ -0,0 +1,704 @@
package api
import (
"context"
"encoding/json"
"fmt"
"log"
"net/http"
"time"
"github.com/google/uuid"
"go.temporal.io/sdk/client"
"github.com/rockliang/poimen/workflows/internal/routing"
"github.com/rockliang/poimen/workflows/pkg/db"
)
// WorkflowNode matches frontend node type
type WorkflowNode struct {
ID string `json:"id"`
Type string `json:"type"` // "activity", "start", "end"
Position map[string]interface{} `json:"position"`
Data struct {
Label string `json:"label"`
Activity string `json:"activity"`
Config map[string]interface{} `json:"config"`
} `json:"data"`
}
// WorkflowEdge matches frontend edge type
type WorkflowEdge struct {
ID string `json:"id"`
Source string `json:"source"`
Target string `json:"target"`
Data map[string]interface{} `json:"data,omitempty"`
}
// WorkflowDef is the request body for creating/updating workflows
type WorkflowDef struct {
Name string `json:"name"`
Description string `json:"description"`
Nodes []WorkflowNode `json:"nodes"`
Edges []WorkflowEdge `json:"edges"`
Status string `json:"status"` // "draft", "active"
}
// WorkflowResponse is the workflow with metadata
type WorkflowResponse struct {
ID string `json:"id"`
Name string `json:"name"`
Description string `json:"description"`
Status string `json:"status"`
Version int `json:"version"`
Nodes []WorkflowNode `json:"nodes"`
Edges []WorkflowEdge `json:"edges"`
CreatedAt string `json:"createdAt"`
UpdatedAt string `json:"updatedAt"`
CreatedBy string `json:"createdBy"`
}
// ExecutionRequest is the request to execute a workflow
type ExecutionRequest struct {
Inputs map[string]interface{} `json:"inputs"`
}
// ExecutionResponse is the execution result
type ExecutionResponse struct {
ID string `json:"id"`
WorkflowID string `json:"workflowId"`
Status string `json:"status"` // "pending", "running", "success", "failed"
StartedAt string `json:"startedAt"`
CompletedAt string `json:"completedAt,omitempty"`
Inputs map[string]interface{} `json:"inputs"`
Outputs map[string]interface{} `json:"outputs,omitempty"`
Errors []string `json:"errors,omitempty"`
Logs []ExecutionLog `json:"logs"`
}
// ExecutionLog is a log entry from execution
type ExecutionLog struct {
Timestamp string `json:"timestamp"`
NodeID string `json:"nodeId"`
Level string `json:"level"` // "info", "warn", "error"
Message string `json:"message"`
}
// WorkflowAPI handles workflow endpoints
type WorkflowAPI struct {
db *db.DB
temporalClient client.Client
logger *log.Logger
customerID string // TODO: Extract from JWT token
}
// NewWorkflowAPI creates new API handler
func NewWorkflowAPI(database *db.DB, tc client.Client, logger *log.Logger) *WorkflowAPI {
return &WorkflowAPI{
db: database,
temporalClient: tc,
logger: logger,
customerID: "default-customer", // TODO: From auth context
}
}
// CreateWorkflow handles POST /workflows
func (api *WorkflowAPI) CreateWorkflow(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
return
}
var req WorkflowDef
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, fmt.Sprintf("Invalid request: %v", err), http.StatusBadRequest)
return
}
if req.Name == "" {
http.Error(w, "Workflow name required", http.StatusBadRequest)
return
}
// Create workflow in database
id := uuid.New().String()
now := time.Now()
// Convert nodes and edges to JSONB
nodesJSON, err := json.Marshal(req.Nodes)
if err != nil {
http.Error(w, fmt.Sprintf("Failed to marshal nodes: %v", err), http.StatusBadRequest)
return
}
edgesJSON, err := json.Marshal(req.Edges)
if err != nil {
http.Error(w, fmt.Sprintf("Failed to marshal edges: %v", err), http.StatusBadRequest)
return
}
status := req.Status
if status == "" {
status = "draft"
}
workflow := &db.Workflow{
ID: id,
CustomerID: api.customerID,
Name: req.Name,
Description: req.Description,
Status: status,
Version: 1,
Nodes: nodesJSON,
Edges: edgesJSON,
CreatedBy: "anonymous", // Use JWT claim in real implementation
CreatedAt: now,
UpdatedAt: now,
}
if err := api.db.SaveWorkflow(r.Context(), workflow); err != nil {
api.logger.Printf("Failed to save workflow: %v", err)
http.Error(w, "Failed to create workflow", http.StatusInternalServerError)
return
}
response := WorkflowResponse{
ID: workflow.ID,
Name: workflow.Name,
Description: workflow.Description,
Status: workflow.Status,
Version: workflow.Version,
Nodes: req.Nodes,
Edges: req.Edges,
CreatedAt: workflow.CreatedAt.Format(time.RFC3339),
UpdatedAt: workflow.UpdatedAt.Format(time.RFC3339),
CreatedBy: workflow.CreatedBy,
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusCreated)
json.NewEncoder(w).Encode(response)
}
// ListWorkflows handles GET /workflows
func (api *WorkflowAPI) ListWorkflows(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
return
}
page := 1
limit := 10
// Parse pagination params if needed
workflows, err := api.db.ListWorkflows(r.Context(), api.customerID, limit, (page-1)*limit)
if err != nil {
api.logger.Printf("Failed to list workflows: %v", err)
http.Error(w, "Failed to list workflows", http.StatusInternalServerError)
return
}
list := make([]WorkflowResponse, 0)
for _, wf := range workflows {
var nodes []WorkflowNode
var edges []WorkflowEdge
json.Unmarshal(wf.Nodes, &nodes)
json.Unmarshal(wf.Edges, &edges)
list = append(list, WorkflowResponse{
ID: wf.ID,
Name: wf.Name,
Description: wf.Description,
Status: wf.Status,
Version: wf.Version,
Nodes: nodes,
Edges: edges,
CreatedAt: wf.CreatedAt.Format(time.RFC3339),
UpdatedAt: wf.UpdatedAt.Format(time.RFC3339),
CreatedBy: wf.CreatedBy,
})
}
response := map[string]interface{}{
"workflows": list,
"total": len(list),
"page": page,
"limit": limit,
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(response)
}
// GetWorkflow handles GET /workflows/{id}
func (api *WorkflowAPI) GetWorkflow(w http.ResponseWriter, r *http.Request, id string) {
if r.Method != http.MethodGet {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
return
}
workflow, err := api.db.FetchWorkflow(r.Context(), id, api.customerID)
if err != nil {
http.Error(w, "Workflow not found", http.StatusNotFound)
return
}
var nodes []WorkflowNode
var edges []WorkflowEdge
json.Unmarshal(workflow.Nodes, &nodes)
json.Unmarshal(workflow.Edges, &edges)
response := WorkflowResponse{
ID: workflow.ID,
Name: workflow.Name,
Description: workflow.Description,
Status: workflow.Status,
Version: workflow.Version,
Nodes: nodes,
Edges: edges,
CreatedAt: workflow.CreatedAt.Format(time.RFC3339),
UpdatedAt: workflow.UpdatedAt.Format(time.RFC3339),
CreatedBy: workflow.CreatedBy,
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(response)
}
// UpdateWorkflow handles PUT /workflows/{id}
func (api *WorkflowAPI) UpdateWorkflow(w http.ResponseWriter, r *http.Request, id string) {
if r.Method != http.MethodPut {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
return
}
// Fetch existing workflow
workflow, err := api.db.FetchWorkflow(r.Context(), id, api.customerID)
if err != nil {
http.Error(w, "Workflow not found", http.StatusNotFound)
return
}
var req WorkflowDef
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, fmt.Sprintf("Invalid request: %v", err), http.StatusBadRequest)
return
}
// Update fields
if req.Name != "" {
workflow.Name = req.Name
}
if req.Description != "" {
workflow.Description = req.Description
}
if req.Nodes != nil {
nodesJSON, _ := json.Marshal(req.Nodes)
workflow.Nodes = nodesJSON
}
if req.Edges != nil {
edgesJSON, _ := json.Marshal(req.Edges)
workflow.Edges = edgesJSON
}
if req.Status != "" {
workflow.Status = req.Status
}
workflow.Version++
workflow.UpdatedAt = time.Now()
if err := api.db.SaveWorkflow(r.Context(), workflow); err != nil {
api.logger.Printf("Failed to update workflow: %v", err)
http.Error(w, "Failed to update workflow", http.StatusInternalServerError)
return
}
var nodes []WorkflowNode
var edges []WorkflowEdge
json.Unmarshal(workflow.Nodes, &nodes)
json.Unmarshal(workflow.Edges, &edges)
response := WorkflowResponse{
ID: workflow.ID,
Name: workflow.Name,
Description: workflow.Description,
Status: workflow.Status,
Version: workflow.Version,
Nodes: nodes,
Edges: edges,
CreatedAt: workflow.CreatedAt.Format(time.RFC3339),
UpdatedAt: workflow.UpdatedAt.Format(time.RFC3339),
CreatedBy: workflow.CreatedBy,
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(response)
}
// DeleteWorkflow handles DELETE /workflows/{id}
func (api *WorkflowAPI) DeleteWorkflow(w http.ResponseWriter, r *http.Request, id string) {
if r.Method != http.MethodDelete {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
return
}
if err := api.db.DeleteWorkflow(r.Context(), id, api.customerID); err != nil {
http.Error(w, "Workflow not found", http.StatusNotFound)
return
}
w.WriteHeader(http.StatusNoContent)
}
// ExecuteWorkflow handles POST /workflows/{id}/execute
func (api *WorkflowAPI) ExecuteWorkflow(w http.ResponseWriter, r *http.Request, id string) {
if r.Method != http.MethodPost {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
return
}
workflow, err := api.db.FetchWorkflow(r.Context(), id, api.customerID)
if err != nil {
http.Error(w, "Workflow not found", http.StatusNotFound)
return
}
var req ExecutionRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, fmt.Sprintf("Invalid request: %v", err), http.StatusBadRequest)
return
}
// Unmarshal nodes and edges
var nodes []WorkflowNode
var edges []WorkflowEdge
json.Unmarshal(workflow.Nodes, &nodes)
json.Unmarshal(workflow.Edges, &edges)
// Convert to workflow response for spec conversion
workflowResp := &WorkflowResponse{
ID: workflow.ID,
Name: workflow.Name,
Description: workflow.Description,
Status: workflow.Status,
Version: workflow.Version,
Nodes: nodes,
Edges: edges,
CreatedBy: workflow.CreatedBy,
}
// Convert nodes/edges to WorkflowSpec
spec := api.nodesToWorkflowSpec(workflowResp, req.Inputs)
// Execute via Temporal RoutingWorkflow
execID := uuid.New().String()
workflowOptions := client.StartWorkflowOptions{
ID: execID,
TaskQueue: "default",
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
_, err = api.temporalClient.ExecuteWorkflow(ctx, workflowOptions, "RoutingWorkflow", spec)
if err != nil {
api.logger.Printf("Failed to execute workflow: %v", err)
http.Error(w, fmt.Sprintf("Execution failed: %v", err), http.StatusInternalServerError)
return
}
// Save execution to database
inputsJSON, _ := json.Marshal(req.Inputs)
now := time.Now()
execution := &db.WorkflowExecution{
ID: execID,
WorkflowID: id,
CustomerID: api.customerID,
TemporalID: execID,
Status: "running",
Inputs: inputsJSON,
StartedAt: now,
}
if err := api.db.SaveExecution(r.Context(), execution); err != nil {
api.logger.Printf("Failed to save execution: %v", err)
http.Error(w, "Failed to save execution", http.StatusInternalServerError)
return
}
// Create execution response
execResp := ExecutionResponse{
ID: execID,
WorkflowID: id,
Status: "running",
StartedAt: now.Format(time.RFC3339),
Inputs: req.Inputs,
Outputs: make(map[string]interface{}),
Logs: []ExecutionLog{},
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusCreated)
json.NewEncoder(w).Encode(execResp)
}
// GetExecution handles GET /executions/{id}
func (api *WorkflowAPI) GetExecution(w http.ResponseWriter, r *http.Request, id string) {
if r.Method != http.MethodGet {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
return
}
execution, err := api.db.FetchExecution(r.Context(), id)
if err != nil {
http.Error(w, "Execution not found", http.StatusNotFound)
return
}
// Get logs from database
logs, err := api.db.FetchExecutionLogs(r.Context(), id)
if err != nil {
api.logger.Printf("Failed to fetch logs: %v", err)
}
execLogs := make([]ExecutionLog, 0)
for _, log := range logs {
execLogs = append(execLogs, ExecutionLog{
Timestamp: log.LoggedAt.Format(time.RFC3339),
NodeID: log.NodeID,
Level: log.Level,
Message: log.Message,
})
}
// Parse inputs/outputs
var inputs map[string]interface{}
var outputs map[string]interface{}
json.Unmarshal(execution.Inputs, &inputs)
if execution.Outputs != nil {
json.Unmarshal(execution.Outputs, &outputs)
}
// Check Temporal workflow status
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
desc, err := api.temporalClient.DescribeWorkflowExecution(ctx, execution.TemporalID, "")
status := execution.Status
if err == nil && desc != nil {
switch desc.Status.String() {
case "RUNNING":
status = "running"
case "COMPLETED":
status = "success"
case "FAILED":
status = "failed"
}
}
completedAtStr := ""
if execution.CompletedAt != nil {
completedAtStr = execution.CompletedAt.Format(time.RFC3339)
}
execResp := ExecutionResponse{
ID: execution.ID,
WorkflowID: execution.WorkflowID,
Status: status,
StartedAt: execution.StartedAt.Format(time.RFC3339),
CompletedAt: completedAtStr,
Inputs: inputs,
Outputs: outputs,
Logs: execLogs,
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(execResp)
}
// ListExecutions handles GET /workflows/{id}/executions
func (api *WorkflowAPI) ListExecutions(w http.ResponseWriter, r *http.Request, workflowID string) {
if r.Method != http.MethodGet {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
return
}
// TODO: Implement query by workflow_id in database
// For now, return empty list (needs DB method for filtering by workflow_id)
list := make([]ExecutionResponse, 0)
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(list)
}
// nodesToWorkflowSpec converts frontend nodes/edges to routing.WorkflowSpec
func (api *WorkflowAPI) nodesToWorkflowSpec(wf *WorkflowResponse, inputs map[string]interface{}) *routing.WorkflowSpec {
spec := &routing.WorkflowSpec{
Name: wf.Name,
Input: inputs,
States: []routing.State{},
}
// Build states from nodes
stateMap := make(map[string]*routing.State)
// Create all states
for _, node := range wf.Nodes {
if node.Type == "activity" {
state := &routing.State{
Name: node.ID,
Type: routing.StateTypeTask,
Resource: node.Data.Activity,
Parameters: node.Data.Config,
End: false,
}
stateMap[node.ID] = state
spec.States = append(spec.States, *state)
}
}
// Wire edges (transitions)
for _, edge := range wf.Edges {
if state, exists := stateMap[edge.Source]; exists {
state.Next = edge.Target
}
}
// Mark last state as End
if len(spec.States) > 0 {
// Find state with no outgoing edge
for i := range spec.States {
hasNext := false
for _, edge := range wf.Edges {
if edge.Source == spec.States[i].Name {
hasNext = true
break
}
}
if !hasNext {
spec.States[i].End = true
}
}
}
return spec
}
// QueryWorkflowGraph handles POST /workflows/{id}/query
func (api *WorkflowAPI) QueryWorkflowGraph(w http.ResponseWriter, r *http.Request, workflowID string) {
ctx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
defer cancel()
var req QueryWorkflowGraphRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, "Invalid request body", http.StatusBadRequest)
return
}
// Defaults
if req.SearchType == "" {
req.SearchType = "edges"
}
if req.ConfidenceFloor == 0 {
req.ConfidenceFloor = 0.5
}
if req.TopK == 0 {
req.TopK = 10
}
if req.MaxPathDepth == 0 {
req.MaxPathDepth = 3
}
if req.RankingProfile == "" {
req.RankingProfile = "default"
}
// Get latest version if not specified
if req.Version == 0 {
wf, err := api.db.GetWorkflow(ctx, workflowID)
if err != nil {
http.Error(w, "Workflow not found", http.StatusNotFound)
return
}
req.Version = wf.Version
}
// Call temporal workflow
run, err := api.temporalClient.ExecuteWorkflow(
ctx,
client.StartWorkflowOptions{
ID: fmt.Sprintf("graph-query-%s-v%d", workflowID, req.Version),
TaskQueue: "poimen",
},
"WorkflowGraphQuery",
map[string]interface{}{
"workflow_id": workflowID,
"query": req.Query,
"search_type": req.SearchType,
"relation_type": req.RelationType,
"version": req.Version,
"confidence_floor": req.ConfidenceFloor,
"top_k": req.TopK,
"find_paths": req.FindPaths,
"target_node_id": req.TargetNodeID,
"max_path_depth": req.MaxPathDepth,
"ranking_profile": req.RankingProfile,
"include_reasoning": req.IncludeReasoning,
},
)
if err != nil {
api.logger.Printf("Failed to start workflow: %v", err)
http.Error(w, "Failed to start query workflow", http.StatusInternalServerError)
return
}
var result map[string]interface{}
if err := run.Get(ctx, &result); err != nil {
api.logger.Printf("Workflow execution failed: %v", err)
http.Error(w, "Query execution failed", http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(result)
}
// QueryWorkflowGraphRequest matches frontend payload
type QueryWorkflowGraphRequest struct {
Query string `json:"query"`
SearchType string `json:"search_type"`
RelationType string `json:"relation_type"`
Version int `json:"version"`
ConfidenceFloor float64 `json:"confidence_floor"`
TopK int `json:"top_k"`
FindPaths bool `json:"find_paths"`
TargetNodeID string `json:"target_node_id"`
MaxPathDepth int `json:"max_path_depth"`
RankingProfile string `json:"ranking_profile"`
IncludeReasoning bool `json:"include_reasoning"`
}
// GetWorkflowRelationVersions handles GET /workflows/{id}/relations/{edge_id}/versions
func (api *WorkflowAPI) GetWorkflowRelationVersions(w http.ResponseWriter, r *http.Request, workflowID, edgeID string) {
ctx, cancel := context.WithTimeout(r.Context(), 10*time.Second)
defer cancel()
// Query relation versions from DB
versions, err := api.db.GetRelationVersions(ctx, workflowID, edgeID)
if err != nil {
api.logger.Printf("Failed to get relation versions: %v", err)
http.Error(w, "Failed to fetch relation versions", http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(map[string]interface{}{
"workflow_id": workflowID,
"edge_id": edgeID,
"versions": versions,
"total_count": len(versions),
})
}
+260 -3
View File
@@ -407,11 +407,265 @@
"dependencies": [],
"notes": "Network-dependent. First activity to run for context-aware routing. Fast timeout."
}
},
{
"name": "AssumeRoleActivity",
"description": "Request temporary JWT token for accessing LLM APIs (like AWS AssumeRole)",
"category": "authentication",
"inputs": {
"identity": {
"type": "string",
"description": "User/service identity requesting access",
"required": true,
"examples": ["[email protected]", "service:poimen-worker"]
},
"clientId": {
"type": "string",
"description": "OAuth2 client ID (from vault if not provided)",
"required": false
},
"clientSecret": {
"type": "string",
"description": "OAuth2 client secret (from vault if not provided)",
"required": false
},
"scope": {
"type": "string",
"description": "Scope of access (e.g., 'llm:read' or 'llm:read llm:write')",
"required": true,
"examples": ["llm:read", "llm:read llm:write", "llm:admin"]
},
"durationSeconds": {
"type": "integer",
"description": "Token validity duration in seconds (default: 3600, max: 86400)",
"required": false,
"default": 3600
},
"authServerUrl": {
"type": "string",
"description": "Auth server URL (from AUTH_SERVER_URL env if not provided)",
"required": false
}
},
"outputs": {
"token": {
"type": "string",
"description": "JWT token for calling api.riotpiao.com"
},
"expiresAt": {
"type": "integer",
"description": "Token expiration time (Unix timestamp)"
},
"expiresIn": {
"type": "integer",
"description": "Seconds until token expires"
},
"tokenType": {
"type": "string",
"description": "Token type (typically 'Bearer')"
}
},
"constraints": {
"defaultTimeout": "30s",
"isFlaky": false,
"recommendedRetries": 2,
"retryBackoff": 1.5,
"dependencies": [],
"notes": "Must run before LLM Router to provide auth token. Call early in workflow."
}
},
{
"name": "LLMInferenceActivity",
"description": "Call LLM API with custom prompt and get response text",
"category": "llm",
"inputs": {
"model": {
"type": "string",
"description": "Model ID (reasoning, ornith:35b, ornith:13b, qwen2.5:3b)",
"required": true,
"examples": ["reasoning", "ornith:35b"]
},
"system_prompt": {
"type": "string",
"description": "System instruction for the model",
"required": false,
"default": ""
},
"user_prompt": {
"type": "string",
"description": "User message to send to the model",
"required": true
},
"temperature": {
"type": "number",
"description": "Sampling temperature (0.0-1.0, higher=more creative)",
"required": false,
"default": 0.7
},
"max_tokens": {
"type": "integer",
"description": "Maximum tokens in response",
"required": false
}
},
"outputs": {
"response": {
"type": "string",
"description": "LLM response text"
},
"model": {
"type": "string",
"description": "Model used for inference"
},
"stop_reason": {
"type": "string",
"description": "Why inference stopped (stop_sequence, length, etc)"
},
"tokens_used": {
"type": "integer",
"description": "Total tokens consumed"
}
},
"constraints": {
"defaultTimeout": "120s",
"isFlaky": true,
"recommendedRetries": 2,
"retryBackoff": 2.0,
"dependencies": [],
"notes": "API-dependent. Network flaky. Use for single prompts. See LLMBatchInferenceActivity for multiple."
}
},
{
"name": "LLMBatchInferenceActivity",
"description": "Call LLM API multiple times sequentially with different prompts",
"category": "llm",
"inputs": {
"model": {
"type": "string",
"description": "Model ID (reasoning, ornith:35b, ornith:13b, qwen2.5:3b)",
"required": true
},
"system_prompt": {
"type": "string",
"description": "System instruction (same for all prompts)",
"required": false
},
"prompts": {
"type": "array",
"description": "List of user prompts to process",
"required": true,
"items": {
"type": "string"
}
},
"temperature": {
"type": "number",
"description": "Sampling temperature (0.0-1.0)",
"required": false,
"default": 0.7
}
},
"outputs": {
"responses": {
"type": "array",
"description": "List of LLM responses (parallel to input prompts)",
"items": {
"type": "string"
}
},
"model": {
"type": "string",
"description": "Model used"
},
"errors": {
"type": "array",
"description": "Error messages for failed prompts",
"items": {
"type": "string"
}
}
},
"constraints": {
"defaultTimeout": "600s",
"isFlaky": true,
"recommendedRetries": 1,
"retryBackoff": 2.0,
"dependencies": [],
"notes": "Sequential processing of multiple prompts. Use for batch analysis, summarization, etc."
}
},
{
"name": "CanvasReasonerActivity",
"description": "Use LLM reasoning to infer and suggest connections between workflow activities",
"category": "workflow",
"inputs": {
"nodes": {
"type": "array",
"description": "Canvas workflow nodes to analyze",
"required": true,
"items": {
"type": "object",
"properties": {
"id": {"type": "string"},
"type": {"type": "string"},
"label": {"type": "string"}
}
}
},
"edges": {
"type": "array",
"description": "Existing edges in the workflow",
"required": false,
"items": {
"type": "object"
}
},
"preserve_existing": {
"type": "boolean",
"description": "If true, only suggest new edges; if false, redesign entire workflow",
"required": false,
"default": true
},
"auth_token": {
"type": "string",
"description": "JWT token for authenticated LLM calls",
"required": false
}
},
"outputs": {
"suggested_edges": {
"type": "array",
"description": "Edges suggested by LLM reasoning",
"items": {
"type": "object",
"properties": {
"source": {"type": "string"},
"target": {"type": "string"}
}
}
},
"reasoning": {
"type": "string",
"description": "LLM explanation of suggested connections"
},
"confidence": {
"type": "number",
"description": "Confidence score (0.0-1.0) of the suggestions"
}
},
"constraints": {
"defaultTimeout": "120s",
"isFlaky": true,
"recommendedRetries": 2,
"retryBackoff": 2.0,
"dependencies": [],
"notes": "Uses reasoning model to analyze workflow logic. Good for understanding data flow and connections between activities."
}
}
],
"metadata": {
"totalActivities": 9,
"lastUpdated": "2025-08-31T00:00:00Z",
"totalActivities": 13,
"lastUpdated": "2025-09-05T00:00:00Z",
"categories": {
"repository": 1,
"analysis": 1,
@@ -421,7 +675,10 @@
"notification": 1,
"approval": 1,
"storage": 1,
"memory": 1
"memory": 1,
"authentication": 1,
"llm": 2,
"workflow": 1
}
}
}
+179
View File
@@ -0,0 +1,179 @@
package routing
import (
"fmt"
"github.com/rockliang/poimen/workflows/pkg/db"
)
// CanvasConverter converts visual canvas to executable WorkflowSpec
type CanvasConverter struct {
validator *CanvasValidator
}
// NewCanvasConverter creates a converter
func NewCanvasConverter() *CanvasConverter {
return &CanvasConverter{
validator: NewCanvasValidator(),
}
}
// CanvasToWorkflowSpec converts canvas to WorkflowSpec
func (cc *CanvasConverter) CanvasToWorkflowSpec(canvas *db.Canvas) (*WorkflowSpec, error) {
// Validate first
if err := cc.validator.ValidateCanvas(canvas); err != nil {
return nil, fmt.Errorf("canvas validation failed: %w", err)
}
// Get topological order
sortedNodes, err := cc.validator.TopoSort(canvas.Nodes, canvas.Edges)
if err != nil {
return nil, fmt.Errorf("topological sort failed: %w", err)
}
// Build states from sorted nodes
states := []State{}
nodeToState := make(map[string]int) // node ID to state index
for i, node := range sortedNodes {
state := cc.nodeToState(node, canvas.Edges)
states = append(states, state)
nodeToState[node.ID] = i
}
// Wire up transitions
for i, node := range sortedNodes {
outgoing := cc.getOutgoingEdges(node.ID, canvas.Edges)
if len(outgoing) == 0 {
// Last state - no transitions
continue
}
if len(outgoing) == 1 {
// Single outgoing edge
targetNode := outgoing[0]
targetIdx := nodeToState[targetNode]
if targetIdx > i {
states[i].Next = states[targetIdx].Name
}
} else {
// Multiple outgoing edges - parallel
states[i].Type = "Parallel"
branches := []interface{}{}
for _, targetNode := range outgoing {
branches = append(branches, map[string]string{
"state": states[nodeToState[targetNode]].Name,
})
}
if states[i].Branches == nil {
states[i].Branches = branches
}
}
}
spec := &WorkflowSpec{
Name: canvas.Name,
Input: map[string]interface{}{},
States: states,
}
return spec, nil
}
// nodeToState converts a canvas node to a workflow state
func (cc *CanvasConverter) nodeToState(node db.WorkflowNode, edges []db.WorkflowEdge) State {
// Map node type to activity name
activityName := cc.mapActivityType(node.Type)
state := State{
Name: node.ID,
Type: TaskActivity,
Activity: activityName,
Retry: &RetryPolicy{MaxAttempts: 3, BackoffSeconds: 2},
Timeout: "300s",
Parameters: node.Data,
}
return state
}
// mapActivityType maps canvas activity type to Poimen activity
func (cc *CanvasConverter) mapActivityType(canvasType string) string {
typeMap := map[string]string{
"clone-repo": "CloneRepoActivity",
"analyze-code": "AnalyzeCodeActivity",
"security-scan": "SecurityScanActivity",
"generate-report": "GenerateReportActivity",
"deployment-precheck": "DeploymentPreCheckActivity",
"notify-status": "NotifyStatusActivity",
"approve-workflow": "ApproveWorkflowActivity",
"archive-results": "ArchiveResultsActivity",
"retrieve-memory": "RetrieveMemoryActivity",
"assume-role": "AssumeRoleActivity",
"llm-inference": "LLMInferenceActivity",
"llm-batch-inference": "LLMBatchInferenceActivity",
"canvas-reasoner": "CanvasReasonerActivity",
}
if mapped, ok := typeMap[canvasType]; ok {
return mapped
}
return canvasType // fallback to type as-is
}
// getOutgoingEdges returns target node IDs for a given source node
func (cc *CanvasConverter) getOutgoingEdges(nodeID string, edges []db.WorkflowEdge) []string {
targets := []string{}
seen := make(map[string]bool)
for _, edge := range edges {
if edge.Source == nodeID && !seen[edge.Target] {
targets = append(targets, edge.Target)
seen[edge.Target] = true
}
}
return targets
}
// CanvasToExecutionPlan converts canvas to sequential activity list
func (cc *CanvasConverter) CanvasToExecutionPlan(canvas *db.Canvas) ([]ExecutionStep, error) {
// Validate first
if err := cc.validator.ValidateCanvas(canvas); err != nil {
return nil, fmt.Errorf("canvas validation failed: %w", err)
}
// Get topological order
sortedNodes, err := cc.validator.TopoSort(canvas.Nodes, canvas.Edges)
if err != nil {
return nil, fmt.Errorf("topological sort failed: %w", err)
}
steps := []ExecutionStep{}
for i, node := range sortedNodes {
step := ExecutionStep{
Index: i,
NodeID: node.ID,
ActivityName: cc.mapActivityType(node.Type),
Label: node.Label,
Parameters: node.Data,
Timeout: "300s",
}
steps = append(steps, step)
}
return steps, nil
}
// ExecutionStep represents one activity in execution plan
type ExecutionStep struct {
Index int `json:"index"`
NodeID string `json:"node_id"`
ActivityName string `json:"activity_name"`
Label string `json:"label"`
Parameters map[string]interface{} `json:"parameters"`
Timeout string `json:"timeout"`
DependsOn []int `json:"depends_on,omitempty"` // Indices of predecessor steps
}
+317
View File
@@ -0,0 +1,317 @@
package routing
import (
"fmt"
"strings"
"github.com/rockliang/poimen/workflows/pkg/db"
)
// CanvasValidator validates React Flow canvas (nodes + edges)
type CanvasValidator struct {
activityRegistry map[string]bool
}
// NewCanvasValidator creates validator with activity registry
func NewCanvasValidator() *CanvasValidator {
return &CanvasValidator{
activityRegistry: map[string]bool{
"clone-repo": true,
"analyze-code": true,
"security-scan": true,
"generate-report": true,
"deployment-precheck": true,
"notify-status": true,
"approve-workflow": true,
"archive-results": true,
"retrieve-memory": true,
"assume-role": true,
"llm-inference": true,
"llm-batch-inference": true,
"canvas-reasoner": true,
},
}
}
// ValidateCanvas checks canvas structure, connectivity, and DAG
func (cv *CanvasValidator) ValidateCanvas(canvas *db.Canvas) error {
if canvas == nil {
return fmt.Errorf("canvas is nil")
}
if len(canvas.Nodes) == 0 {
return fmt.Errorf("canvas has no nodes")
}
// Step 1: Validate nodes
if err := cv.validateNodes(canvas.Nodes); err != nil {
return fmt.Errorf("node validation failed: %w", err)
}
// Step 2: Validate edges
if err := cv.validateEdges(canvas.Nodes, canvas.Edges); err != nil {
return fmt.Errorf("edge validation failed: %w", err)
}
// Step 3: Check for cycles (must be DAG)
if err := cv.detectCycles(canvas.Nodes, canvas.Edges); err != nil {
return fmt.Errorf("cycle detected: %w", err)
}
// Step 4: Check connectivity (all nodes reachable from start)
if err := cv.validateConnectivity(canvas.Nodes, canvas.Edges); err != nil {
return fmt.Errorf("connectivity check failed: %w", err)
}
return nil
}
// validateNodes checks each node has required fields and valid type
func (cv *CanvasValidator) validateNodes(nodes []db.WorkflowNode) error {
if len(nodes) == 0 {
return fmt.Errorf("no nodes in canvas")
}
nodeIds := make(map[string]bool)
for i, node := range nodes {
// Check required fields
if node.ID == "" {
return fmt.Errorf("node[%d] has empty ID", i)
}
if nodeIds[node.ID] {
return fmt.Errorf("node[%d] has duplicate ID: %s", i, node.ID)
}
nodeIds[node.ID] = true
if node.Label == "" {
return fmt.Errorf("node[%d] (%s) has empty label", i, node.ID)
}
if node.Position == nil {
return fmt.Errorf("node[%d] (%s) has no position", i, node.ID)
}
// Check activity type (if present)
if node.Type != "" && !cv.activityRegistry[strings.ToLower(node.Type)] {
return fmt.Errorf("node[%d] (%s) has unknown activity type: %s", i, node.ID, node.Type)
}
// Check data structure
if node.Data == nil {
return fmt.Errorf("node[%d] (%s) has no data", i, node.ID)
}
}
return nil
}
// validateEdges checks edges reference valid nodes
func (cv *CanvasValidator) validateEdges(nodes []db.WorkflowNode, edges []db.WorkflowEdge) error {
nodeIds := make(map[string]bool)
for _, node := range nodes {
nodeIds[node.ID] = true
}
for i, edge := range edges {
// Check required fields
if edge.Source == "" {
return fmt.Errorf("edge[%d] has empty source", i)
}
if edge.Target == "" {
return fmt.Errorf("edge[%d] has empty target", i)
}
// Check source node exists
if !nodeIds[edge.Source] {
return fmt.Errorf("edge[%d] references unknown source node: %s", i, edge.Source)
}
// Check target node exists
if !nodeIds[edge.Target] {
return fmt.Errorf("edge[%d] references unknown target node: %s", i, edge.Target)
}
// Check self-loops (discouraged but allow for now)
if edge.Source == edge.Target {
// Could warn here but not fail
}
}
return nil
}
// detectCycles checks for cycles in the DAG (must be acyclic)
func (cv *CanvasValidator) detectCycles(nodes []db.WorkflowNode, edges []db.WorkflowEdge) error {
// Build adjacency list
graph := make(map[string][]string)
inDegree := make(map[string]int)
for _, node := range nodes {
graph[node.ID] = []string{}
inDegree[node.ID] = 0
}
for _, edge := range edges {
graph[edge.Source] = append(graph[edge.Source], edge.Target)
inDegree[edge.Target]++
}
// Kahn's algorithm: topological sort
queue := []string{}
for _, node := range nodes {
if inDegree[node.ID] == 0 {
queue = append(queue, node.ID)
}
}
processed := 0
for len(queue) > 0 {
// Dequeue
current := queue[0]
queue = queue[1:]
processed++
// Visit neighbors
for _, neighbor := range graph[current] {
inDegree[neighbor]--
if inDegree[neighbor] == 0 {
queue = append(queue, neighbor)
}
}
}
// If we didn't process all nodes, there's a cycle
if processed != len(nodes) {
return fmt.Errorf("graph has cycle (processed %d/%d nodes)", processed, len(nodes))
}
return nil
}
// validateConnectivity checks all nodes are reachable from start nodes
func (cv *CanvasValidator) validateConnectivity(nodes []db.WorkflowNode, edges []db.WorkflowEdge) error {
if len(nodes) == 0 {
return nil
}
// Build adjacency list
graph := make(map[string][]string)
inDegree := make(map[string]int)
for _, node := range nodes {
graph[node.ID] = []string{}
inDegree[node.ID] = 0
}
for _, edge := range edges {
graph[edge.Source] = append(graph[edge.Source], edge.Target)
inDegree[edge.Target]++
}
// Find start nodes (in-degree 0)
startNodes := []string{}
for _, node := range nodes {
if inDegree[node.ID] == 0 {
startNodes = append(startNodes, node.ID)
}
}
if len(startNodes) == 0 {
return fmt.Errorf("no start nodes found (all nodes have incoming edges)")
}
// BFS from all start nodes
visited := make(map[string]bool)
queue := startNodes
for len(queue) > 0 {
// Dequeue
current := queue[0]
queue = queue[1:]
if visited[current] {
continue
}
visited[current] = true
// Visit neighbors
for _, neighbor := range graph[current] {
if !visited[neighbor] {
queue = append(queue, neighbor)
}
}
}
// Check all nodes were visited
if len(visited) != len(nodes) {
unreached := []string{}
for _, node := range nodes {
if !visited[node.ID] {
unreached = append(unreached, node.ID)
}
}
return fmt.Errorf("unreachable nodes: %v", unreached)
}
return nil
}
// TopoSort returns nodes in topological order (execution order)
func (cv *CanvasValidator) TopoSort(nodes []db.WorkflowNode, edges []db.WorkflowEdge) ([]db.WorkflowNode, error) {
if len(nodes) == 0 {
return []db.WorkflowNode{}, nil
}
// Build adjacency list and in-degree map
graph := make(map[string][]string)
inDegree := make(map[string]int)
nodeMap := make(map[string]db.WorkflowNode)
for _, node := range nodes {
graph[node.ID] = []string{}
inDegree[node.ID] = 0
nodeMap[node.ID] = node
}
for _, edge := range edges {
graph[edge.Source] = append(graph[edge.Source], edge.Target)
inDegree[edge.Target]++
}
// Kahn's algorithm
queue := []string{}
for _, node := range nodes {
if inDegree[node.ID] == 0 {
queue = append(queue, node.ID)
}
}
result := []db.WorkflowNode{}
processed := make(map[string]bool)
for len(queue) > 0 {
// Dequeue
current := queue[0]
queue = queue[1:]
result = append(result, nodeMap[current])
processed[current] = true
// Visit neighbors
for _, neighbor := range graph[current] {
inDegree[neighbor]--
if inDegree[neighbor] == 0 {
queue = append(queue, neighbor)
}
}
}
if len(result) != len(nodes) {
return nil, fmt.Errorf("topological sort failed: graph has cycle")
}
return result, nil
}
-16
View File
@@ -53,12 +53,6 @@ type LLMAuth struct {
// HeaderValue is the custom header value for Custom auth
HeaderValue string `json:"headerValue,omitempty"`
// TenantID is the tenant/customer ID for multi-tenant federated access
TenantID string `json:"tenantId,omitempty"`
// Scopes are the OAuth2 scopes (space-separated)
Scopes string `json:"scopes,omitempty"`
}
// LLMClient is a simple LLM client for routing
@@ -216,16 +210,6 @@ func (c *LLMClient) applyAuth(req *http.Request) error {
req.Header.Set(c.auth.HeaderName, c.auth.HeaderValue)
}
// Add tenant ID if specified (for multi-tenant federated access)
if c.auth.TenantID != "" {
req.Header.Set("X-Tenant-ID", c.auth.TenantID)
}
// Add scopes if specified (for OAuth2 flows)
if c.auth.Scopes != "" {
req.Header.Set("X-OAuth-Scopes", c.auth.Scopes)
}
return nil
}
-1
View File
@@ -69,7 +69,6 @@ type LLMRouterConfig struct {
Validators []WorkflowValidator
ParamBinder ParameterBinder
Auth *LLMAuth // Authentication config for LLM API
TenantID string // Tenant ID for multi-tenant isolation
}
// NewLLMRouter creates a new LLM router with custom config
View File
+2 -2
View File
@@ -9,6 +9,6 @@ metadata:
app.kubernetes.io/name: poimen
app.kubernetes.io/component: orchestrator
data:
GIT_COMMIT: "30644a8e" # Updated automatically by CI/CD
GIT_COMMIT: "84b4ca120" # Updated automatically by CI/CD
GIT_BRANCH: "main"
DEPLOYMENT_DATE: "2026-09-04"
DEPLOYMENT_DATE: "2026-09-05"
+11 -7
View File
@@ -4,15 +4,19 @@ kind: Kustomization
namespace: poimen
resources:
- worker-deployment.yaml
- configmap.yaml
- poimen-application.yaml
commonLabels:
app.kubernetes.io/name: poimen
app.kubernetes.io/component: worker
secretGenerator:
- name: poimen-secrets
envs:
- secrets.env
behavior: create
images:
- name: forgejo.riotpiao.com/rock/poimen-memory
newName: forgejo.riotpiao.com/rock/poimen-memory
newTag: latest
- name: forgejo.riotpiao.com/rock/poimen-workflows
newName: forgejo.riotpiao.com/rock/poimen-workflows
newTag: latest
- name: forgejo.riotpiao.com/rock/poimen-frontend
newName: forgejo.riotpiao.com/rock/poimen-frontend
newTag: latest
+283
View File
@@ -0,0 +1,283 @@
apiVersion: v1
kind: Namespace
metadata:
name: poimen
---
apiVersion: v1
kind: Secret
metadata:
name: poimen-registry
namespace: poimen
type: kubernetes.io/dockercfg
data:
.dockercfg: eyJmb3JnZWpvLnJpb3RwaWFvLmNvbSI6eyJhdXRoIjoiYmFzZTY0LWVuY29kZWQtY3JlZGVudGlhbHMifX0=
---
# Poimen Memory Service
apiVersion: apps/v1
kind: Deployment
metadata:
name: poimen-memory
namespace: poimen
spec:
replicas: 1
selector:
matchLabels:
app: poimen-memory
template:
metadata:
labels:
app: poimen-memory
spec:
imagePullSecrets:
- name: poimen-registry
containers:
- name: memory
image: forgejo.riotpiao.com/rock/poimen-memory:latest
imagePullPolicy: Always
ports:
- containerPort: 8000
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: poimen-db-credentials
key: memory-url
- name: REDIS_URL
valueFrom:
configMapKeyRef:
name: poimen-config
key: redis-url
- name: JWT_SECRET
valueFrom:
secretKeyRef:
name: poimen-secrets
key: jwt-secret
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: 8000
initialDelaySeconds: 5
periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
name: poimen-memory
namespace: poimen
spec:
selector:
app: poimen-memory
ports:
- port: 8000
targetPort: 8000
type: ClusterIP
---
# Poimen Workflows Service
apiVersion: apps/v1
kind: Deployment
metadata:
name: poimen-workflows
namespace: poimen
spec:
replicas: 2
selector:
matchLabels:
app: poimen-workflows
template:
metadata:
labels:
app: poimen-workflows
spec:
imagePullSecrets:
- name: poimen-registry
containers:
- name: workflows-server
image: forgejo.riotpiao.com/rock/poimen-workflows:latest
imagePullPolicy: Always
command: ["/app/workflows", "server"]
ports:
- containerPort: 8080
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: poimen-db-credentials
key: workflows-url
- name: TEMPORAL_HOST
valueFrom:
configMapKeyRef:
name: poimen-config
key: temporal-host
- name: MEMORY_SERVICE_URL
valueFrom:
configMapKeyRef:
name: poimen-config
key: memory-service-url
- name: JWT_SECRET
valueFrom:
secretKeyRef:
name: poimen-secrets
key: jwt-secret
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
- name: workflows-worker
image: forgejo.riotpiao.com/rock/poimen-workflows:latest
imagePullPolicy: Always
command: ["/app/workflows", "worker"]
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: poimen-db-credentials
key: workflows-url
- name: TEMPORAL_HOST
valueFrom:
configMapKeyRef:
name: poimen-config
key: temporal-host
- name: MEMORY_SERVICE_URL
valueFrom:
configMapKeyRef:
name: poimen-config
key: memory-service-url
---
apiVersion: v1
kind: Service
metadata:
name: poimen-workflows
namespace: poimen
spec:
selector:
app: poimen-workflows
ports:
- port: 8080
targetPort: 8080
type: ClusterIP
---
# Poimen Frontend Service
apiVersion: apps/v1
kind: Deployment
metadata:
name: poimen-frontend
namespace: poimen
spec:
replicas: 2
selector:
matchLabels:
app: poimen-frontend
template:
metadata:
labels:
app: poimen-frontend
spec:
imagePullSecrets:
- name: poimen-registry
containers:
- name: frontend
image: forgejo.riotpiao.com/rock/poimen-frontend:latest
imagePullPolicy: Always
ports:
- containerPort: 3000
env:
- name: NEXT_PUBLIC_WORKFLOWS_API
valueFrom:
configMapKeyRef:
name: poimen-config
key: workflows-api-url
- name: NEXT_PUBLIC_MEMORY_API
valueFrom:
configMapKeyRef:
name: poimen-config
key: memory-api-url
- name: NEXT_PUBLIC_AUTH_URL
valueFrom:
configMapKeyRef:
name: poimen-config
key: auth-url
- name: OAUTH_CLIENT_ID
valueFrom:
secretKeyRef:
name: poimen-secrets
key: oauth-client-id
- name: OAUTH_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: poimen-secrets
key: oauth-client-secret
livenessProbe:
httpGet:
path: /
port: 3000
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
path: /
port: 3000
initialDelaySeconds: 5
periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
name: poimen-frontend
namespace: poimen
spec:
selector:
app: poimen-frontend
ports:
- port: 80
targetPort: 3000
type: LoadBalancer
---
# ConfigMap for shared configuration
apiVersion: v1
kind: ConfigMap
metadata:
name: poimen-config
namespace: poimen
data:
temporal-host: "temporal:7233"
redis-url: "redis://redis:6379"
workflows-api-url: "http://poimen-workflows:8080"
memory-api-url: "http://poimen-memory:8000"
auth-url: "https://auth.riotpiao.com"
memory-service-url: "http://poimen-memory:8000"
---
# Secrets placeholder - should be created separately
apiVersion: v1
kind: Secret
metadata:
name: poimen-db-credentials
namespace: poimen
type: Opaque
stringData:
memory-url: "postgresql://user:pass@postgres:5432/poimen_memory"
workflows-url: "postgresql://user:pass@postgres:5432/poimen_workflows"
---
apiVersion: v1
kind: Secret
metadata:
name: poimen-secrets
namespace: poimen
type: Opaque
stringData:
jwt-secret: "your-jwt-secret-here"
oauth-client-id: "your-oauth-client-id"
oauth-client-secret: "your-oauth-client-secret"
+2 -2
View File
@@ -13,8 +13,8 @@ spec:
labels:
app: poimen-worker
annotations:
git-commit: "30644a8e" # ✅ Updated on each push, triggers rolling restart
deployment-date: "2026-09-04"
git-commit: "84b4ca120" # ✅ Updated on each push, triggers rolling restart
deployment-date: "2026-09-05"
spec:
containers:
- name: worker
+192
View File
@@ -0,0 +1,192 @@
-- Poimen Workflows Schema
-- Tables: workflows, workflow_executions, execution_logs, workflow_memory_links
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "vector";
-- Workflows (canvas definitions)
CREATE TABLE IF NOT EXISTS workflows (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
customer_id TEXT NOT NULL,
name TEXT NOT NULL,
description TEXT,
status TEXT NOT NULL CHECK (status IN ('draft', 'active', 'archived')) DEFAULT 'draft',
version INT NOT NULL DEFAULT 1,
-- Canvas data
nodes JSONB NOT NULL DEFAULT '[]'::jsonb, -- WorkflowNode[]
edges JSONB NOT NULL DEFAULT '[]'::jsonb, -- WorkflowEdge[]
-- Metadata
created_by TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
last_executed_at TIMESTAMPTZ,
CONSTRAINT workflow_name_per_customer UNIQUE (customer_id, name)
);
CREATE INDEX idx_workflows_customer ON workflows(customer_id);
CREATE INDEX idx_workflows_status ON workflows(status);
CREATE INDEX idx_workflows_created_at ON workflows(created_at DESC);
-- Workflow executions (runs triggered by user)
CREATE TABLE IF NOT EXISTS workflow_executions (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
workflow_id UUID NOT NULL REFERENCES workflows(id) ON DELETE CASCADE,
customer_id TEXT NOT NULL,
-- Temporal details
temporal_id TEXT NOT NULL UNIQUE, -- Temporal workflow execution ID
status TEXT NOT NULL CHECK (status IN ('pending', 'running', 'success', 'failed', 'cancelled')) DEFAULT 'pending',
-- Input/Output
inputs JSONB NOT NULL,
outputs JSONB,
-- Timing
started_at TIMESTAMPTZ NOT NULL DEFAULT now(),
completed_at TIMESTAMPTZ,
duration_ms INT,
-- Error tracking
error_message TEXT,
error_count INT DEFAULT 0,
CONSTRAINT duration_when_completed CHECK (
(status IN ('success', 'failed') AND completed_at IS NOT NULL) OR
(status IN ('pending', 'running', 'cancelled'))
)
);
CREATE INDEX idx_executions_workflow ON workflow_executions(workflow_id);
CREATE INDEX idx_executions_customer ON workflow_executions(customer_id);
CREATE INDEX idx_executions_status ON workflow_executions(status);
CREATE INDEX idx_executions_temporal_id ON workflow_executions(temporal_id);
CREATE INDEX idx_executions_started_at ON workflow_executions(started_at DESC);
-- Execution logs (detailed activity logs)
CREATE TABLE IF NOT EXISTS execution_logs (
id BIGSERIAL PRIMARY KEY,
execution_id UUID NOT NULL REFERENCES workflow_executions(id) ON DELETE CASCADE,
-- Node/Activity info
node_id TEXT NOT NULL, -- "activity-123" from canvas
activity_name TEXT NOT NULL, -- "CloneRepo", "AnalyzeCode", etc.
-- Log entry
level TEXT NOT NULL CHECK (level IN ('info', 'warn', 'error', 'debug')),
message TEXT NOT NULL,
metadata JSONB, -- Arbitrary structured data (duration, result, etc.)
-- Timing
logged_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT log_order UNIQUE (execution_id, logged_at, id)
);
CREATE INDEX idx_logs_execution ON execution_logs(execution_id);
CREATE INDEX idx_logs_node ON execution_logs(execution_id, node_id);
CREATE INDEX idx_logs_level ON execution_logs(level);
CREATE INDEX idx_logs_logged_at ON execution_logs(logged_at DESC);
-- Memory links (connect executions to memory/lessons learned)
CREATE TABLE IF NOT EXISTS workflow_memory_links (
execution_id UUID NOT NULL REFERENCES workflow_executions(id) ON DELETE CASCADE,
memory_node_sha TEXT NOT NULL, -- SHA256 from memory.memory_node
relationship TEXT NOT NULL CHECK (relationship IN ('generated', 'used', 'learned', 'failed_on')),
-- Context
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
notes TEXT,
PRIMARY KEY (execution_id, memory_node_sha, relationship)
);
CREATE INDEX idx_memory_links_memory_node ON workflow_memory_links(memory_node_sha);
CREATE INDEX idx_memory_links_execution ON workflow_memory_links(execution_id);
-- Activity execution trace (detailed per-activity metrics)
CREATE TABLE IF NOT EXISTS activity_traces (
id BIGSERIAL PRIMARY KEY,
execution_id UUID NOT NULL REFERENCES workflow_executions(id) ON DELETE CASCADE,
node_id TEXT NOT NULL,
-- Activity details
activity_name TEXT NOT NULL,
parameters JSONB NOT NULL,
result JSONB,
-- Timing
started_at TIMESTAMPTZ NOT NULL,
completed_at TIMESTAMPTZ,
duration_ms INT,
-- Retry info
attempt INT DEFAULT 1,
retry_reason TEXT,
-- Status
status TEXT NOT NULL CHECK (status IN ('running', 'success', 'failed', 'skipped')),
error_message TEXT
);
CREATE INDEX idx_traces_execution ON activity_traces(execution_id);
CREATE INDEX idx_traces_activity ON activity_traces(activity_name);
CREATE INDEX idx_traces_status ON activity_traces(status);
CREATE INDEX idx_traces_started_at ON activity_traces(started_at DESC);
-- Workflow stats (materialized for fast dashboard queries)
CREATE TABLE IF NOT EXISTS workflow_stats (
workflow_id UUID PRIMARY KEY REFERENCES workflows(id) ON DELETE CASCADE,
customer_id TEXT NOT NULL,
total_runs INT DEFAULT 0,
successful_runs INT DEFAULT 0,
failed_runs INT DEFAULT 0,
avg_duration_ms NUMERIC,
min_duration_ms INT,
max_duration_ms INT,
last_30d_runs INT DEFAULT 0,
last_30d_success_rate NUMERIC,
updated_at TIMESTAMPTZ DEFAULT now()
);
CREATE INDEX idx_stats_customer ON workflow_stats(customer_id);
-- View: Recent executions with workflow context
CREATE OR REPLACE VIEW v_recent_executions AS
SELECT
we.id,
we.workflow_id,
w.name as workflow_name,
we.customer_id,
we.status,
we.started_at,
we.completed_at,
we.duration_ms,
we.error_message,
(SELECT COUNT(*) FROM execution_logs WHERE execution_id = we.id) as log_count,
(SELECT COUNT(*) FROM activity_traces WHERE execution_id = we.id) as activity_count
FROM workflow_executions we
JOIN workflows w ON we.workflow_id = w.id
ORDER BY we.started_at DESC;
-- View: Execution timeline (for state machine visualization)
CREATE OR REPLACE VIEW v_execution_timeline AS
SELECT
el.execution_id,
el.logged_at,
el.node_id,
el.activity_name,
el.level,
el.message,
at.duration_ms as activity_duration,
at.status as activity_status
FROM execution_logs el
LEFT JOIN activity_traces at ON el.execution_id = at.execution_id
AND el.node_id = at.node_id
ORDER BY el.execution_id, el.logged_at;
+190
View File
@@ -0,0 +1,190 @@
-- Poimen Workflows schema
-- Tables: workflows, workflow_executions, execution_logs, activity_traces, workflow_stats, workflow_memory_links
-- Integrates with temporal workflow orchestrator and memory service
-- Workflows (canvas definitions with JSONB nodes/edges)
CREATE TABLE IF NOT EXISTS workflows (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
customer_id TEXT NOT NULL,
name TEXT NOT NULL,
description TEXT,
status TEXT NOT NULL CHECK (status IN ('draft', 'active', 'archived')) DEFAULT 'draft',
version INT NOT NULL DEFAULT 1,
-- Canvas data (React Flow format)
nodes JSONB NOT NULL DEFAULT '[]'::jsonb, -- WorkflowNode[]
edges JSONB NOT NULL DEFAULT '[]'::jsonb, -- WorkflowEdge[]
-- Metadata
created_by TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
last_executed_at TIMESTAMPTZ,
CONSTRAINT workflow_name_per_customer UNIQUE (customer_id, name)
);
CREATE INDEX idx_workflows_customer ON workflows(customer_id);
CREATE INDEX idx_workflows_status ON workflows(status);
CREATE INDEX idx_workflows_created_at ON workflows(created_at DESC);
-- Workflow executions (runs triggered by user)
CREATE TABLE IF NOT EXISTS workflow_executions (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
workflow_id UUID NOT NULL REFERENCES workflows(id) ON DELETE CASCADE,
customer_id TEXT NOT NULL,
-- Temporal details
temporal_id TEXT NOT NULL UNIQUE, -- Temporal workflow execution ID
status TEXT NOT NULL CHECK (status IN ('pending', 'running', 'success', 'failed', 'cancelled')) DEFAULT 'pending',
-- Input/Output
inputs JSONB NOT NULL,
outputs JSONB,
-- Timing
started_at TIMESTAMPTZ NOT NULL DEFAULT now(),
completed_at TIMESTAMPTZ,
duration_ms INT,
-- Error tracking
error_message TEXT,
error_count INT DEFAULT 0,
CONSTRAINT duration_when_completed CHECK (
(status IN ('success', 'failed') AND completed_at IS NOT NULL) OR
(status IN ('pending', 'running', 'cancelled'))
)
);
CREATE INDEX idx_executions_workflow ON workflow_executions(workflow_id);
CREATE INDEX idx_executions_customer ON workflow_executions(customer_id);
CREATE INDEX idx_executions_status ON workflow_executions(status);
CREATE INDEX idx_executions_temporal_id ON workflow_executions(temporal_id);
CREATE INDEX idx_executions_started_at ON workflow_executions(started_at DESC);
-- Execution logs (detailed activity logs)
CREATE TABLE IF NOT EXISTS execution_logs (
id BIGSERIAL PRIMARY KEY,
execution_id UUID NOT NULL REFERENCES workflow_executions(id) ON DELETE CASCADE,
-- Node/Activity info
node_id TEXT NOT NULL, -- "activity-123" from canvas
activity_name TEXT NOT NULL, -- "CloneRepo", "AnalyzeCode", etc.
-- Log entry
level TEXT NOT NULL CHECK (level IN ('info', 'warn', 'error', 'debug')),
message TEXT NOT NULL,
metadata JSONB, -- Arbitrary structured data (duration, result, etc.)
-- Timing
logged_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT log_order UNIQUE (execution_id, logged_at, id)
);
CREATE INDEX idx_logs_execution ON execution_logs(execution_id);
CREATE INDEX idx_logs_node ON execution_logs(execution_id, node_id);
CREATE INDEX idx_logs_level ON execution_logs(level);
CREATE INDEX idx_logs_logged_at ON execution_logs(logged_at DESC);
-- Activity execution trace (detailed per-activity metrics)
CREATE TABLE IF NOT EXISTS activity_traces (
id BIGSERIAL PRIMARY KEY,
execution_id UUID NOT NULL REFERENCES workflow_executions(id) ON DELETE CASCADE,
node_id TEXT NOT NULL,
-- Activity details
activity_name TEXT NOT NULL,
parameters JSONB NOT NULL,
result JSONB,
-- Timing
started_at TIMESTAMPTZ NOT NULL,
completed_at TIMESTAMPTZ,
duration_ms INT,
-- Retry info
attempt INT DEFAULT 1,
retry_reason TEXT,
-- Status
status TEXT NOT NULL CHECK (status IN ('running', 'success', 'failed', 'skipped')),
error_message TEXT
);
CREATE INDEX idx_traces_execution ON activity_traces(execution_id);
CREATE INDEX idx_traces_activity ON activity_traces(activity_name);
CREATE INDEX idx_traces_status ON activity_traces(status);
CREATE INDEX idx_traces_started_at ON activity_traces(started_at DESC);
-- Workflow stats (materialized for fast dashboard queries)
CREATE TABLE IF NOT EXISTS workflow_stats (
workflow_id UUID PRIMARY KEY REFERENCES workflows(id) ON DELETE CASCADE,
customer_id TEXT NOT NULL,
total_runs INT DEFAULT 0,
successful_runs INT DEFAULT 0,
failed_runs INT DEFAULT 0,
avg_duration_ms NUMERIC,
min_duration_ms INT,
max_duration_ms INT,
last_30d_runs INT DEFAULT 0,
last_30d_success_rate NUMERIC,
updated_at TIMESTAMPTZ DEFAULT now()
);
CREATE INDEX idx_stats_customer ON workflow_stats(customer_id);
-- Memory links (connect executions to memory/lessons learned)
CREATE TABLE IF NOT EXISTS workflow_memory_links (
execution_id UUID NOT NULL REFERENCES workflow_executions(id) ON DELETE CASCADE,
memory_node_sha TEXT NOT NULL REFERENCES memory_node(sha256) ON DELETE CASCADE,
relationship TEXT NOT NULL CHECK (relationship IN ('generated', 'used', 'learned', 'failed_on')),
-- Context
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
notes TEXT,
PRIMARY KEY (execution_id, memory_node_sha, relationship)
);
CREATE INDEX idx_memory_links_memory_node ON workflow_memory_links(memory_node_sha);
CREATE INDEX idx_memory_links_execution ON workflow_memory_links(execution_id);
-- View: Recent executions with workflow context
CREATE OR REPLACE VIEW v_recent_executions AS
SELECT
we.id,
we.workflow_id,
w.name as workflow_name,
we.customer_id,
we.status,
we.started_at,
we.completed_at,
we.duration_ms,
we.error_message,
(SELECT COUNT(*) FROM execution_logs WHERE execution_id = we.id) as log_count,
(SELECT COUNT(*) FROM activity_traces WHERE execution_id = we.id) as activity_count
FROM workflow_executions we
JOIN workflows w ON we.workflow_id = w.id
ORDER BY we.started_at DESC;
-- View: Execution timeline (for state machine visualization)
CREATE OR REPLACE VIEW v_execution_timeline AS
SELECT
el.execution_id,
el.logged_at,
el.node_id,
el.activity_name,
el.level,
el.message,
at.duration_ms as activity_duration,
at.status as activity_status
FROM execution_logs el
LEFT JOIN activity_traces at ON el.execution_id = at.execution_id
AND el.node_id = at.node_id
ORDER BY el.execution_id, el.logged_at;
+55
View File
@@ -0,0 +1,55 @@
-- Workflow Relations Schema
-- Stores semantic relations between workflow canvas nodes with versioning support
-- Table for storing workflow relations with relation wording
CREATE TABLE IF NOT EXISTS workflow_relations (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
workflow_id VARCHAR(255) NOT NULL,
version INTEGER NOT NULL,
source_node_id VARCHAR(255) NOT NULL,
target_node_id VARCHAR(255) NOT NULL,
relation_type VARCHAR(50) NOT NULL, -- data-flow, dependency, conditional, parallel
label TEXT NOT NULL,
relation_wording JSONB NOT NULL, -- {verb, source_output, target_input, connection_type, confidence, semantic_match}
metadata JSONB,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (workflow_id) REFERENCES workflows(id) ON DELETE CASCADE,
UNIQUE(workflow_id, source_node_id, target_node_id, version)
);
-- Table for versioned history of relation changes
CREATE TABLE IF NOT EXISTS workflow_relation_versions (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
workflow_id VARCHAR(255) NOT NULL,
edge_id VARCHAR(255) NOT NULL,
version_num INTEGER NOT NULL,
operation VARCHAR(10) NOT NULL, -- CREATE, UPDATE, DELETE
snapshot JSONB NOT NULL,
changed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
changed_by VARCHAR(255),
fields_changed JSONB, -- ["field1", "field2"]
FOREIGN KEY (workflow_id) REFERENCES workflows(id) ON DELETE CASCADE,
UNIQUE(workflow_id, edge_id, version_num)
);
-- Table for GraphRAG indexing metadata
CREATE TABLE IF NOT EXISTS workflow_rag_index (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
workflow_id VARCHAR(255) NOT NULL UNIQUE,
version INTEGER NOT NULL,
indexed_status VARCHAR(20) NOT NULL DEFAULT 'pending', -- pending, indexed, partial, failed
embedding_model VARCHAR(100),
last_indexed_at TIMESTAMP,
index_metadata JSONB,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (workflow_id) REFERENCES workflows(id) ON DELETE CASCADE
);
-- Indexes for performance
CREATE INDEX idx_workflow_relations_workflow_version ON workflow_relations(workflow_id, version);
CREATE INDEX idx_workflow_relations_type ON workflow_relations(relation_type);
CREATE INDEX idx_workflow_relations_nodes ON workflow_relations(source_node_id, target_node_id);
CREATE INDEX idx_workflow_relation_versions_workflow_edge ON workflow_relation_versions(workflow_id, edge_id);
CREATE INDEX idx_workflow_relation_versions_operation ON workflow_relation_versions(operation);
CREATE INDEX idx_workflow_rag_index_status ON workflow_rag_index(indexed_status);
+535
View File
@@ -0,0 +1,535 @@
package db
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"os"
"time"
_ "github.com/lib/pq"
)
// DB wraps the database connection
type DB struct {
conn *sql.DB
}
// New creates a new database connection to memory-db (K8s CNPG)
// Expected DSN format: postgresql://app:[email protected]:5432/memory?sslmode=disable
func New(dsn string) (*DB, error) {
if dsn == "" {
// Fallback: try to construct from K8s env vars
host := os.Getenv("DATABASE_HOST")
port := os.Getenv("DATABASE_PORT")
name := os.Getenv("DATABASE_NAME")
user := os.Getenv("DATABASE_USER")
password := os.Getenv("DATABASE_PASSWORD")
if host != "" && port != "" && name != "" && user != "" && password != "" {
dsn = fmt.Sprintf("postgresql://%s:%s@%s:%s/%s?sslmode=disable",
user, password, host, port, name)
} else {
return nil, fmt.Errorf("DATABASE_URL or K8s env vars (DATABASE_HOST, DATABASE_PORT, DATABASE_NAME, DATABASE_USER, DATABASE_PASSWORD) required")
}
}
conn, err := sql.Open("postgres", dsn)
if err != nil {
return nil, fmt.Errorf("failed to open database: %w", err)
}
// Test connection
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := conn.PingContext(ctx); err != nil {
return nil, fmt.Errorf("failed to ping database: %w", err)
}
// Set connection pool settings
conn.SetMaxOpenConns(25)
conn.SetMaxIdleConns(5)
conn.SetConnMaxLifetime(5 * time.Minute)
return &DB{conn: conn}, nil
}
// Close closes the database connection
func (db *DB) Close() error {
return db.conn.Close()
}
// SaveWorkflow saves or updates a workflow with canvas
func (db *DB) SaveWorkflow(ctx context.Context, wf *Workflow) error {
query := `
INSERT INTO workflows (id, customer_id, name, description, status, version, nodes, edges, created_by, created_at, updated_at)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)
ON CONFLICT (id) DO UPDATE SET
name = $3,
description = $4,
status = $5,
version = $6,
nodes = $7,
edges = $8,
updated_at = $11
`
_, err := db.conn.ExecContext(ctx, query,
wf.ID,
wf.CustomerID,
wf.Name,
wf.Description,
wf.Status,
wf.Version,
wf.Nodes,
wf.Edges,
wf.CreatedBy,
wf.CreatedAt,
wf.UpdatedAt,
)
return err
}
// SaveCanvasUpdate saves canvas (nodes + edges) for a workflow
func (db *DB) SaveCanvasUpdate(ctx context.Context, workflowID, customerID string, canvas *Canvas) error {
nodesJSON, err := json.Marshal(canvas.Nodes)
if err != nil {
return fmt.Errorf("failed to marshal nodes: %w", err)
}
edgesJSON, err := json.Marshal(canvas.Edges)
if err != nil {
return fmt.Errorf("failed to marshal edges: %w", err)
}
query := `
UPDATE workflows
SET nodes = $1, edges = $2, updated_at = now()
WHERE id = $3 AND customer_id = $4
`
result, err := db.conn.ExecContext(ctx, query, nodesJSON, edgesJSON, workflowID, customerID)
if err != nil {
return fmt.Errorf("failed to update canvas: %w", err)
}
rows, err := result.RowsAffected()
if err != nil {
return err
}
if rows == 0 {
return fmt.Errorf("workflow not found: %s", workflowID)
}
return nil
}
// FetchWorkflow retrieves a workflow by ID
func (db *DB) FetchWorkflow(ctx context.Context, workflowID, customerID string) (*Workflow, error) {
query := `
SELECT id, customer_id, name, description, status, version, nodes, edges, created_by, created_at, updated_at, last_executed_at
FROM workflows
WHERE id = $1 AND customer_id = $2
`
wf := &Workflow{}
err := db.conn.QueryRowContext(ctx, query, workflowID, customerID).Scan(
&wf.ID,
&wf.CustomerID,
&wf.Name,
&wf.Description,
&wf.Status,
&wf.Version,
&wf.Nodes,
&wf.Edges,
&wf.CreatedBy,
&wf.CreatedAt,
&wf.UpdatedAt,
&wf.LastExecutedAt,
)
if err != nil {
if err == sql.ErrNoRows {
return nil, fmt.Errorf("workflow not found: %s", workflowID)
}
return nil, fmt.Errorf("failed to fetch workflow: %w", err)
}
return wf, nil
}
// FetchCanvas retrieves canvas (nodes + edges) for a workflow
func (db *DB) FetchCanvas(ctx context.Context, workflowID, customerID string) (*Canvas, error) {
query := `
SELECT nodes, edges
FROM workflows
WHERE id = $1 AND customer_id = $2
`
var nodesJSON, edgesJSON []byte
err := db.conn.QueryRowContext(ctx, query, workflowID, customerID).Scan(&nodesJSON, &edgesJSON)
if err != nil {
if err == sql.ErrNoRows {
return nil, fmt.Errorf("workflow not found: %s", workflowID)
}
return nil, fmt.Errorf("failed to fetch canvas: %w", err)
}
var nodes []WorkflowNode
var edges []WorkflowEdge
if err := json.Unmarshal(nodesJSON, &nodes); err != nil {
return nil, fmt.Errorf("failed to unmarshal nodes: %w", err)
}
if err := json.Unmarshal(edgesJSON, &edges); err != nil {
return nil, fmt.Errorf("failed to unmarshal edges: %w", err)
}
return &Canvas{Nodes: nodes, Edges: edges}, nil
}
// ListWorkflows retrieves all workflows for a customer
func (db *DB) ListWorkflows(ctx context.Context, customerID string, limit, offset int) ([]Workflow, error) {
query := `
SELECT id, customer_id, name, description, status, version, nodes, edges, created_by, created_at, updated_at, last_executed_at
FROM workflows
WHERE customer_id = $1
ORDER BY created_at DESC
LIMIT $2 OFFSET $3
`
rows, err := db.conn.QueryContext(ctx, query, customerID, limit, offset)
if err != nil {
return nil, fmt.Errorf("failed to list workflows: %w", err)
}
defer rows.Close()
var workflows []Workflow
for rows.Next() {
wf := Workflow{}
err := rows.Scan(
&wf.ID,
&wf.CustomerID,
&wf.Name,
&wf.Description,
&wf.Status,
&wf.Version,
&wf.Nodes,
&wf.Edges,
&wf.CreatedBy,
&wf.CreatedAt,
&wf.UpdatedAt,
&wf.LastExecutedAt,
)
if err != nil {
return nil, fmt.Errorf("failed to scan workflow: %w", err)
}
workflows = append(workflows, wf)
}
return workflows, rows.Err()
}
// DeleteWorkflow deletes a workflow
func (db *DB) DeleteWorkflow(ctx context.Context, workflowID, customerID string) error {
query := `
DELETE FROM workflows
WHERE id = $1 AND customer_id = $2
`
result, err := db.conn.ExecContext(ctx, query, workflowID, customerID)
if err != nil {
return fmt.Errorf("failed to delete workflow: %w", err)
}
rows, err := result.RowsAffected()
if err != nil {
return err
}
if rows == 0 {
return fmt.Errorf("workflow not found: %s", workflowID)
}
return nil
}
// SaveExecution saves a workflow execution record
func (db *DB) SaveExecution(ctx context.Context, exec *WorkflowExecution) error {
query := `
INSERT INTO workflow_executions (id, workflow_id, customer_id, temporal_id, status, inputs, outputs, started_at, completed_at, duration_ms, error_message, error_count)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)
ON CONFLICT (id) DO UPDATE SET
status = $5,
outputs = $7,
completed_at = $9,
duration_ms = $10,
error_message = $11,
error_count = $12
`
_, err := db.conn.ExecContext(ctx, query,
exec.ID,
exec.WorkflowID,
exec.CustomerID,
exec.TemporalID,
exec.Status,
exec.Inputs,
exec.Outputs,
exec.StartedAt,
exec.CompletedAt,
exec.DurationMs,
exec.ErrorMessage,
exec.ErrorCount,
)
return err
}
// FetchExecution retrieves a workflow execution
func (db *DB) FetchExecution(ctx context.Context, executionID string) (*WorkflowExecution, error) {
query := `
SELECT id, workflow_id, customer_id, temporal_id, status, inputs, outputs, started_at, completed_at, duration_ms, error_message, error_count
FROM workflow_executions
WHERE id = $1
`
exec := &WorkflowExecution{}
err := db.conn.QueryRowContext(ctx, query, executionID).Scan(
&exec.ID,
&exec.WorkflowID,
&exec.CustomerID,
&exec.TemporalID,
&exec.Status,
&exec.Inputs,
&exec.Outputs,
&exec.StartedAt,
&exec.CompletedAt,
&exec.DurationMs,
&exec.ErrorMessage,
&exec.ErrorCount,
)
if err != nil {
if err == sql.ErrNoRows {
return nil, fmt.Errorf("execution not found: %s", executionID)
}
return nil, fmt.Errorf("failed to fetch execution: %w", err)
}
return exec, nil
}
// SaveExecutionLog saves an activity log entry
func (db *DB) SaveExecutionLog(ctx context.Context, log *ExecutionLog) error {
query := `
INSERT INTO execution_logs (execution_id, node_id, activity_name, level, message, metadata, logged_at)
VALUES ($1, $2, $3, $4, $5, $6, $7)
`
_, err := db.conn.ExecContext(ctx, query,
log.ExecutionID,
log.NodeID,
log.ActivityName,
log.Level,
log.Message,
log.Metadata,
log.LoggedAt,
)
return err
}
// FetchExecutionLogs retrieves all logs for an execution
func (db *DB) FetchExecutionLogs(ctx context.Context, executionID string) ([]ExecutionLog, error) {
query := `
SELECT id, execution_id, node_id, activity_name, level, message, metadata, logged_at
FROM execution_logs
WHERE execution_id = $1
ORDER BY logged_at ASC
`
rows, err := db.conn.QueryContext(ctx, query, executionID)
if err != nil {
return nil, fmt.Errorf("failed to fetch execution logs: %w", err)
}
defer rows.Close()
var logs []ExecutionLog
for rows.Next() {
log := ExecutionLog{}
err := rows.Scan(
&log.ID,
&log.ExecutionID,
&log.NodeID,
&log.ActivityName,
&log.Level,
&log.Message,
&log.Metadata,
&log.LoggedAt,
)
if err != nil {
return nil, fmt.Errorf("failed to scan log: %w", err)
}
logs = append(logs, log)
}
return logs, rows.Err()
}
// SaveActivityTrace saves per-activity execution trace
func (db *DB) SaveActivityTrace(ctx context.Context, trace *ActivityTrace) error {
query := `
INSERT INTO activity_traces (execution_id, node_id, activity_name, parameters, result, started_at, completed_at, duration_ms, attempt, retry_reason, status, error_message)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)
ON CONFLICT (id) DO UPDATE SET
status = $11,
result = $5,
completed_at = $7,
duration_ms = $8,
error_message = $12
`
_, err := db.conn.ExecContext(ctx, query,
trace.ExecutionID,
trace.NodeID,
trace.ActivityName,
trace.Parameters,
trace.Result,
trace.StartedAt,
trace.CompletedAt,
trace.DurationMs,
trace.Attempt,
trace.RetryReason,
trace.Status,
trace.ErrorMessage,
)
return err
}
// FetchActivityTraces retrieves all activity traces for an execution
func (db *DB) FetchActivityTraces(ctx context.Context, executionID string) ([]ActivityTrace, error) {
query := `
SELECT id, execution_id, node_id, activity_name, parameters, result, started_at, completed_at, duration_ms, attempt, retry_reason, status, error_message
FROM activity_traces
WHERE execution_id = $1
ORDER BY started_at ASC
`
rows, err := db.conn.QueryContext(ctx, query, executionID)
if err != nil {
return nil, fmt.Errorf("failed to fetch activity traces: %w", err)
}
defer rows.Close()
var traces []ActivityTrace
for rows.Next() {
trace := ActivityTrace{}
err := rows.Scan(
&trace.ID,
&trace.ExecutionID,
&trace.NodeID,
&trace.ActivityName,
&trace.Parameters,
&trace.Result,
&trace.StartedAt,
&trace.CompletedAt,
&trace.DurationMs,
&trace.Attempt,
&trace.RetryReason,
&trace.Status,
&trace.ErrorMessage,
)
if err != nil {
return nil, fmt.Errorf("failed to scan trace: %w", err)
}
traces = append(traces, trace)
}
return traces, rows.Err()
}
// GetWorkflowRelations retrieves all relations for a workflow version
func (db *DB) GetWorkflowRelations(ctx context.Context, workflowID string, version int) ([]WorkflowRelation, error) {
var relations []WorkflowRelation
query := `
SELECT id, workflow_id, version, source_node_id, target_node_id,
relation_type, label, relation_wording, metadata, created_at
FROM workflow_relations
WHERE workflow_id = $1 AND version = $2
ORDER BY created_at DESC
`
rows, err := db.conn.QueryContext(ctx, query, workflowID, version)
if err != nil {
return nil, fmt.Errorf("failed to query relations: %w", err)
}
defer rows.Close()
for rows.Next() {
var rel WorkflowRelation
if err := rows.Scan(
&rel.ID,
&rel.WorkflowID,
&rel.Version,
&rel.SourceNodeID,
&rel.TargetNodeID,
&rel.RelationType,
&rel.Label,
&rel.RelationWording,
&rel.Metadata,
&rel.CreatedAt,
); err != nil {
return nil, fmt.Errorf("failed to scan relation: %w", err)
}
relations = append(relations, rel)
}
return relations, rows.Err()
}
// GetRelationVersions retrieves version history for a specific relation
func (db *DB) GetRelationVersions(ctx context.Context, workflowID string, edgeID string) ([]WorkflowRelationVersion, error) {
var versions []WorkflowRelationVersion
query := `
SELECT id, workflow_id, edge_id, version_num, operation, snapshot,
changed_at, changed_by, fields_changed
FROM workflow_relation_versions
WHERE workflow_id = $1 AND edge_id = $2
ORDER BY version_num ASC
`
rows, err := db.conn.QueryContext(ctx, query, workflowID, edgeID)
if err != nil {
return nil, fmt.Errorf("failed to query relation versions: %w", err)
}
defer rows.Close()
for rows.Next() {
var v WorkflowRelationVersion
if err := rows.Scan(
&v.ID,
&v.WorkflowID,
&v.EdgeID,
&v.VersionNum,
&v.Operation,
&v.Snapshot,
&v.ChangedAt,
&v.ChangedBy,
&v.FieldsChanged,
); err != nil {
return nil, fmt.Errorf("failed to scan version: %w", err)
}
versions = append(versions, v)
}
return versions, rows.Err()
}
+140
View File
@@ -0,0 +1,140 @@
package db
import (
"time"
)
// WorkflowNode represents a React Flow node in the canvas
type WorkflowNode struct {
ID string `json:"id"`
Label string `json:"label"`
Type string `json:"type"` // "activity"
Position map[string]interface{} `json:"position"`
Data map[string]interface{} `json:"data"`
}
// WorkflowEdge represents a React Flow edge in the canvas
type WorkflowEdge struct {
ID string `json:"id"`
Source string `json:"source"`
Target string `json:"target"`
Data map[string]interface{} `json:"data"`
}
// Canvas represents the full React Flow canvas (nodes + edges)
type Canvas struct {
Nodes []WorkflowNode `json:"nodes"`
Edges []WorkflowEdge `json:"edges"`
}
// Workflow represents a workflow definition in the database
type Workflow struct {
ID string `db:"id"`
CustomerID string `db:"customer_id"`
Name string `db:"name"`
Description string `db:"description"`
Status string `db:"status"` // "draft", "active", "archived"
Version int `db:"version"`
Nodes []byte `db:"nodes"` // JSONB stored as []byte
Edges []byte `db:"edges"` // JSONB stored as []byte
CreatedBy string `db:"created_by"`
CreatedAt time.Time `db:"created_at"`
UpdatedAt time.Time `db:"updated_at"`
LastExecutedAt *time.Time `db:"last_executed_at"`
}
// WorkflowExecution represents a workflow execution run
type WorkflowExecution struct {
ID string `db:"id"`
WorkflowID string `db:"workflow_id"`
CustomerID string `db:"customer_id"`
TemporalID string `db:"temporal_id"` // Temporal execution ID
Status string `db:"status"` // "pending", "running", "success", "failed", "cancelled"
Inputs []byte `db:"inputs"` // JSONB
Outputs []byte `db:"outputs"` // JSONB
StartedAt time.Time `db:"started_at"`
CompletedAt *time.Time `db:"completed_at"`
DurationMs *int `db:"duration_ms"`
ErrorMessage string `db:"error_message"`
ErrorCount int `db:"error_count"`
}
// ExecutionLog represents a detailed activity log entry
type ExecutionLog struct {
ID int64 `db:"id"`
ExecutionID string `db:"execution_id"`
NodeID string `db:"node_id"` // From canvas node ID
ActivityName string `db:"activity_name"` // "CloneRepo", "AnalyzeCode", etc
Level string `db:"level"` // "info", "warn", "error", "debug"
Message string `db:"message"`
Metadata []byte `db:"metadata"` // JSONB
LoggedAt time.Time `db:"logged_at"`
}
// ActivityTrace represents per-activity execution metrics
type ActivityTrace struct {
ID int64 `db:"id"`
ExecutionID string `db:"execution_id"`
NodeID string `db:"node_id"`
ActivityName string `db:"activity_name"`
Parameters []byte `db:"parameters"` // JSONB
Result []byte `db:"result"` // JSONB
StartedAt time.Time `db:"started_at"`
CompletedAt *time.Time `db:"completed_at"`
DurationMs *int `db:"duration_ms"`
Attempt int `db:"attempt"`
RetryReason string `db:"retry_reason"`
Status string `db:"status"` // "running", "success", "failed", "skipped"
ErrorMessage string `db:"error_message"`
}
// WorkflowStats represents aggregated workflow metrics
type WorkflowStats struct {
WorkflowID string `db:"workflow_id"`
CustomerID string `db:"customer_id"`
TotalRuns int `db:"total_runs"`
SuccessfulRuns int `db:"successful_runs"`
FailedRuns int `db:"failed_runs"`
AvgDurationMs float64 `db:"avg_duration_ms"`
MinDurationMs *int `db:"min_duration_ms"`
MaxDurationMs *int `db:"max_duration_ms"`
Last30dRuns int `db:"last_30d_runs"`
Last30dSuccessRate float64 `db:"last_30d_success_rate"`
UpdatedAt time.Time `db:"updated_at"`
}
// WorkflowMemoryLink represents a connection between execution and memory nodes
type WorkflowMemoryLink struct {
ExecutionID string `db:"execution_id"`
MemoryNodeSha string `db:"memory_node_sha"`
Relationship string `db:"relationship"` // "generated", "used", "learned", "failed_on"
CreatedAt time.Time `db:"created_at"`
Notes string `db:"notes"`
}
// WorkflowRelation represents a semantic relation between two canvas nodes
type WorkflowRelation struct {
ID string `db:"id"`
WorkflowID string `db:"workflow_id"`
Version int `db:"version"`
SourceNodeID string `db:"source_node_id"`
TargetNodeID string `db:"target_node_id"`
RelationType string `db:"relation_type"` // "data-flow", "dependency", "conditional"
Label string `db:"label"` // Human-readable relation description
RelationWording []byte `db:"relation_wording"` // JSONB with verb, outputs, inputs, confidence
Metadata []byte `db:"metadata"` // JSONB for extensibility
CreatedAt time.Time `db:"created_at"`
}
// WorkflowRelationVersion represents versioned history of relation changes
type WorkflowRelationVersion struct {
ID string `db:"id"`
WorkflowID string `db:"workflow_id"`
EdgeID string `db:"edge_id"`
VersionNum int `db:"version_num"`
Operation string `db:"operation"` // "CREATE", "UPDATE", "DELETE"
Snapshot []byte `db:"snapshot"` // JSONB full state at this version
ChangedAt time.Time `db:"changed_at"`
ChangedBy string `db:"changed_by"`
FieldsChanged []byte `db:"fields_changed"` // JSONB array of changed field names
}
+106
View File
@@ -0,0 +1,106 @@
package statemachine
import (
"context"
"time"
"go.temporal.io/sdk/workflow"
"github.com/rockliang/poimen/workflows/action"
"github.com/rockliang/poimen/workflows/pkg/db"
)
type WorkflowGraphQueryInput struct {
WorkflowID string `json:"workflow_id"`
Query string `json:"query"`
SearchType string `json:"search_type"`
RelationType string `json:"relation_type"`
Version int `json:"version"`
ConfidenceFloor float64 `json:"confidence_floor"`
TopK int `json:"top_k"`
FindPaths bool `json:"find_paths"`
TargetNodeID string `json:"target_node_id"`
MaxPathDepth int `json:"max_path_depth"`
RankingProfile string `json:"ranking_profile"`
IncludeReasoning bool `json:"include_reasoning"`
}
type WorkflowGraphQueryOutput struct {
WorkflowID string `json:"workflow_id"`
Query string `json:"query"`
Version int `json:"version"`
ExecutionTimeMs int64 `json:"execution_time_ms"`
Results []action.EdgeWithWording `json:"results"`
Paths []QueryPath `json:"paths"`
TotalCount int `json:"total_count"`
HasMore bool `json:"has_more"`
RankingProfile string `json:"ranking_profile"`
}
type QueryPath struct {
SourceID string `json:"source_id"`
TargetID string `json:"target_id"`
Distance int `json:"distance"`
PathCount int `json:"path_count"`
NodeIDs []string `json:"node_ids"`
Confidence float64 `json:"total_confidence"`
}
func WorkflowGraphQuery(ctx workflow.Context, input WorkflowGraphQueryInput) (WorkflowGraphQueryOutput, error) {
startTime := time.Now()
output := WorkflowGraphQueryOutput{
WorkflowID: input.WorkflowID,
Query: input.Query,
Version: input.Version,
RankingProfile: input.RankingProfile,
Results: []action.EdgeWithWording{},
Paths: []QueryPath{},
}
opts := workflow.ActivityOptions{
StartToCloseTimeout: 120 * time.Second,
RetryPolicy: &workflow.RetryPolicy{
InitialInterval: 2 * time.Second,
BackoffCoefficient: 2.0,
MaxInterval: 10 * time.Second,
MaxAttempts: 3,
},
}
ctx = workflow.WithActivityOptions(ctx, opts)
// Fetch canvas + relations
var canvasData action.CanvasWithRelationsData
err := workflow.ExecuteActivity(ctx, action.FetchCanvasRelationsActivity,
action.FetchCanvasRelationsInput{
WorkflowID: input.WorkflowID,
Version: input.Version,
},
).Get(ctx, &canvasData)
if err != nil {
return output, err
}
// Query Memory System via unified endpoint
var graphResults action.GraphRAGQueryOutput
err = workflow.ExecuteActivity(ctx, action.QueryGraphRAGActivity,
action.GraphRAGQueryInput{
WorkflowID: input.WorkflowID,
Query: input.Query,
SearchType: input.SearchType,
RelationType: input.RelationType,
ConfidenceFloor: input.ConfidenceFloor,
TopK: input.TopK,
RankingProfile: input.RankingProfile,
Canvas: canvasData,
},
).Get(ctx, &graphResults)
if err != nil {
return output, err
}
output.Results = graphResults.Edges
output.TotalCount = graphResults.TotalCount
output.HasMore = graphResults.HasMore
output.ExecutionTimeMs = time.Since(startTime).Milliseconds()
return output, nil
}