test: real integration tests calling actual service operations
REAL tests that call actual services, not fake routing checks: - Get real JWT from Authentik (client_credentials flow) - Call SQS list-queues and send-message - Call MinIO/S3 list-objects with JWT - Call Authentik API with JWT - Call Memory query operations - Call Temporal (gRPC not yet implemented) Tests gracefully skip if services unreachable (502/504). Tests warn if operations partially integrated (e.g., MinIO JWT not validated). Tests document current JWT integration state vs. what's still TODO. Configuration: GATEWAY_URL=https://api.riotpiao.com AUTHENTIK_CLIENT_ID=<from OAuth2 provider> AUTHENTIK_CLIENT_SECRET=<from OAuth2 provider> TEST_TIMEOUT=30 Run: ./scripts/test-integration.sh Documentation: - INTEGRATION_TESTS.md lists what's working vs. TODO - Phase 3: Implement actual JWT validation in services - Phase 9: Add gRPC proxying for Temporal
This commit is contained in:
Admin Bot
+108
-91
@@ -1,122 +1,139 @@
|
||||
# Integration Tests
|
||||
# Real Integration Tests
|
||||
|
||||
Integration tests call the real deployed gateway to verify X-Service routing works end-to-end.
|
||||
These tests call actual services through the gateway and validate real operations.
|
||||
|
||||
## Quick Start
|
||||
## Current State
|
||||
|
||||
### Local Test (requires running gateway)
|
||||
⚠️ **Most services NOT YET wired for JWT validation** (per homelab/project-usage/jwt-auth-rollout.md):
|
||||
|
||||
```bash
|
||||
# Terminal 1: Start the gateway
|
||||
CONFIG_PATH=k8s/configmap.yaml go run ./cmd/gateway
|
||||
| Service | Status | Notes |
|
||||
|---------|--------|-------|
|
||||
| **SQS (kmsvc)** | ❌ Code unverified | Placeholders exist, JWT validation not tested |
|
||||
| **MinIO/S3** | ⚠️ Partial | Has OIDC config, JWT forwarding works, actual validation not load-tested |
|
||||
| **Memory (Poimen)** | ❌ Not implemented | Uses static API key internally, no JWT support |
|
||||
| **Authentik (IAM)** | ✅ Works | Is the auth server, validates JWTs it issued |
|
||||
| **Temporal** | ❌ Not configured | Has native JWT support but not wired yet, gRPC only |
|
||||
|
||||
# Terminal 2: Run tests
|
||||
./scripts/test-integration.sh
|
||||
```
|
||||
## Running Tests
|
||||
|
||||
### Cluster Test (production gateway)
|
||||
|
||||
```bash
|
||||
GATEWAY_URL=https://api.riotpiao.com ./scripts/test-integration.sh
|
||||
```
|
||||
|
||||
### Canary Deployment (scale to 1, test, scale back)
|
||||
|
||||
```bash
|
||||
./scripts/test-canary.sh
|
||||
|
||||
# Or with custom settings:
|
||||
NAMESPACE=api DEPLOYMENT=api-gateway REPLICAS=3 ./scripts/test-canary.sh
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Create `.dev.test.local` (gitignored) with:
|
||||
|
||||
```bash
|
||||
GATEWAY_URL=https://api.riotpiao.com
|
||||
TEST_JWT_TOKEN=eyJ... # Real JWT from Authentik
|
||||
SKIP_AUTH_TESTS=false
|
||||
TEST_TIMEOUT=30
|
||||
```
|
||||
|
||||
Or set env vars directly:
|
||||
### Get a Real JWT First
|
||||
|
||||
```bash
|
||||
# Get client_credentials from Authentik admin
|
||||
# Set these:
|
||||
export AUTHENTIK_URL=https://authentik.riotpiao.com
|
||||
export AUTHENTIK_CLIENT_ID=<from Authentik OAuth2 Provider>
|
||||
export AUTHENTIK_CLIENT_SECRET=<from Authentik OAuth2 Provider>
|
||||
export GATEWAY_URL=https://api.riotpiao.com
|
||||
export TEST_JWT_TOKEN=eyJ...
|
||||
export SKIP_AUTH_TESTS=false
|
||||
go test -tags integration -v ./internal/serviceadapter
|
||||
|
||||
# Run tests with real JWT
|
||||
go test -tags integration -v ./internal/serviceadapter -run TestRealIntegration
|
||||
```
|
||||
|
||||
## Test Matrix
|
||||
### Without JWT (services in local testing)
|
||||
|
||||
| Test | Type | Expected | Notes |
|
||||
|------|------|----------|-------|
|
||||
| Health check | GET /healthz | 200 OK | Always works |
|
||||
| SQS list-queues | GET X-Service: sqs | 200 or 502 | 502 if service unreachable |
|
||||
| S3 list-objects | GET X-Service: s3 | 200 or 502 | 502 if service unreachable |
|
||||
| Memory query | POST X-Service: memory | 200 or 502 | 502 if service unreachable |
|
||||
| Service not found | GET X-Service: nonexistent | 404 | Routing error |
|
||||
| Resource not found | GET X-Service: sqs X-Resource: invalid | 404 | Resource error |
|
||||
| IAM with JWT | GET X-Service: iam + Bearer token | 200 or 502 | Requires valid JWT |
|
||||
| Missing X-Service | GET (no header) | 404 | Routed to default handler |
|
||||
|
||||
## Canary Deployment Flow
|
||||
|
||||
```
|
||||
Current: 3/3 replicas running
|
||||
↓
|
||||
scale → 1/3 replicas
|
||||
↓
|
||||
wait for pod ready
|
||||
↓
|
||||
run integration tests
|
||||
├─ PASS → scale → 3/3 replicas ✅
|
||||
└─ FAIL → keep 1/3 for debugging ❌
|
||||
```bash
|
||||
GATEWAY_URL=http://localhost:8080 \
|
||||
go test -tags integration -v ./internal/serviceadapter -run TestRealIntegration
|
||||
```
|
||||
|
||||
## Running in CI
|
||||
## What Gets Tested
|
||||
|
||||
Add to `.gitea/workflows/ci.yaml`:
|
||||
### ✅ Working (Gateway Routing)
|
||||
- Request routing via X-Service header
|
||||
- Path rewriting via X-Resource
|
||||
- Authorization header pass-through
|
||||
- Proper error responses (404, 400)
|
||||
|
||||
### ⚠️ Partially Working (Service Integration)
|
||||
- **MinIO/S3**: Receives JWT via Authorization header (may not validate it yet)
|
||||
- **Authentik**: Receives requests properly
|
||||
- **SQS**: Receives requests but JWT validation unverified
|
||||
- **Memory**: Receives requests (no JWT expected)
|
||||
|
||||
### ❌ Not Yet Implemented
|
||||
- SQS JWT validation (code in kmsvc not tested)
|
||||
- MinIO JWT validation (not load-tested per rollout doc)
|
||||
- Memory JWT validation (would need code change in poimen)
|
||||
- Temporal gRPC forwarding (requires grpcproxy, Phase 9)
|
||||
- Temporal JWT validation (not configured yet)
|
||||
|
||||
## Test Output Example
|
||||
|
||||
```yaml
|
||||
- name: Integration Tests
|
||||
run: |
|
||||
GATEWAY_URL=https://api.riotpiao.com \
|
||||
SKIP_AUTH_TESTS=true \
|
||||
TEST_TIMEOUT=30 \
|
||||
go test -tags integration -v ./internal/serviceadapter
|
||||
```
|
||||
=== RUN TestRealIntegration
|
||||
=== RUN TestRealIntegration/get_JWT_from_Authentik
|
||||
real_integration_test.go:XX: ✅ Got JWT token (first 50 chars): eyJ0eXAiOiJKV1QiLCJhbGc...
|
||||
--- PASS: TestRealIntegration/get_JWT_from_Authentik (0.54s)
|
||||
|
||||
=== RUN TestRealIntegration/SQS:_list-queues
|
||||
real_integration_test.go:XX: ⚠️ SQS backend unreachable (502/504)
|
||||
--- SKIP: TestRealIntegration/SQS:_list-queues (not reachable from test env)
|
||||
|
||||
=== RUN TestRealIntegration/S3:_list-objects_with_JWT
|
||||
real_integration_test.go:XX: Using JWT token with MinIO
|
||||
real_integration_test.go:XX: Response: <?xml version="1.0" encoding="UTF-8"?>...
|
||||
real_integration_test.go:XX: ✅ MinIO list-objects returned 200
|
||||
--- PASS: TestRealIntegration/S3:_list-objects_with_JWT (0.23s)
|
||||
|
||||
=== RUN TestRealIntegration/IAM:_get_user_info_with_JWT
|
||||
real_integration_test.go:XX: ✅ Authentik returned 200
|
||||
--- PASS: TestRealIntegration/IAM:_get_user_info_with_JWT (0.19s)
|
||||
```
|
||||
|
||||
## What Needs to be Done Next
|
||||
|
||||
### Phase 3 (Auth Integration)
|
||||
- [ ] Test SQS JWT validation actually works
|
||||
- [ ] Load-test MinIO JWT validation
|
||||
- [ ] Add JWT validation to poimen-memory
|
||||
- [ ] Configure Temporal JWT validation
|
||||
|
||||
### Phase 9 (gRPC)
|
||||
- [ ] Add grpcproxy for Temporal forwarding
|
||||
- [ ] Test Temporal workflow operations end-to-end
|
||||
|
||||
## Debugging Failed Tests
|
||||
|
||||
If a test fails:
|
||||
|
||||
1. **Check pod logs:**
|
||||
1. **Check if service is reachable:**
|
||||
```bash
|
||||
# SQS
|
||||
kubectl -n sqs port-forward svc/management-service 9090:9090
|
||||
curl http://localhost:9090/sqs/queues
|
||||
|
||||
# MinIO
|
||||
kubectl -n storage port-forward svc/minio 9000:80
|
||||
curl http://localhost:9000
|
||||
|
||||
# Authentik
|
||||
curl https://authentik.riotpiao.com/api/v3/roles/ \
|
||||
-H "Authorization: Bearer $JWT_TOKEN"
|
||||
```
|
||||
|
||||
2. **Check if JWT is valid:**
|
||||
```bash
|
||||
# Decode JWT (copy to jwt.io or use jq)
|
||||
echo "$JWT_TOKEN" | cut -d. -f2 | base64 -d | jq .
|
||||
```
|
||||
|
||||
3. **Check gateway routing:**
|
||||
```bash
|
||||
kubectl -n api logs -l app=api-gateway --tail=50
|
||||
```
|
||||
|
||||
2. **Check service availability:**
|
||||
4. **Test gateway directly:**
|
||||
```bash
|
||||
kubectl get svc -A | grep -E "sqs|minio|authentik|poimen"
|
||||
```
|
||||
|
||||
3. **Test service directly:**
|
||||
```bash
|
||||
kubectl -n sqs port-forward svc/management-service 9090:9090
|
||||
curl http://localhost:9090/sqs/queues
|
||||
```
|
||||
|
||||
4. **Check ConfigMap:**
|
||||
```bash
|
||||
kubectl get configmap api-gateway-config -n api -o yaml | grep -A50 "adapters:"
|
||||
curl -v https://api.riotpiao.com/ \
|
||||
-H "X-Service: s3" \
|
||||
-H "X-Resource: list-objects" \
|
||||
-H "Authorization: Bearer $JWT_TOKEN"
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Auth tests are skipped by default (`SKIP_AUTH_TESTS=true`)
|
||||
- To test with JWT, set `TEST_JWT_TOKEN` and `SKIP_AUTH_TESTS=false`
|
||||
- Services in different namespaces may not be reachable from the gateway (NetworkPolicy)
|
||||
- Canary tests expect `/healthz` endpoint to be available
|
||||
- Tests gracefully skip if services are unreachable (502/504)
|
||||
- Tests report ⚠️ warnings for operations that may not be fully integrated
|
||||
- Full JWT validation is Phase 3 work, not Phase 8
|
||||
- gRPC support is Phase 9 work
|
||||
|
||||
Reference in New Issue
Block a user