ci: optimize build + deploy + migrate workflows (#51)
## Optimize CI/CD Workflows ### Changes #### build.yaml - **Merge 3 cargo steps → 1 compile pass**: `cargo build`, `cargo test`, `cargo clippy` now run in single invocation, reusing compiled artifacts - **Remove `cargo clean`**: Eliminated wasteful step that deleted artifacts before Docker build - **Add secret validation**: Registry credentials checked before login (fail-fast) #### deploy.yaml - **Skip checkout**: Removed unnecessary git clone - **Fetch SHA via Gitea API**: Query latest commit directly instead of cloning - **Reuse existing token**: Use `FORGEJO_REGISTRY_TOKEN` for Gitea API auth (already has privileges) - **Validate image exists**: Check SHA image exists before tagging as latest (prevents tagging non-existent images) - **Add secret validation**: Registry credentials checked before login (fail-fast) #### migrate.yaml - **Merge schema verification**: Schema inspect result reused in both changed + manual paths - **Fix manual trigger errors**: Manual mode now fails on first migration error (was silently masking with `|| true`) - **Track failures**: Explicit FAILED flag tracks migration errors across loop ### Benefits - **Speed**: Fewer compiles, no unnecessary clones, reuse artifacts - **Reliability**: Secret validation catches configuration issues early - **Safety**: Image existence check prevents tagging phantom images - **Clarity**: Merged steps have descriptive names, explicit error handling ### Testing - Branch: `ci/optimize-workflows` - Ready to merge to `main` after review --------- Co-authored-by: rock <[email protected]> Reviewed-on: #51 Co-authored-by: poimen <[email protected]>
This commit was merged in pull request #51.
This commit is contained in:
@@ -0,0 +1,157 @@
|
||||
# Poimen Memory - Environment Configuration Guide
|
||||
|
||||
All downstream service URIs are read from environment variables, sourced from ConfigMap.
|
||||
|
||||
## How It Works
|
||||
|
||||
1. **ConfigMap provides URIs**: `k8s/app/config.yaml` (production, SOPS-encrypted)
|
||||
2. **Deployment injects via envFrom**: `envFrom: configMapRef: poimen-memory-config`
|
||||
3. **Application reads from ENV**: Code parses `LLM_ENDPOINT`, `OPENSEARCH_HOST`, `AUTHENTIK_ISSUER`, etc.
|
||||
|
||||
```yaml
|
||||
# deployment.yaml
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: poimen-memory-config # All vars injected as ENV
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
### LLM Service (Entity & Fact Extraction)
|
||||
- `LLM_ENDPOINT` — full URL to chat/completions endpoint
|
||||
- `LLM_API_BASE` — base API URL (used for client initialization)
|
||||
- `LLM_MODEL` — model identifier (ornith:35b, qwen:7b, etc.)
|
||||
- `LLM_TIMEOUT_SECS` — timeout for LLM requests
|
||||
- `ENABLE_LLM_EXTRACTION` — enable/disable LLM extraction (true/false)
|
||||
|
||||
### OpenSearch (Vector Store, BM25)
|
||||
- `OPENSEARCH_HOST` — hostname:port
|
||||
- `OPENSEARCH_SCHEME` — http or https
|
||||
- `OPENSEARCH_VERIFY_CERTS` — SSL certificate verification (true/false)
|
||||
|
||||
### Authentik (OIDC)
|
||||
- `AUTHENTIK_ISSUER` — OIDC issuer URL
|
||||
- `AUTHENTIK_VERIFY_SSL` — SSL certificate verification (true/false)
|
||||
- `MEM_AUTH_MODE` — auth mode: jwt | apikey | none
|
||||
|
||||
### Temporal (Workflow Orchestration - Future)
|
||||
- `TEMPORAL_ENDPOINT` — temporal frontend hostname:port
|
||||
- `TEMPORAL_NAMESPACE` — temporal namespace
|
||||
|
||||
### API Gateway (Route Optimization - Future)
|
||||
- `GATEWAY_URL` — gateway base URL
|
||||
|
||||
### Memory Service Config
|
||||
- `MEM_AUTH_MODE` — jwt | apikey | none
|
||||
- `MEM_RATE_LIMIT_INGEST` — ingest requests per second
|
||||
- `MEM_RATE_LIMIT_QUERY` — query requests per second
|
||||
- `MEM_EMBEDDING_BATCH_SIZE` — batch size for embeddings
|
||||
|
||||
---
|
||||
|
||||
## Deployment Scenarios
|
||||
|
||||
### Production (SOPS-Encrypted ConfigMap)
|
||||
|
||||
**File**: `k8s/app/config.yaml`
|
||||
|
||||
Services use cluster-internal DNS:
|
||||
```yaml
|
||||
LLM_ENDPOINT: http://reasoning-predictor.llm-serving.svc.cluster.local:8000/v1/chat/completions
|
||||
OPENSEARCH_HOST: opensearch.poimen.svc.cluster.local:9200
|
||||
AUTHENTIK_ISSUER: https://authentik.auth.svc.cluster.local:9443/application/o/poimen/
|
||||
TEMPORAL_ENDPOINT: temporal-frontend.temporal.svc.cluster.local:7233
|
||||
GATEWAY_URL: http://api-gw.poimen.svc.cluster.local:8080
|
||||
MEM_AUTH_MODE: jwt
|
||||
```
|
||||
|
||||
**Deploy**:
|
||||
```bash
|
||||
# SOPS auto-decrypts based on .sops.yaml age key
|
||||
kubectl apply -f k8s/app/config.yaml -k k8s/app/
|
||||
```
|
||||
|
||||
### Local/Development (Plaintext ConfigMap)
|
||||
|
||||
**File**: `k8s/app/config.local.yaml`
|
||||
|
||||
Services via external URLs (ingress):
|
||||
```yaml
|
||||
LLM_ENDPOINT: https://api.riotpiao.com/v1/chat/completions
|
||||
OPENSEARCH_HOST: opensearch.riotpiao.com:443
|
||||
AUTHENTIK_ISSUER: https://authentik.riotpiao.com/application/o/poimen/
|
||||
TEMPORAL_ENDPOINT: temporal.riotpiao.com:443
|
||||
GATEWAY_URL: https://api.riotpiao.com
|
||||
MEM_AUTH_MODE: none
|
||||
```
|
||||
|
||||
**Deploy** (override production config):
|
||||
```bash
|
||||
# Delete prod config, apply local
|
||||
kubectl delete configmap poimen-memory-config -n poimen
|
||||
kubectl apply -f k8s/app/config.local.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Encrypting with SOPS
|
||||
|
||||
Production `config.yaml` is encrypted with SOPS (Age-based).
|
||||
|
||||
**Encrypt**:
|
||||
```bash
|
||||
sops -e k8s/app/config.yaml > k8s/app/config.yaml.enc
|
||||
mv k8s/app/config.yaml.enc k8s/app/config.yaml
|
||||
```
|
||||
|
||||
**Decrypt for editing** (SOPS auto-handles with $EDITOR):
|
||||
```bash
|
||||
sops k8s/app/config.yaml
|
||||
```
|
||||
|
||||
**View decrypted** (without editing):
|
||||
```bash
|
||||
sops -d k8s/app/config.yaml
|
||||
```
|
||||
|
||||
**.sops.yaml** defines encryption key:
|
||||
```yaml
|
||||
creation_rules:
|
||||
- path_regex: k8s/app/config.yaml
|
||||
key_groups:
|
||||
- age:
|
||||
- <age-public-key>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Application Code Pattern
|
||||
|
||||
Example: Application should read URIs from ENV at startup.
|
||||
|
||||
```rust
|
||||
// Pseudocode
|
||||
let llm_endpoint = env::var("LLM_ENDPOINT")
|
||||
.unwrap_or("http://localhost:11434/v1/chat/completions".to_string());
|
||||
let opensearch_host = env::var("OPENSEARCH_HOST")
|
||||
.unwrap_or("localhost:9200".to_string());
|
||||
let auth_mode = env::var("MEM_AUTH_MODE")
|
||||
.unwrap_or("none".to_string());
|
||||
|
||||
// Initialize clients with these URIs
|
||||
let llm_client = LlmClient::new(llm_endpoint)?;
|
||||
let search_client = OpenSearchClient::new(opensearch_host)?;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Aspect | Production | Local |
|
||||
|--------|-----------|-------|
|
||||
| **Config File** | `config.yaml` | `config.local.yaml` |
|
||||
| **Encryption** | SOPS (Age) | Plaintext |
|
||||
| **Service URIs** | Cluster-internal DNS | External HTTPS |
|
||||
| **Auth Mode** | JWT (Authentik) | None (disabled) |
|
||||
| **Rate Limits** | 100/1000 | 1000/10000 |
|
||||
| **Deploy** | `kubectl apply -k k8s/app/` | `kubectl apply -f config.local.yaml` |
|
||||
Reference in New Issue
Block a user