riotpiao-poimen/homelab
2
0
Fork 0
Code Issues 23 Pull Requests 1 Actions Packages Projects 1 Releases Wiki Activity

chore: remove intermediate progress markdown files

Browse Source 
Deleted:
- PHASE1_AND_QUEUE_COMPLETE.md (intermediate summary)
- OAUTH2_PROVIDERS_STATUS.md (reference - move to wiki if needed)

Kept:
- AUTH_INTEGRATION.md (architecture)
- TROUBLESHOOTING.md (operations)
- USAGE.md (user guide)
- README.md (repo index)
This commit is contained in:
rock
2026-09-13 05:25:13 +09:00
parent 03044614b7
commit e34232cc98
2 changed files with 0 additions and 778 deletions
Download Patch File Download Diff File Expand all files Collapse all files
OAUTH2_PROVIDERS_STATUS.md
-430
View File
@@ -1,430 +0,0 @@
# OAuth2 Providers Status — Complete Reference
**Last Updated**: 2026-09-12
**Status**: ALL 6 PROVIDERS ✅ COMPLETE
---
## Summary Table
| Provider | PK | Client ID | Status | Grant Types | Secrets | Scope Mappings | Service Account |
|----------|----|-----------| -------|-------------|---------|----------------|-----------------|
| **api-gw** | 2 | api-gw | ✅ | authz_code, implicit, password, client_creds | ✅ Stored | ✅ 9 mappings | api-gateway |
| **minio** | 3 | minio | ✅ | authz_code, implicit, password, client_creds | ✅ Stored | ✅ 9 mappings | minio service |
| **poimen** | 4 | poimen | ✅ | authz_code, implicit, password, client_creds | ✅ Stored | ✅ 9 mappings | poimen service |
| **paperless** | 5 | paperless | ✅ | authz_code, implicit, password, client_creds | ✅ Stored | ✅ 9 mappings | paperless service |
| **grafana** | 6 | grafana | ✅ | authz_code, implicit, password, client_creds | ✅ Stored | ✅ 9 mappings | grafana service |
| **queue** | 13 | queue-sqs | ✅ | authz_code, implicit, password, client_creds | ✅ Stored | ✅ 9 mappings | temporal-worker-agent |
**TOTAL: 6/6 Complete** ✅
---
## Detailed Provider Specs
### 1. api-gw (PK=2)
**Purpose**: Core API gateway authentication
**Client ID**: `api-gw`
**Grant Types**:
- ✅ authorization_code (browser login)
- ✅ implicit (SPA)
- ✅ password (CLI/scripts)
- ✅ client_credentials (service-to-service)
**Redirect URIs**:
- `http://localhost:3000/callback`
- `https://api.riotpiao.com/callback`
**Scope Mappings** (9/9):
```
roles, permissions, minio_buckets, paperless_doctypes,
memory_projects, memory_visibility, authorized_models,
sqs_queues, grafana_org_role
```
**Application**: `api-gw` (bound)
**Secret**: In `~/.env` as `AUTHENTIK_PROVIDER_API_GW_SECRET`
**Last Rotated**: 2026-09-12
---
### 2. minio (PK=3)
**Purpose**: MinIO S3 console OIDC login
**Client ID**: `minio`
**Grant Types**:
- ✅ authorization_code
- ✅ implicit
- ✅ password
- ✅ client_credentials
**Redirect URIs**:
- `http://localhost:9000/auth/sso/oauth2/code`
- `https://minio.riotpiao.com/auth/sso/oauth2/code`
**Scope Mappings** (9/9): All fine-grained claims
**Application**: `minio` (bound)
**Secret**: In `~/.env` as `AUTHENTIK_PROVIDER_MINIO_SECRET`
**Last Rotated**: 2026-09-12
**Special Claims** (JWT):
```json
{
"policy": "consoleAdmin" // for homelab-admins
// or "readonly" for others
}
```
---
### 3. poimen (PK=4)
**Purpose**: Memory/semantic search service OIDC
**Client ID**: `poimen`
**Grant Types**:
- ✅ authorization_code
- ✅ implicit
- ✅ password
- ✅ client_credentials
**Redirect URIs**:
- `http://localhost:3000/callback`
- `https://poimen.riotpiao.com/callback`
**Scope Mappings** (9/9): All fine-grained claims
**Application**: `poimen` (bound)
**Secret**: In `~/.env` as `AUTHENTIK_PROVIDER_POIMEN_SECRET`
**Last Rotated**: 2026-09-12
**Special Claims** (JWT):
```json
{
"memory_projects": ["*"], // User's allowed projects
"memory_visibility": "private", // User's visibility level
"memory_role": "admin" | "user" // User's role
}
```
---
### 4. paperless (PK=5)
**Purpose**: Paperless-ngx document manager OIDC
**Client ID**: `paperless`
**Grant Types**:
- ✅ authorization_code
- ✅ implicit
- ✅ password
- ✅ client_credentials
**Redirect URIs**:
- `http://localhost:8000/auth/complete`
- `https://paperless.riotpiao.com/auth/complete`
**Scope Mappings** (9/9): All fine-grained claims
**Application**: `paperless` (bound)
**Secret**: In `~/.env` as `AUTHENTIK_PROVIDER_PAPERLESS_SECRET`
**Last Rotated**: 2026-09-12
**Special Claims** (JWT):
```json
{
"paperless_doctypes": ["invoices", "receipts", "expenses"],
"paperless_access": "admin" | "readonly"
}
```
---
### 5. grafana (PK=6)
**Purpose**: Grafana dashboards OIDC
**Client ID**: `grafana`
**Grant Types**:
- ✅ authorization_code
- ✅ implicit
- ✅ password
- ✅ client_credentials
**Redirect URIs**:
- `http://localhost:3000/login/generic_oauth`
- `https://grafana.riotpiao.com/login/generic_oauth`
**Scope Mappings** (9/9): All fine-grained claims
**Application**: `grafana` (bound)
**Secret**: In `~/.env` as `AUTHENTIK_PROVIDER_GRAFANA_SECRET`
**Last Rotated**: 2026-09-12
**Special Claims** (JWT):
```json
{
"grafana_org_role": "Admin" | "Editor" | "Viewer"
}
```
---
### 6. queue (PK=13) — NEW
**Purpose**: Kafka/SQS queue operations
**Client ID**: `queue-sqs`
**Grant Types**:
- ✅ authorization_code
- ✅ implicit
- ✅ password
- ✅ **client_credentials** ← Used by temporal-worker-agent
**Redirect URIs**:
- `http://localhost:8080/callback`
- `https://queue.riotpiao.com/callback`
**Scope Mappings** (9/9): All fine-grained claims
**Application**: `queue` (bound)
**Secret**: In `~/.env` as `AUTHENTIK_PROVIDER_QUEUE_SECRET`
**Created**: 2026-09-12
**Special Claims** (JWT):
```json
{
"sqs_queues": ["default", "events", "*"] // Allowed topics
}
```
**Service Account**:
- `temporal-worker-agent` can authenticate via client_credentials
- Has roles: llm:inference, workflow:execute, memory:read, memory:write, queue:send
- Claims: sqs_queues=*, authorized_models=[reasoning, ornith:35b, qwen2.5:3b]
---
## Scope Mappings Reference (9 Total)
All 6 providers linked to these 9 scope mappings:
| Scope Name | Expression | Values |
|------------|------------|--------|
| `roles` | user.attributes.get("roles", []) | ["llm:inference", "queue:send", "workflow:execute", "memory:read", "memory:write"] |
| `permissions` | ["*"] if superuser else groups | All groups or ["*"] |
| `minio_buckets` | user.attributes.get("minio_buckets", []) | ["*"], ["paperless"], ["immich"], ["backups"] |
| `paperless_doctypes` | user.attributes.get("paperless_doctypes", []) | ["*"], ["invoices"], ["contracts"], ["employment"] |
| `memory_projects` | user.attributes.get("memory_projects", []) | ["*"], ["prod"], ["dev"], ["staging"] |
| `memory_visibility` | user.attributes.get("memory_visibility", "public") | "private", "internal", "public" |
| `authorized_models` | user.attributes.get("authorized_models", []) | ["reasoning"], ["ornith:35b"], ["qwen2.5:3b"] |
| `sqs_queues` | user.attributes.get("sqs_queues", []) | ["*"], ["default"], specific topics |
| `grafana_org_role` | user.attributes.get("grafana_org_role", "Viewer") | "Admin", "Editor", "Viewer" |
---
## Flows Configuration
All 6 OAuth2 providers use these flows:
| Flow | Slug | Type | Purpose |
|------|------|------|---------|
| Authorization Flow | `default-provider-authorization-implicit-consent` | UUID | Token generation for all grant types |
| Invalidation Flow | `default-provider-invalidation-flow` | UUID | Token revocation/logout |
---
## Grant Type Matrix
| Grant Type | Use Case | Providers | Example |
|------------|----------|-----------|---------|
| **authorization_code** | Browser login + redirect | All 6 | User clicks "Login with Authentik" |
| **implicit** | SPA/JavaScript apps | All 6 | Frontend JavaScript fetching token |
| **password** | CLI login | All 6 | `core auth login` device code |
| **client_credentials** | Service-to-service | All 6 | temporal-worker-agent → API gateway |
---
## Credentials Storage
**Location**: `~/.env` (gitignored)
**Format**: `export AUTHENTIK_PROVIDER_{NAME}_SECRET="..."`
**All Secrets Status** (as of 2026-09-12):
```
AUTHENTIK_PROVIDER_API_GW_SECRET ✅ Rotated (256-bit)
AUTHENTIK_PROVIDER_MINIO_SECRET ✅ Rotated (256-bit)
AUTHENTIK_PROVIDER_POIMEN_SECRET ✅ Rotated (256-bit)
AUTHENTIK_PROVIDER_PAPERLESS_SECRET ✅ Rotated (256-bit)
AUTHENTIK_PROVIDER_GRAFANA_SECRET ✅ Rotated (256-bit)
AUTHENTIK_PROVIDER_QUEUE_SECRET ✅ Created (256-bit)
```
**Backup Location**: SOPS-encrypted in git
**Rotation Schedule**: Every 90 days (next: 2026-12-11)
---
## Service Account Access
### Temporal Worker Agent
```
OAuth2 Provider: queue
Grant Type: client_credentials
Can authenticate as: temporal-worker-agent
Has roles: llm:inference, workflow:execute, memory:read, memory:write, queue:send
Can access:
- LLM models (via api-gateway)
- Memory projects (*)
- Queue topics (*)
- Workflows (Temporal)
```
### Paperless AI Agent
```
OAuth2 Provider: api-gw (implicit)
Can authenticate as: paperless-ai-agent
Has roles: llm:inference, memory:write, paperless:admin
Can access:
- LLM models (reasoning, qwen2.5:3b)
- Memory projects (*)
- Paperless admin (*)
- MinIO buckets (paperless)
```
### Memory Agent
```
OAuth2 Provider: api-gw (implicit)
Can authenticate as: memory-agent
Has roles: llm:inference, memory:read, memory:write
Can access:
- LLM models (all 3)
- Memory projects (*)
- Private visibility documents
```
### Portfolio Agent
```
OAuth2 Provider: api-gw (implicit)
Can authenticate as: portfolio-agent
Has roles: llm:inference, memory:read
Can access:
- LLM models (ornith:35b)
- Memory projects (homelab, portfolio)
- Public visibility documents
- MinIO backups bucket
```
---
## Application Bindings
All providers bound to OAuth2 Applications:
```
Authentik Admin Console
→ Providers (6)
→ Applications (6)
→ api-gw → api-gw provider
→ minio → minio provider
→ poimen → poimen provider
→ paperless → paperless provider
→ grafana → grafana provider
→ queue → queue provider (NEW)
```
**Public Endpoint**: `https://authentik.riotpiao.com/application/o/token/`
---
## Testing Credentials
**Test User**: `rock`
**Test Email**: `[email protected]`
**Test Group**: `homelab-admins`
**Test JWT Claims**:
```json
{
"sub": "rock",
"groups": ["homelab-admins"],
"roles": ["*"],
"permissions": ["*"],
"minio_buckets": ["*"],
"memory_projects": ["*"],
"authorized_models": ["*"],
"sqs_queues": ["*"],
"grafana_org_role": "Admin"
}
```
---
## Troubleshooting
### Issue: "Invalid client_secret"
**Solution**: Regenerate secret in Authentik UI
```bash
# Or via API
AUTHENTIK_BOOTSTRAP_TOKEN=... python3 scripts/iam/provision-rbac.py
```
### Issue: Token validation fails at gateway
**Solution**: Verify JWT claim in Authorization header
```bash
# Decode token
core auth token | jq -R 'split(".")[1] | @base64d | fromjson'
```
### Issue: X-Forwarded-User not reaching backend
**Solution**: Check api-gateway proxy configuration
```bash
kubectl logs -n api svc/api-gateway | grep "X-Forwarded"
```
### Issue: Queue provider returns 404
**Solution**: Management service API endpoints not finalized yet
```bash
# Expected in Phase 2:
# POST /api/v1/messages
# GET /api/v1/topics
```
---
## Commands for Verification
```bash
# List all OAuth2 providers
curl -s -H "Authorization: Bearer $AUTHENTIK_BOOTSTRAP_TOKEN" \
https://authentik.riotpiao.com/api/v3/providers/oauth2/ | jq '.results | length'
# Get specific provider
curl -s -H "Authorization: Bearer $AUTHENTIK_BOOTSTRAP_TOKEN" \
https://authentik.riotpiao.com/api/v3/providers/oauth2/13/ | jq '{name, client_id, grant_types}'
# Get all applications
curl -s -H "Authorization: Bearer $AUTHENTIK_BOOTSTRAP_TOKEN" \
https://authentik.riotpiao.com/api/v3/core/applications/ | jq '.results | length'
# Test client_credentials flow
curl -X POST https://authentik.riotpiao.com/application/o/token/ \
-d "grant_type=client_credentials" \
-d "client_id=queue-sqs" \
-d "client_secret=$AUTHENTIK_PROVIDER_QUEUE_SECRET" \
-d "scope=openid"
```
---
## Related Documentation
- [PHASE1_AND_QUEUE_COMPLETE.md](PHASE1_AND_QUEUE_COMPLETE.md) - Phase 1 CLI + Queue setup
- [provision-rbac.py](scripts/iam/provision-rbac.py) - Idempotent provisioning script
- [rotate-secrets.sh](scripts/iam/rotate-secrets.sh) - 90-day rotation schedule
- [AUTH_INTEGRATION.md](AUTH_INTEGRATION.md) - Three-layer auth architecture
---
**OAUTH2 PROVIDERS COMPLETE** ✅
**ALL 6 CONFIGURED AND TESTED** ✅
**READY FOR PRODUCTION** ✅
Verified 2026-09-12.
PHASE1_AND_QUEUE_COMPLETE.md
-348
View File
@@ -1,348 +0,0 @@
# Phase 1 Complete + Queue OAuth2 Setup ✅✅✅
**Date**: 2026-09-12
**Status**: PRODUCTION READY
---
## Executive Summary
**Core CLI Phase 1 fully implemented and tested.** Auth + LLM modules working. Queue OAuth2 provider configured. All 6 OAuth2 providers ready. Ready for Phase 2.
---
## Phase 1: Core CLI Complete
### ✅ Authentication Module
```bash
core auth login # Device code flow → Authentik
core auth status # Show token + expiry
core auth token # Export JWT for scripts
core auth logout # Delete token
core auth refresh # Refresh token
```
**Token Storage**: `~/.riotpiao/token.json` (0600 perms, HMAC-signed)
### ✅ LLM Module
```bash
core llm models # List 5 models (tested ✓)
core llm chat "2+2?" # Single-turn inference
core llm chat --model reasoning "prompt"
core llm complete --model reasoning --prompt "..." --max-tokens 512
```
**Result**: 5 models returned, JWT auth working, X-Forwarded-User headers propagated
### ✅ Auth Stack
```
User/Script
↓
core auth login (device code flow)
↓
Authentik OAuth2 (client_credentials or browser flow)
↓
~/.riotpiao/token.json (JWT, 24h expiry)
↓
core llm/queue/workflow commands
↓
Load token → POST/GET api.riotpiao.com with JWT + X-Forwarded-User
↓
Gateway validates → Backend services
```
---
## Queue OAuth2 Setup Complete
### ✅ Authentik Configuration
| Component | Status | Details |
|-----------|--------|---------|
| OAuth2 Provider `queue` | ✅ Created (pk=13) | client_id=queue-sqs, all grant types |
| Grant Types | ✅ Complete | `authorization_code`, `implicit`, `password`, `client_credentials` |
| Scope Mappings | ✅ Linked | roles, permissions, minio_buckets, paperless_doctypes, memory_projects, memory_visibility, authorized_models, **sqs_queues**, grafana_org_role |
| Service Account | ✅ Updated | `temporal-worker-agent` now has `queue:send` role + `sqs_queues=*` claim |
| Groups | ✅ Active | `sqs-users` (read default), `sqs-writers` (read/write all) |
| Application | ✅ Bound | queue app linked to provider |
### ✅ CLI Implementation
```
src/cmd/queue/mod.rs (240 lines, ready)
- send --topic "message"
- list-topics
- receive --topic --count N --group GROUP
- describe --topic
```
**Status**: Awaiting management-service API endpoint finalization
### ✅ Credentials
```bash
# ~/.env
export AUTHENTIK_PROVIDER_QUEUE_SECRET="qHPbhENUq70V6fbXl10O..." (43 chars)
```
---
## All OAuth2 Providers Status
| Provider | pk | Client ID | Status | Grant Types |
|----------|----|-----------| -------|-------------|
| api-gw | 2 | api-gw | ✅ | authz_code, implicit, password, client_credentials |
| minio | 3 | minio | ✅ | authz_code, implicit, password, client_credentials |
| poimen | 4 | poimen | ✅ | authz_code, implicit, password, client_credentials |
| paperless | 5 | paperless | ✅ | authz_code, implicit, password, client_credentials |
| grafana | 6 | grafana | ✅ | authz_code, implicit, password, client_credentials |
| queue | 13 | queue-sqs | ✅ | authz_code, implicit, password, **client_credentials** |
**TOTAL: 6/6 Complete**
---
## Test Results
### Auth
```
$ core auth status
🔐 Authentication Status:
Client: rock-user
Scope: openid
Expires in: 24h 0m
✅ Token valid
```
### LLM
```
$ core llm models
📦 Available LLM Models:
• BAAI/bge-reranker-base (api.riotpiao.com)
• nomic-ai/nomic-embed-text-v2-moe (api.riotpiao.com)
• ornith:35b (api.riotpiao.com)
• qwen2.5:3b-instruct (api.riotpiao.com)
• reasoning (api.riotpiao.com)
✅ Total: 5 models
```
### Queue (OAuth2 only, API TBD)
```
$ core queue list-topics
error: management-service API spec not finalized
(OAuth2 provider configured, CLI ready)
```
---
## Architecture Diagram
```
┌─────────────┐
│ User/Script │
└──────┬──────┘
│
├─ core auth login
│ ↓
│ [Device Code Flow]
│ ↓
│ Authentik OAuth2
│ ↓
│ ~/.riotpiao/token.json (JWT, 24h)
│
├─ core llm models
│ ↓
│ Load token
│ ↓
│ POST api.riotpiao.com/v1/models
│ + Authorization: Bearer JWT
│ + X-Forwarded-User: client_id
│ ↓
│ api-gateway validates JWT
│ ↓
│ Backend (vLLM, Ollama, TEI)
│ ↓
│ 5 models returned ✅
│
└─ core queue send
↓
Load token
↓
POST management-service/api/v1/messages
+ Authorization: Bearer JWT
+ X-Forwarded-User: queue-sqs
↓
Queue service validates sqs_queues claim
↓
Message enqueued ✅ (pending API spec)
```
---
## Security
✅ **Token Storage**
- File: `~/.riotpiao/token.json`
- Perms: 0600 (owner read/write only)
- No token in environment variables
- No token logging in output
✅ **JWT Security**
- Signed by Authentik (HMAC-SHA256)
- Expiry: 24 hours
- Refresh token support (Phase 2)
- Revocation via logout
✅ **Authorization**
- X-Forwarded-User header for audit
- Fine-grained scopes (sqs_queues, memory_projects, etc)
- Service account roles (llm:inference, queue:send, etc)
- Group-based permissions (sqs-users, sqs-writers)
---
## Files Changed
### Core CLI
```
✅ src/auth/mod.rs (Token module)
✅ src/auth/token_manager.rs (Token lifecycle)
✅ src/auth/legacy.rs (Backward compat)
✅ src/cmd/auth/mod.rs (Auth commands)
✅ src/cmd/llm/mod.rs (LLM commands)
✅ src/cmd/queue/mod.rs (Queue commands)
✅ src/cmd/iam_stub.rs (Phase 2 placeholder)
✅ src/cmd/minio_stub.rs (Phase 2 placeholder)
✅ src/main.rs (Updated CLI)
✅ Cargo.toml (Dependencies)
✅ build.rs (Build script)
```
### Homelab
```
✅ scripts/iam/provision-rbac.py (Queue provider + temporal-worker)
```
### Documentation
```
✅ PHASE1_COMPLETE.md (CLI Phase 1 details)
✅ PHASE1_IMPLEMENTATION.md (Implementation status)
✅ QUEUE_SETUP_COMPLETE.md (Queue OAuth2 setup)
✅ API_INTEGRATION.md (Auth stack architecture)
```
---
## Deployment
### 1. Build
```bash
cd ~/workplace/core
cargo build --release
# Binary: target/release/core (3.2 MB)
```
### 2. Install
```bash
cp ~/workplace/core/target/release/core /usr/local/bin/
chmod +x /usr/local/bin/core
```
### 3. Test
```bash
core auth login
core auth status
core llm models
core queue list-topics # Once API is ready
```
### 4. Deploy in Cluster (Optional)
```bash
# Copy binary to agent-pod
kubectl cp ~/workplace/core/target/release/core \
agent-pod-XXXX:/usr/local/bin/core -n agent-pod
# Or rebuild in-cluster via CI/CD
```
---
## What's Next (Phase 2)
### Week 1: Core Modules
- [ ] Workflow module (Temporal API)
- `core workflow submit --name job "python"`
- `core workflow list`, `describe`, `cancel`
- [ ] Memory module (Poimen)
- `core memory search --project prod "query"`
- `core memory store --project prod "text"`
- [ ] Streaming LLM
- `core llm chat --stream "prompt"`
### Week 2: Finalization
- [ ] S3 module (replaces bucket command)
- [ ] IAM/MinIO refactor (real implementations)
- [ ] Integration tests
- [ ] Documentation
### Week 3: Deprecation
- [ ] Mark old cluster commands as deprecated
- [ ] Migrate to legacy/talos-cli branch
- [ ] Archive kubectl/talosctl wrappers
---
## Known Limitations
1. **Chat 403 for some users** - Permission scoping to be investigated
2. **Queue API pending** - management-service endpoints not finalized
3. **Refresh token unused** - Auto-refresh in Phase 2
4. **No config file** - ~/.riotpiao/config.yaml in Phase 2
5. **No streaming** - Deferred to Phase 2
---
## Success Metrics ✅
| Metric | Target | Actual | Status |
|--------|--------|--------|--------|
| CLI builds | No warnings | 0 warnings | ✅ |
| Auth flow | Device code | Working | ✅ |
| Token storage | 0600 perms | Verified | ✅ |
| LLM models | 5 returned | 5 returned | ✅ |
| JWT auth | X-Forwarded-User | Headers sent | ✅ |
| OAuth2 providers | 6 total | 6 created | ✅ |
| Queue OAuth2 | client_credentials | Configured | ✅ |
| End-to-end test | Pass | Passed | ✅ |
---
## Critical Context
**Authentik URL**: https://authentik.riotpiao.com
**API Gateway**: https://api.riotpiao.com
**Bootstrap Token**: `kubectl -n iam get secret authentik-secrets -o jsonpath='{.data.AUTHENTIK_BOOTSTRAP_TOKEN}' | base64 -d`
**OAuth2 Providers** (6 total):
- pk 2-6: api-gw, minio, poimen, paperless, grafana
- pk 13: queue (NEW)
**Service Accounts** (4 total):
- paperless-ai-agent (llm:inference, memory:write, paperless:admin)
- portfolio-agent (llm:inference, memory:read)
- memory-agent (llm:inference, memory:read, memory:write)
- **temporal-worker-agent** (NEW: queue:send added)
---
## Commits
```
5b06ec3 cli: phase 1 complete + queue oauth2 setup
641ab8b iam: add queue oauth2 provider + temporal-worker queue access
```
---
**PHASE 1 AND QUEUE SETUP COMPLETE** ✅✅✅
**PRODUCTION READY FOR TESTING**
**NEXT: Phase 2 (Workflow, Memory, S3 modules)**
Verified 2026-09-12. All systems functional.