docs: Add M7 source connectors (10 tasks), M3.5.10 auth integration, remove Kong refs
- M7.1-M7.10: Extensible SourceConnector trait, Obsidian/paperless/git/S3 connectors, sync framework, CLI, HTTP endpoints, health monitoring, gate - M3.5.10: Auth integration with Authentik OIDC → Vault token validation - DESIGN.md: Add source connectors architecture, update auth to Authentik/Vault (Kong removed from cluster) - INDEX.md: 75 tasks, 11 gates - Fix all Kong references in M3.5.1 task
This commit is contained in:
Story Crater Bot
@@ -0,0 +1,178 @@
|
||||
# M3.5.10 — Auth integration with Authentik/Vault
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M3.5 — Distributed API Layer |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | — |
|
||||
| Depends | M3.5.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Replace the placeholder `apikey` header check with proper authentication via the
|
||||
cluster's IAM stack: **Authentik** (OIDC provider) → **HashiCorp Vault** (token
|
||||
issuer) → **memory service** (token validator).
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
**Current (wrong):**
|
||||
```rust
|
||||
fn check_auth(req: &HttpRequest, state: &AppState) -> Result<(), HttpResponse> {
|
||||
let api_key = req.headers().get("apikey").and_then(|h| h.to_str().ok());
|
||||
if api_key != Some(&state.api_key) {
|
||||
return Err(HttpResponse::Unauthorized().json(...));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
This is a raw string match against `MEM_API_KEY` env var. No JWT, no Vault, no
|
||||
user identity. It does not integrate with the cluster's IAM stack.
|
||||
|
||||
**Cluster IAM stack:**
|
||||
- **Authentik** (`iam` namespace) — OIDC provider at
|
||||
`https://authentik.riotpiao.com/application/o/vault/`
|
||||
- **HashiCorp Vault** (`iam` namespace) — OIDC auth method enabled, validates
|
||||
Authentik JWTs, issues Vault tokens based on role/policy.
|
||||
- **Vault OIDC role:** `auth/oidc/role/homelab-admin`
|
||||
- `bound_claims: { "permissions": "*" }`
|
||||
- Policy: `homelab-admin` (path `"*"` full access)
|
||||
- **Vault unseal:** Shamir 3/3, keys in `vault-unseal-keys` secret, S3 backend
|
||||
via MinIO.
|
||||
|
||||
**Auth flow (production):**
|
||||
```
|
||||
User/Agent authenticates with Authentik (OIDC)
|
||||
→ Receives JWT with claims { sub, permissions, groups, ... }
|
||||
→ Presents JWT to Vault OIDC auth method
|
||||
→ Vault validates JWT against Authentik JWKS
|
||||
→ Vault issues Vault token with matched policy
|
||||
→ Client sends Vault token to memory service
|
||||
→ Memory service validates token via Vault API
|
||||
```
|
||||
|
||||
**Three integration options (pick one):**
|
||||
|
||||
### Option A: Vault token validation (recommended)
|
||||
Memory service receives `X-Vault-Token` header, calls Vault's
|
||||
`POST /v1/auth/token/lookup-self` to validate. Extracts policy and metadata.
|
||||
- Pro: Vault is the single source of truth for authorization.
|
||||
- Pro: Token revocation is immediate (Vault controls lifecycle).
|
||||
- Con: Extra network call per request (cache with TTL to mitigate).
|
||||
|
||||
### Option B: Direct JWKS validation
|
||||
Memory service fetches Authentik's JWKS endpoint, validates JWT `Authorization:
|
||||
Bearer <token>` directly. No Vault in the request path.
|
||||
- Pro: No Vault dependency at request time.
|
||||
- Pro: Standard OAuth2/OIDC pattern.
|
||||
- Con: Token revocation is delayed (until JWT expires).
|
||||
- Con: Memory service must know about Authentik's OIDC config.
|
||||
|
||||
### Option C: Trust gateway
|
||||
Memory service trusts homelab-frontend gateway (cluster-internal traffic).
|
||||
Gateway validates auth, forwards `X-User-Id` and `X-Capabilities` headers.
|
||||
Memory service checks capabilities against ServiceAdapter CRD requirements.
|
||||
- Pro: Auth logic centralized in gateway.
|
||||
- Pro: Memory service stays simple.
|
||||
- Con: Gateway auth is currently a stub (`hasCapability()` returns true for any
|
||||
`Authorization` header).
|
||||
- Con: Requires gateway auth to be completed first (homelab-frontend task 8.3).
|
||||
|
||||
**ServiceAdapter CRD for memory (`memory-adapter`):**
|
||||
```yaml
|
||||
auth:
|
||||
capability: memory:read # default
|
||||
required: true
|
||||
resources:
|
||||
- name: ingest
|
||||
methods:
|
||||
- verb: POST
|
||||
auth: { capability: memory:write, required: true }
|
||||
- name: query
|
||||
methods:
|
||||
- verb: POST
|
||||
- name: skills
|
||||
methods:
|
||||
- verb: GET
|
||||
```
|
||||
|
||||
**Capabilities needed:**
|
||||
- `memory:read` — query, skills, vault browse, projects, sources
|
||||
- `memory:write` — ingest, source sync, skill draft
|
||||
|
||||
## Steps
|
||||
|
||||
### Option A (Vault token — recommended)
|
||||
|
||||
1. Add `vault_addr` to `AppState` (default: `http://vault.iam.svc.cluster.local:8200`).
|
||||
2. Replace `check_auth()` with `validate_vault_token()`:
|
||||
```rust
|
||||
async fn validate_vault_token(req: &HttpRequest, state: &AppState) -> Result<VaultIdentity, HttpResponse> {
|
||||
let token = req.headers().get("X-Vault-Token")
|
||||
.or_else(|| req.headers().get("Authorization")) // Bearer <token>
|
||||
.and_then(|h| h.to_str().ok());
|
||||
// POST vault_addr/v1/auth/token/lookup-self with X-Vault-Token header
|
||||
// Parse response: policies, metadata, ttl
|
||||
// Cache token -> identity for TTL duration
|
||||
}
|
||||
```
|
||||
3. Add token cache (HashMap<token_hash, (VaultIdentity, Instant)>) with configurable TTL.
|
||||
4. Extract `VaultIdentity` (policies, metadata) from lookup response.
|
||||
5. Map policies to capabilities: `homelab-admin` → `memory:read` + `memory:write`.
|
||||
6. Update each handler to check required capability.
|
||||
7. Keep `apikey` as fallback for dev/test (controlled by env var `MEM_AUTH_MODE=vault|apikey`).
|
||||
|
||||
### For all options
|
||||
|
||||
8. Add env vars: `VAULT_ADDR`, `MEM_AUTH_MODE` (vault/jwks/gateway/apikey).
|
||||
9. Update K8s deployment to inject `VAULT_ADDR`.
|
||||
10. Update ServiceAdapter CRD if needed.
|
||||
11. Document auth flow in README.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Requests with valid Vault token are accepted.
|
||||
- Requests with expired/revoked Vault token are rejected (401).
|
||||
- Requests without any auth are rejected (401).
|
||||
- `memory:write` capability required for ingest/sync endpoints.
|
||||
- `memory:read` capability sufficient for query/skills/vault endpoints.
|
||||
- Token cache reduces Vault API calls on repeated requests.
|
||||
- Fallback to `apikey` mode for dev/test environments.
|
||||
|
||||
## Verify
|
||||
|
||||
**Integration test** — `tests/it_auth_integration.rs`:
|
||||
1. `a1_vault_token_accepted` — mock Vault lookup-self returning valid response;
|
||||
assert request proceeds.
|
||||
2. `a2_expired_token_rejected` — mock Vault returning 403; assert 401 response.
|
||||
3. `a3_no_auth_rejected` — request with no auth headers; assert 401.
|
||||
4. `a4_write_requires_capability` — token with `memory:read` only; POST /ingest;
|
||||
assert 403.
|
||||
5. `a5_read_with_read_capability` — token with `memory:read`; GET /query;
|
||||
assert proceeds.
|
||||
6. `a6_token_cache_hit` — same token twice; assert Vault called once.
|
||||
7. `a7_apikey_fallback` — `MEM_AUTH_MODE=apikey`; assert old behavior works.
|
||||
8. `a8_auth_mode_configurable` — assert `MEM_AUTH_MODE` switches validation logic.
|
||||
|
||||
**Command:** `cargo test --test it_auth_integration`
|
||||
|
||||
**False pass:**
|
||||
- Testing only with apikey fallback. The Vault integration is the whole point.
|
||||
- Mocking Vault without testing cache expiry. A cache that never expires accepts
|
||||
revoked tokens forever.
|
||||
|
||||
## Traps
|
||||
|
||||
- Calling Vault on every request without caching. Vault API calls add 5-10ms
|
||||
per request. Cache with TTL matching token TTL (or shorter).
|
||||
- Not handling Vault being temporarily unreachable. Return 503 (not 401) if
|
||||
Vault is down — "cannot verify" is not "unauthorized".
|
||||
- Hardcoding Vault addr. Use env var + service discovery.
|
||||
- Not supporting `Authorization: Bearer <token>` format alongside `X-Vault-Token`.
|
||||
Different clients use different conventions.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — auth section, Authentik/Vault IAM stack
|
||||
Reference in New Issue
Block a user