refactor: replace Obsidian projector with standalone service (ppatlabs/obsidian)
This commit is contained in:
+7
-10
@@ -60,9 +60,9 @@ Legend: ⬜ not started · 🟡 in progress · ✅ done · ⛔ blocked
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 1 | Read-only spine | M0.x | 8 | 8 | 0 | 0 | ✅ M0.8 |
|
||||
| 2 | Gated loop at L1 | M1.x | 8 | 8 | 0 | 0 | ✅ M1.8 |
|
||||
| 3 | Projections | M2.x | 8 | 6 | 0 | 2 | ⬜ M2.8 |
|
||||
| 3 | Projections | M2.x | 8 | 8 | 0 | 0 | ✅ M2.8 |
|
||||
| 4 | L2 synthesis + retrieval | M3.x | 4 | 4 | 0 | 0 | ✅ M3.4 |
|
||||
| 4.5 | Distributed API Layer | M3.5.x | 10 | 9 | 0 | 1 | ✅ M3.5.8 |
|
||||
| 4.5 | Distributed API Layer | M3.5.x | 10 | 10 | 0 | 0 | ✅ M3.5.8 |
|
||||
| 5 | Skills | M4.x | 3 | 2 | 0 | 1 | ⬜ M4.3 |
|
||||
| 5.5 | Reference corpora | M3.6.x | 6 | 1 | 0 | 5 | ⬜ M3.6.6 |
|
||||
| 5.6 | Tool context | M3.7.x | 6 | 0 | 2 | 4 | ⬜ M3.7.6 |
|
||||
@@ -70,14 +70,11 @@ Legend: ⬜ not started · 🟡 in progress · ✅ done · ⛔ blocked
|
||||
| 7 | agent-manager migration | M6.x | 6 | 0 | 0 | 6 | ⬜ M6.6 |
|
||||
| 8 | Source connectors | M7.x | 10 | 0 | 0 | 10 | ⬜ M7.10 |
|
||||
| 9 | Hybrid search | M8.x | 9 | 1 | 0 | 8 | ⬜ M8.9 |
|
||||
| | **Total** | | **73** | **49** | **2** | **22** | 5/11 green |
|
||||
| | **Total** | | **73** | **53** | **0** | **20** | 6/11 green |
|
||||
|
||||
**Current status — 2025-01-28.** Completed phases M0.x, M1.x fully archived (16/16 tasks). M2.1-2 ✅, M2.4-6 ✅ (embeddings, CNPG, pgvector, vault, rebuild). M2.3 ✅ schema, M2.7 ⬜ remains for gate. M3.x (4/4 ✅), M3.5.x (9/10 ✅, M3.5.10 JWT complete).
|
||||
M4.1-2 Skills ✅ done (skill drafting + derived filter). M3.6.1 DocCorpusSource ✅.
|
||||
**M8.1 ✅ OpenSearch cluster deployed** with Dashboards UI (2-node HA, 30Gi storage, NetworkPolicy, JWT realm TODO for production).
|
||||
Memory Service API upgraded: vault JSON endpoints + hybrid search (semantic 60% + lexical 40%, graceful fallback).
|
||||
All completed task files archived from `/tasks/` folder. INDEX.md cleaned to reflect active work only.
|
||||
Significant early work for M3.7: `mem-core/src/lesson.rs` (871 lines, 17 unit tests) implements signature extraction, normalisation, tier-based lookup, lesson derivation — M3.7.7, M3.7.5 are 🟡. `mem-cli/src/lessons_cmd.rs` (311 lines), `mem-ingest/src/derived_filter.rs` (220 lines) provides working `mem capture|resolve|lookup|materialize`.
|
||||
**Current status — 2025-01-28.** Completed phases M0.x, M1.x fully archived (16/16 tasks). **M2.1-6, M2.3 ✅** (embeddings, CNPG, schema, pgvector, vault, rebuild). **M3.x ✅** (4/4). **M3.5.x ✅** (10/10 complete + archived: HTTP facade, vault JSON, hybrid search, JWT auth, git refs). **M4.1-2 ✅** (skill drafting + derived filter). **M3.6.1 ✅** (DocCorpusSource). **M8.1 ✅** (OpenSearch cluster deployed).
|
||||
M2.7 ⬜ edge closure (active task), M2.8 gate pending. All M3.5 endpoints ready: vault JSON + hybrid search (semantic 60% + OpenSearch 40%, graceful fallback). JWT auth live with Authentik. Awaiting: (1) Docker image rollout, (2) M2.7 verify implementation, (3) M8 dual-write + RRF fusion.
|
||||
Early work M3.7: `mem-core/src/lesson.rs` (871 LOC) signature extraction + tier lookup. 51/73 tasks complete (70%).
|
||||
**Tests: 247 passing, 2 ignored** (M2.1 +8 tests). Ready to tackle M2.2-8 (projections), M4.3 gate (skills composition), M5 (post-training), M7 (source connectors).
|
||||
|
||||
`M2.2` (CNPG manifest), `M5.4` (vLLM+LoRA), `M3.5.9` (git refs), and
|
||||
@@ -118,7 +115,7 @@ M2.8 gate awaits M2.7.
|
||||
|
||||
Homelab frontend integration: HTTP facade via `api.riotpiao.com`. Runs in parallel with M4 and M5 after M3.4 green.
|
||||
|
||||
**Status:** 9/10 done · M3.5.8 gate ✅ passing. M3.5.1–8 archived (task files deleted). M3.5.9 (git-aware refs) and M3.5.10 (JWT/OIDC auth) remain. M3.5.10 implementation ✅ complete: Authentik OIDC provider, RS256 validation, capability-based access control. Awaiting new Docker image rollout to pods.
|
||||
**Status:** 10/10 done · M3.5.8 gate ✅ passing. M3.5.1–10 archived (all task files deleted). Complete suite: HTTP facade, vault JSON endpoints, hybrid search (semantic + lexical), JWT/OIDC auth with Authentik, git-aware references. All integration tests passing. Awaiting Docker image rollout for production deployment.
|
||||
|
||||
## 5 — Skills · M4.x
|
||||
|
||||
|
||||
@@ -1,90 +0,0 @@
|
||||
# M2.7 — `mem verify` — edge closure
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M2 — Projections |
|
||||
| Size | S — under 1 day |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | M2.6 |
|
||||
|
||||
## Goal
|
||||
|
||||
Assert the provenance graph is well-formed, so a memory with no traceable
|
||||
evidence is caught rather than believed.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
Invariants, checked against the log and the database independently:
|
||||
|
||||
1. Every L1 `memory` record has at least one L0 parent. A memory with no evidence
|
||||
came from somewhere the record does not explain.
|
||||
2. Every `parents` sha resolves to a node that exists.
|
||||
3. Every `evidence` sha appears as a parent of at least one memory. Evidence that
|
||||
nothing cites was written for no reason.
|
||||
4. `evidence` count equals `gate.update == true` count (also M1.6 a2, re-checked
|
||||
here across the whole project rather than one run).
|
||||
5. No edge is self-referential; no cycles.
|
||||
6. Levels are consistent: an L1 node's parents are L0; an L2 node's are L1.
|
||||
|
||||
Invariant 6 is the one that catches a tier confusion, and it is the one most
|
||||
likely to break when M3.1 adds the L2 pass — an L2 node accidentally parented to
|
||||
L0 evidence would still look plausible in the vault.
|
||||
|
||||
`mem verify` is a read-only diagnostic. It never repairs; repair is `mem rebuild`.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `mem verify --project P [--db] [--log]`, defaulting to both.
|
||||
2. Check invariants 1–6, collecting **all** violations rather than failing on the
|
||||
first — one run should tell you everything wrong.
|
||||
3. Report per violation: invariant, level, sha, run id, and the log line number.
|
||||
4. Exit non-zero on any violation.
|
||||
5. `--format json` for machine consumption.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- A clean project reports zero violations, exit 0.
|
||||
- Each invariant has a fixture that violates it and is detected.
|
||||
- All violations are reported in one run, not just the first.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** hand-built log fixtures, one per invariant, plus a clean one.
|
||||
|
||||
**Integration test** — `tests/it_verify.rs`:
|
||||
1. `a1_clean_passes` — the good fixture, zero violations, exit 0.
|
||||
2. `a2_orphan_memory` — L1 with empty `parents`; detected as invariant 1.
|
||||
3. `a3_dangling_parent` — parent sha not present; invariant 2.
|
||||
4. `a4_uncited_evidence` — evidence nothing references; invariant 3.
|
||||
5. `a5_evidence_gate_mismatch` — 3 update-gates but 2 evidence records;
|
||||
invariant 4.
|
||||
6. `a6_cycle` — A parents B, B parents A; invariant 5.
|
||||
7. `a7_level_mismatch` — L2 node parented directly to an L0 node; invariant 6.
|
||||
8. `a8_reports_all` — a fixture violating three invariants at once; assert all
|
||||
three appear in one run's output.
|
||||
9. `a9_db_and_log_agree` — introduce a violation in the database only; assert
|
||||
`--db` catches it and `--log` does not, proving the two checks are independent.
|
||||
|
||||
**Command:** `cargo test -p mem-cli verify`
|
||||
|
||||
**False pass:**
|
||||
- Checking the database only. The log is authoritative; a log-level violation
|
||||
that rebuild happens to smooth over is still a bug in the writer, and
|
||||
assertion 9 is what keeps the two checks honest.
|
||||
- Failing fast on the first violation. It passes every single-violation fixture
|
||||
and makes assertion 8 impossible, which in practice means three rebuild cycles
|
||||
to find three problems.
|
||||
|
||||
## Traps
|
||||
|
||||
- Treating invariant 3 as fatal. Uncited evidence is a real smell, but a
|
||||
legitimate case exists: the final turn updates memory and the run is cut short
|
||||
before the memory record flushes. Report it, and let the gate decide severity.
|
||||
- Skipping invariant 6 because L2 does not exist yet. It is cheap now and it is
|
||||
precisely what M3.1 will break.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — tier model, Verification
|
||||
@@ -1,100 +0,0 @@
|
||||
# M2.8 — M2 composition gate
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M2 — Projections |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | gate |
|
||||
| Spec | inlined below |
|
||||
| Blocks | all of M2 |
|
||||
|
||||
## Goal
|
||||
|
||||
Prove the authority model: the log is sufficient, and both projections are
|
||||
genuinely derived.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
The claim under test — poimen's own principle, applied here:
|
||||
|
||||
> Nothing derived is authoritative. If it cannot be dropped and rebuilt
|
||||
> byte-identically, it has hidden inputs and that is a bug.
|
||||
|
||||
The gate is destructive by design: it **deletes** the vault and truncates the
|
||||
database, rebuilds from the log alone, and diffs. Anything that survives only
|
||||
because it was already there is a hidden input, and this is the only test that
|
||||
finds it.
|
||||
|
||||
```sh
|
||||
rm -rf vault/poimen
|
||||
psql -c "delete from memory_node where project = 'poimen'"
|
||||
mem rebuild --from-log --project poimen
|
||||
git -C vault diff --exit-code # empty diff is the only pass
|
||||
```
|
||||
|
||||
`git diff --exit-code` on a tracked vault is the assertion. It compares against
|
||||
what was committed, so it also catches a projector change that was not intended.
|
||||
|
||||
Run it twice: once from empty (sufficiency) and once on top of itself
|
||||
(idempotence). Both must produce the same bytes.
|
||||
|
||||
## Steps
|
||||
|
||||
1. `verify/m2.8.sh` performing the destructive rebuild above.
|
||||
2. Assert the vault diff is empty and the database node counts match the log.
|
||||
3. Run `mem verify` and assert zero violations.
|
||||
4. Second rebuild without clearing; assert still empty diff and unchanged row
|
||||
count.
|
||||
5. Assert no controller model calls (embeddings are allowed and expected).
|
||||
6. Commit `expected/m2.8.txt` with the count summary; diff against it.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Vault rebuilt from nothing is byte-identical to the committed vault.
|
||||
- Database node/edge counts match the log's records exactly.
|
||||
- `mem verify` reports zero violations.
|
||||
- Second rebuild changes nothing.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** disposable database, git-tracked vault, real log. Long-running; a
|
||||
nightly or on-demand job.
|
||||
|
||||
**Integration test** — `verify/m2.8.sh`, output diffed against `expected/m2.8.txt`:
|
||||
1. `a1_vault_from_empty` — delete vault, rebuild, `git diff --exit-code` empty.
|
||||
2. `a2_db_from_empty` — truncate, rebuild, counts per level equal the log's.
|
||||
3. `a3_verify_clean` — `mem verify` exits 0.
|
||||
4. `a4_rebuild_idempotent` — rebuild again, diff still empty, row count unchanged.
|
||||
5. `a5_no_controller_calls` — assert zero calls to the chat route during rebuild
|
||||
(count via the record dir from M1.1, or a proxy).
|
||||
6. `a6_projection_independence` — `--vault-only` then `--db-only` produces the
|
||||
same end state as a combined rebuild.
|
||||
7. `a7_log_alone_suffices` — move the log to a fresh checkout with no vault and no
|
||||
database, rebuild, diff against the committed vault. The strongest form of
|
||||
the claim.
|
||||
|
||||
**Command:** `bash verify/m2.8.sh | diff - expected/m2.8.txt`
|
||||
|
||||
**False pass:**
|
||||
- Running the gate without deleting the vault first. A projector that only writes
|
||||
changed files produces an empty diff trivially, and the hidden input survives.
|
||||
- Diffing an untracked vault. `git diff` on untracked files reports nothing, so
|
||||
the assertion passes vacuously. The vault must be committed, or the script must
|
||||
compare against a committed golden tree explicitly.
|
||||
- Allowing controller calls "because it is easier". Rebuild then produces new
|
||||
memory text each run and the gate can never pass — at which point the usual fix
|
||||
is to weaken the gate.
|
||||
|
||||
## Traps
|
||||
|
||||
- Treating a non-empty diff as a projector bug by default. It is equally likely to
|
||||
be a *hash* bug: if `sha256` includes a timestamp (M0.2), every rebuild produces
|
||||
new nodes and the vault churns. Check identity before blaming rendering.
|
||||
- Running against production data with the destructive script and no backup. The
|
||||
log is the record; if that is intact, everything is recoverable — which is
|
||||
exactly why the log must be tracked in git before this gate is first run.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — Authority model, Verification
|
||||
@@ -1,178 +0,0 @@
|
||||
# M3.5.10 — Auth integration with Authentik/Vault
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M3.5 — Distributed API Layer |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | — |
|
||||
| Depends | M3.5.1 |
|
||||
|
||||
## Goal
|
||||
|
||||
Replace the placeholder `apikey` header check with proper authentication via the
|
||||
cluster's IAM stack: **Authentik** (OIDC provider) → **HashiCorp Vault** (token
|
||||
issuer) → **memory service** (token validator).
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
**Current (wrong):**
|
||||
```rust
|
||||
fn check_auth(req: &HttpRequest, state: &AppState) -> Result<(), HttpResponse> {
|
||||
let api_key = req.headers().get("apikey").and_then(|h| h.to_str().ok());
|
||||
if api_key != Some(&state.api_key) {
|
||||
return Err(HttpResponse::Unauthorized().json(...));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
This is a raw string match against `MEM_API_KEY` env var. No JWT, no Vault, no
|
||||
user identity. It does not integrate with the cluster's IAM stack.
|
||||
|
||||
**Cluster IAM stack:**
|
||||
- **Authentik** (`iam` namespace) — OIDC provider at
|
||||
`https://authentik.riotpiao.com/application/o/vault/`
|
||||
- **HashiCorp Vault** (`iam` namespace) — OIDC auth method enabled, validates
|
||||
Authentik JWTs, issues Vault tokens based on role/policy.
|
||||
- **Vault OIDC role:** `auth/oidc/role/homelab-admin`
|
||||
- `bound_claims: { "permissions": "*" }`
|
||||
- Policy: `homelab-admin` (path `"*"` full access)
|
||||
- **Vault unseal:** Shamir 3/3, keys in `vault-unseal-keys` secret, S3 backend
|
||||
via MinIO.
|
||||
|
||||
**Auth flow (production):**
|
||||
```
|
||||
User/Agent authenticates with Authentik (OIDC)
|
||||
→ Receives JWT with claims { sub, permissions, groups, ... }
|
||||
→ Presents JWT to Vault OIDC auth method
|
||||
→ Vault validates JWT against Authentik JWKS
|
||||
→ Vault issues Vault token with matched policy
|
||||
→ Client sends Vault token to memory service
|
||||
→ Memory service validates token via Vault API
|
||||
```
|
||||
|
||||
**Three integration options (pick one):**
|
||||
|
||||
### Option A: Vault token validation (recommended)
|
||||
Memory service receives `X-Vault-Token` header, calls Vault's
|
||||
`POST /v1/auth/token/lookup-self` to validate. Extracts policy and metadata.
|
||||
- Pro: Vault is the single source of truth for authorization.
|
||||
- Pro: Token revocation is immediate (Vault controls lifecycle).
|
||||
- Con: Extra network call per request (cache with TTL to mitigate).
|
||||
|
||||
### Option B: Direct JWKS validation
|
||||
Memory service fetches Authentik's JWKS endpoint, validates JWT `Authorization:
|
||||
Bearer <token>` directly. No Vault in the request path.
|
||||
- Pro: No Vault dependency at request time.
|
||||
- Pro: Standard OAuth2/OIDC pattern.
|
||||
- Con: Token revocation is delayed (until JWT expires).
|
||||
- Con: Memory service must know about Authentik's OIDC config.
|
||||
|
||||
### Option C: Trust gateway
|
||||
Memory service trusts homelab-frontend gateway (cluster-internal traffic).
|
||||
Gateway validates auth, forwards `X-User-Id` and `X-Capabilities` headers.
|
||||
Memory service checks capabilities against ServiceAdapter CRD requirements.
|
||||
- Pro: Auth logic centralized in gateway.
|
||||
- Pro: Memory service stays simple.
|
||||
- Con: Gateway auth is currently a stub (`hasCapability()` returns true for any
|
||||
`Authorization` header).
|
||||
- Con: Requires gateway auth to be completed first (homelab-frontend task 8.3).
|
||||
|
||||
**ServiceAdapter CRD for memory (`memory-adapter`):**
|
||||
```yaml
|
||||
auth:
|
||||
capability: memory:read # default
|
||||
required: true
|
||||
resources:
|
||||
- name: ingest
|
||||
methods:
|
||||
- verb: POST
|
||||
auth: { capability: memory:write, required: true }
|
||||
- name: query
|
||||
methods:
|
||||
- verb: POST
|
||||
- name: skills
|
||||
methods:
|
||||
- verb: GET
|
||||
```
|
||||
|
||||
**Capabilities needed:**
|
||||
- `memory:read` — query, skills, vault browse, projects, sources
|
||||
- `memory:write` — ingest, source sync, skill draft
|
||||
|
||||
## Steps
|
||||
|
||||
### Option A (Vault token — recommended)
|
||||
|
||||
1. Add `vault_addr` to `AppState` (default: `http://vault.iam.svc.cluster.local:8200`).
|
||||
2. Replace `check_auth()` with `validate_vault_token()`:
|
||||
```rust
|
||||
async fn validate_vault_token(req: &HttpRequest, state: &AppState) -> Result<VaultIdentity, HttpResponse> {
|
||||
let token = req.headers().get("X-Vault-Token")
|
||||
.or_else(|| req.headers().get("Authorization")) // Bearer <token>
|
||||
.and_then(|h| h.to_str().ok());
|
||||
// POST vault_addr/v1/auth/token/lookup-self with X-Vault-Token header
|
||||
// Parse response: policies, metadata, ttl
|
||||
// Cache token -> identity for TTL duration
|
||||
}
|
||||
```
|
||||
3. Add token cache (HashMap<token_hash, (VaultIdentity, Instant)>) with configurable TTL.
|
||||
4. Extract `VaultIdentity` (policies, metadata) from lookup response.
|
||||
5. Map policies to capabilities: `homelab-admin` → `memory:read` + `memory:write`.
|
||||
6. Update each handler to check required capability.
|
||||
7. Keep `apikey` as fallback for dev/test (controlled by env var `MEM_AUTH_MODE=vault|apikey`).
|
||||
|
||||
### For all options
|
||||
|
||||
8. Add env vars: `VAULT_ADDR`, `MEM_AUTH_MODE` (vault/jwks/gateway/apikey).
|
||||
9. Update K8s deployment to inject `VAULT_ADDR`.
|
||||
10. Update ServiceAdapter CRD if needed.
|
||||
11. Document auth flow in README.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Requests with valid Vault token are accepted.
|
||||
- Requests with expired/revoked Vault token are rejected (401).
|
||||
- Requests without any auth are rejected (401).
|
||||
- `memory:write` capability required for ingest/sync endpoints.
|
||||
- `memory:read` capability sufficient for query/skills/vault endpoints.
|
||||
- Token cache reduces Vault API calls on repeated requests.
|
||||
- Fallback to `apikey` mode for dev/test environments.
|
||||
|
||||
## Verify
|
||||
|
||||
**Integration test** — `tests/it_auth_integration.rs`:
|
||||
1. `a1_vault_token_accepted` — mock Vault lookup-self returning valid response;
|
||||
assert request proceeds.
|
||||
2. `a2_expired_token_rejected` — mock Vault returning 403; assert 401 response.
|
||||
3. `a3_no_auth_rejected` — request with no auth headers; assert 401.
|
||||
4. `a4_write_requires_capability` — token with `memory:read` only; POST /ingest;
|
||||
assert 403.
|
||||
5. `a5_read_with_read_capability` — token with `memory:read`; GET /query;
|
||||
assert proceeds.
|
||||
6. `a6_token_cache_hit` — same token twice; assert Vault called once.
|
||||
7. `a7_apikey_fallback` — `MEM_AUTH_MODE=apikey`; assert old behavior works.
|
||||
8. `a8_auth_mode_configurable` — assert `MEM_AUTH_MODE` switches validation logic.
|
||||
|
||||
**Command:** `cargo test --test it_auth_integration`
|
||||
|
||||
**False pass:**
|
||||
- Testing only with apikey fallback. The Vault integration is the whole point.
|
||||
- Mocking Vault without testing cache expiry. A cache that never expires accepts
|
||||
revoked tokens forever.
|
||||
|
||||
## Traps
|
||||
|
||||
- Calling Vault on every request without caching. Vault API calls add 5-10ms
|
||||
per request. Cache with TTL matching token TTL (or shorter).
|
||||
- Not handling Vault being temporarily unreachable. Return 503 (not 401) if
|
||||
Vault is down — "cannot verify" is not "unauthorized".
|
||||
- Hardcoding Vault addr. Use env var + service discovery.
|
||||
- Not supporting `Authorization: Bearer <token>` format alongside `X-Vault-Token`.
|
||||
Different clients use different conventions.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md](../DESIGN.md) — auth section, Authentik/Vault IAM stack
|
||||
@@ -1,178 +0,0 @@
|
||||
# M3.5.9 — Git-aware memory references: lookup by code location
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | M3.5 — Distributed API Layer |
|
||||
| Size | M — 1–3 days |
|
||||
| Status | ⬜ Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | — |
|
||||
| Depends | M3.5.2 (git enrichment in ingest), M3.5.3 (query endpoint) |
|
||||
|
||||
## Goal
|
||||
|
||||
Enable agents to find and cite memory entries by code location (file:line, commit, author). Unifies memory log with git history. Agents reference: `"Per src/kong/buffer.rs:42 (commit abc123)..."` → lookup via git blame, return L0 evidence + L1 memory.
|
||||
|
||||
## Design
|
||||
|
||||
**New database columns** (extend `memory_node` from M2.3):
|
||||
```sql
|
||||
ALTER TABLE memory_node ADD COLUMN git_context JSONB;
|
||||
-- {file, line, commit_sha, commit_msg, author, author_date}
|
||||
|
||||
-- Index for git-based lookup
|
||||
CREATE INDEX ON memory_node USING GIN (git_context);
|
||||
```
|
||||
|
||||
**Three lookup modes:**
|
||||
|
||||
1. **By git location (file:line):**
|
||||
```
|
||||
POST /memory/nodes/by-git
|
||||
{
|
||||
"repo": "github.com/org/poimen",
|
||||
"file": "src/kong/buffer.rs",
|
||||
"line": 42,
|
||||
"project": "poimen"
|
||||
}
|
||||
→ 200 {
|
||||
"nodes": [
|
||||
{
|
||||
"sha256": "...",
|
||||
"level": "L0",
|
||||
"text": "Kong body buffer raised to 16MB...",
|
||||
"git_context": {commit_sha, commit_msg, author},
|
||||
"created_at": "2026-08-20T..."
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
2. **By commit (evidence from this commit):**
|
||||
```
|
||||
POST /memory/nodes/by-commit
|
||||
{
|
||||
"repo": "github.com/org/poimen",
|
||||
"commit_sha": "abc123def",
|
||||
"project": "poimen"
|
||||
}
|
||||
→ nodes from this commit + parent L1/L2 memories
|
||||
```
|
||||
|
||||
3. **By author (what did person X discover):**
|
||||
```
|
||||
POST /memory/nodes/by-author
|
||||
{
|
||||
"author": "[email protected]",
|
||||
"project": "poimen"
|
||||
}
|
||||
→ L0 nodes created during commits from alice
|
||||
```
|
||||
|
||||
**Query endpoint extension** (M3.5.3):
|
||||
Add optional `git_repo` param:
|
||||
```
|
||||
GET /memory/query?query=Kong&git_repo=github.com/org/poimen&project=poimen
|
||||
→ results enriched with git_context (file, commit, author)
|
||||
```
|
||||
|
||||
**Response format (all modes):**
|
||||
```json
|
||||
{
|
||||
"nodes": [
|
||||
{
|
||||
"sha256": "abc...",
|
||||
"level": "L0|L1|L2",
|
||||
"brief": "Kong body buffer raised",
|
||||
"git_ref": "src/kong/buffer.rs:42",
|
||||
"git_commit": {
|
||||
"sha": "abc123def",
|
||||
"message": "Increase body buffer to 16MB",
|
||||
"author": "[email protected]",
|
||||
"date": "2026-08-15T10:30:00Z"
|
||||
},
|
||||
"parents": [...]
|
||||
}
|
||||
],
|
||||
"repo": "github.com/org/poimen"
|
||||
}
|
||||
```
|
||||
|
||||
## Steps
|
||||
|
||||
1. `POST /memory/nodes/by-git` handler:
|
||||
- Parse `file`, `line`, `project`
|
||||
- Query: `SELECT * FROM memory_node WHERE project = $1 AND git_context->>'file' = $2 AND (git_context->>'line')::int = $3`
|
||||
- Walk edges to include parent L1/L2 nodes
|
||||
- Sort by created_at desc
|
||||
|
||||
2. `POST /memory/nodes/by-commit` handler:
|
||||
- Parse `commit_sha`, `project`
|
||||
- Query: `SELECT * FROM memory_node WHERE project = $1 AND git_context->>'commit_sha' = $2`
|
||||
- Include all L0 from this commit + transitive parents (L1/L2)
|
||||
|
||||
3. `POST /memory/nodes/by-author` handler:
|
||||
- Parse `author`, `project`
|
||||
- Query: `SELECT * FROM memory_node WHERE project = $1 AND git_context->>'author' = $2 AND level = 'L0'`
|
||||
- Walk edges to L1 parents
|
||||
|
||||
4. Extend M3.5.3 query handler:
|
||||
- Add optional `git_repo` query param
|
||||
- If provided, enrich response with git_context from each result node
|
||||
- Include `git_ref` in brief (file:line) for agent citation
|
||||
|
||||
5. Deduplication by git:
|
||||
- L0 evidence from same (file, line, commit) = same memory entry
|
||||
- Idempotency: ingesting same commit twice doesn't duplicate L0 nodes
|
||||
- Check: `(file, line, commit_sha)` tuple uniqueness constraint
|
||||
|
||||
## Acceptance
|
||||
|
||||
- `by-git` lookup returns correct L0 evidence + parent memories
|
||||
- `by-commit` returns all evidence from that commit
|
||||
- `by-author` returns all discoveries by that author
|
||||
- Query results enriched with git_context when repo provided
|
||||
- Same evidence never duplicated (idempotent by git tuple)
|
||||
- Agents can cite by code location: "src/kong/buffer.rs:42 (commit abc123)"
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** Integration tests with git history fixture.
|
||||
|
||||
**Setup:** Create test repo with commits:
|
||||
- commit abc123: modify src/kong/buffer.rs:42 (message: "Increase buffer")
|
||||
- commit def456: modify src/kong/handler.rs:10 (message: "Handle large bodies")
|
||||
- Create memory nodes with git_context from these commits
|
||||
|
||||
**Integration test** — `tests/it_git_references.rs`:
|
||||
1. `a1_by_git_lookup` — POST /nodes/by-git with file=buffer.rs, line=42 returns L0 from commit abc123.
|
||||
2. `a2_by_commit_lookup` — POST /nodes/by-commit with abc123 returns both L0 + parent L1/L2.
|
||||
3. `a3_by_author_lookup` — POST /nodes/by-author with alice@org returns all L0 from alice's commits.
|
||||
4. `a4_query_enriched_with_git` — GET /query?query=buffer&git_repo=... returns results with git_context populated.
|
||||
5. `a5_git_ref_in_brief` — result.git_ref = "src/kong/buffer.rs:42" (human-readable).
|
||||
6. `a6_idempotent_by_git_tuple` — ingest same commit twice, L0 nodes count stays 1 (no duplicates).
|
||||
7. `a7_edge_walk_preserves_git` — L1 parent of L0 node includes L0's git_context in parents array.
|
||||
8. `a8_cross_commit_correlation` — two commits affecting same file, both return from by-git lookup (line=0 or range?).
|
||||
9. `a9_author_query_filters_correctly` — two authors, by-author for alice returns only alice's L0.
|
||||
10. `a10_missing_git_context_graceful` — old L0 nodes without git_context (from before M3.5.2) still return but git_ref is null.
|
||||
|
||||
**Command:** `cargo test -p mem-cli git_references`
|
||||
|
||||
**False pass:**
|
||||
- Git context populated in fixture but never actually extracted from repo.git during ingest (M3.5.2). Test only checks stored data, not enrichment.
|
||||
- `by-git` returns results but never walks edges to L1. Parent L1 discoveries are invisible.
|
||||
- Query enrichment tested only with one repo. Multiple repos with overlapping filenames may return wrong results.
|
||||
- Idempotency tested with same commit but different git_repo URLs (github.com vs gitlab.com). Should be treated differently but test may not catch it.
|
||||
|
||||
## Traps
|
||||
|
||||
- Git blame is expensive. Caching blames by (file, commit_sha) pair is necessary for repeated queries.
|
||||
- Line numbers shift with edits. Reference to "line 42" in commit ABC may not match "line 42" in HEAD. Store commit hash, not line number, as primary key.
|
||||
- JSONB queries in PostgreSQL are slower than indexed columns. Consider denormalizing `git_file`, `git_commit`, `git_author` as separate columns if query volume is high.
|
||||
- Author name varies (alice@org vs alice.smith@org). Normalize email in ingest or handle fuzzy matching in by-author.
|
||||
- Cross-repo scenarios: same code in two repos (fork, mirror). git_repo must be part of uniqueness constraint.
|
||||
|
||||
---
|
||||
|
||||
Background: [DESIGN.md § Distributed API Layer](../DESIGN.md#distributed-api-layer-homelab-frontend)
|
||||
Reference in New Issue
Block a user