poimen-memory/DESIGN.md

Poimen Memory — gated recurrent knowledge extraction from agent context

Target repo: /Users/rockliang/workplace/Poimen/memory (empty git repo, no remote, no commits).

Context

Agent sessions accumulate faster than they can be read, and almost all of the volume is noise. One pi session in this project measured:

assistant   1445
toolResult  1261   <- 43%, mostly evidence-free (ls output, file reads)
user         196
+ 8 compaction events

Compaction already fires 8 times per session, which means context is being discarded rather than retained — the knowledge produced (root causes, decisions, gotchas) evaporates when the window rolls. Meanwhile the corpus is already project-partitioned on disk:

• ~/.pi/agent/sessions/--Users-rockliang-workplace-Poimen--/*.jsonl — cwd in the session header gives the project key
• ~/.claude/projects/<dash-encoded-cwd>/<uuid>.jsonl — 15 MB, 7 transcripts
• Poimen/agent-rust/tasks/artifacts/<TaskId>/ — coder/reviewer JSONL per attempt

Intended outcome: a durable, queryable project memory built by reading that history chunk-by-chunk and keeping only what answers standing questions — surfaced as an Obsidian vault a human reads and a pgvector index an agent queries.

The mechanism is GRU-Mem (arXiv 2602.10560). Its two gates map directly onto the problem: the update gate refuses to write evidence-free chunks into memory (the 43% of toolResults), and the exit gate stops the scan once evidence is sufficient. The paper reports up to 400% speedup and better accuracy than ungated recurrent memory, because unchecked memory growth actively degrades later updates.

Verified facts this plan depends on

Fact	Value	How known
Embedding dims	768	probed /v1/embeddings with nomic-ai/nomic-embed-text-v2-moe
pgvector in CNPG	available, v0.7.0, not yet installed	pg_available_extensions on forgejo-db-2, stock image ghcr.io/cloudnative-pg/postgresql:16.2
CNPG operator	1.30.0, supports declarative Database.spec.extensions	kubectl explain database.spec.extensions
Ollama context cap	32768 (OLLAMA_CONTEXT_LENGTH)	k8s/apps/llm-serving/ornith.yaml
Controller model	qwen2.5:3b-instruct	is Qwen2.5-3B-Instruct — the paper's exact 3B backbone
Gateway	nginx ingress + homelab-frontend (Go, ServiceAdapter CRDs)	Kong is removed from cluster
IAM	Authentik (OIDC) → Vault (OIDC auth method)	Authentik issues JWT, Vault validates via OIDC, homelab-admin role bound on permissions: "*" claim
Auth (placeholder)	apikey: header in mem-cli	To be replaced — does not integrate with Authentik/Vault chain

The 32K cap is the binding constraint and it fits: paper uses 5000-token chunks, 8192 max prompt, 2048 max response.

Side fix: ~/.pi/agent/models.json declares contextWindow: 131072 for ornith. Wrong — Ollama caps at 32768, so long prompts truncate silently. Correct to 32768.

The tier model

Memory is levelled, and every event in the log carries its level. The paper has one flat memory; a project knowledge base needs three, plus a fourth tier that sits outside the recurrence entirely (R, below).

Level	What it is	Produced by	Bounded
L0	evidence chunk — the verbatim source span the update gate accepted	update gate opening at L1	no, but sparse (~17 of 412 chunks)
L1	per-query memory — GRU-Mem's M_t for one standing question	gated loop over L0 chunk stream	1024 tokens
L2	project synthesis — memory across the L1 memories of one project	gated loop over L1 memories	1024 tokens
R	reference text — documentation the models are weak at, not evidence of anything	corpus ingest, no gate	no, bounded by corpus size

The tiering is not new machinery. L2 is the same controller, same prompt, same two gates — run with the L1 memories as its chunk stream and a project-level question. The recurrence is the algorithm applied to its own output, so mem-core implements one loop and the level is a parameter. Two consequences worth having on purpose:

• Exit gate flips by level. Off at L1 (see below), reasonably on at L2, where the input is a handful of memories rather than hundreds of chunks and "enough evidence" is actually decidable.
• Levels form a provenance graph, not a pile. Each L1 node records the L0 nodes that produced it; each L2 node records its L1 parents. That graph is the Obsidian link structure and the parent_id edges in Postgres — one relationship expressed in both projections.

Architecture

                            api.riotpiao.com (nginx ingress + homelab-frontend gateway)
                                    │
                  ┌─────────────────┼─────────────────┐
                  │                 │                 │
            /ingest            /query              /skills
              (async)          (sync)           (read-only)
                  │                 │                 │
                  ▼                 ▼                 ▼
        ┌─────────────────────────────────────────────────────┐
        │  API Server (Rust httpd)  mem-store / mem-llm      │
        │  - ingest_id dedup + queue                          │
        │  - query → HNSW + rerank + edge-walk               │
        │  - skill catalog (excludes _drafts)                │
        └──────────────────┬──────────────────────────────────┘
                          │
        pi/claude CLI ─────┼────── agents in-session
        local or CI/CD     │     (embedded queries)
                          │
       ┌──────────────────┴──────────────────┐
       │                                     │
       ▼                                     ▼
  [ Ingest Queue ]                   [ CNPG Cluster ]
  (redis or local)                   (pgvector, HNSW)
       │                                     │
       ├─────────────────────────────────────┤
       │
       ▼
pi sessions / claude transcripts / loop.sh artifacts
        │
        ▼  project resolver (cwd -> project id)
   [ Chunk Parser ]  5000-token chunks, split on message boundaries
        │
        ▼
┌──────────────────────────────────────────────────────┐
│  GRU-Mem Controller   qwen2.5:3b-instruct            │
│  <think>  reason about chunk vs question             │
│  <check>  yes|no   -> update gate U_t                │
│  <update> candidate memory M̂_t                       │
│  <next>   continue|end -> exit gate E_t              │
└───────────────┬──────────────────────────────────────┘
      U_t=yes   │   U_t=no
        ┌───────┴────────┐
        ▼                ▼
  emit L0 evidence   discard chunk
  M_t <- M̂_t         M_t <- M_{t-1}
        │
        ▼  L1 memory per standing query
        └──────► same loop, input = L1 memories ──► L2 project synthesis
        │
        ▼
┌──────────────────────────────────────────────────────┐
│  JSONL event log         AUTHORITATIVE               │
│  Obsidian vault          derived projection          │
│  pgvector index          derived projection          │
└──────────────────────────────────────────────────────┘

Authority invariant unchanged: API is stateless demultiplexer. JSONL log is authoritative; vault and pgvector projections are droppable. API caches read-only state (embeddings, L2 synthesis); ingest writes only to log.

Authority model — the load-bearing decision. The JSONL log is the only source of truth. The vault and the vector index are projections that must be droppable and rebuildable byte-identically from the log. This is poimen's own §1 principle ("nothing derived is authoritative; if it cannot be dropped and rebuilt, it has hidden inputs and that is a bug") applied here, and it buys three things: re-embedding after a model change is a rebuild not a migration, Obsidian edits cannot corrupt the record, and the post-training corpus is the log itself.

Standing queries

The paper's memory agent is φθ(Q, C_t, M_{t-1}) — it requires a question. The update gate is defined as "does this chunk contain useful information about the problem". Without Q the gate has no referent, and r_update becomes undefinable, which forecloses post-training.

So each project declares durable questions in YAML. One query = one L1 memory = one Obsidian note.

# queries/poimen.yaml
project: poimen
sources:
  - pi:--Users-rockliang-workplace-Poimen-agent-rust--
  - claude:-Users-rockliang-workplace-Poimen
queries:
  - id: architecture-decisions
    question: What architectural decisions were made, with reasoning and rejected alternatives?
  - id: infra-root-causes
    question: What infrastructure bugs were found, what was the root cause, how was it isolated?
  - id: open-questions
    question: What questions were raised and left unresolved?
synthesis:                      # the L2 pass
  question: What is the current state of this project, and what should someone know before working on it?
  exit_gate: true

Exit gate defaults off at L1. Paper §3.3 makes this call itself: for "what are all the X" questions you cannot know evidence is sufficient without reading everything, so they provide a w/o-EG inference mode. L1 extraction is exactly that shape. Keep the gate recorded (its signal is needed for RL) but do not act on it. Enable at L2 and for interactive retrieval, where the paper measures 4× speedup.

Separate weights — yes, specifically a LoRA adapter

Confirming the instinct, with the reason that actually matters here:

• Base: Qwen2.5-3B-Instruct, already resident as qwen2.5:3b-instruct
• Memory policy: LoRA adapter, rank 16–32, ~30–60 MB

Why an adapter rather than a fine-tuned model:

1. VRAM. One GPU, OLLAMA_MAX_LOADED_MODELS=2, currently holding ornith:35b + qwen2.5:3b. A separate full memory model evicts something, and eviction here is a weights reload measured in tens of seconds — we already watched ornith cold-start blow a 60s gateway timeout. A LoRA rides on the resident base for near-zero extra VRAM.
2. Iteration. Post-training emits a ~50 MB adapter, not a 6 GB model. Swap without redeploying.
3. Reversibility. A gate-behaviour regression reverts by pointing at the previous adapter.

Infra consequence to accept up front: Ollama cannot hot-swap LoRA adapters. Serving one means either moving the memory model to a vLLM InferenceService with --enable-lora (the reasoning predictor is already vLLM v0.11.0, so the pattern exists), or merging into a GGUF for Ollama and losing swappability. P1–P4 run prompted-only on the stock model, so this is deferred, not dodged.

Repository layout

Rust workspace at Poimen/memory:

Cargo.toml                    workspace
crates/
  mem-core/     domain types; Level enum; gate-response parser; the gated loop
  mem-chunk/    RecordSource trait; chunking policy; flush triggers  (see below)
  mem-llm/      gateway client — chat completions, embeddings, rerank
  mem-ingest/   source adapters: pi sessions, claude transcripts, loop.sh artifacts
  mem-store/    JSONL log writer/reader; pgvector repo; Obsidian projector
  mem-cli/      binary `mem`: ingest | synthesize | rebuild | query | verify | skill | label
memory-tasks/   the task board — INDEX.md + one file per task, tracked
queries/        standing query YAML, one file per project
vault/          Obsidian output (own git repo or gitignored)
log/            JSONL event log — authoritative, tracked

Crates: sqlx (postgres, runtime-tokio-rustls) + pgvector (sqlx feature), tokenizers for chunk sizing against the real Qwen2 tokenizer, futures, serde/serde_json, clap, reqwest.

Reuse rather than reinvent: mem-llm's request shape and the apikey header convention are proven in agent-rust/loop.sh; response parsing mirrors its extract_text().

mem-chunk — its own crate, stream-shaped from day one

Chunking is split out because it is the seam where new input kinds arrive. Sources today are files with an EOF; telemetry, a live session tail, or a broker will not have one. Designing the boundary as a stream now means a future source implements a trait rather than forcing the loop to be rewritten — and it costs nothing, since the rest of the stack is already tokio (sqlx runtime-tokio, reqwest).

// mem-chunk
pub trait RecordSource {
    /// Normalised records: role, text, timestamp, provenance. Sources decide
    /// how to produce them; the chunker never learns about pi vs claude vs a socket.
    fn records(self) -> impl Stream<Item = Result<Record>>;
}

pub struct ChunkPolicy {
    pub max_tokens: usize,        // 5000, paper default
    pub split_on: Boundary,       // never mid-message
    pub flush: FlushTrigger,      // see below
}

pub fn chunks<S: RecordSource>(src: S, p: ChunkPolicy) -> impl Stream<Item = Chunk>;

Batch sources become streams for free via futures::stream::iter, so mem-ingest gets no more complex today.

The one thing that genuinely differs for streams is the flush trigger. A file chunker emits a partial chunk at EOF; a stream has no EOF, so a partially-filled chunk would sit forever. FlushTrigger is therefore Tokens(n) today and gains OrIdle(Duration) when a stream source lands — carrying it in the policy now means the later change is one enum variant, not a signature change through the loop.

Worth noting for whenever that happens: the exit gate changes meaning on an unbounded source. At L1 over a finite transcript it is switched off because "read everything" is well defined (paper §3.3). Over a live stream there is no everything, so the gate stops being an optimisation and becomes the only termination condition — which is an argument for keeping it trained even while it is switched off.

Storage schemas

JSONL event log — authoritative

log/<project>/<query-id>/<run-id>.jsonl. Every record carries level.

{"type":"run","level":"L1","project":"poimen","query_id":"infra-root-causes","input_level":"chunk","model":"qwen2.5:3b-instruct","chunk_tokens":5000,"memory_budget":1024,"exit_gate":false,"ts":"..."}
{"type":"chunk","level":"L0","t":1,"source":"pi:2026-07-21T16-23-59_019f857d","span":[0,42],"sha256":"..."}
{"type":"gate","level":"L1","t":1,"update":false,"exit":false,"think":"...","latency_ms":812}
{"type":"evidence","level":"L0","t":7,"source":"pi:...","text":"...","sha256":"..."}
{"type":"memory","level":"L1","t":7,"text":"...","tokens":142,"parents":["<L0 sha>"],"sha256":"..."}
{"type":"run_end","level":"L1","chunks_seen":412,"chunks_used":17,"final_memory_sha":"..."}

The L2 pass writes the same record types with "level":"L2", "input_level":"L1", and parents holding L1 shas. evidence records appear only when the update gate opened, so update-rate is directly measurable and memory at any t is replayable.

pgvector — projection

One table across all levels, because retrieval wants to search them together and filter:

CREATE TABLE memory_node (
  id        BIGSERIAL PRIMARY KEY,
  level     TEXT NOT NULL CHECK (level IN ('L0','L1','L2','R')),
  project   TEXT NOT NULL,
  query_id  TEXT,                        -- null at L2 and R
  run_id    TEXT NOT NULL,
  t         INT  NOT NULL,
  source    TEXT,                        -- set at L0; source URI at R
  text      TEXT NOT NULL,
  sha256    TEXT NOT NULL UNIQUE,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE memory_edge (               -- provenance: child <- parent
  child_sha  TEXT NOT NULL REFERENCES memory_node(sha256),
  parent_sha TEXT NOT NULL REFERENCES memory_node(sha256),
  PRIMARY KEY (child_sha, parent_sha)
  -- No row may point at an R node as parent. R is not evidence; see
  -- "Reference corpora" below and the M3.6.6 assertion that enforces it.
);
CREATE INDEX ON memory_node (project, level);

-- One node, several vectors. 'symptom' is a generated projection describing the
-- failures a memory would explain -- see Retrieval below for why one vector per
-- node does not work.
CREATE TABLE memory_vector (
  node_sha  TEXT NOT NULL REFERENCES memory_node(sha256) ON DELETE CASCADE,
  kind      TEXT NOT NULL CHECK (kind IN ('text','symptom')),
  embedding vector(768) NOT NULL,
  PRIMARY KEY (node_sha, kind)
);
-- Partial per kind: one index over both forces post-filtering and starves
-- recall. The predicate must be a literal or the planner ignores the index.
CREATE INDEX ON memory_vector USING hnsw (embedding vector_cosine_ops) WHERE kind = 'text';
CREATE INDEX ON memory_vector USING hnsw (embedding vector_cosine_ops) WHERE kind = 'symptom';

-- Exact-match tier. Failures repeat verbatim; prose does not.
CREATE TABLE failure_signature (
  sig_sha    TEXT PRIMARY KEY,           -- hash of the NORMALISED signature
  node_sha   TEXT NOT NULL REFERENCES memory_node(sha256) ON DELETE CASCADE,
  tool       TEXT NOT NULL,
  raw        TEXT NOT NULL,
  seen_count INT  NOT NULL DEFAULT 1,    -- a fold over log occurrences, not state
  last_seen  TIMESTAMPTZ NOT NULL
);

-- A lesson about Kong is wrong now, not merely old.
CREATE TABLE memory_supersede (
  old_sha TEXT NOT NULL REFERENCES memory_node(sha256) ON DELETE CASCADE,
  new_sha TEXT NOT NULL REFERENCES memory_node(sha256) ON DELETE CASCADE,
  reason  TEXT,
  PRIMARY KEY (old_sha, new_sha)
);

erDiagram
    MEMORY_NODE ||--o{ MEMORY_EDGE : "child_sha -> sha256"
    MEMORY_NODE ||--o{ MEMORY_EDGE : "parent_sha -> sha256"

    MEMORY_NODE {
        bigserial id PK
        text level "L0 | L1 | L2"
        text project
        text query_id "NULL at L2"
        text run_id
        int t
        text source "set at L0"
        text text
        text sha256 UK "content identity"
        vector_768 embedding
        timestamptz created_at
    }

    MEMORY_EDGE {
        text child_sha PK,FK
        text parent_sha PK,FK
    }

One table across L0/L1/L2 (not three) — retrieval searches all levels together and filters by level. memory_edge is the provenance graph: L1 rows point back at the L0 chunks that produced them, L2 rows point back at L1 parents. sha256 UNIQUE is content identity (dedup key, hash excludes run id/timestamp — M0.2) and is what memory_edge actually references, not the surrogate id.

Retrieval: HNSW recall filtered by level, then bge-reranker-base via /v1/rerank for precision — that endpoint scored 0.98 vs 0.00009 on a discrimination probe, so it earns its place. Default query searches L1+L2 and walks memory_edge down to L0 for citations.

Infra — new CNPG cluster with the extension managed declaratively (CNPG 1.30 supports this, so no manual psql, consistent with the GitOps hard rule). Follows k8s/infra/databases/temporal-db.yaml exactly:

# k8s/infra/databases/memory-db.yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata: { name: memory-db, namespace: memory }
spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:16.2
  bootstrap: { initdb: { database: memory, owner: app, encoding: UTF8, localeCollate: C, localeCType: C } }
  enableSuperuserAccess: false
  storage: { size: 10Gi, storageClass: longhorn-cnpg }
  monitoring: { enablePodMonitor: true }
  affinity:
    podAntiAffinityType: preferred
    tolerations: [{ key: node-role.kubernetes.io/control-plane, operator: Exists, effect: NoSchedule }]
---
apiVersion: postgresql.cnpg.io/v1
kind: Database
metadata: { name: memory-db-vector, namespace: memory }
spec:
  name: memory
  owner: app
  cluster: { name: memory-db }
  extensions: [{ name: vector, ensure: present }]

Obsidian vault — projection

The tier graph becomes the note graph:

vault/poimen/
  index.md                     <- L2 synthesis, links to every L1 note
  infra-root-causes.md         <- L1
  architecture-decisions.md    <- L1
  evidence/                    <- L0, optional (--emit-evidence-notes, default off)
vault/skills/
  _drafts/<name>/SKILL.md      <- machine-written, never auto-loaded
  <name>/SKILL.md              <- human-promoted, loadable

---
project: poimen
level: L1
query_id: infra-root-causes
updated: 2026-08-17
chunks_seen: 412
chunks_used: 17
---
# Infra root causes — poimen

<final memory text>

## Provenance
- [[pi-2026-07-21-019f857d]] chunk 66 — Kong body buffer

L0 defaults to inline citations rather than notes — 17 per query is manageable but grows unbounded across projects. The flag exists for when the graph view is worth the file count.

Skills — the procedural projection

A skill is a projection, not a level. L0/L1/L2 are all descriptive — what happened. A skill is procedural — what to do next time. That is a change of modality, not a further compression, so the gated loop does not produce it: the update gate's question ("does this chunk contain evidence for Q") has no meaning when the output is an instruction.

The format is free. SKILL.md is YAML frontmatter plus markdown, which is exactly an Obsidian note — verified against ~/.claude/skills/seo-geo-claude-skills/research/keyword-research/SKILL.md, whose frontmatter carries name, description, when_to_use, argument-hint. So vault/skills/<name>/SKILL.md is simultaneously a vault note and a loadable skill, with no conversion step:

pi --skill vault/skills/            # or set skillsPath in ~/.pi/config.json
ln -s .../vault/skills/<name> ~/.claude/skills/<name>

mem skill draft --from poimen/infra-root-causes reads an L1 or L2 note and writes a draft. Emission should follow the existing authoring rubric rather than inventing one — the installed grafana-core:skill-authoring skill encodes Anthropic's Agent Skills guidance and a four-dimension rubric (conciseness, actionability, workflow clarity, progressive disclosure). Description quality is the whole game: a skill whose description does not match how the user actually phrases the request never fires.

Drafts are never auto-loaded, and promotion is manual. This is the one place the system can close a loop on itself, and the failure is subtle:

a memory-derived skill is auto-loaded → it appears in future session transcripts → those transcripts are ingested as evidence → the memory that produced the skill is reinforced by its own output

No external verifier breaks that cycle. It is the same hazard poimen §16 names when it gates "automatic workflow mutation without human approval" by default, and it is why _drafts/ is a separate directory rather than a frontmatter flag — a directory cannot be accidentally globbed into --skill.

Two mechanical guards:

1. Promotion is a human move out of _drafts/, reviewable as a diff.
2. Provenance marks derived text. Every emitted skill carries generated_from: <L2 sha> in frontmatter, and mem-ingest tags chunks matching a known emitted artifact as derived: true and excludes them from evidence. Without this the corpus slowly becomes its own training data.

Source connectors — extensible multi-source ingestion

Memory service is a cluster-wide RAG serving multiple agents and workflows. Knowledge does not live in one place — it is spread across an Obsidian vault, a paperless-ngx instance, agent session logs, and whatever document stores appear next. The connector architecture makes adding a new source a single-trait implementation rather than a pipeline rewrite.

The problem with hardcoded sources

Today mem-ingest has three concrete sources: PiSessionSource, ClaudeTranscriptSource, DocCorpusSource. Each knows its own file format and emits Records through the RecordSource trait. This works, but:

1. Every new source requires Rust code. paperless-ngx has a REST API; Confluence has another; S3 is a third protocol. Each lands as a new .rs file compiled into the binary.
2. No runtime discovery. The CLI must know about every source at compile time. A homelab that adds Bookstack next month needs a code change, a rebuild, and a redeploy.
3. No shared sync logic. Change detection (sha-based skip), tombstoning, drift reporting — each source will reimplement these unless the framework provides them.

Design: SourceConnector trait + registry

A connector is anything that can enumerate documents and yield their content. The framework handles chunking, embedding, change detection, and storage.

/// A connector to an external document source.
#[async_trait]
pub trait SourceConnector: Send + Sync {
    /// Unique connector kind identifier (e.g., "obsidian", "paperless", "s3")
    fn kind(&self) -> &str;

    /// Human-readable name for this connector instance
    fn name(&self) -> &str;

    /// Enumerate all documents available from this source.
    /// Returns (doc_id, doc_metadata) pairs.
    async fn list_documents(&self) -> Result<Vec<SourceDocument>>;

    /// Fetch content for a single document by its source-specific ID.
    async fn fetch_document(&self, doc_id: &str) -> Result<DocumentContent>;

    /// Health check — can we reach this source?
    async fn health_check(&self) -> Result<SourceHealth>;
}

/// Metadata about a source document (before fetching content)
pub struct SourceDocument {
    pub doc_id: String,           // source-specific unique ID
    pub title: String,            // human-readable title
    pub source_uri: String,       // canonical URI (file://, https://, paperless://)
    pub content_hash: Option<String>,  // if source provides hash, skip fetch on match
    pub mime_type: String,        // text/markdown, application/pdf, etc.
    pub updated_at: Option<OffsetDateTime>,
}

/// Fetched document content ready for chunking
pub struct DocumentContent {
    pub doc_id: String,
    pub text: String,             // extracted text (markdown preferred)
    pub source_uri: String,
    pub content_hash: String,     // sha256 of text content
    pub metadata: HashMap<String, String>,  // source-specific metadata
}

/// Source health status
pub struct SourceHealth {
    pub reachable: bool,
    pub document_count: Option<u64>,
    pub last_error: Option<String>,
}

Connector registry

Connectors are registered at startup from a YAML config file. Adding a new source is: implement the trait, register the kind, add a config block.

# connectors.yaml — source connector configuration
connectors:
  - kind: obsidian
    name: homelab-vault
    config:
      root: /data/vault                # local path (PVC mount or git-sync)
      extensions: [md, markdown, txt]
      exclude_dirs: [.obsidian, .trash]

  - kind: paperless
    name: homelab-paperless
    config:
      base_url: http://paperless-ngx.paperless.svc.cluster.local:8000
      token_secret: paperless-api-token  # k8s secret name
      tags: [reference, manual, runbook] # only sync docs with these tags
      format: markdown                   # request markdown export

  - kind: git_repo
    name: infra-docs
    config:
      repo_url: https://forgejo.riotpiao.com/rock/homelab-docs.git
      branch: main
      sync_interval: 1h
      paths: [docs/, runbooks/]

  - kind: s3
    name: backup-docs
    config:
      endpoint: https://minio.riotpiao.com
      bucket: knowledge-base
      prefix: docs/
      access_key_secret: minio-credentials

Sync framework

The connector trait provides enumeration and fetch. The sync framework handles everything else:

1. Change detection: Compare content_hash from list_documents() against last-known hash in the manifest. Skip unchanged docs (zero embed calls).
2. Chunking: Route through existing ChunkPolicy — heading-based for markdown, paragraph-based for plain text, configurable per connector.
3. Tombstoning: Documents that vanish from list_documents() get tombstone records in the log. Append-only, never delete.
4. Drift reporting: mem source status shows per-connector: docs total, changed, new, removed — without mutating anything.
5. Rate limiting: Configurable fetch concurrency per connector (don't DDoS paperless with 500 parallel fetches).
6. Resume: Sync is resumable. Crash mid-sync, restart, only unseen docs are processed.

mem source sync --all                    # sync all registered connectors
mem source sync --name homelab-vault     # sync one connector
mem source status                        # drift report, no mutations
mem source list                          # registered connectors + health
mem source add --kind paperless ...      # register new connector
mem source rm --name old-source          # deregister + tombstone

Built-in connectors (shipped with binary)

Kind	Source	Protocol	Status
obsidian	Local Obsidian vault	filesystem (walkdir)	M7.2
paperless	paperless-ngx	REST API (/api/documents/)	M7.3
git_repo	Git repository	git clone/pull	M7.4
s3	S3-compatible storage	S3 API (minio, AWS)	M7.5
pi_session	Pi agent sessions	filesystem (existing)	M7.1 (wrap existing)
claude	Claude transcripts	filesystem (existing)	M7.1 (wrap existing)

How connectors interact with levels

Session connectors (pi, claude) produce evidence that flows through the gated loop → L0/L1/L2. These are the existing RecordSource implementations, wrapped in SourceConnector for unified management.

Document connectors (obsidian, paperless, git, s3) produce reference material → Level R. These bypass the gated loop entirely (no standing question, no gate decision). This is the existing M3.6 design, now generalized.

The connector kind determines the pipeline:

Session connector → RecordSource → ChunkPolicy → GatedLoop → L0/L1/L2
Document connector → SourceConnector → ChunkPolicy → Level R (no gate)

paperless-ngx integration (concrete example)

paperless-ngx is already running in the cluster. It has:

• REST API at http://paperless-ngx.paperless.svc.cluster.local:8000/api/
• Documents with tags, correspondents, document types
• Full-text content available via API
• Thumbnail and original file access

pub struct PaperlessConnector {
    base_url: String,
    token: String,
    tag_filter: Vec<String>,
}

#[async_trait]
impl SourceConnector for PaperlessConnector {
    fn kind(&self) -> &str { "paperless" }
    fn name(&self) -> &str { &self.name }

    async fn list_documents(&self) -> Result<Vec<SourceDocument>> {
        // GET /api/documents/?tags__name__in=reference,manual
        // Paginate through results
        // Return doc_id, title, checksum (paperless provides this)
    }

    async fn fetch_document(&self, doc_id: &str) -> Result<DocumentContent> {
        // GET /api/documents/{id}/
        // Extract content field (full text)
        // Or GET /api/documents/{id}/download/ for original
    }

    async fn health_check(&self) -> Result<SourceHealth> {
        // GET /api/ — check 200
    }
}

Obsidian vault as a connector

The Obsidian vault is the primary reference source. It is human-maintained, git-backed, and the canonical location for runbooks, procedures, and domain knowledge that agents need.

Deployment options:

1. Git-sync sidecar: A sidecar container clones the vault repo into a shared PVC. Memory service reads from the PVC.
2. Local mount: For development, mount the vault directory directly.
3. API upload: Push vault changes to memory service via HTTP.

The Obsidian connector wraps the existing DocCorpusSource (M3.6.1) with the SourceConnector interface, adding change detection and registry management.

Future extensibility

Adding a new source requires:

1. Implement SourceConnector trait (~100-200 lines)
2. Register the kind in the connector factory
3. Add config block to connectors.yaml
4. Run mem source sync --name new-source

No changes to the ingest pipeline, chunking, embedding, storage, or query layers. The connector is the only new code.

Reference corpora — the non-evidential tier

RecordSource takes session transcripts, and the update gate asks "does this chunk contain evidence for Q". A kubectl or tea cheatsheet answers neither question: it has no session, no turn, and no evidence. Left alone the system faithfully retains what happened when a model used a tool badly and never learns the tool. Level R closes that gap, and the shape of the fix matters more than the fact of it.

Reference text bypasses the gated loop entirely. It is not evidence, so it gets no gate decision, never becomes L0, and never parents an L1. It is embedded, indexed, retrievable, and inert with respect to the recurrence. The update-rate that M1.8 watches must not move when a corpus is added — M3.6.6 asserts exactly that, because a design where documentation quietly enters the gate is the memory-explosion failure wearing a different hat.

Four rules, each with an assertion behind it:

1. R is never a parent. An L1 -> R or L2 -> R edge is a bug, not a provenance nuance — it lets upstream doc prose be cited as evidence for what happened in this cluster. mem verify rejects it.
2. R is opt-in at query time. Default levels stay L1,L2. --levels R is an explicit ask. A default that blends manual pages into project answers makes the memory sound like documentation, which is precisely what the tier model exists to prevent.
3. The log stays authoritative. Corpus ingest writes Reference records carrying source URI and content sha; the pgvector rows and vault notes are projections and must survive mem rebuild --from-log byte-identically, same as every other level.
4. Re-ingest replaces, never appends. Upstream docs change. Identity is (source_uri, sha256): an unchanged sha is a no-op, a changed one tombstones the prior node in the log and writes its successor. Skip this and the index accumulates every historical revision of a cheatsheet, with recall drifting toward the oldest copy.

The fork is structural, not a flag. run_loop takes a Query, and M1.2 makes an empty question a load error — the update gate is defined as "does this chunk contain useful information about the problem", so with no Q it has no referent. A corpus has no standing question, therefore a corpus ingest cannot construct a legal call into the recurrence at all. mem ingest and mem ref add are separate pipelines sharing RecordSource and mem-chunk and diverging before the controller. Encode that in the types — a reference chunk has no run_loop overload — rather than a skip_gate: bool, which is one careless default away from feeding documentation to the gate.

Why that matters to M1.8 specifically. Update-rate is chunks_used / chunks_seen over a run, and the M1 gate fails above 30%. Documentation is evidence-free with respect to almost any project question, so routing a corpus through the controller would lower update-rate and make the threshold easier to clear while the system got worse. A gate metric that improves when you add unrelated text has stopped measuring what it claims. M3.6.6 asserts that a corpus ingest leaves chunks_seen untouched, and that is the assertion protecting M1.8 from being gamed by accident.

The retrieval cycle is the skills cycle wearing a different hat. A retrieved R section lands in an agent's context, appears verbatim in that session's transcript, and returns as L0 evidence — the exact loop M4.2 exists to break, with upstream docs in place of emitted skills. R text therefore registers in the same artifact manifest M4.2 reads, and this is why the phase is ordered after skills rather than before: the shingle matcher already exists by then. Without it, memory learns the man page as though it were a project finding.

No exit gate, no memory budget, no M_t. R has none of the recurrence's state. Nothing about a corpus is bounded by 1024 tokens, nothing records E_t, and nothing appears in the M5 training corpus as a gate decision — because no decision was made. A corpus that shows up in mem label output is a bug in the export filter, not a labelling question.

Abstention comes with it. A corpus multiplies the documents that are somewhat related to any question, so an unconditional top-k starts returning confident-looking prose for questions the memory cannot answer. mem query gains a relevance floor: below threshold it reports insufficient recall and returns nothing rather than the least-bad row. Worth having independently of R; M3.6.5 builds it here because R is what makes it urgent.

What this does not do. It does not make a 35B model competent at a tool in the abstract — retrieval puts the right page in context, nothing more. Skills (M4) remain the procedural path, and a skill drafted from a session where the tool actually failed still beats a retrieved man page. R is the floor, not the ceiling.

Tool context — the assembly surface

The consumer that made this necessary is the orchestrator (Poimen/workflows): its ImplementerActivity renders a prompt ending in "use the available tools to implement this task" while naming none, and PrepareSkillsActivity clones one static skill list for every task regardless of what the task is. Both models behind it — reasoning and ornith:35b — are then asked to operate tooling they were never told about. Memory owns the fix because the alternative is a second retrieval stack inside the orchestrator, indexing the same corpus against the same embedder, drifting immediately.

Memory describes tools. It never executes them. The orchestrator holds the sandbox, the credentials and the blast radius. This repo holds the catalog, the knowledge and the history.

No tool catalog lives here. An earlier draft had memory serving MCP schemas so a prompt builder could enumerate tools. That is duplication — pi and any other caller already hold their own MCP connections and schemas, and a second copy drifts. Memory answers what do we know about this, keyed by a tool name, a task, or a raw failure. The caller knows what tools it has.

Retrieval: three tiers, cheapest first

The input is usually not a question. It is a 50KB CI log, or a tool name. Three problems follow, and plain top-k cosine handles none of them.

Query/document asymmetry. An L1 is written as an answer — "requests over 10KB failed because Kong buffered the body; fixed with proxy-body-size: 0". The query arrives as a symptom — 413 Request Entity Too Large. Same incident, different register, mediocre cosine neighbours. This is the main reason retrieval that passes its unit tests disappoints in use. The fix is a second vector per memory (kind='symptom', M3.7.8) generated at write time, describing the failures that memory would explain. Query-time HyDE solves the same problem by putting an LLM call on every lookup; writes are rare here because the gate keeps acceptance sparse, so paying once at write is the right side of the trade.

The input needs reducing before it can be embedded. Signature extraction (M3.7.7) strips run ids, timestamps, workspace paths, shas, line numbers and durations, then hashes. Normalisation quality decides whether the exact tier ever fires — and when it silently does not, vector search still returns something, so the failure is invisible without the ablation the gate runs.

Failures repeat verbatim; prose does not. npm ERR! ERESOLVE unable to resolve dependency tree is byte-identical across occurrences, so it deserves a hash lookup rather than a vector search.

Tier	Mechanism	What a hit means
1	sig_sha primary key on failure_signature	this exact failure happened here before
2	vector over kind='symptom', then kind='text', reranked	something similar happened
3	R reference corpus	nobody here has hit this; here are the docs

Tier 1 leads, it does not short-circuit — an exact hit plus two related memories beats an exact hit alone, and the extra tiers cost milliseconds against the caller's own inference. The tier is a field in the response, because "we hit this in July" and "the manual says" must not arrive in the same register.

Scope differs by tier. Signature and symptom lookups federate across projects — an ERESOLVE lesson is not homelab-specific — while task-shaped queries stay project-scoped. Project match is a rank boost, not a filter.

Superseded memories are excluded, not demoted. Kong is retired; a lesson about its buffer settings is wrong rather than stale, and memory_supersede surfaces the successor instead.

The bundle is a composition, and stores nothing new

POST /memory/context takes any of tool, task, signature_source and merges the tiers above with matched skills. Each leg degrades independently — a skills timeout returns [] and a 200. No leg failure justifies a 5xx: a thinner answer beats no answer when someone is mid-incident.

Learned beats documented, and that ordering is the whole point

When a task mentions kubectl, the bundle must surface "--all is not a flag — it failed on 2026-08-19, the working form was --all-namespaces" above the generic cheatsheet section. L1/L2 outrank R at equal rerank score, deliberately and by rule.

Without that ordering this whole repo reduces to a documentation server, and the gated recurrence — the expensive part, the part with a 3B controller and a post-training phase behind it — contributes nothing at the moment a task is actually being implemented. R is the fallback for what nobody here has learned yet.

The feedback loop is the actual answer to "make ornith better at tools"

A cheatsheet is a floor. The mechanism that improves is this one:

implementer emits a bad invocation
        → fails, judge rejects, lessons injected, retry
        → session ingested at end
        → standing query `tool-failures` gates it in as evidence   <- real evidence, unlike docs
        → L1 memory: what failed, the error, the working form
        → next task's /memory/context surfaces it above the docs

Note where this sits relative to the gate: tool failures are genuine evidence about what happened in this project, so unlike reference corpora they belong inside the recurrence and pass through the update gate normally. No bypass, no special casing — the only new artifact is a standing question (M3.7.5) whose answers happen to be operationally useful at task time.

Budget is a hard contract, not a hope

The bundle is injected into every implementer prompt and OLLAMA_CONTEXT_LENGTH is 32768 — a verified constraint above, not a theoretical one. The bundle carries an explicit token budget with a fixed truncation order:

1. drop tier 3 (R) first — upstream docs are the most replaceable content here
2. then trim tier 2 toward the relevance floor
3. then drop skills
4. never drop tier 1 — an exact prior occurrence is the smallest and most valuable thing in the response

A response that cannot fit its tier-1 hits inside the budget is an error, not a truncation.

Phases

P1 — Read-only spine. mem-ingest implements RecordSource for pi sessions and Claude transcripts; mem-chunk chunks to 5000 tokens on message boundaries; mem ingest --dry-run prints the chunk plan with no model calls. Both sources are batch, but they go through the stream interface so the seam is exercised from the first commit rather than retrofitted.

P2 — Gated loop at L1, prompted only. mem-llm + gate-response parser (paper Figure 10a prompt, adapted for standing queries). Writes the JSONL log with L0 evidence and L1 memory records. Exit gate recorded, not acted on. This is the paper's "w/o RL" baseline, which Figure 9 shows already works.

P3 — Projections. Obsidian projector and pgvector repo, both rebuildable via mem rebuild --from-log. Infra commit for memory-db.

P4 — L2 synthesis and retrieval. mem synthesize runs the same loop over L1 memories with the exit gate on. mem query — embed, HNSW recall by level, rerank, return with provenance walked through memory_edge.

P5 — Skill drafting. mem skill draft --from <note> emits vault/skills/_drafts/<name>/SKILL.md with generated_from provenance; mem-ingest grows the derived: true exclusion filter. Promotion stays manual. Cheap to build and it is the phase that makes the memory do something rather than only be read.

P5.5 — Reference corpora (board M3.6, ordered after skills). DocCorpusSource implements RecordSource over a documentation tree, chunked on heading boundaries rather than message boundaries; R nodes land in log, index and vault; mem ref add/list/sync/rm manages corpora with replace-on-change identity; R text registers in M4.2's artifact manifest so retrieved docs cannot re-enter as evidence; mem query gains filter-then-recall over levels and a relevance floor. Ordered after P5 for two reasons: skills are the better answer to the same problem and should be built first, and the cycle guard is an extension of M4.2 rather than a parallel mechanism. The gate proves update-rate is unmoved, no L1 acquired an R parent, and no R text reached the controller.

P5.6 — Tool context (board M3.7). Signature extraction reduces a failure log to a stable hash; a symptom projection gives every L1/L2 a second vector so an error message can find an answer written as prose; POST /memory/context serves the three tiers — exact signature, symptom similarity, reference docs — with skills matched alongside, under a hard budget. A tool-failures standing query feeds real invocation failures back through the gate, so tier 2 becomes tier 1 the second time something breaks. Consumers are pi, curl, or an MCP call; this phase ships no tool execution and no tool catalog.

P6 — Post-training (separate, Python). Boundary is the JSONL. mem label uses the 32B reasoning model as an offline evidence labeler to produce per-chunk U_t ground truth (the paper had synthetic NIAH labels; we do not, and this is the honest cheapest substitute). Then verl trains a LoRA with the paper's rewards: r_update ±1, r_exit {0, −0.5 late, −0.75 early}, strict r_format, α=0.9 mixing trajectory- and turn-level advantage. Requires the vLLM decision above.

P7 — Source connectors (board M7). Generalizes the ingestion layer from hardcoded file sources to an extensible SourceConnector trait with a YAML-driven registry. Existing sources (pi sessions, Claude transcripts, DocCorpusSource) are wrapped in the new interface; new sources (paperless-ngx, git repos, S3) implement the trait directly. The sync framework handles change detection, tombstoning, drift reporting, and resumable sync for all connectors. The connector is the only new code when a source is added — no changes to chunking, embedding, storage, or query layers. Ordered after M3.6 because it generalizes mem ref rather than duplicating it, and after M4.2 because document connectors must register in the derived-content manifest to prevent cycle contamination.

Task breakdown

Board lives in memory-tasks/ at the repo root. Format follows agent-rust/tasks/: one file per task, self-contained, each with Acceptance / Verify (harness, numbered assertions, command) / False pass / Traps, a Status field as source of truth, and memory-tasks/INDEX.md mirroring it. Ids are M<phase>.<n> and frozen once written — phase order is declared in INDEX.md, never derived from the id.

memory-tasks/
  INDEX.md                          board + phase order + progress mirror
  M0.1-cargo-workspace.md
  M0.2-domain-types.md
  ...
  M5.6-m5-gate.md

Note agent-rust/.gitignore excludes tasks, which silently untracks the whole board there. Naming this memory-tasks/ sidesteps that pattern — and it should be tracked, since the task files carry the acceptance criteria.

M0 — Read-only spine (no model calls anywhere in this phase)

id	task	size	deps
M0.1	Cargo workspace + six crate skeletons, CI builds clean	S	—
M0.2	mem-core domain types: Level, Record, Chunk, MemoryNode, sha256 identity	S	M0.1
M0.3	mem-chunk: RecordSource trait, ChunkPolicy, FlushTrigger::Tokens	M	M0.2
M0.4	Tokenizer-backed sizing against the real Qwen2 tokenizer	M	M0.3
M0.5	mem-ingest: pi session adapter (~/.pi/agent/sessions/<encoded-cwd>/*.jsonl)	M	M0.3
M0.6	mem-ingest: claude transcript adapter (~/.claude/projects/**/<uuid>.jsonl)	S	M0.5
M0.7	mem ingest --dry-run — chunk plan, token histogram, source breakdown	S	M0.4, M0.6
M0.8	M0 gate — both adapters through one RecordSource, no source-specific code past the trait	M	gate

M1 — Gated loop at L1

id	task	size	deps
M1.1	mem-llm chat client — apikey header, retry, timeout	M	M0.1
M1.2	Standing-query YAML loader + schema validation, unresolved id fails at load	M	M0.2
M1.3	GRU-Mem prompt template (paper Fig 10a), memory + chunk + question assembly	M	M1.2
M1.4	Gate-response parser: <think>/<check>/<update>/<next>, strict, malformed = hard error	M	M1.3
M1.5	The gated loop — U_t mutate-or-retain, E_t recorded not acted on, 1024-token budget	L	M1.4
M1.6	JSONL event log writer, level on every record, parents on memory	M	M1.5
M1.7	mem ingest end to end + update-rate reported on stdout	M	M1.6
M1.8	M1 gate — full run on poimen, update-rate < 30%, memory tokens flat not climbing	M	gate

M2 — Projections

id	task	size	deps
M2.1	mem-llm embeddings client, 768-dim, batched	S	M1.1
M2.2	k8s/infra/databases/memory-db.yaml — CNPG Cluster + Database with extensions: [vector]	M	—
M2.3	memory_node / memory_edge schema + sqlx migrations, HNSW indexes	M	M2.2
M2.4	mem-store pgvector repo — upsert by sha, edge insert	M	M2.3, M2.1
M2.5	Obsidian projector — frontmatter, wikilinks, L0 citations, --emit-evidence-notes	M	M1.6
M2.6	mem rebuild --from-log — drop and rebuild both projections	M	M2.4, M2.5
M2.7	mem verify — every L1 has ≥1 L0 parent, every parent sha resolves	S	M2.6
M2.8	M2 gate — rebuild is byte-identical (git -C vault diff --exit-code empty)	M	gate

M3 — L2 synthesis and retrieval

id	task	size	deps
M3.1	L2 pass — same loop, input Stream<L1 memory>, exit gate on	M	M1.5
M3.2	mem-llm rerank client (bge-reranker-base)	S	M1.1
M3.3	mem query — embed, HNSW recall filtered by level, rerank, walk edges to L0	M	M3.2, M2.4
M3.4	M3 gate — known-answer query returns the right L1 node with a real L0 citation	M	gate

M4 — Skills

id	task	size	deps
M4.1	mem skill draft --from <note> → _drafts/, frontmatter incl. generated_from	M	M3.1
M4.2	derived: true ingest filter — emitted artifacts excluded from evidence	M	M4.1, M0.5
M4.3	M4 gate — draft absent from --list-skills; no L0 node matches an emitted artifact	M	gate

M3.5 — Distributed API Layer (Homelab Frontend integration)

id	task	size	deps
M3.5.1	HTTP server + router (actix-web or axum), auth hook, request metrics	M	M0.1
M3.5.2	POST /ingest endpoint — ingest_id dedup, async queue, git context enrichment	M	M1.7, M3.5.1
M3.5.3	GET /query endpoint — embed query, HNSW recall by level, rerank, walk edges to L0	M	M3.3, M3.5.1
M3.5.4	Federation: single query across projects, fan+merge results, deduplicate	M	M3.5.3
M3.5.5	GET /skills and /skills/{name} — loadable skills only, exclude _drafts, YAML frontmatter in JSON	M	M4.1, M3.5.1
M3.5.6	GET /projects and /projects/{id}/status — metadata, metrics, synthesis timestamps	S	M3.5.1
M3.5.7	Rate limiting (apikey-scoped per endpoint) + idempotency by sha256	M	M3.5.2
M3.5.8	M3.5 composition gate — end-to-end ingest→query via HTTP, load from cli and from agent simul	M	gate
M3.5.9	Git-aware references: lookup by code location (file:line, commit, author)	M	M3.5.2

M5 — Post-training (Python, separate from the Rust workspace; boundary is the JSONL)

id	task	size	deps
M5.1	mem label — 32B reasoning as offline evidence labeler, writes U_t ground truth	M	M1.6
M5.2	Labeler calibration — hand-label a holdout, measure agreement before trusting it	M	M5.1
M5.3	Training corpus export from the log to verl's expected format	M	M5.1
M5.4	vLLM InferenceService for Qwen2.5-3B with --enable-lora (GitOps, homelab)	L	—
M5.5	verl loop — r_update ±1, r_exit {0,−0.5,−0.75}, strict r_format, α=0.9	L	M5.3, M5.4
M5.6	M5 gate — adapter beats prompted baseline on held-out update accuracy	L	gate

M7 — Source connectors (extensible multi-source ingestion)

id	task	size	deps
M7.1	SourceConnector trait + SourceDocument / DocumentContent types + connector registry	M	M0.3, M3.6.1
M7.2	Obsidian vault connector — wraps DocCorpusSource, adds change detection, config-driven	M	M7.1, M3.6.1
M7.3	paperless-ngx connector — REST API client, tag filtering, markdown export	M	M7.1
M7.4	Git repo connector — clone/pull, path filtering, branch tracking	M	M7.1
M7.5	S3 connector — S3-compatible API, prefix filtering, content-type routing	M	M7.1
M7.6	Sync framework — change detection, tombstoning, drift report, resume	L	M7.1, M3.6.3
M7.7	mem source CLI — sync/status/list/add/rm subcommands	M	M7.6
M7.8	GET /memory/sources + POST /memory/sources/sync HTTP endpoints	M	M7.7, M3.5.1
M7.9	Connector health monitoring + observability	S	M7.8
M7.10	M7 composition gate — two connectors sync, change detection skips unchanged, tombstone works, rebuild parity	M	gate

Total 74 tasks, 11 gates. M0 and M2.2 have no model dependency and can start immediately; M5.4 is homelab work independent of everything else in M5 and can run in parallel. M3.5 depends on M2 (pgvector store exists) and M1 (ingest loop exists); can run in parallel with M4 and M5. M3.5.9 (git-aware references) is optional, depends on M3.5.2. M7 depends on M3.6 (reference corpora) and M4.2 (derived filter) for cycle prevention.

Verification

# P1 — corpus parses, chunk plan sane, zero model calls
cargo run -p mem-cli -- ingest --project poimen --dry-run

# P2 — one query end to end
cargo run -p mem-cli -- ingest --project poimen --query infra-root-causes
jq -r 'select(.type=="gate" and .level=="L1") | .update' log/poimen/infra-root-causes/*.jsonl | sort | uniq -c
#   expect: far more false than true. Update-rate > ~30% means the gate is not
#   discriminating — that is the paper's memory-explosion failure, Figure 6 is
#   the reference shape.
jq -r 'select(.type=="memory") | .tokens' log/.../*.jsonl | tail -1
#   expect: <= 1024 and roughly flat over t, not monotonically climbing

# levels are well-formed and edges close
jq -r '.level' log/poimen/**/*.jsonl | sort | uniq -c        # L0/L1 present
cargo run -p mem-cli -- verify --project poimen
#   asserts: every L1 memory has >=1 L0 parent; every parent sha exists

# P3 — projections truly derived
cargo run -p mem-cli -- rebuild --from-log --project poimen
git -C vault diff --exit-code        # empty: rebuild is byte-identical
psql -c "select level, count(*) from memory_node group by level;"

# P3.5 — API server online
cargo run -p mem-cli -- serve --port 8080 &
sleep 1
curl -H "apikey: test-key" http://localhost:8080/memory/projects
#   expect: ["poimen", ...]
curl -H "apikey: test-key" \
  "http://localhost:8080/memory/query?query=kong+body&level=L1,L2&project=poimen&limit=3"
#   expect: 200, array of memory nodes with score + parents
#
# ingest via HTTP (async):
jq -n '{project:"poimen", source:"test:local", records:[...]}' | \
  curl -X POST -H "apikey: test-key" \
    http://localhost:8080/memory/ingest -d @-
#   expect: 202, {"job_id": "ingest-<uuid>", "status_url": "/memory/ingest/ingest-<uuid>"}
#
# idempotency: same request twice with same ingest_id returns same job_id, no re-enqueue
# rate limit: 11th req in 1 second gets 429 Retry-After
# auth missing: 401 Unauthorized

# P4 — synthesis and retrieval
cargo run -p mem-cli -- synthesize --project poimen     # expect exit gate to fire
cargo run -p mem-cli -- query "why did requests over 10KB fail?"
#   expect: infra-root-causes L1 node, Kong body-buffer passage, L0 citation
#
# Via API (same result):
curl -H "apikey: test-key" \
  "http://localhost:8080/memory/query?query=why+did+requests+over+10KB+fail"
#   expect: identical results

# P5 — skill drafts land unloadable, and the cycle stays open
cargo run -p mem-cli -- skill draft --from poimen/infra-root-causes
ls vault/skills/_drafts/                    # draft here, NOT in vault/skills/
pi --skill vault/skills/ --list-skills      # draft must not appear
curl -H "apikey: test-key" http://localhost:8080/memory/skills
#   expect: no drafts in list
cargo run -p mem-cli -- verify --derived-filter --project poimen
#   asserts: no L0 evidence node text matches an emitted skill artifact

The decisive P2 metric is update-rate, the one number distinguishing a working gate from an expensive summarizer. toolResults are 43% of records and mostly evidence-free, so a correct gate rejects the large majority of chunks.

P3.5 API gate: Ingest and query work over HTTP with correct idempotency, auth, and rate limiting. CLI and agents both submit to same endpoint; no duplication or ordering issues.

Distributed API Layer (Homelab Frontend)

Gateway: api.riotpiao.com routes requests through nginx ingress to homelab-frontend gateway (Go, ServiceAdapter CRDs). Kong is removed from the cluster.

Auth: Authentik (OIDC provider) → Vault (OIDC auth method, auth/oidc/role/homelab-admin, bound on permissions: "*" claim) → Vault token → service validates token. Memory service currently uses placeholder apikey header — M3.5.10 replaces this with Vault token validation.

Architecture assumption: Memory services run in CNPG cluster; API layer is HTTP facade exposing read/write workflows to distributed agents. Authority remains JSONL—API is a request demultiplexer, not a cache or alternative source of truth.

REST API Endpoints

POST   /memory/ingest                  <- async, idempotent by sha256
GET    /memory/query                   <- semantic search + rerank
GET    /memory/projects                <- list projects with L2 synthesis
GET    /memory/projects/{id}/status    <- ingest/synthesis status
GET    /memory/projects/{id}/notes     <- L1/L2 notes (Obsidian export)
GET    /memory/skills                  <- loadable skills (excludes _drafts)
GET    /memory/skills/{name}           <- one skill frontmatter + body
POST   /memory/context                 <- 3-tier lookup: signature, symptom, docs

Request/Response contract:

# POST /memory/ingest (idempotent, async)
{"project": "poimen", "source": "agent:uuid", "records": [...], "ingest_id": "sha256-of-batch"}
→ 202 Accepted
  {"job_id": "ingest-<uuid>", "ingest_id": "...", "status_url": "/memory/ingest/ingest-<uuid>"}

# GET /memory/query (semantic search)
{"query": "why did requests over 10KB fail?", "level": ["L1", "L2"], "project": "poimen", "limit": 5}
→ 200 OK
  [
    {"level": "L1", "sha256": "...", "text": "...", "score": 0.92,
     "parents": [{"level": "L0", "source": "pi:...", "text": "..."}]},
    ...
  ]

# GET /memory/skills?loadable=true
→ 200 OK
  [
    {"name": "infra-root-causes", "description": "...", "when_to_use": "...",
     "generated_from": null, "promoted_at": "2026-08-20"}
  ]

Distributed Behavior

Ingestion: mem-ingest CLI submits batches to POST /memory/ingest via ingest_id (sha256 of batch text). Duplicate ingest_id returns same job_id without re-enqueuing — jobs are idempotent by content hash, not request. Server stores the mapping; HTTP 409 means already ingested (user caller resubmits without retry).

Query federation: Agents query single endpoint; server fans requests to appropriate project (selected by metadata or query text). Results walk memory_edge down to L0 server-side, so client gets complete citation graph in one round-trip.

Skills as cargo: GET /memory/skills returns YAML frontmatter in JSON so agent UIs can inspect description and when_to_use without fetching the file. Body is optional (fetch separately if needed to load).

Status & observability:

• GET /memory/projects/{id}/status → {"last_ingest": "...", "chunks_total": N, "chunks_used": M, "synthesis_ran": "...", "next_synthesis_at": "..."}
• Metrics: ingest latency (p50/p99), query latency, update-rate per project, memory size trends

Scaling Constraints

Single points of failure:

• CNPG cluster (mitigated by ≥3 replicas + Longhorn)
• Ollama inference (separate from memory store; ingest is offline, query caches embeddings)

Throughput limits:

• Ingest: one gated loop per project sequentially (5000 tokens/chunk, gate latency 812ms); ~7 chunks/min = 35k tokens/min per project
• Query: HNSW recall is O(log n), rerank O(k log k), each << embedding roundtrip to Ollama (typically 200ms)

Caching strategy:

• Memory nodes are immutable (sha256 content hash) — safe to cache indefinitely post-write
• L2 synthesis is project-scoped and regenerated on mem synthesize — TTL 1h or explicit purge
• Embeddings cached per-query hash (same embedding twice = cache hit, save 200ms Ollama call)
• Client-side: ETag: <sha256> on all read endpoints, no conditional logic server-side (it's stateless)

Auth & rate limits:

Auth chain (production):

User/Agent → Authentik (OIDC login)
  → JWT with claims (including `permissions`)
  → Vault validates via OIDC auth method
  → Vault issues token based on role match
  → Services validate Vault token or trust gateway-forwarded identity

Authentik OIDC: https://authentik.riotpiao.com/application/o/vault/
Vault OIDC role: auth/oidc/role/homelab-admin
  bound_claims: { "permissions": "*" }
  policy: homelab-admin (path "*" full access)

Current (placeholder, to be replaced):

• apikey: header, raw string match — does not integrate with Authentik/Vault
• Must be replaced with one of:
  1. Vault token validation (call Vault's auth/token/lookup-self)
  2. Authentik JWT validation via JWKS
  3. Trust gateway-forwarded headers (X-User-Id, X-Capabilities)

Rate limits (unchanged):

• Per-key limits: ingest 100 jobs/hour, query 1000 req/hour, skill fetch unlimited
• Burst allowance: 10 req/sec per key (ingest waits in queue; query returns 429 Retry-After if burst exceeded)

Git-Aware References (M3.5.9)

Memory entries are anchored in code. Agents reference by git location, not sha256.

Ingest enrichment (M3.5.2): If repo.git available, auto-populate:

"git_context": {
  "file": "src/kong/buffer.rs",
  "line": 42,
  "commit_sha": "abc123def",
  "commit_msg": "Increase body buffer to 16MB",
  "author": "[email protected]",
  "author_date": "2026-08-15T10:30:00Z"
}

Lookup endpoints (M3.5.9):

• POST /memory/nodes/by-git — find evidence by (file, line)
• POST /memory/nodes/by-commit — all discoveries in this commit
• POST /memory/nodes/by-author — what did this person find
• GET /memory/query?git_repo=github.com/org/poimen — enrich results with git context

Agent citation: "Per src/kong/buffer.rs:42 (commit abc123): ..." instead of sha256.

Integration with Existing Flows

From mem-cli (local or CI/CD):

mem ingest --project poimen --query infra-root-causes --gateway https://api.riotpiao.com --git-repo /path/to/repo/.git

Client computes ingest_id locally (sha256 of all records), enriches with git context, submits batch, polls /memory/ingest/<job_id> until done.

From agents (in-session via Pi or Claude):

# Query within agent:
curl -H "apikey: $MEM_APIKEY" \
  "https://api.riotpiao.com/memory/query?query=why+did+X+fail&project=poimen&level=L1,L2"
  
# Ingest at session end:
{session_transcript_chunk} | curl -X POST -H "apikey: $MEM_APIKEY" \
  https://api.riotpiao.com/memory/ingest \
  -d @- -H "Content-Type: application/jsonl"

Skill loading in agent systems:

# Discovery:
curl -H "apikey: $MEM_APIKEY" https://api.riotpiao.com/memory/skills?loadable=true \
  | jq -r '.[] | .name' | xargs -I {} \
    curl https://api.riotpiao.com/memory/skills/{} > ~/.claude/skills/{}/SKILL.md

Error Taxonomy

200 OK — query succeeded, memory node found (or empty result)
202 Accepted — ingest accepted, job queued
204 No Content — query matched no nodes; not an error
400 Bad Request — malformed query or invalid project/level
401 Unauthorized — missing/invalid apikey
409 Conflict — ingest_id already processed (idempotent, safe retry)
429 Too Many Requests — rate limit exceeded, Retry-After header set
500 Internal Server Error — CNPG offline or embedding service down
503 Service Unavailable — gated loop busy (queue building), retry in 5s

Risks

• 3B gate quality unmeasured on this corpus. The paper evaluates on QA benchmarks with clean evidence labels; agent transcripts are messier. Mitigation: P2's update-rate is a cheap early read, and the 32B reasoning model can spot-audit a sample before committing to P5.
• L2 inherits L1's errors with no path back to source. Synthesis over memories cannot recover evidence the L1 gate wrongly discarded. memory_edge makes the omission visible (an L1 note with suspiciously few parents) but not recoverable without a re-run.
• Self-reinforcement through skills. The only cycle in the system: emitted skill → future session context → ingested as evidence → reinforces the memory that emitted it. Guarded by manual promotion plus the derived: true ingest filter, and both must hold. Audit it by checking that no L0 evidence node's text matches an emitted artifact.
• No ground-truth evidence labels. r_update needs them. Distant supervision from the 32B labeler inherits its bias; hold out a hand-labelled set to measure agreement before trusting it.
• Vault/log divergence. Hand edits are overwritten on rebuild. Either make the vault read-only or add an ## Notes region the projector preserves. Decide before anyone starts editing.
• Reference corpora are inert by construction, and that is a real limit. M3.6 makes documentation retrievable as level R, but R never becomes evidence and never parents an L1, so it can improve recall and nothing else — synthesis quality is untouched by adding a corpus. Tool competence still arrives mainly through M4 skills drafted from real sessions. Skipping M3.6 entirely leaves a working system that simply has nothing to say about a tool until someone has used it badly in a logged session.
• R inflates the index against a fixed recall width. mem query recalls 10×k before reranking. A large corpus competes for those slots with genuine L1/L2 answers even when R is excluded by the level filter, unless the filter is pushed into the HNSW query rather than applied after it. Filter-then-recall, not recall-then-filter; M3.6.4 asserts the ordering.
• Ollama has no LoRA path. P5 forces the vLLM decision. Do not discover this at P5.
• API latency at scale. Query federation fans requests to multiple projects; slowest project wins. Mitigation: query timeout 5s, client-side fallback to local JSONL search, async synthesis keeps L2 warm (cache hit 95%+).
• Ingest race on concurrent writes. Two agents submit overlapping session chunks to same project simultaneously. Mitigation: ingest_id based on content hash prevents duplicate evidence in log; gated loop is single-threaded per project, queues serialize. Allowed cost: cold-start ingest delay ~5m for backlog.