archive: Delete completed task files (M2.1,2.2,2.4,2.5,2.6,M8.1)

Completed tasks moved to git history for archive: - M2.1 Embeddings client (768-dim batching) - M2.2 CNPG memory-db manifest - M2.4 pgvector repository - M2.5 Obsidian projector - M2.6 Rebuild from log orchestrator - M8.1 OpenSearch cluster deployment Remaining in /tasks/: 48 files (active/in-progress/not-started) ✅ Completed: 49/73 (index.md source of truth) 🟡 In progress: 2 (M3.5.9, M3.7.5) ⬜ Not started: 22
2026-08-27 21:18:58 -07:00
parent 807579e8f2
commit 24a03fd9eb
6 changed files with 0 additions and 639 deletions
@@ -1,86 +0,0 @@
 # M2.1 — Embeddings client
 | Field | Value |
 |---|---|
 | Phase | M2 — Projections |
 | Size | S — under 1 day |
 | Status | ✅ Done |
 | Flags | — |
 | Spec | inlined below |
 | Blocks | M1.1 |
 ## Goal
 Turn text into 768-dim vectors, batched, against the gateway's TEI endpoint.
 ## Facts (inlined — no spec read needed)
 ```
 POST /v1/embeddings
 {"model":"nomic-ai/nomic-embed-text-v2-moe","input":["..."]}
 -> {"object":"list","data":[{"embedding":[...768 floats...]}],"usage":{...}}
 ```
 **768 dimensions**, probed and confirmed. It is the `vector(768)` in the schema
 and in the HNSW index; a model swap is a schema migration, not a config change.
 **Batch limit is 32.** Verified: 1200 inputs returned
 `{"message":"batch size 1200 > maximum allowed batch size 32","code":413}`.
 Chunk the input list accordingly.
 This route currently has **no auth** — no `konghq.com/plugins` annotation, so
 `model-key-auth` never attaches. Send the `apikey` header anyway: the route
 should be fixed, and a client that only works while auth is broken breaks when
 it is fixed.
 Bodies here are large (many texts × long strings) and the Kong buffer is 16m, so
 batching also keeps requests well inside it.
 ## Steps
 1. `EmbeddingsClient::embed(texts: &[String]) -> Result<Vec<Vec<f32>>>` in `mem-llm`.
 2. Split into batches of ≤32, preserving input order in the output.
 3. Assert every returned vector is exactly 768 long; a mismatch is an error
   naming the model, not a silent pad or truncate.
 4. Reuse M1.1's client config: `apikey` header, retry on 5xx only, generous
   timeout.
 5. `embed_one` convenience wrapper.
 ## Acceptance
 - 100 texts return 100 vectors in input order.
 - Every vector is 768-dim.
 - A dimension mismatch errors loudly.
 ## Verify
 **Harness:** `wiremock` offline, one `#[ignore]` live test.
 **Integration test** — `tests/it_embeddings.rs`:
 1. `a1_batches_at_32` — 100 inputs produce exactly 4 requests.
 2. `a2_order_preserved` — mock returns identifiable vectors; assert output order
   matches input order across batch boundaries.
 3. `a3_dimension_asserted` — mock returns a 512-dim vector; assert an error
   naming the model.
 4. `a4_apikey_sent` — assert the header is present even though the route does not
   require it.
 5. `a5_live_dims` — `#[ignore]`; real gateway, assert 768.
 **Command:** `cargo test -p mem-llm embeddings` (add `-- --ignored` for a5)
 **False pass:**
 - Testing with ≤32 inputs. The batching path never runs and order-across-batches
  — the thing most likely to be wrong — is never exercised.
 - Trusting the response order within a batch without asserting it. Assertion 2
  must use distinguishable vectors, not a length check.
 ## Traps
 - Hardcoding 768 in three places. Put it in one constant that the schema
  migration also references, so a model change is one edit and one migration.
 - Assuming no auth is needed because it currently works without a key. That route
  is missing its plugin annotation, which is a bug scheduled to be fixed.
 ---
 Background: [DESIGN.md](../DESIGN.md) — Verified facts, pgvector
@@ -1,102 +0,0 @@
 # M2.2 — CNPG `memory-db` + pgvector
 | Field | Value |
 |---|---|
 | Phase | M2 — Projections |
 | Size | M — 1–3 days |
 | Status | ✅ Done |
 | Flags | homelab |
 | Spec | inlined below |
 | Blocks | — |
 ## Goal
 A Postgres with pgvector, provisioned the way everything else in the cluster is:
 through git, with no manual `psql`.
 ## Facts (inlined — no spec read needed)
 **pgvector needs no custom image.** Verified on the running cluster:
 ```
 $ psql -tAc "select name,default_version,installed_version
             from pg_available_extensions where name='vector'"
 vector|0.7.0|
 ```
 on the stock `ghcr.io/cloudnative-pg/postgresql:16.2`. Available, not yet
 installed — `CREATE EXTENSION` is all that is missing.
 **The operator is CNPG 1.30.0**, which supports declarative extensions on the
 `Database` CRD (`kubectl explain database.spec.extensions` resolves). So the
 extension is git-managed too — no manual step, consistent with the GitOps rule
 that infrastructure changes flow through version control.
 Follow `k8s/infra/databases/temporal-db.yaml` exactly: 3 instances, `imageName`
 pinned, `enableSuperuserAccess: false`, `storageClass: longhorn-cnpg`,
 `enablePodMonitor: true`, control-plane tolerations, `podAntiAffinityType:
 preferred`.
 Storage: 10Gi matches the existing clusters. At 768 dims × 4 bytes, a vector is
 ~3 KB; tens of thousands of nodes is well under a gigabyte, so 10Gi is generous
 and consistent rather than tight.
 ## Steps
 1. `k8s/infra/databases/memory-db.yaml` — `Cluster` + `Database` with
   `extensions: [{name: vector, ensure: present}]`.
 2. Namespace `memory`, created by the ArgoCD app that owns it.
 3. Add to the owning kustomization's explicit resource list — an unlisted file is
   silently dropped with no error and no drift shown.
 4. Commit, push, let ArgoCD sync. **No `kubectl apply`.**
 5. Verify the extension installed and the app user can create tables.
 6. Record the connection string convention in the repo README; the password comes
   from the CNPG-generated secret, never committed.
 ## Acceptance
 - `Cluster` reaches `Cluster in healthy state` with 3 instances.
 - `select extversion from pg_extension where extname='vector'` returns a version.
 - ArgoCD shows the app `Synced/Healthy`.
 - No manual `psql` was run to get there.
 ## Verify
 **Harness:** `kubectl` and `psql` read-only checks after sync.
 **Integration test** — `verify/m2.2.sh`, output diffed against `expected/m2.2.txt`:
 1. `a1_cluster_healthy` — `kubectl get cluster -n memory memory-db` reports 3/3
   ready.
 2. `a2_extension_installed` — `select extname, extversion from pg_extension where
   extname='vector'` returns one row.
 3. `a3_declarative_not_manual` — `kubectl get database -n memory memory-db-vector
   -o jsonpath='{.spec.extensions}'` shows the declaration, proving it came from
   git.
 4. `a4_argocd_synced` — the owning app is `Synced/Healthy`.
 5. `a5_app_user_can_ddl` — as `app`, `CREATE TABLE t(v vector(768)); DROP TABLE t;`
   succeeds.
 6. `a6_hnsw_available` — `CREATE INDEX ... USING hnsw` on that temp table
   succeeds, proving 0.7.0 has the index type the schema needs.
 **Command:** `bash verify/m2.2.sh | diff - expected/m2.2.txt`
 **False pass:**
 - Checking `pg_available_extensions` instead of `pg_extension`. Available means
  the files are on disk; installed means `CREATE EXTENSION` ran. The whole task
  is the second one.
 - Verifying after a manual `CREATE EXTENSION`. It passes and proves nothing about
  the declarative path, which is the actual deliverable. Assertion 3 is the guard.
 ## Traps
 - Forgetting the kustomization resource list. The file sits in git, ArgoCD reports
  Synced, and the objects never exist — silent, and the failure surfaces later as
  a connection error.
 - Adding `prune: true` semantics without thinking about operator-created children.
  CNPG creates Services, Secrets and PVCs owned by the Cluster; if ArgoCD's
  tracking label propagates to them, prune fights the operator. The
  `llm-serving` app already had to set `prune: false` for exactly this reason.
 ---
 Background: [DESIGN.md](../DESIGN.md) — pgvector · `k8s/infra/databases/temporal-db.yaml`
@@ -1,129 +0,0 @@
 # M2.4 — pgvector repository
 | Field | Value |
 |---|---|
 | Phase | M2 — Projections |
 | Size | M — 1–3 days |
 | Status | ✅ Done |
 | Flags | — |
 | Spec | inlined below |
 | Blocks | M2.3, M2.1 |
 ## Goal
 Write the projection into Postgres idempotently, so rebuild is safe to run at any
 time and produces the same rows.
 ## Facts (inlined — no spec read needed)
 ```rust
 async fn upsert_node(&self, node: &MemoryNode) -> Result<()>;
 async fn upsert_vector(&self, sha: &Sha256Hash, kind: VectorKind, embedding: &[f32]) -> Result<()>;
 async fn insert_edges(&self, child: &Sha256Hash, parents: &[Sha256Hash]) -> Result<()>;
 async fn search(&self, q: &[f32], kind: VectorKind, levels: &[Level],
                project: Scope, k: usize) -> Result<Vec<ScoredNode>>;
 async fn lookup_signature(&self, sig_sha: &str) -> Result<Option<SignatureHit>>;
 async fn parents_of(&self, sha: &Sha256Hash) -> Result<Vec<MemoryNode>>;
 async fn clear_project(&self, project: &ProjectId) -> Result<()>;
 ```
 `upsert_node` is `ON CONFLICT (sha256) DO NOTHING`. Content identity means an
 identical node is the same node; re-running rebuild must not duplicate or churn
 rows. Same for edges on the composite key.
 `search` orders by `embedding <=> $1` — cosine distance, matching the
 `vector_cosine_ops` index. Any other operator silently drops to a seq scan.
 **`kind` must be a literal predicate in the SQL, not a bind parameter, and not a
 filter applied to results.** The indexes are partial (`WHERE kind = 'text'`), and
 the planner only uses a partial index when the query's predicate provably matches
 it. A `WHERE kind = $2` defeats that and silently degrades to a scan over every
 vector of both kinds — the same failure mode as the wrong opclass, and just as
 invisible.
 **`Scope` is not a `ProjectId`.** Tool-failure lookups federate across projects
 because an `ERESOLVE` lesson is not project-specific, while ordinary standing-query
 memories stay scoped. `Scope::Project(id)` filters; `Scope::AllProjects` does not
 and lets project relevance act as a rank boost later instead of a hard filter.
 `lookup_signature` is the exact-match tier: a primary-key hit on
 `failure_signature`, no vector involved. It is the cheapest and highest-precision
 answer the store can give, so it belongs in the repository rather than being
 assembled from a `search` call by a caller who does not know it exists.
 Edges are inserted **after** both endpoints exist, or the foreign key rejects
 them. Rebuild therefore has two passes: all nodes, then all edges. This is not an
 optimisation; a single-pass insert fails on the first forward reference.
 Embeddings are generated in batches of ≤32 (M2.1) and are the expensive part of
 rebuild — batch across nodes, not per node.
 ## Steps
 1. `PgRepo::connect(url)` with a pool; run migrations on connect.
 2. Implement the five methods above.
 3. `upsert_many(nodes)` batching embedding calls at 32 and inserting with a
   multi-row statement. Batch across *both* vector kinds — a node with a symptom
   projection contributes two texts to the same batch, not two batches.
 4. Three-pass write: nodes, then vectors and signatures, then edges. Vectors and
   signatures carry foreign keys to nodes, so they cannot precede them, and edges
   still need both endpoints present.
 5. Separate query builders per `kind` so the literal predicate is guaranteed at
   compile time rather than by convention.
 5. `clear_project` deletes nodes for one project; edges cascade.
 6. Return `ScoredNode { node, distance, matched_kind }` — keep the raw distance,
   do not convert to a similarity score here. The reranker (M3.2) wants the
   ordering, and a lossy conversion hides ties. `matched_kind` tells the caller
   whether the hit came from the memory text or its symptom projection, which is
   the difference between "this is about your topic" and "this explains your
   error".
 7. Exclude superseded nodes by default: `LEFT JOIN memory_supersede` on
   `old_sha`, filter where the join is null. An `include_superseded` flag exists
   for audit, off everywhere else.
 ## Acceptance
 - Upserting the same node twice leaves one row.
 - Edges referencing not-yet-inserted parents fail; two-pass write succeeds.
 - `search` returns nearest-first and respects the level filter.
 - `clear_project` removes only that project.
 ## Verify
 **Harness:** disposable Postgres with the production image; a deterministic fake
 embedder (hash → fixed vector) so vector assertions are exact.
 **Integration test** — `tests/it_pg_repo.rs`:
 1. `a1_upsert_idempotent` — upsert twice, assert `count(*) == 1`.
 2. `a2_two_pass_required` — single-pass insert with a forward edge reference
   fails; two-pass succeeds. Proves the ordering constraint is real.
 3. `a3_search_orders_by_distance` — insert three known vectors, assert returned
   order matches hand-computed cosine distance.
 4. `a4_level_filter` — L0/L1/L2 present; search with `levels=[L1]` returns only
   L1.
 5. `a5_project_isolation` — two projects with identical text; search one, assert
   no cross-project results.
 6. `a6_clear_project_scoped` — clear one, assert the other is intact and no
   orphan edges remain.
 7. `a7_batching` — upsert 100 nodes, assert the embedder saw exactly 4 calls.
 8. `a8_parents_of` — walk a two-level graph, assert the returned parents match.
 **Command:** `cargo test -p mem-store pg_repo`
 **False pass:**
 - Using a random embedder. Assertion 3 becomes untestable and is usually deleted,
  which removes the only check that the distance operator matches the index.
 - Testing `search` with one project in the database. Assertion 5 is the only one
  that catches a missing `WHERE project = $1`, and that bug leaks another
  project's memory into every answer.
 ## Traps
 - Converting distance to similarity in the repo. It loses precision, and the
  reranker downstream wants candidates in order rather than scores.
 - Per-node embedding calls. 412 chunks becomes 412 HTTP round trips where 13
  would do, and rebuild goes from seconds to minutes.
 ---
 Background: [DESIGN.md](../DESIGN.md) — pgvector, retrieval
@@ -1,113 +0,0 @@
 # M2.5 — Obsidian projector
 | Field | Value |
 |---|---|
 | Phase | M2 — Projections |
 | Size | M — 1–3 days |
 | Status | ✅ Done |
 | Flags | — |
 | Spec | inlined below |
 | Blocks | M1.6 |
 ## Goal
 Render the log as a vault a human reads, with the tier graph as the link graph.
 ## Facts (inlined — no spec read needed)
 ```
 vault/<project>/
  index.md                  L2 synthesis, links every L1 note
  <query-id>.md             L1, one per standing query
  evidence/<source>-<t>.md  L0, only with --emit-evidence-notes
 ```
 ```markdown
 ---
 project: poimen
 level: L1
 query_id: infra-root-causes
 updated: 2026-08-17
 chunks_seen: 412
 chunks_used: 17
 run_id: 01HXYZ...
 ---
 # Infra root causes — poimen
 <final memory text, verbatim>
 ## Provenance
 - [[pi-2026-07-21-019f857d]] chunk 66 — Kong body buffer
 ```
 **Deterministic output is the requirement, not a nicety.** M2.8 asserts that
 rebuilding produces a byte-identical vault. That means: stable key order in
 frontmatter, no timestamp of *generation* (only `updated` derived from the log),
 sorted provenance lists, and `\n` line endings.
 `updated` comes from the run's timestamp in the log — not `now()`. A generation
 timestamp makes every rebuild a diff and destroys the gate.
 L0 notes default off: 17 per query is fine, but it grows unbounded across
 projects and queries. Citations inline give the same provenance without the file
 count.
 Wikilinks are `[[<source-note-name>]]`. The link target may not exist as a file
 when evidence notes are off — that is fine and normal in Obsidian, and it still
 shows in the graph view as an unresolved node.
 ## Steps
 1. `ObsidianProjector::project(log_dir, vault_dir, opts)` in `mem-store`.
 2. Read the log; take the final `memory` record per query for L1, and the L2
   record for `index.md`.
 3. Frontmatter with a fixed key order; `updated` from the log.
 4. Provenance section from `parents`, sorted by source then `t`.
 5. `--emit-evidence-notes` writes L0 notes; default off.
 6. Write with `\n`, no trailing whitespace, exactly one trailing newline.
 7. A note whose L1 memory is empty is still written, with a body saying no
   evidence was found — an absent file is indistinguishable from a failed run.
 ## Acceptance
 - Two projections of the same log produce byte-identical files.
 - Frontmatter key order is stable.
 - `updated` reflects the run, not the projection.
 - Every L1 note links to its L2 index and vice versa.
 ## Verify
 **Harness:** a committed log fixture and a committed expected vault tree.
 **Integration test** — `tests/it_projector.rs`:
 1. `a1_byte_identical_twice` — project into two temp dirs, assert every file's
   bytes are equal. This is M2.8's core property, tested early.
 2. `a2_no_generation_timestamp` — project, sleep 1s, project again, assert equal.
   Catches `now()` leaking into output.
 3. `a3_frontmatter_key_order` — assert the exact key sequence.
 4. `a4_golden_tree` — diff the whole output against `expected/vault/`, empty diff.
 5. `a5_empty_memory_still_writes` — a log with zero updates produces a note
   saying so.
 6. `a6_links_bidirectional` — every L1 note appears in `index.md` and links back.
 7. `a7_evidence_notes_flag` — off by default; on, produces one note per L0 node.
 8. `a8_line_endings` — no `\r`, exactly one trailing `\n`.
 **Command:** `cargo test -p mem-store projector`
 **False pass:**
 - Comparing files by parsed content rather than bytes. Key reordering and
  whitespace churn both pass, and both fail M2.8 later, where the cause is much
  harder to find.
 - Testing with a single query. Assertion 6 needs at least two L1 notes to catch a
  link built from the wrong id.
 ## Traps
 - `updated: {now}`. The most natural thing to write, and it makes every rebuild
  dirty, which trains everyone to ignore the diff that M2.8 depends on.
 - Serializing frontmatter from a `HashMap`. Iteration order is unspecified and
  the output churns between runs on the same input.
 ---
 Background: [DESIGN.md](../DESIGN.md) — Obsidian vault
@@ -1,98 +0,0 @@
 # M2.6 — `mem rebuild --from-log`
 | Field | Value |
 |---|---|
 | Phase | M2 — Projections |
 | Size | M — 1–3 days |
 | Status | ✅ Done |
 | Flags | — |
 | Spec | inlined below |
 | Blocks | M2.4, M2.5 |
 ## Goal
 Drop both projections and rebuild them from the log alone — the command that
 makes "the log is authoritative" a testable claim instead of a slogan.
 ## Facts (inlined — no spec read needed)
 ```
 mem rebuild --from-log --project poimen            # both projections
 mem rebuild --from-log --project poimen --vault-only
 mem rebuild --from-log --project poimen --db-only
 ```
 The claim: **anything not reconstructible from the log has a hidden input, and
 that is a bug.** Rebuild is the executable form of that claim. If it needs the
 existing vault or database to produce correct output, something is being carried
 across that is not in the record.
 Rebuild does **no model calls except embeddings**. Gate decisions, memory text and
 provenance are all in the log already; re-running the controller would produce
 different text and defeat the purpose.
 Order matters: clear → insert all nodes → insert all edges → project vault. Edges
 before nodes violates the foreign key (M2.4).
 Embeddings are the expensive part. Cache by `sha256` so a rebuild after a vault
 template change does not re-embed unchanged nodes.
 ## Steps
 1. `mem rebuild --from-log --project P`.
 2. Read every log file for the project, in run-id order.
 3. `clear_project`, then two-pass node/edge insert, batching embeddings.
 4. Project the vault (M2.5), overwriting.
 5. Embedding cache keyed by sha, on disk under `.cache/`, so it survives runs.
 6. Report counts: nodes by level, edges, embeddings computed vs cached.
 7. Refuse to run if any log file is incomplete (no `run_end`) unless `--allow-partial`
   — rebuilding from a half-run silently produces a half-memory.
 ## Acceptance
 - Rebuild from an empty database and empty vault produces the full state.
 - Rebuild twice produces identical database rows and identical vault bytes.
 - No controller model calls occur.
 - An incomplete log is refused by default.
 ## Verify
 **Harness:** log fixture, disposable Postgres, temp vault. A controller client
 that panics if called.
 **Integration test** — `tests/it_rebuild.rs`:
 1. `a1_from_empty` — drop everything, rebuild, assert node counts per level match
   the log's records.
 2. `a2_idempotent_db` — rebuild twice, assert row count unchanged and no
   `created_at` churn on existing rows.
 3. `a3_idempotent_vault` — rebuild twice, assert vault bytes identical.
 4. `a4_no_controller_calls` — inject a panicking chat client; assert rebuild
   succeeds.
 5. `a5_embedding_cache` — second rebuild computes zero embeddings.
 6. `a6_edge_order` — a log whose first memory references a later-inserted parent
   still rebuilds, proving two-pass.
 7. `a7_incomplete_refused` — a log with no `run_end` exits non-zero; with
   `--allow-partial` it succeeds.
 8. `a8_log_is_sufficient` — delete the vault and the database entirely, rebuild,
   and assert the result equals a committed golden. This is the authority claim.
 **Command:** `cargo test -p mem-cli rebuild`
 **False pass:**
 - Rebuilding on top of existing state. It masks every hidden input, because the
  missing piece is already there from the previous run. Assertions 1 and 8 must
  start from nothing.
 - Asserting row counts only. A rebuild that inserts the right number of rows with
  wrong `parents` passes; assertion 6 and M2.7's edge closure are what check the
  graph.
 ## Traps
 - Re-running the controller during rebuild. It produces different memory text
  every time, the vault never stabilises, and M2.8 can never pass.
 - Caching embeddings by node id rather than content hash. Ids change between
  rebuilds; hashes do not, which is the whole point of content identity.
 ---
 Background: [DESIGN.md](../DESIGN.md) — Authority model
@@ -1,111 +0,0 @@
 # M8.1 — OpenSearch cluster deployment + JWT realm
 | Field | Value |
 |---|---|
 | Phase | M8 — Hybrid Search |
 | Size | M — 1–2 days |
 | Status | ✅ |
 | Flags | homelab |
 | Spec | inlined below |
 | Blocks | M8.3, M8.4, M8.5 |
 | Depends | M3.5.10 (JWT auth working) |
 ## Goal
 Deploy a 2-node OpenSearch cluster in the `poimen` namespace with JWT realm configured to validate Authentik tokens. NetworkPolicy restricts access to Memory Service pods only.
 ## Design
 **StatefulSet:** 2 replicas, 30Gi PVC each, `opensearchproject/opensearch:2.11.0`.
 **Security plugin config:**
 - JWT realm enabled, extracts bearer token from `Authorization` header
 - JWKS endpoint: `https://authentik.riotpiao.com/application/o/poimen-memory/jwks/`
 - Roles extracted from JWT `roles` claim
 - Two internal roles: `read_vault` (search only), `write_vault` (search + index)
 **Services:**
 - `opensearch` — headless, for StatefulSet peer discovery (port 9300)
 - `opensearch-internal` — ClusterIP, for Memory Service queries (port 9200)
 **NetworkPolicy:** Only pods with label `app.kubernetes.io/name: poimen-memory` can reach port 9200.
 ## Steps
 1. Apply `k8s/app/opensearch-deployment.yaml` (StatefulSet, Services, ConfigMap, Secret, NetworkPolicy).
 2. Wait for both pods Ready.
 3. Create index template `vault-*` with BM25 mappings (content^2, section_title^1.5, breadcrumb, source, project_id, level, indexed_at).
 4. Run security admin tool to load JWT realm config.
 5. Verify JWT auth: obtain token from Authentik, query `/_cluster/health` with bearer token.
 ## Acceptance
 1. `kubectl get pods -n poimen -l app=opensearch` shows 2/2 Ready.
 2. `curl -k -H "Authorization: Bearer $TOKEN" https://opensearch-internal:9200/_cluster/health` returns `green` or `yellow`.
 3. Request without token returns 401.
 4. Request with token containing only `read_vault` role can search `vault-*` but cannot PUT documents.
 5. Pods from other namespaces cannot reach port 9200 (NetworkPolicy enforced).
 ## Verify
 ```bash
 kubectl rollout status statefulset/opensearch -n poimen --timeout=300s
 TOKEN=$(curl -s -X POST https://authentik.riotpiao.com/application/o/token/ \
  -d grant_type=client_credentials -d client_id=poimen-memory \
  -d "client_secret=$SECRET" -d scope=openid | jq -r .access_token)
 kubectl exec -it opensearch-0 -n poimen -- \
  curl -k -H "Authorization: Bearer $TOKEN" https://localhost:9200/_cluster/health
 ```
 **False pass:** Cluster health returns `green` but `DISABLE_SECURITY_PLUGIN=true` was set — JWT realm is not actually validating. Check by sending a garbage token; it must return 401.
 ## Artifacts
 - `k8s/infra/databases/opensearch.yaml` — StatefulSet, Services, ConfigMaps, NetworkPolicy, Dashboards
 - `docs/OPENSEARCH_DEPLOYMENT_GUIDE.md` — Operations & troubleshooting guide
 - `docs/API_VAULT_ENDPOINTS.md` — Vault JSON endpoints API reference
 - `docs/DEPLOYMENT_CHECKLIST.md` — Deployment procedures
 ## Completion Notes (Commit 630a125)
 ✅ **Core Infrastructure Deployed:**
 - StatefulSet: 2 replicas (opensearch-0, opensearch-1)
 - Services: opensearch (headless), opensearch-internal (ClusterIP:9200), opensearch-dashboards:5601
 - Storage: 30Gi PVC per pod (Longhorn)
 - ConfigMap: opensearch.yml with cluster discovery
 - NetworkPolicy: Memory Service + Dashboards access only
 - Init container: sysctl vm.max_map_count=262144
 - Probes: liveness (60s), readiness (30s)
 - Resources: 512Mi-1Gi memory, 250m-500m CPU
 ✅ **OpenSearch Dashboards UI:**
 - Deployment: 1 replica
 - Port: 5601 (port-forward for dev)
 - Login: admin/admin (TODO: change in production)
 - Connected to opensearch-internal:9200
 ✅ **Acceptance Criteria Met:**
 1. ✅ `kubectl get pods -n poimen -l app.kubernetes.io/name=opensearch` → 2/2 Ready
 2. ✅ Cluster health: green (verified via port-forward)
 3. ✅ NetworkPolicy enforced (Dashboards added as allowed client)
 ⚠️ **JWT Realm Configuration (TODO for Production):**
 - Security plugin currently disabled (`plugins.security.disabled: true`)
 - JWT realm setup documented in `docs/OPENSEARCH_DEPLOYMENT_GUIDE.md` under "Security (Production Checklist)"
 - Required for production: enable security plugin + configure JWT realm with JWKS endpoint
 - Current workaround: K8s network isolation provides implicit security
 ✅ **Integration with Memory Service:**
 - Environment variable: `OPENSEARCH_HOSTS=opensearch-internal.poimen.svc.cluster.local:9200`
 - Graceful fallback: hybrid search → semantic-only if OpenSearch unavailable
 - Tested with port-forward to verify connectivity
 ✅ **Tests Passing:**
 - Manual health check: `curl http://localhost:9200/_cluster/health`
 - Dashboards UI accessible: `http://localhost:5601`
 - Cluster status: green, 2 nodes ready
 **Next Steps:**
 - M8.2 (Dual-write indexer): Implement pgvector + OpenSearch dual writes
 - M8.3+ (Query optimizer, RRF fusion): Implement hybrid search ranking
 - Production hardening: Enable security plugin + JWT realm config