15 Commits

Include renames

Author	SHA1	Message	Date
Story Crater Bot	dfdcfa5d3a	feat(M5.3): Add training corpus export infrastructure for verl ... M5.3 — Training Corpus Export (verl format): - Trajectory struct: trajectory_id, turns[], r_exit, r_format, r_outcome - TrajectoryTurn: t, prompt, response, r_update, parsed - CorpusStats: total_trajectories, total_turns, positive/negative split, r_format pass rate, r_exit distribution Reward computation: - r_update_t: +1 if label matches U_t, -1 if mismatch (per turn) - r_exit: 0 if exit == last_evidence_t, -0.75 if earlier, -0.5 if later - r_format: 1.0 if all turns parsed, 0.0 if any unparsed (strict) - r_outcome: null (no answer correctness signal available) Files created: crates/mem-core/src/trajectory.rs (280 LOC) - Trajectory construction and reward calculation - CorpusStats aggregation from trajectories - Serialization for JSONL output tests/it_export.rs (280 LOC, 12 tests) - a1: Trajectory grouping by run - a2: r_update signs correct - a3: r_format strict (any unparsed = 0) - a4: r_exit distribution (perfect/early/late) - a5: Prompts are exact byte recordings - a6: CorpusStats aggregation - a7: r_outcome null - a8: Turn ordering preserved - a9: Multiple trajectories - a10: Serde roundtrip - a11: CorpusStats structure complete - a12: Mixed exit rewards Unit tests: - crates/mem-core/src/trajectory.rs: 8/8 passing Integration tests: - tests/it_export.rs: 12/12 passing Architecture: Log + Labels → Trajectories → JSONL for verl Each trajectory = one run with multiple turns Per-turn rewards enable trajectory-level loss + turn-level loss Blocks: M5.4 (vLLM setup), M5.5 (verl training) Depends: M5.1 ✓, M5.2 ✓	2026-08-25 12:45:15 -07:00
Story Crater Bot	6a873088e6	feat(M5.1-M5.2): Add evidence labeler and calibration infrastructure ... M5.1 — Evidence Labeler (distant supervision): - EvidenceLabel struct: chunk_sha, t, label, why, model, ts - LabelerConfig: configurable model_id, max_tokens, max_context - make_label_prompt(): question + chunk in 16K context budget - parse_label_response(): extract yes/no + 1-sentence justification - fits_context_budget(): verify prompt fits reasoning model limits - Unit tests: 8/8 passing M5.2 — Labeler Calibration (Cohen's kappa): - CalibrationResults: tp/tn/fp/fn, accuracy, kappa, precision, recall, f1 - Cohen's kappa formula (corrects for class imbalance, unlike accuracy) - CalibrationSample: blind worksheet (hides labeler answers from human) - stratified_sample(): 50/50 positive/negative (not corpus-proportional) - passes_gate(): kappa >= 0.6 threshold - Unit tests: 6/6 passing Integration tests: tests/it_labeler.rs: 11 tests, all passing - a1: One label per chunk - a2: Keyed by sha (survives re-chunking) - a3: Context budget respected - a4: Justifications preserved - a5: Label structure correct - a6: No tools in prompt (reasoning model requirement) - a7: Parse variations (YES/no/Yes/No) - a8-a11: Serialization, rate reporting, edge cases tests/it_calibration.rs: 12 tests, all passing - a1: Worksheet blind (labeler answers hidden) - a2: Stratified sampling (attempts 50/50) - a3: Kappa perfect agreement = 1.0 - a4: Kappa vs accuracy (high accuracy ≠ good kappa) - a5: Confusion matrix (all 4 cells tracked) - a6: Precision/recall separated - a7: Gate threshold kappa >= 0.6 - a8: F1 score computed - a9-a12: Roundtrips, disagreement analysis, formula validation Files created: crates/mem-llm/src/labeler.rs (250 LOC) crates/mem-llm/src/calibration.rs (280 LOC) tests/it_labeler.rs (200 LOC) tests/it_calibration.rs (300 LOC) Architecture: M5.1: Question + Chunk → Reasoning Model → Label + Why M5.2: Labeler Labels + Human Labels → Kappa + Confusion Matrix → Gate Blocks: M5.3 (corpus export) Depends: M4.3 ✓	2026-08-25 12:44:23 -07:00
Story Crater Bot	383d5ae0d1	feat(M4.2): Implement shingle-based cycle guard (derived filter) ... Adds normalized shingle matching to prevent feedback loops where emitted skills are re-ingested as evidence: Files created: crates/mem-core/src/shingle.rs (250 LOC) - Shingle: normalized n-gram wrapper - ShingleConfig: configurable threshold (default 0.80) and size (default 4) - normalize(): removes markdown, code fences, collapses whitespace - get_shingles(): overlapping token n-grams - jaccard_similarity(): Jaccard index for text comparison - matches_artifact(): detect if record matches any artifact above threshold tests/it_derived_filter.rs (11 tests, all passing) - a1: Verbatim artifact copies detected - a2: Reformatted copies (whitespace/markdown) detected - a3: Mere mentions of skill names NOT excluded (false positive guard) - a4: Unrelated text NOT excluded - a5: Multiple artifacts handled correctly - a6: Threshold configurable - a7: Similarity score returned - a8: No artifacts is safe (empty list) - a9: Empty text is safe - a10: Case-insensitive matching - a11: Partial coverage detection Files modified: crates/mem-core/src/lib.rs - Add shingle module - Export ShingleConfig, jaccard_similarity, matches_artifact Architecture: During ingest: compare record against vault/.artifacts.jsonl If overlap >= threshold: tag derived=true, exclude from evidence Log exclusion event for auditability Threshold tuning: - 0.80: strict, catches verbatim + reformatted - 0.70: moderate, catches variants - 0.60: permissive, catches substantial overlap Default 0.80 prevents false positives (mentioning skill != using skill text) Tests: ✓ 11/11 passing ✓ Unit tests in shingle module: 11/11 passing ✓ Integration tests: 11/11 passing Blocks: M4.3 gate (needs ingest integration) Depends: M4.1 ✓ (skill draft)	2026-08-25 12:41:26 -07:00
Story Crater Bot	b54585d8f4	feat(M4.1): Add mem skill draft CLI command with integration tests ... Adds command to generate SKILL.md drafts from memory notes: Files modified: crates/mem-cli/src/main.rs - Add SkillCommand enum with Draft variant - Add Commands::Skill variant to Commands enum - Add cmd_skill_draft() handler function - Parse project/query-id input - Generate SKILL.md with YAML frontmatter - Include name, description, when_to_use fields - Include generated_from: <sha> provenance - Include generated_at: <timestamp> - Support --dry-run flag (print without writing) - Enforce _drafts/ directory (no direct skills/ writes) - Create directory structure automatically Files created: tests/it_skill_draft.rs - 7 unit tests (all passing): a1: Parses input format (project/query-id) a2: Rejects invalid formats (wrong separators, empty) a3: Creates _drafts directory structure a4: Generates YAML frontmatter with all required fields a5: Includes generated_from provenance link a6: Enforces _drafts/ directory (not skills/) a7: Dry-run mode doesn't write files Status: ✓ All 7 tests pass ✓ Command works end-to-end (tested manually) ✓ Dry-run mode verified ✓ Directory enforcement working Next (TODO in code): - Read L1/L2 memory node from database - Use LLM to convert descriptive → procedural memory - Retrieve real sha256 from memory_node (replace placeholder) - Skill authoring rubric in LLM prompt (name, description, when_to_use) Blocks: M4.2 (cycle guard), M4.3 (gate) Depends: M3.4 ✓ (composition gate)	2026-08-25 12:27:29 -07:00
Story Crater Bot	ff28eac91f	feat(M3.3): Implement mem query CLI command with reranking ... Adds semantic search with vector recall + reranking + provenance walking: Changes to crates/mem-cli/src/main.rs: - Add Query command variant with flags: --project, --levels, --k, --format, --explain - Add cmd_query handler: embed → recall → rerank → format output - Support both text and JSON output formats Changes to crates/mem-cli/src/query_worker.rs: - Implement reranking in QueryWorker::query() - Recall 10×k candidates (capped at 50), rerank to top-k - Fall back to vector similarity if reranker fails - Handle reranker index mapping correctly (bare array format) Changes to crates/mem-store/src/pgvector.rs: - Add pool() method for test access to connection pool New file: tests/it_query.rs - 8 integration tests (6 ignored, require live DB + gateway): a1_known_answer: query returns correct L1 node first a2_provenance_resolves: every hit's parents exist in DB a3_default_excludes_l0: default output has no L0 a4_levels_flag: --levels L0 returns evidence a5_rerank_reorders: pre/post rerank order differs a6_project_isolation: no cross-project hits a7_no_project_errors: bad project returns empty a8_l2_two_hop_provenance: L2→L1→L0 chain resolves - Seeded test DB fixture with L0/L1/L2 nodes Pipeline: embed question → HNSW recall (10×k, cap 50) → rerank → top-k → render Blocked on: M3.2 (✅ done), M2.1 (✅ done), M2.4 (✅ done)	2026-08-25 12:13:21 -07:00
rock	a4a4053d57	feat: add Obsidian vault projection with Longhorn storage (#13)	2026-08-24 01:58:39 +00:00
rock	b10c0b9c53	fix: resolve module imports and rerank test format (#12)	2026-08-24 01:45:47 +00:00
rock	e6e39cf6fd	feat(core): implement full memory pipeline (#11)	2026-08-24 01:37:16 +00:00
Story Crater Bot	ae778e3478	Implement M3.5.2: POST /ingest endpoint with idempotent async queue (204 tests)	2026-08-23 16:33:34 -07:00
Story Crater Bot	43239d24ce	Implement M3.6.1: DocCorpusSource with heading-boundary chunking (196 tests)	2026-08-23 09:42:09 -07:00
Story Crater Bot	ae606a0685	Fix LLM gateway path, update M1.8 gate test to load real chunks (Option B)	2026-08-23 00:32:27 -07:00
Story Crater Bot	a0ebc1183c	Add K8s app deployment, Dockerfile, and CI workflow (Option A)	2026-08-23 00:01:30 -07:00
Story Crater Bot	d3be7f6fd4	Deploy Poimen Memory K8s cluster with ArgoCD tracking (M2.2, M3.5-M3.7)	2026-08-22 23:13:42 -07:00
Story Crater Bot	6e6de869be	feat: complete M0 phase - read-only spine (8/51 tasks) ... M0.1 - Cargo workspace + crate skeletons (4 tests) ✅ 6-crate workspace with enforced dependency direction ✅ GitHub Actions CI pipeline M0.2 - Domain types and sha256 identity (6 tests) ✅ Level, Role, Record, Chunk, MemoryNode types ✅ Content-hash identity (sha256) ensuring rebuild idempotence ✅ Newtypes (ProjectId, QueryId, RunId) without Default M0.3 - RecordSource trait + ChunkPolicy (6 tests) ✅ RecordSource streaming trait ✅ Chunk policy with token budgets and record boundaries ✅ Chunking stream that respects budgets without splitting records M0.4 - Tokenizer-backed chunk sizing (3 tests + 1 ignored) ✅ Vendored Qwen2 tokenizer with hash verification ✅ QwenTokenCounter for accurate token counting ✅ mem tokens CLI subcommand M0.5 - pi session adapter (5 tests) ✅ PiSessionSource implementing RecordSource ✅ Project key extraction from cwd field ✅ Content flattening for various shapes ✅ Shared flatten_content helper module M0.6 - Claude transcript adapter (4 tests) ✅ ClaudeTranscriptSource implementing RecordSource ✅ Identical content flattening as pi source ✅ Cross-source project key agreement M0.7 - ingest --dry-run (2 tests) ✅ mem ingest --project --dry-run command ✅ Zero network calls guarantee M0.8 - M0 composition gate (5 tests) ✅ Both sources compose through chunker identically ✅ Sources are swappable via RecordSource trait ✅ All role types properly emitted ✅ Chunk boundaries respected, t values contiguous Summary: - 35 integration tests (34 passing, 1 ignored) - Zero clippy warnings with -D warnings - All phases compose and verify correctly - Read-only spine foundation proves extensibility	2026-08-22 23:13:42 -07:00
Story Crater Bot	631cbfa3e9	feat: complete M0.1-M0.4 phases ... M0.1 - Cargo workspace + crate skeletons - 6-crate workspace with correct dependency direction - CI/CD pipeline with GitHub Actions - Integration tests verifying build and dependency structure M0.2 - Domain types and sha256 identity - Level (L0, L1, L2) enum with proper serde formatting - Role enum (User, Assistant, ToolResult, System) - Record, Chunk, and MemoryNode domain types - Content-hash identity system ensuring rebuild idempotence - Newtypes (ProjectId, QueryId, RunId) with validation - Round-trip serde tests for all types M0.3 - RecordSource trait + ChunkPolicy - RecordSource trait for streaming record sources - Chunk policy with token budgets and boundary modes - TokenCounter trait with CharsOverFourCounter stub - Chunking stream that respects budgets without splitting records - VecSource for testing - Integration tests verifying lossless chunking and budget adherence M0.4 - Tokenizer-backed chunk sizing - Vendored Qwen2 tokenizer with hash verification - QwenTokenCounter implementing proper token counting - Hash guard that fails on modified tokenizer - mem tokens CLI subcommand for token counting - Integration tests with known string counts, hash guards, and budget verification Total: 19 integration tests passing, all phases verified to compose correctly Workspace builds cleanly with no clippy warnings	2026-08-22 23:13:42 -07:00