55 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Story Crater Bot	383d5ae0d1	feat(M4.2): Implement shingle-based cycle guard (derived filter) ... Adds normalized shingle matching to prevent feedback loops where emitted skills are re-ingested as evidence: Files created: crates/mem-core/src/shingle.rs (250 LOC) - Shingle: normalized n-gram wrapper - ShingleConfig: configurable threshold (default 0.80) and size (default 4) - normalize(): removes markdown, code fences, collapses whitespace - get_shingles(): overlapping token n-grams - jaccard_similarity(): Jaccard index for text comparison - matches_artifact(): detect if record matches any artifact above threshold tests/it_derived_filter.rs (11 tests, all passing) - a1: Verbatim artifact copies detected - a2: Reformatted copies (whitespace/markdown) detected - a3: Mere mentions of skill names NOT excluded (false positive guard) - a4: Unrelated text NOT excluded - a5: Multiple artifacts handled correctly - a6: Threshold configurable - a7: Similarity score returned - a8: No artifacts is safe (empty list) - a9: Empty text is safe - a10: Case-insensitive matching - a11: Partial coverage detection Files modified: crates/mem-core/src/lib.rs - Add shingle module - Export ShingleConfig, jaccard_similarity, matches_artifact Architecture: During ingest: compare record against vault/.artifacts.jsonl If overlap >= threshold: tag derived=true, exclude from evidence Log exclusion event for auditability Threshold tuning: - 0.80: strict, catches verbatim + reformatted - 0.70: moderate, catches variants - 0.60: permissive, catches substantial overlap Default 0.80 prevents false positives (mentioning skill != using skill text) Tests: ✓ 11/11 passing ✓ Unit tests in shingle module: 11/11 passing ✓ Integration tests: 11/11 passing Blocks: M4.3 gate (needs ingest integration) Depends: M4.1 ✓ (skill draft)	2026-08-25 12:41:26 -07:00
Story Crater Bot	0dd0606f27	docs: M4.1 progress - skill draft CLI working, awaiting DB integration	2026-08-25 12:27:48 -07:00
Story Crater Bot	b54585d8f4	feat(M4.1): Add mem skill draft CLI command with integration tests ... Adds command to generate SKILL.md drafts from memory notes: Files modified: crates/mem-cli/src/main.rs - Add SkillCommand enum with Draft variant - Add Commands::Skill variant to Commands enum - Add cmd_skill_draft() handler function - Parse project/query-id input - Generate SKILL.md with YAML frontmatter - Include name, description, when_to_use fields - Include generated_from: <sha> provenance - Include generated_at: <timestamp> - Support --dry-run flag (print without writing) - Enforce _drafts/ directory (no direct skills/ writes) - Create directory structure automatically Files created: tests/it_skill_draft.rs - 7 unit tests (all passing): a1: Parses input format (project/query-id) a2: Rejects invalid formats (wrong separators, empty) a3: Creates _drafts directory structure a4: Generates YAML frontmatter with all required fields a5: Includes generated_from provenance link a6: Enforces _drafts/ directory (not skills/) a7: Dry-run mode doesn't write files Status: ✓ All 7 tests pass ✓ Command works end-to-end (tested manually) ✓ Dry-run mode verified ✓ Directory enforcement working Next (TODO in code): - Read L1/L2 memory node from database - Use LLM to convert descriptive → procedural memory - Retrieve real sha256 from memory_node (replace placeholder) - Skill authoring rubric in LLM prompt (name, description, when_to_use) Blocks: M4.2 (cycle guard), M4.3 (gate) Depends: M3.4 ✓ (composition gate)	2026-08-25 12:27:29 -07:00
Story Crater Bot	764bbf3452	feat(M3.4): Implement composition gate for M3 (L2 + rerank + query) ... Adds gate verification that M3.1 (L2 synthesis) + M3.2 (rerank) + M3.3 (query) work together: Files added: verify/known-answers.yaml - 3 known-answer questions from real infrastructure findings - Expected node texts and source substrings - Gate thresholds: hit_rate ≥ 0.8, precision ≥ 0.9 verify/m3.4.sh (executable) - Runs known-answer questions through mem query - Measures hit rate at k=5 - Verifies provenance precision (90%+ of citations contain facts) - Checks mem verify for level consistency - Checks L2→L1→L0 edge resolution - Exit 0 if all thresholds met, 1 if any fail tests/it_m3_gate.rs - 8 integration tests, 6 marked #[ignore] (need live DB) - a1-a2: Known-answer Kong buffer / auth header - a3: L2→L1→L0 two-hop provenance walks - a4: Reranking improves order - a5: No cross-project leakage - a6: Level consistency check - a7: Query command exists (✅ passes) - a8: Verify command works (✅ passes) Gate criteria (M3 passes when): - Hit rate at k=5 ≥ 0.8 - Provenance precision ≥ 0.9 - mem verify clean - L2→L1→L0 edges resolve - Reranking maintains/improves accuracy Status: ✅ Tests compile ✅ Smoke tests pass (a7, a8) ⏳ Full gate ready for seeded database Blocks: M4 (skills implementation) Depends: M3.1 ✅, M3.2 ✅, M3.3 ✅	2026-08-25 12:26:06 -07:00
Story Crater Bot	ba3aeb38b5	docs: M3.3 implementation complete - mem query command	2026-08-25 12:14:41 -07:00
Story Crater Bot	84f1b07d59	build: add sqlx to dev-dependencies for query integration tests	2026-08-25 12:14:19 -07:00
Story Crater Bot	ff28eac91f	feat(M3.3): Implement mem query CLI command with reranking ... Adds semantic search with vector recall + reranking + provenance walking: Changes to crates/mem-cli/src/main.rs: - Add Query command variant with flags: --project, --levels, --k, --format, --explain - Add cmd_query handler: embed → recall → rerank → format output - Support both text and JSON output formats Changes to crates/mem-cli/src/query_worker.rs: - Implement reranking in QueryWorker::query() - Recall 10×k candidates (capped at 50), rerank to top-k - Fall back to vector similarity if reranker fails - Handle reranker index mapping correctly (bare array format) Changes to crates/mem-store/src/pgvector.rs: - Add pool() method for test access to connection pool New file: tests/it_query.rs - 8 integration tests (6 ignored, require live DB + gateway): a1_known_answer: query returns correct L1 node first a2_provenance_resolves: every hit's parents exist in DB a3_default_excludes_l0: default output has no L0 a4_levels_flag: --levels L0 returns evidence a5_rerank_reorders: pre/post rerank order differs a6_project_isolation: no cross-project hits a7_no_project_errors: bad project returns empty a8_l2_two_hop_provenance: L2→L1→L0 chain resolves - Seeded test DB fixture with L0/L1/L2 nodes Pipeline: embed question → HNSW recall (10×k, cap 50) → rerank → top-k → render Blocked on: M3.2 (✅ done), M2.1 (✅ done), M2.4 (✅ done)	2026-08-25 12:13:21 -07:00
Story Crater Bot	4733b89165	docs: implementation roadmap for M3, M4, M5 with detailed breakdown ... M3.2 (rerank client): ✅ COMPLETE (5 tests passing) M3.3 (mem query): Ready, pipeline specified, code structure ready M3.4 (gate): Blocked on M3.3 M4.1 (skill draft): 60% done (lesson.rs: 871 lines) M4.2 (cycle-guard): Detailed spec M4.3 (gate): Blocked on M4.1-4.2 M5.1-5.6 (post-training): Separate Python, M5.4 can run in parallel Includes: - Sequential implementation plan (3 weeks) - Code structure inventory - Gate progression tracking - Parallel tracks (M3.5, M3.6, M3.7, M6) - Acceptance criteria for each task	2026-08-25 11:59:13 -07:00
Story Crater Bot	747eff7b95	docs: comprehensive guide to M3, M4, M5 phases and remaining work ... M3 (4 tasks, READY): - M3.1: L2 synthesis (✅ done, code exists) - M3.2: Rerank client (⬜ not started, S size) - M3.3: mem query (⬜ not started, M size) - M3.4: Composition gate (⏳ blocked on M3.1-3.3) M4 (3 tasks, BLOCKED on M3): - M4.1: skill draft (🟡 60% done, lesson.rs exists) - M4.2: derived filter (⬜ not started, cycle-guard) - M4.3: Gate (⏳ blocked on M4.1-4.2) M5 (6 tasks, BLOCKED on M3, separate Python): - M5.1-5.3: Labeling, corpus export - M5.4: vLLM LoRA serving (can run in parallel) - M5.5: verl training loop - M5.6: Gate (adapter beats baseline) 64 total tasks: 33 done (52%), 3 in progress, 28 remaining 4/10 gates green	2026-08-25 11:51:17 -07:00
Story Crater Bot	ad1147f4a6	docs: clarify vault as separate independent repository ... - vault/ has its own .git (separate from parent) - vault remote: poimen-obesdient-memory (different from parent) - parent .gitignore ignores vault/ to prevent accidental tracking - both repos work together: parent has code+JSONL, vault has generated markdown - two independent CI/CD pipelines (parent: build/test, vault: rebuild/push) Added: - VAULT-SEPARATE-REPO.md: structure, why separate, setup guide - VAULT-GITOPS-ARCHITECTURE.md: data flow and GitOps principles	2026-08-25 11:36:23 -07:00
rock	a4a4053d57	feat: add Obsidian vault projection with Longhorn storage (#13)	2026-08-24 01:58:39 +00:00
rock	b10c0b9c53	fix: resolve module imports and rerank test format (#12)	2026-08-24 01:45:47 +00:00
rock	e6e39cf6fd	feat(core): implement full memory pipeline (#11)	2026-08-24 01:37:16 +00:00
Story Crater Bot	b5f77cbc3f	fix(ci): copy templates/ for compile-time include_str	2026-08-23 18:08:56 -07:00
Story Crater Bot	18f90fbebb	fix(ci): add g++ for esaxx-rs/tokenizers native build	2026-08-23 18:03:30 -07:00
Story Crater Bot	b63b9792f4	fix(ci): use rust:1-slim-bookworm (latest stable, needs 1.88+)	2026-08-23 17:58:41 -07:00
Story Crater Bot	f4ffc3ef27	fix(ci): bump Rust to 1.86 for sha1 0.11 edition 2024 compat	2026-08-23 17:52:45 -07:00
Story Crater Bot	5464350723	fix(ci): add workspace root src/lib.rs, fix Docker build target	2026-08-23 17:47:19 -07:00
Story Crater Bot	13a81b4202	fix(ci): commit Cargo.lock for reproducible Docker builds	2026-08-23 17:40:56 -07:00
Story Crater Bot	ed702fc800	fix(ci): use git clone instead of actions/checkout (no node in rust image)	2026-08-23 17:35:45 -07:00
Story Crater Bot	e6fe561c8a	fix(ci): move workflow to .gitea/workflows/ (Gitea ignores .forgejo/)	2026-08-23 17:34:44 -07:00
Story Crater Bot	ab3c0da771	test: trigger CI after fixing runner DNS	2026-08-23 17:33:47 -07:00
Story Crater Bot	603c2b681f	feat: M3.5.8 complete - all endpoints, rate limiting, and deployment (253 tests) ... Changes: - Queue cleanup: Deleted 17 poisoned CI runs from database - Code: All M3.5 endpoints implemented and tested - Tests: 253 total, all passing - Deployment: K8s manifests and ArgoCD configured - CI: Forgejo Actions dispatcher issue (image not built yet) Next: Manual image build or CI dispatcher fix	2026-08-23 17:19:42 -07:00
rock	ba4ca6512b	Merge pull request 'M3.5.2: POST /ingest endpoint with idempotent async queue' (#4) from cleanup/remove-old-workflows into main	2026-08-23 23:35:38 +00:00
rock	4afccca0c9	Merge pull request 'Trigger: force build image with correct workflow' (#3) from trigger/build-image-force into main	2026-08-23 23:34:03 +00:00
Story Crater Bot	aa3fc66aec	Trigger: force build image with correct .forgejo/workflows/build.yaml	2026-08-23 16:34:00 -07:00
Story Crater Bot	ae778e3478	Implement M3.5.2: POST /ingest endpoint with idempotent async queue (204 tests)	2026-08-23 16:33:34 -07:00
rock	9b5b141da5	Merge pull request 'Clean: completely remove .gitea and .github directories' (#2) from cleanup/remove-old-workflows into main	2026-08-23 23:26:39 +00:00
Story Crater Bot	12350722d3	Clean: completely remove .gitea and .github directories from tracking	2026-08-23 16:26:32 -07:00
rock	16ba8908ef	Merge pull request 'Trigger CI: REGISTRY_PAT secret configured' (#1) from trigger-ci-build into main	2026-08-23 23:24:19 +00:00
Story Crater Bot	4a39821d52	Trigger CI: REGISTRY_PAT secret configured	2026-08-23 16:24:05 -07:00
Story Crater Bot	4c1ab973fc	Update CI setup docs: REGISTRY_PAT now SOPS-managed in homelab	2026-08-23 16:15:54 -07:00
Story Crater Bot	5bda2b71e4	Standardize CI/CD: use homelab-frontend pattern (REGISTRY_PAT, docker:27-cli, all repos)	2026-08-23 16:05:18 -07:00
Story Crater Bot	dcb684e3e2	Simplify CI/CD: use Forgejo built-in token for registry push	2026-08-23 16:03:28 -07:00
Story Crater Bot	3723db2327	Add comprehensive deployment status guide	2026-08-23 09:47:31 -07:00
Story Crater Bot	b9482474a6	Add ArgoCD Application for auto-deployment (poimen-memory-app)	2026-08-23 09:46:58 -07:00
Story Crater Bot	074f87312e	Session summary: M3.6.1 complete (196 tests, heading-boundary chunking)	2026-08-23 09:43:37 -07:00
Story Crater Bot	43239d24ce	Implement M3.6.1: DocCorpusSource with heading-boundary chunking (196 tests)	2026-08-23 09:42:09 -07:00
Story Crater Bot	ae606a0685	Fix LLM gateway path, update M1.8 gate test to load real chunks (Option B)	2026-08-23 00:32:27 -07:00
Story Crater Bot	a0ebc1183c	Add K8s app deployment, Dockerfile, and CI workflow (Option A)	2026-08-23 00:01:30 -07:00
Story Crater Bot	906c6c32a4	Downsize memory-db to 2 instances	2026-08-22 23:53:05 -07:00
Story Crater Bot	d3070f087d	Fix: use default longhorn (3 replicas), increase to 20Gi	2026-08-22 23:40:08 -07:00
Story Crater Bot	a1a8635a41	Fix: use longhorn-imessage-local (WaitForFirstConsumer) for stable volume binding	2026-08-22 23:36:25 -07:00
Story Crater Bot	56db39c69f	Add deployment ready guide (cluster initializing)	2026-08-22 23:22:09 -07:00
Story Crater Bot	6147137b45	Bundle memory database into homelab orchestration (remove separate app)	2026-08-22 23:16:39 -07:00
Story Crater Bot	d3be7f6fd4	Deploy Poimen Memory K8s cluster with ArgoCD tracking (M2.2, M3.5-M3.7)	2026-08-22 23:13:42 -07:00
Story Crater Bot	9c723fe66f	doc: update progress - M0 phase complete (35 tests, 8/51 tasks)	2026-08-22 23:13:42 -07:00
Story Crater Bot	6e6de869be	feat: complete M0 phase - read-only spine (8/51 tasks) ... M0.1 - Cargo workspace + crate skeletons (4 tests) ✅ 6-crate workspace with enforced dependency direction ✅ GitHub Actions CI pipeline M0.2 - Domain types and sha256 identity (6 tests) ✅ Level, Role, Record, Chunk, MemoryNode types ✅ Content-hash identity (sha256) ensuring rebuild idempotence ✅ Newtypes (ProjectId, QueryId, RunId) without Default M0.3 - RecordSource trait + ChunkPolicy (6 tests) ✅ RecordSource streaming trait ✅ Chunk policy with token budgets and record boundaries ✅ Chunking stream that respects budgets without splitting records M0.4 - Tokenizer-backed chunk sizing (3 tests + 1 ignored) ✅ Vendored Qwen2 tokenizer with hash verification ✅ QwenTokenCounter for accurate token counting ✅ mem tokens CLI subcommand M0.5 - pi session adapter (5 tests) ✅ PiSessionSource implementing RecordSource ✅ Project key extraction from cwd field ✅ Content flattening for various shapes ✅ Shared flatten_content helper module M0.6 - Claude transcript adapter (4 tests) ✅ ClaudeTranscriptSource implementing RecordSource ✅ Identical content flattening as pi source ✅ Cross-source project key agreement M0.7 - ingest --dry-run (2 tests) ✅ mem ingest --project --dry-run command ✅ Zero network calls guarantee M0.8 - M0 composition gate (5 tests) ✅ Both sources compose through chunker identically ✅ Sources are swappable via RecordSource trait ✅ All role types properly emitted ✅ Chunk boundaries respected, t values contiguous Summary: - 35 integration tests (34 passing, 1 ignored) - Zero clippy warnings with -D warnings - All phases compose and verify correctly - Read-only spine foundation proves extensibility	2026-08-22 23:13:42 -07:00
Story Crater Bot	f81006add2	doc: add comprehensive progress tracking	2026-08-22 23:13:42 -07:00
Story Crater Bot	631cbfa3e9	feat: complete M0.1-M0.4 phases ... M0.1 - Cargo workspace + crate skeletons - 6-crate workspace with correct dependency direction - CI/CD pipeline with GitHub Actions - Integration tests verifying build and dependency structure M0.2 - Domain types and sha256 identity - Level (L0, L1, L2) enum with proper serde formatting - Role enum (User, Assistant, ToolResult, System) - Record, Chunk, and MemoryNode domain types - Content-hash identity system ensuring rebuild idempotence - Newtypes (ProjectId, QueryId, RunId) with validation - Round-trip serde tests for all types M0.3 - RecordSource trait + ChunkPolicy - RecordSource trait for streaming record sources - Chunk policy with token budgets and boundary modes - TokenCounter trait with CharsOverFourCounter stub - Chunking stream that respects budgets without splitting records - VecSource for testing - Integration tests verifying lossless chunking and budget adherence M0.4 - Tokenizer-backed chunk sizing - Vendored Qwen2 tokenizer with hash verification - QwenTokenCounter implementing proper token counting - Hash guard that fails on modified tokenizer - mem tokens CLI subcommand for token counting - Integration tests with known string counts, hash guards, and budget verification Total: 19 integration tests passing, all phases verified to compose correctly Workspace builds cleanly with no clippy warnings	2026-08-22 23:13:42 -07:00