[Phase 2.4] Compaction activities (4 activities) #16

New Issue

2026-09-09T01:04:29Z

poimen commented

2026-09-09 01:04:29 +00:00

Goal

Implement 4 activities for the compaction workflow. Tier 1 is pure DB, Tier 2 calls LLM.

Activities

1. ExactDedupActivity

Query memory service: POST /memory/query for edges with same (source, target, relation, fact)
Group duplicates, keep highest confidence / newest
Return count of soft-deleted edges

2. StaleGCActivity

Query edges where t_invalid IS NOT NULL AND age > 30 days
Hard-delete via memory service
Return count cleaned

3. SemanticDedupActivity

Pre-filter: query pairs with embedding cosine > 0.9
For each candidate pair, call LLM: "Are these two facts semantically equivalent?"
LLM model: qwen2.5:3b (cheap, fast)
If yes (confidence > 0.95): auto-merge (keep superset)
If maybe (0.7-0.95): queue for human review
Return counts: { auto_merged, queued_review, skipped }

4. AuditCompactionActivity

Write compaction summary to memory service via POST /memory/ingest
Include: timestamp, tier, counts, duration, dry_run flag

Acceptance Test

// activity/compaction_test.go
func TestExactDedup_FindsDuplicates(t *testing.T) {
    // Mock memory client returns 3 duplicate groups
    // Assert: returns 3 deduped
}

func TestSemanticDedup_CallsLLM(t *testing.T) {
    // Mock memory client returns 2 candidate pairs (cosine > 0.9)
    // Mock LLM returns "equivalent" for pair 1, "different" for pair 2
    // Assert: auto_merged=1, skipped=1
}

func TestStaleGC_CleansOldEdges(t *testing.T) {
    // Mock returns 5 stale edges
    // Assert: 5 cleaned
}

Integration Test (local worker)

# With local Temporal + memory service running:
TEMPORAL_HOSTPORT=localhost:7233 \
MEMORY_SERVICE_URL=http://localhost:8080 \
LOCAL_LLM_BASE_URL=http://localhost:11434 \
  go run cmd/starter/main.go --workflow CompactionWorkflow \
  --input {"project":"test","dry_run":true,"tier":2}

# Should see in worker logs:
# ExactDedup: 0 deduped (empty DB)
# StaleGC: 0 cleaned
# SemanticDedup: 0 candidates
# Audit: logged dry_run=true

Effort

~200 LOC, 3 days

## Goal Implement 4 activities for the compaction workflow. Tier 1 is pure DB, Tier 2 calls LLM. ## Activities ### 1. ExactDedupActivity - Query memory service: `POST /memory/query` for edges with same (source, target, relation, fact) - Group duplicates, keep highest confidence / newest - Return count of soft-deleted edges ### 2. StaleGCActivity - Query edges where `t_invalid IS NOT NULL AND age > 30 days` - Hard-delete via memory service - Return count cleaned ### 3. SemanticDedupActivity - Pre-filter: query pairs with embedding cosine > 0.9 - For each candidate pair, call LLM: "Are these two facts semantically equivalent?" - LLM model: `qwen2.5:3b` (cheap, fast) - If yes (confidence > 0.95): auto-merge (keep superset) - If maybe (0.7-0.95): queue for human review - Return counts: { auto_merged, queued_review, skipped } ### 4. AuditCompactionActivity - Write compaction summary to memory service via `POST /memory/ingest` - Include: timestamp, tier, counts, duration, dry_run flag ## Acceptance Test ```go // activity/compaction_test.go func TestExactDedup_FindsDuplicates(t *testing.T) { // Mock memory client returns 3 duplicate groups // Assert: returns 3 deduped } func TestSemanticDedup_CallsLLM(t *testing.T) { // Mock memory client returns 2 candidate pairs (cosine > 0.9) // Mock LLM returns "equivalent" for pair 1, "different" for pair 2 // Assert: auto_merged=1, skipped=1 } func TestStaleGC_CleansOldEdges(t *testing.T) { // Mock returns 5 stale edges // Assert: 5 cleaned } ``` ## Integration Test (local worker) ```bash # With local Temporal + memory service running: TEMPORAL_HOSTPORT=localhost:7233 \ MEMORY_SERVICE_URL=http://localhost:8080 \ LOCAL_LLM_BASE_URL=http://localhost:11434 \ go run cmd/starter/main.go --workflow CompactionWorkflow \ --input {"project":"test","dry_run":true,"tier":2} # Should see in worker logs: # ExactDedup: 0 deduped (empty DB) # StaleGC: 0 cleaned # SemanticDedup: 0 candidates # Audit: logged dry_run=true ``` ## Effort ~200 LOC, 3 days

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: riotpiao-poimen/poimen-workflows#16