(tasks) add tasks for harness
This commit is contained in:
@@ -0,0 +1,125 @@
|
||||
# T3.1 — `WorkflowDef` IR + canonicalization
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | P3 — Workflow as data |
|
||||
| Size | L — over 3 days |
|
||||
| Status | Not started |
|
||||
| Flags | — |
|
||||
| Spec | inlined below |
|
||||
| Blocks | T3.2–T3.6 |
|
||||
|
||||
## Goal
|
||||
|
||||
The validated intermediate representation every workflow format parses into, and
|
||||
the canonicalization that makes `WorkflowVersion` a content hash of *meaning*
|
||||
rather than of source text.
|
||||
|
||||
## Facts (inlined — no spec read needed)
|
||||
|
||||
```rust
|
||||
pub struct WorkflowDef {
|
||||
pub id: WorkflowId,
|
||||
pub schema: SchemaVersion,
|
||||
pub steps: Vec<StepDef>,
|
||||
pub transitions: Vec<Transition>,
|
||||
pub rubric: RubricDef,
|
||||
pub budget: BudgetDef,
|
||||
}
|
||||
|
||||
pub struct StepDef {
|
||||
/// Author-assigned, stable across versions.
|
||||
pub id: StepId,
|
||||
pub kind: StepKind,
|
||||
pub tools: ToolSelector,
|
||||
pub verify: Vec<VerifierRef>,
|
||||
pub retry: RetryPolicy,
|
||||
pub timeout: Duration,
|
||||
}
|
||||
|
||||
pub enum StepKind {
|
||||
Model { prompt: PromptTemplate, effort: ReasoningEffort },
|
||||
Tool { tool: ToolId, args: ArgTemplate },
|
||||
Parallel { branches: Vec<StepId>, join: JoinPolicy },
|
||||
Conditional { on: Predicate, then: StepId, otherwise: Option<StepId> },
|
||||
SubWorkflow { workflow: WorkflowId, version: VersionSelector },
|
||||
}
|
||||
```
|
||||
|
||||
- **`WorkflowVersion` is the Blake3 hash of the canonicalized IR, not of the
|
||||
source text.** Two YAML files differing only in key order produce the same
|
||||
version — which is what makes "did this change affect results" answerable.
|
||||
- Validation and canonicalization live in the **kernel** and run on the IR, so a
|
||||
new format (T3.2) inherits every check without reimplementing one.
|
||||
- Domain states are open: a workflow declares its own step names, ordering and
|
||||
transitions as data. The kernel validates the declaration, then executes it.
|
||||
Adding a domain state is a config change, never a recompile.
|
||||
- Versions form a DAG: content-addressed, parent-pointered, **never edited**.
|
||||
Editing a version in place destroys every result already attributed to it.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Define the IR types above. Keep them free of parser concerns — no source
|
||||
spans, no format-specific fields.
|
||||
2. Write the canonicalizer: sort every unordered collection by a defined key,
|
||||
normalize whitespace inside templates only where semantically irrelevant,
|
||||
drop optional fields that equal their default, and emit a deterministic byte
|
||||
encoding.
|
||||
3. Hash the canonical bytes with Blake3 into `WorkflowVersion`.
|
||||
4. Add the parent pointer: a new version records its parent's hash. Nothing
|
||||
mutates an existing version record.
|
||||
5. Decide and document what is *not* canonicalized — prompt template text is
|
||||
semantic and must hash as written. Put that decision in a comment at the
|
||||
canonicalizer, since the next person will otherwise "improve" it.
|
||||
6. Test: build two structurally identical IRs from different field orders and
|
||||
assert equal hashes; change one prompt character and assert different hashes.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Two YAML files differing only in key order and whitespace produce identical
|
||||
`WorkflowVersion`.
|
||||
|
||||
## Verify
|
||||
|
||||
**Harness:** pairs of source files that differ only in ways that must not matter,
|
||||
and pairs that differ in ways that must.
|
||||
|
||||
**Integration test** — `tests/it_canonical_version.rs`:
|
||||
1. **Must match:** two YAML files with reordered top-level keys, reordered map
|
||||
entries, different indentation, and trailing whitespace. Assert identical
|
||||
`WorkflowVersion`.
|
||||
2. **Must differ:** change one character inside a prompt template. Assert a
|
||||
different version. This is the boundary — prompt text is semantic.
|
||||
3. **Must differ:** change a `timeout`, a `retry` count, a `StepId`. One test per
|
||||
field, table-driven, so a canonicalizer that drops a field is caught by the
|
||||
field it drops.
|
||||
4. Round trip: canonicalize twice; assert the byte output is stable.
|
||||
5. Parent pointer: build v2 from v1; assert v2 records v1's hash and that v1's
|
||||
record is unchanged on disk.
|
||||
6. Property test: generate random IRs, permute the order of every unordered
|
||||
collection, assert hash invariance.
|
||||
|
||||
**Command:** `cargo test -p workflow canonical`
|
||||
|
||||
**False pass:**
|
||||
- Only testing step 1. A canonicalizer that discards fields it does not
|
||||
understand passes every "these should match" test and silently makes two
|
||||
different workflows the same version. Step 3's per-field table is the guard.
|
||||
- Hashing the source bytes, which passes step 2 and fails step 1 — check both
|
||||
directions.
|
||||
- A property test that permutes only the collection the implementation already
|
||||
sorts.
|
||||
|
||||
## Traps
|
||||
|
||||
- Hashing the source bytes. It is one line simpler and makes every reformat look
|
||||
like a behaviour change.
|
||||
- Canonicalizing prompt text. A whitespace change inside a prompt *is* a
|
||||
behaviour change.
|
||||
- Allowing a version record to be updated in place.
|
||||
|
||||
---
|
||||
|
||||
Background (not required to do this task):
|
||||
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §2, §4.1, §12.2 ·
|
||||
[rust-agentic-task.md](../../../rust-agentic-task.md)
|
||||
Reference in New Issue
Block a user