128 lines
5.3 KiB
Markdown
128 lines
5.3 KiB
Markdown
# T2.2 — Effect-class recovery
|
|||
|
|
|
||
|
|
| Field | Value |
|
||
|
|
|---|---|
|
||
|
|
| Phase | P2 — Durability hard parts |
|
||
|
|
| Size | M — 1 to 3 days |
|
||
|
|
| Status | Not started |
|
||
|
|
| Flags | — |
|
||
|
|
| Spec | inlined below |
|
||
|
|
| Blocks | T6.5, T8.3 |
|
||
|
|
|
||
|
|
## Goal
|
||
|
|
|
||
|
|
Resolve a `Dispatched`-but-unresolved intent by the tool's declared effect class.
|
||
|
|
`Indeterminate` becomes a real terminal state with an operator path.
|
||
|
|
|
||
|
|
## Facts (inlined — no spec read needed)
|
||
|
|
|
||
|
|
```rust
|
||
|
|
pub struct ToolRegistration {
|
||
|
|
pub id: ToolId,
|
||
|
|
/// No Default. The author is the only party who knows this.
|
||
|
|
pub effects: EffectClass,
|
||
|
|
pub caps: CapabilitySet,
|
||
|
|
pub timeout: Duration,
|
||
|
|
}
|
||
|
|
|
||
|
|
/// Each class carries what its recovery path actually needs. A bare
|
||
|
|
/// discriminant would let a tool claim `Idempotent` while withholding the one
|
||
|
|
/// thing that makes the claim actionable.
|
||
|
|
pub enum EffectClass {
|
||
|
|
Idempotent { key: KeyDerivation },
|
||
|
|
Queryable { lookup: RequestIdLookup },
|
||
|
|
Unsafe,
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
Resolution for a `Dispatched` intent:
|
||
|
|
|
||
|
|
| Class | Recovery |
|
||
|
|
|---|---|
|
||
|
|
| `Idempotent` | retry with the same idempotency key; the provider deduplicates |
|
||
|
|
| `Queryable` | ask the provider whether the request id landed, then complete or retry |
|
||
|
|
| `Unsafe` | **never auto-retry.** Attempt becomes `Indeterminate`, operator notified |
|
||
|
|
|
||
|
|
- The third row is the honest one. Some effects cannot be made safe by any
|
||
|
|
protocol. The intent log's value is converting an invisible unknown into a
|
||
|
|
recorded one: `Indeterminate` is a fact a grader and an operator can both use;
|
||
|
|
a silently retried payment is not.
|
||
|
|
- A tool that cannot state whether it is safe to retry **does not register**.
|
||
|
|
- Declaration is not enforcement — that is T8.3's keyed capability, and the
|
||
|
|
sandbox (T6.5) is the third layer. This task builds layer one plus recovery.
|
||
|
|
- Until built-in tools are audited, default `Unsafe` and never auto-retry.
|
||
|
|
|
||
|
|
## Steps
|
||
|
|
|
||
|
|
1. Define `EffectClass` with payloads as above — no bare discriminants. A
|
||
|
|
registration declaring `Idempotent` without a `KeyDerivation` must not compile.
|
||
|
|
2. Extend `ToolRegistration` and reject registration of a tool with no declared
|
||
|
|
effect class. No `Default` impl.
|
||
|
|
3. Implement the recovery pass, driven by T2.1's restart scan: for each
|
||
|
|
`Dispatched` intent, dispatch on the registered class.
|
||
|
|
4. `Idempotent`: re-derive the key from the recorded arguments, re-issue, record
|
||
|
|
the outcome. Same key, so the provider deduplicates.
|
||
|
|
5. `Queryable`: call the registered `RequestIdLookup` with the recorded request
|
||
|
|
id; complete from the provider's answer or retry if it never landed.
|
||
|
|
6. `Unsafe`: transition the attempt to `Indeterminate`, emit the operator
|
||
|
|
notification, and **stop**. No retry path exists for this arm — do not add one
|
||
|
|
behind a flag.
|
||
|
|
7. Build three test tools, one per class, each with an injectable crash point
|
||
|
|
mid-call.
|
||
|
|
|
||
|
|
## Acceptance
|
||
|
|
|
||
|
|
- Three tools, one per class; crash mid-call for each.
|
||
|
|
- `Idempotent` retries and produces no duplicate effect.
|
||
|
|
- `Queryable` reconciles against the provider and completes or retries correctly.
|
||
|
|
- `Unsafe` becomes `Indeterminate` and raises, rather than retrying.
|
||
|
|
|
||
|
|
## Verify
|
||
|
|
|
||
|
|
**Harness:** three real test tools, one per class, each writing to an
|
||
|
|
**external** side-effect ledger (a file or a counter service) so effect count is
|
||
|
|
observable independently of the log. Plus T2.1's child-process kill.
|
||
|
|
|
||
|
|
**Integration test** — `tests/it_effect_recovery.rs`:
|
||
|
|
1. **`Idempotent`:** tool appends to the ledger keyed by its idempotency key.
|
||
|
|
Kill mid-call, restart, let recovery retry. Assert the ledger holds
|
||
|
|
**exactly one** entry and the attempt completes.
|
||
|
|
2. **`Queryable`:** tool records a request id; the fake provider is configurable
|
||
|
|
to answer "landed" or "never arrived".
|
||
|
|
- Provider says landed → assert recovery **completes** without re-issuing;
|
||
|
|
ledger count stays 1.
|
||
|
|
- Provider says never arrived → assert recovery **retries**; ledger count
|
||
|
|
becomes 1 (from the retry, not two).
|
||
|
|
3. **`Unsafe`:** kill mid-call. Assert the attempt becomes `Indeterminate`, the
|
||
|
|
operator notification fires, and the ledger count is **unchanged** — no retry
|
||
|
|
was issued.
|
||
|
|
4. Registration: a tool with no declared effect class is rejected; a
|
||
|
|
`Idempotent` registration without a `KeyDerivation` fails to compile
|
||
|
|
(`trybuild`).
|
||
|
|
5. Key stability: run recovery twice; assert the derived key is identical both
|
||
|
|
times.
|
||
|
|
|
||
|
|
**Command:** `cargo test -p durability --features test-hooks effect_recovery -- --test-threads=1`
|
||
|
|
|
||
|
|
**False pass:**
|
||
|
|
- Counting effects only in the framework's own log. The log is what you are
|
||
|
|
testing; the external ledger is the witness.
|
||
|
|
- The `Idempotent` case passing because the provider stub deduplicates on
|
||
|
|
*content* rather than on the supplied key — it then passes even when the key
|
||
|
|
derivation is broken. Make the stub key-only.
|
||
|
|
- Testing `Unsafe` by asserting an error is returned. `Indeterminate` is a state,
|
||
|
|
not an error, and the distinction is the deliverable.
|
||
|
|
|
||
|
|
## Traps
|
||
|
|
|
||
|
|
- A "retry anyway with a warning" path for `Unsafe`. The class exists to say the
|
||
|
|
answer is no.
|
||
|
|
- Deriving the idempotency key at recovery time from something that changed — it
|
||
|
|
must derive from the recorded arguments, deterministically.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
Background (not required to do this task):
|
||
|
|
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §8.4, §13.1, §13.2 ·
|
||
|
|
[rust-agentic-task.md](../../../rust-agentic-task.md)
|