112 lines
4.9 KiB
Markdown
112 lines
4.9 KiB
Markdown
|
# T7.8 — P7 composition gate
|
|||
|
|
|
|||
|
|
| Field | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Phase | P7 — Distribution |
|
|||
|
|
| Size | L — over 3 days |
|
|||
|
|
| Status | Not started |
|
|||
|
|
| Flags | gate |
|
|||
|
|
| Spec | inlined below |
|
|||
|
|
| Blocks | P8 |
|
|||
|
|
|
|||
|
|
## Goal
|
|||
|
|
|
|||
|
|
Prove **deployment mode is interchangeable**: the same behavioural suite passes
|
|||
|
|
in embedded mode and distributed mode, and two nodes survive a partition with no
|
|||
|
|
double-execution.
|
|||
|
|
|
|||
|
|
**Phase gate criterion:** two nodes surviving a partition with no
|
|||
|
|
double-execution.
|
|||
|
|
|
|||
|
|
## Facts (inlined — no spec read needed)
|
|||
|
|
|
|||
|
|
- **Two deployment modes, one set of ports.**
|
|||
|
|
|
|||
|
|
| Mode | Log + state | Blobs | Coordination |
|
|||
|
|
|---|---|---|---|
|
|||
|
|
| **Embedded** — single binary, no services | `redb` | `redb` table | in-process |
|
|||
|
|
| **Distributed** — multi-node, multi-tenant | Postgres | S3-compatible | Postgres advisory locks, or Redis if leases dominate |
|
|||
|
|
|
|||
|
|
- **Embedded mode is a first-class product, not a test harness.** That constraint
|
|||
|
|
is what keeps the ports honest — and it is only enforced if the same suite runs
|
|||
|
|
in both modes.
|
|||
|
|
- The properties no single P7 task owns:
|
|||
|
|
- a run that survives a partition must also survive it **while grading is in
|
|||
|
|
flight** — leases, the join stage and the outbox interact;
|
|||
|
|
- the outbox relay must ship correctly **across a reclaim**: entries committed
|
|||
|
|
by node A, relayed after node B takes over;
|
|||
|
|
- **partition keys and per-run ordering must hold across a node handover** —
|
|||
|
|
the same run's events must not split partitions because a different node
|
|||
|
|
published them;
|
|||
|
|
- exactly-once is achieved **at the fold**, not in transport, so redelivery
|
|||
|
|
during a partition must be a no-op.
|
|||
|
|
- Per-run total order, nothing promised across runs. Everything up to
|
|||
|
|
verification is embarrassingly parallel; grading is a join; decide is
|
|||
|
|
single-writer.
|
|||
|
|
|
|||
|
|
## Steps
|
|||
|
|
|
|||
|
|
1. Extract the behavioural suite — P1's composition matrix plus P4's and P5's
|
|||
|
|
gates — into a mode-parameterized harness.
|
|||
|
|
2. Run it in embedded mode and in distributed mode. Both must pass unmodified.
|
|||
|
|
3. Build the compound distributed scenario: partition **during** grading, with
|
|||
|
|
the broker down, across a lease handover.
|
|||
|
|
4. Assert exactly-once effects and no lost runs across every scenario.
|
|||
|
|
5. Make this the required CI job gating P8.
|
|||
|
|
|
|||
|
|
## Acceptance
|
|||
|
|
|
|||
|
|
- The behavioural suite passes **unmodified** in both deployment modes.
|
|||
|
|
- Two nodes survive a partition with **no double-execution** and no lost run.
|
|||
|
|
- The outbox ships every event in order across a lease handover, exactly once at
|
|||
|
|
the fold.
|
|||
|
|
|
|||
|
|
## Verify
|
|||
|
|
|
|||
|
|
**Harness:** `turmoil` two-node topology; real Postgres and MinIO; the external
|
|||
|
|
side-effect ledger as the double-execution witness; a consumer that folds on
|
|||
|
|
`(BranchKey, Lsn)`.
|
|||
|
|
|
|||
|
|
**Integration test** — `tests/it_p7_composition.rs`:
|
|||
|
|
1. **Mode interchange:** parameterize the behavioural suite over
|
|||
|
|
`[embedded, distributed]`. Both pass with **zero** suite edits. Any
|
|||
|
|
`if mode == ...` branch inside the suite is a finding about the ports.
|
|||
|
|
2. **Compound scenario:** start a run, begin grading, take the broker down,
|
|||
|
|
partition node A mid-step. Assert node B reclaims, the run and its grading
|
|||
|
|
complete, and the ledger shows **exactly one** effect per intended effect.
|
|||
|
|
3. **Relay across handover:** assert outbox entries committed by A are shipped
|
|||
|
|
after B takes over — in `(BranchKey, Lsn)` order, exactly once at the fold.
|
|||
|
|
4. **Partition-key stability across nodes:** assert every event of the handed-over
|
|||
|
|
run landed on **one** broker partition, despite two different nodes publishing.
|
|||
|
|
5. **Fencing under load:** heal the partition while B is executing. Assert A's
|
|||
|
|
writes are rejected by fence and A does not resume.
|
|||
|
|
6. **No lost runs:** across all scenarios, count runs spawned versus runs reaching
|
|||
|
|
a terminal state. Assert equality.
|
|||
|
|
7. **Grading yields:** during the scenario, assert agent-call latency stays within
|
|||
|
|
bound of baseline while the join stage is saturated.
|
|||
|
|
8. **Regression:** re-run P0–P6 gates in embedded mode in the same job.
|
|||
|
|
|
|||
|
|
**Command:** `cargo test -p distribution --test it_p7_composition -- --test-threads=1 --nocapture`
|
|||
|
|
|
|||
|
|
**False pass:**
|
|||
|
|
- A mode-parameterized suite containing mode-specific branches. At that point the
|
|||
|
|
two modes are not proven equivalent, which is the entire acceptance criterion —
|
|||
|
|
and embedded mode is a shipped product, not a fixture.
|
|||
|
|
- Asserting runs completed. **Completing twice also completes**; only the external
|
|||
|
|
ledger distinguishes them.
|
|||
|
|
- Step 3 tested without a handover, where the relay never changes owner — that is
|
|||
|
|
T7.4's test, already green.
|
|||
|
|
- Step 6 omitted: a run silently dropped during a partition produces no error and
|
|||
|
|
passes every other assertion here.
|
|||
|
|
|
|||
|
|
## Traps
|
|||
|
|
|
|||
|
|
- Skipping embedded mode in CI for speed. It is the mode that keeps the ports
|
|||
|
|
honest, and it is the cheap one to run.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Background (not required to do this task):
|
|||
|
|
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §7, §9 ·
|
|||
|
|
[rust-agentic-task.md](../../../rust-agentic-task.md)
|