112 lines
4.9 KiB
Markdown
112 lines
4.9 KiB
Markdown
# T7.8 — P7 composition gate
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Phase | P7 — Distribution |
|
||
| Size | L — over 3 days |
|
||
| Status | Not started |
|
||
| Flags | gate |
|
||
| Spec | inlined below |
|
||
| Blocks | P8 |
|
||
|
||
## Goal
|
||
|
||
Prove **deployment mode is interchangeable**: the same behavioural suite passes
|
||
in embedded mode and distributed mode, and two nodes survive a partition with no
|
||
double-execution.
|
||
|
||
**Phase gate criterion:** two nodes surviving a partition with no
|
||
double-execution.
|
||
|
||
## Facts (inlined — no spec read needed)
|
||
|
||
- **Two deployment modes, one set of ports.**
|
||
|
||
| Mode | Log + state | Blobs | Coordination |
|
||
|---|---|---|---|
|
||
| **Embedded** — single binary, no services | `redb` | `redb` table | in-process |
|
||
| **Distributed** — multi-node, multi-tenant | Postgres | S3-compatible | Postgres advisory locks, or Redis if leases dominate |
|
||
|
||
- **Embedded mode is a first-class product, not a test harness.** That constraint
|
||
is what keeps the ports honest — and it is only enforced if the same suite runs
|
||
in both modes.
|
||
- The properties no single P7 task owns:
|
||
- a run that survives a partition must also survive it **while grading is in
|
||
flight** — leases, the join stage and the outbox interact;
|
||
- the outbox relay must ship correctly **across a reclaim**: entries committed
|
||
by node A, relayed after node B takes over;
|
||
- **partition keys and per-run ordering must hold across a node handover** —
|
||
the same run's events must not split partitions because a different node
|
||
published them;
|
||
- exactly-once is achieved **at the fold**, not in transport, so redelivery
|
||
during a partition must be a no-op.
|
||
- Per-run total order, nothing promised across runs. Everything up to
|
||
verification is embarrassingly parallel; grading is a join; decide is
|
||
single-writer.
|
||
|
||
## Steps
|
||
|
||
1. Extract the behavioural suite — P1's composition matrix plus P4's and P5's
|
||
gates — into a mode-parameterized harness.
|
||
2. Run it in embedded mode and in distributed mode. Both must pass unmodified.
|
||
3. Build the compound distributed scenario: partition **during** grading, with
|
||
the broker down, across a lease handover.
|
||
4. Assert exactly-once effects and no lost runs across every scenario.
|
||
5. Make this the required CI job gating P8.
|
||
|
||
## Acceptance
|
||
|
||
- The behavioural suite passes **unmodified** in both deployment modes.
|
||
- Two nodes survive a partition with **no double-execution** and no lost run.
|
||
- The outbox ships every event in order across a lease handover, exactly once at
|
||
the fold.
|
||
|
||
## Verify
|
||
|
||
**Harness:** `turmoil` two-node topology; real Postgres and MinIO; the external
|
||
side-effect ledger as the double-execution witness; a consumer that folds on
|
||
`(BranchKey, Lsn)`.
|
||
|
||
**Integration test** — `tests/it_p7_composition.rs`:
|
||
1. **Mode interchange:** parameterize the behavioural suite over
|
||
`[embedded, distributed]`. Both pass with **zero** suite edits. Any
|
||
`if mode == ...` branch inside the suite is a finding about the ports.
|
||
2. **Compound scenario:** start a run, begin grading, take the broker down,
|
||
partition node A mid-step. Assert node B reclaims, the run and its grading
|
||
complete, and the ledger shows **exactly one** effect per intended effect.
|
||
3. **Relay across handover:** assert outbox entries committed by A are shipped
|
||
after B takes over — in `(BranchKey, Lsn)` order, exactly once at the fold.
|
||
4. **Partition-key stability across nodes:** assert every event of the handed-over
|
||
run landed on **one** broker partition, despite two different nodes publishing.
|
||
5. **Fencing under load:** heal the partition while B is executing. Assert A's
|
||
writes are rejected by fence and A does not resume.
|
||
6. **No lost runs:** across all scenarios, count runs spawned versus runs reaching
|
||
a terminal state. Assert equality.
|
||
7. **Grading yields:** during the scenario, assert agent-call latency stays within
|
||
bound of baseline while the join stage is saturated.
|
||
8. **Regression:** re-run P0–P6 gates in embedded mode in the same job.
|
||
|
||
**Command:** `cargo test -p distribution --test it_p7_composition -- --test-threads=1 --nocapture`
|
||
|
||
**False pass:**
|
||
- A mode-parameterized suite containing mode-specific branches. At that point the
|
||
two modes are not proven equivalent, which is the entire acceptance criterion —
|
||
and embedded mode is a shipped product, not a fixture.
|
||
- Asserting runs completed. **Completing twice also completes**; only the external
|
||
ledger distinguishes them.
|
||
- Step 3 tested without a handover, where the relay never changes owner — that is
|
||
T7.4's test, already green.
|
||
- Step 6 omitted: a run silently dropped during a partition produces no error and
|
||
passes every other assertion here.
|
||
|
||
## Traps
|
||
|
||
- Skipping embedded mode in CI for speed. It is the mode that keeps the ports
|
||
honest, and it is the cheap one to run.
|
||
|
||
---
|
||
|
||
Background (not required to do this task):
|
||
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §7, §9 ·
|
||
[rust-agentic-task.md](../../../rust-agentic-task.md)
|