104 lines
4.6 KiB
Markdown
104 lines
4.6 KiB
Markdown
# T7.1 — Postgres `EventLog`
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Phase | P7 — Distribution |
|
|
| Size | L — over 3 days |
|
|
| Status | Not started |
|
|
| Flags | — |
|
|
| Spec | inlined below |
|
|
| Blocks | — |
|
|
|
|
## Goal
|
|
|
|
The same `EventLog` port over Postgres, partitioned by tenant and time. Not a
|
|
second port — the same one.
|
|
|
|
## Facts (inlined — no spec read needed)
|
|
|
|
- Two deployment modes, **one set of ports**:
|
|
|
|
| Mode | Log + state | Blobs | Coordination |
|
|
|---|---|---|---|
|
|
| **Embedded** — single binary, no services | `redb` | `redb` table | in-process |
|
|
| **Distributed** — multi-node, multi-tenant | Postgres | S3-compatible | Postgres advisory locks, or Redis if leases dominate |
|
|
|
|
- Postgres transactions supply crash-atomicity exactly as `redb`'s shadow paging
|
|
does; `commit`'s four writes stay in one transaction.
|
|
- This is why the port was async from the first line: a sync signature over a
|
|
network call is impossible, and the local implementation paid only a negligible
|
|
poll for the compatibility.
|
|
- **Every port test runs against both backends or the abstraction is not real.**
|
|
The conformance suite from T0.5/T0.7 is the deliverable being reused here.
|
|
- Open at this scale: partitioning by tenant and time, index strategy for branch
|
|
scans, and whether the outbox is a table or a logical replication slot. Decide
|
|
and record the decision; it is not pre-settled.
|
|
|
|
## Steps
|
|
|
|
1. Schema: log table keyed `(tenant, run, branch, lsn)`; state, consumer position
|
|
and outbox tables mirroring the embedded key shapes.
|
|
2. Partition the log by tenant and by time. Choose declarative partitioning and
|
|
document the retention interaction with T8.4's tiering.
|
|
3. Index for the access patterns that exist: branch scans from an LSN, recent
|
|
runs by `RunId` prefix (ULIDs sort by creation time), outbox drain per
|
|
`BranchKey` ascending `Lsn`.
|
|
4. Implement `commit` as one transaction with all four writes. LSN allocation
|
|
stays inside the transaction — a sequence is not sufficient, since LSNs are
|
|
per branch and must be gap-free.
|
|
5. Implement the checkpoint and outbox methods against the same schema.
|
|
6. Run the **unmodified** P0 conformance suite against this backend in CI.
|
|
7. Record the outbox decision (table vs replication slot) with its reasoning.
|
|
|
|
## Acceptance
|
|
|
|
- The P0 conformance suite passes **unmodified** against Postgres.
|
|
- Every port test runs against both backends.
|
|
|
|
## Verify
|
|
|
|
**Harness:** the **same** conformance function from T0.5, instantiated with a
|
|
Postgres factory. A real Postgres in CI (testcontainers or a service container) —
|
|
not an in-memory fake, which would test nothing about the backend.
|
|
|
|
**Integration test** — `tests/it_postgres_conformance.rs`:
|
|
1. Call `conformance(|| PostgresEventLog::new(...))` with **zero modifications**
|
|
to the suite. Any needed change is a finding about the port, not the test.
|
|
2. Run the identical suite against `redb` in the same CI job, so a divergence
|
|
between backends fails immediately rather than months later.
|
|
3. **Concurrent LSN allocation** across separate **connections** — the redb test
|
|
used tasks; here contention is across processes and is the real case. Assert
|
|
`1..=n` with no gaps or repeats.
|
|
4. Assert `commit`'s four writes are in one transaction: kill the connection
|
|
mid-commit and assert all-or-nothing on reconnect.
|
|
5. Partitioning: insert across several tenants and time ranges; assert a branch
|
|
scan hits the expected partitions (check the query plan, not just the result).
|
|
6. Outbox: assert `drain_outbox` returns `(BranchKey, Lsn)` ascending across
|
|
partition boundaries.
|
|
7. Record the outbox decision (table vs replication slot) and assert the chosen
|
|
one is the one under test.
|
|
|
|
**Command:** `cargo test -p storage-postgres --test it_postgres_conformance`
|
|
|
|
**False pass:**
|
|
- A modified copy of the conformance suite. The moment it is edited "just for
|
|
Postgres", the two backends are no longer proven equivalent — which is the
|
|
entire acceptance criterion.
|
|
- Step 3 with a single connection, where the sequence is uncontended.
|
|
- A `SERIAL`/sequence-backed LSN. It passes casual reads and is neither per-branch
|
|
nor gap-free under rollback.
|
|
- Running against SQLite or an in-process fake for CI speed.
|
|
|
|
## Traps
|
|
|
|
- A Postgres-only method added to the trait "temporarily". The embedded backend
|
|
then has a stub and the abstraction is fiction.
|
|
- A global sequence for LSNs. It serializes every run through one atomic and
|
|
breaks per-branch gap-freedom.
|
|
|
|
---
|
|
|
|
Background (not required to do this task):
|
|
[rust-agentic-sys.md](../../../rust-agentic-sys.md) §7, §8.3, §18 ·
|
|
[rust-agentic-task.md](../../../rust-agentic-task.md)
|