The numbers behind it.

Claims about money should come with measurements. These are the chaos campaigns and the benchmarks as they came out, read straight from the result files in the repository, including the parts that didn't go my way.

Chaos

Payments run while processes are killed with SIGKILL, Postgres and Kafka crash, the network misbehaves and the bank lies. After each run an independent checker reads the books.

Runs

Journeys and journals

74,388

Kills

616

Runs with a violation

A zero only means something if the harness can see a violation. So each guard gets a mutant: the same code with that one guard removed. Every one has to be caught, except where another guard is meant to cover for it.

Mutant	Guard removed	Harness	Result
M1	no unique reference on journals and attemptspayments: survived 5 runs	payments	survived 5 runs
M1+M2	both of the abovepayments: caught in run 1 by S3, S4, S8	payments	caught in run 1 by S3, S4, S8
M2	no conditional update on payment statepayments: caught in run 1 by S3, S8	payments	caught in run 1 by S3, S8
M3	no inbox dedupe in the consumerspayments: caught in run 1 by L3	payments	caught in run 1 by L3
M4	the relay marks events published before producing thempayments: caught in run 1 by L2, L3	payments	caught in run 1 by L2, L3
M5	no request hash check on idempotency keyspayments: caught in run 1 by S5	payments	caught in run 1 by S5
M6	no amount check on the bank's answerspayments: caught in run 5 by S4	payments	caught in run 5 by S4
M6	no amount check on the bank's answerspayments: caught in run 1 by S4	payments	caught in run 1 by S4
M7	a balance read without its lock and written back (a lost update)ledger: caught in run 1 by S2	ledger	caught in run 1 by S2
M7	a balance read without its lock and written back (a lost update)payments: caught in run 1 by S2, S6	payments	caught in run 1 by S2, S6
M7	a balance read without its lock and written back (a lost update)ledger: caught in run 1 by S2	ledger	caught in run 1 by S2
M7a	only FOR UPDATE dropped from the lock (control)ledger: survived 3 runs	ledger	survived 3 runs
M7a	only FOR UPDATE dropped from the lock (control)payments: caught in run 2 by S6	payments	caught in run 2 by S6
M8	webhooks applied as sent instead of pulledpayments: caught in run 1 by S4	payments	caught in run 1 by S4
M9	a snapshot horizon that counts running transactionsledger: caught in run 1 by S2	ledger	caught in run 1 by S2
M9	a snapshot horizon that counts running transactionsledger: caught in run 1 by S2	ledger	caught in run 1 by S2
M10	the settlement journal leaves the fee outpayments: caught in run 1 by L5	payments	caught in run 1 by L5
M11	the leaf encoding leaves the reference outpayments: caught in run 1 by S7	payments	caught in run 1 by S7
M12	the checkpointer sends the wrong rootpayments: caught in run 1 by S7, L4	payments	caught in run 1 by S7, L4

M1 survives on purpose: the conditional state update still stops a double capture, and M1 with M2 together is caught at once. M7a is a control: in the ledger harness it changes nothing, and in the payments harness S6 still catches the order it gets wrong. The first M6 campaign needed five runs, because wrong amounts at 0.2% were too rare to test the guard; at 0.5% it was caught in every run.

The hot account

Every payment touches the same fee and receivable accounts, so a plain balance row becomes a lock every payment waits on. Five ways to keep that balance, offered from 50 to 3,200 journals a second, each rate measured for 15 s, three times.

Connections

Journals a second, achievedLocked rowAsync commitSerializableDerivedSharded, 16

p99 latencyLocked rowAsync commitSerializableDerivedSharded, 16

The numbers as a table

Offered	Locked row	Async commit	Serializable	Derived	Sharded, 16
50	51 · 31 ms	51 · 15 ms	51 · 60 ms	51 · 16 ms	51 · 17 ms
100	102 · 59 ms	102 · 18 ms	95 · 77 ms	102 · 16 ms	102 · 19 ms
150	149 · 1.5 s	151 · 25 ms	133 · 72 ms	151 · 17 ms	151 · 18 ms
200	149 · 5.2 s	200 · 36 ms	—	200 · 18 ms	200 · 19 ms
300	—	307 · 68 ms	—	307 · 19 ms	307 · 17 ms
400	—	406 · 57 ms	—	406 · 20 ms	406 · 15 ms
800	—	530 · 4.3 s	—	802 · 81 ms	788 · 285 ms
1,200	—	—	—	915 · 2.7 s	743 · 3.9 s

Each cell is achieved journals a second and p99, the median of three runs.

A locked row tops out near 150 journals a second, whatever the connection count, because each commit waits for the one before it to flush. Serializable is worse at every count. Derived balances and a balance split into 16 rows both sustain about 700 at 32 connections, with p99 under 50 ms. The ledger uses the 16 rows, because their reads stay cheap.

The posting path

One journal of three entries, one connection, the locked mode. The same work through four ways of talking to Postgres.

How	Round trips	p50	p99	Journals a second
Prisma query API	8	17.4 ms	30.4 ms	61
Prisma, raw statements	6	13.9 ms	21.1 ms	77
Prisma, one CTE	4	10.8 ms	21.7 ms	92
pg, one CTE	4	10.1 ms	24.8 ms	101

Outbox lag

Events are written in the same transaction as the money and a relay publishes them to Kafka. How long that takes, at three steady rates (five runs each, median and range), and what happens when the broker is killed for 30 s at 500 events a second.

Events a second	To Kafka, p50	To Kafka, p99	To the consumer, p99	Transactions an event
100	20.2 ms	46.1 ms (45–47)	64.9 ms (63–67)	5.48
500	16.4 ms	36.0 ms (35–38)	94.3 ms (91–101)	2.68
2,000	21.1 ms	46.0 ms (45–279)	146.0 ms (134–349)	1.35

Broker killed, run 1
Back after 44.1 s, first publish 6.5 s later. 25,084 events waited and drained in 1.9 s at 13,131 a second; latency was normal again 54.9 s after the broker came back.
Broker killed, run 2
Back after 43.8 s, first publish 6.7 s later. 24,912 events waited and drained in 1.9 s at 13,304 a second; latency was normal again 57.2 s after the broker came back.
Broker killed, run 3
Back after 43.6 s, first publish 7.3 s later. 24,772 events waited and drained in 1.8 s at 13,959 a second; latency was normal again 24.4 s after the broker came back.

Audit log

Every committed journal becomes a leaf in a Merkle tree, and a witness outside the database signs its roots. How fast that runs as the log grows, how big a proof is, and whether tampering is caught.

Journals	Sequenced a second	Full verification	Proof size	Bytes a journal
100,000	25,462	2.8 s	320–544 B	405
500,000	23,172	13.9 s	448–608 B	404
1,000,000	21,718	31.5 s	480–640 B	404

Checking only what was added on top of a million: 1,000 journals in 33 ms, 10,000 journals in 228 ms, 100,000 journals in 2,236 ms. Under 200 payments a second, a journal waited 510 ms for the sequencer at the median and 1,026 ms at p99.

Tampering	Trials	Caught	Right journal named	Caught without the witness
class 1, covered	917	917	917	917
class 1, above the last checkpoint	83	83	83	83
class 2, covered	900	900	900	0
class 2, above the last checkpoint	100	0	0	0
class 3, covered	887	887	887	887
class 3, above the last checkpoint	113	113	113	113

Reconciliation

Settlement files with faults injected at known rates, over three seeds. With references, everything is found. Without them, matching on the amount guesses, and a car wash sells five prices.

Payments a day	Fallback	Missing line	Duplicate line	Wrong amount	Fee difference	Settles next day	Dropped reference	No reference: placed wrong
50	unique	100%	100%	100%	100%	100%	100%	0%
50	oldest	95.1%	100%	100%	100%	100%	87%	13%
500	unique	100%	100%	100%	100%	100%	100%	0%
500	oldest	63.9%	100%	100%	100%	100%	51.3%	49%
2,000	unique	100%	100%	100%	100%	100%	100%	0%
2,000	oldest	53.9%	100%	100%	100%	100%	23.5%	77%

Still to measure

These need a quiet machine and run at night. Their numbers will appear here as they come in.

Schema change
Adding a column to 20 million entries under live payments, naive against expand and contract.
A long chaos campaign
A few hundred runs instead of five, on the final code.