Writes got a lot faster, and we built the rig that proved it
Summary: Writes got a lot faster, and we built the rig that proved it (v0.6.0b14).
The block compressor changed from zlib to lz4, and it is the largest single
performance change this engine has had. Profiling the daemon under sustained
write load found 65% of server CPU inside zlib's deflate, on
WiredTiger's page-reconciliation path — which is also why tail latency was
CPU-bound rather than I/O-bound. Swapping the compressor measured +86%
write throughput and a 97% cut in p99.9 latency on incompressible data, and
+15% / −88% on compressible data where zlib's ratio is at its best. End to end
on the three-droplet rig that is now part of the release process, that took
SecantusDB from 3,993 to 11,099 operations per second and cut p99.9 latency
from 1,303 ms to 37 ms — closing the gap to a real mongod from 0.27x
throughput to 0.75x, and from 72x its tail latency to 2.0x. Existing
data is safe: block_compressor is recorded per table at create time, so
stores written before this release keep their zlib tables and zlib stays
linked permanently to read them.
Finding it needed a better instrument than the single-host benchmarks this
project had. There is now a three-droplet DigitalOcean harness that runs
SecantusDB and a real mongod back-to-back on identical hardware — one server
machine, two client machines, a real network between them — and reports
medians across interleaved passes with a spread column, so a number comes with
evidence about whether it is worth quoting. docs/benchmark.md publishes that
comparison honestly, tail-latency weakness included, and the release process
now re-measures it so the published figures cannot quietly go stale.
The PostgreSQL wire surface also grew substantially this cycle: stored
procedures (CREATE PROCEDURE / CALL / DROP PROCEDURE), DROP TRIGGER,
statement timeouts, anonymous record constructors, composite and
composite-array catalog types, and a long list of cast, length and comparison
fixes that bring psql, JDBC and SQLAlchemy closer to behaving as they do
against a real PostgreSQL.
Note on 0.6.0b13: it was tagged but never published — its macOS
wheel failed delocate's deployment-target check, because Homebrew's
liblz4 targets macOS 14 while the wheel targets macOS 11. Accepting it
would have dropped Apple Silicon users on macOS 11–13, so the lz4
dependency is now built from source at the wheel's own target. No
0.6.0b13 artefacts exist on PyPI.