RecordId storage on both servers, faster writes, a decoupled oplog, and an Ops Board
Summary: RecordId storage on both servers, faster writes, a decoupled oplog, and an Ops Board (v0.6.0b1).
The headline change is that both servers now use the same on-disk document
layout, byte for byte. The Python server's document table is keyed by RecordId —
a monotonic insertion counter — exactly as the Rust server has been since its own
RecordId work, and its index entries carry the RecordId too. That drops a write
per insert and a lookup per index-scan result (measured Python inserts ~15%
faster, unsorted scans ~35%, $group ~24%), makes an unsorted find(), tailable
cursors, and multi:false updates follow true insertion order like mongod, and —
because the two servers now agree on the whole format, documents and indexes both
— lets a store written by one server be read, backed up, and point-in-time
restored by the other. (There is no in-place upgrade from the older layout: a
pre-RecordId data directory is refused at open with a clear error naming the
mismatch; start from a fresh directory or downgrade to the build that wrote it.)
This release is also a Rust-server write-performance push plus a new tool for
running the project. The oplog — the write every change stream, point-in-time
recovery, and local.oplog.rs read depends on, and which a bare standalone
mongod doesn't keep at all — stopped re-encoding documents it had already
serialized for the collection write. That one change brought single-writer inserts
to within ~10% of a real mongod (1.1×) and updates and deletes to ~1.3×, on the
same WiredTiger engine mongod ships.
For concurrent writers there's a new opt-in async oplog (SECANTUS_OPLOG_ASYNC=1):
it moves the oplog write off the writer's critical path onto a background drainer,
lifting multi-writer write throughput ~1.4× while keeping change streams correct —
validated exactly-once under concurrent writers. The investigation behind it pinned
the remaining multi-writer ceiling squarely to WiredTiger's own aggregate write
rate: a parallel drainer pool (also shipped) does not beat it, and neither does WT
config tuning beyond ~10%, so the honest guidance for sustained concurrent-write
workloads stays "run a real mongod, or drop the oplog if you don't need change
streams." The trade for the async path is that the oplog is no longer atomic with
the data — a hard crash loses drainer-queued entries (the data itself stays fully
durable; a clean shutdown flushes) — which is why it is off by default.
The other headline is the Ops Board: a local web app that drives the whole
build / test / release / validate cycle from one place — a matrix of all thirteen
driver-conformance gauges with per-driver scores, a CI monitor with version-drift
detection and startable runs, graphical job progress with full-tree cancel, a
confirm-gated release page, and per-task explanations and time estimates. Alongside
these, a batch of correctness fixes brought more edges in line with mongod:
capped collections now evict in true FIFO order, indexes on arrays of subdocuments
actually get used, malformed wire frames surface as typed errors instead of raw
exceptions, and a panic in one collection can no longer wedge it until restart.