Changelog

All notable changes to SecantusDB are documented here. This file is the system of record for what shipped in each release — the per-release blog posts on secantusdb.com are generated from these entries via tools/generate_blog_post.py.

Format follows Keep a Changelog with one extension: each release carries a one-to-three-paragraph prose lede between the date line and the structured #### Added / #### Changed / #### Fixed subsections. The prose lede is what the blog generator lifts verbatim as the marketing-post body, so it should read as a self-contained narrative — not as “v0.5.1bN ships X.”

This project adheres roughly to Semantic Versioning, but while we’re in beta the patch number bN rolls forward on every PyPI-visible push; the API surface itself is shaped by Semantic Versioning intent.

Unreleased

[0.7.0b3] — 2026-10-14

The per-operation benchmark measures its servers interleaved

Three droplet runs of one build put the Rust server’s multi-stage aggregation at 2.8×, 3.6× and 4.1× of mongod. The harness measured each server’s reps back to back, so minutes separated the two halves of every ratio and a noisy neighbour on the shared-CPU droplet moved one column only.

Changed

• bench.compare_servers runs every server inside each rep, rotating the order, and invoke do-perf takes fifteen reps instead of five. Two droplet runs of the same build that way agreed on multi-stage aggregation to 0.24× (was 1.24×) and on delete_many to 0.02× (was 1.02×).

• The results file records, per workload, the lowest and highest ratio among the individual passes (rust_x_range, py_x_range), so a noisy run shows as one.

• The published per-operation figures now come from that harness. Across three interleaved droplet runs the Rust server is at 1.0×–3.2× of mongod (published before as 1.0×–4.1×), the Python server at 2.0×–26.7×, and the Rust server 1.9×–13.5× faster than the Python one. A third run widened the agreement on multi-stage aggregation to 0.63×: better than 1.24×, and still the row that moves most.

scripts/detached_run.py: a stopped run always reads as finished

On Windows CI, status could crash on every call after stop, so a stopped run never reported “finished”. The supervisor could be killed while writing its child’s exit code, leaving that file empty.

Fixed

• The exit file and the state file are written beside their name and renamed into place, so a kill or a concurrent reader never sees one half written.

• A missing, empty or garbled exit file reads as exit code -1 instead of raising.

• stop records the run’s end itself instead of leaving it to the next status.

detached_run.py: an exit that came before the state file

Fixed

• A command started with scripts/detached_run.py start that ended at once could end before its state file was written. Its exit was then recorded nowhere a reader of <name>.json would see it, and the file said exit_code: null until status or wait was called. start now records an exit that is already there.

The droplet benchmark: a task for every command, and one for PostgreSQL

Added

• invoke do-destroy and invoke do-ssh, and --server-size / --client-size on invoke do-up. Every do-cluster command now has a task.

• invoke do-pg-bench (bench/do_pg_bench.py): the measurement behind the site’s Rust PostgreSQL table, a released secantusd-pg against PostgreSQL 16 on the server droplet. It was done by hand, and so had not been repeated since 0.1.0-beta.7. The raw trials are kept under bench/results/pg-droplet/.

• That measurement runs its clients on the two client droplets, over the private network (bench/pg_remote_client.py). It used to run them beside the server, where a lone client and the server share CPUs: one trial of one or four clients then moved by up to 21%.

Changed

• bench/pg_concurrency.py checks that the server binary is current when it starts the server, where it used to check on import.

The Java driver gauge no longer deletes its own test data

The Java gauge reported 495 of 496 for the Rust MongoDB server, with a different test failing on each run, always a read that was missing documents inserted a moment earlier. That reads like lost writes. It was the gauge.

The driver’s test fixture drops the shared JavaDriverTest database when each test JVM exits, and the gauge overrode the driver’s own one-JVM setting to run twelve at once against one server. A JVM that finished deleted the data under the ones still running. A logging proxy in front of the server showed a dropDatabase from another connection 2 ms before the failing read.

Fixed

• invoke validate-java runs one test JVM at a time by default, as the driver itself does. SECANTUS_GAUGE_PARALLEL_FORKS still raises it for timing experiments.

• The Rust server’s Java panel is corrected to 496 of 496.

• invoke validate-kotlin does the same: its tests use the same fixture. It had not shown a failure, and one JVM at a time reports the same 340 of 340 on both servers.

The PostgreSQL droplet benchmark refuses a slow disk and records why a trial was slow

Two things went wrong in the measurement behind the Rust PostgreSQL page’s table on 2026-10-13, and neither can now happen quietly.

Added

• invoke do-pg-up creates the three droplets, probes the server’s disk, and destroys and creates them again while it completes fewer than 5,000 fdatasync calls a second, four times at most. Two of three c-16 droplets that day completed about 1,700, where both servers wait for the disk and the write rows come out near 1.0x.

• invoke do-pg-bench makes the same probe first and refuses a slow disk. bench/do_pg_bench.py --check-disk runs the probe alone, and --min-fdatasync 0 measures anywhere.

• Every trial now stores what the server droplet did during it: the disk’s flushes and the time each took, and the server’s CPU a statement. A row whose trials differ by more than 20% of their median is printed under the table trial by trial with those figures. Three trials of one row had fallen by a third with nothing recorded that could say why; fifty-four further trials did not repeat it.

The Rust PostgreSQL server no longer loses a cancel that arrives before execution

A CancelRequest that reached secantusd-pg while a statement was still being parsed, bound or described was silently dropped, and the statement ran to completion. The server cleared its pending-cancel flag at the moment a statement began executing, on the reasoning that a cancel received while idle targets nothing. That reasoning is right for an idle backend and wrong for one that has already started on the client’s message.

Probed against PostgreSQL 15.19 with a statement that takes seconds to parse: PostgreSQL answered 57014 for a cancel sent anywhere from 0 to 640 ms after the statement, and this server ran the full sleep at every offset. This is how psycopg’s test_generators.py::test_cancel ran its pg_sleep(180) to the end on the Windows gauge runner.

Fixed

• A cancel is now dropped only if it arrives before the first message of a cycle (the backend is idle). One that arrives later stays pending until the next cancellation point, whichever message is being processed.

• A statement nested inside another (a DO block’s inner statements) no longer clears a cancel aimed at the statement that contains it.

• A cancel is consumed by the statement it interrupts, so a handler that catches query_canceled is not cancelled again by the same request.

The Rust PostgreSQL server zero-fills its log on Linux

On Linux every durable commit ends in an fdatasync of the log, and WiredTiger’s log file is reserved on disk but never written, so each of those syncs also had the filesystem record newly written blocks. The server now writes zeros through a new log file when it creates it, as PostgreSQL does for a WAL segment.

Changed

• secantusd-pg on Linux, when commits sync (the default; not under SECANTUS_TEST_FAST_STORAGE=1), creates its log files zero-filled and 16 MB each where they were 128 MB. A store written by an earlier version opens unchanged, and an earlier version opens a store written by this one.

• macOS and the MongoDB server are unchanged.

The Rust PostgreSQL server no longer reuses a plan across a folded parameter

A prepared statement whose parameter sat inside an expression the planner folds into a value – WHERE k = $1 % 100000 – had its plan reused with the raw parameter substituted, so a later execution returned the wrong row. This was in 0.1.0-beta.3.

Fixed

• secantusd-pg reuses a prepared statement’s plan only when every parameter stands alone where its value is used: compared with a column in the WHERE (k = $1, k IN ($1, $2)), an item of an INSERT’s VALUES, the whole of SET v = $1, or one operand of SET v = v + $1. A parameter anywhere else (k = $1 % 100000, k = -$1, $1::int2, BETWEEN, LIMIT $1) is planned on every execution.

• The same reuse would have stored the wrong value for INSERT ... VALUES ($1 % 1000) and UPDATE ... SET v = $1 % 1000; that part was introduced after 0.1.0-beta.3 and never released.

The Rust PostgreSQL server reuses the plan of a prepared INSERT or UPDATE

A prepared statement’s plan was reused only when its parameters landed in a WHERE clause. An INSERT with bound values, an UPDATE ... SET v = $1 and an UPDATE ... SET v = v + 1 were planned again on every execution, which was a quarter to a third of the server’s CPU time for those statements.

Changed

• secantusd-pg reuses the plan when a parameter is stored into an int4, int8, float8 or text column from a value of the matching kind, and when a parameter is read by an expression the server evaluates per row. Server CPU per statement, macOS, release build: INSERT 49 to 42 us, UPDATE SET v = $1 59 to 47 us, UPDATE SET v = v + 1 69 to 57 us, an UPDATE matching no row 45 to 29 us.

• A column with a declared width, a narrower range, a domain, an enum or a generated value is planned per execution as before, so its checks see every value.

Fixed

• secantusd-pg holds a smallint to its range. 40000::smallint answered 40000 and a smallint column stored it; both are now PostgreSQL’s 22003 smallint out of range. A float outside integer’s range (or NaN) cast to an integer is 22003 where it used to saturate, and digits too large for the type ('99999999999'::int) are 22003 value ... is out of range for type integer where they were 22P02.

The Rust PostgreSQL server reuses a plan that stores a small integer in a bigint column

A driver sends a small integer as int2 or int4. Bound for a bigint column, the planner widens it, and a prepared INSERT or UPDATE ... SET that did so was planned again on every execution.

Changed

• secantusd-pg reuses the plan when an int2 or int4 parameter is stored in a plain bigint column, carrying the widening with the value.

• The secantusd-pg binary allocates with mimalloc, as secantusd-rs does. It is the default mimalloc feature of the secantus-pg crate; --no-default-features --features bin builds without it. A program that embeds secantus_pg::PgServer keeps its own allocator.

The Rust PostgreSQL server’s release build is optimised across the whole program, and by a profile

The released secantusd-pg binary was compiled at Cargo’s defaults: no link-time optimisation and no profile. Measured on a 16-vCPU droplet with one client selecting by primary key, that binary answered 12,052 statements a second. Built with link-time optimisation it answered 14,402, and with a profile as well 17,855.

Changed

• crates/secantus-pg builds its release profile with lto = "fat" and one codegen unit. cargo install secantus-pg gets this too. A release build takes longer: about seven minutes on 16 vCPUs.

• The released binaries are built in two stages by scripts/build_pg_release.sh: an instrumented build runs bench/pgo_workload_pg.py, and the build that ships uses the profile it wrote. cargo install cannot have the profile. The two stages take 16 minutes on 16 vCPUs.

Fixed

• invoke do-pg-bench measured PostgreSQL over TLS and secantusd-pg in the clear. Ubuntu’s PostgreSQL has ssl = on and psycopg asks for TLS when the server offers it, so PostgreSQL paid for encryption the other side did not. Every DSN the benchmark uses now says sslmode=disable, the run records pg_stat_ssl for its own connection, and it stops if that says TLS was on.

• A benchmark droplet’s disk decides the write rows, and one droplet of the same size completed a fifth of the commits a second of the one before it. The run now records the processor and pg_test_fsync’s rate, and the table’s footnote prints them.

The psycopg gauge removes its server’s store

Fixed

• psycopg_validation.runner made the server’s storage directory, and the TLS run’s certificate directory, with mkdtemp and never removed them: about 100 MB of WiredTiger left in the temporary directory by every gauge run. Both are removed when the runner exits.

The Python servers trim their log while they run

The Python store checkpointed when it closed, in a backup and in fsync, and at no other time. WiredTiger removes a log file only after a checkpoint, so a long-running secantusd-py or secantusd-py-pg kept every log file it had written and replayed them all after a crash. The Rust servers had the same gap and were fixed first.

Fixed

• A durable store checkpoints every 60 seconds. SECANTUS_CHECKPOINT_SECONDS sets the interval, as it does for the Rust servers, and Storage takes checkpoint_seconds; 0 means never.

• A store opened for tests (SECANTUS_TEST_FAST_STORAGE=1, or durable=False) does not checkpoint while open, as before.

A release no longer strands its tag when main moves during the test run

release-prepare pushed the release commit and the tag in one git push, which is not atomic. When another PR merged during the 30-minute test run, the tag was accepted and main was rejected, leaving a published tag on a commit main did not have. It happened on 0.7.0b1 and again on 0.7.0b2, and the version bump had to be landed by hand each time.

Fixed

• release-prepare pushes main first and the tag only once main carries the release commit. If main moved, it merges origin/main in and retries; the tag stays on the commit the tests ran against.

• A merge conflict, or any other refused push, stops before the tag is pushed, so nothing is published.

secantusd-rs starts with a 128 MB store, not a 2 GB one

WiredTiger allocates a log file in full when it creates one, and secantusd-rs asked for 2 GB files. A server that had taken no write had a 2 GB store.

Changed

• --log-file-max (and [storage] log_file_max) defaults to 128MB, as the embedded handle already did. Pass --log-file-max 2GB for the old behaviour. An existing store opens as before.

• Measured on a 16-vCPU droplet with the clients on two others, half inserts and half updates of 1 KiB documents: 22,701 operations a second at 2 GB and 22,422 at 128 MB with 16 writers, 16,010 and 16,102 with 64. The three trials of one size differ by more than that.

A partial TTL index no longer deletes what it does not hold, and three Rust MongoDB server gaps close

The Python MongoDB server expired documents outside a partial TTL index’s filter: three expired documents, one matching the filter, and all three were deleted. It now deletes the one, as mongod 8.2.11 does.

The rest is the Rust MongoDB server. An _id is unique by the collection’s default collation, a $out that fails leaves its target alone, and a time-series collection has its buckets.

Fixed

• Python server: a partial TTL index deleted documents its filter leaves out. It also expired documents by a compound index that collMod had given an expireAfterSeconds, and missed a date held in an array. collMod now refuses a TTL on a compound index or on _id, and a negative one (72).

• Rust server: _id was not unique by a collection’s default collation. With strength: 2, "A" went in beside "a". Insert, upsert, rename and a transaction all follow the rule now. A collection created with a collation before this release is not protected for the documents already in it.

• Rust server: a $out that failed destroyed its target. The target was dropped before the new documents went in, so a duplicate _id or a unique index the output broke left it empty or half-written. $out now builds beside the target and swaps it in. A duplicate _id in the output used to be dropped without an error; it is reported (11000).

• Rust server: $merge matched by a simple comparison, whatever the target’s collation, and so inserted a second spelling of an _id. It matches under the target’s collation and refuses to change _id (66).

• Rust server: a duplicate key on a clustered collection names no index and carries foundValue, as on mongod.

Added

• Rust server: a time-series collection’s buckets. system.buckets.<name> and system.views can be read and are listed. A scan returns measurements bucket by bucket with the time and meta fields first. validate, collStats and $collStats report buckets. The server still stores measurements, not buckets, and derives the buckets on request; tasks/backlog.md section 7.071 has what that leaves out.

• Rust server: convertToCapped, cloneCollectionAsCapped, compact and reIndex. They answered no such command.

Changed

• Rust server: validate reports repairMode, the collection’s uuid and each index’s spec, in mongod’s field order.

• Rust server: $out to a capped collection is refused (17152).

Capped collections on the Rust MongoDB server hold their bounds within one insert

A capped collection on the Rust MongoDB server could grow past its cap. The server never evicted a document that belonged to the batch being inserted, so insert_many of five documents into a max: 3 collection left all five, and forty 500-byte documents into a size: 4096 collection left all forty. mongod 8.2.11 leaves three and seven. The server now evicts from the batch like any other documents and always keeps the newest one, as mongod does. A sweep of 34 capped-collection results (sizes, payloads, one batch against single inserts, duplicates) went from 17 different from mongod to none.

Two changes make the same server faster. Aggregations decode fewer fields: the server read only the fields a $group needs, but only when $group was the first stage after a leading $match. It now works out which fields the leading stages read, through $unwind, $sort, $match, $skip and $limit, up to the first $group, $sortByCount or $count. And an insert opens its _id index cursor once per batch, not once per document.

Measured on a Mac over 10,000 five-field documents, old and new binary interleaved: the benchmark’s multi-stage pipeline went from 13.1 ms to 10.1 ms (mongod 8.2.11: 4.6 ms), $match then $count from 6.2 ms to 3.0 ms (1.4 ms), and a 10,000-document insert_many from 48 ms to 43 ms (38 ms).

Fixed

• A capped collection on the Rust MongoDB server is held to size and max after every insert, including for the documents of the batch that overflows it. The newest document is kept even when it alone is larger than size.

Changed

• aggregate on the Rust MongoDB server decodes only the fields read by a pipeline’s leading $unwind / $sort / $match / $skip / $limit stages and the $group, $sortByCount or $count they lead to.

• insert on the Rust MongoDB server reuses one _id index cursor per batch and no longer collects the batch’s keys.

Capped collections on the Rust MongoDB server follow mongod in six more places

A probe of capped-collection behaviour against mongod 8.2.11 (tools/probes/capped_collections.py) found the Rust MongoDB server different on 67 of 97 results. It now differs on none. Two of the causes lost or over-kept data. An upsert was never held to the cap, so a max: 3 collection that received upserts grew without limit. And max: 0, which means “no document limit” on mongod, was read as a limit of zero: every insert evicted everything except the newest document.

Not covered by that probe, and still different: convertToCapped is not implemented (the server answers CommandNotFound), and after a refused statement aborts a transaction the server words the follow-up error differently from mongod. The Python MongoDB server differs on 51 of the 97 and was not changed.

Fixed

• An upsert into a capped collection (update, findAndModify, a replacement, several in one command) evicts the oldest documents to stay within size and max, as an insert does.

• max of zero or less on a capped collection means no document limit. It is stored and reported as 2147483647, as on mongod. A collection created earlier with max: 0 is no longer emptied on insert.

• create refuses a capped size under 1 or over 1 PB and a max of 2^31 or more, with mongod’s messages. They were stored as given, and a max over 2^31 wrapped to a negative number.

• size and max are stored and reported as integers. pymongo sends size as a double, and listCollections echoed 1000.0.

• collMod with cappedSize or cappedMax re-bounds the collection, checks the same ranges, and answers InvalidOptions for a collection that is not capped. Both options were accepted and ignored.

• collStats reports max and maxSize for a capped collection.

• A write to a capped collection inside a multi-document transaction is refused and the transaction aborted: insert and update report write error 263, delete write error 20, findAndModify fails with 263. They used to run.

Change streams on the Rust MongoDB server, checked command by command against mongod

A comparison of 115 scripted change-stream scenarios with mongod 8.2.11 (tools/probes/change_stream_commands.py): what $changeStream accepts, every event’s shape, getMore, invalidation, resuming, the three scopes, pre-images and transactions. The previous release answered 111 of them differently; this one answers 5 differently.

Fixed

• A write in a transaction is reported without its transaction. Its events now lead with txnNumber and lsid, as mongod’s do.

• A resume token that is not found resumed anyway. mongod re-reads the token’s event and refuses (280, NonResumableChangeStreamError) when it is not the first one seen: a token from another collection, or a pipeline that filters its event out. The server started after the token’s position whatever it named.

• startAtOperationTime in the future delivered everything, and one far in the past delivered nothing when the oplog was long.

• A stream on the target of renameCollection ... dropTarget saw nothing; it sees the rename and is invalidated. The replaced target no longer gets a drop event of its own.

• A cluster-wide stream reported writes to admin, config and local.

• After a stream ended, its cursor could still be read, and a getMore with cursor id 0 found a cursor.

• An insert that created its collection produced no create event under showExpandedEvents. It does now, and so does an upsert.

• collMod’s modify event listed options the command did not give (validationLevel, validationAction); createIndexes and dropIndexes events left an index’s options (unique) out and had mongod’s field order wrong.

• A small event carried splitEvent: {fragment: 1, of: 1} under $changeStreamSplitLargeEvent. mongod adds the field only to an event it split.

Changed

• The $changeStream spec is checked. An unknown field (40415), a wrong type (14), a bad fullDocument or fullDocumentBeforeChange value (2), resumeAfter with startAfter (50865), a token whose _data is not a string (40647) or not hex (9), allChangesForCluster anywhere but admin with no collection (72), and a stream on admin, local, config or a system.* collection (73) are refused with mongod’s messages. A stage mongod knows but does not allow after $changeStream is “not permitted” (20); $changeStream after the first stage is 40602; $changeStreamSplitLargeEvent must be last (7182802) and needs a change stream (20).

• The cursor document’s field order is mongod’s: postBatchResumeToken before id, and invalidated: true on the batch that ends a stream. A 280 error leads with its errorLabels.

• Resume tokens are upper-case hex, as mongod’s are. A token from an earlier release still resumes.

Rust MongoDB server: collation that orders and matches as mongod does

The Rust server’s collation was a hand-written approximation. Measured against mongod 8.2.11 over 764 scenarios it gave a different answer in 462: range queries compared code points, $sort, $group and every expression ignored the collation, a collection’s default collation was never used, and a malformed collation document was accepted. It now compares through ICU4X, the Rust implementation of the algorithm and locale data mongod gets from ICU, and 8 scenarios differ.

Three bugs found beside it are fixed too. renameCollection dropped the collection’s options, so a validator stopped being enforced after a rename. A unique index with a collation compared bytes, so it stored "a" and "A" together at strength 2. And $group’s $min / $max kept the first of two equal values where mongod keeps the last.

Added

• Collation by ICU4X: every strength, caseLevel, caseFirst, numericOrdering, alternate, maxVariable and backwards, and the 109 locales mongod accepts with their @collation= variants (five variants are refused by name: search, searchjl, big5han, gb2312han, phonetic).

• A collation applies to aggregation expressions ($eq, $cmp, $in, the set operators, $max / $min, $sortArray, $switch), to $group, $sortByCount, $bucket and $addToSet keys, to the $sort stage, $lookup, $graphLookup, window and $fill partitions, and to the update operators $pull, $pullAll, $addToSet, $push with $sort, $min, $max and arrayFilters.

• A collection’s default collation: stored in full, shown by listCollections and on the _id index, inherited by its indexes and by every command that names no collation. A view has its own.

• $alwaysTrue and $alwaysFalse in a query.

Fixed

• renameCollection keeps the collection’s options and UUID.

• A unique index with a collation enforces uniqueness by that collation, on insert, update and build, and its partial filter is read under it.

• A malformed collation is refused on find, aggregate, count, distinct, findAndModify, update, delete, create and createIndexes, with mongod’s code and message.

• {locale: "simple"} means no collation; it was read as one.

• createIndexes stores a collation spelled out as mongod does; two indexes on one key may differ by collation, and dropIndexes by that key answers 181. An _id index with a collation other than the collection’s is refused. collMod refuses an unknown index field.

• $group $min / $max keep the later of two equal values; $maxN lists the later of equal values first.

• A sort inside $top / $bottom, $setWindowFields and $fill reads an array by its smallest (or largest) element, as every other sort does.

• $fill with locf writes null where there is nothing to carry forward, and emits partitions in key order.

• $setWindowFields refuses an array partition key (14); $group refuses _id: {field: 1} (17390).

Changed

• An index created with a collation holds collation sort keys and is marked so in the catalog. A query with a collation still scans the collection. An index with a collation built by an earlier version keeps comparing by value until it is dropped and rebuilt.

Rust MongoDB server: the cursor commands checked against mongod

We compared find’s cursor options, getMore, killCursors, tailable cursors, count and distinct on the Rust MongoDB server with mongod 8.2.11: 85 scripts of commands. 49 differed. 5 do now.

Five of the differences were wrong results. find with singleBatch returned every matching document whatever batchSize said. A tailable cursor applied its filter and projection to the first batch only, so every document inserted later came back whole. min and max compared documents where mongod compares index keys, and left out a document whose array had one element inside the bounds, or which lacked the field. distinct kept one value per BSON type, so 1, 1.0 and a Decimal128 1 were three values, and it did not look through an array of documents. count ignored a negative limit.

Fixed

• find: singleBatch returns one batch of batchSize; batchSize: 0 opens a cursor even when nothing matches (so does an aggregate, over a missing collection too); a negative batchSize, an unknown field, a non-boolean cursor flag, awaitData without tailable and a malformed $natural hint are refused; an empty hint is no hint; showRecordId keeps the projection.

• min / max bound the hinted index’s keys, and are refused with mongod’s codes when no index is hinted (291), when the index does not fit (51174) and when min is not below max (51175).

• let: a name must start with a lower-case letter, and a value may not read a field.

• Tailable cursors: the filter and projection apply to every batch; a tailable find on an empty capped collection or a missing one returns a closed cursor; a sort other than {$natural: 1} and singleBatch are refused; no postBatchResumeToken outside a change stream.

• getMore: the cursor id is an int64; a getMore for another namespace is 13 and names both; maxTimeMS is refused on a cursor that is not awaitData; a negative batchSize and an unknown field are refused. None of these consumes the cursor.

• A cursor whose collection is dropped or renamed, or whose database is dropped, answers its next getMore 175 with mongod’s reason. It used to answer CursorNotFound, or go on returning documents after a dropDatabase.

• killCursors: ids are int64, a null is passed over, an unknown field is refused.

• count: a negative limit counts as its size; a negative skip, a fractional limit and an unknown field are refused.

• distinct: values are distinct by BSON equality and come back in BSON sort order; a dotted key reads through an array of documents; an unknown field and a key with a NUL are refused.

• aggregate, listIndexes, listCollections: an unknown field, an unknown cursor option and a negative batchSize are refused.

• A collection name that is not a string, or is empty, is 73 with mongod’s message on every command that takes one.

• The parser’s errors 40414 and 40415 are named IDLFailedToParse and IDLUnknownField everywhere.

The Rust MongoDB server holds DDL and admin commands to mongod’s rules

secantusd-rs now refuses the database names, create options and command arguments mongod 8.2.11 refuses, and reports what mongod reports from listCollections, dbStats, collStats and listDatabases. A comparison of 809 replies went from 268 different (of the first 670) to 2.

Several of the differences could leave a test passing on something mongod would have stopped: any database name was accepted, dropDatabase dropped the database whatever its argument, and collMod with dryRun: true applied the change.

Fixed

• Rust MongoDB server: an invalid database name is refused. a.b, a b, the empty name and names with /, \ or " were accepted, and a write created the database. A name with $ is refused for writes, as on mongod.

• dropDatabase takes 1. 0, 2.5, a string or null used to drop the database; they are refused (5255100, 14, 40414).

• renameCollection checks both namespaces first: an empty or dotless namespace, a rename onto itself (20), onto system.x (20) or onto a name past 255 bytes.

• create parses its options before creating anything: validationLevel and validationAction must be strings, changeStreamPreAndPostImages a document with a boolean enabled, expireAfterSeconds a number, storageEngine a known engine, idIndex a {_id: 1} key; autoIndexId is unknown; a namespace past 255 bytes is refused. A view keeps only what defines it.

• collMod: dryRun: true no longer applies the change; a null option is an option not given; expireAfterSeconds is refused on a collection that is not clustered; an index named by both name and keyPattern is refused; the reply ends with ok.

• Converting an index to unique takes mongod’s two steps. unique: true is refused until prepareUnique: true has been set; the replies carry prepareUnique_old / prepareUnique_new and unique_new; unique: false is refused.

• Dropping system.views drops every view. It answered ok and left them in place.

• listCollections: no ns on idIndex, no uuid on a view, options in mongod’s order, defaults left out; with nameOnly the filter sees the name and type only; an unknown operator in the filter is refused.

Changed

• dbStats, collStats and listDatabases have mongod’s fields, order and number types: views, avgObjSize, totalSize and scaleFactor on dbStats; numOrphanDocs, freeStorageSize, indexBuilds, totalSize and scaleFactor on collStats; totalSizeMb on listDatabases. scale and freeStorage are honoured and checked. A database is empty when it holds no collection.

• dataSize is a command.

• An unknown field on drop, dropDatabase, dbStats, collStats, listDatabases or renameCollection is refused (40415).

• listDatabases outside admin is refused (13).

Rust MongoDB server: handshake, diagnostic and session commands checked against mongod

The commands a driver or a tool sends about the server and the connection (hello, buildInfo, serverStatus, getParameter, dataSize, $collStats, the session commands and the fields every command takes) were sent to the Rust MongoDB server and to mongod 8.2.11, 1,141 of them, and each reply compared. 492 differed; 160 do now (tasks/backlog.md 7.082).

Fixed

• dataSize with min and max sizes that key range. It sized the whole collection for any range.

• The storageStats of a $collStats stage are the collection’s sizes. They were 0.

• serverStatus reports the process id, the uptime, the operations run (opcounters) and the bytes read and written. Each was a constant 0.

• A wrong-typed $readPreference, $clusterTime, apiVersion, apiStrict, readConcern.afterClusterTime or atClusterTime is refused. Each was read as absent. A read at an atClusterTime more than five minutes old is SnapshotTooOld; it was answered with the current documents.

• A command given a field it does not have is refused where mongod refuses it ({ping: 1, bogus: 1}).

• hello answers isWritablePrimary and the two older names answer ismaster, in mongod’s key order, with an int32 connectionId. The client metadata is checked, and refused on a second hello.

• Which commands are in API Version 1, and which run only against admin, comes from mongod’s own command table. collStats, createUser and eight more were taken under apiStrict: true.

• whatsmyuri gives the client’s address. killOp takes any whole number and does not say what it found. getParameter refuses a request that names no parameter it has, and answers showDetails.

• profile, validate, endSessions, killSessions, killAllSessions, usersInfo, rolesInfo, saslStart, saslContinue, $collStats, $indexStats, $currentOp, $listSessions and {aggregate: 1} check their arguments.

• $listSessions and $listLocalSessions list the sessions commands have come in. They answered a row describing an operation.

Added

• listCommands, logout, getDefaultRWConcern, replSetGetConfig, shardingState, fsyncUnlock, and the lower-case collstats and datasize.

• tools/probes/diagnostic_commands.py.

Changed

• A filter document on profile is refused. It was taken and ignored.

• killOp no longer takes a string of digits, and writeConcern no longer takes wtimeoutMS: mongod takes neither.

• getParameter no longer reports version, which is not a parameter.

The Rust MongoDB server’s explain follows mongod’s planner

explain on secantusd-rs now reports the plan mongod 8.2.11 reports for the same command: the index bounds, what the fetch rechecks, whether an index serves the sort or covers the projection, and the stages count, distinct, update, delete and findAndModify have of their own. A comparison of 786 explain replies went from 771 different to 72.

Two defects outside explain turned up and are fixed. An operator the query language does not have was accepted over an empty or missing collection by every command but find. And count or delete by _id on a collection that was never written could answer an internal storage error.

Changed

• Rust MongoDB server: explain is derived from mongod’s planning rules. indexBounds on every index scan, in mongod’s notation and for every operator that bounds an index; EXPRESS_IXSCAN and IDHACK for a point lookup; PROJECTION_COVERED; SORT_MERGE; COUNT_SCAN and RECORD_STORE_FAST_COUNT; DISTINCT_SCAN; UPDATE, DELETE, BATCHED_DELETE, EXPRESS_UPDATE and EXPRESS_DELETE; EOF for a missing collection or a filter that matches nothing; a view explained through its pipeline. The reply has mongod’s envelope (explainVersion, the planner flags, command, serverParameters).

• executionStats counts what the reported plan examines, stage by stage: a LIMIT stops its input, a multikey scan drops repeated documents, and a scan over several intervals examines the key that ends each one.

• explain checks its arguments as mongod does: the verbosity, unknown fields, writeConcern, commands that cannot be explained, a write batch of more or fewer than one statement. With no verbosity it runs at allPlansExecution.

• explain describes the plan mongod runs. This server’s storage layer may read the same documents another way; tasks/backlog.md 7.075 lists where. rejectedPlans is always empty.

Fixed

• Rust MongoDB server: an unknown query operator is refused over an empty or missing collection. {a: {$nope: 1}} was accepted by count, distinct, aggregate, update, delete and findAndModify when there was no document to match it against, and answered as if the filter were valid. find refused it as unsupported query operator. All now answer unknown operator: $nope (2), as mongod does.

• Rust MongoDB server: count and delete by _id on a collection that was never written answered WiredTiger error: No such file or directory. They find nothing, as mongod does.

• An explain reply no longer carries atClusterTime under readConcern: {level: "snapshot"}.

The Rust MongoDB server accepts sort: {$natural: ±1} on find again

Since the build of 2026-10-05, every find().sort("$natural", ...) against secantusd-rs failed with 16410 FieldPath field names may not start with '$'. A sort validator shared between find and the aggregation $sort stage applied the stage’s rule to both, and only the stage reads $natural as a field path. The driver-gauge refresh caught it: pymongo’s test_to_list_tailable, which reads the newest oplog entry this way, failed on the Rust server in both the sync and async suites.

Measured against mongod 8.2.11 with the same commands on both servers: 38 cases, no differences after the fix.

Fixed

• find with sort: {$natural: 1} or {$natural: -1} returns documents in storage order, forwards or backwards, and composes with filter, skip and limit. explain reports it as a COLLSCAN with a direction.

• A $natural sort value other than exactly 1 or -1, or $natural beside another key, answers mongod’s 2 $natural sort cannot be set to a value other than -1 or 1.

• A hint alongside a $natural sort follows mongod: the same-direction $natural hint is accepted, the opposite direction and any index hint are refused with mongod’s messages.

• findAndModify with sort: {$natural: -1} used to act on the FIRST document. It now acts on the last, as mongod does.

Rust MongoDB server: documents come back in the order mongod returns them

A query with no sort has no promised order, and a client still sees one: the order of the index the server read. That order also decides which document a limit: 1 delete removes and which one an update without multi changes. The Rust MongoDB server read through its own index router and returned its own order, so the same write could change a different document from mongod’s.

It now takes the order from the plan mongod runs. The store says which documents match, and the planner behind explain says in what order mongod reads them.

Found by sending 6,531 commands to the server and to mongod 8.2.11 and comparing the _ids each read returned, in order, and what each write of one document left behind. 1,189 differed; 97 do now (tasks/backlog.md 7.080).

Fixed

• {a: {$gt: 1, $lt: 9}} over an index on a did not return {a: [1, 9]}. One element is over 1 and another under 9, and the scan read only the keys between the two.

• {"p.q": null} over an index on p.q did not return a document whose array p has an element without q.

• A sort by p.q ranked that document by its other elements. mongod ranks it as null.

• find, the front of an aggregate pipeline, distinct and findAndModify return documents in the order mongod’s plan reads them: an index’s order for $ne, $nin, $exists, $type, $not, $mod and a regex, _id order under a hinted _id index, and equal keys last first on a backward scan.

• delete with limit: 1, update without multi and findAndModify change the first document in that order. findAndModify and update now read their hint.

Changed

• A read whose order the plan has to work out costs more than it did. Measured against the build before, on 20,000 documents: a read through the index the router also picks, a collection scan and a read by _id cost the same; {a: {$ne: 5}} over an index on a costs about three times as much, since mongod reads it in the index’s order and the router scans the collection for it. A sort is unchanged unless documents compare equal.

Rust MongoDB server: geo queries checked against mongod

The Rust MongoDB server tested every geo shape as if coordinates were a plane, and treated $near as a filter. mongod tests $geometry on the sphere, where an edge is the great-circle arc between its ends, and returns a $near nearest first from a geo index it requires.

Found by sending 1,278 geo commands to the server and to mongod 8.2.11 and comparing each reply. 726 differed; 137 do now, about 100 of them the order of documents under a geo index (tasks/backlog.md 7.081).

Fixed

• $geoWithin and $geoIntersects with $geometry are tested on the sphere. [1, 1] is not on the line from [0, 0] to [2, 2], and a polygon across the antimeridian is the small one.

• $box, $polygon and $center return points only. They returned a line or a polygon that lay inside them.

• A value holding several locations ([[1, 1], [30, 30]]) matches through any of them, and is found through a 2d index.

• $near and $nearSphere return documents nearest first, measure a line or a polygon by its nearest point, and need a geo index on the field that the form of the query can read. They are refused inside $or, $nor, $not and $match.

• $geoNear takes its units from near: metres for GeoJSON, radians for a legacy point with spherical: true. It no longer needs distanceField, refuses limit and num, and must be the first stage.

• A 2dsphere index refuses a value that is not a geometry, and a 2d index a value that is not a legacy pair: on insert, on update, and when the index is built over such a document.

• $within is read as $geoWithin.

• About ninety argument errors have mongod’s code and text.

Changed

• A $geoNear with a maxDistance reads every document of the collection. It used to fetch candidates through the index.

Index management on the Rust MongoDB server follows mongod

dropIndexes given the key {_id: 1} dropped the _id index on the Rust MongoDB server. mongod refuses, by name, by key or in a list, and so does the Rust server now.

That came out of a probe of index management against mongod 8.2.11 (tools/probes/index_admin.py): the Rust server differed on 68 of 87 results and now differs on 9. Seven of the nine are text and hashed indexes, which it does not build, and what follows from their absence; the other two are below.

Fixed

• dropIndexes never drops the _id index. It accepts a list of names, checks every name before dropping any, reports a missing collection, and answers non-_id indexes dropped for collection for "*".

• createIndexes checks every spec before it builds anything or creates the collection. Refused with mongod’s errors: a key value of 0 or a boolean, an empty or $-prefixed field name, more than 32 fields, an unknown option, an empty or * name, index version 3, partialFilterExpression with sparse, a negative, oversized or compound TTL, any option on the _id index, and an empty indexes list. All were accepted.

• unique: 1 builds a unique index. A number was read as “not unique”.

• The createIndexes reply has mongod’s fields: commitQuorum, the _id index counted for a collection the build creates, index already exists when some of the specs existed, and no createdCollectionAutomatically when all did.

• collMod with index: {hidden: ...} hides or shows the index and reports hidden_old / hidden_new. It answered ok and did nothing. Hiding the _id index is refused.

• A hidden index is passed over by the query planner, by sorts and by hints (on reads and writes), whether hidden by collMod or created hidden. It is still maintained, and a hidden unique index still enforces uniqueness.

• $geoWithin with $box, $polygon or a $geometry polygon includes a point that lies on the boundary. It was left out, with and without an index; mongod includes it in all 15 combinations probed.

• collMod of a TTL reports both values as int64 and no old value for an index that had none, and names the missing index or option in its errors.

• listIndexes lists _id_ first.

Rust MongoDB server: insert, delete, count and distinct checked against mongod

The Rust MongoDB server stored two documents whose _id mongod holds to be one: after {_id: {a: 1}} it accepted {_id: {a: 1.0}}. It also lost a document when a $merge with whenMatched: "replace" was refused by a unique index, because it deleted the old document before inserting the new one. Both are fixed. They were found by sending 905 insert, delete, count and distinct commands to the server and to mongod 8.2.11 and comparing each reply and the documents left behind; 162 replies differed and 32 still do (tasks/backlog.md 7.078).

No stored key changes. A document _id takes a second claim, by value, beside its row in the _id index. The first time this version opens a store it gives every stored document _id its claim, so a duplicate by value of an older document is refused too. Two documents an earlier version already stored as different both stay.

Fixed

• A document _id is unique by value: {a: 1}, {a: 1.0} and {a: NumberLong(1)} are one _id.

• $merge with whenMatched: "replace" replaces the document in place. A replacement that fails leaves the document as it was.

• A delete statement with no limit is refused. It removed every match.

• delete checks every statement before it runs any.

• A limit: 1 delete removes the first document its hint reads.

• An $in over an index returns documents in the index’s order, for find and for delete.

• insert refuses a $-prefixed field at any depth of an _id document, a document nested past 180 levels, and a batch holding an element that is not a document; it drops a null element.

• A duplicate-key message prints dates, documents, binary and timestamps as mongod does. They were printed as Rust debug text.

• A document that repeats both an _id and a unique index key is reported for the _id.

• count reads limit and skip as mongod does: a decimal skip is rounded, a limit that is not a whole number is refused.

• insert, delete, count and distinct refuse fields they do not take, and count and distinct refuse a writeConcern, as do 38 other commands measured.

• A readConcern that is not a document is refused.

• A write concern’s j and fsync take any number, are refused together, and wtimeout is accepted whatever its type.

• An apiVersion other than “1” is refused in mongod’s words.

• $expr with an unknown operator answers 168, and $text with no text index answers 27, on delete, count and distinct.

Rust MongoDB server: $out and $merge checked against mongod

$merge on the Rust MongoDB server stored a different document from mongod’s. With whenMatched: "merge", the default, it merged sub-documents field by field: {o: {p: 1}} merged into {o: {q: 2}} gave {o: {q: 2, p: 1}}. mongod sets each top-level field whole and stores {o: {p: 1}}. The stage is now written with the update command, as mongod writes it, and stores what mongod stores.

Found by sending 376 $out and $merge pipelines to the server and to mongod 8.2.11 and comparing each reply and what the source and target held afterwards. 191 of the first 327 differed; 4 of 376 do now (tasks/backlog.md 7.079).

Fixed

• whenMatched: "merge" sets each top-level field whole. A field name with a dot in it is a path, and new fields are stored in the order $set stores them.

• $out and $merge inside $facet, $lookup or $unionWith are refused. They ran, and wrote to the target.

• A write the target refuses no longer stops the others: every document of the batch is attempted and the first failure is reported.

• The target’s validator is asked about each document as it is written. One failing document used to stop all of them from being written.

• A whenMatched pipeline no longer inserts a document the validator refuses, and sees the stage’s let variables.

• whenMatched: "delete" is refused; mongod has no such mode. So are the pairs of modes mongod does not have.

• The on fields need a unique index that is not partial and has the pipeline’s collation.

• A target that does not exist is not created by a pipeline that writes nothing.

• $out refuses a target name no collection can have, a system collection, and a document an insert would refuse, and leaves the target as it was.

• A field set to $$variable.path that leads nowhere is removed. It was set to null, in aggregate and in a pipeline update.

• A replacement cannot change an _id to one a collation holds equal.

• {$set: {$c: 2}} over a document whose $c is already 2 is accepted.

• listIndexes shows a clustered index’s expireAfterSeconds.

• A duplicate key reported for a whole command carries keyPattern and keyValue.

The Rust servers trim their log while they run

Neither Rust server checkpointed between opening its store and closing it, and WiredTiger removes a log file only after a checkpoint. A server under writes kept every log file it had ever written, and after a crash replayed all of them.

Fixed

• secantusd-rs and secantusd-pg checkpoint every 60 seconds (SECANTUS_CHECKPOINT_SECONDS, and --checkpoint-seconds on secantusd-rs). Measured on secantusd-pg with eight clients updating for 13 minutes: the store was 2.1 GB before this change and 69 MB after, and a restart after a SIGKILL took 36.8 s before and 1.4 s after. Throughput was the same, 13,815 statements a second before and 13,864 after.

• A store opened for tests (SECANTUS_TEST_FAST_STORAGE=1, or durable=false) does not checkpoint while open, as before.

Rust PostgreSQL server: built-in functions, checked one by one

A new probe calls every immutable or stable function in PostgreSQL 15’s pg_proc on PostgreSQL and on the Rust PostgreSQL server, with one sample value for each argument type. Of the 1,935 functions it can call, 619 answered the same before this change and 1,439 answer the same after it. The corpora written by hand had not reached most of what it found.

Fixed

• sum(interval) answered zero and avg(interval) answered NULL, over a table and over literals. Both add the months, days and time apart, as PostgreSQL does.

• substr, substring, overlay, md5, btrim, ltrim, rtrim and trim over a bytea ran the text function over a rendering of the bytes, or were refused. They are the bytea functions now.

• bytea LIKE bytea answered NULL. It matches bytes, with % and _ as the wildcards.

• mod(7.5, 2) answered 1 and div(9.9, 0.3) was a division by zero: both truncated their arguments to integers first.

• smallint arithmetic answered integer, and never overflowed. It answers smallint, and a result past its range is 22003. A prefix - or abs of a real or a smallint keeps the type too.

• to_tsvector over json or jsonb indexed the document’s text, keys and numbers included. It indexes the string values only.

• array_agg over an array column was typed varchar; it is the array’s own type.

• diagonal(box) and a box cast to lseg ran from the lower left corner to the upper right. PostgreSQL’s runs the other way.

• A regular-expression function took any letter as an option. One that is no option is 22023 invalid regular expression option, for regexp_like, regexp_count, regexp_instr, regexp_substr, regexp_replace, regexp_matches, regexp_split_to_array and regexp_split_to_table.

• pg_size_bytes of a bad size is 22023 and ts_rank with a short weight array is 2202E, where both were other codes. A bad range bound flag, is_normalized with an unknown form and json_array_elements of a non-array carry PostgreSQL’s text.

Added

• regexp_match.

• regexp_replace with a start position, and with the number of the match to replace.

• OVERLAPS, over dates, times and timestamps, with an end or a length.

• bit_xor; bit_and and bit_or over bit strings; string_agg over bytea.

• date + time.

• get_bit, set_bit and bit_count over bytea, and length(bytea, encoding).

• num_nulls and num_nonnulls, parse_ident, convert_to and convert (UTF8, LATIN1 and SQL_ASCII), multirange(range), trunc(macaddr), trunc(macaddr8) and macaddr8_set7bit.

• A built-in called by the name of the function behind an operator: int4pl(a, b) is a + b, texteq, int48lt, textlike and about 740 more, from PostgreSQL 15’s pg_operator. Also the *larger / *smaller functions behind max / min, the numeric and date *cmp functions, and names such as textlen, dexp and numeric_sqrt.

• The embedded Python handle, _secantus_server.PgServer, takes tls_cert_file, tls_key_file, tls_ca_file, tls_client_cert, tls_require, refuse_unknown_users, hba_file and ident_file, and has reload(). It is a tool for this repository’s tests; the wheel on PyPI does not carry it.

• ./inv validate-psycopg-rust [--tls] runs psycopg’s suite against the Rust PostgreSQL server, and with --tls again over TLS. A new workflow runs both on Linux each week and on a pull request that touches the server’s TLS code. It fails when a run stops short, or when a test fails over TLS that passes in the clear.

Changed

• A statement that passed an unknown letter as a regular-expression option ran before and is refused now, as on PostgreSQL.

• smallint arithmetic that overflowed gave a number before and is an error now.

tools/probes/pg_builtin_coverage.py is the probe. tools/probes/pg_corpora/builtin_coverage.sql holds 153 statements over what changed, with PostgreSQL 15.19’s answers recorded beside it, and tests/test_rust_pg_builtin_coverage.py holds the server to them in CI. What the probe still reports is in tasks/backlog.md.

Rust PostgreSQL server: pg_hba.conf

The Rust PostgreSQL server applied one login rule to every connection, set by command-line options. --hba-file takes a pg_hba.conf in their place, read as PostgreSQL 15 reads it.

Added

• --hba-file (PgBuilder::hba_file, ServerOptions::hba_file). The first line whose type, database, user and address match a connection decides it, and a connection no line matches is refused with PostgreSQL’s “no pg_hba.conf entry”. Read: host, hostssl, hostnossl; all, sameuser, samerole, names and lists for the database; all, names, +role and lists for the user; all, samehost, and an address with a prefix length or a mask; the trust, reject, scram-sha-256, md5, password and cert methods; clientcert=, clientname= and map=.

• --ident-file (PgBuilder::ident_file), the pg_ident.conf a map= is looked up in.

• The password method: the password in the clear, checked against a stored SCRAM verifier or md5 hash.

• The file is read again on SIGHUP, select pg_reload_conf() and PgServer::reload.

Changed

• With a pg_hba.conf, a role with no password cannot pass a scram-sha-256 line and a user name with no role is refused, as on PostgreSQL. Without one, both still connect, as before.

• With a pg_hba.conf, a login is proved before the database is looked up: a wrong password for a database that is not there is a wrong password.

• PgServer::reload_tls and TlsReloader, added earlier in this release cycle and not yet released, are PgServer::reload and Reloader.

A line the server cannot apply stops it from starting, with the line’s number: a method that needs an outside service (peer, ident, gss, sspi, ldap, radius, pam, bsd), a host name for the address, samenet, and @file inclusion.

Run against PostgreSQL 15.19 with the same two files: 118 of 118 scenarios answer the same.

Rust PostgreSQL server: the login comparisons run in CI

The login rules of the Rust PostgreSQL server were measured against PostgreSQL 15 by scripts that lived outside the repository, so nothing would have failed had a later change moved them.

Added

• tools/probes/pg_login_options.py, pg_login_tls_files.py and pg_login_hba.py: 817 logins put to a throwaway PostgreSQL 15 cluster and to secantusd-pg, each printing how many answer the same.

• tests/test_rust_pg_login_differential.py runs the three and requires every scenario to match, and each probe to hold the number of scenarios it is known to. It needs SECANTUS_PG15_BIN; CI’s pg-oracle lane installs PostgreSQL 15 and sets it.

• SECANTUS_GAUGE_TLS=1 runs psycopg’s suite with the Rust server started with a certificate and libpq told sslmode=verify-full. Measured on 2026-10-12: 5,544 pass and none fail of the 5,731 that ran, the same counts as the plain run.

Rust PostgreSQL server: --refuse-unknown-users

The Rust PostgreSQL server lets a user name it has no role for connect without a password. --refuse-unknown-users stops that. The default is unchanged, because test fixtures connect as roles nobody created.

Added

• secantusd-pg --refuse-unknown-users, PgBuilder::refuse_unknown_users and ServerOptions with bind_with for a caller that binds the server itself. A user there is no role for is asked for a password and the login fails as a wrong password does, which is what PostgreSQL answers under a scram-sha-256 line. Under --tls-client-cert cert the certificate is checked and the role is then reported missing. A role with no password still connects without one.

Run against PostgreSQL 15.19 in the five login configurations of the TLS comparison, with 65 unknown-user scenarios added: 219 of 219 answer the same with the flag, and 171 without it.

Rust PostgreSQL server: public off the search_path hides its tables, and grouping sets take expressions in any clause

A table in public is stored under its bare name, so a bare name went on finding it after public was taken off the search_path. PostgreSQL answers 42P01 relation "t" does not exist there; this server read the table, and an INSERT, UPDATE or DELETE wrote it. It is 42P01 now.

Under ROLLUP, CUBE and GROUPING SETS, a HAVING term or a select-list expression computed over the keys was refused when it was more than a key compared with a constant. Those queries run now.

Fixed

• A relation in public named without its schema is not found while public is off the search_path: a SELECT, INSERT, UPDATE, DELETE, TRUNCATE, COPY, ALTER TABLE, CREATE INDEX, COMMENT or DROP of it is 42P01, as PostgreSQL 15 has it. A name written with its schema (public.t) works, and a view created over such a table keeps reading it. Still found, and listed in the backlog: a name looked up from a string (nextval('s'), 't'::regclass) and DROP ... IF EXISTS.

• An unterminated string, quoted identifier, dollar-quoted string or comment reports where it opens (select 'abc is position 8). The error had no position.

Added

• Under ROLLUP / CUBE / GROUPING SETS: a HAVING term over a key (having g || 'x' = 'ax', having coalesce(g, z) = 'a'), an expression over an expression key (select upper(g) || 'x' ... group by rollup (upper(g))), GROUPING() inside an expression (case when grouping(g) = 1 then 'total' else g end), and ORDER BY over an expression of the keys or an aggregate. Each was 0A000.

• GROUP BY (a, b) groups by a and b. It was a 42803 naming a.

Tests

• tools/probes/pg_corpora/search_path_public.sql (37 statements) and grouping_set_exprs.sql (45, was 25) against PostgreSQL 15.19. What still differs is listed at the end of each file and in tasks/backlog.md.

Rust PostgreSQL server: client certificates, channel binding and pg_stat_ssl

The Rust PostgreSQL server could offer TLS, and that was all it could do with it. A client that required SCRAM channel binding gave up, a client certificate was never asked for, and nothing could make the server refuse a connection in the clear. It has no pg_hba.conf, so three options now stand for the lines of one that matter here.

Each behaviour was measured against PostgreSQL 15.19 in the matching pg_hba.conf configuration: 154 scenarios through psycopg 3.3.4 and a hand-written SCRAM client, of which 150 answer the same. The four that differ are one known case: this server trusts a user it has never heard of.

Added

• SCRAM channel binding. A password login over TLS is offered SCRAM-SHA-256-PLUS (tls-server-end-point), so channel_binding=require connects. A client that says it can bind but was not offered it is refused when the server did offer it, with PostgreSQL’s SQLSTATE, message and detail.

• --tls-ca-file (TlsConfig::ca_file): a TLS client is asked for a certificate, and one it sends must be signed by a CA in that file.

• --tls-client-cert verify-ca | verify-full | cert (TlsConfig::client_cert): a certificate must be sent; its common name must also be the user name; or that, with no password asked for.

• --tls-require (TlsConfig::require_tls): a connection that is not TLS is refused with PostgreSQL’s “no pg_hba.conf entry … no encryption”.

• pg_stat_ssl, which answered 42P01: a row for each connection with the protocol version, the cipher under OpenSSL’s name, and the client certificate’s subject, serial number and issuer.

• PgBuilder::tls_config, which takes a whole TlsConfig.

• pg_backend_pid() inside an expression (where pid = pg_backend_pid()). It worked only as a select-list item on its own.

Changed

• Selecting a SASL mechanism the server did not offer is 08P01 “client selected an invalid SASL authentication mechanism”, as PostgreSQL has it. It was 28000 with other words.

• A SCRAM client-final message whose c= attribute does not repeat the header of the first message is refused. It was not looked at.

• TlsConfig is #[non_exhaustive]: build it with TlsConfig::new.

Rust PostgreSQL server: revocation lists, user name maps and a TLS reload

The Rust PostgreSQL server’s TLS could not refuse a revoked client certificate, map a certificate to a role other than the one it names, or pick up a renewed certificate without a restart. It can now.

Added

• --tls-crl-file (TlsConfig::crl_file), PostgreSQL’s ssl_crl_file. A client certificate a revocation list names fails the handshake. As OpenSSL requires of PostgreSQL, every certificate in the chain needs a list from its issuer that is in date. A list with no extensions, which openssl ca -gencrl writes by default, is read.

• --tls-ident-file and --tls-user-map (TlsConfig::user_map), a pg_ident.conf and map=. Under --tls-client-cert cert a certificate logs in as the roles the named map gives it, by exact name or by a regular expression with \1.

• --tls-client-name DN (TlsConfig::client_name), clientname=DN: the certificate’s whole subject, CN=alice,O=Example, is what is compared with the user name or handed to the map.

• The TLS files are read again on SIGHUP, on select pg_reload_conf() from a superuser, and on PgServer::reload_tls / RunningPgServer::tls_reloader. Files that cannot be used leave the old ones serving.

Changed

• A server certificate whose signature is RSA-PSS is offered SCRAM channel binding, with the hash its parameters name. It was offered none.

• A server certificate signed with Ed25519 is offered channel binding too, and a login that takes it up fails with “could not find digest for NID UNDEF”, as on PostgreSQL. libpq takes it up by default, so a client that connected to such a server before now needs channel_binding=disable.

• SIGHUP no longer stops a server that was started with TLS. It reloads.

Run against PostgreSQL 15.19: 480 of 480 scenarios answer the same, and the 219 of the earlier login comparison still do.

The Rust PostgreSQL server offers TLS

secantusd-pg answered every TLS request with “not supported”, so a client set to sslmode=require could not connect at all. Given a certificate and its key, it now completes the handshake.

Run against PostgreSQL 15.19 with ssl = on and the same certificate, psycopg (libpq 18) gets the same answer from both servers for sslmode=disable, prefer, require, verify-ca and verify-full, for a certificate signed by another CA, for a TLS 1.2 ceiling, for a SCRAM-SHA-256 login over TLS, and for SHOW ssl, RESET ALL and SET ssl.

Added

• secantusd-pg --tls-cert-file PATH --tls-key-file PATH: a PEM certificate chain and its private key (PKCS#8, PKCS#1 or SEC1, no passphrase). A client that asks for TLS gets it; one that does not is still served in the clear.

• PgBuilder::tls(cert_file, key_file) and secantus_pg::bind_tls for the embedded server, and secantus_pg::TlsConfig.

• SHOW ssl reads on when the server has a certificate.

• A TLS round trip (sslmode=verify-full) in the release workflow’s smoke test, so the published binary’s TLS is exercised before it ships.

Changed

• An option secantusd-pg does not know (--tls-cert, any other --name) is now an error. It used to be taken as the storage path, and the server started over a directory of that name.

• A server that cannot use its certificate does not start: a missing file, a key that does not match, or one flag of the pair without the other.

Not included

• SCRAM channel binding (SCRAM-SHA-256-PLUS): a client with channel_binding=require and a password is refused by its own library, where PostgreSQL connects.

• Client certificates (clientcert, cert authentication), pg_stat_ssl, and a way to refuse plaintext connections.

Rust PostgreSQL server: an unprepared statement reuses a plan

A statement sent with its values written into the text (select v from t where k = 5, then ... k = 6) was a new text each time, so the server parsed and planned every one from the start. It now reads such a statement as the same statement with the values taken out, and after the second of a shape it reuses the plan with the new values put in, as it already did for a prepared statement. On the development machine a primary-key read with a different literal each time went from 118 to 43 microseconds, against 42 for PostgreSQL 15 on the same machine and 42 for the same text repeated.

Changed

• A plan is kept for the shape of an unprepared SELECT, INSERT, UPDATE or DELETE whose integer and plain string literals stand where a prepared statement’s parameters may: compared with a column in the WHERE, a value of an INSERT, the right-hand side of a SET. It is learned the way a prepared statement’s is, by planning the text twice with stand-in literals and keeping the plan only if it follows them, so a literal the planner reads (a LIMIT, a cast, arithmetic, a value for a varchar(n) or a date column) is planned every time as before.

• A plan template is learned the second time a statement is planned, not the first. Learning plans the statement twice more, which a statement sent once never earned back.

• A single command is no longer parsed once to split it from its neighbours and again to plan it. Its syntax error is unchanged: the position counts from the start of what the client sent.

Fixed

• A string literal compared with a bytea column (x = 'abc', x = '\x616263') matched no row. It is read as bytea now, in either input format.

• Under GROUPING SETS, ROLLUP or CUBE, a select-list expression with no aggregate in it (a constant, g || 'x') was refused. So was an aggregate over such a query in FROM (select count(*) from (select ... group by cube (g, z)) q), because the outer query needs none of the inner columns. Both work, and a column that is not a grouping key is PostgreSQL’s 42803.

Still open

• Under grouping sets, a HAVING term over a key (having g || 'x' = 'ax') and an expression over an expression key (rollup (upper(g))) are still refused.

• A decimal or exponent literal, and a negative number written against its operator (k =-5), are not taken out of the text; such a statement is planned every time.

Rust MongoDB server: sessions, transactions and retryable writes follow mongod’s rules

We compared the Rust MongoDB server with mongod 8.2.11 on 251 scripts of session, transaction and retryable-write commands. 189 differed. 11 do now.

Two of the differences let a write through that should have been refused. A command whose session fields were malformed, such as autocommit: true or a txnNumber with no session, was run as an ordinary write. And a collection created inside a transaction with a validator took any document for the rest of that transaction. A third gave a wrong reply: a retried batch that had failed part-way was run again from the start, so the retry reported a duplicate key for the document its own first attempt had inserted.

Fixed

• The session fields (lsid, txnNumber, autocommit, startTransaction) are checked on every command, with mongod’s codes and messages. A txnNumber must be an int64.

• A transaction’s state errors are mongod’s: 225 for an older number on any request, 117 for startTransaction at the active number, 251 naming the session and its active number or saying the transaction was aborted, with TransientTransactionError first in the reply. Transaction number 0 is valid.

• A command a transaction does not allow (count, listCollections, ping and the rest) is refused without aborting the transaction, and as a first statement starts none. A statement that fails while running still aborts.

• commitTransaction and abortTransaction need the admin database and a transaction (13, 125).

• In a transaction: $unionWith, a tailable find, create of a view or a time-series collection, createIndexes on an existing or non-empty collection, an insert into a time-series collection, the config and local databases and system collections are refused as mongod refuses them. An unordered batch stops at its first error.

• A collection created in a transaction is held to its validator by the transaction’s later statements.

• A retryable write is remembered per statement and only if it wrote. A retry answers retriedStmtIds (retriedStmtId for findAndModify) and runs only the statements that had not been applied. stmtId and stmtIds are honoured. An older number is 225, a multi update or a limit: 0 delete is refused, and a txnNumber on a command that is not a write is 50768.

• A cursor can be read on only by the session, and the transaction, that opened it (50736 to 50741). killSessions kills the session’s cursors.

• endSessions no longer aborts the session’s open transaction; mongod leaves it to the client or to the transaction lifetime.

• A transaction idle for its lifetime (60 seconds) is aborted about a second later, whatever else the server is doing. It used to be aborted only when another transaction command arrived, so a writer waiting on a document an abandoned transaction held could wait for ever.

• A commitTransaction whose commit failed in the storage engine answered ok: 1. It now reports the error and the transaction is aborted.

• A malformed readConcern gets mongod’s error, and a command that is not a read refuses a level above local.

• A write sent with w: 0 reports n: 0 and no write errors.

• An insert, update or delete reply lists writeErrors before upserted.

• $geoNear with no geo index answers 27 IndexNotFound.

The Rust MongoDB server logs slow operations

A benchmark run on a cloud machine refused a row because one writer had not finished an insert_many within ten seconds. Nothing could say whether the server had stalled or the machine had: the server logged nothing about it, and the harness had discarded the server’s output and removed its store.

The Rust MongoDB server now writes a Slow query line for every operation that runs for 100 ms or longer, as mongod does. Under heavy disk contention on a Mac, both servers stall an insert for seconds at a time (mongod up to 4.1 s, the Rust server up to 3.3 s over two 60-second runs each); mongod’s own line attributes its stall to waiting on the WiredTiger cache. The ten-second stall itself was not reproduced.

Added

• secantusd-rs logs Slow query with the namespace, command, outcome and duration of any operation at or over slowms (default 100), at INFO and in the getLog buffer. Time an awaitData getMore spends waiting is not counted.

• The benchmark harness (bench/concurrency.py) keeps the server’s log and store when a row fails, prints where they are, and shows the end of the log. It runs the Rust server at INFO so the slow lines are in it.

Fixed

• profile’s slowms and sampleRate are server-wide on the Rust MongoDB server, as on mongod 8.2.11. They were kept per database, so a threshold set through one database read back as 100 through another.

• The Rust server’s documentation gave the --cache-size default as 1G. It is 4G.

Rust MongoDB server: time-series collections checked as mongod checks them, and $out keeps the target’s validator

A time-series collection on the Rust server was a plain collection with a label. Measured against mongod 8.2.11 over 187 scenarios, 139 answers differed: any timeseries document was accepted, a measurement with no date in its time field was stored, and updates were either let through or refused with one message for every case. 24 now differ, nearly all of them because mongod stores buckets in system.buckets.<name> and this server does not.

One bug found on the way affects every collection: $out dropped its target and created it again bare, so the target’s validator and indexes were gone after the first $out, and the next one wrote documents the validator would have refused.

Added

• create validates a timeseries definition (timeField, metaField, granularity, bucketMaxSpanSeconds / bucketRoundingSeconds), refuses it beside capped, validator, viewOn, clusteredIndex and changeStreamPreAndPostImages, and stores it with its defaults, as listCollections shows it on mongod.

• A time-series collection with a meta field is created with the <meta>_1_<time>_1 index, and lists no _id index.

• collMod on a time-series collection: a wider granularity or bucket span, expireAfterSeconds (a number or "off"), and mongod’s refusals for the rest.

• $out with {db, coll, timeseries} creates a time-series collection, or writes into an existing one with that definition.

Fixed

• $out keeps the target collection’s options and indexes.

• An insert into a time-series collection refuses a document whose time field is missing or not a date (2).

• An update of a time-series collection follows mongod’s rules, each with its own message (72): multi only, no upsert, no pipeline or replacement, a query on the meta field only, and changes to the meta field only.

• findAndModify on a time-series collection takes an unsorted remove and refuses everything else (72).

• Unique and text indexes are refused on a time-series collection, and a TTL index needs a partial filter on the meta field.

• A time-series collection cannot be renamed or renamed onto, watched by a change stream, read with tailable, min or max, or written to by $merge.

• dropDatabase answers {ok: 1} with no dropped field when the server runs as a replica-set member, as mongod does.

• create words an unknown field with its final full stop.

Rust MongoDB server: TTL indexes and the bulkWrite command, measured against mongod

Two more surfaces of the Rust MongoDB server were compared with mongod 8.2.11, scenario by scenario, and the differences fixed. TTL indexes had two that lose or hide data. A partial TTL index expired every old document in the collection, including the ones its filter leaves out. And an expiry was written nowhere: no oplog entry, so a change stream never saw the delete and recovery from the oplog could not replay it. Expiry is now an ordinary delete with the index’s filter, which fixes both and also expires a document whose field is an array holding an old date.

The bulkWrite command (the 8.0 server command, not a driver’s helper) checked each operation as it reached it, so a malformed third operation left the first two applied. It now checks the whole command before writing anything, with mongod’s codes and messages. The probe also found four bugs in plain insert and update, listed below.

Added

• setParameter / getParameter take ttlMonitorSleepSecs and ttlMonitorEnabled, and they act on the running monitor: a test can set the sleep to 1 second instead of waiting a minute.

• serverStatus reports metrics.ttl (passes, deletedDocuments), so a test can wait for a pass instead of sleeping.

• update statements take c (constants) and upsertSupplied; bulkWrite update operations take constants and upsertSupplied.

• tools/probes/bulk_write_command.py (141 scenarios) and tools/probes/ttl_indexes.py (50).

Fixed

• A partial TTL index no longer deletes documents outside its partialFilterExpression.

• A TTL expiry is written to the oplog and reaches change streams.

• A document expires when its TTL field is an array holding an expired date, directly or through a dotted path.

• Only a single-field index expires documents. collMod refuses to put a TTL on a compound index or on _id, refuses a negative or non-numeric expireAfterSeconds, and stores one past int32 as int32’s largest; createIndexes refuses NaN. expireAfterSeconds is stored as an int32.

• The same index name and key with a different expireAfterSeconds is 85 IndexOptionsConflict, as an equivalent index, where it was 86.

• A TTL sweep that failed on one collection reported success.

• create refuses expireAfterSeconds without clusteredIndex or timeseries.

• insert refuses an array or regex _id (53), and so does an upsert that would produce one. Both were stored.

• An upsert onto a taken _id reports the collection, index and key, where it answered a bare E11000 duplicate key error.

• A $-prefixed field in an inserted _id document is code 52, where it was 2.

• bulkWrite: nothing is written when any operation, nsInfo entry or option is malformed; an operation on a view fails with 166 where it was accepted; a failed operation’s entry carries errInfo and, for an update, nModified; an unacknowledged write returns no per-operation results; a namespace index may be any numeric type.

• createIndexes refuses a partialFilterExpression using an operator a partial index cannot hold ($ne, $nin, $not, $nor, $exists: false, $regex, $mod, $size, $elemMatch, the $bits* operators, $expr, $where, $text, $jsonSchema).

Rust MongoDB server: update and findAndModify checked against mongod

An update of a document on the Rust MongoDB server could rewrite a field it did not name. A stored sub-document shaped like extended JSON, such as {$numberLong: "1"} or {$oid: "..."}, came back from the server’s BSON decoder as the value it would stand for, so writing the document back replaced the sub-document with a number or an ObjectId. Every decode now keeps it as the document it is.

It was found by a new probe, tools/probes/update_commands.py, which sends 1,295 update and findAndModify commands to the server and to mongod 8.2.11 and compares each reply and the documents left behind. The server differed on 213 of the first 1,096 and differs on 17.

Fixed

• A stored sub-document shaped like extended JSON is no longer rewritten when its document is updated or read through aggregate.

• An update adds new fields in path order across its operators, as mongod does, not in the order written.

• The positional operator $ names the one array position mongod names when a query has more than one array predicate, applies to any array in the update, and is refused on an upsert.

• A pipeline update can no longer change the _id. An operator that writes the _id without changing it is accepted.

• {$set: {$x: 1}}, and the same in a pipeline or in aggregate, no longer stores a top-level $x field.

• $bit keeps a long a long. $addToSet, $pull, $pullAll and $push with $sort store what mongod stores for seven shapes they got wrong.

• A pipeline $set or $addFields of a dotted path follows the aggregation rules: through a scalar, through an array, and for a nested document.

• An upsert whose query names both a and a.b, or uses $expr, is refused.

• An update with no upsert no longer creates the collection it names.

• Timestamp(0, 0) at the top level of an inserted or replacement document is filled with the current time.

• A document may not nest deeper than 180 levels.

• findAndModify reads upsert: 1 as true, refuses arrayFilters with remove or a pipeline, and checks a pipeline and a projection when nothing matches.

• update and findAndModify refuse unknown fields, and update checks the type of updates, of each statement and of upsert.

• Error messages for malformed operator arguments are mongod’s, where the server had answered that it does not support the construct.

Added

• The expression operator $const.

Rust MongoDB server: user and role commands answer as mongod does

The sixteen user and role management commands on the Rust MongoDB server, and usersInfo and rolesInfo, were compared with mongod 8.2.11 reply by reply. Of 399 replies 277 differed; 7 do now, each of them the order of a set of roles.

Fixed

• dropRole removes the role from every user and role that held it. It was left named there.

• A built-in role exists only in the databases mongod has it in: root on a database other than admin is RoleNotFound, where it was accepted.

• A role name given as {role: "read"} with no db is refused. It was taken to mean the current database.

• createUser refuses a mechanism it does not know, where it dropped the name, and refuses an empty password, a user on local, and a password for a user on $external.

• A role on a database other than admin cannot be granted privileges or roles of another database, and a grant that would make a role inherit from itself is refused.

• Every field of these commands is checked for its type, and an unknown field is refused, with mongod’s codes and messages.

Added

• rolesInfo answers for the built-in roles, with the privileges mongod gives each, and takes showBuiltinRoles, showPrivileges (also "asUserFragment") and showAuthenticationRestrictions.

• usersInfo takes filter, showPrivileges, showCustomData and showAuthenticationRestrictions, and lists userId and customData.

• createUser and updateUser store customData and authenticationRestrictions; updateUser can change mechanisms; a password the client digested is taken for SCRAM-SHA-1.

• createRole and updateRole store authenticationRestrictions, and check privileges against mongod’s 170 action names.

Changed

• A plain rolesInfo no longer lists privileges; ask with showPrivileges: true, as on mongod.

• A role that mongod has and enforcement here does not know (hostManager, enableSharding) can now be granted. It permits nothing yet (tasks/backlog.md 7.083).

Rust MongoDB server: validate checks the collection

validate on the Rust MongoDB server answered valid: true without reading anything, and reported every index as holding one key per document. It now reads every document, compares each index with the entries its documents should have, in both directions, and checks that the _id index leads to each document and to none that is gone.

Fixed

• validate answers valid: false, with the counts in errors and the index marked in indexDetails, when a document cannot be read, an index entry is missing, or an entry belongs to no document.

• keysPerIndex is the number of keys the documents give the index, as mongod 8.2.11 counts them: an array gives one per distinct element.

Added

• Storage::check_collection, the check itself, with tests over a collection the server wrote and one damaged by hand.

An ordered insert on the Rust MongoDB server keeps the documents before a validation failure

When a document in an ordered insert batch failed the collection’s validator, the Rust MongoDB server inserted none of the batch and reported n: 0. mongod 8.2.11 inserts the documents before the failing one and stops there. A three-document batch whose second document fails now leaves the first in the collection, as on mongod.

That was the one data difference in a probe of document validation against mongod (tools/probes/validators.py): the Rust server differed on 15 of 68 results and now differs on none. Error 121’s errInfo already matched.

Fixed

• An ordered insert stops at the document that fails validation (or has a $-prefixed key in its _id) and keeps the documents before it. Write errors are reported in batch order.

• create and collMod refuse a validationLevel or validationAction that is not one of mongod’s words. They were stored, and then read as the default.

• A validator holding $where, $text, $near, $nearSphere or $geoNear is refused.

• collMod of any validation option leaves listCollections showing both validationLevel and validationAction; an empty validator removes the validator and is not listed.

• A validation failure raised by $out or $merge carries errInfo and mongod’s message.

Views on the Rust MongoDB server are read-only, and read through

A probe of views against mongod 8.2.11 (tools/probes/views.py) found the Rust MongoDB server different on 52 of 70 results. Several were wrong answers with no error:

• an insert, update or delete aimed at a view was acknowledged, and stored or changed rows under the view’s own name;

• distinct on a view returned nothing;

• $lookup, $graphLookup and $unionWith from a view matched nothing;

• collMod of a view’s pipeline or viewOn answered ok and changed nothing.

It now differs on 2 of the 70, both the system.views collection, which the server does not expose.

The same work found that $count over no documents returned {n: 0}. mongod returns no document, and so does the Rust server now. The phantom document also appeared in every $lookup, $unionWith and $facet sub-pipeline that counted no match.

Fixed

• A view refuses writes (insert, update, delete, findAndModify), index commands, collStats, validate, $collStats, a change stream, a rename, and $out / $merge into it, with mongod’s errors.

• distinct, $lookup, $graphLookup and $unionWith read a view through its pipeline.

• collMod redefines a view: pipeline and viewOn each replace their own half. Collection options are refused on a view, and view options on a collection.

• create and collMod check a view’s definition: a pipeline that is not an array, an unknown stage, $out / $merge / $changeStream, an empty viewOn, and a view defined on itself are refused. They were stored.

• find on a view accepts a $natural sort (it failed with a field-path error), refuses tailable, and refuses a collation other than the view’s.

• Dropping a view answers {ns, ok} with no index count.

• $count over no documents emits no document.

A Rust version bump now moves the install commands in the docs with it

The README, the docs and the crate READMEs print cargo install secantus-mdb --version <ver> with the version spelled out, because every release so far is a pre-release and a bare cargo install picks the 0.0.0 placeholder. Nothing moved those versions when the crates moved, so the next Rust release would have left every page telling readers to install the previous one.

Changed

• ./inv rust-version-bump rewrites the version in every documented cargo install secantus-mdb --version … and secantus-mdb = "…" (and the secantus-pg forms under --line pg), and fails if any page still names another version. A version mentioned in passing is left alone.

• A test fails when a documented install command names a version other than the one the crates carry.

./inv rust-wt-build recovers from a build directory left by an older checkout

The Rust-side WiredTiger build moved its source from vendor/wiredtiger to a patched copy on 2026-10-07. A checkout that had built before then failed at the CMake configure step with “the source … does not match the source … used to generate cache”, and stayed broken until build/rust-wt/wt-build was deleted by hand.

Fixed

• The build now removes a WiredTiger build directory whose CMake cache names a different source tree, and rebuilds.

• The 0.7.0b0 changelog entry and its blog post gave cargo install secantus-mdb without --version, which installs nothing. They now name the version.

One log sync now covers every commit already written

When several clients commit at once and each commit is synced (fdatasync on Linux), WiredTiger recorded a sync as covering only the commit of the thread that made it. The sync had flushed every commit written before it began, but the next thread synced again all the same. A sync is now credited with everything written before it started, as PostgreSQL credits a WAL flush.

Changed

• A new WiredTiger patch, cmake/patch_wt_fsync_group.py, applied by the wheel’s build and the secantus-wiredtiger-sys crate. It changes method=fsync only: the Rust PostgreSQL server on Linux, and either MongoDB server when it is run with a sync per commit. Measured on Linux with the Rust PostgreSQL server, durable writes at eight and sixteen clients are 10 to 18 percent faster; one to four clients are unchanged.

• A commit is still acknowledged only after a sync that began after its bytes were written. Checked by hard-rebooting a server under sixteen remote writers six times: no acknowledged row was missing.

[0.7.0b2] — 2026-10-08

The cargo install instructions now work

Every place that told you how to install a Rust server from crates.io printed cargo install secantus-mdb, and that command fails. Both crates carry a 0.0.0 name-reservation release and every real release is a pre-release, so cargo picks 0.0.0 and stops with “there is nothing to install”. With a version it works: cargo install secantus-mdb --version 0.5.3-beta.173, and cargo install secantus-pg --version 0.1.0-beta.3 for the Rust PostgreSQL server.

Fixed

• The README, both docs trees, the two crates’ READMEs and the site’s Rust MongoDB page give the install command with --version.

• The README now says that both Rust servers install from crates.io; it still said the PostgreSQL one was not published.

• A test fails if a published page prints the bare command again.

secantus-pg is on crates.io

The Rust PostgreSQL server’s first crates.io release is 0.1.0-beta.3: cargo install secantus-pg --version 0.1.0-beta.3 builds secantusd-pg and the WiredTiger it links, and secantus_pg::PgServer embeds it in a Rust test. It shipped with secantus-mdb 0.5.3-beta.173, which it pins, so a crates.io build carries the same storage and WiredTiger as the released binaries.

Changed

• The site’s PostgreSQL page offers cargo install secantus-pg and links the 0.1.0-beta.3 binaries; the MongoDB page links 0.5.3-beta.173.

[0.7.0b1] — 2026-10-08

The source distribution is back under PyPI’s file-size limit

0.7.0b0 reached PyPI as wheels only. Its source distribution came out at 573 MB, against 14 MB for 0.6.0b17, and PyPI refuses any file over 100 MB. A cargo build directory, crates/secantus-storage/target-dev/, had been committed to the repository: 6,875 files and 1.77 GB, all of which the sdist picked up.

Fixed

• The build directory is no longer tracked, and every cargo target directory under crates/ is now ignored by git and excluded from the sdist.

• A test fails if a cargo target directory, or any file over 8 MB, is tracked.

The Rust PostgreSQL server’s crate is linted in CI

crates/secantus-pg was built and tested in CI but never run through cargo fmt --check or cargo clippy, and six clippy::drop_non_drop errors had accumulated in it. They are fixed and the crate joins the lint step the other WiredTiger-linked crates already have.

Fixed

• Six redundant drop() calls on closures, in grace_join.rs, stream_join.rs and lib.rs. No behaviour change: a closure with no Drop releases its borrows at its last use.

Changed

• test.yml’s rust-storage job runs cargo fmt --check and cargo clippy --all-targets -- -D warnings in crates/secantus-pg.

Concurrent durable commits share log writes on the Rust PostgreSQL server (macOS)

On macOS the Rust PostgreSQL server syncs each commit with WiredTiger’s method=dsync, and stock WiredTiger gives every synced commit a log write of its own. Commits that arrive while a log write is in flight now join the next one. With 8 clients inserting, each into its own table, throughput went from 31.5k to 37.9k statements a second (PostgreSQL 15 on the same machine: 41.5k); UPDATE went from 29.6k to 33.5k. One to four clients are unchanged. A commit is still acknowledged, and still becomes visible to other sessions, only after its log record is written.

This is the first patch to the vendored WiredTiger that changes behaviour rather than the build. It applies to method=dsync commits only. The MongoDB servers, and the PostgreSQL server on Linux, use method=fsync and run the unpatched code.

Changed

• cmake/patch_wt_dsync_group.py patches src/log/log.c and src/log/log_slot.c: a dsync commit copies its record into the active log slot without closing it, and the first waiter to find every earlier slot written closes the slot and writes the group. It is applied by the wheel’s CMake build, the secantus-wiredtiger-sys crate’s bundled source and ./inv rust-wt-build.

• A dsync commit no longer takes log_sync_lock to check for a directory sync that has already happened. Losing that try-lock stalled the commit for 10 ms.

Fixed

• An existing build directory now rebuilds WiredTiger when a cmake/patch_wt_*.py script is added or edited. The patch step’s recorded command never changed, so it kept the library it had.

• ./inv rust-wt-build built WiredTiger from the unpatched submodule; it now applies the same patches as the crate’s bundled copy.

• secantus-wiredtiger-sys relinks when a prebuilt libwiredtiger.a named by SECANTUS_WT_LIB is rebuilt in place.

• bench/pg_concurrency.py’s update and select workloads addressed a row that was never seeded, so they measured statements that matched nothing. Both benchmarks also measured the main checkout’s binary from any worktree.

• Both PG benchmarks remove their temporary store on exit; twenty runs had left 1.3 GB of WiredTiger stores in the temp directory.

[0.7.0b0] — 2026-10-07

The PyPI package is the Python servers; the Rust servers ship as crates

pip install SecantusDB now installs the two Python reference servers (MongoDB and PostgreSQL) and WiredTiger, and nothing else. The Rust servers no longer ride inside the wheel. The Rust MongoDB server is the secantus-mdb crate on crates.io: cargo install secantus-mdb --version 0.5.3-beta.173 builds it, WiredTiger included, and puts secantusd-rs on your PATH. From a Rust test, secantus_mdb::Server starts it in-process. Prebuilt binaries of both Rust servers stay on GitHub Releases for machines without a Rust toolchain.

Two things drove the split. Bundling the Rust servers multiplied the size of every wheel. That growth pushed the project past PyPI’s 10 GB storage quota, and 0.6.0b18’s upload was refused. It also meant a Python user downloaded two Rust servers they might never run.

Because this changes what pip install SecantusDB gives you, the release starts the 0.7 line: it is 0.7.0b0, not 0.6.0b19.

Changed

• publish.yml and wheels.yml build the wheel with SECANTUS_BUILD_STORAGE_ENGINE off, so it carries no _secantus_server, _secantus_storage or secantusd-rs.

• If you used the embedded handle (from _secantus_server import RustServer) from a PyPI install, you have two options. You can run secantusd-rs as a subprocess and point pymongo at it, or you can use SecantusDBServer, which has the same pymongo surface. The handle remains available in a source build.

• The README, the server comparison, the installation pages and the website now give cargo install secantus-mdb --version 0.5.3-beta.173 as the way to get the Rust MongoDB server.

The Rust PostgreSQL server gives each connection its own thread

The Rust PostgreSQL server now runs every client connection on a thread of its own, the way PostgreSQL gives every connection a backend process. A statement runs directly on the thread that owns its client, instead of being handed off from a shared runtime worker to a blocking thread first, and a connection that waits (on a lock, in pg_sleep) holds up only itself.

Profiling eight inserting clients showed the server’s time going to locks, not work: the per-statement hand-off contended on the async runtime’s blocking-pool mutex, and every table lookup took one process-wide catalog cache lock. With both gone, INSERT throughput at 8 clients (one table each, no journal sync) rose from about 45k to about 61k statements a second, and scaling from 2.8x to 3.6x of one client.

Changed

• secantus-pgserver: each accepted connection runs on its own OS thread with a one-worker runtime of its own (server::accept_loop). Synchronous statement work goes through blocking_wait, which runs it in place on a connection’s runtime and uses block_in_place elsewhere. A notice sent mid-statement still hands the worker off so the socket keeps being driven.

• secantus-pgserver: the table-definition cache (lookup_inner) and the committed-catalog cache (committed_cached) keep a per-thread copy of what the shared cache holds as current, so a hit takes no shared lock.

The psycopg gauge now runs the Rust PostgreSQL server on Windows, in CI

psycopg 3’s own test suite had never finished against secantusd-pg on Windows. A new workflow builds the server on windows-latest and runs the gauge there, so that result now comes from a real run. The run completes, and every test it starts reports a result.

Added

• .github/workflows/psycopg-windows.yml: builds WiredTiger and a debug secantusd-pg on windows-latest, then runs SECANTUS_GAUGE_SERVER=rust python -m psycopg_validation.runner. The job fails when the run is truncated, meaning the tests that started outnumber the ones that reported a result. It runs weekly, on demand, and on PRs that touch crates/secantus-pg*, psycopg_validation/ or the workflow itself.

• SECANTUS_PSYCOPG_GAUGE_TIMEOUT raises the runner’s wall-clock cap on slower hosts.

Fixed

• The Windows gauge stopped at about 65% of the run. One test, test_type_error_shadow, takes 30 s against PostgreSQL itself on that runner, which is more than the gauge’s 20 s limit per test. On Windows, pytest-timeout can only use its thread method, so a test that runs over the limit ends the whole pytest process. The test is now deselected on Windows, with that reason written down.

• Two test_right_exception_on_session_timeout tests are deselected on Windows. They expect PostgreSQL’s Windows-only behaviour, where an abortive close destroys the server’s FATAL. secantusd-pg closes the connection gracefully and delivers the real 25P03.

• The Windows runner image exports PGPASSWORD. That made test_used_password expect a password challenge that a trusted role never gets. The variable is now unset for the gauge step.

• psycopg’s refcount marker is now excluded on Windows, as psycopg’s own scheduled Windows CI does. These are client-side checks that count live Python objects to find leaks. On the runner they passed in one run and failed 75 test_leak cases in the next, some with negative counts.

The Rust PostgreSQL server is ready for crates.io as secantus-pg

The Rust PostgreSQL server’s crate is now secantus-pg, the name it will have on crates.io, and it can be embedded in a Rust test in one line: secantus_pg::PgServer::start() opens a temporary store, binds a free port and hands back dsn() / url() for tokio-postgres or any other client; dropping it stops the server, checkpoints the store and removes it. It is the PostgreSQL counterpart of secantus_mdb::Server, and like it is safe to start and drop inside #[tokio::test]. The secantusd-pg binary is unchanged.

All four PostgreSQL-side crates are now publishable, and the release pipeline publishes them from a secantusd-pg-v* tag. Nothing has been published yet; the first publish happens at the next PostgreSQL server release.

Added

• secantus_pg::PgServer and PgBuilder (.storage_path, .host, .port, .databases, .cache_size, 256M by default), with address(), port(), dsn(), url(), stop() and Drop. A temporary store is removed only once RunningPgServer::store_closed() confirms WiredTiger closed it. Tests in crates/secantus-pg/tests/pg_server.rs (both tokio runtime flavours, a persistent store across a restart, 50 servers in parallel), a doc-test, and examples/quickstart.rs.

• secantus_pg::open_storage_with_cache, the PG server’s storage open with a cache size other than the daemon’s 4G.

• secantus-pgwire: the vendored pgwire 0.40.7 fork, moved from crates/vendor/pgwire to crates/secantus-pgwire and made a publishable crate. crates.io ignores [patch], and the fork has diverged too far from upstream for one small upstream PR to replace it. The library keeps the name pgwire.

• crates.io metadata, exact = version pins, READMEs and rust-version on secantus-pg, secantus-pgplan, secantus-pgcatalog and secantus-pgwire; cargo binstall secantus-pg metadata pointing at the secantusd-pg-v* release archives.

• scripts/crates_package_check.py --line {mdb,pg,all}, and the crates-package.yml gate now packages and builds both lines from their tarballs. publish-crates.yml fires on secantusd-pg-v* tags as well and publishes the PG line (crates_publish.py --line pg), refusing to start if a MongoDB-side crate it pins is not on crates.io yet.

• ./inv rust-version-bump --line pg --to <ver> bumps the PG version line.

Changed

• crates/secantus-pgserver is now crates/secantus-pg (package secantus-pg, library secantus_pg), licensed GPL-2.0-only as the plan decided; secantus-pgplan / secantus-pgcatalog stay Apache-2.0. Every path in the workflows, invoke tasks, gauges, probes, benches and tests moved with it.

• The binary’s signal-handling dependencies sit behind a default-on bin feature; the Python extension builds the library without them.

• secantusd-pg --version prints source: crates.io for a build from a packaged crate, as secantusd-rs does.

The oplog pruner stops monopolising the Rust PostgreSQL server’s journal

In durable mode, concurrent INSERT throughput on the Rust PostgreSQL server stopped growing at about 2x with more clients. Most of that ceiling came from the background oplog pruner. It deleted old entries one autocommit at a time, so every delete was a synced journal write of its own, competing with the commits clients were waiting on. It now deletes in transactions of 1,024 rows. Measured at 8 clients, release build: 24.4k to 27.5k inserts/s, with 2 clients at 1.95x the single-client rate (was 1.55x).

FROM-less literal SELECTs (select 1) now run on the connection’s worker thread instead of being handed off to another one, which saves about 2 µs a statement.

A new test SIGKILLs the server under eight concurrent writers and a reader. Every row a writer saw acknowledged, and every row the reader saw, must survive the restart. As in PostgreSQL, a committed row is visible to other sessions only once it is durable.

Changed

• secantus-storage: the oplog prune sweep deletes in transactions of 1,024 rows instead of one autocommit per row. The MongoDB server benefits the same way under --sync-on-commit.

• secantus-pgserver: literal-only FROM-less SELECTs skip the block_in_place hand-off.

[0.6.0b18] — 2026-10-07

Every MongoDB backlog item closed, and a Rust PostgreSQL server that runs real SQL

This release bundles 131 slices. Three themes stand out.

The first is the Rust PostgreSQL server, which went from a protocol and type system with a thin query language to most of what an application writes. It gained general JOINs, subqueries and CTEs, window functions, GROUPING SETS, ON CONFLICT, sequences, ALTER TABLE, views and indexes, user functions and triggers, MERGE, row-level security, full-text search, and PostgreSQL’s own catalogs. Much of it now streams in bounded memory. Every piece was measured against a real PostgreSQL rather than written from the documentation. Several fixes were silent wrong answers. A DELETE ... USING deleted every row, an UPDATE ... FROM wrote to every row, and an autocommit write could lose a concurrent update.

The second is the MongoDB servers. Every open backlog item for the Rust and Python MongoDB servers was reproduced against mongod 8.2.11 and either fixed, closed by re-measurement, or given a written disposition. The Python reference server caught up with the Rust one. On every differential probe, both now give mongod’s answers, apart from documented, deliberate differences. Seven of the fixes were silent writes, for example a failed updateMany that rolled back work mongod keeps, and a $set past the end of an array that stored nothing. Both servers now also read date strings with mongod’s own parser, a port of timelib.

The third is honesty in the published numbers. A Go gauge had reported 100% over a run that stopped three-quarters of the way through, and a crashed pymongo gauge had published a fabricated report. The driver panels could only ever show the Python server. All of these now refuse to publish a number they did not measure. A stale build is caught for every artifact, and reports are stamped with the date of the data behind them.

The admin screenshots regenerate correctly on Windows

The 22 admin-UI screenshots had not been regenerated since 0.6.0b11 (2026-08-15), so the published images still showed a 0.6.0b10 version badge six releases later. tests/test_docs_screenshots.py cannot catch this — it checks that every documented page has an image, and nothing checks what is in one.

Regenerating on Windows then surfaced two faults in the capture script itself, both of which would have shipped a worse image than the stale one they replaced.

Fixed

• scripts/admin_screenshots.py scrubbed machine-specific paths by rewriting only their PREFIX, so on Windows every segment after the placeholder kept its backslash and the docs would have shown /var/lib/secantus\backups and /home/user\.secantus\embedded-data. The scrubber now re-slashes the run of path characters it has just written, and leaves backslashes anywhere else on the page alone.

Changed

• Regenerated all 22 admin screenshots at 0.6.0b17.

apiStrict: true now rejects a command outside the Stable API

A client that declares serverApi: {version: "1", strict: true} is asking the server to refuse anything outside MongoDB’s Stable API Version 1. SecantusDB accepted those commands and ran them, which is the one thing a strict client has explicitly asked you not to do — a driver using strict mode to catch non-portable calls in CI got a clean run and a false sense of portability.

Both servers now answer mongod’s APIStrictError (323), with mongod’s own message, for a command outside Version 1. The rule has three branches rather than the two the specification’s wording suggests, and the third is the one that matters: a command the server has but that sits outside Version 1 is rejected, a command inside Version 1 runs, and a command that does not exist at all still answers CommandNotFound (59) even under apiStrict. A check that simply refused everything it did not recognise would get that last case wrong.

The membership list was measured, not transcribed, by sending every command SecantusDB implements to a real mongod 8.2.11 under apiStrict: true and recording which it refused. That is how distinct, buildInfo, isMaster and serverStatus turned out to be outside the Stable API while count and hello are inside it — a set assembled by reading the manual would have had several of those the wrong way round.

Fixed

• commands.py / crates/secantus-commands: apiStrict: true answers APIStrictError (323) for a command outside Stable API Version 1, with mongod’s verbatim message including the dochub link.

• The aggregation-stage message was wrong on both servers. mongod says $listLocalSessions is not allowed with 'apiStrict: true' in API Version 1; we said Provided aggregation pipeline stage ... is not in API Version 1.

• Closes the apiStrict failure in two driver gauges — pymongo’s TestVersionedApiTestCommandsStrictMode and the Java driver’s VersionedApiTest strict-mode case. Two unrelated drivers failing the same behaviour is what marked it as a real gap rather than a harness artifact.

Changed

• Replaced the narrow _API_V1_REJECTED_BY_NAME gate (just distinct, with a non-mongod message) and the Rust name == "distinct" canary with one measured allowlist per server. The old gate’s comment said a full allowlist “would reject count”; probing mongod showed count is in Version 1, so that objection was a guess rather than a measurement.

Testing

• tests/test_mongod_differential.py: eleven cases asserting an exact match — code, codeName and errmsg — against a live mongod, covering all three branches. testVersion2 is deliberately excluded: it is gated behind enableTestCommands, so including it would assert the fixture’s configuration rather than our conformance. The driver gauges cover that one.

A stale build was only caught for one of four artifacts

#1489 added a collection-time check that refuses to run when the installed _secantus_core was built from different sources than the checkout. It covered that one extension. _secantus_server, secantusd-pg and secantusd-rs had no such protection, and on 2026-09-27 that cost nine false-regression diagnoses in a single session:

• a stale _secantus_server — six days behind, predating #1569 — failed a colleague’s entire new failpoint test file with 3 failures and a hang that killed three xdist workers at 99%, surfacing as a bare rc=70 with no summary line. It was diagnosed as “main is broken” and reported as such. 13 of 13 passed after a rebuild, with no code change.

• a stale secantusd-pg produced failures on six separate occasions, each time in the test file whose fix the binary predated, each time reading as somebody else’s regression.

The binaries already carried a source stamp from #1492 and nothing read it — the worst arrangement, with the evidence sitting in --version output that no automated thing looked at.

Added

• secantus-server-py: build.rs stamps the crates tree hash, exposed as _secantus_server.__source_tree__, mirroring secantus-core-py.

• tests/conftest.py: the collection-time check now covers all four artifacts, reading the binaries’ stamps via --version.

• tests/test_build_provenance.py: cases for the generic check, weighted toward the configurations where it must stay silent — an unstamped artifact, a checkout git cannot read, a binary that is not built — because a false positive is how a check gets disabled.

Notes on the design

The identifier is a git tree hash, not the commit SHA (which moves on every commit and would cry stale constantly) and not the package version (constant for these path dependencies — #1489 measured beta.163 reported by a build stale enough to need rebuilding twice in a day). This is why no Python packaging tool solves it: pip and uv compare versions, and uv sync reuses a cached build for a path dependency whose version has not moved.

_secantus_core keeps its narrower two-crate stamp so an unrelated crate’s change does not read as stale there; the server extension links most of the workspace and takes the whole crates tree, which is also what both binaries stamp. Conflating them would either over-report on core or under-report on the server.

The rebuild commands are asserted against rust_tasks.py rather than trusted: the first draft pointed secantusd-rs at rust-server-build, which rebuilds the embedded extension instead of the binary. A check that prints the wrong remedy sends the reader on a detour and teaches them to distrust it.

The C driver gauge covers change streams again

Running libmongoc’s suite against a --standalone server fixed four tests that assert standalone semantics, but libmongoc then skipped every test that needs a replica set – most of its change-stream suite. The gauge now runs the suite against the default single-node replica set and re-runs just the six standalone-only tests against a standalone server, then merges the two. On the Rust server that is 790 passed, 2 failed (the known IPv6 tests), up from 768.

Changed

• c_validation runs two passes and merges them; the standalone tests are listed in c_validation/include_paths.py as STANDALONE_ONLY.

cbrt is judged against the platform’s libm, not a hardcoded 3.0

Three cbrt tests asserted that cbrt(27.0) is exactly 3.0. That is what Windows and macOS answer — and it is not what Linux answers. PostgreSQL’s dcbrt is a bare libm cbrt() call, libm is not correctly rounded, and glibc 2.39 returns 3.0000000000000004 for that input: exactly one ULP high, while 8, 64, 125 and 1e6 come out exact. So PostgreSQL on Linux returns 3.0000000000000004 too, and the assertion was pinning the wrong platform’s answer.

The tests now assert to within one ULP, plus a separate check that we hand libm’s value through byte-for-byte — which is the property that matters, since being more exact than libm would move us away from PostgreSQL. _real_cbrt itself is unchanged; its docstring already named this trade-off correctly.

Worth recording how it hid for so long: a push or PR run tests only Python 3.10 on Linux, and 3.10 has no math.cbrt, so it takes a Newton-refined fallback that rounds the ULP away and passes. Only the full 3.10–3.13 matrix — a workflow_dispatch or the weekly cron — runs the Python versions that fail. It had been red in the cron before anyone looked.

Fixed

• tests/test_sql_missing_builtins.py: cbrt expectations are no longer hardcoded floats. Verified by emulating glibc’s cbrt(27.0) on a non-Linux box: the old assertions fail with exactly the three names CI reported, the new ones pass.

Added

• A test that a 1-ULP libm value survives the SQL layer unrounded, so nothing between the function and the result row quietly “corrects” it.

• Two backlog entries: the ≤1 ULP divergence the 3.10 fallback causes against PostgreSQL on Linux (a record, not a task — it disappears when 3.10 support does), and the CI matrix gap that let a 3.11+ Linux failure stay invisible to every pull request.

The MongoDB-side Rust crates are publishable

The ten crates behind the Rust MongoDB server now package for crates.io, and a CI gate keeps them that way.

Changed

• The secantusdb crate is now named secantus-mdb. Its directory, the secantusd-rs binary and the secantusdb-v* release tags are unchanged.

• secantus-wt now builds through the bundled WiredTiger by default. The SECANTUS_WT_INCLUDE / SECANTUS_WT_LIB override still links a prebuilt WiredTiger, which is how the wheel and the release binaries build. zlib and lz4 are now linked statically from bundled sources everywhere: zlib from libz-sys, and lz4 from lz4.c alone, vendored in secantus-wiredtiger-sys. It is not lz4-sys, because that crate’s bundled xxhash collides with libpg_query’s when both link into the PostgreSQL server.

• Regenerating the bindings with bindgen is now opt-in (--features bindgen). ./inv rust-wt-test turns it on, so the drift check still runs in CI.

• secantus-wiredtiger-sys joins the MongoDB server’s version line.

Added

• Every one of the ten crates has crates.io metadata, a README, and exact =version pins on its sibling crates. The internal crates say they carry no semver promise.

• secantusd-rs --version prints source: crates.io for a build from a packaged crate, which has no git tree to stamp.

• docs.rs can document the crates without compiling WiredTiger.

• scripts/crates_package_check.py packages all ten crates together and builds each one from its own tarball. The new crates-package.yml workflow runs it on every PR that touches crates/.

Rust release tooling for crates.io

A Rust release now bumps, checks and publishes the crates in one consistent path.

Added

• ./inv rust-version-bump --to <version> bumps the Rust MongoDB server’s lockstep version. It rewrites every version, every exact = pin between the crates, and every Cargo.lock that records one. It then fails if the old version survives anywhere or a lockfile no longer resolves --locked, which the release builds require.

• cargo binstall secantus-mdb installs the prebuilt, PGO-optimised secantusd-rs from the GitHub release instead of compiling WiredTiger.

Changed

• The crates.io publish step publishes one crate at a time and skips any version that is already published. A run that failed part way is fixed by re-running it on the same tag.

featureCompatibilityVersion said 7.0 while the server said 8.2.11

getParameter reported featureCompatibilityVersion: {version: "7.0"} on both servers. The handshake right next to it reported buildInfo.version 8.2.11, versionArray [8, 2, 11, 0] and maxWireVersion 27 — so the value contradicted the same server’s own answers, not merely mongod’s. A client gating a feature on FCV was told this is a 7.0 deployment.

Measured against a mongod 8.2.11, which answers {"version": "8.2"} — the major.minor of the running binary.

Fixed

• Both servers now derive the string from SERVER_VERSION_ARRAY instead of carrying a second copy of the version. As a literal it survived the retarget from 6.0 to 8.x untouched while every other version surface moved; deriving it is what stops that recurring.

Added

• A differential test asserting FCV equals mongod’s, and a value assertion in the Rust get_parameter_named_and_all unit test.

  The two tests that already looked at this parameter asserted only that the KEY was present — which is exactly the assertion a stale literal survives, and why nothing caught a wrong value for the whole life of the 8.x retarget.

The Go gauge was hiding 16 failures behind a 100% score

It had been reporting 100.0% over a run that never finished. go test killing itself on -timeout panics the binary without emitting a terminal event for the tests still in flight, so 476 of 481 reported and the summariser scored the survivors as perfect. A truncated run looks better the more tests go missing, which is why it survived review — in every committed report back through 2026-09-21.

With the run completing: 594 passed / 16 failed / 49 skipped over 659 tests — 97.3%, on both servers. The 16 failures are not new; they simply never ran.

Fixed

• TestInitialDNSSeedlistDiscoverySpec/replica_set excluded — the narrow branch, not the whole spec, whose sharded, load_balanced and non-RS cases all run.

  It blocks forever: getServerByAddress calls topo.SelectServer(context.Background(), ...) — no deadline — waiting for the SRV-resolved localhost:27017 to join the topology. Our daemon binds an ephemeral port, so it never joins. The subtest is gated on the replica-set persona this gauge deliberately keeps for change streams, so --standalone cannot dodge it the way it does for the C gauge.

  What is lost: the Go driver’s own SRV/TXT resolver. No SecantusDB code path. Measured: 803 started / 798 finished / 5 hung before, 659 / 659 / 0 in ~6 minutes after — the timeout had also been costing 30 minutes per run.

Found

Eight distinct leaf failures, triaged in tasks/backlog.md. Two are worth naming here: setParameter is not implemented on either server (mongod has it), and replSetStepDown answers 59 where a standalone mongod answers 76 NoReplicationEnabled — a divergence rather than an unimplemented feature. Two more, the SDAM pool-clearing pair, assert the pool is not cleared on a timeout or cancelled context, and look most likely to be real.

The Go gauge reported 100.0% over a run that stopped three-quarters of the way through

go test killing itself on -timeout panics the binary without emitting a terminal event for the tests still in flight. The summariser therefore counted only the tests that finished, found no failures among them, and printed 100.0% — while the package-level event in the same file said fail. The two disagreed, and nothing compared them.

Every recorded run of this gauge had been truncated at the same point: 476 of 481 tests, on both servers, including the committed 2026-09-21 report. The five that never complete are TestInitialDNSSeedlistDiscoverySpec and two others, which resolve SRV/TXT records against mongodb.test.build.10gen.cc — DNS tests that never open a connection to SecantusDB, and which cost 30 minutes of wall clock per run before the binary gives up.

So the published “Go: 100%” has never described the whole include set. What it did describe is real: 439 of 439 tests that ran passed.

Fixed

• The report generator now detects a truncated run — a test that emitted run with no terminal event, or a package reporting fail with no failing test beneath it — and refuses to present the rate as a result: the report opens with a banner naming the tests that never finished, and the process exits non-zero.

  The banner goes above the table on purpose. A reader who meets the numbers first has already formed a view of the pass rate by the time a footnote reaches them.

• An ordinary red run (a package fail with a failing test under it) is deliberately not flagged. A guard that cried truncation at every failing gauge would be ignored exactly when it mattered.

Added

• tests/test_go_gauge_truncation.py, the Go counterpart of the pgjdbc guard in tests/test_pgjdbc_gauge_truncation.py — which already existed, and which the Go gauge simply never had.

How it was found

A fresh Rust-server run and a stale report from a different server, a different version and a week earlier carried identical numbers. Agreement between a fresh artifact and a stale one reads as confirmation; here it meant both were cut off at the same DNS hang. The honest signal was the wall clock: 30m 09s for a gauge whose own timeout is 30m is a truncation, not a slow run.

---

A crashed pymongo gauge published a fabricated report

Separate bug, same class, found in the same run. The two pymongo tasks run pytest with warn=True — correctly, since a partially-red suite still owes us a report — and then generated the report unconditionally. Unlike _run_gauge, they never cleared the raw artifact first.

So invoke validate --server rust died in one second on a missing _secantus_server, collected zero tests, and rewrote docs/validation-report-rust-server.md as “Generated 2026-09-28 — SecantusDB 0.6.0b17”, 99.4% — over a raw artifact from 30 August.

The figures had even drifted against the previous report (99.5% → 99.4%, with a new failing test) because the generator changed under the same data. A reader diffing the file would have concluded a fresh run had caught a regression.

Fixed

• Both pymongo tasks now clear the raw before the run and refuse to generate a report when the run produced none, leaving the previous report untouched and naming the usual cause (invoke rust-server-build).

Also found, not fixed here

Seven -rust-server reports claim “Generated 2026-09-21 — 0.6.0b16” over raw artifacts dated 10–30 August: pymongo, pymongo-async, c, cxx, dotnet, php-ext and php-lib. The date records when the generator ran, not when the tests did. Filed in tasks/backlog.md.

Drivers now speak modern hello instead of falling back to legacy isMaster

A driver puts helloOk: true in its opening handshake to ask whether the server understands the modern hello command; a server that does echoes helloOk: true back, and the driver uses hello from then on. Neither SecantusDB server echoed it, so every modern driver concluded the server predated hello and fell back to the legacy isMaster — on every connection it opened, for the life of that connection.

That was visible on the wire rather than inferred: captured through a logging proxy, mongo-go-driver’s SDAM monitor sent isMaster exhaustAllowed to SecantusDB and hello exhaustAllowed to a real mongod, and did no RTT monitoring against us at all. Both servers now echo the flag, and only when the client asks for it — mongod omits it otherwise, so echoing unconditionally would be its own divergence.

Fixed

• hello / isMaster echo helloOk: true when the client’s request carries it, over both the initial OP_QUERY handshake and OP_MSG, matching mongod 8.2.11.

Indexes order embedded documents and arrays by value

An index keyed a document or an array by its raw BSON, which begins with the value’s length, so byte order was size order: {a: 2, b: [3]} sorted above {a: 5} because it is longer. The Rust server refused to use an index for a range or sort over such values; the Python server used it anyway and scanned the wrong part of the index. Both servers now key documents and arrays the way mongod compares them, element by element, and use the index.

Changed

• Index entry format 4. A store whose indexes were written by an earlier build is refused at open with IncompatibleStorageFormatError; drop and recreate the indexes (the documents themselves are untouched – _id keys keep their encoding).

• Range queries, partial-index use and sorts over document-valued fields use the index on the Rust server again; on the Python server they now return the right documents.

• Strings inside documents and arrays follow the index’s collation, as they do in mongod.

Added

• tools/probes/value_order_encoding.py: the index encoder against mongod’s $cmp over 16,110 random pairs.

A LIKE pattern ending in an escape character matched instead of nothing

A trailing escape has no character to escape. Falling through to the default branch treated it as a literal backslash, so a pattern of a + backslash matched a value of a + backslash — where PostgreSQL returns no rows.

Measured on PostgreSQL 14 (2026-09-25) with a matching row present and the pattern bound as a parameter: no rows, and no error.

Fixed

• _like_to_regex returns a never-matching regex when the escape character is the last character of the pattern.

The check happens during the pattern scan, not with endswith. A pattern of a + two backslashes also ends with the escape character, but there the first consumes the second and it matches one literal backslash perfectly well — a first version of this guard used endswith and broke exactly that case. Both sides are pinned by the tests, along with the same pair under a custom ESCAPE character and the ESCAPE '' case where escaping is disabled and a trailing backslash really is literal.

Found from pgjdbc’s DatabaseMetaDataTest::escaping.

./inv rust-gate lints the same scope as CI again

When CI’s lint step was widened from src tests to ., rust_tasks.py’s gate was left at the old scope — so the gate would have passed while CI failed, which is exactly the failure its own comment says it exists to prevent. Found by the documentation pass at session close rather than by anything automated.

Fixed

• rust-gate now runs ruff check . and ruff format --check ., matching CI.

Added

• test_the_local_gate_lints_the_same_scope_as_ci pins the two together, so widening one without the other fails a test instead of a push.

The macOS release binary builds again

The macOS secantusd-rs for 0.5.3-beta.166 was not published, because its profile-guided-optimisation (PGO) build failed. 0.5.3-beta.167 restores it.

Fixed

• The cc crate copied rustc’s PGO flag onto clang, so Apple clang instrumented the bundled lz4.c. Its profile format no longer matches the one rustc 1.99 writes, and the instrumented binary failed with “Runtime and instrumentation version mismatch”. The lz4 build no longer takes rustc’s flags.

• zlib links the system library again where there is one (always on macOS) instead of being compiled with the same inherited flags. The Linux release lanes still link it statically.

The macOS release binary builds again, for real this time

0.5.3-beta.166 and 0.5.3-beta.167 published no macOS secantusd-rs. 0.5.3-beta.168 restores it.

Fixed

• The macOS profile-guided build instrumented C as well as Rust. The cc crate copies rustc’s profiling flags onto clang, so ring (used by rustls) was instrumented by Apple clang. Apple clang’s profile format no longer matches the one rustc 1.99 reads. The macOS release build now compiles C through a wrapper that drops those flags, so only Rust is profiled. The 0.5.3-beta.167 fix covered only the bundled lz4, which was not enough.

Start the Rust MongoDB server from a Rust test

secantus-mdb is now a library as well as the secantusd-rs binary. A Rust test suite can start a server in one line and connect the official mongodb driver to it.

let server = secantus_mdb::Server::start()?;
let client = mongodb::sync::Client::with_uri_str(server.uri())?;

Added

• Server::start() gives each server a temporary store, which is removed on drop, and an OS-assigned port. It advertises the secantus single-node replica set and enables test commands, as the Python embedded handle does.

• Server::builder() sets a persistent store, the host, the port, the replica-set name (or none), auth, TLS, the WiredTiger cache (256M by default) and test commands. A server provides uri(), address(), port() and stop(), and dropping it stops it.

• A temporary store is removed only after the storage is confirmed closed. If a connection outlives the shutdown drain, the directory is left in place with a warning, rather than removed while WiredTiger still has it open.

• A runnable quickstart example, a doc-test, and tests through the official driver, including use inside #[tokio::test] and 50 servers in parallel.

Changed

• The binary’s own dependencies (ctrlc, env_logger, mimalloc) are behind a default-on bin feature, so a test that uses only the library can turn it off. Release builds with --no-default-features now pass --features bin.

Both MongoDB servers close out their backlog against mongod 8.2.11

Every open backlog item for the Rust and Python MongoDB servers was reproduced against mongod 8.2.11 and fixed, closed by re-measurement, or given a written disposition (tasks/backlog.md §7.05). Most of the work landed on the Python reference server, which had fallen behind the Rust server on error replies, write concern, failpoints, maxTimeMS, change streams and collation. On every differential probe in tools/probes/ the two servers now give the same answers as mongod, except for differences that are documented and deliberate.

Several of the Python fixes were silent writes, where a client got an acknowledgement for a result mongod would not produce:

• a failed updateMany rolled back the documents it had already rewritten;

• $set: {"b.2.c": 1} over b: [] stored nothing;

• multi: true with a replacement applied it;

• delete with limit: 5 deleted;

• createIndexes built a duplicate index under a new name, and built an index over an empty key;

• a$b, "" and system.foo became collections;

• $sort tied true with 1.

Fixed

• Python server, writes:

  • A multi-update that fails part-way keeps the documents written before the failure. The reply is still n: 0, as mongod reports it.

  • An intermediate array index past the end is created and padded with nulls.

  • A multi replacement is a per-statement 9.

  • A numeric delete limit other than 0 or 1 is 9.

  • Duplicate keys on update / upsert / findAndModify carry the executor prefix.

  • An _id changed by a replacement, $rename onto _id, and an unmatched positional $ use mongod’s words.

• Python server, indexes:

  • 85 / 86 index conflicts, in mongod’s text.

  • An empty key, an unknown plugin and a missing name are refused with mongod’s codes.

  • A unique build over duplicates fails the build with mongod’s wrapper, and names the colliding key.

• Python server, namespaces and commands:

  • Namespace validation on create and on every write command.

  • renameCollection outside admin is 13.

  • collMod refuses unknown fields (40415).

  • find sort values follow the $sort stage’s rule: a double truncates and a decimal rounds half to even.

  • An invalid regex answers 51091 with PCRE2’s message.

  • $sort orders every number before every bool.

• Python server, rules ported from the Rust server:

  • Validation errInfo, and the validator checked at create / collMod.

  • The writeConcern.w rules.

  • The awaitable hello.

  • failCommand labels and code names (mongod_codes.py).

  • 388 for an oversized transaction.

  • maxTimeMS executor prefixes. The index-build envelope and the write prefixes are given only when running standalone, as mongod does.

  • PCRE \Z / \z.

  • Type-first document comparison.

  • startOfWeek.

  • Decimal $range / $log / $pow / $bucketAuto.

  • Negative $slice.

  • $project: {_id: 1}.

  • The 32-bit index-argument rules, including the $range memory limit (146).

• Both servers:

  • Collated ordering expands a compatibility character (fi) at the primary and secondary levels.

  • ß sorts as ss plus a secondary weight. This also makes the PostgreSQL server’s ORDER BY ... COLLATE agree with PostgreSQL on Straße.

• Rust server:

  • renameCollection checks argument types before the admin check.

  • --block-compressor lz4|zlib|none is exposed on the daemon.

• Tooling:

  • detached_run.py records the exit in its state file.

  • Gauge reports are stamped with the date of the raw artifact.

  • max_time_expiry.py covers createIndexes and follows mongod’s topology.

  • Two load-sensitive tooling tests now poll.

Two test harnesses stopped hiding what the servers were telling them

Both changes are to harnesses rather than to a server, and both had the same shape: a real signal from a database was being discarded before anyone could read it.

Fixed

• A crashed oracle is now an error, not silence. The differential gate spawns a real mongod and compares against it, but sent that server’s output to DEVNULL and discarded its exit status — so a mongod that died mid-module was indistinguishable from one the harness stopped on purpose, while every comparison made after it died was against nothing at all and the gate went on reporting agreement. This is not hypothetical: a mongod on the Windows dev box really did abort on 2026-09-22, and the only surviving evidence was a minidump with no log beside it (its exception stream reads 0xE0000001, EXCEPTION_NONCONTINUABLE — mongod’s own fatal-assertion path, not a memory fault — but the assertion message was gone for good). The gate now captures the oracle’s output to <dbpath>/mongod.log, and an unexpected exit preserves that log outside the doomed dbpath and raises with its tail attached.

• The Rust PG server was being killed, not stopped, on Windows. The slice tests’ _Server.__exit__ called proc.terminate() — SIGTERM on POSIX, but TerminateProcess on Windows, an immediate kill that runs no handler, so WiredTiger never closed and anything not yet checkpointed was gone. Seven tests failed as a result, including the cross-language hand-off cases where the Python server opened the store and found it empty. The binary already installed a Windows console control handler (the ctrlc crate with termination); it was simply never sent a signal it could catch. It is now spawned with CREATE_NEW_PROCESS_GROUP and stopped with CTRL_BREAK_EVENT, the pattern the Rust binary smoke tests already used. On the Windows dev box that moves the file from 10 failures to 3; the remaining three are a different root cause (a FATAL racing the socket close) and are recorded in tasks/backlog.md.

Added

• Twelve $geoWithin / $centerSphere cases in the differential gate, run both over a 2d index and without one. $centerSphere’s radius is in radians, so anything at or past pi covers the whole sphere and must match every document — which is what the mongo-java-driver’s own fixture asserts with r=4. That test failed once in the 2026-09-21 gauge and passed again on 2026-09-27; it was a flake, but nothing pinned the behaviour, so a real regression would have looked identical to one. Probed against 8.2.11: 0 of 12 divergent.

Changed

• The gate’s mongod is spawned with CREATE_NO_WINDOW on Windows. A console executable with no creation flags allocates its own console, so under -n auto every xdist worker popped a terminal window on the desktop.

Two more ways the driver panels could publish a number they had not measured

Both found while rendering the first Rust-server grid, and both the same shape as the Go and pymongo bugs fixed earlier the same day: an artifact that looks current and isn’t.

Fixed

• An EMPTY artifact passed the freshness check. It tests mtime, and an empty directory keeps a fresh one — so java-results-rust-server/, cleared by a re-run and never refilled, had the newest timestamp of the whole set and rendered a blank panel beside twelve real ones. The report on disk still described the data that had gone. The guard now refuses an artifact that exists but holds no results, naming it.

• The panel showed a rate for a truncated run. #1613 stopped the Go report claiming one; the panel had no such check and published 100.0% — over a run cut short at 476 of 481 tests by a 30-minute DNS hang. passed / ran over the part that finished looks better the more tests went missing, which is what made the number so convincing. GaugeStats.truncated now carries the signal and the panel renders an em-dash instead.

The Go collector detects truncation by the same two signatures the report generator uses: a test that emitted run with no terminal event, or a package reporting fail with no failing test beneath it. Verified against both real Go artifacts — python and rust — which are both flagged.

The published driver panels could only ever show the Python server

Every gauge has written a -rust-server artifact for months, and nothing read them: all thirteen collectors hardcoded the Python filenames. The page’s own prose carried a note saying the Rust numbers were not shown — a documented limitation rather than a fixed one.

Added

• --server python|rust on validation_summary.driver_panels. The Rust grid now renders from the Rust artifacts.

• A mixed-age guard. The grid is published as ONE snapshot with one implied date, so a panel built from a months-old artifact reads as current to everyone who sees it. There was no check at all: the first Rust render would have put a 10 August pymongo-async rate beside twelve numbers measured that morning.

  It tests SPREAD rather than absolute age — a deliberately old but consistent sweep is honest, and it is the mixing that misleads. --allow-stale overrides it for the rare case where a mixed grid is genuinely wanted.

• GAUGE_ARTIFACTS, one map from panel name to artifact filename, so the freshness check reaches the same files the collectors do. A test pins it against the collector registry: a gauge added to one and not the other would be silently exempt from the check, which is the exact failure the guard exists to prevent.

The Python PostgreSQL server on a store the Rust server shares: honest refusals, faithful indexes

The two PostgreSQL servers share one on-disk format. The Python server silently mishandled three things the Rust server stores in it, each a wrong answer rather than an error. Batch 16 makes the Python server either handle them as PostgreSQL does or refuse them with PostgreSQL’s 0A000.

Fixed

• Triggers it cannot run were skipped. The Python server runs only BEFORE INSERT FOR EACH ROW triggers, and it silently ignored every other kind the Rust server stores, so a write bypassed an audit trigger or an enforced invariant.

  • A write that would fire such a trigger is now refused. This covers INSERT, UPDATE, DELETE, TRUNCATE, MERGE, ON CONFLICT, UPDATE FROM and DELETE USING.

  • A multi-event trigger is now read from all of its events, not only the first.

• Partitions read as empty. The Rust server keeps a partitioned table’s rows in the root’s collection, so the Python server read a partition as having no rows and accepted root writes no partition covers.

  • Reading the root is allowed, since every row is there.

  • Anything else touching a partition or writing the root is refused.

• ALTER TABLE erased what the Rust server recorded. A Python catalog rewrite dropped every key it does not model: partition_by, owner, a column’s collation. Those keys now round-trip verbatim.

• CREATE INDEX matches PostgreSQL on all 48 lines of indexes.sql, up from 35:

  • a unique index over colliding rows, and an UPDATE into one, are 23505, not XX000;

  • default names follow PostgreSQL’s <table>_<cols>_idx;

  • an index name collides with table and view names in both directions;

  • dropping a constraint’s index is 2BP01;

  • DROP INDEX a, b works;

  • USING hash and INCLUDE render in pg_indexes.

A terminated connection now hears why

pg_terminate_backend, an idle-session timeout and an idle-in-transaction timeout all end a connection with a FATAL error, and the client is supposed to see that error on its next round trip. It often saw nothing of the kind: on Linux the 57P01 surfaced only afterwards, buried in psycopg’s rollback warning, and on Windows it vanished entirely behind a bare Software caused connection abort (10053) with no SQLSTATE at all.

The error was being sent correctly and then destroyed in transit. After the FATAL the connection loop breaks, process_socket returns, and the socket is dropped — closed outright. The client’s next statement is a write to a closed socket, which draws a TCP RST, and an RST discards whatever the client has not yet read. The FATAL was sitting in exactly that buffer.

Fixed

• The server now ends a connection with a lingering close (crates/vendor/pgwire/src/tokio/server.rs): it shuts the write half down first, sending a FIN so the client reads the error and then a clean EOF, and drains reads afterwards so the socket stays half-open long enough for the client’s in-flight bytes to be consumed rather than reset. Applied to both process_socket and process_socket_unix.

  The drain budget is a ceiling rather than a delay — it ends at the client’s EOF, so an ordinary Terminate-and-hang-up costs nothing measurable. On the Rust PG server’s slice tests the change is a straight improvement in both directions: 229.8s with three failures becomes 95.1s with none, because the connection aborts had been burning time in retries.

• The same tests were also killing the server rather than stopping it on Windows. proc.terminate() is SIGTERM on POSIX but TerminateProcess there — an immediate kill that runs no handler, so WiredTiger never closed and anything not yet checkpointed was lost, which is why the cross-language hand-off tests opened an empty store. The binary already installed a console control handler; it was never sent a signal it could catch.

Four built-in types were missing from pg_type

char, json, pg_lsn and txid_snapshot had no pg_type row, so a client enumerating types never saw them. pgjdbc’s getTypeInfo() is checked against a fixed list of 38 built-in names, and those four were the gap.

char is worth calling out: it is PostgreSQL’s internal one-byte character type (oid 18), a different type from bpchar / character(n) (1042). The names invite conflating them, so the test pins both rows.

Oids, typarray and typlen all measured against PostgreSQL 14 on 2026-09-27.

Added

• pg_type rows for char (18), json (114), pg_lsn (3220) and txid_snapshot (2970).

Noted, not fixed

Probing this surfaced a separate divergence: a json column reports atttypid 3802 (jsonb) rather than 114. The two are distinct types in PostgreSQL — json keeps its text verbatim, jsonb normalises — and this server folds json onto the jsonb tag. The pg_type row added here is correct on its own terms, since PostgreSQL has both; only the column identity is wrong. Filed in tasks/backlog.md rather than fixed, because reporting 114 over jsonb storage would trade one wrong answer for another — the value semantics are the harder half.

Found from pgjdbc’s DatabaseMetaDataTest::types.

The Rust PostgreSQL server can be started and stopped inside an async test

secantus_pgserver::bind and the handle it returns could only be used from a plain thread. Called inside a tokio runtime – a #[tokio::test], which is how a Rust program writes a database test – bind panicked before returning (“Cannot start a runtime from within a runtime”), and so did stopping or dropping the handle (“Cannot drop a runtime in a context where blocking is not allowed”). The Python embedding never hit either, because it calls from a plain thread.

Fixed

• bind no longer blocks on the server’s runtime: the listener is bound with the standard library and then handed to the runtime, which works from any context.

• stop() and Drop run their blocking half – the connection drain, the runtime shutdown and the store’s close-checkpoint – on a thread of their own when called inside a runtime, and wait for it, so they still return only once the store is closed.

• New tests start, write, drop, reopen and read back inside both a current-thread and a multi-thread #[tokio::test].

The stale-artifact check now covers probes, gauges and benchmarks — not just pytest

PR #1586 added a collection-time check that refuses to run when a built artifact was made from a different crates/ tree than the checkout. It works, and it only ever ran under pytest.

That left the gap where it matters most. This repo’s primary bug-finding method is not the test suite — CLAUDE.md says to run behaviour against the reference server rather than reason about the source — and an ad-hoc probe, a driver gauge and a benchmark all launch these artifacts directly, with nothing checking them. On 2026-09-28 a probe of secantusd-pg duly ran against a binary built from a different tree, and the only thing that caught it was a hand-read of --version: exactly the state #1586’s own docstring complains about, where “the evidence was sitting in --version output that no automated thing read”.

Changed

• The comparison, the rebuild commands and the per-artifact cost strings moved from tests/conftest.py into tools/provenance.py, importable by anything that runs with the repo root on sys.path. conftest.py is now one caller rather than the owner; its behaviour is unchanged.

• Both probe launch paths check before starting a server: tools/probes/_servers.py (the embedded _secantus_server) and tools/probes/pg_differential.py (secantusd-pg). A stale artifact aborts the probe instead of producing divergences that are the artifact’s age rather than the server’s behaviour.

• The Windows .exe fallback moved into the shared resolve_binary, so the fourth caller cannot forget it. Getting it wrong once already made the guard watch a filename that cannot exist on Windows, and made 1,194 pgserver tests skip.

Added

• Eight tests in tests/test_build_provenance.py, including two that pin the gap this closes: each probe launcher must call the check, and tools.provenance must import with nothing but the repo root on the path — no pytest, no conftest.

Note

conftest.py discovers tools/provenance.py by walking up from its own location and abstains when it is not there, rather than importing it unconditionally. A verbatim copy of conftest.py is loaded from a temp directory by tests/test_crash_stall_watchdog.py (deliberately, so the nested session exercises the real watchdog), where no checkout sits above it — a hard module-scope import failed to LOAD the conftest there, which is not one test failing but every test in the lane. Pinned by test_a_copied_conftest_still_loads.

The stale-artifact check reaches the gauges and benchmarks, and the lint gate reaches the whole repo

Two gaps left open by the previous change, both of the same shape: a guard that existed but only covered part of what it was meant to.

Changed

• Every launcher checks provenance, not just pytest and the probes. Six more sites: gauge_common.rust_binary() (all thirteen MongoDB gauges), psycopg_validation/runner.py, and the bench/ harnesses behind the published concurrency chart, the latency-vs-mongod table and the two PostgreSQL cost benchmarks. These are the paths that PUBLISH a number — on 2026-09-18 the psycopg gauge was measured against a checkout 116 crate-commits behind, reported 73.8% where the truth was ~99.98%, and that figure reached the live website.

• SECANTUS_ALLOW_STALE_ARTIFACT=1 overrides the check, loudly. Measuring an old build on purpose — a bisect, a before/after against a previous release — is legitimate, and without a supported way to say so the person who needs it comments the check out, which removes it for everybody. The override still prints the mismatch, so a stale run cannot look like a clean one in a log.

• CI lints . instead of src tests. That scope left tools/, bench/, all nineteen gauge runners, the invoke tasks and website/ unchecked. A named path list was the first fix and was wrong the same way — an enumeration grows a hole the moment a directory is added, and the test written against it immediately found 27 locations missing. ruff already skips gitignored trees and the vendored submodules, so . reaches our Python and nothing else.

Fixed

• A dict key written twice in pgtest_validation/include_paths.py, silently discarding one session’s findings — Python keeps only the last. Two sessions had each documented the procedure stanza; only one was visible. Both are now merged into a single entry, including the detail they disagree on (the offending line is cited as procedure:66 by one and procedure:68 by the other), rather than one being picked. This was invisible until ruff was pointed at that directory.

• 44 further lint findings across the newly-covered directories: unused imports, import ordering, long lines, a zip() without strict=, a try/except/pass.

Added

• Twelve tests: one per launcher asserting it calls the check, two for the override’s behaviour, and two pinning the lint gate’s scope.

Note

[tool.ruff] extend-exclude now names vendor and crates/vendor. Linting . reaches every checked-out submodule, so without it the gate reports upstream’s style as our failure — CI failed on a WiredTiger analytics notebook. It passed locally because a fresh worktree has no submodules checked out, so . reached nothing vendored and the exclusion looked unnecessary; the difference was the environment, not the config. Pinned by test_ruff_excludes_every_vendored_tree, which fails if a newly-added submodule is not covered.

The Rust MongoDB server publishes to crates.io on release

A secantusdb-v* release tag now also publishes the Rust MongoDB server’s crates to crates.io, with no stored token.

Added

• .github/workflows/publish-crates.yml:

  • It refuses a tag that disagrees with the crate version.

  • It packages the ten crates and builds each one from its tarball.

  • It publishes them in dependency order with cargo publish --workspace.

  • It authenticates by crates.io trusted publishing, in a crates-io environment that accepts only release tags.

  • A manual run defaults to a dry run.

Four wrong answers the Python PostgreSQL server gave about arrays

The Rust PostgreSQL server’s array work last release left a corpus that could be pointed at the Python server too. It was not flattering: 18 divergences out of 32 against PostgreSQL 14.13, where the Rust server had none. Four of them were wrong answers rather than refusals, and one of those was wrong in two independent places.

ARRAY[1,NULL] @> ARRAY[NULL] answered true. So did &&, and ARRAY[1,NULL] <@ ARRAY[1,NULL]. PostgreSQL says false to all three: the containment operators use the element type’s =, under which a NULL matches nothing — not even another NULL. Python’s None == None is True, and that stood in for the SQL semantic. In a WHERE clause it returned rows PostgreSQL excludes.

The same predicate had a second copy of the bug in the index path, which had to be fixed separately: field @> ARRAY[...] lowers to a Mongo bare-equality filter so it can use a multikey index, and a bare equality against null matches a missing field as well. Fixing the evaluator alone would have left the indexed form of the same query still wrong.

Separately, array_position(arr, elem, start) ignored its third argument entirely, so array_position(ARRAY[1,2,3,2], 2, 3) answered 2 where PostgreSQL answers 4 — a wrong number, which is why nothing had caught it. sqlglot files that argument under a slot named zero_based, which it reuses; the obvious place to look for it holds nothing at all.

And a multidimensional subscript — m[1], m[1:2][2] — leaked a bare Python ValueError to the wire with no SQLSTATE at all, reachable from a plain SELECT.

Fixed

• @> / <@ / && no longer match a NULL to a NULL, in both the per-row evaluator and the index pushdown.

• @> / <@ / && now flatten both operands, so ARRAY[[1,2],[3,4]] @> ARRAY[3] is true as PostgreSQL has it; previously containment was wrong for every multidimensional array.

• A NULL array operand to any of the three is now NULL rather than 42883 function array_contains_all() does not exist — an internal fallthrough re-labelled as a missing function the user never called.

• array_position(arr, elem, start) honours start, and a NULL start is PostgreSQL’s 22004 initial position must not be null.

• array_cat(NULL, NULL) is NULL, not the empty array.

• array_length(arr, NULL) is NULL rather than a 42883 naming array_size(...).

• string_to_array(s, '') keeps the whole string as one element rather than raising, and string_to_array('', sep) is the empty array rather than {''}.

• array_positions is declared integer[], which is what PostgreSQL reports, not bigint[].

• A subscript chain shorter than the array’s dimensionality is NULL — (ARRAY[[1,2],[3,4]])[1], which used to return the inner row and then fail to coerce it, leaking a Python ValueError with no SQLSTATE.

• Once any subscript in a chain is a slice, a bare index means “1 to n” as PostgreSQL defines it, so m[1:2][2] is the whole second dimension. The result type is now taken over the whole chain, so a nested slice reports the array type rather than the element type.

Added

• UPDATE t SET a[i] = v and SET a[lo:hi] = v, matching the Rust server: a subscript past the end extends the array with NULLs, assigning into a NULL column builds it from nothing, and two assignments to one column in one statement compose rather than racing.

• A subscript below 1 is refused by name. PostgreSQL answers it by moving the array’s lower bound, which neither server models; writing it at 1 instead would silently shift every other subscript into the array.

Security

• SET a[1000000000] = 1 is one statement and would have allocated every slot it named. PostgreSQL caps an array at 134217727 elements and says so; the cap is now checked before allocation.

Python PostgreSQL server: constraint triggers and INSTEAD OF triggers

CREATE CONSTRAINT TRIGGER now runs on the Python PG server. Each behaviour below was checked against PostgreSQL 15.

Added

• A DEFERRABLE INITIALLY DEFERRED constraint trigger queues its row events for COMMIT. If the trigger raises there, the whole block rolls back.

• SET CONSTRAINTS name | ALL IMMEDIATE runs the queued events and switches the trigger to fire at once. Setting a non-deferrable constraint trigger answers 42809.

• Outside a transaction block, a deferred constraint trigger runs at the end of the statement, after the immediate triggers.

• CREATE CONSTRAINT TRIGGER rejects a BEFORE trigger, a statement-level trigger, and NOT DEFERRABLE INITIALLY DEFERRED with PostgreSQL’s 42601.

• INSTEAD OF row triggers on views. Each row an INSERT, UPDATE or DELETE names goes to the trigger instead of the base table. A NULL return skips the row, the command tag counts the rest, and RETURNING projects what the trigger returned.

• CREATE TRIGGER rejects the shapes PostgreSQL rejects for views and tables, with the same SQLSTATE, message and detail. Dropping a view drops its triggers.

Python PostgreSQL server: date/time input follows PostgreSQL’s DecodeDateTime

Date, timestamp and timestamptz text input on the Python PG server now goes through a port of PostgreSQL’s ParseDateTime / DecodeDateTime (src/secantus/sql/dtparse.py). Corpus dt_input went from 117 divergences of 212 to 0 against PostgreSQL. wide_timestamptz, time_input and tz_abbrevs each improved, and no corpus got worse.

Fixed

• Date and time input now accepts everything PostgreSQL accepts:

  • month and weekday names, in any order (Jan 5 2020 10:30 PM, Sat Jan 05 10:30:00 2020 CET);

  • two-digit years (1/5/69 is 2069, 1/5/70 is 1970);

  • Julian days (J2458854), year.doy (2020.005) and y2020m01d05;

  • AD / BC, 24:00:00 rollover;

  • time-zone abbreviations, IANA zone names and numeric offsets.

• Errors split as PostgreSQL’s do: 22007 for bad syntax, 22008 for an out-of-range field, 22009 for an invalid zone. A timestamptz error names the type timestamp with time zone.

• A time in a daylight-saving gap or overlap resolves to the same offset PostgreSQL picks.

• AT TIME ZONE accepts interval zones, abbreviations and POSIX offsets (+05, utc+3, -03:30). timezone(zone, ts) is added.

• A timestamptz outside Python’s year range renders in the session zone with PostgreSQL’s offset spelling (+00, -12).

Python PostgreSQL server: full-text search follows PostgreSQL

The Python PG server’s text search was a simplified stand-in. It is now a port of PostgreSQL’s own (tsvector / tsquery I/O, the document parser, ranking, headlines, phrase search). Corpora fts, fts2 and fts_langs went from 106 divergences to 0 against PostgreSQL, and no corpus got worse.

Added

• A new fts extra: pip install "SecantusDB[fts]" installs snowballstemmer==2.2.0, the Snowball release PostgreSQL 15 ships. With it, the 26 non-English text search configurations stem and drop stop words exactly as PostgreSQL does. Without it they refuse with 0A000 rather than answering differently.

• setweight (two and three arguments), ts_delete, ts_filter, tsquery_phrase and get_current_ts_config are implemented.

Fixed

• tsvector and tsquery input and output follow PostgreSQL’s rules: ordering, positions, weights, quoting, and the exact error codes and messages. 'x'::tsquery is no longer stemmed.

• The document parser recognises emails, hosts, URLs, numbers, versions and hyphenated words. The English stop-word list is complete.

• ts_rank / ts_rank_cd match PostgreSQL’s arithmetic and return real.

• Operators: !!tsquery, tsquery <-> tsquery, tsvector @@ 'literal' and text @@ text now work as PostgreSQL defines them.

• An unknown text search configuration answers 42704.

Python PostgreSQL server: the range and multirange operators

The range_ops corpus went from 260 divergences of 771 to 0 against PostgreSQL 15.

Fixed

• Range ordering (<, <=, >, >=) follows PostgreSQL’s range order: empty first, then lower bound, then upper bound. It raised 42883 for an unbounded or empty range.

• <<, >>, &< and &> work on ranges and multiranges. &< / &> answered 0A000, and << / >> fell through to the bit-shift path.

• Multirange +, * and - are implemented, and pg_typeof reports the multirange type.

• A non-contiguous range union or difference answers 22000, where it raised an internal error.

• A multirange against a plain range answers PostgreSQL’s 42883 operator does not exist: int4multirange + int4range.

• daterange @> date no longer raises an internal error.

Python PostgreSQL server: fixes from a full PostgreSQL corpus sweep

Running every corpus in tools/probes/pg_corpora/ against the Python PG server turned up these divergences from PostgreSQL 15. They are fixed, and the sweep total drops from 2,985 to 2,955 with no corpus getting worse.

Fixed

• A join whose alias equals a column of the FROM table (FROM a o JOIN a n) returned the whole joined row for o.n, silently. It now returns the value.

• IS [NOT] DISTINCT FROM works in WHERE, JOIN ... ON, UPDATE and DELETE. It answered 0A000 there before.

• string_agg(<expression>, ...) over a join now works. It answered 0A000 expected a column.

• CREATE OR REPLACE VIEW refuses to drop, rename or retype a view column with 42P16, as PostgreSQL does. It used to replace the view silently.

• Dates with a month name ('Jan 5, 2020', '5 Jan 2020', '2020-Jan-05') are accepted.

• A date with a day past its month’s end ('2020-02-30', 'Feb 29 2021') answers 22008, where it answered 22007.

• SELECT t FROM t describes its column with the table’s row type rather than generic record, so a client reads it as it does from PostgreSQL.

• pg_trigger lists triggers (it was always empty), including the deferrability of constraint triggers.

• SET CONSTRAINTS naming no constraint answers 42704.

• Transition tables with one name for OLD and NEW answer 42P17.

Python PostgreSQL server: trigger transition tables, and three string_agg fixes

AFTER triggers can now declare REFERENCING OLD TABLE AS ... NEW TABLE AS ... and query the statement’s affected rows as a relation. Each change was checked against PostgreSQL 15.

Added

• Transition tables on every write path: INSERT, UPDATE, DELETE, ON CONFLICT, UPDATE FROM, DELETE USING and MERGE. They are available to both statement-level and row-level AFTER triggers.

• CREATE TRIGGER rejects the shapes PostgreSQL rejects, with the same SQLSTATE and message: a BEFORE trigger, more than one event, a column list, the wrong OLD/NEW side for the event, and TRUNCATE.

Fixed

• A string_agg nested in an expression no longer drops its in-call ORDER BY. For example, coalesce(string_agg(v, ',' ORDER BY id DESC), '') came back in insertion order with no error.

• A string_agg nested in an expression over a JOIN no longer fails with 0A000 unsupported aggregate.

• || with a non-text operand inside an aggregate (string_agg(id || '=' || v, ',')) now casts the operand to text. It used to fail with $concat only supports strings.

Python PostgreSQL server: trigger arguments, UPDATE OF and TRUNCATE triggers

The Python PG server now runs three trigger shapes it used to refuse, each checked against PostgreSQL 15’s output.

Added

• Trigger arguments: EXECUTE FUNCTION f('x', 7) is stored as text and exposed to PL/pgSQL as TG_ARGV (subscripted from 0) and TG_NARGS.

• UPDATE OF col, ... triggers. One fires when any listed column is a target of the SET list, whether or not its value changes.

• BEFORE / AFTER TRUNCATE statement triggers. A truncate that fires one runs as a single transaction, so a trigger that raises leaves the rows in place.

The Python PostgreSQL server runs AFTER, statement-level, UPDATE and DELETE triggers

Until now the Python PostgreSQL server ran only BEFORE INSERT FOR EACH ROW triggers. It now runs the trigger kinds most applications use, as PostgreSQL does, and refuses with 0A000 the ones it still cannot run.

Added

• CREATE TRIGGER:

  • BEFORE and AFTER, FOR EACH ROW and FOR EACH STATEMENT (the default), on INSERT, UPDATE and DELETE, several events joined with OR;

  • WHEN (...) conditions over NEW / OLD.

• Inside a trigger function:

  • TG_OP, TG_WHEN, TG_LEVEL, TG_NAME and TG_TABLE_NAME;

  • OLD / NEW are NULL where the event has none, and a field of a NULL record is NULL;

  • SQL statements in the body can read new.x / old.x, so an audit trigger can insert them into another table.

• Firing:

  • a BEFORE ROW trigger can change NEW, or skip the row with NULL;

  • statement-level triggers fire even when no row matches.

• Triggers the Rust server stores in the shared catalog fire here too.

Fixed

• A write whose trigger wrote, then failed, left the trigger’s writes committed. A single INSERT / UPDATE / DELETE / MERGE now runs as one transaction whenever the database has a trigger, as on PostgreSQL.

Still refused, with 0A000:

• UPDATE OF column lists, trigger arguments, transition tables (REFERENCING), constraint triggers, INSTEAD OF and TRUNCATE;

• a trigger on a path that fires none: ON CONFLICT, MERGE, UPDATE ... FROM and DELETE ... USING.

The firing order and logged values are checked against PostgreSQL 15’s output for the same statements.

Python PostgreSQL server: triggers fire on ON CONFLICT, UPDATE FROM, DELETE USING and MERGE

These write paths used to refuse any table with a trigger. They now fire triggers in PostgreSQL’s order, checked against PostgreSQL 15.

Added

• INSERT ... ON CONFLICT fires BEFORE INSERT ROW for every proposed row. EXCLUDED sees the row as that trigger returned it. A conflicting DO UPDATE then fires BEFORE UPDATE ROW.

• UPDATE ... FROM and DELETE ... USING fire their statement and row triggers, including UPDATE OF.

• MERGE fires BEFORE STATEMENT for each action its WHEN clauses name, row triggers per action, and the AFTER STATEMENT triggers in reverse.

• On every path, AFTER ROW events queue to the end of the statement. A BEFORE ROW trigger can change the row or skip it.

Python PostgreSQL server: statement-level triggers on views

BEFORE / AFTER ... FOR EACH STATEMENT triggers on a view are now accepted, and they fire in PostgreSQL’s order: around the view’s INSTEAD OF rows, and not for a write through an automatically-updatable view, which fires the base table’s triggers instead. Checked against PostgreSQL 15. This closes the last trigger shape the Python server refused.

The Python server reads date strings with mongod’s own parser

The Python MongoDB server now parses date strings the way mongod 8.2.11 does, because it runs the same parser: a pure-Python port of timelib 2022.13 – the library mongod vendors – together with mongod’s own wrapper around it. That covers $dateFromString with and without a format, $toDate, and $convert to a date. Before this, the Python server used Python’s ISO parser and strptime, and on mongod’s differential probe of 433 date strings it answered 323 differently: wrong values (every 12-hour time, every zone abbreviation), strings mongod accepts refused, and error text that matched nothing mongod says. It now matches on all 433, values and full error text.

Fixed

• secantus/timelib/ (new): a literal port of timelib’s free-form scanner (parse_date.re, re2c longest-match semantics), its parse-from-format under mongod’s kDateFromStringFormatMap, timelib_update_ts, and the 1,127-entry zone abbreviation table (generated from mongod’s timezonemap.h by tools/timelib/gen_zone_tables.py --python). Pure Python, no Rust in the request path.

• $dateFromString: evaluated in mongod’s order – format type (40684) and validity (18535 / 18536) first, then timezone (40517 / 40485), then onNull, then onError around the dateString type check and the parse (241, now “found: int with value 5”). %j is zero-based as in mongod; %y is refused (not a mongod specifier); a lone %Y / %j is incomplete; a zone in the string together with a timezone argument is refused; a DST gap or overlap resolves the way mongod’s does; a present but null format gives null; and a literal bad format is reported before a bad literal timezone.

• $toDate / $convert of a string: every shape timelib accepts (relative forms, @ timestamps, ISO weeks, month names, zone abbreviations, 12-hour times) with mongod’s value, and mongod’s exact error text otherwise. Dates outside years 1-9999 are returned as raw BSON dates rather than refused; one beyond about +-292,000 years is mongod’s 159 DurationOverflow.

replSetStepDown, and a hello that waits when it is asked to

SecantusDB advertises itself as a single-node replica-set primary and already answered replSetGetStatus with a full status, but replSetStepDown was CommandNotFound — so a driver could read the topology and not act on it. It is implemented now on both servers, reproducing what a real single-node replica set does: the command returns immediately, the node reports itself a secondary for the requested period with primary and electionId dropped from hello, writes are refused 10107 "not primary" while reads keep working, and then it is primary again.

Two halves of that were not guessable from the command’s description. The refusal has to carry topologyVersion, or the driver marks the server Unknown and the very next read fails server selection. And the monitoring hello stream has to push the change the moment it happens rather than waiting out maxAwaitTimeMS, or the driver does not learn about it until its next heartbeat and the refusal then looks newer than its whole view of the server.

Alongside it, a plain awaitable hello — topologyVersion plus maxAwaitTimeMS, without exhaustAllowed — now holds its reply for the requested budget instead of answering in a fraction of a millisecond. The streaming exhaustAllowed form was already implemented; this is the other half of the same protocol, and a driver that polls it was spinning.

Election timing is deliberately not reproduced: mongod’s return to primary is driven by its election machinery rather than the step-down period alone, and modelling that means modelling election timeouts — the multi-node machinery this project puts out of scope. The window here is exactly the period requested.

Added

• replSetStepDown on both servers, with mongod’s refusals: 2 BadValue for a period under secondaryCatchUpPeriodSecs or a negative one, 262 when a non-forced step-down has no electable secondary to hand over to, 10107 when already a secondary, and 76 NoReplicationEnabled on a server not advertising a set.

• hello reports secondary, which mongod always includes in a replica-set reply and SecantusDB never emitted.

Fixed

• A plain awaitable hello holds its reply for maxAwaitTimeMS rather than returning at once.

• maxAwaitTimeMS without a topologyVersion is refused 31368, as mongod refuses it; both servers used to accept it.

• topologyVersion.counter advances when the topology changes. It was pinned at 0 on the grounds that the topology never changes, which stopped being true here — and a frozen counter makes a step-down look stale to a driver.

Change events and error replies on the Rust MongoDB server match mongod

Three new probes compare the Rust MongoDB server with mongod 8.2.11: random write sequences watched through a change stream, every update operator’s updateDescription, and the full reply to 58 real failures. Everything they found is fixed, including several writes that stored nothing or the wrong thing.

Fixed

• $set through an array index past the end (b.0.c on []) creates the element instead of silently writing nothing.

• A positional update whose query names the array itself ({a: 2} with $set: {"a.$": 9}) works; it was refused.

• updateMany keeps the documents it updated before a failure, as mongod does.

• multi with a replacement document, a delete limit other than 0 or 1, a duplicate index under a new name, an index with an empty key or no name, and collection names mongod refuses are now refused instead of carried out.

• Change events: no fullDocumentBeforeChange on inserts; updateDescription reports what each update operator touched, as mongod does; pipeline updates use mongod’s own diff and become replace events when mongod’s would.

• Error replies: duplicate-key messages in mongod’s form with keyPattern and keyValue, and mongod’s code and message for unknown operators, invalid regexes, unmatched positional updates, unused array filters, index and collection DDL conflicts, bad sort specifications, negative skip / limit, aggregate without cursor, and renameCollection outside admin.

Added

• tools/probes/change_stream_fuzz.py, tools/probes/update_description.py and tools/probes/write_error_replies.py.

$dateFromString with a format now parses the way mongod does, on the Rust server

$dateFromString with a format on the Rust MongoDB server is now a literal port of the parser mongod uses: timelib’s parse-from-format, run under mongod’s own table of format specifiers and its % prefix. Before, it used a hand-written strptime. That refused most of what mongod accepts (%L milliseconds, ISO weeks, %z zones, and every malformed string) and got some answers wrong. %j came out a day early, because mongod counts the day of the year from zero, and a lone %Y returned a date where mongod reports the string as incomplete.

Against mongod 8.2.11, 433 date-string cases (now including 115 with a format) give 0 divergences, in both the values and the full error text.

Fixed

• $dateFromString with a format on the Rust server: %L, %G / %V / %u, %z, %Z (an offset in minutes), %b / %B and %% all parse. %j is zero-based. Invalid dates and times, trailing data, missing data and literal mismatches return mongod’s 241 message, with timelib’s positions and characters.

• A format with an unknown specifier or a trailing % now returns mongod’s 18536 / 18535. A non-string format returns 40684, and a non-string timezone returns 40517. These checks run in mongod’s order, so a bad format is reported even when dateString is null.

• format and timezone are evaluated as expressions, so "$field" references work.

• A non-string dateString is a 241 that onError catches, as on mongod. The Rust server used to refuse it.

Rust MongoDB server: 12-hour times and zone names in date strings

$toDate and $dateFromString returned wrong times on the Rust MongoDB server for every 12-hour time and every zone abbreviation, with no error. They now match mongod 8.2.11 on all 123 probed strings; 61 differed before.

Fixed

• A 12-hour time came back hours out, often on another day. "2024-01-01 10:00 PM" was 01:00, and "12:00 AM" was 23:00 the previous day. Times are now read as mongod reads them: hours 1–12, with am/pm or a.m./p.m..

• A zone abbreviation came back hours out: "… 22:30 UTC" was twelve hours off, and GMT, EST and PST were wrong too. There are 51 abbreviations, each with the offset mongod gives it; some are timelib’s historical values, such as IST +2. Abbreviations mongod does not know are refused.

• Second 60 ("23:59:60") is refused, as mongod refuses it. It used to roll over into the next day.

Added

• These date-string forms now parse as on mongod:

  • an offset after a space ("22:30 +02:00");

  • a time before the date;

  • a month-name date followed by a time;

  • ctime order ("Jan 1 10:00:00 2024");

  • month and year ("Jan 2024", "Sept 2024");

  • a leading weekday, which moves the date forward to that weekday, as mongod does.

• tools/probes/date_string_parsing.py, a 123-string probe that compares against mongod.

A hint on a collection that does not exist was rejected

mongod validates a hint during query planning, and there is nothing to plan against when the namespace does not exist — so it accepts any hint there and returns an empty result. This server refused it, on every command that takes a hint.

Measured against mongod 8.2.11 (2026-09-28), across three collection states:

collection	mongod	before
does not exist	ok	BadValue (2)
exists, empty	BadValue (2)	BadValue (2)
has documents	BadValue (2)	BadValue (2)

Fixed

• Two sites, because reads and writes validate in different places: the storage layer’s resolve_hint covers find / count / aggregate / distinct, and validate_write_hint covers findAndModify / update / delete. Fixing only the first left the three write commands still diverging, which is the sort of half-fix a single-command test would have missed.

Found by

mongo-c-driver’s /find_and_modify/hint, which runs against a collection it never creates. One failing driver test; seven diverging commands once the siblings were probed.

Also

• The C gauge now spawns its daemon with --standalone, as the Java gauge already did. libmongoc’s MONGOC_TEST_URI carries no replicaSet=, so its tests assert standalone semantics — four /Client/select_server* tests select with a SECONDARY read preference and assert standalone_or_rs_secondary_or_mongos, which our single-node replica-set persona answered with RSPrimary. With the flag they pass; without it they fail.

The Rust server ignored maxTimeMS

It was parsed and validated exactly as mongod validates it — every type error, every bound — and then never checked. The operation ran to completion and answered ok. Measured against mongod 8.2.11: a createIndexes over 100,000 documents with maxTimeMS: 1 returned ok: 1.0 here and code 50 MaxTimeMSExpired there.

That is invisible on a fast operation, which is why validating the argument so carefully never exposed it.

Added

• secantus_core::deadline — a thread-local budget armed by run_handler around the whole handler, because that is the span mongod bounds: the operation, not any one loop inside it. Mirrors src/secantus/deadline.py deliberately, so the two servers overrun in the same places for the same reasons.

• Polling in the loops whose length is driven by the data: the document scan, the two predicate passes behind find and count, and the three index-build loops. Every 64 documents, which bounds the overrun to 64 rows rather than the whole table.

  getMore is excluded, as on the Python server — there maxTimeMS is the awaitData wait, not a limit, and arming a deadline would make every tailable poll report a timeout.

Measured

• Against mongod 8.2.11: 0 divergent of 4 — createIndexes, find + sort and count all answer code 50 where they previously answered ok, and an operation with no budget is untouched.

• Cost on the unarmed fast path: none detectable. A 100,000-document COLLSCAN with no maxTimeMS: 34.9 ms before, 34.7 ms after (−0.7% on the min of seven). The plan warned these polls sit in the flagship’s hottest loops and said not to assume the cost, so it was measured rather than asserted.

The embedded Rust server expires TTL documents

secantus_mdb::Server, the MongoDB server you start from a Rust test, now runs the same background sweepers as the secantusd-rs daemon. A TTL index expires documents every 60 seconds, as on mongod. Before this, documents in an embedded server never expired. The noop oplog heartbeat, which keeps a quiet change stream’s resume token advancing, is available and off by default, as on the daemon.

Added

• Builder::ttl_sweep(Option<Duration>) sets the TTL sweep interval (default 60 seconds, None to disable). Set a shorter one in a test that waits for an expiry.

• Builder::noop_heartbeat(Option<Duration>) sets the noop oplog heartbeat, which also prunes the oplog (default off).

The Rust MongoDB server labels injected errors the way mongod does

A driver decides whether to retry a write, resume a change stream or replay a transaction from the errorLabels on an error, not from the code alone. Errors injected with failCommand – which is how every driver’s retry and resume tests work – carried a label in only one of the places mongod puts one. Every code from 1 to 520 was swept against mongod 8.2.11 and the Rust server now answers each one the same way.

Fixed

• ChangeStreamFatalError (280) and ChangeStreamHistoryLost (286) carry NonResumableChangeStreamError on every command, a plain cursor’s getMore included (the PHP driver’s bug1419-001.phpt).

• The aggregate that opens a change stream carries ResumableChangeStreamError for mongod’s 28 resumable codes; a retryable write carries RetryableWriteError for its 23 codes, which it never did; the transaction and commit label sets gain the codes they were missing; and four overload codes add SystemOverloadedError.

• failCommand refuses a code whose error needs extra information (duplicate key, stale config and 22 more) with mongod’s 40671, and drops the connection on the four codes where mongod does.

• An injected error reports mongod’s real codeName for 452 codes that used to come back as Location<code>.

• An oversized multi-document transaction now fails with TransactionTooLargeForCache’s real code, 388. It said 313, which on mongod is a different error.

• The C++ driver gauge started its server without test commands, so every failpoint test in it failed on the harness.

Added

• tools/probes/error_labels.py: code, code name and labels for injected errors across nine command contexts, compared against mongod.

The Rust MongoDB server stops losing rows through its indexes, and matches mongod on twenty more shapes

A sweep of every differential probe against mongod 8.2.11 found three ways an index made the Rust server return fewer documents than it should. A sort over a partial compound index, a sort or hint over a multikey index holding an empty array, and an index range scan bounded by an array or a document each dropped rows silently. All three are fixed, and the probe that should have caught them now does.

The same sweep turned up a run of wrong answers and missing pieces: sorting embedded documents by their encoded length instead of their value, $project: {_id: 1} returning whole documents, $slice / $indexOf* / $range argument handling, maxTimeMS not interrupting aggregations, and an aggregate result over 16 MB sent as a message too large for any driver to accept. Each was measured against mongod, fixed, and pinned by a probe that now reports zero divergences.

Fixed

• Silent data loss through indexes (Rust server). find({}).sort({a: 1, b: 1}) over a compound index partial on a returned only the rows the index holds; a sort or a hint over a multikey index dropped every document whose field is an empty array (count({}, hint: ...) too); and {x: {$gt: [1, 2, 3]}} with an index on x dropped {x: [9]}.

• Sort order of embedded documents and nested arrays. find().sort() compared raw-BSON sort keys, which lead with the value’s length, so {a: 2, b: [3]} sorted above {a: 5}. Documents and arrays now compare by value, as the aggregation $sort always did.

• $project: {_id: 1} alone is an inclusion projection.

• $slice accepts a whole Decimal128, refuses a count outside int32 (28726 / 28728) or not positive (28729), and counts a negative start from the right place; $substrCP and $indexOfCP / $indexOfBytes / $indexOfArray enforce int32 and accept whole decimals; $range is bounded by mongod’s 100 MiB memory estimate (146) instead of refusing past 100,000 elements.

• maxTimeMS interrupts aggregation stages, distinct, sorted find and index walks; read commands report an execution-time expiry under mongod’s executor prefix, and update / delete fail the command rather than reporting a per-statement write error.

• An aggregate result over 16 MB answers 10334 BSONObjectTooLarge instead of an 84 MB reply the driver rejects; decoded first batches respect the 16 MB budget.

• Regex $ and \Z match before a final newline, as PCRE does; \Z was refused.

• $jsonSchema refuses type: "integer" (9) and unknown type names (2).

• $bucketAuto POWERSOF2 answers an int for a whole power of two.

• A writeConcernError carries mongod’s errInfo.writeConcern (w, j, wtimeout, provenance) and sits before ok; a write whose w names an unknown tag runs and then reports 79, instead of being refused.

• drop reports the collection’s real index count as nIndexesWas.

• SASL errors match mongod (Authentication failed., 334 for an unknown mechanism, 17 with no conversation), and usersInfo {forAllDBs: true} lists every user instead of none.

• The streaming hello a driver’s server monitor sends is held as mongod holds it. A monitor that already had the current topology got its first streamed reply at once, one extra heartbeat per stream: the Go driver’s heartbeats_processed_more_frequently test counted 12 messages against a limit of 10. A monitor with an out-of-date topology is now answered at once instead of held, returning to primary after replSetStepDown moves topologyVersion so the monitor hears about it immediately, and a malformed topologyVersion or maxAwaitTimeMS gets mongod’s error instead of being accepted.

• A document that fails its validator gets mongod’s full explanation in errInfo.details: every broken $jsonSchema rule in mongod’s order, and every failing query clause, where it used to get {operatorName: "$jsonSchema"}.

Added

• SCRAM-SHA-1 on the Rust server, created for every user by default next to SCRAM-SHA-256 as mongod 8.2 does; hello’s saslSupportedMechs names the user’s own mechanisms.

• The localhost exception: a fresh --auth Rust server lets a loopback connection create the first user, as mongod does. It refused every createUser, so it could never be given one.

• Decimal128 operands for $pow, $atan2 and $bucketAuto granularity, correctly rounded and with mongod’s special values and quanta.

• A multi-field filter rides a single-field index on one of its fields, and a sort under an unindexed filter walks the sort index – the plans mongod picks.

• Probes: max_time_expiry.py, int32_arguments.py, nested_value_sort.py, bucket_auto_granularity.py, validation_error_details.py, scram_auth.py, awaitable_hello.py; index_result_sets.py now covers empty filters, compound sorts, hints, partial compound indexes and array / document bounds.

Changed

• Differential probes stop their embedded server before deleting its store; three of them ended in a WT_PANIC at exit.

• The driver gauges pass --noop-heartbeat-seconds and the other tuning flags through to the Rust server, which accepts them all.

ALTER TABLE on the Rust PostgreSQL server

ALTER TABLE did not exist in any form — AlterTableStmt is not supported yet — which put every migration tool out of reach. ALTER TABLE ... RENAME was a separate refusal (RenameStmt), because PostgreSQL’s parser puts it in a different node.

Both work now. Against a live PostgreSQL 14.13 the DDL corpus went from 28 divergences out of 41 to 7, and every one of the seven left is CREATE INDEX or CREATE VIEW — a different slice. A second corpus of 85 lines written for this change, covering the shapes the first one never reached, is clean.

What works: ADD COLUMN (with DEFAULT, NOT NULL, IF NOT EXISTS), DROP COLUMN, ALTER COLUMN SET/DROP DEFAULT, SET/DROP NOT NULL, ALTER COLUMN TYPE, ADD CONSTRAINT ... CHECK, DROP CONSTRAINT, RENAME COLUMN and RENAME TO — several actions in one statement, applied in order, so add column m int, alter column m set default 5 works.

Three decisions worth stating, because each is a place where a plausible implementation is silently wrong:

The rows are rewritten, not left short a field. ADD COLUMN fills every existing row and DROP COLUMN removes the field. Leaving the field behind would be invisible while the catalog no longer named it — and then adding a column of the same name later would resurrect the old values, a wrong answer no error would flag. It makes an ALTER O(table) where PostgreSQL can often avoid the rewrite; that is the right trade here, where tables are fixtures.

ALTER COLUMN TYPE follows PostgreSQL’s cast rule, which is about the TYPES and not the values. text -> int is refused with 42804 even when every value would convert cleanly; it needs a USING clause. Casting per row instead made the same statement succeed or fail depending on the data, which is not what PostgreSQL does either way. The rule was measured across 31 type pairs on 14.24.

An ALTER is validated before a row is touched. Every action is checked against the catalog first, so a second action failing cannot leave the table in a shape neither the old nor the new catalog describes. SET NOT NULL over a column that has NULLs, and ADD CONSTRAINT CHECK that existing rows fail, are both refused with PostgreSQL’s own SQLSTATE.

Added

• secantus-pgplan / secantus-pgserver: ALTER TABLE and ALTER TABLE ... RENAME, in the forms above. An action outside them is refused by its own name (ALTER TABLE OWNER TO, ... VALIDATE CONSTRAINT) rather than under one catch-all.

• Savepoint capture for the new statements, so a ROLLBACK TO puts back the rewritten ROWS as well as the catalog.

Fixed

• secantus-pgplan: CREATE TABLE t (id int, id int) was accepted silently, producing a table whose second column was unreachable because every lookup resolves a name to the first match. It is PostgreSQL’s 42701 now.

The Rust PostgreSQL server learns arrays

Arrays were the largest hole left in the Rust PostgreSQL server’s query language. Storing and returning one already worked — int[] columns, ARRAY[...] literals, multidimensional values, || — but almost nothing could be done with one: fifteen array functions, the three containment operators, and subscripting itself all answered 0A000 … is not supported yet, and UPDATE t SET a[2] = 99 answered cannot cast int4 to int4[], an error about a cast the statement never asked for.

All of that now works. The array corpus this release was measured against went from 24 divergences out of 29 to 6, and a second corpus of 32 further shapes runs clean; every one of the six remaining is an honest, named refusal rather than a wrong answer, and five of them are one missing feature (set-returning functions — unnest, generate_subscripts, a function in FROM).

Two things are deliberately refused rather than approximated. This server does not model array lower bounds — every array starts at subscript 1, which is what every array a client can build actually does — so array_fill(v, dims, lbounds) with a bound other than 1, and SET a[i] = v below subscript 1, both answer 0A000 by name. PostgreSQL answers the second by moving the array’s lower bound, leaving an [0:5]={…}; a server that silently re-based it to 1 would hand back the right value under subscripts that then lie about it, which is the kind of quiet divergence this project treats as data loss.

Added

• The array functions: array_length, array_ndims, array_dims, array_lower, array_upper, cardinality, array_cat, array_append, array_prepend, array_to_string, string_to_array, array_position, array_positions, array_remove, array_replace and array_fill, each measured against PostgreSQL 14.13 including its NULL rules — which are not one rule: array_cat(NULL, ARRAY[3]) is {3} while array_remove(NULL, 1) is NULL, and array_position(ARRAY[1,NULL], NULL) finds the NULL where ARRAY[1,NULL] @> ARRAY[NULL] is false.

• The containment operators @>, <@ and &&, which flatten both sides, so ARRAY[[1,2],[3,4]] @> ARRAY[3] is true.

• Array subscripting: a[1], a[2:3], a[2:], a[:2], m[1][2], m[1:2][1:1], and a bound that reads the row (a[n]). An out-of-range element is NULL where an out-of-range slice is the empty array, and a subscript list shorter than the array’s dimensionality selects nothing — (ARRAY[[1,2],[3,4]])[1] is NULL, not the inner row.

• Subscript assignment: SET a[i] = v and SET a[lo:hi] = v, extending the array with NULLs past its end, building it from a NULL column, and letting a second assignment in one statement see the first.

Fixed

• A multidimensional array reported its type as text[] instead of the element array (int4[], oid 1007). A binary-format client refused the integer rows it was handed as _text.

• An array whose first element was NULL took its element type from that NULL and reported text[], so a client decoded the values beside it as NULL.

• An array column was sampled as NULL at plan time, so every expression over one was described from a NULL: length(ta[1]) was typed text and the integer it computed went out as the string '1'.

• UPDATE with only a subscripted assignment took the bulk write path, wrote an empty $set and still reported UPDATE 1 — a statement that claimed success and changed nothing.

Security

• array_fill(1, ARRAY[1000000000]) and SET a[1000000000] = 1 each size an array from a number the client supplies. PostgreSQL caps an array at 134217727 elements and says so; the cap is now checked before any allocation, so a single statement can no longer exhaust the server’s memory.

Rust PostgreSQL server: system types, all PG 15 settings, session-private pg_temp functions

pgjdbc against the Rust PostgreSQL server went from 22 failures to 3 (7,354 tests). The remaining three are an UPDATE of pg_class, a LANGUAGE C function, and a client-socket check. pgx stays at 377 passed / 0 failed, including two runs over one store. Every fix is checked against PostgreSQL 15 and pinned in the b40_fixes and b40_types corpora.

Added

• The macaddr, macaddr8, pg_lsn, txid_snapshot, pg_snapshot, xid, xid8 and cid types.

• All 345 PostgreSQL 15 settings are SHOW-able with PG’s defaults. pg_settings carries their metadata.

• Standalone composite types appear in pg_class.

Fixed

• pg_temp functions are private to their session and dropped at disconnect.

• ADD PRIMARY KEY USING INDEX keeps the index’s name and its INCLUDE columns.

• A LIKE pattern ending in its escape character raises 22025 only when matching reaches it.

• _custom array type names follow PostgreSQL’s rule. Columns of composite or enum array types appear in pg_attribute.

• Two-dimensional enum and composite arrays render correctly.

• A BC date keeps a UTC offset written after it.

• Describe of a statement reports text formats. The time-zone name is case-canonical.

• System relations carry initdb’s relacl.

• Startup ParameterStatus no longer sends search_path.

• The pgjdbc runner gives its test role a password, as pgjdbc’s CI does.

Rust PostgreSQL server: a privilege bypass, has_table_privilege, and backlog leftovers

Security

• A table named with its schema (s.t) skipped the privilege check entirely: any role could SELECT and INSERT on it. The check now resolves the qualified relation.

• has_table_privilege answered true for every role. It now answers from grants, owners, role membership and PUBLIC.

Fixed

• Two cases accepted duplicate rows silently on a dotted column: ALTER TABLE ADD UNIQUE over existing rows, and UNIQUE NULLS NOT DISTINCT. Both now reject them.

• Binary input of xid, cid, xid8, txid_snapshot, pg_snapshot and their arrays was stored as raw bytes. It is now decoded.

• A pg_temp function called without its pg_temp. prefix is refused (42883). Leftovers from a killed server are dropped at open.

• min / max over types with no such aggregate in PostgreSQL 15 are refused.

• SET of server-start or reload-only settings is refused with 55P02.

• Index comments are kept per schema.

• pg_get_viewdef qualifies other schemas’ tables.

• 't'::regclass and to_regclass walk search_path.

• An aggregate FILTER containing a subquery works for array_agg, json_agg, jsonb_agg, json_object_agg and window aggregates.

• The SQLAlchemy gauge runner deletes its temp directory.

Rust PostgreSQL server: durable commits, schema and object privileges, snapshot functions

Changed

• In durable mode (the default), every acknowledged COMMIT on secantusd-pg is now synced to disk. Before, a commit acknowledged just before a process kill was lost: 20 of 20 times in a kill test, against 0 of 20 now. This matches PostgreSQL’s default. Durable autocommit writes cost about 10x more. Test fast-storage mode (SECANTUS_TEST_FAST_STORAGE=1) is unchanged. The Rust MongoDB server is unaffected.

Fixed

• Schema USAGE is enforced: reading s.t without it is 42501.

• GRANT / REVOKE on schemas, sequences and functions is recorded.

• has_schema_privilege, has_sequence_privilege, has_function_privilege and has_column_privilege answer from grants and owners, not true for everyone.

• set_config(x, v, true) outside a block lasts only for its statement.

• Large-object descriptors opened in autocommit close at statement end.

• Other sessions’ temp tables are listed in pg_class.

• A notice raised before pg_sleep is sent during the sleep, and the sleep can be cancelled.

Added

• information_schema.role_table_grants and table_privileges.

• The snapshot functions: pg_snapshot_xmin / xmax / xip, pg_visible_in_snapshot, pg_current_snapshot, and their txid_ forms.

Rust PostgreSQL server: privilege enforcement, and the per-statement cost halved

Fixed

• Privileges that were recorded but never enforced now are:

  • A schema without USAGE is left out of the search path.

  • Schema CREATE is checked when an object is created.

  • nextval / currval / setval and SELECT from a sequence check its privileges.

  • Function EXECUTE is checked at the call.

• Function grants are per signature. GRANT ... ON ALL SEQUENCES / FUNCTIONS IN SCHEMA takes effect.

• A dropped sequence’s or function’s grants no longer carry over to a recreated object of the same name.

• A PL/pgSQL or DO-block RAISE NOTICE is sent as it is raised.

Performance

Per-statement overhead had tripled since the full PostgreSQL 15 setting list was added. Release build, psycopg, median of 5 × 1,000:

µs per statement	before	after
simple select 1	78	37
extended select 1	152	60
extended primary-key read	205	103

The setting map is reinstalled only when it changes. Roles, row-level-security flags, inheritance and expression-index lists are cached against the catalog version.

Rust PostgreSQL server: READ COMMITTED waits for a row instead of failing

Fixed

• A READ COMMITTED block that met another session’s uncommitted row used to fail at once with 40001. It now waits for that session, then re-runs its statement against the committed row, as PostgreSQL does. The final result matches PostgreSQL in every scenario tested. The wait still ends on:

  • cancel;

  • statement_timeout;

  • lock_timeout (55P03);

  • a deadlock (40P01).

• REPEATABLE READ and SERIALIZABLE still raise 40001.

• An UPDATE that leaves a row unchanged (n = n*2 over 0) now takes the row lock. Before, it ignored another session’s uncommitted update: it left 7 where PostgreSQL gives 14.

• A failed block rolls back its storage transaction immediately, so a deadlock’s survivor need not wait for the loser’s ROLLBACK.

• Privilege checks:

  • a SERIAL column’s default nextval is checked for the inserting role;

  • lastval() is checked against its sequence;

  • EXECUTE on the trigger function is checked at CREATE TRIGGER;

  • grants on built-in functions are per signature.

Performance

A statement’s Describe result is cached per session while the catalog, settings and role are unchanged. Release build, before and after:

statement	before	after
primary-key read	104 µs	77 µs
autocommit update	101 µs	84 µs

Rust PostgreSQL server: operator type checks, built-in EXECUTE, view definitions

Fixed

• Four comparisons across types used to give a silent wrong answer. Each now raises PostgreSQL’s error (42883 / 22007):

  • arr = 1 returned the row;

  • a = ANY(ARRAY['x']) returned no rows;

  • text BETWEEN 1 AND 2 returned no rows;

  • current_date = 'x' returned no rows.

• Other type and placement errors:

  • sum(text) returned 0;

  • an aggregate or window function in WHERE was accepted;

  • json -> / ->> operand types were not checked;

  • int + float8 typing now matches PostgreSQL.

• After REVOKE EXECUTE on a built-in function, a direct call is refused. The check uses the overload chosen for the argument types.

• pg_get_viewdef prints cross-type date/time comparisons, [NOT] MATERIALIZED CTEs and RANGE offsets. Invalid RANGE offsets are refused at planning.

• Error positions and hints for the ungrouped-column and invalid-reference errors.

Changed

• A test now proves the catalog-cache gate is needed: without it, one connection’s uncommitted event trigger fired on another connection’s DDL.

Rust PostgreSQL server: PostgreSQL-equivalent commit sync on macOS, PL/pgSQL error context

Changed

• On macOS, durable commits sync the log with O_DSYNC, which is what PostgreSQL’s macOS default open_datasync does. They no longer use a full drive-cache flush (F_FULLFSYNC). An acknowledged commit still survives a process kill (0 losses in 20 kill runs), but not a power loss, the same as PostgreSQL. Durable autocommit UPDATE dropped from 7.9 ms to 149 µs, against PostgreSQL 15’s 82 µs. Linux and the Rust MongoDB server are unchanged.

Fixed

• a LIKE 'x' over an integer in the select list returned NULL. It is now 42883 with a position.

• A FROM-list subquery that references a sibling (FROM t x, (SELECT x.a) s) is 42P01 with PostgreSQL’s hint.

• PL/pgSQL errors carry PostgreSQL’s CONTEXT stack, through nested calls, EXECUTE, triggers and DO blocks.

• pg_get_viewdef:

  • prints implicit casts for the resolved overload;

  • prints EXTRACT in its own syntax;

  • wraps by column width.

Rust PostgreSQL server: system indexes, SQL-function error context, SECURITY DEFINER

Fixed

• Writes to system catalogs were silently wrong: UPDATE and DELETE reported 0 rows, and INSERT created a hidden table. Now:

  • an UPDATE that leaves every matched row unchanged reports its rows, as PostgreSQL does;

  • a real change, a DELETE that matches rows, or an INSERT is refused with 0A000.

• pg_index, pg_indexes and pg_get_indexdef list PostgreSQL’s own 162 system indexes. Catalog result columns carry their table oid and column number.

• Errors in LANGUAGE sql functions carry PostgreSQL’s CONTEXT: statement N, during inlining, or no frame, following PostgreSQL’s inlining rules.

• CREATE FUNCTION ... SECURITY DEFINER and SET clauses were dropped. They are now stored and applied: a definer function runs as its owner, and a SET clause holds for the call only.

Performance

• Extended-protocol replies are buffered until Sync or Flush rather than flushed once per message. Extended select 1 went from 61 to 53 µs.

Rust PostgreSQL server: precise row waits, plan reuse, streaming portals, PG 15 catalog columns

Fixed

• A transaction waiting on a row now waits only on that row’s holder. The holder is tracked through each block’s written rows, including rows written by triggers and FK cascades. A chain of waits no longer reports a false deadlock (40P01); real cycles still do.

• SELECT * over 51 system catalogs returns exactly PostgreSQL 15’s columns, in PostgreSQL 15’s order. 21 differed before.

• An int8 value too large for an int4 column answers 22003, not 22P02.

Performance

• A prepared SELECT / UPDATE / DELETE that is safe to template reuses its plan across Executes, substituting the bound values into its WHERE filter.

• An extended-protocol SELECT outside a transaction block streams from one snapshot in 256-row batches on a pooled reader thread. Server memory for a 400 MB result grew 436 MB instead of 1,620 MB.

µs per statement (release)	before	after	PostgreSQL 15
extended primary-key read	74	55	34
simple select 1	37	32	23
extended scan, 1,000 rows	891	755	188

Rust PostgreSQL server: psycopg back to zero failures, COMMIT no longer rebuilds the catalog

The psycopg suite against the Rust PostgreSQL server had regressed from 1 failure (2026-09-18) to 61. It is now 5,544 passed / 0 failed. Every fix is checked against PostgreSQL 15 and pinned in the b49_psycopg corpus.

Fixed

• An untyped literal or parameter compared with a range or multirange takes the range’s type ('empty' = $1::int4range).

• An explicit cast to bpchar with no length keeps trailing blanks (chr(32)::bpchar is ' ').

• An untyped argument no longer forces a text overload (array[set_byte('x', 0, $2)]).

• Temporary tables:

  • a temp table can REFERENCES itself;

  • constraint errors on a temp table report its bare name, with pg_temp_N as the schema.

• SET client_encoding inside a block keeps working after COMMIT.

Performance

• COMMIT and ROLLBACK rebuilt every connection’s catalog view, even when the transaction changed no catalog. 60 select; commit over 100 tables took 31.9 s; they now take 0.4 s. The full psycopg run dropped from 1,539 s to 1,010 s.

Rust PostgreSQL server: SELECT FOR UPDATE, row-lock residuals, type checks, faster correlated subqueries

Fixed

• SELECT ... FOR UPDATE / FOR NO KEY UPDATE was accepted and ignored. Two sessions could both read and then update a row: a lost update. Row locks are now taken, including NOWAIT (55P03) and SKIP LOCKED.

• Row waits and deadlocks:

  • Autocommit statements, a running statement and primary-key INSERT conflicts now hold their rows while waiting, so cycles through them are 40P01.

  • ROLLBACK TO SAVEPOINT releases the rows written after the savepoint.

  • REPEATABLE READ waits for the holder before deciding on 40001.

  • A block re-run after a conflict no longer writes its failed attempt twice.

• Type mismatches that returned a value now raise PostgreSQL’s error:

  • CASE / COALESCE / GREATEST / LEAST / NULLIF over mismatched types;

  • x IN (SELECT ...) across types.

• greatest(date, timestamptz) returns a timestamptz.

• pg_get_viewdef matches PostgreSQL on 60 more view shapes.

• txid_current() and pg_current_xact_id() work. pg_current_snapshot() reports running transactions.

• DDL notices (“skipping”, the CASCADE list) are sent as they are raised.

Performance

• Correlated subqueries with a text key, a JOIN or an aggregate under a filter are hash-indexed: 2,000 × 2,000 rows went from about 1.2 s to 0.01-0.1 s.

• UPDATE / DELETE no longer decode the whole catalog to find foreign keys. Planner setup is cached per session.

µs per statement (release)	before	after	PostgreSQL 15
extended select 1	49	42	27
extended autocommit UPDATE	86	74	56

Rust PostgreSQL server: shared row locks, FOR UPDATE through joins and views, streaming cursors in a block

Fixed

• FOR SHARE / FOR KEY SHARE took no lock. They are now shared row locks following PostgreSQL’s conflict table: sharers coexist, writers and FOR UPDATE wait for them, and two sharers that both upgrade get 40P01.

• FOR UPDATE / FOR SHARE over a join, FROM-subquery or view locked nothing. It now locks the base rows behind the returned rows and honours OF.

• PostgreSQL’s refusals of row locking are added: with DISTINCT, GROUP BY, aggregates, windows, UNION, or on the nullable side of an outer join.

• ROLLBACK TO SAVEPOINT releases later rows under REPEATABLE READ, after DDL, and for lock-only rows. When it does rewrite, a commit by another session no longer makes it fail.

Performance

• Inside a transaction block, an extended-protocol portal or a DECLARE CURSOR (with or without HOLD) over a plain one-table SELECT now streams in batches. Server memory for a 200 MB result grew 16-24 MB instead of 670-900 MB.

Rust PostgreSQL server: faster correlated subqueries, sum(bigint) overflow, pg_collation_for

Fixed

• sum(bigint) past the int64 range silently wrapped (-9223372036854775806 where PostgreSQL answers 9223372036854775810). It now sums exactly and answers numeric.

• pg_collation_for:

  • raises 42804 over a non-collatable type;

  • answers NULL for an untyped literal;

  • keeps its column name.

• A COPY or Describe that arrived before any statement had run on its worker thread could miss user-defined types.

Performance

Correlated subqueries, 2,000 × 2,000 rows, release build:

shape	before	after	PostgreSQL 15
nested correlated EXISTS	36.7 s	1.21 s	0.13 s
sum / avg under a filter	1.2 s	0.03-0.04 s	0.15 s
inner ORDER BY ... LIMIT 1	0.30 s	0.008 s	0.13 s

• A WHERE with one non-lowerable conjunct now lowers the rest instead of evaluating the whole clause per row.

• BindComplete and CloseComplete wait for Sync or Flush. A primary-key read went from 54 to 50 µs.

Rust PostgreSQL server: FOR UPDATE on tables without a primary key, faster nested subqueries

Fixed

• FOR UPDATE through a subquery on a table with no primary key locks only the rows behind the result.

• sum / avg of double precision returned Infinity on overflow. They now raise 22003, as PostgreSQL does, for plain aggregates and window functions.

• A user error inside a subquery reached the client as XX000 “could not read a subquery”. It now keeps its own SQLSTATE.

• The psycopg gauge runner restores default SIGINT handling. A run launched in the background inherited SIGINT ignored, so psycopg’s test_ctrl_c could never send its cancel.

Performance

2,000 × 2,000 rows, release build:

shape	before	after	PostgreSQL 15
nested correlated EXISTS	1.23 s	0.147 s	0.128 s
grouping in a FROM subquery	0.32 s	0.045 s	0.128 s
sum / avg of bigint, numeric or float under a filter	~1.24 s	0.03-0.07 s	~0.146 s

Rust PostgreSQL server: savepoint-safe UNIQUE indexes, READ COMMITTED reads after a write, EXCEPTION blocks

Fixed

• A UNIQUE index created in a transaction became a plain index if the transaction was later moved: by ROLLBACK TO SAVEPOINT, a conflict re-run, or a snapshot refresh. Duplicates were then accepted after COMMIT, silently. The index’s options are now kept.

• A READ COMMITTED block that has written now sees other sessions’ later commits in its following statements, as PostgreSQL does. This applies while its write set is at most 256 entries.

• A PL/pgSQL EXCEPTION block releases the rows it undoes immediately.

• Every transaction move keeps the block’s rows held across the move, closing a window where another writer could take them.

• FOR UPDATE through a JOIN on a table without a primary key locks only the rows returned.

• sum(real) adds in single precision. It raises 22003 on overflow instead of returning Infinity.

• round / trunc(numeric, n) honour a negative n and handle NaN and Infinity.

• A correlated subquery no longer raises a data error from a select-list expression on a row no outer row reaches.

Performance

• More correlated-subquery shapes run once per statement rather than once per outer row:

  • numeric ordering filters;

  • sum / avg of real;

  • immutable built-ins in the select list;

  • generate_series / unnest of constants.

Rust PostgreSQL server: SIGTERM, long READ COMMITTED blocks, sequences read live

Fixed

• secantusd-pg started with SIGTERM blocked, for example as a background job, could not be stopped by SIGTERM. It now unblocks and resets SIGINT and SIGTERM at startup. This is POSIX only.

• A block’s SELECT last_value on a sequence missed another session’s nextval. Sequences are now read live unless the block wrote them.

• READ COMMITTED blocks with more than 256 writes no longer keep their first snapshot. A block of 400 serial INSERTs alternating with SELECTs went from 2.54 s to 0.44 s. The block now moves to a fresh snapshot only for commits that can change its answer.

Performance

• Correlated subqueries with collated-text ordering filters, or with numeric min / max under a filter, are hashed instead of run per row. At 2,000 × 2,000 rows they take 0.03-0.04 s, against 0.12-0.68 s before.

Tooling

• The PostgreSQL differential probe compares against a C-collation reference database when the reference cluster’s default collation is not C.

Rust PostgreSQL server: linear long READ COMMITTED blocks and sorted portals in bounded memory

A READ COMMITTED transaction block that had written and then read another table while other sessions kept committing there was moved onto a fresh snapshot before every such read, replaying its whole write set each time – quadratic in the block’s length (800 insert/read pairs took 54 s). A plain read of a table the block has not written now runs in a fresh read-only transaction of its own, which is exactly the per-statement snapshot PostgreSQL takes, and the same 800 pairs take 1.3 s.

An extended-protocol SELECT ... ORDER BY outside a block is now streamed: the reader sorts the rows in runs of at most 16 MB, spills them to anonymous temporary files and merges them (a LIMIT keeps only the rows it can return), so a 300,000-row, 600 MB ordered result costs the server what an unordered one does instead of ~800 MB more.

Fixed

• secantus-pgserver: a READ COMMITTED block’s plain one-table read of a table it has not written (nor any catalog) reads in its own transaction instead of replaying the block’s write set (rc_reads_apart, read_apart).

• secantus-pgserver: an IMMUTABLE LANGUAGE sql function called over constants is run as itself, as PostgreSQL’s evaluate_function does before inlining, so its error carries SQL function "f" statement 1 (it said during inlining).

Changed

• secantus-pgserver: streamed portals accept an ORDER BY of stored columns (external_sort); per-statement work trimmed – the DateStyle and TimeZone are parsed only when the session’s settings change, the catalog collections are checked once per connection, the type catalog is read through a per-thread copy of the shared cache, and sql_relations remembers its parse-tree walk per statement text (names are still resolved per call).

Rust PostgreSQL server: READ COMMITTED joins and aggregates without a replay, and pulled-up constants

A READ COMMITTED transaction block that has written, then reads other tables by a join or an aggregate while other sessions commit, no longer replays its whole write set before each read. Errors from inlined SQL functions over a one-row constant FROM item now carry PostgreSQL’s during inlining context.

Fixed

• secantusd-pg: a READ COMMITTED block’s read by a JOIN, an aggregate, GROUP BY or an ORDER BY over an expression now runs in a fresh read-only transaction when every table it names is an ordinary table the block has not written. Before, the block was moved to a new snapshot with its writes replayed for every such statement. The read also has to call no function except pure built-ins, and have no subquery, no CTE and no row lock. 800 insert-then-join pairs beside a committing writer (debug build): 66.5 s -> 11.8 s.

• secantusd-pg: a one-row VALUES or constant SELECT in FROM is pulled up into a user function call’s arguments, as PostgreSQL’s planner does. An error raised by an inlined LANGUAGE sql function over that column now says SQL function "f" during inlining. Before, the error had no CONTEXT.

The Rust servers: ordered cursors in bounded memory, cheaper primary-key reads, fewer READ COMMITTED replays, and a SIGTERM fix for secantusd-rs

Inside a transaction block, a cursor or portal over one table with an ORDER BY of stored columns no longer holds its whole result in memory. A primary-key read is about 3 µs faster. A READ COMMITTED block that reads a table it has written no longer replays its writes when only other tables changed.

Changed

• Rust PostgreSQL server: a DECLARE CURSOR, or an extended-protocol portal inside a block, over one table with an ORDER BY of stored columns is sorted at its first fetch in bounded memory. It spills sorted runs to temporary files and merges them. Fetching 300,000 rows of 2 KB in a block grew the server by 47 MB, down from 2.7 GB.

• Rust PostgreSQL server: a read by primary-key equality runs on its tokio worker directly, without block_in_place’s hand-off. In a release build a primary-key read went from about 49 µs to 45 µs.

• Rust PostgreSQL server: in a READ COMMITTED block that has written, a plain read of tables the block wrote takes a fresh snapshot only when another session committed to one of those tables (or to a catalog). It used to move whenever anything had committed. Then it replayed the block’s whole write set, which made a long block quadratic: 800 insert-then-read pairs beside another table’s writer took 52.7 s, and now take 3.2 s.

Fixed

• secantusd-rs (the Rust MongoDB server binary) ignored SIGTERM when its parent had left SIGTERM blocked. It also lost a SIGTERM sent the moment it printed its “listening on” line, because the stop handler was installed after that line. It now resets the signal mask and installs the handler before opening storage, as secantusd-pg does.

The Rust PostgreSQL server: hashed DISTINCT, bounded-memory DISTINCT, expression sorts and large GROUP BYs, and cheaper primary-key reads

DISTINCT, GROUP BY, INTERSECT and EXCEPT no longer compare each row against every distinct row kept so far. That comparison was quadratic, and a SELECT DISTINCT over 300,000 distinct rows did not finish. Rows now go through a hash on the same value identity. DISTINCT, DISTINCT ON and an ORDER BY over an expression now stream in bounded memory, as an ORDER BY of stored columns already did. So does a GROUP BY whose input is larger than 64 MB. A primary-key read is about 3 µs faster.

Changed

• Rust PostgreSQL server: SELECT DISTINCT, DISTINCT ON, GROUP BY (grouping sets included), INTERSECT [ALL], EXCEPT [ALL] and agg(DISTINCT ...) use a hash on the value identity. Equality is still decided by == inside a bucket, so the answers are unchanged (distinct_set.rs).

• Rust PostgreSQL server: a DECLARE CURSOR, a block’s portal and an extended-protocol portal outside a block now stream with DISTINCT, DISTINCT ON or an ORDER BY over an expression. Each one sorts in bounded memory and keeps the first of each run of equal rows. Fetching 300,000 rows of 2 KB 1,000 at a time after a restart (release build), the server’s RSS growth went down as follows:

  • expression ORDER BY, outside a block: 1,719 → 40 MB;

  • expression ORDER BY, named cursor: 2,384 → 44 MB;

  • DISTINCT: 1,371 → 33 MB;

  • DISTINCT ON: 1,372 → 37 MB.

• Rust PostgreSQL server: a plain GROUP BY over one stored table whose input passes 64 MB sorts on the group key and aggregates one group at a time. This also works inside a transaction block. RSS growth for the 300,000-row table went from 1,430 MB to 106–110 MB. Smaller inputs are grouped as before. SECANTUS_PG_GROUP_MEMORY_BYTES lowers the threshold for tests.

• Rust PostgreSQL server: a primary-key read no longer copies the role list, the session user, the role and the temp-schema name on every statement. It also no longer resolves its relations through the search path on every statement. Release build, bench43.py, two interleaved runs: 47.4 / 49.0 → 45.4 / 45.3 µs. PostgreSQL 15.19 takes 33.1 µs.

Fixed

• Rust PostgreSQL server: DISTINCT / GROUP BY over an array compares its elements by value. {1.0} and {1.00} are now one numeric array, and {NaN} equals itself in a float array. A set operation now matches an integer literal against a numeric column by value, so select n ... union select 1 no longer returns 1 twice. These are PostgreSQL 15.19’s answers (corpus b59_distinct).

Rust PostgreSQL server: jsonb compares by value, ungrouped aggregates in bounded memory

jsonb values were compared by their stored text, which keeps each number’s scale, so {"x": 1} and {"x": 1.0} came out as two different values. PostgreSQL compares jsonb numbers as numerics. The wrong answers were not limited to DISTINCT and GROUP BY: WHERE j = '{"x":1.0}' returned no rows at all, even for a row stored with exactly that text. An aggregate with no GROUP BY now runs in bounded memory.

Fixed

• jsonb equality is by value everywhere it is decided: WHERE = / <> / IN / = ANY / IS [NOT] DISTINCT FROM, joins, IN and EXISTS semi-joins, SELECT DISTINCT, DISTINCT ON, GROUP BY (including its sorted spill), count(DISTINCT j), UNION / INTERSECT / EXCEPT, a window’s PARTITION BY, and UNIQUE / PRIMARY KEY constraints on INSERT and UPDATE. Checked against PostgreSQL 15.19 with corpus b60_jsonb_eq.

• An UPDATE now enforces a UNIQUE constraint over a column with a nondeterministic collation. Before, only INSERT checked it.

• SELECT DISTINCT count(*) ... GROUP BY k ORDER BY 1 (or ORDER BY the aggregate’s alias) returned an error. It now returns the distinct aggregate values in order.

Changed

• An aggregate with no GROUP BY over one stored table (with no filter or a collection-scan filter) reads its input one chunk at a time and combines the partial results. This covers count, min, max, bool_and, bool_or, and an exact integer or numeric sum. On 300,000 rows of 2 KB the server’s memory growth fell from 1755 MB to 58 MB.

• An equality on a jsonb column is now checked row by row, so it no longer uses a btree index on that column. A jsonb UNIQUE constraint reads the whole table once per write statement.

READ COMMITTED reads its own rows without replaying, and avg streams

A READ COMMITTED transaction block on the Rust PostgreSQL server that reads a table it has written, while other sessions keep committing to that same table, no longer replays its whole write set onto a new snapshot for every statement. The read runs in a fresh snapshot with the block’s own rows laid over it, so it sees every commit since and its own uncommitted writes, as PostgreSQL does: 800 insert-then-aggregate pairs beside a committing writer went from 24.7 s to 8.6 s on a debug build.

avg over any numeric or floating-point column, and sum over a float column, with no GROUP BY, are now computed a chunk at a time in bounded memory, with answers bit-identical to the one-pass result: 300,000 rows of 2 KB went from about 1.75 GB of server memory growth to about 64 MB.

Added

• secantus-storage: Storage::block_overlay, TableOverlay, with_read_overlay and Storage::oplog_async. find_matching_with, scan_batch_after, scan_matching_batches, count_matching and find_by_id honour an installed overlay; nothing installs one on the MongoDB server.

Changed

• secantus-pgserver: a READ COMMITTED SELECT of the narrowed shapes over tables the block wrote reads apart under the block’s overlay (rc_overlays) instead of moving the block. Ungrouped avg and float sum stream (ungrouped_in_bounded_memory): a float total is carried from chunk to chunk, an exact average keeps its sum and count.

Rust PostgreSQL server: subqueries see READ COMMITTED commits; sub-millisecond timestamps sort and group; simple-protocol SELECTs stream

A batch of fixes and memory work on secantusd-pg. Every answer was checked against PostgreSQL 15.19.

Fixed

• A READ COMMITTED block’s uncorrelated subquery missed rows committed since the block’s snapshot. A subquery like this runs while the statement is planned. Planning came before the block moved onto a fresh snapshot. So the first read after another session’s commit answered the subquery from the old snapshot: select (select max(x) from t) gave 2 where PostgreSQL gives 300. A statement with a subquery or a WITH item now gets its snapshot before planning.

• Timestamps that share a millisecond were not told apart in five places. A timestamp / timestamptz is stored as a millisecond date plus a hidden microsecond remainder. Five places compared the date alone:

  • ORDER BY ts put .002006 before .002000.

  • GROUP BY ts merged every value in one millisecond into one group.

  • DISTINCT ON (ts) kept one row per millisecond.

  • min(ts) / max(ts) lost the microseconds.

  • count(DISTINCT ts) undercounted.

  All of these now use the full value, and window peers and ranks use the same comparison. Corpus b62_distinct_ts gives 0 divergences in 35 checks, with and without SECANTUS_PG_GROUP_MEMORY_BYTES=1000.

Changed

• A READ COMMITTED read with uncorrelated subqueries or plain-SELECT CTEs no longer replays the block’s write set. Such a read used to move the block onto a new snapshot whenever another session had committed. Now it reads in a fresh transaction, with the block’s own rows laid over it, exactly as a read without a subquery already did. The planning-time subquery and the statement share that one transaction. Measured on a debug build with 200 / 400 / 800 insert-then-subquery-read pairs, beside a session committing to the same table: the replay path takes 4.0 / 15.5 / 59.8 s and the new path 1.1 / 3.3 / 12.2 s.

• A lone SELECT sent through the simple query protocol now streams. This covers a SELECT with no WHERE, outside any transaction. Its rows come from a reader thread as pgwire sends them, as an extended portal’s already did. Measured on 300,000 rows of 2 KB after a restart, with the WiredTiger cache capped at 64 MB so the result is not hidden behind the cache: peak RSS went from 1884 MB to 175 MB.

• DISTINCT over a timestamp / timestamptz column streams in bounded memory. It used to be materialised. The hidden microsecond remainder is now part of the row’s identity.

Rust PostgreSQL server: read-only STABLE functions, and two silent wrong answers fixed

The Rust PostgreSQL server now runs a STABLE or IMMUTABLE function the way PostgreSQL does: read-only. An INSERT, UPDATE, DELETE, DDL, SET, SHOW, NOTIFY or row-locking SELECT inside one is refused with PostgreSQL’s 0A000 “… is not allowed in a non-volatile function” – for a SQL function before any of its statements runs, for PL/pgSQL at the offending statement. Before, the write simply happened.

That guarantee is what lets a READ COMMITTED block call such a function without replaying its writes: a read that calls only STABLE / IMMUTABLE functions whose bodies read plain tables now reads in a fresh snapshot with the block’s own rows laid over it. Doing that exposed a silent wrong answer that was already there – a function over constants is run while the statement is planned, which was before the block took its fresh snapshot, so select f() missed a row another session had just committed. Such a read now gets its snapshot before planning.

A subquery inside INSERT ... VALUES works (it was “SubLink is not supported yet”), and a simple-protocol SELECT whose WHERE would be answered by a collection scan anyway now streams its rows instead of building the whole result first.

Fixed

• secantus-pgserver: a write (or any statement but a plain SELECT) inside a STABLE / IMMUTABLE function is 0A000, with PostgreSQL’s CONTEXT (SQL function "f" during startup, or the PL/pgSQL statement frame). A VOLATILE function such a function calls may still write, as in PostgreSQL.

• secantus-pgserver: in a READ COMMITTED block that had written, a user function folded while planning read the block’s old snapshot and missed rows other sessions had committed since (select f() answered 2 where PostgreSQL 15 answers 3).

• secantus-pgplan: a subquery in an INSERT ... VALUES row.

Changed

• secantus-pgserver: a READ COMMITTED read calling STABLE / IMMUTABLE user functions (SQL, or PL/pgSQL without dynamic SQL, cursors, CALL or EXCEPTION blocks, transitively, with no VOLATILE function anywhere) reads apart from the block under its overlay instead of replaying the block’s write set.

• secantus-pgserver: a simple-protocol SELECT whose WHERE the storage would answer by a collection scan streams from a reader thread (300,000 rows of 2 KB: server RSS growth ~700-1000 MB -> ~20 MB).

Rust PostgreSQL server: indexed and in-block reads stream, and four wrong answers fixed

The Rust PostgreSQL server streams more reads in bounded memory. A SELECT with an indexed WHERE now walks its index a batch at a time instead of building the whole result first, and so does an aggregate over an indexed filter. A simple-protocol SELECT inside a REPEATABLE READ or SERIALIZABLE block streams through the block’s own transaction. A DISTINCT over a tsvector or tsquery column streams too. Over 300,000 rows of 2 KB, server memory growth fell from 1.3 GB to 27 MB for an indexed range, and from 2.6 GB to 47 MB for a whole-table read inside a REPEATABLE READ block. The answers are byte-identical.

Reading one row by primary key no longer asks the storage for a query plan first. That makes a PK read about 3 microseconds faster.

Probing these paths against PostgreSQL 15.19 found four wrong answers, all now fixed. exists(select 1/0) raised an error, but PostgreSQL never evaluates an EXISTS subquery’s select list and answers true. A SELECT DISTINCT whose ORDER BY was not in the select list returned rows where PostgreSQL raises 42P10. Float power returned NaN or Infinity where PostgreSQL raises an error. Float sqrt, ln and log raised errors with the wrong SQLSTATE.

Added

• secantus-storage: Storage::scan_routed_batches, a batched scan routed through the same index find_matching would use.

• tools/probes/plpgsql_expr_context.py: compares the CONTEXT of PL/pgSQL expression errors against PostgreSQL over 60 expression shapes.

Changed

• secantus-pgserver: a simple-protocol or whole-result extended SELECT with an indexed WHERE streams; a primary-key point lookup keeps the direct path.

• secantus-pgserver: an aggregate over an indexed filter is computed in bounded memory, like one over a collection scan.

• secantus-pgserver: a simple-protocol SELECT inside a REPEATABLE READ / SERIALIZABLE block streams through the block’s transaction.

• secantus-pgserver: a DISTINCT over a tsvector / tsquery column streams, de-duplicating on the value’s text.

Fixed

• secantus-pgplan: an EXISTS subquery’s select list, DISTINCT and ORDER BY are dropped before it runs, so exists(select 1/0) is true (PostgreSQL’s simplify_EXISTS_query).

• secantus-pgplan: SELECT DISTINCT ... ORDER BY <something not in the select list> is 42P10 for SELECT DISTINCT, ORDER BY expressions must appear in select list.

• secantus-pgplan: the float math functions follow PostgreSQL’s float.c:

  • sqrt of a negative raises 2201F.

  • ln / log of zero or a negative raises 2201E, with PostgreSQL’s two messages.

  • exp overflow and underflow raise 22003.

  • power raises 2201F for a negative to a fraction or zero to a negative, and 22003 on overflow or underflow. Its NaN and infinity cases match.

• secantus-pgserver: a PL/pgSQL expression error carries the SQL expression "..." CONTEXT frame exactly when PostgreSQL’s planner would raise it. That happens when a constant subexpression fails as it is folded: 1/0 + x gets the frame and x/0 does not. An unreadable literal never gets the frame. 58 of 60 probed shapes now match, up from 50.

• tests: the nested pytest runs no longer sweep the stale temp backlog at session start. That sweep was the cause of a 300 s timeout in test_tmp_retention_guard.py.

Rust PostgreSQL server: joins and READ COMMITTED reads stream in bounded memory

A join of stored tables no longer builds every joined row before the first one goes out. The Rust PostgreSQL server reads and hashes each right side first, within a 64 MB bound, then reads the leftmost table a batch at a time on the statement’s own thread, joining, filtering and handing rows on as it goes; an aggregate over the join folds them in bounded memory. A 300,000-row join that held 3.5 GB now holds 25 MB, with byte-identical output. The narrow two-table path’s nested loop over every pair is a hash on the ON equality, so a join it falls back on is fast too.

A plain SELECT inside a READ COMMITTED transaction block streams through the block’s transaction as well, except when the statement reads apart from the block in a fresh snapshot of its own. Three missing errors are raised now: a COLLATE on a type that takes none is caught at planning, a DISTINCT or DISTINCT ON whose ORDER BY does not match it is 42P10, and rows skipped by OFFSET still have their select list computed.

Changed

• secantus-pgserver (stream_join): joins of stored tables, and aggregates over them, stream outside a transaction block; a right side over SECANTUS_PG_JOIN_INNER_BYTES (default 64 MB) falls back to the materialised path before any row is sent.

• secantus-pgserver: the narrow join path matches through a hash on its ON equality instead of a nested loop.

• secantus-pgserver: a simple-protocol SELECT inside a READ COMMITTED block streams unless it runs apart from the block.

Fixed

• secantus-pgplan: ORDER BY v COLLATE "C" over an integer column is 42804 at planning, over an empty table too.

• secantus-pgplan: SELECT DISTINCT t ... ORDER BY t COLLATE "C" and a DISTINCT ON whose leading ORDER BY items are not its expressions are 42P10.

• secantus-pgserver: rows skipped by OFFSET have their select list computed, so exists(select a/b ... offset 5) raises 22012 as PostgreSQL does.

The Rust PostgreSQL server skips subquery outputs nothing reads, and streams the joins it used to hold whole

PostgreSQL never computes a FROM-subquery output that nothing reads, so select count(*) from (select a/b from t) s answers even when b is zero. The Rust PostgreSQL server computed every output and raised a division by zero; it now removes an unread output first, under PostgreSQL’s own rules.

Joins that batch 65 still built whole now read in bounded memory: a join with a subquery, function or LATERAL side, a join inside a transaction block, GROUPING SETS / ROLLUP / CUBE, and a join whose right side is too big to hold (that side is now split into partitions on disk). On 300,000 rows of 2 KB with a 64 MB WiredTiger cache, peak server memory for these queries dropped from 2.9-5.9 GB to 52-290 MB, with byte-identical output.

Fixed

• secantus-pgplan (prune_outputs): a FROM-subquery output (or an output of an inlined CTE, or of a UNION ALL arm) that the enclosing query cannot read is replaced by NULL before planning, so its errors are not raised. As in PostgreSQL, volatile and set-returning outputs are kept, and so are outputs under a plain DISTINCT or named by the subquery’s own ORDER BY / GROUP BY. A subquery that does not plan on its own keeps every output, so parse-analysis errors still surface.

• secantus-pgserver: a two-table join of a float8 or numeric column to an int column (ON a.f = b.k) returned no rows on the narrow join path. It now compares the values numerically.

Changed

• secantus-pgserver (stream_join, grace_join): streamed joins now cover subquery, function and LATERAL sides, transaction blocks (read through the block’s session), and right sides past SECANTUS_PG_JOIN_INNER_BYTES (a grace hash join that keeps the materialised order). GROUPING SETS are aggregated one set at a time in bounded memory. Aggregates over a join no longer encode the joined rows only to decode them again. SECANTUS_PG_JOIN_STREAM=0 turns streamed joins off.

The Rust PostgreSQL server places an aggregate at the query level its columns belong to, and groups ROLLUP / CUBE from one read

In PostgreSQL an aggregate belongs to the lowest query level whose columns it reads. So in select (select max(s.x) from t) from s, max(s.x) is the OUTER query’s aggregate: the outer query returns one row, and the subquery fails with “more than one row returned” when t has several rows. The Rust PostgreSQL server computed the aggregate once per outer row instead, and returned a confident wrong answer. It now follows PostgreSQL’s rule in select lists, HAVING, ORDER BY, EXISTS / IN / ARRAY subqueries and nested subqueries. Such an aggregate placed in WHERE, a JOIN condition, GROUP BY or a LATERAL FROM item is now rejected with 42803, as in PostgreSQL.

GROUPING SETS / ROLLUP / CUBE over an input too big for memory read that input once per grouping set (batch 66), so a ROLLUP over a join ran the join three times. The input is now read once, and only the keys and aggregate inputs are kept. A small input is read once too; batch 66 read it twice. Joins whose right side spills to disk no longer re-sort the whole joined rows to restore their order.

Fixed

• An aggregate inside a subquery that reads only outer columns is the outer query’s aggregate (PostgreSQL’s agglevelsup), at any depth. One misplaced in WHERE / JOIN ON / GROUP BY / a LATERAL subquery raises 42803.

• A correlated scalar subquery that returns an outer column ((select s.x) from s) reported its column as text instead of the column’s type.

• A window function in a SELECT with no FROM (select max(1) over (), or a subquery’s max(s.x) over ()) was refused with 0A000.

• A SELECT ... FOR SHARE that waited for another transaction’s update could return the row as it was BEFORE that update. The other transaction’s rows stopped counting as locked just before its commit became visible, and a waiter re-reading in that instant saw the old row and no lock. A transaction now holds its rows until its commit is visible.

• An unread LATERAL subquery output is not computed, so (select a/b x ...) s, lateral (select s.x) l answers where it raised 22012.

Changed

• GROUPING SETS / ROLLUP / CUBE past the memory bound read their input once (unless the aggregate inputs are about as wide as the rows). On 300,000 rows of 2 KB (release build, 64 MB WiredTiger cache), a ROLLUP over a join went from 7.6 s to 2.2 s, and CUBE from 5.8 s to 1.7 s. Below the bound, the input is read once again.

• A grace hash join restores row order by sorting an index of the joined rows, not the rows themselves.

Rust PostgreSQL server: joins past the memory bound skip the spill when the other side fits, and GROUPING() works over joins and in HAVING

A join whose right side is too big to hold in memory used to write that whole side to partition files before it had looked at the left side. The server now reads the left side first. If the left side fits, it keeps it in memory and reads the right table again in chunks, so nothing goes to disk. Only when neither side fits does it partition, as before. Grouping sets over wide aggregate inputs (a max() over a long text column, say) used to read the input once per set. Now they read it once: each set’s groups are hashed and their partial results combined, so the memory used grows with the number of groups, not with the rows.

Running queries against PostgreSQL 15.19 turned up three divergences in GROUPING() and ORDER BY, all now fixed. GROUPING() was refused with a grouping error over any join. It was also refused in a HAVING clause. And an output alias inside an ORDER BY expression was taken for the output column where PostgreSQL resolves it to an input column.

Changed

• secantus-pgserver grace_join.rs / stream_join.rs: a right side past SECANTUS_PG_JOIN_INNER_BYTES triggers a read of the left side; a left side within the bound is held and joined against the right table a chunk at a time (left_built_join), in the materialised path’s row order. Otherwise the left side is spooled and the right side partitioned (grace_join). Both the narrow two-table path and the general join planner use it.

• secantus-pgserver: grouping sets whose slim rows are as wide as the rows are grouped from one read when every aggregate’s partials combine exactly (count, min / max, bool_and / bool_or, integer / numeric sum). When more groups than the bound holds turn up, it falls back to one read per set.

• secantus-pgplan: infer_param_types remembers its answer per statement text and declared types (per thread). libpq declares no parameter types, so a prepared statement was parsed again at every Execute (~0.9 us of a primary-key read).

Fixed

• secantus-pgplan: GROUPING(...) over a join (in the select list, ORDER BY or HAVING) was 42803 “arguments to GROUPING must be grouping expressions”; the expression walker now reaches its arguments.

• secantus-pgplan: GROUPING(...) in HAVING was 0A000; it is computed for the test like a hidden aggregate. A misplaced GROUPING(k) in HAVING is PostgreSQL’s 42803 with its position.

• secantus-pgplan: over a join, an ORDER BY expression (-g, name || 'x', grouping(name)) now resolves names to input columns. Only a bare ORDER BY key may name an output column, as in PostgreSQL. An output alias used inside an expression is 42703 (it was a confusing 42803 naming a hidden column).

• Corpus b68_wide_sets (36 lines, 0 against PostgreSQL 15.19, also with the join and group bounds at 20,000 bytes and at 1 byte, and with streaming off); slice test test_batch68_left_built_joins_and_hashed_grouping_sets.

The Rust PostgreSQL server scales with readers

Two point-read clients against the Rust PostgreSQL server used to get LESS done together than one did alone: 21,000 statements a second for one client fell to 14,400 for two. Each statement re-read the whole user catalog and copied every session setting whenever a worker thread switched from one connection to another, because both per-thread caches were keyed by the connection. They are now keyed by what they actually depend on, and the same benchmark scales like PostgreSQL 15 does: 1.81x at two clients (PostgreSQL 1.86x) and 3.6x at eight (PostgreSQL 3.6x).

Fixed

• secantus-pgserver: the planner’s per-session catalog tables are shared between connections with the same role, database and session user; a session with its own temporary tables or functions, or a database with row-level security, keeps a private copy.

• secantus-pgserver / secantus-pgplan: session settings reach the planner as a shared reference instead of a full copy per statement.

• secantus-pgserver: pg_get_serial_sequence on a table that does not exist raises 42P01, and on a column that does not exist 42703, as PostgreSQL does; both used to answer NULL.

• psycopg gauge: on Windows it deselects the proxy, timing and mypy markers, as psycopg’s own Windows CI does, and clears a crash flag left by an earlier interrupted run that otherwise stopped every later run.

• bench/pg_concurrency.py honours SECANTUSD_PG, so a worktree’s build can be measured.

Rust PostgreSQL server: citext, and the PostgreSQL gauges can drive it

Added

• CREATE EXTENSION citext on the Rust PG server, matching PostgreSQL 15. citext columns and casts compare, order, group, dedup and enforce UNIQUE / PRIMARY KEY case-insensitively.

  • LIKE / ~ on citext are case-insensitive, and citext’s own string functions (strpos, replace, split_part, the regexp family) match case-insensitively.

  • citext against a typed text value compares as text, as PostgreSQL resolves it.

  • Corpus citext: 79 lines, 0 divergences.

• The SQLAlchemy, pgjdbc, pgx and pgtest gauges can run against the Rust server through a shared pg_gauge_server.py switch.

  • Set SECANTUS_GAUGE_SERVER=rust. Results go to a -rust-server report beside the Python server’s, and a binary built from a different source tree is refused.

  • The SQLAlchemy dialect suite, run against the Rust server for the first time, found the schema gap now in the backlog.

Fixed

• A JOIN ON condition over a nondeterministic collation now matches case-insensitively. It used to match only rows with the same case.

• A grouped nondeterministic-collation column used inside an expression no longer raises 42803.

The Rust PostgreSQL server answers pg_constraint, and a regclass list no longer silently matches nothing

pg_constraint was the last catalog table a client could not read on the Rust PostgreSQL server: every query against it answered 42P01 relation "pg_constraint" does not exist, even though the server already enforced all four constraint kinds and recorded each one’s name. It is now a virtual table with PostgreSQL 14’s full 25-column shape, projected from the catalog the server already keeps.

Measuring it turned up a second, quieter bug one level down. A regclass value is carried as a document holding its oid, and only the scalar comparison path unwrapped it — so WHERE conrelid IN ('t'::regclass, 'u'::regclass) compared documents against numbers, matched nothing, and returned zero rows with no error, while the same predicate spelled with OR returned the right ones. That is the shape catalog reflection actually emits (SQLAlchemy and pgjdbc both use IN / = ANY), so it read as “this server has no constraints” rather than as a defect.

Added

• pg_constraint on the Rust PostgreSQL server, with PostgreSQL 14.24’s column set, attnum order and wire-type oids — including the two types the server had no vocabulary for: the internal single-byte "char" (18) that contype and the three conf*type columns use, and pg_node_tree (194) for conbin. Rows cover PRIMARY KEY (p, named <table>_pkey as PostgreSQL names an implicit one), UNIQUE (u), EXCLUDE (x), CHECK (c) and FOREIGN KEY (f) with its one-letter ON UPDATE / ON DELETE action codes and confkey resolved against the parent table.

• pg_constraint is listed in pg_tables under schemaname = 'pg_catalog', like the other catalog relations the server answers for.

Fixed

• A regclass or regtype operand inside IN (...) or = ANY(ARRAY[...]) is now compared by its oid, as the scalar = path already did. Previously such a predicate matched no rows at all and reported no error.

Notes

• NOT NULL deliberately produces no row. PostgreSQL records a not-null as pg_attribute.attnotnull rather than as a pg_constraint entry, so a table with a NOT NULL column, a CHECK, an FK, a UNIQUE and a PK has exactly five rows here — measured, and pinned by a test.

• Two columns are honestly empty rather than fabricated: conbin is NULL because the server keeps a CHECK predicate as SQL text, not as PostgreSQL’s serialised parse tree, and conindid is 0 because there are no pg_index rows for it to point at. Constraint oids are synthetic — distinct and stable per table, but not PostgreSQL’s numbers.

Rust PostgreSQL server: correlated subqueries over equalities run once

A correlated subquery used to be planned and run again for every distinct outer value. When every outer reference is an equality (WHERE t.x = o.a AND t.y = o.b), the inner query now runs once per statement and each outer row is answered by a hash lookup, the semi-join PostgreSQL itself plans.

Changed

• Timings at 2,000 × 2,000 rows (debug build):

  • EXISTS over two equalities: 7.3 s → 36 ms.

  • NOT EXISTS over two equalities: 7.4 s → 47 ms.

  • A correlated count(*): 1.1 s → 227 ms.

  • A correlated LIMIT 1 subquery: 7.6 s → 36 ms.

• Answers are unchanged. A shape the lookup cannot reproduce exactly falls back to the per-row path. That includes a non-equality correlation, a numeric key, a nondeterministic collation, and an inner ORDER BY, DISTINCT or function.

• New corpus correlated_hash (23 edge cases) matches PostgreSQL. Every corpus still matches.

Rust PostgreSQL server: correlated subqueries with comparisons, and subquery column names

Changed

• The once-per-statement path for correlated subqueries now also takes comparisons (<, <=, >, >=, <>) against the outer row, for numbers and timestamps. EXISTS (... t.x = o.a AND t.y > o.b) at 2,000 × 2,000 rows went from 7.6 s to 40 ms (debug build), with the same answers as PostgreSQL.

Fixed

• A scalar subquery’s column is named after its inner column, as in PostgreSQL. EXISTS and ARRAY(...) are named exists and array. Before, all three were ?column?.

• With the names fixed, ORDER BY id over two output columns named id answers PostgreSQL’s 42702 ORDER BY "id" is ambiguous. It used to sort by one of them silently.

Rust PostgreSQL server: column names with a dot or a leading $

A column named like "dot.s" or "$x" was read as a nested MongoDB-style path. WHERE matched nothing, UPDATE / DELETE reported rows but changed nothing, and UNIQUE was never enforced. These columns now behave as in PostgreSQL 15, pinned by the new dotted_columns corpus. The SQLAlchemy dialect suite against the Rust server is now 978 passed / 0 failed.

Fixed

• WHERE over such a column is evaluated against the literal key.

• UPDATE and ON CONFLICT DO UPDATE write the literal key.

• UNIQUE constraints and unique indexes over such columns are enforced. An automatic index name containing a dot no longer breaks its hidden index field.

ORDER BY over an expression, and a WHERE that cannot lower to a filter

Two more of the refusals the 2026-09-28 survey found. The predicate corpus against PostgreSQL 14.13 goes 14/17 → 17/17.

Added

• ORDER BY <expression> — order by n * -1, order by upper(a), and several expression keys in one clause. The expression is materialised per row into a synthetic field just before sorting, so the comparison stays one routine.

• A WHERE that does not lower to an MQL filter (where (case ... end)) is evaluated per row as a residual instead of being refused. Only TRUE keeps a row, so SQL’s three-valued logic is preserved.

Fixed

• The build-provenance stamp went stale in a git worktree. build.rs declared rerun-if-changed=../../.git/HEAD, but in a worktree .git is a FILE, so that path does not exist, cargo never re-ran the build script, and the stamp reported the tree of whichever checkout last built it — a staleness checker that is itself stale, which is the failure the block exists to prevent. All four stamping build scripts now resolve the path with git rev-parse --git-path.

Rust PostgreSQL server: aggregate FILTER with a subquery

count(*) FILTER (WHERE EXISTS (SELECT ...)), and any other FILTER condition holding a subquery, used to answer 0A000 SubLink is not supported yet.

Fixed

• An aggregate that skips NULL inputs now accepts a subquery in its FILTER, correlated or not. This covers count, sum, avg, min, max, string_agg, the bool_* and bit_* aggregates, and the variance family. The filter is rewritten to the CASE argument it is equivalent to.

• It works in the select list, HAVING and ORDER BY, and with GROUP BY and DISTINCT. Corpus filter_sublink matches PostgreSQL 14 on all 13 lines.

• NULL-keeping aggregates such as array_agg still refuse with 0A000.

GROUPING SETS, ROLLUP and CUBE on the Rust PostgreSQL server

The clause used to be parsed and then discarded: the GroupingSet node fell through to the expression arm, failed to resolve as a column, and the statement died with 42803 column "a" must appear in the GROUP BY clause — an error blaming the user’s own query for a clause the server had dropped. That is the second of the two such cases the 2026-09-28 survey found, and the last one.

Added

• GROUP BY GROUPING SETS (...), ROLLUP (...) and CUBE (...), including several constructs in one GROUP BY (their sets are crossed). Each set groups on its own keys and NULL-pads the others; sets concatenate in declared order and duplicate sets emit duplicate rows, as PostgreSQL 14.13 does.

• The empty set () groups the whole input into one row, and still returns that row when the input is empty.

Known limitation

• The GROUPING(col) function is refused 0A000, and the refusal names it rather than answering the generic “this target is not supported yet”. It reports which set produced a row, which needs the producing set carried through the group.

information_schema and the catalog views on the Rust PostgreSQL server

information_schema.columns — the single most-read relation in any PostgreSQL deployment, because every ORM, migration tool and \d starts there — did not exist. Nor did .tables, .table_constraints, .key_column_usage, .sequences, pg_class, pg_namespace, pg_index, pg_indexes or pg_attrdef; pg_attribute existed but listed only composite types, so attrelid = 't'::regclass found nothing for a table.

Against a live PostgreSQL 14.13 the catalog corpus went from 19 divergences of 22 to 4, and the sequences corpus finished at 0. A second corpus of 34 lines written for this change is clean.

Two findings worth stating:

Seven catalog functions already worked — but only as a bare select-list target, and the two halves have to be kept apart. Three of them (current_database, current_catalog, current_setting) must still DEFER to the connection when they stand alone: the server’s value is the live one, and current_setting has to see a set_config from earlier in the session. Making the expression form work by adding them to the scalar evaluator’s name list silently broke that — the bare-target gate matched them first and folded them, which a planner unit test with no session installed saw as “unrecognized configuration parameter” for a GUC that exists. CI’s rust job caught it; a test now pins both halves together.

The seven: version(), current_schema(), current_database(), current_setting(), format_type(), obj_description() and pg_get_expr() become a value the server resolves when they stand alone. Reached inside an EXPRESSION — version() LIKE 'PostgreSQL%', current_setting('x') ~ '...', obj_description(oid) IS NULL — the constant evaluator handled them instead and had nowhere to ask, so every one answered 0A000. Which is precisely how a client writes them.

information_schema’s views are called tables, columns and sequences — names a user table may perfectly well have, and a virtual relation wins over the catalog. Registering them bare would have made a user’s own columns table unreachable, so they keep their schema in the name and both resolve.

Added

• information_schema.columns / .tables / .table_constraints / .key_column_usage / .sequences, and pg_class / pg_namespace / pg_index / pg_indexes / pg_attrdef.

• The catalog functions reachable inside an expression, with current_database() and current_setting() reading a per-statement session snapshot the way the session user and the timezone already do.

• server_version_num, the numeric form every client that gates on a server version actually reads.

Fixed

• pg_attribute listed only composite types, so a table’s columns were missing entirely — and a table’s ROW TYPE is a composite under the same name and the same relation oid, so once tables were added every column appeared TWICE, once with attnotnull true and once false.

• pg_attribute gained attnotnull, atttypmod and atthasdef, without which a client cannot tell a nullable column from a NOT NULL one.

• format_type(oid, NULL) answered NULL. It is not null-propagating in its second argument: a NULL typmod means “no modifier”, and PostgreSQL answers integer.

• information_schema.table_constraints under-counted every table by one: PostgreSQL records a NOT NULL check for the PRIMARY KEY column too.

Rust PostgreSQL server: pgx is clean, pgjdbc runs to completion

The pgx gauge against the Rust PostgreSQL server went from 357 passed / 20 failed to 377 / 0. Before this change the pgjdbc gauge hung forever. It now completes: 5,475 of 5,637 tests pass and 134 fail. Each fix was checked against PostgreSQL 15 and pinned in the new jdbc_pgx corpus.

Fixed

• Several silent wrong answers:

  • A row error raised while rows streamed (select 0/0 from t) left the batch’s earlier writes committed and the block not aborted.

  • DEFERRABLE UNIQUE was never enforced.

  • A data-modifying CTE ran again at Parse and Describe.

  • 5/count(*) over no rows answered NULL instead of 22012.

• Two hangs:

  • nextval deadlocked on its own transaction after a table was dropped and re-created in one block.

  • An extended-protocol group kept its table and advisory locks after Sync.

• Temporary tables are per session (pg_temp_N): they shadow permanent tables and are dropped at disconnect and at DISCARD TEMP.

• Prepared statements and portals:

  • DEALLOCATE and DISCARD ALL also drop wire-level statements.

  • A failed Bind aborts the block.

  • A changed result shape reports cached plan must not change result type.

  • A suspended portal resumes.

  • ROLLBACK TO SAVEPOINT reports the right transaction status.

• Startup:

  • Startup parameters and options -c are applied.

  • server_version is reported as 15.0.

  • application_name is reported.

  • Protocol 3.2 requests are negotiated down to 3.0.

Added

• DISCARD.

• current_schemas(bool).

• regproc over the built-in functions.

• Empty-query responses.

• 22021 for a NUL byte in a text parameter.

• Binary COPY ending without its trailer.

• The pgjdbc runner creates its test database.

normalize() on the Rust PostgreSQL server

The one string function left after the last release needed something the others did not: a dependency. Unicode normalisation is table-driven — the composition and decomposition mappings are data rather than an algorithm — so there is no honest way to approximate it in a few lines. An ASCII-passthrough version would have answered most inputs correctly and a minority silently wrongly, which is exactly the kind of divergence this project treats as unacceptable, so it stayed refused by name until the dependency was agreed.

It is implemented now, on unicode-normalization, and all four forms match PostgreSQL 14.13: NFC (the default), NFD, NFKC and NFKD. A precomposed á decomposes to two codepoints under NFD and recomposes to one under NFC; the compatibility forms fold the fi ligature to fi where the canonical ones leave it alone.

Added

• normalize(text [, form]). The form arrives as an ordinary string constant — NFD and its siblings are grammar keywords, so an unknown one is a syntax error before the evaluator ever sees it.

Known limitations

• IS NORMALIZED and IS NORMALIZED NFD are a separate construct and remain unimplemented.

INSERT ... ON CONFLICT works on the Rust PostgreSQL server

The clause used to be parsed and then silently discarded, so insert ... on conflict do nothing raised 23505 where PostgreSQL inserts nothing and succeeds, and do update never upserted. A dropped clause is worse than an unimplemented one — the client gets a confident wrong answer instead of an honest 0A000 — and it was one of two such cases the 2026-09-28 survey found.

Added

• ON CONFLICT DO NOTHING and ON CONFLICT ... DO UPDATE SET ... [WHERE ...], with excluded.* reading the proposed row, assignments that mix the existing row and the proposed one (set v = t.v * 100 + excluded.v), a column-list / ON CONSTRAINT / bare arbiter, RETURNING over exactly the rows affected, and PostgreSQL’s row counts (a skipped row counts 0).

• An arbiter that matches no unique constraint is a PLAN-time error, as in PostgreSQL: 42P10 for a column list, 42704 for an unknown constraint name. The two codes differ, measured against PostgreSQL 14.13 rather than assumed.

Known limitation

• A partial-index arbiter (ON CONFLICT (a) WHERE ...) is refused 0A000. Inferring it needs partial indexes, which this server does not have; widening it to the unconditional index would absorb a conflict the predicate excludes, which is the silent divergence this change exists to remove.

LIKE, ILIKE, the regex operators and CASE on the Rust PostgreSQL server

Four of the seventeen refusals the 2026-09-28 survey found, closed together. 0/17 → 14/17 on a differential corpus against PostgreSQL 14.13.

Added

• LIKE / ILIKE / NOT LIKE / NOT ILIKE, anchored as SQL requires, with % and _ wildcards, every other character literal, and an explicit ESCAPE (which the parser folds into a like_escape(p, e) call rather than a third operand).

• The regex operators ~, ~*, !~, !~*.

• Both work as predicates and as values (select a like 'a%').

• CASE in the searched and simple forms, with lazy branch evaluation, NULL falling through as not-true, and a missing ELSE yielding NULL — including in a FROM-less SELECT, which routed targets through a separate allow-list.

Fixed

• n LIKE 'x' over a non-text column is now 42883 operator does not exist: integer ~~ unknown, as PostgreSQL has it. Lowering it to a regex anyway returned no rows, silently.

Known limitations

• CASE used directly as a bare WHERE predicate is still 0A000: a CASE does not lower to an MQL filter, and lower_where has no arm for it.

• ORDER BY over an expression is unchanged, still 0A000.

Rust PostgreSQL server: timestamptz, temp-table race, pgjdbc metadata

pgjdbc against the Rust PostgreSQL server went from 84 failures to 22 (7,354 tests); pgx stays at 377 passed / 0 failed. Every fix is checked against PostgreSQL 15 and pinned in the b39_fixes, b39_datetime and b39_catalog corpora.

Fixed

• A timestamptz copied from a column, a bound parameter or an array parameter was re-read as a wall clock. Under summer time that moved the value by an hour, silently.

• A newly created temp table was intermittently not found (42P01) under parallel sessions. The catalog version was bumped before the creating transaction committed.

• ROLLBACK TO SAVEPOINT undoes SET / SET LOCAL.

• pg_typeof(nextval(...)) no longer advances the sequence twice.

• Unary - / + on money is refused.

• Binary money parameters are decoded.

• Notices are sent while a statement waits on a table lock. LOCK TABLE works inside a function.

• A comma-join with no join keys in FROM now hashes the WHERE equalities. pgjdbc’s foreign-key metadata query no longer runs for minutes.

• Date/time:

  • fractional seconds round half-to-even;

  • +hhmm offsets are accepted;

  • timetz ± interval works;

  • BC binary dates decode.

• Binary varchar[] / bpchar[] / name[] results.

• LIKE with a parameter is described as boolean.

Added

• PostgreSQL’s read-only internal settings, such as max_index_keys.

• pg_get_keywords().

• Expression and partial indexes in pg_index / pg_get_indexdef.

• ADD PRIMARY KEY USING INDEX.

• COMMENT ON DOMAIN.

• Pseudo-types in pg_type.

• lseg ?# box.

• COPY ... HEADER.

• PL/pgSQL $n argument references.

• The pgjdbc runner creates a test role, as pgjdbc’s own CI does.

Rust PostgreSQL server: money, large objects, refcursors and database settings

The Rust PostgreSQL server gained the PostgreSQL 15 features pgjdbc’s suite was missing. pgjdbc now runs 7,354 tests, up from 5,637, because three classes no longer fail in setup. 84 fail, down from 133. pgx stays at 377 passed / 0 failed. Each feature is pinned by a new corpus checked against PostgreSQL 15.

Added

• The money type, transcribed from cash.c. It is stored as Decimal128, as on the Python server.

• Large objects:

  • every lo_* function, from SQL and over the Fastpath function-call protocol;

  • stored in the Python server’s collections;

  • pg_largeobject_metadata.

• PL/pgSQL OPEN for refcursors: FOR query, FOR EXECUTE and bound cursors. refcursor is now a type.

• ALTER DATABASE ... SET / RESET, applied at connect, and a pg_settings view.

• SET LOCAL and set_config(..., true) are undone at transaction end. A SET inside a rolled-back block is undone. ParameterStatus is re-sent when a value changes back.

• information_schema._pg_expandarray, and unnest over int2vector / oidvector.

• Named protocol portals appear in pg_cursors.

• Smaller additions:

  • getdatabaseencoding and pg_encoding_to_char / pg_char_to_encoding;

  • negative numeric scale;

  • every DateStyle keyword;

  • interval + datetime;

  • timestamp-to-time assignment casts;

  • inet / cidr / bit / varbit in pg_type;

  • INSERT ... SELECT with an uncorrelated subquery.

Rust PostgreSQL server: schema-qualified tables

The Rust PG server dropped the schema from every relation name, so CREATE TABLE test_schema.users was silently created in public and collided with public.users. Relations are now schema-qualified as in PostgreSQL 15, using the same catalog keys as the Python server (a non-public relation is schema.name), so each server reads the other’s tables.

Fixed

• schema.table works in CREATE / DROP / ALTER / RENAME / TRUNCATE, DML with RETURNING and ON CONFLICT, joins and subqueries, serial and identity sequences, UNIQUE / CHECK / FOREIGN KEY, indexes and views.

• Unqualified names resolve through search_path, and an unqualified CREATE lands in the path’s first existing schema. A missing schema answers 3F000.

• DROP SCHEMA RESTRICT refuses a non-empty schema with 2BP01, and CASCADE drops its contents.

• The catalogs report the real schema: pg_class / pg_namespace, pg_tables, pg_indexes, pg_views, pg_constraint, information_schema, regclass and to_regclass.

• nextval('s.seq') used the public sequence of the same name. It now resolves the schema.

• information_schema.sequences no longer lists identity-column sequences, matching PostgreSQL 15.

• Corpus schemas: 75 lines, 0 divergences. Two cross-server tests prove each server reads the other’s schema tables. The SQLAlchemy dialect suite against the Rust server went from 421 passed / 782 errors to 811 passed / 0 errors.

Sequences and identity columns on the Rust PostgreSQL server

CREATE SEQUENCE, nextval, currval, setval, ALTER SEQUENCE, DROP SEQUENCE and GENERATED ... AS IDENTITY were all missing. The machinery was already there — serial columns have drawn from a stored sequence document since the beginning — but none of it was reachable from SQL.

Against a live PostgreSQL 14.13 the sequences corpus went from 24 divergences of 26 to 1, and the one left is information_schema.sequences, part of the catalog-introspection gap. A second corpus of 70 lines written for this change is clean.

A sequence is also a RELATION, so SELECT last_value, is_called FROM s reads it like a one-row table, and pg_get_serial_sequence('t', 'id') names the sequence a serial column owns.

Three behaviours were measured rather than assumed, and each is a place a plausible implementation goes wrong:

The bound a sequence runs into depends on its DIRECTION. A descending sequence starts at its MAXIMUM and exhausts at its MINIMUM. Checking only max_value let CREATE SEQUENCE s INCREMENT -3 MINVALUE -10 run past its floor for ever, and named the wrong bound when it did stop. CYCLE wraps to the far bound instead of failing.

currval is keyed to the SESSION, not to the sequence. It is 55000 before any nextval in that session even when another session has advanced the sequence — reading the stored value instead would hand one session another’s number.

The identity overriding matrix has four live cases, measured on 14.24: GENERATED ALWAYS refuses a hand-written value with 428C9; OVERRIDING SYSTEM VALUE lets it through; OVERRIDING USER VALUE discards it and draws from the sequence for EITHER kind; and an explicit NULL is a not-null violation for both kinds, even under OVERRIDING SYSTEM VALUE — the override decides whose value wins, not whether the column may be null.

Added

• CREATE / ALTER / DROP SEQUENCE with START, INCREMENT, MINVALUE, MAXVALUE, CYCLE, RESTART [WITH], OWNED BY, IF NOT EXISTS and IF EXISTS; nextval / currval / setval (both arities); pg_get_serial_sequence; and a sequence read as a relation.

• GENERATED ALWAYS / BY DEFAULT AS IDENTITY columns, with OVERRIDING SYSTEM VALUE and OVERRIDING USER VALUE.

Fixed

• secantus-pgcatalog: a column’s catalog keys that this server does not model — identity, enum_type, domain_type, generated, comment, default_expr, composite_type, json_plain — were written back as unconditional NULLs, so any rewrite of a catalog row here ERASED what the Python server had recorded. That was unreachable while nothing rewrote an existing row; ALTER TABLE made it reachable last week. They are kept verbatim now, and identity is modelled outright because this server enforces it.

• tools/probes/pg_corpora/sequences.setup.sql: the identity tables were never dropped, so the REFERENCE server accumulated them across runs — CREATE TABLE idt11 answered 42P07 and count(*) grew by three every time. Four scenarios were comparing against that debris rather than against PostgreSQL’s behaviour.

Rust PostgreSQL server: the sqllogictest gauge, and what it found

The sqllogictest gauge can now run against the Rust PG server (SECANTUS_GAUGE_SERVER=rust). Its report goes to a separate .validation/slt-raw-rust-server.json, and it refuses a binary built from a different source tree. The first run passed 28 of 60 lane-files. These fixes came from it, each checked against PostgreSQL 15.

Fixed

• BETWEEN in every shape: NULL BETWEEN …, NOT x BETWEEN …, expression bounds and BETWEEN SYMMETRIC. These answered 0A000 this operator form is not supported yet. x NOT BETWEEN NULL AND hi now keeps the rows above hi (it matched nothing).

• SELECT DISTINCT over an aggregate or a grouped expression (it was 0A000). SELECT * / t.* with GROUP BY is fixed too (it was a 42803 naming an empty column).

• A correlated subquery whose inner FROM aliases the outer table’s own name (FROM t1 AS x WHERE x.b < t1.b) failed with 42P01. It now reads the outer row, as in PostgreSQL.

• An aggregate inside CASE, COALESCE, NULLIF or an IN (...) list is now computed. It was left out, and the expression answered NULL.

• Arithmetic over a numeric aggregate (- avg(x), avg(x) / count(*)) failed with 42883 operator does not exist: integer - text. It now works.

• A scalar subquery column takes its inner column’s name.

• The “must appear in the GROUP BY clause” error names the column qualified, as PostgreSQL does (tab2.col1).

Rust PostgreSQL server: SQLAlchemy reflection works

The SQLAlchemy dialect suite against the Rust PostgreSQL server went from 811 passed / 167 failed to 977 / 1. Every fix was checked against PostgreSQL 15 and pinned in the new sqlalchemy_catalog corpus.

Fixed

• pg_table_is_visible always returned true, so table and view listings included information_schema relations. It now follows search_path.

• Index names are per schema, not global. DROP INDEX s.ix and COMMENT ON INDEX s.ix resolve within that schema.

• pg_get_constraintdef quotes identifiers the way PostgreSQL does.

• Constraint comments now appear in pg_description / obj_description.

• PRIMARY KEY column order and INCLUDE columns are reported correctly in the catalogs.

• A DECLARE cursor now sees rows written earlier in the same transaction.

• Index listing reads inside the open transaction.

• A sequence created in the transaction is visible to regclass.

• Casts over joined columns and scalar subqueries in the select list now get PostgreSQL’s column names and types.

• LIMIT $1 inside a scalar subquery no longer fails at Describe.

FROM unnest(...) on the Rust PostgreSQL server

SELECT * FROM unnest(ARRAY[1,2,3]) is how a client turns an array into rows, and the Rust PostgreSQL server answered 0A000 … is not supported yet. So did regexp_split_to_table and generate_subscripts, and so did every clause a client puts around one of them — the ORDER BY, the WHERE, the count(*).

All of that works now. A set-returning function in FROM supports table and column aliases, * expansion, filtering, ordering, limiting and aggregates; an empty or NULL array yields no rows rather than an error; and a multidimensional array unnests to its leaves in row-major order, so unnest(ARRAY[[1,2],[3,4]]) is four rows rather than two. The same functions work as a bare target with no FROM at all.

The interesting part is how little code it took. The executor already had the right shape for this — its own comment explains that a generated source “is a SOURCE rather than its own statement” precisely because everything downstream works on documents and does not care where they came from. So a set-returning function is now planned as the same thing FROM (SELECT …) s produces, and the FROM-subquery path supplies all the clause handling for free. Nothing in the executor changed, and the two spellings of unnest(ARRAY[1,2]) — as a target and as a FROM item — share one implementation, so they cannot drift apart.

generate_series keeps the lazy path it already had. It is a range rather than a bounded list, and materialising generate_series(1, 10000000) into a vector to gain uniformity would have been a clear regression.

Added

• unnest, generate_subscripts and regexp_split_to_table as a FROM item, with WHERE, ORDER BY, LIMIT, aggregates (count(*), array_agg), AS t(col) and AS t aliasing, and * expansion.

• The same three as a bare select-list target with no FROM.

• unnest over a multidimensional array, yielding the leaves in row-major order.

Known limitations

• A set-returning function in the select list over a column — SELECT unnest(ia) FROM t — is still refused by name. That form changes the row count mid-pipeline rather than supplying the source, which is a different mechanism from the one added here.

• unnest(a, b) with several arrays zips and NULL-pads them; it is refused by name rather than answered as a single column.

• LATERAL forms are unaffected: a function in FROM that references the row to its left needs correlation machinery, not a materialised source.

Ten string functions, and three that were quietly wrong

Padding a string, converting one to hex, translating characters, overlaying a run, quoting a literal — the Rust PostgreSQL server could do none of them. Each answered 0A000 … is not supported yet, which is at least honest. Three others were worse than missing: they answered, and the answer was wrong.

split_part('a,b,c', ',', -1) was refused outright. PostgreSQL has counted from the end since version 14, so this rejected a form that works — and the error for a zero field named the wrong rule besides (“must be greater than zero”, where PostgreSQL says “must not be zero”).

regexp_replace('abc', 'b', '\&\&') returned a\&\&c. In PostgreSQL’s replacement text \& means the whole match, so the answer is abbc. The escape was passing through as literal characters.

And substring('abc' from '(b)') — the regex form every client actually writes — answered 42601 function substring does not exist with that argument list. That was not a missing function but a missing overload: the second argument’s type is what decides whether it is an offset or a POSIX pattern, and it was always read as an offset. An error naming the user’s call as malformed is worse than one naming a gap.

Measured against PostgreSQL 14.13: the strings corpus went 18 divergences out of 35 to 8, a new strings2 corpus of 27 shapes runs clean, and char_padding and jsonpath_number improved on the way past. Of the eight that remain, one is a locale artifact rather than a defect (this reference server runs lc_ctype = C, which does no non-ASCII case mapping), two are functions PostgreSQL 14 does not itself have, and two need set-returning functions.

Added

• lpad and rpad, which count characters rather than bytes, truncate when the target is shorter than the input, and cannot pad at all with an empty fill.

• to_hex, whose width follows the argument’s type: an int4 -1 is ffffffff and an int8 -1 is ffffffffffffffff.

• translate, which deletes the characters its target string does not cover.

• overlay(s placing r from p [for n]), with n defaulting to the length of r. A p below 1 answers PostgreSQL’s own 22011 substring_error — its own class, not the generic 22P02 a malformed value gets.

• quote_literal and quote_nullable. The pair differ only on NULL: quote_literal(NULL) is NULL, quote_nullable(NULL) is the four-character string NULL.

• regexp_split_to_array, unistr (all four escape spellings, with PostgreSQL’s 42601 for a bad one) and convert_from.

• substring(s FROM pattern) and substring(s FROM pattern FOR escape) — the POSIX and SQL-standard regex forms.

Fixed

• split_part accepts a negative field, counting from the end, and its zero-field error is PostgreSQL’s wording.

• regexp_replace expands \& to the whole match. An unknown escape such as \q still passes through as written, which PostgreSQL also does.

• substring with a negative length now reports 22011 rather than 22P02, matching the class PostgreSQL gives.

Subqueries and CTEs on the Rust PostgreSQL server

The Rust PostgreSQL server refused every form of subquery. (SELECT ...) as a value, EXISTS, IN (SELECT ...), a subquery in FROM, and WITH all answered 0A000 ... is not supported yet, which gated most real application SQL — a table of gaps in tasks/backlog.md named them the biggest single lever left on this server. Measured against a live PostgreSQL 14.13 with a 33-line corpus, 32 of 33 diverged.

They work now, in every uncorrelated form: scalar subqueries, EXISTS / NOT EXISTS, IN / NOT IN, ANY / ALL, ARRAY(SELECT ...), a subquery in FROM with an alias and an optional column list, and non-recursive WITH including several CTEs where one reads another. A widened 50-line corpus is at five divergences, and all five are correlated subqueries — refused by name (a correlated subquery ...), because a subquery whose value depends on the outer row has no single set of values to substitute, and substituting one would be a wrong answer rather than a missing feature.

Three bugs were found while building it, two of them wrong answers rather than errors, and all three reachable before this change or made reachable by it. The one that matters most is the correlation check itself: the lowering resolves a column by the last part of its name and ignores the qualifier, so EXISTS (SELECT 1 FROM emp e WHERE e.dept_id = d.id) bound the outer d.id to emp’s own id, planned cleanly, and answered true for every row.

Added

• secantus-pgplan / secantus-pgserver: uncorrelated subqueries in every form. An uncorrelated subquery is evaluated once during planning and replaced by the values it returned — which is what PostgreSQL does with one too — so IN (SELECT ...) lowers through the same ANY/ALL path that already had SQL’s three-valued rules right: an empty ANY matches nothing, an empty ALL matches everything, and a NULL in a NOT IN subquery makes the whole predicate NULL, so it returns no rows.

• secantus-pgplan: FROM (SELECT ...) s as a source in its own right, for plain and aggregate selects alike, with s(a, b) column aliases.

• secantus-pgplan: non-recursive WITH, rewritten into the FROM-subqueries it is shorthand for. WITH RECURSIVE and a data-modifying WITH are refused rather than inlined: the first has no subquery to expand into, and the second must run exactly once however many times it is referenced.

• secantus-pgplan: 21000 (cardinality_violation) for a scalar subquery that returns more than one row, as PostgreSQL reports it.

Fixed

• secantus-pgplan: a qualified aggregate argument read the FIRST part of the column’s name, so max(t.n) answered 42703 column "t" does not exist — over a plain table as much as over a subquery, and for as long as aggregates have existed.

• secantus-pgserver: an UNQUALIFIED column reference across a join whose left side is a subquery was assumed to belong to the right side, because a subquery side’s columns could not be probed by name. They can now, from the side’s own plan; before, select x, y from (select 1 as x) a, (select 2 as y) b answered (NULL, 2).

• secantus-pgserver: a subquery runs during planning, which is outside the transaction scope the statement’s execution runs in, so WiredTiger served it its own snapshot — insert; select count(*) from t where id in (select id from t) counted the rows from before the insert while the same query without a subquery counted correctly. The read now enters the open transaction.

• secantus-pgplan: walk_column_refs and the new subquery walk are one traversal rather than two over the same dozen node kinds — the drift the former’s own comment was already warning about.

Window functions on the Rust PostgreSQL server

sum(v) OVER (PARTITION BY g ORDER BY id), row_number(), rank(), lag, lead — none of them worked, and most did not even fail honestly. Ten were refused by name, but the seventeen most common shapes answered 42803 column "id" must appear in the GROUP BY clause: the planner matched sum on its NAME, never looked at the OVER beside it, and routed the query into the aggregate planner, which demanded a GROUP BY the query neither has nor needs. The client got an error blaming its own SQL for a feature the server did not have.

Measured against a live PostgreSQL 14.13, the existing window corpus diverged on 27 of 27 lines. Across four corpora totalling 89 lines it is now 86, and the three that differ are all refusals that name what is missing: a window over an aggregate, over a JOIN, and over a generated source.

Three of those corpora were written for this change, and the third one earned its place: the first two agreed with PostgreSQL on all 75 lines while a RANGE frame whose bounds sat on ONE side of the current row — RANGE BETWEEN 1 FOLLOWING AND 20 FOLLOWING — returned the whole partition on every row. Neither corpus contained such a frame. A corpus that agrees completely is evidence about the shapes it holds and nothing else.

The whole family works: row_number / rank / dense_rank / percent_rank / cume_dist / ntile, lag / lead / first_value / last_value / nth_value, and the aggregates as windows; PARTITION BY and ORDER BY over columns or expressions; ROWS, RANGE and GROUPS frames including value offsets; all three EXCLUDE forms; named WINDOW clauses; and FILTER.

One behaviour is worth stating because it is the one an implementation usually gets wrong: the DEFAULT frame is RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and under RANGE a bound at CURRENT ROW means the current row and its peers. So two rows that tie on the ORDER BY get the SAME running total, and a window with no ORDER BY at all sees the whole partition — the second falls out of the first, because with nothing to order by every row is a peer.

Added

• secantus-pgplan / secantus-pgserver: window functions, computed over the materialised rows after the WHERE and before DISTINCT / ORDER BY / LIMIT, which is PostgreSQL’s evaluation order. Each lands in a synthetic __winN field the select list projects, the same way a computed ORDER BY key already used __orderN.

• The aggregate windows go through the ORDINARY aggregate accumulator rather than a second implementation, so sum(int4) widens to int8, sum(numeric) stays exact and avg divides as numeric exactly as a GROUP BY does.

• 22014 for ntile(0), which PostgreSQL gives its own class rather than the generic 22023.

Fixed

• secantus-pgplan: ORDER BY now sees the select list’s output names, so select id * 2 as d from t order by d works. Independent of windows — it answered 42703 column "d" does not exist for any aliased column — but it is what makes a window usable, since ORDER BY rn is the only way to sort by one.

• secantus-pgplan: a window’s synthetic column reaches the def a subquery or CTE publishes, so select rn from (select row_number() over (...) as rn from t) s can name what the window computed.

• secantus-pgserver: the window columns are added to the def select_docs RETURNS rather than only in select_def. Every reader of a select’s schema takes the former, and without it a window value went over the wire as TEXT while its value was right — which a row comparison alone does not catch.

• secantus-pgplan: a window function beside a GROUP BY, over a JOIN, or over a generated source is refused by that name, instead of the false function sum() is not supported yet / function row_number() is not supported yet those paths used to produce.

• secantus-pgplan: a window function in a WHERE answers PostgreSQL’s own 42P20 window functions are not allowed in WHERE, and a RANGE offset over more than one ORDER BY column answers 42P20 with PostgreSQL’s wording.

The Rust PostgreSQL server: user aggregates and operators, table inheritance, hash partitioning, pg_trgm, and operators resolved by type

Several queries answered an empty result where PostgreSQL refuses them or returns rows. Each was silent: no error, just a wrong answer.

• Comparisons across type categories. A text column compared with an integer (WHERE s = 1) answered an empty result, as did a date compared with a number or a boolean compared with an integer. PostgreSQL refuses each at plan time with 42883 operator does not exist: text = integer. The Rust server now resolves operators by type as PostgreSQL does.

• A date column against a timestamp. WHERE d < now() matched nothing. The date is now promoted to midnight, in the session zone for a timestamptz.

• A binary FETCH from some cursors. Over VALUES, an aggregate or a constant select, the FETCH sent text bytes that a binary client decoded as garbage integers.

• A partition’s own constraints were not enforced. A PRIMARY KEY, UNIQUE or CHECK declared on a partition was accepted and never checked, so duplicate keys went in silently. They now hold for the rows the partition takes, whether written through the parent or the partition itself.

• text || numeric printed the value’s internal form (aDecimal128(96000...)) instead of a1.50.

• col::numeric(5,1) and col::timestamp(0) over a column ignored the modifier and kept every digit.

Errors now carry PostgreSQL’s position, so a client prints the caret under the token at fault.

psql’s describe commands work. With psql 15, the output of \d, \d+, \dt, \di, \dv, \dm, \ds, \df, \df+, \sf, \dT, \dD, \dp, \dn and \dx is byte-for-byte what PostgreSQL 15 prints, including the index, check, foreign-key, referenced-by, trigger and policy footers. \dt+’s Size column is the one exception: PostgreSQL counts its TOAST index pages, which this server does not have.

SQLAlchemy 2.1’s reflection matches. Its inspector answers the same as against PostgreSQL 15 for columns, keys, foreign keys, indexes (DESC ordering included), unique and check constraints, and comments. That covers pg_opclass, and pg_index.indclass / indoption / indcollation.

Added

• CREATE AGGREGATE, CREATE OPERATOR, CREATE STATISTICS, CREATE PUBLICATION, CREATE TABLESPACE and SECURITY LABEL, each with its catalog (pg_aggregate, pg_operator, pg_statistic_ext, pg_publication*, pg_tablespace).

• Table inheritance (INHERITS):

  • a read of a parent reads its descendants;

  • UPDATE / DELETE / the recursing ALTERs reach them too;

  • tableoid works on every table.

• PARTITION BY HASH, routed by PostgreSQL’s own lookup3 hash, so a row lands in the partition PostgreSQL would choose. Partition keys may be expressions, and a multi-column key compares in declared order.

• pg_trgm: similarity, word_similarity, the % family of operators and their thresholds, and trigram opclasses.

• ALTER VIEW (owner, rename, rename column, defaults, options, security_invoker). Views follow the renames of what they read. A view’s * is frozen at creation. pg_views and information_schema.views are populated.

• Other additions:

  • table locks held by a block’s statements, deadlock detection (40P01), and pg_locks;

  • DROP TABLE ... CASCADE;

  • ADD COLUMN ... serial;

  • ON CONFLICT arbiters on unique and partial indexes, and key-changing upserts;

  • DISTINCT on the extended aggregates;

  • STRICT functions;

  • function-style casts.

• RANGE window frames with interval offsets over dates and times, and with fractional offsets over numbers. ROWS and GROUPS offsets are read as bigint.

• A Bind asking for MIXED per-column result formats is honoured column by column. A format count that does not match the columns is 08P01.

• DDL waits for readers: ALTER TABLE, DROP TABLE, a rename and CLUSTER take ACCESS EXCLUSIVE, and CREATE INDEX takes SHARE, so each waits for (or, under lock_timeout, fails against) a session still using the table.

• A partition’s own column options in PARTITION OF (...): NOT NULL, DEFAULT, and column CHECK / UNIQUE / PRIMARY KEY.

• pg_get_viewdef(view [, pretty]), by oid, regclass or name.

• For psql and ORMs:

  • the relations pg_collation, pg_am, pg_policy, pg_depend, pg_sequence, pg_description and pg_publication_namespace;

  • the missing columns of pg_class, pg_index, pg_attribute, pg_type and pg_extension, with pg_attribute rows for view and index columns;

  • the functions pg_get_indexdef, pg_get_triggerdef, pg_get_partkeydef, pg_get_functiondef, pg_get_function_arguments / _identity_arguments / _result, pg_get_function_sqlbody, pg_get_userbyid, pg_partition_ancestors, pg_relation_is_publishable and the pg_*_is_visible family;

  • OPERATOR(pg_catalog.op) syntax;

  • PostgreSQL’s catalog relation oids;

  • a sequence resolves as a regclass, and nextval('s'::regclass) works.

Fixed

• Silent wrong answers, each measured against PostgreSQL 15:

  • ON UPDATE CASCADE now checks the child row’s other foreign keys, so a cascade onto a key another parent lacks is 23503 instead of committing.

  • Two unnamed foreign keys over one column are named t_a_fkey and t_a_fkey1, not both t_a_fkey. A named ADD CONSTRAINT that clashes is 42710.

  • int8 values above 2^53 compare exactly in ORDER BY, DISTINCT and window peers, and in RANGE frames.

  • A timestamp(p) / timestamptz(p) column stores its value rounded to p digits.

  • A stored timestamptz cast to timestamp or date is the session zone’s wall clock inside a WHERE, a nested cast and an aggregate, not only a plain projection.

  • An UPDATE onto an equal wide numeric primary key is 23505, and the error names t_pkey.

• null::no_such_type is 42704.

• Error codes and messages now match PostgreSQL:

  • abs(text) and the other numeric built-ins given a string are 42883, as are upper(1) and the other text built-ins given a number;

  • an unknown function is 42883 at plan time, even over an empty table;

  • negative frame offsets are 22013;

  • in a FROM-less select, a WHERE naming an output alias is 42703, and a WHERE naming no column is evaluated once;

  • a recursive CTE whose anchor is narrower than its UNION’s type is 42804;

  • CREATE AGGREGATE over a built-in operator function that does not take the declared types is 42883;

  • ORDER BY a name that two different output columns carry is 42702;

  • operators are also resolved by type inside a subquery.

• || renders each side as its ::text: a float4 in its own digits, a timestamptz in the session zone. An unknown literal beside a timestamp or interval is text there, not that type.

• A function parameter’s DEFAULT was accepted and ignored: a call that left the argument out was 42883. It now fills the call, and a later parameter without a default is refused (42P13).

• A domain’s CHECK constraints are listed in pg_constraint.

• An aggregate whose WHERE compares two columns over a join, such as SELECT count(*) FROM a, b WHERE a.id = b.id, was refused. So was a subquery correlated through a FROM function’s argument, such as ARRAY(SELECT ... FROM unnest(t.opts) x).

• pg_get_constraintdef(oid, true) drops a CHECK’s outer parentheses, and a CHECK shows PostgreSQL’s implicit casts (v > 0::numeric, '-3'::integer).

• pg_class.relhastriggers is true for both sides of a foreign key.

• A failed autocommit INSERT consumes the serial values it drew, as PostgreSQL’s non-transactional sequences do, so the next id matches. The statement’s own transaction used to roll the sequence back with it.

• A folded DEFAULT takes the column’s modifier as stored: numeric(4,1) DEFAULT 1.25 stores 1.3, on INSERT and on ADD COLUMN’s backfill.

• timestamp(p) / timestamptz(p) casts round to p digits, as PostgreSQL’s AdjustTimestampForTypmod does.

• The Python PG server has pg_catalog.pg_tablespace and the default_table_access_method setting, which SQLAlchemy 2.1’s table reflection reads.

• An untyped literal compared with a date column is read as a date.

• A parameterised recursive CTE (SELECT $1::int UNION ALL SELECT n + 1 ...) typed its column as text and failed with 42883. A VALUES column holding NULL under a cast now takes the cast’s type.

• Date and time handling:

  • time-only input forms and PostgreSQL 15’s full default zone-abbreviation set;

  • DateStyle in text casts and arrays;

  • to_date / to_timestamp stop at the end of the input, read ISO weeks, and number negative years as PostgreSQL does;

  • wide-year and BC timestamptz render in the session zone;

  • numeric and POSIX TimeZone settings are accepted;

  • daylight saving is applied past 2037.

• Float NaN compares above every number. Stored ranges render in the session zone, and a range casts to a multirange.

• information_schema.columns.column_default prints expressions as PostgreSQL’s ruleutils does: negative constants typed by the literal, implicit numeric casts shown, NOT / AND / OR kept rather than folded, and arrays as typed literals.

• MERGE fires each action’s statement triggers once, and acts on identical keyless rows as separate rows. Row-level security on a table read through a view applies the view owner’s policies.

The Rust PostgreSQL server verifies passwords, and gains LATERAL and EXPLAIN

A role created with a password now has to prove it. The Rust PostgreSQL server stored the SCRAM-SHA-256 verifier CREATE ROLE ... PASSWORD gives it, and then trusted every connection anyway – a wrong password, or none, logged in as the role. It now runs the SCRAM-SHA-256 exchange against the stored verifier, and refuses a NOLOGIN role. A role with no password, and a user the server has never heard of, are still trusted, which is what test fixtures connecting as a password-less postgres rely on.

LATERAL works, for subqueries and for functions (which are implicitly lateral, as in PostgreSQL), in comma, CROSS and LEFT joins. A set-returning function over a column in the select list – SELECT unnest(tags) FROM t – works, planned as the lateral join it means. EXPLAIN answers with the plan’s shape in PostgreSQL’s layout; it prints zero costs rather than invented ones.

Added

• SCRAM-SHA-256 password verification (secantus-auth gains the PostgreSQL form of the exchange: an empty client user name and the y,, header).

• LATERAL subqueries and functions; select-list set-returning functions; jsonb_array_elements[_text], jsonb_each[_text], jsonb_object_keys and their json_ twins; regexp_matches; multi-array unnest(a, b).

• EXPLAIN [ANALYZE], (COSTS OFF), (FORMAT JSON).

• COLLATE "C" (and POSIX / default / ucs_basic).

Fixed

• regexp_count / _instr / _substr / _like and to_ascii answer as PostgreSQL 14 does (42883, 0A000) rather than “not supported yet”.

• unnest(a) AS x names its column x, as PostgreSQL’s rule for a function returning one column has it.

Added (foreign keys)

• Multi-column FOREIGN KEYs, to a composite PRIMARY KEY or to a UNIQUE constraint; ON DELETE / ON UPDATE with CASCADE, SET NULL and the new SET DEFAULT. MATCH SIMPLE: a key with a NULL column references nothing.

The Rust PostgreSQL server: two silent data-loss fixes, rules, event triggers, foreign data, CREATE CAST and COLLATION, pgcrypto’s PGP, and view definitions as PostgreSQL prints them

Batch 11 started from a re-measured backlog: every open Rust-server entry was probed against PostgreSQL 15. Two of them were losing committed data without an error; both are fixed. The batch then closes the refusals and wrong answers that probe found, and adds CREATE CAST and pgcrypto’s encryption functions. Every change is measured against PostgreSQL 15 (and 14 where a corpus needs it).

Fixed

• A block’s earlier write could vanish. An explicit transaction that wrote, then collided with another session on a later statement, retried that statement on a fresh transaction and silently discarded its earlier writes, while every statement and the COMMIT reported success. The “has this transaction written?” check read a flag before the snapshot refresh that sets it. The block now fails whole with 40001.

• A prepared transaction lost its locks across a restart. Recovered from its record, it held nothing: another session could update one of its rows and commit, and COMMIT PREPARED then overwrote that commit. A prepared transaction is now revived as a live transaction when the store opens, so a conflicting write waits (55P03 under lock_timeout), as in PostgreSQL.

• inet / cidr ordered and compared as text. 10.0.0.1 sorted before 9.0.0.1, a < '9.255.0.0'::inet compared strings, min / max were wrong, and a = '10.0.0.1' missed the stored 10.0.0.1/32. They now follow PostgreSQL’s network_cmp.

• Deep and long expressions. A 10-term a || b || ... never finished (typing re-walked each operand per arm, about 5^depth). A 24-term chain overflowed a worker’s stack and aborted the whole server, and past 50 levels the parse tree failed to decode. Typing is now linear. Workers have a large reserved stack, and a statement nested past PostgreSQL’s own limit is its 54001, never a crash.

• now() is the transaction’s start, as in PostgreSQL, where it was the statement’s. statement_timestamp() and clock_timestamp() keep their own meanings.

• Window RANGE frames over numeric values past 15 significant digits compare exactly.

• greatest / least keep an array’s lower bounds.

• An aggregate over a derived source (VALUES, a subquery, a function) with a WHERE that is not a plain filter works; it was 0A000.

• DISTINCT ON over a grouped query works; it was 0A000.

• Set operations type their columns as PostgreSQL does:

  • varchar UNION name is name;

  • null::text UNION 1 is 42804, with its position.

• Error codes and messages now match PostgreSQL:

  • max(boolean), md5(1), substr(1, 1), array_length(1, 1) and to_hex(text) are 42883;

  • a cross-type comparison on a subquery or CTE column is 42883, as it already was for a table’s;

  • 1::inet is 42846.

• to_tsvector follows the C.UTF-8 locale the server reports: every letter lowercases, and only letters make words.

• SQL functions returning composites. A function returning a composite returns the whole row. A composite-returning function in FROM, and (f()).*, expand to its fields.

• Composite operands. A composite beside another type in an operator is 42883, as in PostgreSQL.

• Dropping a type that a function’s signature names now refuses (2BP01) or cascades, as PostgreSQL does.

• DROP FUNCTION f(), g() with several functions works. It runs as one transaction: when one is missing, none is dropped (it was 0A000).

• A PL/pgSQL record’s array field keeps its array type. r.a of a text[] column was typed text, so storing it failed with 42804.

• A computed column’s type modifier. x::numeric(5,2) in a select list or a view now reports numeric(5,2), in the row description and in pg_attribute, where it was bare numeric.

• pg_table_size counts a table’s TOAST index (8192 bytes) when a column can be TOASTed, as PostgreSQL does.

• Time zone abbreviations tied to a zone (MSK, VOLT, YAKT, …) mean that zone’s offset at the time given: 2012-01-01 12:00 MSK is UTC+4, as it was then. A zone abbreviation on a timetz ('12:00 EST'::timetz) is read; it was dropped.

• pg_class.relacl shows the grants on a table (grantee=arwd/owner, * for WITH GRANT OPTION, per privilege); it was always NULL. relhasrules is true for a table with a rule.

Added

• Collations:

  • ICU ones: any <tag>-x-icu name, and CREATE COLLATION ... provider = icu with the locale’s ks / kn / kf / co / ka keywords (de-u-co-phonebk, und-u-kn-true).

  • Applied in ORDER BY, comparisons, IN, min / max, greatest / least, DISTINCT, GROUP BY and UNIQUE indexes.

  • Nondeterministic (case-insensitive) collations compare equal where their strength says so; LIKE under one is refused, as in PostgreSQL.

  • A column’s COLLATE, which was silently dropped, is recorded and honoured.

  • COLLATE in an index key, DROP COLLATION with its column dependencies, and the 42P21 / 42P22 conflicts.

  • Catalog: pg_collation, pg_attribute.attcollation, information_schema.columns.collation_name, regcollation and pg_collation_for / COLLATION FOR.

  • Built on ICU4X, over the same CLDR data as the ICU PostgreSQL links.

• DISTINCT ON over an expression (DISTINCT ON (lower(x))).

• CREATE CAST / DROP CAST:

  • function, WITH INOUT and binary casts;

  • explicit, AS ASSIGNMENT and AS IMPLICIT contexts, through INSERT and UPDATE too;

  • pg_cast, with PostgreSQL’s built-in rows;

  • PostgreSQL’s validation and dependency errors.

• pgcrypto’s PGP functions:

  • pgp_sym_encrypt / pgp_sym_decrypt (and _bytea) with all of pgcrypto’s options;

  • pgp_pub_encrypt / pgp_pub_decrypt over RSA and Elgamal keys, password-protected keys included;

  • armor, dearmor and pgp_key_id.

  • Messages decrypt on PostgreSQL and the other way round. Blowfish and CAST5 are checked against GnuPG, since the reference servers’ OpenSSL 3 builds lack them.

• pgcrypto’s raw ciphers: encrypt / decrypt / encrypt_iv / decrypt_iv, byte for byte with PostgreSQL.

• The network functions: host, masklen, network, broadcast, netmask, hostmask, family, abbrev, text, set_masklen, inet_same_family, inet_merge, and the containment operators <<, <<=, >>, >>= and &&.

• CREATE RULE / DROP RULE on INSERT, UPDATE and DELETE: DO ALSO, DO INSTEAD, DO INSTEAD NOTHING and a WHERE, with pg_rewrite and pg_rules. ALTER TABLE ... ENABLE / DISABLE RULE and TRIGGER (including ALL and USER) work too.

• Event triggers: CREATE / ALTER / DROP EVENT TRIGGER on ddl_command_start, ddl_command_end and sql_drop, with TG_EVENT, TG_TAG, a WHEN TAG IN filter, pg_event_trigger_dropped_objects(), pg_event_trigger_ddl_commands() and pg_event_trigger.

• Foreign data: CREATE / ALTER / DROP FOREIGN DATA WRAPPER, SERVER, USER MAPPING and FOREIGN TABLE, with their options, dependencies and CASCADE, and the pg_foreign_* catalogs and information_schema views. There is no FDW handler here, so reading or writing a foreign table, and IMPORT FOREIGN SCHEMA, answer PostgreSQL’s own 55000 for a wrapper without one.

• View definitions as PostgreSQL prints them. pg_get_viewdef and pg_views.definition now follow ruleutils’ layout: qualified columns, typed literals, implicit casts written out, and parenthesised operators. The pretty form pg_get_viewdef(v, true) (which psql’s \d+ uses) works too. pg_rules.definition, and pg_get_expr over generated columns, CHECK constraints and policies, use the same printer.

• LANGUAGE internal wrappers are callable when the C function they name has a SQL form here (int4pl, textlen, upper, …).

• New corpora: user_casts, pgcrypto_ciphers, expr_depth, clocks, triage_fixes, collations, rules, event_triggers, fdw, catalog_b11, views_ruleutils, expr_ruleutils and internal_functions.

The Rust PostgreSQL server: rules as a real rewriter, a DELETE USING that deleted every row, overload choice by type, and HAVING without GROUP BY

Batch 12 started from the re-measured backlog, as batch 11 did: every open Rust-server entry was probed against PostgreSQL 15. Two findings were silent wrong answers, and both are fixed. Most of the rest of the batch replaces approximations with PostgreSQL’s own rules, transcribed from its source or its catalogs: the rewriter for rules, func_select_candidate for choosing a function overload, and pg_proc / pg_type for types. Each change is measured against PostgreSQL 15.

Fixed

• DELETE ... USING (subquery) deleted every row. A derived table inside the correlated subquery that USING is rewritten to was not recognised as local, so the correlation never bound and the predicate matched every row.

• HAVING without GROUP BY was ignored. select 1 from t having count(*) > 5 returned every row of t. The whole input is now one group, so the query yields one row or none.

• Rules ran as per-row triggers. A DO ALSO action ran once per row, and an UPDATE rule’s action ran after the update and saw the new rows. Rules now rewrite the statement as PostgreSQL’s rewriteHandler does: once per statement, actions before the original for UPDATE / DELETE, conditional INSTEAD rules negated into the original, and recursive rules refused (42P17).

• Function results had the wrong type. round / ceil / floor / trunc of an integer are double precision, as PostgreSQL picks the float8 overload (its category’s preferred type). abs(real) is real. Over a table column these came back as numeric.

• generate_series(smallint, smallint [, smallint]) is 42725 ... is not unique, as on PostgreSQL. A bigint series is bigint, and select from generate_series(...) is rows of no columns.

• A 2-D varchar[] / bpchar[] / name[] rendered as a 1-D array of the sub-arrays’ text.

• inet[] / cidr[] to text prints each element as inet_out does, dropping a full host mask.

• aclitem resolves roles made with CREATE ROLE. aclitem::oid is 42846, and oid has no arithmetic operators (42883).

• Under LATIN1 / LATIN9, a binary text array is transcoded element by element. A column name the encoding cannot hold is 22P05.

• Date and time input errors carry their position, and a month or day overflow carries PostgreSQL’s datestyle hint. to_date has PostgreSQL’s DETAIL lines and refuses an invalid day name. jsonpath keyvalue() ids follow the position of the value.

Added

• Functions resolve by argument type against PostgreSQL 15’s pg_proc signatures and implicit casts: a call no overload takes is 42883, and a call’s result types the expressions around it.

• Expression typing covers CASE, COALESCE, NULLIF, GREATEST / LEAST, ||, date and numeric arithmetic and scalar subqueries. Compared literals are coerced when the statement is analysed.

• Aggregates with no FROM (select sum(1), select count(*) having false).

• EXPLAIN prints Index Cond, Filter, Hash Cond, Join Filter and scan aliases.

• CREATE TYPE takes internallength, passedbyvalue, alignment, storage, category, receive / send and the rest, with PostgreSQL’s validation. pg_type carries the physical and I/O columns and array rows for the built-ins.

• information_schema lists the pg_catalog and information_schema relations and columns.

• Corpora builtin_overloads, assign_types, explain_quals, create_type_options, infoschema_system, func_types and b12_residuals, all at 0 divergences against PostgreSQL 15.

• The vendored pgwire fork’s two TLS unit tests generate their certificate per run. They used to fail on a key file that the repository’s *.key ignore rule keeps out.

The Rust PostgreSQL server: procedures, ALTER FUNCTION, column privileges, every RENAME, and PostgreSQL’s own catalogs

Batch 13 continued from the re-measured backlog. Two of its findings were security bugs, and both are fixed: column privileges acted as table privileges, and dropped tables’ grants survived. It also fills in a family of missing statements: procedures, ALTER FUNCTION and the RENAME forms. Every change is measured against PostgreSQL 15.

Fixed

• A column-level GRANT acted as a table-level one. After GRANT SELECT (v) ON t TO r, the grantee could read every column of t. Column privileges are now recorded in the shared __sql_column_grants__ catalog (the Python server’s shape) and checked against the columns a statement uses.

• Grants outlived their table. A table created under a dropped table’s name inherited its grants. DROP TABLE / DROP VIEW now drop them, a rename moves them, and a dropped column’s grants go with it.

• A procedure could be called with SELECT, which ran it as a function. That is now 42809 ... is a procedure, as on PostgreSQL.

• A PRIMARY KEY or UNIQUE constraint over a nondeterministic collation accepted apple beside Apple. Keys are now compared by the collation’s sort key, and count(DISTINCT ...) counts them the same way.

• A named CONSTRAINT ... PRIMARY KEY was reported as <table>_pkey.

• relacl lost the owner’s entry once every grant was revoked. relacl and attacl are now aclitem[], and attacl shows column grants.

• A function with OUT parameters returned NULL (plpgsql) or a single record column from select * from f().

• A void function’s value is now '', not NULL.

Added

• CREATE / CALL / DROP PROCEDURE, with OUT and INOUT parameters.

• ALTER FUNCTION / PROCEDURE / ROUTINE: RENAME, SET SCHEMA, OWNER TO, volatility, STRICT, SECURITY, LEAKPROOF, COST, ROWS, PARALLEL, and SET / RESET.

• CREATE FUNCTION without RETURNS (the result comes from the OUT parameters), and function schemas. Functions are stored in the shared shape, every parameter with its mode.

• ALTER ... RENAME for indexes, constraints, sequences, types (with enum values and composite attributes), domains (and their constraints), schemas, triggers and rules. Each rewrites every stored reference to the object.

• Rules: a rule action with several VALUES rows, DEFAULT in VALUES, and CREATE RULE "_RETURN" ... ON SELECT turning an empty table into a view. A set-operation action is refused at CREATE RULE (42P10).

• Event triggers:

  • table_rewrite fires, with pg_event_trigger_table_rewrite_oid() and _reason().

  • pg_event_trigger_ddl_commands() covers schemas, comments, renames, grants, routines, domains, sequences, types, triggers, rules, policies, statistics and publications, and has a command column.

  • The event functions raise 39P03 outside their event.

• Catalogs:

  • pg_class and pg_attribute list PostgreSQL 15’s own relations and their columns, and those relations resolve through ::regclass.

  • pg_type has a row type for each view and an array type for every user type, and pg_rewrite lists each view’s _RETURN rule.

  • pg_proc has argument modes, namespaces and owners.

  • pg_indexes prints COLLATE and ruleutils’ parenthesised expressions.

• Corpora column_privileges, rule_shapes, nondeterministic_keys, procedures, alter_routines, event_trigger_coverage, renames and catalog_system, all at 0 divergences against PostgreSQL 15.

The Rust PostgreSQL server: transaction control in procedures, semi-joins, join pushdown, recursive views

Batch 14 adds transaction control to procedures and DO blocks, and plans EXISTS and selective joins far faster. It also fixes three silent-data bugs. Each change was probed against PostgreSQL 15, and the corpora run against PostgreSQL 14.

Fixed

• A failing DO block left its earlier writes committed. A block run on the function interpreter wrote each statement on its own. Its statements now join the session’s transaction.

• A caught exception did not undo its block’s writes. A BEGIN ... EXCEPTION block now runs as a subtransaction, as PostgreSQL’s does.

• COMMIT in a SQL-language procedure crashed the connection. It is now 0A000, as on PostgreSQL.

• VALUES columns take the rows’ common type. A string beside a numeric was summed as 0.

• NaN and -0 group as PostgreSQL groups them in GROUP BY, DISTINCT, count(DISTINCT) and window partitions.

• A COLLATE inside a VALUES list, subquery or CTE carries out to the derived column. GROUP BY under a nondeterministic collation answers the group’s first value.

• An aggregate under a subscript, (array_agg(x))[1], is an aggregate.

• A view over SELECT * FROM (VALUES ...) v had no columns.

• A table column-alias list, FROM t r(a, b), answered 42703.

Added

• COMMIT / ROLLBACK [AND CHAIN] in procedures and DO blocks. They are allowed only outside a transaction block, as on PostgreSQL; anywhere else they are 2D000.

• CALL inside PL/pgSQL, writing OUT / INOUT values back to variables.

• CREATE VIEW ... WITH RECURSIVE, and recursive CTEs inside FROM subqueries.

• pg_get_viewdef prints PostgreSQL’s form for more view shapes:

  • VALUES;

  • ROWS / GROUPS frames with EXCLUDE;

  • LATERAL and column-alias lists;

  • a set operation’s ORDER BY / LIMIT / OFFSET;

  • WITH RECURSIVE and CTE column lists;

  • renamed relations (t t_1).

Performance

• A join applies each single-table WHERE condition to that table’s own scan, so the scan can use an index. Only tables on the preserved side of an outer join are filtered early. A selective join over 20,000 rows went from 731 ms to 1.7 ms.

• A WHERE-level EXISTS / NOT EXISTS over one equality runs as the uncorrelated IN / null-safe NOT IN it equals. Over 2,000 x 2,000 rows, EXISTS went from 7.2 s to 46 ms.

• Window partitions are hashed rather than scanned.

New corpora: values_common_type, derived_collation, procedure_transactions, join_pushdown, semi_join, viewdef_shapes and srf_fields. All are at 0 divergences, and so is every other corpus.

The Rust PostgreSQL server: expression indexes that work, numeric and NOT IN at index speed, error positions

Batch 15 makes several common shapes fast instead of quadratic or full-scanning. It also corrects error positions and static type checks. Each change was probed against PostgreSQL 15, and the corpora run against PostgreSQL 14.

Fixed

• A UNIQUE expression index (lower(email)) cost a full table evaluation per write. An INSERT took 1.6 s at 20,000 rows; it now takes 0.9 ms at any size.

  • Each row keeps the expression’s value in a hidden field, and the storage index on that field enforces UNIQUE.

  • Rows another writer left (the Python server knows no expression indexes) are recomputed when the server opens the store.

• An error raised while rows stream, or for a record without the named field, now carries PostgreSQL’s position.

• integer + boolean, string arithmetic and jsonb arithmetic on a composite field are rejected at plan time (42883), as on PostgreSQL, even over an empty table.

Added

• ON CONFLICT (expression) arbitrates on a unique expression index.

Performance (debug builds)

• A WHERE over an indexed expression uses the index: lower(t) = 'x' at 20,000 rows went from 2.4 s to 3.7 ms.

• A numeric predicate uses the column’s index: n = 5 went from 820 ms to 1.2 ms, and BETWEEN from 1.9 s to 17 ms.

  • Storage answers an $or whose every branch has an index as the union of the branches’ lookups (mongod’s OR plan).

  • Each indexable $or in an AND is intersected with the others.

• NOT IN (subquery) and NOT EXISTS went from 7.3 s to 34 ms over 2,000 x 2,000 rows.

  • NOT (x = ANY (...)) now lowers to an index-aware filter.

  • A scan hashes a large $in / $nin list once instead of comparing every row against every element.

The MongoDB server

The storage and matcher changes above apply to the Rust MongoDB server too:

• an indexed $or uses its indexes;

• a long $in / $nin list is hashed;

• explain reports the OR plan as mongod 8.2.11 does: SUBPLAN, FETCH, OR, and an IXSCAN per branch.

New corpora: not_in_large, numeric_index, error_positions, expr_unique, expr_where. All are at 0 divergences.

The Rust PostgreSQL server compares quoted literals correctly, and gains expression defaults

A quoted literal compared with a column is PostgreSQL’s unknown-typed constant, and it takes the column’s type. The Rust PostgreSQL server left it a string, so WHERE n > '5', WHERE ok = 't' or WHERE created > '2026-03-01' compared a string against a number, boolean or timestamp and matched nothing – a silent empty answer, on every quoted number, boolean, timestamp and interval in a WHERE, an IN list or a BETWEEN. The literal is now coerced to the column’s type, and a timestamp comparison accounts for the sub-millisecond remainder the storage keeps beside the millisecond value.

Column DEFAULTs may now be expressions – now(), CURRENT_DATE, gen_random_uuid(), nextval('s') – evaluated for each inserted row (and each existing row, for ALTER TABLE ADD COLUMN) rather than refused. INSERT ... DEFAULT VALUES, VALUES (DEFAULT, ...) and UPDATE ... SET c = DEFAULT work. Sequence functions work anywhere in an expression, not only as a bare select-list target. Window functions work over an aggregate, over generate_series, and inside an expression.

Added

• secantus-pgplan / secantus-pgserver: expression column defaults, stored in the Python server’s default_expr catalog key; DEFAULT VALUES, the DEFAULT keyword in VALUES and SET.

• nextval / currval / setval / lastval inside any expression, through a sequence hook the executor installs; lastval().

• gen_random_uuid() / uuid_generate_v4(), random(), and CURRENT_DATE / CURRENT_TIME / LOCALTIME / LOCALTIMESTAMP anywhere in an expression.

• A window function over an aggregate (sum(sum(v)) OVER (...)), over generate_series, and nested in an expression (v - avg(v) OVER ()).

Fixed

• A quoted literal compared with a non-text column matched nothing.

• A timestamp comparison ignored the stored microsecond remainder, so t = '...123456' and t > '...123' answered wrongly.

• A volatile SET value (SET n = nextval('s')) was evaluated once for every row instead of once per row.

• now()::date and the other timestamptz casts to a zone-less type failed in a constant expression.

Added (composite keys)

• A composite PRIMARY KEY, stored as the Python server’s subdocument _id (its columns in table order), so either server reads and enforces the other’s. A duplicate names the key’s columns in its DETAIL.

Fixed (grouping)

• A QUALIFIED grouped column in the select list (select c.a, count(*) from t c group by c.a) was a 42803 naming the alias c: the target read the first name part rather than the column.

The Rust PostgreSQL server: UPDATE … FROM, updatable views, exact numeric math, and PostgreSQL’s full date/time input

UPDATE ... FROM and DELETE ... USING used to ignore their extra FROM and write every row the statement’s own WHERE allowed. They now touch only the joined rows. RETURNING may read the joined item, and an ambiguous unqualified column is 42702, as in PostgreSQL.

A view over a single table is now automatically updatable. INSERT / UPDATE / DELETE through it are rewritten onto the base table, a computed view column is read-only, and WITH [LOCAL | CASCADED] CHECK OPTION refuses a row the view would not show (44000). A READ ONLY transaction, or default_transaction_read_only, now refuses every write, nextval() included (25006); before, it wrote.

Integer arithmetic overflows as PostgreSQL does (22003 integer out of range). Before, an int4 result silently widened to int8. The numeric transcendentals (sqrt, exp, ln, log, power, ^) are exact, at PostgreSQL’s result scale, instead of float approximations.

Date/time input now goes through a transcription of PostgreSQL’s DecodeDateTime. Jan 5, 2020, 5 January 2020 10:30 PM, 1/5/2020 (in the session’s DateStyle order), 20200105T103000, J2458854, y2020m01d05, zone names and abbreviations, and today / tomorrow all read as they do on PostgreSQL. AT TIME ZONE is implemented.

Added

• bit(n) / bit varying(n): literals, casts to and from integers and text, & | # ~ << >> ||, get_bit / set_bit / bit_count / length / position / substring / overlay, binary wire format; the integer bitwise operators.

• AT TIME ZONE / timezone(), with named zones, abbreviations, POSIX offsets and intervals.

• scale(), min_scale(), trim_scale(), numeric_send(), and numeric(p,s) rounding and overflow (22003) on casts and assignment.

• Expression indexes, CREATE [UNIQUE] INDEX ... (lower(email)): a unique one is enforced on INSERT and UPDATE. Index keys may declare NULLS FIRST / LAST.

• ORDER BY over an aggregate result (ORDER BY count(*) DESC, by alias or position); GROUPING(); an aggregate over a WHERE that does not lower to a filter (WHERE lower(email) = ...).

• generate_series over numeric, and over date / timestamp / timestamptz with an interval step.

• EXPLAIN (FORMAT YAML | XML), and Parent Relationship / Alias / Parallel Aware in the structured formats.

• SET (a, b) = (1, 2), subqueries in INSERT ... RETURNING, has_*_privilege(), to_regclass(), IS [NOT] TRUE / FALSE / UNKNOWN, and the "char" type.

Fixed

• UPDATE ... RETURNING / DELETE ... RETURNING with bound parameters sent rows without a RowDescription, which a client rejects.

• A float4 is rounded to single precision and prints as float4out does (0.33333334).

• CASE / COALESCE / greatest / least answer the branches’ common type; real + int is double precision; a negative LIMIT / OFFSET is 2201W / 2201X.

• upper / lower / initcap use the simple case mapping (upper('ß') is ß); lc_collate / lc_ctype report C.UTF-8.

• A DST-ambiguous or skipped local time resolves to the offset PostgreSQL picks.

• The information_schema views report varchar / name / "char" column types, and column_default shows a folded default as written ((1 + 2)).

• A wide-year or BC timestamptz prints with its zone offset.

The Rust PostgreSQL server runs user-defined functions and triggers

CREATE FUNCTION ... LANGUAGE sql and LANGUAGE plpgsql now define functions the Rust PostgreSQL server can call — in the select list, in WHERE, in FROM as a set-returning function, and from each other (recursion included). PL/pgSQL runs on a new interpreter over the real PostgreSQL grammar (libpg_query’s PL/pgSQL parser), so a body the reference server rejects is rejected here at CREATE time too. Functions are stored in the Python server’s __sql_functions__ shape, so either server can call the other’s.

CREATE TRIGGER works: BEFORE and AFTER, FOR EACH ROW and FOR EACH STATEMENT, over INSERT, UPDATE, DELETE and (statement-level) TRUNCATE, with UPDATE OF, WHEN (...), TG_ARGV and every TG_ variable. A BEFORE row trigger can rewrite the row (NEW.x := ...) or skip it (RETURN NULL); an AFTER trigger can write elsewhere; an error raised in any trigger aborts the whole statement. pg_trigger lists them.

DO blocks outside the small subset they already supported — DECLARE, IF, loops, EXCEPTION handlers, SELECT ... INTO — now run on the same interpreter instead of being refused.

Added

• CREATE [OR REPLACE] FUNCTION in LANGUAGE sql / plpgsql: scalar, SETOF, RETURNS TABLE and OUT parameters.

• A PL/pgSQL interpreter: blocks and EXCEPTION handlers (condition names and SQLSTATEs), assignment, IF / CASE, LOOP / WHILE / integer FOR / query FOR / FOREACH, EXIT / CONTINUE, RETURN / RETURN NEXT / RETURN QUERY, RAISE with USING, PERFORM, SELECT ... INTO, EXECUTE, GET DIAGNOSTICS, FOUND, ASSERT.

• CREATE [OR REPLACE] TRIGGER, DROP TRIGGER [IF EXISTS], pg_trigger.

Fixed

• DROP FUNCTION refuses (2BP01) while a trigger calls the function, and CASCADE drops the trigger; DROP TABLE drops the table’s triggers.

• A statement that runs user code is atomic outside a block too: a DO block or trigger that raises after writing takes those writes with it (a DO that ran EXECUTE 'insert ...' and then raised used to leave the row).

• ROLLBACK TO SAVEPOINT undoes what a trigger or function wrote to tables other than the statement’s own target.

• A call at an arity no user function has answers PostgreSQL’s 42883 function f(integer) does not exist.

The Rust PostgreSQL server gains general JOINs, correlated subqueries, views and indexes

The Rust PostgreSQL server (secantusd-pg) used to support a JOIN only in the narrow two-table shape psycopg’s catalog queries take. It now plans any JOIN as a source: inner, LEFT, RIGHT, FULL and CROSS joins, USING and NATURAL, joins of three or more relations, subqueries and set-returning functions as join sides, and every clause over them (WHERE, GROUP BY / HAVING, window functions, DISTINCT, SELECT * and t.*). A bare column name that both sides have is PostgreSQL’s 42702 rather than the left side’s value taken silently.

Correlated subqueries (EXISTS, NOT EXISTS, IN, ANY / ALL, ARRAY(...) and scalar subqueries that read the outer row) now answer per row instead of being refused, including in an UPDATE’s SET and in an UPDATE or DELETE WHERE. CREATE [UNIQUE] INDEX / DROP INDEX and CREATE [OR REPLACE] VIEW / DROP VIEW are implemented, in the same on-disk shape the Python SQL server uses, so either server reads the other’s views and indexes.

Added

• secantus-pgplan / secantus-pgserver: general JOIN planning (joins.rs), hash-joined on the ON clause’s column equalities and always decided by the full ON predicate.

• secantus-pgplan / secantus-pgserver: correlated subqueries, evaluated per outer row and memoised by the outer values (correlated.rs).

• CREATE [UNIQUE] INDEX (btree / hash, DESC, INCLUDE, a partial WHERE, IF NOT EXISTS, PostgreSQL’s default <table>_<cols>_idx naming) and DROP INDEX, reported by pg_indexes. A unique index admits many NULLs, and a violation names the index.

• CREATE [OR REPLACE] VIEW (with a column list and WITH CHECK OPTION) and DROP VIEW [CASCADE], with PostgreSQL’s OR REPLACE column rules (42P16) and dependency errors (2BP01) for a view’s table or view.

• IS [NOT] DISTINCT FROM, as a filter and as a value.

Fixed

• A WHERE on the nullable side of a LEFT JOIN ran before the join and kept the NULL-extended rows PostgreSQL drops.

• The narrow join path took a bare column name both sides have from the left side, and accepted a qualifier naming neither side.

• A subquery calling nextval() (or another volatile function) ran once per PLAN – a Describe and an Execute advanced the sequence twice. It now runs only in the plan that executes.

• NULL > ALL (empty set) answered NULL; PostgreSQL answers TRUE.

• A missing qualified column is reported as column e.nosuch does not exist, as PostgreSQL words it.

The Rust PostgreSQL server: MERGE, row-level security, table locks and timeouts, and autocommit writes that no longer lose updates

An autocommit UPDATE ... SET n = n + 1 now runs in a transaction of its own. Before, a row changed by another session between this statement’s read and its write was silently overwritten: the committed value was lost, where PostgreSQL waits for the other writer and builds on its result. A write that collides with another transaction’s now waits, and re-evaluates against the row that transaction left, as READ COMMITTED does. The wait honours a cancel, statement_timeout and lock_timeout.

A primary key column accepted NULL, and did so repeatedly. It is now NOT NULL, as in PostgreSQL. A table rewrite (ALTER COLUMN TYPE, CLUSTER) and an autocommit COPY FROM are now atomic. A duplicate key midway through either used to leave the table half-written.

The server reports PostgreSQL 15.0, and now has what 15 added: MERGE, regexp_count / regexp_instr / regexp_substr / regexp_like, and UNIQUE NULLS NOT DISTINCT. SQL/JSON syntax from 16 and 17 is answered as 15 answers it. The differential probe can run a corpus against a PostgreSQL 15 reference (# reference-version: 15).

Added

• MERGE INTO ... USING ... ON ... WHEN [NOT] MATCHED [AND ...] THEN UPDATE | DELETE | INSERT | DO NOTHING, with PostgreSQL 15’s 21000 for a target row matched twice.

• Row-level security enforced: permissive and restrictive policies per command, WITH CHECK on new rows, FORCE, BYPASSRLS, and the SELECT policies on an UPDATE or DELETE that reads its rows.

• Table privileges read off the SQL as written. A view is checked as the view, and its base tables as the view’s owner. A subquery anywhere in the statement, correlated or not, is checked with it.

• Arrays whose lower bound is not 1 ('[0:1]={a,b}', array_fill(v, dims, lbounds), an assignment below or past the bounds). They keep their bounds through text and binary I/O, subscripts, array_lower / array_dims, equality, and the functions that carry them.

• Timeouts and table locks:

  • statement_timeout (57014) and lock_timeout (55P03).

  • LOCK TABLE ... IN <mode> MODE [NOWAIT], with PostgreSQL’s conflict table.

• Maintenance statements:

  • CLUSTER, which rewrites a table in an index’s order.

  • VACUUM, ANALYZE, CHECKPOINT and REINDEX, with PostgreSQL’s validation.

• ALTER TABLE forms:

  • ADD UNIQUE, ADD PRIMARY KEY (the rows are re-keyed) and ADD FOREIGN KEY, each checked against the existing rows.

  • ADD COLUMN with inline constraints.

  • ALTER COLUMN TYPE ... USING.

  • ADD CHECK ... NOT VALID / VALIDATE CONSTRAINT.

  • CLUSTER ON / SET WITHOUT CLUSTER.

• SQL PREPARE / EXECUTE / DEALLOCATE, listed in pg_prepared_statements with from_sql.

• WITH ORDINALITY, and ROWS FROM (f(...), g(...)).

• Several set-returning functions in one select list, run in lockstep.

• A bare VALUES with ORDER BY / LIMIT / OFFSET.

• (a, b) IN (SELECT ...) and NOT IN.

• New functions and aggregates:

  • range_agg and range_intersect_agg.

  • trim_array.

  • IS [form] NORMALIZED.

• Triggers:

  • INSTEAD OF triggers on views.

  • Constraint triggers and SET CONSTRAINTS.

  • Transition tables (REFERENCING OLD/NEW TABLE).

• User-defined functions:

  • Overloads.

  • VARIADIC.

  • SQL-standard bodies.

• Types:

  • The geometric types.

  • regnamespace, regrole, regproc and regprocedure.

• pgcrypto and jsonpath .datetime().

• Roles:

  • Role membership: IN ROLE, GRANT role, pg_has_role, pg_auth_members.

  • md5 passwords and VALID UNTIL.

Fixed

• A parameter compared with a column (WHERE id = $1, SET n = $1) is described with the column’s type.

• WITH ORDINALITY and ROWS FROM silently dropped a column, and a bare VALUES ignored its ORDER BY. A FROM subquery with one output name twice answered the second column for both.

• An array_cat with an untyped literal beside an array now resolves. A bare boolean column works in a FILTER, and NOT of one too.

• POSIX character classes ([[:alpha:]]) follow the UTF-8 ctype, and pg_database reports the C.UTF-8 the server behaves as. A strict jsonpath datetime template reports trailing or missing input.

• A storage insert’s rejected rows were dropped by several callers. Every one now reports the first.

The Rust PostgreSQL server gains full-text search, formatting, datetime functions, SQL/JSON paths and the statistical aggregates

Full-text search now works on the Rust PostgreSQL server: to_tsvector, to_tsquery and its plainto_ / phraseto_ / websearch_ forms, @@, phrase operators, weights, ts_rank / ts_rank_cd, ts_headline and the tsvector editing functions. It follows PostgreSQL 14’s english configuration (Snowball stemmer, stop list, parser token types) and its simple one. A tsvector column written by either server can be read and searched by the other.

to_char / to_number for numbers and to_char / to_date / to_timestamp for dates and times are transcribed from PostgreSQL’s formatting.c, and so is the rest of the datetime family: extract / date_part, date_trunc, age, justify_*, make_*, date_bin, isfinite, BC dates, and time / timestamp arithmetic.

SQL/JSON path is implemented: the jsonpath type, jsonb_path_query (as a set-returning function), jsonb_path_query_array / _first / _exists / _match, @? and @@, in lax and strict modes, with vars and silent.

Added

• Statistical aggregates: var_* / stddev_* / variance / stddev (exact over numeric), corr, covar_*, every regr_*, and bit_and / bit_or.

• Ordered-set and hypothetical-set aggregates: percentile_cont / percentile_disc (scalar and array), mode(), and rank / dense_rank / percent_rank / cume_dist WITHIN GROUP.

• json_agg / jsonb_agg / json[b]_object_agg, to_json[b], json[b]_build_object / _build_array, json_object, array_to_json, row_to_json, json[b]_strip_nulls, with PostgreSQL’s per-function spacing.

• A whole-row reference (SELECT t FROM t, row_to_json(t)) and named record fields.

• char(n) semantics: values pad on output (including GROUP BY keys and record fields) and compare blank-insensitively. An assignment that is too long is 22001, unless the excess is blanks, which are dropped.

• IN / NOT IN and SIMILAR TO over constants; a constant WHERE (true / false / NULL / a parameter).

Fixed

• The Python PostgreSQL server reads a tsvector written by the Rust server, and treats a string opposite a tsquery in @@ as a tsvector.

The Rust PostgreSQL server: partitioning, row-level security, WITH RECURSIVE, domains, materialized views, READ COMMITTED, the xml type, and every text-search language

Inside a transaction block, each statement now sees what other connections committed before it, as PostgreSQL’s default READ COMMITTED does. That covers rows, and tables created since the block began. An UPDATE of a row another connection changed since the block began no longer fails with an internal WriteConflict. One limit comes from WiredTiger: once a block has written, it keeps its snapshot, as REPEATABLE READ would. A write that still conflicts now reports PostgreSQL’s retryable 40001, not XX000.

Added

• Declarative partitioning, PARTITION BY RANGE | LIST:

  • CREATE TABLE ... PARTITION OF ... FOR VALUES FROM / TO / IN / DEFAULT, including multi-column ranges, MINVALUE / MAXVALUE, NULL list members and sub-partitions.

  • A row written to the parent goes to its partition. A row that fits none is 23514 no partition of relation ... found for row. A row written to a partition must satisfy the partition’s bound.

  • An UPDATE that changes the key moves the row between partitions.

  • ATTACH / DETACH PARTITION move the rows. DROP and TRUNCATE of a partition remove its rows. COPY works in and out of partitions.

  • ALTER TABLE ADD COLUMN on the parent reaches every partition.

  • Overlapping, empty and mismatched bounds are refused with PostgreSQL’s errors.

  • Reflected in the catalog: relkind 'p', relispartition, pg_get_expr(relpartbound), pg_inherits, pg_partitioned_table, and per-partition indexes.

  • The tableoid system column names the partition a row is in.

  • A duplicate key names the partition’s own constraint.

• CREATE INDEX ... USING gin | gist | brin | spgist, with PostgreSQL’s operator-class rules. A type with no default class is 42704, and an explicit class such as jsonb_path_ops or text_pattern_ops shows in pg_indexes. The btree_gin and btree_gist extensions add the scalar classes. Dropping one refuses (2BP01) or cascades to the indexes that use it.

• UPDATE of a PRIMARY KEY column, single or composite. Uniqueness is checked row by row as PostgreSQL checks it, so SET id = id + 1 over (1, 2) collides, and nothing is written when it does.

• Row-level security:

  • ALTER TABLE ... ENABLE | DISABLE | FORCE | NO FORCE ROW LEVEL SECURITY, reflected in pg_class.relrowsecurity / relforcerowsecurity.

  • CREATE / ALTER / DROP POLICY, listed in pg_policies with PostgreSQL’s rendering of USING and WITH CHECK.

  • Policies are recorded, not yet enforced; see the backlog.

• ALTER TYPE ... ADD VALUE [BEFORE | AFTER] and RENAME VALUE. Enum values order by declared position in ORDER BY, comparisons, BETWEEN, min / max and GREATEST / LEAST.

• Generated columns (GENERATED ALWAYS AS (...) STORED), recomputed on INSERT, UPDATE and COPY. Assigning one directly is 428C9. Reflected in information_schema.columns and pg_attrdef.

• CREATE TABLE ... (LIKE t INCLUDING ...).

• The jsonb functions: jsonb_set, jsonb_set_lax, jsonb_insert, jsonb_pretty, jsonb_typeof, jsonb_array_length, jsonb_extract_path[_text], json[b]_to_record[set] and json[b]_populate_record, and the -, #- and || operators.

• Trigonometric, hyperbolic and degree functions, cbrt, gcd, lcm, factorial, width_bucket, setseed, and the @, |/ and ||/ operators.

• sha224 / sha256 / sha384 / sha512, string_to_table, and format() widths (%-10s, %*s).

• Advisory locks: pg_advisory_lock and its whole family, session-level and transaction-level, shared and exclusive, blocking and try_.

• WITH RECURSIVE, iterated to a fixed point: UNION stops a cycle, UNION ALL keeps every row, and a self-reference without a non-recursive term is 42P19.

• CREATE DOMAIN / ALTER DOMAIN / DROP DOMAIN. A domain’s NOT NULL, CHECK constraints and DEFAULT apply on INSERT, UPDATE and casts. The wire carries the base type, as PostgreSQL’s does, and domains are reflected in pg_type (typtype 'd'), information_schema.domains, and information_schema.columns.domain_name.

• Materialized views: CREATE MATERIALIZED VIEW [WITH NO DATA], REFRESH MATERIALIZED VIEW, DROP MATERIALIZED VIEW, pg_matviews, and relkind 'm'. A write to one is refused (42809).

• COMMENT ON a table, column, index, constraint, function and more, read back through obj_description / col_description.

• GRANT / REVOKE on tables (recorded, as the Python server records them) and on other objects, GRANT role TO role, and ALTER DEFAULT PRIVILEGES. Missing roles (42704), missing relations (42P01) and bad privileges (0LP01) are refused.

• TABLESAMPLE SYSTEM | BERNOULLI, pg_size_pretty, pg_size_bytes, pg_column_size, and the relation-size functions.

• pg_type gains typtype, typbasetype, typnotnull, typnamespace, typtypmod and typdefault. pg_namespace uses PostgreSQL’s fixed oids.

• The xml type: input checked for well-formedness (2200N / 2200M), xmlelement / xmlattributes, xmlforest, xmlconcat, xmlagg, xmlcomment, xmlpi, xmlroot, xmlparse, xmlserialize, IS DOCUMENT, the xml_is_well_formed* functions, and XPath 1.0 through xpath, xpath_exists and XMLEXISTS, namespaces included.

• Every PostgreSQL text-search configuration (french, german, russian, spanish and 23 more), each with its Snowball stemmer and stop-word list. russian sends ASCII words to the English stemmer, as PostgreSQL does.

• EXCLUDE constraints over any operator (23P01), MATCH FULL foreign keys, and pg_get_constraintdef for every constraint kind.

• pg_index lists every index: CREATE INDEX, UNIQUE constraints and expression indexes, not just the primary key. Each row has a real indexrelid that pg_class and ::regclass agree on, and indkey is an int2vector (2 3, subscripted from 0).

• Subqueries in a grouped query’s select list, HAVING or ORDER BY, including an outer aggregate inside the subquery.

• A grouped query that computes over its groups (n::text ... GROUP BY n, -n, count(*)), which used to be refused.

• DROP EXTENSION ... CASCADE drops the columns of the extension’s types, and DROP SCHEMA ... CASCADE drops the schema’s types.

• Range and multirange operators, box operators, and binary results for box, bit, varbit, regtype and regclass.

Fixed

• CREATE TABLE ... PARTITION OF used to create a table with no columns, and every row stayed in the parent. INHERITS was ignored. It is now refused by name.

• array_agg(DISTINCT x) over no rows answered {} instead of NULL. A DISTINCT array_agg / string_agg ignored its ORDER BY ... DESC and NULLS placement. An ORDER BY that is not the argument is now PostgreSQL’s 42P10.

• jsonb || jsonb concatenated the two as text.

• FILTER (WHERE ...) over generate_series ignored the filter.

• An unknown function is 42883 with PostgreSQL’s message, argument types included (function foo(integer, unknown) does not exist), rather than 0A000.

• FROM t, LATERAL (SELECT ... WHERE x = t.id) s gave every row the same answer: the subquery was planned on its own, and t.id resolved to its own id column. A qualified column naming nothing in the FROM clause (sv_c.id inside FROM (SELECT ... FROM sv_o)) was silently read as the inner table’s column. Both are now 42P01, as in PostgreSQL.

• A select-list t.* returned the row as one composite column instead of the table’s columns.

• WHERE c.col IN (...) with a qualified column failed with column "c" does not exist.

• INSERT ... SELECT, CREATE TABLE AS and COPY (query) TO over an aggregate or a join were refused.

• CREATE TABLE with an unknown column type succeeded, silently storing text. It is now 42704, as in PostgreSQL.

• The catalog kept only part of a table’s metadata when rewriting it. Table-level and constraint comments, and a named primary key, recorded by the Python server were erased.

• ORDER BY 1 and ORDER BY <alias> over a computed column sorted by the column it was computed from. SELECT n::text ... ORDER BY 1 came back in numeric order where PostgreSQL gives text order.

• A recursive PL/pgSQL function overflowed the worker’s stack within a few levels and aborted the whole server. The stack now grows on demand, and runaway recursion is 54001.

• A function’s result is coerced to its declared return type, so a recursive RETURNS numeric function no longer overflows as an integer.

• nextval / setval are no longer rolled back with the transaction.

• A cast to an unknown type is 42704, and isempty('...') with an untyped argument is 42725, as in PostgreSQL.

• Text search treats every non-ASCII character as a letter, as PostgreSQL does under the C locale, so a—b and Devanagari words tokenise as they do there.

Rust MongoDB server: timelib’s date parser, ported

$toDate and $dateFromString on the Rust MongoDB server now parse date strings with a port of timelib, the same parser mongod uses, so they match mongod 8.2.11 exactly, error messages included.

Changed

• Free-form date strings go through a literal port of timelib 2022.13’s scanner and post-processing, and of mongod’s wrapper that turns its errors into messages. On 318 probed cases, values and full error text match mongod: every error and warning, with the position and character mongod reports. Before this, 26 strings got only a generic “incomplete date/time” message.

• The zone abbreviation table is generated from mongod’s own copy of timelib’s timezonemap.h, which has 1,127 entries.

Fixed

• $dateFromString without a format refused every string mongod parses except strict ISO-8601.

• $dateFromString’s onError was never applied to a string that failed to parse.

• With a named timezone, a local time in a daylight-saving overlap resolved to the earlier instant, and one in a daylight-saving gap was an error. Both now resolve as on mongod.

• $toDate and $dateFromString refused dates outside the years 1–9999. mongod accepts any 64-bit millisecond value, such as year 0 or year −100000, and reports an overflow only at its own limit (code 159).

The Rust server’s transaction errors carried the wrong label, and sometimes the wrong name

A driver decides what to do after a transaction fails by reading the error’s codeName and errorLabels. The Rust server got both wrong in ways that left a driver’s retry machinery either idle or running when it should not, and the drivers’ own specification suite said so in ten tests.

Everything below was measured against a single-node replica set mongod 8.2.11 over a raw OP_MSG socket. Both qualifiers earned their place: transactions need a replica set, so the standalone used for most probing here cannot answer these at all; and a driver in the path is not safe to measure through, because pymongo retries commitTransaction itself and converts the NotPrimary family into a client-side exception whose reply is never read. Two earlier passes did exactly that and produced two confidently wrong columns.

Fixed

• A failCommand-injected transaction error carried no label at all. The failpoint short-circuit returns before the handler runs and before the transaction is resolved, so it never reached the labelling code that has been there all along. Every test in the drivers’ transaction error-label suite injects its error that way, so every one of them saw errorLabels: [].

• The label depends on WHICH COMMAND failed, which no server here modelled. A statement inside a transaction gets TransientTransactionError; a commitTransaction or abortTransaction splits the same set in two, giving RetryableWriteError to the thirteen codes that are about reaching the node and keeping TransientTransactionError for the five that are about the transaction. Telling a driver the wrong one of those asks it to replay an entire transaction where mongod asks it to retry just the commit.

• Six codes had no name and fell through to Location<code> — 24, 112, 246, 251, 267 and 11601. This is what made the spec’s commitTransaction fails after Interrupted assert Interrupted and receive Location11601: the behaviour was right and the name was not.

• An explicit errorLabels: [] was indistinguishable from the key being absent, because both parsed to an empty list. mongod treats a supplied list as authoritative and adds nothing to it, so an explicit [] means “no labels” rather than “you decide”. Found by this change breaking two previously-passing spec tests, which is the argument for running the suite rather than the targeted tests.

• Code 100 was CannotSatisfyWriteConcern on both servers, a name mongod uses in neither context. A w: 5 write against a single-node replica set and a failCommand injecting 100 both answer UnsatisfiableWriteConcern. The Python server’s message for it also named the requested w, where mongod’s is the bare Not enough data-bearing nodes. tests/test_crud.py asserted the old name, so the test was pinning our bug rather than mongod’s behaviour.

• Both servers were missing 134 and 262 from the transient set, together. The Rust comment cited commands.py::_TRANSIENT_TXN_CODES as its authority, which is precisely how two engines stay in perfect agreement while both differ from the server they imitate. Neither the parity suite nor an engine-vs-engine sweep can see that; only the reference server can.

Added

• failpoints::COMMIT_RETRYABLE_WRITE_CODES, the measured commit/abort split, with unit tests that encode the mongod table as data — so a future divergence fails a fast Rust test instead of surfacing as an unlocalised red driver gauge.

Measured

• Rust server versus mongod 8.2.11: 0 divergent of 63 across codeName and errorLabels for 21 codes on insert / commitTransaction / abortTransaction, plus the explicit-errorLabels rule.

• test_transactions_unified against the standalone secantusd-rs: 10 failures to 3, and the three survivors are the secondary-read cases already recorded as an explicit non-goal — there is no secondary on a single node.

Validation failures, validators and write concern now answer as mongod does, on the Rust server

A write that leaves a document failing the collection validator now gets mongod’s full reply on every update path: update (operator, replacement, pipeline, multi, upsert) and findAndModify. That means the Plan executor error during <command> prefix and the errInfo explaining which schema rule failed. Before this, updates carried no errInfo at all.

create and collMod now parse the validator. An invalid $jsonSchema or an unknown operator is refused when it is set, as on mongod, instead of being stored and then failing every write. writeConcern.w follows mongod’s parsing. Decimal $log is answered rather than refused.

Fixed

• Rust server: validation failures on update and findAndModify carry mongod’s message prefix and errInfo (failingDocumentId and the schemaRulesNotSatisfied tree).

• Rust server: create / collMod refuse a validator mongod cannot parse. This covers $jsonSchema type: "integer", an unknown keyword and an unknown operator; collMod uses mongod’s Parsing of collection validator failed prefix.

• Rust server: writeConcern.w accepts a double or decimal (truncated), null (the empty tag) and a tag set, and refuses bool / array / an empty tag set with mongod’s 9.

• Rust server: $log with a decimal operand returns a decimal result.

• Rust server: documents sort as mongod sorts them, comparing each element’s value type before its field name. GeoJSON with a crs member used to sort after a point without one.

Rust MongoDB server: decimal sort directions, write namespaces and arrayFilters

A decimal sort direction sorted the wrong way on the Rust MongoDB server. Writes could also create collections mongod refuses. Both now match mongod 8.2.11, along with the rest of sort-spec, arrayFilters and write-reply validation.

Fixed

• find with a Decimal128 sort direction returned the wrong order: sort: {a: Decimal128("-1")} sorted ascending. Sort directions now follow mongod’s rule in find and in the aggregation $sort stage. A double is truncated and a decimal is rounded half to even.

• An insert, update, delete, findAndModify or createIndexes into a name mongod refuses now fails with code 73 and mongod’s message. That covers a$b, system.foo, system.views, system.profile, and a namespace over 255 characters. An insert used to create such a collection.

• A bad $sort key, value or $meta spec now gets mongod’s own error code, at the top level and inside $facet, $lookup and $unionWith. These used to report “stage not supported”, and {a: 1.5} was refused although mongod accepts it.

• Invalid arrayFilters get mongod’s code and message on update and findAndModify. A non-document filter fails the whole command, as on mongod.

• On a replica set, insert, update and delete replies now carry electionId and opTime. All write replies order their fields as mongod does.

Added

• tools/probes/write_and_sort_validation.py covers 84 shapes against mongod.

setParameter works, and refuses the way mongod refuses

Both servers reported their parameters through getParameter and had no way to change one: setParameter answered 59 CommandNotFound. It is implemented now, matching mongod 8.2.11 on every outcome for the parameters SecantusDB registers — the previous value in was, and four distinct refusals that are easy to conflate. A parameter that exists but is startup-only is 20 IllegalOperation, not the 72 InvalidOptions an unknown name gets; answering 72 there would claim the parameter does not exist while getParameter was still reporting it.

The coercion rules are more permissive than their types suggest, and every one below was measured rather than inferred. logLevel accepts any number or bool, truncates toward zero (1.9 → 1), clamps at 5 (99 → 5), and refuses only a negative that does not truncate to zero. A boolean parameter accepts every BSON type and stores its truthiness — null, 0 and 0.0 are false, while an empty string, an array and a document are all true.

A parameter SecantusDB does not implement is refused rather than accepted and ignored. mongod’s ingressConnectionEstablishment* family tunes a connection rate limiter this server has no equivalent of; accepting those names would let a client ask for rate limiting and silently get none.

Added

• setParameter on both servers, with getParameter reporting whatever was last set. The store is server-wide, so a value set on one connection is visible on every other.

• A setParameter privilege action, granted to clusterAdmin as mongod grants it to hostManager.

Fixed

• getParameter and setParameter can no longer disagree: the getter overlays the changed values onto its defaults instead of keeping a second copy.

Eleven driver gauges re-measured against the Rust server

Step 5 of the driver-conformance plan, run against secantusd-rs built from d971ef9e — the first sweep since the transaction fixes landed.

gauge	passed	failed	skipped	rate
C++, Kotlin, .NET	892 / 340 / 228	0	9 / 198 / 0	100.0%
php-lib	3,101	1	27	99.9%
php-ext	679	1	35	99.8%
Node	357	1	6	99.7%
Ruby	293	1	24	99.6%
pymongo	1,205	5	290	99.5%
Java	493	3	404	99.3%
mongo-rust-driver	100	1	0	99.0%
Go	439	0	37	withheld — truncated run

pymongo hit the predicted 1,205 / 5 exactly, and its five remaining failures are precisely the declared non-goals: hashed indexes, text indexes, $where twice, and test_to_list_csot_applied. The Rust server went 15 failures to 5 on that gauge and is now level with the Python server.

Changed

• Every -rust-server report regenerated from a run of its own. Seven of them previously carried a September date over raw artifacts from 10–30 August.

• The two Go reports now carry the truncation banner added in #1613, rather than a bare 100.0% over a run that stopped at 476 of 481 tests.

Found

Three server-side defects, all filed in tasks/backlog.md — and none of them in pymongo, which is the argument for running the other-language gauges at all: $geoIntersects returning nothing, change-stream resume losing the original read preference, and eight C-driver failures clustered in the connection-string and server-selection surface.

configureFailPoint is now gated behind enableTestCommands

W6 in docs/security-reports/2026-08-10.md: an unauthenticated client that can reach the port could arm a server-wide failCommand with closeConnection: true, dropping the socket of every subsequent operation on every connection — a cross-tenant DoS with no privilege required. mongod ships the same command behind a startup parameter, off by default. SecantusDB had no equivalent gate; the feature was always live.

Measured against mongod 8.2.11 (2026-09-28): started without the parameter it answers 59 CommandNotFound :: no such command: 'configureFailPoint' — not Unauthorized, not a no-op — and getParameter reports enableTestCommands: false. Both servers now answer identically.

Changed

• The standalone daemons default it OFF, as mongod does: --enable-test-commands on secantusd-rs and secantusd-py, or [server] enable_test_commands in secantusd.toml.

• The embedded SecantusDBServer defaults it ON. Constructing one in a test is the entire use case for that class, and every driver failpoint suite needs it. The daemon is the thing an operator exposes on a port; the embedded handle is not.

• getParameter reports the real value instead of a hardcoded true. Drivers gate their failpoint suites on this flag — while it merely said false, pymongo skipped ~1,080 unified-spec tests — so a server that refuses the command must not claim to accept it.

The gate is the registry’s absence

A server that did not enable test commands wires no FailPointRegistry, and the missing registry is what makes the command report CommandNotFound. That keeps the decision at server startup, where the operator made it, instead of threading a flag down to the dispatch table.

One choke point for the gauges, not thirteen

tasks/driver-conformance-followups-plan.md sized the cost as “every gauge task must pass the flag. Miss one and that gauge silently loses its failpoint coverage.” gauge_common.spawn_daemon already rewrites the port and log level, so it forces the flag too — all thirteen daemon gauges get it from one place, and the failure mode the plan predicted cannot happen.

Transaction failures reported Location11601 where mongod says Interrupted

A failCommand failpoint renders its injected error code through the server’s code-name table, and that table had none of the transaction or replication failure codes in it. So every injected transaction error came back named Location<code> — Location11601 for Interrupted, Location251 for NoSuchTransaction, Location112 for WriteConflict. Drivers assert on codeName, so the specs’ commitTransaction fails after Interrupted failed not because the behaviour was wrong but because the name was.

Seventeen names now come from a probe rather than the error-codes list, read off a single-node replica-set mongod 8.2.11 — transactions need one, so the standalone used for most probing in this file could not answer it.

The same probe settled a question that looked like one item and was two. The driver gauges failed on Interrupted (11601) and PreparedTransactionInProgress (267) together, and the obvious move was to add both to the transient-label set. mongod labels 267 TransientTransactionError and gives 11601 no labels at all — so adding both would have turned one test green and the other red. 267 is in; 11601 stays out, which is where it already was.

Fixed

• commands.py: 17 transaction / replication codes now render mongod’s real codeName instead of Location<code> — Interrupted, NoSuchTransaction, WriteConflict, PreparedTransactionInProgress, SnapshotUnavailable, LockTimeout, NotWritablePrimary, InterruptedAtShutdown, InterruptedDueToReplStateChange, ShutdownInProgress, PrimarySteppedDown, SocketException, NetworkTimeout, HostUnreachable, HostNotFound, NotPrimaryNoSecondaryOk, NotPrimaryOrSecondary.

• 267 PreparedTransactionInProgress added to _TRANSIENT_TXN_CODES, measured rather than assumed.

• test_transactions_unified.py: 5 failures -> 3, and the remaining three are the secondary-read cases already recorded as an explicit non-goal (single node, no secondary to read from). Both retry-semantics failures are closed.

A transaction error injected by a failpoint carried no errorLabels

A driver decides whether to retry a whole transaction by reading the TransientTransactionError label off the error. SecantusDB attached that label to errors it raised itself, but not to errors produced by a failCommand failpoint — and since every test in the drivers’ transaction error-label suite injects its error that way, the entire suite saw errorLabels: []. A driver’s retry loop, exercised exactly as the specification intends, silently did not engage.

The labelling logic already existed and was already correct. failCommand’s short-circuit simply returned before reaching it: it answers before the handler runs, and before the transaction is even resolved. So this is six lines routing one path through machinery the other path had been using all along, not new semantics — the third time in this file that a gap turned out to be an existing mechanism one caller never reached.

Fixed

• commands.py: a failpoint-injected error whose code is in _TRANSIENT_TXN_CODES now carries TransientTransactionError when the statement is part of a transaction, matching what a naturally-occurring error of the same code already did. Detected via autocommit: false — the signal mongod itself uses, present on every statement of a transaction including commit and abort — so the failpoint check does not have to move below the transaction resolution and reorder failpoint-versus-transaction error precedence, which no probe currently justifies.

• Takes test_transactions_unified.py from 9 failures to 5, including all three of TestUnifiedErrorLabels (NoSuchTransaction, NoSuchTransaction on commit, WriteConflict).

Changed

• Reading from a secondary is now an explicit non-goal rather than a gap. SecantusDB is single-node, so there is no secondary to read from and anything whose behaviour is defined by one is out of scope. Three driver-gauge tests fail permanently on this and are accepted with the reason written down in tasks/backlog.md §4, rather than deselected to tidy the number.

Deferred

• Two failures remain, and the two error codes want OPPOSITE treatment: Interrupted (11601) where the test expects the commit to fail, and PreparedTransactionInProgress (267) where it expects the transaction to be retried. Adding both to the transient set would make one green and the other red, so tasks/backlog.md §5 says to probe a single-node replica-set mongod — transactions need one, so a standalone cannot answer it — before moving a label set on a guess.

$where is refused the way a mongod without a script engine refuses it

$where runs user-supplied JavaScript, and SecantusDB embeds no script engine. Both servers already declined it, but each invented its own answer: the Rust server said BadValue: query uses a construct the Rust server does not support — our implementation leaking onto the wire — and the Python server said unknown top level operator: $where, which is wrong twice over, because mongod knows $where perfectly well.

mongod --noscripting is a supported configuration with exactly the property we have, and it answers 6108304 / Location6108304 / "no globalScriptEngine in $where parsing". Both servers now answer that, on every command that takes a filter, with the per-statement writeErrors shape on the two batch writes that mongod uses. Inside an aggregation $match the answer is different and deliberately so — 2 / BadValue / "$where is not allowed in this context", which mongod gives whether or not it has a script engine, so it is a pipeline rule rather than a scripting one.

This does not make $where work: a query that needs the JavaScript to run still fails, exactly as it does against a real --noscripting mongod. What it fixes is that a client reading the error now learns something true.

Fixed

• $where in a query context answers mongod’s 6108304 Location6108304 "no globalScriptEngine in $where parsing" on both servers, for find, count, distinct, findAndModify, and per-statement in writeErrors for delete / update, at parse time so an empty or nonexistent collection refuses too.

• $where inside an aggregation $match answers 2 BadValue "$where is not allowed in this context", and is checked before the leading-$match lift — previously the first stage picked up the query-context refusal while every other position answered correctly.

• command_error_during / read_exec_error sent the bare sentinel codeName: "Location" for any error code with no symbolic name, where mongod sends Location<n>. Affected every such code reaching a non-batch command through the storage error path, not just $where.

Added

• mongod_noscripting_uri fixture in the differential gate, so a refusal that only a script-engine-less mongod can demonstrate is compared against a live one rather than against hardcoded values.

Committed WiredTiger bindings

secantus-wt can now build without libclang.

Added

• secantus-wt ships its WiredTiger bindings as src/bindings.rs. A build without the bindgen feature uses that file and needs no libclang.

• The bindgen feature, on by default, regenerates the bindings and fails the build if they differ from the committed file. ./inv wt-bindings-refresh updates the file.

WiredTiger builds from a crate

The Rust servers can now build WiredTiger from bundled source, the first step towards publishing them on crates.io.

Added

• crates/secantus-wiredtiger-sys: builds the same WiredTiger as the wheel (mongodb-7.0.33 with SecantusDB’s patches) as a static library with CMake and a C compiler, with zlib and lz4 linked statically. No Python is needed.

• secantus-wt has a bundled feature that uses it. It is off by default, so existing builds still link the prebuilt WiredTiger.

• ./inv wt-sys-refresh regenerates the bundled source. A test fails if the source drifts from vendor/wiredtiger and the patch scripts.

[0.6.0b17] — 2026-09-26

The errors are the feature

For three weeks the work was almost entirely about what SecantusDB says when something is wrong. Rejecting a malformed command is easy; rejecting it the way mongod does — same code, same message, validated in the same order, so a driver’s own error handling behaves identically — is what makes a surrogate usable rather than merely available. Three hundred and forty-one slices landed in this release, and the largest group of them move an error from “generic” to “mongod’s”.

Fourteen were worse than a wrong message. A null argument to createIndexes, malformed cursor arguments, non-numeric input to $stdDevPop, a NaN in $expr, an integer overflow in the update path — these crashed the server rather than raising. They are fixed, and the wrong-type sweep across the command surface now reports zero divergences. A crash is not a bad error message; it is the absence of one, and in a database it takes the connection with it.

The Rust PostgreSQL server grew up over the same period — client_encoding, composite types over the binary wire, column precision and scale, backend termination and cancellation, two-phase commit — and a column-level UNIQUE that had been accepted and then silently not enforced, which is data corruption rather than a missing feature. Both Rust binaries now stamp the git tree they were built from and report it in --version, because an artifact older than the tree it is tested against produces failures that look like somebody else’s regression, and the evidence is never in the diff.

The binaries themselves were the last thing to get that treatment, and they needed it. Both published macOS archives linked liblz4 by its absolute Homebrew path and would not launch on a Mac that did not happen to have Homebrew lz4 installed there; the Linux archives wanted liblz4.so.1 and libz.so.1 from the host, which a slim container image often does not carry. Both are fixed — every archive now links only its platform’s own C runtime — but the more useful change is that the release no longer takes anyone’s word for it. Before an archive is packaged, the workflow reads the binary’s real dependency table (Mach-O load commands, ELF DT_NEEDED, or the PE import directory) and refuses to publish one that reaches outside a per-platform allowlist. The macOS bug had survived every build, test and smoke run for two releases because the machine that builds the artifact is the one machine where it works.

Failpoints stay with the client that set them, and drivers can now see them

Driver test suites inject errors with configureFailPoint, usually scoped to one client by appName, and often on that client’s hello so its connections drop. Both servers ignored the scope, so the failpoint reached every connection — including the test runner’s own, which then could not switch it off. One such test in the Go driver’s unified suite left the server unusable for every test that came after it.

Both servers also told drivers that test commands were disabled, although configureFailPoint works. pymongo reads that flag and skipped every failpoint test: about 1,080 of its unified-spec tests never ran against SecantusDB.

Fixed

• failpoints.py / secantus-commands failpoints.rs: failCommand honours data.appName, matching the connection’s client metadata. A handshake hello is matched by the client.application.name it carries itself, so a failpoint can fail the first command on a new connection, as the SDAM spec tests require. Commands from other clients no longer spend the failpoint’s skip / times budget.

• Both servers: failCommands compares a command’s canonical name, so isMaster also matches the legacy lower-case ismaster handshake that pymongo sends (likewise findAndModify / findandmodify).

• Both servers: a closeConnection failpoint now drops the socket on every path. The Rust server replied {ok: 1} to a legacy OP_QUERY handshake, which a driver rejects as “wire version 0”. Both servers answered a streamed (awaitable) heartbeat as a clean end of stream; the Python server also logged it as an error. The Rust server’s exhaust getMore stream ignored it too.

• Rust server: endSessions, killSessions, killAllSessions and killAllSessionsByPattern abort the sessions’ open transactions, as the Python server’s already did. They were no-ops, so a transaction left open kept every later write to its documents in a WriteConflict retry loop. Driver test runners call killAllSessions between tests for exactly this. Once the failpoint tests ran, this failed 68 pymongo tests and crashed four gauge workers.

• Both servers: a change stream opened or resumed with events already waiting no longer returns a postBatchResumeToken past events it has not sent.

  • The Rust server took the token from after the whole poll, so when batchSize held events back for getMore, a driver that resumed after the first batch skipped them silently.

  • The Python server did the same for batchSize: 0.

  • The token is now the last sent event’s _id, or the resume point when nothing was sent.

Changed

• Both servers: getParameter reports enableTestCommands: true. Driver test harnesses now run their failpoint tests instead of skipping them. What the server accepts is unchanged: configureFailPoint was already served, and is still RBAC-gated when auth is on.

maxTimeAlwaysTimeOut failpoint

Both servers now honour mongod’s maxTimeAlwaysTimeOut failpoint, which driver test suites use to provoke MaxTimeMSExpired deterministically.

Added

• Both servers: with configureFailPoint: "maxTimeAlwaysTimeOut" on (alwaysOn or {times: N}), any operation that has a maxTimeMS fails with 50 MaxTimeMSExpired before doing its work; operations without one run normally. A getMore counts as time-limited when its cursor was opened with maxTimeMS, because mongod bounds a non-tailable cursor’s getMores by the originating command’s limit and drivers do not resend it. The failpoint was previously accepted and ignored, so four pymongo tests failed with “ExecutionTimeout not raised”.

The probe-store sweep no longer deletes a database that is still in use

The sweep that reclaims abandoned probe WiredTiger homes deleted a running mongod’s data directory. The directory had been named by hand with a shell’s $$ — the shell’s pid, not the server it launched — so the name pointed at a process that had exited seconds later. The sweep saw a dead pid, believed it, and removed the files underneath a live database, which died on a fatal WiredTiger assertion: log pre-alloc server error ... the process must exit and restart.

The pid in a store’s name is written by whoever created the directory. It is a hint, not proof of ownership, and nothing else was asked before deleting.

Fixed

• A dead pid is no longer sufficient grounds. A store is reaped only once it has also sat untouched for half an hour — a store being served is written to constantly, so a recent mtime means hands off, whatever the name claims. An abandoned store is still reclaimed on the next run after that, which is ample for a problem measured in days.

• probe_store() is documented as the only way to name one of these directories, since it fills in os.getpid() and cannot name the wrong process.

WiredTiger.lock looks like the better authority and is not, which CI proved before this merged: POSIX advisory locks are held per process, so a check made from the process that opened the store reports the file as free. That passed on Windows — which locks mandatorily at the handle — and failed on macOS, where the sweep then deleted a live store and took WT_PANIC through the worker. Worse, merely opening and closing a descriptor to a file the process holds an fcntl lock on releases that lock, so the “safe” probe can break the database it is inspecting. The Windows os.remove probe is kept as an extra gate there, where it is genuinely decisive.

The regression tests build stores directly rather than running a server, so a future regression fails an assertion instead of panicking WiredTiger inside the test worker. They were confirmed to fail with the guard removed.

$stdDevPop and $stdDevSamp work in an expression on the Rust MongoDB server

{$stdDevSamp: [1, 2]} in a $project / $addFields / $set answered 168 Unrecognized expression where mongod answers 0.7071067811865476. The $group accumulator forms were fine; only the expression forms were affected.

The evaluator had implemented both for three weeks, with unit tests. The names were simply absent from KNOWN_EXPR_OPS, which the pipeline validator consults before the evaluator runs — so a correct, tested engine sat behind a gate that refused to let any pipeline reach it. The Python server was never affected.

Fixed

• Both operators are on the validator’s list, so pipelines reach the evaluator that already implemented them. Verified against mongod 8.2.11: $stdDevPop over [1, 2, 3] is 0.816496580927726 and $stdDevSamp over [1, 2] is 0.7071067811865476 on both. Across the expression sweep this took the Rust server from 96 code divergences to 2, and those two are the documented Decimal128 $atan2 / $pow deferrals.

• The list’s drift guard actually guards something now. The old test looped over KNOWN_EXPR_OPS asserting a function whose entire body is a lookup in KNOWN_EXPR_OPS accepted each name — the list agreeing with itself, unable to fail however far the code drifted. The replacement reads apply_op’s match arms from the source and asserts every dispatched operator is listed; it was confirmed to fail, naming both operators, when the fix is reverted.

The Rust PostgreSQL server computes with aggregate results, and pads char(n) where PostgreSQL does

count(*) + 1 was refused. So were sum(n) + 0 and coalesce(sum(n), 0) — anything that did arithmetic on what an aggregate returned. The shape of the plan was the reason: an output column named either a grouping key or an aggregate, with no room for a computation over them.

Added

• Aggregates inside an expression. The aggregates within are extracted and planned as ordinary items — computed once per group as usual — and the expression then runs over their results, reading each from its own slot. An identical aggregate written twice in one expression is computed once.

• char(n) reaches array_agg, min and max blank-padded, which is what PostgreSQL does: those take the column’s own type. string_agg does not, its argument being coerced to text, and the counting aggregates cannot tell the difference either way. Measured on PostgreSQL 14.24 rather than reasoned about — the three behave differently and the difference is not guessable.

Fixed

• min(c) || '|' over a char(n) no longer answers with the padding. min/max carry the column’s atttypmod so the encoder pads on output; padding the stored value instead put blanks through ||, which PostgreSQL strips. The first implementation did exactly that, and the concatenation case is what caught it.

$toDecimal of a long numeric string answers what mongod answers

Converting a numeric string with more than 34 significant digits to a decimal, through $toDecimal or $convert, failed with an internal server error on both SecantusDB servers, with or without onError. mongod accepts it: it keeps the first 34 digits, dropping the rest without rounding, so "1.99999999999999999999999999999999999" becomes 1.999999999999999999999999999999999. Both servers now do exactly that.

A string outside the range a decimal can hold now fails the way mongod fails, as a conversion error that onError catches, with mongod’s own reason: “would overflow” above about 1E+6144, and “would underflow” for a value too small to keep all its digits (1E-6176 still converts; 1E-6177 does not).

A constant conversion that fails is now reported under the error name ConversionFailure, as mongod reports it, instead of Location241.

Fixed

• expressions.py: $convert / $toDecimal of a string parse as mongod does: 34 digits rounded toward zero, IEEE overflow and underflow as conversion failures.

• crates/secantus-core: the same for the Rust engine (decimal128_from_str), which relied on a parser that refuses a 35th digit.

• commands.py: error code 241 is named ConversionFailure.

Testing

• tests/test_mongod_differential.py: eleven cases against a real mongod 8.2.11: truncation, both range failures, the boundaries, and onError.

• crates/secantus-core: unit tests with the same mongod-measured values.

• A probe of 66 cases across both engines found no differences from mongod.

detached_run stop says when it fails to stop anything

On Windows the stop path ran taskkill with its output sent to DEVNULL and its exit code ignored, then printed stopped <name> and returned 0 whatever happened. A kill that failed left no trace at all, and a caller that believed the success and started a replacement would get two of whatever it was running.

stop now captures what taskkill said, retries once if the process is still there after ten seconds, and — if it is still alive after that — reports the failure with the diagnostics and exits non-zero instead of claiming success. The POSIX path already escalated to SIGKILL; it now verifies that worked too.

Prompted by test_stop_ends_a_running_command failing once on Windows CI and passing on a re-run, with nothing in the log to say why. That test now prints what stop and status reported when it fails.

Fixed

• scripts/detached_run.py: cmd_stop reports a kill that did not work.

• tests/test_detached_run.py: the assertion carries the evidence.

A dropped table’s privileges came back with the next table of that name

Dropping a table removed its definition and left everything that governed access to it behind: the table’s grants, its column grants, its relacl state and its RLS policies. Creating a table under the same name silently reattached all four — including grants to other roles.

That is a privilege-escalation shape rather than a metadata nit. Dropping a table is how someone revokes all access to it; before this, the replacement table still granted what the old one did.

Measured against PostgreSQL 14 on 2026-09-20: after DROP TABLE + CREATE TABLE the new table reports relacl NULL and has no grants, column grants or policies. Comments and the RLS-enabled flag were already cleared correctly; the other four were not.

Fixed

• DROP TABLE clears the table’s grants, column grants, relacl state and RLS policies along with its definition.

Found from pgjdbc’s DatabaseMetaDataTest::tablePrivileges, which passed or failed depending on run order: the class’s noTablePrivileges test revokes everything on metadatatest, and the fixture drops and recreates that table between every test, so on a real server the revoke cannot carry over. The initial diagnosis — “a view’s privileges resolve and a table’s do not” — was wrong, and checking it rather than building on it is what led here: the same test was passing in one parameterised run and failing in the other.

Measured on pgjdbc’s DatabaseMetaDataTest (194 tests): 31 failures → 30, 18 distinct → 17, no regressions. tablePrivileges goes green. The single-test delta understates the change; the leak it exposed is the substance.

A full-suite run here also surfaced a parallel-worker race in test_pg_numeric_wide_grouping.py (every xdist worker created a table named t in the shared reference PostgreSQL’s public schema, so workers raced on drop/create). That turned out to be fixed independently in #1536 while this branch was in flight, so nothing for it is carried here — #1536’s per-case schema is what ships.

An array-of-enum column reported the enum, not the array type

A column declared test_schema.test_enum[] recorded the element enum’s oid. pg_attribute’s enum branch had no array handling — the composite branch beside it already did — so a client resolving the column found a typtype = 'e' enum where PostgreSQL 14 reports the typtype = 'b' array type. pgjdbc’s getColumns returned TYPE_NAME "test_schema"."test_enum" and the element’s DATA_TYPE instead of "test_schema"."_test_enum" and ARRAY.

Fixing that exposed a second defect underneath it. Array type names were made unique against a single global set of every type name in the catalog, but a PostgreSQL type name is unique only within its own namespace. So test_schema.test_enum’s array dodged the entirely unrelated public._test_enum and came out __test_enum, and the miscount compounded — an array in public whose name was already taken landed on four underscores where PostgreSQL uses three.

Both fixed, and every name was measured against PostgreSQL 14 on 2026-09-20 rather than reasoned about — including the three-underscore case, which looks like an off-by-one and is not: _test_enum is taken by an enum of that literal name, and __test_enum by that enum’s own array, which is created first because it has the lower oid.

Fixed

• An array-of-enum column reports the array type’s oid.

• Array type-name collisions are resolved per namespace.

Measured on pgjdbc’s DatabaseMetaDataTest (194 tests): 33 failures → 31, 20 distinct → 18, no regressions. getCorrectSQLTypeForOffPathTypes and getCorrectSQLTypeForShadowedTypes go green.

A foreign key reported the wrong referenced constraint

pg_constraint.conindid — the index backing the constraint a foreign key references — was hardcoded to the referenced table’s PRIMARY KEY index, and the comment beside it stated that as the rule. A foreign key may reference any UNIQUE constraint, so REFERENCES pkt(b) reported pkt_pk_a where PostgreSQL 14 reports pkt_un_b (measured 2026-09-20).

pgjdbc’s getImportedKeys reads PK_NAME by joining pkic.oid = con.conindid, so this is the name an ORM sees for the key a foreign key points at — naming the wrong constraint misdescribes the relationship rather than merely omitting it.

Fixed

• conindid resolves from the foreign key’s confkey, matching whichever PRIMARY KEY or UNIQUE constraint covers those columns, and falling back to the primary key when nothing matches.

Columns are matched as a sorted set, because a foreign key may name the referenced columns in a different order than the constraint declares them. All three shapes are pinned by the test: the UNIQUE case that was wrong, the PRIMARY KEY case that was right and must stay so, and a composite unique.

Measured on pgjdbc’s DatabaseMetaDataTest (194 tests): 23 failures → 21, 14 distinct → 13, no regressions. foreignKeysToUniqueIndexes goes green.

A function created in a schema was invisible in it

CREATE FUNCTION hasfunctions.addfunction(...) stored the function and then reported it in public. Only a pg_temp_ qualifier was preserved at creation; every other schema was dropped. The function existed, worked, and could not be found by any client that filters by schema — which pgjdbc’s getFunctions and getProcedures both do.

Underneath that were two more defects on the same rows. pg_proc.proname returned the dotted storage key rather than the bare name, and prokind was hardcoded 'f' — so every procedure was reported as a function, and getProcedures(), which filters on prokind = 'p', returned nothing at all regardless of schema.

Fixing only the catalog half would have made things worse: the routine is stored under a dotted key, so recording the schema without teaching the call path about it made hf.addf(1, 2) raise “function addf does not exist” — the namespace correct and the function unusable. A qualified call now resolves against the dotted key first, then the bare name, so builtins (pg_catalog.now()) and unqualified calls are unaffected.

Fixed

• A schema qualifier is preserved for any schema, not just pg_temp_. public stays unqualified: it is the default search_path schema, so its functions must keep resolving from a bare name.

• pg_proc.pronamespace resolves to the routine’s real schema, and proname is the bare name.

• pg_proc.prokind is 'p' for a procedure.

• A schema-qualified call resolves the dotted key.

Changed

• A bare call to a function homed in a non-search_path schema now raises function addf(integer, integer) does not exist. This server used to answer it; PostgreSQL 14 raises exactly that error (measured 2026-09-19), so the new behaviour is the faithful one.

Measured on pgjdbc’s DatabaseMetaDataTest (194 tests): 36 failures → 34, 23 distinct → 21, no regressions. getFunctionsInSchemaForFunctions and getProceduresInSchemaForProcedures go green.

CREATE FUNCTION f(int, int) recorded its arguments as void

pg_proc.proargtypes read '2278 2278' for a function declared with unnamed parameters — 2278 is void, and an OID this catalog does not even define, so a client resolving it found nothing. Named parameters were unaffected: CREATE FUNCTION g(a int, b text) recorded '23 25' correctly all along, which is why a catalog full of working functions never showed it.

Fixed

• _function_param_types now resolves a bare parameter type. sqlglot parses f(a int) as a ColumnDef carrying a DataType, but bare f(int, int) as a plain Identifier whose name is the type spelling; the DataType branch missed it and the tag fell through to None → void. Multi-word builtins resolve too (double precision → float8).

Found from pgjdbc’s DatabaseMetaDataTest::funcWithoutNames(), which is exactly the test for functions declared without parameter names. Measured effect on that gauge: 82 → 80 standing failures, and the baseline is tightened to match.

$lookup joins numbers by value, and SQL set operations dedup by value

A $lookup with localField / foreignField returned no match for Decimal128("1.5") against Decimal128("1.500"), or for 2 against Decimal128("2.0"). On the Rust server it also missed 2 against 2.0. The Python server joined true to 1. mongod 8.2.11 joins all of those pairs by value except true/1. This only happened when the foreign field had no index: that path is a hash join, and it keyed on the raw value. Python’s Decimal128 hashes by representation, and the Rust server compared with structural Bson equality. With an index, the query goes through the query engine and was already right.

The PostgreSQL server lowers a join to $lookup, so the same bug meant select ... from t join u on t.v = u.v returned no rows for numerics that differed only in scale, or for a numeric against an int. Separately, UNION / INTERSECT / EXCEPT, recursive-CTE dedup, evaluated DISTINCT, DISTINCT ON and window PARTITION BY built each row’s identity from repr(). So 1.5 and 1.50 were different rows, INTERSECT of the pair was empty, and 0.0 / -0.0 split too.

Fixed

• aggregate.py: the $lookup hash join keys on the $group bucket key (numerics by value, one NaN, bool apart from numbers).

• crates/secantus-commands: lookup_match compares with the canonical BSON order instead of ==.

• sql/numeric.py: eq_key, a by-value row identity, used by the set operations, DISTINCT, DISTINCT ON and PARTITION BY.

Testing

• tests/test_lookup_numeric_equality.py: both servers, with and without an index on the foreign field.

• tests/test_mongod_differential.py: lookup-numeric-equality-by-value.

• tests/test_pg_numeric_value_equality.py: joins, set operations, signed zero, NaN, DISTINCT ON and PARTITION BY.

The macOS binaries only ran on a machine that had Homebrew

Both published macOS archives — secantusd-rs and secantusd-pg — linked liblz4 by its absolute Homebrew path, /opt/homebrew/opt/lz4/lib/liblz4.1.dylib. On a Mac without Homebrew lz4 installed at exactly that path, neither binary launches at all. Nothing in the build said so: the release runner has Homebrew by definition, the smoke test passes there, and otool -L on the shipped archive is the only thing that ever reported it.

The cause was one word. secantus-wt’s build script emitted cargo:rustc-link-lib=dylib=lz4, and because Homebrew ships liblz4.a and liblz4.dylib side by side in the same directory, dylib= meant the linker took the dylib every time even though a static archive sat next to it. macOS now picks the link kind where it already picks the search path, and prefers the static archive wherever one exists.

The fix for the release lanes was already in the tree and only the wheel build used it: tools/build_lz4_macos.sh builds a static-only liblz4 at the right deployment target, which is why the published macOS wheel has never had this problem. Both binary workflows now build lz4 the same way instead of brew install lz4. The Linux archives are unaffected — they link liblz4.so.1 and libz.so.1 through the normal loader path — and the Windows archive was already self-contained.

Fixed

• macOS secantusd-rs and secantusd-pg archives are now self-contained: they link only /usr/lib and /System/Library system libraries, with liblz4 statically linked. Verified with otool -L and a 3,000-document round trip through lz4-compressed tables.

• A plain developer cargo build on macOS now produces a portable binary too, rather than one pinned to the builder’s Homebrew prefix.

Changed

• release-binaries.yml and release-pg-binaries.yml build lz4 from source via tools/build_lz4_macos.sh and set CMAKE_PREFIX_PATH + SECANTUS_WT_EXTRA_LIBDIR, matching what wheels.yml has always done.

Wide numerics group by value

On the Python PostgreSQL server, select v, count(*) from t group by v over a numeric column returned one row per stored text rather than one per value. A number past 34 significant digits does not fit a Decimal128, so it is stored with its Postgres text beside a scale-free key — which made 1e40, 1e40.0 and 1e40.00 three documents and therefore three groups where PostgreSQL has one. SELECT DISTINCT and count(DISTINCT v) split the same way.

Such a column now groups on the value, and the group carries the first row’s value for display, which is what PostgreSQL prints: insert …890.00 first and GROUP BY answers …890.00, insert the bare form first and it answers that (measured against PostgreSQL 14.24).

Still split, and recorded in tasks/backlog.md: a join whose key is wide, and a value stored narrow in one row and wide in another (1.5 against 1.5 written with 39 digits).

Fixed

• sql/numeric.py: group_key_expr, the value identity for a numeric column.

• sql/planner.py: every $group that keys on a numeric column — plain GROUP BY, GROUPING SETS / ROLLUP / CUBE, the window and join planners, SELECT DISTINCT and the DISTINCT-over-grouped-output dedup — keys on that identity and restores the display value.

Testing

• tests/test_pg_numeric_wide_grouping.py: pinned cases, plus ten that run the same statements against a live PostgreSQL and compare the text, so a difference in scale fails rather than passing silently.

pg_class.relpages and pg_type.typlen did not exist

Two catalog columns a JDBC client reads were simply absent, and each failed a query outright rather than returning a wrong value.

pg_class.relpages is what pgjdbc’s getIndexInfo selects as PAGES, so its absence failed that whole query with column "relpages" does not exist before any row came back. pg_type.typlen is read by getMaxNameLength(), which selects it for typname = 'name' and treats a missing row as fatal — and name is not a type this server stores, so the row had to be added for the lookup to resolve at all.

All values measured against PostgreSQL 14 on 2026-09-19.

Added

• pg_class.relpages. A fresh table reports 0; a fresh index reports 1, because an index has its metapage from the moment it exists — 0 would be the obvious guess and is wrong.

• pg_type.typlen, for built-in, array, composite, range, domain and enum rows. An enum is a fixed 4-byte oid reference; everything else user-defined is varlena (-1). A domain inherits its base type’s width. TypeInfoCache filters its array lookup on typlen = -1, so the array rows’ value is load-bearing rather than decorative.

• a pg_type row for name (oid 19), a type this server never stores but a client still resolves by name.

Fixed

• A comment added with the declared-char-type change claimed this catalog emits no _<type> array rows for any type. That was inferred from PG_TYPENAME holding no _ names and is false — array rows are synthesised from typarray, and _text (1009) has always been served.

Measured on pgjdbc’s DatabaseMetaDataTest (194 tests): 36 failures → 35, 23 distinct → 22, no regressions. getClientInfoProperties goes green. relpages also unblocks getIndexInfo far enough to reach the next blocker in the same query, now filed in tasks/backlog.md with both of its halves.

COPY TO exports a timestamp’s microseconds

COPY <table> TO on the Python PostgreSQL server cut every timestamp and timestamptz down to the millisecond: a stored 10:00:00.412661 was exported as 10:00:00.412, in both the text and binary formats. SELECT and COPY (SELECT …) TO were exact. COPY TO is how a table is exported or backed up, so every export taken that way was quietly less precise than the data.

The server keeps a timestamp’s sub-millisecond part beside the stored date, and the table form of COPY TO read the date without it. It now reads each value the way SELECT does. Found by psycopg’s test_copy_table_across, which passed or failed depending on whether its random timestamps had microseconds.

Fixed

• sql/engine.py: copy_extract and copy_extract_raw read each column through executor._with_subms, the same merge SELECT uses.

Testing

• tests/test_pg_copy_to_subms.py: text and binary exports, and a table copied across through COPY TO / COPY FROM in both formats. All four fail without the fix.

varchar and char(n) columns pointed at a type that did not exist

Both spellings fold to the text storage tag, but a column still records the declared oid — 1043 for varchar, 1042 for bpchar. pg_type is built from a tag-keyed table, which can name only one of the three, so every varchar and char(n) column in the catalog referenced an oid with no pg_type row. A client joining pg_attribute to pg_type — which is what a JDBC or psycopg metadata call does — resolved those columns to nothing.

Function AND procedure parameters had the same distinction missing one level further in: they carried no equivalent of a column’s decl_oid, so CREATE FUNCTION f(int, varchar) recorded proargtypes = '23 25' where PostgreSQL 14 records '23 1043', and a client reading the argument back was told it was text. For a procedure the same gap reached the wire: an INOUT varchar parameter’s CALL result column was described as oid 25 where PostgreSQL 14 sends 1043.

Found from pgjdbc’s DatabaseMetaDataTest::functionColumns, which asserts a bare varchar parameter reports as varchar. All values here were measured against PostgreSQL 14 on 2026-09-19.

Fixed

• pg_type now carries rows for varchar (1043) and bpchar (1042), so a declared-char column resolves to a real type row instead of dangling.

• pg_proc.proargtypes records the declared oid for a varchar / char(n) parameter, in both the named and the unnamed signature form, for functions and procedures alike.

• A CALL’s RowDescription reports the declared oid for an OUT / INOUT varchar / char(n) parameter — the wire-visible half of the same gap.

• A bare bpchar parameter recorded 2278 (void) — it has no storage tag at all, so it was not merely the wrong string type but an oid this catalog does not define. It now records 1042.

• A function’s RETURN type kept the declaration too: RETURNS varchar had recorded prorettype 25, and pg_get_function_result / information_schema.routines.data_type both said text.

• information_schema.columns.data_type names the declared type for a varchar / char(n) column instead of reporting the text tag for every string column.

• pg_get_function_arguments and information_schema.parameters.data_type render character varying / character for those parameters — the SQL spelling PostgreSQL uses there, which is deliberately not the pg_type.typname (varchar / bpchar) the same type reports elsewhere.

The Rust Postgres server was checked for the same defect and does not carry it: its BUILTIN_TYPES already lists varchar (1043) and bpchar (1042) with the same array oids measured here, and it has no pg_proc function reflection at all. This one is Python-server-only.

Measured against pgjdbc’s DatabaseMetaDataTest (jdbc2 + jdbc4 + jdbc42, 194 tests), by diffing a run with and without this change: 42 failures → 36, 26 distinct → 23, no regressions. functionColumns, getColumnsCharOctetLength and informationAboutArrayTypes all go green — the latter two because a varchar column now resolves to a real pg_type row, which is what the driver’s own type lookup needs.

HAVING compares a numeric sum at full precision

On the Python PostgreSQL server, select g from t group by g having sum(v) = <the value the select list returned> could return no rows. The select list sums a numeric column exactly. HAVING compared an in-pipeline running total instead, which rounds at 34 significant digits and ignores a value too wide for a Decimal128. A group whose sum held a 50-digit value compared as if that value were not there.

For a single-table GROUP BY, including with window functions, and for an aggregate with no GROUP BY, a numeric sum / min / max in HAVING is now evaluated after the exact fold. Joins, grouping sets and FILTER terms still compare in the pipeline (tracked in tasks/backlog.md).

Fixed

• sql/planner.py: _having_to_match sends a numeric sum / min / max term to the per-grouped-row HAVING residual when the caller has one.

Testing

• tests/test_pg_having_exact_numeric.py: 37-digit and 50-digit sums, min / max, BETWEEN, IS NULL, a mixed AND, ORDER BY + LIMIT, a window over the groups, and a whole-table aggregate.

numeric is exact at any width on the Python PostgreSQL server

The Python PostgreSQL server stored every numeric as a BSON Decimal128, which holds 34 significant digits. A wider value was silently rounded, and a very small one was changed to a different number entirely: 1E-7000 was stored as 1E-6176. That was documented as a permanent limit. It is gone: numeric now holds any value PostgreSQL does, exactly, with its display scale.

It uses the same representation the Rust server adopted, so both servers store a numeric the same way. A value that fits a Decimal128 exactly is stored as one, so a Mongo client reading the same collection sees exactly what it saw before for ordinary numbers. Anything wider is stored in the column as a small document holding PostgreSQL’s text for the value and a sort key, and every part of the server that compares, sorts or adds numbers understands both forms.

Getting there turned up several older bugs, now fixed. +, - and * on a numeric rounded at 28 digits, below even Decimal128’s 34, and % converted its operands to floating point. ORDER BY over a numeric column holding NaN failed with an internal error, and so did UPDATE of a numeric primary key. numrange bounds printed large values in exponent notation.

Fixed

• sql/numeric.py (new): the two-form representation, PostgreSQL’s canonical text, a byte-sortable key with NaN above everything, and exact WHERE filters that work for either form. Ported from crates/secantus-pgplan/src/numeric.rs, with the Decimal128 neighbour calculation corrected for values near the smallest exponent (recorded for the Rust side in tasks/backlog.md).

• sql/typemap.py: writes, literals, typmod enforcement, rendering and sort keys understand the wide form; unary minus no longer rounds.

• sql/planner.py: =, <>, <, <=, >, >=, IN, BETWEEN and = ANY(ARRAY[...]) on a numeric column are exact; sum / min / max / avg over a numeric are computed exactly after the pipeline; an ORDER BY on a numeric in a grouped, distinct or joined query is done after the pipeline.

• sql/scalar.py: +, -, *, abs, round and % are exact.

• sql/executor.py: a numeric primary key rejects a duplicate by value (1e40 and 1e40.0), ON CONFLICT finds it by value, and an UPDATE that changes a numeric key no longer fails.

• sql/window.py: a window ORDER BY over a wide numeric no longer orders it as JSON.

• sql/ranges.py: numrange bounds accept the wide form and render in plain notation.

Testing

• tests/test_sql_numeric_wide.py: the Rust server’s own wide-numeric tests, whose expectations PostgreSQL 16 produced, running against the Python server: round trips in text and binary, arithmetic, every comparison operator, NaN placement, exact aggregates, and numeric primary keys.

oid[] casts parse, and regclass columns no longer crash

On the Python PostgreSQL server, a cast to oid[], regclass[], regproc[] or regtype[] ('{1,2}'::oid[]) was rejected as a syntax error, for SQL PostgreSQL accepts. psycopg writes an oid[] value exactly that way when it fills parameters in on the client side, so inserting such a value through a client-side cursor failed. These casts now work, and a regtype[] column can be created.

Creating a table with a regclass or regproc column failed with an internal error. The server does not support those column types; it now says so with a “not supported” error instead of crashing.

Found by replaying psycopg’s own random-data tests through the parser: 47 of 3,000 random schemas hit it before the fix, none after.

Fixed

• sql/planner.py: oid[] / regclass[] / regproc[] / regtype[] in a type position are rewritten to a form the SQL parser accepts and restored after parsing, using the parser’s own tokens so string literals are never touched.

• sql/planner.py: column-type checks no longer assume every parsed type is an enum member (regclass is kept as a plain string).

Testing

• tests/test_pg_oid_array_casts.py: every cast and column shape, a string literal left alone, the array type reported back, a client-side-bound oid list, and regclass / regproc columns failing cleanly. Fifteen of the sixteen tests fail without the fix.

The wide-numeric oracle tests stop colliding in CI

tests/test_pg_numeric_wide_grouping.py compares against a live PostgreSQL, and every xdist worker shares that one server. Each test dropped and recreated a table named t, so two parametrisations raced each other and CI saw both halves: relation "t" does not exist in one worker and a duplicate key on pg_type_typname_nsp_index in another.

Each test now works in a schema of its own, which leaves the cases’ SQL (which names a bare t) unchanged and drops the schema afterwards.

Fixed

• tests/test_pg_numeric_wide_grouping.py: a per-test schema on the oracle side.

Parse reports errors when it should, and ABORT works

The PostgreSQL server accepted a Parse it should have rejected. A statement naming a table that does not exist, or input that is not SQL at all, got a ParseComplete, and the error only came back later, at Execute. PostgreSQL checks both when it parses, so a client that prepares a statement now and runs it later (pgx’s Prepare, JDBC with server-side prepare) saw the error on the wrong message. Both are now reported in reply to Parse, and a syntax error names the token PostgreSQL names: this is not sql is an error “at or near this”.

ABORT, PostgreSQL’s synonym for ROLLBACK, was rejected as a syntax error. That was more than cosmetic: inside a transaction that had already failed, the ABORT meant to end it was itself refused, and the session stayed stuck until the client sent ROLLBACK. CHECKPOINT was rejected the same way and is now accepted.

lower() and upper() of a range now have the type PostgreSQL gives them: timestamp for a tsrange (not timestamp with time zone), date for a daterange (not a midnight timestamp), and a tstzrange bound keeps its +00 when cast to text.

scripts/detached_run.py, the helper for long runs, now works on Windows.

Fixed

• sql/pgextended.py, sql/planner.py: Parse resolves every relation with planning’s own resolver (planner.missing_relation, which also accepts views and sequences) and answers 42P01 for a missing one.

• sql/engine.py: any bare expression that parses in place of a statement is a 42601 (this is not sql parsed as NOT (this IS sql) and slipped past a list of node types), and the error names the leading input token.

• sql/planner.py: ABORT [WORK | TRANSACTION] parses as ROLLBACK.

• sql/engine.py: CHECKPOINT answers its command tag (SecantusDB has no user-driven checkpoint to force).

• sql/ranges.py, sql/planner.py, sql/scalar.py, sql/typemap.py: the result type of a range bound follows the range type, and ::text of a lower() / upper() looks through to the range column’s type.

• scripts/detached_run.py: a Windows liveness check (os.kill(pid, 0) is not one there), stop without process groups, and a bare python resolved through PATH, so it is the venv’s interpreter rather than the base one.

Testing

• tests/test_pg_parse_analysis.py: missing relations fail the Parse; 22 valid relation kinds (views, sequences, temp tables, CTEs, catalogs, UPDATE … FROM, …) still parse; ABORT ends a failed block.

• tests/test_pg_range_bound_types.py: types and text for every range type, plus a real-PostgreSQL comparison in the pg-oracle lane.

• tests/test_detached_run.py: three tests that run on every platform (the rest are POSIX-only by nature, which is how the Windows breakage went unnoticed).

Timestamp arrays keep their microseconds

A timestamp[] or timestamptz[] column on the Python PostgreSQL server stored every element to the millisecond: 10:00:00.826829 came back as 10:00:00.826, with no error. The same array written as an expression was exact, so the loss was only in storage.

A plain timestamp column keeps its sub-millisecond part in a hidden field beside the stored date. Array columns never had one, and each element of the array is stored as a date too. They now keep a parallel list of remainders, written only when some element needs one, and cleared when an update no longer does.

Found by psycopg’s random-data COPY tests, which failed whenever a draw put microseconds inside an array.

Fixed

• sql/subms.py: split / merge work element by element on arrays (nested arrays too); carries_subms names the column types that keep the companion.

• sql/planner.py, sql/executor.py: INSERT, COPY FROM, UPDATE and the whole-row read paths (including COPY TO) use it for array columns.

Testing

• tests/test_pg_timestamp_array_subms.py: parameters with NULL elements, a two-dimensional literal, updates that set and clear remainders, and COPY in and out in text and binary.

Filters and grouping see a timestamp’s microseconds

On the Python PostgreSQL server, some queries compared timestamps only to the millisecond, and so returned the wrong rows when two values differed only in their microseconds:

• A WHERE on a timestamp[] column (a = ARRAY[...], x = ANY(a), a @> ...) found nothing when the literal had microseconds, and a <> ... returned the equal row as well.

• A query with EXISTS or a correlated subquery that compared timestamps (EXISTS (... WHERE s2.t = s.t)) never matched.

• DISTINCT and GROUP BY on a timestamp[] column merged arrays that differed only in their microseconds, so counts came out too low.

All three now compare the full value. The data was always stored correctly; these paths read it without the stored microseconds.

Fixed

• sql/planner.py: a WHERE that reads a timestamp-array column is evaluated row by row; DISTINCT / GROUP BY on one groups on the full value.

• sql/executor.py, sql/scalar.py: the row-by-row WHERE reads columns with their microseconds, for the outer query and for the rows of a subquery.

Testing

• tests/test_pg_timestamp_subms_predicates.py: the array predicates, a correlated EXISTS, and DISTINCT / GROUP BY counts and values. All six tests fail without the fix.

Prepared pg_sleep / pg_notify / advisory-lock / lo_creat calls keep working

On the Python PostgreSQL server, select 1, pg_sleep(0.25) failed on its seventh run with 0A000 “cached plan must not change result type”. psycopg prepares a statement after five runs. After that, each Bind compares the type Describe reported with the type Execute produced, and for these functions the two were different.

Describe cannot run a volatile function, so it reads the type from a table. Execute types the call through the planner, which fell back to text. The two disagreed for pg_sleep, pg_notify, the blocking pg_advisory_lock* forms, pg_advisory_unlock_all, lo_creat and lo_create. Both now report what PostgreSQL reports: void for the first four, oid for the large-object creators.

Fixed

• sql/planner.py: return types for these functions match pg_proc.

• sql/engine.py: the Describe table uses void for pg_sleep and the advisory locks, and now covers the _shared / _xact / try variants.

Testing

• tests/test_pg_volatile_fn_describe.py: each function prepared and repeated, asserting PostgreSQL’s oid on every run, plus the original select 1, pg_sleep(0) shape under psycopg’s default prepare threshold.

pgjdbc gauge re-baselined

The committed baseline dated from 2026-08-16 and was taken over a materially different test population: today’s run executes 5,819 tests against that baseline’s 5,570. A regression count computed across different populations is not evidence in either direction, and the check had started reporting 23 “regressions” that were nothing of the sort.

Changed

• pgjdbc_validation/baseline.json regenerated from a complete current run: 82 standing failures across 49 entries (was 58 across 36). The verdict is clean again, so a real regression will stand out instead of being lost in a permanent delta.

The largest cluster — 35 failures in DatabaseMetaDataTest, 21 of them the same rs.next()-returned-nothing signature — is recorded in tasks/backlog.md with the six hypotheses already eliminated, so accepting them into the baseline does not quietly retire the question.

The Rust PostgreSQL server starts in-process now

Starting a SecantusDB server in a test has always been one or two lines with no external processes to manage — except for the Rust PostgreSQL server, which every test and the psycopg gauge had to spawn as a secantusd-pg subprocess, wait for a readiness line from, and remember to terminate. That server’s accept loop lived in its main.rs, so nothing else could reach it. The loop has moved into the library as secantus_pgserver::bind, and _secantus_server grows a PgServer handle over it that mirrors the Mongo server’s RustServer: PgServer(path) binds a socket on an OS-assigned port, .dsn is handed straight to psycopg, and .stop() shuts it down. Python is only the launcher — no statement ever enters it.

The interesting half of that move is who owns the store. WiredTiger’s close-checkpoint runs when Storage is dropped, and without it every write acknowledged since the previous checkpoint is gone — measured last month as a CREATE TABLE plus INSERT that the client had been told succeeded and that were not there afterwards. An Arc<Storage> parameter would have let a caller hold a second reference, skip the checkpoint, and hear nothing about it. So bind takes the Storage by value and the returned handle owns it for the rest of its life: stopping the server drains the connections, tears down the runtime, and drops the last reference, which is where the checkpoint happens. “Stopping checkpoints the store” is a property of the type rather than a rule someone has to remember, and if a wedged connection ever defeats the drain, stop says so on stderr instead of returning as though the data were safe.

Shutdown also had to learn to tell the connections, not just wait for them. A pooled PostgreSQL connection sits idle indefinitely — a real backend never hangs up on one — so a stop that only drained blocked for its full timeout whenever a caller left a connection open, which in an embedded test is most of the time: 10.8 seconds, measured. Every connection now selects its socket against a shutdown signal and lets go when it fires, which brings the same stop to 0.11 seconds with the writes still landing on disk.

secantusd-pg is now a thin CLI wrapper around the same bind — same readiness line, same --database handling, same clean SIGINT/SIGTERM shutdown — so there is one serve path instead of two, and a durability or shutdown fix lands in both the binary and the embedded handle by construction.

Added

• secantus-pgserver: bind() / RunningPgServer (address(), dsn(), idempotent stop(), Drop) — a synchronous, runtime-free entry point that owns its own tokio runtime and resolves the bound address before returning, so 127.0.0.1:0 works from a plain (non-async) caller.

• _secantus_server.PgServer: the embedded Python lifecycle handle — PgServer(storage_path, port=0, host="127.0.0.1", databases=None), with address / port / dsn / version properties, stop(), and the context-manager protocol. Behind the default-on pgserver cargo feature; _secantus_server.HAS_PGSERVER reports whether a build carries it.

Changed

• Stopping the PostgreSQL server now signals its live connections instead of only waiting for them, so an open client connection no longer holds the shutdown for the full drain timeout (10.8s → 0.11s). The old binary aborted the accept task and dropped a still-shared Arc<Storage>, which skipped the close-checkpoint silently in exactly that case.

• secantusd-pg is a CLI wrapper over secantus_pgserver::bind rather than carrying its own accept loop. Observable behaviour — the secantusd-pg listening on {bound} storage={home} readiness line, the ephemeral-port form, --database, and clean SIGINT/SIGTERM shutdown — is unchanged.

A WiredTiger session per statement, for statements that never read a row

Every statement the Rust PostgreSQL server runs outside a transaction block opened a WiredTiger session and closed it again — measured at exactly one per statement, including SELECT 1, which touches no data at all. Sessions are cheap but not free, and a client such as psycopg sends every statement through the path that pays for one.

A session whose transaction has finished holds nothing: the transaction is over, and any cursor closed with the scope that opened it. So rather than closing them, finished sessions are now parked and handed to the next transaction that asks. The pool is bounded, because a session is a WiredTiger resource the connection’s own limit governs, and over the cap a session is closed as before — the pool can never grow beyond what concurrent work actually needed. A session whose commit failed is never returned: closing it is what rolls its dead transaction back.

An extended-protocol SELECT 1 now costs 51.4 microseconds where it cost 55.2.

Changed

• Finished user-transaction sessions are pooled and reused rather than closed, bounded, with failed commits excluded.

A cache that switched itself off for the length of every transaction

The Rust PostgreSQL server keeps a process-wide cache of the type catalog so that a statement need not re-read and re-decode it. Opening a transaction block recorded the catalog version the block had seen; the statement that opened it then advanced that version, as every transaction-control statement does. BEGIN cannot change a catalog, but the bump happened anyway, and from that moment the recorded version and the live one disagreed — which is the exact condition under which the cache refuses to be refilled, on the reasonable ground that a block whose view has diverged should not publish its reads as committed truth.

The consequence was that no statement inside any transaction could ever use the cache. Counted directly, a short block took two cache hits and fifty misses, re-reading every catalog collection from storage each time; re-anchoring the recorded version to the bump that just happened turns that into twenty-six hits and eight, the remainder being first-time reads.

Worth being plain about the size of it: this recovers about three microseconds of a seventy-microsecond gap between statements inside a block and the same statements outside one. The catalog scan sat at the top of the sampled profile and looked like the cost; it was the most visible symbol rather than the dominant one, and with it gone the profile is flat and the gap is still there. What accounts for the rest is not yet known, and the backlog now says so rather than repeating the story that turned out to be worth three microseconds.

Fixed

• The type-catalog cache is usable inside a transaction block again, instead of being disabled from the block’s first statement onward.

It was never the transaction block; it was the uncommitted CREATE TABLE

For a while the Rust PostgreSQL server looked as though it charged a standing tax for working inside a transaction: the same select 1 cost about 125 microseconds inside an explicit block against 55 outside one, where PostgreSQL 16 is flat at 30 either way. Two rounds of profiling had gone looking for the hot symbol that owned the difference and had come back with candidates worth a few microseconds each. The mistake was in the question. Running the statement both ways and varying one ingredient at a time shows that a bare block is slightly cheaper than autocommit, as it should be — it has no per-statement transaction to begin and commit. What costs is a block that is holding uncommitted DDL, and the benchmark had been creating its table inside the block all along.

Once a block has created a table or a type, the session carries an overlay of catalog rows nobody has committed yet. Two separate caches back away from that overlay, and counting calls rather than sampling time showed how far: a statement in such a block re-read the type catalog from storage twenty times where the same statement in autocommit read it zero times, and rebuilt the planner’s user-type tables twice per statement where autocommit rebuilt them never. Both were reacting to the same thing for the same reason — a view that carries one session’s uncommitted DDL must not be published to the others — and both were using the catalog version number to express it, which cannot tell “another connection changed the catalog” apart from “I changed it myself”. A block’s own first DDL statement bumps that version, so from then on the test could never pass again and both caches stayed off for the life of the block.

The fix says what was actually meant in each case. The process-wide catalog cache is filled only from a read that did not run on the transaction’s own WiredTiger session, since that session is the one that can see the block’s uncommitted writes; a read on a fresh session sees exactly the committed catalog and is publishable wherever it was issued. The planner’s per-thread type tables now record which session owns a view that carries an overlay, so a session can reuse its own and no other session can pick it up — a worthwhile distinction, because two connections share worker threads, and dropping the owner from that key lets one connection cast to a type another connection has not committed. Inside a block that created a table, select 1 goes from 153 to 53 microseconds and select v from t where k = 1 from 134 to 62, with autocommit unchanged; a statement in a block is now marginally cheaper than the same statement outside one, which is the shape PostgreSQL has.

Fixed

• crates/secantus-pgserver: a statement inside a transaction block that had done any DDL re-read the whole type catalog twenty times and republished the planner’s type tables twice, costing ~100us per statement. Both caches now distinguish a session’s own uncommitted view from the committed one instead of switching off for the rest of the block.

• crates/secantus-pgserver: the planner’s thread-local user-type tables are keyed by the session that owns an uncommitted-type overlay, so a view holding one connection’s uncommitted CREATE TYPE can no longer be reused by another connection that lands on the same worker thread.

Changed

• bench/pg_statement_cost.py honours SECANTUSD_PG, so a worktree can measure its own binary instead of silently measuring the main checkout’s, and its docstring now says that --in-transaction varies “block holding uncommitted DDL”, not “block”.

getProcedureColumns could not describe a function’s arguments

pg_proc.proargmodes and proallargtypes were always NULL, so nothing recorded that a parameter was OUT or INOUT. Around that sat four more defects on the same rows, each of which independently broke a client reading a routine’s shape:

• proargtypes listed every parameter, but it is the function’s call signature — an OUT-only parameter is excluded. f3(IN a int, INOUT b varchar, OUT c timestamptz) advertised a three-argument signature where PostgreSQL 14 records 23 1043.

• prorettype was 2278 (void, an oid this catalog does not even define) for any function whose return type came from its outputs. With one output column it is that column’s type; with two or more it is 2249 (record).

• RETURNS TABLE (...) recorded nothing about its columns. PostgreSQL lists them in the same three arrays with mode t.

• RETURNS <composite> also reported void: a user type has no storage tag, so the name is now recorded at CREATE and resolved to the type’s oid at reflection time, where the catalog is in scope.

proallargtypes is declared oid[] rather than text[] because pgjdbc casts that array to Long[] — a text array throws ClassCastException inside the driver before any assertion runs.

Fixed

• proargmodes / proallargtypes are populated when any parameter is not a plain IN, and stay NULL otherwise — PostgreSQL’s own rule, which the driver switches on.

• proargtypes is the call signature, excluding OUT-only parameters.

• prorettype is derived from output parameters, RETURNS TABLE columns, or a named composite type.

• pg_type.typtype reports p for record and m for a multirange; both read b. Not cosmetic — pgjdbc decides whether to emit a leading returnValue row by switching on typtype, so record reading b gave a function with OUT parameters a spurious extra row. That one surfaced only after the argmodes were already correct, which is why it is in this change.

All values measured against PostgreSQL 14 on 2026-09-20.

Measured on pgjdbc’s DatabaseMetaDataTest (194 tests): 33 failures → 25, 20 distinct → 16, no regressions. funcWithDirection, funcReturningComposite, funcReturningTable and droppedColumns go green.

One narrowing worth recording: sqlglot parses RETURNS refcursor as a USERDEFINED type name — the same shape as RETURNS <composite> — even though type_tag_for_sql resolves it perfectly well. The first version of the composite-return fix claimed every USERDEFINED name and so dropped refcursor’s tag, making SELECT getref() describe its column as text (25) instead of refcursor (1790). The discriminator is whether the type resolves to a storage tag, not what sqlglot called it. Caught by the full suite, not by the gauge.

Rust server: numeric filters near the smallest exponent select the right rows

On the Rust PostgreSQL server, comparing a numeric column against a constant with more than 34 significant digits and a very small exponent (around 1E-6150) could return the wrong rows. For n > 1.2345678901234567890123456789012345E-6150, a stored 5E-6160 came back even though it is smaller, and n < missed it.

To compare a wide constant against the ordinary stored values, the server finds the nearest values on either side that the storage format can hold. Near the bottom of that format’s exponent range, that calculation gave up and answered “between 0 and 1E-6176”, which does not contain the constant at all. It now finds the true neighbours. Found while porting the same code to the Python server, which already had the corrected version.

Fixed

• crates/secantus-pgplan/src/numeric.rs: decimal128_bracket keeps only the digits that still fit above Decimal128’s exponent floor.

Testing

• crates/secantus-pgplan/src/tests.rs: the bracket contains its value for a 35-digit value at every exponent from -6180 to -6100 (and around 0 and the top), both signs, plus short values at the floor.

• tests/test_rust_pgserver_differential.py: every comparison operator on such a constant, against a real PostgreSQL.

The Rust PostgreSQL server supports FILTER on an aggregate

count(*) FILTER (WHERE n > 8) was 0A000 FILTER on an aggregate is not supported yet. Only the rows matching the filter now contribute, and a group where none match is the empty input — count is 0 and every other aggregate NULL, which is what PostgreSQL answers.

Each aggregate carries its own filter, so count(*) filter (where n > 8), count(*) computes both from one pass over the group, and a filter works inside HAVING and beside DISTINCT.

Added

• crates/secantus-pgplan: AggItem.filter, lowered with the same WHERE-lowering the rest of the planner uses.

• crates/secantus-pgserver: compute_aggregate applies it.

Testing

• tests/test_rust_pgserver_slice.py: grouped and ungrouped, a filter that matches nothing, two aggregates with different filters, one inside HAVING, and one beside count(DISTINCT ...) — measured against PostgreSQL 14.24.

The Rust PostgreSQL server: bool_and, bool_or, and honest refusals

Found by sweeping the existing pg_corpora/ corpora against the Rust server.

bool_and and bool_or were not recognised as aggregates at all, so they reached the per-row scalar evaluator. They now work — grouped or not, over a column or a comparison, skipping NULLs, and answering NULL over an empty input.

array_agg over an empty input answered an empty array where PostgreSQL answers NULL. Every other aggregate over an empty input was already right.

The rest of that cluster is still missing — string_agg, and an aggregate wrapped in an expression such as count(*) + 1 — but they now refuse while planning. That matters beyond the message: they used to be planned as a plain SELECT with a computed column, so the refusal came from evaluating a row, and over an EMPTY table nothing was evaluated, nothing refused, and the client got zero rows instead of the one row PostgreSQL returns.

Fixed

• crates/secantus-pgplan: bool_and / bool_or are aggregates; string_agg and a nested aggregate are refused while planning.

• crates/secantus-pgserver: the two boolean aggregates, and array_agg over an empty input.

Testing

• tests/test_rust_pgserver_slice.py: the boolean aggregates, array_agg over nothing, and that the unsupported shapes refuse the same way whether the table is empty or not.

The Rust PostgreSQL server supports avg()

avg() was refused with a note that it “returns PostgreSQL numeric with its own scale rules; approximating it would be a wrong answer”.

Those rules turn out to be the ones numeric DIVISION already follows here: a small quotient gets 16 decimal places, and an input carrying more keeps more — avg of two values with 20 decimals answers 20. So avg is the exact sum over the count, divided through the same code the / operator uses, and the scale is right without anyone deciding it twice.

A float input averages as float8; the integers and numeric answer numeric; an empty input is NULL. It composes with DISTINCT, FILTER and HAVING.

Added

• crates/secantus-pgplan: AggFunc::Avg and avg_result_type.

• crates/secantus-pgserver: the computation, over the exact sum.

Testing

• tests/test_rust_pgserver_slice.py: integer, float and numeric inputs, a 20-decimal input, the reported oids, an empty input, and avg beside DISTINCT, FILTER and HAVING — measured against PostgreSQL 14.24.

The Rust PostgreSQL server keeps a declared width

A char(4) column was described to the client as an unsized bpchar, and its value came back as ab where PostgreSQL sends ab  . Underneath was a sharper problem: the shared catalog records a declared string type as type: "text" with decl_oid: 1042 and an atttypmod, and the Rust side modelled neither field. So a char(n) or varchar(n) column created by the Python server read back as plain text with no width — and one created by the Rust server was written in a form the Python server read as text in turn. The values always survived; the declared type did not, in either direction.

Both servers now agree on what a declared width is and where it lives. A char(n) value is blank-padded on the way out (it is stored unpadded, so length() and a cast to text keep PostgreSQL’s meaning for free), and a comparison against one ignores trailing blanks on both sides — varchar stays blank-sensitive, as PostgreSQL has it.

The remaining char(n) rules — the functions that see the padded form, pattern matching, and ::char(n) truncation — are mapped in tasks/backlog.md.

Fixed

• crates/secantus-pgcatalog: Column carries typmod, reads and writes decl_oid the way the Python server does.

• crates/secantus-pgplan: CREATE TABLE records the declared modifier; a literal compared against a char(n) column is stripped.

• crates/secantus-pgserver: the row description carries the width, and char(n) output is blank-padded.

Testing

• tests/test_rust_pgserver_slice.py: padding and the described width, blank-insensitive comparison (with varchar unaffected), and a table created by the Python server read back by the Rust one with its declared type intact.

The Rust PostgreSQL server learns DISTINCT and the set operations

SELECT DISTINCT was ignored: select distinct s from t returned the duplicates, and DISTINCT ON (...) was dropped the same way. UNION, INTERSECT and EXCEPT were not recognised at all — a set operation has no FROM clause of its own, so it fell through to the constant planner and answered a single empty row. Both returned wrong results without an error.

All of them now work, matching PostgreSQL 14.24 across 33 differential cases: NULLs group as one value, 1.5 and 1.50 are one row, DISTINCT ON keeps the first row per key in ORDER BY order, and ALL keeps multiplicities — INTERSECT ALL pairs each right-hand row with one left-hand row and EXCEPT ALL subtracts them.

A set operation’s columns take their names from the left side and their types from both: within a type category PostgreSQL widens (int4 with int8 is bigint, numeric with float8 is double precision), across categories it refuses, and this now says the same thing in the same words — UNION types text and integer cannot be matched (42804). A count mismatch is 42601, and an untyped NULL takes the other side’s type.

count(DISTINCT col) works too, on every aggregate that has one — the group’s values are deduped by value before the function runs. array_agg(DISTINCT x) is the one that keeps NULL as a value and returns its result sorted, as PostgreSQL does. SELECT DISTINCT over aggregate output collapses groups that produced the same row.

Added

• crates/secantus-pgplan: Distinct on a planned select, and SetOpSelect as a statement of its own, with the ORDER BY / LIMIT / OFFSET that belong to the combined result.

• crates/secantus-pgserver: dedup for DISTINCT and DISTINCT ON, DISTINCT inside an aggregate, and set_op_rows — all matching rows by value, so numerics that differ only in scale count once.

Testing

• tests/test_rust_pgserver_slice.py: DISTINCT, DISTINCT ON, every set operation with and without ALL, ordering and limits, and the type rules, with values measured against PostgreSQL 14.24.

The Rust PostgreSQL server supports HAVING

select g, count(*) from t group by g having count(*) > 1 was refused outright with 0A000 HAVING is not supported yet, which ruled out most grouped reporting queries.

HAVING now filters the groups after the aggregates are computed and before ORDER BY, DISTINCT and LIMIT see them. An aggregate written only in HAVING is computed for the test and never projected, so group by g having count(s) = 0 works without selecting that count. Comparisons match PostgreSQL’s rules: a NULL on either side makes the test UNKNOWN rather than true, numerics compare by value, and a constant may be written on either side of the operator.

The accepted shape is deliberately narrow — a comparison or NULL test on an aggregate or grouping key against a constant, combined with AND / OR / NOT. Anything else is refused while planning, because HAVING decides which rows come back and a half-understood predicate would answer wrongly.

Added

• crates/secantus-pgplan: a Having predicate on the aggregate plan, and the parser that builds it (appending any aggregate the SELECT list omits).

• crates/secantus-pgserver: having_holds, applied to the grouped rows.

Testing

• tests/test_rust_pgserver_slice.py: the connectives, NULL handling, an aggregate only HAVING asks for, a grouping key, a constant on the left, and HAVING with no GROUP BY — all measured against PostgreSQL 14.24.

• crates/secantus-pgplan: the plan reuses an existing aggregate item or adds one, and an unsupported shape is still refused.

The Rust PostgreSQL server groups numerics by value

select v, count(*) from t group by v split a numeric column by its stored form rather than its value, so 1.5 and 1.50 were different groups — and so were 1e40, 1e40.0 and 1e40.00. Seven rows came back where PostgreSQL returns three, with each group’s sum wrong to match. Grouping compared the BSON values structurally, and a numeric carries its display scale.

A grouping key is now reduced to the value PostgreSQL groups on, while the group keeps its first row for display — the text PostgreSQL prints. Integers, strings and every other type keep their own equality.

Fixed

• crates/secantus-pgserver: group_key_ident normalises a numeric grouping key; GROUP BY matches on it.

Testing

• tests/test_rust_pgserver_slice.py: grouping over scale variants and a wide value, with the counts, sums and printed texts PostgreSQL 14.24 returns.

• That file’s server binary is now found on Windows (.exe), where all 1,194 of its tests silently skipped.

The Rust PostgreSQL server answers UPDATE and DELETE RETURNING

update t set ... returning id, n and delete from t ... returning * did the write, reported the right tag, and returned no rowset at all — a client that asked which rows it had just changed got nothing back. The clause was parsed and dropped: neither plan carried it. UPDATE now returns each row as it is after the update and DELETE as it was before, which is what PostgreSQL does.

Found by sweeping the existing pg_corpora/ corpora against the Rust server, which the differential probe can now drive.

Fixed

• crates/secantus-pgplan: Update and Delete carry a returning list.

• crates/secantus-pgserver: both project their affected rows.

• crates/secantus-pgserver: an aliased PRIMARY KEY kept its type. A key is stored as _id, and the row description looked the column up by its stored field and then by its output name, so select id as k matched neither and fell back to varchar — the client decoded an integer as a string. Only aliased primary keys were affected.

Added

• tools/probes/pg_differential.py --server=rust runs any existing corpus against the Rust server instead of the Python one.

Testing

• tests/test_rust_pgserver_slice.py: UPDATE / DELETE RETURNING including a no-match statement, and an aliased key’s type through both SELECT and RETURNING.

The Rust PostgreSQL server supports string_agg, and ORDER BY inside an aggregate

string_agg(s, ',') was refused because it takes TWO arguments and the aggregate planner rejected anything but one. It now joins the non-NULL values in group order, answers NULL over an empty input, treats a NULL separator as joining with nothing between the values, and — under DISTINCT — dedups and sorts, which is how PostgreSQL implements a DISTINCT aggregate. A non-text argument is reported as the missing FUNCTION PostgreSQL calls it (42883), not as an unsupported feature.

ORDER BY written inside an aggregate now works too: array_agg(s ORDER BY id DESC) and string_agg(s, ',' ORDER BY s) sort the group’s rows before the values are collected, which is the only thing that gives either aggregate a defined order. An ORDER BY over an expression rather than a column is still refused, rather than quietly answered in a different order.

Added

• crates/secantus-pgplan: AggFunc::StringAgg with its separator, and plan_aggregate_order for the in-call ORDER BY.

• crates/secantus-pgserver: both, over the group’s rows.

Testing

• tests/test_rust_pgserver_slice.py: grouped and ungrouped, empty input, a NULL separator, DISTINCT, FILTER, the 42883 for a non-text argument, and the ordering cases — all measured against PostgreSQL 14.24.

sum() of a float column no longer breaks the client

On the Rust PostgreSQL server, select sum(f) over a float8 column described the result as int8 and then sent 1.5, so psycopg raised invalid literal for int() with base 10: '1.5' — an exception, on an ordinary query, rather than a number.

sum()’s result type was “numeric if the input was numeric, otherwise int8”. PostgreSQL’s widening is not uniform, and now neither is ours (measured against 14.24):

input	sum
int2, int4	int8
int8	numeric
float4	float4
float8	float8
numeric	numeric
money	money
interval	interval

So sum(int8) was wrong too: it claimed int8 where PostgreSQL returns numeric.

Fixed

• crates/secantus-pgplan: sum_result_type, shared by the planner’s output definition and the wire type.

• crates/secantus-pgserver: the row description uses it.

Testing

• tests/test_rust_pgserver_slice.py: every numeric input type’s sum oid and value.

Ruling out the explanation that would have been ours

Two psycopg tests once failed in forty milliseconds where they should have taken four seconds, and an earlier note blamed the machine’s network for answering an unroutable address immediately. That note was wrong and has been withdrawn. The question it left behind was better: an immediate connect() failure is exactly what a harness that has run out of sockets produces, and that would be a defect here rather than a property of the machine.

Measured during a full gauge run, it is not what happens. Ephemeral port use peaks at eleven hundred of the sixteen thousand available and sawtooths rather than climbing, sockets in TIME_WAIT rise and fall with the work, and the descriptor limit on this platform is a million. Nothing is running out.

The failure is therefore still unexplained, with its most plausible cause eliminated by measurement instead of argument, and the backlog now says both halves of that. The probe is kept so the next person measures rather than reasons.

Added

• tools/probes/socket_pressure.py: samples descriptors, TIME_WAIT sockets and ephemeral-port use while a long run proceeds, and reports whether any of them climbs.

The standalone binaries need nothing but libc, and CI now proves it

The Linux archives linked liblz4.so.1 and libz.so.1 — WiredTiger builds both compressors as builtin extensions, so libwiredtiger_static references them — which meant “download, extract, run” quietly assumed the host already had two libraries a slim container image often does not. Both are now linked from their static archives on the binary release lanes, so secantusd-rs and secantusd-pg depend on the glibc family and nothing else. The wheel lanes deliberately keep the shared objects: auditwheel bundles those, and changing what a published wheel contains is not a side effect worth taking.

The larger problem was that nothing in the pipeline could tell. A published archive can be broken in a way the build, the tests and the smoke test all miss — the macOS binaries of 2026-09-19 linked liblz4 by its absolute Homebrew path and would not launch on a Mac without Homebrew, while every stage passed because the runner has Homebrew. otool -L on the artifact was the only thing that ever reported it, and nobody ran otool -L.

So the release workflows now run it themselves. Before an archive is packaged, .github/scripts/check_binary_deps.py reads the binary’s real dependency table — Mach-O load commands, ELF DT_NEEDED, or the PE import directory — and fails the release if anything falls outside a per-platform allowlist. Checked against the four artifacts that matter: it fails the published macOS binary naming the Homebrew dylib, fails the published Linux one naming both compressors, and passes the Windows archive and the fixed macOS build.

Added

• .github/scripts/check_binary_deps.py, run by both binary release workflows before packaging. The allowlist is glibc-family on Linux, /usr/lib + /System/Library on macOS, and OS DLLs on Windows — so a VCRUNTIME140 dependency would fail the release too.

• SECANTUS_WT_STATIC_COMPRESSORS=1 in secantus-wt’s build script, linking zlib and lz4 from their static archives on Linux. Opt-in, set only by the binary release lanes.

Changed

• The Linux release lanes install zlib1g-dev alongside liblz4-dev, for the .a archives.

Test temp dirs are reclaimed on the way out, and probe stores are reclaimed at all

A dev box filled its disk — 935 GiB, down to 50 MB free — with every janitor in the tree working exactly as designed. Three things were wrong at once, and none of them was the reaping rule.

The retained-run count was set against a suite that no longer exists: the code said “~1.7 GiB a run”, a measured full run leaves ~104 GiB, and keeping three of those is ~300 GiB of WiredTiger homes. The sweep also ran only at session start, so whatever the last run left stayed on disk until somebody ran pytest again — which may be days, or may be impossible because the disk is already full. And the ~17 differential probes under tools/probes/ each took a bare tempfile.mkdtemp() that nothing ever deleted: invisible to pytest’s janitor, which only manages pytest-of-<user>/, and to the gauge sweep, which only matches secantus-*-gauge-*. One session left 385 of them, ~50 GiB.

Fixed

• Keep one abandoned pytest run instead of three, and re-derive the per-run cost the comment quotes. The newest run is what a post-mortem needs; the two behind it were 200 GiB nobody read.

• Reap at session finish as well as session start, so the common case — run the suite, walk away — leaves the disk as the retention policy intends. The current run is protected by both existing rules (newest dir, live PID in its .lock).

• Probe stores are deleted when the probe exits, and carry their creating PID so an abandoned one can be reaped later. The atexit delete cannot be the whole answer — on Windows an open file cannot be deleted, so a probe killed mid-flight leaves its home behind with WiredTiger still holding it — hence _sweep_stale_probe_tmp, which applies the same PID-liveness rule the pytest sweep already used, and now runs from both invoke clean and pytest.

Added

• tools/probes/_servers.probe_store(): one self-cleaning, PID-tagged WiredTiger home for a probe, replacing the bare mkdtemp() in every probe.

• A test pinning that conftest.py defines pytest_sessionfinish and pytest_sessionstart exactly once each. The reap was first written as a second pytest_sessionfinish; Python silently kept only the later definition, so it never ran and nothing failed to say so.

secantusdb.com publishes a sitemap

The site had no sitemap.xml — /sitemap.xml answered 404 — so search engines had only in-page links to work from. Pelican now renders one from the content it already knows about: the four product pages, the blog index, and all 62 posts, each at the same .html URL its rel="canonical" declares, with lastmod taken from the post date.

Deliberately narrow. The paginated /blogN.html listings are duplicates of content already listed; /docs/ is excluded because the two Sphinx trees are grafted into the output after Pelican runs, so Pelican cannot enumerate them and a hand-written guess would rot silently.

Added

• sitemap.xml, generated at build time from the Pelican content tree.

Accumulator spec errors depend on WHERE the operator appears

$firstN, $lastN, $minN, $maxN, $median, $percentile, $topN and $bottomN all take a document spec. Given something else, mongod’s code depends on the operator’s position — a $group output field is accumulator position and an $addFields value is expression position, and they do not agree:

{$group:     {x: {$median: 5}}}   ->  7436100
{$addFields: {x: {$median: 5}}}   ->  7436201

Both servers carried one table and applied it in both positions, which was right for four of the eight operators and wrong for four. Measured across all eight × {scalar, array} × {accumulator, expression} against mongod 8.2.11: 16 of 32 cells diverged, now 0.

Fixed

• Accumulator position has its own codes — $median 7436100, $percentile 7429703, $topN / $bottomN 5788001, where the expression table says 7436201 / 7436200 / 168.

• An array spec in accumulator position is one message for all eight: 40237 The <op> accumulator is a unary operator, not the per-operator “specification must be an object”.

• $topN / $bottomN are accumulator-only. As an expression mongod does not recognise the name and never looks at the spec, so it answers Unrecognized expression '$topN'. A unit test had asserted the object-spec wording here and passed, because “Unrecognized expression” carries code 168 too and only the code was ever checked.

• $median / $percentile are validated in mongod’s IDL field order — input, then p, then method. Checking method first made {$median: {}} name method where mongod names input, and let a bad method outrank a bad p. The Python side’s ordering came from a 7.0.12 probe.

• The bad-method message was missing an article: mongod says “used as a percentile ‘method’”.

• $percentile’s p has three distinct codes, not one. A non-array or an EMPTY array is 7750301 naming the array, a non-number element is 7750302, and a number outside [0, 1] is 7750303. An empty p was accepted outright and produced an empty result; elements rendered as a and None rather than mongod’s "a" and null.

• On the Rust server the whole family deferred, which surfaces as 2 aggregation pipeline uses a stage or operator not supported by the Rust server — blaming the operator for a bad argument. Its $median / $percentile accumulator and expression forms now share one validator.

• These errors are raised at PARSE time, which is what fixes the wrapper. mongod reports them while parsing, so $addFields prefixes Invalid $addFields :: caused by :: and $group prefixes nothing at all. Raising them from the evaluator instead produced Failed to optimize pipeline on one server and Executor error during aggregate command on the other — right code, right sentence, wrong wrapper, which no code comparison can see.

Note

Adding the accumulator-only rule to the shared expression walker initially rejected every valid {$group: {x: {$topN: {…}}}} — the walker recursed into the accumulator and did not know which stage it was in. The stage name is now threaded through the whole walk rather than passed only at the top.

A failing $expr in a pipeline’s first $match lost its error prefix

MongoDB wraps an aggregation’s runtime failures in Executor error during aggregate command on namespace: <ns> :: caused by ::. SecantusDB did too — except for the very first stage. A leading $match is lifted into the initial fetch so it can use an index, which put it outside the block that adds the prefix, so {$match: {$expr: {$divide: ["$a", 0]}}} reported a bare can't $divide by zero as the first stage and the full wrapped message anywhere else in the same pipeline.

$sqrt’s domain error also carried a , but is -1 suffix that MongoDB does not emit — the one operator in that family that omits it, where $ln and $log10 keep it.

Of twelve probed runtime-error shapes, eleven now match MongoDB 8.2.11 exactly. The twelfth is MongoDB re-coding an error raised inside a $group accumulator (4848401 for a division by zero, 7157706 for $ln) — its execution engine substituting its own code per operator, with no rule derivable short of probing every accumulator, and deliberately not guessed at.

Fixed

• commands.py: the lifted leading $match gets the executor prefix.

• expressions.py: $sqrt’s domain message matches MongoDB.

Aggregation no longer invents fields or empty buckets

Two aggregation results carried data MongoDB never sends.

A field path that doesn’t resolve — {$project: {z: "$nope"}}, or "$n.k" where n has no k — came back as z: null. MongoDB omits the key entirely. The difference is invisible until code asks whether a field is present: "z" in doc answered yes where a real server answers no, and every document in the result carried an extra key. $addFields did the same, and a document literal went further, turning {$project: {z: {w: "$nope"}}} into {z: {w: null}} where MongoDB gives {z: {}}.

The rule being restored is narrower than “missing means null”, which is why the bug survived: a missing path is null when it’s an argument to an operator — {$add: ["$nope", 1]} is 1, not an error — and only missing when it’s the value of a projected field. Both behaviours are now pinned by tests so a future fix to one can’t quietly break the other.

Separately, $bucket emitted buckets that nothing landed in. An unused default came back as a bare {_id: "other"} with no count field at all, and empty boundary buckets were emitted too. MongoDB omits any empty bucket.

Both fixes landed on the Python and Rust engines together, checked against a live mongod.

Fixed

• A field path that resolves to nothing is omitted from $project / $addFields output rather than emitted as null, including through nested document literals. Missing paths remain null as operator arguments, matching MongoDB.

• $bucket emits a bucket only when at least one document falls in it — boundary buckets and the default bucket alike.

A $switch that can never match is now rejected, not silently ignored

An aggregation whose $switch had no matching branch and no default was supposed to fail. When every one of its cases was a constant — {case: false}, or {$gt: [1, 99]} — MongoDB rejects the pipeline while optimizing it, before a single document is read, so the error arrives even over an empty collection. SecantusDB instead waited until execution found a document to fail on. Over an empty collection there was none, so the pipeline reported success and returned an empty cursor: a query that could not possibly have done what was asked came back looking like it had. Adding one document to the same collection surfaced the error, which made the behaviour depend on the data rather than the query.

$replaceRoot was a second case of the same kind of drift. When newRoot evaluated to something other than a document it reported the generic type error, code 14, with a message no real server emits. MongoDB answers a specific code — 40228 — and a message that names the offending value, its BSON type and the input document that produced it. That last part is not the stored document but the pruned one, narrowed to just the fields the expression reads, which is what MongoDB’s dependency analysis hands the stage; $replaceWith shares the shape but names a different subject.

All of it was found by running the operations against a live MongoDB 8.2.11 and comparing the replies verbatim, and the thirty cases that pin it are now part of the differential gate.

Fixed

• $switch with only constant cases and no default is rejected during pipeline optimization (40069), matching MongoDB — including over an empty collection, where SecantusDB previously returned an empty cursor and ok: 1. A single field-referencing case, or a default, correctly prevents folding.

• $switch that finds no matching branch at execution time now reports MongoDB’s 40066 and its wording, rather than a generic 14 TypeMismatch.

• $replaceRoot / $replaceWith whose expression does not evaluate to a document now report MongoDB’s 40228 and its full message, including the value, its BSON type, and the dependency-pruned input document.

• $bucket’s out-of-range-value error (7158303) is now wrapped in MongoDB’s Executor error during aggregate command on namespace: … :: caused by :: prefix, which marks it as an execution failure rather than a parse error.

Malformed aggregation stages report MongoDB’s error, not a generic one

A malformed $sample, $unwind, $bucket, $densify or $fill spec reported BadValue on the Rust server, whatever was actually wrong with it. The engine signals “cannot do this” without a code, so a typo was indistinguishable from an unimplemented feature — a driver matching on the code saw the wrong error, and a caller could not tell which of the two had happened.

All five now report MongoDB’s own code and message: a negative $sample size is 28747, an unprefixed $unwind path is 28818, a one-element $bucket boundary list is 40192, a zero or negative $densify step is 5733401, and an unknown $fill method is 6050202.

Validation happens before the pipeline runs, which is where $facet already validated its own spec, so the engine’s error type did not have to widen.

Two of these were wrong on the Python server too, in a way that only a message-level comparison shows: $sample said “must not be negative” where MongoDB says “must be a positive integer”, and $bucket reported “found 1.” instead of “found 1 value(s).” The codes had always matched, which is why an earlier comparison of codes alone reported them as correct.

Expression arity and spec shape are PARSE errors, not fold errors

tools/probes/agg_expressions.py had never been reported on. Running it found 551 wrong error codes across 58 operators on a 3,968-case corpus — the largest unaddressed surface left. It is now 50.

The systematic cause

mongod builds the expression tree before it folds anything, so an argument-count or spec-shape mistake is a parse error, reported under the stage’s wrapper (Invalid $addFields :: caused by ::). We folded first and reported the optimizer’s (Failed to optimize pipeline), so 279 shapes carried both the wrong wrapper and the wrong code. aggregate._expression_shape_problem is the new pre-pass, alongside the literal-timezone one.

Fixed — a wrong answer

• An ObjectId or Timestamp IS a date. mongod accepts every BSON type carrying a timestamp, so {$year: ObjectId("64b7f9a2…")} answers 2023; we raised 16006 and refused the document. A one-element array is also the argument itself — {$year: [<date>]} — which we rejected too.

Fixed — codes and messages

• Arity ($indexOfArray / $indexOfBytes / $indexOfCP / $range / $slice): 28667, naming the bounds and the count.

• Date extractors given an array of any length but one: 40536.

• Object-spec expressions ($firstN, $lastN, $minN, $maxN, $median, $percentile, $topN, $bottomN): each carries its own Location code — 5787801, 5787900, 7436201, 7436200, 168 — not one shared code.

• Unrecognised date-spec arguments: eight operators, eight different codes, and only three append what they expected.

• $rand, $getField, $indexOfCP, $bitNot, $setEquals each answer mongod’s codes now. $getField with an unknown argument, and $rand with a non-container spec, previously answered ok.

The families that look uniform and are not

Two fixes had to be narrowed after they made things worse:

• $bitNot splits its message by whether the operand is a number at all (28765 vs a bare 14). Its three siblings $bitAnd / $bitOr / $bitXor name no type whatsoever and always answer 14.

• $setEquals checks arity before types and numbers the offending argument (2-th argument). $setUnion and $setIntersection accept a single array and say “One argument”, with their own codes.

Applying either rule to the family cost 46 shapes before the probe caught it.

Also

agg_expressions.py no longer compares $rand’s value — two correct servers disagree by design, and the moment $rand started succeeding the probe reported it as a wrong answer.

And a panic

Pointing the probe at the Rust server for the first time crashed a request thread: {$setEquals: []} reached &arrays[1..] on a zero-length slice. arrays.first() handled the empty case; the next line did not. Any client could take a thread down with a two-character argument. Fixed with mongod’s arity check (17045), which is also what makes the slice safe.

That measurement also put a number on the Rust server’s expression error surface for the first time — 1,376 of 3,968, with zero wrong values, almost all of it the engine deferring. Recorded in tasks/backlog.md.

Six aggregation stages answered differently from MongoDB

Follow-up to the aggregation-results probe: both servers now match MongoDB on all 48 stages it measures, up from 42 (Python) and 41 (Rust).

Fixed

• $setWindowFields returned rows in the wrong order. MongoDB emits partition by partition, in first-seen partition order, and within each partition in sortBy order. Both servers preserved the input order — the docstring even claimed that was correct. Wrong order is wrong results as soon as a $limit follows.

• {$count: {}} failed in a $group with “Unrecognized expression ‘$count’”. The accumulator existed and evaluated correctly all along; nothing ever reached it. The constant folder got there first — {$count: {}} reads no field, so it looked like a constant expression and was handed to the expression evaluator, which doesn’t know $count. It now works in $group and $setWindowFields, the two accumulator positions, and is still refused as an expression elsewhere. The $count stage ({$count: "<field>"}) is unaffected, including inside $facet.

• A dotted $project dropped the surviving parent. {$project: {"sub.k": 1}} over {sub: {}} emits sub: {} on MongoDB, and an array of documents is pruned element-wise. find’s projection had this right; the $project stage was a second implementation that only checked the leaf. It now delegates, so there is one implementation of the rule.

• $bucket put a Decimal128 in the default bucket. The placement used Python’s comparison operators, which have no ordering between a Decimal128 and an int — and the resulting TypeError was silently swallowed. It now uses the same BSON ordering the boundary validation already used.

• $bucketAuto split equal values across buckets and misreported max. 1.5 and Decimal128("1.5") are the same value to MongoDB, which never puts equal values in different buckets; and the remainder goes to the earlier buckets (8 values into 3 is 3/3/2, not 2/3/3).

• $bucket and $bucketAuto were fixed on the Rust server too, and had to be: fixing only the Python half turned six engine-parity cases red, because the two servers then genuinely disagreed. Parity doesn’t say which side is right, but it does say they move together.

• On the Rust server: $max over {NaN, Infinity} answered NaN — its bson_lt treated NaN as unordered, following Python’s < rather than MongoDB’s order, so the accumulator never moved off the first NaN it saw. And a $group keyed on a MinKey, Timestamp, Binary, Regex or Code failed the whole stage; each of those buckets by exact value.

Measured against mongod 8.2.11.

A sort no longer fails on a NaN, and true no longer groups with 1

aggregation_stage_specs and its tests cover the errors aggregation stages reject. Nothing covered the documents they emit — which is where a silently wrong answer lives: the pipeline succeeds, the shape looks right, and the rows are wrong. A new probe found real divergences on its first run, as the same gap did twice before for queries and updates.

Fixed

• A plain {$sort: {v: 1}} failed outright on the Rust server for any collection holding a NaN, a MinKey or a MaxKey — ordinary data, ordinary query, whole pipeline refused with “aggregation pipeline uses a stage or operator not supported”. It took $group, $bucket and $topN down with it, since those sort too. The sort gate had been kept deliberately narrow to protect the comparator from types with no ordering arm; every one of those arms now exists, so the gate was only firing a refusal.

• true shared a $group bucket with 1. MongoDB keeps them apart — a bool is not a number to it — while merging 1 with 1.0, and merging the signed zeros:

  bucket	members
  0	0, 0.0, -0.0
  false	false
  true	true
  1	1, 1.0

  The Python server merged true into the numbers; the Rust server was worse, keying by truthiness, so six documents collapsed into two buckets.

• A Decimal128 group key sat in its own bucket on the Python server, where MongoDB merges Decimal128("1") and Decimal128("1.0") with 1 and 1.0. On the Rust server it failed the whole $group instead.

Added

• tools/probes/aggregation_stage_results.py — 48 stages, comparing emitted documents and field order, which content comparison silently drops.

Measured against mongod 8.2.11.

Aggregation stages validate their specs, and four BSON types became queryable again

A sweep of every aggregation stage crossed with every pathological argument — 725 shapes, a surface no earlier sweep covered — found 167 divergences against mongod 8.2.11. It is now 22, and everything left is a message or validation-order difference rather than a wrong answer.

Fixed

• $unset validated nothing. A non-string, non-array spec raised a bare TypeError and reached the client as internal server error; an empty string and an empty array were accepted and did nothing; a document spec silently iterated its keys. mongod has four distinct codes for those.

• $out: "" and $merge: "" were accepted, writing to a nameless collection. Both are InvalidNamespace on mongod. Neither stage validated its field names either, so a typo was silently ignored — and $out takes {db, coll} where $merge takes {into: ...}.

• $documents is a collection-less stage. Run against a collection, mongod refuses it before looking at the argument at all; we ran it happily.

• Four BSON types were unreachable by $type. javascript, javascriptWithScope, timestamp, minKey and maxKey all validated as aliases but had no predicate, so the query silently matched nothing — and $type: "string" matched JavaScript values.

• $group on a field holding a bson.Code crashed, as did $count, $out and $merge given one as their spec, and $unset silently accepted one. Code subclasses str and defines __eq__ without __hash__; bsontypes.is_bson_string is now the check to use where the distinction is a BSON one.

• $set reported itself as $addFields, whose handler it shares.

Changed

• mongod has two value renderings and they are not interchangeable: they differ in six places ([ 1 ] vs [1], BinData(0, 7A) vs BinData(0, "7A"), ObjectId('…') vs bare hex, new Date(ms) vs ISO-8601, and two more). Everything else is identical, which is what made the difference easy to miss. bsontypes now holds one function per family, retiring four more partial copies.

Aggregation stage specs: mongod’s validation ORDER

tools/probes/aggregation_stage_specs.py is now 0 of 725 on the Python server, down from 22 (and 167 when the probe was written).

Fixed

mongod parses a stage spec field by field, so an unknown or specifically-missing field is reported before the generic “requires X and Y” — {$bucket: {a: 1}} names a rather than listing what is absent. Each stage has its own code and wording: $sample 28748, $bucket 40197 with a trailing period, $bucketAuto 40245 without one, and the IDL’s 40415 for $replaceRoot / $densify / $fill / $unionWith.

Three stages invert the rule:

• $lookup reports a missing from first (must specify 'pipeline' when 'from' is empty), and only checks unknown fields once from is present.

• $graphLookup does the same, and its message echoes the whole spec in mongod’s spaced document rendering — { a: 1 }, not the value renderer’s {a: 1}.

• $geoNear reports a missing near before objecting to the spec’s own type. An array is a document in BSON, so {$geoNear: []} gets the near message while a scalar gets the type error.

Two were not message differences at all:

• {$unionWith: ""} returned the outer documents unchanged — a wrong answer — where mongod rejects the empty namespace with 73.

• {$documents: {}} is rejected while the stage is desugared into a projection, so it answers 51270 even against a collection, where every other argument gets the namespace error.

Changed

• The probe grew a PROBE_SERVER column so it measures the Rust server too. It had only ever compared the Python one, which is why the Rust server was 219 of 725 divergent on the same corpus with nobody the wiser. That is now 0 as well.

Fixed — Rust server

Three of them were wrong answers: $out and $merge with an empty target namespace returned ok and wrote to a nameless collection — a $out that silently did nothing — and $unionWith: "" and $unset: "" likewise reported success having done nothing.

The rest were three families:

• A third value renderer. Both render_stage_value and render_value_compact ended in other => format!("{other:?}"), so every type without an explicit arm reached the client as Rust Debug — Regex { pattern: "a", options: "" } where mongod says /a/. And the two are not interchangeable: the 40228 / 17053 family quotes binary (BinData(0, "7A")) and wraps code as Code("x=1") where $limit’s renders BinData(0, 7A) and the code text bare.

• map_err(|_| Fallback::Defer) on seven stage dispatches discarded the real error — the same shape as the storage layer’s map_err(|_| QueryUnsupported). On the standalone server that reads as “the stage is unsupported” for what is a bad argument to a supported one.

• Validation order, the same rule as above, ported stage by stage.

$set also reported its errors as $addFields, because the two share an implementation and the message was hard-coded to the alias.

Numeric and cursor command arguments no longer crash the server

Extending the wrong-type sweep past document-valued arguments found 24 more slots that crashed with internal server error instead of returning a parse error: find’s limit / skip / batchSize, aggregate’s cursor and cursor.batchSize, listIndexes.cursor, createIndexes.indexes, and a $match stage whose spec isn’t a document.

mongod’s strictness here is per-slot, not per-class. find.limit: {} is a type error, while delete.deletes.limit: {} is accepted and means “no limit” — so a blanket “validate every numeric argument” rule would have fixed one and broken the other. Every slot was probed individually.

Fixed

• find.limit / .skip / .batchSize and cursor.batchSize return mongod’s BSON field '<path>' is the wrong type '<t>', expected types '[long, int, decimal, double']. find’s are reported under mongod’s internal IDL name — FindCommandRequest.limit, not find.limit. Booleans are rejected explicitly, since Python otherwise reads true as 1.

• aggregate’s cursor uses its own wording, cursor field must be missing or an object; listIndexes.cursor and createIndexes.indexes use the BSON-field form with 'object' and 'array' respectively.

• A $match stage whose spec isn’t a document returns 15959 / the match filter must be an expression in an object.

• delete’s limit is deliberately not type-checked: anything that isn’t a numeric 1 means “no limit”, matching mongod — including true, which mongod does not treat as 1. We used to call int() on it and crash.

Verified against mongod 6.0.16 and 8.3.4: zero crashes on both, and the same behaviour on both.

Wrong-typed command arguments are no longer accepted in silence

Nine argument slots took a value of the wrong type without complaint and reported success. find accepted collation: 5 and let: "x", create accepted storageEngine: true, collMod accepted index: 5, aggregate accepted let: 5, and the boolean slots — find’s singleBatch, findAndModify’s upsert, update’s multi — took objects and arrays. A driver sending a malformed command was told the operation had succeeded, which is the worst of the three ways this class of bug shows up: the earlier tranches either crashed the command or answered the wrong error code, and both of those are at least visible.

All nine now answer what MongoDB answers, and find’s maxTimeMS — the one slot in the sweep that is not a type error at all — answers its three distinct BadValue messages for a non-number, a non-integral number, and a negative one. The let option on update, delete and findAndModify is validated too.

Two related divergences turned up while probing and are fixed with them: an explicit null was accepted for find’s filter, sort and projection, and for aggregate’s cursor, where MongoDB rejects it. An absent option is still fine — the two cases had been indistinguishable in the code.

Nine slots needed six different message families, because MongoDB’s strictness here is per-slot rather than per-class. findAndModify.upsert accepts a bool or any number, so upsert: 1 and even upsert: 1.5 are valid, while the adjacent update.multi is a strict bool that rejects multi: 1. find.let is reported under MongoDB’s internal name FindCommandRequest.let while the same option on every other command uses the command’s own name. Six slots accept an explicit null and three reject it. Every one of these was measured against a live mongod rather than inferred from its neighbour, and delete’s limit — which MongoDB genuinely does not type-check — is pinned by a test so a future tidy-up can’t sweep it in.

Malformed pipeline stages report MongoDB’s error code

A $lookup, $group, $sort or $unwind stage whose specification was the wrong type returned an error — just not the one MongoDB returns. $lookup answered TypeMismatch where MongoDB answers FailedToParse; $group and $sort answered generic codes in place of their own; $unwind answered a message about a path when the problem was that the specification was not a path at all. Codes are what a driver branches on, so a client checking for a parse failure saw a type error instead.

find’s min and max cursor bounds had the same shape: a wrong-typed bound reached the index bound-checker and came back as an index error, where MongoDB type-checks the argument first and never gets that far.

Two of these were one condition doing two jobs. $sort used a single test for “not an object, or empty”, so a spec of 5 was reported as “must have at least one sort key” — true of {}, meaningless for a number. $unwind reported “expected a string as the path” for a spec with no path key at all, and the missing-$ message for an empty one, where both are “no path specified”. Splitting them means an empty $sort and a missing $unwind path keep their own distinct codes, which is why both are now pinned by tests.

Also fixed while separating those cases: {$unwind: {other: 1}} now reports the unrecognized option rather than complaining about the path it also lacks.

With this the wrong-typed-argument sweep reports 87 of 87 cases clean, from 24 crashes and 44 divergences when it started.

A null command argument no longer crashes createIndexes

createIndexes with indexes: null returned an internal server error instead of rejecting the argument. Several neighbouring slots were wrong in quieter ways: distinct reported a null key as the wrong type where MongoDB reports it as a missing required field, and on the Rust server a null was accepted in places MongoDB rejects it and rejected in two places MongoDB allows it.

These were found by widening the differential test corpus. It had been feeding three values per argument slot, none of them null — so a sweep reporting every case clean was accurate and, at the same time, silent about an entire class of input. The corpus now also covers fractional and negative numbers, decimals, ObjectIds and dates, and both servers match MongoDB across all 244 resulting shapes.

Worth knowing if you rely on this behaviour: null is not uniform. MongoDB accepts it for find’s let and listIndexes’ cursor, rejects it for find’s min and max and aggregate’s cursor, and gives createIndexes a different error code for a null list than for a wrong-typed one.

The wrong-type sweep reaches zero code divergences

The last of the Python server’s wrong-typed-argument work. Across the 685-shape corpus measured against mongod 8.2.11, code divergences are now zero (from 36 at the start of this slice), and message divergences are down to six.

Fixed

• create’s capped options were four separate defects. size or max present without capped: true was silently accepted where mongod answers 72; our “the ‘size’ field is required” message carried a trailing clause mongod does not have; a negative size answered 72 where mongod answers 2 BSON field 'size' value must be >= 1 (a floor of one, not zero); and max has no lower bound at all — 0, -1 and 2.5 are all accepted — where we rejected anything at or below zero. Two tests pinned the old behaviour and were rewritten.

• dropIndexes could not drop an index by key pattern. Every non-string index was rejected as a type error, so {index: {a: 1}} never worked. mongod resolves the key pattern, and answers 27 can't find index with key: { z: 1 } when nothing matches. Dropping the _id index is 72, not the 67 both arms returned.

• An explicit hint: null was accepted on all six commands that take a hint, where mongod answers 9 Hint must be a string or an object. An absent hint remains fine — the two are now distinguished.

• A null in a required field is “missing”, not “wrong type” — getMore.collection and renameCollection.to answer 40414. An optional typed field is the other way round: renameCollection.dropTarget: null is a 14.

• An empty write batch was accepted on update and delete (updates: [] or {}), where mongod answers 16 Write batch sizes must be between 1 and 100000. Got 0 operations.

• bypassDocumentValidation was unchecked on insert, update and findAndModify, and listIndexes.cursor.batchSize accepted a boolean through a bare int().

• Message families corrected to mongod’s own text: insert.writeConcern (was “writeConcern must be a document”), distinct (the IDL struct name, distinctCommandRequest.query), aggregate.pipeline (“A pipeline must be an array of objects”), renameCollection.to, dropIndexes.index (whose expected-type list differs for an array), and $facet, which rendered its spec with Python’s repr.

Known gap

The message for a hint naming an index that does not exist is mongod’s multi-line planner dump, which renders its parsed match-expression tree ({a: 1} becomes Tree: a $eq 1). Reproducing it in general means porting mongod’s MatchExpression::debugString; the error code already matches, and a partial renderer would be wrong on anything nested. Recorded rather than approximated.

A change-stream test asserted which 1-second window an event arrived in

test_absent_max_time_ms_still_waits_and_delivers inserted after 150 ms and asserted that one getMore came back with the event. That holds only while the inserting thread is scheduled inside the server’s 1 s default tailable wait. On a loaded runner it isn’t: the window closes empty and the test fails with [] == ['insert']. Seen on macOS CI, and reproduced here by delaying the insert past 1 s.

Delivering the event on the next getMore is correct behaviour, not a bug — so the test was asserting something the server never promised. The property it exists to protect is “an absent maxTimeMS waits for an event and delivers it”, and that is now pinned as two separate assertions instead of one race:

1. It waits. With nothing pending, an absent maxTimeMS must block rather than poll once — the inverse of the explicit maxTimeMS: 0 pinned by the test above it.

2. It delivers. An insert issued while a getMore is blocking comes back, in that window or the one after it.

The guard was checked both ways, because a flake “fixed” by loosening an assertion is worse than the flake:

• with the insert delayed to 1.4 s — the exact CI failure — the test now passes;

• with the server not blocking (simulated by sending maxTimeMS: 0), it still fails: an absent maxTimeMS returned after 0ms; it must WAIT.

Its siblings were checked for the same shape: test_change_streams.py already polls to a deadline, so this single-window assumption was unique to this test.

Fixed

• tests/test_getmore_maxtimems_zero.py: assert the blocking property directly and let delivery span more than one window.

--version says which source a binary was built from

secantusd-rs --version and secantusd-pg --version reported only a crate version — and this repo forbids version bumps in feature PRs, so hundreds of commits share one string. A user reporting “0.1.0-beta.0” had told you almost nothing about which code they were running.

Added

• Both binaries now print the git tree hash of the crates/ sources they were built from, on a second line:

  $ secantusd-pg --version
  secantusd-pg 0.1.0-beta.0
  tree: 30bfc40839ab7200307814e7b38a5267d6f73bbd
  

  Line 1 stays short for the common “what did I install?”; line 2 is what makes a bug report actionable. The tree: line is omitted, not blank, when the binary was built without git (an sdist, a release tarball).

• The binary smoke tests compare that stamp against HEAD:crates and fail when SECANTUSDB_BIN / SECANTUSD_PG_BIN is set — the variable the release workflow uses to say “smoke THIS artifact”, which is exactly when a stale binary would ship. A locally discovered binary only reports, because failing someone mid-edit is how a check gets switched off.

  This extends the _secantus_core provenance check to the two distributed artifacts, which accounted for the majority of the day’s staleness incidents.

Fixed

• secantusd-rs --version printed without a trailing newline, so its output ran into the next shell prompt.

A tree hash, not the commit SHA: HEAD:crates changes only when a crate’s content changes — measured stable across three consecutive commits touching only docs and tests — whereas a commit SHA moves constantly and a check built on it would be disabled within a week.

A stale build no longer looks like a regression

On 2026-09-18 eight separate incidents came from a built artifact being older than the tree it was tested against. Every one presented as somebody else’s regression, because the tests were current and only the artifact was old: a conformance gauge reported 1,084 failures and a 73.8% pass rate from a binary 50 crate-commits behind, and that wrong number reached the public website; a full suite produced 174 parity failures that were all fictional, the same files passing 207 of 207 after a rebuild with no code change; and three failures landed in a colleague’s new test file, reading as their bug, from a binary built 33 minutes before the fix those tests assert.

Reading the diff cannot catch this, because the evidence is not in the diff. The suite now checks at collection whether the installed _secantus_core was built from the sources in the checkout, and stops with the rebuild command if it was not.

The identifier is the git TREE hash of the crates, not the commit SHA and not the package version. The commit SHA moves on every commit, so that check would report stale constantly and be switched off; the version moves at release cadence, and was measured reporting beta.163 for an extension stale enough to need rebuilding twice in a day — green for exactly the builds that were wrong. A tree hash changes if and only if the crate’s content changes.

Added

• secantus-core-py: build.rs stamps the source tree hash, exposed as _secantus_core.__source_tree__. Absent without git history (an sdist, a build container), in which case the check abstains.

• tests/conftest.py: the collection-time check, and tests/test_build_provenance.py pinning its decisions — weighted toward the cases where it must stay SILENT, since a false positive is how a check gets disabled.

bulkWrite, sort on updates, and an honest 8.x version

SecantusDB advertised itself as MongoDB 7.0 while its error surface had already been retargeted to 8.x. That gap was not cosmetic: the advertised version is a capability contract, and drivers gate features on it. client.bulk_write() — MongoDB 8.0’s command for writing across several collections at once — was simply unavailable, and had a caller sent the command by hand they would have got CommandNotFound.

Three things landed together, because they cannot land apart. The driver spec suites assert in both directions: that a pre-8.0 server rejects sort on an update, and that an 8.0 one honours it. Implementing without advertising fails the first set; advertising without implementing fails the second.

Added

• bulkWrite — inserts, updates and deletes across multiple namespaces in one command, with MongoDB’s cursor-shaped reply and summary counters. Ordered batches stop at the first failing operation and unordered ones continue; upserts report the generated _id; errorsOnly suppresses successful results. A failing operation is reported against that operation, leaving the rest of the batch to run — an early version let it fail the whole command, so a driver saw no partial result at all.

• sort on updateOne / replaceOne — matches in sort order and updates the first. Rejected in combination with multi, as MongoDB rejects it, and an upsert that matches nothing still upserts.

• nsType on change-stream create events — collection, timeseries or view.

Changed

• The advertised server version moves from 7.0.0 (wire 17) to 8.2.11 (wire 27), matching the MongoDB the project targets and tests against.

Every shape was measured against a live MongoDB 8.2.11 rather than derived from documentation: the command’s reply layout, its five rejection codes, the ordered/unordered split, and the three nsType values.

bulkWrite reports mongod’s errors for a malformed command

A differential sweep of 47 bulkWrite shapes against mongod 8.2.11 found five divergences, all of them error shape — the command’s behaviour on well-formed input already matched.

Fixed

• A missing nsInfo was reported as a wrong type (2) rather than a missing required field (40414), and an explicit null was not recognised as missing at all.

• A non-array nsInfo reported a batch-size problem. mongod validates nsInfo before the operation count, so {ops: [], nsInfo: 5} is a type error and not “Got 0 operations” — our check order had it the other way round.

• nsInfo entries were unvalidated: a non-document entry, an unknown field inside one, and a non-string ns now answer mongod’s codes. The entry error carries its index (bulkWrite.nsInfo.0) while the field errors do not (bulkWrite.nsInfo.x) — mongod’s own inconsistency, reproduced.

• An invalid namespace ("nodot", "", ".") answered “invalid nsInfo index” instead of 73 Invalid namespace specified for bulkWrite: '<db>'.

• A negative namespace index answered our own wording instead of 2 BSON field 'insert' value must be >= 0 — which names the bare op kind, not the IDL path — and a wrong-typed index is now a 14 rather than a generic index error. A double index (0.0) is valid and still accepted.

• An unknown op kind now names the offending key (BSON field 'bulkWrite.frobnicate' is an unknown field.).

• filter is required on update and delete. It defaulted to {}, silently turning a malformed operation into a match-all — the only one of these that could change which documents a write touched.

bulkWrite sent a 32-bit cursor id, and refused a field every 8.x driver sends

The Go driver refused every bulkWrite reply with id should be an int64 but it is a BSON 32-bit integer. A cursor id is an int64 on the wire; this one was a bare 0, which BSON encodes as a 32-bit integer. A permissive driver cannot see the difference — pymongo accepts either, so the pymongo gauge had never noticed — but a type-strict one refuses outright.

Fixed

• bulkWrite’s reply cursor id is now Int64. Every other cursor reply in the codebase was already wrapped; this was the one that was not.

• bulkWrite now accepts bypassEmptyTsReplacement, which mongod 8.2.11 takes on bulkWrite, insert and update alike, and which the 8.x drivers append by default. insert and update already accepted it here; only bulkWrite refused, answering 40415 ... is an unknown field. It is accepted and ignored — the flag governs a Timestamp() substitution neither server implements. A genuinely unknown field is still refused with 40415.

Both fixes apply to the Python server; the Rust server shared the bypassEmptyTsReplacement refusal and already had a correctly typed cursor id, now pinned by a test so it cannot regress. The mongo-go-driver gauge goes from 30 bulkWrite failures to 3, the remainder being a genuine feature gap recorded in tasks/backlog.md.

bulkWrite results page through a real cursor

The reply cursor was always {id: 0, firstBatch: [...everything...]}, so a driver never issued a getMore — three of the mongo-go-driver’s CRUD prose tests assert that it does, and the C driver fails the same three numbered cases.

Added

• bulkWrite now returns a real cursor when its results do not fit one batch, on both servers. cursor.batchSize is honoured, the remainder pages out through {getMore: <id>, collection: "$cmd.bulkWrite"} against admin, and killCursors closes it.

• The first batch is limited by size as well as count: two upserts with 8MB _ids produce results that cannot share one 16MB reply, so one comes back and the rest follow — which is what mongod does and what a count-only rule misses.

Boundaries match mongod 8.2.11 exactly, including the ones that are easy to get backwards: a batch filled exactly keeps no cursor (unlike find), batchSize: 0 opens a cursor with an empty first batch, and errorsOnly with no errors pages nothing.

Change streams: two fatal errors that shared one wrong code, and events whose fields came out in the wrong order

Change streams had never been compared against a real MongoDB, because mongod refuses them on a standalone server and the differential harness spawns a standalone. Run against a single-node replica set, 14 of 41 cases disagreed.

Fixed

• A change stream configured with fullDocument: "required" or fullDocumentBeforeChange: "required" failed with code 280 ChangeStreamFatalError when the image was not available. MongoDB answers that with code 47 NoMatchingDocument and no error labels — a different condition, and a different code, from the 280 it uses when a pipeline strips the resume token. Both now match, message included. The two used to share one exception class, which is why one wrong code covered both; 280 is still correct for the stripped-token case and is asserted there by the driver spec suite.

• The stripped-token error itself was missing MongoDB’s Executor error during getMore :: caused by :: prefix and the trailing Expected: … but found: …, which names the resume token that was expected and what the pipeline left behind. Dropping the token, rewriting it with $literal, and rewriting it with $addFields all now produce MongoDB’s message exactly.

• Change events emitted their fields in the wrong order: wallTime came after documentKey instead of directly after clusterTime, and fullDocument was hoisted ahead of _id — so _id was not the first field. Field order is invisible to a document comparison, which is how nine event-construction sites drifted out of it without any test noticing. Events are now assembled in MongoDB’s order in one place rather than nine, which took the ordering divergences from 28 cases to 5 — and the five that remain are all explained by a missing field that is recorded in the backlog, not by ordering.

All three are fixed on both servers, and both now diverge from MongoDB on exactly the same remaining cases.

Known, recorded, not fixed

updateDescription still reports array edits in a shape MongoDB never produces: we emit truncatedArrays where MongoDB reports either the whole new array or the positional path of an appended element. This is measured across eight mutations and four array sizes in the backlog. It is not fixed here because the diff walk is mirrored in both engines and roughly nineteen assertions pin the current behaviour — that is its own change, not a tail-end addition to this one.

Added

• tools/probes/change_streams.py, the probe behind all of the above. It compares event contents and field order, and documents the replica-set requirement that kept this area unprobed.

A change stream could quietly ignore the options you asked it for

Phase 2 of tasks/remaining-work-plan.md, fifth and last surface: 13 change-stream shapes against a real replica-set mongod 6.0.16. All 13 diverged.

The dominant failure was arguments accepted and ignored, which is a worse shape here than almost anywhere else in the server, because the caller believes they asked for something. parse_spec guarded every field with isinstance and silently skipped a wrong-typed value, so a client that asked for fullDocument: "updateLookup" — or to resume from a token — got a stream that did neither and reported success.

Fixed

• A crash. A resume token that is valid hex but not valid BSON ({"_data": "aa"} — two hex digits, one byte) raised InvalidBSON straight out of the handler and escaped as internal server error (code 1).

• An unknown fullDocument or fullDocumentBeforeChange value is rejected (BadValue) instead of falling back to the default. Misspelling updateLookup used to give you a stream without lookups and no indication.

• An unknown $changeStream field is rejected (Location40415).

• A wrong-typed resumeAfter / startAfter / startAtOperationTime is rejected (TypeMismatch). These were the most consequential of the ignored arguments: the stream started from the beginning rather than the requested position, and said it had succeeded.

• $changeStream anywhere but the first stage is rejected (Location40602). We built an ordinary aggregation and answered an exhausted cursor — a “stream” that never yields an event and never says why.

• $changeStream: 5 answers mongod’s Location6188500 rather than a generic BadValue, and a non-hex resume token reports mongod’s own wording instead of our prefix wrapping Python’s fromhex() complaint.

• Event field order. fullDocument now sits immediately after operationType, where mongod puts it, rather than being appended at the end. The event contents already matched exactly; only the order differed, and the event’s field order is the contract drivers read off the wire.

Known gap

The $project-drops-_id fatal error carries the right code (280) and message body but not mongod’s Executor error during getMore :: caused by :: prefix — the same wrapper-prefix class already tracked for aggregation errors.

The differential cases for this surface are deliberately not added to tests/test_mongod_differential.py in this PR: another session is editing that file to generalise the mongod-version gating, and the coverage here lives in a dedicated test file instead so the two do not collide.

Change streams report array edits the way MongoDB does

An update that changed an array produced an updateDescription MongoDB would never send. Popping one element reported a truncatedArrays truncation; pushing one reported the entire array. MongoDB does the opposite of both: it resends the whole array for a $pop, and reports arr.5 for a $push.

The reason it was wrong in both directions is that the shape depends on the operation, not on what changed. $set: {arr: [1,2,3,4,5,6,7]} and $push: {arr: {$each: [6,7]}} produce an identical document, and MongoDB reports the first as a whole-array replacement and the second as two indexed appends. No comparison of the before and after documents can tell those apart, so the update itself is now an input to the diff.

Measured against MongoDB 8.2.11 and applied to both servers:

• $push / $addToSet report arr.<i> for each appended index;

• $set, $unset, $inc, $mul, $min and $max of an indexed path report exactly that path — $set: {"arr.7": 77} on a five-element array reports arr.7 and not the two nulls it silently creates;

• $pop, $pull, $pullAll, a sliced or sorted $push, and a whole-field $set report the whole array;

• an aggregation-pipeline update is diffed by value, and that is the one shape where truncatedArrays really is emitted.

That last point corrects a claim made when this was filed. The earlier sweep concluded MongoDB “never” emits truncatedArrays, having probed only operator updates; it emits it for pipeline updates, which is what the driver spec suite had been asserting.

Ten of the fourteen remaining differences in the change-stream sweep close with this. The four that remain are a separate, recorded gap in expanded events.

The Windows storage-engine job filled a small drive while a 147 GB one sat idle

storage-engine (windows-latest) failed with No space left on device and WinError 112 on a change that added no storage at all — the second Windows disk exhaustion on this job.

The suite genuinely needs the room. tests/conftest.py refuses a tmp_path_retention_policy that deletes a passed test’s directory mid-session, because one was tried and raced WiredTiger’s background threads into a WT_PANIC cascade, so every test’s WiredTiger home stays on disk for the whole session by design. Both engines already ship prealloc=false, the lever that tripwire names, and Rust’s larger file_max is a documented production-throughput choice — none of that is the problem.

The problem was which drive it needed the room on. Python’s tempfile reads TMP/TEMP, which on this image default to C:\Users\runneradmin\AppData\Local\Temp. RUNNER_TEMP is D:\a\_temp, and D: has ~147 GB free. The job filled C: while D: sat at 3 GB used of 150.

That was found by adding the reporting first — and the first version of that reporting was itself wrong, running df -h ., which measures the workspace drive (D: on Windows) and would cheerfully have shown 147 GB free on the run that died of a full C:. It now reports the directory the tests actually write to.

Fixed

• .github/workflows/test.yml, storage-engine job: TMP / TEMP / TMPDIR point at RUNNER_TEMP on Windows, so the per-test WiredTiger homes land on the drive with the space.

Changed

• Reclaim the Android SDK on Windows, mirroring the reclaim the test job already has on Linux (MSVC / CMake / Python / LLVM all live elsewhere).

• Report free space for the test temp directory before and after the storage tests, so the delta sizes the suite’s real appetite. The “after” step is if: always(), because a report that runs only on success never prints on the run that needs it.

The Rust PostgreSQL server honours client_encoding

secantusd-pg stores and evaluates everything in UTF-8, but until now it ignored SET client_encoding and hard-coded client_encoding = UTF8 in its startup reply. A client that asked for LATIN9 still received UTF-8 bytes, and — worse — nothing told the client its request had been dropped, so the two sides silently disagreed about the wire encoding. The one thing the server was careful not to do was report a client_encoding it did not actually honour: a ParameterStatus for an unhonoured GUC makes libpq switch its own codec and mis-decode every value, which is how a comparable DateStyle change once regressed 160 tests.

The server now transcodes result text to the client encoding and decodes incoming text parameters from it, in both the text and binary wire formats, for LATIN1 and LATIN9; UTF8 and SQL_ASCII (and every other accepted name) keep the internal UTF-8 bytes exactly as before, so a UTF-8 client is byte-for-byte unaffected. A character with no representation in the target encoding raises PostgreSQL’s 22P05 untranslatable-character error, with the same message text a real server emits. SET client_encoding (and set_config('client_encoding', …)) now validates the name against PostgreSQL’s encoding table — an unknown name is 22023, MULE_INTERNAL is 0A000 — stores the canonical spelling, and reports it via ParameterStatus only once output genuinely respects it.

The encoding is honoured on the way in as well. The query text itself — a literal select 'café €' or an alias as "prix €" typed by a LATIN9 client — used to be decoded as UTF-8 and reach the planner as U+FFFD mojibake, because the wire library discarded the original bytes before the server saw them. The vendored pgwire now keeps the raw Query / Parse bytes and lets the server decode them in the session encoding, and RowDescription column names travel back in that encoding too. A client_encoding in the startup packet (libpq’s PGCLIENTENCODING, psycopg’s client_encoding= option) is applied before the first query and re-reported under its canonical name; an unknown one fails the connection with PostgreSQL’s FATAL 22023.

Added

• crates/secantus-pgserver/src/encoding.rs: client-encoding name canonicalisation (PostgreSQL’s clean_encoding_name rules plus the ISO-8859-N / UNICODE aliases) and LATIN1 / LATIN9 byte transcoding, with unit tests.

Fixed

• crates/secantus-pgserver/src/lib.rs: track client_encoding per session from SET / set_config / (validated, canonicalised); transcode text output in encode_field_value and COPY-OUT text, decode text parameters in decode_parameter, and report client_encoding via ParameterStatus now that output honours it.

• crates/secantus-pgserver/src/lib.rs: decode the simple-query and Parse SQL text in the session client_encoding (decode_query_text), send RowDescription column names in it (transcoded_name), and apply a startup client_encoding parameter in post_startup. SHOW <guc> no longer holds the settings lock while building its result column (a deadlock once that column name went through the encoding lookup).

• crates/vendor/pgwire: Query / Parse carry query_raw (the undecoded C-string bytes) and both query handlers gain a decode_query_text hook; FieldInfo / FieldDescription gain name_raw so a handler can send a column name in a non-UTF-8 client encoding. The cursor example is updated for the earlier parameter_oids patch.

• crates/secantus-pgserver/src/lib.rs: json and jsonb gain their PostgreSQL binary wire form (the text verbatim; jsonb behind its one-byte format version), transcoded to the client encoding like text. Before this a binary-format COPY ... TO STDOUT of a json column failed with 22P03 after the CopyOutResponse had gone out, which psycopg reported as “you cannot mix COPY with other operations”. A binary-format json / jsonb PARAMETER now takes the same cast as a text one, so a jsonb sent as psycopg’s ASCII-escaped dump is normalised to the character rather than stored escaped.

$strLenCP measured a JavaScript value, and the $to* family parsed one

bson.Code subclasses str. A previous fix addressed the surfaces that NAME a type — $type and the shared type-name helper — and stopped there. Naming a type correctly and DISPATCHING on it correctly are different things, and the operators that consume a string still admitted a JavaScript value:

• $strLenBytes, $strLenCP and $binarySize returned 3 — the length of x=1 — where mongod refuses the argument outright. Wrong values, not wrong messages. Each already had an error branch naming the type correctly; the guard isinstance(s, str) meant it was never reached.

• $toInt / $toLong / $toDouble / $toDecimal / $toDate / $toObjectId fed the JavaScript source text to string parsers, so $toInt complained “Did not consume whole string” and $toDate tried to read x=1 as a date, where mongod answers 241 Unsupported conversion from javascript to <target>.

$toString had been fixed for Code already — at one site, with the type name hardcoded as the literal "javascript". That is the second one-site fix for this root cause, and it is why a scoped Code was still misnamed there.

$toBool is the measured exception. {$toBool: Code("x=1")} is true on mongod; every other target refuses. A single uniform “reject Code in $convert” guard would have been simpler, wrong, and would have looked correct against seven of the eight targets. All eight were probed (8.2.11, 2026-09-04/05) rather than assumed.

Against mongod the probe goes from 31 wrong codes and 148 message differences to 28 and 142 — exactly the nine attributable to this cause. The three Code cases that remain ($ifNull, $rand, $setEquals) are stage-wrapper differences that diverge for any value.

The Rust engine was already correct on all eleven: its Bson enum has a distinct JavaScriptCode variant, so the subclass trap cannot occur there.

Fixed

• expressions.py: $strLenBytes / $strLenCP / $binarySize test is_bson_string; the five non-bool arms of _convert_value do too, so a Code falls through to the ConversionFailure the function already raised; $toString’s rejection derives the type name instead of hardcoding it.

Changed

• tests/test_bson_code_type_and_signed_zero.py: 22 more cases covering both surfaces, scoped and unscoped, including the $toBool exception and why the fix is per-arm.

$type called JavaScript a string, and signed zero was dropped by five operators

Six defects, found by classifying the wrong-value bucket of tools/probes/agg_expressions.py instead of trusting the backlog’s summary of it. That summary said “5 wrong values, all a documented Decimal128 precision limitation”; the probe reported 24, so nineteen were uncharacterised.

bson.Code read as a string. Code subclasses str, so anything dispatching on isinstance(v, str) classifies a JavaScript value as a string. expressions._type_name was a FOURTH partial copy of mongod’s type vocabulary — the exact drift bsontypes.py was created to end — and it had the two bugs a hand-rolled copy gets: $type of a Code answered "string", and a compiled pattern answered "object". It is now deleted and delegated.

A scoped Code is a different BSON type. Code("x", {}) is type 15, javascriptWithScope, not type 13. The query language already drew this line; the shared helper did not, so every error message naming a scoped Code’s type was wrong. That one change closed 25 message divergences.

Signed zero. IEEE keeps the sign when a rounding lands on zero, and mongod does too. math.ceil returns an int, which has no -0.0, so $ceil / $floor / $trunc answered 0.0 where mongod answers -0.0 — and more than the probe showed, since its corpus held only a literal -0.0 while -0.5 is an ordinary input that was equally wrong.

The accumulator asymmetry. $add / $sum / $avg fold from a ZERO accumulator, so +0 + -0 makes a lone -0 come back POSITIVE; $multiply folds from ONE and keeps the sign. Both engines returned -0.0 from {$add: [-0.0]}, and the Rust shortcut justified itself with a comment citing the Python engine rather than the server. Neither $add bug was in the probe’s list — both were found by re-probing the neighbouring operators after the first hit.

Against mongod 8.2.11 the probe goes from 24 wrong values and 173 message differences to 20 and 148. All 20 remaining are the documented Decimal128-computed-in-float limitation, which is what the backlog now says.

Fixed

• expressions.py: _type_name delegates to bsontypes.bson_type_name; $ceil / $floor / $trunc keep IEEE signed zero; $add and $avg fold from a zero accumulator.

• bsontypes.py: a Code carrying a scope is javascriptWithScope.

• crates/secantus-core: the $add single-operand shortcut folds instead of returning the operand unchanged.

Changed

• tests/test_rust_expressions_parity.py: _same distinguishes signed zeros and recurses into arrays and documents, and all sixteen assertion sites go through it — twelve used a bare ==, under which -0.0 == 0.0. The suite had -0.0 in its fuzz pool and exercised $ceil 5,000 times a run without ever being able to see the difference. The remaining seven parity files still compare with ==; filed in tasks/backlog.md §7.

Collated sorting now orders the way MongoDB orders

Collation was implemented for matching and only for matching. Each string was folded down to a single value, and anything the fold did not distinguish fell back to comparing whole codepoints — so a collated sort put every accented word after z instead of beside its base letter, had no tertiary case order at all, and accepted caseFirst, backwards and numericOrdering while ignoring them. Sorting ["a", "á", "ä", "az", "b"] under {locale: "en"} gave a az b á ä where MongoDB gives a á ä az b.

Sorting is a different problem from matching, and it now has its own key: three levels in the shape ICU uses. The primary level is the base letters with accents removed and case folded (or split into digit runs when numericOrdering is on, so a2 sorts before a10). The secondary level is the accents, one entry per base character, weighted by an order measured against MongoDB rather than by codepoint — acute sorts before grave even though the codepoints run the other way — and reversed when backwards is set, which is what makes French cote < côte < coté. The tertiary level is case, flipped by caseFirst. strength truncates the key and caseLevel re-adds the case rank.

Measuring it turned up something worse than wrong order: a collated sort returned a different order depending on whether a collated index existed, because the index’s byte order is the single-level normalisation and could never express the levels. The index is still used to fetch — explain still reports IXSCAN and the scan stays narrow — but it no longer counts as having satisfied the sort. An index must change speed, never results.

Seventeen of nineteen cases in the new sweep now match MongoDB 8.2.11 exactly. The two that do not are locale-specific: Swedish sorts ä after z and Danish sorts å last, which is CLDR data rather than something decomposition can derive, and needs an ICU dependency.

Fixed

• collation.py: sort_levels builds the three-level ordering key; Collation gained caseFirst and backwards.

• ordering.py: sort_docs takes a collation and uses it for string comparison.

• storage.py: find_matching sorts with the query’s collation, and no longer treats a collated index walk as already sorted.

• aggregate.py: the $sort stage sorts with the pipeline’s collation.

Added

• tools/probes/collation_order.py: the standing sweep, run with and without a collated index so the two can never diverge again.

Invalid collation specs are rejected instead of silently ignored

Every malformed collation spec was accepted: a missing locale, strength: 9, strength: "x", a misspelled field name. The query then ran under a different collation than the caller asked for and reported success — so a typo in a collation silently changed which documents matched.

Fixed

• find, aggregate, count, distinct and findAndModify now validate the collation spec’s contents, matching mongod 8.2.11 exactly across 28 spec shapes: a required locale (an explicit null counts as missing, 40414), its type, strength’s type and range, strict-bool caseLevel / normalization / numericOrdering / backwards, the caseFirst / alternate / maxVariable enumerations, and unknown fields (40415).

  The rules are not symmetric and each was probed rather than inferred: an empty {} is accepted while {strength: 2} is not; strength: 0 is an enumeration error where 6 is a range error; strength: 2.5 is accepted but strength: true is not; and backwards uses a different wrong-type message from the boolean fields beside it.

• update and delete deliberately still accept any spec contents, because mongod does — validating them for consistency would reject specs a real server runs.

Known gaps

Collation matching is correct at every strength; collation ordering is not ICU — accents sort after z rather than beside their base letter, and caseFirst, numericOrdering, backwards and locale-specific rules are ignored. An invalid locale name is also still accepted. Both are measured and recorded in tasks/backlog.md §5, with the reason each was left rather than approximated.

The Rust PostgreSQL server reports column precision, scale and size

cursor.description on the Rust secantusd-pg server left precision, scale, display_size and internal_size blank (or, for internal_size, a bogus 0) for every column. psycopg derives those four attributes from two fields the server sends in each RowDescription — the type modifier (atttypmod) and the fixed byte width (typlen) — and the server was sending the pgwire defaults (type_modifier = -1, type_size = 0) for all of them. So a numeric(10,2) column reported no precision, a varchar(42) no display size, and an int4 an internal size of 0 rather than 4.

The server now emits the real typlen for every wire type from a per-type table (int4 -> 4, time -> 8, timetz -> 12, interval -> 16, every varlena type -> -1), and threads the declared modifier of a select null::type(mod) cast through to the wire: numeric(p,s), varchar(n) / char(n), bit(n) / varbit(n), and time / timestamp / interval precision are all encoded exactly as PostgreSQL encodes them (measured field-for-field against PostgreSQL 16). bit and varbit also now carry their own oids (1560 / 1562) instead of falling through to varchar. This is description metadata only — no value is decoded any differently.

The whole of psycopg’s test_column.py now passes against the Rust server (53 of 53, up from 7); the fix clears all 46 previously-failing cases, including the 30 test_details and 15 test_details_time parametrisations.

Fixed

• secantus-pgserver: RowDescription now sends a per-type typlen (type_size) and a declared atttypmod (type_modifier); bit / varbit map to their own oids.

• secantus-pgplan: a ::type(mod) cast in a constant SELECT carries its type modifier through to the column description.

Wrong-typed command arguments are parse errors, not crashes

Twice in one session a caller-supplied scalar reached code that assumed a document and crashed as internal server error: pipeline: [42] called len() on an int, update: 5 called .keys() on one. Sweeping that shape across every document-valued command argument showed the pattern was systemic — 45 of 56 probed argument slots crashed, where mongod returns a parse error.

A crash is worse than a wrong answer here: the client learns nothing about what it got wrong, and a bare internal server error is indistinguishable from a genuine server fault.

Fixed

Wrong-typed arguments now return mongod’s parse errors, matching its per-command message families rather than one blanket check:

• find — filter / sort / projection return Expected field filterto be of type object (mongod’s own missing space, reproduced deliberately).

• count.query, distinct.query, delete.deletes.q, update.updates.q, findAndModify.query / .sort / .fields return BSON field '<path>' is the wrong type '<type>', expected type 'object'.

• aggregate.pipeline returns 'pipeline' option must be specified as an array.

• update’s u accepts an object or an array: a scalar is 9 FailedToParse (Update argument must be either an object or an array), while an array of non-documents gets the pipeline-element error (14), since an array u is a pipeline.

Verified 56/56 against mongod 6.0.16 — the version the live differential gate spawns — and cross-checked identical on 8.3.4.

Comparison operators accept every BSON type

$cmp / $gt / $gte / $lt / $lte / $eq / $ne — and $expr, which routes through them — were gated on the Rust engine’s order::is_sortable. That predicate guards the sort engines, where a type lacking a transitive same-type arm would corrupt an ordering, and it deliberately excludes NaN, Binary, Timestamp, Regex, JavaScript and Min/MaxKey.

A single comparison needs no transitivity, and mongod compares every BSON type by its canonical rank. Gating on the narrow predicate turned each of those types into 2 … not supported by the Rust server, so one BinData document made an entire $expr query fail.

Measured against mongod 8.2.11 over 19 value classes × 3 operands × 7 operators: 120 of 399 cells diverged, now 0.

Fixed

• The comparison operators use a new order::is_comparable, which admits every type order::cmp ranks. is_sortable is unchanged, so the sort engines keep their narrower guarantee.

• order::cmp placed NaN equal to every other number. mongod ranks it below them — {$cmp: [NaN, 5]} is -1 — and this project’s own storage sort already did, so the comparison operators and the sort disagreed with each other. A unit test had pinned the Equal behaviour; it asserted Python’s <-is-false-both-ways rather than anything measured.

• order::cmp gained the missing same-type arms for JavaScript code, which previously fell through to the numeric branch and compared equal.

The Python server was already correct here — its _bson_lt covers the wider set — so this was a Rust-only gap, and it was found by sweeping the Rust server against mongod rather than against the other engine.

Composite field selection and NULL-blind composite equality on the Rust PostgreSQL server

The Rust PostgreSQL server (secantusd-pg) now understands (expr).field — the field-selection syntax psycopg emits when a composite is registered — and it now compares composite VALUES the way PostgreSQL does, which is not the same as comparing two row constructors. A composite with a NULL field, compared for equality against another composite, is equal when their NULL fields line up; only a bare ROW(...) = ROW(...) keeps SQL’s three-valued rule where a NULL makes the result NULL. Composite fields that are themselves composites compare by the same rule, recursively.

Together these close the composite-planner half of psycopg’s test_composite.py gauge: field selection ((mycol).foo, (ROW(1,2)).f1), composite =/<>/< with NULL fields, and comparison of nested records all now match a real PostgreSQL 16 oracle. What remains — binary anonymous-record result encoding and reserved-keyword quoting in regtype::text output — is tracked in the backlog; neither is a planner-comparison gap.

Added

• secantus-pgplan: (expr).field field selection (AIndirection) in the scalar expression evaluator and the const-column projection allow-list. A named composite resolves the field name through its declared field list (and reports the field’s declared type in the row description); an anonymous record names its fields f1, f2, … by position. A field of a NULL composite is NULL; an unknown field is 42703, with PostgreSQL’s own message (column "x" not found in data type t / could not identify column "x" in record data type).

Fixed

• secantus-pgplan: record_compare now takes the operand kind into account. Comparing composite VALUES treats two NULL fields as equal and a NULL as sorting larger than any non-NULL (so the result is always true/false), while a bare row constructor keeps the three-valued NULL rule. The row-constructor case is detected from the AST (N::RowExpr on both sides) in the expression evaluator; every other record comparison is a value comparison.

• secantus-pgplan: record_compare recurses into a field that is itself a record, so comparing composites whose fields are composites (ce2 = ce2) no longer fails with “comparing document with document using =”.

Composite values in the binary wire format, and arrays of composites

The Rust PostgreSQL server (secantusd-pg) now hands a composite value back the way PostgreSQL does when a client asks for it in binary. A SELECT row('hi', 10, 20)::mytype on a binary cursor used to come back as the text (hi,10,20) — every field a string — because the composite had no binary encoder. It now goes out in PostgreSQL’s binary record format (a field count, then each field’s oid, length and bytes), so psycopg’s binary composite loader decodes each field to its real Python type: the float8 is a float, the int8 an int. Nested composites recurse, so a composite whose field is itself a composite round-trips in binary too.

An array of composites — SELECT array[row('hi', 10, 30)::mytype] — now reports the composite’s array oid rather than falling back to varchar. Before, the whole array arrived as the 17-character string {"(hi,10,30)"}; now a registered client parses it into a one-element list of composites, in both the text and the binary wire formats.

Added

• secantus-pgserver: composite / composite-array binary result encoding (encode_binary / element_binary / a new record_binary helper) and a composite ARRAY arm in user_wire_type reporting the derived typarray oid. Composite wire types now carry Kind::Composite with their declared field types, so the binary encoder knows each field’s oid.

Fixed

• secantus-pgserver: a composite-array result in a TEXT cursor was double-escaped (its element oid is a user oid the generic array path does not know, so it hit the catch-all) — it now renders each element’s (...) text and escapes once, matching PostgreSQL.

Composite VALUES round-trip on the Rust PostgreSQL server

With CompositeInfo.fetch already reading the catalog, the Rust PostgreSQL server can now round-trip a composite VALUE on the wire. A '(1,x)'::testcomp text cast and a row(1,'x')::testcomp record cast — the first used to answer invalid input value for enum testcomp, the second a record cast to testcomp is not supported yet — both parse into a composite datum whose result column carries the composite type’s own oid, so psycopg’s register_composite loader fires and hands back the registered namedtuple. A composite param cast on the wire (%s::testcomp), an INSERT/SELECT through a column of the composite type, and an array of composites all round-trip, with field escaping (NULL, the empty string, and fields carrying commas, quotes, backslashes, parentheses or whitespace) matching PostgreSQL exactly across all 255 single-byte characters.

On psycopg’s own composite test suite this moves the Rust server from 25 to 45 of 79 passing. The remaining families — the composite BINARY wire format, a composite parameter typed with its own oid (rather than explicitly cast), pg_typeof / field access of such a parameter, and per-element loading of an array of composites — are tracked in tasks/backlog.md.

Added

• secantus-pgplan: a composite arm in cast_value that parses PostgreSQL composite TEXT (parse_composite_text) or coerces a record’s fields into a composite value (composite_value), plus a PLAN_USER_COMPOSITES thread-local (set_user_composites / user_composite / user_composite_oid) carrying each composite’s field metadata to the planner.

• secantus-pgserver: install_user_types now pushes composite field metadata, and user_wire_type reports a composite result column under its own oid so a client that ran register_composite decodes it.

Fixed

• secantus-pgplan: an array element that is a record / composite now renders as its (...) text instead of leaking Rust’s {:?} debug form ({"Document({...})"}) into the array literal.

find evaluates a computed projection field, instead of dropping it

find({}, {total: {$multiply: ["$price", "$qty"]}}) returned documents with no total in them — and reported success. The client asked for a computed field, was told ok: 1, and got documents without it.

Computed projections are now evaluated per document. Almost every rule involved had to be measured against a real server rather than reasoned about, and the ones that were reasoned about first were all wrong:

• Only a BSON number or bool is an include/exclude flag. A string, null, an array, a date, an ObjectId, BinData, a regex, a Timestamp, MinKey/MaxKey are literal constants that replace the field on every document. Decimal128("1.5") includes a field and Decimal128("0") excludes one — bool() of either is True in Python, so the obvious test is backwards.

• A plain sub-document is a sub-projection, classified per leaf. {o: {p: 1, z: "$b"}} returns the stored o.p alongside the computed o.z, so the sub-document cannot be classified as a whole.

• A computed _id replaces the stored one and moves to the end: {_id: "$b", a: 1} gives {a: …, _id: …}.

• A bare field reference that resolves to nothing omits the output field, while an expression over a missing field yields null.

• An evaluation failure is an execution-time error carrying the namespace, the way mongod reports a per-document failure: $size on a non-array is Executor error during find command: <db>.<coll> :: caused by :: ….

Mixing a computed field with an exclusion has three different errors, picked by the first offending field in spec order: 31253 for an inclusion flag, 31310 for a literal (with the value rendered as a BSON debug string, n: [ 1, 2 ]), and 31252 for an operator expression. Swapping two fields in the spec swaps which code comes back. An empty sub-document is its own error, 51270, naming the leaf.

A test in tests/test_projection.py asserted that {_id: None} and {_id: ""} were includes, under a docstring reading “Oracle-pinned against real mongod”. Nothing in that file can reach a mongod, so the claim had never been run — the server returns the constant. Corrected in place. The Rust engine carried the same wrong rule, with the same comment, and the projection parity suite had been green throughout because the two engines agreed with each other on the wrong answer — the “parity is not correctness” shape, caught here by the parity suite turning red only after the Python side was corrected.

Unchanged, and deliberately: the Rust server still answers 2 BadValue: projection is not supported by the Rust server. Refusing is the better of the two behaviours when the feature is absent, and the two servers are now honest-refusal versus correct rather than honest-refusal versus silently wrong. The Rust half is filed with the same measured semantics.

Fixed

• secantus.projection: an expression-valued projection field is evaluated per document; the spec is flattened to dotted leaves so a sub-document is classified per leaf and errors name the field mongod names; the flag/literal split follows the BSON type rather than Python truthiness.

• secantus-core’s projection engine defers a non-numeric _id spec value instead of returning the stored _id, so the two engines agree on mongod’s answer rather than on each other’s.

• secantus.commands: a computed projection’s evaluation failure is wrapped as Executor error during find command: <ns> :: caused by :: …, which needs the namespace only this layer knows.

The Rust PostgreSQL server terminates and aborts like a real backend

The Rust secantusd-pg server now speaks the connection-lifecycle side of the wire protocol the way a real PostgreSQL backend does: it can tell you its own process ID, terminate a backend, and — crucially — surface the failure of a statement inside a transaction so that the block is poisoned until it ends. A pymongo-style permissive client never noticed the gaps here, but psycopg does, and five of its test_connection.py lifecycle checks now pass against the Rust server.

pg_terminate_backend(pid) behaves as it should on both sides of a connection. Called on your own backend it ends the connection with a FATAL 57P01 (AdminShutdown) and closes the socket, so the client learns the connection is gone rather than believing it is still usable — a distinction that only shows up over the simple query protocol, where PostgreSQL sends no ReadyForQuery after a FATAL. Called on another backend it arms a flag that the victim notices at its next statement and ends with the same 57P01; terminating a PID that is not a live backend returns false. pg_backend_pid() reports the PID pgwire assigned during startup.

The other half is the aborted-transaction rule. A statement that fails inside a transaction — whether it fails while the simple protocol splits it or while the extended protocol describes it — now poisons the block, so every later statement gets 25P02 (InFailedSqlTransaction) until COMMIT/ROLLBACK, and a COMMIT of a failed block rolls back. Previously the failure went unrecorded and the aborted block kept accepting commands.

Added

• crates/secantus-pgplan / crates/secantus-pgserver: pg_backend_pid() and pg_terminate_backend(pid) (self- and cross-connection), resolved from per-connection state through two new ConstCol variants and a process-wide backend registry.

Fixed

• crates/secantus-pgserver: an error inside a transaction now poisons the block from every path that can raise it — the simple protocol’s statement split and the extended protocol’s Describe, not only Execute — so the next statement correctly gets 25P02.

• crates/secantus-pgserver: a FATAL error over the simple query protocol now closes the socket without a trailing ReadyForQuery, so the client sees the connection break instead of thinking it is still open.

An error in a constant expression says so — Failed to optimize pipeline, not Executor error

mongod carries an expression error under one of two prefixes, chosen by when it failed. A constant expression is folded at optimization time and reports Failed to optimize pipeline :: caused by ::; one that reads the document fails per document under Executor error during aggregate command on namespace: … :: caused by ::.

Both servers always used the executor form. That was 618 of the Python server’s message-only differences and 148 of the Rust server’s — the single largest item left in the expression sweep, and a plan entry deferred three times as “modelling constant folding”.

The rule, measured

It is statically decidable, which is what made this cheap. Probed on mongod 8.2.11, the predicate is simply does the expression read the document:

folds	does not fold
literals, $literal	a field path ($s)
$$NOW, $$CLUSTER_TIME	$$ROOT, $$CURRENT
the command’s own let values	a variable bound from the input ($$this in $map)
$let whose bindings are constant	$let bound from a field
nested constants	$rand

Fixed

Both servers now fold constant sub-expressions before running the pipeline — which is what mongod’s optimizer does — and report an error in one under its own prefix. Python’s message-only differences fall 669 → 143, and the Rust server’s 148 → 4.

The predicate is conservative: anything unrecognised counts as non-constant, which keeps the previous behaviour rather than risking the wrong prefix.

Two stages are deliberately not folded, both found by the differential gate:

• $switch folds to a different error than it raises at execution — 40069 Cannot execute a switch statement where all the cases evaluate to false without a default, not 40066 — and a dedicated path already models that. Folding it here reported the execution-time error under the optimization-time prefix, which is neither answer.

• $redact, whose decision variables are bound by the stage itself with marker values the folding evaluator does not hold; it reads the document anyway, so there is nothing to fold.

What is left of the message-only residue

The remaining 143 on Python are a different family: number rendering inside the message. mongod prints 1.09951e+12 where we print 1099511627776, and 0 where we print 0.0. Recorded.

17 cases added to tests/test_mongod_differential.py, each a pair — the same error reached both ways, so the two prefixes are pinned against each other.

$convert and the $toX operators now agree with MongoDB on 480 shapes

$toInt: " 5 " returned 5. Python’s int() strips surrounding whitespace and accepts PEP-515 underscores, so "1_0" became 10 — MongoDB’s parser accepts neither, and both are wrong values, not wrong messages. $toDate: 1 returned an epoch date where MongoDB refuses an int32 outright, and $convert of an empty string to bool returned false where every BSON string is true. And Decimal128("Infinity") to an integer target reached int(Decimal("Infinity")), whose OverflowError escaped the handler and reached the client as 1 internal server error.

Underneath all of it was the same shape: two implementations of one conversion. $toInt / $toLong / $toDouble / $toDecimal each carried their own copy of the logic alongside $convert’s, and the copies had drifted — different overflow messages, different unsupported-type errors, and none of them knew MongoDB separates NaN from infinity from merely out of range. The shorthands now delegate to $convert, so they cannot drift again.

String parsing is strict and reports MongoDB’s reason for refusing: No digits, Did not consume whole string., Overflow, Leading whitespace, Empty string, Did not consume any digits, Failed to parse string to decimal — which are not the same set for integers, doubles and decimals, and are not interchangeable. Hexadecimal input gets MongoDB’s other message shape entirely. $toObjectId, which did not exist, now does. And the conversion shorthands accept the single-element array form ({$toInt: ["$field"]} — the way a field reference is naturally written) with MongoDB’s own wrong-arity error for anything else.

A sweep of 480 $convert and $toX shapes plus 51 date and objectId shapes against MongoDB 8.2.11 is now at zero divergences.

Fixed

• expressions.py: strict string→number parsing with MongoDB’s per-target reasons; NaN / infinity / overflow separated; unsupported source types answer ConversionFailure naming both ends; $toString uses BSON spellings (true, Infinity, NaN); every string converts to true.

• expressions.py: to: "date" accepts a long / double / decimal / ObjectId / Timestamp and rejects an int32, and returns a naive UTC datetime that compares with a stored date instead of raising TypeError against one.

• expressions.py: the four $toX numeric shorthands delegate to $convert.

Added

• expressions.py: the $toObjectId operator, which did not exist.

• expressions.py: $stdDevPop and $stdDevSamp in EXPRESSION position — over an array argument in $project / $addFields. The accumulator forms shipped long ago; the expression forms answered Unknown expression. They share the accumulator’s fold, so the two forms cannot disagree. Found by the same sweep, which also confirmed that most of the “missing operators” list in docs/feature-comparison.md had quietly started working.

Fixed (Rust engine, to keep the two in step)

• secantus-core: $convert: {to: "bool"} returns true for every string. The $toBool shorthand already did — the two copies had drifted inside one engine, the same shape as the Python split this change collapses.

• secantus-core: to: "date" refuses an int32, matching MongoDB; it used to read one as epoch milliseconds.

COPY BINARY on the Rust PostgreSQL server encodes real binary, not text bytes

The Rust secantusd-pg server accepted COPY ... (FORMAT BINARY) in both directions, but on the way out it only knew how to write the four types whose value happens to encode the same in text and binary — int4, int8, float8, bool, text. Every other column was handed to the binary encoder as its PostgreSQL text rendering, so a smallint went out four bytes wide, a numeric / date / time / timestamp / bytea / array column went out as ASCII digits behind a binary length prefix, and a strict client reading the stream back mis-parsed it (psycopg’s binary array loader read a garbage dimension count and raised DataError). A round-trip only survived because our own decoder read the same wrong bytes back.

COPY binary out now goes through the exact encode_binary codec the SELECT binary path uses — a DataRowEncoder over a binary-format schema produces the [i32 len][bytes] layout a COPY field wants byte-for-byte — extended with the three temporal encoders (date as an i32 day count, time as i64 microseconds since midnight, timestamp / timestamptz as i64 microseconds since 2000). A timestamp column’s hidden sub-millisecond companion is folded back before encoding, so .ffffff fractional seconds survive the round-trip. Measured against PostgreSQL 16 as the oracle, every common scalar and array type (int2/int4/int8, float4/float8, bool, text/varchar, numeric, date, time, timestamp, timestamptz, bytea, and int4[] / float8[] / text[]) now emits bytes identical to the real server’s.

Fixed

• crates/secantus-pgserver: COPY ... TO STDOUT (FORMAT BINARY) now routes every field through encode_binary (the shared SELECT-binary codec) instead of a text fallback, fixing the wire form for int2, float4, numeric, date, time, timestamp, timestamptz, bytea, and array columns. The psycopg test_read_rows[*-1] binary cases (a float8[] COPY read) now pass.

• crates/secantus-pgserver: a timestamp / timestamptz column’s __us_ sub-millisecond companion is reassembled into the value COPY reads, so microsecond-precision timestamps no longer lose their last three digits.

Added

• crates/secantus-pgplan: date_to_pg_days, time_to_pg_micros, and timestamp_bson_to_pg_micros — the inverses of the existing render_date_from_pg_days / render_time_from_micros / render_timestamp_from_pg_micros decoders, used by the binary COPY encoder.

The Rust PostgreSQL server closes three COPY gaps

secantusd-pg (the Rust PostgreSQL server) gained three fixes that psycopg’s own test_copy.py suite was catching, taking that gauge from 54 failures to 24 against a live PostgreSQL 14 oracle.

A serial column now stores integers. serial is a PostgreSQL pseudo-type that resolves to int4; the catalog was keeping it typed serial, so a COPY field parsed as text and 40010 came back as the string "40010" — the shape the whole test_copy_in_* cluster hit. smallserial/serial2, serial/serial4, and bigserial/serial8 now normalise to int2/int4/ int8 at CREATE TABLE time. (The implicit sequence default a real serial carries is still a separate, unimplemented feature.)

COPY (VALUES ...) TO STDOUT reads every row. A bare multi-row VALUES list was planned through the single-row constant path and produced no rows at all, so copy (values (1),(2)) to stdout returned an empty body; it is now planned as a ValuesConstant that both COPY and a directly-executed VALUES query read.

Text COPY stopped corrupting control characters. The unescaper handled only \t \n \r \\, so PostgreSQL’s \b/\f/\v escapes were read back as literal b/f/v — and a second, redundant unescape pass also halved an escaped backslash twice, dropping it from a value like \end. The path now decodes the full \b \f \n \r \t \v \\ set plus octal (\ooo) and hex (\xHH) byte escapes, over bytes, once. Silent data loss for any text carrying those bytes.

Fixed

• secantus-pgplan: serial/bigserial/smallserial (and the serialN aliases) normalise to their integer type at CREATE TABLE time.

• secantus-pgplan: a bare VALUES (...), (...) query plans as a new ValuesConstant statement instead of an empty single-row constant.

• secantus-pgserver: text COPY unescaping is complete (\b \f \v, octal, hex) and no longer runs twice, ending control-character and backslash corruption on COPY FROM; the text COPY-TO encoder escapes \b \f \v to match.

Wrong-typed createIndexes and collMod options

The second half of the Python server’s wrong-type sweep, measured the same way against mongod 8.2.11: 97 code divergences down to 37, with every createIndexes.* and collMod.* case now matching mongod byte-for-byte.

Fixed

• collMod silently accepted a wrong-typed validator, changeStreamPreAndPostImages or viewOn — a viewOn: 5 reached the catalog. All three now answer 14 TypeMismatch under mongod’s IDL path, and all three still accept an explicit null.

• Wrong-typed per-index options on createIndexes were ignored — collation, partialFilterExpression, expireAfterSeconds, unique and sparse. These do not use the plain BSON field '<path>' form: mongod echoes the offending index spec and appends the reason after :: caused by ::, in three distinct shapes (14 for the object slots, 67 for the TTL one, 14 with a different wording for the bool ones). Each is now reproduced verbatim — including the two unbalanced quotes that are mongod’s own, and the fact that unique / sparse accept a double.

• wildcardProjection reported one error where mongod reports three. All three arms answered 67 CannotCreateIndex; mongod answers 14 for a wrong type, 9 for an empty object (with its own wording, “can’t be an empty object”), and 2 for a non-wildcard base index. The spec in those messages was also rendered with Python’s repr ({'a': 1}) instead of mongod’s shell syntax, and omitted the index name.

Added

• secantus.bsontypes.render_bson, which renders a value the way mongod echoes it inside an error message — inner spaces on non-empty documents and arrays but not empty ones, unescaped strings, new Date(<ms>), BinData(0, ABCD), UUID("...") for subtype 4, /pattern/flags, Timestamp(t, i), MinKey / MaxKey. Every rule probed, not inferred.

A slow npm registry failed CI where the test harness says it should skip

Every _ensure_* helper in the cross-driver smokes is documented as returning False when its toolchain cannot be provisioned — offline, no registry, a cold cache — so that the smoke needing it skips. They only ever checked the exit code, while _run passes timeout straight to subprocess.run, which raises TimeoutExpired. So a registry that merely responded slowly failed the build, which is not what any of those helpers claim to do.

Four CI shards went red on npm install exceeding its 300s budget, across two test files and both Linux and macOS, on a commit that touched nothing near them. A rerun cleared three and not the fourth — it is not a coin-flip flake, the macOS runner is simply slower than the budget.

Provisioning subprocesses now go through _run_provision, which reports a timeout the same way it reports a non-zero exit: no usable toolchain right now. This is deliberately not folded into _run — inside a test body a timeout is a real failure and stays one. All eight call sites are inside _ensure_* helpers; no test body changed.

Fixed

• tests/test_cross_driver_features.py, tests/test_geo_cross_driver.py: _run_provision returns None on timeout, and every provisioning helper treats that as “toolchain unavailable” and skips.

Malformed cursor arguments crashed the server instead of being rejected

Phase 2 of tasks/remaining-work-plan.md, second surface: 51 cursor / getMore / killCursors shapes run against a live mongod 6.0.16. 22 diverged, four of them crash-class — a malformed argument reached a bare int() and the ValueError / TypeError escaped as internal server error (code 1), which is the shape that tells a client nothing and looks like a server fault rather than their own bad request.

Fixed

• Four crashes. getMore with a string cursor id or a string batchSize, and killCursors with a non-array cursors or a wrong-typed element, all answered code 1. They now answer mongod’s TypeMismatch (14) naming the field and both types.

• getMore answered CursorNotFound (43) for parse errors — for a cursor that existed. A missing collection (mongod: 40414), a non-string one (14), an unknown top-level field (40415) and an int32 cursor id (14; mongod requires a long, the same int64 strictness the Go and C drivers enforce on the reply side) all reported a plausible-looking lie about the cursor instead of the parse error mongod reports before it looks a cursor up.

• Negative batchSize / limit / skip were silently accepted on find, getMore and aggregate’s cursor spec. A negative batchSize fell through or DEFAULT and quietly became the default; a negative limit returned the whole collection. mongod answers Location51024, and — unlike the type error on the same slot — names the field bare rather than by its IDL path.

• maxTimeMS on a getMore for a non-awaitData cursor was accepted and ignored. It is the awaitData wait budget, so mongod refuses it on a cursor that cannot wait, tailable-but-not-awaitData included (probed both ways). Accepting it hides a client bug: a caller who believes it has bounded a blocking read has bounded nothing.

• aggregate ran without a cursor option, which mongod requires except with explain. A client that forgot the option never learned it had. Unknown keys inside the cursor spec are now 40415 too.

• killCursors with no cursors field returned a cheerful all-empty success reply; mongod requires the field (40414). A null element is skipped, and a null cursors takes mongod’s older Location10065 — both probed rather than assumed.

• awaitData without tailable was accepted and ran an ordinary find, so a client that asked to block got a plain batch back with no indication its option had been dropped.

The accepted-type matrix is unchanged and was re-verified against mongod while adding the range check: int, long, double, decimal and null are all taken, and a fractional double truncates toward zero (batchSize: 2.5 yields two documents).

Changed

• tests/test_crud.py::test_tailable_drop_closes_pymongo_cursor_cleanly no longer calls max_await_time_ms on a plain TAILABLE cursor. It passed only because we accepted maxTimeMS there; the same code against a real mongod raises. pymongo sends the option despite documenting it as ignored for non-await cursors, because its guard is a bitmask test (_query_flags & CursorType.TAILABLE_AWAIT is 2 & 6 == 2, truthy, for a plain TAILABLE cursor).

tests/test_mongod_differential.py grew 29 cases (93 → 122).

Cursor round-trip counts now match mongod

A cursor whose result count was an exact multiple of batchSize closed one getMore early. mongod closes a cursor only when it knows the result is finished — a batch that exactly fills the requested size proves nothing about what follows, so mongod keeps the cursor open and the client spends one more getMore to see an empty batch. SecantusDB buffers the whole result and so could close early: fewer round trips, but a count drivers observe directly.

The rule was probed against a real mongod rather than inferred, and it is not uniform across commands. find and aggregate keep the cursor open on an exact-fill batch; listIndexes and listCollections close, because they enumerate a catalog whose size is known up front. A find with limit or singleBatch is bounded again and closes without the extra trip, and a getMore with no batchSize means “server default”, where mongod drains and closes.

Fixed

• find and aggregate cursors stay open past a batch that exactly drains the result, matching mongod’s round-trip count. Verified across 12 shapes against mongod 8.3.4, on both the Python and Rust servers.

• limit / singleBatch still close immediately, with no trailing empty batch.

• listIndexes / listCollections still close on an exact-fill batch.

• A getMore with a non-positive batchSize drains the cursor and closes it.

The Rust PostgreSQL server exposes pg_cursors and constant select-list columns

psycopg’s server cursor (ServerCursor / RawServerCursor) does two things the Rust secantusd-pg server could not answer. It reads the pg_cursors catalog — directly, to confirm a cursor is gone after CLOSE, and internally as SELECT 1 FROM pg_catalog.pg_cursors WHERE name = ... before closing a cursor it did not declare — and every server-cursor test seeds its rows with a bare literal, SELECT 1 FROM generate_series(...). The first raised relation "pg_cursors" does not exist; the second raised AConst is not supported yet, because the select-list planner accepted a column, a cast, or a scalar call but not a plain constant.

pg_cursors is now a virtual catalog table over the connection’s open cursors, with PostgreSQL’s columns (name, statement, is_holdable, is_binary, is_scrollable, creation_time); a CLOSE removes the row, and a cursor declared NO SCROLL / WITH HOLD / BINARY reports it faithfully. A literal in the select list — SELECT 1 FROM t, over a table or a generate_series — is now a constant column named ?column? (or its alias), one value per row, which is how a client counts a source’s rows through cursor.rowcount without reading their values.

Measured against psycopg 3’s own cursor suite (test_cursor*.py), the change turns 38 previously-failing cases green, among them test_context, test_close, test_stolen_cursor_close, test_init_params, test_rownumber, and test_scrollable.

Added

• secantus-pgserver: a pg_cursors virtual catalog table backed by the connection’s open-cursor map; CursorState now carries the catalog columns, captured at DECLARE from the cursor’s options.

• secantus-pgplan: a ColumnExpr::Const select-list column, planned from an AConst target over a table or a generate_series source.

Date operators honour their timezone

The backlog carried this as one Rust-only gap in $dateFromString that “genuinely needs a timezone database — a dependency decision, not an afternoon’s port”. Measuring it found all three parts wrong: chrono-tz is already a dependency with the IANA database bundled (which is why $hour with a named zone already worked), it is $dateFromParts too, and the sweep turned up three silent wrong answers on BOTH servers that the entry never mentioned.

New standing probe tools/probes/date_timezones.py — 409 shapes across the whole date family, every zone kind, and both DST directions.

	Before	After
Python server	142 divergent	0
Rust server	191 divergent	0

Fixed — silent wrong answers

• $dateTrunc ignored timezone entirely, bucketing on UTC boundaries. A daily rollup for America/New_York bucketed at 00:00Z instead of 04:00Z, quietly attributing four hours of every day to the wrong bucket. It also binned from year 1 rather than mongod’s 2000-01-01 reference, and defaulted weeks to Monday where mongod uses Sunday.

• $dateDiff ignored it too, and computed “whole units elapsed” where mongod counts boundary crossings: 02:00Z→23:00Z is 1 day in New York and 0 in UTC. Both operators now share one bin-index function so they cannot drift apart.

• $dateAdd / $dateSubtract ignored it as well. A calendar shift moves the local wall clock: noon Eastern plus one day is noon Eastern — 23 real hours across a spring-forward — while +24 hour is 24. Every calendar shift across a DST boundary was an hour out (30 minutes for Australia/Lord_Howe).

Two arithmetics, both mongod’s, probed across the 2026-03-08 spring-forward: calendar units land on a local wall-clock boundary; sub-day units bin by real elapsed time, so binSize: 5 hours stays 5 real hours apart rather than re-aligning.

Fixed — Rust capability

• $dateFromString and $dateFromParts accept named IANA zones. Only the instant→wall-clock direction had been wired up; these two need the reverse, which is now tz_instant_from_local_ms.

• An unusable zone answers mongod’s 40485 instead of deferring — which on the standalone server reported the operator as unsupported for what is a bad argument.

Fixed — error surface

• Zone names are case-sensitive. zoneinfo resolves through the filesystem, so America/new_york loaded on macOS and failed on Linux — the answer depended on the host.

• A literal timezone is validated at pipeline optimization time, which is where mongod reports it, and $dateTrunc / $dateDiff name the parameter in the message as mongod does.

Correction: mongod’s decimal transcendentals are largely reproducible after all

An earlier entry concluded from one measurement that mongod is “not correctly rounded, so no independent implementation can match it”. That was too strong, and probing further found the actual rule:

• $sqrt is plain correctly-rounded decimal128 — IEEE 754 requires it for square root — and matches on 6 of 6 inputs. Exactly reproducible today, with no dependency.

• $exp / $ln go through binary128: rounding the true value to a 113-bit significand and converting back to 34 decimal digits reproduces mongod where correct decimal rounding does not.

• It is not one uniform rule — across 24 covered pairs the binary128 route matched 20 and missed 4, and the trig/hyperbolic family was not covered.

So there is a real pure-Rust path that needs no C dependency, though it needs a high-precision core and per-operator verification. The backlog entry now records the rule, the counter-examples, and the exact next step, instead of a blocker that was not one.

Decimal128 transcendentals, measured over 96 pairs instead of guessed

This backlog entry was rewritten three times in one day, and the first two rewrites were wrong because each inferred a mechanism from four or five data points. A 96-pair sweep (12 operators × 8 inputs) settles it:

rule	pairs
correctly-rounded decimal128	76
matches a binary128 round-trip instead	7
neither — mongod 1–2 ULP off the correctly-rounded value	13

$sqrt and $asinh are correctly-rounded decimal on 8 of 8 — IEEE 754 requires it for square root — so those are exactly reproducible today with no dependency. Everything else is mixed.

The strays are real, not harness error; the reference series is stable at 60, 130 and 190 digits (sin(2.5) is 1 ULP low, tan(2.5) 2 ULP, cos(2.5) exact).

So implementing correctly-rounded decimal transcendentals in pure Rust is a legitimate option — ~79% exact, $sqrt/$asinh fully exact — rather than the trap an earlier write-up called it. The alternatives (link Intel RDFP; keep refusing) are unchanged.

The sweep is kept as tools/probes/decimal_transcendental_rule.py so the next session measures rather than re-theorises.

Decimal128 transcendentals: one of the three options is now ruled out

The previous write-up costed three options for the finite-operand transcendentals and flagged “add a pure-Rust decimal crate” as the risky one. A further measurement rules it out entirely.

Computing at 60 digits and rounding correctly to 34 matches mongod for $sqrt and $exp — and does not for $ln or $log10:

ln(2.5)    = 0.9162907318741550651835272117680110|714501…
             correctly rounded → …680111      mongod → …680110   (below)

log10(2.5) = 0.3979400086720376095725222105510139|464636…
             correctly rounded → …510139      mongod → …510140   (above)

mongod lands below the true value for one and above it for the other, so it is neither correctly rounded nor consistently truncated — it carries Intel RDFP’s own approximation error. Matching it means reproducing that error, which a more accurate library cannot do at any precision.

So the choice is now binary: link Intel RDFP (bit-identical, at the cost of a C dependency across five wheel platforms), or keep refusing (today’s behaviour). The backlog entry records the evidence.

The last two Rust-server backlog items, measured and decided

Neither is a coding task. Both entries asserted a blocker; probing replaced the assertions with data, and in both cases the data changed the conclusion.

Decimal128 transcendentals — a decision, not a dependency gap. The entry said this “needs a 34-digit decimal library” and suggested tabling the zero column as a cheap partial win. Measured against mongod 8.2.11:

• The zero column is already done — 0 divergent of 18, as are all six error cases. There was nothing to win.

• The library is not the blocker. The Python server already has one (stdlib decimal at full precision) and still differs from mongod by exactly one unit in the last place on $ln, $log10 and $sin, while matching on six others. mongod uses Intel’s RDFP, whose transcendental rounding differs from every other implementation in the final digit.

So the real choice is: link Intel RDFP (exact, but a C dependency across five wheel platforms), add a pure-Rust decimal crate (cheap, but silently wrong in the last digit), or keep refusing. The middle option is the one to resist without an explicit decision — it trades a visible refusal for an invisible wrong answer.

$toDate of an unparseable string — won’t fix. mongod’s message is timelib’s re2c scanner talking. Across 25 measured strings there is no small rule: 'a1' errors at position 1 while '1a' errors at position 0, '123' emits one error per position, 'junk here' emits two with a “Double timezone specification”, and '2020-01-01X' parses because X is the military timezone UTC+11. Reproducing it means reproducing the lexer, its timezone abbreviation tables and its per-position error accumulation; a partial job would emit messages mongod never sends.

Both entries are rewritten with the measurements so the next session inherits evidence instead of an assertion.

The Rust server answers decimal trigonometry instead of refusing it

Ask the Rust server for {$sin: <Decimal128>} — or $cos, $tan, $asin, $acos, $atan, $sinh, $cosh, $tanh, $acosh — and it used to reply “a construct the Rust server does not support”. MongoDB returns a number. That is the least faithful outcome available, and it covered 108 of 285 measured shapes. All ten now answer, sharing the Python engine’s results.

Getting there settled a question that had been treated as a matter of taste. Compared against a 60-digit reference, MongoDB is correctly-rounded only about 78% of the time for these operators — it carries Intel’s decimal-library error in the last digit. Exact agreement is therefore capped, and no choice of working precision reaches it. What follows is the practical part: being correct is the closest we can get to MongoDB, because wherever we round correctly our agreement equals MongoDB’s own accuracy.

That condemned an older strategy in the Python engine, which computed the hyperbolics at 34 digits throughout in order to reproduce MongoDB’s accumulation. It had been adopted on the strength of one $cosh case:

	agreement at 34 digits	computed wide
$tanh	3/20	12/20
$sinh	8/20	12/20
$acosh	12/15	14/15
$cosh	16/20	16/20

$cosh did not even lose, so the case behind the strategy did not support it.

Fixed

• The Rust server implements the whole decimal trig and hyperbolic family. sinh / cosh / tanh / acosh / atanh from the existing high-precision exp / ln / sqrt; atan / asin / acos from an argument-reduced Taylor series; sin / cos / tan by reduction modulo an embedded 2π. Zero refusals, and identical to the Python server throughout.

• The Python hyperbolics compute with guard digits and round once. $tanh, $sinh and $acosh were losing accuracy — and agreement — to compounded 34-digit rounding.

Agreement with mongod 8.2.11 rose from 209 to 224 of 285 on the Python server and from 90 to 224 on the Rust server. The remaining 61 are MongoDB’s own last-digit error.

Added

• tools/probes/decimal_transcendental_rounding.py — the only probe here that asks “is the answer right?” alongside “does it match MongoDB?”, against an mpmath reference. That pairing is what distinguishes our bug from MongoDB’s error, and it fails the run only on the former.

Known gap

sin / cos / tan on the Rust server reduce against 2π embedded to ~1200 digits, so an argument beyond about 1e1100 still defers. MongoDB carries π to the full decimal128 range.

$ln, $log10, $exp and $asinh compute on a finite Decimal128

These four used to refuse a finite non-zero decimal outright. On the Rust server that is a BadValue — a deferral has no Python behind it there — so a collection holding Decimal128 values could not take a logarithm at all. They now compute at decimal128’s 34 significant digits, on both servers, from a new arbitrary-precision layer in secantus-core.

Added

• A high-precision decimal core (hp_mul / hp_add / hp_div / hp_sqrt) that does the same arithmetic as the 34-digit operators at a caller-chosen width. The existing add / mul round to 34 digits at every step, which is right for arithmetic and useless for a series: an argument reduction can cancel thirty digits away and leave nothing behind.

• ln by argument reduction (x = m·10^k, m = r·2^j) onto an atanh series; exp by reduction onto a power of ten and a Taylor series; log10 from ln, except for an exact power of ten, which answers the integer with no series at all — the only way $log10 of 1E+400 comes out as 400.

• Rounding is verified rather than assumed: the working precision widens (80 → 140 → 260 digits) until the guard digits actually decide the 34-digit answer, instead of trusting a fixed guard.

Fixed

• $asinh of a small argument lost most of its digits on both engines. asinh(x) = ln(x + sqrt(x²+1)) cancels to 1 + x for small x, so at any fixed precision the answer collapses: the Rust server returned 0 for Decimal128("1E-100") and the Python engine had eleven digits of error at 1E-10. Below 1E-18 the correction term falls past all 34 digits and the answer is the argument itself; above it, Python now widens its working precision with the exponent.

Changed

• These four are correctly rounded, which means they differ from mongod on about a fifth of finite inputs. Over 290 measured pairs mongod is correctly rounded on 231; it carries Intel RDFP’s approximation error in the last digit on the rest, and matching that would mean linking RDFP. $ln of Decimal128("2.5") is now …117680111 where 8.2.11 answers …117680110 — the true value is …1176801107145…. Measured, deliberate, and recorded in tasks/backlog.md.

$avg gave a wrong number, and a decimal zero got the wrong quantum

Two families that both looked like “needs 34-digit decimal math” and were not.

$avg was a wrong answer on the Python server. mongod converts the integer total to a double and then divides — it does not do exact integer division. The two agree until the total passes 2⁵³, and then they do not:

$avg: [2**53+1, 2**53+3, 2**53+5]
    mongod  9007199254740994.0     (float(sum) / n)
    before  9007199254740996.0     (sum / n, correctly rounded)

Python’s int / int is correctly rounded over the exact quotient — a better answer, and the wrong one, because the conformance target is mongod’s arithmetic rather than the most accurate arithmetic. The Rust engine deferred above 2⁵³ with a comment reading “defer to Python int/int divide”, so it was deferring to that wrong answer — a comment justifying behaviour by the other engine instead of by the oracle.

A decimal zero answers a constant, and the quantum is load-bearing. No series has to run, which is exactly what separates these from the finite decimals in the same operators. Both the per-operator quantum and the sign rule are unguessable:

	mongod	Rust before	Python before
$tan(0)	0E-40	deferred	0
$asinh(0)	0E-6176	deferred	0
$cos(0)	1.000000000000000000000000000000000	deferred	1
$sin(-0)	-0	deferred	0 — sign lost
$degreesToRadians(0)	0E-35	deferred	0E-50

The odd functions carry -0 through and the even ones drop it. Every cell was generated from mongod 8.2.11 rather than derived.

Triaging the family by measurement rather than by its label is what found this: of the 35 shapes where mongod answered and the Rust engine deferred, 19 genuinely needed decimal transcendentals and 16 did not. Rust refusals in this corpus go 35 → 19, and the remaining 19 are now exclusively the finite transcendentals — the one genuine dependency question.

Fixed

• secantus.expressions: $avg divides the way mongod does; the trig, hyperbolic and angle-conversion operators answer a decimal zero from a measured table instead of running their series.

• secantus-core: the same table, plus $sqrt and $exp at zero; the $avg precision guard is gone.

Decimal128 through the math operators: 13 crashes, and half the digits

Decimal128 is one of the types this project keeps documents as opaque BSON to preserve. The aggregation math operators were converting it to float, or failing outright: 13 of them crashed the Python server with internal server error, because math rejects a Decimal128 and the TypeError escaped. The rest silently narrowed 34 significant digits to 17.

Measured against mongod 8.2.11: 21 of 49 shapes correct before, 38 after.

Fixed

• The 13 crashes are gone. $abs, $ceil, $floor, $trunc, $round, $exp, $ln, $log10, $sqrt, $mod, $pow, $log and $avg now compute in decimal at decimal128’s 34-digit precision and return a Decimal128.

• The hyperbolics keep full precision — $sinh, $cosh, $tanh, $asinh, $acosh, $atanh are exact identities over exp / ln / sqrt, which decimal provides.

• $degreesToRadians / $radiansToDegrees use a decimal π instead of the float one.

• $mod uses Decimal’s truncate-toward-zero, which is mongod’s rule; Python’s % floors, so widening the existing expression would have been wrong for negative operands.

• $pow computes exp(e * ln(b)), not b ** e. That looks like the worse choice and is the right one: 2.5 ** 2 is exactly 6.25, and mongod answers 6.249999999999999999999999999999999.

A finding worth keeping: do not be more accurate than the reference

Computing the hyperbolic identities with guard digits — wide, then rounded back to 34 — is more accurate and matched mongod less, moving $cosh from agreeing to differing in the final digit. mongod accumulates its own rounding at decimal128 precision throughout, so fidelity means reproducing that arithmetic rather than improving on it. The extra precision was reverted.

Still open (recorded)

The circular functions — $sin, $cos, $tan, $asin, $acos, $atan, $atan2 — have no identity over the operations decimal provides and would need series expansions; they still narrow to float. $acosh agrees to 33 of 34 digits. And cmp(string, Decimal128) is still inverted in the cross-type order.

27 cases added to tests/test_mongod_differential.py.

Decimal128 NaN and infinity now answer in the math operators

$sqrt, $exp, $ln, $log10, $degreesToRadians and $radiansToDegrees refused a Decimal128 NaN or ±Infinity on the Rust server — 2 BadValue: aggregation pipeline uses a stage or operator not supported by the Rust server — where mongod answers. The Python server answered, but wrongly in five places.

These values carry no precision, so they need none of the 34-digit decimal math a finite decimal would; that is what makes them separable from the rest of the family, which still defers on the Rust server.

Four of the rules defeat a guess, and all were measured against 8.2.11:

• $ceil / $floor of a Decimal infinity are NaN, not the infinity;

• $ln / $log10 of a Decimal NaN come back as a double nan — the one place in this family where the argument’s type is not kept;

• $cosh(-Infinity) is +Infinity;

• $abs(-0) is 0 while $trunc(-0) is -0.

The Python engine’s bug was one shape repeated: its domain guards tested isinstance(v, (int, float)), so a Decimal128 slipped past them into the decimal path and came back NaN — $sqrt(Decimal128("-Infinity")) returned NaN where mongod raises 28714, and $ln / $log10 of a Decimal -Infinity returned NaN where mongod raises 28766 / 28761.

The parity suite was green throughout. Neither engine’s corpus contained these shapes, so the two engines were pinned to each other while one deferred and the other answered wrongly. Only comparing against a real mongod separated them — the “parity is not correctness” case, again.

Fixed

• secantus.expressions: $sqrt / $ln / $log10 apply their domain checks to a Decimal128 as mongod does, instead of letting it reach the decimal path and return NaN.

• secantus-core: the same six operators answer a special Decimal128 rather than deferring, keeping the argument’s type.

$densify no longer crashes on a null or missing field

Running $densify over a collection where any document’s target field was null or absent returned an internal server error. The stage sorted documents by that field without checking it existed, so Python’s own comparison raised — '<' not supported between instances of 'NoneType' and 'int' — and the failure escaped as a generic “internal server error” rather than anything actionable. A single document with a missing field was enough to take down the whole aggregation.

MongoDB simply doesn’t densify those documents: they pass through unchanged, in their normal sort position ahead of the numbers, and the remaining values densify as usual. A field holding something that is neither a number nor a date is rejected outright with a specific error rather than a crash.

Both servers now behave that way. The Rust server had been declining the stage entirely whenever such a document was present — its code carried a comment explaining that it deferred because the Python side raised — so it inherited the bug as an error rather than a crash. Both are fixed together.

Fixed

• $densify emits a document whose target field is null or missing unchanged instead of failing the aggregation, and densifies only the documents that have a usable value.

• A non-numeric, non-date value in the densify field is rejected with MongoDB’s 5733201 (Densify field type must be numeric or a date) instead of an internal error.

Long runs survive the shell that launched them

A multi-minute test or gauge run started in a background shell died three times in one session — twice at about ninety per cent, once at a third of the way through — with no failing test, no out-of-memory kill and nothing in the log but the word killed. The cause was not the run: a supervisor reaping the shell it had spawned signalled the entire process group, and the work shared that group.

scripts/detached_run.py gives the work its own session and process group, so reaping the launcher leaves it running. Start a command under a name, then poll it, wait on it, or stop it; the pid, the command line, the log and the eventual exit code live in .detached-runs/<name>.json beside <name>.log, which is what makes the exit code readable at all once the launching process is gone.

The exit code is recorded by a small Python supervisor rather than a sh -c wrapper, and that detail is load-bearing on macOS: with a shell in the middle of the process tree, every child’s connection to a Postgres.app server timed out, so the 825 PostgreSQL differential tests reported themselves as skipped behind an otherwise green run. A test pins the absence of the shell.

Added

• scripts/detached_run.py with start / status / wait / stop subcommands for long-running commands that must outlive their launcher.

• tests/test_detached_run.py, which pins the property that matters by signalling the launcher’s own process group and asserting the child is untouched, plus exit-code capture and the refusal to reuse a live name.

The gate that compares us against a real mongod now runs in CI

tests/test_mongod_differential.py runs every supported operation against a real mongod and asserts an exact match. It gates on shutil.which("mongod") — and the CI lanes installed mongosh and the Database Tools but no server, so it skipped everywhere except whichever dev box happened to run it.

That is a load-bearing gap rather than a cosmetic one: for any change to the error surface, a green CI was no evidence at all. The 908-divergence operand campaign that landed immediately before this rewrote error text across ~40 operators, and nothing in CI could have judged a single one of them.

The Linux lanes now install mongodb-org-server. 614 tests that were skipping now run.

Changed

• test, test-durable and record-durations install mongodb-org-server, and their apt repo moves from 8.0 to 8.2.

8.2 because that is the series every expectation in the file was measured from (8.2.1 / 8.2.11). The gate compares only the MAJOR, so 8.0 would have run — and this file has already been bitten by a patch-level difference, an expected-type list whose order changed between 8.2.1 and 8.2.11 and needed _sort_type_lists. Pointing CI at a series the expectations were never taken from would manufacture exactly the false failures the file’s whole-file skip exists to avoid.

No separate step was needed: the suite’s addopts excludes perf / online / slow but not differential, and the file’s mongod fixture is module-scoped, so it spawns one server for the whole file.

Two things worth knowing, both found by checking rather than assuming

• There is no server-8.2.asc key. It 404s at both mongodb.org and pgp.mongodb.com. MongoDB signs both series with one packaging key — verified by fingerprint, not by hope: 8.2’s InRelease is signed by 4B0752C1BCA238C0B4EE14DC41DE058A4E7DCA05, which is exactly what server-8.0.asc serves. The keyring file is now named mongodb-server.gpg rather than -8.0, so the two do not have to agree; a plausible-looking “fix” to a versioned 8.2 URL will 404 the job.

• The patch CI installs is verified, not assumed. The repo tracks the newest patch (8.2.12 today) while the expectations came from 8.2.11. All 614 cases were run against both before landing this, so the version CI actually gets is known-clean. If patch churn ever becomes noise rather than signal, the answer is to pin mongodb-org-server=8.2.11 — not to loosen the gate.

Not changed

macOS and Windows install no server and still skip. macOS runs only on the cron cell, and the Homebrew route for a server is the messier half of an already-awkward tap; Windows has no packaging story worth the flake. The value is in the PR lane, and the PR lane is Linux.

The mongod comparison suite runs on any 8.x server, not just one

The suite that compares SecantusDB against a real MongoDB server, operation by operation, was pinned to the exact release its expectations were taken from. That was deliberate caution when the expectations came from a server two majors behind, but it meant a developer running any other build got the whole suite skipped and no signal at all. MongoDB’s error surface is stable within a major version, so the suite now runs against any 8.x server and only skips across a major boundary.

The practical effect is that a difference on a nearby release now shows up as a failure to investigate rather than a silent skip — which is the right default, since a mismatch there is far more likely to be a real divergence than version drift.

Changed

• The differential gate keys on the mongod major version rather than major-and-minor.

A double in an error message was rendered the wrong way — in both servers

mongod has two renderings for a double inside an error message, and they are not interchangeable. Measured one value at a time against 8.2.11:

value	value form	spec form
-0.0	-0	-0.0
-1.0	-1	-1.0
1234567.0	1.23457e+06	1234567.0
-2147483648.0	-2.14748e+09	-2147483648.0
0.000123456789	0.000123457	0.000123456789

The value form (Value::toString) is C’s %g at precision 6 and is what $mergeObjects, $replaceRoot, $ln, $log and $log10 echo. The spec form is %.16g with a .0 appended when that leaves no . or e, and is what a stage’s echoed specification uses — $firstN/$lastN/$maxN/$median’s “specification must be an object” and $graphLookup’s missing-from message.

The spec form is not the shortest round-trip form, though the two agree for every ordinary value, which is why it was written that way first and why the unit tests — written from the same assumption — passed. They part company at the bottom of the range, where the shortest string that round-trips is shorter than sixteen significant digits: 1e-308 echoes as 9.999999999999999e-309 and 5e-324 as 4.940656458412465e-324. Only the differential gate against a real mongod caught that, which is the argument for putting a finding there rather than in a test that drives our own servers.

A single renderer was serving both, so every value message rendered a double the spec way. Switching that renderer is not enough on its own: it fixes $mergeObjects and $replaceRoot and silently breaks $graphLookup, which is a spec echo that happened to share the function. $graphLookup now takes the spec renderer explicitly.

The Rust server was wrong in a louder way. Its value renderer cast an integral double to i64, which lost the sign of -0.0 (printing 0) and saturated past i64::MAX — so 1e308 came back as 9223372036854775807, a flatly wrong number shown to the user. Its spec renderer used {:.1}, which expanded 1e308 into its full 309-digit decimal value; Rust has no %g at all, so both precisions are now reproduced by one routine.

Across the expression corpus this takes the Rust server’s message-only divergences from 13 to 1 and the Python server’s from 173 to 9.

Note $ln renders its operand as a double whatever its BSON type: an Int32 -2147483648 comes back as -2.14748e+09, not as itself.

tests/test_mongod_differential.py gains 52 cases covering both vocabularies, so a future simplification that collapses the two renderers fails on one side or the other. tools/probes/double_error_rendering.py is the three-server harness.

Fixed

• secantus.bsontypes: new fmt_double_value (C’s %g); fmt_double_parse becomes %.16g rather than repr, which fixes the denormal end of the range; bson_value_repr_stage renders a double the value way.

• secantus.aggregate: $graphLookup’s missing-from echo uses the spec renderer, which also removes the manual brace re-spacing it needed.

• secantus.expressions: $ln / $log / $log10 domain messages render the operand as a double in the value form.

• secantus-core: new format_double_g / format_double_spec; render_value_compact and the three log-family messages use the first, and the command layer’s render_stage_value uses the second.

$elemMatch traversed an array twice

mongod applies implicit array traversal once per path step, and $elemMatch spends that step choosing the element — so inside it the element is a terminal value and nothing descends into it again. Both servers got that wrong, in opposite directions.

The operator form matched through an element that was itself an array, so {$elemMatch: {$gt: 1}} returned documents holding [[5]] and [1, [2, [3]]]; mongod returns neither, because an array is not greater than 1. The criteria form had the mirror-image gap: it considered only document elements, so {$elemMatch: {}} — a criteria that imposes no field requirement — missed every document whose array holds an array element, which mongod returns.

The fix is a descend flag threaded through the operator dispatch, off for the element match, so one rule covers every operator rather than each one growing its own special case. Three operators needed reaching individually because they recurse or iterate the array themselves, and each was a separate miss in the first version of the fix — $in has its own candidate path, $not re-enters the field matcher, and $all walks the array. Two were caught by writing the test and one by probing before believing the fix was complete.

Two neighbouring divergences from the same 8.2.11 sweep are not fixed here and are filed with their measured rules, because they live in path resolution rather than in matching: a dotted POSITIONAL path (x.0) descends one level too far, and a dotted sort key does not descend at all.

Fixed

• secantus.query / secantus-core: $elemMatch’s operator form treats the element as terminal, so it no longer matches through a nested array. Applies to every operator it can carry — comparison, $in / $nin, $type, $regex, the $bits* family — via one flag rather than per-operator logic.

• secantus.query / secantus-core: $elemMatch’s criteria form reaches array elements when the criteria names no field, so {$elemMatch: {}} returns what mongod returns. A non-empty criteria still requires a document element.

• secantus.query / secantus-core: $not, $in / $nin and $all carry the flag through their own recursion, so {$elemMatch: {$not: {$gt: 1}}} no longer inverts a descended match and {$elemMatch: {$all: [5]}} no longer matches a nested array. $size and a nested $elemMatch were already right and are pinned so they stay that way.

Expanded change events were missing the collection UUID

With showExpandedEvents, MongoDB puts collectionUUID on every event that has a collection. SecantusDB set it only on inserts, updates and deletes, so a consumer watching DDL — create, createIndexes, dropIndexes, collMod, drop, rename — could not tell which collection a UUID-keyed event belonged to without a second lookup.

invalidate is the deliberate exception: it derives from an event that does carry a UUID, and MongoDB still omits it there. That exclusion is pinned by a test, because it is exactly the sort of asymmetry a later refactor would tidy away.

collMod events also gained stateBeforeChange — the collection’s options as they stood before the modification, which is what lets a consumer see what a collMod actually replaced rather than only what it set.

With these, the change-stream differential sweep is at zero divergences from MongoDB 8.2.11 across all 41 cases, on both servers. The one remaining difference is a field-order quirk in MongoDB’s own rename event, recorded and deliberately not copied.

explain reports the stage tree and the query MongoDB actually parsed

explain answered with a single flat node — COLLSCAN, or FETCH wrapping IXSCAN — and echoed your filter back verbatim as parsedQuery. MongoDB does neither. It wraps the scan in the stages that describe the rest of the query, and it reports the match expression after normalisation. The practical cost of the flat node was that a client running explain to ask “is my sort served by an index?” could not tell, because the blocking SORT stage that answers the question was never emitted.

Both halves now match. winningPlan carries SORT (with sortPattern, memLimit, and the limit it absorbed), SKIP, LIMIT and PROJECTION_SIMPLE / PROJECTION_DEFAULT in MongoDB’s nesting, which is not the order the command’s fields are written in. The IXSCAN node carries multiKeyPaths, isUnique, isSparse, isPartial and indexVersion; COLLSCAN carries direction; FETCH carries only the residual filter and omits the key entirely when the index bounds already cover the predicate, which is how you tell a fully index-served query from one that re-checks documents.

parsedQuery is now the normalised match expression: bare equality grows an explicit $eq, several clauses fold into an $and whose children are sorted, $ne becomes $not/$eq, $type becomes numeric BSON codes, $bitsAllSet becomes a bit-position list, and so on. The child order inside $and is the part with nothing documented behind it — it is MongoDB’s internal match-type ordinal, and it disagrees with the enum in MongoDB’s own source about where $not sits, so it was derived from ninety-one pairwise probes instead. All fifty-six filters in the sweep now agree.

What is left out is left out on purpose. indexBounds, rejectedPlans and the specialised IDHACK / COUNT_SCAN / DISTINCT_SCAN executors describe MongoDB’s cost model and its plan cache, neither of which this project has ever claimed to reproduce; and the stage tree is emitted for find only, because count and distinct use a vocabulary that has not been measured and inventing one would be worse than the flat node they get today.

Added

• secantus/explain.py: canonical_match and build_stage_tree, both pure functions over parsed input.

• tools/probes/explain_shapes.py: the sweep, including the pairwise derivation of the match-type ordering.

Fixed

• commands.py: explain emits the stage tree, the normalised query, the full IXSCAN metadata, COLLSCAN direction, isCached and explainVersion.

• storage.py: explain_plan reports sorted_by_index, which is what decides whether a blocking SORT appears.

The expression language’s comparisons and truthiness, measured operator by operator

The first sweep of the aggregation expression operators — tools/probes/agg_expressions.py, 143 operators against mongod 8.2.11. The argument, change-stream, update and findAndModify surfaces had all been swept; this one, the largest operator family in the server, never had.

It found four defects that produce silent wrong answers, all on both servers.

Fixed

• Every cross-type comparison answered false. $gt / $gte / $lt / $lte compared with the host language’s own operators and swallowed the error a cross-type pair raises:

  try:
      return bool(a > b)
  except TypeError:
      return False
  

  So {$gt: ["abc", 1]} was false where mongod says true — a string sorts after a number in BSON’s canonical order — and {$lt: [null, 1]} likewise. $cmp, two thousand lines away, had used the correct comparator all along. The four now share it. This reaches rows: the expression language drives $expr, $cond, $filter, $switch and $bucket.

• $and / $or iterated a non-array argument. {$and: "$s"} is a one-element list to mongod; iterating it directly walked the string character by character, so "$s" became '$' and 's' and the first parsed as an empty field path. Short-circuiting is preserved — mongod does short-circuit at runtime, so a false $and operand hides a later error.

• Truthiness. Only null, missing, false and zero are false. Every string is true, the empty one included ({$or: ""}, {$toBool: ""}), as are empty arrays and documents. Missing was reading as true.

• A bool is not a number. {$eq: [true, 1]} is false on mongod; the host language’s True == 1 made it true. The same root cause was collapsing $addToSet’s 0 and false into one element, making {$in: [false, [0]]} true, and — in the oplog update-diff — reporting a field that changed from true to 1 as no change at all, so a change stream never saw it.

Also fixed, uncovered by the above

• decimal.Decimal was not ranked as a number in the BSON order. The SQL layer’s numerics are native Decimals, so once the relational operators started using that order, price < cost * 1.5 compared by TYPE and returned rows where the comparison is false. Caught by the SQL suite.

• Booleans were excluded from the Rust engine’s sortable set, so every comparison involving one deferred — a generic BadValue on a server with no Python. type_rank had ranked them all along.

Notes

Two tests asserted the limitation rather than the behaviour (is_sortable(true) is false; $maxN over booleans errors) and were rewritten against mongod, which handles both.

The engine-parity fuzz caught two mistakes mid-change: a lost short-circuit, and the bool-equality rule landing in Rust before Python. One definition of BSON equality now lives in ordering.bson_equal and is used by both the expression language and the diff.

30 cases added to tests/test_mongod_differential.py. The sweep is not finished — its remaining findings are recorded in tasks/backlog.md.

Conversions, decimal $divide / $mod, and missing-parameter errors

Re-running the 6,628-case expression corpus against the Rust server found 28 shapes answering a different code from mongod. Extending the same cases to the pure Python engine found the same family plus three of its own. The grid of mongod × Rust server × Python engine over these 76 cases is now 0 divergent.

Fixed

• $dateFromString, $dateToParts, $dateTrunc and $dateDiff answered null for a missing required parameter where mongod raises. {$dateTrunc: {}} was null, not an error — a wrong value, not a wrong message. The codes share no pattern (40542 / 40522 / 5439009 / 5166303) so they are a measured table, not a rule.

• $getField named the wrong missing field. mongod checks field before input, the reverse of the order it reads them in, so {$getField: {}} is 3041702 and not 3041703.

• $divide by a decimal zero raised decimal.DivisionByZero out of the Python evaluator — an internal server error. b == 0 is False for Decimal128("0"), since the BSON wrapper defines no comparison against int, so a decimal zero divisor walked past the guard.

• $mod of Decimal128("1E+6144") by 7 answered NaN. The quotient needs 6,145 digits and Decimal.__mod__ raises InvalidOperation past the working precision. It is now computed at whatever width the quotient needs, and the Rust side does it as an exact integer remainder.

• $mod by zero used the wrong code with a decimal operand: 16610 is for int / double, and it is 5733415 once a decimal is on either side.

• $toDate of a fractional value did not truncate. A BSON date holds whole milliseconds, so {$toDate: 1.5} is 1ms; the Python engine built a datetime with 1500 microseconds, a value BSON cannot hold.

Added

• $divide and $mod accept Decimal128 on the Rust server, which refused them outright. $divide carries the decimal spec’s ideal exponent, so 100 / 10 is 10, 2.50 / 1.0 is 2.5 and 1 / 8 is 0.125; $mod takes its sign from the dividend and its quantum from min(e1, e2), so 7.5 % 2.5 is 0.0.

• binData conversions on both servers. $toInt / $toLong reinterpret the bytes as a little-endian integer (BinData(0, "01020304") is 67305985, not 16909060), $toDouble reinterprets 4 bytes as an IEEE single and 8 as a double, and $toString is base64. Each target accepts its own set of lengths and names the rest.

• $toDecimal of inf / -inf / nan, which was a 241.

Fixed (regression from this session)

• $asinh of a decimal below 1E-4966 returned the value where mongod underflows to a bare 0. mongod’s own implementation gives up there — and gives progressively wrong answers for a decade above it, $asinh(1E-4965) being 1.295…E-4965 where the true value is 1E-4965. This server had been changed to answer the mathematically correct value because the two engines were compared to each other rather than to mongod. The Rust server’s exemplar is mongod; the threshold is bisected from 8.2.11 and both servers now follow it.

A NaN no longer crashes $expr, and it now equals itself

find({"$expr": {"$gt": ["$v", 0]}}) over a collection holding a Decimal128("NaN") answered internal server error on the Python server. The expression language widens a Decimal128 to a Python Decimal before comparing, and Decimal("NaN") < 0 raises decimal.InvalidOperation — not the TypeError the comparison fallback was catching. One such document made an ordinary query fail.

The same probe found a second, quieter one: {$eq: [NaN, NaN]} is true on MongoDB, whose canonical order ranks the two as equal — which is also why find({a: NaN}) matches a stored NaN. Both servers said false. The Rust server’s Decimal128 path had it right and its plain-double path did not, so the operator’s answer depended on which numeric type happened to hold the NaN.

Both were found by tools/probes/query_result_sets.py the first time it ran with a Python column — the throwaway original had compared MongoDB against the Rust server only, and promoting it to the shared two-server harness surfaced a crash on the first run.

Fixed

• $gt / $gte / $lt / $lte / $cmp against a Decimal128("NaN") no longer error. NaN ranks below every number, so $lt is true and $gt is false, matching the plain-double NaN that already worked.

• {$eq: [NaN, NaN]} is true and $ne false, for all four pairings of double and Decimal128 NaN, on both servers.

• The neighbouring rules are unchanged and now pinned: a bool is still not a number ({$eq: [true, 1]} is false), signed zeros are still equal, and change detection still answers differently from equality — $set of a NaN over the same-typed NaN reports modifiedCount: 0, while a double NaN replaced by a Decimal128 NaN reports 1.

Added

• tools/probes/query_result_sets.py, tools/probes/update_result_documents.py and tools/probes/upsert_seeding.py — three sweeps that had been throwaway scripts, now comparing both servers against MongoDB (266, 527 and 120 shapes; 0 divergent).

Measured against mongod 8.2.11.

Aggregation operators answered the wrong BSON type, and an integer overflow crashed the update path

A sweep of all 143 aggregation expression operators against a real mongod 8.2.11 found 42 cases where both servers returned a value and the value was wrong. Nearly all of them were one of three rules the engines did not implement.

Fixed

• The rounding operators are type-preserving. $ceil / $floor / $trunc / $round answered a Python int for a double operand, so {$ceil: 1.5} returned 2 where mongod returns 2.0 — a different BSON type, which then compares and sorts differently downstream.

• long is contagious through arithmetic. $add / $subtract / $multiply / $mod / $pow / $abs and the four rounding operators all narrowed a 64-bit operand back to int when the result happened to fit in 32 bits, so Int64(1) + 1 answered an int where mongod answers a long. An int32 result that outgrows its width now widens to long on its own, as mongod’s does ({$abs: -2147483648}).

• An integral result past int64 no longer fails the command. It saturates to a double in an aggregation ({$pow: [2, 64]}), matching mongod. It previously reached bson.encode as an unbounded Python int, whose OverflowError surfaced to the client as internal server error.

• $inc / $mul past int64 now fail the write with mongod’s error (code 2, Failed to apply $inc operations to current value ((NumberLong)…) for document {_id: …}) rather than crashing. The same unencodable int was reaching bson.encode from inside the storage layer’s update transaction.

• $mod truncates toward zero. It used Python’s flooring %, which gives the wrong sign whenever an operand is negative — {$mod: [-5, 2]} answered 1 where mongod answers -1. Three of the four sign combinations were wrong.

• $toLower / $toUpper coerce their operand to a string first. They passed a non-string straight through, so {$toLower: 1.5} answered the number 1.5 rather than the string "1.5". That conversion is deliberately not $toString’s: it accepts a javascript value but rejects a bool and an ObjectId (Location16007), renders a double with six significant digits where $toString round-trips it, and turns null and missing into "" where $toString gives null.

• $toString renders mongod’s forms, not Python’s str(): true / false for a bool (it answered True / False), ISO-8601 for a date, base64 for binary, and ConversionFailure (241) for the types mongod refuses.

• Doubles inside error messages use mongod’s six-significant-digit rendering, so {$acos: 1099511627776} names 1.09951e+12 rather than 1099511627776.0.

• $degreesToRadians / $radiansToDegrees multiply by a single precomputed constant the way mongod does; computing x * pi / 180 differed in the last bit. They also now report a non-numeric operand as Location28765 like the rest of the math operators.

• $sin / $cos / $tan / $asin / $acos / $atan / $atan2 keep a Decimal128 operand’s type. decimal has no circular functions, so these fell through to the double path and narrowed a 34-digit operand to 17 digits.

After the change the Rust server has no value differences left across the 3,884-case sweep (was 33) and the Python server has five (was 42), all of them a last-digit disagreement on a Decimal128 transcendental where mongod’s own answer is 1-2 ulp below the correctly-rounded value.

25 operators that require a document argument, and the last of the sweep’s crashes

The largest uniform block left in the expression sweep: 25 operators, 675 shapes where mongod says the argument must be a document and the Rust server answered its generic BadValue (2) aggregation pipeline uses a stage or operator not supported by the Rust server.

Fixed

• $convert, $dateAdd, $dateDiff, $dateFromParts, $dateFromString, $dateSubtract, $dateToParts, $dateToString, $dateTrunc, $filter, $let, $ltrim, $map, $reduce, $regexFind, $regexFindAll, $regexMatch, $replaceAll, $replaceOne, $rtrim, $setField, $sortArray, $switch, $trim, $zip now answer mongod’s own code and wording on both servers. It is a table because the wording is five different phrasings that are not interchangeable — found: <type> versus found <type> versus no type at all — and the codes range from 9 to 5439007.

• The last of the sweep’s crashes. {$trunc: []} reached arg[0] (IndexError); an unrecognised key in $cond / $dateToString was a bare KeyError; and $exp / $sinh / $cosh of a large value raised OverflowError where mongod saturates to infinity. Each surfaced as internal server error.

  That takes the Python server from 274 crashes to zero across this sweep.

• $trunc / $round have a ranged arity (1–2 arguments) with mongod’s own 28667, distinct from the fixed-arity 16020.

• An unrecognised key is reported at parse time, so it takes mongod’s Invalid $<stage> :: caused by :: wrapper rather than the executor prefix.

Also fixed

Code 9 is FailedToParse, not Location9. The parse-time checks were naming every code Location<n>; the codes they return are a mix, and the differential gate caught it. Both servers now use their existing code-name tables.

Where the sweep stands

	start	now
Python crashes	274	0
Python code differences	2288	682
Rust code differences	2288	1556
Rust message-only	689	0

32 cases added to tests/test_mongod_differential.py.

Fixed-arity expression operators answer mongod’s 16020 — and a one-element list is one argument

The largest block left by the expression sweep: ~907 shapes where mongod answers 16020 Expression $x takes exactly N arguments. M were passed in. and we answered a mix of 14 / 28765 / 51044 / 51276 — 233 of them by crashing with internal server error, an operator indexing arg[0], arg[1] on a scalar.

Fixed

• 65 fixed-arity operators now check their argument count, with mongod’s own code and wording, on both servers. It is a PARSE error: an empty or missing collection reports it, as mongod does.

• A one-element list is ONE argument — and getting this wrong was producing silent wrong values, not merely wrong errors:

  expression	mongod	before
  {$size: [[1, 2]]}	2	1 (counted the outer list)
  {$size: ["$arr"]}	2	1
  {$toUpper: ["a"]}	"A"	["a"]
  {$type: [5]}	"int"	"array"
  {$first: ["$arr"]}	1	the whole array
  {$abs: [5]}	5	an error

  This was found by a test written for the arity fix, not by the sweep — the sweep’s corpus never paired an operator with a single-element list.

How the table was built

By asking mongod, not from documentation: each of the 143 operators was called with 0-4 arguments and the arity read out of its own error message. That also surfaced the three rules a table alone would have missed — $cond’s object form ({if, then, else}) carries all three arguments and is exempt; $substr is reported under its canonical name $substrBytes; and the count is len for a list but 1 for anything else, including a nested expression document.

The table lives in the engine rather than the command layer, because the evaluator needs it for the unwrap as well as the command layer for the error.

Result

	crashes	wrong 16020	message-only
Python before / after	274 → 21	~907 → 0	689 → 669
Rust before / after	274 → 1	~907 → 0	689 → 0

The remaining Python crashes are the Decimal128 family, already recorded. The remaining Rust code differences are the engine’s per-operator operand-type errors — a separate family, also recorded.

21 cases added to tests/test_mongod_differential.py, including the exempt forms and the parse-time (missing-collection) case.

83 expression errors carried the wrong wrapper, with the right message inside

mongod has three wrappers for an expression that fails inside $addFields / $project / $set: Invalid $addFields :: caused by :: for a parse error, Failed to optimize pipeline :: caused by :: for a constant-fold failure, and Executor error during aggregate command on namespace: … :: caused by :: for a runtime one.

Eighty-three shapes carried the wrong one — and their message body was byte-identical to mongod’s, so only the prefix was wrong. Nothing that compared error codes could see it, which is why they survived every earlier sweep of this surface.

Seventy-six of the eighty-three were two operators. $ifNull and $setEquals have their own arity code and their own wording, so they could not ride the generic arity table and were reaching the evaluator instead of the parser. The rest were missing required keys in a spec document ($convert without input, $dateDiff without startDate, $dateFromParts without a year, and the n-operator family without n).

The ordering is load-bearing. mongod reports an unrecognised key before a missing required one, so {$firstN: {k: 1}} is “Unknown argument for ‘n’ operator: k” and only {$firstN: {}} is “Missing value for ‘n’”. A first version of this change ran the missing-key checks first, which fixed the wrapper on all 83 and silently changed the CODE on six shapes that were already correct. Both checks fire on the same document, so only their order separates them; tests/test_expression_parse_time_wrappers.py pins it.

Against mongod 8.2.11 the probe goes from 142 message differences to 59, with the wrong-code set byte-identical to before.

The Rust server has the same defect and worse — it never emits the stage wrapper at all. Filed in tasks/backlog.md §7 rather than guessed at, since a fresh worktree cannot build its binary.

Fixed

• aggregate.py: _expression_shape_problem recognises the per-operator minimums ($ifNull, $setEquals) and the required-key specs, so they are classified as parse errors and take the stage’s wrapper. Both mongod wordings are reproduced verbatim, including the comma $ifNull has and $setEquals does not.

Changed

• tools/probes/agg_expressions.py, findandmodify_shapes.py, update_operators.py: a __main__ guard, so the corpus can be imported by a focused harness instead of the import running the whole sweep.

• tests/test_expression_parse_time_wrappers.py (new): 19 cases covering the classification, the unknown-before-missing ordering, and valid specs that must fall through to folding.

A second differential pass over findAndModify — 14 of 49 option combinations diverged from mongod

Phase 2 of tasks/remaining-work-plan.md asks for the differential harness to be pointed at surfaces it has never covered. findAndModify was one of them: 49 option combinations run against a live mongod 6.0.16, comparing the raw command reply rather than the driver wrapper, so lastErrorObject’s shape and the field order of an upserted document were compared too.

Fourteen diverged. Two were silent wrong data, and both turned out to be shared with the plain update command:

Fixed

• An empty update document silently kept every field. update: {} is a replacement with an empty document — mongod reduces the stored document to its _id and reports nModified: 1. Both write commands short-circuited on a falsy update and returned the document untouched, with ok: 1 and no error, so every field the caller asked to drop stayed. An empty pipeline ([]) is the genuine no-op and remains one.

• An upsert from a dotted query stored a literal dotted key. {"sub.k": 77} upserted a document with a key that has a dot in it — one mongod cannot produce, most drivers refuse to send, and which then never matched the very query that created it. It now builds the nesting mongod builds, at any depth, merging with any dotted paths the update itself sets.

• An update that would create a field under a non-document did nothing at all. $set: {"n.x": 1} against {n: 5} reported success and wrote no change. mongod answers PathNotViable (28) with Cannot create field 'x' in element {n: 5}. Creation only: $unset down the same path is still a no-op, and an out-of-range array index still pads with nulls.

• findAndModify reported every update failure as 14 TypeMismatch. The errors escaped to the generic dispatch handler, so an unknown modifier (mongod: 9), a changed _id (66), a path conflict (40) and a non-viable path (28) all arrived under one wrong code, and the drivers’ canonical handling keyed on those codes never fired. findAndModify also — uniquely on 6.0.16 — wraps its execution errors in Plan executor error during findAndModify :: caused by :: while leaving parse errors bare; both halves now match.

• codeName no longer contradicts code. Any user-facing exception that named its own code was reported with that code and codeName: "TypeMismatch" — a pair mongod never sends. The name now follows the code.

• findAndModify.new was never type-checked. A string went through Python’s truthiness, so new: "no" returned the post-image. It now takes the same bool-or-number rule as upsert and remove (numbers and null accepted, arrays / documents / strings rejected), and a zero-valued Decimal128 is false rather than truthy.

• An unknown top-level findAndModify field is rejected with Location40415, as mongod does. A misspelled option — field for fields, returnNew for new — was silently dropped, and the caller got a correct-looking reply computed under options they had not asked for.

• findAndModify.hint was accepted and ignored, so hinting an index that does not exist got a silent collection scan and an ok: 1 reply. It is now honoured, an unresolvable hint is BadValue (2), a non-string / non-object one is FailedToParse (9), and $natural — which mongod does not accept on this command, unlike find — is refused.

• arrayFilters type errors named a field path that does not exist on this command (update.updates.arrayFilters.0), with the wrong type. They now name findAndModify.arrayFilters, and an explicit null takes mongod’s older Location10065.

• An update path referencing an undeclared array filter answered 9 FailedToParse with hand-written wording; mongod answers 2 BadValue with No array filter found for identifier 'e' in path 'arr.$[e]', naming the path.

• The “unknown modifier” message is mongod’s, and there is one of it. Unknown $-operators and documents mixing operators with replacement fields are the same complaint to mongod, which names the offending key — Unknown modifier: z for a bare field. We had two different sentences, and neither is one any real server emits.

• Field order matches on the wire. An upserted document leads with _id, then the query-seeded fields, then the update’s, each group in field-name order; the update reply puts upserted / writeErrors before nModified. BSON keeps field order, and drivers do compare raw reply bytes.

The Rust side carries the same fixes: the core engine’s empty-update short-circuit is gone, its “unknown modifier” wording matches, and a non-viable path now defers with a named PathNotViable validator wired to code 28 for the standalone server, mirroring the arith_type_error template.

tests/test_mongod_differential.py grew 36 cases (57 → 93) so none of this can drift back.

findAndModify argument validation and reply shape match mongod

Differential-probing 18 findAndModify shapes against a real mongod found six divergences: a crash, two commands mongod rejects that we silently performed, and a reply that described a delete as an update.

findAndModify with a non-document update — {update: 5} — reached the update engine, which called .keys() on it and raised AttributeError. That surfaced as a bare internal server error (code 1) rather than mongod’s parse error.

remove: true combined with new: true or with upsert: true was accepted and the document removed. mongod rejects both: a remove has no “after” document to return, and upserting while removing is contradictory.

lastErrorObject for a remove carried updatedExisting, which describes an update. mongod omits it, so a driver reading that field saw an update-shaped reply for a delete.

Fixed

• A non-document, non-array update returns 9 FailedToParse (Update argument must be either an object or an array) instead of crashing.

• remove + new and remove + upsert are rejected with 9 FailedToParse, and the document is left in place.

• A remove’s lastErrorObject is {n: 1} (or {n: 0} when nothing matched). Update and upsert replies keep updatedExisting and upserted as before.

• Cannot specify both an update and remove=true gains the article mongod uses.

Probed on mongod 6.0.16 — the version the live differential gate spawns — and cross-checked on 8.3.4. Every behaviour above is identical on both; only the error wording differs (8.3 quotes the field names), and SecantusDB advertises 7.0, so 6.0’s wording ships.

The pymongo gauge measured the two servers against different bson versions

The gauge is the project’s headline MongoDB-compatibility number and it is routinely used to compare the two servers. It was not comparing like with like:

python mode   pymongo -> vendor/pymongo-tests/pymongo   (4.17.0)
              bson    -> site-packages/bson             (4.18.0)   <- MIXED
rust mode     pymongo -> vendor/pymongo-tests/pymongo
              bson    -> vendor/pymongo-tests/bson

The split is import order, not configuration. _start_server("python") does from secantus import SecantusDBServer, and secantus imports bson — at a point before pytest has inserted the vendored tree into sys.path, so the name binds to site-packages and stays bound for the rest of the run. Rust mode never imports Python bson at that moment (_secantus_server is a compiled extension), so bson resolves later, to the vendored copy.

That produced a phantom server difference: test_default_exports::test_bson failed on the Python server and passed on the Rust one, because vendored bson takes Generator from typing (which the test skips) and site-packages bson takes it from collections.abc (which it does not). Nothing to do with either server.

The plugin now puts vendor/pymongo-tests on sys.path in its pytest_load_initial_conftests hook, before the server import. Both modes load the same vendored bson / pymongo pair, and the two servers’ numbers became identical:

server	before	after
rust	1277 passed / 5 failed	1277 / 5 (unchanged — it already used the vendored pair)
python	1276 passed / 6 failed	1277 / 5

Both failure lists are now byte-identical and are exactly the five known-standing ones (test_index_hashed, test_index_text, test_where, all out of scope; test_maxtime_ms_message / test_to_list_csot_applied, pymongo’s client-side CSOT formatting). The Python server’s headline figure moves 99.53% to 99.61% by removing a harness artifact, not by any change in behaviour.

Fixed

• pymongo_validation/plugin.py: the vendored test tree goes on sys.path before the embedded server is imported, so bson cannot bind to site-packages first.

A test was excluded for a reason that turned out not to be true

Two psycopg tests that connect to reserved, unroutable addresses failed twice during full gauge runs, and were excluded with a note explaining that the host’s network stack had answered the address immediately instead of letting the attempt time out. That explanation was plausible, was written as though it were a measurement, and was never checked.

It does not hold. Connecting to those addresses in a loop fails six times out of six after four seconds with exactly the error the test expects, and re-enabling both tests in a full gauge run — with the machine simultaneously busy running another test suite — passed them. The exclusion has been removed.

What remains true is that the two failures happened, took forty milliseconds rather than four seconds, and are unexplained. That is now an open question with a note on how to answer it: capture the exception text, because the errno distinguishes a routing answer from resource exhaustion, and resource exhaustion would make it a defect in this project’s own test harness rather than a property of the machine it ran on.

Fixed

• test_connect_error_multi_hosts_each_message_preserved and its async twin run in the psycopg gauge again.

Changed

• The unexplained failures are recorded as an open backlog item rather than an exclusion, with instructions to capture the error before forming a theory.

A failing gauge run could print a perfect pass rate

docs/validation-report-php-lib.md published this, live:

| **Overall** | **3089** | **1** | **40** | **3130** | **100.0%** |

3089 passed, one failed, scored 100.0%. Every report formatted its rate with f"{...:.1f}%", which rounds — and 3089/3090 is 99.9676%. The website’s driver panels are generated from those reports, so the claim was on the public site.

The threshold is one failure in ~2000 tests, which is exactly where the healthy gauges now sit: the better the servers get, the likelier their own reports are to overstate them.

Fixed

• One shared validation_summary.rates.pass_rate that floors to the displayed precision, so only a genuinely clean run can print 100.0%. Raising the precision was considered and rejected — 99.98% still reads as “perfect” to someone skimming, whereas 99.9% beside a visible failure count does not.

• Adopted across 16 per-gauge report generators, the cross-driver summary (validation_summary/generate.py, including its adjusted rate) and the website panels (driver_panels.py), replacing nineteen copies of the same rounding idiom.

• tests/test_validation_pass_rate.py pins both halves: the arithmetic, and that a real generator actually uses it — verified by reverting one generator and watching the test fail.

$group gave every NaN its own bucket

Two documents with a: NaN grouped as two buckets of one where mongod reports one bucket of two. A wrong aggregation result rather than an error, so nothing surfaced it — the kind that gets absorbed into a report and never questioned.

Python dicts key a NaN by IDENTITY: hash(nan) is 0 so it is hashable, but nan != nan, so it never matches itself on lookup. The group-key canonicaliser returned hashable values unchanged, and every NaN became its own key.

Both engines were wrong, differently, and both cited the other as their authority. The Rust side deferred on NaN — safe while the pure engine is behind it, but the standalone Rust server has no Python behind a defer, so $group by a NaN could not group at all there. Its comment justified this with “NaN never equals itself in a dict probe”, and the key type’s doc header described itself as mirroring “Python dict equality”. Both describe Python, not the server.

mongod’s rule is simpler than either implementation assumed (probed 8.2.11, 2026-09-05): every NaN is one key, a Decimal128 NaN merges with a double NaN into that same key, and the rule applies inside arrays and subdocuments.

Neighbours on the same key path were checked: $sortByCount and compound group keys had it too and are fixed with it. $addToSet, $setUnion and sorting were already correct, which is why nothing had surfaced this.

Fixed

• aggregate.py: _hashable_scalar maps every NaN — float or Decimal128 — to one canonical bucket key, before the plain-hashable path that returned it unchanged.

• crates/secantus-core/src/group.rs: GKey::Nan replaces the defer, so the standalone Rust server groups a NaN instead of failing on it. A non-NaN Decimal128 still defers, which is a separate gap.

The published benchmark table is generated, not pasted

docs/benchmark.md’s “Over a real network” head-to-head table was the last benchmark surface a human had to update by hand.

Added

• bench.head_to_head_chart rewrites it from bench/results/do/<run>/comparison.md, the artifact release-benchmark already writes, and --check exits non-zero when the page is stale.

• tests/test_benchmark_table_fresh.py runs that check, so a forgotten refresh fails the suite instead of silently publishing old numbers. It went stale twice: once leaving a post-lz4 droplet section above a pre-lz4 latency table, and once running two releases behind while the header directly above it named a different mongod version.

delete and update performed the write when the hint named no index

Phase 2 of tasks/remaining-work-plan.md, third surface: index and query planning, probed against a live mongod 6.0.16. The headline is not a missing error message — it is an operation MongoDB declines to run being executed.

Fixed

• delete and update ignored their per-statement hint entirely. mongod refuses the statement when a hint names no index — n: 0 plus a writeError — rather than falling back to a collection scan. Both commands dropped the field and performed the write, so a caller who hinted a typo’d index name had their delete applied. Now a per-statement writeError (code 2), with the batch stopping or continuing per ordered, exactly like the other per-statement errors on those commands.

• explain did not validate a hint — and explain is what you run to check one. find, count and aggregate all rejected an unresolvable hint already; explain alone reported a COLLSCAN, which tells the caller their hint is fine and that it is being ignored, in the same breath.

• explain fabricated plans for commands that do not exist. {explain: {nosuchcmd: "c"}} answered a plausible COLLSCAN with ok: 1; so did {explain: {}} and a non-document explain argument, the latter inventing a namespace (<db>.$cmd) to report it against. They now answer mongod’s CommandNotFound (59) and TypeMismatch (14). Confidently wrong is worse than absent here: a client explaining a mistyped command got an answer about a query that could never run.

• {$natural: -1} did not reverse the scan. Both directions resolved to the same "$natural" token, dropping the sign, so a caller asking for reverse insertion order silently got forward order.

• distinct accepted any field and ignored it, so a misspelled option was silently dropped. Unknown fields now answer Location40415.

• explain.verbosity conflated two errors. mongod separates a wrong type (TypeMismatch, 14) from an invalid enum value (BadValue, 2, with its own wording); we emitted one hand-written message for both.

Known divergences, recorded rather than changed

Both are deliberate and are described in tasks/backlog.md:

• hint: "$natural" as a string is accepted here and rejected by mongod, which takes only the document form. It is a documented SecantusDB convenience with existing tests, and pymongo’s .hint("$natural") produces exactly that string.

• distinct.hint is accepted although 6.0.16 rejects it, because a later MongoDB release added the option — accepting is the safe direction for a field whose status changed between versions.

The explain stage vocabulary (SORT / LIMIT / SKIP / PROJECTION_SIMPLE wrappers, COLLSCAN’s direction, IDHACK, COUNT_SCAN, DISTINCT_SCAN, populated rejectedPlans, explainVersion) is a separate and much larger piece of work, now scoped as one backlog item rather than half-built — a partial stage tree would be more misleading than an honestly flat one.

Every command that takes a hint now resolves it, on both servers

A hint tells MongoDB which index to use, and MongoDB refuses a command whose hint names no index rather than quietly scanning instead. SecantusDB got that right for six of the seven commands that accept one. distinct was the exception: it accepted the field and then ignored it, so a typo’d index name returned a full result set where a real server returns an error.

Probing that gap across all seven commands turned up three more divergences, two of them on the Rust server only — the kind that a Python-versus-Rust parity test cannot see, because both servers were consistent with each other and both were wrong. One of them returned wrong data rather than an error: the Rust server ignored the direction in {$natural: -1} and walked the collection forwards.

The hint surface now matches mongod 8.2.11 exactly on all eleven shapes probed, on the Python server, the Rust server, and a real mongod side-by-side.

Fixed

• distinct resolves its hint like every other read. A valid index name or key spec is honoured (including a sparse index’s reduced document set); one naming no index answers BadValue (code 2) instead of returning every value.

• The $natural string is rejected on both servers. MongoDB accepts $natural only in the document form, and findAndModify already enforced that here while find did not.

• Rust server: {$natural: -1} walks the collection backwards. It previously resolved to the same token as {$natural: 1} and returned forward insertion order — the Python server had fixed this and the port had not.

• Rust server: an unresolvable hint on delete / update is a per-statement writeErrors entry with ok: 1, matching mongod, instead of failing the whole batch command.

Changed

• hint="$natural" is now an error. This is a behaviour change for anyone passing the string, including pymongo’s .hint("$natural") — but it is the same error a real MongoDB server gives, which is the point. Use .hint([("$natural", 1)]) or hint={"$natural": 1}; both work against SecantusDB and MongoDB alike. docs/indexes.md and docs/feature-comparison.md record the document form.

invoke sync now produces a environment that can actually run the suite

The task that sets up a development environment installed only the base test dependencies. That left out the compiled Rust engine, and because the ~1700 tests that compare the Rust and Python engines skip themselves when it is missing — on purpose, so the pure-Python parts still work anywhere — the suite would run to completion, report success, and be about 1700 tests short. Nothing failed; the tests simply were not there.

It also never rebuilt that engine when its source changed, so pulling a change to the Rust side left the comparison running against a stale build. That surfaced as 31 failures on a perfectly healthy main branch, which looks exactly like someone broke something.

Both are fixed, and the reasoning is written down next to the task and in the project guide so the next person does not have to rediscover it.

Fixed

• invoke sync installs all extras and forces a rebuild of the Rust engine.

Merging two jsonb objects with || produces JSON again

'{"x":1}'::jsonb || '{"y":2}'::jsonb returned {'x': 1}{'y': 2} — the two values rendered as Python dictionaries and glued together as text. Single quotes, no merge, and not valid JSON. Nothing errored; the wrong value simply came back, and any code parsing the result failed somewhere further along.

Two objects now merge, with the right-hand operand winning on conflicting keys, as PostgreSQL does.

Concatenations where either side is an array or a scalar are unchanged and still differ from PostgreSQL in how the result is typed — the values are right but render as an array literal rather than JSON. That is tracked separately.

Fixed

• jsonb || jsonb merges two objects instead of concatenating their Python representations as text.

Two internal server errors replaced with real answers

Deleting a key from a jsonb value — data - 'key' — returned “internal server error”. The operator fell through to Python’s subtraction, which has no meaning for a dictionary, and the resulting failure surfaced with no indication of what had gone wrong. It now deletes the key, along with the rest of PostgreSQL’s rules for the operator: removing a key that isn’t there changes nothing, deleting from an array works by index (counting from the end for a negative one), a list of keys deletes each, and the combinations PostgreSQL rejects — an integer index into an object, or deleting from a scalar — are rejected the same way.

Separately, an arithmetic expression whose operands have no matching operator ('\x01'::bytea + 1) also returned “internal server error”, for the same underlying reason: the raw Python operation was attempted and its failure escaped. It now reports operator does not exist: bytea + integer, naming both operand types the way PostgreSQL does.

Fixed

• jsonb - key, jsonb - index and jsonb - key[] work instead of returning an internal server error.

• An arithmetic operation with no operator for its operand types reports 42883 operator does not exist instead of an internal server error.

Opening a database no longer builds two tables nothing uses

Every time a SecantusDB store opened, it created two WiredTiger tables that no current code ever writes to: the single documents table that predates per-collection sharding, and the old forward index that insertion order replaced. They existed only so a one-time migration and the collection-drop paths could look at them and find nothing. Each table costs about ten milliseconds to create, on every open, forever.

They are no longer created. A store that already has them still migrates and drops from them correctly — every reader treats a missing table as an empty one — so existing databases are unaffected and remain readable by both servers. Opening a store is about 9% faster, which shows up wherever a server is started: tests, short-lived tools, and anything that opens a database per operation.

Changed

• The legacy secantus_documents and secantus_natural tables are no longer created when a store is opened, on either server.

Fixed

• A pre-existing clippy warning in secantus-storage, which sits outside the clean-workspace lint gate and so had gone unreported.

listIndexes can be paginated again

A listIndexes cursor was registered under a db.$cmd.listIndexes.<coll> pseudo-namespace, but drivers put the plain collection name in the follow-up getMore’s collection field. The getMore ownership check — which compares the caller’s claimed namespace against the cursor’s stored one, and is right to exist — therefore rejected every continuation with CursorNotFound. In practice any collection with more indexes than the batch size could not have its index list read to the end.

Real mongod reports that cursor under the plain db.coll namespace. Note this is not a blanket “drop the $cmd prefix”: probed on mongod 8.3.4, listCollections really is db.$cmd.listCollections and the collectionless aggregate: 1 form really is db.$cmd.aggregate. Both were already correct; only listIndexes was wrong.

Fixed

• listIndexes cursors are registered and reported under db.coll, so a getMore continuation succeeds and every index is returned. Fixed on both the Python and Rust servers.

• The getMore cross-namespace ownership check is unchanged — a continuation claiming a different collection is still rejected with CursorNotFound.

$graphLookup stopped following a chain at the first null link

Phase 2 of tasks/remaining-work-plan.md, fourth surface: 27 $lookup / $graphLookup shapes against a live mongod 6.0.16. 20 diverged.

The worst was a short answer with no error. $graphLookup treated a null connectFromField as “no further links”, so a four-document chain came back with one document in it. Nothing failed; the traversal simply stopped early, which is the hardest kind of wrong to notice.

Fixed

• A null link no longer ends the traversal. mongod follows it to documents whose connectToField is explicitly null; only a missing link stops the walk. Both halves were wrong in the same place, because the value was tested for None — which conflated missing with null.

• A null link no longer reaches documents that lack the field. Missing and null are different values here; comparing get_path’s None for both made every field-less document reachable from a null.

• An empty-array localField matches null. mongod unwinds the local array for matching and an empty one still joins against the null-valued foreign rows; we produced no lookup keys at all and matched nothing. Both join paths had it — the hash join and the index-driven one — and the index path carried a comment asserting mongod’s $in: [] semantics that the oracle contradicts. A $lookup’s localField is not an $in.

• as is a path, not a key. as: "a.b" now produces {a: {b: [...]}} instead of a literal key with a dot in it — the same bug, and the same fix, as the dotted-equality upsert seed fixed earlier in this campaign. Wherever a user-supplied path is used as a key, it has to go through set_path.

• Two crashes. $lookup with let: 5 reached .items() and with pipeline: 5 was iterated; both raised bare exceptions that escaped as internal server error (code 1).

• Argument errors name the argument. mongod answers FailedToParse (9) with a per-argument message; we answered TypeMismatch (14) with one of two generic sentences that named neither the field nor the problem. Which message applies to a half-specified field pair depends on whether a pipeline is present — probed both ways.

• Unknown arguments are rejected on both stages ($lookup → 9, $graphLookup → Location40104). They were accepted and ignored, so a misspelled foreignFeild silently became a join over the whole foreign collection.

• A negative maxDepth is rejected (Location40101). It was accepted and matched nothing, so every document got an empty array — which reads as “no connections” rather than “bad option”.

Correction

$graphLookup was first reported here as “does not recurse at all”. It does. The original fixture happened to put a null link on the very first hop, so one narrow bug looked like a missing feature; a chain with no nulls showed the traversal, the maxDepth cut-off and an array startWith all working. The fix is a guard, not an implementation.

Malformed aggregation pipeline elements now fail like mongod

A pipeline array element that isn’t a document — pipeline: [42] — crashed the Python server. _apply_stage called len() on the raw element, so a scalar raised TypeError: object of type 'int' has no len() and the client got a bare internal server error (code 1) with no indication of what was wrong. Real mongod answers 14 TypeMismatch with a specific message, and libmongoc’s /change_stream/accepts_array asserts on it verbatim.

Chasing that surfaced two further divergences in the same area. Our arity error for a stage that is a document but isn’t a single {operator: spec} pair used the generic code 14 and our own wording, where mongod uses a dedicated Location40323. Worse, the leading-$match optimisation matched on "$match" in stage alone: a malformed two-key stage such as {"$match": {...}, "$count": "n"} had its filter hoisted into the initial fetch and the stage dropped, so the $count was silently discarded and the aggregate returned wrong results instead of an error.

All the malformed-pipeline responses are now byte-identical to mongod, verified by differential probe against 6.0.16 and 8.3.4 (which agree with each other).

Fixed

• aggregate with a non-document pipeline element (42, "str", [...], null, a bool, a double) returns mongod’s 14 TypeMismatch / “Each element of the ‘pipeline’ array must be an object” instead of crashing with an unhandled TypeError behind a generic internal server error.

• A stage document with the wrong number of fields — an empty {} or a multi-key stage — returns mongod’s 40323 Location40323 / “A pipeline stage specification object must contain exactly one field.”, replacing our own code-14 wording.

• The leading-$match initial-filter lift now requires a single-key stage, so a malformed multi-key stage is rejected rather than partially applied and dropped.

• The Rust server had the same gaps on its plain-aggregate path, where both malformed shapes were skipped by continue during validation, and had no arity check on its change-stream path (an empty stage was misreported as 40324 “Unrecognized pipeline stage name: ‘’”). Both now match mongod and share the message constants with the change-stream path, which already had the element-type error right.

maxTimeMS actually times operations out

maxTimeMS was parsed and validated exactly the way MongoDB validates it — and then ignored. The operation ran to completion and answered ok where MongoDB aborts it with MaxTimeMSExpired. That is invisible on a fast operation, which is why it survived: a sweep at a one-second budget shows nothing, because nothing takes a second. At a two-millisecond budget it shows up everywhere.

A deadline cannot be a parse-time check, because the thing being bounded is elapsed time inside the handler. It is now a thread-local budget armed around the command and polled from the loops whose length tracks the data: the storage scan and its predicate pass, count’s own scan, the write commands’ candidate selection, the index build, and the aggregation pipeline between stages. An expired budget answers 50 MaxTimeMSExpired, and createIndexes wraps it in MongoDB’s index-build envelope, both reproduced from a probe of 8.2.11.

Enforcement is cooperative and polls once every 64 documents, so an operation can overrun by up to that many — MongoDB’s own enforcement is interrupt-point based and has the same property. getMore is deliberately excluded: there maxTimeMS is the awaitData wait budget, and arming a deadline would make every tailable poll report a timeout the moment it waited out its budget.

The write path polls candidate selection only, which happens before anything is written, so an expired budget leaves nothing half-applied.

Added

• secantus/deadline.py: the thread-local budget, armed by dispatch.

Fixed

• commands.py / storage.py / aggregate.py: maxTimeMS is enforced on find, count, distinct, aggregate, update, delete and createIndexes.

maxTimeMS is checked on every command, not just find

maxTimeMS is a generic command field — MongoDB’s IDL validates it on everything from find to ping — but SecantusDB checked it in find alone. On the other twenty-three commands that accept it, a wrong-typed value was taken without complaint: db.command({"aggregate": "c", "pipeline": [], "cursor": {}, "maxTimeMS": "x"}) ran the pipeline and reported success. That is the silently-accepted failure mode, where a driver bug reaches the application as a correct-looking answer. The check now runs once in dispatch, beside the readConcern and apiVersion checks, so every command gets it.

The four behaviours it checked were also still the MongoDB 6.0 ones, and 8.x honours none of them. A wrong type is now a TypeMismatch (14) naming the IDL struct, not a BadValue (2); a fractional value is FailedToParse (9); the negative and out-of-range messages changed wording and gained an upper bound of 2147483647; and an explicit null is now accepted, meaning the field was not sent. The old code’s own docstring described the slot as “the only one in this sweep that is not a TypeMismatch”, which is exactly what stopped being true.

Three of the rules would have been wrong if reasoned about rather than measured, so all of them were probed across 24 commands against a live mongod 8.2.1 — 436 cases, matching exactly. The IDL struct name in the type error is the command’s own name for all twenty-four except find, which reports FindCommandRequest. The check order matters: -1.5 is both non-integral and negative, and MongoDB answers the integral error rather than the range one. And a fractional Decimal128 gets different wording from a fractional double for the same numeric value. Where the check sits in dispatch was measured too — CommandNotFound takes precedence over it, while it takes precedence over the authorization check.

Separately, createIndexes with an explicit indexes: null answered MongoDB 6.0’s 10065 where 8.x treats an explicit null as the field being absent and answers 40414, identical to omitting it — the same null-means-absent rule already applied to findAndModify’s arrayFilters and killCursors’ cursors.

Both fixes land on both servers. The Rust server had ported the same 6.0 contract, comment and all, and called it from three commands rather than one — so it shared the bug in a slightly milder form. Neither the engine-parity suites nor the driver gauges would have caught the drift, because this is command-layer behaviour that no parity suite covers.

Fixed

• maxTimeMS is validated on every command that accepts it, not only find; aggregate, count, distinct, insert, update, delete and 17 others previously accepted a wrong-typed value silently.

• maxTimeMS follows the MongoDB 8.x contract: TypeMismatch (14) with the IDL struct name for a wrong type, FailedToParse (9) for a non-integral or unrepresentable number, BadValue (2) for a value below 0 or above 2147483647, and an explicit null accepted as absent.

• createIndexes with indexes: null answers 40414 IDLFailedToParse, the same reply as omitting the field, instead of 10065.

• The Rust server gets both fixes, having carried the same MongoDB 6.0 contract and validated maxTimeMS on three commands rather than all of them.

A $meta projection no longer discards the document

Asking for a metadata field — find({}, {score: {$meta: "recordId"}}) — came back with just _id. Every other field was gone. The projection was being treated as an inclusion list, so naming one metadata field silently excluded everything else.

MongoDB treats $meta the way it treats $slice: as something that reshapes a value, not as an instruction about which fields to keep. A projection containing only $meta returns the whole document; combining it with a real inclusion or exclusion field lets that field decide, and _id: 0 still drops the identifier.

The metadata value itself is still not computed, so the requested field is absent from the result — but the document it was asked about now comes back intact.

Fixed

• A $meta projection returns the document instead of reducing it to _id. {_id: 0}, and combining $meta with an ordinary inclusion or exclusion field, all behave as MongoDB does.

$meta projections return real values

{$meta: "recordId"}, "sortKey" and "indexKey" validated cleanly and then produced nothing — the field was simply absent from every document. MongoDB returns real metadata for all three, and SecantusDB has the data behind each of them (a RecordId per row, the sort specification, the index the query chose); it was never plumbed through to the projection.

All three now compute. sortKey reports the sort fields in order, taking the same representative element for an array-valued field that the sort itself took, and null for a missing one. indexKey reports the indexed fields’ values, and only for a real secondary-index scan — a collection scan and the _id fast path both omit the field, as MongoDB does. Asking for sortKey without a sort is now the error MongoDB gives rather than a silently missing field, and an unrecognised $meta argument reports MongoDB’s wording (Unsupported $meta field: zzz) in place of our own paraphrase, which a test had been pinning.

One deliberate difference: SecantusDB’s RecordId is a store-wide insertion counter where MongoDB restarts it per collection, so the numbers differ for a second collection in the same store. The properties a caller can use — unique per row, ascending in insertion order, unchanged by sorting the output — hold. Matching the numbers would mean a per-collection counter, and that counter is the document table’s key, shared byte-for-byte with the Rust server.

The metadata SecantusDB has no machinery for at all — text scoring, Atlas Search, $geoNear distances — still validates and then omits its field, which is graceful degradation rather than a wrong value.

Fixed

• projection.py / storage.py: $meta values are computed for recordId, sortKey and indexKey; sortKey without a sort is rejected; the unknown- argument message matches MongoDB.

A MinKey / MaxKey bound skipped documents that had no such field

$gt / $gte / $lt / $lte are type-bracketed — {x: {$gt: 3}} matches numbers greater than 3 and nothing else. MinKey and MaxKey are the two bounds that escape that bracketing, because mongod compares them against every type, and an absent field is one of the things they must reach: mongod’s query language treats a missing field as null, which ranks above MinKey and below MaxKey.

Both servers skipped an absent field outright in the range comparison, so {x: {$gt: MinKey()}} and {x: {$lt: MaxKey()}} — the idiomatic “match everything” bounds — left out every document that lacked the field entirely.

The fix is to compare an absent field as null rather than skip it, which is safe for the ordinary bounds precisely because they are bracketed: {x: {$gt: 3}} then compares null against a number, the brackets differ, and the document is dropped exactly as before. Both halves are pinned, along with the null bounds, whose own rule ($gte / $lte match null and missing, $gt / $lt match nothing) is unchanged.

Fixed

• secantus.query / secantus-core: a range comparison treats an absent field as null, so a MinKey / MaxKey bound returns the documents that have no such field, as mongod does.

Filed while probing the same surface, and materially worse than the entry that described it: computed projections ({lit: {$literal: 7}}) are unimplemented, and the Python server drops the field silently — ok: 1, and a document without the field the client asked for. The Rust server refuses honestly, which is the better of the two behaviours. Eleven measured shapes are in the backlog with the two semantics that are easy to get wrong.

“Missing” now propagates through $cond, $switch, $let and $ifNull

mongod distinguishes a missing value from an explicit null, and the operators whose result is one of their sub-expressions pass that missing-ness along. A $cond whose taken branch is missing is itself missing — so $addFields omits the field rather than writing a null. Both servers wrote the null.

Measured against mongod 8.2.11: 1 of 7 shapes were correct before this ($getField alone).

Fixed

• $cond, $switch, $let (its in) and $ifNull now return the missing marker when the sub-expression they select is missing, on both servers. {"$addFields": {"z": {"$cond": [true, "$nosuch", 1]}}} omits z, as mongod does, instead of writing z: null.

• It matters in operator position too, not just as a field value. {"$eq": [{"$cond": [true, "$nosuch", 1]}, null]} is false on mongod, because the result is missing rather than null — we answered true. The comparison operators already distinguished the two for a bare field path; now a value reaching them through a control-flow operator is distinguished as well.

• $ifNull skips a missing argument exactly as it skips a null one, and returns missing when every argument is.

The operators that compute a value — $add, $concat, $arrayElemAt, $first — still collapse missing to null, which both engines already had right. That split is the whole rule: return a sub-expression and missing-ness travels, compute one and it does not.

Notes

• expressions.evaluate_or_missing was a second copy of the field-value rule, and the copies had already drifted once: the $$REMOVE fix had to be written twice. It now delegates to the single implementation, so this change did not need writing twice too.

• The engine-parity fuzz suite caught the mid-port state (Python propagating, Rust not) on a seeded expression — exactly what it is for. Worth recording that the expression it failed on is one mongod itself rejects, so the parity gate is pinning the two engines to each other there, not to the reference server; the isolated question it raised was then settled against mongod directly.

25 cases added to tests/test_mongod_differential.py, covering both positions and both families (propagating and computing), so the split cannot quietly move.

A missing field compared equal to null in every aggregation comparison

A targeted sweep, motivated by the Phase 2 differential campaign: get_path returns None for both an absent field and an explicit null, so anywhere the code tested a resolved value for is None it had conflated them. That confusion produced both $graphLookup bugs and the $lookup let gap found earlier in the campaign, so the question was how far it spread.

25 shapes probed against mongod 6.0.16. 22 were already right — the query language deliberately does treat them alike ({a: null} matches a missing field, and that is mongod’s behaviour too). The divergence was confined to the comparison operators.

Fixed

• $eq: ["$absent", null] answered true; mongod answers false, while $eq: ["$explicitNull", null] is true. Every comparison against null matched documents that did not have the field at all, and $cond built on $eq inherited it.

• A missing field now ranks below every real value, MinKey included ($cmp: ["$absent", MinKey] is -1), and equals only another missing field. $ne / $lt / $lte / $gt / $gte / $cmp all follow.

• $let bound a missing field as null. A var bound from an absent field now stays missing, so $eq: ["$$v", null] is false the way mongod’s is.

• $lookup’s let had the same bug, which is how it was found: a document without the local field joined foreign rows mongod excludes. This closes the gap recorded when $lookup was swept.

The Rust engine carries the identical rule — the parity fuzz caught the divergence within seconds of the Python change, which is exactly its job.

Corrected

_eval_field_value’s docstring (and its Rust mirror) claimed {$add: ["$nope", 1]} is 1. mongod answers null; arithmetic over null is null. The behaviour was already right — only the note about it was wrong, and a test written from that comment failed against the real server. That is the sixth instance in this campaign of a comment asserting something the oracle contradicts.

Eight more test files share a server

The same treatment as the previous two rounds, applied to files that build their server, client and database in a single fixture: one server per file instead of one per test, with each test dropping what it created so the next sees a clean server. Their combined runtime falls from 26 seconds to 3.5.

Five otherwise-eligible files were deliberately left alone because their fixture inserts seed data. Sharing the fixture would run that seed once and the per-test cleanup would then delete it out from under every later test — the tests would still pass individually and fail together, which is the worst kind of change to make casually.

Changed

• Eight further test modules share a module-scoped server.

The seeded test files share a server too, without sharing their data

Five test files were held back from the previous rounds because their fixture inserts seed data before each test. Sharing that fixture would have seeded once and then let one test’s writes reach the next.

They now split the difference: the server is created once per file, but each test still gets its own client, its own fresh seed, and drops its database on the way out. Only the cost is shared, never the state. Their combined runtime falls from 40 seconds to 3.

Changed

• Five further test modules share a module-scoped server while keeping their per-test seeding and isolation.

Nine test files share one database server instead of standing up dozens

Most test files start a fresh database server for every single test, which is the safe default but costs about a quarter of a second each time. Where a file’s tests already write to separate collections, none of that isolation is actually being used — the tests cannot see each other regardless.

Nine such files now start one server per file. Their combined runtime drops from 101 seconds to 14. The selection was made by checking that every collection name in the file is unique and that nothing in it depends on a private server, rather than by judgement, so files involving change streams, the oplog, reopening a store, capped collections, TTL clocks or authentication are untouched.

Changed

• Nine test modules take a module-scoped server fixture.

Fixed

• invoke rust-test’s description claimed it tested the whole Rust workspace. It tests the clean workspace, which excludes the six WiredTiger-linked crates — the entire storage layer and the shipped binary — so it could report success having never compiled them.

Nine more test files share a server, by cleaning up after themselves

A previous change let test files share one database server, but only where every test happened to use a differently-named collection. That ruled out most files — not because they genuinely needed their own server, but because two tests happened to pick the same collection name.

Those files now share a server too, and get their clean slate a cheaper way: each test drops the databases it created when it finishes, so the next test sees a server that looks new. That turns out to cost far less than starting a fresh server, and it needed no changes to any test’s body. The nine files run in 6 seconds instead of 23.

The cleanup runs after the test rather than before, so a failing test leaves its data in place to be inspected.

Changed

• Nine further test modules share a module-scoped server, with a per-test database reset providing isolation.

Error messages and codes now match a current MongoDB server

SecantusDB’s job is to be indistinguishable from a real MongoDB server, and that includes being wrong in the same places it is. Its error surface had been matched against MongoDB 6.0 — closely enough to reproduce a genuine quirk where 6.0 put a closing quote inside a bracket in one of its type-mismatch messages. MongoDB has moved on since, and the differences had quietly accumulated: negative batch sizes reported a different code, update failures were missing the wrapper newer servers put around execution errors, an explicitly null argument was rejected where a current server treats it as simply absent, and $lookup and distinct had both switched to generated argument parsing with a different family of messages.

The surface is now matched against MongoDB 8.2, verified case by case against a real server rather than transcribed. Applications that key on error codes will see the current server’s codes: a negative limit, skip or batchSize now reports BadValue rather than the old internal location code, and malformed $lookup arguments report the same missing-field and unknown-field codes a real server sends. The differential test suite that compares the two servers operation-by-operation passes in full, and the same changes have landed in the Rust server so both stay in step.

Changed

• Negative cursor-sizing values report 2 BadValue instead of 51024.

• Type-mismatch messages carry the current server’s per-field type lists.

• Execution-time update and aggregate failures are wrapped in the server’s executor-error prefixes; parse-time failures are not.

• An explicit null for findAndModify.arrayFilters / killCursors.cursors is treated as an absent field.

• $lookup and distinct argument errors use the generated-parser messages and codes (40414 / 40415 / 14), with IDLFailedToParse / IDLUnknownField code names.

Custom range types expose their auto-created multirange companion

CREATE TYPE testrange AS RANGE (...) on the Rust PostgreSQL server now creates the companion MULTIRANGE type that PostgreSQL auto-creates alongside every range type, and psycopg’s MultirangeInfo.fetch resolves it. Previously the companion was entirely absent: MultirangeInfo.fetch(conn, "testmultirange") errored column "rngmultitypid" does not exist, because pg_range had no rngmultitypid column, no per-range multirange oid was minted, and no multirange pg_type row existed.

The multirange’s name follows PostgreSQL’s rule — substitute multirange for the first range in the range’s name (testrange → testmultirange, int4range → int4multirange, rangetest → multirangetest), or append _multirange when the name contains no range (foo → foo_multirange). Each custom range mints a multirange oid and a multirange array oid, gets a multirange row in pg_type (its typname bare, as in PostgreSQL), and a pg_range.rngmultitypid pointing back at it. to_regtype resolves the multirange name in bare, schema.name, and quoted sql.Identifier forms, and a schema-qualified range’s multirange is distinct from the public one’s — mirroring the range schema-qualification from the previous release. All four MultirangeInfo.fetch forms resolve to the right oid and subtype, verified against a live PostgreSQL 14 oracle.

Added

• pg_range carries a rngmultitypid column, populated for both builtin ranges (pointing at the builtin int4multirange/etc. types) and custom ranges (pointing at a minted multirange oid, range_oid + 200_000; its array type + 300_000).

• A multirange pg_type row is synthesized for every custom range, with a bare typname derived by PostgreSQL’s range→multirange naming rule and its own array oid, so MultirangeInfo.fetch finds it after resolving to_regtype and joining pg_range.

• to_regtype resolves a custom multirange name (bare in public, schema.name otherwise), distinct per schema; the reverse oid::regtype::text rendering resolves a multirange oid back to its name.

A partial index hid every NaN row, and $min/$max ranked NaN by IEEE rules

Widening two probe corpora to include values no earlier sweep had ever contained — NaN, the infinities, Decimal128, MinKey/MaxKey, -0.0 — found two defects, present identically on both servers.

A partial index silently dropped rows. A partial index on {b: {$lte: 1.5}} made find({b: NaN}) return nothing; the same query on the same documents returned the row with no index, and MongoDB returns it in both cases. The implication check decides whether a user’s query is narrow enough to be answered from a partial index, and it compared encoded sort keys — where NaN orders below every number — concluding that a NaN equality was covered by a $lte: 1.5 filter that in fact excludes it. NaN is the value that separates the two orderings MongoDB maintains: sort places it below -Infinity, but every range operator excludes it while equality matches it. The type-bracket gate added for the previous instance of this bug class cannot catch it, because NaN is inside the numeric bracket.

$min/$max used the wrong comparison. Both compared with IEEE semantics, under which every NaN comparison is false, so {$min: {a: NaN}} over a: 5 left the field at 5. MongoDB ranks the operands by sort order, where NaN is below -Infinity, and writes the NaN. Both engines now compare through the sort-key encoder they already had — the operators were simply bypassing it.

index_result_sets.py is 0 of 1,803 on both servers after the fix (it found the first defect at 1 of 1,788), and update_operators.py closes the Python column at 0 of 70 shapes.

Fixed

• storage.py, crates/secantus-storage: the partial-index implication check refuses any non-equality bound involving a NaN on either side.

• update.py, crates/secantus-core: $min/$max rank operands by sort order, not IEEE comparison.

Changed

• tools/probes/index_result_sets.py: 11 more values in the corpus (NaN, the infinities, Decimal128, MinKey/MaxKey, ObjectId, a date, an Int64), and PROBE_SEED / PROBE_TRIALS overrides so a sweep can be re-run at a different seed rather than only deeper.

• tools/probes/update_operators.py: 34 more shapes, 36 → 70, covering the same value classes across $inc/$mul/$min/$max/$set — including the NaN-versus-infinity $min/$max pairs, which a test had asserted against our own servers only.

$near distance bounds are validated like mongod’s

{$near: {..., $minDistance: null}} ran as an unbounded query instead of being rejected, and negative bounds were accepted outright. The bound parser could not distinguish key absent from key present with null, so an explicit null was silently treated as “no bound”. Strings and bools were already refused, which is why the gap looked covered.

Found by differential-probing $near against a real mongod: 4 of 8 cases diverged.

While fixing it, a rationale recorded in the Rust engine turned out to be false. A comment there justified accepting null by claiming the Java driver “sends $minDistance: null when the caller passes no minimum”. The driver source disproves that — it omits the field entirely, in both serialisation paths (Filters.java, if (minDistance != null) { ... }). So null-tolerance was never needed for the Java gauge, and it made both servers silently run a query mongod errors on. The comment and its three tests have been corrected.

Fixed

• $near / $nearSphere reject a null or negative $minDistance / $maxDistance, matching mongod’s messages (must be a number, must be non-negative).

• The codes match per form, which differ: the nested GeoJSON form uses BadValue (2), while the legacy sibling form ({geo: {$near: [x, y], $maxDistance: …}}) uses mongod’s dedicated 16895 ($maxDistance) and 16893 ($minDistance).

• An absent bound still means unbounded — the fix distinguishes it from an explicit null rather than rejecting both.

A document holding a NaN was rewritten by updates that touched nothing

The write guard asked “did this document change” with a value comparison, and NaN != NaN. So an update that changed nothing at all — an $unset of a missing field, a $rename of a missing source, a $pull that matched nothing — looked like a change on any document containing a NaN anywhere, nested in a subdocument or inside an array included. The document was rewritten, an oplog entry emitted, a change-stream event delivered to every watcher, and nModified came back 1. mongod reports 0 and writes nothing.

The Rust server carried this in full. The Python server was hidden from it by an accident — container equality short-circuits on object identity, so an untouched value compared equal — which was never a safe thing to rely on and is now gone from both. Both servers compare the encoded BSON, which sees a signed zero and a numeric type change while treating two identical NaNs as identical.

The encoding is not quite all of mongod’s rule, and the remainder is the interesting part. mongod counts an arithmetic write whose result is a NaN: {$inc: {a: 1}} over a: NaN reports 1, while {$min: {a: 5}} over the same document reports 0 because $min declined to write, and {$set: {a: NaN}} reports 0 because it wrote an equal value. All five have byte-identical before and after images, so no document comparison can separate them — the discriminator is exactly “an arithmetic operator produced a NaN”, and that is now its own narrow check rather than a side effect of how != treats NaN.

Fixed

• secantus.storage / secantus-core: the write guard compares the encoded BSON, so an update that touches nothing no longer rewrites a NaN-holding document, emits an oplog entry, or fires a change-stream event.

• secantus.update / secantus-core: arith_wrote_nan carries mongod’s per-operator half of nModified, so $inc / $mul over a NaN still count as modifications while $min / $max that decline to write, and $set of an equal value, do not.

• secantus-core: diff::same_encoding’s catch-all compares values rather than BSON variant discriminants, which was only sound while it was called behind an equality check.

• secantus.update: arith_wrote_nan accepts a pipeline update (a list) rather than raising, which had turned findAndModify with update: [] into an InternalError. Caught by the mongod differential gate.

Query and update operators answer mongod’s errors — and $bits* answers the right documents

A systematic sweep crossed every query and update operator with every pathological argument — 2,226 shapes — against mongod 8.2.11. 583 disagreed. A hand-picked sample of 32 had found only 12 of those, and had missed $bits* entirely; $bits* turned out to be returning the wrong documents, not the wrong message.

Fixed

• $bitsAllSet / $bitsAnySet / $bitsAllClear / $bitsAnyClear took a plain integer and nothing else. They silently skipped array elements (which mongod matches element-wise, like every other multikey operator), doubles, Decimal128, and BinData values — and rejected BinData masks outright. A document holding [1, 4] or 5.0 was invisible to a query that should match it. 35 of 44 probed shapes were wrong; now none.

• Two arguments crashed with internal server error. {v: {$regex: BinData}} compiled as a bytes regex and then raised on a string subject, and {v: {$type: Code(...)}} hit an unhashable-key set test. Both now report mongod’s error.

• {v: {$not: {a: 1}}} matched. An ordinary field name inside $not degraded to an equality test which $not then negated, returning the document; mongod refuses the query. Every key inside $not must be an operator.

• $rename applied when its target was a bson.Code, because Code subclasses str and passed the type check.

• {v: {$type: []}} answered an empty result set where mongod refuses an empty alias list, and Decimal128 is a valid numeric type code.

• $currentDate reported the wrong problem for an unrecognized option key: mongod names the key before it looks at $type, so {$type: "date", a: 1} names a.

• A whole Decimal128 is a valid $pop direction, $size and $bits* mask. All three rejected it.

Changed

• The value renderer mongod uses to echo an offending argument had drifted into five partial copies. $inc / $mul / $pop / $rename all used the scalar-only one, so an array printed [1] where mongod prints [ 1 ] and a sub-document printed {'a': 1} where mongod prints { a: 1 }. There is now one, in bsontypes, alongside the type-name helper that had already been consolidated for the same reason.

• $pop, $size and the $bits* mask share mongod’s numeric-argument ladder — NaN, out of range, fractional and non-integral Decimal128 each get their own sentence — extracted as one helper.

• Message wording now matches mongod throughout: $and argument must be an array, unknown operator: $x, malformed mod, needs to be an array (which mongod distinguishes from a too-short array), and the $bit family’s four distinct texts.

tools/probes/operator_error_surface.py is the standing cover.

Projected fields came back in the wrong order — in both engines, differently

mongod emits projected fields in the stored document’s order and ignores the projection spec’s order entirely. Neither engine did that:

	order emitted
mongod	document
pure Python	spec
Rust	alphabetical (its spec trie is a BTreeMap)

So find({}, {"c": 1, "b": 1}) returned fields in an order no mongod produces. Field order is behaviour — a driver renders it, and a wire-level test can assert it — and this was wrong on every projection either engine has ever done.

Why it survived. The parity suites compare the two engines, and == on a dict ignores key order, so the dimension was invisible. The Rust variant was invisible even to a probe for a subtler reason: alphabetical order coincides with document order for any document whose keys happen to be sorted, and the fuzz corpus was keyed a, b, c. Only a document keyed z, a, m separates the three orderings, and nothing had one.

The comparator is now shared. tests/parity_compare.py holds the one same() — NaN equals NaN, signed zeros are distinct, key order compared, recursing into arrays and documents — and all eight test_rust_*_parity.py suites use it. Previously only the expressions suite had a sharpened comparator, and the other seven compared with ==. Routing them through it is what surfaced the projection bug within seconds.

Fixed

• projection.py, crates/secantus-core/src/projection.rs: _include_doc / include_doc iterate the DOCUMENT, not the spec trie. Nested levels follow their own sub-document’s order. exclude_doc needed no change — it removes keys in place and so preserves order already.

Changed

• tests/parity_compare.py (new): the shared parity comparator.

• All eight parity suites route their value comparisons through it — 39 sites that used a bare ==.

• Signed zero added to the update, projection and aggregate corpora, and the measured field-order cases to the projection corpus.

• tests/test_projection_field_order.py (new): pins the measured mongod order and explicitly excludes BOTH wrong answers, using documents that are not already alphabetical — the only shape that can tell them apart.

secantusd-pg can be released

The Rust PostgreSQL server was a headline product with no way to ship it: release-binaries.yml builds the MongoDB binary only, no workflow referenced the crate, and no tag scheme existed for it.

Added

• release-pg-binaries.yml — a tag-triggered track for secantusd-pg, modelled on the MongoDB binary’s. Builds static-WiredTiger archives for x86_64-unknown-linux-gnu and aarch64-apple-darwin, smokes the exact artifact it is about to publish, and attaches tar.gz + .sha256 to a GitHub pre-release. Triggered by secantusd-pg-v<crate-version> — a third tag namespace, verified disjoint from the PyPI v[0-9]+… and the MongoDB secantusdb-v* patterns.

• tests/test_rust_pg_binary_smoke.py — psycopg → secantusd-pg → WiredTiger against the release artifact, including a restart to prove the data is on disk. Registered in CI’s pg-oracle lane, the only lane that builds the binary.

Fixed

• secantusd-pg --version and --help now answer. --version previously fell through to the positional storage-path argument, so the server tried to open a WiredTiger database in a directory called --version and reported WT_TRY_SALVAGE: database corruption detected.

• secantusd-pg creates its storage directory when missing, which secantusd-rs already did. A first run against a fresh path failed inside WiredTiger with WiredTiger.lock: handle-open: open: No such file or directory — an alarming answer to pointing it at a new directory.

The PG server is not on the MongoDB crates’ lockstep version (0.1.0-beta.0 against 0.5.3-beta.163), so its tag carries its own number and the workflow asserts against crates/secantus-pgserver/Cargo.toml alone.

The Rust PostgreSQL server’s scaling problem was not the one on the list

A backlog item had stood for over a week saying the Rust PostgreSQL server serialised its writes behind a global mutex, and that removing it was the next piece of performance work. Reading the twenty-two places that mutex is taken shows what it actually guards: checkpoints, archives, collection options, renames, index builds and catalog records. Not one row insert, update or delete takes it. The single site near a write path holds it just long enough to open a storage session, and per-connection transactions — the thing the entry proposed building — were already there.

Measuring says the same. Against PostgreSQL 16 on the same machine, write throughput scales at 2.14x on four connections where PostgreSQL manages 2.36x, and putting every client on one contended table costs this server almost nothing extra. The read path, which takes no lock at all, behaves the same way. There is no collapse to fix.

What the measurements do show is a statement that costs too much. A single uncontended connection spends about sixty-six microseconds of server CPU on a SELECT that PostgreSQL serves in thirteen — roughly five times the cost, before any concurrency is involved, and that ratio is most of the gap at every connection count. It is a profiling problem, not a locking one, and the item now says so. The benchmark that settled it ships as invoke pg-concurrency so the next person re-measures rather than re-reasons.

Added

• bench/pg_concurrency.py and invoke pg-concurrency: statement throughput versus connection count for secantusd-pg and PostgreSQL 16, with repeated trials, a median, and the run-to-run spread that says whether two rows differ.

• tests/test_pg_concurrency_bench.py, covering the summary arithmetic and the release-binary assumption a debug build would quietly invalidate.

Changed

• The storage-concurrency backlog entry now records what the lock guards, the measured scaling and CPU-per-operation figures, the confound that makes the high-connection rows untrustworthy on a four-performance-core box, and an explicit warning not to touch the lock as a concurrency fix.

• Six over-length hint= strings in tasks.py wrapped.

A skipped test now says why it skipped

Six test suites diff SecantusDB’s SQL against a live PostgreSQL. When that server is unreachable they skip — and until now they skipped with the words no local PostgreSQL oracle, which is indistinguishable from PostgreSQL is not installed. A suite disabled by a transient connection failure looked exactly like a suite deliberately switched off, so nobody could tell whether ~109 tests were reporting green because they passed or because they never ran. Answering that question required a full 20-minute suite run.

The six suites now share one probe, tests/pg_oracle.py, and its skips name the DSN and the underlying exception. They also stop drifting: the copies had grown three different default DSNs between them, one of which omitted the user.

For the record, the suites are not skipping — a full run executes all 195 of those tests. The backlog entry claiming otherwise described a Postgres.app permission gate on a box that runs Homebrew’s PostgreSQL, and has been corrected. The documented location of this machine’s mongod builds was corrected the same way: the reference-server notes named a version that is not installed and called two that are “gone”.

Fixed

• The PostgreSQL reference-server suites (test_sql_search_path, test_sql_subms_timestamps, test_sql_operator_types, test_sql_result_type_tags, test_sql_float_rendering, test_sql_isolation_level) skip with the DSN and the connection error instead of a bare “no local PostgreSQL oracle”.

• Three drifted default DSNs collapsed into one; the probe is cached, so a worker opens one connection rather than nine. That bounds collection-time cost at one connect_timeout when the server hangs rather than refuses.

Changed

• CLAUDE.md and tasks/remaining-work-plan.md record the three mongod builds actually installed (6.0.16, 8.2.11 on PATH, 8.3.4) with Cellar paths, replacing claims that PATH gives 8.2.1 — which is installed nowhere — and that 6.0.16 and 8.3.4 “are gone”. Version-difference probing described by older entries is reproducible on this box today.

• A docstring in tests/test_rust_pgserver_differential.py asserted that the old skipif shape leaked a connection per worker. Measured: it does not — CPython collects the unreferenced connection immediately. The claim is corrected rather than repeated.

Timestamp ranges keep their microseconds, and pipelined replies wait for Sync

The PostgreSQL server stored tsrange and tstzrange bounds to the millisecond, so a range written as […49.338943, …) read back as […49.338000, …), with no error. Plain timestamp columns were fixed for this some time ago; range bounds live inside the range value and never got the fix. They now round-trip exactly on every write path: INSERT, bound parameters, INSERT … SELECT, UPDATE, COPY (text and binary), multiranges and arrays of ranges.

Two older range bugs turned up along the way. Comparing a stored range with a constructed one (r && tsrange(…), r * tsrange(…)) failed with an internal error, and WHERE r = '<range literal>' could miss a row holding exactly that range. Both work now.

The server also sent each extended-protocol reply as soon as it had it. Real PostgreSQL holds them until the client’s Sync or Flush, so a client counting round trips could see one pipeline turn into two. Replies are now buffered the same way.

All three were found by re-running the driver conformance gauges with the CI runners moved off UTC; none of them is time-zone related.

Fixed

• sql/ranges.py, sql/typemap.py, sql/planner.py: a timestamp range bound’s sub-millisecond remainder is stored inside the range subdocument (ranges.pack) and restored on every read (ranges._bound, and the lower_bound / upper_bound accessors now used by the binary wire encoders, the range sort key and SQL lower() / upper()).

• sql/ranges.py: a naive (stored) bound orders as UTC against an aware (constructed) one; stored && tsrange(…) and * were XX000.

• sql/planner.py: range = / <> is evaluated per row via ranges.canonical rather than pushed down as a whole-subdocument BSON match.

• sql/pgserver.py: extended-protocol replies are buffered until Sync or Flush, or past PostgreSQL’s 8 KB send buffer; an ErrorResponse is still sent at once.

Testing

• tests/test_pg_range_subms.py: every write path, containment and equality within one millisecond, the naive-vs-aware crash, and a comparison against a real PostgreSQL that runs in the pg-oracle CI lane.

• tests/test_pg_pipeline_flush.py: wire-level; nothing before Sync, Flush delivers without Sync, errors are not held back, and a large result streams.

Where the Rust PostgreSQL server’s statement time actually goes

The previous release established that this server costs roughly five times more CPU per statement than PostgreSQL, and that no lock was responsible. It could not say which layer owned the time. It can now, and the answer is that almost none of it belongs to the query.

A statement that touches no table at all — select 1 — costs fifty-five microseconds more than a bare protocol round trip, where PostgreSQL pays nine. The wire layer is not to blame: an empty round trip is within two microseconds of PostgreSQL’s. Reading one row by primary key adds only twelve more, so storage is not to blame either. The overhead sits in catalog bookkeeping done once per statement, by two different mechanisms. Outside a transaction, every statement opens seven fresh WiredTiger sessions to re-confirm that catalog collections which are created once and never dropped still exist. Inside a transaction block it is worse rather than better — around forty microseconds worse — because holding a block makes the catalog cache ineligible and each statement re-scans the catalog from storage instead.

One finding from this work has been retracted rather than shipped. A single run suggested prepared statements cost seventy-seven microseconds more than unprepared ones, which would have been a serious and surprising defect; four further runs put the two within noise of each other. The number was an artifact, and the backlog now says so explicitly, because a plausible phantom that nobody can reproduce is more expensive than no finding at all.

Added

• bench/pg_statement_cost.py: a layer bisect from a bare protocol round trip up to a prepared row read, run against both servers, with --in-transaction for the block path.

• tests/test_pg_statement_cost_bench.py, pinning the stage set the attribution is a difference between.

Changed

• The backlog entry now carries the measured per-layer figures, the two call paths behind them, the retraction above, and the instruction to repeat a measurement before quoting it.

The Rust PostgreSQL server passes psycopg’s array and cursor suites

psycopg’s tests/types/test_array.py and tests/test_cursor_common.py went from 52 failures to 2 against the Rust PG server (secantusd-pg), every change measured against PostgreSQL 16 first. The array literal scanner is now a port of array_in (only the six array_isspace characters are trimmed, quotes and backslashes are handled at the level they occur, the [lo:hi]= decoration is validated, and each malformed shape is the 22P02 PostgreSQL gives it); multidimensional arrays round-trip in text and binary both ways; || beside an array follows array_cat / array_append / array_prepend. INSERT … RETURNING returns named, computed and * columns typed from the row — including executemany(..., returning=True) — and serial / bigserial columns draw from a real <table>_<column>_seq sequence. INSERT … SELECT writes the query’s rows (it used to answer INSERT 0 0 with nothing written), literal column DEFAULTs are stored and applied, and cur.stream() works because Describe now distinguishes a zero-field RowDescription from NoData.

The box type landed with its ; array delimiter — '{(1,2),(3,4);(5,6), (7,8)}'::box[] parses and renders as PostgreSQL does, corners re-ordered per coordinate — and pg_type reports typdelim per type. Doubles now render as float8out (1e+20, 1e-07, Infinity, {1.5,2}) instead of Rust’s shortest form (1e20, inf, {1.5,2.0}), a float8 with a numeric operand is float8 arithmetic, unary minus keeps a double’s signed zero and numeric has none. user / current_user / session_user / current_role answer the role the client connected as, and an unaliased cast, array[…], row(…), coalesce, greatest / least and nullif column is named as PostgreSQL names it (int4, float8, bpchar, array, row, …) rather than ?column?. A cast PostgreSQL has no definition for (5::box, 1.5::bool, true::int8) is 42846 cannot cast type …, not a 22P02 parse failure.

Fixed

• crates/secantus-pgplan: parse_array is a port of PostgreSQL’s array_in — whitespace set, nested quoting, unquoted NULL, the [lo:hi]= prefix, per-element delimiter (; for box), and every 22P02 message; array_concat follows array_cat; nested arrays typed by depth.

• crates/secantus-pgserver: binary array parameters with ndim > 1 are reshaped into nested arrays; binary array results nest likewise.

• crates/secantus-pgplan / -pgserver: INSERT … RETURNING (named, computed, *), executemany(..., returning=True), an empty bound list binds as an empty array; serial / bigserial sequences in __sql_sequences__ (the Python server’s shape), dropped with the table; INSERT … SELECT; literal column DEFAULTs in the catalog, applied to every omitted column; an expression DEFAULT is refused (0A000) instead of dropped.

• crates/secantus-pgserver: SHOW completes with a bare SHOW tag; a FROM-less select … where <const> honours the predicate (false / NULL is zero rows, a non-boolean is 42804, 'x' is 22P02).

• crates/secantus-pgplan: pg_sleep(seconds) — a void (2278) column, waits on the connection thread; copy (select <expr> …) to stdout evaluates its expressions; a crossed range inside an array literal is 22000 from the range parser.

• crates/secantus-pgplan: expressions over generate_series rows (arithmetic, casts, calls, date arithmetic, comparisons) evaluate per row and are named / typed as PostgreSQL does.

• crates/secantus-pgserver / crates/vendor/pgwire: Describe answers NoData only for statements that return no rows; a zero-field RowDescription (select) yields one empty row, so cur.stream() works.

• crates/secantus-pgplan/src/geo.rs (new): the box type — parse_box (every input spelling, per-coordinate corner ordering, NaN to the high corner, "1e400" is out of range for type double precision 22003), box_text, float8_text (PostgreSQL’s float8out); pgtypes::typdelim. pg_type rows carry the real typdelim; oid 603 / 1020; a bound text parameter of oid 603 casts.

• crates/secantus-pgplan: float8 text rendering is float8out everywhere (column, ::text, float8[]); a float8 with a numeric operand is float8 arithmetic (0.1::float8 + 0.2); unary minus negates a double outright (-(0.0::float8) is -0); numeric has no negative zero (- 0.0 is 0.0) and a zero mantissa with an exponent renders 0 (0.00e3).

• crates/secantus-pgplan: user / current_user / session_user / current_role are the connecting role (name, oid 19); current_catalog / current_schema; "user" is an ordinary missing column (42703).

• crates/secantus-pgplan: FROM-less select columns are named as PostgreSQL’s FigureColname: a cast after its target type (int4, float8, bpchar, numeric, char, box), a nested cast after the outer type, array / row / coalesce / greatest / least / nullif after the keyword, and a cast of any of those keeps the inner name.

• crates/secantus-pgplan: Error::CannotCoerce (42846) for casts PostgreSQL does not define — 5::box, box::float8, 1.5::bool, 1::int8::bool, 1::int2::bool, '2021-01-01'::date::bool, true::int8, true::int2; true::int is 1. The cast site’s source type decides, so '2021-01-01'::bool stays the 22P02 text failure.

The Rust PG server answers every type psycopg draws in binary, byte-for-byte

psycopg reads every column of a row in the format of column 0, so a result set that mixes one text-only type in with binary ones makes the client run text loaders over binary payloads. The psycopg gauge’s randomised “faker” tests — test_leak, test_copy_to_leaks, test_random and their async twins — draw a random assortment of types per run and were failing on roughly ninety parameterisations for exactly that reason. The Rust PG server now encodes the whole faker family in binary — the date / time / timetz / timestamp / timestamptz family (including 24:00:00, years past 9999 and the BC era), interval, every range and multirange, json / jsonb and their arrays, and empty arrays — and each layout was pinned against PostgreSQL 16 by comparing the raw bytes, not by asking whether the client could decode them. A dozen smaller things fell out of the same runs: a timestamptz result in a named session zone (Europe/Rome) now carries the right offset; set_config reports the zone the way SET does; jsonb renders Unicode escapes as PostgreSQL does; COPY inside a transaction leaves the session in it, fills omitted serial / default columns, and parses array literals; and an empty-multirange binary parameter with no declared type is typed from the column it is inserted into. The remaining faker failures are all the documented 34-significant-digit numeric limit.

Fixed

• secantus-pgserver: binary result encoders for the datetime family, interval, ranges, multiranges, json/jsonb and their arrays, and empty arrays, each pinned to PostgreSQL 16’s bytes (test_binary_results_cover_every_faker_type).

• secantus-pgplan: timestamp_text_to_pg_micros handles wide years, the BC era and an explicit offset; time_to_pg_micros accepts 24:00; a timestamptz result is rendered in the session’s named zone (utc_text_in_zone); datetime array text is PostgreSQL’s, not a debug dump.

• secantus-pgplan: catalog_param_types infers an INSERT’s parameter type by column position when the statement has no column list (ColumnRef), so an untyped binary parameter is typed from its target column either way.

• secantus-pgserver: COPY inside a transaction keeps INTRANS; COPY with a column list fills the omitted columns from their defaults and sequences; COPY FROM parses array literals; set_config('TimeZone', ...) reports the new zone; jsonb text uses PostgreSQL’s Unicode escaping.

• vendored pgwire: put_cstring stops at the first NUL and ErrorResponse / NoticeResponse length their fields the same way, so an error message that quotes a binary parameter no longer drops the connection with “message contents do not agree with length”.

secantusd-pg reports the address it actually bound

The readiness line exists so a harness can wait for the server and then connect, but it echoed the address that was requested rather than the one the listener ended up on. That made 127.0.0.1:0 useless: the kernel picks the port and nothing told the caller which one, so callers had to probe for a free port themselves and pass it in — a race no caller can win, because the probe socket must be closed before the child can bind it.

Fixed

• secantusd-pg’s listening on … line now reports local_addr(), so 127.0.0.1:0 is a supported way to start the server and read back its port.

• The test harness now starts every secantusd-pg that way instead of guessing a port. Under pytest -n auto two workers could be handed the same port; the loser’s child exited, but a liveness probe fired in that window connected to the winner’s server, and the test then ran against another worker’s database until that worker shut it down — surfacing as server closed the connection unexpectedly on an opening CREATE TABLE.

The Rust PostgreSQL server enforces NOT NULL, CHECK and FOREIGN KEY

CREATE TABLE on the Rust PostgreSQL server now records its NOT NULL, CHECK and FOREIGN KEY constraints in the shared catalog and enforces them on every INSERT, UPDATE and DELETE, answering what PostgreSQL 16 answers: 23502 with the failing column, 23514 with the constraint’s name, 23503 on the child side and — for NO ACTION / CASCADE / SET NULL — the parent side, each with the Failing row contains (...) / Key (...)=(...) detail and the schema, table, column and constraint diagnostic fields a driver reads. A DEFERRABLE INITIALLY DEFERRED key is checked at COMMIT: the COMMIT reports the violation, the transaction rolls back and the connection is left idle, as it is on PostgreSQL. Unnamed constraints take PostgreSQL’s generated names (<table>_<column>_check, <table>_check1, <table>_<column>_fkey), and a CHECK naming a missing column or a foreign key without a unique target is refused at CREATE (42703, 42830).

Added

• Rust PostgreSQL server: NOT NULL (23502), CHECK (23514) and FOREIGN KEY (23503, immediate and INITIALLY DEFERRED to COMMIT) enforcement with PostgreSQL’s messages, details and diagnostic fields; ON DELETE CASCADE / SET NULL; temp tables report a pg_temp schema in diagnostics.

• secantus-pgcatalog: TableDef carries temp, check_constraints and foreign_keys in the Python server’s document shape.

• Rust PostgreSQL server: CancelRequest interrupts the running statement (57014 canceling statement due to user request, connection left idle); pg_stat_activity shows each backend’s state and running query; and idle_in_transaction_session_timeout / idle_session_timeout are validated, rendered (60000 shows as 1min) and enforced — the session ends with FATAL 25P03 / 57P05 and the connection closes, as on PostgreSQL 16.

• Rust PostgreSQL server: databases. The startup packet’s dbname is checked before AuthenticationOk — an unknown name is FATAL 3D000 database "x" does not exist (a failed connect in libpq, not a failed first query) and template0 is 55000 — against a registry of postgres / template1 / template0, the daemon’s --database NAME flags and CREATE DATABASE; DROP DATABASE (with PostgreSQL’s 25001, 42P04, 3D000 / IF EXISTS notice, 55006 and 42809) drops the data too; pg_database lists the set and current_database() / current_catalog name the connected one.

• Rust PostgreSQL server: GROUP BY over an expression (length(data), col is null, n + 1) or a select-list position (GROUP BY 1, 2), with the key matched to the projected expression by structure and to ORDER BY by position, alias or expression; IS [NOT] NULL as a value, including PostgreSQL’s row rule (row(1, null) is neither null nor not null); and a FROM-less select unnest(array) as one row per element in a column named unnest of the element type.

• Rust PostgreSQL server: a table is also its row type, as on PostgreSQL — CREATE TABLE rtt (...) registers the composite rtt, so '(1,foo)'::rtt, '{"(1,foo)"}'::rtt[], row(1,'x')::rtt, pg_typeof, to_regtype('rtt') and the pg_type / pg_attribute rows psycopg’s TypeInfo.fetch reads all see it; DROP TABLE removes it; DROP TYPE rtt is 2BP01 cannot drop type rtt because table rtt requires it with the You can drop table rtt instead. hint; and a CREATE TYPE / CREATE TABLE over an existing type or relation is 42710 type "x" already exists (with PostgreSQL’s hint when a relation collides with a type) or 42P07 relation "x" already exists. to_regtype also resolves a user type’s array — mood[], rtt[] or the internal _rtt spelling — which was NULL for every enum and composite.

• Rust PostgreSQL server: the aclitem type (oid 1033, array 1034) with PostgreSQL 16’s parser and renderer — grantee=privileges/grantor with the group / user key words, quoted names, * grant options and the canonical arwdDxtXUCTcsA order — and its errors (role "x" does not exist, invalid mode character, unrecognized key word with its hint, extra garbage at the end of the ACL specification). The roles it knows are the session user (there is no role catalog); an omitted grantor defaults to it with PostgreSQL’s defaulting grantor to user ID 10 WARNING, and a planner WARNING now reaches the client as a NoticeResponse.

• Rust PostgreSQL server: SET standard_conforming_strings TO off is honoured and reported. A plain '...' literal is then read with the pre-9.1 backslash escapes ('a\'b', 'p\nq', '\\'), E'...', B'...', X'...' and $$...$$ keep their own rules, a statement is read under the setting in force when it is prepared, and U&'...' is refused as PostgreSQL refuses it (0A000 unsafe use of string constant with Unicode escapes). escape_string_warning (default on) raises PostgreSQL’s 22P06 nonstandard use of \' / \\ / escape in a string literal WARNING per literal, with its hint and position. Both GUCs are validated as Booleans in every spelling PostgreSQL accepts (22023 parameter "x" requires a Boolean value), and the standard_conforming_strings value is sent as a ParameterStatus — which is what libpq’s PQescapeString follows — because the server now obeys it.

Fixed

• Rust PostgreSQL server: a long statement no longer stalls every other connection — execution runs off the async runtime’s I/O thread, so a cancel request (or any other client) is served while pg_sleep runs.

• Rust PostgreSQL server: every extended-protocol statement between two Syncs runs in one transaction that the Sync commits, as on PostgreSQL — an error in a pipeline now rolls back the earlier statements of its group (libpq’s PIPELINE_ABORTED batch is all-or-nothing), BEGIN inside a group turns it into a block, and DECLARE in a group is still 25P01.

• Rust PostgreSQL server: a statement prepared without parameter types (libpq PQprepare with nParams = 0) sizes its parameters from the lexer, so insert into t values ($1, $2) no longer fails with there is no parameter $1 — pg_query’s node walk skips a VALUES list.

• Rust PostgreSQL server: the first DDL on a fresh store no longer fails a second connection with a WiredTiger WriteConflict. The __sql_* catalog collections were registered lazily inside whichever block first needed one, and a block that began before the row landed could not see it; they are now created before a transaction handle opens. The enum / composite type oid counter is likewise advanced outside the block, like PostgreSQL’s OID counter, so two open CREATE TYPE blocks no longer conflict (a rolled-back block just skips an oid).

• Rust PostgreSQL server: an assignment with no assignment cast is refused as PostgreSQL refuses it — a text / varchar-typed expression (an explicit cast, or a parameter the client declared, which is how psycopg sends a binary-format string) into a jsonb, integer, date, … column, or boolean into an integer column, is 42804 column "data" is of type jsonb but expression is of type text with the You will need to rewrite or cast the expression. hint, on INSERT and UPDATE. Before this the value was coerced through the column’s parser and stored.

• Rust PostgreSQL server: the TimeZone / DateStyle ParameterStatus sent at startup now carries the session’s values (UTC, ISO, MDY) rather than the wire library’s defaults, so what a client caches at connect is what SHOW reports.

• Rust PostgreSQL server: a record literal of the wrong width reports PostgreSQL’s message and detail — 22P02 malformed record literal: "(1)" with Too few columns. / Too many columns., and row(1)::rtt is 42846 cannot cast type record to rtt with Input has too few columns. / Input has too many columns.; a planner error’s Detail: line now travels in the error’s detail field rather than its message.

• Rust PostgreSQL server: a syntax error’s message is PostgreSQL’s alone — syntax error at or near "selct", unterminated quoted string at or near "'q" — without the Error splitting: / Invalid statement: label the libpg_query Rust binding prefixes it with, which reached the client’s message_primary on every syntax error.

Datetime arithmetic and special values on the Rust PostgreSQL server

The Rust secantusd-pg server now speaks the everyday datetime idioms a PostgreSQL client reaches for. date + int, date - date, timestamp + interval, interval + interval and interval * n all evaluate — and, just as importantly, describe their result column with PostgreSQL’s own type, so a client that picks its loader from the row description (psycopg does) decodes a timestamp + interval as a timestamp rather than as text. The epoch special literal is accepted on date and timestamptz alongside the timestamp it already handled, and 24:00:00 is recognised as PostgreSQL’s valid end-of-day time.

Out-of-range results are rendered in PostgreSQL’s own text — a year past 9999, or the BC era — so the client’s own loader is what rejects a value it cannot hold, exactly as it does against a real server. This closes 38 of the remaining cases in psycopg’s vendored test_datetime.py suite; the only ones left need a full IANA time-zone database (named-zone DST result loading), which stays out of scope.

Added

• crates/secantus-pgplan: date +/- int, int + date, and date - date (→ int4) arithmetic; the epoch literal on date and timestamptz; and 24:00:00 as a valid end-of-day time.

Fixed

• crates/secantus-pgplan: datetime arithmetic (timestamp + interval, interval + interval, interval * n, …) is now typed from its operands, so the result column is described with the correct OID even at DESCRIBE time when every value is NULL — previously it was described as text/int4 and clients decoded it wrongly, and interval / timestamp / date overflow was never surfaced to the client’s loader.

The Rust PostgreSQL server runs DO blocks and speaks more of the wire

psycopg’s own test suite exercises corners of the protocol that no application reaches on purpose: a DO $$ ... $$ block that raises a notice, a connection set to latin9 reading an error message with a euro sign in it, a ROW(...) fetched in binary, an enum array sent as bytes. Each of those now answers the way PostgreSQL 16 does, measured against it. The largest piece is a small executor for inline plpgsql blocks — RAISE at every level with USING options and % formatting, PERFORM, EXECUTE, NULL and plain SQL — with the context, position and internal_query fields a client sees on failure. It is deliberately a subset: variables, control flow and EXCEPTION handlers are still refused with 0A000 rather than half-run.

Around it, a cluster of smaller fidelity fixes: an anonymous record’s binary form now carries each field’s real type (an untyped literal is unknown, a cast one its cast type), quote_ident and regtype quote reserved keywords, format() arrives with its %s / %I / %L specifiers, a WHERE clause works over generate_series, inet values drop their host mask on COPY TO, and oid, oid[] and enum-array parameters resolve to their own types in both text and binary.

Fixed

• secantus-pgserver: DO [LANGUAGE plpgsql] $$ ... $$ inline blocks (plpgsql_do.rs parser + do_block.rs executor). RAISE notices reach the client as NoticeResponse with PL/pgSQL function inline_code_block line N at RAISE context; RAISE EXCEPTION carries P0001 / a named condition / an errcode, plus detail / hint / column / constraint / datatype / table / schema; an error inside PERFORM stacks the SQL statement under the block frame, and one inside EXECUTE carries the executed text in internal_query / internal_position; a bad body is 42601 positioned inside the statement; an unknown condition name is 42704 with the compilation context; LANGUAGE sql is 0A000.

• secantus-pgserver / pgwire: every error and notice now carries the V (severity_nonlocalized) field; 42P01 for an undefined table carries the P position of the relation’s first mention; error messages are re-encoded in the client’s encoding (latin9 reads bad €).

• secantus-pgplan: ROW(...) records each field’s type, so a binary anonymous-record result is byte-identical to PostgreSQL (unknown 705 for a bare literal, the cast type otherwise); -'NaN'::numeric and the infinities negate; regtype and quote_ident quote reserved keywords ("order"); format() with %s / %I / %L / %% / positional %n$, NULL handling, and the 22023 / 22004 errors (with PostgreSQL’s hint); concat(true) is t; a WHERE over generate_series — constant or on the series column — filters the rows, and a non-boolean constant is 42804.

• secantus-pgserver: COPY ... TO STDOUT renders inet without a /32 / /128 host mask (::ffff:102:300/128 → ::ffff:1.2.3.0) while cidr keeps its mask; COPY FROM stores numeric text at its written scale.

• secantus-pgserver: an Oid parameter keeps type 26 (pg_typeof is oid), an oid[] sent as text comes back binary as 1028; a parameter typed with a user enum’s oid is checked against its labels (22P02 invalid input value for enum ... with the unnamed portal parameter $1 context); enum ARRAY parameters parse in text and binary, and a binary enum-array result carries the array oid with each element as its label.

Seven session opens per statement, for catalogs that were already there

Profiling the Rust PostgreSQL server found that a SELECT 1 — a statement that touches no table at all — cost fifty-five microseconds more than an empty protocol round trip, against PostgreSQL’s nine. The call tree put most of that somewhere unexpected: before running anything, every statement opened seven fresh WiredTiger sessions, each with a cursor to open, search and close, to re-confirm that the seven catalog collections still existed. They are created once and never dropped except with the whole database.

The check itself has to stay. One catalog write — the row recording a new table — has no guard of its own and relies on that blanket pass; removing it would turn a CREATE TABLE in a fresh database into a storage error rather than a no-op. What can go is the repetition. A connection that has established a collection exists cannot subsequently be wrong about it, so the verdict is now remembered for the life of the connection.

Deliberately per connection rather than process-wide: the test suite builds many servers over many storage paths that all call their database postgres, and a process-wide verdict would cheerfully report a collection as present in a store that had never seen one.

A SELECT 1 now costs 55.9 microseconds where it cost 76.3, and a single-row primary-key read 68.0 where it cost 89.2 — about a fifth off every statement, with the conformance suite unchanged at 5542 passing.

Changed

• ensure_collection on the Rust PostgreSQL server caches its verdict per connection instead of re-probing storage before every statement.

Rust pgserver: extensions (hstore, postgis) and a role catalog

The Rust PG server now takes CREATE EXTENSION for the two extensions the psycopg test-suite reaches for. hstore brings the key/value map type with PostgreSQL’s operator set — ->, ?, ?|, ?&, @>, <@, ||, - — and postgis brings a geometry type whose EWKB round-trips through ST_GeomFromEWKB / ST_AsEWKB and shapely. Each extension registers its type under the oid a client discovers through pg_type / pg_extension, and DROP EXTENSION removes it again; forty-three gauge tests that skipped on “extension unavailable” now run.

Roles landed alongside: CREATE / ALTER / DROP ROLE (and the USER spellings) keep a cluster-wide catalog. pg_roles and pg_user mask every password as ********, pg_authid carries the SCRAM-SHA-256 verifier — the one derived from a plaintext PASSWORD (4096 iterations, a fresh 16-byte salt, exactly what password_encryption = scram-sha-256 produces) or the one libpq’s PQchangePassword sends ready-made — and every error and notice was measured on PostgreSQL 16: 42710, 42704 and the IF EXISTS notice, 2BP01 for the bootstrap superuser, 55006 for the session’s own user, 22007 for a bad VALID UNTIL, the empty-password notice, and a multi-name DROP ROLE that is all or nothing. Passwords are recorded, never checked: every connection is still trusted, which is what keeps the gauges connecting as user=postgres with no password.

Added

• secantus-pgplan / secantus-pgserver: CREATE EXTENSION [IF NOT EXISTS] / DROP EXTENSION [IF EXISTS] for hstore and postgis, the pg_extension catalog, the hstore type and operators (hstore.rs), the geometry type over EWKB (geometry.rs).

• secantus-pgserver: CREATE / ALTER / DROP ROLE|USER with a persisted role catalog; pg_roles, pg_authid, pg_user virtual tables; SCRAM-SHA-256 verifier derivation via secantus-auth.

Fixed

• secantus-pgplan: a stored timestamptz column’s ::text dropped the offset and rendered in UTC regardless of the session zone — the cast chain now carries the column’s declared type, so t::text renders 2026-01-01 13:00:00+01 under Europe/Berlin as PostgreSQL does.

• secantus-pgplan: ALTER ROLE ... PASSWORD $1 is the 42601 syntax error PostgreSQL raises (its grammar takes only a string constant there) instead of silently clearing the password.

The Rust PostgreSQL server delivers notifications, terminates backends, and runs the whole psycopg suite

The psycopg gauge measured a third of psycopg’s test suite against the Rust PostgreSQL server: the synchronous modules, with the asynchronous twins, notifications, pipelining, two-phase commit, concurrency and the libpq-level tests left for “a later lane”. This release runs all of them, and the server was fixed until the widened gauge is clean.

The largest piece is LISTEN / NOTIFY. A notification is delivered the way PostgreSQL delivers it: to every listening connection, the sender included, before the statement’s ReadyForQuery outside a transaction block, at COMMIT inside one (queued, deduplicated, dropped on ROLLBACK), and unprompted to a connection sitting idle. That idle path is what pg_terminate_backend needed too — it used to set a flag the victim noticed on its next statement, so a session asleep in pg_sleep or idle never went away. Both it and the new pg_cancel_backend now reach the victim within milliseconds, whether it is running a statement or waiting for one.

The rest is throughput and shape. One psycopg test timed out because every statement re-read the type catalog from storage, re-looked-up its table and re-parsed its SQL up to four times; a process-wide catalog cache and a parse memo bring a trivial statement from 0.86 ms to under 0.2 ms and a 20,000-row executemany from 15 s to 3.4 s. CREATE TABLE AS, functions in FROM, aggregates over expressions, pg_tables, now() and its siblings, and the extended protocol’s Describe replies for undeclared parameters all landed because psycopg’s tests asked for them, each measured against PostgreSQL 16.

Added

• LISTEN / UNLISTEN [*] / NOTIFY / pg_notify() / pg_listening_channels() with asynchronous cross-connection delivery on the Rust PG server. The vendored pgwire gains an idle_event hook the connection loop races against the next frontend message, and a before_ready_for_query hook.

• pg_cancel_backend(pid); pg_terminate_backend(pid) now ends an idle or sleeping session immediately, and an unknown pid answers false with PostgreSQL’s 01000 warning.

• CREATE [TEMP] TABLE [IF NOT EXISTS] t [(cols)] AS query [WITH [NO] DATA].

• A function call as the row source in FROM (select 'ok' from pg_sleep(0.5), select * from pg_listening_channels()).

• Aggregates over expressions (max(length(data)), sum(col1 * 2)).

• The pg_tables catalog view.

• now() / transaction_timestamp() / statement_timestamp() / clock_timestamp() and current_timestamp, typed timestamptz and rendered with the session-zone offset.

• A process-wide catalog cache (type-catalog documents and table lookups, versioned by every catalog-changing statement and transaction control) and a memo of pg_query parse trees and parameter counts.

• The psycopg gauge now includes every test_*_async.py twin, test_notify, test_pipeline, test_concurrency, test_tpc, test_xid, test_conninfo_attempts, test_waiting, test_module and tests/pq, with a per-platform pytest marker filter (MARKER_EXPR) that excludes psycopg’s proxy and timing markers on macOS as psycopg’s own CI does. Three tests are deselected with their reasons recorded in include_paths.py: one reads a foreign libpq’s PGconn and two connect to RFC 5737 unroutable addresses and depend on the host network timing out rather than answering at once.

Fixed

• Describe of a prepared statement reports each undeclared parameter’s type from the statement instead of unknown, and integer arithmetic over parameters describes as the operands’ type.

• begin; declare cur ... in one simple-query batch keeps the transaction and the cursor open; a wire Close of a portal closes the DECLAREd cursor of that name; a missing portal is 34000, a missing statement 26000.

• now()::text rendered the wall clock with no zone suffix; it now carries the session-zone offset like every other timestamptz.

• current_timestamp inside an expression (current_timestamp::text) was refused; it evaluates.

• Per-statement cost grew with every table the store had ever held: each statement re-decoded every row type from BSON for the planner and again per described column, so a used store ran select 1 twice as slowly as a fresh one and test_type_error_shadow overran its 20 s budget late in the gauge. The type catalog is now shared by reference, the planner’s tables are published once per thread per catalog version, and a described column decodes only the one composite it names.

• password_encryption reports scram-sha-256; ALTER ROLE of a role other than the session user is 42704.

The Rust PostgreSQL server passes psycopg’s prepared-statement, uuid and string suites

psycopg’s tests/test_prepared.py, tests/types/test_uuid.py and tests/types/test_string.py went from 45 failures to none against the Rust PG server (secantusd-pg), every change measured against PostgreSQL 16 first. pg_prepared_statements now lists the connection’s named protocol-level statements — statement text, prepare_time, parameter_types and result_types as regtype names, and the generic / custom plan counts psycopg’s tests read — and a protocol Close, DEALLOCATE <name> or DEALLOCATE ALL removes the row. A COPY TO STDOUT that fails mid-stream no longer follows its error with a CopyFail, which had left the client believing a COPY was still in progress (“you cannot mix COPY with other operations”) for the rest of the connection.

uuid_in accepts every spelling PostgreSQL does (braces, upper case, a hyphen after any group of four hex digits), a text parameter bound to a $n::uuid is canonicalised, and uuid, bytea, inet, cidr and their arrays are sent in binary when the client asks for it — psycopg decodes a whole row with the first column’s format, so a single text column in an otherwise binary row was undecodable. A NUL byte inside a binary text parameter is the 22021 PostgreSQL gives it, and the any-typed builtins (concat, concat_ws, format, num_nulls, json_build_*, …) refuse an untyped parameter with 42P18, naming the parameter PostgreSQL names. UPDATE … SET col = <expression over the row> — num = num * 2, s = upper(s), num = coalesce(num, 0) — evaluates per matched row, and ROLLBACK TO a savepoint on a store with no committed tables now undoes a CREATE TYPE issued after it.

Fixed

• crates/secantus-pgserver: a pg_prepared_statements registry of named Parse statements (name, statement, prepare_time timestamptz, parameter_types / result_types regtype[], from_sql, generic_plans, custom_plans), maintained by on_parse / on_close, DEALLOCATE <name> (26000 when missing) and DEALLOCATE ALL; NOTIFY completes with its tag.

• crates/secantus-pgserver: Describe of a zero-oid Parse sizes the parameter list from the highest $n in the text, so select $1::uuid describes instead of “there is no parameter $1”.

• crates/vendor/pgwire: a COPY OUT / COPY BOTH handler error is returned as the error alone — no trailing CopyFail (local patch in api/copy.rs).

• crates/secantus-pgplan: parse_uuid accepts a hyphen after any group of four hex digits and the brace form; uuid_to_wire for the binary encoding; catalog_param_types / max_param_number / static_text_type for the registry; refuse_untyped_any_args (42P18, skipping format’s and concat_ws’s leading text argument).

• crates/secantus-pgserver: uuid, bytea, inet, cidr and their array types are binary-encodable; an inet[] / cidr[] in text renders each element through inet_out (a host address drops its full mask); a text parameter for a uuid target casts through uuid_in; a NUL byte in a binary text-family parameter is 22021.

• crates/secantus-pgplan / -pgserver: UPDATE … SET values that read the row are planned as row expressions (Update::set_exprs) and evaluated per matched row before the first write (update_row_sets).

• crates/secantus-storage: collection_exists reads on the active user transaction’s session, so a savepoint restore sees a collection the same transaction created.

The Rust PostgreSQL server names each result column’s source table

A RowDescription now says where each column came from: the relation oid and 1-based attribute number libpq exposes as PQftable / PQftablecol, through an alias, a two-table FROM, a JOIN, SELECT * and INSERT ... RETURNING, over the simple and the extended protocol alike. A computed column reports 0 / 0, exactly as PostgreSQL 16 does. The relation oid is the one the table’s row type already carried in pg_type and pg_attribute.attrelid, so the three agree.

The regclass type arrives with it: 't1'::regclass resolves a relation name to that oid (unquoted parts fold to lower case, quoted ones keep theirs, a public. / pg_temp. / pg_catalog. prefix is honoured, the fixed catalog relations such as pg_class answer their PostgreSQL oids), renders as the name, casts to oid / int / text, and refuses what PostgreSQL refuses with the same codes – 42P01 for an unknown relation, 42602 for a malformed name, 42846 for a cast to regtype. A comma-separated FROM t1, t2 plans as a CROSS join, and a select list may repeat an output name.

Added

• crates/secantus-pgplan: the regclass type (oid 2205) – resolution against the session’s published relations, text rendering, casts to and from it, and PostgreSQL’s error surface; comma-FROM cross joins; constant targets in a join’s select list.

• crates/secantus-pgserver: RowDescription ftable / ftablecol for pass-through base-table columns; each table’s columns carry their source from the catalog lookup and from CREATE TABLE within the transaction.

Fixed

• crates/secantus-pgserver: two join outputs with the same name (select 'a'::regclass::oid, 'b'::regclass::oid from t1, t2) no longer collapse to one value.

Shell types and base types on the Rust PostgreSQL server

CREATE TYPE "a-b"; used to be refused outright (DefineStmt is not supported yet), which closed the door on the sequence psycopg’s own suite uses to build a custom base type: a shell type, two LANGUAGE internal I/O functions that name it, and the full CREATE TYPE "a-b" (input=invin, output=invout, like=text) that completes it. The Rust server now runs that sequence the way PostgreSQL 16 does. A shell is a pg_type row with no array type that nothing can be cast to (type "a-b" is only a shell), CREATE FUNCTION ... LANGUAGE internal registers the wrapper with PostgreSQL’s notices for a shell argument or return type (and creates a shell for an unknown return type), and the full CREATE TYPE checks its shell and both I/O functions — existence, exact signature, return type — with PostgreSQL’s messages before completing the type.

A completed type carries its values as text: 'hello-inv'::"a-b" and '{hello-inv}'::"a-b"[] come back described with the type’s own oid and its typarray, TypeInfo.fetch finds both, and a name that needs quoting (€, order, foo bar, FooBar) renders quoted through regtype. DROP TYPE and DROP FUNCTION know the dependency between a type and its I/O functions: RESTRICT refuses with 2BP01 and PostgreSQL’s per-dependent DETAIL lines, CASCADE drops them with the drop cascades to ... notice, and both roll back inside a transaction. DROP TYPE IF EXISTS on a missing type is now the does not exist, skipping notice rather than silence.

Added

• secantus-pgplan: plans DefineStmt of kind TYPE (shell and full forms), CreateFunctionStmt (LANGUAGE internal only) and DROP FUNCTION; DropType carries CASCADE; a base type registry beside the enum / composite / range ones, with a defined type casting text to itself and a shell refused as is only a shell.

• secantus-pgserver: the __sql_base_types__ catalog (oid band 71000, typarray = oid + 100000), CREATE FUNCTION rows in the shared __sql_functions__ shape, pg_type rows for shells and base types, and the dependency-aware DROP TYPE / DROP FUNCTION with PostgreSQL 16’s errors and notices. A table column typed as a shell is refused.

• psycopg gauge: test_sql.py::TestLiteral::test_invalid_name passes for all five spellings.

The Rust PostgreSQL server prepares transactions

Two-phase commit is how a transaction manager coordinates one commit across several databases: each participant is asked to PREPARE TRANSACTION 'gid', which must make the work durable without making it visible, and only once every participant has said yes does the manager COMMIT PREPARED 'gid' on each of them — from whichever connection it happens to hold at the time, possibly after the original one has gone, possibly after the server has been restarted in between. The Rust PostgreSQL server now does all of that. PREPARE TRANSACTION records the block’s write set durably and ends the block; the writes stay invisible to everyone; COMMIT PREPARED and ROLLBACK PREPARED resolve the transaction from any connection, and a prepared transaction found on disk at startup — data and DDL alike — is listed in pg_prepared_xacts and replayed when it is committed. max_prepared_transactions reports 100, so psycopg’s two-phase-commit suite, which had skipped itself against the server, now runs: 38 tests pass.

The error surface is PostgreSQL 16’s, each case measured: a PREPARE outside a block is a warning and a ROLLBACK, every failed PREPARE ends the block, a duplicate identifier is 42710, one of 200 bytes or more is 22023, a block that created a temporary table, opened a WITH HOLD cursor or used LISTEN / NOTIFY cannot be prepared (0A000), and COMMIT PREPARED inside a block is 25001. TRUNCATE landed on the way, because psycopg’s fixture uses it: TRUNCATE [TABLE] a, b [RESTART IDENTITY] [CASCADE], with PostgreSQL’s refusal of a foreign-key parent (detail and hint included) and its per-table cascade notices.

Added

• PREPARE TRANSACTION / COMMIT PREPARED / ROLLBACK PREPARED and the pg_prepared_xacts view on the Rust PG server; max_prepared_transactions is 100. secantus-storage gains a secantus_prepared_xacts table, prepare_user_transaction / commit_prepared / rollback_prepared / list_prepared_xacts, and a replay of a recorded write set for a transaction prepared before a restart.

• TRUNCATE [TABLE] name [, ...] [RESTART IDENTITY] [CASCADE].

Fixed

• Storage::outside_user_transaction (the oid counter a CREATE TABLE mints mid-block) let its inner autocommit statement drain the enclosing block’s parked oplog seq ranges — and, in async oplog mode, clear its buffered entries — so the block’s earlier writes were lost to the in-flight window and to the prepared write set. It now stashes and restores the thread’s oplog bookkeeping around the bookkeeping call.

The Rust PostgreSQL server keeps numeric exact past 34 digits

PostgreSQL’s numeric is arbitrary precision; the Rust PostgreSQL server stored it as Decimal128, which holds 34 significant digits and an exponent no wider than ±6144. A value beyond that was refused with a 22003 where PostgreSQL simply returns it — a loud refusal, but psycopg’s exhaustive numeric round-trip tests, and any application that keeps a 40-digit key or a 1000-digit constant, could not run at all.

Wide values now persist as their canonical PostgreSQL text alongside a byte-sortable key, and everything that fits Decimal128 stays Decimal128. The two forms compare and sort by value — 1.50 still ties 1.5, a 40-digit value still lands after a 35-digit one, and NaN takes PostgreSQL’s place above infinity — in every WHERE operator, in ORDER BY, through a numeric PRIMARY KEY, and inside sum / min / max. Arithmetic is exact at any width, with PostgreSQL’s division-scale rule, and a computed numeric column is described as numeric rather than as an integer. Every expectation was measured on PostgreSQL 16.

Fixed

• secantus-pgplan/src/numeric.rs (new): canonical-text parsing and rendering, the {__numeric, __numkey} wide representation, Decimal128 bracketing, numeric_filter WHERE lowering (Decimal128 arm + key arm + NaN arm), exact BigInt-backed + - * /, sum_numeric_texts, and a value comparison that ranks NaN above Infinity as PostgreSQL does.

• secantus-pgplan: sum(numeric) is typed numeric; a computed numeric column (n * 2) is numeric (oid 1700), not int4 — the client’s integer loader used to choke on 3.0.

• secantus-pgserver: text and binary encoders, array elements, record fields, parameter decoding and casts all accept and emit wide values; sum over numeric is exact; ORDER BY and every comparison use the value order; a numeric PRIMARY KEY rejects a duplicate that differs only in display scale (1e40 vs 1e40.0) with 23505, and resolves equality / range / UPDATE / DELETE by value through the _id index.

Composite values bound as parameters over the Rust PG server’s extended protocol

The Rust PostgreSQL server can now accept a user CREATE TYPE ... AS (...) composite bound as a query PARAMETER. When psycopg’s register_composite sends a composite value it puts the type’s OID in the Parse message and the value in the Bind — but pgwire’s StoredStatement::parse maps every parameter OID through Type::from_oid, which only knows builtins and returns None for a composite. The raw OID was gone before our parser ran, so a bound composite had no resolvable type and every such query failed with could not determine data type of parameter $1.

The fix vendors pgwire 0.40.7 into the tree (crates/vendor/pgwire, wired in through secantus-pgserver’s own [patch.crates-io]) with a single-field addition: StoredStatement::parameter_oids preserves the raw Parse OIDs alongside the mapped types. The vendored copy is byte-identical to upstream but for that one patch to src/api/stmt.rs; its licenses travel with it. The server then resolves a parameter’s raw OID against its own composite catalog, gives the parameter a declared type (so pg_typeof($1) answers the type name), and decodes the value into the same record BSON a '(..)'::type literal produces — from the TEXT (a,b) form and the binary RECORD form alike, recursing for a composite-typed field.

Added

• crates/vendor/pgwire: a locally-patched copy of pgwire 0.40.7 whose StoredStatement carries parameter_oids: Vec<u32> (the raw Parse OIDs), wired in via secantus-pgserver’s [patch.crates-io].

Fixed

• secantus-pgserver: a user composite bound as a parameter now resolves its type from the raw Parse OID and decodes to the correct record value, so register_composite round-trips (text and binary) instead of failing with could not determine data type of parameter $1.

php-library’s text-index failure reads as the declared gap it is

IndexInfoFunctionalTest::testIsText fails with the server’s own text indexes are not supported by SecantusDB. Text indexes are permanently out of scope, and the same gap was already declared for the node and pymongo gauges — php-library was the one left reading as an unexplained failure, which is what made its report look like it had something to chase.

Changed

• docs/validation-report-php-lib.md now carries Expected and Adjusted columns, and an “Expected failures” section naming the test and why it fails: 3089 passed / 0 failed / 1 expected — 99.9% plain, 100.0% adjusted.

  Both columns ship. The plain rate still counts the failure, so it stays visible; the adjusted one answers “how much of the conformable surface conforms”. Declaring a gap is not the same as hiding it, and the gauge is never told to skip the test.

A numeric path component over an array matched the wrong documents

mongod reads {"v.0": …} both ways and matches on either:

• the element at that index — and having spent the path step on the index, it does not re-apply the implicit array traversal, so {"v.0": 1} does not match {v: [[1, 2]]} even though 1 is inside v.0;

• the field of that name in each element, so {"v.0": 9} does match {v: [{"0": 9}]}.

Both servers had both halves wrong, in opposite directions — they applied membership after the index and never tried the field reading — so a query could both return documents mongod would not and miss ones it would, silently and with no error.

Measured against mongod 8.2.11 over 22 filters × a 14-document corpus: 11 of 22 diverged, now 0.

Fixed

• The matcher’s path resolver now produces both readings, and marks a value reached positionally so the implicit one-level traversal skips it. In Python that is a list subclass, so the value is still an array for $size, $type, $elemMatch and whole-array equality; in Rust it is a Cand carrying the flag beside the borrowed value.

• The Rust raw-BSON fast lane carries the same provenance. Without it the two lanes would answer the same query differently depending on which ran.

Combining the two readings with a plain OR would have been wrong: $ne and $nin mean no candidate matches, over the combined set, not either reading’s negation holds. The flag travels with each candidate for that reason.

Four probes were only ever asking half the question

Every differential probe here asks “does SecantusDB match mongod?” — and SecantusDB is two servers. Five probes only ever asked the Python one. That is how the aggregation stage corpus came to hide 219 Rust divergences until a PROBE_SERVER column was added to it last week.

tools/probes/_servers.py is now the shared probe_targets() helper, and update_operators, arg_types_documents, update_path_conflicts and findandmodify_shapes all compare both servers. It found 21 divergences on the Rust server, 0 on the Python one.

Fixed — the Rust server accepted malformed writes and reported success

• A non-document q, or a non-document/array u, on update / delete fell through every match arm: the statement applied nothing and answered ok. mongod refuses the command (14 for a bad filter, 9 for a bad update, 40414 for an absent one, and 14 for a non-document element inside a pipeline-form u). 12 shapes.

• findAndModify with remove alongside new or upsert, or with an update that is neither a document nor a pipeline, ran the delete where mongod refuses the command outright.

Fixed — reply shape

• A remove’s lastErrorObject carried updatedExisting, which mongod reports only for an update. Drivers read that field by field.

• findAndModify’s remove+update message said “both update and remove=true”; mongod says “both an update and remove=true”.

Fixed — a defer that should have been an answer

• $rename with the same source and target deferred, which on the standalone Rust server reports “a construct the Rust server does not support” for an ordinary bad argument. It now names it, as mongod does.

Two shapes remain, recorded in tasks/backlog.md: the Rust update path has no parse-time / execution-time distinction, so it omits mongod’s Plan executor error during update :: caused by :: wrapper. Code and message body already match.

Widening the probe corpus found thirteen crashes and six wrong values

tools/probes/agg_expressions.py had run tens of thousands of times against both servers. Its value list contained no infinity, no NaN, no signed zero, no numeric boundary, and none of MinKey / MaxKey / Binary / Timestamp / Regex / Code.

A value class that is absent is invisible in exactly the way a passing test is: it costs nothing and it proves nothing. Adding those classes took the sweep from 3,968 cases to 6,628 and immediately surfaced thirteen crash-class bugs — each an internal server error reachable from any query — plus six wrong values on a server the campaign had held at zero for days.

	Before	After
Crash-class bugs, Python server	13	0
Wrong values, Rust server	6	0
agg_expressions.py cases	3,968	6,628

The crashes, by root cause

Two causes, both closed at the class level rather than per instance:

• Python’s decimal contexts trapped InvalidOperation instead of producing a value. decimal128’s own answer for Inf + -Inf is NaN, and for the trig series on a non-finite operand it is the operator’s limit. Both _DEC128_CTX and _DEC_TRIG_CTX now trap nothing.

• math.ceil / math.trunc / datetime.fromtimestamp raise on values BSON accepts. $ceil of a double infinity, $trunc of NaN, and $toDate of Int64(2**63 - 1) (year 292278994, outside Python’s datetime) all escaped as internal errors where mongod answers.

Wrong values

• The accumulators did not unwrap a one-element array. {$sum: [[1]]} answered 0 where mongod answers 1 and {$max: [[1]]} answered [1] where mongod answers 1 — five shapes across $sum / $avg / $min / $max / $stdDevPop, on both engines. {$sum: [[1, 2]]} sums the inner array; {$sum: [[1], [2]]} has two operands, both arrays, both ignored.

• Integer rounding went through f64, so $trunc and $round lost 9223372036854775807 entirely. Both now compute in integers.

Rules probed, not derived

• NaN is never a domain error — it answers NaN for every trig operator in both numeric types, including the range-limited ones where -1 <= nan <= 1 is trivially false.

• A decimal infinity takes the operator’s limit, which is a table rather than a series: $atan is pi/2 to all 34 digits, $tanh is exactly 1, $cosh is Infinity from either side. So the Rust server can answer these exactly despite having no decimal transcendentals.

• Rounding up out of int64 is an error (51080), not a widening to double.

• A date outside int64 is 241, not a saturation — which is what Rust’s as i64 was silently doing for {$toDate: 1e308}.

• Python’s % is floor-based where Rust’s truncates. The identical n - (n % scale) truncates toward zero in Rust and away from it in Python, so $trunc of -12345 at place -1 differed by 10 between the engines.

A mongod bug, deliberately not reproduced

{$round: [-2147483648, -1]} answers positive 2147483646 on mongod — an exact int32 wrap of the correct -2147483650. The same overflow is handled three different ways: widened for a positive int32, wrapped for a negative one, and detected (51080) at int64 width. $trunc on the same input is correct.

Both servers answer -2147483650. Reproducing the wrap would mean writing arithmetic known to be wrong to match a defect, and it is recorded in tasks/backlog.md §7 with the arithmetic rather than chased.

Also

The probe’s clients now use datetime_conversion="DATETIME_AUTO". Without it the default codec raises InvalidBSON while decoding a legitimate out-of-range date, killing the probe rather than reporting a divergence — and it did so identically for mongod, so it is the client that cannot cope, not either server.

Projections come back in MongoDB’s field order, and refuse what MongoDB refuses

apply_projection sits on the read path of every find and had never been compared against MongoDB. A new probe found three divergences on its first run, all shared by both servers.

Fixed

• Projected fields came back in the wrong order when $slice or $elemMatch was involved. MongoDB emits _id first, then the document’s own key order — not the projection spec’s — with computed fields appended. Those two operators were applied after the plain inclusions, so their fields landed at the end. Field order is what a driver renders, and comparing documents for equality ignores it entirely, so nothing else could see this.

• {$elemMatch: {$gt: 2}} failed with “unknown top level operator: $gt” whenever the array held documents. A criterion of only $-operators is an element-value predicate — each element tested as a value — and the code branched on the element’s type instead of the criterion’s shape. The value predicate also suppresses the implicit one-level array traversal, so a nested array is not descended:

  array	criterion	result
  [1, 2, 3]	{$gt: 2}	[3]
  [1, [3, 4], 5]	{$gt: 2}	[5]
  [[1, 2], [3, 4]]	{$gt: 2}	field omitted
  [[1, 2], [3]]	{$size: 2}	[[1, 2]]

• A projection naming both a path and its ancestor was accepted and returned a truncated document. MongoDB refuses the pair, with the code depending on the order: {a: 1, "a.x": 1} is 31249 Path collision at a.x remaining portion x, and {"a.x": 1, a: 1} is 31250 Path collision at a. Siblings, a shared string prefix that isn’t a path component, and the same path twice remain legal.

• On the Rust server, a named projection error is no longer rewritten to a generic BadValue — only the positional cases whose exact code it cannot reproduce still fall back.

Added

• tools/probes/projection_results.py — 37 shapes, comparing field order as well as content. 0 divergent on both servers.

Measured against mongod 8.2.11.

Wrong-typed command arguments on the Python server: the message sweep

The wrong-type sweep that closed the Rust server compared error CODES only. Comparing MESSAGES over the same 685 shapes against mongod 8.2.11 found the Python server divergent on 409 of them, including 18 that answered internal server error — an int() or an iteration over a value whose type was never checked. This closes 311 of the code divergences and 31 of the message-only ones; createIndexes and collMod are the measured remainder.

Fixed

• A wrong-typed count.limit, count.skip, listCollections.cursor.batchSize or update.updates.arrayFilters crashed the command handler and answered 1 internal server error. All 18 crashing shapes now answer mongod’s code. count’s two slots are deliberately not symmetric: limit answers 2 limit value is not a valid number (and 9 Expected an integer for a non-integral double), skip answers 14 with the IDL expected-types list — which is what mongod does.

• A negative count limit was ignored where mongod counts its absolute value, so limit: -3 returned the whole collection instead of 3.

• Python class names leaked into error messages — 'ObjectId' for objectId, 'datetime' for date, 'bytes' for binData. Three partial copies of the type-name mapper have been collapsed into secantus.bsontypes, probed against mongod and matching the Rust engine’s existing vocabulary. This corrects the type name in roughly 90 message sites across every engine.

• Wrong-typed hint answered 2 invalid hint type: int instead of mongod’s 9 Hint must be a string or an object, on find / count / aggregate / update / delete. On the batch commands it is a command-level error, so nothing is written.

• Wrong-typed document options were silently accepted — collation, readConcern, validator, timeseries, filter and cursor across find / aggregate / update / delete / findAndModify / create / distinct / listCollections now answer 14 under mongod’s own IDL path.

• Aggregation stages reported a generic 14 for a non-document spec. Each now answers mongod’s own code and wording: $project 15969, $addFields 40272, $replaceRoot 40229, $bucket 40201, $bucketAuto 40240, $sample 28745, $geoNear 10065, $redact 17053 (a runtime executor error, with the wrapper), and $densify / $fill / $setWindowFields / $unionWith 9.

• Boolean options were ignored or misreported — find’s tailable, awaitData, returnKey, showRecordId and allowDiskUse, aggregate.allowDiskUse, ordered on all three write commands, and renameCollection.dropTarget. A string tailable used to reach the cursor machinery and answer “tailable cursor requested on non capped collection”.

Every wrong argument to $rand carried the wrong wrapper

$rand takes no arguments, and both ways of getting that wrong are PARSE errors on mongod, so they take the stage’s wrapper (Invalid $addFields :: caused by ::) rather than the executor’s. The pure engine already produced the right code and the right sentence for all of them — only the routing was wrong, on 45 shapes, which is 76% of every remaining message difference on this surface.

The rule has a shape worth stating, because a single “must be a document” check gets it wrong three ways (measured against mongod 8.2.11, 2026-09-05):

argument	mongod
{} or []	valid — returns a double
non-empty document or array	3040501 “$rand does not currently accept arguments”
any scalar	10065 “invalid parameter: expected an object ($rand)”

An empty array being accepted is the easy one to miss, and the two wrong arguments get two different codes for what reads as one mistake.

The Rust engine already had all of this in its parse-time scanner, correctly, from an earlier pass — this brings the pure engine to it.

With this, the probe’s message differences fall from 59 to 14, and the wrong-code set is byte-identical to before. What remains on this surface is a flat tail: 28 codes across 28 different operators with one shape each, and 14 messages likewise. No further rule-shaped win is visible.

Fixed

• aggregate.py: _expression_shape_problem classifies both $rand argument errors as parse errors.

Changed

• tests/test_expression_parse_time_wrappers.py: 10 more cases covering every measured $rand shape, including the two valid ones.

A read-path sweep: queries that returned the wrong documents

A 385-case differential sweep of find filters, projections and sorts against mongod 8.2.11 — comparing full result sets, not counts, so a missing document is visible rather than a wrong tally. It found 27 divergences on the Python server and 20 on the Rust one. This release fixes thirteen of them, in four families, and every one was a wrong answer rather than a wrong message.

A NaN range bound matched nothing. mongod’s comparison order treats NaN as equal to NaN — which is why find({x: NaN}) works — so an inclusive bound matches it: {$gte: NaN} and {$lte: NaN} return the NaN document while {$gt: NaN} and {$lt: NaN} return nothing. IEEE says every NaN comparison is false, so both servers returned nothing for all four.

Decimal128 could not be compared with an infinity. The numeric bridge bailed out whenever either operand was non-finite, and only NaN needed excluding — Decimal orders ±Infinity perfectly well. The bail-out left float > Decimal128 to raise TypeError, which the caller swallows into a silent no-match, so {x: {$gt: Decimal128("5")}} skipped a document holding Infinity and {x: {$lt: Infinity}} skipped one holding Decimal128("5"). $all had the same shape from a different cause: it compared elements with a bare ==, so {$all: [5]} missed a stored Decimal128("5") that $eq: 5 matched.

NaN had no place in the sort order. The comparator answered “not less” in both directions, so a sort left a NaN wherever the algorithm happened to put it — between 5.5 and Infinity in a measured case. mongod ranks it below every other number, -Infinity included. BinData was mis-ordered too: mongod compares by length first and then by bytes, so b"\x02" sorts before b"\x01\x02".

$type accepted eight aliases it could never match. On the Rust server javascript, minKey, maxKey, timestamp, undefined, symbol, dbPointer and javascriptWithScope all validated as arguments and then matched no document, because the type table had no arm for them. Argument validation passing is what made the gap invisible.

Fixed

• secantus.query / secantus-core: $gte / $lte with a NaN bound match the NaN document; $gt / $lt still match nothing.

• secantus.query: the numeric bridge admits ±Infinity, so Decimal128 compares against them under every range operator instead of silently matching nothing.

• secantus.query: $all compares elements with the same numeric-bridging equality $eq uses.

• secantus.ordering: NaN sorts below every other number and ties with another NaN; BinData sorts by length, then bytes.

• secantus-core: $type matches the eight BSON types its alias table already accepted.

The test suite reclaims its own disk

Every test in this suite builds a real WiredTiger database — never a mock — so every run leaves a few gigabytes of per-test stores behind. pytest normally sweeps old runs itself, but it stops the moment a run dies without running its exit hooks: the dead run’s lock file makes pytest treat the directory as live for three days. The backlog then compounds, because the bigger it gets the longer each run’s exit-time cleanup takes, so more runs get killed mid-cleanup, each leaving another stale lock. One machine reached 241 directories and 391 GiB that way; another reached 48 GiB in a day.

invoke clean has been able to fix this for a while. The problem was that it only ran when somebody remembered, so the backlog kept coming back. The sweep now also runs automatically when a pytest session starts.

It deletes only directories whose owning pytest process is gone, decided from the PID in the lock file, and never the newest few. A live run is protected twice over: its directory is the newest, and its lock names a running process.

Added

• tests/conftest.py reaps abandoned pytest temp trees at session start. It runs on the xdist controller only, skips the byte-counting walk that invoke clean does for its summary line, and swallows every error — reclaiming disk must never fail a test run. Set SECANTUS_NO_TMP_REAP=1 to turn it off.

• _sweep_stale_pytest_tmp takes measure=False for callers that want the deletion without sizing every tree first.

Note

Deleting a passed test’s own tmp_path mid-session is a different thing and remains forbidden — it races WiredTiger’s background threads into WT_PANIC, and tests/conftest.py refuses to start under a retention policy that does it. This sweep only ever touches trees from runs that have already exited.

$redact returned data it exists to withhold — three defects, both servers

$redact is MongoDB’s content-based access-control stage: an expression decides, per document and per sub-document, whether to keep it, drop it, or recurse. Three separate defects made it hand back documents mongod withholds. All three were present on both servers, and all three were found by running the stage against mongod 8.2.11 rather than by reading the code — which had described the decisions as “sentinel strings”, the assumption the first two bugs rest on.

Fixed

• A stored string could impersonate a decision. The stage compared the evaluated result against the string "$$KEEP", so a document whose own field held that string satisfied the test: {"$redact": "$tag"} over caller-supplied content kept a document — with its secrets — that mongod refuses to keep. $$KEEP / $$PRUNE / $$DESCEND are now variables bound only while $redact’s expression is evaluated, and the stage dispatches on the bound marker, so no value carried in a document can be mistaken for a decision.

• Redaction skipped nested arrays. The descent walked documents and arrays-of-documents but passed a nested array through untouched, so a sub-document one array deeper — [[{level: 9}]] — was returned in full where mongod prunes it and leaves the emptied inner array in place.

• The decision names leaked outside $redact. {"$project": {"x": "$$KEEP"}} returned the marker string as ordinary data. mongod binds these three names only inside $redact and answers Use of undefined variable: KEEP (17276) anywhere else, which both servers now do.

• A non-decision result now answers mongod’s error. 17053 with its own wording, wrapped in Executor error during aggregate command on namespace: … :: caused by ::, and the offending value rendered mongod’s compact way — the Python server answered a generic TypeMismatch (14) with its own text, and the Rust server reported the stage as unsupported.

• An undefined variable takes the wrapper mongod gives it. Invalid $<stage> :: caused by :: inside $project / $addFields / $set, and a bare message in every other stage; the Python server applied the executor prefix to both.

Notes

• $redact was never missing: it worked on both servers for every valid pipeline, which is why only its error path had been noticed. The backlog entry calling it “unimplemented on the Rust server” was wrong and has been corrected.

• The 17053 value rendering is mongod’s compact Value::toString ({k: 1}, [1, "a"], a bare ObjectId, an ISO-8601 date) — a different renderer from the shell form other messages use ({ k: 1 }, ObjectId('…'), new Date(<ms>)). Both are mongod’s; they are not interchangeable.

• The Rust server had no executor wrapper at all. It now applies one where mongod does, via a standalone validator that names the error the engine could only signal as Fallback — the update::arith_type_error template, so Fallback itself is untouched.

• 21 cases added to tests/test_mongod_differential.py, so this is measured against a real mongod from now on rather than pinned to our own belief.

A regex is a value, not only a pattern — and $addToSet dedups by BSON equality

Continues the measure-against-mongod-8.2.11 sweep, taking the last Rust-engine capability defers. Two new standing probes cover the ground: tools/probes/regex_value_semantics.py (104 shapes) and tools/probes/addtoset_membership.py (30).

Three of the fixes are silent WRONG ANSWERS on the Python server, not errors — found while reproducing a Rust-only backlog item, which is the whole argument for running a probe against both servers rather than reading either one.

Fixed

• A bare regex filter never matched a stored regex. mongod matches /ab/i against a string by pattern and against a stored regex by equality, so find({v: /ab/i}) returns {v: /ab/i}. Both servers returned nothing. Equality is exact-pattern plus options-as-a-SET: /ab/im is /ab/mi, and /ab/i is not /ab/mi. $in, $nin and $regex carry the same rule.

• A regex was applied to JavaScript. bson.Code subclasses str, so find({v: /ab/}) matched {v: Code("ab")}. mongod never regex-matches code.

• $max / $min over two regexes never moved. bson.Regex defines no __lt__, so the comparison fell to a type-name fallback that reported every pair EQUAL — and both Rust arms had that accident written in as a comment citing what Python did. mongod orders by pattern, then option string.

• $eq with a regex operand errored on the Rust server for every document in the collection. On mongod $eq with a regex is equality only — the opposite of a bare regex, which matches — and a defer is an error on the standalone server.

• $addToSet of a bool, a document, or a Code errored on the Rust server. It deferred on the strength of a stale comment; py_eq had since grown the bool and Code rules, leaving only document field ORDER to add.

• A Code value made $set fail on the Rust server — the oplog update-diff walks every field through py_eq, which deferred on a JavaScript value paired with anything else.

• $pop’s error quoted the argument as written, so Decimal128("-0") read found: -0. mongod reports the coerced integer (1E+2 is found: 100).

Changed

• The regex option-character table is now shared (bsontypes.REGEX_OPTION_CHARS / regex_options_string) between the in-memory sort and the index-entry encoder, on both servers. Two functions disagreeing about a sort key is how an index comes to change the sort answer. No entryFormat bump: the index encoder was the half that was already right.

$$REMOVE is the missing value, not a marker of its own

Probed 9-for-9 against mongod 8.2.11: in every position — a projected field, an array element, a nested document, $ifNull, $type, $eq, $cond, $concat, $sum, a $group _id — $$REMOVE answers exactly what the equivalent absent field path answers. Both engines instead gave it a marker of its own, which then escaped the places that knew about it.

Fixed

• A crash. {"$addFields": {"arr": [1, "$$REMOVE", 2]}} put the marker object into the result, where bson.encode failed and the command returned internal server error (code 1). mongod returns [1, null, 2].

• $type: "$$REMOVE" answered "object" — the marker’s own Python type — where mongod says "missing".

• $concat with $$REMOVE raised 16702; mongod returns null.

• $ifNull: ["$$REMOVE", 9] omitted the field instead of returning 9.

• The Rust engine deferred the variable entirely, which on a server with no Python surfaced as a generic BadValue (2) for all 15 shapes probed.

$$REMOVE now follows the same two-position rule the engines already apply to an absent path: the missing marker in field-value position (so $project / $addFields omit the key), null as an operator argument.

Also fixed, found next door

• $setField with an absent path wrote a null where mongod removes the field. It evaluated its value in operator position, so only the literal $$REMOVE removed anything; {"value": "$nosuch"} set null. mongod treats both the same — probed.

• $setField rejected value: null outright ($setField requires field, input, value) because a present-but-null argument was tested with is None and read as absent. That is the one form that distinguishes write a null from remove the field.

Notes

The two names for the marker are now one: MISSING was already an alias of the $$REMOVE sentinel, which is what made the conflation easy to miss.

23 cases added to tests/test_mongod_differential.py, five of them twins that pair a $$REMOVE shape with its absent-path equivalent — the two must answer identically, which is the claim this change rests on.

$rename no longer rewrites paths MongoDB refuses

$rename will not move a field into or out of an array element, and it will not accept a positional token in either path. Both servers accepted several of those updates and applied them — and one of them, {$rename: {"items.$[].a": "items.$[].b"}}, was renaming every element of the array, which MongoDB does not do at all.

That last one had a test asserting it worked. The test drove our own server, the only server it could reach, so it recorded what the code did rather than what MongoDB does; it has been corrected along with the behaviour.

The two refusals differ in when MongoDB can decide them, and that changes how they arrive: a dynamic component ($, $[], $[id]) is a parse error, raised without looking at the document and sent bare, while a path that indexes into an array is discovered per document, arrives under Plan executor error during update :: caused by ::, and is skipped altogether when the source field is absent — because then the rename is simply a no-op.

Fixed

• A dynamic component in a $rename path is refused — The source field for $rename may not be dynamic: <path>, or its destination twin. Raised before the document is read, so an absent source still errors. Precedence, measured: source-dynamic before destination-dynamic, and both before an array-element path; the general No array filter found for identifier check comes first of all.

• The array-element messages name the field holding the array and carry MongoDB’s executor wrapper. deep.n.0.a reports 'n', not the whole path, and the _id in the message uses MongoDB’s value rendering — a string _id is quoted, an ObjectId is wrapped — where the Python server printed it raw.

• A $rename whose source does not resolve is a no-op again, so {"v.9.a": "q"} and {"v.0.zz": "q"} succeed rather than reporting a traverse failure.

• cannot use the part (…) to traverse the element was sent bare by the Python server. Every code-28 traverse failure carries the executor wrapper; the Rust server already did.

• The Rust server failed the whole update command when an arrayFilters identifier was missing, instead of recording one entry in writeErrors. A driver saw the wrong exception class, and an unordered batch lost the statements that had nothing wrong with them.

Measured against mongod 8.2.11 over 35 shapes: 0 divergent on both servers.

A rename change event now leads with to, as MongoDB’s does

MongoDB puts to first on a rename change event — before _id, the only event where the resume token is not the leading field. SecantusDB put it after ns.

This was measured earlier and deliberately left alone, on the reasoning that leading with to looked like an artifact of how MongoDB assembles that event and might not survive a version change. Re-measured against MongoDB 8.2.11, the version SecantusDB targets, it is identical — with and without showExpandedEvents. Stable across two major versions is a contract, not an artifact, so it is now replicated.

With this, the change-stream differential sweep is at zero divergences across all 41 cases and zero field-order differences, on both servers, against a live MongoDB 8.2.11.

A retryable write’s retry no longer replays the original attempt’s error

A writeConcernError describes one attempt: the write itself succeeded and only its durability acknowledgement failed. The retryable-write replay cache stored the reply verbatim — deliberately, so a replay would be byte-identical — which meant a driver’s retry was handed the very error it retried because of. The operation then surfaced as an error even though the write was safely applied.

Separately, the failpoint-configured writeConcernError was being embellished with a synthesised errmsg and codeName. Real mongod echoes the failpoint’s document verbatim, and the synthesised codeName was wrong anyway: it rendered 91 as Location91 where 91 is ShutdownInProgress.

Together these were the last genuine failure in the mongo-c-driver gauge, /command_monitoring/unified/writeConcernError, which now passes. The remaining C-gauge failures are the documented inherent ones (IPv6 listener, and server selection asserting a standalone/secondary that a single-node surrogate has not got).

Fixed

• The retryable-write replay drops writeConcernError and errorLabels, so a retry reports the write’s actual outcome. Byte-identical replay is right for the write result and wrong for the attempt’s own condition. Fixed on both the Python and Rust servers.

• A failpoint’s writeConcernError is echoed exactly as configured — {code: 91} stays {code: 91} — matching mongod. The Rust server was already correct here.

• A non-retryable write (no lsid/txnNumber) still reports its writeConcernError as before; only the replay is stripped.

The Rust server gains the MongoDB 8.0 features and an honest version

The Python server advertised MongoDB 8.2.11 and supported bulkWrite, sort on updateOne / replaceOne, and change-event nsType. The Rust server had none of them and still advertised 7.0, so which features you got depended on which server you started. It now has all three and advertises the same version.

The sequencing is the same as the Python side’s, and it is the point: the advertised version is a capability contract, so it moved only once the features it promises existed. Raising it first would have made drivers send bulkWrite and receive CommandNotFound.

Each operation in a bulkWrite runs through the ordinary insert / update / delete handler with the database rebound to that operation’s namespace, so bulk semantics cannot drift from single-write semantics — the same structure the Python implementation uses.

Verified against a live MongoDB 8.2.11: the twelve-shape bulkWrite differential agrees exactly, as it does for the Python server, and the pymongo conformance gauge run against the Rust server sits at 99.4% with the same five out-of-scope failures the Python server has.

Runtime aggregation errors on the Rust server report mongod’s codes

A pipeline that fails while processing documents — as opposed to failing its spec, which the command layer already validates — answered a single generic 2 BadValue on the Rust server, because the engine can only signal such a failure as “defer to Python” and the Rust server has no Python to defer to. Measured against mongod 8.2.11, six of seven probed cases were divergent.

Fixed

• $densify over a non-numeric, non-date field now answers 5733201 Densify field type must be numeric or a date.

• $bucket with a value outside every boundary and no default now answers 7158303. mongod implements $bucket over $switch, so it reports the $switch sentence under $bucket’s own code — reproduced.

• $switch with no matching branch and no default now answers 40066.

All three carry mongod’s executor wrapper (Executor error during aggregate command on namespace: <ns> :: caused by ::).

Known gaps

Three cases still answer the generic message, each needing machinery this pattern does not provide, and each recorded in tasks/remaining-work-plan.md:

• $replaceRoot with a scalar newRoot (40228), whose message quotes the input document pruned to the fields the expression reads — mongod runs dependency analysis before the stage.

• $arrayToObject (40386) and $concatArrays (28664), which are expression type errors rather than stage errors, and are the head of a per-operator long tail.

The Python server already matched mongod on all seven cases; this release changes the Rust server only.

The last of the deferred operand errors, and three renderings that look like one

The operand-error campaign took the Rust server’s agg_expressions sweep from 908 divergences to 117. This closes it to 32 — and the remaining 32 are two deliberate deferrals, not defers standing in for ordinary argument complaints. That class is gone.

	Campaign start	After the first pass	Now
agg_expressions.py, Rust server	902 code + 6 message	117 + 0	32 + 0
agg_expressions.py, Python server	50 code + 212 message	28 + 80	0 + 71
Wrong values, either server	0	0	0

The Python server now has zero code divergences on this sweep, down from 50.

Fixed

• The trigonometric domain errors (50989) across $acos / $asin / $atanh / $acosh / $sin / $cos / $tan, each with its own range string.

• $mergeObjects names the offending value, and flattens an evaluated array: a field path resolving to an array is the operand list, so {$mergeObjects: "$arr"} over [3, 1, 2] reports “input 3” rather than naming the whole array.

• Six arithmetic type guards, no two alike — $add and $multiply name the first offender, $divide and $mod name both operands, $subtract inverts them into can't $subtract X from Y, and $atan2 carries a different code per position (51044 / 51045).

• The unknown-argument family — eighteen operators, two sentences, eighteen codes that share nothing, plus the n-operator family and $median / $percentile’s IDL wording.

• $round and $trunc had no arity check at all: {$round: [3, 1, 2]} answered 3.

• $range was checking 64 bits, not 32 — it accepted a long and built a range of a trillion elements, answering [].

• $substr* and $strcasecmp coerce their operands rather than requiring a string, using the same rule $toLower does.

Rules a reasonable guess gets wrong

• NaN is not a domain error. {$tan: NaN} answers NaN, for all three of sin/cos/tan; only the infinities are refused. Both engines spelled the guard not isfinite(x) — which rejects NaN, and no probe corpus had asked.

• Three value renderings that look like one. $acos converts to double first (1.09951e+12), $range keeps the integer (1099511627776), and a Decimal128 keeps its own representation in both — 2.50 does not become 2.5.

• $subtract’s Date capitalisation is positional, not per-type: can't $subtract string from Date but can't $subtract date from int.

• A Decimal128 is numeric. Reporting a type error for it would be wrong; both engines defer on it for a different reason, and the guards had to be written to let it through.

The test that was right all along

Capitalising Date on both sides of $subtract was caught by test_arithmetic_date_semantics, which held the second-operand case. Three times this session an old expectation turned out to be a stale citation and the new measurement won; here both were right, for different positions. Re-probing rather than pattern-matching on “old test, new measurement” is the only thing that told them apart.

Also

The parity fuzz called _rust_eval bare and let a named error escape as an exception — it was written when the engine could only answer or defer, and naming mongod’s error is a third outcome it had no branch for. It now compares the named error against the pure engine, which is what the rest of the suite already did.

The wrong-typed-argument sweep, widened from codes to messages — 76 slots on the Rust server

The existing sweep compared error codes over 244 argument shapes, and both servers read 244/244 clean. Comparing messages as well, over 685 shapes, against mongod 8.2.11, found the Rust server diverging on 76 argument slots. Almost all of them were silently accepted: the wrong-typed value did not merely return the wrong status, it made the server do something other than what the caller asked and report success.

The probe’s reach is exactly its case list, which is the recurring lesson here — “244/244 clean” only ever meant “these 244 shapes are”.

Fixed

• Argument slots that took a wrong-typed value and ran anyway — hint, collation, readConcern, writeConcern, arrayFilters, ordered, bypassDocumentValidation, let, fields, sort, new, remove, filter, cursor / cursor.batchSize, validator, timeseries, capped / size / max, viewOn, changeStreamPreAndPostImages, partialFilterExpression, unique, sparse, expireAfterSeconds, dropTarget and tailable / awaitData / returnKey / showRecordId / allowDiskUse, across find, count, distinct, aggregate, insert, update, delete, findAndModify, create, collMod, createIndexes, dropIndexes, renameCollection, listCollections, listIndexes, getMore and killCursors. Each now answers mongod’s own code and wording.

• Thirteen aggregation stages whose spec type was never checked — $addFields / $set (40272), $project (15969), $replaceRoot / $replaceWith (40229), $facet (40169), $bucket (40201), $sortByCount (40147 / 40148 / 40149), $geoNear (10065), $graphLookup, $unionWith, $setWindowFields, $densify, $fill (9) and $sample (28745).

• aggregate with a non-array pipeline ran the whole collection through no stages and answered ok. Now A pipeline must be an array of objects (14).

• An empty updates / deletes batch answered ok:1 with n: 0, telling the caller a batch it never sent had been applied. Now InvalidLength, like insert.

• killCursors with a non-array cursors reported the named cursors killed while killing none.

• A hint naming no index was ignored by update / delete / findAndModify, so the write ran unhinted where mongod fails the command.

• $[identifier] with no matching arrayFilters entry was accepted when the target field was not an array — the engine’s walk returns early for a non-array, so {$set: {"a.$[e]": 1}} against {a: 1} wrote nothing and reported success. mongod decides this from the update document alone.

• dropIndexes by key pattern answered code 1 (InternalError) — the crash code — for a shape mongod handles routinely. It now drops the matching index, or answers IndexNotFound (27).

• findAndModify: {remove: 1} was rejected: remove accepts a bool or a number, and a nonzero number is true.

• InvalidLength is code 16, not 4 (4 is NoSuchKey) — on both servers. insert had carried the wrong code under a comment asserting that driver tests gate on “this exact code/codeName combo”; they gate on the codeName, and bulkWrite in the same codebase already answered 16.

Added

• tools/probes/arg_types_messages.py — the wide sweep, comparing (code, errmsg) rather than codes alone. It normalises the ORDER of mongod’s expected-type lists, which differ between 8.2.1 and 8.2.11 (a patch bump) and so pin a build rather than a behaviour.

• tests/test_rust_arg_types_sweep.py (42 tests) and nine Rust unit tests, pinning the per-slot asymmetries: count.limit rejects an explicit null while count.skip beside it accepts one, getMore.collection reads null as absent (40414) rather than wrong-typed, createIndexes’ unique accepts 1.5 and quotes the spec back with mongod’s own unclosed quote, and $densify capitalises “The” where $setWindowFields does not.

The Rust server validates command arguments

A wrong-typed command argument was accepted without complaint, and the server then did the wrong thing and reported success. createIndexes with a non-array indexes answered ok: 1 and created no index — a driver was told an index existed that did not. An update whose multi was a document updated a single document instead of all matches. A findAndModify whose upsert was an array skipped the upsert. A find with a non-numeric limit returned everything.

Measured against a real MongoDB across 87 argument shapes, the server diverged on 78 of them; it now matches on all 87, as the Python server already did.

A second group answered a generic BadValue where MongoDB has a specific code for the stage in question — $lookup, $group, $match, $sort, $limit, $skip, $count and $unwind each report their own code and message now. That one had a structural cause worth naming: the shared engine signals “this construct needs the Python implementation”, which the Python server honours by running it, and this server — having no Python — was reporting as a generic error. Stages it can identify are now named before that point is reached.

Both servers are now at 87 of 87 on this sweep.

The Rust server errored on case-insensitive queries, and had no collated order

tools/probes/collation_order.py was the one Rust-aware probe never swept against the Rust server. Sweeping it found two things, the first of them a hard failure on ordinary input.

A case-insensitive query or sort over any non-ASCII text errored.

find().sort("v", 1).collation({locale: "en", strength: 1})   over ["a","A","á","B","b"]
    mongod / python  ['a', 'A', 'á', 'B', 'b']
    rust             2 BadValue: an indexed value is of a type the
                     Rust server does not support

Not only accents — ß and 日 triggered it too, on match filters as well as sorts. normalize_index_bytes had if !s.is_ascii() { return None }, meaning “defer to the pure engine”; that is right on the Python server and wrong on the Rust one, which has no Python behind a defer. It now folds properly: NFKD, drop the combining marks, full case fold.

There were two such bails, not one. Fixing the index encoder left the query path — normalize, behind collation::equal / compare — still deferring, so collated equality and range queries on non-ASCII text kept answering 2 BadValue: query uses a construct the Rust server does not support after the sort was fixed. Both now share one fold helper so they cannot drift apart again.

The fold has to be case folding, not to_lowercase: folding maps ß to ss, and mongod sorts ["ß","s","t"] as ["s","ß","t"] — which only comes out right if ß compares as ss.

Collated ordering was not implemented at all — strings sorted by codepoint, so every accented word landed after z. collation.py’s sort_levels is now ported to collation::sort_level_bytes as a byte-comparable three-level key: the measured MARK_ORDER table (acute before grave, which is not codepoint order), the backwards reversal that makes cote < côte < coté, the caseFirst flip, and numericOrdering digit runs so a2 < a10.

The probe goes from dying at case 10 to 0 unexpected divergences of 19 — the 2 remaining are the documented Swedish / Danish locale gaps that need CLDR data. A wider three-way sweep across every collation option shape is 0 of 64 against mongod.

Nothing on disk changes, and the encoders are now split so that stays true by construction. sortkey::encode_value remains the INDEX encoder — single-level fold, byte-identical to Python’s encode_value(collation=), which is what the entries table holds and what the other server reads back. The three-level ordering key lives in a new encode_sort_value, used only by the in-memory sort-key builder. Collapsing both roles into encode_value is what broke test_collation_encoding_parity: the same function name means index bytes in Python and had come to mean the sort key in Rust. The probe’s own invariant confirms the result — index and non-index results agree on every case, so an index still changes speed and never results.

Mark filtering uses the predicate Python uses, per site. These are three different sets and the difference is measurable: _strip_accents filters general category Mn alone, while sort_levels filters on a nonzero canonical combining class. unicode_normalization::char::is_combining_mark is neither — it is true for all of M*, so using it for the accent strip dropped a Devanagari vowel sign (U+093E, category Mc) that the Python engine keeps. Measured 2026-09-07.

Adds direct unicode-normalization and unicode-properties dependencies to secantus-core; both were already in the lock tree transitively.

Filed, not fixed: the Python server orders ligatures wrongly under a collation (["fi","fi","fj"]), which this port surfaced. sort_levels decomposes with NFD, which does not split a compatibility ligature, so the secondary level compares a one-group key against a two-group one. The obvious fixes are unmeasured — see tasks/backlog.md.

Fixed

• secantus-core: normalize_index_bytes and the query-path normalize both fold non-ASCII instead of deferring; new sort_level_bytes three-level ordering key behind a new encode_sort_value, keeping the on-disk encode_value untouched; accent stripping filters category Mn (not all of M*); Collation carries caseFirst and backwards.

The Rust server refused every computed projection

find and findAndModify answered 2 BadValue: projection is not supported by the Rust server for any projection whose value was an EXPRESSION rather than an include/exclude flag:

find({}, {x: {$add: ["$a", "$b"]}})
    mongod  {_id: 1, x: 5}
    rust    2 BadValue: projection is not supported by the Rust server

Not just operator expressions — a bare rename {x: "$a"} failed too, as did a string literal, a null, an array, and a Decimal128 flag. The engine deferred those shapes to the pure evaluator, and a defer has no Python behind it on this server, so the refusal reached the client. The backlog had recorded this as a deliberate “honest refusal”; it is now implemented instead.

The semantics, all measured against mongod 8.2.11

• Only a number or a bool is a FLAG. Everything else is a value: {x: "plain"} writes the string on every document, and {x: {$literal: 0}} yields 0 rather than excluding. A Decimal128 is a BSON number, so Decimal128("1.5") includes and Decimal128("0") excludes.

• A sub-document is classified PER LEAF. {n: {p: 1, z: "$b"}} includes n.p and computes n.z, so the spec is flattened to dotted leaves before anything is decided. An empty sub-document at any depth is 51270.

• A bare reference and an expression differ on a missing field. {x: "$absent"} omits x entirely; {x: {$add: ["$absent", 1]}} yields null. The field-value evaluator is what draws that line.

• A computed field forces inclusion mode, so a companion b: 0 is the 31254 mix, and _id: 0 still just drops _id.

• Dotted output keys build nesting through set_path — never a key with a literal dot in it, the shape CLAUDE.md warns about.

A projection error carrying a mongod code is now surfaced verbatim instead of being flattened to BadValue; only a bare defer becomes the generic refusal.

Sweep: 42 shapes at 0 divergences against mongod 8.2.11, comparing the raw find reply including FIELD ORDER, plus 7 shapes on findAndModify — the sibling command shares the engine and carried the same refusal.

Fixed

• secantus-core: computed projections implemented (is_flag_value / is_computed_spec / flatten_projection_spec / with_computed); spec_truthy accepts Decimal128.

• secantus-commands: find and findAndModify no longer refuse them, and a coded projection error keeps its code and message.

Decimal128 arithmetic and rounding on the Rust server, and five crash-class bugs on the Python one

decimal.rs already had add, mul and div_int — the primitives $sum and $avg accumulate with. They were never wired into the expression path, so {$add: [Decimal128("2.5"), 1]} told the client the Rust server could not do $add, while {$sum: …} over the same values answered. The backlog had this filed as “a dependency decision”; sizing it from a probe rather than the text showed most of it needed no new numerics at all.

	Before	After
agg_expressions.py, Rust server	32 code + 0 message	22 + 0
Wrong values, either server	0	0

Fixed on the Rust server

• $add / $subtract / $multiply over a Decimal128, at 34 digits, with the quantum preserved: 2.5 * 2 is 5.0 and not 5, and Decimal128("2.50") + 2 is 4.50.

• $ceil / $floor / $trunc / $round, including the place argument. New decimal::round_to_exp with the four modes.

• $log validates its base before deferring a decimal argument — the four checks run argument-type, base-type, argument-domain, base-domain, so a bad base is named even when the argument is one this engine cannot compute with.

Five crash-class bugs on the Python server

Each reached the client as internal server error, from any query:

• {$add: [Decimal128("-Infinity"), Decimal128("Infinity")]} — Python’s decimal context traps InvalidOperation; decimal128’s own answer is NaN. The context now traps nothing, which closes the class rather than the instance.

• $trunc / $round of a decimal infinity — quantize refuses a non-finite operand by IEEE rule.

• $ceil / $floor / $trunc of a double infinity or NaN — math.ceil(inf) raises OverflowError and math.trunc(nan) raises ValueError.

The last of those was caught by this change’s own control assertion — a line included only to contrast the double case with the decimal one. The probe corpus contains no infinities, so none of this surface had ever been asked about.

Two more Python defects

• The decimal fold ran in Python’s default context: 28 digits, so six of decimal128’s 34 were silently lost. 1.000000000000000000000000000000001 + 1 answered 2.000000000000000000000000000.

• It converted a double with Decimal(v), giving 4.5 where mongod gives 4.50000000000000. Arithmetic takes a double at 15 significant digits, and that precision enters the quantum — a different conversion from the one the accumulators use.

Rules probed, not derived

• The place sets the quantum whether or not it changed the value: {$round: [Decimal128("2.5"), 2]} is 2.50. Returning it unchanged was wrong on 40 of 210 shapes.

• When the whole value sits below the target place, the deciding digit is an implicit leading zero, not the coefficient’s first digit — {$round: [Decimal128("9.995"), -3]} is 0E+3, not 1E+3.

• $ceil / $floor of a decimal infinity is NaN, while $trunc / $round pass it through and a double infinity passes through all four. An asymmetry, measured rather than reasoned.

Why the transcendentals are still deferred

Not for want of an implementation: the Python engine computes them at 60 digits. It still disagrees with mongod in the last digit on 5 shapes, because mongod uses Intel’s decimal library — and a prior session verified at 80 digits that our value is the correctly-rounded one and mongod’s is 1–2 ULP low.

The operations landed here are the ones IEEE 754-2008 defines as correctly rounded, which is exactly why they match exactly. Porting series to Rust would trade 16 loud “unsupported” errors for silent last-digit-wrong values, and this campaign has held wrong values at zero throughout. tasks/backlog.md §7 records the decision with the measurement.

$abs and the $toX conversions accept Decimal128 on the Rust server

$abs, $toBool, $toInt, $toLong and $toDouble all deferred on a Decimal128 operand — and a defer on the standalone server is an error, so a collection holding decimals could not be converted or absolute-valued at all.

70 conversion shapes now match mongod on both servers (the probe corpus counts 5 of them: agg_expressions.py Rust 907 → 902, zero wrong values).

Rules that would have been wrong by assumption

Every one of these was probed against mongod 8.2.11 rather than derived:

• $toBool of NaN is true — not an error, and not false. So is Infinity.

• -0 is false, so zero-ness ignores the sign.

• $abs preserves the quantum: Decimal128("-2.50") gives 2.50, not 2.5. No arithmetic happens, so it is implemented as a sign-strip and re-parse rather than a trip through the decimal engine.

• Three distinct messages under code 241 — NaN, infinity, and overflow are not one failure — and the overflow message echoes the decimal’s own rendering (…no onError value: 1E+30), not a normalised form.

• $toInt of 2147483648 overflows where $toLong succeeds.

Not in this change

$add / $subtract / $multiply / $divide / $mod on decimals still defer. Those go through the decimal engine where the quantum is load-bearing — 2.5 × 2 is 5.0, and Decimal + double yields 4.50000000000000, not 4.5 — and that deserves its own pass. tasks/backlog.md carries the probed semantics so it starts from measurement.

Decimal128 values outside f64’s range no longer take the wrong branch

decimal128 spans 1E-6176 to 9.999…E+6144; f64 spans about 1E-308 to 1E+308. Eight operators on the Rust server asked their classifying questions — is this infinite? is this zero? — of an f64 rendering of the argument, and that rendering saturates: a finite Decimal128("1E+6144") reads back as f64::INFINITY and a finite Decimal128("1E-6176") as 0.0. Each took the branch for a special value and answered confidently.

Measured against mongod 8.2.11: the Rust server diverged on 79 of 180 cells (18 operators × 10 extreme inputs) and the Python engine on 64 of 364 (14 × 26). Both grids are now 0 apart from $exp in its middle range on the Rust server, which needs a decimal exponential series.

Fixed

• $sqrt of a large finite decimal answered Infinity. It now computes in decimal — correctly rounded, which IEEE 754 requires of square root, with the decimal spec’s ideal exponent so $sqrt of 4 is 2 and of 1E+6144 is 1.00000000000000000E+3072. Verified against mongod on 75 values.

• $toDouble of an out-of-range decimal returned inf / 0.0 where mongod raises 241 ConversionFailure. A decimal converts only when the double is normal; the boundary is IEEE’s tininess-after-rounding cut at 2^-1022 − 2^-1076, a quarter of a subnormal ULP below f64::MIN_POSITIVE (bisected against 8.2.11), so a value representable as a subnormal double is still a 241.

• $toBool of a decimal below f64’s range was false. 1E-6176 is not zero, and is now true.

• $floor / $ceil of a decimal needing more than 34 integer digits returned the value; they are the decimal spec’s quantize, so mongod answers NaN. $trunc / $round deliberately do not share the rule.

• $degreesToRadians / $radiansToDegrees answered Infinity above 1E+309 and a zero below 1E-324. Both are now one correctly-rounded decimal multiply by mongod’s own 34-digit constant, reaching the subnormal range: 1E-6176 radians is 5.7E-6175.

• $exp in the two regions that need no series: |x| ≥ 1E+5 is decided by sign alone (Infinity, or 0E-6176 at the minimum quantum), and |x| ≤ 1E-40 is exactly 1.

• Arithmetic results outside decimal128’s exponent range were refused rather than clamped. They now follow the format’s rules — overflow to ±Infinity, and rounding to the minimum quantum on the way down, which is where subnormal results come from.

Fixed (Python server)

The same sweep run against the pure Python engine found the same family, plus three of its own:

• Three crashes. $degreesToRadians / $radiansToDegrees of a decimal whose exact product falls outside decimal128’s range raised a raw decimal.Inexact or decimal.Overflow out of the evaluator — an internal server error where mongod returns a value (5.7E-6175, Infinity). Constructing a Decimal128 now clamps to the format instead of refusing.

• The angle conversions computed x * pi / 180, two roundings where mongod does one — the association the double path’s own comment warns against. They now use mongod’s 34-digit constants, which fixes results that came back with 32 significant digits.

• $trunc / $round of a decimal past 34 integer digits answered NaN. They do not quantize; mongod expresses the value at the finest quantum that fits.

• Plus the shared items above: $sqrt of a negative decimal too small for float to keep the sign, $toBool, $toDouble, and $floor / $ceil.

Decimal128 takes part in the numeric order on the Rust server

mongod treats Decimal128 as one of the numeric types for comparison and sorting: a mixed field sorts Decimal128("1") < 2 < Decimal128("2.5") < 3.0, and {$gt: [Decimal128("2.5"), 2]} is true.

order::cmp had always handled that — rank 3 routes through numeric::classify, which understands decimals. Only order::is_sortable excluded them, and that one predicate is what every comparison consults first. So on the Rust server every comparison involving a decimal deferred, and a defer there is a generic BadValue with no Python behind it.

The practical effect was larger than the probe count suggests: $gt / $lt / $cmp, sort(), and range queries like find({v: {$gt: 2}}) were all unusable on a collection holding decimals.

	Before	After
agg_expressions.py codes, Rust	912	907
Decimal sort / compare / range query	deferred	matches mongod

Five corpus shapes; a whole capability in practice. tests/test_decimal128_ordering.py pins it against both servers, because the defect was on the Rust one and a Python-only test would have proved nothing about it.

A note on the test that had to change

order::tests::sortable_gating asserted !is_sortable(Decimal128) — pinning a gating decision rather than a behaviour. Its own comment records the same thing having happened with bools, which were excluded for the same wrong reason. mongod’s actual sort order was probed before the assertion was changed.

A collection holding a Decimal128 was un-updatable on the Rust server

{$set: {z: 1}} against a document containing a Decimal128 anywhere — a top-level field, a nested one, an array element, or the _id — failed with query uses a construct the Rust server does not support. Nothing about the update touched the decimal.

The oplog update-diff walks every field of the old and new document through py_eq to work out what changed, and py_eq deferred on Decimal128. On the Python server that defer is a real fallback; on the standalone Rust server there is no Python behind it, so the whole update failed. replace_one and $unset worked, which is what made it look like an obscure edge rather than “this collection is read-only”.

numeric::classify has handled Decimal128 all along — only the fast-path comparison declines it — so the comparison was already available and equality simply was not asking for it.

A descending sort put every prefix chain in ascending order

sort({x: -1}) over ["", "a", "ab", "abc", "b"] came back from the Rust server as ["", "b", "a", "ab", "abc"] — the empty string first, and the whole a/ab/abc chain ascending, inside a descending result. mongod answers ["b", "abc", "ab", "a", ""].

The cause is a good idea used one step too far. A descending column is stored in the B-tree by inverting its key bytes, which is the only way to express a direction where the storage engine sorts by raw bytes. The Rust server reused that trick to build an in-memory sort key — and inversion is not a descending comparator, because it does not reverse a prefix relationship. "" encodes to a strict prefix of "a"’s key, and a shorter byte string sorts first both before and after inversion.

Direction is now applied when the keys are compared rather than by inverting them, which needs no trick at all: prefix-shorter-first is exactly right ascending, and its reverse is exactly right descending. Nothing persisted changes, so no stored index is affected — a descending index gives the same answer it always did, and the same answer as no index at all.

The Python server was never affected, because its in-memory sort goes through ordering.sort_docs rather than the encoder. It shares the encoder, though, so both definitions now carry the warning that the inverted form is for B-tree placement only and must never be used to order values.

Fixed

• secantus-storage: sort_key emits ascending per-field parts and compare_sort_keys applies each field’s direction, so a descending sort orders prefix chains correctly. Per-field direction in a compound sort is unchanged, as is []’s place between MinKey and null.

• secantus-commands: the min / max cursor-bound comparison compared inverted keys and had the same flaw; it now compares ascending encodings and negates.

• secantus.sortkey / secantus-core: encode_value_directed documents that it is for physical B-tree placement only, with the measurement that shows why.

The Rust server answers mongod’s argument errors

tools/probes/operator_error_surface.py crosses every query and update operator with every pathological argument. Against the Rust server it reported 1,053 divergent shapes, 999 of which answered BadValue: "query uses a construct the Rust server does not support" for an argument mongod names precisely — Unknown type name alias: x, Expected a number in: n: "x", and so on. It is now 17, on a corpus that grew 38% during the work.

840 of those 999 had the right code by accident, because mongod’s parse errors are BadValue and so is the generic refusal. A code-only comparison had made this look like 159 problems.

Fixed

• Every query operator names its own argument errors: $mod, $size, $type, $in / $nin, $all, $elemMatch, $regex, $not, and $and / $or / $nor.

• Every update operator likewise: $pop, $rename, $bit, $currentDate, $push, $pull, $pullAll, $addToSet.

• The storage layer’s update path threw the named error away with map_err(|_| QueryUnsupported) — the same erasure the query path had before it gained query_fault. That one line was why the engine’s messages could not reach a client.

• $exists read every Decimal128 as truthy, so {v: {$exists: Decimal128("0")}} matched where mongod matches nothing. This was wrong on both servers, and the sweep missed it until the corpus gained a zero Decimal128 — a value that is falsy in one BSON type and truthy in another is exactly what such a corpus needs.

• {v: {$type: NaN}} crashed the Rust server with internal server error.

• Decimal128("-0") was rejected as unrepresentable everywhere it appeared, because whole-ness was tested by comparing parsed forms structurally and -0 is not structurally 0.

• bson_type_name reported object for MinKey, MaxKey, Timestamp, Undefined, Symbol, DbPointer and JavaScript — everything its match arms did not name.

The Rust server’s explain echoed the filter instead of normalising it

mongod does not echo the filter you sent back in queryPlanner.parsedQuery — it echoes the MatchExpression tree after normalisation. A bare equality grows an explicit $eq, several top-level fields become an $and whose children are sorted by mongod’s internal match-type ordinal, $ne becomes $not/$eq, an $in of one collapses to $eq, an $in of none becomes $alwaysFalse, $all splits into equalities, $type becomes numeric BSON codes, a bitmask becomes a bit-position array, and $comment disappears.

The Rust server answered with the filter as sent, so a client reading parsedQuery — the usual way to ask “how did the server understand my query?” — got its own input back. It diverged from mongod on 44 of the 56 shapes in tools/probes/explain_shapes.py; the Python server matched all 56, because only it had secantus.explain.canonical_match.

secantus-core::explain is now the port of that module, rule for rule, and the Rust server is at 0 of 56.

This was found by sweeping the sixteen Rust-aware probes against a server built from main. Twelve of them are completely clean — 740 aggregation-stage shapes, 3,074 operator-error shapes, 409 date/timezone shapes, 244 extended argument types with no crashes, and more — which is what made the explain result stand out rather than blend into a general noise level.

Still open, and filed with its measurement: the winningPlan plan-node FIELDS. mongod’s COLLSCAN carries direction and isCached, and its IXSCAN carries nine keys where the Rust server emits four. The Python server reproduces those and sits at its documented floor of 7 of 25 (four indexBounds, which this project deliberately does not reproduce, and three genuine cost-model differences); the Rust server is at 25 of 25.

Fixed

• secantus-core: new explain module — a port of secantus.explain’s canonical_match, including the match-type rank table, $type alias codes and the $nor decomposition rule.

• secantus-commands: explain’s parsedQuery and the COLLSCAN stage filter report the normalised expression.

The Rust server’s explain plan nodes were missing most of mongod’s fields

mongod reports a fixed set of keys on each plan node, and a client reads them to answer real questions. The Rust server’s IXSCAN emitted four of the nine:

key	what it answers	Rust before
isUnique	can a reader assume one document per key?	absent
isSparse	does the index omit documents?	absent
isPartial	does a filter restrict what it holds?	only when true
multiKeyPaths	which fields made it multikey?	absent
indexVersion	the index format	absent

Three more shapes were wrong rather than missing:

• FETCH echoed the whole filter. mongod carries only the RESIDUAL predicate there and omits the key entirely when the index bounds covered the query — which is exactly how a reader tells a fully-index-served query from one that re-checks every document. Echoing the whole filter erased that signal.

• COLLSCAN had no direction and emitted an empty filter where mongod omits the key.

• isCached was never set. It is a whole-plan property, so it belongs on the outermost node only, as its first key.

The Rust server’s IXSCAN node is now identical to the Python server’s, and the probe’s winningPlan count drops from 25 of 25 to 22 of 25.

Still open, and filed with the shapes that show it: the SORT / SKIP / LIMIT / PROJECTION_SIMPLE|PROJECTION_DEFAULT stage tree (secantus.explain.build_stage_tree), which the Rust server does not build at all — so {filter: …, limit: 3} reports a bare COLLSCAN where mongod reports LIMIT above one. That is what the remaining 22 are, and the blocking-SORT half of it needs a sorted_by_index flag that ExplainPlan does not currently carry.

Fixed

• secantus-commands: explain’s IXSCAN node carries mongod’s nine keys in mongod’s order; FETCH carries only the residual filter; COLLSCAN reports direction and omits an empty filter; isCached leads the outermost node.

The Rust server’s explain now builds mongod’s stage tree

mongod wraps the scan in the stages that describe the rest of the query. The Rust server reported the bare scan node, so a client got:

query	mongod	Rust before
limit: 3	LIMIT → COLLSCAN	COLLSCAN
skip: 3	SKIP → COLLSCAN	COLLSCAN
limit: 3, skip: 2	LIMIT → SKIP → COLLSCAN	COLLSCAN
projection: {a: 1}	PROJECTION_SIMPLE → COLLSCAN	COLLSCAN
sort: {zzz: 1}	SORT → COLLSCAN	COLLSCAN

The most useful consequence of getting this right: a client asking “is my sort served by an index?” reads the answer off the presence of a blocking SORT, which is the question explain is usually run to answer. Without the tree there was nothing to read.

The nesting is mongod’s own and is not the order the command’s fields are written in — a blocking SORT sits directly above the scan and ABSORBS the limit (as limitAmount, counting the documents the skip will later discard, so no separate LIMIT appears); SKIP sits above that; the projection above the skip; an unabsorbed LIMIT outermost.

Deciding whether a sort needs a blocking stage takes a sorted_by_index flag, which ExplainPlan::IxScan did not carry — the walk comes out in sort order only when the index’s LEADING field is the one being sorted on. It is now plumbed from make_ixscan_plan and both hint branches through the storage adapter.

The Rust server now diverges from mongod on exactly the same seven shapes as the Python server, which are that server’s documented floor: four are indexBounds (which this project deliberately does not reproduce, along with rejectedPlans and the IDHACK / EXPRESS_IXSCAN / COUNT_SCAN / DISTINCT_SCAN executors) and three are a genuine cost-model difference where mongod picks an IXSCAN and we pick COLLSCAN + SORT, returning identical documents. explain is at parity between the two servers.

Fixed

• secantus-core: build_stage_tree and projection_stage_name, ported from secantus.explain.

• secantus-storage: ExplainPlan::IxScan carries sorted_by_index, set at every construction site; the storage adapter surfaces it.

• secantus-commands: explain wraps the scan node in the stage tree for find (count / distinct keep their flat node — their COUNT_SCAN / DISTINCT_SCAN vocabulary has not been measured).

Expression parse errors on the Rust server

aggregate._expression_shape_problem — the parse-time pass that fixed 279 shapes on the Python server — ported to the Rust command layer. Arity and spec shape are errors mongod raises while building the expression tree, before it folds anything, so they carry the stage’s wrapper rather than the optimizer’s.

tools/probes/agg_expressions.py (3,968 cases)	Before	After
Wrong error codes	1,376	981
Message-only differences	4	6
Wrong values	0	0

Fixed

• Arity ($indexOfArray / $indexOfBytes / $indexOfCP / $range / $slice): 28667, with each operator’s own bounds.

• Date extractors given an array of any length but one: 40536.

• Object-spec expressions: $firstN/$lastN 5787801, $minN/$maxN 5787900, $median 7436201, $percentile 7436200, $topN/$bottomN 168.

• Unrecognised date-spec arguments across eight operators, eight codes — several of which the Rust server had been silently ignoring, answering ok for a spec mongod refuses.

Left

946 of the remaining 981 are the engine deferring on a bad argument — “operator not supported by the Rust server” for what is a bad operand. They are spread across ~120 operators and driven by the operand type, so they are a per-operator campaign rather than another systematic pass. Recorded in tasks/backlog.md with the operand-type breakdown.

The Rust server leaked Rust type names into the bad-hint error

A hint that names no index came back with Rust’s Debug formatting, so a MongoDB client was shown Rust’s own type names:

hint String("x") does not correspond to an existing index
hint Document({"nope": Int32(1)}) does not correspond to an existing index

String(…), Document(…) and Int32(…) mean nothing to a client. Both now name the value: hint "x" and hint { nope: 1 }. The string form leaked from the command layer (update / delete / findAndModify) and the key-spec form from the storage layer (every command that takes a hint).

The rest of this message is a deliberate difference and is unchanged: mongod answers a bad hint with a multi-line planner diagnostic that names the whole query plan, and this project reproduces the CODE and names the hint instead — the wording moved between 6.0.16 and 8.2.11, so the gate asserts the rejection rather than the text. That decision is why the divergence stays in the probe’s message-only column on both servers; what was not deliberate was leaking Rust’s internals into it.

Fixed

• secantus-commands: the bad-hint error renders the hint value, not format!("{hint:?}") on a Bson.

• secantus-storage: the key-spec branch of resolve_hint likewise.

The Rust server had the same four index defects

The index-layer fixes that landed for the Python server — a sparse index answering queries that match a missing field, a compound sparse index that under-indexed, a partial index whose implication check compared across BSON type brackets, and a query naming only a partial filter’s own fields — were fixes to storage.py. The Rust server has its own storage layer with its own port of those helpers, and it still had all four. Three are silent data loss: the query succeeds, the shape is right, and rows are simply missing.

Nothing would have caught them. The engine-parity suites that keep the two engines honest pin query, update, expressions, projection, sortkey, diff and aggregate — the pure operator engines. None of that is the storage layer, so a divergence there is invisible to the one mechanism built to catch divergence.

One of the four behaves differently on this side and worse. Where the Python server raised IndexError out of the command handler for a query covering only a partial filter’s own fields — loud, and visible as an internal error — Rust built an empty key prefix and scanned for the bare separator, which matches no key. It returned nothing, quietly.

Added

• tools/probes/index_result_sets.py: the sweep, which compares the server against itself with and without the index rather than only against MongoDB. That isolates the index from the sort engine, and it is what makes the output diagnostic: indexed=[1] no-index=[1,2] mongod=[1,2] names the dropped row. Against the pre-fix Rust server it reports 12 of 15 curated and 53 of 1692 randomised; after, zero on both servers.

Fixed

• secantus-storage: sparse_covers (at least one indexed field present), sparse_index_usable / predicate_may_match_missing (a sparse index is unusable for a query that could match a missing field — including any comparison against null, not just $eq), type_bracket on the partial implication check, and a whole-index scan where there is no key prefix to pin.

$abs only supports numeric types, not string — the Rust engine’s largest error family

The biggest single block left in the expression sweep: 24 unary operators, 220 shapes where mongod answers 28765 $OP only supports numeric types, not <type> (or 51081 for $round / $trunc) and the Rust server answered its generic BadValue (2).

The Python server already had these right, so this is a Rust-side gap: its operators know the operand’s type when they evaluate it, but their only failure signal is Fallback, which carries no code — the comment on op_abs read “Python raises 28765 -> defer”, which is true on a server that has Python and useless on one that does not.

Fixed

$abs, $acos, $acosh, $asin, $asinh, $atan, $atanh, $bitNot, $ceil, $cos, $cosh, $degreesToRadians, $exp, $floor, $ln, $log10, $radiansToDegrees, $round, $sin, $sinh, $sqrt, $tan, $tanh, $trunc now answer mongod’s code and sentence on the Rust server, for a constant operand and for a field reference alike.

It works by re-evaluating just the argument — not the operator — against the documents the stage sees, which is enough to name the error without widening Fallback. That is the update::arith_type_error template the other validators in this module use. A null operand is not an error, and Decimal128 is numeric: it defers for a different reason, so reporting a type guard for it would be wrong.

A note on the gate

may_name_runtime_error decides whether to keep a copy of the input documents for this naming pass, and it is deliberately narrow because the copy is taken on the success path too. It now also fires when a numeric-guard operator appears anywhere in a stage — a spec scan, so it costs nothing for a pipeline without one.

Sweep status, and one thing that got more visible

Rust code differences 1556 → 1336. Rust message-only went 0 → 148, which is these same cases moving from “wrong code” to “right code, wrong wrapper”: for a constant operand mongod folds at optimization time and says Failed to optimize pipeline :: caused by :: where we say Executor error during aggregate command on namespace: … :: caused by ::.

That is the long-deferred constant-folding item — but it is now measured rather than guessed, and the rule turns out to be statically decidable: mongod folds exactly when the argument contains no field-path reference ($$NOW folds too). Recorded in tasks/backlog.md with the probe, since it is a much cheaper proposition than “modelling constant folding” implied.

14 differential cases added.

A wrong-typed operand names mongod’s error instead of “not supported”

An aggregation operator that had to refuse an argument had only one way to say so in the Rust engine — Fallback::Defer — and a defer on the standalone Rust server has no Python behind it. It surfaces as 2 BadValue, “aggregation pipeline uses a stage or operator not supported by the Rust server”. So {$size: 1} told the client this server cannot do $size. It can; 1 is not an array.

tools/probes/agg_expressions.py measured 908 such divergences against mongod 8.2.11 across ~120 operators, with zero wrong values — the whole surface was error-shaped. Roughly 900 of them were one defer standing in for an ordinary argument complaint.

	Before	After
agg_expressions.py, Rust server	902 code + 6 message	117 code + 0 message
agg_expressions.py, Python server	50 code + 212 message	28 code + 80 message
Wrong values, either server	0	0

Every expectation was measured against 8.2.11 rather than derived, and several defeat a reasonable guess.

Fixed

• The thirteen date extractors ($year / $month / $hour / …) report 16006 can't convert from BSON type <T> to Date instead of deferring — 169 shapes from one helper. Their {date, timezone} options form is now validated too: an unrecognised key is 40535 and reports the first offender even when date is present and valid, and a spec with no date is 40539, echoing the document back.

• The eight $convert shorthands. An array of any length but one is 50723 $toInt requires a single argument, got 3; an unsupported pair is 241 Unsupported conversion from objectId to int; overflow, NaN and infinity each carry their own sentence under that same code. {$toInt: [1]} now unwraps to the single argument mongod reads.

• Fourteen type-guarded operators — $size, $first/$last, $strLenCP/$strLenBytes, $reverseArray, $arrayToObject/$objectToArray, $bsonSize, $binarySize, $tsSecond/$tsIncrement, $allElementsTrue/$anyElementTrue — plus $concat, $concatArrays, $in, $arrayElemAt, $slice, $indexOfArray, $indexOfBytes, $indexOfCP, $split, $toLower/$toUpper and the $bit* family.

• $ifNull, $setEquals, $rand and $getField are refused at parse time, under the stage’s Invalid $addFields :: wrapper rather than a pipeline one.

Capabilities the Rust server was missing

Four conversions mongod performs were reported as unsupported: $toDate of a date string, an ObjectId or a Timestamp, and $toLong / $toDouble / $toDecimal of a date ($toInt of a date really is refused — the arms are not interchangeable). $convert’s onError now catches an unsupported pair, which it previously ignored; the one form that exists to survive a bad conversion was the one that could not.

Bugs this found on the Python server too

These sit outside the probe corpus, so nothing had ever compared them:

• $setUnion / $setIntersection / $setDifference answer null for a null operand; both servers raised. Operands are scanned left to right, so {$setUnion: [null, 1]} is null while {$setUnion: [1, null]} raises on the int. $setEquals and $setIsSubset refuse null instead — two rules, not one.

• $setDifference and $setIsSubset carry a different code per position (17048/17049 and 17046/17042); both reported the first-argument code either way.

• $getField’s bare form is an expression, not a literal field name. A plain string still evaluates to itself, so {$getField: "s"} is unchanged, but {$getField: "$n"} resolves the path and then refuses the int — where taking it literally looked for a field named $n and answered missing. A literally-dollared name goes through $literal, which both engines had been reading as the options form and rejecting as an unknown argument. The object form requires input and does not fall back to $$CURRENT.

• $toDecimal of a date, $tsSecond/$tsIncrement’s wording, and $slice’s “but is of type:” phrasing.

• $split’s second argument is 10503900 on 8.2.11, not the 40086 recorded here; the first keeps 40085. Only one of the pair moved.

• The four $bit* operators do not agree on how to refuse a bool: {$bitOr: [1, true]} is the fold family’s bare 14 ... only supports int and long operands. with no type named, while $bitNot calls a bool non-numeric (28765). One sentence had been standing in for both.

• One value renderer was standing in for two mongod serializers. specification must be an object; found $firstN: [ 3, 1, 2 ] renders in mongod’s shell form — inner spaces, ObjectId('…'), new Date(1767323045000), BinData(0, 7A) — while $replaceRoot’s Input document: {n: 1} uses the compact form with none of those. Both call sites went through bson_value_repr_stage, so one of them was always wrong — and neither wholesale choice works: rendering everything compact leaves 66 shapes wrong, rendering everything shell-form leaves 13. Probing the two families side by side is what separated them.

Rules worth recording

• mongod says missing for an absent field path where it says null for an explicit null. One eval collapses both, so the distinction is recovered when a message is built rather than threaded through evaluation.

• Null-tolerance is per operator: $size and $strLenCP refuse null; $first and $reverseArray answer null.

• $bitNot uses two codes — 28765 for a non-numeric operand, 14 for a numeric one it cannot use — where $bitAnd / $bitOr / $bitXor use one sentence for both.

• $getField never folds, not even with a wholly literal input, because it reads $$CURRENT. That now lives in is_constant_expression, which both engines consult.

• The wordings are not interchangeable: “found: {}” vs “but is {}” vs “but was of type: {}”, $setEquals’s literal “1-th argument”, and $tsSecond’s verbatim leading space.

Caught by the parity suite

Two of these were found by tests/test_rust_expressions_parity.py after the Rust half moved and the Python half had not — which is what that suite is for. It also caught a defect in this change: a finite double too large for the target ({$toLong: 1e30}) fell into the non-finite arm and was reported as “Attempt to convert infinity value”, where mongod says it overflowed and echoes 1e+30. Parity flagged the drift within seconds; the oracle said which side to move.

Two parity tests asserted the Rust engine raw is None for these operators — pinning the gating decision rather than the behaviour, so they broke the moment the operators learned to name their own errors. They now assert what actually matters: defer or name it, but the client sees mongod’s code either way.

A stale citation, not a regression

$split’s second-argument code was recorded as 40086 in three places, one of them a test whose docstring read “mongod 7.0.12-verified.” It was right for 7.0.12 — 8.2.11 answers 10503900, while the first argument keeps 40085 — and the expectation survived the 8.x retarget unchecked. Re-probed and re-dated. This is the shape CLAUDE.md warns about: a version citation records when something was measured, not what the server does now.

tests/test_ci_runs_rust_server_tests.py also did its job — the new test file is gated on importorskip("_secantus_server"), so it would have run nowhere until named in the storage-engine job. Now wired in.

And wiring it in immediately found the next thing: a server per test is ~380 WiredTiger stores across this file, which filled the disk on the Windows runner. Nothing here needs a private server, so it takes one per engine per module and re-seeds the collection between tests — 192 cases in 1.6s rather than 82s, and two stores rather than 380.

Also fixed

Conv::Failed was serving three different mongod messages at once — overflow, non-finite, and unsupported-pair — which was invisible while all three deferred. One of its own comments already described a case as “Unsupported conversion” while the code classified it as a failure.

tests/test_expression_operand_errors.py pins all of it against both servers.

Two existing claims turned out to be untested rather than wrong-by-regression. Three tests asserted $getField’s non-string field was 5654602; mongod answers 3041704. And {$getField: 0} was listed as a parse error, which nothing could check — it only reached the parse pass because the constant-fold pre-pass happened to evaluate it and surface the exception. On an empty collection mongod returns no documents and no error, so it is a runtime error; that test now drives the evaluator.

Still open

117 shapes, characterised in tasks/backlog.md: Decimal128 arithmetic and the transcendentals (the deliberately deferred half), $mergeObjects and the trigonometric domain errors (both need mongod’s value rendering), and a handful of value gaps in $substr* / $strcasecmp / $range.

The Rust server reported four spec errors under the wrong wrapper

mongod uses Invalid $addFields :: caused by :: for an expression that fails at PARSE time, and the optimizer’s or executor’s prefix otherwise. The Rust server’s parse-time scanner already classified most of that correctly — but a spec document missing a REQUIRED key fell through it, so {$convert: {to: "int"}} and its siblings were reported as fold or runtime failures instead.

Now covered: $convert without input, $dateDiff without startDate, the n-operator family ($firstN / $lastN / $maxN / $minN) without n, and $dateFromParts with neither year nor isoWeekYear. The Python engine got the same set in the preceding change.

Ordering matters and is pinned by its own test. mongod reports an unrecognised key before a missing required one, so {$firstN: {k: 1}} is “Unknown argument for ‘n’ operator: k” while only {$firstN: {}} is “Missing value for ‘n’”. Both checks fire on the same document, so only their order separates them — and getting it backwards changes the CODE on shapes that are already correct, which is what happened on the Python side before it was corrected.

A note on how this was sized. The backlog entry this closes claimed the Rust server “never emits the stage wrapper at all”, from a grep over two files. It does: the wrapper, the wrapping-stage list, and a parse-time scanner covering $ifNull and $setEquals — 76 of the 83 shapes the Python fix was worth — were all already there. Running the scanner in a throwaway #[test] (possible because secantus-commands is a clean-workspace crate needing no WiredTiger) showed the true gap in one command. The entry has been corrected in place rather than left beside a newer one.

Fixed

• crates/secantus-commands/src/argtypes.rs: REQUIRED_SPEC_KEY and the $dateFromParts year check, evaluated after every unknown-key table.

Changed

• Three tests in argtypes: the classification, the unknown-before-missing ordering, and valid specs that must fall through to folding.

Rust pgserver: ANY / ALL array operators

scalar <op> ANY(array) and scalar <op> ALL(array) work now — the form psycopg renders an IN-list into (col = ANY(%s)), so a whole family of cursor and array tests turned on it. All six comparison operators are supported, in both a SELECT expression and a WHERE clause, with PostgreSQL’s exact three-valued logic: ANY is true on the first match, ALL false on the first mismatch, a NULL element or NULL scalar yields NULL, an empty array is false for ANY and true for ALL. An untyped array parameter (which arrives as array literal text) is coerced to the column’s element type, matching how PostgreSQL resolves an unknown ANY operand. A scalar compared to an array with no ANY/ALL is 42883 (no such operator), as on PostgreSQL — an array-to-array comparison is unaffected.

Added

• = / <> / < / <= / > / >= with ANY(array) and ALL(array), in SELECT and WHERE, over array literals and array parameters (including untyped ones).

Fixed

• A scalar compared to an array without ANY/ALL now raises 42883 instead of silently matching nothing (WHERE) or reporting a wrong operand type.

An array takes its type from its elements

array[%s::float4] came back as an array of strings. So did array[%s], and so did every array built over a parameter. The values were computed correctly and then described wrongly: the array’s type was read off the values it held, and the pass that describes a statement to the client sees no values at all — every parameter is NULL there — so it settled on text[], and the client decoded floats as text because the row description is what it believes.

An array’s type now comes from its elements’ expressions, which are the same whether or not there are values to hand. Mixed numerics widen in PostgreSQL’s own order (array[1, 1.5] is numeric[], array[1::float4, 1.5] is float4[]), a bare NULL contributes no type at all, and the element conversion on the way out follows the column’s type rather than the first element’s — which is what turned the 1.5 in array[1, 1.5] into a NULL.

Two smaller things fell out. An array of dates was described as varchar, so a client read back strings where PostgreSQL hands it dates; the remaining array types now have their real oids. And a quoted brace was being treated as the start of a nested array, so '{"{"}'::text[] answered “malformed array literal” — only an unquoted { opens a sub-array, and { is an ordinary member of any corpus that walks the ASCII range.

A third: two characters disappeared from any text array that carried them. U+0085 and U+00A0 are whitespace to Rust’s trim and not to PostgreSQL, so an unquoted element that was one of them came back as the empty string — a character in, nothing out, and invisible to any test whose alphabet is ASCII.

Fixed

• An array built over a parameter was described as text[] and its values handed back as strings.

• array[1, 1.5] turned its decimal element into a NULL.

• An array of dates, timestamps, intervals or json was described as varchar.

• A quoted { inside an array literal was read as a nested array.

• U+0085 and U+00A0 were trimmed out of array elements entirely.

Arrays on the Rust PostgreSQL server, and the bug that correct types exposed

int[] is a different type from int, and the difference is easy to lose: PostgreSQL’s own parser keeps the array-ness of a declared type in a separate array_bounds field rather than in the type’s name, so code that reads only the name types every array column — and every array cast — as its element type.

That mistake is quiet in a way worth recording. While a cast to text[] was degrading to text, comparing two arrays rendered both sides to their text form and compared the resulting strings, which agrees with PostgreSQL often enough to look correct. Typing the casts properly is what revealed that array comparison had never been implemented at all — so the feature that looked like a regression was really a gap that the wrong types had been hiding.

Array NULLs then turn out not to follow scalar NULL rules, and all four rules here were probed against a live PostgreSQL rather than reasoned out: inside an array two NULLs compare equal, a NULL sorts after every non-NULL, a shared prefix makes the shorter array the smaller one, and empty equals empty. Scalar NULL = NULL is NULL, so an elementwise comparison written by analogy with the scalar path gets every one of them wrong.

Multidimensional arrays are refused rather than answered. The encoder beneath this handles a single dimension, and the flattening it produced turned {{1,2},{3,4}} into two elements whose text read {1,2} and {3,4} — an answer a client has no way to tell apart from a real one.

Added

• Arrays as column types, cast targets and literals: PostgreSQL’s text form with its quoting rules, NULL elements, empty arrays, and the array type oids, so a client reads a list rather than a string.

• Array comparison and ordering, including the four NULL and length rules above.

Fixed

• An array cast lost its brackets, so %s::text[] was planned as %s::text.

• Array values were reported under their element type’s oid, so ARRAY[1,2,3] arrived at the client as the string {1,2,3}.

• Comparing two arrays raised “comparing these operands” instead of comparing them.

Changed

• A multidimensional array now raises 0A000 instead of being silently flattened into its rendered inner literals.

Rust pgserver: binary array parameters for bytea/inet/cidr/uuid

bytea[], inet[], cidr[], and uuid[] now round-trip as BINARY array parameters (oids 1001 / 651 / 1041 / 2951) and as array columns read in either wire format. The per-element decode reuses the scalar bytea / inet / cidr / uuid decoders, and the array types report their own oid instead of falling through to varchar (which had made a binary result hit the binary-varchar encoder and a text result hand back strings). A bytea array element also renders as \x… hex in the array text form.

Added

• Binary array-parameter decode and correct result typing for bytea[], inet[], cidr[], uuid[].

Fixed

• A bytea[] cast to text rendered each element as Rust debug output; it now renders \x… hex like PostgreSQL.

Bound parameters in the binary format

Client libraries do not send parameters as text. psycopg, like most modern drivers, sends numbers, dates, timestamps and arrays in PostgreSQL’s binary format by default, and falls back to text only where it must. The Rust PostgreSQL server decoded integers, floats, booleans and strings that way and refused everything else — so binding a Decimal, a date, a datetime or a list failed, even though the same values written as SQL literals worked.

Binary decoding now covers numeric, date, time, timestamp, and arrays of every element type the server knows. Each one decodes to the same canonical text a literal would have, so a bound value takes exactly the same path through the planner as a written one; the alternative — a second, parallel set of conversions for the binary format — is how the two formats drift apart and start disagreeing about the same value.

The text format had a bug of its own that this work surfaced. A numeric parameter was being parsed as a floating-point number, so a client binding 1.50 got back a float that had already lost both the exactness and the scale that make it a different value from 1.5.

Separately, a timestamp constant answered NULL. A stored timestamp is reassembled from its column plus a hidden field carrying sub-millisecond digits, and a constant never passes through a row — so it reached the encoder in a shape nothing matched, while the identical value read from a column, or cast to text, came back correctly. Three routes to the same value, one of them silently empty.

Added

• Binary-format decoding for bound numeric, date, time, timestamp and array parameters, including NULL elements and empty arrays.

Fixed

• A numeric parameter sent as text was parsed as a float, losing exactness and scale.

• SELECT '2026-01-01 12:00'::timestamp answered NULL, though the same value through a table column or cast to text was correct.

Changed

• A multidimensional array sent as a binary parameter is refused with 0A000 rather than decoded, matching what the server does when returning one.

Binary means binary, and a cursor is a portal

Two things a client asks for and this server quietly answered differently.

A cursor opened for binary results got its rows in text. The values were right — the format travels per column in the row description, so the client dutifully decoded text and handed back the right Python objects — which is exactly why it went unnoticed: nothing was ever wrong except the thing the client asked for. Result columns now go out in the requested format for the types this server can render exactly in PostgreSQL’s binary layout: booleans, the integer and float widths, the string types, numeric, and arrays of those. A column outside that set is still described as text, which the client reads correctly, and the gap is written down rather than hidden.

The other is server cursors. PostgreSQL exposes a declared cursor as a portal of the same name, and psycopg describes that portal straight after the DECLARE — before it fetches anything — to learn the columns. This server’s cursors were its own, so the describe found nothing and every server cursor died on its first row with “portal not found”. They now answer for the cursor of that name.

numeric was the interesting half of the format work: its binary layout is base-10000 groups aligned on the decimal point rather than on the digit string, so 0.00001 is a single group of 1000 two places below the point, not a group that straddles it.

Added

• Binary result columns for bool, int2/int4/int8, float4/float8, text/varchar/bpchar/name/char, numeric, and arrays of those.

• An answer for Describe portal naming a declared cursor, which is how psycopg’s server cursors learn their columns.

Fixed

• A cursor asking for binary results received text.

• Every psycopg server cursor failed with “portal not found”.

Rust pgserver: bytea comparison operators

bytea values could be stored, concatenated and sliced but not COMPARED — the Rust PostgreSQL server answered 0A000 for bytea = bytea and its siblings, which failed the psycopg tests that filter or order by a bytea. The comparison operators (=, <>, <, <=, >, >=) now work, ordering by unsigned byte value lexicographically exactly as PostgreSQL does (a prefix sorts before the longer value).

Added

• bytea comparison operators, ordering by unsigned byte value.

Rust pgserver: the bytea type

The Rust PostgreSQL server now supports bytea — as a cast, a bound parameter (text or binary wire format), and a real column type (oid 17). A value is stored as a Binary (the representation the Python server uses, since the two share one store), accepts both PostgreSQL input forms (the \x… hex form and the octal-\ooo escape form), and renders back as the \x… hex text a modern server emits. Because psycopg sends and reads a bytea in the binary format by default, both the binary parameter decoder and the binary result encoder handle it.

The byte-level functions come with it: length / octet_length / bit_length count bytes, get_byte / set_byte read and replace a byte, encode / decode convert to and from hex / base64 / escape, and bytea || bytea concatenates. The error surface matches PostgreSQL: a bad hex digit or odd length is 22023, a malformed escape is 22P02, and a byte index out of range is 2202E.

Added

• bytea cast, column type (oid 17), and text/binary parameter and result wire formats.

• get_byte, set_byte, encode, decode, bytea || bytea, and bytea-aware length / octet_length / bit_length.

Exact decimals, NaN’s place in the order, and one message hiding five bugs

The Rust PostgreSQL server could not compare a great many pairs of values, and said so with a single message — “comparing these operands” — that named neither of them. Making that message name the types turned one entry on the failure list into five distinct causes, four of which were the same rule and none of which were guessable from the text.

Arithmetic on decimals had stopped working entirely. When decimal literals became exact numerics rather than floating-point numbers in the previous release, every arithmetic operator on them began refusing outright: 1.5 + 1.5 was an error. It is fixed here, and fixed exactly — 0.1 + 0.2 is 0.3, and the result’s scale is part of the answer, so 1.50 + 1.5 is 3.00 where 1.5 + 1.5 is 3.0. Division stays refused rather than guessed at, because its result scale depends on the operands in a way that has not been measured.

Comparing decimals no longer goes through a floating-point number. A numeric carries 34 significant digits and a float holds about 15, so two visibly different twenty-digit numbers were the same float — and compared equal.

NaN has a place in PostgreSQL’s ordering, which the IEEE rules it inherits from do not give it: NaN equals itself and sorts above every number, infinity included. The underlying comparison reports “no answer” for each of those, which this server passed on to the client as an error where PostgreSQL has a result.

An unknown literal takes the type of the operand beside it — in comparisons just as in arithmetic. That type then decides both the parse and the error, which is why comparing an interval to '2020-01-01' reports a bad interval rather than false. Implementing this rule for arithmetic alone in the previous release left four of the five failures above.

Added

• Exact +, - and * on decimals, with PostgreSQL’s result scales.

• Comparison of decimals (on their digits), timestamps, and NaN / infinity.

Fixed

• Arithmetic on decimal literals refused outright since they became exact.

• Two different numerics with more digits than a float can hold compared equal.

• An unknown literal beside a typed operand was not resolved for comparison.

• The “cannot compare” error now names both operand types.

Composite types: the DDL and the catalog

CREATE TYPE name AS (field type, ...) and DROP TYPE work now, written in the same __sql_composites__ catalog the Python server uses — a doc per type with its ordered fields and a monotonically minted oid (base 67000, the enum rule), so an enum created by one server resolves on the other under the same oid.

Composites appear in pg_type (with typrelid set to their own oid), to_regtype and regtype, and a new pg_attribute virtual table exposes each composite’s fields keyed on that oid — attname, atttypid, attnum. This is the catalog CompositeInfo.fetch reads; its query’s nested-subquery join is a separate slice (a JOIN whose right arm is itself an aggregate subquery), so the fetch itself does not resolve yet.

Added

• CREATE TYPE … AS (…), DROP TYPE for composites, composites in pg_type/to_regtype/regtype, and the pg_attribute virtual table.

psycopg can read composite types back (CompositeInfo.fetch)

Registering a composite type with psycopg — CompositeInfo.fetch(conn, name) — now returns its fields against the Rust PostgreSQL server, matching a real PostgreSQL server exactly. The query psycopg emits is the most demanding catalog read the server has faced: pg_type LEFT JOIN (SELECT array_agg(attname), array_agg(atttypid) FROM (pg_attribute JOIN pg_type) GROUP BY attrelid), with a coalesce(..., '{}') around each aggregate column. It combines four things the server could not previously do at once — a JOIN whose side is a subquery materialised from an aggregate, an oid[] array column, a coalesce projected as an output column, and a base type surviving the LEFT JOIN with no matching fields.

Three of those were silent wrong answers rather than errors. array_agg(atttypid) came back tagged as text, so psycopg read the field-type list as the raw string "{23,25}" instead of a list of oids. A base type with no fields (or any type whose composite subquery found nothing) dropped out of the result entirely, or returned None for field_names/field_types where PostgreSQL returns two empty arrays — because the coalesce fallback never ran on a LEFT-JOIN miss, which is the one case it exists for. And a composite field whose type is itself a user type (CREATE TYPE t AS (sub other_composite)) was silently dropped from pg_attribute, since the field-type-to-oid lookup consulted only builtin types.

Added

• CompositeInfo.fetch is supported: the aggregate-subquery join side, the oid[] (_oid, 1028) array column type, the coalesce-as-output-column, and the coalesce-fills-'{}'-as-an-empty-array-on-a-LEFT-JOIN-miss all work together.

Fixed

• array_agg(atttypid) (and any oid[] column) carries the _oid array oid instead of falling through to varchar, so a client parses it as a list of oids rather than a string.

• A composite field whose type is itself a composite, enum, or range type appears in pg_attribute with that type’s own oid, instead of being dropped.

• A coalesce(col, '{}') over a LEFT-JOIN miss yields an empty array, not NULL and not the text "{}".

COPY in every format, and where NULL hides

COPY now works in all three of PostgreSQL’s formats, in both directions, and from a query as well as a table.

The three formats differ in more than punctuation. They differ in how they spell NULL, and that is where every bug in this work turned out to live. Text writes \N for NULL, so an empty field is an empty string. CSV writes NULL as an unquoted empty field, which forces it to quote the empty string as "" to keep the two apart. Binary writes a length of minus one, where an empty string is a length of zero. Each of those distinctions was broken at some point while writing this, and each broke the same way — NULL and empty string became indistinguishable, which is silent, survives a round trip in one format, and corrupts data in another.

The binary case is worth recording because the cause was ordinary and easy to repeat: a catch-all match arm sat above the NULL arm and swallowed it, rendering NULL as text and so writing a zero-length field. Match arms are tried in order, and a catch-all has to come after every case it must not absorb.

COPY (query) TO STDOUT reuses the ordinary query path rather than reading rows a second way, so anything a select can do — ordering, limits, a generated series — works inside a COPY too. That includes a query with no FROM at all: clients use copy (select 1) to stdout to check how a server reports a bad query, so refusing it failed a whole file of tests that were not about COPY.

Binary input decodes each value with the same code that decodes a bound binary parameter. The bytes on the wire are identical, so a second implementation could only drift from the first.

Added

• COPY ... TO STDOUT and FROM STDIN in text, CSV and binary formats.

• COPY (query) TO STDOUT, including queries with no FROM.

Fixed

• COPY ... TO STDOUT in text format wrote an empty field for NULL instead of \N, losing the distinction between NULL and the empty string.

COPY FROM STDIN on the Rust PostgreSQL server

Bulk loading with COPY table FROM STDIN now works, in PostgreSQL’s text format. This is how pgbench populates its tables and how most bulk-load tooling gets data in, so it matters out of proportion to the number of tests it moves.

The escaping is the substance of the format rather than a detail of it. A null value arrives as \N, which has to stay distinct from an empty string, and a tab inside a value arrives escaped so that it is not mistaken for the separator between fields. Data also arrives in chunks whose boundaries fall wherever the client’s buffer happened to end, including halfway through a row, so nothing can be parsed until the client says it has finished sending.

COPY TO STDOUT was still refused when this was written; it landed shortly afterwards, along with the CSV and binary formats — see the companion entry.

Added

• COPY <table> [(columns)] FROM STDIN, with \N nulls, escaped tabs, newlines and backslashes, an optional column list, and chunk boundaries that fall mid-row.

• PostgreSQL’s COPY n completion tag, and its errors for a row with the wrong number of columns or a value that will not parse.

Rust pgserver: CREATE TYPE AS RANGE

Custom range types work now: CREATE TYPE name AS RANGE (subtype = int4) registers the type in the catalog (a __sql_ranges__ collection paralleling the enum and composite catalogs), and a value casts, renders, and compares using the subtype’s own machinery — '[1,5)'::myrange. psycopg’s RangeInfo.fetch resolves the subtype through the existing pg_type / pg_range join, and DROP TYPE removes it. Unlike a builtin int4range, a custom range with no canonical function is not auto-canonicalised, matching PostgreSQL ('[1,4]' stays [1,4]).

Added

• CREATE TYPE … AS RANGE (subtype = …) DDL, catalog, and DROP TYPE.

• Casts, rendering, and comparison for custom range types over int/numeric/ date/timestamp subtypes; RangeInfo.fetch support via pg_range.

Cursors, and reading one in a loop

DECLARE, FETCH, MOVE and CLOSE now work on the Rust PostgreSQL server. A cursor is declared inside a transaction — outside one it would be closed again the moment the statement ended, and PostgreSQL refuses it for that reason — and its result is read at declaration time, which is what makes it scrollable in both directions afterwards.

PostgreSQL’s idea of where a cursor is has two positions that are easy to overlook, and both change the answers. The cursor sits on a numbered row, but it can also sit before the first one or after the last. Fetching past the end parks it after the last row rather than on it, so backing up two from there lands on the last row and not the one before it. Getting that wrong is off by exactly one, in the case people are most likely to try.

Three more rules that only a real server tells you: a backward fetch returns its rows nearest-first rather than in table order; RELATIVE and ABSOLUTE fetch a single row — the n-th from here, or the n-th from the start — where FORWARD and BACKWARD fetch a run of them; and FETCH ALL arrives as a count so large that any arithmetic on it has to be written not to overflow.

The bug worth naming is not in cursors at all. Clients prepare a statement they run repeatedly — psycopg after five times — and a prepared statement is described once and then executed. The describe path had no answer for FETCH, so it reported that the statement returned no columns; the sixth read in a loop then sent rows the client had no description for, which is a protocol violation rather than a wrong answer. Reading a cursor in a loop is the ordinary way to use one, so this was the normal case rather than an edge of it.

Added

• DECLARE ... CURSOR FOR, FETCH, MOVE and CLOSE, with PostgreSQL’s position model, reverse-order backward fetches, and the FORWARD / BACKWARD / ABSOLUTE / RELATIVE directions.

Fixed

• A prepared FETCH described no columns, so reading a cursor in a loop broke once the client prepared the statement.

Date, time and timestamp columns on the Rust PostgreSQL server

date and time now work as column types and as casts. They are stored in the same canonical text form the Python server uses — the two servers share one database, so the representation is a contract rather than a choice — but are reported over the wire with their real type identifiers, which is what lets a client hand back a date object rather than a string.

Writing a test that used date as a column type, rather than only as a cast, turned up something larger and unrelated to dates. A value assigned to a column was stored exactly as written instead of being converted to the column’s type, so INSERT INTO t (d) VALUES ('2026-9-1') stored 2026-9-1 and a client reading it back could not parse it. PostgreSQL converts on assignment; now so do we, for INSERT and UPDATE alike and for every type, not just dates.

Two error codes that are easy to conflate are kept apart, because PostgreSQL keeps them apart: a value that is not a date at all is one error, and a well-formed date naming a day that does not exist — the thirtieth of February — is another.

Timestamps needed more care than the other two. PostgreSQL keeps microseconds; the underlying document format keeps only milliseconds. The Python server already solved this by storing the truncated time and keeping the lost microseconds in a hidden companion field beside it, and this server now writes exactly the same thing — so a timestamp written by one is read back at full precision by the other. The rule that makes it safe is that every write must either set that companion or remove it: leaving a stale one behind would report a time nobody ever stored, which is worse than losing the precision would have been. Overwriting a precise time with a whole-millisecond one, and back again, is covered by a test for exactly that reason.

Added

• date, time and timestamp as column types and cast targets, accepting the spellings PostgreSQL accepts and storing the single canonical form it stores.

• Microsecond precision for timestamps, stored compatibly with the Python server so a value written by either is read correctly by both.

Fixed

• Values assigned by INSERT or UPDATE were stored as written rather than converted to the column’s declared type.

Rust pgserver: honour DateStyle in date/time text output

The Rust PostgreSQL server (secantusd-pg) now renders date, timestamp and timestamptz values in the session’s DateStyle — the four display formats (ISO, Postgres, SQL, German) crossed with the YMD / MDY / DMY field order — instead of always rendering ISO. A SET datestyle = SQL, DMY followed by SELECT '2026-09-08'::timestamp now answers 08/09/2026 12:34:56, byte-for-byte what PostgreSQL 14 answers, and Postgres style gains the spelled-out day-of-week and month (Tue Sep 08 12:34:56.789 2026) it requires.

Because the output now genuinely respects the style, the server once again reports DateStyle over ParameterStatus (in PostgreSQL’s canonical spelling) — which a previous change had to disable, since reporting a style the output ignored made psycopg switch its parser to a layout the server never produced and mis-parse every datetime. Reporting is safe now that the two agree: psycopg’s loaders and the server’s output speak the same style. Binary datetime output is DateStyle-independent and is left untouched.

Added

• secantus-pgplan: DateStyle (format + field order) with parse / canonical, and render_date_styled / render_timestamp_styled / render_timestamptz_styled (plus the *_value_text_styled Bson helpers).

Fixed

• secantus-pgserver: date / timestamp / timestamptz TEXT output renders in the session DateStyle; SET datestyle is stored and reported in canonical form. Fixes the 12 psycopg test_overflow_message[timestamptz-*] cases (a non-ISO style must make the driver’s timestamptz loader raise NotImplementedError).

A table you have just created, in a transaction that has not ended

Creating a table and then using it, without committing in between, reported that the relation did not exist. That is the ordinary shape of a test fixture and of a migration, and it accounted for 184 failures in psycopg’s suite by itself.

The cause is worth stating because of what hid it. Resolving a table name is part of planning, planning reads the catalogue, and the catalogue is an ordinary table — so a CREATE TABLE that had not committed was invisible to it. Execution already ran inside the transaction, so anything that used a table which already existed worked perfectly. Only the combination failed, and only for a client that had not committed.

Fixing it exposed a second problem underneath, which had never been reachable: COPY inside a transaction wrote its rows outside that transaction, where they blocked against its own locks and hung the connection outright. Rows from a COPY now go through the transaction like every other write.

ORDER BY 1 also works now. It means the first output column — an ordinal into the select list, so select b, a from t order by 1 orders by b — rather than the constant one, and a position with no such column gets PostgreSQL’s own error for it.

One thing this does not fix, and it is worth being plain about: rolling back does not undo the CREATE. DDL is not transactional on this server, where in PostgreSQL it is. That needs schema operations to participate in the transaction down in the storage layer, and it is recorded as an open divergence rather than quietly left to be discovered.

Fixed

• A table created in a transaction was invisible to later statements in the same transaction, and one dropped in a transaction was still found.

• COPY inside a transaction deadlocked the connection.

Added

• ORDER BY <position>, with PostgreSQL’s error for a position that is not in the select list.

DROP TABLE and type casts on the Rust PostgreSQL server

DROP TABLE, DROP TABLE IF EXISTS and casts ('1'::int, 1::text, $1::float8) now work. Both came straight off the ranked list that psycopg’s test suite produced when it was first pointed at the Rust server — measuring against someone else’s tests turns out to be a much better guide to what to build next than deciding for oneself.

Casts brought a subtler problem than converting values. A client asks what columns a query returns before it supplies any parameter values, so a column’s type cannot be read off the value it happens to hold — at that point $1::int has no value at all. Inferring from the value typed that column as text, and the client then decoded a perfectly good integer as a string. Types now come from the cast that declares them. The same fix corrected text columns, which were being reported as varchar: PostgreSQL treats those as different types, and while Python clients decode both to strings, the Java and Go drivers do not.

The gauge moved from 694 to 746 of psycopg’s 4,238 tests. The modest jump is itself informative: a test that was blocked by a missing DROP TABLE usually needs several other things too, so removing one obstacle mostly reveals the next. Expressions in a SELECT list are now the single largest blocker.

Added

• DROP TABLE, including IF EXISTS and multiple tables in one statement.

• Casts to the integer, floating-point, boolean and text types, with PostgreSQL’s invalid input syntax error for values that cannot convert.

Fixed

• text columns were reported over the wire as varchar.

Rust pgserver: enum columns report their own oid

A SELECT of an enum COLUMN was described over the wire as varchar (oid 1043) instead of the enum’s own type oid. psycopg reads that oid to decide whether to apply a registered enum loader, so with varchar it handed back a bare string where a register_enum’d client expected the Python enum member — the enum value-mismatch and non-ASCII case-fold failures. The column-description paths now consult the user-type catalog (as the parameter path already did), so an enum column — scalar or array — carries the enum’s minted oid, and the label is returned verbatim.

Fixed

• Enum columns (and enum-array columns) are described with the enum type’s oid, not varchar, so psycopg’s registered enum loader applies and returns the enum member.

Types you make yourself

CREATE TYPE ... AS ENUM and DROP TYPE work on the Rust PostgreSQL server — which matters more than it sounds, because psycopg’s entire enum test file (207 tests) died in one session fixture running exactly that DDL, and nothing behind it was even measurable.

The enum catalog is written in the Python server’s representation, because the two servers share one store and the doc shapes, collection names and oid-minting rule are a contract, not an implementation choice: oids are minted monotonically from a shared counter and never reused (renumbering types would strand any client that registered a decoder by oid), and a type’s array oid is derived — its own oid plus 100 000 — never stored. An enum created by one server resolves on the other under the same oid, and the next type minted by either continues the same sequence.

Enum values work too: 'sad'::mood validates the label with PostgreSQL’s own error for a miss (invalid input value for enum mood: "nope"), the column carries the enum’s minted oid so a client that registered the type decodes it, pg_typeof answers the type, and a case-sensitive name renders quoted through regtype ("CamelCase"), exactly as measured. The catalog reads behind psycopg’s TypeInfo.fetch see user enums beside the builtins.

DROP of an unsupported object kind also stopped leaking Rust debug formatting into its message — “DROP of Ok(ObjectType)” is now “DROP of a schema” and friends, naming the kind.

Added

• CREATE TYPE ... AS ENUM, DROP TYPE [IF EXISTS], enum value casts, enum rows in pg_type / to_regtype / regtype, cross-server with shared oids.

Fixed

• DROP of a non-table leaked a protobuf enum’s debug form into the message.

Registering an enum, end to end

EnumInfo.fetch works now — psycopg’s own enum-discovery API, unmodified — and with it the whole slice of the driver that hangs off it: fetching a type’s labels, registering a Python enum against it, casting values in and out.

The query it sends is the reason this took a batch of its own: a FROM subquery whose body is a LEFT JOIN of pg_type to pg_enum, with array_agg and GROUP BY on the outside. So this adds a joined subquery as an aggregate source, a two-table join (inner or left, one ON equality, an optional filter and one ORDER BY — the shape every catalog query actually uses, nothing more), the pg_enum virtual table, and array_agg — which keeps NULLs, the way a left-join miss surfaces as [None] rather than [] and lets a client tell a non-enum apart from an empty one.

Enum values round-trip in both wire formats: a label’s binary form is just its UTF-8, so binary cursors get the format they asked for, and an unspecified-oid parameter carrying an enum label decodes as that label. An enum array reports its own derived array oid rather than varchar, so a client that registered the array type decodes it.

Added

• EnumInfo.fetch: a joined subquery as an aggregate source, pg_enum, array_agg, and a two-table LEFT/INNER join limited to the catalog query shape.

• Enum values in both wire formats, and enum arrays with their real oid.

Arithmetic and string expressions in the Rust PostgreSQL server

SELECT 1+1, SELECT 'a'||'b', SELECT 7/2 and their relatives now work. Expressions in a select list were the single largest thing standing between the Rust PostgreSQL server and psycopg’s test suite, and the score moved from 746 to 853 of 4,238 — the largest jump so far.

The corners were measured against a real PostgreSQL rather than assumed, and two of them are easy to get wrong. Integer division truncates, so 7/2 is 3 rather than 3.5, and dividing by zero is an error rather than a null or an infinity. Adding anything to NULL gives NULL, and concatenating a number to a string converts the number.

Arithmetic on decimals is deliberately still refused. PostgreSQL treats 1 + 1.5 as its numeric type with particular scale rules, not as a floating-point number; returning a double would give the right value under the wrong type, and that is exactly the class of bug that recently made a correctly-converted integer arrive at the client as a string. Explicit floating-point casts work, because then the type genuinely is floating point.

Added

• Arithmetic (+ - * / %), string concatenation (||), comparison operators and unary minus in a SELECT list, including over bound parameters.

• PostgreSQL’s division by zero and integer out of range errors.

Rust pgserver: only report the TimeZone GUC, not DateStyle

The timestamptz-columns change began emitting a ParameterStatus for several GUC_REPORT variables on SET, including DateStyle. But this server always renders dates in ISO regardless of DateStyle, so reporting a DateStyle change made the client (psycopg) switch its date parser to a style our output never uses — mis-parsing every datetime. Reporting is now limited to TimeZone, the one GUC whose change the output actually honours (a timestamptz renders in it).

Fixed

• SET datestyle no longer breaks datetime parsing: the server reports only TimeZone via ParameterStatus, not GUCs it does not honour in output.

The Rust PostgreSQL server can be scored against psycopg’s own test suite

SecantusDB’s conformance gauges run real client libraries’ own test suites, unmodified, against the server. Until now the SQL gauges could only measure the Python PostgreSQL server; setting SECANTUS_GAUGE_SERVER=rust now points the psycopg gauge at the Rust one instead, so the same tests score both.

Getting there needed two things that only a real client asks for. Clients check which server they have connected to before doing anything else, and the gauge refuses to score a server that does not identify itself — so SELECT version() and the other session functions had to work, which meant supporting a SELECT with no table at all. Clients also wrap their work in transactions, so BEGIN, COMMIT and ROLLBACK had to work, backed by real storage transactions rather than accepted and ignored: a ROLLBACK that quietly kept the changes would be worse than refusing the statement.

The first score is low, and deliberately published rather than buried: 694 of psycopg’s 4,238 tests pass. Every measurement before this compared the server against expectations written alongside it; this is the first one where someone else’s tests decide. The ranked list of what they trip over is the useful part.

Added

• SELECT without a FROM clause, and the version(), current_database(), current_schema() and current_user session functions.

• BEGIN / COMMIT / ROLLBACK, backed by real storage transactions.

• SECANTUS_GAUGE_SERVER=rust support in the psycopg gauge, writing its results to a separate file so the two servers’ scores cannot overwrite each other.

generate_series, and the rows a cursor needs to scroll over

generate_series(start, stop) — with an optional step, which may count downwards — now works as a source in FROM. It is a source rather than a statement of its own, and that is the whole design: ORDER BY, LIMIT, OFFSET and the aggregates all operate on the generated rows without knowing where they came from, so nothing had to be reimplemented for them.

Two rules are worth stating because they read the other way round. Counting up towards a smaller stop produces nothing — generate_series(5, 1) is empty rather than reversed, and counting down needs a negative step. And a step of zero is refused rather than looped over, with the code PostgreSQL uses for an argument whose value cannot work, which it keeps distinct from its general data-error class.

The reason this was worth doing now is not the function itself. Cursors landed in the previous release, complete and matching PostgreSQL across every operation probed — and almost every test that used one still failed, because the usual way to give a cursor rows to scroll over, without inventing a table first, is to select from a generated series. A feature can be finished and still be unreachable.

A WHERE clause over a generated source is refused rather than ignored. The filter machinery here is built against stored columns, and quietly dropping a predicate would return rows the client asked to exclude.

Added

• generate_series as a FROM source, with an optional and possibly negative step, column aliases (AS g, AS g(x)), ORDER BY, LIMIT, OFFSET, and count / sum / min / max over it.

Rust pgserver: inet and cidr

The Rust PostgreSQL server now supports the network address types inet (oid 869) and cidr (oid 650) — as casts, bound values, and real column types. Both are carried as the canonical addr/masklen text the Python server stores, so the two servers share one representation. inet keeps its host bits; cidr is strict and rejects any host bit set below the netmask. IPv4 and IPv6 are both handled, with IPv6 compressed to its canonical form.

psycopg sends and reads these types in the binary wire format, so the round trip goes through PostgreSQL’s [family][bits][is_cidr][nb][addr] layout in both directions: a /32 (or /128) host comes back as an IPv4Address / IPv6Address, a shorter prefix as an Interface, and a cidr as a Network, exactly as against a real server. The ::text cast keeps the mask (network_show), while an inet column read in text drops a full-host mask (inet_out).

Added

• inet (869) and cidr (650) casts, bound values (text + binary), and column types; malformed input is 22P02, and a cidr with host bits set is 22P02 invalid cidr value.

Intervals, and why they refuse to be one number

An interval is a duration, and PostgreSQL keeps it as three separate numbers: months, days and microseconds. That looks like an implementation detail until you try to collapse it. A month is 28 to 31 days depending on where you start, and a day is 23, 24 or 25 hours across a daylight-saving boundary — so 2026-01-31 plus one month is 2026-02-28, a result no fixed count of microseconds can express. Adding thirty days to the same date lands on March 2nd instead, and both answers are correct for what was asked.

Comparison is the exception, and it goes the other way: PostgreSQL flattens the parts using thirty-day months and twenty-four-hour days, so '1 mon' and '30 days' compare equal while adding them takes you to different dates. The Rust PostgreSQL server now does both — ordering through the flattened value, arithmetic through the parts.

Intervals arrive in three written forms, all of which are now accepted: the verbose one (1 year 2 months, with abbreviations and a week that becomes seven days), a bare time (02:03:04.5, which carries its own sign and may run past twenty-four hours), and ISO 8601 (P1Y2M3D, where M means months before the T and minutes after it). They combine, and each part keeps its own sign: 1 day -02:03:04 is a positive day and a negative time.

The output is PostgreSQL’s, including the spelling that looks wrong: a value pluralises whenever it is not exactly one, so -1 day prints as -1 days.

Added

• interval as a cast target, a literal and a bound parameter in both wire formats, with its own type oid.

• Interval comparison and ordering, flattened as PostgreSQL flattens it.

• timestamp ± interval with end-of-month clamping, interval ± interval, and scaling an interval by a number, where a fractional result spills from months into days and from days into time.

Fixed

• Beside an interval, a bare unquoted-type literal now resolves to an interval rather than to a timestamp, as PostgreSQL resolves it — so '2020-01-01' + interval '1 day' reports a bad interval instead of quietly doing date arithmetic, and '1 day' + interval '1 day' is two days.

User types are visible in the transaction that creates them

A user type created on the Rust PostgreSQL server is now visible to later statements in the same transaction, before it is committed. Previously CREATE TYPE t AS (...); SELECT 't'::regtype in one transaction failed with 42704 type "t" does not exist, because planning resolves type names against the catalog and reads it OUTSIDE the open transaction, so an uncommitted CREATE TYPE was invisible to the read that needed it.

This is the transaction-visibility gap that tables already closed, now closed for types. It mattered far beyond one statement: psycopg’s composite, enum, and range test fixtures run on a non-autocommit connection and create a type then immediately use it — CompositeInfo.fetch, EnumInfo.fetch, and RangeInfo.fetch all query the catalog in that same transaction — so the invisible type returned None and cascaded into TypeError: no info passed and FeatureNotSupported across the whole composite/enum/range gauge surface. A per-connection overlay of the transaction’s uncommitted type creates and drops, consulted before the committed catalog exactly as the table one is, makes the type resolve, cast, and fetch inside its own transaction; ROLLBACK discards it, COMMIT persists it, and ROLLBACK TO SAVEPOINT undoes a type created after the savepoint. Wrapping the catalog read in the transaction was rejected for the same reason the table fix rejected it — it deadlocks COPY, which opens its own transaction context.

Fixed

• A CREATE TYPE (composite, enum, or range) issued inside a transaction is now visible to later statements in that transaction: to_regtype, a value cast to the type, and psycopg’s CompositeInfo / EnumInfo / RangeInfo .fetch helpers all resolve it before it is committed.

• DROP TYPE inside a transaction hides the type from later statements in the same transaction, before the drop is committed.

• ROLLBACK and ROLLBACK TO SAVEPOINT correctly discard a type created in the rolled-back span, and COMMIT persists one; the type-catalog collections are captured by savepoint pre-images so a rolled-back create cannot survive a later commit.

Rust pgserver: multi-predicate JOIN/subquery WHERE

A JOIN (or aggregate-subquery) WHERE clause now accepts several ANDed predicates on either side — WHERE t.oid = $1 AND a.attnum > 0 AND NOT a.attisdropped — with the operators =, >, >=, <, <=, and NOT <boolcol>, where before only a single left-side equality was allowed. A right-side predicate now filters the right rows (correct for the INNER joins these catalog queries use) rather than being refused.

This is the first piece of the composite CompositeInfo.fetch campaign (its inner subquery joins pg_attribute to pg_type with exactly this multi-predicate WHERE); the remaining piece — a subquery as a LEFT-JOIN side — is tracked in tasks/backlog.md.

Added

• = / > / >= / < / <= / NOT <bool> predicates, ANDed, on either side of a JOIN’s WHERE (JoinSelect.filter is now a Vec<JoinPred>).

Reaching into a json document

'{"a": 1}'::json ->> 'a' was an error. So was every other way of getting at part of a json value: the Rust PostgreSQL server could parse json, store it, cast it and hand it back whole, and could not reach inside it.

All of the navigation and key operators work now — -> and ->> by name or by array index, #> and #>> down a path, ?, ?| and ?& for keys, and @> / <@ for containment. A negative index counts from the end, which is PostgreSQL’s rule and not most JSON libraries’; a lookup that does not apply — a missing key, an index past the end, a name against an array — is SQL NULL rather than an error, which is the whole reason these operators are usable; and ->> reads a json string without its quotes while a json null becomes a SQL NULL.

Containment compares by value rather than by text, so key order and whitespace do not count, and neither does a number’s scale: {"a": 1.0} contains {"a": 1}.

A json value is carried here as its text, so by the time two operands reach the evaluator there is nothing to tell {"a": 1} from any other string. The left operand’s static type is what makes these json operators at all — the same mechanism that resolves a range parameter from the operand beside it.

Added

• ->, ->>, #>, #>>, ?, ?|, ?&, @> and <@ over json and jsonb, with their result types (-> keeps the json flavour, ->> is text, the key and containment tests are boolean).

json and jsonb, and the number that gives the difference away

json and jsonb look interchangeable and are not. json validates its input and stores the text it was given, so whitespace, key order and even duplicate keys all survive a round trip. jsonb stores a parsed structure, so what comes back is normalised: keys sorted, the last of any duplicate pair kept, one canonical spacing throughout.

Key order is the part worth knowing. jsonb sorts keys by byte length first and only then bytewise, so z comes before é — one byte against two — and b comes before aa. It is neither alphabetical nor by character count, and no amount of reasoning gets you there; it was measured.

Numbers are where a shortcut would have shown. A jsonb number is a numeric, and prints as one: an exponent expands, so -1.5e10 comes back as -15000000000, while a trailing zero written in the literal survives, so 1.10 stays 1.10 rather than becoming 1.1. That second half is what rules out reading numbers into floating-point values, which every general-purpose JSON parser does by default — it gets the exponent right and silently drops the zero. Number text is therefore kept as written and normalised the way numeric is.

One related fix came out of testing this. A bound parameter whose type the client leaves unspecified is guessed from its text, and the guess used to accept 01 as the number 1 — so '01'::json, which PostgreSQL rejects, was quietly turned into valid JSON before the cast ever ran. A guess made on the client’s behalf must never make a value more acceptable than the client wrote it, so a number is now only inferred when it round-trips to the same text.

Added

• json and jsonb as cast targets, literals and bound parameters in both wire formats, with their own type oids.

Fixed

• An unspecified-type parameter was inferred as a number even when the text was not how that number is written, which could turn invalid input valid.

Rust pgserver: multidimensional arrays

Multidimensional arrays now cross the wire. They already stored and text- rendered, but returning a nested array to a client was refused — so SELECT ARRAY[[1,2],[3,4]], reading a 2-D int[] column, and casting a nested text literal all failed. The server now builds PostgreSQL’s multidimensional array binary wire form by hand (an ndims header, per-dimension bounds, and the leaves in row-major order) and also emits the {{1,2},{3,4}} text form, so a column reaches the client correctly whether the cursor asked for text (the psycopg default) or binary. A nested ARRAY[...] constructor now reports the array type (oid 1007), not varchar; the array cast recurses through the nesting; and a ragged array is rejected exactly as PostgreSQL rejects it — 2202E from a constructor, 22P02 from a text literal.

Added

• Multidimensional array results (2-D, 3-D, …) in both the text and binary wire formats, for int / bigint / smallint / float / numeric / bool / text element types.

• A nested array constructor is typed as its array type (oid 1007), and the array cast handles nested elements.

Fixed

• A ragged multidimensional array is now rejected (2202E / 22P02) instead of being silently accepted.

Multiranges, and the difference between touching and overlapping

A multirange is a set of ranges, and PostgreSQL keeps it in one normal form: members sorted, empty ones dropped, and any two that meet folded into a single member. So {[10,20),[1,5)} comes back sorted, {empty} comes back as {}, and {[1,5),[3,8)} comes back as {[1,8)}.

The interesting rule is the last one, and it is not quite “overlapping”. {[1,5),[5,8)} also collapses to {[1,8)} — the two do not overlap at all, but nothing lies between them either, so they are one continuous stretch. {[1,5),[6,8)} stays two members, because 5 is missing from it. The test is whether the next member starts at or before the previous one ends, and at the exact meeting point it comes down to the bounds: over a continuous type, [1.0,2.0) and [2.0,3.0) join, while [1.0,2.0) and (2.0,3.0) do not, because the second leaves 2.0 out.

Members are canonicalised before any of this happens, so {[1,5]} is stored as {[1,6)} and merging sees the same bounds a client would.

All six multirange types are supported as literals, constructors, cast targets and bound parameters, each with its own type oid.

Added

• int4multirange, int8multirange, nummultirange, datemultirange, tsmultirange and tstzmultirange, with sorting, empty-member removal, merging of overlapping and adjacent members, and their own type oids.

Several commands in one query, and a flake that was a missing feature

PostgreSQL’s simple query protocol takes any number of commands separated by semicolons and answers with one result each — create table ...; insert ...; select ... in a single round trip. The Rust PostgreSQL server refused the whole string, which is why so much client code failed before reaching the query it cared about: the setup was the batch.

The part that cannot be added later is the transaction. A multi-command batch runs as one implicit transaction, so a failure in the third command discards what the first two wrote — and an explicit COMMIT inside the batch ends that transaction, so whatever it committed survives a later failure. Both rules were measured against a live PostgreSQL rather than assumed, and both fall out of reusing the session’s own transaction slot instead of tracking a second one.

DEALLOCATE ALL is now accepted too, and it is worth saying why it mattered. Clients issue it to reset their prepared-statement cache, but only when the connection happens to have one — so refusing it failed a scattered handful of tests that varied from run to run. That looked exactly like flakiness, and it was a missing feature the whole time.

pg_typeof(x) answers the type’s display name — integer rather than int4, timestamp without time zone rather than timestamp — reported as a regtype. It reports the type the server would give the expression, not one read off the value, which is why pg_typeof(NULL) is unknown.

Added

• Multi-command simple queries, one result per command, run as a single implicit transaction with PostgreSQL’s commit and rollback behaviour.

• DEALLOCATE ALL.

• pg_typeof(), and the type display names behind it.

Fixed

• Casting a decimal to a float failed outright (1.5::float8 raised “invalid input syntax”), and casting one to an integer failed the same way. Decimal literals became exact numerics in the previous release and these two cast paths were never taught about them.

• 2.5::float8::int answered 3. PostgreSQL rounds float-to-integer half to even and numeric-to-integer half away from zero; one rule was being used for both. A large numeric is now rounded on its digits rather than through a float, which cannot represent every value a numeric can hold.

Changed

• Several commands in one prepared statement now raise PostgreSQL’s own 42601 “cannot insert multiple commands into a prepared statement” instead of reporting the batch as an unsupported feature. The extended protocol has one parameter list and one row description, so this is a real error rather than a gap.

• An unsupported function now names itself (function chr() is not supported yet), where every one of them used to report the same FuncCall.

Exact decimal numbers on the Rust PostgreSQL server

PostgreSQL’s numeric type is for numbers that must be exact — money, quantities, anything where a rounding error is a bug rather than a rounding error. It is not a floating-point type, and the difference shows in two ways that matter to a client: 1.50 and 1.5 are different values, and reading one back gives a decimal rather than a float.

The Rust PostgreSQL server had been refusing decimals rather than storing them as floats, on the grounds that returning the right magnitude under the wrong type is worse than declining — the same reasoning that had a cast integer arriving at a client as text earlier in this work. That refusal can now be lifted: decimals are stored in a format that keeps the exact digits and the trailing zeros, and are reported as the type they are.

There is a limit. PostgreSQL’s decimals have no fixed size; the storage format here holds 34 significant digits. A number needing more is refused rather than quietly rounded, because a silently shortened number is indistinguishable from a correct one.

Added

• numeric as a column type, a cast target, and the type of a decimal literal, keeping exact digits and trailing zeros (1.50 stays 1.50).

Fixed

• A decimal literal such as 1.5 was reported as a floating-point number. PostgreSQL reports it as numeric, and clients read that to decide whether they get an exact decimal or a float.

A PostgreSQL server written in Rust, sharing one database with the Python one

SecantusDB already speaks the PostgreSQL wire protocol, and it already has a Rust server for the MongoDB side. This is the first slice of the third: a PostgreSQL server written entirely in Rust, with no Python anywhere in the request path. It handles CREATE TABLE, INSERT and single-table SELECT today — a deliberately thin slice, because the point of it is to prove the architecture end to end rather than to be useful yet.

The part that matters is that the two servers share one database. A table created by the Rust server can be read and written by the Python server, and a table created by the Python server can be read by the Rust one, because both write the same catalog documents into the same WiredTiger store. That contract is pinned by golden vectors captured from the Python server, since a catalog written subtly wrong by one server is not an error the other reports — it is a table with the wrong columns.

SQL is parsed by PostgreSQL’s own parser, statically linked, rather than by a general-purpose SQL parser. That is a correctness decision as much as a performance one: the shapes SecantusDB currently has to work around, like COPY … WITH (freeze on), parse correctly by construction. Anything the new server cannot yet handle answers the PostgreSQL error code for “feature not supported” rather than guessing, so an unsupported query is always a refusal and never a wrong answer.

Added

• secantus-pgcatalog, secantus-pgplan and secantus-pgserver crates, plus the standalone secantusd-pg binary.

• invoke rust-pgserver-build and invoke rust-pgserver-test; the latter is now part of invoke rust-gate.

• Cross-server round-trip tests covering both directions, PostgreSQL-oracle checked predicates, and the error codes for unknown columns, unknown tables, duplicate tables, duplicate keys and unsupported constructs.

Fixed

• A duplicate primary key reported MongoDB’s E11000 duplicate key error through the PostgreSQL connection. It now reports what PostgreSQL reports, down to the DETAIL: Key (id)=(1) already exists. line.

The Rust PostgreSQL server learns ORDER BY, aggregates, UPDATE, DELETE and SQL’s three-valued logic

The Rust PostgreSQL server now handles ORDER BY (with LIMIT and OFFSET), UPDATE, DELETE, and the predicates that make SQL SQL: IS NULL, IN, NOT IN, BETWEEN and NOT. All of it is checked against a real PostgreSQL rather than against our own idea of what PostgreSQL does — a new differential suite runs 83 identical statements against both and compares the answers.

That comparison earned its keep immediately, because SQL and MongoDB disagree about NULL in ways that are easy to miss and produce wrong rows rather than errors. PostgreSQL sorts NULLs last when ascending and first when descending, while MongoDB sorts them low; pushing a sort down to the storage engine would have quietly reordered every nullable column. n <> 1 must not return a row whose n is NULL, because comparing anything to NULL yields NULL rather than true — but MongoDB’s equivalent operator matches it. x NOT IN (1, NULL) returns nothing at all, for the same reason. Each of those is now handled explicitly, and each has a test that would have caught it.

NOT is pushed down into the individual comparisons rather than wrapped around them, since MongoDB has no operator that means what SQL’s NOT means. De Morgan’s laws hold in SQL’s three-valued logic, so the transformation is exact. Anything the server still cannot express — joins, aggregates, LIKE, sorting by an expression — continues to answer PostgreSQL’s “feature not supported” rather than guessing at an answer.

Aggregates arrived with the same care. count(*) counts rows including those whose columns are all NULL, while count(col), sum, min and max skip NULLs — and over an empty result every one of them except count returns NULL rather than zero. A NULL forms its own GROUP BY group. avg is deliberately absent: PostgreSQL returns it as numeric with particular scale rules, and a close-enough answer would be worse than an honest refusal.

Added

• count(*), count, sum, min and max, with or without GROUP BY, including PostgreSQL’s result types (count and sum are bigint; min and max take the column’s own type).

• ORDER BY with per-column direction and NULLS FIRST / NULLS LAST, including PostgreSQL’s direction-dependent defaults; LIMIT and OFFSET.

• UPDATE and DELETE, with PostgreSQL’s row counts (UPDATE reports rows matched, not rows whose value changed).

• IS NULL, IS NOT NULL, IN, NOT IN, BETWEEN, NOT BETWEEN and NOT.

• A differential test suite comparing 109 statements against a live PostgreSQL.

Fixed

• <> returned rows whose column was NULL, where PostgreSQL excludes them.

• Selecting the same aggregate name twice (SELECT count(*), count(n)) reported the second result in both columns.

Parameterised queries work against the Rust PostgreSQL server

Client libraries switch to PostgreSQL’s extended query protocol the moment a query carries a parameter — WHERE n > %s rather than WHERE n > 5. The Rust PostgreSQL server only implemented the simple protocol, so those queries were answered with a success status and no rows at all. That is worse than an unimplemented feature: an application would have seen an empty result for a query that should have returned data, with nothing to indicate anything had gone wrong. Prepared statements, portals, parameter binding and Describe are now implemented, so parameterised queries, prepared statements and the row limits that clients set on execution all behave.

Adding them immediately exposed a second, quieter bug that had nothing to do with parameters. Comparing anything to NULL in SQL is never true — WHERE n = NULL returns no rows, even for rows where n really is NULL, because only IS NULL tests for it. The server was treating that comparison as a match. It went unnoticed because every existing test wrote its comparisons as literals, and it took binding a NULL parameter to make the case obvious enough to write down.

Both protocols now run through one shared execution path, so they cannot drift apart, and the differential suite that compares the server against a real PostgreSQL grew to 141 statements — 28 of them over the extended protocol.

Added

• The PostgreSQL extended query protocol: Parse, Bind, Describe, Execute and Close, with parameters in both text and binary form, and support for the row limit a client can set on execution.

Fixed

• Comparing a column to NULL (= NULL, <> NULL, > NULL) returned rows where PostgreSQL returns none. Affected literal SQL as well as parameters.

The type a parameter was sent as

A bound parameter carries a type the client declared, and this server was throwing it away — reading each parameter’s meaning back out of its decoded value instead. Most of the time the two agree. Where they don’t, the answers were wrong in ways that print correctly.

pg_typeof(%s) with a small integer said integer, because that is what the value looks like; PostgreSQL says smallint, because that is what psycopg declared. And a parameter with no declared type at all has no type to report: PostgreSQL answers an error rather than guessing, where this server guessed text. The declared types now reach the planner, on the describe path as well as the execute one — the describe runs first, so a describe that did not know them answered for the whole statement.

Ranges had a sharper version of the same problem. A range over a discrete element type has one true spelling — PostgreSQL rewrites every bound, so [10,20] is stored and printed as [10,21) — and a range parameter kept whatever the client wrote. int4range(10, 20, '[]') = %s with that very range bound was false, while both sides printed identically. Two routes needed fixing: a range parameter that arrives with its type now decodes through the same cast a literal takes, and one that arrives untyped takes its type from the operand beside it, which is what PostgreSQL does at analysis time.

Added

• Declared parameter types reach the planner, so pg_typeof reports them.

• 42P18 for a parameter whose type neither the client nor the context gives.

Fixed

• A range or multirange bound as a parameter compared unequal to the same range written any other way.

A catalog to ask about types

psycopg discovers a type by sending one query — pg_type joined to to_regtype(), with a ::regtype::text cast in the select list — and the Rust PostgreSQL server had none of its five ingredients: no pg_type, no to_regtype, no regtype cast worth the name, no table aliases and no cast of a column in a select list. Type discovery failed wholesale, and with it the whole family of client features built on it — registering an enum, a composite, a custom range.

pg_type is now a virtual table: a definition and rows computed on read from the same builtin-type catalog that names oids everywhere else, so the two can never disagree. to_regtype() resolves either spelling of a name (int4 or integer) and answers NULL — not an error — for one it does not know, which is the whole reason clients prefer it to the ::regtype cast, which errors.

A regtype value itself carries two natures no single scalar holds: it prints as the type’s display name while comparing as its oid. select to_regtype('text') shows text; where t.oid = to_regtype('text') compares 25. Casting one onward follows the same split — to text as the name, to an integer as the oid.

The vehicle for the oid::regtype::text select item — a chain of casts applied per column of a table read — works for any casts, not just these, and the described column type follows the last cast in the chain, so a client decodes the rows it was promised. pg_prepared_statements rides along as an empty virtual table, which is what psycopg’s pipeline tests count rows in.

The oid type came with it — psycopg’s numeric tests cast to it constantly — with PostgreSQL’s own edges: a negative literal wraps ((-1)::oid is 4294967295), a value past 2³²−1 is out of range rather than wrapped, and the binary format is the 4-byte unsigned form.

Added

• The pg_type and pg_prepared_statements virtual catalog tables.

• The oid type: casts from integers and text, both wire formats, unsigned wrap-around and range errors as PostgreSQL reports them.

• to_regtype(), a real regtype (prints as a name, compares as an oid), and cast chains over columns in a table select list.

• Table aliases (FROM pg_type t ... WHERE t.oid).

The Rust PostgreSQL server moves to a current wire-protocol library

The library that speaks the PostgreSQL wire protocol was pinned to a version from nine releases ago. Two limitations that had been written down as costs of using that library — errors that could not carry the name of the constraint they violated, and no way to send data for COPY ... TO STDOUT — turned out to have been fixed upstream months earlier. They were costs of the pin, not of the library.

With the upgrade, both are closed. A duplicate key error now carries the same constraint, table and schema names PostgreSQL sends, which is what the Java driver reads when an application asks which constraint failed. COPY table TO STDOUT produces output byte-identical to PostgreSQL’s, so data copied out of one server loads straight into the other.

The lesson is worth more than the features: a dependency pinned below 1.0 stops receiving even compatible updates, and nothing announces that. Checking for a newer release takes seconds and should happen before limitations get written down as permanent.

Added

• COPY <table> TO STDOUT in text format, round-tripping with COPY FROM.

• Constraint, table and schema names on duplicate-key errors.

Changed

• The wire-protocol library moves from 0.31 to 0.40, clearing its deprecated calls at the same time.

Rust pgserver: multirange arrays report their element type

An ARRAY of multiranges was typed as varchar on the Rust PostgreSQL server, where an array of ranges already carried its real element type. That broke the value in BOTH wire formats: in text the client read back a bare string like {{[1,5)},{}} instead of parsing it into Multirange objects, and in binary a varchar column stays on the binary path — where an array value cannot be sent as a binary varchar at all (22P03). The six multirange array types now report their own array oids (int4multirange[] is 6150, and so on), which keeps them, like range arrays, on the text-format path the row description already downgrades a non-binary-encodable type onto — so the client parses them.

The scalar range/multirange binary wire codec was already correct: typed empty, unbounded and populated range/multirange values round-trip in binary against a real server; the only remaining range gaps are CREATE TYPE ... AS RANGE (custom range types) and array-of-range element comparison, both separate work.

Fixed

• ARRAY-of-multirange results now report the multirange’s array type (int4multirange[] = 6150, int8multirange[] = 6157, nummultirange[] = 6151, datemultirange[] = 6155, tsmultirange[] = 6152, tstzmultirange[] = 6153) instead of varchar, in both wire formats.

• Binary parameters of those array oids decode through the multirange element decoder.

RangeInfo.fetch, and a plain join

RangeInfo.fetch works now — psycopg’s range-type discovery, unmodified. Its query is pg_type joined to pg_range on rngtypid, but without the aggregate wrapper EnumInfo uses: a plain top-level JOIN in an ordinary SELECT. So the join source, which the enum work put on the aggregate planner, now lives on plain selects too, and pg_range joins the virtual catalog — six rows, each builtin range type paired with its element type’s oid, read from the same table the range casts use so the two can never disagree.

Added

• The pg_range virtual table (rngtypid, rngsubtype).

• A top-level two-table JOIN in a plain (non-aggregate) SELECT, so RangeInfo.fetch resolves.

The Rust PostgreSQL server passes psycopg’s range and multirange suites

psycopg’s test_range.py and test_multirange.py drive the whole range family through the wire in every parameter format: lists of ranges bound as int4range[], untyped Range(empty=True) values that arrive as a single binary flag byte, custom CREATE TYPE ... AS RANGE types registered through RangeInfo.fetch, and a quoting sweep over every awkward bound character. The Rust PG server failed 72 of those tests; it now fails one, a reserved-keyword quoting nit that is not about ranges at all. Every behaviour was measured against PostgreSQL 16 and matched exactly — values, canonical renderings, error messages and SQLSTATEs.

Four things were wrong. A range-array parameter had no name for its oid, so a literal beside it was compared as a string against an array. An untyped range parameter bound in binary was read as text because nothing inferred its type from the operand it was compared with (or from a $1::int4range cast). Custom range types could be created but their values could not travel: the constructor, the binary codec, and the schema-qualified name each resolved to the wrong type or to text. And the literal parser and renderer disagreed with PostgreSQL’s range_out on doubled quotes, backslashes, non-ASCII whitespace, and the always-exclusive infinite bound.

Added

• secantus-pgplan: lower / upper / lower_inc / upper_inc / lower_inf / upper_inf / isempty over ranges and multiranges, statically typed by the subtype (lower(NULL::int4range) describes as integer).

• secantus-pgplan: AND / OR / NOT in a FROM-less SELECT, three-valued, with PostgreSQL’s 42804 argument of AND must be type boolean, not type integer and 22P02 for an untyped literal that is not a boolean.

• secantus-pgplan: infer_param_types gives an undeclared parameter the type of a $1::<range type> cast, as it already did for a comparison operand.

• secantus-pgserver: binary decoding of custom range and multirange parameters (binary_multirange factored out of the builtin arm).

Fixed

• secantus-pgplan: a literal beside a range / multirange / range-array parameter is cast to the parameter’s declared type ('{empty,"(,)"}' = [Int4Range(...)] was text = text[], 42883).

• secantus-pgserver: an untyped binary range / multirange parameter takes its type from context before decoding (the \x01 empty flag was read as text and every comparison answered False).

• secantus-pgplan: custom range constructors resolve their own type (testrange('a', 'c'), testschema.testrange(1.5, 2.5)), the one-argument form is the literal cast (testrange('a') is 22P02 malformed range literal), and a schema-qualified type name keeps its schema ('[1.5,2.5)'::testschema.testrange reported the oid of public.testrange).

• secantus-pgplan range.rs: "" inside a quoted bound is a literal quote; " and \ are doubled on output; only C isspace characters are whitespace (U+0085 / U+00A0 are bound text); an infinite bound is exclusive in canonical form for every subtype ('[,foo)'::testrange is (,foo)).

• secantus-pgserver: COPY ... FROM STDIN stores ranges in canonical form ({empty} → {}, [1,5] → [1,6)), and ascii(%s) describes as int4.

Range types, and the rewrite that makes two of them the same range

A range like [1,5) has bounds that may each be inclusive or exclusive, or absent entirely. Over a type whose values are discrete — integers, dates — that leaves several ways to write the same thing, so PostgreSQL picks one: every bound is rewritten to [). [1,5] becomes [1,6), (1,5) becomes [2,5), and the two compare equal because they are, in fact, the same range.

Over a continuous type there is no such rewrite, because there is no next number to move a bound to: [1.0,2.0]::numrange stays inclusive at both ends. The Rust PostgreSQL server now supports both families — int4range, int8range and daterange on the discrete side, numrange, tsrange and tstzrange on the other — as literals, constructors and cast targets, each with its own type oid so a client builds a range object rather than reading text.

Some details that only a real server tells you. An absent bound prints as nothing at all, so an unbounded range is (,5) rather than anything spelled with infinity. A range whose bounds meet without including each other contains nothing and is the empty range, so int4range(1,1) prints as empty. And a bound gets quoted when its own text would be ambiguous between the brackets, which a timestamp always is, because it has a space in the middle.

Three different mistakes get three different error classes, which is worth keeping distinct even though all three refuse the query: a crossed bound is a data error, a malformed literal is an invalid-text one, and unrecognised bound flags are a syntax error.

Added

• int4range, int8range, daterange, numrange, tsrange and tstzrange as literals, constructors and cast targets, with canonicalisation, empty ranges, unbounded ends, bound quoting and their own type oids.

Rust pgserver: ROW / record expressions

ROW(...) and the bare parenthesised list (a, b, ...) build an anonymous record in the Rust PostgreSQL server now — reported as oid 2249, which psycopg decodes to a Python tuple. The text form follows PostgreSQL’s composite rules ((a,b,c), a NULL field empty, a field with a comma/quote/backslash/space double-quoted, a bool printed t/f), and record comparison is exactly PostgreSQL’s three-valued logic: =/<> examine every field (a non-null unequal field decides, else a NULL field makes the result NULL) while the ordering operators short-circuit left to right on the first NULL or unequal field.

Added

• ROW(...) / (a, b, ...) record construction (oid 2249), its ::text render, and the =/<>/</<=/>/>= record comparison operators.

Savepoints, and the nested blocks built on them

The Rust PostgreSQL server refused SAVEPOINT outright, and that refusal was quietly expensive: every client builds a nested transaction block out of savepoints, so with conn.transaction(): inside another one failed even though nothing in the user’s code mentions the word.

They work now — SAVEPOINT, RELEASE, and ROLLBACK TO, with PostgreSQL’s rules: a repeated name shadows rather than replaces, rolling back to an outer savepoint discards the ones nested inside it, releasing one keeps its writes while leaving the enclosing savepoint still able to undo them, and rolling back to a savepoint recovers a block that an error had aborted.

WiredTiger has no savepoint of its own, so one here is a set of pre-images: before a statement writes a table, every open savepoint that has not yet captured that table captures it, and rolling back puts the captured contents back. Capturing lazily is what keeps it affordable — a savepoint nobody writes through costs nothing at all.

Two more things fell out of the work. CREATE TABLE IF NOT EXISTS on an existing table raised 42P07 instead of doing nothing, so the ordinary “create it if it is missing” fixture failed the second time a session ran it. And the aborted-block check now runs before the planner’s answer, as PostgreSQL’s does: in an aborted block select nosuchcolumn is 25P02, not 42703 — though a syntax error is still reported as itself, because the parser runs first there too.

Added

• SAVEPOINT, RELEASE [SAVEPOINT] and ROLLBACK TO [SAVEPOINT], and so nested client transaction blocks.

Fixed

• CREATE TABLE IF NOT EXISTS raised 42P07 on an existing table.

• A statement in an aborted block reported its own error rather than 25P02.

Scalar functions, of which the Rust PostgreSQL server had none

upper, length, abs, round, coalesce — none of these worked. Not a gap in a corner: the Rust PostgreSQL server had no scalar function table at all, so every built-in taking an argument answered “not supported yet”. A survey of thirty-seven common ones found thirty-seven missing.

They are here now, along with COALESCE, NULLIF, GREATEST and LEAST, which a user writes like functions but which arrive as their own kinds of expression and so needed handling of their own.

The result type turns out to be as much of the answer as the value, and it is where the surprises live. sign answers a floating-point number even when given an integer. div answers an exact numeric, because that is the type it is defined on, not the integer its arithmetic suggests. round splits by argument type exactly as casting does — half away from zero for an exact numeric, half to even for a floating-point one — so round(2.5) is 3 and round(2.5::float8) is 2. And a handful of functions ignore NULL arguments rather than propagating them: concat skips them, and greatest and least pick the extreme of whatever is left, so greatest(1, NULL) is 1.

nullif deserves its own note. It answers the type of its left argument even when the answer itself is NULL — and a NULL cannot tell you what type it is, so reading the type from the value gave text where PostgreSQL gives integer. A literal carries its type in the query, and that is where it is now read from.

Added

• Forty-odd scalar built-ins: string, numeric, and the conditional expressions.

Fixed

• A literal’s type was inferred from its value, so an expression producing NULL reported text regardless of what it was computed from.

A function of a column, and the error that hid the gap

regexp_replace(statement, 'prepare _pg3_\d+ as ', '', 'i') in a select list — how psycopg reads back its own prepared statements — failed with “column “name” must appear in the GROUP BY clause”. The gap was real but the error was wrong twice over: the router treated any function call in a target list as an aggregate, sent the statement to the aggregate planner, and the planner’s refusal came out wearing a grouping error’s clothes.

The router now names the aggregates it means (count, sum, avg, min, max), and a scalar call over a column is a real computed column: the value is worked out per row by the executor, and the type is fixed at plan time — the describe pass sees no rows and has to name the column’s type anyway. This is the same machinery the catalog work built for cast chains, widened from casts to calls.

regexp_replace itself came with it, with PostgreSQL’s rules rather than a regex library’s defaults: only the first match is replaced unless the g flag is given, \1 references capture groups, i folds case, and a malformed pattern is 2201B — its own error class, because the pattern is broken, not the value being matched. And current_setting(NULL) answers NULL rather than refusing the argument.

Added

• Scalar calls over columns in a table select list, typed at plan time.

• regexp_replace, constant or over a column, with g / i flags and group references.

pg_typeof now answers a real regtype value rather than a display-name string, so pg_typeof(x)::oid reads the type’s oid and ::text its name — psycopg’s wrapper tests read the oid form for every numeric wrapper, and a name is not a number. And a range array carries its own oid (int4range[] is 3905, not varchar), so a client builds Range objects from it in either format.

Fixed

• Any function call in a select list was routed to the aggregate planner, so a plain gap surfaced as a grouping error.

• current_setting(NULL) refused the argument instead of answering NULL.

• pg_typeof(x)::oid failed trying to parse a type name as a number.

• A range array was described as varchar.

Schema-qualified composite types are distinct types

CREATE TYPE testschema.testcomp AS (...) on the Rust PostgreSQL server now creates a type that is genuinely distinct from a bare testcomp in public, matching PostgreSQL. Previously a qualified type name was resolved to its last part, so testschema.testcomp collided with testcomp and the second CREATE failed with 42710 type already exists.

This mattered far beyond one CREATE: psycopg’s composite-type test fixture is session-scoped and creates testschema.testcomp beside a bare testcomp in one script, so the collision failed the fixture and cascaded to every composite test. With the two types now coexisting, CompositeInfo.fetch resolves each name to its own type — bare testcomp, testschema.testcomp, and the quoted "testschema"."testcomp" form a sql.Identifier renders all reach the right oid, while typname stays unqualified as PostgreSQL keeps it. DROP TYPE testschema.testcomp is schema-aware and leaves the bare type standing.

Added

• Schema-qualified composite type names (CREATE TYPE schema.name AS (...)) are stored and resolved per (schema, name); a bare name and a schema-qualified one with the same last part are distinct types.

• to_regtype resolves a schema-qualified type reference in bare, schema.name, and quoted "schema"."name" forms; a bare name resolves only the public type (matching the default search_path).

• DROP TYPE schema.name targets the schema-qualified type; public.name normalises to the bare name.

CREATE SCHEMA

CREATE SCHEMA and DROP SCHEMA are accepted now, tracked in the same __sql_schemas__ catalog the Python server uses so the two agree on which schemas exist. Schema-qualified names already resolved by their last part — s2.t finds table t — so a table or type living in a schema works the moment the schema DDL is allowed.

A duplicate is 42P06 (distinct from a table’s 42P07 and a type’s 42710), a missing DROP SCHEMA is 3F000, IF NOT EXISTS / IF EXISTS are no-ops on the wrong-existence case, and CASCADE is accepted (this server does not track which objects belong to a schema — names carry none — so it drops the schema record; the objects are dropped by name elsewhere).

Added

• CREATE SCHEMA [IF NOT EXISTS], DROP SCHEMA [IF EXISTS] … [CASCADE], with PostgreSQL’s error classes.

A series whose bounds are parameters

select * from generate_series(1, %s) — the way clients actually write a series — was refused outright by the Rust PostgreSQL server. A bound parameter with no declared type arrives as text, and PostgreSQL resolves it against the function’s own signature; this server saw text where it wanted an integer and said the feature was unsupported. Every parameterised series failed, which is most of them, and an explicit %s::int4 failed too.

A NULL bound is now an empty result rather than an error, matching PostgreSQL, where a series with any NULL bound produces no rows at all.

The probe that established those two also caught the server being more permissive than PostgreSQL in one place: generate_series(1, 3::float8) has no matching overload there and is refused with 42883, while this server truncated the bound to an integer and answered rows. Truncating an argument a real server rejects is a wrong answer, so it is now the same refusal.

Added

• Series bounds given as parameters, in every position, including a NULL bound.

Fixed

• generate_series with a float8 bound truncated it instead of refusing.

Session settings on the Rust PostgreSQL server

SET, SHOW, RESET and the current_setting() / set_config() functions now work, with the settings held per connection as PostgreSQL holds them. This was the largest remaining thing psycopg’s test suite asked for, and the score moved from 853 to 899 of 4,238.

Small details decide whether a client is satisfied here. SHOW datestyle answers a column named DateStyle, not datestyle — lookups ignore case but the reported name does not, and clients match on it. Asking for a setting that does not exist is an error, while asking with the “missing is fine” flag returns null instead. RESET restores a setting to its default rather than deleting it, so a client reading it back afterwards sees the default rather than an error. A fresh connection starts from the defaults, not from whatever the last one did.

Added

• SET / SET LOCAL, SHOW, RESET, RESET ALL.

• current_setting(name [, missing_ok]) and set_config(name, value, is_local).

• The settings a client expects to read before it has set anything: client_encoding, DateStyle, TimeZone, IntervalStyle, standard_conforming_strings, integer_datetimes, transaction_read_only, search_path, application_name, server_encoding, server_version.

Set-returning functions where clients actually put them

select generate_series(1, 10) — the function in the select list, with no FROM at all — now returns ten rows rather than an error. It is the shortest way to conjure rows without a table, and it is what client test suites reach for constantly; a server cursor over exactly that form was the single most common use in psycopg’s.

It is planned as an ordinary select over a generated source, which is the same shape the FROM generate_series(...) form already produced. That means ordering, limits and offsets came along for free rather than being written a second time for a second syntax that means the same thing.

A set-returning function beside another output column — select 1, generate_series(1,3), which repeats the constant across the generated rows — is refused rather than half-supported. It needs the other columns carried into each generated row, and a shape that silently dropped one would be worse than an honest refusal.

Multiranges can also be sent as bound parameters in the binary format now. The layout is a count of ranges followed by each one in the range’s own binary form, so it reuses the range decoder rather than restating the flags-and-bounds layout a second time and risking the two drifting apart.

Added

• generate_series in the select list of a FROM-less query, with aliases, ORDER BY, LIMIT and OFFSET.

• Multirange bound parameters in the binary wire format.

A parameter should not mean different things in different wire formats

A client may send each parameter as text or in PostgreSQL’s binary format, and picks per value — psycopg sends most things binary and falls back to text. The two are decoded by separate code here, and only the binary side had learned arrays, intervals and timestamps. Sent as text, those values fell through to a plain string, so array[...] = %s compared an array against a string and reported that it could not compare them. The message pointed at comparison; the cause was one layer earlier, in decoding.

Both formats now produce the same value for the same declared type, and the mapping from an array type to its element type is shared between them, so they cannot drift apart again.

The other half is subtler. A client may leave a parameter’s type unspecified and let the server work it out — psycopg does this for lists and datetimes — and PostgreSQL then resolves it from whatever it is being compared to, exactly as it resolves an unquoted literal. That rule was already implemented for literals; extending it to parameters is what makes '2026-01-01 12:00'::timestamp = %s answer true rather than complain about comparing a timestamp to a string.

Fixed

• Array, interval, date, time and timestamp parameters sent in the text format decoded to plain strings, so comparing one to a value of its own type failed.

• A parameter whose type the client left unspecified was not resolved from the operand beside it, though a literal in the same position was.

Rust pgserver: a timestamp drops a trailing time-zone offset

PostgreSQL’s timestamp (WITHOUT time zone) input accepts a trailing offset and drops it, keeping the wall-clock reading — '2000-01-01 03:02:03+02' is 2000-01-01 03:02:03. The Rust server rejected the offset (22007 / 22008), which surfaced when psycopg dumps a tz-aware datetime that reaches the naive timestamp parser (its offset now reflects the session zone, so it can even be a second-precision -01:02:03).

Fixed

• timestamp input now accepts and drops a trailing ±HH[:MM[:SS]] offset, matching PostgreSQL.

Rust pgserver: infinity, epoch and out-of-range dates

PostgreSQL’s date and timestamp domain is far wider than a Python date or datetime: infinity / -infinity are real values, and so are years past 9999 and dates BC. The Rust PostgreSQL server now accepts all of them — storing the canonical text and letting the client’s loader decide what it can hold, exactly as a real server does (psycopg’s overflow tests want the “date too large” the loader raises, not a server error). A wide-year or BC plain timestamp is rendered the way PostgreSQL renders it, with the time canonicalised to HH:MM:SS. The one constant special input keyword, epoch, now resolves to 1970-01-01 00:00:00.

Added

• infinity / -infinity, years > 9999, and BC dates are accepted on ::date and ::timestamp casts and passed through as canonical text.

• 'epoch'::timestamp resolves to 1970-01-01 00:00:00.

Fixed

• A wide-year (> 9999) or BC date no longer errors on the Rust pgserver; a datetime-shaped value with an impossible field is 22008 and a non-date is 22007, matching PostgreSQL.

Rust pgserver: timestamptz columns

timestamptz is a real column type now. It is stored as a UTC INSTANT — the same date-plus-microsecond-companion carrier a timestamp uses — and rendered in the session’s zone on the way out, so the same stored instant reads back correctly under any SET timezone rather than in whichever zone happened to write it. This closes a deliberate refusal that had stood because storing session-rendered text was a wrong answer for every other session.

Making it correct end to end also meant the server now reports GUC changes: after a SET, it emits a ParameterStatus for the variables PostgreSQL marks GUC_REPORT (TimeZone, DateStyle, client_encoding, …). libpq and psycopg track the session TimeZone from that message and re-express a timestamptz in it — without the report a stored instant displayed in the client’s stale startup zone.

Added

• timestamptz column type (oid 1184): UTC-instant storage, session-zone rendering, sub-millisecond precision, and the ::timestamptz::text cast.

• ParameterStatus reports for GUC_REPORT variables on SET / RESET.

Timestamps that know their time zone

timestamptz is not a timestamp with an offset stapled to it. It is an instant, and what you see is the session’s view of that instant — so the same stored value prints as 12:00+01 in Rome and 06:00-05 in Chicago, and the same literal read under two zones names two different moments. The Rust PostgreSQL server now supports it, along with timetz, SET TimeZone for both fixed offsets and named IANA zones, and the regtype that pg_typeof had already been answering with.

Two sign conventions meet in this type and they run in opposite directions. In SET TimeZone TO '+02:00' the sign is POSIX: positive means west of Greenwich, so that setting renders timestamps as -02. In a literal like '2026-01-01 12:00+02' the sign is the ordinary one, two hours east. Both were measured against a live PostgreSQL rather than reasoned out, because getting either backwards is completely invisible under UTC and wrong by hours everywhere else.

Named zones carry their daylight-saving rules, so 2026-01-01 12:00 and 2026-07-01 12:00 resolve to different offsets in Europe/Rome and to the same one under a fixed +02:00. Offsets may also carry minutes and seconds: +01:02:03 is a real offset that appears in client test suites, and a comment in this work asserting otherwise was contradicted by the first probe that looked.

Added

• timestamptz and timetz, as casts, literals and bound parameters in both wire formats, with their own type oids.

• SET TimeZone for fixed offsets ('+02:00') and named IANA zones ('Europe/Rome'), including daylight-saving rules.

• regtype as a cast target: 'int4'::regtype is integer.

• Offsets with minute and second precision.

Fixed

• A doc comment that had come adrift from the function it described.

Changed

• A timestamptz or timetz column is now refused rather than accepted. The types are stored as canonical text, and a timestamptz renders in the session’s zone — so a row written under UTC read back under another zone showed the right instant with the wrong wall clock and the wrong offset, which no client could detect. They remain available everywhere they are a value rather than storage.

A transaction you can see, and a failed one that says no

Every connection to the Rust PostgreSQL server reported itself as idle. Inside a transaction, after a failed statement, in the middle of a block — always idle. The status rides on the message that ends every statement rather than being something a client selects, so nothing in a row comparison could ever see it, and clients that steer on it were steering blind.

The half that was more than cosmetic: PostgreSQL aborts a transaction block at the first error and refuses everything after it until the block ends. This server carried on. A client that shrugged off a mid-transaction error went on writing and committed work PostgreSQL would have discarded — a wrong answer, not a missing feature. Statements after an error in a block now get 25P02 until the block ends, and a COMMIT there is a rollback that says ROLLBACK in its command tag, exactly as PostgreSQL’s does.

Two smaller spellings came along: START TRANSACTION answers with its own command tag rather than BEGIN’s, and COMMIT AND CHAIN / ROLLBACK AND CHAIN open the next block immediately instead of leaving the connection idle — a client that chained was previously left autocommitting its next statements one at a time.

Added

• AND CHAIN on COMMIT and ROLLBACK.

• START TRANSACTION’s own command tag.

Fixed

• Every connection reported IDLE whatever the transaction state.

• Statements after an error inside a transaction were executed rather than refused with 25P02, and a COMMIT of a failed block reported COMMIT.

Rust pgserver: transaction settings psycopg reads

psycopg reads a handful of transaction GUCs to learn a connection’s defaults, and the Rust PostgreSQL server answered 42704 (unrecognized parameter) for all of them, which failed a swathe of cursor and connection tests before they could begin. The server now reports the fixed values a single-node server gives: max_prepared_transactions is 0 (no two-phase commit), transaction_isolation / default_transaction_isolation are read committed, and transaction_deferrable / default_transaction_read_only are off.

Added

• SHOW / SET recognise max_prepared_transactions, transaction_isolation, default_transaction_isolation, transaction_deferrable and default_transaction_read_only.

The Rust PostgreSQL server enforces UNIQUE

CREATE TABLE t (tag text UNIQUE) was accepted and then ignored: the constraint was recorded nowhere and a second equal value inserted where PostgreSQL answers 23505. That is worse than a missing feature. A refusal is honest and you meet it on the first run; this let a user believe a uniqueness guarantee the server was not providing, and the duplicates were already stored by the time anyone noticed.

Enforcement is a storage unique index rather than a check before each write, because a probe read cannot see a value another transaction committed after the writer’s snapshot, nor one a second writer is inserting right now — so WiredTiger arbitrates instead. SQL NULLs stay distinct, as SQL requires: the index carries a partial filter excluding NULL from every column, which a sparse index would not achieve, because a SQL NULL is stored as an explicit null rather than a missing field.

The error surface was measured against PostgreSQL 14.13 — the constraint PostgreSQL would have generated (<table>_<column>_key, or the declared name), and DETAIL: Key (a, b)=(1, 2) already exists. for a multi-column one.

Added

• secantus-pgplan: column-level and table-level UNIQUE, with PostgreSQL’s constraint-name generation and DEFERRABLE / INITIALLY DEFERRED.

• secantus-pgcatalog: UniqueConstraint, in the Python server’s on-disk shape key for key, so a table created by one server reads back in the other.

Fixed

• secantus-pgserver: an UPDATE that violated any unique constraint — including the pre-existing PRIMARY KEY one — surfaced as could not update: E11000 duplicate key error on index ..., leaking the MongoDB persona through the PostgreSQL one with no SQLSTATE a client could branch on. Both write paths now render PostgreSQL’s 23505.

• secantus-pgplan: DEFERRABLE attached to the most recent FOREIGN KEY regardless of what it actually qualified, so UNIQUE ... DEFERRABLE would have marked an unrelated foreign key deferrable.

Rust pgserver: uuid binary parameters

A uuid sent as a BINARY parameter (16 raw bytes, oid 2950 — psycopg’s default for a Python UUID) is now decoded to its canonical lowercase text, the same value the text path stores. Previously only the text form was accepted, so a bound UUID failed with binary parameters of type oid Some(2950) are not supported.

Added

• Binary decode of uuid (oid 2950) parameters.

Rust pgserver: uuid, and timetz columns

The Rust PostgreSQL server now supports uuid — as a cast, a bound value, and a real column type. Any spelling PostgreSQL accepts (uppercase, surrounding braces, no hyphens, or hyphens at the standard group boundaries) canonicalises to lowercase 8-4-4-4-12, and the column reports its true oid (2950) so a client hands back a UUID object rather than a string. A malformed uuid is 22P02, matching PostgreSQL.

timetz also works as a column now. Unlike timestamptz — whose text is session-relative and which stays refused until its stored form is an instant — a timetz offset is literal (12:34:56+02 renders the same under any zone), so its canonical text is a safe column.

Added

• uuid cast, bound value, and column type (oid 2950), with PostgreSQL’s input leniency and 22P02 on malformed input.

• timetz as a column type (oid 1266), stored as canonical text.

Errors the Rust server lost, or answered as a value, inside a pipeline

Four defects, found by probing outward from the required-argument fix and all measured against mongod 8.2.11.

$group and friends discarded every named error. group.rs, fill.rs, densify.rs and windowfields.rs typed their errors as Result<T, ()>, so an error the expression engine had already named was thrown away at the module boundary and the client got the generic refusal instead:

{$group: {_id: null, x: {$first: {$ln: 0}}}}
    mongod  28766 $ln's argument must be a positive number ...
    before  2     aggregation pipeline uses a stage or operator not supported

Carrying Fallback lets a named error through; Fallback::Defer is the same “cannot reproduce this” signal the unit () was, so all 57 existing sites keep their exact behaviour. A parse error is reported bare here, which mongod does inside $group / $expr / $redact and which the new Fallback::bare() marks.

$sortByCount rejected every expression. An unconditional early match arm answered 40147 for any document argument, shadowing a later arm seventy lines below that already implemented mongod’s three codes correctly. {$sortByCount: {$add: ["$n", 1]}} is valid and now works.

Removing the shadowing arm was only half the rule, and the 740-shape stage-spec probe caught the other half: a document is an EXPRESSION only when its first key is $-prefixed, so {a: 1} and {a: {$add: [...]}} are literal documents and get 40147 like {} does. Without that they fell through to the engine and deferred.

$arrayElemAt answered null for a non-numeric index — a silent wrong value. Only bool was checked, so {$arrayElemAt: [[1, 2], "x"]} came back as null where mongod raises 28690 naming the type. Measured across 19 index shapes: null and a missing field really are null, every numeric works (including Decimal128, which was rejected), and everything else is 28690. “Representable as a 32-bit integer” is also enforced now, so a whole 1e40 is 28691 rather than a missing field.

$divide and $mod by zero deferred, each with a comment justifying it by what “Python raises” — the shape CLAUDE.md catalogues, and wrong here because a defer has no Python behind it. They now answer mongod’s 2 can't $divide by zero and 16610 can't $mod by zero.

Measured

$group / $bucket / $sortByCount over twelve named-error expressions plus four valid $sortByCount forms: 31 of 40 matching, from 18 of 36. $arrayElemAt index types: 0 divergences of 19, and the 740-shape aggregation stage-spec probe goes to 0 divergent (it was the only probe of twelve still reporting one). The 6,628-case expression corpus is unchanged at 38 different-code divergences with 0 wrong values and no regressions (it does not cover these shapes, which is why they survived so long).

Fixed

• secantus-core: group / fill / densify / windowfields carry Fallback; Fallback::bare() for parse errors mongod sends unwrapped; $arrayElemAt index typing and int32 range; $divide / $mod by zero named.

• secantus-commands: the shadowing $sortByCount arm removed; a bare error is no longer given a pipeline wrapper.

• secantus (the PYTHON server): the same $arrayElemAt index rules — it had the identical silent-null defect, and rejected a decimal index too.

BOTH servers answered a value where mongod rejects the expression

{$regexMatch: {}} returned false. {$filter: {}} returned null. So did {$trim: {}}, {$reduce: {}}, {$map: {}} and {$dateAdd: {}} — cases where an operator was missing a REQUIRED argument and the server answered a value a caller can branch on, from an expression mongod refuses to run at all.

Both servers had it, and the PYTHON one was worse: 31 of its 57 cases were silently wrong against the Rust server’s 25. That only came to light because the Rust fix turned the parity suite red — parity is equally satisfied by both engines being wrong, and here they had been. The fix moves both to mongod’s answer rather than moving either to the other.

The cause is the missing-vs-null conflation CLAUDE.md catalogues: the operators read their required fields with the evaluator’s optional-field helper, which reports an absent key as null. {$trim: {input: null}} really is legal and really does yield null — only an ABSENT key is an error — so the fix tests key presence, never the value.

A further thirty-two cases answered the generic 2 BadValue: aggregation pipeline uses a stage or operator not supported by the Rust server, which blamed the operator when the argument was at fault.

The rules, all measured against mongod 8.2.11

• Every (operator, missing field) pair has its own code and wording, and they are not derivable from a pattern: $filter says Missing 'input' parameter to $filter (28648) where $reduce says $reduce requires 'input' to be specified (40077), and $replaceAll’s three fields descend 51749 / 51748 / 51747 as you read them left to right. Hence a table of 40 measured pairs.

• An UNKNOWN argument outranks a missing one. {$trim: {k: 1}} is 50694 $trim found an unknown argument: k, and so is {$trim: {input: "a", k: 1}}. The operators already emit those correctly, so the new check stands aside whenever a key is unrecognised. The corpus caught this: a first version got the precedence backwards and traded twenty fixed cases for twenty broken ones at an unchanged total.

• The stage decides the wrapper. A parse error from $addFields / $project / $set is wrapped Invalid <stage> :: caused by :: …; the same error inside $match’s $expr is BARE. So validation runs per stage, where the stage name is known, mirroring mongod’s own parse-time check.

• $switch with branches: [] and $zip with inputs: [] get the same code as omitting the field entirely.

Measured

Both servers at 0 divergences of 57 — from 57 with 25 silently wrong (Rust) and 57 with 31 silently wrong (Python). The 6,628-case expression corpus improves from 58 different-code divergences to 38, with 0 wrong values and no regressions. Parity: 730 tests, and the $zip fuzz now asserts that a named error matches between engines rather than tripping over one.

Fixed

• secantus-core: check_required_fields + validate_expression_args, gated on all_fields_recognised so unknown-argument errors keep precedence. The field-value dispatch path ($cond / $switch / $let / $ifNull, which bypasses apply_op) validates too.

• secantus-commands: validate_stage_expr_args applies mongod’s per-stage wrapper at parse time.

• secantus (the PYTHON server): the same table in expressions.py, reported through aggregate.py’s existing parse-time scanner so it picks up the Invalid $<stage> :: caused by :: wrapper that machinery already knew about.

Known gap

$group still answers the generic refusal for these and for errors that were already named elsewhere ($ln: 0’s 28766): group.rs types its module as Result<T, ()> and discards the error. Pre-existing and independent; filed in tasks/backlog.md.

Both servers answer mongod’s argument errors, and the range operators bracket by type

A measured sweep of both servers against mongod 8.2.11 found the same shape of bug over and over: an operator that works perfectly well was refusing, or silently mis-answering, a bad argument. The Rust server had the worse version of it — with no Python engine behind it, any refusal it could not name reached the client as “not supported by the Rust server”, so {$round: ["$n", 1.5]} reported that the server cannot do $round.

Fixed

• Range operators are type-bracketed. {v: {$gt: 3}} matches numbers greater than 3 and nothing else. Only three brackets were enforced, so a collection holding a MaxKey returned that document for every $gt query. 96 of 112 probed (bound, operator, collation) shapes disagreed with mongod; all 112 now agree. A MinKey / MaxKey bound remains the one exception and compares across every type.

• A JavaScript value is not a string. bson.Code subclasses str, so it took the string type rank, sorted among the strings, and matched a string bound. Being unhashable, it also crashed the cached collation key, so an ordinary collated sort over a collection holding one answered internal server error. mongod ranks JavaScript between Regex and MaxKey, and it now sorts there — in the in-memory comparator and in the persisted index-key encoder, which had to move together or an index would change the sort answer.

Storage format

• Index entries are entryFormat 3. Giving JavaScript its own type rank shifts MaxKey’s rank byte from 13 to 14, so every index key for a JavaScript or MaxKey value changes. A store written by an earlier build is refused at open rather than read back in the old order; there is no migration, as with the two format bumps before it. Start from a fresh data directory, or drop and recreate the indexes.

• $round / $trunc validate their precision the way mongod does — three ordered checks with three different codes. A fractional Decimal128, an out-of-int32 integer, a string and a bool were all silently accepted: {$round: ["$n", -25]} answered 0.0.

• $indexOfArray has its own error codes (9711600 / 9711601), and a negative index is an error rather than a clamp — {$indexOfArray: [[1,2,3], 3, -1]} answered 2.

• $toDouble follows C’s strtod. It accepts the hexadecimal spellings mongod accepts ("0X1f" is 31.0) and reports an unrepresentable magnitude as an error rather than saturating ("1e400" answered inf). The "0x" gate is the literal lower-case prefix mongod uses, not a case- and sign-insensitive one.

• $toObjectId names the offending character when the length is right, instead of reporting “expected 24 but found 24”.

• $dateToString’s format language is not strftime. mongod accepts exactly %b %d %j %m %u %w %z %B %G %H %L %M %S %U %V %Y %% and refuses the rest; a typo’d %a used to render a value instead of erroring. %z and %Z were empty and are now the offset (+0000) and the offset in minutes (0), and month names no longer come from the machine’s locale.

• $toUpper / $toLower / $strcasecmp map ASCII only, and $trim’s default strips mongod’s fixed 20-code-point table. Python’s Unicode case folding turned "straße" into "STRASSE" where mongod answers "STRAßE".

• A regex as the bound of a range operator or $ne is refused at parse time, where an empty result set used to hide the malformed query.

• $range with a zero step, and the domain guards on $sqrt / $ln / $log10 / $log / $pow, carry mongod’s codes.

Changed

• The Rust engines share one Fallback type that can carry a mongod error, so a refusal reaches the client verbatim instead of collapsing into a generic “not supported”. The command layer picks mongod’s constant-folding wrapper by where the failure came from, and StorageError gained the matching variant for query-side refusals.

• The six $toX conversion shorthands now route through one $convert implementation on the Rust side, as they already did in Python. They had drifted: $toBool and $convert {to: "bool"} disagreed on the empty string inside a single engine, $toObjectId was not registered at all, and the numeric ones still deferred on a string source.

$stdDevPop / $stdDevSamp work in expression position on the Rust server

The $group accumulator forms shipped long ago; the expression forms — over an array argument in $project / $addFields — never did. All 56 shapes in the probe corpus answered “operator not supported by the Rust server” where mongod computes a number, and on the standalone server that is an error, not a fallback.

They share group::std_dev with the accumulators, so the two forms cannot answer different numbers, and drop non-numeric members exactly as the accumulators do: mongod counts int / long / double / decimal and silently skips bool, null, string, array and document.

tools/probes/agg_expressions.py against the Rust server: 981 → 925, still zero wrong values.

Also recorded

Analysing the remaining defers turned up that 116 of them were valid input being refused — mongod answers and the Rust server errors. 56 were these; the other 60 are almost all a Decimal128 operand reaching a math operator, declined with comments reading “→ Python” on a server that has no Python. In practice a collection holding Decimal128 values cannot use most math operators there. Recorded in tasks/backlog.md with what a fix would need.

An ObjectId or Timestamp is a date on the Rust server too

mongod accepts every BSON type that carries a timestamp wherever a date is expected — a Date, an ObjectId (its 4-byte generation time) or a Timestamp (its seconds field) — and treats a one-element array as the argument itself. So {$year: ObjectId("64b7f9a2…")} answers 2023.

The Rust engine deferred on all of them, and a defer on the standalone server is an error: 13 shapes refused input mongod answers. The Python engine took this fix in the previous change; this is the port.

tools/probes/agg_expressions.py against the Rust server: 925 → 912, still zero wrong values.

Also scoped

The remaining valid-input refusals are 38 operators declining a Decimal128 operand — the whole math, comparison and conversion family — each with a comment reading “→ Python” on a server that has no Python. In practice a collection holding Decimal128 values cannot use most math operators there.

tasks/backlog.md now carries the scoping rather than a guess: decimal.rs already represents sign / coefficient / exponent with add, mul, div_int and trunc_to_i64, so about 19 of the 38 are reachable with what exists ($abs is a sign flip, $subtract is add negated, the comparisons need one cmp). The other ~17 are transcendental and need real decimal math. It is 19 individually-probed operators, not one change — result type, precision and overflow all differ per operator.

The Rust engine ignored TZ on Windows, and no CI lane could see it

$toLower / $toUpper of a Timestamp is the one conversion mongod renders in the server process’s local time. The Rust engine resolved that zone through chrono::Local, which on Windows reads GetDynamicTimeZoneInformation and ignores the TZ environment variable entirely — so a server started with TZ=UTC on a Europe/London host rendered summer instants an hour late, while mongod and the pure-Python evaluator both answered UTC. The two SecantusDB servers disagreed with each other and one of them disagreed with mongod.

Three independent things hid it. GitHub’s Windows runners are UTC, where the host zone and TZ=UTC coincide; the Windows test lane has no _secantus_core, so the test that would have caught it skipped; and the storage-engine job runs only three smoke files. It took building and running on a non-UTC Windows box in July to surface — London is UTC in winter, so three of the four instants in the existing corpus agree even with the bug present.

The fix routes the Windows render through the MSVC CRT (_tzset + _localtime64_s), which is the same function mongod and CPython’s time.localtime reach. That is exact by construction rather than an emulation of the CRT’s tzn[+|-]hh[:mm[:ss]][dzn] grammar — a grammar that is not the IANA one, and observably so: measured against mongod 8.2.11 on Windows 11, TZ=America/New_York parses as a zone name with a zero offset plus a daylight rule, so mongod answers UTC+1 in July, not New York’s UTC-4. Unix keeps chrono::Local, which already matched.

Fixed

• crates/secantus-core: $toLower / $toUpper of a Timestamp now honours TZ on Windows, matching mongod and the Python engine. New expressions::render_local_asctime splits the render per platform; the Windows half calls the CRT via a cfg(windows) libc dependency.

Added

• tests/test_mongod_differential.py: test_timestamp_local_render_matches_mongod spawns a mongod per zone and compares both engines against it, with no hardcoded expectations — so unlike the sibling unit test, which pins Unix-measured values and must skip zone-shifted cases on Windows, it needs no platform skip and asserts whatever the local mongod actually does. The engine is a parameter, so the Rust half is a visible skip where that extension is absent rather than a silent one.

• tests/test_mongod_differential.py: _start_mongod factors the spawn, readiness poll and port-race guard out of the module fixture, so cases that need a differently-configured server reuse them instead of re-deriving them.

$toDate rejected every timestamp carrying milliseconds

Chasing the error message for an unparseable date string turned up a much larger bug behind it: parse_iso required exactly 19 characters before a Z, so every ISO timestamp with a fractional second failed — 2020-01-01T00:00:00.123Z, the ordinary form for a BSON date. It came back as an error on the Rust server where mongod parses it.

Measured against mongod 8.2.11 and now reproduced: a fractional second takes 1..n digits and is truncated to milliseconds (.1 is 100 ms, .1234567 is 123), with or without a Z or a ±HH:MM offset. YYYY-MM is the first of that month; a bare YYYY is not a date at all.

The error surface, too

A failed parse used to return Conv::Failed, which on this server surfaces as 2 BadValue: aggregation pipeline uses a stage or operator not supported by the Rust server — false, since $toDate is supported and the string was at fault, and a different code from mongod’s 241 ConversionFailure. It now carries 241 always, with mongod’s exact text for the two reproducible shapes: an empty string names a literal NUL, everything else gets the incomplete-string message.

Whitespace-only is not empty for mongod — '' is “Empty string” but '  ' is the incomplete message — which the Python server had backwards because it tested the stripped text. Fixed on both servers.

Still not reproduced, and now shared rather than divergent

mongod’s per-position timelib diagnostic ('abc' names the offending character and where its scanner stopped) needs timelib’s own lexer, its timezone abbreviation tables and its per-position error accumulation. Both servers give the same message instead, so they agree with each other while the gap stays documented.

Measured

34 strings: 18 exact, 16 message-only, 0 with a wrong value or code — from 0 exact with all 25 failure cases carrying the wrong code. Rust and Python agree on 23 of 23 failure strings.

The Rust server names an unknown expression operator

Nine of thirteen measured shapes came back as 2 BadValue "aggregation pipeline uses a stage or operator not supported by the Rust server" — which told the client the server could not do $addFields, when what it could not do was the unrecognised operator inside it. A Fallback::Defer has no Python behind it on the standalone server, so it reaches the client as that blanket refusal.

Fixed

• An unknown operator in $addFields / $set / a nested $project position now answers 168 Invalid $<stage> :: caused by :: Unrecognized expression '$x', and in $group / $replaceWith / $match’s $expr the same 168 with no envelope — both mongod’s.

• A nested unknown inside $project answered 31325 (the projection parser’s code) because the check recursed. mongod uses 31325 only for the top-level value of a $project field; anything deeper is 168.

• codeName for code 168 is now InvalidPipelineOperator rather than Location168.

Both servers are now 0 of 13 divergent on tools/probes/unknown_expression_errors.py.

Update errors the Rust server named wrongly, or not at all

Two defects, measured against mongod 8.2.11.

$inc / $mul with a non-numeric operand carried a wrapper mongod does not send. mongod has two shapes for the same code 14 and wraps only one:

{$inc: {n: "x"}}    operand bad, readable from the spec
    mongod  Cannot increment with non-numeric argument: {n: "x"}
    before  Plan executor error during update :: caused by :: Cannot increment ...

{$inc: {s: 1}}      stored field bad, needs the document
    mongod  Plan executor error during update :: caused by :: Cannot apply $inc ...

arith_type_error now returns (message, exec) and the storage layer threads it through, instead of hard-coding every code 14 as execution-time.

$position / $slice / $bit with a bool argument answered the generic refusal. The guards were already there and the code (2) was already right — they simply deferred, which on this server reads as “the operator is not supported” when the argument was at fault. $position and $slice are worded differently by mongod (“not of type:” vs “but was given type:”), so each is measured rather than shared.

Also: two backlog entries were stale

The query matcher’s three “still deferred where faithful” residuals — an exotic-text value under a collation, an exotic type range-compared inside an array, and a Decimal128-valued $mod field — all match mongod now (0 of 6 shapes). Both entries are marked resolved rather than left advertising finished work as remaining.

Fixed

• secantus-core: arith_type_error reports which of mongod’s two wrappers applies; $position / $slice / $bit bool arguments carry mongod’s text.

• secantus-storage / -storage-adapter: the exec flag is threaded instead of assumed.

The Rust server stores the zero you asked it to store

{$set: {a: -0.0}} over a stored 0.0 was silently dropped by the Rust server. Its storage write guard asked new != doc, and Rust’s f64 == calls the two zeros equal, so the write was skipped: nModified came back 0, no oplog entry was emitted, no change-stream event fired, and reading the document back gave the old zero. The value the caller asked to store was never stored, and nothing reported a problem. The Python server was fixed for this a release ago; the Rust storage layer kept the bare comparison, and nothing covered it — the parity suites pin the pure operator engines, not storage.

The same batch closes the two ways the Rust server misdescribed an update it refused. An $inc or $mul past int64 could only defer, and a defer has no Python engine behind it on the Rust server, so five real overflow shapes told the client query uses a construct the Rust server does not support — which says the server cannot do $inc, when it can and it was the result that did not fit. Separately, every execution-time update error came back bare: mongod reports the failures that depend on the stored document under Plan executor error during <command> :: caused by :: and leaves the parse errors readable from the update spec alone unwrapped, and the Rust server had the message bodies right and the wrapper on none of them.

All three were measured against mongod 8.2.11, and the Rust server now matches it on every shape in the sweep that the Python server matches.

Fixed

• secantus-storage: an update whose only difference is the sign of a zero is stored and counted, instead of being silently skipped. The write guard is now secantus_core::diff::doc_changed, which falls back to the encoded BSON when == says the documents match — the same rule storage._doc_changed applies on the Python server, rather than a third copy of it.

• secantus-core: an $inc / $mul that overflows int64 reports mongod’s Failed to apply $inc operations to current value ((NumberLong)…) for document {_id: …} (code 2) instead of a generic “not supported”.

• secantus-core / secantus-storage / secantus-commands: execution-time update errors carry mongod’s Plan executor error during <command> :: caused by :: wrapper, with the command name interpolated (update and findAndModify report their own), and parse-time errors stay bare. Ten wrapped and seven bare shapes probed against 8.2.11.

The same upsert, run twice, inserted two documents

On the Rust server, an upsert whose filter used a dotted equality stored a document with a literal dotted key. update({"a.b": 5}, {$set: {z: 1}}, upsert: true) inserted {"a.b": 5, "z": 1} — a document mongod cannot produce, and one that does not match the query that created it. So running the same upsert again inserted a second document. An idempotent upsert is the canonical use of the feature, and it was silently broken: no error, just a growing pile of near-duplicate rows. mongod builds the nesting, storing {a: {b: 5}}, and matches it on the next call.

The Python server has used a path-aware seed here since it hit the same bug; the Rust port kept a plain insert. This is the “user-supplied path used as a dict key” shape the project’s own notes call out, and it is the third instance of it.

The same batch gives the Rust upsert mongod’s field order. mongod emits _id first, then the fields seeded from the query, then the fields the update added, sorted by name; the Rust path had no ordering at all and emitted them in insertion order. BSON field order travels on the wire and drivers compare raw bytes, so this was visible to clients. Deliberately not imitated: mongod’s order for the query-seeded fields is an internal hash order that varies between runs for identical input, so both servers sort those instead — an approximation the Python server already made, and one the two servers now share exactly.

Fixed

• secantus-storage: an upsert seeded from a dotted equality builds the nested document mongod builds, so the upserted document matches its own filter and the operation is idempotent again.

• secantus-storage: an operator upsert emits mongod’s field order (_id, the query-seeded fields, then the update-added ones sorted by name); a replacement upsert keeps the document’s own order.

• secantus-storage: the leftover any(k.starts_with('$')) form test that selects the oplog entry’s shape now uses the shared is_operator_form predicate, so it cannot drift from the engine’s rule.

• secantus-core: update::set_document_path is the public, properly-typed way to build a dotted path, replacing a bare Result<(), ()> on the crate surface.

rustls bumped past RUSTSEC-2026-0285

Fixed

• rustls moved to 0.23.45 (from 0.23.40 / 0.23.43, depending on the lockfile) across all four Cargo lockfiles that carry it, clearing RUSTSEC-2026-0285 — TLS 1.3 handshake messages incorrectly accepted across encryption level boundaries, medium severity, fixed in 0.23.45. rustls-webpki came along to 0.103.15.

Schema-qualified RANGE and ENUM types are distinct types

CREATE TYPE testschema.testrange AS RANGE (...) on the Rust PostgreSQL server now creates a type genuinely distinct from a bare testrange in public, matching PostgreSQL. Previously a schema-qualified CREATE TYPE ... AS RANGE (and AS ENUM) resolved to its last name part, so testschema.testrange collided with testrange and the second CREATE failed 42710 type already exists.

This mattered beyond one CREATE: psycopg’s session-scoped range fixture creates testschema.testrange beside a bare testrange in one script, so the collision failed the fixture and cascaded to every range test. With the two types now coexisting, RangeInfo.fetch resolves each name to its own type — bare testrange, testschema.testrange, and the quoted "testschema"."testrange" form a sql.Identifier renders all reach the right oid, each carrying its own subtype, while typname stays unqualified. DROP TYPE testschema.testrange is schema-aware and leaves the bare type standing. The same fix is applied to CREATE TYPE ... AS ENUM, which had the identical last-part-only bug.

Fixed

• Schema-qualified CREATE TYPE s.t AS RANGE / AS ENUM no longer collides with a bare t; the two are distinct types, stored and duplicate-checked per (schema, name) with a schema-qualified catalog _id. CreateRange and CreateEnum thread the schema qualifier through the planner instead of dropping it, mirroring the composite-type template.

• to_regtype resolves a schema-qualified range or enum reference in bare, schema.name, and quoted "schema"."name" forms; a bare name resolves only the public type (matching the default search_path). DROP TYPE schema.name targets the schema-qualified type; public.name normalises to the bare name.

Rust PostgreSQL server: WITH HOLD / NO SCROLL enforcement and binary server cursors

The Rust secantusd-pg server now enforces the two cursor contracts a psycopg ServerCursor leans on. A cursor declared WITHOUT HOLD is closed at COMMIT (a later FETCH answers 34000 cursor does not exist), a WITH HOLD cursor survives it with its position intact, and ROLLBACK closes every cursor, holdable included — where before all cursors silently outlived the transaction that made them. A NO SCROLL cursor now rejects any backward FETCH/MOVE with 55000 cursor can only scan forward, matching the server it emulates.

Binary server cursors return real binary rows. A server cursor materialises its rows once, in text, at DECLARE — but psycopg requests BINARY on the FETCH, not on the DECLARE, and the frozen text bytes cannot be turned back into binary. The server now keeps the resolved per-column values behind a SELECT-sourced cursor and re-encodes them in the FETCH’s format through the same codec the live query path uses, so a binary FETCH FORWARD decodes correctly (before it handed the client text bytes tagged as binary, which decoded to garbage). The FETCH’s row description reports the binary format to match.

Along the way select generate_series(1, 2)::int4 — a cast applied to a set-returning function in a FROM-less target list — is planned instead of refused, carried as an ordinary per-row cast with the described column type taken from the cast. A WHERE clause over such a series is now refused rather than silently dropped, the same contract the FROM generate_series(...) form already held.

Added

• crates/secantus-pgserver: server cursors keep the resolved typed values of a SELECT source so a BINARY FETCH re-encodes them in binary; the FETCH’s RowDescription reports the requested format.

• crates/secantus-pgplan: select generate_series(...)::type (a cast over a FROM-less set-returning target) is planned as a per-row cast.

Fixed

• crates/secantus-pgserver: COMMIT closes non-holdable cursors and ROLLBACK closes all cursors, so a WITHOUT HOLD cursor is correctly invalid after COMMIT (34000) and cannot read discarded rows.

• crates/secantus-pgserver: a NO SCROLL cursor rejects a backward FETCH/MOVE with 55000 cursor can only scan forward.

• crates/secantus-pgplan: a WHERE clause over a FROM-less generate_series target is refused (0A000) instead of being silently ignored, which had returned rows the client asked to exclude.

The Rust PostgreSQL server honours transaction characteristics

psycopg exposes a connection’s isolation_level, read_only and deferrable as first-class attributes, and applies them by tacking the transaction characteristics onto the BEGIN it emits before each block (BEGIN ISOLATION LEVEL SERIALIZABLE READ ONLY DEFERRABLE), reading them back through current_setting('transaction_isolation') and friends. The Rust secantusd-pg server refused those characteristics, so sixteen of psycopg’s test_set_transaction_param_* conformance tests failed on the very first statement.

The server now parses the characteristics on BEGIN / START TRANSACTION, SET TRANSACTION, and SET SESSION CHARACTERISTICS AS TRANSACTION, and reflects them in the transaction_* / default_transaction_* GUCs a client reads back. SecantusDB is single-node, so the isolation level is accepted and reported but not enforced — every level behaves as the one snapshot the storage engine offers, which is exactly what a single-node PostgreSQL does. Opening a block resets transaction_* to the session default and overlays the named modes; ending it reverts to the default, matching PostgreSQL 14 exactly. A companion fix lets set_config($1, $2, false) plan over the extended protocol, where the name parameter is still unbound at DESCRIBE time.

All sixteen test_set_transaction_param_* tests now pass.

Added

• secantus-pgplan: TransactionModes on BEGIN / START TRANSACTION, and the SetTransaction / SetSessionCharacteristics statements for the two SET forms (SET TRANSACTION, SET SESSION CHARACTERISTICS AS TRANSACTION).

• secantus-pgserver: applies the modes to the transaction_* GUCs for the life of a block and reverts on commit / rollback; a default_transaction_deferrable default GUC.

Fixed

• secantus-pgplan: set_config(name, value, is_local) folds a NULL name to NULL during a DESCRIBE instead of erroring, mirroring current_setting, so a fully-parameterised set_config round-trips over the extended protocol.

$setWindowFields rejects options it used to ignore

An unknown field in a $setWindowFields stage was accepted and dropped. A caller who misspelled partitionBy got their accumulators computed over the whole collection as a single partition, and one who wrote range at the top level — where it looks plausible, but belongs inside a window — silently got the default window covering the entire partition. Both returned ok with a wrong answer rather than an error, which is the worst shape for an option: the caller believes they asked for something.

The same silence applied one level down, to unknown keys inside window, where a misspelled documents widened the window without saying so.

Both now match MongoDB: an unknown top-level field is 40415, a missing output is 40414, and an unknown window key is 9. Fixed on both servers.

Found by probing aggregation validation errors against MongoDB 8.2.11 while working a filed item about error codes — the silent acceptance was the more serious problem sitting next to it.

A $set of -0.0 over 0.0 was silently not stored

update_one({...}, {"$set": {"a": -0.0}}) on a stored 0.0 did nothing: modifiedCount: 0, no change-stream event, and a read-back of 0.0. The value the caller asked to store was never stored. mongod stores it and reports the update (probed 8.2.11, 2026-09-05).

storage.py guarded the write with if new != doc, and Python’s == treats -0.0 and 0.0 as equal — and an int 0 as equal to a float -0.0, so a numeric type change was dropped the same way.

Fixing it un-masked a second bug that had been cancelling it. Our $mul computed -0.0 for 0.0 * -1 where mongod keeps 0.0; the wrong product was previously discarded by the very comparison being fixed. mongod’s rule, measured across 15 shapes: a stored double or decimal zero keeps its own sign whatever the multiplier, while a non-zero result (0.0 * inf is NaN) writes normally and an int zero promotes and follows IEEE.

The change-stream diff was blind the same way, in two places: a $set of -0.0 over 0.0 produced an update event with an EMPTY updatedFields — the consumer was told the document changed and never told which field — and so did a numeric type change.

Equality and change detection are different questions, which is why this is not a one-line fix to the shared helper. mongod calls 0.0 and -0.0 the same value for $eq ($cmp is 0, find({a: -0.0}) matches a stored 0.0) and different for change detection. Folding the rule into bson_equal / expressions::py_eq would have broken $eq and query matching; both engines get a separate change-detection predicate instead.

A Rust test had canonised the bug as the specification: numeric_bridge_no_change asserted that {a: 1} -> {a: 1.0} emits nothing, justified by the comment 1 == 1.0 -> no update emitted — Python’s equality rule cited as though it were the server’s. It is replaced by tests asserting the measured behaviour, plus a guard that an unchanged value still emits nothing.

Remaining and deliberately untouched: $set of a NaN over the same NaN emits a spurious event here where mongod emits none. That is the per-OPERATOR nModified rule (mongod’s $set skips a byte-identical NaN while $inc: 1 on that same NaN writes, the stored bit pattern 000000000000f87f either way), already tracked in tasks/backlog.md.

Fixed

• storage.py: the write guard counts a difference in the ENCODED BSON as a change, so signed zero and numeric type changes are stored.

• update.py, crates/secantus-core/src/update.rs: $mul leaves a stored double / decimal zero’s sign alone.

• ordering.py (bson_same_stored_value), diff.py, crates/secantus-core/src/diff.rs (same_stored_value): change detection distinguishes signed zero and numeric type, recursing into arrays and subdocuments so the array fast path sees them too.

A dotted sort key skipped the documents it should have ranked

sort({"x.y": 1}) over array-of-subdocument data came back in the wrong order. mongod ranks x: [{y: 1}] among the documents that have an x.y — by 1, its representative element — and both servers ranked it with the documents that have none. Wrong order becomes wrong results as soon as a limit is involved.

The sort was resolving the path with a resolver that deliberately does not walk through an array; that behaviour is right for $set and projection and wrong here. The array-descending resolver already existed for index-key generation and already has mongod’s semantics, including stopping at one level — x: [[{y: 5}]] has no x.y on mongod either. Using it for the sort makes the in-memory order agree with an index walk by construction, which is the property that matters: an index must change speed, never results.

Nearly shipped with a regression, caught by re-measuring the undotted sort against the pre-change binary rather than assuming only the dotted case had moved: the representative-element rule was briefly applied twice, which put x: [[5]] among the numbers instead of the arrays.

Still divergent and filed: a dotted positional component (x.0) is ambiguous. mongod tries it as an array index and as a literal field name, descends only for the second reading, and raises 16746 Ambiguous field name found in array when a sort hits both. That is a separate piece of work in path resolution.

Fixed

• secantus.ordering / secantus-storage: a dotted sort key walks one array level, so sort({"x.y": 1}) ranks array-of-subdocument values by their representative element — the minimum ascending, the maximum descending — and a two-level path still counts as absent. The undotted sort is unchanged, and a unit test pins that it descends exactly once.

A sort by x.0 no longer reads an array as a number

When MongoDB sorts by a dotted path, it walks that path through arrays — and what it does at the last step depends on how that step got there. An array-valued key reached by a field name is descended one level; one reached by an index is used exactly as it stands. Both servers descended in every case, so {x: [[5]]} sorted by x.0 ranked with the numbers (as 5) instead of with the arrays (as [5]). That is the wrong order — and, the moment a limit is involved, the wrong documents.

Nothing had probed it. The engine-parity suites pin the Rust and Python engines to each other, and here they were wrong together; the standing index probe compares _id sets on purpose, so ordering stays out of its way. It took a probe that compares ORDER against a real mongod — now tools/probes/sort_path_resolution.py — to see it.

The same probe settled a second rule: mongod refuses, rather than guesses, a sort path whose numeric component names both an array index and a key of that array’s elements.

Fixed

• A sort key reached by an array INDEX is no longer descended. {x: [[5]]} sorted by x.0 sorts by [5]; {x: [{y: [1, 2]}]} sorted by x.y still sorts by 1, because that step is a field name. Both servers, find and $sort, ascending and descending.

• An ambiguous sort path is now the 16746 refusal mongod gives. A component is ambiguous when it is a valid index of the array and some element document carries that exact key — so {x: [{"0": 5}]} sorted by x.0 is refused, while x.1 over [{"1": 5}] (index past the end) and x.0 over [{"00": 5}] ("00" is not the key "0") are answered normally. The element carrying the key need not be the one at that index. The same paths in a filter are still resolved both ways, as mongod resolves them.

• The Rust aggregation $sort stage resolved its keys without walking arrays at all, so it disagreed with the same server’s find on 9 of 48 measured shapes. Both Rust sort paths now share one resolver with the storage layer.

• The Rust find handler reported this refusal under the update-side executor wrapper with an empty command name (Plan executor error during ::). Read commands now get mongod’s own Executor error during find command: <db>.<coll> :: caused by ::.

Measured against mongod 8.2.11 over 204 ambiguity shapes and 48 resolution shapes: 0 divergent on both servers.

An index could change which documents a query returned

Four defects in the index layer, three of them silent data loss: the query came back with fewer documents than the collection held, no error, and the same query on the same data returned everything as soon as you dropped the index. All four were found by running SecantusDB and MongoDB 8.2.11 side by side over randomised documents and diffing the result sets, and all four are now covered by that sweep plus a regression suite.

A sparse index omits documents that are missing the indexed field — but a query for null MATCHES them, because MongoDB’s query language treats an absent field as null. The planner used the index anyway, so find({a: null}) on a collection with a sparse index on a skipped every document that had no a at all. The same applied to $in lists containing null, and to any sort: a sort walks the whole index, so a sparse one truncated the result set outright. A sparse index is now only used when some indexed field carries a predicate that guarantees the field is present — and “could match a missing field” covers any comparison against null, not only $eq: {a: {$lte: null}} matches an absent field too, which the first version of the gate missed.

A compound sparse index was under-populated. MongoDB indexes a document that has at least one of the indexed fields, keying the missing ones as null; we required all of them, so {a: 1} (no b) never reached a sparse {a: 1, b: 1} index and find({a: 1}) lost it.

A partial index’s “does this query imply the filter?” check compared values in BSON sort order, where a string sorts above every number. MongoDB’s range operators are type-bracketed — {$gt: 0} matches numbers and nothing else — so the check concluded that {b: "x"} implied {b: {$gt: 0}}, used an index that does not contain those documents, and find({a: 5, b: "x"}) returned nothing.

Finally, a query naming only fields covered by a partial index’s filter ({b: 5} against an index on a partial on {b: {$gt: 0}}) left no key prefix to pin, built an empty lookup key and raised IndexError out of the command handler — which reached the client as an internal error rather than an answer.

Fixed

• storage.py: a sparse index holds an entry for any document with at least one of its indexed fields (_sparse_covers), matching MongoDB’s compound-sparse rule. An index built before this change under-indexes until it is dropped and recreated; nothing rewrites existing entries.

• storage.py: the index pickers refuse a sparse index for a query whose predicates could match a missing field (_sparse_index_usable / _predicate_may_match_missing), including the sort-acceleration pickers.

• storage.py: partial-filter implication is type-bracketed (_op_implies_bound), so a cross-type comparison no longer claims coverage the index does not have.

• storage.py: a query fully covered by a partial index’s filter scans that index instead of crashing.

Arithmetic and casts that overflow now say so

Python’s int is unbounded and its float saturates to infinity, so a result that no Postgres type can hold was computed silently. i + 1 on an int column answered 2147483648 — and sent it under oid 23, four bytes that cannot carry it — where PostgreSQL answers 22003 integer out of range.

The width has to come from the declared operand types, not from the value: s + 1 on a smallint is int4 arithmetic in PostgreSQL, so 32768 is a correct answer there, while 32767::smallint * 2::smallint overflows. The two are indistinguishable by value alone. The planner already implements PostgreSQL’s promotion table for the RowDescription, so it now stamps that width on each arithmetic node and the evaluator checks against it.

1e39::float4 was worse than wrong: struct.pack('!f', …) raised OverflowError and reached the wire as an XX000 internal error.

Fixed

• 22003 for integer overflow at int2 / int4 / int8 — through +, -, *, /, abs() and unary minus, and in every clause that reaches the scalar evaluator: projections, INSERT, UPDATE, subqueries, CTEs, GROUP BY, HAVING and ORDER BY, with or without a FROM.

• 22003 for float8 overflow and underflow, following PostgreSQL’s CHECKFLOATVAL rule — an infinite result is an error unless an operand was already infinite, and a zero result is an error unless zero was a legal answer.

• 22003 instead of XX000 for a float cast out of range, in both spellings PostgreSQL uses: casting from numeric or text quotes the input, while narrowing an existing double reports value out of range: overflow.

Still divergent

Arithmetic inside a WHERE clause. It lowers to a Mongo $expr evaluated by the operator engine the MongoDB server shares, and teaching that engine PostgreSQL’s integer widths would break the layer boundary.

array_agg told “no rows” and “a row with no value” apart

Fixed

• array_agg over zero contributing rows is NULL, not {} — which a caller could not recover from, since {} is itself a legal value. The same applies to json_agg and to a FILTER that matches nothing.

• An unmatched outer-join row contributes a NULL element, so array_agg(k.v) over a LEFT JOIN gives {NULL} rather than {}.

Both halves had to land together: $push of a missing field pushes nothing, so the pushed array could not tell the two cases apart until the value was wrapped to leave an explicit null behind. The wrap is per-aggregate — array_agg, json_agg and jsonb_agg keep null elements while string_agg skips them and is correct to answer NULL for a group of nothing but nulls.

• array_agg over a JOIN reports its element’s array type, where it had reported jsonb; the same call over a single table already did.

UPDATE … SET col = value rejects a value PostgreSQL would not assign

UPDATE t SET int_col = text_col and UPDATE t SET bool_col = 1 were silently coerced. PostgreSQL rejects both with 42804 column "…" is of type … but expression is of type …, because assignment goes through assignment casts — a different, more permissive rule than the implicit casts a comparison gets, which is why this needed its own analysis rather than reusing the existing cross-category comparison check.

The rule is the one PostgreSQL actually applies, measured across 25 shapes: a string target accepts anything (there is an assignment cast to text / varchar / char from every type involved), an expression whose type is not statically certain is left alone so a bad value still fails at runtime where PostgreSQL fails it, and otherwise a type-category mismatch is the error. That keeps bigint_col = int_col, int_col = real_col and int_col = 1.7 working while rejecting int_col = text_col.

Two coercion errors turned up in the same probe run and are fixed with it.

Fixed

• UPDATE t SET numeric_col = 'abc' reached the client as a raw [<class 'decimal.ConversionSyntax'>] — a Python exception on the wire. It now reports 22P02 invalid input syntax for type numeric: "abc".

• Date, time and timestamp values that cannot be parsed now report PostgreSQL’s 22007 invalid input syntax for type date: "nope" rather than an internal message, and timestamp coercion uses 22007 rather than 22P02.

Changed

• The SERIALIZABLE isolation-level divergence is now named in the SQL capability matrix, not only in the transactions section, and the documentation records why accepting the level and documenting it was chosen over rejecting it or reporting repeatable read.

Infrastructure

• CI now runs the PostgreSQL-oracle suites against a real PostgreSQL. They compare SecantusDB’s answers against a live server, and until now only ever executed on a developer machine that happened to have one — in CI they skipped silently. A Linux-only job stands up a pinned postgres:14.13 and fails if the server is unreachable, so the suites cannot go back to skipping unnoticed.

AT TIME ZONE, LIKE ALL/ANY, and the timezone_* extract fields

Three features the sixth sweep found refused, taking that sweep from 20 of 33 matching PostgreSQL 14.13 to 30.

Added

• AT TIME ZONE — the SQL operator, not a function. It reads both ways, and which way depends on the operand: a naive timestamp is interpreted as being in the zone and becomes an instant, while an aware one is converted into the zone and loses it. So '2020-06-15 12:00'::timestamp AT TIME ZONE 'America/New_York' is 16:00 UTC, and the same instant back through it is 08:00. An unknown zone is 22023.

• LIKE ALL(<array>) / LIKE ANY(<array>), and their ILIKE and NOT LIKE spellings. The scalar LIKE worked; the quantified form was 0A000 unsupported scalar expression.

• extract(timezone …) / timezone_hour / timezone_minute, reporting the session zone’s offset. PostgreSQL normalises a timestamptz into the session zone before extracting, so '…+05'::timestamptz gives 0 under a UTC session rather than 5.

Fixed

• The identity and generated-column insert errors put their explanation in DETAIL, as PostgreSQL does, instead of folding it into the message — which had made a message no client could match on. The HINT matches too.

Still divergent

to_ascii() and IS NORMALIZED (which parses as a column reference), and regexp_count / regexp_instr, which PostgreSQL 14 does not have either — only the error differs.

WHERE id IN %s is a syntax error, not zero rows

IN takes a parenthesised list or a subquery. Given a bare right-hand side — WHERE id IN %s, which is a common slip in psycopg where the working spelling is = ANY(%s) — SecantusDB compared against it and quietly matched nothing. PostgreSQL rejects it outright.

Returning no rows is the worst available answer for this: it looks like data rather than a mistake, so the query silently reports that nothing matched.

Fixed

• x IN <expr> without parentheses reports 42601 syntax error, as PostgreSQL does. Lists, single-element lists, NOT IN, subqueries and = ANY(…) are unaffected.

character(n) now pads the way PostgreSQL does

char(n) is the one string type whose stored form differs from its form in an expression: PostgreSQL stores it blank-padded but strips those blanks on every conversion to text. SecantusDB got the two halves from two different models — the column path stored unpadded and padded at the wire, while a ::char(n) cast padded eagerly into the value — so everything downstream of a cast saw blanks PostgreSQL had already removed.

Fixed

• ::char(n) no longer pads into the value, so 'a'::char(3) || '|' is 'a|' and length('a'::char(3)) is 1. A bare cast still reaches the client padded; the padding is applied at the wire, as it already was for columns.

• A cast of a non-string to char(n) applies the length limit: 123::char(2) is '12', not '123'. A char(n) target’s type tag is plain text, so the number-to-text conversion returned before the char-length check ever ran.

• LIKE, ILIKE, SIMILAR TO and ~ against a char(n) column match the blank-padded value, as PostgreSQL does. They are not blank-insensitive the way = is, so a char(5) holding 'ab' does not match LIKE 'ab' — SecantusDB was returning a row PostgreSQL excludes, in the WHERE clause as well as the select list.

• octet_length, concat, concat_ws, format, to_json, to_jsonb and a cast to bytea see the padded value, because they take it through the type’s output function rather than as text. length, upper, md5, left and position continue to see the stripped value.

ORDER BY … COLLATE "en_US.UTF-8" actually collates

A COLLATE clause on ORDER BY was accepted and then ignored — the rows came back in byte order, so a query that asked for a locale ordering silently got a different one. An unknown collation name was accepted too, where PostgreSQL rejects it.

The default ordering is unchanged and was never wrong: SecantusDB sorts text by bytes, which is exactly what a PostgreSQL database created with the C collation does. It now says so, rather than reporting an empty collation name.

Naming a locale gives the ordering you would expect — case and punctuation stop dominating, and accented letters sort beside their base letter instead of after z:

default / COLLATE "C"     ABC, Abc, ZZZ, a b, a-b, aBc, ab, abc, zzz
COLLATE "en_US.UTF-8"     a b, a-b, ab, abc, aBc, Abc, ABC, zzz, ZZZ

This needs no ICU library: it reuses the collation ordering already built for the MongoDB side. Two differences from PostgreSQL remain and are documented in the tests — ß is not expanded to ss, and - and _ take different relative weights.

Fixed

• ORDER BY … COLLATE "<locale>" orders by that collation instead of by bytes.

• An unknown collation reports 42704 collation "…" for encoding "UTF8" does not exist, as PostgreSQL does.

• SHOW lc_collate and SHOW lc_ctype report C instead of an empty string.

• pg_collation lists the collations that can be used, instead of being empty.

An abandoned COPY … TO STDOUT aborts the transaction, and INSERT checks its value types

A client that started a large COPY … TO STDOUT, read one row and gave up left the transaction usable. PostgreSQL leaves it in the aborted state, so the next statement gets “current transaction is aborted” — SecantusDB ran it instead. The copy sent its whole result in one write, so there was no moment at which it could notice the client had gone. It now sends in batches and checks for a cancelled statement between them.

The row count is why this went unnoticed: a small copy fits in the buffer and finishes before the client can abandon it, on PostgreSQL too. The difference only shows on a copy large enough that the server is still sending.

INSERT INTO t (col) VALUES (…) also now rejects a value PostgreSQL would not assign, the way UPDATE … SET already did — INSERT INTO t (jsonb_col) VALUES (42) is an error rather than a silent coercion. json and jsonb columns are type-checked at all for the first time, so comparing one against text is now reported instead of quietly answering false.

Fixed

• COPY … TO STDOUT is a cancellation point: abandoning one aborts the enclosing transaction block, as PostgreSQL does. A copy that completes still returns every row in text, CSV and binary formats.

• INSERT applies PostgreSQL’s assignment-cast rules, reporting 42804 for a value that cannot be assigned to the target column.

• A json / jsonb column compared against text or a number now reports operator does not exist, matching PostgreSQL, instead of silently evaluating to false.

Notes

Three long-standing entries turned out to be already fixed and were closed by measurement rather than code: sub-millisecond timestamps round-trip exactly (200 random values, zero loss) since the companion-field work; query-pipeline aborts already discard the rest of the batch and match PostgreSQL’s statuses and rows exactly; and index changes are already undone by ROLLBACK TO SAVEPOINT.

numeric beyond 34 significant digits was re-checked and does still round — that one is a real, permanent storage ceiling, already documented as such.

COPY learns FORCE_QUOTE, FORCE_NULL and FORCE_NOT_NULL

Added

• FORCE_QUOTE (col, ...) and FORCE_QUOTE * on COPY ... TO in CSV mode quote the named columns even where quoting is not required. A NULL is not force-quoted, which is what keeps it distinguishable from the empty string under FORCE_QUOTE *, and the HEADER line is not force-quoted either.

• FORCE_NULL (col, ...) on COPY ... FROM in CSV mode reads a quoted empty field as NULL; FORCE_NOT_NULL (col, ...) reads an unquoted empty field as the empty string rather than NULL.

Each option is valid in exactly one direction, and they disagree about which: FORCE_QUOTE is COPY TO only, FORCE_NULL and FORCE_NOT_NULL are COPY FROM only. All three are CSV-only, and an unknown column raises 42703.

sqlglot cannot parse any of them — it raises a hard error on the whole statement — so they are lifted out of the SQL text before parsing and re-attached to the syntax tree. That rewrite applies only to a statement beginning with COPY, and never inside a string literal.

Fixed

• DELIMITER '"' in CSV mode is refused with 22023 COPY delimiter and quote must be different, as PostgreSQL does. It was accepted, and produced CSV that cannot be parsed back.

to_char on an interval crashed, and day names ignored their case and padding

to_char(interval '3 days', 'DD') reported an internal error — to_char assumed a date, and an interval is stored differently, so it fell through to the date parser. Interval templates now work: DD, HH24, MI, SS, MM, YYYY and combinations of them. Calendar-name templates like Day are rejected with PostgreSQL’s own message, since an interval is not tied to a calendar date.

to_char(date, 'Day') returned Thursday where PostgreSQL returns Thursday — the full day and month names are padded to nine characters — and DY returned Thu rather than THU. The token’s own spelling decides both the padding and the capitalisation, and neither survived the conversion the formatter was doing.

Added

• to_date() and to_timestamp(), in both the format-string and epoch-seconds forms. They previously reported that a function called str_to_date was unsupported — a name the query never used.

• extract(century …), extract(millennium …) and extract(decade …).

Fixed

• to_char() on an interval.

• Day and month names pad to nine characters, and follow the case of the template (DAY upper, Day capitalised); FM suppresses the padding.

• to_date() reports a date and to_timestamp() a timestamp, rather than text.

Known limitation

An all-lower-case day or dy template renders capitalised. The parser maps the leading D before we see it, so day and Day are indistinguishable by then; DAY and DY are unaffected.

ALTER TABLE works outside public, and DDL errors name the right relation

ALTER TABLE never looked at the schema. It read the bare relation name and threw away the qualifier — whether that qualifier came from search_path or was written out as schema.table — so every form of the statement (ADD COLUMN, DROP COLUMN, RENAME COLUMN, RENAME TO, ALTER COLUMN, ADD CONSTRAINT, ADD PRIMARY KEY) answered relation "…" does not exist for any table outside public. TRUNCATE, CREATE INDEX, DROP TABLE and COMMENT ON had always resolved correctly, which is what kept it hidden.

Behind that sat a second defect: because the rename target was also read bare, a successful ALTER TABLE s.a RENAME TO b would have written the table back as public.b — moving it out of its schema.

The same probe run turned up three more, all found by running each shape against PostgreSQL 14.13 rather than reading the code. DROP SCHEMA … CASCADE dropped a schema’s tables and types but left its views, materialized views and sequences behind, and a bare DROP SCHEMA did not count them as dependants — so they outlived the schema and then collided with a later CREATE. A bare CREATE SEQUENCE ignored search_path and always created in public, so a following CREATE SEQUENCE schema.s saw a free name and quietly made a second sequence where PostgreSQL raises 42P07. And an error naming a relation reported the resolved catalog key instead of what the statement said, so a missing onlypub under SET search_path TO sa came back as relation "sa.onlypub" does not exist — naming a schema the user never typed.

Fixed

• ALTER TABLE resolves its target through search_path and honours an explicit schema.table qualifier, across all seven statement forms.

• ALTER TABLE … RENAME TO keeps the relation in its own schema, rejects a schema-qualified new name with 42601 (as PostgreSQL does), and answers 42P07 when the new name is already taken — including a rename to the relation’s current name.

• DROP SCHEMA … CASCADE drops the schema’s views, materialized views and sequences; without CASCADE those objects now raise 2BP01 like any other dependant.

• A bare CREATE SEQUENCE is created in the first schema on search_path.

• A “does not exist” error names the relation as the statement wrote it, and keeps an explicit public. qualifier.

• DROP TABLE on a missing relation says table "x" does not exist rather than relation "x", matching the noun PostgreSQL uses for each DROP verb.

• A DROP naming a relation that exists under a different kind answers 42809 "x" is not a table instead of claiming the object is absent — which had made DROP TABLE IF EXISTS <a view> succeed silently while the view survived.

• 42P07 relation "…" already exists names the bare relation, as PostgreSQL does, instead of the schema-qualified catalog key.

• Materialized views are schema-aware. Every matview path read the bare name, so a matview was created, catalogued and stored unqualified whatever schema the statement gave — SELECT … FROM schema.mv raised 42P01, and two schemas could not hold same-named matviews. CREATE / DROP / REFRESH / ALTER … RENAME TO and the not-populated check all resolve properly now, and DROP VIEW and DROP MATERIALIZED VIEW no longer reach each other’s relations.

• SELECT … INTO t works. PostgreSQL’s older spelling of CREATE TABLE t AS SELECT … was not dispatched at all: the target was resolved as if it were a source table, so every such statement failed with relation "t" does not exist.

• A sequence can be read as a relation — SELECT last_value, is_called FROM s — as it can in PostgreSQL. last_value reports the value actually handed out; reading the stored counter would have shown the pre-allocated batch’s high-water mark, which runs ahead of the sequence’s real position. (log_cnt is reported as 0: it counts values PostgreSQL has pre-logged to WAL and has no counterpart here.)

Changed

• CREATE TABLE AS, SELECT INTO and CREATE MATERIALIZED VIEW now report the number of rows they wrote as the driver’s row count, and send no row description. They carry a SELECT n command tag, and the wire layer took that tag alone as meaning “rows follow” — so each sent an empty row description and clients read the row count as 0.

Finishing the enum story

The previous batch fixed enum comparisons in a WHERE clause by rewriting a range comparison into the set of labels that satisfy it. A comparison in the SELECT list has to yield a boolean instead, and is evaluated by the scalar evaluator, which has no catalog — so SELECT m > 'ok' still answered by spelling while WHERE m > 'ok' did not. Two halves of one operator disagreeing is a worse state than either being wrong on its own.

Fixed

• An enum comparison in a projection. The planner stamps the declared label list on the comparison node for the evaluator to read, at the umbrella planner so every single-table shape it dispatches to is covered.

• enum_range(), enum_first() and enum_last(). They take their enum type from the argument’s cast — the argument is a NULL — so they cannot go through the value-only builtin table and were 0A000. An unknown type is 42704.

A plain text column still compares by spelling, which is what PostgreSQL does; only a column whose declared type is an enum is reordered.

Extended query protocol: three error-surface fixes

The Parse/Bind/Describe/Execute path is what psycopg, JDBC and most ORMs actually speak, and it is a different server path from the interpolated SQL a literal test corpus produces. Its value surface came back clean — every scalar type through Bind, parameters in predicates, select lists, LIMIT and RETURNING, and server-side cursors — but three statements were accepted that PostgreSQL refuses.

Fixed

• EXPLAIN with a parameter no longer violates the wire protocol. It described itself as returning no rows and then sent rows anyway, which clients report as server sent data ("D" message) without prior row description. Only a parameterised EXPLAIN reaches this path, because a parameter is what makes a driver switch away from the simple protocol.

• DECLARE CURSOR outside a transaction block now raises 25P01, as PostgreSQL does, instead of being accepted and then failing the following FETCH with 34000 — which reported the problem one statement late and blamed the wrong statement. DECLARE ... WITH HOLD is still allowed, since a holdable cursor survives the commit. The check applies to wire sessions only: the embedded run_sql API has no implicit commit, so a cursor declared there without a transaction stays usable, and the rule’s rationale does not reach it.

• Parameters in DDL now raise 42P02 there is no parameter $1. PostgreSQL binds parameters only into statements whose body is planned, so CREATE TABLE t AS SELECT $1 is accepted while CREATE VIEW v AS SELECT $1, CREATE INDEX, ALTER TABLE and a DEFAULT or CHECK holding a placeholder are not.

Changed

• tests/test_tmp_retention_guard.py gives its nested pytest subprocesses their own --basetemp and a timeout. Without a basetemp a nested pytest registers an exit-time cleanup of every stale pytest-of-<user> directory, which can run for minutes after its tests have all passed; under -n auto the outer worker is killed waiting and xdist reports only node down: Not properly terminated, naming no test. Same diagnosis and same fix as the nested runs in tests/test_crash_stall_watchdog.py.

Floats picked the wrong notation, on both float types

SELECT CAST(80 AS REAL) put 8e+01 on the wire where PostgreSQL puts 80. The value was never wrong — only its spelling — which is exactly why it survived so long: a driver decodes 80 and 8e+01 to the same Python float, so a value-level comparison cannot see it at all. It took registering a raw text loader and comparing the bytes.

Underneath was one confusion in two places. Emitting a float is two decisions — how many significant digits round-trip, and whether to print fixed or scientific — and both renderers derived the second from the first. PostgreSQL derives it from the exponent alone: scientific iff exp < -4 or exp >= 6 for float4, >= 15 for float8.

Fixed

• float4 went scientific far too early. The renderer searched for the shortest round-trip with f"{value:.{p}g}" and returned that string, but %g switches to exponent form whenever exp >= precision — so 80, which round-trips on a single digit, printed as 8e+01. Values needing more digits (3, 64, 123.456) were unaffected, which is why this looked value-specific rather than systematic.

• float8 went scientific too late. It used Python’s repr, whose threshold is 16 where PostgreSQL’s is 15, so 1e15 printed as 1000000000000000 and 9007199254740992 as itself, where PG gives 1e+15 and 9.007199254740992e+15. Found by sweeping float8 after fixing float4, on the suspicion that the same conflation lived there too.

Both now share typemap._shortest_round_trip, so the digit search and the notation rule cannot drift apart again. Verified against a live PostgreSQL 14 over 637 float4 and 620 float8 values — 400 of each random bit patterns across the whole range — with zero mismatches.

This also closes a sqllogictest gauge failure: random/select/slt_good_1.test compares raw text and now passes, taking that lane back to the committed 52/60.

Also

Two backlog corrections, both found by re-probing rather than reading: _pg_expandarray(...).x was still filed as returning TEXT but was fixed on 2026-08-29 (value and OID match PG for int, text and numeric arrays), and pg_typeof() over a set-returning function returns one row where PostgreSQL returns N — pre-existing, now recorded with its cause.

Full-text search: tsvector and tsquery stop leaking as JSON

A tsvector and a tsquery are dictionaries internally, so every path that did not know their type treated them as JSON and sent the internal representation to the client.

Fixed

• tsvector::text and tsquery::text render PostgreSQL’s form ('fat':2, 'fat' & 'cat') instead of {"tsvector": {"fat": [2]}}.

• length(tsvector) counts distinct lexemes. It was measuring the internal dictionary’s JSON — 45 for a two-lexeme vector.

• tsvector || tsvector concatenates. It fell into the hstore merge branch and returned only the right operand, silently dropping half the document; the second operand’s positions are now shifted past the first’s, as PostgreSQL does, so phrase queries over the result stay correct.

• && on two tsqueries is tsquery AND rather than array overlap, and tsquery || tsquery is OR.

• to_tsvector('simple', ...) honours its configuration and keeps stop-words. It dropped them under every configuration, losing tokens the caller had explicitly asked to index.

Added

• strip, numnode, querytree, tsvector_to_array, array_to_tsvector. querytree returns T for a query with no positive term, as PostgreSQL does.

The hypothetical-set aggregates

Added

• rank, dense_rank, percent_rank and cume_dist in their f(value) WITHIN GROUP (ORDER BY expr) form — what value would rank if it were inserted into the group. All four were 0A000.

They share the ordered-set plumbing percentile_cont / percentile_disc / mode already used, and needed a different payload and finish rather than new machinery.

The sort direction is part of the answer and NULLs take part in the ordering, so rank(20) WITHIN GROUP (ORDER BY v) and ... ORDER BY v DESC give different results on the same data — ASC defaults to NULLS LAST and DESC to NULLS FIRST. percent_rank and cume_dist use different denominators (N and N + 1), and only cume_dist counts the hypothetical row itself. On an empty group the four answer 1, 1, 0.0 and 1.0 rather than NULL.

The multi-column form (one argument per ORDER BY expression) is refused with 0A000 rather than answered approximately.

ROLLBACK TO SAVEPOINT now undoes index changes

An index created after a savepoint survived a rollback to it, and an index dropped after one stayed dropped. Everything else already behaved: tables, columns and views created or dropped after a savepoint were all reverted correctly, as was a full ROLLBACK of any index change. Indexes were the gap, because savepoints work by snapshotting table contents and an index is not stored as table contents.

Restored indexes keep what they were declared with — a unique index comes back still rejecting duplicates, and a partial index keeps its filter — rather than returning as a plain index of the same name.

The SQL guide previously stated the opposite, that DDL in general is not undone by ROLLBACK TO SAVEPOINT and gave CREATE TABLE as the example. That was wrong for every case it named and right only for the one it didn’t; it now describes the measured behaviour.

Integer columns accepted values they cannot hold

INSERT INTO t (i) VALUES (2147483648) into an int column succeeded and stored the value. The column’s declared type and its contents then disagreed, and the row description still advertised a four-byte integer for it. smallint and bigint behaved the same way, and 1e10::int returned a ten-digit number instead of failing.

PostgreSQL rejects all of these with “integer out of range”, and now so does SecantusDB — on INSERT, on UPDATE, and on an explicit cast. An expression that overflows cannot reach a column by any of those routes. Values at the exact boundaries are still accepted.

Fixed

• smallint, integer and bigint reject out-of-range values with 22003 … out of range, instead of storing them.

Known limitation

Arithmetic that overflows without being stored — SELECT 2147483647 + 1 — still returns the wide result rather than failing. Storing it anywhere is rejected.

sum(interval) answered 0 and avg(interval) answered NULL

An interval rides as a subdocument, and Mongo’s $sum over subdocuments is 0 while its $avg is NULL — so both came back as silently wrong data rather than an error. PostgreSQL gives 3 days and 1 day 12:00:00.

min / max were unaffected: Mongo’s BSON order over the subdocument happens to agree with duration order.

Fixed

• sum(interval) and avg(interval), whole-table and grouped, on the join planners as well as the single-table ones, and whether the aggregate stands alone or sits inside a computed projection. NULLs are skipped and zero contributing rows is NULL, as every SQL aggregate is. The fold is componentwise for the sum (PostgreSQL’s interval_pl) and carries months into days and days into micros for the average, which a per-field divide would get wrong.

• An interval inside an array rendered as its raw subdocument — ARRAY[interval '1 day']::text gave {"{\"interval\": {\"months\": 0, …}}"} where PostgreSQL gives {"1 day"}. The text cast defaulted every array element to text rather than inferring the element type from the values; the two shapes that carry real element identity (an inner array cast, an array of element casts) still take precedence.

The isolation level the SQL server provides is now documented and tested

No behaviour changes here — this records what the engine actually does, because it does not match what it reports.

Every explicit transaction runs on snapshot isolation, which is what PostgreSQL calls REPEATABLE READ, while BEGIN ISOLATION LEVEL echoes back whichever level was asked for. Measured against a real PostgreSQL, that lands differently for each level:

• Autocommit statements match PostgreSQL exactly. Two clients updating the same row both land their write; the second waits for the first and re-reads. This is the common path and it was already correct.

• READ COMMITTED inside an explicit transaction diverges. PostgreSQL blocks the second writer and completes it; we report a serialization failure, so a client that does not retry loses its write.

• SERIALIZABLE is over-claimed. We accept it and report it, but provide snapshot isolation, which permits write skew — two transactions can each read what the other is about to change and both commit. PostgreSQL aborts one.

All four cases are now pinned by tests, with the two divergent ones named so they read as known divergences rather than as conformance. Closing the READ COMMITTED gap needs a fresh snapshot per statement, which the storage engine does not offer within one transaction; what to do about SERIALIZABLE is a decision about what the server should promise, and is recorded rather than quietly chosen.

The jsonb function family: four no-ops and a family of wrong renderings

jsonb_set('{"a":1}','{b}','2') returned its input unchanged. So did jsonb_strip_nulls. Both are implemented — they only ever worked when the argument carried an explicit ::jsonb cast. A bare '{"a":1}' literal is PostgreSQL’s unknown, and there the function’s declared parameter type resolves it; here it stayed a Python str, the navigation had nothing to walk, and the call returned the input. A no-op that looks like a success.

Fixed

• jsonb_set, jsonb_insert, jsonb_strip_nulls, jsonb_typeof, jsonb_pretty and jsonb_array_length coerce an untyped string argument, as PostgreSQL’s parameter types do.

• jsonb_build_array(1,'x',true)::text renders [1, "x", true] rather than the PostgreSQL array {1,x,t}, and to_jsonb('x'::text)::text renders "x". Their values are ordinary Python lists, dicts and strings, so only the call says the rendering should be JSON. For json_agg the call is no longer visible by the time the cast runs — its operand is a synthetic column — so the planner marks the cast instead.

• jsonb_object_keys yields PostgreSQL’s storage order (shorter keys first, then bytewise). json_object_keys keeps the input’s own order and was right.

• jsonb_typeof(v->'arr') was 0A000 unsupported scalar expression: inside a function call, v -> 'arr' looks like an arrow-lambda to sqlglot’s parser, and only reaches JSONExtract when the left side is something an identifier cannot be. PostgreSQL has no lambda syntax, so a lambda there is always that misparse.

• jsonb_array_length(NULL) is NULL, not an error.

• to_char’s ISO-week tokens IYYY / IW / ID — 'IYYY-IW-ID' came out as the literal I20Y-IW-I3, the lone Y and D having matched and the Is not. IYY / IY / I and IDDD have no strftime directive and are recorded rather than guessed at.

• A json_agg inside a computed projection is typed json again. It had been typed by its ELEMENT since the nested-array_agg fix, which those two share a registrar with.

Still divergent

string_agg(DISTINCT x, sep) and jsonb_agg(...) inside a computed projection — sqlglot models the latter as an anonymous call, which the aggregate collector does not look for.

jsonb_path_query returned one row; to_number returned NULL

Fixed

• jsonb_path_query is set-returning and was not registered as such, so a path matching many values produced a single row — SELECT count(*) FROM jsonb_path_query('{"a":[1,2,3]}', '$.a[*]') answered 1 where PostgreSQL answers 3. Rows were silently missing, not values wrong.

• A jsonpath predicate can now be used as a whole path expression, so jsonb_path_match(j, 'exists($.a)') works. Note the rule this exposes: a predicate path yields one boolean item, so jsonb_path_exists(doc, 'exists($.zz)') is true even when $.zz is absent, while jsonb_path_query of the same path returns false.

• to_number(text, format) answered NULL for every input. sqlglot gives it a dedicated node rather than an anonymous call, so the name-keyed dispatch never saw it. It now parses under the format mask: decoration (, G L $ %) is dropped, the sign may lead, trail or be angle brackets, excess decimals are truncated rather than rounded, and input with no digits raises 22P02.

LISTEN / NOTIFY: a notification delivered twice

Fixed

• pg_notify() sent its notification twice when the call carried a parameter. Describe evaluates a FROM-less SELECT to learn its column shape, and the table that exists so a volatile call’s shape is derived statically instead of run — engine._VOLATILE_FN_TAGS — was missing pg_notify, so Describe sent the notification and Execute sent it again. Only the extended protocol reaches it, because a parameter is what stops a driver using the simple one, so the literal spelling always looked correct. nextval, pg_sleep, the advisory locks, and INSERT with parameters were checked and were already correct.

• An identical (channel, payload) signalled more than once in a transaction is now delivered once, as PostgreSQL does, so a loop that notifies per row wakes a listener once rather than once per row. Distinct payloads on the same channel are still all delivered. Both NOTIFY and pg_notify() collapse — the two spellings of one operation had different semantics.

• A payload of 8000 bytes or more raises 22023 payload string too long instead of being delivered. 7999 is accepted.

isfinite, scale, cbrt on a numeric, and justify_* on a time

Looking for the opposite of the last round — shapes PostgreSQL accepts and SecantusDB refused — turned up 21, across five functions.

isfinite() and scale() were missing entirely. cbrt() worked on a whole number but not on a decimal one, and justify_hours() / justify_days() refused a time, which PostgreSQL reads as an interval of that length.

cbrt() was also inaccurate: cbrt(1000000) returned 99.99999999999997 rather than 100.

Added

• isfinite() and scale().

Fixed

• cbrt() accepts a decimal argument, and is exact for perfect cubes (math.cbrt on Python 3.11+, a Newton-refined fallback on 3.10).

• justify_hours(), justify_days() and justify_interval() accept a time.

Still unsupported

numnode() and strip() (full-text search), and hashtext() — PostgreSQL’s internal hash, whose values cannot be reproduced, so returning a different number would be worse than refusing.

A nested array_agg silently dropped its ORDER BY

SELECT array_agg(i ORDER BY i DESC) FROM t sorted. Wrap it in anything at all — a cast, an operator, a subscript, another function — and it returned insertion order instead, with no error.

array_to_string(array_agg(x ORDER BY y), ',') is the shape that makes this look like ordinary SQL and get a wrong answer.

The top-level projection path pushed {v, k} pairs and sorted them in a post-aggregate step. The registrar used for an aggregate nested inside a computed projection registered a plain $push and never carried the ordering at all — and EvaluatedSelectPlan had nowhere to carry it, so the fix adds the same post_aggregates channel the pipeline plan already had.

Fixed

• An in-call ORDER BY is honoured wherever the array_agg appears: under a cast, an operator, a subscript, a function call, alongside other aggregates, grouped or not, and on the join planners as well as the single-table ones. Multi-key and per-key directions included.

Still refused

GROUPING SETS with any computed projection — count(*)::text is rejected just as array_agg(…)::text is, so this is not about ordering. It stays an honest 0A000.

A function given the wrong kind of value no longer reports an internal error

SELECT abs('abc'), SELECT date_trunc(1, 2), SELECT repeat(ARRAY[1,2], 'x') and several hundred shapes like them reported internal error. A Python error from inside the function was reaching the client unchanged.

They now report function abs(unknown) does not exist, which is what PostgreSQL says for the large majority of them.

This came out of a deliberate hunt rather than another accident: two of these had already turned up by chance in consecutive rounds, so every function was tried against every value type. 397 shapes reported an internal error; none do now. The guard sits at the two points where functions are evaluated, not inside each function — per-function guards are exactly what left the holes.

A function that already reports a proper error keeps it: to_char on an interval with a day-name template still reports the format error, rather than being flattened into a generic one.

Fixed

• A scalar function applied to a type it does not support reports 42883 function … does not exist instead of internal error.

avg, stddev and variance over exact types answered the wrong type

PostgreSQL accumulates N, sum(X) and sum(X²) as numerics and finishes in numeric arithmetic, so an integer or numeric input gets an exact numeric answer whose scale comes from select_div_scale. This engine used Mongo’s float accumulators — $avg, $stdDevSamp — and squared the stddev for the variances. Every one of them came back float8 where PostgreSQL says numeric, with the last digits wrong and no scale at all: 2.333333333333333 for PostgreSQL’s 2.3333333333333333.

Fixed

• avg, stddev, stddev_samp, stddev_pop, variance, var_samp and var_pop over smallint / integer / bigint / numeric are computed exactly and reported as numeric, at PostgreSQL’s derived scale — avg(i) over 1, 2, 4 is 2.3333333333333333, and over a single 1 it is 1.00000000000000000000.

• A non-positive numerator short-circuits to a plain 0, as PostgreSQL’s const_zero does. Dividing instead would have given 0.00000000000000000000 for a constant column.

• variance and var_pop inside a computed projection (variance(i)::text) work at all; they were 0A000 aggregate variance is not supported.

• A float input now reports float8, which is what PostgreSQL answers. The variances claimed numeric for a value that had been a float all along — an existing test asserted that, and has been corrected against the reference server.

Still divergent

stddev / variance over a float input differ in the last bit or two: PostgreSQL uses the Youngs-Cramer update there, which is a different rounding path from summing the squares. avg(DISTINCT …) and an aggregate with a FILTER clause keep the float path.

numeric(p, s) never applied its declared scale

PostgreSQL rounds a stored value to the declared scale: 0.12345 into a numeric(10,3) column is stored as 0.123, and a bare 1 as 1.000. This engine kept whatever scale the literal happened to carry, so the stored value itself was wrong — not merely its rendering — and every sum, min / max, avg and arithmetic result over the column inherited the error.

sum(a) over 1, 2.5 and 0.12345 in a numeric(10,3) column answered 3.62345 where PostgreSQL answers 3.623.

Fixed

• A declared numeric(p, s) rounds to s on write — INSERT, UPDATE, parameterised statements and COPY alike — half away from zero, matching PostgreSQL both signs.

• A value whose integer part does not fit after rounding is 22003 numeric field overflow, with PostgreSQL’s DETAIL line naming the precision, the scale and the limit. The check runs on the rounded value, so 9999.999 into a numeric(6,2) overflows.

• NaN and NULL are stored untouched, and an unconstrained numeric keeps its own scale, as PostgreSQL does.

Three operator-typing gaps on the PostgreSQL interface, and one sqlglot mis-parse behind them

Follow-on to the result-type work: the three items that fix left recorded rather than closed, each measured against a live PostgreSQL 14 before being worked. Two turned out to have a different root cause than the note predicted, and one uncovered a second bug sitting next to it.

Fixed

• pg_typeof(...) wired as text (25) instead of regtype (2206). Universal, not specific to any argument: the value was right every time and only the declared type was wrong, so nothing that compared values could see it. regtype is now a real type tag. The literal rewrite_pg_typeof mints is tagged rather than wrapped in a ::regtype cast — the cast is evaluated, and 'int4'::regtype evaluates to the type’s OID number, so wrapping changed the answer from integer to 23. An explicit 'int4'::regtype now wires as regtype too; the ::text and ::oid forms are unchanged.

• A typed text operand in arithmetic answered the wrong code. Postgres defines no arithmetic operator on text at all, so the content is irrelevant: '1'::text + 1 errors exactly as 'a'::text + 1 does. We coerced both, so the first silently answered 2 and the second reported the coercion’s 22P02 instead of 42883. An explicit cast is decided in sql/scalar.py (a cast is unambiguously typed, and a constant-only statement never reaches the plan-time analysis); a text column is decided in sql/typecheck.py, which owns the exemptions a declared type needs — reflected schema-on-read tables above all, where a column typed text from a 50-document sample may hold numbers. Postgres names the declared type, so a varchar column reports “character varying”.

• interval '1 day' + 1 answered 1 day 00:00:01. Not an evaluator bug: sqlglot absorbs a following NUMBER into its multi-part interval form (INTERVAL '1' DAY '2' HOUR), a syntax Postgres does not have, so the 1 was read as one second. The distinction cannot be recovered after parsing — sqlglot rewrites the numeric token into a string literal inside the synthesised Interval, leaving + 1 and + '1' byte-identical in the AST — so it is corrected at the parser, where the token is still visible. + 'string' is deliberately untouched: Postgres resolves the unknown literal to an interval there, which is what the continuation already computed.

• interval '1 day' - 1 answered 22023 cannot delete from scalar — a jsonb error for an interval. Intervals and ranges ride as tagged subdocuments, so the jsonb - key branch claimed them. Found while probing the item above.

tests/test_sql_operator_types.py pins the constant forms over the real wire, asserting the declared OID and the SQLSTATE alongside the value, and compares 22 shapes against a live PostgreSQL when one is reachable. tests/test_sql_typecheck.py grows the column-typed half, including the reflected-table exemption.

ORDER BY over a jsonb or range column was an internal error

Both ride as bare Python subdocuments, so the sort’s x < y raised TypeError: '<' not supported between instances of 'dict' and 'dict' and the client saw XX000.

PostgreSQL’s jsonb order was measured against 14.13 rather than taken from the manual, which matters: a top-level empty array sorts before everything, null included. That is not a documented rule but a consequence of storage — a top-level scalar is held as a one-element array, so [] is simply the shorter container. Nested, [] is an ordinary array.

The key has to be decided from the column, not the value. Keying only the values that fail to compare gives an order that is not even transitive, because Python compares False < 1 quite happily — false landed between two numbers.

Fixed

• ORDER BY over jsonb, ascending and descending, with either NULL placement: the full type order, arrays by length then element-wise, objects by pair count then key/value pairs walked in PostgreSQL’s storage order.

• ORDER BY over a range type: empty first, then by lower bound (unbounded lowest), then by upper (unbounded highest).

• OVER (ORDER BY …) and array_agg(x ORDER BY x) over a partition of structured values.

Still divergent

'null'::jsonb and SQL NULL are both Python None here, so a JSON null sorts where SQL NULL does rather than inside the jsonb order. And array_agg(x ORDER BY <mixed jsonb>) builds its sort key inside the pipeline, where the column type is not available — it no longer errors, but a column mixing scalars with containers is ordered inconsistently there.

ORDER BY on a numeric column was an internal error

SELECT n FROM t ORDER BY n answered XX000 internal error. Sorting compares stored values directly, and Decimal128 — which is every numeric and money value — implements no Python numeric protocol at all, so the comparison raised a bare TypeError. An interval rides as a subdocument and had the same problem.

It reached every sort path that does not delegate to Mongo: a plain ORDER BY, a window’s OVER (ORDER BY …), array_agg(x ORDER BY x), WITHIN GROUP (ORDER BY …), and every window aggregate except count, the one that never looks at the value. DISTINCT, GROUP BY and UNION were unaffected, because those sorts do go to Mongo — which is how a bug this plain went unnoticed.

Decimal128 was wrong even where it did not raise: its equality compares the BID encoding, so 1.0 and 1.00 were different values and rank() made two peers into two ranks.

Fixed

• ORDER BY over numeric, money and interval, ascending and descending, with either NULL placement. Intervals order by duration, which is what PostgreSQL compares.

• OVER (ORDER BY …) for every window function, and rank() / dense_rank() now tie equal-but-differently-scaled numerics as the peers they are.

• Window sum / avg / min / max over numeric, money and interval. min and max follow PostgreSQL’s fold, where an equal value replaces the running one — over 2.5, 1.0, 1.00 the minimum is 1.00.

• array_agg(x ORDER BY x) and percentile_cont / percentile_disc / mode WITHIN GROUP (ORDER BY x) over the same types.

Ordering keys are now computed once per row rather than on every comparison, which is where the normalisation lives.

Still unsupported

ORDER BY over jsonb or a range type. Both ride as subdocuments, and PostgreSQL’s total order over them is a slice of its own rather than a coercion.

array_agg(t ORDER BY t) returns the microseconds it stored

An ordered aggregate over a timestamp column sorted by microseconds and then returned every element rounded to the millisecond — times that were never stored. The sub-millisecond remainder was attached to the sort key but not to the value being collected.

This was invisible from the shape that already worked: array_agg(x ORDER BY t) aggregates a different column, so only the key mattered there. It only appears when the column being collected is the timestamp itself.

Fixed

• array_agg(t ORDER BY …) and the ordered-aggregate path generally return microsecond-exact timestamps.

A partial index reports its WHERE clause

pg_indexes.indexdef and pg_get_indexdef() rendered a partial index as though it covered the whole table — the predicate was simply missing from the generated CREATE INDEX statement. Anything that recreates an index from that string, which is the usual reason to read it, built a full index instead of a partial one.

The predicate is now reconstructed, matching PostgreSQL’s own rendering exactly across the comparison operators, IS NOT NULL, AND/OR combinations, and string literals with their ::text cast. WHERE b <> 5 round-trips too, which takes a little care: it is stored internally as “not equal and not null”, and PostgreSQL reports the predicate you wrote.

A predicate shape that cannot be reproduced exactly still renders without the WHERE, as before. That is deliberate — an approximate predicate in a statement meant for recreating an index would build the wrong index quietly, which is worse than an obviously incomplete one.

A record rendered as JSON, and half of strip_nulls

Fixed

• record::text renders PostgreSQL’s record literal — ('a', 1)::text is (a,1), not {"f1": "a", "f2": 1}. The record renderer already existed and is what the wire uses for a composite column; only the cast did not route to it. Field quoting, doubled quotes, NULL and empty-string fields, and a blank-padded char(n) field all match.

• json_strip_nulls answered NULL for every input, while jsonb_strip_nulls worked. sqlglot gives the non-b spelling its own node and leaves the b spelling an anonymous call, so the name-keyed dispatch served only one of them.

ORDER BY works over _pg_expandarray, and its value column has a type

Selecting the value field of a record set-returning function — (information_schema._pg_expandarray(arr)).x, the shape JDBC’s metadata queries use — had two problems.

Ordering the result did not work. ORDER BY <alias> and ordering by the subscript field both failed outright with “column does not exist”, because the name only exists in the expanded output, not in the source row. ORDER BY 1 was worse: it was accepted and then ignored, so the query reported success and returned the array’s own order. All of these now sort, matching PostgreSQL.

The value column also reported no type at all rather than the array’s element type, so a client asking what it had just selected got nothing usable. It now reports int4 for an integer array, text for a text array, numeric for a numeric one — the subscript field was always correct.

One test that pinned the old ordering behaviour has been rewritten. It was named so it could not be mistaken for intended behaviour, and it failed as soon as the behaviour improved, which is what it was there for.

substring(text from pattern) crashed, and LIKE ignored its default escape

substring('abc123' from '[0-9]+') — the pattern-matching form — reported an internal error. The pattern was being read as a starting position.

'a_c' LIKE 'a\_c' was false. Backslash is PostgreSQL’s default escape character in LIKE, so the backslash should make the _ match literally; SecantusDB only honoured an escape when one was written out with ESCAPE. ESCAPE '', which genuinely turns escaping off, still does.

Three more expressions reported the wrong type, so their values arrived as text: BETWEEN and EXISTS sent 't'/'f' instead of booleans, and a scalar subquery sent the string '1' instead of an integer.

Fixed

• substring(text FROM pattern) returns the first capture group when the pattern has one, the whole match when it does not, and NULL when it does not match — instead of failing.

• LIKE treats backslash as its escape character by default, as PostgreSQL does.

• BETWEEN, EXISTS and scalar subqueries report their real types.

• array_replace() keeps the array’s type instead of rendering it as text.

Added

• regexp_match, regexp_split_to_array, string_to_array, array_replace.

The two-argument statistical aggregates

Added

• corr, covar_pop, covar_samp, and regr_avgx, regr_avgy, regr_count, regr_intercept, regr_r2, regr_slope, regr_sxx, regr_sxy, regr_syy — all previously 0A000.

They are one feature rather than twelve: every one is derived from the same six sums, so they share a single accumulator set and a single post-aggregate, and differ only in the finishing arithmetic. A pair contributes only when both arguments are non-null, which is what makes regr_count disagree with count(*) and what keeps the means over the same population the count reports.

regr_count returns int8 and is the only one defined over an empty input (0, where the others are NULL); the rest return float8.

A computed SQL column declared a type its own value could not be decoded as

Seven wire-level divergences on the PostgreSQL interface, all one shape: the server computed the right value, declared a type taken from the wrong operand, and the client raised while decoding it. Found by probing our PG server against a live PostgreSQL 14 and comparing the result OID as well as the value — an in-process comparison of the same statements showed nothing, because the values were never wrong.

Fixed

• jsonb || jsonb and jsonb - key lost the jsonb type. || typed as text and - typed from its right operand (int4 / numeric), so a jsonb payload went out under a numeric OID. Every jsonb - anything was a hard client-side failure — psycopg raised invalid literal for int() with base 10: '{1,3}' — and || answered a PG array literal ({1,2,3}) instead of jsonb ([1, 2, 3]). #- (JSONBDeleteAtPath) was already correct. Result: 13 shapes now match PG byte-for-byte, OID included.

• An unknown literal widened instead of resolving to the other operand’s type. Postgres coerces an untyped literal to the other operand’s type before choosing an operator, so that type decides both the parse and the error. '1.5' + 1 is integer input and fails 22P02; we answered 2.5 under the int4 OID the literal 1 had already fixed, and psycopg raised. The 22P02 message now names the target type, as PG’s does.

• A date-shaped literal did date arithmetic. '2020-01-01' + 1 is integer input in PG (22P02); we answered '2020-01-02' under an int4 OID — again a client-side crash. Only a bare literal is judged: a ::date cast or a date column keeps its date arithmetic.

• Beside an interval, the unknown literal must become an interval. '2020-01-01' + interval '1 day' is 22007 in PG; we read the literal as a date and answered a timestamp under the interval OID, so psycopg raised can't parse interval '2020-01-02 00:00:00'. Restricted to + / -: for * / / PG resolves the unknown to a number instead, so interval '1 day' * '2' is still two days.

• Boolean arithmetic was accepted. Postgres defines no arithmetic operator on boolean, but Python’s bool is an int, so nothing raised: true + 1 quietly answered 2 and true - false answered 1. Both now answer 42883.

tests/test_sql_result_type_tags.py pins all of it over the real wire, asserting the declared OID alongside the value, and compares 20 shapes against a live PostgreSQL when one is reachable.

Two more internal errors, and three wrong answers, from a second sweep

(1,2) < (1,3) was XX000. A record rides as a dict of f1..fN, and a dict has no < — equality worked, so only the ordering comparisons failed.

FETCH FIRST 2 ROWS ONLY — the SQL-standard spelling of LIMIT that PostgreSQL accepts — was XX000 twice over. It parses as exp.Fetch, whose count lives in count rather than expression, and the “unsupported” error built from that None then raised AttributeError on None.sql(). So the query could not even say why it failed.

3 BETWEEN SYMMETRIC 5 AND 1 answered FALSE: the keyword was parsed and then ignored, so every reversed-bound test was wrong.

Fixed

• Records compare field by field, left to right.

• FETCH FIRST n ROW[S] ONLY, with or without an OFFSET.

• BETWEEN SYMMETRIC orders its bounds first. Plain BETWEEN still does not.

• jsonb ? 'k' and its ?| / ?& siblings work in a SELECT list — they reported function jsonb_contains() is not supported, a name the user never wrote, and for the two-key forms a name mangled out of the node class. They worked inside a WHERE all along. An object is asked about its keys, an array about its string elements, and a jsonb string about equality, which is PostgreSQL’s rule.

• Every containment and key-existence operator (@>, <@, &&, ?, ?|, ?&) types as boolean. They typed as text, so a driver was sent 't' under oid 25 where PostgreSQL sends a boolean under oid 16.

Nine missing functions, and four expressions that reported the wrong type

md5, btrim, quote_ident, quote_literal, quote_nullable, concat_ws, starts_with, width_bucket and div were unavailable. The error said the function was “not supported in this context”, which was misleading — they were unreachable in every context.

Four expressions returned the right value with the wrong type, which is the worse half because nothing reports it. coalesce(NULL, NULL, 3) sent the string '3' typed as text where PostgreSQL sends 3 as an integer, and IS DISTINCT FROM sent 't' as text where PostgreSQL sends a boolean — so a driver reading the column got a string that is always truthy. power() and sign() reported numeric where PostgreSQL reports double precision.

Added

• md5, btrim, quote_ident, quote_literal, quote_nullable, concat_ws, starts_with, width_bucket and div.

Fixed

• coalesce(…) reports the type of its arguments instead of text.

• IS DISTINCT FROM / IS NOT DISTINCT FROM report boolean instead of text.

• power() and sign() report double precision, as PostgreSQL does.

Fixed (aggregates)

• bool_and(n > 0) and bool_or(n > 0) return the right answer instead of NULL. Over a plain boolean column they were always correct; given a comparison they silently returned nothing at all. A row where the comparison is NULL is skipped, as PostgreSQL does, rather than counting as false.

• sum(CASE WHEN … THEN … ELSE … END) — the counting idiom — works.

• An aggregate argument that cannot be handled now says “unsupported aggregate argument” instead of naming array_agg for a sum() or min() call.

SET search_path now decides which schema a bare table name resolves to

The setting was recorded — SHOW search_path returned it — but ignored when resolving an unqualified relation. With the same table name in two schemas, the answer never moved: SET search_path TO sa still read public.t, and sa, public and public, sa gave the same result, so the order asked for made no difference.

Three behaviours change, each checked against a real PostgreSQL:

• The path is walked in order, and the first schema holding the name wins.

• A relation in no schema on the path is invisible rather than lower priority — selecting a public-only table with search_path set elsewhere now reports that the relation does not exist, where it previously returned rows.

• CREATE TABLE creates into the path’s first schema, so a same-named relation there is now a conflict. Previously the table was created in public while every read of that same name resolved to the other schema — writes and reads landing in different places.

Two existing tests asserted the old behaviour and have been rewritten. One of them contradicted its own docstring, which described PostgreSQL’s rule correctly.

One difference remains, recorded rather than silently left: when a bare name resolves nowhere, the error names the schema that was tried ("sa.onlypub") rather than the name as written ("onlypub"). The error code is correct.

SERIALIZABLE is documented as snapshot isolation

The SQL server accepts all three isolation levels and reports back whichever was requested, but every explicit transaction runs on the storage engine’s snapshot isolation — what PostgreSQL calls REPEATABLE READ. For SERIALIZABLE that difference has teeth: snapshot isolation permits write skew, so two transactions can each read what the other is about to change and both commit, where PostgreSQL aborts one.

The SQL guide now spells this out, with the four-way comparison against a real PostgreSQL, a worked write-skew example, and what to use instead when an invariant genuinely needs protecting (an explicit lock, or a constraint the database can check).

Mapping SERIALIZABLE onto snapshot isolation is deliberate and has precedent — Oracle has long done the same — and it keeps drivers and ORMs that request the level working. Being silent about it was the problem, not the mapping.

Full-text search now stems

to_tsvector('english', …) did not stem, so cats did not match cat and a search for quick did not find a row whose title is Running quickly. That is the worst class of search defect: a query that should match returns nothing, with no error.

Added

• secantus.sql.snowball implements the English (Porter2) algorithm PostgreSQL’s english configuration uses. It is written out rather than taken from a dependency, since SecantusDB ships self-contained wheels and a stemmer is a closed, fully specified algorithm. It is pinned against 6,094 words stemmed by PostgreSQL itself (tests/data/english_stems.txt), and matches on every one.

Fixed

• Documents and queries both stem, which is the point — a query’s running and a document’s runs meet at run. Prefixes stem too, so running:* renders as 'run':*.

• to_tsquery drops stop-words, as the document side always did. Keeping them produced a query that could never match, since no document indexes them.

• simple still neither stems nor drops stop-words.

Nine divergences a broad sweep against PostgreSQL turned up

A corpus of ordinary SQL run against both servers and diffed — not read off the backlog. Three were silently wrong answers rather than errors.

trim(both 'x' from 'xxabxx') answered 'xxabxx': the trim characters and the position were both ignored, and every spelling ran a plain str.strip(). Only the SQL keyword form was affected — btrim / ltrim / rtrim take their characters as an ordinary second argument and were always right.

substr('abcdef', -1, 3) answered 'abc'. The start was clamped to 1 and the length counted from there; PostgreSQL counts from the original start, so positions -1, 0 and 1 leave just 'a'.

unnest('{1,2,3}'::int[]) handed a driver '1', '2', '3' — the elements were typed any and went out as text.

Fixed

• TRIM([LEADING|TRAILING|BOTH] [chars] FROM string) honours both the character set and the position.

• substr() counts the length from the original start, and a negative length is 22011 negative substring length not allowed.

• unnest() without a FROM types its elements: from the cast where there is one, otherwise from the values, with PostgreSQL’s own rule that an ARRAY[…] of integer literals is integer unless one does not fit.

• A NULL array is empty in a || concatenation, so NULL::int[] || 9 is {9} where it used to be NULL — while a NULL of any other type still makes the whole || NULL, and a NULL element stays a NULL element. A column’s array-ness comes from the type-checking pass, a cast or ARRAY[…] from the node itself.

• to_hex(), make_date(), make_time() and make_timestamp(), each of which was 0A000 function … is not supported — and two of them under a name the user never wrote, because sqlglot renames them.

Still divergent

SIMILAR TO; date_part() returns numeric where PostgreSQL returns float8 (sqlglot parses it and extract, which is numeric, to the same node); and two-argument log() returns float8 where PostgreSQL returns a numeric whose scale comes from an estimator in log_var that has not been ported.

min() and max() on a timestamp no longer answer a time that was never stored

min(t) over a column holding 12:00:00.123456 returned 12:00:00.123000. Not a rounding of the display — the query answered a timestamp that is not in the table and never was. max() did the same, and array_agg(x ORDER BY t) and string_agg(x, ',' ORDER BY t) ordered their output at millisecond granularity, leaving rows that share a millisecond in storage order.

These are computed inside the aggregation pipeline rather than in the SQL layer, which is why the read-side fix for SELECT and ORDER BY never reached them: a BSON date holds whole milliseconds and the microseconds live in a separate hidden field the accumulator never saw. They now accumulate the two together as a single sortable value and recombine it on the way out. FILTER (WHERE …), all-NULL groups, and the join form are covered.

HAVING moved with them, and it is worth saying why. A clause like HAVING min(t) > '12:00:00.1230' compares against the accumulator’s output, so the literal had to be handled in the same representation. Measured against PostgreSQL across five operators before and after: three were already wrong, one — min(t) = '…122000' — was right only because its literal happened to fall on a whole millisecond, and all five are right now. Anything that changes this area should re-measure HAVING rather than assume it follows.

Still not fixed, and recorded: GROUP BY on a timestamp merges rows that differ only in microseconds, because the group key is still the truncated date.

SELECT DISTINCT on a timestamp keeps rows a microsecond apart

Two rows recorded at .123100 and .123500 came back as a single row reading .123000 — collapsed together, and reported as a time neither of them held. DISTINCT deduplicates on the projected value, and the projection was dropping the microseconds before the deduplication ever ran.

count(DISTINCT t) was wrong for the same underlying reason but by a different route, and answered 2 where PostgreSQL answers 3.

Both are fixed, along with DISTINCT *, multi-column DISTINCT, and ordering over deduplicated output.

This completes the sub-millisecond work: comparisons, reads, ORDER BY, min / max, ordered aggregates, GROUP BY keys, HAVING, DISTINCT and count(DISTINCT …) are now all microsecond-exact, verified across 39 query shapes against a live PostgreSQL.

GROUP BY on a timestamp no longer merges rows that differ in microseconds

Grouping by a timestamp column used the whole-millisecond value as the key, so rows recorded at .123100 and .123500 landed in the same group. Three distinct times became two groups; count(*) answered 3 where PostgreSQL answers 2 and 1, sum(id) answered 6 where PostgreSQL answers 4 and 2, and the group key came back as .123000 — a time none of the rows held.

The aggregate values are the important part. A merged group does not just label itself wrongly, it sums and counts rows that belong to different groups, and nothing about the result looks suspicious.

HAVING over a grouping column moved with it, the same way it did for min and max: the clause compares against the group key, so the literal is handled in the same representation. All three HAVING shapes measured against PostgreSQL were wrong before and are right now.

Still not fixed, and now measured precisely: SELECT DISTINCT on a timestamp collapses rows the same way and returns a truncated value. It takes a third route through the planner — a projection that drops the microseconds before the deduplication runs — so neither this fix nor the earlier ones reach it.

A join on a timestamp column returned rows that do not match

… a JOIN b ON a.t = b.t compared the stored dates, which hold whole milliseconds, so every timestamp inside the same millisecond joined to every other. Over four distinct times a self-join returned ten rows where PostgreSQL returns four — a wrong answer, not a lost digit. Joining now compares the microseconds too, and a joined timestamp comes back with the microseconds it was stored with rather than rounded to the millisecond.

Two other timestamp routes are fixed with it, both found in the same sweep. string_agg(t::text, …) rendered the rounded time, even though t::text on its own was already exact. And a timestamp rendered as text padded the fractional seconds to six digits — 00:00:00.123100 where PostgreSQL prints 00:00:00.1231 — which also made concat(t, '') and t::text disagree with each other.

Fixed

• A join on a timestamp column matches only equal times, and its result keeps microseconds.

• string_agg(t::text, …) keeps microseconds.

• A timestamp rendered as text prints PostgreSQL’s shortest form, and every route that renders one (::text, concat, ||) agrees.

ORDER BY on a timestamp is microsecond-exact

Rows whose timestamps differed only below the millisecond came back in storage order. Sorting four rows at .122000, .123100, .123500 and .123900 gave the first row correctly and then the remaining three in the order they were inserted, because the sort key was built from the stored millisecond value and the microseconds live in a hidden companion field that the key never consulted.

This is easy to under-rate as a display nit, so it is worth being precise about what it affected. LIMIT reads off the sorted list, so ORDER BY t LIMIT 2 returned the wrong rows. DISTINCT ON keeps the first row per group in the ORDER BY order, so it silently picked a different row than PostgreSQL does. Neither looks like a sorting bug from the outside — they look like wrong answers.

It was two code paths, not one. Beyond the plain-column sort key, a query ordering by an expression or a column ordinal — SELECT id, t FROM x ORDER BY 2 — runs through a separate route that evaluates against the source row, and that route never restored the microseconds at all. So that shape both sorted at millisecond granularity and returned truncated times. Both paths now read through one helper.

Verified against a live PostgreSQL 14 across plain, DESC, aliased, ordinal, DISTINCT ON and LIMIT shapes.

Not fixed, and now recorded with its measurement: min(), max() and the in-call array_agg(x ORDER BY t) form are computed inside the aggregation pipeline, which reads the stored date and never sees the companion. min(t) and max(t) therefore still answer a whole-millisecond time for a stored microsecond one.

sum() answered 0 for a group that contributed nothing

SELECT j.id, sum(k.v) FROM j LEFT JOIN k ON k.jid = j.id GROUP BY j.id answered 0 for a j row with no match, where PostgreSQL answers NULL — a common reporting query, silently wrong, and one a caller could not defend against, since coalesce(sum(...), -1) also saw the 0.

Fixed

• The NULL guard on sum counted contributions with a bare $ne: [value, null], which is true for a missing field. An unmatched outer-join row carries no key at all for the non-driving side, so the guard counted a contribution that never happened. It now collapses missing into null first, exactly as COUNT(col) does. This is why only the unmatched-row case was wrong while a group holding a genuinely NULL value was already right.

• HAVING sum(x) IS NULL could never be true — neither the single-table nor the join HAVING path applied the guard at all. Both now do.

An eighth SQL sweep: four silently wrong answers, and a whole type that answered NULL

263 statements run against PostgreSQL 14.13 through the same psycopg client on both sides — so client-side type mapping is identical and every difference is the server’s. The four worst findings all returned a plausible answer rather than an error.

Every navigation over a json value answered NULL. The json type keeps the client’s exact text — whitespace, key order and duplicate keys preserved, which is what separates it from jsonb — so a ::json value arrives as a str subclass. The -> / ->> / #> / #>> walker descends dict and list only, so it fell straight through to “not a container” and answered NULL for the entire type. '{"a":1}'::json -> 'a' was NULL where PostgreSQL says 1.

ORDER BY b.id sorted by a.id. Two joined tables routinely project same-named columns; the order term was resolved by its bare name against the output list, which finds the first of them. In a RIGHT JOIN it also misplaced the unmatched rows, because their a.id is NULL and NULLs sort to one end.

SELECT jsonb_each(x) returned no rows at all. The SELECT-list record-SRF expansion was written for _pg_expandarray and treats the argument as an array; jsonb_each’s argument is an object, so it expanded to zero elements and the statement answered an empty result.

ORDER BY 99 was accepted and ignored. Each planning path gates on 1 <= n <= len(select list) and, when that fails, leaves the literal alone — which sorts by a constant, i.e. not at all.

Fixed

• json (non-b) navigation. ->, ->>, #>, #>> and json_extract_path all descend a json value again instead of answering NULL.

• #> keeps jsonb; #>> returns text. The type inference named #> in its comment but tested only the JSONExtract classes — #> parses to JSONBExtract, which is not a subclass — so #> went out under oid 25 and '{"a":{"b":[1,2]}}'::jsonb #> '{a,b}' rendered the PostgreSQL array literal {1,2} instead of the jsonb [1, 2].

• ORDER BY <n> out of range raises 42P10 (including 0 and a negative ordinal, which parses as Neg(Literal) and never reached the range gate).

• A qualified ORDER BY term matches the select list by its full text before falling back to the bare column name, so ORDER BY b.id no longer sorts by a.id.

• JOIN ... ON a.id = b.id - 1. The fast-path “is this a simple equality?” detector raised instead of answering no, so any ON term that is not a bare column was a fatal 0A000 ON must compare columns — even though the general pipeline form lowers arithmetic perfectly well.

• SELECT jsonb_each(x) / jsonb_each_text(x) expand to one row per member, in the composite and the (...).key / (...).value field forms; the composite renders a jsonb field as JSON ((b,"x"), not (b,x)).

• An incomparable pair (ARRAY[1,2] > 1, '{"a":1}'::jsonb > 1) raises 42883 naming the operator, rather than leaking a Python TypeError to the client as XX000 internal error.

• initcap treats a digit as part of a word, as PostgreSQL does: initcap('a1b c') is A1b C, not A1B C.

• quote_ident quotes a keyword that is not UNRESERVED (quote_ident('select') → "select"), and format’s %I is now that same rule instead of always quoting — format('%I', 'tbl') is a bare tbl. A NULL to %I raises 22004.

• LIKE ... ESCAPE types as boolean. The ESCAPE clause wraps the predicate in a node that is not itself a boolean class, so adding it to a working LIKE flipped its column from oid 16 to oid 25 and sent a 't'.

• A timestamp literal’s fractional seconds, at any width. The support matrix disagreed with itself in both directions and neither half matched PostgreSQL: before Python 3.11 fromisoformat accepted only 3 or 6 fractional digits, so TIMESTAMP '2020-01-15 10:30:45.5' — a perfectly good literal — raised on 3.10 while parsing on 3.12; from 3.11 it accepts any width but truncates beyond six digits, where PostgreSQL rounds (.1234567 → .123457). Normalising the fraction before the parse makes every supported Python answer the same microseconds, and the same ones PostgreSQL does. Only the CI matrix could show this — a local 3.12 run sees neither half.

• An unresolvable function name answers 42883 function f(...) does not exist rather than 0A000 ... is not supported in this context, which claimed the call site was the problem when the name was unreachable everywhere. A set-returning function in scalar position keeps 0A000 and now says what the actual limit is.

Added

• SIMILAR TO / NOT SIMILAR TO, with ESCAPE. SQL’s regex flavour is LIKE’s wildcards plus | * + ? {} () []; every other character is literal, so 'abc' SIMILAR TO 'a.c' is false.

• Quantified comparisons under every operator. ANY / ALL worked only under = and <>; 1 < ALL(ARRAY[2,3]) was an outright error. The set form (x = ANY (SELECT ...)) works too, and the three-valued rules match PostgreSQL — including that an empty array settles the answer before the needle is looked at (NULL = ALL('{}') is true).

• DISTINCT inside array_agg / string_agg / jsonb_agg. PostgreSQL dedupes by sorting, so the result comes back ascending even with no ORDER BY written; an ORDER BY naming anything but the argument raises 42P10, as it does there.

• jsonb_extract_path, jsonb_extract_path_text, jsonb_path_query_first, array_to_json, trim_array. sqlglot folds only the json_ spelling of the extract-path pair into a dedicated node, which is why one of each pair worked and the other did not.

• extract: isoyear, julian, microseconds, milliseconds (the last two fold the seconds in, as PostgreSQL does). date_trunc: decade, century, millennium, milliseconds, microseconds — centuries and millennia start at year 1, so 2026 truncates to 2001.

• generate_series over date / timestamp bounds with an interval step. The bounds arrive as canonical text, so the temporal overload was rejecting its own arguments; date bounds type as timestamptz and timestamp bounds as timestamp, which is how PostgreSQL resolves the overloads.

An eleventh SQL sweep: a multidimensional array is one array, not nested ones

Sequences and identity columns came back strong (23 of 26 shapes already matching PostgreSQL 14.13). Arrays did not — 20 of 29, with the misses clustered on multidimensional arrays, which PostgreSQL does not treat as nested arrays at all: int[][] is ONE array with two dimensions, and every whole-array operation walks it flat.

array_to_string leaked Python syntax. Joining only the top level rendered each inner list through str(), so array_to_string(ARRAY[[1,2],[3,4]], ',') answered [1, 2],[3, 4] where PostgreSQL says 1,2,3,4.

unnest of a 2-D array crashed with a bare invalid literal for int(): '{1,2}' and no SQLSTATE — the inner lists went out as elements and the int4 output coercion died on them.

Every scalar subquery over a VALUES source failed with 42P01 relation "" does not exist. Not an array bug at all — IN (SELECT … FROM (VALUES …)), EXISTS, ARRAY(…) and the bare scalar form were all affected, because a VALUES-derived source is not a relation and the inner-table lookup found nothing.

Fixed

• array_to_string and unnest walk a multidimensional array flat, in row-major order. unnest had three separate copies of the same one-level logic — the exp.Unnest node, the Anonymous spelling, and the select-list expansion — which is why fixing one of them was not enough; they share a helper now.

• A VALUES source works inside any scalar subquery, with the row’s cells evaluated on reference so an expression in the VALUES list works too, and the alias’s column names (or PostgreSQL’s default column1, column2 …) both resolve.

• A scalar subquery and ARRAY(subquery) keep their element type. Everything came back as text before: (SELECT n FROM (VALUES(7)) t(n)) is int4, and ARRAY(SELECT n …) is int4[].

• pg_get_serial_sequence returns the schema-qualified sequence name instead of NULL. The column already recorded it (Column.sequence, which nextval and the information_schema view both read) — this just never looked, so ORM reflection saw every serial column as plain.

Added

• ARRAY(SELECT ...), the array-subquery constructor, which parses as an Array whose single element is the Select and so tried to evaluate a Select as a scalar.

A fifth sweep: a dropped column DEFAULT, and enums compared by spelling

28 of 36 shapes matched PostgreSQL 14.13 across arrays, enums, domains, ranges and DDL. Two of the misses were silently wrong rather than refused.

ALTER TABLE t ADD COLUMN c text DEFAULT 'z' dropped the DEFAULT on the floor. Existing rows kept NULL — PostgreSQL backfills — and, worse, a later insert that omitted the column got NULL too, because the default was never recorded in the catalog at all. NOT NULL DEFAULT 7 therefore left a NOT NULL column holding NULL.

An enum comparison answered by spelling. An enum’s order is its declared label order, so 'happy' > 'ok' is true for mood AS ENUM ('sad','ok','happy') and false as text — and WHERE m > 'ok' returned sad. Sorting already knew the declared order (enum_orders); comparison did not.

Fixed

• ADD COLUMN … DEFAULT records the default and backfills existing rows. A non-literal default is evaluated once, as PostgreSQL does.

• Range comparisons on an enum column in WHERE (and so in UPDATE / DELETE), both operand orders. An enum has a finite label set, so a range comparison is exactly a set membership — no ordinal needed at query time.

• array_positions(), and result types for array_fill() (an array of its value’s type, not text) and range_merge() over ::int4range casts — only the range constructor was recognised as a range operand.

• version() nested in an expression. It worked as a bare projection through the session-function path but reported function current_version() is not supported — sqlglot’s node name, not one the user wrote — the moment it was nested.

Still divergent

enum_range() / enum_first() / enum_last(), and an enum comparison in the SELECT list (SELECT m > 'ok') rather than a WHERE — both need the catalog inside the scalar evaluator.

A fourth sweep: two badly wrong answers, three wrong types

25 of 35 shapes matched PostgreSQL 14.13 across windows, aggregates, bytea and subqueries.

substring(b from 1 for 1) over a bytea answered the string 'b' — the first character of the Python repr b'\x01\x02' — where PostgreSQL answers the byte \x01.

every(n > 5) answered NULL. every is the standard-SQL spelling of bool_and, and the mapping kept the name but dropped an expression argument; bool_and(n > 5), the same aggregate, was right all along.

Fixed

• substring / substr over bytea slice bytes and report bytea.

• every(<expression>).

• A scalar subquery takes its projection’s type: (SELECT count(*) FROM t) is bigint, and a correlated aggregate takes its column’s type. It was text, so a driver was sent the string '3' under oid 25.

• A ::numeric(p, s) cast rounds to its declared scale, as the column path already did — 10::numeric(5,2) is 10.00, and a value that no longer fits is 22003.

• round() / floor() / ceil() report their argument’s numeric type; round(2.345::float8) claimed numeric for a float result.

• GROUPS window frames, whose offset counts peer groups rather than rows. They were not handled at all — they fell through to the RANGE branch and reported RANGE with a numeric offset requires a numeric ORDER BY key, an error about a clause the user had not written.

Still divergent

sqrt(numeric) returns float8 (PostgreSQL returns numeric at a scale its sqrt_var estimator picks), corr / covar_pop / regr_* are unsupported, min/max accept a bytea PostgreSQL rejects, and a scalar subquery under GROUP BY cannot be typed because the synthetic resolver cannot see the inner table.

A ninth SQL sweep: a named window was evaluated as OVER ()

Window functions were the richest surface probed here yet, and four of the findings were silent — a plausible number in every row, computed from a window that was not the one written.

A named window lost its whole definition. sqlglot keeps a WINDOW w AS (...) definition on the SELECT and leaves the reference as a bare alias with no partition, no order and no frame — which is exactly what every consumer downstream reads. So sum(v) OVER w with w AS (ORDER BY id) returned the whole-partition total on every row instead of a running one, and a PARTITION BY in the definition was dropped just as quietly.

EXCLUDE was parsed and then ignored. It rides on the frame spec as args["exclude"] and nothing read it, so EXCLUDE CURRENT ROW answered the unexcluded frame — a running sum that still counted the current row.

NULLS FIRST in a window ORDER BY was ignored. NULLs were placed by direction alone, which happens to reproduce PostgreSQL’s defaults (last for ASC, first for DESC) — so the flag looked right until somebody wrote it explicitly, and then every rank in the partition was wrong.

sum and avg were typed by rules the GROUP BY path had long since got right. A window sum(int4) declared int4 where PostgreSQL promotes to int8, and avg declared float8 and divided as a float where PostgreSQL answers numeric at select_div_scale’s scale.

Fixed

• Named windows (WINDOW w AS (...)) are resolved into their references before a planning path is chosen, so the rest of the engine only ever sees a fully specified window. Definitions may chain (w2 AS (w1 ORDER BY x)); a reference may add an ORDER BY and a frame but not override the definition’s ORDER BY (42P20), and an unknown name is 42704.

• EXCLUDE CURRENT ROW / GROUP / TIES / NO OTHERS, on aggregate and value windows alike. A frame is now an explicit index list, because EXCLUDE punches a hole in the middle that an [lo, hi] pair cannot express.

• NULLS FIRST / NULLS LAST in a window ORDER BY.

• Window sum / avg result types, via the same _sum_tag / _avg_tag helpers the GROUP BY path uses, and avg over an exact input now finishes in numeric arithmetic rather than float — avg over 10, 20, 20 is 16.6666666666666667, not 16.666666666666668.

• ntile types as int4; it is the one integer window function PostgreSQL does not make bigint.

• agg(...) FILTER (WHERE ...) OVER (...) no longer fails with 42803 naming an ordinary column. The FILTER node sits between the aggregate and its Window, and the “is this a window aggregate?” guard looked only one level up, so the aggregate was mistaken for a grouped one.

• A subquery in RETURNING — INSERT ... RETURNING id, (SELECT count(*) FROM t) — crashed with XX000 internal error: the RETURNING scalar context was built with catalog=None, so resolving the subquery’s table raised AttributeError.

• COALESCE, GREATEST and LEAST resolve a common type instead of taking the first argument’s. coalesce(1::int, 2.5::numeric) declared int4 and then coerced the numeric result with int('2.5') — a bare Python ValueError that reached the client with no SQLSTATE at all. The old code carried a comment claiming first-wins “is what PG’s common-type resolution amounts to for the shapes we can decide”; it is not, and a probe says so. Note the precedence is not arithmetic’s: int + real is double precision, but greatest(int, real) is real.

• concat, concat_ws and format’s %s render a boolean as t / f. They go through the type’s output function, not ::text — which spells it true — so concat(1, 2.5, true) answered 12.5true where PostgreSQL says 12.5t. Bool is the only type where the two spellings differ.

• format rejects too few arguments with 22023 instead of substituting an empty string, positional (%3$s) forms included.

• split_part with an empty delimiter returns the whole string as field 1; Python’s str.split("") raises, and the ValueError escaped as a confusing function split_part(unknown) does not exist. A zero field position is 22023, as it is there.

• A compound INTERVAL literal lost everything after its first two tokens. sqlglot parses INTERVAL '1 day 3:45:00' as Interval(this='1', unit=DAY) and discards the rest of the string — it round-trips as INTERVAL '1 DAY', so three hours and forty-five minutes were gone before any of this engine’s code ran, and INTERVAL '2 days ago' came back positive. It only truncates when the text starts <number> <unit>; a bare '3:45:00' and a many-worded '1 year 2 mons 3 days 04:05:06' both survive, which is why it hid. Compound literals are now rewritten to a cast before parsing, beside the other repairs to sqlglot’s parsing.

• INTERVAL '1-2' (the ISO year-month form) reached the wire as XX000; so did the full '1-2 3 4:05:06', whose bare 3 is the days field rather than a value awaiting a unit.

• Negating an interval COLUMN raised a bare decimal.ConversionSyntax with no SQLSTATE: - col fell through to a numeric default for any non-numeric operand, and the output coercion then fed the interval subdocument to Decimal. The literal form was always fine.

• time ± interval is a time, wrapping at midnight, not an interval — TIME '13:45' - INTERVAL '14 hours' came back as a 23:45 duration under the interval oid.

• A ::date cast compares equal to CURRENT_DATE. The cast yields the canonical text while CURRENT_DATE yields a datetime.date, so now()::date = CURRENT_DATE answered FALSE on a day when both plainly named the same one. Two casts, or two literals, were always fine.

Added

• string_agg and array_agg as window functions, including under FILTER, with the array typed from its element rather than declared numeric.

• LOCALTIME and LOCALTIMESTAMP, the tz-naive twins of CURRENT_TIME / CURRENT_TIMESTAMP; sqlglot gives them their own nodes and neither was handled, so both answered 42883 function localtime() does not exist.

• min_scale and trim_scale, beside the scale that was already there — scale is the digits a numeric carries, min_scale the smallest that keeps the value exactly, and trim_scale the value re-scaled to it.

• UPDATE ... SET (a, b) = (x, y), in all three row-constructor spellings ((a, b), a one-element (a), and ROW(...)), expanded to single-column assignments at parse time. An arity mismatch is 42601. A row subquery right-hand side is still unsupported and says so.

SQL: char(n) comparison, WITH ORDINALITY aliases, and greatest/least typing

A seventh differential sweep against PostgreSQL 14.13 — this one over char(n) semantics, INSERT … ON CONFLICT, GROUPING/ROLLUP/CUBE, CTEs, row locking and set operations — scored 20 of 24 and turned up four defects, one of them a silently wrong answer.

Fixed

• char(n) columns now compare blank-insensitively, as Postgres does. bpchar comparison strips trailing spaces from both operands, so a char(5) holding 'ab' matches = 'ab' and = 'ab   '. SecantusDB stores the value unpadded, which got the stored side right for free but left the literal side padded — so the second form quietly answered false where Postgres answers true, in WHERE, in a projection, and in IN. varchar is genuinely blank-sensitive and is deliberately left alone.

• UNNEST(…) WITH ORDINALITY AS t(v, i) now names its ordinality column i. sqlglot hoists an UNNEST’s last alias column into a separate offset slot rather than leaving it in the alias list, so the column fell back to the default name ordinality and SELECT i failed with 42703 column "i" does not exist.

• greatest() / least() now report their arguments’ type rather than text. greatest(NULL, 1) was sent as the string '1' under oid 25 where Postgres sends the integer 1.

Added

• num_nonnulls() and num_nulls().

A sixth sweep: age() borrowed from the wrong month

20 of 33 shapes matched PostgreSQL 14.13 across constraints, identity columns, time zones, GUCs and string functions. The important miss was silently wrong arithmetic.

age('2021-03-15', '2020-01-20') answered 1 year 1 mon 23 days where PostgreSQL answers 26 days. When the day difference goes negative, the borrow takes the length of the start date’s month — January’s 31 here — not the month before end (February’s 28), and not a flat 31. Eight probed cases discriminate all three readings, including age('2020-04-01','2020-01-15') (January’s 31, not April’s 30) and age('2020-03-01','2020-02-28') (February’s 29, not 31).

Fixed

• age()’s day borrow. Sixteen cases now match, and two existing tests that had recorded the old answer are corrected against the reference server.

• format('%1$s-%1$s-%2$s', 'a', 'b') — positional argument specifiers, which may repeat one. Unrecognised, the whole directive was copied through as literal text, so the format string came back unformatted.

• current_setting('nope', true) is NULL and current_setting('nope') is 42704 unrecognized configuration parameter. Both answered the empty string, which reads as a setting that exists and is blank.

• parse_ident(), unistr() and normalize().

• localtimestamp nested in an expression, and in the session’s time zone. It used the machine’s wall clock, a different instant whenever the two zones differ — with the default UTC session on a UTC+1 host, localtimestamp <= now() was FALSE.

Still divergent

AT TIME ZONE, extract(timezone_hour …), LIKE ALL/ANY(array), and to_ascii().

A tenth SQL sweep: a function that was legal at the top of a SELECT and nowhere else

DDL came back strong — 38 of 41 shapes already matched PostgreSQL 14.13, and catalog introspection 15 of 22 — so this is a short batch. Three findings.

A session function stopped being resolvable one level down. current_setting('x') worked; current_setting('x') ~ '…' answered 42883 function current_setting(text) does not exist. Those functions were only ever reached from the constant-SELECT planner, so any operand position, WHERE clause or wrapping call lost them — the same shape as a set-returning function that only works as a row source, but here with no reason for the restriction.

has_table_privilege ignored the owner. It consulted recorded GRANTs only, so a table the caller had just created and could plainly read reported FALSE. PostgreSQL’s owner holds every privilege implicitly — measured on 14.13 by creating a table, granting SELECT to another role, and asking as the creator, which answers true.

Fixed

• Session functions are legal wherever an expression is. current_setting, current_database, current_schema, current_query, version, pg_is_in_recovery and the inet_server_* / pg_postmaster_start_time pair now resolve in any position. Deliberately not on that list: anything with a side effect (set_config, the advisory locks, pg_terminate_backend), which keeps its existing explicit handling. Widening where they are reachable does not widen what they accept — an unknown setting is still 42704.

• has_table_privilege grants the owner everything implicitly, and honours a REVOKE that targets the owner (which materializes the ACL). This is the reporting function; the authz gate has its own path and already permitted the owner, which is exactly why the read worked while this denied it. The rule hands nothing to anyone else — a stranger still reports false.

• CREATE TABLE (id int, id int) is rejected with 42701 column "id" specified more than once, instead of creating a relation whose second id was unreachable. Names fold, so (id int, ID int) collides too.

• ALTER TABLE ... DROP COLUMN names the relation in its 42703, as PostgreSQL does: column "x" of relation "t" does not exist.

An existing test asserted that the two-argument has_table_privilege was false for the session user “(default ‘secantus’, no grant)”. That is not the rule, and PostgreSQL disagrees; the test now carries the measured value and the reason.

to_char(numeric, …) matched PostgreSQL on 63 of 300 shapes

A sweep of 30 templates × 10 values against PostgreSQL 14.13 found four rules the implementation did not have at all, and a fifth problem upstream of it.

• Overflow prints #. A value too wide for the digit slots fills every one of them — to_char(1234.5, '999') is ' ###', not ' 1235'. Printing the number anyway silently violated the template’s own declared width.

• The sign sits against the digits, immediately left of the first one, not in front of the padded field: to_char(-12, '999') is ' -12'.

• A 0 slot zero-fills everything to its right: '0999' over 12 is ' 0012'.

• An all-9 integer part renders blank when the value has none — to_char(0.5, '999.9') is '    .5'.

And the template never reached the numeric formatter intact: sqlglot’s postgres dialect part-converts it to strftime first, so MI999 arrived as %M999 and 9999D99 as 9999%u99 — the tokens the numeric formatter did not recognise were simply dropped, which is why D produced no decimal point at all.

All 300 shapes now match.

Fixed

• to_char(numeric, …): overflow # fill, sign placement (S / MI / PR, leading and trailing), 0 zero-fill, blank integer parts, G / D / L locale tokens, $ in front of the sign, FM (which drops trailing fractional zeros but keeps the point), and RN Roman numerals.

• IN (subquery) and NOT IN (subquery) in UPDATE and DELETE. The identical predicate has always worked in a SELECT — the DML planners simply never published the subquery context the SELECT planner does.

Corrected tests

Six to_char expectations recorded this engine’s own output rather than PostgreSQL’s, and were re-probed against 14.13: FM drops trailing fractional zeros (FM$9,999.99 over 1234.5 is $1,234.5), a value too wide for its slots is ###.##, and L is the locale currency symbol — empty here, not $.

Still divergent

EXISTS (subquery) in UPDATE / DELETE, UPDATE … FROM (VALUES …), and string_agg / array_agg with DISTINCT.

A twelfth SQL sweep: LIMIT 0 returned every row

Transactions came back perfect — 40 of 40: savepoints, the aborted-transaction state, isolation levels, READ ONLY, and transactional DDL rollback all match PostgreSQL 14.13. Two clusters did not.

LIMIT 0 returned the whole table. A sentinel collision: the planner used 0 to mean “no LIMIT”, and every consumer tested the value for truthiness, so a real LIMIT 0 was indistinguishable from its own absence. It matters because LIMIT 0 is how a client asks for a result’s column metadata without rows — ORMs and BI tools do it constantly.

The fix runs deeper than the sentinel. The storage layer reads limit=0 as “no limit” too (Mongo’s convention), so a genuine LIMIT 0 must never reach it; and Mongo’s $limit stage rejects zero outright (54000 the limit must be positive), so the pipeline emits a match-nothing stage instead.

Rows containing NULL ignored SQL’s three-valued rules. (1,NULL) = (1,NULL) answered true where PostgreSQL says NULL — Python’s == treats two Nones as equal — (NULL,NULL) IS NULL answered false where it says true, and (1,2) < (1,NULL) raised 42883 naming integer[], the record having been compared as an array.

NATURAL JOIN was a cross join. sqlglot records it as args["method"] with no on and no using, and nothing in join planning read it — so the join lost its condition entirely and returned every pair. This is precisely the bug desugar_join_using was written for one step later in the same chain, which is now where the fix lives: NATURAL resolves to the common columns, and that function turns them into the ON.

Fixed

• LIMIT 0 returns no rows, on every path: the plain scan, the aggregation pipeline, a derived table, and a FROM-less SELECT.

• LIMIT NULL and LIMIT ALL mean no limit, as they do in PostgreSQL; a negative limit is 2201W; and FETCH FIRST ROW ONLY (the standard’s optional count, defaulting to one) works instead of returning everything.

• A FROM-less SELECT honours LIMIT / OFFSET at all — SELECT 1 OFFSET 1 is empty.

• Row comparison follows the three-valued rule: fields are compared left to right, the first pair that decides wins, and a NULL pair reached before then makes the whole comparison NULL. (1,NULL) = (2,3) is still false, because the first pair decided it.

• NATURAL JOIN / NATURAL LEFT JOIN join on the relations’ common columns instead of returning every pair (NATURAL LEFT JOIN previously errored outright with “LEFT JOIN requires an ON clause”). A NATURAL join with no common column is a cross join in PostgreSQL too, so that case needs no special handling.

• An aggregate’s argument may be a function call. sum(abs(n)), string_agg(coalesce(s,'-'), ',' ORDER BY id) and array_agg(upper(s)) all answered 0A000 unsupported aggregate argument — the lowerer handled columns, literals, comparisons, CASE and arithmetic, but no function calls at all (14 of 16 probed shapes failed on it). upper, lower, abs, floor, ceil, sqrt, length, coalesce and || now lower to their Mongo operators, each wrapped in a NULL guard because PostgreSQL’s scalar functions are strict and Mongo’s are not — $toUpper maps null to the empty string and $strLenCP rejects it outright.

  round is deliberately absent from that table: Mongo’s $round rounds half-to-even where PostgreSQL rounds half-away-from-zero, so lowering it would make sum(round(x)) over 1.5 and 2.5 answer 4 instead of 5 — a silent wrong answer in place of an honest “unsupported argument”.

• row IS NULL / row IS NOT NULL are each true only when every field qualifies — so a row with one NULL is false for both. They are not negations of each other, and treating them as such made (1,NULL) IS NOT NULL true.

SQL: remove the superseded to_char strftime machinery, correct the docs

Follow-up to the datetime template engine. Housekeeping found by reconciling the documentation against what the code now does.

Changed

• Removed 98 lines of dead to_char machinery that the template parser superseded (_repair_time_format, _render_word_token, _PG_WORD_TOKENS, _WORD_TIME_TOKEN_RE, _WORD_TIME_DIRECTIVES, _WORD_TIME_PAD). Nothing referenced it, and its comment still asserted that IYY / IY / I / IDDD “have no strftime directive and are not handled” — a limitation that no longer exists, pointing at a backlog entry that is now resolved.

Fixed (documentation)

• Three of the four worked to_char numeric examples in docs/sql.md showed pre-FM-fix output: FM999,999.99 on 1234.5 is 1,234.5, not 1,234.50. All examples on the page are now verified against PostgreSQL 14.13.

• docs/sql.md claimed RN (Roman numerals) was unimplemented. It is implemented and matches Postgres.

• Added a to_char / to_date / to_timestamp datetime section — the new template engine had no user documentation.

SQL: a real to_char / to_date / to_timestamp template engine

An eighth differential sweep against PostgreSQL 14.13, over datetime formatting, scored 55 of 122. The datetime half of to_char was built by converting the template through sqlglot’s Postgres TIME_MAPPING and handing the result to strftime — a mapping that knows a handful of tokens and matches single letters anywhere they appear. Two sweeps now score 122/122 and 190/192.

Fixed

• Tokens that rendered as their own spelling now render: Q, W, WW, CC, J, MS, US, SSSS, HH, RM, Y,YYY, YYY, Y, IYY, IY, I, IDDD, FF1–FF6, TZH, TZM, OF, quoted "literals", and the TM prefix.

• Tokens matched inside other tokens. The D in AD rendered the weekday, so to_char(ts, 'AD') answered 'A3' and 'A.D.' answered 'A.3.'.

• Case-sensitive token matching, which is how Postgres works and is observable: Ddth is D + d + th ('44th'), not DD + th ('02nd'). day/dy/am/bc now render lower-case.

• FM prefixes one token, rather than latching on: FMHH12:MI is '2:07'.

• D is 1=Sunday..7=Saturday, not the ISO weekday — it was off by one for every day of the week.

• th gives the right ordinal ('02nd', '01st', '03rd'), not always 'th'.

• to_date / to_timestamp parse word templates. Mon, Month, Dy, AM, MS, IYYY IW, J and DDD templates raised 22007 invalid input syntax because the same lossy mapping was used to build a strptime directive.

• to_timestamp returns a timestamptz, not a naive timestamp — it rendered without the +00 offset Postgres sends.

Known limitation

A year-less template defaults to 1 BC on Postgres. Python’s datetime has no era and a minimum year of 1 AD, so SecantusDB answers 1 AD. Recorded in tasks/backlog.md; it is the only divergence left in the 314-case sweep.

UPDATE … SET col = DEFAULT

Setting a column back to its default reported column "default" does not exist. The DEFAULT keyword was being read as the name of a column.

A column with no default becomes NULL, as in PostgreSQL, and a quoted "default" still means a column called default.

Fixed

• UPDATE … SET col = DEFAULT sets the column to its default, for a literal default, an expression default, or NULL where there is none. A NOT NULL column with no default reports a not-null violation, as PostgreSQL does.

• The same mis-reading sat under the guard that decides whether a generated column may be updated, so that check was treating every SET gen = DEFAULT as a non-DEFAULT value.

Known limitation

SET serial_col = DEFAULT is refused rather than guessed at: a serial’s default draws from its sequence, which the statement planner cannot do.

A table alias can stand for the whole row

Fixed

• row_to_json(t) FROM (SELECT ...) t — one of the commonest ways to get a row out as JSON — answered 42703 column "t" does not exist. So did to_json(r), row_to_json(<table>), (<table>)::text and SELECT t FROM t. A table or sub-select alias now stands for the whole row, as it does in PostgreSQL.

A real column of the same name still wins, which is what PostgreSQL does.

One gap remains: SELECT t FROM t reports the generic RECORD oid where PostgreSQL reports the table’s own rowtype oid. The field values are correct; minting per-table rowtype oids is a catalog feature, recorded in tasks/backlog.md.

cume_dist() and percent_rank()

Both window functions were unavailable. Every other window function already worked, so these two were the gap.

Both are peer-aware: rows that tie under the ORDER BY share a value, so cume_dist() counts the whole tied group rather than the row’s own position.

Added

• cume_dist() and percent_rank(), including partitions, tied rows, and a single-row partition.

ORDER BY works over unnest

Sorting the rows produced by unnest didn’t. SELECT unnest(ARRAY[9,8,7]) AS u FROM src ORDER BY 1 returned the elements in array order, and ordering by the column’s own name — ORDER BY u, the form most queries use — failed outright with “feature not supported”.

Both came from the same thing: the sort key was computed once per source row, before the function expanded it into many. Every expanded row therefore carried an identical key, and sorting them left the original order untouched. Keys for a set-returning column are now taken from the expanded row.

DISTINCT ON is unaffected: its key is deliberately row-level and still computed before expansion, which is what PostgreSQL does.

The equivalent field-selection form, (information_schema._pg_expandarray(arr)).x, plans differently and is not covered by this change.

Fixed

• ORDER BY over unnest sorts the expanded rows, by output ordinal or by column name, ascending or descending.

$limit and $skip argument errors match MongoDB exactly

A malformed $limit or $skip reported the right error code with the wrong text, and in a few cases the wrong outcome entirely. MongoDB echoes the value it rejected in shell form — "x", true, [ 1, "a" ], { a: 1 } — where the Python server printed a Python representation, and the Rust server printed a Rust one.

Probing the two servers against a real MongoDB across the whole value space turned up more than the wording. $skip: 1.5 and $skip: -1 came back as a generic error from the Rust server rather than the specific one; so did $limit: 0. A decimal argument was rejected by both servers where MongoDB accepts it — $skip: Decimal128("2") is a valid skip of one document — and a fractional decimal has its own message, distinct from a fractional double’s.

All 44 shapes now agree across MongoDB, the Python server and the Rust server, including ObjectId and date arguments.

$stdDevPop / $stdDevSamp no longer crash on non-numeric input

A group containing a value that wasn’t a number — a string, an array, a document — returned an internal server error. The accumulator added every value it was handed, so Python’s own arithmetic raised (unsupported operand type(s) for +=: 'float' and 'str') and the failure escaped as a generic “internal server error”. MongoDB simply ignores those values and computes the deviation over the numbers that are there.

A quieter problem sat next to it. When a group held no numeric value at all, both servers omitted the output field entirely; MongoDB always emits it, with null. Code reading doc["s"] got a KeyError where a real server hands back None. Booleans were also being counted as 0 and 1, which MongoDB does not do — a group of booleans is null, not zero.

Fixed

• $stdDevPop / $stdDevSamp skip non-numeric values (string, array, document, boolean, null) instead of failing the aggregation, matching MongoDB’s numeric domain of int / long / double / decimal.

• The output field is always present, holding null when the group contained no numeric value, rather than being omitted.

• Decimal input answers a double, as MongoDB does.

Timestamp comparisons are microsecond-exact

A timestamp or timestamptz holding sub-millisecond precision could not be found by a query for it. Comparisons looked only at the stored millisecond, not the microsecond remainder kept alongside it, and the results were wrong in both directions: a row storing 12:00:00.123456 did not match WHERE t = '12:00:00.123456' — an equality on its own stored value — while it did match WHERE t = '12:00:00.123', a value it is not equal to. Range comparisons inside the same millisecond were wrong the same way.

Reads were always precise, so a value could be inserted, selected back correctly, and still be unfindable by a predicate on the value just returned.

Comparisons now consider the remainder: the millisecond is compared first and the remainder only within it. Every shape — =, <>, <, <=, >, >= — was checked against a live PostgreSQL across 42 predicate/literal combinations with no divergence, and that comparison now runs as a test wherever a PostgreSQL server is reachable.

ORDER BY within a single millisecond is still millisecond-granular; sorting needs the remainder as a tiebreaker and is not part of this change.

Fixed

• WHERE comparisons on timestamp / timestamptz account for sub-millisecond precision, so a row matches an equality on its own stored value and no longer matches a truncated one it differs from.

More of the test suite starts from a cloned database

An earlier change had tests copy a prebuilt WiredTiger database instead of building a fresh one per test, but it only covered the files that named their storage location in one particular way. The most common remaining form was the same thing with a subdirectory, so this extends the same treatment to 54 more files — cutting their combined runtime from 350 to 253 seconds.

Left alone deliberately: the backup, restore and point-in-time-recovery tests. Those stand up several databases with distinct roles, and a restore target in particular often needs to start empty, so handing them a pre-populated copy would change what they actually prove.

Changed

• 54 further test files take the cloned-home fixture instead of creating a WiredTiger database per test.

Tests start from a cloned WiredTiger home instead of building one each time

The test suite spent most of its time waiting on WiredTiger rather than running Python. Measuring the per-test fixture floor showed that of the ~281 ms it cost to stand a server up, ~234 ms was inside WiredTiger’s C library — and ~137 ms of that was WiredTiger creating the same dozen empty tables over and over, once per test, at roughly 9.7 ms per table. That work is identical every time: every test begins from the same empty schema.

So each worker now builds one pristine database home at the start of a session and copies it per test, rather than asking WiredTiger to construct a new one from scratch. Where the filesystem supports copy-on-write the copy is nearly free and uses less disk than the old approach did. Across the 22 test files converted so far this cut their runtime from 260 s to 195 s, and the equivalence a change like this depends on — that a copied database behaves exactly like a freshly built one — is pinned by tests that compare the two directly rather than taking it on faith.

Added

• tests/wt_template.py (build_template / clone_template) and the session-scoped _wt_template + per-test wt_home fixtures in tests/conftest.py.

• tests/test_wt_template.py, pinning cloned-vs-created equivalence, clone isolation, and durable close-and-reopen.

• tasks/rust-test-harness-investigation.md, recording the measurements behind this (and why reimplementing the harness in Rust was rejected).

Changed

• 22 test files now take the wt_home fixture instead of creating a WiredTiger home in tmp_path.

A tz-aware datetime bound as a timestamptz parameter now round-trips

Binding a Python timezone-aware datetime as a timestamptz parameter on the Rust PostgreSQL server (secantusd-pg) and comparing it to the same instant written as a literal — '<expr>'::timestamptz = %s, the shape psycopg’s own test_dump_datetimetz asserts — returned false. The binary wire form of a timestamptz parameter hands the server an absolute instant (i64 microseconds since 2000-01-01 UTC), but the decoder rendered it to session-zone TEXT and shipped THAT string as the value. The zone offset was then dropped the moment anything re-coerced the string as a bare timestamp, so the parameter landed the session offset away from the literal it was meant to equal and compared unequal. Both %b (binary) and psycopg’s default %s (which sends a datetime in binary) were wrong across the corpus — year 0001 through 9999, sub-second fractions, and seconds-carrying offsets; %t (forced text) was already correct.

The binary decoder now stores the instant directly, on the same carrier a ::timestamptz literal produces (a BSON date, or a sub-millisecond composite), so the binary and text paths share one representation. A redundant timestamptz → timestamptz cast ($1::timestamptz over a parameter already declared timestamptz) is now a no-op instead of re-parsing the instant’s offset-less wall clock and applying the session zone a second time — the timestamp → timestamptz cast still applies the zone, since its source type is timestamp, not timestamptz. The psycopg gauge gains +33 passing tests (3557 → 3590) with no regressions; test_dump_datetimetz goes from 30 failing to fully green.

Fixed

• secantus-pgserver decode_parameter: a binary timestamptz parameter (oid 1184) is decoded to the instant carrier, not session-rendered text; the extreme i64 infinities are kept as infinity / -infinity text.

• secantus-pgplan const_value: a timestamptz → timestamptz cast over a stored instant is a no-op, so it no longer double-applies the session zone.

• secantus-pgplan: new timestamptz_value_from_micros builds the stored instant carrier, shared by the binary decoder.

A network timeout has a direction, and the TLS tests had it backwards

Two tests failed intermittently on the Windows CI runner and nowhere else, three weeks apart, in different suites: a pymongo client pinging a TLS-enabled Rust server, and a raw socket completing a PostgreSQL TLS handshake. Both were recorded as flakes and passed on rerun, which is how they survived — and the same entry was filed in the backlog twice, once for each sighting.

Neither was a race against the listener. The server binds and listens before it hands back an address, so a connection lands in the accept backlog whether or not the accept thread has been scheduled; there is no readiness signal to wait for, because the readiness signal is the handshake itself. What both tests shared was a five-second budget that had to cover thread scheduling, a protocol round trip and an RSA-2048 handshake on the slowest machine in the matrix while the rest of the suite ran in parallel.

The fix is a distinction rather than a bigger number. A budget on a path expected to succeed should be generous, because it costs nothing when the test passes and buys only failures on slow machines; a budget on a path expected to be refused must stay short, because the test waits out the whole of it. Those are now two named constants, applied across every suite that negotiates TLS — one of which had already arrived at the same split on its own.

Added

• tests/net_timeouts.py: CONNECT_TIMEOUT_S and SERVER_SELECTION_TIMEOUT_MS for paths expected to connect, and REJECTED_SELECTION_TIMEOUT_MS for paths expected to be refused.

Fixed

• The intermittent Windows failures of test_tls_against_rust_server and test_pgserver_auth.py::test_tls_request_accepted_and_query_over_tls.

• Budgets in test_tls.py, test_x509_auth.py, test_pgserver_auth.py, test_pgserver_pg8000.py and test_rust_server_smoke.py now come from the shared module instead of five sets of literals.

$toDate parses the dates MongoDB parses

{$toDate: "12/31/2020"} is an ordinary call. Both servers answered 241 ConversionFailure, because both implemented a small ISO-8601 subset while MongoDB runs timelib — a parser with a much larger format table. 15 of 19 measured shapes diverged.

Three of its rules are worth knowing, because none is guessable:

• The slash form is US-first by rule, not by ambiguity-resolution. 31/12/2020 is refused outright, so MM/DD/YYYY wins and day-first is not a fallback.

• A trailing letter is a military timezone, not the ISO separator. "2020-01-01T" is 07:00:00, because T is UTC−7. This is deterministic, not host-local — a TZ=UTC server answers the same. J is the one letter timelib rejects.

• An out-of-range component is a parse failure, not a rollover. 13/01/2020 and 12/32/2020 are both refused.

Added

Both servers now accept, and agree with MongoDB on:

form	example
US slash, padded or not, optional time	12/31/2020, 1/2/2020, 12/31/2020 10:30
year-first slash	2020/12/31
non-padded ISO	2020-1-1, 2020-1-1 10:30
month names, either order	Dec 31 2020, 31 December 2020, Dec 31, 2020
Unix seconds	@1577836800, @-1, @1577836800.5
compact	20200101, 20200101T120000
ISO week date	2020-W01-1
hour with no minutes	2020-01-01T00
military timezone suffix	2020-01-01T, …A, …Z
surrounding whitespace	"  2020-01-01", "2020-01-01 "

Measured against mongod 8.2.11 over 45 shapes — including the refusals, which matter as much as the acceptances: a parser that takes too much is as wrong as one that takes too little. 0 divergent on both servers.

Still not reproduced

MongoDB’s per-position diagnostic for a string its scanner got partway through ('abc' names the offending character and where it stopped) needs timelib’s own lexer and timezone-abbreviation tables. Both servers give the same error code and a general message rather than inventing a position.

The Rust server renders $toLower / $toUpper of a Timestamp

mongod puts a Timestamp through a legacy asctime-like path rather than the $dateToString format language, and renders it in the server process’s local timezone. The Python server has always matched that; the Rust server answered 16007 can't convert from BSON type timestamp to String instead — a refusal where mongod returns a string, and one with no Python behind it on the standalone server.

Fixed

• $toLower / $toUpper of a Timestamp now render on the Rust server, as %b %e %H:%M:%S:<increment> — the day space-padded (jul  3), the increment unpadded (:0, :12), the whole string then ASCII-cased.

• The rendering is DST-correct, resolved against a real timezone database at the instant rather than a fixed offset: re-probed against mongod 8.2.11 across three zones, America/New_York is 5h behind in November and 4h in July. Verified end-to-end over the wire on secantusd-rs under both TZ=UTC and TZ=America/New_York.

top reports real per-namespace operation counters

top answered with mongod’s shape but every {time, count} was a hard zero, so mongotop rendered an idle server no matter how much load it was under. Both servers now instrument per-namespace operation timing and report it.

The section mapping was probed against a real mongod rather than inferred, which was worth doing: the obvious mapping is wrong in four places. aggregate, count, distinct and findAndModify all land in commands, not in queries/update — mongod’s queries section is essentially just find. Counts are per command rather than per document, so a 50-document insert bumps the count by one. And a successful drop resets a namespace’s counters instead of carrying its history forward.

Added

• top’s total / readLock / writeLock and per-operation sections now carry real microsecond times and operation counts, per namespace, on both the Python and Rust servers. Verified against mongod 8.3.4 over a mixed workload: 8 of the 9 sections match exactly.

• The Python server reuses the profiler’s existing clock read rather than adding a second one to the dispatch path.

Fixed

• A successful drop now resets that namespace’s counters, matching mongod.

• Commands that name no collection (ping, hello, serverStatus, listCollections) are no longer attributed to a namespace; explain is attributed to the namespace of the command it explains.

top reports real timings on Windows

The top command measured how long each operation took with a clock whose resolution on Windows before Python 3.11 is about 15.6 milliseconds. Anything faster than that measured as zero elapsed time, so the reported per-namespace timings were almost always 0 on that platform — the counts were right, the times were not.

Timing now uses the high-resolution performance counter, which is the correct clock for measuring an interval and is precise on every supported platform.

Fixed

• top reports non-zero operation times on Windows / Python 3.10, where the previous clock’s granularity rounded almost every measurement to zero.

CI broke on a crate nobody here depends on, because two lockfiles were ignored

tinyvec 1.13.0 shipped a library that does not compile — TinyVec::Heap(vec![ ...]) with vec! not in scope — and took the whole of CI with it on a commit that changed no dependency.

test.yml’s rust-storage job runs cargo fmt / clippy / test in each of six WT-linked crate directories, which makes every one of them its own dependency resolution root. Four of the six commit a Cargo.lock and were unaffected — secantusdb among them, which pulls tinyvec and pins 1.11.0. The two whose locks were gitignored re-resolved to 1.13.0 and failed, and so did the storage-engine wheel build on both Linux and Windows.

The exclusion had a comment justifying it: these crates are “built only by CI’s CMake path, so a stray per-crate target/ + Cargo.lock should never be committed”. The first half is not true — the workflow builds them directly — and the second conflated a build artifact with a lockfile. The target/ directories stay ignored; the two locks are now tracked, which makes the set of six consistent.

Nothing about this was specific to one branch: main would have failed on its next run.

Fixed

• .gitignore: crates/secantus-storage-adapter/Cargo.lock and crates/secantus-server-py/Cargo.lock are no longer ignored, with the reason recorded where the old claim used to be.

• Both lockfiles committed, pinning tinyvec 1.12.0.

An undefined $$variable answers 17276 on every surface, not just aggregation

The parse-time check for an undefined $$variable reached the aggregation pipeline. Every other surface that takes a filter or an update still fell through to the storage layer’s generic BadValue (2) — query uses a construct the Rust server does not support — on the Rust server, and the Python server failed whole write batches that mongod fails one statement of.

Measured on mongod 8.2.11: the Rust server was wrong on all 8 surfaces, the Python server on 4.

Fixed

• find, count, distinct and findAndModify now answer 17276 for an undefined variable inside a filter’s $expr, instead of the generic unsupported-construct error.

• update and delete report it per STATEMENT, in writeErrors with code 17276 — not as a command error. This matters beyond the code: mongod applies the earlier statements in the batch and fails only the offending one (n: 1 with the error at index: 1), where the Python server failed the whole batch and the Rust server reported code 2.

• A pipeline-form update carries the stage wrapper. {"u": [{"$set": {"b": "$$NOPE"}}]} is Invalid $set :: caused by :: Use of undefined variable: NOPE; a filter never takes a wrapper.

How it is checked

A filter is query language, not an expression: {s: "$$NOPE"} matches the literal string and must not be flagged. Only $expr holds an expression, and only $and / $or / $nor nest further filters — so the new filter walker descends into exactly those and leaves everything else alone, the same conservative rule the pipeline walker follows. A false positive would reject a valid query, which is worse than the wrong code being fixed.

Command-level let binds, and is threaded through on every surface.

14 cases added to tests/test_mongod_differential.py, four of them false-positive guards (a literal "$$NOPE" in a filter, a bound let, and two ordinary queries), plus the per-statement case that pins n: 1.

An undefined $$variable is a parse error, and an empty collection proves it

{"$project": {"x": "$$NOPE"}} is rejected by mongod before a single document is read — it fails the same way on a collection that is empty, and on one that does not exist. Neither server did that, because both only discovered the problem while evaluating the expression against a document. With no documents there was nothing to evaluate, so both answered ok: 1 and an empty cursor.

Fixed

• An empty or non-existent collection now reports the undefined variable (17276) instead of silently succeeding — on both servers.

• The Rust server answers 17276 at all, where it previously gave a generic BadValue (2) aggregation pipeline uses a stage or operator not supported by the Rust server for every one of the seven stages probed. The Python server had the code but, until the previous change, the wrong wrapper.

• The wrapper is per stage, as mongod’s is: Invalid $<stage> :: caused by :: inside $project / $addFields / $set, and a bare message everywhere else ($group, $redact, $replaceRoot, $replaceWith, $match’s $expr, $sortByCount, $bucket).

How it is checked, and why conservatively

A static walk of the pipeline, run once before it executes — the only design that can report a parse-time error, since the engine never evaluates anything when there are no documents.

It reports only from positions known to hold expressions, and ignores any stage it does not recognise. A false negative leaves the previous behaviour; a false positive would reject a valid pipeline, which is far worse than the wrong code it replaces. The rules were probed rather than assumed:

• a $match filter is query language, so {"$match": {"s": "$$NOPE"}} matches the literal string and must not be flagged — only its $expr holds an expression;

• $literal’s argument is data, not an expression;

• $let bindings are evaluated in the outer scope (they cannot see each other) and do not escape the in;

• $map / $filter bind as, defaulting to this; $reduce binds this and value;

• $lookup’s let binds only inside that stage’s own sub-pipeline — naming it in a later stage is undefined;

• $redact binds $$KEEP / $$PRUNE / $$DESCEND for its own expression;

• $$CLUSTER_TIME, $$SEARCH_META and $$JS_SCOPE are defined variables that answer their own errors (10071200 / 6347902 / 51144), so they are left to those paths rather than reported as undefined.

Testing

29 cases added to tests/test_mongod_differential.py — 15 that must error and 14 false-positive guards that must not — plus 6 Rust unit tests. The full suite (9405 tests, many of them valid $let / $map / $filter / $reduce pipelines) is itself the broadest false-positive check, and stayed green.

An unknown expression operator gets mongod’s code, name and envelope

mongod does not answer this one way, and the discriminator is position: the top-level value of a $project field is parsed by the projection parser, which has its own code and wording, while anywhere deeper the generic expression parser answers. The same $project therefore gives two different codes depending on how deep the unknown operator sits.

Fixed

• {$project: {n: {$nosuch: 1}}} now answers 31325 Invalid $project :: caused by :: Unknown expression $nosuch — mongod’s code and its wording, with the operator unquoted — instead of 168. Nested deeper, and in $addFields / $set, it still answers 168 ... Unrecognized expression '$nosuch', which is also what mongod does. $count / $topN / $bottomN follow the same rule.

• codeName for code 168 is now InvalidPipelineOperator rather than the generic Location168. This was wrong for every unknown-expression error.

• The message envelope is now mongod’s: Invalid $addFields :: caused by :: where the stage supplies one, and no envelope at all in $group / $replaceWith / $expr. An unknown operator was previously found by the constant folder, which stamped Failed to optimize pipeline :: caused by :: on all of them.

Swept by tools/probes/unknown_expression_errors.py (13 shapes, Python 0 divergent, down from 13); gated by tests/test_mongod_differential.py -k unknownexpr.

unnest over a non-integer array no longer fails the query

SELECT unnest(ARRAY['a', 'b']) raised ValueError: invalid literal for int() with base 10: 'a' in the client. The server described the output column as int4 regardless of what the array actually held, then sent the text a — so the driver tried to decode a string as an integer and the query died before the application saw a row. Text, numeric and boolean arrays were all affected; integer arrays worked, and only because the hardcoded guess happened to be right.

The column is now described with the array’s element type. Subscript-producing functions (generate_subscripts, and _pg_expandarray’s .n field) stay integers, since a position is an integer whatever the array holds.

Checked against a live PostgreSQL for text, numeric, boolean and integer arrays.

Fixed

• unnest(...) declares the array’s element type instead of always int4, so a non-integer array can be unnested at all.

An update path may not be empty

{$set: {"": 1}} used to succeed on both servers and store {"": 1} — a document mongod cannot produce, and one the very query that created it then fails to match. The rule applied uniformly to ten operators: $set, $unset, $inc, $mul, $min, $max, $push, $addToSet, $pop and $bit all either wrote an empty field name or answered the wrong code, twenty shapes in all. mongod rejects every one of them with 56, and it distinguishes a wholly empty path (An empty update path is not valid.) from one with an empty component (The update path 'a.' contains an empty field name, which is not allowed.).

Getting the code right also meant getting the ORDER right. mongod validates an update spec in a single walk in document order — the operator’s name, then each of its paths for emptiness, then for a conflict with a path claimed earlier — and reports the first offender it meets. Both servers had run those checks as separate passes, so an unknown modifier anywhere in the spec preempted an empty path or a conflict that mongod reports first. All three checks now share one walk on both servers, which is also how the Rust command layer stopped keeping its own private copy of the modifier list.

Two smaller $each fixes ride along, both measured the same day: $addToSet with a non-array $each answers 14 rather than the 2 it had borrowed from its $push sibling, and on the Rust server a non-array $each under $push now reports mongod’s message instead of claiming the server cannot do $push.

Fixed

• secantus.update / secantus-core: an empty update path, or a path with an empty component, is rejected with mongod’s code 56 and its two messages, across every operator and both ends of a $rename. Replacement-style updates are deliberately exempt — mongod really does store an empty field name for replace_one({_id: 1}, {"": 1}).

• secantus.update / secantus-core: the unknown-modifier, empty-path and path-conflict checks share one document-order walk, so the first offender wins, as mongod’s does. These are parse errors: they are reported even when the filter matches nothing, and they come back bare.

• secantus.update / secantus-core: $addToSet with a non-array $each answers TypeMismatch (14), not BadValue (2).

• secantus-core: $push with a non-array $each reports The argument to $each in $push must be an array but it was of type: <type> instead of deferring, which on the standalone Rust server surfaced as “query uses a construct the Rust server does not support”.

Update errors that depend on the stored document match mongod’s

Errors an update raises against a particular stored document — $inc on a non-numeric field, an array operator on a non-array field — used our own codes and wording for $push and $addToSet. Worst of the set: $pop on a non-array was a silent no-op, reporting n: 1 with no write error for an update mongod refuses.

Found by differential-probing the update operator family against a real mongod.

One note on message shape: mongod 8.3 wraps these in Plan executor error during update :: caused by :: and 6.0 does not, while the codes and bodies are identical either way. SecantusDB advertises 7.0, and the repo’s live differential gate runs whatever mongod is on PATH, so the bare body is what ships. The classification is kept in the code as a single switch point.

Fixed

• $push on a non-array returns code 2 with mongod’s message (The field 'a' must be an array but is of type int in document {_id: 1}), replacing our code 9 and our own wording. $addToSet likewise.

• $pop on a present non-array now errors with mongod’s code 14 and message instead of silently succeeding. A missing field or an empty array remain no-ops, as on mongod.

• The Rust engine had the same $pop gap — a non-array fell through its if let Some(Bson::Array(..)) — and now defers so the exact error is raised.

• $inc / $mul / $pull were already correct on codes and bodies and are unchanged.

Overlapping update operator paths are rejected, as mongod does

An update whose operators target the same path — or where one path is a prefix of another — was applied anyway. {$set: {a: 2}, $inc: {a: 1}} produced {a: 3}; real mongod refuses it outright, because $set replaces the very subtree $inc wants to walk into. We accepted 8 of the 12 overlapping shapes mongod rejects, returning documents mongod would never produce, with no error to notice.

Found by differential-probing the update operator family against a real mongod rather than by any failing test.

Fixed

• An update whose operators touch equal or prefix-overlapping paths returns mongod’s ConflictingUpdateOperators (code 40) with its exact message, Updating the path 'X' would create a conflict at 'Y'. Verified byte-identical across all six message shapes against mongod 8.3.4, on both servers.

• Sibling and disjoint paths are unaffected — {$set: {"a.b": 2}, $inc: {"a.c": 1}} still applies, as do sibling array indexes.

• The check splits on dots rather than comparing strings, so ab is not treated as overlapping a.

• A $rename claims both its source and destination against other operators, but its two endpoints are not compared with each other: mongod gives an overlapping pair its own error (“must not be on the same path”, code 2), and a self-rename its own too (“must differ”).

Also fixed

• The Rust server now attaches a failpoint’s errorLabels alongside its writeConcernError. Without RetryableWriteError a driver never classifies the write as retryable and never retries, which left /command_monitoring/unified/writeConcernError failing on the Rust server even after the replay fix cleared it on the Python one.

The first key decides whether an update is operators or a replacement

{z: 2, $set: {a: 1}} is not an operator update. mongod reads the first key alone to decide an update’s form, so that document is a replacement that happens to contain a $-prefixed field — which it refuses with DollarPrefixedFieldName (52) and a message pointing at $replaceWith. Both servers instead asked “does any key start with $”, called it an operator update, and answered 9 Unknown modifier: z. Reverse the two keys and mongod agrees with the old answer, which is what made the difference easy to miss.

The two errors are not just worded differently; they fire at different times, and that is observable. The operator-form 9 is a parse error, reported even when the filter matches nothing and on an upsert. The replacement-form 52 is an execution error: with no matching document the statement is a silent no-op (n: 0), and an upsert inserts the document verbatim, $-key and all. Both servers used to reject all three cases up front, so a legitimate no-match update failed and a legitimate upsert never happened.

Only the top level is restricted. mongod 8.x stores {a: {$bad: 1}}, {a: [{$bad: 1}]} and even a literal dotted key {"a.b": 1} without complaint, and insert accepts all of those too — all already correct on both servers, and now pinned so they stay that way.

Fixed

• secantus.update / secantus-core: an update’s form is decided by its first key, via one shared is_operator_form predicate rather than three separate any(...) tests.

• secantus.update / secantus-core: a $-prefixed top-level key in a replacement is mongod’s DollarPrefixedFieldName (52), raised at execution time and carrying the Plan executor error during <command> :: caused by :: wrapper, with the first such key named. A no-match update is a silent no-op and an upsert inserts the document unchanged.

• secantus.storage: an upserted replacement keeps the document’s own field order (_id first, then as sent) instead of being re-sorted by field name, which had put a $-prefixed key ahead of a plain one. Operator upserts are unaffected.

Update operators wrote where mongod refuses

tools/probes/update_operators.py compares update errors. Nothing compared the document a successful update produces, which is where a silently wrong write hides. A sweep of 31 updates × 17 seed value classes (527 cells) against mongod 8.2.11 found 30 divergent in four families; all 527 now agree.

Most of it is one shape CLAUDE.md already names — missing conflated with null. get_path(doc, path, default=None) returns None for a field that is absent and for one that is present and null, so three operators treated a null field as an absence and created a value over it.

Fixed — data was being written

• $push / $addToSet over a null field created an array. {$push: {v: 4}} over {v: null} wrote {v: [4]}, destroying the null, where mongod raises 2 The field 'v' must be an array but is of type null. An absent field is still created, which is the half that has to keep working.

• $bit over a null field wrote a number. The Python side read the current value as get_path(..., default=0) or 0, which turned every falsy present value — a null, a -0.0, an empty array — into the integer 0 so it passed the integral check. mongod refuses all three.

• $pull with a scalar emptied arrays of arrays. {$pull: {v: 1}} over {v: [[1, 2]]} left {v: []}; mongod leaves the document untouched, because [1, 2] is not 1. The scalar criterion was routed through the query engine, which adds implicit array traversal — the same membership-after-nesting family as the positional path fix. An operator or regex criterion does traverse, and still does: {$pull: {v: {$gt: 1}}} empties that same document.

Fixed — an invalid update reported success

• $rename through a non-document path silently no-opped. {$rename: {"v.k": "v.j"}} over any non-document v is 28 cannot use the part (v of v.k) to traverse the element ({v: 1}) on mongod. A genuinely absent path stays a no-op, which is the distinction the check has to preserve.

• $pop over a null field no-opped on the Python server where mongod and the Rust server both raise 14 Path 'v' contains an element of non-array type 'null'.

An upsert seeds every equality the query implies, not just the bare ones

When an upsert finds no match, mongod builds the new document from the QUERY — and it reads more than bare equality. Both servers seeded only bare equality, so five query forms lost their fields entirely: a silently wrong insert, since the document mongod would have written is missing a field.

Measured against mongod 8.2.11 over 20 queries × 6 updates: 24 of 120 divergent, now 0.

Fixed

query	mongod seeds	was
{a: {$eq: 1}}	a: 1	nothing
{a: {$in: [1]}} (one element)	a: 1	nothing
{a: {$all: [1]}} (one element)	a: 1	nothing
{$and: [{a: 1}, {b: 2}]}	a: 1, b: 2	nothing
{$or: [{a: 1}]} (one branch)	a: 1	nothing

A longer $in, a two-branch $or, $nor, and the range / $exists / $type / $not / $elemMatch operators seed nothing on mongod either, and still seed nothing here.

• Two clauses implying the same path are an error, not a silent pick: {$all: [1, 2]} and {$and: [{a: 1}, {a: 1}]} are both 54 cannot infer query fields to set, path 'a' is matched twice.

Deliberately not reproduced

mongod emits the seeded fields in its own hash-table order — {a: 1, b: 2} gives b, a, {aa: 1, ab: 2} gives aa, ab, {one, two, three} gives three, one, two. It ignores the query’s own order and is neither sorted nor reversed, so it is an implementation detail rather than a contract; CLAUDE.md already records that it changed between 6.0.16 (sorted) and newer servers. Both servers keep emitting the seeded fields sorted, and the tests compare field/value pairs rather than order.

$toDouble of a date was off by the server’s time zone

The Python server converted a date to a double by reading it as the host’s local time, so on any server not running in UTC, $toDouble of a date came back off by the host’s UTC offset. $toLong and $toDecimal were already correct, which is how it went unnoticed: every CI runner is UTC, and a London box is UTC in winter.

That is now also something CI can see. Both Windows lanes run on a Newfoundland host (never UTC, observes DST, a half-hour offset), and this bug is the first thing they caught. The same change gives the Rust server a local-time rendering test that runs on Windows, where the Rust engine’s TZ bug (#1468) used to hide.

Fixed

• expressions.py: $toDouble / $convert to double of a date pins UTC instead of reading the naive BSON date as local time.

Testing

• .github/workflows/test.yml: the test-windows and Windows storage-engine lanes set the host zone to Newfoundland and assert the change took.

• tests/test_rust_server_timestamp_local_time.py: drives the real Rust server through pymongo for $toLower of a Timestamp, with and without TZ, and runs in the storage-engine job.

A let variable made a pipeline update fail

Passing let to update or findAndModify and referring to it from a pipeline update — {update: "c", let: {x: 5}, updates: [{q: {}, u: [{$set: {v: "$$x"}}]}]} — was refused with Use of undefined variable: x, for a variable the command itself defines. update reported it as a per-statement write error and applied nothing; findAndModify failed the whole command.

The variable was bound. What went wrong is in the parse-time check that reproduces MongoDB’s constant folding: it decides an expression is constant because every $$name in it is bound, then evaluates it — and the update path passed the bound names without the values, so the evaluation hit an unbound variable and reported that as the query’s error. aggregate passed both and was unaffected, which is why this only ever showed up on writes. The fold now declines to fold a variable it has no value for, and both write paths pass the values the way aggregate does.

Two more differences surfaced alongside it. A statement’s own c constants map was never bound at all, so {u: [...], c: {y: 7}} failed the same way — and MongoDB rejects c outright on a non-pipeline update, which we accepted. And a constant that genuinely does fail ({$abs: "$$cv"} with cv: "x") takes MongoDB’s executor prefix inside a pipeline update, naming the command, rather than aggregate’s optimizer prefix.

Finally, neither write command validated its statements’ field names. MongoDB answers 40415 for anything it does not know inside a delete or update statement — including $-prefixed names, which get no envelope carve-out there — and refuses the whole command rather than the one statement, because it parses every statement before running any.

Found by the pymongo gauge: three *_with_let_option tests in test_crud_unified.py.

Fixed

• aggregate.py: the constant-fold check no longer reports a bound-but-unvalued variable as undefined, and wrap_expression_problem takes the command name for the update-pipeline executor prefix.

• commands.py: update and findAndModify pass the real let values to the fold; a statement’s c constants are bound (and win over let); c on a non-pipeline update answers 51198; unknown fields in a delete / update statement answer 40415.

macOS builds broke when SWIG shipped 4.5.0

Building from source on macOS started failing partway through the vendored WiredTiger step, with a wall of errors about calls to undeclared functions named PyInt_AsLong and PyString_InternFromString — names from the Python 2 C API, which have not existed since Python 3 was released.

The cause is upstream and outside this project. WiredTiger’s own SWIG typemaps still call those functions, and it has been getting away with it because SWIG used to paper over them: every generated wrapper carried a block of compatibility macros redefining the Python 2 spellings in terms of their Python 3 equivalents. SWIG 4.5.0 removed those macros, so the day Homebrew started serving it, the same unchanged source stopped compiling.

It looked intermittent, which is worth explaining. The wrapper is only generated when the WiredTiger build is not restored from cache, so a job with a warm cache skipped the whole step and passed while a sibling with a cold one failed — the same commit, green and red on adjacent machines, drifting redder as caches expired.

The typemaps now call the Python 3 functions directly, applied to the vendored tree at build time by the same mechanism as the project’s other WiredTiger patches. That removes the dependency on SWIG’s compatibility layer entirely, so it no longer matters which version generates the wrapper.

Fixed

• Source builds failed against SWIG 4.5.0 with undeclared Python 2 C API functions in WiredTiger’s Python bindings.

[0.6.0b16] — 2026-08-26

The drivers found the bugs this time

Six of the nine entries below started as a driver-conformance failure rather than a failing unit test, and two of them could only have been found that way. The C driver asserts on the exact text of the error a change stream returns when a pipeline modifies the event _id; SecantusDB returned the right code with its own paraphrase, so the assertion failed and pymongo — which never checks that string — had been passing it for months. The Go driver sends maxTimeMS: 0 to mean “tell me what is ready right now”; SecantusDB treated an explicit zero the same as an absent field and waited a second, long enough for an unrelated event to land inside the wait and be handed back to a client that had asked what was ready when it asked.

Both are the sort of divergence a permissive client never notices and a strict one fails on immediately. In each case the Rust server was already right and the Python server carried its own approximation, which is a useful thing to know: when the two disagree, the Rust side has usually been checked against mongod more recently.

Alongside those, arithmetic got stricter. $inc and $mul now reject non-numeric operands with mongod’s own error text instead of returning 500s or writing silently, and Decimal128 arithmetic is exact on both servers rather than routed through binary floating point. A differential harness that compares every operator against a live mongod — the thing that turned up nine of these — is now part of the test suite rather than a one-off script.

Tail latency improved as well: p99.9 relative to mongod moved from 1.48× to 1.18×, measured on dedicated cloud instances as part of cutting this release.

Tail latency improved again

Re-measured as part of cutting secantusdb-v0.5.3-beta.163, on dedicated cloud instances against a real mongod.

Changed

• p99.9 latency relative to mongod: 1.48× → 1.18×. In absolute terms our p99.9 fell from 49.10 ms to 37.34 ms (−24%) while mongod’s moved −5%, so the gain is ours rather than the reference moving. Throughput is flat at 0.74× (was 0.73×, inside the 3.1% pass spread).

• Two per-operation rows now beat mongod — indexed range 0.91×, full scan 0.96× — with filtered scan and the change-stream drain at parity. The Rust server’s overall range narrowed from 0.8×–2.9× to 0.9×–2.4×.

• The reference moved from mongod 8.0.29 to 8.0.31 (the gauge installs the latest 8.0.x). The results file records the version, so the change is traceable rather than silent.

Decimal128 arithmetic is exact on both servers

Stored decimals were quietly losing precision. $inc, $mul, $sum and $avg ran in Python’s default decimal context, which carries 28 significant digits — but Decimal128 carries 34. Every arithmetic result on a decimal field was silently truncated by six digits: incrementing 1.000000000000000000000000000000001 by one answered 2.000000000000000000000000000, dropping the trailing digit a real MongoDB server keeps. Nothing errored and nothing warned; the value simply came back shorter than it went in. The same four operators failed outright on the Rust server, which had no decimal arithmetic at all and rejected the write or the pipeline rather than answering.

Both servers now compute decimals exactly. The Rust engine gained a sign/coefficient/exponent implementation that adds, multiplies and divides without an intermediate binary float, rounding half-even to 34 digits exactly once and only when a result genuinely needs it. Crucially it preserves the quantum: Decimal128 distinguishes 5.00 from 5, so 2.50 + 0.10 is 2.60 and 2.50 * 2 is 5.00, matching MongoDB rather than collapsing to a normalised form.

Getting there turned up a genuine MongoDB quirk worth knowing about: the server uses two different rules for turning a double into a decimal. The accumulators take the double’s exact binary value, so $sum of 0.1 contributes 0.1000000000000000055511151231257827, while $inc, $mul and $toDecimal round to 15 significant digits and contribute 0.100000000000000. Both rules are now reproduced on both servers, $toDecimal included — it had been using neither. All of this is verified against a live mongod 6.0.16 rather than asserted from documentation, and the two engines are pinned to each other by several hundred thousand randomised comparisons.

Fixed

• $inc / $mul / $sum / $avg no longer truncate Decimal128 results to 28 significant digits; all 34 are kept, with IEEE 754-2008 preferred exponents so the quantum survives (2.50 + 0.10 → 2.60, 2.50 * 2 → 5.00).

• The Rust server computes these four operators over Decimal128 instead of failing. Previously a $group over a collection containing a single decimal value failed the whole pipeline, and $inc on a decimal field returned a write error.

• $toDecimal and $convert: {to: "decimal"} convert a double at MongoDB’s 15 significant digits (0.1 → 0.100000000000000, 4.125 → 4.12500000000000) rather than its shortest round-trip text, and round from the exact binary value so denormals match (5e-324 → 4.94065645841247E-324). All four implementations — both operators on both servers — were previously wrong, each in its own way.

• $sum / $avg convert a double by its exact binary value, matching MongoDB’s separate accumulator rule.

Added

• crates/secantus-core/src/decimal.rs — exact decimal128 arithmetic (add / mul / div_int, parse and render, both double-conversion rules).

• Decimal arithmetic cases in tests/test_mongod_differential.py, plus corpus cases in the update and aggregate Rust/Python parity suites.

• tests/test_rust_decimal_parity.py — a seeded generative parity fuzz over decimal arithmetic, the accumulators, and the conversions. It found three of the bugs fixed here that review and hand-written cases both missed, and it asserts two properties: that the engines agree, and that the Rust engine never defers on a decimal (a deferral is fatal on the standalone Rust server, which has no Python to fall back to). Scale it up with SECANTUS_DECIMAL_FUZZ_SCALE=50 when hunting.

The Go gauge can keep its data directory for debugging

SECANTUS_GAUGE_KEEP_STORAGE=1 invoke validate-go leaves the daemon’s storage behind instead of deleting it.

Added

• A driver-side assertion tells you a test failed but not what the server sent, and some gauge failures only reproduce under the whole run — so by the time there is a failure worth explaining, the oplog that produced it has already been removed. Keeping it is the difference between reading the offending entry and guessing at it.

An explicit maxTimeMS: 0 no longer blocks

getMore treated an explicit maxTimeMS: 0 the same as an absent one, because doc.get("maxTimeMS", 0) yields 0 for both — so a poll that asked for no waiting got the one-second default anyway.

Fixed

• mongod distinguishes the two: an explicit zero is a non-blocking poll, an absent field means wait. Drivers rely on it, and blocking there does not merely slow the call down — it changes the answer, because an event occurring during the wait comes back to a client that asked what was ready now. Found via mongo-go-driver’s change-stream suite, where a collection dropped by test teardown surfaced as a drop event the client had just been told did not exist. The Rust server already behaved correctly, so this also closes a two-server parity gap.

The test suite is warning-free again

starlette’s TestClient prefers httpx2 and warns on every construction when only httpx is importable. Six of those warnings came from the admin websocket tests, and they were the last ones left in the default suite.

Adding httpx2 to the dev extra silences them. It is test-only on purpose: nothing under src/secantus imports httpx at runtime.

Changed

• httpx2>=2.12 added to the dev extra. Refreshing the lock also pulled in its dependencies (httpcore2, httpx2-jsfetch, truststore) and moved idna 3.13 → 3.19.

$inc / $mul type errors now match MongoDB exactly

Incrementing a non-numeric field is an error on every MongoDB server, and both SecantusDB servers already refused it — but neither reported it the way MongoDB does, in two different ways.

The Rust server answered BadValue (code 2) where MongoDB answers TypeMismatch (14). The cause is structural: the shared operator engine signals “I can’t handle this, run the Python engine” with a single opaque value, which is the right answer on the Python server and a dead end on the standalone Rust server, where there is no Python to fall back to. Every such signal collapsed into one generic code. The fix adds a small validator that names the errors we can name — the same shape an existing $jsonSchema validator already uses — leaving the opaque signal for constructs that genuinely aren’t implemented.

Chasing that turned up a second problem nobody had recorded: the Python server’s message was wrong. MongoDB identifies the offending document by its _id ({_id: 1} has the field 'n' of non-numeric type string); we printed the field path instead ({n} has the field 'n' …). The code was right, so it had gone unnoticed — the text simply wasn’t one any real server produces.

Both servers now match MongoDB byte-for-byte on code and message, checked three-way against a live mongod and the standalone Rust binary.

Fixed

• The Rust server answers TypeMismatch (14), not BadValue (2), for $inc / $mul against a non-numeric field or with a non-numeric argument.

• The Python server’s type-error message identifies the document by _id, as MongoDB does, including ObjectId('…') rendering and the leaf field name for dotted paths.

The differential that found nine bugs is now part of the suite

The 2026-08 backlog audit found nine real defects — three that surfaced an unhandled exception as “internal server error”, three that silently wrote or returned wrong data, one where adding an index changed query results, and two missing capabilities. Every one came from running the same operation against SecantusDB and a real mongod and comparing. None came from reading the backlog, whose entries for those areas were absent, stale, or wrong.

That comparison lived in a scratchpad script. It is now tests/test_mongod_ differential.py: small independent cases, mongod as the oracle, errors compared as values because a wrong error code is a real divergence too.

It skips when no mongod is on PATH — the same convention the mongosh and database-tools tests already use — so it is free on machines without MongoDB installed and gives real coverage where it exists.

Added

• tests/test_mongod_differential.py, 16 cases covering every bug the audit found, plus a differential pytest marker (pytest -m differential).

The resume-token error now carries mongod’s wording

A change stream whose pipeline modifies the event _id is rejected with error 280 on both servers — but the Python server’s message was its own paraphrase, ending “makes it unusable for resuming”.

Fixed

• The Python server now returns mongod’s exact text, including the sentence drivers assert on: “Only transformations that retain the unmodified _id field are allowed.” The Rust server already carried it. libmongoc checks that string, so the paraphrase failed /change_stream/live/missing_resume_token and /invalid_resume_token in the C gauge — 758 passed / 10 failed → 760 passed / 8 failed (98.7% → 99.0%). pymongo does not assert on the message, so only a stricter driver could surface it.

$inc on a string no longer 500s, and $inc on a bool no longer writes silently

A three-way differential against a real mongod turned up three defects in the update operators.

$inc against a string field raised an unhandled ValueError that reached the client as “internal server error”; mongod answers TypeMismatch. $inc against a boolean silently computed a number, because Python treats bool as a subclass of int — that one wrote wrong data rather than failing, which is worse. $mul had both defects identically. Every non-numeric type is now refused with mongod’s code 14, and the document is left untouched.

$addToSet compared documents with Python ==, which ignores field order. mongod does not: {y: 2, x: 1} is a different value from {x: 1, y: 2} and gets appended as a separate element. Our query matcher already had this right, so $addToSet was disagreeing with our own equality rule; it now delegates to the matcher so the two cannot drift.

$min and $max are deliberately unchanged — unlike $inc/$mul they accept any type and use BSON cross-type ordering, which was verified against the same mongod.

Fixed

• $inc / $mul against a string, bool, null, array or document answer TypeMismatch (14) instead of an internal error or a silent write.

• $addToSet treats field-reordered documents as distinct, matching mongod and our own query matcher. tests/test_update_type_rules.py.

[0.6.0b15] — 2026-08-23

Restores stopped writing 2 GB for a 2 MB backup

A point-in-time restore used to write 2 GB to disk no matter how little data it was restoring. WiredTiger preallocates its log file to 2 GiB, that file is almost entirely zeros, and zeros compress to nothing but expand to everything — so a database holding 100 documents archived to 2.0 MB and restored to 2.0 GB. Restores now write sparsely: the same bytes, holes instead of zero-runs, a restored directory of 276 KB, and a restore that took 858 seconds on a busy disk now takes under three.

The second thread in this release is how performance gets measured at all. The published figures used to be whatever someone last remembered to re-run, on a developer laptop — where a background build or an OS indexer shifts every column at once and nothing in the output says so. They are now measured on dedicated cloud instances as part of cutting the release they describe, against a current mongod. Tail latency relative to mongod improved from 2.04x to 1.48x, three per-operation workloads now beat it outright, and the numbers on the site are the numbers for the build you can download.

The rest is correctness work across both servers: $avg on Decimal128, _id: NaN lookups, $arrayElemAt inventing nulls, array-field sorts that depended on whether an index happened to exist, a first batch that ignored mongod’s 16 MB reply cap, IPv6 binding, top on the Rust server, and replSetGetStatus agreeing with hello.

Sorting on an array field no longer depends on whether an index exists

MongoDB sorts an array-valued field by one representative element: its minimum ascending, its maximum descending. SecantusDB compared whole arrays, which placed every array after every scalar.

That was wrong against mongod, but the sharper problem was closer to home. A multikey index writes one entry per element, so an index scan already produced mongod’s ordering — meaning the same query returned a different order depending on whether an index happened to exist. An index is supposed to change speed, never results.

The in-memory sort now uses the same representative element the index path does, and an empty array sorts between MinKey and null as mongod places it.

Fixed

• sort on an array-valued field orders by the array’s minimum element ascending and its maximum descending, so indexed and unindexed sorts agree with each other and with mongod in both directions. tests/test_array_sort_order.py.

• The index scan no longer lets a multikey index’s whole-array entry decide sort position. Those entries exist to answer equality against a whole array; they were being hit first on a backward walk and steering the descending order.

• The Rust server matches, on all four paths. Its find sort builds a byte key through the same encoder that writes index entries, so the empty-array case is handled locally in the sort key rather than by renumbering the persisted ranks.

$avg no longer crashes on Decimal128, and $arrayElemAt stops inventing nulls

A $group whose $avg saw a Decimal128 alongside any other numeric type threw an unhandled TypeError out of the accumulator, which reached the client as a bare “internal server error”. $sum had always used the type-preserving bson_add; $avg used a raw += and was simply missed.

Fixing the crash uncovered a second bug beneath it: the average came back with 27 significant digits where mongod gives 34, because Python’s default decimal context is 28 while Decimal128 carries 34. Widened for the division, the result is now byte-identical to mongod.

Separately, $arrayElemAt with an out-of-range index returned null on both servers. mongod evaluates it to missing, so $project omits the field entirely — we were adding a field mongod does not send. A missing or null input array really is null, and that is unchanged.

Fixed

• $avg over Decimal128 returns a full-precision Decimal128 instead of raising TypeError. tests/test_avg_decimal128.py.

• $arrayElemAt out of range evaluates to missing on both servers, so $project omits the field as mongod does.

Benchmark numbers now come from the release they describe

The published performance figures were re-measured as part of cutting secantusdb-v0.5.3-beta.162, on dedicated cloud instances against mongod 8.0.29 — so for the first time they describe the build you can actually download, rather than whatever was current when someone last remembered to re-run them.

Changed

• Tail latency improved substantially: p99.9 against mongod moved from 2.04× to 1.48× on the head-to-head. Throughput is flat at 0.73× (was 0.75×, inside the 2.1% run-to-run spread).

• Three per-operation rows now beat mongod — filtered scan 0.85×, indexed range 0.92×, change-stream drain 0.93× — and full scan is at parity.

• Concurrency figures are now measured on the same cloud instance as everything else rather than a workstation. Absolute throughput is lower because those cores are slower; the scaling ratios the page reports are better, and mongod — the control — scales 4.96× there against 4.65× on the workstation, confirming the instance is not core-starved.

Concurrency sweep gives each row its own store

bench.concurrency shared one store across the whole writer sweep, so every row measured a different database. With 8,192-byte documents the 1-, 2- and 4-writer rows leave tens of gigabytes behind, and the 8-writer row wrote into a tree several times the size the 1-writer row saw. In a measurement whose sole purpose is to isolate writer count, that is a confound — and it biases scaling downwards, because later rows look worse partly for having a bigger tree.

It was also a hard failure. On a 48 GB droplet the accumulated store exhausted the disk mid-sweep and WiredTiger took its documented ENOSPC panic — “the process must exit and restart: WT_PANIC” — killing the writers. The harness correctly refused to report a row measured with a missing writer rather than publishing a silently-low number.

Each row now provisions a fresh store and server, bounding peak disk to a single row.

Changed

• Published concurrency figures are not comparable across this change: rows after the first previously carried the accumulated weight of every row before them, so scaling was understated. The next published sweep re-baselines.

Published benchmarks now measure on dedicated hardware

The per-operation latency and concurrent-writer scaling figures on docs/benchmark.md, docs/concurrency.md and the website’s performance page were measured on a developer laptop. That is not a trustworthy place to measure one, and the failure mode is silent: a background build or an OS indexer moves every column at once and nothing in the output says so. One run taken straight after a parallel compile recorded mongod itself at 2.5x its own baseline, and because the workloads run sequentially while load decays, the ratios were skewed too — it reported a fabricated 0.3x where the honest figure was 0.8x. The table looked entirely normal.

Added

• do-cluster perf / invoke do-perf — runs both Python benchmark harnesses on a DigitalOcean droplet and pulls bench/results/latency.json and bench/results/concurrency.json back for the chart generators. Uses only the server droplet: both harnesses spawn all three engines and drive them over loopback, which is what makes them per-operation engine measurements rather than network measurements, so a client droplet would add nothing but a NIC.

• bench.compare_servers --json PATH — writes results directly in the latency.json schema. Publishing these numbers previously required hand-transcribing 27 figures into that file.

• bench/DO_CLUSTER.md documents the command, and records why mongod is the control: it is measured in the same run and does not change between releases, so if its numbers drift from the previous results file, the machine moved and not the engine.

A find’s first batch now respects mongod’s 16MB reply cap

getMore already stopped at mongod’s 16MB reply budget; a find or aggregate first batch did not. It was capped on document count alone, so find with batchSize: 25 over 1MB documents assembled a 25MB reply and exhausted the cursor. Measured against a real mongod 6.0.16 on the same data, mongod returns 15 documents (15.0 MiB) and hands back a live cursor id.

Both servers now apply the same budget the cursor registry already used: stop before the document that would overflow, but always take at least one, so a single oversized document still makes progress rather than returning an empty batch forever. On the Rust server the blob lengths are already known, so this costs nothing on the hot path.

Fixed

• find / aggregate first batches stop under the 16MB reply cap and keep the cursor open for the remainder, matching mongod. Python and Rust both, covered by tests/test_first_batch_byte_cap.py and unit tests in secantus-commands::find.

A document with _id: NaN can be found again

Writing {_id: NaN} succeeded and then the document was unreachable by its own key: find({_id: NaN}) matched nothing while the row sat in the collection. The same held for any field — {x: NaN} never matched a stored NaN.

IEEE 754 says NaN is not equal to itself and Python and Rust both follow it, but mongod matches {x: NaN} against a stored NaN. Storage was never at fault: sortkey.encode_value already gives NaN a stable encoding, so the index entry was correct all along. Only the equality matcher was wrong, on both servers.

The rule is confined to equality. Range operators and sort keep IEEE semantics, so NaN still sorts below every other number and $gt: NaN still matches nothing.

Fixed

• {field: NaN} and {_id: NaN} match a stored NaN on both servers, across double and Decimal128 and between the two, matching mongod 6.0.16. Covered by tests/test_nan_equality.py and a secantus-core unit test that also pins the ordering behaviour left unchanged.

PITR restore no longer writes 2 GB for a tiny database

Every point-in-time restore wrote 2 GB to disk regardless of how much data it was restoring. WiredTiger preallocates its log file to log_file_max (2 GiB) and that file is almost entirely zeros, so it compressed to nothing inside the backup and expanded to full size on the way out: a database holding 100 documents archived to 2.0 MB and restored to 2.0 GB.

Fixed

• Backup extraction writes files sparsely, seeking past runs of zeros instead of writing them. The restored bytes are unchanged — holes read back as zeros — so this changes only how a restore reaches the disk, never what WiredTiger subsequently reads. A restored directory drops from 2.0 GB to 276 KB, and a restore that took 858 seconds on a busy disk now takes seconds.

replSetGetStatus now agrees with hello on the Python server

SecantusDB advertises itself as a single-node secantus replica-set primary so that drivers accept change streams. The Python server said so in hello — and then, asked replSetGetStatus, replied “not running with –replSet”. Real mongod is never both, and drivers notice: libmongoc’s test framework counts the member roster to decide what kind of server it is talking to, saw nothing, concluded standalone, and ran standalone-only tests against a server presenting itself as a replica set.

The Rust server fixed this months ago; the Python server never got the port. It does now, gated the same way — with a set name configured, replSetGetStatus reports the one-member PRIMARY roster matching hello; started with replica_set_name=None it is a genuine standalone and the honest NoReplicationEnabled error stands.

Fixed

• replSetGetStatus on the Python server reports a one-member PRIMARY roster consistent with hello, instead of the standalone NoReplicationEnabled error, whenever a replica-set name is configured. Ported from crates/secantus-commands/src/handshake.rs.

• Two mongo-c-driver gauge failures on the Python server — /Client/last_write_date_absent and its pooled variant — now report skip rather than failing. They are standalone-only tests that libmongoc should never have run against us, and it only did because the roster was empty.

Republished the performance numbers, measured properly

Every figure on the benchmark and concurrency pages has been re-measured. The per-operation latency table now comes from a dedicated cloud instance against mongod 8.0.29; the writer-scaling sweep from a quiet 12-core workstation, verified by two independent runs agreeing to within 1.2%.

Changed

• Latency ratios look worse, and SecantusDB is not the reason. The reference moved from mongod 6.0.16 to 8.0.29. Every “×mongod” figure is a ratio, so a faster denominator lowers the score: insert reads 2.0× where it read 1.4×, while SecantusDB’s own absolute timing barely moved. Publishing against a three-major-version-old mongod flattered us. The results file now records the mongod version so this cannot silently recur.

• Concurrency scaling looks better, and that is a fixed measurement rather than a faster engine. The sweep used to share one store across every writer count, so later rows carried everything the earlier rows wrote. Removing that lifted every engine — including mongod, unchanged code, by 14% at eight writers. The Rust server now measures 3.0× scaling at eight writers.

top works on the Rust server

mongotop failed outright against the Rust server: top answered code 59 CommandNotFound, so the tool errored instead of rendering a table. The backlog entry describing this said “counters are always zero”, which read as though it covered both servers and hid the fact that one of them did not implement the command at all.

It does now, ported from commands.py::_top — one totals entry per namespace, the note key mongo-tools skips, total/readLock/writeLock plus the per-op sections each {time, count}, and the same code-13 refusal outside the admin database.

The counters themselves are still zero on both servers: nothing instruments per-namespace operation timing, so mongotop renders an idle server. That half stays open and is recorded as such.

Fixed

• top on the Rust server returns the mongod-shaped reply instead of CommandNotFound, so mongotop runs against it. Covered by a secantus-commands unit test pinning the non-admin refusal.

The server is no longer IPv4-only

SecantusDBServer created its listening socket with a hardcoded socket.AF_INET, so an IPv6 host was not merely unserved — it failed at bind with a bare gaierror (“nodename nor servname provided”) that gave no clue the address family was the problem. Nothing in the suite bound a non-IPv4 host, so it went unnoticed.

The family now comes from getaddrinfo, which also handles hostnames and gives the correct wildcard address for an empty host. host="::1" serves a full round-trip; IPv4 behaviour is unchanged.

Fixed

• SecantusDBServer(host="::1", ...) binds and serves instead of raising gaierror. Covered by tests/test_server_bind_family.py, which drives a real insert and read-back over both families.

[0.6.0b14] — 2026-08-22

Writes got a lot faster, and we built the rig that proved it

The block compressor changed from zlib to lz4, and it is the largest single performance change this engine has had. Profiling the daemon under sustained write load found 65% of server CPU inside zlib’s deflate, on WiredTiger’s page-reconciliation path — which is also why tail latency was CPU-bound rather than I/O-bound. Swapping the compressor measured +86% write throughput and a 97% cut in p99.9 latency on incompressible data, and +15% / −88% on compressible data where zlib’s ratio is at its best. End to end on the three-droplet rig that is now part of the release process, that took SecantusDB from 3,993 to 11,099 operations per second and cut p99.9 latency from 1,303 ms to 37 ms — closing the gap to a real mongod from 0.27x throughput to 0.75x, and from 72x its tail latency to 2.0x. Existing data is safe: block_compressor is recorded per table at create time, so stores written before this release keep their zlib tables and zlib stays linked permanently to read them.

Finding it needed a better instrument than the single-host benchmarks this project had. There is now a three-droplet DigitalOcean harness that runs SecantusDB and a real mongod back-to-back on identical hardware — one server machine, two client machines, a real network between them — and reports medians across interleaved passes with a spread column, so a number comes with evidence about whether it is worth quoting. docs/benchmark.md publishes that comparison honestly, tail-latency weakness included, and the release process now re-measures it so the published figures cannot quietly go stale.

The PostgreSQL wire surface also grew substantially this cycle: stored procedures (CREATE PROCEDURE / CALL / DROP PROCEDURE), DROP TRIGGER, statement timeouts, anonymous record constructors, composite and composite-array catalog types, and a long list of cast, length and comparison fixes that bring psql, JDBC and SQLAlchemy closer to behaving as they do against a real PostgreSQL.

Note on 0.6.0b13: it was tagged but never published — its macOS wheel failed delocate’s deployment-target check, because Homebrew’s liblz4 targets macOS 14 while the wheel targets macOS 11. Accepting it would have dropped Apple Silicon users on macOS 11–13, so the lz4 dependency is now built from source at the wheel’s own target. No 0.6.0b13 artefacts exist on PyPI.

getTablePrivileges reflects relation ownership and ACLs

pgjdbc’s getTablePrivileges (and getViewPrivileges / matview variants) returned nothing: pg_class.relowner was hardcoded to PG’s bootstrap-superuser oid 10, so the driver’s c.relowner = r.oid join against the minted role oid found no rows. relowner now resolves to the connecting user’s role oid (every relation is owned by its creator), and pg_class.relacl reflects the relation’s ACL — NULL while untouched (a driver reads that as the owner holding every privilege implicitly), and a materialized aclitem array once a GRANT/REVOKE touches it. REVOKE ALL … FROM <owner> empties the owner’s entry, so getTablePrivileges then reports no rows (keeping noTablePrivileges correct).

Fixed

• pg_class.relowner on tables / views / materialized views resolves to the owning role’s oid (was 10), so getTablePrivileges’ owner join works.

• pg_class.relacl materializes from recorded grants (owner-implicit privileges + per-grantee grants; REVOKE ALL FROM owner → {}).

Anonymous record constructors and binary composite parameters

The (a, b, …) parenthesized tuple is now an anonymous record constructor, like ROW(a, b, …) — SELECT (1::int2, 2::int4, 3::int8, null) builds a RECORD value (OID 2249) that encodes and renders correctly in both the binary and text wire formats, preserving each field’s declared type OID (so the binary record embeds int2/int4/int8, not a collapsed int8). A COLLATE applied to a value expression is now a no-op on the value (it only affects comparison/sort order).

Binary composite bind parameters ($1::my_type sent in the binary format) are decoded through the declared type’s field layout and validated with PostgreSQL’s exact wire errors — a truncated header or element is 08P01, a wrong column count or element-type mismatch is 42804, and a declared element length that overruns the message is 22P03. A parameter declared as the generic anonymous RECORD type (OID 2249) is rejected with 0A000 (input of anonymous composite types is not implemented), matching PG. This closes the pgtest tuple corpus file.

Added

• scalar.py: the (a, b, …) tuple record constructor and COLLATE value-expression handling; record field OIDs derive from the argument AST (a bare NULL field is the unknown type, OID 705).

• pgextended.py: a validating binary-composite parameter decoder (_decode_binary_composite) raising PG’s 08P01 / 42804 / 22P03; a RECORD (2249) parameter is rejected with 0A000.

Fixed

• planner.py: a $1::user_type cast infers the type’s minted OID, so a binary composite parameter decodes through the record layout instead of being mis-read as text.

• pgextended.py: a NULL record field carries its declared type OID in the binary encoding (was always text).

--payload random, and the 19x disk finding it produced

do-client filled every document with a single repeated character. Both engines compress their tables — zlib here, snappy in mongod — so that payload compresses to almost nothing, and any measurement of bytes-on-disk becomes a measurement of the compressor rather than the engine. --payload random fills the same document shape with incompressible bytes instead. The default is unchanged, so existing comparisons stay comparable.

Measuring with it immediately turned up something worth fixing: SecantusDB leaves 674 MB on disk where mongod leaves 41 MB for the same 320 MB of documents. The data files are not at fault — they are smaller than mongod’s. It is the write-ahead log: 639 MB against mongod’s 11-16 MB.

The cause is the daemon’s 2GB --log-file-max default. WiredTiger reclaims only completed log files, so a workload writing 639 MB of WAL has produced exactly one still-active file and nothing can be freed until 2 GB is reached, where mongod’s 100 MB files rotate and are removed after each checkpoint. Setting --log-file-max 128MB drops the total 19x, to 35 MB — better than mongod — and a separate experiment shows the 2GB default is worth under 1% of throughput. The recommendation and its evidence are in tasks/backlog.md; the default is deliberately left unchanged here, as a durability-adjacent default deserves its own reviewed slice.

Added

• --payload repeat|random on do-client, defaulting to repeat.

Cached-plan revalidation happens at Bind

A named prepared statement whose result shape changed under DDL now raises cached plan must not change result type (0A000) during Bind, replacing BindComplete, so no portal is created — matching PostgreSQL, where the revalidation is part of planning rather than execution. Previously the error arrived at Execute, after the client had already been told the bind succeeded. Checking at Bind also keeps the revalidation ahead of any side effect, which is what a data-modifying CTE needs. The pgtest prepared_stmt_invalidation corpus file pins the exact reply and is now green.

Fixed

• The cached-plan error was reported after BindComplete instead of instead of it.

Composite columns and composite-array type names in the catalog

A composite-typed column (c custom) reported pg_attribute.atttypid 2249 (generic RECORD) instead of the composite’s minted oid, so getColumns / psycopg composite reflection couldn’t resolve its type name. Composite columns now report their minted oid (and composite-array columns the array-companion oid). Array type names also avoid collision the way real PG does: when _custom is already a type (a composite named _custom), the array type of custom becomes __custom rather than shadowing it. pgjdbc’s customArrayTypeInfo.

Fixed

• pg_attribute.atttypid for composite / composite-array columns is the minted type oid, not RECORD/text.

• Array type names in pg_type avoid collisions with existing type names by prepending underscores in element-oid (creation) order (custom[] → __custom, _custom[] → ___custom).

The block compressor is the largest performance lever in the engine

Profiling put 65% of the daemon’s CPU inside zlib’s deflate, on WiredTiger’s page-reconciliation path. That prompted the obvious question nobody had asked: zlib is not the only option, and MongoDB defaults to snappy.

WiredTiger can now be built with the snappy / lz4 / zstd builtin extensions alongside zlib (-DSECANTUS_WT_EXTRA_COMPRESSORS=ON, opt-in — the default build and its dependencies are unchanged), and the block compressor swept on the document and oplog tables. 8 writers, 1 GB cache, 8 KiB documents:

payload	compressor	throughput	p99.9 latency	disk
incompressible	lz4	+86%	−97%	1.9×
incompressible	zstd	+73%	−93%	1.67×
compressible	lz4	+15%	−88%	1.14×
compressible	zstd	+12%	−74%	1.11×

lz4 nearly doubles write throughput and cuts p99.9 latency by 30× on incompressible data, and still wins clearly on compressible data where zlib’s ratio should be at its best. The cost is disk footprint.

This closes a long investigation: it is why the tail was CPU-bound rather than I/O-bound, why a bigger cache helped (fewer evictions to compress), why no eviction tuning helped, and a large part of why a real mongod measured 3.7× faster with a 72× better tail in the three-droplet comparison.

It also refines an earlier conclusion rather than contradicting it. “Never turn oplog compression off — throughput craters” remains true; uncompressed pages mean more eviction I/O. But that compared zlib against none. The real axis is which compressor, and zlib sits at the wrong point on the CPU/IO curve for this engine.

lz4 is now the default. New document and oplog tables are created with it; snappy and zstd measured close enough that they stay behind an opt-in build flag rather than becoming two more link dependencies.

Existing data is safe. block_compressor is recorded per table at create time, so a store written before this change has zlib tables — and zlib therefore stays linked, deliberately and permanently. Verified end to end: a store created by the previous release was opened by the new build, 201,982 documents read back with zero errors, and subsequent writes added an lz4 table alongside the existing zlib ones. A unit test records why the zlib extension must not be removed as a tidy-up.

The trade is disk: expect roughly 1.9x the footprint on incompressible content and 1.14x on compressible. A --block-compressor flag for deployments that would rather have the space is tracked in tasks/backlog.md.

Changed

• The default block compressor is now lz4 for the document and oplog tables. zlib remains built and linked so existing stores stay readable.

Added

• SECANTUS_WT_EXTRA_COMPRESSORS CMake option and matching build.rs env hook (with SECANTUS_WT_EXTRA_LIBDIR) to additionally build snappy and zstd.

• SECANTUS_DATA_TABLE_EXTRA, a create-time config hook for the document tables, mirroring the existing SECANTUS_OPLOG_TABLE_EXTRA.

Stored procedures: CREATE PROCEDURE / CALL / DROP PROCEDURE

SecantusDB now supports PL/pgSQL stored procedures. CREATE [OR REPLACE] PROCEDURE name(params) LANGUAGE plpgsql AS $$ … $$ parses (including the a INOUT int argmode that sqlglot rejects — procedures are parsed by a dedicated handler), stores the body with its per-parameter modes, and CALL name(args) runs it over both the simple and extended wire protocols. A procedure’s OUT / INOUT parameters form its result row, so CALL proc(1) on proc(a INOUT int) returns 1; a procedure with no output parameters returns no row. RAISE NOTICE inside a procedure body surfaces as wire NoticeResponse messages, and COMMIT / ROLLBACK inside a procedure are accepted (execution continues). DROP PROCEDURE [IF EXISTS] removes a stored procedure.

Added

• engine.py / planner.py / plpgsql.py: CREATE PROCEDURE (parsed by a regex handler that accepts IN/OUT/INOUT/VARIADIC argmodes in either order), CALL returning OUT/INOUT parameters as the result row, DROP PROCEDURE, and PL/pgSQL COMMIT / ROLLBACK statements. parameter_count scans a raw Command’s tail so CALL proc($1) binds its parameter in the extended protocol.

Known limitation

• COMMIT / ROLLBACK inside a procedure are accepted but do not create a mid-body transaction boundary in the CALL’s autocommit context — a procedure that relies on committing part of its work and rolling back the rest is not modelled.

Declared column widths are enforced, and a cast to text works in a WHERE

A varchar(3) column would accept 'abcd' and store it. The declared width was recorded, reported to clients, and used when reflecting the schema — but never actually applied, so a column could hold data that contradicted its own definition, and the value came back out at full length. Over-length input is now refused with 22001 value too long for type character varying(3), on both INSERT and UPDATE. Postgres’ one exception is preserved: an overflow made up only of trailing spaces is trimmed to fit rather than rejected, so 'abc  ' still lands in a varchar(3) as 'abc'.

The second fix is a query that quietly returned the wrong answer. WHERE n::text = '2' found nothing, even for a row where n is 2. The cast was applied when a value was on its way out to the client, but not when it was used in a predicate — there, the stored number was compared against the string, so nothing ever matched. Such comparisons are now evaluated per row, where the cast is applied properly.

That surfaced a related trap worth knowing about: numeric values are stored in a decimal form that the conversion did not recognise, so WHERE d::text = '2.50' failed for its own separate reason even after the first repair. Both are fixed, and 2.50 still renders as 2.50 — the declared scale survives, as it does in Postgres.

Fixed

• char(n) / varchar(n) reject an over-length value with 22001 instead of storing it, on INSERT and UPDATE; a trailing-blank-only overflow is trimmed, as Postgres does.

• A comparison against a cast to text (WHERE n::text = '2') matches the rows Postgres matches, including for numeric columns.

Length functions, and an interval cast that leaked internals

octet_length and bit_length were answering for bit strings no matter what you gave them, so octet_length('abc') came back as 1 and bit_length('abc') as 3 — Postgres says 3 and 24. Both now measure a string’s encoded bytes, which also makes them correct for multi-byte text (octet_length('é') is 2 while length('é') is 1), and both keep their bit-string meaning for actual bit values.

Casting an interval to text handed back the internal storage form — {"interval": {"months": 0, "days": 1, ...}} — rather than 1 day. It now renders the same way an interval column already did on its way to a client.

Fixed

• octet_length() / bit_length() return Postgres’ answers for text and bytea, and keep bit-string semantics for bit values.

• interval::text renders Postgres’ interval text instead of the internal representation.

A three-machine benchmark harness for the Rust server

Every performance harness in this project has shared one host with the server under test. That means the load generator competes with the database for the same cores and page cache, and the “network” between them is loopback — latency that no real deployment will ever see, and a throughput ceiling set by whatever the client can drive on the leftover cores.

crates/secantus-bench measures the other shape. It provisions three DigitalOcean droplets — one running the standalone secantusd-rs daemon, two driving load at it across a private VPC — runs a coordinated benchmark with both clients loading the server over the same wall-clock window, collects the per-client results plus the server’s own CPU and memory trace, and tears the cluster down afterwards. One command does the lot: invoke do-bench.

The load agent speaks the MongoDB wire protocol directly through the project’s secantus-wire crate rather than through a driver. For a server benchmark that is the right instrument: driver overhead is the usual reason a client machine saturates before the database does. Measured against the same local secantusd-rs, it drives about 1.6x the operations per second of an equivalent pymongo agent at the same worker count.

The report is built to say when it should not be believed. It warns when a client saturated its own CPU (the client, not the server, was the limit), when operations errored (throughput counts only successes), when the two clients’ load windows failed to overlap, when the server sat idle enough that the bottleneck must be elsewhere, and when the server process died during the run — which invalidates the run outright.

Verified against a live DigitalOcean account: one c-4 server and two c-2 clients in lon1 sustained 6,230 operations per second with zero errors over 120 seconds, with the server at 82.9% mean CPU and the clients at 5-7% — the shape a valid server benchmark should have, since the instrument was nowhere near its own limit. Two consecutive runs agreed to within 0.16%.

Added

• crates/secantus-bench, a new WiredTiger-free workspace member shipping two binaries: do-cluster (provisioning, deployment, orchestration, reporting and teardown against the DigitalOcean v2 API) and do-client (the load agent and the server-side resource sampler).

• A log-linear latency histogram that merges by adding counts, so one report combines every worker across both client droplets into a single set of percentiles without shipping raw samples between machines.

• invoke do-bench / do-up / do-deploy / do-run / do-suspend / do-status wrappers, and a full operator guide at bench/DO_CLUSTER.md.

• Three teardown modes. The default is destroy, because a powered-off DigitalOcean droplet still bills at full price; snapshot keeps the installed software as a cheap image so the next run skips deployment, and power-off trades cost for a fast resume. do-status prints the live hourly rate from the API rather than a hardcoded price table that would go stale and quietly lie.

The three-droplet benchmark now measures SecantusDB against real MongoDB

The DigitalOcean harness gained the comparison it existed for. --engine both (the new default) installs MongoDB Community from MongoDB’s own apt repository alongside secantusd-rs, runs the identical workload against each in turn on the same droplets, and prints them side by side with a ratio.

Everything that could confound the comparison is held fixed: the same cores, the same clients, the same private network, the same operation mix, the same WiredTiger cache size — one --cache-size drives SecantusDB’s --cache-size and mongod’s --wiredTigerCacheSizeGB — and an empty data directory for each. The engines run sequentially, never side by side, because two databases sharing four cores would measure contention rather than either engine.

The first live comparison, on one c-4 server and two c-2 clients with 8 KiB documents and a 70/20/10 insert/find/update mix:

engine	version	ops/s	p50 ms	p99 ms	p99.9 ms	server CPU
secantusdb	0.5.3-beta.160	8,244	2.33	25.07	114.03	83.2%
mongod	v8.0.29	17,984	1.49	6.94	11.58	84.5%

SecantusDB reaches 0.46x MongoDB’s throughput on this workload, and the gap widens sharply in the tail — 3.6x at p99 and 9.8x at p99.9. Both engines saturated the same server while the clients idled at 12-15%, so both figures are server-bound and the comparison is fair; a second back-to-back pass reproduced the ratio exactly, with each engine within 1% of its first result.

Added

• --engine both | secantus | mongod and --mongod-version (default 8.0).

• A comparison.md artifact per run, and per-engine summaries, client reports, resource traces and journals alongside it.

• Ratios reported in their own senses — throughput above 1.0 is faster, latency below 1.0 is quicker — because conflating those directions is how benchmark tables mislead.

Fixed

• up reported every droplet it touched as needing a deploy, including ones it had just woken from a power-off or restored from a snapshot — which already carry their software. It now distinguishes freshly created droplets (“run deploy before run”) from woken ones (”run directly, no deploy needed”).

• The harness’s state and results directories were resolved relative to the current directory, so running it through cargo from crates/ created a second bench/.do-state there (one copy reached git). Both now anchor to the repository root.

Repeat a benchmark and report the median, not a single roll of the dice

The three-droplet harness could only measure each engine once per run, so nothing in the report said whether the number was stable. --repeat N fixes that.

The engines interleave within each pass rather than each running to completion in turn — pass 1 measures SecantusDB then MongoDB, pass 2 does the same, and so on. Thermal drift, a noisy neighbour, or anything else that changes over the run therefore lands on both engines roughly equally, instead of penalising whichever happened to go last.

Every figure is then a median rather than a mean: one pass disrupted by a checkpoint stall should not drag the headline, and with a small number of passes a mean is exactly what an outlier hijacks. Alongside it the report adds a spread column — (max - min) / median — which is the number that says whether the median deserves to be quoted, plus a per-pass table showing the raw figures in the order they ran.

Three 60-second passes on a c-4 server measured SecantusDB at a 3.4% spread and MongoDB at 1.1%, putting the throughput ratio at 0.49x — consistent with the 0.46x measured earlier on a different cluster at a different duration.

Added

• --repeat N on do-cluster run / all, and --repeat on invoke do-bench and invoke do-run.

• Median, spread and per-pass reporting in comparison.md; per-pass artifacts are named <engine>-pass<N>-… so every individual measurement is kept.

Record slow operations with timestamps, and what that found

do-client --slow-ms MS records every operation at or above a threshold with its completion timestamp, worker id and operation type. A latency histogram deliberately throws time away, which is precisely what a tail investigation needs back: whether slow operations arrive periodically (a checkpoint, a prune, a flush) or at random (lock contention, eviction) is the first fork in the diagnosis, and only timestamps answer it. It costs nothing when off, which is the default.

It was added to chase the p99.9 gap the MongoDB comparison exposed — SecantusDB runs at roughly half MongoDB’s throughput but with 9.8-11.3x its p99.9 latency, a far worse ratio than at the median. The traces localised it quickly, and the full finding with its evidence is recorded in tasks/backlog.md:

• The read path is clean — 8 concurrent readers see a 2x spread from p50 to p99.9. There is no read tail.

• The write path owns it, and it is a concurrency effect rather than a per-write cost: the tail jumps 7x between 2 and 4 concurrent writers (0.68ms → 4.83ms) while the median barely moves.

• Every large stall hits all workers within the same few milliseconds, at irregular intervals — a convoy behind a shared resource, not a periodic background task.

• Checkpoints, WiredTiger log preallocation, and log file size were each ruled out by experiment; oplog pruning turned out to drive the worst single outlier (126ms → 26ms when disabled) but not p99.9.

• The root cause is WiredTiger cache pressure: the tail scales inversely with cache headroom (52x spread at a 512M cache, 7x at 8G) while throughput and median latency stay flat. Shrinking the documents instead of growing the cache does the same thing, so it is the rate dirty data fills the cache that governs the tail.

• There is no WiredTiger config-only fix. Every eviction knob tried either did nothing (thread count) or made the tail worse while costing throughput (dirty and updates thresholds).

• On the droplets both engines ran the same 4G cache, yet mongod held p99.9 at 10.75ms where SecantusDB reached 121ms. The structural difference is admission control: MongoDB bounds concurrent storage-engine write transactions so excess writers queue outside the engine, while SecantusDB lets every connection thread dive straight into WiredTiger. The harness data prices the trade — capping writers at 4 rather than 16 costs 23% of throughput and halves the tail.

• --oplog-async recovers a further 24% of the tail and removes 42% of the stalls, independently of the cache story.

Added

• --slow-ms on do-client run, surfaced as slow_ops in the client report.

getColumns reports the size of domain columns

A domain over a length/precision-qualified base type (CREATE DOMAIN d AS varbit(3) / numeric(8,3)) now carries that typmod on its pg_type row (typtypmod + typbasetype), which is where JDBC’s getColumns reads a domain column’s COLUMN_SIZE. Previously the domain’s typtypmod was always -1, so a varbit(3) domain column reported an unbounded size instead of 3 (pgjdbc’s domainColumnSize).

Fixed

• Domain pg_type.typtypmod / typbasetype reflect the base type’s declared length/precision, so getColumns reports COLUMN_SIZE / DECIMAL_DIGITS for domain columns.

DROP TRIGGER

DROP TRIGGER [IF EXISTS] name ON table is now supported, removing a BEFORE INSERT FOR EACH ROW trigger so it stops firing on subsequent inserts. Dropping a trigger that doesn’t exist raises 42704 (or is silently tolerated with IF EXISTS). This completes the trigger DDL surface (CREATE TRIGGER already worked) and greens the pgtest schema_changes_implicit_txn corpus file, whose triggers subtest prepares CREATE TRIGGER and DROP TRIGGER over the extended protocol.

Added

• engine.py / catalog.py: DROP TRIGGER [IF EXISTS] name ON table (catalog.drop_trigger), routed through the DROP dispatch.

Fix: DROP TABLE no longer pins itself against the active-cursor guard

The active-portal DROP guard (which refuses DROP TABLE with 55006 while a suspended cursor in the same session still reads the table) counted the DROP TABLE statement’s own extended-protocol portal as a “query using the table” — so a plain DROP TABLE t via the extended protocol refused itself. This broke pgjdbc’s DatabaseMetaDataTest at setup (its DROP TABLE IF EXISTS bestrowid CASCADE failed, aborting the whole class). Only an active READ cursor now pins a table: a write portal (DML / DDL, including the DROP being executed) or a not-yet-executed portal never does. The real suspended-SELECT pin (55006) is unchanged.

Fixed

• DROP TABLE over the extended protocol no longer self-pins with 55006.

Fix --payload random, and find an uncompressed WAL

do-client --payload random generated one random payload per worker and reused it for every document. Each document was incompressible on its own and every document was identical — and WiredTiger compresses blocks spanning many records, so it crushed them anyway: 20,000 “random” 8 KiB documents produced an 8.8 MB table. The option existed precisely to make storage measurements honest and it was doing the opposite. The payload is now derived per document.

Re-measuring with it found something: SecantusDB writes byte-identical WAL volume for random and repeated payloads — 2.04x the logical data either way. That is proof the write-ahead log is uncompressed, where mongod defaults to snappy. On compressible data the gap is 2.04x against 0.05x, roughly 40x the write I/O for the same workload.

Enabling compressor=zlib on the log — zlib is already linked into the vendored WiredTiger and is what the data tables use — cuts p99.9 latency by 22% for 5% of throughput. That is a much better trade than the admission control prototype, and it composes with --oplog-async. The recommendation is in tasks/backlog.md; no default is changed here.

WiredTiger’s own statistics were then used to compare the two engines’ cache behaviour, which looked like it answered the question the investigation had left open. It did not: the bytes dirty in the cache cumulative counter turns out not to cover every table’s writes — SecantusDB’s documents table writes 12.4 GB from cache while that counter reads 0.24 GB — so any cross-engine comparison built on it measures instrumentation coverage rather than work. Both the “42x more dirty bytes than mongod” and the “oplog is 95% of dirty bytes” findings are retracted, with the evidence for the retraction recorded in tasks/backlog.md.

What survives is the counter that behaves consistently: bytes written from cache shows the oplog and the documents table each writing ~1x the logical data, so ~2x in total — which independently agrees with the 2.04x measured from WAL file sizes. The cache sweep (tail versus cache size) is a black-box measurement and is unaffected. Why mongod holds a better tail at the same cache size remains unexplained.

Fixed

• --payload random now varies per document, so a random dataset is actually incompressible across records rather than only within one.

Fix: pytest-tmp reaper’s liveness check works on Windows

The abandoned-pytest-tmp reaper (invoke clean) probed whether a tmp-dir’s owning run was still alive with os.kill(pid, 0) — the POSIX existence idiom. On Windows os.kill rejects signal 0 with OSError [WinError 87], a plain OSError the ProcessLookupError/PermissionError handlers never caught, so it errored instead of detecting a dead run (failing test_clean_pytest_tmp on the Windows CI lane). The liveness check is now cross-platform: Windows queries the process handle (OpenProcess + GetExitCodeProcess); POSIX keeps the signal-0 idiom.

Fixed

• _pytest_tmp_owner_alive liveness check is cross-platform (Windows CI no longer errors with WinError 87).

Comma-joins are keyed, not cartesian — getImportedKeys no longer OOMs

A multi-table comma-join (FROM a, b, c WHERE a.id = b.aid AND …) compiled each table to an UNKEYED $lookup that returned the whole foreign collection per outer row — a cartesian product filtered only by the terminal $match. Over several catalog tables that intermediate is astronomical: pgjdbc’s getImportedKeys for multi-column foreign keys (a 9-way comma-join over the catalogs) ballooned to 183GB and OS-killed the server.

The planner now pushes the WHERE equalities of the form joined.col = available.col onto each comma-join’s ON before building stages, so the $lookup is keyed (comma-joins are INNER, so this is result-preserving; residual predicates like single-table filters and array-subscript joins stay in WHERE). The getImportedKeys query now completes in milliseconds. A hard per-stage materialization cap (MAX_PIPELINE_DOCS, 5M, env-overridable) is the backstop: any query that still degenerates into an unbounded product fails with a clean SQLSTATE 54000 instead of exhausting memory.

Fixed

• Comma-join $lookups are keyed from WHERE equalities (getImportedKeys and similar catalog joins complete instead of cross-producting).

• Pipeline stages cap materialization at MAX_PIPELINE_DOCS, surfacing 54000 (program_limit_exceeded) rather than OOMing the server.

HAVING accepts any predicate, and a cast to text produces text

Postgres evaluates HAVING over the grouped rows, so anything that yields a boolean is legal there. SecantusDB lowered what it could into the aggregation engine’s $match and rejected the rest outright: HAVING NOT (count(*) > 1), HAVING count(*) BETWEEN 2 AND 5, HAVING count(*) * 2 > 3, a CASE, a call to abs() or coalesce() — all of them a flat “not supported”. Of ten common shapes surveyed, nine were refused. They now fall back to the same per-grouped-row evaluation the HAVING subquery case already used, and all ten return exactly what PostgreSQL 14 returns. A genuine mistake still fails properly: a bare column that is neither grouped nor aggregated is still the error Postgres raises, not a silently deferred predicate.

Fixing that surfaced something worse underneath. A cast to text was not producing text — it passed the value through unchanged — so count(*)::text = '2' compared the number 2 against the string '2' and quietly matched nothing. Rendering had hidden the bug for years, because a number and its text form go onto the wire as identical bytes; only a comparison could see the difference, and when it did, the answer was silently wrong rather than an error. Casts to text now convert, using Postgres’ own spellings: 2.0::float8 renders 2, 2.50::numeric keeps its scale, and true::text is true — not the t that appears in a result row.

Fixed

• HAVING predicates outside the lowerable set (NOT, BETWEEN, arithmetic on aggregates, CASE, function calls) no longer raise 0A000.

• A cast to text converts numbers, decimals and booleans to their Postgres text spellings, so comparing a cast result against a string literal works.

Multi-table joins filter each table before combining it

A query like SELECT … FROM a, b, c WHERE a.x = 1 AND b.y = 2 AND c.z = 3 has no join condition at all — it is a cross product with a filter on each table. Postgres narrows each table first, so the combination is over a handful of rows. SecantusDB was building the whole cross product and applying the filters at the end, which made the work grow with the product of the table sizes rather than with the answer: three 100-row tables meant a million intermediate rows for a single-row result.

Each single-table condition now moves to the stage that produces its rows — the base table’s own conditions run before the first lookup, and a joined table’s run inside its lookup. A condition combining tables with OR moves too, as long as every column it mentions belongs to the same table. On three tables with one condition each, time was previously cubic in table size (a million-row product took 2.5 seconds); it is now flat, and a 343-million-row product answers in hundredths of a second.

Separately, a join written the way SQL test suites usually write it — WHERE a3 = b9, with no table prefixes — was not being recognised as a join condition at all, so those tables were combined exhaustively and filtered afterwards. Unqualified columns are now resolved to the table that declares them (only when the name is unambiguous), so the join is performed as a join.

One case is deliberately left alone: a condition on the right-hand table of a LEFT JOIN is not moved into the lookup. WHERE is applied after the join, so such a condition has to remove the unmatched row entirely; filtering earlier would leave it in the result with empty columns. Every shape here was checked against a real PostgreSQL 14, including that trap and its ON-clause counterpart.

Changed

• A multi-table join filters each table as it enters the pipeline instead of after the full cross product.

• Unqualified equality conditions between tables are recognised as join keys.

A parameter’s declared type is resolved at Parse, like Postgres

Postgres picks the operator for a comparison while it is still analysing the statement, before a single row is read. That means a prepared statement comparing a varchar column against a parameter declared as uuid is rejected outright — operator does not exist: character varying = uuid — rather than quietly becoming a predicate that never matches. SecantusDB already restored that behaviour for literals, but a parameter was treated as undecidable, so the same query returned zero rows and no error. A client that declares its parameter types (psycopg does, from the Python object it dumps) could silently get an empty result for a query Postgres would have refused to prepare.

The declared parameter types are only known at Parse, so the check now runs there too, with the OIDs the client sent. The error message names the declared type rather than the internal storage tag: a varchar column folds to text internally, and reporting text = uuid would have been a message no real Postgres ever emits. Both the SQLSTATE and the wording were probed against a real PostgreSQL 14.

The analysis stays deliberately sound-but-incomplete — an undeclared parameter (Postgres’ unknown, which takes the other operand’s type) and any type pair the categories don’t model are left alone, because a spurious error breaks a working query, which is far worse than the lenient old behaviour.

Fixed

• A comparison against a parameter whose declared type has no operator against the other operand now raises 42883 at Parse, instead of silently matching no rows. The message names the declared type (character varying, not text).

pgjdbc wire fidelity: DISCARD tags, implicit-txn commit, OUT-param procedures

Four wire-protocol fidelity fixes surfaced by CockroachDB’s pgtest pgjdbc corpus (which greens as a result). DISCARD now echoes its target in the command tag (DISCARD ALL / DISCARD PLANS / DISCARD SEQUENCES / DISCARD TEMP) instead of a bare DISCARD. A simple Query message received mid-pipeline now commits any pending extended-protocol implicit transaction before it runs — pgjdbc’s autosave pattern interleaves a simple query between extended Executes and relies on the earlier statement committing (so re-executing a unique insert then conflicts), and the reported transaction status becomes idle; an explicit BEGIN block is left open. And stored procedures now support OUT parameters: a procedure is keyed by its total parameter count (CALL supplies a placeholder for every parameter, OUT included), CALL returns the OUT / INOUT parameters as its result row, and an extended-protocol Describe of a CALL portal reports that shape — a RowDescription when the procedure has output parameters, NoData when it doesn’t — without running the body (so a procedure that COMMITs internally emits no stray RowDescriptions).

Fixed

• engine.py: DISCARD <target> command tags; procedures keyed by total parameter count with OUT parameters forming the CALL result row and the extended Describe-portal shape (describe_statement CALL branch).

• pgserver.py: a simple Query mid-pipeline commits a pending extended-protocol implicit transaction (explicit blocks untouched).

Faithful errors for out-of-scope binary parameters

An untyped (unknown, oid 0) binary bind parameter whose bytes aren’t valid text — for example an EWKB GEOMETRY value bound to $1::GEOMETRY, a type SecantusDB’s core-PostgreSQL SQL layer doesn’t model — no longer leaks a UnicodeDecodeError as a generic XX000 internal error. It now surfaces PostgreSQL’s 22P03 (invalid binary representation), so the connection recovers cleanly with an honest error. PostGIS (GEOMETRY / BOX2D) and pgvector (VECTOR) remain out of scope for the surrogate; the pgtest gauge records them as documented divergences rather than unexpected failures.

Fixed

• pgextended.py: a non-text binary payload for an untyped parameter raises a faithful 22P03 instead of leaking a UnicodeDecodeError (→ XX000).

Changed

• pgtest gauge: spatial, box2d, and pgvector are documented expected divergences (extension types outside the surrogate’s core-PostgreSQL scope).

Wire fidelity for void returns, custom GUCs, binary bit params, and unknown params

Four PostgreSQL wire-protocol fidelity fixes surfaced by CockroachDB’s pgtest corpus. pg_sleep now reports its result as the void type (OID 2278, typlen 4) with a NULL value, matching real PostgreSQL, instead of an untyped text column. A custom (extension) GUC spelled namespace.name survives a round trip: SET custom_option.session_setting = 'abc' followed by SHOW custom_option.session_setting returns abc — previously the SET dropped the namespace prefix, so SHOW never found the value. A binary bit / varbit bind parameter is now decoded through PostgreSQL’s varbit_recv framing: an empty payload (too short for the 4-byte bit-length header) raises 08P01, and a payload whose declared bit length leaves trailing bytes unconsumed raises 22P03, where before the raw bytes were silently accepted as text. And a parameter a client declares as the unknown type (OID 705) is now resolved from context — the INSERT target column, a cast, a compared operand — exactly like an undeclared parameter, so ParameterDescription reports the resolved types instead of echoing 705.

Fixed

• functions.py / typemap.py / pgwire.py: pg_sleep returns the void type (OID 2278, size 4) with a NULL value (pgtest void).

• engine.py: SET namespace.name = … preserves the full dotted custom-GUC name so SHOW namespace.name resolves it (pgtest set).

• pgextended.py: binary bit / varbit parameters decode via varbit_recv framing — short payloads are 08P01, unconsumed trailing bytes are 22P03 (pgtest varbit).

• planner.py: an explicitly-declared unknown (OID 705) parameter is treated like an undeclared one, so parse-analysis type inference resolves it (pgtest unknown).

Publish the three-droplet MongoDB comparison, measured honestly

docs/benchmark.md carried only single-host loopback numbers, where the load generator competes with the database for the same cores. It now also carries the deployment-shaped measurement: one server droplet, two separate client droplets, real NICs between them, and SecantusDB run back-to-back against a real mongod on the same hardware.

The number is not flattering, and it is the one we would rather publish. On 8 KiB incompressible documents with a 70/20/10 mix, three interleaved 90-second passes on a c-4 server:

engine	ops/s	p50	p99	p99.9
SecantusDB 0.5.3-beta.160	3,993	2.41 ms	64 ms	1,303 ms
mongod 8.0.29	14,937	1.73 ms	10 ms	18 ms

A quarter of MongoDB’s throughput, and 72x the p99.9 latency. Both engines saturated the same server while the clients sat idle, so both figures are server-bound and the comparison is fair; run-to-run spread was under 4%.

This supersedes an earlier 0.46x figure that was measured with the harness’s default repeated-character payload. Both engines compress that away, and it flattered SecantusDB — MongoDB’s snappy-compressed journal benefits far more than SecantusDB’s uncompressed one. Re-measuring on incompressible data before publishing was the difference between a citable number and a misleading one.

The docs state the weak point plainly: SecantusDB’s p50 is within 1.4x of MongoDB’s, so typical operations are competitive, but under sustained write saturation the worst 0.1% of operations are far slower — and if that matters for your workload, run a real mongod. Two identified, unfixed, configuration-level causes are linked from there.

Added

• A “Over a real network, against a real MongoDB” section in docs/benchmark.md, reproducible with invoke do-bench --repeat 3 --payload random.

• --payload threaded through do-cluster and the invoke do-bench / do-run wrappers, so the orchestrator can select payload entropy (it previously existed only on the client agent).

Re-measure the published benchmark at release time

docs/benchmark.md publishes a head-to-head comparison against a real mongod. It is prose with numbers in it, so it goes stale silently: nothing in the test suite fails when the engine gets faster, and a release that improves performance ships a page that understates it.

That is not hypothetical. The figures published there were measured the day before lz4 replaced zlib as the block compressor — so without this step, the release that nearly doubled write throughput would have shipped the old numbers.

invoke release-benchmark provisions the three droplets, deploys both engines, runs three interleaved passes on incompressible payloads, prints the comparison table, and destroys the cluster. Two settings are deliberate: --payload random (both engines compress, so the default payload measures the compressor and flatters whichever side compresses harder — this alone moved the published ratio from 0.46x to 0.27x) and --repeat 3 (a single pass carries no spread, so nothing tells you whether the median is worth quoting).

Added

• invoke release-benchmark, and an “At release time” section in bench/DO_CLUSTER.md describing when to run it and why those two settings are not negotiable.

Result columns keep their identity across joins, views, and retypes

Every row Postgres describes on the wire says where each column came from: the relation it belongs to and its position within that relation. Tools lean on that. A JDBC updatable ResultSet resolves a result column back to its base column through those two fields, and reporting nothing left updateRow() emitting broken SQL. SecantusDB filled them in for a single-table SELECT, but a join has no single base table, so every column of a joined result reported no provenance at all. Joined columns now carry their own table’s identity — in SELECT tab1.a, tab2.c, both columns are the first column of their own table and both say so, and a computed expression alongside them no longer strips the identity from its plain siblings.

Views got the same treatment, and fixing them turned up a worse bug underneath. A view declared with a column list — CREATE VIEW v (v1, v2) AS SELECT … — was filed in the catalog under an empty name while CREATE VIEW still reported success, so every later reference to it failed as an undefined relation. The declared names are now applied to the stored definition the way Postgres applies them (positionally, surplus outputs keeping their own names, more names than columns raising 42601), a SELECT * body has its columns resolved once at creation, and a view’s result columns report the view’s own oid and its own positions rather than the underlying tables’.

Two smaller fidelity gaps closed alongside them. A char(n) value now goes out blank-padded to its declared width, as a blank-padded type should, while the semantics that read it — length(), comparison, casting to text — still see it unpadded, matching Postgres on both halves. And ALTER TABLE … ALTER COLUMN … TYPE now recomputes the column’s declared type identity instead of inheriting the old one: a column retyped from char(8) to text kept describing itself as bpchar with the old width, phantom padding included.

Added

• Result columns from a JOIN report their source table’s oid and 1-based attribute number, resolving qualified (tab1.a) and unqualified column references alike, including through SELECT *.

• Result columns selected from a view report the view’s own pg_class oid and the column’s position within the view.

• CREATE VIEW v (cols…) applies the declared column names to the stored definition, resolving a SELECT * body’s columns at creation time (so a column added to the underlying table afterwards does not appear in the view, as in Postgres).

Fixed

• CREATE VIEW with a column list registered the view under an empty name while still reporting success; every later reference failed with relation "v" does not exist.

• CREATE VIEW with more column names than the query has columns now raises 42601 CREATE VIEW specifies more column names than columns instead of silently ignoring the surplus.

• char(n) values are blank-padded to the declared width on the wire.

• ALTER TABLE … ALTER COLUMN … TYPE now replaces the column’s declared type oid and modifier; retyping away from char(n)/varchar(n) left the old declaration in place, so the column kept describing itself as the old type.

Comparing a text column against an integer now errors, like Postgres

Postgres resolves comparison operators while it analyses a statement, not while it reads rows, so SELECT * FROM t WHERE text_col = 42 never returns an empty result — it fails with operator does not exist: text = integer before the first row is touched. SecantusDB’s SQL layer evaluated comparisons with Python’s == on decoded BSON values, which absorbed the mismatch and quietly answered “no rows”. A query with a genuine type bug in it looked like a query with no matching data.

The Postgres front end now performs that operator resolution at plan time. A comparison whose two operands are both confidently typed, and whose types fall in two different categories — numeric, string, boolean, date-time — raises 42883 with Postgres’ message and no rows read. Everything the analysis cannot decide with certainty is left exactly as it was: an untyped string literal still takes the other operand’s type (text_col = '42' and int_col = '42' both keep working), and bound parameters, subqueries, arithmetic, unrecognised functions, CTEs, derived tables and views are all passed through untouched. A false positive here would break a working query, so the rule is sound rather than complete.

Schema-on-read tables are exempt on purpose. A reflected collection’s column types are inferred by sampling fifty documents, so a heterogeneous BSON field can be declared text while holding integers — comparing across BSON types is the whole point of the dual-protocol path, and it stays lenient there.

Added

• sql/typecheck.py: plan-time comparison-operator resolution, raising 42883 undefined_function for a cross-category comparison on declared tables. Reflected tables, unmodelled type categories (bytea, uuid, json, money, interval, time, arrays, ranges, geo, network, bit), untyped literals, parameters, and subquery scopes are all left lenient.

statement_timeout is enforced

The statement_timeout GUC is now enforced, not just accepted: a statement that runs longer than the configured timeout is cancelled with 57014 canceling statement due to statement timeout. The timeout is a per-statement / per-message-batch deadline on the session that pg_sleep (and other cancellation points) check, so a runaway query — for example SELECT pg_sleep(5) under SET statement_timeout='1s' — returns control to the client at the deadline instead of blocking. A bare numeric value is milliseconds (PG’s default unit); s / min / h / ms suffixes are honoured; 0 disables it.

Added

• session.py / functions.py / pgserver.py / pgextended.py: statement_timeout enforcement — the deadline is armed at the first statement of a query / extended-protocol batch and checked in pg_sleep.

Known limitation

• A portal paged with Execute MaxRows:N is materialised eagerly, so statement_timeout cancels the whole materialisation rather than emitting some rows and then timing out on a later page (that incremental behaviour would need lazy per-row portal evaluation). Simple queries and single statements time out correctly.

Timestamps keep their microseconds

MongoDB has no sub-millisecond date. A BSON date is a count of whole milliseconds, and the other date-like type — Timestamp — is coarser still, being seconds plus a counter for replication ordering. A Postgres timestamp carries microseconds, so inserting 12:00:00.123456 and reading it back gave 12:00:00.123000: the last three digits were dropped, silently, with no error.

Those digits are now kept. The BSON date still holds the whole milliseconds — a Mongo client reading the same collection sees exactly the date it always saw — and the leftover microseconds ride alongside it in a hidden __us_<field> companion, written only when they are non-zero, so a timestamp that lands on a whole millisecond adds nothing to the document. The companion is not a column: it never appears in SELECT *, in information_schema, or among the columns reflected from a schema-on-read collection.

The rule that matters if you are reading documents directly: every write resolves the companion. Updating a timestamp to a whole-millisecond value removes it rather than leaving the previous value’s microseconds attached, because a stale remainder would report a time that was never stored. A remainder outside 0–999 is ignored on read rather than trusted.

One limitation is unchanged and is now documented rather than implied: WHERE and ORDER BY on a timestamp column still compare whole milliseconds, because that is what the stored date holds. A sub-millisecond literal matches nothing, and rows within the same millisecond sort in an unspecified order — exactly as before. Reads are now precise; predicates are not yet.

Fixed

• timestamp / timestamptz columns round-trip microseconds through INSERT, UPDATE, SELECT and RETURNING instead of truncating to milliseconds.

The p99.9 tail is CPU-bound, and two thirds of that CPU is zlib

Profiling the three-droplet cluster on Linux settled what the tail investigation had left open.

Sampling every server thread’s scheduler state every 20 ms and intersecting it with the clients’ recorded stall windows shows that during a stall the threads are running, not blocked: 75.8% in state R inside stalls against 2.1% outside, with uninterruptible disk I/O at 0.9% and lock waiting actually falling. That retires the earlier “blocked on disk reads” reading.

A CPU profile then names the work. By shared object, 65.5% of the server’s CPU is inside libz — zlib block compression — with every top symbol being deflate under WiredTiger’s page reconciliation.

That single fact ties the whole investigation together: application threads conscripted into eviction spend their time compressing, which is why a bigger cache helps, why smaller documents help more, why eviction thread count does not help on a 4-vCPU box, and why bounding concurrency did not help either.

It also points at a lever nobody has pulled. Earlier work established that turning oplog compression off craters throughput, and that is true — but it compared zlib against none. The real choice is zlib against a cheaper compressor: MongoDB defaults to snappy, roughly an order of magnitude cheaper to compress at a worse ratio. SecantusDB pays zlib CPU on every page reconciliation. The build currently enables only the zlib WiredTiger extension, so selecting an alternative is a build-flag change before it is a tuning question; the work is scoped in tasks/backlog.md.

timetz accepts sub-minute zone offsets

A timetz value can now carry a zone offset with a seconds component — '00:00:00+01:01:03'::timetz parses and round-trips as 00:00:00+01:01:03 rather than being rejected with 22007. Postgres uses sub-minute offsets for historical LMT zones, and the offset is preserved (and only trailing all-zero groups are dropped, so +01:00 still renders +01). This greens the pgtest timezone corpus file — its remaining behaviour (session-TimeZone-aware timestamptz rendering with historical LMT offsets, GMT-N upper-casing, and binary time-type result encodings) already worked.

Fixed

• datetimes.py: the timetz parser accepts an HH:MM:SS zone offset, and the offset normaliser / renderer / splitter handle the sub-minute form.

Length-qualified character casts truncate (and char pads)

A cast to a length-qualified character type now applies the declared length, matching PostgreSQL. 'bar'::varchar(2) yields ba (was bar — the length modifier was parsed but silently ignored), and 'a'::char(4) yields a   (blank-padded to width). Bare text / varchar still impose no limit.

Fixed

• scalar.py: varchar(n) / char(n) casts truncate the value to the declared length (char(n) also right-pads with spaces); previously the length was never applied to the cast value.

--write-tickets: admission control for storage-engine writes

A new server flag bounds how many writes are inside WiredTiger at once, with the rest queueing outside it — MongoDB’s ticket system in miniature. Off by default (0), so a server that does not opt in behaves exactly as before.

It was built to test a specific hypothesis about the p99.9 tail gap against MongoDB, and the honest result is that it does not fix it. Three interleaved passes under maximum cache pressure, 16 concurrent writers: --write-tickets 4 cost 22% of throughput to buy 12% off p99.9 and 32% off the worst single stall. At a larger cache the benefit shrank to nothing, and two tickets made the tail worse.

Capping engine concurrency relocates the queue rather than removing it: a client with sixteen requests in flight still waits for all sixteen, whether they are stalled inside WiredTiger’s eviction or parked in a condvar. The flag therefore stays off by default and is documented as a diagnostic and a predictability knob — with tickets the tail became perfectly repeatable across passes — rather than as a performance fix. The full investigation, including the hypotheses it rules out, is in tasks/backlog.md.

Added

• --write-tickets N on secantusd-rs, and write_tickets under [storage] in the config file.

• secantus-storage::admission, a permit pool with a thread-local re-entrancy guard so a multi-document transaction rides its outer ticket instead of deadlocking against itself, and RAII release so a panicking write cannot leak a permit and wedge the pool.

[0.6.0b12] — 2026-08-17

Three privilege-escalation holes closed, and the PostgreSQL wire corpus from 10 files to 37

Three ways a caller could reach data its grants didn’t cover are now shut. An aggregation pipeline could sidestep role-based access control outright, so a user with no read grant on a collection could still read it through $lookup; a SQL write statement could read tables it held no SELECT grant on, because only the write target was authorized; and large-object writes honoured neither RBAC nor read-only transactions. Each is closed with tests that assert the denial, not just the happy path. The admin UI also stopped treating collection names as markup — a name containing a script tag is now escaped wherever it is rendered — and the Rust storage crates moved to PyO3 0.29, clearing two RUSTSEC advisories.

The PostgreSQL wire protocol had its largest single push so far, measured by running CockroachDB’s byte-exact pgtest corpus unmodified: 37 of 64 corpus files now pass, up from 10, with unexpected failures down from 43 to 18. Portals behave like PostgreSQL’s — they die at transaction end, refuse a duplicate name inside a block, block a DROP of a table they still read, and suspend on exactly their row limit instead of guessing that the data ran out. GUC reporting follows PG’s ordering and spellings, including savepoint-scoped reverts and the report that accompanies an error. Parameter typing follows PG’s parse analysis, so a placeholder compared against a column takes that column’s type and genuinely unresolvable placeholders are rejected rather than silently coerced. COPY reached byte-exact CSV fidelity, and jsonpath, ltree, and PostgreSQL’s internal one-byte "char" joined the type system.

Driver-facing depth grew alongside it: BEFORE INSERT row triggers with plpgsql NEW records, plpgsql refcursors (OPEN … FOR, CLOSE, FETCH by portal name), set-returning functions in the select list, EXCLUDE constraints, and a long list of catalog and metadata gaps that pgjdbc’s DatabaseMetaData suite reads. Two measurement fixes are worth naming because they change what the numbers mean: the pgtest gauge now runs each corpus file against a fresh server, as upstream does — sharing one server let debris from an earlier file fail two later ones — and the crash-diagnostic test harness now recognises a nested run that finished its tests before hanging in teardown, which had been failing releases on a passing suite.

Aborted-transaction pipeline semantics match PG

Two wire-protocol fundamentals pinned by CockroachDB’s pgtest corpus (the aborted_txn file). Inside an ABORTED explicit transaction, every extended-protocol step — Parse, Bind, Describe, Execute — now fails with 25P02 until the transaction ends, with PG’s transaction-exit carve-out (COMMIT / ROLLBACK statements still parse and execute, or a client could never leave the aborted block). Previously a Parse in an aborted transaction quietly succeeded.

And an errored extended-protocol pipeline now discards everything until Sync — including interleaved simple Query messages, matching PG’s ignore_till_sync. Previously a Query slipped through the discard and executed (answering 25P02 and an extra ReadyForQuery the client never expected).

Fixed

• sql/pgextended.py: 25P02 for extended steps in an aborted explicit transaction, with the COMMIT/ROLLBACK exemption.

• sql/pgserver.py: ignore_till_sync covers interleaved simple Query (and Fastpath) messages in an errored pipeline.

Admin UI: collection names can no longer inject script

The collections page built its per-row modify/rename toggle keys by splicing the collection name straight into an Alpine.js directive (@click, x-show). Jinja HTML-escaped the name, but the browser decodes those entities back before Alpine compiles the attribute as a JavaScript expression via new Function, so a collection name containing a single quote — no character restriction exists on collection names — broke out of the toggle-key literal and ran arbitrary script in the admin operator’s authenticated session on page render. Because the Mongo wire port is unauthenticated by default, this was a pivot from an anonymous wire client to the admin UI’s session.

The toggle keys are now the loop’s row index (an integer), so no attacker-controlled string ever reaches the Alpine expression context. The name still renders, safely escaped, in the row’s link, form actions, and rename field.

Security

• Stored XSS on the admin collections page: a collection name with a ' executed script in the operator’s session on render (#835). Row toggle keys are now integer indices, keeping the name out of every Alpine.js directive.

ALTER TABLE … DROP without the COLUMN keyword

ALTER TABLE t DROP name — the keywordless column-drop form real Postgres accepts — no longer errors. sqlglot parses the action as a raw Command; the executor now recognises DROP [IF EXISTS] <col> [CASCADE|RESTRICT] there and applies the standard drop-column action. (pgjdbc’s DatabaseMetaDataTest droppedColumns.)

Fixed

• Keywordless ALTER TABLE … DROP <col> (quoted or bare, IF EXISTS, CASCADE/RESTRICT tails).

Multi-statement batches are one implicit transaction, like Postgres

A multi-statement simple query now runs in a single implicit transaction, matching real PG: a mid-batch error rolls back the earlier statements’ writes (their result rows were already streamed — PG streams too), an explicit BEGIN inside the batch takes the transaction over — its characteristics included, so BEGIN READ ONLY makes a following write fail 25006 and poison the block — and COMMIT/ROLLBACK end it with the remainder starting a fresh implicit transaction. Previously each statement ran in its own autocommit transaction, so a failed batch left earlier writes behind — a recorded semantic divergence, now closed (pinned by the pgtest batch_stmt corpus, which is fully green).

In support: a batch whose statements only parse individually (BEGIN READ ONLY mid-batch needs the regex fallback) now splits at top-level semicolons — respecting quotes, dollar-quotes, and comments — and parses each segment through the full entry point.

Fixed

• sql/engine.py: run_sql wraps multi-statement batches in an implicit transaction with PG’s takeover/settle rules; BEGIN takeover applies the BEGIN’s characteristics; the read-only gate poisons an open block.

• sql/planner.py: top-level-semicolon segment fallback when a batch fails to parse as one string.

Binary numeric wire fidelity

Two fixes in the binary numeric parameter decoder, both pinned by the pgtest decimal corpus file (now fully green). A zero encoded with thousands of zero digit-groups — CockroachDB’s #38139 regression payload uses 8192 of them — now renders as 0 instead of 0.000…0: the decoder quantizes to the declared display scale unconditionally, where previously a dscale of 0 skipped the quantize and scaleb on a zero kept the huge negative exponent. And a declared scale outside PostgreSQL’s NUMERIC_DSCALE_MASK (for example 0xFFF0, a negative int16 reinterpreted) is rejected at Bind with PG’s 22P03 invalid scale in external "numeric" value, instead of silently producing an absurd quantization.

Fixed

• Binary numeric zeros with non-canonical digit-group padding render as 0.

• Out-of-range binary numeric dscale raises 22P03 at Bind (was accepted).

Bind validates its parameter count

A Bind message must supply exactly as many parameters as the prepared statement has, and the parameter types the client declared at Parse count even when the query text uses fewer placeholders — declaring three and binding one now raises PostgreSQL’s 08P01 instead of silently executing with whatever arrived. COPY keeps its own 08P01 (with PG’s statement-summary detail) for the same mistake. The pgtest prepare corpus file pins both and is now green.

Fixed

• Bind accepted a parameter count that disagreed with the prepared statement’s.

citext gets a real wire oid

citext columns and parameters now report oid 90008 — the stable placeholder CockroachDB uses for the extension type, mirroring how hstore rides 16935 — instead of collapsing into text’s oid 25. ParameterDescription infers it for INSERT targets and for unknown parameters compared against a citext column (citext only: it ships its own operator family, unlike types that resolve comparisons through text), RowDescription reports it for citext columns, and binary parameter/result formats carry the text bytes. Case-insensitive matching is unchanged. The pgtest citext corpus file pins the whole exchange byte-for-byte and is now green.

Changed

• citext’s wire oid: 25 → 90008 (drivers treat the unknown oid as text, so text-mode round-trips are unaffected).

Added

• Binary param/result codecs for citext (raw text bytes).

• Comparison-against-citext-column parameter inference (oid 90008).

COMMENT ON DOMAIN / INDEX and obj_description()

Two more COMMENT targets and the lookup function that reads them. COMMENT ON DOMAIN stores on the domain (surfacing in pg_description under classoid pg_type), COMMENT ON INDEX stores by index name (resolved to the index relation’s oid at read time), and obj_description(oid[, 'catalog']) / col_description(oid, attnum) look comments up the way pgjdbc’s getUDTs and getIndexInfo do. IS NULL removes a comment on both new targets.

Added

• COMMENT ON DOMAIN d IS '…' | NULL (sqlglot Command fallback path).

• COMMENT ON INDEX i IS '…' | NULL with pg_description reflection.

• obj_description / col_description scalar functions.

COMMENT ON FUNCTION and table row types as column types

COMMENT ON FUNCTION f(args) (and the bare-name form) stores the comment on the function’s catalog entry, and a column may now be declared with a table’s name as its type — in PostgreSQL every table is also a composite row type, so CREATE TABLE t (col other_table) stores that row shape and supports (col).field access. These were the two setup blockers behind four pgjdbc DatabaseMetaData/ResultSetMetaData/ RefCursor test classes whose entire suites died in class setup; ResultSetMetaDataTest alone now runs its 60-test matrix.

Added

• COMMENT ON FUNCTION (parenthesised and bare-name forms; 42883 for unknown functions, 42725 for ambiguous overloads).

• Columns typed by a table’s row type.

COPY reaches byte-exact CSV fidelity

COPY’s CSV codec now mirrors PostgreSQL’s own parser: quoting decides NULL-ness (a quoted "N" under NULL 'N' is the string N, a quoted empty field is the empty string, and only unquoted cells match the null token), custom ESCAPE characters work inside quoted fields, a \. line terminates the data stream, an unterminated quoted field raises 22P04, and COPY TO CSV force-quotes empty strings so they stay distinct from NULL. The legacy un-parenthesized option syntax (CSV NULL 'NS' DELIMITER '|') parses correctly, ESCAPE and HEADER outside CSV mode raise PG’s 0A000, and text-format \xHH / octal byte escapes decode on COPY FROM. COPY (query) TO STDOUT also works through the extended query protocol now — Describe answers NoData and Execute streams CopyOutResponse/CopyData/CopyDone — with PG’s exact error shapes for parameters (COPY takes none: binding any is 08P01 with the statement-summary detail; an unbound placeholder at Execute is 42P02). The pgtest copy corpus file (1187 lines) pins all of it and is fully green.

Added

• Extended-protocol COPY (query) TO STDOUT (Parse/Bind/Describe/Execute).

• CSV ESCAPE character support and \. terminator handling.

• Text-format \xHH / \OOO byte-escape decoding on COPY FROM.

• crdb-style inline INDEX (...) table elements are accepted (index skipped); ADD COLUMN ... NOT VISIBLE parses as a normal column.

Fixed

• Quoted CSV cells equal to the NULL token no longer read back as NULL.

• COPY TO CSV writes empty strings as "" (previously indistinguishable from NULL).

• Legacy COPY option lists (CSV NULL 'NS' DELIMITER '|') no longer mis-parse into default delimiters.

• ESCAPE/HEADER without CSV raise 0A000 instead of silently entering copy mode (which deadlocked the connection).

COPY option validation and custom CSV quote characters

COPY now validates its option keywords against PostgreSQL’s grammar: an unknown keyword (for example CockroachDB’s WITH destination = '…' upload extension) raises PG’s 42601 syntax error at parse time, before the target table is resolved — previously it fell through to a misleading 42P01. The QUOTE option is also implemented for real: a custom CSV quote character applies to both COPY FROM parsing and COPY TO rendering, is rejected outside CSV mode with PG’s 0A000, and a multi-character quote raises 22023. The pgtest copy_file_upload corpus file pins the 42601 shape and is now green.

Added

• COPY … CSV QUOTE 'x' — custom quote characters in both directions; ENCODING/FREEZE/OIDS accepted as no-ops.

Fixed

• Unknown COPY option keywords raise 42601 (syntax error), not 42P01.

DatabaseMetaData catalog gaps: max_index_keys, proargmodes, reltuples, output-alias ORDER BY

Four of DatabaseMetaDataTest’s failure clusters, all catalog or planner gaps. pg_settings now carries max_index_keys (32 — pgjdbc reads it once per connection and every foreign-key metadata call died without it). pg_proc gains proargmodes / proallargtypes (NULL — no OUT params recorded) so getFunctionColumns runs. pg_class gains reltuples (−1, PG’s “no estimate yet”) so getIndexInfo’s CARDINALITY reads. And ORDER BY naming a computed output alias in a join query — pgjdbc’s getTables sorts by the CASE-computed "TABLE_TYPE" — now resolves through the select list like the single-table and grouped paths already did, instead of raising 42703.

Added

• pg_settings.max_index_keys (32), pg_proc.proargmodes / proallargtypes, pg_class.reltuples (−1).

Fixed

• Evaluated-join ORDER BY resolves computed output aliases and ordinals (ORDER BY "TABLE_TYPE", ORDER BY 1) instead of erroring 42703.

Describe stops executing volatile functions, regclass resolves, and more

The extended protocol’s Describe no longer executes volatile functions to learn a statement’s shape: Describe select pg_sleep(5) actually slept (and a CancelRequest arriving mid-sleep was swallowed into a NoData reply while Execute later emitted a DataRow — the protocol violation JDBC’s setQueryTimeout crashed on), and a Describe of nextval(...) drew a sequence value. Known volatile session functions now describe from a static type table, cancellation interrupts Execute where it belongs, and the sequence is untouched until execution.

'name'::regclass resolves to the relation’s pg_class oid — including schema-qualified spellings and search_path resolution for bare names — while still rendering as the relation name, so metadata queries joining c.oid = ?::regclass work (pgjdbc’s SearchPathLookupTest). Slash-format timestamp input ('8/10/7777') parses per the session’s DateStyle field order (MDY/DMY), array_fill(value, ARRAY[dims]) lands, and geometric results (point, lseg, box, path, polygon, line, circle) have binary-mode encoders, completing pgjdbc’s binary PGpoint/PGbox round-trips.

Added

• regclass casts resolve names to pg_class oids (42P01 when unknown); regtype casts resolve base types and table row types to pg_type oids, with search_path resolution and bare-typname rowtype rows in pg_type.

• Slash-format DateStyle-aware timestamp input.

• array_fill (value + dimensions form).

• Binary result encoders for the seven geometric types.

Fixed

• Describe evaluates no volatile function (pg_sleep, nextval, setval, set_config, lo_*, advisory locks); cancels land in Execute with 57014.

Enum casts describe like PostgreSQL’s

An unaliased cast to a user-defined type now names its output column after the type — SELECT 'hi'::te yields a column named te, matching PG’s rule for enums, composites, and domains — and RowDescription reports DataTypeSize 4 for enum-typed columns, mirroring how PostgreSQL stores enum values as 4-byte oids. The pgtest enum corpus file pins both shapes and is now green.

Fixed

• SELECT 'x'::myenum reported ?column? with size -1; now the type name with typlen 4.

Constraint violations carry PG’s error-identity fields, and EXCLUDE lands

Every constraint violation the SQL engine raises now attaches the ErrorResponse identity fields real PostgreSQL sends — schema, table, column, constraint, and datatype — which drivers surface through psycopg’s diag and pgjdbc’s ServerErrorMessage. Duplicate keys name the violated constraint (the declared PK name, not a synthesized one), NOT NULL violations name the column, domain CHECK failures name the domain and its check constraint, and foreign-key violations name the referencing table. pgjdbc’s ServerErrorTest — which asserts exactly these fields for six violation kinds — passes in full.

EXCLUDE (col WITH =, ...) table constraints are supported in their equality-only form: enforcement is a unique index under the hood, but a conflict raises PostgreSQL’s 23P01 exclusion_violation with the <table>_<col>_excl constraint name. Non-equality exclusion operators (the GiST range forms) keep the honest “not supported” error.

Added

• ErrorResponse diagnostic fields (s/t/c/n/d) on unique, not-null, check, domain-check, foreign-key, and exclusion violations.

• Equality-only EXCLUDE table constraints, violating with 23P01.

Fixed

• Duplicate-key errors name the violated constraint instead of the table; schema-qualified tables report schema and bare name separately.

EXECUTE portals describe their underlying statement

Describing a portal bound to a wire-parsed EXECUTE name(args) — the SQL-level PREPARE/EXECUTE flow driven through the extended protocol — now resolves the underlying prepared statement and reports its result shape: a prepared SELECT answers with its RowDescription instead of NoData, so clients no longer receive DataRows without a preceding row description. The pgtest execute corpus file pins the exchange and is now green.

Fixed

• Describe(P) of an EXECUTE portal returned NoData for row-returning prepared statements.

Set-returning functions in the SELECT list, and search_path-aware visibility

information_schema._pg_expandarray now works in the SELECT list — bare (a composite (x, n) column, one output row per array element) and with immediate field access ((SRF(arr)).n), with multiple references to the same call expanding in lockstep and empty arrays eliminating the row, as PostgreSQL does. Composite field access (col).x also lowers inside JOIN ON conditions. Together these are the exact call sites pgjdbc’s DatabaseMetaData.getPrimaryKeys / getPrimaryUniqueKeys queries emit, which back JDBC updatable ResultSets.

pg_table_is_visible() now honours the session’s search_path instead of a hardcoded default-namespace list — SET search_path TO schema1 previously made every user-schema relation invisible to the predicate, so a same-named table in two schemas could not be disambiguated (the pgjdbc updatable-resultset probe hit exactly this). The function also works as a projected value in any expression context, alongside current_database() / current_schema().

Added

• Record SRFs in the SELECT list (_pg_expandarray), FROM-ful and FROM-less, with lockstep multi-reference expansion.

• (col).field composite access in JOIN ON.

• current_database() / current_catalog / current_schema / pg_table_is_visible() in per-row expression contexts.

Fixed

• pg_table_is_visible WHERE lowering follows the session search_path (was a hardcoded public/pg_catalog list).

Foreign-key metadata: conindid, referential-action codes, and a 3x faster $unwind

pgjdbc’s getImportedKeys/getExportedKeys/getCrossReference returned zero rows: a foreign key’s pg_constraint.conindid was 0, and the metadata query joins pkic.oid = con.conindid to name the referenced PK index — the join silently emptied every FK result. FK rows now point conindid at the referenced table’s PK index and carry the one-letter confupdtype / confdeltype referential-action codes (CASCADE ‘c’, SET NULL ‘n’, SET DEFAULT ‘d’, RESTRICT ‘r’, NO ACTION ‘a’).

The same query exposed an aggregation hot spot: $unwind deepcopied every fanned-out doc, dominating high-fanout join pipelines (the FK query is a 9-way join). When no stage in the pipeline (including nested $lookup/$facet/$unionWith sub-pipelines) mutates docs in place, unwind now fans out with shallow top-level copies — every writing stage deepcopies its input first, so shared subtrees are never corrupted. ~3x on the FK metadata query; benefits MongoDB-side aggregations with the same shape.

Fixed

• pg_constraint.conindid on FK rows (was 0); new confupdtype / confdeltype columns.

• $unwind shallow fast path gated on pipeline mutation analysis ($fill / $densify anywhere in the pipeline keep the deepcopy path).

Single-precision float fidelity

float4 values now behave like PostgreSQL’s: a cast to float4 narrows the value to single precision, and its text form is the shortest decimal that round-trips at that precision — (1/3.0)::float4 prints 0.33333334, in arrays too — while float8 keeps the full double form. The extra_float_digits GUC’s negative range now works for both widths (%.{15+n}g / %.{6+n}g, as PG renders when shortest-output is turned down), which also uncovered that SET silently dropped the minus sign from negative values. Bare ARRAY[…] and ROW(…) constructors now name their output columns array and row like PG. The pgtest float corpus file pins all of it and is now green.

Fixed

• float4 rendered at double precision (0.3333333333333333 instead of 0.33333334).

• SET guc = -1 stored 1 — the sign vanished in value extraction.

• Unaliased ARRAY/ROW constructor columns were named ?column?.

Binary inet parameters reject malformed payloads like PostgreSQL

A malformed binary inet parameter now raises PostgreSQL’s error classes instead of leaking an internal XX000: a truncated header is 08P01 (insufficient data), and a bad address family or address length is 22P03 (invalid binary representation), matching inet_recv. The pgtest inet corpus file pins all four shapes and is now green.

Fixed

• Empty / truncated / bad-family binary inet parameters surfaced XX000.

Binary int2vector results encode as int2 arrays

Requesting an int2vector column (pg_index’s indkey/indoption) in binary result format now yields PostgreSQL’s wire form — an int2 array with element oid 21, 2-byte elements, and lower bound 1 — where previously the text rendering leaked through the binary format. Binary pgwire clients decoding index metadata get well-formed arrays. The pgtest int2vector corpus file pins the encoding byte-for-byte (its expected indoption VALUE is CockroachDB’s NULLS-FIRST 2 where PostgreSQL — and SecantusDB — report 0; recorded as an expected divergence).

Fixed

• Binary-format int2vector results carried text bytes.

PG’s internal one-byte “char” type

The quoted "char" spelling now names PostgreSQL’s internal one-byte type (oid 18, typlen 1) instead of collapsing into char(n)/text: casts report a column named char with oid 18, table columns declared "char" describe and bind parameters with oid 18, input values truncate to one character, an empty string or zero byte stores SQL NULL, 0::"char" produces the zero byte (rendered as one 0x00 byte in binary result format), and binary parameter/result codecs carry the raw byte. sqlglot loses the quoting — the quoted and unquoted spellings both parse as plain CHAR — so the planner rewrites the quoted form to an internal sentinel type name before parse, token-context aware: it fires after ::, after AS only inside CAST(...), and in a CREATE/ALTER column-type position, never on aliases, column names, or string literals. The pgtest char corpus file pins the whole surface byte-for-byte (its one remaining stanza expects CockroachDB’s deterministic TableOID and cannot pass against any non-crdb server; recorded as an expected divergence).

Added

• "char" (quoted, oid 18) as a first-class column and cast type: 1-char truncation, NULL for empty/zero-byte input, int::"char" as chr(i) with 22003 out-of-range, binary param/result wire codecs, typlen 1 in RowDescription.

Fixed

• SELECT 'a'::"char" reported oid 25 with a bpchar column name; it now reports oid 18, size 1, named char (pgtest char:42).

The jsonpath type

::jsonpath casts now produce a real jsonpath value: oid 4072 on the wire, PostgreSQL’s canonical text form ($.abc renders $."abc", subscripts and filters re-render canonically), a 42601 syntax error on an empty path, and the binary format PostgreSQL sends — a version byte followed by the canonical text. jsonb_path_query accepts a string first argument by coercing it to jsonb like PG’s implicit cast, and unaliased function-call output columns are now named after the function (SELECT jsonb_path_query(…) yields a column named jsonb_path_query, PG’s rule) instead of ?column?. The pgtest jsonpath corpus file pins everything except its two binary stanzas, which expect CockroachDB’s single-quoted binary wrapping where PostgreSQL (and SecantusDB) send the unquoted text — recorded as an expected divergence.

Added

• The jsonpath type: canonical text rendering, oid 4072, binary codec.

Fixed

• jsonb_path_query('{"a": true}', '$.a') returned NULL — the string document never coerced to jsonb.

• Unaliased function-call columns were named ?column?.

The ltree type

ltree columns and casts now work: stored as a validated dotted label path (alphanumeric/underscore labels), reported on the wire at oid 90010 — the stable placeholder CockroachDB uses for the extension type, mirroring citext’s 90008 — with ParameterDescription inference for INSERT targets and for unknown parameters compared against an ltree column, and the binary parameter/result format PostgreSQL’s extension uses (a version byte followed by the text). The pgtest ltree corpus file pins the whole exchange byte-for-byte and is now green.

Added

• The ltree type: text storage with label validation, oid 90010, binary codec, parameter inference.

Multidimensional arrays keep their base type; JSON[] keeps oid 199

Two array-typing fixes from the pgtest corpus. A nested array constructor (ARRAY[ARRAY[1], ARRAY[2]]) typed as text[] — its binary wire form carried text elements where PG uses ONE array oid per element type regardless of dimensionality, so clients read integer arrays as strings. The tag inference now recurses into nested constructors, and the multidimensional binary encoding carries the element type’s oid and binary cells.

And ::JSON[] now keeps the plain-json array identity end-to-end — parameter descriptions and row descriptions report oid 199 (not jsonb-array’s 3807), and the binary array header carries element oid 114 — extending the earlier scalar ::json → 114 rule to arrays. The one remaining json_array corpus divergence (a hand-spaced json element re-rendering compact where PG echoes the client’s text verbatim) is the documented parsed-storage tradeoff, now recorded as an expected divergence in the gauge.

Fixed

• sql/planner.py: nested array constructors type as their base array type; $1::JSON / $1::JSON[] parameter inference keeps the plain-json oids.

• sql/typemap.py: cast_type_identity reports 199 for ::JSON[].

• pgtest_validation/include_paths.py: json_array recorded as an expected divergence (verbatim-json, deliberate).

Multi-name DROP TABLE and Bind format-code validation

DROP TABLE a, b, c now behaves as the single statement it is in PostgreSQL: one CommandComplete tag instead of one per table, and — without IF EXISTS — every name must resolve before anything is dropped, so a missing table aborts the whole statement with 42P01 and leaves the others intact. Separately, Bind now rejects parameter/result format codes other than 0 (text) and 1 (binary) with PG’s 08P01 protocol violation before BindComplete. Both shapes are pinned by the pgtest errors corpus file, now green (its foreign-key ConstraintName error-field expectations already passed unchanged).

Fixed

• Multi-name DROP TABLE emitted one tag per table and dropped left-to-right before failing on a missing name.

• Invalid Bind format codes were silently accepted.

Nested BEGIN warns like PostgreSQL

Issuing BEGIN inside an already-open transaction block now completes with the BEGIN tag while emitting PostgreSQL’s exact warning — a NoticeResponse with severity WARNING, SQLSTATE 25001, “there is already a transaction in progress”, and PG’s source-identity fields (File xact.c, Routine BeginTransactionBlock) — and the open block survives untouched. The notice plumbing gained optional sqlstate/file/routine fields along the way. The pgtest implicit_txn corpus file reads the warning’s fields byte-for-byte and is now green.

Fixed

• A nested BEGIN in an explicit block completed silently, with no warning.

Reply-shape fidelity: repeated ?column?, array-cast names, int4 series

Three small divergences that pgx’s byte-exact network-usage test caught in one reply:

Real PostgreSQL repeats duplicate output column names verbatim — select 'a', 'b' describes as ?column?, ?column? — where we suffixed them (?column?_2). The evaluated-select path now keeps PG’s names (its row extraction is positional, so uniquifying was never load-bearing there). An unaliased array cast is named after its ELEMENT typname, like PG ('{a}'::text[] yields a column named text). And generate_series with int4-range bounds now yields int4 rows (oid 23, 4-byte binary cells) instead of int8, matching PG’s overload selection — with describe-time (unbound parameter) and execute-time typing agreeing, so a RowDescription never claims int8 over int4 cells.

Fixed

• sql/planner.py: the evaluated-select path stops uniquifying display names; _cast_output_name names array casts after the element type.

• sql/srf.py: generate_series types int4 for int32-range bounds (int8 otherwise), consistently between Describe and Execute.

Malformed parameters get proper SQLSTATEs, never internal errors

Two parameter-decode crashes surfaced by the pgtest byte-exact corpus: a binary array parameter with a structurally-bogus header (bad element oid, missing element data) and an empty-string text parameter cast to an array type (''::JSON[]) both escaped as internal XX000 errors. A malformed parameter is client input, and PG classifies it precisely: truncated binary data is 08P01 (insufficient data left in message), and a bad array literal is 22P02 (malformed array literal). Both paths now raise the right SQLSTATE through the normal error machinery.

Fixed

• sql/pgextended.py: structurally-invalid binary array parameters raise 08P01 instead of an internal error.

• sql/scalar.py: a malformed array-literal cast raises 22P02 instead of an internal error.

• sql/pgwire.py: the test-side build_bind helper accepts binary parameter format codes.

GUC reporting matches PostgreSQL

ParameterStatus messages now follow the command’s CommandComplete (both the simple and extended protocol paths) rather than preceding it, and the values themselves are reported the way PostgreSQL reports them: a numeric SET TIME ZONE becomes a POSIX zone spec (+6 → <+06>-06, -11.5 → <-11:30>+11:30), DateStyle always reads <style>, <order> no matter which order it was written in, and IntervalStyle lowercases. IntervalStyle and is_superuser are now reported GUCs, so a role switch tells the client its superuser status changed — once, on a real change. Transaction-scoped reverts report too: savepoints snapshot GUC state so ROLLBACK TO SAVEPOINT restores and re-reports whatever changed after it, an error that aborts a block reverts its SET LOCALs immediately (with the reports alongside the error, as PG sends them), and every unwind list is ordered case-insensitively by name. SET LOCAL TIME ZONE also honours LOCAL scope instead of leaking past the transaction. The pgtest param_status corpus file pins all of this and is now green.

Added

• IntervalStyle and is_superuser as reported GUCs; savepoint-scoped GUC snapshots.

Fixed

• ParameterStatus was sent before CommandComplete.

• Numeric time-zone offsets were echoed verbatim instead of as POSIX specs.

• DateStyle echoed the written component order.

• SET LOCAL TIME ZONE persisted past the transaction.

Parameter typing follows PostgreSQL’s parse analysis

Prepared-statement parameters now take their types the way PostgreSQL’s parse analysis assigns them. A parameter compared or assigned against a column gets that column’s type — UPDATE t SET ts = $1 WHERE id = $2 describes as timestamptz and uuid rather than text — where previously only citext and ltree columns did this. Conflicting uses are rejected instead of silently succeeding: because a parameter has exactly one type, SELECT lower($1), $1::int raises 42883 (no lower(integer)), a gap in the numbering (SELECT $2 > 0, with no $1) raises 42P18, and a bare parameter as a CASE’s only result raises 42P18 too — while a CASE with a typed sibling branch still resolves. Unaliased column names also follow PG’s FigureColname precedence: a cast takes its operand’s name when it has one (n::int4 is n), falling back to the type name only for nameless operands (2::int8 is int8). The pgtest parameter_description corpus file pins all of it and is now green.

Added

• Column-derived parameter types for assignments and comparisons.

• 42883 / 42P18 rejections for unresolvable parameter typings.

Fixed

• A cast of a column reported the type name instead of the column name.

getCatalogs lists the postgres maintenance database

pg_database now reports the connected database plus postgres — the maintenance database every real PG cluster carries and the one JDBC clients enumerate through. pgjdbc’s getCatalogs asserts both are present and sorted. MongoDB-wire namespace names (e.g. local) are deliberately kept out of the PG catalog — a PG client must never see them as a connectable catalog.

Fixed

• pg_database / getCatalogs includes postgres (deduped when the connection is already to postgres).

DatabaseMetaDataTest setup unblocked: pg_description DML, operator DDL, PK USING INDEX

pg_description now carries COMMENT ON FUNCTION rows (classoid pg_proc), 'name'::regproc resolves a user function to its pg_proc oid (still rendering as the bare name), and UPDATE / DELETE statements targeting pg_description work — persisted as a delta over the derived comment rows. Real Postgres lets a superuser edit the catalog directly; pgjdbc’s DatabaseMetaDataTest setup does exactly that (moving a function comment onto a table’s oid to prove the metadata queries’ classoid guards), and the whole ~90-test class aborted at setup without it. The same setup also needed CREATE OPERATOR / DROP OPERATOR (registered DDL; expression evaluation doesn’t consult user operators) and ALTER TABLE … ADD PRIMARY KEY USING INDEX (promotes an existing unique index to the primary key, taking the index’s name).

Added

• Function-comment rows in pg_description (objoid = pg_proc oid, classoid 1255).

• 'name'::regproc resolves unique user functions to their minted oid, comparing equal to both the oid and the name.

• UPDATE / DELETE against pg_description (suppress + re-emit delta, persisted per database, savepoint-aware via the catalog snapshot set).

• CREATE OPERATOR name (LEFTARG = …, RIGHTARG = …, PROCEDURE = …) and DROP OPERATOR [IF EXISTS] name (left, right).

• ALTER TABLE … ADD PRIMARY KEY USING INDEX idx — validates the index is unique, re-keys rows, reflects the PK constraint under the index’s name in pg_constraint.

• Array-of-composite columns (custom[]) — stored as subdocument lists, reporting the composite’s minted array-companion oid.

Fixed

• OUT-only parameters no longer count toward a function’s signature: f3(IN a int, INOUT b varchar, OUT c timestamptz) is f3(int, varchar) to DROP FUNCTION and callers, matching PG’s identity rule.

pg_get_keywords(), aggregates over SRF row sources, and <> ALL(array)

pg_get_keywords() joins the SRF family (word / catcode / barelabel / catdesc / baredesc; ~100 PG-specific keywords incl. reindex), the record-SRF machinery now handles any column count (it assumed two), and aggregates over an SRF row source work — SELECT string_agg(word, ',') FROM pg_get_keywords() is pgjdbc’s getSQLKeywords query, previously “not supported in this context”. The planner rewrites an aggregate-over-SRF select into the derived-subquery shape the pipeline already handles; scalar subqueries whose FROM is an SRF route through the engine the way ordered/grouped subqueries do; x <> ALL(array) / = ALL(array) evaluate in scalar contexts; and a function-wrapped string_agg (decode(string_agg(…), 'hex')) registers like the plain form instead of erroring.

Added

• pg_get_keywords() SRF; = ALL / <> ALL over array values.

Fixed

• Aggregates over FROM generate_series(…) / other SRF sources (previously only count(*) worked).

• Scalar subqueries with an SRF FROM ((SELECT string_agg(…) FROM generate_series(…))).

• Function-wrapped string_agg in the computed-projection paths.

• Record SRFs with more than two columns.

LISTEN connections no longer desync on a fragmented client write

A connection holding an active LISTEN waits for its next command in short 0.25-second slices so queued notifications flush promptly. That wait was a socket read timeout, which could fire in the middle of reading a frame — after the type byte or partway through the length or payload — and the bytes already read were silently discarded. The next read then re-synchronized on the wrong byte offset and misread the rest of the stream, so a legitimate but slightly slow or fragmented client write (ordinary network jitter, not just a malicious client) got the connection dropped with a spurious protocol error.

The idle-poll wait now uses select to wait for readability and only reads a complete frame once the socket has data, with a blocking recv — so a poll wakeup can never truncate a frame. A frame whose tail is delayed simply blocks the recv until it arrives, exactly as the non-listening default path already did. Async NOTIFY delivery to an idle listener is unchanged.

Fixed

• A LISTEN-holding connection could desync its PostgreSQL wire stream when a frame’s bytes straddled the 0.25s notification-poll window, dropping the connection on ordinary network jitter (#882). The poll now waits with select and reads whole frames only.

plpgsql refcursors: OPEN … FOR, CLOSE, and FETCH by portal name

plpgsql functions can now declare refcursor variables, bind them with OPEN <cursor> FOR <query>, CLOSE them, and return them. An OPEN materializes the query into a session cursor named like PG’s unnamed portals (<unnamed portal N>); the returned name is typed refcursor (oid 1790) on the wire, which is what tells a driver to fetch the result set with FETCH ALL IN "<name>" — the exact round-trip pgjdbc’s CallableStatement performs for {? = call f()} on a refcursor-returning function. FETCH/MOVE now also accept quoted cursor names containing spaces, which the unnamed-portal naming requires.

Added

• plpgsql OPEN <cursor> FOR <query> and CLOSE <cursor> statements; refcursor declarations and returns (the OPEN … FOR EXECUTE form stays unsupported).

• The refcursor type (oid 1790) in result descriptors.

Fixed

• FETCH/MOVE with a double-quoted cursor name containing spaces no longer truncates the name at the last space.

PostgreSQL portal semantics

Named portals now behave like PostgreSQL’s: re-binding a portal name that is still live inside the same explicit transaction raises 42P03 “portal already exists” (the unnamed portal keeps its silent replace), portals are destroyed at transaction end — a suspended portal resumed after its implicit transaction settled at Sync answers 34000 — and DROP TABLE refuses with 55006 while an undrained portal in the session still reads the table, poisoning the block. Interleaved suspended portals (multiple active portals draining alternately under MaxRows) work across the board. Nine of the pgtest multiple_active_portals subtests pin these shapes; the file’s remaining subtests need row-lazy portal execution (tracked in the backlog).

Fixed

• Re-binding a live named portal silently replaced it.

• Suspended portals survived transaction end.

• DROP TABLE succeeded under active portals reading the table.

Portal Execute suspension and row counts

An Execute that delivers exactly its MaxRows now always answers PortalSuspended, even when the portal happens to be exhausted — PostgreSQL cannot know it reached the end until a later Execute fetches past the last row, and clients that loop until CommandComplete depend on that. Each Execute’s CommandComplete also reports the number of rows that Execute returned rather than the portal’s running total, so the final drained Execute reports SELECT 0. The pgtest portals corpus file exercises 1182 of its 1550 lines against this (it stops at a stanza that pins CockroachDB’s CHECK-violation message text where we emit PostgreSQL’s; recorded as an expected divergence).

Fixed

• An Execute delivering exactly MaxRows sent CommandComplete instead of PortalSuspended when no rows remained.

• Portal CommandComplete counted the portal’s total rows, not the rows the Execute delivered.

PyO3 bindings on 0.29, clearing two RUSTSEC advisories

The three PyO3 binding crates (secantus-core-py, secantus-server-py, secantus-storage-py) move from PyO3 0.22 to 0.29, clearing RUSTSEC-2025-0020 (buffer overflow in PyString::from_object, fixed ≥0.24.1) and RUSTSEC-2026-0177 (missing Sync bound on PyCFunction::new_closure, fixed ≥0.29.0). Neither vulnerable API was called anywhere in the tree, so this was dependency-currency debt rather than a reachable vector — but it retires the two advisories the cargo audit CI gate had been baselining, so the gate now runs with an empty ignore list and fails on any newly disclosed advisory.

Because the bindings were already written against PyO3’s Bound<'py, T> smart-pointer API, the migration was mechanical: the deprecated PyBytes::new_bound / get_type_bound methods drop their _bound suffix and Python::allow_threads becomes Python::detach (0.29’s attach/detach terminology). The _secantus_core engine bindings stay byte-for-byte identical to pure Python — the full parity corpus (1705 cases) passes unchanged.

Changed

• PyO3 bumped 0.22 → 0.29 across the three binding crates; cargo audit’s --ignore RUSTSEC-2025-0020 --ignore RUSTSEC-2026-0177 entries removed (#584).

Security

• Cleared RUSTSEC-2025-0020 and RUSTSEC-2026-0177 (both in PyO3 <0.29).

Worker-death root cause: a silently overwritten pytestmark

The remaining xdist worker deaths (“Not properly terminated”, killing the whole suite) traced to a Python footgun: tests/test_rust_binary_ pitr.py assigned pytestmark twice, and the second assignment (the binary-availability skipif) silently discarded the first — the timeout(1200, method="signal") mark added by the original worker- death fix. The file’s disk-bound PITR tests therefore still ran under the global 600s thread-method timeout, whose expiry os._exits the worker mid-test: no signal trace (nothing catchable is delivered), no faulthandler dump, just a dead worker. Diagnosed with the env-gated signal tracer on a quiet machine after co-load theories were falsified. The marks now live in one combined list, and a meta-test walks every test module’s AST rejecting double pytestmark assignment.

Fixed

• test_rust_binary_pitr.py: both marks (signal-method 1200s timeout + skipif) applied via a single pytestmark list.

• New tests/test_meta_pytestmark.py guard against the overwrite pattern anywhere in the suite.

• test_rust_binary_pitr.py tests now schedule on a single xdist worker (xdist_group), so the machine-wide serialization flock never contends within one suite — cross-worker queuing on it starved tests to the fixture’s 480s deadline under full-suite disk contention.

Aggregation pipelines can no longer sidestep RBAC

With access control enabled, aggregate’s privilege check covered only the primary collection — a principal holding nothing but find on one collection could overwrite any namespace in any database via $out or $merge, and read foreign namespaces via the $lookup family, with no grant on the target. Both servers now resolve a pipeline’s secondary-namespace requirements before execution, the same model mongod uses: $out demands insert+remove on its target, $merge insert+update, and $lookup / $graphLookup / $unionWith demand find — with sub-pipelines and $facet branches walked recursively. An unauthorized stage is rejected with Unauthorized (13) before the pipeline touches anything.

configureFailPoint — a server-wide fault-injection lever that could close every client’s connection — previously required no privilege at all under --auth. It now demands a cluster-admin grant on both servers, mirroring mongod’s rule that test commands require a privileged role.

Security

• aggregate $out/$merge could write to (drop and replace) any namespace, and $lookup/$graphLookup/$unionWith could read any same-db namespace, with only a find grant on the primary collection (#783). Both servers now check per-stage privileges pre-execution.

• configureFailPoint was missing from both servers’ RBAC action tables, so any authenticated principal — even one with zero roles — could arm a server-wide DoS failpoint (#806). It now requires a cluster-admin grant (clusterAdmin or root).

reg* pseudo-types on the wire

The registry pseudo-types — regclass, regtype, regproc, regprocedure, regnamespace, regrole — now use PostgreSQL’s oid wire representation: a 4-byte unsigned integer in binary format, and DataTypeSize 4 in RowDescription. Previously a binary parameter of one of these types passed its raw bytes through untouched, so a client sending the standard 4-byte form got the bytes back instead of the numeric value. A payload of any other length is rejected with 08P01, matching PG’s oidrecv. The pgtest oid corpus file pins all of it and is now green.

Fixed

• Binary reg* / oid parameters echoed raw bytes instead of decoding.

• reg* columns reported variable width instead of typlen 4.

Regclass parameter oids and bind-time portal snapshots

A $1::REGCLASS parameter (and the other reg-pseudotype casts — regtype, regproc, regprocedure, regnamespace, regrole, oid) now describes with its real oid (2205 for regclass) in ParameterDescription instead of falling through to text. And a portal bound inside an explicit transaction block now captures its results at Bind, matching PG’s portal-snapshot semantics: a later same-transaction DDL statement (for example ALTER TABLE … RENAME) no longer changes what a held portal returns at Execute. Execution errors still surface at Execute, after BindComplete, and cached-plan revalidation still raises 0A000 at Execute. Both shapes are pinned byte-for-byte by the pgtest bind_and_resolve corpus file, now fully green.

Fixed

• $1::REGCLASS and sibling reg-pseudotype casts report their parameter oids in ParameterDescription (pgtest bind_and_resolve:29).

• Portals bound inside an explicit transaction execute eagerly at Bind (read-only SELECTs only), so later same-transaction DDL is invisible to the held portal (pgtest bind_and_resolve:132).

BEFORE INSERT row triggers, with plpgsql NEW records

CREATE TRIGGER … BEFORE INSERT ON t FOR EACH ROW EXECUTE PROCEDURE fn() now works end-to-end: a plpgsql RETURNS trigger function receives the row as its NEW record, may read fields (new.t), assign them (new.ts := to_tsvector(new.t)), and RETURN NEW — or RETURN NULL to skip the row, exactly PG’s BEFORE-trigger semantics. Triggers fire on every insert path (INSERT and COPY FROM), die with their table, and pg_temp.-qualified trigger functions resolve to the session’s private namespace. This is the tsvector-maintenance shape pgx’s COPY test exercises — the last stable failure in the pgconn package.

Every other trigger shape — AFTER, UPDATE/DELETE events, statement-level — stays faithfully rejected rather than stored-and-never-fired. In support: RETURNS trigger parses (sqlglot rejects the bare pseudo-type; the planner quotes it pre-parse), and to_tsvector now refuses words longer than 2046 characters like real PG, so a 10 kB token yields an empty tsvector instead of a giant lexeme.

Added

• sql/engine.py / sql/catalog.py: CREATE TRIGGER (BEFORE INSERT ROW) with catalog storage, function validation (42P17 for non-trigger functions, 42710 duplicates), and trigger-drops-with-table.

• sql/plpgsql.py: record-field assignment (new.f := …), qualified record reads (new.f), and invoke_trigger with PG’s NEW/NULL-return semantics.

• sql/executor.py: BEFORE INSERT row triggers fire over every planned row in the shared insert path (INSERT and COPY).

Fixed

• sql/fts.py: lexemes longer than 2046 characters are not indexed, like real PG.

ResultSetMetaData fidelity: STRUCT oids, base columns, declared typmods

Three metadata gaps JDBC’s ResultSetMetaData surfaces. A column typed by a table’s row type describes with the table’s rowtype oid (typtype ‘c’ — drivers map it to Types.STRUCT; the generic RECORD oid mapped to OTHER). Bare-column outputs carry their source table oid and attnum even when the SELECT list mixes in computed expressions, and an aliased column resolves to its base column — getBaseColumnName works. Declared type identities ride the descriptors: varchar(n) reports its real oid and length typmod (display size), timestamp(p) its precision, numeric(p,s) its packed precision/scale.

Fixed

• Table-rowtype columns: rowtype oid (typtype ‘c’), not RECORD/2249.

• Base-column table_oid/attnum on evaluated (computed-projection) selects; aliases resolve to their source column.

• varchar/bpchar report their declared oid + typmod; timestamp(p) and numeric(p,s) typmods flow to RowDescription.

• Extended-protocol Describe (PreparedStatement.getMetaData) reports the same base-column identity and typmods as execution — the pipeline Describe path sent 0/0/-1, so prepared-statement metadata lost what the simple protocol carried.

Rust storage: bounded async-oplog buffers and poison-tolerant oplog locks

Two defence-in-depth fixes to the Rust storage engine’s oplog path.

In async-oplog mode a multi-document transaction buffers every statement’s oplog entry — and, when pre-images are enabled, the pre-image bytes — on the transaction handle for its whole lifetime. The entry bytes were already charged to the transaction dirty budget (so an oversized transaction trips TransactionTooLargeForCache), but the pre-image bytes were not, leaving that half of the buffer able to grow unbounded within the 60-second transaction-lifetime window. Pre-image bytes are now charged to the same budget, so the whole buffer is bounded.

Two self.oplog.lock().unwrap() sites in the oplog-prune path used the non-poison-tolerant form that the rest of the codebase had been swept clear of — a panic while that lock was held would have permanently poisoned the single mutex every write’s oplog emission (and therefore change streams) depends on. Both now use the poison-tolerant unwrap_or_else(|e| e.into_inner()), and a source-scanning test guards crates/secantus-storage and crates/secantus-commands against the pattern’s reintroduction.

Fixed

• Async-oplog transactions now charge buffered pre-image bytes to the transaction dirty budget, so pending_async cannot grow the heap without bound (#750).

• The two oplog-prune mutex lock sites are poison-tolerant, matching every other lock on the server-wide oplog mutex; a held-lock panic no longer wedges oplog emission server-wide (#593). A tests/ guard fails CI if the bare .lock().unwrap() pattern returns to either crate.

Large-object writes now honour RBAC and read-only transactions

PostgreSQL’s large-object API reaches the server two ways that both skip the ordinary statement pipeline — the Fastpath sub-protocol (pgjdbc’s LargeObjectManager) and the SQL-callable lo_* scalars — so neither the RBAC gate nor the read-only-transaction check applied to them. A session with no write privilege, or one inside BEGIN READ ONLY, could still create, write, truncate, or unlink large objects.

Mutating Fastpath calls (lo_creat/lo_create/lowrite/lo_truncate/ lo_unlink) now pass the same write-privilege check and read-only gate a table write goes through (large objects are database-scoped, so RBAC is at db granularity — a write action such as insert, which readWrite grants). The SELECT lo_unlink(...) scalar path is likewise classified as a write, so it needs a write grant and is refused inside a read-only transaction. Read calls and ordinary queries are unaffected.

Security

• The Fastpath large-object sub-protocol dispatched lo_* writes with no authorization or read-only-transaction check (#836). Mutating calls are now gated in _handle_fastpath.

• The SQL-callable lo_creat/lo_create/lo_unlink scalars slipped the read-only gate (a bare SELECT reads as non-write) and the write-RBAC check. They are now classified as writes on both paths.

SQL writes can no longer read tables they weren’t granted

The SQL server’s per-statement RBAC authorized only a write statement’s primary target table — the table a subquery, FROM, USING, or AS SELECT clause read from was never checked. A principal holding nothing but an INSERT grant on one table could run INSERT INTO granted SELECT * FROM secret RETURNING * and receive secret’s rows in the response, defeating the finer-grained table/column grant model the SQL layer exists to provide.

Every table a write statement reads as a source now requires its own find (SELECT) grant — db-wide role or table-level GRANT SELECT — across INSERT ... SELECT, UPDATE ... FROM, DELETE ... USING, CREATE TABLE ... AS SELECT, and subqueries. CTE names are excluded (query-local, not base tables) and a self-referential INSERT INTO a SELECT ... FROM a is not charged an extra read grant (the actor already writes a, so no other table leaks). Plain SELECTs are unaffected — their reads, including column-level grants, are authorized exactly as before.

Security

• INSERT ... SELECT, UPDATE ... FROM, DELETE ... USING, and subqueries checked RBAC only on the primary write target, so a write-only grant leaked an unrelated table’s rows through the source clause (#785).

• CREATE TABLE ... AS SELECT authorized only CREATE on the new table, never a read grant on its SELECT source, so a create-capable role could copy out any table (#881).

dropIndex is race-safe and a failed oplog emit no longer freezes change streams

Two storage-layer robustness fixes on the Python server.

drop_index and drop_all_indexes took only the global storage lock, not the per-collection lock that CRUD writers coordinate against — the same gap that was closed for create_index but left open on the drop side. A write landing between an index’s entry-table snapshot and its deletion could survive as an orphaned entry row. Both now take _coll_lock before _lock, the canonical order.

The bare (autocommit) oplog-emit path deregistered its minted sequence range at the end of the method with no surrounding try/finally. Every DDL write goes through it, and if the cursor-write loop or the opportunistic prune raised — a WiredTiger write error or WT_ROLLBACK under contention, both expected — the minted range was never removed from the in-flight set, so the change-stream visible tail clamped at that sequence for the life of the process and change streams server-wide silently stopped advancing. The mint-to-deregister region is now exception-safe.

Fixed

• drop_index / drop_all_indexes now hold the per-collection lock, so a concurrent insert/update/delete can’t leave an orphaned index entry behind (#635).

• A failed bare oplog emit no longer strands its minted sequence range in the in-flight set — the change-stream visible tail recovers instead of freezing server-wide until restart (#714).

Temp-namespacing fallout: bare diag names, self-referencing FKs

The psycopg gauge caught two regressions from the per-session temp-table namespacing. Error diagnostics leaked the pg_temp_<n>. catalog prefix into the TABLE NAME field, where real PG reports the bare relation name (the schema rides in its own field). And a SELF-referencing foreign key inside CREATE TEMP TABLE captured its target by the pre-rewrite bare name — the table doesn’t exist yet when references resolve — so the constraint pointed at a nonexistent relation and never fired, including DEFERRABLE INITIALLY DEFERRED checks at COMMIT.

Diagnostics now report the bare relname, and a self-referencing FK is pointed at the table’s own final (rewritten) name at plan time.

Fixed

• sql/executor.py: NOT NULL / CHECK diagnostics report the bare relation name; the temp schema stays in the schema field.

• sql/planner.py: CREATE TABLE re-points self-referencing FKs at the table’s final name after the temp-namespace rewrite.

Transaction-scoped GUC unwinding, and SAVEPOINT survives the parse guard

Three PostgreSQL-semantics fixes surfaced by re-baselining the pgjdbc gauge after 0.6.0b11. A plain SET inside a transaction block now unwinds on ROLLBACK (and on COMMIT of a failed block) while surviving a successful COMMIT, exactly as PostgreSQL scopes it — and whenever a SET LOCAL or rolled-back SET unwinds a GUC_REPORT parameter, the server re-reports it via ParameterStatus so the client’s cached view reverts too (pgjdbc reads getParameterStatus straight from that cache; all three of its transactionalParameters tests now pass, and the extended-protocol Sync response delivers the revert for autocommit implicit transactions).

The Parse-time bare-expression guard introduced with the pgx parse-error work no longer rejects SAVEPOINT name / RELEASE SAVEPOINT name — sqlglot parses both as a bare alias expression, which the guard mistook for garbage, breaking JDBC’s Connection.setSavepoint outright. Bare START TRANSACTION (no characteristics tail) now opens a block like BEGIN; only the READ ONLY-style suffixed forms parsed before.

Fixed

• Plain SET in a transaction block: kept on COMMIT, unwound on ROLLBACK / failed-block COMMIT, with ParameterStatus re-reports.

• SET LOCAL unwind re-reports GUC_REPORT parameters (Sync-response delivery on the extended protocol).

• SAVEPOINT / RELEASE SAVEPOINT pass the extended protocol’s garbage guard (pgjdbc setSavepoint regression, #876).

• Bare START TRANSACTION opens a transaction block.

Plain json echoes the client’s bytes verbatim; jsonb params validated

The pgtest corpus’ json files pinned four fidelities. PG’s plain json preserves input text byte-for-byte — SELECT $1::JSON returns exactly what the client sent, spacing and all, where jsonb normalises. Casts of recoverable text (a string literal, or a json parameter’s substituted form) now validate the JSON parses and then carry the client’s own text through to the wire, for scalars and for JSON[] array elements alike. This narrows the previously-documented verbatim-json divergence to table-stored json columns only (their parsed storage shape is what powers ->> filter pushdown); parameter echoes and literals now round-trip byte-exact like real PG.

Binary jsonb parameters are validated like PG: an empty payload (no version byte) or an unknown version number is 08P01, and invalid UTF-8 inside is 22021 — all previously accepted silently. And a binary array parameter whose embedded element oid names a KNOWN type that disagrees with the declared array type (a jsonb[] payload bound as json[]) is PG’s 42804 datatype mismatch, while a garbage element oid stays the structural 08P01.

The json and json_array corpus files are fully green — json_array’s expected-divergence entry is removed because the divergence no longer exists.

Fixed

• sql/scalar.py / sql/typemap.py: verbatim JsonText carry-through for plain-JSON casts, scalar and array-element.

• sql/pgextended.py: jsonb binary version/empty/UTF-8 validation; known-type element-oid mismatch → 42804.

• pgtest_validation/include_paths.py: json_array expected-divergence entry removed.

Newer wire-protocol minors negotiate down instead of dropping the connection

A client that opened with any protocol version other than exactly 3.0 had its connection dropped at the startup packet. pgx 5.6+ offers protocol 3.2 by default configuration (MaxProtocolVersion), and real PostgreSQL answers a newer minor with NegotiateProtocolVersion — the newest minor it speaks plus any unrecognized _pq_.* startup options — then both sides continue at the negotiated version.

The server now accepts any major-3 startup, sends NegotiateProtocolVersion as the first response when the client asked for a newer minor (or sent unknown _pq_.* options), and continues the handshake at 3.0. SHOW server_version_num is also supported (150000, matching the advertised 15.0), which protocol-aware clients read to pick expected wire shapes.

Fixed

• sql/pgwire.py: startup packets accept any major-3 protocol and carry the requested version; negotiate_protocol_version builds the ‘v’ message.

• sql/pgserver.py: the handshake answers a newer minor / unknown _pq_.* options with NegotiateProtocolVersion first, like real PG.

• sql/session.py: server_version_num GUC (150000).

Any run-time GUC works as a startup parameter

The startup packet’s parameter list applied only the reportable GUCs (TimeZone, client_encoding and friends); everything else was dropped. Real PostgreSQL applies any run-time GUC sent at startup as the session default — pgx’s target_session_attrs=read-write probe relies on it, shipping default_transaction_read_only=on in the startup packet and expecting SHOW transaction_read_only to answer on so the validator can reject the connection.

All startup parameters are now applied as session GUCs (user / database / options / replication and the _pq_.* protocol options excepted), with the existing TimeZone and client_encoding canonicalization preserved.

Fixed

• sql/pgserver.py: the startup-parameter loop applies every GUC, not just the reportable set.

[0.6.0b11] — 2026-08-15

A silent lost update caught in the act, pipelined batches with real transaction semantics, and query cancellation

The headline of this release is a data-integrity fix that took three CI platforms, a paired A/B sampler, and a deterministic race harness to pin down. Statements a client pipelines before a single Sync now run in one implicit transaction, exactly as PostgreSQL treats them — pgjdbc’s 184-variant BatchFailureTest depends on a failed batch rolling back its earlier inserts, and both batch classes now pass in full. Making that correct exposed a genuine race: an implicit transaction’s commit lands outside the statement lock, and a concurrent autocommit computed update (SET n = n + 1) whose read-compute-write window it straddled would silently overwrite it while still reporting UPDATE 1. The fix runs the whole computed update as one storage snapshot transaction, so a mid-window commit surfaces as a write conflict and the statement retries from a fresh read — proven by a paired fix-on/fix-off sampler catching the loss in the wild on the exact runner conditions that first exposed it.

The PostgreSQL surface took two more long-standing steps. The wire cancel sub-protocol is honoured — a CancelRequest verifies the BackendKeyData secret and interrupts the target’s running statement with PG’s 57014, which is what JDBC’s setQueryTimeout and pgx’s context cancellation lean on. And named prepared statements revalidate their cached plan under DDL, raising cached plan must not change result type with the routine field pgjdbc’s transparent re-prepare matches on — its 1056-variant AutoRollbackTest matrix now passes in full. Around them: dollar-quoted strings and nested block comments parse everywhere, CREATE TABLE AS SELECT works, now() is transaction-stable, LISTEN/NOTIFY pushes to idle connections asynchronously, and per-session temp-table namespacing matches PG’s pg_temp_<n> scheme.

On the MongoDB side, the default getMore batch now fills mongod’s 16MB envelope instead of stopping at 101 documents — a large cursor drain that took dozens of round trips now takes one — and retryable writes are idempotent across reconnects on both servers, with failGetMore carrying the resumable-error label change streams expect.

Prepared statements revalidate their cached plan, like Postgres

A named server-prepared statement whose result shape changed under DDL (a SELECT * after ALTER TABLE ADD COLUMN) now raises PostgreSQL’s cached plan must not change result type (0A000) instead of silently re-planning — and it raises at planning time, before any side effect, so a data-modifying CTE’s INSERT does not run. The ErrorResponse carries ROUTINE=RevalidateCachedQuery, which is the field (not the SQLSTATE) pgjdbc’s transparent re-prepare-and-retry matches on; without it every recoverable case surfaced the raw error. Unnamed statements re-plan per Bind and never raise, matching PG. pgjdbc’s AutoRollbackTest — the 1056-variant autosave × DDL × transaction matrix — now passes in full.

Fixed

• Named prepared statements: result-shape changes under DDL raise 0A000 with the RevalidateCachedQuery routine field, before side effects; first execution captures the plan identity; unnamed statements are exempt.

Query cancellation: the wire CancelRequest is honoured

The PostgreSQL cancel sub-protocol — a client opens a fresh connection and sends the (pid, secret) pair from BackendKeyData to cancel the query running on its main connection — was parsed and silently dropped, and pg_sleep ran as an uninterruptible time.sleep. Drivers lean on this machinery for statement timeouts and context cancellation: pgx sends a CancelRequest whenever a context is cancelled mid-query.

CancelRequest now fires the target session’s cancel event (after verifying the secret), and cancellation points observe it and raise PG’s 57014 canceling statement due to user request while the connection stays fully usable — cancel is not terminate. pg_sleep is such a point in every context (FROM-less and per-row), pg_cancel_backend now cancels instead of closing the target’s connection (matching real PG; pg_terminate_backend still closes), and a cancel that lands while the session is idle is discarded, like real PG. In support of pgx’s liveness-poll shape, pg_stat_activity now reports an extended-protocol statement’s original text with $1 placeholders intact — the bound render inlined parameter values, which both leaked them and made a query like $1 poll match its own row.

Fixed

• sql/pgserver.py / sql/session.py: CancelRequest verifies the BackendKeyData secret and fires the target session’s cancel event; stale cancels are discarded at the next statement’s start.

• sql/functions.py / sql/scalar.py: pg_sleep waits interruptibly and raises 57014 on cancel, in FROM-less and per-row contexts alike (per-row numeric arguments arrive as Decimal128 and are now coerced); pg_cancel_backend cancels the target’s running query without closing its connection.

• sql/pgextended.py: pg_stat_activity.query shows the prepared statement’s original text (placeholders intact), not the parameter-inlined render.

The legacy COPY … BINARY keyword selects the binary format

COPY t FROM STDIN BINARY — the pre-9.0 bare-keyword spelling that pgx still emits — parses as a value-less COPY parameter, which the option reader did not recognise. The format stayed “text”, so the client’s PGCOPY binary stream was fed to the text parser and rejected with 22021 invalid byte sequence for encoding "utf-8". The same applied to COPY t TO STDOUT BINARY.

The bare BINARY keyword now selects the binary format on both COPY directions, riding the existing PGCOPY parse/encode machinery that the WITH (FORMAT binary) spelling already used.

Fixed

• sql/engine.py: _copy_options recognises the value-less BINARY COPY parameter (legacy pre-9.0 syntax) as FORMAT binary for both COPY FROM and COPY TO.

Stray COPY frames no longer poison the connection

Drivers that stream COPY data concurrently with the command — pgx’s CopyFrom pumps CopyData without waiting for the server’s CopyInResponse — kept sending frames after the COPY command itself had already failed (a syntax error, a missing table). The wire server routed those stray frames into the extended-protocol dispatch, answered 08P01 unexpected message type 'd', and left the connection in a discard-until-Sync state that a simple-protocol client can never clear: one failed COPY wedged the connection for good.

Real PostgreSQL accepts and silently discards CopyData, CopyDone, and CopyFail messages that arrive outside a COPY operation, exactly so that this optimistic-streaming pattern stays safe. The wire server now does the same, so a failed COPY reports its error and the connection remains fully usable — including an immediately following valid COPY.

Fixed

• sql/pgserver.py: CopyData / CopyDone / CopyFail frames arriving outside a COPY operation are accepted and discarded, matching PostgreSQL’s PostgresMain behaviour, instead of raising 08P01 and poisoning the extended-protocol state.

Change streams resume when the server says a getMore failed resumably

Drivers decide whether to resume a change stream by looking for the ResumableChangeStreamError label on the error a getMore returns. mongod attaches that label from inside its change-stream machinery, which is why the drivers’ own test suites reach for the failGetMoreAfterCursorCheckout failpoint to provoke one. SecantusDB’s Python server ignored that failpoint entirely — the getMore simply succeeded, no error was raised, and no resume ever happened. The Rust server already handled it, so the two servers disagreed about whether a stream should recover.

The distinction between the two failpoints is deliberate and is now pinned by tests: failGetMoreAfterCursorCheckout with a resumable code resumes the stream, while plain failCommand with the same code does not, because it short-circuits before the change-stream path and carries only the labels the failpoint itself named. Stamping the label unconditionally would silently swallow errors that callers expect to see.

Fixed

• The Python server honours failGetMoreAfterCursorCheckout and stamps ResumableChangeStreamError on the sixteen error codes mongod treats as resumable, matching the Rust server’s table exactly. libmongoc’s change-streams-resume-errorLabels now passes.

generate_series accepts an untyped parameter as a bound

select generate_series(1, $1) — with the parameter sent without a type OID, as clients routinely do — was rejected outright. Nothing upstream had coerced the value, because the wire never said what type it was, so the bound arrived as text and the function refused it as a non-numeric range. Real PostgreSQL infers the parameter’s type from the argument position and reads it as an integer.

The cost of this was out of all proportion to the gap. The pgx driver’s test suite calls that exact query in a helper that runs at the end of 66 of its connection tests, to check the connection is still usable. Every one of those tests failed at the final step, regardless of what it was actually testing — which made a single missing coercion look like sixty-six unrelated bugs.

Fixed

• A numeric-looking text bound (or step) is parsed as a number, so generate_series works with untyped parameters. Bounds that are genuinely not numbers still raise, and numeric bounds are now accepted alongside int and float. The pgx pgconn package goes from 86 failures to 29.

A full scan now drains in two round trips, like mongod

MongoDB’s 101-document cursor default applies only to a query’s first batch — a getMore with no batchSize fills its reply with as many documents as fit in 16MB. Both servers were reusing the 101-document default on every getMore, so draining a collection cost one round trip per 101 documents; a 10,000-document full scan paid ~100 round trips where mongod pays two. That round-trip tax was the entirety of the benchmark’s remaining full-scan gap to mongod: with the corrected default the Rust server’s find full scan lands at parity with mongod on the same box.

Both servers now fill an unspecified getMore batch up to mongod’s 16MB budget (always at least one document, so a drain makes progress), and an explicit batchSize is byte-capped the same way — a batch stops before the document that would push the reply past 16MB, and the cursor stays open for the remainder. Tailable change-stream cursors keep the small incremental default.

Fixed

• getMore without batchSize returned 101 documents per batch on both servers instead of filling the reply to mongod’s 16MB budget — a full collection scan paid count / 101 wire round trips instead of ~2. (crates/secantus-commands/src/cursors.rs, src/secantus/commands.py::_get_more)

• A getMore with an explicit batchSize could assemble a reply of unbounded size; batches are now capped at 16MB of documents with the cursor kept open for the remainder, matching mongod.

INSERT rows can fill a column prefix, like Postgres

INSERT INTO t VALUES (1, 2) into a three-column table is legal PostgreSQL — the row fills a prefix of the columns and the rest take their defaults. The SQL server required an exact arity match, which broke pgjdbc’s rewritten batch inserts (reWriteBatchedInserts=true collapses a repeated INSERT into one multi-VALUES statement without a column list) and several JDBC tests that insert partial rows. Too many expressions is still an error, and an explicit column list still requires an exact match, both with PostgreSQL’s wording. pgjdbc’s BatchedInsertReWriteEnabledTest (60) and TimeTest now pass in full.

Fixed

• A VALUES row shorter than the table’s column list (no explicit column list) fills the leading columns; remaining columns take DEFAULT/NULL.

Dollar quotes, nested comments, CTAS, stable now(), and the JDBC escape functions

A grab-bag of SQL-surface fixes driven by pgjdbc’s StatementTest and PreparedStatementTest. Dollar-quoted string literals now work in any expression position — including digit-bearing tags ($A0$) and tag-vs-content ambiguity ($B$;$b$B$) — and nested block comments (/* /* */ */, which PostgreSQL nests) parse correctly. With standard_conforming_strings = off, plain string literals honour backslash escapes exactly like E'' strings.

now() and CURRENT_TIMESTAMP are now transaction-stable: every call in a statement (and across an explicit transaction block) returns the same instant, as in PostgreSQL — so interval round-trips like extract(second from ((interval '3s' + now()) - now())) are exact. CREATE [TEMP] TABLE … AS SELECT ships with PG’s SELECT <n> command tag, and TRUNCATE resolves schema-qualified and session-temp table names. The scalar-function surface behind JDBC’s {fn …} escapes is complete: the trig family (acos through atan2, hyperbolics, degree variants), replace, numeric-aware power and trunc(x, n), and to_char’s word tokens (Day / Dy / Month / Mon).

Added

• Dollar-quoted string literals ($$…$$, $tag$…$tag$) in expressions.

• CREATE [TEMP] TABLE … AS SELECT (CTAS) with IF NOT EXISTS.

• Trig/hyperbolic scalar functions, atan2, cot, degree variants, replace(text, from, to).

• standard_conforming_strings = off backslash-escape semantics.

Fixed

• Nested block comments mis-tokenized into stray operators.

• now() / CURRENT_TIMESTAMP drifted between calls in one statement.

• power / trunc raised TypeError on numeric-vs-double operand mixes.

• TRUNCATE dropped the schema qualifier, missing temp and schema-qualified tables.

• to_char rendered 'Day' as '5ay' (the D token matched first).

Plain json columns render compact, jsonb keeps its canonical spacing

PostgreSQL treats json and jsonb output differently: a json value’s text is preserved verbatim from input, while jsonb re-renders in its canonical spaced form ({"a": 1, "b": 2}). SecantusDB rendered both from the parsed stored value with jsonb’s spacing, so a client that inserted compact JSON into a json column — which is what every machine-serialized payload looks like — got visibly different bytes back from SELECT and COPY TO.

A plain json (oid 114) column now renders compact ({"abc":"def"}) across the simple protocol, the extended protocol (text and binary formats), and both COPY TO forms, reproducing typical input byte-for-byte; jsonb keeps PG’s canonical spacing. Full verbatim text preservation is deliberately out of scope: the parsed-subdocument storage shape is what lets json-path filters push down to indexed storage lookups, so a hand-spaced json literal still re-renders normalised.

Fixed

• sql/typemap.py / sql/pgserver.py / sql/pgextended.py / sql/engine.py: plain json (oid 114) result columns render compact in DataRows (text + binary), COPY table TO STDOUT, and COPY (SELECT …) TO STDOUT; jsonb rendering is unchanged.

The notification-push check stays off the hot path

The async LISTEN/NOTIFY push decides per message read whether a session is a listener. That check took the server-wide notify-hub lock and scanned the channel registry — on every message, on every connection, a shared lock on the whole server’s hottest path. It now reads a per-session counter maintained by the hub at LISTEN / UNLISTEN time: a plain attribute read, no lock, no scan.

Changed

• sql/pgnotify.py / sql/session.py: is_listening reads Session.listen_count (maintained under the hub lock by listen / unlisten / unlisten_all) instead of locking and scanning the channel registry per message read.

65535-parameter statements work; the 1 MB statement cap was too small

The parser guarded against oversized statements with a 1 MB length cap (a parse-cost DoS guardrail). The premise — “1 MB is far above any real query” — turned out to be false: a statement using the extended protocol’s full 65535 parameters (values ($1::text), … ($65535::text), the shape pgx’s max-parameter tests exercise) is ~1.04 MB of SQL, and real PostgreSQL accepts statements up to its 1 GB message limit. The cap now stands at 16 MB — the same ceiling as the MongoDB document size — which keeps parse cost bounded while accepting every legitimate shape.

ParameterDescription also now wraps its int16 parameter count for 65536-and-up parameters exactly like real PG does (pq_sendint16), instead of crashing the encoder: preparing a 65536-parameter statement succeeds server-side, with the client responsible for the 65535-parameter execution limit, matching PostgreSQL’s behaviour.

Fixed

• sql/planner.py: MAX_SQL_LENGTH raised 1 MB → 16 MB.

• sql/pgwire.py: parameter_description wraps the int16 count for ≥65536 parameters instead of raising struct.error.

Notifications reach idle connections without waiting for a query

LISTEN/NOTIFY delivery was piggybacked on the query cycle: a queued notification was written to the listener’s socket only when that connection next issued a command. A client that just blocks reading the socket — pgx’s WaitForNotification, psycopg’s notifies() — waited forever, because real PostgreSQL pushes notifications to idle connections asynchronously.

Listening sessions now wait for their next command in short slices and flush queued notifications between them, from the connection’s own thread so socket writes stay serialized. Sessions with no LISTENs — the overwhelming default — keep the pure blocking read, so there is no busy-wake cost for ordinary connections, and the idle-in-transaction-session-timeout deadline is preserved across the poll slices.

Fixed

• sql/pgserver.py: the idle read loop pushes queued notifications to listening sessions (~250 ms delivery latency) instead of holding them until the next query cycle.

• sql/pgnotify.py: NotifyHub.is_listening — the poll applies only to sessions with at least one active LISTEN.

Garbage SQL fails at parse time, and multi-statement errors stream partial results

sqlglot is a permissive parser: it reads bad as a column reference and SYNTAX ERROR as an aliased expression, so preparing or executing a non-statement quietly “succeeded” where real PostgreSQL raises a syntax error. A bare expression at the top level is now rejected at parse time with PG’s 42601 syntax error at or near "..." across every entry point — simple protocol, extended-protocol Parse, and pipelined Parse.

A multi-statement simple query (select 1; select 1/0; select 2) also now matches PG’s streaming shape: the completed statements’ results are delivered before the ErrorResponse, and the statements after the error never run. Previously a mid-batch error discarded the already-computed results, so the client saw only the error.

Fixed

• sql/engine.py: top-level bare expressions raise 42601; the expression-shaped commands sqlglot mis-parses the same way (CLOSE, DISCARD, DEALLOCATE) are exempted and keep working.

• sql/pgextended.py: the extended protocol’s Parse applies the same check, so Prepare and pipelined SendPrepare error at parse time like real PG.

• sql/engine.py / sql/pgserver.py: a mid-batch SQLError carries the completed statements’ results, and the wire layer renders them before the ErrorResponse, like real PG.

pg_type grows real array-type rows

Every type that advertises a typarray now has the paired array-type row in pg_catalog.pg_type — _int4 with typelem = 23 and friends, for built-ins, enums, domains, composites and table row types — where before the advertised oid resolved to nothing. The typelem column exists at all now, 'pg_catalog.array_in'::regproc strips the schema the way PostgreSQL renders search-path-visible functions (so pgjdbc’s standard is-array probe matches), and pg_class carries a relacl column (null, single-user server). pgjdbc’s EnumTest enum-array resolution now works; psycopg’s TypeInfo.fetch finds array types by oid.

Added

• Array-type rows in pg_type (typname _<elem>, typelem, typinput = array_in) for every type with a typarray.

• pg_type.typelem, pg_class.relacl columns; ::regproc casts.

The pgjdbc weekly lane’s red now means something

The weekly pgjdbc gauge returned gradle’s raw exit code, and gradle exits non-zero while any test fails — so with ~200 documented standing failures the lane was red by construction and its conclusion carried no signal. The lane now compares the run’s failures against a committed baseline (pgjdbc_validation/baseline.json, seeded from the latest weekly run) and fails only on regression: a failing test the baseline doesn’t list, or a parameterized test failing more times than recorded. Runs with fewer failures stay green and print the newly-passing entries so the baseline can be tightened (python -m pgjdbc_validation.baseline --update).

Changed

• pgjdbc_validation/runner.py exits by baseline comparison, not gradle’s raw code; a gradle failure that produced no test results at all is still a hard failure, and a truncated run still refuses a verdict (124).

Added

• pgjdbc_validation/baseline.py (compare / verdict / --update CLI) and the committed baseline.json (204 standing failures, 2026-08-11 weekly).

Pipelined statements run in one implicit transaction, like Postgres

Statements a client pipelines before a single Sync now run in ONE implicit transaction, exactly as PostgreSQL treats them: a mid-pipeline error rolls back the earlier statements’ effects, a clean Sync commits them, and an explicit BEGIN inside the pipeline takes the transaction over. pgjdbc’s batch semantics depend on this — a failed batch must not leave its earlier inserts behind — and its 184-variant BatchFailureTest and 140-variant BatchExecuteTest both now pass in full. The first statement of a pipeline retries internally on write-write races, so single-statement autocommit behaves exactly as before.

Two describe/planner gaps closed alongside: SELECTs joining derived VALUES tables (no real table anywhere) now Describe their shape instead of answering NoData before emitting rows (a protocol violation pgjdbc rejects), and CrystalReports’ {oj ((( … )))} grouping-paren join chains plan correctly. pgjdbc’s OuterJoinSyntaxTest passes in full.

Fixed

• Extended protocol: implicit transaction from first pipelined statement to Sync (commit / rollback-on-error at Sync; BEGIN takeover; transaction-control and VACUUM-class statements exempt).

• Describe over joins of derived VALUES tables returns the row shape.

• Grouping parens around join chains unwrap through multiple layers, and an aliased VALUES parsed as a Table-wrapped node normalizes.

• A mixed-mode lost-update race the implicit transaction exposed: a pipeline’s Sync-commit could land inside a bare autocommit computed update’s read-compute-write window and be silently overwritten (every statement still reported UPDATE 1). Computed updates outside a transaction block now run their whole read-compute-write as one storage snapshot transaction, so a mid-window commit surfaces as a write conflict and the statement retries from a fresh read. Pinned by a deterministic regression test.

Changed

• The feature is ON by default (SECANTUS_PIPELINE_TXN=0 is an escape hatch).

A retried write no longer applies twice

Every official MongoDB driver retries a failed write automatically, resending it with the same session id and transaction number after a network blip or a write-concern error. Real MongoDB remembers that it already ran the statement and hands back the original answer. SecantusDB did not: it ran the write a second time.

For an insert this was noisy — the retry collided with its own first attempt and raised a duplicate-key error. For anything non-idempotent it was silent and much worse. A retried {$inc: {n: 1}} incremented twice, a retried $push appended twice, and in both cases the client was told exactly one document had been modified. Nothing surfaced an error; the data was simply wrong.

The Python server now keeps a record of each completed retryable write and replays it when the same write arrives again. Only writes that fully took effect are recorded — a failed one must genuinely re-run, or a momentary error would become a permanent one.

Fixed

• Retryable writes are idempotent on the Python server: insert, update, delete and findAndModify carrying a session’s transaction number are executed once, and a retry replays the original reply.

Known limitations

• The Rust server still applies retried writes twice; the same fix has yet to be ported. See tasks/backlog.md §5.

• Records are whole-command, not per-statement, so a partially-failed batch re-runs in full rather than retrying only its missing documents.

• Records expire after 30 minutes, matching MongoDB’s own sweep.

Rust server: renaming a huge collection can no longer wedge the engine

renameCollection re-keyed every row in one WiredTiger statement transaction — the same unbounded-dirty-content livelock class as the (already fixed) one-transaction drop purge, and the last DDL path that could wedge the engine on a collection larger than the cache’s dirty budget. The rename is now a chunked two-phase move that reuses the drop tombstones: tombstone the destination, copy the rows across in bounded transactions (fresh RecordIds preserving insertion order, index entries and unique claims rebuilt per batch), then one small switch transaction registers the destination, unregisters the source, moves the tombstone, and emits the rename oplog entry, and the source’s rows purge in bounded batches. Both crash windows recover through the existing open-time tombstone recovery — on either server — as a plain drop: a crash mid-copy purges the partial destination (the rename never happened); a crash after the switch purges the leftover source (it did). A deterministic regression test renames a collection larger than a small cache — the shape that previously spun forever.

Fixed

• Rust server: renameCollection of a collection larger than the WT cache’s dirty budget livelocked the engine (one unbounded re-key transaction + unbounded write-conflict retry); now a chunked two-phase move with crash-safe tombstone recovery. Inside a user transaction the atomic single-transaction path remains, bounded by the transaction dirty-budget guard. The batched copy also drops the old whole-collection in-memory materialization.

The Rust server stops applying retried writes twice

The Python server learned to recognise a retried write; the Rust server had not, so the two disagreed about something as basic as whether a write happened once or twice. A driver that retried after a network blip — which every official driver does automatically — would silently double a $inc against the Rust server while the Python server handled it correctly.

Both servers now keep the same record and apply the same rules, so a retry replays the original reply rather than re-running the write.

Fixed

• Retryable writes are idempotent on the Rust server, matching the Python server: insert, update, delete and findAndModify carrying a session’s transaction number execute once, and a retry replays the stored reply. Verified over the wire against a release build — a retried $inc leaves 1 where it previously left 2.

Concurrent sessions get their own temp-table namespaces

Postgres gives every backend a private pg_temp_<n> schema, so two open connections can each CREATE TEMPORARY TABLE bar without colliding. SecantusDB’s SQL server shared one namespace: the second concurrent create failed with 42P07 relation "bar" already exists, a real divergence that connection-pooled applications and driver test suites hit immediately.

Each session now allocates its own pg_temp_<n> namespace the first time it creates a temp table. Unqualified names resolve against the session’s temp namespace ahead of public — so a temp table shadows a permanent one of the same name, exactly like real Postgres — and an explicit pg_temp.<name> qualifier resolves to the session’s own namespace (CREATE TABLE pg_temp.t creates a temp table, and CREATE TEMP TABLE aimed at any other schema is rejected with 42P16). COPY and extended-protocol Describe resolve through the same path, temp-table SERIAL sequences are per-session too, and pg_class / information_schema.tables report the bare relation name under its session’s temp schema.

Fixed

• sql/session.py: per-session pg_temp_<n> namespace, allocated lazily on first temp-table creation (pid-seeded so a crashed process’s stale entries can’t collide with a new one’s).

• sql/planner.py: qualify_from_search_path resolves the session temp namespace first (unless pg_temp is placed explicitly on search_path), rewrites pg_temp.<name> to the session’s namespace, and qualify_temp_create_target homes CREATE TEMP TABLE targets there; pg_table_is_visible lowers against bare relnames.

• sql/engine.py: copy_plan and extended-protocol Describe apply the same search-path / temp-namespace resolution as execution.

• sql/executor.py: duplicate temp-table errors name the bare relation (relation "bar" already exists); error diagnostics report the session’s actual pg_temp_<n> schema.

Read and write concerns inside a transaction are refused, not ignored

A transaction’s concerns are settled when it begins: its read concern rides the statement that starts it, and its write concern belongs to the commit. Attaching either to a statement in the middle is meaningless, and real MongoDB says so with an InvalidOptions error. SecantusDB accepted them and quietly did nothing, so a caller could believe a statement had run at a durability or isolation level it never had.

Drivers already refuse this on the client side, which is why no driver test suite ever caught it. It surfaces for anyone issuing raw commands — the one audience with no other way to tell us apart from a real server.

Fixed

• A writeConcern on an in-transaction statement is rejected with InvalidOptions (72) on both servers, matching MongoDB’s wording.

• A readConcern on a statement that continues (rather than starts) a transaction is likewise rejected. The starting statement may still carry one, since that is how a transaction’s read concern is chosen.

JDBC clients get their real time zone

A pgjdbc connection tells the server its JVM time zone through a TimeZone startup parameter — and the PG server dropped it, leaving every JDBC session on UTC. For clients west of Greenwich that shifted date reads back a day (1950-02-07 came back 1950-02-06). Startup GUC parameters are now applied and echoed in the opening ParameterStatus burst, the way PostgreSQL treats them.

Four smaller conformance gaps closed with it: SET timezone = 'gmt-3' now reports the normalized GMT-3 spelling (pgjdbc’s ParameterStatus parser is case-sensitive and silently fell back to UTC on the lowercase echo); POSIX-style zone specs accept minutes (GMT+3:30 is UTC-03:30, pgjdbc’s half-hour-zone test); tstz::text casts render the session-zone offset and tz::text renders PostgreSQL’s +01 spelling; and a BC-era timestamptz literal without an offset is stamped with the session zone’s offset so the stored instant is correct. pgjdbc’s TimezoneTest is now 16/16 and DateTest 192/192 — and this time measured with a fixed tally (the release-note claim that DateTest was already clear traced to an XML-parsing bug in the measurement script, not the server).

Fixed

• TimeZone (and other reportable GUCs) sent as startup parameters are applied to the session and reported in the initial ParameterStatus burst.

• TimeZone values normalize to PostgreSQL’s reported spelling (gmt-3 → GMT-3).

• POSIX GMT/UTC offsets accept minutes and seconds (GMT+3:30).

• timestamptz::text renders the session-zone offset; timetz::text renders whole-hour offsets as +01.

• An out-of-range (BC) timestamptz literal without an offset takes the session zone’s offset instead of UTC.

SELECT * over a USING join merges the join column, like Postgres

SELECT * FROM a JOIN b USING (k) now returns k once — from the left side (the right for RIGHT joins, COALESCE for FULL) — followed by each source’s remaining columns, exactly as PostgreSQL expands it. Previously the star emitted the column once per side, a long-pinned divergence. The fix is one AST rewrite before the USING-to-ON desugar, not a change to every star-expansion path. tbl.* items over joins also work now (they previously crashed with column "*" does not exist), and — matching Postgres — tbl.* does NOT merge; only the bare * does.

Fixed

• Bare * over USING joins merges the join columns (left / right / coalesce per join side; chained USING joins in the all-inner case).

• tbl.* in a join select expands to the table’s columns instead of crashing.

validationLevel finally does something

A collection can tell the server how strictly to apply its validator, and SecantusDB recorded the answer and then ignored it. validationLevel: "off" — an explicit request for no validation at all — still had every write checked. "moderate" behaved like "strict", which defeats the reason the level exists: it lets you attach a validator to a collection that already holds rows predating it, without freezing those rows. Under our behaviour those legacy documents became un-updatable.

Both levels now work, on both servers. off disables validation outright. moderate exempts a document that ALREADY failed the validator from update-time checks, while a document that currently satisfies it is still held to it — so an update can no longer turn a valid document invalid, and inserts are validated as before.

Fixed

• validationLevel: "off" disables document validation on the Python and Rust servers.

• validationLevel: "moderate" exempts already-invalid documents from update-time validation on both servers, on the single-document and multi-document update paths and through findAndModify.

[0.6.0b10] — 2026-08-13

Large objects over Fastpath, a drop that can’t wedge the engine, and TCP_NODELAY everywhere

This release closes two of the oldest gaps a PostgreSQL client could hit. The PG server now implements the Large Object API the way pgjdbc’s LargeObjectManager (and therefore JDBC Blob/Clob) actually drives it — the Fastpath sub-protocol dispatching lo_open / loread / lowrite and friends by their real pg_proc OIDs, backed by chunked sparse storage that joins the session’s transaction. Around it landed the pieces callable statements need: user-defined functions in FROM position typed by their declared return type, Describe that derives result shapes without executing a side-effecting function body, PostgreSQL’s void-argument convention for the JDBC OUT-parameter slot, and plpgsql RAISE. Four pgjdbc test classes that were previously zeroed — BlobTest, BlobTransactionTest, CallableStmtTest, CleanupSavepointsWithFastpathTest — now pass in full. (A claim in the original release notes that DateTest was cleared at 192/192 traced to an XML-parsing bug in the measurement script — the real remaining failures are fixed in the next release’s time-zone slice.)

On the storage side, dropping a very large collection could livelock the Rust server: the whole row purge ran as one WiredTiger transaction, and once its delete volume exceeded the cache’s dirty budget the engine rolled it back for cache pressure and the retry loop re-ran it forever — an eviction storm that survived client disconnects and ignored SIGTERM. Drops are now chunked and two-phase, with a tombstone that makes a crash mid-purge recover cleanly at the next open, on both servers. Both wire servers also now set TCP_NODELAY on every connection, which removes Linux delayed-ACK stalls worth ~40ms per round-trip — one pgjdbc generated-keys batch test went from 41.5 seconds to 0.2. The performance and concurrency reports on secantusdb.com were re-measured from scratch with hardened harnesses, and the Rust server gained a background oplog pruner and a 4G embedded cache default that keep sustained write throughput off the request path.

Document validation you can actually stage, and an admin UI that reaches the rest of the server

Setting validationAction: "warn" on a collection is how you stage a validator against live traffic — mongod logs the violations and stores the document anyway. The Python server accepted the option, reported success, and then rejected the write with code 121 regardless, so the one workflow the setting exists for was the one it broke. collMod had the same shape of problem from the other end: it replied ok: 1 to validationAction and validationLevel and quietly discarded both, leaving callers convinced they had relaxed enforcement that was still fully armed. Both are fixed, on every write path, and the Rust server — which already got this right — is now matched exactly.

The admin UI also stopped hiding features the server has shipped for a while. Collections can be created with validators and capped options, modified with collMod, and renamed (across databases, with an optional dropTarget); custom roles can be created and dropped. The change-stream page gained the options that make it a real debugging tool: fullDocument and fullDocumentBeforeChange, all three start points, and a pipeline filter — plus a Resume from here button on every event, which finally closes a loop the page had left open by offering a “Copy resume token” button with nowhere to paste the token.

Added

• Admin: create / collMod / rename panels on the collection list, and create / drop for custom roles on /roles. Options are entered as one Extended-JSON document, so any option the target server understands works without waiting for a matching form field.

• Admin: fullDocument, fullDocumentBeforeChange, resumeAfter, startAfter, startAtOperationTime and pipeline controls on the change-stream page, with a Resume from here action per event. Options round-trip through the URL, so a shared link reproduces the same stream.

Fixed

• validationAction: "warn" and "off" now accept violating writes instead of rejecting them with DocumentValidationFailure (121), on insert, update, replace and findAndModify alike. Only the default "error" rejects.

• collMod now applies validationAction and validationLevel rather than accepting and discarding them.

• Admin: a rejected change-stream option is reported as a readable error frame instead of a bare websocket close, and the message is no longer overwritten by the disconnect handler that followed it.

Advisory locks now actually exclude

pg_advisory_lock and friends used to be session-local bookkeeping that always granted — two connections could both “hold” the same exclusive lock, so leader-election and migration-fencing patterns (alembic’s lock, cron fencing) silently provided no mutual exclusion. The PG server now runs a server-wide advisory-lock table shared by every connection: exclusive and shared modes with PostgreSQL’s grant rules, re-entrant holds, blocking pg_advisory_lock waits with deadlock detection (40P01 deadlock detected), truthful pg_try_* results, and release on unlock, at transaction end for xact locks, and when a connection ends.

Added

• secantus.sql.pgadvisory.AdvisoryLockHub: the server-wide lock table, attached to every wire session; per-session state remains the pg_locks reflection layer. Pinned by cross-connection tests covering exclusion, blocking waits, shared/exclusive interaction, deadlock detection, transaction-end and connection-teardown release — including a wire-level two-connection psycopg test.

Large batch inserts no longer risk a storage livelock

A single insert message can carry up to 48MB of documents, and the Python server used to write the entire batch — document rows, their full-document oplog entries, and every index entry, roughly two to three times the message bytes — inside one WiredTiger transaction. A transaction’s dirty content is unevictable, so a large enough batch could pin the storage cache past its dirty-stall threshold and livelock the engine: every thread drafted into eviction, nothing evictable, and only the stuck writer’s own commit able to free the cache. This is what wedged the mongo-rust-driver conformance gauge’s large_insert test (35,000 tweet-sized documents) in weekly CI — and once wedged, the server never recovered.

Inserts now commit in bounded chunks of at most 1,000 documents or 4MB per statement transaction, mirroring what real mongod does with its internal insert batches. MongoDB batch inserts are per-document atomic only — a batch has never been all-or-nothing — so the extra commit points are invisible to clients: ordered batches still stop at the first error with the correct per-document index, unordered batches still report every error, and capped collections still never evict documents from the batch being inserted.

Fixed

• secantus.storage.insert: one wire batch no longer runs as one WiredTiger statement transaction; chunks are bounded at 1,000 docs / 4MB with the write-conflict retry scoped per chunk. Reproduced and pinned by test_storage.py::test_large_batch_insert_survives_a_small_cache (35k × 1.1KB documents against a deliberately small 128M cache) plus ordered/unordered cross-chunk semantics tests.

A finished job no longer shows an empty log

The opsboard runs each job as a detached child on a pseudo-terminal and tees everything it prints to a logfile the UI tails. That tee loop asked the pty whether it had anything to read and, on a quiet answer, left as soon as the child had exited. A child that wrote its output and exited inside that window left its bytes sitting in the pty buffer, and leaving discarded them — so the job finished with exit 0 and a completely empty log. The shorter the job, the likelier it was to lose everything it said.

The loop now drains the buffer before it leaves. The comment that used to justify the old behaviour (“a timed-out select with the child reaped means everything has been drained”) was simply untrue, and is gone.

Fixed

• jobkit’s pty tee no longer discards output written in the window between polling the terminal for readability and observing that the child has exited. This is the second race of its kind on this path; the regression test forces the losing interleaving deterministically rather than relying on timing.

Numeric division carries PostgreSQL’s result scale

Dividing numerics now produces the display scale real PostgreSQL derives: SELECT 5.52 / 2.4 answers 2.3000000000000000 (scale 16), 1/3::numeric answers 0.33333333333333333333, and a driver reading getBigDecimal().scale() sees exactly what it would on Postgres. The rule is select_div_scale from Postgres’ own numeric.c, ported into the numeric division path and verified against a live PostgreSQL 14.13 across twenty division cases — every text render byte-identical. Values were already exact after the numeric-exactness work; this closes the last recorded divergence, the displayed scale. Integer division still truncates and float8 mixes still coerce to float8, as before.

Fixed

• secantus.sql: numeric / numeric results are quantized to PG’s derived division scale (typemap.numeric_div, half-away-from-zero rounding). Pinned by tests/test_sql_numeric_div_scale.py — a twenty-case battery whose expectations are byte-exact captures from PostgreSQL 14.13.

The oplog prune moves off the write path entirely

Under sustained write load the oplog reaches its entry cap within seconds, and from then on the opportunistic prune has to delete rows as fast as they arrive. That sweep — a key merge across the shard tables, PITR archiving, per-row deletes — ran inline on whichever thread crossed the cadence: the writer itself in the default mode (measured at roughly a third of the whole insert path under cap pressure), or a drainer in async mode. A dedicated background pruner now owns the sweep in both modes; write paths just set a flag. mongod does the same job on its OplogCapMaintainerThread, for the same reason.

Oplog reads got cheaper alongside: shard tables are created lazily and most never exist, but every oplog merge probed all sixteen plus the legacy table, paying a failed cursor-open per absent table per read. A shard-existence mask seeded at open skips them outright. The embedded Rust Storage library also now defaults to the same 4G WiredTiger cache cap as the daemon and the Python handle (the cache fills lazily, so small test instances stay small) — closing the gap where a library user hit eviction pressure at 256M that the daemon never would.

Measured on the standard concurrency methodology (8 KiB docs, batch 100, sync oplog, interleaved A/B): +7.7% single-writer and +2.9% at eight writers, with every single-writer rep separating cleanly.

Changed

• Rust storage: the opportunistic oplog prune runs on a dedicated background pruner thread (signalled by the write-path cadence, with a 10s retention backstop) instead of inline on writer / drainer threads. Explicit prune_oplog calls are unchanged (synchronous).

• Rust storage: oplog merges (reads, floor, prune scans, archiving) skip shard tables known absent via an existence mask seeded at open.

• Rust storage: Storage::open’s default WiredTiger cache is a 4G cap (was 256M), matching the daemon and the embedded Python handle.

Fresh performance and concurrency numbers — and honest harnesses

Both benchmark reports are re-measured on current code (post TCP_NODELAY, batched sequences, and the parse cache). Per-operation latency: the Rust server runs at 0.7×–2.2× of mongod, with three workloads now beating mongod outright (change-stream drain 0.7×, delete and single-stage $group 0.9×); the Python server spans 1.2×–24×. Write scaling is unchanged in shape and confirmed healthy: the Rust server scales monotonically to 2.5× at eight writers (~93k docs/s fully durable), the async oplog stack reaches ~107k.

Getting trustworthy numbers surfaced two real defects. The concurrency harness handed writer 0 a drop that raced the other writers’ insert stream — a drop starved behind continuous batches for the whole window, died summary-less on SIGTERM, and rows silently averaged a dead writer, manufacturing a 3.4× phantom regression. Writers now target fresh per-row collections (no drops near the measurement), install signal handlers before any I/O, and a missing writer summary fails the run instead of shipping a corrupt row. Second, dropping a heavily-churned collection can wedge the Rust server behind a WiredTiger eviction storm that survives client disconnect and SIGTERM — captured with native stacks and filed in tasks/backlog.md for its own slice.

Added

• bench/latency_chart.py + bench/results/latency.json: the latency chart, markdown table, and site table are now regenerated mechanically from one results file (they were hand-edited SVGs).

The PG server now ships a default idle-in-transaction timeout

A PostgreSQL client that opens a transaction and then goes quiet — a failed test that never rolls back, a leaked pooled connection — used to pin the storage engine’s oldest snapshot indefinitely. WiredTiger then had to keep every subsequent write’s history reachable, so each operation got slower in proportion to total churn until a large statement (a 100k-row TRUNCATE in pgjdbc’s own suite) stalled in page reads and wedged the whole server. That single mechanism was the root cause of the pgjdbc conformance lane’s two-hour hang.

SecantusPGServer now applies a server-config default of 120 seconds for idle_in_transaction_session_timeout (PG ships 0/disabled, but PG’s MVCC degrades gracefully where WiredTiger’s cache-bound history does not). The GUC hierarchy is faithful: a session SET overrides the server default, SET … = 0 opts out entirely, RESET falls back to the server value, and SHOW reports the effective setting. The secantusd-py-pg daemon grows a matching --idle-in-transaction-timeout flag.

Added

• Session.server_gucs — a postgresql.conf-tier defaults layer between session SET overrides and the built-in GUC defaults, honoured by get_setting, SHOW, SHOW ALL, and RESET.

• SecantusPGServer(idle_in_transaction_timeout_s=…) constructor knob and the --idle-in-transaction-timeout daemon flag (default 120s, 0 disables).

Fixed

• An abandoned open transaction on a live connection no longer degrades all later writes without bound (linear-with-churn slowdown, ending in a server-wide page-read stall). Idle-in-transaction sessions are terminated with PG’s own FATAL 25P03 after the timeout, unpinning the snapshot.

PostgreSQL Large Object API over Fastpath

The PG server now implements PostgreSQL’s Large Object surface the way pgjdbc’s LargeObjectManager (and therefore JDBC Blob/Clob) drives it: the Fastpath sub-protocol (‘F’ FunctionCall / ‘V’ FunctionCallResponse) dispatching lo_open / lo_close / loread / lowrite / lo_lseek / lo_creat / lo_create / lo_tell / lo_unlink / lo_truncate and their 64-bit variants by their real pg_proc OIDs, reflected into pg_catalog.pg_proc so drivers can resolve them by name. Object bytes live in chunked, sparse per-database collections (a 2GB lo_truncate extension stores nothing and reads back as zeros, like PG’s own representation), and reads/writes join the session’s open transaction so ROLLBACK discards lowrite data. lo_creat / lo_create / lo_unlink are also SQL-callable.

Around it, the pieces pgjdbc’s CallableStatement and Blob tests need: a user-defined function call in FROM position (select * from f($1) as result, pgjdbc’s rewrite of {? = call f(?)}) evaluates as a one-row source typed by the function’s declared return type; extended-protocol Describe derives that shape from the catalog without executing the function body (a side-effecting UDF in a pgjdbc batch previously ran twice — once at Describe, once at Execute); a NULL parameter declared void (oid 2278) is dropped from the call’s argument list, matching PG’s accommodation of the JDBC OUT-parameter slot; plpgsql gains RAISE (NOTICE/WARNING/etc. flow to the wire as NoticeResponse, EXCEPTION raises P0001); and contrib/lo’s lo_manage trigger DDL is accepted as a recognized no-op. pgjdbc’s BlobTest (28), BlobTransactionTest, CallableStmtTest (14), and CleanupSavepointsWithFastpathTest (10) all pass fully — all four were previously zeroed.

Added

• secantus/sql/largeobjects.py: chunked sparse LO store + Fastpath dispatch with PG’s real pg_proc OIDs; per-session descriptors.

• Fastpath sub-protocol handling in the PG wire server (parse_function_call / function_call_response).

• plpgsql RAISE statement (levels, % formatting, notice delivery over both simple and extended protocol).

• UDF and built-in function calls (now(), version()) as one-row FROM sources, typed by declared return type.

Fixed

• Extended-protocol Describe no longer executes side-effecting UDF bodies to derive the result shape (pgjdbc batched {call f(?)} ran every insert twice).

• Storage.use_user_transaction is re-entrant (nested entry from a SQL-callable lo_creat inside a transactional INSERT no longer breaks the outer transaction).

Read-only transactions are enforced; isolation level round-trips

Writes inside a read-only transaction now fail with PostgreSQL’s 25006 (cannot execute INSERT in a read-only transaction) — whether the read-only-ness came from BEGIN READ ONLY, SET TRANSACTION READ ONLY, or SET SESSION CHARACTERISTICS AS TRANSACTION READ ONLY. DML, DDL, TRUNCATE, MERGE, and GRANT are gated; reads are untouched. And SHOW TRANSACTION ISOLATION LEVEL — the multi-word spelling pgjdbc’s getTransactionIsolation issues verbatim (previously resolving to an unknown GUC and an empty string) — now reports the level a SET SESSION CHARACTERISTICS planted, as does SHOW TIME ZONE.

pgjdbc: ConnectionTest 15/15 (was 12/15), DatabaseMetaData TransactionIsolationTest 14/14 (was 8/14), AutoSaveTransactionSettings 4/6 (was 0/6). Known divergences, none gauge-exercised, recorded in tasks/backlog.md: temp-table writes are also blocked (PG allows them under read-only), and SELECT … FOR UPDATE / nextval() are not yet gated (PG blocks both).

A conformance run that was cut short no longer reads as a clean sweep

The pgjdbc gauge wiped its results directory at startup and only aggregated them once Gradle returned, so a run that hit the wall-clock budget reported zero tests — which looks identical to a flawless run at a glance, and was read that way once. A truncated run now keeps whatever did complete, records that it was cut short, and exits with the conventional timeout status.

The report generator refuses to render a truncated run at all. That is the important half: a partial run’s per-class numbers are every bit as correct as a complete one’s, and only the set of classes is short — so publishing it produces a healthy-looking pass rate quietly measured over less of the suite. There is no caveat that reliably survives being pasted into a summary, so the artifact simply isn’t produced.

The budget itself is raised and made overridable via SECANTUS_PGJDBC_TIMEOUT, because CI hardware runs several times slower than a development machine and the suite legitimately grew once the crashes that used to end tests in milliseconds were fixed.

Added

• SECANTUS_PGJDBC_TIMEOUT overrides the gauge’s Gradle budget (default two hours, up from one).

Fixed

• A timed-out pgjdbc gauge aggregates partial results and reports the run as truncated instead of silently summarising zero tests.

• generate_report refuses to publish a conformance rate computed from a truncated run.

The pgjdbc lane runs sharded — CI wall clock drops from ~70 to ~20 minutes

The pgjdbc conformance gauge runs ~5,500 JUnit tests over a real wire and took the better part of an hour as a single CI job. The lane now fans out as four parallel jobs, each running a deterministic round-robin quarter of the class list (the vendored suite stays byte-for-byte unmodified — only Gradle’s --tests selection differs per shard), and a merge job combines the shards’ JUnit results into the same single conformance report.

The merge enforces the same publish discipline as the truncation guard: a missing, duplicate, or truncated shard refuses the report outright rather than rendering a pass rate measured over part of the suite. only=pgjdbc dispatches select all four shards; a single shard is addressable as only=pgjdbc-1. Locally, invoke validate-pgjdbc is unchanged (one full run), with --shard K/N + validate-pgjdbc-report available for the split flow.

Changed

• .github/validate-lanes.json gains lane groups; the plan job’s filter matches groups as well as names.

• pgjdbc_validation.runner honours SECANTUS_PGJDBC_SHARD=K/N; generate_report merges a complete shard set (refusing anything less); shard-math and merge-guard tests in tests/test_pgjdbc_gauge_truncation.py.

Tailable cursors wait, rewritten resume tokens are refused, and _id leads again

Three unrelated fidelity gaps, each found by mongo-php-library’s suite asking a question no unit test had thought to ask.

A TAILABLE_AWAIT cursor is supposed to park on the server until data arrives or maxAwaitTimeMS expires. SecantusDB’s capped-collection tailables returned in about a fifth of a millisecond, because the wait’s wake condition was keyed on a change-stream position counter that plain tailables never maintain — leaving it permanently satisfied. Clients polling a capped collection were spinning instead of waiting.

A change-stream pipeline may not tamper with an event’s _id: that field is the resume token, and an altered one silently breaks resumption. The server already rejected a pipeline that removed it, but a pipeline that rewrote it passed straight through, and the error surfaced client-side in the driver rather than from the server. Both are fatal now, as they are in mongod.

Finally, a replacement-style update put the preserved _id at the end of the stored document rather than the front. BSON keeps field order on the wire, so the bytes a client got back differed from mongod’s for the same operation — invisible until something compared raw documents, which is exactly what the PHP codec tests do.

Fixed

• getMore on a capped-collection tailable cursor with awaitData now blocks for up to maxTimeMS instead of returning immediately.

• A change-stream pipeline that modifies (not only removes) an event’s _id now fails server-side with ChangeStreamFatalError, matching the Rust server, which already did this.

• Replacement updates place _id first in the resulting document, on both the Python and Rust engines.

The mongo-php-library gauge goes from 42 failures to 1 — and the one that remains is a text-index test, a feature that is explicitly out of scope.

updateMany and deleteMany commit in bounded chunks on the Python server too

The Python server gains the same bounded multi-document write transactions the Rust server just did: updating or deleting everything a broad filter matches no longer runs as a single WiredTiger transaction whose unevictable dirty content grows with the matched set. Chunks re-read their documents inside their own transaction, every document is transformed exactly once even across conflict retries, and single-document writes, upserts, and writes inside multi-document transactions are unchanged. With this, the storage-engine livelock class is closed on both servers across all three surfaces: batch inserts, multi-document updates and deletes, and multi-document transactions.

Fixed

• secantus.storage: update_matching (multi) and delete_matching (unbounded) run chunked statement transactions (≤1000 docs / ≤4MB each) instead of one unbounded transaction — mongod-faithful, since updateMany and deleteMany are per-document write units and documented non-atomic. Pinned by a 35,000-document rewrite + deleteMany against a deliberate 128M cache, exactly-once $inc across chunk boundaries, and unchanged bounded paths.

The pymongo gauges now separate “unsupported” from “broken”

Both pymongo gauges — the sync suite and the AsyncMongoClient one — had a handful of red tests that were never going to go green, because every one of them exercises something SecantusDB deliberately does not implement: hashed indexes, text indexes, and $where, which needs the embedded JavaScript runtime mongod ships and SecantusDB does not. The server already answers each with a faithful “not supported” error; the tests fail because they asked, not because anything is wrong.

Two of them are worth naming precisely, because their titles suggest otherwise. test_maxtime_ms_message and test_to_list_csot_applied are about timeouts, not about $where — they merely use $where to make a query slow enough to time out. Since the query is rejected up front, they never reach the behaviour they are named for. They are recorded as unverified rather than as passing: the gauge tells us nothing about maxTimeMS message shape or CSOT either way.

Changed

• The six pymongo / pymongo-async failures are now classified as expected, each with its rationale, so the summary counts them separately from failures that need a fix. Both gauges report zero actionable failures.

• The async gauge is wired to the shared expected-failures list; it runs the same upstream tests and hit the same gaps under different node IDs.

A 2dsphere index reports its format version

MongoDB stamps every 2dsphere index with the index format version it was built at, and drivers read it back through listIndexes — the PHP library exposes it as IndexInfo::is2dSphere() and $index['2dsphereIndexVersion']. The Rust server left the field off entirely, so a client asking which 2dsphere format an index used got no answer. It now reports version 3, matching both the Python server and MongoDB 3.2 onwards. A 2d index carries no such field and still doesn’t.

Fixed

• listIndexes reports 2dsphereIndexVersion for a 2dsphere index on the Rust server.

A change stream survives a transient error, as it should

A change stream is meant to be durable across a hiccup: when the server hits a transient problem mid-stream, the client is supposed to quietly reconnect and carry on from where it left off. That never happened here, because the server gave the client no way to tell a transient failure from a fatal one.

MongoDB marks the errors a change stream may recover from, and drivers act on that marking alone — never on the error code by itself. The Rust server sent neither the marking nor, in fact, the errors: the mechanism test suites use to provoke a mid-stream failure was accepted and then ignored, so nothing ever went wrong to recover from. Both halves are now in place, so a change stream interrupted by a transient error resumes instead of surfacing the failure to the application.

The distinction MongoDB draws is preserved: an error injected inside the change-stream path is recoverable, while the same error code injected at the command boundary is not, and a fatal error stays fatal.

Fixed

• A change stream resumes after a transient server error rather than failing.

A change stream stops forgetting where it got to

Reading to the end of a change stream threw away the position it had reached. While events were arriving, the stream reported each one’s position faithfully; the moment a read came back empty it replaced that with a bare positional marker — one that named neither the collection nor the document last seen. A client that then reconnected resumed from something less precise than it had already been told, and the token it had been carefully tracking went backwards.

The position now only ever moves forward. An idle stream still advances as the server’s clock does, so a quiet collection doesn’t strand a reader behind the oplog window, but it never rewinds past an event already delivered.

Separately, $currentOp did not report which application a connection belonged to, so tools that look up their own operation — by the appName given in the connection string — found nothing to inspect.

Both were invisible until now: the C++ driver’s suite is the one that covers them, and it had never been run against this server because its tests bind a fixed port.

Fixed

• A change stream’s resume position no longer regresses to a positional marker when a read returns no events.

• $currentOp reports appName and the connection’s driver metadata.

Transactional DDL and consistent scans for the Rust server

The Rust server’s storage engine now runs every namespace-level DDL — createIndexes, dropIndexes (single and "*"), create, drop and rename collection, and dropDatabase — inside the same per-statement WiredTiger transaction machinery its CRUD path has used since the collection-locks work. Registry rows, index entries, collection options and the DDL’s oplog entry now commit or vanish together, so a crash mid-DDL can no longer strand orphan index-entry rows behind a missing registry row. dropDatabase commits one transaction per collection — the same unit real mongod uses — so a huge database can’t blow the storage cache with a single monolithic transaction.

That atomicity also closes the long-standing DDL-vs-scan wobble: a lock-free read racing a drop or rename could previously return a partial result set, splicing rows read before the DDL with the post-DDL view. Reads now run under a seqlock-style namespace-generation check — DDL holds the generation counter odd for its duration, and a scan that observed an odd or moved generation re-runs against the settled state, so every result is a point-in-time answer. A concurrent-stress test pins the new invariant: scans racing drops and renames observe the full collection or none of it, never a partial splice.

Two smaller items land alongside: single-document updates no longer clone the post-image document unless the caller actually asked for it (only findAndModify does — plain updates skip a full per-document clone), and the anyhow dependency moved past RUSTSEC-2026-0190 in all four lockfiles.

Changed

• secantus-storage: create_collection[_with_options] / drop_collection / drop_database / rename_collection / create_index / drop_index / drop_all_indexes wrap their row writes in with_statement_txn + retry_write_conflicts; dropDatabase is per-collection transactions. DDL invoked inside a user (multi-document) transaction now joins it uniformly and rolls back with it (pinned by tests/ddl_txn.rs).

• secantus-storage: update_matching / update_matching_pipeline (and the secantus-commands storage seam’s update_matching_array_filters / update_matching_pipeline) take a want_post_image flag; UpdateOutcome::post_image is captured only for findAndModify, sparing every plain single-doc update a full Document clone.

• anyhow 1.0.102 → 1.0.104 in crates/, secantusdb, secantus-storage and secantus-storage-py lockfiles, clearing the RUSTSEC-2026-0190 unsoundness advisory from the cargo-audit log.

Fixed

• secantus-storage: a lock-free find_matching_with / count_matching racing a renameCollection / dropCollection / dropDatabase / dropIndexes can no longer return a partial result set. Namespace DDL runs under a drop-guarded seqlock generation (ddl_generation_scope, serialised by the global lock) and readers re-run a scan whose generation was odd or moved (bounded, so a DDL storm can’t livelock a reader). Pinned by tests/concurrent_reads.rs::scans_racing_namespace_ddl_are_never_partial.

Rust server: dropping a huge collection can no longer wedge the engine

Dropping a collection ran its whole row purge as one WiredTiger statement transaction. Because collections share the sharded document tables, a drop is a row-by-row purge — and a collection whose delete volume exceeds the cache’s dirty budget got a cache-pressure WT_ROLLBACK, which the write- conflict retry loop re-ran forever while the eviction threads spun. That is the livelock the 2026-08-11 concurrency sweep hit: a drop that sat for 40+ minutes at full CPU, survived client disconnect, and ignored SIGTERM. The same unevictable-dirty-content class was already fixed for batch inserts, updateMany, and deleteMany; drop (and dropDatabase) were the remaining unbounded transactions.

Drops are now chunked and two-phase. A small first transaction unregisters the collection, writes a drop tombstone, and emits the drop oplog entry — after it commits the namespace is gone for every reader and writer. The row purge then runs in bounded 4000-row transactions and finally clears the tombstone. A crash mid-purge is finished at the next open, before any traffic can re-create the name, so leftover rows can never resurface inside a re-created collection. Inside a user transaction, drops keep the old atomic single-transaction path, which the transaction dirty-budget guard (TransactionTooLargeForCache) already bounds. A deterministic regression test drops a collection larger than a deliberately small cache — the exact shape that previously wedged — and a recovery test pins the crash-left tombstone path.

Fixed

• Rust server: drop / dropDatabase of a collection larger than the WT cache’s dirty budget livelocked the engine (unbounded purge transaction + unbounded write-conflict retry); now chunked, with crash-safe tombstone recovery at open.

Added

• table:secantus_drop_tombstones (additive to the shared on-disk layout): pending-drop markers that make the chunked purge crash-safe.

The Rust server matches the Python one on the C and Ruby driver suites

Four more behaviours the Rust server was missing, found by regenerating the gauges rather than reasoning about the code — every one of them was invisible from the source and obvious from a single line of driver output.

serverStatus omitted its connections section entirely, so a driver asking how many connections had been created got no answer. That is what the C driver’s exhaust-cursor tests were failing on all along: they open a cursor and check that a connection was created, and the failure looked for all the world like an exhaust-cursor bug. It took three passes to fix properly — the section was missing, then present but the wrong integer width for a driver that type-checks rather than coerces, then present and correctly typed but always zero, which cannot satisfy a test asserting the count went up. It now reports the server’s real counters.

A capped collection’s $collStats still didn’t report its bounds, because the values arrive as 32-bit integers and were read as 64-bit only. And listIndexes accepted a negative batchSize instead of rejecting it, which is the deliberate failure a Ruby session spec uses to check that errors surface.

Fixed

• serverStatus reports connections (with live counts), opcounters and network.

• $collStats reports maxSize / max for a capped collection regardless of the integer width the driver used.

• listIndexes rejects a negative batchSize rather than accepting it.

The Rust server rejects the specs it should, and owns up to Atlas-only commands

Three behaviours the Python server had and the Rust one didn’t, found by splitting the C and Ruby driver-conformance failures against the Python server’s own results so only the Rust-specific ones remained.

Unknown fields on create and on an index spec were silently accepted rather than rejected. Real MongoDB fails them, and drivers rely on that: three mongo-ruby-driver specs deliberately pass invalid: true and assert the operation fails, which is how a typo in an index option gets caught at the point it is made rather than becoming an index that quietly isn’t what was asked for. Both now answer with the same unknown-field error MongoDB gives.

The Atlas Search index commands — createSearchIndexes, updateSearchIndex, dropSearchIndex — went unanswered entirely, so a client heard “no such command” rather than “this needs Atlas”. A non-Atlas MongoDB registers them and fails them with a message naming Atlas, which is the difference between a driver reporting a missing feature and reporting a broken server. Finally, $collStats reported that a capped collection was capped but not what its bounds were; the max and maxSize fields are now present.

Fixed

• create rejects unknown top-level options, and createIndexes rejects unknown fields on an index spec, with MongoDB’s Location40415.

• createSearchIndexes / updateSearchIndex / dropSearchIndex report CommandNotSupported naming Atlas, instead of CommandNotFound.

• $collStats reports maxSize and max for a capped collection alongside capped.

The Rust server’s batch inserts are chunk-committed too

The Rust storage engine had the same latent hazard the Python server’s large_insert CI wedge exposed: one wire message’s inserts ran as one WiredTiger statement transaction, whose unevictable dirty content could in principle cross the cache’s dirty-stall threshold and livelock the engine. Its 4G embedded default cache kept the worst 48MB-message case comfortably inside the budget — but a daemon configured with a smaller --cache-size had no such protection.

Batch inserts now commit in the same bounded chunks as the Python server (at most 1,000 documents or 4MB per statement transaction), keeping the dirty footprint independent of the client’s batch size on any cache configuration. As on the Python side, MongoDB batch inserts are per-document atomic only, so the commit points are invisible to clients.

Fixed

• secantus-storage: Storage::insert chunks one wire batch into bounded statement transactions (write-conflict retry per chunk; capped-FIFO fresh-key protection spans the whole client batch). Pinned by batch_insert.rs::large_batch_insert_survives_a_small_cache (35k × 1.1KB documents against a deliberate 128M cache) plus ordered/unordered cross-chunk semantics tests.

updateMany and deleteMany commit in bounded chunks on the Rust server

The last unbounded-transaction surface on the Rust server is closed: updating or deleting every document a broad filter matches used to run as a single WiredTiger transaction, whose unevictable dirty content grows with the matched set — the same storage-livelock class the chunked batch inserts and the transaction dirty budget already closed. Multi-document updates and deletes now commit in bounded chunks (at most 1,000 documents or 4MB per statement transaction), each chunk re-reading its documents inside its own transaction so concurrent transaction commits are never overwritten from a stale read, and each document is transformed exactly once even across conflict retries.

Real MongoDB’s updateMany and deleteMany are per-document write units and documented as non-atomic, so the chunk boundaries match its semantics — single-document writes, upserts, and writes inside multi-document transactions are unchanged.

Fixed

• secantus-storage: update_matching (multi) and delete_matching (unbounded) run chunked statement transactions instead of one unbounded transaction. Pinned by multiwrite_chunk.rs: a 35,000-document rewrite and deleteMany against a deliberately small 128M cache, exactly-once $inc across chunk boundaries, and unchanged bounded paths.

The C driver can finally exercise change streams

replSetGetStatus said this server was a standalone while hello, on the very same connection, described a single-node replica set. Real MongoDB is never both, and the disagreement had a cost: the C driver’s test fixture reads the member roster to decide whether replica-set behaviour is available, saw an empty one, and skipped every change-stream test as inapplicable. The strictest wire-protocol suite we run had no change-stream coverage at all.

replSetGetStatus now reports the same one-member primary that hello already advertised. A server started without a replica-set name still answers as a standalone, which is the honest reply for one.

Thirty-two change-stream tests run as a result, and four real defects came out of them: the error for a pipeline that discards the resume token had the wrong message, the error for a malformed pipeline stage had the wrong code and message, and — the substantive one — a pipeline that rewrote the resume token rather than removing it was accepted. MongoDB permits only transformations that leave the token untouched, so a rewritten token now fails the same way a removed one does, instead of reaching the client as a confusing driver-side error.

Fixed

• replSetGetStatus reports a one-member primary roster when a replica-set name is configured, agreeing with hello.

• A change-stream pipeline that modifies the resume token is rejected, not just one that removes it.

• The resume-token and pipeline-stage errors carry MongoDB’s own codes and messages.

A tailing cursor is told why its collection went away

Dropping a collection while a client is tailing it left the client with “cursor not found” — technically true, but it doesn’t say what happened, and a tailing application can’t tell a dropped collection from an expired cursor or a server restart. MongoDB reports that the query plan was killed and names the dropped namespace, and the Rust server now does the same.

The three kinds of cursor a drop can hit are handled differently, matching the Python server. An ordinary cursor is discarded, so the next fetch reports the cursor is gone. A tailing cursor is kept just long enough to explain itself. Change streams are left alone entirely: they already announce a drop through their own invalidation event, and turning that into an error would replace a normal end-of-stream with a failure.

The cursors are also now killed before the collection is removed rather than after — a tail parked waiting for new data is woken by the drop itself, and it has to find the explanation already in place or it goes back to waiting on a collection that no longer exists.

Fixed

• A getMore on a tailable cursor whose collection was dropped reports QueryPlanKilled naming the collection, instead of CursorNotFound.

The Rust server disables Nagle too

Mirror of the Python servers’ TCP_NODELAY fix: the Rust server’s accept loop now calls set_nodelay(true) on every accepted connection, closing the same ~40ms-per-round-trip delayed-ACK stall on Linux that cost pgjdbc’s chatty batch tests a 200x slowdown in CI against the Python server. Best-effort (a failed setsockopt on a dying socket never kills the accept loop), matching mongod’s and PostgreSQL’s own unconditional NODELAY.

The Rust server rejects oversized transactions too

The Rust server now enforces the same transaction dirty budget the Python server gained: a multi-document transaction whose written volume exceeds a cache-derived threshold (about 15% of the storage cache, mirroring real MongoDB’s TransactionTooLargeForCache guard) fails with code 313 before its unevictable content can stall WiredTiger. The error carries no transient label and the transaction aborts, matching mongod. With this, the storage-engine livelock class is closed on both servers: batch inserts commit in bounded chunks, and transactions are bounded by the cache budget.

Added

• secantus-storage: StorageError::TransactionTooLargeForCache + a per-transaction dirty budget (~15% of the configured cache_size, default 4G) enforced across with_user_transaction statements; mapped to mongod’s 313 by the command seam. Pinned by txn_budget.rs::transaction_dirty_budget_guard against a deliberate 128M cache.

Unique indexes hold across a transaction

A unique index on the Rust server could be persuaded to accept two documents with the same value. If one writer was inside a transaction and another was not, each checked for a clash by reading its own snapshot of the data — and neither snapshot showed the other’s pending write. Both were told they were fine, both were written, and the index that was supposed to guarantee uniqueness quietly held a duplicate. Nothing failed, nothing was logged; the damage only became visible later, in the data.

The clash check no longer relies on reading. Each unique value is now claimed in a table keyed by the value itself, so the storage engine refuses the second claim outright, whoever makes it and whenever they started. A writer that arrives while a transaction holds the value now waits for it, exactly as MongoDB does, and is then told the value is taken — or, if the transaction was rolled back, quietly takes it.

Claims are released when the row that owns them is deleted, and cleared when the collection, database or index they belong to is dropped, so a value can always be used again once nothing is using it.

Fixed

• A unique index no longer accepts a duplicate when one of the writers is inside a transaction.

Sequences allocate in batches — bulk SERIAL ingest is 3x faster

Every nextval used to pay a full read plus durable-update transaction against the sequence’s stored document, which dominated bulk-ingest profiles: a 100k-row COPY into a SERIAL table spent roughly three quarters of its time advancing the sequence, capping ingest around 5,000 rows/s. nextval now pre-allocates a batch of 128 values with a single persisted write and hands the rest out from memory under the same statement-write lock that already serialized it — PostgreSQL’s own CACHE mechanism applied server-side. The same COPY now runs at 13,000–15,800 rows/s, and per-statement SERIAL inserts gain about 20%.

Values remain gapless while the server runs (the cache is server-wide, not per-backend). The stored document carries the batch’s high-water mark, so a restart resumes past the unhanded values — the identical gap PostgreSQL’s CACHE and crash semantics produce. setval, ALTER SEQUENCE, DROP, and re-CREATE all discard the prefetched run, so their effects stay immediate.

Changed

• Catalog.sequence_nextval allocates SEQUENCE_ALLOC_BATCH (128) values per persisted write; tests/test_sql_sequences.py pins the gapless run, the high-water persistence and reopen gap, and the invalidation on setval / ALTER … RESTART / re-create paths.

serverStatus tells drivers which storage engine it is

Real driver test suites branch on serverStatus.storageEngine.name before they will even attempt a transaction. SecantusDB never reported the field, so mongo-php-library’s skipIfTransactionsNotSupported helper threw UnexpectedValueException: Could not determine server storage engine and took roughly twenty-seven transaction tests down with it — not because any transaction misbehaved, but because the suite could not establish what it was talking to. One absent sub-document read as dozens of independent failures.

Both servers now report the engine, and the answer is the true one: SecantusDB is WiredTiger-backed, the same engine mongod uses. The persistent flag is wired to the actual store rather than hard-coded, so an :memory: instance reports itself as non-persistent instead of claiming durability it does not have.

Fixed

• serverStatus now carries the storageEngine sub-document (name, supportsCommittedReads, supportsPendingDrops, supportsSnapshotReadConcern, readOnly, persistent, backupCursorOpen) on both the Python and Rust servers. The mongo-php-library gauge goes from 42 failures to 4 over the same 3130 tests.

BC timestamps, and parameters that kept their declared type

A date before year 1 stored in a timestamp without time zone column came back carrying a time-zone offset it should never have had — 0101-01-01 00:00:00+00 BC where Postgres writes 0101-01-01 00:00:00 BC. Ordinary dates already dropped the offset; only the ones outside the range Python can represent kept it.

Separately, a parameter the client declared as timestamp with time zone lost that declaration on the way to the column. Stored into a timestamp column it was treated as though it had been typed out as a literal — offset discarded, clock face kept — instead of being converted through the connection’s zone, so the value moved by the zone’s offset. A client in New York writing midnight got five in the morning back.

Fixed

• A BC or far-future timestamp in a timestamp without time zone column no longer reports an offset.

• A timestamp with time zone parameter keeps its type when stored into a timestamp column, and converts through the session’s zone.

A date written with a time-zone offset and no clock time

1950-02-07 -05 — a calendar date, an offset, and no time of day — is what a JDBC client sends for a date when it has been given a calendar. We read the offset as though it were the time itself, so the value quietly became five in the morning with no zone at all, and a timestamp column stored it that way.

Postgres reads the implicit midnight, and so do we now: the date lands on the day it names, a timestamp column keeps midnight, and a timestamp with time zone column keeps the instant that midnight refers to.

Dates at the very edge of the representable range are handled alongside this. Now that the offset is understood, shifting one of those to UTC can fall off the end of the calendar — the first instant of year 1 is in year zero once you move it west. Those keep their clock face rather than failing.

Fixed

• A date literal carrying a time-zone offset but no time of day is read as midnight at that offset, rather than as a time.

Unquoted identifiers fold to lower case, as Postgres does

SELECT r.table_name FROM (SELECT id AS TABLE_NAME …) r reported that the column did not exist. Postgres lower-cases an unquoted identifier, so writing an alias in upper case and reading it back in lower case names the same column; we compared every spelling exactly, so the two forms were two different names.

Quoted identifiers keep their spelling exactly, which is what quoting is for — "Mixed" and mixed remain different columns.

Matching case always worked, which is why this went unnoticed: code that writes an alias one way and reads it back the same way never trips it. Generated SQL, and anything written in the SQL-standard upper case, does — JDBC’s metadata queries are how it surfaced.

Folding happens once, immediately after parsing, so table names, column references and aliases all agree on one canonical spelling.

Note for existing databases: a table created unquoted with a mixed-case name is now addressed lower-cased, matching what Postgres would have stored in the first place.

Fixed

• An unquoted identifier written in one case and read in another now names the same table, column or alias.

Five ways a SQL connection could drop with “internal error”

Every crash the PostgreSQL front end reported as a bare internal error came from a distinct, small cause, and each one killed the connection rather than returning a message the client could act on. All five are fixed, along with a quadratic cost in parameter binding that made large statements look like hangs.

The protocol’s 16-bit count fields — parameter counts, column counts — were read and written as signed. Postgres allows up to 65535 parameters in a single Bind, and a JDBC driver rewriting a batch into one statement really does send tens of thousands; above 32767 the count came back negative, walked the parse offset backwards, and the connection died. Binding those parameters was also O(N²), because each placeholder was replaced one at a time and the expression library re-parents every sibling on each replacement. A statement with 40000 parameters took over two minutes; it now takes well under a second.

Geometric values had no binary decoder at all, so a point, box or polygon sent in the binary format — which drivers do by default — arrived at the text parser as raw bytes and failed as “no coordinate pairs in geometry”. The line type could not be parsed even as text: its canonical form is three coefficients {A,B,C} rather than coordinate pairs, and the branch that handled it sat after the pair parse it could never survive. time + interval was simply missing, and an interval inside a WHERE clause was pushed down into an aggregation expression that has no interval type, where it surfaced as a $multiply type error. Finally, the catalog builders behind pg_class and friends enumerated the table list twice — once to assign OIDs and once to emit rows — so a table created by another session in between produced a KeyError part-way through a catalog scan.

Added

• Binary parameter decoders for every geometric type: point, lseg, path, box, polygon, line, circle.

• pggeo.line_from_points, converting the two-point spelling of a line to its {A,B,C} canonical text the way Postgres does.

• virtual._tables_with_oids, the single-snapshot accessor catalog builders use instead of enumerating the tables twice.

Fixed

• 16-bit count fields are read and written unsigned, so a Bind carrying more than 32767 parameters no longer drops the connection. Fields that can legitimately be negative — attnum, type size, format codes — stay signed.

• Binding N parameters is linear rather than quadratic.

• line values parse, and an open path keeps its […] spelling through a round trip instead of being rewritten as closed.

• time ± interval returns a time, wrapping into a single day and dropping the month/day components, as Postgres does. timetz ± interval does the same and carries the zone offset through untouched.

• A date compared against a computed timestamp promotes to midnight the way Postgres does, instead of failing to compare ISO text against a datetime.

• '23:59:60'::time carries forward to 24:00:00 rather than storing a second that nothing downstream could parse — which had made time - time fail too.

• An unknown-type operand beside an interval resolves numerically, so $1 * $2::interval works with the typeless parameters JDBC drivers bind.

• Interval arithmetic in a WHERE clause falls back to per-row evaluation instead of lowering to an aggregation expression that cannot express it.

• Catalog builders take one snapshot of the table list, so concurrent DDL no longer aborts a pg_class / pg_attribute / pg_attrdef / pg_description / pg_index scan.

Leap seconds are accepted, and a bad timestamp says what is wrong

'2015-06-30 23:59:60' — a real leap second, and a value Postgres accepts by rolling it forward to the next minute — crashed with an internal error, because Python has no room for a second numbered 60. It now rolls forward the same way, carrying across the minute, day and year boundaries.

The same path had a wider problem: any timestamp that could not be parsed reached the client as an internal error rather than saying so. Even 'not-a-date' did. Unparseable timestamps now report invalid input syntax, naming the value, and the out-of-range near-misses Postgres also rejects — 23:59:61, or a fractional leap second like 23:59:60.5 — are among them.

Fixed

• A :60 leap second in a timestamp literal no longer fails with an internal error.

• An unparseable timestamp reports invalid input syntax instead of an internal error.

numeric is exact again — 0.1 + 0.2 = 0.3 is true

Decimal literals were read as floats, so Postgres’ arbitrary-precision exact numeric behaved like a double. 0.1 + 0.2 = 0.3 answered false, SELECT 0.000000 came back as 0 with its scale discarded, and a value wider than a double silently dropped digits — 12345678901234567890.12345 + 1 returned 1.2345678901234567E+19, which for money-shaped data is corruption rather than rounding.

A literal is now the same exact decimal a numeric column already stored, so values written, computed and read back all agree. Integers are unaffected, and so is integer division.

Comparisons involving a decimal were wrong in a quieter way: the operators could not compare a decimal against an int or a float at all, and answered false instead. Any predicate mixing the two — a column against a decimal expression, a stored numeric against a literal — silently matched nothing.

Added

• typemap.number_literal, the single mapping from a numeric literal to its Postgres type. The planner and the scalar evaluator carried separate copies of this, which is why an earlier attempt at this fix left arithmetic on floats.

• typemap.unwrap_numeric / typemap.negate / typemap.to_decimal128.

Fixed

• Decimal literals are exact and keep their scale, so numeric arithmetic no longer inherits floating-point error or loses digits.

• Comparison operators handle decimals instead of silently answering false.

Repeated SQL statements skip the parser

planner.parse now caches parsed statements by text, handing out fresh copies of the cached trees (Expression.copy() measures 3–4× cheaper than a parse, which profiled at ~29% of embedded statement time). Entries are cached on second sight — the first occurrence only leaves a marker — so workloads of mostly-unique statements (sqllogictest’s corpus, inline-literal DML) pay nothing beyond a dict probe, while repeated text (per-connection re-parse of prepared statements, fixture DDL repeated across thousands of tests) hits from the second occurrence on: +26% embedded statement throughput on repeated-text workloads, no measurable cost on unique-text ones. The cached trees never leave the cache uncopied, so downstream mutation cannot poison them — pinned by tests/test_sql_parse_cache.py alongside second-sight, eviction, and error-path semantics.

_pg_expandarray and composite field access in the select list

information_schema._pg_expandarray(arr) yields one (x, n) record per array element — the value and its 1-based subscript. JDBC’s metadata queries lean on it heavily, selecting it two ways in the same statement: the whole record, and a single field via (…).n. Neither shape was recognised, so those queries failed outright rather than returning primary-key or index information.

Both now work, including the schema-qualified spelling, and the record stays a composite rather than being flattened to text so that a field can still be read from it a level up — which is exactly how the driver uses it, producing the record in a subquery and selecting a field from it in the outer query.

Added

• information_schema._pg_expandarray in the select list, whole or by field.

• (expr).field against a record-returning function.

Fixed

• Set-returning functions are recognised when written with a schema qualification in the select list.

Result columns report the table and column they came from

Every result column described itself as having no source: the table OID and column number that Postgres puts in each field of a row description were sent as zero. JDBC clients use exactly those to map a result column back to the column it was selected from, so an updatable ResultSet could not name the column it was asked to update — it built UPDATE t SET "" = ? and the server rejected it.

Columns selected from a table now carry their source table and position, and they keep it through aliasing and reordering, since the position describes the table rather than the select list. Computed columns still report none, which is what Postgres reports for them.

Fixed

• Updating a row through a JDBC updatable ResultSet no longer fails with column "" does not exist.

SET TIME ZONE actually sets the time zone

Written the two-word way — SET TIME ZONE 'Europe/Dublin' — the statement did nothing at all. It takes no = or TO, so it slipped past the handler that reads name-and-value settings, and SHOW TIME ZONE answered with an empty string because that spelling was not recognised either. A client that pinned its connection’s zone this way, as JDBC drivers do, silently stayed on the default and had no way to tell.

Both spellings now set and report the same setting, DEFAULT resets it, and the change is announced to the client the way other tracked settings are.

Worth being clear about the limit: this makes the setting stick. Values of type timestamp with time zone are still stored and displayed without regard to it — that conversion is a larger piece of work and is written up in the backlog.

Fixed

• SET TIME ZONE <value> sets the TimeZone setting; SHOW TIME ZONE reports it.

timestamp with time zone respects the session’s time zone

A value written without an offset — '2005-01-01 12:00:00' — was read as UTC rather than as local time in the connection’s own time zone, so it was stored at the wrong instant by however far that zone sits from Greenwich. Reading it back showed the same skew, which for a value near midnight moved it to the previous or the following day.

Such a value is now interpreted in the session’s zone, as Postgres does, and displayed back in that zone. A value that arrives carrying its own offset is already unambiguous and is left alone.

Two smaller things came with it. Zone names written with an offset, like GMT+13, previously resolved to nothing and fell back to UTC; they now resolve, keeping the POSIX convention Postgres follows where GMT+13 means thirteen hours behind UTC. And offsets are written the way Postgres writes them — +00 and -05 rather than +00:00, widening to +05:30 only where the minutes matter — which clients that compare the rendered text depend on.

Values of type date and timestamp without time zone are unaffected, as they should be: neither has an instant behind it to move.

Fixed

• A timestamptz written without an offset is interpreted in the session’s time zone instead of UTC, and displayed in that zone.

• Zone settings of the form GMT±N resolve, with Postgres’ sign convention.

• Offsets render in Postgres’ spelling.

A failed statement now aborts its transaction, whatever raised it

Postgres aborts a transaction block on any error: every later statement fails until the block is rolled back. That held for errors raised while running a statement, but not for errors the protocol layer raised on its own — asking for a prepared statement or portal that no longer exists, or a parameter that could not be decoded. Those left the block looking healthy, so work that a client believed had been discarded went on to commit.

Rolling back, including to a savepoint, still recovers the block, and statements outside a transaction are unaffected.

DEALLOCATE ALL also now reports the command tag Postgres reports — DEALLOCATE ALL rather than a bare DEALLOCATE. Drivers watch for that exact tag to learn their server-side statement cache has been discarded and to re-prepare; without it they kept using names the server had already dropped.

The two go together. Aborting the transaction on its own made a JDBC driver’s recovery worse, not better: the block now died where the driver expected to carry on, because it still had no idea its cache was stale.

Fixed

• An error raised by the extended query protocol aborts the open transaction.

• DEALLOCATE ALL reports the DEALLOCATE ALL command tag.

UNIQUE constraints are enforced by the storage engine

A UNIQUE constraint was upheld by looking for a clashing row before writing one. That look happens against the snapshot the writing transaction is reading, so it could not see a value another transaction had just committed, nor one a second writer was inserting at that moment. Either way a duplicate was stored, and the constraint quietly did not hold.

Declaring a constraint now creates the index that enforces it, so the storage engine decides: a value already present is refused whoever wrote it and whenever, and two transactions reaching for the same value collide so that only one keeps it. Adding a constraint to an existing table does the same, and dropping it removes the index.

The SQL rules around NULL are preserved: any number of NULLs satisfy a UNIQUE constraint, and a constraint over several columns does not apply to a row where any of them is NULL.

Constraints declared DEFERRABLE are unchanged. Those are allowed to be violated part-way through a transaction and are judged when it commits — a swap of two values being the usual case — so they continue to be checked at commit rather than on every write.

Fixed

• A UNIQUE constraint no longer admits a duplicate written by a transaction that began before the value was committed, or by two transactions at once.

Unique indexes are enforced by the storage engine

A unique index used to be upheld by looking for a clashing value before writing one. That look happens against the snapshot the writer is reading, which cannot show a value another transaction committed a moment earlier, and cannot show a value a second writer is inserting right now. Both cases stored a duplicate.

Unique indexes now also record each indexed value under a key that is the value itself, so WiredTiger decides. A value already present is refused by the engine whoever wrote it and whenever; two writers reaching for the same value collide and only one keeps it. Creating a unique index over rows that already exist claims their values too.

Nothing else changes: the existing index entries, and every query path that reads them, are untouched, and a database written by an earlier version stays readable.

This covers unique indexes as used through the MongoDB interface. A UNIQUE constraint declared in SQL is still upheld the older way and keeps the same two gaps; the groundwork for closing that is now in place.

Fixed

• A unique index no longer admits a duplicate written by a transaction that began before the value was committed, or by two writers at once.

Both wire servers disable Nagle — a 200x CI stall on chatty round-trips

Neither server set TCP_NODELAY on accepted sockets. Reply paths write small frames back-to-back (a reply then ReadyForQuery, one batch item’s result then the next), and with Nagle enabled the second write waits for the peer’s delayed ACK — roughly 40ms per round trip on Linux, invisible on macOS loopback where ACKs are immediate. pgjdbc’s generated-keys batch tests, which perform 1,000 single-row round trips each, measured 41.5 seconds per test in CI against 0.2 seconds locally from exactly this — about 20 minutes of the pgjdbc lane’s in-test time on ~30 tests. Both servers now set TCP_NODELAY unconditionally on every accepted connection, as mongod and PostgreSQL do.

Oversized transactions are rejected before they can stall the engine

A multi-document transaction’s statements all join a single WiredTiger transaction, whose written content stays unevictable from the storage cache until commit. A client that pushed enough data through one transaction could therefore pin the cache past its dirty threshold and livelock the engine — the same stall class the chunked-insert fix closed for plain batch writes, where chunking cannot apply.

The Python server now enforces the guard real MongoDB has for this exact condition: a transaction whose buffered write volume exceeds a budget derived from the cache size (about 15%, mirroring mongod’s threshold) fails with TransactionTooLargeForCache (code 313). The error carries no TransientTransactionError label — retrying the same oversized transaction would hit the same wall — and, as with any failed in-transaction statement, the transaction is aborted server-side. Transactions under the budget, and plain writes of any size, are unaffected.

Added

• secantus.storage: TransactionTooLargeError + a per-transaction dirty-bytes budget (~15% of cache_size) enforced in the oplog-buffering path; surfaced by the command layer as mongod’s TransactionTooLargeForCache (313, unlabeled). Pinned at both the storage and wire levels against a deliberately small cache.

Unique-key claims no longer survive their table

The storage-backed unique-index enforcement introduced a week ago kept its claims table alive across namespace teardown: dropping a table (or index, or database) left the dropped namespace’s unique-key claims behind, so recreating the table and inserting a previously-used value was falsely rejected as a duplicate. Caught by the weekly conformance sweep — the sqllogictest corpus cycles drop/create with unique indexes constantly — and reproduced in eight lines. Every teardown path now releases the namespace’s claims: drop table, drop index, drop all indexes, drop database, and rename.

Fixed

• secantus.storage: table:secantus_unique_keys rows are purged wherever their index or collection dies. Pinned by per-path regression tests (TestClaimsDieWithTheirNamespace) and the previously-failing index/delete sqllogictest file, which passes again in both protocols.

Idle connections can no longer pin WiredTiger’s transaction horizon

The pgjdbc conformance lane’s two-hour hang had a second, deeper cause beyond the idle-in-transaction timeout shipped previously: a connection whose last statement left its cached WiredTiger session with a positioned cursor held an implicit transaction — invisible to every PostgreSQL-level accounting — and pinned the storage engine’s oldest-transaction horizon while it idled. Every write after that pin kept its history unevictable, so per-operation cost grew linearly with churn until a 100k-row TRUNCATE stalled in page reads and wedged the server. The wedge needed a specific mix of prior traffic to arm, which is why it only appeared mid-way through the full pgjdbc suite.

Both wire servers now call the new Storage.release_thread_snapshot() before blocking for the next client message: WT_SESSION.reset() releases the snapshot and every cursor position in one cheap call, so an idle connection holds nothing by construction. Inside an open user transaction the release is a deliberate no-op — a transaction’s pinned snapshot is its semantics, and the transaction-lifetime / idle-in-transaction timeouts bound that case. The previously-deterministic pgjdbc wedge reproduction now runs clean with the pinned-transaction-range statistic flat at zero.

Added

• Storage.release_thread_snapshot() — releases the calling thread’s WT read snapshot and cursor positions; called by both the PG and Mongo wire servers at the end of every request, before the idle wait.

• tests/test_storage_snapshot_release.py — statistics-backed regression tests: a positioned cursor measurably pins the horizon and the release clears it; the release is a no-op inside a user transaction; a wire-level invariant that an idle PG connection never accumulates a pinned range.

Fixed

• An idle connection’s stale read snapshot no longer degrades all later writes without bound (the pgjdbc CopyLargeFileTest wedge / 2-hour CI lane timeout). The Rust server’s equivalent idle-session behaviour is tracked as a follow-up in tasks/backlog.md.

[0.6.0b9] — 2026-08-01

Async oplog hardened: transactions can no longer leak ghost events

The Rust server’s opt-in async oplog (RustServer(oplog_async=True) / secantusd-rs --oplog-async) closed out its prototype caveats. The important one was a correctness bug the hardening audit caught: a write inside a multi-document transaction handed its oplog entry to the background drainer before the transaction committed, so a rollback left a persisted entry for data that never existed — a phantom change event and a wrong PITR row. Entries now buffer on the transaction handle and reach the drainer only after the commit succeeds; a rolled-back transaction leaves no oplog trace.

Two smaller async-mode gaps closed with it. Reading local.oplog.rs now drains the writer’s queue first, so a client that just got its write acknowledged sees the entry in the oplog view — read-your-own-write, as on mongod. And the opportunistic prune cadence moved from the write path to the drainers themselves: the old trigger could only prune rows already persisted, so a lagging drainer queue escaped every sweep and a burst of writes could leave the oplog over its cap until the next explicit prune. CI gains an async-oplog lane that runs the whole storage suite with the drainer pool live.

Fixed

• Async oplog: multi-document transaction writes minted + enqueued their oplog entries mid-transaction; a rollback persisted a ghost entry (phantom change-stream event, wrong PITR). Entries now buffer on the transaction handle and are minted + enqueued only after a successful commit; rollback / commit-failure / handle drop discard them (crates/secantus-storage/tests/async_txn.rs pins both directions).

• Async oplog: local.oplog.rs reads raced the drainer — an acknowledged write’s entry could be missing from the view. The view read path now flushes the drainer first (no-op in sync mode; skipped inside a user transaction, where mongod forbids reading local anyway).

• Async oplog: the opportunistic prune fired on minted volume but could only doom persisted rows, so drainer-queue lag escaped the sweep and the counter reset deferred the retry a full interval — an oplog temporarily unbounded past oplog_max_entries under bursts. The cadence now lives with the drainers (triggered as rows land).

• Async oplog: an explicit prune_oplog call racing the drainer pruned a timing-dependent subset of acknowledged writes (cap-excess rows still queued escaped the sweep, shifting the pruned count and the resulting oplog floor / PITR segment contents). The public entry point now drains the queue first, so explicit prunes deterministically cover every acknowledged write.

Changed

• tests/oplog_visibility.rs pins oplog_async: Some(false) (it tests the sync in-flight-mint window, which async mode does not have) and storage-crate oplog tests pin the async read-after-write contract with explicit flush_oplog() calls, so the whole suite is meaningful in both modes.

Added

• CI: an async-oplog parity lane in the rust-storage job — cargo test re-run under SECANTUS_OPLOG_ASYNC=1 + SECANTUS_OPLOG_NONLOGGED=1 — the stated precondition for the mode ever becoming a default.

Change streams no longer skip an event that commits mid-lookup

Resuming a change stream from a point in time could permanently miss an event. Mapping a startAtOperationTime to a position scans the committed oplog and then checks that nothing is still in flight below the answer — but it read those two things in the wrong order. A write that committed between the scan and the check produced a stale answer naming the position above it, while the check had already advanced to cover that position, so the answer was accepted and the event was never delivered.

The two reads are now ordered so the in-flight check is sampled first, which is conservative in the safe direction: the visible position only ever moves forward, so an earlier reading can only make the check stricter, never let an unresolved write slip past.

The window was narrow enough to surface only as an intermittent CI failure on Windows, where the coarser scheduling quantum happened to land inside it. It is reproducible on demand once the interleaving is forced, and the regression test does exactly that rather than racing for it. Both the Python and the Rust storage engines carried the same ordering and both are fixed.

Fixed

• startAtOperationTime could resolve to a position past an in-flight write whose entry qualified, permanently skipping that event once it committed.

Set-returning functions work as join and derived-table sources

generate_series, unnest and friends worked only as the sole FROM item. Used anywhere else — joined to a table, or inside a derived table — they failed with relation "" does not exist, an error naming a relation nobody had written. The empty name was the tell: sqlglot models a table function as a table whose name lives in a function node rather than an identifier, so the planner fell through to a catalog lookup for the empty string.

Such a source is now reduced to the base-less shape the engine already knows how to materialize, and handed to the executor as a raw sub-plan. That matters for more than tidiness: the rows are produced at execution time, so an SRF whose arguments read session state — generate_series(1, array_upper(current_schemas(false), 1)) — resolves against the real session instead of being guessed at while planning.

pg_type also gained typinput, the column drivers compare against array_in to decide whether a type is an array.

Together these let the JDBC driver’s type-lookup query run, which had been the single largest source of failures in its conformance suite; the gauge moves from 92.5% to 93.7%.

Added

• pg_catalog.pg_type.typinput.

Fixed

• A set-returning function in JOIN position, or in the body of a derived table, no longer fails with relation "" does not exist.

UNIQUE constraints hold against rows committed after your snapshot

A UNIQUE constraint could be violated from inside a transaction. Enforcement worked by looking for an existing row through the transaction’s own snapshot, which is fixed when the transaction begins — so a row another connection committed after that point was invisible, the check passed, and the duplicate was stored. PostgreSQL rejects the same sequence, because a unique index is checked against committed data even though your reads stay on your snapshot.

Enforcement now consults committed state as well as the transaction’s own view. Both are needed: the transaction’s view sees rows it has inserted itself and respects rows it has deleted, and the committed view sees what everyone else has landed in the meantime.

Autocommit statements were never affected — each is its own short transaction — which is why this went unnoticed. Nothing changes for them, and the extra check costs nothing outside a transaction.

Two narrower cases still get through and are recorded in the backlog: a transaction that has already written to the table before inserting, and two transactions inserting the same value simultaneously. Both are closed properly by making unique index entries collide in the storage engine, which is a change to the on-disk layout.

Added

• Storage.find_matching_committed, a committed-state read for constraint enforcement (not for user-visible reads, which must keep their snapshot).

Fixed

• A UNIQUE constraint no longer accepts a value another transaction committed after the inserting transaction’s snapshot was taken.

[0.6.0b8] — 2026-08-01

A kill -9 crash window in the data-nonlogged mode could lose acknowledged writes — fixed

The opt-in log-only-the-oplog mode (data_nonlogged) wrote its stable marker — the seq recovery replays from — before running the checkpoint it describes. The marker lives in an always-WAL-logged table, so it became crash-durable immediately: a kill -9 landing after the marker’s WAL write but before the checkpoint completed recovered with a marker above what the last checkpoint actually contained, and replay started too high — every acknowledged write between the old checkpoint and the marker was silently lost as a mid-history hole (the oplog rows themselves all survived). The window is a few milliseconds on an idle machine but stretches with checkpoint duration under load, which is how the hard-kill harness caught it live: 2,300 of 7,200 acknowledged documents missing after recovery, with all 7,200 oplog entries present.

Both checkpoint sites (the periodic anchor thread and explicit/close-time stable_checkpoint) now checkpoint first and write the marker after. A crash between the two leaves the old marker, and replay covers extra already-applied entries — the idempotent-replay path that has always existed absorbs exactly that. Stale-marker is safe; eager-marker loses data. The hard-kill harness also gained self-diagnosis: on any future loss it reports whether the missing documents’ oplog entries survived, separating WAL loss from replay-window bugs at a glance.

Fixed

• secantus-storage: stable-marker row written after (not before) its checkpoint in both the periodic checkpoint thread and stable_checkpoint; the recovery floor can now only ever be conservative.

• tests/test_crash_recovery.py: loss assertions carry a diagnosis dict (doc count, oplog row count and tail, whether the first missing id’s oplog entry exists).

Numeric comparisons stop allocating on the hot path

Every numeric comparison in the Rust engines — a find filter’s $gt/$eq/range test, a sort comparator call, an $expr compare — used to build the value’s exact decimal-digit form on the heap (a String plus a digit vector per operand) before comparing. A new allocation-free fast path answers the common int32/int64/double pairs directly, falling back to the digit form only for Decimal128 and for int64↔double pairs beyond ±2^53 (where the engines’ shortest-repr decimal semantics and exact binary comparison can diverge — the boundary is proven and pinned by an edge-corpus equivalence test). Measured on COLLSCAN drains: +11% on an integer range filter, +49% when an integer query bound meets a double field; all seven Rust↔Python parity suites unchanged.

Changed

• secantus-core: numeric::fast_cmp / fast_eq / fast_cmp_numberish answer int/double comparisons without allocating; the query matcher, order::cmp / bson_lt, and the expression engine’s compare/eq paths try them first. Decimal128 and out-of-range pairs keep the exact digit-form path; verdicts are byte-for-byte unchanged.

Change-stream and exhaust replies stop re-encoding every document

The last two survivors of the reply-path materialization (Finding 2) are gone. A change stream’s tailable getMore decoded every event blob into a document and re-encoded it onto the wire — even though the only thing the handler needed from the batch was the last event’s _id for the postBatchResumeToken. And the exhaust streamer round-tripped every batch through an owned document array (plus a full clone of each batch) between pulling it from the cursor registry and framing it. Both now splice the pre-encoded blobs straight onto the wire like the ordinary find/getMore path has since the RecordId era. Measured: change-stream drain +22% (105k → 128k events/s), exhaust-cursor drain +26% (1.20M → 1.52M docs/s).

Changed

• secantus-commands: the tailable getMore hands its event blobs to the wire encoder undecoded; the postBatchResumeToken decodes only the final blob (as it always did).

• secantus-server: the exhaust streamer threads the pre-encoded batch through every moreToCome frame (encode_cursor_reply splice) instead of materialising and cloning it per frame; materialize_batch is gone.

The Python server compiles a projection once per cursor, not once per document

Every projected document re-ran the whole projection front-end: meta validation, spec partitioning, inclusion/exclusion mode detection, and — worst — rebuilding the dotted-path trie from scratch, per row. The spec is constant for a cursor’s lifetime, so all of that now compiles once into a projection plan and only the per-document work runs per document. Alongside it, the expression engine stops shallow-copying the entire document on every $field reference (the copy only existed to satisfy a type annotation — the path walk is read-only), the matcher stops rebuilding a constant frozenset per operator clause, and the pure-Python FNV shard-name hash is memoised. Measured on the Python server: projected find drain +46%, exclusion projection +19%, a $group pipeline +2.8%.

Changed

• secantus.projection: new compile_projection / apply_projection_plan split; apply_projection and the batch path are unchanged in behaviour (all seven Rust↔Python parity suites pass untouched — the Python engine stays the oracle).

• secantus.expressions: $field resolution no longer copies the document; secantus.query: _SIBLING_MODIFIERS hoisted to module scope; secantus.storage: shard-name lookup memoised, projected reads use the batch (compile-once) path.

Write ops decode the collection-options row once, not three times

Every insert decoded the collection-options blob twice (the timeseries check, then the UUID fetch for the oplog entry), and every replace/delete decoded it twice more (UUID, then the pre/post-image flag) — the same tiny BSON row, searched and decoded repeatedly within one operation. A one-decode CollMeta view now feeds all three consumers; the collection UUID stays lazily minted only when the oplog actually needs it, so a server running with the oplog disabled mints exactly as few UUIDs as before. Measured paired A/B on batch inserts into a two-index collection: +2.3% (5/5 positive pairs).

Changed

• secantus-storage: coll_meta / meta_uuid replace the per-op is_timeseries + collection_uuid + pre_post_images_enabled call chains on the insert/replace/delete paths. Behaviour is unchanged — same facts, one decode.

The benchmark page now covers the paths that differentiate — and the PGO profile catches up

The published nine-workload latency table gains two rows the old six-row table never measured: a filtered collection scan (the per-document compare path — the one the new allocation-free numeric fast path accelerates; the unfiltered scan and the indexed range never touch it) and a change-stream drain, where the Rust server now clocks 0.8× of mongod — faster than mongod at its own change streams — after the reply path stopped re-encoding event blobs. The aggregate multi-stage workload joins the published table too. The committed PGO profile is regenerated on the post-review hot paths (a stale profile silently forfeits its 12–19%), and every surface that quotes the ×mongod ranges — the benchmark page, the website performance page, the Rust-server docs, the README — is re-baselined from the same fresh five-rep run.

Changed

• bench/compare_servers.py: new find_filtered_scan and change_stream_drain workloads; the change-stream reference spawns a single-node replica-set mongod (its change streams require one) while every other row keeps the standalone reference; the Rust server arm advertises the replica-set persona to match the Python server.

• crates/pgo/_secantus_server.profdata.tar.gz: retrained via invoke rust-pgo-refresh on the post-micro-opt hot paths.

• docs/benchmark.md, docs-rust/index.md, README.md, website performance page: nine-row table + refreshed charts and ×mongod ranges (Rust ~0.8×–2.3×; three rows beat mongod outright).

Concurrency graphs are now generated, refreshed per release

The N-writer scaling charts on secantusdb.com/performance and in the docs’ concurrency deep-dive are no longer hand-authored SVG. A new invoke concurrency-refresh task re-measures all four series (Python server, Rust server, Rust async stack, mongod) with bench.concurrency — now able to drive the async-oplog stack directly (--server rust-async), take medians over interleaved runs (--runs), and write machine-readable results (--json) — and bench.concurrency_chart regenerates the chart and data-table blocks in both surfaces from those results. The committed results live at bench/results/concurrency.json, and a test pins the committed charts to exactly what that file renders to, so the graphs can no longer silently drift from the measurements. The refresh is part of the per-release website update.

Added

• bench.concurrency: --server rust-async (async + non-logged oplog stack), --runs N interleaved-median sweeps, and --json PATH structured output; --server all now sweeps four servers.

• bench.concurrency_chart: renders the website and docs concurrency chart + table blocks from the results JSON into marker-delimited regions.

• invoke concurrency-refresh: benchmark + regenerate in one step (--skip-bench re-renders from the committed results).

• tests/test_concurrency_chart.py: pins the render/replace logic and fails if the committed charts are stale relative to the committed results JSON.

The wire-protocol gauge lands — CockroachDB’s pgtest corpus runs verbatim

The SQL server’s conformance portfolio gains its strictest instrument: G3, the pgwire message-level gauge. invoke validate-pgtest drives CockroachDB’s pkg/sql/pgwire/testdata/pgtest corpus — ~54 datadriven files of raw Parse/Bind/Describe/Execute/COPY/error exchanges with byte-exact expected responses — using CockroachDB’s own pkg/testutils/pgtest runner, completely unmodified. It is the SQL analogue of the mongo-c-driver gauge: where the driver gauges tolerate server slop, this one asserts the framing itself.

The monorepo problem is solved by not vendoring at all: both corpus and runner are fetched at a pinned commit through a sparse, blob-filtered clone (about 25 MB, cached) at gauge time — the same fetch-at-runtime pattern as the sqllogictest runner’s cargo install — which also keeps the CockroachDB Software License outside the repository tree. The only committed Go code is a thin go test driver and a ten-line shim for one internal helper the runner imports. SecantusDB presents as non-CockroachDB, so the corpus’ crdb_only exchanges skip themselves.

The opening baseline is 8 of 58 files — honest and low by design, since every file stops at its first byte-level mismatch; the number climbs cluster-by-cluster the way the psycopg gauge went from 42% to 91%. The first finding is already fixed: an unaliased cast’s output column is now named after the type’s typname (SELECT 2::int8 → column int8), where it previously reported ?column?.

Added

• pgtest_validation/ (pinned-commit sparse fetch, verbatim upstream runner staging, Go driver module, report generator), invoke validate-pgtest, weekly validate.yml row sharing the Go toolchain step.

Fixed

• sql/planner.py: unaliased top-level cast projections are named after the cast target’s typname like real PG, across the constant, single-table, grouped, and RETURNING paths.

The JDBC driver’s own suite now measures the SQL server — and one fix moved it nine points

pgjdbc, the official PostgreSQL JDBC driver, joins the portfolio as the G5 gauge: invoke validate-pgjdbc runs the driver’s own test suite — unmodified, from a vendored submodule at REL42.7.13 — against a daemon SecantusDB server. Targeting uses pgjdbc’s stock build.local.properties mechanism, which the project itself gitignores, so pointing the suite at us leaves the vendored tree pristine. Scope opens at the jdbc2 core package (75 test classes, 5,500-odd tests) and grows package by package.

The opening baseline was 4,462 passed / 1,068 failed (80.7%) — and half of those failures were a single protocol bug. Describe answered NoData for any query with a CTE, then Execute sent DataRows anyway; pgjdbc refuses that outright with “Received resultset tuples, but no field structure for them”, and a data-modifying CTE (WITH x AS (INSERT … RETURNING …) SELECT * FROM x) tripped it every time. Describe now derives a CTE query’s shape by planning the outer SELECT against synthetic tables standing in for each CTE — the data-modifying ones described from their RETURNING clause, nothing executed, no side effects. That one fix took the gauge to 4,962 passed / 568 failed (89.7%).

This is the third distinct form of the same protocol violation the SQL gauges have surfaced this week (computed WHERE clauses, views, now CTEs), each caught by a different client — which is exactly the argument for running several strict drivers rather than one.

Added

• pgjdbc_validation/ (runner with JDK-21 discovery, per-class enumeration so exclusions are effective, JUnit-XML aggregation, report generator), vendor/pgjdbc submodule at REL42.7.13, invoke validate-pgjdbc, and a weekly validate.yml row reusing the java/kotlin JDK + Gradle cache steps.

Fixed

• sql/engine.py: extended-protocol Describe reported NoData for every CTE query while Execute emitted rows — a protocol violation that made data-modifying CTEs unusable from strict clients.

The sqllogictest gauge grows a second protocol lane — and catches a wire bug doing it

invoke validate-slt now runs every corpus file through both PostgreSQL wire protocols: sqllogictest-rs’s postgres engine (simple query) and postgres-extended (Parse/Bind/Execute), completing the two-lane design the gauge plan called for. 52 of 60 lane-files pass; the only failures are the four declared SQLite-vs-Postgres divergences, doubled across lanes.

The new lane immediately earned its keep: a SELECT from a view over the extended protocol answered Describe with NoData and then sent DataRows — a protocol violation strict libpq clients reject outright. Describe now expands view references (on a copy, leaving the stored prepared statement pristine) so the declared row shape always precedes the rows.

Added

• slt_validation/: the postgres-extended lane (both engines per include file, lane-tagged report).

Fixed

• sql/engine.py: extended-protocol Describe of a SELECT-from-view answered NoData while Execute emitted DataRows.

pgbench and psql run clean — the SQL server’s stress smoke lands

Unmodified pgbench now drives SecantusDB end to end: the full init cycle (multi-table DROP TABLE, table creation, a 100,000-row client-side COPY, VACUUM, and ALTER TABLE … ADD PRIMARY KEY), then the TPC-B transaction script in all three protocol modes — simple, extended, and prepared — plus a concurrent select-only lane. psql’s catalog family (\dt, \d table, \di, \l, \dn) runs without error. All of it is packaged as invoke sql-stress (the G7 gauge of the SQL conformance portfolio), weekly in CI, with the invariant that any error or dropped connection is a bug.

Getting there closed a string of real gaps: multi-name DROP TABLE a, b, c; VACUUM accepted; ALTER TABLE ADD PRIMARY KEY as a true migration (validates NOT NULL and uniqueness, then re-keys every existing row onto the column value); PG’s unknown-type literal coercion in arithmetic (abalance + $1 with an untyped text parameter — how pgbench binds everything); the OPERATOR(pg_catalog.~) regex spelling with COLLATE; schema-qualified array_to_string; comma-join scalar subqueries (psql’s collation lookup); literal IN lists in JOIN ON; and the pg_catalog surface psql reads — owner/toast/statistics columns on pg_class, encoding and collation on pg_database, pg_policy, and present-but-empty pg_trigger / pg_statistic_ext / pg_inherits / pg_rewrite / publication catalogs.

One documented boundary: under concurrent writers to the same row, WiredTiger’s optimistic concurrency surfaces a PG-SERIALIZABLE-style 40001 serialization failure rather than blocking like READ COMMITTED. Retry-capable clients handle this normally; the smoke keeps its write lanes single-client and the retry-semantics question is tracked in the backlog.

Added

• sqlstress_validation/ + invoke sql-stress + weekly validate.yml row (installs postgresql-contrib for pgbench/psql).

• sql/executor.py: ALTER TABLE … ADD [CONSTRAINT] PRIMARY KEY with row re-keying and 23502/23505/42P16 validation.

• sql/planner.py + sql/engine.py: multi-name DROP TABLE; VACUUM; OPERATOR(pg_catalog.~ / ~*) (+ negations) rewritten to regex matches; literal IN lists in join ON.

• sql/scalar.py: unknown-text numeric coercion in arithmetic (22P02 on garbage), pg_get_userbyid, pg_encoding_to_char, schema-qualified array_to_string, comma-join (cartesian) scalar subqueries.

• sql/virtual.py: pg_class owner/toast/check/flag columns, pg_database encoding/collation/ACL, pg_namespace owners, pg_index validity flags, pg_policy, and empty pg_trigger / pg_statistic_ext / pg_inherits / pg_rewrite / pg_publication* catalogs.

Chasing the JDBC driver’s failures turns up six real server bugs

Working the pgjdbc conformance gauge’s failure clusters took it from 89.7% to 92.4% of the driver’s jdbc2 suite — but the point is what the failures were hiding. Six of them were genuine correctness bugs, two of which produced wrong answers rather than errors.

The starkest: an ungrouped aggregate returned no rows when its WHERE excluded everything. SELECT count(*) WHERE 1=2 answered “no rows” where PostgreSQL answers 0, and SELECT max(3) WHERE 1=2 answered nothing where PostgreSQL answers one NULL row. This was verified against a real PostgreSQL 14.13 rather than from memory — and it means SELECT 0/count(*) WHERE 1=2 now raises division-by-zero, which is precisely how pgjdbc’s batch tests inject a runtime failure. A pre-existing test had encoded the wrong behaviour; it has been corrected with the verification noted in place.

Also fixed: BC-era timestamps are accepted with the era marker either side of the zone offset (pgjdbc sends 0101-01-01 BC +00, PostgreSQL’s datetime input is field-order flexible), and a BC value stored in a date column no longer silently loses its era and becomes an AD date. time and timetz accept a full timestamp and keep the time-of-day, as PostgreSQL does. Multi-dimensional enum arrays (flag[][]) no longer crash the server, and nested arrays render with nested braces instead of quoted JSON. x = ANY(…) works in per-row evaluation, current_schemas() is implemented, and ALTER DATABASE … SET stores database-level GUC defaults applied to new sessions with PostgreSQL’s precedence. Finally, extended-protocol Describe no longer needs parameter values: SELECT $1::inet has a shape fixed by its cast target.

Added

• sql/scalar.py: current_schemas(include_implicit), x = ANY(<array>) in per-row evaluation, pg_encoding_to_char.

• sql/engine.py + sql/catalog.py + sql/session.py: ALTER DATABASE … SET / RESET [ALL] database-level GUC defaults, merged into new sessions (explicit session settings still win).

• sql/engine.py: value-free Describe fallback for cast projections over unbound parameters.

Fixed

• sql/planner.py: an ungrouped aggregate now yields exactly one row when the WHERE excludes the implicit row (COUNT 0, others NULL) — previously zero rows, a wrong answer. Verified against PostgreSQL 14.13.

• sql/datetimes.py: the BC era marker is accepted before or after a zone offset; a BC/out-of-range value with a time part keeps its era in a date column (previously became an AD date); time / timetz accept a full timestamp and a trailing offset.

• sql/scalar.py: multi-dimensional enum arrays (flag[][]) raised an internal error; labels are now validated at every depth.

• sql/typemap.py: nested array text rendering inferred its element type from the outer list, rendering sub-arrays as quoted JSON instead of nested braces.

Unqualified SQL names resolve through search_path

The PostgreSQL front end resolved an unqualified relation name to the public schema and nowhere else. SET search_path TO reporting followed by SELECT * FROM orders raised relation "orders" does not exist even though reporting.orders was right there — the schema was addressable only by spelling it out on every reference. Unqualified names now walk search_path in order and bind to the first schema that holds them, which is what every tool that sets a search path and then writes plain SQL expects.

Resolution only consults the path when the bare name misses, so a relation that already resolved is never redirected, and the rewrite happens on the statement itself — a read and a write of the same unqualified name are guaranteed to address the same schema. CREATE TABLE is deliberately exempt: Postgres creates into the path’s first schema rather than binding to a same-named relation further along it.

Separately, a fixed wrong answer: a nested SELECT inside a FROM-less one had its aggregates folded against the outer statement’s single implicit row, so SELECT (SELECT count(*) FROM t) reported 1 for any table regardless of its contents, and the other aggregates raised column … does not exist. The subquery now aggregates over its own rows.

Added

• Session.search_path, the resolution-ordered schema list ("$user" collapsed to public, repeats dropped). Session.current_schema is now its first entry.

• planner.qualify_from_search_path, which binds unqualified table references to a search_path schema in place, skipping CTE names and CREATE TABLE / CREATE VIEW targets.

Fixed

• Unqualified relation names now resolve through every search_path entry instead of only public.

• Aggregates in a subquery nested inside a FROM-less SELECT are no longer folded against the outer implicit row.

[0.6.0b7] — 2026-07-31

The async oplog stack graduates to first-class options

The Rust server’s storage write-path modes — the background oplog drainer, non-logged oplog tables, and the mongod-style log-only-the-oplog data mode with its stable-checkpoint cadence — were until now reachable only through process-wide SECANTUS_* environment variables. They are now real, per-store options at every layer: a StorageOptions struct on the storage crate, RustServer(oplog_async=…, oplog_nonlogged=…, data_nonlogged=…, checkpoint_seconds=…) kwargs on the embedded handle, and --oplog-async / --oplog-nonlogged / --data-nonlogged / --checkpoint-seconds flags plus matching [storage] TOML keys on the secantusd-rs daemon. Unset options defer to the environment variables, so existing env-driven workflows are unchanged; an explicit option wins for that store only.

Two async-mode gaps closed on the way: an async store now prunes its oplog opportunistically from write volume (the every-1000-emits cadence the sync path always had — previously an async store only pruned on explicit calls), and create_archive drains the oplog queue before its checkpoint so a backup taken under the async drainer can no longer miss acknowledged writes.

Added

• secantus_storage::StorageOptions + Storage::open_with_options — per-store wt_config / durable / oplog_async / oplog_nonlogged / data_nonlogged / checkpoint_seconds; None defers to the matching SECANTUS_* env var.

• RustServer kwargs oplog_async / oplog_nonlogged / data_nonlogged / checkpoint_seconds (embedded handle).

• secantusd-rs flags --oplog-async / --oplog-nonlogged / --data-nonlogged / --checkpoint-seconds N and [storage] keys oplog_async / oplog_nonlogged / data_nonlogged / checkpoint_seconds (Rust-daemon-only; secantusd-py rejects them).

Fixed

• Async-mode change streams could surface pre-open events: a write acknowledged before watch() could still be queued at the drainer, so the open position (seeded at the drainer’s watermark) sat below it and the event leaked into the new stream (pymongo’s test_kill_cursors, async-only). The open path now waits (bounded) for the drainer to reach the minted tail captured at open (Storage::oplog_open_seq); sync mode is unchanged — an open transaction’s pinned visible tail is already the correct open position, and flushing there would block opens behind long transactions.

• Async-oplog stores never pruned the oplog from write volume; the drain path now mirrors the sync emit path’s opportunistic every-1000-emits prune.

• create_archive under the async drainer could snapshot before queued oplog entries landed; it now calls flush_oplog() first.

• docs/rust/embedded.md documented replica_set_name=None as defaulting to the replica-set persona; the embedded handle’s default is a plain standalone hello (pass replica_set_name="secantus" for change streams).

The SQL server gets its ORM gauge — and a primary-key fidelity fix to go with it

SQLAlchemy’s own dialect-compliance suite now runs against SecantusDB’s PostgreSQL server as a first-class conformance gauge (invoke validate-sqlalchemy), joining the psycopg and sqllogictest gauges in the weekly validation run — the sqllogictest gauge itself also graduates to weekly CI in the same stroke. Nothing is vendored: the suite ships inside the sqlalchemy package, pointed at a daemon server through the stock postgresql+psycopg dialect, with SecantusDB’s capabilities declared in a requirements class the suite is designed to read. The opening baseline is 572 of 738 executed tests passing (77.5%), published in the new docs/validation-report-sqlalchemy.md.

Standing the gauge up flushed out a real correctness bug: a table-level CONSTRAINT <name> PRIMARY KEY (…) was silently dropped — the column was never mapped to the document _id, so primary-key uniqueness was not enforced and duplicate keys were accepted. Declared PK constraint names are now honored end-to-end: enforcement, catalog reflection (in place of the synthesized <table>_pkey), and duplicate-key error messages. The suite’s provisioning also forced two smaller statement gaps closed: CREATE / DROP EXTENSION (citext, hstore, and plpgsql accepted — the extensions whose functionality ships built in; anything else is honestly unavailable) and COMMENT ON CONSTRAINT for check, unique, foreign-key, and primary-key constraints.

Added

• sqlalchemy_validation/: the G6 ORM gauge of tasks/sql-gauges-plan.md — runner, capability declarations (requirements.py), report generator, and an invoke validate-sqlalchemy task; weekly in validate.yml.

• .github/workflows/validate.yml: the sqllogictest gauge (validate-slt) runs weekly too, with a pinned cached sqllogictest-bin 0.29.1.

• sql/engine.py: CREATE EXTENSION [IF NOT EXISTS] / DROP EXTENSION [IF EXISTS] for citext / hstore / plpgsql (no-op success); unknown extensions raise 0A000, unknown drops 42704.

• sql/engine.py + sql/planner.py: COMMENT ON CONSTRAINT <c> ON <t> (check / unique / FK / PK), stored in the catalog; IS NULL removes.

Fixed

• sql/planner.py: a table-level CONSTRAINT <name> PRIMARY KEY (…) was silently ignored — no _id mapping, no uniqueness enforcement. The PK now applies regardless of clause position, and the declared constraint name is recorded (TableDef.pk_name) and surfaced by pg_constraint / pg_class reflection and duplicate-key errors instead of the synthesized <table>_pkey.

The SQLAlchemy compliance gauge climbs from 77% to 97% — and takes a pile of SQL fixes with it

One day after the SQLAlchemy dialect-compliance gauge landed at 77.5%, a sweep through its failure clusters brought the PostgreSQL server to 713 of 735 executed suite tests passing (97.0%), with zero errors. As with the gauge’s first landing, the score is a by-product: each cluster traced to a real server gap, and each fix is ordinary engine behavior any client benefits from.

The catalog now tells the truth about more things: temp tables carry relpersistence 't', are visible only to their creating session (pg_table_is_visible is session-aware), and are dropped when that session’s connection closes — real Postgres temp-table lifecycle. Declared type modifiers survive into reflection (varchar(52) reports its length and character varying(52) from format_type; numeric precision/scale likewise), pg_get_expr returns stored default expressions (so a SERIAL column reflects its nextval default and autoincrement), plain views expose their output columns through pg_attribute, constraint comments reflect through pg_description, pg_get_constraintdef quotes identifiers the way quote_ident does (fixing every “bizarro character” reflection case), and a composite primary key reflects its declared column order.

The expression engine grew LIKE … ESCAPE (with PG’s 22025 invalid-escape error and ESCAPE '' disabling escaping), computed LIKE patterns over the extended protocol (Describe no longer fails on a WHERE that will be evaluated per-row), IS [NOT] DISTINCT FROM in per-row evaluation, exact numeric division for int-to-numeric casts (CAST(15 AS NUMERIC) / 10 is 1.5, while 15 / 10 stays integer division), float⊕numeric operand harmonization, constant expressions in LIMIT / OFFSET, INSERT … DEFAULT VALUES, and CREATE SEQUENCE … NO MINVALUE NO MAXVALUE.

Added

• sql/planner.py + sql/scalar.py: LIKE … ESCAPE (pushdown + per-row), IS [NOT] DISTINCT FROM (per-row), constant expressions in LIMIT/OFFSET, INSERT … DEFAULT VALUES, CREATE SEQUENCE NO MINVALUE / NO MAXVALUE.

• sql/virtual.py: plain-view columns in pg_attribute; constraint comments in pg_description; quote_ident semantics in pg_get_constraintdef; temp tables report relpersistence 't' / pg_temp_1.

• sql/session.py + sql/engine.py + sql/pgserver.py: session-scoped temp-table lifecycle — visibility limited to the creating session, drop at connection teardown.

• sqlalchemy_validation/requirements.py: temp-table, constraint-index, and include-columns capabilities declared.

Fixed

• sql/scalar.py: CAST(<int> AS NUMERIC) now yields numeric, so division is exact instead of silently truncating; mixed float/Decimal arithmetic no longer raises TypeError (float8 wins, as in PG); pg_get_expr returned NULL for every stored default, hiding SERIAL defaults and autoincrement from reflection; format_type ignored type modifiers.

• sql/engine.py: extended-protocol Describe failed outright on a WHERE clause that Execute would evaluate per-row (computed LIKE patterns over bound parameters errored under psycopg); CREATE SEQUENCE … NO MINVALUE crashed with an internal error.

• sql/planner.py: a composite PK’s declared column order (PRIMARY KEY (name, id, attr)) was lost in reflection.

The SQLAlchemy compliance gauge reaches 100% — every executed suite test passes

The final round on the SQLAlchemy dialect-compliance gauge closes the residual tail: 731 of 731 executed suite tests pass, with zero failures and zero errors, up from 77.5% at the gauge’s first landing. Nothing is deselected; the only declared divergence is datetime_microseconds (BSON datetimes are int64 milliseconds, and the shared dual-protocol document store is the product), closed through the suite’s own capability mechanism — the same switch MySQL-family dialects close.

As before, the score is a by-product of real engine work. A FROM-less SELECT … WHERE EXISTS (…) now routes through the constant path with the subquery evaluated against real storage; parenthesized set-operation arms carry their own ORDER BY and LIMIT; derived tables can be set operations, VALUES lists with column aliases (the shape SQLAlchemy’s insertmanyvalues sentinel emits), or FROM-less selects; INSERT accepts any constant expression in a VALUES cell (nextval('seq') included); and covering indexes (CREATE INDEX … INCLUDE (…)) store their columns and reflect through pg_index’s indnkeyatts split.

One fix in this round was a silent wrong-answer, the worst kind: a scalar subquery ignored its ORDER BY and LIMIT, so (SELECT id FROM t ORDER BY id DESC LIMIT 1) returned the first row in storage order instead of the last. Ordered, limited, grouped, or joined scalar subqueries now run through the full query engine, and a subquery returning more than one row raises PG’s 21000 instead of picking one arbitrarily.

Added

• sql/engine.py + sql/planner.py + sql/executor.py: FROM-less WHERE EXISTS; parenthesized union arms; set-operation / VALUES / FROM-less derived tables; constant expressions (incl. nextval) in INSERT VALUES cells; covering-index INCLUDE metadata + reflection (pg_index.indnkeyatts).

• sqlalchemy_validation/requirements.py: supports_distinct_on opened; datetime_microseconds closed with the BSON-millisecond rationale.

Fixed

• sql/scalar.py: a scalar subquery with ORDER BY / LIMIT / GROUP BY / joins silently ignored them (wrong row returned); it now runs through the engine, and >1 result row raises 21000.

• sql/engine.py: extended-protocol Describe answered NoData for a set operation whose first arm was parenthesized, then Execute sent DataRows — a protocol violation that crashed libpq clients.

The pgx gauge lands — Go’s strictest pgwire client now measures the SQL server

jackc/pgx joins the SQL server’s conformance portfolio as its fourth external gauge. invoke validate-pgx runs the vendored pgx v5.9.2 pgconn and pgproto3 test packages — the hand-rolled wire client and message codecs, the Go analogue of the mongo-c-driver gauge on the Mongo side — completely unmodified, pointed at a daemon server through PGX_TEST_DATABASE. It runs weekly in CI alongside the psycopg, sqllogictest, and SQLAlchemy gauges.

The opening baseline is 291 passed / 87 failed / 22 skipped (77.0%): the pgproto3 wire codecs pass 99.4%, while pgconn (55.7%) exposes two clear feature clusters worth their own follow-ups — pipeline mode (Sync-less extended-protocol batching) and CancelRequest handling — now recorded in the backlog as the next levers.

Added

• pgx_validation/ (runner, package list, go test -json report generator), vendor/pgx submodule pinned at v5.9.2, invoke validate-pgx, and a weekly validate.yml row sharing the Go gauge’s toolchain step.

Tables get real schemas — and NOT LIKE stops lying

Relations in a user schema are now first-class: CREATE TABLE test_schema.users coexists with public.users, resolves qualified from any statement (DML, views, sequences, indexes, comments, foreign keys), reflects under its own pg_namespace row, and is invisible to unqualified lookups — pg_table_is_visible now enforces the default search path, exactly like real Postgres. DROP SCHEMA … CASCADE takes the schema’s tables with it, creating into a nonexistent schema raises 3F000, and a cross-schema foreign key renders its target as separately-quoted identifiers in pg_get_constraintdef. Internally a schema-qualified relation stores under a dotted catalog key (test_schema.users) — the same mapping user-defined types adopted — so the dual-protocol Mongo view addresses the backing collection as db["test_schema.users"].

With that in place the SQLAlchemy compliance gauge’s schemas capability opens, unlocking the suite’s entire schema-qualified surface: 978 of 978 executed tests pass (100%), up from 731 executed before, still with zero failures and zero errors.

Standing the schema surface up flushed out a genuine wrong-answer bug: sqlglot parses NOT LIKE as Like(negate=True) rather than wrapping it in NOT, and both the pushdown translator and the per-row evaluator ignored the flag — so WHERE n NOT LIKE 'pg_%' silently behaved as LIKE. Both engines now honor the negation.

Added

• sql/planner.py (qualified_table_name) + resolution sites across engine/executor: schema-qualified tables, views, and sequences stored under dotted catalog keys; 3F000 on unknown target schemas; DROP SCHEMA CASCADE drops contained tables.

• sql/virtual.py: relations split into (schema, relname) for pg_class / information_schema reflection; pg_temp_1 namespace row; cross-schema FK targets quoted per part in pg_get_constraintdef.

• sql/planner.py: pg_table_is_visible lowers to a search-path check (default namespaces + the session’s own temp tables).

• sqlalchemy_validation/: the schemas capability opens; the runner pre-provisions test_schema / test_schema_2 (SQLAlchemy’s documented DBA setup step).

Fixed

• sql/planner.py + sql/scalar.py: NOT LIKE behaved as LIKE — sqlglot encodes the negation as Like(negate=True), which both engines ignored.

• sql/executor.py: COMMENT ON a schema-qualified table or column landed on the same-named public relation.

• sql/planner.py: an auto-named foreign key on a schema-qualified table minted schema.table_col_fkey instead of PG’s table_col_fkey.

[0.6.0b6] — 2026-07-31

Log-only-the-oplog becomes crash-safe: replay-on-open recovery lands

The SECANTUS_DATA_NONLOGGED mode — the mongod storage architecture, where only the oplog is WAL-journaled and the data tables are checkpoint-durable — graduates from a measure-only benchmark probe to a recoverable configuration. A periodic stable checkpoint (60s cadence, the mongod default; SECANTUS_CHECKPOINT_SECONDS overrides) anchors a marker in the always-logged oplog-meta table, the oplog prune never touches entries above the marker (they are the recovery source), and Storage::open replays the oplog above the marker through the ordinary write paths — idempotently, so the deliberately conservative marker can never double-apply work. A clean close anchors a final checkpoint even under the fast-storage test environment, whose skip-the-close-checkpoint optimisation would otherwise lose unlogged tables’ data with no crash involved.

The contract is proven by a hard-kill harness (tests/test_crash_recovery.py): a writer subprocess is SIGKILLed mid-load and every acknowledged write must be present after the reopen — including with no checkpoint ever taken, where the entire dataset comes back from oplog replay alone. Durability matches the logged default at each sync_on_commit setting: with per-commit fsync every acknowledged write survives a hard kill; without it, a hard crash can lose the unsynced WAL tail — in either mode, exactly as before.

The default is unchanged, and deliberately so: with the durability anchoring live, the mode’s own measurements moved. A single writer gains ~5%, and a workload whose oplog stays under the retention cap keeps the probe-era headroom (~122k docs/s at eight writers measured with anchoring idle) — but a sustained eight-writer load at cap pressure pays the periodic checkpoint of a hot, unlogged working set and lands at roughly half the logged default’s throughput. Finding 14 records the decomposition; the default flip stays parked until the checkpoint cost is tamed. The mode is correct and recoverable today; choose it for read-heavy, single-writer, or bounded workloads.

Added

• Rust server: replay-on-open crash recovery for SECANTUS_DATA_NONLOGGED stores — stable-checkpoint marker + periodic checkpoint thread + idempotent oplog replay + prune clamp; the mode is recorded per-store at create time (existing stores are unaffected by the env var).

• tests/test_crash_recovery.py: the hard-kill recovery harness (SIGKILL mid-load → reopen → every acknowledged write present), plus WT-level stable-marker tests.

[0.6.0b5] — 2026-07-30

The Rust server’s change streams get a real oplog visibility point

Concurrent writers on different collections could permanently lose a change event. The Rust server’s tailable cursors treated the highest minted oplog seq as the readable tail, but a seq is minted inside its writer’s still-open transaction — so a writer on one collection could commit a later seq while an earlier one was still in flight, and a change stream that polled in that window advanced its resume position past the hole. When the in-flight transaction then committed, its event sat behind the stream’s position: dropped from the live stream and unreachable on resume. Same-collection writers never hit this (the per-collection lock serializes them), which is why it survived every single-collection test.

The fix is the analogue of WiredTiger/mongod’s all_durable timestamp: an in-flight window tracks every minted-but-unresolved seq range, and readers — wait_for_oplog, read_oplog, change-stream open positions, post-batch resume tokens — are bounded by its floor. A commit releases its range and the tail advances; a rollback releases it silently, leaving a permanent seq hole the shard merge already tolerates, so an aborted transaction can never stall the stream. flush_oplog in sync mode now genuinely waits for the window to drain, and abandoned transaction handles release their ranges on drop so a reaped session cannot pin the tail.

Fixed

• Rust server: change streams no longer lose events when writers on different collections commit out of oplog-mint order (live and on resume). New Storage::oplog_visible_tail_seq() is the bound every reader uses; three WT-level pinning tests and a cross-collection database-watch exactly-once test guard the invariant.

Added

• tests/test_mongo_server_concurrency.py::test_db_change_stream_exactly_once_across_collections — N per-collection writers under a database-wide watch, asserting exactly-once delivery (no duplicates, no losses), on both servers.

The Python server’s change streams get the oplog visibility point too

The Rust server’s oplog visibility fix has a twin on the Python server. Since the per-collection lock split, Python writers on different collections mint their oplog sequence numbers and commit their WiredTiger transactions independently — so a writer could commit a later sequence while an earlier one was still inside an open transaction, and a change stream polling in that window advanced its resume position (the scan_high skip bound) past the hole. When the in-flight transaction committed, its event sat behind the stream’s position: dropped live and unreachable on resume. Multi-document transactions were already protected by commit-time minting, but the flush between mint and WiredTiger commit had the same narrow window.

The fix is the same all_durable-style design: an in-flight mint window pins the visible tail at its floor, registered when sequences are minted and released when the owning transaction commits or rolls back (batch transactions, the user-transaction commit flush, and bare autocommit emits each resolve at their own point). Every reader is bounded by it — the tailable-getMore wake predicate, change-stream open positions, read_oplog and its scan_high, the PITR archive head, and startAtOperationTime (which now waits briefly for the window to drain past its answer instead of finalising a position an in-flight event could land behind). A rolled-back mint leaves a permanent, tolerated hole and can never stall the stream.

Fixed

• Python server: change streams no longer lose events when writers on different collections commit out of oplog-mint order (live and on resume); startAtOperationTime can no longer skip an event minted inside a still-open transaction. Five WT-level pinning tests (tests/test_oplog_visibility.py) mirror the Rust suite.

The oplog prune stops taxing every write — +25% single-writer, +62% at eight writers

Phase-0 profiling of the concurrency-parity program (Finding 12) caught the Rust server’s opportunistic oplog prune consuming ~36% of the sustained write path: once a workload passes the 100k-entry oplog cap — about four seconds into any sustained run — every sweep re-read the full 8 KiB value of every doomed row through the shard merge, copying ~8 MB per sweep just to learn which seqs to delete, on the writer’s own thread. The sweep now walks keys only, peeking a row’s timestamp just in the retention tail beyond the cap excess, and the emit path stops re-running WiredTiger’s schema-locked create for its oplog shard on every batch (a first-touch bitmask remembers what exists). Measured on the Finding-12 baseline rig: sync single-writer 25.4k → 31.6k docs/s (+25%), eight writers 41.5k → 67.3k (+62%), lifting durable-path scaling from 1.65× to 2.13× and oplog retention from 22% to 36% of the no-oplog ceiling.

The slice also closes the startAtOperationTime residual recorded by the visibility-point fix: find_seq_for_ts no longer finalises a resume position past a minted-but-uncommitted oplog entry whose timestamp qualifies — it waits (bounded) for the in-flight window to drain past its committed-view answer and rescans, so a transaction committing mid-open surfaces the earlier event instead of losing it.

Fixed

• Rust server: startAtOperationTime can no longer skip an event whose oplog entry was minted inside a still-open transaction (bounded wait on the in-flight window; falls back to the committed view at the deadline — today’s behaviour — only for long-open transactions).

Changed

• Rust server: the opportunistic oplog prune identifies doomed rows with a key-only shard merge (values peeked only for the retention tail), and oplog shard tables are created on first touch instead of per-batch.

The Finding-13 winners become the defaults — another +14% at eight writers, no knobs required

The oplog append-path sweep’s measured winners now ship as defaults on the Rust server. Oplog writes route across two shard tables instead of sixteen — the wide split existed to spread a rightmost-page append hotspot that the RecordId keying and the prune fix eliminated, and the sweep measured every narrower width beating sixteen; the read side still scans all sixteen, so stores written under any width stay fully readable and interchangeable. The oplog and pre-image btrees are created append-tuned (split_pct=100, leaf_page_max=128KB — rows arrive in ascending seq order and are never updated, so pages fill completely before splitting), and the daemon and the Python RustServer handle raise their WiredTiger cache default from a 1G to a 4G cap — WiredTiger fills cache lazily, so idle test servers stay as small as before while sustained writers stop thrashing eviction (--cache-size / cache_size= still override; the low-level Storage::open library default is unchanged).

Interleaved A/B against the previous defaults on the reference box: sync single-writer 31.8k → 35.1k docs/s (+10%), eight writers 78.1k → 88.7k (+14%) — on top of the prune-fix release’s +62%. Oplog block compression stays on deliberately: the sweep measured turning it off cratering throughput to a fifth of the ceiling (bigger uncompressed pages mean more eviction IO, and IO volume — not CPU — is the constraint).

Changed

• Rust server: oplog write routing defaults to 2 shard tables (was 16); SECANTUS_OPLOG_SHARDS still overrides 1–16; reads scan all tables regardless, so on-disk compatibility is unaffected.

• Rust server: oplog/pre-image tables are created with split_pct=100,leaf_page_max=128KB (fresh stores; existing stores keep their config).

• Rust server: secantusd-rs and the Python RustServer handle default cache_size to 4G (a lazy cap, was 1G); docs/concurrency.md’s tuning guidance updated (log prealloc now hurts at eight writers post-prune-fix; never disable oplog compression).

Oplog experiment hooks and the sweep that found 54%-retention durable writes

Three measure-oriented env hooks land on the Rust server so oplog append-path experiments no longer need a rebuild: SECANTUS_OPLOG_SHARDS overrides how many shard tables the write path routes across (1–16; reads always consider all, so any store stays fully readable), SECANTUS_OPLOG_TABLE_EXTRA appends last-key-wins WiredTiger config to the oplog/preimage table creates (the SECANTUS_WT_CONFIG_EXTRA trick at table scope), and SECANTUS_DATA_NONLOGGED is a loudly-documented, crash-unsafe, measure-only probe of the mongod architecture (journal only the oplog). bench/oplog_sweep.py drives the arms interleaved and reports retention against the same-session no-oplog ceiling.

The sweep’s headlines (recorded as Finding 13): the 16-way oplog sharding is now pure overhead — every lower shard count beats it at eight writers; turning oplog compression off craters throughput to 19% retention (zlib is load-bearing under write pressure); cache size is the strongest single knob; and the winning stack (2 shards + append-tuned oplog pages + 4G cache) reaches 102.8k docs/s at eight writers fully durable — 54% of the no-oplog ceiling, up from 43% on the defaults. The mongod-architecture probe adds only the last +11% on top (60%, matching mongod’s own 61% oplog-retention ratio), so the replay-on-open recovery project is parked until the config winners ship.

Added

• Rust server: SECANTUS_OPLOG_SHARDS, SECANTUS_OPLOG_TABLE_EXTRA, and SECANTUS_DATA_NONLOGGED (measure-only, crash-unsafe) experiment hooks — all default-off, create/routing-time only.

• bench/oplog_sweep.py: the interleaved oplog append-path sweep runner.

Rust concurrency: steal telemetry, a read-under-load bench, and the CI that runs it

The per-collection write-lock work turned the Rust server’s multi-writer story into real scaling — it climbs to about 2.6× its single-writer rate at four concurrent writers before a WiredTiger ceiling (specifically the oplog’s WAL append and checkpoint share) bends the curve back down, and the opt-in async + non-logged oplog stack lifts even that to a monotonic ~2.4× at eight writers. That ceiling lives inside WiredTiger, not in a SecantusDB lock; docs/concurrency.md carries the measured curve and the attribution. This slice adds the tooling and telemetry around that result rather than the measurement itself.

The new bench/read_concurrency.py harness measures the property the lock split most directly buys and that a raw write-throughput curve hides: a read-heavy workload keeps 60–75% of its standalone query throughput while eight writers saturate the server, where before every read queued behind every write. The findAndModify steal telemetry makes a concurrent-steal storm on a hot job-queue document visible in the server log instead of surfacing only as CPU. And the Rust parametrization of the #451 concurrency stress suite now actually runs in CI, where it previously skipped itself in every lane.

Added

• Rust server: findAndModify logs a warning every few seconds of continuous re-picking (a concurrent writer repeatedly stealing the matched document), so a steal storm on a hot job-queue document is visible in the server log instead of surfacing only as CPU — mirroring the storage layer’s write-conflict retry telemetry.

• bench/read_concurrency.py: a read-under-write-load benchmark for the Rust server — measures the query throughput a read-heavy client retains while N writers saturate the server.

• CI: the Rust parametrization of the #451 concurrency stress suite (exactly-one-winner races, exact final counts, typed-errors-only) now runs in the storage-engine job — previously it importorskiped in every lane because no other job builds the embedded Rust server.

Refreshed PGO profile for the reworked write path

The committed profile-guided-optimization profile (crates/pgo/_secantus_server.profdata.tar.gz) is regenerated against the current hot paths — the oplog visibility point, the key-only prune, and the new routing defaults all reshaped the write path since the profile was last trained, and a stale profile silently forfeits PGO’s gains (measured: the refresh recovered update_many 1.2×→1.1×, $group 1.3×→1.0×, delete_many 1.5×→0.9× of mongod on the six-workload benchmark).

Changed

• crates/pgo/_secantus_server.profdata.tar.gz retrained on the post-#702 write path (wheel builds consume it; a stale profile is safe but slower).

The admin console, documented in pictures — and four bugs it was hiding

The admin UI’s documentation has always described 22 pages in prose and shown none of them. It does now: every page in the console has a screenshot, generated rather than hand-captured. invoke admin-screenshots starts a throwaway SecantusDB on a fixed port, seeds it with a fictional shop — invented customers, example.com addresses, public landmark coordinates, indexes of every shape, users, profiler entries, backup archives — then drives all 22 pages through a real browser with Playwright, filling and submitting forms where a bare page load would only show an empty one. Machine-specific strings are rewritten out of the DOM before each shot, so a committed image carries nothing about the machine that made it. The same run publishes the four shots the marketing site uses, so the docs, the README and secantusdb.com can’t drift apart.

Driving the console through a real browser turned out to be the first time anyone had. It found four live bugs, all invisible to the existing tests because the templates render identically whether or not their JavaScript runs. Alpine was loading before Chart.js, and since this Alpine build starts the moment its script executes, the dashboard threw Chart is not defined during init() — which aborted the component before it opened the metrics websocket. The dashboard has been showing zeros, no charts, and a permanent “connecting…” status. Behind that sat three more: every Alpine page called its own init() twice, so the change-stream tail opened two sockets and displayed every event twice; the sparkline canvases had no sized parent and grew until they filled the viewport; and the geo map’s markers 404’d on Leaflet image assets this package doesn’t vendor, so map pins rendered as broken images. All four are fixed, and each is pinned by a regression test.

Regenerating the screenshots is now a release step. A browser-free test keeps every documented page wired to an image on disk, but it can’t tell a fresh screenshot from a stale one — so the release procedure regenerates them, and the capture itself fails loudly if any page logs a JavaScript error or is photographed showing an empty state.

Added

• scripts/admin_screenshots.py and invoke admin-screenshots: Playwright-driven capture of all 22 admin-UI pages against a seeded throwaway server, with DOM anonymisation, JS-error detection, empty-state detection, and publication of the website-tagged subset into the Pelican theme. Flags: --only, --list, --headed, --scale, --server-port, --keep-data, --skip-website, and --from-checkout for rendering a working tree’s templates and static assets instead of the installed package’s.

• A screenshots optional extra carrying Playwright (kept out of dev so CI lanes don’t install a browser stack they never drive).

• Screenshots throughout docs/admin.md, an admin-UI section in the README, and an admin console section on the secantusdb.com landing page.

• tests/test_docs_screenshots.py and tests/test_admin_asset_order.py.

Fixed

• The admin dashboard never worked. alpine.min.js loaded before chart.umd.min.js, and this Alpine build calls Alpine.start() as soon as its own deferred script runs, so Chart was undefined when the dashboard’s init() executed. The thrown error aborted the component before _connect(), so the live-metrics websocket never opened: every tile read 0 and the status stayed on “connecting…”. Alpine now loads last.

• Every Alpine page (dashboard, changestream, query, insert) carried a redundant x-init="init()" alongside a component that already defines init(), which Alpine invokes itself. Each page therefore initialised twice — two Chart instances per canvas, two metrics websockets, two change-stream sockets (so every event appeared twice), and duplicate collection-suggestion fetches.

• The dashboard’s sparkline canvases had no fixed-height positioned parent, so Chart.js’s maintainAspectRatio: false sizing loop grew each chart until it overflowed the viewport.

• Chart instances were stored in Alpine’s reactive state; reached through its Proxy, Chart.js’s internal per-chart lookups missed and every update() threw Cannot set properties of undefined (setting 'fullSize'). They now live in the component factory’s closure.

• The geo map drew points with Leaflet’s default marker, which loads images/marker-icon.png and images/marker-shadow.png relative to leaflet.css — files this package doesn’t vendor. Every point 404’d twice and rendered broken; points are now vector circleMarkers needing no assets.

Changed

• docs/admin.md no longer claims the UI “never makes outbound network calls of its own”. It makes one: the Geo page fetches basemap tiles from OpenStreetMap. That’s now stated up front and called out in a note on the page’s own section, since it tells a third party your IP and roughly where your data is.

A standalone Windows binary for the Rust server

The secantusd-rs standalone binary now ships for Windows alongside Linux and macOS. Every secantusdb-v* release attaches an x86_64-pc-windows-msvc archive — a .zip for Explorer, and a .tar.gz for anyone who’d rather use the same command on all three platforms — each with a .sha256 beside it. The Windows build links the C runtime statically, so the .exe runs on a clean machine with no Visual C++ redistributable installed.

Windows had been listed as blocked on the MSVC WiredTiger build “producing no static library”. That turned out to be a statement about a filename rather than a capability: MSVC emits wiredtiger.lib where Unix emits libwiredtiger.a, and build.rs grew the second name some time ago. CI had quietly been linking that static library and building secantusd-rs.exe on every push ever since — the note simply outlived its cause. Enabling the release lane was mostly packaging.

Two Windows-specific problems did surface, and both were worth finding. The linker had been warning LNK4098: defaultlib 'LIBCMT' conflicts, which is not cosmetic: WiredTiger’s static library uses the static C runtime while Rust’s MSVC target defaults to the dynamic one, and two C runtimes in one process means two heaps — memory allocated inside WiredTiger and freed on the Rust side is undefined behaviour. Building with +crt-static matches them. Separately, the binary’s smoke test asserted a clean exit after SIGTERM, which can’t work on Windows at all: send_signal maps SIGTERM to TerminateProcess, an immediate kill that exits 1 and runs no handler. It now sends CTRL_BREAK_EVENT to a process-group child, which the binary’s console handler turns into the same graceful shutdown Unix gets — and that test now runs on Windows in CI on every push, so the release binary is exercised continuously rather than only at tag time.

Added

• x86_64-pc-windows-msvc in the release-binaries matrix, publishing secantusdb-<version>-x86_64-pc-windows-msvc.zip + .tar.gz (each with a .sha256). Built with +crt-static; no PGO on this target yet, so it is a few percent slower on write-heavy paths than the other two archives and otherwise identical.

• The wheel-bundled secantusd-rs smoke test now runs on Windows in test.yml’s storage-engine job (it was skipped on the stale grounds that the binary wasn’t built there).

Fixed

• CRT mismatch in the Windows binary (LNK4098): WiredTiger’s static CRT vs Rust’s default dynamic CRT put two C runtimes, and two heaps, in one process. Now built with -Ctarget-feature=+crt-static.

• tests/test_rust_binary_smoke.py asserted a graceful SIGTERM exit that Windows cannot deliver; it now uses CTRL_BREAK_EVENT with CREATE_NEW_PROCESS_GROUP there, leaving the Unix path unchanged.

Changed

• docs-rust/installation.md, docs-rust/releases.md, the marketing site’s Rust-server page, and the two “Windows is blocked” comments in release-binaries.yml / test.yml all corrected — they described a limitation that no longer existed.

[0.6.0b4] — 2026-07-29

Restore full PGO on the arm64-macOS standalone binary

The arm64-macOS secantusd-rs binary’s on-target profile-guided optimization is working again, after a run of macos-14-runner-specific failures. Two quirks (both Linux-immune) are now handled: the PGO instrumented build drops mimalloc (whose instrumented allocator internals segfault at startup), and — the crux — the instrumented binary writes its own profile on shutdown, because the profiling runtime on that runner never wires up its LLVM_PROFILE_FILE at-exit write (a clean --version exit produced no .profraw at all). A CI diagnostic pinned that down; the binary now calls __llvm_profile_set_filename + __llvm_profile_write_file to a known path (behind the instrumented-only pgo-instrument feature), yielding a valid, mergeable profile (23.9k functions). Both binary targets ship with full two-stage PGO again.

Fixed

• secantusdb arm64-macOS binary: full on-target PGO restored — the instrumented stage self-writes its profile (the runtime’s env-driven at-exit write is inert on the macos-14 runner) and drops mimalloc to avoid an instrumentation segfault.

Fix the arm64-macOS binary’s PGO build crashing on mimalloc instrumentation

The standalone secantusd-rs binary’s two-stage profile-guided-optimization build segfaulted at startup on the arm64-macOS CI runner: the PGO instrumented stage compiles mimalloc’s own allocator internals with profiling counters, and __llvm_profile_instrument_target faults (EXC_BAD_ACCESS) when it runs inside mimalloc’s first page allocation (from LogBuffer::new in the server bind path), re-entering the half-initialized global allocator. The instrumented stage now builds with the system allocator (--no-default-features, gating mimalloc behind a default-on mimalloc cargo feature); the optimized final binary still ships mimalloc, and the collected profile — a hint whose unmatched functions are ignored — is unaffected by the allocator swap. The PyPI wheel’s embedded server was never affected (it consumes a committed profile, not on-target instrumentation).

Fixed

• secantusdb binary: the arm64-macOS two-stage PGO release build no longer segfaults in the instrumented stage. mimalloc is now a default-on cargo feature so the instrumented build can opt out (--no-default-features) while the shipped binary keeps mimalloc.

Flush the PGO profile explicitly so the arm64-macOS binary build completes

With mimalloc no longer instrumented, the arm64-macOS binary’s PGO instrumented stage ran the profiling workload cleanly but wrote no .profraw: the LLVM profiling runtime’s atexit flush doesn’t fire under the release workflow’s SIGTERM shutdown on that runner (Linux flushes normally), so the profile-merge step had nothing to merge. The instrumented stage-1 build now compiles an explicit __llvm_profile_write_file() into the shutdown path (behind a pgo-instrument cargo feature, off for every normal build), flushing the profile deterministically before exit.

Fixed

• secantusdb binary: the arm64-macOS two-stage PGO build now writes its profile under the workflow’s SIGTERM shutdown (explicit __llvm_profile_write_file() behind the instrumented-only pgo-instrument feature), so the profile-merge and optimized stages complete.

[0.6.0b3] — 2026-07-28

Fix drop (and other ops) on a never-written collection under lazy shards

Lazy shard creation (0.6.0b2) makes a collection’s documents shard exist only once something is written to it, and the Rust server’s drop_collection ran purge_collection_tables unconditionally — which opened the collection’s shard cursor and failed with a WiredTiger “No such file or directory” error when the collection had never been written (dropping an empty / never-created collection, a no-op in MongoDB). The purge now treats an absent shard as “no rows to remove”. The gap escaped the test suite because tests always create a collection before dropping it; it was caught by the standalone binary’s PGO-profile workload, whose first operation is coll.drop() on a fresh collection. A regression test now exercises every operation (drop / find / count / distinct / delete / update / aggregate / findAndModify / create-index-then-drop) against never-written collections on the Rust server.

Fixed

• Rust server: drop on a collection whose shard was never written no longer errors with a WiredTiger ENOENT; purge_collection_tables tolerates an absent documents shard (matching the other lazy-shard read/scan/merge paths).

[0.6.0b2] — 2026-07-28

Rust server: 2.2× multi-writer throughput with the async + non-logged oplog stack

The Rust server’s concurrent-write ceiling was the oplog: every write paid a WAL-logged oplog append, holding 8-writer throughput to ~56k docs/s (sync default) or ~88k with the async-oplog prototype, against a ~191k no-oplog ceiling. Two new opt-in levers close most of that gap. Setting SECANTUS_OPLOG_NONLOGGED=1 creates the oplog and pre-image tables with WiredTiger WAL logging disabled, so oplog rows are checkpoint-durable only — in async mode that removes the drainer’s WAL volume from the writers’ path and lifts 8-writer throughput to ~125k docs/s, 2.2× the sync default (and ~1.9× a single writer’s async rate), while change streams stay exactly-once. The async drainer also now coalesces queued batches into one WiredTiger transaction (up to 32 batches / 16 MB; SECANTUS_OPLOG_ASYNC_COALESCE=0 disables it). The durability trade is explicit and opt-in: a hard crash loses the oplog tail written since the last checkpoint (data tables stay fully logged and durable; a clean shutdown flushes and checkpoints a complete oplog). Defaults are unchanged — the synchronous, fully-logged oplog remains the out-of-the-box behaviour.

Added

• SECANTUS_OPLOG_NONLOGGED=1 — create the oplog + preimage tables with log=(enabled=false) (checkpoint-durable oplog; data unaffected). Applies at table-create time on a fresh store.

• Async-oplog drainer batch coalescing: queued DrainBatches are written in a single WT transaction (caps: 32 batches / 16 MB), on by default in async mode; SECANTUS_OPLOG_ASYNC_COALESCE=0 restores per-batch commits.

• SECANTUS_DISABLE_OPLOG=1 on secantusd-rs — run the daemon with oplog emission off entirely (the “drop the oplog” throughput lever from docs/concurrency.md, previously reachable only via the embedded API).

Lazy shard creation cuts Storage open cost ~2×, and a shutdown-hang fix

Opening a store eagerly created all ~37 WiredTiger tables up front — 16 per-collection document shards plus 16 oplog shards dominating — even for an ephemeral store that touches a single collection. At ~10.6 ms per WT create that is ~500 ms and 51 files per open, and under a highly parallel test run it saturated disk I/O badly enough to stall (workers stuck in uninterruptible I/O wait). Both servers now create shards on demand: a document shard is made on first creation of a collection that hashes to it, and an oplog shard on first write to it (the Python server, which never writes the sharded oplog, creates no oplog shards at all). A fresh store now creates ~13 base tables plus only the shards actually used, roughly halving open cost (open + one collection + insert dropped from ~500 ms to ~300 ms; 51 files → 20). Every read / scan / merge / drop / rename / $out / delete path on both servers treats an absent shard as empty (Python _cursor_optional; Rust WtError::is_missing_table), so a store written with a subset of shards stays byte-compatible with an eager store and across servers for backup / PITR — a missing shard simply reads as empty.

Separately, Storage.close() could hang forever inside WiredTiger’s WT_CONNECTION->close: it joined the background TTL sweeper (on by default, ttl_sweep_seconds=60) with only a 2-second timeout and then tore WiredTiger down anyway, so under load — when a sweep outran the 2 s budget — it closed the sweeper’s still-live WT session from the wrong thread and conn.close() blocked. It now joins the sweeper and heartbeat threads to completion before any WiredTiger teardown, and each loop closes its own thread-local session on exit.

Changed

• Both the Python and Rust servers create documents / oplog shard tables lazily (on first write; the Python server creates no oplog shards at all) instead of all ~37 eagerly at open, cutting Storage open cost ~2× and the per-open file count from 51 to ~20. Read, scan, merge, drop, rename, $out, and delete paths on both servers tolerate an absent shard, so the on-disk layout stays cross-server byte-compatible (a missing shard reads empty).

Fixed

• Storage.close() no longer hangs in WT_CONNECTION->close when the TTL sweeper or noop-heartbeat thread is active: the threads are joined to completion before WiredTiger teardown, and each closes its own thread-local WiredTiger session on exit rather than leaving it for a cross-thread close.

Internal

• Test harness: the session _hang_watchdog now routes its traceback to the per-worker crash file (SECANTUS_FAULTHANDLER_DIR) and stays armed through shutdown, so a shutdown-time wedge self-diagnoses instead of dying anonymously as “node down”. Hang timeout tunable via SECANTUS_HANG_SECONDS.

Fixed

• Wheel builds: the embedded extension’s PGO wiring passed cargo a fresh RUSTFLAGS (-Cprofile-use=…), clobbering the ambient flags cibuildwheel sets for the Linux containers — -Ctarget-feature=-crt-static, without which the musllinux target cannot produce a cdylib. Every manylinux/musllinux wheel build since the PGO change failed with “cannot produce cdylib … does not support these crate types”. The PGO flags now append to the ambient RUSTFLAGS instead of replacing them (an empty ambient value composes to the previous behaviour, so macOS/Windows builds are unchanged).

A much faster Rust server: mimalloc, thin LTO, and profile-guided optimization

The Rust server — both the standalone secantusd-rs binary and the copy embedded in the Python wheel — is now built with the mimalloc allocator, thin link-time optimization, and profile-guided optimization (PGO). Profiling had pinned malloc/realloc/free churn from BSON materialization as a top CPU cost on every write and aggregate path; a faster allocator attacks that directly, LTO inlines across the crate boundaries the hot path crosses, and PGO lays the final build out around the branches a benchmark run showed are hot.

Together they move single-client writes to beat standalone mongod, aggregate $group to parity, and the whole six-workload benchmark into the ~0.8×–2.1× band — a cumulative ~30–40% off the write and aggregate paths since the previous release, with no change in behaviour (it is entirely an allocator and compiler-optimization story). This is primarily a Rust-server speed release — it ships in both the wheel’s embedded server and the standalone binary.

It also carries a change-stream correctness fix on the Python server: a stream could silently skip a write that committed while it was polling (or one whose sequence number was assigned but not yet committed), because the empty-poll skip was bounded by the oplog tail rather than by what the poll had actually examined. And it hardens the driver-conformance gauges so a gauge that never ran can no longer leave behind a report that looks like it passed.

Added

• invoke rust-pgo-refresh regenerates the committed PGO profile for the embedded extension (instrument → run the benchmark workloads → merge → commit → rebuild). Needs rustup component add llvm-tools-preview.

• crates/pgo/_secantus_server.profdata.tar.gz: the committed, sparse PGO profile the wheel build consumes.

Changed

• Rust server: mimalloc is the global allocator for the embedded _secantus_server extension and the standalone secantusd-rs binary.

• Rust server: release builds use lto = "thin" and codegen-units = 1 (the allocator is the dominant lever; LTO is roughly additive on top).

• Rust server: both distributions are built with PGO. The embedded extension build (CMake) applies a committed profile via -Cprofile-use (SECANTUS_PGO_DISABLE=1 turns it off, SECANTUS_PGO_GENERATE=<dir> switches to instrumentation); the standalone binary is built two-stage per architecture in its release workflow for an on-target profile. Cumulative measured gain over the previous release (mongod-normalized): writes and aggregates ~−30–40%, moving writes past mongod and aggregate $group to parity. Raw-scan reads are unchanged (already at the wire floor).

• Java and Kotlin conformance gauges move from an exit-code guard to the shared artifact guard, so a legitimately failing run reports instead of being suppressed.

Fixed

• Python server: a change stream no longer skips past a write that commits while it is polling, or one whose sequence number has been assigned but not yet committed. The empty-poll skip is now bounded by the highest position the poll actually examined, not the oplog tail, so a write racing the poll is delivered on the next poll instead of being stepped over permanently. The Rust server was never affected.

• Driver-conformance gauges no longer regenerate a validation report from a previous run’s results — a gauge that cannot start (port in use, missing toolchain, failed build) now exits non-zero and leaves the prior report untouched, instead of restamping stale numbers under today’s date. Applies to all thirteen gauges (go, node, ruby, rust, java, kotlin, php-lib, php-ext, c, cxx, dotnet, psycopg, slt).

[0.6.0b1] — 2026-07-26

RecordId storage on both servers, faster writes, a decoupled oplog, and an Ops Board

The headline change is that both servers now use the same on-disk document layout, byte for byte. The Python server’s document table is keyed by RecordId — a monotonic insertion counter — exactly as the Rust server has been since its own RecordId work, and its index entries carry the RecordId too. That drops a write per insert and a lookup per index-scan result (measured Python inserts ~15% faster, unsorted scans ~35%, $group ~24%), makes an unsorted find(), tailable cursors, and multi:false updates follow true insertion order like mongod, and — because the two servers now agree on the whole format, documents and indexes both — lets a store written by one server be read, backed up, and point-in-time restored by the other. (There is no in-place upgrade from the older layout: a pre-RecordId data directory is refused at open with a clear error naming the mismatch; start from a fresh directory or downgrade to the build that wrote it.)

This release is also a Rust-server write-performance push plus a new tool for running the project. The oplog — the write every change stream, point-in-time recovery, and local.oplog.rs read depends on, and which a bare standalone mongod doesn’t keep at all — stopped re-encoding documents it had already serialized for the collection write. That one change brought single-writer inserts to within ~10% of a real mongod (1.1×) and updates and deletes to ~1.3×, on the same WiredTiger engine mongod ships.

For concurrent writers there’s a new opt-in async oplog (SECANTUS_OPLOG_ASYNC=1): it moves the oplog write off the writer’s critical path onto a background drainer, lifting multi-writer write throughput ~1.4× while keeping change streams correct — validated exactly-once under concurrent writers. The investigation behind it pinned the remaining multi-writer ceiling squarely to WiredTiger’s own aggregate write rate: a parallel drainer pool (also shipped) does not beat it, and neither does WT config tuning beyond ~10%, so the honest guidance for sustained concurrent-write workloads stays “run a real mongod, or drop the oplog if you don’t need change streams.” The trade for the async path is that the oplog is no longer atomic with the data — a hard crash loses drainer-queued entries (the data itself stays fully durable; a clean shutdown flushes) — which is why it is off by default.

The other headline is the Ops Board: a local web app that drives the whole build / test / release / validate cycle from one place — a matrix of all thirteen driver-conformance gauges with per-driver scores, a CI monitor with version-drift detection and startable runs, graphical job progress with full-tree cancel, a confirm-gated release page, and per-task explanations and time estimates. Alongside these, a batch of correctness fixes brought more edges in line with mongod: capped collections now evict in true FIFO order, indexes on arrays of subdocuments actually get used, malformed wire frames surface as typed errors instead of raw exceptions, and a panic in one collection can no longer wedge it until restart.

Added

• Python server: indexes record entryFormat in the catalog. Like the internal multikey flag it is stripped from listIndexes, so clients never see it.

• invoke validate --jobs N / invoke validate-pymongo-async --jobs N: run the gauge on N xdist workers, each with its own embedded server (--dist loadfile, so files stay whole). Default 1 — unchanged serial behaviour and an unchanged published number.

• pymongo_validation/plugin.py: SECANTUS_GAUGE_PER_WORKER makes each xdist worker start (and tear down) its own embedded server and overwrite DB_IP / DB_PORT with its own address before pymongo’s conftest import. The controller still runs the full identity tripwire, so every process verifies the server it is about to measure.

• explain reports isMultiKey on the IXSCAN stage, on both servers.

• /ci page: recent workflow runs (bounded, cached) plus per-server version drift.

• secantus.opsboard.github: read-only gh wrapper with an injectable runner, TTL cache, bounded limits and graceful degradation.

• secantus.opsboard.versions: local version + latest-tag reader for both servers (no network).

• /gauges page: 13 gauges × 2 servers, data-driven from registry.GAUGES, with per-gauge toolchain requirements, time estimates and info dialogs.

• secantus.opsboard.reports: validation-report parser + gauge/server filename mapping, handling all three report shapes.

• Per-gauge, per-server scores on the /gauges matrix.

• secantus.opsboard.activity: merged local + CI feed with an explicit origin.

• GitHubClient.workflows() / .dispatch() and a confirm-gated /ci/dispatch.

• Journal.list(include_running=False) and a self-refreshing running block.

• secantus.opsboard.progress: log→progress parser (phase markers + pytest %) driving an overall bar + phase stepper in the job view.

• Cancel all running control; per-job cancel now tears down the whole process group + escaped descendants (SIGINT→SIGTERM→SIGKILL) and reaps children.

• py-gate / rust-gate emit ==> [k/N] label phase-step banners.

• All gauges button per server (Python / Rust) running validate-all, with a parallelism input that sets --jobs N (dispatches the gauges over a thread pool; capped, 4 or fewer recommended).

• /release page: readiness checklist (fail-safe — unknown blocks), version + typed-confirmation gate, explicit override for blocking checks.

• secantus.opsboard.readiness: local git/changelog checks with an advisory CI check that never blocks.

• secantus.opsboard.discovery: Tier-3 process-table scan for untracked build processes, filtered against journal-tracked pids.

• Per-task info dialogs on the dashboard with long-form detail, the exact command, an irreversibility warning for release-class tasks, and a time estimate.

• secantus.opsboard.estimates: median-of-past-successful-runs estimation with an explicit measured / rough / unknown provenance shown in the UI.

• Journal.completed_durations(): bounded, exact-argv duration history (successful runs only, so an early-aborting failure can’t skew the estimate).

• secantus.jobkit: shared job runner + sqlite journal (cursor-paginated) with a pty-tee run_tracked; the ./inv wrapper routes through it (import-light, SECANTUS_NO_TRACK=1 to bypass).

• secantus.opsboard: FastAPI + HTMX + pywebview app — dashboard cards per server, job history, live log tail, cancel; token middleware; invoke opsboard task; opsboard extra; secantus-opsboard console script.

• secantus.opsboard.config: layered configuration (CLI > env > saved ~/.secantus/opsboard.json > default) with an env var for every persistable setting and --save / --print-config.

• SECANTUS_OPLOG_ASYNC=1 (Rust server, opt-in, prototype): oplog entries are persisted by a background drainer off the writer’s transaction. Storage::flush_oplog blocks until the drainer has caught up (read-after-write oplog visibility).

• secantusd-rs --log-file-max SIZE and [storage] log_file_max in the config file (unit-suffixed, e.g. 128MB / 1GB / 2GB), threaded into the WiredTiger connection config. Daemon default: 2GB.

• SECANTUS_WT_CONFIG_EXTRA (Rust server / daemon): raw WiredTiger connection-config appended to the built config string, overriding defaults via last-key-wins.

Changed

• Python server: the documents table is keyed (db, collection, RecordId) with a framed [u32-LE id_key_len][id_key][blob] value, matching the Rust server’s on-disk format exactly. Insert write-amplification drops from four rows to three, and an unsorted scan no longer walks a side index to find insertion order.

• Python server: an unindexed update/delete candidate scan now visits documents in insertion order rather than _id order, so multi: false picks the oldest matching document like mongod.

• Python server: index entries store the document’s RecordId (8-byte big-endian) instead of its encoded _id, removing a lookup per result from every index scan. Matches the Rust server’s entry format byte for byte.

• Python server: uniqueness checks exclude the document being updated by RecordId rather than by encoded _id.

• docs/benchmark.md, the README and the Rust server’s docs index carry the new figures; the latency chart is regenerated from them.

• listIndexes no longer echoes the internal multikey catalog flag, matching mongod. The admin console’s multikey badge is gone with it — the flag isn’t wire-visible, and the console reports what the wire says; isMultiKey in the explain visualiser carries the same information.

• secantus/storage.py: _prune_oplog_locked streams only the oldest oplog entries and stops early instead of scanning and decoding the entire oplog; a new in-memory _oplog_live_count (seeded by a one-time key-only count on open) drives the entry-cap decision without a counting scan. Doomed rows are deleted from the one table the oldest-first walk found them in, not probed across all shard tables.

• The Rust Storage holds its Connection / oplog state behind Arc so the drainer thread can share them; wait_for_oplog blocks on the drainer’s written_seq watermark in async mode and on next_seq - 1 (unchanged) in the synchronous default.

• The daemon’s WiredTiger WAL log=(file_max=...) defaults to 2GB instead of 128MB. wt_config takes the value as a parameter; the embedded RustServer handle and Storage::open’s test-default config are unchanged at 128MB.

• Storage::emit_oplog (Rust) takes an OplogEntry that is either an owned Document (the rare DDL / noop / findAndModify paths, encoded as before) or a pre-assembled raw RawDocumentBuf (the hot insert / update / delete / capped-eviction paths). The raw builder writes op / ns / ui / o / o2 in mongod field order and splices the pre-encoded o / o2 bytes, so the document body is never re-serialised; ts and wall are appended last, matching the historical byte layout.

• secantus_core::diff::compute_update_description walks the pre-/post-images directly (walk_docs) instead of wrapping each in an owned Bson::Document clone.

• Storage::find_matching short-circuits an empty filter (find({})) instead of running a foregone RawDocument::from_bytes + matches_raw per document, and the read-only collection scan (scan_blobs_natural) reuses each value’s allocation rather than cloning the blob a second time.

• Rust server oplog persisted across sixteen shard tables (secantus_oplog_sh0..15), each write batch routed to one shard by its start sequence; all oplog readers merge the shards (plus the legacy single table) in seq order via a k-way merge.

• Opportunistic oplog prune bounded via a maintained live-entry count: an early-out (one timestamp read) when under the cap with the oldest row still in-window, and a bounded walk of only the doomed rows otherwise — replacing the full-oplog scan that dominated the single-writer write path.

• Python Storage oplog readers (read_oplog, oplog_floor_seq, find_seq_for_ts, prune, recovery, PITR archive) merge the sharded oplog so the Python server can read/recover/prune a Rust-written store; Python writes stay on the legacy single table.

• The Rust server routes an insert’s OP_MSG kind-1 documents sequence to the handler as un-decoded byte slices (a new CommandContext.raw_insert_documents side-channel), skipping the merge-decode and the command-layer re-encode. The handler pre-checks _id $-prefixed keys over RawDocument and passes the raw bytes to storage; documents inline in the command body, and collections with a validator, still take the decoded path.

• Storage::insert / Storage::insert_one (Rust) store the caller’s BSON verbatim when _id already leads the document, skipping the encode_doc re-serialisation; they fall back to encode_doc when an ObjectId is assigned or _id must be reordered to the front. Stored bytes are unchanged in every case (verified byte-for-byte against the client-sent encoding across ObjectId / string / int _id, nested documents, arrays, Decimal128, dates, and binary), and the pymongo conformance gauge is non-regressing (1020/1500, 99.5%).

Fixed

• Python server: a store written before this change is now refused at open with a clear message naming the format mismatch, instead of being opened and read with the wrong key format.

• Python server: a store whose indexes predate this change is refused at open with a message naming the index and the format mismatch, instead of running index scans that quietly match nothing.

• Python server: a tailable cursor on a capped collection with non-monotonic _id values no longer drops documents inserted after the cursor opened, and no longer redelivers documents it already returned.

• Python server: capped-rollover detection (CappedPositionLost) is based on the oldest-inserted document rather than the smallest _id, matching what capped eviction actually removes.

• Python server: capped-collection eviction is now strict FIFO (insertion order) even for non-monotonic user _id values, matching mongod, instead of evicting in _id byte order.

• Rust server: per-collection write locks, the cursor registry, and per-statement transaction locks are poison-tolerant, so a panic inside a critical section no longer leaves the collection, cursor, or transaction permanently unusable.

• paths.py / secantus-core: new get_path_values resolves a dotted path through arrays, returning every reachable value plus whether an array was traversed.

• storage.py / secantus-storage: index-key generation, the multikey flag, and the sparse-index gate all use that walk, so a path descending into an array is indexed per element.

• storage.py / secantus-storage: unique enforcement (_unique_conflict, the createIndexes pre-check, and find_index_duplicates) probes every key a document contributes instead of one canonical key, and a duplicate-key error reports the value actually behind the conflicting key.

• $jsonSchema with pathological schema nesting returns FailedToParse (code 9) rather than a generic internal error (security review I21).

• A malformed PostgreSQL startup packet (short CANCEL, non-UTF-8 parameter) surfaces as PGProtocolError instead of a raw struct.error / UnicodeDecodeError (I16).

• The Mongo wire framing translates a RecursionError during body parse into a BadValue reply, keeping the connection alive (I1).

• secantus-storage (Rust): Storage’s close (Drop) now checkpoints when durable is set, mirroring Python Storage.close; the flag is resolved on open via resolve_durable with Python’s precedence. Adds Storage::open_with_config_durable for an explicit override.

• Oplog retention prune deletes each doomed row from its exact source table (WiredTiger cursors are overwrite-mode, so a remove() of an absent key silently succeeds — a shard-then-legacy fallback would have leaked pruned rows on a Python-written store).

• secantus-storage (Rust): WiredTiger connection config now sets log=(prealloc=false), eliminating the ~256 MB per-instance pre-allocated journal that was exhausting CI runner disks.

• Admin UI: the change-stream tail stops polling and releases its cursor as soon as the client disconnects, instead of only on the next event send.

• Bounded the per-poll awaitData wait so an orphaned poll thread left behind on disconnect frees promptly, hardening against an intermittent CI worker crash under load.

Removed

• Python server: the forward natural-order table (secantus_natural) is no longer written. The reverse table (secantus_natural_seq) remains as the _id index.

[0.6.0b0] — 2026-07-19

Say no like mongod: an operator-fidelity sweep, and a Rust server that clears every gauge

The single largest theme in this release is learning to fail correctly. Roughly sixty slices went through the query, update, and aggregation operators asking one question of each: when the argument is wrong, does SecantusDB raise the same error, with the same code, that mongod raises? Very often it did not — $all silently mis-matched a bad argument, $rename could corrupt a document, $concat quietly coerced non-strings, $bucket dropped out-of-range documents on the floor, and a long tail of operators leaked a raw Python exception to the wire. Those are now mongod-shaped errors with mongod’s codes. Alongside the rejections, numeric fidelity got the same treatment: whole-number doubles are accepted where mongod accepts them, $toInt and $convert enforce int32/int64 overflow bounds, $mod matches on floats and bools, and $substrBytes is genuinely byte-based — including refusing a range that would split a UTF-8 character.

The Rust server crossed its headline milestone: it now clears all thirteen driver-conformance gauges, the same unmodified upstream suites the Python server runs. Getting there took both correctness work (MONGODB-X509 authentication, $project error fidelity, full $min/$max ordering) and a sustained performance push — writers no longer queue behind each other, reads no longer queue behind writers, oplog bookkeeping came off the write path, and four separate paths (scanning, filtering, projection, and the wire reply) now work over raw BSON instead of decoding whole documents just to re-encode them.

New surface landed too. $median and $percentile ship on both servers without a t-digest; $sum / $avg / $max / $min work as expression operators, not just accumulators; $toLong joins the conversion family; $jsonSchema grew mongod’s full keyword surface; and expanded change-stream events now match mongod field-for-field. The SQL server kept its own pace with binary COPY, composite records end to end, binary and scrollable server-side cursors, DO blocks, savepoint rollback that reverts DDL, type modifiers on the wire, and idle_in_transaction_session_timeout. The admin console caught up with all three servers, the benchmark pages gained charts, and new concurrency stress suites found and fixed real races.

Highlights

• Operator fidelity: ~60 slices aligning argument validation and error codes with mongod across query, update, and aggregation operators — rejections, numeric coercion rules, and byte-vs-codepoint string semantics.

• Rust server: clears all thirteen driver-conformance gauges; MONGODB-X509 auth; per-collection write locks, lock-free reads, oplog off the write path, raw-BSON scan / match / projection / reply paths, and $group field pushdown so wide documents decode only what the stage reads.

• New operators: $median, $percentile, $toLong, $sum / $avg / $max / $min as expressions, full $jsonSchema keyword surface, $bucketAuto preferred-number granularity.

• Change streams: expanded events match mongod field-for-field.

• SQL server: binary COPY, composite records, binary + scrollable server-side cursors, DO blocks, savepoint DDL rollback, wire type modifiers, idle_in_transaction_session_timeout.

• Tooling: concurrency stress suites (and the races they caught), benchmark charts, a three-server admin console, and a green docs build.

The Rust server decodes only the fields a $group actually reads

The aggregate_group workload was the worst of the six in the post-raw-BSON profile (3.1× of mongod) for a structural reason: $group received fully-decoded documents while reading only its _id and accumulator-argument fields, so input materialization survived into the heavy stage. secantus_core::referenced_top_level_fields now walks a $group spec and returns the top-level fields it reads, and the command layer pushes that field set down into the fetch when the first heavier stage of a pipeline is such a $group. Wide documents no longer pay to materialize fields the pipeline never looks at.

The analysis bails to a full decode wherever the referenced set can’t be determined statically — $$ROOT / $$CURRENT, computed-field access, and non-simple accumulators like $top / $topN whose sortBy names a field by bare key.

Changed

• Rust server: a leading $group pushes its referenced top-level field set into the fetch, decoding only those fields from wide documents.

Admin console docs catch up, and the docs build goes green again

The admin web UI documentation still described the console as it was before the Rust server and the SQL server existed. Most conspicuously it carried a per-server feature table asserting that the Rust server lacked archive restore, oplog and TTL pruning, role grant/revoke, killOp, logs, and profiling — a table that had been wrong for months, and whose in-code counterpart has since been removed. The page now explains why there is no such table any more, and what replaced it: SecantusDB targets start permissive, and a feature is withdrawn only when the server itself reports the command missing.

The rest of the page caught up with what shipped alongside that — point-in-time recovery, launching the Rust server from the embedded control, index collation, target-sourced roles, the wider set of _id types the collection browser can paginate, and the fact that admin.db is a credential store and is now permissioned like one. Several stale claims went with it, including a “one target server per launch” line that predated the target hot-swap, and a limitations entry for a saved- connections page that has since shipped.

Separately, invoke docs had been failing. Eleven Rust-server driver validation reports were never added to the toctree, and since docs-only commits are deliberately skipped by CI, nothing caught it. All thirteen reports are now listed and the build is clean again.

Fixed

• invoke docs builds warning-free. Eleven validation-report-*-rust-server pages were missing from the toctree, failing the warnings-as-errors build.

Changed

• The admin UI docs describe the current capability model, the PITR panel, the Rust embedded-server option, index collation, target-sourced roles, admin.db permissions, and the supported _id types for pagination.

The admin console catches up with three servers

The admin console was written when SecantusDB had one server. Since then a Rust server and a PostgreSQL-wire SQL server shipped, and the console quietly fell behind: a hardcoded table of “what the Rust server can’t do yet” went stale within days of being written and spent months hiding six feature groups — archive restore, oplog and TTL pruning, role grant/revoke, killOp, the server log, and profiling — behind disabled buttons, on a server that implements every one of them.

That table is gone. Only a real mongod keeps a static capability profile, because its negatives are definitional rather than a snapshot of a moving target: no mongod will ever serve the proprietary secantusAdmin.* commands. Both SecantusDB servers now start fully permissive, and a feature is withdrawn only when the target itself answers CommandNotFound — negative knowledge learned from the live server instead of guessed in advance, so the console cannot drift out of step with either server again.

The same review closed the rest of the gap. Point-in-time recovery, the largest shipped subsystem with no interface at all, gets a panel on the backup page. The embedded-server button can start the Rust server, not just the Python one. The role picker asks the connected target what roles exist rather than consulting the Python server’s own table. Collections keyed by Decimal128, UUID, or Binary _id can be browsed at last, and a tampered pagination cursor now returns a clean error instead of an unhandled crash.

Added

• Point-in-time recovery on /backup: take a base snapshot, and recover an archive to a wall-clock moment into a fresh directory.

• The embedded-server control can launch either the Python or the Rust server; the picker appears when the Rust extension is installed.

• Collation input on the index-create form, and a collation badge in the index list.

• Decimal128, UUID, and Binary _id values are supported by the collection browser’s pagination cursor, Binary round-tripping its subtype.

Changed

• Capability detection no longer keeps a per-flavour feature table for SecantusDB servers. Features are hidden only after the target reports CommandNotFound.

• The role picker and /roles catalogue are sourced from the connected target via rolesInfo, falling back to the built-in names. Role names submitted from the form are no longer filtered against a local table, so a valid custom role is no longer silently discarded.

Fixed

• ~/.secantus/admin.db, which stores target URIs verbatim including credentials, is created 0600 with its directory 0700. Previously it was left at the process umask while the token file beside it was already locked down.

• A malformed pagination cursor returns a ValueError-shaped error rather than an unhandled InvalidId / InvalidOperation from a bson constructor.

• Pointing the console at a postgresql:// URI explains that the SQL server has no admin UI, instead of failing with an opaque pymongo parse error.

$all validates its argument instead of silently mis-matching

$all accepted a malformed argument. A non-array leaked / mis-parsed, and a $-expression element that wasn’t the all-$elemMatch form — mixing $elemMatch with a scalar, or using another $-operator document — was silently treated as an equality clause (matching nothing) rather than erroring. mongod rejects both with BadValue: “$all needs an array” and “no $ expressions in $all”. Both servers now match.

A pure-scalar $all, an all-$elemMatch form, regex elements, and plain subdocument elements remain valid. The Python server carries mongod’s BadValue; the Rust core defers these cases so the Rust server rejects them too. Three-way mongod 7.0.12-verified.

Fixed

• $all rejects a non-array argument (“needs an array”) and a $-expression element outside the all-$elemMatch form (“no $ expressions in $all”) with BadValue, instead of silently mis-matching (both servers).

$all against a scalar field now matches, like mongod

The $all array query operator silently missed documents whose field held a scalar value rather than an array — {tags: {$all: ["red"]}} matched {tags: ["red", ...]} but not {tags: "red"}, on both the Python and Rust servers. mongod treats a scalar field like a one-element array for $all (equality and regex elements alike), so those documents should have matched. This dual-server correctness bug was found while triaging the driver-gauge results and is verified fixed against a live mongod 7.0.12 probe (three-way: Python == Rust == mongod). In the same fix, $all: [] now correctly matches nothing (it previously matched every array-valued document), and $elemMatch clauses inside $all still correctly require an actual array.

Fixed

• $all matches a scalar field value like a one-element array (both servers).

• $all: [] matches nothing rather than every array-valued document.

Range operators order array elements by full BSON type order, like mongod

A third comparison bug from the driver-gauge triage: $gt / $lt against an array bound compared elements pairwise, but a cross-type element pair made both servers return no match — {a: {$gt: [1, 2]}} skipped {a: [1, "x"]} even though mongod matches it (a string element outranks a number element in BSON order, so [1, "x"] > [1, 2]). Python’s native list comparison raises TypeError on "x" > 2 (swallowed to a no-match) and the Rust matcher returned no-match on any incomparable element pair. Both now order array elements by full BSON order (type rank first) via the shared _bson_lt comparator, verified three-way against a live mongod 7.0.12 probe.

Fixed

• Range operators order two arrays element-by-element in full BSON order, so a cross-type element pair still orders instead of silently no-matching (both servers).

Array operators reject a non-array input instead of silently yielding null

$first, $last, $reverseArray, $concatArrays, $slice, $map, $filter, and $reduce all silently returned null when their input wasn’t an array. mongod errors on a non-array (non-null) input, each with its own code: $first / $last 28689, $reverseArray 34435, $concatArrays 28664, $slice 28724, $map 16883, $filter 28651, $reduce 40080. A null or missing input still yields null. Both servers now match.

The Python server carries mongod’s codes; the Rust core defers a non-array input (so the Rust server rejects it) and now distinguishes a null input (→ null) from a non-array one. Three-way mongod 7.0.12-verified.

Fixed

• $first / $last / $reverseArray / $concatArrays / $slice / $map / $filter / $reduce reject a non-array input with the operator’s mongod code, instead of silently returning null; a null / missing input still yields null (both servers).

Array / set / string operator type-guards match mongod’s error codes (and stop silently accepting)

A discovery sweep of ~120 aggregation-operator error cases against real mongod 7.0.12 found a bounded set of type-guard divergences, several of them silent accepts — operators that returned a value where mongod errors. They now match mongod: $arrayElemAt, $in, and $regexMatch/$regexFind/$regexFindAll were silently returning null/false/[] on a bad argument and now error, and the rest returned a generic TypeMismatch (14) where mongod uses a specific Location code. Both the Python and Rust servers are fixed (the Rust core defers each case — $in, $arrayElemAt, and the regex ops needed Rust-side fixes to stop computing a value), verified against real mongod.

Fixed

• Silent accepts, now errors (both engines): $arrayElemAt non-array → Location28689; $in non-array second argument → Location40081; $regexMatch / $regexFind / $regexFindAll non-string input → Location51104 (a null/missing input stays valid — false/null/[]).

• Generic TypeMismatch (14) → mongod’s Location code: $size (17124), $indexOfArray (40090), $setUnion (17043), $setIntersection (17047), $setDifference (17048), $setIsSubset (17046), $anyElementTrue (17041), $allElementsTrue (17040), $mergeObjects (40400), $range non-numeric bound (34443), $indexOfBytes non-string (40091/40092), $binarySize (51276), $bsonSize (31393).

arrayFilters nested-identifier extraction

arrayFilters identifiers are now extracted recursively through $and / $or / $nor, completing the arrayFilters validation. A filter like {$and: [{"x.a": {…}}, {"x.b": {…}}]} correctly resolves the single identifier x (so a $[x] update path applies to the matching elements), and mongod’s “exactly one identifier per filter” rule is now enforced: a filter carrying two distinct identifiers — top-level or nested — is rejected, as is a bare $expr. Both the Python and Rust servers behave identically, verified against real mongod 7.0.12.

Fixed

• A single arrayFilter identifier nested inside $and/$or/$nor (e.g. {$and: [{"x.score": {$lt: 50}}]} for $[x]) is now extracted and applied, instead of failing with “arrayFilters has no entry for identifier x”.

• An arrayFilter referencing two or more distinct identifiers now raises code 9 (“Expected a single top-level field name, found ‘x’ and ‘y’”) — previously a second top-level identifier was silently ignored.

• An arrayFilter that is a bare $expr (no field identifier) now raises code 224 (“$expr is not allowed in this context”).

arrayFilters validation

arrayFilters (the $[<identifier>] filter documents passed to an update) are now validated the way real mongod validates them, instead of silently accepting malformed input. A filter that isn’t an object, is empty, carries an identifier that isn’t a lowercase-letter-led alphanumeric name, repeats an identifier, or isn’t actually referenced by any $[<id>] path in the update is rejected with mongod’s exact error code. Covered on both the Python and Rust servers (the Rust core defers each invalid case), verified against real mongod 7.0.12.

Fixed

• A non-object array filter now raises code 14 (“BSON field ‘update.updates.arrayFilters.N’ is the wrong type …, expected type ‘object’”).

• An empty array filter ({}) now raises code 9 (“Cannot use an expression without a top-level field name in arrayFilters”).

• An identifier that isn’t an alphanumeric string beginning with a lowercase letter (e.g. 1x, X) now raises code 2 (“Error parsing array filter …”).

• Two array filters with the same top-level identifier now raise code 9 (“Found multiple array filters with the same top-level field name …”).

• An array-filter identifier that no $[<id>] path in the update references now raises code 9 (“The array filter for identifier ‘’ was not used in the update …”).

Notes

• An array filter whose only top-level keys are $-operators (e.g. {$and: [{x: …}]}) carries a nested identifier that SecantusDB doesn’t extract yet; such a filter is left unvalidated rather than wrongly rejected (tracked in tasks/backlog.md).

$bitsAllSet / $bitsAllClear / $bitsAnySet / $bitsAnyClear validate their argument, like mongod

The bitwise query operators mishandled several non-integer arguments. A negative bit position ($bitsAllSet: [-1]) raised an uncaught ValueError (from 1 << -1) that surfaced without a code, a negative or fractional non-array bitmask was silently accepted or reported the wrong code, and the Rust server rejected a valid whole-number-double mask/position ($bitsAllSet: 6.0) because its coercion didn’t accept doubles.

All four operators now match mongod on both servers: a whole-number double is accepted (truncated), and a fractional double, a bool, or a negative value is rejected — a bad bit position with code 2, a bad non-array mask with code 9 — on the Python server with mongod’s messages, the Rust core deferring to BadValue. Three-way mongod 7.0.12-verified.

Fixed

• $bits* accept a whole-number-double mask / bit position and reject a fractional / negative / bool one with mongod’s exact code, instead of raising an uncaught ValueError on a negative position, silently accepting a negative mask (Python), or rejecting a valid 6.0 (Rust server).

$bucket errors on an out-of-range value instead of silently dropping the document

$bucket did almost no validation. Worst of all, a document whose groupBy value fell outside every bucket and had no default was silently dropped — silent data loss. mongod errors (7158303). It now does too, on both servers.

$bucket also now validates the rest of its spec like mongod, instead of silently accepting it: missing groupBy (40198), non-array boundaries (40200), fewer than two boundaries (40192), boundaries of mixed type (40193) or not strictly ascending / duplicated (40194), a default that falls inside the bucket range (40199), and a non-document output (40196). The Python server carries mongod’s codes; the Rust core defers those cases to BadValue. Valid buckets are unaffected. Three-way mongod 7.0.12-verified.

Fixed

• $bucket errors (7158303) on an out-of-range value with no default instead of silently dropping the document, and rejects an invalid spec (missing groupBy, bad/unsorted/mixed boundaries, in-range default, non-doc output) with mongod’s codes instead of silently accepting it (both servers).

Argument validation for $bucketAuto, projection $elemMatch, and $pull / $pullAll

Three more type-guard divergences from real mongod are closed. $bucketAuto now validates its buckets argument (a bool or non-numeric value, a fractional double, a non-positive count, or a missing groupBy/buckets each raise mongod’s exact code, while a whole-double count is accepted); a non-document $elemMatch projection argument is rejected; and $pull / $pullAll against a field that is present but not an array now errors instead of silently doing nothing. All three are covered on both the Python and Rust servers (the Rust core defers each invalid case) and verified against real mongod 7.0.12.

Fixed

• $bucketAuto buckets now raises Location40241 (non-numeric or bool), Location40242 (fractional double — not representable as a 32-bit integer), Location40243 (not greater than 0), and Location40246 (missing groupBy or buckets) instead of silently accepting buckets: true or leaking an uncoded error. A whole-double buckets (e.g. 2.0) is accepted, matching mongod.

• A non-document $elemMatch projection argument (e.g. {arr: {$elemMatch: 5}}) now raises Location31274 instead of being silently accepted.

• $pull / $pullAll on a field that exists but is not an array (a scalar or null) now raises code 2 (“Cannot apply $pull to a non-array value”) instead of silently doing nothing. A missing field remains a no-op.

$bucketAuto granularity validation

The optional granularity argument to $bucketAuto is now validated against real mongod’s rules instead of being silently ignored. A non-string value raises code 40261, and an unknown series name raises 40257 — both matching mongod 7.0.12 exactly. A valid preferred-number series (R5, R10, …, POWERSOF2, 1-2-5, E6, …) is rejected as not-yet-supported (code 2) rather than silently producing count-chunked, unrounded boundaries: reproducing mongod’s boundary rounding byte-for-byte would require its exact internal float series constants (its 6.3 is the f64 6.3000000000000007, not float("6.3")), which aren’t recoverable by black-box probing — so a faithful error is preferred over a silently-divergent result (see tasks/backlog.md). Both the Python and Rust servers behave identically.

Fixed

• $bucketAuto with a non-string granularity now raises Location40261, and with an unknown series name Location40257, instead of accepting them.

Changed

• $bucketAuto with a valid but unsupported granularity series now raises an explicit “not yet supported” error (code 2) instead of silently ignoring the field and returning unrounded boundaries.

$bucketAuto granularity rounds boundaries to preferred-number series

$bucketAuto now honours the granularity option (R5/R10/R20/R40/R80, E6/E12/E24/E48/E96/E192, 1-2-5, and POWERSOF2), rounding bucket boundaries to the ISO preferred-number series exactly as mongod does — instead of rejecting a valid series as unsupported. The rounding is hex-exact against real mongod 7.0.12, including mongod’s non-standard floating-point results (its R5 boundary at 6.3 is the double 6.300000000000001, i.e. 63 * 0.1).

The rounder and the boundary walk were ported verbatim from mongod’s granularity_rounder_preferred_numbers.cpp and document_source_bucket_auto.cpp to both the Python server and the Rust core (which backs the Rust server), so the two agree bit-for-bit and both match mongod. A granularity groupBy value must be a non-negative number: a non-numeric value, a NaN, or a negative number is rejected (mongod codes 40258 / 40259 / 40260 on the Python server). A Decimal128-valued groupBy is deferred (the standing Decimal128 precision limitation); the int/double path is complete.

Added

• $bucketAuto granularity boundary rounding for every preferred-number series and POWERSOF2, hex-exact to mongod 7.0.12 on both the Python and Rust servers.

Fixed

• A $bucketAuto granularity groupBy value that is non-numeric (40258), NaN (40259), or negative (40260) is now rejected with mongod’s code instead of the previous blanket “unsupported” error.

CI stops cancelling its own answers

Two blind spots on the same theme — checks that quietly produced no result, so a breakage could sit on main looking green.

The docs had no CI at all. test.yml and both wheel workflows carry paths-ignore: ['**.md', 'LICENSE*', 'docs/**'], which is right for a test matrix but means a docs-only commit skips CI entirely — and that is exactly how the Sphinx build came to be failing on main through many green pushes. A new Docs workflow builds both trees with warnings-as-errors. It has no paths filter on purpose: conf.py runs autodoc over the package, so a malformed docstring in a code-only commit can break the docs build, and filtering to docs/** would recreate the same blind spot facing the other way. The build compiles nothing — the WiredTiger extension is mocked — so running it on everything is the cheapest job in the repo.

The second was subtler. All three of Tests, Build wheels and Build secantus-core wheels cancelled in-progress runs per ref. On a feature branch that is what you want, since the newest push should win. On main every merge lands on the same ref, so on a busy day each merge cancelled the previous commit’s post-merge run: several consecutive merges each showed cancelled, meaning the default branch went long stretches with no completed result and a wheel-only regression would not have surfaced until a release tag. Cancellation is now disabled on main only, so those runs queue and each finishes, while pull requests keep newest-push-wins.

Added

• A Docs workflow building docs/ and docs-rust/ with sphinx-build -W, covering the gap left by every other workflow’s paths-ignore.

Fixed

• Tests, Build wheels and Build secantus-core wheels no longer cancel their own post-merge runs on main. Runs on the default branch queue and complete; pull-request branches still cancel superseded runs.

$concat rejects non-string operands instead of coercing them

$concat silently str()-coerced any operand — {$concat: ["x=", 5]} produced "x=5" — and treated a null / missing operand as an empty string. mongod requires every operand to be a string: a non-string operand is Location16702 (“$concat only supports strings, not ”), and a null or missing operand short-circuits the whole expression to null (evaluated left-to-right, so a non-string that precedes a null still errors). Both servers now match.

The Python server carries mongod’s code; the Rust core defers a non-string operand (so the Rust server rejects it) and now returns null on a null operand rather than skipping it. Three-way mongod 7.0.12-verified.

Fixed

• $concat rejects a non-string operand with Location16702 and returns null for a null / missing operand, instead of str()-coercing operands or treating null as an empty string (both servers).

Concurrency stress suites hammer the servers — and the races they caught are fixed

Two new concurrency harnesses hammer the servers with barrier-synchronized thread storms — one drives the Mongo-wire servers (the Python server and the embedded Rust server, every test parametrized over both) through real pymongo clients, the other drives the PostgreSQL-wire server through psycopg — same-key insert races, transactional increment hammers, bank-transfer invariants under concurrent readers, findAndModify ticket dispensers, unique- index races, DDL churn against live writers, and connection churn under load. Every test asserts a hard integrity invariant (exact counts, exactly one race winner, a conserved total) plus error hygiene: the only errors a loser may see are the typed, retriable signals a real server would send.

The harnesses caught five real concurrency bugs, now fixed on every affected server. A SQL write-write conflict escaped as a generic XX000 internal error; it now surfaces as SQLSTATE 40001 serialization_failure, the retriable signal drivers key their retry loops on, and the losing connection stays fully usable. SQL DML statements are read-modify-write sequences spanning several storage calls, so concurrent inserts could double-satisfy a UNIQUE constraint (134 rows landed for 30 distinct values in the reproducer) and concurrent SET n = n + 1 updates lost increments (83 of 400 survived); DML statements — and bare nextval() draws — now serialize per shared storage, closing both. findAndModify {new: true} on both Mongo-wire servers re-found the document after updating it, so two concurrent callers could be handed the same post-image (8 duplicate tickets in 400 measured); the write now captures its own post-image while it holds the storage lock, on the Python server and the Rust server alike. And findAndModify’s write now re-asserts the original query (keyed by the matched _id, in a re-pick loop) on both servers, so job-queue claims and fam removes are exclusive — two workers can no longer both “take” the same document.

Added

• tests/test_pgserver_concurrency.py — 11 psycopg-driven stress tests: autocommit insert storms, same-PK and UNIQUE-constraint races (exactly one winner, losers see 23505), transactional and autocommit increment hammers, a deterministic two-transaction 40001 conflict, bank transfers conserving the total under concurrent readers, concurrent nextval(), DDL churn alongside DML, connection churn under write load, extended-protocol prepared statements across threads, and a bounded txn-vs-autocommit stall check.

• tests/test_mongo_server_concurrency.py — one pymongo-driven harness parametrized over BOTH Mongo-wire servers (the pure-Python SecantusDBServer and the embedded Rust server): insert storms, $inc hammers, findAndModify ticket dispensers, upsert races, unique-index races, readers paginating (getMore) through churn, index builds under write load, multi-collection writers, delete/insert churn, client connection churn, and a change stream observing every insert from four concurrent writers, plus job-queue claim-exclusivity and remove-exclusivity storms. Every test runs against both servers.

• Storage.update_matching(..., return_post_images=True) — returns the post-image of each write, captured while the statement holds the storage lock, so command handlers never re-read what they just wrote.

Fixed

• SQL: a storage-level write-write conflict (WriteConflictError / WT_ROLLBACK) now maps to SQLSTATE 40001 serialization_failure on both the simple and extended protocol paths, instead of escaping as XX000 internal error; the losing connection survives, ROLLBACK works, and retry converges.

• SQL: DML statements serialize per shared storage, so concurrent inserts can no longer double-satisfy a UNIQUE constraint and concurrent computed updates (SET n = n + 1) no longer lose increments. Bare SELECT nextval('seq') draws are serialized the same way and never repeat a value.

• findAndModify {new: true} — on BOTH Mongo-wire servers — returns the post-image of its own write instead of a racy re-read, so concurrent callers can no longer be handed the same ticket. Python captures it via Storage.update_matching(..., return_post_images=True); the Rust storage grew the matching primitive (UpdateOutcome::post_image). The upsert path returns the upserted document from the write itself.

• findAndModify (both servers) re-asserts the original query at write time — the update/delete is keyed by the matched _id plus the query, in a re-pick loop — so the job-queue pattern ({state: "new"} → {$set: {state: "taken"}}) can no longer double-claim a document, and a fam remove checks its deleted-count so two removes can never both claim the same pre-image. This mirrors mongod re-evaluating the predicate when it acquires the document write.

mongod-specific error codes for conversion / string-length / sort expressions

Several aggregation expressions raised a generic TypeMismatch (code 14) where real mongod returns a specific error code. They now match mongod 7.0.12, so a pymongo client sees the same code on a failed operation. This is a Python server refinement — the Rust server already surfaced BadValue for these.

Fixed

• $toInt / $toLong / $toDouble / $toDecimal on an unparseable numeric string now raise ConversionFailure (241) instead of 14.

• $convert with an unknown target type name now raises code 2 (“Unknown type name: …”) and is not swallowed by onError (a query-compile error, matching mongod), instead of 14.

• $sortArray on a non-array input now raises Location2942504 instead of 14.

• $strLenCP / $strLenBytes on a non-string argument now raise Location34471 / Location34473 instead of 14.

$toInt and $convert enforce int32 / int64 overflow bounds

$toInt and $convert (to int / long) never range-checked their result: $toInt: 1e30 returned an unbounded Python integer, and a value larger than the target type silently widened instead of overflowing. mongod errors (241, “Conversion would overflow target type in $convert”) — or routes to $convert’s onError. SecantusDB now does the same on both servers.

$toInt also now yields an int32 (a plain int on the wire) rather than preserving an int64 input’s type, matching mongod, which always narrows to int. Non-finite doubles (inf / nan) overflow rather than raising an uncaught Python error. The Python server carries mongod’s 241 code; the Rust core defers the overflow cases to BadValue. Valid in-range conversions are unaffected. Three-way mongod 7.0.12-verified.

Fixed

• $toInt / $convert (int/long) error on an out-of-range or non-finite value (mongod 241, caught by $convert’s onError) instead of returning an unbounded / silently-widened integer, and $toInt narrows an int64 input to int32 like mongod (both servers).

Expanded change-stream events now match mongod field-for-field

showExpandedEvents change streams now reproduce mongod 7.0.12’s event shapes exactly, on both servers. Expanded update events always carry disambiguatedPaths — an empty document when nothing was ambiguous — and plain streams never do. dropIndexes events describe the dropped index in full ({v, key, name}, with the key spec captured at drop time), matching createIndexes. And the Rust server now emits dropIndexes events on the dropIndexes: "*" path at all — its drop_all_indexes previously skipped the oplog, so drop_indexes() from a driver produced no event.

Fixed

• Expanded update events carry disambiguatedPaths (both servers); the key is correctly absent without showExpandedEvents.

• dropIndexes events describe the dropped index in full on both servers, and the Rust server emits them for dropIndexes: "*".

$dateAdd / $dateSubtract / $dateTrunc validate their integer arguments

The date-arithmetic operators mishandled a non-integer amount / binSize: a whole double (2.0) was over-rejected, and a bool was silently coerced to 1. mongod accepts an integer or a whole double, and rejects everything else: a fractional double / bool / non-numeric amount is Location5166405 (“$dateAdd expects integer amount of time units”), a non-integer binSize is Location5439017, and a non-positive binSize is Location5439018. Both servers now match.

The Python server carries mongod’s codes; the Rust core (via a new date_int helper) now accepts a whole-double argument rather than deferring — so the Rust server no longer rejects a valid amount: 2.0 — and defers the invalid cases. Three-way mongod 7.0.12-verified.

Fixed

• $dateAdd / $dateSubtract accept a whole-double amount and reject a fractional / bool / non-numeric one (Location5166405); $dateTrunc accepts a whole-double binSize and rejects a non-integer (Location5439017) or non-positive (Location5439018) one (both servers).

Date and misc aggregation operators match mongod’s error codes

Continuing the operator error-code sweep, a set of date and miscellaneous aggregation operators now raise mongod 7.0.12’s exact error code instead of a generic TypeMismatch (14) — and two more silent accepts are closed ($dateToString on a non-date, and $dateDiff with a missing endDate parameter, both of which returned a value where mongod errors). Both the Python and Rust servers are fixed (the Rust core defers each case — $dateToString and $dateDiff needed Rust-side fixes to stop computing null).

Fixed

• $dateToString / $dateToParts on a non-date date → Location16006 ($dateToString was a silent null; a null/missing date stays valid).

• $dateFromString with a non-string dateString → ConversionFailure (241).

• $dateAdd / $dateSubtract / $dateTrunc with an unknown unit → code 9 (“unknown time unit value”).

• $dateDiff with a missing startDate/endDate/unit parameter → Location5166303/5166304/5166305 (was a silent null for a missing endDate; a present-but-null parameter still yields null).

• $let referencing an undefined variable → Location17276.

• $switch with no branches → Location40068.

• $ifNull with fewer than two arguments → Location1257300.

• $getField / $setField with a non-string field → Location5654602 / Location4161107.

• $sortArray with an invalid sortBy → Location2942507.

• $convert with a missing input/to parameter → code 9.

$densify validates its range spec

The $densify stage didn’t validate its range. A date unit applied to a numeric field leaked a raw Python TypeError (adding a timedelta to an int), a bool step was silently coerced to 1, a non-positive step and malformed bounds (a bad string, a wrong-length array, a descending array) were quietly accepted or mis-handled. mongod rejects each with a specific code: a numeric value under a date unit is 6053600, a bool step is 14, a non-positive step is 5733401, a bounds string that isn’t "full" / "partition" is 5946802, a bounds array that isn’t exactly two elements is 5733403, and a non-ascending bounds array is 5733402. A fractional step (1.5) is still accepted. Both servers now match.

The Python server carries mongod’s codes; the Rust core defers every invalid case (bool step included) so the Rust server rejects them too. Three-way mongod 7.0.12-verified.

Fixed

• $densify rejects a date unit on a numeric value (6053600), a bool step (14), a non-positive step (5733401), and a malformed bounds string / array (5946802 / 5733403 / 5733402), instead of leaking a Python TypeError, coercing a bool, or silently mis-handling the range (both servers).

Range operators now order embedded documents, like mongod

$gt / $gte / $lt / $lte against an embedded-document bound returned nothing on both the Python and Rust servers — {a: {$gt: {x: 1}}} matched no documents at all — because Python’s operator.gt raises TypeError on two dicts (swallowed to a silent no-match) and the Rust matcher treated any document operand as an unconditional no-match. mongod orders embedded documents field-by-field (first differing key compares as a string, else recurse into the value, else the shorter document sorts first), so those queries should have matched. Found while triaging the driver-gauge results and verified against a live mongod 7.0.12 probe; now three-way parity (Python == Rust == mongod). The type bracket is preserved — a document-valued field still never matches a scalar bound, and vice versa.

Fixed

• Range operators order two embedded documents field-by-field (both servers) instead of matching nothing.

Docs: the benchmark pages get charts

docs/benchmark.md and docs/concurrency.md now open their result sections with inline SVG charts — grouped latency-multiplier bars against a mongod = 1x reference, and the three-server concurrency-scaling lines — theme-aware for furo’s light and dark modes (palette validated for both surfaces), with native tooltips and the tables kept as the data view. Matches the charts on secantusdb.com/performance.html.

Aggregation expressions reject a bool where a number is expected, like mongod

The bool-as-int cluster reaches the aggregation expression engine. Because Python’s bool is an int subclass (and the Rust core mapped Boolean straight to 0/1), a bool argument slipped through the numeric checks of eight operators and was computed instead of rejected: $round/$trunc treated true as a decimal place, $arrayElemAt/$slice/$indexOfArray/$substrCP treated it as an index, and $sortArray as a sort direction. Every one is a parse error in real mongod — a bool is not a number — and both servers now say so.

The Python server reports mongod’s exact per-operator codes ($round/$trunc 16004, $arrayElemAt 28690, $slice 28725/28727, $sortArray 2942507, $substrCP 34450/34452, $range 34443/34445/34447, $indexOfArray 40096) and messages; the Rust server surfaces BadValue. Found while sweeping the aggregation surface for the same root cause as the $inc/$mul and $pop/$position/$slice/$bit clusters; three-way mongod 7.0.12-verified. $range already rejected a bool (with a generic code) and now carries mongod’s per-argument code.

Fixed

• $round, $trunc, $arrayElemAt, $slice, $sortArray, $substrCP, $range, and $indexOfArray reject a bool argument with mongod’s exact error code instead of coercing it to 0/1 (both servers).

$sum / $avg / $max / $min as expression operators

MongoDB 5.0 made $sum, $avg, $max, and $min usable as ordinary expression operators — over an array or a single value, anywhere an expression is accepted (e.g. inside $project/$addFields), not only as $group accumulators. SecantusDB previously rejected these as unknown expression operators (code 168); they now compute, matching real mongod 7.0.12.

Added

• $sum / $avg / $max / $min as expression operators. An array argument reduces over its elements; a scalar is a single value; a missing/absent argument contributes nothing. $sum/$avg ignore non-numeric elements ($sum of an empty/all-non-numeric input is 0, $avg is null); $max/$min order by BSON cross-type order and ignore null (empty → null).

Notes

• Implemented on the Python server (the pymongo conformance target). The Rust core defers these to Python, so the embedded Rust server does not yet compute them — tracked in tasks/backlog.md.

$facet validates its spec instead of leaking on a malformed sub-pipeline

The $facet stage didn’t validate its spec. A sub-pipeline element that wasn’t a stage document ({a: [5]}) leaked a raw Python TypeError, an empty {} spec and a nested $facet were silently accepted, and a non-array sub-pipeline gave a generic error. mongod rejects each: an empty / non-object spec is 40169, a non-array sub-pipeline is 40170, a non-object stage element is 40171, and a $facet nested inside a $facet is 40600. Both servers now match.

An empty sub-pipeline ({a: []}) remains valid. The Python server carries mongod’s codes; the Rust core defers every invalid case (empty spec and nested $facet included) so the Rust server rejects them too. Three-way mongod 7.0.12-verified.

Fixed

• $facet rejects an empty / non-object spec (40169), a non-array sub-pipeline (40170), a non-object stage element (40171), and a nested $facet (40600), instead of leaking a Python TypeError or silently accepting the malformed spec (both servers).

Array-index operators accept a whole-number double, like mongod

$arrayElemAt, $slice, and $indexOfArray rejected every non-integer index, but mongod accepts a whole-number double ($arrayElemAt: [[...], 2.0] → element 2; -1.0 → last element) and rejects only a fractional one. SecantusDB now matches: a whole double is coerced to the integer index (so both servers compute the same element), and a fractional double raises mongod’s per-operator code — $arrayElemAt 28691, $slice 28726 (second arg) / 28728 (third arg), $indexOfArray 40096.

This mattered beyond fidelity: the fix had to land on both servers together — had only the Python engine learned to accept 2.0, the two servers would have disagreed on a valid index (Python returning the element, the Rust server erroring). Ground truth probed against mongod 7.0.12; three-way verified.

Fixed

• $arrayElemAt, $slice, and $indexOfArray accept a whole-number double index (coerced to int) instead of returning null/-1, and reject a fractional double with mongod’s exact per-operator error code (both servers).

Whole-number-double acceptance completed for $substrCP, $range, and $round/$trunc

Finishes the whole-number-double sweep begun for the array-index operators. Like mongod, $substrCP (start/length), $range (start/end/step), and the precision argument of $round/$trunc now accept a whole-number double (coerced to the integer) and reject a fractional one with mongod’s exact per-argument code rather than rejecting every non-integer. So $range: [0.0, 5.0, 1.0] yields [0,1,2,3,4] on both servers, while $range: [0, 5.7] raises 34446.

The Python server carries mongod’s codes — $substrCP 34451/34453, $range 34444/34446/34448, $round/$trunc place 51082 — and the Rust core coerces the whole double (computing the same result) or defers the fractional case to BadValue. As with the array operators, the coercion had to land on both engines together so they never disagree on a valid whole-double argument. Three-way mongod 7.0.12-verified.

Fixed

• $substrCP, $range, and $round/$trunc (precision) accept a whole-number double argument (coerced to int) and reject a fractional one with mongod’s exact error code, instead of rejecting all non-integer values (both servers).

$group accumulators stop coercing non-numeric values

$sum and $avg accumulated whatever the input expression produced — a bool folded in as 1, and other non-numeric values either coerced or leaked a Python error. mongod ignores non-numeric operands entirely: $sum of a group with no numeric value is 0, $avg is null. Both servers now match.

$min and $max compared values with Python’s native < / >, which raises on a cross-type pair (a number vs a string) — so a mixed-type field errored where mongod returns a real extreme. They now ignore null / missing and order every other value by BSON cross-type order (number < string < bool < …), so $max over [10, "hi", true] is true, matching mongod. On the Rust server these mixed-type groups previously deferred to a BadValue; they now compute. Three-way mongod 7.0.12-verified.

Fixed

• $sum / $avg ignore non-numeric operands (string / bool / null / missing) instead of coercing or erroring — an all-non-numeric group yields 0 / null like mongod (both servers).

• $min / $max order mixed-type values by BSON cross-type order and skip null / missing, instead of raising on a cross-type comparison (both servers).

$in / $nin validate their argument instead of leaking or silently no-matching

$in and $nin never checked their argument. A non-array ({a: {$in: 5}}) leaked a raw Python TypeError, and an array element that was a document with a $-prefixed key ({$regex: …} or {$x: 1}) silently matched nothing. mongod rejects both with BadValue: “$in needs an array” and “cannot nest $ under $in”. Both servers now do the same.

A BSON regex literal (/x/) and a plain subdocument element remain valid. The Python server carries mongod’s BadValue; the Rust core defers these cases so the Rust server rejects them too. Three-way mongod 7.0.12-verified.

Fixed

• $in / $nin reject a non-array argument (“needs an array”) and an array element that is a document with a $-prefixed key (“cannot nest $ under $in”) with BadValue, instead of leaking a Python TypeError or silently matching nothing (both servers).

$inc / $mul reject a non-numeric argument, like mongod

$inc and $mul silently computed with a bool argument — {$inc: {n: true}} added 1 and {$mul: {n: false}} multiplied by 0, because Python’s bool is an int subclass — and the Python engine raw-raised a ValueError/TypeError on a string or null argument instead of a clean coded error. mongod rejects any non-number argument with Cannot increment with non-numeric argument: {field: value} (code 14). Both servers now reject it: the Python server raises mongod’s exact message and code; the Rust server surfaces BadValue (the standing update error-code gap), but neither silently computes a wrong result. Found while triaging the driver-gauge update operators; three-way mongod-verified.

Fixed

• $inc / $mul by a bool, string, or null argument is rejected instead of computing a wrong value (both servers); the Python server reports mongod’s code 14 and message.

$indexOfBytes / $indexOfCP validate their start / end index

The $indexOfBytes and $indexOfCP operators mishandled a non-integer start / end index: a whole double (2.0) was silently ignored (the whole expression returned -1), and a bool was coerced to an integer. mongod accepts an integer or a whole double, and rejects everything else: a fractional double, a bool, or a non-numeric index is Location40096 (“requires an integral … index”), and a negative index is Location40097 (“requires a nonnegative … index”). Both servers now match.

The Python server carries mongod’s codes (reproducing its verbatim missing-space message quirk); the Rust core defers the invalid cases and now computes a whole-double index rather than returning -1. Three-way mongod 7.0.12-verified.

Fixed

• $indexOfBytes / $indexOfCP accept a whole-double start / end index and reject a fractional / bool / non-numeric index (Location40096) or a negative one (Location40097), instead of silently returning -1 or coercing a bool (both servers).

$jsonSchema grows mongod’s full keyword surface — and rejects what mongod rejects

The $jsonSchema query operator now covers every keyword real mongod accepts, on both servers, with semantics pinned by a live probe against mongod 7.0: multipleOf (fmod semantics, fractional divisors included), tuple-form items with additionalItems (false / schema / absent), and the title / description metadata keywords (accepted and ignored, with mongod’s string type check). Exclusive bounds move to the draft-4 semantics mongod actually implements — exclusiveMinimum / exclusiveMaximum are booleans that sharpen minimum / maximum to a strict bound, and the draft-6 numeric form is rejected at parse time (the previous numeric treatment was a silent divergence).

Just as important is what gets rejected: schema keywords are now validated at parse time, recursively through every sub-schema, with mongod’s verbatim codes and messages — an unknown keyword or a known-but-unsupported one ($ref, $schema, default, definitions, format, id) is 9 FailedToParse, and a type violation (non-number multipleOf, non-boolean exclusive bound, non-string metadata, non-object schema) is 14 TypeMismatch — before a single document is scanned, even on an empty collection. Previously both servers silently ignored anything they didn’t recognise, so a typo’d keyword matched everything.

Added

• $jsonSchema keywords multipleOf, tuple-form items + additionalItems, and title / description, on both servers, with curated parity coverage.

• Parse-time recursive keyword validation on both servers (query._check_json_schema_keywords / secantus_core::query:: json_schema_keyword_error), with mongod’s verbatim errors.

• QueryError carries a code / code_name (default 2 BadValue), so parse-time errors with documented distinct codes surface faithfully through find, update, and delete write-error paths.

Changed

• $jsonSchema exclusive bounds follow draft-4 (boolean) semantics, matching mongod; the draft-6 numeric form now errors instead of silently applying.

Fixed

• A $jsonSchema with a mistyped or unsupported keyword no longer silently matches every document — it errors exactly as mongod does.

$limit and $skip validate their argument like mongod

The $limit and $skip stages coerced their argument naively (int(spec)), so a range of invalid inputs silently produced wrong results instead of the error mongod raises: $limit: 0 returned nothing (mongod: “the limit must be positive”), $limit: -1 did a Python negative-slice, and a bool or fractional double was quietly truncated/coerced. The Rust server had the mirror-image bug — it rejected a valid whole-number double ($limit: 2.0) because its integer coercion didn’t accept doubles, while still coercing a bool to 1.

Both stages now match mongod: a whole-number double is accepted (coerced to the count), and a bool, a fractional double, or a negative value is rejected — the Python server with mongod’s exact codes ($limit 5107201, $skip 5107200, plus 15958 for a zero $limit), the Rust core deferring to BadValue. $skip: 0 stays valid. Three-way mongod 7.0.12-verified.

Fixed

• $limit / $skip accept a whole-number double and reject a bool / fractional / negative argument (and $limit a zero) with mongod’s exact error code, instead of silently coercing it (Python) or rejecting a valid 2.0 (Rust).

Log-family domain errors now match mongod

An out-of-domain argument to the log family — $ln or $log10 of a non-positive number, $log with a non-positive argument or a base that is non-positive or 1, $sqrt of a negative number — now raises mongod’s exact Location error (28766 / 28761 / 28758 / 28759 / 28714, messages verbatim from a mongod 7.0.12 probe) on both servers, instead of silently returning null. NaN inputs now propagate as NaN (IEEE, matching mongod); null and missing still yield null.

Fixed

• $ln / $log / $log10 / $sqrt out-of-domain arguments error exactly as mongod does, on both servers (the Rust engine defers those cases so the Python error surfaces).

$log rejects a non-numeric argument or base

$log type-checked neither of its operands: a string argument or base leaked a raw Python TypeError, and a bool was silently coerced to 1 / 0. mongod rejects both — a non-numeric argument is Location28756, a non-numeric base is Location28757 — before the positive-domain check, while null still passes through as null. Both servers now match, completing the math-operator type-guard family (the unary ops landed in the previous slice).

The Python server carries mongod’s codes; the Rust core defers these cases (bool included, reusing the math_float helper) so the Rust server rejects them too. Three-way mongod 7.0.12-verified.

Fixed

• $log rejects a non-numeric (incl. bool) argument (Location28756) or base (Location28757) instead of coercing a bool or leaking a Python TypeError (both servers).

$median and $percentile land on both servers — no t-digest required

The $median and $percentile group accumulators — and their expression forms over arrays — now run on both the Python and Rust servers, with semantics pinned by a live probe against real mongod 7.0.12. On bounded data mongod’s “approximate” method resolves to a discrete percentile — sorted[max(0, ceil(p·n) − 1)], returned as a double — so no approximate t-digest sketch is needed and the two engines agree exactly: values collect from int, long, double, and Decimal128 inputs (as doubles), bool and NaN are excluded, and an empty input yields null (median) or per-p nulls (percentile), all exactly as mongod behaves.

Spec validation carries mongod’s verbatim codes and messages: a missing method / input / p field is 40414, a method other than "approximate" is rejected with mongod’s exact wording, a non-array p is 7750301, and an out-of-range p value is 7750303.

Added

• $median / $percentile as $group accumulators and as expression operators, on both servers, with curated parity coverage, unit tests, and wire tests against each server.

$mod matches mongod on floats, bools, and error cases

A fourth query-operator bug from the driver-gauge triage, this time in $mod. mongod truncates both the field value and the divisor toward zero to integers, excludes bool (bool is not a number for $mod), and uses C-style truncated modulo (sign of the dividend). Both servers diverged: they matched a bool field ({a: {$mod: [2, 1]}} wrongly matched a: true), didn’t truncate non-integer floats, and Python’s floored % disagreed on negatives — and the Rust server outright errored (BadValue) on a double-valued field, aborting the whole query. Now both engines truncate value and divisor, exclude bool, compute C-style modulo, and raise mongod’s errors for a zero divisor and a malformed spec — verified three-way against a live mongod 7.0.12 probe.

Fixed

• $mod truncates float values and divisors toward zero, excludes bool, and uses C-style (truncated) modulo, matching mongod on both servers.

• The Rust server no longer errors on a $mod query against a double-valued field.

• $mod with a zero divisor or a malformed spec raises like mongod.

More mongod error codes for $zip / $arrayToObject / $replaceOne / $dateDiff

A second batch of aggregation expressions that raised a generic TypeMismatch (code 14) now return mongod 7.0.12’s specific error code, so a pymongo client sees the same code. This clears the named-operator error-code rows from the divergence catalog’s Tier 3. Python-server refinement (the Rust core already defers each case, so the Rust server surfaced BadValue — unchanged).

Fixed

• $zip with a non-array inputs now raises Location34461, and with a non-array element inside inputs Location34468, instead of 14.

• $arrayToObject on a non-array input now raises Location40386, and $objectToArray on a non-document input Location40390, instead of 14.

• $replaceOne / $replaceAll with a non-string argument now raise mongod’s per-argument code — input → 51746, find → 51745, replacement → 51744 — instead of a single generic 51745.

• $dateDiff with an unknown unit now raises code 9 (“unknown time unit value: …”) instead of 14.

$not and $elemMatch validate their arguments

$not accepted any argument: {$not: 5} silently degraded to “not equal to 5” instead of erroring, and an empty {$not: {}} was accepted. $elemMatch accepted a non-object argument and mis-parsed it. mongod rejects both with BadValue: $not “needs a regex or a document” (a non-empty one — an empty document is “cannot be empty”), and $elemMatch “needs an Object”. Both servers now match.

A regex or an operator document under $not, and an object under $elemMatch, remain valid. The Python server carries mongod’s BadValue; the Rust core defers these cases so the Rust server rejects them too. Three-way mongod 7.0.12-verified.

Fixed

• $not rejects a scalar / array / bool / empty-document argument (“needs a regex or a document” / “cannot be empty”), and $elemMatch rejects a non-object argument (“needs an Object”), with BadValue — instead of silently degrading to an equality check or mis-parsing (both servers).

The SQL server reports IntervalStyle at startup

IntervalStyle is one of postgres’s GUC_REPORT parameters — a real server announces it in the startup ParameterStatus set so clients that decode intervals themselves know which style to parse. SecantusDB’s SQL server tracked the setting internally, defaulting to postgres, but never announced it.

That gap is invisible to psycopg’s binary backend, because libpq keeps its own copy of the value, which is why the pinned test configuration never noticed. psycopg’s pure-Python backend trusts the server instead: with the parameter absent it sees a style of unknown and raises NotImplementedError on any query returning an interval, so a client configuration that works against real postgres failed against SecantusDB. Anywhere psycopg falls back to the pure-Python implementation — a platform with no binary wheel, or an explicit psycopg install without the [binary] extra — hit it.

The server now sends IntervalStyle, and the guarantee is pinned at the wire level rather than through a client, so it holds regardless of which psycopg implementation is installed.

Fixed

• The SQL server reports IntervalStyle in its startup ParameterStatus, as real postgres does. Without it, psycopg’s pure-Python backend could not decode an interval value and raised NotImplementedError.

$pow no longer crashes on a negative base with a fractional exponent

$pow with a negative base and a fractional exponent (e.g. $pow: [-2, 0.5]) produced a Python complex number, which is unencodable — it crashed BSON serialization of the response. It now returns NaN, matching mongod. $pow also now validates its operands like mongod: a non-numeric base raises 28762, a non-numeric exponent (including a bool) raises 28763, and a zero base with a negative exponent raises 28764 — instead of silently coercing a bool, or leaking a raw Python TypeError/ZeroDivisionError. Both servers; the Rust core already returned NaN for the complex case (f64::powf) and now defers the bool / zero-negative-exponent cases so both servers agree. Three-way mongod 7.0.12-verified.

This is the first fix from a parallel divergence sweep (recorded in tasks/divergence-catalog.md) that probed the full operator surface against real mongod and turned up a queue of type-coercion / argument-validation gaps.

Fixed

• $pow returns NaN for a negative base with a fractional exponent instead of crashing BSON encode, and rejects a non-numeric / bool operand (28762 / 28763) or a zero base with a negative exponent (28764) instead of coercing or leaking a raw Python exception (both servers).

Projection $slice validates its argument

A projection $slice silently accepted a malformed argument and returned the full or a wrong array. mongod validates it: the valid forms are a number (first / last n) or [skip, limit] with a positive limit; anything else is evaluated as the aggregation $slice expression and errors. A non-number scalar or an array with fewer than two / more than three elements is Location28667 (wrong argument count); a two- or three-element array whose first element isn’t a number is Location28724. A negative [skip, limit] limit and a three-element array are rejected the same way. Both servers now match.

Valid forms — a number, and [skip, limit] with a positive limit (the skip may be negative) — are unaffected. The Python server carries mongod’s codes; the Rust core defers the invalid shapes so the Rust server rejects them too. Three-way mongod 7.0.12-verified.

Fixed

• A projection $slice rejects a non-number scalar / short / long array (Location28667) and a two/three-element array that isn’t [skip, positive limit] (Location28724), instead of silently returning the full or a wrong array (both servers).

$push $sort direction validation and $currentDate boolean acceptance

Two more update-operator divergences from real mongod are closed. A $push with a $sort modifier now rejects any direction that isn’t exactly 1 or -1 (previously an out-of-range value such as 2 silently sorted anyway), and $currentDate now accepts a boolean false (like true, it sets the current Date) instead of wrongly rejecting it. Both are verified against real mongod 7.0.12.

Fixed

• $push $sort now raises code 2 when the whole-element sort direction is a number other than ±1 (“The $sort element value must be either 1 or -1”), when a {field: dir} direction is not ±1 (“The sort element value must be either 1 or -1”), or when the spec is a non-numeric value such as a string, bool, or array (“The $sort is invalid: use 1/-1 …”). A whole-double ±1 (e.g. 1.0, or {field: 1.0}) is accepted and sorts, matching mongod — previously a whole-double scalar sort was wrongly rejected.

• $currentDate: {field: false} now sets the current Date (a boolean false is the same set-Date form as true) instead of raising. A non-boolean scalar argument and a bad or missing {$type: …} now raise code 2 with mongod’s exact message instead of an uncoded “not understood” error.

$gte/$lte: null and $exists match mongod’s semantics

Two query-match correctness bugs that silently returned the wrong documents (no error): a range comparison against null and $exists’s argument truthiness.

{f: {$gte: null}} (and $lte: null) matched nothing; mongod matches documents where f is null or missing — the same set as $eq: null — because null only orders equal to null. Both now do. ($gt/$lt: null correctly match nothing.)

$exists used Python’s truthiness for its argument, so {$exists: ""} / {$exists: []} / {$exists: {}} were read as $exists: false. mongod uses its own truthiness — only false, 0, and null are falsy; an empty string, array, or document is truthy — so those all mean $exists: true. Both servers now match (the Rust truthy no longer treats empty containers as falsy, and its comparison routes $gte/$lte: null to null-equality). Three-way mongod 7.0.12-verified.

Fixed

• $gte/$lte: null match null and missing (like $eq: null) instead of nothing.

• $exists uses mongod’s argument truthiness (empty string/array/document are truthy), not Python’s, so {$exists: ""} means exists-true (both servers).

The Rust server clears all thirteen driver-conformance gauges

Every one of SecantusDB’s thirteen driver-conformance gauges now runs against the Rust server, not just pymongo — pymongo (sync + async), the mongo-go / node / java / kotlin / ruby / rust / php-library / php-driver / c / c++ / .NET drivers — and the Rust server reaches effective conformance parity with the mature Python server across every ecosystem. Three gauges pass perfectly (mongo-rust-driver, .NET, kotlin at 100%), nothing scores below 98%, and no failure is a new Rust-specific divergence: each one traces to a gap that is already out of scope for a single-node surrogate (text / hashed indexes, $where, multi-node transactions and sessions, Atlas search-index management, IPv6) or to a documented driver-side or test-harness artifact the Python server exhibits too. The per-gauge scoreboard and the follow-up triage notes (a handful of assertion failures to diff against the Python-server runs) live in tasks/backlog.md under the R8 entry; each run’s full report is committed as docs/validation-report-<driver>-rust-server.md.

Added

• Committed Rust-server conformance reports for all thirteen gauges under docs/, and a full-sweep scoreboard in the backlog’s R8 entry.

$regex / $options validate their arguments instead of silently ignoring

A $regex / $options query condition wasn’t validated. An unknown option flag ({$options: "z"}) was silently ignored, a non-string $options was interpreted as raw regex flags, $options with no sibling $regex silently matched, and a non-string $regex value leaked a Python error. mongod rejects each: an unknown flag is Location51108 (“invalid flag in regex options: X”), and the other three are BadValue (“$options has to be a string” / “$options needs a $regex” / “$regex has to be a string”). Both servers now match.

Valid flags (imsxu), an empty option string, a plain $regex string, and a BSON regex literal are all unaffected. The Python server carries mongod’s codes; the Rust core defers these cases so the Rust server rejects them too. Three-way mongod 7.0.12-verified.

Fixed

• A $regex query validates its options: an unknown flag is rejected with Location51108, and a non-string $options, an $options without a sibling $regex, or a non-string $regex value with BadValue — instead of silently ignoring the flag, coercing, matching, or leaking a Python error (both servers).

$rename validates its spec instead of corrupting the document

$rename performed no validation, so several invalid specs silently corrupted data or leaked a raw Python exception rather than raising mongod’s error:

• {$rename: {"arr.0": "x"}} (source is an array element) rewrote the array to [null, 2, 3]; {a: "arr.0"} (destination an array element) wrote into the array. mongod rejects both (code 2) — the field cannot be an array element.

• {a: "a"} (same field) and {a: "a.b"} (source/target on the same path) were applied; mongod rejects both (code 2).

• {a: ""} (empty target) created a field named ""; mongod → code 56.

• {a: 5} / {a: true} (non-string target) leaked an AttributeError ('int' object has no attribute 'split'); mongod → code 2.

All now raise mongod’s codes on the Python server (2 for the field/path/type cases, 56 for the empty path) and defer to BadValue on the Rust server — the document is left untouched. Valid renames (including into a new nested path) are unaffected. Three-way mongod 7.0.12-verified.

Fixed

• $rename rejects an array-element source/destination, a same-field or same-path rename, an empty target, and a non-string target — instead of silently corrupting the document or leaking a Python exception (both servers).

The Rust server’s writers stop queueing behind each other

Writes on the Rust server now serialise per collection instead of per server. Each collection gets its own write lock (created on first reference, stable across drop-and-recreate); inserts, updates, deletes, replaces and TTL prunes take only their collection’s lock, so writers to different collections run in parallel where previously every write in the process queued on one global mutex — the flat ~0.5× concurrency scaling the three-way benchmark measured. DDL (index builds, create/drop/rename, collMod) takes the global lock plus the affected collection lock(s), so an index build excludes in-flight writes on its namespace — which the Rust server genuinely needs, because its write path is autocommit per operation and WiredTiger would not surface a DDL-vs-write overlap as a conflict the way the Python server’s per-statement transactions do.

Getting the exactness guarantees right under real overlap took two more pieces, both ported from the Python server’s concurrency work. Every write statement now runs inside its own WiredTiger snapshot transaction — without one, an update could read a document in one implicit transaction and write it in another, and a competitor committing in between was silently overwritten with a value computed from the stale read (a lost update the new stress tests catch reliably). The statement transaction also makes each write atomic: document row, index entries, natural-order rows and oplog rows commit or vanish together, closing a crash-mid-statement window that could previously leave a dangling index entry. And a write that loses a race retries: statement-level WT_ROLLBACK and the bare-EINVAL commit-time conflict (a competitor marking the transaction rollback-only after its last operation) both map to the typed WriteConflict, which plain writes retry unbounded — matching mongod’s writeConflictRetry, with a warning logged every few seconds of continuous retrying — while statements inside a user transaction surface it immediately so the client sees mongod’s statement-time WriteConflict with the TransientTransactionError label.

Changed

• Rust server: CRUD writes serialise per collection (own lock per namespace); DDL takes the global lock plus the affected collection lock(s); the opportunistic oplog prune moved to its own mutex so the write path never takes the global lock. Lock-order rules are documented on the storage struct.

• Rust server: every write statement runs in its own WiredTiger snapshot transaction — statement atomicity (doc row + index entries + natural-order rows + oplog rows commit together) and write-conflict detection across the statement’s read-modify-write.

• Rust server: tailable change-stream waiters are woken after the statement (or user-transaction) commit makes the oplog rows visible, not at emit time inside the still-open transaction.

Fixed

• Rust server: a plain write racing a multi-document transaction on the same document can no longer be lost to a stale-snapshot read-modify- write — the statement transaction turns it into a detected conflict and the write retries to completion, so both increments land.

• Rust server: a commit-time transaction conflict (bare EINVAL from WiredTiger after a competitor marked the transaction rollback-only) now surfaces as the retriable WriteConflict instead of a generic internal error, matching the Python server’s #444 mapping.

• Rust server: two callers racing the lazy collection-UUID mint can no longer mint different UUIDs for the same namespace — the mint runs under the collection write lock with a double-check, and the already-minted fast path is a lock-free read.

Added

• crates/secantus-storage/tests/concurrent_writes.rs — cross-collection writer storms (exact counts), same-collection $inc hammers (exact final value), unique-index races (exactly one winner, typed loser errors), createIndex under write load (index-routed reads must reach every document), a plain-write-vs-transaction race (retries to completion, both effects land), and a transaction-vs-transaction statement conflict (typed WriteConflict, no retry inside the transaction).

$sum / $avg / $max / $min expression operators on the Rust server

The expression-operator forms of $sum, $avg, $max, and $min (added to the Python server in the previous release) now also compute on the embedded Rust server, so both servers support the MongoDB 5.0+ feature. The Rust implementation reuses the group-accumulator numeric-width logic (int32 → int64 → double promotion) and BSON cross-type ordering, so its result — value and type — is byte-for-byte identical to the Python server (pinned by the parity suite).

Added

• $sum / $avg / $max / $min as expression operators in the Rust core. An array argument reduces over its elements, a scalar is a single value, and a missing/absent argument contributes nothing; a Decimal128 element or an extreme that isn’t BSON-orderable still defers to Python.

The Rust server’s reads stop queueing behind writers

Every read the Rust server served used to take the same global storage lock as every write — concurrent readers serialised not just against writers but against each other. Read-only storage methods (find, count, the _id point lookup, collection scans, listers, planners and stats) now run lock-free: each call’s own WiredTiger session gets a consistent MVCC view without blocking, so reads no longer queue behind a bulk insert and a mixed read/write workload stops paying the writer’s lock hold. All mutable Rust state lives in WiredTiger tables or under the dedicated oplog mutex, so the lock was buying readers nothing — correctness under concurrency is carried by the storage schema instead (a fixed set of shared tables that DDL never drops) plus the invariant that index-routed candidates are always re-verified by the exact matcher and doc fetches tolerate not-found.

Making that invariant airtight surfaced three write-ordering fixes worth having on their own. Index maintenance for updates is now a set diff — an update inserts only the entry keys the new document adds and removes only the ones it drops, so a $set of an unindexed field performs zero index-table operations (the old scheme deleted and rewrote every entry, opening a window where a document vanished from an index whose value the update never touched). createIndex backfills its entry rows before writing the registry row, so a reader can never route through a half-built index and miss documents. And every delete-shaped path (delete, TTL prune, capped eviction) removes the document row first and its index entries after, so a stale entry resolves to a skipped not-found rather than an index miss of a still-live document.

Changed

• Rust server: read-only storage methods no longer take the global storage lock — reads run concurrently with writes and with each other under WiredTiger MVCC. The lock now serialises only writes and DDL.

• Rust server: update index maintenance writes the set difference of entry keys (additions before the doc-row write, removals after) instead of delete-all-then-rewrite; updates that don’t change indexed values do no index writes at all.

Fixed

• Rust server: createIndex now makes the index-registry row the commit point of a fully-backfilled index (entries first, registry last), so a concurrent reader can no longer route through a half-built index and return incomplete results.

• Rust server: delete-shaped writes (delete, TTL prune, capped-collection eviction) remove the doc row before its index/natural-order entries, so a concurrent index-routed read can no longer miss a document that is still live.

Added

• crates/secantus-storage/tests/concurrent_reads.rs — four reader threads hammer find / findOne / scans / counts / listIndexes against a live writer doing replaces, delete/re-insert churn and drop/recreate index churn; every served document must decode and match the filter it was returned for.

The Rust server’s oplog bookkeeping gets off the write path

The Rust server’s oplog housekeeping no longer taxes the hot paths. The opportunistic oplog prune that runs on the write path (every 1000 emits, under the writer’s lock) used to decode every oplog row in full just to read its timestamp — an O(entire-oplog) stall for every concurrent writer each time it fired. The retention scan now peeks the timestamp out of the raw BSON bytes without materialising the document, stops dating rows at the first in-window entry (timestamps are monotone in seq, so the expired rows form a prefix), and walks the rest keys-only. hello replies stopped writing to storage entirely: the per-call oplog-meta persist that every driver heartbeat used to pay — a single-row WiredTiger hotspot every concurrent writer contended on — is gone, matching the cure the Python server shipped in 0.5.4b236.

Crash recovery got structurally safer at the same time. The oplog meta row is now written once at close, and recovery treats it as a hint, not the truth: the recovered counters are clamped up past what the oplog and natural-index tables actually contain, so a crash can never lead to a re-minted (duplicate) oplog seq — previously a stale meta row could overwrite live oplog rows after an unclean shutdown. Restart monotonicity of the cluster clock is guaranteed the same way the Python server does it: recovery bumps the clock one full second past everything it can see, which covers any hello-minted timestamp that was never persisted.

Changed

• Rust server: current_cluster_time (every hello reply under the replica-set persona) no longer persists the oplog meta row, and no longer takes the global storage lock — it is a pure in-memory mint under the dedicated oplog mutex.

• Rust server: the write-path opportunistic oplog prune peeks timestamps from raw BSON (no full-document decode), early-stops at the first in-retention row, and collects the remainder of the walk keys-only; startAtOperationTime seq resolution uses the same raw peek.

• Rust server: oplog-meta recovery reconstructs from the newest oplog row with a single reverse cursor step instead of a full-table decode walk.

Fixed

• Rust server: a stale oplog-meta row (the on-disk state a crash leaves behind, since the meta snapshot is written at close, not per emit) can no longer rewind next_seq / next_nat_seq — recovery clamps both counters up past the table maxima, so a reopen can never re-mint an already-used oplog seq (which would silently overwrite a live oplog row) or collide a natural-order entry.

• Rust server: the cluster clock can no longer step backwards across an unclean restart — recovery bumps it one second past the recovered timestamp, the wall clock, and the oplog tail, covering mints that were never persisted.

The Rust server thins aggregation input before decoding it

The aggregation pipeline fetched its input, then decoded every document into an owned bson::Document before the pipeline ran — even documents a leading $skip / $limit / $match was about to drop. A pipeline that narrows its input early (a limit-then-group, a sample, a second filter after the lifted one) paid to materialize rows it never used.

The leading pass-through prefix of a pipeline — $skip, $limit, and a non-leading $match — now runs over the raw BSON before anything is decoded ($match via the same query::matches_raw find uses), so only the survivors that actually reach the first heavier stage ($group / $sort / computed $project / $unwind / …) are decoded. On a 5000-row scan of wide documents feeding [{$limit: 50}, {$group: …}], that measured ~4× faster (it decoded 50 rows instead of 5000). The result is identical to decoding everything and running the same stages — the prefix is order-preserving and reuses the parity-pinned matcher — and any stage the prefix doesn’t handle (or a $match filter the raw matcher defers on) flows through the full decode-and-run path unchanged. The heavier stages themselves still materialize; accelerating those is separate, larger work.

Changed

• Rust server: an aggregation pipeline’s leading $skip / $limit / $match prefix is applied over raw BSON before decode_docs, so the heavier stages decode only the documents that survive the prefix.

The Rust server projects simple field lists without decoding whole documents

Phase 3 of the raw-BSON serving-path work takes the return path off materialization for the common projection shape. A find with a projection still decoded every returned document into an owned bson::Document before projecting it — the last big materialization site now that projection-free find and count run fully on raw BSON.

A pure top-level inclusion projection ({a: 1, b: 1}, optionally with _id: 0) now projects straight off the raw document, decoding only the included fields rather than the whole thing. On a 5000-row scan of wide documents projecting two of twelve fields, that measured ~2× faster. The fast path is byte-identical to the full projection — same fields, same order — so no result changes; anything it doesn’t cover (exclusion, dotted paths, $slice / $elemMatch / $meta, positional, mixed inclusion/exclusion) transparently falls back to the full projection on a decoded document.

Changed

• Rust server: a pure top-level inclusion projection is applied over raw BSON (projection::apply_projection_raw), decoding only the projected fields instead of the whole document. All other projection shapes fall back to the full decode + apply_projection unchanged.

Added

• secantus_core::projection::apply_projection_raw(&RawDocument, spec) — the raw-BSON inclusion-projection fast path, exposed to the parity harness as _secantus_core.apply_projection_raw and cross-checked byte-for-byte against apply_projection on every projection parity case.

The Rust server stops decoding documents just to re-encode them for the wire

The Rust server used to decode every document it served into an owned bson::Document purely to build the wire reply — then immediately re-encode it. The storage scan and the cursor registry already speak raw BSON bytes end-to-end, so a find’s firstBatch and every getMore’s nextBatch were round-tripping through a decode→IndexMap→re-encode step that produced exactly the bytes they started from. That reply-path materialization was one of the two dominant hot spots the profiler found (tasks/rust-perf-findings.md).

Cursor replies now splice the pre-encoded document blobs straight onto the wire. A new secantus_wire::encode_cursor_reply assembles the cursor.firstBatch / cursor.nextBatch BSON array from the stored blobs without decoding them, and the find (no-projection) and non-tailable getMore handlers hand their batches to the server as raw bytes instead of an owned array. The output is byte-for-byte identical to the old path (pinned by a unit test), so no driver can tell the difference — the work saved is pure overhead. This is Phase 1 of the raw-BSON serving-path plan; the change-stream (tailable) path, projected find, and exhaust-cursor streaming keep their existing behaviour for now.

Changed

• Rust server: find (without a projection) and non-tailable getMore no longer materialize their document batch into the reply — the pre-encoded blobs are spliced onto the wire by secantus_wire::encode_cursor_reply, eliminating the reply-path decode→re-encode round-trip. The batch is carried to the server out-of-band via CommandContext::pending_batch (the same idiom as close_connection); the exhaust-getMore streamer reconstructs the batch it needs to reframe.

The Rust server stops decoding whole documents just to filter them

Phase 2 of the raw-BSON serving-path work takes the scan path off owned-BSON materialization. The Rust server’s collection scan used to decode every candidate document into an owned bson::Document — a heap allocation per field — purely to run the query filter over it, then throw that document away for any document the filter rejected. For a selective filter over wide documents that is almost all wasted work: the profiler put this scan-side materialization (with the reply-path decode fixed in the previous release) at the larger share of the serving path’s on-CPU time.

The filter now runs over the raw BSON bytes. A new secantus_core::query::matches_raw walks the document by field name and decodes only the fields the filter actually reaches — a filter on one field of a ten-field document never touches the other nine — reusing every existing operator unchanged. Documents the filter rejects are never fully decoded, and a filter-only find (no in-memory sort) decodes nothing at all; only the matched documents of a post-sorted find are decoded, for their sort keys. The raw matcher is pinned bool-for-bool to the owned matcher (and transitively to the pure-Python matcher) across the entire curated + fuzz + regex + collation parity corpus, so no query answers change.

A selective filter over wide documents that this optimizes — a count or find that scans many rows and keeps few — measured ~2.8× faster on a 5000-row collection scan (one filter field of eleven, rejecting 99.8%).

Changed

• Rust server: find and count scan filtering runs over raw BSON (query::matches_raw) instead of materialising each candidate into an owned Document. Selective filters over wide documents skip decoding the fields they don’t touch; rejected documents and no-sort finds skip the owned-document build entirely. (The update/delete candidate scans still materialise, since a matched document is needed for the write — a later phase.)

Added

• secantus_core::query::matches_raw(&RawDocument, …) — the raw-BSON query matcher, exposed to the parity harness as _secantus_core.query_matches_raw and cross-checked against query_matches on every parity case.

The Rust server matches update/delete candidates over raw BSON too

Completing the raw-BSON match work: update and delete scanned their candidate documents by decoding each one in full to run the filter, then discarded that document for every candidate the filter rejected — the same waste find and count already avoid. Both now match candidates over raw BSON (query::matches_raw, decoding only the filter’s fields) and decode the full document only for a candidate that actually matches (which the write needs anyway). A selective update / delete over a collection scan of wide documents no longer decodes the rows it isn’t going to touch.

On a 5000-row collection scan of wide documents (11 fields) updating the ~10 that match one unindexed filter field, this measured ~4× faster — the baseline decoded all 5000 candidates to update ten. This reuses the same matcher find and count use, already pinned bool-for-bool to the owned matcher (and pure Python) across the query parity corpus — so no query semantics change.

Changed

• Rust server: update and delete candidate scans filter over raw BSON (query::matches_raw) instead of decoding every candidate; only a matched candidate is fully decoded (for the write / oplog). Every scan-matching path — find, count, update, delete — now skips decoding the documents a selective filter rejects.

Rust-server residues closed: $project error fidelity, WT knobs, full $min/$max order

Three long-tracked Rust-server gaps from the rewrite backlog are closed. An unknown expression operator inside an aggregation $project now reports mongod’s stage-specific Location31325 on the Rust server too (a parse-time scan that never mislabels the projection-only $slice / $elemMatch / $meta shapes), completing the context-specific unknown-operator codes on both servers. The embedded RustServer handle grew the WiredTiger knobs the daemons already exposed — cache_size, session_max, and sync_on_commit constructor parameters — so tests can drive non-default storage configs in-process. And the Rust engine’s $min/$max update operators moved onto a direct port of Python’s _bson_lt (order::bson_lt), a single strict-less relation that needs none of the $sort comparator’s transitivity guarantees: bool, Decimal128, NaN, Binary, Timestamp, Regex, Min/MaxKey, and the decoded exotic text types all compute natively now, with only a DBPointer operand still deferring. Range operators accept the exotic text types the same way (Symbol / JS code compare as strings, mirroring pymongo’s decode), so otherwise-fine queries no longer error with BadValue on the Rust server.

Added

• RustServer(..., cache_size=, session_max=, sync_on_commit=) — the embedded handle’s WiredTiger knobs, threading into wt_config exactly like secantusd-rs’s --cache-size / --session-max / --sync-on-commit.

• order::bson_lt in secantus-core — the direct ordering._bson_lt port backing $min/$max, covering the types is_sortable must bar from sorts.

Fixed

• Rust server: an unknown expression operator inside $project reports Location31325 (“Invalid $project :: caused by :: Unknown expression $op”) instead of a generic 2 BadValue, matching the Python server and mongod.

• Rust matcher: JS code / Symbol range operands compare as text and DBPointer / undefined operands are a clean no-match, instead of erroring BadValue.

• Rust engine: $min/$max with bool / Decimal128 / NaN / Binary / Timestamp / Regex operands compute instead of deferring; curated parity cases pin every newly-computed shape.

• invoke rust-parity installs shapely / s2sphere / python-dateutil into its isolated environment, so the geo curated cases run instead of erroring with ModuleNotFoundError.

Rust server: MONGODB-X509 authentication actually works now

The Rust server’s TLS, mTLS, and MONGODB-X509 machinery was in place, but the first end-to-end test showed the peer-certificate DN was extracted in x509-parser’s raw display form — least-specific-first with comma-space separators — so the identity a client’s certificate asserted never matched the user record provisioned for it, and X509 authentication always failed. The extraction now produces the mongod-style RFC 4514 string (most-specific-first, bare commas, short OID names, value escaping), byte-identical to the Python server’s conversion, so a user provisioned against either server authenticates on the other. A full two-stage bootstrap-then-authenticate test now runs against the Rust server, mirroring the Python suite.

Fixed

• Rust server: the MONGODB-X509 peer DN matches mongod’s RFC 4514 form; X509 auth verified end-to-end (TLS handshake, mTLS client-cert requirement, DN identity, and the no-client-cert refusal path).

• Rust server: tailable cursors on capped collections verified live and pinned by a smoke test (the shipped find.rs producer had outlived its “still deferred” note).

• secantus-wt’s build probe also accepts MSVC’s wiredtiger.lib (and .dylib), unblocking the standalone secantusd-rs build on Windows.

Changed

• A 2026-07-17 backlog audit closed five stale Rust-rewrite entries: the find command entry (shipped long since), Phase-4 sub-phase 5e and the storage-keystone engine-selection half (superseded by the two-server model), the keystone wheel-flag flip (already ON in the shipping matrix), and the standalone-binary half of the Rust-package entry (secantusd-rs ships). The crates.io publish and the “recommend Rust by default” call are explicitly flagged as product decisions.

• R8: the mongo-go-driver gauge now runs against the Rust server — 398 passed / 3 failed / 52 skipped (99.3%), unified suite 42/42; the single real failure is the documented, accepted go-harness try_next load-timing artifact. Report at docs/validation-report-go-rust-server.md.

$sample validates its size argument, like mongod

The Python server’s $sample stage coerced its size with a naive int(), so a bool size was treated as 1 and a negative size crashed with a raw ValueError. mongod rejects both — a non-number size with 28746 (“size argument to $sample must be a number”) and a negative size with 28747 (“must not be negative”) — while accepting a fractional double and truncating it. The Python server now matches; the Rust server already rejected these (its $sample lives in the server crate and validated), so this closes the gap on the Python side.

With this, the aggregation numeric-argument trio — $limit, $skip, and $sample — matches mongod on both servers.

Fixed

• $sample rejects a bool size (28746) and a negative size (28747) instead of coercing the bool to 1 or crashing on the negative (Python server).

$size validates its argument like mongod

$size accepted or silently ignored arguments mongod rejects: a negative size returned no match instead of erroring, a bool was accepted as 1 (Python’s bool is an int), and an integer-valued float like 2.0 was wrongly rejected even though mongod accepts it as 2. Both engines now validate the argument the way mongod 7.0.12 does — it must be a number, integer-valued, and non-negative — raising the corresponding parse error (code 2) otherwise, and accepting an integer-valued float. Found while triaging the driver-gauge results; three-way mongod-verified.

Fixed

• $size errors on a negative, non-integer, string, or bool argument (code 2) instead of silently matching nothing or accepting a bool, and accepts an integer-valued float — on both servers.

$sort stage validates its direction values

The $sort aggregation stage didn’t validate its spec. A string direction ({v: "asc"}) leaked a raw Python ValueError, a bool was silently coerced to ascending, a numeric value other than ±1 (0, 2) was treated as ascending, and an empty {} spec was a silent no-op. mongod rejects each: a non-numeric direction is Location15974 (“Illegal key in $sort specification”), a numeric non-±1 is Location15975 (“must be 1 … or -1”), and an empty spec is Location15976 (“must have at least one sort key”). A whole double (1.0) is still accepted as ±1. Both servers now match.

The Python server carries mongod’s codes; the Rust core defers these cases (bool included — it no longer coerces true to 1) so the Rust server rejects them too. Three-way mongod 7.0.12-verified.

Fixed

• The $sort stage rejects a non-numeric direction (15974), a numeric non-±1 direction (15975), and an empty spec (15976), instead of leaking a Python ValueError, coercing a bool, or silently no-op-ing (both servers).

$split validates its arguments instead of leaking a Python error

$split with an empty separator leaked a raw Python ValueError (empty separator), and its type / arity errors surfaced with a generic code. mongod rejects each with a specific Location code: an empty separator is 40087, a non-string first / second argument is 40085 / 40086, and the wrong number of arguments is 16020; a null string or separator still yields null. The Python server now carries these codes; the Rust core already defers every invalid case, so the Rust server rejects them too. Three-way mongod 7.0.12-verified.

Fixed

• $split reports mongod’s Location codes — empty separator 40087, non-string first / second argument 40085 / 40086, wrong arity 16020 — instead of leaking a Python ValueError or using a generic code (both servers).

SQL server: binary COPY, Parse-time inference, full array-literal grammar

COPY … (FORMAT binary) works in both directions: COPY OUT emits the PGCOPY stream (signature/flags header bundled with the first row the way real PG frames it, one CopyData per row, int16 -1 trailer) with each field encoded by its column type through the existing binary result encoders, and COPY IN parses the same layout, decoding fields by the target column’s type.

Untyped parameters get real Parse-analysis type inference — a client that binds a value in binary format with no declared type (psycopg’s Range(empty=True) dump sends OID 0) takes its type from the AST (a cast on the parameter, a cast or range-constructor operand it’s compared with) — and an untyped parameter fed straight to a VARIADIC “any” function (concat) raises 42P18 like real Postgres.

The array machinery reaches PG’s full literal grammar: nested {{…}} multi-dimensional arrays parse, render, and encode/decode in binary (row-major with per-dimension headers), [l:u]= bounds prefixes parse, box[]’s ; delimiter is honoured both ways, int[][][] collapses to the one array type like PG, '{a,b}'::text[] casts materialise real lists (so subscripting and unnest work), array concatenation || concatenates lists, and = any('{1,2}') accepts array-literal operands. E-string literals are now decoded by the engine before sqlglot parses (sqlglot’s half-decoding was lossy for E'\\x5c'), json and jsonb columns carry their distinct OIDs (114/3802 — plain json’s binary form has no version byte), table row types appear in pg_type (typtype c, with typarray, resolvable via regtype so psycopg’s TypeInfo.fetch(conn, "<table>") works), and the name/aclitem types exist with their real OID pairs.

Range text literals follow PG’s quoting rules exactly: quoted bound tokens with ""/\X escapes parse, embedded quotes/backslashes double on render, ASCII-only whitespace trimming (Python’s unicode-aware .strip() corrupted NBSP/NEL bounds), and a user-declared range constructor result describes with its minted OID. psycopg’s range, multirange, json, array, and string suites all pass.

Added

• GROUP BY 1 positional references resolve to the select-list expression (42P10 when out of range); computed GROUP BY keys beyond the aggregation engine’s operators (GROUP BY col = ascii(x), substr(…)) evaluate per-doc in Python before the pipeline, typed from the source expression.

• bytea params substitute through ::bytea / ::bytea[] casts so equality against computed bytea values compares bytes (text-format arrays decode at Bind); (x::box[])::text renders with the element’s rules.

Fixed

• Multi-statement COPY strings raise 42601 (ProgrammingError), bytea cast failures 22P02 (was XX000), and the binary array encoder covers varchar[]/bpchar[] (1015/1014).

SQL server: composite records end to end, typed array parameters, NaN semantics

The composite/record machinery closes its remaining conformance gaps. Array-of-composite elements binary-encode as real records (not JSON), a binary composite parameter with a NESTED composite field recurses by the field’s minted OID instead of crashing, a registered dumper’s record text literal ('("(foo,10)",20)'::"-x-€") parses on INSERT — including nested and zero-field ('()') types — and a declared-composite table column reports its minted OID so a registered psycopg loader parses nested fields by their reflected types. Anonymous row(…) records now carry each field’s SQL type OID from the source expression: an untyped literal embeds unknown (705, loads as bytes like real PG), ::text embeds 25, ::bytea 17 with real binary bytea, and int literals type int4. VALUES rows of records describe as RECORD, and array_agg types as the element’s real array type (jsonb_agg/json_agg keep json) — psycopg’s CompositeInfo.fetch of a zero-field type depends on it. Parse, Bind, and Describe now run inside the open transaction’s storage scope, so a type created earlier in an uncommitted block is visible to parameter-OID resolution.

Typed array parameters generalize: a text- or binary-format array param with a known array OID (int2[]/numeric[]/inet[]/bytea[]/…) decodes into a typed list and substitutes through a ::tag[] cast, so equality against array[…] constructor values compares element-wise (this closed the numpy, uuid, and network dump/load clusters wholesale). Array casts coerce LIST values to the element’s canonical form, bare array[x::inet, …] constructors describe as the element’s array type, uuid/inet/cidr/macaddr text parameters canonicalise at Bind (psycopg dumps uuids as bare hex), and NaN = NaN is true like Postgres. psycopg’s composite, numpy, uuid, net, and numeric suites now pass (numeric’s exhaustive wide-digit test remains — the Decimal128 34-digit cap).

SQL server: DO blocks, backend termination, richer diagnostics, typed-param polish

A batch of protocol-conformance closers across the psycopg gauge’s error, connection, prepared-statement, cursor, and adapter suites.

DO $$ … $$ blocks run through a minimal plpgsql interpreter: RAISE NOTICE/WARNING/INFO surface as NoticeResponse messages (via the new SQLResult.notices), RAISE EXCEPTION raises with its USING ERRCODE (default P0001), and EXECUTE format(…) runs dynamic SQL whose errors keep their real SQLSTATE. pg_terminate_backend / pg_cancel_backend close the target connection through the live-session registry. ErrorResponse now carries the optional diagnostic identity fields (schema/table/column/ constraint) and a statement position, so a CHECK violation reports its constraint name and a name error renders the LINE 1: … caret context.

Typed parameters and introspection sharpen: pg_prepared_statements reports each statement’s original query text, real prepare time, and regtype parameter names (with array typing); DEALLOCATE ALL clears the extended-protocol registry; INSERT parameters infer their type from the target column at Parse (so an untyped %s into a jsonb column types correctly); ->/#> type as jsonb and ->>/#>> as text, and integer JSON subscripts (-> 1) index arrays. numeric renders in plain positional form (1.1E+2 → 110, matching numeric_out), NaN = NaN holds, generate_series(…)::int4 casts each element, the East-Asian client encodings Python can convert are accepted, format('%s/%I/%L', …) works, and max_prepared_transactions reports non-zero so drivers’ 2PC probes pass.

Added

• pgwire.error_response diagnostic fields + statement position; notice_response full severity/sqlstate.

• errors.SQLError carries diag / position.

• SQLResult.notices, rendered as NoticeResponse in both protocols.

• TableDef.temp (CREATE TEMP TABLE — reflected in error schema).

SQL server: idle_in_transaction_session_timeout

A connection left idle inside an open transaction block longer than the idle_in_transaction_session_timeout GUC (milliseconds; 0 = disabled, the default) is now terminated with a FATAL 25P03 — the connection’s blocked read for the next command is bounded by the timeout, and exceeding it aborts the open transaction and closes the socket, exactly as Postgres does. psycopg’s test_right_exception_on_session_timeout (which sets the GUC, sleeps, and expects IdleInTransactionSessionTimeout) passes.

Added

• idle_in_transaction_session_timeout GUC (default 0); the wire loop bounds the next-command read by it while a transaction is open and terminates on timeout.

SQL server: savepoint rollback reverts DDL; wide binary numerics round-trip

ROLLBACK TO SAVEPOINT now undoes schema changes made after the savepoint, not just data writes. A CREATE TYPE / CREATE TABLE / DROP / ALTER inside a savepoint snapshots every catalog collection (they’re tiny), so the rollback restores the pre-savepoint schema — a re-CREATE of the same type then succeeds, and a DROP is undone. Previously only DML target collections were snapshotted, so the catalog change leaked past the abort (psycopg’s test_change_type_savepoint, which creates and rolls back an enum three times, hit “type already exists”).

The binary numeric decoder handles arbitrarily wide values: a wide integral magnitude combined with a large declared scale sized the Decimal context too small and made the final quantize raise InvalidOperation (surfacing to the client as an internal error). The context now spans the full integer + fractional digit count, so test_dump_numeric_exhaustive’s 50-plus-digit values round-trip.

Added

• catalog.ALL_CATALOG_COLLECTIONS; engine._is_ddl drives the catalog snapshot for savepoint rollback.

Fixed

• pgextended._decode_numeric context precision spans the whole value.

SQL server: binary and scrollable server-side cursors

psycopg’s ServerCursor / RawServerCursor (DECLARE … CURSOR / FETCH / MOVE over the wire) work in binary as well as text, and honour scroll semantics. A binary FETCH … FROM <name> rides the extended protocol, so Describe on the FETCH portal now reports the cursor’s columns instead of NoData — previously the server sent DataRows with no prior RowDescription, a protocol violation the client rejected. DECLARE … NO SCROLL is enforced (backward movement raises 55000, a psycopg OperationalError) and SCROLL allows it; a negative bare count (FETCH -2, MOVE -1) scans backward in the default direction, FORWARD -n / BACKWARD -n flip direction, and MOVE ABSOLUTE 0 repositions before the first row (not at the end) like Postgres. A DECLARE body that isn’t a row-returning query (wat, a DDL statement) raises 42601 (ProgrammingError) rather than 0A000. psycopg’s test_cursor_server.py goes from 15 failed / 7 errored to 0.

Added

• _Cursor.scrollable (SCROLL / NO SCROLL); _moves_backward gates NO SCROLL cursors.

• describe_statement reports a FETCH portal’s columns (binary server cursors) and NoData for MOVE.

Fixed

• Negative FETCH/MOVE counts scan backward; MOVE ABSOLUTE 0 positions before-first; a non-query DECLARE body is a syntax error.

SQL server: type modifiers on the wire, E-string fidelity, transaction characteristics

RowDescription now carries real PG type modifiers: select null::varchar(42) describes with typmod 46 (and varchar/bpchar keep their distinct OIDs 1043/1042 instead of folding onto text), numeric(p,s) packs precision and scale — including negative scales, which sqlglot can’t parse and the engine now pre-rewrites through a sentinel — and the bit/varbit/time-family precisions all flow through, so psycopg’s Column.display_size / precision / scale / type_display report like real Postgres. psycopg’s test_column.py goes 35 failed → 0.

Two more conformance holes close alongside. E'…' escape strings interpolated by psycopg’s ClientCursor (any string containing a backslash) were double-unescaped — sqlglot already decodes the simple escapes, and the second pass corrupted \\b into a backspace or raised 0A000 in the INSERT value path; the decoder now finishes only the octal/hex/unicode forms sqlglot leaves raw. And transaction characteristics are honoured end to end: BEGIN ISOLATION LEVEL … / READ ONLY / DEFERRABLE (every spelling, including the ones sqlglot rejects), SET TRANSACTION, and SET SESSION CHARACTERISTICS AS TRANSACTION apply to the transaction (via the SET LOCAL revert machinery) or the session defaults, and the transaction_* GUCs mirror their default_transaction_* values until overridden — psycopg’s set_isolation_level / set_read_only / set_deferrable suite goes 13 failed → 0.

Added

• result.py / pgwire.py: ColumnDesc.typmod carried through both the simple-query and extended-protocol RowDescription emitters.

• typemap.py: cast_type_identity — (oid, typmod) for modifier-bearing cast targets, including arrays (element typmod with the array OID).

• engine.py: _parse_txn_characteristics shared by BEGIN / SET TRANSACTION / SET SESSION CHARACTERISTICS; session.get_setting falls back dynamically from transaction_* to default_transaction_*.

Fixed

• scalar._unescape_estring no longer re-decodes escapes sqlglot already resolved (the test_leak corruption); planner._literal accepts ByteString values in INSERT position.

Aggregation stage-argument validation for $count, $project, and $sortByCount

Three more aggregation stages now reject malformed arguments with mongod’s exact error code instead of silently computing a wrong result. $count enforces that its field is a non-empty string that is neither $-prefixed, dotted, nor the reserved _id; an empty $project specification is rejected up front; and $sortByCount requires a $-prefixed path string or a single-$-key expression object. Both the Python and Rust servers are covered (the Rust core defers each invalid case so the exact code is raised), and each is verified against real mongod 7.0.12.

Fixed

• $count now raises Location40156 (non-string field), Location40157 (empty), Location40158 ($-prefixed), Location40160 (contains .), and Location15948 (_id) instead of accepting the malformed field name.

• $project with an empty specification now raises Location51272 (“projection specification must have at least one field”) instead of returning the input documents unchanged.

• $sortByCount now raises Location40149 (non-string/non-object argument), Location40148 (bare, non-$ path string), and Location40147 (non-expression object) instead of grouping on a constant.

$strcasecmp coerces its operands like mongod

$strcasecmp previously required both operands to be strings and raised a generic TypeMismatch (14) otherwise. Real mongod $toString-coerces each operand first — a number becomes its string form, null becomes the empty string — and rejects only a boolean. SecantusDB now does the same, matching mongod 7.0.12.

Fixed

• $strcasecmp now coerces a non-string operand to a string (null → "", numbers → their string form) instead of raising, so {$strcasecmp: [5, "a"]} returns -1 like mongod. A boolean operand raises Location16007 (mongod’s code). Integer coercion computes on both the Python and Rust servers; the Rust core defers double/date coercion to Python.

$substrBytes rejects a bool index, and $substr is byte-based like mongod

Completing the aggregation bool-as-int sweep: $substrBytes computed a bool start/length index (as_int_like(Boolean) → 0/1) instead of rejecting it. Both servers now reject — the Python server with mongod’s exact codes (16034 for the starting index, 16035 for the length), the Rust core defers to BadValue.

While verifying against mongod, $substr turned out to be mis-aliased: mongod treats $substr as a deprecated alias of $substrBytes (byte-based), but SecantusDB routed it to $substrCP (code-point-based). On multi-byte strings the two diverge, and a bool index reported the wrong code (34450 instead of 16034). $substr now aliases $substrBytes on both servers, fixing the byte semantics and the bool code together. Three-way mongod 7.0.12-verified.

Fixed

• $substrBytes rejects a bool start/length argument with mongod’s codes (16034 / 16035) instead of coercing it to 0/1 (both servers).

• $substr is now a byte-based alias of $substrBytes (matching mongod), rather than code-point-based $substrCP.

$substrBytes rejects a byte range that splits a UTF-8 character

A $substrBytes (or its $substr alias) range whose start or end falls inside a multi-byte UTF-8 character used to return a Unicode replacement character (Python server) or an empty string (Rust server) rather than the error mongod raises. Both servers now reject: the Python server with mongod’s exact codes — 28656 when the starting index is a UTF-8 continuation byte, 28657 when the ending index lands in the middle of a character — and the Rust core defers to BadValue.

The subtlety a fuzz run surfaced: mongod rejects a continuation-byte start even for an empty (length 0) range, which the Rust core’s “is the slice valid UTF-8?” check missed (an empty slice is always valid), so both engines needed an explicit boundary check. A negative start keeps the legacy slice semantics on both engines. Clean character boundaries and clamped past-the-end ranges still compute. Three-way mongod 7.0.12-verified.

Fixed

• $substrBytes / $substr reject a byte range that splits a UTF-8 character (mongod’s 28656 / 28657 on the Python server, BadValue on the Rust server), including an empty range that starts on a continuation byte, instead of returning a replacement character or empty string.

$substrBytes truncates a double index, completing substr numeric fidelity

$substrBytes rejected a non-integer start/length, but mongod accepts any double there and truncates it toward zero (1.7→1, 2.9→2, 0.9→0) — unlike $substrCP, which rejects a fractional double. Both servers now truncate and compute the same substring; a truncated-negative start (-1.7→-1) still falls into the negative-start rejection (50752), and a negative length still means “to the end”.

With this, $substrBytes / $substrCP numeric-argument handling matches mongod across the board — bool rejection, byte-vs-code-point aliasing, whole-double / fractional / truncation semantics, UTF-8-split rejection, and negative indices. Three-way mongod 7.0.12-verified.

Fixed

• $substrBytes accepts a double start/length and truncates it toward zero (matching mongod), instead of rejecting all non-integer values (both servers).

$substr* reject a negative index, like mongod

A negative start (or, for $substrCP, a negative length) silently produced a Python-style negative-index slice — usually an empty or wrong substring — instead of the error mongod raises. Both servers now reject: the Python server with mongod’s exact codes, the Rust core defers to BadValue.

• $substrBytes / $substr negative start → 50752 (“starting index must be non-negative”). A negative length is still fine — it means “to the end”.

• $substrCP negative start → 34455, negative length → 34454.

This completes $substrBytes / $substrCP numeric-argument fidelity (bool rejection, byte-vs-code-point aliasing, whole-double / fractional handling, UTF-8-split rejection, and now negative indices). Three-way mongod 7.0.12-verified.

Fixed

• $substrBytes / $substr / $substrCP reject a negative start (and $substrCP a negative length) with mongod’s exact error code instead of returning a Python-style negative-index slice (both servers).

$toDate rejects a bool instead of coercing it to a date

$toDate (and $convert to date) silently coerced a bool to a date by treating it as 1 / 0 milliseconds. mongod rejects it: a bool is a ConversionFailure (241, “Unsupported conversion from bool to date”), which $convert’s onError still catches. Every other supported source (int / long / double / string / objectId / decimal → date, null → null) is unchanged. Both servers now match.

The Python server carries mongod’s 241 code (through $toDate, which previously re-wrapped it as a generic error); the Rust core now classifies bool → date as a supported-but-failed conversion so $convert’s onError applies on the Rust server too. Three-way mongod 7.0.12-verified.

Fixed

• $toDate / $convert to date reject a bool with ConversionFailure (241) instead of coercing it to a date, and $convert’s onError handles the failure (both servers).

$toLong aggregation operator

The $toLong conversion operator is now implemented, completing the $to* conversion family ($toInt / $toDouble / $toDecimal / $toBool / $toString / $toDate were already present, and $convert: {to: "long"} too). It converts numbers (truncating a double toward zero), numeric strings, and booleans to a 64-bit long, matching real mongod 7.0.12 — so a value beyond the 32-bit range that $toInt rejects converts cleanly, while a value beyond the 64-bit range overflows (code 241, catchable by $convert’s onError). Covered on both the Python and Rust servers (the Rust core computes the numeric cases and defers string / Decimal128 parsing to Python).

Added

• $toLong — previously an unrecognized expression operator (code 168), now converts int / long / double (truncating toward zero) / bool / numeric string to a BSON long; a result outside [-2^63, 2^63-1], or a non-finite double, overflows with code 241.

$trim / $ltrim / $rtrim validate their input and chars arguments

The trim operators silently ignored a non-string chars argument (falling back to whitespace trimming) and reported a non-string input with a generic error. mongod validates both: a non-string input is Location50699 and a non-string chars is Location50700 (each message names the offending value and type). A null / missing input yields null, and — unlike the whitespace default — an explicit chars: null also yields null. Both servers now match.

The Python server carries mongod’s codes; the Rust core defers the non-string cases (so the Rust server rejects them) and now returns null for a chars: null rather than deferring. Three-way mongod 7.0.12-verified.

Fixed

• $trim / $ltrim / $rtrim reject a non-string input (Location50699) or chars (Location50700) instead of erroring generically or silently ignoring chars, and yield null for a chars: null (both servers).

$type validates its argument and accepts whole-double codes

The $type query operator didn’t validate its argument: an unknown alias, an out-of-range or fractional numeric code, and a bool all silently matched nothing instead of erroring, and the Rust engine additionally rejected a valid whole-double code ({$type: 2.0}) that mongod accepts. mongod validates it: a known alias or a numeric code in {-1, 1..19, 127} (a whole double counts) is valid; an unknown alias or an out-of-range / fractional code is BadValue (2, with a {$exists: false} hint for code 0), and a bool / other type is TypeMismatch (14). Both servers now match.

The Python server carries mongod’s codes; the Rust core defers the invalid cases so the Rust server rejects them too, and now computes whole-double codes rather than deferring (so the Rust server no longer rejects a valid {$type: 2.0}). All 22 aliases (including the deprecated ones and number) are recognised. Three-way mongod 7.0.12-verified.

Fixed

• $type rejects an unknown alias / out-of-range / fractional code (BadValue) and a bool (TypeMismatch) instead of silently no-matching, and accepts a valid whole-double numeric code on both servers (previously the Rust server rejected it).

Unary math operators reject non-numeric operands instead of coercing or crashing

$abs, $ceil, $floor, $sqrt, $exp, $ln, $log10, $round, and $trunc never type-checked their operand. A string leaked a raw Python TypeError (surfacing as a generic error, not a clean server error), and a bool was silently coerced to 1 / 0 and computed on. mongod rejects both: a non-numeric operand is Location28765 ($round / $trunc use 51081), while null still passes through as null. Both servers now match.

The Python server carries mongod’s exact codes; the Rust core defers these cases to BadValue (so the Rust server rejects them too, rather than coercing a bool). Whole-double operands and every valid numeric input are unaffected. Three-way mongod 7.0.12-verified.

Fixed

• $abs / $ceil / $floor / $sqrt / $exp / $ln / $log10 reject a string or bool operand with Location28765, and $round / $trunc with 51081, instead of coercing a bool to 1/0 or leaking a Python TypeError (both servers).

$unwind validates its path, includeArrayIndex, and preserveNullAndEmptyArrays

The $unwind stage silently accepted a malformed spec: a non-$-prefixed path, a non-string path, a non-string / empty / $-prefixed includeArrayIndex, and a non-bool preserveNullAndEmptyArrays (which it coerced with Python’s bool()). mongod rejects each: a non-string path is Location28808, a bare path is Location28818, a non-string / empty includeArrayIndex is Location28810, a $-prefixed one is Location28822, and a non-bool preserveNullAndEmptyArrays is Location28809. Both servers now match.

The Python server carries mongod’s codes (including its verbatim double-space message quirk for 28810); the Rust core defers the invalid cases so the Rust server rejects them too. Three-way mongod 7.0.12-verified.

Fixed

• $unwind rejects a non-string / bare path (28808 / 28818), a non-string / empty / $-prefixed includeArrayIndex (28810 / 28822), and a non-bool preserveNullAndEmptyArrays (28809), instead of silently accepting or coercing them (both servers).

$pop / $position / $slice / $bit reject a bool argument, like mongod

A cluster of update-operator bugs from the same root cause as $inc/$mul: Python’s bool being an int subclass. $pop: true was treated as $pop: 1 (pop the last element) on both servers, and $push with $position: true or $slice: true computed on the Rust server (insert at index 1 / keep 1) — all of these are parse errors in mongod. Every one now rejects a bool argument: the Python server reports mongod’s exact codes (9 for $pop, 2 for $position / $slice / $bit) and messages, and the Rust server surfaces BadValue. $pop now also errors on a number other than ±1 (it silently did nothing before). Found while triaging the driver-gauge update operators; three-way mongod 7.0.12-verified.

Fixed

• $pop, $push $position / $slice, and $bit reject a bool argument instead of coercing it to 1 (both servers); $pop errors on a non-±1 value.

[0.5.4b237] — 2026-07-17

SQL: user-defined range types, and enum OIDs in every plan shape

A focused SQL-server release, both slices driven by the psycopg conformance gauge. CREATE TYPE ... AS RANGE brings user-defined range types end to end — catalog rows, the v3 extended protocol’s binary codec path, and range operators over the new types — building on the range/multirange parameter support that shipped in the previous release. And enum result OIDs now survive every plan shape: joins, subqueries, set operations, and computed projections all report the enum’s real pg_type OID in RowDescription (previously only simple scans did), with mood[]-style enum-array columns decoding correctly on the binary path.

Added

• CREATE TYPE ... AS RANGE / DROP TYPE: catalog rows, binary codecs on the extended protocol, range operators over user-defined range types.

• Enum result OIDs in RowDescription across joins, subqueries, set operations, and computed projections; enum-array (mood[]) columns on the binary path.

SQL server: CREATE TYPE … AS RANGE

User-declared range types land: CREATE TYPE textrange AS RANGE (subtype = text) mints the type and its auto-created companion multirange (textmultirange, following Postgres’ naming rule) with allocation-stable OIDs, reflects both through pg_type (typtype r / m, real typarray) and pg_range (rngtypid / rngsubtype / rngmultitypid), and wires the full value path: literal casts ('[a,b)'::textrange, '{[a,b)}'::textmultirange) parse with the declared subtype’s coercion, the type gets its constructor (textrange(lo, hi, bounds)), parameters a registered psycopg dumper declares with the minted OID round-trip in text and binary (PG’s range wire layout), results describe with the minted OID and render/encode as ranges in both formats, and DROP TYPE removes the pair. psycopg’s RangeInfo.fetch → register_range → typed Range values works end-to-end, as does the multirange counterpart.

The statement itself exceeds sqlglot’s parser (it falls back to a raw Command), so the engine intercepts the command tail — the same pattern CREATE DOMAIN uses. Together with the earlier waves this clears psycopg’s custom-range fixtures, which previously errored out of thirty-one range and multirange tests before any assertion ran.

Added

• catalog.py: create_range_type / get_range_type (by range or companion multirange name) / drop_range_type / list_range_types, minted from the stable user-type OID counter; multirange_name_for (Postgres’ rename rule).

• engine.py: the CREATE TYPE … AS RANGE (…) Command interception (subtype resolved via regtype spelling; collation/opclass options accepted, ignored); DROP TYPE drops range types.

• ranges.py: custom_elem parsing/construction for non-builtin subtypes.

• scalar.py: custom range/multirange casts and constructors.

• pgextended.py: binary custom-range/multirange parameter decode and the generic binary range/multirange result encoders; user-type binary params route by catalog kind.

• virtual.py: pg_type + pg_range rows for user ranges; regtype / user_type_* resolution covers them.

SQL server: enum OIDs through every plan shape, and enum-array columns

The minted enum OID now survives every SELECT plan shape. GROUP BY keys, JOIN projections, SELECT DISTINCT, and per-row-evaluated selects (a scalar function alongside the enum column) previously described enum result columns as plain text (25) because the pipeline/evaluated planners flatten output columns to string type tags; the enum identity now travels in a parallel out_enum_types position map so RowDescription reports the mint — and a psycopg register_enum loader fires on those results — in both the simple and extended protocols. array['sad'::mood, …] constructors describe with the minted array-companion OID like the ::mood[] cast already did.

mood[] table columns land too: an array of a declared enum type was previously rejected outright (unsupported column type); it now stores a text array, validates every element against the enum’s labels at write time (22P02), and reports the array-companion OID so a registered loader returns lists of enum members. An array of an undeclared type raises 42704.

Added

• planner.py: out_enum_types on PipelineSelectPlan / EvaluatedSelectPlan, populated by the DISTINCT / GROUP BY / JOIN / evaluated builders; _enum_array_element_name recognises mood[] column declarations; the constant-select array-constructor override gains an enum branch.

• executor.py: _tagged_out_column_descs resolves minted enum OIDs for the string-tag plans (shared by Execute and Describe); _out_column_descs reports the array-companion OID for enum-array columns; enum write validation checks each element of an array value.

[0.5.4b236] — 2026-07-17

Measured everywhere: a concurrent test suite that caught real bugs, three-way benchmarks, and self-hosted docs

SecantusDB now measures itself against real mongod in both dimensions. The per-operation benchmark became a three-way comparison — the Rust server lands at 2.1×–4.5× of mongod, roughly 2.7×–5.2× faster than the Python server workload-for-workload — and a new concurrent test suite (bench.concurrency --server all) sweeps 1–8 parallel writers across the Python server, the Rust server, and mongod. That harness paid for itself immediately: its first runs caught a write-path bug where a WiredTiger transaction marked rollback-only by a concurrent competitor surfaced to clients as a generic InternalError instead of a retryable write conflict, plus a Windows-only ordering bug in the admin console’s recent-connections list.

The conflict story now matches mongod end to end. Commit-time conflicts map to the same retryable WriteConflict machinery as operation-time ones; plain writes retry without a deadline — a client never sees WriteConflict outside a multi-document transaction, exactly like mongod’s writeConflictRetry; and the two hot-path writes to the shared oplog-metadata row (per oplog emit, and per cluster-time mint — the latter ran on every driver heartbeat) are gone, with restart monotonicity guaranteed structurally by a one-second recovery bump of the cluster clock.

The documentation moved home. Both docs trees — the main reference and a new dedicated Rust-server tree — are now built and deployed with the website at secantusdb.com/docs/ and /docs/rust/, wearing the site’s banner; readthedocs.org keeps a pointer banner to the new location, and the release pipeline dropped its four Read-the-Docs legs. The SQL server kept marching on the psycopg gauge: range/multirange parameters as first-class values, enum result OIDs in RowDescription, composite-type materialization, JSON/datetime binary codec fidelity, and a batch of protocol quick-wins.

Highlights

• Concurrency: bench.concurrency --server python|rust|mongod|all with failure-diagnosis instruments (writer log tails, --server-log); measured table in Concurrency.

• Fixed by it: commit-time WT conflicts are retryable WriteConflict (was InternalError); unbounded writeConflictRetry for plain writes; heartbeats no longer write the oplog meta row; admin recent-targets ordering is deterministic under timestamp ties.

• Benchmarks: three-way docs/benchmark.md — Rust server 2.1×–4.5× of mongod per operation.

• Docs: self-hosted at secantusdb.com/docs (main) and /docs/rust (the new nine-page Rust-server tree); RTD carries a moved banner; [project.urls] metadata on PyPI; the weekly validate run regenerates the cross-driver summary and the Rust-server gauge reports.

• SQL: range/multirange parameters, enum OIDs in RowDescription, composite materialization, psycopg JSON/datetime codec fidelity, protocol quick-wins.

Benchmark: the Rust server measured — 2.1×–4.5× of mongod

docs/benchmark.md is regenerated as a three-way comparison (bench.compare_servers): real mongod, the Rust server, and the Python server, six workloads end-to-end through pymongo on on-disk WiredTiger. The Rust server lands at 2.1×–4.5× of mongod per operation and ~2.7×–5.2× faster than the Python server workload-for-workload; the Python server sits at 6×–20.5× of mongod on this run. The Rust docs tree cites the numbers, and its releases page now links the secantusdb-v-filtered GitHub listing (binary releases are pre-releases, so the bare releases page leads with the source-only PyPI release).

Concurrent writers: heartbeats stop writing the oplog meta row; conflict retries are unbounded

Two follow-ups from the concurrency harness. current_cluster_time() no longer persists the oplog meta row on every call — it runs on every hello reply under the replica-set persona (driver heartbeats) and on change-stream high-water-mark minting, so it was a single-row write hotspot; restart monotonicity is now guaranteed structurally (recovery bumps the cluster clock one second past the meta hint, the oplog tail, and the wall clock, so mints that were never persisted can’t be re-minted). And the non-transaction write-conflict retry loop loses its 5-second deadline: real mongod’s writeConflictRetry loops until the write goes through, so a client never sees WriteConflict (112) for a plain write — ours now matches, logging a warning during long retry stretches. Post-fix sweeps show zero client-visible conflict errors at 1–8 concurrent writers.

Fixed

• Storage.current_cluster_time() is write-free; recovery bump keeps restart cluster time strictly monotonic (regression-tested with a simulated crash).

• _retry_write_conflicts: unbounded retry with capped backoff outside user transactions (inside one, conflicts still surface immediately as mongod’s statement-time WriteConflict).

Concurrent writers: commit-time conflicts retry instead of erroring

Under concurrent writers, a WiredTiger batch transaction can be marked rollback-only by a competitor after its last operation succeeded; the conflict then surfaces at commit_transaction as a bare EINVAL with no WT_ROLLBACK marker, which escaped the write-conflict retry wrapper and reached clients as a generic InternalError (code 1). Found by the new three-server bench.concurrency harness. Commit failures now map to the retryable WriteConflictError when WiredTiger reports a rollback reason or the documented rollback-required EINVAL shape; commit failures that are neither (I/O errors, panics) stay loud, per the never-swallow rule. The remaining structural contention (every batch transaction updates the shared oplog-meta row, so writers on different collections still conflict) is recorded in tasks/backlog.md for the WT concurrency plan.

Fixed

• storage._commit_batch_transaction: commit-time rollback-required failures become WriteConflictError (retried outside user transactions; mongod’s statement-time WriteConflict inside them) instead of InternalError.

The concurrency benchmark measures all three servers

bench.concurrency grows a --server python|rust|mongod|all switch — all sweeps the three back-to-back and prints a combined throughput-vs-writers table — plus two diagnosis instruments the first run immediately paid for: failing writers dump their log tails instead of silently zeroing, and --server-log captures the server’s own stdout/stderr. Concurrency now carries the measured end-to-end table (mongod scales to 4.1× at 8 writers; the Rust server holds flat behind its global write mutex; the Python server degrades under the shared oplog-meta hotspot — with conflicts retried and surfaced honestly since the commit-conflict fix this harness uncovered).

Docs: the site banner tops every self-hosted docs page

Both self-hosted docs trees (secantusdb.com/docs/ and /docs/rust/) now carry the standard site banner — SecantusDB · Python DB · Rust DB · Blog · Python docs · Rust docs — via furo’s announcement bar, so the documentation reads as part of secantusdb.com rather than a detached sub-site. The readthedocs.io copies keep their “docs have moved” banner instead (the two are the same announcement slot, switched on the READTHEDOCS build env var).

Docs move to secantusdb.com; Read the Docs carries a pointer banner

The documentation is now self-hosted: the main tree at secantusdb.com/docs/ and the Rust server’s tree at secantusdb.com/docs/rust/, both deployed atomically with every website publish. The release pipeline drops its four Read the Docs legs (release-finalize now waits only for the publish workflow and the PyPI listing, and no longer requires READTHEDOCS_TOKEN); README, and the new [project.urls] PyPI metadata, point at the self-hosted locations. The readthedocs.io copies stay online but every page there now carries a banner linking to the up-to-date docs (furo’s announcement bar, enabled only when READTHEDOCS=True is in the build environment, so the self-hosted build never shows it).

Docs: a dedicated Rust-server documentation tree

The Rust server gets its own Sphinx tree (docs-rust/, built with invoke docs-rust): installation from the prebuilt secantusdb-v* binary archives, the full secantusd-rs CLI-flag and secantusd.toml reference, the embedded RustServer handle, security (SCRAM / X509 / RBAC / rustls TLS), backup and point-in-time recovery via secantusd-rs restore, the crates architecture, conformance numbers, and the binary release track. The tree is pure Markdown (no autodoc — its version is read from the lockstep crate version), so it builds in any bare worktree, and it deploys to secantusdb.com alongside the main docs.

SQL server: composite types materialize — row(), record casts, typed field access

Composite values were half-real: a '(foo,42,3.14)'::testcomp cast passed raw text through with a text OID, so psycopg’s register_composite loaders never fired, row(…) didn’t exist, and (value).field access failed on anything but a table column. The whole path is now materialized: row(a, b, …) builds an anonymous record (rendered (a,b), described as RECORD 2249, with the PG binary record layout on binary cursors); casts to a declared composite parse the record text literal — including quoted/escaped fields and nested records — into the typed, field-named subdocument; a parameter a registered psycopg dumper declares with the minted composite OID round-trips in both text and binary formats; array[…::testcomp] describes with the paired array OID; pg_typeof prints the type’s name; and ('…'::testcomp).bar types as the declared field, not text.

Composite and domain OIDs also switched to the allocation-stable mint that enums got earlier (assigned at CREATE TYPE/CREATE DOMAIN from a persisted counter, never renumbered or reused) — positional minting shifted every type’s OID whenever a lexically-earlier name appeared, sending registered client loaders decoding the wrong type. oid::regtype output now also double-quotes reserved words ("order"), which psycopg’s sql.Literal pastes verbatim. psycopg’s tests/types/test_composite.py goes from 66 failing to 17 (the remainder: binary record edge samples and suite-order effects).

Added

• scalar.py: row(…) anonymous records; composite cast materialization (_composite_from_text / _composite_from_seq with positional remap for row(…)::type); typemap.parse_pg_record_literal.

• pgextended.py: PG binary record encode (_encode_record) and param decode (_binary_record_to_text); minted user-type binary params keep raw payloads until the catalog resolves them at Bind; pg_typeof($N) resolves minted user-type OIDs.

• catalog.py: allocation-stable composite_type_oids / domain_type_oids (shared _mint_user_type_oid counter machinery).

Fixed

• planner.py: constant-select RowDescription overrides for composite casts (minted OID, composite tag), array[…::testcomp] (paired array OID), and composite field access (the field’s declared tag); user-defined type names build as udt DataTypes in parameter substitution.

• virtual.quote_type_name: reserved words double-quote in regtype output.

SQL server: a round of protocol-fidelity fixes from the psycopg gauge

A batch of small wire-protocol and error-surface divergences, each found by running psycopg’s own test suite against the server: to_regtype now accepts double-quoted identifiers (psycopg’s TypeInfo.fetch(conn, sql.Identifier("text")) was returning None); garbage input ("wat") raises a real syntax error (42601) instead of feature-not-supported, so clients map it to ProgrammingError; a non-numeric string bound to an integer or float column surfaces 22P02 invalid input syntax instead of an internal error; COPY TO STDOUT sends one CopyData message per row like a real server (a single all-rows blob made every row after the first vanish in psycopg’s Copy.rows()); a client’s CopyFail aborts the enclosing transaction (INERROR); a bare VALUES (…) answers extended-protocol Describe with its row shape instead of NoData-then-DataRows (a protocol violation that crashes libpq’s stream mode); RowDescription reports fixed-width types’ typlen and encodes column names in the client’s encoding; pg_sleep() sleeps; pg_tables exists; and the transaction-characteristics GUCs (transaction_isolation etc.) report their honest single-node constants.

Together these clear ~60 tests across psycopg’s test_typeinfo (18 → 0), test_cursor_common (27 → 3), test_copy (37 → 26) and test_column (42 → 35) files.

Fixed

• typemap.oid_for_regtype / planner._to_regtype: double-quoted identifier resolution with Postgres case rules (quoted names keep case; built-ins only match lowercase).

• engine.py: bare expression statements → 42601; extended-protocol Describe of VALUES returns the row shape.

• typemap.coerce: int/float coercion failures raise 22P02 (as an exception that is also a ValueError, so soft-fallback callers keep their behaviour).

• pgserver.py: COPY OUT chunks per row; CopyFail marks the transaction failed.

• pgwire.row_description: static typlen table; client-encoding column names (threaded from both the simple and extended paths).

• functions.py: pg_sleep (capped at 30s — our connection threads have no cancel path); session.py: transaction-characteristics GUC defaults.

• virtual.py: the pg_tables system view.

SQL server: the psycopg JSON and datetime suites go fully green

Two of the three biggest failure clusters in the psycopg conformance gauge — tests/types/test_json.py (181 failing) and tests/types/test_datetime.py (259 failing) — now pass completely, taking the gauge headline from 2900 passed (70.3%) to 3473 passed (84.2%) under deterministic test order. The third cluster, test_typing.py (125), was purely environmental: it shells out to a bare mypy, which the gauge venv didn’t carry — mypy now rides the dev extra and all 125 pass.

The JSON cluster came down to one root cause with wide blast radius: json and jsonb values were never parsed at ingress. A '{"a":1}'::jsonb cast passed raw text through, so ->/->> navigation returned NULL and output double-encoded. Casts and json-declared parameters now parse into real JSON values, array[…]::text renders Postgres’ array_out literal instead of a JSON list, E'…' escape strings evaluate (psycopg’s sql.Literal emits them for any string containing a backslash), and the plain-json OIDs (114/199) alias the jsonb tag.

The datetime cluster decomposed into seven root causes, all fixed: temporal parameters substituting as bare text (a datetime param silently compared false against an equal cast literal); interval literals rejecting PG’s unit abbreviations (1s, 5 min, 1d 3h); parser gaps for epoch, infinity, BC dates, non-padded fields and loose UTC offsets; the session TimeZone GUC being ignored on both input and output (including POSIX-inverted numeric zones and set_config(), which now emits ParameterStatus); DateStyle-aware text rendering (German/SQL/Postgres orders); binary encoders using float seconds (a 1µs error at year 9999) and lacking infinity sentinels; and PG-range values beyond Python’s datetime limits, now carried as text via proleptic-Gregorian ordinal math so '9999-12-31'::date + 1 returns 10000-01-01 like a real server. Intervals also gained PG’s justified duration comparison (-1 day +23:59:59.999999 = -0.000001s).

Added

• datetimes.py: proleptic-Gregorian ordinal helpers valid outside [year 1, 9999], infinity/-infinity/epoch sentinels, wide/BC timestamp canonical text + binary wire values, TimeZone-GUC tzinfo resolution (POSIX sign convention, zoneinfo names), loose-input widening.

• intervals.py: PG unit abbreviations (s/sec/min/h/d/w/y/ ms/us, attached forms like 1d), justified total_micros.

• typemap.py: session-bound render context (TimeZone/DateStyle GUCs honoured at output), typed parameter carriers (JsonText/DateText/TimeText/ TimeTzText) that substitute as casts, json OID aliases.

• session.py: case-insensitive GUC name canonicalization (set timezone hits TimeZone); set_config() on a reportable GUC emits ParameterStatus.

• pyproject.toml: mypy in the dev extra (psycopg’s test_typing.py shells out to it).

Fixed

• planner._value_to_node: datetime / date / time / timetz / interval / json parameters substitute as typed casts, not bare string literals — the same treatment Decimal already had.

• pgextended.py: temporal text params convert per their declared OID; binary interval params decode to the interval subdoc; binary timestamp/date encoders use integer-µs arithmetic and PG’s infinity sentinels.

• scalar.py: 'nope'::timestamp raises 22007 instead of silently passing raw text into the binary encoder; ts::text renders through the session-aware renderer; mixed naive/aware datetime comparisons treat naive as UTC; multi-value SET name = v1, v2 (DateStyle) is stored and reported.

• engine.py / functions.py: client_encoding canonicalises on every SET path (utf-8 → UTF8 in ParameterStatus).

SQL server: range and multirange parameters become first-class values

Range-typed parameters used to arrive as raw text and never become range values: select 'empty'::int4range = %s with a psycopg Range parameter silently compared a subdocument against a string and returned false. A parameter declared with a range or multirange OID (or their array forms) now travels as tagged text and substitutes as a ::type cast, so the existing cast coercion turns it into the structured value. Array casts ('{empty,"[1,3)"}'::int4range[]) coerce their elements, untyped literals compared against a range value take the range’s type (Postgres’ context inference), and range::text renders the [a,b) literal.

Equality itself also got Postgres semantics: range bounds store in the subtype’s canonical form regardless of construction path (a daterange(date, date) constructor bound now matches the text cast’s bound; numrange bounds unify int / Decimal / Decimal128), and comparisons go through a representation-independent canonical identity. psycopg’s range and multirange suites drop from 149 failing + 31 errors to 10 + 31 — the remainder being untyped binary parameters (psycopg dumps a bound-less Range(empty=True) with OID 0 in binary; needs Parse-time parameter-type inference) and CREATE TYPE … AS RANGE (both recorded in tasks/backlog.md).

Added

• ranges.canonical / canonical_multirange: representation-independent range identity used by comparisons.

• typemap.TaggedText: the typed-parameter carrier for range/multirange (and array-of-range) declared OIDs.

Fixed

• pgextended.py: text and binary range/multirange parameters (and binary arrays of them) substitute as typed casts; ParameterDescription resolves an undeclared parameter to text like Postgres’ parse analysis instead of echoing 0.

• ranges.make_range: bounds coerce to the subtype’s canonical storage form.

• scalar.py: untyped-literal context coercion against range values; ::range[] element coercion; range::text rendering.

SQL server: enum result OIDs in RowDescription, and real array OIDs for user types

An enum-typed result column used to describe itself as plain text (OID 25), which broke the catalog-driven type registration flow every Postgres driver builds on: psycopg’s EnumInfo.fetch would find the type’s minted OID in pg_type, but no result column ever carried it, so register_enum loaders never fired and enum values always came back as bare strings. RowDescription now reports the same minted OID that pg_type / pg_enum / pg_attribute reflect — the mint moved onto Catalog.enum_type_oids so reflection and the wire layer cannot drift — and the full psycopg round-trip works: fetch the type, register a Python enum.Enum, and SELECT / RETURNING rows come back as enum members.

Chasing the conformance numbers surfaced a second, far larger bug: every user-declared type (enum / domain / composite) reported pg_type.typarray = 0. Clients key array-type registrations on that value, and 0 is INVALID_OID — psycopg’s own suite pops the loader registered under array_oid, which deleted psycopg’s global unknown-oid fallback loader and poisoned every subsequent unknown-OID text load in the process. User types now mint a derived paired array OID (oid + 100000).

Enum values also flow through expressions and parameters, not just table columns, so the cast and Bind paths grew the same fidelity: SELECT %s::mood describes with the enum OID and validates the label (22P02 invalid input value for enum on a label the type doesn’t have), a parameter a registered psycopg dumper declares with the enum OID is label-validated at Bind, oid::regtype::text quotes mixed-case type names the way real Postgres does (psycopg’s ClientCursor pastes that string verbatim as a cast suffix), and %s::mood[] round-trips as a list through the minted array OID in both text and binary formats. psycopg’s enum-adaptation suite (tests/types/test_enum.py, 197 tests) passes completely. On the full psycopg conformance gauge the work takes the headline from 2554 passed (61.9%) to 2900 passed (70.3%) under deterministic test order — +346 tests, including the entire 212-test “unknown oid loader not found” cluster and all 152 enum failures.

Added

• catalog.py: Catalog.enum_type_oids(db) — the single enum-OID mint, shared by pg_catalog reflection (virtual._enum_oids now delegates) and result-column description. OIDs are allocation-stable: assigned from a persisted counter at CREATE TYPE, kept across ALTER TYPE … ADD VALUE, and never renumbered or reused after DROP TYPE — the previous positional mint (base + sorted-name index) shifted every enum’s OID whenever a lexically-earlier type appeared, which would send a client’s registered loader decoding the wrong type.

• executor._out_column_descs: enum-aware (name, Column) → ColumnDesc resolution, used by SELECT (plain, correlated), INSERT / UPDATE / DELETE / MERGE RETURNING, and extended-protocol Describe (statements and RETURNING).

• virtual._pg_type: enum / domain / composite rows carry a derived typarray (oid + USER_TYPE_ARRAY_OID_OFFSET) instead of 0.

• scalar.py / planner.py: casts to a declared enum ('ok'::mood, %s::mood, %s::mood[]) validate labels (22P02) and describe with the enum’s OID (arrays: the paired array OID) in constant selects.

• pgextended.py: a Bind parameter declared with an enum OID is label-validated (22P02); binary array parameters and results handle user-type array OIDs (elements travel as text — an enum’s wire form is its label).

Fixed

• executor.py / engine.py: enum columns in RowDescription report the enum’s OID instead of 25 across the simple-SELECT, RETURNING, and Describe paths. JOIN / GROUP BY / evaluated-expression plans still describe enum outputs as text (their column shape drops the enum tag at plan time) — recorded in tasks/backlog.md.

• scalar.py: oid::regtype::text of a user type quotes names that need it ("CamelCaseEnum") — an unquoted mixed-case name pasted back as a cast suffix folds to lowercase and misses the type.

• virtual.user_type_oid: ::regtype / to_regtype() resolution of a user-declared type name now applies Postgres identifier folding — an unquoted part folds to lowercase ('StrTestEnum'::regtype finds strtestenum), a quoted part keeps its case. psycopg’s EnumInfo.fetch(conn, "MixedCaseName") was returning None, which poisoned its entire enum-adaptation suite.

Admin UI: recent-connections order is deterministic under timestamp ties

TargetStore.recent() ordered by last_used_at alone; on Windows, time.time()’s ~15.6ms resolution makes back-to-back records tie, so the recent-targets list (and the trim that caps the table) could return them in arbitrary order — caught as a Windows-only CI flake. Both queries now break ties by rowid DESC (the later insert), pinned by a regression test that freezes the clock.

[0.5.4b235] — 2026-07-16

Point-in-time recovery, a SQL server with its own gauges, and operator parity across both servers

This is the largest SecantusDB release to date — a month of parallel work, 125 changelog entries. The headline capability is point-in-time recovery: every write already flowed through the oplog, and secantusAdmin.restoreToTimestamp now replays it to reconstruct the database exactly as it stood at any moment inside the retention window — with hot backup archives, base snapshots, and archives portable between the Python and Rust servers. The other headline is housekeeping with teeth: the daemons got distinguishable names (secantusd-py, secantusd-rs, secantusd-py-pg — the old secantusdb console script is gone), and Python 3.10 is now genuinely supported and genuinely tested in CI.

The PostgreSQL-wire SQL server graduated from experiment to measured surface. It now has two external conformance gauges of its own — psycopg 3’s unmodified test suite and the SQLite-originated sqllogictest corpus — and the long tail they surfaced landed alongside them: server-side cursors over the wire, COPY inside transaction blocks, CREATE SCHEMA, LANGUAGE plpgsql function bodies, the full binary codec surface with real Postgres type OIDs, per-statement RBAC reusing the Mongo role model, and SQL’s three-valued NULL semantics carried all the way down the filter-pushdown path.

On the MongoDB side, both servers picked up a wide operator-fidelity batch — the $setWindowFields operator set completed ($derivative / $integral with time units, $locf, $linearFill, $expMovingAvg, range windows), the N-ary accumulators, trigonometric and set expressions, a much larger date toolbox, and dozens of exact-error-code alignments — and the Rust server reached pymongo-suite parity with the Python server (99.5% each). Measurement grew to match: sixteen driver-conformance gauges now run weekly (C, C++, C#, Kotlin, pymongo async, psycopg, and the Rust-server gates joined this cycle), feeding a regenerated cross-driver summary and a new three-way feature-comparison page. One genuine bug that machinery caught — an awaitData wake race that could delay change-stream delivery by a full maxTimeMS — is fixed, alongside security hardening (two admin-UI CVEs, SCRAM-credential leak paths closed, constant-time token comparison).

Highlights

• Point-in-time recovery: secantusAdmin.backupArchive / archiveBaseSnapshot / restoreArchive / restoreToTimestamp on the Python server; the same archives restore on the Rust server via secantusd-rs restore. See Recovery.

• Daemon renames: secantusd-py (MongoDB wire, Python), secantusd-rs (MongoDB wire, Rust), secantusd-py-pg (PostgreSQL wire). The legacy secantusdb / secantus console scripts are removed.

• SQL server conformance gauges: invoke validate-psycopg (psycopg 3’s own suite) and invoke validate-slt (sqllogictest corpus), with the wire-protocol, codec, cursor, COPY, schema, and plpgsql work they drove.

• Both servers: completed $setWindowFields operators, $topN / $bottomN / $firstN / $lastN / $maxN / $minN accumulators, $mergeObjects as an accumulator, trigonometric / set / bitwise expression operators, $dateFromParts, $toDate, timezone-aware date extraction, and mongod-exact error codes for unknown operators.

• Rust server catch-up to parity: index-driven $lookup, views, getLog, killOp, role grants, oplog maintenance commands, IANA timezones in date formatting, and the pymongo suite at 99.5% — level with the Python server.

• Change streams: the awaitData wake race is fixed (a write landing between the producer drain and the wait could stall delivery a full maxTimeMS); resume tokens advance per event even at batchSize 1.

• Security: two admin-UI CVEs fixed plus a stored-XSS; admin.system.users no longer leaks SCRAM credentials; constant-time secret comparison in the PostgreSQL SCRAM and admin-token checks.

• Conformance measurement: sixteen gauges (C, C++, C#, Kotlin, pymongo async, psycopg, sqllogictest, and the two Rust-server gates joined), weekly report + cross-driver-summary refresh, and the Feature comparison page.

• Python 3.10 support, tested per-version in CI.

• Process: changelog fragments (changelog.d/) and release-time version assignment ended cross-PR conflicts on docs/changelog.md and pyproject.toml.

Change streams: awaitData wake no longer misses a write landing mid-getMore

A tailable getMore baselined its awaitData wake predicate on a fresh oplog-tail snapshot taken after draining the change-stream producer. A write landing in the gap between the drain and the wait was counted into that snapshot and never tripped the predicate — the getMore slept its full maxTimeMS with the event already in the oplog, surfacing it only on the post-wait re-drain. On a loaded machine that pushed delivery past the client’s await window (seen as a one-off test_await_data_blocks_then_wakes_on_insert failure in the durable CI lane). The predicate now baselines on the producer’s own consumed position (entry.position_seq, which the drain advances to the tail it actually observed) — any write after that observation wakes or skips the wait, mirroring the Rust server’s wait_for_oplog(position, ...), which was never affected. A regression test pins the interleaving deterministically by landing an insert inside the former race window. A side benefit: a resuming cursor that drains a full filtered batch no longer sleeps its whole maxTimeMS before fetching the next backlog page.

Fixed

• commands.py tailable getMore: wake predicate compares the oplog tail against entry.position_seq instead of a post-drain tail snapshot.

$bit update applies multiple operations (both servers)

The $bit update operator now accepts more than one bitwise operation per field and applies them in order, matching mongod: {$bit: {n: {and: X, or: Y}}} computes (n & X) | Y. Both servers previously rejected any $bit document with more than a single sub-operation. Found by a three-way update differential vs real mongod 6.0.

Fixed

• update.py / secantus-core: $bit iterates every and/or/xor entry in the per-field document (in order) instead of requiring exactly one; an empty $bit document is still rejected, and the int32/int64 result width is preserved as before.

Changelog fragments and release-time version assignment

Concurrent development got much less painful. Previously every PR edited the top of docs/changelog.md’s [Unreleased] section and bumped the single version line in pyproject.toml — two shared lines that made any two in-flight PRs conflict, so merging one forced the others to rebase and hand-resolve the same files. Feature PRs now add a changelog.d/<slug>.md fragment (one entry per file) instead of touching docs/changelog.md, and they no longer bump the Python package version at all — the version is assigned once, at release time, by release-prepare. New fragment files never collide, so parallel sessions stay independent.

Added

• changelog.d/ fragment convention (changelog.d/README.md), a changelog.fragments collator, and an invoke changelog-collate task that folds fragments into ## [Unreleased]. release-prepare runs the collation automatically before it stamps the version.

Changed

• Feature PRs no longer bump the Python version / __version__ (assigned at release) or edit docs/changelog.md directly. The Rust crate version is still bumped per-PR (its buildInfo traceability handle; rare same-session-only collisions). See the Versioning and Conventions sections of CLAUDE.md.

Date extractors error on a non-date input (both servers)

All thirteen date-component extractors — $year / $month / $dayOfMonth / $hour / $minute / $second / $dayOfWeek / $dayOfYear / $week / $isoWeek / $isoDayOfWeek / $isoWeekYear / $millisecond — now raise mongod’s Location16006 (“can’t convert from BSON type … to Date”) when given a present non-date value (a string, a number, a bool, …), instead of silently returning null. A null or a missing field still yields null, as before.

Fixed

• expressions.py / secantus-core: the shared date-operand resolver (_date_operand / date_operand_millis) distinguishes a null / missing operand (→ null) from a present non-date value. The Python server raises Location16006; the Rust server surfaces a generic BadValue on that path (the documented error-code gap). Verified three-way vs real mongod 6.0 (Python zero divergences).

Six more date-component extractors and $dateToParts ISO mode

The aggregation date toolbox picks up the components MongoDB exposes but SecantusDB was still missing: $dayOfYear (1-366), $week (US week number, 0-53, weeks starting Sunday), $isoWeek (ISO-8601 week 1-53), $isoDayOfWeek (1=Monday … 7=Sunday), $isoWeekYear (the ISO week-numbering year), and $millisecond. Each slots in alongside the existing extractors and accepts the same two shapes — a bare date expression or a {date, timezone} object — so a fixed ±HH:MM offset or a named IANA zone (America/New_York) shifts the instant before the component is read. The year-boundary edge cases match mongod: 2026-01-01 (a Thursday) is US week 0, and 2027-01-01 (a Friday) is ISO week 53 of ISO year 2026.

$dateToParts now honours iso8601: true, returning {isoWeekYear, isoWeek, isoDayOfWeek, hour, minute, second, millisecond} instead of the calendar {year, month, day, …} shape. The timezone option applies in both modes, and iso8601: false (or absent) keeps the existing output unchanged.

Both servers gain the operators together, pinned byte-for-byte by the Rust ↔ Python expression parity harness. The named-IANA-zone cases compute natively on the Rust side via chrono-tz.

Added

• expressions.py / secantus-core: $dayOfYear, $week, $isoWeek, $isoDayOfWeek, $isoWeekYear, and $millisecond aggregation-expression operators, each supporting the {date, timezone} object form.

• expressions.py / secantus-core: $dateToParts now supports iso8601: true, emitting the ISO week-based parts document.

Docs: three-way feature comparison (MongoDB vs Python server vs Rust server)

A new Feature comparison docs page decomposes the validation-report pass rates into a per-feature matrix: commands, query/update/expression operators, aggregation stages, accumulators and window functions, index types, collections, change streams, transactions, auth, backup/PITR, and the SQL frontend — each marked supported / partial / missing for real mongod, the Python server, and the Rust server.

Changed

• docs/servers.md: refreshed the stale “what the Rust server doesn’t support” list — the pymongo-suite gap has closed to parity (99.5% both) and the DDL-change-stream-event, large-event-splitting, and timeseries-_id bullets described already-shipped features; replaced with the current gap set (SQL frontend, mapReduce/top, wire-level restoreToTimestamp, session lifecycle no-ops, oracle-deferred operator edges, thinner diagnostics). Dropped the out-of-scope claim that RBAC is unimplemented (both servers enforce it).

• docs/index.md: added the new page to the toctree and quick links, and included the previously-orphaned psycopg validation report in the toctree (it was failing the -W docs build as toc.not_included).

$getField on an absent field now resolves to missing, not null

Reading a field that doesn’t exist with $getField used to hand back an explicit null. Real MongoDB treats an absent field as the missing value — and a $project or $addFields computed field that resolves to missing is omitted from the output document entirely, rather than emitted as null. Both SecantusDB servers now match that: {$project: {r: {$getField: {field: "k", input: "$sub"}}}} over [{sub: {k: 1}}, {sub: {j: 2}}, {}] yields [{r: 1}, {}, {}] — the documents with no sub.k carry no r field at all. A field that is present with an explicit null still returns null and is emitted, so the missing-vs-null distinction is preserved.

The same change makes $$REMOVE behave correctly as a $project / $addFields computed value: the field is dropped instead of leaking the internal removal sentinel.

Fixed

• expressions.py / secantus-core: $getField returns the missing/$$REMOVE marker (not null) for a field absent from its input; on the Rust side that case defers to the pure-Python engine, keeping the parity harness green.

• aggregate.py: $project and $addFields computed fields that evaluate to the missing marker are omitted from the output (an existing $addFields target set to the marker is removed), matching mongod.

$inc / $mul on an explicit-null field now errors like mongod

Applying $inc or $mul to a field that is present with an explicit null value now raises a TypeMismatch (error code 14), exactly as real MongoDB does — “Cannot apply $inc to a value of non-numeric type … of non-numeric type null”. Previously both servers silently coerced the null to 0 and applied the delta, so {$inc: {n: 5}} against {n: null} returned {n: 5} instead of failing. A missing (absent) field is still treated as 0 and the operation applied — that has always matched mongod and is unchanged.

The fix distinguishes an absent field from a present-but-null one: the pure-Python engine raises the coded error directly, and the Rust core defers the null case to the Python oracle so the exact error code is preserved (the Rust server surfaces a generic BadValue, the documented error-code gap).

Fixed

• update.py / secantus-core: $inc / $mul on a field present with an explicit null now errors with code 14 (TypeMismatch) instead of coercing the null to 0. A missing field is still treated as 0 and the operation applied.

Docs: the Rust-server Java gauge report joins the site

invoke validate-java --server rust has been writing docs/validation-report-java-rust-server.md — the mongo-java-driver suite pointed at the standalone Rust server — but the report had never been committed or added to the docs toctree. It now ships alongside the other validation reports (445/2 passed, 99.6%; the two failures are the mapReduce tests, consistent with the Rust server not implementing mapReduce).

Fixed

• java_validation/generate_report.py: the generator emitted the Python-server title and refresh command for both servers; a -rust-server output now gets a (Rust server) title, the --server rust refresh command, and a note that the two-phase spawn boots secantusd-rs.

$jsonSchema uniqueItems bridges cross-type numerics recursively (both servers)

{$jsonSchema: {properties: {arr: {uniqueItems: true}}}} now detects duplicate array elements using MongoDB value equality, which treats int / long / double / Decimal128 as equal when their values match — and does so recursively inside sub-documents and sub-arrays. So an array like [{a: 1}, {a: 1.0}] is correctly rejected (the two documents are equal), matching real mongod 6.0.

Previously only top-level scalar arrays collapsed cross-type numerics ([1, 1.0] was already a duplicate); a cross-type-equal numeric nested inside a document or array element ([{a: 1}, {a: 1.0}]) was wrongly treated as distinct on both servers, because duplicate detection keyed off a raw BSON encoding that differs for int 1 versus double 1.0.

Fixed

• query.py / secantus-core: uniqueItems duplicate detection uses a recursive canonical key (_unique_items_key / unique_items_key) that normalises numerics to a common value form at every nesting level and recurses into sub-documents and sub-arrays, instead of Python structural == or a raw sort-key/BSON encoding.

$mergeObjects as a $group / $setWindowFields accumulator

MongoDB’s $mergeObjects was already available as a $project expression, but not as an accumulator inside $group or $setWindowFields. It now is: SecantusDB merges each group member’s operand document into a single accumulated document, with later documents’ keys overriding earlier ones. A null or missing operand is skipped, a group whose operands are all missing/null yields an empty document {}, and a non-null, non-document operand raises the same Location40400 error mongod returns — so {$group: {_id: "$g", merged: {$mergeObjects: "$sub"}}} now behaves exactly like a real server.

The accumulator ships on both the Python server and the Rust server, pinned byte-for-byte by the aggregation parity harness.

Added

• aggregate.py / secantus-core (group.rs): $mergeObjects accumulator for $group and $setWindowFields — merge operand documents across the group (later keys win), skip null/missing, empty group → {}, non-document operand → Location40400.

$meta projection faithful error codes (both servers)

find()’s {field: {$meta: <arg>}} projection now returns the same errors real mongod does. A {$meta: "textScore"} projection without a $text predicate in the query fails with Location40218 (query requires text score metadata, but it is not available), and any unrecognized $meta argument fails with Location17308 (Unsupported argument to $meta: <arg>). Both errors are raised at parse time — before matching — so they fire even against an empty collection, matching mongod. Verified against real mongod 6.0.

For a recognized-but-unsupported $meta keyword (indexKey, recordId, sortKey, and the search/geo/vector variants) SecantusDB degrades gracefully: rather than emitting a wrong metadata value, it omits the projected field entirely, leaving the rest of the projection intact. Previously the Python server mis-handled the $meta value as a truthy inclusion flag and the Rust server errored generically on it.

Fixed

• projection.py / secantus-core / secantus-commands: {$meta: "textScore"} without a $text query raises Location40218, and an unknown $meta argument raises Location17308, on both servers with mongod’s exact codes and wording. A recognized-but-unsupported $meta arg is validated clean and the field is omitted from the result (partial — SecantusDB doesn’t compute the metadata).

$min / $max compare by BSON order — no more traceback leak (both servers)

The $min and $max update operators now compare the incoming value against the current field value by MongoDB’s BSON canonical-type order, instead of Python’s native < / >. This fixes two bugs found by a three-way update differential against real mongod 6.0:

• A leaked traceback. A cross-type compare — e.g. {$max: {a: "str"}} on a numeric a — raised a raw TypeError ('>' not supported between 'str' and 'int') that surfaced to the client. Now it orders like mongod: a string out-ranks a number, so $max sets "str"; $max of an ObjectId, a date, or a bool over a number likewise picks the higher-ranked value.

• Explicit null treated as “no current”. An explicit-null field is a real value (BSON rank 2, below numbers), not an absent field. {$min: {a: 9}} on {a: null} now keeps null (null < 9); a genuinely missing field is still set unconditionally.

Fixed

• update.py / secantus-core: $min/$max use ordering._bson_lt (Python) / order::cmp (Rust) with a missing-vs-present split. The Rust engine handles the sortable subset (null / number / string / objectId / date / doc / array) natively and defers a bool / Decimal128 / NaN / exotic operand to the Python oracle (whose _bson_lt covers the full order).

PG server: connection teardown releases the thread’s WiredTiger session

Every PG connection thread that wrote data leaked its cached WT session on disconnect (the Mongo server’s teardown has always released it; _handle_client’s never did). Dead threads’ positioned cursors kept cache pages pinned, and after a few hundred connections WiredTiger’s eviction livelocked — an application thread wedged in __wt_cache_eviction_worker while holding the storage lock, queueing every other connection forever. The full psycopg gauge’s single-daemon run hung at ~test 420 three times out of three; with the fix it completes in ~125s (faster than the ~550s baseline, since sessions no longer pile up). Verified by an 8-writer-connection leak probe (unfixed: 2 → 10 sessions; fixed: flat) pinned as a regression test. Also: a binary/garbage COPY payload now raises SQLSTATE 22021 (invalid byte sequence) instead of escaping as an internal error.

Fixed

• pgserver.py: _handle_client’s finally releases the thread’s WT session and cached cursors via Storage._reset_thread_session(), mirroring the Mongo server; _copy_in guards decode_text with a faithful 22021.

• psycopg gauge headline after the day’s slices (COPY transactionality, CREATE SCHEMA, server-side cursors, this fix), on the standard single-daemon protocol: 2554 passed / 61.9%, up from 2465 / 59.8% — report refreshed.

Positional $ projection (both servers)

find()’s positional projection operator now works on both servers: find({"items.k": "b"}, {"items.$": 1}) returns only the first array element that matched the query on that path — items: [{k: "b", …}] — instead of the whole array stripped to empty documents, which is what both servers previously produced. The matched element is resolved from the query’s clause on the array (a dotted items.sub field, a direct value/range on items, or an items: {$elemMatch: …}), so it works for arrays of documents and arrays of scalars alike. Found by a three-way projection differential against real mongod 6.0; all value cases match exactly.

Fixed

• projection.py / secantus-core: the positional $ projection resolves and returns the first query-matched array element. The find command threads the filter into the projection engine so the operator has the query context it needs. Validation is parse-time (matching mongod), so an invalid positional — more than one (Location31276), an exclusion form (Location31395), or an array field the query doesn’t reference (Location51246) — errors even when the query matches nothing. The Python server reproduces mongod’s exact Location codes; the Rust server surfaces a generic BadValue on these error paths (the documented cross-cutting error-code gap). ($meta projection remains deferred — tasks/backlog.md §7.5.)

The psycopg conformance gauge: SecantusDB’s SQL server gets its headline number

The SQL server now has what the Mongo server has had for a year: an external conformance gauge running a real driver’s own unmodified test suite. invoke validate-psycopg vendors psycopg 3.3.4 (pinned in lockstep with the dev-extra wheel), spawns a SecantusPGServer daemon on an ephemeral port, verifies it actually is SecantusDB (a stray real Postgres would inflate the numbers), runs the full sync half of psycopg’s suite over PSYCOPG_TEST_DSN, and renders docs/validation-report-psycopg.md with the per-file pass/fail/skip breakdown. It joins the weekly validate.yml matrix as the fourteenth gauge — and the first for the SQL side. The opening baseline over the full sync suite is 2415 passed of ~4100 run (58.6%); the six-file subset that drove this month’s conformance work stands at 91%.

Added

• psycopg_validation/ (runner, include list, report generator), invoke validate-psycopg, a psycopg lane in validate.yml, and the vendor/psycopg submodule @ 3.3.4. psycopg[binary] is now pinned exactly so the vendored suite and the installed wheel stay in lockstep.

Python 3.10 actually works — and CI actually tests it

The CI test matrix’s python-version never took effect: uv sync honours the repo’s .python-version pin (3.12), so every matrix cell — including the scheduled 3.10–3.13 sweep — was silently testing 3.12. With the interpreter genuinely pinned per cell (a job-level UV_PYTHON, which outranks the pin file for every uv invocation in the job), the first real 3.10 run surfaced three breakers that the gap had been hiding, all now fixed: the config loader’s module-level tomllib import (stdlib only from 3.11) crashed secantus.config / the secantusd-py CLI on 3.10; datetime.UTC (a 3.11+ alias) in fifteen test call sites; and datetime.fromisoformat on 3.10 rejecting Postgres’s short UTC offsets (+00 / +0000), which PG text rendering emits and timestamptz literals carry.

Fixed

• config.py: fall back to the API-identical tomli backport on Python 3.10 (tomli>=2.0; python_version < '3.11' added to the core dependencies).

• sql/datetimes.py: new parse_iso_datetime — fromisoformat fast path (a no-op passthrough on 3.11+) that widens a trailing short UTC offset to +HH:MM only on failure; wired into scalar._as_datetime, intervals, and both typemap.coerce timestamp branches.

• .github/workflows/test.yml: the three matrix jobs set a job-level UV_PYTHON: ${{ matrix.python-version }} so uv sync and every uv run agree on the matrix interpreter (a sync-only --python flag is not enough — a later bare uv run re-resolves against .python-version and recreates the venv without the dev extras).

• Tests: datetime.UTC → datetime.timezone.utc in test_indexes / test_expressions / test_crud.

Range operators are type-bracketed, matching mongod

MongoDB’s range operators ($gt / $gte / $lt / $lte) are type-bracketed: a scalar bound only ever matches values in the same BSON type bracket. SecantusDB now honours that on both the Python and the Rust server, closing two divergences that a three-way probe against real mongod surfaced.

A document-valued (or array-of-documents) field no longer errors on the Rust server when compared against a scalar bound — {a: {$gt: 2}} against a document-valued a, and {items: {$elemMatch: {$gt: n}}} over an array of sub-documents, now cleanly no-match (as they always did on the Python server and on mongod) instead of the Rust server returning a BadValue. And bool is its own bracket: a boolean-valued field no longer spuriously matches a numeric bound (Python’s bool is an int subclass, so True < 2 used to match on both engines), while bool-vs-bool comparisons (True > False) still work. Both the collection-scan and index-scan paths agree with mongod on every case.

Fixed

• Range operators ($gt/$gte/$lt/$lte) are now type-bracketed on both servers. A document/array operand against a scalar bound no-matches instead of erroring on the Rust server; a boolean field no longer matches a numeric bound (bool compares only with bool). Verified against real mongod 6.0 with a three-way probe (collection-scan and index-scan paths both).

Rust server compares array-vs-array range bounds lexicographically

A range query whose bound is an array — {a: {$gt: [1, 2]}} — now evaluates on the Rust server instead of erroring. The Rust matcher previously deferred any array operand to a Fallback, which the Rust server surfaced as a BadValue; it now compares the two arrays whole-array lexicographically, exactly as the Python server (via Python’s native list < list) and real mongod do.

The comparison recurses element-by-element: the first decisive element pair wins, equal leading elements continue to the next pair, and if one array is a prefix of the other the shorter one sorts first. A cross-type element pair (where Python’s < would raise TypeError) yields a clean no-match rather than an error, and an array field compared against a scalar bound still rides the multikey element path ({a: [1, 3]} matches {a: {$gt: 2}} because 3 > 2). Only the exotic BSON types (JS code / symbol / dbpointer / undefined) as a range operand still defer to the Python engine.

Verified against real mongod 6.0 and pinned to the Python oracle by new curated parity cases and Rust unit tests.

Fixed

• Rust server: $gt / $gte / $lt / $lte with an array bound (e.g. {a: {$gt: [1, 2]}}) now compares whole-array lexicographically instead of returning BadValue, matching the Python server and mongod. Array-vs-scalar bounds continue to match via the multikey element path; a cross-type element pair no-matches cleanly.

$log10 now evaluates natively on the Rust server

The $log10 aggregation-expression operator is now computed natively by the Rust engine, so the Rust server evaluates it instead of rejecting the pipeline with a BadValue. The rest of the transcendental family ($exp / $ln / $log) was already native; $log10 had simply been left out. Rust’s f64::log10 and CPython’s math.log10 share the platform libm, so the two servers agree bit-for-bit (pinned by the expression parity corpus). Found by a three-way differential sweep against real mongod 6.0.

Fixed

• $log10 is evaluated by the Rust secantus-core expression engine (was a Fallback → BadValue on the Rust server). Matches the Python server and mongod for positive inputs; a non-positive input yields null on both servers (see tasks/backlog.md §7 for the pre-existing log-domain divergence from mongod).

Tooling: the sqllogictest conformance gauge (invoke validate-slt)

The SQL server gets its correctness gauge (tasks/sql-gauges-plan.md G1): the SQLite-originated sqllogictest corpus — 622 files, millions of records — is vendored pristine at vendor/sqllogictest and executed by sqllogictest-rs over real pgwire, one fresh SecantusPGServer daemon per file. A preprocessing pass (never touching the vendored tree) bridges the three corpus/runner incompatibilities established empirically: trailing comments on skipif/onlyif lines, value-per-line expected blocks for nosort/rowsort multi-column records, and sqlite’s implicit hash-threshold 8 default. The curated 30-file include list currently passes 26/30 end-to-end; the 4 failures are declared EXPECTED_DIVERGENCES (SQLite read-only views, SQLite’s division-by-zero→NULL vs PG’s 22012, and the runner’s missing query I type coercion), so the gauge is green in its own terms and reports loudly if a divergence resolves.

Added

• vendor/sqllogictest (shallow submodule, dev-only, excluded from sdist/wheel), the slt_validation/ gauge package (preprocessor, per-file daemon runner with identity verification, report generator, include + expected-divergence lists), the invoke validate-slt task, and docs/validation-report-slt.md in the Sphinx toctree. Requires the sqllogictest binary (cargo install sqllogictest-bin).

• pyproject.toml: sdist excludes for vendor/sqllogictest / slt_validation — and the previously-missing vendor/psycopg / psycopg_validation entries.

SQL: COPY runs inside the open transaction block

The COPY sub-protocol handler never entered the session’s user transaction: COPY after a same-block CREATE TABLE failed with UndefinedTable (psycopg’s standard fixture shape, cascading through ~190 of its COPY-backed tests), COPY TO STDOUT couldn’t see rows inserted earlier in the block — and worst, COPY FROM STDIN rows were written outside the transaction, so they survived a ROLLBACK. Plan resolution, the copy-in insert, and the copy-out extract now all run under use_user_transaction when a block is open, and a failed COPY marks the block aborted like Postgres does.

Fixed

• pgserver.py: _handle_copy / _copy_in / _copy_out wrap their engine calls in the session’s open user transaction (no-op outside a block); a COPY error inside a block sets txn_failed. The three copy-heavy psycopg suites (test_copy / test_range / test_multirange) move 230 → 374 passing.

SQL server: bare COPY options and computed projections over row sources

Two more gauge-driven fixes. COPY … TO STDOUT (FORMAT csv) — the options spelling psycopg emits, without WITH — now parses (sqlglot only accepts the WITH (…) form, so parse() inserts it, anchored on the STDIN/STDOUT target and a known option keyword). And projections that compute over a set-returning or catalog row source — SELECT x * 2 FROM generate_series(1,3) AS t(x), SELECT 1 FROM pg_namespace — run through the per-row evaluated plan instead of failing with “expected a column”. The psycopg-gauge subset stands at 685 of 979 (70%), from 42% at the first run.

Fixed

• planner.py: COPY … TO STDOUT/FROM STDIN (options) normalizes to the WITH (options) spelling sqlglot parses; both the table and the query form take options.

• engine.py: SRF and virtual-catalog row sources route computed projections (arithmetic, literals, scalar functions) through the evaluated-select plan — execution and Describe agree on the shape.

SQL: CREATE SCHEMA and schema-qualified user types

CREATE SCHEMA [IF NOT EXISTS] / DROP SCHEMA [IF EXISTS] [CASCADE] land, with user-declared types (enum / domain / composite) creatable and droppable under a schema (CREATE TYPE testschema.testcomp AS (…)). Qualified names resolve everywhere psycopg’s type machinery needs them: to_regtype, the 'schema.name'::regtype literal cast (previously an internal error — the pushdown’s cast coercion knew regclass but not regtype), oid::regtype rendering, and TypeInfo/CompositeInfo fetches by dotted string or sql.Identifier spelling. pg_namespace carries user schemas with minted oids and pg_type reports the bare typname under the schema’s typnamespace. Dropping a non-empty schema without CASCADE is a 2BP01 dependency error, CASCADE drops the contained types, and DROP TYPE IF EXISTS tolerates a missing schema. This clears the psycopg gauge’s entire “CREATE SCHEMA is not supported” cluster and unblocks the schema-gated composite/range/typeinfo fixtures. (Schema-qualified tables remain 0A000 — tasks/backlog.md; user-defined CREATE TYPE … AS RANGE likewise.)

Added

• catalog.py: schema registry (create_schema / schema_exists / drop_schema / list_schemas); engine.py: CREATE/DROP SCHEMA routing, qualified-name extraction for CREATE/DROP TYPE; virtual.py: user-schema pg_namespace rows, dotted-name splitting in pg_type, quote-normalized qualified lookups; planner.py: the ::regtype literal cast resolves built-ins and user types (42704 on unknown, like PG).

SQL server: arbitrary WHERE expressions and three-valued logic

The sqllogictest random corpus writes SQL the way a fuzzer does — `WHERE

• col2 + col1 IS NOT NULL, WHERE 1 IN (2), SELECT + + 90 * a * - b— and the planner used to reject anything its Mongo-filter pushdown couldn't express. Untranslatable WHERE clauses now route to per-row evaluation automatically (a dry-run of the lowering decides), computed unary projections type correctly instead of crashing tag inference,ORDER BY resolves to the output expression on the evaluated path, and the scalar evaluator's NOT/AND/OR/BETWEEN implement SQL's three-valued logic (NOT NULLis NULL,NULL AND FALSE` is FALSE — visible under NOT).

Fixed

• planner.py: where_needs_per_row dry-runs the pushdown lowering and falls back to per-row evaluation when it raises; the DISTINCT plan path consults it too; _infer_scalar_tag types - col from its operand; ORDER BY 1 resolves the output ordinal (except SRF outputs, which sort post-expansion).

• scalar.py: three-valued NOT/AND/OR and a decomposed BETWEEN whose definitively-false arm dominates a NULL bound.

• A predicate the pushdown can’t lower no longer errors 0A000; cross-type comparisons under per-row evaluation match nothing instead of raising Postgres’ 42883 — a documented divergence (tasks/backlog.md).

SQL server: client_encoding, wire-protocol fixes, and binary-format hardening from the psycopg gauge

Running psycopg 3’s own unmodified test suite against the SQL server (tasks/sql-gauges-plan.md) surfaced a batch of wire-protocol and type-handling divergences beyond the type-OID work. The headline is client_encoding support: the server now honours the startup parameter and SET client_encoding (LATIN1/LATIN2/LATIN5/LATIN9, WIN1250-1252, SQL_ASCII pass-through), converting query text, text and binary parameters, text and binary results, arrays, COPY data, and error messages at the wire boundary while the engine stays UTF-8 throughout. Alongside it, a real protocol-ordering bug: Describe answered NoData for DML with RETURNING while Execute then emitted DataRows — a violation that crashed psycopg’s pipelined executemany. The measured effect on the fixed psycopg-gauge subset (six files, psycopg 3.3.4): 409 → 637 passed of 979 (42% → 65%) across this and the preceding type-OID release.

Added

• client_encoding (startup parameter and SET, with canonical ParameterStatus reporting and 22023 on unknown encodings); an untranslatable result character raises 22P05 like Postgres instead of degrading to ?, and a NUL byte in a text parameter is rejected with 22021.

• Quoted built-in type names in DDL (CREATE TABLE t (c "cidr"), the form psycopg’s fixtures emit via sql.Identifier) resolve as built-ins — including array spellings — instead of failing as undeclared enums.

Fixed

• pgextended.py: Describe on INSERT/UPDATE/DELETE/MERGE … RETURNING answers with the RETURNING columns’ RowDescription (was NoData followed by DataRows — a protocol violation).

• engine.py: Describe on a set-returning row source (FROM generate_series(…) / bare SELECT generate_series(…)) resolves the result shape instead of erroring — this is what failed every cursor.stream() (libpq single-row mode) call.

• Array round-trips, all six param/result format combinations: a binary array parameter’s Python list is rendered as a Postgres array literal (was the Python repr); the array-literal parser strips only Postgres’ whitespace set (\x1c–\x1f are str.isspace() to Python but data to Postgres); the renderer quotes every whitespace character; binary array elements coerce to native values (bytea hex, bool 't'/'f') before encoding. chr(1)–chr(255) plus € now round-trip byte-exact in text and bytea arrays.

• Binary numeric handles ±Infinity in both directions (signs 0xD000/0xF000; encoding previously crashed, decoding produced garbage).

SQL server: oid/regtype, declared-parameter typing, and the full binary codec surface

Three parallel work streams off the psycopg gauge, landing together. The oid type (26, arrays 1028) is now first-class — columns, casts, binary codecs, pg_type rows — and 21::regtype resolves an OID to its type name the way Postgres does. Parameter typing got the same discipline on every path: the OID a client declares in Parse now governs the value whether it arrives in text or binary format, '19.99'::numeric-style scalar casts convert instead of passing strings through, and Execute encodes DataRows with the same column OIDs Describe reported (the mismatch fed text bytes to clients parsing binary numerics). And the binary result/parameter codec surface now covers what psycopg’s full-type faker exercises: time, timetz, interval, uuid, inet, cidr, macaddr, json, and every range and multirange type — including new tstzrange/tstzmultirange registration, PG-exact multirange rendering, JSON integers beyond int64, and Decimal128-safe numeric handling at any width. psycopg’s test_leak (the full-type CRUD matrix) went from 72 failures to 72 passes; the six-file gauge subset stands at 887 of 979 (91%), from 42% at the first external run.

Added

• oid type end-to-end; N::regtype OID resolution (42704 on unknown OIDs); pg_typeof(x)::oid resolves to the type’s OID.

• Binary codecs (both directions) for time/timetz/interval/uuid/inet/cidr/ macaddr/json and all range/multirange types; tstzrange/tstzmultirange types; oid[]/json[]/multirange array OIDs.

Fixed

• Execute now applies the same declared-parameter OID overrides to its DataRow encoding that Describe applies to RowDescription — the divergence sent int4/text bytes in fields announced as int2/numeric.

• Text-format parameters with a declared scalar OID convert to the native type (declared type governs, matching the binary twin; garbage raises 22P02); scalar casts to int/float/numeric/bool convert with PG rounding semantics.

• Binary numeric survives values wider than Python’s default 28-digit context (wide-context decode, context-free negate/abs); >34-digit numerics round into Decimal128 range instead of erroring on INSERT.

• Numeric/bytes/±inf parameters keep their types through statement binding (typed cast nodes / hex literals instead of bare string literals).

• Multirange text rendering drops the “, “ separator Postgres doesn’t print; daterange bounds render date-only; bool coercion of ‘f’/’false’ strings; JSON top-level scalars render as JSON.

SQL: HAVING IS NULL forms, constant JOIN ON, duplicate join group keys

Round four of the sqllogictest corpus tail. HAVING <operand> IS [NOT] NULL now lowers for bare-column, aggregate, and computed-over-group-key operands (HAVING (- col2) IS NOT NULL) on both the single-table and join HAVING lowerers. A constant JOIN ON condition (LEFT JOIN tab0 ON 80 = 70) folds three-valued — TRUE joins every foreign row, FALSE/unknown joins none (INNER drops the row, LEFT null-pads). And two join GROUP BY wrong-answer bugs: the same bare column name grouped from two aliases (GROUP BY cor1.col1, cor0.col1) collapsed to a single group key, and SELECT DISTINCT over grouped join output never deduplicated.

Fixed

• planner.py: _having_to_match / _join_having_to_match lower IS [NOT] NULL over bare columns, aggregates, and computed group-key expressions (the last via _to_agg_expr over a group-key resolver, correct through any NOT nesting); [NOT] <expr> IN (<exprs over group keys>) lowers three-valued; always-unknown NULL-operand predicates (HAVING NOT NULL IN (- col1), NOT NULL NOT BETWEEN - col0 AND NULL) fold to match-nothing; _to_agg_expr learns unary minus over non-literals.

• planner.py: an always-unknown JOIN ON (ON NOT NULL < expr) folds like a constant-false ON instead of raising.

• planner.py: _lookup_stage folds a constant ON via _constant_predicate_filter instead of raising “ON must compare columns”.

• planner.py: duplicate bare column names in a join GROUP BY mint distinct grouped fields on both the join-group and join-group-window paths (qualified references rewrite/resolve onto the minted key); grouped SELECT DISTINCT over a join dedups with the same second $group the single-table planner uses.

SQL server: join-path aggregate expressions and WHERE residuals

The JOIN planners catch up with the single-table paths from the last two rounds: aggregate arguments over joins can be expressions (MAX(cor0.col0 + 1), SUM(- 83) over a CROSS JOIN), lowered through the join resolver with identity decorations stripped, and a join WHERE the $match lowering can’t express routes to the per-row residual the join pipelines already carry (a dry-run probe, the join twin of the single-table one) instead of erroring.

Fixed

• planner.py: computed-over-aggregate outputs over a join (COUNT(*) * 32 FROM a CROSS JOIN b) route to the group-then-evaluate builder instead of failing per-row; _join_accumulator lowers expression arguments for sum/avg/min/max; _agg_key identifies expression aggregates by SQL text instead of crashing the resolver; _join_where_lowerable dry-runs _expr_to_filter and the inner/outer join builders plus both join residual sites consult it.

SQL server: RowDescription reports real Postgres type OIDs for computed columns

A libpq client keys its result decoding off the type OID in each RowDescription column, and SecantusDB’s SQL server used to fall back to text (25) for most computed results — CASE expressions, array[...] constructors, array casts, integer arithmetic, bound parameters — and widened smallint/real to integer/double precision everywhere. The first external-gauge run (psycopg 3’s own test suite plus the sqllogictest corpus, see tasks/sql-gauges-plan.md §6) flagged this as the single highest-leverage divergence. Computed and derived columns now describe with the OID real Postgres would use, so typed loaders in psycopg / pg8000 / SQLAlchemy decode results without special-casing.

Added

• pg_typeof() and 'name'::regtype: the type-introspection pair psycopg’s type suite leans on (select pg_typeof(%s::int2) = 'smallint'::regtype). pg_typeof resolves at plan time from the same static inference that types RowDescription; ::regtype normalizes any accepted spelling (int4, varchar, float4) to the canonical pretty form pg_typeof prints.

• typemap.py: first-class int2 (21) and float4 (700) type tags — smallint / real columns, casts, arrays (1005 / 1021), catalog pg_type rows, and information_schema spellings; SMALLSERIAL columns now describe as int2 instead of text.

Fixed

• planner.py: type inference for computed SELECT columns — CASE types from its result branches; array[...] and array casts report the array OID; integer arithmetic stays integer (int + int → int4, matching _pg_div’s truncating division) instead of numeric; an unadorned decimal constant (SELECT 1.5) is numeric, matching Postgres; sum(int2/int4) → int8, sum(int8) → numeric, avg(integer) → numeric per Postgres’ aggregate result types; CAST($1 AS SMALLINT) coerces its text-bound value numerically.

• pgextended.py: SELECT $1 describes with the parameter OID the client declared in Parse (psycopg binds a small Python int as int2), instead of re-inferring from the substituted Python value.

• pgextended.py: binary result format and binary parameters now cover arrays (the real ndim/hasnull/elemoid wire layout, both directions). The correct array OIDs engage a libpq client’s binary array parser, which the text-bytes fallback would have fed garbage.

SQL: server-side cursors over the wire, pg_cursors, pg_prepared_statements

psycopg’s ServerCursor works end-to-end. A DECLAREd cursor is a portal in the v3 protocol, and psycopg’s first move after the DECLARE is a wire Describe('P', name) — which our extended-protocol session answered with 34000 portal does not exist. The portal Describe (and Close) now fall back to the session’s DECLAREd cursors, parameterized declarations substitute their $N placeholders inside the raw DECLARE … FOR SELECT $1 command text, and the session’s cursors and prepared statements surface in new pg_cursors / pg_prepared_statements catalog tables. psycopg’s test_cursor_server + test_prepared move 26 → 102 passing.

Added

• pg_catalog.pg_cursors (name / statement / is_holdable / is_binary / is_scrollable / creation_time, from the session’s open cursors) and pg_catalog.pg_prepared_statements (SQL-level PREPAREd plus the connection’s wire-Parse statements, exposed via Session.wire_prepared).

Fixed

• pgextended.py: Describe('P', name) on a DECLAREd cursor returns its RowDescription; Close('P', name) destroys the cursor.

• planner.py: substitute_parameters also substitutes $N textually inside a raw exp.Command tail (DECLARE bodies aren’t parsed trees).

SQL: three-valued NULL semantics on the pushdown, and the aggregate long tail

The SQL server’s Mongo-filter pushdown now honours SQL’s three-valued logic: <>, NOT (...), NOT BETWEEN, and NOT IN no longer match rows whose operand column is NULL (Mongo’s $ne/$nor/$nin are two-valued and matched them), a NULL candidate in an IN list can no longer match a NULL row, and x NOT IN (…, NULL) correctly matches nothing. SUM over zero non-null inputs returns NULL instead of Mongo’s 0, on every plan path. Alongside, a round of sqllogictest-corpus aggregate and planner shapes: FROM-less aggregates (SELECT COUNT(*) is 1), COUNT(<expr>) counting non-null evaluations, expression DISTINCT aggregate arguments (SUM(DISTINCT 77)), computed and constant projections under GROUP BY, SELECT * grouped by every column, SELECT DISTINCT over grouped output, parenthesized join sources (FROM (a CROSS JOIN b)), constant-LHS IN (list and subquery forms), division by zero raising SQLSTATE 22012, and Postgres-exact float8 wire text (12, not 12.0; NaN/Infinity spellings). Three files of the corpus’s random/ suites now pass end-to-end that previously failed on their first record.

Fixed

• planner.py: _negated_filter lowers NOT by pushing the negation into the tree (De Morgan, comparison-operator flips, null-guarded single-field fallback) instead of Mongo’s two-valued $nor; <> is null-guarded; $in lists drop NULL candidates; constant-LHS IN/NOT IN fold three-valued (list + subquery); a NULL comparison operand folds to match-nothing even when wrapped (51 <> (NULL), - CAST(NULL AS INT) <> x); computed comparisons lowered to $expr guard both sides non-null (BSON total order is two-valued — NULL <> 19 matched every row).

• planner.py: a join WHERE the $match lowering can’t express routes to the per-row evaluated join / the pre-group residual instead of being silently dropped, on both the plain-join and the join-group-window paths (WHERE (NULL) BETWEEN NULL AND NULL returned every row).

• planner.py: two different expression aggregates of the same function (MAX(3) and MAX(-94 - -16)) no longer collide on the (func, None) accumulator-dedup key and share one value; integer / inside aggregate arguments and $expr lowers with PG’s truncate-toward-zero semantics (MIN(col1 / -99) was computed with real division).

• planner.py / executor.py: SUM over only-NULL inputs is NULL on the plain group, group-window, join, join-window, and DISTINCT paths; the evaluated group path synthesizes the one implicit-aggregate row over empty input like the pipeline path already did.

• planner.py: FROM-less SELECTs fold aggregates over their one implicit row; COUNT(<literal>) no longer misroutes to the lone-COUNT(*) fast path; COUNT(<expr>) counts non-null evaluations (COUNT(NULL) is 0).

• planner.py: expression DISTINCT aggregate arguments push the lowered expression into the distinct set (single-table, group-window, join, join-window registrars); computed-over-aggregate outputs over a JOIN route to the group-then-evaluate builder (COUNT(*) * COUNT(*)).

• planner.py: grouped SELECTs with computed/constant projections route to the evaluated group path; SELECT * under GROUP BY expands when every column is a group key; SELECT DISTINCT over grouped output dedups; ORDER BY <ordinal> resolves on the group-then-evaluate path.

• planner.py: FROM (a CROSS JOIN b) unwraps grouping parens instead of erroring “a derived table requires an alias”.

• scalar.py: division / modulo by zero raise SQLSTATE 22012 instead of leaking an internal error; COALESCE evaluates lazily like Postgres, so a division-by-zero in a never-reached argument no longer raises; operand-form CASE x WHEN v uses SQL equality (a NULL operand or WHEN value never matches, where Python == matched NULL to NULL).

• planner.py / executor.py: a constant HAVING (HAVING NOT NULL IS NULL) folds three-valued to match-all / match-nothing; DISTINCT aggregates over zero input rows synthesize their NULL row instead of crashing on the $addToSet reduction (“$size requires an array”).

• typemap.py: float8 text output uses Postgres’ shortest form (12, -0, 1e+20, NaN, Infinity).

• planner.py: _infer_scalar_tag is memoized per statement — deep arithmetic chains were exponential (a 20-term sqllogictest expression took ~0.5s; whole corpus files timed out).

SQL: psycopg TypeInfo catalog fidelity (typarray, pg_range, to_regtype)

psycopg’s type-registration machinery works end-to-end: TypeInfo.fetch, RangeInfo.fetch, MultirangeInfo.fetch, EnumInfo.fetch (with labels), and CompositeInfo.fetch (with field names) all resolve against the virtual catalog. pg_type gains typarray / typdelim, a pg_range table maps range oids to their declared subtype and multirange oids, to_regtype() is implemented (built-ins and user-declared enum/domain/composite types, returning NULL for unknown names), and oid::regtype::text renders user-declared type names. Catalog-table WHEREs that can’t lower now evaluate per-row with the real catalog in scope, and a context-dependent function call (to_regtype('mood')) is no longer folded as if it were a NULL literal.

Added

• pg_type.typarray / typdelim columns; the pg_catalog.pg_range virtual table (rngtypid / rngsubtype / rngmultitypid, declared subtypes — tsrange advertises timestamp, daterange advertises date); to_regtype(name) (scalar + FROM-less + pushdown-constant paths).

Fixed

• oid::regtype on a user-declared type’s oid resolves its name through the catalog instead of raising 42704.

• The catalog-table fast path publishes the planning subquery context and routes non-lowerable WHEREs through per-row evaluation with the real catalog (a synthetic catalog over the row backend knew no user types).

• The NULL-operand comparison folds no longer treat an Anonymous function call as a NULL literal (WHERE t.oid = to_regtype('mood') matched nothing).

Docs: compatibility / authentication / index pages caught up with shipped features

Three docs pages still described the server as it was several releases ago. compatibility.md’s stub table claimed getLog returns an empty array, hostInfo / whatsmyuri / buildInfo are hardcoded, sessions are untracked, and serverStatus is all zeros — all of those return real data now, so the table shrinks to the honest remainder (top’s zero counters, buildInfo’s deliberate 7.0.0 compatibility identity, connectionStatus’s empty privileges expansion, serverStatus’s zeroed fallback for bare CommandContext embedders). The $lookup stopgap section described the pre-index-join hash-only implementation; the date-format section listed ISO-week tokens as missing; the TTL-index row said there was no background sweeper. All rewritten to match the code.

authentication.md and index.md both still said authorization (RBAC) is not implemented and that an authenticated principal is fully privileged — RBAC has been enforced for a while (built-in and custom roles, checked on every command when --auth is on). authentication.md gains an Authorization section documenting the enforcement model, the built-in role list, and the custom-role / grant-revoke command set; both scope lists now credit SCRAM-SHA-1 and MONGODB-X509 correctly.

Changed

• docs/compatibility.md: stub table rewritten to current behaviour; $lookup section describes the index-driven join (IXSCAN on a matching foreign-field index, hash-join fallback); date-format token list updated (%G %V %j %U %u %w all supported); TTL row documents the 60-second background sweeper; out-of-scope auth bullet updated (SCRAM-SHA-1 implemented, RBAC enforced); Rust-server note updated to conformance parity with a pointer to the feature comparison.

• docs/authentication.md: RBAC documented as enforced (new Authorization section: built-in roles, custom roles, grant/revoke quartet, code-13 behaviour); createUser example uses a real role binding.

• docs/index.md: in-scope and out-of-scope auth bullets updated to SCRAM (SHA-1/SHA-256) + MONGODB-X509 + enforced RBAC.

$toDate conversion expression on both servers

The $toDate aggregation expression now works on the pure-Python and Rust servers. $toDate: <expr> is the shorthand for $convert: {input: <expr>, to: "date"}, and SecantusDB implements it as exactly that — a date is returned unchanged, an int/long/double is read as milliseconds since the Unix epoch, and an ISO-8601 string is parsed, while null or a missing field yields null.

Because $toDate delegates straight to the existing $convert-to-date path, it inherits precisely the same supported inputs and errors: whatever $convert can turn into a date, so can $toDate, with no separate conversion code to drift. The Rust engine’s $convert-to-date was also widened to convert an int / long / double (epoch milliseconds) to a date natively, so both $convert and $toDate now compute the numeric case on the Rust server rather than deferring; ISO-string and ObjectId inputs still defer to the Python oracle (matching $dateFromString’s partial Rust support). The two engines stay byte-for-byte in step (pinned by the expression parity harness).

Added

• expressions.py / secantus-core: $toDate aggregation expression operator, delegating to the existing $convert-to-date conversion; the Rust $convert-to-date path gains native int/long/double → epoch-millis conversion.

Unrecognized aggregation-expression operators report mongod’s error codes

When a query or pipeline references an aggregation-expression operator that doesn’t exist (e.g. a typo like $notreal, or an operator MongoDB itself hasn’t shipped), SecantusDB now rejects it with the same context-specific error code and message that real mongod returns, instead of a generic one.

An unknown operator inside a query $expr — find({$expr: {$notreal: [...]}}) — now surfaces 168 InvalidPipelineOperator with the message Unrecognized expression '$notreal' on both the Python and the Rust server (previously the Python server returned 14 TypeMismatch and the Rust server a generic 2 BadValue). An unknown operator inside an aggregation $project — aggregate([{$project: {y: {$notreal: [...]}}}]) — returns Location31325 Invalid $project :: caused by :: Unknown expression $notreal on the Python server. mongod emits these same “unknown expression” errors even for operators it recognises by name but hasn’t implemented, so SecantusDB simply matches that behaviour for any operator it doesn’t recognise.

Fixed

• Query $expr with an unrecognized expression operator returns 168 InvalidPipelineOperator "Unrecognized expression '$op'" on both servers (was 14 TypeMismatch on Python, 2 BadValue on Rust).

• Aggregation $project with an unrecognized expression operator returns Location31325 "Invalid $project :: caused by :: Unknown expression $op" on the Python server (was 14 TypeMismatch). The Rust server still returns a generic BadValue here — faithful $project detection needs to distinguish the projection-only operators ($slice / $elemMatch / $meta) from expressions, tracked in tasks/backlog.md §7.

CI: the Java-vs-Rust-server gauge joins the weekly validate run

docs/validation-report-java-rust-server.md was only refreshable by hand — the weekly validate.yml run regenerated every other committed report but not this one, so it would have gone stale. A java-rust-server matrix entry now runs invoke validate-java --server rust weekly alongside the other gauges: it reuses the java gauge’s JVM/Gradle toolchain plus the storage-engine sync, and points gauge_common.rust_binary at the venv-staged secantusd-rs via SECANTUSDB_BIN (the default search only covers the cargo target dir).

CI: the cross-driver summary regenerates with the weekly validate run

docs/validation-summary.md had been frozen since 2026-06-20 (“the 11 gauges”) while the per-driver reports refreshed weekly. Each gauge job now uploads its raw output (.validation/) as an artifact alongside its report, and the aggregate job reassembles them and regenerates the summary in the same refresh PR — no WiredTiger build needed there, because the generator now reads the package version straight from src/ and resolves vendored-driver SHAs from the superproject’s gitlinks (git ls-tree) instead of requiring checked-out submodules.

Added

• validation_summary.generate: collectors for the mongo-kotlin-driver gauge (JUnit XML from :driver-kotlin-sync:integrationTest) and the pymongo (async) gauge (AsyncMongoClient suite), bringing the summary to 13 gauges; the gauge count in the prose is computed, not hand-written.

$push / $addToSet skip missing field values (both servers)

A three-way aggregate differential against real mongod 6.0 found that the $push and $addToSet group accumulators were adding null for a document whose accumulated field is absent, where mongod skips it entirely. They now match mongod: a missing field is not accumulated, while an explicit null still is — so {$push: "$s"} over ["x", <missing>, null, "x"] yields ["x", null, "x"], and an all-missing field still produces [] (not a list of nulls). The distinction is drawn by a new missing-aware evaluate helper (evaluate_or_missing / eval_or_missing) that surfaces an absent field path as a distinct sentinel rather than null.

Fixed

• aggregate.py / secantus-core: $push / $addToSet (in both $group and $setWindowFields) skip a missing accumulator value. (The differential also surfaced three items deferred to tasks/backlog.md §7.5: $mergeObjects as a $group accumulator, $getField returning null instead of missing for an absent field, and a last-ULP $stdDevPop difference vs mongod.)

PG server shutdown drains client handlers before storage close

SecantusPGServer.stop() now tracks its per-connection handler threads and joins them (bounded, 5 s) after closing their sockets. Previously stop() returned while a handler could still be mid-request on its per-thread WiredTiger session, so an embedder’s natural stop() → storage.close() sequence raced the handler and corrupted the WT session handle (a logged WT session close failed during close / Session__freecb TypeError during teardown — visible whenever a client was abandoned mid-transaction, e.g. a failing test). A handler that outlives the drain window is logged by name.

Fixed

• sql/pgserver.py: stop() joins live handler threads before returning, so storage.close() immediately after stop() can no longer close a WT session a handler thread is still using. Regression test: abandon a client mid-transaction, stop(), assert every handler exited and close() logs no session-close error.

• Test docstrings in test_pgserver.py / test_pgserver_copy.py / test_sql_aggregate.py still described the deleted FakeStorage in-memory mock; they now state the real WT-backed Storage these suites run on.

$in/$nin regex candidates and $all + $elemMatch (both servers)

A three-way query differential against real mongod 6.0 turned up two match-operator gaps present on both servers, now fixed and three-way verified.

A regex inside $in (or $nin) now matches string values by pattern, as mongod does — {s: {$in: [/^h/i]}} matches "hello" and "HELLO". Previously both servers treated the regex as a literal value to compare by equality, so it silently matched nothing (and on the Rust server it errored). And $all now accepts $elemMatch clauses ({a: {$all: [{$elemMatch: {$gt: 1, $lt: 3}}]}}) — each clause requires some array element to satisfy its sub-query, so an array is matched against several independent element predicates at once.

Fixed

• query.py / secantus-core: $in / $nin route a regex candidate through the regex matcher (_in_candidate_matches / in_candidate_matches); $all handles a {$elemMatch: …} entry by delegating to the $elemMatch matcher over the whole array. (A related Rust-only gap — $gt/$lt against a cross-type operand such as a document-valued array element deferring instead of comparing by BSON type order — is tracked in tasks/backlog.md §7.5.)

$pull query semantics, $pullAll, and $push $sort on the Rust server

A three-way differential against real mongod 6.0 turned up two array-update correctness bugs present on both servers, plus a Rust-server feature gap — all now fixed and three-way verified.

$pull previously removed only elements literally equal to the criterion, so a predicate like {$pull: {a: {$gte: 10}}} silently removed nothing and a sub-document criterion like {$pull: {a: {x: 5}}} never matched. $pull now applies the criterion under full query semantics — an operator-only criterion ({$gte: 10}, {$in: […]}) is an element-value predicate; any other document criterion is a sub-document match against each element; a scalar is BSON-aware equality (so 1 matches 1.0 but not true, exactly as mongod does — the old literal-== path wrongly conflated 1 and true). $pullAll, which removes every element equal to any value in a list, was entirely unimplemented (both servers rejected it as an unknown modifier) and now works. On the Rust server the $push $sort modifier (1 / -1 whole-element or {field: dir}, in BSON order) now computes natively instead of deferring.

Fixed

• update.py / secantus-core: $pull now matches via the query engine (query.matches / query::matches) instead of literal equality; $pullAll added to both engines and to the Rust KNOWN_UPDATE_OPS validator.

Added

• secantus-core: $push $sort (whole-element and {field: dir} forms) on the Rust server, via the shared order::cmp / is_sortable contract; an element outside the sortable subset still defers. $inc / $mul with a Decimal128 operand remains a Rust-side defer (decimal arithmetic parity is out of scope — tasks/backlog.md §7.5).

Trigonometric expression operators (both servers)

The full trigonometric family lands on both servers — circular, inverse, and hyperbolic — matched to real mongod 6.0 via a three-way probe (mongod vs Rust vs Python server) with zero value divergences on the numeric path:

$sin · $cos · $tan · $asin · $acos · $atan · $atan2 · $sinh · $cosh · $tanh · $asinh · $acosh · $atanh

Inputs are int / long / double (result: double); null / missing propagate to null. Domain violations raise exactly as mongod does (Location50989): $asin / $acos / $atanh need [-1, 1], $acosh needs [1, ∞), and $sin / $cos / $tan reject ±Infinity / NaN. $atanh(±1) returns ±Infinity (not a domain error). A non-numeric argument raises Location28765 (Location51044 for $atan2).

Added

• expressions.py / secantus-core: the operators above. Both servers compute through the platform libm — Rust f64::sin and CPython math.sin share it, so they agree bit-for-bit (the same basis as the already-shipped $exp / $ln). Decimal128 inputs are float-cast on the Python server (SecantusDB does not reproduce mongod’s decimal-precise transcendental result) and defer to the Python oracle on the Rust side — the documented generic-code gap (tasks/backlog.md §7.5).

Set-expression and utility operators (both servers)

The aggregation set-expression family lands on both servers, plus a handful of comparison / size / angle utilities — all matched to real mongod 6.0 via a three-way probe (mongod vs Rust vs Python server) with zero value divergences:

• $setUnion / $setIntersection / $setDifference — set algebra over arrays. Union and intersection return their result in BSON sort order (matching mongod); difference preserves first-array order. All three dedup by BSON-order equality, so 1 and 1.0 collapse but 1 and true do not.

• $setEquals / $setIsSubset — set membership predicates over two-or-more / exactly-two arrays.

• $allElementsTrue / $anyElementTrue — truthiness reductions over an array.

• $cmp — three-way comparison (-1/0/1) using the full BSON cross-type order.

• $binarySize (UTF-8 byte length of a string / length of Binary; null → null) and $bsonSize (encoded BSON byte size of a document; null → null).

• $degreesToRadians / $radiansToDegrees — angle conversions.

Added

• expressions.py / secantus-core: the operators above. Set ops share a _set_dedup_sorted / set_dedup_sorted helper that sorts by BSON order and dedups adjacent equal values; a non-array argument (or an element the Rust core can’t cross-type-order) raises on the Python server with mongod’s code and defers to the Python oracle on the Rust side (documented generic-code gap, tasks/backlog.md §7.5).

Batch of aggregation expression operators (both servers)

Nine more aggregation expression operators land on both servers, all matched to real mongod 6.0 via a three-way probe (mongod vs Rust vs Python server):

• $tsSecond / $tsIncrement — the seconds / increment fields of a BSON Timestamp (as longs); null/missing → null, non-timestamp → error.

• $dateFromParts ISO-week form — {isoWeekYear, isoWeek, isoDayOfWeek} (the calendar form shipped earlier); starts at the Monday of ISO week 1 and rolls over (isoWeek 53 → next ISO year).

• $type — the BSON type string of a value, with "missing" for an absent field (distinct from "null").

• $isNumber (int/long/double/decimal, not bool) and $isArray.

• $strcasecmp — case-insensitive string comparison (-1/0/1; null → empty string).

• $replaceOne / $replaceAll — substring replacement; any null input/find/replacement → null, a non-string one → error.

Added

• expressions.py / secantus-core: the operators above. $strcasecmp and $replaceOne/$replaceAll follow the existing string-op contract (Rust computes ASCII and defers non-ASCII case mapping to the Python oracle); the ISO-week form uses chrono’s ISO calendar. The Python server reproduces mongod’s error codes exactly (5687301/5687302, 40515/40516/40523, 51745); the Rust server errors on the same inputs but with a generic code (its core defers error-raising — documented gap, tasks/backlog.md §7.5).

$dateFromParts expression (both servers)

Both servers now build dates from calendar components with $dateFromParts: {$dateFromParts: {year, month, day, hour, minute, second, millisecond, timezone}}. Components default to month/day = 1 and time = 0, and out-of-range values roll over exactly as mongod does — month 13 → next January, month 0 → previous December, day 0 → last day of the previous month, hour 25 → next day, millisecond 1500 → +1.5 s. year is required and must be 1–9999. A timezone interprets the components as local time in that zone (local→instant). Any null component yields null. Matched to real mongod 6.0 via a three-way probe (mongod vs Rust vs Python server): all values, rollovers, and the validation error codes (40515 non-integral component, 40516 missing year, 40523 year out of range) confirmed.

Added

• expressions.py / secantus-core: $dateFromParts — month-carry + day/time timedelta arithmetic for the rollover. The Python server also resolves named IANA timezones (via zoneinfo); the Rust server computes fixed-offset zones natively and defers named zones to the Python oracle (the local→instant direction is DST-ambiguous, as with $dateFromString). The ISO-week form (isoWeekYear / isoWeek / isoDayOfWeek) is not yet supported.

$top / $bottom / $topN / $bottomN accumulators (both servers)

Both servers now support MongoDB 5.2’s sort-key $group (and $setWindowFields) accumulators: {$topN: {n, sortBy, output}} sorts the group’s documents by sortBy and returns the top n documents’ output, $bottomN the bottom n; $top / $bottom are the single-value forms and take no n. The sortBy is a multi-key spec with per-field directions, matching $sort’s cross-type BSON order. Matched to real mongod 6.0 via a three-way probe (mongod vs Rust vs Python server) — values, multi-key sort, array output, integral-double n, and the validation error codes (5788002-5788005, 5787908, 10065) all confirmed.

Added

• aggregate.py / secantus-core (group.rs): $top / $bottom / $topN / $bottomN accumulators — collect (sortBy-values, output) per doc, stable-sort by the sortBy directions at finalize (via the same _SortKey / order::cmp contract as $sort, deferring an unsortable sort key to the Python oracle), and take the top/bottom output(s). Usable in $group and $setWindowFields.

Server stop names the connection thread when a shutdown drain wedges

SecantusDBServer.stop() already drains its per-connection handler threads to zero before closing WiredTiger (polling the active-connection count, re-closing sockets each poll), and storage’s per-op _closed fences make that teardown safe even if the drain times out. What it lacked was observability: on a drain timeout it logged only a count (“N connection thread(s) still active”), which is exactly what made the intermittent xdist-worker-death race in this area hard to pin down. Connection threads are now named secantus-conn-<host>:<port>, and the timeout warning dumps the live stack of each still-active one — so a genuine shutdown wedge names its own culprit instead of surfacing as an opaque number.

Changed

• server.py: per-connection handler threads are named secantus-conn-<addr>; the stop-drain timeout warning now includes each stuck thread’s stack (_format_stuck_conn_stacks).

$top / $bottom / $topN / $bottomN accumulators (both servers)

Both servers now support MongoDB 5.2’s sort-key $group (and $setWindowFields) accumulators: {$topN: {n, sortBy, output}} sorts the group’s documents by sortBy and returns the top n documents’ output, $bottomN the bottom n; $top / $bottom are the single-value forms and take no n. The sortBy is a multi-key spec with per-field directions, matching $sort’s cross-type BSON order. Matched to real mongod 6.0 via a three-way probe (mongod vs Rust vs Python server) — values, multi-key sort, array output, integral-double n, and the validation error codes (5788002-5788005, 5787908, 10065) all confirmed.

Added

• aggregate.py / secantus-core (group.rs): $top / $bottom / $topN / $bottomN accumulators — collect (sortBy-values, output) per doc, stable-sort by the sortBy directions at finalize (via the same _SortKey / order::cmp contract as $sort, deferring an unsortable sort key to the Python oracle), and take the top/bottom output(s). Usable in $group and $setWindowFields.

$firstN / $lastN / $maxN / $minN as $group accumulators (both servers)

Both servers now support the N-element operators as $group (and $setWindowFields) accumulators, completing the family whose expression forms shipped earlier: {$firstN: {n, input}} collects the first n per-doc input values across the group, $lastN the last n, and $maxN / $minN the n largest / smallest by BSON order. Matched to real mongod 6.0 via a three-way probe (mongod vs Rust server vs Python server): $firstN / $lastN keep null values (they’re the first/last values seen), while $maxN / $minN drop them; {n, input} validation (integral-double n accepted, mongod error codes) is shared with the expression forms.

Added

• aggregate.py / secantus-core (group.rs): $firstN / $lastN / $maxN / $minN accumulators (shared nelem_parse_n validator; $maxN/$minN sort via the order::cmp/is_sortable contract, deferring bool/Decimal128 elements to Python’s _SortKey). Usable in $group and $setWindowFields.

Storage close-path race fixed; opt-in fast test storage

The WiredTiger-backed storage close path carried a latent use-after-free race. A connection thread opening or resetting its per-thread session (_session / _reset_thread_session) and the cross-thread oplog readers touched the WiredTiger connection outside the storage lock, so under rapid concurrent teardown they could race close()’s own session/connection teardown — a double-close or an open-on-a-closed-connection that segfaults. Every one of those paths is now fenced against the closed flag under the storage lock. The shipped server always checkpoints on close, whose timing masked the race in practice, and the pure-Rust server already drains its connection threads to zero before closing the connection — so neither production server was exposed — but the race was real and is now closed.

Separately, Storage and SecantusDBServer gained a durable parameter so the test suite can run against a faster non-durable storage mode (journal on, close-checkpoint skipped — every table is still created on disk, so schema, persistence and within-session behaviour stay real). It is opt-in and defaults to durable, so the shipped server is unchanged. SECANTUS_FORCE_DURABLE=1 forces full journal + checkpoint durability everywhere and overrides the fast default; a CI lane runs the whole suite that way on every push, so the checkpoint-durability path (schema, close-and-reopen, PITR / backup) stays continuously exercised even though the default local suite now runs fast.

Added

• Storage / SecantusDBServer: a durable parameter (defaults to durable), with SECANTUS_FORCE_DURABLE (force durable, wins over everything) and the conftest-set SECANTUS_TEST_FAST_STORAGE (fast default for the test suite) environment overrides.

• CI: a SECANTUS_FORCE_DURABLE=1 full-suite lane so durability paths run every push.

Fixed

• Storage close-path use-after-free / double-close race: _session, _reset_thread_session, and the oplog readers (read_oplog, read_preimage, oplog_floor_seq, find_seq_for_ts, scan helpers) now open/close WiredTiger sessions only under the storage lock and only while the store is open.

$firstN / $lastN / $maxN / $minN as $group accumulators (both servers)

Both servers now support the N-element operators as $group (and $setWindowFields) accumulators, completing the family whose expression forms shipped earlier: {$firstN: {n, input}} collects the first n per-doc input values across the group, $lastN the last n, and $maxN / $minN the n largest / smallest by BSON order. Matched to real mongod 6.0 via a three-way probe (mongod vs Rust server vs Python server): $firstN / $lastN keep null values (they’re the first/last values seen), while $maxN / $minN drop them; {n, input} validation (integral-double n accepted, mongod error codes) is shared with the expression forms.

Added

• aggregate.py / secantus-core (group.rs): $firstN / $lastN / $maxN / $minN accumulators (shared nelem_parse_n validator; $maxN/$minN sort via the order::cmp/is_sortable contract, deferring bool/Decimal128 elements to Python’s _SortKey). Usable in $group and $setWindowFields.

Date-operator timezone errors now report mongod’s exact codes (Python server)

A three-way conformance probe (real mongod 6.0 vs the Rust server vs the Python server) found that the date operators’ timezone errors used a generic code. The Python server now reports mongod’s exact codes: an unrecognized time zone ({$dateToString: {…, timezone: "Not/AZone"}}, and likewise for the $hour/… extractors and $dateToParts) is Location40485 “unrecognized time zone identifier: “…””, and a non-string timezone is Location40517 “timezone must evaluate to a string, found …”. (The Rust server raises on the same inputs but with a generic code — its core defers error-raising to Python.)

Fixed

• expressions.py (_resolve_timezone): pin the unknown-zone / non-string-timezone errors to mongod’s 40485 / 40517, shared by every timezone-aware date operator.

N-element array expressions $firstN / $lastN / $maxN / $minN (both servers)

Both servers now support MongoDB 5.2’s N-element array aggregation expressions: $firstN / $lastN return the first / last n elements of an array, and $maxN / $minN the n largest / smallest by MongoDB’s cross-type BSON sort order (descending for $maxN, ascending for $minN, with null elements ignored). When the array has fewer than n elements all are returned. Neither server recognised these before.

The {n, input} validation is matched to real mongod 6.0 (via a three-way probe against mongod): n may be any positive integral number — an integral double like 2.0 is accepted — and a missing n / input, a non-integral or non-positive n, or a null / missing / non-array input each raises mongod’s exact error code (Location5787902-5787908 / Location5788200). In particular a null or missing input is an error, not null — an earlier draft (and the $firstN/$lastN that first shipped under this Unreleased section) returned null there, which diverged from mongod; this is now corrected.

Added

• expressions.py / secantus-core: $firstN / $lastN / $maxN / $minN expression operators over a shared, mongod-faithful {n, input} validator (_nelem_n_and_input / nelem_n_and_input). $maxN / $minN sort via the same order::cmp / is_sortable contract $sortArray uses (an element outside the sortable subset — bool, Decimal128, … — defers to the Python _SortKey oracle), so the two engines agree on cross-type order. The Python server reproduces mongod’s error codes exactly; the Rust server raises on the same inputs but with a generic code (its core defers error-raising to Python). The $group accumulator forms remain a follow-on (tasks/backlog.md §7.5).

Distinguishable daemon names: secantusd-py, secantusd-rs, secantusd-py-pg

The two servers used to collide on the command name secantusdb, and the Rust binary went by the confusable secantusdb-rs. Each daemon now has a clear name under a shared secantusd-<engine>[-<protocol>] scheme: the Python MongoDB server is secantusd-py, the Rust MongoDB server is secantusd-rs, and the PostgreSQL-wire server gets its first console script, secantusd-py-pg. This is a clean break with no backwards-compatibility shim: the old secantusdb and secantus console scripts are gone, and the two utility commands are renamed to the bare-secantus import-name prefix (they aren’t daemons, so the secantusd- prefix doesn’t fit them) — secantusdb-admin → secantus-admin and secantusdb-restore-archive → secantus-restore-archive. The PyPI project name (SecantusDB / pip install secantusdb) and the secantus import package are unchanged.

The shared configuration file is renamed to match: both Mongo daemons read secantusd.toml (auto-discovered in the cwd, ~/.secantus/, and /etc/secantus/). The legacy secantusdb.toml name is still discovered at each location — the new name wins on a tie — so an existing config file keeps working.

Added

• secantusd-py / secantusd-py-pg console scripts; secantusd-py-pg is a new CLI entry point (main() / build_parser()) for SecantusPGServer.

• Config auto-discovery now probes secantusd.toml ahead of the legacy secantusdb.toml at every location, in both the Python and Rust loaders.

Changed

• The standalone Rust binary is now emitted as secantusd-rs (was secantusdb); its --version / --help / startup banner and the secantusd-rs restore usage text follow suit.

• Utility console scripts renamed: secantusdb-admin → secantus-admin, secantusdb-restore-archive → secantus-restore-archive (the argparse prog= / help text follow).

• secantusdb.toml.example renamed to secantusd.toml.example.

Removed

• The secantusdb and secantus console-script aliases of the Python server. Use secantusd-py (or python -m secantus).

Bitwise aggregation operators $bitAnd / $bitOr / $bitXor / $bitNot (both servers)

Both servers now support MongoDB 6.3’s bitwise aggregation expressions. $bitAnd / $bitOr / $bitXor fold a list of int/long operands with the corresponding bitwise operator; $bitNot complements a single operand. The result is a long when any operand is a long and an int otherwise; a null or missing operand makes the whole result null; an empty operand list yields the operator’s identity (all-ones for $bitAnd, 0 for $bitOr / $bitXor). A non-integer operand (double, bool, decimal, …) raises, matching mongod. Neither server recognised these before.

Added

• expressions.py / secantus-core: $bitAnd / $bitOr / $bitXor / $bitNot aggregation expressions, with int32/int64 result-width tracking. (The $group accumulator forms remain a follow-on — see tasks/backlog.md §7.5.)

Rust server: $stdDevPop / $stdDevSamp group accumulators

The Rust server now supports the $stdDevPop and $stdDevSamp accumulators in $group (and $setWindowFields), matching the Python server — population standard deviation (÷n, 0 for a single value) and sample standard deviation (÷n-1, null for fewer than two values). Previously the Rust server rejected these accumulators; the Python server already had them.

To keep the two engines bit-for-bit identical, both now compute the deviation with the same fixed sequence of correctly-rounded IEEE operations — a naive left-fold float sum, multiply-based squaring, and hardware sqrt. CPython 3.12’s sum() builtin switched to Neumaier compensated summation for floats, which is more accurate but would diverge from the Rust engine’s naive fold by a last ULP, so the Python _std_dev now sums with an explicit loop instead of sum(). (mongod uses an online Welford-style algorithm, so neither server matches it to the last ULP; aligning the two SecantusDB engines is the goal.) A parity fuzz seed caught the divergence before it shipped.

Added

• secantus-core (group.rs): $stdDevPop / $stdDevSamp accumulators (numeric values folded to f64, non-numeric defers to Python), shared by $group and $setWindowFields.

Changed

• aggregate.py (_std_dev): compute with a naive float fold + multiply + sqrt (no sum() / ** 2 / ** 0.5) so the Python and Rust engines agree bit-for-bit.

Stable hello topologyVersion — no more spurious connection-pool churn (both servers)

hello’s topologyVersion.processId was minted fresh on every call (ObjectId from now() on the Python server, ObjectId::new() on the Rust server). The SDAM spec treats a changed processId as “the server restarted”, so drivers reacted to nearly every monitoring heartbeat by invalidating and clearing the connection pool — closing the live connection and reconnecting. Both servers now pin processId once per process, so a driver sees a stable topology and keeps its connections. This was surfaced by the Java driver’s connection-pool-logging and client-metadata event-count tests (which observed the extra connectionClosed / “Connection pool cleared” events), but the churn affected every driver’s SDAM.

Fixed

• Python server (commands.py) and Rust server (secantus-commands handshake.rs): hello returns a process-stable topologyVersion.processId instead of a fresh ObjectId per call. Regressions: tests/test_hello_topology.py (pymongo, both processId stability and cross-connection identity) and a secantus-commands unit test.

$dateToParts honours a timezone (both servers)

$dateToParts now accepts a timezone and returns the year/month/day/hour/… parts read off the wall clock in that zone, instead of always in UTC. {$dateToParts: {date: "$d", timezone: "America/New_York"}} on a 16:30Z instant returns hour 11 in winter (EST); Asia/Tokyo rolls the day forward. Fixed-offset zones (+05:30, UTC) and named IANA zones both resolve; no timezone still reads UTC. Like the date extractors, this was a gap in both servers — each previously ignored timezone on $dateToParts.

The Rust server resolves named zones via the shared timezone_offset_ms helper (the unambiguous instant→wall-clock direction, matching Python zoneinfo); an unknown zone name defers to the Python oracle. $dateTrunc / $dateDiff timezone remain deferred — $dateTrunc truncates to a local boundary that must convert back to an instant (local→instant, DST-ambiguous), and $dateDiff’s day/week already count elapsed duration rather than local-calendar boundaries, so timezone there would compound an existing divergence (see tasks/backlog.md §7.5).

Added

• expressions.py / secantus-core: $dateToParts reads timezone (fixed-offset

  • named IANA), shifting the instant into the zone before splitting into parts. The Rust side factors the fixed-offset/named-zone resolution into a shared timezone_offset_ms helper now used by $dateToString, the {date, timezone} extractors, and $dateToParts.

Date component extractors honour a timezone (both servers)

The date component operators — $year, $month, $dayOfMonth, $dayOfWeek, $hour, $minute, $second — now accept mongod’s {date: <expr>, timezone: <expr>} object form, reading the component off the wall clock in the requested zone rather than always in UTC. {$hour: {date: "$d", timezone: "America/New_York"}} on a 16:30Z instant returns 11 in winter (EST) and 12 in summer (EDT); a shift that crosses midnight moves $dayOfMonth too. Both fixed-offset zones (+05:30, UTC) and named IANA zones resolve, and a bare date expression still reads UTC as before.

This closes a conformance gap that was present in both servers — previously each ignored timezone on these operators and a {date, timezone} argument silently returned null. The Rust server resolves named zones via the same chrono-tz path as $dateToString (the unambiguous instant→wall-clock direction, matching Python zoneinfo); an unknown zone name defers to the Python oracle.

Added

• expressions.py / secantus-core: a shared date-operand resolver (_date_operand / date_operand_millis) that both the bare-date and {date, timezone} object forms of the seven date component extractors route through, shifting the instant into the requested zone before the component is read. Fixed-offset and named IANA zones both supported on both servers.

Rust server: $dateToString now formats dates in named IANA timezones

The Rust server can now render $dateToString in a named IANA timezone — {$dateToString: {date: "$d", timezone: "America/New_York"}} — with the correct daylight-saving offset for the instant being formatted, exactly as the Python server (and real mongod) do. A summer date shifts by -04:00 (EDT), a winter one by -05:00 (EST); Europe/Dublin, Asia/Tokyo, and every other zone resolve the same way. Previously the Rust server accepted only fixed-offset zones (+05:30, UTC) and errored on a named zone, since it has no Python zoneinfo to defer to.

This is the unambiguous instant-to-wall-clock direction: a UTC instant maps to exactly one local time in any zone, so the bundled chrono-tz database and Python zoneinfo agree. ($dateFromString’s named-zone form — naive-local-to-instant, which is ambiguous across a DST gap/overlap — still defers to the Python oracle.)

Added

• secantus-core: chrono-tz dependency and a named-zone offset resolver (named_tz_offset_ms) wired into $dateToString. A named zone resolves its DST-correct UTC offset at the rendered instant; an unknown zone name still defers. Both servers now agree on named-timezone $dateToString.

Rust server: connection-auth mutex locks are poison-tolerant

The three production sites that lock the per-connection auth mutex in secantus-commands used .lock().expect("conn auth mutex poisoned"), which would panic if the mutex had been poisoned by an earlier panic while the lock was held. They now use the same .lock().unwrap_or_else(|e| e.into_inner()) pattern as the rest of the crate (logbuf, transactions), recovering the guard instead of panicking — so a poisoned auth mutex degrades to a caught error at the catch_unwind dispatch boundary rather than compounding into a second panic. No behaviour change on the healthy path. Found by the nightly security review (2026-07-04 §I14).

Changed

• secantus-commands (lib.rs ×2, auth.rs ×1): connection-auth mutex locks are poison-tolerant. The two remaining .lock().unwrap() sites the review flagged are in #[cfg(test)] mock stores and are intentionally left panicking (a test that poisons its own mock lock should fail loudly).

Rust server: $dateToString now formats dates in named IANA timezones

The Rust server can now render $dateToString in a named IANA timezone — {$dateToString: {date: "$d", timezone: "America/New_York"}} — with the correct daylight-saving offset for the instant being formatted, exactly as the Python server (and real mongod) do. A summer date shifts by -04:00 (EDT), a winter one by -05:00 (EST); Europe/Dublin, Asia/Tokyo, and every other zone resolve the same way. Previously the Rust server accepted only fixed-offset zones (+05:30, UTC) and errored on a named zone, since it has no Python zoneinfo to defer to.

This is the unambiguous instant-to-wall-clock direction: a UTC instant maps to exactly one local time in any zone, so the bundled chrono-tz database and Python zoneinfo agree. ($dateFromString’s named-zone form — naive-local-to-instant, which is ambiguous across a DST gap/overlap — still defers to the Python oracle.)

Added

• secantus-core: chrono-tz dependency and a named-zone offset resolver (named_tz_offset_ms) wired into $dateToString. A named zone resolves its DST-correct UTC offset at the rendered instant; an unknown zone name still defers. Both servers now agree on named-timezone $dateToString.

Projection inclusion/exclusion mix reports mongod’s specific error code

Mixing an inclusion and an exclusion in the same find() projection (e.g. {a: 1, b: 0}) now fails with mongod’s specific per-field error instead of a generic TypeMismatch. An exclusion inside an inclusion projection returns Location31254 (“Cannot do exclusion on field b in inclusion projection”); an inclusion inside an exclusion projection returns Location31253 (“Cannot do inclusion on field b in exclusion projection”). The offending field is named in the message, matching real mongod and the Rust server (which already emitted these codes). Drivers’ projection-error tests assert both the code and the exact wording, so this closes a pymongo-gauge fidelity gap.

Fixed

• projection.py: _detect_inclusion validates field-by-field in order and raises ProjectionError with code/code_name pinned to 31254/31253 (the offending field named); ProjectionError gained a code/code_name constructor so the dispatch layer surfaces the specific code instead of 14.

The Rust server prunes its oplog from write volume alone

The standalone Rust secantusd-rs server used to prune its oplog only from the noop-heartbeat thread — and that thread is off by default (--noop-heartbeat-seconds 0). A long-lived, busy server with the default configuration could therefore grow its oplog past the retention window and entry cap without bound, since nothing was trimming it. The Rust storage engine now self-prunes on the write path, exactly as the Python server has always done: every 1000 emitted oplog entries it runs an opportunistic prune_oplog, so the oplog stays bounded from writes alone, with or without a heartbeat. Document data is never touched by a prune.

Fixed

• secantus_storage::emit_oplog bumps an in-memory emit counter and runs an opportunistic prune_oplog every OPLOG_PRUNE_INTERVAL (1000) entries — mirroring the Python server’s _emit_oplog. The prune reuses the write path’s already-held storage lock (prune_oplog_inner(..., take_lock: false)) and is best-effort, so it never fails an otherwise successful write.

Storage is now a context manager

Storage gained the __enter__ / __exit__ protocol, so you can write with Storage(path) as store: and have WiredTiger torn down (background threads joined, oplog meta persisted, connection closed) on block exit — even if the body raises — instead of relying on the embedder to remember close(). close() remains idempotent, so an explicit close inside the block is still safe. Found by the nightly security review (2026-07-04 §I12).

Added

• Storage.__enter__ / Storage.__exit__ — context-manager support that calls close() on exit. Tests in tests/test_storage_ctxmgr.py.

The standalone Rust server now honours every Python-server CLI flag

The pure-Rust secantusd-rs binary — the standalone server you run with no Python interpreter in the process — used to accept only a subset of the daemon flags the Python server understands. That gap meant a secantusd.toml or flag set tuned for one server wouldn’t drive the other. It’s now closed: the Rust binary accepts the same eight configuration flags the Python launcher does, with real behaviour behind each, so a single configuration drives either server identically.

--config loads a secantusd.toml with the same discovery order (./, ~/.secantus/, /etc/secantus/), the same defaults < file < CLI flag precedence, table renames, and strict unknown-key/table rejection as the Python loader. --log-level initialises a logger to the requested level; --cache-size, --session-max, and --sync-on-commit tune the WiredTiger connection; and --oplog-retention-seconds, --oplog-max-entries, and --noop-heartbeat-seconds control oplog retention plus a background heartbeat that keeps quiet change-stream cursors’ resume tokens inside the retention window. The heartbeat thread and a TTL sweeper also give the standalone binary the periodic oplog-prune and expired-document maintenance the Python daemon has always run.

One behavioural alignment worth calling out: the standalone daemon’s default WiredTiger cache is now 1G, matching python -m secantus (the embedded Rust handle’s default is unchanged).

Added

• The --config / --log-level / --cache-size / --session-max / --sync-on-commit / --noop-heartbeat-seconds / --oplog-retention-seconds / --oplog-max-entries flags on the standalone secantusd-rs binary, with a faithful TOML config loader (secantus-server’s new config module) mirroring src/secantus/config.py’s precedence and validation.

• Background noop-heartbeat (with opportunistic prune_oplog) and TTL-sweep threads in the secantusd-rs binary, sharing one Arc<Storage> and joining on a shutdown flag before WiredTiger closes.

• secantus_storage::wt_config builds the WiredTiger connection string from the resolved cache-size / session-max / sync-on-commit knobs.

Changed

• The standalone secantusd-rs daemon defaults its WiredTiger cache to 1G (matching the Python server) instead of the engine’s 256M default.

Projection inclusion/exclusion mix reports mongod’s specific error code

Mixing an inclusion and an exclusion in the same find() projection (e.g. {a: 1, b: 0}) now fails with mongod’s specific per-field error instead of a generic TypeMismatch. An exclusion inside an inclusion projection returns Location31254 (“Cannot do exclusion on field b in inclusion projection”); an inclusion inside an exclusion projection returns Location31253 (“Cannot do inclusion on field b in exclusion projection”). The offending field is named in the message, matching real mongod and the Rust server (which already emitted these codes). Drivers’ projection-error tests assert both the code and the exact wording, so this closes a pymongo-gauge fidelity gap.

Fixed

• projection.py: _detect_inclusion validates field-by-field in order and raises ProjectionError with code/code_name pinned to 31254/31253 (the offending field named); ProjectionError gained a code/code_name constructor so the dispatch layer surfaces the specific code instead of 14.

Admin UI: stop passing the raw (credential-bearing) MongoDB URI into the server page

The admin server page’s render context included current_uri_raw, the raw mongo_uri (which can carry a username/password), even though no template ever referenced it — a latent credential-exposure surface with no live leak. Removed the dead context variable; the page continues to show the scrubbed current_uri_display. Found by the nightly security review (2026-07-04 §I13).

Security

• admin/routers/server.py: drop the unused current_uri_raw template context variable so the unscrubbed connection URI is no longer handed to the renderer.

PostgreSQL/SQL server: malformed messages get an error reply instead of a dropped connection

The PG/SQL server’s simple-query loop now answers a malformed message with a proper ErrorResponse rather than letting the exception escape to the outer handler and silently drop the connection. Most notably, a SQL syntax error over the simple-query protocol previously escaped (the parse ran outside the handler’s try) and dropped the connection with no reply; it now returns 42601 and the connection stays alive. Invalid UTF-8 in a query message returns 08P01 and the connection survives (the message was length-framed, so the byte stream is still in sync), and a genuine framing error (an implausible length prefix) sends a FATAL 08P01 before closing instead of dropping silently.

Found by the nightly security review (2026-07-04 §I16).

Fixed

• sql/pgserver.py: planner.parse moved inside _handle_query’s try so a syntax error returns 42601 instead of dropping the connection; _query_loop now catches PGProtocolError (framing) and UnicodeDecodeError (query text) and replies with 08P01. Regression: tests/test_pgserver_framing.py.

PostgreSQL/SQL server: a malformed SCRAM client-first is a typed auth error

A truncated SCRAM gs2 header (e.g. a client-first message of just "n,", with no bare message after the header) made ScramExchange.server_first raise a bare ValueError from an unpack, caught only by the connection’s outer generic handler. It now raises the typed PGAuthError, consistent with the rest of the PG-auth path — the connection still fails cleanly with no leak, just via the right exception type. Found by the nightly security review (2026-07-04 §I21).

Fixed

• sql/pgauth.py: ScramExchange.server_first guards the gs2-header split and raises PGAuthError on a truncated header instead of an unpack ValueError. New unit tests in tests/test_pgauth.py.

$push / $addToSet honour the $each modifier

$push and $addToSet now unwrap the $each modifier and append every element, instead of storing the {$each: […]} document as a single array element. So {$push: {scores: {$each: [90, 85, 82]}}} appends three scores, and {$addToSet: {tags: {$each: ["a", "b", "a"]}}} adds each not-already-present tag — matching MongoDB. $push also honours the companion modifiers $position (insert at an index, negative from the end), $slice (keep the first N / last |N| / none), and $sort (order the array — whole elements by 1/-1, or documents by a {field: dir} spec, in BSON order).

Previously both operators appended the modifier document verbatim — a silent data-shape bug for one of the most common update forms. It ships on both the Python and Rust servers, pinned by the update parity suite; the Rust engine computes $each / $position / $slice natively and defers $sort (BSON-order array sort) to the Python oracle.

Fixed

• $push / $addToSet $each is now unwrapped (multi-element append / add), with $push $position / $slice / $sort modifiers, instead of the $each document being stored as a single element. Ships on both servers (update.apply_update + secantus-core::update).

Rust server: correct error codes for unrecognized / Atlas aggregation stages

The Rust server now validates aggregation stage names up-front, matching the Python server (and mongod): an unrecognized stage ({$badStage: …}) is rejected with Location40324 (“Unrecognized pipeline stage name”), and an Atlas-only stage ($search / $vectorSearch / $searchMeta / $listSearchIndexes) with CommandNotSupported (115) and the Atlas configuration message. Previously both surfaced as a generic BadValue (2), so drivers couldn’t tell an unknown stage from any other unsupported construct.

Fixed

• Rust server: an unrecognized aggregation stage now returns code 40324 (was 2), and an Atlas-only stage returns 115 with the Atlas message (aggregate::validate_stage_names, recognized-stage set kept in sync with the Python aggregate._STAGES registry). Regression: tests/test_rust_server_smoke.py::test_aggregate_stage_name_validation_against_rust_server.

PostgreSQL/SQL server: an unexpected internal error no longer leaks its Python text to the client

When a statement hit an unexpected exception (not a curated SQLError), the PostgreSQL/SQL server sent the raw str(exc) back to the client as internal error: <text>. That text could disclose internal file paths, type names, or document values. The server now logs the full exception server-side (as it already did) but answers the client with a generic XX000 internal error, matching the Mongo dispatch’s discipline of never leaking a Python traceback to the wire. Curated SQLErrors are unaffected — they still surface their real SQLSTATE and user-facing message.

Found by the nightly security review (2026-07-04 §I17).

Security

• sql/pgserver.py (simple-query + COPY paths) and sql/pgextended.py (extended protocol) no longer interpolate the raw exception into the ErrorResponse for an unexpected internal error. Regression: tests/test_pgserver_error_hygiene.py.

Reads on a view resolve its pipeline

find, aggregate, and count_documents on a view now run the view’s stored pipeline against its base collection instead of returning nothing. Create a view with db.createView("active", "users", [{$match: {status: "active"}}]) and a find / aggregate on active reads exactly the matching users — with the caller’s filter, sort, skip, limit, and projection applied on top (a find is translated into the equivalent aggregate over the base collection). A view defined on another view resolves recursively.

Previously only the count command resolved views; find and aggregate (and therefore count_documents, which pymongo implements via aggregate) treated the view as an empty collection.

Fixed

• find / aggregate on a view now resolve the view’s viewOn + pipeline (recursively for a view-on-a-view) via commands._resolve_view, applying the request’s own filter/sort/skip/limit/projection over the result. Fixes count_documents on a view returning 0.

Rust server: reads on a view resolve its pipeline

The Rust server now mirrors the Python server: find, aggregate, and count_documents on a view resolve the view’s viewOn + stored pipeline against its base collection (recursively for a view-on-a-view) instead of returning nothing. The aggregate command’s initial fetch resolves the view chain (next to the leading-$match / $geoNear lifts, keeping the view name for the reply ns); find on a view is translated into the equivalent aggregate. Previously the Rust server treated a view as an empty collection.

Fixed

• Rust server: find / aggregate / count on a view resolve viewOn + viewPipeline (aggregate::resolve_view), applying the request’s own filter/sort/skip/limit/projection on top. Regression: tests/test_rust_server_smoke.py::test_view_reads_resolve_against_rust_server.

Rust server: refuses direct writes to synthetic read-only views

The Rust server now rejects a direct insert / update / delete on local.oplog.rs or admin.system.users with Unauthorized (13), matching the Python server (and mongod’s RBAC-denial code). These namespaces are synthetic read-only views — local.oplog.rs projects the oplog WT table (written only via oplog emission) and admin.system.users is fronted by createUser / updateUser / dropUser — so a direct write would land in the wrong table or break the view’s invariants. Previously the Rust server silently accepted such writes. A regular collection write is unaffected.

Fixed

• Rust server: direct insert / update / delete on local.oplog.rs / admin.system.users is now rejected with code 13 (matching the Python server), instead of being silently accepted. Regression: tests/test_rust_server_smoke.py:: test_synthetic_view_write_rejected_against_rust_server.

PostgreSQL/SQL server: per-statement RBAC, reusing the Mongo role model

The PostgreSQL/SQL server gained an authorization layer. Until now an authenticated SQL client — or, in the documented trust default, any client — could run any statement against any database it named at connect time, so a SQL client had broader effective access than an equivalently-authenticated Mongo client on the same data. The SQL surface now enforces the same RBAC engine the Mongo server uses (secantus.rbac): each statement maps to one action on the connection’s database and is checked against the authenticated user’s roles.

Authorization is opt-in and backward-compatible. Start the server with require_auth=True and per-user role bindings via the new user_roles argument ({"analyst": [{"role": "read", "db": "shop"}]}) to turn it on; without user_roles, and for the embedded run_sql API, the surface stays unrestricted exactly as before. Built-in roles (read / readWrite / dbAdmin / dbOwner / root) resolve directly and custom roles resolve through the shared roles table, so a role defined once governs both protocols on a shared Storage. A denied statement returns SQLSTATE 42501 and the connection survives.

Found by the nightly security review (issue #193).

Added

• SecantusPGServer(user_roles=...): per-user RBAC role bindings, enforced per-statement when require_auth is on. New pure sql/authz.py maps each statement to an rbac action and calls rbac.check_privilege; transaction control, SET/SHOW, and cursor navigation need no privilege. New errors.insufficient_privilege (SQLSTATE 42501).

Security

• SQL clients on a Storage shared with a Mongo server are now gated by the same roles as Mongo clients, closing the “authenticated SQL client has unrestricted access” gap. Regressions in tests/test_sql_authz.py (engine-level, built-in + custom roles) and tests/test_pgserver_auth.py (over the wire).

Rust server: rejects a malformed writeConcern

The Rust server now validates a write command’s writeConcern before running it, matching the Python server (and mongod). A negative or too-large integer w is a FailedToParse (9), a string w other than "majority" is an UnknownReplWriteConcern (79), and a bool / non-number-or-string w — or a non-bool/int j or non-number wtimeout — is a TypeMismatch (14). A well-formed writeConcern is accepted as before, and a satisfiable-but-too-wide w > 1 still succeeds with the single-node writeConcernError attached. The check runs in dispatch for every write command (insert / update / delete / findAndModify / create / collMod / createIndexes / drop / dropIndexes / dropDatabase / renameCollection).

Fixed

• Rust server: malformed writeConcern values are now rejected with mongod’s codes (9 / 79 / 14) before the write runs, instead of being silently accepted. Regression: tests/test_rust_server_smoke.py:: test_write_concern_validation_against_rust_server.

Rust server: $lookup drives the foreign index (and fixes its result order)

The Rust server’s simple-form $lookup now drives a per-outer-doc index probe when the foreign collection has a leading-field index on foreignField (single-field, compound-prefix, or multikey — all resolve to an IXSCAN via Storage::find), falling back to the full-scan hash-join only when no such index exists. This mirrors the Python server’s $lookup path.

Besides the performance win, this fixes a two-server divergence: the previous hash-join returned the joined as array in foreign-collection scan order, while the Python server (and the index probe) return it in index order. For an indexed $lookup the two servers now produce byte-for-byte identical results, including the order of documents within as. A regression test drives the Rust server and asserts the index-ordered output.

Fixed

• Rust server: simple-form $lookup over an indexed foreignField now returns the as array in index order (matching the Python server) instead of foreign-scan order, and rides the index instead of materialising the whole foreign collection. Regression: tests/test_rust_server_smoke.py:: test_lookup_index_order_against_rust_server.

PostgreSQL/SQL server: connection cap, cursor caps, and a parser guard

The PostgreSQL wire server gained the resource limits its MongoDB counterpart already had, closing a compounding denial-of-service surface (all reachable pre-auth, since require_auth defaults off):

• Connection cap. SecantusPGServer now enforces max_connections (default 1000, matching the Mongo server’s DEFAULT_MAX_CONNECTIONS). An over-cap accept is closed immediately instead of spawning a handler thread.

• Per-session cursor caps. DECLARE CURSOR (which eagerly materializes its whole result set) now rejects a session’s 101st open cursor, and a cursor whose result exceeds 1,000,000 rows, with program_limit_exceeded (SQLSTATE 54000). Combined with the connection cap, total cursor memory is bounded.

• Parser guard. planner.parse rejects a statement longer than 1 MB before handing it to sqlglot, and converts the RecursionError from a deeply-nested statement (e.g. hundreds of parentheses — a ~600-byte trigger) into a clean 54000 error instead of relying on the connection loop’s broad except.

Found by the nightly security review (issue #194).

Security

• sql/pgserver.py: max_connections cap (default DEFAULT_MAX_CONNECTIONS).

• sql/engine.py: MAX_CURSORS_PER_SESSION (100) + MAX_CURSOR_ROWS (1e6) caps on DECLARE CURSOR.

• sql/planner.py: MAX_SQL_LENGTH (1 MB) cap + RecursionError guard in parse. New errors.program_limit_exceeded (SQLSTATE 54000). Regressions in tests/test_sql_cursors.py + tests/test_pgserver.py.

$geoNear rides the geo index instead of scanning

A bounded $geoNear — one with a maxDistance — no longer walks the whole collection computing a distance for every document. When it’s the leading stage and the collection has a matching geo index on the queried field, SecantusDB now lifts the search into a conservative $geoWithin candidate query and serves it through the same geo-index path $near and $geoWithin already use, then computes exact distances and sorts over just the candidates.

The candidate radius is inflated by a negligible epsilon so the fetched set is a strict superset of the exact within-maxDistance set; the $geoNear stage then re-applies the exact distance filter, so results — the documents, their order, and the attached distanceField — are byte-for-byte identical to the brute-force path. Only the number of documents fetched shrinks. An unbounded $geoNear (no maxDistance) must still return every document in distance order, so it keeps scanning; the optimization is scoped to the bounded case, and a mismatched index type falls back to the full scan. A randomized regression test asserts the optimized output equals the scan output across many queries.

Changed

• $geoNear with a maxDistance and a matching 2dsphere / 2d index now fetches candidates through the geo index (aggregate._geo_near_index_filter, lifted into the aggregate command’s initial fetch) instead of a full collection scan. Output is unchanged. (Rust-server mirror below.)

Rust server: $geoNear rides the geo index for a bounded search

The Rust server now mirrors the Python server’s $geoNear index optimization. A leading $geoNear with a maxDistance and a matching 2d / 2dsphere index no longer scans the whole collection: the search is lifted into a conservative $geoWithin candidate fetch in the aggregate command’s initial fetch (next to the existing leading-$match lift), and the $geoNear stage then re-applies the exact distance filter over just the candidates.

The candidate radius is inflated by a negligible epsilon so the fetched set is a strict superset of the exact within-maxDistance set, keeping the output — docs, order, and attached distanceField — byte-for-byte identical to the brute-force path. An unbounded $geoNear still scans, and a mismatched index type falls back to the full scan. A regression test drives the Rust server directly, asserting the optimized output equals the scan output across many random queries.

Changed

• Rust server: $geoNear with a maxDistance and a matching geo index fetches candidates through the index (aggregate::geo_near_index_filter in secantus-commands) instead of a full collection scan. Output is unchanged. Regression: tests/test_rust_server_smoke.py:: test_geo_near_index_optimization_against_rust_server.

Constant-time secret comparison in the PostgreSQL SCRAM + admin-token checks

Two authentication comparisons used a plain != / == on secret material, which CPython does not guarantee to run in constant time — a timing side-channel that can narrow a secret a byte at a time. The PostgreSQL/SQL server’s SCRAM proof check (sql/pgauth.py) compared the recomputed stored-key digest with !=, and the admin console’s token middleware (admin/middleware.py) compared the presented token with != / == on both the HTTP and WebSocket paths. All three now use hmac.compare_digest (bytes-encoded, so a non-ASCII presented value is rejected rather than raising), matching the Mongo-side SCRAM check in secantus.auth which was already constant-time. Behaviour is otherwise unchanged — valid credentials/tokens are accepted, wrong or missing ones rejected. Found by the nightly security review (issue #195).

Security

• sql/pgauth.py: SCRAM stored-key comparison uses hmac.compare_digest.

• admin/middleware.py: HTTP TokenAuthMiddleware and verify_websocket_token compare the admin token with hmac.compare_digest (and reject a missing/non-ASCII token without raising). Regression: tests/test_admin_skeleton.py::test_verify_websocket_token_is_constant_time_and_robust.

Rust server: bool-as-int range comparison in the query matcher

The Rust server’s query matcher now compares a boolean field against a numeric $gt / $lt / $gte / $lte bound (and vice versa) natively, instead of deferring the whole match to a BadValue. Following the Python oracle — which compares with Python’s <, where bool is an int subclass — True counts as 1 and False as 0: {x: {$gt: 0}} now matches x: true on the Rust server, matching the Python one. A bool compared against a non-numeric type (string, Decimal128, date, …) is TypeError in Python, i.e. no match, and is reproduced as such. Boolean equality is unaffected (a bool stays distinct from 1).

Pinned by the query parity suite (curated bool-vs-number / bool-vs-non-numeric / multikey cases, plus the existing randomized fuzz whose scalar corpus already includes booleans — 0 divergences).

Added

• Rust server: bool-as-int $gt / $lt / $gte / $lte comparison in the secantus-core query matcher (numeric compare vs int / long / double; no match vs any other type), matching the Python oracle’s < semantics.

Rust server: getLog returns a real in-memory log

The Rust server’s getLog was a stub returning an empty array; it now surfaces a real bounded in-memory log ring buffer (secantus_commands::logbuf::LogBuffer, 5000-entry cap, the Rust port of logbuf.py). The server records a startup line and a "connection accepted" NETWORK line per connection, and getLog: "global" returns them as mongod-shaped pre-formatted strings ("<ts> <level> <component> <msg>") with totalLinesWritten. The admin console’s Logs page now shows activity against a Rust target instead of an empty table. Slice 5 of the Rust admin-command parity work (issue #163).

This resolves the last genuine gap in #163: with getLog real, killOp and the role grant/revoke + native backup/restore/prune commands all landed, and the profile command was already at parity with the Python server (both persist the level / slowms / sampleRate config; neither captures slow ops — out of scope for the surrogate). currentOp / serverStatus remain intentionally minimal.

Added

• Rust server: secantus_commands::logbuf::LogBuffer + a getLog-backed log ring buffer wired through the server (connection-accept + startup lines) and a logs handle on CommandContext. Regressions: crates/secantus-commands unit tests (buffer append/tail/capacity; getLog formatting; empty without a buffer) and tests/test_rust_server_smoke.py::test_get_log_returns_connection_lines.

Rust server: $min / $max / $addToSet / $pull update operators

The Rust server now applies four more update operators natively instead of rejecting them: $min and $max (keep the smaller / larger of the current and given value), $addToSet (append to an array only if not already present), and $pull (remove the array elements equal to a value). Previously each surfaced as a BadValue error on the Rust server; they now match the Python server.

Fidelity follows the Python oracle exactly. $min/$max compare with Python’s < semantics for numeric / string / date pairs (bool counts as its integer value; a cross-type comparison Python would raise on defers to Python), and an absent-or-null field is treated as “no current value”. $addToSet and $pull use Python == element equality — cross-type-equal numerics (1 == True) and structural document/array equality included — via the shared value-equality helper. It’s pinned by the update parity suite with curated edge cases and a fuzz corpus (0 divergences). $bit was already native.

Added

• Rust server: $min / $max / $addToSet / $pull update operators in the secantus-core update engine, matching the Python oracle’s comparison / == element semantics.

Rust server: $dateFromString format (strptime)

The Rust server now parses a $dateFromString format string natively for the numeric-directive subset (%Y %y %m %d %H %M %S %j %%, literals, and whitespace), instead of the whole expression being unsupported. So $dateFromString: {dateString: "15/01/2024", format: "%d/%m/%Y"} now evaluates on the Rust server, matching the Python one.

Fidelity is structural: the format is translated into a regex built from CPython’s own _strptime per-directive sub-patterns, so field matching — the 2-digit-year pivot (00–68 → 2000s, 69–99 → 1900s), the value-range digit rules, single-digit leniency, day-of-year, and full-input consumption — is identical to Python’s by construction. It’s pinned by the expression parity suite with a 6000-case fuzz corpus (0 divergences). Directives outside the subset (%z / %Z / %a / %b / %p / …), a %j combined with %m/%d, a leap second, or any input Python would reject still defer to the Python oracle.

Added

• Rust server: $dateFromString format (strptime) for the numeric-directive subset, built from CPython _strptime’s exact regex fragments for byte-faithful field matching. Combined with the fixed-offset timezone support, a naive strptime result is interpreted in the given offset zone.

Rust server: killOp

The Rust server now implements killOp. Real mongod signals a per-op interrupt flag; SecantusDB’s faithful analogue (in both servers) is “close the socket” — since we model one in-flight op per connection, the op a caller passes is the connection’s conn_id (readable off hello’s connectionId). The server gained a live-connection registry (conn_id → socket clone, populated on accept and cleared on disconnect); killOp shuts down the target socket, so the connection thread’s next read returns 0 and the connection ends. The opid is accepted as Int32 / Int64 / integral Double / numeric string, and the reply mirrors the Python handler: {info: "operation killed" | "no operation with that opid" | "no connection registry", ok: 1}, or TypeMismatch for a non-integer op. The admin console’s Connections → Kill button now works against a Rust target. Slice 4 of the Rust admin-command parity work (issue #163).

Remaining #163 gaps: fleshing out the getLog / profile stubs (both need real capture infrastructure — a log ring buffer and slow-op timing).

Added

• Rust server: killOp (gated by A_KILLOP under --auth), backed by a new server-side connection registry + a ConnectionKiller handle on CommandContext. Regressions: crates/secantus-commands unit tests (registry-present/absent, found/not-found, numeric-string opid, non-integer → TypeMismatch) and tests/test_rust_server_smoke.py::test_kill_op_closes_a_connection.

Rust server: fixed-offset timezone on $dateToString / $dateFromString

The Rust server now handles a fixed-offset timezone on the $dateToString and $dateFromString aggregation-expression operators natively, instead of the whole expression being unsupported. $dateToString with timezone: "+05:30" shifts the wall clock before formatting; $dateFromString with a timezone interprets a naive date string as being in that zone (and ignores it when the string already carries its own offset), matching the Python server exactly. The UTC / GMT aliases and both the ±HHMM and ±HH:MM offset spellings are accepted.

Named IANA zones (America/New_York) still defer — they need a bundled tz database — as does a $dateFromString format (strptime). This narrows the gap between the two servers on the date operators; the behaviour is pinned against the Python oracle by the expression parity suite.

Added

• Rust server: fixed-offset / UTC / GMT timezone support on $dateToString and $dateFromString (offset arithmetic in the shared secantus-core expression engine). Named IANA zones and format strptime still defer.

$jsonSchema gains uniqueItems

The $jsonSchema query operator now understands uniqueItems. With uniqueItems: true on an array schema, a document matches only when every element of the array is distinct — the way to say “this list has no repeats” in a validator or a $jsonSchema query. Elements are compared by value: cross-type numerics that are numerically equal (1 and 1.0) count as duplicates, and documents compare field-by-field in order, matching how mongod compares them.

The check ships on both the Python and Rust servers, pinned by the query parity suite; it reuses the shared byte-sortable value encoding as the equality key, so both engines agree bit-for-bit. (One documented gap: cross-type-equal numerics nested inside a document element — [{a: 1}, {a: 1.0}] — are treated as distinct on both servers; top-level scalar arrays are fully faithful.)

Added

• $jsonSchema uniqueItems: true — rejects arrays with duplicate elements (value equality, cross-type-numeric aware for top-level scalars). false is a no-op.

Rust server: grantRolesToUser / revokeRolesFromUser

The Rust server now implements the two user-role management commands it was missing: grantRolesToUser adds roles to a user’s assignment list (deduped by (role, db)), and revokeRolesFromUser removes them. Both validate the requested roles against the built-in + custom role catalogue (RoleNotFound), require the user to exist (UserNotFound), and refresh the calling connection’s effective roles so a privilege change takes effect immediately — matching the Python handlers. Previously the Rust server could only set a user’s roles at createUser / updateUser time; the admin console’s Users → Roles editor can now grant/revoke against a Rust target. Slice 3 of the Rust admin-command parity work (issue #163).

Remaining #163 gaps: killOp, and fleshing out the getLog / profile stubs.

Added

• Rust server: grantRolesToUser / revokeRolesFromUser commands (gated by A_GRANT_ROLE / A_REVOKE_ROLE under --auth). Regressions: crates/secantus-commands unit tests (grant/revoke, dedup, UserNotFound, RoleNotFound) and tests/test_rust_server_smoke.py::test_secantus_grant_revoke_roles_to_user.

Rust server: secantusAdmin.restoreArchive

The Rust server now implements secantusAdmin.restoreArchive, completing the proprietary backup/restore + prune command family (backupArchive and the prune commands already shipped). It extracts a backup .tar.gz (produced by backupArchive) into a fresh targetDir the operator then points a new server at — the running server’s storage is untouched, the same side-channel restore model as the Python command and real mongod’s “stop, swap dbpath, start”. It rejects a non-empty target unless allowExisting: true, verifies the archive is a genuine SecantusDB / WiredTiger backup before extracting (so a malformed archive can’t pollute the target), and returns {targetDir, fileCount, archive, ok}, mirroring the Python handler. This is slice 2 of the Rust admin-command parity work (issue #163); the admin console’s Backup page can now drive native archive restore against a Rust target.

Remaining #163 gaps: the standard admin commands grantRolesToUser / revokeRolesFromUser / killOp, plus fleshing out the getLog / profile stubs.

Added

• Rust server: secantusAdmin.restoreArchive wire command (Storage::restore_archive on the command trait → secantus_storage::extract_backup_archive_ex, with the target-emptiness / WiredTiger-metadata guards and abs-path + file-count reply). Regressions: tests/test_rust_server_smoke.py::test_secantus_admin_restore_archive_roundtrips and ::test_restore_archive_rejects_nonempty_target.

Rust server: secantusAdmin.pruneOplog and pruneTtl maintenance commands

The Rust server now implements two of the proprietary secantusAdmin.* maintenance commands the Python server already had: pruneOplog (force an immediate oplog-retention sweep) and pruneTtl (run TTL pruning across every collection now). Both return {pruned: <count>, ok: 1}, mirroring the Python handlers. The underlying storage already prunes on its own cadence; these wire commands let an operator — or the admin console’s Maintenance page — drive a deterministic pass on demand. This closes the first slice of the Rust admin-command parity gap (issue #163); the admin UI’s capability probe will now enable the “Prune oplog / TTL” buttons against a Rust target.

Remaining #163 gaps (tracked, not in this slice): secantusAdmin.restoreArchive and the standard admin commands grantRolesToUser / revokeRolesFromUser / killOp, plus fleshing out the getLog / profile stubs.

Added

• Rust server: secantusAdmin.pruneOplog / secantusAdmin.pruneTtl wire commands (Storage::prune_oplog / prune_ttl_all on the command Storage trait, forwarded by the WT adapter to the real storage engine). Regression: tests/test_rust_server_smoke.py::test_secantus_admin_prune_commands.

$derivative and $integral gain a time unit

The $setWindowFields rate operators $derivative and $integral now accept a time unit. Over a date-valued sortBy, $derivative: {input: "$v", unit: "hour"} reports the change in v per hour — the x-axis is the date’s epoch milliseconds scaled into the requested unit — and $integral computes the trapezoidal area with the same unit-scaled x-axis. This is how you express a rate of change in meaningful units (per second, per hour, per day) instead of the raw millisecond spacing between timestamps.

As with time-unit range windows, the unit requires a date sortBy (a numeric sort with a unit is rejected) and the fixed-duration units week/day/hour/minute/second/millisecond are supported; variable-length month/quarter/year defer. The feature ships on both servers, pinned by the $setWindowFields parity suite — the millisecond-to-unit scaling runs in IEEE double on both sides so the results match bit-for-bit. This completes the $setWindowFields time-unit surface.

Added

• $setWindowFields $derivative / $integral accept a fixed-duration time unit over a date sortBy, scaling the x-axis into that unit so the rate / area is expressed per unit. unit requires a date sortBy; variable-length units defer.

$setWindowFields range windows over a date sortBy

A $setWindowFields value-range window can now span a time interval. Give the window a unit — week, day, hour, minute, second, or millisecond — and the numeric range: [lower, upper] bounds are measured in that unit against a date-valued sortBy field. A range: [-2, 0], unit: "day" window is the trailing three-day span ending at each row, so {$sum: "$v"} over it is a rolling 3-day total regardless of how the dates are spaced.

The rule mongod enforces holds both ways: a unit requires a date sortBy (a numeric sort with a unit is rejected), and a date sortBy in a range window requires a unit (there is no implicit millisecond arithmetic on dates). Variable-length units — month, quarter, year — are still rejected, since their span depends on the calendar position. The feature ships on both the Python and Rust servers, pinned by the $setWindowFields parity suite; the date x-axis is carried as epoch milliseconds so both engines compute identical window bounds.

Added

• $setWindowFields range windows accept a fixed-duration time unit (week/day/hour/minute/second/millisecond) over a date sortBy, offsetting the bounds in that unit against the date’s epoch millis. unit and a date sortBy are mutually required; variable-length month/quarter/year defer with an error.

$jsonSchema gains patternProperties and dependencies

The $jsonSchema query operator now understands patternProperties and dependencies. patternProperties applies a sub-schema to every field whose name matches a regular expression — the way to say “every s_* field must be a string” without listing them — and it also tells additionalProperties: false which keys are legitimately covered, so pattern-matched fields are no longer flagged as unexpected. dependencies expresses conditional structure: when a trigger field is present, either a list of other fields must also be present (property dependency) or the whole document must satisfy a sub-schema (schema dependency) — e.g. “if card is set, billing is required.”

Both ship on the Python and Rust servers, pinned by the query parity suite; patternProperties reuses the shared regex engine.

Added

• $jsonSchema patternProperties (regex-keyed sub-schemas, also honoured by additionalProperties) and dependencies (property-list and schema forms).

$setWindowFields completes its time-series window operators with $derivative and $integral

$setWindowFields now supports $derivative and $integral, the last two of MongoDB’s time-series window operators. Over the window, $derivative reports the rate of change — the slope between the first and last points, (yₙ − y₀) / (xₙ − x₀), with the sortBy value as x and input as y — and $integral reports the area under the curve by the trapezoidal rule. Together with $shift, $expMovingAvg, $locf, and $linearFill, that rounds out the full set of time-series window functions: rates, running smoothing, lag/lead, and gap-filling all now run in-process.

Both operate over any window (documents or range), require a single ascending numeric sortBy as the x-axis, and run the same IEEE-double arithmetic on the Python and Rust servers (pinned bit-for-bit by the aggregation parity suite). A $derivative window with fewer than two points is null. A time unit (for a date x-axis) is not yet modelled and raises a clear error.

Added

• $setWindowFields $derivative and $integral window operators ({input}) — slope / trapezoidal area over the sortBy x-axis, over any window.

$setWindowFields gains $locf and $linearFill

$setWindowFields now supports the two gap-filling window operators, $locf and $linearFill. {$locf: <expr>} (“last observation carried forward”) replaces a null with the most recent non-null value seen in sort order — the standard way to hold a reading steady until the next sample. {$linearFill: <expr>} instead draws a straight line between the surrounding non-null anchors and reads off the missing values along it, using the sortBy value as the x-axis. Leading nulls (for $locf) and leading/trailing nulls (for $linearFill, which has nothing to interpolate between) stay null.

Both run on the Python and Rust servers alike, pinned by the aggregation parity suite (the interpolation is IEEE-double, so the two agree bit-for-bit). With these plus $shift and $expMovingAvg, only $derivative and $integral remain of the time-series window operators.

Added

• $setWindowFields $locf and $linearFill gap-fill window operators (<expr>) — prefix/partition-based, require a sortBy ($linearFill a single ascending numeric one).

Rust server: an interior-NUL db/collection/index name no longer panics the connection

A well-formed BSON command whose database, collection, or index name carried an embedded NUL byte (BSON strings are length-prefixed and may legally contain one) reached the Rust server’s WiredTiger key encoder (secantus-wt’s cstr, CString::new(..).expect(..)) and panicked. Because the storage layer serialises WiredTiger operations under a std::sync::Mutex, that panic unwound while the lock was held and poisoned it for every connection — turning a single crafted command into a whole-server denial of service, and dropping the offending socket with no {ok: 0, ...} reply (unlike the Python server, which catches every error in dispatch).

The Rust server now rejects an interior-NUL database / collection / index name during command validation — before it reaches storage — with the same InvalidNamespace error mongod returns, so the connection survives with a clean wire reply. As defense-in-depth, the per-connection dispatch call is now wrapped in catch_unwind: any future unguarded panic in a handler produces a wire-level InternalError reply instead of a silent disconnect. This is the Rust-side analogue of the earlier Python-server OP_QUERY hardening. Found by the nightly security review (issue #139).

Security

• Rust server: interior-NUL db / coll / index names are rejected with InvalidNamespace before reaching the WiredTiger key encoder (which would otherwise panic and poison the shared storage mutex — a whole-server DoS). Per-connection dispatch is wrapped in catch_unwind so any residual panic surfaces as an InternalError wire reply rather than a dropped socket. Regressions in crates/secantus-commands unit tests.

admin.system.users no longer leaks SCRAM credentials via find/count/aggregate

A query against admin.system.users — reachable through ordinary find / count / aggregate / explain / distinct / mapReduce, which need only the standard collection-read action — used to return each user record including its SCRAM credentials blob (storedKey, serverKey, salt, iteration count). That’s the sensitive artifact: even without the plaintext password it enables offline dictionary/brute attacks (salt + iterations are the /etc/shadow equivalent) and, via serverKey, lets an attacker stand up a rogue server that completes the SASL server side. A principal holding only read / readAnyDatabase on admin — a routine monitoring/backup grant — could read every user’s credential material, which is exactly what usersInfo intentionally gates behind A_VIEW_USER + showCredentials.

The generic read path now strips credentials unconditionally, before the filter runs, so it is never returned and can’t be used as a match-oracle either. Credentials remain reachable only through usersInfo with showCredentials and the A_VIEW_USER privilege — the one intentionally-gated path, unchanged. Found by the nightly security review (issue #167).

Security

• Storage._find_system_users / _count_system_users: the SCRAM credentials blob is stripped from the generic admin.system.users read path (find / count / aggregate / …) before filtering, so a low-privilege reader can no longer harvest credential material. usersInfo (gated by A_VIEW_USER + showCredentials) is unchanged. Regressions in tests/test_system_users_view.py.

$setWindowFields gains $expMovingAvg

$setWindowFields now supports $expMovingAvg, the exponential moving average — the second time-series window operator (after $shift). {$expMovingAvg: {input: <expr>, N: <n>}} smooths a series over the sorted partition, weighting recent rows more heavily: each output is input·α + previous·(1−α), with α = 2/(N+1). An explicit {alpha: <0…1>} may be given instead of N. It’s the standard smoothing for noisy time-series — a trend line that reacts faster than a flat rolling average.

The recurrence runs in IEEE double on both the Python and Rust servers, so the two agree bit-for-bit (pinned by the aggregation parity suite). The remaining time-series operators ($derivative, $integral, $linearFill, $locf) still raise a clear error.

Added

• $setWindowFields $expMovingAvg window operator ({input, N | alpha}) — prefix-accumulated per partition, requires a sortBy.

$jsonSchema gains logical combinators and additionalProperties

The $jsonSchema query operator now understands the JSON-Schema logical combinators — allOf, anyOf, oneOf, and not — plus additionalProperties. allOf/anyOf/oneOf compose sub-schemas (all / at-least-one / exactly-one must hold), not inverts one, and additionalProperties controls fields not named in properties: false forbids them outright, while a sub-schema validates each one. Together they cover the structural side of collection validators that go beyond flat field constraints — “exactly one of these shapes”, “none of that”, “no unexpected fields”.

These keywords were previously absent from both servers; they now ship on the Python and Rust servers alike, pinned by the query parity suite. (patternProperties is still not modelled.)

Added

• $jsonSchema allOf / anyOf / oneOf / not combinators and additionalProperties (true / false / sub-schema).

$setWindowFields gains the $shift window operator

$setWindowFields now supports $shift — the first of MongoDB’s time-series window operators. $shift reaches across the sorted partition to read a value from a neighbouring row: {$shift: {output: <expr>, by: <n>, default: <expr>}} evaluates output on the row n positions away (negative looks back, positive looks ahead), falling to default (or null) when that position is past the partition edge. It’s the idiomatic way to compute a delta from the previous row, peek at the next value, or lag/lead a series — without a self-join. Shifts never cross a partition boundary.

The same semantics ship in both the Python and Rust servers, pinned by the aggregation parity suite. The remaining time-series operators ($derivative, $integral, $expMovingAvg, …) still raise a clear error.

Added

• $setWindowFields $shift window operator ({output, by, default?}) — position-based, per-partition, requires a sortBy.

$setWindowFields learns value-based (range) windows

The $setWindowFields aggregation stage now supports value-based windows — window: {range: [lower, upper]} — alongside the position-based documents windows it already had. Where a documents window counts rows relative to the current one, a range window is defined by the sortBy value: it includes every row whose value falls in [current + lower, current + upper]. That’s the natural way to express “sum everything within 10 units of this point” or a gap-aware running total, and it’s what analytics pipelines reach for. Bounds may be a number, "current" (this row’s value), or "unbounded".

The window resolves against a single ascending numeric sortBy field; a range window with a time unit, or over a descending / multi-field / non-numeric sort, still raises a clear error rather than guessing. The same semantics ship in both the Python and Rust servers, pinned together by the aggregation parity suite.

Added

• $setWindowFields value-based windows (window: {range: [lo, hi]}) over a single ascending numeric sortBy, with "unbounded" / "current" / numeric bounds. Time-unit ranges and non-ascending / multi-field / non-numeric sorts remain deferred with a clear AggregateError.

Smaller on-disk footprint: WiredTiger log pre-allocation disabled

Each on-disk SecantusDB instance used to reserve ~30 MB of WiredTiger log files regardless of how little data it held — the active WiredTigerLog plus two 10 MB WiredTigerPreplog files WT pre-allocates ahead of the active log. That pre-allocation is a write-latency optimisation for long-running, high-throughput servers; SecantusDB’s instances are small, ephemeral, in-process test databases, so it bought nothing and cost disk — acutely on CI, where a full ~2000-test on-disk run retained thousands of instances and exhausted the Windows runner’s disk (No space left on device → WT_PANIC). Disabling log=(prealloc=false) drops each instance’s log footprint from ~30 MB to ~10 MB with no durability change (recovery replays the same log records; WT just allocates each segment on demand). file_max stays 10 MB so a near-maxBsonObjectSize document still fits in one segment.

Changed

• Storage: on-disk WiredTiger opens with log=(...,prealloc=false) — ~3x smaller per-instance log footprint, no durability impact.

A malformed OP_QUERY frame returns BadValue instead of dropping the connection

A legacy OP_QUERY frame whose fullCollectionName carried no NUL terminator made bytes.index(b"\x00", ...) raise an uncaught ValueError that escaped the wire layer’s (InvalidBSON, _BodyBoundsError) handler, killed the connection handler without sending a reply, and logged a Python traceback. The same gap existed for several siblings on the same attacker-controlled path: a non-UTF-8 collection name (UnicodeDecodeError), a frame truncated before the skip/return/query fields (struct.error), and a negative/oversized declared query-doc length (the OP_MSG path already guarded this with _check_doc_len, but OP_QUERY did not). The OP_MSG kind-1 section-identifier .index() / .decode() had the identical unterminated-cstring gap.

_parse_op_query is now hardened the same way _parse_op_msg already was — every read on the network buffer raises _BodyBoundsError / struct.error, which read_message translates into a BadValue (2) wire reply while keeping the connection alive (matching mongod). read_message also now catches struct.error as a backstop so no malformed frame can escape as an uncaught exception. Found by the nightly security review (issue #116).

Security

• wire._parse_op_query / _parse_op_msg: malformed OP_QUERY/OP_MSG frames (missing cstring NUL, invalid UTF-8, truncation, bad BSON length) now yield a BadValue reply and a surviving connection instead of a dropped socket + logged traceback. Regression: tests/test_wire_malformed.py (5 new OP_QUERY cases).

Admin UI security hardening (two CVEs + a stored-XSS fix)

Three findings from the nightly security review, all confined to the optional [admin] extra (the loopback FastAPI console):

• CVE-2026-48710 “BadHost” (issue #114): the admin token middleware gated its /healthz + /static/ bypass allowlist on request.url.path, which pre-starlette 1.0.1 is rebuilt from an unvalidated Host header — a request with Host: x/healthz?t= could shift request.url.path to a bypass prefix and reach protected admin endpoints unauthenticated. Bumped the starlette floor to >=1.0.1, and the middleware now reads the ASGI scope["path"] (immune to Host spoofing) as defence-in-depth.

• CVE-2026-53539 (issue #113): python-multipart <0.0.30 has a quadratic-CPU DoS in its urlencoded-form parser. Bumped the floor to >=0.0.30.

• Stored XSS in the geo viewer (issue #115): the geo page injected sampled document data into an inline <script> via json_util.dumps — a document whose string _id contained </script><script>… closed the block and injected arbitrary JS (with access to pywebview’s js_api). Feature JSON is now escaped for the script context (</>/&/U+2028/ U+2029 → \uXXXX) so a payload can never break out of the block.

Security

• admin/middleware.py: token-bypass check reads scope["path"], not the Host-derived request.url.path.

• admin/routers/extras.py: _json_for_script escapes geo feature JSON for safe inline-<script> embedding.

• pyproject.toml [admin]: starlette>=1.0.1, python-multipart>=0.0.30.

• Regressions: tests/test_admin_security.py.

Capped-collection eviction survives a backup taken mid-stream

A capped collection restored from a backupArchive could evict the wrong document on its next insert — dropping the freshly-inserted row instead of the oldest one. The cause was a stale recovery hint: the oplog-meta row that records the next insertion sequence is only refreshed on hello, prune_oplog, and close (the per-write path stopped re-persisting it because it WT-rollbacks under concurrent writers), so a checkpoint taken between two refreshes captures a sequence counter that lags the actual data. On reopen the server trusted that stale value, re-minting an already-used natural-order sequence; the collision overwrote a live document’s entry in the insertion-order index and corrupted capped FIFO eviction. The symptom was load- and timing-dependent — it surfaced only when the driver’s background topology hello happened to fall before the last insert rather than after it.

Recovery now treats the persisted next_seq / next_nat_seq as a hint that can only be corrected upward: it clamps each counter to what the oplog and natural-order tables actually contain, so a lagging meta row can never lower the sequence and re-mint a used value. The oplog maximum is read with a single prev() (the table is keyed on the bare sequence), keeping reopen cheap.

Fixed

• Capped-collection FIFO eviction after restoring a backupArchive that was taken between oplog-meta refreshes — the recovered insertion-sequence counter is now clamped up to the natural-order table’s maximum rather than trusting a stale persisted value, eliminating the sequence collision that dropped a just-inserted document. The same clamp guards the oplog next_seq against re-minting a used sequence (which would silently overwrite an oplog row).

Storage close no longer swallows durability errors; admin history/backup never persist credentials

Two security findings from the nightly review are closed. Storage.close() used to wrap its final teardown — the last oplog-meta persist, the shutdown checkpoint, and every WiredTiger session/connection close — in bare contextlib.suppress(Exception), discarding a checkpoint or connection-close failure with no trace at all. In a database that is a durability signal, not noise: the embedder had no way to know the last on-disk image might be incomplete. The teardown now logs every caught failure via log.exception(...) (matching the TTL-sweep and noop-heartbeat loops) while still completing idempotently.

The admin console’s query-history store and backup helpers no longer let a credentialed connection string reach disk or the UI in plaintext. HistoryStore scrubs the URI to its password-free display_uri() form at the store boundary before it becomes a SQLite lookup key, so ~/.secantus/admin.db can never hold a mongodb://user:pass@host string (and a caller passing the raw URI can’t reintroduce the leak). The mongodump/mongorestore helpers still hand the live credential to the subprocess — they need it to authenticate — but now redact the password from any captured stdout/stderr, closing the path where a tool that echoes the connection string on error surfaces the secret in the rendered backup result.

Fixed

• Storage.close() teardown errors (oplog-meta persist, shutdown checkpoint, WT session/connection close) are logged instead of silently suppressed (security issue #138).

• Admin HistoryStore persists the scrubbed display_uri() form instead of the raw credentialed mongo_uri; mongodump/mongorestore captured output has the password redacted (security issue #140).

Admin console detects the target server and gates features it can’t do

The admin UI is a plain pymongo client, so it can point at any of the three MongoDB-wire servers: the SecantusDB Python server, the SecantusDB Rust server, or a real mongod. They differ in which commands they implement — most visibly the four proprietary secantusAdmin.* backup/maintenance commands (no mongod has them) and a handful of standard admin commands the Rust server hasn’t ported yet. Until now the console advertised every button regardless of target, so clicking “native checkpoint backup” against a mongod, or “prune oplog” against the Rust server, returned a bare CommandNotFound.

The console now probes the target once at connect (and on every target swap) via buildInfo + serverStatus, classifies it as Python / Rust / MongoDB from the server’s own self-identification (serverStatus.secantus.server and buildInfo.secantusVersion), and derives a capability set the templates consult. A detected-server pill appears next to the target badge, and features the target can’t honour are disabled with an explanatory tooltip instead of failing on click: native backup archive / restore and manual oplog/TTL prune (all secantusAdmin.*, SecantusDB-only), plus role grant/revoke and connection-kill (killOp) where the Rust server hasn’t ported them. An unreachable or not-yet-probed target stays fully permissive, so a transiently-down server never hides a working button.

Added

• secantus.admin.capabilities: server-capability probe + classifier (classify / probe / ServerCapabilities / UNKNOWN), wired into the app lifespan startup and swap_target, exposed to templates as request.app.state.capabilities.

• Admin templates gate secantusAdmin.* backup/maintenance buttons, role grant/revoke, and connection-kill to the detected server’s capabilities, and show a server-type badge.

Atlas Search index commands are rejected with an “Atlas” error

Atlas Search index management — the createSearchIndexes, updateSearchIndex, and dropSearchIndex commands plus the $listSearchIndexes aggregation stage (and its $search / $searchMeta / $vectorSearch siblings) — is an Atlas-only feature. A real non-Atlas mongod registers these but fails them at execution with a message naming Atlas. SecantusDB now does the same (CommandNotSupported, message mentioning Atlas) instead of returning CommandNotFound / an “unrecognized pipeline stage” error. Closes the mongo-c-driver /index-management/{list,drop,update,create}SearchIndex tests, which assert the error contains “Atlas”.

Added

• createSearchIndexes / updateSearchIndex / dropSearchIndex command handlers and $listSearchIndexes / $search / $searchMeta / $vectorSearch aggregation-stage rejection, all returning the shared aggregate.SEARCH_INDEX_ATLAS_MSG (mirrors mongod’s not-on-Atlas error).

Dropping a collection under a tailable cursor reports “collection dropped”

When a capped collection is dropped while a tailable cursor is open on it, the next getMore now fails with QueryPlanKilled (175) and a “collection dropped” message — exactly what mongod surfaces to a tailing client. Previously the cursor was simply removed, so the follow-up getMore returned a bare CursorNotFound (43). Regular (non-tailable) cursors are unchanged: dropping their collection still yields CursorNotFound, which the strict wire gauges rely on. Closes mongo-php-driver’s cursor-tailable_error-001, which asserts the dropped-collection error mentions “collection dropped”.

Fixed

• CursorRegistry.kill_namespace tombstones tailable cursors (sets a dropped flag, keeps the entry) instead of deleting them; non-tailable cursors are still deleted. _get_more returns the new QueryPlanKilled “collection dropped” reply for a tombstoned tailable cursor, and _drop tombstones before the storage drop so a parked awaitData getMore (woken by the drop’s oplog write) observes the flag.

A tailable cursor’s filter now applies to docs inserted after the find

A tailable + awaitData cursor on a capped collection re-polls the collection for documents inserted after the find. That poll was returning every new row, ignoring the cursor’s query filter — so a tailable cursor watching for {a: 1} would surface unrelated inserts (and even pre-existing non-matching docs) the moment it ran a getMore. The producer now re-applies the find filter (with the same let vars and collation) to each scanned row, exactly as the oplog-tailing variant already did. The watermark still advances past every scanned row, matched or not, so non-matching docs aren’t re-examined on the next poll. Closes the mongo-c-driver gauge’s /Collection/tailable/timeout/single.

Fixed

• commands._find_tailable: the capped-collection tailable producer filters follow-up inserts through query.matches instead of returning them unconditionally.

A malformed $and / $or / $nor is a clean parse error, not a crash

$and / $or / $nor require a non-empty array of sub-documents. A query passing a non-array ({$or: true}), an empty array, or a non-document element used to crash the query engine — for c in condition raised a Python TypeError that escaped the parse-error handling and surfaced over the wire as a generic InternalError (1) with the traceback logged server-side. It now matches mongod: a BadValue (2) parse error (“$or must be an array” / “must be a nonempty array” / “entries need to be full objects”). Surfaced while triaging the mongo-c-driver gauge’s malformed-input command-monitoring tests.

Fixed

• query._match_clause: $and / $or / $nor validate their argument is a non-empty list of documents and raise QueryError (→ BadValue 2) for any malformed shape, instead of letting a TypeError leak out as InternalError.

An unrecognised index-key string is rejected as an unknown plugin

A string value in an index key names a special index type (“plugin”) — 2d, 2dsphere, text, hashed. SecantusDB already accepted the geo plugins and rejected text/hashed as out-of-scope, but it let any other string through: createIndex({abc: "hallo thar"}) silently created a broken index instead of erroring. It now matches mongod and rejects an unrecognised plugin name with CannotCreateIndex (67) “Unknown index plugin ‘’”.

This closes the mongo-c-driver gauge’s /Collection/index_w_write_concern test, which (after the 0.5.4b13 write-concern fix) was failing on its invalid-index assertion — it creates {abc: "hallo thar"} and expects the server to reject it. The test name is misleading; the failure had nothing to do with write concern.

Fixed

• storage.create_index: a string index-key value that isn’t a recognised plugin (2d / 2dsphere / 2dsphere_bucket / geoHaystack) is rejected with CannotCreateIndex (67) “Unknown index plugin ‘’”, alongside the existing text / hashed out-of-scope rejection.

A numeric write-concern w above 50 is now a parse error

A writeConcern with a numeric w greater than 50 (or negative) is now rejected at parse time with FailedToParse (9) and the message “w has to be a non-negative number and not greater than 50” — matching mongod, which caps w at the maximum number of voting replica-set members (50). Previously SecantusDB treated any w above 1 the same way (a CannotSatisfyWriteConcern writeConcernError, code 100, attached to a successful reply). That’s only correct for 1 < w <= 50 — satisfiable on a multi-node deployment but not on our single node. Above 50 the value is simply invalid, and mongod errors the whole command.

This closes the mongo-c-driver gauge’s last cluster of failures — /Collection/{drop,rename,index} and /Database/drop, which each run a DDL op with w: 99 and assert the assert_wc_oob_error shape (code 9, the message above) for a server advertising version >= 4.3.3 (we advertise 7.0). The “state-ordering” label these carried in the backlog was a misdiagnosis: the failure is deterministic, not dependent on test order.

Fixed

• commands._validate_write_concern: a numeric writeConcern.w outside [0, 50] is rejected with FailedToParse (9) before the command runs, instead of falling through to the satisfiability check (code 100). _drop_database and _rename_collection now run this validation too (they previously skipped it, relying only on the dispatch-level _unsatisfiable_wc_error).

The $currentOp aggregation stage now surfaces the connecting driver’s handshake metadata — the full clientMetadata document (driver name/version, OS, application name) plus a top-level appName — on its self-row, matching real mongod. Previously only the currentOp command echoed this back; the aggregation form (db.aggregate([{$currentOp: {}}])) returned a bare stub, so a client couldn’t find its own operation by appName or read back the metadata it sent on connect.

This was the last real divergence the mongo-cxx-driver gauge’s “integration tests for client metadata handshake feature” exercised — it connects with ?appName=xyz, scans $currentOp for the matching op, and verifies its clientMetadata.{application,driver,os}. With this and the 0.5.4b11 resume-token fix, the cxx gauge’s remaining real failures are closed.

Fixed

• aggregate._stage_current_op: the $currentOp self-row now carries clientMetadata (the connection’s hello.client subdoc) and a top-level appName lifted from application.name, threaded in via a new PipelineContext.client_metadata field that the aggregate command handler populates from the connection registry. The currentOp command already did this; the aggregation stage now matches.

Change-stream resume tokens now advance per event, even at batchSize 1

A change stream’s postBatchResumeToken now tracks the resume token of the last event actually returned in each batch, not the last event the server happened to prefetch. The producer reads up to 200 oplog rows ahead while the cursor hands them back batchSize at a time, so with a small batch size the token reported on each getMore was stale — three single-event reads all carried the same token. Drivers that resume off the per-batch token (the whole point of postBatchResumeToken) would resume from the wrong place. Alongside it, an empty getMore over a quiet collection no longer re-mints the token with a fresh cluster time when the oplog tail hasn’t actually moved, so an exhausted stream reports the same resume token as its last event rather than drifting.

This was surfaced by the mongo-cxx-driver gauge’s spec prose test “ChangeStream must continuously track the last seen resumeToken” (batchSize=1, read three events, assert each token differs, then assert the post-exhaustion token equals the last event’s). The fix is server-side and driver-agnostic — every change- stream driver benefits.

Fixed

• commands._change_stream_cursor_doc: postBatchResumeToken is now the _id (resume token) of the last event in the returned batch, not the producer’s prefetch-tail last_token — so per-batch tokens advance correctly under any batchSize.

• commands change-stream producer: an empty getMore only advances / re-mints the resume token when the oplog tail has genuinely moved past the cursor’s position, preserving the go resume_token_updated_on_empty_batch advance while fixing the mongocxx no-change-equals-last-token case. (On a truly quiet collection the token now advances via the oplog’s periodic noop heartbeats, mirroring mongod’s periodicNoopIntervalSecs — not a per-getMore clock tick.)

• The change-stream batch builder (_change_stream_cursor_doc) is shared with capped-collection tailable cursors, whose documents carry plain _id values and no resume token. The new postBatchResumeToken = last-event-_id logic is gated to change-stream cursors only, so a capped tailable getMore no longer emits a non-document PBRT (strict drivers — the Java driver — rejected an int32 there).

Two more conformance gauges: the Kotlin driver and pymongo async

SecantusDB now also measures itself against the official MongoDB Kotlin driver and against pymongo’s native async (AsyncMongoClient) suite — bringing the gauge count to thirteen. Both reuse infrastructure already in the tree rather than vendoring new submodules: the Kotlin driver ships inside the mongo-java-driver monorepo (driver-kotlin-sync), so its gauge runs the :driver-kotlin-sync:integrationTest Gradle task against an embedded SecantusDB daemon over the same JDK/Gradle toolchain the Java gauge already needs; and the async gauge points pymongo’s test/asynchronous/ suite at the same embedded-server plugin the sync gauge uses, run under pytest-asyncio with asyncio_mode=auto. The async gauge is the more interesting of the two — it exercises the async/await wire path that replaced Motor, which drives cursors and change-stream getMore polling through a different event-loop code path than the synchronous client, catching divergences a sync-only gauge can’t see.

Run them with invoke validate-pymongo-async and invoke validate-kotlin. Both join the weekly validate.yml matrix and invoke validate-all. Neither touches the shipped secantus package — the gauge directories are dev-only, excluded from the wheel and sdist like every other gauge.

Bringing the Kotlin gauge up also surfaced and fixed a latent break in the shared gauge plumbing: every daemon-subprocess gauge (go / node / java / ruby / rust / c / cxx / dotnet) passes --log-level WARNING, but gauge_common. spawn_daemon learns the daemon’s kernel-assigned port by grepping its listening on <host>:<port> line — which the Python server logs at INFO, so WARNING suppressed it and the spawn waited the full timeout (and, with a blocking read, could hang for hours — one scheduled CI run was cancelled at 6h). The spawn now forces the Python daemon to INFO (per-request logging is at DEBUG, so this adds only the one readiness line, no noise) and reads the daemon’s output under a hard deadline so a missing line times out instead of hanging. This is why those gauges were red in the weekly run; they go green again with this fix.

Added

• pymongo_async_validation/ gauge package (include_paths / generate_report) and the invoke validate-pymongo-async task — pymongo’s native AsyncMongoClient suite against an embedded SecantusDB, reusing pymongo_validation.plugin and the vendor/pymongo-tests submodule. Adds a pytest-asyncio dev dependency.

• kotlin_validation/ gauge package (include_modules / runner / generate_report / init.gradle.kts) and the invoke validate-kotlin task — the official Kotlin driver’s :driver-kotlin-sync:integrationTest suite against a standalone SecantusDB daemon, sharing the Java gauge’s JVM toolchain and vendor/mongo-java-driver submodule.

• Both gauges wired into invoke validate-all / validate-all-servers and the CI validate.yml matrix.

Fixed

• gauge_common.spawn_daemon: the Python daemon is now forced to --log-level INFO so its listening on … readiness line (logged at INFO) is visible even though gauges pass --log-level WARNING, and the readiness read is bounded by the spawn deadline instead of a blocking readline(). Unbreaks the daemon-subprocess gauges (go / node / java / ruby / rust / c / cxx / dotnet), which were failing/hanging in CI because WARNING suppressed the line.

Closing the gaps the C, C++, and C# gauges opened

Adding the mongo-c-driver, mongo-cxx-driver, and mongo-csharp-driver gauges turned up a cluster of small conformance divergences in the Python server, and this release fixes the actionable ones. None were show-stoppers on their own — they’re the kind of edge that a permissive driver like pymongo glosses over but a strict C extension or a spec-faithful CRUD suite pins exactly — and together they tighten how faithfully SecantusDB answers the corners of the wire protocol.

The headline is document-validation error detail: a write that fails a collection validator now returns mongod’s full per-operator errInfo.details (operatorName / specifiedAs / reason / consideredValue / consideredType) instead of a bare placeholder, so a driver can tell you why a document was rejected. Alongside it: $out / $merge are now rejected unless they’re the final pipeline stage, they enforce the destination collection’s validator (honouring bypassDocumentValidation), a change stream opened with an invalid $match errors immediately rather than on the first batch, batchSize accepts any BSON number type, over-long database names are rejected, dropping or renaming a collection invalidates its open cursors, and collMod can stage an index prepareUnique and convert it to unique — reporting the duplicate _id groups as violations when the conversion can’t proceed.

Added

• collMod {index: {prepareUnique: true}} arms an existing index so new uniqueness-violating writes are rejected (11000) while pre-existing duplicates remain, and collMod {index: {unique: true}} converts it — refusing with CannotConvertIndexToUnique (359) plus a violations: [{ids: [...]}] array when duplicates exist (storage.set_index_options / find_index_duplicates).

Fixed

• Document validation now synthesises mongod’s per-operator errInfo.details for query-expression validators (commands._validation_failure_details), used by both the insert path and findAndModify upsert simulation.

• $out / $merge are rejected with Location40601 (40601) unless they are the final pipeline stage, and they enforce the destination collection’s validator unless bypassDocumentValidation is set (DocumentValidationFailure, 121).

• Change streams validate $match filter syntax at open time, so an unknown query operator errors at .begin() (aggregate) rather than the first getMore.

• find / aggregate accept a batchSize encoded as any BSON number, including Decimal128.

• Commands targeting a database whose name exceeds 63 bytes are rejected with InvalidNamespace (73).

• Dropping or renaming a collection now kills its open cursors, so a later getMore fails with CursorNotFound (43) instead of serving stale rows.

createIndexes now rejects conflicting index definitions

createIndexes previously accepted a re-creation that collided with an existing index, silently treating it as a no-op. It now matches mongod: re-using an index name for a different key spec is rejected with IndexKeySpecsConflict (code 86), re-using a name with the same key but different options is rejected with IndexOptionsConflict (code 85), and an identical re-create returns note: "all indexes already exist" so drivers report it as a no-op rather than a fresh build. This was surfaced by the mongo-cxx-driver gauge (create_index tests/fails, index_view/fails for same name, fails for same keys and options), and applies to every driver’s createIndex / index-view API.

Fixed

• createIndexes: same-name-different-key now errors IndexKeySpecsConflict (86); identical re-creates now carry note: "all indexes already exist" (storage.create_index / commands._create_indexes). Same-name-different- options continues to error IndexOptionsConflict (85).

An eleventh conformance gauge: the MongoDB C# / .NET driver

SecantusDB now also measures itself against the official MongoDB C# / .NET driver — the one the .NET, Unity, and Xamarin ecosystems build on — bringing the gauge count to eleven. The gauge runs the driver’s own xUnit suite via dotnet test against an embedded SecantusDB daemon, with MONGODB_URI pointed at it, scoped to the CRUD specification conformance tests (MongoDB.Driver.Tests.Specifications.crud). MongoDB.Driver.Tests as a whole is enormous and dominated by non-server unit tests and external-service suites (client-side encryption, Atlas Search, multi-node transactions), so the CRUD spec runner is the focused, bounded conformance slice — expandable to more spec families over time. The driver’s [RequireServer] attribute self-skips tests whose server-version or topology requirements a single node doesn’t meet.

Run it with invoke validate-dotnet (needs the .NET SDK and gpg — the driver’s encryption project verifies a downloaded libmongocrypt during build). It joins the weekly validate.yml matrix and the cross-driver summary.

Added

• dotnet_validation/ gauge package (runner / generate_report / include_paths) and the invoke validate-dotnet task; wired into invoke validate-all, the cross-driver summary (validation_summary), and CI.

Two new conformance gauges: the MongoDB C and C++ drivers

SecantusDB now measures itself against the official MongoDB C (libmongoc) and C++ (mongocxx) drivers — bringing the gauge count to ten. libmongoc is the lowest-level official client, the one the PHP, Ruby, and PyMongo C-extensions ultimately wrap; mongocxx is the modern C++ driver built on top of it. Both gauges build the driver’s own test suite from source and run it, unmodified, against an embedded SecantusDB daemon — test-libmongoc (curated CRUD / cursor / aggregation / command / GridFS / index suites over MONGOC_TEST_URI) for C, and the mongocxx test_driver Catch2 suite for C++. Strict native clients surface type- and wire-shape divergences that more permissive drivers accept silently.

Two wrinkles were worth solving. First, libmongoc’s test fixture probes the server with replSetGetStatus and aborts the whole run on an unexpected error, so SecantusDB now answers it like a standalone mongod — NoReplicationEnabled (code 76), “not running with –replSet” — which driver harnesses special-case as “standalone, skip the replica-set-only paths”. Second, mongocxx’s tests hard-wire mongodb://localhost:27017 with no environment override, so the C++ gauge binds its daemon on port 27017 and refuses to run if something else already holds it (it won’t gauge a foreign server).

Run them with invoke validate-c / invoke validate-cxx (both need cmake and a C/C++ toolchain; the first run builds the drivers, later runs reuse the cached builds). Both join the weekly validate.yml matrix and the cross-driver summary.

Added

• c_validation/ and cxx_validation/ gauge packages (runner / generate_report / include_paths each) and the invoke validate-c / invoke validate-cxx tasks; wired into invoke validate-all, the cross-driver summary (validation_summary), and CI.

• replSetGetStatus command: returns the standalone-mongod NoReplicationEnabled error so single-node-aware driver test fixtures skip replica-set-only behaviour instead of aborting.

The in-process Rust engine selection is fully removed

The transitional in-process engine selection — SECANTUS_ENGINE=python|rust|auto, the --engine CLI flag, and the SecantusDBServer(engine=...) constructor parameter — has been removed entirely. It was already inert since 0.5.3b3 (no operator module delegated to the _secantus_core extension any more), so this is a no-op for behaviour: the Python server has been pure-Python end to end for several releases. This change deletes the dead surface so there is one obvious way things work.

SecantusDB’s Rust implementation lives in the separate Rust server (and the standalone secantusdb binary), not in this package’s request path. The secantus-core wheel remains as the engine library and the parity-test oracle that pins each Rust engine byte-for-byte against its pure-Python counterpart.

Removed

• The --engine CLI flag and the engine= parameter on SecantusDBServer. Passing engine= now raises TypeError (the flag had no effect since 0.5.3b3).

• The secantus.engine module (the inert compatibility stub) and its tests/test_engine.py.

• The SECANTUS_ENGINE=rust full-suite CI step, which re-ran the test suite under a now-dead env var and was a source of flaky worker-crash failures.

Point-in-time recovery: restore the database as it was at any moment in its oplog

SecantusDB already kept a mongod-shaped oplog and could take consistent WiredTiger backup archives; this release joins the two into real point-in-time recovery. Given a backup (or a stopped server’s data directory), you can now rebuild a fresh database as it was at any target timestamp by replaying the oplog forward — documents, in-place updates, deletes, collection options (capped / size / max / validator / viewOn / …), and index / collMod / rename DDL are all reconstructed through the ordinary write paths, so the result is indistinguishable from the live database at that instant.

Recovery is offline, matching real mongod: it writes a fresh data directory you then start a new server on. Drive it from the CLI (secantusd-py restore --source <archive|dir> --target-dir <dir> [--to-time <ISO> | --to-timestamp <secs,ord>]) or the secantusAdmin.restoreToTimestamp admin command. A multi-document transaction is always replayed all-or-nothing — its statements share one commit timestamp, so a recovery point never lands mid-transaction. Out of the box the recovery window is the oplog retention window (tune --oplog-retention-seconds / --oplog-max-entries for the horizon you need). To reach further back, turn on oplog archiving (--oplog-archive-dir) and take periodic base snapshots: SecantusDB then keeps the dropped oplog on disk and recovery stitches the newest snapshot before your target together with the archived oplog, so any moment in the archived history is reachable without keeping the whole oplog live. See Backup & point-in-time recovery.

Added

• secantus.diff.apply_update_description — applies a $v: 2 updateDescription back to a document (the inverse of compute_update_description); the keystone of oplog replay.

• secantus.oplog_replay — replay() / restore_to_timestamp() / restore_archive_to_timestamp(): replays an oplog source into a fresh store, stopping at a target ts / wall-clock time.

• Storage.replay_mode() — a context manager that suppresses oplog emission so replay drives the real write paths without regenerating the oplog.

• secantusd-py restore CLI subcommand and the secantusAdmin.restoreToTimestamp wire command.

• Backup archives now embed a pitr-manifest.json describing their recoverable oplog range (Storage._pitr_manifest).

• Collection options (capped / size / max / validator / viewOn / …) now ride the create oplog entry, so PITR replay reconstructs them on the restored collection (previously only documents and indexes were). Storage.create_collection gained an options= argument; the same options now also surface in the show_expanded_events change-stream create event’s operationDescription.

• --preserve-oplog (secantusd-py restore) / preserveOplog: true (secantusAdmin.restoreToTimestamp) carries the replayed oplog onto the restored directory verbatim, so a change stream on the restored server can resume from a token minted before the restore point. The default still starts a fresh oplog timeline, matching mongorestore. Backed by Storage.import_oplog_segment.

• PITR v2 — arbitrary-window recovery (secantus.pitr_archive). A server started with --oplog-archive-dir <dir> ([oplog] archive_dir) archives the oplog rows prune_oplog is about to drop into durable segment files first; secantusAdmin.archiveBaseSnapshot / Storage.archive_base_snapshot take base snapshots into the same directory. Recovery then accepts that archive directory as the restore source (CLI / wire auto-detect it): it picks the newest base snapshot at or before the target time and stitches the archived oplog forward onto it. This lifts the v1 genesis-intact restriction — a restore can reach a time before the live oplog floor, without keeping the entire oplog live. Base snapshots are taken on demand (no background scheduler, matching prune_ttl / prune_oplog).

[0.5.3b13] — 2026-06-16

find() with no sort now returns documents in insertion order

An unsorted find() now returns documents in insertion order, matching mongod’s natural (storage) order. Previously SecantusDB returned them in _id order — which coincides with insertion order for the default monotonic ObjectId _ids, but diverged whenever a collection mixed _id types or used non-monotonic _ids (an int 1, a string "foo", a sub-document — BSON sorts those very differently from the order you inserted them). Code and drivers that read back rows in the order they were written — a common, reasonable assumption that mongod honours — now see the same order here.

Internally this adds a small natural-order index (a monotonic insertion sequence → document map) that an unsorted scan and the $natural hint walk; the document store itself is unchanged, so every _id lookup, secondary index, and uniqueness check is untouched. Capped-collection eviction and equal-key sort tie-breaks also follow insertion order now. (Multi=false updateOne/deleteOne without a sort still pick the _id-order-first match rather than the insertion-first one — a smaller remaining divergence, tracked in the backlog.)

Fixed

• Unsorted find() (and the $natural hint) return documents in insertion order, matching mongod — including for collections with mixed-type or non-monotonic _ids. Clears the PHP BulkWrite::testInserts / bulkwrite-insert-004 conformance tests.

[0.5.3b12] — 2026-06-16

count honours a hint, and 2dsphere indexes report their version

Two driver-conformance fixes. count now respects a hint: pass an index name or key pattern and the count walks that index — so hinting a sparse index counts only the documents present in it. count({}, hint: "sparse_idx") returns the number of docs that have the indexed field, not the whole collection, exactly as mongod does. Previously count ignored the hint and always counted every document.

Separately, 2dsphere indexes now carry a 2dsphereIndexVersion in their listIndexes output (version 3, mongod’s format since 3.2), so drivers that introspect geo indexes — like the PHP library’s IndexInfo::is2dSphere() — read the field they expect.

Added

• 2dsphereIndexVersion (3) on 2dsphere indexes, surfaced via listIndexes.

Fixed

• count now honours hint (index name or key pattern), counting via the hinted index — including sparse-index semantics (missing-field docs excluded).

[0.5.3b11] — 2026-06-16

serverStatus reports a live open-cursor count

serverStatus now includes metrics.cursor.open.total — the number of cursors currently registered on the server. It rises by one when a batched query leaves a cursor open for getMore and returns to its baseline once the cursor is exhausted or killed (via killCursors, which drivers send when a cursor object is destroyed). Tools and drivers that watch cursor lifecycle — including the PHP extension’s cursor-destruct test — can now see cursors open and close.

The value is read live from the server’s CursorRegistry, so it reflects the true set of not-yet-exhausted, not-killed cursors at the moment of the call.

Added

• serverStatus.metrics.cursor.open.total (live open-cursor count from the CursorRegistry).

[0.5.3b10] — 2026-06-16

collMod can retune a TTL index’s expiry

collMod now handles its index modification form: pass {collMod: "<coll>", index: {keyPattern: {...}, expireAfterSeconds: N}} (or {index: {name: "<idx>", expireAfterSeconds: N}}) and SecantusDB rewrites the TTL index’s expiry in place, returning the expireAfterSeconds_old and expireAfterSeconds_new pair that mongod echoes. The new value takes effect immediately — prune_ttl reads the expiry from the same index options — so a retuned TTL window applies on the next prune.

Previously collMod accepted the command but ignored the index form, returning a bare {ok: 1} with neither the old nor new expiry. The PHP library’s ModifyCollection and Database::modifyCollection tests assert both values; both now pass.

Added

• collMod index form for TTL retuning: resolves the target index by keyPattern or name, updates expireAfterSeconds, and returns expireAfterSeconds_old / expireAfterSeconds_new. Backed by a new Storage.set_index_expiry.

[0.5.3b9] — 2026-06-16

Duplicate-key errors now read exactly like mongod

When a write hits a unique-index collision, the E11000 error message now matches mongod verbatim: E11000 duplicate key error collection: <db>.<coll> index: <indexName> dup key: { <field>: <value> }. Previously SecantusDB emitted a terser, non-standard form (… in index <name>: _id=<value>) that worked for permissive drivers but failed the type-strict ones that pin the message text — the PHP extension’s WriteError::getMessage() and WriteResult::getWriteErrors() assert the full string, down to the shell-formatted dup key fragment.

The structured fields drivers parse — code, index, keyPattern, keyValue — were already correct; this is purely the human-readable message catching up, across every path that can raise a duplicate key (batch insert, update/upsert, and unique-index builds). The fix clears the PHP extension’s writeError and writeResult suites.

Fixed

• Duplicate-key (E11000) error messages now use mongod’s exact wording, including the collection: <ns> index: <name> dup key: { … } shape with shell-formatted key values, consistently across all duplicate-key raise sites.

[0.5.3b8] — 2026-06-16

explain now speaks allPlansExecution, and aggregate’s inline explain flag works

explain gained the last two pieces the official MongoDB drivers probe for. At the most verbose level, verbosity: "allPlansExecution", the reply now carries an allPlansExecution array inside executionStats — empty, because a single-node query is always served by a single candidate plan with no rejected alternatives, which is exactly what real mongod reports when there’s no multi-planning to summarise. Drivers that assert the key is present (and absent at lower verbosities) now see the shape they expect.

The aggregate command also learned to honour its legacy inline explain: true flag — the form drivers send when you call explain() on an aggregation rather than wrapping it in the top-level explain command. SecantusDB previously ran the pipeline and returned data; it now returns the explain plan instead, and critically does not execute a trailing $out or $merge write stage under explain, matching mongod’s dry-run behaviour. Together these clear the PHP library’s entire ExplainFunctionalTest and aggregate-explain suite.

Fixed

• explain with verbosity: "allPlansExecution" now includes an executionStats.allPlansExecution array (empty for single-solution plans), on both find-style and aggregate explains.

• The aggregate command’s inline explain: true flag now returns the explain document (stages / queryPlanner) instead of running the pipeline, and suppresses $out / $merge writes under explain.

[0.5.3b7] — 2026-06-15

$exists: true rides a sparse index instead of scanning the collection

A query of the form {field: {$exists: true}} now uses a sparse single-field index on field when one exists, instead of falling back to a full collection scan. A sparse index holds an entry for exactly the documents where the field is present — missing-field documents are omitted, present-but-null and array values keep an entry — so the complete set of index entries is the $exists: true match set. The planner walks the whole index (no value bound), and explain reports IXSCAN accordingly. A non-sparse index still can’t serve $exists: true (it has an entry per document, including the absent ones), and $exists: false never uses a sparse index — both correctly stay on COLLSCAN. Results were always correct; this is the missing fast path.

Added

• {field: {$exists: true}} uses a sparse single-field index (IXSCAN) when one is present, via Storage._sparse_index_for_exists + _all_id_keys_for_index, mirrored in explain_plan. Non-sparse indexes and $exists: false stay on COLLSCAN.

Fixed

• The three pymongo DBRef-spec tests (test_dbref.py::TestDBRefSpec) are now deselected from the gauge. They are pure client-side BSON codec tests that never exercise SecantusDB; they pass under plain unittest but crash the gauge’s -n1 xdist worker because execnet can’t pickle the ObjectId in their subTest params (DumpError). Deselecting them keeps the gauge run clean and stops three spurious failures from being attributed to the server.

Fixed a shutdown race that could crash the server process

Stopping a SecantusDBServer now drains its in-flight per-connection threads before tearing down WiredTiger. Previously stop() joined only the accept thread and then closed the storage engine — so a connection handler still mid-WiredTiger-operation (e.g. a change-stream tailable getMore reading the oplog) had its WT connection freed underneath it: a use-after-free that surfaced as an intermittent native crash (the pytest-xdist worker death seen near the end of the full suite under churn). stop() now closes every connection socket to unblock reads, wakes any tailable getMore parked on the oplog condition variable, and waits for the active-connection count to reach zero before calling storage.close(). A 200-iteration stress that reliably tripped the use-after- close now runs clean.

Waking those parked reads is platform-specific, and the first cut got it wrong on both ends. On POSIX, shutdown(SHUT_RDWR) wakes a recv blocked in another thread while leaving the descriptor valid; calling close() from the stopping thread instead does not wake the parked recv and frees the fd number for immediate reuse, leaving the handler blocked forever on a recycled descriptor — so the drain barrier timed out. On Windows the opposite holds: shutdown does not interrupt an already-blocked recv, so closesocket is required. The wake is now shutdown-only on POSIX and shutdown-then-close on Windows. The drain barrier also re-runs the socket wake on every poll, not just once up front: the accept thread bumps the active-connection count and spawns the handler before the handler registers its socket, so a connection accepted in the instant before stop() could register after the initial sweep and never be woken — re-sweeping catches it within milliseconds.

Fixed

• SecantusDBServer.stop() drains in-flight connection threads before closing WiredTiger (via ConnectionRegistry.close_all + Storage.signal_shutdown + an active-connection drain barrier), eliminating a use-after-free / native crash on teardown under load.

• The stop-time socket wake is now platform-correct: shutdown-only on POSIX (closing the fd from another thread left handlers blocked on a recycled descriptor and timed out the drain), shutdown+close on Windows (where shutdown alone doesn’t interrupt a blocked recv). The drain barrier re-sweeps each poll so a connection that registers its socket just after stop() begins is still woken.

Tailable cursors over local.oplog.rs

A client can now tail the oplog the way replication does: local.oplog.rs accepts TAILABLE_AWAIT find cursors and streams oplog entries as they’re written. Two pieces landed for this — the synthetic oplog view is now reported as a capped collection by collection_is_capped (so a tailable cursor isn’t rejected), and a dedicated oplog tailable producer reads new entries by oplog seq (oplog documents have no _id, so the ordinary capped-collection tail path doesn’t apply). find().sort("$natural", ...) is honoured against the view — the oplog’s only meaningful order.

To match mongod — whose oplog is never empty (its first entry is the replica set’s “initiating set” noop) — a freshly-started server now seeds one bootstrap noop into the oplog, so a client can tail local.oplog.rs before any user write. The seed is an op: "n" entry (skipped by change-stream projection, so it never surfaces as a change event) and only fires on a truly fresh oplog. Closes the pymongo gauge’s test_cursor.test_to_list_tailable.

Added

• TAILABLE_AWAIT find over local.oplog.rs (via _find_tailable_oplog), and $natural sort on the oplog view.

• A bootstrap oplog noop seeded at server start (Storage.ensure_oplog_bootstrap) so local.oplog.rs is never empty, matching mongod.

The Python server is pure Python — no Rust dependency — and preserves numeric types

The secantus package no longer imports or calls any Rust component. The original in-process engine-swap — where each operator module could delegate to the optional _secantus_core extension under SECANTUS_ENGINE=rust — has been retired in favour of the two-separate-servers model: the Python server is the pure-Python implementation, end to end, and the Rust engines live only in the standalone Rust server (and in the parity-oracle test suites, which import the extension directly rather than through this package). secantus.engine remains as an inert compatibility stub so SecantusDBServer(engine=...) keeps working.

Decoupling the engines let the Python operator engines adopt MongoDB’s numeric type promotion (int32 < int64 < double < decimal128) without being pinned to a not-yet-updated Rust port. $inc, $mul, and the $sum accumulator now preserve the BSON numeric type of their result — Int64(5) incremented by 3 stays Int64(8) instead of narrowing to int32 on the wire — so a client codec that keys on the BSON 64-bit type round-trips correctly. This closes the pymongo gauge’s test_custom_types aggregate/findAndModify decoder cases.

Changed

• secantus is now pure Python with no Rust import in the request path; the SECANTUS_ENGINE in-process accelerator is retired (the Rust engines moved to the standalone Rust server). secantus.engine.available() / enabled() always report Python.

Fixed

• $inc / $mul / $sum preserve the BSON numeric type per mongod’s promotion rules (int32 < int64 < double < decimal128) via the new secantus.numerics helpers, instead of narrowing 64-bit results to int32.

find honours returnKey and showRecordId

find now supports the returnKey and showRecordId cursor options. With returnKey: true each result is reduced to just the keys of the index that serves the query — the index’s key-pattern fields plus the sort fields (a sort by _id, served by the document table’s natural order, yields {_id: <value>}). With showRecordId: true each document is tagged with a $recordId; when returnKey is also set, showRecordId adds nothing, matching mongod. Closes the pymongo gauge’s command-monitoring find with showRecordId and returnKey.

Added

• returnKey (project results down to the serving index’s key fields) and showRecordId ($recordId tag) options on the find command.

createIndexes accepts and ignores the deprecated dropDups option

dropDups was removed in MongoDB 3.0, but modern mongod still accepts it on the wire and silently ignores it rather than rejecting the index spec. SecantusDB now matches that: passing dropDups no longer trips the unknown-field guard. The practical upshot is that building a unique index over data that already contains a duplicate fails on the duplicate with DuplicateKey (11000) — a DuplicateKeyError to the driver — exactly as a real server does, instead of an unrelated “unknown field” error. The collection is left untouched and no index is created. Closes the pymongo gauge’s test_collection.test_index_dont_drop_dups.

Changed

• createIndexes accepts dropDups and strips it from the stored index options (deprecated, ignored — never drops duplicates).

Partial indexes serve range-on-indexed-field queries with a residual clause

A query that puts a range on a partial index’s indexed field and an extra clause that the index’s partial filter absorbs now uses the index — e.g. find({x: {$gt: 1}, a: 1}) against an index on x with partialFilterExpression: {a: {$lte: 1.5}}. The x range rides the index, the a: 1 clause is implied by the partial filter (so the index’s existence already guarantees it) and is rechecked by the exact post-scan matcher, and explain reports IXSCAN with isPartial: true. Previously any multi-field filter fell off the single-field index path to a COLLSCAN.

The relaxation is deliberately conservative: only partial indexes get this treatment, and only when every residual field is a partial-filter field, so a non-partial residual still keeps the query on a collection scan. This closes the last open assertion in the pymongo gauge’s test_collection.test_index_filter.

Changed

• The single-field index lookup and its explain mirror now accept a multi-field filter when the non-indexed fields are absorbed by an implied partial filter, via a shared _single_field_partial_residual_match selector.

Tailable cursors die on capped-collection rollover

A tailable cursor over a capped collection now dies with CappedPositionLost when the collection rolls over and evicts the document the cursor was anchored on — exactly as mongod does. Before, the cursor would blithely keep streaming the post-rollover documents instead of recognising it had been lapped. The server detects this by comparing the cursor’s last-returned position against the collection’s current oldest document; if the anchor has been evicted it returns error 136, which pymongo swallows for tailable cursors (the cursor reports alive == False and the in-flight read yields nothing). Closes the pymongo gauge’s test_cursor.test_tailable.

Fixed

• Tailable cursors on capped collections now surface CappedPositionLost (code 136) when rollover evicts their anchor document, instead of continuing to stream the rolled-over documents.

Change streams report create, modify, and richer DDL events

Change streams opened with showExpandedEvents: true now surface the full set of expanded DDL events that mongod 6.0+ emits. A createCollection (including views) produces a create event, a collMod produces a modify event, and rename events carry an operationDescription with the destination namespace and the dropped target’s UUID under dropTarget. CRUD events (insert / update / delete / replace) on an expanded stream also carry the watched collection’s collectionUUID, the way a real server tags them.

Previously only createIndexes / dropIndexes were emitted as expanded events; create and modify had no oplog entry at all, so a stream waiting for them blocked indefinitely. This completes the showExpandedEvents spec surface that single-node SecantusDB can support (sharding-only events like shardCollection remain out of scope), taking the pymongo change-stream gauge from 102 to 106 passing — a clean sweep of test_change_stream.py.

Added

• create (createCollection / views) and modify (collMod) change-stream events under showExpandedEvents, both gated off by default like the other expanded events.

• operationDescription.{to,dropTarget} on expanded rename events, and collectionUUID on expanded CRUD events.

Resumed change streams return their backlog on open

Opening a change stream with resumeAfter, startAfter, or startAtOperationTime now returns the already-committed backlog — the events between the resume point and now — in the aggregate’s firstBatch, exactly as mongod does. Previously every change-stream open returned an empty firstBatch and deferred all events to the first getMore. That was invisible to most consumers, but a driver that inspects the cursor for buffered data before issuing any getMore (pymongo’s CommandCursor._has_next(), which never sends one itself) saw nothing and reported the stream as empty.

A fresh tail watch has no backlog, so it still opens with an empty firstBatch — the change is scoped to the resuming forms. And because a non-empty firstBatch means pymongo doesn’t overwrite its cached resume token from the open response, an uniterated resumed stream now correctly reports resume_token equal to the token the caller passed in. Closes the pymongo gauge’s test_resumetoken_uniterated_nonempty_batch_* (change-streams prose test #14), lifting the change-stream gauge from 100 to 102 passing.

Fixed

• Resumed change-stream opens (resumeAfter / startAfter / startAtOperationTime) return their committed backlog in firstBatch instead of deferring every event to the first getMore, so a driver that checks for buffered data before any getMore sees the events and an uniterated resumed stream reports the correct resume_token.

Profiler op-class for distinct and count

system.profile entries for distinct and count are now recorded under op: "command", matching mongod — where only find carries op: "query". The previous bucketing filed both under op: "query", so a profile query like {op: "command", "command.distinct": "<coll>"} found nothing. Monitoring tooling that slices the profiler by operation class now sees the same shape it would against a real server.

This closes the pymongo gauge’s test_cursor.test_comment. The OP_MSG exhaust-cursor mid-stream-fault hardening shipped earlier this cycle also gained a dedicated regression test (a synthetic mid-stream getMore fault must terminate the stream with a moreToCome-clear reply, never drop the connection).

Fixed

• distinct / count profiler entries use op: "command" (were op: "query"), so system.profile queries that filter by operation class find them.

OP_MSG exhaust cursors

Exhaust cursors (CursorType.EXHAUST) now stream over the wire the way a real mongod does. When a driver sets the OP_MSG exhaustAllowed flag on a getMore, SecantusDB streams every remaining batch back over the same socket using the moreToCome flag — one round trip instead of a getMore per batch — and closes the stream with a trailing empty reply carrying id: 0. That trailing empty batch is what makes a real server keep the cursor alive until the client has drained it; pinning it faithfully is why pymongo’s command monitor sees find, getMore, getMore, getMore for three documents at batchSize: 1, and why exhaust-pinned connections return to the pool at exactly the right moment.

This closes the last wire-protocol gap behind the pymongo gauge’s test_exhaust / test_exhaust_cursor_db_set cases. The streaming is driven entirely in the connection loop (SecantusDBServer._stream_exhaust_getmore) off the existing cursor registry, so no operator engine or storage path changed; find / aggregate replies that open a cursor are still sent as a single message (mongod streams only on getMore).

Added

• OP_MSG exhaust-cursor streaming: a getMore with the exhaustAllowed flag streams all remaining batches with moreToCome, ending in a trailing empty id: 0 reply (mongod parity). Tailable / awaitData cursors that yield nothing fall back to ordinary getMore rather than spin the stream. A mid-stream getMore that raises unexpectedly still terminates the stream with a moreToCome-clear reply, so the client never sees “Server ended moreToCome unexpectedly”.

Parse-time update validation, partial-index range implication

update now rejects an unknown modifier ($thismodifierdoesntexist) at parse time with code 9, even against an empty collection — matching mongod, which validates the update before matching any document (the per-document apply path would never see an unmatched update). createIndexes rejects a malformed partialFilterExpression (a non-document, an unknown operator, a logical operator with a non-array argument). And a partial index whose filter uses a range operator ({a: {$lte: 1.5}}) is now used when the query provably implies it (an equality a: 1, or a: {$lt: 1}) — a sound, conservative range implication that errs to a full scan rather than risk missing documents; explain flags such a scan with isPartial.

Added

• Sound range implication for partial indexes ($eq/$lt/$lte/$gt/ $gte), with isPartial in the explain IXSCAN stage.

Fixed

• update rejects an unknown modifier at parse time (code 9), even on an empty collection.

• createIndexes rejects a malformed partialFilterExpression.

Upsert subdocument _id, and idempotent drop with write concern

Two real correctness fixes. An upsert whose filter pins _id to a subdocument value ({_id: {f: ..., f2: ...}}) now seeds that _id into the inserted document instead of generating a fresh ObjectId — the seed extraction was skipping every dict-valued filter field to avoid copying operator expressions ({$gt: 5}), but a literal subdocument is a real equality and must be kept. And drop of a non-existent collection now returns {ok: 1} (idempotent, as modern mongod does) rather than NamespaceNotFound, which also lets an unsatisfiable write concern surface its writeConcernError on the reply.

Fixed

• Upsert seeds a subdocument _id from the filter (operator expressions are still correctly excluded).

• drop of a non-existent collection is idempotent ({ok: 1}) and honours an unsatisfiable write concern.

Cursor min() / max() index bounds

The find command’s min / max cursor options are now honoured: they bound a hinted index scan, with max an exclusive upper bound and min an inclusive lower bound on the index key (mongod semantics). Bounds and documents are compared with the same direction-aware byte-sortable key encoder the indexes use, so cross-type ordering and per-field direction are correct. A bound whose field order doesn’t match the hinted index’s key pattern is rejected with mongod’s 51174.

Added

• Cursor min / max index-bound options on find (oracle-pinned against mongod; 51174 on a key-pattern mismatch).

• Rust server: change streams (R3b-a) — aggregate with a leading $changeStream now opens a tailable oplog cursor instead of rejecting, and tailable getMore projects insert / update / replace / delete events (with documentKey, updateDescription, updateLookup fullDocument, pre-images, and a resume token under _id). The projector runs behind a new WT-free Storage trait seam (change_stream_poll / wait_for_oplog / oplog accessors) so the command crate stays WiredTiger-free. Measured +58 on the R8 rust-server gauge (936 → 994 of 1713, zero regressions; 52 are test_change_stream.py). awaitData blocking, resume tokens, and invalidation cursor-close land in R3b-b.

Clustered collections

The clusteredIndex create option is now supported. mongod uses it to make _id the collection’s clustering key — which is exactly SecantusDB’s WiredTiger layout already (the document table is keyed by _id), so this is a metadata-and-reporting feature: the option is validated at create (only {_id: 1} with unique: true, mongod’s two rejection codes), echoed in listCollections.options.clusteredIndex with its v and defaulted name, and reported by listIndexes as a single entry carrying clustered: true (a clustered collection has no separate _id_ index). Secondary indexes coexist normally.

Added

• clusteredIndex create option (create / listCollections / listIndexes), oracle-pinned against mongod.

Matcher correctness, the validate command, and upsert _id fidelity

Continuing the honest-gauge triage, this slice fixes two genuine correctness bugs the gauge surfaced. Embedded-document equality is now field-order-sensitive and exact, recursively — {size: {h: 14, w: 21}} matches a document only when size is exactly that, in that key order (a documented mongod gotcha that Python’s order-insensitive dict == got wrong). And an upsert whose resulting _id is None now reports did_upsert correctly: None was doubling as the “no upsert” sentinel, so a legitimate {_id: null} upsert looked like a no-op to the driver.

The validate command is implemented — a clean, mongod-shaped consistency report (real record and index counts; SecantusDB’s WiredTiger-backed storage has nothing to repair), including mongod’s rejection of full + background together.

Added

• validate command (collection consistency check; full/background/ scandata options, full+background rejected with InvalidOptions).

Fixed

• Embedded-document equality is order-sensitive and exact, recursively, with numeric-bridged leaves (matcher correctness; both query engines — the Rust core already deferred Document/Array equality to Python).

• Upsert with a None _id reports did_upsert and the upserted _id correctly (update and findAndModify paths).

• Rust server: cluster-time gossip — the Rust server now attaches $clusterTime (keyless signature) and operationTime to every reply when the replica-set persona is on, matching mongod and the Python server (shipped in 0.5.2b19). Reads observe the clock via the new secantus_storage::Storage::peek_cluster_time without advancing it; standalone mode stays gossip-free. Measured +6 on the R8 rust-server gauge (930 → 936 of 1713, zero regressions): the $clusterTime-gossip, causal-consistency, and transaction-commit tests that read operationTime. Closes a documented Rust-server gap (backlog §7).

The honest-gauge triage: projection, size caps, snapshot reads, and change-stream fidelity

The first honest pymongo-gauge run (94.8%) left a 64-failure triage list; this slice clears the bulk of it. Projection gained mongod’s exact semantics for three long-standing divergences — {_id: 1}-only specs are inclusion projections, dotted paths fan out over arrays (with {}-skeleton preservation), and $slice interacts with explicit _id correctly — fixed in both the Python and Rust engines with the parity corpus extended to pin every oracle-checked case. Writes now enforce maxBsonObjectSize server-side with mongod’s codes and wording (10334 on insert and update-growth, 17420 on upsert).

Snapshot sessions work end-to-end: readConcern: {level: snapshot} is accepted on find/aggregate/distinct (and their cursor continuations) under the replica-set persona, with atClusterTime stamped on replies for session pinning — and still rejected like a real standalone when the persona is off. The $$NOW system variable landed as part of the same path, seeded per-operation for every command’s let scope.

Change streams got the biggest batch: events that project out the resume token now fail with mongod’s 280 ChangeStreamFatalError and the NonResumableChangeStreamError label instead of being silently swallowed; fullDocument: required/whenAvailable follow post-image semantics (error/null when changeStreamPreAndPostImages is off); resumeAfter rejects invalidate-event tokens (260) while startAfter accepts them; readConcern: local on $changeStream is rejected; unknown pipeline stages return mongod’s 40324 at aggregate time; pipeline-form updates emit update events (with truncatedArrays) instead of replace; and updateDescription.disambiguatedPaths is computed for ambiguous numeric-string field names — in both engines, parity-pinned.

Added

• $$NOW aggregation system variable (constant per operation, all command let scopes).

• updateDescription.disambiguatedPaths on change-stream update events (Python + Rust diff engines).

• atClusterTime on snapshot-read replies (cursor and top-level).

Fixed

• Projection: _id-only inclusion, dotted-path array fan-out, dict skeletons, $slice+_id interaction (both engines).

• Server-side maxBsonObjectSize enforcement (10334 / 17420).

• Change streams: 280 + non-resumable label for projected-out resume tokens, post-image semantics for required/whenAvailable, invalidate tokens rejected by resumeAfter (260), local readConcern rejected, 40324 for unknown stages at create time, pipeline updates as diff events, disambiguatedPaths.

• AggregateError can carry mongod-specific codes (40324).

Real multi-document transactions

commitTransaction and abortTransaction were the last true stubs in the Python server: they returned {ok: 1} while every operation “inside” a driver transaction took effect immediately and could never roll back. They’re real now. Each transaction owns a dedicated WiredTiger session — not the connection thread’s, because pymongo can legally send a transaction’s statements and its retryable commit on different pooled connections — and every statement runs with that session swapped into the storage layer, so snapshot isolation, read-your-own-writes, and rollback all come straight from the same engine mongod uses. Oplog entries are buffered and flushed at commit with one shared commit timestamp plus lsid/txnNumber, so change streams never see uncommitted writes and transaction events carry their session identity, exactly as in mongod.

The server-side state machine (secantus.transactions) pins the spec’s resolution table: statements against unknown or aborted transactions get 251 NoSuchTransaction with the TransientTransactionError label, committed ones get 256, stale txnNumbers get 225 TransactionTooOld, commit is idempotent (driver commit retries depend on it), and any failed statement aborts the transaction server-side. Write-write conflicts between transactions surface as statement-time 112 WriteConflict + transient label; count inside a transaction gets mongod’s 263 OperationNotSupportedInTransaction. Transactions idle past 60s (transaction_lifetime_seconds) are reaped, endSessions/ killSessions abort their session’s transaction, and readConcern: "snapshot" is now accepted inside transactions (every in-transaction read runs against the pinned WT snapshot anyway).

The whole MongoDB CLI toolchain now runs against SecantusDB

The MongoDB Database Tools are strict Go-driver clients, and two of them couldn’t talk to SecantusDB at all: mongostat crashed with a Go nil-pointer panic because serverStatus had no mem section (the tool dereferences mem.supported unguarded), and mongotop failed outright because the top command didn’t exist. Both work now — serverStatus reports a real resident-set size under mem, and top returns mongod’s exact per-namespace shape (counters are zero pending per-namespace instrumentation; mongotop renders it like an idle server).

Every connectable tool in the toolchain is pinned by an end-to-end test in the default suite: mongosh, mongodump/mongorestore, mongoimport/mongoexport (NDJSON + CSV, plus canonical-extended-JSON type fidelity for ObjectId / datetime / Decimal128 / Int64 / Binary), bsondump, mongofiles (GridFS put/get/list/delete against pymongo’s gridfs), and single-iteration mongostat / mongotop probes. The Go tools also exposed two connection-lifecycle nits, now fixed: an RST-style hang-up (how Go’s pool drops connections) no longer dumps a traceback through the catch-all handler, and a request racing stop()’s socket close no longer raises OSError reading the server address.

Compass gets the same treatment, headlessly: every command the GUI issues — the connect-time instance probes, $collStats storage figures, $sample schema analysis, $indexStats, both explain verbosities, and the performance-tab polls — is pinned by tests. That sweep caught explain’s executionStats reporting hardcoded zeroes (Compass would render “0 documents returned” for any query); the server now really executes the query at executionStats verbosity, and aggregate-explain lifts a leading $match so it reports the same IXSCAN decision the real pipeline run uses.

Added

• Multi-document transactions: real commitTransaction / abortTransaction, per-transaction WiredTiger sessions (Storage.begin/use/commit/abort_user_transaction), the secantus.transactions.TransactionRegistry state machine (251/256/225/50911/263/112 + TransientTransactionError labels, idempotent commit, implicit abort on a newer txnNumber, 60s lifetime reaping via SecantusDBServer(transaction_lifetime_seconds=…)), oplog buffering with a shared commit timestamp, and lsid / txnNumber on change-stream events for transactional writes. Conformance: tests/test_transactions.py, tests/test_transaction_registry.py, tests/test_storage_user_txn.py; divergence notes in backlog §3.4.

• Cluster-time gossip: every command reply in replica-set mode now carries $clusterTime (unsigned-cluster placeholder signature, as mongod without auth keys) and operationTime, via the non-minting Storage.peek_cluster_time(). Drivers track these per session and echo readConcern.afterClusterTime on causally consistent reads and transaction starts — the wire shape the transactions / causal-consistency unified specs assert.

• top command — mongod-shaped per-namespace reply (totals with total/readLock/writeLock/per-op {time, count} sections, RBAC top action granted via clusterMonitor); counters are zero (no per-namespace timing instrumentation yet, see backlog §2).

• serverStatus.mem section (bits/resident/virtual/supported) — resident is real (getrusage max-RSS).

• CLI-tool conformance tests: tests/test_mongoimport_export.py, tests/test_mongofiles.py, tests/test_mongostat_mongotop.py, and a bsondump dump-format test in tests/test_mongodump_restore.py.

• Compass headless coverage: tests/test_compass_commands.py pins the full command surface MongoDB Compass issues (instance probes, $collStats/$sample/$indexStats, explain at both verbosities, performance-tab polls, atlasVersion → CommandNotFound).

• serverStatus now carries a secantus subdocument ({server: "python"|"rust", version: ...}) on both servers — categorical self-identification that real mongod never has. The conformance-gauge tripwire checks it over the wire before any test runs, so the gauge can never again silently measure a foreign server.

• Cluster-time gossip: every reply (success or error) now carries $clusterTime (keyless signature) and operationTime when the replica-set persona is on, exactly like a real replica-set mongod; standalone mode stays gossip-free. Reads observe the cluster clock via the new Storage.peek_cluster_time() without advancing it. Clears the startAtOperationTime / causal-consistency bucket of the honest pymongo gauge (Rust-server port tracked in backlog §7).

Changed

• CI: the Linux and macOS test cells install mongosh + MongoDB Database Tools, so the CLI-tool conformance tests run continuously instead of skipping on runners (Windows omitted — mongosh tests skip on win32 by design).

• CI: all actions/* workflow actions bumped to their Node-24 majors (checkout v5, setup-python v6, upload-artifact v6, download-artifact v7, cache v5, setup-go v6, setup-java v5, setup-node v5) ahead of GitHub’s June 16th 2026 forced Node 20 → 24 switch.

Fixed

• Arithmetic expressions ($add / $subtract / $multiply / $divide / $mod) now raise mongod’s type errors instead of silently producing Python-flavoured results: non-numeric operands error with mongod’s exact messages and codes (verified against a real mongod 8.2 oracle), $divide/$mod by zero error (codes 2 / 16610) instead of returning null, bool operands are rejected (BSON arithmetic has no bool), $add/$subtract date semantics follow mongod (date ± millis, date − date → long, two dates in $add → 16612), and Decimal128 operands widen the fold to decimal. The Rust engine defers all error-shaped cases to Python (parity corpus extended first; 536 parity tests green).

• Timeseries collections no longer enforce _id uniqueness, matching mongod (measurements are bucketed by time; _id is not a key there). Doc-table keys for timeseries rows carry a uniqueness suffix so equal _ids coexist; index entries point at the actual row key, updates and deletes preserve it, and the _id point-lookup fast path falls back to a collection scan for timeseries. Closes the last E11000 item from the honest-gauge triage.

• Aggregation-pipeline updates (update_one(filter, [{"$set": ...}])) now project as update change-stream events with a computed updateDescription, matching mongod. The replacement classifier iterated the pipeline list (whose elements are stage documents, not $-prefixed keys) and emitted a full-document oplog entry, so pymongo’s “Test array truncation” unified spec saw replace.

• Stale WT read snapshots made the mutating scanners (drop_collection / drop_database / rename_collection / drop_index / drop_all_indexes, plus index_sizes) silently miss rows committed by other connection threads — a pinned snapshot from an earlier positioned cursor turned drop into a partial or complete no-op, surfacing in the pymongo gauge as drop-then-reinsert E11000 duplicate-key errors. All six now refresh the session snapshot on entry, the same discipline the public read paths already had.

• mongostat no longer panics against SecantusDB (missing serverStatus.mem); mongotop no longer fails with CommandNotFound.

• explain with executionStats / allPlansExecution verbosity now really executes the query and reports actual nReturned / totalDocsExamined / totalKeysExamined / executionTimeMillis instead of hardcoded zeroes; aggregate-explain lifts a leading $match into the reported plan, matching the real pipeline run’s index decision.

• Abrupt client resets (RST close, routine for Go-driver tools) are treated as normal disconnects instead of logging unhandled error on connection N tracebacks.

• Shutdown race: a request arriving while stop() closes the listen socket no longer raises OSError: Bad file descriptor from the address probe.

• The pymongo conformance gauge was not measuring SecantusDB. pymongo’s test helpers freeze DB_IP/DB_PORT at conftest-import time, before the gauge plugin’s pytest_configure wrote them — so local runs silently targeted whatever listened on localhost:27017 (a real mongod, which produced the previous “100.0%” headline) and CI runs, with nothing on 27017, mass-skipped 1100+ tests. The plugin now starts the embedded server in pytest_load_initial_conftests (before any conftest import), aborts via tripwire if the helpers captured the wrong address or the target lacks the secantus marker, and the regenerated honest report shows the real number.

• The weekly validate.yml aggregate never opened its report PR: upload-artifact@v4 strips the docs/ parent from single-file artifacts, so the staging glob matched nothing and untracked new reports were invisible to git diff. Staging now fails loudly on an empty match and git add --intent-to-adds new report files.

• The gauge now runs under one xdist worker (-n1) with a 120s per-test deadline, so a hung test is recorded as a crash and the run continues, instead of pytest-timeout killing the whole process and losing the JSON report.

• Editable storage-engine rebuilds shipped stale Rust extensions: the CMake custom command had no dependency on the crate sources, so once the staged .so existed cargo never re-ran. The build now always invokes cargo (its own dependency tracking decides freshness) and stages with copy_if_different.

[0.5.2b15] — 2026-05-22

WT session leak fix unblocks the rust crud unified runner

SecantusDB cached a WiredTiger session per connection thread in threading.local() but never released it when the thread died. Aggressive driver pools (mongo-rust-driver’s spec runners are the canonical case) opened thousands of short-lived connections; once cumulative connections crossed WT’s 1024-session pool limit, hello started failing mid-handshake with WT_ERROR: out of sessions, which downstream surfaced as a checkpoint stat-error on WiredTigerHS.wt. This release calls Storage._reset_thread_session() in SecantusDBServer._handle_client’s finally block, releasing the session/cursors on disconnect so the pool stays bounded by the live connection count.

The fix also closes a small aggregate validation gap: $out and $merge under readConcern: "linearizable" now return InvalidOptions (72) to match mongod’s invariant (the aggregate-out-readConcern unified spec asserts the rejection).

Together these unblock test::spec::crud::run_unified in the rust gauge — ~80 subtests across find / insert / update / delete / aggregate / countDocuments / distinct / findOne* / replaceOne / bypassDocumentValidation / collation / hints / comments / let bindings / readConcern levels / dots-and-dollars keys, running end-to-end in ~75s. Rust gauge moves from 100 → 101 filters passing.

Fixed

• WT session pool exhaustion under high connection churn: per- connection-thread WT session is now released on disconnect instead of leaking until the engine’s 1024-session pool fills.

• aggregate with $out / $merge under readConcern: "linearizable" now errors with InvalidOptions (72) instead of silently returning an empty array.

Changed

• Rust conformance gauge: test::spec::crud::run_unified is now in the include list. test::spec::collection_management::run_unified and test::spec::sessions::run_unified remain deferred for separate gaps (time-series collections, snapshot read concern under fake replica-set topology).

[0.5.2b14] — 2026-05-22

Change-stream split-event implementation: real {fragment: N, of: M}

The splitLargeChangeStreamEvents opt-in previously stamped every event with {fragment: 1, of: 1} regardless of size — correct from the driver’s reassembly perspective for events under 16 MB, but wrong for events that genuinely exceed the BSON wire limit (the typical case being an update with fullDocumentBeforeChange: required where the pre-image plus a large $set value together push the projected event past 16 MB).

This slice ships real splitting. When an event’s BSON-encoded size exceeds 16 MB, stamp_split_event distributes any top-level field larger than 1 MB into its own fragment; light metadata (resume token, operationType, clusterTime, ns, documentKey, wallTime, …) is copied verbatim into every fragment so each is a valid change event the driver can process independently. Fragments share the same _id resume token; drivers reassemble by combining fields across fragments with matching _id. The split is size-based, not field-name-based: any heavy field qualifies (in practice fullDocument, fullDocumentBeforeChange, and updateDescription.updatedFields are the candidates).

Two opt-in paths now both light up the producer flag: the original $changeStream: {splitLargeChangeStreamEvents: true} spec field plus the pipeline-stage form [{$changeStreamSplitLargeEvent: {}}] that the rust / node / java drivers use from their high-level watch() APIs. Either signals to the producer that fragmentation should run.

mongo-rust-driver’s test::change_stream::split_large_event — which constructs a 10 MB pre-image + 10 MB update value and asserts events[0].splitEvent == {fragment: 1, of: 2} and events[1].splitEvent == {fragment: 2, of: 2} — now passes end- to-end. The rust gauge moves from 92 → 93 (still 100%).

Added

• src/secantus/aggregate.py: $changeStreamSplitLargeEvent registered in _STAGES as a pass-through marker. The stage itself is a no-op in the pipeline (real splitting happens upstream at event-projection time); accepted spec is {}.

• src/secantus/changestreams.py:

  • _HEAVY_FIELD_BYTES = 1 MB and _SPLIT_THRESHOLD_BYTES = 16 MB.

  • stamp_split_event(event) -> list[dict] rewritten to compute the event’s BSON size, identify heavy top-level fields by per-field encoding, and emit one fragment per heavy field with light metadata duplicated. Returns one event (no split) when the original is under 16 MB.

• src/secantus/commands.py: change-stream aggregate handler detects the $changeStreamSplitLargeEvent pipeline stage and sets cs_spec.split_large_events = True so the producer fragments on that opt-in path too. Producer call sites changed from events.append(stamp_split_event(ev)) to events.extend(stamp_split_event(ev)).

• tests/test_change_stream_split_stage.py (5 tests): pipeline parses cleanly; bad-spec rejected standalone; stage works outside change-stream context (no-op pass-through); 10 MB pre-image + 10 MB $set value produces two fragments with correct {fragment: N, of: 2} envelopes and shared resume token, heavy fields distributed one per fragment; small event with opt-in still produces single {fragment: 1, of: 1} fragment.

Changed

• rust_validation/include_paths.py adds test::change_stream::split_large_event to INCLUDE (rust gauge 92 → 93). The previous EXCLUDED entry’s rationale is removed.

Point lookups by _id stop scanning the whole collection

Every MongoDB collection has an _id index, and looking a document up by its _id is the single most common read an application makes. In SecantusDB that lookup was quietly walking the entire collection: the _id_ index is virtual — the documents table is itself keyed by the encoded _id, so there’s no separate entries table for it — and the query planner’s index pickers only ever consulted the stored secondary indexes. With nothing matching _id, every find({_id: …}) fell back to a COLLSCAN that got linearly slower as the collection grew.

find, findOne, updateOne, and deleteOne filtered on _id now take a direct primary-key point lookup on the documents table instead. On a 5,000-document collection that turns a 45 ms read into a 0.6 ms read — about 74× faster — and the gap widens with collection size. explain reports the lookup honestly as an IXSCAN on the _id_ index. Equality ({_id: x}), {_id: {$eq: x}}, and {_id: {$in: […]}} are all accelerated; range, regex, and multi-field filters keep their existing routing. The cross-numeric _id collision (1 == 1.0 == Decimal128("1")) is preserved because the fast path encodes the query value with the same encode_value used for the stored key.

Fixed

• find / findAndModify / single-document update / delete filtered on _id equality ({_id: v}, {_id: {$eq: v}}, {_id: {$in: [...]}}) now do an O(1) primary-key point lookup on the documents table instead of a COLLSCAN, and explain reports IXSCAN on the _id_ index. Discovered with the new bench/rw_harness.py concurrent read/write validator, whose interleaved _id read-backs collapsed throughput on growing collections.

[0.5.2b7] — 2026-05-21

Rust driver gauge — 6th conformance gauge alongside the rest

mongo-rust-driver is now the 6th driver gauge alongside pymongo / go / node / java / ruby. The runner spawns SecantusDB on an ephemeral port and runs cargo test --lib -p mongodb against a curated include set with MONGODB_URI explicitly overridden in the subprocess env — the rust driver’s fallback chain ($MONGODB_URI → ~/.mongodb_uri → localhost:27017) is short-circuited at the first step so a stray ambient URI in the user’s shell can’t route the gauge at a real mongod. A belt-and-braces hello.setName == "secantus" probe at runner start adds a second layer of confirmation.

Initial baseline: 12 curated handshake + single-collection CRUD filters expand to 24 actual test runs (libtest substring matching fans test::coll::find out across find_allow_disk_use etc.). The first cut surfaced two real conformance gaps; both fixed in the same release:

• listDatabases now populates sizeOnDisk per database (sum of bson-encoded doc bytes across the db’s collections — same accounting collStats / dbStats use). empty is derived from the size (size == 0). totalSize reports the actual sum across all dbs. Previously every entry carried a placeholder sizeOnDisk: 0 and empty: false.

• hello.client subdoc captured per connection in the registry and surfaced back via currentOp as clientMetadata. Drivers use it to identify their own connections in admin tooling — they send the subdoc on handshake and expect to read it back. Previously we threw the subdoc away on hello and currentOp emitted no clientMetadata field.

After the fixes the rust gauge runs 24/24 (100%).

Added

• rust_validation/ package — __init__.py / include_paths.py / runner.py / generate_report.py, mirrors the ruby_validation/ shape.

• vendor/mongo-rust-driver submodule (7th vendored driver).

• invoke validate-rust task; validate-all GAUGES extended with the 6th entry.

• .github/workflows/validate.yml matrix entry for rust; toolchain via dtolnay/rust-toolchain@stable; cargo cache key on vendor/mongo-rust-driver/Cargo.lock.

• validation_summary integration — _collect_rust, PANEL_PROSE entry, stale “pending” marker removed.

• docs/validation-report-rust.md (new) + toctree entry + index.md prose update referencing all six drivers.

• tests/test_list_databases_size.py (4 tests): populated db has non-zero sizeOnDisk + empty: false; totalSize sums per-db sizes; nameOnly skips the size walk; filter scopes against the full descriptor.

• tests/test_hello_client_metadata.py (2 tests): pymongo’s driver / OS / appname metadata round-trips through hello → currentOp; clientMetadata is a dict shape when present.

Changed

• commands._list_databases: computes sizeOnDisk per db as sum(collection_data_size(...) for coll in list_collections); empty derived from size; totalSize is real.

• commands._hello: captures doc.get("client") and stashes via ctx.connections.set_client_metadata(...).

• commands._current_op: emits clientMetadata on each in-progress op when the connection’s registry entry has it.

• connreg.ConnInfo grows client_metadata: dict | None; ConnectionRegistry.set_client_metadata(conn_id, metadata) added; get() and snapshot() thread the new field through their fresh-copy semantics.

[0.5.2b5] — 2026-05-21

$setWindowFields rank functions — $rank / $denseRank / $documentNumber

Closes one of the explicit deferred surfaces from the b35 $setWindowFields minimum-viable subset. Driver test suites probe all three regularly; the previous wire-level response was an explicit “rank functions and time-series operators are not yet implemented” AggregateError.

The three functions share one linear walk per partition. They sit in output: {<field>: {$rank: {}}} alongside the accumulator functions but evaluate differently — no window argument (mongod rejects it), no function argument (the spec is just {$rank: {}}), and the value is computed once per partition slot rather than rolled up over a windowed subset.

• $documentNumber — 1-indexed position within the partition. Independent of ties; happy with or without sortBy.

• $rank — 1-indexed position with gaps after ties: tied rows share the lower rank, next non-tied row jumps by the number of ties ([10, 20, 20, 30] → [1, 2, 2, 4]). Requires sortBy.

• $denseRank — 1-indexed position without gaps: tied rows share, next row is +1 ([10, 20, 20, 30] → [1, 2, 2, 3]). Requires sortBy.

Tie detection is sort-key tuple equality: compound sortBy specs work uniformly. Rank counters reset at every partition boundary, same as the accumulator functions.

Added

• src/secantus/aggregate.py: _RANK_FUNCS frozenset; the validation branch in _stage_set_window_fields recognises the three rank ops, rejects window / non-empty arg, and requires sortBy for $rank / $denseRank. The per-row loop branches: rank functions look up a precomputed array, accumulators take the existing windowed path.

• _compute_rank_state helper does one linear walk over each partition’s sort-key tuples and emits per-slot vectors for whichever of the three functions are referenced. _sort_key_values extracts the tuple the tie comparison runs on.

• tests/test_window_rank_functions.py (13 new tests) — covers $documentNumber with and without sort, per-partition reset, $rank gaps with ties, $rank == $documentNumber without ties, compound sort tie detection, $denseRank no-gap semantics, all three together in one stage, partition-resets, plus four validation tests (window rejected, sortBy required for $rank / $denseRank, non-empty arg rejected).

Changed

• _stage_set_window_fields docstring rewritten to document the rank-function surface.

• tests/test_set_window_fields.py: the b35 placeholder test test_unsupported_rank_function_raises is replaced by test_unsupported_time_series_function_raises, which now probes with $derivative to keep the deferred-surface guard alive.

apiStrict: true rejects distinct (narrow command-name gate)

The Stable API v1 contract rejects a list of commands when apiStrict: true is set. SecantusDB already rejected non-v1 aggregation stages inside aggregate pipelines (lights up mongo-java-driver’s versioned-api/aggregate on database test that probes with $listLocalSessions). The matching command-name gate had been intentionally left off in a previous attempt: a broader whitelist invert reportedly caused 6 cascade failures via MongoConnectionPoolClearedException.

A focused Java-gauge run with a narrow gate (_API_V1_REJECTED_BY_NAME = {"distinct"}) tells a different story. Rejecting only distinct produces +1 pass for the canary crud-api-version-1-strict.yml distinct appends declared API version test and zero new failures across the 900-test mongo-java-driver suite — no pool-clear symptoms anywhere in the JUnit XML. The cascade the previous attempt observed was not pool-clear semantics; it was the broader invert also rejecting count (used internally by estimatedDocumentCount) and other handshake-adjacent internal commands. The narrow gate sidesteps that mechanism entirely.

Added

• src/secantus/commands.py: _API_V1_REJECTED_BY_NAME frozenset (one entry: distinct); the dispatch apiStrict block grew a command-name check that runs before the aggregation-stage check. The rejection’s errmsg matches mongod’s "Provided command distinct is not in API Version 1" so the unified test runner’s errorContains assertion fires cleanly.

• tests/test_api_strict.py (5 new tests): distinct rejected under apiStrict: true with code 323; distinct allowed without apiStrict; count still allowed under apiStrict (the cascade-avoidance check); find still allowed; aggregate with a v1 stage still allowed (gates compose).

Changed

• Backlog §5 entry on apiStrict pool-clear struck through with the empirical resolution path. The previous theory turned out to be wrong about the mechanism — narrow rejection works.

Pymongo gauge: +80 passing tests from five newly-includable files

Cross-gauge audit of currently-excluded test files against the work shipped in this development cycle (0.5.2b1 + the rank-functions and apiStrict slices above) identified five pymongo test files that pass cleanly now and had been excluded purely because the supporting features hadn’t shipped. Adding them to pymongo_validation/include_paths.py bumps the gauge from 959 → 1039 passing with zero new failures, +25 new skips (genuine feature gaps the suite self-skips on), overall pass rate stays at 100%.

• test_collation.py (16 new tests) — unlocked by per-index collation work (single-field, compound, sort acceleration).

• test_versioned_api.py (4 tests) + test_versioned_api_integration.py (36 tests) — unlocked by the apiStrict aggregation-stage gate and the new distinct command-name gate.

• test_command_logging.py (20 tests) + test_logger.py (4 tests) — command monitoring / logging format conformance; no SecantusDB-specific blocker.

The audit also confirmed no flip-worthy candidates in the go / node / java / ruby gauges — every remaining exclusion in those gauges is a feature genuinely out of scope (replica sets, transactions, encryption, text indexes, GridFS, time-series, etc.).

Changed

• pymongo_validation/include_paths.py — five test files added to INCLUDE. Inline comments name the slice that unlocked each.

[0.5.2b1] — 2026-05-20

MONGODB-X509 auth — cert subject DN as the username

The natural sequel to the b22 mTLS slice. mTLS gives you a transport-layer “approved client” gate; MONGODB-X509 turns the client cert’s subject DN into the user identity directly, no SCRAM step. Same flow MongoDB Atlas X509 deployments use: create the user on $external with mechanisms: ["MONGODB-X509"] and the cert DN as the username, connect with ?authMechanism=MONGODB-X509&authSource=$external, the server matches the DN from the verified cert against the user record. No password to rotate, no SCRAM round-trip, no shared secret on disk.

Mixed mechanisms work too — a user record can carry both SCRAM-SHA-256 and MONGODB-X509 in mechanisms for migration or to keep a SCRAM fallback. The driver picks per-connection from saslSupportedMechs.

Closes the “transport-layer gate only” caveat the production + configuration docs called out when mTLS shipped; documentation updated to point at the worked X509 example as the alternative to SCRAM-on-top.

Added

• secantus.auth.MONGODB_X509 constant, X509_CREDENTIAL_MARKER for the user record’s credentials doc (no password to hash — the credential IS the cert), and secantus.auth.subject_dn_from_peercert() which converts Python’s ssl.SSLSocket.getpeercert() tuple-of-tuples into the mongod-style RFC 4514 DN string (short attribute names, most-specific-first, special-char escaping).

• CommandContext.peer_cert_dn — server captures the verified client cert’s DN once per connection (right after the TLS handshake in _handle_client), replays it into every CommandContext so the auth handlers can read it.

• _sasl_start_x509 and the legacy authenticate command handler — pymongo / Java / Go / Node all use the legacy command path for X509, not saslStart. Both are wired up and refuse cleanly on plaintext connections / non-X509 users / payload-DN mismatch.

• createUser accepts mechanisms=["MONGODB-X509"] with no password (cert IS the credential). Mixed ["SCRAM-SHA-256", "MONGODB-X509"] works too — SCRAM creds are derived from pwd, X509 marker is written alongside.

• tests/test_x509_auth.py — 9 tests: DN extraction unit tests (reversal, short names, escaping, empty), end-to-end happy path via pymongo, refused-with-no-matching-user, refused-for-SCRAM-only user, SCRAM still works on mTLS-required server, X509 refused on plaintext connection.

Changed

• saslSupportedMechs now includes MONGODB-X509 when a user has that mechanism in its credentials doc. SCRAM is still listed first when both are available (drivers pick the strongest).

• _PRE_AUTH_COMMANDS includes authenticate so the legacy X509 command path bypasses the require-auth gate (same as saslStart / saslContinue already did for SCRAM).

• docs/authentication.md — new MONGODB-X509 section with the provisioning + connection examples; the stale “what’s not here yet” list rewritten (RBAC, updateUser, grantRolesToUser, TLS, SCRAM-SHA-1 all shipped slices ago and shouldn’t have been listed as gaps).

• docs/production.md + docs/configuration.md — mTLS sections now offer two routes (SCRAM-on-top vs MONGODB-X509) instead of the “transport-layer only, MONGODB-X509 is a follow-on” caveat.

Per-index collation — case- and accent-insensitive lookups at IXSCAN

The last entry on the compatibility doc’s “Deferred” list is gone. Before this slice, the per-query collation infrastructure already honoured collation for find / count / distinct / findAndModify via matches() — but any query that carried a collation argument fell through to COLLSCAN by design, because index entries were written in raw BSON codepoint order. The storage-layer comment said as much: “we don’t support per-index collation yet, so the safe path is always-COLLSCAN-when-collation.”

That comment is gone. createIndexes with a collation option now writes index entries under collation-normalised bytes — strings that compare-equal under the collation produce the same key, so a query carrying a matching collation hits the same row at IXSCAN. Strength 1/2/3 + caseLevel are supported; numericOrdering still falls back to COLLSCAN (would need a length-prefixed digit-run encoding to stay byte-sortable, deferred until a workload needs it).

Two indexes on the same field with different collations are allowed — the picker walks every candidate and uses the one whose collation exactly matches the query’s. Useful for collections that mix case-sensitive and case-insensitive lookups against the same column. Unique indexes with a collation enforce uniqueness under the collation: two docs differing only by case collide against a strength: 2 unique index. Only the single-field equality / range / $in picker threads collation through today; multi-field filters combined with a collation still fall back to COLLSCAN. Worth widening case-by-case when a workload needs it.

Added

• sortkey.encode_value(value, *, collation=None), encode_value_directed, encode_compound, and the bound helpers (gt_bound / gte_bound / lt_bound / lte_bound) all take an optional collation kwarg. When set and the value is a string, normalisation runs through secantus.collation.normalize_for_index_bytes before encoding, so equal-under-collation strings produce equal bytes.

• Collation.supports_index_encoding — True for strength 1/2/3 + caseLevel, False for numericOrdering. The picker treats numericOrdering as “no index available for this collation.”

• secantus.collation.normalize_for_index_bytes(s, collation) — bytes form of the collation-normalised string (strips accents for strength 1, casefolds for strength ≤ 2, UTF-8 encodes).

• _parse_index_collation helper in storage.py — reads an index’s stored collation option blob into a Collation, returning None for collations that don’t support index encoding.

• tests/test_per_index_collation.py — 11 tests covering routing (matching collation → IXSCAN, mismatch → COLLSCAN, no-collation query against collation-having index → COLLSCAN), correctness on equality / range / $in / update_one, numericOrdering fallback, unique-index-under-collation, and two indexes on the same field with different collations.

Changed

• _index_key / _index_key_variants (the byte-key builders for index writes) accept a collation kwarg; the storage writers load it from the index’s stored options and pass it through.

• _find_leading_field_index + _pick_index_for_filter + _try_index_lookup + _try_index_id_keys thread a collation kwarg. Indexes whose stored collation doesn’t exactly equal the query’s are skipped — the caller falls back to COLLSCAN, which is the safe semantics. _pick_compound_eq_index / _pick_compound_range_index skip collation-having indexes entirely; compound pickers don’t yet support collation, and picking a collation-having index for a no-collation multi-field filter would return wrong rows.

• explain_plan takes a collation kwarg, and the explain command extracts it from the wrapped command. Mismatched collations report COLLSCAN in winningPlan; matched ones report IXSCAN with the index name.

• find_matching’s “if collation present, always COLLSCAN” gate has been rewritten — now tries the collation-aware index path first, falls back to COLLSCAN only when no matching index exists.

• docs/compatibility.md field-options table: collation is now Honoured rather than Accepted-but-ignored. The Deferred list is now empty.

• docs/indexes.md: new “Per-index collation” section with examples and rules; the “What’s still missing” list updated to call out compound-index collation as the next widening.

• tasks/backlog.md §2: the per-index-collation stopgap entry is struck through with a one-line summary of what shipped and the remaining compound-index limitation.

Compound-index collation — multi-field filters light up under matching collation

The b25 per-index collation slice closed the single-field path but left the compound pickers (_pick_compound_eq_index / _pick_compound_range_index) skipping any collation-having index — a multi-field filter combined with a collation argument fell back to COLLSCAN even when a compound collation index could have served it. This slice closes that gap.

Both compound pickers now thread collation through and gate by exact match against each index’s stored collation, the same rule the single-field path already used. The lookup builders thread collation into every encode_value_directed call (leading-equality prefix bytes and the trailing operator’s bound bytes), so the lookup hits the same byte rows the index-write path produced. Strength 1/2/3 + caseLevel apply uniformly across single- and compound-field indexes; numericOrdering still falls back to COLLSCAN at every level. The unique-probe path now reads the index’s stored collation too, so a unique compound index with {strength: 2} correctly rejects a second insert whose values collide under the collation.

After this slice, every CRUD pattern that the single-field collation path covers — equality / range / $in / update / unique enforcement — covers under compound indexes too.

Changed

• _pick_compound_eq_index + _try_compound_eq_id_keys thread collation through; the compound-eq lookup builds the prefix bytes under the same collation as the index.

• _pick_compound_range_index + _try_compound_range_id_keys thread collation through; the trailing operator’s $eq / $in / $gt / $gte / $lt / $lte bounds are all encoded under the collation.

• _try_index_id_keys no longer short-circuits compound pickers when collation is set — they’re called with the collation kwarg and use the exact-match gate.

• _pick_index_for_filter (the explain planner) mirrors the same threading, so explain reports IXSCAN for collation-matching multi-field queries.

• _unique_conflict reads each index’s stored collation via _parse_index_collation and threads it to _index_key, so the unique probe collides on byte-equal canonical keys (the bug that let ("Alice","Boston") and ("ALICE","BOSTON") both land in a unique strength-2 compound index).

• docs/indexes.md “Per-index collation” section rewritten to cover the compound case with examples; “What’s still missing” drops the compound-collation entry.

• tests/test_compound_index_collation.py (10 new tests): compound bare-eq IXSCAN under matching collation, leading-prefix-only scan, mismatch → COLLSCAN, no-collation-vs-collation index selection across two indexes on the same fields, compound prefix + trailing-operator ($gt, $in) under collation, update via compound collation index, unique compound collation enforcement, numericOrdering fallback.

Sort acceleration with collation — index walk replaces Python sort

The third collation slice closes a quieter gap left by the preceding two. The b25 + b27 slices wired up filter-side collation routing — equality / range / $in / compound bare-eq / compound prefix + trailing-operator all light up at IXSCAN when the query’s collation matches an index’s stored collation. But the sort path stayed on COLLSCAN + Python sort_docs: any query carrying a collation argument fell into a single branch that never tried sort acceleration, even when an index whose collation matched the query’s would have given the requested order for free just by walking it.

That branch is gone. The collation and non-collation paths through find_matching are now unified, and every sort-picker call (_find_leading_field_index for single-field sorts, _compound_index_for_sort for multi-field) threads collation_obj through with the same exact-match gate as the filter side. A find().sort("name", 1).collation({strength: 2}) walks a {name: 1} strength-2 collation index forward; -1 walks it backward; multi-field sorts that exactly match (or fully invert) a compound collation index’s key spec walk it forward or backward respectively, and no Python sort runs in either case. The same gate keeps no-collation sorts off collation indexes (walking would give the wrong order) and vice versa.

After this slice the collation domain is structurally complete: every CRUD pattern that hits an index without collation — filter lookup, range, $in, multi-field filter, sort, compound sort, unique enforcement — hits the index when a matching collation is in play, and falls back to COLLSCAN + matches() + sort_docs when no matching index exists.

Changed

• find_matching’s elif collation_obj is not None: ... branch removed; the no-collation branch’s sort logic now runs for both cases, with collation=collation_obj (which is None when no collation set) threaded through every picker call. Single-field sort + filter on the sort field, single-field sort with empty filter, and multi-field sort (compound key match) all collation-gate.

• _compound_index_for_sort takes an optional collation kwarg and gates by exact match against each index’s stored collation (same rule as _find_leading_field_index and the compound filter pickers). Multikey indexes are still excluded from sort acceleration regardless of collation.

• explain_plan mirrors the threading: _find_leading_field_index and _compound_index_for_sort both receive collation=collation_obj, so explain reports IXSCAN with the right direction for collation-matching sort queries and COLLSCAN otherwise.

• docs/indexes.md “Per-index collation” section grows a “sort acceleration honours the same gate” subsection with worked forward / backward / mismatch examples.

• tests/test_sort_with_collation.py (8 new tests): single-field ASC + DESC sort with matching collation walks index forward / backward; no-collation sort against collation index → COLLSCAN; strength-2 index + strength-3 query → COLLSCAN; filter on sort field with matching collation hits index in order; multi-field sort that matches a compound collation index walks forward; the full-inverse sort walks backward; multi-field mismatch falls back to Python sort.

$type: "int" / "long" distinguishes by BSON type tag, not value range

A quieter long-standing bug in the $type query operator. The _TYPE_PREDS table used a Python value-range check (-2**31 <= v <= 2**31 - 1) to distinguish int32 from int64. A doc inserted as Int64(5) — value fits in int32 numerically, but its BSON tag is int64 — was matched by $type: "int" instead of $type: "long", contradicting mongod.

pymongo’s BSON decoder already preserves the int32/int64 distinction by class: int32 round-trips as plain int, int64 round-trips as bson.Int64 (a subclass of int). The fix keys on isinstance(v, bson.Int64) for “long” and isinstance(v, int) and not isinstance(v, (bool, Int64)) for “int” — type-tag-faithful, no value-range arithmetic.

$convert: {to: "long"} had a paired bug: it returned a plain int so its output couldn’t be matched by $type: "long" on a downstream $match. Now wraps the result in Int64 for code 18 (int64); to: "int" (code 16) still returns plain int.

Changed

• src/secantus/query.py: replaced _is_bson_int(... ranged=...)

  • _INT32_RANGE with three named predicates (_is_int32, _is_int64, _is_bson_number). _TYPE_PREDS entries for int / 16 / long / 18 / number now route through them.

• src/secantus/expressions.py: _convert_value code 18 path wraps its result in Int64 (codes 16 and 18 share the input coercion logic but the wrapper diverges).

• tests/test_type_int32_int64.py (8 new tests): Int64(5) → $type: "long" (not int); plain int(5) → $type: "int"; large int (2**40) round-trips as Int64 → long; $type: "number" accepts both; numeric $type codes (16, 18) agree with their string aliases; array-form $type matches either; $convert: {to: "long"} output matches $type: "long"; $convert: {to: "int"} output matches $type: "int".

$unionWith aggregation stage

A v1 stable-API stage that wasn’t yet wired up. $unionWith concatenates docs from a second collection — optionally filtered through a sub-pipeline — onto the current pipeline’s input. Driver test suites probe it routinely; the prior wire-level response was a generic “unsupported aggregation stage” error.

Both spec shapes ship:

• Shorthand: {$unionWith: "<coll>"}

• Full form: {$unionWith: {coll: "<coll>", pipeline: [...]}}

Outer docs land first, then the union docs in the order the sub-pipeline produced them. No deduplication — duplicates across the boundary survive, matching mongod. The sub-pipeline runs in a fresh :class:PipelineContext; outer $lookup let variables are deliberately not visible (mongod doesn’t accept a let field on $unionWith). Chained $unionWith stages accumulate; downstream $sort / $group / $count / $limit see the combined set.

A non-existent target collection is treated as empty (mongod’s behaviour). Bad specs (non-string shorthand, missing coll, non-array pipeline) surface as AggregateError to the client.

Added

• src/secantus/aggregate.py: _stage_union_with handler; wired into _STAGES next to $geoNear. ~30 LOC + docstring.

• tests/test_union_with.py (11 new tests): shorthand form; full form with and without sub-pipeline; outer-first ordering; no-dedup across boundary; chained $unionWith; downstream $group / $sort+$limit; missing collection treated as empty; empty outer + non-empty union; bad-spec rejection (numeric spec, missing coll, non-array pipeline).

• docs/aggregation.md stages table grows a row.

admin.system.users is a synthetic read-only view onto the user store

Credentials live in a dedicated WT table (secantus_users) that createUser / updateUser / dropUser / usersInfo own. But find / aggregate / count against admin.system.users — mongod’s canonical user-storage namespace — searched the empty regular doc table and returned nothing. Tools and a few driver tests that introspect the user list via db.system.users.find() saw an empty collection on SecantusDB even after a createUser landed.

This slice mirrors the oplog pattern (local.oplog.rs is a synthetic view onto secantus_oplog). admin.system.users is now read-only-surfaced: find / aggregate / count route through _find_system_users / _count_system_users, which scan the user table on a fresh WT session for cross-thread visibility and apply the standard filter / sort / skip / limit / projection / collation pipeline against the decoded records.

The stored records already carry the mongod-shaped fields (_id = <db>.<user>, user, db, credentials, roles, mechanisms), so the view requires no schema synthesis. Users created against any database all surface under admin.system.users (matching mongod — every user record lives in admin.system.users regardless of its auth db, and the per-record db field names the auth database). Querying any other db’s system.users returns empty rows (also mongod’s behaviour).

Writes are rejected with code 13 (Unauthorized) and a clear errmsg pointing users at createUser / updateUser / dropUser. The existing _reject_oplog_rs_write helper grew a clause for admin.system.users — it was already wired into every write command (insert / update / delete / findAndModify / drop / create / createIndexes) so the rejection lands everywhere implicitly. Function name kept (_reject_oplog_rs_write) for churn reasons, with the docstring updated to cover both views.

Added

• storage._is_system_users / _scan_user_records / _find_system_users / _count_system_users — the synthetic view helpers, modelled directly on the oplog view’s pattern.

• storage.find_matching + count_matching route through the new helpers when (db, coll) == ("admin", "system.users").

• tests/test_system_users_view.py (13 new tests): find / count / projection / aggregate against the view; users created across multiple databases all visible; filter on db field; other-db system.users is empty; write rejection on insert / update / delete / drop with code 13; dropUser / updateUser mutations reflected in the view.

Changed

• commands._reject_oplog_rs_write grew a second case for admin.system.users. Docstring rewritten to cover both views. Existing call sites pick up the new behaviour with no further edits.

$redact aggregation stage

The largest v1 stable-API aggregation stage still missing. $redact implements content-based document and sub-document pruning — the pipeline analogue of mongod’s field-level access control. The stage’s expression evaluates against each (sub-)doc and returns one of three sentinel strings; the result drives include / exclude / recurse behaviour. Driver test suites probe it routinely.

• "$$KEEP" — include the sub-doc as-is, no recursion into nested sub-docs. Useful for “trusted” sub-docs whose interior shouldn’t be re-evaluated.

• "$$PRUNE" — drop the sub-doc. At the top level the doc leaves the pipeline entirely; in a nested context the sub-doc is removed from its parent field, or from its array element slot (with the surrounding array preserved).

• "$$DESCEND" — recurse into every dict-valued field and every dict-valued list element. Non-dict scalars and non-dict list elements pass through unchanged.

The three sentinels are wired into the expression evaluator as system variables (alongside $$ROOT, $$CURRENT, $$REMOVE); their resolved value is the literal "$$NAME" string the stage handler dispatches on. Returning anything else from the expression raises AggregateError — matches mongod.

The stage uses the standard $cond / $switch / $let / $ifNull plumbing that the rest of the expression engine already provides, so the typical pipeline shape works straight out:

[{"$redact": {
    "$cond": {
        "if": {"$eq": [{"$ifNull": ["$classified", False]}, True]},
        "then": "$$PRUNE",
        "else": "$$DESCEND",
    },
}}]

Added

• src/secantus/aggregate.py: _stage_redact handler + private _redact_subdoc / _redact_descend recursive helpers, wired into _STAGES next to $unionWith. The _redact_descend walker preserves non-dict scalars and non-dict list elements; pruned sub-docs are dropped from their parent field or array.

• src/secantus/expressions.py: _resolve_var recognises $$KEEP / $$PRUNE / $$DESCEND and returns the literal "$$NAME" string — same pattern as $$REMOVE for $setField.

• tests/test_redact.py (11 new tests): unconditional KEEP and PRUNE; conditional KEEP-vs-PRUNE access-control canon; DESCEND with nested sub-doc pruning; DESCEND into arrays of sub-docs with non-dict elements preserved; multi-level deep recursion; KEEP short-circuits descent (nested PRUNE never fires); chained with $match; non-sentinel return rejected; null / empty expression rejected; array-element KEEP preserves nested sub-docs unchanged.

admin.system.version returns the auth-schema doc

The companion to the b31 admin.system.users view. Some user-management tools (and a handful of driver tests) read admin.system.version.find({_id: "authSchema"}) on startup to gate which user-management features they offer; pre-slice that namespace was empty and tools either skipped features or assumed the lowest schema version.

The view returns one hard-coded doc:

{"_id": "authSchema", "currentVersion": 5}

currentVersion: 5 is the SCRAM-SHA-256 baseline (MongoDB 4.0+), which is what SecantusDB actually implements — so the answer is honest, not just placating. Other databases’ system.version still returns empty. Writes are rejected with code 13 (Unauthorized) via the same _reject_oplog_rs_write helper that gates admin.system.users and local.oplog.rs.

Added

• storage._is_system_version / _system_version_docs / _find_system_version / _count_system_version — same pattern as the b31 admin.system.users view; the doc set is fixed at one entry rather than scanned from a table.

• storage.find_matching + count_matching route through the new helpers when (db, coll) == ("admin", "system.version").

• commands._reject_oplog_rs_write grew a third case for admin.system.version; existing call sites pick up the rejection with no further edits.

• tests/test_system_version_view.py (10 new tests): find / find_one / count / aggregate read paths; non-matching filter returns empty; other-db system.version is empty; write rejection on insert / update / delete / drop with code 13.

renameCollection cross-process safety — pinned by WiredTiger.lock

A backlog item (“renameCollection: atomic per the storage RLock, but no protection against concurrent writers across worktrees”) turns out to be structurally addressed by WiredTiger itself. wiredtiger_open takes an exclusive lock on the data directory at open time; a second open on the same path fails with WT_ERROR Resource busy before any state is touched, so the “concurrent writers across processes” scenario can’t exist in the first place.

Within-process atomicity is the storage RLock. Cross-process exclusion is WiredTiger.lock. The two layers compose: rename is safe under both. The backlog entry is struck through.

Added

• tests/test_storage_exclusion.py (2 new tests) pinning the guarantee: a second Storage(path=...) on the same on-disk directory raises a WiredTigerError whose message contains "busy"; the first instance keeps working unaffected. rename_collection survives a close + reopen round-trip — the renamed namespace is visible to a fresh Storage instance.

$setWindowFields aggregation stage — minimum viable subset

The largest v1 stable-API stage that wasn’t yet wired up. $setWindowFields is mongod’s windowed-analytics surface — running totals, rolling averages, per-partition rankings — all expressed as a partition + sort + per-row windowed accumulator over the input. Driver test suites probe it heavily.

Spec shape::

{
    partitionBy: <expression>,         # optional; default = single partition
    sortBy: <sort spec>,               # optional; default = input order
    output: {
        <field>: {
            <$accumulator>: <expr>,
            window: {documents: [<lower>, <upper>]},  # optional
        },
    },
}

For each output field, the accumulator runs over the rows inside that row’s window — within the row’s partition, in the partition’s sorted order. Original input order is preserved in the result; the partition / sort dance is purely internal to compute the new fields.

Shipped (first-cut subset)

• The nine $group accumulators: $sum, $avg, $min, $max, $first, $last, $push, $addToSet, $count. The dispatch reuses _ACC_DISPATCH from $group — same per-doc accumulator semantics, just applied over a per-row windowed subset.

• Position-based windows via window: {documents: [<lower>, <upper>]}. Bound forms: integer offsets relative to the current row, "current" (= 0), and "unbounded" (partition edge).

• Default window (omit window) covers the whole partition. [unbounded, current] gives running-total semantics; [-1, 1] gives a 3-doc rolling window; etc.

• Empty-window output values: 0 for $sum/$count, [] for $push/$addToSet, null for the rest (matches mongod).

Deferred (raise AggregateError with a clear message)

• Range-based windows (window: {range: [...]}, optionally with unit: for date ranges). Needs value-based bounds + date arithmetic; out of scope for the first cut.

• Time-series functions: $derivative, $integral, $linearFill, $locf, $shift, $expMovingAvg. Each is its own slice and not in the common driver-test surface.

• Rank functions: $rank, $denseRank, $documentNumber. These need sort-key equality detection (tied rows get the same rank). Worth a dedicated slice when a workload needs them.

Added

• src/secantus/aggregate.py: _stage_set_window_fields handler

  • helpers _window_bounds (resolves documents: [<lower>, <upper>] to inclusive partition indices, with clamping to partition edges) and _empty_window_value (mongod-matching defaults). Wired into _STAGES. Reuses _ACC_DISPATCH + _finalize from $group so the accumulator semantics stay aligned across the two stages.

• tests/test_set_window_fields.py (15 new tests): no-partition totals; partitionBy splits totals correctly; rolling 3-doc sum with edge clamping; [unbounded, current] running total; [unbounded, unbounded] per-partition total; $avg / $min / $max / $first / $last over [-1, 1]; $count over [-1, 1]; $push / $addToSet accumulating across rows; sortBy controls running-total order independently of input order; original input order preserved on output; rank function raises; range window raises; missing output rejected; multiple accumulators in one output rejected; empty input → empty out.

0.5.1b24 — 2026-05-19

Geo: legacy $near sibling form, 2d quadtree covering, java gauge

Three geo improvements that close the long-standing tail of the phase 1/2 geo work and lift the mongo-java-driver gauge into the geo surface for the first time.

Legacy mongod 2d shape — {geo: {$near: [x, y], $maxDistance: r, $minDistance: r2}} with the distance bounds at sibling level rather than nested inside $near — now matches end-to-end through both the operator matcher and the 2d-index picker. This is exactly what mongo-java-driver’s Filters.near(field, x, y, max, min) and Filters.nearSphere(...) build. Unit conventions match mongod: legacy $near takes the bound in input units (planar Pythagoras); legacy $nearSphere takes radians on the unit sphere (picker converts to meters for 2dsphere and to degrees for 2d).

The 2d range scan picks tighter Z-order ranges via a quadtree decomposition of the bbox: each 2^k × 2^k power-of-2-aligned quadtree cell that lands fully inside the bbox emits one contiguous Z-range (the invariant that makes Z-order indexes work). Partial-overlap cells recurse; pure-outside cells are skipped. Falls back to the single coarse range if the decomposition would exceed max_ranges=32. Tightens the WT range scan on wider query polygons; correctness is unchanged (per-doc verifier filters false positives either way).

mongo-java-driver’s GeoJsonFiltersFunctionalSpecification and GeoFiltersFunctionalSpecification (driver-core functional) joined the java gauge include list and both pass 10/10. They exercise $geoWithin / $geoIntersects / $near / $nearSphere through the driver’s Filters builder against a real 2d and 2dsphere index — the kind of integration coverage neither the pymongo conformance gauge nor our in-tree pymongo tests reach.

Added

• secantus.geo_index.planar_2d_covering_ranges() — quadtree Z-order range decomposition for 2d index scans. Returns up to 32 tight (lo, hi) ranges; falls back to a single coarse range on cap overflow.

• 6 new tests in tests/test_geo_query.py / tests/test_geo.py: sibling-form $near with $maxDistance, sibling-form annulus (max+min), sibling-form $nearSphere with radians convention, single-range quadtree for an aligned bbox, multi-range quadtree for an off-axis bbox, fallback to single range under cap.

• _DRIVER_CORE_FUNCTIONAL_INCLUDES in java_validation/include_modules.py: brings the two upstream geo functional specs into the java gauge as :driver-core:test filtered runs.

• docs/geospatial.md — dedicated reference page: operator-by-operator, both index types, doc-side shapes accepted, the legacy / GeoJSON / spherical distance-unit conventions, a worked deployment example, validation surface summary. Linked from the Highlights list and added to the Sphinx toctree.

• docs/indexes.md — new geospatial section pointing at the dedicated page; the “Acceleration summary across index types” table now covers 2d, 2dsphere, and compound geo + scalar.

Changed

• _parse_near_spec now returns a 5-tuple (center, max_d, min_d, spherical, legacy_form); consumers use the new legacy_form flag to pick the right unit conversion (legacy+spherical → radians; legacy+planar → input units; GeoJSON → meters).

• 2d-index picker uses the multi-range coverer; existing single- range planar_2d_covering kept as the coarse fallback.

• docs/indexes.md — “What’s still missing” list rewritten. Multi-field sort acceleration, multikey indexing, and basic collation all shipped long ago and shouldn’t have been on the gap list; the actual remaining gaps (per-index collation, TTL background sweeper, text / hashed indexes) replace the stale entries.

• docs/production.md — added a paragraph on per-write writeConcern: {j: true} routing as the finer-grained alternative to the daemon-wide sync_on_commit = true knob.

Fixed

• Legacy mongod {geo: {$near: [x, y], $maxDistance: r}} previously raised unsupported query operator: $maxDistance because the dispatcher treated the sibling bound as a standalone operator. The matcher now skips the sibling keys when iterating and passes them into _op_geo_near.

• 2d-index picker no longer over-filters on $nearSphere legacy form: the radians bound is converted to degrees before building the planar disk, matching mongod’s behaviour against a 2d index.

0.5.1b23 — 2026-05-19

Native TLS + mTLS + per-write j:true — production gaps closed

Three slices land together against the production-readiness gaps called out in the docs/production.md page.

[tls] cert_file + [tls] key_file (in secantusdb.toml) or --tls-cert-file / --tls-key-file (CLI) makes the daemon wrap every accepted socket in TLS before the wire protocol starts. Clients connect with mongodb://host:port/?tls=true&tlsCAFile=<ca> and SecantusDB negotiates the TLS handshake itself; the connection thread then sees an encrypted socket-like object and serves mongo wire frames over it unchanged. This closes one of the biggest production-deployment gaps the docs/production.md page called out — operators no longer need to terminate TLS at an nginx / HAProxy / stunnel reverse proxy that becomes part of the trust boundary.

mTLS lands as a layer on top: set [tls] ca_file and the daemon asks connecting clients for their own X.509 cert during the TLS handshake, verifying it against the configured CA bundle. Set [tls] require_client_cert = true to reject clients that don’t present a cert; the default (false, CERT_OPTIONAL) verifies a cert if presented and accepts clients without one — useful for staged rollouts. mTLS is a coarse-grained “you’re someone we approved of” gate; SCRAM-SHA-256 still identifies the specific user on top. mongod’s MONGODB-X509 auth mechanism (cert-subject-DN as the username, no SCRAM step) is a separate follow-on slice.

Python’s PROTOCOL_TLS_SERVER (TLS 1.2+, no SSLv2/3 fallback, default cipher list) is the only protocol mode. The SSLContext is built once at startup and cached — hot cert rotation requires a daemon restart. certbot renew --post-hook 'systemctl reload secantusdb' is the standard pattern. Without the cert / key kwargs the daemon stays plaintext exactly as before — no regression risk for the 1300+ existing tests.

The b20 sync_on_commit knob enabled per-commit fsync at the connection level — every write on the daemon shared the same durability mode. The third slice finishes the story: the per-write writeConcern.j flag now threads from the wire layer through Storage.insert / update_matching / delete_matching (and all four findAndModify paths) into _batch_transaction(sync=True), which calls session.commit_transaction("sync=on"). A client can now mix j: true and j: false writes against one daemon: the j:true subset pays the per-commit fsync cost (closes the durability gap), the rest stays fast.

Added

• [tls] table in secantusdb.toml (cert_file, key_file, ca_file, require_client_cert). Half-configured TLS (only one of cert/key set) raises ValueError at startup so deployment mistakes can’t silently fall back to plaintext.

• --tls-cert-file / --tls-key-file / --tls-ca-file / --tls-require-client-cert CLI flags. Standard precedence: SecantusConfig defaults < TOML < explicit CLI.

• SecantusDBServer(tls_cert_file=..., tls_key_file=..., tls_ca_file=..., tls_require_client_cert=...) kwargs. When cert/key are set an ssl.SSLContext is built in __init__ and used to wrap accepted sockets in _serve_forever. When ca_file is also set, the context asks clients for an X.509 cert during the handshake and verifies it against that CA.

• tests/test_tls.py: 12 tests via trustme for ephemeral CA + client cert fixtures. Covers TLS round-trip, non-TLS-client rejection, no-args plaintext path (no regression), half-configured raises, missing-cert startup error, active_conns leak guard, and the four mTLS modes (required + valid cert / required + no cert / required + foreign-CA cert / optional + both modes).

• journal: bool = False kwarg on Storage.insert / update_matching / delete_matching. When True, the WT transaction commits with session.commit_transaction("sync=on") — forces a per-commit fsync of the log regardless of the connection’s transaction_sync config.

• _batch_transaction(*, sync: bool = False) context-manager kwarg. The per-commit-fsync escape hatch the new journal write kwargs route through.

• tests/test_write_concern_journal.py: 10 tests covering the storage-layer kwarg threading (_batch_transaction is invoked with sync=True/False appropriately), wire-level happy paths on insert / update / delete / findAndModify, and the positive + negative routing assertions.

Changed

• TLS / mTLS handshake errors are logged + the socket closed + the active-connection slot released; the daemon keeps serving everyone else.

• writeConcern: {j: true} is now honoured per-write: the wire layer extracts the flag and threads it through to _batch_transaction(sync=True). Previously the flag was accepted on the wire but had no effect — only the daemon-wide sync_on_commit knob (b20) could enable per-commit fsync.

• docs/production.md updated: “Native TLS” is no longer in the gaps list; the dedicated TLS section now shows the in-process config plus the mTLS opt-in instead of an nginx-stream-module example.

• docs/configuration.md documents the full [tls] schema (cert / key / ca / require_client_cert), the hot-rotation caveat, and the cipher-suite “out of scope for v1” note.

Dependencies

• trustme>=1.2 added to the dev extra for the test CA fixture (transitively pulls cryptography).

0.5.1b20 — 2026-05-19

secantusdb.toml config file, native checkpoint restore, j:true durability knob

Two production-shaping slices land together. A new secantusdb.toml configuration file exposes every CLI flag plus the WT and oplog knobs that were previously hard-coded — including cache_size (so you can size the engine for your dataset instead of running with the 1 GB test default) and a sync_on_commit switch that closes the long-standing writeConcern: {j: true} durability gap by enabling WT’s per-commit fsync. The loader auto-discovers ./secantusdb.toml, ~/.secantus/secantusdb.toml, and /etc/secantus/secantusdb.toml; an explicit --config PATH overrides the search. CLI flags still win over file values, so the file is a deployment baseline rather than a lock-in.

A new secantusAdmin.restoreArchive wire command and matching secantusdb-restore-archive offline CLI close out the backup story started in b18 — extract a backup .tar.gz into a target directory the operator then points a fresh SecantusDB process at. The admin UI’s per-row Restore button now adapts to backup type: mongodump directories still call mongorestore; native .tar.gz archives surface an inline target-dir field and an Extract action that hits the new endpoint. Restore intentionally doesn’t try to swap the WT home under a running server (the connection-thread session-caching layer would need a wholesale rework first), and matches how real mongod restore tooling already trains operators.

Drive-by fix: the admin UI’s “Existing backups” list now also includes .tar.gz files. The native archives created by the b18 backup button were previously invisible because list_backups only enumerated directories.

The new Running in production doc page ties the config-file, native-backup, and restore work together — honest comparison vs single-node Postgres (the more useful framing than “SecantusDB vs mongod”), the gaps you have to accept, and a concrete systemd / TLS / backup / monitoring deployment shape.

Added

• Running in production docs page — honest comparison vs single-node Postgres (the more useful framing than “SecantusDB vs mongod-for-prod”), the gaps you must accept (no native TLS, no PITR, no replication, beta maturity), and a concrete deployment shape: systemd unit, secantusdb.toml with sync_on_commit = true, SCRAM auth provisioning, nginx stream TLS termination, hourly native checkpoint backups with off-host sync, the restore drill, serverStatus scraping for Prometheus / Datadog, and capacity sizing notes for cache_size.

• secantusdb.toml configuration file (see Configuration for the full schema). Auto- discovered from ./secantusdb.toml, ~/.secantus/secantusdb.toml, /etc/secantus/secantusdb.toml; --config PATH disables discovery and loads a specific file. Unknown keys / unknown top-level tables fail loudly at startup so typos can’t silently leave the engine running on the hard-coded default.

• secantus.config.SecantusConfig dataclass + load_config() / apply_overrides() helpers. CLI flags’ argparse defaults are now None (the “user did not pass this” sentinel) so the precedence chain is SecantusConfig defaults < secantusdb.toml < explicit CLI flag — file is a per-deployment baseline, the CLI overrides for one-off runs.

• New CLI flags exposing previously-hard-coded knobs: --cache-size, --session-max, --sync-on-commit, --oplog-retention-seconds, --oplog-max-entries. Each has a matching [storage] / [oplog] key in the config file.

• Storage.__init__ accepts cache_size, session_max, sync_on_commit kwargs. The WT engine config string is built from these instead of being a hard-coded literal.

• secantusAdmin.restoreArchive wire command. Accepts archivePath (server-side path to .tar.gz), targetDir (extraction destination), and optional allowExisting (overlay into a non-empty dir). Returns {targetDir, fileCount, archive, ok: 1}. RBAC: fsync action, cluster scope.

• secantus.storage.extract_backup_archive(archive_path, target_dir, *, allow_existing=False) — module-level helper shared by the wire command, the admin route, and the CLI. Validates that the archive contains a WiredTiger metadata file before unpacking, so a malformed tarball can’t pollute the target.

• secantusdb-restore-archive console script (new [project.scripts] entry). Same validation as the wire command, no server needed.

• Admin UI per-row Extract action on .tar.gz rows, posting to POST /backup/restore-archive with editable target-dir form field; the existing Restore button still handles mongodump directories.

Changed

• writeConcern: {j: true} is now honourable end-to-end via [storage] sync_on_commit = true (or --sync-on-commit), which sets WT’s transaction_sync=(enabled=true,method=fsync). Closes the long-standing durability gap previously documented in the backlog. Off by default (matches mongod’s default {w:1, j:false}) since the throughput cost is significant.

• secantus.admin.backup.list_backups() now includes *.tar.gz files alongside directories. Native-archive backups produced by b18’s backup button were previously invisible in the admin UI’s “Existing backups” list.

• MongoFacade.restore_archive(archive_path, target_dir, *, allow_existing=False) — new admin client facade method.

Fixed

• “Existing backups” table on /backup was silently dropping every .tar.gz produced by the native checkpoint backup path introduced in v0.5.1b18 (only dump directories were listed). Both kinds now render with the correct per-row restore action.

0.5.1b18 — 2026-05-18

Native WT-checkpoint backups, admin UI /oplog page, and change-stream fidelity wins

The natural follow-on to v0.5.1b17’s local.oplog.rs synthetic collection lands as the admin UI /oplog page: a paged entry browser with a window selector (last 50 / 500 / 5000), op-checkbox filter (i / u / d / c / n), ns substring filter, and a per-row expandable JSON body. Auto-refreshes every 5 s. The data source is just client.local.oplog_rs.find() — no new server-side surface needed, only the page chrome and an _rows partial that follows the same pattern as /connections + /cursors.

showExpandedEvents on change streams now matches mongod: the flag defaults to false, and DDL “expanded” events (createIndexes, dropIndexes) are suppressed unless the user opts in via coll.watch(show_expanded_events=True). Previously these surfaced unconditionally — more permissive than mongod, and broke the conformance contract for tests that assume the stable v1 event set.

killOp lands as a real wire command that closes the target connection’s socket via shutdown(SHUT_RDWR). Any in-flight command finishes, the per-connection thread’s next recv returns 0, the loop exits, and the connection unregisters cleanly. Real mongod uses a per-op interrupt flag, which would need cancellation infrastructure SecantusDB doesn’t carry — but “close the socket” is the visible end-state users care about, and the kill-and-reap admin button on /connections is now functional.

$sample becomes deterministic when SECANTUS_SAMPLE_SEED=<n> is set in the environment. Builds a dedicated random.Random(seed) instance at module load instead of mutating the global random state, so other code sharing the process keeps its own entropy. Closes the long-standing test-flake source where $sample results varied run-to-run.

Added

• Admin UI /oplog page (routers/oplog.py + templates/pages/oplog.html + templates/partials/oplog_rows.html): window / op / ns filters, expandable per-row JSON, 5 s auto-refresh, sidebar entry between Profiler and Maintenance.

• killOp wire command + kill(conn_id) on ConnectionRegistry (shuts down the socket via shutdown(SHUT_RDWR)). Per-connection sockets are now stashed on the registry at _handle_client time.

• A_KILLOP privilege action in secantus.rbac; granted by clusterAdmin and root.

• Admin UI /connections Kill button (was a placeholder), typed-confirm modal (partials/connection_kill_modal.html), facade kill_connection(conn_id) method.

• ChangeStreamSpec.show_expanded_events parsed from $changeStream.showExpandedEvents; threaded into changestreams.project.

• SECANTUS_SAMPLE_SEED env var (read at aggregate module import) — $sample uses a dedicated random.Random(seed) when set.

• secantusAdmin.backupArchive wire command + Storage.create_archive

  • admin UI “Run native checkpoint backup” button: forces a WT checkpoint then tars the storage directory into a single .tar.gz. Faster + atomic vs mongodump; restore is “extract

  • start a new SecantusDB pointing at it”. Rigorous round-trip test coverage in tests/test_backup_restore.py (doc identity at scale, every non-default index shape, oplog tail continuity, capped collection options + FIFO state, SCRAM users / roles, concurrent-writes consistency, archive portability, repeated- backup idempotency).

• $densify month / quarter / year units via dateutil.relativedelta. quarter is canonically 3 months. Adds python-dateutil>=2.8 to the runtime dependencies (pure Python, available almost everywhere as a transitive dep).

Changed

• changestreams.project suppresses createIndexes / dropIndexes events unless the caller passed show_expanded_events=True (mongod-faithful default-off). The three existing tests + cross-driver DDL smokes (mongosh / node / go / java) all set the opt-in.

Fixed

• Closes backlog entry $sample uses random.sample without a fixed seed — deterministic via env var.

• Closes backlog entry killOp / connection-close command — admin UI Kill button is functional.

• Closes backlog entry showExpandedEvents — accepted, ignored.

• Closes backlog entry Admin UI /oplog page.

• updateDescription.truncatedArrays now emits for any array shrink (not just strict head-prefix), with indexed updatedFields for kept-prefix changes — matches mongod’s $v:2 in-place diff rather than wholesale-replacing on any reshape. Same-length-with- changes arrays also produce indexed arr.<i> updates now (previously wholesale). Closes the §3.2 backlog entry.

0.5.1b17 — 2026-05-17

local.oplog.rs queryable from pymongo, $merge pipeline form + $fill stage + $$var.path resolution

Real mongod exposes the oplog as a queryable collection at local.oplog.rs — pymongo clients can db.oplog.rs.find() against it the same way they would against any collection. Until this release, SecantusDB’s oplog was internal only: Storage.read_oplog / oplog_floor_seq / oplog_tail_seq were Python methods but had no wire surface. Now local.oplog.rs is a synthetic read-only view — list_collections("local") surfaces it, find / count / listCollections.options route to a reader that walks the oplog WT table directly, and write attempts (insert, update, delete, findAndModify, drop, create, createIndexes) refuse with code 13 (Unauthorized) like mongod does. The deferred admin UI /oplog page is unblocked as a follow-up; for now, debugging an in-flight change-stream pipeline is as simple as client.local.oplog_rs.find({"op": "u"}).sort("ts", -1).limit(20).

The aggregation expression library picks up two of the three remaining stages on most “more stages” wishlists. $merge was partly implemented; this batch fills in the rest: whenMatched: [<pipeline>] runs a sub-pipeline against the matched target doc with $$new bound to the source doc and any user let vars threaded through; whenMatched: "delete" (MongoDB 5.0+) removes the matched doc; a unique-index guard refuses non-_id on fields without a unique: true index covering them, matching mongod’s rule against silent on-field collapse.

$fill lands fresh — the 5.3+ stage for filling missing/null fields. Three modes per output field: {value: <expr>} replaces with an evaluated expression; {method: "locf"} carries the last observation forward within the partition’s sortBy order; {method: "linear"} interpolates between bracketing non-null anchors along the sortBy field (works for numbers and datetimes — timedelta arithmetic divides cleanly to float and multiplies back to timedelta). Partitioning via partitionByFields or partitionBy; sortBy required when any output uses method.

The $merge pipeline form was the first thing in the repo to exercise $$var.path (e.g. $$new.delta), and surfaced that the expression evaluator only did exact-name var lookup. Fixed in the same batch: $$var.field.path now walks the dotted path into the resolved value across $$ROOT.f / $$CURRENT.f / user-let vars.

Added

• local.oplog.rs synthetic collection: queryable via find / count / listCollections. Walks the existing oplog WT table via a private session for cross-thread visibility. list_databases surfaces local whenever the oplog is enabled.

• $merge whenMatched: [<pipeline>] with $$new binding + let clause for user-defined vars (aggregate._stage_merge).

• $merge whenMatched: "delete" (MongoDB 5.0+).

• $merge unique-index guard on non-_id on fields.

• $fill stage with value, locf, and linear modes (aggregate._stage_fill).

• $$var.field.path dotted-path resolution in expressions._resolve_var.

• docs/changelog.md as the system of record (see the changelog itself and the changelog/ Python package that generates blog posts from it).

Changed

• Writes to local.oplog.rs (insert / update / delete / findAndModify / drop / create / createIndexes) refuse with code 13 (Unauthorized).

• $merge validates whenMatched / whenNotMatched against the allowed string sets — typos surface as AggregateError instead of silently falling through to the default merge.

0.5.1b16 — 2026-05-16

Sidebar grouping, auto-refreshing connections and cursors, Roles in the nav

The /connections and /cursors admin pages have always been live-data views — they read currentOp and render the connection / cursor list each time the page is requested — but they didn’t refresh. The dashboard polls 1 Hz over a WebSocket; these two felt stale next to it. v0.5.1b16 extracts each table’s tbody into an HTMX partial and lets the tbody itself swap every 5 s via hx-trigger="every 5s" hx-get="/connections/_rows". The page chrome and column headers stay fixed; only the rows refresh. /connections also gains a (disabled) Actions column with a tooltip explaining that connection-kill is deferred until SecantusDB grows killOp — purely a layout-symmetry fix so the page mirrors the shape of /cursors.

The sidebar gets two structural fixes. A Roles entry now lives directly under Users with a sub-nav indent (it was reachable only via the breadcrumb on /users, and roles.html was setting active="users" so the wrong sidebar item highlighted while you were on the page). A second visual separator above Change stream marks the boundary between per-target data pages and operational-state pages, mirroring the existing separator below Server.

A separate fix: 15 *_via_mongosh cross-driver smoke tests are now grouped into a single xdist worker. Mongosh launches a full Node-based shell, and under heavy parallel load the PBKDF2 work inside SCRAM-SHA-256 auth could blow past mongosh’s connect timeout. Tagged with @pytest.mark.xdist_group(name="mongosh_smokes") so they serialize.

Added

• Sidebar Roles entry under Users with sub-nav indent + correct active highlight on /roles.

• Sidebar visual separator above the operational-state group (nav-ops-start CSS class).

• /connections + /cursors auto-refresh tbody (hx-trigger="every 5s") with new _rows partial endpoints.

• /connections disabled Actions column for layout symmetry with /cursors; tooltip explains killOp is deferred.

Fixed

• 15 *_via_mongosh cross-driver smoke tests serialized via xdist_group="mongosh_smokes" to dodge PBKDF2-handshake timeouts under parallel-test CPU contention.

0.5.1b15 — 2026-05-16

One scaffold for every confirmation modal — escape, focus-trap, restored focus

The secantus-admin UI has nine confirmation / edit modals (drop-database, drop-collection, drop-index, drop-user, change-password, manage-roles, edit-document, delete-document, kill-cursor). They were assembled at slightly different times and drifted in five different ways — different destructive-button copy, different typed-confirm targets (the delete-document modal asked the user to type the collection name shared by every row; the kill-cursor modal asked for the giant int cursor id), no Escape-to-close, no focus restoration to the trigger element, no focus trap so Tab leaked back into the page behind, and aria-label="Close" only on two of nine close buttons.

v0.5.1b15 consolidates all nine on a shared scaffold: a new modal-shell.js exposes openModal(url) / closeModal() / setupModal(el) plus a global htmx hook that captures the trigger element so closeModal() can restore focus. Each modal partial has the same overlay shape — x-init="setupModal($el)", @click.self="closeModal()", @keydown.escape.window="closeModal()", role="dialog", aria-modal, aria-labelledby — and Tab / Shift+Tab cycle within the modal’s focusable children rather than escaping into the page behind.

Three substantive fixes ride along with the scaffolding: destructive button copy now always restates action+noun (Kill cursor / Delete document / Drop index / Drop user / Drop database / Drop collection); the delete-document typed-confirm asks for the doc’s _id value rather than the collection name; the kill-cursor typed-confirm asks for the collection ns rather than the unguessable cursor id. None of these change SecantusDB’s wire-protocol behaviour.

Added

• static/js/modal-shell.js: openModal(url), closeModal(), setupModal(el), htmx hook for trigger-element capture.

• [x-cloak] CSS helper to prevent Alpine flash on first paint.

Changed

• All 9 confirmation / edit modal partials use the shared overlay shape with role="dialog" / aria-modal / aria-labelledby.

• Destructive button copy restates action+noun across the board.

• delete-document typed-confirm uses the doc’s _id value (was the collection name).

• kill-cursor typed-confirm uses the collection ns (was the cursor id).

Fixed

• Escape now closes every modal.

• Focus restored to the triggering element after modal close.

• Tab focus-trap inside modals.

• aria-label="Close" on all 9 close buttons (was on 2).

0.5.1b14 — 2026-05-15

Admin UI punch list — five silent-failure modes fixed

The May 2026 end-to-end review of the secantus-admin web UI catalogued five P0s — bugs that didn’t crash anything but presented wrong information to the user. v0.5.1b14 fixes all five. None require any database-level change; this is purely admin-UI plumbing, but each one was either lying to the user or hiding a real error behind cheerful copy.

The biggest was the profiler page swallowing every exception while reading system.profile. A bare except Exception: rendered “no entries yet — run an operation to see one appear here” no matter what the underlying error was, including the target server being completely unreachable. The clause is now narrowed to PyMongoError and the friendly error message gets funnelled into the page’s normal error banner. The same page also had a flash keyword argument that the template never rendered — every settings change returned HX-Redirect and the user saw zero confirmation that anything had happened. The POST handler now re-renders the page inline with a flash banner that names the new level / slowms / sampleRate values.

The other three are dead-code cleanups: the doc tour in docs/admin.md walked the user through a /console page that was renamed to /query two refactors ago; the Maintenance “Drop collection” form had an hx-get pointing at a route that never existed; and the dashboard router still exposed a GET /_partials/dashboard-tiles endpoint from before the WebSocket dashboard landed.

Fixed

• Profiler page: narrowed bare except Exception: to PyMongoError so server-down errors surface (routers/profiler.py).

• Profiler page: added flash banner block to template + POST handler re-renders inline instead of HX-Redirect.

• Maintenance “Drop collection” form: dropped dead hx-get="/maintenance/drop-collection-redirect" attribute.

• Dashboard router: deleted unused GET /_partials/dashboard-tiles endpoint, partial template, and the two tests that exercised them.

• docs/admin.md: replaced stale ### Console section with ### Query (/query) + ### Insert (/insert) + new ### Server (/server) subsection.

0.5.1b13 — 2026-05-15

Zero actionable failures — every driver gauge classified, every gap explained

Over the past few releases the cross-driver gauge pass rate has been climbing — 99.5% at v0.5.1b4, 99.9% by last week’s refresh. The last 0.1% was a handful of failures that either could not be fixed in SecantusDB (a Java-driver SDAM cascade triggered by a server-side APIStrictError), reproduced only under heavy parallel load (two mongo-go-driver flakes), or assumed a multi-node replica-set deployment SecantusDB deliberately doesn’t simulate (Ruby’s w: 2 write-concern test). Reporting them as plain “failures” overstated the gap — but silently dropping them would let real regressions hide in the same column.

v0.5.1b13 introduces validation_summary/expected_failures.py — a small per-gauge registry of (pattern, rationale) entries. The cross-driver summary now separates “Failed” (unexpected, a real bug we need to fix) from “Expected” (a documented gap with a one-line reason that ships in the report). A new Adjusted column reports the rate excluding expected failures from the denominator — “how much of the conformable surface actually conforms.” Current numbers: 7,186 tests, 6,254 passed, 0 unexpected failures, 5 expected failures, 927 skipped — 100.0% adjusted across every driver.

This release also bundles the gauge improvements that landed since v0.5.1b4: mapReduce returns a graceful empty result for non-canonical bodies, $changeStream against a standalone topology is rejected with code 40573, Node CSOT explain-plus-timeoutMS tests pass via a new block_connection / block_time_ms failpoint pair, getParameter advertises authenticationMechanisms: ["SCRAM-SHA-256"], and createIndexes / create reject unknown options up-front.

Added

• validation_summary/expected_failures.py: per-gauge registry of documented-known failures with rationales.

• Cross-driver summary “Expected” + “Adjusted pass rate” columns.

• block_connection / block_time_ms failpoint fields (failpoints._FailCommand).

Changed

• mapReduce returns a graceful empty result for non-canonical map/reduce bodies (wire-shape probes pass).

• $changeStream on a standalone topology is rejected with code 40573.

• getParameter advertises authenticationMechanisms: ["SCRAM-SHA-256"].

• createIndexes rejects unknown per-index options (_INDEX_SPEC_KNOWN_OPTIONS whitelist).

• create rejects unknown collection options (_CREATE_KNOWN_OPTIONS whitelist).

• validate-all serialized (max_workers=1) to dodge load-induced inter-gauge flakes.

0.5.1b4 — 2026-05-12

Cross-driver conformance summary — 99.5% across 7,186 tests on one page

Until this release, comparing SecantusDB’s conformance across the five driver gauges (pymongo / mongo-java-driver / mongo-go-driver / mongo-node-driver / mongo-ruby-driver) required opening five different reports and squinting at five different per-category breakdowns whose denominators came from incompatible units of count — JUnit <testcase> versus Mocha test versus RSpec example versus go test event versus pytest item.

v0.5.1b4 ships docs/validation-summary.md — a single table that normalises on test count, one row per gauge, the same five columns across the board: tests run, passed, failed, skipped, pass rate. A new validation_summary Python module reads each gauge’s raw artifact under .validation/ directly and renders the table; a new invoke validate-summary task refreshes it.

Current numbers: 7,186 tests, 6,232 passed, 33 failed, 921 skipped — 99.5% pass rate across all five drivers. Java is biggest by raw count (4,710 tests, 4,242 passed); Node smallest (364).

This release also rolls up two driver-gauge fixes that landed since v0.5.1b1: a Java widening to 21 of 112 driver-sync functional classes (+34 passes), and a snapshot-read-concern rejection that turned three SessionsTest snapshot-error scenarios from “expected error, got success” into “expected error, got SnapshotUnavailable (code 246)”.

Added

• docs/validation-summary.md cross-driver normalized table.

• validation_summary/ Python module (raw-artifact reader + renderer).

• invoke validate-summary task.

• snapshot readConcern rejected with code 246 (SnapshotUnavailable).

• Java gauge: ChangeStreamsTest, UnifiedWriteConcernTest, VersionedApiTest unified-spec runners (21 of 112 driver-sync functional classes total).

Fixed

• RTD build for v0.5.1b3 failed on a missing toctree entry for the new summary file; b4 is the first release where the docs match what’s on PyPI.

0.5.1b1 — 2026-05-12

Java gauge scope made honest — 18 of 112 driver-sync classes, five named follow-ups

The Java gauge passing rate had been reported at “100%” — but only across the 13 driver-sync functional classes the gauge was running. v0.5.1b1 widens the include set to 18 of 112 and adds an explicit Scope section to docs/validation-report-java.md that surfaces the “X of 112 driver-sync functional classes” denominator so the headline number isn’t misleading.

The widened set surfaced five real failures, all named and tracked in tasks/backlog.md §5: Java apiStrict pool-clear cascade, mapReduce non-canonical bodies, snapshot reads on standalone, distinct apiStrict — none are SecantusDB bugs, but they’re now documented expected-fail entries.

Added

• Java gauge include set widened to 18 of 112 driver-sync functional classes (java_validation/include_modules.py waves 1 + 2).

• “Scope” section in Java validation report exposing the include-set denominator (java_validation/generate_report.py).

0.5.0b18 — 2026-05-12

Ruby gauge climbs to 99%, completing the cross-driver 99–100% band

The Ruby gauge had been the weakest of the five at ~95% — a handful of real SecantusDB gaps the Ruby driver exercises but the others don’t. v0.5.0b18 closes the high-value ones: writeConcernError is now attached on w > 1 (CannotSatisfyWriteConcern code 100), invalid wildcardProjection is rejected on createIndexes, commitQuorum is validated at the top level, listIndexes rejects negative batchSize (code 51024), and $collStats surfaces capped-collection bounds (storageStats.{capped, max, maxSize}).

Net: Ruby gauge from 94.6% → 99.7%, 13 net passes. All five driver gauges now sit in the 99–100% band.

Added

• writeConcernError attached on w > 1 (CannotSatisfyWriteConcern code 100).

• createIndexes validates wildcardProjection shape.

• commitQuorum validated at top-level.

• $collStats surfaces capped bounds (storageStats.{capped, max, maxSize}).

Changed

• listIndexes rejects negative batchSize with code 51024.

Older releases

Releases before v0.5.0b18 (the v0.3.0aN and v0.4.0bN lines, and v0.5.0b1 through v0.5.0b3) shipped before this changelog was the system of record. See the GitHub Releases page for the auto-generated commit-list notes from those tags.