Cross-Driver Conformance Summary

Generated 2026-10-15 — SecantusDB 0.7.0b3. Each per-driver gauge runs the driver vendor’s own integration test suite (unmodified) against a SecantusDB daemon and emits its raw output to .validation/. This summary normalises on test count so the 13 gauges compare like for like — every row counts one assertion outcome, whether it landed as a JUnit <testcase>, a Mocha test, an RSpec example, a go test event, or a pytest collected item.

Failures split into two columns: Failed counts tests that actually need a fix on SecantusDB; Expected counts tests with a documented reason for failing (driver-side cascade, out-of-scope feature, single-node-topology assumption, known intermittent flake). The expected list lives in validation_summary/expected_failures.py and each entry carries a rationale. Adjusted pass rate = passes ÷ (passes + actual failures).

Summary by driver

Driver	Language	Driver version	Tests run	Passed	Failed	Expected	Skipped	Pass rate	Adjusted
pymongo	Python	f2103a95870a	1501	1206	0	5	290	99.5%	100.0%
pymongo (async)	Python	f2103a95870a	1423	1111	0	6	306	99.4%	100.0%
mongo-java-driver	Java	cb45be6bb147	900	496	0	0	404	100.0%	100.0%
mongo-kotlin-driver	Kotlin	cb45be6bb147	538	340	0	0	198	100.0%	100.0%
mongo-go-driver	Go	fd85a834c40e	659	598	12	0	49	98.0%	98.0%
mongo-node-driver	Node.js	7e53685952f2	364	357	0	1	6	99.7%	100.0%
mongo-ruby-driver	Ruby	f68d676643c1	283	258	0	1	24	99.6%	100.0%
mongo-rust-driver	Rust	12dd49bf18bb	105	104	1	0	0	99.0%	99.0%
mongo-php-library	PHP	12e56461166d	2221	2197	1	0	23	99.9%	99.9%
mongo-php-driver	PHP	06be1f01bb4c	271	254	0	0	17	100.0%	100.0%
mongo-c-driver	C	57dba9c04991	841	790	1	2	48	99.6%	99.8%
mongo-cxx-driver	C++	24852b68a3d1	901	892	0	0	9	100.0%	100.0%
mongo-csharp-driver	C#	8297e62d7f2b	228	228	0	0	0	100.0%	100.0%
All drivers	—	—	10235	8831	15	15	1374	99.6%	99.8%

Per-driver scope

• pymongo — curated server-touching pytest paths under vendor/pymongo-tests/test/.

• pymongo (async) — AsyncMongoClient suite under vendor/pymongo-tests/test/asynchronous/.

• mongo-java-driver — driver-sync functional integration tests.

• mongo-kotlin-driver — driver-kotlin-sync integrationTest (ships in the mongo-java-driver monorepo).

• mongo-go-driver — vendor/mongo-go-driver/internal/integration/….

• mongo-node-driver — curated test/integration/ spec set.

• mongo-ruby-driver — curated spec/mongo/*.rb spec files.

• mongo-rust-driver — curated driver/src/test/ in-tree tests.

• mongo-php-library — curated functional tests (Operation / Collection / Database / Command).

• mongo-php-driver — curated .phpt wire-protocol tests (bson serialization units excluded).

• mongo-c-driver — curated test-libmongoc wire-protocol suites (CRUD / cursor / aggregate / command).

• mongo-cxx-driver — curated mongocxx test_driver Catch2 suite (CRUD / cursor / aggregate / gridfs).

• mongo-csharp-driver — curated MongoDB.Driver.Tests CRUD specification suite (xUnit).

Expected failures

These tests fail for documented reasons that have no SecantusDB-side fix (driver-internal behaviour we can’t influence, features intentionally out of scope, single-node topology assumptions in tests that assume a 3-node replica set, etc.). Each entry has a rationale in validation_summary/expected_failures.py. If you fix one of these gaps, delete its entry there.

pymongo (5)

• vendor/pymongo-tests/test/test_collection.py::TestCollection::test_index_hashed — Hashed indexes are intentionally out of scope per CLAUDE.md. createIndexes rejects them explicitly with CannotCreateIndex (67) ‘hashed indexes are not supported by SecantusDB’ — a faithful not-supported error, which is the documented preference over a half-implemented index type.

• vendor/pymongo-tests/test/test_collection.py::TestCollection::test_index_text — Text indexes are intentionally out of scope per CLAUDE.md (same gap as the node gauge’s text-search test). createIndexes rejects them with CannotCreateIndex (67) ‘text indexes are not supported by SecantusDB’.

• vendor/pymongo-tests/test/test_cursor.py::TestCursor::test_maxtime_ms_message — Blocked by $where, not by the behaviour under test. The test builds a deliberately slow query with find({'$where': delay(...)}) and asserts the resulting timeout error names the configured timeouts. SecantusDB rejects $where up front (BadValue 2), so the command fails before any timeout can elapse. NOTE: this leaves the maxTimeMS error-message shape unverified by this gauge rather than known-good — it is untested here, not proven correct.

• vendor/pymongo-tests/test/test_cursor.py::TestCursor::test_to_list_csot_applied — Blocked by $where, same as test_maxtime_ms_message: the test delays the query with find({'$where': delay(1)}) and asserts the raised error carries .timeout == True. $where is rejected up front, so the error is a BadValue rather than a timeout. NOTE: CSOT behaviour is therefore unverified by this gauge, not confirmed.

• vendor/pymongo-tests/test/test_cursor.py::TestCursor::test_where — $where runs server-side JavaScript and SecantusDB ships no JS runtime, so it is rejected with BadValue (2) ‘unsupported top-level operator: $where’. Out of scope per tasks/backlog.md §4 — supporting it would mean embedding a JS engine as mongod does.

pymongo (async) (6)

• vendor/pymongo-tests/test/asynchronous/test_collection.py::AsyncTestCollection::test_index_hashed — Hashed indexes are intentionally out of scope per CLAUDE.md. createIndexes rejects them explicitly with CannotCreateIndex (67) ‘hashed indexes are not supported by SecantusDB’ — a faithful not-supported error, which is the documented preference over a half-implemented index type.

• vendor/pymongo-tests/test/asynchronous/test_collection.py::AsyncTestCollection::test_index_text — Text indexes are intentionally out of scope per CLAUDE.md (same gap as the node gauge’s text-search test). createIndexes rejects them with CannotCreateIndex (67) ‘text indexes are not supported by SecantusDB’.

• vendor/pymongo-tests/test/asynchronous/test_cursor.py::TestCursor::test_maxtime_ms_message — Blocked by $where, not by the behaviour under test. The test builds a deliberately slow query with find({'$where': delay(...)}) and asserts the resulting timeout error names the configured timeouts. SecantusDB rejects $where up front (BadValue 2), so the command fails before any timeout can elapse. NOTE: this leaves the maxTimeMS error-message shape unverified by this gauge rather than known-good — it is untested here, not proven correct.

• vendor/pymongo-tests/test/asynchronous/test_cursor.py::TestCursor::test_to_list_csot_applied — Blocked by $where, same as test_maxtime_ms_message: the test delays the query with find({'$where': delay(1)}) and asserts the raised error carries .timeout == True. $where is rejected up front, so the error is a BadValue rather than a timeout. NOTE: CSOT behaviour is therefore unverified by this gauge, not confirmed.

• vendor/pymongo-tests/test/asynchronous/test_cursor.py::TestCursor::test_where — $where runs server-side JavaScript and SecantusDB ships no JS runtime, so it is rejected with BadValue (2) ‘unsupported top-level operator: $where’. Out of scope per tasks/backlog.md §4 — supporting it would mean embedding a JS engine as mongod does.

• vendor/pymongo-tests/test/asynchronous/test_read_preferences.py::TestMongosAndReadPreference::test_read_preference_hedge_deprecated — Async-only, and never reaches the wire: the test constructs PrimaryPreferred(hedge={'enabled': True}) and asserts a DeprecationWarning is raised by the driver’s own constructor. Purely client-side pymongo behaviour, dependent on the ambient warning filters — no server can influence the outcome.

mongo-node-driver (1)

• Find should correctly sort using text search in find — Text indexes ($text, $meta: textScore, text-index creation) are intentionally out of scope per CLAUDE.md — would require a full-text index implementation. Documented in tasks/backlog.md §4.

mongo-ruby-driver (1)

• Mongo::Collection#create when the collection has options when the collection has a write concern when write concern passed in as an option applies the write concern passed in as an option — The test passes w: 2 and expects success — it assumes the canonical multi-node replica-set test cluster the Ruby driver’s own CI runs against. SecantusDB advertises as a single-node replica set, so w: 2 returns CannotSatisfyWriteConcern (the correct mongod emulation). Documented in tasks/backlog.md §5.

mongo-c-driver (2)

• /Client/ipv6/single — Requires an IPv6 listener (MONGOC_TEST_IPV6); the gauge daemon binds IPv4 127.0.0.1 only. Environment-specific, not a protocol gap.

• /Client/ipv6/single — Requires an IPv6 listener (MONGOC_TEST_IPV6); the gauge daemon binds IPv4 127.0.0.1 only. Environment-specific, not a protocol gap.

Per-driver reports

Each gauge ships its own detailed report — per-category breakdown, named failures for triage, and the gauge’s own setup notes. Open the one whose pass / fail counts you want to dig into:

• pymongo

• mongo-java-driver

• mongo-go-driver

• mongo-node-driver

• mongo-ruby-driver

• mongo-rust-driver

• mongo-php-library

• mongo-php-driver

• mongo-c-driver

• mongo-cxx-driver

• mongo-csharp-driver

Refreshing

Run all 13 gauges plus this summary:

uv run python -m invoke validate-all
uv run python -m invoke validate-summary

Run a single gauge (still updates that one report) plus the summary:

uv run python -m invoke validate-java       # or validate / validate-go / etc.
uv run python -m invoke validate-summary

The summary reads whatever is currently in .validation/; a gauge that’s never been run is silently omitted from the table.