#35713 Remove boost as a unit test runner
https://github.com/bitcoin/bitcoin/pull/35713 · · +1696/-264 in 163 files, 12 commits · labels: none
Goal
- Eliminate Boost.Test to improve failure diagnostics and reduce build maintenance overhead for developers
- Unblock future test runner concurrency and tailored testing tools for the project
This pull request replaces the Boost.Test unit test framework with an in-house, header-only test runner in `src/test/util/framework.h`. It migrates all unit test files to the new framework headers while providing backwards-compatible macro aliases, introduces assertion expression decomposition for improved failure diagnostics, and removes the Boost.Test library from depends and vcpkg.
Problem: Boost.Test requires managing an external dependency with awkward workarounds, exhibits inconsistent failure diagnostics across different assertion macros (`BOOST_CHECK`, `BOOST_CHECK_EQUAL`, `BOOST_TEST`), and lacks tailoring for Bitcoin Core-specific testing requirements such as concurrency.
Category: Build and CI (#45 of 55)
P4 · cleanup
- P3 because it simplifies build system scripts and depends configurations
- It removes maintenance quirks without affecting production builds or binary release integrity
P4 because removing the Boost.Test component from depends and vcpkg is a minor dependency cleanup that does not remove Boost completely (Boost Multi-Index is retained).
Membership: Modifies depends/packages/boost.mk, vcpkg.json, and contrib/guix/libexec/build_linux.sh to drop the Boost.Test package and its build workarounds.
Factors: security/stability 0, bug 0, performance 0, user value 0, leverage 1
Category: Test infrastructure (#5 of 45)
P3 · cleanup
- P2 because it fixes chronic developer pain with test assertion macros and diagnostics
- It unblocks future test infrastructure enhancements including harness parallelization
P3 because replacing the test runner provides clear maintainability value and cleaner assertions across test files, but is reasonably deferrable as it does not fix an active test breakage or immediately speed up CI.
Membership: Introduces a complete custom test runner in src/test/util/framework.h and migrates the entire unit test suite away from Boost.Test.
Factors: security/stability 0, bug 0, performance 0, user value 0, leverage 2
Reviewability: Ready
- Ready to review
- CI is passing and recent pushes have resolved prior review comments
The PR is rebased and mergeable, CI is passing, and the author addressed all concrete issues raised in the latest review.
Author status: active, addressed reviewer feedback and force-pushed updates
Resolved concerns:
- maflcko noted that comparing unsupported types should remain a compile-time static assertion rather than falling back to printing runtime type names; resolved by adding a static assertion and a `CHECK_NO_DISPLAY` opt-out macro.
- jeanpablojp identified several regressions including segfaults on null C-strings, lost test output during crashes due to missing `std::cout` flushes, non-reproducible test ordering with `BOOST_TEST_RANDOM`, and dropping `-Wl,--fatal-warnings` from Linux Guix builds; resolved by author pushes on 2026-09-10.
Agreement: Strong
- Strong sentiment favoring the removal of Boost.Test for maintainability
- Strong support citing customizability and future concurrent test execution needs (josibake)
- Concept approval with extensive testing of runner flags, signals, and seed reproduction (jeanpablojp)
- Feedback on stringification fallbacks and static assertions incorporated cleanly (maflcko)
Strong concept agreement from multiple reviewers; all technical review objections have been addressed.
There is substantive Concept ACK support from josibake and jeanpablojp. Technical concerns raised by maflcko and jeanpablojp regarding string formatting, null checks, buffering, and random seeds were promptly addressed by the author.
- josibake provided a Strong Concept ACK endorsing in-house test framework customization.
- jeanpablojp gave a Concept ACK while providing rigorous technical feedback.
- Author resolved all identified behavior gaps in subsequent pushes.
Objections:
| Reviewer | Kind | Harm | Status | Blocking | Author replied | Quote |
|---|---|---|---|---|---|---|
| maflcko | approach | silently printing mangled type names instead of failing compilation when formatting is missing | resolved | no | yes | 2026-07-13: 'This should remain a compile failure, like it is on current master.' Settled: 2026-07-20: Author pushed commit adding static_assert: '05397f7c0881265f1b108e8e2544c889d7c8b830 moves to a static_assert' |
| jeanpablojp | correctness | segfaulting if either side of a string comparison is null | resolved | no | yes | 2026-08-30: 'With either side null this segfaults instead of reporting the failure.' Settled: 2026-09-10: 'Added a check if either side is null so it does not go through strcmp' |
Support:
- josibake: Having our own simple, in house test framework gives us the ability to customise it for exactly our needs, removing barriers to writing tests and unblocking concurrency improvements
- jeanpablojp: Confirmed test suite passes and scripted-diff reproduces correctly
Participants: josibake (support), maflcko (objection), fanquake (neutral), jeanpablojp (objection)
State derived from the lists: substantive support, no open objection (josibake, jeanpablojp)
Review verdicts (DrahtBot): 0
- Concept ACK: josibake, jeanpablojp
Dependencies
Enables:
- Future concurrency in unit testing runner to accelerate CI
Files
1939 lines under test/bench/ci.
- src/test/util/framework.h +1209/-0
- src/test/util/common.cpp +146/-0
- src/test/util/common.h +55/-0
- src/test/kernel/test_kernel.cpp +36/-2
- src/test/main.cpp +10/-17
- src/test/validation_chainstatemanager_tests.cpp +11/-11
- cmake/module/AddBoostIfNeeded.cmake +0/-12
- src/test/dbwrapper_tests.cpp +11/-1
- src/test/descriptor_tests.cpp +5/-7
- src/test/serialize_tests.cpp +6/-6
- src/test/baseindex_tests.cpp +5/-5
- src/test/miniscript_tests.cpp +5/-5
- src/test/skiplist_tests.cpp +5/-5
- src/test/sock_tests.cpp +5/-5
- src/wallet/test/spend_tests.cpp +6/-4
- contrib/guix/libexec/build_linux.sh +1/-7
- src/test/feefrac_tests.cpp +4/-4
- src/wallet/test/wallet_interfaces_tests.cpp +6/-1
- src/test/addrman_tests.cpp +3/-3
- src/test/arith_uint256_tests.cpp +3/-3
- src/test/bip324_tests.cpp +3/-3
- src/test/blockpolicyestimator_tests.cpp +3/-3
- src/test/mempool_fee_estimator_tests.cpp +3/-3
- src/test/private_broadcast_tests.cpp +3/-3
- src/test/script_tests.cpp +3/-3
- vcpkg.json +1/-4
- src/test/argsman_tests.cpp +2/-2
- src/test/cluster_linearize_tests.cpp +2/-2
- src/test/result_tests.cpp +3/-1
- src/test/sighash_tests.cpp +2/-2
- src/test/system_tests.cpp +2/-2
- src/test/transaction_tests.cpp +2/-2
- src/test/txdownload_tests.cpp +2/-2
- src/test/txgraph_tests.cpp +2/-2
- src/test/txvalidationcache_tests.cpp +2/-2
- src/test/versionbits_tests.cpp +2/-2
- src/wallet/test/coinselector_tests.cpp +2/-2
- depends/packages/boost.mk +1/-1
- src/ipc/test/ipc_tests.cpp +1/-1
- src/test/allocator_tests.cpp +1/-1
- src/test/amount_tests.cpp +1/-1
- src/test/banman_tests.cpp +1/-1
- src/test/base32_tests.cpp +1/-1
- src/test/base58_tests.cpp +1/-1
- src/test/base64_tests.cpp +1/-1
- src/test/bech32_tests.cpp +1/-1
- src/test/bip328_tests.cpp +1/-1
- src/test/bip32_tests.cpp +1/-1
- src/test/blockchain_tests.cpp +1/-1
- src/test/blockencodings_tests.cpp +1/-1
- src/test/blockfilter_index_tests.cpp +1/-1
- src/test/blockfilter_tests.cpp +1/-1
- src/test/blockmanager_tests.cpp +1/-1
- src/test/bloom_tests.cpp +1/-1
- src/test/bswap_tests.cpp +1/-1
- src/test/btcsignals_tests.cpp +1/-1
- src/test/caches_tests.cpp +1/-1
- src/test/chain_tests.cpp +1/-1
- src/test/chainstate_write_tests.cpp +1/-1
- src/test/checkqueue_tests.cpp +1/-1
- src/test/coins_tests.cpp +1/-1
- src/test/coinscachepair_tests.cpp +1/-1
- src/test/coinstatsindex_tests.cpp +1/-1
- src/test/coinsviewoverlay_tests.cpp +1/-1
- src/test/common_url_tests.cpp +1/-1
- src/test/compress_tests.cpp +1/-1
- src/test/crypto_tests.cpp +1/-1
- src/test/cuckoocache_tests.cpp +1/-1
- src/test/denialofservice_tests.cpp +1/-1
- src/test/disconnected_transactions.cpp +1/-1
- src/test/feerounder_tests.cpp +1/-1
- src/test/fees_util_tests.cpp +1/-1
- src/test/flatfile_tests.cpp +1/-1
- src/test/fs_tests.cpp +1/-1
- src/test/getarg_tests.cpp +1/-1
- src/test/hash_tests.cpp +1/-1
- src/test/headers_sync_chainwork_tests.cpp +1/-1
- src/test/httpserver_tests.cpp +1/-1
- src/test/i2p_tests.cpp +1/-1
- src/test/interfaces_tests.cpp +1/-1
- src/test/key_io_tests.cpp +1/-1
- src/test/key_tests.cpp +1/-1
- src/test/logging_tests.cpp +1/-1
- src/test/mempool_tests.cpp +1/-1
- src/test/merkle_tests.cpp +1/-1
- src/test/merkleblock_tests.cpp +1/-1
- src/test/miner_tests.cpp +1/-1
- src/test/miniminer_tests.cpp +1/-1
- src/test/minisketch_tests.cpp +1/-1
- src/test/multisig_tests.cpp +1/-1
- src/test/net_peer_connection_tests.cpp +1/-1
- src/test/net_peer_eviction_tests.cpp +1/-1
- src/test/net_tests.cpp +1/-1
- src/test/netbase_tests.cpp +1/-1
- src/test/node_init_tests.cpp +1/-1
- src/test/node_warnings_tests.cpp +1/-1
- src/test/orphanage_tests.cpp +1/-1
- src/test/pcp_tests.cpp +1/-1
- src/test/peerman_tests.cpp +1/-1
- src/test/pmt_tests.cpp +1/-1
- src/test/pool_tests.cpp +1/-1
- src/test/pow_tests.cpp +1/-1
- src/test/prevector_tests.cpp +1/-1
- src/test/psbt_tests.cpp +1/-1
- src/test/random_tests.cpp +1/-1
- src/test/rbf_tests.cpp +1/-1
- src/test/rest_tests.cpp +1/-1
- src/test/reverselock_tests.cpp +1/-1
- src/test/rpc_tests.cpp +1/-1
- src/test/sanity_tests.cpp +1/-1
- src/test/scheduler_tests.cpp +1/-1
- src/test/script_assets_tests.cpp +1/-1
- src/test/script_p2sh_tests.cpp +1/-1
- src/test/script_parse_tests.cpp +1/-1
- src/test/script_segwit_tests.cpp +1/-1
- src/test/script_standard_tests.cpp +1/-1
- src/test/scriptnum_tests.cpp +1/-1
- src/test/serfloat_tests.cpp +1/-1
- src/test/settings_tests.cpp +1/-1
- src/test/sigopcount_tests.cpp +1/-1
- src/test/span_tests.cpp +1/-1
- src/test/streams_tests.cpp +1/-1
- src/test/sync_tests.cpp +1/-1
- src/test/system_ram_tests.cpp +1/-1
- src/test/testnet4_miner_tests.cpp +1/-1
- src/test/threadpool_tests.cpp +1/-1
- src/test/timeoffsets_tests.cpp +1/-1
- src/test/torcontrol_tests.cpp +1/-1
- src/test/translation_tests.cpp +1/-1
- src/test/txindex_tests.cpp +1/-1
- src/test/txospenderindex_tests.cpp +1/-1
- src/test/txpackage_tests.cpp +1/-1
- src/test/txreconciliation_tests.cpp +1/-1
- src/test/txrequest_tests.cpp +1/-1
- src/test/txvalidation_tests.cpp +1/-1
- src/test/uint256_tests.cpp +1/-1
- src/test/util_check_tests.cpp +1/-1
- src/test/util_expected_tests.cpp +1/-1
- src/test/util_string_tests.cpp +1/-1
- src/test/util_tests.cpp +1/-1
- src/test/util_threadnames_tests.cpp +1/-1
- src/test/util_trace_tests.cpp +1/-1
- src/test/validation_block_tests.cpp +1/-1
- src/test/validation_chainstate_tests.cpp +1/-1
- src/test/validation_flush_tests.cpp +1/-1
- src/test/validation_tests.cpp +1/-1
- src/test/validationinterface_tests.cpp +1/-1
- src/wallet/test/coinselection_tests.cpp +1/-1
- src/wallet/test/db_tests.cpp +1/-1
- src/wallet/test/feebumper_tests.cpp +1/-1
- src/wallet/test/group_outputs_tests.cpp +1/-1
- src/wallet/test/init_tests.cpp +1/-1
- src/wallet/test/ismine_tests.cpp +1/-1
- src/wallet/test/psbt_wallet_tests.cpp +1/-1
- src/wallet/test/scriptpubkeyman_tests.cpp +1/-1
- src/wallet/test/wallet_crypto_tests.cpp +1/-1
- src/wallet/test/wallet_rpc_tests.cpp +1/-1
- src/wallet/test/wallet_tests.cpp +1/-1
- src/wallet/test/wallet_transaction_tests.cpp +1/-1
- src/wallet/test/walletdb_tests.cpp +1/-1
- src/wallet/test/walletload_tests.cpp +1/-1
- test/lint/lint-includes.py +0/-2
- src/test/util/CMakeLists.txt +1/-0
Card
This pull request removes Boost.Test across the unit test suite and introduces an in-house header-only testing framework in src/test/util/framework.h. It solves long-standing issues with Boost test assertion consistency, provides cleaner expression decomposition on test failures, and drops Boost.Test from depends. There is strong conceptual support from reviewers, and critical issues regarding output flushing, null string checks, and random test order reproduction have been resolved. The PR is currently ready for review without blocking dependencies.