#35713 Remove boost as a unit test runner

ReviewabilityReviewsAgreementSize
#35713 rustaceanrob Remove boost as a unit test runnerReady0Mild3 + 1,433 tests
  • Eliminate external Boost.Test dependencies and packaging overhead for unit tests
  • Provide clearer failure diagnostics for test authors via automatic expression decomposition
  • Make unit tests easier to run, maintain, and integrate with development tooling
  • Ready for review
  • The latest push addressing string formatting and macros is ready for reviewer feedback
  • Concept approval because an in-house runner lowers test barriers and enables CI concurrency (josibake)
  • Concept approval after verifying the test suite passes and scripted diffs reproduce (jeanpablojp)
  • Answered objection regarding static assertions for unprintable optionals awaits reviewer reply (jeanpablojp)
  • Open question on whether failure-path tests for the framework belong in a follow-up (jeanpablojp)
  • 3 lines added or modified outside tests
  • 1,433 lines added or modified in tests
  • 218 lines removed in total
  • 167 files, 9 commits
Build and CI#24 of 56
P3 · cleanup
  • P3 because it reduces packaging friction by dropping Boost.Test from depends and Guix
  • Simplifies dependency management and improves tooling integration with clang-tidy and IWYU
  • It is deferrable maintenance rather than a fix for broken builds or release integrity
Ranking notes
Test infrastructure#7 of 54
P2 · cleanup
  • P2 because replacing Boost.Test with an in-tree runner fulfills a long-standing project goal
  • Expression decomposition automatically reports operand values during assertion failures
  • An in-house framework removes friction for test authors and enables future test concurrency
Ranking notes
Reviewability: ReadyStalePaused
Agreement: StrongPositivePositive w/ caveatsNeutralMildDisputedBlockedCrickets
Reviews: current code-review ACKs, then (+stale ACKs) and -NACKs; greener = more ACKs.
Size: lines added or modified outside tests, then in tests; greener = smaller.

Goal

Replaces the Boost.Test library with a custom in-tree C++ unit test runner and assertion framework in src/test/util/framework.{h,cpp}. The new runner introduces expression decomposition to display operand values automatically upon assertion failures, while preserving legacy BOOST_* macro shims to prevent merge conflicts. Boost.Test is removed from depends, Guix, vcpkg, and CMake build configuration.

Problem: Boost.Test introduces external build and packaging complexity across depends and Guix, interferes with tooling such as IWYU and clang-tidy, and has suffered from platform bugs like undersized signal stacks. Furthermore, its variety of assertion macros (BOOST_CHECK, BOOST_CHECK_EQUAL, BOOST_TEST) handle operand value reporting inconsistently, complicating test debugging.

Evidence of importance:

  • Replacing Boost Test has been an open objective discussed in RFC #34666 and issue #8670.
  • Author notes practical issues with Boost.Test signal stack sizing previously addressed in #36045.
  • Author notes Boost.Test complicates depends and Guix packaging and hinders clang-tidy/IWYU integration across tests.

Scope: The vast majority of the 167 modified files are mechanical single-line include changes applied via a scripted diff. The primary substantive logic is in the newly introduced framework files (src/test/util/framework.{h,cpp}, totaling ~1,240 lines).

Risk of a mistake: low (build, tests). Bugs in the custom test framework could produce false passes or mask test failures in CI.

Changed since the previous assessment: Added commits to resolve CI failures, including inlining CRecipient construction to bypass a GCC 12 uninitialized variable warning, addressing a clang-tidy warning in miniminer_tests, and removing rarely used macros like BOOST_FAIL and BOOST_TEST_INFO.

Reviewability

Ready

Nothing known that would invalidate a review now.

Waiting on (from the thread read):

  • reviewer: review of the push simplifying stringify to CHECK_NO_DISPLAY and punting optional static_assert handling 2026-10-16
  • author: answer question on failure path unit tests for framework.h

Agreement

Mild

objection that needs a change was answered, no reply from the objector (jeanpablojp: Comparing two optional<T> without operator<< bypasses static_assert and prints type names instead of failing compilation)

Thread notes: Author pushed an update migrating BOOST_CHECK/BOOST_CHECK_MESSAGE to CHECK_NO_DISPLAY to simplify stringify, removing mangled type name printing, and replied punting the optional static_assert handling to macro migration.

Objections, suggestions, and questions:

ReviewerTypeHarm, request, or questionStatusMust beAuthor repliedQuote
maflcko (member)objection, approachSilently printing mangled or compile-time type names at runtime instead of failing compilation when formatting is missing loses compile-time safety and differs from master.fixed · fix pushed · author added static_assert and CHECK_NO_DISPLAY macrochangedyes2026-07-13: 'This should remain a compile failure, like it is on current master.'
Status from: 2026-07-16: maflcko: 'CHECK_NO_DISPLAY sounds fine. The benefit of the static assert is that it can print the problem and solution at compile'
fanquake (member)question, approachHow could Boost library missing instrumentation cause issues when Boost has been header-only since #24301?answered · author verified CI and dropped the commitansweredyes2026-07-14: 'not sure how the library missing instrumentation could be the cause of issues here?'
Status from: 2026-07-14: rustaceanrob: 'Looks like an older state of the framwork or commit history was causing this, dropped the commit.'
maflcko (member)suggestion, maintenanceDrop the unused boost workaround in guix.fixed · fix pushed · author removed the unused Boost workaround in guixchangedno2026-07-30: 'Could rebase and drop the unused boost workaround in guix?'
Status from: 2026-07-30: maflcko: 'Could rebase and drop the unused boost workaround in guix?'
jeanpablojp (contributor)objection, correctnessDropping BOOST_TEST_RUN_FILTERS causes test coverage scripts to run the whole test suite instead of specific suites.fixed · fix pushed · author re-added filter supportchangedyes2026-08-30: 'BOOST_TEST_RUN_FILTERS is no longer read, so deterministic-unittest-coverage runs the whole binary instead of the suite it was given.'
Status from: 2026-09-10: rustaceanrob: 'Fixed'
jeanpablojp (contributor)objection, correctnessDropping Boost.Test flags causes downstream CI scripts like libmultiprocess to fail immediately.fixed · fix pushed · author added compatibility flags and separated summary from log levelchangedyes2026-08-30: 'bitcoin_core_ci.sh in the libmultiprocess subtree still passes three Boost.Test flags, and any one of them now makes it exit with code 2 without running anything.'
Status from: 2026-09-10: rustaceanrob: 'I added a catch_system_error flag for compatibility as well as a separation between the summary and log level.'
jeanpablojp (contributor)question, scopeShould unit tests covering failure paths in framework.h be added or left for a follow-up?openansweredno2026-08-30: 'That's 1096 new lines in framework.h and I couldn't find anything covering their failure paths. util_check_tests covers both paths for Assert. Would something similar work here, or is it worth a follo'
jeanpablojp (contributor)objection, correctnessComparing null C-strings segfaults rather than reporting a test failure.fixed · fix pushed · author added null check before strcmpchangedyes2026-08-30: 'With either side null this segfaults instead of reporting the failure.'
Status from: 2026-09-10: rustaceanrob: 'Added a check if either side is null so it does not go through strcmp'
jeanpablojp (contributor)objection, correctnessUnflushed output causes test logs before a crash to be lost when output is redirected.fixed · fix pushed · author added cout flushchangedyes2026-08-30: 'This never flushes. With a crash inside a test case and output redirected, master keeps the preceding lines and here the file comes back empty.'
Status from: 2026-09-10: rustaceanrob: 'I think a cout flush should solve these. Updated in latest push.'
jeanpablojp (contributor)objection, correctnessUnflushed output causes duplicate output on child process exits when tests fork.fixed · fix pushed · author added cout flushchangedyes2026-08-30: 'The same buffer causes a second, smaller thing. test_LockDirectory forks and the child exits with exit(0), reprinting what it inherited, so the output comes out duplicated.'
Status from: 2026-09-10: rustaceanrob: 'I think a cout flush should solve these. Updated in latest push.'
jeanpablojp (contributor)objection, approachDropping the entire guix linker block drops -Wl,--fatal-warnings from all Linux builds instead of only dropping the riscv64 exception.fixed · fix pushed · author re-added -Wl,--fatal-warningschangedyes2026-08-30: 'Dropping the whole block means no Linux host links with -Wl,--fatal-warnings any more, while build_win.sh and build_win_gui.sh keep it.'
Status from: 2026-09-10: rustaceanrob: 'Added Wl, --fatal-warnings on head commit, lmk if thats what you mean'
jeanpablojp (contributor)objection, correctnessBOOST_TEST_RANDOM is treated as a boolean without parsing the seed value or logging the used seed, making test run ordering non-reproducible.fixed · fix pushed · author fixed seed handling and printingchangedyes2026-08-30: 'Here the value is only checked for presence, never read, so 0 and an empty string both turn shuffling on and a fixed seed no longer reproduces.'
Status from: 2026-09-10: rustaceanrob: 'Fixed by using a random seed and printing it if equal to 1, and setting the seed directly if > 1.'
jeanpablojp (contributor)objection, correctnessComparing two optional<T> without operator<< bypasses static_assert and prints type names instead of failing compilation.answered · author removed mangled type names and punted optional static_assert handling to macro migrationchangedyes2026-08-30: 'This overload wins over the primary by partial ordering, so the static_assert never reaches std::optional.'
Status from: 2026-10-16: rustaceanrob: 'I think we can punt on this for the time being and address this during migration of the macros'
jeanpablojp (contributor)objection, correctnessTEST_CASE inside a TEST_SUITE does not run the suite fixture, deviating from Boost and the PR description.fixed · fix pushed · author updated TEST_CASE to use suite fixturechangedyes2026-08-30: 'TEST_SUITE_BEGIN(suite, Fixture) sets btc_suite_fixture, but TEST_CASE registers a plain function and never uses it.'
Status from: 2026-09-10: rustaceanrob: 'Seems fine. I would think it very strange to update a global as part of a test fixture, but this is a code reduction and'
maflcko (member)suggestion, approachSeparate stringify for optional truthiness checks from stringify for optional value comparison checks.answered · author agreed with the ideaansweredyes2026-10-14: 'would it be possible to have a `stingify_optional_truthy` which is called on truty checks and a separate `stingify_optional`, which is called on value cmp checks?'
Status from: 2026-10-14: rustaceanrob: 'Yeah I was thinking something along those lines'

Support:

  • josibake (member) (Concept ACK, 2026-07-13) [read]: Having an in-house customizable test framework removes barriers to writing tests and enables concurrency improvements to speed up CI.
  • jeanpablojp (contributor) (Concept ACK, 2026-08-30) [both]: The whole suite passes and the scripted-diff reproduces cleanly.

Participants:

  • josibake (member, support, 1 statements 2026-07-13 to 2026-07-13): Strong Concept ACK endorsing in-house customizable test runner
  • maflcko (member, objection, 17 statements 2026-07-13 to 2026-10-14): objected to fallback printing type names; suggested dropping guix workaround and handling optional stringification
  • fanquake (member, question, 2 statements 2026-07-14 to 2026-07-22): asked why Boost library instrumentation was relevant given Boost has been header-only
  • jeanpablojp (contributor, objection, 7 statements 2026-08-30 to 2026-08-30): Concept ACK, but raised multiple behavioral regressions and bug reports

Dependencies

Enables:

  • Incremental migration of unit tests from legacy BOOST_* macros to native CHECK, REQUIRE, and TEST_CASE macros.

Files

1632 lines under test/bench/ci.

  • src/test/util/framework.h +783/-0
  • src/test/util/framework.cpp +458/-0
  • src/test/main.cpp +9/-18
  • cmake/module/AddBoostIfNeeded.cmake +0/-12
  • src/test/baseindex_tests.cpp +5/-5
  • src/wallet/test/spend_tests.cpp +6/-4
  • src/test/util/CMakeLists.txt +9/-0
  • contrib/guix/libexec/build_linux.sh +1/-7
  • src/test/descriptor_tests.cpp +3/-5
  • src/test/miniminer_tests.cpp +3/-3
  • src/test/kernel/test_kernel.cpp +1/-4
  • vcpkg.json +1/-4
  • src/test/result_tests.cpp +3/-1
  • src/test/system_tests.cpp +2/-2
  • depends/packages/boost.mk +1/-2
  • src/ipc/test/ipc_tests.cpp +1/-1
  • src/test/addrman_tests.cpp +1/-1
  • src/test/allocator_tests.cpp +1/-1
  • src/test/amount_tests.cpp +1/-1
  • src/test/argsman_tests.cpp +1/-1
  • src/test/arith_uint256_tests.cpp +1/-1
  • src/test/banman_tests.cpp +1/-1
  • src/test/base32_tests.cpp +1/-1
  • src/test/base58_tests.cpp +1/-1
  • src/test/base64_tests.cpp +1/-1
  • src/test/bech32_tests.cpp +1/-1
  • src/test/bip324_tests.cpp +1/-1
  • src/test/bip328_tests.cpp +1/-1
  • src/test/bip32_tests.cpp +1/-1
  • src/test/bip352_tests.cpp +1/-1
  • src/test/blockchain_tests.cpp +1/-1
  • src/test/blockencodings_tests.cpp +1/-1
  • src/test/blockfilter_index_tests.cpp +1/-1
  • src/test/blockfilter_tests.cpp +1/-1
  • src/test/blockmanager_tests.cpp +1/-1
  • src/test/blockpolicyestimator_tests.cpp +1/-1
  • src/test/bloom_tests.cpp +1/-1
  • src/test/bswap_tests.cpp +1/-1
  • src/test/btcsignals_tests.cpp +1/-1
  • src/test/caches_tests.cpp +1/-1
  • src/test/chain_tests.cpp +1/-1
  • src/test/chainstate_write_tests.cpp +1/-1
  • src/test/checkqueue_tests.cpp +1/-1
  • src/test/cluster_linearize_tests.cpp +1/-1
  • src/test/coins_tests.cpp +1/-1
  • src/test/coinscachepair_tests.cpp +1/-1
  • src/test/coinstatsindex_tests.cpp +1/-1
  • src/test/coinsviewoverlay_tests.cpp +1/-1
  • src/test/common_url_tests.cpp +1/-1
  • src/test/compress_tests.cpp +1/-1
  • src/test/crypto_tests.cpp +1/-1
  • src/test/cuckoocache_tests.cpp +1/-1
  • src/test/dbwrapper_tests.cpp +1/-1
  • src/test/denialofservice_tests.cpp +1/-1
  • src/test/disconnected_transactions.cpp +1/-1
  • src/test/feefrac_tests.cpp +1/-1
  • src/test/feerounder_tests.cpp +1/-1
  • src/test/fees_util_tests.cpp +1/-1
  • src/test/flatfile_tests.cpp +1/-1
  • src/test/fs_tests.cpp +1/-1
  • src/test/getarg_tests.cpp +1/-1
  • src/test/hash_tests.cpp +1/-1
  • src/test/headers_sync_chainwork_tests.cpp +1/-1
  • src/test/httpserver_tests.cpp +1/-1
  • src/test/i2p_tests.cpp +1/-1
  • src/test/interfaces_tests.cpp +1/-1
  • src/test/key_io_tests.cpp +1/-1
  • src/test/key_tests.cpp +1/-1
  • src/test/logging_tests.cpp +1/-1
  • src/test/mempool_fee_estimator_tests.cpp +1/-1
  • src/test/mempool_tests.cpp +1/-1
  • src/test/merkle_tests.cpp +1/-1
  • src/test/merkleblock_tests.cpp +1/-1
  • src/test/miner_tests.cpp +1/-1
  • src/test/miniscript_tests.cpp +1/-1
  • src/test/minisketch_tests.cpp +1/-1
  • src/test/multisig_tests.cpp +1/-1
  • src/test/net_peer_connection_tests.cpp +1/-1
  • src/test/net_peer_eviction_tests.cpp +1/-1
  • src/test/net_tests.cpp +1/-1
  • src/test/netbase_tests.cpp +1/-1
  • src/test/node_init_tests.cpp +1/-1
  • src/test/node_warnings_tests.cpp +1/-1
  • src/test/orphanage_tests.cpp +1/-1
  • src/test/pcp_tests.cpp +1/-1
  • src/test/peerman_tests.cpp +1/-1
  • src/test/pmt_tests.cpp +1/-1
  • src/test/pool_tests.cpp +1/-1
  • src/test/pow_tests.cpp +1/-1
  • src/test/prevector_tests.cpp +1/-1
  • src/test/private_broadcast_tests.cpp +1/-1
  • src/test/psbt_tests.cpp +1/-1
  • src/test/random_tests.cpp +1/-1
  • src/test/rbf_tests.cpp +1/-1
  • src/test/rest_tests.cpp +1/-1
  • src/test/reverselock_tests.cpp +1/-1
  • src/test/rpc_tests.cpp +1/-1
  • src/test/sanity_tests.cpp +1/-1
  • src/test/scheduler_tests.cpp +1/-1
  • src/test/script_assets_tests.cpp +1/-1
  • src/test/script_p2sh_tests.cpp +1/-1
  • src/test/script_parse_tests.cpp +1/-1
  • src/test/script_segwit_tests.cpp +1/-1
  • src/test/script_standard_tests.cpp +1/-1
  • src/test/script_tests.cpp +1/-1
  • src/test/scriptnum_tests.cpp +1/-1
  • src/test/serfloat_tests.cpp +1/-1
  • src/test/serialize_tests.cpp +1/-1
  • src/test/settings_tests.cpp +1/-1
  • src/test/sighash_tests.cpp +1/-1
  • src/test/sigopcount_tests.cpp +1/-1
  • src/test/skiplist_tests.cpp +1/-1
  • src/test/sock_tests.cpp +1/-1
  • src/test/span_tests.cpp +1/-1
  • src/test/streams_tests.cpp +1/-1
  • src/test/sync_tests.cpp +1/-1
  • src/test/system_ram_tests.cpp +1/-1
  • src/test/testnet4_miner_tests.cpp +1/-1
  • src/test/threadpool_tests.cpp +1/-1
  • src/test/timeoffsets_tests.cpp +1/-1
  • src/test/torcontrol_tests.cpp +1/-1
  • src/test/transaction_tests.cpp +1/-1
  • src/test/translation_tests.cpp +1/-1
  • src/test/txdownload_tests.cpp +1/-1
  • src/test/txgraph_tests.cpp +1/-1
  • src/test/txindex_tests.cpp +1/-1
  • src/test/txospenderindex_tests.cpp +1/-1
  • src/test/txpackage_tests.cpp +1/-1
  • src/test/txreconciliation_tests.cpp +1/-1
  • src/test/txrequest_tests.cpp +1/-1
  • src/test/txvalidation_tests.cpp +1/-1
  • src/test/txvalidationcache_tests.cpp +1/-1
  • src/test/uint256_tests.cpp +1/-1
  • src/test/util_check_tests.cpp +1/-1
  • src/test/util_expected_tests.cpp +1/-1
  • src/test/util_not_null_tests.cpp +1/-1
  • src/test/util_string_tests.cpp +1/-1
  • src/test/util_tests.cpp +1/-1
  • src/test/util_threadnames_tests.cpp +1/-1
  • src/test/util_trace_tests.cpp +1/-1
  • src/test/validation_block_tests.cpp +1/-1
  • src/test/validation_chainstate_tests.cpp +1/-1
  • src/test/validation_chainstatemanager_tests.cpp +1/-1
  • src/test/validation_flush_tests.cpp +1/-1
  • src/test/validation_tests.cpp +1/-1
  • src/test/validationinterface_tests.cpp +1/-1
  • src/test/versionbits_tests.cpp +1/-1
  • src/wallet/test/coinselection_tests.cpp +1/-1
  • src/wallet/test/coinselector_tests.cpp +1/-1
  • src/wallet/test/db_tests.cpp +1/-1
  • src/wallet/test/feebumper_tests.cpp +1/-1
  • src/wallet/test/group_outputs_tests.cpp +1/-1
  • src/wallet/test/init_tests.cpp +1/-1
  • src/wallet/test/ismine_tests.cpp +1/-1
  • src/wallet/test/psbt_wallet_tests.cpp +1/-1
  • src/wallet/test/rescan_tests.cpp +1/-1
  • src/wallet/test/scriptpubkeyman_tests.cpp +1/-1
  • src/wallet/test/wallet_crypto_tests.cpp +1/-1
  • src/wallet/test/wallet_interfaces_tests.cpp +1/-1
  • src/wallet/test/wallet_rpc_tests.cpp +1/-1
  • src/wallet/test/wallet_tests.cpp +1/-1
  • src/wallet/test/wallet_transaction_tests.cpp +1/-1
  • src/wallet/test/walletdb_tests.cpp +1/-1
  • src/wallet/test/walletload_tests.cpp +1/-1
  • test/lint/lint-includes.py +0/-2
  • src/test/CMakeLists.txt +1/-0
  • src/test/kernel/CMakeLists.txt +1/-0

Card

The short text the ranking pass compares across PRs.

This PR replaces Boost.Test with an in-tree test runner and assertion framework implemented in src/test/util/framework.{h,cpp}. Boost.Test adds packaging and build overhead across depends and Guix, conflicts with tools like clang-tidy and IWYU, and has inconsistent value reporting across assertion macros. Replacing the Boost test framework has been discussed across RFC #34666 and issue #8670, with signal stack workarounds required as recently as #36045. Most file changes are mechanical include updates handled by a scripted diff, while backward-compatible BOOST_* macro shims avoid disrupting concurrent pull requests. Boost.Test is completely dropped from CMake, Guix, depends, and vcpkg.

Data

dossier JSON · extract JSON · model ledger, generated 2026-10-29T01:22, confidence high, input hash 25d9f18d1f1cacb6
thread read through 2026-10-16 (52 statements), head 3f96c1b0ad
labels: none

Categories

Build and CI

P3 · cleanup

Removes Boost.Test from depends, Guix, and CMake packaging recipes, reducing build and tooling friction. While it simplifies dependency management and improves tooling integration like clang-tidy and IWYU, it is reasonably deferrable maintenance rather than a fix for broken builds or release integrity.

Membership: Removes Boost.Test configuration from cmake/module/AddBoostIfNeeded.cmake, depends/packages/boost.mk, vcpkg.json, and contrib/guix/libexec/build_linux.sh

Factors: security/stability 0, bug 0, performance 1, user value 0, leverage 2

Test infrastructure

P2 · cleanup

Replaces the external Boost.Test runner with an in-house assertion framework and test runner, realizing long-standing project goals from RFC #34666 and issue #8670. Reviewers highlight that having an in-house customizable framework removes barriers to writing tests, improves assertion failure output via expression decomposition, and enables future test concurrency.

Membership: Adds an in-tree unit test runner and assertion framework in src/test/util/framework.{h,cpp} and updates src/test/main.cpp

Factors: security/stability 0, bug 0, performance 1, user value 0, leverage 2