#35762 test: optionally run functional tests via CTest

ReviewabilityReviewsAgreementSize
#35762 willcl-ark test: optionally run functional tests via CTestWaiting on author1Strong14 + 233 tests
  • Run Python functional tests directly through CTest alongside unit tests
  • Enable fine-grained test filtering, parallel scheduling, and CDash dashboard reporting for developers
  • Ready to review now
  • Needs rebase due to conflicts with master, but the rebase is mechanical
  • Strong support for CMake integration and fine-grained test control (BrandonOdiwuor)
  • Verified test discovery, execution, fixtures, and CDash submission on Ubuntu (BrandonOdiwuor)
  • Concerns about CMake complexity, CI duplication, and win64 cross testing answered (maflcko)
  • Unanswered suggestion to rebase on PR 35846 for improved error handling (maflcko)
  • 14 lines added or modified outside tests
  • 233 lines added or modified in tests
  • 36 lines removed in total
  • 8 files, 6 commits
Build and CI#41 of 56
P3 · new feature
  • P3 because it is an optional workflow enhancement requiring CMake 4.4 or newer
  • Benefits developers seeking CTest integration and CDash reporting rather than core build outputs
  • Ranking pass: Moved down from 31: opt-in developer workflow gated on CMake 4.4, affects nobody by default
Ranking notes
Test infrastructure#19 of 54
P3 · new feature
  • P3 because it expands runner execution options without changing test coverage or assertions
  • Helps developers schedule unit and functional suites together and prevents concurrent temp dir collisions
Ranking notes
Reviewability: ReadyStalePaused
Agreement: StrongPositivePositive w/ caveatsNeutralMildDisputedBlockedCrickets
Reviews: current code-review ACKs, then (+stale ACKs) and -NACKs; greener = more ACKs.
Size: lines added or modified outside tests, then in tests; greener = smaller.

Goal

Adds opt-in CMake registration to run Python functional tests directly through CTest when configured with -DBUILD_FUNCTIONAL_TESTS=ON (requiring CMake 4.4 or newer). It adds --dump-ctest and --ctest-direct modes to test_runner.py, models shared cache setup and teardown as CTest fixtures, and avoids directory collisions across concurrent variants with unique temporary paths. The macOS native CI job is updated to exercise this combined unit and functional CTest execution workflow.

Problem: Functional tests can only be orchestrated through test_runner.py rather than standard CTest commands. This prevents developers from scheduling unit and functional tests together, filtering tests or rerunning failures through standard CTest and IDE interfaces, and reporting individual test results and timings to dashboards like CDash.

Evidence of importance:

  • PR description: 'Scheduling unit and functional tests together may improve parallel execution. It also enables per-test results and timings to be uploaded to our Bitcoin Core CDash instance, helping track failures and runtime changes across builds.'
  • Commit 7d3bbcffaa: 'Native discovery makes functional cases available to CTest scheduling, filters and dashboards alongside unit tests.'
  • Commit f574198736 notes real-world collisions: 'Concurrent variants of the same script can observe the same wall-clock value on macOS and select an identical temporary directory. The second test then fails before framework setup completes.'

Scope: Requires CMake 4.4+ only when BUILD_FUNCTIONAL_TESTS=ON is set; builds with the option disabled keep the existing CMake minimum.

Risk of a mistake: low (tests, build). Mistakes are confined to test harness execution and the native macOS CI job, with no impact on production node binaries or consensus logic.

Changed since the previous assessment: Dropped the fallback discovery script for older CMake versions in favor of strictly requiring CMake 4.4+ when the option is enabled; switched temporary test directory generation to UUID suffixes to prevent collision races during concurrent execution; and set allow_abbrev=False on the test runner argument parser.

Reviewability

Paused · Waiting on author

Open objection from maflcko with no author reply for 7 days; review would repeat what has been asked.

Waiting on (from the thread read):

  • author: rebase suggestion on PR 35846 2026-08-17
  • reviewer: re-review from maflcko on addressed objections 2026-10-07

Agreement

Strong

substantive support, no open objection (BrandonOdiwuor)

Thread notes: Recorded a substantive Tested ACK from BrandonOdiwuor detailing thorough testing across test discovery, execution, cache fixture ordering, and CDash submission on Ubuntu 24.04.

Objections, suggestions, and questions:

ReviewerTypeHarm, request, or questionStatusMust beAuthor repliedQuote
maflcko (member)question, scopeWhether the argument filtering modification is an unrelated changeanswered · Author explained why the handling was needed for direct mode and noted a potential separate bugansweredyes2026-07-21: 'Unrelated change?'
Status from: 2026-07-23: willcl-ark: 'Not quite, although it was incorrect (thanks!)'
maflcko (member)suggestion, interfaceCall the test runner from the build directory instead of adding a new command-line optionfixed · fix pushed · Addressed in push by invoking the generated test_runner.py link/copy from the build directoryansweredyes2026-07-21: 'Why not follow the docs and call the test_runner from the build dir, which avoids having to pass the new option?'
Status from: 2026-10-07: willcl-ark: 'The added runner-level --configfile option is gone.'
maflcko (member)suggestion, maintenanceDrop the change or provide a rationale for resolving the source-tree path and setting PYTHON_GILfixed · fix pushed · Author removed the change in latest pushansweredyes2026-07-21: 'Could drop this, or add least add a short rationale for stuff like this?'
Status from: 2026-10-07: willcl-ark: 'now removed.'
maflcko (member)objection, maintenanceExposing CTest execution in CI creates redundant workflows and risks silently ignoring TEST_RUNNER_EXTRAanswered · Author dropped the duplicate selector in CI pushansweredyes2026-07-30: 'not sure about exposing this in CI. This will silently ignore `TEST_RUNNER_EXTRA`, and having two ways to achieve the same seems confusing anyway.'
Status from: 2026-10-07: willcl-ark: 'Also dropped the duplicate selector.'
maflcko (member)objection, approachAdds 500 lines of complex CMake logic when an alternative in Python would be far simpleranswered · Author reduced complexity by dropping support for older CMake versionsansweredyes2026-08-04: 'This pull is adding 500 LOC of logic complexity, so my preference would be to at least keep the overhead low. Recall the alternative to this would be to write it in a few trivial lines of Python'
Status from: 2026-10-07: willcl-ark: 'Agreed this change was getting too large. About half was supporting this new feature on old cmake versions, so I dropped'
maflcko (member)objection, approachCTest cannot run functional tests for win64 cross builds whereas a Python runner cananswered · Author acknowledged limitation and deferred win64 cross support to a follow-upansweredyes2026-08-04: 'Also, recall that Python would work when testing win64-cross builds, but CTest does not.'
Status from: 2026-10-07: willcl-ark: 'Windows cross remains a limitation for now. It can be made to work without too much tinkering, but I don't think that's '
maflcko (member)suggestion, maintenanceRebase on top of PR 35846 for better error message handlingopenansweredno2026-08-17: 'You may want to rebase this on top of https://github.com/bitcoin/bitcoin/pull/35846, for better error (message) handling'

Support:

  • BrandonOdiwuor (contributor) (Tested ACK, 2026-10-25) [tested: build and discovery, execution, cache fixture behavior, CDash submission]: An opt-in way to run the functional tests through CTest is valuable for developers integrating Bitcoin Core into CMake/CTest/CDash workflows, and for anyone who wants finer-grained control (filtering, parallel scheduling, fixtures, dashboard reporting) while replacing the existing `test_runner.py` u

Participants:

  • maflcko (member, objection, 8 statements 2026-07-21 to 2026-10-06): concerned about 500 LOC complexity vs Python, lack of win64 cross-test support with CTest, and CI duplication
  • BrandonOdiwuor (contributor, support, 1 statements 2026-10-25 to 2026-10-25): Tested ACK with extensive verification of discovery, execution, cache fixtures, and CDash submission on Ubuntu 24.04

Files

268 lines under test/bench/ci.

  • test/functional/test_runner.py +111/-33
  • test/functional/CMakeLists.txt +53/-0
  • test/README.md +47/-0
  • ci/test/03_test_script.sh +15/-2
  • CMakeLists.txt +14/-1
  • test/CMakeLists.txt +4/-0
  • ci/test/00_setup_env_mac_native.sh +2/-0
  • ci/test/00_setup_env.sh +1/-0

Card

The short text the ranking pass compares across PRs.

Adds opt-in CMake integration to register Python functional tests as individual CTest targets alongside unit tests when built with CMake 4.4 or newer. Functional tests previously required orchestration through test_runner.py, preventing developers from using standard CTest flags or IDE runners and blocking per-test reporting to dashboards like CDash. The change introduces test manifest emission and single-test execution modes in the runner, manages the shared test cache via CTest fixtures, and enables the workflow in the macOS native CI job. The existing test runner remains the default for standard local builds.

Data

dossier JSON · extract JSON · model ledger, generated 2026-10-26T01:20, confidence high, input hash c82736e72d7cfb33
thread read through 2026-10-25 (20 statements), head f574198736
labels: Tests, Needs rebase

Categories

Build and CI

P3 · new feature (ranking pass: Moved down from 31: opt-in developer workflow gated on CMake 4.4, affects nobody by default)

Enables opt-in CTest scheduling and reporting for functional tests within the CMake build system and CI. As noted in the description, this allows developers to schedule tests together and upload per-test results to CDash, though it remains an optional workflow enhancement requiring CMake 4.4+.

Membership: Adds CMake integration via CMakeLists.txt and updates CI scripts to test running functional tests via CTest.

Factors: security/stability 0, bug 0, performance 1, user value 1, leverage 1

Test infrastructure

P3 · new feature

Extends test framework capabilities by adding direct CTest execution modes and concurrent temp directory handling. As stated in the description, this allows 'scheduling unit and functional tests together' and reporting granular metrics to dashboards.

Membership: Has the Tests label and updates test_runner.py to support direct CTest discovery and execution.

Factors: security/stability 0, bug 0, performance 1, user value 1, leverage 1