#35762 test: optionally run functional tests via CTest

full analysis

https://github.com/bitcoin/bitcoin/pull/35762 · willcl-ark · +449/-45 in 9 files, 6 commits · labels: Tests

Goal

  • Allow developers to run functional tests alongside unit tests using standard CTest commands
  • Enables test filtering, rerun flags, and native IDE integration for functional test workflows

This pull request adds support for discovering and executing Python functional tests through CTest alongside existing unit tests. It adds a `--dump-ctest` manifest export and `--ctest-direct` single-test execution mode to `test_runner.py`, implements CTest test discovery via CMake 4.4's `discover_tests()` with a backward-compatibility shim for older CMake versions, and enables this runner in the macOS native CI job.

Problem: Bitcoin Core's functional tests currently require running `test_runner.py` directly, separating them from CTest unit tests and preventing developers from using standard CTest features such as `-R`, `-L`, `--rerun-failed`, and CTest-aware IDE integrations.

Category: Build and CI (#31 of 55)

P3 · new feature

  • P3 because it standardizes test invocation under CMake without affecting release integrity
  • Improves developer build ergonomics rather than fixing platform build failures

Integrating functional tests into CTest standardizes how tests are registered and invoked across the CMake build system. It is developer-facing ergonomics work that does not touch release integrity or fix platform breakages, making it P3.

Membership: Adds CMake functional test registration logic, discovery templates, and updates macOS native CI configuration.

Factors: security/stability 0, bug 0, performance 0, user value 1, leverage 1

Category: Test infrastructure (#25 of 45)

P3 · new feature

  • P3 because it improves test runner convenience rather than adding critical test coverage
  • Enhances test execution workflows locally without addressing test suite reliability

Allows developers to execute functional tests using standard CTest flags like -R, -L, and --rerun-failed, and surfaces tests to CTest-aware IDEs. While valuable for testing workflows, it does not fix test flakiness or add missing coverage to critical areas, placing it in P3.

Membership: Modifies test_runner.py to provide CTest manifest generation and single-test direct execution mode.

Factors: security/stability 0, bug 0, performance 0, user value 2, leverage 1

Reviewability: Ready

  • Ready to review: CI is passing and the branch is up to date with feedback

The branch is clean, CI passes, and the author addressed recent review feedback.

Author status: Active; addressed reviewer nits and force-pushed updates on 2026-08-17.

Open concerns:

  • maflcko questioned the 500 lines of added logic complexity versus an alternative in Python, noting that CTest does not work for win64 cross-testing.
  • maflcko noted that maintaining both CTest and legacy test runner workflows in CI can be confusing and overlapping.

Resolved concerns:

  • Author removed the need to pass `--configfile` by invoking the generated test runner copy in the build directory as suggested by maflcko.
  • Author added logic to fail explicitly when `TEST_RUNNER_EXTRA` is supplied under CTest mode so options are not silently ignored.

Agreement: Mild

  • No approvals yet with reservations about the added complexity (maflcko)
  • Questioned whether adding CMake orchestration is justified over a Python alternative (maflcko)
  • Drawback: CTest execution does not support win64 cross-testing workflows (maflcko)

Mild: maflcko questioned added complexity and noted win64 cross-testing limitations; no ACKs.

Only one reviewer has commented, raising questions about whether 500 lines of CMake orchestration logic is justified given Python alternatives and win64 cross-testing constraints.

  • maflcko: 'This pull is adding 500 LOC of logic complexity, so my preference would be to at least keep the overhead low. Recall the alternative to this would be to write it in a few trivial lines of Python'
  • maflcko: 'Also, recall that Python would work when testing win64-cross builds, but CTest does not.'

Review verdicts (DrahtBot): 0

Files

475 lines under test/bench/ci.

  • test/functional/test_runner.py +137/-30
  • test/functional/functional_discovery.cmake.in +96/-0
  • test/functional/CMakeLists.txt +87/-0
  • test/README.md +58/-0
  • ci/test/03_test_script.sh +42/-14
  • CMakeLists.txt +18/-1
  • ci/test/00_setup_env_mac_native.sh +4/-0
  • test/CMakeLists.txt +4/-0
  • ci/test/00_setup_env.sh +3/-0

Card

This PR integrates Python functional tests into CTest, adding manifest export and direct-execution modes to test_runner.py along with CMake discovery rules and a macOS CI demonstration. It allows developers to run unit and functional tests through a uniform CTest interface and leverage standard CTest filtering and IDE tools. It offers worthwhile workflow ergonomics but does not address test suite flakiness or missing test coverage. maflcko raised concerns regarding 500 lines of added logic complexity versus Python alternatives and noted that CTest does not support win64 cross-testing. The PR is ready for review with no open dependency blockers.

Data

dossier JSON · extract JSON · model openrouter/google/gemini-3.8-flash, generated 2026-09-17T16:20, confidence high, input hash 6b7f0160b987c23e