{
 "number": 35762,
 "repo": "bitcoin/bitcoin",
 "url": "https://github.com/bitcoin/bitcoin/pull/35762",
 "title": "test: optionally run functional tests via CTest",
 "author": "willcl-ark",
 "author_association": "MEMBER",
 "created_at": "2026-07-21T11:04:45Z",
 "updated_at": "2026-09-04T23:13:05Z",
 "age_days": 58,
 "draft": false,
 "labels": [
  "Tests"
 ],
 "milestone": null,
 "base": "master",
 "head_sha": "d3cf1fa313df9f3aff3b28b70c1878accb586db1",
 "head_ref": "ctest-functional-v6",
 "head_repo": "willcl-ark/bitcoin",
 "head_history": [
  {
   "t": "2026-07-22T19:04:50Z",
   "sha": "16c4245b58308121ebef2446b9e841844f4b1981"
  },
  {
   "t": "2026-07-23T21:10:11Z",
   "sha": "121c9aeb1c6f8b2025d31ba0489d323f87cc45b4"
  },
  {
   "t": "2026-07-23T21:25:26Z",
   "sha": "b7ad3bff61c4e466ae6e16c605d6b0aef2e318e6"
  },
  {
   "t": "2026-08-04T10:19:59Z",
   "sha": "e05e5ad0f7016c5430da3c75685c7749e3b02de1"
  },
  {
   "t": "2026-08-17T09:40:48Z",
   "sha": "d3cf1fa313df9f3aff3b28b70c1878accb586db1"
  }
 ],
 "additions": 449,
 "deletions": 45,
 "changed_files": 9,
 "commit_count": 6,
 "size_bucket": "L",
 "mergeable_state": "clean",
 "bot": {
  "drahtbot": {
   "present": true,
   "reviews": {},
   "conflicts": [
    {
     "number": 35395,
     "title": "doc: Improve test suite dependencies documentation",
     "author": "hebasto"
    },
    {
     "number": 31349,
     "title": "ci: detect outbound internet traffic generated while running tests",
     "author": "vasild"
    }
   ]
  }
 },
 "acks_parsed": {},
 "acks_tally": {
  "ack": 0,
  "stale_ack": 0,
  "concept_ack": 0,
  "approach_ack": 0,
  "nack": 0,
  "concept_nack": 0,
  "approach_nack": 0
 },
 "reviews": {
  "approved": 0,
  "changes_requested": 0,
  "distinct_reviewers": [
   "maflcko"
  ]
 },
 "signals": {
  "needs_rebase": false,
  "ci_failed": false,
  "mergeable_state": "clean",
  "last_author_activity": "2026-08-17T09:40:48Z",
  "last_reviewer_activity": "2026-08-17T06:51:03Z",
  "last_reviewer": "maflcko",
  "author_silent_days": 31,
  "waiting_on_author_days": 0,
  "days_since_update": 12
 },
 "refs": {
  "mentioned": [],
  "depends_on": [],
  "fixes": [],
  "linked_issues": [],
  "references": [],
  "conflicts": [
   35395,
   31349
  ]
 },
 "stack": {
  "shares_commits_with": [],
  "based_on": [],
  "base_for": []
 },
 "review_paths": [
  "ci/test/00_setup_env.sh",
  "test/functional/test_runner.py"
 ],
 "body": "Running the functional tests via CTest lets CI and developers can use the same standard CTest interface to run all unit and enabled functional tests!\n\nIndividual tests become easy to inspect and rerun with standard CTest features such as `-R`, `-L`, `--rerun-failed`, `--output-on-failure`, and `-j`.\n\nThis also lets functional tests (and their output) become visible to CTest-aware IDEs, CTest dashboards, CDash, and all other downstream tooling that already understands CTest metadata.\n\nExercise the ctest invocation in a single CI job (native macos) to begin with and keep this from going stale.\n\n#### Design decisions\n\n- Functional tests become first-class CTest tests alongside the existing unit tests. CTest replaces `test_runner.py` as the top-level orchestrator, while `test_runner.py` remains (and will always need to remain as) the discovery adapter and direct-execution wrapper.\n- `test_runner.py --dump-ctest` emits a six-field, line-oriented manifest containing the test name, script, port seed, optional label, optional argument, and cost.\n  - The manifest is structured directly for native CMake 4.4 [`discover_tests()`](https://cmake.org/cmake/help/latest/command/discover_tests.html#command:discover_tests) capture groups.\n- CMake 4.4+ uses native discovery; older CMake versions use a compatibility shim with the same manifest format and a matching five-second discovery timeout.\n- Both paths execute tests through `test_runner.py --ctest-direct`.\n- Direct mode forwards CTest\u2019s exact `--portseed`, `--tmpdir`, `--cachedir`, and per-test arguments; preserves skip status 77; and matches the legacy runner by treating exit status 0 with nonempty stderr as failure.\n- The combined log renderer is extracted into a shared runner function so direct CTest execution can reuse the normal test runner's combined-log rendering.\n- Test costs are emitted with the inventory so CTest can preserve the test runner's intended scheduling.\n- The native macOS CI job enables functional test registration and runs unit and functional tests together in one CTest invocation. The existing CI timeout factor is forwarded to framework-internal waits.\n\n#### Known differences from `test_runner.py` orchestration\n\nCTest mode is per-test rather than one aggregate `test_runner.py` invocation. Consequently, CTest owns:\n\n- parallelism instead of `test_runner.py --jobs`\n- test selection instead of `--filter` and `--exclude`\n- failure reporting instead of the runner\u2019s aggregate summary\n- scheduling and stop-on-failure policy\n\nThe following runner features are not currently wired into the CTest command:\n\n- `--coverage`;\n- `--resultsfile`;\n- `--nocleanup`;\n- `--tmpdirprefix`;\n- interactive/debugging-oriented options such as `--pdbonfailure`;\n- arbitrary multi-argument test specifications.\n\nCombined logs are enabled through `CTEST_FUNCTIONAL_COMBINED_LOGS_LEN` rather than test_runner.py's `--combinedlogslen` option.\n\n#### Testing\n\nTo test the CTest runner, configure and build with `BUILD_FUNCTIONAL_TESTS=ON`:\n\n```sh\ncmake -B build -DBUILD_FUNCTIONAL_TESTS=ON\ncmake --build build --parallel\nctest --test-dir build --parallel --output-on-failure\n\n# Or functional tests only\nctest --test-dir build --label-regex '^functional$' --parallel --output-on-failure\n```\n\n#### Disclaimer\n\nCodex `gpt-5.6-sol medium` wrote the entire re-implementation of CMake's discover_tests() in cmake in file test/functional/functional_discovery.cmake.in. The python, shell and docs changes are my own.",
 "commits": [
  {
   "sha": "710c1e2e5b790005133a1e1e71166dd9c4fe1c9f",
   "date": "2026-08-17T09:14:47Z",
   "message": "test: emit ctest manifest\n\nGive test_runner.py the ability to emit a line-based test manifest which\ncan be consumed by ctest."
  },
  {
   "sha": "39299351da1a970da2853aa4fe8a73374f913367",
   "date": "2026-08-17T09:14:48Z",
   "message": "test: reuse combined log renderer in test runner\n\ntest runner already emits combined logs when a test fails. Keeping that\nformatting in one helper lets single-test CTest execution reuse the same\noutput."
  },
  {
   "sha": "dfdaa540886d3393366473bfa94644298911bec0",
   "date": "2026-08-17T09:15:31Z",
   "message": "test: add direct CTest runner mode\n\nCTest schedules each discovered functional test independently, so let the\nrunner execute exactly one script without its normal suite orchestration.\nPropagate failures and skip status 77 so CTest can report each result."
  },
  {
   "sha": "a0620ed2c78afeb768b4c5dc8a9ea1ce9bd55bd9",
   "date": "2026-08-17T09:15:32Z",
   "message": "build: register functional tests with CTest\n\nAllow functional tests to run alongside unit tests through CTest while\nkeeping registration opt-in. Use CMake 4.4's discover_tests() when available\nand retain an equivalent compatibility implementation for older supported\nCMake versions.\n\nRun the build-tree test runner so discovery and execution use the generated\nconfig for that build without a separate override."
  },
  {
   "sha": "d868ca08da3b3c540fe011142e9d9f2b9d9f7377",
   "date": "2026-08-17T09:15:33Z",
   "message": "doc: document CTest functional testing\n\nExplain how to enable functional-test registration, select base or extended\ntests, filter by CTest names and labels, and request combined failure logs.\nAlso record the shared-build-tree constraint and discovery behavior so local\nruns do not interfere with each other."
  },
  {
   "sha": "d3cf1fa313df9f3aff3b28b70c1878accb586db1",
   "date": "2026-08-17T09:15:34Z",
   "message": "ci: exercise functional tests through CTest on macOS\n\nUse the existing macOS native job to run unit and functional tests together\nthrough CTest. This exercises the new registration path without changing the\nfunctional-test runner used by the remaining CI jobs."
  }
 ],
 "timeline": [
  {
   "t": "2026-07-21T12:43:40Z",
   "kind": "comment",
   "who": "willcl-ark",
   "assoc": "MEMBER",
   "text": "The `MacOS native` CI job shows ctest orchestrating the unit and functional tests altogether: https://github.com/bitcoin/bitcoin/actions/runs/29824697919/job/88615235560?pr=35762#step:9:3919"
  },
  {
   "t": "2026-07-21T14:54:58Z",
   "kind": "review_comment",
   "who": "maflcko",
   "assoc": "MEMBER",
   "path": "test/functional/test_runner.py",
   "commit": "d3cf1fa313df9f3aff3b28b70c1878accb586db1",
   "in_reply_to": null,
   "text": "d2b4bd40f23067cad97ba6553fe0832d57424c26: Unrelated change?"
  },
  {
   "t": "2026-07-21T14:55:38Z",
   "kind": "review_comment",
   "who": "maflcko",
   "assoc": "MEMBER",
   "path": "test/functional/test_runner.py",
   "commit": "d2b4bd40f23067cad97ba6553fe0832d57424c26",
   "in_reply_to": null,
   "text": "d2b4bd40f23067cad97ba6553fe0832d57424c26: Why not follow the docs and call the test_runner from the build dir, which avoids having to pass the new option?"
  },
  {
   "t": "2026-07-21T14:57:52Z",
   "kind": "review_comment",
   "who": "maflcko",
   "assoc": "MEMBER",
   "path": "test/functional/test_runner.py",
   "commit": "a9e0c71ca4dee40abce66aad0bbb27682ccdcd7f",
   "in_reply_to": null,
   "text": "a9e0c71ca4dee40abce66aad0bbb27682ccdcd7f: Could drop this, or add least add a short rationale for stuff like this?"
  },
  {
   "t": "2026-07-22T19:04:50Z",
   "kind": "force_push",
   "who": "willcl-ark",
   "commit": "16c4245b58308121ebef2446b9e841844f4b1981"
  },
  {
   "t": "2026-07-23T21:10:11Z",
   "kind": "force_push",
   "who": "willcl-ark",
   "commit": "121c9aeb1c6f8b2025d31ba0489d323f87cc45b4"
  },
  {
   "t": "2026-07-23T21:16:55Z",
   "kind": "review_comment",
   "who": "willcl-ark",
   "assoc": "MEMBER",
   "path": "test/functional/test_runner.py",
   "commit": "d3cf1fa313df9f3aff3b28b70c1878accb586db1",
   "in_reply_to": 3623201465,
   "text": "Not quite, although it was incorrect (thanks!)\n\n(somethign like this) is needed because native/upstream `discover_tests()` preserves the empty `\\5` replacement for tests without an optional argument. Without filtering it, `--ctest-direct` sees the script plus an empty second \u201ctest\u201d and rejects the invocation.\n\nThe explicit `--` separator is needed separately because `argparse` allows option abbreviation by default: CTest\u2019s child option `--tmpdir` (tdestined for test_framework.py) can otherwise be consumed as the runner\u2019s `--tmpdirprefix` option instead of being forwarded. The `arg != \"--\"` filter removes that separator before invoking the child test.\n\n(in fact, I think this is a seperate bug and we could disable argparse [allow_abbrev](https://docs.python.org/3/library/argparse.html#allow-abbrev) to fix it properly. Or perhaps makes better sense to fix that first, before these changes...)"
  },
  {
   "t": "2026-07-23T21:21:07Z",
   "kind": "review_comment",
   "who": "willcl-ark",
   "assoc": "MEMBER",
   "path": "test/functional/test_runner.py",
   "commit": "d2b4bd40f23067cad97ba6553fe0832d57424c26",
   "in_reply_to": 3623206731,
   "text": "Addressed in latest push by invoking the generated `build/test/functional/test_runner.py` link/copy. The runner can therefore continue locating `test/config.ini` relative to itself, without adding a runner-level `--configfile` option.\n\nNice, thanks. I didn't think of it because I always run everythign from the source directory, for no good reason."
  },
  {
   "t": "2026-07-23T21:25:26Z",
   "kind": "force_push",
   "who": "willcl-ark",
   "commit": "b7ad3bff61c4e466ae6e16c605d6b0aef2e318e6"
  },
  {
   "t": "2026-07-23T21:27:10Z",
   "kind": "review_comment",
   "who": "willcl-ark",
   "assoc": "MEMBER",
   "path": "test/functional/test_runner.py",
   "commit": "a9e0c71ca4dee40abce66aad0bbb27682ccdcd7f",
   "in_reply_to": 3623224266,
   "text": "added a short comment documenting the source-tree resolution (and dropped `PYTHON_GIL=1` which is set by _test/CMakeLists.txt_ on each test already."
  },
  {
   "t": "2026-07-30T19:44:22Z",
   "kind": "review_comment",
   "who": "maflcko",
   "assoc": "MEMBER",
   "path": "ci/test/00_setup_env.sh",
   "commit": "d3cf1fa313df9f3aff3b28b70c1878accb586db1",
   "in_reply_to": null,
   "text": "not sure about exposing this in CI.  This will silently ignore `TEST_RUNNER_EXTRA`, and having two ways to achieve the same seems confusing anyway.\n\nHowever, not exposing it in CI will also make it less useful ..."
  },
  {
   "t": "2026-08-04T10:19:59Z",
   "kind": "force_push",
   "who": "willcl-ark",
   "commit": "e05e5ad0f7016c5430da3c75685c7749e3b02de1"
  },
  {
   "t": "2026-08-04T11:39:55Z",
   "kind": "review_comment",
   "who": "willcl-ark",
   "assoc": "MEMBER",
   "path": "ci/test/00_setup_env.sh",
   "commit": "d3cf1fa313df9f3aff3b28b70c1878accb586db1",
   "in_reply_to": 3685751131,
   "text": "Thank you for taking a look, and fair point!\n\nI don't consider these strict duplicates: `CTEST_TESTS` is a coarse CTest label selection: all, functional, or nonfunctional whereas `TEST_RUNNER_EXTRA` contains runner-specific behavior and test arguments.\n\nHowever they do certainly overlap, and I think (for now) I will amend the commit to fail if `TEST_RUNNER_EXTRA` is non-empty in `CTEST` mode as a check on silently doing something else than the tests might have indented.\n\nLonger-term, say we switch all tests over to ctest, I would see us dropping `RUN_FUNCTIONAL_TESTS_WITH_CTEST` (it would be default/on for all and unneeded), probably alos remove `CTEST_TESTS` and instead have each job just construct its own ctest command directly, and maybe even remove `TEST_RUNNER_EXTRA` (depening on whether we can get all supported options ported to ctest)."
  },
  {
   "t": "2026-08-04T14:31:56Z",
   "kind": "review_comment",
   "who": "maflcko",
   "assoc": "MEMBER",
   "path": "ci/test/00_setup_env.sh",
   "commit": "d3cf1fa313df9f3aff3b28b70c1878accb586db1",
   "in_reply_to": 3685751131,
   "text": "Why would porting be hard? The leftover from \"${TEST_RUNNER_EXTRA[@]}\" would just end up in `passon_args` and be exported to the test list, no?\n\nThis pull is adding 500 LOC of logic complexity, so my preference would be to at least keep the overhead low.\n\nRecall the alternative to this would be to write it in a few trivial lines of Python (https://github.com/bitcoin/bitcoin/issues/32770#issuecomment-3714857697)"
  },
  {
   "t": "2026-08-04T14:37:13Z",
   "kind": "review_comment",
   "who": "maflcko",
   "assoc": "MEMBER",
   "path": "ci/test/00_setup_env.sh",
   "commit": "d3cf1fa313df9f3aff3b28b70c1878accb586db1",
   "in_reply_to": 3685751131,
   "text": "Also, recall that Python would work when testing win64-cross builds, but CTest does not."
  },
  {
   "t": "2026-08-17T06:51:03Z",
   "kind": "review_comment",
   "who": "maflcko",
   "assoc": "MEMBER",
   "path": "test/functional/test_runner.py",
   "commit": "e05e5ad0f7016c5430da3c75685c7749e3b02de1",
   "in_reply_to": null,
   "text": "unrelated nit: You may want to rebase this on top of https://github.com/bitcoin/bitcoin/pull/35846, for better error (message) handling"
  },
  {
   "t": "2026-08-17T09:40:48Z",
   "kind": "force_push",
   "who": "willcl-ark",
   "commit": "d3cf1fa313df9f3aff3b28b70c1878accb586db1"
  }
 ],
 "labels_log": [
  {
   "t": "2026-07-21T11:04:49Z",
   "action": "labeled",
   "label": "Tests",
   "who": "DrahtBot"
  },
  {
   "t": "2026-07-23T21:26:05Z",
   "action": "labeled",
   "label": "CI failed",
   "who": "DrahtBot"
  },
  {
   "t": "2026-07-30T08:47:12Z",
   "action": "unlabeled",
   "label": "CI failed",
   "who": "maflcko"
  },
  {
   "t": "2026-07-30T09:04:11Z",
   "action": "labeled",
   "label": "CI failed",
   "who": "DrahtBot"
  },
  {
   "t": "2026-08-04T11:32:16Z",
   "action": "unlabeled",
   "label": "CI failed",
   "who": "DrahtBot"
  }
 ],
 "state_log": [],
 "text_chars": 8726,
 "text_tokens_estimate": 2181,
 "changed_paths": [
  "CMakeLists.txt",
  "ci/test/00_setup_env.sh",
  "ci/test/00_setup_env_mac_native.sh",
  "ci/test/03_test_script.sh",
  "test/CMakeLists.txt",
  "test/README.md",
  "test/functional/CMakeLists.txt",
  "test/functional/functional_discovery.cmake.in",
  "test/functional/test_runner.py"
 ],
 "files": [
  {
   "path": "CMakeLists.txt",
   "add": 18,
   "del": 1
  },
  {
   "path": "ci/test/00_setup_env.sh",
   "add": 3,
   "del": 0
  },
  {
   "path": "ci/test/00_setup_env_mac_native.sh",
   "add": 4,
   "del": 0
  },
  {
   "path": "ci/test/03_test_script.sh",
   "add": 42,
   "del": 14
  },
  {
   "path": "test/CMakeLists.txt",
   "add": 4,
   "del": 0
  },
  {
   "path": "test/README.md",
   "add": 58,
   "del": 0
  },
  {
   "path": "test/functional/CMakeLists.txt",
   "add": 87,
   "del": 0
  },
  {
   "path": "test/functional/functional_discovery.cmake.in",
   "add": 96,
   "del": 0
  },
  {
   "path": "test/functional/test_runner.py",
   "add": 137,
   "del": 30
  }
 ],
 "test_lines": 475,
 "git": {
  "head": "d3cf1fa313df9f3aff3b28b70c1878accb586db1",
  "head_matches_backup": true,
  "base": "e0992599a6388822b959ea6286cb32637aa0ec5c",
  "commits": [
   {
    "sha": "710c1e2e5b",
    "subject": "test: emit ctest manifest",
    "files": 1,
    "add": 67,
    "del": 9
   },
   {
    "sha": "39299351da",
    "subject": "test: reuse combined log renderer in test runner",
    "files": 1,
    "add": 14,
    "del": 10
   },
   {
    "sha": "dfdaa54088",
    "subject": "test: add direct CTest runner mode",
    "files": 1,
    "add": 54,
    "del": 9
   },
   {
    "sha": "a0620ed2c7",
    "subject": "build: register functional tests with CTest",
    "files": 5,
    "add": 207,
    "del": 3
   },
   {
    "sha": "d868ca08da",
    "subject": "doc: document CTest functional testing",
    "files": 1,
    "add": 58,
    "del": 0
   },
   {
    "sha": "d3cf1fa313",
    "subject": "ci: exercise functional tests through CTest on macOS",
    "files": 3,
    "add": 49,
    "del": 14
   }
  ],
  "patch_truncated": false
 },
 "input_hash": "6b7f0160b987c23e",
 "extracted_at": "2026-09-17T16:15:31+00:00"
}