#36147 ci: add [free|open|net]BSD ci test jobs
https://github.com/bitcoin/bitcoin/pull/36147 · · +222/-141 in 14 files, 4 commits · labels: Tests, Needs rebase · draft
Goal
- Catch BSD-specific regressions in PR CI before code merges to master
- Reduce reactive bug fixing for maintainers monitoring nightly BSD test runs
Replaces the existing cross-compile CI jobs for FreeBSD, NetBSD, and OpenBSD with native compilation and test execution inside QEMU-based virtual machines using vmactions. Also includes small functional test framework adjustments to suppress FreeBSD lsof warnings and bound HTTPConnection socket timeouts.
Problem: Bitcoin Core cross-compiles for several BSD targets but does not execute tests against them in PR CI. Regressions on these platforms frequently slip into master and must be resolved reactively after nightly test failures.
Category: Build and CI (#24 of 55)
P3 · test coverage
- P3 because catching secondary platform regressions pre-merge is helpful but deferrable
- Benefits BSD platform maintainers but adds CI overhead for non-primary targets
Testing BSD builds directly in CI catches platform-specific regressions before merge instead of after the fact in nightly jobs. As author willcl-ark notes, 'This leads to regressions on master which are noticed in nightly builds and fixed here, recent examples include #36078 and #36145.' While valuable for platform maintainers, adding CI testing for secondary platforms is deferrable.
Membership: Adds GitHub Actions workflows and configuration scripts to compile and execute tests on BSD platforms via VMs.
Factors: security/stability 0, bug 1, performance 0, user value 1, leverage 1
Reviewability: Stale: Needs rebase
- Needs rebase due to merge conflicts
- Draft PR pending decision on whether to reduce scope to FreeBSD
The PR has merge conflicts with master (mergeable_state dirty) and is in draft while its final scope is being decided.
Author status: Active; offered to narrow the PR to FreeBSD only or close the PR if consensus leans against running nested VMs in CI.
Open concerns:
- maflcko pointed out that running nested VMs for three BSD variants risks high false-positive rates and flakiness on the primary CI path, recommending a trial or limiting scope to FreeBSD.
Resolved concerns:
- Investigated historical nightly failure rates on hebasto's repository, determining that FreeBSD is very stable while OpenBSD and NetBSD experience higher test failure rates.
Agreement: Mild
- Nonblocking objection: running three BSD VMs risks flaky tests and CI noise (maflcko)
- Concept approval for adding a test job on at least FreeBSD (hebasto)
- Author offered to narrow scope to FreeBSD or close the PR if VM CI is unwanted (willcl-ark)
Mild: maflcko cautioned that VM flakiness across all three BSDs could add noise to CI; hebasto supports a FreeBSD job.
maflcko raised a nonblocking objection regarding the flakiness and maintenance overhead of running all three BSD VMs in the PR hot path. hebasto supported starting with FreeBSD only, which the author is willing to adopt.
- maflcko noted that vm-in-a-vm tests are brittle and could lead to devs ignoring CI if false-positive rates are high.
- hebasto gave a Concept ACK on running tests on at least one *BSD platform, proposing FreeBSD.
- willcl-ark analyzed failure statistics and offered to start with FreeBSD only or drop the PR before taking it out of draft.
Review verdicts (DrahtBot): 0
- Concept ACK: hebasto
Files
228 lines under test/bench/ci.
- .github/workflows/ci.yml +117/-18
- ci/test/00_setup_env_netbsd_cross.sh +0/-38
- ci/test/00_setup_env_openbsd_cross.sh +0/-38
- ci/test/00_setup_env_freebsd_cross.sh +0/-36
- ci/test/00_setup_env_netbsd_native.sh +24/-0
- ci/test/00_setup_env_openbsd_native.sh +24/-0
- ci/test/03_test_script.sh +16/-6
- ci/test/00_setup_env_freebsd_native.sh +21/-0
- ci/README.md +12/-0
- ci/test/02_run_container.py +3/-1
- ci/test/00_setup_env.sh +2/-1
- ci/test_run_all.sh +1/-1
- test/functional/interface_rest.py +1/-1
- test/functional/test_framework/netutil.py +1/-1
Card
This PR replaces cross-compilation CI tasks for FreeBSD, NetBSD, and OpenBSD with VM-based native builds and functional test runs. It aims to eliminate a recurring cycle where BSD regressions pass PR review and are only caught by external nightly builds. Review is currently stale due to merge conflicts and open discussion regarding whether nested VM flakiness makes running all three BSDs on every PR too noisy. The author and reviewers have converged toward testing only FreeBSD if the PR proceeds.