{ "number": 36147, "repo": "bitcoin/bitcoin", "url": "https://github.com/bitcoin/bitcoin/pull/36147", "title": "ci: add [free|open|net]BSD ci test jobs", "author": "willcl-ark", "author_association": "MEMBER", "created_at": "2026-09-02T11:07:50Z", "updated_at": "2026-09-17T09:53:51Z", "age_days": 15, "draft": true, "labels": [ "Tests", "Needs rebase" ], "milestone": null, "base": "master", "head_sha": "31a2ac55c0dc967b064a2a3e4e91240fb70e7f3c", "head_ref": "bsd-ci", "head_repo": "willcl-ark/bitcoin", "head_history": [], "additions": 222, "deletions": 141, "changed_files": 14, "commit_count": 4, "size_bucket": "M", "mergeable_state": "dirty", "bot": { "drahtbot": { "present": true, "reviews": { "concept_ack": [ { "login": "hebasto", "url": "https://github.com/bitcoin/bitcoin/pull/36147#issuecomment-5525915036" } ] }, "conflicts": [ { "number": 36100, "title": "ci: use LLVM 23 in *san, fuzz, *cross jobs", "author": "fanquake" }, { "number": 36082, "title": "ci: add Guix builds to CI", "author": "willcl-ark" }, { "number": 36052, "title": "ci: Doc: Move all config comments right next to the option they explain", "author": "maflcko" }, { "number": 36042, "title": "build: Bump g++ minimum supported version to 13", "author": "maflcko" }, { "number": 35774, "title": "ci: test cross-built macos arm64 binaries", "author": "willcl-ark" }, { "number": 35762, "title": "test: optionally run functional tests via CTest", "author": "willcl-ark" } ] } }, "acks_parsed": { "hebasto": { "kind": "concept_ack", "hash": null, "t": "2026-09-03T12:41:20Z", "stale": false } }, "acks_tally": { "ack": 0, "stale_ack": 0, "concept_ack": 1, "approach_ack": 0, "nack": 0, "concept_nack": 0, "approach_nack": 0 }, "reviews": { "approved": 0, "changes_requested": 0, "distinct_reviewers": [ "hebasto", "maflcko" ] }, "signals": { "needs_rebase": true, "ci_failed": false, "mergeable_state": "dirty", "last_author_activity": "2026-09-03T12:24:02Z", "last_reviewer_activity": "2026-09-17T09:53:51Z", "last_reviewer": "maflcko", "author_silent_days": 14, "waiting_on_author_days": 0, "days_since_update": 0 }, "refs": { "mentioned": [ 36078, 36145 ], "depends_on": [], "fixes": [], "linked_issues": [], "references": [ { "number": 36078, "type": "pull", "state": "closed", "merged": true, "merged_at": "2026-08-25", "title": "qa: Reduce `-maxconnections` in the functional test framework" }, { "number": 36145, "type": "pull", "state": "closed", "merged": true, "merged_at": "2026-09-03", "title": "qa: Use IP_PORTRANGE_HIGH on OpenBSD for dynamic port allocation" } ], "conflicts": [ 36100, 36082, 36052, 36042, 35774, 35762 ] }, "stack": { "shares_commits_with": [], "based_on": [], "base_for": [] }, "review_paths": [], "body": "We currently cross compile various BSD binaries but do not test them. This leads to regressions on master which are noticed in nightly builds and fixed here, recent examples include #36078 and #36145.\n\nBuild **and test** the binaries in this repo as part of our CI instead to avoid this regression/fix flip-flopping.\n\nAs we don't have native *BSD runners we require BSD VMs to test the binaries. https://github.com/vmactions offers these (and are currently being used in various nightly jobs).\n\nUnfortunately it's not possible to run all of the cross-compiled binaries directly in these VMs because the VM image builds (via anyvm) [are syspatching](https://github.com/anyvm-org/openbsd-builder/blob/457118c7ea96bba2aa15579457efc17701210061/hooks/vm_postBuild.sh#L18-L19) their images, which cause some function calls to be missing when executing the (pure) cross-compiled binaries.\n\nTherefore here, and to keep all BSD builds following a single CI process, build and test all BSDs inside their VM.\n\nThe VM caches itself after its `prepare` step (installing dependencies) so operates _similarly_ to an unchanged docker buildkit image stage.\n\nA clean (no caches) build and test inside the VM on a free GHA runner takes ~ 60 minutes. With the various caches seeded this drops to about 20-30 minutes. On a larger Warp runner I'd expect both of these runtimes to be much shorter, but this is untested currently.\n\nThere is some slowness when the tests are completed: `rsync` copies various cache directories back out from the VM in case they want to be GHA-cached later. It may be possible to optimise this slowness a little in the future.\n\nThe patchset also includes two small fixes, one needed to get the tests running, and one to prevent an overly-long REST request hang observed during testing (it may be possible/preferable to drop the REST request timeout reduction before merging).\n\nIt also omits `feature_reindex_init` and `p2p_private_broadcast_retry_v1` from the functional tests; the former fails, and the latter seems to take an extraordinarily long time. Both might be good candidates for investigation and followup PRs.\n\nThese jobs will require enabling 3 more actions:\n\nvmactions/freebsd-vm@v1\nvmactions/netbsd-vm@v1\nvmactions/openbsd-vm@v1\n\nI have tried to re-use or at least mimic the current CI execution model/flow wherever possible.", "commits": [ { "sha": "4b890c5b306472e52b0202b60ff4364483fd88ba", "date": "2026-09-02T08:35:01Z", "message": "test: silence lsof warnings on FreeBSD\n\nFreeBSD lsof emits warnings even though the requested socket data remains\nusable. Suppress them to keep functional test output focused." }, { "sha": "9e72629be68ec7ba4b039064424544c2e6b36e91", "date": "2026-09-02T08:35:34Z", "message": "test: bound REST request timeout\n\nA stalled REST response can otherwise hang the entire functional test job.\nUse a finite socket timeout so the request fails with a diagnostic." }, { "sha": "6be2f517702d07e423defce0974d0e76b98500e3", "date": "2026-09-02T09:05:14Z", "message": "ci: support native BSD test runs\n\nReuse the existing host-mode CI driver instead of maintaining a second entry\npoint. BSD hosts arrive pre-provisioned, while the shared scripts handle the\nnative tools and resource limits." }, { "sha": "31a2ac55c0dc967b064a2a3e4e91240fb70e7f3c", "date": "2026-09-02T09:05:15Z", "message": "ci: replace BSD cross jobs with native jobs\n\nThe BSD VMs update their base systems, so cross-built binaries can target\nstale interfaces. Build and test within each VM while retaining the standard\ntest runner and repository caches." } ], "timeline": [ { "t": "2026-09-02T12:11:11Z", "kind": "comment", "who": "maflcko", "assoc": "MEMBER", "text": "[quoted text omitted]\n\nI think this requires a stronger motivation, to explain that this will actually reduce overall workload.\n\nThose vm-in-a-vm tests are extremely brittle (https://github.com/hebasto/bitcoin-core-nightly/issues/230#issuecomment-4311704411, https://github.com/hebasto/bitcoin-core-nightly/issues/81, ... etc)\n\nSo if those tasks are added here, we need to understand that this is the \"hot path\" (for all pull requests) and before this is merged, there needs to be several people who monitor those intermittent failures on those platforms and then categorize them into \"intermittent\", \"unrelated\", etc.\n\nIf this isn't done, then devs will just ignore all CI results because the false-positive rate is too high and just re-run it until it is green. This may then degrade non-*BSD code at a slight benefit of avoid some trivial test-only BSD fixups.\n\nMaybe I am wrong, but I wanted to raise the point to not miss the large picture here." }, { "t": "2026-09-02T12:24:11Z", "kind": "comment", "who": "hebasto", "assoc": "MEMBER", "text": "[quoted text omitted]\n\nWe might begin with FreeBSD only, which has shown high stability over a [long period](https://github.com/hebasto/bitcoin-core-nightly/actions/workflows/freebsd.yml?query=event%3Aworkflow_run)." }, { "t": "2026-09-02T12:38:23Z", "kind": "comment", "who": "maflcko", "assoc": "MEMBER", "text": "[quoted text omitted]\n\nYour link shows an unrelated failure for the 35th build (second page). Not sure if this is accurate, but if every 35th build fails and Bitcoin Core has 35 pushes per day, then every day there will be one additional unrelated CI failure.\n\nMaybe there can be a one-month trial period and then a summary is presented with all CI failures and each of them annotated with \"intermittent\" (y/n), \"unrelated\" (y/n)?\n\nAlso, even if a failure is related, it can still be intermittent, so merging this pull request does not guarantee to prevent fixups such as https://github.com/bitcoin/bitcoin/pull/36145, because the failure happens intermittently." }, { "t": "2026-09-02T14:18:17Z", "kind": "comment", "who": "willcl-ark", "assoc": "MEMBER", "text": "I did not understand these VMs to be flaky. I didn't observe any flakiness in testing, but probably only ran about 15 runs (x3 jobs), so perhaps I'm due a random flake-failure soon \ud83d\ude0b\n\nThe actions _are_ used by a few other high-profile projects like [tokio](https://github.com/tokio-rs/tokio/actions/runs/32341804131/job/96978505535) and [sccache](https://github.com/mozilla/sccache/actions/runs/33376171138/job/99438039126) as well as:\n\n - tukaani-project/xz using OpenBSD (workflow)\n - hashcat/hashcat using OpenBSD\n - acmesh-official/acme.sh using OpenBSD\n - aws/aws-lc-rs using FreeBSD (workflow)\n - mozilla/sccache using FreeBSD\n - aristocratos/btop using OpenBSD\n - stalwartlabs/stalwart using FreeBSD\n - deskflow/deskflow using FreeBSD\n - FrameworkComputer/framework-system using FreeBSD\n\nBut I'd imagine our functional tests put significantly more strain on the vm than any of those projects are doing.\n\nI asked codex to take a look at the last months-worth of runs on hebasto's nightly repo:\n\nDetails\n\n[`hebasto/bitcoin-core-nightly`](https://github.com/hebasto/bitcoin-core-nightly)\nfrom 2026-08-02 through 2026-09-02. Pull-request runs were excluded. The\nnightly sample consists of runs with the `workflow_run` event. Manual\n`workflow_dispatch` failures are listed separately where they add useful\ncontext.\n\n## Summary\n\n| Workflow | Nightly runs | Successful | Failed | Cancelled |\n| --- | ---: | ---: | ---: | ---: |\n| FreeBSD | 33 | 33 | 0 | 0 |\n| NetBSD | 33 | 28 | 4 | 1 |\n| OpenBSD | 33 | 10 | 19 | 4 |\n\nThe logs for all 29 failed runs were downloaded and inspected.\n\nFreeBSD looks stable in this sample. NetBSD has a small number of nightly\nfailures, mostly caused by platform resource limits or platform-sensitive\ntests. OpenBSD has a high failure rate, but most failures are repeatable test\nor platform problems rather than failures to boot the VM.\n\n## NetBSD\n\n### Nightly failures\n\n- [Run 906](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/32799434384),\n 2026-08-25: `feature_block.py` timed out waiting for the expected peer\n disconnect. Other jobs showed the NetBSD file-descriptor limit.\n- [Run 899](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/32322734783),\n 2026-08-20: `feature_block.py` timed out waiting for a peer disconnect after\n the node had rejected the block.\n- [Run 892](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/31765546925),\n 2026-08-14: `wallet_listtransactions.py` failed when `generateblock`\n returned `bad-txns-inputs-missingorspent`.\n- [Run 888](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/31350308484),\n 2026-08-10: `wallet_listtransactions.py` failed with the same\n `TestBlockValidity` error.\n\nThe `feature_block.py` failures look like delayed networking or scheduling in\nthe guest: the node log records the rejection and disconnect, while the test\ndoes not observe the disconnect before its timeout. The wallet failures are\nfunctional test failures and should not be counted as generic VM boot\nfailures.\n\n### Manual failures\n\nThe following manual runs on 2026-08-25 failed during test setup because the\nguest did not have enough file descriptors:\n\n- [Run 909](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/32837640603)\n- [Run 911](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/32842679103)\n- [Run 912](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/32843365189)\n- [Run 913](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/32852803381)\n- [Run 914](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/32867589050)\n\nThe common messages were:\n\n```text\nNot enough file descriptors available. 256 available, 290 required.\nWarning: Reducing -maxconnections ... because of system limitations.\n```\n\nThese are reproducible NetBSD configuration or test-environment failures, not\nrandom VM failures.\n\nThere was one additional manual failure:\n\n- [Run 895](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/31943115495),\n 2026-08-16: the VM boot probe timed out and QEMU was restarted. The run also\n contained `feature_anchors.py` and `interface_http.py` timing failures.\n\nThis is the clearest NetBSD example of actual VM or infrastructure flakiness.\n\n## OpenBSD\n\n### Repeated `interface_gui.py` failures\n\n`interface_gui.py` failed in runs [738](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/30731534646),\n[739](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/30783225699),\n[740](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/30875541319),\n[743](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/30972882503),\n[744](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/31069287625),\n[745](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/31144016684),\n[746](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/31234889921),\n[747](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/31290532291),\n[748](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/31350308514),\n[749](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/31452609174),\nand [751](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/31558633967).\n\nThe recurring error was:\n\n```text\nGLib-CRITICAL: g_main_context_pop_thread_default:\nassertion 'stack != NULL' failed\n```\n\nThis is highly repeatable across OpenBSD 7.8 and 7.9 GUI jobs. It looks like\nan OpenBSD/Qt shutdown or test-environment problem, not QEMU randomly failing.\n\n### Repeated P2P test timeouts\n\n`p2p_private_broadcast.py` or\n`p2p_private_broadcast_retry_v1.py` failed in runs [738](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/30731534646),\n[754](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/31765547000),\n[757](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/31986512649),\n[758](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/32089854630),\n[761](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/32322734906),\n[765](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/32681596291),\n[766](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/32799434355),\n[772](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/33060952442),\nand [779](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/33591911980).\n\nThe test waited exactly 480 seconds for all addresses to advertise\n`NODE_P2P_V2`:\n\n```text\nPredicate ... not true after 480.0 seconds\n```\n\nThe node remained responsive and answered `getnodeaddresses` RPC requests, but\nthe expected service flags never appeared. Because the same test fails in the\nsame way across many consecutive OpenBSD runs, this looks like an OpenBSD/test\ninteraction or severe guest-performance problem rather than a generic boot\nfailure.\n\n### Package and VM infrastructure failures\n\n- [Run 754](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/31765547000):\n OpenBSD package installation failed with `Premature end of archive` while\n installing `nghttp2`.\n- [Run 761](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/32322734906),\n [765](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/32681596291),\n and [766](https://github.com/hebasto/bitcoin-core-nightly/actions/runs/32799434355):\n package downloads included 404s or connection timeouts.\n\nThese failures are package-mirror or network infrastructure failures. They are\nseparate from the repeated functional test failures.\n\n## Conclusion\n\nFor this one-month sample:\n\n- FreeBSD: no observed nightly failures.\n- NetBSD: some real guest or scheduling sensitivity, but most failures are\n resource-limit or functional-test issues. The one clear VM boot failure was\n on 2026-08-16.\n- OpenBSD: 19 failed nightly runs out of 33. The failure count is dominated by\n repeatable `interface_gui.py` and `p2p_private_broadcast*` failures, with a\n few package-mirror failures. It would be misleading to attribute all 19 to\n vmactions or QEMU flakiness.\n\nThe next useful experiment would be to fix or disable the known OpenBSD GUI\nand P2P test incompatibilities, raise the NetBSD file-descriptor limit, and\nthen repeat the same nightly comparison. That would leave a much cleaner\nmeasurement of VM-level reliability.\n\nI have not observed the interface_gui.py failures that hebasto's repo sees on OpenBSD. I did see a failure on p2p_private_broadcast_retry_v1.py once or twice and did have that test disabled on *BSD in one iteration; if that test in particular is flaky it could be disabled on these jobs.\n\nMost of the other failures at first glance appear to be GHA/package archive 404 or other network infra failures and I wouldn't attribute them to the VMs being flaky, so I'm not 100% convinced that critique is warranted.\n\nThat said I don't want to add known-flaky jobs and you are correct to call out they could be more work to maintain than porting over a few fixes, if they are unreliable. I suppose the only unclear bit is at what cost level do we prefer to merge buggy code and fix later, vs run more jobs on every PR and merge to prevent that happening in the first place...\n\n[quoted text omitted]\nHappy to start with FreeBSD only if that's preferable." }, { "t": "2026-09-02T14:47:39Z", "kind": "comment", "who": "maflcko", "assoc": "MEMBER", "text": "[quoted text omitted]\n\nSure, I am not against this pull request, but then the motivation is off, because the two motivating issues are about netbsd and openbsd, which for some reason have a lot more false positive intermittent failures?\n\nIt still seems that the remaining fixups required for FreeBSD are less effort than maintaining the FreeBSD ci task here? (Recall that codex only looked at the last month with daily builds, but it missed the failure from 35 days ago and recall that Bitcoin Core has a higher CI volume than just one build per day, so any rare failure will amplify)\n\nAgain, seems fine to merge this, but as long as the false positive rate is higher than the true positive rate, it doesn't seem worth it overall." }, { "t": "2026-09-03T12:24:02Z", "kind": "comment", "who": "willcl-ark", "assoc": "MEMBER", "text": "Scanning the last 90 days (where logs still exist) in hebasto's repo I really only see this one where the vm failed to boot: https://github.com/hebasto/bitcoin-core-nightly/actions/runs/29139756629/job/86510586228#step:2:188\n\nThere are a handful of \"random\" functional test issues, which could be attributable to the vm-in-vm setup, but I'd recommend [disabling those few tests](https://github.com/willcl-ark/bitcoin/blob/31a2ac55c0dc967b064a2a3e4e91240fb70e7f3c/ci/test/00_setup_env_freebsd_native.sh#L16). There are a number of legitimate functional test failures repeated over many days (e.g interface_gui.py) and few semi-legitimate-looking errors which could have been vm-flakiness.\n\nIncluding the ~legit test timeout-type failures, but excluding real test failures by my count I see 18 \"flaky failures\" out of 812 jobs over the 90 days, which or 2.2%.\n\n[quoted text omitted]\nOur CI jobs have between a 7 and 2% fail rate (assuming all current failures are true-positives), per our CI [job statistics](https://willcl-ark.github.io/bitcoin-core-ci-stats/stats/).\n\nSo for references 2.2% as a potential false-positive rate is ~ in the same ballpark as the true-failures we see on jobs today (assuming all are true-failures). I agree a flakiness ~= to the bug-catching ability of non-flaky tests isn't great.\n\nIf we are more happy to backport fixes from broken nightly/master builds than run CI jobs here as we are worried they'll be too flaky (which is more like a regular CI/CD process to be honest) then that works for me too.\n\nI'll leave this in draft for a few days in case anyone else wants to weigh in, before closing." }, { "t": "2026-09-03T12:41:20Z", "kind": "comment", "who": "hebasto", "assoc": "MEMBER", "text": "Concept ACK on having at least one CI job that runs tests on a *BSD platform." }, { "t": "2026-09-17T09:53:51Z", "kind": "comment", "who": "maflcko", "assoc": "MEMBER", "text": "[quoted text omitted]\n\nIs this the average over all BSDs? Because I see 5 OpenBSD failures in the last 10 days alone, which is more than 10%.\n\nI guess we could add the FreeBSD one only, but I wonder what issues it would have caught this year. Mostly only https://github.com/bitcoin/bitcoin/issues/34331?\n\nI'd say we don't care about FreeBSD failures that happen on other platforms as well, so true positive rate is meant as the \"marginal\" (additional or exclusive) true positive rate.\n\nI guess it could be tried for a month or two, to see what happens in reality." } ], "labels_log": [ { "t": "2026-09-02T11:07:54Z", "action": "labeled", "label": "Tests", "who": "DrahtBot" }, { "t": "2026-09-03T11:54:53Z", "action": "labeled", "label": "Needs rebase", "who": "DrahtBot" } ], "state_log": [], "text_chars": 17246, "text_tokens_estimate": 4311, "changed_paths": [ ".github/workflows/ci.yml", "ci/README.md", "ci/test/00_setup_env.sh", "ci/test/00_setup_env_freebsd_cross.sh", "ci/test/00_setup_env_freebsd_native.sh", "ci/test/00_setup_env_netbsd_cross.sh", "ci/test/00_setup_env_netbsd_native.sh", "ci/test/00_setup_env_openbsd_cross.sh", "ci/test/00_setup_env_openbsd_native.sh", "ci/test/02_run_container.py", "ci/test/03_test_script.sh", "ci/test_run_all.sh", "test/functional/interface_rest.py", "test/functional/test_framework/netutil.py" ], "files": [ { "path": ".github/workflows/ci.yml", "add": 117, "del": 18 }, { "path": "ci/README.md", "add": 12, "del": 0 }, { "path": "ci/test/00_setup_env.sh", "add": 2, "del": 1 }, { "path": "ci/test/00_setup_env_freebsd_cross.sh", "add": 0, "del": 36 }, { "path": "ci/test/00_setup_env_freebsd_native.sh", "add": 21, "del": 0 }, { "path": "ci/test/00_setup_env_netbsd_cross.sh", "add": 0, "del": 38 }, { "path": "ci/test/00_setup_env_netbsd_native.sh", "add": 24, "del": 0 }, { "path": "ci/test/00_setup_env_openbsd_cross.sh", "add": 0, "del": 38 }, { "path": "ci/test/00_setup_env_openbsd_native.sh", "add": 24, "del": 0 }, { "path": "ci/test/02_run_container.py", "add": 3, "del": 1 }, { "path": "ci/test/03_test_script.sh", "add": 16, "del": 6 }, { "path": "ci/test_run_all.sh", "add": 1, "del": 1 }, { "path": "test/functional/interface_rest.py", "add": 1, "del": 1 }, { "path": "test/functional/test_framework/netutil.py", "add": 1, "del": 1 } ], "test_lines": 228, "git": { "head": "31a2ac55c0dc967b064a2a3e4e91240fb70e7f3c", "head_matches_backup": true, "base": "dc0395c5858a1d55239b82a834e5075cf2069219", "commits": [ { "sha": "4b890c5b30", "subject": "test: silence lsof warnings on FreeBSD", "files": 1, "add": 1, "del": 1 }, { "sha": "9e72629be6", "subject": "test: bound REST request timeout", "files": 1, "add": 1, "del": 1 }, { "sha": "6be2f51770", "subject": "ci: support native BSD test runs", "files": 8, "add": 103, "del": 9 }, { "sha": "31a2ac55c0", "subject": "ci: replace BSD cross jobs with native jobs", "files": 4, "add": 117, "del": 130 } ], "patch_truncated": false }, "input_hash": "d216145dd8cc6582", "extracted_at": "2026-09-17T16:15:31+00:00" }