{ "number": 35751, "repo": "bitcoin/bitcoin", "url": "https://github.com/bitcoin/bitcoin/pull/35751", "title": "validation: use parallel input prevout fetching in TestBlockValidity", "author": "andrewtoth", "author_association": "MEMBER", "created_at": "2026-07-19T23:30:37Z", "updated_at": "2026-09-10T12:48:12Z", "age_days": 59, "draft": false, "labels": [ "Validation" ], "milestone": null, "base": "master", "head_sha": "2ba5628cb90ea2ae960d3823742412741704e4ca", "head_ref": "testblockvalidity-startfetching", "head_repo": "andrewtoth/bitcoin", "head_history": [ { "t": "2026-07-29T02:51:50Z", "sha": "2046c94a9b013651535c100d7bcca9c54e6568ec" }, { "t": "2026-07-29T03:01:40Z", "sha": "ff4dd421f9020e85711c6e4069e62abf31fc3c0f" }, { "t": "2026-08-08T17:21:10Z", "sha": "2ba5628cb90ea2ae960d3823742412741704e4ca" } ], "additions": 39, "deletions": 7, "changed_files": 3, "commit_count": 2, "size_bucket": "S", "mergeable_state": "unknown", "bot": { "drahtbot": { "present": true, "reviews": { "concept_ack": [ { "login": "Sjors", "url": "https://github.com/bitcoin/bitcoin/pull/35751#issuecomment-5061422051" } ], "stale_ack": [ { "login": "sedited", "url": "https://github.com/bitcoin/bitcoin/pull/35751#pullrequestreview-4753791958" } ] }, "conflicts": [ { "number": 36066, "title": "validation: Separate check-only version of ConnectBlock", "author": "optout21" }, { "number": 35646, "title": "RFC: Separate out runtime errors from BlockValidationState using `util::Expected`", "author": "yuvicc" }, { "number": 35570, "title": "refactor: Change some validation.cpp methods to return BlockValidationState", "author": "optout21" }, { "number": 35511, "title": "RFC: consensus: Make `CAmount` a class", "author": "hodlinator" }, { "number": 29700, "title": "kernel, refactor: return error status on all fatal errors", "author": "ryanofsky" } ] } }, "acks_parsed": { "Sjors": { "kind": "concept_ack", "hash": null, "t": "2026-07-23T17:30:19Z", "stale": false } }, "acks_tally": { "ack": 0, "stale_ack": 0, "concept_ack": 1, "approach_ack": 0, "nack": 0, "concept_nack": 0, "approach_nack": 0 }, "reviews": { "approved": 1, "changes_requested": 1, "distinct_reviewers": [ "Sjors", "l0rinc", "optout21", "sedited" ] }, "signals": { "needs_rebase": false, "ci_failed": false, "mergeable_state": "unknown", "last_author_activity": "2026-08-23T21:42:50Z", "last_reviewer_activity": "2026-08-26T21:30:14Z", "last_reviewer": "l0rinc", "author_silent_days": 24, "waiting_on_author_days": 21, "days_since_update": 7 }, "refs": { "mentioned": [ 31132, 35295, 35671 ], "depends_on": [], "fixes": [], "linked_issues": [], "references": [ { "number": 31132, "type": "pull", "state": "closed", "merged": false, "merged_at": null, "title": "validation: fetch block inputs on parallel threads" }, { "number": 35295, "type": "pull", "state": "closed", "merged": true, "merged_at": "2026-07-09", "title": "validation: fetch block input prevouts in parallel during ConnectBlock" }, { "number": 35671, "type": "pull", "state": "open", "merged": false, "merged_at": null, "title": "mining: add TxCollection to bandwidth-efficiently validate external block templates" } ], "conflicts": [ 36066, 35646, 35570, 35511, 29700 ] }, "stack": { "shares_commits_with": [], "based_on": [], "base_for": [] }, "review_paths": [ "src/bench/block_assemble.cpp", "src/validation.cpp" ], "body": "Use CoinsViewOverlay to speed up ConnectBlock in TestBlockValidity. This also has the benefit of not inserting entries fetched from db into the main cache.\n\ncs_main must be held for each place where the overlay is being used, and the reset guard ensures it will be reset for the next caller.", "commits": [ { "sha": "8ebb32e69f8a4959e4d73481c64d7c1282fc9e33", "date": "2026-07-29T03:01:34Z", "message": "bench: fill assembled block to max weight in AssembleBlock" }, { "sha": "2ba5628cb90ea2ae960d3823742412741704e4ca", "date": "2026-08-08T17:20:59Z", "message": "validation: use parallel input prevout fetching in TestBlockValidity\n\nUse CoinsViewOverlay to speed up ConnectBlock in TestBlockValidity.\nThis also has the benefit of not inserting entries fetched from db\ninto the main cache.\n\ncs_main must be held for each place where the overlay is being used,\nand the reset guard ensures it will be reset for the next caller." } ], "timeline": [ { "t": "2026-07-22T11:18:43Z", "kind": "review", "who": "sedited", "assoc": "MEMBER", "state": "APPROVED", "commit": "985d248383b43205f98a6971eb887ffa239adf85", "text": "~ACK 985d248383b43205f98a6971eb887ffa239adf85~\n\nI was too hasty here, looks like we need to understand the regression a bit better." }, { "t": "2026-07-22T21:17:24Z", "kind": "comment", "who": "l0rinc", "assoc": "MEMBER", "text": "For the record: This is a follow up to https://github.com/bitcoin/bitcoin/pull/31132#discussion_r2645502461 and https://github.com/bitcoin/bitcoin/pull/34164#discussion_r2657511963.\n\n----\n\nhttps://corecheck.dev/bitcoin/bitcoin/pulls/35751 claimed `AssembleBlock` slowed down considerably but I can't reproduce it locally (though I can confirm it's called from the benchmark).\nDetails\n\n```\n[quoted text omitted]\n\n| ns/op | op/s | err% | total | benchmark\n|--------------------:|--------------------:|--------:|----------:|:----------\n| 82,662.62 | 12,097.37 | 0.3% | 1.07 | `AssembleBlock`\n\n[quoted text omitted]\n| ns/op | op/s | err% | total | benchmark\n|--------------------:|--------------------:|--------:|----------:|:----------\n| 82,993.67 | 12,049.11 | 0.4% | 1.08 | `AssembleBlock`\n```\n\nCould you provide a reproducible way to test the performance claim, ideally using a deterministic set of mainnet block proposals or a microbenchmark?\nAnd since this change targets block-template validation, maybe we should ping @Sjors to get input from reviewers familiar with mining workloads." }, { "t": "2026-07-23T02:43:39Z", "kind": "comment", "who": "andrewtoth", "assoc": "MEMBER", "text": "[quoted text omitted]\n\n@l0rinc is that the correct benchmark data you meant to post in the details? It doesn't seem to show the a slowdown...\n\n[quoted text omitted]\nThere have been many benchmarks performed already by you in #31132 for this change. Is this not obvious to you it will increase performance?" }, { "t": "2026-07-23T12:13:06Z", "kind": "comment", "who": "sedited", "assoc": "MEMBER", "text": "[quoted text omitted]\n\nI see it in the report too, and can measure the slowdown locally as well." }, { "t": "2026-07-23T12:23:57Z", "kind": "comment", "who": "sedited", "assoc": "MEMBER", "text": "If I fill up the block properly in the `AssembleBlock` benchmark, this does indeed then speed up connection. Seems like we should be doing that anyway to make the benchmark realistic?\n\nI'm guessing there is a cutover where the synchronization overhead outweighs getting the coins from cache (or maybe even from disk?). Maybe it would be worthwhile to introduce a heuristic where we skip background fetching if the block does not spend a lot of coins?" }, { "t": "2026-07-23T17:30:19Z", "kind": "comment", "who": "Sjors", "assoc": "MEMBER", "text": "Concept ACK on (indirectly) making the `checkBlock()` IPC faster. This is intended to be called frequently by the Job Declarator Server role in Stratum v2.\n\nAlthough #35671 introduces a different approach for that, its `makeTemplate()` method also goes through `TextBlockValidity`. I wonder if `TxCollection` could be made even faster if pre-loads coins it needs, _before_ a client calls `makeTemplate()`to check the full block. _Update: see https://github.com/Sjors/bitcoin/pull/122_\n\nI haven't studied the earlier work, so for that reason alone a quick benchmark might be nice to demonstrate to others too." }, { "t": "2026-07-29T02:51:50Z", "kind": "force_push", "who": "andrewtoth", "commit": "2046c94a9b013651535c100d7bcca9c54e6568ec" }, { "t": "2026-07-29T03:01:40Z", "kind": "force_push", "who": "andrewtoth", "commit": "ff4dd421f9020e85711c6e4069e62abf31fc3c0f" }, { "t": "2026-07-29T13:40:28Z", "kind": "comment", "who": "andrewtoth", "assoc": "MEMBER", "text": "Yes, I see the regression in the in corecheck as well. I first read @l0rinc's comment as containing the regression. My apologies.\n\n@Sjors I've included an additional commit at the start that improves the `AssembleBlock` benchmark by filling the block to max weight with transactions. The new benchmark shows a 10% speedup on my local machine when comparing the first and second commits.\n\nBefore this change\n\n| ns/op | op/s | err% | ins/op | cyc/op | IPC | bra/op | miss% | total | benchmark\n|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------\n| 16,370,829.20 | 61.08 | 0.5% | 169,414,150.61 | 36,723,542.62 | 4.613 | 21,344,193.75 | 0.2% | 5.71 | `AssembleBlock`\n| 16,157,284.00 | 61.89 | 2.9% | 169,412,308.63 | 36,308,394.28 | 4.666 | 21,345,426.37 | 0.2% | 5.61 | `AssembleBlock`\n| 16,328,205.58 | 61.24 | 1.6% | 169,374,584.69 | 36,629,119.06 | 4.624 | 21,343,993.70 | 0.2% | 5.57 | `AssembleBlock`\n\nAfter this change\n\n| ns/op | op/s | err% | ins/op | cyc/op | IPC | bra/op | miss% | total | benchmark\n|--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------\n| 14,860,137.50 | 67.29 | 1.8% | 160,474,005.09 | 35,797,692.66 | 4.483 | 21,636,973.66 | 0.2% | 5.42 | `AssembleBlock`\n| 14,322,912.28 | 69.82 | 1.1% | 160,432,472.41 | 34,525,565.16 | 4.647 | 21,656,523.97 | 0.2% | 5.41 | `AssembleBlock`\n| 14,767,059.78 | 67.72 | 0.7% | 160,474,240.27 | 35,583,122.53 | 4.510 | 21,644,652.59 | 0.2% | 5.49 | `AssembleBlock`\n\n[quoted text omitted]\n@sedited There is likely a cutover, but it would be dependent on the system as well as other factors like how many coins are in the cache vs have to be fetched from db.\nre: a heuristic - what we really care about is that slow blocks are faster. A block spending few coins is already very fast, and so speeding it up from 2ms -> 1ms will not really benefit anything in the grand scheme IMO." }, { "t": "2026-08-07T07:21:24Z", "kind": "review_comment", "who": "optout21", "assoc": "CONTRIBUTOR", "path": "src/validation.cpp", "commit": "ff4dd421f9020e85711c6e4069e62abf31fc3c0f", "in_reply_to": null, "text": "[quoted text omitted]\n\nA note on \"dummy\" naming: before the change, `view_dummy` was a local-only instance, so the \"dummy\" naming makes sense. However, now it's a reference to a longer-living instance, so I find \"dummy\" misleading. I suggest renaming it to `view_shared`. (There are not many usages to change)." }, { "t": "2026-08-07T07:27:49Z", "kind": "review", "who": "optout21", "assoc": "CONTRIBUTOR", "state": "COMMENTED", "commit": "ff4dd421f9020e85711c6e4069e62abf31fc3c0f", "text": "Another performance improvement, a must-have follow-up of #35295/#34164/#31132, looks good.\n\nTwo comments:\n\nI am confused by the description of the 2nd (last) commit. It states:\n\n- \"Use CoinsViewOverlay to speed up ConnectBlock in TestBlockValidity.\" -- CoinsViewOverlay was used before; the change seems not about using it, but using an existing shared instance, instead of always a new one.\n\n- \"This also has the benefit of not populating entries in the main cache.\" -- that was true also without the change.\n\nThere is also another usage of `m_connect_block_view`, at line 3055:\n\n```\n3055 CoinsViewOverlay& view{*m_coins_views->m_connect_block_view};\n3056 const auto reset_guard{view.StartFetching(*block_to_connect)};\n```\n(https://github.com/andrewtoth/bitcoin/blob/ff4dd421f9020e85711c6e4069e62abf31fc3c0f/src/validation.cpp#L3055-L3056)\n\nCould this also use the newly introduced `ConnectBlockView()`?" }, { "t": "2026-08-08T17:21:10Z", "kind": "force_push", "who": "andrewtoth", "commit": "2ba5628cb90ea2ae960d3823742412741704e4ca" }, { "t": "2026-08-08T17:22:17Z", "kind": "review_comment", "who": "andrewtoth", "assoc": "MEMBER", "path": "src/validation.cpp", "commit": "ff4dd421f9020e85711c6e4069e62abf31fc3c0f", "in_reply_to": 3734115342, "text": "I renamed it to overlay_view." }, { "t": "2026-08-08T17:26:01Z", "kind": "comment", "who": "andrewtoth", "assoc": "MEMBER", "text": "Thanks for your review @optout21.\n\n[quoted text omitted]\nIt was using a `CCoinsViewCache` before as an ephemeral cache on top of the main cache. This patch changes it to use the `CoinsViewOverlay`. The previous version would insert cache misses into the cache before returning to the `view_dummy` cache, while `CoinsViewOverlay` does not do that. I updated the commit message and PR description to try and clarify that.\n\n[quoted text omitted]\nDone." }, { "t": "2026-08-12T04:11:11Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/bench/block_assemble.cpp", "commit": "2ba5628cb90ea2ae960d3823742412741704e4ca", "in_reply_to": null, "text": "nit: we could `std::move` a few of these (please check the other cases as well):\n```suggestion\n txs.at(b) = MakeTransactionRef(std::move(tx));\n```" }, { "t": "2026-08-12T04:11:58Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/bench/block_assemble.cpp", "commit": "2ba5628cb90ea2ae960d3823742412741704e4ca", "in_reply_to": null, "text": "nit: I don't usually mind this iteration, but above we're using standard loop which makes the two iterations assymetric" }, { "t": "2026-08-16T05:00:43Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/bench/block_assemble.cpp", "commit": "8ebb32e69f8a4959e4d73481c64d7c1282fc9e33", "in_reply_to": null, "text": "The commit message claims we `fill assembled block to max weight`, but we still have leftover transactions - can we assert the claim (and adjust the `FAN_OUT` to make it true? Just make sure the number is not `110`, we're still sensitive to that...)\n\n```suggestion\n assert(GetBlockWeight(*PrepareBlock(test_setup->m_node, options)) > MAX_BLOCK_WEIGHT - DEFAULT_BLOCK_RESERVED_WEIGHT);\n}\n```" }, { "t": "2026-08-16T05:11:16Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/bench/block_assemble.cpp", "commit": "8ebb32e69f8a4959e4d73481c64d7c1282fc9e33", "in_reply_to": null, "text": "[quoted text omitted]\n\nnit: and if we're modifying this we might as well add braces and format it" }, { "t": "2026-08-16T05:13:52Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/bench/block_assemble.cpp", "commit": "8ebb32e69f8a4959e4d73481c64d7c1282fc9e33", "in_reply_to": null, "text": "[quoted text omitted]\n\nwe don't need to copy the existing style:\n```suggestion\n const auto res{test_setup->m_node.chainman->ProcessTransaction(MakeTransactionRef(tx))};\n```" }, { "t": "2026-08-16T05:15:58Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/validation.cpp", "commit": "2ba5628cb90ea2ae960d3823742412741704e4ca", "in_reply_to": null, "text": "[quoted text omitted]\n\n`TestBlockValidity()` previously used a local cache. This PR changes it to reuse `ConnectBlockView()`, which a synchronous `BlockChecked` callback can reach while `ConnectTip()` still has a fetch active.\n\nCould we first record that both paths now select the same overlay?\n\n```C++\nBOOST_AUTO_TEST_CASE(testblockvalidity_with_active_connect_view)\n{\n auto tip_hash{WITH_LOCK(cs_main, return Assert(m_node.chainman)->ActiveTip()->GetBlockHash())};\n auto outer_block{BadBlock(tip_hash)};\n auto candidate_block{GoodBlock(tip_hash)};\n LOCK(cs_main);\n auto& chainstate{Assert(m_node.chainman)->ActiveChainstate()};\n auto reset_guard{chainstate.ConnectBlockView().StartFetching(*outer_block)};\n BOOST_CHECK(&chainstate.TestBlockValidityView() != &chainstate.ConnectBlockView());\n BOOST_CHECK(TestBlockValidity(chainstate, *candidate_block, /*check_pow=*/false, /*check_merkle_root=*/false).IsValid());\n}\n```" }, { "t": "2026-08-16T05:33:47Z", "kind": "review", "who": "l0rinc", "assoc": "MEMBER", "state": "CHANGES_REQUESTED", "commit": "2ba5628cb90ea2ae960d3823742412741704e4ca", "text": "I think the reentrancy issue needs to be addressed before this can be merged.\nI also left a few comments on making the benchmark match its stated workload, plus some nits (feel free to just resolve those, but since we need to push anyway, we might as well clean this up a bit)." }, { "t": "2026-08-23T21:42:50Z", "kind": "review_comment", "who": "andrewtoth", "assoc": "MEMBER", "path": "src/validation.cpp", "commit": "2ba5628cb90ea2ae960d3823742412741704e4ca", "in_reply_to": 3791003663, "text": "[quoted text omitted]\n\nI don't think there is a reentrancy issue? I see though that `BlockChecked` callbacks in `ConnectTip` are synchronous, so a future change could theoretically call `TestBlockValidity` from the callback on the same thread. Nothing does that today. We can prevent this by calling `Assert(overlay_view.GetCacheSize() == 0);` inside `TestBlockValidity`. I'm not sure I see the value of this unit test." }, { "t": "2026-08-26T21:30:14Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/validation.cpp", "commit": "2ba5628cb90ea2ae960d3823742412741704e4ca", "in_reply_to": 3791003663, "text": "Agree, the test is just a demonstration, an assert is the ideal solution here!" } ], "labels_log": [ { "t": "2026-07-19T23:30:41Z", "action": "labeled", "label": "Validation", "who": "DrahtBot" }, { "t": "2026-07-29T03:02:17Z", "action": "labeled", "label": "CI failed", "who": "DrahtBot" }, { "t": "2026-07-29T04:03:21Z", "action": "unlabeled", "label": "CI failed", "who": "DrahtBot" } ], "state_log": [], "text_chars": 10357, "text_tokens_estimate": 2589, "changed_paths": [ "src/bench/block_assemble.cpp", "src/validation.cpp", "src/validation.h" ], "files": [ { "path": "src/bench/block_assemble.cpp", "add": 25, "del": 2 }, { "path": "src/validation.cpp", "add": 6, "del": 5 }, { "path": "src/validation.h", "add": 8, "del": 0 } ], "test_lines": 27, "git": { "head": "2ba5628cb90ea2ae960d3823742412741704e4ca", "head_matches_backup": true, "base": "18c05d93016b28a9afd4c716dfe00b6e0accb30b", "commits": [ { "sha": "8ebb32e69f", "subject": "bench: fill assembled block to max weight in AssembleBlock", "files": 1, "add": 25, "del": 2 }, { "sha": "2ba5628cb9", "subject": "validation: use parallel input prevout fetching in TestBlockValidity", "files": 2, "add": 14, "del": 5 } ], "patch_truncated": false }, "input_hash": "a7a387bbb7a45e1e", "extracted_at": "2026-09-17T16:15:31+00:00" }