{ "number": 36496, "repo": "bitcoin/bitcoin", "url": "https://github.com/bitcoin/bitcoin/pull/36496", "title": "coins: avoid redundant prevout preparation allocations", "author": "l0rinc", "author_association": "MEMBER", "created_at": "2026-10-10T15:14:07Z", "updated_at": "2026-10-31T05:40:59Z", "description_hash": "0b1f49167b995f98", "age_days": 22, "draft": false, "labels": [ "UTXO Db and Indexes" ], "milestone": null, "base": "master", "head_sha": "0977948b35524130ac972efa1c8bf3c35d7d7b85", "head_ref": "l0rinc/coins-reuse-prevout-preparation", "head_repo": "l0rinc/bitcoin", "head_history": [ { "t": "2026-10-10T15:22:19Z", "sha": "b0debcf4daad3c227be25f64ad192ce9663dedc6" }, { "t": "2026-10-15T07:04:27Z", "sha": "483c9d0f448a5bcd6d24e24b00d45a8f22975fb7" }, { "t": "2026-10-22T10:26:09Z", "sha": "45b675358c52a3f8172b0568932291140935c481" }, { "t": "2026-10-22T10:27:47Z", "sha": "0977948b35524130ac972efa1c8bf3c35d7d7b85" } ], "additions": 32, "deletions": 27, "changed_files": 3, "commit_count": 4, "size_bucket": "S", "mergeable_state": "clean", "bot": { "drahtbot": { "present": true, "reviews": { "concept_ack": [ { "login": "hodlinator", "url": "https://github.com/bitcoin/bitcoin/pull/36496#pullrequestreview-5501669620" } ], "stale_ack": [ { "login": "andrewtoth", "url": "https://github.com/bitcoin/bitcoin/pull/36496#pullrequestreview-5578945452" } ] }, "conflicts": [] } }, "acks_parsed": { "hodlinator": { "kind": "concept_ack", "hash": "b0debcf4daad3c227be25f64ad192ce9663dedc6", "t": "2026-10-13T10:01:08Z", "stale": false }, "andrewtoth": { "kind": "ack", "hash": "483c9d0f448a5bcd6d24e24b00d45a8f22975fb7", "t": "2026-10-19T23:23:23Z", "stale": true } }, "acks_tally": { "ack": 0, "stale_ack": 1, "concept_ack": 1, "approach_ack": 0, "nack": 0, "concept_nack": 0, "approach_nack": 0 }, "reviews": { "approved": 1, "changes_requested": 0, "distinct_reviewers": [ "andrewtoth", "hodlinator" ] }, "signals": { "needs_rebase": false, "ci_failed": false, "mergeable_state": "clean", "last_author_activity": "2026-10-22T10:28:04Z", "last_reviewer_activity": "2026-10-22T11:46:30Z", "last_reviewer": "hodlinator", "author_silent_days": 10, "waiting_on_author_days": 10, "days_since_update": 1 }, "refs": { "mentioned": [], "depends_on": [], "fixes": [], "linked_issues": [], "references": [], "conflicts": [] }, "stack": { "shares_commits_with": [], "based_on": [], "base_for": [] }, "review_paths": [ "src/coins.cpp" ], "body": "**Problem:** Block preparation inserts an unused final transaction ID, repeatedly constructs salted transaction-ID sets, and allocates a worker-task vector. Small input queues also submit more tasks than can do useful work.\n\n**Fix:** Insert each transaction ID before scanning the following transaction's inputs. Reuse the set's buckets and hash salt and the worker-task vector across blocks. Limit submitted tasks to the smaller of the worker and queued-input counts.", "commits": [ { "sha": "9c2bcaf10420d3c50122cf6ea7f270a613b6eb63", "date": "2026-10-15T06:37:49Z", "message": "coins: omit final txid from prevout classification\n\nInsert each transaction ID before scanning the following transaction's inputs, omitting the unused final ID.\nThis reduces txid insertions from N to N-1 and replaces the separate coinbase seed and trailing insertion with one insertion at the start of the loop." }, { "sha": "905772446199bc04ad670dccea157523c6d17b09", "date": "2026-10-15T06:37:49Z", "message": "coins: reuse prevout buckets and hash salt\n\nRetain the classification set across blocks to reuse its buckets and avoid reseeding its hasher.\nThe set still allocates and frees its nodes for each block." }, { "sha": "0fdf36a36547672cff02c33b70d2e1457aa7d97d", "date": "2026-10-22T10:27:32Z", "message": "coins: reuse prevout task vector storage\n\n`ThreadPool::Submit` binds its range by reference and moves the individual callables without taking the vector's storage, so keep the vector on the overlay for reuse across blocks." }, { "sha": "0977948b35524130ac972efa1c8bf3c35d7d7b85", "date": "2026-10-22T10:27:32Z", "message": "coins: cap prevout tasks at the input count\n\nEach task drains the shared input queue, so tasks beyond the number of inputs cannot do useful work.\nFor an empty queue, submit an empty range and skip the worker notification inside `ThreadPool::Submit` instead of guarding submission at the caller." } ], "timeline": [ { "t": "2026-10-10T15:22:19Z", "kind": "force_push", "who": "l0rinc", "commit": "b0debcf4daad3c227be25f64ad192ce9663dedc6", "id": "push:b0debcf4daad3c227be25f64ad192ce9663dedc6" }, { "t": "2026-10-13T09:44:35Z", "kind": "review_comment", "who": "hodlinator", "assoc": "MEMBER", "path": "src/coins.cpp", "commit": "696ddc5f281c7701bd3e1576b19c9a15f37a6642", "in_reply_to": null, "text": "[quoted text omitted]\n\nThe vector is cast to an r-value reference. (The template type becoming an l-value is banned through `!std::is_lvalue_reference_v`).\nThe argument to `Submit()` is never used to construct a new vector inside the method, which means the storage of the underlying vector is never currently \"reaped\".\nMy inference is that the `ThreadPool` was written that way since we move from the individual *elements* of the range.\n\nIt would probably be more honest/correct to allow non-const l-values to be passed into `Submit()` and remove the `move()` here. `!std::is_const_v>` could be sufficient, and the l-value/r-value nature of `R` would be up to the caller as was intended when r-value references were introduced.\n\nCompiling diff\n\n```diff\ndiff --git a/src/coins.cpp b/src/coins.cpp\nindex e32e7f21d8..bf8c2bd766 100644\n--- a/src/coins.cpp\n+++ b/src/coins.cpp\n@@ -397,7 +397,7 @@ CCoinsViewCache::ResetGuard CoinsViewOverlay::StartFetching(const CBlock& block\n m_tasks.assign(std::min(workers_count, m_inputs.size()), [this] {\n while (ProcessInput()) {}\n });\n- if (auto futures{m_thread_pool->Submit(std::move(m_tasks))}) {\n+ if (auto futures{m_thread_pool->Submit(m_tasks)}) {\n m_futures = std::move(*futures);\n } else {\n // Submit can fail if a shared owner of the thread pool outside of this class calls Stop() or\ndiff --git a/src/util/threadpool.h b/src/util/threadpool.h\nindex 6c16851525..421f956367 100644\n--- a/src/util/threadpool.h\n+++ b/src/util/threadpool.h\n@@ -216,7 +216,7 @@ public:\n * uncaught exceptions, as they would otherwise be silently discarded.\n */\n template \n- requires(!std::is_lvalue_reference_v)\n+ requires(!std::is_const_v>)\n [[nodiscard]] util::Expected>, SubmitError> Submit(R&& fns) noexcept EXCLUSIVE_LOCKS_REQUIRED(!m_mutex)\n {\n std::vector> futures;\n```", "id": "rc:4257097025", "url": "https://github.com/bitcoin/bitcoin/pull/36496#discussion_r4257097025", "hash": "0710d4dc38f87d12", "edited": "2026-10-13T10:40:49Z" }, { "t": "2026-10-13T10:01:08Z", "kind": "review", "who": "hodlinator", "assoc": "MEMBER", "state": "COMMENTED", "commit": "b0debcf4daad3c227be25f64ad192ce9663dedc6", "text": "Concept ACK b0debcf4daad3c227be25f64ad192ce9663dedc6", "id": "r:5501669620", "url": "https://github.com/bitcoin/bitcoin/pull/36496#pullrequestreview-5501669620", "hash": "a44d36b484aae32f", "edited": null }, { "t": "2026-10-15T04:23:31Z", "kind": "comment", "who": "andrewtoth", "assoc": "MEMBER", "text": "ACK b0debcf4daad3c227be25f64ad192ce9663dedc6", "id": "c:6184363683", "url": "https://github.com/bitcoin/bitcoin/pull/36496#issuecomment-6184363683", "hash": "5a6851a43e6a41a2", "edited": null }, { "t": "2026-10-15T07:04:27Z", "kind": "force_push", "who": "l0rinc", "commit": "483c9d0f448a5bcd6d24e24b00d45a8f22975fb7", "id": "push:483c9d0f448a5bcd6d24e24b00d45a8f22975fb7" }, { "t": "2026-10-15T07:05:20Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/coins.cpp", "commit": "696ddc5f281c7701bd3e1576b19c9a15f37a6642", "in_reply_to": 4257097025, "text": "Thanks, added you as coauthor and also handled the empty tasks internally", "id": "rc:4280049366", "url": "https://github.com/bitcoin/bitcoin/pull/36496#discussion_r4280049366", "hash": "3d9f8a2171784960", "edited": "2026-10-15T07:05:27Z" }, { "t": "2026-10-19T23:23:23Z", "kind": "review", "who": "andrewtoth", "assoc": "MEMBER", "state": "APPROVED", "commit": "483c9d0f448a5bcd6d24e24b00d45a8f22975fb7", "text": "reACK 483c9d0f448a5bcd6d24e24b00d45a8f22975fb7", "id": "r:5578945452", "url": "https://github.com/bitcoin/bitcoin/pull/36496#pullrequestreview-5578945452", "hash": "9e23e819dbc20d25", "edited": null }, { "t": "2026-10-22T07:48:59Z", "kind": "review_comment", "who": "hodlinator", "assoc": "MEMBER", "path": "src/coins.cpp", "commit": "696ddc5f281c7701bd3e1576b19c9a15f37a6642", "in_reply_to": 4257097025, "text": "Upon further consideration, influenced by on Ralph's feedback in https://git.fish.foo/bitcoin/bitcoin/pulls/36496#issuecomment-610179; it may be safer to revert my suggestion and instead opt for adding `Assume(m_tasks.capacity() >= std::min(workers_count, m_inputs.size()))` with an explanatory comment like so:\n\n```diff\ndiff --git a/src/coins.cpp b/src/coins.cpp\nindex cdc5a8858d..88bea6c391 100644\n--- a/src/coins.cpp\n+++ b/src/coins.cpp\n@@ -395,7 +395,10 @@ CCoinsViewCache::ResetGuard CoinsViewOverlay::StartFetching(const CBlock& block\n m_tasks.assign(std::min(workers_count, m_inputs.size()), [this] {\n while (ProcessInput()) {}\n });\n- if (auto futures{m_thread_pool->Submit(m_tasks)}) {\n+ if (auto futures{m_thread_pool->Submit(std::move(m_tasks))}) {\n+ // We expect to be able to reuse the backing array.\n+ Assume(m_tasks.capacity() >= std::min(workers_count, m_inputs.size()));\n+\n m_futures = std::move(*futures);\n } else {\n // Submit can fail if a shared owner of the thread pool outside of this class calls Stop() or\ndiff --git a/src/test/threadpool_tests.cpp b/src/test/threadpool_tests.cpp\nindex 3f802b754f..4811d73215 100644\n--- a/src/test/threadpool_tests.cpp\n+++ b/src/test/threadpool_tests.cpp\n@@ -469,7 +469,6 @@ BOOST_AUTO_TEST_CASE(submit_range_of_tasks_complete_successfully)\n std::array, static_cast(num_tasks)> array_tasks;\n std::vector> vector_tasks;\n vector_tasks.reserve(static_cast(num_tasks));\n- BOOST_CHECK(Assert(threadPool.Submit(vector_tasks))->empty());\n for (const auto i : std::views::iota(int32_t{1}, num_tasks + 1)) {\n array_tasks.at(static_cast(i - 1)) = [i, square] { return square(i); };\n vector_tasks.emplace_back([i, square] { return square(i); });\n@@ -477,7 +476,7 @@ BOOST_AUTO_TEST_CASE(submit_range_of_tasks_complete_successfully)\n\n auto futures{std::move(*Assert(threadPool.Submit(std::move(array_tasks))))};\n BOOST_CHECK_EQUAL(futures.size(), static_cast(num_tasks));\n- std::ranges::move(*Assert(threadPool.Submit(vector_tasks)), std::back_inserter(futures));\n+ std::ranges::move(*Assert(threadPool.Submit(std::move(vector_tasks))), std::back_inserter(futures));\n BOOST_CHECK_EQUAL(futures.size(), static_cast(num_tasks * 2));\n\n auto squares_sum{0};\ndiff --git a/src/util/threadpool.h b/src/util/threadpool.h\nindex 80971009ee..c3c11c53ac 100644\n--- a/src/util/threadpool.h\n+++ b/src/util/threadpool.h\n@@ -216,7 +216,7 @@ public:\n * uncaught exceptions, as they would otherwise be silently discarded.\n */\n template \n- requires(!std::is_const_v>)\n+ requires(!std::is_lvalue_reference_v)\n [[nodiscard]] util::Expected>, SubmitError> Submit(R&& fns) noexcept EXCLUSIVE_LOCKS_REQUIRED(!m_mutex)\n {\n std::vector> futures;\n```\n\nRan functional tests to affirm that the `Assume()` holds.", "id": "rc:4345793434", "url": "https://github.com/bitcoin/bitcoin/pull/36496#discussion_r4345793434", "hash": "a238235c8080bb70", "edited": "2026-10-22T08:02:26Z" }, { "t": "2026-10-22T09:34:25Z", "kind": "comment", "who": "hodlinator", "assoc": "MEMBER", "text": "Performed some measurements which ended up with an increased average run time but a decreased minimum run time. They leave me not worrying too much about unexpected pessimization.\n\n#### Before/649bc0a645eb127a9b1e6d078f8d129be6380c87 (PR base)\n\n```\n\u20bf hyperfine --runs=3 \"./build/bin/bitcoind -stopatheight=700000 -reindex-chainstate\"\nBenchmark 1: ./build/bin/bitcoind -stopatheight=700000 -reindex-chainstate\n Time (mean \u00b1 \u03c3): 5747.993 s \u00b1 8.252 s [User: 12214.062 s, System: 866.596 s]\n Range (min \u2026 max): 5738.535 s \u2026 5753.728 s 3 runs\n```\n\n#### After/483c9d0f448a5bcd6d24e24b00d45a8f22975fb7 (PR tip)\n\n```\n\u20bf hyperfine --runs=3 \"./build/bin/bitcoind -stopatheight=700000 -reindex-chainstate\"\nBenchmark 1: ./build/bin/bitcoind -stopatheight=700000 -reindex-chainstate\n Time (mean \u00b1 \u03c3): 5781.568 s \u00b1 117.178 s [User: 12371.366 s, System: 875.421 s]\n Range (min \u2026 max): 5705.545 s \u2026 5916.514 s 3 runs\n```", "id": "c:6337553644", "url": "https://github.com/bitcoin/bitcoin/pull/36496#issuecomment-6337553644", "hash": "b448b6864a384d9f", "edited": null }, { "t": "2026-10-22T10:26:09Z", "kind": "force_push", "who": "l0rinc", "commit": "45b675358c52a3f8172b0568932291140935c481", "id": "push:45b675358c52a3f8172b0568932291140935c481" }, { "t": "2026-10-22T10:27:47Z", "kind": "force_push", "who": "l0rinc", "commit": "0977948b35524130ac972efa1c8bf3c35d7d7b85", "id": "push:0977948b35524130ac972efa1c8bf3c35d7d7b85" }, { "t": "2026-10-22T10:28:04Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/coins.cpp", "commit": "696ddc5f281c7701bd3e1576b19c9a15f37a6642", "in_reply_to": 4257097025, "text": "Changed back to keep task consumption explicit without broadening the API.\nSkipped the capacity check since `Submit` preserves the storage and `assign()` handles any allocation needed.", "id": "rc:4347226109", "url": "https://github.com/bitcoin/bitcoin/pull/36496#discussion_r4347226109", "hash": "f8100128e4bcbe7b", "edited": "2026-10-22T10:28:05Z" }, { "t": "2026-10-22T11:35:20Z", "kind": "review_comment", "who": "hodlinator", "assoc": "MEMBER", "path": "src/coins.cpp", "commit": "696ddc5f281c7701bd3e1576b19c9a15f37a6642", "in_reply_to": 4257097025, "text": "Still think it's worth adding the `Assume()` to catch potential changes in behavior of how `Submit()` uses the argument. Agree the comment was overkill as the reason for `m_tasks` being a member is already documented in the header. Not going to insist again though.", "id": "rc:4347819028", "url": "https://github.com/bitcoin/bitcoin/pull/36496#discussion_r4347819028", "hash": "30874255027dbbfe", "edited": "2026-10-22T11:54:36Z" }, { "t": "2026-10-22T11:46:30Z", "kind": "review_comment", "who": "hodlinator", "assoc": "MEMBER", "path": "src/coins.cpp", "commit": "0977948b35524130ac972efa1c8bf3c35d7d7b85", "in_reply_to": null, "text": "nit: We will now perform more work when we have blocks with only have a coinbase tx (previously we would not call `ThreadPool::Submit()`. In the case that bitcoin as a whole succeeds those blocks are outliers, so I agree with that part of the change.\n\nHowever, the warning in the `else`-block becomes somewhat weird when `m_tasks` is empty (failing to submit 0 tasks). How about bringing the `m_inputs.size()` check back for that?\n```suggestion\n } else if (m_inputs.size()) {\n // Submit can fail if a shared owner of the thread pool outside of this class calls Stop() or\n // Interrupt() on a different thread after we call WorkersCount() above. In that case parallel\n // fetching will not make progress, so we clear the inputs to fall back to single threaded fetching.\n LogWarning(\"Failed to submit prevout fetch tasks (%s); falling back to single-threaded fetching for this block.\", SubmitErrorString(futures.error()));\n m_inputs.clear();\n StopFetching(); // Assert nothing changed if we failed to start tasks.\n }\n```", "id": "rc:4347916622", "url": "https://github.com/bitcoin/bitcoin/pull/36496#discussion_r4347916622", "hash": "c33b01dc340d91c4", "edited": "2026-10-22T11:54:36Z" } ], "labels_log": [ { "t": "2026-10-10T15:14:10Z", "action": "labeled", "label": "UTXO Db and Indexes", "who": "DrahtBot" }, { "t": "2026-10-10T15:22:50Z", "action": "labeled", "label": "CI failed", "who": "DrahtBot" }, { "t": "2026-10-10T16:46:34Z", "action": "unlabeled", "label": "CI failed", "who": "DrahtBot" }, { "t": "2026-10-22T10:28:50Z", "action": "labeled", "label": "CI failed", "who": "DrahtBot" }, { "t": "2026-10-22T11:24:55Z", "action": "unlabeled", "label": "CI failed", "who": "DrahtBot" } ], "state_log": [ { "t": "2026-10-11T23:49:40Z", "kind": "ready_for_review", "who": "l0rinc" } ], "text_chars": 9471, "text_tokens_estimate": 2367, "changed_paths": [ "src/coins.cpp", "src/coins.h", "src/util/threadpool.h" ], "files": [ { "path": "src/coins.cpp", "add": 21, "del": 24 }, { "path": "src/coins.h", "add": 10, "del": 2 }, { "path": "src/util/threadpool.h", "add": 1, "del": 1 } ], "test_lines": 0, "git": { "head": "0977948b35524130ac972efa1c8bf3c35d7d7b85", "head_matches_backup": true, "base": "649bc0a645eb127a9b1e6d078f8d129be6380c87", "patch_id": "d9a240d6974d2d72", "commits": [ { "sha": "9c2bcaf104", "subject": "coins: omit final txid from prevout classification", "files": 1, "add": 5, "del": 6 }, { "sha": "9057724461", "subject": "coins: reuse prevout buckets and hash salt", "files": 2, "add": 12, "del": 6 }, { "sha": "0fdf36a365", "subject": "coins: reuse prevout task vector storage", "files": 2, "add": 6, "del": 3 }, { "sha": "0977948b35", "subject": "coins: cap prevout tasks at the input count", "files": 2, "add": 13, "del": 16 } ], "patch_truncated": false }, "patch_id": "d9a240d6974d2d72", "input_hash": "5c560eb319b47dd3", "extracted_at": "2026-11-02T01:09:49+00:00" }