{ "number": 33922, "repo": "bitcoin/bitcoin", "url": "https://github.com/bitcoin/bitcoin/pull/33922", "title": "mining: add getMemoryLoad() and track template non-mempool memory footprint", "author": "Sjors", "author_association": "MEMBER", "created_at": "2025-11-21T15:34:14Z", "updated_at": "2026-09-17T08:02:32Z", "age_days": 300, "draft": true, "labels": [ "Mining" ], "milestone": null, "base": "master", "head_sha": "1647b291ccae0c0c5c79f0117adcbb0d73231221", "head_ref": "2025/11/ipc-memusage", "head_repo": "Sjors/bitcoin", "head_history": [ { "t": "2025-11-21T16:04:54Z", "sha": "f22413f31f32b6985b0090d10ab5d62ba91595cb" }, { "t": "2025-11-21T16:22:21Z", "sha": "3b77529f74b76ea52c1b7b4ac4edacb0947ff202" }, { "t": "2025-11-25T15:47:05Z", "sha": "24592b737fb80f07276a88fbcf8eaf0ee64c7e49" }, { "t": "2025-11-25T17:21:57Z", "sha": "03dcfaeb5892fa877eaae0fe6d6942943b7d9b95" }, { "t": "2025-11-25T18:13:13Z", "sha": "ac1e97a59279b2407e5f64bd684170587bae88dc" }, { "t": "2025-12-03T09:36:11Z", "sha": "e8f8f7f677bcde0179526be3ed9a657c44998b93" }, { "t": "2025-12-05T11:09:09Z", "sha": "e75406dccc39b1f5d988994083e7fd959fa2a49a" }, { "t": "2025-12-05T11:13:53Z", "sha": "9a03d3837f2532619f8249e4a062453b333c0692" }, { "t": "2025-12-05T11:18:27Z", "sha": "7f4249b0d0a126cfaddba48bc1bd4966c14fbd0d" }, { "t": "2025-12-19T10:24:15Z", "sha": "71795a21e6ec854f2f450ce04cc6ff9e108a2c89" }, { "t": "2026-01-14T09:54:27Z", "sha": "6fb639a089d83a2e4e4eeca26f3041d2da15d0c7" }, { "t": "2026-02-03T13:32:38Z", "sha": "c518fdf2dec42b6e52eff172816a097738389d16" }, { "t": "2026-02-19T11:04:55Z", "sha": "6cfd0f29e544c85c175a4ea6bf9db64fe57cdc05" }, { "t": "2026-02-23T10:51:15Z", "sha": "2ba0f0ba36a23665c23e190b60282dade1108bbf" }, { "t": "2026-02-24T12:17:03Z", "sha": "c37d715c1d32020fe16ddc92f7ca95f585cc6b4b" }, { "t": "2026-03-05T19:58:25Z", "sha": "04553cd6612f08d757fb20287fe111007bfb6db1" }, { "t": "2026-04-08T09:28:48Z", "sha": "430990c0c5bc72f8b64e76056b304b88821b9af0" }, { "t": "2026-04-08T09:36:05Z", "sha": "10acf818ed70d4cd2af864278a35a6b64de0bb16" }, { "t": "2026-04-14T14:56:16Z", "sha": "d1b9df5805b9219217b7b22eedaf1fcb5704e35d" }, { "t": "2026-04-14T18:24:33Z", "sha": "cd1fce108dca9c02abf76cbeb009ed6c2cfa8cc9" }, { "t": "2026-04-14T18:39:33Z", "sha": "e8aaada3f2f8396b47eda9fad576ab89dbdf80ab" }, { "t": "2026-04-17T13:01:06Z", "sha": "bf49247b35f43ed9bcb769ea0f9a8eba2ce082d5" }, { "t": "2026-04-17T13:02:18Z", "sha": "4673836a656a2638292fd02453e5d8ef42cd8eab" }, { "t": "2026-04-17T13:10:21Z", "sha": "5094af74bc5280b89835654ff7a7604d57a31d8c" }, { "t": "2026-04-17T13:21:17Z", "sha": "ceb5b70a53c43fd31dbaaaf5737e6550c2fe51e6" }, { "t": "2026-04-17T13:29:41Z", "sha": "e23a8f3015d8b3c682c45adb3cb39db912f7d1a9" }, { "t": "2026-04-17T14:42:01Z", "sha": "79a74da860a557af73788f277306a4aee67bc50a" }, { "t": "2026-04-17T15:26:55Z", "sha": "b89942d29f1e71429070b2d1140d2792eb33bf47" }, { "t": "2026-04-17T19:49:18Z", "sha": "d37e0fc16c235ff511bd9485ef505f0c5615d9e4" }, { "t": "2026-05-22T07:40:04Z", "sha": "9f353d56735293677c1db92e639ef034c615eedd" }, { "t": "2026-05-26T16:03:27Z", "sha": "39254e9edf50781fffa613966f886809595aa998" }, { "t": "2026-05-28T17:58:47Z", "sha": "c82ccf896759b0d89a953dfb0c66fda176f22afc" }, { "t": "2026-06-24T09:45:43Z", "sha": "3cb6c4779e20f183347c3c1d63d8abb9d2e4c4ee" }, { "t": "2026-06-24T19:37:28Z", "sha": "8b4967e3570a7adce2e73c3476bcdd5d676cf090" }, { "t": "2026-07-08T07:58:27Z", "sha": "837060b034e65ceabed52e35f9e4bcc276a4bafb" }, { "t": "2026-07-08T10:19:38Z", "sha": "d1f9b11b40661bb49d19333e9fcc43d5a969eb3f" }, { "t": "2026-07-16T13:55:00Z", "sha": "ef901fa30831b99ca1892bf7949e7bd92a42460b" }, { "t": "2026-07-16T15:14:19Z", "sha": "239af38f706835a5a5ca10176a4bcda45196d919" }, { "t": "2026-07-23T17:37:26Z", "sha": "794a8a04dd6bfeac3553a71c24f21be9d0afa38b" }, { "t": "2026-07-23T18:04:26Z", "sha": "1f569895ea25814e46ba53f94efa618c7bc9eacc" }, { "t": "2026-07-24T13:25:55Z", "sha": "de2c7a8f975f0309bb7eb9c86675ce825342cf4f" }, { "t": "2026-08-20T09:23:04Z", "sha": "c10e77f9897ed74ca9f3cf3085cd1eabb956fbe3" }, { "t": "2026-09-09T07:44:37Z", "sha": "4259dde8f195b71e0282bb8f82d5009107945aa2" }, { "t": "2026-09-14T09:56:27Z", "sha": "11739697d0b6d8015c2746dd3d42866a694bcae3" }, { "t": "2026-09-14T12:47:55Z", "sha": "1647b291ccae0c0c5c79f0117adcbb0d73231221" } ], "additions": 1087, "deletions": 463, "changed_files": 34, "commit_count": 19, "size_bucket": "XL", "mergeable_state": "clean", "bot": { "drahtbot": { "present": true, "reviews": { "concept_ack": [ { "login": "ismaelsadeeq", "url": "https://github.com/bitcoin/bitcoin/pull/33922#pullrequestreview-3493499544" }, { "login": "ryanofsky", "url": "https://github.com/bitcoin/bitcoin/pull/33922#issuecomment-3608940825" } ], "stale_ack": [ { "login": "enirox001", "url": "https://github.com/bitcoin/bitcoin/pull/33922#pullrequestreview-4061578431" }, { "login": "vasild", "url": "https://github.com/bitcoin/bitcoin/pull/33922#pullrequestreview-4179948443" } ] }, "conflicts": [ { "number": 36257, "title": "qa: assert_equals -> assert_true/assert_false", "author": "hodlinator" }, { "number": 36244, "title": "validation, net: Process blocks asynchronously and reduce cs_main contention", "author": "w0xlt" }, { "number": 36188, "title": "crypto: plug hardware optimized SHA256 into libsecp256k1's context", "author": "furszy" }, { "number": 36109, "title": "test: harden arbitrary-parent block creation", "author": "l0rinc" }, { "number": 36097, "title": "mining: replace interrupt methods with cancellation arguments", "author": "xyzconstant" }, { "number": 36047, "title": "rpc: handle createNewBlock() returning nullptr during shutdown", "author": "chriszeng1010" }, { "number": 36000, "title": "validation: prefetch blocks while connecting", "author": "l0rinc" }, { "number": 35671, "title": "mining: add TxCollection to bandwidth-efficiently validate external block templates", "author": "Sjors" }, { "number": 35646, "title": "RFC: Separate out runtime errors from BlockValidationState using `util::Expected`", "author": "yuvicc" }, { "number": 35581, "title": "node: add block template manager and track waitNext fee inflow", "author": "ismaelsadeeq" }, { "number": 35569, "title": "Encapsulation for CTransaction", "author": "purpleKarrot" }, { "number": 35511, "title": "RFC: consensus: Make `CAmount` a class", "author": "hodlinator" }, { "number": 32468, "title": "rpc: generateblock to allow multiple outputs", "author": "polespinasa" }, { "number": 31117, "title": "miner: Reorg Testnet4 minimum difficulty blocks", "author": "fjahr" }, { "number": 29700, "title": "kernel, refactor: return error status on all fatal errors", "author": "ryanofsky" } ] } }, "acks_parsed": { "ismaelsadeeq": { "kind": "concept_ack", "hash": null, "t": "2025-11-21T16:22:59Z", "stale": false }, "ryanofsky": { "kind": "concept_ack", "hash": "e8f8f7f677bcde0179526be3ed9a657c44998b93", "t": "2025-12-03T21:36:31Z", "stale": false }, "vasild": { "kind": "ack", "hash": "d37e0fc16c235ff511bd9485ef505f0c5615d9e4", "t": "2026-04-27T10:04:18Z", "stale": true }, "enirox001": { "kind": "ack", "hash": "04553cd", "t": "2026-04-06T10:52:35Z", "stale": true } }, "acks_tally": { "ack": 0, "stale_ack": 2, "concept_ack": 2, "approach_ack": 0, "nack": 0, "concept_nack": 0, "approach_nack": 0 }, "reviews": { "approved": 4, "changes_requested": 0, "distinct_reviewers": [ "brunoerg", "enirox001", "hebasto", "ismaelsadeeq", "maflcko", "ryanofsky", "vasild" ] }, "signals": { "needs_rebase": false, "ci_failed": false, "mergeable_state": "clean", "last_author_activity": "2026-09-14T12:47:55Z", "last_reviewer_activity": "2026-04-27T10:04:18Z", "last_reviewer": "vasild", "author_silent_days": 3, "waiting_on_author_days": 0, "days_since_update": 0 }, "refs": { "mentioned": [ 33421, 33819, 33899, 33965, 33966, 34003, 34020, 34184, 34422, 34452, 34568, 34644, 34646, 34806, 34896, 35073, 35101, 35403, 35581, 35598, 35675 ], "depends_on": [], "fixes": [], "linked_issues": [], "references": [ { "number": 33899, "type": "issue", "state": "open", "merged": false, "merged_at": null, "title": "Block template memory management (for IPC clients)" }, { "number": 35675, "type": "pull", "state": "open", "merged": false, "merged_at": null, "title": "mining: add block template manager" }, { "number": 33421, "type": "pull", "state": "closed", "merged": false, "merged_at": null, "title": "node: add `BlockTemplateCache`" }, { "number": 33819, "type": "pull", "state": "closed", "merged": true, "merged_at": "2026-01-13", "title": "mining: getCoinbase() returns struct instead of raw tx" }, { "number": 33965, "type": "pull", "state": "closed", "merged": true, "merged_at": "2026-02-12", "title": "mining: fix -blockreservedweight shadows IPC option" }, { "number": 33966, "type": "pull", "state": "closed", "merged": true, "merged_at": "2026-05-26", "title": "refactor: disentangle miner startup defaults from runtime options" }, { "number": 34003, "type": "pull", "state": "closed", "merged": true, "merged_at": "2025-12-16", "title": "test: interface_ipc.py minor fixes and cleanup" }, { "number": 34020, "type": "pull", "state": "closed", "merged": true, "merged_at": "2026-07-07", "title": "mining: add getTransactions(ByWitnessID) IPC methods" }, { "number": 34184, "type": "pull", "state": "closed", "merged": true, "merged_at": "2026-02-24", "title": "mining: add cooldown to createNewBlock() immediately after IBD" }, { "number": 34422, "type": "pull", "state": "closed", "merged": true, "merged_at": "2026-03-03", "title": "Update libmultiprocess subtree to be more stable with rust IPC client" }, { "number": 34452, "type": "pull", "state": "closed", "merged": true, "merged_at": "2026-02-07", "title": "test: split interface_ipc.py" }, { "number": 34568, "type": "pull", "state": "closed", "merged": true, "merged_at": "2026-02-20", "title": "mining: Break compatibility with existing IPC mining clients" }, { "number": 34644, "type": "pull", "state": "closed", "merged": true, "merged_at": "2026-05-28", "title": "mining: add submitBlock to IPC Mining interface" }, { "number": 34646, "type": "pull", "state": "closed", "merged": true, "merged_at": "2026-02-23", "title": "Fix two issues in p2p_private_broadcast.py" }, { "number": 34806, "type": "pull", "state": "closed", "merged": true, "merged_at": "2026-05-22", "title": "refactor: logging: Various API improvements" }, { "number": 34896, "type": "pull", "state": "closed", "merged": true, "merged_at": "2026-03-25", "title": "ci: Upgrade IWYU to 0.26 compatible with Clang 22" }, { "number": 35073, "type": "pull", "state": "closed", "merged": false, "merged_at": null, "title": "iwyu: map std::hash to " }, { "number": 35101, "type": "pull", "state": "closed", "merged": true, "merged_at": "2026-06-10", "title": "refactor: disable default std::hash for CTransactionRef" }, { "number": 35403, "type": "pull", "state": "closed", "merged": true, "merged_at": "2026-06-23", "title": "mining: pr 33966 followups (disentangle miner startup defaults)" }, { "number": 35581, "type": "pull", "state": "open", "merged": false, "merged_at": null, "title": "node: add block template manager and track waitNext fee inflow" }, { "number": 35598, "type": "pull", "state": "open", "merged": false, "merged_at": null, "title": "test: cover feeThreshold = MAX_MONEY in interface_ipc_mining.py" } ], "conflicts": [ 36257, 36244, 36188, 36109, 36097, 36047, 36000, 35671, 35646, 35581, 35569, 35511, 32468, 31117, 29700 ] }, "stack": { "shares_commits_with": [ 35581, 35675 ], "based_on": [ 35675 ], "base_for": [] }, "review_paths": [ "src/node/context.h", "src/node/interfaces.cpp", "src/node/miner.h", "src/node/types.h", "src/primitives/transaction.h", "src/rpc/mining.cpp", "test/functional/interface_ipc.py" ], "body": "Implements a way to track the memory footprint of all non-mempool transactions that are still being referenced by block templates, see discussion in #33899. It does not impose a limit.\n\nIPC clients can query this footprint (total, across all clients) using the `getMemoryLoad()` IPC method. Its client-side usage is demonstrated here:\n- https://github.com/stratum-mining/sv2-tp/pull/63\n\nAdditionally, the functional test in `interface_ipc.py` is expanded to demonstrate how template memory management works: templates are not released until the client drops references to them, or calls the template destroy method, or disconnects. The destroy method is called automatically by clients using [libmultiprocess](https://github.com/bitcoin-core/libmultiprocess), as [sv2-tp](https://github.com/stratum-mining/sv2-tp) does. In the Python tests it also happens when references are destroyed or go out of scope.\n\nBased on:\n\n- #35675\n\nCommits:\n\n- _refactor: move CTransactionRefComp to util/hasher_\n- _mining: track non-mempool memory usage_: add `TxTemplateMap` to `BlockTemplateManager` to track how many templates contain any given transaction. This map is updated by the `BlockTemplate` constructor and destructor.\n- _mining: add GetTemplateMemoryUsage()_ - loops over this map and sums up the memory footprint for transactions outside the mempool (includes a fuzzer)\n- _ipc: add getMemoryLoad()_ expose this information to IPC clients and add test coverage", "commits": [ { "sha": "482a5d27124c4241c27ed323659914cd38409650", "date": "2026-09-13T10:33:47Z", "message": "node: introduce BlockTemplateManager\n\nAdd BlockTemplateManager, a wrapper around\nBlockAssembler::CreateNewBlock(), and store it in NodeContext.\n\nWire it into node init and test setup so it is reset before its\nmempool/chainman dependencies, and update tests and fuzz setups that\nrebuild chainman or mempool. Add a unit test that verifies a block\ntemplate can be created through the manager." }, { "sha": "bcdadf2b7111a061d2bbeba350233db7b258a787", "date": "2026-09-13T11:28:23Z", "message": "node: move mining_args to BlockTemplateManager\n\nPass the parsed mining args to BlockTemplateManager at construction\nand expose them via BlockCreateArgs(), so the manager owns the\ninit-time block create options instead of NodeContext." }, { "sha": "07964c525d4769d00004b9942225a2cb1306e09d", "date": "2026-09-13T11:28:23Z", "message": "miner: move SubmitBlock into BlockTemplateManager\n\nMove SubmitBlockStateCatcher and SubmitBlock from miner.cpp into\nBlockTemplateManager as a member function. This groups block submission\nwith block creation in the same class. The function uses m_chainman\ndirectly instead of taking it as a parameter." }, { "sha": "037d3519f2ff9937aa82060a42939aafab4179f1", "date": "2026-09-13T11:28:25Z", "message": "node: move tip and wait helpers into BlockTemplateManager\n\nMove the mining tip lookup and block-template waiting helpers (GetTip,\nWaitTipChanged, WaitAndCreateNewBlock, InterruptWait,\nCooldownIfHeadersAhead) into BlockTemplateManager so the manager owns\nthe template waiting flow. The manager now takes KernelNotifications\nat construction." }, { "sha": "400ac328c91b653e793bd140b89e8fabfb42e3ac", "date": "2026-09-13T11:28:26Z", "message": "interfaces: create block template via BlockTemplateManager\n\nRoute the Mining interface's createNewBlock() through\nBlockTemplateManager::CreateNewTemplate() instead of constructing a\nBlockAssembler directly. Merging the init-time defaults into unset\noptions now happens inside CreateNewTemplate(), so every caller gets\nthem applied." }, { "sha": "011cbfbcd2e3a3282ae7e5afdfd923bc444d1845", "date": "2026-09-13T11:28:27Z", "message": "rpc: route getblocktemplate internals through node\n\nAdd EnsureBlockTemplateManager() and use it in getblocktemplate for tip\nlookup and longpoll waiting, while using ChainstateManager directly for\ntest-chain and IBD checks." }, { "sha": "0998c0bbdfcc3c3f7c105a1fb7223bace529556a", "date": "2026-09-13T11:28:28Z", "message": "rpc: build getblocktemplate via BlockTemplateManager" }, { "sha": "951fac807d37302e7de5e289c0ab05f8dfcb1e1c", "date": "2026-09-13T11:28:29Z", "message": "rpc: do not copy template data in getblocktemplate\n\nThe transactions, fees and sigops costs, and the coinbase outputs\nare only read, so reference them from the template instead of copying\nthem out of it." }, { "sha": "38556f817e060187b48cd31490f578cdf63134ee", "date": "2026-09-13T11:28:29Z", "message": "rpc: only copy the header in getblocktemplate\n\nThe block copy exists solely to apply header adjustments (time, nonce,\nversion bits) that must not mutate the cached template. Now that the\ntransactions are read directly from the template, copy only the header\ninstead of the entire block." }, { "sha": "8372721f20e2ea2c15bae38a046476736169d801", "date": "2026-09-13T11:28:30Z", "message": "rpc: build generation templates via BlockTemplateManager\n\ngenerateblock is no longer safe for fuzzing: it previously threw at\nEnsureMining (node.mining is never set in the fuzz setup), but with\nEnsureBlockTemplateManager it now executes, mining and submitting a\nreal block. Submission mutates the chain state shared across fuzz\niterations and writes the block to disk." }, { "sha": "ec4f24ef181a7987bcea72a21e507f59dc1b8126", "date": "2026-09-13T11:28:31Z", "message": "rpc: wait for tips via BlockTemplateManager\n\nThe waitforblock, waitforblockheight and waitfornewblock RPCs are no\nlonger safe for fuzzing: they previously threw at EnsureMining\n(node.mining is never set in the fuzz setup), but with\nEnsureBlockTemplateManager they now really wait. The tip never changes\nduring fuzzing and nothing interrupts the wait, so a call without a\ntimeout blocks forever." }, { "sha": "6c71166bcfa4339d5019f2b27d477eb45ce7fe86", "date": "2026-09-13T11:28:32Z", "message": "test: create templates via BlockTemplateManager" }, { "sha": "ffe3a631f38b1224db8b0ea246a3c976f358c59e", "date": "2026-09-13T11:28:33Z", "message": "node: remove NodeContext::mining and EnsureMining" }, { "sha": "3d7d190b6995b3c0adb6528c2d9eac81a8f839d9", "date": "2026-09-13T11:28:34Z", "message": "ci: enforce iwyu for block template manager" }, { "sha": "e8b283c35a656f808250e4b32d3494f49a000b08", "date": "2026-09-13T11:28:35Z", "message": "test: fuzz BlockTemplateManager" }, { "sha": "f9d9aac99a40dce53cce243d1652d76e646a92ef", "date": "2026-09-14T12:44:19Z", "message": "refactor: move CTransactionRefComp to util/hasher" }, { "sha": "4b10fc327fd1e1c1c397bc5a8f07b54bd7e61193", "date": "2026-09-14T12:44:19Z", "message": "mining: track non-mempool memory usage\n\nIPC clients can hold on to block templates indefinately, which has the\nsame impact as when the node holds a shared pointer to the\nCBlockTemplate. Because each template in turn tracks CTransactionRefs,\ntransactions that are removed from the mempool will not have\ntheir memory cleared.\n\nThis commit adds bookkeeping to the block template constructor and\ndestructor that will let us track the resulting memory footprint.\n\nCo-authored-by: Vasil Dimov " }, { "sha": "063ae8400ac71c190c1328f19afa7c3eff789df5", "date": "2026-09-14T12:44:46Z", "message": "mining: add GetTemplateMemoryUsage()\n\nCalculate the non-mempool memory footprint for template transaction\nreferences.\n\nAdd bench logging to collect data on whether caching or simplified\nheuristics are needed, such as not checking for mempool presence.\n\nCheck the calculation in the block template manager fuzz test." }, { "sha": "1647b291ccae0c0c5c79f0117adcbb0d73231221", "date": "2026-09-14T12:44:47Z", "message": "ipc: add getMemoryLoad()\n\nAllow IPC clients to inspect the amount of memory consumed by\nnon-mempool transactions in blocks.\n\nReturns a MemoryLoad struct which can later be expanded to e.g.\ninclude a limit.\n\nExpand the interface_ipc.py test to demonstrate the behavior and\nto illustrate how clients can call destroy() to reduce memory\npressure." } ], "timeline": [ { "t": "2025-11-21T15:36:54Z", "kind": "comment", "who": "Sjors", "assoc": "MEMBER", "text": "I haven't benchmarked this yet on mainnet, so I'm not sure if checking every (unique) transaction for mempool presence is unacceptably expensive.\n\nIf people prefer, I could also add a way for the `getblocktemplate` RPC to opt-out of the memory bookkeeping, since it holds on to one template max and no longer than a minute." }, { "t": "2025-11-21T16:04:54Z", "kind": "force_push", "who": "Sjors", "commit": "f22413f31f32b6985b0090d10ab5d62ba91595cb" }, { "t": "2025-11-21T16:17:32Z", "kind": "review_comment", "who": "Sjors", "assoc": "MEMBER", "path": "src/node/interfaces.cpp", "commit": "8b4967e3570a7adce2e73c3476bcdd5d676cf090", "in_reply_to": null, "text": "b9306b79b8f5667a2679236af8792bb1c36db817: in addition, we might be wiping the dummy coinbase from the template later: https://github.com/Sjors/bitcoin/pull/106" }, { "t": "2025-11-21T16:22:21Z", "kind": "force_push", "who": "Sjors", "commit": "3b77529f74b76ea52c1b7b4ac4edacb0947ff202" }, { "t": "2025-11-21T16:22:59Z", "kind": "review", "who": "ismaelsadeeq", "assoc": "MEMBER", "state": "COMMENTED", "commit": "3b77529f74b76ea52c1b7b4ac4edacb0947ff202", "text": "Concept ACK\n\nI think it would be better if we have internal memory management for the mining interface IPC, since we hold on to the block templates.\n\nI would suggest the following approach:\n\n- Add memory budget for the mining interface.\n- Introduce a tracking list of recently built block templates and total memory usage.\n- Add templates to the list and increment the memory usage after every `createnewblock` or `waitnext` return.\n- Whenever the memory budget is exhausted, we should release templates in FIFO order.\n\nI think since we create a new template after a time interval elapses even if fees increase and that interval is usually enough for the client to receive and distribute the template to miners, this mechanism should be safe as the miners have long switch to most recent template when the budget elapsed because of the time interval being used in between returns of waitnext.\n\nMining interface clients should also handle their own memory internally.\n\nCurrently, I don\u2019t see much use for the exposed `getMemoryLoad` method. In my opinion, we should not rely on the IPC client to manage our memory." }, { "t": "2025-11-21T16:34:13Z", "kind": "comment", "who": "Sjors", "assoc": "MEMBER", "text": "[quoted text omitted]\n\n[quoted text omitted]\nIt seems counter intuitive, but from a memory management perspective IPC clients are treated no different than our own code. And if we started FIFO deleting templates that are used by our own code, we'd crash.\n\nSo I think FIFO deletion should be a last resort (not implemented here).\n\nThere's another reason why we should give clients an opportunity to gracefully release templates in whatever order they prefer. Maybe there's 100 downstream ASIC's, one of which is very slow at loading templates, so it's only given a new template when the tip changes, not when there's a fee change. In that scenario you have a specific template that the client wants to \"defend\" at all cost.\n\nIn practice I'm hoping none of this matters and we can pick and recommend defaults that make it unlikely to get close to a memory limit, other than during some weird token launch." }, { "t": "2025-11-21T17:38:04Z", "kind": "comment", "who": "ismaelsadeeq", "assoc": "MEMBER", "text": "[quoted text omitted]\n\nIMHO I think we should separate that, and treat clients differently from our own code, because they are different codebases and separate applications with their own memory.\n\n[quoted text omitted]\nI see your point but I don\u2019t think that\u2019s a realistic scenario, and I think we shouldn\u2019t design software to be one-size-fits-all.\nIf you want to use only single block templates, then use `createnewblock` and create a new block template and mine that continuously until the chain tip changes or you mine a block.\n\n`waitNext` returning indicates that we assume your miners are switching from the block they are currently mining to the new one they receive.\nDepending on the budget (which I assume is large), many templates would need to be returned before we exhaust it.\n\nDelegating template eviction responsibility to the client can put us in a situation where they handle it poorly and cause us to OOM (but I guess your argument is that we rather take that chance than being in a situation where we make miners potentially lose on rewards).\nHowever I think if there is a clean separation of concerns between the Bitcoin Core node and its clients and clear interface definition and expectations that should not happen, and I believe the mining interface should not differ in that respect.\nOtherwise, if we do want a one-size-fits-all solution capable of handling the scenario you described, we should rethink the design entirely and revert to an approach where we do not retain block templates." }, { "t": "2025-11-24T10:49:07Z", "kind": "comment", "who": "Sjors", "assoc": "MEMBER", "text": "[quoted text omitted]\n\nNote that it's _already_ the clients responsibility, that's inherent to how multiprocess works.\n\nIn the scenario where they handle it poorly, we can use FIFO deletion. All `getMemoryLoad()` does is give clients an opportunity to handle it better. If they're fine with FIFO, then they never have to call this method.\n\n[quoted text omitted]\nWe currently don't track whether any given `CBlockTemplate` is owned by an IPC client or by our internal code. Once we introduce FIFO deletion all call sites will have to check if it's been deleted since, or we need to exempt them from the memory accounting.\n\n[quoted text omitted]\nAfaik that means revalidating the block from scratch, removing one advantage the `submitBlock()` approach has over the `submitblock` RPC (I haven't benchmarked this though)." }, { "t": "2025-11-24T16:56:34Z", "kind": "comment", "who": "Sjors", "assoc": "MEMBER", "text": "I tracked the non-mempool transaction memory footprint for half a day on mainnet, using fairly aggressive template update criteria (minimum fee delta 1 sat and no more than once per second). So far the footprint is minuscule, but of course this depends on the mempool weather:\n\nThe memory spike after each new block is because `sv2-tp` holds on to templates from previous blocks for 10 seconds. Those ~3 MB spikes may look impressive, but keep in mind that the default mempool is 300 MB." }, { "t": "2025-11-25T15:47:05Z", "kind": "force_push", "who": "Sjors", "commit": "24592b737fb80f07276a88fbcf8eaf0ee64c7e49" }, { "t": "2025-11-25T15:56:12Z", "kind": "comment", "who": "Sjors", "assoc": "MEMBER", "text": "I restructured the implementation and commits a bit.\n\nThe `TxTemplateMap` now lives on the `NodeContext` rather than `MinerImpl` (interface). This reflects the fact that we want to track the global memory footprint instead of per client. It's a lightweight member `template_tx_refs` which should be easy to fold into a block template manager later.\n\nIt's also [less code churn](https://github.com/bitcoin/bitcoin/compare/3b77529f74b76ea52c1b7b4ac4edacb0947ff202..24592b737fb80f07276a88fbcf8eaf0ee64c7e49#diff-0ef8ae12c6e9ef2accc78537f42612b3267e8a7c45dc7e9eb998f797e79f2e95) because I don't have to touch the `BlockTemplateImpl` constructor.\n\nIt also made it easier to move `GetTemplateMemoryUsage` from `interface.cpp` to `miner.cpp`, where it's more reusable.\n\nThis in turn let me split out a separate commit that introduces the actual `getMemoryLoad()` interface method. So even if we decide against including that method, the rest of the PR should be useful. However I do think it's worth keeping, it's already been a helpful debugging and monitoring tool.\n\nI added some comments to point out that we don't hold a `mempool.cs` lock during the calculation because we don't need an accurate result (mempool drift) and we don't want to bog down transaction relay with a potentially long lock (1-3ms in my testing so far)." }, { "t": "2025-11-25T17:21:57Z", "kind": "force_push", "who": "Sjors", "commit": "03dcfaeb5892fa877eaae0fe6d6942943b7d9b95" }, { "t": "2025-11-25T17:22:05Z", "kind": "comment", "who": "Sjors", "assoc": "MEMBER", "text": "`mining_getblocktemplate_longpoll.py` triggered a `stack-use-after-return`, due to `block_template` being `static` (to allow template reuse between RPC calls). I added a commit d752dccaa56b663001d1bb29ab8b9a50628602a9 to move this longpoll template to the node context. This seems more appropriate anyway since `BlockTemplate` has a `m_node` member, so it shouldn't be able to outlive the node.\n\nOne caveat is that `gbt_template` has to be cleared before `template_tx_refs`, so I swapped them and added a comment (cde248a6613b6e37f7f7e35c1aabeb75347ffe95 -> 9c667c362a1639b48113a3657882b751f475082c.\n\n---\n\nExpanded the PR description." }, { "t": "2025-11-25T18:13:13Z", "kind": "force_push", "who": "Sjors", "commit": "ac1e97a59279b2407e5f64bd684170587bae88dc" }, { "t": "2025-12-02T13:17:51Z", "kind": "review_comment", "who": "brunoerg", "assoc": "MEMBER", "path": "src/node/interfaces.cpp", "commit": "8b4967e3570a7adce2e73c3476bcdd5d676cf090", "in_reply_to": null, "text": "Just ran a mutation testing for this PR and this is the only unkilled mutant - feel free to ignore if it doesn't make sense to address:\n\n```diff\ndiff --git a/src/node/interfaces.cpp b/src/node/interfaces.cpp\nindex f734296b24..13ddbb672f 100644\n--- a/src/node/interfaces.cpp\n+++ b/src/node/interfaces.cpp\n@@ -880,7 +880,7 @@ public:\n for (const CTransactionRef& tx : m_block_template->block.vtx | std::views::drop(1)) {\n auto ref_count{m_node.template_tx_refs.find(tx)};\n if (!Assume(ref_count != m_node.template_tx_refs.end())) break;\n- if (--ref_count->second == 0) {\n+ if (1==1) {\n m_node.template_tx_refs.erase(ref_count);\n }\n }\n```" }, { "t": "2025-12-03T08:19:54Z", "kind": "comment", "who": "Sjors", "assoc": "MEMBER", "text": "Here's a slightly more realistic plot from last night on a well connected node running on an Intel i5-8400:\n\nIt's connected to DMND pool, declaring custom templates and getting them approved, but not actually mining. Due to their rate limiting I set `-sv2interval=20`, so if fees go up, it waits at least 20 seconds before generating a new template. It does not wait when the tip changes.\n\nThe machine also runs a lightning node and BTCPay so the moment block comes in the system is quite busy." }, { "t": "2025-12-03T09:36:11Z", "kind": "force_push", "who": "Sjors", "commit": "e8f8f7f677bcde0179526be3ed9a657c44998b93" }, { "t": "2025-12-03T09:36:17Z", "kind": "review_comment", "who": "Sjors", "assoc": "MEMBER", "path": "src/node/interfaces.cpp", "commit": "8b4967e3570a7adce2e73c3476bcdd5d676cf090", "in_reply_to": 2581145624, "text": "@brunoerg that's useful. So far `interface_ipc.py` never had a transaction appear in multiple templates, so the destructor would always remove the last reference.\n\nI adjusted the test so that it does. Now your mutation causes a crash during this test." }, { "t": "2025-12-03T21:36:31Z", "kind": "comment", "who": "ryanofsky", "assoc": "MEMBER", "text": "Concept ACK e8f8f7f677bcde0179526be3ed9a657c44998b93. All the changes here seem good and mostly straightforward. The `getMemoryLoad()` function seems useful by itself and the underlying tracking would seem to provide almost everything needed to limit memory used by block templates.\n\nI am a little concerned about the idea of proactively deleting block templates in FIFO order on behalf of clients, since it seems like this could increase complexity server-side, and client-side if clients have to deal with templates disappearing without being notified. Just not returning new templates after a certain amount of memory has been used would like a simpler approach.\n\nre: https://github.com/bitcoin/bitcoin/pull/33922#issue-3652141565\n\n[quoted text omitted]\nWould be good if this said templates are also released if the python references are destroyed or go out of scope. (This stood out because I tested this yesterday in https://github.com/bitcoin/bitcoin/issues/33940#issuecomment-3604081211.)" }, { "t": "2025-12-04T06:06:14Z", "kind": "review_comment", "who": "brunoerg", "assoc": "MEMBER", "path": "src/node/interfaces.cpp", "commit": "8b4967e3570a7adce2e73c3476bcdd5d676cf090", "in_reply_to": 2581145624, "text": "Nice, thank you!" }, { "t": "2025-12-04T10:46:46Z", "kind": "comment", "who": "Sjors", "assoc": "MEMBER", "text": "[quoted text omitted]\n\nIt is, but refusing to make new templates doesn't stop the footprint of existing templates from growing. The worst case extra memory footprint for _existing_ templates is the full size of the mempool.\n\nThis is rather unlikely though, it would only happen if between two blocks the entire mempool was gradually RBF'd in such a way that each transaction was at the top of the mempool briefly, and thus made it into a template.\n\n[quoted text omitted]\nAdded a sentence to the PR description." }, { "t": "2025-12-04T17:37:39Z", "kind": "review_comment", "who": "ryanofsky", "assoc": "MEMBER", "path": "src/rpc/mining.cpp", "commit": "c37d715c1d32020fe16ddc92f7ca95f585cc6b4b", "in_reply_to": null, "text": "In commit \"rpc: move static block_template to node context\" (a5eee29fd7d177f57c78da9773cb656a129de839)\n\nI think it would actually be nice to move all these static variables to a struct or class like @ismaelsadeeq's BlockTemplateCache from #33421. But this could be a followup, and doesn't need to complicate this PR." }, { "t": "2025-12-04T17:55:02Z", "kind": "review_comment", "who": "ryanofsky", "assoc": "MEMBER", "path": "src/node/context.h", "commit": "e8f8f7f677bcde0179526be3ed9a657c44998b93", "in_reply_to": null, "text": "In commit \"mining: track non-mempool memory usage\" (7c4d03d7b23417612fbca2f22e5bb1a198c9e5a2)\n\nThis map can updated from multiple threads, so it needs a mutex to be used safely. I think I'd suggest combining `template_tx_refs` and `gbt_template` variables and a mutex into single struct called something like `BlockTemplateState` and adding a `unique_ptr` to that struct as a member here. The struct could be replaced with a cache class in #33421." }, { "t": "2025-12-04T18:11:07Z", "kind": "review_comment", "who": "ryanofsky", "assoc": "MEMBER", "path": "test/functional/interface_ipc.py", "commit": "7f4249b0d0a126cfaddba48bc1bd4966c14fbd0d", "in_reply_to": null, "text": "In commit \"ipc: add getMemoryLoad()\" (e8f8f7f677bcde0179526be3ed9a657c44998b93)\n\nSeems ok to assert this log message is logged, but I'm wondering if there was a particular reason for doing this. Was the idea to pair the LOG_TIME_MILLIS_WITH_CATEGORY and assert_debug_log calls together?" }, { "t": "2025-12-04T18:23:00Z", "kind": "review", "who": "ryanofsky", "assoc": "MEMBER", "state": "COMMENTED", "commit": "e8f8f7f677bcde0179526be3ed9a657c44998b93", "text": "Code review e8f8f7f677bcde0179526be3ed9a657c44998b93. This looks good except for a thread safety issue I think you can address by adding a mutex.\n\nre: https://github.com/bitcoin/bitcoin/pull/33922#issuecomment-3611468373\n\n[quoted text omitted]\nSorry, I should have made a more specific suggestion. The problem is is that this sentence is not accurate: \"templates are not released until the client disconnects or calls the destroy() method.\" Templates will be released if the client drops references to them, even if it never disconnects or calls destroy. I would just change it to \"templates are not released until the client drops references to them, or calls the template destroy method, or disconnects\"" }, { "t": "2025-12-05T09:26:08Z", "kind": "review_comment", "who": "Sjors", "assoc": "MEMBER", "path": "test/functional/interface_ipc.py", "commit": "7f4249b0d0a126cfaddba48bc1bd4966c14fbd0d", "in_reply_to": 2590089590, "text": "This just checks that bench logging happens." }, { "t": "2025-12-05T11:09:09Z", "kind": "force_push", "who": "Sjors", "commit": "e75406dccc39b1f5d988994083e7fd959fa2a49a" }, { "t": "2025-12-05T11:09:14Z", "kind": "review_comment", "who": "Sjors", "assoc": "MEMBER", "path": "src/node/context.h", "commit": "e8f8f7f677bcde0179526be3ed9a657c44998b93", "in_reply_to": 2590041765, "text": "To limit the scope of this PR, I only added the mutex, but called it `template_state_mutex` in anticipation." }, { "t": "2025-12-05T11:13:53Z", "kind": "force_push", "who": "Sjors", "commit": "9a03d3837f2532619f8249e4a062453b333c0692" }, { "t": "2025-12-05T11:18:27Z", "kind": "force_push", "who": "Sjors", "commit": "7f4249b0d0a126cfaddba48bc1bd4966c14fbd0d" }, { "t": "2025-12-19T10:24:15Z", "kind": "force_push", "who": "Sjors", "commit": "71795a21e6ec854f2f450ce04cc6ff9e108a2c89" }, { "t": "2025-12-19T10:25:37Z", "kind": "comment", "who": "Sjors", "assoc": "MEMBER", "text": "Rebased after #34003. Dropped c548d6f0e8ecc0da6e29256c7085d48d10e10216 _test: destroy templates more carefully_. That commit also added coverage for `feeThreshold == MAX_MONEY`, so I moved that into a new commit - not really related to this PR though." }, { "t": "2026-01-14T09:54:19Z", "kind": "comment", "who": "Sjors", "assoc": "MEMBER", "text": "Rebased after #33819" }, { "t": "2026-01-14T09:54:27Z", "kind": "force_push", "who": "Sjors", "commit": "6fb639a089d83a2e4e4eeca26f3041d2da15d0c7" }, { "t": "2026-02-03T13:32:38Z", "kind": "force_push", "who": "Sjors", "commit": "c518fdf2dec42b6e52eff172816a097738389d16" }, { "t": "2026-02-19T11:04:55Z", "kind": "force_push", "who": "Sjors", "commit": "6cfd0f29e544c85c175a4ea6bf9db64fe57cdc05" }, { "t": "2026-02-19T11:06:26Z", "kind": "comment", "who": "Sjors", "assoc": "MEMBER", "text": "Rebased after #34452 and #33965." }, { "t": "2026-02-23T10:51:15Z", "kind": "force_push", "who": "Sjors", "commit": "2ba0f0ba36a23665c23e190b60282dade1108bbf" }, { "t": "2026-02-23T11:09:07Z", "kind": "comment", "who": "Sjors", "assoc": "MEMBER", "text": "Rebased after #34568.\n\nAncestor CI failure is probably related to #34646." }, { "t": "2026-02-24T12:17:03Z", "kind": "force_push", "who": "Sjors", "commit": "c37d715c1d32020fe16ddc92f7ca95f585cc6b4b" }, { "t": "2026-02-24T12:17:35Z", "kind": "comment", "who": "Sjors", "assoc": "MEMBER", "text": "Rebased after #34184." }, { "t": "2026-03-03T14:35:53Z", "kind": "review_comment", "who": "vasild", "assoc": "MEMBER", "path": "src/node/context.h", "commit": "c37d715c1d32020fe16ddc92f7ca95f585cc6b4b", "in_reply_to": null, "text": "Is \"gbt_template\" supposed to mean \"get block template template\"? Maybe \"gbt_result\" or \"get_block_template_result\" would be better." }, { "t": "2026-03-03T14:48:24Z", "kind": "review_comment", "who": "vasild", "assoc": "MEMBER", "path": "src/node/interfaces.cpp", "commit": "9f353d56735293677c1db92e639ef034c615eedd", "in_reply_to": null, "text": "In the commit message of ee259ca6b4dd4986be45e31aeee8d73bad8a115a `mining: track non-mempool memory usage`:\n\n\"transactions that are removed from the mempool will ~have~ not have their memory cleared.\"" }, { "t": "2026-03-03T15:26:49Z", "kind": "review_comment", "who": "vasild", "assoc": "MEMBER", "path": "src/node/types.h", "commit": "c37d715c1d32020fe16ddc92f7ca95f585cc6b4b", "in_reply_to": null, "text": "How many entries will be supposedly stored in this map? `std::map` has lookup `O(log(size))` whereas `std::unordered_map` has `O(1)`. Here we do not need the entries to be ordered.\n\nnit: start the comment with `/**` to make doxygen recognize it and attach it to the following code in the documentation." }, { "t": "2026-03-03T15:43:36Z", "kind": "review_comment", "who": "vasild", "assoc": "MEMBER", "path": "src/node/context.h", "commit": "c37d715c1d32020fe16ddc92f7ca95f585cc6b4b", "in_reply_to": null, "text": "I think the comment warrants an elaboration. `gbt_template` is not explicitly cleared anywhere, so that happens at the destructor of `NodeContext`. `struct` members are destroyed in reverse order of their declaration. Is this comment intended to prevent swapping the declaration order of `template_tx_refs` and `gbt_template`?\n\nMaybe:\n\n```cpp\n //! Cache latest getblocktemplate result for BIP 22 long polling. Must be cleared\n //! before template_tx_refs because the destructor of this decrements the count\n //! in `template_tx_refs` of each transaction in the template. If it does not find\n //! some of its transactions in `template_tx_refs` then it will abort.\n```" }, { "t": "2026-03-03T16:38:18Z", "kind": "review_comment", "who": "vasild", "assoc": "MEMBER", "path": "src/node/miner.h", "commit": "c37d715c1d32020fe16ddc92f7ca95f585cc6b4b", "in_reply_to": null, "text": "```\nsrc/node/miner.h:17 col 1 warning: Included header map is not used directly\n```" }, { "t": "2026-03-03T16:40:47Z", "kind": "review_comment", "who": "vasild", "assoc": "MEMBER", "path": "src/node/miner.h", "commit": "c37d715c1d32020fe16ddc92f7ca95f585cc6b4b", "in_reply_to": null, "text": "In the commit message of ce781758df81bca7bc3b60b8e02e36064634439e `ipc: add getMemoryLoad()`\n\n\"which can later be expand to\" ->\n\"which can later be expanded to\"" }, { "t": "2026-03-03T16:49:34Z", "kind": "review", "who": "vasild", "assoc": "MEMBER", "state": "APPROVED", "commit": "c37d715c1d32020fe16ddc92f7ca95f585cc6b4b", "text": "ACK c37d715c1d32020fe16ddc92f7ca95f585cc6b4b" }, { "t": "2026-03-05T19:43:04Z", "kind": "review_comment", "who": "Sjors", "assoc": "MEMBER", "path": "src/node/context.h", "commit": "c37d715c1d32020fe16ddc92f7ca95f585cc6b4b", "in_reply_to": 2878639103, "text": "I went for `gbt_result`" }, { "t": "2026-03-05T19:43:35Z", "kind": "review_comment", "who": "Sjors", "assoc": "MEMBER", "path": "src/node/interfaces.cpp", "commit": "9f353d56735293677c1db92e639ef034c615eedd", "in_reply_to": 2878715526, "text": "Fixed" }, { "t": "2026-03-05T19:44:47Z", "kind": "review_comment", "who": "Sjors", "assoc": "MEMBER", "path": "src/node/types.h", "commit": "c37d715c1d32020fe16ddc92f7ca95f585cc6b4b", "in_reply_to": 2878931110, "text": "Done both.\n\nAssuming one template per second for two hours (long block interval), less than 10k." }, { "t": "2026-03-05T19:56:10Z", "kind": "review_comment", "who": "Sjors", "assoc": "MEMBER", "path": "src/node/context.h", "commit": "c37d715c1d32020fe16ddc92f7ca95f585cc6b4b", "in_reply_to": 2879025033, "text": "I took your comment.\n\nIt\u2019s not only about declaration order. The intent is to destroy all BlockTemplate instances, including the one in gbt_result, while template_tx_refs is still alive so destructors can decrement reference counts. Once all templates are gone (and the map is empty), template_tx_refs can be destroyed." }, { "t": "2026-03-05T19:58:25Z", "kind": "force_push", "who": "Sjors", "commit": "04553cd6612f08d757fb20287fe111007bfb6db1" }, { "t": "2026-03-05T19:58:29Z", "kind": "comment", "who": "Sjors", "assoc": "MEMBER", "text": "Rebased after #34422 for easier testing with Rust, see https://github.com/2140-dev/bitcoin-capnp-types/pull/13.\n\nImplemented all of @vasild's nits." }, { "t": "2026-03-06T08:55:44Z", "kind": "review_comment", "who": "vasild", "assoc": "MEMBER", "path": "src/node/types.h", "commit": "c37d715c1d32020fe16ddc92f7ca95f585cc6b4b", "in_reply_to": 2878931110, "text": "Just mentioning - it is the same for `map` and `unordered_map` when used with `shared_ptr` (or `CTransactionRef`) - they compare pointers. So, two distinct objects that have the same values for all their members will be considered different. I am not sure if this is a problem in our code. Grepping the code for `(map|set).*CTransactionRef`, it looks like this will be the first case in non-test code where we use `CTransactionRef` as a key without providing custom comparator/hasher.\n\nTo illustrate with an explicit example:\n\n```cpp\n CMutableTransaction mutable_tx;\n // mutable_tx.vin = ...\n // mutable_tx.vout = ...\n\n CTransaction tx1{mutable_tx};\n CTransaction tx2 = tx1; // tx2 is a copy of tx1, same transaction _logically_\n\n CTransactionRef tx1_ref{MakeTransactionRef(tx1)};\n CTransactionRef tx2_ref{MakeTransactionRef(tx2)};\n\n std::map m;\n assert(m.emplace(tx1_ref, 5).second); // inserted\n assert(m.emplace(tx2_ref, 6).second); // inserted\n assert(m.size() == 2); // has 2 elements\n```" }, { "t": "2026-03-06T08:57:16Z", "kind": "review", "who": "vasild", "assoc": "MEMBER", "state": "APPROVED", "commit": "04553cd6612f08d757fb20287fe111007bfb6db1", "text": "ACK 04553cd6612f08d757fb20287fe111007bfb6db1\n\nWould be good to figure out if https://github.com/bitcoin/bitcoin/pull/33922#discussion_r2894597519 needs addressing." }, { "t": "2026-03-06T11:16:11Z", "kind": "review_comment", "who": "vasild", "assoc": "MEMBER", "path": "src/node/types.h", "commit": "c37d715c1d32020fe16ddc92f7ca95f585cc6b4b", "in_reply_to": 2878931110, "text": "If this needs to be resolved, then the below should do it:\n\n```diff\ndiff --git i/src/node/types.h w/src/node/types.h\nindex 164667772a..7bb187ac1c 100644\n--- i/src/node/types.h\n+++ w/src/node/types.h\n@@ -11,21 +11,23 @@\n //! files.\n\n #ifndef BITCOIN_NODE_TYPES_H\n #define BITCOIN_NODE_TYPES_H\n\n #include \n-#include \n-#include \n-#include \n #include \n #include \n #include