{ "number": 34565, "repo": "bitcoin/bitcoin", "url": "https://github.com/bitcoin/bitcoin/pull/34565", "title": "refactor: extract BlockDownloadManager from PeerManagerImpl", "author": "w0xlt", "author_association": "CONTRIBUTOR", "created_at": "2026-02-11T18:54:52Z", "updated_at": "2026-09-14T22:48:33Z", "age_days": 217, "draft": false, "labels": [ "Refactoring", "Needs rebase" ], "milestone": null, "base": "master", "head_sha": "d867ea88fe8152db814772c3b3634ac73313ba08", "head_ref": "refactor/extract-block-download-manager", "head_repo": "w0xlt/bitcoin", "head_history": [ { "t": "2026-02-17T19:28:29Z", "sha": "a24d36e75c0f602fbef6106928f5631be41c150f" }, { "t": "2026-06-11T19:56:39Z", "sha": "78c890442e9a3e553d6c0bb0c1251bef1ea50086" } ], "additions": 1947, "deletions": 586, "changed_files": 10, "commit_count": 4, "size_bucket": "XL", "mergeable_state": "dirty", "bot": { "drahtbot": { "present": true, "reviews": { "concept_ack": [ { "login": "sedited", "url": "https://github.com/bitcoin/bitcoin/pull/34565#issuecomment-3887406021" }, { "login": "0xbrito", "url": "https://github.com/bitcoin/bitcoin/pull/34565#issuecomment-3888994209" } ] }, "conflicts": [ { "number": 35852, "title": "scripted-diff: Use inline const(expr) over static constexpr in headers", "author": "maflcko" }, { "number": 35839, "title": "p2p: select another peer after empty headers", "author": "l0rinc" }, { "number": 35591, "title": "[DO NOT MERGE] Erlay: bandwidth-efficient transaction relay protocol (Full implementation)", "author": "sr-gi" }, { "number": 35561, "title": "net: move some CNodeState fields to Peer", "author": "Crypt-iQ" }, { "number": 35558, "title": "p2p: Prefill compact blocks", "author": "davidgumberg" }, { "number": 35522, "title": "refactor: Extract per-message helpers from SendMessages() (move-only)", "author": "pablomartin4btc" }, { "number": 35229, "title": "refactor: Use CBlockIndex parameters as reference", "author": "optout21" }, { "number": 34743, "title": "p2p: don't disconnect manual peers for block stalling", "author": "willcl-ark" }, { "number": 34075, "title": "fees: Introduce Mempool Based Fee Estimation to reduce overestimation", "author": "ismaelsadeeq" }, { "number": 33854, "title": "fix assumevalid is ignored during reindex", "author": "Eunovo" }, { "number": 33637, "title": "refactor: optimize block index comparisons (1.4-6.8x faster)", "author": "l0rinc" }, { "number": 27052, "title": "test: rpc: add last block announcement time to getpeerinfo result", "author": "LarryRuane" } ] } }, "acks_parsed": { "sedited": { "kind": "concept_ack", "hash": null, "t": "2026-02-11T21:48:34Z", "stale": false }, "0xbrito": { "kind": "concept_ack", "hash": null, "t": "2026-02-12T06:49:40Z", "stale": false } }, "acks_tally": { "ack": 0, "stale_ack": 0, "concept_ack": 2, "approach_ack": 0, "nack": 0, "concept_nack": 0, "approach_nack": 0 }, "reviews": { "approved": 0, "changes_requested": 0, "distinct_reviewers": [ "0xbrito", "Crypt-iQ", "maflcko", "sedited" ] }, "signals": { "needs_rebase": true, "ci_failed": false, "mergeable_state": "dirty", "last_author_activity": "2026-09-14T22:48:33Z", "last_reviewer_activity": "2026-09-14T14:58:52Z", "last_reviewer": "Crypt-iQ", "author_silent_days": 2, "waiting_on_author_days": 0, "days_since_update": 2 }, "refs": { "mentioned": [ 30110, 35561 ], "depends_on": [], "fixes": [], "linked_issues": [], "references": [ { "number": 30110, "type": "pull", "state": "closed", "merged": true, "merged_at": "2024-10-29", "title": "refactor: TxDownloadManager + fuzzing" }, { "number": 35561, "type": "pull", "state": "open", "merged": false, "merged_at": null, "title": "net: move some CNodeState fields to Peer" } ], "conflicts": [ 35852, 35839, 35591, 35561, 35558, 35522, 35229, 34743, 34075, 33854, 33637, 27052 ] }, "stack": { "shares_commits_with": [], "based_on": [], "base_for": [] }, "review_paths": [], "body": "### Motivation\n\n`net_processing.cpp` is the largest file in the codebase (~6200 lines) and `PeerManagerImpl` mixes several largely independent subsystems into a single class: transaction relay, address gossip, headers sync, compactblocks, and block download. This makes the file difficult to review, test in isolation, and reason about lock ordering.\n\n #30110 successfully extracted transaction download logic into `TxDownloadManager`. This PR applies the same approach to block download, continuing the incremental decomposition of `PeerManagerImpl`.\n\n### What this PR does\n\nExtract all block download state and scheduling logic into a new `BlockDownloadManager` module:\n\n**Global state moved** (from `PeerManagerImpl`): `mapBlocksInFlight`, `mapBlockSource`, `nSyncStarted`, `m_block_stalling_timeout`, `m_last_tip_update`, `m_num_preferred_download_peers`, `m_peers_downloading_from`.\n\n**Per-peer state moved** (from `CNodeState`): `pindexBestKnownBlock`, `hashLastUnknownBlock`, `pindexLastCommonBlock`, `pindexBestHeaderSent`, `fSyncStarted`, `vBlocksInFlight`, `m_downloading_since`, `m_stalling_since`, `fPreferredDownload`.\n\n**Methods moved** (from `PeerManagerImpl`): `FindNextBlocksToDownload`, `TryDownloadingHistoricalBlocks`, `ProcessBlockAvailability`, `UpdateBlockAvailability`, `BlockRequested`, `RemoveBlockRequest`, `IsBlockRequested`, `IsBlockRequestedFromOutbound`, `TipMayBeStale`.\n\n### The result:\n* `net_processing.cpp` shrinks from 6193 to 5751 lines (\u2212442 net)\n* `CNodeState` loses 10 fields, retaining only compact block relay and chain sync timeout state\n* Block download logic becomes unit-testable and fuzz-testable in isolation (8 unit test cases and 1 fuzz target added)\n\n### Design\n\nFollows the `TxDownloadManager` pimpl pattern:\n* `blockdownloadman.h` \u2014 public interface\n* `blockdownloadman_impl.h` \u2014 implementation class with per-peer state\n* `blockdownloadman_impl.cpp` \u2014 implementation\n\n Unlike `TxDownloadManager`, no new mutex is introduced. Block download is inherently tied to chain state through `CBlockIndex*` pointers, so `cs_main` remains the synchronizing lock. The extraction is for code organization and testability, not lock granularity.\n\nA fuzz target is included to exercise the extracted manager directly, covering peer lifecycle, availability updates, in-flight request tracking, scheduling, stalling state, and counter consistency.\n\n ### Commits\n\n1. **Add BlockDownloadManager class with pimpl skeleton**: all new module files, no call sites changed\n2. **Add unit tests**: exercises the module in isolation\n3. **Migrate call sites**: ~150 call sites in `net_processing.cpp` rewired through `m_blockdownloadman`, pure mechanical replacement\n4. **Add fuzz target**: exercises `BlockDownloadManager` state transitions and invariants through the new module boundary", "commits": [ { "sha": "6958f980942d922e844b934bf859451ec142fa63", "date": "2026-06-11T19:28:26Z", "message": "refactor: add BlockDownloadManager class with pimpl pattern\n\nCreate a new BlockDownloadManager module following the established\nTxDownloadManager pimpl pattern:\n\n- blockdownloadman.h: public interface with QueuedBlock, options/info\n structs, and full BlockDownloadManager API\n- blockdownloadman_impl.h: BlockDownloadManagerImpl with\n PeerBlockDownloadState and per-peer map\n- blockdownloadman_impl.cpp: forwarding wrappers and implementations\n for block request tracking, availability processing, download\n scheduling, stalling detection, and tip staleness checks\n\nThe module manages all block download state: in-flight tracking\n(mapBlocksInFlight), block source attribution (mapBlockSource),\nper-peer download state, sync coordination, and scheduling algorithms\n(FindNextBlocksToDownload, TryDownloadingHistoricalBlocks).\n\nKey design decisions:\n- cs_main provides synchronization (no separate mutex), since block\n download is inherently tied to chain state via CBlockIndex pointers\n- Two-phase peer registration: ConnectedPeer handles both initial\n registration (conservative defaults) and re-registration (real\n connection properties from VERSION handshake)\n- TipMayBeStale preserves lazy initialization of m_last_tip_update\n- CompareExchangeBlockStallingTimeout preserves CAS atomicity for\n the stalling timeout accessed from multiple threads\n\nNo call sites are migrated yet; that happens in a subsequent commit." }, { "sha": "bd59d0300894ded605091b7e2c1d8af784d2537c", "date": "2026-06-11T19:53:41Z", "message": "test: add unit tests for BlockDownloadManager\n\nExercise BlockDownloadManager in isolation:\n\n- peer_lifecycle: registration, sync state (including idempotency\n and explicit unsetting), blocks in flight, and disconnect cleanup\n for all counters (preferred, sync, downloading)\n- block_source_tracking: set/get/erase block sources\n- stalling_and_tip_staleness: last tip update get/set, and\n TipMayBeStale lazy initialization and staleness detection\n- block_requested_basic: request/remove lifecycle, outbound\n detection, duplicate rejection, in-flight counts\n- remove_block_request_from_specific_peer: peer-targeted removal\n leaves other peers' requests intact\n- find_block_in_flight: verify BlockInFlightInfo fields across\n no-flight, single-peer, and multi-peer scenarios with and\n without partialBlock\n- tip_not_stale_with_blocks_in_flight: blocks in flight suppress\n staleness even with an old tip update\n- connected_peer_reregistration: two-phase ConnectedPeer upgrade\n from preferred=false to preferred=true without double-counting\n- compare_exchange_stalling_timeout: CAS success/failure semantics\n- find_next_blocks_to_download: download scheduling against\n headers-only blocks - height ordering, count cap, in-flight\n skipping, last-common-block update, and the accept/refuse paths\n of TryDownloadingHistoricalBlocks" }, { "sha": "78c890442e9a3e553d6c0bb0c1251bef1ea50086", "date": "2026-06-11T19:53:55Z", "message": "refactor: migrate block download state and methods into BlockDownloadManager\n\nMove all block download state and logic from PeerManagerImpl and\nCNodeState into BlockDownloadManager, completing the extraction.\n\nMigrated from PeerManagerImpl:\n- mapBlocksInFlight, mapBlockSource, nSyncStarted\n- m_block_stalling_timeout, m_last_tip_update\n- m_num_preferred_download_peers, m_peers_downloading_from\n- IsBlockRequested, IsBlockRequestedFromOutbound\n- RemoveBlockRequest, BlockRequested, TipMayBeStale\n- ProcessBlockAvailability, UpdateBlockAvailability\n- FindNextBlocksToDownload, TryDownloadingHistoricalBlocks\n\nMigrated from CNodeState:\n- pindexBestKnownBlock, hashLastUnknownBlock\n- pindexLastCommonBlock, pindexBestHeaderSent\n- fSyncStarted, vBlocksInFlight, m_downloading_since\n- m_stalling_since, fPreferredDownload\n\n~150 call sites updated to go through m_blockdownloadman.\n\nMAX_CMPCTBLOCKS_INFLIGHT_PER_BLOCK is removed from net_processing.h;\nits definition moved to blockdownloadman.h with the module.\n\nIn EvictExtraOutboundPeers, the manager is only queried after the\nCNodeState null check: GetBlocksInFlight() asserts that the peer is\nregistered, so querying it first would turn the previously tolerated\nunknown-peer case into an abort. Keeping the query behind the null\ncheck preserves the existing short-circuit behavior." }, { "sha": "d867ea88fe8152db814772c3b3634ac73313ba08", "date": "2026-06-14T00:12:56Z", "message": "fuzz: add BlockDownloadManager target\n\nExercise BlockDownloadManager through arbitrary sequences of its public\nAPI, against a fixture chainstate consisting of 20 mined blocks, a\nheaders-only extension of the tip, and a headers-only fork branching\nfrom below the tip. This covers the block download scheduler\n(FindNextBlocksToDownload, TryDownloadingHistoricalBlocks) across fork\ntopologies, the in-flight request bookkeeping whose multimap entries\nhold iterators into per-peer lists, availability tracking with both\nknown and unknown hashes, and the stalling/staleness logic.\n\nThe harness keeps a small per-peer mirror to respect the documented\npreconditions (registration before per-peer calls, the\nMAX_CMPCTBLOCKS_INFLIGHT_PER_BLOCK call-site guard) and checks\ninvariants throughout:\n\n- global counters (peers downloading from, preferred peers, sync\n started, total blocks in flight) always agree with per-peer state\n- scheduled blocks are capped by count, height-ascending, never\n already stored and never already in flight\n- a stale tip implies nothing is in flight\n- FindBlockInFlight results are internally consistent\n- stalling timeout compare-exchange follows CAS semantics\n- after disconnecting all peers, all data structures are empty" } ], "timeline": [ { "t": "2026-02-11T21:48:34Z", "kind": "comment", "who": "sedited", "assoc": "MEMBER", "text": "Concept ACK" }, { "t": "2026-02-12T06:49:40Z", "kind": "comment", "who": "0xbrito", "assoc": "NONE", "text": "Concept ACK" }, { "t": "2026-02-14T16:50:53Z", "kind": "comment", "who": "Crypt-iQ", "assoc": "MEMBER", "text": "Only glanced at the changes -- does this change make it easier to fuzz individual components?" }, { "t": "2026-02-17T19:28:29Z", "kind": "force_push", "who": "w0xlt", "commit": "a24d36e75c0f602fbef6106928f5631be41c150f" }, { "t": "2026-02-17T20:48:35Z", "kind": "comment", "who": "w0xlt", "assoc": "CONTRIBUTOR", "text": "[quoted text omitted]\n\nYes, this is one of the main benefits. Currently block download logic is embedded in `PeerManagerImpl` and `CNodeState`, which require standing up the full net_processing stack (`connman`, `addrman`, `mempool`, etc.) to test. There is no way to exercise block scheduling, request tracking, or stalling detection without that heavy setup.\n\nAfter this PR, `BlockDownloadManager` can be constructed with just a `ChainstateManager` reference:\n\n```c++\nBlockDownloadManager bdm(BlockDownloadOptions{*m_node.chainman});\nbdm.ConnectedPeer(peer1, info);\nbdm.BlockRequested(peer1, *genesis);\n```\n\nThe fuzz target itself isn't included here (keeping scope focused on the extraction), but the architecture now makes it straightforward to add as a follow-up." }, { "t": "2026-06-11T19:56:39Z", "kind": "force_push", "who": "w0xlt", "commit": "78c890442e9a3e553d6c0bb0c1251bef1ea50086" }, { "t": "2026-06-14T00:48:20Z", "kind": "comment", "who": "w0xlt", "assoc": "CONTRIBUTOR", "text": "I initially left the fuzz target out to keep this PR focused on the extraction and unit coverage, but added it now since it seems to provide useful coverage without expanding the production-code scope.\n\nThe target exercises `BlockDownloadManager` directly through the new module boundary: peer registration/disconnection, block availability, in-flight request tracking/removal, scheduling, stalling/stale-tip state, block source tracking, and global counter consistency. This is the kind of state-machine behavior that was hard to fuzz while it lived inside `PeerManagerImpl`.\n\nIt intentionally stays at the `BlockDownloadManager` API level and respects the same preconditions as the `net_processing` call sites. I think this makes it a useful follow-up commit for validating the extraction while keeping the main refactor unchanged." }, { "t": "2026-08-27T13:24:19Z", "kind": "comment", "who": "Crypt-iQ", "assoc": "MEMBER", "text": "Is separating components out of `PeerManagerImpl` really desirable? Before #30110, I liked that the transaction downloading code was all in one place, but now you have to look in multiple places (to each their own, I understand why it was moved). The PR is mostly move-only stuff, but is large and therefore a bit intimidating for me to even attempt reviewing. This approach does work for fuzzing some subset of the block download logic, but I also think we could focus on mocking out the stuff in net_processing first so we don't need as many components to fuzz in the first place. If we mock out validation code, ideally we wouldn't even need the fuzz test in this PR since we'd be able to hit all of the cases that this PR's fuzz test hits (though maybe we could still have both just for extra assurances). I also wonder if the interface for the block download manager could be slimmed down here, is everything really needed?" }, { "t": "2026-09-01T14:55:41Z", "kind": "comment", "who": "maflcko", "assoc": "MEMBER", "text": "[quoted text omitted]\n\nHaven't looked deeply, but it looks like the coupling mostly still remains (compare the `cs_main` lock requirements). I wonder if moving out in this case makes it easier to review code around lock requirements. Compare also https://github.com/bitcoin/bitcoin/pull/36077#pullrequestreview-5029955365\n\nPure fuzzing coverage (aka the code is called with some random input and doesn't crash) doesn't hurt, but I also wonder if there were any past bugs that would be caught by the new fuzz test?" }, { "t": "2026-09-14T14:58:52Z", "kind": "comment", "who": "Crypt-iQ", "assoc": "MEMBER", "text": "This PR conflicts with https://github.com/bitcoin/bitcoin/pull/35561 in spirit and I don't really know which way to go on this. #35561 tries to make a step to directly trim `CNodeState` (and I have a couple more patches that move other fields) instead of moving to a class and trimming later. I saw a few older PRs linked in https://github.com/bitcoin/bitcoin/issues/19398 that took a similar approach to mine. I think additional feedback from others on which approach to take would be helpful since I'm biased. Separately, I think more fuzz coverage of the block download code is good, but I haven't run the fuzz test yet and will do that at some point.\n\n[quoted text omitted]\nI guess because things are encapsulated in a class it's easier to reason about what sections of code need which locks? For me personally, I find the indirection itself more confusing even if the locking requirements are more clearly stated by using the class. Just my personal preference though and I don't know how others feel." }, { "t": "2026-09-14T22:48:33Z", "kind": "comment", "who": "w0xlt", "assoc": "CONTRIBUTOR", "text": "Thanks for the review. This PR focuses on refactoring and testing. To keep the scope manageable, the extraction deliberately preserves the existing locking requirements.\n\nI don\u2019t think this necessarily conflicts with #35561 (which I also ACKed). This PR extracts block-download state and logic into a separate component. #35561 moves independent peer flags into `Peer` and removes their dependency on `cs_main`. Both changes can be combined with some adjustments to where the state lives.\n\nFor example, in https://github.com/w0xlt/bitcoin/tree/refactor/block-download-peer-state, I preserved all of #35561\u2019s changes and adapted this extraction accordingly:\n\n- `Peer` owns the compact-block negotiation flags and the preferred-download flag, using atomics.\n- `PeerManagerImpl` keeps the atomic preferred-peer counter and decides whether to download from a peer.\n- `BlockDownloadManager` determines which blocks to request and tracks block availability and outstanding requests, under the existing `cs_main` requirement.\n\nThis is still an early draft branch, but I think it demonstrates how both changes can coexist. I see it as an incremental step toward a clearer separation of responsibilities, with room to refine the boundaries further." } ], "labels_log": [ { "t": "2026-02-11T18:54:59Z", "action": "labeled", "label": "Refactoring", "who": "DrahtBot" }, { "t": "2026-03-11T13:43:24Z", "action": "labeled", "label": "Needs rebase", "who": "DrahtBot" }, { "t": "2026-06-11T21:56:04Z", "action": "unlabeled", "label": "Needs rebase", "who": "DrahtBot" }, { "t": "2026-08-14T17:53:43Z", "action": "labeled", "label": "Needs rebase", "who": "DrahtBot" } ], "state_log": [], "text_chars": 13621, "text_tokens_estimate": 3405, "changed_paths": [ "src/CMakeLists.txt", "src/net_processing.cpp", "src/net_processing.h", "src/node/blockdownloadman.h", "src/node/blockdownloadman_impl.cpp", "src/node/blockdownloadman_impl.h", "src/test/CMakeLists.txt", "src/test/blockdownload_tests.cpp", "src/test/fuzz/CMakeLists.txt", "src/test/fuzz/blockdownloadman.cpp" ], "files": [ { "path": "src/CMakeLists.txt", "add": 1, "del": 0 }, { "path": "src/net_processing.cpp", "add": 146, "del": 584 }, { "path": "src/net_processing.h", "add": 0, "del": 2 }, { "path": "src/node/blockdownloadman.h", "add": 226, "del": 0 }, { "path": "src/node/blockdownloadman_impl.cpp", "add": 644, "del": 0 }, { "path": "src/node/blockdownloadman_impl.h", "add": 132, "del": 0 }, { "path": "src/test/CMakeLists.txt", "add": 1, "del": 0 }, { "path": "src/test/blockdownload_tests.cpp", "add": 466, "del": 0 }, { "path": "src/test/fuzz/CMakeLists.txt", "add": 1, "del": 0 }, { "path": "src/test/fuzz/blockdownloadman.cpp", "add": 330, "del": 0 } ], "test_lines": 798, "git": { "head": "d867ea88fe8152db814772c3b3634ac73313ba08", "head_matches_backup": true, "base": "c117bbc467f80b4f28bb8c4d97a4181cd714bfc9", "commits": [ { "sha": "6958f98094", "subject": "refactor: add BlockDownloadManager class with pimpl pattern", "files": 4, "add": 1003, "del": 0 }, { "sha": "bd59d03008", "subject": "test: add unit tests for BlockDownloadManager", "files": 2, "add": 467, "del": 0 }, { "sha": "78c890442e", "subject": "refactor: migrate block download state and methods into BlockDownloadManager", "files": 2, "add": 146, "del": 586 }, { "sha": "d867ea88fe", "subject": "fuzz: add BlockDownloadManager target", "files": 2, "add": 331, "del": 0 } ], "patch_truncated": true }, "input_hash": "1d6846c427844fed", "extracted_at": "2026-09-17T16:15:31+00:00" }