{ "number": 35354, "repo": "bitcoin/bitcoin", "url": "https://github.com/bitcoin/bitcoin/pull/35354", "title": "net: wait for validation queue flush for missing compact filter for an already known block", "author": "randomlogin", "author_association": "FIRST_TIME_CONTRIBUTOR", "created_at": "2026-05-21T22:07:53Z", "updated_at": "2026-09-07T10:36:09Z", "age_days": 118, "draft": false, "labels": [ "P2P" ], "milestone": null, "base": "master", "head_sha": "4849c0555cd8675d666f32c061011e7d2c8ad77b", "head_ref": "p2p-cbf-wait-for-indexed", "head_repo": "randomlogin/bitcoin", "head_history": [ { "t": "2026-05-22T10:50:09Z", "sha": "b68a721e645d0449fe01aebaa1c28c8d13eda3f2" }, { "t": "2026-08-05T01:29:56Z", "sha": "6c325d03b5d18127eb01dddb618685d9d0cae7f5" }, { "t": "2026-08-05T11:49:40Z", "sha": "b68a721e645d0449fe01aebaa1c28c8d13eda3f2" }, { "t": "2026-08-05T11:50:28Z", "sha": "21c65a024c9c0752b9fa53512f77604298f85e7c" }, { "t": "2026-08-14T15:47:17Z", "sha": "0d0750894d53cd14be3866d43db0292c3ec41b45" }, { "t": "2026-08-14T15:56:10Z", "sha": "3a8f89e8281de0e54cb9eac3344f23286d281f7f" }, { "t": "2026-08-14T22:54:27Z", "sha": "cef32b96cb7bf38135239c1eca4abdc4d7334245" }, { "t": "2026-08-26T14:26:51Z", "sha": "1526b81e16508ac6f938c6ea004c2abf67ed912a" }, { "t": "2026-08-27T07:11:08Z", "sha": "4849c0555cd8675d666f32c061011e7d2c8ad77b" } ], "additions": 563, "deletions": 13, "changed_files": 3, "commit_count": 3, "size_bucket": "L", "mergeable_state": "clean", "bot": { "drahtbot": { "present": true, "reviews": {}, "conflicts": [] } }, "acks_parsed": {}, "acks_tally": { "ack": 0, "stale_ack": 0, "concept_ack": 0, "approach_ack": 0, "nack": 0, "concept_nack": 0, "approach_nack": 0 }, "reviews": { "approved": 0, "changes_requested": 0, "distinct_reviewers": [ "ajtowns", "mzumsande", "sedited" ] }, "signals": { "needs_rebase": false, "ci_failed": false, "mergeable_state": "clean", "last_author_activity": "2026-08-27T07:11:08Z", "last_reviewer_activity": "2026-08-21T00:33:23Z", "last_reviewer": "ajtowns", "author_silent_days": 21, "waiting_on_author_days": 0, "days_since_update": 10 }, "refs": { "mentioned": [ 29655 ], "depends_on": [], "fixes": [], "linked_issues": [], "references": [ { "number": 29655, "type": "issue", "state": "open", "merged": false, "merged_at": null, "title": "CFilters reject and disconnect peers with a valid block hash" } ], "conflicts": [] }, "stack": { "shares_commits_with": [], "based_on": [], "base_for": [] }, "review_paths": [ "src/index/base.h", "src/index/blockfilterindex.cpp", "src/net_processing.cpp" ], "body": "This PR is another approach to fix a race condition bug when a yet-unconstructed compact filter is requested for an already known block.\n\nFixes: https://github.com/bitcoin/bitcoin/issues/29655, https://github.com/bitcoin/bitcoin/issues/27085\nRelated discussion: [1](https://bnoc.xyz/t/request-block-filter-peers-being-disconnected-due-to-invalid-block-requests/101/2), [2](https://github.com/lightninglabs/neutrino/issues/338)\nAnother approach which received NACK: https://github.com/bitcoin/bitcoin/pull/35262\n\n**The problem:**\nBitcoin Core receives a new block and processes it, then advertises it; however, by that moment the compact filter might not yet have been constructed. If a peer asks for a filter in that time window, Bitcoin Core simply does not respond. From the peer's perspective this is a misbehaviour (we advertised a block and don't provide filters for it), which results in disconnects or bans. While it does not critically affect core nodes, it breaks the network topology for BIP 157 clients.\n\nMore precisely, [BIP157](https://github.com/bitcoin/bips/blob/master/bip-0157.mediawiki#getcfilters) specifies that for `getcfilters` requestor:\n\n```\nStopHash MUST be known to belong to a block accepted by the receiving peer. This is the case if the peer had previously sent a headers or inv message with that block or any descendents. A node that receives getcfilters with an unknown StopHash SHOULD NOT respond.\n```\n\nCurrent behaviour clearly contradicts the above specification.\n\n**Proposed solution:**\n\nAs [was suggested](https://github.com/bitcoin/bitcoin/pull/35262#pullrequestreview-4278902223), we can:\n1. attempt to respond to a request (`getcfilters`, `getcfheaders`, `getcfcheckpt` which uses `LookupFilterHeader`); if it fails,\n2. check that we know the requested block hash and that the height difference between the last constructed filter block and the requested block is small \u2014 from which we deduce that pending validation events would fix the miss,\n3. wait for validation-queue flush and then retry.\n\nWaiting for the validation queue flush parks the single `msghand` thread, which blocks all peers' `ProcessMessages` for the duration of the drain. So the trigger has to be tight: an external actor can otherwise force us into that wait.\n\nThis PR introduces `BaseIndex::WaitForRacingWrite(target, max_ahead)`, which only drains when:\n1. the index is synced (`m_synced == true`),\n2. we have a non-null target block index,\n3. the target is at most `max_ahead` blocks ahead of the index's last processed block (and not below it).\n\n`BlockFilterIndex` calls this from all four lookup paths (`LookupFilter`, `LookupFilterHeader`, `LookupFilterRange`, `LookupFilterHashRange`) with `max_ahead = CF_MAX_BLOCKS_AHEAD_RACE_WAIT = 2`, retrying the lookup once after the drain.\n\n**Reorg handling:**\n\nThe same primitive cleanly handles the reorg cases the bug originally exposed:\n\n- _New chain extends past the old tip_ (`ahead > 0`): the queued `BlockConnected` events for the new branch will, in order, run `BaseIndex::BlockConnected`'s internal Rewind (which calls `CustomRemove` on the disconnected blocks \u2014 `BlockFilterIndex::CustomRemove` migrates each old filter from its height-keyed slot to a hash-keyed slot) and then `CustomAppend` for the new blocks. After the drain, both the old (hash-keyed) and new (height-keyed) filters are reachable.\n- _Same-height sibling reorg_ (`ahead == 0`): the index's last processed block and the requested block sit at the same height on different branches. The first lookup misses (height-keyed slot still holds the old block's data, hash-keyed slot for the new block doesn't exist yet). The `ahead == 0` branch of the helper drains the queue, which runs the disconnect/connect for the sibling reorg, after which the retry succeeds.\n- _Stale-branch lookup after the reorg has settled_ (`ahead < 0`): the helper does not wait \u2014 there is no in-flight callback that would help. The lookup falls back to the hash-keyed slot independently via the existing dual-key storage.\n\nThis PR does not affect block propagation, as the very root of this bug is the independence of block processing and filter construction in `CustomAppend`. Nor does it affect IBD: during IBD `m_synced` is false, the helper returns immediately, and we simply don't respond to compact filter messages (the writes are happening on the sync thread, not via the validation queue, so draining wouldn't help anyway). The IBD short-circuit also closes the DoS surface during the period when the node is most resource-constrained.\n\n**Drive-by:**\n\n`LookupFilterHeader` previously held `m_cs_headers_cache` across the DB lookup. Now that the lookup path can call `SyncWithValidationInterfaceQueue` (which must not be invoked under a lock that a queued callback might need), the cache lock is narrowed to wrap only the cache reads/writes, and the cache insert is switched to `try_emplace` since the check-then-insert is no longer atomic.\n\n**Tests:**\n\nA new `cfilter_race_tests` suite uses a `PreFilterBlocker` `CValidationInterface` registered before the filter index, so its own `BlockConnected` fires first on the scheduler thread and stalls it before the filter index gets to run `CustomAppend`. While the scheduler is parked, the test issues the lookup on a worker thread and asserts:\n\n1. the worker remains blocked (proves the wait path is engaged \u2014 a regression that bypasses it would let the worker return immediately with `false`);\n2. after releasing the scheduler, the worker returns the correct filter/header/range.\n\nCases covered:\n\n- `cfilter_available_during_append_window` \u2014 `LookupFilter` for the new tip racing with `BlockConnected`.\n- `cfilter_range_available_during_append_window` \u2014 `LookupFilterRange` covering indexed history plus the racing tip.\n- `cfilter_header_available_during_append_window` \u2014 `LookupFilterHeader` on the racing tip.\n- `cfilter_available_during_same_height_reorg` \u2014 same-height sibling reorg path (`ahead == 0`).\n\nI've used Claude Code for this PR.", "commits": [ { "sha": "a345a746f94261efb14dc9ac6ba940b4e9f2675c", "date": "2026-08-26T00:30:55Z", "message": "net: refactor: split serving compact filter requests from message parsing\n\nMove the serving logic of the getcfilters/getcfheaders/getcfcheckpt\nhandlers into ServeGetCFilters()/ServeGetCFHeaders()/ServeGetCFCheckPt(),\nso they can be reused when the request gets parked in the next commit.\nNo behavior change." }, { "sha": "ac49cd5a0ae7a45f7b53ad7dc04a8ec26a196fe4", "date": "2026-08-26T00:30:55Z", "message": "net: defer response to racing compact filters requests\n\nPreviously compact filters requests might falsely be ignored because\nthe requested block has been connected to the chain, but the filters\nhave not yet been constructed.\n\nThis commit adds deferring of such racing requests. We store the pending\nrequest in a per-peer `m_pending_request`, and defer processing further\nmessages from that peer until the pending request can be dealt with.\n\nChecking if we have a race condition is done via inspecting `GetSummary`\nof the index. Pending requests are resolved when the index catches up\nwith the tip, or when the requested stop hash is covered by the index.\n\nAlso adds a unit test for the race-condition." }, { "sha": "4849c0555cd8675d666f32c061011e7d2c8ad77b", "date": "2026-08-27T07:10:47Z", "message": "net: add IBD guard for parked cfilter request" } ], "timeline": [ { "t": "2026-05-22T10:50:09Z", "kind": "force_push", "who": "randomlogin", "commit": "b68a721e645d0449fe01aebaa1c28c8d13eda3f2" }, { "t": "2026-05-22T10:50:56Z", "kind": "comment", "who": "randomlogin", "assoc": "NONE", "text": "Squashed two commits in one (previously there was a distinct commit for tests addition)." }, { "t": "2026-06-21T21:47:20Z", "kind": "comment", "who": "sedited", "assoc": "MEMBER", "text": "Reading through this, I don't think this should be handled on the index level. While I'm still not sold that this is not something that is just a bit flawed in the spec, I would prefer an approach where net_processing dealt with this directly in the case of an index query returning nothing by checking if the requested block lies in the two block tolerance range and if so syncing the validation interface queue." }, { "t": "2026-07-24T09:56:37Z", "kind": "comment", "who": "sedited", "assoc": "MEMBER", "text": "@randomlogin how do you want to proceed here?" }, { "t": "2026-07-28T15:52:51Z", "kind": "comment", "who": "randomlogin", "assoc": "NONE", "text": "@sedited sorry for a long response.\n\nThere are two ways: the first one is simply to move race-window check logic to `net_processing` and then again wait for `SyncWithValidationInterfaceQueue`.\n\nOn the other hand, instead of blocking the caller (`msghand`) on `SyncWithValidationInterfaceQueue` completion, on a request and hitting the race-window (known block, no filters yet) we could remember the peer awaiting response, move on to other work happening and once the filters are constructed send a response.\n\nSomething like `PeerManagerImpl::ProcessPendingCFilterRequests`.\n\nThis will not block other p2p messages while we wait for filters being constructed.\n\nHowever I'm not sure that it's worth, as with the window gap being small enough we'd save only fractions of second. Additionally we could use `WakeMessageHandler` to avoid waiting next loop iteration (up to 100ms).\n\nDoes that sound right to you?" }, { "t": "2026-07-29T09:03:29Z", "kind": "comment", "who": "ajtowns", "assoc": "MEMBER", "text": "[quoted text omitted]\n\nThat seems like a plausible approach to me, fwiw. Just set a flag in `ProcessMessage` when you receive the request that this peer wants a a cfilter response, then in `SendMessages` attempt to handle the cfilter requests, deferring it to a later call if the block is pending.\n\nThat will mean cfilter responses for that peer may be out of order vs other messages on that connection, but I think that's okay?" }, { "t": "2026-07-29T10:13:50Z", "kind": "review", "who": "mzumsande", "assoc": "MEMBER", "state": "COMMENTED", "commit": "b68a721e645d0449fe01aebaa1c28c8d13eda3f2", "text": "The problem formulation in the OP is not precise: Currently, when we receive a block, we first build the filter and then advertise the block, so there usually is no race.The race can only happen if the client asks for the newest headers themselves unprompted. This could happen because they just newly connected, or because they do active `getheaders` polling for some other reason. But this explains why this is very rare.\n\nIn my opinion, both a `SyncWithValidationInterfaceQueue` call in `net_processing` (it's not like we have anything more urgent to do than processing the block anyway?) and to schedule it for later sound fine." }, { "t": "2026-07-29T13:31:10Z", "kind": "comment", "who": "randomlogin", "assoc": "NONE", "text": "[quoted text omitted]\n\nOP is precise for BIP152 block transmission, when we go through BIP152, we transmit compact block header to peers before we run `BlockConnected` which internally calls filters construction. (To be accurate, I wrongly called it advertising, but rather it is an actual sending of the header).\n\nAlso I took a moment to review how other requests are handled when we announced compact block header:\n- if a peer asks `getheaders` in the moment we have not yet connected the block, they will receive stale response\n- `getdata` for a block - we synchronously connect the block and then respond (which blocks all other peers for the time block connects)\n\nThen maybe it is better to have some kind of a unified approach for both `getdata` and `getcfilters`, for example both block and wait for block connection/filters constructions to complete?\n\nDeferring and responding later does not increase blocking time, and personally is more appealing and I'd go this way." }, { "t": "2026-07-29T13:40:12Z", "kind": "comment", "who": "mzumsande", "assoc": "MEMBER", "text": "[quoted text omitted]\n\nBut how does this make sense / how is this a relevant use case? To be chosen as a high-bandwidth block relay peer, you need to be a full node. In this case you don't need to download the blockfilters for new blocks from peers - this is a feature meant for light clients." }, { "t": "2026-07-29T14:39:41Z", "kind": "comment", "who": "randomlogin", "assoc": "NONE", "text": "@mzumsande you are right, I confused BIP152 with BIP130 (light clients use BIP130). And as you pointed, this bug cannot happen with BIP130 header transmission." }, { "t": "2026-07-30T06:06:59Z", "kind": "comment", "who": "ajtowns", "assoc": "MEMBER", "text": "[quoted text omitted]\n\nMy understanding is that the way we currently work is:\n\n * send the compact block info to peers that have requested high bandwidth cb relay ASAP\n * pause the message handling thread to validate the block\n * queue the cfilter updates for another thread\n * send `headers` and `inv` messages to non-cb and low-bandwidth cb peers\n * resume the message handling thread\n * message handling thread \"immediately\" processes a cfilter request, and we attempt to respond before the cfilter updates have finished\n\nApart from the hb compact block path, we don't announce blocks until we've validated them, but we do announce them before we've calculated their compact filter.\n\nI don't think we should block the message handling thread further, because we do have more important things to do: responding to `getblocktxn` requests that came in while we were validating the block (ideally we'd be responding to those requests while validating the block), and `getdata` requests that came in after our `headers` message.\n\nThose \"immediate\" requests could come from peers that selected us as hb for cb, or could just be a very quick response to a headers announcement compared to the calculations required to update the compact filters. I think those calcs are slow enough that this is likely to occur occasionally even for peers that aren't getting the high-bandwidth compact block path, so is worth handling, but it would probably good to confirm the exact flow and timings? (How long does calculating the compact filters actually take? How much does that vary across blocks? What's the real turn-around time for real compact filter clients between seeing a header and requesting the compact filter?)" }, { "t": "2026-07-30T12:19:41Z", "kind": "comment", "who": "mzumsande", "assoc": "MEMBER", "text": "Yes, that makes sense - I hadn't thought of `getblocktxn` requests.\n\n[quoted text omitted]\nI'd say it's\n- queue the headers and inv message to non-cb and low-bandwidth cb peers for the other thread (`UpdatedBlockTip()`)\n- resume the message handling thread which sends out headers and inv messages to non-cb and low-bandwidth cb peers once the queue entry has been processed" }, { "t": "2026-08-05T01:29:56Z", "kind": "force_push", "who": "randomlogin", "commit": "6c325d03b5d18127eb01dddb618685d9d0cae7f5" }, { "t": "2026-08-05T01:30:18Z", "kind": "comment", "who": "randomlogin", "assoc": "NONE", "text": "#### Changes\n\nAdded deferring of a response, we store the racing request in a `PendingCFilterRequest` struct in a `Peer` and then check it in `SendMessages`.\n\nTo avoid code repetition, added a helper `TryRespondToCFilterRequest` which is used in both original `ProcessGetCFilters` and `SendMessages` handlings of a request.\n\nRewrote the test.\n\nIf we receive two requests while the first one has not yet been responded, we do not overwrite it, thus responding only to the first one.\n\n#### Benchmarks\n\nAdded benchmarks commit (b1d4868485770777e46950460852d61f9ad4b6c2) to this PR (I'll squash it later).\n\nHere is the data for the initial filter construction since genesis as well as for processing during normal operation, i.e. I have an already running node in which I turned on filters and measured the times. Initial blocks' measurements are shown over latest 100k blocks, since ~861k).\n\n| Metric | mean | median | p90 | p99 | max |\n| ------------------------------------ | -------- | ------- | ------- | --------- | ---------- |\n| Cfilters construction (from 861k initial) | 4.50ms | 4.50ms | 6.74ms | 8.04ms | 112.86ms |\n| Cfilters construction (212 live blocks) | 5.92ms | 6.10ms | 8.35ms | 11.33ms | 13.96ms |\n| Connect block (212 live blocks) | 172.85ms | 51.14ms | 63.89ms | 4408.39ms | 10423.97ms |\n\nMaximum time (112 ms for filters construction) is perhaps some I/O problem or something else not connected to normal behaviour. All times include I/O.\n\nWe can say filters processing takes around 10% of block processing.\n\nComputer that runs benchmarks has the following specs:\n\n- CPU: Intel Xeon E5-2686 v4 (Broadwell) \u2014 2 vCPUs, 2.30GHz, 1 thread/core\n- RAM: 3.8GB\n\n#### Going further\n\n1. Currently only added handling `getcfilters`, but the same bug stands for other bip157 requests: `getcfheaders` and `getcfcheckpt`. It is even less likely for them to encounter a race, do we want them handled in the same way?\n\n2. Should we call `m_connman.WakeMessageHandler()` in block connection, or the current approach is fine?" }, { "t": "2026-08-05T11:49:40Z", "kind": "force_push", "who": "randomlogin", "commit": "b68a721e645d0449fe01aebaa1c28c8d13eda3f2" }, { "t": "2026-08-05T11:50:28Z", "kind": "force_push", "who": "randomlogin", "commit": "21c65a024c9c0752b9fa53512f77604298f85e7c" }, { "t": "2026-08-11T02:09:10Z", "kind": "review_comment", "who": "ajtowns", "assoc": "MEMBER", "path": "src/index/blockfilterindex.cpp", "commit": "b1d4868485770777e46950460852d61f9ad4b6c2", "in_reply_to": null, "text": "The benchmarking commit should be removed, not squashed, presumably? It doesn't seem like it'll be useful beyond developing this PR?\n\nI think this PR should be marked as draft if it has commits that need updating/replacing prior to merge." }, { "t": "2026-08-11T02:11:09Z", "kind": "review_comment", "who": "ajtowns", "assoc": "MEMBER", "path": "src/net_processing.cpp", "commit": "21c65a024c9c0752b9fa53512f77604298f85e7c", "in_reply_to": null, "text": "Rather than an optional, I'd suggest considering `std::list` and using `push_back` to add requests to the end, and `front` to get the oldest request in order to attempt to service it. Allows handling multiple requests, though still needs a limit to avoid growing unbounded. Also wastes a little less memory than an optional if there's nothing queued, which should be true for most peers for most of the time." }, { "t": "2026-08-11T16:18:33Z", "kind": "review_comment", "who": "mzumsande", "assoc": "MEMBER", "path": "src/index/base.h", "commit": "21c65a024c9c0752b9fa53512f77604298f85e7c", "in_reply_to": null, "text": "Races in `net_processing` shouldn't be the business of indexes, and `GetSummary()` should have enough info (or we could add more if not), so I'd prefer if the index code was not touched at all for this and the `net_processing` code would decide whether there is a possible race." }, { "t": "2026-08-14T15:47:17Z", "kind": "force_push", "who": "randomlogin", "commit": "0d0750894d53cd14be3866d43db0292c3ec41b45" }, { "t": "2026-08-14T15:47:47Z", "kind": "review_comment", "who": "randomlogin", "assoc": "NONE", "path": "src/index/blockfilterindex.cpp", "commit": "b1d4868485770777e46950460852d61f9ad4b6c2", "in_reply_to": 3754782852, "text": "Originally I thought it might be useful on its own.\nRemoved benchmark commit." }, { "t": "2026-08-14T15:48:13Z", "kind": "review_comment", "who": "randomlogin", "assoc": "NONE", "path": "src/net_processing.cpp", "commit": "21c65a024c9c0752b9fa53512f77604298f85e7c", "in_reply_to": 3754789371, "text": "Changed an optional in favor of a list with up to `MAX_PENDING_CFILTER_REQUESTS=2` entries." }, { "t": "2026-08-14T15:48:35Z", "kind": "review_comment", "who": "randomlogin", "assoc": "NONE", "path": "src/index/base.h", "commit": "21c65a024c9c0752b9fa53512f77604298f85e7c", "in_reply_to": 3759737064, "text": "Removed `IsRacing`, now using `GetSummary`. No changes in `GetSummary` were required." }, { "t": "2026-08-14T15:56:10Z", "kind": "force_push", "who": "randomlogin", "commit": "3a8f89e8281de0e54cb9eac3344f23286d281f7f" }, { "t": "2026-08-14T22:54:27Z", "kind": "force_push", "who": "randomlogin", "commit": "cef32b96cb7bf38135239c1eca4abdc4d7334245" }, { "t": "2026-08-16T05:31:31Z", "kind": "comment", "who": "ajtowns", "assoc": "MEMBER", "text": "Sorry for repeatedly requesting changes.\n\n[quoted text omitted]\nBased on the logs in #29655 this seems backwards -- there were more disconnects for getcfheaders than getcfilters? So I think this ought to be handled for all the cases? Not at all clear to me why it's not, so maybe I've missed something?\n\nI wonder if it wouldn't be better to adopt the \"pause this peer\" approach, rather than queueing messages but continuing. Here's a somewhat vibecoded branch that demos that (and delays all the messages when needed): https://github.com/ajtowns/bitcoin/commits/202608-cfilter-park/\n\nThe idea is that you add a single \"hey, I'm working on an answer to this request\" queued item for the peer, and refrain from processing any further messages from that peer (without interfering with outgoing messages like block announcements or other peers in general). Not sure if checking the index for updates every msgproc loop is okay or there should be an additional pause there." }, { "t": "2026-08-18T17:27:12Z", "kind": "comment", "who": "randomlogin", "assoc": "NONE", "text": "[quoted text omitted]\nI didn't implement it as wanted to receive feedback for the approach itself first.\n\nI also checked my logs when I encountered this bug with https://github.com/2140-dev/kyoto, and indeed I also encountered it with `getcfheaders`.\n\nYes, all three cbf messages should be handled the same way.\n\n[quoted text omitted]\nIt seems this approach is conceptually simpler, but as we defer all other messages from a peer including `ping` or `getdata` my concerns are:\n1) We potentially slow down block propagation, as we don't respond to a \"legitimate\" `getdata` during the race window. (Though it's quite questionable why bip157 peer would ask for a full block without having filters/cfheaders for it first, but it's a presupposition what a bip157 peer has to be.)\n2) We logically mix handling of compact filter related messages with other ones. While per se it is not harmful, it feels not very appealing.\n\nIf we try to specify which messages we still should process after a request is parked during the racing window (e.g. `getdata`), it defeats the whole idea of simplicity. It seems it's the same as to specify which requests we should defer (as in my commit).\n\nAlso I'm not sure that preserving the order of responses is important, it seems any (reasonable) client matches the response by its content.\n\n[quoted text omitted]\nI'll try to benchmark that." }, { "t": "2026-08-21T00:33:23Z", "kind": "comment", "who": "ajtowns", "assoc": "MEMBER", "text": "[quoted text omitted]\n\nI think that's very unlikely -- compact block filters are for light clients; so any node requesting compact filters isn't going to have every block and isn't going to be a reliable part of the fast path for block relay. I'd suspect most such nodes wouldn't even store full blocks for the ones they request, instead just keeping the txs they're interested in and a merkle path back to the block header?\n\n[quoted text omitted]\nI would say that by pausing that peer until we can reply to its request we avoid mixing handling of compact filter messages with other ones.\n\n[quoted text omitted]\nRight, the point is to pause **all** message processing for that peer, so that message processing for that peer remains in order.\n\n[quoted text omitted]\nPreserving the order is valuable for our own tests (if we send message X followed by PING, we won't receive the PONG corresponding to our PING until X has been dealt with), and possibly for state updates.\n\nI've been thinking about it in terms of separating out block validation into a separate thread from message processing (suggested by @sipa offline, see also [A](https://github.com/bitcoin/bitcoin/issues/35023#issuecomment-4332885607) and [B](https://github.com/bitcoin/bitcoin/issues/35453#issuecomment-4632593610)), where it would be a nice win if we could respond to compactblock requests while validating the block, while still pausing nodes that send us messages that need to wait for the validation to finish (so that observable behaviour doesn't change at a logical level, and in particular so that our own tests don't get heavily impacted). I think getting the \"pause\" part of that change in here would be nice, particularly if it's simple." }, { "t": "2026-08-26T14:26:51Z", "kind": "force_push", "who": "randomlogin", "commit": "1526b81e16508ac6f938c6ea004c2abf67ed912a" }, { "t": "2026-08-26T14:53:24Z", "kind": "comment", "who": "randomlogin", "assoc": "NONE", "text": "I took some time to gain more understanding on how `net_processing` works and your comments make a perfect sense.\n\nSome suggestions on top of your commit:\n\n1. Added a guard for initial block download.\n On a fresh node with enabled compact filters `PeerManagerImpl::CFilterIndexMayBeRacing` would return true, meaning we pause the peer for the whole duration needed to construct filters from the genesis block, which takes hours.\n Also we could store a timestamp when parking occured and release the pause if it takes too long, though I am not sure if it is really needed.\n\n2. Now we have several cases where we process peer's messages not in a \"usual\" way, when we might return early and not poll any new peer's messages.\n\n These cases are:\n - continue responding to a huge `getdata` message\n - parked a compact filter related message\n - we processed an orphan tx and have a result\n - in future here will be a case when we validate a block broadcast by this peer, we early exit if we haven't finished yet\n\n Should we combine all of the above into something like `PreprocessMessages` or `ProcessAwaitingMessages` or `ProcessPausedPeer`?\n\n3. Made minor renamings in your code.\n\n4. Here are some [claude-coded benchmarks](https://github.com/randomlogin/bitcoin/commit/6a3e332b504d9fb2bdbffa32ab68433b8bc5afa7), I ran them locally on my computer.\n\n- `AttemptParkedRequest()` is retried on every `ProcessMessages()` pass for a peer with a pending\n request, not gated on new traffic \u2014 so n parked peers add n extra `cs_main` acquisitions to every\n message-handler sweep for as long as they stay parked.\n\n Per-call cost (`bench_bitcoin`, isolated \u2014 both paths take `cs_main`, differ in what's done under it):\n\n | path | ns/op |\n |---|---:|\n | not racing (`Tip()` compare only) | 66 |\n | racing (2 lookups + ancestor walk) | 99 |\n\n Negligible alone \u2014 microseconds/sec even at high parked-peer counts. The real cost is repetition,\n not the call.\n\n- Retry-rate impact (regtest, P2P sim). Natural race windows are sub-ms, too short to observe\n anything here, so the filter index's per-block indexing was artificially delayed ~200ms to hold the window open long enough to measure \u2014 this is a stress test of what happens if the window ever widens (slow index, edge case), not a description of current behavior.\n\n With that delay in place: `ThreadMessageHandler`'s idle sleep is 100ms, so n parked peers add up to 10\u00b7n acquisitions/sec. n peers parked on `getcfilters` vs. n peers sending an ordinary `getdata` (control \u2014 resolves immediately, no repeat cost), measuring an unrelated peer's ping RTT:\n\n | n | +cs_main acq/s | racing max | control max |\n |---:|---:|---:|---:|\n | 10 | 100 | 101ms | 51ms |\n | 50 | 500 | 152ms | 52ms |\n | 100 | 1,000 | 302ms | 51ms |\n\n Control stays flat; racing tail grows with the acquisition rate \u2014 isolates the effect to parking, not to connection count. Since this depends on an artificially widened window rather than natural behavior, it supports bounding max parked duration with a timeout rather than leaving it open-ended, but isn't evidence of a problem today." }, { "t": "2026-08-27T07:11:08Z", "kind": "force_push", "who": "randomlogin", "commit": "4849c0555cd8675d666f32c061011e7d2c8ad77b" } ], "labels_log": [ { "t": "2026-05-21T22:07:56Z", "action": "labeled", "label": "P2P", "who": "DrahtBot" }, { "t": "2026-05-22T10:47:20Z", "action": "labeled", "label": "CI failed", "who": "DrahtBot" }, { "t": "2026-05-22T15:24:56Z", "action": "unlabeled", "label": "CI failed", "who": "DrahtBot" }, { "t": "2026-08-05T21:26:38Z", "action": "labeled", "label": "CI failed", "who": "DrahtBot" }, { "t": "2026-08-06T00:26:23Z", "action": "unlabeled", "label": "CI failed", "who": "DrahtBot" }, { "t": "2026-08-14T17:52:21Z", "action": "labeled", "label": "Needs rebase", "who": "DrahtBot" }, { "t": "2026-08-15T00:39:21Z", "action": "unlabeled", "label": "Needs rebase", "who": "DrahtBot" }, { "t": "2026-08-17T09:11:37Z", "action": "labeled", "label": "CI failed", "who": "DrahtBot" }, { "t": "2026-08-17T12:20:28Z", "action": "unlabeled", "label": "CI failed", "who": "DrahtBot" }, { "t": "2026-08-26T17:47:10Z", "action": "labeled", "label": "CI failed", "who": "DrahtBot" }, { "t": "2026-08-27T09:03:10Z", "action": "unlabeled", "label": "CI failed", "who": "DrahtBot" } ], "state_log": [ { "t": "2026-05-21T22:08:15Z", "kind": "ready_for_review", "who": "randomlogin" }, { "t": "2026-08-27T07:05:52Z", "kind": "closed", "who": "maflcko" }, { "t": "2026-08-27T07:06:00Z", "kind": "reopened", "who": "maflcko" } ], "text_chars": 23719, "text_tokens_estimate": 5929, "changed_paths": [ "src/net_processing.cpp", "src/test/CMakeLists.txt", "src/test/cfilter_race_tests.cpp" ], "files": [ { "path": "src/net_processing.cpp", "add": 176, "del": 13 }, { "path": "src/test/CMakeLists.txt", "add": 1, "del": 0 }, { "path": "src/test/cfilter_race_tests.cpp", "add": 386, "del": 0 } ], "test_lines": 387, "git": { "head": "4849c0555cd8675d666f32c061011e7d2c8ad77b", "head_matches_backup": true, "base": "031175197f1b7f90397b838a381f0892a74ca62a", "commits": [ { "sha": "a345a746f9", "subject": "net: refactor: split serving compact filter requests from message parsing", "files": 1, "add": 35, "del": 13 }, { "sha": "ac49cd5a0a", "subject": "net: defer response to racing compact filters requests", "files": 3, "add": 530, "del": 6 }, { "sha": "4849c0555c", "subject": "net: add IBD guard for parked cfilter request", "files": 1, "add": 17, "del": 13 } ], "patch_truncated": false }, "input_hash": "70aa1624b591797f", "extracted_at": "2026-09-17T16:15:31+00:00" }