#26966 index: initial sync speedup, parallelize process

ReviewabilityReviewsAgreementSize
#26966 furszy index: initial sync speedup, parallelize processAuthor silent 272d0 (+1) -1Disputed569 + 462 tests
  • Speed up initial index synchronization for block filter and transaction indexes
  • Allow node runners to process block batches across multiple worker threads during initial sync
  • Needs rebase
  • Author is splitting out the thread pool into #33689 before updating this PR
  • Verified significant block filter index speedups on SSD (Sjors, pinheadmz, mzumsande)
  • Unaddressed objection: txindex is slower on SSD and parallel sync adds too much complexity (l0rinc)
  • Unaddressed objection: LevelDB write contention slows txindex sync on SSD by ~7% (andrewtoth)
  • Concept and approach approval for the parallel processing design (ryanofsky, sedited, w0xlt)
  • Unanswered question on how parallel indexing interacts with AssumeUTXO (maflcko)
  • 569 lines added or modified outside tests
  • 462 lines added or modified in tests
  • 153 lines removed in total
  • 14 files, 11 commits
Indexes#5 of 11
P2 · speedup
  • P2 because parallel block digestion dramatically cuts initial sync time for block filter indexes
  • Node runners building block filter indexes see multi-hour builds drop to under an hour on SSDs
  • Transaction index write contention on SSDs partially offsets gains but filter speedups are substantial
Ranking notes
Utilities (logging, arguments, libraries)#24 of 59
P3 · new feature
  • P3 because it introduces a generic thread pool utility that other node subsystems can reuse
  • Serves as a helpful concurrent building block rather than solving an urgent deficiency
  • Ranking pass: Moved up: a 7h-to-1h initial index build is the largest user-visible performance claim in this band.
Ranking notes
Reviewability: ReadyStalePaused
Agreement: StrongPositivePositive w/ caveatsNeutralMildDisputedBlockedCrickets
Reviews: current code-review ACKs, then (+stale ACKs) and -NACKs; greener = more ACKs.
Size: lines added or modified outside tests, then in tests; greener = smaller.

Goal

Splits block index synchronization into parallel block digestion and sequential post-processing phases, and introduces a general-purpose ThreadPool utility. Indexes that implement concurrent digestion (currently blockfilterindex and txindex) can process batches of blocks across multiple worker threads during initial sync when enabled via the new `-indexworkers=<n>` startup option.

Problem: Constructing block filter and transaction indexes during initial sync is slow because blocks, undo data, and headers are read from disk and processed sequentially one block at a time on a single thread.

Evidence of importance:

  • Author benchmark from the PR description: 'with the node in debug mode and on IBD, with -indexworkers=4, the block filter index generation took less than an hour. While, in master, the sync took more than 7 hours.'

Scope: The PR introduces a new general-purpose `ThreadPool` class in `src/util/threadpool.h` with dedicated unit and fuzz tests, in addition to the indexing changes. Part of the `BaseIndex` sync loop refactor in `src/index/base.cpp` was truncated from the patch snippet.

Reviewability

Stale · Author silent 272d

No author activity for 272 days; a review may wait a long time for a response.

Waiting on (from the thread read):

  • author: extract ThreadPool via #33689 and rebase this PR once included 2026-02-01
  • author: answer maflcko's question regarding AssumeUTXO compatibility 2023-10-25
  • author: splitting ThreadPool into #33689 and rebasing before continuing with index parallel sync 2026-02-01
  • author: answering maflcko's question about how this works with AssumeUTXO 2023-10-25

Agreement

Disputed

objection that needs a change, the objector pushed back after a reply (l0rinc)

Thread notes: Added unrecorded suggestions and questions from mzumsande, w0xlt, Sjors, maflcko, ryanofsky, and Eunovo under schema 2 rules. l0rinc's Approach NACK was contested following pushback on review complexity and deadlocks. The PR has been marked draft by the author while the ThreadPool is decoupled into #33689. / Updated claim classifications under current schema rules. Evaluated l0rinc's Approach NACK as contested following their pushback on deadlock risks and chunking. Added open question from mafl

Objections, suggestions, and questions:

ReviewerTypeHarm, request, or questionStatusMust beAuthor repliedQuote
mzumsande (member)suggestion, approachimplement parallelization in a general way so other indices like txindex can benefitfixed · fix pushed · author implemented generic workers pool and txindex parallelizationansweredyes2023-01-29: 'did you attempt to implement parallelization in a more general way so that other indices could benefit from it as well?'
Status from: 2023-02-28: furszy: 'Implemented generic workers pool.'
w0xlt (contributor)suggestion, approachseparate parallelization and index logic so it can be reused across other indexesfixed · fix pushed · author separated generic ThreadPool in later pushansweredno2023-01-30: 'Perhaps it could have separate parallelization and index logic. So it could be reused in other indexes.'
Status from: 2023-02-28: furszy: 'Implemented generic workers pool.'
maflcko (member)question, approachhow parallel indexing interacts with AssumeUTXO (AU)openansweredno2023-10-25: 'Also, how would this work with AU?'
Sjors (member)suggestion, maintenancespin down or destroy worker threads once initial index sync is completefixed · fix pushed · author agreed and destructed thread pool once initial sync completesansweredyes2024-03-21: 'Maybe the threads should spin down if there's not much work.'
Status from: 2024-03-21: furszy: 'The thread pool can be destructed once all indexes initial sync finish'
ryanofsky (member)objection, safetytaking a reference to a temporary CBlockUndo object at block height 0 causes undefined behaviorfixed · fix pushed · author agreed and simplified to Assert(block.undo_data)changedyes2025-04-09: 'I'm not sure this is safe. It looks like if block height is 0 this is taking a reference to a temporary object that will go out scope.'
Status from: 2025-06-05: furszy: 'yeah, great. Done as suggested.'
ryanofsky (member)suggestion, approachopportunistically run post-processing in worker threads to prevent idle threadsfixed · fix pushed · implemented opportunistic post-processing pipeline in worker threadsansweredyes2025-04-09: 'It seems like this idleness could be avoided by moving the CustomPostProcessBlocks calls into the worker threads. So that whenever each worker thread finishes processing blocks, it then opportunistically calls CustomPostProcessBlocks to post-process any blocks that are available'
Status from: 2025-06-23: furszy: 'Spent a few days implementing this suggestion... The process runs faster now.'
ismaelsadeeq (member)objection, safetyintermittent segfault during index sync caused by worker threads accessing an index Sync() local variable post-destructionfixed · fix pushed · author fixed subtle race condition where worker thread accessed destroyed local variablechangedyes2025-08-12: 'Fwiw I encountered a segfault previously while testing on master on previous PR HEAD'
Status from: 2025-08-12: furszy: 'There was a very subtle bug on which the worker threads might have accessed an index Sync() local variable post-destruction.'
sedited (member)objection, correctnessthread pool fuzzer times out after 1608 seconds during thread destruction and joiningfixed · fix pushed · author added fuzz harness and fixed shutdown synchronization bugchangedyes2025-09-16: 'It seems to have found something: ... ERROR: libFuzzer: timeout after 1608 seconds'
Status from: 2025-10-09: furszy: 'Updated with a fuzz test case from @TheCharlatan. Thanks!'
mzumsande (member)objection, interfaceparallel indexing on HDDs causes a 2x slowdown compared to sequential master due to disk seek penalties without warning in helpfixed · fix pushed · author updated -indexworkers help documentation warning against HDD usechangedyes2025-10-07: 'observed a slowdown on a HDD compared to master (by a factor 2)... Should it be mentioned in the -indexworkers help'
Status from: 2025-10-07: furszy: 'Updated, thanks for testing!'
Eunovo (member)question, approachwhether ThreadPool should support assigning tasks in batches to threadsanswered · author explained task granularity should be handled by caller before submissionansweredyes2025-10-08: 'Is it not beneficial for a generic ThreadPool to support assigning tasks in batches to threads?'
Status from: 2025-10-08: furszy: 'I don't think that's thread pool responsibility. Generic thread pools are designed to execute tasks efficiently, but they don't dictate their granularity'
mzumsande (member)suggestion, approachextract ThreadPool into a separate pull request firstanswered · author agreed to split ThreadPool to #33689 and set this PR to draftansweredyes2025-10-10: 'It could make sense to introduce the threadpool in an extra PR first, since it is very generic and can be used for multiple things'
Status from: 2025-10-21: furszy: 'will push the thread pool on an isolated PR in the coming week'
mzumsande (member)objection, correctnessspecifying -indexworkers=0 triggers an assertion failure in ThreadPool on startupfixed · fix pushed · author fixed initialization check when indexworkers is 0changedyes2025-10-10: '-indexworkers=0 is allowed according to the doc, but it will trigger the assert num_workers > 0 in the ThreadPool.'
Status from: 2025-10-11: furszy: 'nice catch. Fixed.'
ryanofsky (member)suggestion, approachavoid std::shared_ptr and std::function overhead in ThreadPool by using std::packaged_task<void()>fixed · fix pushed · author adopted packaged_task<void()> wrapper in ThreadPool queueansweredyes2025-10-10: 'Would be nice to avoid shared_ptr here. Apparently it is needed because std::function objects are required to be copyable and packaged tasks aren't copyable. Could fix this by avoiding std::function'
Status from: 2025-10-11: furszy: 'ha, that's awesome... I see the trick now... That's elegant.'
l0rinc (member)objection, approachtxindex parallelization causes a 3-13% slowdown on SSDs, and combining a generic shared thread pool with complex out-of-order index synchronization introduces excessive complexitycontested · l0rinc reiterated deadlock and review complexity concerns after author's reply; author later agreed to split ThreadPool to #33689 and put PR into draftchangedyes2025-10-12: 'The txindex parallelization appears to be broken (showing 3-13% slowdowns rather than speedups)... Approach NACK from me.'
Status from: 2025-10-15: l0rinc: 'The review effort would be the same for 2 or nproc threads, but the implications would be limited with the former'
andrewtoth (member)objection, approachtxindex sync on SSD is ~7% slower than master due to LevelDB write contention across worker threads offsetting parallel digest gainsanswered · author demonstrated 2.6x speedup on Signet and offered to investigate SSD slowdown or disable parallel sync for txindexchangedyes2025-10-13: 'master was ~7% faster than this PR with 5 workers on an SSD. Maybe we don't enable parallel sync for txindex?'
Status from: 2025-10-14: furszy: 'I'm still working on the current approach but synced the tx index on the same environment as before... Parallel Mode (5 workers): ~5 minutes... Sequential Mode: 13 minutes.'

Support:

  • w0xlt (contributor) (Concept ACK, 2023-01-30) [read]: supportive of general concept and suggested separating parallelization and index logic so it can be reused
  • Sjors (member) (2023-02-07) [tested]: tested building the block filter index on SSD, confirming speedup from 35m15s down to 4m26s with 32 workers
  • sedited (member) (Concept ACK, 2023-11-25) [read: src/index/base.cpp, src/test/threadpool_tests.cpp, src/util/threadpool.h]: promising approach to parallelization, provided thorough first pass review
  • ryanofsky (member) (Approach ACK, 2025-04-09) [read: src/index/base.cpp, src/util/threadpool.h]: seems like a nice design and good approach after code review
  • ismaelsadeeq (member) (Concept ACK, 2025-07-21) [both]: reviewed and performed mainnet testing
  • pinheadmz (member) (ACK, 2025-07-25) [both: src/index/base.cpp, src/index/blockfilterindex.cpp, src/util/threadpool.h]: tested on macos/arm64 and debian/x86, praised ThreadPool utility for broader reuse across the codebase
  • pinheadmz (member) (ACK, 2025-07-29) [both]: re-ACK after testing on mainnet node (>900k blocks) with -indexworkers=16
  • mzumsande (member) (2025-10-07) [tested]: tested signet parallel sync on SSD showing significant speedups
  • brunoerg (member) (2025-10-10) [tested: src/test/fuzz/threadpool.cpp]: tested the ThreadPool fuzz harness overnight, confirming reasonable performance and no leaks

Participants:

  • Sjors (member, support, 7 statements 2023-01-26 to 2025-07-17): tested block filter index sync performance on SSD and HDD, confirming major speedup on SSD
  • mzumsande (member, support, 7 statements 2023-01-29 to 2025-10-10): tested on signet SSD, reported HDD slowdown and ThreadPool zero-worker assertion bug, both fixed
  • w0xlt (contributor, support, 1 statements 2023-01-30 to 2023-01-30): Concept ACK; suggested separating parallelization and index logic
  • maflcko (member, question, 1 statements 2023-10-25 to 2023-10-25): asked about CI failure and AssumeUTXO compatibility
  • sedited (member, support, 20 statements 2023-11-24 to 2025-09-16): Concept ACK, code review, contributed fuzz harness uncovering timeout bug that was fixed
  • andrewtoth (member, objection, 26 statements 2024-10-23 to 2025-10-15): measured txindex slowdown on SSD and suggested disabling parallel sync for txindex or splitting out ThreadPool
  • yancyribbens (contributor, neutral, 1 statements 2025-02-26 to 2025-02-26): suggested comment nit in unit tests
  • ryanofsky (member, support, 14 statements 2025-04-09 to 2025-10-10): Approach ACK, suggested opportunistic post-processing pipeline and packaged_task queue simplification
  • ismaelsadeeq (member, support, 9 statements 2025-07-21 to 2025-08-12): Concept ACK, tested on mainnet, reported intermittent shutdown crash which was fixed
  • pinheadmz (member, support, 15 statements 2025-07-24 to 2025-07-29): ACK, benchmarked mainnet speedup with 16 workers, suggested ThreadPool naming and logging improvements
  • brunoerg (member, support, 2 statements 2025-09-17 to 2025-10-10): ran ThreadPool fuzz target overnight and generated coverage report
  • Eunovo (member, question, 4 statements 2025-10-08 to 2025-10-08): asked about task batching and contributed test for recursive task submission
  • l0rinc (member, objection, 8 statements 2025-10-12 to 2025-10-15): Approach NACK; measured txindex slowdowns and requested breaking into smaller PRs

Dependencies

Enables:

Files

539 lines under test/bench/ci.

  • src/index/base.cpp +265/-62
  • src/test/threadpool_tests.cpp +267/-0
  • src/util/threadpool.h +192/-0
  • src/test/blockfilter_index_tests.cpp +51/-77
  • src/test/fuzz/threadpool.cpp +100/-0
  • src/index/base.h +55/-2
  • src/test/txindex_tests.cpp +42/-0
  • src/index/blockfilterindex.cpp +17/-9
  • src/init.cpp +22/-1
  • src/index/blockfilterindex.h +8/-2
  • src/index/txindex.h +7/-0
  • src/node/context.h +3/-0
  • src/test/CMakeLists.txt +1/-0
  • src/test/fuzz/CMakeLists.txt +1/-0

Card

The short text the ranking pass compares across PRs.

Parallelizes initial block indexing across multiple worker threads for the compact block filter and transaction indexes, configured by a new `-indexworkers=<n>` flag. Today, initial index builds are strictly single-threaded and read blocks and undo data sequentially from disk, causing initial sync to take many hours. The author reports that building the block filter index took under 1 hour with 4 workers compared to over 7 hours on master. Work is related to and proceeding alongside PR #34489.

Data

dossier JSON · extract JSON · model ledger, generated 2026-09-25T01:27, confidence high, input hash 873f91544e006101
thread read through 2026-02-01 (213 statements), head c5dad86523
labels: UTXO Db and Indexes, Needs rebase · draft

Categories

Indexes

P2 · speedup

Parallelizing block digestion dramatically speeds up initial index synchronization for block filters. Benchmarks cited by the author showed block filter index generation dropping from over 7 hours to under 1 hour with 4 workers, and Sjors confirmed a drop from 35m down to 4m26s with 32 workers. Although txindex exhibited minor write contention slowdowns on SSDs, drastically accelerating filter builds provides substantial, durable performance value for index users.

Membership: Modifies src/index/base.cpp and index implementations to parallelize initial index sync.

Factors: security/stability 1, bug 0, performance 2, user value 2, leverage 1

Utilities (logging, arguments, libraries)

P3 · new feature (ranking pass: Moved up: a 7h-to-1h initial index build is the largest user-visible performance claim in this band.)

The PR adds a generic ThreadPool utility intended for concurrent task execution across node components. Reviewers recognized its potential for broader reuse, though concerns over synchronization complexity and deadlocks indicate it serves primarily as a worthwhile building block rather than solving an urgent utility deficiency.

Membership: Introduces a general-purpose ThreadPool utility in src/util/threadpool.h.

Factors: security/stability 1, bug 0, performance 2, user value 2, leverage 1