#26966 index: initial sync speedup, parallelize process

• Speed up initial construction of optional indexes by processing blocks in parallel across worker threads
• Dramatically cut the hours needed to build block filters and transaction indexes on multi-core systems

This PR introduces a multi-threaded execution model for the initial synchronization of optional indexes (block filter index and txindex). It introduces a two-phase block processing architecture in BaseIndex (CustomProcessBlock and CustomPostProcessBlocks) and adds a user-configurable -indexworkers option backed by a thread pool.

Problem: Building optional indexes sequentially takes many hours during initial sync because single-threaded block reading and filter computation cannot utilize multi-core processors.

P2 · speedup

• P2 because initial index builds take hours and tests show multi-fold speedups for block filters
• It provides durable value for index sync and lays groundwork for batched database writes

P2 because it addresses the multi-hour initial synchronization bottleneck of optional indexes by parallelizing block processing. Node operators building compact block filters experience a 2x to 6x speedup on SSDs, dropping sync times from over seven hours to under one hour.

Membership: Modifies BaseIndex, BlockFilterIndex, and TxIndex to support concurrent block digestion and out-of-order execution

Factors: security/stability 0, bug 0, performance 3, user value 2, leverage 1

P3 · new feature

• P3 because the thread pool is an internal utility supporting other subsystems
• It is reasonably deferrable on its own without direct user-facing benefit

P3 because adding a general-purpose ThreadPool utility provides clear value to background worker operations across the node. However, this component was split into #33689 and already merged into master, meaning it will be dropped from this branch upon rebase.

Membership: Introduces ThreadPool under src/util/threadpool.h for concurrent task execution

Factors: security/stability 0, bug 0, performance 1, user value 1, leverage 2

• Review #33689 first
• Drafted by the author and needs rebase

The author converted the pull request to draft and noted they are waiting to rebase after the extracted ThreadPool PR (#33689) is merged. The PR is also marked with merge conflicts.

Author status: paused; author converted PR to draft to rebase after #33689 merged

Open concerns:

• l0rinc submitted an Approach NACK stating that txindex benchmarks show a 3-13% slowdown rather than a speedup, and requested breaking the large PR into smaller steps
• andrewtoth corroborated the txindex slowdown on SSDs, observing that LevelDB write contention offsets parallel digest gains

Resolved concerns:

• mzumsande noted HDD performance degradation due to head seeking; furszy added an explicit warning to the -indexworkers help text
• ismaelsadeeq reported an intermittent crash on shutdown; furszy resolved a lifetime race condition in Sync() worker access
• Concerns regarding ThreadPool ownership and interrupt handling were addressed by extracting ThreadPool into #33689

• Strong support for parallelizing sync, but contested over diff complexity and txindex regressions
• Concept approval for the design and approach (ryanofsky)
• Verified sync time dropped from over 48 hours to 16 hours in testing (pinheadmz)
• Approach objection over diff complexity and performance regressions on txindex (l0rinc)
• Observed slight txindex slowdowns on SSD and questioned parallelizing it (andrewtoth)

Disputed: l0rinc holds an Approach NACK over txindex slowdowns and PR complexity, while author paused to rebase after #33689

The PR has strong concept support and ACKs from several contributors for its block filter performance gains, but l0rinc logged an Approach NACK pointing out that txindex is slower when parallelized and urging that the PR be broken down. The author agreed to split the ThreadPool work (which merged in #33689) and is investigating the txindex performance before rebasing.

• l0rinc: 'Approach NACK from me. Breaking this into smaller, more manageable chunks would significantly improve the chances of getting this important work merged'
• andrewtoth: 'master was ~7% faster than this PR with 5 workers on an SSD. Maybe we don't enable parallel sync for txindex?'
• furszy: 'Drafted while work continues on #33689. Once the thread pool is included, I'll rebase this one and continue moving forward.'

Objections:

Support:

• ryanofsky: Approach ACK, praised design and suggested opportunistic post-processing pipeline
• pinheadmz: ACK after code review and mainnet benchmarks demonstrating significant sync time reduction
• Sjors: Tested and confirmed block filter sync speedup from 35m down to 4m26s with 32 workers
• mzumsande: Concept ACK and verified performance improvements on SSDs
• sedited: Concept ACK and confirmed promise in parallelization approach
• ismaelsadeeq: Concept ACK and mainnet testing
• brunoerg: Fuzz testing execution and coverage reporting
• w0xlt: Concept ACK [not substantive]

Participants: Sjors (support), mzumsande (support), w0xlt (support), maflcko (question), sedited (support), andrewtoth (objection), yancyribbens (neutral), ryanofsky (support), ismaelsadeeq (support), pinheadmz (support), brunoerg (support), Eunovo (neutral), l0rinc (objection)

State derived from the lists: blocking objection open, author engaging (l0rinc)

Review verdicts (DrahtBot): 0 (+1) -1

• Stale ACK: pinheadmz
• Approach ACK: ryanofsky
• Concept ACK: w0xlt, sedited, ismaelsadeeq, mzumsande
• Approach NACK: l0rinc

Depends on: #33689

Enables:

• #34489

539 lines under test/bench/ci.

• src/index/base.cpp +265/-62
• src/test/threadpool_tests.cpp +267/-0
• src/util/threadpool.h +192/-0
• src/test/blockfilter_index_tests.cpp +51/-77
• src/test/fuzz/threadpool.cpp +100/-0
• src/index/base.h +55/-2
• src/test/txindex_tests.cpp +42/-0
• src/index/blockfilterindex.cpp +17/-9
• src/init.cpp +22/-1
• src/index/blockfilterindex.h +8/-2
• src/index/txindex.h +7/-0
• src/node/context.h +3/-0
• src/test/CMakeLists.txt +1/-0
• src/test/fuzz/CMakeLists.txt +1/-0

PR 26966 introduces multi-threaded block processing for initial index synchronization in BaseIndex, BlockFilterIndex, and TxIndex via a new -indexworkers flag. Benchmarks from multiple reviewers confirm dramatic speedups for the block filter index (from over 7 hours to under 1 hour on SSDs). However, reviewers identified that txindex experiences slight slowdowns due to LevelDB write contention, leading to an Approach NACK from l0rinc. The PR is currently paused in draft while the author rebases after the ThreadPool utility was merged upstream in #33689.