#34489 index: batch db writes during initial sync
https://github.com/bitcoin/bitcoin/pull/34489 · · +300/-74 in 11 files, 10 commits · labels: UTXO Db and Indexes, Needs rebase · draft
Goal
- Speed up initial sync of optional indexes by batching database writes across block ranges
- Reduce heavy disk I/O and cs_main lock contention that degrades overall node responsiveness
Batches LevelDB writes across configurable block ranges during index initial sync rather than committing block by block. Modifies NextSyncBlock to return ranges of blocks, decoupling block traversal from cs_main lock acquisition. Adds unit tests covering interrupted sync and reorg handling with batched writes.
Problem: During initial sync, indexes flush writes to disk after every block and hold cs_main frequently, causing heavy I/O overhead on mechanical disks and lock contention that impairs node responsiveness.
Category: Indexes (#3 of 7)
P2 · speedup
- P2 because it speeds up initial index sync by 11% to 27% on spinning disks
- Significantly reduces database file churn and lock contention for node operators running indexes
- Unblocks foundational architecture needed for parallel index processing
Initial sync of optional indexes takes many hours. Benchmarks demonstrate an 11% to 28% sync speedup on HDD for txindex and txospenderindex, alongside reducing coinstatsindex LevelDB file counts from 144 to 10. Decoupling block iteration from lock acquisition also reduces cs_main contention and is an explicit prerequisite for parallelizing index sync in #26966.
Membership: Touches src/index/base.cpp, src/index/base.h, src/index/blockfilterindex.cpp, and src/index/txindex.cpp to alter how optional indexes sync.
Factors: security/stability 0, bug 0, performance 2, user value 2, leverage 2
Reviewability: Stale: Needs rebase
- Needs rebase due to merge conflicts and recent architectural changes to indexes
The branch has merge conflicts against master and requires adaptation to recent index refactors (#34897, #35531, #35847).
Author status: Active; furszy acknowledged the race condition on 2026-08-23 and is coordinating the rebase with arejula27.
Open concerns:
- l0rinc requested changes noting that batching lets in-memory state advance ahead of persisted data, causing Commit() during an interrupt to write advanced state (e.g. m_muhash in coinstatsindex) with a stale locator and corrupt the index on restart
- l0rinc noted blockfilterindex writes flat files too eagerly relative to LevelDB batching
- Merge conflicts and semantic collisions with recently merged index changes (#34897, #35531, #35847)
Resolved concerns:
- arejula27 caught an issue where early exit in ProcessBlocks bypassed interrupt handling; furszy added an explicit check and unit test coverage
- sipa and furszy clarified that iterator reordering in an early commit does not alter execution behavior
Agreement: Blocked
- Concept supported for batching writes and preparing parallel index sync (arejula27, polespinasa, l0rinc)
- Blocking objection: interrupts can save advanced in-memory state with stale locators (l0rinc)
- Author acknowledged the race condition and plans to resolve it alongside the rebase (furszy)
Blocked: blocking objection open with no author reply (l0rinc)
Broad support exists for the concept and benchmarked gains, but l0rinc has an open blocking review regarding state corruption on shutdown which the author has acknowledged but not yet patched.
- l0rinc requested changes on 2026-03-09 regarding m_muhash and locator desync on interrupted sync
- furszy noted on 2026-08-23 that 'there is just one missing check regarding the race condition that we should consider properly'
- optout21 gave crACK on 2026-02-09
- arejula27 gave concept ACK on 2026-02-12 and 2026-02-14
Objections:
| Reviewer | Kind | Harm | Status | Blocking | Author replied | Quote |
|---|---|---|---|---|---|---|
| l0rinc | correctness | In-memory index state advances ahead of disk writes, causing shutdown Commit() to persist advanced state with a stale locator and corrupt the index on restart | open | yes | no | 2026-03-09: 'The most serious issue seems to me that Commit() during interrupt writes advanced m_muhash alongside a stale locator, which causes CoinStatsIndex::CustomInit to reject the index as corrupted on restart.' |
| arejula27 | correctness | Early return in ProcessBlocks bypassed the interrupt handling path, skipping expected recovery | resolved | no | yes | 2026-02-14: 'Returning false here causes the interrupt to be handled incorrectly... As a result, the expected recovery path is skipped.' Settled: 2026-02-14: arejula27 confirmed 'Nice catch. Pushed an update that fixes it and adds test coverage... I think the test implemented for the scenario is enough.' |
Support:
- optout21: crACK after explanations on batching logic and lock contention reduction
- arejula27: Concept ACK citing massive upgrade potential when paired with parallelization in #26966
- polespinasa: Concept ACK during code review [not substantive]
Participants: fjahr (question), l0rinc (objection), hebasto (neutral), maflcko (neutral), Jhackman2019 (support), bvbfan (neutral), optout21 (support), arejula27 (support), sedited (question), sipa (neutral), polespinasa (support)
State derived from the lists: blocking objection open with no author reply (l0rinc) (model's own read: Disputed)
Review verdicts (DrahtBot): 0 (+1)
- Stale ACK: optout21
- Concept ACK: arejula27, l0rinc, polespinasa
Dependencies
Enables:
- #26966 index: initial sync speedup, parallelize process
Files
File list not available for this run.
Card
This PR batches LevelDB writes across configurable block ranges during index initial sync, replacing per-block writes and reducing cs_main lock contention. Benchmarks show an 11% to 28% speedup on HDD for txindex and txospenderindex, a reduction in coinstatsindex LevelDB files from 144 to 10, and groundwork for multithreaded sync in #26966. The PR is in draft and needs a rebase against recent index changes. While concept agreement is positive, l0rinc holds an open blocking review regarding state corruption if interrupted mid-batch, which the author is currently coordinating to resolve.