#36480 txospenderindex: hash keys and pack positions to reduce disk usage

ReviewabilityReviewsAgreementSize
#36480 arejula27 txospenderindex: hash keys and pack positions to reduce disk usageCI failing0Strong402 + 387 tests
  • Shrink the transaction output spender index disk footprint by roughly a third
  • Reduce index sync time and help prepare spender indexing for pruned nodes
  • CI is failing; check whether the failure is caused by this PR before doing a deep review
  • Strong support from tests showing spender index drops from ~84GB to ~62GB with faster sync (andrewtoth)
  • Concept approval favoring future pruning support and merging #36002 first (l0rinc)
  • Concept approval without stated reasons (w0xlt, sedited)
  • 402 lines added or modified outside tests
  • 387 lines added or modified in tests
  • 264 lines removed in total
  • 15 files, 6 commits
Indexes#7 of 11
P2 · speedup
  • P2 because it cuts spender index disk usage by 33%, saving over 20GB on a full node
  • Significantly lowers initial indexing time and helps enable pruned node support
Reviewability: ReadyStalePaused
Agreement: StrongPositivePositive w/ caveatsNeutralMildDisputedBlockedCrickets
Reviews: current code-review ACKs, then (+stale ACKs) and -NACKs; greener = more ACKs.
Size: lines added or modified outside tests, then in tests; greener = smaller.

Goal

Adapts txospenderindex to use the compact on-disk key format introduced for txindex in PR #35531. Key entries are shortened by replacing raw disk transaction positions with block sequence numbers and a 5-byte salted hash prefix of the spent outpoint. Common block-sequence tracking and sync locator logic are factored out into shared index utilities.

Problem: Each txospenderindex entry consumes roughly 18 bytes on disk using 8-byte SipHash-2-4 keys and raw disk file positions (CDiskTxPos), leading to larger disk usage and preventing the index from easily referencing blocks instead of local files (a prerequisite for pruned-mode support).

Evidence of importance:

  • PR description: 'Each entry is about 33% smaller (key bytes): before: 's' (1) + SipHash-2-4 of the outpoint (8) + CDiskTxPos, as varints nFile (2) + nPos (4) + nTxOffset (3) = 18 bytes; after: 'x' (1) + SipHash-1-3-UJ prefix (5) + block sequence varint (3) + tx offset (3) = 12 bytes'
  • PR description: 'The motivation is to save space and to allow replicating what #36002 does for txindex, by referencing blocks instead of local files.'

Scope: Extracts block sequence and locator utilities out of txindex into new files src/index/block_seq.{h,cpp} and src/index/base.{h,cpp} before updating txospenderindex.

Risk of a mistake: medium (rpc_api, other). Bugs in index migration, serialization, or active-chain filtering could cause gettxspendingprevout to return incorrect spending data or corrupt the spender index DB.

Reviewability

Ready · CI failing

CI is red; check whether the failure is the PR's before a deep review.

Waiting on (from the thread read):

  • nothing: no items pending

Agreement

Strong

substantive support, no open objection (andrewtoth)

Thread notes: Added Concept ACK from sedited.

Objections, suggestions, and questions:

ReviewerTypeHarm, request, or questionStatusMust beAuthor repliedQuote
andrewtoth (member)suggestion, scopeEnable pruning in txospenderindex in this PR alongside the key changes, similar to #36002.accepted · andrewtoth withdrew the suggestion in favor of waiting for #36002 to merge firstansweredyes2026-10-08: 'Do we want to also enable pruning in one go like we are with #36002? I don't think it would be controversial so we could do it all in 1 PR.'
Status from: 2026-10-08: andrewtoth: 'On second thought, if we merge #36002 first we can reuse the TxLookupResult'

Support:

  • andrewtoth (member) (Concept ACK, 2026-10-08) [tested]: Previous experiments showed database size reduced from ~84GB to ~62GB and indexing time was much lower
  • l0rinc (member) (Concept ACK, 2026-10-09): supports pruning support and merging #36002 first [not substantive]
  • w0xlt (contributor) (Concept ACK, 2026-10-15): (no reason given) [not substantive]
  • sedited (member) (Concept ACK, 2026-10-23): (no reason given) [not substantive]

Participants:

  • andrewtoth (member, support, 2 statements 2026-10-08 to 2026-10-08): Concept ACK based on prior experiments; suggested bundling pruning before retracting the idea to wait for #36002
  • l0rinc (member, support, 1 statements 2026-10-09 to 2026-10-09): Concept ACK; favors pruning support and merging #36002 first
  • w0xlt (contributor, support, 1 statements 2026-10-15 to 2026-10-15): Concept ACK
  • sedited (member, support, 1 statements 2026-10-23 to 2026-10-23): Concept ACK

Dependencies

Enables:

Files

394 lines under test/bench/ci.

  • src/test/txospenderindex_tests.cpp +283/-5
  • src/index/txospenderindex.cpp +109/-105
  • src/index/block_seq.h +121/-0
  • test/functional/feature_txospenderindex_compatibility.py +100/-0
  • src/index/txindex_key.h +9/-78
  • src/index/txospenderindex_key.h +78/-0
  • src/index/txindex.cpp +8/-43
  • src/index/txospenderindex.h +23/-16
  • src/index/block_seq.cpp +26/-0
  • src/index/base.cpp +14/-3
  • src/index/base.h +10/-3
  • src/rpc/mempool.cpp +3/-9
  • test/functional/rpc_gettxspendingprevout.py +3/-2
  • src/CMakeLists.txt +1/-0
  • test/functional/test_runner.py +1/-0

Card

The short text the ranking pass compares across PRs.

Adopts the compact on-disk key scheme from txindex (#35531) for txospenderindex, reducing per-entry key size by approximately 33% (from 18 to 12 bytes). Replacing file-based disk positions with block sequence numbers also prepares txospenderindex to eventually support pruned mode. The change refactors shared block-sequencing helpers out of txindex, handles hash prefix collisions by filtering candidates against the active chain, and supports reading legacy index entries.

Data

dossier JSON · extract JSON · model ledger, generated 2026-10-24T01:18, confidence high, input hash 36169e4bf5b15e32
thread read through 2026-10-23 (7 statements), head fadf210822
labels: CI failed · draft

Categories

Indexes

P2 · speedup

Reduces txospenderindex entry sizes by 33%, dropping overall database size from ~84GB to ~62GB with noticeably faster indexing time. Reviewer notes confirm significant disk savings and performance improvements, while decoupling block indexing to facilitate pruned mode support.

Membership: Modifies src/index/txospenderindex.*, src/index/txindex.*, and introduces shared index sequence helpers in src/index/block_seq.*

Factors: security/stability 0, bug 0, performance 2, user value 2, leverage 1