#36480 txospenderindex: hash keys and pack positions to reduce disk usage
| Reviewability | Reviews | Agreement | Size | ||||
|---|---|---|---|---|---|---|---|
| #36480 txospenderindex: hash keys and pack positions to reduce disk usage | CI failing | 0 | Strong | 402 + 387 tests | |||
|
|
|
|
| |||
Reviewability: ReadyStalePaused Agreement: StrongPositivePositive w/ caveatsNeutralMildDisputedBlockedCrickets Reviews: current code-review ACKs, then (+stale ACKs) and -NACKs; greener = more ACKs. Size: lines added or modified outside tests, then in tests; greener = smaller. | |||||||
Goal
Adapts txospenderindex to use the compact on-disk key format introduced for txindex in PR #35531. Key entries are shortened by replacing raw disk transaction positions with block sequence numbers and a 5-byte salted hash prefix of the spent outpoint. Common block-sequence tracking and sync locator logic are factored out into shared index utilities.
Problem: Each txospenderindex entry consumes roughly 18 bytes on disk using 8-byte SipHash-2-4 keys and raw disk file positions (CDiskTxPos), leading to larger disk usage and preventing the index from easily referencing blocks instead of local files (a prerequisite for pruned-mode support).
Evidence of importance:
- PR description: 'Each entry is about 33% smaller (key bytes): before: 's' (1) + SipHash-2-4 of the outpoint (8) + CDiskTxPos, as varints nFile (2) + nPos (4) + nTxOffset (3) = 18 bytes; after: 'x' (1) + SipHash-1-3-UJ prefix (5) + block sequence varint (3) + tx offset (3) = 12 bytes'
- PR description: 'The motivation is to save space and to allow replicating what #36002 does for txindex, by referencing blocks instead of local files.'
Scope: Extracts block sequence and locator utilities out of txindex into new files src/index/block_seq.{h,cpp} and src/index/base.{h,cpp} before updating txospenderindex.
Risk of a mistake: medium (rpc_api, other). Bugs in index migration, serialization, or active-chain filtering could cause gettxspendingprevout to return incorrect spending data or corrupt the spender index DB.
Reviewability
Ready · CI failing
CI is red; check whether the failure is the PR's before a deep review.
Waiting on (from the thread read):
- nothing: no items pending
Agreement
Strong
substantive support, no open objection (andrewtoth)
Thread notes: Added Concept ACK from sedited.
Objections, suggestions, and questions:
| Reviewer | Type | Harm, request, or question | Status | Must be | Author replied | Quote |
|---|---|---|---|---|---|---|
| andrewtoth (member) | suggestion, scope | Enable pruning in txospenderindex in this PR alongside the key changes, similar to #36002. | accepted · andrewtoth withdrew the suggestion in favor of waiting for #36002 to merge first | answered | yes | 2026-10-08: 'Do we want to also enable pruning in one go like we are with #36002? I don't think it would be controversial so we could do it all in 1 PR.' Status from: 2026-10-08: andrewtoth: 'On second thought, if we merge #36002 first we can reuse the TxLookupResult' |
Support:
- andrewtoth (member) (Concept ACK, 2026-10-08) [tested]: Previous experiments showed database size reduced from ~84GB to ~62GB and indexing time was much lower
- l0rinc (member) (Concept ACK, 2026-10-09): supports pruning support and merging #36002 first [not substantive]
- w0xlt (contributor) (Concept ACK, 2026-10-15): (no reason given) [not substantive]
- sedited (member) (Concept ACK, 2026-10-23): (no reason given) [not substantive]
Participants:
- andrewtoth (member, support, 2 statements 2026-10-08 to 2026-10-08): Concept ACK based on prior experiments; suggested bundling pruning before retracting the idea to wait for #36002
- l0rinc (member, support, 1 statements 2026-10-09 to 2026-10-09): Concept ACK; favors pruning support and merging #36002 first
- w0xlt (contributor, support, 1 statements 2026-10-15 to 2026-10-15): Concept ACK
- sedited (member, support, 1 statements 2026-10-23 to 2026-10-23): Concept ACK
Dependencies
Enables:
Files
394 lines under test/bench/ci.
- src/test/txospenderindex_tests.cpp +283/-5
- src/index/txospenderindex.cpp +109/-105
- src/index/block_seq.h +121/-0
- test/functional/feature_txospenderindex_compatibility.py +100/-0
- src/index/txindex_key.h +9/-78
- src/index/txospenderindex_key.h +78/-0
- src/index/txindex.cpp +8/-43
- src/index/txospenderindex.h +23/-16
- src/index/block_seq.cpp +26/-0
- src/index/base.cpp +14/-3
- src/index/base.h +10/-3
- src/rpc/mempool.cpp +3/-9
- test/functional/rpc_gettxspendingprevout.py +3/-2
- src/CMakeLists.txt +1/-0
- test/functional/test_runner.py +1/-0
Card
The short text the ranking pass compares across PRs.
Adopts the compact on-disk key scheme from txindex (#35531) for txospenderindex, reducing per-entry key size by approximately 33% (from 18 to 12 bytes). Replacing file-based disk positions with block sequence numbers also prepares txospenderindex to eventually support pruned mode. The change refactors shared block-sequencing helpers out of txindex, handles hash prefix collisions by filtering candidates against the active chain, and supports reading legacy index entries.
Data
thread read through 2026-10-23 (7 statements), head fadf210822
labels: CI failed · draft
Categories
Indexes
P2 · speedup
Reduces txospenderindex entry sizes by 33%, dropping overall database size from ~84GB to ~62GB with noticeably faster indexing time. Reviewer notes confirm significant disk savings and performance improvements, while decoupling block indexing to facilitate pruned mode support.
Membership: Modifies src/index/txospenderindex.*, src/index/txindex.*, and introduces shared index sequence helpers in src/index/block_seq.*
Factors: security/stability 0, bug 0, performance 2, user value 2, leverage 1