#36629 Add output script index and corresponding script history RPC

ReviewabilityReviewsAgreementSize
#36629 andrewtoth Add output script index and corresponding script history RPCReady1Strong1,174 + 1,060 tests
  • Allow querying historical transactions by script or address without wallet imports
  • Avoid requiring external indexing stacks or Electrum servers to sync separate databases
  • Ready to review
  • Strong support across reviewers, with one Tested ACK verifying fixes on the updated push (kravens)
  • Concept approval from multiple contributors (sedited, l0rinc, w0xlt, romanz, ViniciusCestarii, xyzconstant)
  • Earlier objections around disk read budgets and library boundaries are resolved (kravens)
  • Nonblocking suggestions remain open regarding mempool error handling and help text (ViniciusCestarii, kravens)
  • 1,174 lines added or modified outside tests
  • 1,060 lines added or modified in tests
  • 31 lines removed in total
  • 30 files, 5 commits
Indexes#6 of 11
P2 · new feature
  • P2 because it adds a long-sought script index functioning on full and pruned nodes
  • Resolves an architectural gap attempted since 2013 to serve external indexers without wallet imports
  • Ranking pass: New index with a stated consumer ranks below integrity fixes and sync-time work per the category ordering, hence 4 places below its own score position.
Ranking notes
RPC / REST / ZMQ#10 of 55
P2 · user request
  • P2 because it exposes a getscripthistory RPC repeatedly requested by downstream tooling
  • Allows querying transaction history and spends by script hash or descriptor window directly from the node
Ranking notes
Reviewability: ReadyStalePaused
Agreement: StrongPositivePositive w/ caveatsNeutralMildDisputedBlockedCrickets
Reviews: current code-review ACKs, then (+stale ACKs) and -NACKs; greener = more ACKs.
Size: lines added or modified outside tests, then in tests; greener = smaller.

Goal

Adds an optional transaction output script index (`txoscriptindex`) that indexes script pubkeys by a five-byte SipHash prefix of their SHA256 hash followed by an order-preserving `OrderedVarInt` block height. Introduces a `getscripthistory` RPC allowing queries for transaction outputs and spending transactions by script hash, address, or ranged descriptor across a block height window. Also exposes transaction offsets in `txospenderindex` and refactors transaction JSON rendering into shared RPC helpers.

Problem: Bitcoin Core cannot query historical transactions by address or script without importing them into a wallet, forcing operators of external tools and Electrum servers to run and sync resource-intensive third-party indexing stacks.

Evidence of importance:

  • Address indexing has been repeatedly attempted in closed PRs #2802, #3652, #6835, #10370, and #14053 dating back to 2013.
  • Author reports that the new index 'can be synced in a few hours and the current disk usage is ~52 GB' and functions on pruned nodes, though these benchmarks have not yet been corroborated by independent reviewers.

Scope: Spans a new index implementation with a custom lexicographically order-preserving VarInt serializer (`OrderedVarIntFormatter`), modifications to `txospenderindex` to export transaction offsets, refactored raw transaction JSON documentation and formatting helpers, and extensive functional test suites.

Risk of a mistake: medium (rpc_api, other). Flaws in index key encoding, LevelDB range seeks, or RPC byte-budget enforcement could cause incorrect transaction history results, LevelDB corruption, or excessive disk I/O DoS on the node.

Changed since the previous assessment: Exposes transaction byte offsets from block headers in `txospenderindex`, extracts shared transaction documentation into `RawTransactionDoc`, and adds test coverage verifying that script history queries are disabled during assumeutxo background validation.

What the assessment lacked:

  • rest_of_diff

Reviewability

Ready

Nothing known that would invalidate a review now.

Waiting on (from the thread read):

  • author: response to observations regarding mempool conflict error and missing release notes file 2026-10-29

Agreement

Strong

substantive support, no open objection (kravens, kravens)

Thread notes: Author pushed an update with a new index layout, budget-enforced undo reads, and header fixes; kravens verified the fixes and gave a Tested ACK while dropping remaining earlier suggestions and adding non-blocking observations.

Objections, suggestions, and questions:

ReviewerTypeHarm, request, or questionStatusMust beAuthor repliedQuote
kravens (contributor)objection, safetyThe lookup budget does not count block and undo reads, allowing queries that fit in the 50 MiB budget to read far more disk than intended.accepted · fix pushed · author bounded reads to budget; reviewer confirmed fixedansweredyes2026-10-23: 'The lookup budget does not count block and undo reads. At verbosity 2, tx_to_json reads the full block and its undo data for every block that holds a non-coinbase match'
Status from: 2026-10-29: kravens: 'From my earlier comments, 1 (block and undo reads outside the budget), 3 (spender ordering without a block read) and 7 (TxToJSON header) are fixed'
kravens (contributor)suggestion, approachUse BigEndianFormatter<3> instead of introducing OrderedVarInt.accepted · reviewer dropped the suggestionansweredno2026-10-23: 'OrderedVarInt could be BigEndianFormatter<3>. Big-endian already sorts by height, and the same key already uses it for the tx offset.'
Status from: 2026-10-29: kravens: '2, 4, 5 and 6 were suggestions; no objection to leaving them.'
kravens (contributor)suggestion, approachSort extra spenders by (height, tx_offset) using CDiskTxPos from txospenderindex instead of reading full blocks.accepted · fix pushed · tx_offset exposed and step-10 block read removed; confirmed by revieweransweredyes2026-10-23: 'Ordering spenders doesn't need a block read. txospenderindex keys already store a CDiskTxPos, and its nTxOffset grows with the transaction's position in the block.'
Status from: 2026-10-29: kravens: 'From my earlier comments, 1 (block and undo reads outside the budget), 3 (spender ordering without a block read) and 7 (TxToJSON header) are fixed'
kravens (contributor)suggestion, approachHave Find take the whole set of script hashes to avoid duplicate transaction reads and simplify post-processing.accepted · reviewer dropped the suggestionansweredno2026-10-23: 'Find could take the whole set of script hashes. Today the RPC calls Find once per hash and merges results by (height, tx_offset).'
Status from: 2026-10-29: kravens: '2, 4, 5 and 6 were suggestions; no objection to leaving them.'
kravens (contributor)suggestion, interfaceUnify index-layer and RPC-layer budget enforcement and error reporting into a single type.accepted · reviewer dropped the suggestionansweredno2026-10-23: 'One budget, one error. Index lookups charge a uint64_t& and fail with "Script history index lookup limit exceeded…". Extra transaction bodies throw a std::runtime_error'
Status from: 2026-10-29: kravens: '2, 4, 5 and 6 were suggestions; no objection to leaving them.'
kravens (contributor)suggestion, maintenanceMove getscripthistory RPC handler into its own file (src/rpc/scripthistory.cpp) with one function per step.accepted · reviewer dropped the suggestionansweredno2026-10-23: 'Could the RPC go in its own file? blockchain.cpp grows by ~530 lines to 4,238. The handler is one ~400-line lambda that opens with a 12-step index comment.'
Status from: 2026-10-29: kravens: '2, 4, 5 and 6 were suggestions; no objection to leaving them.'
kravens (contributor)objection, interfaceDeclaring TxToJSON in rawtransaction_util.h violates library boundaries because the header belongs to bitcoin_common while TxToJSON takes a Chainstate and is defined in bitcoin_node.accepted · fix pushed · author moved TxToJSON to rpc/rawtransaction.h; reviewer confirmed fixedchangedyes2026-10-23: 'TxToJSON is now declared in rawtransaction_util.h. That header belongs to bitcoin_common, which the wallet also links. TxToJSON takes a Chainstate and is still defined in rawtransaction.cpp in bitcoin'
Status from: 2026-10-29: kravens: 'From my earlier comments, 1 (block and undo reads outside the budget), 3 (spender ordering without a block read) and 7 (TxToJSON header) are fixed'
romanz (contributor)suggestion, approachAvoid SHA256 hashing for script indexing since Electrum 1.7.x protocol now uses scriptPubKey-based queries directly.answered · author replied keeping it for backwards compatibilityansweredyes2026-10-26: 'Electrum 1.7.x protocol is now using scriptPubKey-based subscription and queries, so maybe it would simpler to avoid SHA256 here?'
Status from: 2026-10-29: andrewtoth: 'I'm not sure it will make it much simpler. I think it's worth it to keep this for backwards compatibility.'
ViniciusCestarii (contributor)suggestion, interfaceState in getscripthistory help that pruned_blocks can be fetched with getblockfrompeer and can contain false positives.openansweredno2026-10-26: 'getscripthistory help could say pruned_blocks blocks can be fetched with getblockfrompeer and then retry. It also could also state that pruned_blocks can contain false positives'
kravens (contributor)suggestion, approachAllow asof queries during initial index sync by skipping BlockUntilSyncedToCurrentChain() when asof is provided.openansweredno2026-10-29: 'Skipping the two BlockUntilSyncedToCurrentChain() calls when asof is given would let a server start paging old history while the index catches up. Fine as a follow-up.'
kravens (contributor)suggestion, interfaceThrow an RPC_INVALID_PARAMETER error when include_mempool=true is explicitly set together with asof.openansweredno2026-10-29: 'Explicit include_mempool=true together with asof is silently ignored (rpc/blockchain.cpp:3890). The help text says so, but an RPC_INVALID_PARAMETER would make the conflict visible to a caller who set both on purpose.'
kravens (contributor)question, maintenanceDid doc/release-notes-36629.md miss the push?openansweredno2026-10-29: 'Release notes. The summary mentions release notes, but the branch only touches doc/files.md. Did a doc/release-notes-36629.md miss the push?'

Support:

  • sedited (member) (Concept ACK, 2026-10-23): (no reason given) [not substantive]
  • l0rinc (member) (Concept ACK, 2026-10-23): (no reason given) [not substantive]
  • w0xlt (contributor) (Concept ACK, 2026-10-23): (no reason given) [not substantive]
  • kravens (contributor) (Concept ACK, 2026-10-23) [both]: Built and ran unit/functional tests and manual regtest testing covering address, Electrum scripthash, ranged wpkh() descriptor, verbosities, and spender linking with expected results
  • romanz (contributor) (Concept ACK, 2026-10-26): (no reason given) [not substantive]
  • ViniciusCestarii (contributor) (Concept ACK, 2026-10-26): (no reason given) [not substantive]
  • xyzconstant (contributor) (Concept ACK, 2026-10-29): (no reason given) [not substantive]
  • kravens (contributor) (Tested ACK, 2026-10-29) [both: src/index/txoscriptindex, rpc/blockchain.cpp]: Built and ran unit and functional test suites; verified compact index layout, 6-byte prefix, and asof paging semantics under reorgs

Participants:

  • kravens (contributor, objection, 2 statements 2026-10-23 to 2026-10-29): Tested ACK on latest push; earlier objections on unbudgeted reads and header boundaries accepted as fixed
  • l0rinc (member, support, 1 statements 2026-10-23 to 2026-10-23): Concept ACK
  • sedited (member, support, 1 statements 2026-10-23 to 2026-10-23): Concept ACK
  • w0xlt (contributor, support, 1 statements 2026-10-23 to 2026-10-23): Strong Concept ACK
  • ViniciusCestarii (contributor, support, 3 statements 2026-10-26 to 2026-10-26): Concept ACK; suggested doc clarifications on pruned_blocks in RPC help
  • romanz (contributor, support, 2 statements 2026-10-26 to 2026-10-26): Concept ACK; suggested avoiding SHA256 (author answered)
  • xyzconstant (contributor, support, 1 statements 2026-10-29 to 2026-10-29): Concept ACK

Files

1064 lines under test/bench/ci.

  • test/functional/rpc_getscripthistory.py +609/-0
  • src/rpc/blockchain.cpp +587/-1
  • src/test/txoscriptindex_tests.cpp +302/-0
  • src/index/txoscriptindex.cpp +200/-0
  • src/index/txoscriptindex.h +101/-0
  • src/index/txoscriptindex_key.h +98/-0
  • src/test/ordered_varint_tests.cpp +95/-0
  • src/index/ordered_varint.h +87/-0
  • src/rpc/rawtransaction_util.cpp +29/-3
  • src/rpc/rawtransaction.cpp +7/-23
  • src/rpc/rawtransaction.h +21/-0
  • src/test/caches_tests.cpp +14/-0
  • src/init.cpp +11/-0
  • test/functional/feature_assumeutxo.py +10/-1
  • test/functional/feature_init.py +11/-0
  • src/test/txospenderindex_tests.cpp +9/-1
  • src/rpc/client.cpp +9/-0
  • src/rpc/rawtransaction_util.h +7/-0
  • src/node/caches.cpp +6/-0
  • src/rpc/node.cpp +5/-0
  • test/functional/rpc_misc.py +3/-2
  • src/test/baseindex_tests.cpp +3/-0
  • src/index/txospenderindex.h +2/-0
  • src/test/CMakeLists.txt +2/-0
  • doc/files.md +1/-0
  • src/CMakeLists.txt +1/-0
  • src/index/txospenderindex.cpp +1/-0
  • src/node/caches.h +1/-0
  • src/test/fuzz/rpc.cpp +1/-0
  • test/functional/test_runner.py +1/-0

Card

The short text the ranking pass compares across PRs.

Adds an optional `txoscriptindex` and a `getscripthistory` RPC to query output and spending transactions by address, script hash, or ranged descriptor. Bitcoin Core currently cannot query script history without wallet keys or descriptors, forcing node operators running external wallets or Electrum servers to maintain third-party indexers. Script and address indexing has been proposed and closed in at least five previous pull requests since 2013. The author states the index syncs in a few hours, requires ~52 GB of disk space, and works with block pruning.

Data

dossier JSON · extract JSON · model ledger, generated 2026-11-01T01:16, confidence high, input hash 644ade2497eee46f
thread read through 2026-10-29 (13 statements), head 1fc5b618e3
labels: none

Categories

Indexes

P2 · new feature (ranking pass: New index with a stated consumer ranks below integrity fixes and sync-time work per the category ordering, hence 4 places below its own score position.)

Adds a long-sought script index allowing pruned and full node operators to track script history. Script indexing resolves an architectural gap attempted since 2013 across PRs #2802, #3652, #6835, #10370, and #14053 to serve external indexers and Electrum servers without requiring wallet imports.

Membership: Implements a new optional index in `src/index/txoscriptindex.{h,cpp}` alongside `OrderedVarInt` key formatting and updates to `txospenderindex`.

Factors: security/stability 0, bug 0, performance 1, user value 3, leverage 1

RPC / REST / ZMQ

P2 · user request

Exposes a public RPC method repeatedly requested by downstream tooling to query historical transactions and spends by script hash or descriptor window. This fills a significant capability gap in Bitcoin Core's programmatic interface that previously forced external consumers to run separate indexing stacks.

Membership: Adds the `getscripthistory` RPC method in `src/rpc/blockchain.cpp` and refactors transaction JSON rendering into `src/rpc/rawtransaction_util.cpp`.

Factors: security/stability 0, bug 0, performance 1, user value 3, leverage 1