#36002 txindex: allow running in pruned mode
| Reviewability | Reviews | Agreement | Size | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| #36002 txindex: allow running in pruned mode | Ready | 1 | Strong | 211 + 166 tests | ||||||
|
|
|
| |||||||
Reviewability: ReadyStalePaused Agreement: StrongPositivePositive w/ caveatsNeutralMildDisputedBlockedCrickets Reviews: current code-review ACKs, then (+stale ACKs) and -NACKs; greener = more ACKs. Size: lines added or modified outside tests, then in tests; greener = smaller. | ||||||||||
Goal
Permits running -txindex concurrently with block pruning (-prune) as long as the index does not contain legacy file-offset entries. When a transaction lookup via txindex or GetTransaction targets a block that has been pruned from disk, the lookup returns candidate pruned block hashes. RPC calls (getrawtransaction, gettxoutproof) and the REST /tx endpoint return structured error information containing those block hashes so callers can fetch the missing blocks on demand via getblockfrompeer.
Problem: Running txindex has historically been blocked on pruned nodes because legacy index entries stored direct file positions on disk. Pruned node operators could not use txindex to locate transactions or service RPC lookups like getrawtransaction without storing the full blockchain on disk.
Evidence of importance:
- Resolves four long-standing issues requesting txindex support on pruned nodes dating back to 2017: #10609, #12651, #16182, and #35240 (PR description).
Risk of a mistake: medium (rpc_api). Modifies JSON-RPC error formatting by adding an optional data field, alters getrawtransaction and gettxoutproof error responses for pruned blocks, and updates transaction lookup behavior across node RPCs and REST endpoints.
Changed since the previous assessment: Added handling in getrawtransaction with verbosity 2 to gracefully omit fee and prevout information when undo data is absent (e.g. after blocks retrieved via getblockfrompeer), and added release notes.
Reviewability
Ready
Nothing known that would invalidate a review now.
Waiting on (from the thread read):
- reviewer: re-review from l0rinc after author implemented all suggestions 2026-10-18
- author: unanswered question on TxIndexMissErrorDetails from fjahr 2026-09-28
Agreement
Strong
substantive support, no open objection (ajtowns, arejula27, optout21, optout21)
Thread notes: Added ACK from optout21 (c:6446697433) reviewing commit 4ab48b827691ade8ceaa78b5125e9bb610d37aa6 and noting the active-pruned vs stale-unpruned edge case is acceptable.
Objections, suggestions, and questions:
| Reviewer | Type | Harm, request, or question | Status | Must be | Author replied | Quote |
|---|---|---|---|---|---|---|
| ajtowns (member) | question, approach | What is the intended behavior when txindex is enabled on an already pruned node? | answered · author explained it triggers the same startup error as coinstatsindex or blockfilterindex | answered | yes | 2026-08-25: 'What's the (intended) behaviour if you enable txindex on a node that is already pruned?' Status from: 2026-08-29: andrewtoth: 'It's the same behavior if you try and start coinstatsindex or blockfilterindex after you've pruned past their last index' |
| ajtowns (member) | suggestion, approach | add a -prunetxindex=1 option to allow looking up txs without extra disk growth | answered · author replied this could be added as a follow-up | answered | yes | 2026-08-25: 'I'd definitely prefer to have a -prunetxindex=1 option I could set, so that my prune=2000 (~1 week worth of blocks) node could lookup txs without needing to be reprovisioned with extra disk though.' Status from: 2026-08-29: andrewtoth: 'I think that could be added as a follow-up, and would not conflict with this approach.' |
| mzumsande (member) | suggestion, approach | Consider a partial index that only indexes non-pruned data so txindex can be enabled on already-pruned nodes | answered · author removed partial indexing to keep scope smaller and defer to a follow-up | answered | yes | 2026-09-07: 'However, it would have the big upside that it could be enabled on an already pruned node.' Status from: 2026-09-29: andrewtoth: 'Partially pruned indexes are out of scope and can be left for a follow-up.' |
| arejula27 (contributor) | objection, interface | Return pruned block hashes as structured data rather than inside the error string | fixed · fix pushed · author returned pruned block hashes in optional data field of JSON error response | answered | yes | 2026-09-13: 'Right now those hashes only reach the caller inside the error string, so automating the fetch and retry means parsing the message. Would returning them as structured data be worth it?' Status from: 2026-09-18: andrewtoth: 'I added returning the pruned block hashes in an optional data field of the JSON error response.' |
| arejula27 (contributor) | suggestion, interface | Use a single enum rather than two separate flags in BaseIndex to express prune policy invariants | fixed · fix pushed · author replaced AllowPrune with GetPrunePolicy returning an enum | answered | yes | 2026-09-13: 'Would a single enum value express that relationship better, so the invariant lives in the type instead of a convention that a future index could break?' Status from: 2026-09-18: andrewtoth: 'I renamed AllowPrune to GetPrunePolicy which returns an enum.' |
| arejula27 (contributor) | suggestion, interface | Report the pruned block hash in gettxoutproof when the transaction has unspent outputs | open | answered | no | 2026-09-24: 'Would it make sense to report that hash too?' |
| arejula27 (contributor) | suggestion, interface | Mention deleting the index directory to rebuild partial history from disk when an index falls behind pruned data | answered · author replied this is no longer relevant after removing partial indexing | answered | yes | 2026-09-24: 'Would it be worth mentioning that for indexes that allow partial history?' Status from: 2026-09-29: andrewtoth: 'Thanks, but this is no longer relevant since we will not support partial indexes in this change.' |
| arejula27 (contributor) | suggestion, interface | Use std::variant<TxFound, TxMiss> rather than one struct to ensure consistent fields | fixed · fix pushed · author implemented std::variant<TxFound, TxMiss> | answered | yes | 2026-09-24: 'My proposal is to use `std::variant<TxFound, TxMiss>` rather than one struct where the outcome is implied by which fields are set.' Status from: 2026-09-29: andrewtoth: 'Done.' |
| arejula27 (contributor) | suggestion, correctness | Add test coverage for pruned blocks outside active chain and returning multiple candidate pruned block hashes | fixed · fix pushed · author added the requested test cases | answered | yes | 2026-09-24: 'I would add two more scenarios, as I have tested them and they are not covered' Status from: 2026-09-29: andrewtoth: 'Added these cases.' |
| arejula27 (contributor) | question, style | Whether result can be returned directly instead of reconstructing the object | answered · author removed this code and noted it was no longer relevant without partial indexing | answered | yes | 2026-09-24: 'Is there a reason not to `return result;` directly?' Status from: 2026-09-29: andrewtoth: 'Removed this, but also it seems no longer relevant since we will not support partial indexes in this change.' |
| arejula27 (contributor) | suggestion, interface | Use a neutral base error message like 'Transaction not found' when appending partial history details in gettxoutproof | answered · author replied this is no longer relevant after removing partial indexing | answered | yes | 2026-09-24: 'Maybe use a neutral base message here too when the partial history detail is appended, e.g. "Transaction not found"?' Status from: 2026-09-29: andrewtoth: 'Thanks, but this is no longer relevant since we will not support partial indexes in this change.' |
| fjahr (member) | suggestion, scope | Split the PR into enabling plain prune mode first and keep partial indexing for a follow-up | fixed · fix pushed · author removed partial indexing from the PR to leave for a follow-up | answered | yes | 2026-09-28: 'Would you mind splitting the PR into enabling plain prune mode and keep the partial index and all the related refactorings for a follow-up to that?' Status from: 2026-09-29: andrewtoth: 'I have updated this to remove the partial index, so this can only work if synced from genesis. Partially pruned indexes ' |
| fjahr (member) | objection, interface | Changing the semantics of synced: true in getindexinfo may break clients that do not check first_block_height | fixed · fix pushed · partial indexing was removed, reverting getindexinfo synced semantics | answered | yes | 2026-09-28: '`synced: true` doesn't really mean the same thing as before. So some clients which are not checking the new `first_block_height` may not get what they think they are getting.' Status from: 2026-09-29: andrewtoth: 'Thanks, but this is no longer relevant since we will not support partial indexes in this change.' |
| fjahr (member) | suggestion, interface | Consider requiring an explicit opt-in flag for partial indexing to communicate the behavior clearly to users | answered · author removed partial indexing from this PR | answered | yes | 2026-09-28: 'Starting with `-prune=XXX -txindex=1` may give you a partial or full txindex depending on if you had the index on before. Possibly the partial index could use an explicit opt-in?' Status from: 2026-09-29: andrewtoth: 'I have updated this to remove the partial index, so this can only work if synced from genesis.' |
| fjahr (member) | question, correctness | Whether all information in TxIndexMissErrorDetails like the synced check can ever be hit given f_txindex_ready | open | answered | no | 2026-09-28: 'Because of what we do with `f_txindex_ready` above I am not sure all the infos in `TxIndexMissErrorDetails` would ever be relevant or possible to hit' |
| optout21 (contributor) | suggestion, interface | Consider mentioning in the message that restarting block download allows rebuilding a full txindex when blocks are pruned | accepted · reviewer acknowledged the original request was mistaken and author addressed an edge case | answered | yes | 2026-09-29: 'If blocks are pruned, the rebuilt txindex will be pruned. The other option is to also restart block download; that way the txindex will be full. If that's the case, is this worth mentioning as well?' Status from: 2026-09-30: optout21: 'Feel free to Resolve.' |
| optout21 (contributor) | question, correctness | In what cases besides duplicate BIP30 coinbases can a transaction hash be present in the active chain more than once | accepted · author explained false positives and duplicate coinbases, which reviewer accepted | answered | yes | 2026-09-29: 'I just wonder in what cases can a TX hash be present in the active chain more than once? Only duplicate coinbase (BIP30)?' Status from: 2026-09-30: optout21: 'Clear, thanks.' |
| optout21 (contributor) | suggestion, interface | Return tx_offset_in_block to the caller as well | accepted · reviewer agreed there was no concrete use case and withdrew the suggestion | answered | yes | 2026-09-29: 'I think here we could also return `tx_offset_in_block`; it may be useful in some usage scenarios to pass this to the caller as well.' Status from: 2026-09-30: optout21: 'In conclusion, I'm not able to come up with a concrete use case for this, and you are right, then it's best to leave it.' |
| optout21 (contributor) | suggestion, maintenance | Add doc comment explaining TxMiss is used both when txid is missing and when the block is pruned, and clarify redundant phrasing | accepted · reviewer recognized distinction cannot be made due to prefix false positives | answered | yes | 2026-09-29: 'This struct is used both when a TXID is not found in the index, and when it is found, but the block is not found (pruned). This could be mentioned in a doc comment on the struct.' Status from: 2026-09-30: optout21: 'I would've argued for a stronger conceptual distinction between "Not in the index" and the "In the index but in a pruned' |
| optout21 (contributor) | suggestion, style | Separate the 3 lookup cases (found, found in pruned block, not found) in comments/code for clarity | accepted · reviewer recognized distinction cannot be made due to prefix false positives | answered | yes | 2026-09-29: 'For clarity it would be better to separate 3 cases: found, found in pruned block, and not found at all in the index.' Status from: 2026-09-30: optout21: 'Same as above: I would've argued for a stronger conceptual distinction between "Not in the index" and the "In the index "' |
| optout21 (contributor) | suggestion, interface | Update REST documentation to clarify that a block may not have been indexed or has been pruned | fixed · fix pushed · author adopted the suggested documentation wording | answered | yes | 2026-09-29: 'What about if it exists in a block that was not indexed? More correct would be "in a block that has not been indexed or that has been pruned"' Status from: 2026-09-30: andrewtoth: 'Taken, thanks.' |
| optout21 (contributor) | suggestion, interface | Update RPC docs to explain the new behavior and returned possible block hashes | fixed · fix pushed · author updated help text for getrawtransaction and gettxoutproof | answered | yes | 2026-10-05: 'The RPC docs are not updated to explain the new behavior, and how possible block hashes are returned.' Status from: 2026-10-09: andrewtoth: 'Updated help text for `getrawtransaction` and `gettxoutproof`.' |
| optout21 (contributor) | suggestion, maintenance | Add a release note for the RPC changes | fixed · fix pushed · author added release notes | answered | yes | 2026-10-05: 'This change (esp. because of the RPC changes) may warrant a release note.' Status from: 2026-10-09: andrewtoth: 'Added release notes.' |
| optout21 (contributor) | suggestion, maintenance | Add missing translation string to bitcoinstrings.cpp | answered · author explained they removed them from bitcoinstrings.cpp as they will be generated later | answered | yes | 2026-10-05: 'This string is not added to `bitcoinstrings.cpp`, while the next one is. May be missing.' Status from: 2026-10-09: andrewtoth: 'I removed them from bitcoinstrings.cpp. I believe that will be generated later.' |
| optout21 (contributor) | suggestion, interface | Fix or document the changed error message when txindex is not set but blockhash is provided or found in the UTXO set | fixed · fix pushed · author updated documentation to reflect this | answered | yes | 2026-10-05: 'In those cases the "Transaction may be in pruned block X." message is not correct. This looks like a changed error message, even in some cases independent of txindex. If the change is kept, it should ' Status from: 2026-10-09: andrewtoth: 'Updated documentation to reflect this.' |
| optout21 (contributor) | question, correctness | Whether the change affects non-txindex cases when a block downloaded by getblockfrompeer has no undo data and blockhash is provided | answered · author explained this fixes a bug without txindex when calling getrawtransaction verbosity 2 without undo data | answered | yes | 2026-10-05: 'I wonder if this change may affect non-txindex cases as well, e.g. when a block has been downloaded by `getblockfrompeer` (block data but no undo), and the blockhash is provided.' Status from: 2026-10-09: andrewtoth: 'Yes, this fixes a bug without `txindex` as well. Previously, calling `getrawtransaction` with verbosity 2 and no undo da' |
| optout21 (contributor) | suggestion, interface | De-duplicate candidate block hashes when multiple false positives match the same block | fixed · fix pushed · author changed TxMiss::pruned_block_hashes to std::set to de-duplicate hashes | answered | yes | 2026-10-05: 'Theoretically multiple false positives are possible for the same block, in which case a block hash will be included multiple times. The list could be de-duplicated.' Status from: 2026-10-09: andrewtoth: 'I changed the `TxMiss::pruned_block_hashes` from a `std::vector` to a `std::set` to reflect this.' |
| arejula27 (contributor) | objection, correctness | feature_pruning.py fails when built without the wallet (-DENABLE_WALLET=OFF) because blocks are never pruned before that step | fixed · fix pushed · author fixed the test by using node 2 which always has pruned blocks | changed | yes | 2026-10-05: 'I ran this test in a build without the wallet (`-DENABLE_WALLET=OFF`) and it fails.' Status from: 2026-10-09: andrewtoth: 'Fixed this by using `self.nodes[2]` instead of `self.nodes[0]`. That node does always has pruned blocks.' |
| arejula27 (contributor) | question, interface | Why gettxoutproof and getrawtransaction return different error messages when given an explicit pruned blockhash | fixed · fix pushed · author fixed it so both cases return the same error | answered | yes | 2026-10-05: 'With an explicit blockhash whose block is pruned, `gettxoutproof` now returns `Transaction may be in pruned block <hash>.`, while `getrawtransaction` still returns `Block not available`. Is there a re' Status from: 2026-10-09: andrewtoth: 'Fixed this so both cases return the same error.' |
| l0rinc (member) | suggestion, maintenance | Update the release notes to refer to v32 instead of v31 for the index rebuild requirement | fixed · fix pushed · author implemented all of l0rinc's suggestions | answered | yes | 2026-10-12: 'This should probably be v32.' Status from: 2026-10-18: andrewtoth: 'Thanks @l0rinc for your detailed review. I took all your suggestions.' |
| l0rinc (member) | suggestion, maintenance | Move tx_lookup_result.h from index/ to node/ since GetTransaction uses it for mempool lookups | fixed · fix pushed · author implemented all of l0rinc's suggestions | answered | yes | 2026-10-12: 'Given that `GetTransaction` also uses the result for mempool lookups we could move this one to `node/`.' Status from: 2026-10-18: andrewtoth: 'Thanks @l0rinc for your detailed review. I took all your suggestions.' |
| l0rinc (member) | suggestion, correctness | Test that prune mode rejects a legacy txindex before pruning and move the AllowPrune assertion to an on-disk test | fixed · fix pushed · author implemented all of l0rinc's suggestions | answered | yes | 2026-10-14: 'Could we move this assertion to the existing on-disk test, where it actually checks the database?' Status from: 2026-10-18: andrewtoth: 'Thanks @l0rinc for your detailed review. I took all your suggestions.' |
| l0rinc (member) | suggestion, correctness | Test pruned UTXO lookups without txindex and verify recovered transaction lookups survive a restart | fixed · fix pushed · author implemented all of l0rinc's suggestions | answered | yes | 2026-10-14: 'With txindex enabled, its fallback returns the same error and can mask a broken UTXO lookup.' Status from: 2026-10-18: andrewtoth: 'Thanks @l0rinc for your detailed review. I took all your suggestions.' |
| l0rinc (member) | suggestion, maintenance | Build the pruned-block JSON-RPC error in a single helper to avoid repeating error construction across call sites | fixed · fix pushed · author implemented all of l0rinc's suggestions | answered | yes | 2026-10-14: 'Returning the whole error here would save repeating the same code, message and data at all four call sites.' Status from: 2026-10-18: andrewtoth: 'Thanks @l0rinc for your detailed review. I took all your suggestions.' |
| l0rinc (member) | suggestion, interface | Reuse CheckBlockDataAvailability in getrawtransaction when blockindex is provided and test header-only blocks | fixed · fix pushed · author implemented all of l0rinc's suggestions | answered | yes | 2026-10-14: '`gettxoutproof` follows the same pruned check with `CheckBlockDataAvailability`, which we could reuse here.' Status from: 2026-10-18: andrewtoth: 'Thanks @l0rinc for your detailed review. I took all your suggestions.' |
| l0rinc (member) | suggestion, maintenance | Add a release note that getrawtransaction verbosity 2 omits fee and prevout when undo data is missing | fixed · fix pushed · author implemented all of l0rinc's suggestions | answered | yes | 2026-10-14: '`getrawtransaction` with verbosity 2 now omits `fee` and `prevout` instead of returning an error when the block is available but its undo data is not' Status from: 2026-10-18: andrewtoth: 'Thanks @l0rinc for your detailed review. I took all your suggestions.' |
| l0rinc (member) | question, correctness | What should happen when the active-chain copy is pruned but a stale copy is still readable | fixed · fix pushed · author implemented all of l0rinc's suggestions | answered | yes | 2026-10-14: 'What should happen when the active-chain copy is pruned but a stale copy is still readable?' Status from: 2026-10-18: andrewtoth: 'Thanks @l0rinc for your detailed review. I took all your suggestions.' |
| l0rinc (member) | objection, correctness | Concurrent pruning could delete a block file between the index scan and OpenBlockFile, causing lookup failure rather than reporting the block as pruned | fixed · fix pushed · author implemented all of l0rinc's suggestions | changed | yes | 2026-10-14: 'could pruning remove the file between the scan above and `OpenBlockFile`?' Status from: 2026-10-18: andrewtoth: 'Thanks @l0rinc for your detailed review. I took all your suggestions.' |
| l0rinc (member) | suggestion, interface | Add GetFound() and GetMiss() member accessors to TxLookupResult | fixed · fix pushed · author implemented all of l0rinc's suggestions | answered | yes | 2026-10-14: 'I find `result.GetFound()` and `result.GetMiss()` easier to read than repeating `std::get_if` at each call site.' Status from: 2026-10-18: andrewtoth: 'Thanks @l0rinc for your detailed review. I took all your suggestions.' |
| l0rinc (member) | suggestion, style | Keep candidate deserialization and hash comparison together inside the try block | fixed · fix pushed · author implemented all of l0rinc's suggestions | answered | yes | 2026-10-14: 'This can be a bit shorter if we construct the transaction from the file and check its hash inside the `try` block.' Status from: 2026-10-18: andrewtoth: 'Thanks @l0rinc for your detailed review. I took all your suggestions.' |
Support:
- l0rinc (member) (Concept ACK, 2026-08-18): no reason given: Big concept ACK :D/ [not substantive]
- ajtowns (member) (Concept ACK, 2026-08-25): I think expected block usage is about 100GB per year, so 5 years worth of history would be half a terabyte, and probably cover most lookups... Concept ACK, looks okay to me at first glance
- arejula27 (contributor) (Concept ACK, 2026-09-13) [read]: I'm excited about this PR and I love the idea and the approach... we can even access txs on pruned nodes, since #35531 the index stores the block 'virtual' reference rather than a file po
- optout21 (contributor) (Concept ACK, 2026-09-29): The ability to have a txindex holding information of _all_ transactions even in case of a pruned node is a huge improvement, it makes pruned nodes much more capable.
- l0rinc (member) (Concept ACK, 2026-10-14) [read]: re-concept ACK, overall LGTM, left a few remaining suggestions, it's looks pretty good already. [not substantive]
- optout21 (contributor) (ACK, 2026-10-27) [read]: Building on previous reviews, looks good! Accepts the edge case where an active pruned block with an unpruned stale block returns Found as acceptable.
Participants:
- l0rinc (member, objection, 19 statements 2026-08-18 to 2026-10-14): Concept ACK; requested changes on race condition, test coverage, and code organization
- ajtowns (member, support, 1 statements 2026-08-25 to 2026-08-25): Concept ACK; asked about behavior on already-pruned nodes and suggested -prunetxindex option
- mzumsande (member, neutral, 1 statements 2026-09-07 to 2026-09-07): suggested partial indexing on already-pruned nodes (deferred to follow-up)
- arejula27 (contributor, objection, 11 statements 2026-09-13 to 2026-10-05): Concept ACK; tested on testnet4; objected to test failure without wallet (fixed); raised questions and suggestions
- fjahr (member, objection, 4 statements 2026-09-28 to 2026-09-28): objected to getindexinfo synced: true semantic change (fixed); suggested splitting plain prune mode from partial indexing
- optout21 (contributor, support, 23 statements 2026-09-29 to 2026-10-27): ACK; reviewed code, edge cases, documentation, and error handling
Files
179 lines under test/bench/ci.
- test/functional/feature_txindex_prune.py +105/-0
- src/index/txindex.cpp +32/-18
- src/test/txindex_tests.cpp +36/-7
- src/rpc/rawtransaction.cpp +22/-20
- src/node/tx_lookup_result.h +39/-0
- src/node/transaction.cpp +14/-16
- doc/release-notes-36002.md +26/-0
- src/rpc/txoutproof.cpp +18/-5
- test/functional/feature_txindex_compatibility.py +19/-4
- src/index/txindex.h +8/-13
- src/rpc/util.cpp +20/-0
- src/rest.cpp +9/-3
- src/init.cpp +7/-3
- src/node/transaction.h +4/-3
- src/rpc/util.h +6/-0
- test/functional/feature_pruning.py +4/-2
- src/rpc/request.cpp +4/-1
- doc/REST-interface.md +1/-1
- src/rpc/request.h +1/-1
- test/functional/rpc_blockchain.py +1/-0
- test/functional/test_runner.py +1/-0
Card
The short text the ranking pass compares across PRs.
Allows Bitcoin Core nodes to run txindex concurrently with block pruning when the index contains no legacy file-offset entries. When a transaction lookup targets a pruned block, GetTransaction, REST, and RPC calls (getrawtransaction, gettxoutproof) surface candidate block hashes in structured error data so callers can fetch the block via getblockfrompeer and retry. It addresses several issues opened since 2017 requesting txindex support on pruned nodes (#10609, #12651, #16182, #35240), building on the index key format changes merged in #35531.
Data
thread read through 2026-10-27 (88 statements), head 4ab48b8276
labels: none
Categories
Indexes
P2 · user request
Resolves four long-standing issues dating back to 2017 requesting txindex support on pruned nodes (#10609, #12651, #16182, #35240). By removing the requirement to store the full raw block history on disk, it provides a major user-facing enhancement for resource-constrained node operators.
Membership: src/index/txindex.cpp, src/index/txindex.h, and associated index tests are modified to allow txindex to operate on pruned nodes.
Factors: security/stability 0, bug 0, performance 0, user value 3, leverage 1
RPC / REST / ZMQ
P3 · new feature
RPC methods getrawtransaction and gettxoutproof, along with the REST /tx endpoint, gain structured error reporting containing pruned block hashes. This fills an interface gap, allowing client applications to fetch missing blocks on demand via getblockfrompeer.
Membership: src/rpc/rawtransaction.cpp, src/rpc/txoutproof.cpp, src/rpc/util.cpp, and src/rest.cpp are modified to return structured error information for pruned transactions.
Factors: security/stability 0, bug 0, performance 0, user value 3, leverage 1