{
 "number": 36002,
 "repo": "bitcoin/bitcoin",
 "url": "https://github.com/bitcoin/bitcoin/pull/36002",
 "title": "txindex: allow running in pruned mode",
 "author": "andrewtoth",
 "author_association": "MEMBER",
 "created_at": "2026-08-18T00:01:24Z",
 "updated_at": "2026-09-16T15:13:10Z",
 "age_days": 30,
 "draft": false,
 "labels": [],
 "milestone": null,
 "base": "master",
 "head_sha": "c92f0aae4ce8448df3666584341beec548591d37",
 "head_ref": "txindex-prune",
 "head_repo": "andrewtoth/bitcoin",
 "head_history": [
  {
   "t": "2026-08-18T00:14:02Z",
   "sha": "6abdd0ce00996ab01bca5e5f4cb23e3c13985bcd"
  },
  {
   "t": "2026-09-09T00:17:35Z",
   "sha": "1f1dcd63b38edc55571a6bc2d50f3c19c1f88291"
  },
  {
   "t": "2026-09-10T01:49:05Z",
   "sha": "ee176d310a960b7ec2fd216df6e688c28fad2ac3"
  },
  {
   "t": "2026-09-10T02:41:12Z",
   "sha": "c92f0aae4ce8448df3666584341beec548591d37"
  }
 ],
 "additions": 371,
 "deletions": 86,
 "changed_files": 23,
 "commit_count": 7,
 "size_bucket": "L",
 "mergeable_state": "clean",
 "bot": {
  "drahtbot": {
   "present": true,
   "reviews": {
    "concept_ack": [
     {
      "login": "l0rinc",
      "url": "https://github.com/bitcoin/bitcoin/pull/36002#issuecomment-5321848392"
     },
     {
      "login": "ajtowns",
      "url": "https://github.com/bitcoin/bitcoin/pull/36002#issuecomment-5405875239"
     },
     {
      "login": "arejula27",
      "url": "https://github.com/bitcoin/bitcoin/pull/36002#pullrequestreview-5190953631"
     }
    ]
   },
   "conflicts": [
    {
     "number": 36244,
     "title": "validation, net: Process blocks asynchronously and reduce cs_main contention",
     "author": "w0xlt"
    },
    {
     "number": 35474,
     "title": "node: move index ownership to NodeContext",
     "author": "w0xlt"
    },
    {
     "number": 34729,
     "title": "Reduce log noise",
     "author": "ajtowns"
    },
    {
     "number": 34132,
     "title": "coins, dbwrapper: remove error catcher, make point-read failures fatal",
     "author": "l0rinc"
    }
   ]
  }
 },
 "acks_parsed": {
  "arejula27": {
   "kind": "concept_ack",
   "hash": null,
   "t": "2026-09-13T13:59:40Z",
   "stale": false
  }
 },
 "acks_tally": {
  "ack": 0,
  "stale_ack": 0,
  "concept_ack": 1,
  "approach_ack": 0,
  "nack": 0,
  "concept_nack": 0,
  "approach_nack": 0
 },
 "reviews": {
  "approved": 0,
  "changes_requested": 0,
  "distinct_reviewers": [
   "ajtowns",
   "arejula27",
   "l0rinc",
   "mzumsande"
  ]
 },
 "signals": {
  "needs_rebase": false,
  "ci_failed": false,
  "mergeable_state": "clean",
  "last_author_activity": "2026-09-10T02:41:12Z",
  "last_reviewer_activity": "2026-09-13T13:59:40Z",
  "last_reviewer": "arejula27",
  "author_silent_days": 7,
  "waiting_on_author_days": 4,
  "days_since_update": 1
 },
 "refs": {
  "mentioned": [
   35531,
   36116
  ],
  "depends_on": [
   35531
  ],
  "fixes": [],
  "linked_issues": [],
  "references": [
   {
    "number": 35531,
    "type": "pull",
    "state": "closed",
    "merged": true,
    "merged_at": "2026-08-15",
    "title": "txindex: hash keys and pack positions to reduce disk usage"
   },
   {
    "number": 36116,
    "type": "pull",
    "state": "closed",
    "merged": true,
    "merged_at": "2026-09-09",
    "title": "iwyu: Fix warnings in `src/rpc` and treat them as errors"
   }
  ],
  "conflicts": [
   36244,
   35474,
   34729,
   34132
  ]
 },
 "stack": {
  "shares_commits_with": [],
  "based_on": [],
  "base_for": []
 },
 "review_paths": [],
 "body": "After #35531, each txindex entry now points to the block hash containing the tx instead of the file location of the block. The block hash is then used to find the file position of the block to read the tx from. This allows us to determine the hash of the containing block even if the block is later pruned, and also determine the location of the block on disk if it is later downloaded again via `getblockfrompeer`.\n\nThis PR enables maintaining a txindex while pruned. If a txid lookup tries to read a tx from a pruned block from before the index was started, the hash of the missing block is surfaced to the caller. Otherwise, a warning is returned that the tx might be in a pruned block that was not indexed.\n\nIf desired, a user may call `getblockfrompeer` using the returned hash to download the block, then try again successfully. The downloaded block will be added to the front of the block queue, so won't be pruned again for at least 288 blocks. Of course a larger prune setting is desired if lots of historical blocks will be read, and there are privacy issues if the user is looking up their own transactions. Ideally the blocks can be fetched from a trusted node or from random peers with decoy blocks fetched occasionally.",
 "commits": [
  {
   "sha": "009e5cd256f8d7280630151b17bd9379f14eae75",
   "date": "2026-09-10T01:48:57Z",
   "message": "txindex: allow prune mode when there are no legacy entries"
  },
  {
   "sha": "768169019ddfb49ad5a3f2192f6bd7a51f44e5ca",
   "date": "2026-09-10T02:41:00Z",
   "message": "txindex: return pruned block hashes from FindTx\n\nFindTx now returns a TxLookupResult so callers can distinguish a\nmissing transaction from one that may only be in pruned blocks."
  },
  {
   "sha": "c2573d5c925eaccdae083064b8a7c06728c54b0a",
   "date": "2026-09-10T02:41:00Z",
   "message": "rpc: report pruned blocks from GetTransaction\n\ngetrawtransaction, gettxoutproof, and REST /tx now fail with an error\nnaming the candidate block(s) when a txindex lookup only matches pruned\nblocks. getrawtransaction verbosity 2 omits fee and prevout when undo\ndata is unavailable instead of throwing.\n\nA caller can call getblockfrompeer with the candidate block hashes and try again."
  },
  {
   "sha": "4d9d787a5ee03dd02d7e539452db3e643cd4e4ce",
   "date": "2026-09-10T02:41:00Z",
   "message": "txindex: allow creating index from an already pruned node\n\nAdds an AllowPartialHistory virtual method on BaseIndex, which returns false but the TxIndex override returns true.\n\nWhen true, on creation we set the best block locator to the earliest unpruned block instead of nullptr. We also persist the best block, and use it to skip reading the checkpoint before the first block."
  },
  {
   "sha": "4a15aa2274807f22a270c21cfe7993ad41b1c9b4",
   "date": "2026-09-10T02:41:00Z",
   "message": "rpc: return index starting height in getindexinfo"
  },
  {
   "sha": "efa6cdc6911325a9cdf2adafcc17fe70653be8a5",
   "date": "2026-09-10T02:41:00Z",
   "message": "rpc,rest: return tx miss error text about unknown pruned blocks\n\nReturn extra context in the error message when we miss fetching a tx via the txindex.\nIf we are still syncing, or if we only have a partial index and the tx could be in a block we don't know about."
  },
  {
   "sha": "c92f0aae4ce8448df3666584341beec548591d37",
   "date": "2026-09-10T02:41:00Z",
   "message": "test: cover txindex lookups of pruned blocks\n\nExtend the prune+txindex functional test to check that\ngetrawtransaction, gettxoutproof, and REST /tx return the pruned block hash,\nand that utxoupdatepsbt skips those previous transactions until the\nblock is fetched.\n\nAlso test partial indexes when starting from a pruned node, and the text responses when we don't know the pruned block hash."
  }
 ],
 "timeline": [
  {
   "t": "2026-08-18T00:14:02Z",
   "kind": "force_push",
   "who": "andrewtoth",
   "commit": "6abdd0ce00996ab01bca5e5f4cb23e3c13985bcd"
  },
  {
   "t": "2026-08-18T00:20:45Z",
   "kind": "comment",
   "who": "l0rinc",
   "assoc": "MEMBER",
   "text": "Big concept ACK \\\\:D/"
  },
  {
   "t": "2026-08-25T05:24:37Z",
   "kind": "comment",
   "who": "ajtowns",
   "assoc": "MEMBER",
   "text": "[quoted text omitted]\n\nHmm, I think that means the txindex starts off at ~30GB and grows at ~2.5GB/year indefinitely, so in 40 years' time might be 130GB. I guess that makes sense for nodes wanting to run a local block-explorer type setup; particularly if they keep a significant range of real blocks around; I think expected block usage is about 100GB per year, so 5 years worth of history would be half a terabyte, and probably cover most lookups.\n\nI'd definitely prefer to have a `-prunetxindex=1` option I could set, so that my `prune=2000` (~1 week worth of blocks) node could lookup txs without needing to be reprovisioned with extra disk though. (With such an option enabled, should be feasible to setup a txindex on an already pruned node without redownloading blocks, I think)\n\nConcept ACK, looks okay to me at first glance.\n\nWhat's the (intended) behaviour if you enable txindex on a node that is already pruned? I think it will just hit `FatalErrorf` in `BaseIndex::ProcessBlock` and shut the node down?"
  },
  {
   "t": "2026-08-29T20:03:54Z",
   "kind": "comment",
   "who": "andrewtoth",
   "assoc": "MEMBER",
   "text": "[quoted text omitted]\n\nIt's the same behavior if you try and start `coinstatsindex` or `blockfilterindex` after you've pruned past their last indexed block - the startup error:\n`Error: <index> best block of the index goes beyond pruned data (including undo data).`\n`Please disable the index or reindex (which will download the whole blockchain again)`\n\n[quoted text omitted]\nI think that could be added as a follow-up, and would not conflict with this approach. We could just delete entries by sequence until the last sequence maps to a block we have not yet pruned. We would have to compact the db afterwards though. That way though you will just get a \"tx not found\" response for a historical lookup where the index has also been pruned past, instead of the block hash."
  },
  {
   "t": "2026-09-07T15:04:12Z",
   "kind": "review",
   "who": "mzumsande",
   "assoc": "MEMBER",
   "state": "COMMENTED",
   "commit": "6abdd0ce00996ab01bca5e5f4cb23e3c13985bcd",
   "text": "I wonder if the current design would be more helpful for users - or a genuine partial index that would only report on data of the non-pruned range and clean up old entries every now and then.\n\nThe alternative design would have the downside that it wouldn't report the possible blocks pruned txns might be in. Plus, it would need changes to `baseindex`, so more complicated.\n\nHowever, it would have the big upside that it could be enabled on an already pruned node."
  },
  {
   "t": "2026-09-07T15:28:51Z",
   "kind": "comment",
   "who": "andrewtoth",
   "assoc": "MEMBER",
   "text": "[quoted text omitted]\n\n[quoted text omitted]\nI think we can come up with something that would be interesting for both use cases, but without the complexity if we removed the \"cleanup\" part.\n\nIf we start with an empty index and are pruned, we set the locator to the last pruned block we have and sync. This way we get all the blocks indexed that we have, and future blocks that are pruned we can still return the block hash. When we want to \"cleanup\", we can just delete the txindex folder and restart.\n\n[quoted text omitted]\nIndeed, this has the downside that some existing txs will return not found, and some will return the pruned block hash. So it introduces some ambiguity into the result if you create the index while already pruned. It also does not bound the txindex disk size, but that has already been reduced significantly via #35531."
  },
  {
   "t": "2026-09-09T00:17:35Z",
   "kind": "force_push",
   "who": "andrewtoth",
   "commit": "1f1dcd63b38edc55571a6bc2d50f3c19c1f88291"
  },
  {
   "t": "2026-09-09T00:34:43Z",
   "kind": "comment",
   "who": "andrewtoth",
   "assoc": "MEMBER",
   "text": "Thanks @ajtowns @mzumsande for your thoughts.\n\nI've updated this change to also allow starting an index with an already pruned node. For any tx lookup miss where we don't get a block hash and started indexing after genesis, the response returns context in the error message that the tx may have been in a block that we never indexed.\n\nWe also return the starting block of the index in `getindexinfo`.\n\nRegarding cleaning up entries in the txindex that are pruned, I don't think the complexity of this is worth it. If you are running pruned, you need at least 11 GB, and the txindex grows at about 2.5 GB per year. So every few months you can just nuke your index directory and rebuild. If you're running with a very low prune target, this shouldn't take more than a few minutes max (likely under a minute for `-prune=2000`).\n\nWdyt?"
  },
  {
   "t": "2026-09-10T01:49:05Z",
   "kind": "force_push",
   "who": "andrewtoth",
   "commit": "ee176d310a960b7ec2fd216df6e688c28fad2ac3"
  },
  {
   "t": "2026-09-10T01:50:25Z",
   "kind": "comment",
   "who": "andrewtoth",
   "assoc": "MEMBER",
   "text": "Rebased due to `include` conflicts with #36116."
  },
  {
   "t": "2026-09-10T02:41:12Z",
   "kind": "force_push",
   "who": "andrewtoth",
   "commit": "c92f0aae4ce8448df3666584341beec548591d37"
  },
  {
   "t": "2026-09-13T13:59:40Z",
   "kind": "review",
   "who": "arejula27",
   "assoc": "CONTRIBUTOR",
   "state": "COMMENTED",
   "commit": "c92f0aae4ce8448df3666584341beec548591d37",
   "text": "Concept ACK.\n\nI'm excited about this PR and I love the idea and the approach.\n\nI have to go deeper into the code (just did a first review to see the idea), but meanwhile I have some questions.\n\n`BaseIndex` now has two prune-related flags, `AllowPrune()` and `AllowPartialHistory()`, and they aren't really independent. Partial history is only ever set up on a pruned node, so (as I understand it) an index with `AllowPartialHistory()` but not `AllowPrune()` can't work. Would a single enum value express that relationship better, so the invariant lives in the type instead of a convention that a future index could break? Something like an `AllowPrune` enum with values `Disallowed`, `FullHistory` and `PartialHistory`.\n\nAnother option would be to move `AllowPartialHistory()` to `TxIndex` instead. As far as I can understand no other current index could set it to true anyway. Maybe your idea was for future ones?\n\nFinally, I think one big addition with this is that we can even access txs on pruned nodes, since #35531 the index stores the block \"virtual\" reference rather than a file position , so the node knows exactly which block is missing. From there the caller can fetch that block with `getblockfrompeer` and retry the lookup. Right now those hashes only reach the caller inside the error string, so automating the fetch and retry means parsing the message. Would returning them as structured data be worth it? Or even an optional flag on `getrawtransaction` that requests the block for us in the same rpc call?"
  }
 ],
 "labels_log": [
  {
   "t": "2026-08-18T00:14:35Z",
   "action": "labeled",
   "label": "CI failed",
   "who": "DrahtBot"
  },
  {
   "t": "2026-08-18T01:30:25Z",
   "action": "unlabeled",
   "label": "CI failed",
   "who": "DrahtBot"
  },
  {
   "t": "2026-09-09T16:17:22Z",
   "action": "labeled",
   "label": "Needs rebase",
   "who": "DrahtBot"
  },
  {
   "t": "2026-09-10T02:41:38Z",
   "action": "labeled",
   "label": "CI failed",
   "who": "DrahtBot"
  },
  {
   "t": "2026-09-10T03:20:41Z",
   "action": "unlabeled",
   "label": "Needs rebase",
   "who": "DrahtBot"
  },
  {
   "t": "2026-09-10T03:44:45Z",
   "action": "unlabeled",
   "label": "CI failed",
   "who": "DrahtBot"
  }
 ],
 "state_log": [],
 "text_chars": 8496,
 "text_tokens_estimate": 2124,
 "changed_paths": [
  "src/index/base.cpp",
  "src/index/base.h",
  "src/index/tx_lookup_result.h",
  "src/index/txindex.cpp",
  "src/index/txindex.h",
  "src/init.cpp",
  "src/node/transaction.cpp",
  "src/node/transaction.h",
  "src/qt/bitcoinstrings.cpp",
  "src/rest.cpp",
  "src/rpc/node.cpp",
  "src/rpc/rawtransaction.cpp",
  "src/rpc/txoutproof.cpp",
  "src/rpc/util.cpp",
  "src/rpc/util.h",
  "src/test/txindex_tests.cpp",
  "test/functional/feature_assumeutxo.py",
  "test/functional/feature_index_prune.py",
  "test/functional/feature_pruning.py",
  "test/functional/feature_txindex_prune.py",
  "test/functional/interface_rest.py",
  "test/functional/rpc_misc.py",
  "test/functional/test_runner.py"
 ],
 "files": [
  {
   "path": "src/index/base.cpp",
   "add": 17,
   "del": 1
  },
  {
   "path": "src/index/base.h",
   "add": 9,
   "del": 0
  },
  {
   "path": "src/index/tx_lookup_result.h",
   "add": 31,
   "del": 0
  },
  {
   "path": "src/index/txindex.cpp",
   "add": 19,
   "del": 12
  },
  {
   "path": "src/index/txindex.h",
   "add": 10,
   "del": 13
  },
  {
   "path": "src/init.cpp",
   "add": 10,
   "del": 3
  },
  {
   "path": "src/node/transaction.cpp",
   "add": 11,
   "del": 10
  },
  {
   "path": "src/node/transaction.h",
   "add": 4,
   "del": 3
  },
  {
   "path": "src/qt/bitcoinstrings.cpp",
   "add": 1,
   "del": 1
  },
  {
   "path": "src/rest.cpp",
   "add": 11,
   "del": 4
  },
  {
   "path": "src/rpc/node.cpp",
   "add": 2,
   "del": 0
  },
  {
   "path": "src/rpc/rawtransaction.cpp",
   "add": 15,
   "del": 19
  },
  {
   "path": "src/rpc/txoutproof.cpp",
   "add": 11,
   "del": 5
  },
  {
   "path": "src/rpc/util.cpp",
   "add": 17,
   "del": 0
  },
  {
   "path": "src/rpc/util.h",
   "add": 8,
   "del": 0
  },
  {
   "path": "src/test/txindex_tests.cpp",
   "add": 39,
   "del": 6
  },
  {
   "path": "test/functional/feature_assumeutxo.py",
   "add": 1,
   "del": 1
  },
  {
   "path": "test/functional/feature_index_prune.py",
   "add": 2,
   "del": 2
  },
  {
   "path": "test/functional/feature_pruning.py",
   "add": 0,
   "del": 4
  },
  {
   "path": "test/functional/feature_txindex_prune.py",
   "add": 150,
   "del": 0
  },
  {
   "path": "test/functional/interface_rest.py",
   "add": 1,
   "del": 1
  },
  {
   "path": "test/functional/rpc_misc.py",
   "add": 1,
   "del": 1
  },
  {
   "path": "test/functional/test_runner.py",
   "add": 1,
   "del": 0
  }
 ],
 "test_lines": 210,
 "git": {
  "head": "c92f0aae4ce8448df3666584341beec548591d37",
  "head_matches_backup": true,
  "base": "7e75b3b922f9ba740240c2204e3e9389524eafdb",
  "commits": [
   {
    "sha": "009e5cd256",
    "subject": "txindex: allow prune mode when there are no legacy entries",
    "files": 6,
    "add": 14,
    "del": 7
   },
   {
    "sha": "768169019d",
    "subject": "txindex: return pruned block hashes from FindTx",
    "files": 6,
    "add": 98,
    "del": 36
   },
   {
    "sha": "c2573d5c92",
    "subject": "rpc: report pruned blocks from GetTransaction",
    "files": 7,
    "add": 53,
    "del": 34
   },
   {
    "sha": "4d9d787a5e",
    "subject": "txindex: allow creating index from an already pruned node",
    "files": 5,
    "add": 34,
    "del": 6
   },
   {
    "sha": "4a15aa2274",
    "subject": "rpc: return index starting height in getindexinfo",
    "files": 5,
    "add": 7,
    "del": 5
   },
   {
    "sha": "efa6cdc691",
    "subject": "rpc,rest: return tx miss error text about unknown pruned blocks",
    "files": 5,
    "add": 19,
    "del": 3
   },
   {
    "sha": "c92f0aae4c",
    "subject": "test: cover txindex lookups of pruned blocks",
    "files": 2,
    "add": 151,
    "del": 0
   }
  ],
  "patch_truncated": true
 },
 "input_hash": "945e4820b72908a6",
 "extracted_at": "2026-09-17T16:15:31+00:00"
}