{
 "number": 36480,
 "repo": "bitcoin/bitcoin",
 "url": "https://github.com/bitcoin/bitcoin/pull/36480",
 "title": "txospenderindex: hash keys and pack positions to reduce disk usage",
 "author": "arejula27",
 "author_association": "CONTRIBUTOR",
 "created_at": "2026-10-08T20:46:11Z",
 "updated_at": "2026-10-31T08:11:11Z",
 "description_hash": "df3f24bf1389c1e8",
 "age_days": 24,
 "draft": true,
 "labels": [
  "CI failed"
 ],
 "milestone": null,
 "base": "master",
 "head_sha": "fadf210822d84dcf278c6d6897c67e189a3c0065",
 "head_ref": "txospenderindex-block-keys",
 "head_repo": "arejula27/bitcoin",
 "head_history": [],
 "additions": 789,
 "deletions": 264,
 "changed_files": 15,
 "commit_count": 6,
 "size_bucket": "XL",
 "mergeable_state": "unstable",
 "bot": {
  "drahtbot": {
   "present": true,
   "reviews": {
    "concept_ack": [
     {
      "login": "andrewtoth",
      "url": "https://github.com/bitcoin/bitcoin/pull/36480#issuecomment-6068799412"
     },
     {
      "login": "l0rinc",
      "url": "https://github.com/bitcoin/bitcoin/pull/36480#issuecomment-6075248071"
     },
     {
      "login": "w0xlt",
      "url": "https://github.com/bitcoin/bitcoin/pull/36480#issuecomment-6185688251"
     },
     {
      "login": "sedited",
      "url": "https://github.com/bitcoin/bitcoin/pull/36480#issuecomment-6370740796"
     }
    ]
   },
   "conflicts": []
  }
 },
 "acks_parsed": {
  "andrewtoth": {
   "kind": "concept_ack",
   "hash": null,
   "t": "2026-10-08T20:50:14Z",
   "stale": false
  },
  "l0rinc": {
   "kind": "concept_ack",
   "hash": null,
   "t": "2026-10-09T05:58:38Z",
   "stale": false
  },
  "w0xlt": {
   "kind": "concept_ack",
   "hash": null,
   "t": "2026-10-15T06:07:50Z",
   "stale": false
  },
  "sedited": {
   "kind": "concept_ack",
   "hash": null,
   "t": "2026-10-23T17:25:24Z",
   "stale": false
  }
 },
 "acks_tally": {
  "ack": 0,
  "stale_ack": 0,
  "concept_ack": 4,
  "approach_ack": 0,
  "nack": 0,
  "concept_nack": 0,
  "approach_nack": 0
 },
 "reviews": {
  "approved": 0,
  "changes_requested": 0,
  "distinct_reviewers": [
   "andrewtoth",
   "l0rinc",
   "sedited",
   "w0xlt"
  ]
 },
 "signals": {
  "needs_rebase": false,
  "ci_failed": true,
  "mergeable_state": "unstable",
  "last_author_activity": "2026-10-08T22:53:54Z",
  "last_reviewer_activity": "2026-10-23T17:25:24Z",
  "last_reviewer": "sedited",
  "author_silent_days": 24,
  "waiting_on_author_days": 9,
  "days_since_update": 1
 },
 "refs": {
  "mentioned": [
   35531,
   36002
  ],
  "depends_on": [],
  "fixes": [],
  "linked_issues": [],
  "references": [
   {
    "number": 35531,
    "type": "pull",
    "state": "closed",
    "merged": true,
    "merged_at": "2026-08-15",
    "title": "txindex: hash keys and pack positions to reduce disk usage"
   },
   {
    "number": 36002,
    "type": "pull",
    "state": "open",
    "merged": false,
    "merged_at": null,
    "title": "txindex: allow running in pruned mode"
   }
  ],
  "conflicts": []
 },
 "stack": {
  "shares_commits_with": [],
  "based_on": [],
  "base_for": []
 },
 "review_paths": [],
 "body": "This applies the txindex key scheme from #35531 to `txospenderindex`. The motivation is to save space and to allow replicating what #36002 does for `txindex`, by referencing blocks instead of local files.\n\nThe first commits refactor the index code so the #35531 scheme can be shared: `BaseIndex::DB` can opt in to the versioned sync locator, and the block sequence keys, sequence assignment and salted hasher move from txindex into `index/block_seq.h`, without changing txindex's on-disk format. The following commits switch txospenderindex to the new keys and add tests. As in txindex, entries of disconnected blocks are kept and lookups only count blocks in the active chain, so `CustomRemove` is gone. Entries written by v31 are still read as a fallback. The last commit makes `FindSpender` skip candidates it cannot read and return `std::optional`, mimicking `TxIndex::FindTx`, since with 5-byte prefixes an unreadable false positive could otherwise hide the real spender.\n\nEach entry is about 33% smaller (key bytes):\n\n- before: 's' (1) + SipHash-2-4 of the outpoint (8) + CDiskTxPos, as varints nFile (2) + nPos (4) + nTxOffset (3) = 18 bytes\n- after: 'x' (1) + SipHash-1-3-UJ prefix (5) + block sequence varint (3) + tx offset (3) = 12 bytes\n\nI did not have time to do a benchmark (my setup is really cheap, laptop where i work with an external SSD connected), will try to do on the next days. But same as #35531 i hope to reduce disk space and indexing time.",
 "commits": [
  {
   "sha": "53566bf2961dd6c592e4a6f6eb63f5e9c3e7364e",
   "date": "2026-10-08T19:47:33Z",
   "message": "index: let indexes opt in to a versioned sync locator\n\nMove the best_block_v2 locator handling from TxIndex::DB into\nBaseIndex::DB behind a constructor flag, so other indexes can reuse\nit when they change their on-disk format."
  },
  {
   "sha": "f3b1bef79290bd165ea5f3b5c84b028c19b07567",
   "date": "2026-10-08T19:47:33Z",
   "message": "refactor: move block sequence keys to index/block_seq.h\n\nMake the key prefix a template parameter so other indexes can reuse\nthem.\n\nNot in db_key.h, which holds the height-based scheme of\nblockfilterindex and coinstatsindex."
  },
  {
   "sha": "e2a3182e457963924e7323e0c586d218b1145d1b",
   "date": "2026-10-08T19:47:33Z",
   "message": "refactor: move block sequence assignment and hasher salt to block_seq\n\nMove the logic that assigns a sequence number to a newly indexed block\nand the creation of the salted hasher out of TxIndex::DB into free\nfunctions in block_seq, so other indexes using the same scheme can\nshare them."
  },
  {
   "sha": "007650b331130996feb45295f342c2f2b3758358",
   "date": "2026-10-08T20:02:43Z",
   "message": "txospenderindex: key entries by block sequence instead of disk position\n\nUse the txindex key scheme: a 5-byte salted hash of the spent outpoint,\nfollowed by the block sequence and the offset of the spending tx.\n\nAs in txindex, entries of disconnected blocks are kept, so CustomRemove\ngoes away and FindSpender only returns spends in the active chain.\n\nFall back to the legacy entries if no spend is found."
  },
  {
   "sha": "21772d62ec8dec7fabf0583eb365be794b3a8ace",
   "date": "2026-10-08T20:02:43Z",
   "message": "test: cover txospenderindex hash collisions, reorgs and legacy entries\n\nPin the encodings of the new and legacy keys, and add unit tests for\nhash prefix false positives, for entries of disconnected blocks being\nkept but not returned, and for the fallback to legacy entries. Add a\nfunctional test that upgrades a v31 datadir, reorgs a block indexed in\nthe legacy format and downgrades again."
  },
  {
   "sha": "fadf210822d84dcf278c6d6897c67e189a3c0065",
   "date": "2026-10-08T20:30:02Z",
   "message": "txospenderindex: skip unreadable candidates in FindSpender\n\nMimic TxIndex::FindTx: skip a candidate that cannot be read and return\nstd::optional. With 5-byte hash prefixes collisions are no longer\nnegligible, and failing on an unreadable false positive would hide the\nreal spender."
  }
 ],
 "timeline": [
  {
   "t": "2026-10-08T20:50:14Z",
   "kind": "comment",
   "who": "andrewtoth",
   "assoc": "MEMBER",
   "text": "Concept ACK!\n\n[quoted text omitted]\nI have done this before. IIRC the db size went from ~84GB -> ~62GB. The time to index is also much lower.\n\nDo we want to also enable pruning in one go like we are with #36002? I don't think it would be controversial so we could do it all in 1 PR.",
   "id": "c:6068799412",
   "url": "https://github.com/bitcoin/bitcoin/pull/36480#issuecomment-6068799412",
   "hash": "862c5d2b996c7474",
   "edited": null
  },
  {
   "t": "2026-10-08T20:56:00Z",
   "kind": "comment",
   "who": "arejula27",
   "assoc": "CONTRIBUTOR",
   "text": "[quoted text omitted]\n\nSorry @andrewtoth, i wasnt aware of it :sweat_smile: , should i compare with ur code and add u as co author?\n[quoted text omitted]\n\nsounds awesome!\n\n[quoted text omitted]\nSure! i will add the commits, but it will take me some time o7",
   "id": "c:6068891650",
   "url": "https://github.com/bitcoin/bitcoin/pull/36480#issuecomment-6068891650",
   "hash": "d48abf251bb4496f",
   "edited": "2026-10-08T21:04:06Z"
  },
  {
   "t": "2026-10-08T22:15:21Z",
   "kind": "comment",
   "who": "andrewtoth",
   "assoc": "MEMBER",
   "text": "[quoted text omitted]\n\nNo worries, it was nothing I published. Let's just keep going with this. I don't need to be added as co-author.\n\n[quoted text omitted]\nOn second thought, if we merge #36002 first we can reuse the TxLookupResult (sorry I have to push an update there)",
   "id": "c:6070112631",
   "url": "https://github.com/bitcoin/bitcoin/pull/36480#issuecomment-6070112631",
   "hash": "2f35677df15e34ef",
   "edited": null
  },
  {
   "t": "2026-10-08T22:53:54Z",
   "kind": "comment",
   "who": "arejula27",
   "assoc": "CONTRIBUTOR",
   "text": "Sounds good to me, i will wait an then add here `TxLookupResult `!",
   "id": "c:6070655289",
   "url": "https://github.com/bitcoin/bitcoin/pull/36480#issuecomment-6070655289",
   "hash": "c480925d6a4a3156",
   "edited": null
  },
  {
   "t": "2026-10-09T05:58:38Z",
   "kind": "comment",
   "who": "l0rinc",
   "assoc": "MEMBER",
   "text": "Concept ACK\n\ud83d\udc4d for pruning support and also for merging https://github.com/bitcoin/bitcoin/pull/36002 first - if you agree, please convert this to draft until then so reviewers know which PRs to prioritize.",
   "id": "c:6075248071",
   "url": "https://github.com/bitcoin/bitcoin/pull/36480#issuecomment-6075248071",
   "hash": "58a12bb2eb195090",
   "edited": null
  },
  {
   "t": "2026-10-15T06:07:50Z",
   "kind": "comment",
   "who": "w0xlt",
   "assoc": "CONTRIBUTOR",
   "text": "Concept ACK",
   "id": "c:6185688251",
   "url": "https://github.com/bitcoin/bitcoin/pull/36480#issuecomment-6185688251",
   "hash": "e8334cc43879b73c",
   "edited": null
  },
  {
   "t": "2026-10-23T17:25:24Z",
   "kind": "comment",
   "who": "sedited",
   "assoc": "MEMBER",
   "text": "Concept ACK",
   "id": "c:6370740796",
   "url": "https://github.com/bitcoin/bitcoin/pull/36480#issuecomment-6370740796",
   "hash": "e8334cc43879b73c",
   "edited": null
  }
 ],
 "labels_log": [
  {
   "t": "2026-10-08T22:15:12Z",
   "action": "labeled",
   "label": "CI failed",
   "who": "DrahtBot"
  }
 ],
 "state_log": [
  {
   "t": "2026-10-09T07:31:57Z",
   "kind": "convert_to_draft",
   "who": "arejula27"
  }
 ],
 "text_chars": 4461,
 "text_tokens_estimate": 1115,
 "changed_paths": [
  "src/CMakeLists.txt",
  "src/index/base.cpp",
  "src/index/base.h",
  "src/index/block_seq.cpp",
  "src/index/block_seq.h",
  "src/index/txindex.cpp",
  "src/index/txindex_key.h",
  "src/index/txospenderindex.cpp",
  "src/index/txospenderindex.h",
  "src/index/txospenderindex_key.h",
  "src/rpc/mempool.cpp",
  "src/test/txospenderindex_tests.cpp",
  "test/functional/feature_txospenderindex_compatibility.py",
  "test/functional/rpc_gettxspendingprevout.py",
  "test/functional/test_runner.py"
 ],
 "files": [
  {
   "path": "src/CMakeLists.txt",
   "add": 1,
   "del": 0
  },
  {
   "path": "src/index/base.cpp",
   "add": 14,
   "del": 3
  },
  {
   "path": "src/index/base.h",
   "add": 10,
   "del": 3
  },
  {
   "path": "src/index/block_seq.cpp",
   "add": 26,
   "del": 0
  },
  {
   "path": "src/index/block_seq.h",
   "add": 121,
   "del": 0
  },
  {
   "path": "src/index/txindex.cpp",
   "add": 8,
   "del": 43
  },
  {
   "path": "src/index/txindex_key.h",
   "add": 9,
   "del": 78
  },
  {
   "path": "src/index/txospenderindex.cpp",
   "add": 109,
   "del": 105
  },
  {
   "path": "src/index/txospenderindex.h",
   "add": 23,
   "del": 16
  },
  {
   "path": "src/index/txospenderindex_key.h",
   "add": 78,
   "del": 0
  },
  {
   "path": "src/rpc/mempool.cpp",
   "add": 3,
   "del": 9
  },
  {
   "path": "src/test/txospenderindex_tests.cpp",
   "add": 283,
   "del": 5
  },
  {
   "path": "test/functional/feature_txospenderindex_compatibility.py",
   "add": 100,
   "del": 0
  },
  {
   "path": "test/functional/rpc_gettxspendingprevout.py",
   "add": 3,
   "del": 2
  },
  {
   "path": "test/functional/test_runner.py",
   "add": 1,
   "del": 0
  }
 ],
 "test_lines": 394,
 "git": {
  "head": "fadf210822d84dcf278c6d6897c67e189a3c0065",
  "head_matches_backup": true,
  "base": "f739c7876b81304bdc72e7eaa307702da269ac31",
  "patch_id": "a7a27d5d4a0b7a20",
  "commits": [
   {
    "sha": "53566bf296",
    "subject": "index: let indexes opt in to a versioned sync locator",
    "files": 4,
    "add": 26,
    "del": 26
   },
   {
    "sha": "f3b1bef792",
    "subject": "refactor: move block sequence keys to index/block_seq.h",
    "files": 2,
    "add": 112,
    "del": 77
   },
   {
    "sha": "e2a3182e45",
    "subject": "refactor: move block sequence assignment and hasher salt to block_seq",
    "files": 5,
    "add": 52,
    "del": 25
   },
   {
    "sha": "007650b331",
    "subject": "txospenderindex: key entries by block sequence instead of disk position",
    "files": 4,
    "add": 207,
    "del": 117
   },
   {
    "sha": "21772d62ec",
    "subject": "test: cover txospenderindex hash collisions, reorgs and legacy entries",
    "files": 4,
    "add": 356,
    "del": 0
   },
   {
    "sha": "fadf210822",
    "subject": "txospenderindex: skip unreadable candidates in FindSpender",
    "files": 4,
    "add": 69,
    "del": 52
   }
  ],
  "patch_truncated": true
 },
 "patch_id": "a7a27d5d4a0b7a20",
 "input_hash": "b854f0cc2ea0a9fb",
 "extracted_at": "2026-11-02T01:09:49+00:00"
}