{ "number": 35827, "repo": "bitcoin/bitcoin", "url": "https://github.com/bitcoin/bitcoin/pull/35827", "title": "bench: add index benchmarks", "author": "arejula27", "author_association": "FIRST_TIME_CONTRIBUTOR", "created_at": "2026-07-27T19:53:40Z", "updated_at": "2026-09-06T16:39:46Z", "age_days": 51, "draft": false, "labels": [ "Tests" ], "milestone": null, "base": "master", "head_sha": "02051528670118ff4f71f3c2a5570ab006614ab0", "head_ref": "index-sync-bench-infra", "head_repo": "arejula27/bitcoin", "head_history": [ { "t": "2026-07-27T20:24:04Z", "sha": "4f79b7275ac78cf88341a83167067c4343270a8d" }, { "t": "2026-07-30T20:18:33Z", "sha": "d46c4b672e43cc8afc8cfab1cd8d0d7ef8204101" }, { "t": "2026-09-06T16:23:37Z", "sha": "77f89c3ad501a1f6d5378f4e63347c41b587692f" }, { "t": "2026-09-06T16:39:46Z", "sha": "02051528670118ff4f71f3c2a5570ab006614ab0" } ], "additions": 452, "deletions": 0, "changed_files": 7, "commit_count": 3, "size_bucket": "L", "mergeable_state": "unstable", "bot": { "drahtbot": { "present": true, "reviews": { "concept_ack": [ { "login": "l0rinc", "url": "https://github.com/bitcoin/bitcoin/pull/35827#pullrequestreview-4791093422" } ] }, "conflicts": [] } }, "acks_parsed": { "l0rinc": { "kind": "concept_ack", "hash": null, "t": "2026-07-27T20:17:13Z", "stale": false } }, "acks_tally": { "ack": 0, "stale_ack": 0, "concept_ack": 1, "approach_ack": 0, "nack": 0, "concept_nack": 0, "approach_nack": 0 }, "reviews": { "approved": 0, "changes_requested": 0, "distinct_reviewers": [ "l0rinc" ] }, "signals": { "needs_rebase": false, "ci_failed": false, "mergeable_state": "unstable", "last_author_activity": "2026-09-06T16:39:46Z", "last_reviewer_activity": "2026-07-28T17:19:22Z", "last_reviewer": "l0rinc", "author_silent_days": 10, "waiting_on_author_days": 0, "days_since_update": 10 }, "refs": { "mentioned": [ 34489, 35531 ], "depends_on": [], "fixes": [], "linked_issues": [], "references": [ { "number": 35531, "type": "pull", "state": "closed", "merged": true, "merged_at": "2026-08-15", "title": "txindex: hash keys and pack positions to reduce disk usage" }, { "number": 34489, "type": "pull", "state": "open", "merged": false, "merged_at": null, "title": "index: batch db writes during initial sync" } ], "conflicts": [] }, "stack": { "shares_commits_with": [], "based_on": [], "base_for": [] }, "review_paths": [ "src/bench/index_blockfilter.cpp", "src/bench/index_sync_util.cpp", "src/bench/index_sync_util.h", "src/bench/index_txindex.cpp", "src/bench/index_txospender.cpp" ], "body": "There is a fair amount of work going into the indices at the moment, and no cheap way to tell whether a change actually improves what it set out to improve: checking end to end means a full IBD. The only index benchmark in the tree, `BlockFilterIndexSync`, syncs coinbase-only blocks, so it cannot show anything that scales with what a block contains, and there is none at all for `TxIndex` or `TxoSpenderIndex`.\n\nThis adds shared helpers in `src/bench/index_sync_util.{h,cpp}` that build a chain of blocks with chained, validly-signed transactions (~2 inputs and ~2 outputs each, every output paying a distinct key), and on top of them, for txindex, txospenderindex and blockfilterindex:\n\n- `SyncDisk` / `SyncMem` \u2014 full sync with the index database on the filesystem, and in memory.\n- `Lookup` \u2014 one query per indexed key (`FindTx`, `FindSpender`).\n\nHow far apart Disk and Mem land depends on the data directory, which defaults to a tmpfs on many Linux systems. Pointing `-testdatadir` at real storage is what makes the difference the cost of the write path:\n\n| Benchmark | tmpfs (default) | ext4 via `-testdatadir` |\n| ----------------------------------- | --------------: | ----------------------: |\n| `TxIndexSyncDisk` | 7.0 ms | 18.1 ms |\n| `TxIndexSyncMem` | 6.5 ms | 6.8 ms |\n| `TxIndexLookup` | 13.4 us | 13.7 us |\n| `TxoSpenderIndexSyncDisk` | 8.3 ms | 23.8 ms |\n| `TxoSpenderIndexSyncMem` | 7.8 ms | 8.1 ms |\n| `TxoSpenderIndexLookup` | 13.8 us | 14.1 us |\n| `BlockFilterIndexSyncRealisticDisk` | 14.7 ms | 31.0 ms |\n| `BlockFilterIndexSyncRealisticMem` | 13.8 ms | 17.8 ms |\n\nThe existing coinbase-only `BlockFilterIndexSync` is left untouched. The whole suite runs in well under a minute.\n\nI have used this on those two pull requests, and in both the interesting number was one the existing benchmark could not produce: #35531 (5-byte siphash keys) trades a 49% smaller index on disk for a lookup regression that grows with index size, and #34489 (batch db writes) cuts `write` syscalls by 91% with CPU unchanged. The numbers are in those threads.\n\nDisclosure: I used claude to help me in some parts, however all code has been verified by me and I ran the bench manually on the scenarios mentioned", "commits": [ { "sha": "5ef138bd0129886e9b80c236de55b7b457a33cf9", "date": "2026-09-06T16:36:26Z", "message": "bench: add reusable index sync benchmark helpers\n\nThe existing BlockFilterIndexSync syncs coinbase-only blocks, which\ncannot exercise anything that scales with the number of inputs or\noutputs per block.\n\nAdd src/bench/index_sync_util.{h,cpp} with the chain setup and timing\nloops shared by index benchmarks. ExtendChainWithSpends() builds blocks\nof validly-signed transactions with ~2 inputs each (the average ratio\nobserved on mainnet), paying every output to a distinct key: block\nfilter elements are deduplicated, so reusing a couple of scripts would\nkeep every filter tiny no matter how many transactions the block holds.\n\nIt also flushes the chainstate once the chain is built. BaseIndex::Commit\nreturns early unless the index tip is an ancestor of the last flushed\nblock, and a test setup never flushes on its own, so without this the\nbenchmarks would write the index entries but never the best-block\nlocator, leaving CustomCommit() outside the measurement." }, { "sha": "d02a3de47b77e4f103514f15969f81801339bbe5", "date": "2026-09-06T16:36:47Z", "message": "bench: add TxoSpenderIndex, TxIndex and BlockFilterIndex sync benchmarks\n\nAdd, on top of the shared helpers:\n- TxoSpenderIndexSync{Disk,Mem} and TxoSpenderIndexLookup (FindSpender)\n- TxIndexSync{Disk,Mem} and TxIndexLookup (FindTx)\n- BlockFilterIndexSyncRealistic{Disk,Mem}\n\nThe Disk variant exercises the file-backed write path, the Mem variant\nkeeps the database in memory. Reading the pair as the cost of that path\nrequires the data directory to be on real storage: it defaults to\nfs::temp_directory_path(), which on many Linux systems is a tmpfs, and\nthere both variants write to RAM and report the same number. Pointing\n-testdatadir at a real filesystem is what makes the Disk figures mean\nwhat their name says.\n\nThe pre-existing coinbase-only BlockFilterIndexSync is left untouched.\nIt scales with the number of blocks over near-empty filters, where the\nnew one scales with what a block contains, so the two are kept as\nseparate axes." }, { "sha": "02051528670118ff4f71f3c2a5570ab006614ab0", "date": "2026-09-06T16:36:47Z", "message": "doc: describe measuring I/O and memory cost of benchmarks\n\nThe framework reports wall-clock time and, where available, cycles and\ninstructions, but not syscalls, bytes written, on-disk size or peak\nmemory. Document how to measure those around bench_bitcoin with strace,\n/usr/bin/time and du, and the -testdatadir knob needed to avoid tmpfs." } ], "timeline": [ { "t": "2026-07-27T20:04:50Z", "kind": "comment", "who": "arejula27", "assoc": "NONE", "text": "I think this PR can be splited into two, one for the doc and another for the index benchmarks" }, { "t": "2026-07-27T20:10:45Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/bench/index_blockfilter.cpp", "commit": "4f79b7275ac78cf88341a83167067c4343270a8d", "in_reply_to": null, "text": "nit: you can use `1_MiB`" }, { "t": "2026-07-27T20:11:29Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/bench/index_sync_util.cpp", "commit": "4f79b7275ac78cf88341a83167067c4343270a8d", "in_reply_to": null, "text": "I don't usually mind long lines but formatting is a bit off on GitHub" }, { "t": "2026-07-27T20:12:37Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/bench/index_sync_util.h", "commit": "4f79b7275ac78cf88341a83167067c4343270a8d", "in_reply_to": null, "text": "we should avoid adding new arch dependent types if possible" }, { "t": "2026-07-27T20:13:13Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/bench/index_sync_util.h", "commit": "4f79b7275ac78cf88341a83167067c4343270a8d", "in_reply_to": null, "text": "locking inside the measured block will likely distort the results. Consider precalculating and maybe using setup block" }, { "t": "2026-07-27T20:14:12Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/bench/index_sync_util.h", "commit": "4f79b7275ac78cf88341a83167067c4343270a8d", "in_reply_to": null, "text": "either assert or doNotOptimizeAway, no need for both. Alternatively have a bool at the begining, and each value and assert at the end" }, { "t": "2026-07-27T20:15:02Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/bench/index_txindex.cpp", "commit": "4f79b7275ac78cf88341a83167067c4343270a8d", "in_reply_to": null, "text": "I'd prefer having all of these at the very end instead" }, { "t": "2026-07-27T20:15:42Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/bench/index_txindex.cpp", "commit": "02051528670118ff4f71f3c2a5570ab006614ab0", "in_reply_to": null, "text": "these are likely small, so even if they're not in memory, the OS will likely make sure they're still actually in memory..." }, { "t": "2026-07-27T20:16:16Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/bench/index_txospender.cpp", "commit": "02051528670118ff4f71f3c2a5570ab006614ab0", "in_reply_to": null, "text": "aren't these two conditions exactly the same?" }, { "t": "2026-07-27T20:17:13Z", "kind": "review", "who": "l0rinc", "assoc": "MEMBER", "state": "COMMENTED", "commit": "6bf3b13e535f51ab13cebd4b616970e7d438d30e", "text": "Concept ACK\n\nThanks for taking care of these, I will review them later in more detail, just quickly glanced at the structure." }, { "t": "2026-07-27T20:24:04Z", "kind": "force_push", "who": "arejula27", "commit": "4f79b7275ac78cf88341a83167067c4343270a8d" }, { "t": "2026-07-28T17:04:53Z", "kind": "review_comment", "who": "arejula27", "assoc": "NONE", "path": "src/bench/index_txindex.cpp", "commit": "02051528670118ff4f71f3c2a5570ab006614ab0", "in_reply_to": 3660596643, "text": "Thats a problem i am having with this PR, based on my results looks like Im not using the disk, not sure if it is only the size or am i doing something wrong elsewhere" }, { "t": "2026-07-28T17:11:34Z", "kind": "review_comment", "who": "arejula27", "assoc": "NONE", "path": "src/bench/index_sync_util.h", "commit": "4f79b7275ac78cf88341a83167067c4343270a8d", "in_reply_to": 3660577747, "text": "mb" }, { "t": "2026-07-28T17:19:22Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/bench/index_txindex.cpp", "commit": "02051528670118ff4f71f3c2a5570ab006614ab0", "in_reply_to": 3660596643, "text": "You can try invalidating the OS file caches" }, { "t": "2026-07-28T17:34:22Z", "kind": "review_comment", "who": "arejula27", "assoc": "NONE", "path": "src/bench/index_txospender.cpp", "commit": "02051528670118ff4f71f3c2a5570ab006614ab0", "in_reply_to": 3660599957, "text": "```\n /**\n * Search the index for a transaction that spends the given outpoint.\n *\n * @param[in] txo The outpoint to search for.\n *\n * @return std::nullopt if the outpoint has not been spent on-chain.\n * std::optional{TxoSpender} if the output has been spent on-chain. Contains the spending transaction\n * and the block it was confirmed in.\n * util::Unexpected{error} if something unexpected happened (i.e. disk or deserialization error).\n */\n util::Expected, std::string> FindSpender(const COutPoint& txo) const;\n```\n\n[`FindSpender`](https://github.com/bitcoin/bitcoin/blob/7dea464d6b51a69bd99a0451be8aaf3a26313eb6/src/index/txospenderindex.h#L68) returns `util::Expected, std::string>`, which might be complex read. My understanding:\n- The first check is \"no read error\".\n- The second check is \"a spender was found\".\n\nHow do you understand it?" }, { "t": "2026-07-28T17:57:59Z", "kind": "review_comment", "who": "arejula27", "assoc": "NONE", "path": "src/bench/index_txindex.cpp", "commit": "02051528670118ff4f71f3c2a5570ab006614ab0", "in_reply_to": 3660596643, "text": "I didn't find a way to do it from inside the benchmark itself, only things the user can do outside the process, but I don't know this area well enough. Do you know a simpler way?\n\nAlso I noticed the writes are not `fsynced` (`fSync=false` in `BaseIndex::Commit`) and the index is smaller than the write buffer, so it never leaves the memtable. I'll try with a bigger chain and see what happens" }, { "t": "2026-07-30T20:18:33Z", "kind": "force_push", "who": "arejula27", "commit": "d46c4b672e43cc8afc8cfab1cd8d0d7ef8204101" }, { "t": "2026-09-06T16:23:37Z", "kind": "force_push", "who": "arejula27", "commit": "77f89c3ad501a1f6d5378f4e63347c41b587692f" }, { "t": "2026-09-06T16:25:51Z", "kind": "comment", "who": "arejula27", "assoc": "NONE", "text": "Invalidating the OS caches wasn't the real issue but the data directory defaults to a tmpfs. So `f_memory=false` was writing to RAM either way. With `-testdatadir` on real storage the Disk/Mem split shows up on its own, around 2.7x for txindex. The helper documents that now, since the default hides it.\n\nDigging into that I also found the benchmarks never reached `BaseIndex::Commit()`, a test setup never flushes the chainstate, so its guard always returned early and the commit path stayed out of the measurement. `ExtendChainWithSpends()` forces a flush now.\n\nThe rest of your comments are applied, thank you for the review @l0rinc" }, { "t": "2026-09-06T16:39:46Z", "kind": "force_push", "who": "arejula27", "commit": "02051528670118ff4f71f3c2a5570ab006614ab0" } ], "labels_log": [ { "t": "2026-07-27T19:53:44Z", "action": "labeled", "label": "Tests", "who": "DrahtBot" }, { "t": "2026-07-29T12:35:04Z", "action": "labeled", "label": "CI failed", "who": "DrahtBot" }, { "t": "2026-07-30T19:47:33Z", "action": "unlabeled", "label": "CI failed", "who": "maflcko" }, { "t": "2026-07-30T19:51:57Z", "action": "labeled", "label": "CI failed", "who": "DrahtBot" }, { "t": "2026-08-17T11:08:06Z", "action": "unlabeled", "label": "CI failed", "who": "DrahtBot" } ], "state_log": [], "text_chars": 7933, "text_tokens_estimate": 1983, "changed_paths": [ "doc/benchmarking.md", "src/bench/CMakeLists.txt", "src/bench/index_blockfilter.cpp", "src/bench/index_sync_util.cpp", "src/bench/index_sync_util.h", "src/bench/index_txindex.cpp", "src/bench/index_txospender.cpp" ], "files": [ { "path": "doc/benchmarking.md", "add": 38, "del": 0 }, { "path": "src/bench/CMakeLists.txt", "add": 3, "del": 0 }, { "path": "src/bench/index_blockfilter.cpp", "add": 33, "del": 0 }, { "path": "src/bench/index_sync_util.cpp", "add": 146, "del": 0 }, { "path": "src/bench/index_sync_util.h", "add": 98, "del": 0 }, { "path": "src/bench/index_txindex.cpp", "add": 66, "del": 0 }, { "path": "src/bench/index_txospender.cpp", "add": 68, "del": 0 } ], "test_lines": 414, "git": { "head": "02051528670118ff4f71f3c2a5570ab006614ab0", "head_matches_backup": true, "base": "7dea464d6b51a69bd99a0451be8aaf3a26313eb6", "commits": [ { "sha": "5ef138bd01", "subject": "bench: add reusable index sync benchmark helpers", "files": 3, "add": 245, "del": 0 }, { "sha": "d02a3de47b", "subject": "bench: add TxoSpenderIndex, TxIndex and BlockFilterIndex sync benchmarks", "files": 4, "add": 169, "del": 0 }, { "sha": "0205152867", "subject": "doc: describe measuring I/O and memory cost of benchmarks", "files": 1, "add": 38, "del": 0 } ], "patch_truncated": false }, "input_hash": "9ace089778b2c3b8", "extracted_at": "2026-09-17T16:15:31+00:00" }