#31868 [IBD] specialize block serialization

ReviewabilityReviewsAgreementSize
#31868 l0rinc [IBD] specialize block serializationAuthor silent 124d0Mild196 + 38 tests
  • Speed up block serialization and Initial Block Download by reducing overhead in primitive serialization
  • Avoid unnecessary buffering and endianness conversions during single-byte writes and size computation
  • Needs rebase
  • Author silent for 86 days
  • No review verdicts recorded
  • Unaddressed objection: single-byte writes in sha256 add complexity without clear benefit (sedited)
  • Questioned whether existing stream buffering already negates the real-world IBD speedup (maflcko)
  • Support for faster block serialization, noting potential value for future kernel users (sedited, theuni)
  • 196 lines added or modified outside tests
  • 38 lines added or modified in tests
  • 98 lines removed in total
  • 11 files, 5 commits
Utilities (logging, arguments, libraries)#28 of 59
P3 · speedup
  • P3 because microbenchmarks show 10-25% faster block serialization latency and a modest ~2% IBD gain
  • Practical impact may be limited because production stream writes are already buffered
  • Ranking pass: Grouped with the other measured performance work rather than with refactors.
Ranking notes
Reviewability: ReadyStalePaused
Agreement: StrongPositivePositive w/ caveatsNeutralMildDisputedBlockedCrickets
Reviews: current code-review ACKs, then (+stale ACKs) and -NACKs; greener = more ACKs.
Size: lines added or modified outside tests, then in tests; greener = smaller.

Goal

Optimizes primitive serialization and serialized size calculation across Bitcoin Core's stream, crypto, and serialization primitives. It adds single-byte write and read overloads using fixed-extent spans (std::span<..., 1>) across AutoFile, DataStream, VectorWriter, and SHA-256 contexts to bypass general buffer allocations and memcpy calls. In addition, it consolidates primitive integer serialization using C++20 concepts and if constexpr, allowing SizeComputer to advance byte offsets directly without performing endianness conversions.

Problem: Generic serialization routines impose small per-element overheads that accumulate during high-volume operations like block serialization and Initial Block Download (IBD). Single-byte writes (e.g. booleans, VarInt prefix bytes, and int8 types) invoke general stream buffering or memcpy, while SizeComputer pointlessly computes endian transformations when only the byte length matters.

Evidence of importance:

  • Author's nanobench results report SerializeBlock latency improving from ~195 µs to ~174 µs on AppleClang and from ~867 µs to ~615 µs on GCC 13, with SizeComputerBlock dropping from ~12.0 µs to ~10.2 µs on AppleClang and ~30.9 µs to ~25.6 µs on GCC.
  • Author's 3-run hyperfine IBD benchmark to block height 880,000 reports bitcoind running '1.02 ± 0.02 times faster' (mean wall time reduced from 33,647s to 33,062s).

Scope: Touches core serialization logic in serialize.h, stream interfaces in streams.{h,cpp}, and cryptographic hash wrappers in sha256.{h,cpp} and hash.h, alongside new microbenchmarks in checkblock.cpp.

Reviewability

Stale · Author silent 124d

No author activity for 124 days; a review may wait a long time for a response.

Agreement

Mild

objection that needs an answer has none yet (sedited)

Thread notes: The PR remains in draft. theuni's request to wait for #31519 was resolved when it merged and the branch was rebased, but nonblocking questions from maflcko regarding whether the modified paths actually affect IBD and from sedited regarding the sha256 optimizations remain open.

Objections, suggestions, and questions:

ReviewerTypeHarm, request, or questionStatusMust beAuthor repliedQuote
theuni (member)objection, approachpremature specialization without static extent support from the std::span replacementfixed · fix pushed · v1, not yet re-checkedansweredyes2025-02-14: 'I think it makes sense to wait for the std::span replacement (#31519) to do this, that way we can specialize for any static extent instead'
Status from: 2025-03-20: l0rinc: 'Now that #31519 was merged, I've rebased an moved it out of draft.'
maflcko (member)objection, usefulnessAutoFile single-byte write optimizations are not hit during production IBD because reads and writes are already bufferedanswered · reply on record; standing not assessed yet · v1, not yet re-checkedansweredyes2025-04-17: 'The Buffered* then passes them to AutoFile, which is modified here, but never receives 1-byte writes anymore'
sedited (contributor)objection, scopeunclear benefit of specializing single-byte writes in crypto/sha256 adds complexityopen · v1, not yet re-checkedansweredno2025-04-17: 'I'm not sure about all these crypto changes. Will this be noticable at all? Why only do it for the sha256 hasher?'

Support:

  • theuni (member) (2025-02-14): These are nice optims [not substantive]
  • sedited (contributor) (2025-04-17): This looks ok, albeit that I'm seeing more modest performance improvements on my end. [not substantive]
  • sedited (contributor) (2025-05-01): I think block serialization being fast could be useful for future kernel users. Reading and de-serializing blocks is one of the common operations I'd expect outside of just pure block validation.

Participants:

  • theuni (member, objection, 1 statements 2025-02-14 to 2025-02-14): suggested waiting for std::span replacement (#31519) before specializing on extents
  • maflcko (member, objection, 7 statements 2025-04-17 to 2026-06-27): questioned real-world IBD speedup and motivation since BufferedReader and BufferedWriter already buffer AutoFile writes
  • sedited (contributor, objection, 6 statements 2025-04-17 to 2025-05-01): questioned sha256 changes and struggled to reproduce speedups, while noting potential usefulness for the kernel

Files

51 lines under test/bench/ci.

  • src/serialize.h +126/-84
  • src/bench/checkblock.cpp +31/-6
  • src/hash.h +23/-1
  • src/streams.h +18/-0
  • src/crypto/sha256.cpp +15/-0
  • src/streams.cpp +13/-0
  • src/test/streams_tests.cpp +3/-3
  • src/test/fuzz/autofile.cpp +2/-2
  • src/bench/rpc_blockchain.cpp +1/-1
  • src/test/crypto_tests.cpp +1/-1
  • src/crypto/sha256.h +1/-0

Card

The short text the ranking pass compares across PRs.

Specializes single-byte stream operations and optimizes SizeComputer paths across serialization primitives. Currently, single-byte serialization invokes generic buffer checks and memcpy operations, and SizeComputer executes unnecessary endianness conversions when calculating sizes. The PR adds fixed-extent single-byte write and read methods to streams and hashers, and short-circuits SizeComputer passes in serialize.h. Microbenchmarks from the author indicate an 11% to 29% reduction in block serialization time and a ~2% speedup in full IBD to height 880,000.

Data

dossier JSON · extract JSON · model ledger, generated 2026-09-18T03:36, confidence high, input hash 0560ab85df5791ad
thread read through 2026-06-30 (26 statements), head 41ef25fcba
Discussion read under schema 1 rules: suggestions and questions may be missing, and an objection may combine several harms (schema history).
labels: Needs rebase · draft

Categories

Utilities (logging, arguments, libraries)

P3 · speedup (ranking pass: Grouped with the other measured performance work rather than with refactors.)

The PR optimizes serialization primitives by adding single-byte read/write overloads and modernizing integer size computation with C++20 concepts. Microbenchmarks show 10–25% latency improvements on block serialization routines and ~2% in IBD benchmarks, though reviewers note production impact on IBD may be limited because stream writes are already buffered.

Membership: Modifies core serialization primitives in src/serialize.h, stream utilities in src/streams.{h,cpp}, and cryptographic SHA-256 helpers in src/crypto/sha256.{h,cpp}.

Factors: security/stability 0, bug 0, performance 1, user value 0, leverage 0