{
 "number": 36548,
 "repo": "bitcoin/bitcoin",
 "url": "https://github.com/bitcoin/bitcoin/pull/36548",
 "title": "net: serve `MSG_BLOCK` by stripping witnesses in place",
 "author": "l0rinc",
 "author_association": "MEMBER",
 "created_at": "2026-10-16T01:08:15Z",
 "updated_at": "2026-11-01T18:25:29Z",
 "description_hash": "87c177e97cf940a7",
 "age_days": 17,
 "draft": false,
 "labels": [
  "P2P"
 ],
 "milestone": null,
 "base": "master",
 "head_sha": "d36943b49ce321cd622a8ae5a885227c7b3189cf",
 "head_ref": "l0rinc/msg-block-serving",
 "head_repo": "l0rinc/bitcoin",
 "head_history": [
  {
   "t": "2026-10-16T13:37:28Z",
   "sha": "0b01ce05e162ee61fb0e3c0d8413e0e1af4c1976"
  },
  {
   "t": "2026-10-31T15:02:36Z",
   "sha": "4ddf35f099a10caf377c0d77e5c64c5d716db388"
  },
  {
   "t": "2026-10-31T15:24:32Z",
   "sha": "d36943b49ce321cd622a8ae5a885227c7b3189cf"
  }
 ],
 "additions": 482,
 "deletions": 48,
 "changed_files": 8,
 "commit_count": 8,
 "size_bucket": "L",
 "mergeable_state": "clean",
 "bot": {
  "drahtbot": {
   "present": true,
   "reviews": {
    "ack": [
     {
      "login": "fametrano",
      "url": "https://github.com/bitcoin/bitcoin/pull/36548#pullrequestreview-5729574972"
     }
    ]
   },
   "conflicts": []
  }
 },
 "acks_parsed": {},
 "acks_tally": {
  "ack": 0,
  "stale_ack": 0,
  "concept_ack": 0,
  "approach_ack": 0,
  "nack": 0,
  "concept_nack": 0,
  "approach_nack": 0
 },
 "reviews": {
  "approved": 0,
  "changes_requested": 0,
  "distinct_reviewers": [
   "fametrano"
  ]
 },
 "signals": {
  "needs_rebase": false,
  "ci_failed": false,
  "mergeable_state": "clean",
  "last_author_activity": "2026-10-31T15:24:32Z",
  "last_reviewer_activity": "2026-11-01T18:25:24Z",
  "last_reviewer": "fametrano",
  "author_silent_days": 1,
  "waiting_on_author_days": 0,
  "days_since_update": 0
 },
 "refs": {
  "mentioned": [
   13011,
   13098,
   13151,
   31551,
   36476
  ],
  "depends_on": [],
  "fixes": [],
  "linked_issues": [],
  "references": [
   {
    "number": 36476,
    "type": "issue",
    "state": "open",
    "merged": false,
    "merged_at": null,
    "title": "pruneassumevalid: avoid downloading witnesses and writing blocks/undos during pruned assumevalid IBD"
   },
   {
    "number": 13151,
    "type": "pull",
    "state": "closed",
    "merged": true,
    "merged_at": "2018-05-23",
    "title": "net: Serve blocks directly from disk when possible"
   },
   {
    "number": 13011,
    "type": "pull",
    "state": "closed",
    "merged": true,
    "merged_at": "2018-05-23",
    "title": "Cache witness hash in CTransaction"
   },
   {
    "number": 13098,
    "type": "pull",
    "state": "closed",
    "merged": false,
    "merged_at": null,
    "title": "Skip tx-rehashing on historic blocks"
   },
   {
    "number": 31551,
    "type": "pull",
    "state": "closed",
    "merged": true,
    "merged_at": "2025-04-16",
    "title": "[IBD] batch block reads/writes during `AutoFile` serialization"
   }
  ],
  "conflicts": []
 },
 "stack": {
  "shares_commits_with": [],
  "based_on": [],
  "base_for": []
 },
 "review_paths": [],
 "body": "**Problem:** Serving blocks to legacy nodes currently decodes and reserializes them just to remove witnesses. `MSG_WITNESS_BLOCK` already avoids this work by serving the stored bytes directly. Faster witnessless IBD, as proposed in [#36476](https://github.com/bitcoin/bitcoin/issues/36476), also needs peers to serve these blocks efficiently.\n\n**Fix:** Read disk-backed, non-Signet `MSG_BLOCK` replies into an owned byte buffer, check the header's proof of work and expected hash, and strip witness ranges from that buffer in place.\n\nExtract a shared helper so decoding transactions and skipping their bytes follow the same encoding rules. Signet keeps `ReadBlock` for its solution check. Header checks now precede transaction parsing, so records corrupted in both places can log a different error.\n\n**Performance:**\n\nTime to read a block from disk and build its non-witness message:\n\n| Block | Before | After | Speedup |\n| --- | ---: | ---: | ---: |\n| `block413567` | 1.31 ms | 0.147 ms | **8.9x** |\n| Same block with synthetic witnesses | 3.47 ms | 0.374 ms | **9.3x** |\n\nThe synthetic fixture adds a 512-byte witness item and an empty item to every input.\n\n**Related work:** This extends the raw witness-block serving added in [#13151](https://github.com/bitcoin/bitcoin/pull/13151). The [#13011 discussion](https://github.com/bitcoin/bitcoin/pull/13011#issuecomment-383145046) and the closed [#13098](https://github.com/bitcoin/bitcoin/pull/13098) identify unnecessary transaction hashing when serving historical blocks. The implementation also builds on the bulk block reads introduced in [#31551](https://github.com/bitcoin/bitcoin/pull/31551).",
 "commits": [
  {
   "sha": "9d0110fb1e99342f1a14a00eb16ee7e4baefeccf",
   "date": "2026-10-31T14:19:40Z",
   "message": "test: cover block serving serialization\n\nCheck exact `MSG_BLOCK` and `MSG_WITNESS_BLOCK` payloads before and after recent-block cache eviction.\nCover successful and failing `ReadBlock` cases before introducing the serialized reader."
  },
  {
   "sha": "0157f6d24f786e28a5657fb992598abeae6fce89",
   "date": "2026-10-31T14:19:55Z",
   "message": "refactor: share block read hash checks\n\nExtract the expected-hash check for reuse by the serialized reader.\nKeep Signet solution validation before the shared check."
  },
  {
   "sha": "b670ec17fc9ccde290f61c2c58c909c5582d76f4",
   "date": "2026-10-31T15:15:44Z",
   "message": "bench: measure non-witness block serving\n\nMeasure disk reads followed by `TX_NO_WITNESS` message serialization before replacing the full block reader.\n\nWitness-free: 1.31 ms/block\nWitness-bearing: 3.47 ms/block"
  },
  {
   "sha": "217c49acac642133bd07ce01bbfabecef2bfc3f7",
   "date": "2026-10-31T15:15:45Z",
   "message": "node: read non-witness block payloads\n\nAdd `ReadBlockWithoutWitness` as a decode-and-reserialize reference implementation.\nIt checks proof of work and the expected header hash before transaction decoding, drops trailing bytes, and requires non-Signet consensus because it omits the Signet solution check.\n\nThe benchmark takes 1.42 ms/block for witness-free data and 3.65 ms/block for witness-bearing data."
  },
  {
   "sha": "c4300125be468f8e370f6d2ebdde1212b34a8d7a",
   "date": "2026-10-31T15:15:45Z",
   "message": "net: serve `MSG_BLOCK` from serialized data\n\nUse `ReadBlockWithoutWitness` for disk-backed `MSG_BLOCK` requests.\nKeep Signet and cached, witness, compact, and filtered block requests on their existing paths."
  },
  {
   "sha": "83ebf0908cd00aa4458940b629bb9928334c149f",
   "date": "2026-10-31T15:15:45Z",
   "message": "refactor: extract transaction decoding rules\n\nExtract marker, optional-data, input, output, and witness handling into `UnserializeTransactionBody` so block witness stripping can reuse the decoder's validation.\nKeep the ordinary decoder's reads, clears, and error order unchanged."
  },
  {
   "sha": "c130a8a74dc9ce0037d669c0f7b3e93e4f38d031",
   "date": "2026-10-31T15:15:45Z",
   "message": "node: strip witnesses without decoding\n\nWalk serialized transactions with `UnserializeTransactionBody` and copy only retained ranges.\nThis shares the ordinary decoder's marker, optional-data, CompactSize, and witness checks without allocating transactions or computing their cached hashes.\nWitness-free encodings are still walked completely, and trailing bytes remain excluded.\n\nThe benchmark falls from 1.41 to 0.161 ms/block for witness-free data and from 3.65 to 0.384 ms/block for witness-bearing data."
  },
  {
   "sha": "d36943b49ce321cd622a8ae5a885227c7b3189cf",
   "date": "2026-10-31T15:15:45Z",
   "message": "node: strip block witnesses in place\n\nReuse the owned input buffer and move retained ranges only after a removed witness gap.\nEach destination is at or before its source, so `memmove` leaves unread bytes intact.\nErase the unused tail after the walk.\n\nThe benchmark falls from 0.161 to 0.147 ms/block for witness-free data and from 0.384 to 0.374 ms/block for witness-bearing data."
  }
 ],
 "timeline": [
  {
   "t": "2026-10-16T13:37:28Z",
   "kind": "force_push",
   "who": "l0rinc",
   "commit": "0b01ce05e162ee61fb0e3c0d8413e0e1af4c1976",
   "id": "push:0b01ce05e162ee61fb0e3c0d8413e0e1af4c1976"
  },
  {
   "t": "2026-10-25T11:41:24Z",
   "kind": "comment",
   "who": "fametrano",
   "assoc": "NONE",
   "text": "tACK 0b01ce05e162ee61fb0e3c0d8413e0e1af4c1976\n\nI checked that the new `MSG_BLOCK` reply is byte for byte the old one. On 214 blocks, the head's `ReadBlockWithoutWitness` matches its `ReadBlock` plus `TX_NO_WITNESS`, that old path matches the base's, and btclib, a library I maintain, agrees. The blocks are mainnet 481824, 709632, 774628, 920000 and the bench's 413567, plus 209 regtest blocks with segwit v0 and v1 spends, 1,000 witness items, empty stacks and items, a 3,990,000-byte annex, 300 inputs, 316 outputs, 260 transactions, or no witness. Over P2P, a head node and a base node give the same `MSG_BLOCK` and `MSG_WITNESS_BLOCK` payloads for all 209: 208 from disk, and the two successive tips from the recent-block cache.\n\nEach of seven mutants of `StripBlockWitness` and `ReadBlockWithoutWitness` fails `blockmanager_tests`. They range from keeping the marker or the witnesses to skipping the proof-of-work or hash check. The new `p2p_getdata` case also passes against a base node, so it guards the payload format rather than the new path.\n\nBuilding the reply got 7.2x to 9.0x faster than on the base for those five mainnet blocks and the two largest regtest blocks (median of three runs, Release, Apple M5). Peak heap fell from 2.9 to 2.0 MB for 481824 and from 7.9 to 4.0 MB for 774628. I measured with a Boost test that writes each block with `WriteBlock` and reads it back both ways, and a functional test running a head and a base node; I can share both as a gist.",
   "id": "c:6397181019",
   "url": "https://github.com/bitcoin/bitcoin/pull/36548#issuecomment-6397181019",
   "hash": "f0e1d961eaecfec3",
   "edited": null
  },
  {
   "t": "2026-10-31T15:02:36Z",
   "kind": "force_push",
   "who": "l0rinc",
   "commit": "4ddf35f099a10caf377c0d77e5c64c5d716db388",
   "id": "push:4ddf35f099a10caf377c0d77e5c64c5d716db388"
  },
  {
   "t": "2026-10-31T15:24:32Z",
   "kind": "force_push",
   "who": "l0rinc",
   "commit": "d36943b49ce321cd622a8ae5a885227c7b3189cf",
   "id": "push:d36943b49ce321cd622a8ae5a885227c7b3189cf"
  },
  {
   "t": "2026-11-01T18:25:24Z",
   "kind": "review",
   "who": "fametrano",
   "assoc": "NONE",
   "state": "COMMENTED",
   "commit": "d36943b49ce321cd622a8ae5a885227c7b3189cf",
   "text": "tACK d36943b49ce321cd622a8ae5a885227c7b3189cf\n\nThe PR's own changes are the ones I tested on 0b01ce05: the range-diff only moves three bench includes into the earlier bench commits and places the new tests before master's new `blockmanager_tests` case, and the net diff is unchanged line for line. The new base widens block file positions to 64 bits, which the new read path goes through, so I re-ran the byte-for-byte check rather than carry the old result over.\n\nOn this head, `ReadBlockWithoutWitness` matches `ReadBlock` plus `TX_NO_WITNESS` for the same five mainnet blocks and for 263 new regtest blocks, and btclib agrees. The regtest blocks include taproot annexes of up to 3,990,001 bytes, witness stacks of 1,001 items, 260 transactions, 300 inputs and 316 outputs, and three blocks with no witness. Over P2P, `MSG_BLOCK` equals the test framework's non-witness serialization for all 263 blocks read from disk and for 22 of them while they were the tip. `MSG_WITNESS_BLOCK` equals `getblock`. `blockmanager_tests`, `p2p_getdata`, `p2p_segwit` and `p2p_compactblocks` pass. I did not re-run the mutants or the timings. Scripts and output: https://gist.github.com/fametrano/740bc1f88f3d5967023c920824bafd70/29865a5e048c740b1ac50fbc311e2f1607bfaa3d",
   "id": "r:5729574972",
   "url": "https://github.com/bitcoin/bitcoin/pull/36548#pullrequestreview-5729574972",
   "hash": "c8b8b2cae58cb30c",
   "edited": null
  }
 ],
 "labels_log": [
  {
   "t": "2026-10-16T01:08:19Z",
   "action": "labeled",
   "label": "P2P",
   "who": "DrahtBot"
  },
  {
   "t": "2026-10-16T02:21:11Z",
   "action": "labeled",
   "label": "CI failed",
   "who": "DrahtBot"
  },
  {
   "t": "2026-10-18T05:41:37Z",
   "action": "unlabeled",
   "label": "CI failed",
   "who": "DrahtBot"
  },
  {
   "t": "2026-10-28T23:18:14Z",
   "action": "labeled",
   "label": "Needs rebase",
   "who": "DrahtBot"
  },
  {
   "t": "2026-10-31T15:25:01Z",
   "action": "labeled",
   "label": "CI failed",
   "who": "DrahtBot"
  },
  {
   "t": "2026-10-31T16:27:54Z",
   "action": "unlabeled",
   "label": "CI failed",
   "who": "DrahtBot"
  },
  {
   "t": "2026-10-31T16:37:15Z",
   "action": "unlabeled",
   "label": "Needs rebase",
   "who": "DrahtBot"
  }
 ],
 "state_log": [
  {
   "t": "2026-10-18T04:49:05Z",
   "kind": "closed",
   "who": "l0rinc"
  },
  {
   "t": "2026-10-18T04:49:10Z",
   "kind": "reopened",
   "who": "l0rinc"
  }
 ],
 "text_chars": 6836,
 "text_tokens_estimate": 1709,
 "changed_paths": [
  "src/bench/readwriteblock.cpp",
  "src/net_processing.cpp",
  "src/node/blockstorage.cpp",
  "src/node/blockstorage.h",
  "src/primitives/transaction.h",
  "src/test/blockmanager_tests.cpp",
  "src/test/fuzz/block.cpp",
  "test/functional/p2p_getdata.py"
 ],
 "files": [
  {
   "path": "src/bench/readwriteblock.cpp",
   "add": 52,
   "del": 0
  },
  {
   "path": "src/net_processing.cpp",
   "add": 11,
   "del": 9
  },
  {
   "path": "src/node/blockstorage.cpp",
   "add": 87,
   "del": 7
  },
  {
   "path": "src/node/blockstorage.h",
   "add": 9,
   "del": 0
  },
  {
   "path": "src/primitives/transaction.h",
   "add": 45,
   "del": 29
  },
  {
   "path": "src/test/blockmanager_tests.cpp",
   "add": 185,
   "del": 2
  },
  {
   "path": "src/test/fuzz/block.cpp",
   "add": 46,
   "del": 0
  },
  {
   "path": "test/functional/p2p_getdata.py",
   "add": 47,
   "del": 1
  }
 ],
 "test_lines": 333,
 "git": {
  "head": "d36943b49ce321cd622a8ae5a885227c7b3189cf",
  "head_matches_backup": true,
  "base": "ab7e668cc6af5d716ad40166d1f88235a2f6d25d",
  "patch_id": "db8008e6cda1fb5c",
  "commits": [
   {
    "sha": "9d0110fb1e",
    "subject": "test: cover block serving serialization",
    "files": 2,
    "add": 86,
    "del": 1
   },
   {
    "sha": "0157f6d24f",
    "subject": "refactor: share block read hash checks",
    "files": 2,
    "add": 19,
    "del": 13
   },
   {
    "sha": "b670ec17fc",
    "subject": "bench: measure non-witness block serving",
    "files": 1,
    "add": 52,
    "del": 0
   },
   {
    "sha": "217c49acac",
    "subject": "node: read non-witness block payloads",
    "files": 5,
    "add": 235,
    "del": 6
   },
   {
    "sha": "c4300125be",
    "subject": "net: serve `MSG_BLOCK` from serialized data",
    "files": 2,
    "add": 5,
    "del": 5
   },
   {
    "sha": "83ebf0908c",
    "subject": "refactor: extract transaction decoding rules",
    "files": 1,
    "add": 45,
    "del": 29
   },
   {
    "sha": "c130a8a74d",
    "subject": "node: strip witnesses without decoding",
    "files": 1,
    "add": 49,
    "del": 5
   },
   {
    "sha": "d36943b49c",
    "subject": "node: strip block witnesses in place",
    "files": 1,
    "add": 7,
    "del": 5
   }
  ],
  "patch_truncated": false
 },
 "patch_id": "db8008e6cda1fb5c",
 "input_hash": "f543aa8a20c1685b",
 "extracted_at": "2026-11-02T01:09:49+00:00"
}