{ "number": 36385, "repo": "bitcoin/bitcoin", "url": "https://github.com/bitcoin/bitcoin/pull/36385", "title": "rest: fix getutxos POST body decoding", "author": "l0rinc", "author_association": "MEMBER", "created_at": "2026-09-30T01:04:18Z", "updated_at": "2026-10-25T11:40:49Z", "description_hash": "8ab544d718d6910e", "age_days": 33, "draft": false, "labels": [ "RPC/REST/ZMQ" ], "milestone": null, "base": "master", "head_sha": "b7622767c6e4100ea2d0ac8fae2d6c98a43d081d", "head_ref": "l0rinc/rest-getutxos-body", "head_repo": "l0rinc/bitcoin", "head_history": [ { "t": "2026-09-30T01:20:41Z", "sha": "0cc326caae461e57c1c26f3e5f6eab96ed6837b5" }, { "t": "2026-10-04T14:35:51Z", "sha": "3617de785cfc6930a0324e95daa9463b07a2df39" }, { "t": "2026-10-04T23:34:17Z", "sha": "b7622767c6e4100ea2d0ac8fae2d6c98a43d081d" } ], "additions": 24, "deletions": 11, "changed_files": 2, "commit_count": 4, "size_bucket": "S", "mergeable_state": "clean", "bot": { "drahtbot": { "present": true, "reviews": { "ack": [ { "login": "fametrano", "url": "https://github.com/bitcoin/bitcoin/pull/36385#issuecomment-6397174832" } ] }, "conflicts": [] } }, "acks_parsed": {}, "acks_tally": { "ack": 0, "stale_ack": 0, "concept_ack": 0, "approach_ack": 0, "nack": 0, "concept_nack": 0, "approach_nack": 0 }, "reviews": { "approved": 0, "changes_requested": 0, "distinct_reviewers": [ "Zeegaths", "b-l-u-e", "fametrano" ] }, "signals": { "needs_rebase": false, "ci_failed": false, "mergeable_state": "clean", "last_author_activity": "2026-10-04T23:35:43Z", "last_reviewer_activity": "2026-10-25T11:40:45Z", "last_reviewer": "fametrano", "author_silent_days": 28, "waiting_on_author_days": 7, "days_since_update": 7 }, "refs": { "mentioned": [], "depends_on": [], "fixes": [], "linked_issues": [], "references": [], "conflicts": [] }, "stack": { "shares_commits_with": [], "based_on": [], "base_for": [] }, "review_paths": [ "src/rest.cpp" ], "body": "**Problem:** Binary GET and POST requests to `/rest/getutxos` for the same outpoints should agree. GET finds the unspent output, but POST incorrectly reports no hit.\nThe POST decoder writes the body into a `DataStream` with `operator<<`, serializing it as a string and prepending a CompactSize length. It then reads the first byte of that added length as the checkmempool flag, misaligning the later request fields. Hex POST bodies use the same decoding and are affected too.\n\n**Fix:** Deserialize the supplied body bytes directly. Separate commits apply the existing 15-outpoint limit during decoding before allocating the vector, and use `SpanReader` to avoid the extra copy into a `DataStream`.", "commits": [ { "sha": "dd7998e248dc48e26c7d631a691076ef2d64bf2f", "date": "2026-10-04T23:28:11Z", "message": "test: characterize getutxos GET and POST disagreement\n\nGET and POST requests for the same outpoints should return the same binary response. The current POST parser incorrectly returns a different result. Record this disagreement before fixing body decoding.\n\nThe existing POST test incorrectly wrote txids in display byte order, so it queried different outpoints. Use COutPoint and ser_vector to encode the same outpoints as the GET request while retaining the chain-tip checks.\n\nCover hex POST bodies as well and record the current acceptance of trailing bytes and malformed hex before tightening request parsing.\n\nCo-authored-by: b-l-u-e " }, { "sha": "57dc676b5911c9c871523c36156e72d85aeb68d4", "date": "2026-10-04T23:29:07Z", "message": "rest: fix getutxos request body decoding\n\nBinary GET and POST requests for the same outpoints incorrectly return different UTXO results. Writing the POST body to DataStream with operator<< serializes it as a string and prepends a CompactSize length, which the parser incorrectly reads as request fields.\n\nInitialize the stream from the supplied body bytes. The existing binary and hex comparisons now require GET and POST responses to match.\n\nReject bytes left after the outpoint vector so a successful POST consumes the complete decoded body.\nReject malformed hex with TryParseHex instead of treating failed decoding as an empty request.\n\nCo-authored-by: b-l-u-e " }, { "sha": "f1512131e984d6d761ea73f5a6ed502532874d60", "date": "2026-10-04T23:29:19Z", "message": "rest: bound getutxos POST vector decoding\n\nThe ordinary vector reader may reserve a batch based on the declared count before enough outpoint bytes are present. Apply the existing 15-outpoint limit during deserialization so an oversized count fails before allocation. The later size check continues to cover URI inputs." }, { "sha": "b7622767c6e4100ea2d0ac8fae2d6c98a43d081d", "date": "2026-10-04T23:29:19Z", "message": "rest: read getutxos POST body in place\n\nThe DataStream span constructor copies the request body, which the HTTP layer permits up to 32 MiB. Use SpanReader to read the already held bytes without another full-body allocation and copy. Deserialization and parse-error handling stay the same." } ], "timeline": [ { "t": "2026-09-30T01:20:41Z", "kind": "force_push", "who": "l0rinc", "commit": "0cc326caae461e57c1c26f3e5f6eab96ed6837b5", "id": "push:0cc326caae461e57c1c26f3e5f6eab96ed6837b5" }, { "t": "2026-10-01T09:14:28Z", "kind": "comment", "who": "Zeegaths", "assoc": "NONE", "text": "I reproduced the issue locally by running this script:\n\n```\n#!/usr/bin/env python3\nimport sys, struct, urllib.request\n\ndef post_body(txid_hex, vout):\n txid_le = bytes.fromhex(txid_hex)[::-1] # RPC displays big-endian, wire format is little-endian\n return bytes([1]) + bytes([1]) + txid_le + struct.pack('\n\nResponse match on PR branch (pr-36385) :", "id": "c:5928408741", "url": "https://github.com/bitcoin/bitcoin/pull/36385#issuecomment-5928408741", "hash": "eea39bd4d7c2cdf9", "edited": "2026-10-02T18:12:07Z" }, { "t": "2026-10-01T17:22:08Z", "kind": "comment", "who": "l0rinc", "assoc": "MEMBER", "text": "@Zeegaths you posted the same image twice, could you please update the first one?", "id": "c:5936734432", "url": "https://github.com/bitcoin/bitcoin/pull/36385#issuecomment-5936734432", "hash": "8589503d076ddc26", "edited": null }, { "t": "2026-10-02T16:41:51Z", "kind": "review_comment", "who": "b-l-u-e", "assoc": "CONTRIBUTOR", "path": "src/rest.cpp", "commit": "0cc326caae461e57c1c26f3e5f6eab96ed6837b5", "in_reply_to": null, "text": "seems like trailing data is silently being accepted... the decoder doesnt check that the body was consumed fully..so any bytes after the outpoint vector are ignored\n\ni reproduced the issue by adding this line in interface_rest.py\n\n```diff\ngit diff test/functional/interface_rest.py\ndiff --git a/test/functional/interface_rest.py b/test/functional/interface_rest.py\nindex eb18569478..9a32c49e3b 100755\n--- a/test/functional/interface_rest.py\n+++ b/test/functional/interface_rest.py\n@@ -213,6 +213,7 @@ class RESTTest (BitcoinTestFramework):\n self.test_rest_request(\"/getutxos\", http_method='POST', req_type=ReqType.JSON, body='{\"checkmempool', status=400, ret_type=RetType.OBJ)\n self.test_rest_request(\"/getutxos\", http_method='POST', req_type=ReqType.BIN, body='{\"checkmempool', status=400, ret_type=RetType.OBJ)\n self.test_rest_request(\"/getutxos/checkmempool\", http_method='POST', req_type=ReqType.JSON, status=400, ret_type=RetType.OBJ)\n+ self.test_rest_request(\"/getutxos\", http_method='POST', req_type=ReqType.BIN, body=bin_request + b'\\x00', status=400, ret_type=RetType.OBJ)\n self.test_rest_request(f\"/getutxos/{spending[0]}_+1\", ret_type=RetType.OBJ, status=400)\n```\n\nhere below shows the node returns success response\n\noutput results\n\n```py\nbuild/test/functional/interface_rest.py\n2026-10-02T16:09:34.675935Z TestFramework (INFO): PRNG seed is: 6815011314856331724\n2026-10-02T16:09:34.726606Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test__mdx5r6s\n2026-10-02T16:09:35.188130Z TestFramework (INFO): Broadcast test transaction and sync nodes\n2026-10-02T16:09:36.216533Z TestFramework (INFO): Test the /tx URI\n2026-10-02T16:09:36.222070Z TestFramework (INFO): Query an unspent TXO using the /getutxos URI\n2026-10-02T16:09:36.240350Z TestFramework (INFO): Query a spent TXO using the /getutxos URI\n2026-10-02T16:09:36.241426Z TestFramework (INFO): Query two TXOs using the /getutxos URI\n2026-10-02T16:09:36.242814Z TestFramework (INFO): Compare binary GET and POST /getutxos responses for the same TXOs\n2026-10-02T16:09:36.244860Z TestFramework (INFO): Test the /getutxos URI with and without /checkmempool\n2026-10-02T16:09:36.277380Z TestFramework (INFO): Check some invalid requests\n2026-10-02T16:09:36.280077Z TestFramework (ERROR): Unexpected exception:\nTraceback (most recent call last):\n File \"/home/user/projects/bitcoin/test/functional/test_framework/test_framework.py\", line 145, in main\n self.run_test()\n File \"/home/user/projects/bitcoin/build/test/functional/interface_rest.py\", line 216, in run_test\n self.test_rest_request(\"/getutxos\", http_method='POST', req_type=ReqType.BIN, body=bin_request + b'\\x00', status=400, ret_type=RetType.OBJ)\n File \"/home/user/projects/bitcoin/build/test/functional/interface_rest.py\", line 90, in test_rest_request\n assert resp.status == status, f\"Expected: {status}, Got: {resp.status} ({resp.reason}) - Response: {str(resp.read())}\"\n ^^^^^^^^^^^^^^^^^^^^^\nAssertionError: Expected: 400, Got: 200 (OK) - Response: b'\\xca\\x00\\x00\\x00>\\xac\\xfa1\\xe1J\\xe7nm\\x8cp\\xe8\\x8e\\x05~\\x050Xx\\xe1\\x14\\xb9\\x0e\\xa8\\x86d\\xab\\xbed\\xd3\\x920\\x01\\x01\\x01\\x00\\x00\\x00\\x00\\xc9\\x00\\x00\\x00\\x80\\x96\\x98\\x00\\x00\\x00\\x00\\x00\"Q h\\xe0|\\xdaU\\xb4<\\x01q\\xa8\\x19\\xeaT\\xb91O=\\x17\\x05\\x821\\x04\\xba\\xc6f\\xa6\\x8eE\\xee\\x0cX\\xd6'\n2026-10-02T16:09:36.333952Z TestFramework (INFO): Not stopping nodes as test failed. The dangling processes will be cleaned up later.\n2026-10-02T16:09:36.334372Z TestFramework (WARNING): Not cleaning up dir /tmp/bitcoin_func_test__mdx5r6s\n2026-10-02T16:09:36.334541Z TestFramework (ERROR): Test failed. Test logging available at /tmp/bitcoin_func_test__mdx5r6s/test_framework.log\n2026-10-02T16:09:36.334838Z TestFramework (ERROR):\n2026-10-02T16:09:36.335156Z TestFramework (ERROR): Hint: Call /home/user/projects/bitcoin/test/functional/combine_logs.py '/tmp/bitcoin_func_test__mdx5r6s' to consolidate all logs\n2026-10-02T16:09:36.335312Z TestFramework (ERROR):\n2026-10-02T16:09:36.335445Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.\n2026-10-02T16:09:36.335608Z TestFramework (ERROR): https://github.com/bitcoin/bitcoin/issues\n2026-10-02T16:09:36.335720Z TestFramework (ERROR):\n[node 1] Cleaning up leftover process\n[node 0] Cleaning up leftover process\n```\n\nso perhaps we could keep reader named and reject leftover bytes like how transaction decoding being handled here: https://github.com/bitcoin/bitcoin/blob/65896ac58ea63e29ad19d41b492e37f8722ac715/src/core_io.cpp#L174-L182\n\n```suggestion\n SpanReader reader{MakeByteSpan(strRequestMutable)};\n reader >> fCheckMemPool >> LIMITED_VECTOR(vOutPoints, MAX_GETUTXOS_OUTPOINTS);\n if (!reader.empty()) return RESTERR(req, HTTP_BAD_REQUEST, \"Parse error\");\n```", "id": "rc:4167801310", "url": "https://github.com/bitcoin/bitcoin/pull/36385#discussion_r4167801310", "hash": "c37a0ad0fd6362a7", "edited": "2026-10-02T17:08:45Z" }, { "t": "2026-10-02T18:10:50Z", "kind": "comment", "who": "Zeegaths", "assoc": "NONE", "text": "[quoted text omitted]\n\nsorry about that, i've updated it", "id": "c:5958449192", "url": "https://github.com/bitcoin/bitcoin/pull/36385#issuecomment-5958449192", "hash": "1f7ca9db278c194b", "edited": null }, { "t": "2026-10-04T14:35:51Z", "kind": "force_push", "who": "l0rinc", "commit": "3617de785cfc6930a0324e95daa9463b07a2df39", "id": "push:3617de785cfc6930a0324e95daa9463b07a2df39" }, { "t": "2026-10-04T14:38:20Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/rest.cpp", "commit": "0cc326caae461e57c1c26f3e5f6eab96ed6837b5", "in_reply_to": 4167801310, "text": "Thanks, added trailing-byte rejection and coverage for binary and hex POST bodies", "id": "rc:4178091417", "url": "https://github.com/bitcoin/bitcoin/pull/36385#discussion_r4178091417", "hash": "441516d88bf7d94c", "edited": null }, { "t": "2026-10-04T19:00:42Z", "kind": "comment", "who": "b-l-u-e", "assoc": "CONTRIBUTOR", "text": "i tested and found this issue below\n```diff\ngit diff test/functional/interface_rest.py\ndiff --git a/test/functional/interface_rest.py b/test/functional/interface_rest.py\nindex 68fb491567..7f7ff5692d 100755\n--- a/test/functional/interface_rest.py\n+++ b/test/functional/interface_rest.py\n@@ -180,6 +180,7 @@ class RESTTest (BitcoinTestFramework):\n\n for req_type, body in [(ReqType.BIN, bin_request + b'\\x00'), (ReqType.HEX, (bin_request + b'\\x00').hex())]:\n self.test_rest_request(\"/getutxos\", http_method='POST', req_type=req_type, body=body, status=400, ret_type=RetType.OBJ)\n+ self.test_rest_request(\"/getutxos\", http_method='POST', req_type=ReqType.HEX, body=bin_request.hex() + '0', status=400, ret_type=RetType.OBJ)\n\n self.log.info(\"Test the /getutxos URI with and without /checkmempool\")\n # Create a transaction, check that it's found with /checkmempool, but\n```\n\noutput results\n\n```py\nbuild/test/functional/interface_rest.py\n2026-10-04T18:53:45.192405Z TestFramework (INFO): PRNG seed is: 8612403223243651630\n2026-10-04T18:53:45.243022Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_0yvdhtym\n2026-10-04T18:53:45.690260Z TestFramework (INFO): Broadcast test transaction and sync nodes\n2026-10-04T18:53:46.718605Z TestFramework (INFO): Test the /tx URI\n2026-10-04T18:53:46.722420Z TestFramework (INFO): Query an unspent TXO using the /getutxos URI\n2026-10-04T18:53:46.737722Z TestFramework (INFO): Query a spent TXO using the /getutxos URI\n2026-10-04T18:53:46.739081Z TestFramework (INFO): Query two TXOs using the /getutxos URI\n2026-10-04T18:53:46.740170Z TestFramework (INFO): Compare binary GET and POST /getutxos responses for the same TXOs\n2026-10-04T18:53:46.745517Z TestFramework (ERROR): Unexpected exception:\nTraceback (most recent call last):\n File \"/home/user/projects/bitcoin/test/functional/test_framework/test_framework.py\", line 145, in main\n self.run_test()\n File \"/home/user/projects/bitcoin/build/test/functional/interface_rest.py\", line 183, in run_test\n self.test_rest_request(\"/getutxos\", http_method='POST', req_type=ReqType.HEX, body=bin_request.hex() + '0', status=400, ret_type=RetType.OBJ)\n File \"/home/user/projects/bitcoin/build/test/functional/interface_rest.py\", line 90, in test_rest_request\n assert resp.status == status, f\"Expected: {status}, Got: {resp.status} ({resp.reason}) - Response: {str(resp.read())}\"\n ^^^^^^^^^^^^^^^^^^^^^\nAssertionError: Expected: 400, Got: 200 (OK) - Response: b'c900000096269ba75ce72cbc9b444672644ecd2b8754d79f8625303504fb9b8b628223440000\\n'\n2026-10-04T18:53:46.799148Z TestFramework (INFO): Not stopping nodes as test failed. The dangling processes will be cleaned up later.\n2026-10-04T18:53:46.799530Z TestFramework (WARNING): Not cleaning up dir /tmp/bitcoin_func_test_0yvdhtym\n2026-10-04T18:53:46.799703Z TestFramework (ERROR): Test failed. Test logging available at /tmp/bitcoin_func_test_0yvdhtym/test_framework.log\n2026-10-04T18:53:46.799976Z TestFramework (ERROR):\n2026-10-04T18:53:46.800302Z TestFramework (ERROR): Hint: Call /home/user/projects/bitcoin/test/functional/combine_logs.py '/tmp/bitcoin_func_test_0yvdhtym' to consolidate all logs\n2026-10-04T18:53:46.800453Z TestFramework (ERROR):\n2026-10-04T18:53:46.800569Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.\n2026-10-04T18:53:46.800735Z TestFramework (ERROR): https://github.com/bitcoin/bitcoin/issues\n2026-10-04T18:53:46.800845Z TestFramework (ERROR):\n[node 1] Cleaning up leftover process\n[node 0] Cleaning up leftover process\n```", "id": "c:5983327618", "url": "https://github.com/bitcoin/bitcoin/pull/36385#issuecomment-5983327618", "hash": "9af1e12df9baad02", "edited": null }, { "t": "2026-10-04T23:34:17Z", "kind": "force_push", "who": "l0rinc", "commit": "b7622767c6e4100ea2d0ac8fae2d6c98a43d081d", "id": "push:b7622767c6e4100ea2d0ac8fae2d6c98a43d081d" }, { "t": "2026-10-04T23:35:43Z", "kind": "comment", "who": "l0rinc", "assoc": "MEMBER", "text": "Not sure how important rejecting extra bytes is - given that nobody even noticed it was fundamentally broken -, but we might as well fix the remaining issues. Added @b-l-u-e as co-author.", "id": "c:5985643679", "url": "https://github.com/bitcoin/bitcoin/pull/36385#issuecomment-5985643679", "hash": "604986a11b998ee2", "edited": null }, { "t": "2026-10-25T11:40:45Z", "kind": "comment", "who": "fametrano", "assoc": "CONTRIBUTOR", "text": "tACK b7622767c6e4100ea2d0ac8fae2d6c98a43d081d\n\nI sent 14 POST bodies, each in binary and in hex, to regtest nodes built at the merge base and at this head, and compared each answer with the binary GET for the same outpoints, or with a 400 where a library I maintain rejects the body. The head matches on all 28 requests and the base on 10. The new test fails on the base at the GET/POST comparison.\n\nA POST body that names no outpoints gets a 200 with an empty result. A URI without outpoints and an empty body get \"Error: empty request\". The body `01 00` (checkmempool, no outpoints) was a 400 on the base only because of the misread length. A hex body of only whitespace, such as `\" \"`, is a 200 on both. Adding `if (vOutPoints.empty()) return RESTERR(req, HTTP_BAD_REQUEST, \"Error: empty request\");` before the max-outpoints check makes these a 400 too. With `01 00` and `\" \"` added to `interface_rest.py`, the test passes with the line and fails without it. I can share the harness.", "id": "c:6397174832", "url": "https://github.com/bitcoin/bitcoin/pull/36385#issuecomment-6397174832", "hash": "03bffba977186584", "edited": null } ], "labels_log": [ { "t": "2026-09-30T01:04:22Z", "action": "labeled", "label": "RPC/REST/ZMQ", "who": "DrahtBot" }, { "t": "2026-09-30T01:22:18Z", "action": "labeled", "label": "CI failed", "who": "DrahtBot" }, { "t": "2026-09-30T02:23:25Z", "action": "unlabeled", "label": "CI failed", "who": "DrahtBot" } ], "state_log": [], "text_chars": 13007, "text_tokens_estimate": 3251, "changed_paths": [ "src/rest.cpp", "test/functional/interface_rest.py" ], "files": [ { "path": "src/rest.cpp", "add": 7, "del": 6 }, { "path": "test/functional/interface_rest.py", "add": 17, "del": 5 } ], "test_lines": 22, "git": { "head": "b7622767c6e4100ea2d0ac8fae2d6c98a43d081d", "head_matches_backup": true, "base": "e7aef7e86da79000aa42f5ba6d13b0d123d9da7c", "patch_id": "51ab6873bb1eb6f3", "commits": [ { "sha": "dd7998e248", "subject": "test: characterize getutxos GET and POST disagreement", "files": 1, "add": 17, "del": 5 }, { "sha": "57dc676b59", "subject": "rest: fix getutxos request body decoding", "files": 2, "add": 10, "del": 8 }, { "sha": "f1512131e9", "subject": "rest: bound getutxos POST vector decoding", "files": 1, "add": 1, "del": 1 }, { "sha": "b7622767c6", "subject": "rest: read getutxos POST body in place", "files": 1, "add": 3, "del": 4 } ], "patch_truncated": false }, "patch_id": "51ab6873bb1eb6f3", "input_hash": "a81c5305662c2f74", "extracted_at": "2026-11-02T01:09:49+00:00" }