{
 "number": 36276,
 "repo": "bitcoin/bitcoin",
 "url": "https://github.com/bitcoin/bitcoin/pull/36276",
 "title": "http: don't register RecvEvent while a request is in-flight",
 "author": "pinheadmz",
 "author_association": "MEMBER",
 "created_at": "2026-09-16T15:34:08Z",
 "updated_at": "2026-09-17T00:20:18Z",
 "age_days": 1,
 "draft": true,
 "labels": [
  "RPC/REST/ZMQ",
  "CI failed"
 ],
 "milestone": null,
 "base": "master",
 "head_sha": "0527c052fee74fad0b30fcfd1f64ce025d042922",
 "head_ref": "http-throttle-req-busy",
 "head_repo": "pinheadmz/bitcoin",
 "head_history": [],
 "additions": 40,
 "deletions": 17,
 "changed_files": 3,
 "commit_count": 2,
 "size_bucket": "S",
 "mergeable_state": "unstable",
 "bot": {
  "drahtbot": {
   "present": true,
   "reviews": {
    "concept_ack": [
     {
      "login": "hodlinator",
      "url": "https://github.com/bitcoin/bitcoin/pull/36276#pullrequestreview-5226138362"
     }
    ]
   },
   "conflicts": []
  }
 },
 "acks_parsed": {
  "hodlinator": {
   "kind": "concept_ack",
   "hash": "0066120eb648a588dc83b1d95247aef0e1135bb4",
   "t": "2026-09-16T17:43:48Z",
   "stale": false
  }
 },
 "acks_tally": {
  "ack": 0,
  "stale_ack": 0,
  "concept_ack": 1,
  "approach_ack": 0,
  "nack": 0,
  "concept_nack": 0,
  "approach_nack": 0
 },
 "reviews": {
  "approved": 0,
  "changes_requested": 0,
  "distinct_reviewers": [
   "davidgumberg",
   "hodlinator"
  ]
 },
 "signals": {
  "needs_rebase": false,
  "ci_failed": true,
  "mergeable_state": "unstable",
  "last_author_activity": "2026-09-16T22:44:49Z",
  "last_reviewer_activity": "2026-09-17T00:20:18Z",
  "last_reviewer": "davidgumberg",
  "author_silent_days": 0,
  "waiting_on_author_days": 0,
  "days_since_update": 0
 },
 "refs": {
  "mentioned": [
   36123,
   36216
  ],
  "depends_on": [],
  "fixes": [
   36216
  ],
  "linked_issues": [
   {
    "number": 36216,
    "type": "issue",
    "state": "open",
    "merged": false,
    "merged_at": null,
    "title": "qa: Intermittent failure in `interface_http.py`"
   }
  ],
  "references": [
   {
    "number": 36216,
    "type": "issue",
    "state": "open",
    "merged": false,
    "merged_at": null,
    "title": "qa: Intermittent failure in `interface_http.py`"
   },
   {
    "number": 36123,
    "type": "pull",
    "state": "closed",
    "merged": true,
    "merged_at": "2026-09-05",
    "title": "http: throttle per-connection reads while a request is in flight"
   }
  ],
  "conflicts": []
 },
 "stack": {
  "shares_commits_with": [],
  "based_on": [],
  "base_for": []
 },
 "review_paths": [
  "src/httpserver.h"
 ],
 "body": "Closes #36216\n\nFollow up to #36123, which implemented a read-side throttle so the server won't drain the socket buffer into application memory without bound. There was a tiny window left unaddressed which allowed one additional read operation of up to 65kB from the socket buffer in rare cases, correctly failing the test.\n\nThe original throttle works by only registering socket `recv` events when EITHER:\n1. A pending request is being built from socket data (`m_req !=  nullptr`)\n2. The receive buffer `m_recv_buffer` is empty.\n\nThe idea is \"don't read from the socket unless we need to, in order to complete an incoming request\".\n\nThe new rule applied in this PR is \"if a request is already busy in a worker thread, don't read at all\".\n\nThe test sends a blocking HTTP request followed by a flood of data, expecting that flood to be blocked at some point within a few seconds by TCP backpressure: the application stops reading from the socket and then the kernel stops reading from the client (`BlockingIOError` in the python test).\n\nThe TCP spec requires clients to periodically probe the server to learn when the window has been reopened:\nhttps://www.rfc-editor.org/rfc/rfc9293.html#name-zero-window-probing\n\nBecause of the bug in the throttle logic from #36123, this probe would discover a non-zero window and send more data, resetting the stall timer in the test. This may repeat until the socket buffer is full again (the kernel replaced the 65kB the application foolishly drained) at which point the stream is truly blocked.\n\nOn linux platforms [this probe is sent around 200ms,](https://codebrowser.dev/linux/linux/include/net/tcp.h.html#_M/TCP_RTO_MIN) so even after a few cycles the server throttle would still register as solid to the test.\n\nOn macos however, given the small size of the open window, [the probe is delayed 5 seconds.](https://github.com/apple-oss-distributions/xnu/blob/main/bsd/netinet/tcp_timer.h#L179) The cycle repeating every 5 seconds breaks the assumptions in the test and is interpreted as no throttle at all.\n\nNote that the worst-case scenario for this bug is only 65kB so this is not a critical OOM fix.\n\nI spent a few days with GPT, kimi, Sonnet and Opus trying to improve the test, or make it fail more reliably on master with no satisfying outcome. Tweaking the `STALL_TIMEOUT` constant, for example, either makes the test less sensitive to regression or more likely to fail on macos. I think we are at the limit of what we can test from the client side. We *could* add more DebugLog messages to the server and test server-side behavior a bit more like the other throttle test `check_slow_read_throttle()` but I chose to leave that alone for now.",
 "commits": [
  {
   "sha": "0066120eb648a588dc83b1d95247aef0e1135bb4",
   "date": "2026-09-16T14:56:40Z",
   "message": "http: don't register RecvEvent while a request is in-flight\n\nWe use TCP backpressure to throttle incoming data from the client.\nWe release the throttle when there is an empty receive buffer and\nno request in the progress of being parsed, but we should continue\nto throttle if there is a \"busy\" request being handled by a worker\nthread.\n\nOn some platforms this would allow up to 65kB of additional data\nto trickle in after the throttle was supposedly engaged."
  },
  {
   "sha": "0527c052fee74fad0b30fcfd1f64ce025d042922",
   "date": "2026-09-16T19:22:08Z",
   "message": "test: use select() to avoid blocking on macos in interface_http.py\n\nThe flood loop uses a non-blocking socket to detect when the server\nstops reading from the connection. When the server stops reading, the\nTCP receive buffer fills up and send() is expected to fail immediately\nwith \"try again\" (EAGAIN/BlockingIOError).\n\nOn macOS 15, when the buffer fills completely to zero, the kernel can\nblock inside send() waiting for the remote side to free up space,\ninstead of returning an error like it should. This causes the test to\nhang when run alongside other tests that put the system under load.\n\nFix: call select() with a short timeout before each send(). select()\nasks the kernel whether there is space to write, and always returns\nwithin the timeout. If there is no space, we never call send() at all,\nso the kernel never gets the chance to block."
  }
 ],
 "timeline": [
  {
   "t": "2026-09-16T17:36:20Z",
   "kind": "review_comment",
   "who": "hodlinator",
   "assoc": "MEMBER",
   "path": "src/httpserver.h",
   "commit": "0527c052fee74fad0b30fcfd1f64ce025d042922",
   "in_reply_to": null,
   "text": "nanonit: Wish we had a more descriptive name... something closer to:\n```suggestion\n    bool WorkerHasRequest() const { return m_req_busy.load(); }\n```"
  },
  {
   "t": "2026-09-16T17:43:48Z",
   "kind": "review",
   "who": "hodlinator",
   "assoc": "MEMBER",
   "state": "COMMENTED",
   "commit": "0066120eb648a588dc83b1d95247aef0e1135bb4",
   "text": "Concept ACK 0066120eb648a588dc83b1d95247aef0e1135bb4\n\nWill have a closer look soon."
  },
  {
   "t": "2026-09-16T19:25:21Z",
   "kind": "comment",
   "who": "pinheadmz",
   "assoc": "MEMBER",
   "text": "Added a second commit to handle this CI failure where macos-15 just hangs forever. My agents and I think what is happening is that macos is blocking inside of `send()` even though the socket is non-blocking. So we'll wrap that in a `select()` call with a timeout to detect the TCP backpressure."
  },
  {
   "t": "2026-09-16T22:44:49Z",
   "kind": "comment",
   "who": "pinheadmz",
   "assoc": "MEMBER",
   "text": "Converting to draft, i think i have this issue again where 50ms is being added to every RPC call..."
  },
  {
   "t": "2026-09-17T00:20:18Z",
   "kind": "comment",
   "who": "davidgumberg",
   "assoc": "MEMBER",
   "text": "Sorry for the dump, if you understand the issue in the PR description then this is not helpful so I'll put this inside of a details block:\n\nI had a bit of trouble following what the issue is here, so I just want to check my understanding:\n\nhttps://github.com/bitcoin/bitcoin/blob/b8215855437288cac95c7fd5d41104c73c0f3495/src/httpserver.cpp#L1034-L1039\n\n1. The server has an open socket.\na. the server's `ReadyToSend` is `False` so the first check fails. (*where `Ready` means `HaveStuffQueuedToSend` not `CouldSendStuffIfAskedTo`*)\nb. There is no current request, but the ReceiveBuffer is empty. so the `else if` branch is taken.\nc. The socket is marked with the receive event, so in `WaitMany()` we'll look for a `POLLIN` event:\n\nhttps://github.com/bitcoin/bitcoin/blob/b8215855437288cac95c7fd5d41104c73c0f3495/src/util/sock.cpp#L171-L173\n\nhttps://github.com/bitcoin/bitcoin/blob/b8215855437288cac95c7fd5d41104c73c0f3495/src/util/sock.cpp#L188-L190\n\n2. The client sends a request like `waitforblockheight`.\na. Now there is a `RecvEvent` in the `events.occurred` for the client's sock.\nb. The client sock is `recv_ready` in `SocketHandlerConnected`:\nhttps://github.com/bitcoin/bitcoin/blob/b8215855437288cac95c7fd5d41104c73c0f3495/src/httpserver.cpp#L904\nand now `Receive()` is called:\nhttps://github.com/bitcoin/bitcoin/blob/b8215855437288cac95c7fd5d41104c73c0f3495/src/httpserver.cpp#L916-L918\nc. `Receive()` fills the `m_recv_buffer` and marks the connection as busy:\nhttps://github.com/bitcoin/bitcoin/blob/b8215855437288cac95c7fd5d41104c73c0f3495/src/httpserver.cpp#L964-L968\nd. `TryReadRequest()` is called a request is constructed:\nhttps://github.com/bitcoin/bitcoin/blob/b8215855437288cac95c7fd5d41104c73c0f3495/src/httpserver.cpp#L1082-L1084\ne. `TryReadRequest()` calls `ReadRequest()` which will read some bytes from the buffer, clearing `m_recv_buffer` as it goes.\nf. Eventually the whole request is read, and turned into a job, the `m_req` is cleared and the client is marked `m_req_busy = true`:\nhttps://github.com/bitcoin/bitcoin/blob/b8215855437288cac95c7fd5d41104c73c0f3495/src/httpserver.cpp#L1133-L1134\n\n3. The next time `WaitForSockets()` is called, `m_req == nullptr` (`m_req_busy == true`) and the receive buffer is empty, so the client is marked as ready for a receive again.\na. `TryReadRequest` will get called again on this client, but the receive buffer won't be drained into a new `HTTPRequest` since the client is still marked as busy with a request until the `waitforblockheight` completes:\nhttps://github.com/bitcoin/bitcoin/blob/b8215855437288cac95c7fd5d41104c73c0f3495/src/httpserver.cpp#L1080\n\nThis is the bug since it means that the client can send the server more data (up to the server's TCP receive window limit which is usually ~64KiB or so).\n\nI think my understanding up to here is right, but I'm a little bit confused about why the functional test was failing, and how it relates to the zero-window probing, but here's what I gather:\n\n4. Because the 64KiB or so get drained from the kernels buffer into our application request buffer, this frees up the OS's receive buffer.\na. the server's kernel's window goes from 0 bytes available, to all bytes available or however many were drained into the application buffer.\nb.But the server's kernel never advertises this to the client ??\nc. Instead the client only learns this at some random time later when it does a zero-window poll.\nd. Whenever it does the zero-window probe, it actually learns that it can send another full window of data, but even after it does that, the next time `WaitForSockets()` is called, the application's receive buffer has contents so we don't mark the socket as ready to receive.\n\nOn MacOS, the zero-window probe timer happens to be ~5 seconds, so about ~5 seconds into the test, the zero-window probe is performed by the client and it learns that it can send another window of bytes, resetting the `stuck_timer`:\n\nhttps://github.com/bitcoin/bitcoin/blob/b8215855437288cac95c7fd5d41104c73c0f3495/test/functional/interface_http.py#L767-L777\n\nwhich we were waiting for to hit 5 seconds:\nhttps://github.com/bitcoin/bitcoin/blob/b8215855437288cac95c7fd5d41104c73c0f3495/test/functional/interface_http.py#L756-L758\n\nBut the test times out after 10 seconds:\n\nhttps://github.com/bitcoin/bitcoin/blob/b8215855437288cac95c7fd5d41104c73c0f3495/test/functional/interface_http.py#L759-L761\n\nSo in some cases there won't be enough time left for the stall to hit 5 seconds, and that's why the test is failing.\n\nAnd the fix here is to just check if `m_req_busy` is true, so we don't have this gap where we mean to stop reading from the socket buffer once we're busy with a request, but we actually accidentally read one more time."
  }
 ],
 "labels_log": [
  {
   "t": "2026-09-16T15:34:12Z",
   "action": "labeled",
   "label": "RPC/REST/ZMQ",
   "who": "DrahtBot"
  },
  {
   "t": "2026-09-16T17:34:49Z",
   "action": "labeled",
   "label": "CI failed",
   "who": "DrahtBot"
  }
 ],
 "state_log": [
  {
   "t": "2026-09-16T22:44:20Z",
   "kind": "convert_to_draft",
   "who": "pinheadmz"
  }
 ],
 "text_chars": 9415,
 "text_tokens_estimate": 2353,
 "changed_paths": [
  "src/httpserver.cpp",
  "src/httpserver.h",
  "test/functional/interface_http.py"
 ],
 "files": [
  {
   "path": "src/httpserver.cpp",
   "add": 13,
   "del": 4
  },
  {
   "path": "src/httpserver.h",
   "add": 3,
   "del": 0
  },
  {
   "path": "test/functional/interface_http.py",
   "add": 24,
   "del": 13
  }
 ],
 "test_lines": 37,
 "git": {
  "head": "0527c052fee74fad0b30fcfd1f64ce025d042922",
  "head_matches_backup": true,
  "base": "51ddab532cb38213e2258c24c492bc8a392ffc90",
  "commits": [
   {
    "sha": "0066120eb6",
    "subject": "http: don't register RecvEvent while a request is in-flight",
    "files": 2,
    "add": 16,
    "del": 4
   },
   {
    "sha": "0527c052fe",
    "subject": "test: use select() to avoid blocking on macos in interface_http.py",
    "files": 1,
    "add": 24,
    "del": 13
   }
  ],
  "patch_truncated": false
 },
 "input_hash": "46a8bd982fa6967d",
 "extracted_at": "2026-09-17T16:15:31+00:00"
}