{
 "number": 36549,
 "repo": "bitcoin/bitcoin",
 "url": "https://github.com/bitcoin/bitcoin/pull/36549",
 "title": "WIP: improve MallocUsage and test in CI",
 "author": "sipa",
 "author_association": "MEMBER",
 "created_at": "2026-10-16T04:27:29Z",
 "updated_at": "2026-10-17T07:19:34Z",
 "description_hash": "851915a011904211",
 "age_days": 16,
 "draft": true,
 "labels": [],
 "milestone": null,
 "base": "master",
 "head_sha": "6140d4c45155987c461b928ad9e94abf714208d6",
 "head_ref": "202610_mallocusage",
 "head_repo": "sipa/bitcoin",
 "head_history": [
  {
   "t": "2026-10-16T04:38:07Z",
   "sha": "24cbd0b979436b7ae76181fc453c3fbb16303cfe"
  },
  {
   "t": "2026-10-16T04:51:51Z",
   "sha": "672e36b962a8363e89ec9b163097289ff9a78a83"
  },
  {
   "t": "2026-10-16T06:10:58Z",
   "sha": "5e3d7e6efcfd6b13073a7c7b4c262a99e1b04c36"
  },
  {
   "t": "2026-10-16T06:43:49Z",
   "sha": "60524a8d93749502632aaa4adfcffe99c71b6683"
  },
  {
   "t": "2026-10-17T00:34:32Z",
   "sha": "7eff0312049aa11c205c3cafca730c8877da037a"
  },
  {
   "t": "2026-10-17T06:11:08Z",
   "sha": "0f0f6c304027cb9d689f28db0023e0844736b0d3"
  },
  {
   "t": "2026-10-17T07:19:34Z",
   "sha": "6140d4c45155987c461b928ad9e94abf714208d6"
  }
 ],
 "additions": 390,
 "deletions": 21,
 "changed_files": 6,
 "commit_count": 4,
 "size_bucket": "L",
 "mergeable_state": "clean",
 "bot": {
  "drahtbot": {
   "present": true,
   "reviews": {},
   "conflicts": []
  }
 },
 "acks_parsed": {},
 "acks_tally": {
  "ack": 0,
  "stale_ack": 0,
  "concept_ack": 0,
  "approach_ack": 0,
  "nack": 0,
  "concept_nack": 0,
  "approach_nack": 0
 },
 "reviews": {
  "approved": 0,
  "changes_requested": 0,
  "distinct_reviewers": []
 },
 "signals": {
  "needs_rebase": false,
  "ci_failed": false,
  "mergeable_state": "clean",
  "last_author_activity": "2026-10-17T07:19:34Z",
  "last_reviewer_activity": null,
  "last_reviewer": null,
  "author_silent_days": 15,
  "waiting_on_author_days": 0,
  "days_since_update": 15
 },
 "refs": {
  "mentioned": [],
  "depends_on": [],
  "fixes": [],
  "linked_issues": [],
  "references": [],
  "conflicts": []
 },
 "stack": {
  "shares_commits_with": [],
  "based_on": [],
  "base_for": []
 },
 "review_paths": [],
 "body": "This is a WIP to improve the `MallocUsage` function, with logic to measure actual usage in the unit test code.",
 "commits": [
  {
   "sha": "02d2134d5f78d71b1fe824947fee368952c361ed",
   "date": "2026-10-16T23:52:24Z",
   "message": "test: make fill_mempool independent of malloc overhead\n\nfill_mempool() relied on exactly 72 of its ~67 kvB transactions fitting\nin a 5 MB mempool, and 75 not fitting, which only holds while each one\ncosts between 66.7 kB and 69.4 kB of mempool memory. That depends on\nhow much memory the platform's malloc uses for the 67443-byte output\nscript of each. Instead, keep sending transactions while two more would\nstill fit, judging by the mempool usage that the last one added.\n\nSimilarly, mempool_limit.py checked that transactions are large enough\nto trigger eviction by comparing their vsize against the maximum\nmempool usage minus the total vsize of the mempool. Compare against the\nremaining mempool usage instead: the usage of these transactions is at\nleast their vsize, as they are padded with output script data."
  },
  {
   "sha": "3d35a71b3db0fdfa4ec70d292cd88b108eb95c2a",
   "date": "2026-10-17T07:17:53Z",
   "message": "memusage: model real malloc implementations in MallocUsage\n\nThe previous formula was measured on glibc 2.19 and overestimates the\nsize of glibc chunks by 16 bytes for half of all request sizes, ignores\nglibc's minimum chunk size, and has no notion of mmapped allocations.\n\nModel the allocators actually used by Linux, macOS and Windows builds:\n- glibc (ptmalloc2, all Linux release builds): chunks of\n  max(MINSIZE, round_up(size + sizeof(size_t), MALLOC_ALIGNMENT)) bytes,\n  with chunks of 128 KiB or more mmapped as whole pages. This matches\n  glibc 2.43 exactly on x86_64, i386, aarch64, armhf, armel, riscv64,\n  ppc64, ppc64le, ppc, s390x, loongarch64, alpha, hppa, m68k, sh4 and\n  sparc64 (over 300000 request sizes each).\n- musl (mallocng, used by the Alpine CI job): a table of the steady-state\n  cost per slot for each of its 48 size classes, including group headers\n  and out-of-band metadata, which matches RSS measurements of musl 1.2.5\n  within 0.2%.\n- macOS (libmalloc): no in-band headers, with requests rounded up to 16\n  bytes up to 1008 bytes (tiny/nano), to 512 bytes up to 32 KiB (small),\n  to 32 KiB up to 8 MiB (medium), and to whole pages beyond. This matches\n  the CI measurements on macOS 15 (arm64) for every size the previous\n  estimate got wrong (1-112, 1025-7200 and 32769-493424); the other sizes\n  up to 4 MiB were within the test's tolerance. The 8 MiB limit of the\n  medium range is libmalloc's default (from its source), not measured.\n- Windows x64 (the Windows heap, used by the UCRT's malloc): blocks of\n  round_up(size + 8, 16) bytes, except that sizes up to 16 KiB come from\n  its low-fragmentation heap, in buckets of 16 to 512 bytes granularity.\n  This matches the CI measurements on Windows Server 2022 up to 512 KiB\n  (beyond that, addresses no longer reflect usage). Blocks of over\n  0xFF000 bytes get their own VirtualAlloc() region, and are counted as\n  their committed pages (not measured).\n\nOther platforms (BSDs) keep the previous estimate until they are\nmeasured.\n\nAlso count the hash that libc++ stores in every std::unordered_map and\nstd::unordered_set node, and the second list pointer of MSVC's STL. The\nprevious MallocUsage() overestimated small allocations enough to hide\nthis, but with the accurate models, pool_tests/memusage_test fails with\nlibc++ (it finds that the pool allocator uses more memory than\nstd::allocator, as the std_map nodes are counted as 24 instead of 32\nbytes), and would with MSVC.\n\nThat test also assumes that malloc has a per-allocation overhead, which\nthe pool allocator avoids. macOS's allocator has none for small blocks,\nso there the pool really uses slightly more memory (for the list of its\nchunks). Only check that the pool saves memory where malloc's overhead\nfor the nodes is significant."
  },
  {
   "sha": "b1fa1a6aa44c90f19cb6a0ccbb8540adb57f1bef",
   "date": "2026-10-17T07:17:53Z",
   "message": "test: compare MallocUsage against measured allocator behaviour\n\nAdd a unit test that measures how much memory malloc() actually consumes\nand fails if MallocUsage() is more than a factor 1+1/16 below, or more\nthan a factor 1+1/8 above, the measured usage. The upper bound is looser\nbecause MallocUsage() may include out-of-band allocator metadata that\nthe measurement does not see (up to 1/8 for musl's smallest size class).\n\nBy default, a random set of sizes up to 4 MiB (512 KiB on Windows, see\nbelow) is tested: for every power of two 2^b, 16 >> (b / 4) (but at\nleast one) sizes in [2^b, 2^(b+1)), so small sizes are tested more\ndensely than large ones. Every mismatching size is reported on its own\nline, with its measured usage.\n\nWhen the ALL_SIZES constant is set to true, the test instead determines\nthe usage of every size up to 4 MiB (searching for the boundaries\nbetween size classes, assuming usage does not decrease with size),\nreports mismatching size ranges, and prints the results in a form meant\nfor writing MallocUsage() code, e.g. for glibc on x86_64:\n\n  1-24: 32\n  25-131048: round_up(size + 8, 16)\n  131049-4194280: round_up(size + 24, 4096)\n\nThis takes about 25 seconds with glibc, and 5 with musl. In this mode,\nglibc's mmap threshold is pinned to its default, as glibc otherwise\nraises it whenever an mmapped block is freed, which would make the\nresults depend on the order of the measurements.\n\nThe usage of a size is measured as the smallest distance between\nneighbouring blocks (in address order), among up to 16384 live\nallocations of that size, that accounts for at least 10% of all pairs.\nThis is robust against a fragmented heap (the test binary's heap holds\nover a thousand regularly spaced free holes at startup, which dominate\nsmaller samples) and against allocators that place blocks at varying\noffsets within their slots (musl).\n\nLarge blocks are often given page-granular memory, with the block at an\noffset within it that varies from block to block (musl cycles it), which\nmakes neighbouring blocks slightly closer than the memory they use. Such\nan allocator returns a different address, less than a page away, when a\nblock is freed and allocated again. If that happens (for distances of a\npage or more), the test also looks at the longest run of neighbours\nwhose distances are within a page of that distance. If all of them lie\nwithin less than a page of a grid whose spacing is a multiple of the\npage size, that spacing is the usage. Blocks that are really spaced by\nsomething else (e.g. glibc's 4080-byte chunks) drift away from any such\ngrid, though not necessarily within a short run (such as within one\nWindows heap segment), which is why this is only done when the offsets\nare seen to vary. Without this, musl's output in the ALL_SIZES mode has\ntwo lines per page above its mmap threshold, one of them 16 bytes too\nlow; with it, the output is:\n\n  1-12: 16\n  13-156: round_up(size + 4, 16)\n  157-188: 192\n  ... (one line per size class) ...\n  104829-131036: 131072\n  131037-131051: 131056\n  131052-131052: 131072\n  131053-4194284: round_up(size + 20, 4096)\n\nOn Windows, only sizes up to 512 KiB are tested: larger blocks often get\ntheir own heap segment or VirtualAlloc() region, so the distances\nbetween them reflect reserved address space rather than usage.\n\nSanitizer builds are skipped, as MallocUsage() does not model their\nallocators."
  },
  {
   "sha": "6140d4c45155987c461b928ad9e94abf714208d6",
   "date": "2026-10-17T07:17:53Z",
   "message": "test: measure every allocation size in memusage_tests\n\nEnable ALL_SIZES, so that the malloc_usage test measures the usage of\nevery allocation size up to 4 MiB (instead of a random sample) and prints\nthe results in a form meant for writing MallocUsage() code. This takes\nabout 25 seconds with glibc, and 5 with musl."
  }
 ],
 "timeline": [
  {
   "t": "2026-10-16T04:38:07Z",
   "kind": "force_push",
   "who": "sipa",
   "commit": "24cbd0b979436b7ae76181fc453c3fbb16303cfe",
   "id": "push:24cbd0b979436b7ae76181fc453c3fbb16303cfe"
  },
  {
   "t": "2026-10-16T04:51:51Z",
   "kind": "force_push",
   "who": "sipa",
   "commit": "672e36b962a8363e89ec9b163097289ff9a78a83",
   "id": "push:672e36b962a8363e89ec9b163097289ff9a78a83"
  },
  {
   "t": "2026-10-16T06:10:58Z",
   "kind": "force_push",
   "who": "sipa",
   "commit": "5e3d7e6efcfd6b13073a7c7b4c262a99e1b04c36",
   "id": "push:5e3d7e6efcfd6b13073a7c7b4c262a99e1b04c36"
  },
  {
   "t": "2026-10-16T06:43:49Z",
   "kind": "force_push",
   "who": "sipa",
   "commit": "60524a8d93749502632aaa4adfcffe99c71b6683",
   "id": "push:60524a8d93749502632aaa4adfcffe99c71b6683"
  },
  {
   "t": "2026-10-17T00:34:32Z",
   "kind": "force_push",
   "who": "sipa",
   "commit": "7eff0312049aa11c205c3cafca730c8877da037a",
   "id": "push:7eff0312049aa11c205c3cafca730c8877da037a"
  },
  {
   "t": "2026-10-17T06:11:08Z",
   "kind": "force_push",
   "who": "sipa",
   "commit": "0f0f6c304027cb9d689f28db0023e0844736b0d3",
   "id": "push:0f0f6c304027cb9d689f28db0023e0844736b0d3"
  },
  {
   "t": "2026-10-17T07:19:34Z",
   "kind": "force_push",
   "who": "sipa",
   "commit": "6140d4c45155987c461b928ad9e94abf714208d6",
   "id": "push:6140d4c45155987c461b928ad9e94abf714208d6"
  }
 ],
 "labels_log": [
  {
   "t": "2026-10-16T04:39:18Z",
   "action": "labeled",
   "label": "CI failed",
   "who": "DrahtBot"
  },
  {
   "t": "2026-10-17T07:12:51Z",
   "action": "unlabeled",
   "label": "CI failed",
   "who": "DrahtBot"
  }
 ],
 "state_log": [],
 "text_chars": 7449,
 "text_tokens_estimate": 1862,
 "changed_paths": [
  "src/memusage.h",
  "src/test/CMakeLists.txt",
  "src/test/memusage_tests.cpp",
  "src/test/pool_tests.cpp",
  "test/functional/mempool_limit.py",
  "test/functional/test_framework/mempool_util.py"
 ],
 "files": [
  {
   "path": "src/memusage.h",
   "add": 94,
   "del": 9
  },
  {
   "path": "src/test/CMakeLists.txt",
   "add": 1,
   "del": 0
  },
  {
   "path": "src/test/memusage_tests.cpp",
   "add": 267,
   "del": 0
  },
  {
   "path": "src/test/pool_tests.cpp",
   "add": 6,
   "del": 2
  },
  {
   "path": "test/functional/mempool_limit.py",
   "add": 4,
   "del": 4
  },
  {
   "path": "test/functional/test_framework/mempool_util.py",
   "add": 18,
   "del": 6
  }
 ],
 "test_lines": 308,
 "git": {
  "head": "6140d4c45155987c461b928ad9e94abf714208d6",
  "head_matches_backup": true,
  "base": "cce32b16874e85e8a00cfa1a2190f0b6f4dcaf66",
  "patch_id": "738427b6df653609",
  "commits": [
   {
    "sha": "02d2134d5f",
    "subject": "test: make fill_mempool independent of malloc overhead",
    "files": 2,
    "add": 22,
    "del": 10
   },
   {
    "sha": "3d35a71b3d",
    "subject": "memusage: model real malloc implementations in MallocUsage",
    "files": 2,
    "add": 100,
    "del": 11
   },
   {
    "sha": "b1fa1a6aa4",
    "subject": "test: compare MallocUsage against measured allocator behaviour",
    "files": 2,
    "add": 268,
    "del": 0
   },
   {
    "sha": "6140d4c451",
    "subject": "test: measure every allocation size in memusage_tests",
    "files": 1,
    "add": 1,
    "del": 1
   }
  ],
  "patch_truncated": false
 },
 "patch_id": "738427b6df653609",
 "input_hash": "2d8a7c0ee4014af6",
 "extracted_at": "2026-11-02T01:09:49+00:00"
}