#36549 WIP: improve MallocUsage and test in CI

ReviewabilityReviewsAgreementSize
#36549 sipa WIP: improve MallocUsage and test in CIReady0Crickets94 + 296 tests
  • Accurately calculate dynamic memory usage across different operating systems and allocators
  • Prevents cache and mempool limits from relying on outdated glibc allocation assumptions
  • Ready for review
  • No review verdicts recorded
  • No reviews or comments yet
  • No substantive discussion has taken place on the pull request
  • 94 lines added or modified outside tests
  • 296 lines added or modified in tests
  • 21 lines removed in total
  • 6 files, 4 commits
Utilities (logging, arguments, libraries)#20 of 59
P3 · bug fix
  • P3 because it fixes outdated memory accounting heuristics that overestimated allocation sizes
  • Ensures mempool and cache sizing track real memory across glibc, musl, macOS, and Windows
  • Ranking pass: Moved up: wrong allocator accounting distorts mempool and cache limits on every non-glibc platform, the broadest correctness issue in the P3 band.
Reviewability: ReadyStalePaused
Agreement: StrongPositivePositive w/ caveatsNeutralMildDisputedBlockedCrickets
Reviews: current code-review ACKs, then (+stale ACKs) and -NACKs; greener = more ACKs.
Size: lines added or modified outside tests, then in tests; greener = smaller.

Goal

Replaces the generic heuristic in `memusage::MallocUsage` with allocator-specific models for glibc, musl, macOS libmalloc, and Windows heap allocators, and accounts for STL container overheads under libc++ and MSVC. Adds unit tests that measure allocator chunk spacing across allocation sizes to check `MallocUsage` accuracy. Updates functional tests (`mempool_limit.py` and `fill_mempool`) to dynamically adapt to varying malloc overhead instead of relying on hardcoded memory sizes.

Problem: `memusage::MallocUsage()` relied on an outdated heuristic measured on glibc 2.19 that overestimates chunk sizes by 16 bytes for half of all requests, ignores minimum chunk sizes and mmapped thresholds, and fails to model non-glibc allocators on macOS, Windows, and musl. This leads to inaccurate dynamic memory accounting across the node (such as the mempool and UTXO cache) and broke tests when allocator overhead assumptions failed.

Evidence of importance:

  • Commit 3d35a71b3d: glibc model 'matches glibc 2.43 exactly on x86_64, i386, aarch64, armhf, armel, riscv64, ppc64, ppc64le, ppc, s390x, loongarch64, alpha, hppa, m68k, sh4 and sparc64 (over 300000 request sizes each)'
  • Commit 3d35a71b3d: musl model 'matches RSS measurements of musl 1.2.5 within 0.2%'
  • Commit 3d35a71b3d: macOS and Windows models were validated against CI measurements on macOS 15 (arm64) and Windows Server 2022

Scope: The unit test introduces custom address-gap analysis to reverse-engineer allocator slot sizes at runtime.

Risk of a mistake: medium (validation, mempool_policy, tests). Inaccuracies in memory usage calculations could lead to premature mempool eviction, incorrect cache flushing, or excessive memory consumption causing out-of-memory crashes.

Changed since the previous assessment: Adapted functional tests in `mempool_limit.py` and `test_framework/mempool_util.py` to handle dynamic allocator overheads during mempool filling and eviction, and enabled exhaustive testing of all sizes (`ALL_SIZES = true`) in `memusage_tests.cpp`.

Reviewability

Ready

Nothing known that would invalidate a review now.

Agreement

Crickets

no substantive comment either way

Objections, suggestions, and questions: none recorded.

Files

308 lines under test/bench/ci.

  • src/test/memusage_tests.cpp +267/-0
  • src/memusage.h +94/-9
  • test/functional/test_framework/mempool_util.py +18/-6
  • src/test/pool_tests.cpp +6/-2
  • test/functional/mempool_limit.py +4/-4
  • src/test/CMakeLists.txt +1/-0

Card

The short text the ranking pass compares across PRs.

This PR replaces the heuristic in memusage::MallocUsage with platform-specific allocator models for glibc, musl, macOS libmalloc, and Windows heap allocators, while adjusting STL node overhead for libc++ and MSVC. The old formula, measured on glibc 2.19, distorted memory accounting across caches and the mempool by overestimating half of glibc allocations by 16 bytes and miscalculating on non-glibc platforms. Author verification shows exact matches across 16 architectures for glibc 2.43 and within 0.2% for musl 1.2.5. Functional tests are also updated to prevent failures from platform-dependent allocator overheads.

Data

dossier JSON · extract JSON · model ledger, generated 2026-10-18T01:14, confidence high, input hash 784d0f47c74a0259
labels: none · draft

Categories

Utilities (logging, arguments, libraries)

P3 · bug fix (ranking pass: Moved up: wrong allocator accounting distorts mempool and cache limits on every non-glibc platform, the broadest correctness issue in the P3 band.)

The PR replaces outdated glibc 2.19 allocation heuristics with accurate models for glibc, musl, macOS, and Windows. The commit notes explain the old logic 'overestimates the size of glibc chunks by 16 bytes for half of all request sizes' and lacked support for other platforms, skewing node memory accounting for caches and the mempool.

Membership: Modifies `memusage::MallocUsage`, a shared memory accounting utility primitive used across the node.

Factors: security/stability 1, bug 1, performance 1, user value 0, leverage 1