#36549 WIP: improve MallocUsage and test in CI
| Reviewability | Reviews | Agreement | Size | ||||
|---|---|---|---|---|---|---|---|
| #36549 WIP: improve MallocUsage and test in CI | Ready | 0 | Crickets | 94 + 296 tests | |||
|
|
|
|
| |||
Reviewability: ReadyStalePaused Agreement: StrongPositivePositive w/ caveatsNeutralMildDisputedBlockedCrickets Reviews: current code-review ACKs, then (+stale ACKs) and -NACKs; greener = more ACKs. Size: lines added or modified outside tests, then in tests; greener = smaller. | |||||||
Goal
Replaces the generic heuristic in `memusage::MallocUsage` with allocator-specific models for glibc, musl, macOS libmalloc, and Windows heap allocators, and accounts for STL container overheads under libc++ and MSVC. Adds unit tests that measure allocator chunk spacing across allocation sizes to check `MallocUsage` accuracy. Updates functional tests (`mempool_limit.py` and `fill_mempool`) to dynamically adapt to varying malloc overhead instead of relying on hardcoded memory sizes.
Problem: `memusage::MallocUsage()` relied on an outdated heuristic measured on glibc 2.19 that overestimates chunk sizes by 16 bytes for half of all requests, ignores minimum chunk sizes and mmapped thresholds, and fails to model non-glibc allocators on macOS, Windows, and musl. This leads to inaccurate dynamic memory accounting across the node (such as the mempool and UTXO cache) and broke tests when allocator overhead assumptions failed.
Evidence of importance:
- Commit 3d35a71b3d: glibc model 'matches glibc 2.43 exactly on x86_64, i386, aarch64, armhf, armel, riscv64, ppc64, ppc64le, ppc, s390x, loongarch64, alpha, hppa, m68k, sh4 and sparc64 (over 300000 request sizes each)'
- Commit 3d35a71b3d: musl model 'matches RSS measurements of musl 1.2.5 within 0.2%'
- Commit 3d35a71b3d: macOS and Windows models were validated against CI measurements on macOS 15 (arm64) and Windows Server 2022
Scope: The unit test introduces custom address-gap analysis to reverse-engineer allocator slot sizes at runtime.
Risk of a mistake: medium (validation, mempool_policy, tests). Inaccuracies in memory usage calculations could lead to premature mempool eviction, incorrect cache flushing, or excessive memory consumption causing out-of-memory crashes.
Changed since the previous assessment: Adapted functional tests in `mempool_limit.py` and `test_framework/mempool_util.py` to handle dynamic allocator overheads during mempool filling and eviction, and enabled exhaustive testing of all sizes (`ALL_SIZES = true`) in `memusage_tests.cpp`.
Reviewability
Ready
Nothing known that would invalidate a review now.
Agreement
Crickets
no substantive comment either way
Objections, suggestions, and questions: none recorded.
Files
308 lines under test/bench/ci.
- src/test/memusage_tests.cpp +267/-0
- src/memusage.h +94/-9
- test/functional/test_framework/mempool_util.py +18/-6
- src/test/pool_tests.cpp +6/-2
- test/functional/mempool_limit.py +4/-4
- src/test/CMakeLists.txt +1/-0
Card
The short text the ranking pass compares across PRs.
This PR replaces the heuristic in memusage::MallocUsage with platform-specific allocator models for glibc, musl, macOS libmalloc, and Windows heap allocators, while adjusting STL node overhead for libc++ and MSVC. The old formula, measured on glibc 2.19, distorted memory accounting across caches and the mempool by overestimating half of glibc allocations by 16 bytes and miscalculating on non-glibc platforms. Author verification shows exact matches across 16 architectures for glibc 2.43 and within 0.2% for musl 1.2.5. Functional tests are also updated to prevent failures from platform-dependent allocator overheads.
Data
labels: none · draft
Categories
Utilities (logging, arguments, libraries)
P3 · bug fix (ranking pass: Moved up: wrong allocator accounting distorts mempool and cache limits on every non-glibc platform, the broadest correctness issue in the P3 band.)
The PR replaces outdated glibc 2.19 allocation heuristics with accurate models for glibc, musl, macOS, and Windows. The commit notes explain the old logic 'overestimates the size of glibc chunks by 16 bytes for half of all request sizes' and lacked support for other platforms, skewing node memory accounting for caches and the mempool.
Membership: Modifies `memusage::MallocUsage`, a shared memory accounting utility primitive used across the node.
Factors: security/stability 1, bug 1, performance 1, user value 0, leverage 1