#36015 txorphanage: bound orphan memory by storing transactions serialized

TxOrphanage is changed to store orphan transactions in serialized byte form instead of deserialized CTransaction objects, deserializing them only when retrieved for validation or RPC. It also adjusts orphan lookup interfaces such as package candidate selection and announcement processing to operate on txids and wtxids without deserializing candidates unless selected.

Problem: The orphanage uses transaction weight as a proxy for memory usage to enforce per-peer and global memory limits. For deserialized transactions with many tiny witness stack elements, vector heap allocation overhead causes real memory consumption to drastically outstrip weight; for example, a standard-weight orphan with 199,000 1-byte witness elements consumes ~11.1MB (28 times its accounted weight limit), allowing peers to bypass orphanage memory caps and mount memory exhaustion attacks.

Evidence of importance:

• "A transaction of 199,000 1-byte witness elements weighs 398,247WU (i.e. it is of standard weight, and witness standardness cannot be checked while the inputs are missing), but uses 11.1MB of memory: 28 times what is accounted for it, and one such orphan can be retained per peer." (PR description and commit 933a9f95af)
• "Otherwise a peer holding ~40MB of witness-heavy orphans (many tiny witness elements, each a separate heap allocation once deserialized) could make a single call cost ~0.9s of CPU and ~700MB of memory." (commit 0557bc81d5)

Scope: Changes several TxOrphanage methods to work on ID structs rather than full CTransactionRef instances to avoid accidental deserialization overhead during 1P1C candidate evaluation. The largest implementation and test files (src/node/txorphanage.cpp and src/test/orphanage_tests.cpp) were truncated from the patch.

Paused · Waiting on author

Open objections from nervana21 with no author reply for 17 days; review would repeat what has been asked.

Waiting on (from the thread read):

• decision: consensus on whether alternative PR #36521 is viable 2026-10-15
• author: respond to fuzz test suggestions from nervana21 2026-09-24

Strong

substantive support, no open objection (jeanpablojp, nervana21)

Thread notes: instagibbs expressed an Approach NACK favoring alternative PR #36521 if it proves viable; author is willing to close if so, and instagibbs suggested holding off until consensus is reached on #36521.

Objections, suggestions, and questions:

Support:

• w0xlt (contributor) (Concept ACK, 2026-08-18): (no reason given) [not substantive]
• l0rinc (member) (Concept ACK, 2026-08-18): (no reason given) [not substantive]
• jeanpablojp (contributor) (Tested ACK, 2026-08-21) [tested]: Reverted `txorphanage` back to master and the new functional test fails like it should, "no orphan was evicted". With 10 peers sending one memory-heavy orphan each, master keeps all 10 and the head no
• Crypt-iQ (member) (Concept ACK, 2026-08-29): (no reason given) [not substantive]
• nervana21 (contributor) (Concept ACK, 2026-09-23): (no reason given) [not substantive]
• nervana21 (contributor) (Tested ACK, 2026-09-24) [both]: the logic is sound based on my review and testing

Participants:

• l0rinc (member, objection, 1 statements 2026-08-18 to 2026-08-18): Concept ACK, but objected to mixing refactors with hardening
• w0xlt (contributor, support, 1 statements 2026-08-18 to 2026-08-18): Concept ACK
• jeanpablojp (contributor, objection, 6 statements 2026-08-21 to 2026-08-26): Tested ACK on original branch, but raised concerns on Find1P1CPackage memory spike and test assertions
• instagibbs (member, objection, 7 statements 2026-08-24 to 2026-10-15): Approach NACK in favor of #36521 if viable; previously raised concerns on BitVM txs and 1P1C memory/CPU spike
• Crypt-iQ (member, support, 1 statements 2026-08-29 to 2026-08-29): Concept ACK
• nervana21 (contributor, support, 15 statements 2026-09-23 to 2026-09-24): Concept ACK and Tested ACK after testing and reviewing logic; left suggestions on fuzz tests

316 lines under test/bench/ci.

• src/node/txorphanage.cpp +146/-54
• src/test/orphanage_tests.cpp +118/-37
• src/test/txdownload_tests.cpp +62/-0
• src/bench/txorphanage.cpp +33/-22
• src/test/fuzz/txorphan.cpp +21/-16
• src/node/txorphanage.h +26/-8
• src/node/txdownloadman_impl.cpp +10/-7
• src/policy/packages.cpp +8/-2
• src/test/txpackage_tests.cpp +7/-0
• src/policy/packages.h +3/-0

The short text the ranking pass compares across PRs.

This PR modifies TxOrphanage to store orphan transactions in serialized form, deserializing them only when handed out for processing. The orphanage relies on transaction weight to bound memory usage, but heap overhead for deserialized witness stacks allows standard-weight transactions with many small witness elements to consume up to 28 times more memory than accounted for (11.1MB instead of ~400KB per peer). By storing serialized bytes, a transaction's memory footprint is strictly bounded by its weight, closing an orphan-based memory exhaustion vector without rejecting valid standard transactions.

P1 · DoS protection

Prevents remote memory exhaustion DoS attacks where peers bypass orphanage memory limits. Using transaction weight as a proxy for memory allows standard-weight orphans with many tiny witness elements to consume 11.1MB each (28 times accounted weight), enabling attackers to exhaust node memory.

Membership: Changes orphan handling and resource accounting in `src/node/txorphanage.cpp` and `src/node/txdownloadman_impl.cpp`.

Factors: security/stability 2, bug 2, performance 1, user value 0, leverage 0