#27865 wallet: Track no-longer-spendable TXOs separately
https://github.com/bitcoin/bitcoin/pull/27865 · · +549/-162 in 18 files, 19 commits · labels: Wallet
Goal
- Stop scanning historical spent outputs during wallet balance checks and coin selection preparation
- Eliminate transaction creation latency for wallets with extensive histories or high activity
This PR optimizes wallet operations by partitioning the in-memory transaction output (TXO) cache into two separate sets: active spendable/unconfirmed TXOs and definitely unusable TXOs. Outputs spent by confirmed transactions or made invalid by conflicting blocks are moved to a dedicated unusable container so that balance checks and coin selection avoid scanning historical spent outputs. It also introduces internal transaction state caching within TXO objects and synchronizes TXO states during reorgs, abandonments, and wallet upgrades.
Problem: Since PR 27286, the wallet caches all transaction outputs owned by the wallet in memory and iterates over all of them during balance checks and coin selection preparation. In wallets with long transaction histories or hundreds of thousands of historical outputs, this iteration takes seconds per transaction creation, causing noticeable latency for high-activity wallet users.
Category: Wallet (#4 of 84)
P2 · speedup
- P2 because it resolves a major performance bottleneck in wallets with long transaction histories
- Reduces balance check and coin selection latency by 250x to 500x on large wallets
- Crucial for high-frequency wallet operators such as payment processors, exchanges, and Lightning nodes
P2 because it addresses a severe, measurable performance bottleneck in large wallets, speeding up balance checks and coin selection preparation by 250x to 500x. remyers noted on 2023-09-12 that 'on a large synthetic test wallet I\'m seeing a 250x speed up... ~5 seconds using v25.99.0... ~20 milliseconds using latest wallet-unspent-txos'. While normal wallets do not suffer as badly, this removal of O(total transactions) iteration is essential for high-activity nodes such as exchanges, payment processors, and Lightning nodes.
Membership: Directly refactors wallet transaction output tracking, spend analysis, and balance calculation in src/wallet/.
Factors: security/stability 1, bug 0, performance 3, user value 2, leverage 1
Reviewability: Ready
- Ready for review
- Clean merge status, passing CI, and recently updated with review feedback
The PR has green CI, clean merge status, and was recently force-pushed to address review feedback.
Author status: Active; force-pushed updates on 2026-09-14 and 2026-09-15 addressing recent review comments.
Open concerns:
- Outputs unconfirmed by reorg and abandoned may be missing from balance and listunspent until wallet reload due to default min_depth in IsSpent check.
- Block-conflicted outputs can trigger an assertion failure in getbalances after removeprunedfunds due to missing state checks during TXO reconstruction.
- ReorderTransactions and m_from_me upgrades run before LoadWallet confirms load success, rewriting intact records even when loading a corrupted wallet.
- Uninitialized m_from_me optional in CWalletTx leads to libc++ hardening aborts and undefined behavior during fuzz testing.
Resolved concerns:
- Optional bool in CachedTxIsTrusted evaluated to true when false, incorrectly treating untrusted zero-conf change as trusted.
- RemoveTxs failed to keep the new m_txos and m_unusable_txos cache sets in sync with deleted transactions.
- MarkTXOUsable inside AddToWallet for inactive transactions re-added confirmed-spent inputs, causing subsequent GetBalance calls to crash on assertion.
Agreement: Blocked
- Strong support for massive performance gains, with benchmarks showing 250x to 500x speedups (remyers, w0xlt)
- Concept approval without detailed reviews (murchandamus, jonatack, rkrux)
- Multiple early state synchronization and assertion crash bugs were fixed (w0xlt)
- Unaddressed objection: crashes in getbalances, reorg balance bugs, and db rewrites on corrupt loads (jeanpablojp)
Blocked: blocking objection open with no author reply (jeanpablojp)
Reviewers strongly support the PR's optimization, with benchmarks confirming 250x to 500x speedups for large wallets. While multiple previous bugs raised by w0xlt were fixed, jeanpablojp identified four concrete correctness edge cases on 2026-08-27; the author has pushed updates but has not yet commented to confirm resolution.
- remyers reported fundrawtransaction latency dropped from ~5s to ~20ms (250x speedup)
- w0xlt benchmarked a ~508x speedup on WalletBalanceManySpent
- w0xlt identified and helped fix several state synchronization bugs
- jeanpablojp raised four edge cases regarding reorg/abandon balance updates, removeprunedfunds asserts, wallet load error handling, and uninitialized fuzz state
Objections:
| Reviewer | Kind | Harm | Status | Blocking | Author replied | Quote |
|---|---|---|---|---|---|---|
| w0xlt | correctness | Populated optional bool evaluated to true in conditional check even when value was false, treating untrusted zero-conf change as trusted | resolved | yes | yes | 2026-05-22: 'm_from_me = false passes because a populated optional converts to true' Settled: 2026-05-26: 'Fixed. The unit test is not meaningfully helpful.' |
| w0xlt | correctness | CWallet::RemoveTxs did not maintain TXO cache sets, leading to desynchronization between mapWallet and m_txos/m_unusable_txos | resolved | yes | yes | 2026-05-23: 'CWallet::RemoveTxs() deletes wallet transactions, but it was not keeping the new TXO caches (m_txos and m_unusable_txos) in sync.' Settled: 2026-07-01: 'Checking the states to revert them was starting to get a bit complicated, so I ended up doing the overkill thing of recomputing all of the TXOs after removing any transaction.' |
| w0xlt | correctness | Committing an inactive double-spend transaction moved confirmed-spent inputs into m_txos, causing assertion aborts in GetBalance | resolved | yes | yes | 2026-07-02: 'AddToWallet()\'s isInactive() branch marks the inputs of an inactive tx usable again without checking whether another confirmed tx still spends them, which lets a confirmed-spent TXO back into m_txos and trips the new Assert(!wallet.IsSpent(outpoint, 1)) in GetBalance()' Settled: 2026-07-02: 'I\'ve changed MarkTXOUsable to always do an IsSpent check. Added a test in an earlier commit as well.' |
| jeanpablojp | correctness | Spendable coins missing from balance and listunspent after reorg and abandontransaction until wallet is reloaded due to min_depth default in IsSpent | open | yes | no | 2026-08-27: 'After a reorg and abandontransaction, an output that should be spendable again is missing from the balance and from listunspent until the wallet is reloaded.' |
Support:
- remyers: Measured a 250x speedup in fundrawtransaction on a large synthetic test wallet
- w0xlt: Verified a 508x speedup on high-history wallet benchmark in PR 34360
- Zeegaths: Reported a 31% speedup in listunspent on high-activity regtest setup
- murchandamus: Concept ACK [not substantive]
- jonatack: Concept ACK if it improves performance for wallets with many transactions
- rkrux: Concept ACK as a natural next step following PR 27286
- jeanpablojp: Concept ACK [not substantive]
Participants: t-bast (neutral), remyers (support), murchandamus (support), jonatack (support), maflcko (neutral), w0xlt (objection), Zeegaths (support), rkrux (support), brunoerg (neutral), jeanpablojp (objection)
State derived from the lists: blocking objection open with no author reply (jeanpablojp) (model's own read: Mild)
Review verdicts (DrahtBot): 0 (+1)
- Stale ACK: w0xlt
- Concept ACK: remyers, murchandamus, jonatack, rkrux, jeanpablojp
Dependencies
Enables:
- Wallet benchmark PR 34360
Files
181 lines under test/bench/ci.
- src/wallet/wallet.cpp +230/-51
- src/wallet/walletdb.cpp +39/-22
- src/wallet/receive.cpp +27/-29
- test/functional/wallet_importprunedfunds.py +53/-0
- src/wallet/transaction.h +37/-12
- test/functional/wallet_balance.py +32/-0
- src/wallet/wallet.h +21/-6
- src/wallet/transaction.cpp +19/-7
- src/wallet/test/util.cpp +13/-12
- src/wallet/spend.cpp +13/-11
- test/functional/wallet_send.py +21/-0
- src/wallet/test/wallet_tests.cpp +12/-6
- test/functional/wallet_backwards_compatibility.py +18/-0
- src/wallet/test/spend_tests.cpp +9/-2
- src/wallet/rpc/transactions.cpp +2/-2
- src/wallet/receive.h +1/-1
- src/wallet/test/util.h +1/-1
- src/wallet/test/fuzz/spend.cpp +1/-0
Card
PR 27865 segregates definitively unusable wallet transaction outputs (those confirmed spent or conflicted by reorgs) into a separate container so that routine wallet operations like balance calculation and coin selection do not scan the entire transaction history. This delivers dramatic performance improvements for high-activity wallets, with benchmarks demonstrating 250x to 500x speedups for fundrawtransaction and balance queries. Concept agreement is strong across multiple contributors and maintainers, though several subtle edge-case bugs in state synchronization and cache invalidation have required repeated refinement. Four nonblocking concerns regarding edge-case balance updates and crash scenarios raised by jeanpablojp remain open pending confirmation of recent force-pushes.