#33324 blocks: add resumable reobfuscation for existing block files
| Reviewability | Reviews | Agreement | Size | ||||
|---|---|---|---|---|---|---|---|
| #33324 blocks: add resumable reobfuscation for existing block files | Waiting on author | 1 | Strong | 299 + 173 tests | |||
|
|
|
|
| |||
Reviewability: ReadyStalePaused Agreement: StrongPositivePositive w/ caveatsNeutralMildDisputedBlockedCrickets Reviews: current code-review ACKs, then (+stale ACKs) and -NACKs; greener = more ACKs. Size: lines added or modified outside tests, then in tests; greener = smaller. | |||||||
Goal
Adds a -reobfuscate-blocks startup option that rewrites existing block and undo files under a new random or user-specified XOR key before chainstate loading. File rewrites occur sequentially in 1 MiB chunks, using staged files and directory commits to guarantee crash safety and resumability across restarts. Also logs a notice recommending reobfuscation when an unobfuscated blocks directory is detected, and exposes progress through kernel notifications for the GUI and RPC warmup.
Problem: Nodes synced prior to Bitcoin Core v28 store block and undo files with an all-zero XOR key (unobfuscated). Node operators currently have no native way to obfuscate existing files, rotate keys, or remove obfuscation without redownloading the entire chain from scratch.
Evidence of importance:
- PR description: "Nodes that synced before v28 may store block and undo files effectively unobfuscated with a zero XOR key. There is no built-in way to obfuscate those existing files, rotate their key, or deliberately replace it with a zero key without resyncing."
- Author benchmarks demonstrate migration speeds on an ~850 GB datadir ranging from ~10 minutes on modern SSDs to over 24 hours on an HDD attached to a Raspberry Pi.
Risk of a mistake: medium (other). A bug in chunk copying, XOR transformation, or directory renaming could corrupt block storage files on disk, requiring an operator to reindex or redownload the chain.
Changed since the previous assessment: Unchanged in high-level approach; includes minor preparatory cleanups that remove an unused BaseIndex database obfuscation parameter and refine directory commit error reporting.
Reviewability
Paused · Waiting on author
Open objection from sedited with no author reply for 25 days; review would repeat what has been asked.
Waiting on (from the thread read):
- author: response to sedited's question on whether an external tool is sufficient given low interest 2026-10-07
Agreement
Strong
substantive support, no open objection (stickies-v, sedited, sedited)
Thread notes: Added sedited's re-ACK review (r:5441220136) as an ACK support entry and recorded their remark questioning whether an external tool is sufficient as an open suggestion.
Objections, suggestions, and questions:
| Reviewer | Type | Harm, request, or question | Status | Must be | Author replied | Quote |
|---|---|---|---|---|---|---|
| stickies-v (member) | objection, approach | Using a startup option for a one-time migration creates redundancy and potential configuration conflicts with existing options like -blocksxor | answered · author replied noting existing tools; reviewer did not follow up | answered | yes | 2025-09-08: 'I don't like using startup options for one-time operations ... maybe we can bundle this e.g. as part of `bitcoin-util` or a separate `bitcoin-xor-blocks` utility?' Status from: 2026-06-18: l0rinc: '@andrewtoth already has a separate tool for this, see https://github.com/bitcoin/bitcoin/pull/33324#issuecomment-3305103' |
| ajtowns (member) | objection, interface | Progress reporting resets when finishing undo files before block files, creating an uneven progress display | fixed · fix pushed · author grouped block and undo files for uniform progress iteration | answered | yes | 2025-09-11: 'I was surprised when it hit 100% of the undo files then restarted on the block files and was much slower' Status from: 2025-09-20: l0rinc: 'grouped the block and undo files for more uniform iteration instead of shuffling' |
| ajtowns (member) | suggestion, approach | Reset the timestamp of rewritten block files to match original timestamps | answered · author initially implemented timestamp preservation, but later removed it in c:5901894922 explaining it breaks incremental backups | answered | yes | 2025-09-11: 'Would be good to try to reset the timestamp of new_blocks to match that of old_blocks here.' Status from: 2026-09-30: l0rinc: 'Removed timestamp preservation to keep backups consistent.' |
| ajtowns (member) | objection, approach | Reading entire block files into memory at once causes high memory usage on memory-constrained systems | fixed · fix pushed · author adopted chunking from reviewer branch | answered | yes | 2025-09-11: 'Rather than reading the entire blockfile into memory at once, consider chunking it' Status from: 2025-09-20: l0rinc: 'took a few suggestion from your branch (chunking, code simplifications)' |
| ajtowns (member) | objection, correctness | Reobfuscation fails or misbehaves on pruned nodes where blk0000.dat has been deleted | fixed · fix pushed · author changed file detection back to regex matching | answered | yes | 2025-09-11: 'This doesn't seem like it would work correctly with pruning, when blk0000.dat has been deleted?' Status from: 2025-09-12: l0rinc: 'Good call, changed it back to regex matching' |
| luke-jr (contributor) | objection, approach | Adding one-time migration logic to bitcoind adds unnecessary code to the daemon when a standalone utility requires very little of the existing codebase | answered · author noted that an external tool already exists | answered | yes | 2025-09-18: 'I agree a separate utility for this seems better - this requires very little of the existing codebase, in theory.' Status from: 2025-09-18: l0rinc: 'Besides, @andrewtoth already has a tool for that, it was mentioned in the PR description.' |
| luke-jr (contributor) | suggestion, approach | Write reobfuscated files to a new directory with original names and atomically rename the directory when complete, rather than renaming every individual file | answered · author raised issues with directory copying and lack of visible progress; reviewer did not follow up | answered | yes | 2025-09-18: 'Also suggest making the files with the same names, but in a new directory, and then atomically rename the directory when complete, rather than every single file.' Status from: 2025-09-18: l0rinc: 'I will think about it, could make sense, but in that case unrelated files should also be copied over (maybe duplicated t' |
| sedited (member) | suggestion, interface | Downgrade or remove the startup warning for unobfuscated block files | fixed · fix pushed · author downgraded startup notice from warning to info | answered | yes | 2026-06-11: 'Not sure if a warning is really warranted here. Feel like this pull request would have gotten more review over the past year if this were a pressing operation.' Status from: 2026-06-18: l0rinc: 'Downgraded the unobfuscated-blocks startup notice from warning to info.' |
| sedited (member) | suggestion, maintenance | Move startup reobfuscation handling out of InitAndLoadChainstate into a dedicated function called before cache sizing in AppInitMain | fixed · fix pushed · author moved logic to ReobfuscateBlocksIfRequested before cache sizing | answered | yes | 2026-06-11: 'How about moving it to right before the cache size calculations in `AppInitMain`. It could also be moved into its little function.' Status from: 2026-06-18: l0rinc: 'Moved the startup reobfuscation handling out of `InitAndLoadChainstate` into `ReobfuscateBlocksIfRequested()`, called be' |
| sedited (member) | objection, interface | Malformed hex key lengths for -reobfuscate-blocks are not treated as fatal errors on startup | fixed · fix pushed · author made key parsing fail early on invalid hex lengths | answered | yes | 2026-06-11: 'It feels a bit off that getting the length wrong is not fatal.' Status from: 2026-06-18: l0rinc: 'Made `-reobfuscate-blocks` parsing stricter: only `0`, empty/`1`, or exact 16-character hex keys are accepted; wrong-len' |
| sedited (member) | question, approach | Where does preserving block file timestamps become relevant? | answered · author explained it was requested by ajtowns to minimize unintended changes | answered | yes | 2026-06-11: 'I don't quite understand what the purpose of this is. Where does the timestamp become relevant?' Status from: 2026-06-18: l0rinc: 'Was explicitly requested to minimize unintended changes' |
| sedited (member) | suggestion, interface | Mark the kernel notification for reobfuscation progress as resumable | fixed · fix pushed · author marked GUI reobfuscation progress as resumable | answered | yes | 2026-06-11: 'Could this be the first kernel notification where resume is possible?' Status from: 2026-06-18: l0rinc: 'Marked GUI reobfuscation progress as resumable and reused a single translated progress title.' |
| sedited (member) | objection, safety | Asserting on filesystem operations aborts the node on exceptional runtime conditions instead of returning or throwing errors | fixed · fix pushed · filesystem failures now propagate instead of aborting on assertions | answered | yes | 2026-07-23: 'It does feel a bit weird to me to assert on what is not an invariant of the program, but rather an exceptional condition. Why not just throw or return here' Status from: 2026-07-25: l0rinc: 'filesystem failures now propagate instead of aborting on assertions.' |
| sedited (member) | suggestion, approach | Consider whether an external tool is sufficient given declining relevance and lack of review interest over time | open | answered | no | 2026-10-07: 'Maybe an external tool is good enough after all?' |
Support:
- sedited (member) (Concept ACK, 2025-09-08): (no reason given) [not substantive]
- stickies-v (member) (Concept ACK, 2025-09-08): this seems like useful functionality to expose.
- sedited (member) (ACK, 2026-06-18) [read]: The migration logic looks nice, and while a C++ standalone utility would be a good candidate later, the PR should not be blocked on pioneering GUI launching and lock handling.
- sedited (member) (ACK, 2026-10-07): I still think the change is good, but this has not gotten a lot of review interest in the year it has been open, and I feel like the feature probably looses relevance over time.
Participants:
- sedited (member, support, 10 statements 2025-09-08 to 2026-10-07): support; Re-ACKed, though wondered if an external tool might be sufficient given low review interest
- stickies-v (member, support, 1 statements 2025-09-08 to 2025-09-08): support; Concept ACK, though suggested bundling into a standalone utility instead of a startup flag
- ajtowns (member, objection, 8 statements 2025-09-11 to 2025-09-15): objection; Raised concerns regarding file chunking, timestamp preservation, and pruning compatibility
- luke-jr (contributor, objection, 1 statements 2025-09-18 to 2025-09-18): objection; Advocated for a separate utility rather than adding startup migration code to the daemon
Files
174 lines under test/bench/ci.
- src/node/blockstorage.cpp +189/-26
- test/functional/feature_reobfuscation.py +134/-0
- src/init.cpp +42/-0
- src/test/streams_tests.cpp +27/-1
- doc/release-notes-33324.md +21/-0
- src/util/obfuscation.h +20/-0
- src/node/blockstorage.h +12/-0
- src/util/fs_helpers.cpp +7/-5
- src/test/fs_tests.cpp +11/-0
- src/util/fs_helpers.h +3/-1
- src/index/base.cpp +1/-2
- src/index/base.h +1/-1
- src/index/txindex.cpp +1/-1
- src/index/txospenderindex.cpp +1/-1
- src/streams.h +1/-1
- test/functional/test_runner.py +1/-0
Card
The short text the ranking pass compares across PRs.
Adds a -reobfuscate-blocks startup option to rewrite existing block and undo files under a random or user-specified XOR key without resyncing. Nodes synced prior to Bitcoin Core v28 have zero-keyed, unobfuscated block files, exposing them to false-positive corruption by antivirus scanners, with no native way to migrate or rotate keys. The migration runs before chainstate loading, stages files with commits to guarantee safe resumption after crashes or interruptions, and reports progress to GUI notifications and logs. Benchmark numbers show migration times of ~10 minutes on NVMe SSD to ~24 hours on HDD for ~850 GB datadirs.
Data
thread read through 2026-10-07 (40 statements), head ec31cf7222
labels: none
Categories
Validation
P3 · new feature
Provides a built-in startup option to obfuscate or rotate keys for existing block and undo files without needing to resync from scratch. The feature offers a clear use case for nodes synced before v28, but as noted in discussion, the need is niche and reasonably deferrable.
Membership: Touches src/node/blockstorage.cpp (+189/-26) and src/node/blockstorage.h to implement resumable block and undo file reobfuscation.
Factors: security/stability 1, bug 0, performance 0, user value 1, leverage 0