{ "number": 35714, "repo": "bitcoin/bitcoin", "url": "https://github.com/bitcoin/bitcoin/pull/35714", "title": "validation: stop writes after flush failure", "author": "l0rinc", "author_association": "MEMBER", "created_at": "2026-07-13T19:37:29Z", "updated_at": "2026-09-16T21:14:31Z", "age_days": 65, "draft": false, "labels": [ "Validation" ], "milestone": null, "base": "master", "head_sha": "c42d7fe1cf2aee821b4f9931ef94f16e09de8459", "head_ref": "l0rinc/flush-state-failure-boundary", "head_repo": "l0rinc/bitcoin", "head_history": [ { "t": "2026-08-08T19:29:06Z", "sha": "0f4c2a7dd37727cd53b0b8c1e31dbc30a1b57c90" }, { "t": "2026-08-08T19:44:10Z", "sha": "9d0a5f2b0d91d03166766dc5e57e2490e8c4304a" }, { "t": "2026-08-09T01:13:08Z", "sha": "c42d7fe1cf2aee821b4f9931ef94f16e09de8459" } ], "additions": 28, "deletions": 3, "changed_files": 2, "commit_count": 2, "size_bucket": "S", "mergeable_state": "clean", "bot": { "drahtbot": { "present": true, "reviews": { "ack": [ { "login": "optout21", "url": "https://github.com/bitcoin/bitcoin/pull/35714#issuecomment-5663353895" } ], "concept_ack": [ { "login": "mzumsande", "url": "https://github.com/bitcoin/bitcoin/pull/35714#pullrequestreview-4714960377" } ], "stale_ack": [ { "login": "arejula27", "url": "https://github.com/bitcoin/bitcoin/pull/35714#issuecomment-4983411974" } ] }, "conflicts": [ { "number": 35731, "title": "Indexes: Harden the flush-error notification invariant", "author": "arejula27" }, { "number": 35307, "title": "blockstorage: keep snapshot base in normal blockfile range", "author": "shuv-amp" }, { "number": 29700, "title": "kernel, refactor: return error status on all fatal errors", "author": "ryanofsky" } ] } }, "acks_parsed": { "arejula27": { "kind": "ack", "hash": "e1a337ee961db4308697de7cddcf4aad9684c7f1", "t": "2026-07-15T17:19:56Z", "stale": true }, "optout21": { "kind": "ack", "hash": "c42d7fe1cf2aee821b4f9931ef94f16e09de8459", "t": "2026-09-14T11:41:55Z", "stale": false } }, "acks_tally": { "ack": 1, "stale_ack": 1, "concept_ack": 0, "approach_ack": 0, "nack": 0, "concept_nack": 0, "approach_nack": 0 }, "reviews": { "approved": 0, "changes_requested": 0, "distinct_reviewers": [ "arejula27", "maflcko", "mzumsande", "optout21" ] }, "signals": { "needs_rebase": false, "ci_failed": false, "mergeable_state": "clean", "last_author_activity": "2026-08-18T17:19:30Z", "last_reviewer_activity": "2026-09-14T11:41:55Z", "last_reviewer": "optout21", "author_silent_days": 29, "waiting_on_author_days": 3, "days_since_update": 0 }, "refs": { "mentioned": [ 27866, 30611, 34897 ], "depends_on": [], "fixes": [], "linked_issues": [], "references": [ { "number": 27866, "type": "pull", "state": "closed", "merged": true, "merged_at": "2023-09-29", "title": "blockstorage: Return on fatal flush errors" }, { "number": 34897, "type": "pull", "state": "closed", "merged": true, "merged_at": "2026-07-09", "title": "indexes: Don't commit ahead of the flushed chainstate" }, { "number": 30611, "type": "pull", "state": "closed", "merged": true, "merged_at": "2025-05-01", "title": "validation: write chainstate to disk every hour" } ], "conflicts": [ 35731, 35307, 29700 ] }, "stack": { "shares_commits_with": [], "based_on": [], "base_for": [] }, "review_paths": [ "src/test/chainstate_write_tests.cpp", "src/validation.cpp" ], "body": "**Problem:** When `FlushChainstateBlockFile()` fails, shutdown is requested, but `FlushStateToDisk()` still writes block-index metadata and coins data, then advances `m_last_flushed_block`.\nThose writes can record a block even though its block or undo data may not be durable.\n\n**Fix:** Return the flush error before those writes, leaving `m_last_flushed_block` at the last successfully flushed block.\n\n**History:**\n* [#27866](https://github.com/bitcoin/bitcoin/pull/27866#discussion_r1256378064) proposed returning here to avoid writing metadata for block data that failed to flush, but review requested a test and caller audit before changing behavior, so it was [deferred](https://github.com/bitcoin/bitcoin/pull/27866#discussion_r1297313148) as the current [TODO](https://github.com/bitcoin/bitcoin/pull/27866#issuecomment-1702186979); this PR now resolves it.\n* [#34897](https://github.com/bitcoin/bitcoin/pull/34897) later introduced `m_last_flushed_block` to prevent persistent indexes from advancing past flushed chainstate; this fix keeps it at the last successful flush.", "commits": [ { "sha": "772b872fc08016193d11017c7026714e9ad96caf", "date": "2026-08-08T19:15:19Z", "message": "test: characterize writes after flush failure" }, { "sha": "c42d7fe1cf2aee821b4f9931ef94f16e09de8459", "date": "2026-08-09T00:57:32Z", "message": "validation: stop writes after flush failure\n\nWhen a block-file or undo-file flush fails, shutdown is requested, but `FlushStateToDisk()` still writes block-index metadata and coins data, then advances `m_last_flushed_block`.\nReturn the error before those writes, keeping `m_last_flushed_block` at the last successful flush.\n\nCo-authored-by: optout <13562139+optout21@users.noreply.github.com>" } ], "timeline": [ { "t": "2026-07-13T20:13:41Z", "kind": "comment", "who": "arejula27", "assoc": "CONTRIBUTOR", "text": "Concept ACK\n\nI still have to dig into the code and the implications of this change (just did a quick review), but in the meantime: would it be worth adding a test case for the undo-file failure path as well? both the commit message and the error string mention block or undo files, but the test only injects a failure on blk*.dat." }, { "t": "2026-07-14T19:28:18Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/test/chainstate_write_tests.cpp", "commit": "c42d7fe1cf2aee821b4f9931ef94f16e09de8459", "in_reply_to": null, "text": "[quoted text omitted]\n\nGood question, thanks - `FlushBlockFile()` returns false for either block or undo flush failures, see: https://github.com/bitcoin/bitcoin/blob/e1a337ee961db4308697de7cddcf4aad9684c7f1/src/node/blockstorage.cpp#L774-L786\nSince this PR only changes how that shared result is handled, testing both failure sources separately would fit better in a lower-level `BlockManager` test." }, { "t": "2026-07-15T17:19:56Z", "kind": "comment", "who": "arejula27", "assoc": "CONTRIBUTOR", "text": "ACK e1a337ee961db4308697de7cddcf4aad9684c7f1\nThis is a small change, closing a gap discussed at #34897 I built it and tested it locally. I tryied to play around it and I did not find anything wrong.\n\nOne small, non-blocking point: the comment `// FlushChainstateBlockFile() already emitted the flush-error notification.` describes an invariant, but nothing checks it . It is true today only because every `false` path inside `FlushBlockFile()`/`FlushUndoFile()` calls `flushError()` first. Would be correct adding an assume to verify this behaviour is keep it in the future?:\n\n```cpp\nif (!m_blockman.FlushChainstateBlockFile(m_chain.Height())) {\n // FlushChainstateBlockFile() already emitted the flush-error notification.\n Assume(m_chainman.m_interrupt);\n return state.Error(\"Failed to flush block or undo file\");\n}\n```\n\nAlso, I want to ask about something I found while testing, not sure if it is related to this PR. `FlushStateToDisk()` does not check if shutdown was already requested before it starts. I wrote a test where a first flush fails and requests shutdown, then I fix the file and call `FlushStateToDisk()` again: it succeeds and `m_last_flushed_block` moves forward, even though shutdown was already requested before this second call. Is this expected? I think it may be needed for the final flush in `Shutdown()` (`init.cpp`), but I am not 100% sure, so I wanted to ask before assuming it is fine" }, { "t": "2026-07-15T17:36:24Z", "kind": "comment", "who": "arejula27", "assoc": "CONTRIBUTOR", "text": "I opened a follow-up PR https://github.com/bitcoin/bitcoin/pull/35731 i did while reviewing the PR with the intention to group this notify + `state.Error()` behaviour into one helper, (making the invariant of the comment harder) and add more low-level tests like the ones you mentioned above." }, { "t": "2026-07-16T11:36:46Z", "kind": "review_comment", "who": "maflcko", "assoc": "MEMBER", "path": "src/test/chainstate_write_tests.cpp", "commit": "0f04fbee2f123c37c4b6fbff7dc10f61dd0b5924", "in_reply_to": null, "text": "nit in 0f04fbee2f123c37c4b6fbff7dc10f61dd0b5924: The `&&` make this a bit harder to read and violate https://github.com/bitcoin/bitcoin/pull/35729" }, { "t": "2026-07-16T14:52:54Z", "kind": "review", "who": "mzumsande", "assoc": "MEMBER", "state": "COMMENTED", "commit": "e1a337ee961db4308697de7cddcf4aad9684c7f1", "text": "It would be good to discuss the real-world consequences of this change a bit more:\n\nAs far as I understand one reason for the status quo is that even if an error would occur, and the data is not durable, there is a good chance of the fsync being done by the OS eventually if the failure is temporary - data is not necessarily lost, so we might as well flush the other state to avoid having to re-do the syncing. If the failure was permanent, the other write operations (chainstate etc.) would likely fail as well, so nothing gets out of sync in that scenario either.\n\nHowever, one point for this change is that block data is often stored separately (`-blocksdir` option). In this case, it is quite possible that actually corruption occurs if the `blocksdir` fsync fails permanently, but flushes of the chainstate etc. all succeed, and `BLOCK_HAVE_DATA` is set in the blocktree db, while the actual block data is missing.\n\nSo I am Concept ACK" }, { "t": "2026-07-16T17:01:35Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/test/chainstate_write_tests.cpp", "commit": "0f04fbee2f123c37c4b6fbff7dc10f61dd0b5924", "in_reply_to": 3595055608, "text": "`flushed` is tied to the `state` being invalid, it makes more sense to test them in groups, see https://github.com/bitcoin/bitcoin/pull/35729#pullrequestreview-4715961548\n\nIf you insist I can remove the `state` validation here." }, { "t": "2026-08-04T10:24:01Z", "kind": "review_comment", "who": "optout21", "assoc": "CONTRIBUTOR", "path": "src/test/chainstate_write_tests.cpp", "commit": "0f04fbee2f123c37c4b6fbff7dc10f61dd0b5924", "in_reply_to": null, "text": "[quoted text omitted]\n\nUnit test observation: Running all 781 unit tests, they all pass, but a single output line is printed on the output:\n`Error: A fatal internal error occurred, see debug.log for details: Flushing block file to disk failed. This is likely the result of an I/O error.`\nThis is noted as this is the only stdout printout from the whole unit test suite. It is emitted from the new unit test (`chainstate_flush_failure_boundary` -> 2ns call to `FlushStateToDisk` -> `FlushChainstateBlockFile` -> `FlushBlockFile`.)\nAny chance to suppress this?" }, { "t": "2026-08-04T10:28:19Z", "kind": "review_comment", "who": "optout21", "assoc": "CONTRIBUTOR", "path": "src/test/chainstate_write_tests.cpp", "commit": "0f04fbee2f123c37c4b6fbff7dc10f61dd0b5924", "in_reply_to": 3595055608, "text": "[quoted text omitted]\n\nWhy not simply `BOOST_CHECK(flushed); BOOST_CHECK(state.IsValid());` ?" }, { "t": "2026-08-04T10:36:31Z", "kind": "review_comment", "who": "optout21", "assoc": "CONTRIBUTOR", "path": "src/validation.cpp", "commit": "e1a337ee961db4308697de7cddcf4aad9684c7f1", "in_reply_to": null, "text": "[quoted text omitted]\n\nWhy is the `LogWarning` removed? I could not convince myself that the `fatalError()`/`state.Error()` calls include the same logging. Is it possible to preserve the log line?\n\nAt first I though that the added comments refers to the reason of removal of `LogWarning`, but then I realized it explains the usage of `state.Error()` instead of `FatalError()`.\n\nRegarding the comment, `FlushChainstateBlockFile` indeed calls `KernelNotifications::flushError` whenever it returns `false`, but this is handled across two levels (`FlushUndoFile`), which is a bit too brittle." }, { "t": "2026-08-04T10:55:18Z", "kind": "review_comment", "who": "optout21", "assoc": "CONTRIBUTOR", "path": "src/validation.cpp", "commit": "c42d7fe1cf2aee821b4f9931ef94f16e09de8459", "in_reply_to": null, "text": "[quoted text omitted]\n\nWith this early exit, the update of `m_next_write` below is also skipped. It looks like that _may_ cause some unwanted side-effects, so maybe it should be kept (e.g. by storing the return value, skipping the steps to be skipped, and keeping the `if (should_write || m_next_write == ...` block)." }, { "t": "2026-08-04T11:04:43Z", "kind": "review", "who": "optout21", "assoc": "CONTRIBUTOR", "state": "COMMENTED", "commit": "e1a337ee961db4308697de7cddcf4aad9684c7f1", "text": "Concept ACK (e1a337ee961db4308697de7cddcf4aad9684c7f1)\n\nThe change is within `FlushStateToDisk`, and changes how an error from `FlushChainstateBlockFile` is handled -- ignored previously, returned upwards with the change.\n\nBuild & unit tests verfied locally.\n\nThe effectiveness of the new test (i.e., it catches the error) is checked by manually reverting the change.\n\nAnalyis of call pattern of `FlushChainstateBlockFile`: directly called in only one place, the one affected in this change.\nIndirectly it is used in several usecases:\n\n- The chain tip advances (connect/disconnect, including during IBD/reorgs),\n- A new block is accepted to disk,\n- The mempool cache is trimmed after accepting transactions/packages,\n- A user manually prunes via RPC,\n- The coins cache is resized,\n- An assumeutxo snapshot finishes background validation,\n- The node shuts down or an RPC/kernel-API caller explicitly forces a flush.\n\nIt's hard to conceptually assess the high-level effects in all the use cases.\n\nReviewing the steps excluded with the early exit, the update of the `m_next_write` seems uneqivocal, it _may_ have unintended consequences (left comment).\n\nLGTM, left some comments." }, { "t": "2026-08-07T20:59:33Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/test/chainstate_write_tests.cpp", "commit": "0f04fbee2f123c37c4b6fbff7dc10f61dd0b5924", "in_reply_to": 3595055608, "text": "Yes, I could also do that, but that would assume that I find it likely that they will fail independently, and since I don't, I consider it just noise." }, { "t": "2026-08-07T20:59:59Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/test/chainstate_write_tests.cpp", "commit": "0f04fbee2f123c37c4b6fbff7dc10f61dd0b5924", "in_reply_to": 3711449210, "text": "Good point, added `ASSERT_DEBUG_LOG`, which suppresses the expected output and verifies the lower-level error message. It fixes it for me, can you please check it? Added you as coauthor." }, { "t": "2026-08-08T19:01:41Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/validation.cpp", "commit": "e1a337ee961db4308697de7cddcf4aad9684c7f1", "in_reply_to": 3711536457, "text": "The [lower layer](https://github.com/bitcoin/bitcoin/blob/fa615bd163ac74c11a8e15ed8513a5b81ed9ef3b/src/node/blockstorage.cpp#L776) already reports the specific failure through `flushError()` before returning `false`, so restoring this generic warning would duplicate the error. I clarified the comment to make that explicit, let me know if it helps.\n\n[quoted text omitted]" }, { "t": "2026-08-08T19:02:52Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/validation.cpp", "commit": "c42d7fe1cf2aee821b4f9931ef94f16e09de8459", "in_reply_to": 3711657659, "text": "I don't think `m_next_write` should advance after a failed write: the timer update was deliberately moved after the chainstate write in [#30611](https://github.com/bitcoin/bitcoin/pull/30611#discussion_r2069273906). The lower layer has already requested shutdown, and the final `FORCE_FLUSH` during shutdown does not depend on the periodic deadline." }, { "t": "2026-08-08T19:29:06Z", "kind": "force_push", "who": "l0rinc", "commit": "0f4c2a7dd37727cd53b0b8c1e31dbc30a1b57c90" }, { "t": "2026-08-08T19:44:10Z", "kind": "force_push", "who": "l0rinc", "commit": "9d0a5f2b0d91d03166766dc5e57e2490e8c4304a" }, { "t": "2026-08-09T01:13:08Z", "kind": "force_push", "who": "l0rinc", "commit": "c42d7fe1cf2aee821b4f9931ef94f16e09de8459" }, { "t": "2026-08-09T01:14:04Z", "kind": "review", "who": "l0rinc", "assoc": "MEMBER", "state": "COMMENTED", "commit": "c42d7fe1cf2aee821b4f9931ef94f16e09de8459", "text": "Thanks for the review, took me a bit longer to react, I'm juggling a few open ones..." }, { "t": "2026-08-18T08:32:24Z", "kind": "review_comment", "who": "maflcko", "assoc": "MEMBER", "path": "src/test/chainstate_write_tests.cpp", "commit": "0f04fbee2f123c37c4b6fbff7dc10f61dd0b5924", "in_reply_to": 3595055608, "text": "not important, but it looks like https://github.com/bitcoin/bitcoin/pull/35729 was merged, so it could make sense to follow the style used in `src/test/coinscachepair_tests.cpp`. Or, if you really want to group them, wrap them in another pair of `()`. I just mention it to avoid a silent merge conflict with https://github.com/bitcoin/bitcoin/pull/35713 and avoid having to touch this line again after merge." }, { "t": "2026-08-18T17:19:30Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/test/chainstate_write_tests.cpp", "commit": "0f04fbee2f123c37c4b6fbff7dc10f61dd0b5924", "in_reply_to": 3595055608, "text": "I gave this exact line as a concrete example for why we shouldn't just blindly split these conditions https://github.com/bitcoin/bitcoin/pull/35729#pullrequestreview-4715961548\nWhy not fix https://github.com/bitcoin/bitcoin/pull/35713 instead?" }, { "t": "2026-08-19T04:33:52Z", "kind": "review_comment", "who": "optout21", "assoc": "CONTRIBUTOR", "path": "src/validation.cpp", "commit": "c42d7fe1cf2aee821b4f9931ef94f16e09de8459", "in_reply_to": 3711657659, "text": "Thanks, it makes sense." }, { "t": "2026-08-19T04:43:53Z", "kind": "review_comment", "who": "optout21", "assoc": "CONTRIBUTOR", "path": "src/test/chainstate_write_tests.cpp", "commit": "0f04fbee2f123c37c4b6fbff7dc10f61dd0b5924", "in_reply_to": 3595055608, "text": "I see your argument about how unrolling is sub-optimal, but it's a weak one for me.\nIf you insist, the proper solution may be to create a new macro with 2 (or N) conditions, `BOOST_CHECK_ALL`, that evaluates the conditions, and if one fails, prints the values of all. After moving away from BOOST it would be easy to add such a macro." }, { "t": "2026-08-19T04:46:01Z", "kind": "review_comment", "who": "optout21", "assoc": "CONTRIBUTOR", "path": "src/validation.cpp", "commit": "e1a337ee961db4308697de7cddcf4aad9684c7f1", "in_reply_to": 3711536457, "text": "Very good, thanks." }, { "t": "2026-08-19T04:55:21Z", "kind": "review_comment", "who": "optout21", "assoc": "CONTRIBUTOR", "path": "src/test/chainstate_write_tests.cpp", "commit": "0f04fbee2f123c37c4b6fbff7dc10f61dd0b5924", "in_reply_to": 3711449210, "text": "Verified, no more output produced, thanks!" }, { "t": "2026-09-14T11:41:55Z", "kind": "comment", "who": "optout21", "assoc": "CONTRIBUTOR", "text": "ACK c42d7fe1cf2aee821b4f9931ef94f16e09de8459\n\nConcluding my previous review. Reviewed, built and unit-tested locally. Also verified that without the behavior change, the new test fails." } ], "labels_log": [ { "t": "2026-07-13T19:37:33Z", "action": "labeled", "label": "Validation", "who": "DrahtBot" }, { "t": "2026-08-08T19:45:56Z", "action": "labeled", "label": "CI failed", "who": "DrahtBot" }, { "t": "2026-08-08T21:08:20Z", "action": "unlabeled", "label": "CI failed", "who": "DrahtBot" } ], "state_log": [], "text_chars": 10474, "text_tokens_estimate": 2618, "changed_paths": [ "src/test/chainstate_write_tests.cpp", "src/validation.cpp" ], "files": [ { "path": "src/test/chainstate_write_tests.cpp", "add": 26, "del": 0 }, { "path": "src/validation.cpp", "add": 2, "del": 3 } ], "test_lines": 26, "git": { "head": "c42d7fe1cf2aee821b4f9931ef94f16e09de8459", "head_matches_backup": true, "base": "128456b62d5e38abea031f97f823d5b28aef9357", "commits": [ { "sha": "772b872fc0", "subject": "test: characterize writes after flush failure", "files": 1, "add": 26, "del": 0 }, { "sha": "c42d7fe1cf", "subject": "validation: stop writes after flush failure", "files": 2, "add": 4, "del": 5 } ], "patch_truncated": false }, "input_hash": "524ffe5716b7db89", "extracted_at": "2026-09-17T16:15:31+00:00" }