{ "number": 35763, "repo": "bitcoin/bitcoin", "url": "https://github.com/bitcoin/bitcoin/pull/35763", "title": "util: write JSON atomically in WriteJson", "author": "kevkevinpal", "author_association": "CONTRIBUTOR", "created_at": "2026-07-21T12:47:38Z", "updated_at": "2026-09-17T05:50:38Z", "age_days": 58, "draft": false, "labels": [ "Utils/log/libs" ], "milestone": null, "base": "master", "head_sha": "ed55f14404baabc9eaff68d25e50e167a2fafc8f", "head_ref": "banlist-atomic-write-json-unsafe", "head_repo": "kevkevinpal/bitcoin", "head_history": [ { "t": "2026-08-02T22:12:15Z", "sha": "e524b772ccd01551d37177031f60599902cfe99c" }, { "t": "2026-09-07T14:24:46Z", "sha": "0e67947755fd7e607b8c4e28acbe2022c4e1efde" }, { "t": "2026-09-07T21:26:02Z", "sha": "0260a681e1f15d57b0ddf57b8bb539d8ac3dc577" }, { "t": "2026-09-10T22:11:42Z", "sha": "ed55f14404baabc9eaff68d25e50e167a2fafc8f" } ], "additions": 73, "deletions": 24, "changed_files": 7, "commit_count": 1, "size_bucket": "S", "mergeable_state": "clean", "bot": { "drahtbot": { "present": true, "reviews": { "approach_ack": [ { "login": "winterrdog", "url": "https://github.com/bitcoin/bitcoin/pull/35763#pullrequestreview-5134750069" } ], "stale_ack": [ { "login": "Herb-ops", "url": "https://github.com/bitcoin/bitcoin/pull/35763#issuecomment-5103784773" } ] }, "conflicts": [ { "number": 34520, "title": "refactor: Add [[nodiscard]] to functions returning bool+mutable ref", "author": "maflcko" } ] } }, "acks_parsed": { "winterrdog": { "kind": "approach_ack", "hash": null, "t": "2026-09-07T19:18:51Z", "stale": false }, "Herb-ops": { "kind": "ack", "hash": "07ec5af152b5bcff1b74decb9c7b729dff6c2283", "t": "2026-07-28T11:52:57Z", "stale": true } }, "acks_tally": { "ack": 0, "stale_ack": 1, "concept_ack": 0, "approach_ack": 1, "nack": 0, "concept_nack": 0, "approach_nack": 0 }, "reviews": { "approved": 0, "changes_requested": 0, "distinct_reviewers": [ "Herb-ops", "winterrdog" ] }, "signals": { "needs_rebase": false, "ci_failed": false, "mergeable_state": "clean", "last_author_activity": "2026-09-10T22:12:56Z", "last_reviewer_activity": "2026-09-08T18:37:00Z", "last_reviewer": "winterrdog", "author_silent_days": 6, "waiting_on_author_days": 0, "days_since_update": 0 }, "refs": { "mentioned": [], "depends_on": [], "fixes": [], "linked_issues": [], "references": [], "conflicts": [ 34520 ] }, "stack": { "shares_commits_with": [], "based_on": [], "base_for": [] }, "review_paths": [ "src/common/settings.cpp" ], "body": "## Summary\nFollow-up to https://github.com/bitcoin/bitcoin/pull/35384#issuecomment-4750866299\n\n- Renamed `common::WriteSettings` to `common::WriteJson`.\n- `WriteJson` writes to `path + \".tmp\"` and `RenameOver()`s into place, so a failed or interrupted write cannot leave a truncated destination. Both `settings.json` and `banlist.json` use this helper.\n- Added a unit test for the write/read roundtrip, that the `.tmp` is gone after success, that rename failure is detected, and that a second failed write overwrites the same `.tmp` rather than creating another.", "commits": [ { "sha": "ed55f14404baabc9eaff68d25e50e167a2fafc8f", "date": "2026-09-10T22:11:27Z", "message": "util: atomically write banlist.json and settings.json" } ], "timeline": [ { "t": "2026-07-21T17:42:12Z", "kind": "comment", "who": "winterrdog", "assoc": "CONTRIBUTOR", "text": "Concept ACK\n\nThanks for picking these up.\n\ncc: @sys-dev @ryanofsky" }, { "t": "2026-07-28T11:52:57Z", "kind": "comment", "who": "Herb-ops", "assoc": "NONE", "text": "ACK 07ec5af152b5bcff1b74decb9c7b729dff6c2283, with two non-blocking observations:\n\nThe test is useful, but it also passes when CBanDB::Write is changed back to direct writing. Negative control proved this. It therefore does not verify the rename behavior named by the test.\n\nThe generalized WriteJsonUnsafe helper still reports \u201csettings file\u201d errors when writing banlist.json.tmp. \u201cJSON file\u201d or \u201cfile\u201d would be accurate for both callers." }, { "t": "2026-07-28T22:14:37Z", "kind": "comment", "who": "winterrdog", "assoc": "CONTRIBUTOR", "text": "[quoted text omitted]\n\nthese need attention. so, i think they are blocking" }, { "t": "2026-07-28T22:36:30Z", "kind": "comment", "who": "winterrdog", "assoc": "CONTRIBUTOR", "text": "while going through this, another thought came to mind. right now `WriteJsonUnsafe` writes directly to whatever path it is given and relies on the caller to do the tmp-file + `RenameOver()` step. both current callers (`args.cpp` and `addrdb.cpp`) do this correctly, but that guarantee only lives in a comment. nothing stops a future caller from _accidentally_ writing straight to a live file\n\ni was wondering if it is worth making this a bit harder to misuse. two possible approaches came to mind\n\napproach A: add WriteJsonAtomic alongside WriteJsonUnsafe\n\nkeep the low-level helper as-is, but add a wrapper that handles the tmp-file + rename step:\n\n```cpp\n// settings.h\nbool WriteJsonUnsafe(const fs::path& path,\n const std::map& values,\n std::vector& errors);\n\nbool WriteJsonAtomic(const fs::path& dest_path,\n const std::map& values,\n std::vector& errors);\n\n// settings.cpp\nbool WriteJsonAtomic(const fs::path& dest_path,\n const std::map& values,\n std::vector& errors)\n{\n const fs::path path_tmp{dest_path + \".tmp\"};\n if (!WriteJsonUnsafe(path_tmp, values, errors)) {\n return false;\n }\n return RenameOver(path_tmp, dest_path);\n}\n```\n\ncallers become:\n\n```cpp\n// either this\nif (!WriteJsonUnsafe(path_tmp, values, errors)) return false;\nif (!RenameOver(path_tmp, path)) return false;\n\n// or this\nif (!WriteJsonAtomic(path, values, errors)) return false;\n```\n\nthis keeps the current API intact while giving callers a safe helper for the common case. existing callers with custom tmp-file handling can keep using `WriteJsonUnsafe`, while new callers can just use `WriteJsonAtomic`. the downside is that the footgun is still there. `WriteJsonUnsafe` is still publicly available, so a future caller can still accidentally bypass the atomic write path. it also means carrying two helpers that are closely related\n\napproach B: make the helper always write atomically\n\n*this is quite similar to the idea @sys-dev suggested [here](https://github.com/bitcoin/bitcoin/pull/35384#discussion_r3329176108).*\n\ninstead of exposing both APIs, make the helper always write to a tmp file and rename it:\n\n```cpp\n// settings.h\nbool WriteJson(const fs::path& dest_path,\n const std::map& values,\n std::vector& errors);\n```\n\n```cpp\n// settings.cpp\nbool WriteJson(const fs::path& dest_path,\n const std::map& values,\n std::vector& errors)\n{\n SettingsValue out(SettingsValue::VOBJ);\n for (const auto& [key, value] : values) out.pushKVEnd(key, value);\n\n const fs::path path_tmp{dest_path + \".tmp\"};\n\n std::ofstream file{path_tmp.std_path()};\n // ...error handling lives here...\n file << out.write(/*prettyIndent=*/4, /*indentLevel=*/1) << std::endl;\n\n return RenameOver(path_tmp, dest_path);\n}\n```\n\nthis makes the safe path the only path. every caller gets atomic writes by default, so it is no longer possible to accidentally write directly to a live file. the tradeoff is that every caller now gets `.tmp` + `RenameOver()` semantics, even if a future use case genuinely just wants \"_write these bytes to this exact path_\" (for example, a one-off dump or export). it also bakes the tmp-file naming scheme into the helper itself\n\n---\n\n**my thinking:** i lean toward approach B unless there is already a use case for exposing the raw write helper. it feels a bit simpler, and it removes the footgun instead of just documenting it. that said, i could easily be missing a future use case where writing directly to the given path is the better choice\n\nany thoughts ?" }, { "t": "2026-08-02T22:12:15Z", "kind": "force_push", "who": "kevkevinpal", "commit": "e524b772ccd01551d37177031f60599902cfe99c" }, { "t": "2026-09-07T14:24:46Z", "kind": "force_push", "who": "kevkevinpal", "commit": "0e67947755fd7e607b8c4e28acbe2022c4e1efde" }, { "t": "2026-09-07T14:26:06Z", "kind": "comment", "who": "kevkevinpal", "assoc": "CONTRIBUTOR", "text": "@Herb-ops @winterrdog, can you review this again?" }, { "t": "2026-09-07T19:18:51Z", "kind": "review", "who": "winterrdog", "assoc": "CONTRIBUTOR", "state": "COMMENTED", "commit": "0e67947755fd7e607b8c4e28acbe2022c4e1efde", "text": "approach ACK\n\n---\nnow, we can update the PR title & description to match what the current code\nactually does. they still mention `WriteJsonUnsafe` and callers doing the\natomic rename, while the current implementation is self-contained and handles\nthe \"temp write + `RenameOver()`\" itself" }, { "t": "2026-09-07T19:21:21Z", "kind": "review_comment", "who": "winterrdog", "assoc": "CONTRIBUTOR", "path": "src/common/settings.cpp", "commit": "ed55f14404baabc9eaff68d25e50e167a2fafc8f", "in_reply_to": null, "text": "**premise**: cleaning up `path_tmp` in case one of the error paths\nis explored\n\ncurrently, `WriteJson()` returns `false` but leaves the `.tmp` file behind.\nthe new test (`write_uses_rename`) actually has to manually remove\n`banlist_atomic.json.tmp` after exercising the error paths, so it looks\nlike we can leave stale temp files around in cases like persistent permission\nproblems, an antivirus file lock on Windows, or repeated rename failures,\nthese temp files are most likely to accumulate indefinitely\n\nwould a _best-effort_ `fs::remove(path_tmp)` on the failure path make sense\nhere, or do you prefer leaving the temp file around?\n\nif implemented, one of the tradeoffs i can easily think of is the loss of\nforensic evidence when debugging persistent failures.\n\nany thoughts ?" }, { "t": "2026-09-07T21:25:51Z", "kind": "review_comment", "who": "kevkevinpal", "assoc": "CONTRIBUTOR", "path": "src/common/settings.cpp", "commit": "ed55f14404baabc9eaff68d25e50e167a2fafc8f", "in_reply_to": 3952183738, "text": "I prefer leaving it.\n\nThe tmp name is stable (path + \".tmp\"), so a later attempt opens the same file and overwrites it. You get at most one leftover per destination, not an accumulating set. That matches the old settings.json write, which also left the .tmp on rename failure.\n\nThe fs::remove(...tmp) in the test is just fixture cleanup.\n\n---\n\nI can amend the commit to add a test that asserts we don't create endless tmp files" }, { "t": "2026-09-07T21:26:02Z", "kind": "force_push", "who": "kevkevinpal", "commit": "0260a681e1f15d57b0ddf57b8bb539d8ac3dc577" }, { "t": "2026-09-07T21:30:07Z", "kind": "review_comment", "who": "kevkevinpal", "assoc": "CONTRIBUTOR", "path": "src/common/settings.cpp", "commit": "ed55f14404baabc9eaff68d25e50e167a2fafc8f", "in_reply_to": 3952183738, "text": "Added this test\n\nhttps://github.com/bitcoin/bitcoin/pull/35763/changes/0260a681e1f15d57b0ddf57b8bb539d8ac3dc577#diff-8e202c9cf7b0629a40c224318626d23656c0366a1f4b98f8a34ed62f7691a161R73-R81\n\n@winterrdog let me know if this looks good to you" }, { "t": "2026-09-08T18:33:22Z", "kind": "review_comment", "who": "winterrdog", "assoc": "CONTRIBUTOR", "path": "src/common/settings.cpp", "commit": "0260a681e1f15d57b0ddf57b8bb539d8ac3dc577", "in_reply_to": null, "text": ":( this has dragged on for quite some time. the approach and code work fine but i spotted an API naming issue that i think is worth discussing.\n\ncurrently, `WriteSettings` was renamed to `WriteJson`, but [`ReadSettings`](https://github.com/bitcoin/bitcoin/blob/0260a681e1f15d57b0ddf57b8bb539d8ac3dc577/src/common/settings.cpp#L73) kept its old name. so we now have a `WriteJson` / `ReadSettings` pair that operates on the same kind of data, but has different names (asymmetry)\n\nso, i see 3 potential directions:\n\n**A. keep the ****`ReadSettings`**** / ****`WriteSettings`**** pair**\n\nprobably the simplest fix for this PR. these functions are not really generic JSON helpers anyway: they assume a flat `map` and handle the `_warning_` key. the downside is that having `WriteSettings` write to something like `banlist.json` feels a little odd\n\n**B. rename both to a ****`ReadJson`**** / ****`WriteJson`**** pair**\n\nthis fixes the asymmetry with a small change. the downside is that `WriteJson` sounds more generic than it actually is, because it would still have settings-specific behavior like adding the `_warning_` key.\n\n**C. extract the generic JSON functionality from the settings functionality into separate functions**\n\nfor example, have generic `ReadJsonFile` / `WriteJsonFile` helpers underneath, then keep `ReadSettings` / `WriteSettings` as thin wrappers for the settings-specific behavior.\n\nthis seems like the cleanest long-term design if we expect more JSON-backed files to use this code, but probably more work than this PR expected.\n\n---\nany thoughts on this ?" }, { "t": "2026-09-08T18:37:00Z", "kind": "review_comment", "who": "winterrdog", "assoc": "CONTRIBUTOR", "path": "src/common/settings.cpp", "commit": "ed55f14404baabc9eaff68d25e50e167a2fafc8f", "in_reply_to": 3952183738, "text": "[quoted text omitted]\n\nmakes sense. i concur\n\n[quoted text omitted]\nyes! good, it is" }, { "t": "2026-09-10T22:11:42Z", "kind": "force_push", "who": "kevkevinpal", "commit": "ed55f14404baabc9eaff68d25e50e167a2fafc8f" }, { "t": "2026-09-10T22:12:56Z", "kind": "review_comment", "who": "kevkevinpal", "assoc": "CONTRIBUTOR", "path": "src/common/settings.cpp", "commit": "0260a681e1f15d57b0ddf57b8bb539d8ac3dc577", "in_reply_to": 3961058832, "text": "Thank you for the review, and yeah, I agree that it's strange to have them different like that. I pushed [ed55f14](https://github.com/bitcoin/bitcoin/pull/35763/commits/ed55f14404baabc9eaff68d25e50e167a2fafc8f) to do path A.\n\nWe can do a follow-up for C if we want after this is merged." } ], "labels_log": [ { "t": "2026-07-21T12:47:42Z", "action": "labeled", "label": "Utils/log/libs", "who": "DrahtBot" }, { "t": "2026-09-10T23:16:30Z", "action": "labeled", "label": "CI failed", "who": "DrahtBot" }, { "t": "2026-09-17T05:50:38Z", "action": "unlabeled", "label": "CI failed", "who": "DrahtBot" } ], "state_log": [ { "t": "2026-09-07T21:17:56Z", "kind": "renamed", "who": "kevkevinpal", "from": "util: atomically write banlist.json and rename WriteSettings to WriteJsonUnsafe", "to": "util: write JSON atomically in WriteJson" } ], "text_chars": 8884, "text_tokens_estimate": 2221, "changed_paths": [ "src/addrdb.cpp", "src/common/args.cpp", "src/common/args.h", "src/common/settings.cpp", "src/common/settings.h", "src/test/argsman_tests.cpp", "src/test/banman_tests.cpp" ], "files": [ { "path": "src/addrdb.cpp", "add": 6, "del": 7 }, { "path": "src/common/args.cpp", "add": 6, "del": 10 }, { "path": "src/common/args.h", "add": 1, "del": 1 }, { "path": "src/common/settings.cpp", "add": 10, "del": 4 }, { "path": "src/common/settings.h", "add": 5, "del": 1 }, { "path": "src/test/argsman_tests.cpp", "add": 1, "del": 1 }, { "path": "src/test/banman_tests.cpp", "add": 44, "del": 0 } ], "test_lines": 46, "git": { "head": "ed55f14404baabc9eaff68d25e50e167a2fafc8f", "head_matches_backup": true, "base": "18c05d93016b28a9afd4c716dfe00b6e0accb30b", "commits": [ { "sha": "ed55f14404", "subject": "util: atomically write banlist.json and settings.json", "files": 7, "add": 73, "del": 24 } ], "patch_truncated": false }, "input_hash": "754b293a0d09f99f", "extracted_at": "2026-09-17T16:15:31+00:00" }