{ "number": 36586, "repo": "bitcoin/bitcoin", "url": "https://github.com/bitcoin/bitcoin/pull/36586", "title": "coins: warn of low disk space before chainstate compaction", "author": "l0rinc", "author_association": "MEMBER", "created_at": "2026-10-20T21:34:41Z", "updated_at": "2026-11-01T18:49:09Z", "description_hash": "103b9fcc020aa7dd", "age_days": 12, "draft": false, "labels": [ "UTXO Db and Indexes" ], "milestone": null, "base": "master", "head_sha": "2fcfa1dae75ccd11062a6e6073600e67eb20164c", "head_ref": "l0rinc/chainstate-compaction-headroom", "head_repo": "l0rinc/bitcoin", "head_history": [ { "t": "2026-10-21T06:46:01Z", "sha": "f404c06a2cc114bbaab98ff5e21026d710de3f21" }, { "t": "2026-10-22T10:33:05Z", "sha": "ed0f6de8104d67454c0b9547fd19113d4e95abf2" }, { "t": "2026-10-23T20:49:00Z", "sha": "2fcfa1dae75ccd11062a6e6073600e67eb20164c" } ], "additions": 28, "deletions": 14, "changed_files": 6, "commit_count": 3, "size_bucket": "S", "mergeable_state": "clean", "bot": { "drahtbot": { "present": true, "reviews": { "ack": [ { "login": "optout21", "url": "https://github.com/bitcoin/bitcoin/pull/36586#issuecomment-6380160065" }, { "login": "stickies-v", "url": "https://github.com/bitcoin/bitcoin/pull/36586#issuecomment-6413372201" } ], "concept_ack": [ { "login": "sedited", "url": "https://github.com/bitcoin/bitcoin/pull/36586#issuecomment-6309945846" } ] }, "conflicts": [] } }, "acks_parsed": { "sedited": { "kind": "concept_ack", "hash": null, "t": "2026-10-21T06:50:51Z", "stale": false }, "stickies-v": { "kind": "ack", "hash": "2fcfa1dae75ccd11062a6e6073600e67eb20164c", "t": "2026-10-26T10:06:22Z", "stale": false }, "optout21": { "kind": "ack", "hash": "2fcfa1dae75ccd11062a6e6073600e67eb20164c", "t": "2026-10-24T05:03:28Z", "stale": false } }, "acks_tally": { "ack": 2, "stale_ack": 0, "concept_ack": 1, "approach_ack": 0, "nack": 0, "concept_nack": 0, "approach_nack": 0 }, "reviews": { "approved": 1, "changes_requested": 1, "distinct_reviewers": [ "andrewtoth", "optout21", "sedited", "stickies-v" ] }, "signals": { "needs_rebase": false, "ci_failed": false, "mergeable_state": "clean", "last_author_activity": "2026-11-01T17:41:17Z", "last_reviewer_activity": "2026-11-01T18:49:09Z", "last_reviewer": "andrewtoth", "author_silent_days": 0, "waiting_on_author_days": 0, "days_since_update": 0 }, "refs": { "mentioned": [ 36585 ], "depends_on": [], "fixes": [ 36585 ], "linked_issues": [ { "number": 36585, "type": "issue", "state": "open", "merged": false, "merged_at": null, "title": "Background chainstate compaction (#35465) filled the disk; node then hung after shutdown (v32.0rc3)" } ], "references": [ { "number": 36585, "type": "issue", "state": "open", "merged": false, "merged_at": null, "title": "Background chainstate compaction (#35465) filled the disk; node then hung after shutdown (v32.0rc3)" } ], "conflicts": [] }, "stack": { "shares_commits_with": [], "based_on": [], "base_for": [] }, "review_paths": [ "doc/release-notes-36586.md", "src/txdb.cpp", "src/validation.cpp" ], "body": "**Problem:** Full chainstate compaction can temporarily require nearly another copy of the database before old files are deleted.\n\n**Fix:** Log automatic full compaction start and finish at info level. Before starting, check the free space on the chainstate filesystem, and if it is below `m_assumed_chain_state_size`, log a warning and skip this compaction. Skipping avoids the temporary extra disk use but leaves obsolete data taking up space.\n\nFixes #36585", "commits": [ { "sha": "f57a6973da92359e91b95f72905e43bbf8ea5e4b", "date": "2026-10-23T20:37:46Z", "message": "logging: share full compaction logs\n\nAutomatic chainstate compaction can use substantial disk space and take minutes, but its start and finish are hidden unless `-debug=coindb` is enabled.\nLog both at info level in `CDBWrapper::CompactFull()` so forced and automatic compaction share the same messages.\n\nCo-authored-by: optout <13562139+optout21@users.noreply.github.com>" }, { "sha": "cef76701a81fde56ad143b4ea559468a6b4dc626", "date": "2026-10-23T20:45:16Z", "message": "coins: skip compaction if disk space is low\n\nFull chainstate compaction can temporarily need about as much extra disk space as the existing database.\nBefore scheduling it, check the chainstate filesystem using `AssumedChainStateSize()` and the standard free-space reserve, and report available space when deferring compaction.\nDeferring avoids the temporary extra disk use but leaves obsolete data taking up space until the operator frees enough space for cleanup.\nSkip periodic compaction for in-memory chainstates, which do not need to reclaim disk space.\n\nCo-authored-by: stickies-v " }, { "sha": "2fcfa1dae75ccd11062a6e6073600e67eb20164c", "date": "2026-10-23T20:45:16Z", "message": "doc: add chainstate compaction release note\n\nCo-authored-by: stickies-v " } ], "timeline": [ { "t": "2026-10-20T23:01:37Z", "kind": "comment", "who": "andrewtoth", "assoc": "MEMBER", "text": "[quoted text omitted]\n\nWould it make more sense to just skip compaction if we don't think there's enough free disk space to do it? This would have the tradeoff that some extra data would remain, but it would let users like the issue reporter continue running their node for likely several years without any problems.", "id": "c:6303711894", "url": "https://github.com/bitcoin/bitcoin/pull/36586#issuecomment-6303711894", "hash": "975769e83357e27b", "edited": null }, { "t": "2026-10-21T06:46:01Z", "kind": "force_push", "who": "l0rinc", "commit": "f404c06a2cc114bbaab98ff5e21026d710de3f21", "id": "push:f404c06a2cc114bbaab98ff5e21026d710de3f21" }, { "t": "2026-10-21T06:47:01Z", "kind": "comment", "who": "l0rinc", "assoc": "MEMBER", "text": "Thanks @andrewtoth, updated as suggested.", "id": "c:6309885819", "url": "https://github.com/bitcoin/bitcoin/pull/36586#issuecomment-6309885819", "hash": "81140489272c9522", "edited": null }, { "t": "2026-10-21T06:50:51Z", "kind": "comment", "who": "sedited", "assoc": "MEMBER", "text": "Concept ACK", "id": "c:6309945846", "url": "https://github.com/bitcoin/bitcoin/pull/36586#issuecomment-6309945846", "hash": "e8334cc43879b73c", "edited": null }, { "t": "2026-10-21T15:26:16Z", "kind": "comment", "who": "stickies-v", "assoc": "MEMBER", "text": "Concept ACK", "id": "c:6319694104", "url": "https://github.com/bitcoin/bitcoin/pull/36586#issuecomment-6319694104", "hash": "e8334cc43879b73c", "edited": null }, { "t": "2026-10-21T17:03:18Z", "kind": "review_comment", "who": "stickies-v", "assoc": "MEMBER", "path": "src/txdb.cpp", "commit": "a3de7919419b2b7dce2478d0e1a220ec7623ce9b", "in_reply_to": null, "text": "My understanding of the worst case here is that this happens when all the data is on a single level L, with a very sparse (<32MiB) level L-1 above it. Because L-1 is so sparse, all L-1 files (with keys across the entire L level) get compacted in one go, causing the entire dense level L (~the entire database) to be temporarily copied, correct?\n\nIt seems like we're already being more adventurous with our LevelDB usage than we perhaps should be (the `CompactRange` docstring states _\"This operation should typically only be invoked by users who understand the underlying implementation.\"_), but perhaps an alternative approach could be to call `CompactRange` on slices, instead of on the entire (`nullptr`, `nullptr`) range in one go?", "id": "rc:4339626383", "url": "https://github.com/bitcoin/bitcoin/pull/36586#discussion_r4339626383", "hash": "f7752a40fe215582", "edited": null }, { "t": "2026-10-22T10:33:05Z", "kind": "force_push", "who": "l0rinc", "commit": "ed0f6de8104d67454c0b9547fd19113d4e95abf2", "id": "push:ed0f6de8104d67454c0b9547fd19113d4e95abf2" }, { "t": "2026-10-22T10:39:01Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/txdb.cpp", "commit": "a3de7919419b2b7dce2478d0e1a220ec7623ce9b", "in_reply_to": 4339626383, "text": "Thanks for the hint, we should investigate that - I tried compacting smaller key ranges locally, which reduced peak disk usage in some cases but could still produce large spikes, defeating the purpose. Partial compaction could be investigated separately, but it seems fairly complicated, and I don't think the benefit is worth adding that complexity to this PR.", "id": "rc:4347320990", "url": "https://github.com/bitcoin/bitcoin/pull/36586#discussion_r4347320990", "hash": "435fa50ed29d5eb1", "edited": null }, { "t": "2026-10-22T10:39:23Z", "kind": "comment", "who": "l0rinc", "assoc": "MEMBER", "text": "I've separated the free-space check from the database operation and unified the compaction logging, as @optout21 suggested in the original PR.", "id": "c:6338743262", "url": "https://github.com/bitcoin/bitcoin/pull/36586#issuecomment-6338743262", "hash": "140f116c6d67b790", "edited": null }, { "t": "2026-10-22T11:18:44Z", "kind": "review_comment", "who": "optout21", "assoc": "CONTRIBUTOR", "path": "src/validation.cpp", "commit": "694c535312d90e31f5a3c47f4db7cc32cfbb61c0", "in_reply_to": null, "text": "[quoted text omitted]\n\nAdditionally logging here the currently observed available space is marginally helpful (guideline: if a log is a result of a decision based on several values, log each).", "id": "rc:4347675764", "url": "https://github.com/bitcoin/bitcoin/pull/36586#discussion_r4347675764", "hash": "645709b6712e81ff", "edited": "2026-10-22T11:30:55Z" }, { "t": "2026-10-22T11:23:33Z", "kind": "review_comment", "who": "optout21", "assoc": "CONTRIBUTOR", "path": "src/validation.cpp", "commit": "694c535312d90e31f5a3c47f4db7cc32cfbb61c0", "in_reply_to": null, "text": "[quoted text omitted]\n\nCould we add some more explanatory and call-to-action info here? Describing that the uncompacted DB takes up more disk space, and that the user should free up some disk space.\nAlso, I suggest changing the wording \"Skipping\" to \"Deferring\", to suggest that it's not without consequences.", "id": "rc:4347717041", "url": "https://github.com/bitcoin/bitcoin/pull/36586#discussion_r4347717041", "hash": "065c4f7ac9220e8b", "edited": "2026-10-22T11:30:55Z" }, { "t": "2026-10-22T11:30:55Z", "kind": "review", "who": "optout21", "assoc": "CONTRIBUTOR", "state": "COMMENTED", "commit": "ed0f6de8104d67454c0b9547fd19113d4e95abf2", "text": "Concept ACK\nLeft minor comments on the wording of the warning.", "id": "r:5613334538", "url": "https://github.com/bitcoin/bitcoin/pull/36586#pullrequestreview-5613334538", "hash": "ab047927a9ce4c81", "edited": null }, { "t": "2026-10-23T09:09:11Z", "kind": "review_comment", "who": "stickies-v", "assoc": "MEMBER", "path": "src/validation.cpp", "commit": "ed0f6de8104d67454c0b9547fd19113d4e95abf2", "in_reply_to": null, "text": "nit: any reason we're not using `CheckDiskSpace()` here?", "id": "rc:4357924174", "url": "https://github.com/bitcoin/bitcoin/pull/36586#discussion_r4357924174", "hash": "ef39eae0d70ef600", "edited": "2026-10-23T09:31:00Z" }, { "t": "2026-10-23T09:15:03Z", "kind": "review_comment", "who": "stickies-v", "assoc": "MEMBER", "path": "src/validation.cpp", "commit": "ed0f6de8104d67454c0b9547fd19113d4e95abf2", "in_reply_to": null, "text": "For an in-memory database, the compaction memory usage problem is arguably even worse than the on-disk one? Since memory-only is quite ephemeral in nature, I think it's uncontroversial to just skip compaction entirely in that case?", "id": "rc:4357973303", "url": "https://github.com/bitcoin/bitcoin/pull/36586#discussion_r4357973303", "hash": "894d55f23e1103fe", "edited": "2026-10-23T09:31:00Z" }, { "t": "2026-10-23T09:23:39Z", "kind": "review_comment", "who": "stickies-v", "assoc": "MEMBER", "path": "doc/release-notes-36586.md", "commit": "ed0f6de8104d67454c0b9547fd19113d4e95abf2", "in_reply_to": null, "text": "nit: I found the \"may need another database copy\" a bit confusing, and think the rest of the phrasing could be a bit more clear as well. Proposed alternative, if you like:\n\n[quoted text omitted]\n temporarily use about as much extra disk space as the chainstate, so it is\n skipped with a warning if there is not enough free space, leaving obsolete\n data to take up unnecessary disk space.", "id": "rc:4358044237", "url": "https://github.com/bitcoin/bitcoin/pull/36586#discussion_r4358044237", "hash": "5cbca677c1f10257", "edited": "2026-10-23T09:31:00Z" }, { "t": "2026-10-23T09:25:20Z", "kind": "review_comment", "who": "stickies-v", "assoc": "MEMBER", "path": "src/txdb.cpp", "commit": "a3de7919419b2b7dce2478d0e1a220ec7623ce9b", "in_reply_to": 4339626383, "text": "[quoted text omitted]\n\nYeah, I think I agree.", "id": "rc:4358057761", "url": "https://github.com/bitcoin/bitcoin/pull/36586#discussion_r4358057761", "hash": "e20f23b79efb7335", "edited": "2026-10-23T09:31:00Z" }, { "t": "2026-10-23T09:31:00Z", "kind": "review", "who": "stickies-v", "assoc": "MEMBER", "state": "APPROVED", "commit": "ed0f6de8104d67454c0b9547fd19113d4e95abf2", "text": "ACK ed0f6de8104d67454c0b9547fd19113d4e95abf2", "id": "r:5626190512", "url": "https://github.com/bitcoin/bitcoin/pull/36586#pullrequestreview-5626190512", "hash": "50aafe84c2e358db", "edited": null }, { "t": "2026-10-23T20:49:00Z", "kind": "force_push", "who": "l0rinc", "commit": "2fcfa1dae75ccd11062a6e6073600e67eb20164c", "id": "push:2fcfa1dae75ccd11062a6e6073600e67eb20164c" }, { "t": "2026-10-23T20:49:08Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/validation.cpp", "commit": "ed0f6de8104d67454c0b9547fd19113d4e95abf2", "in_reply_to": 4357924174, "text": "Guess the extra 50 MiB reserve is fine here (seemed imprecise to me) - changed.", "id": "rc:4364106218", "url": "https://github.com/bitcoin/bitcoin/pull/36586#discussion_r4364106218", "hash": "fab1c04c88a63bc2", "edited": null }, { "t": "2026-10-23T20:49:17Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/validation.cpp", "commit": "694c535312d90e31f5a3c47f4db7cc32cfbb61c0", "in_reply_to": 4347717041, "text": "Done, thanks", "id": "rc:4364107414", "url": "https://github.com/bitcoin/bitcoin/pull/36586#discussion_r4364107414", "hash": "19bc0c1e9189a04c", "edited": "2026-10-23T20:49:18Z" }, { "t": "2026-10-23T20:49:23Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/validation.cpp", "commit": "694c535312d90e31f5a3c47f4db7cc32cfbb61c0", "in_reply_to": 4347675764, "text": "Done", "id": "rc:4364107987", "url": "https://github.com/bitcoin/bitcoin/pull/36586#discussion_r4364107987", "hash": "11a6767d5674c7e4", "edited": null }, { "t": "2026-10-23T20:49:32Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/validation.cpp", "commit": "ed0f6de8104d67454c0b9547fd19113d4e95abf2", "in_reply_to": 4357973303, "text": "Sure, I was also wondering if someone would comment on bubbling this check up - thanks, done.", "id": "rc:4364109056", "url": "https://github.com/bitcoin/bitcoin/pull/36586#discussion_r4364109056", "hash": "2320cae8bbbe9665", "edited": null }, { "t": "2026-10-23T20:49:36Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "doc/release-notes-36586.md", "commit": "ed0f6de8104d67454c0b9547fd19113d4e95abf2", "in_reply_to": 4358044237, "text": "Did something similar, thanks.", "id": "rc:4364109542", "url": "https://github.com/bitcoin/bitcoin/pull/36586#discussion_r4364109542", "hash": "5e8edaa11f1943db", "edited": null }, { "t": "2026-10-23T20:51:53Z", "kind": "comment", "who": "l0rinc", "assoc": "MEMBER", "text": "Thanks for the reviews, updated to skip automatic in-memory compaction, use `CheckDiskSpace()` with 50 mb additional overhead, include available space and recovery guidance in the warning, and clarify the release note.", "id": "c:6374496215", "url": "https://github.com/bitcoin/bitcoin/pull/36586#issuecomment-6374496215", "hash": "8bb1d00c6d925099", "edited": null }, { "t": "2026-10-24T04:56:47Z", "kind": "review_comment", "who": "optout21", "assoc": "CONTRIBUTOR", "path": "src/validation.cpp", "commit": "ed0f6de8104d67454c0b9547fd19113d4e95abf2", "in_reply_to": 4357924174, "text": "`AssumedChainStateSize()` is also rather imprecise here.", "id": "rc:4366046792", "url": "https://github.com/bitcoin/bitcoin/pull/36586#discussion_r4366046792", "hash": "f4a8573c5e6ceacc", "edited": "2026-10-24T04:56:48Z" }, { "t": "2026-10-24T05:03:28Z", "kind": "comment", "who": "optout21", "assoc": "CONTRIBUTOR", "text": "crACK 2fcfa1dae75ccd11062a6e6073600e67eb20164c", "id": "c:6380160065", "url": "https://github.com/bitcoin/bitcoin/pull/36586#issuecomment-6380160065", "hash": "60186fd9b116949c", "edited": null }, { "t": "2026-10-26T10:06:22Z", "kind": "comment", "who": "stickies-v", "assoc": "MEMBER", "text": "ACK 2fcfa1dae75ccd11062a6e6073600e67eb20164c", "id": "c:6413372201", "url": "https://github.com/bitcoin/bitcoin/pull/36586#issuecomment-6413372201", "hash": "36ff6b135b3b7427", "edited": null }, { "t": "2026-10-31T19:46:01Z", "kind": "review_comment", "who": "andrewtoth", "assoc": "MEMBER", "path": "src/validation.cpp", "commit": "2fcfa1dae75ccd11062a6e6073600e67eb20164c", "in_reply_to": null, "text": "I'm not sure we should warn about this. Running on a system with < 2x the chainstate disk space should be supported. If that is the case there is nothing the user can do, and they shouldn't be getting warnings about it.", "id": "rc:4433066699", "url": "https://github.com/bitcoin/bitcoin/pull/36586#discussion_r4433066699", "hash": "5e4cc19dc34b6940", "edited": "2026-10-31T19:48:23Z" }, { "t": "2026-10-31T19:48:09Z", "kind": "review_comment", "who": "andrewtoth", "assoc": "MEMBER", "path": "doc/release-notes-36586.md", "commit": "2fcfa1dae75ccd11062a6e6073600e67eb20164c", "in_reply_to": null, "text": "I don't think this change warrants a release note.\nIf others disagree, at least the last part of this line should be struck.\"obsolete data\" is not something a user reading the release notes will understand.", "id": "rc:4433072777", "url": "https://github.com/bitcoin/bitcoin/pull/36586#discussion_r4433072777", "hash": "993d552032199292", "edited": "2026-10-31T19:48:23Z" }, { "t": "2026-10-31T19:48:23Z", "kind": "review", "who": "andrewtoth", "assoc": "MEMBER", "state": "CHANGES_REQUESTED", "commit": "2fcfa1dae75ccd11062a6e6073600e67eb20164c", "text": "", "id": "r:5724645351", "url": "https://github.com/bitcoin/bitcoin/pull/36586#pullrequestreview-5724645351", "hash": "e3b0c44298fc1c14", "edited": null }, { "t": "2026-11-01T17:40:41Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "src/validation.cpp", "commit": "2fcfa1dae75ccd11062a6e6073600e67eb20164c", "in_reply_to": 4433066699, "text": "[quoted text omitted]\n\nWe can't expect a node to run without compactions, so we have to let the user know that it's a temporary allowance and that they should do something about it: relocate to a bigger disk, delete some files, prune harder, etc.", "id": "rc:4436676560", "url": "https://github.com/bitcoin/bitcoin/pull/36586#discussion_r4436676560", "hash": "387e0d4ee0cd0c7d", "edited": "2026-11-01T17:41:27Z" }, { "t": "2026-11-01T17:41:17Z", "kind": "review_comment", "who": "l0rinc", "assoc": "MEMBER", "path": "doc/release-notes-36586.md", "commit": "2fcfa1dae75ccd11062a6e6073600e67eb20164c", "in_reply_to": 4433072777, "text": "Do they need to understand more than garbage collection needs a bit more space?", "id": "rc:4436679534", "url": "https://github.com/bitcoin/bitcoin/pull/36586#discussion_r4436679534", "hash": "23452cad09398ca3", "edited": "2026-11-01T17:41:27Z" }, { "t": "2026-11-01T18:49:09Z", "kind": "review_comment", "who": "andrewtoth", "assoc": "MEMBER", "path": "src/validation.cpp", "commit": "2fcfa1dae75ccd11062a6e6073600e67eb20164c", "in_reply_to": 4433066699, "text": "[quoted text omitted]\n\nWhy not? From my understanding the compactions are just a way to reduce disk usage in the chainstate db, but don't affect node operation otherwise.\n\nRight now the chainstate db is ~11 GB. With pruning we can run it on an instance with < 20 GB of disk space. In this scenario the user will get a warning for every attempted compaction. What is the benefit of having this user relocate to a bigger disk, just to save disk space? Is there a way to determine the total disk size and skip warning if it is < 2x chainstate size?", "id": "rc:4436896379", "url": "https://github.com/bitcoin/bitcoin/pull/36586#discussion_r4436896379", "hash": "93eeb8486cddd2bb", "edited": null } ], "labels_log": [ { "t": "2026-10-20T21:34:44Z", "action": "labeled", "label": "UTXO Db and Indexes", "who": "DrahtBot" } ], "state_log": [], "text_chars": 6459, "text_tokens_estimate": 1614, "changed_paths": [ "doc/release-notes-36586.md", "src/dbwrapper.cpp", "src/dbwrapper.h", "src/txdb.cpp", "src/txdb.h", "src/validation.cpp" ], "files": [ { "path": "doc/release-notes-36586.md", "add": 6, "del": 0 }, { "path": "src/dbwrapper.cpp", "add": 9, "del": 8 }, { "path": "src/dbwrapper.h", "add": 2, "del": 2 }, { "path": "src/txdb.cpp", "add": 0, "del": 2 }, { "path": "src/txdb.h", "add": 2, "del": 0 }, { "path": "src/validation.cpp", "add": 9, "del": 2 } ], "test_lines": 0, "git": { "head": "2fcfa1dae75ccd11062a6e6073600e67eb20164c", "head_matches_backup": true, "base": "df0dc1e2d2059213e49a9f115fd55a9f0119d26c", "patch_id": "f6c4f490f42b94ad", "commits": [ { "sha": "f57a6973da", "subject": "logging: share full compaction logs", "files": 3, "add": 11, "del": 12 }, { "sha": "cef76701a8", "subject": "coins: skip compaction if disk space is low", "files": 2, "add": 11, "del": 2 }, { "sha": "2fcfa1dae7", "subject": "doc: add chainstate compaction release note", "files": 1, "add": 6, "del": 0 } ], "patch_truncated": false }, "patch_id": "f6c4f490f42b94ad", "input_hash": "90a07bd8fb370ce1", "extracted_at": "2026-11-02T01:09:49+00:00" }