{
 "number": 31888,
 "repo": "bitcoin/bitcoin",
 "url": "https://github.com/bitcoin/bitcoin/pull/31888",
 "title": "contrib: Support re-writing linearize-data dumps",
 "author": "midnightmagic",
 "author_association": "CONTRIBUTOR",
 "created_at": "2025-02-17T00:32:35Z",
 "updated_at": "2026-06-19T13:15:08Z",
 "age_days": 577,
 "draft": false,
 "labels": [
  "Scripts and tools"
 ],
 "milestone": null,
 "base": "master",
 "head_sha": "e9dc16194b47921ea6c5453e71c5c97b9232ba8c",
 "head_ref": "fix-linearize-gjpyn",
 "head_repo": "midnightmagic/bitcoin",
 "head_history": [
  {
   "t": "2025-02-19T09:54:51Z",
   "sha": "923218bda44c1b616bb4292364786552bb41035c"
  },
  {
   "t": "2025-02-19T09:58:26Z",
   "sha": "84b039c404d68877c4f04d21aa5c479bddf2e51d"
  },
  {
   "t": "2025-02-19T10:03:31Z",
   "sha": "e9dc16194b47921ea6c5453e71c5c97b9232ba8c"
  }
 ],
 "additions": 6,
 "deletions": 1,
 "changed_files": 1,
 "commit_count": 1,
 "size_bucket": "S",
 "mergeable_state": "blocked",
 "bot": {
  "drahtbot": {
   "present": true,
   "reviews": {
    "ack": [
     {
      "login": "sedited",
      "url": "https://github.com/bitcoin/bitcoin/pull/31888#pullrequestreview-4533117386"
     }
    ]
   },
   "conflicts": []
  }
 },
 "acks_parsed": {
  "sedited": {
   "kind": "ack",
   "hash": "e9dc16194b47921ea6c5453e71c5c97b9232ba8c",
   "t": "2026-06-19T13:15:04Z",
   "stale": false
  }
 },
 "acks_tally": {
  "ack": 1,
  "stale_ack": 0,
  "concept_ack": 0,
  "approach_ack": 0,
  "nack": 0,
  "concept_nack": 0,
  "approach_nack": 0
 },
 "reviews": {
  "approved": 1,
  "changes_requested": 0,
  "distinct_reviewers": [
   "bitcoin31888",
   "laanwj",
   "maflcko",
   "sedited",
   "theStack"
  ]
 },
 "signals": {
  "needs_rebase": false,
  "ci_failed": false,
  "mergeable_state": "blocked",
  "last_author_activity": "2026-03-05T20:32:40Z",
  "last_reviewer_activity": "2026-06-19T13:15:04Z",
  "last_reviewer": "sedited",
  "author_silent_days": 195,
  "waiting_on_author_days": 90,
  "days_since_update": 90
 },
 "refs": {
  "mentioned": [],
  "depends_on": [],
  "fixes": [],
  "linked_issues": [],
  "references": [],
  "conflicts": []
 },
 "stack": {
  "shares_commits_with": [],
  "based_on": [],
  "base_for": []
 },
 "review_paths": [],
 "body": "The linearize-data contrib script does not quite work correctly in the case of a chain reorg, and it fully-truncates files with an incorrect open mode.\n\nThis small patch clips off incorrect trailing data and corrects file open mode type.\n\n(small patch being passed around which I did not write but thought would be helpful to the community)",
 "commits": [
  {
   "sha": "e9dc16194b47921ea6c5453e71c5c97b9232ba8c",
   "date": "2025-02-19T10:03:15Z",
   "message": "correct the linearize-data script (minor)\n\nthis clips off unwanted final data in event of reorg and fixes the file open\nmode to prevent the truncation of files reopened\n\nsmall patch being passed around which I did not write but thought would be\nhelpful to the community at large"
  }
 ],
 "timeline": [
  {
   "t": "2025-02-17T11:42:47Z",
   "kind": "comment",
   "who": "laanwj",
   "assoc": "MEMBER",
   "text": "Lint is failing on the formatting of the commit message:\n```\n[10:12:47.126] The subject line of commit hash 0cf4a6245d2bc8d57f91dd2b4f5bd5a0eed18ed4 is followed by a non-empty line. Subject lines should always be followed by a blank line.\n[10:12:47.126] ^^^\n[10:12:47.126]\n[10:12:47.126] ^---- \u26a0\ufe0f Failure generated from lint check 'commit_msg' (Check that commit messages have a new line before the body or no body at all.)!\n```\n\nIt also looks like the changes somehow break the `feature_loadblock.py` test, which uses the linearize script"
  },
  {
   "t": "2025-02-19T09:54:51Z",
   "kind": "force_push",
   "who": "midnightmagic",
   "commit": "923218bda44c1b616bb4292364786552bb41035c"
  },
  {
   "t": "2025-02-19T09:58:26Z",
   "kind": "force_push",
   "who": "midnightmagic",
   "commit": "84b039c404d68877c4f04d21aa5c479bddf2e51d"
  },
  {
   "t": "2025-02-19T09:59:55Z",
   "kind": "comment",
   "who": "maflcko",
   "assoc": "MEMBER",
   "text": "It would be good to include a test, or modify the existing one. Otherwise, it will be harder to test and see what behavior actually changed and future changes could break the fix again."
  },
  {
   "t": "2025-02-19T10:03:31Z",
   "kind": "force_push",
   "who": "midnightmagic",
   "commit": "e9dc16194b47921ea6c5453e71c5c97b9232ba8c"
  },
  {
   "t": "2025-03-12T14:12:03Z",
   "kind": "comment",
   "who": "theStack",
   "assoc": "CONTRIBUTOR",
   "text": "Can you give more context on what problem this PR is trying to solve? Looking at the description and the diff I can't follow, e.g. it's unclear to me what an \"incorrect open mode\" is."
  },
  {
   "t": "2025-04-23T05:01:32Z",
   "kind": "comment",
   "who": "midnightmagic",
   "assoc": "CONTRIBUTOR",
   "text": "The changes function to make linearize \"less-perturb\" on-disk data. Altogether, in the event of a block reorganization, the reorg'd blocks are sometimes shorter than prior blocks' data in individual output files when split on e.g. size. When the writer potentially moves on to the next blk data file therefore, the rest of the block data that pre-existed the reorganization might need to be truncated off the end of that file before moving on to the next one.\n\nFor the \"wb\" Python open mode, all pre-existing block files as it writes out blk* files are destroyed and re-created by an open with \"wb\", which in python will auto-truncate any pre-existing file to 0 bytes\u2014for many large files, something pulling data from the source will always see rapidly-changing source dataset all the time.\n\nWith the change, the attempt to verify that the file doesn't pre-exist and then re-write in-place from 0 will be much less likely to break anything trying to read the files, most of them will effectively never change, and instead we just get the (much less likely) chance of a reorg write racing a potential midstream read, plus the much less-likely truncate-eliminating-a-bit-of-end-file data.\n\nIt's not really perfect when considered in conjunction with other tools, but it's much better than the current behaviour. Probably some form of atomic file modification would be \"better\" in the sense of guaranteeing on-disk state consistency, or, perhaps a trigger to external tools to identify \"current\" consistent state, but in the meanwhile I can't really see this change hurting because of how shallow reorgs usually are and the inordinate size of the dataset we have to manipulate.\n\nThe use case is for people who repeatedly re-run linearize and make use of the results on an ongoing basis."
  },
  {
   "t": "2025-05-19T02:57:48Z",
   "kind": "comment",
   "who": "bitcoin31888",
   "assoc": "NONE",
   "text": "Consider the following bash script\n\n`while true;do ./linearize-hashes.py linearize.cfg > hashlist.txt; ./linearize-data.py linearize.cfg; sleep 120; done`\n\nThis will continuously write out the bootstrap files, without this patch the files will continuously be truncated to zero length and then re-written, that behavior is undesirable."
  },
  {
   "t": "2026-03-04T07:54:55Z",
   "kind": "comment",
   "who": "sedited",
   "assoc": "CONTRIBUTOR",
   "text": "There's not been a response from the author to a review comment [here](https://github.com/bitcoin/bitcoin/pull/31888#issuecomment-2889473592), in quite some time. There has also been no further review. Maybe this should be closed and marked as up for grabs?"
  },
  {
   "t": "2026-03-04T08:50:13Z",
   "kind": "comment",
   "who": "maflcko",
   "assoc": "MEMBER",
   "text": "My reading from https://github.com/bitcoin/bitcoin/pull/31888#issuecomment-2889473592 seems to give an example of why this pull is needed. However, I haven't looked closely and reviewed/tested this myself"
  },
  {
   "t": "2026-03-04T09:11:25Z",
   "kind": "comment",
   "who": "sedited",
   "assoc": "CONTRIBUTOR",
   "text": "[quoted text omitted]\n\nHuh, yes, I did read that the wrong way around. Will take a look myself then :)"
  },
  {
   "t": "2026-03-04T12:03:07Z",
   "kind": "review",
   "who": "sedited",
   "assoc": "CONTRIBUTOR",
   "state": "COMMENTED",
   "commit": "e9dc16194b47921ea6c5453e71c5c97b9232ba8c",
   "text": "I'm not sure about this patch being framed as a correction of a wrong assumption. As described in a later comment, this adds support for writing to the same set of files again. I can see why that might be useful, but the description both for the PR and the commit should clarify this. Would also be good to capture this behaviour in the README."
  },
  {
   "t": "2026-03-04T19:21:59Z",
   "kind": "comment",
   "who": "midnightmagic",
   "assoc": "CONTRIBUTOR",
   "text": "The use case described in that comment is precisely demonstrating a logical outcome of the regressive, drive-destroying behaviour for anyone who wants to use this script more than once, so, I'd thought it was a good example of what the goal of the diff is and didn't need comment. Does it need further comment? :-)"
  },
  {
   "t": "2026-03-05T09:36:45Z",
   "kind": "comment",
   "who": "sedited",
   "assoc": "CONTRIBUTOR",
   "text": "[quoted text omitted]\n\nTo be clear, I agree it is a desirable thing to patch and the examples in the discussion here clarify that. I'm just saying that this should be clearly described in the pull request description and the commit message."
  },
  {
   "t": "2026-03-05T20:32:40Z",
   "kind": "comment",
   "who": "midnightmagic",
   "assoc": "CONTRIBUTOR",
   "text": "Sure, I can flesh it out more if you like and make it more explicit."
  },
  {
   "t": "2026-03-09T13:13:28Z",
   "kind": "comment",
   "who": "sedited",
   "assoc": "CONTRIBUTOR",
   "text": "[quoted text omitted]\n\nYes, that's all that's required to make progress here."
  },
  {
   "t": "2026-04-18T20:04:33Z",
   "kind": "comment",
   "who": "sedited",
   "assoc": "CONTRIBUTOR",
   "text": "@midnightmagic , would be good to get this in finally, but I'd still like the description to say explicitly that this change targets behaviour around re-writing the files."
  },
  {
   "t": "2026-06-19T13:15:04Z",
   "kind": "review",
   "who": "sedited",
   "assoc": "CONTRIBUTOR",
   "state": "APPROVED",
   "commit": "e9dc16194b47921ea6c5453e71c5c97b9232ba8c",
   "text": "ACK e9dc16194b47921ea6c5453e71c5c97b9232ba8c\n\nSome of the reasons stated here don't make much sense to me, i.e. this doesn't seem to materially change disk wear. But I don't think this hurts existing users and if it can make things easier for the few upstream users this script has, that seems like a good thing."
  }
 ],
 "labels_log": [
  {
   "t": "2025-02-17T01:22:08Z",
   "action": "labeled",
   "label": "CI failed",
   "who": "DrahtBot"
  },
  {
   "t": "2025-02-17T11:37:28Z",
   "action": "labeled",
   "label": "Scripts and tools",
   "who": "laanwj"
  },
  {
   "t": "2025-02-19T11:32:22Z",
   "action": "unlabeled",
   "label": "CI failed",
   "who": "DrahtBot"
  }
 ],
 "state_log": [
  {
   "t": "2026-05-22T09:59:58Z",
   "kind": "renamed",
   "who": "sedited",
   "from": "correct wrong assumptions in the contrib linearize data script",
   "to": "contrib: Support running linearize-data multiple times"
  },
  {
   "t": "2026-05-22T10:05:59Z",
   "kind": "renamed",
   "who": "sedited",
   "from": "contrib: Support running linearize-data multiple times",
   "to": "contrib: Support re-generating linearize-data dumps"
  },
  {
   "t": "2026-06-19T13:03:44Z",
   "kind": "renamed",
   "who": "sedited",
   "from": "contrib: Support re-generating linearize-data dumps",
   "to": "contrib: Support re-writing linearize-data dumps"
  }
 ],
 "text_chars": 5799,
 "text_tokens_estimate": 1449,
 "changed_paths": [
  "contrib/linearize/linearize-data.py"
 ],
 "files": [
  {
   "path": "contrib/linearize/linearize-data.py",
   "add": 6,
   "del": 1
  }
 ],
 "test_lines": 7,
 "git": {
  "head": "e9dc16194b47921ea6c5453e71c5c97b9232ba8c",
  "head_matches_backup": true,
  "base": "d6fe5b28dff4c512bad7c9576232d6def10608d4",
  "commits": [
   {
    "sha": "e9dc16194b",
    "subject": "correct the linearize-data script (minor)",
    "files": 1,
    "add": 6,
    "del": 1
   }
  ],
  "patch_truncated": false
 },
 "input_hash": "6aa689bfc1113ade",
 "extracted_at": "2026-09-17T16:15:31+00:00"
}