{
 "number": 36121,
 "repo": "bitcoin/bitcoin",
 "url": "https://github.com/bitcoin/bitcoin/pull/36121",
 "title": "fuzz: Corpus sharding in parallel runs",
 "author": "maflcko",
 "author_association": "MEMBER",
 "created_at": "2026-08-29T14:07:05Z",
 "updated_at": "2026-09-01T15:33:12Z",
 "age_days": 19,
 "draft": true,
 "labels": [
  "Fuzzing"
 ],
 "milestone": null,
 "base": "master",
 "head_sha": "633456aa8b7bcf3b4d0a5bd4720bd6cf023b8379",
 "head_ref": "2608-fuzz-sharding",
 "head_repo": "maflcko/bitcoin-core",
 "head_history": [
  {
   "t": "2026-09-01T09:04:13Z",
   "sha": "fa2d43f7eabbbb6b052c5c5d484bbff13452fd81"
  }
 ],
 "additions": 70,
 "deletions": 29,
 "changed_files": 2,
 "commit_count": 2,
 "size_bucket": "S",
 "mergeable_state": "clean",
 "bot": {
  "drahtbot": {
   "present": true,
   "reviews": {
    "concept_ack": [
     {
      "login": "jeanpablojp",
      "url": "https://github.com/bitcoin/bitcoin/pull/36121#pullrequestreview-5067237541"
     }
    ]
   },
   "conflicts": [
    {
     "number": 35417,
     "title": "ci: isolate container test networks",
     "author": "willcl-ark"
    }
   ]
  }
 },
 "acks_parsed": {
  "jeanpablojp": {
   "kind": "concept_ack",
   "hash": null,
   "t": "2026-08-31T13:47:21Z",
   "stale": false
  }
 },
 "acks_tally": {
  "ack": 0,
  "stale_ack": 0,
  "concept_ack": 1,
  "approach_ack": 0,
  "nack": 0,
  "concept_nack": 0,
  "approach_nack": 0
 },
 "reviews": {
  "approved": 0,
  "changes_requested": 0,
  "distinct_reviewers": [
   "jeanpablojp"
  ]
 },
 "signals": {
  "needs_rebase": false,
  "ci_failed": false,
  "mergeable_state": "clean",
  "last_author_activity": "2026-09-01T12:55:38Z",
  "last_reviewer_activity": "2026-09-01T15:28:38Z",
  "last_reviewer": "jeanpablojp",
  "author_silent_days": 16,
  "waiting_on_author_days": 16,
  "days_since_update": 16
 },
 "refs": {
  "mentioned": [],
  "depends_on": [],
  "fixes": [],
  "linked_issues": [],
  "references": [],
  "conflicts": [
   35417
  ]
 },
 "stack": {
  "shares_commits_with": [],
  "based_on": [],
  "base_for": []
 },
 "review_paths": [
  "test/fuzz/test_runner.py"
 ],
 "body": "Currently, the fuzz runner accepts a `--par` option to schedule fuzz runs in parallel. This is fine. However, when only a single target is selected, `--par` will not speed up the run. Moreover, when multiple targets are selected, the run-time of the longest target dominates.\n\nFix both issues by splitting the corpus to into `--par` equal-sized shards by default. The sharding can be disabled, if needed.\n\nCan be tested e.g. by running a single target:\n\n```\ntime ./bld-cmake/test/fuzz/test_runner.py --par 1  -l DEBUG ./qa-assets/fuzz_corpora/ utxo_snapshot  # slow\ntime ./bld-cmake/test/fuzz/test_runner.py --par 99 -l DEBUG ./qa-assets/fuzz_corpora/ utxo_snapshot  # fast",
 "commits": [
  {
   "sha": "fa2d43f7eabbbb6b052c5c5d484bbff13452fd81",
   "date": "2026-09-01T09:00:29Z",
   "message": "fuzz: Corpus sharding in parallel runs\n\nTo better utilize the given --par option, split each corpus into shards by default.\n\nThe diff can be reviewed via --ignore-all-space"
  },
  {
   "sha": "633456aa8b7bcf3b4d0a5bd4720bd6cf023b8379",
   "date": "2026-09-01T11:23:31Z",
   "message": "ci: tmpfs"
  }
 ],
 "timeline": [
  {
   "t": "2026-08-29T15:35:39Z",
   "kind": "comment",
   "who": "maflcko",
   "assoc": "MEMBER",
   "text": "Hmm,\n\n* `macOS native, fuzz` seems 3 min slower than master?? (https://github.com/bitcoin/bitcoin/actions/runs/33243456910/job/99076657343 vs https://github.com/bitcoin/bitcoin/actions/runs/33256746608/job/99111704437?pr=36121#logs)\n* asan is 9 min faster (nice) (https://github.com/bitcoin/bitcoin/actions/runs/33243456910/job/99076657348 vs https://github.com/bitcoin/bitcoin/actions/runs/33256746608/job/99111704584?pr=36121)\n* msan is 15 min faster (https://github.com/bitcoin/bitcoin/actions/runs/33243456910/job/99076657192 vs https://github.com/bitcoin/bitcoin/actions/runs/33256746608/job/99111704553?pr=36121)\n* Windows is 7 min faster (https://github.com/bitcoin/bitcoin/actions/runs/33243456910/job/99076657181 vs https://github.com/bitcoin/bitcoin/actions/runs/33256746608/job/99111704453?pr=36121)"
  },
  {
   "t": "2026-08-31T13:47:21Z",
   "kind": "review",
   "who": "jeanpablojp",
   "assoc": "CONTRIBUTOR",
   "state": "COMMENTED",
   "commit": "d0000027dbecbdf553258a25be8a30a13787852c",
   "text": "Concept ACK\n\nI had a look at the logs. Looks like on macOS the pool was already saturated. Summing the process times and dividing by `--par` gives 613s there, against 200s for the slowest single target, so there was no tail to cut. On Windows and MSan it's the other way round, and that is where the win comes from.\n\nAlmost all of the macOS side comes from one target. `utxo_total_supply` builds a `ChainTestingSetup` inside the target body, so a datadir per input, and three shards of 680 take around 240s each against 200s for all 2040 in one process. Same on Windows.\n\nIs it worth gating this on something, or does the win on the other jobs already outweigh it?"
  },
  {
   "t": "2026-08-31T13:47:21Z",
   "kind": "review_comment",
   "who": "jeanpablojp",
   "assoc": "CONTRIBUTOR",
   "path": "test/fuzz/test_runner.py",
   "commit": "633456aa8b7bcf3b4d0a5bd4720bd6cf023b8379",
   "in_reply_to": null,
   "text": "nit: `--help` ends up with two defaults, the one in the help string and a `(default: None)` that `ArgumentDefaultsHelpFormatter` appends."
  },
  {
   "t": "2026-08-31T13:47:21Z",
   "kind": "review_comment",
   "who": "jeanpablojp",
   "assoc": "CONTRIBUTOR",
   "path": "test/fuzz/test_runner.py",
   "commit": "d0000027dbecbdf553258a25be8a30a13787852c",
   "in_reply_to": null,
   "text": "This gets created before we know whether there will be any sharding, so with a read-only parent the run dies in `mkdtemp`, and `--corpus-shards 1` doesn't avoid it. It also survives a `SIGTERM`, left next to the corpus.\n\nAnd since the `with` wraps the collection loop too, the removal only happens after the last job, with everything else idle. 31s on macOS."
  },
  {
   "t": "2026-08-31T13:47:21Z",
   "kind": "review_comment",
   "who": "jeanpablojp",
   "assoc": "CONTRIBUTOR",
   "path": "test/fuzz/test_runner.py",
   "commit": "d0000027dbecbdf553258a25be8a30a13787852c",
   "in_reply_to": null,
   "text": "This hits every target that has a corpus, none has fewer than three inputs. `cmpctblock` sometimes does the same kind of per-input work through `ResetChainmanAndMempool`.\n\nThe hard links are one per corpus input, so that part doesn't shrink with more workers. On macOS that takes 155s before the first job gets collected."
  },
  {
   "t": "2026-08-31T13:47:21Z",
   "kind": "review_comment",
   "who": "jeanpablojp",
   "assoc": "CONTRIBUTOR",
   "path": "test/fuzz/test_runner.py",
   "commit": "d0000027dbecbdf553258a25be8a30a13787852c",
   "in_reply_to": null,
   "text": "nit: this path is already gone by the time anyone reads the log, it's inside the shard dir. The file name survives, the path doesn't.\n\nAnd with the `sys.exit(1)` in here, the dir disappears under the shards that are still running, so on the standalone builds they die on the `read_file` `Assert` instead of on the input that actually broke."
  },
  {
   "t": "2026-08-31T13:47:21Z",
   "kind": "review_comment",
   "who": "jeanpablojp",
   "assoc": "CONTRIBUTOR",
   "path": "test/fuzz/test_runner.py",
   "commit": "d0000027dbecbdf553258a25be8a30a13787852c",
   "in_reply_to": null,
   "text": "`Summary` goes from 237 lines to 3754 on your asan run, and `cov:` becomes per process. `coins_view_db` reports sixteen values between 12033 and 13709 where master reports 14509, and since it's a union there's no way to put it back together. So you can't compare coverage across a change any more. Could the shards be folded back into one line per target?"
  },
  {
   "t": "2026-09-01T09:04:13Z",
   "kind": "force_push",
   "who": "maflcko",
   "commit": "fa2d43f7eabbbb6b052c5c5d484bbff13452fd81"
  },
  {
   "t": "2026-09-01T09:04:19Z",
   "kind": "review_comment",
   "who": "maflcko",
   "assoc": "MEMBER",
   "path": "test/fuzz/test_runner.py",
   "commit": "d0000027dbecbdf553258a25be8a30a13787852c",
   "in_reply_to": 3895026591,
   "text": "I don't think it is possible to recover the coverage from the shards. I can only see compiling with coverage instrumentation and then using that. This could possibly also help with https://github.com/bitcoin-core/qa-assets/issues/166"
  },
  {
   "t": "2026-09-01T09:04:21Z",
   "kind": "review_comment",
   "who": "maflcko",
   "assoc": "MEMBER",
   "path": "test/fuzz/test_runner.py",
   "commit": "d0000027dbecbdf553258a25be8a30a13787852c",
   "in_reply_to": 3895026588,
   "text": "Hmm, the file disappearing should be fine, because the basename of the file (sha1 hash) should be important/sufficient.\n\nThe `read_file` assertions aren't great. I guess this is fixed by having per-`job` tmp dirs for the shards."
  },
  {
   "t": "2026-09-01T09:04:22Z",
   "kind": "review_comment",
   "who": "maflcko",
   "assoc": "MEMBER",
   "path": "test/fuzz/test_runner.py",
   "commit": "d0000027dbecbdf553258a25be8a30a13787852c",
   "in_reply_to": 3895026583,
   "text": "Hmm, I guess this is a combination of some OSs not having a tmpfs at all and the GHA VMs having a slow storage. :("
  },
  {
   "t": "2026-09-01T09:04:24Z",
   "kind": "review_comment",
   "who": "maflcko",
   "assoc": "MEMBER",
   "path": "test/fuzz/test_runner.py",
   "commit": "d0000027dbecbdf553258a25be8a30a13787852c",
   "in_reply_to": 3895026578,
   "text": "read-only parent should be rare, but I think I fixed this in the latest push by having a temp dir for each `job`.\n\nSIGTERM will probably still leave them around.\n\nThe creation/removal is now interleaved, but I don't expect this to improve the runtime, unless there is over-subscription, but then there could be even more storage thrashing..."
  },
  {
   "t": "2026-09-01T09:04:25Z",
   "kind": "review_comment",
   "who": "maflcko",
   "assoc": "MEMBER",
   "path": "test/fuzz/test_runner.py",
   "commit": "633456aa8b7bcf3b4d0a5bd4720bd6cf023b8379",
   "in_reply_to": 3895026569,
   "text": "Yeah, saw that too. Not sure how to fix. I guess it is fine to leave as-is."
  },
  {
   "t": "2026-09-01T09:04:56Z",
   "kind": "comment",
   "who": "maflcko",
   "assoc": "MEMBER",
   "text": "[quoted text omitted]\n\nHmm, if the slowest single target happened to be scheduled toward the end, there would be a tail to cut, no?\n\n[quoted text omitted]\nHmm, I guess this means the bottleneck is storage IO and not CPU, so this pull request won't help and may even make it worse due to storage thrashing.\n\nIn theory, all of the fuzzing should be in a tmpfs (default on modern Linux), but neither macOS nor Windows have that option :(\n\n[quoted text omitted]\nin theory, the macOS config (and possibly Windows) could disable it, but if this gets too complicated, I may close this pull."
  },
  {
   "t": "2026-09-01T12:55:38Z",
   "kind": "comment",
   "who": "maflcko",
   "assoc": "MEMBER",
   "text": "Ok, so tmpfs helped a bit on macos, but still not enough. On Linux, it didn't really make a difference, it seems."
  },
  {
   "t": "2026-09-01T15:28:38Z",
   "kind": "comment",
   "who": "jeanpablojp",
   "assoc": "CONTRIBUTOR",
   "text": "[quoted text omitted]\n\nYou're right. `utxo_total_supply` is the last to finish in all 20 master runs I checked, and it runs for the last 86 to 214s of them, so my sentence was wrong.\n\n[quoted text omitted]\nmacOS on master runs anywhere from 632 to 976s and this head is 767s, so it is inside what master does on its own. I guess two runs can't really separate that."
  }
 ],
 "labels_log": [
  {
   "t": "2026-08-29T14:07:09Z",
   "action": "labeled",
   "label": "Fuzzing",
   "who": "DrahtBot"
  }
 ],
 "state_log": [
  {
   "t": "2026-09-01T10:11:53Z",
   "kind": "convert_to_draft",
   "who": "maflcko"
  }
 ],
 "text_chars": 5947,
 "text_tokens_estimate": 1486,
 "changed_paths": [
  "ci/test/02_run_container.py",
  "test/fuzz/test_runner.py"
 ],
 "files": [
  {
   "path": "ci/test/02_run_container.py",
   "add": 14,
   "del": 0
  },
  {
   "path": "test/fuzz/test_runner.py",
   "add": 56,
   "del": 29
  }
 ],
 "test_lines": 99,
 "git": {
  "head": "633456aa8b7bcf3b4d0a5bd4720bd6cf023b8379",
  "head_matches_backup": true,
  "base": "05e49b342faa1412266951429c135e9f5daa30c2",
  "commits": [
   {
    "sha": "fa2d43f7ea",
    "subject": "fuzz: Corpus sharding in parallel runs",
    "files": 1,
    "add": 56,
    "del": 29
   },
   {
    "sha": "633456aa8b",
    "subject": "ci: tmpfs",
    "files": 1,
    "add": 14,
    "del": 0
   }
  ],
  "patch_truncated": false
 },
 "input_hash": "505ed2733f6dd474",
 "extracted_at": "2026-09-17T16:15:31+00:00"
}