A scripted merge train for the Zephyr main tree

[nashif]

Problem Description

The merge list already decides which pull requests are mergeable: it evaluates multiple gates per PR (conflict-free, assignee approval, minimum review window, 2 approvals, ...) and labels the result ready, waiting, or blocked. What happens next is still manual. A release engineer opens the page, scans roughly 20-40 ready rows, and for each ready PR opens GitHub, re-checks the gates by eye, and presses Rebase and merge. On a typical day that is 20–30 PRs against main at least 3-4 times plus a handful of backports.

Few things are wrong with this beyond how tedious this is:

• Every PR is tested alone. PR CI runs against the PR's own merge with main at the time it was pushed. Two ready PRs that each pass can still break the tree together, and that breakage is discovered only after both are merged, on main, by everyone.
• The process is not reproducible. There is no record of what the list said when the merges happened, what order they went in, or why a PR was skipped. Post-mortems start from GitHub timestamps.
• It does not scale with the queue. The cost is linear in the number of ready PRs, and it lands entirely on the few people with merge rights, at the time of day they happen to be awake.
• PRs CI testing is limited depending on what has changed in a PR. A one line change in a PR might pass the few tests associated with the files changed but might have side-effects that impacts other areas.
• Some PR changes do not trigger any tests or coverage is missed due to other factors (i.e. in PRs we run twister with --integration, so only integration platforms are scoped, when merged, more platforms are scoped and issues might arise

Proposed Change (Summary)

Replace the manual loop of reading the merge list in a browser, opening each ready pull request, and clicking Merge one at a time with a reproducible pipeline: build one branch with every ready PR merged in, push it and let full CI run, verify it is green, then merge exactly the PRs that rode that branch, from the terminal.

Proposed Change (Detailed)

Treat the merge list as a data source rather than a web page, and run a short, scripted merge train against it. Each run snapshots the list, builds an integration branch containing every ready PR for a target, runs the test suite on that branch, and then do a scripted batch merge that merges those PRs that passed the integration test.
Nothing in the merge policy changes; the gates the merge list already enforces remain the only criteria for inclusion. What changes is that the batch is tested as a batch before it lands, and every step leaves an artifact.

possible policy items:

• The merge list is the sole source of eligibility. The tools never compute readiness themselves; they read the gates the list publishes. If the list is wrong, fix the list.
• A PR is merged only from a green train. Being on the list is necessary, not sufficient. A PR that cannot be integrated (conflict) or whose inclusion turns the train red waits for the next one.
• Merge method is rebase, matching current project practice, so commit hygiene requirements on contributors are unchanged.
• Order is deterministic: ascending PR number. Older PRs get the earlier seat; a newer PR that conflicts with an older one is the one that has to rebase.

Dependencies

• need the https://merge-list.zephyrproject.io/ to generate machine readable queue with more data that can be consumed by scripts
• need to maintain store the data as artifcats to enable reproducibility and tracking
• scripts to create the integration branch in a reproducible way
• a gh extension that works with all of the above and allow merges of verified PRs

Concerns and Unresolved Questions

• What lands is not byte-for-byte what was tested
• A PR can change after it was tested
• A PR can stop being ready after it was tested
• Attribution when the train is red
• ....

Alternatives Considered

• GH Merge queues: Solves the "tested together" problem natively but replaces the merge list's gate logic with branch-protection rules, has no notion of the review window, and removes the human checkpoint before landing. Could be revisited if the merge list's rules can be expressed as required checks.

[nashif]

@zephyrproject-rtos/release FYI

[dkalowsk]

Overall I like the idea and the direction.

A slight variation to the idea, convert the merge-list into more of a Kanban style board with column A being all pending PRs and column B being the PRs on the current (reproducible) merge train. While the train is running, column B can be "locked down" meaning no additional PRs can be added until it completes. When the train is successful, all the PRs are merged, the column B is emptied and "unlocked".

This would keep the merge list as a data source, but instead of making it a pull system, it becomes a push system. It leaves it still slightly manual which may be not what you're envisioning.

[teburd]

This is exactly what bors does already with some rather nice ways of controlling what goes into a batch to test and what gets into the merge queue independently to retest on merge.

Should we try using bors? It can also queue up batches into temporary merge, ci test, then merge type queues.

Or does this not work with github CI, I guess that would be a significant issue. Either way, once you start doing batches of merge items, temporarily merged stacked on main and run a CI test then you'll probably want to be able to queue batches or run some things independently and bors does all that as I understand it.

For example looking at https://bors-prod.rust-lang.net/queue/rust you'll often see some rollups (batch of PRs being done together in a CI run) and independent PRs.

[jhedberg]

Great proposal! The exception path (when the train is red) needs some more detailed definition. E.g. how do you determine which PR broke the train (I assume that's the "attribution" bullet in the Concerns/Unresolved section). Also "PR that cannot be integrated (conflict) or whose inclusion turns the train red waits for the next one" is too simplistic, I think. There should be some automatic "DNM" style label that gets added or similar solution to prevent the PR from being picked into a future train until it's actually believed to be ready again.

[fabiobaltieri]

Sounds good, main concern is having a relatively complex bit of infrastructure with merge permissions to maintain and troubleshoot. How would it operate in practice? Where is it going to be hosted, who is going to debug it, how do we dry run it etc.

As always, resulting an existing, developed, maintained, tested piece of software would be better than vibe coding ^w^wwriting one from scratch, we are here to make a world class rtos not to a world class patch management system, but hey if that's what we need. @teburd maybe you can give us an overview of bors? though it seems fairly intuitive.

@nashif on practical terms: if you know, say, rebase a bunch of PRs on a branch and then fast forward that on main, github is smart enough to close those PRs as merged, right?

[nashif]

ok, I did some prototyping, the last batch I did was using this method...

create integration PR / branch (scripted)

• pull the list of all ready PRs from https://merge-list.zephyrproject.io/
• create a branch off main, merge all PRs into the branch (my test branch: https://github.com/zephyrproject-rtos/zephyr-testing/commits/collab-integration/)
• this can be submitted via push to a special namespaced branch on the main repo i.e.. integration/date-uuid-whatever, triggering full CI as we have it with push
• alternatively this can be pushed as a draft PR with some special handling, so it does run the same way we do push with all coverage and CI.
• Wait for results, if something fails, it is either easy to spot or if it is complex, the culprit PR is easier to bisect. If a check fails, attribute it with --no-ff which give a one-PR-per-commit bisection.
• if all is green, you will have the processed list already downloaded from https://merge-list.zephyrproject.io/ with SHAs, you launch the merge script / extension and you can go 1 by 1 or in batch and merge them all...

the merge step:

select (e.g. 1 3 5-7, or 'all'): 2-22
About to merge 21 PR(s) into zephyrproject-rtos/zephyr (--rebase):
  #112258  [main] Mcuxa lpdac test harness enhancement for second serial console checking with latest multi-dut support in twister-pytest
  #112715  [main] tests: posix: common: split POSIX_SYSTEM_DATABASE into standalone
  #113235  [main] drivers: ethernet: dwc_mac: add multicast filtering
  #114720  [main] ci: update docker images to v0.29.3.20260727
  #115373  [main] drivers: power: fix the SAM SUPC wakeup source ID accessor
  #115526  [main] kvss: add @file Doxygen blocks to NVS and ZMS headers
  #115703  [main] twister: add console monitor
  #115727  [main] drivers: i2s: honour the configured timeout in i2s_buf_write()
  #115830  [main] scripts: dts: Add info on how to test edtlib.py and dtlib.py
  #115846  [main] drivers: timer: arm_arch: use the generic timer core
  #116105  [main] MCTP over I2C+GPIO fixes
  #116261  [main] Clear CCC and transfer status in Cadence driver
  #116262  [main] drivers: can: mcan: code cleanup and control flow streamlining
  #116291  [main] devmux: Fix userspace support
  #116317  [main] dap: report a connect failure when the port cannot be enabled
  #116478  [main] drivers: display: uc81xx: fix init and refresh handling
  #116504  [main] testsuite: ztest: benchmark: make the headers usable from C++
  #116721  [main] kernel: stack: fix allocation size overflow
  #116856  [main] MAX32666FTHR: usb and feather SPI/I2C support
  #116863  [main] MAINTAINERS: defer vendor owned UART drivers from generic area
  #116871  [main] MAINTAINERS: defer vendor owned I3C drivers from generic area

Proceed? [y/N] y

merging #112258 Mcuxa lpdac test harness enhancement for second serial console checking with latest multi-dut support in twister-pytest
✓ Rebased and merged pull request zephyrproject-rtos/zephyr#112258 (Mcuxa lpdac test harness enhancement for second serial console checking with latest multi-dut support in twister-pytest)
  #112258 merged

merging #112715 tests: posix: common: split POSIX_SYSTEM_DATABASE into standalone
✓ Rebased and merged pull request zephyrproject-rtos/zephyr#112715 (tests: posix: common: split POSIX_SYSTEM_DATABASE into standalone)
  #112715 merged

merging #113235 drivers: ethernet: dwc_mac: add multicast filtering
✓ Rebased and merged pull request zephyrproject-rtos/zephyr#113235 (drivers: ethernet: dwc_mac: add multicast filtering)
  #113235 merged

merging #114720 ci: update docker images to v0.29.3.20260727
GraphQL: refusing to allow an OAuth App to create or update workflow `.github/workflows/bsim-tests.yaml` without `workflow` scope (mergePullRequest)
  #114720 failed (exit 1)

merging #115373 drivers: power: fix the SAM SUPC wakeup source ID accessor
✓ Rebased and merged pull request zephyrproject-rtos/zephyr#115373 (drivers: power: fix the SAM SUPC wakeup source ID accessor)
  #115373 merged

merging #115526 kvss: add @file Doxygen blocks to NVS and ZMS headers
✓ Rebased and merged pull request zephyrproject-rtos/zephyr#115526 (kvss: add @file Doxygen blocks to NVS and ZMS headers)
  #115526 merged

merging #115703 twister: add console monitor
✓ Rebased and merged pull request zephyrproject-rtos/zephyr#115703 (twister: add console monitor)
  #115703 merged```


From infrastructure side we should be good, we will be just reusing what we have.

[nashif]

Should we try using bors? It can also queue up batches into temporary merge, ci test, then merge type queues.

https://bors.tech/
Heads up! Bors-NG's public instance is being phased out in favor of GitHub Merge Queues.

For example looking at bors-prod.rust-lang.net/queue/rust you'll often see some rollups (batch of PRs being done together in a CI run) and independent PRs.

that looks pretty much like our merge-queue but seems to be running as a service compared with the "static" html page we use, so it is more involved. My proposal above starts with what we have, so we can start using it immediatly after we agreed and improved a few things.

[kartben]

How / when will the release engineer looking after a given batch exercise the "re-checks the gates by eye"? I.e if 1 (or more) PR should not be part of a given batch how/when are they being pulled out?

[nashif]

create integration PR / branch (scripted)

• pull the list of all ready PRs from merge-list.zephyrproject.io
• create a branch off main, merge all PRs into the branch (my test branch: zephyrproject-rtos/zephyr-testing@collab-integration (commits))
• this can be submitted via push to a special namespaced branch on the main repo i.e.. integration/date-uuid-whatever, triggering full CI as we have it with push

all of the above can be done in a workflow, so release engineers do not have to do that manually and locally which might lead to inconsistencies and conflicts...

[nashif]

How / when will the release engineer looking after a given batch exercise the "re-checks the gates by eye"? I.e if 1 (or more) PR should not be part of a given batch how/when are they being pulled out?

that is the part we need to agree on before any of the above happens. Step 0 before any of this is actually kicked off is reviewing the merge list and what would be merged, once all clear, press the button, create the branch and continue looking while CI builds... So that is the first gate.

The CI verdict and removing anything that is red or causing trouble is the second gate... but the first is more important.

There is the other major gap we have which is not really related to the operation side of things, was a pr approved by the right people, do we need more review time on some PRs etc etc, Ideally this would be step -1 and covered more with policies and "automation" than a human factor, so the merge train becomes more of a "no-brainer".

[aescolar]

that is the part we need to agree on before any of the above happens. Step 0 before any of this is actually kicked off is reviewing the merge list and what would be merged, once all clear, press the button, create the branch and continue looking while CI builds... So that is the first gate.

Do you mean then that we would manually go over each PR still?
So effectively what we'll be doing is creating an extra branch in which we run a full main-merge CI job? (which mostly avoid breaks in main).

once all clear

Note the list is changing under us (the list you started from will have different entries once you are done going thru it)
That includes people adding rejections, the PR content changing, other PRs being merged that conflict (we'll need to keep merging by hand).

There is the other major gap we have which is not really related to the operation side of things, was a pr approved by the right people,...

Was it re-assigned incorrectly, does it have a bogus hotfix/trivial label..
(I also do a very quick visual scan of the diff to avoid merging anything that looks horrible even if it was approved)

Also note most of us release engineers should not have rights to push to main directly, but only to press the merge button.