GitHub Community
Parallel Steps #14484
|
Hi, Can you please raise a request to engineering for parallel steps support? Eg. for logging in to multiple docker registries in parallel, or other actions which cannot be done in separate jobs because you require the local cache / side effects. Uploading/downloading artifacts from adjacent jobs is not suitable for this use case. Thanks Hari |
Replies: 47 comments 84 replies
|
I'd like that too. There are scenarios where it doesn't make sense juggling artifacts between jobs. An idea how it could be used: steps:
- uses: actions/checkout@v3
- run: echo "Sequential step"
- parallel:
branch_id_1:
steps:
- run: echo "branch 1 sequential step 1"
- run: echo "branch 1 sequential step 2"
# Executes steps in parallel to branch1
branch_id_2:
steps:
- run: echo "branch 2 sequential step 1"
- run: echo "branch 2 sequential step 2"
- run: echo "More sequential steps after the whole parallel part is done." |
|
This is a good approach as it mirrors Jenkins well-known solution for parallelizing stages. |
|
Response #1 |
|
https://github.blog/news-insights/product-news/lets-talk-about-github-actions/
Thanks to @tempacc21 for noticing! Putting here as well for better visibility. |
|
Putting here for visiblitiy https://github.com/orgs/community/discussions/14484#discussioncomment-17437990
|
|
Alternative solution: |
Alternative solution: |
|
Sorry I'm missed how putting steps in a composite action would allow you to parallelize steps such as multiple logins to different registries? Could you please elaborate further with regards to how this solves step parallelization? |
|
Parallel steps or sub jobs would save us 50% of time waiting for the CI. |
|
GH would lose a lot of money if they let people halve the time they charge for. |
|
Logging into docker in parallel is probably going to result in things exploding. Each docker login command tries to mutate Today, you could probably do something like: (To make these work, you'd want to have them rely on env vars and set the env vars using your workflow based on secrets)
HOME=$HOME/.docker-docker
mkdir -p $HOME
docker login ...
HOME=$HOME/.docker-gcr
mkdir -p $HOME
docker login ...
HOME=$HOME/.docker-ghcr
mkdir -p $HOME
docker login ...
for task in ./log-into-*.sh; do
$task &
done
wait
jq '. * input' ~/.docker*/config.js > my.config.js && mv my.config.js ~/.docker/config.js
jq '. * input' ~/.docker*/plaintext-credentials.config.json > my.plaintext.js && mv my.plaintext.js ~/.docker/plaintext-credentials.config.json |
|
If the |
|
Talking to the docker folks is an exercise for the brave. I wouldn't recommend it to my worst enemy. |
|
@jsoref I think instead of for loop with
About docker I totally agree. Talking to docker devs is PITA and nothing that improve dev work is actually done e.g. multiple dockerignore files or mounting volumes as non root to container and so on. |
|
@piotrekkr: thanks on both points. |
|
can we have the parallel steps please |
|
YES, this is the most useful option and it's obvious. Downvote must come from some GitHub dev who does not like to have some work or what? (nah, downvote was about something else, sorry, it's valid) |
Nope, it's coming from me, because this answer doesn't bring any value. Basically a spam. |
|
Because here no-one can request for anything because GitHub action devs don't see it? Okay, I'm not aware where requests can be made. It seems to be anyways useless because there are some many open requests at various pages and I also saw GitHub action devs saying, they will probably not do it |
|
The main post is the by
Unfortunately yes :(
Link please? |
|
It's from memory, so cannot tell, sorry. I think it was either about better caching or about their unwillingness to make "include yml files for steps" or some other of the obvious shortcomings. But then again, not sure if Github even claims to be a CI/CD solution or just code repo with benefits, so maybe we should not be so hard |
|
The BitBucket has parallel step options. Why not add this option to GitHub actions? |
|
This is the biggest pain point we've run into coming from Jenkins. You can approximate this feature with |
|
How come it's error prone? |
|
Every project has different needs about what artifacts they need available, how they're shaped, fs permissions, size costs, etc... |
|
Need parallel steps support for our scenario: deploy multi clusters concurrently within the same VPN environment. So it is convinient to achive in steps rather than actions or jobs |
|
Using a yarn monorepo I'd like to:
yarn install is slow, and I currently have to repeat it for every workflow like |
|
We've been iterating over an action that does some of the work to make this better: https://github.com/GarnerCorp/build-actions/blob/main/yarn/action.yml You can use actions/upload-artifact and actions/download-artifact to share a build product (e.g. the output tarball from the above action) between jobs (especially a matrix of jobs to run tests). |
|
Are you using yarn workspaces? If you are, you should be able to run |
|
I was looking for a solution for exactly the same problem @raphaelbadia. I did get caching to work but I'm using PNPM, which I would highly recommend as an easy time saver compared to Yarn. It uses symlinks to a system level package cache, instead of installing dependencies in all your monorepo projects. |
|
Check out this approach: Making pnpm run jobs in parallel instead of finding a way to run GitHub steps in parallel: https://stackoverflow.com/questions/68427503/how-can-i-run-pnpm-workspace-projects-as-parallel-jobs-on-github-actions |
|
@michielvanderros-tomtom speaking of PNPM :) Yarn supports pnpm linker out of the box, in addition to pnp and regular node_modules; yet it's developed by a team vs. single person behind pnpm, and has great design with plugin system. |
|
If its easier, GitHub could also allow sharing jobs workdirs |
|
You can use |
|
I'm aware of that. |
|
Have a look here: https://circleci.com/docs/caching/ |
It's super slow. Impractical for anything larger. |
|
I'm in the minority, but I'd actually like this way better too— the handling of logs is more natural because each job has its own, and you already have the dag visualization of how the jobs fit together. The obvious syntax would be instead of Main question mark would be what happens to those post-action steps, if you've got a bunch of torn down auth state, that's mildly annoying, though re-authing is not nearly as bad as starting from scratch. This doesn't cover the "multiple preparatory steps in parallel" ask that some others have been making, but I'm honestly not sure if that should be the responsibility of GitHub Actions anyway. Yes, you can do that kind of thing in your Jenkinsfile, but really it belongs in a proper build tool. Let |
|
I just want to start a resource in a background, which needs some time to start, eg a docker container or a simulator. This resource should start during the job start. My test step should wait until the test resource is available, or fail after an (optional) timeout. - jobs:
- build
- resources:
- test-db
run: docker start someSlowImage
- steps:
- run: assemble
- run: test
resources:
- test-db
timeout: 5min |
|
I think you can do this using service containers: https://docs.github.com/en/actions/using-containerized-services/about-service-containers |
|
GitLab CI does a similar thing: https://docs.gitlab.com/ci/services/ |
Just add a
|
|
They might not, but each Google Cloudbuild step runs a separate disposable container and you can't persist state from one to another. Thus anyone making a comparison to them is making a very uniformed comparison. I'm well aware of both. We're in the process of migrating some builds from Cloudbuild to GitHub Workflows and I've been using GitHub Workflows/Actions for years. |
|
@jsoref you can pass state between steps in Cloud Build just as easily as in GitHub Actions — via the shared A common workflow like:
works naturally in Cloud Build because:
steps:
# Checkout source
- name: gcr.io/cloud-builders/git
args: ["clone", "https://github.com/org/repo.git"]
# Setup Node + install deps
- name: node:18
entrypoint: yarn
args: ["install"]
# Build API and App in parallel
- id: build-api
name: node:18
entrypoint: yarn
args: ["workspace", "api", "build"]
- id: build-app
name: node:18
entrypoint: yarn
args: ["workspace", "app", "build"]
# Deploy in parallel, but only after *both* builds succeed
- id: deploy-api
name: node:18
entrypoint: yarn
args: ["workspace", "api", "deploy"]
waitFor: ["build-api", "build-app"]
- id: deploy-app
name: node:18
entrypoint: yarn
args: ["workspace", "app", "deploy"]
waitFor: ["build-api", "build-app"]Here, deployment of From developer feedback (e.g. Reddit, issues, discussions), a recurring criticism of GitHub Actions is not lack of capability per se, but that core concepts like parallelism and step dependencies require significantly more ceremony (jobs, matrices, A simple Docs: https://docs.cloud.google.com/build/docs/configuring-builds/pass-data-between-steps |
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
# Install dependencies
- run: yarn install
# Build the API and web front-end (app)
- id: build-api-app
run: |
yarn workspace api build &
yarn workspace app build
fg %1 || true
# Deploy
- id: deploy-api-app
run: |
yarn workspace api deploy &
yarn workspace app deploy
fg %1 || true |
|
@jsoref In your example, the build is effectively treated as a single step. There are no independent build steps per package/project, each with its own lifecycle, success/failure status, and logs — and that distinction is the core of this issue. What’s missing in GitHub Actions is true step-level parallelism with explicit dependencies. Without it, workflows quickly devolve into large monolithic steps or awkward job graphs with duplicated setup, artifact passing, and extra indirection. I can’t overstate how painful this is (at least for me) when working with multi-package repos. A simple, declarative way to run independent steps in parallel — with clear dependencies and isolated logs — would make a huge difference in both readability and maintainability. 😫 😖 😣 😡 💔 😭 😩 💢 🤬 💥 💢 😤 |
|
You can make that if you want, just create logs for each thing in the main step and then pass the various job outputs to individual steps for reporting. It really isn't terribly hard |
|
As a workaround I was able to use this ` name: Distributed Tasks This way I was able to run our tests parallel |
|
Are you able to see the logs separately for different parallel commands in GitHub Actions? |
|
@koistya : no, can not see logs separately , but if one of the cmd fails the can see logs |
|
That’s an interesting workaround, @malindajudo. Running parallel steps in GitHub Actions can definitely simplify workflows, especially when dealing with tasks like multiple deployments or integrating third-party APIs. For example, in a web app development company, managing Docker containers and parallelizing tasks can help eliminate the complexity of sequential executions. This capability would allow teams to scale their processes faster, reduce bottlenecks, and enhance overall efficiency in their CI/CD pipelines. |
This comment was marked as off-topic.
This comment was marked as off-topic.
|
Hi. Are there any updates on this topic from a Github team member ? |
|
This is an excellent proposal! The approach of introducing A few thoughts on the design:
The examples provided really help illustrate the use cases this would solve. Looking forward to seeing this feature ship before mid-2026! |
|
WRT to composite actions: composite actions generally is an area we are looking at. There's some other highly requested issues we are working on as well and as mentioned in my other comment, some of that work will probably overlap. With error handling, I want to clarify a little bit. Let's use the following example: steps:
- run: echo "Setup complete"
- name: Build frontend
run: npm run build:frontend
background: true
- name: Build backend
run: npm run build:backend
background: true
- name: Build docs
run: npm run build:docs
background: true
continue-on-error: true
- wait-all:
- run: echo "All builds complete"In this we would first kick off the frontend build, then the backend, then the docs, all of which would run in the background until we hit the wait. Let's say the frontend build failed immediately, I would still expect the backend and docs builds to continue and the job to still be in a pending state. Once we hit the |
|
This looks great, I would add a (more niche) use case: having a background step that can wait for another step without blocking the "main thread" of foreground steps. steps:
- name: Starts A in background
id: a
run: ...
background: true
- name: Does not wait for A
id: b
run: ...
background: true
- name: Waits for A but not B
run: |
wait-step a
...
background: true
- name: Wait for A and B but no C
run: |
wait-step a b
... |
|
Do you have a real world example for this? (Obviously the idea exists, but it's much more interesting with real flows) |
|
Yes I create 2 docker images, the 2nd after the first one to reuse most of the cached steps, and have different tests for each one so I get this graph for ideal parallelism: |
|
That feels like: - name: Checkout
uses: actions/checkout@latest
- name: Build A
run: ...
- name: Build B
id: build-b
run: ...
background: true
- name: Test A
run: ...
- name: Test B
wait: build-b
background: true
run: ...
- name: Report
run: ...
wait-all: ...Which I think should technically work based on the described primitives |
|
tl;dr small clarification on how So - name: Build A
id: build-a
run: ...
background: true
- name: Build B
id: build-b
run: ...
background: true
- name: Build C
id: build-c
run: ...
background: true
- wait: [build-a, build-b, build-c]
- name: Test
run:...We start Build A, then Build B, then Build C. Once we get to the I think there are multiple ways to handle the scenario described with background steps, probably depending on what you want to do. This is how you would do it with a - name: Build A
run: ...
- parallel
- name: Test A
run: ...
- uses: myorg/b-action@v1 # this is a composite action that does Build B, then Test B
- name: Report
run: ...Alternatively, if you wanted to just use background steps, you could do something like this (@jsoref's example would have had Test B wait until Test A finished, regardless of whether Build B had finished, which I don't think is the desired behavior): - name: Build A
run: ...
id: build-a
- name: Test A
run: ...
id: test-a
background: true
- name: Build B
id: build-b
run: ...
- name: Test B
run: ...
- wait-all: # Could also be wait: test-a, since we know test-b would have to be finished before we got here
- name: Report
run: ...I think that if you are trying to run multiple parallel work streams which may eventually converge, using - name: Build A
run: ...
id: build-a
- name: Build B
id: build-b
run: ...
background: true
- name: Test A
run: ...
id: test-a
background: true
- wait: build-b
- name: Test B
run: ...
- wait-all: # Could also be wait: test-a, since we know test-b would have to be finished before we got here
- name: Report
run: ...I think |
|
@salilsub Thank you for your guidance. I prepared the main workflow in my repository to use the new features (Adding |
|
What I really miss in the parallel step design is to have the matrix strategy on step level like this (I create sometimes ca. 30 matrix jobs that run only < 30s) - run: echo Process the following in parallel / generate the matrix in this step dynamically
- strategy:
max-parallel: 2
matrix:
artifacts:
- name: pkg1
path: pkg1/out
- name: pkg2
path: pkg2/out
steps:
# Here you could also add the build command to run sequentially before the upload artifact
# upload-artifact usually only uploads all files into a single artifacts this is bad if you want to download only a single package
- uses: actions/upload-artifacts@v6
with:
name: ${{ stepMatrix.artifacts.name }}
path: ${{ stepMatrix.artifacts.path }}Yes I can do this example with javascript and the @actions/artifact package, but then I need much more code (On the other hand the runner needs more code for this) In my opinion Parallel Syntactic Sugar weakness is that the count of parallel steps is constant, with a matrix strategy you can adjust the called actions count and parallelism depending on the reusable workflow input parameters. Just like in job.strategy. |
|
+1 |
|
Planned to be shipped between April-June 2026 github/roadmap#1191 |
Source: https://github.blog/news-insights/product-news/lets-talk-about-github-actions/ |
|
I see that support for it was already implemented in the actions LSP: actions/languageservices@00d83b8 Looking forward to it! |
|
it landed on the runner itself too actions/runner#4476 |
|
Sounds like a cool feature. When is it shipped and can be used? |
|
Hi everyone, wanted to give an update on this. We've been working on implementing background/parallel steps, some of you have noticed changes in some of our other repos like runner and language services. We're currently dogfooding background steps internally and will publicly ship once that process is completed. I'll post another update once it's shipped. |
|
Hi everyone, these changes have been shipped, you can start adding in |
|
It'd be great for The feature is very bare bones, but it is already immensely useful as is. |
|
@nazar-pc can you give an example? |
|
For example here I have And here I had to repeat Often steps as such do not have useful names, they just share the load (like in the first example above), so it makes much more sense to name |
|
For things like |
|
Tested this feature in some of my orgs reusable workflows and it seems not to be working in composites. Is that intended? Is support for composites expected in the future? |
|
Yep seeing the same - was so excited 😭 |
|
Workaround that isn't too terrible - parallel:
- uses: ./.github/actions/setup-mise
- uses: ./.github/actions/setup-node
- uses: ./.github/actions/setup-aws |
|
Composite was out of scope for our first release :) https://github.com/orgs/community/discussions/14484#discussioncomment-15480826 Will I be able to have background steps within composite actions? What I would <3 is that I am going to close this issue as 'done' and if y'all can open a new one and start voting 'up' on Composite Actions support so we can track further increments outside of the noise that is likely to exist here for some time. It will really help us understand where to put our energy next rather than chasing through 'quite a lot' of sub comments and threads in here :D Appreciate the engagement already on this. |
So that is the primary workflow calling in parallel the composite? |
|
Yeah basically we split up our composite action into the steps we want to run in parallel and run the composites in parallel! Great feature! |
This comment was marked as spam.
This comment was marked as spam.
|
This is now shipped :) as a few folks have called out, its a v1 and is not 'everything we could have possibly done'. I would ask for folks to create new specific feedback issues for improvements (such as composite Action support) so we can get votes on those items specifically to know the priority, get our focus, and not get lost in threads in here. Thank you for the patience, feedback and engaging. More to come <3 |
|
Looks like 1 was created |
|
Ooh thank you, noted and marked as 'backlog' so it auto-adds to my lists of things |
|
This is so cool! Thanks for this feature 😄 Just wondering: would it be possible to define my parallel steps in a matrix-like kind of way? Useful for cases where the steps sequences are the same but too simple/lightweight for using a proper job Matrix E.g.: jobs:
build-openapi-spec:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0
- name: Get deployed tags and add a worktree for each environment
id: get-deploy-tags
run: |
for environment in dev stage prod; do
deploy_tag="$(git tag --sort=creatordate | grep "^${environment}-" | tail -n1)"
git worktree add worktree-"$environment" "$deploy_tag"
done
- parallel-matrix:
vars:
environment: [dev, stage, prod]
steps:
- uses: ruby/setup-ruby@v1
with:
working-directory: worktree-${{ parallel-matrix.vars.environment }}
bundler-cache: true
- name: Build OpenAPI specs
working-directory: worktree-${{ parallel-matrix.vars.environment }}
run: bundle exec rake rswag |
|
Hi @Drowze this is not supported with this release but this is a request that has come up before. I'd recommend opening a discussion post for this specific feature, upvotes there will help us prioritize the requests going forward. Thanks for the feedback! |
|
I just tried it in a composite action and it doesn't seem to be supported there, sadly: Error: D:\a\msys2-tests\msys2-tests\./action.yml (Line: 6, Col: 7): Unexpected value 'parallel' |
|
For windows, |
|
@koppor Can you provide more information on the steps being run? |
|
I did not see your reply earlier - too many things going on. -- We switched to gradle parallel builds so that we do not run into this issue any more. Claude found something - I think, it is https://github.com/JabRef/jabref/blob/9bb0c77da53a5fb86187b337d16d009440e61b38/.github/workflows/binaries.yml#L648 Its for from a minimal workflow file. I can try to reproduce later this month with an MWE... - However, currently not urgent on our side, because we don't use this kind of parallism any more. Failing run (attempt 1): https://github.com/JabRef/jabref/actions/runs/28589706603/attempts/1
Caveat: raw logs are already purged (API returns HTTP 410), so the message text can't be re-quoted from the run — only the step-level failure and your comment survive. |
|
It appears that calling the same external workflow multiple times within a single step can lead to conflicts, depending on the timing. First run: Prepare all required actions
Getting action download info
Download action repository 'pnpm/action-setup@fe02b34f77f8bc703788d5817da081398fad5dd2' (SHA:fe02b34f77f8bc703788d5817da081398fad5dd2)
Download action repository 'actions/cache@3edfce9056124e459a23f683a21433670d47daca' (SHA:3edfce9056124e459a23f683a21433670d47daca)
Error: Directory not empty : '/home/runner/work/_actions/actions/cache/3edfce9056124e459a23f683a21433670d47daca/.github'Second run: Prepare all required actions
Getting action download info
Download action repository 'denoland/setup-deno@22d081ff2d3a40755e97629de92e3bcbfa7cf2ed' (SHA:22d081ff2d3a40755e97629de92e3bcbfa7cf2ed)
Download action repository 'actions/cache@3edfce9056124e459a23f683a21433670d47daca' (SHA:3edfce9056124e459a23f683a21433670d47daca)
Error: Can't find 'action.yml', 'action.yaml' or 'Dockerfile' for action 'actions/cache/restore@3edfce9056124e459a23f683a21433670d47daca'. |
|
Yep, external workflows getting nuked before all parallel jobs have completed work with it is happening to us exactly. Any remedies yet? |
and others
Hi everyone, these changes have been shipped, you can start adding in
backgroundsteps andparallelblocks into your jobs.