EphemeralRunners created through an AutoscalingRunnerSet do not have labels set in the GitHub.com API

[ivucica]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

ghcr.io/actions/gha-runner-scale-set-controller 0.14.0

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

See #4424.

Ensure that AutoscalingRunnerSet has the labels set:

$ kubectl get autoscalingrunnerset  -n arc-runners -o json | jq .items[0].spec.runnerScaleSetLabels
[
  "self-hosted",
  "linux",
  "arc-runner-set"
]

(I wish I could easily prove that running scale set has this set server side, but the API is not documented and the [scaleset client](https://github.com/actions/scaleset) does not have an obvious endpoint documented beyond [`_apis/runtime/runnerscalesets`](https://github.com/actions/scaleset/blob/ebedf0bbbf6245cf5b1ff3c8a1297a08e133caf8/client.go#L24) (however I have not managed to guess what the host is). It's there on the web.

Let an ephemeral runner be created -- it won't have labels. Apply to runners (for example like this, starting with 0th one, repeat as needed)


curl -X POST -d '{"labels":["arc-runner-set"]}' \
    -H "Authorization: Bearer $(grep github_token: 03-pat.secrets.yaml | cut -d':' -f 2)" \
    https://api.github.com/repos/PERSONUSER/REPONAME/actions/runners/$(curl \
        -H "Authorization: Bearer $(grep github_token: 03-pat.secrets.yaml | cut -d':' -f 2)" \
    https://api.github.com/repos/PERSONUSER/REPONAME/actions/runners |  jq -r .runners[0].id)/labels


Run an action.

See the runner get nuked / recycled.

See a new runner get created again without labels.

Describe the bug

Runners created with ARC do not seem to get any labels attached to them.

Discovered at the same time as #4424, which was more about "when you edit labels on the chart and therefore on set, they do not get applied to autoscaling runner set in the API". But, it turns out that applying the labels in AutoscalingRunnerSet is also not sufficient for the runners to get labels applied.

The output from the API doesn't even show default labels that are meant to be applied, and in the web UI, there is no OS mentioned etc.

https://docs.github.com/en/actions/how-tos/manage-runners/self-hosted-runners/use-in-a-workflow#using-default-labels-to-route-jobs

This means that tricks to route to the "best" runner by enumerating the runners, filtering by label, availability and busi-ness (such as what https://github.com/mikehardy/runner-fallback-action does) don't work, as no runner has the appropriate label attached.

Describe the expected behavior

If an AutoscalingRunnerSet has labels applied (via chart's scaleSetLabels [sic] which populates the object's spec.runnerScaleSetLabels, or directly setting that property), it would likely be a sane operation to have the runner get those labels applied, either by ARC or by the API's backends themselves.

It would also be a good idea to have the EphemeralRunnerSet¹ store the intended labels in a runnerSetLabels as a copy of runnerScaleSetLabels from the parent.

EphemeralRunnerSet should likely propagate them into an individual EphemeralRunner as runnerLabels, which should then be the actual ones to create the runner with the labels specified (defaulting to letting API decide what to do if the list of labels is not present, as opposed to being empty).

There's also the precedent in some Kubernetes objects of not putting it into a dedicated field in the CRD, but into annotations or labels with a well-defined prefix: see how Service objects pick up some important configuration for various loadbalancer implementations.

From what I can tell, the CRD for EphemeralRunner does not seem to include runner labels as a standalone field, so if you choose to allow overriding the labels per runner (which seems to be the API's intention), either approach would make sense.

Additional Context

not applicable, beyond `scaleSetLabels` [sic] has been populated correctly

Controller Logs

not applicable

Runner Pod Logs

not applicable

Footnotes

1. It might also, perhaps, make sense for EphemeralRunnerSet to be the entity that causes the controller to create a runner set in the API, not sure. ↩

[nikola-jokic]

Hey, just to confirm, the runners are executing workflows selected through labels? It is not an issue that the runners are not being picked up, but rather that you would like to see these labels when you query them through the API?

[pauloconnor]

I'm also seeing this issue. I've created scale sets with labels, but the actual runners themselves are listing no labels, and have "Configuration: unknown unknown" when viewed through /settings/actions/runners/XX. These runners are not receiving jobs

[WyriHaximus]

Could also confirm this issue. However, after doing a clean install and forcing a replacement of the CRDs the extra labels are now showing up in the UI and runners start based on those extra labels.

[ivucica]

Hey, just to confirm, the runners are executing workflows selected through labels? It is not an issue that the runners are not being picked up, but rather that you would like to see these labels when you query them through the API?

I was not 100% sure. But it looks like they actually scheduled the "determine who will run the jobs" job¹.

So I decided to spin up the cluster again and test again:

For context, the first steps I have are this:

jobs:
  determine-runner:
    if: ${{ github.event_name != 'workflow_run' || github.event.workflow_run.conclusion == 'success' }}
    runs-on: arc-runner-set # TODO: roll back to ubuntu-latest past Apr 01 2026
    concurrency:
      # Runner choice must happen serially for the "primaries-required" logic
      # to be up to date in the context of one self-hosted runner that may be
        id: set-runner
        uses: mikehardy/runner-fallback-action@v1
        with:
          # "There are three ways runners can be allowed to run against a repo: User, Organization, Enterprise"
          # organization: "SOMEUSER" # This is not an org, so we get 404. User is not in documentation of the action.
          # commenting out, hopefully leaving it to be repo-specific.
          # list of tags a runner must match to be considered a primary
          primary-runner: "arc-runner-set"
          # a single tag that will select a runner to fallback to
          fallback-runner: "arc-runner-set" #"ubuntu-latest"  # TODO: undo on  Apr 01 2026
          # optional, fallback if fewer available, big batch jobs or multiple workflows perhaps
          primaries-required: 1
          # optional, fallback if token expires or github API fails
          # TODO: This is set up as 'Repository Secret', but it may be prudent to hide it from most jobs,
          # and set it up as an Environment secret, because some jobs are somewhat autonomous.
          github-token: ${{ secrets.SOMEUSER_ORG_ADMIN_KEY }}

  build:
    needs: determine-runner
    runs-on: ${{ fromJson(needs.determine-runner.outputs.runner) }}
    if: ${{ github.event_name != 'workflow_run' || github.event.workflow_run.conclusion == 'success' }}
    container:
      image: ghcr.io/SOMEUSER/SOMEREPO-ci:latest

    steps:
    - name: Checkout repository
      uses: actions/checkout@v4
      with:
        submodules: recursive

Please note that the job:

1. runs
2. determines the same set is not available (per-worker labels are missing)
3. falls back to the same set solely because I temporarily made it happen so

3s
Current runner version: '2.333.1'
Runner name: 'arc-runner-set-r62fk-runner-wlhx6'
Runner group name: 'Default'
Machine name: 'arc-runner-set-r62fk-runner-wlhx6'
GITHUB_TOKEN Permissions
Secret source: Actions
Prepare workflow directory
Prepare all required actions
Getting action download info
Download action repository 'mikehardy/runner-fallback-action@v1' (SHA:dc7732f9e532c451b2527b288ba8d58fd4b0a4ca)
Complete job name: determine-runner

===

15s
Run sleep 15

===

1s
Run mikehardy/runner-fallback-action@v1
  
Primary runner is online: false
Sufficient primary runners available: false
Using runner: ["arc-runner-set"]
0s
Evaluate and set job outputs
Set output 'runner'
Cleaning up orphan processes
Warning: Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: mikehardy/runner-fallback-action@v1. Actions will be forced to run with Node.js 24 by default starting June 2nd, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

From looking at the API, it is not exactly clear to me how one would query labels of the runnerset for a given worker (i.e. how would the action be patched to correctly determine that GHA is willing to schedule things on this worker). I'd just expect the runners to have some labels -- either specific to runners, or the same as runnerset.

Yes, I can work around this, in my case, but I truly do expect AutoscalingRunnerSet to propagate labels into EphemeralRunnerSet, which then propagates labels into EphemeralRunner at creation time (and at update time either destroys ephemeral runners or updates their labels), which then propagates labels into the API-side runner (and updates them in API if updated). Even if it's an additional field on AutoscalingRunnerSet so that those get copied, not the main ARS ones. [And I should be able to manually create an EphemeralRunner object with labels, so that really should be a field on the ER.]

Footnotes

1. That's the action that actually depends on the existence of labels. Use case: I ran out of monthly minutes, so I wanted to see if I can (experimentally) make my home machines execute jobs when available. While in my particular situation I could roll my own API request that just assumes all selfhosted runners are the same (which in my case they are, for now), and checks if there is at least one online+nonbusy one, I would prefer not rolling a custom action nor ask action developer to support "labelless workers", when the right thing is to fix the assignment of labels. ↩

[WyriHaximus]

Could also confirm this issue. However, after doing a clean install and forcing a replacement of the CRDs the extra labels are now showing up in the UI and runners start based on those extra labels.

Things got more interesting. After that comment, I started playing with adding several more labels, they are not updated. However, doing a reinstall (helm uninstall and then helm install again) resolves this; labels show up on GitHub.com again and jobs are being executed.

[ryanb74]

I also have this issue with one of my customers.

I tried deleting everything including the CRDs and then reinstalling the charts but it seems like that didn't work (maybe I did something wrong while doing so).

The labels don't appear on github entreprise UI and the jobs targeting these labels can't be picked up as well

[jburgos115]

I also have this issue with one of my customers.

I tried deleting everything including the CRDs and then reinstalling the charts but it seems like that didn't work (maybe I did something wrong while doing so).

The labels don't appear on github entreprise UI and the jobs targeting these labels can't be picked up as well

same behavior here, custom labels don't appear and listener won't assign jobs despite recognizing them as available. 0.14.0 appears to be broken when targeting an enterprise instance, but the same deployment works as expected when simply redirecting the scale set to serve a github.com repository. sticking to 0.13.1 for now

[nikola-jokic]

Hey, I'm not sure what you mean exactly. The scale set should have labels. The runners belonging to that scale set doesn't necessarily have these labels in terms of UI. The runner gets provisioned for that particular scale set, therefore, the job assignment will inspect labels applied to the scaleset, and assign the job to the runner.

If the complaint is that the UI or the API when you query runners doesn't show these labels, then it is out of scope of ARC and you should definitely reach out using the Community Support Forum

[oznu]

I tried deleting everything including the CRDs and then reinstalling the charts but it seems like that didn't work (maybe I did something wrong while doing so).

This did work for me. GitHub Enterprise Cloud (not EMU).