Skip to content

Mac Signers Maintenance

Jump to bottom
Heitor Neiva edited this page Jun 8, 2026 · 2 revisions

Mac Maintenance

This document describes the manual steps required to maintain the Mac signers.

The list of available machines can be found here. Please keep the machine statuses up to date on that page.

Before performing maintenance, consider quarantining the machine to prevent jobs from being scheduled on it.

Scriptworkers are currently installed to /usr/local/builds. See modules/roles_profiles/manifests/profiles/mac_v3_signing.pp in the ronin_puppet repo for more information.

Deploying scriptworker updates

Deploying iscript code changes and updating the workers' Python packages use the same procedure: bump the pinned scriptworker-scripts revision so the mac signers pull the new commit. The data/update-scriptworker-revisions.py script rewrites the scriptworker_scripts_revision key in data/common.yaml to point at the latest scriptworker-scripts master commit.

  1. Check out the ronin_puppet repository.

  2. Run the revision update script:

    ./data/update-scriptworker-revisions.py

  3. Review the changes and confirm that scriptworker_scripts_revision now points to the expected scriptworker-scripts commit.

  4. Commit the changes and open a pull request against master.

  5. After the pull request is merged, update the macos-signer-latest branch:

    git fetch origin
git push origin origin/master:macos-signer-latest

  6. No manual worker intervention is required. After macos-signer-latest is updated, the workers pick up the change automatically during their next self-update check, which runs every 15 minutes.

SSH Access

We use Duo for authentication on the Mac signers. To access a Mac via SSH, your account must be added to the Duo allowlist.

If you’d like to cache your ssh authentication for 10 minutes per host, add the following to your ~/.ssh/config:

Host dep-mac-v4-signing* adhoc-mac-v4-signing* fx-mac-v4-signing* tb-mac-v4-signing* vpn-mac-v4-signing*
    ControlMaster auto
    ControlPath ~/.ssh/ssh-%C
    ControlPersist 10m

Updating the Widevine Repository

The Widevine repository is not revision-pinned and is not updated automatically. To update it manually:

  1. Update the Widevine repository.

  2. Remove the existing Widevine directory on the signing workers:

    # ff-prod or tb-prod
rm -rf /usr/local/builds/scriptworker/widevine

# dep environments
rm -rf /usr/local/builds/{dep1,dep2,tb-dep,etc}/widevine

  3. When you run Puppet (see below), it will repopulate and reinstall Widevine with the latest code.

Pausing Periodic Puppet

If you’re performing maintenance on a Mac (remember to quarantine it first!), you may want to temporarily disable periodic Puppet runs.

You can do this in one of two ways:

  • Modify /usr/local/bin/periodic-puppet.sh to exit early before running Puppet.
  • Unload the com.mozilla.periodic.plist LaunchDaemon (see below).

Ensure Puppet is not already running before unloading it, otherwise it may revert your changes.

Unloading Agents and Puppet

There are two types of LaunchDaemons on the signers:

  1. Puppet periodic daemon: /Library/LaunchDaemons/com.mozilla.periodic.plist
  2. Scriptworker daemons: /Library/LaunchDaemons/org.mozilla.scriptworker.*.plist

To unload the Puppet periodic daemon:

launchctl unload /Library/LaunchDaemons/com.mozilla.periodic.plist

To unload all Scriptworker daemons:

for d in /Library/LaunchDaemons/org.mozilla.scriptworker.*.plist; do launchctl unload "$d"; done

Wiping Secrets

Secrets must be wiped whenever a machine is being moved, repurposed, or retired — or if there’s any chance it may have been compromised.

Secrets are stored in the certs/ directory inside the scriptworker install location.

When wiping secrets, remove the following files (using rm -P to securely overwrite them):

  • ed25519_privkey (production only)
  • *.keychain
  • widevine*.crt

Provisioning Profiles

Some apps require Apple provisioning profiles to be installed manually on the mac signers. These profiles may need periodic updates—for example, when signing certificates are re-issued or when app entitlements change.

At the time of writing, the provisioning profile path is hardcoded in the signing code to the ./provisionprofiles directory. See the reference in hardened_sign.py

The authoritative list of provisioning profiles is managed in AppStoreConnect - Developer Profiles

When updating or rotating provisioning profiles, ensure the new profiles are deployed to the mac signers and placed in the expected directory.

Clone this wiki locally