Publish to a new registry repo intermittently 500s (Quay 401): parallel blob pushes with no shared token cache

[aktech]

Publishing a workspace to an OCI registry (e.g. Quay) intermittently fails with HTTP 500. Retrying the same publish usually succeeds.

Server log for a failure:

ERROR unhandled service error || publish failed: failed to push pixi.toml:
POST "https://quay.io/v2/<namespace>/<repo>/blobs/uploads/":
response status code 401: unauthorized: access to the requested resource is not authorized

Cause: the three layer blobs (config, pixi.toml, pixi.lock) are pushed in parallel, and the auth client is built with no token cache, so each parallel push performs its own bearer-token exchange. On the first publish to a repo that doesn't exist yet, Quay auto-creates it on first push and the concurrent token requests race: one leg gets a token that isn't yet push-authorized and returns 401. There is no 401 retry, so the whole publish fails fast (~0.5s vs ~3s for a successful push). The failing leg varies (sometimes pixi.toml, sometimes pixi.lock), matching the race. Once the repo exists, retries succeed.

Parallel push with the un-cached client:

nebi/internal/oci/publisher.go

Lines 313 to 326 in c8696ef

	if c := newAuthClient(reg.Username, reg.Password); c != nil {
	remoteRepo.Client = c
	}
	blobs := make([]blobJob, 0, 3+len(assetDescs))
	blobs = append(blobs,
	blobJob{desc: configDesc, label: "config"},
	blobJob{desc: tomlDesc, label: "pixi.toml"},
	blobJob{desc: lockDesc, label: "pixi.lock"},
	)
	for i, d := range assetDescs {
	blobs = append(blobs, blobJob{desc: d, label: assets[i].RelPath})
	}
	if err := pushBlobsParallel(ctx, fs, remoteRepo, blobs, cfg.concurrency, cfg.progress); err != nil {

Auth client built without Cache:

nebi/internal/oci/browser.go

Lines 125 to 139 in c8696ef

	func newAuthClient(username, password string) *auth.Client {
	if username == "" && password == "" {
	return nil
	}
	return &auth.Client{
	Credential: func(ctx context.Context, hostname string) (auth.Credential, error) {
	return auth.Credential{
	Username: username,
	Password: password,
	}, nil
	},
	}
	}
	// ListRepositories queries the /v2/_catalog endpoint for a registry

Suggested fix (any/all):

1. Set a shared token cache on the auth client (client.Cache = auth.NewCache()) so the parallel blobs reuse one token.
2. Push the first blob serially to trigger repo creation, then fan out the rest.
3. Retry the push on 401 with a short backoff.

Workaround: click Publish again; the retry succeeds once the repo exists.