s3: publish update spends most of its time renaming indexes

[barnumbirr]

Setup

A public Debian archive published to Cloudflare R2 (apt.pkg.haus), three suites of about 68 packages each, published with aptly publish update on every ingest.

What I see

Per suite, publish update takes about 51s, and roughly 31s of that sits between Clearsigning file 'Release' and Cleaning up published repository, with no progress output in between. Across three suites that is about 93s, close to 60% of the whole publish step.

That window is RenameFiles(). FinalizeAll runs at deb/publish.go:1172 under "Finalizing metadata files...", signing follows, and return indexes.RenameFiles() is line 1256, before the cleanup message. RenameFiles is a serial loop over renameMap, and s3.RenameFile is a CopyObject followed by a Remove.

Reproduced locally against MinIO with logLevel: debug, publishing a small repo. Seven index files cost twenty one S3 operations:

S3: PutFile  'dists/bench/main/binary-amd64/Packages.tmp'
S3: PutFile  'dists/bench/main/binary-amd64/Packages.tmp.gz'
...  (7 PutFile in total, all to .tmp keys)
S3: RenameFile  dists/bench/main/Contents-amd64.tmp.gz
S3: Remove      'dists/bench/main/Contents-amd64.tmp.gz'
S3: RenameFile  dists/bench/main/binary-amd64/Release.tmp
S3: Remove      'dists/bench/main/binary-amd64/Release.tmp'
...  (7 RenameFile + 7 Remove, strictly after all uploads, one at a time)

Why it became noticeable

A package carrying about 19,500 files entered the archive, so Contents-<arch>.gz grew to 108 KB and there are four per suite (dist root and per component, two architectures). The publish step's median moved from 79s to 130s, n=8 either side. The index payload is what changed; the object count did not.

Measurements

Against MinIO behind a proxy adding 50ms each way, publishing a repository whose Contents is dominated by one package with 19,500 files, publish update, n=5 each:

	median
master	2703ms	range 2659-2760
renames concurrent, 4 in flight	1624ms	range 1622-1679, 40% faster
staging removed entirely	1240ms	range 1185-1333

The third row is a ceiling, not a proposal. Writing indexes straight to their final names gives up the property the .tmp set exists for: with staging, a failure part way through the upload leaves the live index set untouched and the publish retryable, and without it the archive is left with some indexes new and some old and no way back. The traces differ exactly there, PutFile 'Packages.tmp' against PutFile 'Packages'.

So concurrency captures about 75% of everything the rename phase could give back, and the remaining 25% is not worth the failure mode.

Also considered

Batching the Remove calls into one DeleteObjects, which already exists in RemoveDirs but not on this path. It can only reach part of what survives the concurrency change, and it needs a change to the PublishedStorage interface that every backend implements, so it looks like a poor trade on its own.

FinalizeAll is serial too, so the upload phase has similar headroom, but it calls the signer and I have not looked at whether that is safe to run concurrently.

Caveat on the size of the prize

In production each operation is averaging about 1.2s, which is higher than a server side copy of objects this size should cost, so something beyond round trip count may be involved there. The benchmark above is clean, but I would not promise that the full 93s is recoverable in every environment.

PR follows.