DRAFT: Add RESTRICT=binpkg support

[Flowdalic]

With the continuing success the Gentoo binhost, it becomes important mark certain packages as unsuitable for the binhost. Typical examples include acct-user/, acct-group/, or virtual/* packages.

This introduces a new RESTRICT key: binpkg. If set by the ebuild, portage will not build a binary package for it.

Implementing RESTRICT=binpkg was easy. This could reduce the number of (unnecessary) binary packages on the Gentoo binhost significantly, which would reduce the pressure on the mirrors and probably even speed up things for the users. RESTRICT=binpkg was also requested in https://bugs.gentoo.org/542480#c1

However, then I read sam's comment at https://bugs.gentoo.org/911825#c1 about how not creating all binpkgs would break FEATURES=usepkgonly.

What we need is to add logic to the dependency resolution to consider a non-binary emerge of a RESTRICT=binpkg ebuild, even if FEATURES=usepkgonly is active. I think depgraph.py is probably the place where this should be added, but I couldn't determine the right location. @zmedico (or others): any suggestion would be welcome.

[eli-schwartz]

How does this interact with bindist? A naive approach would be to say, bindist is obsolete and no longer needed?

[mgorny]

What makes them "unsuitable"? Even if they are effectively no-ops, they should work fine from binary packages, and installing them from binary packages should still be more efficient than rebuilding from source.

[eli-schwartz]

Something that would be more interesting to me would be restricting the creation of binpkgs for packages named games-*/*-data, that are inevitably downloading multi-GB asset tarballs and extracting them directly to ${D}

Significant load on mirrors if it ends up being built. Practically zero utility. Most overhead from installing is the bandwidth used and raw file I/O, not executing src_* phases.

I agree that acct-* packages don't have a compelling use case here. They cause effectively no server load, they're absolutely tiny. Arguing that avoiding them is "faster" is also weird to me, because binpackages of an acct-user package use extremely trivial amounts of bandwidth in exchange for not forking out to bash and running phase functions nearly as much. Background downloads will guarantee the bandwidth use is a good tradeoff.

[Flowdalic]

I agree that acct-* packages don't have a compelling use case here.

I wouldn't be so sure about that. Most work in acct-*/* packages is done in pkg_ phases and there is a general additional overhead to processing binpkgs by portage (e.g., parsing the binpkg format).

They cause effectively no server load, they're absolutely tiny.

So your argument is that since their binpkgs are tiny, there is no harm in having acct-*/* as binpkgs, even though there is no real need to have them as binpkgs. But why shouldn't we stop creating binpkgs for acct-*/* packages once we have RESTRICT=binpkg?

But yes, there a packages with a clearer win here. You already mentioned games-*/*-data, but I also assume many font packages fall into this category. And, of course, probably all */*-bin packages.

How does this interact with bindist? A naive approach would be to say, bindist is obsolete and no longer needed?

Maybe RESTRICT=binpkg restricts the creation of a binary package, whereas RESTRICT=bindist restricts the (public?) distribution of a binary package, but not its creation. Not sure if that makes sense, but from a cursory look at it, they seem to be different.

[eli-schwartz]

I wouldn't be so sure about that. Most work in acct-*/* packages is done in pkg_ phases

Most work is done in python as part of portage: slowly, slowly slooooooooooooooooooooooooooowly spinning wheels to do dependency graph calculation inefficiently, profiling points to continual recreation of identical portage.dep.Atom objects for one-shot calculations followed by throwing them out, keeping the original strings around, and creating new ones on the fly the next time they are needed.

Most of the remaining work is process invocation overhead. EbuildPhase isn't exactly cheap (and nor is saving and restoring the environment). Although QA postinst checks do their fair share of the work, there are five src_* phases to run.

The question is not even about where more overall time is spent. The question is whether binpkgs save more time off the edges than they cost.

and there is a general additional overhead to processing binpkgs by portage (e.g., parsing the binpkg format).

This doesn't sound like a scientific measurement at all. More like "I assume so because gosh, file formats are funky and complicated".

I cannot follow the logic towards where this suggests meaningful single-threaded time being spent.

[thesamesam]

I think there's an argument to be made for it being a waste of time for the binpkg producer to compress something like signal-desktop-bin or similar.

But I want us to be really clear on what the point of RESTRICT=bindist and RESTRICT=binpkg are. I don't want to introduce another state if it turns out we can make bindist work with tweaks (like Kangie's #1440). Let's maybe discuss it more on the bug, given this is a serious design discussion that we'll want to refer to in future.