Multi-server support

[pfitzseb]

Juliaup currently does not support switching between multiple JULIAUP_SERVERs with different versiondbs, which makes it hard to implement third-party juliaup servers as a delivery mechanism for custom Julia builds.

Consider the following:

1. A user starts out with the official juliaup server and installs e.g. release
2. sets JULIAUP_SERVER to a URL that serves a versiondb containing custom Julia channels (with a newer versiondb version)
3. installs one of those new channels (this triggers a new versiondb to be written)
4. uses this newly added channel for a while
5. switches back to the original (or yet another server) and uses juliaup (or even just julia, because of auto-update checks)

In this situation, the custom channel will vanish without a trace, because the versiondb of the original server does not contain it.

A possible solution to this issue would be to mimic Pkg.jl's multi-registry design -- essentially, each juliaup server provides its own versiondb (independently versioned) and juliaup merges the contents of all of them when the user wants to use/install a channel.

[StefanKarpinski]

If we go with a merge approach, we may want JULIAUP_SERVER to be a :-separated list of servers. But I'm not sure about that—I think what people will want is more that they switch between servers. Another approach would be to key the whole set of channels and versions off of the server and when you switch servers it behaves like you have a completely separate juliaup install. We could share the actually version if they are identical, but I'm not sure if we have that information since versiondb is not (IIRC) content-addressed. Another issue with that approach would be garbage collection: suppose you use the default juliaup server and it installs a Julia version; then you change JULIAUP_SERVER and never change it back; do Julia versions installed for the default server never get deleted?

[StefanKarpinski]

Back on the merge approach: if we merge versiondb from multiple servers, what do we do about channel name collisions? Since JULIAUP_SERVER is ordered, we can always let the earlier ones win. That may work... What's a little weird about that is that the meaning of a channel can change if you change the value of JULIAUP_SERVER. Supporting multiple servers mainly makes it less necessary to change the server value ever.

[DilumAluthge]

Should channels have UUIDs? And then channel name is just a convenient way of accessing a channel, but the true definition of a channel is from its UUID?

[pfitzseb]

Another approach would be to key the whole set of channels and versions off of the server and when you switch servers it behaves like you have a completely separate juliaup install.

This is effectively what the VS Code extension implements, but imo it's bad UX to split the juliaup db like that.

The way I'd like the merge approach to work is that the local juliaup config file lists installed juliaup DBs (juliaup registries?) with associated URLs. If we give those registries a name then that would also be a way to uniquely address a version/channel. For clean merging/client-side deduplication it would be very nice if julia versions were addressed by content hash instead of an arbitrary version string (so the mapping would be registry->channel->hash->tarball path instead of registry->channel->versionstring->tarball path)

[DilumAluthge]

it would be very nice if julia versions were addressed by content hash

We should do this, regardless of what other changes we make.

[StefanKarpinski]

Agree. Content-address all the things.

Another way to do this would be to change from a model where users set JULIAUP_SERVER and instead have a list of "installed servers" (and presumably deprecate JULIAUP_SERVER). Then a local channel would be a name for a (server, channel) pair where server would have to be in the set of installed servers.

[pfitzseb]

Then a local channel would be a name for a (server, channel) pair where server would have to be in the set of installed servers.

I don't think there's a need for the concept of a local channel. Either all registries agree that $channel points to a certain tarball or the user needs to disambiguate by server anyways.

[davidanthoff]

What is the use-case for this? I never thought of the JULIAUP_SERVER as something that users would commonly switch, the only reason we introduced it was to accommodate folks behind country-level firewalls so that they could use a mirror.

[pfitzseb]

The goal is to distribute custom julia binaries as juliaup channels. And yes, JULIAUP_SERVER is the wrong tool for that, but also the only one that exists at the moment 😅 A registry-like design as proposed above would be much better.

[davidanthoff]

Is the idea that these are alternative official Julialang builds? In that case I would just add them to the version db? I'm still not understanding who this is targeting...

[StefanKarpinski]

It turns out that using juliaup to let people download and use custom Julia sysimgs is a pretty useful thing to do. We're doing this for Dyad and got the idea from Pumas.ai who are, I believe, doing the same thing for their product.

[StefanKarpinski]

No, it's Julia + a bunch of packages that otherwise take an absurd amount of time to precompile — think MKL + graphics stack + a bunch of product packages. We can tell customers to set JULIAUP_SERVER=juliahub.com and then run julia +dyad to get a Julia REPL with all the Dyad stuff baked in.

[StefanKarpinski]

What the heck... I posted sevreral messages here and GitHub just dropped them.

[StefanKarpinski]

Well, they showed up eventually, but man, GitHub is so bad lately.

[DilumAluthge]

it's Julia + a bunch of packages that otherwise take an absurd amount of time to precompile

And some of the packages are proprietary, so it doesn't make sense to put them in the default open-source Juliaup versiondb.