Ship a built-in CLI for Pkg.jl

[MilesCranmer]

Julia should ship with a shell command-line interface for Pkg's pkg> mode.

Many other modern programming languages ship with a CLI for their package manager:

• Rust ships with cargo
• Node.js ships with npm
• Python ships with pip
• Go ships with go get/mod
• Ruby ships with gem
• Julia ships with... julia --project=. -e 'using Pkg; Pkg. 😕

I think this creates a lot of unnecessary friction.

Current workarounds

jlpkg is one such attempt to make this, but it exists as a third-party package. This means:

• It needs to be installed separately
• It is not easily discoverable (I only found it when searching for related issues)
• It needs to configured to point at the specific Julia to use (--julia flag)
• It's another dependency to manage

I find it easier to put the following in my bashrc instead:

jlpkg() {
  julia --project=. -e "using Pkg; Pkg.REPLMode.pkgstr(\"$*\")"
}

However, I do not think users should have to write such workarounds; such a feature should just ship with Julia.

Reasoning

1. Pkg.jl is already part of Julia

Pkg.jl is a standard library component, not a third-party package. It has a CLI already: pkg>. But it requires you to launch Julia to use it, which is really inconvenient when working with other command line tools like git.

2. It's trivially simple

Here is an entire pure-Julia implementation of this:

import Pkg
Pkg.REPLMode.pkgstr(join(ARGS, " "))

All flags can be forwarded to the underlying standard Julia binary.

3. Ecosystem consistency

When teaching Julia or onboarding developers, we have to explain why Julia uniquely lacks this tooling. I worry that it makes Julia feel less mature than it actually is.

Implementation Options

I think one of the following would be nice:

Option A: Ship a minimal script

Include a dead-simple jlpkg script with Julia, such as the following

import Pkg
Pkg.REPLMode.pkgstr(join(ARGS, " "))

That's it. No dependencies, no complexity.

Option B: Add a julia --pkg flag

Build it into julia itself: julia --pkg-{cmd} {arg}. For example, julia --pkg-add LinearAlgebra. This keeps everything integrated in one binary.

However, this is a bit less convenient, and probably overloads too much into one CLI.

Option C: Include with juliaup

Since many users install Julia via juliaup, include a jlpkg command there.

Edit: JuliaLang/juliaup#1230

Option D: Precompiled binary

Use PackageCompiler to create a fast-startup binary version.

Prior Art & Related Work

• Will Pkg3.jl support CLI? Pkg.jl#168 (2017): Previous request asking for this
• Flag to run Pkg command from command line #30943 (2019): Another similar request
• https://github.com/fredrikekre/jlpkg: Third-party CLI implementation by @fredrikekre with some usage by the community
• https://github.com/ararslan/julia-pkg: Third-party CLI implementation by @ararslan, now archived

Summary

Many modern programming languages provide a CLI for package management out of the box. Julia has an excellent package manager in Pkg.jl, and actually already has a CLI for it. It just needs to expose it directly to the shell, with a separate executable. This would significantly improve the developer experience with nearly zero maintenance burden.

[MilesCranmer]

Here is an example implementation of juliapkg within juliaup: JuliaLang/juliaup#1230

To try it out:

cargo install --git https://github.com/MilesCranmer/juliaup.git --branch jlpkg \
              --features juliapkg \
              --bin juliapkg

and then, e.g.,

juliapkg add LinearAlgebra
juliapkg +1.11 test

It shares the same methods with juliaup, so you can set a channel the same way. All Julia flags get forwarded as well.

[ararslan]

• https://github.com/ararslan/julia-pkg: Third-party CLI implementation by @ararslan, now archived

lol, I had forgotten about this

Option B: Add a julia --pkg flag

Build it into julia itself: julia --pkg-{cmd} {arg}. For example, julia --pkg-add LinearAlgebra. This keeps everything integrated in one binary.

However, this is a bit less convenient, and probably overloads too much into one CLI.

I actually like this the best, but I'd spell it as julia --pkg add LinearAlgebra, i.e. take everything after --pkg to be input as it would be entered in the Pkg REPL mode. That feels like pleasant symmetry with -e/--eval and doesn't add an additional file to the mix.

Perhaps Frederik's jlpkg is evidence to the contrary, but it feels like there are enough command line flags that one could plausibly want to pass to the Julia process executing the Pkg commands that you'd end up largely duplicating the command line interface. Or maybe you already can get that from jlpkg with something like jlpkg --julia="path/to/julia --options --here" (I've never actually used it, so that's a guess based on the README).

[MilesCranmer]

In retrospect I don't think that option is feasible because some of the Pkg.jl commands have flags of their own. e.g., here is the shell completion menu for status:

[MilesCranmer]

but it feels like there are enough command line flags that one could plausibly want to the Julia process executing the Pkg commands that you'd end up largely duplicating the command line interface. Or maybe you already can get that from jlpkg with something like jlpkg --julia="path/to/julia --options --here" (I've never actually used it, so that's a guess based on the README).

The really nice thing about JuliaLang/juliaup#1230 is that you get this for free. It is actually calling the same launch command as julialauncher.rs is calling, and forwarding all flags. So the following will work:

juliapkg +1.11 --threads=2 precompile

This will not only use the same machinery to select the 1.11 channel binary, it will also forward --threads=2 to the launch call, and finally pass precompile to pkgstr.

[ararslan]

In retrospect I don't think that option is feasible because some of the Pkg.jl commands have flags of their own.

Sure it is:

take everything after --pkg to be input as it would be entered in the Pkg REPL mode.

So e.g. julia --pkg status --outdated would provide --outdated to Pkg since it follows --pkg. Any command flags intended for the Julia process would need to be put before --pkg, which is not unusual as far as command line interfaces go. The thread flag example would be julia --threads=2 --pkg precompile.

[MilesCranmer]

Sorry, I was referring specifically to shell completions. I don't think any shell completion generation tool would know how to handle order-dependent flag completions, where, e.g., --outdated is allowed as a flag, but only after --pkg status.

More generally I worry that adding subcommands via a --pkg flag would make the Julia command CLI menu feel semantically overloaded?

Also worth noting that in other languages when both package management and execution are done with one binary, it is usually split by subcommands, like:

npm install ...
npm run ...

But having julia ... for running but julia --pkg ... for package management feels unusual. So one natural split would be to have a separate command for package management, such as juliapkg

[mbauman]

Regardless of any "nicer" alias bikeshedding, this seems like this is a good candidate for what julia -m Pkg should do, yeah?

[MilesCranmer]

Definitely. See JuliaLang/juliaup#1230 (comment) for further comments on this idea

[MilesCranmer]

Now, the other side of your question: is julia -m Pkg practical as a CLI? I think this depends on letting it set suitable startup defaults for package manager contexts, because julia --project=. --startup-file=no -m Pkg status is too unwieldy to be practical.

However, the flip side is that designing dynamic defaults, like julia -m Pkg defaulting to --project=. rather than --project=@v1.11, again feels like we are shoving too much into a single command. So I think an advantage of a separate juliapkg is that it can alter those defaults by design (as in the juliaup PR) without messing with user's mental model for the original command.

In any case I would say a dedicated juliapkg entry point is also useful as you get shell completions and fewer pinky strains. Similar to other cases: python -m pip has pip, npm exec -- has npx, ruby -S gem has gem

[visr]

Another possible alternative to having separate julia* tools may be a single jl tool with different commands that get added over time. We could take inspiration from the uv commands, so perhaps adding a package can for example become jl add Example. Actually I see Ruby folks were similarly inspired to announce rv yesterday.

This would be for more than just Pkg, but also run, build and more.

[MilesCranmer]

I like that too.

It's funny how simple this is to add; Pkg.jl's existing REPL mode is like 95% of the way there, it just needs a shell CLI. juliapkg actually has nearly everything except run and fmt. So all you do is branch to a regular Julia call on run, keeping the --project=. --startup-file=no defaults, and you basically have a full uv/npm/rv-like CLI for development.