nix: why does it use fetchTree at eval time?

[colemickens]

cc: @the-mikedavis

Quick question about the nix setup for helix. I reported some "bad behavior" internally at Determinate Systems, (to Eelco no less). I was surprised at things not quite being cached and optimistically skipped the way I might have expected.

Anyway, he (and now I) found it surprising that it utilizes fetchTree (eval time) rather than flake inputs.

Is there a reason for this?

Thanks,
Cole

[the-mikedavis]

Although I have used Nix for a while, I can't say I'm very knowledgeable about it 😄

Is the suggestion to have the tree-sitter grammar repos in the inputs section in flake.nix? That seems alright to me if we can do that and also ensure that they're locked at the revisions pinned in languages.toml. But I'm not sure that's possible - wouldn't the locked revisions then live in flake.lock? I don't want to have two sources of truth for the locked revisions (languages.toml and flake.lock).

[colemickens]

cc: @edolstra any chance I could get you to chime in here?

@the-mikedavis can you render out the languages.toml? You can access the locked revs from the flake and interpolate them into the languages.toml, potentially?

[the-mikedavis]

Ideally I want languages.toml to be the source of truth itself rather than have Nix generate / render it since I want the contribution process (for language-support changes especially) to be as straightforward as possible.

In the long run we want to avoid depending on so many different git repos for language support. We would have either a separate repository or an orphan branch in this repository where we would generate the parser.c, vendor the scanner.c (if it exists) and hold the query files for all languages together. Then Helix would clone that repository and compile the grammars you choose. This would cut way down on the disk footprint of a from-source install and probably make the flake much much simpler as well.

[colemickens]

The biggest downside of this is that it means these not-flake-inputs are not cacheable at all by Nix.

Which also means that anytime any of the tree-sitter grammars are unavailable (increasingly... quite common), I cannot rebuild my systems.

(I also acknowledge that I haven't put any effort in to help address this either. To be honest, I keep hoping for a release but... idk Helix is basically solely responsible for my CI taking so incredibly long on every single run.)

[archseer]

The grammars are pinned in languages.toml which acts as the canonical manifest. I'm not sure if you could somehow ingest that as the flake input.

[archseer]

For what it's worth, in regular builds we now skip failing grammars at build time (except in releases) which loses some consistency but at least doesn't completely block the build: 26a2d55

[colemickens]

Appreciate your thoughts here, a lot, @archseer!

In order:

1. builtins.fetchTree is a builtin fetcher - it is not a derivation (not cacheable), and executes sequentially, and blockingly at eval time.

2. Sadly, I suspect that fix doesn't affect the Nix side of things, since the helix nix code is evaluated (and thus fetchTree'd) far before helix binary is built or run.

side note: flake.nix is a special-case of Nix, it really can't do things like import from other files to fill out the inputs stanza.

My latest thought:

• after nix: Clean up grammar building #15884
• add some nix-compatible hash, or just flatly nix's hash into languages.toml
• change the Nix code to:
  • use the hash from the languages.toml file
  • switch builtins.fetchurl to pkgs.fetchurl where possible, use the hash from languages.toml

That keeps languages.toml as the source of truth, that lets inputs be maximally cacheable and build-time fetched... I'm just typing but maybe there's something here.

[archseer]

I think the issue is just the additional maintenance burden of maintaining yet another hash and leaking Nix specific details into the project: while me and Mike use Nix, most contributors don't.

From my usage, I use the flake but actually only leverage the dev shell: since I rebuilt often and test different branches I just manually compile and I keep the target/ directory in my $PATH.

The grammar situation has become a bit untenable and we are considering support tiers/disabling some on the default build. You can already configure your setup to only build certain grammars to cut on build time.

Grammars are technically standalone packages and we really shouldn't need to rebuild them outside of a languages.toml bump, it's a waste of build time. We bundle them together for convenience, but Alpine for example has packaged them separately and ships hx with no grammars by default. That's another option: disable grammars in the Nixbuild entirely, then manually build them via hx --grammar fetch && hx --grammar build (it just needs a C compiler present)

[colemickens]

I want to explicitly ack that (1) this is a niche problem (2) very, VERY small subset of users likely and especially, (3) that forcing Nix stuff into weird places is a bad idea. Both from maintenance/burden and also ... I try really hard to not be an "annoying pushy Nix person".

I don't have a lot more thoughts right now. But thanks again @archseer !