Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
0b39689
WIP: source file selection tutorial
infinisil Nov 19, 2023
197782d
Updates
infinisil Nov 19, 2023
836b6cd
Updates
infinisil Nov 19, 2023
239f2d9
Updates
infinisil Nov 20, 2023
6582681
Expand on file sets
infinisil Nov 20, 2023
de1e742
Rewrite to be about file sets
infinisil Nov 21, 2023
988945c
Move file
infinisil Nov 21, 2023
02ae281
Complete content
infinisil Nov 21, 2023
33ff09d
Polish
infinisil Nov 21, 2023
436b4aa
Briefly mention problems with builtin features
infinisil Nov 21, 2023
f8427e5
Minor change
infinisil Nov 21, 2023
bd3a71f
Update source/tutorials/file-sets.md
infinisil Nov 21, 2023
17e859e
more informative title
fricklerhandwerk Nov 22, 2023
285fd17
shorten introduction
fricklerhandwerk Nov 22, 2023
6a61be9
clarify wording
fricklerhandwerk Nov 22, 2023
7e7c6d7
reword learning objective
fricklerhandwerk Nov 22, 2023
8402ecf
Basics -> File sets
fricklerhandwerk Nov 22, 2023
06caaae
clarify wording
fricklerhandwerk Nov 22, 2023
d388d6d
add link to command reference
fricklerhandwerk Nov 22, 2023
76905ab
shorten introduction to `trace`
fricklerhandwerk Nov 22, 2023
9fd9700
use log ellipsis as in the other tutorials
fricklerhandwerk Nov 22, 2023
5ca3739
shorten introduction to path arguments
fricklerhandwerk Nov 22, 2023
568925e
move tip before the other notes
fricklerhandwerk Nov 22, 2023
97bbe4f
restructure notes on copying files
fricklerhandwerk Nov 22, 2023
fb6f630
add link to flakes documentation, add emphasis
fricklerhandwerk Nov 22, 2023
767155b
trim down phrasing
fricklerhandwerk Nov 22, 2023
152953e
A local directory -> Example project
fricklerhandwerk Nov 22, 2023
51a78a0
channel -> channel branch
fricklerhandwerk Nov 22, 2023
a6e5f26
package.nix -> build.nix
fricklerhandwerk Nov 22, 2023
501a5f7
add instruction to add a source file
fricklerhandwerk Nov 22, 2023
ba4989c
follow convention from niv guide
fricklerhandwerk Nov 22, 2023
efe4ba4
simplify conclusion
fricklerhandwerk Nov 22, 2023
3d6ac0a
review pass on tutorial sections
fricklerhandwerk Nov 22, 2023
fa037e6
fix omission
fricklerhandwerk Nov 22, 2023
429bbb0
Update source/tutorials/file-sets.md
infinisil Nov 22, 2023
16db564
Update source/tutorials/file-sets.md
infinisil Nov 22, 2023
993bbc1
Apply suggestions from code review
infinisil Nov 22, 2023
8e3032a
Merge pull request #806 from fricklerhandwerk/source-file-selection-r…
infinisil Nov 22, 2023
a2ac364
Add Git to Vale's vocab
infinisil Nov 22, 2023
bc00f00
More consistency in using diffs and prompts
infinisil Nov 22, 2023
0d60fac
inputs -> sources
infinisil Nov 22, 2023
3befcc4
Apply suggestions from code review
infinisil Nov 22, 2023
3b2dfd8
Minor changes after review
infinisil Nov 22, 2023
b3f67ce
Combine two paragraphs
infinisil Nov 22, 2023
67b688a
Don't use nix.conf
infinisil Nov 22, 2023
e4700af
Update source/tutorials/file-sets.md
infinisil Nov 22, 2023
6afd231
Some updates
infinisil Nov 22, 2023
69e6dd6
Some changes from a full read
infinisil Nov 22, 2023
9ecfd01
Remove unecessary non-fileset part
infinisil Nov 22, 2023
fb3fb0c
Update source/tutorials/file-sets.md
infinisil Nov 23, 2023
b6700a9
Review in meeting
infinisil Nov 23, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 34 additions & 30 deletions source/tutorials/file-sets.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,10 @@ instead of https://nixos.org/manual/nixpkgs/unstable/#sec-fileset
-->

To build a local project in a Nix derivation,
its source files must be [accessible](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-sandbox)
its source files must be accessible
to the [derivation builder](https://nixos.org/manual/nix/stable/language/derivations#attr-builder).
Since by default the `builder` runs in an isolated environment that only allows reading from the Nix store,
Since by default, derivations get built in an [isolated environment](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-sandbox)
that only allows reading from the Nix store,
the Nix language has built-in features to copy local files to the store and expose the resulting store paths.

Using these features directly can be tricky however:
Expand All @@ -23,23 +24,27 @@ Using these features directly can be tricky however:
can address these problems.
However, it's often hard to express the desired path selection using the `filter` function interface.

In this tutorial you'll learn how to use the Nixpkgs [`fileset` library](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset) to work with local files in derivations.
It abstracts over built-in functionality and offers a safer and more convenient interface.
In this tutorial you'll learn how to use Nixpkgs' [`lib.fileset` library](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset) to work with local files in derivations.
It abstracts over built-in functionality with a safer and more convenient interface.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd revert both sentences. Apostrophising Nixpkgs looks really weird, and abstracting over X with Y, while grammatically correct can be misleading to be read as "built in functionality having a safer interface".


## File sets

The file set library is based on the concept of _file sets_,
The basic concept is that of a _file set_,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revert too (or if you want file set as singular, say "based on the concept of a file set". "The basic concept is" - of what? What's the subject? And starting with "the basic concept" even if you say "of the file set library" is probably not even important, and the original sentence said what we want: that the file set revolves around this basic concept.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the docs team meeting right now, changing it to

A file set is a data type representing a collection of local files.
File sets can be created, composed, and manipulated with the various functions of the library.

a data type representing a collection of local files.
File sets can be created, composed, and manipulated with the various functions of the library.

You can explore and learn about the library with [`nix repl`](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-repl):

```shell-session
$ nix repl -f channel:nixos-unstable
$ nix repl -f https://github.com/NixOS/nixpkgs/tarball/master
...
nix-repl> fs = lib.fileset
```

:::{note}
We're using the `master` branch here, because the covered library features are part of NixOS 23.11 and up, which has not been released at the time of writing.
:::

The [`trace`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.trace) function pretty-prints the files included in a given file set:

```shell-session
Expand All @@ -48,7 +53,7 @@ trace: /home/user (all files in directory)
null
```

All functions that expect a file set for an argument also accept a [path](https://nixos.org/manual/nix/stable/language/values#type-path).
All functions that expect a file set for an argument also accept a [path](https://nixos.org/manual/nix/stable/language/values#type-path) instead.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revert, because "instead" can be misinterpreted as "mutually exclusive".

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changing it to

All functions that expect a file set for an argument can also accept a path.

Such path arguments are then [implicitly turned into sets](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset-path-coercion) that contain _all_ files under the given path.
In the previous trace this is indicated by `(all files in directory)`.

Expand All @@ -64,14 +69,14 @@ trace: /home/user (all files in directory)
:::

Even though file sets conceptually contain local files, these files are *never* added to the Nix store unless explicitly requested.
You don't have to worry as much about accidentally copying secrets into the world-readable store.
So You don't have to worry as much about accidentally copying secrets into the world-readable store.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capitalisation. Not sure if the connective is really needed here. I would (and did) skip it.

@infinisil infinisil Nov 23, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changing to

Therefore you don't have to worry as much about accidentally copying secrets into the world-readable store.


In this example, although we pretty-printed the home directory, no files were copied.
This is in contrast to coercion of paths to strings such as in `"${./.}"`,
which copies the whole directory to the Nix store on evaluation!

:::{warning}
With [experimental Flakes](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), a local directory containing `flake.nix` is always copied into the Nix store *completely* unless it is a Git repository!
When using [experimental Flakes](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), local directories with a `flake.nix` are always copied into the Nix store *completely* unless it is a Git repository!

@fricklerhandwerk fricklerhandwerk Nov 23, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are "experimental Flakes"? Are there non-experimental flakes?

I still think that should be "With the flakes experimental feature enabled, ...", and if you insist it also needs nix-command, then I suggest writing that out.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When using the flakes and nix-command experimental features,
a local directory within a Flake is always copied into the Nix store completely unless it is a Git repository!

In the meeting

:::

This implicit coercion also works for files:
Expand Down Expand Up @@ -101,9 +106,6 @@ $ cd fileset
$ nix-shell -p niv --run "niv init --nixpkgs nixos/nixpkgs --nixpkgs-branch master"
```

:::{note}
We're using the `master` branch here, because the covered library features are part of NixOS 23.11 and up, which has not been released at the time of writing.
:::

Then create a `default.nix` file with the following contents:

Expand All @@ -130,13 +132,14 @@ $ echo hello > hello.txt
$ echo world > world.txt
```

Start with a minimal `build.nix` based on `stdenv.mkDerivation`:
Start with a minimal `build.nix` based on [`stdenv.mkDerivation`](https://nixos.org/manual/nixpkgs/stable/#part-stdenv):

```{code-block} nix
:caption: build.nix
{ stdenv }:
stdenv.mkDerivation {
name = "fileset";
# This doesn't use the file set library just yet!
src = ./.;
postInstall = ''
mkdir $out
Expand All @@ -158,15 +161,16 @@ building '/nix/store/36p7xhc0rr8jvslban0zba0f7aij8cmb-fileset.drv'...
/nix/store/zdljz7v5bhv1nnh5mdh3xf418cf7z622-fileset
```

You may have already noticed a problem:
Only one file is needed to build the derivation.
But passing the path to `hello.txt` will result in an error, because `mkDerivation` expects a directory:
Since only `hello.txt` is needed to build the derivation,
we can try passing it directly in `src`.
But this will result in an error, because `mkDerivation` expects a directory:

```{code-block} diff
:caption: build.nix
{ stdenv }:
stdenv.mkDerivation {
name = "fileset";
# This doesn't use the file set library just yet!
- src = ./.;
+ src = ./hello.txt;
postInstall = ''
Expand Down Expand Up @@ -312,7 +316,7 @@ result -> /nix/store/xknflcvjaa8dj6a6vkg629zmcrgz10rh-fileset
Since `src` refers to the whole directory, and its contents change when `nix-build` succeeds, Nix will have to start over every time.

:::{note}
This will also happen without the file set library, e.g. when setting `src = ./.;` directly.
This will also happen with `src = ./.`, without using the file set library.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence structure is potentially confusing. What you probably mean here is "... i.e., without using the file set library", but that makes it a lot more clunky. I'd revert that too.

:::

The [`difference`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.difference) function subtracts one file set from another.
Expand Down Expand Up @@ -370,7 +374,7 @@ Removing the `./result` symlink creates a new problem, though:
$ rm result
$ nix-build
error: lib.fileset.difference: Second argument (negative set)
(/home/user/select/result) is a path that does not exist.
(/home/user/fileset/result) is a path that does not exist.
To create a file set from a path that may not exist, use `lib.fileset.maybeMissing`.
```

Expand All @@ -392,7 +396,7 @@ This now works, using the whole directory since `./result` is not present:

```
$ nix-build
trace: /home/user/select (all files in directory)
trace: /home/user/fileset (all files in directory)
this derivation will be built:
/nix/store/zr19bv51085zz005yk7pw4s9sglmafvn-fileset.drv
...
Expand Down Expand Up @@ -427,7 +431,7 @@ Again, Nix will start from scratch:

```shell-session
$ nix-build
trace: /home/user/select
trace: /home/user/fileset
trace: - default.nix (regular)
trace: - nix (all files in directory)
trace: - build.nix (regular)
Expand Down Expand Up @@ -460,9 +464,7 @@ Create a file set containing a union of the files to exclude (`fs.unions [ ... ]
in
```

Changing any of the excluded files now doesn't necessarily cause a new build anymore.

Check it and modify one of the excluded files again:
Changing any of the excluded files now doesn't necessarily cause a new build anymore:

```
$ echo >> build.nix
Expand Down Expand Up @@ -597,7 +599,7 @@ this derivation will be built:
Only the specified files are used, even when a new one is added:

```shell-session
$ touch src/select.o src/README.md
$ touch src/select.o README.md

@fricklerhandwerk fricklerhandwerk Nov 23, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
$ touch src/select.o README.md
$ touch src/select.o

Why do we need another one? It's not used anywhere else. Not important though...

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

discussed in meeting, it's fine to have it


$ nix-build
trace: - build.sh (regular)
Expand All @@ -611,13 +613,13 @@ trace: - world.txt (regular)

## Matching files tracked by Git

If a directory is part of a Git repository, [`gitTracked`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource) automatically filters for files that are tracked by Git.
If a directory is part of a Git repository, passing it to [`gitTracked`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource) gives you a file set that only includes files tracked by Git.

Create a local Git repository and add all files except `src/select.o` and `./result` to it:

```shell-session
$ git init
Initialized empty Git repository in /home/user/select/.git/
Initialized empty Git repository in /home/user/fileset/.git/
$ git add -A
$ git reset src/select.o result
```
Expand Down Expand Up @@ -649,8 +651,9 @@ Building we get

```shell-session
$ nix-build
warning: Git tree '/home/user/select' is dirty
warning: Git tree '/home/user/fileset' is dirty
trace: /home/vg/src/nix.dev/fileset
trace: - README.md (regular)
trace: - build.nix (regular)
trace: - build.sh (regular)
trace: - default.nix (regular)
Expand All @@ -669,7 +672,8 @@ this derivation will be built:
This includes too much though, as not all of these files are needed to build the derivation as originally intended.

:::{note}
With [experimental Flakes](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), it's [not really possible](https://github.com/NixOS/nix/issues/9292) to use this function, even with
When using [experimental Flakes](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), it's [not really possible](https://github.com/NixOS/nix/issues/9292) to use this function, even with

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above for "experimental Flakes"



```shell-session
$ nix build path:.
Expand All @@ -681,7 +685,7 @@ However it's also not needed, because by default, `nix build` only allows access
## Intersection

This is where `intersection` comes in.
It allows creating a file set that consists only of files that are in _both_ of the two given file sets.
It allows creating a file set that consists only of files that are in _both_ of two given file sets.

@fricklerhandwerk fricklerhandwerk Nov 23, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
It allows creating a file set that consists only of files that are in _both_ of two given file sets.
It allows creating a file set that consists only of files that are in _both_ of the two given file sets.

This is more idiomatic, "both of the two x" seems to be a fixed expression.

Also, if you think about it, union takes exactly two file sets, they're definite; it's not arbitrary that it's two. "Both of the two it takes"

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed, it's fine without :)


Select all files that are both tracked by Git *and* relevant for the build:

Expand All @@ -707,7 +711,7 @@ This will produce the same output as in the other approach and therefore re-use

```shell-session
$ nix-build
warning: Git tree '/home/user/select' is dirty
warning: Git tree '/home/user/fileset' is dirty
trace: - build.sh (regular)
trace: - hello.txt (regular)
trace: - src
Expand Down