Skip to content

Commit 05405cb

Browse files
infinisilinfinisil
committed
Updates
1 parent 0b39689 commit 05405cb

1 file changed

Lines changed: 191 additions & 23 deletions

File tree

‎source/tutorials/source-file-selection.md‎

Lines changed: 191 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -6,45 +6,199 @@ To build a local project in a Nix derivation, its source files must be accessibl
66
But since the builder runs in an isolated environment (if the [sandbox](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-sandbox) is enabled),
77
it won't have access to the local project files by default.
88

9-
To make this work, the Nix language has certain builtin features to copy local paths to the Nix store,
9+
To make this work regardless, the Nix language has certain builtin features to copy local paths to the Nix store,
1010
whose paths are then accessible to derivation builders [^1].
1111

1212
[^1]: Technically only Nix store paths from the derivations inputs can be accessed,
1313
but in practice this distinction is not important.
1414

15-
These builtin features are very limited in functionality and are not recommended if you need to do anything more advanced.
15+
These builtin features are very limited in functionality and are not recommended for anything non-trivial. For more advanced use cases, the file set library should be used instead.
1616

17-
For advanced uses, use the file set library instead.
17+
In this tutorial you'll learn both how to use the builtins and the file set library.
1818

19-
## Builtins
19+
## Setting up a local experiment
2020

21-
### Path coercion to strings
21+
To experiment with source file selection, we'll set up a local project.
2222

23-
The most transparent builtin feature is coercion of paths to strings, such as:
24-
- Interpolating paths in strings:
23+
To start out, create a new directory, enter it, and set up `niv` to manage the Nixpkgs dependency:
24+
```shell-session
25+
$ mkdir select
26+
$ cd select
27+
$ nix-shell -p niv --run "niv init --nixpkgs nixos/nixpkgs --nixpkgs-branch nixos-unstable"
28+
```
29+
30+
<!-- TODO: Switch to 23.11 once out -->
31+
:::{.note}
32+
For now we're using the nixos-unstable channel, since no stable channel has all the features we need yet.
33+
:::
34+
35+
Then create a `default.nix` file with these contents:
36+
```nix
37+
{
38+
system ? builtins.currentSystem,
39+
sources ? import ./nix/sources.nix,
40+
}:
41+
let
42+
pkgs = import sources.nixpkgs {
43+
# Ensure purity
44+
config = { };
45+
overlays = [ ];
46+
inherit system;
47+
};
48+
in
49+
pkgs.callPackage ./package.nix { }
50+
```
51+
52+
In this tutorial we'll experiment with different `package.nix` contents, while keeping `default.nix` the same.
53+
54+
For now, let's have a simple `package.nix` to verify everything works so far:
55+
56+
```nix
57+
{ runCommand }:
58+
runCommand "hello" { } ''
59+
echo hello world
60+
''
61+
```
62+
63+
And try it out:
64+
```shell-session
65+
$ nix-build
66+
this derivation will be built:
67+
/nix/store/kmf9sw8fn7ps3ndqs31hvqwsa35b8l3g-hello.drv
68+
building '/nix/store/kmf9sw8fn7ps3ndqs31hvqwsa35b8l3g-hello.drv'...
69+
hello world
70+
error: builder for '/nix/store/kmf9sw8fn7ps3ndqs31hvqwsa35b8l3g-hello.drv'
71+
failed to produce output path for output 'out'
72+
```
73+
74+
We could also add `touch $out` to make the build succeed,
75+
but we'll omit that for the sake of the tutorial, since we only need the build logs.
76+
This also makes it easier to build it again, since successful derivation builds would get cached.
77+
From now on we'll also make build outputs a bit shorter for the sake of brevity.
78+
79+
## Builtin coercion of paths to strings
80+
81+
The easiest way to use local files in builds is using the built-in coercion of [paths](https://nixos.org/manual/nix/stable/language/values.html#type-path) to strings.
82+
83+
Let's create a local `string.txt` file:
84+
```
85+
$ echo "This is a string" > string.txt
86+
```
87+
88+
:::{.note}
89+
Flakes in Git directories requires git add.
90+
:::
91+
92+
The two main ways to coerce paths to strings are:
93+
- Interpolating paths in strings. To try that, change your `package.nix` file to:
94+
```nix
95+
{ runCommand }:
96+
runCommand "file-interpolation" { } ''
97+
(
98+
set -x
99+
cat ${./string.txt}
100+
)
101+
''
102+
```
103+
104+
:::{.note}
105+
Interpolation into bash scripts generally requires [`lib.escapeShellArg`](https://nixos.org/manual/nixpkgs/stable/#function-library-lib.strings.escapeShellArg) for correct escaping.
106+
In this case however, the interpolation results in a Nix store path of the form `/nix/store/<hash>-<name>`,
107+
and all valid characters of such store paths don't need to be escaped in bash.
108+
:::
109+
110+
- Using paths as derivation attributes. To try that, change your `package.nix` file to:
25111
```nix
26-
{ runCommandCC }:
27-
runCommandCC "test" { } ''
28-
cc ${./main.c} -o $out
112+
{ runCommand }:
113+
runCommand "file-interpolation" {
114+
stringFile = ./string.txt;
115+
} ''
116+
(
117+
set -x
118+
cat "$stringFile"
119+
)
29120
''
30121
```
31-
- Using paths as derivation attributes:
122+
123+
:::{.note}
124+
Nowadays using the explicit `env` attribute is recommended to set environment variables.
125+
`env` doesn't implicitly coerce paths to strings, so it requires using string intepolation instead:
32126
```nix
33-
{ stdenv }:
34-
stdenv.mkDerivation {
35-
name = "test";
36-
src = ./.;
37-
}
127+
{ runCommand }:
128+
runCommand "file-interpolation" {
129+
env.stringFile = "${./string.txt}";
130+
} ''
131+
(
132+
set -x
133+
cat "$stringFile"
134+
)
135+
''
38136
```
137+
:::
138+
139+
These all do the same when built:
140+
```shell-session
141+
$ nix-build
142+
building '/nix/store/9fi0khrkmqw5srjzjsfa0b05hf8div4c-file-interpolation.drv'...
143+
++ cat /nix/store/j5lwpnlfrngks3bpidfr5hcrhgq0fy78-string.txt
144+
This is a string
145+
```
146+
147+
As you can see, the `string.txt` file was hashed and added to the store,
148+
which then allowed the build to access it.
39149

40-
In both of these cases, the path gets imported into the Nix store
41-
and transparently converted to its Nix store path.
150+
The underlying functionality is the same as `nix-store --add` on an absolute path:
151+
```shell-session
152+
$ nix-store --add $(pwd)/string.txt
153+
/nix/store/j5lwpnlfrngks3bpidfr5hcrhgq0fy78-string.txt
154+
```
42155

43-
Some problems with this approach include:
44-
- The base name of the path influences the store path, even for `./.`.
45-
- All files in directories are unconditionally imported, even if they're unnecessary, impure or even meant to be private.
156+
## Built-in path coercion on directories
46157

47-
### `builtins.path`
158+
This path coercion also works on directories the same as it does on files, let's try it out:
159+
160+
```nix
161+
{ runCommand, tree }:
162+
runCommand "directory-interpolation" {
163+
# To nicely show path contents
164+
nativeBuildInputs = [ tree ];
165+
} ''
166+
tree ${./.}
167+
''
168+
```
169+
170+
Running it gives us:
171+
```shell-session
172+
$ nix-build
173+
building '/nix/store/6ybg4v48xy8azhrnfdccdmhd2gr938f5-directory-interpolation.drv'...
174+
/nix/store/xdfchqpfx20ar9jil9kys99wc6hnm9zx-select
175+
|-- default.nix
176+
|-- nix
177+
| |-- sources.json
178+
| `-- sources.nix
179+
|-- package.nix
180+
`-- string.txt
181+
```
182+
183+
But here we can get into some subtle trouble:
184+
- Note how the name of the store path ends with `-select`.
185+
So the name of the local directory influenced the result.
186+
187+
This means that whenever you rename the project directory
188+
or a collegue runs it in a different directory name,
189+
you're going to get different build results!
190+
191+
- All files in the directory are unconditionally added to the Nix store.
192+
193+
This means that:
194+
- Even if your build only needs a few files,
195+
changing _any_ file in the directory requires rebuilding the derivation,
196+
potentially wasting a lot of time.
197+
198+
- If you have any secrets stored in the current directory,
199+
they get imported into the Nix store too, exposing them to all users on the system!
200+
201+
## `builtins.path`
48202

49203
The above problems can be fixed by using [`builtins.path`](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-path) instead.
50204
It allows customising the name of the resulting store path with its `name` argument.
@@ -58,7 +212,7 @@ builtins.path {
58212
baseNameOf pathString != "default.nix";
59213
```
60214

61-
This function is notoriously hard to use correctly by itself.
215+
However, this function is notoriously hard to use correctly by itself.
62216

63217
<!--
64218
@@ -70,4 +224,18 @@ Section on file sets:
70224
- Using files from a file set as a derivation source
71225
- Migrate/integrate with lib.source-based filtering
72226
227+
A file structure to show?
228+
- `.envrc`:
229+
- `README.md`
230+
- `Makefile`
231+
- `nix`
232+
- `package.nix`
233+
- `sources.nix`
234+
- `sources.json`
235+
- `src`
236+
- `main.c`
237+
- `main.o`
238+
- `.gitignore`
239+
- `.git`
240+
73241
-->

0 commit comments

Comments
 (0)