Updates

Author: infinisil

source/tutorials/source-file-selection.md

@@ -2,6 +2,8 @@
 # Source file selection
 <!-- Note on title choice: While there's more uses outside of sources, it's by far the most prominent one -->
 
+<!-- TODO: Switch all mentions of unstable to stable once 23.11 is out -->
+
 To build a local project in a Nix derivation, its source files must be accessible to the builder.
 But since the builder runs in an isolated environment (if the [sandbox](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-sandbox) is enabled),
 it won't have access to the local project files by default.
@@ -27,7 +29,6 @@ $ cd select
 $ nix-shell -p niv --run "niv init --nixpkgs nixos/nixpkgs --nixpkgs-branch nixos-unstable"
 ```
 
-<!-- TODO: Switch to 23.11 once out -->
 :::{note}
 For now we're using the nixos-unstable channel, since no stable channel has all the features we need yet.
 :::
@@ -46,7 +47,10 @@ let
     inherit system;
   };
 in
-pkgs.callPackage ./package.nix { }
+{
+  package = pkgs.callPackage ./package.nix { };
+  inherit pkgs;
+}
 ```
 
 In this tutorial we'll experiment with different `package.nix` contents, while keeping `default.nix` the same.
@@ -62,7 +66,7 @@ runCommand "hello" { } ''
 
 And try it out:
 ```shell-session
-$ nix-build
+$ nix-build -A package
 this derivation will be built:
   /nix/store/kmf9sw8fn7ps3ndqs31hvqwsa35b8l3g-hello.drv
 building '/nix/store/kmf9sw8fn7ps3ndqs31hvqwsa35b8l3g-hello.drv'...
@@ -76,7 +80,9 @@ but we'll omit that for the sake of the tutorial, since we only need the build l
 This also makes it easier to build it again, since successful derivation builds would get cached.
 From now on we'll also make build outputs a bit shorter for the sake of brevity.
 
-## Builtin coercion of paths to strings
+## Builtins
+
+### Coercion of paths to strings
 
 The easiest way to use local files in builds is using the built-in coercion of [paths](https://nixos.org/manual/nix/stable/language/values.html#type-path) to strings.
 
@@ -134,7 +140,7 @@ The two main ways to coerce paths to strings are:
 
 These all do the same when built:
 ```shell-session
-$ nix-build
+$ nix-build -A package
 building '/nix/store/9fi0khrkmqw5srjzjsfa0b05hf8div4c-file-coercion.drv'...
 ++ cat /nix/store/j5lwpnlfrngks3bpidfr5hcrhgq0fy78-string.txt
 This is a string
@@ -149,7 +155,7 @@ $ nix-store --add $(pwd)/string.txt
 /nix/store/j5lwpnlfrngks3bpidfr5hcrhgq0fy78-string.txt
 ```
 
-## Built-in path coercion on directories
+### Coercion of directory paths to strings
 
 This path coercion also works on directories the same as it does on files, let's try it out:
 
@@ -165,7 +171,7 @@ runCommand "directory-coercion" {
 
 Running it gives us:
 ```shell-session
-$ nix-build
+$ nix-build -A package
 building '/nix/store/6ybg4v48xy8azhrnfdccdmhd2gr938f5-directory-interpolation.drv'...
 /nix/store/xdfchqpfx20ar9jil9kys99wc6hnm9zx-select
 |-- default.nix
@@ -195,7 +201,7 @@ Setting `src` will copy the resulting store path into the build directory and ma
 For the many commands that expect to be able to write to the current directory, this is great:
 
 ```shell-session
-$ nix-build
+$ nix-build -A package
 building '/nix/store/2cqd93fpnb4vqwkwmbl66dbxhndq1vhh-directory-coercion.drv'...
 unpacking sources
 unpacking source archive /nix/store/178fbwa8iwdl6b85yafksdbwlxf6mjca-select
@@ -228,7 +234,7 @@ However there are some subtle problems with this approach:
   - If you have any secrets in the current directory,
     they get imported into the Nix store too, exposing them to all users on the system!
 
-## `builtins.path`
+### `builtins.path`
 
 <!-- TODO: Use lib.cleanSourceWith instead and only briefly mention builtins.path? -->
 
@@ -239,7 +245,7 @@ And it allows selecting the files that should be included with its `filter` argu
 ```nix
 { runCommand, tree, lib }:
 let
-  source = builtins.path {
+  src = builtins.path {
     # The convention is to use "source"
     name = "source";
     path = ./.;
@@ -251,17 +257,20 @@ let
       || ! lib.hasSuffix ".nix" pathString;
   };
 in
-runCommand "builtins-path" {
+stdenv.mkDerivation {
+  name = "builtins-path";
+  inherit src;
   nativeBuildInputs = [ tree ];
-} ''
-  tree ${source}
-''
+  postInstall = ''
+    tree
+  '';
+}
 ```
 
 The shown filter will recurse into all directories and filter out all `.nix` files:
 
 ```shell-session
-$ nix-build
+$ nix-build -A package
 building '/nix/store/3x051rr6fainqi3a4mmmb06145m0j0mw-builtins-path.drv'...
 /nix/store/95mlqjmm13vd4ambw2pac5gj6i4wxcx4-source
 |-- nix
@@ -272,6 +281,146 @@ building '/nix/store/3x051rr6fainqi3a4mmmb06145m0j0mw-builtins-path.drv'...
 Writing more complex `filter` functions however is notoriously tricky,
 which is why it's not recommended to use this function directly.
 
+## File sets
+
+A better way to select local files is to use the [file set library](https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-fileset).
+This library is an abstraction built on top of `builtins.path`, with essentially the same functionality, but a much easier to use and safer interface.
+
+The library is based on the concept of file sets,
+a data type representing a collection of local files.
+File sets can be created, composed and used with the various functions of the library.
+
+The easiest way to experiment with the file set library is to first use it through `nix repl`.
+We can use the `pkgs` attribute exposed by `default.nix`
+to define `fs = pkgs.lib.fileset` as a shorthand alias for the library:
+
+```shell-session
+$ nix repl -f .
+[...]
+nix-repl> fs = pkgs.lib.fileset
+```
+
+### Basics
+
+It's probably the easiest to just jump right in
+by using the [`trace`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.trace) function,
+which pretty-prints the files included in a given file set:
+
+```shell-session
+nix-repl> fs.trace ./. null
+trace: /home/user/select (all files in directory)
+null
+```
+
+You might wonder where the file set here is, because we just passed a _path_ to the function!
+
+The key is that for all file set functions that expect a file set for an argument, it also accepts a path instead.
+Such a path argument is then [implicitly coerced](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset-path-coercion)
+to a file set that contains _all files under the given path_.
+We can see this from the trace `/home/user/select (all files in directory)`
+
+Note that in contrast to [coercion of paths to strings](#coercion-of-paths-to-strings),
+file sets _never_ add their files to the Nix store unless explicitly requested.
+So you don't have to worry about accidentally copying secrets into the store.
+
+:::{tip}
+The `trace` function pretty-prints its first agument and returns its second argument.
+But since you often just need the pretty-printing in `nix repl`, you can omit the second argument:
+
+```shell-session
+nix-repl> fs.trace ./.
+trace: /home/user/select (all files in directory)
+«lambda @ /nix/store/1czr278x24s3bl6qdnifpvm5z03wfi2p-nixpkgs-src/lib/fileset/default.nix:555:8»
+```
+:::
+
+This implicit coercion also works for files:
+
+```shell-session
+nix-repl> fs.trace ./string.txt
+trace: /home/user/select
+trace: - string.txt (regular)
+```
+
+We can see that in addition to the included file,
+it also prints its [file type](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-readFileType).
+
+And if we make a typo for a path that doesn't exist, the file set library adequately complains about it:
+
+```shell-session
+nix-repl> fs.trace ./string.nix
+error: lib.fileset.trace: Argument (/home/user/select/string.nix) is a path that does not exist.
+```
+
+### Adding files to the Nix store
+
+The file set library wouldn't be very useful if you couldn't also add its files to the Nix store for use in derivations.
+That's where [`toSource`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource) comes in.
+It allows us to create a Nix store path containing exactly the files in the file set, with its root at a specific path.
+
+Let's switch back to editing our `package.nix` file to try it out:
+
+```nix
+{ stdenv, tree, lib }:
+let
+  fs = lib.fileset;
+in
+stdenv.mkDerivation {
+  name = "filesets";
+  src = fs.toSource {
+    root = ./.;
+    fileset = ./string.txt;
+  };
+  nativeBuildInputs = [ tree ];
+  postInstall = ''
+    tree
+  '';
+}
+```
+
+Building this, we can see that indeed only `string.nix` was included:
+```
+$ nix-build
+building '/nix/store/1cdgj8b1jrg2k751jidj9564r66lpvj9-filesets.drv'...
+unpacking sources
+unpacking source archive /nix/store/083k2phzhdakc649z5ql8f7cyisws6wp-source
+[...]
+.
+`-- string.txt
+```
+
+We can also avoid using `tree` by using the `trace` function instead:
+
+```nix
+{ stdenv, lib }:
+let
+  fs = lib.fileset;
+  sourceFiles = ./string.txt;
+in
+fs.trace sourceFiles
+stdenv.mkDerivation {
+  name = "filesets";
+  src = fs.toSource {
+    root = ./.;
+    fileset = sourceFiles;
+  };
+}
+```
+
+And building it:
+
+```
+$ nix-build
+trace: /home/user/select
+trace: - string.txt (regular)
+```
+
+### Combinators
+
+The real benefit of the file set library lies in its combinator functions.
+
+
+
 <!--
 
 Mention lib.cleanSource, it's kind of the only function there's no good replacement for yet