Linking error when building //oak_containers/syslogd:syslogd

[hvolkmer]

Based on latest checkout from today:

$ nix develop .#githubBuildShell --command just oak_containers_tests

(or)

$ nix develop .#githubBuildShell --command bazel build //oak_containers/syslogd:syslogd

[...]
  = note: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: warning: libcap.so.2, needed by /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so, not found (try using -rpath or -rpath-link)
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: warning: libgcrypt.so.20, needed by /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so, not found (try using -rpath or -rpath-link)
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: warning: liblzma.so.5, needed by /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so, not found (try using -rpath or -rpath-link)
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: warning: libzstd.so.1, needed by /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so, not found (try using -rpath or -rpath-link)
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: warning: liblz4.so.1, needed by /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so, not found (try using -rpath or -rpath-link)
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `gcry_md_read@GCRYPT_1.6'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `lzma_stream_decoder@XZ_5.0'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `gcry_md_close@GCRYPT_1.6'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `gcry_md_setkey@GCRYPT_1.6'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `ZSTD_DStreamOutSize'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `gcry_md_copy@GCRYPT_1.6'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `gcry_md_ctl@GCRYPT_1.6'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `LZ4_versionNumber'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `gcry_md_open@GCRYPT_1.6'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `ZSTD_getFrameContentSize'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `gcry_control@GCRYPT_1.6'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `ZSTD_decompressStream'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `lzma_code@XZ_5.0'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `gcry_md_write@GCRYPT_1.6'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `ZSTD_freeDCtx'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `ZSTD_createDCtx'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `ZSTD_isError'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `LZ4_decompress_safe'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `gcry_md_get_algo_dlen@GCRYPT_1.6'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `gcry_md_reset@GCRYPT_1.6'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `LZ4_decompress_safe_partial'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `ZSTD_getErrorCode'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `lzma_end@XZ_5.0'
          /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /home/hvolkmer/.cache/bazel/_bazel_hvolkmer/4c21be3a181200ae522617c823dd8903/external/oak_cc_toolchain_sysroot//usr/lib/x86_64-linux-gnu/libsystemd.so: undefined reference to `gcry_check_version@GCRYPT_1.6'
          collect2: error: ld returned 1 exit status
[...]

This can be fixed by applying:

diff --git i/oak_containers/syslogd/BUILD w/oak_containers/syslogd/BUILD
index 2940e2ce0d..d2acacae71 100644
--- i/oak_containers/syslogd/BUILD
+++ w/oak_containers/syslogd/BUILD
@@ -38,6 +38,12 @@ rust_binary(
         "@oak_crates_index//:signal-hook-tokio",
         "@oak_crates_index//:tokio",
         "@oak_crates_index//:tokio-stream",
+         ":zstd",
+         ":cap",
+         ":lzma",
+         ":lz4",
+         ":gcrypt",
+         ":gpg-error",
     ],
 )

@@ -45,3 +51,50 @@ alias(
     name = "syslogd",
     actual = ":oak_containers_syslogd",
 )
+
+cc_library(
+  name ="zstd",
+  srcs=[],
+  hdrs=[],
+  linkopts = ["-lzstd"],
+  visibility = ["//visibility:public"],
+)
+
+cc_library(
+  name ="cap",
+  srcs=[],
+  hdrs=[],
+  linkopts = ["-lcap"],
+  visibility = ["//visibility:public"],
+)
+
+cc_library(
+  name ="lzma",
+  srcs=[],
+  hdrs=[],
+  linkopts = ["-llzma"],
+  visibility = ["//visibility:public"],
+)
+
+cc_library(
+  name ="lz4",
+  srcs=[],
+  hdrs=[],
+  linkopts = ["-llz4"],
+  visibility = ["//visibility:public"],
+)
+cc_library(
+  name ="gcrypt",
+  srcs=[],
+  hdrs=[],
+   linkopts = ["-lgcrypt"],
+  visibility = ["//visibility:public"],
+)
+
+cc_library(
+  name ="gpg-error",
+  srcs=[],
+  hdrs=[],
+  linkopts = ["-lgpg-error"],
+  visibility = ["//visibility:public"],
+)

Is there a better way of fixing this?

[conradgrobler]

I am unable to reproduce this locally using the latest commit. The missing references should all be in the sysroot of the toolchain, so I am not sure what is causing this on your side.

What version of Bazel are you using?

Also, if you have Docker installed could you try running the following to see whether that works for you?

./scripts/docker_pull
./scripts/docker_run nix develop .#githubBuildShell --command bazel build //oak_containers/syslogd:syslogd

[hvolkmer]

Bazel version:

$ nix develop .#githubBuildShell --command bazel version
warning: Git tree '/home/hvolkmer/oak' is dirty
warning: input 'crane' has an override for a non-existent input 'nixpkgs'
Starting local Bazel server and connecting to it...
Build label: 6.5.0- (@non-git)
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Jan 1 00:00:00 1980 (315532800)
Build timestamp: 315532800
Build timestamp as int: 315532800

Running this via docker still gives the same result:

$ ./scripts/docker_run nix develop .#githubBuildShell --command bazel build //oak_containers/syslogd:syslogd
[...]
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
error: linking with `external/gcc_toolchain_x86_64/bin/gcc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="bazel-out/k8-fastbuild/bin/external/rust_linux_x86_64__x86_64-unknown-linux-gnu__nightly_tools/rust_toolchain/lib/rustlib/x86_64-unknown-linux-gnu/bin:bazel-out/k8-fastbuild/bin/ext
  = note: /root/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: cannot find -lzstd: No such file or directory
          /root/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: cannot find -lcap: No such file or directory
          /root/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: cannot find -llzma: No such file or directory
          /root/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: cannot find -llz4: No such file or directory
          /root/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: cannot find -lgcrypt: No such file or directory
          /root/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: cannot find -lgpg-error: No such file or directory
          collect2: error: ld returned 1 exit status

[...]

The lib files are indeed there, it just seems like the linker isn't finding them.

[hvolkmer]

I did a clean checkout and then the docker build worked, using "nix develop .#githubBuildShell --command bazel build //oak_containers/syslogd:syslogd" without docker still shows the error.

[conradgrobler]

Based on the warning above (warning: input 'crane' has an override for a non-existent input 'nixpkgs') I think the issue might be with your nix configuration/installation/version.

What version of nix are you using? What OS are you using on your development machine? Did you follow the nix installation instructions on https://github.com/project-oak/oak/blob/main/docs/development.md#install-nix?

[hvolkmer]

I tried this on two machines:

• my work box: Centos 9 + nix install as per the doc
• my home box: Nixos 24.11 + nix install as per the doc

[conradgrobler]

That is suprising. My best guess is that something from the host OS is leaking into the build process because of nix develop. All the test machines I have available are running Debian. The Docker container we use and the toolchain sysroot are also based on Debian, which is presumably why it works consistently. I will try to find other ways to reproduce, but in the meantime I would recommend using the docker_run script for the builds to avoid the issue.

[hvolkmer]

The docker build works as a workaround but it's very strange. The files exist, it's just the linker doesn't find them. See my initial comment. If you mess with the BUILD file it finds them (but I don't think that should be necessary). Bazel versions are the same and I don't see how OS environment via nix develop would influence the linker arguments/env - but that does seem to be what is happening.