Integrate llama into meson & ninja build system

[esneyder2]

I reproduced all the steps from this blog "https://github.com/nelhage/llama" to build LLVM with cmake and ninja without any problems. Now I’ m trying to integrate llama into our meson & ninja based build system. We build embedded applications for four different architecture and rely on crossfiles to specify the compilers and other tools used per architecture. We have our own SDK for each processor architecture that contains all the tools required for building applications for that architecture. Currently I'm trying to integrate llamacc and llamac++ into the crossfile for just one architecture (PowerPC). At this stage I have managed to use llama update_function to create a debian image that contains our PowerPC sdk push it to AWS ECR and set up a corresponding Lambda container function. The sdk lives in /opt/poky/ppc in the container, which is the same path that it can it be found in on local dev machines. However, I’m struggling to edit our cross-file to use llamacc and llamac++ as the c and cpp compilers.

It is not quite clear for me, how environment variables can be used to control the behaviour of llamacc and llamac++ such that they know where to look for the c and c++ compilers inside the lambda function. Should I be using the LAMACC_LOCAL_CC/CXX environment variables to control that ? . Any recommendation?

These are the steps I've gone through:

1. Created a dockerfile in images/ppc_gcc

FROM ghcr.io/nelhage/llama as llama
FROM debian:buster-slim
RUN apt-get update
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-unauthenticated \
        xz-utils \
        python
RUN mkdir -p /opt/poky/
COPY ./poky-ppcsdk.sh /opt/poky/ // poky-ppcsdk.sh is the script to install powerpc sdk
RUN /opt/poky/poky-ppcsdk.sh -y -d /opt/poky/ppc  // install sdk to /opt/poky/ppc
ENTRYPOINT ["/llama_runtime"]

2. Run llama update function
   llama update-function --create --build=images/ppc_gcc ppc_gcc

3. Updated cross-file for ppc architecture in our build system (ppc.txt)

#Cross-file for building for ppc architecture 

[binaries]
c = ['llamacc', '-mtune=e300c3', '--sysroot=/opt/poky/ppc-poky-linux']
cpp = ['llamac++', '-mtune=e300c3', '--sysroot=/opt/poky-poky-linux']
ar = '/opt/poky/ppc/powerpc-poky-linux/powerpc-poky-linux-ar'
strip = '/opt/poky/ppc/powerpc-poky-linux-strip'
pkgconfig = '/opt/poky/ppc/usr/bin/pkg-config'


[built-in options]

# Properties used by the compilers
cpp_args = ['-fno-exceptions', '-ffunction-sections', '-fdata-sections', '-fno-working-directory', '-Werror=poison-system-directories', '-Wall', '-std=c++17', '-Wnon-virtual-dtor', '-fdiagnostics-color=always',$
cpp_link_args = ['-fuse-ld=bfd', '-Wl,--no-keep-memory']
c_args = ['-fno-exceptions', '-ffunction-sections', '-fdata-sections', '-fno-working-directory', '-Werror=poison-system-directories', '-Wall', '-std=c99', '-fdiagnostics-color=always', '-g', '-gsplit-dwarf', '-$
c_link_args = ['-fuse-ld=bfd', '-Wl,--no-keep-memory']

[host_machine]
system = 'linux'
cpu_family = 'ppc'
cpu = 'e300c3'
endian = 'big'

4. Set up build directory using meson
   meson . .ppcbuild --cross-file ppc.txt

The error message that i'm getting is:
ERROR: Compiler llamacc --sysroot=/opt/poky/ppc-poky-linux can not compile programs.

[nelhage]

It is not quite clear for me, how environment variables can be used to control the behaviour of llamacc and llamac++ such that they know where to look for the c and c++ compilers inside the lambda function. Should I be using the LAMACC_LOCAL_CC/CXX environment variables to control that ?

Currently llamacc is hard-coded to look for cc and c++ inside the Lambda function, under the theory that since you control the Docker images it's easy enough to set up a symlink inside the image. If that doesn't work, we could potentially add in LLAMACC_REMOTE_{CC,CXX} options; I was just trying to avoid unnecessary configuration surface.

LLAMACC_VERBOSE=1 during a build should help debug what's going on a bit; I would also expect llamacc to emit an error along the lines of "cc: no such file" if you're able to get log output from the ./configure or whatever it is that's emitting that error. For configure it's usually in config.log IIRC, but I can't tell if you're using autoconf or something else.

[esneyder2]

Hi Nelhage, thanks for that info. We're not using autoconf, we're simply using meson to generate the build.ninja file which is then used by ninja to build.

I'm currently testing using a single compilation step used by my build system. When I run it with LLAMACC_VERBOSE=1, the console output makes we wonder if llamacc is even using my lambda function, looks like it is using my local cc and c++. Is there a way I can find out which lambda function llamacc is using?

Context:
I have used llama update_function to set up a lambda function called ppc-sdktest using my Dockerfile. My container contains our ppc sdk in /opt/poky/2.5.3/ppc and the c and c++ compilers in our sdk have been symlinked to c and c++. I try to force llamacc to use this function using the LLAMACC_FUNCTION env variable but it looks like it is the local c and c++ on my machine.

me@my-machine: ~/src/firmware/.ppcbuild$ LLAMACC_VERBOSE=1 LLAMACC_FUNCTION=ppc-sdktest llamac++ -v -mtune=e300c3 --sysroot=/opt/poky/2.5.3/ppc/sysroots/mpc83xx-poky-linux -IlibTextLog_ppc.a.p -I. -I.. -I../TextLog/include -IErrors -I../Errors -I../BOS/include -I../BOS/include/BOS/Test -I../BOS/platform/linux/include -I../Util/include -I../UnitTest/include -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -fno-exceptions -ffunction-sections -fdata-sections -fno-working-directory -Werror=poison-system-directories -Wall -std=c++17 -Wnon-virtual-dtor -fdiagnostics-color=always -g -gsplit-dwarf -DLINUX -DLABRINTH -DNDEBUG -Wno-unused -Wno-uninitialized -O2 -MD -MQ libTextLog_ppc.a.p/TextLog_source_StringRef.cpp.o -MF libTextLog_ppc.a.p/TextLog_source_StringRef.cpp.o.d -o libTextLog_ppc.a.p/TextLog_source_StringRef.cpp.o -c ../TextLog/source/StringRef.cpp

2021/08/16 23:35:43 run cpp -MM: ["c++" "-v" "-mtune=e300c3" "--sysroot=/opt/poky/2.5.3/ppc/sysroots/mpc83xx-poky-linux" "-fdiagnostics-color=always" "-Wall" "-Winvalid-pch" "-Wnon-virtual-dtor" "-fno-exceptions" "-ffunction-sections" "-fdata-sections" "-fno-working-directory" "-Werror=poison-system-directories" "-Wall" "-std=c++17" "-Wnon-virtual-dtor" "-fdiagnostics-color=always" "-g" "-gsplit-dwarf" "-Wno-unused" "-Wno-uninitialized" "-O2" "-MQ" "libTextLog_ppc.a.p/TextLog_source_StringRef.cpp.o" "-D" "_FILE_OFFSET_BITS=64" "-D" "LINUX" "-D" "LABRINTH" "-D" "NDEBUG" "-I" "libTextLog_ppc.a.p" "-I" "." "-I" ".." "-I" "../TextLog/include" "-I" "Errors" "-I" "../Errors" "-I" "../BOS/include" "-I" "../BOS/include/BOS/Test" "-I" "../BOS/platform/linux/include" "-I" "../Util/include" "-I" "../UnitTest/include" "-M" "-MF" "/dev/stdout" "../TextLog/source/StringRef.cpp"]

Using built-in specs.
COLLECT_GCC=c++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/8/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 8.3.0-6' --with-bugurl=file:///usr/share/doc/gcc-8/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-8 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 8.3.0 (Debian 8.3.0-6) 
COLLECT_GCC_OPTIONS='-fdiagnostics-color=always' '-v' '-mtune=e300c3' '-Winvalid-pch' '-fno-exceptions' '-ffunction-sections' '-fdata-sections' '-fno-working-directory' '-Werror=poison-system-directories' '-Wall' '-std=c++17' '-Wnon-virtual-dtor' '-g' '-gsplit-dwarf' '-Wno-unused' '-Wno-uninitialized' '-O2' '-MQ' 'libTextLog_ppc.a.p/TextLog_source_StringRef.cpp.o' '-D' '_FILE_OFFSET_BITS=64' '-D' 'LINUX' '-D' 'LABRINTH' '-D' 'NDEBUG' '-I' 'libTextLog_ppc.a.p' '-I' '.' '-I' '..' '-I' '../TextLog/include' '-I' 'Errors' '-I' '../Errors' '-I' '../BOS/include' '-I' '../BOS/include/BOS/Test' '-I' '../BOS/platform/linux/include' '-I' '../Util/include' '-I' '../UnitTest/include' '-M' '-MF' '/dev/stdout' '-shared-libgcc' '-march=x86-64'
 /usr/lib/gcc/x86_64-linux-gnu/8/cc1plus -E -quiet -v -I libTextLog_ppc.a.p -I . -I .. -I ../TextLog/include -I Errors -I ../Errors -I ../BOS/include -I ../BOS/include/BOS/Test -I ../BOS/platform/linux/include -I ../Util/include -I ../UnitTest/include -imultiarch x86_64-linux-gnu -isysroot /opt/poky/2.5.3/ppc/sysroots/mpc83xx-poky-linux -M -MF /dev/stdout -MQ libTextLog_ppc.a.p/TextLog_source_StringRef.cpp.o -D_GNU_SOURCE -D _FILE_OFFSET_BITS=64 -D LINUX -D LABRINTH -D NDEBUG ../TextLog/source/StringRef.cpp -mtune=e300c3 -march=x86-64 -std=c++17 -Winvalid-pch -Werror=poison-system-directories -Wall -Wnon-virtual-dtor -Wno-unused -Wno-uninitialized -fdiagnostics-color=always -fno-exceptions -ffunction-sections -fdata-sections -fno-working-directory -g -gsplit-dwarf -O2

cc1plus: error: -Werror=poison-system-directories: no option -Wpoison-system-directories
ignoring duplicate directory "/usr/include/x86_64-linux-gnu/c++/8"
ignoring nonexistent directory "/opt/poky/2.5.3/ppc/sysroots/mpc83xx-poky-linux/usr/local/include/x86_64-linux-gnu"
ignoring nonexistent directory "/opt/poky/2.5.3/ppc/sysroots/mpc83xx-poky-linux/usr/local/include"
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/8/../../../../x86_64-linux-gnu/include"
ignoring nonexistent directory "/opt/poky/2.5.3/ppc/sysroots/mpc83xx-poky-linux/usr/include/x86_64-linux-gnu"
#include "..." search starts here:
#include <...> search starts here:
 libTextLog_ppc.a.p
 .
 ..
 ../TextLog/include
 Errors
 ../Errors
 ../BOS/include
 ../BOS/include/BOS/Test
 ../BOS/platform/linux/include
 ../Util/include
 ../UnitTest/include
 /usr/include/c++/8
 /usr/include/x86_64-linux-gnu/c++/8
 /usr/include/c++/8/backward
 /usr/lib/gcc/x86_64-linux-gnu/8/include
 /usr/lib/gcc/x86_64-linux-gnu/8/include-fixed
 /opt/poky/2.5.3/ppc/sysroots/mpc83xx-poky-linux/usr/include
End of search list.
cc1plus: error: bad value (‘e300c3’) for ‘-mtune=’ switch
cc1plus: note: valid arguments to ‘-mtune=’ switch are: nocona core2 nehalem corei7 westmere sandybridge corei7-avx ivybridge core-avx-i haswell core-avx2 broadwell skylake skylake-avx512 cannonlake icelake-client icelake-server bonnell atom silvermont slm knl knm intel x86-64 eden-x2 nano nano-1000 nano-2000 nano-3000 nano-x2 eden-x4 nano-x4 k8 k8-sse3 opteron opteron-sse3 athlon64 athlon64-sse3 athlon-fx amdfam10 barcelona bdver1 bdver2 bdver3 bdver4 znver1 btver1 btver2 generic native
Running llamacc: Detecting dependencies: exit status 1

[nelhage]

Hm, yeah, it looks like you're running into trouble before you even make it to Lambda. The cpp -MM call llamacc is printing out is a local run of the preprocessor in order to detect the header files needed by a source file, so we know which files to bundle up and send to the cloud. That run will use the LLAMACC_LOCAL_CC/CXX compiler to decide which compiler to run locally.

[esneyder2]

Thanks nelhage, that makes sense :)