Build GCC with Musl for Raspbian
Musl is a libc replacement designed for embedded systems. It tries to be compatible with POSIX and where possible with glibc, the default c standard library for GCC. Its code is designed to be better for small memory systems.
One of the main differences with Musl, is that its still possible to build true statically linked binaries. With glibc, you can link statically, but glibc itself will still link dynamically to other libraries, particularly to libnss and related libraries. Libnss is a runtime configurable library that handles interfacing to a bunch of standard linux files ( like /etc/passwd ), and some services ( notably DNS resolution ).
Musl is the default libc on some embedded Linux’s like Alpine.
Musl, GLibc and Docker
I’ve been trying to get some of my c++ projects working in docker containers on Raspbian. Its pretty easy to get docker containers to work if you use standard tools that have been figured out for docker, and then write your scripts to use those tools ( SQL, Java, python etc. ) But if you want to run your own self written C++ programs its much harder.
If you link dynamically with glibc then to run your program successfully within docker you need to include all of your programs’ dynamic library dependencies, which in turn have their own dependencies. You may end up having to track down a lot of libraries. Not all dependencies are visible with ldd, e.g. you won’t see the dependencies on libnss for example. And even when you have it running – some services that depend on missing dependencies may not work right ( in my case DNS resolution in docker containers. )
One option is to link your program against musl. If you can achieve a complete static link, then by definition all the code you need other than the kernel is linked statically into your program and everything should work inside a docker container.
Several ways of getting Musl on Raspbian
1. Wrapper: If all you need is static linking of c programs against musl, then you can just install libmusl on Raspbian:
sudo apt-get install musl-dev musl-tools
This will install musl, along with some wrapper scripts that let you run gcc in a way that links with musl instead of glibc. Instead of calling gcc, you just call musl-gcc.
This is a good quick solution for c. It will not work for c++, because it will not link with a libstdc++ that is compatible with musl.
2. Pre-built cross compilers:
If you are willing to compile on x86, and copy your binaries to the target platform, then you can install pre-built GCC cross compilers. This will mean installing any other libraries you need on the cross compilation environment, which is not as easy to manage as if you compile in the target environment.
Its also possible to use buildroot to get a pre-built Gcc/Musl toolchain. I didn’t investigate this too deeply.
3. Build Gcc With Musl on Raspbian: This is what I decided to try.
How to build Gcc with Musl on Raspbian
Building this manually, by getting all the packages and configuring them seems pretty daunting. There is a lot to figure out.
Thankfully there are at least two scripts that are designed to build the entire thing with just a small set of configuration parameters:
It did it with the Gregor’s script. Its the one that is used to generate the x86 pre-built toolchains I found. That the only reason I chose it over the other one – which seems similarly good.
First download it:
git clone https://github.com/GregorR/musl-cross.git
It will put everything in the directory musl-cross, which will contain just a few script files and make files.
GCC requires a few additional packages to build: gmp, mpfr, mpc. musl-cross can be configured to build them, but the build for these failed for me. Instead of debugging it, I just installed them, since you can get pre-built ones for raspbian:
sudo apt-get install libgmp-dev libmpfr-dev libmpc-dev
In the musl-cross directory there is a script file config.sh. Here is now I configured mine for Raspbian:
# ARCH will be auto-detected as the host if not specified #ARCH=i486 #ARCH=x86_64 #ARCH=powerpc ARCH=arm #ARCH=microblaze #ARCH=mips #ARCH=mipsel CC_BASE_PREFIX=/opt/cross # If you use arm, you may need more fine-tuning: # arm hardfloat v7 TRIPLE=arm-linux-musleabihf GCC_BOOTSTRAP_CONFFLAGS="--with-arch=armv7-a --with-float=hard --with-fpu=vfpv3-d16" GCC_CONFFLAGS="--with-arch=armv7-a --with-float=hard --with-fpu=vfpv3-d16" # arm softfp #TRIPLE=arm-linux-musleabi #GCC_BOOTSTRAP_CONFFLAGS="--with-arch=armv7-a --with-float=softfp" #GCC_CONFFLAGS="--with-arch=armv7-a --with-float=softfp" MAKEFLAGS=-j8 # Enable this to build the bootstrap gcc (thrown away) without optimization, to reduce build time GCC_STAGE1_NOOPT=1 # GCC_BUILTIN_PREREQS=yes
If you uncomment the last line you don’t have to preinstall those three libraries, but doing it that way failed for me.
The first few times I tried compiling on a Raspberry Pi 2, the build failed with errors like:
g++ -c -O0 -g0 -DIN_GCC -DCROSS_DIRECTORY_STRUCTURE -fno-exceptions -fno-rtti /bin/bash: line 1: 26162 Killed build/genautomata... Makefile:2137: recipe for target 's-automata' failed make[2]: *** [s-automata] Error 137 make[2]: *** Waiting for unfinished jobs.... make[2]: Leaving directory '/home/rafael/muslgcc/musl-cross/gcc-5.3.0/build1/gcc' Makefile:4105: recipe for target 'all-gcc' failed make[1]: *** [all-gcc] Error 2 make[1]: Leaving directory '/home/rafael/muslgcc/musl-cross/gcc-5.3.0/build1' Makefile:858: recipe for target 'all' failed make: *** [all] Error 2 + die Failed to build gcc-5.3.0 + echo Failed to build gcc-5.3.0 Failed to build gcc-5.3.0 + exit 1
The key hint here is where it says /bin/bash: line 1: 26162 Killed. This is a sign that the program didn’t crash – but was killed by the operating system. This usually means the process ran out of memory.
To get past this I increased the swap space like this:
sudo dd if=/dev/zero of=/extraswap bs=1M count=4096 mkswap /extraswap sudo swapon /extraswap swapon -s
The command just creates a large empty file to use as swap space. The second command create a swap file system on it. The third command actually makes it into swap space. The last command is just to verify that the swap partition is successfully used. You should get output like:
Filename Type Size Used Priority /var/swap file 102396 32284 -1 /extraswap file 4194300 5020 -2
Then to actually compile run:
sudo ./build.sh 1> build.out 2> build.err & tail -f build.err build.out
You could just run it as build.sh, but then the output just goes to the shell, and if you lose connection you can no longer see the output of the build.
The build takes about 24 hours on a RPI2. This is mostly because a few files get into massive swapping. They seem to require something like 3GB+ to compile and the Pi only has a little less than 1GB of memory available for user processes. When this happens a single file can take over an hour to build.
Once the build is done you’ll want to get rid of the giant swap area:
sudo swapoff -v /extraswap sudo rm /extraswap
Using custom GCC with Musl
The new toolchain will not replace the default GCC that you can install on Raspbian – nor do you want it to. The new toolchain, with the configuration I used is build in /opt/cross/arm-linux-musleabihf. Everything is in there, the compiler, associated tools, and the musl standard c libraries and headers.
To use the new toolchain, first add it to your path:
PATH=/opt/cross/arm-linux-musleabihf/bin:$PATH
And call gcc tools with prefix:
arm-linux-musleabihf-gcc main.c -o main.o
The musl toolchain has some advantages and disadvantages. The main advantage is that you can build full static programs again. These run great in docker containers. They are also much more space efficient. For example one of my projects requires 45MB of binaries and shared libraries when built with glibc, is just 12MB built with musl, as a single executable. On the other hand my executable with glibc was 6MB vs 12MB for musl. So if you need multiple executables, sharing the same libraries will be much more efficient. If you have just one executable musl will be more space efficient.
But there are some significant disadvantages to using musl:
- There is no
execinfo.h. This is not a POSIX standard, its something specific to glibc. That means you can’t easily get stack traces. I’ve read that its possible to get your own stack traces withlibunwind, but this is something I did not explore. dlopen()does not supportRTLD_DEEPBINDM. This is another glibc extension. The project I need this for requires plugins loaded withdlopen, and doesn’t work right without deepbind. I had to give up on using musl for this project because of this. But I’ll keep the built toolchain around in case for projects that don’t require plugins.
