Building and packaging a mipsel-linux cross-compilation toolchain
The goal here is to build, from scratch, a suitable GNU compilation toolchain for some of our targets, based on little endian MIPS processors and running Linux (hence the mipsel-*-linux-gnu
, or mipsel-linux
for short architecture), though it is likely that the following would also work to build (big endian) mips-linux cross toolchains too.
The host computer will be a regular PC running Arch Linux. When building a cross compiler toolchain, a more complex terminology is used to refer to the involved architectures:
build
: the architecture on which the program is built (herei686-pc-linux-gnu
);host
: the architecture on which the program will be run (i686-pc-linux-gnu
, except for the libc for which it will bemipsel-linux
);target
: the architecture for which the program will operate (mipsel-linux
).
Versions
We want to reproduce the building environment of the rest of the target computer. We stick to the same version of the tools:
Building
As we plan future packaging, the files will be laid out in /usr
for the host tools, and /usr/mipsel-linux
for the needed target files (i.e. mostly the libc and include files).
binutils
The binutils are the simplest tools to build, only specifying the target and prefix will marvelously do the job.
binutils-2.14$ ./configure --target=mispel-linux --prefix=/usr binutils-2.14$ make binutils-2.14$ sudo make install
Compilation problem
When compiling for the target using the freshly built binutils
, one may enconter the following error message:
as: unrecognized option `-EL'
The problem is simple, as this only means that the binutils
have not been correctly installed (or the autotools haven't found them in the $PATH
), and the toolchain is using the host's binutils
instead of the mipsel-linux
ones. A simple solution is to check the proper installation of the new utilities, eventually rebuilding them with the correct prefixes.
gcc
The C cross-compiler has to be compiled first, in order to be able to build the glibc for the target.
It is mandatory to build gcc out of the original source tree (i.e. not even in a subdirectory of the source tree as it is “unsupported”). The building dir will be, originally, called gcc-build
and share the same parent directory as the sources.
This will be a lightweight version of the compiler (no shared libraries nor threads support). The following assumes that the source tree has been extracted from the gcc-core package. In case the complete gcc source package has been fetched instead, be sure to provide the additional –enable-languages=c
flag to the configure
script.
gcc-build$ ../gcc-3.3.4/configure --target=mipsel-linux --prefix=/usr \ > --with-gnu-as --with-gnu-ld \ > --disable-shared --disable-nls --disable-threads gcc-build$ make gcc-build$ sudo make install
It may be advisable to make symbolic links to the binaries in the target files directory:
/usr/mipsel-linux/bin$ ls -l total 4328 (...) lrwxrwxrwx 1 root root 25 2007-03-22 16:06 cpp -> /usr/bin/mipsel-linux-cpp lrwxrwxrwx 1 root root 25 2007-03-22 16:06 gcc -> /usr/bin/mipsel-linux-gcc lrwxrwxrwx 1 root root 31 2007-03-22 16:06 gcc-3.3.4 -> /usr/bin/mipsel-linux-gcc-3.3.4 lrwxrwxrwx 1 root root 28 2007-03-22 16:06 gccbug -> /usr/bin/mipsel-linux-gccbug lrwxrwxrwx 1 root root 26 2007-03-22 16:06 gcov -> /usr/bin/mipsel-linux-gcov (...)
glibc
The shared libraries support has been disabled in the bootstrap compiler, but we want to build a shared-libs enabled glibc. This will lead to problems as the linker will unsuccessfully search for libgcc_eh.a
. A simple solution is to first make a dummy symbolic link to libgcc.a
which will trick the linker.
ln -s libgcc.a /usr/lib/gcc-lib/mipsel-linux/3.3.4/libgcc_eh.a
As stated in this post on the linux-mips mailing list, applying a patch may be be necessary before compiling the c library. Practically, it turned out that this wasn't necessary.
As for gcc, glibc has to be built in a separate directory. We'll be using the same layout. After unpacking the glibc sources, one must remember, as the target is a Linux host, to extract the linuxthreads sources too.
The C library also needs kernel headers for the target machine. In this cas, this is Linux 2.4.27. It is necessary to configure the kernel for the headers to be in a valid state. The default parameters will be sufficient.
linux-2.4.27$ make ARCH=mips config
One can then start the building process.
glibc-build$ ../glibc-2.3.3/configure --build=i686-pc-linux-gnu \ > --host=mipsel-linux --prefix=/usr/mipsel-linux \ > --with-headers=../linux-2.4.27/include \ > --enable-add-ons=linuxthreads glibc-build$ make glibc-build$ sudo make install
Note that the –build
and –host
tags have to be specified instead of the –target
one, which is not relevant. Also note that the prefix is /usr/mipsel-linux
instead of the usual /usr
as we do not want to overwrite our host's libc with another version not compiled for the right architecture.
To complete the installation, it is necessary to copy some of the headers from the Linux source tree to the filesystem.
linux-2.4.27$ sudo cp -R include/linux/ /usr/mipsel-linux/include/ linux-2.4.27$ sudo cp -R include/asm-mips/ /usr/mipsel-linux/include/asm
Linux-specific behavior
When building for a Linux host with a prefix set to /usr
, instead of installing the libraries in /usr/lib
, the building system will put them as the base libraries for the OS, in /lib
, and update the linker scripts accordingly. You may get error messages like the following due to this.
ld: cannot find //libc.so.6 collect2: ld returned 1 exit status
This would be a sign that something went wrong and you may have to edit the linker script (like libc.so
, GROUP
line) to the correct paths… Or recompile your libc with correct parameters.
This problem, however, should not occur following the above building method.
gcc (the full monty)
This time, the full gcc source archive is needed. As before, the building process will take place in a directory different from that of the sources.
gcc-build$ ../gcc-3.3.4/configure --target=mipsel-linux --prefix=/usr \ > --with-gnu-as --with-gnu-ld \ > --enable-languages=c,c++ \ > --disable-nls --enable-threads gcc-build$ make gcc-build$ sudo make install
While trying to configure or build gcc, you may encounter these errors:
checking for object suffix... configure: error: installation or configuration problem; compiler does not work
cc1: error: bad value (i686) for -march
They may be due to a wrongly set C(XX)FLAGS
environment option containing the -march=i686
flag. Be sure to properly set this variable (i.e. by removing said flag).
Packaging
binutils
The binutils provide a version of libiberty.a
which is already present in the system. It is not necessary (and even possibly dangerous) to overwrite the preexisting version. It should be removed from the package.
gcc
The GNU Compiler Collection allso builds a version of libiberty.a
, which should be removed from the package, as for the binutils. Also, some of the manpages are already present in the system and should be either removed or renamed in order not to cause conflicts when installing the package.
Thou shalt not strip my binaries
Beware of auto-stripping of the binaries. This package contains libraries for the target system like libgcc.a
which provides basic functions too big to be inlined. Usually, strip
recognizes binaries that are not in the correct format and prints a warning message before ignoring them. It happens, however, that strip
actually does its jobs on some files on which it shouldn't, overwriting the ELF header of these files with the wrong architecture.
$ file _muldi3.o # INCORRECT FORMAT _muldi3.o: ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped
$ file _mul_df.o # CORRECT FORMAT _mul_df.o: ELF 32-bit LSB relocatable, MIPS, MIPS-I version 1 (SYSV), not stripped
When trying to cross compile for the target architecture and link against the defected libgcc.a
, the problem can be spotted when this error message is displayed
ld: skipping incompatible /usr/lib/gcc-lib/mipsel-linux/libgcc.a when searching for -lgcc
Using objdump
and grep
to remove all correctly headed objects confirms the problem:
$ mipsel-linux-objdump -a /usr/lib/gcc-lib/mipsel-linux/libgcc.a | grep format | grep -v elf32-tradlittlemips _muldi3.o: file format elf32-little
To solve the issue, it is necessary to prevent the packaging system from stripping all the binaries and libraries (options=(NOSTRIP)
for Arch Linux), eventually stripping manually those for which strip
is safe (i.e. binaries and libs for the host machine).
glibc
When installing the libc in a alternative root, one must not use the usual
DESTDIR
make variable, but pass the path using install_root=…
.
Some binaries for the target got creating during the build, and have been installed in /usr/mipsel-linux/{bin,libexec,sbin}
. They can be removed as the host computer will never be able nor need to run them. Infopages and locales information has also been generated but are not strictly necessary, it has then be decided to remove them from the final package
gcc
As before, libiberty.a
and some manpage have to be removed or renamed from the final package
Errors when using the cross-compiler
Cannot find library whereas it is present
Sometimes, linking applications for the target arch may fail with the following message.
mipsel-linux-ld: skipping incompatible /lib/libc.so.6 when searching for /lib/libc.so.6 mipsel-linux-ld: cannot find /lib/libc.so.6
This can also be the cause of configure failing pretending the compiler cannot create executables.
checking for C compiler default output file name... configure: error: C compiler cannot create executables See `config.log' for more details.
This happens when using a relocated build environment for the target. That is, libraries have been built to be installed in /
but where installed in, say, /sysroot/
on the build system. This usually does not cause any problem, except it some of the supposed library files are linker scripts (this seems to be really usual for libc6.so
and libpthread.so
), e.g. /sysroot/usr/lib/libc.so
:
/* GNU ld script Use the shared library, but some functions are only in the static library, so try that secondarily. */ OUTPUT_FORMAT(elf32-tradlittlemips) GROUP ( /lib/libc.so.6 /usr/lib/libc_nonshared.a )
Those script point the linker to the actual location of the desired libraries, but they may sometimes include full path to the object file, rather than relative location. Scripts with full path would then point the linker to the build system's libraries instead of the target's.
The cleanest solution to fix that is to modify those scripts to point to the actual location of the libraries on the build system. Here is an example shell script doing so. It depends on the file
command identifying linker scripts as “ASCII C program text”, and relocates the scripts to point to $targetfs
.
for FILE in $targetfs/lib/* $targetfs/usr/lib/*; do if test -f $FILE && file $FILE | grep -q ASCII; then if grep -q $targetfs $FILE; then continue else echo "Relocating ld script $FILE to use $targetfs..." sed -i "s# \(/lib/\)# $targetfs\1#g;s# \(/usr/lib/\)# $targetfs\1#g" $FILE fi fi done
Libtool replaces a -lLIB flag with an absolute path to the build system's native library
(Thanks to Sylvestre Ledru for pointing me in the very right direction.)
When linking using libtool, it sometimes happens that a command line that should have worked perfectly well, is replaced by one with some of the -lLIB
arguments replaced by an absolute path to the system library /usr/lib/libLIB.so
.
/bin/sh ../libtool --tag=CC --mode=link mipsel-linux-gcc -I.. '-DDEBUG_TRAP=__asm__("int $3")' -DAVAHI_DAEMON_RUNTIME_DIR=\"/var/run/avahi-daemon/\" -DAVAHI_SOCKET=\"/var/run/avahi-daemon/socket\" -DAVAHI_SERVICE_DIR=\"/etc/avahi/services\" -DAVAHI_CONFIG_FILE=\"/etc/avahi/avahi-daemon.conf\" -DAVAHI_HOSTS_FILE=\"/etc/avahi/hosts\" -DAVAHI_DBUS_INTROSPECTION_DIR=\"/usr/share/avahi/introspection\" -DAVAHI_CONFIG_DIR=\"/etc/avahi\" -I/tmp/cube-target/usr/include -Wall -W -pedantic -pipe -Wformat -Wfloat-equal -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wredundant-decls -Wmissing-noreturn -Wshadow -Wendif-labels -Wpointer-arith -Wbad-function-cast -Wcast-qual -Wcast-align -Wwrite-strings -Winline -L/tmp/cube-target/lib -L/tmp/cube-target/usr/lib -o avahi-daemon avahi_daemon-main.o avahi_daemon-simple-protocol.o avahi_daemon-static-services.o avahi_daemon-static-hosts.o avahi_daemon-ini-file-parser.o avahi_daemon-setproctitle.o avahi_daemon-check-nss.o ../avahi-common/libavahi-common.la ../avahi-core/libavahi-core.la /tmp/cube-target/usr/lib/libdaemon.so -lexpat -ldl mkdir .libs mipsel-linux-gcc -I.. "-DDEBUG_TRAP=__asm__(\"int \$3\")" -DAVAHI_DAEMON_RUNTIME_DIR=\"/var/run/avahi-daemon/\" -DAVAHI_SOCKET=\"/var/run/avahi-daemon/socket\" -DAVAHI_SERVICE_DIR=\"/etc/avahi/services\" -DAVAHI_CONFIG_FILE=\"/etc/avahi/avahi-daemon.conf\" -DAVAHI_HOSTS_FILE=\"/etc/avahi/hosts\" -DAVAHI_DBUS_INTROSPECTION_DIR=\"/usr/share/avahi/introspection\" -DAVAHI_CONFIG_DIR=\"/etc/avahi\" -I/tmp/cube-target/usr/include -Wall -W -pedantic -pipe -Wformat -Wfloat-equal -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wredundant-decls -Wmissing-noreturn -Wshadow -Wendif-labels -Wpointer-arith -Wbad-function-cast -Wcast-qual -Wcast-align -Wwrite-strings -Winline -o .libs/avahi-daemon avahi_daemon-main.o avahi_daemon-simple-protocol.o avahi_daemon-static-services.o avahi_daemon-static-hosts.o avahi_daemon-ini-file-parser.o avahi_daemon-setproctitle.o avahi_daemon-check-nss.o /tmp/cube-target/usr/lib/libdaemon.so -L/tmp/cube-target/lib -L/tmp/cube-target/usr/lib ../avahi-common/.libs/libavahi-common.so ../avahi-core/.libs/libavahi-core.so /usr/lib/libexpat.so -ldl /usr/lib/libexpat.so: could not read symbols: Invalid operation
This is quite similar to the problem above and due to the relocation of the target's sysroot
as a subtree of the build host. Libtool is being wrongly redirected to an inapropriate library directory due to a script LIB.la
, usually /usr/lib/libLIB.so
like the following.
# libLIB.la - a libtool library file # Generated by ltmain.sh - GNU libtool 1.5.10 (1.1220.2.131 2004/09/19 12:46:56) # # Please DO NOT delete this file! # It is necessary for linking the library. # The name that we can dlopen(3). dlname='libLIB.so.0' # Names of this library. library_names='libLIB.so.0.0.1 libLIB.so.0 libLIB.so' # The name of the static archive. old_library='libLIB.a' # Libraries that this one depends upon. dependency_libs=' -L//tmp/staging/mipsel-linux/lib' # Version information for libLIB. current=0 age=0 revision=1 # Is this an already installed library? installed=yes # Should we warn about portability when linking against -modules? shouldnotlink=no # Files to dlopen/dlpreopen dlopen='' dlpreopen='' # Directory that this library needs to be installed in: libdir='/usr/lib'
The source of the problem is on the last line, the libdir='/usr/lib'
statement, which should obviously be replaced to point to the actual sysroot of the target system. Here is a sample bash script doing the work.
for FILE in $targetfs/lib/*.la $targetfs/usr/lib/*.la; do if grep -q $targetfs $FILE; then continue else echo "Relocating libtool script $FILE to use $targetfs..." sed -i "s#^libdir='\(/lib\)'#libdir=$targetfs\1#g;s#^libdir='\(/usr/lib\)'#libdir=$targetfs\1#g" $FILE fi done