Building and packaging a mipsel-linux cross-compilation toolchain

The goal here is to build, from scratch, a suitable GNU compilation toolchain for some of our targets, based on little endian MIPS processors and running Linux (hence the mipsel-*-linux-gnu, or mipsel-linux for short architecture), though it is likely that the following would also work to build (big endian) mips-linux cross toolchains too.

The host computer will be a regular PC running Arch Linux. When building a cross compiler toolchain, a more complex terminology is used to refer to the involved architectures:

• build: the architecture on which the program is built (here i686-pc-linux-gnu);
• host: the architecture on which the program will be run (i686-pc-linux-gnu, except for the libc for which it will be mipsel-linux);
• target: the architecture for which the program will operate (mipsel-linux).

We want to reproduce the building environment of the rest of the target computer. We stick to the same version of the tools:

• GNU binutils 2.14;
• GNU Compiler Collection (gcc) 3.3.4;
• GNU standard C library (glibc) 2.3.3.

As we plan future packaging, the files will be laid out in /usr for the host tools, and /usr/mipsel-linux for the needed target files (i.e. mostly the libc and include files).

The binutils are the simplest tools to build, only specifying the target and prefix will marvelously do the job.

binutils-2.14$ ./configure --target=mispel-linux --prefix=/usr
binutils-2.14$ make
binutils-2.14$ sudo make install

When compiling for the target using the freshly built binutils, one may enconter the following error message:

as: unrecognized option `-EL'

The problem is simple, as this only means that the binutils have not been correctly installed (or the autotools haven't found them in the $PATH), and the toolchain is using the host's binutils instead of the mipsel-linux ones. A simple solution is to check the proper installation of the new utilities, eventually rebuilding them with the correct prefixes.

The C cross-compiler has to be compiled first, in order to be able to build the glibc for the target. It is mandatory to build gcc out of the original source tree (i.e. not even in a subdirectory of the source tree as it is “unsupported”). The building dir will be, originally, called gcc-build and share the same parent directory as the sources.

This will be a lightweight version of the compiler (no shared libraries nor threads support). The following assumes that the source tree has been extracted from the gcc-core package. In case the complete gcc source package has been fetched instead, be sure to provide the additional –enable-languages=c flag to the configure script.

gcc-build$ ../gcc-3.3.4/configure --target=mipsel-linux --prefix=/usr \
> --with-gnu-as --with-gnu-ld \
> --disable-shared --disable-nls --disable-threads
gcc-build$ make
gcc-build$ sudo make install

It may be advisable to make symbolic links to the binaries in the target files directory:

/usr/mipsel-linux/bin$ ls -l
total 4328
(...)
lrwxrwxrwx 1 root root     25 2007-03-22 16:06 cpp -> /usr/bin/mipsel-linux-cpp
lrwxrwxrwx 1 root root     25 2007-03-22 16:06 gcc -> /usr/bin/mipsel-linux-gcc
lrwxrwxrwx 1 root root     31 2007-03-22 16:06 gcc-3.3.4 -> /usr/bin/mipsel-linux-gcc-3.3.4
lrwxrwxrwx 1 root root     28 2007-03-22 16:06 gccbug -> /usr/bin/mipsel-linux-gccbug
lrwxrwxrwx 1 root root     26 2007-03-22 16:06 gcov -> /usr/bin/mipsel-linux-gcov
(...)

The shared libraries support has been disabled in the bootstrap compiler, but we want to build a shared-libs enabled glibc. This will lead to problems as the linker will unsuccessfully search for libgcc_eh.a. A simple solution is to first make a dummy symbolic link to libgcc.a which will trick the linker.

ln -s libgcc.a /usr/lib/gcc-lib/mipsel-linux/3.3.4/libgcc_eh.a

As stated in this post on the linux-mips mailing list, applying a patch may be be necessary before compiling the c library. Practically, it turned out that this wasn't necessary.

As for gcc, glibc has to be built in a separate directory. We'll be using the same layout. After unpacking the glibc sources, one must remember, as the target is a Linux host, to extract the linuxthreads sources too.

The C library also needs kernel headers for the target machine. In this cas, this is Linux 2.4.27. It is necessary to configure the kernel for the headers to be in a valid state. The default parameters will be sufficient.

linux-2.4.27$ make ARCH=mips config

One can then start the building process.

glibc-build$ ../glibc-2.3.3/configure --build=i686-pc-linux-gnu \
> --host=mipsel-linux --prefix=/usr/mipsel-linux \
> --with-headers=../linux-2.4.27/include \
> --enable-add-ons=linuxthreads
glibc-build$ make
glibc-build$ sudo make install

Note that the –build and –host tags have to be specified instead of the –target one, which is not relevant. Also note that the prefix is /usr/mipsel-linux instead of the usual /usr as we do not want to overwrite our host's libc with another version not compiled for the right architecture.

To complete the installation, it is necessary to copy some of the headers from the Linux source tree to the filesystem.

linux-2.4.27$ sudo cp -R include/linux/ /usr/mipsel-linux/include/
linux-2.4.27$ sudo cp -R include/asm-mips/ /usr/mipsel-linux/include/asm

When building for a Linux host with a prefix set to /usr, instead of installing the libraries in /usr/lib, the building system will put them as the base libraries for the OS, in /lib, and update the linker scripts accordingly. You may get error messages like the following due to this.

ld: cannot find //libc.so.6
collect2: ld returned 1 exit status

This would be a sign that something went wrong and you may have to edit the linker script (like libc.so, GROUP line) to the correct paths… Or recompile your libc with correct parameters.

This problem, however, should not occur following the above building method.

This time, the full gcc source archive is needed. As before, the building process will take place in a directory different from that of the sources.

gcc-build$ ../gcc-3.3.4/configure --target=mipsel-linux --prefix=/usr \
> --with-gnu-as --with-gnu-ld \
> --enable-languages=c,c++ \
> --disable-nls --enable-threads
gcc-build$ make
gcc-build$ sudo make install

While trying to configure or build gcc, you may encounter these errors:

checking for object suffix... configure: error: installation or configuration problem; compiler does not work

cc1: error: bad value (i686) for -march

They may be due to a wrongly set C(XX)FLAGS environment option containing the -march=i686 flag. Be sure to properly set this variable (i.e. by removing said flag).

The binutils provide a version of libiberty.a which is already present in the system. It is not necessary (and even possibly dangerous) to overwrite the preexisting version. It should be removed from the package.

• Arch Linux PKGBUILD.

The GNU Compiler Collection allso builds a version of libiberty.a, which should be removed from the package, as for the binutils. Also, some of the manpages are already present in the system and should be either removed or renamed in order not to cause conflicts when installing the package.

• Arch Linux PKGBUILD

Beware of auto-stripping of the binaries. This package contains libraries for the target system like libgcc.a which provides basic functions too big to be inlined. Usually, strip recognizes binaries that are not in the correct format and prints a warning message before ignoring them. It happens, however, that strip actually does its jobs on some files on which it shouldn't, overwriting the ELF header of these files with the wrong architecture.

$ file _muldi3.o # INCORRECT FORMAT
_muldi3.o: ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped

$ file _mul_df.o # CORRECT FORMAT
_mul_df.o: ELF 32-bit LSB relocatable, MIPS, MIPS-I version 1 (SYSV), not stripped

When trying to cross compile for the target architecture and link against the defected libgcc.a, the problem can be spotted when this error message is displayed

ld: skipping incompatible /usr/lib/gcc-lib/mipsel-linux/libgcc.a when searching for -lgcc

Using objdump and grep to remove all correctly headed objects confirms the problem:

$ mipsel-linux-objdump -a /usr/lib/gcc-lib/mipsel-linux/libgcc.a | grep format | grep -v elf32-tradlittlemips
_muldi3.o:     file format elf32-little

To solve the issue, it is necessary to prevent the packaging system from stripping all the binaries and libraries (options=(NOSTRIP) for Arch Linux), eventually stripping manually those for which strip is safe (i.e. binaries and libs for the host machine).

When installing the libc in a alternative root, one must not use the usual DESTDIR make variable, but pass the path using install_root=….

Some binaries for the target got creating during the build, and have been installed in /usr/mipsel-linux/{bin,libexec,sbin}. They can be removed as the host computer will never be able nor need to run them. Infopages and locales information has also been generated but are not strictly necessary, it has then be decided to remove them from the final package

• Arch Linux PKGBUILD

As before, libiberty.a and some manpage have to be removed or renamed from the final package

• Arch Linux PKGBUILD

Sometimes, linking applications for the target arch may fail with the following message.

mipsel-linux-ld: skipping incompatible /lib/libc.so.6 when searching for /lib/libc.so.6
mipsel-linux-ld: cannot find /lib/libc.so.6

This can also be the cause of configure failing pretending the compiler cannot create executables.

checking for C compiler default output file name... 
configure: error: C compiler cannot create executables
See `config.log' for more details.

This happens when using a relocated build environment for the target. That is, libraries have been built to be installed in / but where installed in, say, /sysroot/ on the build system. This usually does not cause any problem, except it some of the supposed library files are linker scripts (this seems to be really usual for libc6.so and libpthread.so), e.g. /sysroot/usr/lib/libc.so:

/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf32-tradlittlemips)
GROUP ( /lib/libc.so.6 /usr/lib/libc_nonshared.a )

Those script point the linker to the actual location of the desired libraries, but they may sometimes include full path to the object file, rather than relative location. Scripts with full path would then point the linker to the build system's libraries instead of the target's.

The cleanest solution to fix that is to modify those scripts to point to the actual location of the libraries on the build system. Here is an example shell script doing so. It depends on the file command identifying linker scripts as “ASCII C program text”, and relocates the scripts to point to $targetfs.

for FILE in $targetfs/lib/* $targetfs/usr/lib/*; do
        if test -f $FILE && file $FILE | grep -q ASCII; then
                if grep -q $targetfs $FILE; then
                        continue
                else
                        echo "Relocating ld script $FILE to use $targetfs..."
                        sed -i "s# \(/lib/\)# $targetfs\1#g;s# \(/usr/lib/\)# $targetfs\1#g" $FILE
                fi
        fi
done

(Thanks to Sylvestre Ledru for pointing me in the very right direction.)

When linking using libtool, it sometimes happens that a command line that should have worked perfectly well, is replaced by one with some of the -lLIB arguments replaced by an absolute path to the system library /usr/lib/libLIB.so.

/bin/sh ../libtool --tag=CC --mode=link mipsel-linux-gcc -I.. '-DDEBUG_TRAP=__asm__("int $3")' -DAVAHI_DAEMON_RUNTIME_DIR=\"/var/run/avahi-daemon/\" -DAVAHI_SOCKET=\"/var/run/avahi-daemon/socket\" -DAVAHI_SERVICE_DIR=\"/etc/avahi/services\" -DAVAHI_CONFIG_FILE=\"/etc/avahi/avahi-daemon.conf\" -DAVAHI_HOSTS_FILE=\"/etc/avahi/hosts\" -DAVAHI_DBUS_INTROSPECTION_DIR=\"/usr/share/avahi/introspection\" -DAVAHI_CONFIG_DIR=\"/etc/avahi\" -I/tmp/cube-target/usr/include -Wall -W -pedantic -pipe -Wformat -Wfloat-equal -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wredundant-decls -Wmissing-noreturn -Wshadow -Wendif-labels -Wpointer-arith -Wbad-function-cast -Wcast-qual -Wcast-align -Wwrite-strings -Winline  -L/tmp/cube-target/lib -L/tmp/cube-target/usr/lib -o avahi-daemon  avahi_daemon-main.o avahi_daemon-simple-protocol.o avahi_daemon-static-services.o avahi_daemon-static-hosts.o avahi_daemon-ini-file-parser.o avahi_daemon-setproctitle.o avahi_daemon-check-nss.o    ../avahi-common/libavahi-common.la ../avahi-core/libavahi-core.la /tmp/cube-target/usr/lib/libdaemon.so -lexpat  -ldl  
mkdir .libs
mipsel-linux-gcc -I.. "-DDEBUG_TRAP=__asm__(\"int \$3\")" -DAVAHI_DAEMON_RUNTIME_DIR=\"/var/run/avahi-daemon/\" -DAVAHI_SOCKET=\"/var/run/avahi-daemon/socket\" -DAVAHI_SERVICE_DIR=\"/etc/avahi/services\" -DAVAHI_CONFIG_FILE=\"/etc/avahi/avahi-daemon.conf\" -DAVAHI_HOSTS_FILE=\"/etc/avahi/hosts\" -DAVAHI_DBUS_INTROSPECTION_DIR=\"/usr/share/avahi/introspection\" -DAVAHI_CONFIG_DIR=\"/etc/avahi\" -I/tmp/cube-target/usr/include -Wall -W -pedantic -pipe -Wformat -Wfloat-equal -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wredundant-decls -Wmissing-noreturn -Wshadow -Wendif-labels -Wpointer-arith -Wbad-function-cast -Wcast-qual -Wcast-align -Wwrite-strings -Winline -o .libs/avahi-daemon avahi_daemon-main.o avahi_daemon-simple-protocol.o avahi_daemon-static-services.o avahi_daemon-static-hosts.o avahi_daemon-ini-file-parser.o avahi_daemon-setproctitle.o avahi_daemon-check-nss.o /tmp/cube-target/usr/lib/libdaemon.so  -L/tmp/cube-target/lib -L/tmp/cube-target/usr/lib ../avahi-common/.libs/libavahi-common.so ../avahi-core/.libs/libavahi-core.so /usr/lib/libexpat.so -ldl
/usr/lib/libexpat.so: could not read symbols: Invalid operation

This is quite similar to the problem above and due to the relocation of the target's sysroot as a subtree of the build host. Libtool is being wrongly redirected to an inapropriate library directory due to a script LIB.la, usually /usr/lib/libLIB.so like the following.

# libLIB.la - a libtool library file
# Generated by ltmain.sh - GNU libtool 1.5.10 (1.1220.2.131 2004/09/19 12:46:56)
#
# Please DO NOT delete this file!
# It is necessary for linking the library.

# The name that we can dlopen(3).
dlname='libLIB.so.0'

# Names of this library.
library_names='libLIB.so.0.0.1 libLIB.so.0 libLIB.so'

# The name of the static archive.
old_library='libLIB.a'

# Libraries that this one depends upon.
dependency_libs=' -L//tmp/staging/mipsel-linux/lib'

# Version information for libLIB.
current=0
age=0
revision=1

# Is this an already installed library?
installed=yes

# Should we warn about portability when linking against -modules?
shouldnotlink=no

# Files to dlopen/dlpreopen
dlopen=''
dlpreopen=''

# Directory that this library needs to be installed in:
libdir='/usr/lib'

The source of the problem is on the last line, the libdir='/usr/lib' statement, which should obviously be replaced to point to the actual sysroot of the target system. Here is a sample bash script doing the work.

for FILE in $targetfs/lib/*.la $targetfs/usr/lib/*.la; do
        if grep -q $targetfs $FILE; then
                continue
        else
                echo "Relocating libtool script $FILE to use $targetfs..."
                sed -i "s#^libdir='\(/lib\)'#libdir=$targetfs\1#g;s#^libdir='\(/usr/lib\)'#libdir=$targetfs\1#g" $FILE
        fi
done