Differences

This shows you the differences between two versions of the page.

users:oliviermehani:2007ie:mipsellinuxtoolchain [2010/01/13 11:18]
Olivier Mehani Page moved from users:oliviermehani:2008phd:mipsellinuxtoolchain to users:oliviermehani:2007ie:mipsellinuxtoolchain
users:oliviermehani:2007ie:mipsellinuxtoolchain [2011/02/10 14:08] (current)
Line 1: Line 1:
 +====== Building and packaging a mipsel-linux cross-compilation toolchain ======
 +The goal here is to build, from scratch, a suitable GNU compilation toolchain for some of our targets, based on little endian MIPS processors and running Linux (hence the ''mipsel-*-linux-gnu'', or ''mipsel-linux'' for short architecture), though it is likely that the following would also work to build (big endian) mips-linux cross toolchains too.
 +The host computer will be a regular PC running [[http://www.archlinux.org|Arch Linux]]. When building a cross compiler toolchain, a more complex terminology is used to refer to the involved architectures:
 +  * ''build'': the architecture on which the program is built (here ''i686-pc-linux-gnu'');
 +  * ''host'': the architecture on which the program will be run (''i686-pc-linux-gnu'', except for the libc for which it will be ''mipsel-linux'');
 +  * ''target'': the architecture for which the program will operate (''mipsel-linux'').
 +
 +===== Versions =====
 +We want to reproduce the building environment of the rest of the target computer. We stick to the same version of the tools:
 +  * [[http://www.gnu.org/software/binutils/|GNU binutils]] [[ftp://ftp.gnu.org/gnu/binutils/binutils-2.14.tar.bz2|2.14]];
 +  * [[http://www.gnu.org/software/gcc/|GNU Compiler Collection (gcc)]] [[ftp://ftp.gnu.org/gnu/gcc/gcc-3.3.4/gcc-3.3.4.tar.bz2|3.3.4]];
 +  * [[http://www.gnu.org/software/glibc/|GNU standard C library (glibc)]] [[ftp://ftp.gnu.org/gnu/glibc/glibc-2.3.3.tar.bz2|2.3.3]].
 +
 +===== Building =====
 +As we plan future packaging, the files will be laid out in ''/usr'' for the host tools, and ''/usr/mipsel-linux'' for the needed target files (//i.e.// mostly the libc and include files).
 +
 +
 +==== binutils ====
 +The binutils are the simplest tools to build, only specifying the target and prefix will marvelously do the job.
 +<code>
 +binutils-2.14$ ./configure --target=mispel-linux --prefix=/usr
 +binutils-2.14$ make
 +binutils-2.14$ sudo make install
 +</code>
 +
 +=== Compilation problem ===
 +When compiling for the target using the freshly built ''binutils'', one may enconter the following error message:
 +<code>
 +as: unrecognized option `-EL'
 +</code>
 +The problem is simple, as this only means that the ''binutils'' have not been correctly installed (or the autotools haven't found them in the ''$PATH''), and the toolchain is using the host's ''binutils'' instead of the ''mipsel-linux'' ones. A simple solution is to check the proper installation of the new utilities, eventually rebuilding them with the correct prefixes.
 +
 +==== gcc ====
 +The C cross-compiler has to be compiled first, in order to be able to build the glibc for the target.
 +It is mandatory to build gcc out of the original source tree (//i.e.// not even in a subdirectory of the source tree as it is "unsupported"). The building dir will be, originally, called ''gcc-build'' and share the same parent directory as the sources.
 +
 +This will be a lightweight version of the compiler (no shared libraries nor threads support). The following assumes that the source tree has been extracted from [[ftp://ftp.gnu.org/gnu/gcc/gcc-3.3.4/gcc-core-3.3.4.tar.bz2|the gcc-core package]]. In case [[ftp://ftp.gnu.org/gnu/gcc/gcc-3.3.4/gcc-3.3.4.tar.bz2|the complete gcc source package]] has been fetched instead, be sure to provide the additional ''--enable-languages=c'' flag to the ''configure'' script.
 +
 +<code>
 +gcc-build$ ../gcc-3.3.4/configure --target=mipsel-linux --prefix=/usr \
 +> --with-gnu-as --with-gnu-ld \
 +> --disable-shared --disable-nls --disable-threads
 +gcc-build$ make
 +gcc-build$ sudo make install
 +</code>
 +
 +It may be advisable to make symbolic links to the binaries in the target files directory:
 +<code>
 +/usr/mipsel-linux/bin$ ls -l
 +total 4328
 +(...)
 +lrwxrwxrwx 1 root root     25 2007-03-22 16:06 cpp -> /usr/bin/mipsel-linux-cpp
 +lrwxrwxrwx 1 root root     25 2007-03-22 16:06 gcc -> /usr/bin/mipsel-linux-gcc
 +lrwxrwxrwx 1 root root     31 2007-03-22 16:06 gcc-3.3.4 -> /usr/bin/mipsel-linux-gcc-3.3.4
 +lrwxrwxrwx 1 root root     28 2007-03-22 16:06 gccbug -> /usr/bin/mipsel-linux-gccbug
 +lrwxrwxrwx 1 root root     26 2007-03-22 16:06 gcov -> /usr/bin/mipsel-linux-gcov
 +(...)
 +</code>
 +
 +==== glibc ====
 +The shared libraries support has been disabled in the bootstrap compiler, but we want to build a shared-libs enabled glibc. This will lead to problems as the linker will unsuccessfully search for ''libgcc_eh.a''. A simple solution is to first make a dummy symbolic link to ''libgcc.a'' which will trick the linker.
 +
 +<code>
 +ln -s libgcc.a /usr/lib/gcc-lib/mipsel-linux/3.3.4/libgcc_eh.a
 +</code>
 +
 +As stated in [[http://www.linux-mips.org/archives/linux-mips/2004-10/msg00068.html|this post on the linux-mips mailing list]], applying [[https://gforge.inria.fr/plugins/scmsvn/viewcvs.php/*checkout*/misc/arch/mipsel-linux-glibc/glibc-2.3.3-mips.diff?rev=990&root=mehani|a patch]] may be be necessary before compiling the c library. Practically, it turned out that this wasn't necessary.
 +
 +As for gcc, glibc has to be built in a separate directory. We'll be using the same layout. After unpacking the glibc sources, one must remember, as the target is a Linux host, to extract the [[ftp://ftp.gnu.org/gnu/glibc/glibc-linuxthreads-2.3.3.tar.bz2|linuxthreads sources]] too.
 +
 +The C library also needs kernel headers for the target machine. In this cas, this is [[ftp://ftp.kernel.org/pub/linux/kernel/v2.4/linux-2.4.27.tar.bz2|Linux 2.4.27]]. It is necessary to configure the kernel for the headers to be in a valid state. The default parameters will be sufficient.
 +<code>
 +linux-2.4.27$ make ARCH=mips config
 +</code>
 +
 +One can then start the building process.
 +<code>
 +glibc-build$ ../glibc-2.3.3/configure --build=i686-pc-linux-gnu \
 +> --host=mipsel-linux --prefix=/usr/mipsel-linux \
 +> --with-headers=../linux-2.4.27/include \
 +> --enable-add-ons=linuxthreads
 +glibc-build$ make
 +glibc-build$ sudo make install
 +</code>
 +Note that the ''--build'' and ''--host'' tags have to be specified instead of the ''--target'' one, which is not relevant. Also note that the prefix is ''/usr/mipsel-linux'' instead of the usual ''/usr'' as we do not want to overwrite our host's libc with another version not compiled for the right architecture.
 +
 +To complete the installation, it is necessary to copy some of the headers from the Linux source tree to the filesystem.
 +<code>
 +linux-2.4.27$ sudo cp -R include/linux/ /usr/mipsel-linux/include/
 +linux-2.4.27$ sudo cp -R include/asm-mips/ /usr/mipsel-linux/include/asm
 +</code>
 +
 +=== Linux-specific behavior ===
 +When building for a Linux host with a prefix set to ''/usr'', instead of installing the libraries in ''/usr/lib'', the building system will put them as the base libraries for the OS, in ''/lib'', and update the linker scripts accordingly. You may get error messages like the following due to this.
 +
 +<code>
 +ld: cannot find //libc.so.6
 +collect2: ld returned 1 exit status
 +</code>
 +
 +This would be a sign that something went wrong and you may have to edit the linker script (like ''libc.so'', ''GROUP'' line) to the correct paths... Or recompile your libc with correct parameters.
 +
 +This problem, however, should not occur following the above building method.
 +
 +==== gcc (the full monty) ====
 +This time, the full gcc source archive is needed. As before, the building process will take place in a directory different from that of the sources.
 +
 +<code>
 +gcc-build$ ../gcc-3.3.4/configure --target=mipsel-linux --prefix=/usr \
 +> --with-gnu-as --with-gnu-ld \
 +> --enable-languages=c,c++ \
 +> --disable-nls --enable-threads
 +gcc-build$ make
 +gcc-build$ sudo make install
 +</code>
 +
 +While trying to configure or build gcc, you may encounter these errors:
 +<code>
 +checking for object suffix... configure: error: installation or configuration problem; compiler does not work
 +</code>
 +<code>
 +cc1: error: bad value (i686) for -march
 +</code>
 +
 +They may be due to a wrongly set ''C(XX)FLAGS'' environment option containing the ''-march=i686'' flag. Be sure to properly set this variable (//i.e.// by removing said flag).
 +
 +===== Packaging =====
 +
 +==== binutils ====
 +The binutils provide a version of ''libiberty.a'' which is already present in the system. It is not necessary (and even possibly dangerous) to overwrite the preexisting version. It should be removed from the package.
 +
 +  * [[https://gforge.inria.fr/plugins/scmsvn/viewcvs.php/*checkout*/misc/arch/mipsel-linux-binutils/PKGBUILD?rev=997&root=mehani|Arch Linux PKGBUILD]].
 +
 +
 +
 +==== gcc ====
 +The GNU Compiler Collection allso builds a version of ''libiberty.a'', which should be removed from the package, as for the binutils. Also, some of the manpages are already present in the system and should be either removed or renamed in order not to cause conflicts when installing the package.
 +
 +  * [[https://gforge.inria.fr/plugins/scmsvn/viewcvs.php/*checkout*/misc/arch/mipsel-linux-gcc-initial/PKGBUILD?rev=1003&root=mehani|Arch Linux PKGBUILD]]
 +
 +=== Thou shalt not strip my binaries ===
 +Beware of auto-stripping of the binaries. This package contains libraries for the target system like ''libgcc.a'' which provides basic functions too big to be inlined. Usually, ''strip'' recognizes binaries that are not in the correct format and prints a warning message before ignoring them. It happens, however, that ''strip'' actually does its jobs on some files on which it shouldn't, overwriting the ELF header of these files with the wrong architecture.
 +<code>
 +$ file _muldi3.o # INCORRECT FORMAT
 +_muldi3.o: ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped
 +</code>
 +<code>
 +$ file _mul_df.o # CORRECT FORMAT
 +_mul_df.o: ELF 32-bit LSB relocatable, MIPS, MIPS-I version 1 (SYSV), not stripped
 +</code>
 +
 +When trying to cross compile for the target architecture and link against the defected ''libgcc.a'', the problem can be spotted when this error message is displayed
 +<code>
 +ld: skipping incompatible /usr/lib/gcc-lib/mipsel-linux/libgcc.a when searching for -lgcc
 +</code>
 +Using ''objdump'' and ''grep'' to remove all correctly headed objects confirms the problem:
 +<code>
 +$ mipsel-linux-objdump -a /usr/lib/gcc-lib/mipsel-linux/libgcc.a | grep format | grep -v elf32-tradlittlemips
 +_muldi3.o:     file format elf32-little
 +</code>
 +
 +To solve the issue, it is necessary to prevent the packaging system from stripping all the binaries and libraries (''options=(NOSTRIP)'' for Arch Linux), eventually stripping manually those for which ''strip'' is safe (//i.e.// binaries and libs for the host machine).
 +
 +==== glibc ====
 +:!: When installing the libc in a alternative root, one must not use the usual ''DESTDIR'' make variable, but pass the path using ''install_root=...''.
 +
 +Some binaries for the target got creating during the build, and have been installed in ''/usr/mipsel-linux/{bin,libexec,sbin}''. They can be removed as the host computer will never be able nor need to run them. Infopages and locales information has also been generated but are not strictly necessary, it has then be decided to remove them from the final package
 +
 +  * [[https://gforge.inria.fr/plugins/scmsvn/viewcvs.php/*checkout*/misc/arch/mipsel-linux-glibc/PKGBUILD?rev=1006&root=mehani|Arch Linux PKGBUILD]]
 +
 +
 +
 +==== gcc ====
 +As before, ''libiberty.a'' and some manpage have to be removed or renamed from the final package
 +
 +  * [[https://gforge.inria.fr/plugins/scmsvn/viewcvs.php/*checkout*/misc/arch/mipsel-linux-gcc/PKGBUILD?rev=1004&root=mehani|Arch Linux PKGBUILD]]
 +
 +
 +===== Errors when using the cross-compiler =====
 +
 +
 +
 +
 +==== Cannot find library whereas it is present ====
 +Sometimes, linking applications for the target arch may fail with the following message.
 +<code>
 +mipsel-linux-ld: skipping incompatible /lib/libc.so.6 when searching for /lib/libc.so.6
 +mipsel-linux-ld: cannot find /lib/libc.so.6
 +</code>
 +
 +This can also be the cause of configure failing pretending the compiler cannot create executables.
 +<code>
 +checking for C compiler default output file name...
 +configure: error: C compiler cannot create executables
 +See `config.log' for more details.
 +</code>
 +
 +This happens when using a relocated build environment for the target. That is, libraries have been built to be installed in ''/'' but where installed in, say, ''/sysroot/'' on the build system. This usually does not cause any problem, except it some of the supposed library files are //linker scripts// (this seems to be really usual for ''libc6.so'' and ''libpthread.so''), //e.g.// ''/sysroot/usr/lib/libc.so'':
 +<code>
 +/* GNU ld script
 +   Use the shared library, but some functions are only in
 +   the static library, so try that secondarily.  */
 +OUTPUT_FORMAT(elf32-tradlittlemips)
 +GROUP ( /lib/libc.so.6 /usr/lib/libc_nonshared.a )
 +</code>
 +Those script point the linker to the actual location of the desired libraries, but they may sometimes include full path to the object file, rather than relative location. Scripts with full path would then point the linker to the build system's libraries instead of the target's.
 +
 +The cleanest solution to fix that is to modify those scripts to point to the actual location of the libraries on the build system. Here is an example shell script doing so. It depends on the ''file'' command identifying linker scripts as "ASCII C program text", and relocates the scripts to point to ''$targetfs''.
 +<code bash>
 +for FILE in $targetfs/lib/* $targetfs/usr/lib/*; do
 +        if test -f $FILE && file $FILE | grep -q ASCII; then
 +                if grep -q $targetfs $FILE; then
 +                        continue
 +                else
 +                        echo "Relocating ld script $FILE to use $targetfs..."
 +                        sed -i "s# \(/lib/\)# $targetfs\1#g;s# \(/usr/lib/\)# $targetfs\1#g" $FILE
 +                fi
 +        fi
 +done
 +</code>
 +
 +
 +
 +
 +
 +
 +
 +
 +
 +==== Libtool replaces a -lLIB flag with an absolute path to the build system's native library ====
 +(Thanks to [[http://sylvestre.ledru.info|Sylvestre Ledru]] for pointing me in the very right direction.)
 +
 +When linking using libtool, it sometimes happens that a command line that should have worked perfectly well, is replaced by one with some of the ''-lLIB'' arguments replaced by an absolute path to the system library ''/usr/lib/libLIB.so''.
 +
 +<code>
 +/bin/sh ../libtool --tag=CC --mode=link mipsel-linux-gcc -I.. '-DDEBUG_TRAP=__asm__("int $3")' -DAVAHI_DAEMON_RUNTIME_DIR=\"/var/run/avahi-daemon/\" -DAVAHI_SOCKET=\"/var/run/avahi-daemon/socket\" -DAVAHI_SERVICE_DIR=\"/etc/avahi/services\" -DAVAHI_CONFIG_FILE=\"/etc/avahi/avahi-daemon.conf\" -DAVAHI_HOSTS_FILE=\"/etc/avahi/hosts\" -DAVAHI_DBUS_INTROSPECTION_DIR=\"/usr/share/avahi/introspection\" -DAVAHI_CONFIG_DIR=\"/etc/avahi\" -I/tmp/cube-target/usr/include -Wall -W -pedantic -pipe -Wformat -Wfloat-equal -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wredundant-decls -Wmissing-noreturn -Wshadow -Wendif-labels -Wpointer-arith -Wbad-function-cast -Wcast-qual -Wcast-align -Wwrite-strings -Winline  -L/tmp/cube-target/lib -L/tmp/cube-target/usr/lib -o avahi-daemon  avahi_daemon-main.o avahi_daemon-simple-protocol.o avahi_daemon-static-services.o avahi_daemon-static-hosts.o avahi_daemon-ini-file-parser.o avahi_daemon-setproctitle.o avahi_daemon-check-nss.o    ../avahi-common/libavahi-common.la ../avahi-core/libavahi-core.la /tmp/cube-target/usr/lib/libdaemon.so -lexpat  -ldl  
 +mkdir .libs
 +mipsel-linux-gcc -I.. "-DDEBUG_TRAP=__asm__(\"int \$3\")" -DAVAHI_DAEMON_RUNTIME_DIR=\"/var/run/avahi-daemon/\" -DAVAHI_SOCKET=\"/var/run/avahi-daemon/socket\" -DAVAHI_SERVICE_DIR=\"/etc/avahi/services\" -DAVAHI_CONFIG_FILE=\"/etc/avahi/avahi-daemon.conf\" -DAVAHI_HOSTS_FILE=\"/etc/avahi/hosts\" -DAVAHI_DBUS_INTROSPECTION_DIR=\"/usr/share/avahi/introspection\" -DAVAHI_CONFIG_DIR=\"/etc/avahi\" -I/tmp/cube-target/usr/include -Wall -W -pedantic -pipe -Wformat -Wfloat-equal -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wredundant-decls -Wmissing-noreturn -Wshadow -Wendif-labels -Wpointer-arith -Wbad-function-cast -Wcast-qual -Wcast-align -Wwrite-strings -Winline -o .libs/avahi-daemon avahi_daemon-main.o avahi_daemon-simple-protocol.o avahi_daemon-static-services.o avahi_daemon-static-hosts.o avahi_daemon-ini-file-parser.o avahi_daemon-setproctitle.o avahi_daemon-check-nss.o /tmp/cube-target/usr/lib/libdaemon.so  -L/tmp/cube-target/lib -L/tmp/cube-target/usr/lib ../avahi-common/.libs/libavahi-common.so ../avahi-core/.libs/libavahi-core.so /usr/lib/libexpat.so -ldl
 +/usr/lib/libexpat.so: could not read symbols: Invalid operation
 +</code>
 +
 +This is quite similar to the problem above and due to the relocation of the target's ''sysroot'' as a subtree of the build host. Libtool is being wrongly redirected to an inapropriate library directory due to a script ''LIB.la'', usually ''/usr/lib/libLIB.so'' like the following.
 +
 +<file>
 +# libLIB.la - a libtool library file
 +# Generated by ltmain.sh - GNU libtool 1.5.10 (1.1220.2.131 2004/09/19 12:46:56)
 +#
 +# Please DO NOT delete this file!
 +# It is necessary for linking the library.
 +
 +# The name that we can dlopen(3).
 +dlname='libLIB.so.0'
 +
 +# Names of this library.
 +library_names='libLIB.so.0.0.1 libLIB.so.0 libLIB.so'
 +
 +# The name of the static archive.
 +old_library='libLIB.a'
 +
 +# Libraries that this one depends upon.
 +dependency_libs=' -L//tmp/staging/mipsel-linux/lib'
 +
 +# Version information for libLIB.
 +current=0
 +age=0
 +revision=1
 +
 +# Is this an already installed library?
 +installed=yes
 +
 +# Should we warn about portability when linking against -modules?
 +shouldnotlink=no
 +
 +# Files to dlopen/dlpreopen
 +dlopen=''
 +dlpreopen=''
 +
 +# Directory that this library needs to be installed in:
 +libdir='/usr/lib'
 +</file>
 +
 +The source of the problem is on the last line, the '' libdir='/usr/lib' '' statement, which should obviously be replaced to point to the actual sysroot of the target system. Here is a sample bash script doing the work.
 +<code bash>
 +for FILE in $targetfs/lib/*.la $targetfs/usr/lib/*.la; do
 +        if grep -q $targetfs $FILE; then
 +                continue
 +        else
 +                echo "Relocating libtool script $FILE to use $targetfs..."
 +                sed -i "s#^libdir='\(/lib\)'#libdir=$targetfs\1#g;s#^libdir='\(/usr/lib\)'#libdir=$targetfs\1#g" $FILE
 +        fi
 +done
 +</code>
 +
 +{{tag>oliviermehani}}