Here be Dragons! How to cross compile a linux kernel for the RPi

Introduction

It goes without saying that if you aren't a serious geek and you have stumbled upon this blog by mistake then I would leave now. This is about as hard core techie as I get these days.

During my recent spell of relaxing in the Alps and watching the world go by I played with docker on my Raspberry Pi (RPi) since those helpful Resin guys had done all the heavy lifting for me. Great, but the linux distribution they had used was Arch linux which just felt too alien to me since the default Raspbian distribution is based on Debian Wheezy. Wouldn't it be great I thought to install docker on the Raspbian distribution? It can't can't be hard can it since those Resin guys have already done it. Hmmmm!

Anyway, this blog isn't going to explain how to get docker running on Raspbian. I will save that story for another day. This post is about how to cross compile the linux kernel for RPi, so let's get started. Why cross compilation? You can compile and build the linux kernel on the RPi directly. This was my first approach, but it takes a very very long time. The second time I did this I got bored and decided to learn how to cross compile and it was so much faster.

In order to work out how to do this I read some useful blogs, which I will reference at the end.

Prerequisites

I assume you have decent development environment machine such as a Macbook. I used my top-spec mini Air for the job, but I'm sure a Windows machine would do it equally well. The key tool to have as a starting point is vagrant and I used an unbuntu base image for this. I won't explain Vagrant and assume you know how to use this tool.

Vagrant and a fast broadband connection are all you need.

If you want to actually deploy your built kernel to your RPi I would take a copy of it's kernel configuration and scp it to the vagrant directory on your host box first. You can get the configuration on the RPi via the command:

zcat /proc/config.gz > .config

A Step by Step Guide

1. First install all the tools required

sudo apt-get install libncurses5-dev gcc make git bc
sudo apt-get install libc6:i386 libgcc1:i386 gcc-4.6-base:i386 libstdc++5:i386 libstdc++6:i386 lib32z1 lib32ncurses5 lib32bz2-1.0 

The 2nd command insures that you have the 32 bit include files required for the RPi as this is a 32 bit processor.

2. Install the Raspbian Tool Chain for Cross Compiling

sudo su
cd /opt
git clone git://github.com/raspberrypi/tools.git

export CCPREFIX=/opt/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-

3. Download the Linux Kernel for the RPi

cd /opt
mkdir raspberrypi
cd raspberrypi
git clone git://github.com/raspberrypi/linux.git
export KERNEL_SRC=/opt/raspberrypi/linux

For this blog I am assuming you will build the most recent kernel.

4. Configure the Kernel

I prefer to ensure I am starting from a clean state first.

ARCH=arm CROSS_COMPILE=${CCPREFIX} make mrproper

Then copy the existing RPI configuration to act as a starting point

cp /vagrant/.config .
ARCH=arm CROSS_COMPILE=${CCPREFIX} make menuconfig

Unless you are feeling brave or need to change the configuration of the kernel you are building, as you would to build docker, then just save the configuration and exit this GUI.

4. Build the Kernel

Since even on a modern machine this will take a while (about 30 minutes on my machine) you might want to use a script and nohup this, but the raw command is:

ARCH=arm CROSS_COMPILE=${CCPREFIX} make &

Assuming all went well you will have a new linux kernel built and now need to make the kernel modules for it.

ARCH=arm CROSS_COMPILE=${CCPREFIX} make modules_install

Now you should have a new linux kernel here:

/opt/raspberrypi/linux/arch/arm/boot/Image

The modules will have been built here: /lib/modules, e.g. at the time I wrote this blog the kernel was at version 3.12.18+

/lib/modules/3.12.18+

Now there is a tool to run to prepare the kernel for the RPi

cd /opt/tools/mkimage

python ./imagetool-uncompressed.py /opt/raspberrypi/linux/arch/arm/boot/Image

Note: This isn't required for the raspberry Pi 2. For that just copy the zImage kernel to image7.img in /boot 

The compressed image will be in your current directory called kernel.img.

You will need to copy this across to /boot directory on the RPi

5. Install The New Kernel on the RPI

IMPORTANT DISCLAIMER: If you follow these instruction it is possible that your RPi may not boot and you will need to enter emergency recovery procedures. You have been warned and I won't be help accountable or responsible.

First backup your old kernel then copy over the kernel modules and new kernel to the new RPI and reboot.

cp /boot/kernel.img /boot/kernel-old.img

cp kernel.img /boot/

- need to install the kernel modules directory built to the RPi in the /lib/modules directory too. I assume you know how to do this using tar & scp.

reboot

That is it!

Hopefully, you will be running ok on a new kernel and uname -a will show that you are running on the correct kernel version

References

1. Ken Cochrane - Getting docker up on a RaspberryPi

2. RPi Kernel Configuration

How to run Docker on a Raspberry Pi

Introduction

The easy way to just run docker on a RPi is just to follow these instructions, however, this requires you to create a dedicated SD card which runs Arch Linux. Since that distribution isn't my cup of tea I decided I wanted to run Docker on the standard Raspbian Debian Wheezy distribution. Although the docker executable can be downloaded in binary form for the RPi it won't run on the standard Raspbian kernel because it requires LXC containers and AUFS which aren't there. This blog post will explain how you can enable these features.

To prevent you or me having to perform the steps below I have copied the resultant kernel onto github. I did notice that the CPU utilisation was a little high, so I added the follwing line to the file: /boot/config.txt to resolve that:

cgroup_disable=memory

Since this requires making changes to the kernel configuration and building a linux kernel I suggest that you first read my other blog on how to do this. Once you have managed to build a linux kernel then the steps below explain how to customise it to support docker.

Downloading AUFS

Assuming you already have the linux source code for the RPi downloaded, in the vagrant ubuntu VM then follow the following steps.

sudo su
cd /opt/raspberrypi/linux
git clone git://aufs.git.sourceforge.net/gitroot/aufs/aufs3-standalone.git

I had a number of challenges getting AUFS to compile with the Linux kernel and I found the path of least resistance was to use the following versions:

• Linux rpi-3.10.y
• AUFS  aufs3.10

Therefore set the appropriate git branches:

git checkout rpi-3.10.y
cd aufs3-standalone
git checkout origin/aufs3.10

Now we need to actually patch the AUFS code into the appropriate linux kernel source code:

cp -rp *.patch ../
cp -rp fs ../
cp -rp Documentation/ ../
cp -rp include/ ../
cd ..

patch -p1 < aufs3-base.patch
patch -p1 < aufs3-mmap.patch
patch -p1 < aufs3-standalone.patch
patch -p1 < aufs3-kbuild.patch
patch -p1 < aufs3-loopback.patch

You should have seen a lot of 'Hunk #x succeeded' messages and two failures. We need to fix one of these as the 2nd can be ignored.

patching file mm/fremap.c 
Hunk #1 FAILED at 202. 1 out of 1 hunk FAILED -- saving rejects to file mm/fremap.c.rej
..
patching file include/uapi/linux/Kbuild
Hunk #1 FAILED at 56.
1 out of 1 hunk FAILED -- saving rejects to file include/uapi/linux/Kbuild.rej

Ok, so now we need to edit mm/fremap.c to fix the issue as follows. I use vi (which shows my age) but with your favourite editor first look at the mm/fremap.c.rej file:

--- mm/fremap.c
+++ mm/fremap.c
@@ -202,11 +202,12 @@
        */
        if (mapping_cap_account_dirty(mapping)) {
                        unsigned long addr;
-                       struct file *file = get_file(vma->vm_file);
+                       struct file *file = vma->vm_file;
+                       vma_get_file(vma);
                        addr = mmap_region(file, start, size,
                                        vma->vm_flags, pgoff);
-                       fput(file);
+                       vma_fput(vma);
                        if (IS_ERR_VALUE(addr)) {
                                err = addr;<
                        } else {

This is a diff format file. The important points are that the changes start at line 202 in the original file and that the lines that start with a '+' need to be added to the original source code and the lines that start with a '-' need to be removed.

Now we understand what we need to do we cn actually edit mm/fremap.c and look at the code at line 202

*/
                if (mapping_cap_account_dirty(mapping)) {
                        unsigned long addr;
                        struct file *file = get_file(vma->vm_file);
                        /* mmap_region may free vma; grab the info now */
                        vm_flags = vma->vm_flags;

                        addr = mmap_region(file, start, size, vm_flags, pgoff);
                        fput(file);
                        if (IS_ERR_VALUE(addr)) {
                                err = addr;
                        } else {
                                BUG_ON(addr != start);
                                err = 0;
                        }
                        goto out_freed;
                }

We can then change this to be:

*/
                if (mapping_cap_account_dirty(mapping)) {
                        unsigned long addr;
                        // Remove struct file *file = get_file(vma->vm_file);
                        struct file *file = vma->vm_file;
                        /* mmap_region may free vma; grab the info now */
                        vm_flags = vma->vm_flags; /* Add */

                        addr = mmap_region(file, start, size, vm_flags, pgoff);
                        // Remove fput(file);
                        vma_fput(vma); /* Add */
                        if (IS_ERR_VALUE(addr)) {
                                err = addr;
                        } else {
                                BUG_ON(addr != start);
                                err = 0;
                        }
                        goto out_freed;
                }

Now we are ready to configure the kernel. I am going to only summarise the changes here since Ken Cochran has provided good screen shots in his blog.

As I mentioned in my last post on how to build a kernel you should first start from the existing RPi configuration file (.config). Assuming you have done this then to configure the linux kernel you need to run:

ARCH=arm CROSS_COMPILE=${CCPREFIX} make menuconfig

The configuration parameters you need to set are as follows:

1. General -> Control Group Support -> Memory Resource Controller for Control Groups (and its three child options)
   1. To reach the Control Group Support just scroll down and then press enter when on it. Whilst here also enable Cpu set Support (see next point). To set press space bar and you will see an asterisk appear next to the option. The escape key brings you up one level of menu.
2. General -> Control Group Support -> cpuset support
3. Device Drivers -> Character Devices -> Support multiple instances of devpts
   1. You will need to hit the escape key several time to reach the main screen to see the original screen and then scroll down to see the Device devices section. As before press space bar with this entry highlighted and scroll down to Character Devices and press space again. The go up one level using escape for the next entry
4. Device Drivers -> Network Device Support -> Virtual ethernet pair device
5. File Systems --> Miscellaneous filesystems ->select "Aufs (Advanced multi layered unification filesystem) support (NEW)" (mine was the the very bottom)
6. Now save the configuration and exist the tool.

I tend to go back into the tool and check that the above configuration items are actually set before kicking off the kernel build as described in my previous blog post, don't forget to ensure that you have set CCPREFIX.

Assuming all is well you will now have a kernel which you should copy to the RPi as I described in my other blog post. Once successfully running on this kernel  you will need to install the LXC libraries used by docker.

I have been able to test the above parts of my blog using a new vagrant image. Now as I don't have a space SD card I am having to rely on the notes that I took when I completed my installation on my RPi. The original blog post that I followed is still useful.

On the RPI:

sudo su
apt 
mkdir /opt/lxc
cd /opt/lxc
git clone https://github.com/lxc/lxc.git
apt-get install automake libcap-dev
cd lxc
./autogen.sh && ./configure && make && make install

Now to check that LXC is working correctly on the RPi type:

pi@raspberrypi /opt $ lxc-checkconfig
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: missing
Network namespace: enabled
Multiple /dev/pts instances: enabled

--- Control groups ---
Cgroup: enabled
Cgroup clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled

--- Misc ---
Veth pair device: enabled
Macvlan: enabled
Vlan: enabled
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/local/bin/lxc-checkconfig

This will show that the kernel is ready for docker to be installed from here. I installed docker by downloading the tar file from resin (https://github.com/resin-io/lxc-docker-PKGBUILD/releases) & extract as root from /

sudo su /
tar xvf docker*.tar.xz

Since I have a hard drive on my RPi and the docker images can be large I symbolically linked /var/lib/docker to /usbmnt1 (my hard drive mount) having first copied the contents of /var/lib/docker to a directory on the hard drive. Whilst this appeared to work the -v flag to mount local directories across to the docker images did't work. Therefore, I removed the symbolic link and repopulated the /var/lib/docker directory and then used the -g flag when starting docker to store the images on /usbmnt1.

Then you can start docker and pull down a Raspbian image to base images from

sudo su -
export LD_LIBRARY_PATH=/usr/local/lib
nohup docker -d &
docker pull  resin/rpi-raspbian
docker run -i -t rpi-raspbian /bin/bash

Finally, here is a screen shot of docker working on my RPi.

Happy dockering on the RPi!

If you want to download an image with Java & Tomcat already installed I prepared one earlier, which you can pull with the tag seahope/rpidockerjavatomcat from my original trying out of docker on the RPi.