Building Linux

Some notes about building the Linux kernel.

To build with clang:

make -j4 W=1 ARCH=x86_64 HOSTCC=clang CC=clang

Yocto

Yocto is a collection of tools and configurations for building Linux images.

The goal of yocto is to build a full Linux distribution. This includes the toolchain, bootloader, kernel and filesystem image. Keep in mind that Yocto is not a particular tool, it is more of an umbrella project under the Linux Foundation. It uses the “OpenEmbedded” framework which provides tools and configurations to build Linux. The tool you use for building is called BitBake, and the reference base distribution you start with is called “poky”.

What makes yocto different from other build systems is the use of “layers”. Layers are directories that contain build instructions, such as BSP (Board Support Package), UI (frameworks like qt), and distro layers (systemd..). You can create, add, remove and modify layers in your build by editing the bblayers.con file. Openembedded already provides a lot of these for you, and you can find and share lots of layers online, like the official raspberry pi layer for example.

By convention, a layer’s directory starts with meta-. Here is an example layout.

meta-my-layer/
  conf/
    layer.conf
    machine/
      raspberrypi.conf
  meta-bsp/
    recipes-bsp/
    recipes-core/
    recipes-graphics/
    recipes-kernel/
      linux-raspberrypi.inc
      linux-raspberrypi-dev.bb

Yocto uses the following file formats:

• Recipes (.bb): describe build instructions for a single package. This includes fetching, dependencies, configuration and compilation, output.

• PackageGroups (special .bb): often used to group packages together for a FS image.

• Classes (.bbclass): inheritance mechanism for common functionality

• Configuration (.conf): drives the overall behaviour of the build process

• Appen files (.bbappend): define additional metadata for a similarly names .bb file, add or override previously set values

• Include files (.inc): files which are used with the include or require directive

Yocto Getting started

Do this to start a new project:

# Setup
mkdir yocto && cd yocto
# Install build dependencies
sudo apt-get install build-essential chrpath cpio debianutils \
                     diffstat file gawk gcc git iputils-ping  \
                     libacl1 locales python3 python3-git      \
                     python3-jinja2 python3-pexpect python3-pip \
                     python3-subunit socat texinfo unzip wget \
                     xz-utils zstd
# Get bitbake
git clone https://git.openembedded.org/bitbake

# Configure
./bitbake/bin/bitbake-setup init
source bitbake-builds/poky-wrynose/build/init-build-env
bitbake-config-build enable-fragment core/yocto/root-login-with-empty-password
bitbake-config-build enable-fragment core/yocto/sstate-mirror-cdn

# Build a recipe
# You can build different images:
# - core-image-minimal: bare essentials
# - core-image-base: console only with drivers and firmware
# - core-image-sato: includes X11 and some UI applications
# - core-image-full-cmdlike: no GUI but lots of cli tools
bitbake core-image-minimal

runqemu snapshot nographic

Now that you have a linux image, to actually develop an out-of-tree kernel module, you can use bitbake to generate the cross compilationm environment for you (or use devtool, more on this later):

bitbake core-image-minimal -c populate_sdk

This generates a shell script in build/tmp/deploy/sdk/ that you can execute to generate the sdk.

. <sdk_dir>/environment-setup-x86-64-v3-poky-linux

You then develop the module “normally”, using a Makefile. To test it, you run a virtual machine and scp the module inside it, or mount a network filesystem, or something similar. With this you can avoid restarting the image each time.

Once you are done developing, you can integrate it to the image by adding the module inside the bitbake build system. You do this by creating a new recipe that uses the module.bbclass. Then to build it with the bitbake toolchain:

bitbake my-driver

Here is an index of many yocto layers:

https://layers.openembedded.org/layerindex/branch/master/layers/

Useful commands:

bitbake-config-build list-fragments
bitbake-layers -h
bitbake-layers create-layer meta-test
bitbake-layers show-layers

If you get errors like ERROR: Fetcher failure for URL: ‘git://git.openembedded.org/bitbake…’. Unable to fetch URL from any source., do this:

git config --global url."https://git.yoctoproject.org/git/".insteadOf "git://git.yoctoproject.org/"
git config --global url."https://git.openembedded.org/".insteadOf "git://git.openembedded.org/"

Devtool

We have seen that you can create the toolchain with yocto, and develop the module with make. To connect these two systems, you can use devtool. It does some setting up for you and it links the module with the yocto toolchain.

To run the following commands you need to have sourced the bitbake environment.

# Generate new recipes of the driver
devtool add <recipe-name> /path/to/driver
# Or modify existing ones
devtool modify <recipe-name>

# Now you can go to the new workspace directory to modify the
# sources
cd build/workspace/<recipe-name>/sources

# Iterative development

devtool build <recipe-name>
devtool deploy-target <recipe-name> root@<target-ip>

# Commit changes
git add .
git commit -m "commit message"
# Update, keep the workspace active
devtool update-recipe <recipe-name> -a <path to custom layer>

# Cleanup

# Close the workspace
devtool finish <recipe-name> meta-custom-project

Kas

To setup the entire yocto project in a repeatable way, you can (and should) use a tool like Kas. It is not only really useful for CI, but also for local development. You can create a single project.yml file that describes your yocto setup, then enter it with:

kas build project.yml
# or ugse bitbake or build
kas shell project.yml

TODO: play with this

Tuxmake

If you just want to build the kernel and you don’t want to deal with dependencies, tuxmake is a great tool for this.

You can go yo the kernel root directory and simply run tuxmake to build it. You can specify some configuration flags. Tuxmake will download a docker container for the build and configure the toolchain and .config based on the falgs.

cd linux
tuxmake

# or
tuxmake --target-arch=arm64 \
        --toolchain=gcc-10  \
        --kconfig-add /path/to/my.config

Buildroot

Buildroot is quite straight forward to use.

# See which configs are available
ls configs/ | grep qemu

make menuconfig
make linux-menuconfig
make <config>

Devicetree

To tell the kernel which hardware is available and where to find it, you need to write a Device Tree Structure (.dts file). You use the interface from the SoC manifacturer (.dtsi file) as a base class that defines specific connectors. Ideally the kernel should be aware only of the minimum possible in order to save power.

Build Configs

Some build configs you should know about:

• CONFIG_KASAN=y CONFIG_KASAN_GENERIC=y #CONFIG_KASAN_SW_TAGS=y #CONFIG_KASAN_HW_TAGS=y CONFIG_KASAN_VMALLOC=y

  Enable the generic kernel address sanitization, then pick the one you want.

• CONFIG_UBSAN=y

  Enable runtime undefined behaviour checker.

• CONFIG_LOCKDEP=y

  Catch deadlocks

• CONFIG_DEBUG_VM=y

  Do this when touching memory management or DMA.

• CONFIG_PROVE_RCU=y

  This ensures you are accessing RCU-protected memory properly

• CONFIG_SLUB_DEBUG_ON=y

  Do more checks on the slab allocator

• CONFIG_DEBUG_STACKOVERFLOW=y

  Check that we did not put huge object on the stack, because it is very limited.

• CONFIG_DETECT_HUNG_TASK=y

  If you accidentally put a process into an uninterruptible sleep, this will dump the stack trace