Linux Embedded

Le blog des technologies libres et embarquées

WhitePaper

BCC integration into buildroot

Par le  •  pas de commentaire

Introduction

After the rise of eBPF as the newly most powerful Linux tracer (available since Linux 3.15), multiple front end tools have been built on top of it. Amongst all of them, BCC (BPF Compiler Collection) is the most prominent one as it makes eBPF programs much more easier to write.

Unlike classical tracers, eBPF is reprogrammable (can be extended) and code safe (as it applies many restrictions on the code such as checking NULL pointers and forbidding infinite loops). Those constraints guarantee that an eBPF program cannot crash the kernel.

A detailed article on eBPF (and BCC tooling) scripts is already available at: http://www.linuxembedded.fr/2019/03/les-secrets-du-traceur-ebpf/ (French article).

However, BCC was not intended to run on embedded systems.

Hopefully in this article, we are going to discuss our work on integrating BCC into Buildroot and deploying it on tiny devices such as Raspberry Pi 3 (aarch64).

Motivations behind BCC

During my internship entitled "Linux debugging tools", I have experimented many ways to get insight into the Linux kernel (as well as the userspace). Tools like Ftrace, LTTng are successful but not customizable (limited features). In addition, they may crash the system if not properly used.

eBPF does not have such limitations and runs code safely.

However, eBPF programs must be written in a flavor of C, a reason why BCC was created (to use C and Python instead, an example of BCC scripts used to trace memory leaks can be found at: https://github.com/iovisor/bcc/blob/master/tools/memleak.py).

Using BCC, just a couple of hours were enough for me to create a ddos detector: https://github.com/iovisor/bcc/blob/master/examples/tracing/dddos.py (which is included now in BCC scripts).

Note: My internship report can be accessed from: https://github.com/jugurthab/Linux_kernel_debug/blob/master/debugging-linux-kernel.pdf.

Buildroot

Buildroot is a build system used to automate the generation of a system's components: toolchain, Kernel, bootloader and filesystem. It is capable to download sources from upstream, apply patches and create a final image quickly. It is highly customizable as one can add his/her own packages. In addition, Buildroot uses a user friendly interface (based on Kconfig) to configure our system. One may get his hands on Buildroot as follows:

The project is hosted at: https://gitlab.com/buildroot.org/buildroot/ as well as https://github.com/buildroot/buildroot (mirror).

BCC (BPF Compiler Collection)

Alexei Starovoitov introduced eBPF (see patch: https://lwn.net/Articles/598545/) to extend the BPF (Berkeley Packet Filter) machine. This change turned BPF (originally made for network packet filtering) into a general purpose machine that can execute any kind of userspace defined programs in kernel context (subject to restrictions).

However, eBPF programs are writen in a flavor of C (which is tedious), which led Brenden Blanco to create BCC.

BCC uses Python (or lua or Golang) and C languages to reduce the time and efforts required to write eBPF based programs.

One may start using BCC using the following ressources:

The BCC project can be found at: https://github.com/iovisor/bcc/blob/master/README.md.

BCC From servers to embedded utility

As we have said previously, our work was focused on BCC integration as a package into Buildroot. In fact, BCC can help a lot to troubleshoot bottlenecks (that are difficult to trace on embedded systems) in a fast and efficient way.

With my colleague @Romain NAOUR (a contributor to Buildroot project), we have created the BCC package in Buildroot using with the following test environment:

  • Target board: Raspberry PI 3 (aarch64)
  • Buildroot: master (2019.05)
  • Toolchain: Arm ARM 2018.11
  • Linux kernel: 4.14.95-v8.
  • LLVM 7.0.1.
  • Luajit-2.1.0-beta3 (with aarch64 support): we're going to discuss later why this version was chosen.

During the integration phase we have noticed the following:

  • BCC dependency hell: Bcc is a complex utility that pulls in a lot of dependencies as shown in the picture below:bcc eBPF depencyA few notes on those dependencies:

    KERNEL CONFIG OPTIONS

    Some kernel options are mandatory to support BCC (at least: CONFIG_BPF=y and CONFIG_BPF_SYSCALL=y). A complete list is available at: https://github.com/iovisor/bcc/blob/master/INSTALL.md

Configuring our environment

We assume the reader to get Buildroot's last version available from: https://gitlab.com/buildroot.org/buildroot/, then apply all patches at: http://patchwork.ozlabs.org/project/buildroot/list/?series=106280&submitter=76185&state=*&q=&archive=&delegate= (to add BCC support to Buildroot).

RPi3 configuration

Buildroot ships with some predefined configurations for the most commonly used platforms (such as Beaglebone Black and Raspberry Pi). Those configurations are available under buildroot/configs/ folder. Let's use the one for Raspberry Pi 3 (64 bits) as follows: 

$ make raspberrypi3_64_defconfig

 

Some changes need to be made to complement the above configuration using make menuconfig as shown below:

  • Toolchain: change the Toolchain type to External toolchain (to speed up build process) and make sure that that Arm AArch64 <version> is selected in Toolchain (as shown below)
  • System configuration: Change /dev management to Dynamic using devtmpfs + eudev. You should also select "Enable root login with password" and set Root password (otherwise your board will login directly on boot).
    Note: You can change the default System hostname (RPI3 in this demo) and System banner.
  • Filesystem images: The filesystem size must be at least to 2G (exact size = 2G) as we are going to include the kernel sources (required by BCC) on the target.

We have finished configuring our board. Now, let's add BCC.

BCC configuration

We need to enable both BCC and python3-bcc. The screenshot below will show you where to find them in the Buildroot configuration menu.

 

Once you enable all BCC's dependencies (as shown in above image), select BCC package as demonstrated below:

The BCC package selection automatically enables python3, Luajit2.1.0-beta3, flex, elfutils and clang packages.

You should also ensure that both llvm and BFP backend have been selected as shown below:

The BPF LLVM (BPF backend in picture above) allows to use the extended BPF instruction (more information about this patch is available at: https://github.com/llvm-mirror/llvm/commit/4fe85c75482f9d11c5a1f92a1863ce30afad8d0d).

It's also recommended to check that python-bcc, python-pytest, python-pytest-runner and python-traittypes are enabled.

Enabling eBPF kernel support

The Linux kernel must be compiled with the required eBPF supporting config options as described at: https://github.com/iovisor/bcc/blob/master/INSTALL.md#kernel-configuration. The following are mandatory: 

CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
# [for Linux kernel versions 4.1 through 4.6]
CONFIG_HAVE_BPF_JIT=y
# [for Linux kernel versions 4.7 and later]
CONFIG_HAVE_EBPF_JIT=y

Note: use make linux-menuconfig to configure the kernel.

Compiling and deploying image to the board

Once everything is configured, we can start compilation process as shown below: 

$ make

Compilation is a long task (it can take more than one hour) especially to compile llvm/clang.

Note: There is no need to use -j option with make as Buildroot will set it properly on its own.

Depending on how the sdcard was connected (use lsblk), copy the image as shown below: 

$ sudo dd if=output/images/sdcard.img of=/dev/device_name status=progress conv=fsync

Adding kernel sources

As we have said previously, BCC requires kernel sources to work (you may take a look at: https://github.com/iovisor/bcc/blob/master/INSTALL.md), so before trying to boot the system, we must add the kernel sources into /lib/modules/<Kernel version>/Build folder in the rootFs on your sdcard.

Navigate to buildroot/output/build and copy the content of linux-custom into /lib/modules/<Kernel version>/Build in your sdcard. The result should look like this:

Once this process is completed, re-insert the sdcard into the RP3 and boot. The credentials are Login: root and password: root (or those that were set on System configuration in Buildroot).

Important remark: Reducing the number of kernel source files required on the target side is possible. However, more time is needed to figure out which specific kernel files are mandatory for BCC (copying all files was only a simple solution to get BCC works quickly).

Execute BCC scripts

We are going to run 2 examples. BCC ships with 2 folders that can be already used to troubleshoot some common problems.

  • examples folder: scripts that are in developement and need more testing by community.
  • tools folder: scripts that are mature.

Before executing BCC scripts on the target, we must mount DebugFs as it is required by BCC as follows: 

# mount -t debugfs none /sys/kernel/debug

One can navigate to /usr/share/bcc/examples/tracing folder and list the available BCC examples as shown below:

So before trying to create your own scripts take a look at one of these two folders (examples and tools). If no script can suit your needs, you must write one on your own. A previous article was dedicated for this purpose: http://www.linuxembedded.fr/2019/03/les-secrets-du-traceur-ebpf/ (French article).

Example 1 -  bcc/tools/execsnoop.py:

This script monitors process creation (more details are available here):

Most of the tools have a text file describing how they work, so try to experiment other tools.

Important: If the reader fails to execute the above script, please scroll down into "Common errors" section.

Example 2 -  bcc/examples/tracing/dddos.py:

This is a contribution that we have made to the BCC project available at: https://github.com/iovisor/bcc/blob/master/examples/tracing/dddos.py.

The script can detect potential ddos attacks and reports alerts to the end user:

[caption id="attachment_6508" align="aligncenter" width="917"] dddos in action[/caption]

On the right console, we have used hping3 (from an attacking machine) to simulate a ddos attack (Want to learn how to do it?), and on the left we have executed dddos.py (on Rasbperry Pi 3) which has successfully detected the ddos.

Note: In this demo, we have connected to the Pi 3 using ssh connection.

Common errors

You may not succeed to execute BCC for the first time and get prompted with some errors. Here is a list of issues that we have encountered as well as their solutions:

ERROR 1

One the most common errors is shown below:

Exception: Failed to attach BPF program hello to kprobe sys_clone is caused by an unmounted DebugFS.

ERROR 2

This shows clearly that BCC did not find the kernel sources in /lib/modules/<Kernel version>/Build. One must copy them to resolve the problem.

ERROR 3

#error "CONFIG_BPF_SYSCALL" is undefined means that BPF kernel config options (like CONFIG_BPF and CONFIG_BPF_SYSCALL) are missing. You must recompile your kernel (using make linux-menuconfig) and set all config options required by eBPF (eBPF config options are available on this page https://github.com/iovisor/bcc/blob/master/INSTALL.md).

Once the new image is generated, reflash the sdcard (don't forget to deploy the kernel again into /lib/modules/<Kernel version>/Build of your target).

In case you encounter other errors, please refer to: https://github.com/iovisor/bcc/blob/master/FAQ.txt.

What about 32 bit architectures

Bcc does support only few architectures (at the time of writting this article). Taking a brief look at test_usdt_args.cc shows the following lines: 

#ifdef __aarch64__
    USDT::ArgumentParser_aarch64 parser("4@[x32,200]");
#elif __powerpc64__
    USDT::ArgumentParser_powerpc64 parser("4@-12(42)");
#elif __s390x__
    USDT::ArgumentParser_s390x parser("4@-12(%r42)");
#elif defined(__x86_64__)
    USDT::ArgumentParser_x64 parser("4@i%ra+1r");
#endif
    USDT::Argument arg;
    REQUIRE(!parser.parse(&arg));

The code above illustrates USDT (user statically defined tracepoint) in BCC, we can notice that only few architectures are available (aarch64, powerpc64, s390x and x86_64):

    • Trying to compile for a different architecture returns a compilation error on line REQUIRE(!parser.parse(&arg));as the variable parser will not be created.
    • 32 bits architectures are not supported at all. We have already reported this issue in the mailing list of BCC: https://lists.iohjvisor.org/g/iovisor-dev/message/1610.

Integrating 32 bit architectures can be quite difficult but we're sure that it will available soon. Here is an example illustrating the addition of a new architecture to BCC: https://github.com/iovisor/bcc/pull/2266/files (some investigation needs to be performed on this topic to extend BCC's supported architectures).

Other works

Some works are being made at current time to avoid adding kernel sources into the target which are required by BCC. One project is called BPFD which aims to use BCC like gdb and gdbserver fashion as illustrated below:

Unfortunately, unlike gdbserver which can make use of ethernet or serial communication as a debugging medium, bpfd only supports ADB (Android Debugging Bridge) at the current time.

So if your target is an Android powered device, a detailed article was provided by the creator of bpfd at: https://lwn.net/Articles/744522/.

Possible improvements for BCC

BCC is such a great and handy tool. However, we need to avoid some bulky work on the target:

  • Being able to cross compile BCC on the developpement workstation and deploying it to the target can make things easier. For example, we need a way to get the BPF bytecode on the development host and inject it into the eBPF machine on the target side.
  • We also need to know which generated kernel source headers BCC relies on. This can save the cost of copying all kernel sources to the target (as maybe only some of them are required).

Conclusion

In this article, we've gone through the steps of integrating BCC into Buildroot. At current time, no 32 bit architecture is supported by BCC; yet, the tool is still relatively new and hopefully will become mature soon.

Adding BCC to Buildroot was an exiting task because it can finally allow to troubleshoot bugs on embedded systems in a modern way. It also opens a new horizon to implement any functionnality that one can think of and get it run in a kernel context. With BCC, security tools (like firewalls or Intrusion prevention systems), observability and even system's configuration utilities can be made efficiently with few efforts.

As closing words, we would say that Buildroot and BCC are just like any other opensource projects, everyone can contribute to them.

Laisser un commentaire

Votre adresse de messagerie ne sera pas publiée.