How to make DRBD compatible to the Linux kernel

The Linux kernel’s interface towards device drivers is not set in stone. It evolves with the evolvement of Linux. Sometimes driven by hardware improvements, sometimes driven by general evolution of the code base. In tree drivers this is not a big issue since they get changed as the interfaces are modified, therefore are called “tree wide changes”.

DRBD® development happens out-of-tree before code gets sent upstream. We at LINBIT® need to track these tree wide changes and follow them in the DRBD device driver. But, not all of our users are using the same kernel.

Some are strictly sticking to the kernel that was shipped with the distribution, running a kernel that is years behind Linus’ version of Linux.

That creates a problem. DRBD should be compatible with both: Years old, carefully maintained “Vendor kernels”, and the latest and greatest Linus kernel.

In order to do that we have a kernel compatibility layer in DRBD. It contains two main parts:

  1. Detecting the capabilities of the kernel we want to build against. The background is that a “Vendor kernel” is not just a random old Linus kernel. It starts as some release of the vanilla kernel, and then the vendors cherry-pick upstream changes and fixes they deem to be relevant to their vendor kernel.
  2. The compatibility layer itself. Up to DRBD-9.0.19 this was a huge file containing many #IFDEFs. It became a maintenance nightmare. It was hard to extend, hard to understand and debug, and hard to remove old compat. Everything was ugly.

Coccinelle is French for Ladybug

The Coccinelle project from INRIA contains a tool to apply semantic patches to the source code, or expand the semantic patches to conventional patches. A few of us DRBD developers practically fell into love with that tool. It allows us to express how some code that is compatible with the upstream kernel needs to be changed in order to be compatible with some older versions of the kernel. 

This allows our DRBD upstream code to be in a form that has clean Linux upstream code, containing no compatibility hacks.

This allows us to automatically transform DRBD to be compatible with random old kernels or vendor kernels. The result, after the transformation, is clean C code without confusing macros and #IFs. It is wonderful.

The new kernel compatibility mechanism:

  1. Detect kernel capabilities (as before)
  2. Create a compat patch using spatch (from Coccinelle)
  3. Apply the compat patch and compile DRBD

Where there is light there must be shadow

The spatch tool is not available on all Linux distributions. For a little older kernels we even require a very recent version of spatch, which is even less available. The researchers at INRIA write the tool in a programming language, “OCaml”, which is right for them and the challenge, but not familiar to many in the open source community.

This complex build dependency makes it harder for community members to build drbd-9.0.20 and higher compared to how it was before.

The shortcut through the maze

For a number of well known vendor kernels (RHEL/Centos, Ubuntu-LTS, SLES, Oracle linux, Amazon Linux) we include the complete compat patches in the distribution source tar.gz. Meaning, during the DRBD build process it executes all the COMPAT tests and calculates a hash value out of that.

If it finds a pre-generated compat.patch for that hash value, the build process can continue without a call to spatch! Complex build dependency avoided!

The hard route through the maze

When you are building from a GIT-checkout, or you are building for a kernel for which we did not include the pre-generated compat.patch you need spatch.

If necessary you can run step 2 (using spatch) on a different machine then approach step 1 (testing kernel capabilites) and step 3 (compiling the drbd kernel module).

Use ‘make’ to start the compilation process. If it fails just after this output:

 [...]

 COMPAT  sock_create_kern_has_five_parameters

 COMPAT  sock_ops_returns_addr_len

 COMPAT  submit_bio_has_2_params

 CHK     /home/phil/src/drbd09/drbd/compat.4.15.18.h

 UPD     /home/phil/src/drbd09/drbd/compat.4.15.18.h

 CHK     /home/phil/src/drbd09/drbd/compat.h

 UPD     /home/phil/src/drbd09/drbd/compat.h

 GENPATCHNAMES   4.15.0-48-generic

 SPATCH   27e10079afbff16b2b82fae9f7dbe676

Please take note of the hash value after “SPATCH”. That is like a fingerprint containing all the results of the countless “COMPAT” tests that were executed just before.

Then you need to copy the results of the COMPAT tests to machine/VM/container that has the same drbd source directory and a recent spatch.

rsync -rv drbd/drbd-kernel-compat/cocci_cache/27e10079afbff16b2b82fae9f7dbe676 \

 user@somewhere:src/drbd-9.0.20/drbd/drbd-kernel-compat/cocci_cache/

Then you run the spatch part of the build process there:

ssh user@somewehre "make -C src/drbd-9.0.20/drbd compat"

After that you copy the resulting compat.patch back:

rsync -rv \ user@somewhere:src/drbd-9.0.20/drbd/drbd-kernel-compat/cocci_cache/ \

  drbd/drbd-kernel-compat/cocci_cache/

Call ‘make’ to restart the build process. If you did it right, it will find the generated compat.patch and finish the compilation process.

Get a Ladybug

If you’d like to get a spatch that is recent enough for building the DRBD driver, use a docker container we published on dockerhub https://hub.docker.com/r/linbit/coccinelle.

docker pull linbit/coccinelle

Then put the following shell script under the name ‘spatch’ into your $PATH.

#!/bin/bash

docker run -it --rm -v "$PWD:$PWD" -w "$PWD" coccinelle:latest spatch "$@"

DRBD compatible to the Linux kernel

All of this is great for making the code more readable, easier to understand, and more likely to contain less bugs. And, having the DRBD code, without backward compatibility clutter, is an important milestone on the path to getting DRBD-9 into Linus’ vanilla kernel and replacing drbd-8.4 with drbd-9 from there.

Philipp Reisner

Philipp Reisner

Philipp Reisner is founder and CEO of LINBIT in Vienna/Austria. His professional career has been dominated by developing DRBD, a storage replication for Linux. Today he leads a company of about 30 employees with locations in Vienna, Austria and Portland, Oregon.

Talk to us

LINBIT is committed to protecting and respecting your privacy, and we’ll only use your personal information to administer your account and to provide the products and services you requested from us. From time to time, we would like to contact you about our products and services, as well as other content that may be of interest to you. If you consent to us contacting you for this purpose, please tick above to say how you would like us to contact you.

You can unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our Privacy Policy.

By clicking submit below, you consent to allow LINBIT to store and process the personal information submitted above to provide you the content requested.

Talk to us

LINBIT is committed to protecting and respecting your privacy, and we’ll only use your personal information to administer your account and to provide the products and services you requested from us. From time to time, we would like to contact you about our products and services, as well as other content that may be of interest to you. If you consent to us contacting you for this purpose, please tick above to say how you would like us to contact you.

You can unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our Privacy Policy.

By clicking submit below, you consent to allow LINBIT to store and process the personal information submitted above to provide you the content requested.