LINBIT and its Software-Defined Storage (SDS) solution LINSTOR has provided integration with Linux containers for quite some time. These range from a Docker volume plugin, to a Flexvolume plugin, and recently, a CSI plugin for Kubernetes. While we always provided excellent integration to the container world, most of our software itself was not available as a container/base image. Containerizing our services is a non-trivial task. As you probably know, the core of the DRBD software consists of a Linux kernel module and user space utilities that interact via netlink with this kernel module. Additionally, our software needs to create LVM devices and DRBD block devices within a container. These tasks are interesting and challenging to put into containers. For this article, we assume 3 nodes, one node that acts as a LINSTOR controller, and two that act as satellites. We tested this with recent Centos7 machines and with a current version of Docker.
In this article, we assume access to our Docker registry hosted on
drbd.io. On all hosts you should run the following commands:
docker login drbd.io
Installing the DRBD kernel modules
We need the DRBD kernel module and its dependencies on the LINSTOR satellites (the controller does not need access to DRBD). For that we provide a solution for the most common platforms, namely Centos7/RHEL7 and Ubuntu Bionic.
docker run --privileged -it --rm \ -v /lib/modules:/lib/modules drbd.io/drbd9:rhel7 DRBD modul sucessfully loaded
What this does is check which kernel is actually executed on the host, then found it the most appropriate package in the container and installed it. We ship the same, unmodified rpm/deb packages in the container as we provide in our customer repositories. If you are using Ubuntu Bionic, you should use the
Running a LINSTOR controller
docker run -d --name=linstor-controller \
-p 3376:3376 -p 3377:3377 drbd.io/linstor-controller
The controller does not have any special requirements, it just needs to be accessible to the client via TCP/IP. Please note that in this configuration the controller’s database is not persisted. One possibility is to bind-mount the directory used for the controller’s database by adding
-v /some/dir:/var/lib/linstor .
Running a LINSTOR satellite
docker run -d --name=linstor-satellite --net=host \
The satellite is the component that creates actual block devices. On one hand the backing devices (usually LVM) and the actual DRBD block devices. Therefore this container needs access to
/dev, and it needs to share the host networking. Host networking is required for the communication between
drbdsetup and the actual kernel module.
Configuring the Cluster
We have to set up LINSTOR as usual, which fortunately, is an easy task and has to be done only once. In the spirit of this blog post, let’s use a containerized LINSTOR client as well. As the client obviously has to talk to the controller, we need to tell the client in the container where to find the controller. This is done by setting the environment variable
docker run -it --rm -e LS_CONTROLLERS=Controller \ drbd.io/linstor-client interactive ... - volume-definition (vd) LINSTOR ==> node create Satellite1 22.214.171.124 LINSTOR ==> node create Satellite2 126.96.36.199 LINSTOR ==> storage-pool-definition create drbdpool LINSTOR ==> storage-pool create lvm Satellite1 drbdpool drbdpool LINSTOR ==> storage-pool create lvm Satellite2 drbdpool drbdpool
Creating a replicated DRBD resource
So far we loaded the kernel module on the satellites, started the controller and satellite containers and configured the LINSTOR cluster. Now it is time to actually create resources.
docker run -it --rm -e LS_CONTROLLERS=Controller \ drbd.io/linstor-client interactive ... - volume-definition (vd) LINSTOR ==> resource-definition create demo LINSTOR ==> volume-definition create demo 1G LINSTOR ==> resource create demo --storage-pool drbdpool --auto-place 2
If you have
drbd-utils installed on the host, you can now see the DRBD resource as usual via
drbdsetup status. But we can also use a container to do that. On one of the satellites you can run a throw-away linstor-satellite container which contains
docker run -it --rm --net=host --privileged \
$ drbdsetup status
Note that by default you will not see the symbolic links for the backing devices created by LVM/udev in the LINSTOR satellite container. That is expected. In the container you will see something like
/dev/drbdpool/demo_00000, while on the host you will only see
lvs will not show the LVs. If you really want to see the LVs on the host, you could execute
lvscan -a --cache, but there is no actual reason for that. One might also map the
lvmetad socket to the container.
As you can see, LINBIT’s container story is now complete. It is now possible to deploy the whole stack via containers. This ranges from the lowest level of providing the kernel modules to the highest level of LINSTOR SDS including the client, the controller, and satellites.