This section goes over core concepts and terms that you will need to familiarize yourself with to understand how LINSTOR works and deploys storage. The section is laid out in a “ground up” approach.

While LINSTOR might be used to make the management of DRBD more convenient, it is often integrated with software stacks higher up. Such integration exist already for Kubernetes, OpenStack, OpenNebula and Proxmox. Chapters specific to deploying LINSTOR in these environments are included in this guide.

The southbound drivers used by LINSTOR are LVM, thinLVM and ZFS.

LINSTOR is packaged in both the .rpm and the .deb variants:

For further detail about these packages see the Installable Components section above.

LINSTOR doesn’t support rolling upgrade, controller and satellites must have the same version, otherwise the controller with discard the satellite with a VERSION_MISMATCH. But this isn’t a problem, as the satellite won’t do any actions as long it isn’t connected to a controller and DRBD will not be disrupted by any means.

If you are using the embedded default H2 database and the linstor-controller package is upgraded an automatic backup file of the database will be created in the default /var/lib/linstor directory. This file is a good restore point if for any reason a linstor-controller database migration should fail, than it is recommended to report the error to Linbit and restore the old database file and downgrade to your previous controller version.

If you use any external database or etcd, it is recommended to do a manually backup of your current database to have a restore point.

So first upgrade the linstor-controller, linstor-client package on you controller host and restart the linstor-controller, the controller should start and all of it’s client should show OFFLINE(VERSION_MISMATCH). After that you can continue upgrading linstor-satellite on all satellite nodes and restart them, after a short reconnection time they should all show ONLINE again and your upgrade is finished.

LINSTOR is also available as containers. The base images are available in LINBIT’s container registry, drbd.io.

In order to access the images, you first have to login to the registry (reach out to [email protected] for credentials):

The containers available in this repo are:

An up to date list of available images with versions can be retrieved by opening http://drbd.io in your browser. Make sure to access the host via “http”, as the registry’s images themselves are served via “https”.

To load the kernel module, needed only for LINSTOR satellites, you’ll need to run a drbd9-$dist container in privileged mode. The kernel module containers either retrieve an official LINBIT package from a customer repository, use shipped packages, or they try to build the kernel modules from source. If you intend to build from source, you need to have the according kernel headers (e.g., kernel-devel) installed on the host. There are 4 ways to execute such a module load container:

Example building from shipped source (RHEL based):

Example using a module shipped with the container, which is enabled by not bind-mounting /usr/src:

Example using a hash and a distribution (rarely used):

Example using an existing repo config (rarely used):

Then run the LINSTOR satellite container, also privileged, as a daemon:

To run the LINSTOR controller container as a daemon, mapping ports 3370, 3376 and 3377 on the host to the container:

To interact with the containerized LINSTOR cluster, you can either use a LINSTOR client installed on a system via packages, or via the containerized LINSTOR client. To use the LINSTOR client container:

From this point you would use the LINSTOR client to initialize your cluster and begin creating resources using the typical LINSTOR patterns.

To stop and remove a daemonized container and image:

We assume that the following steps are accomplished on all cluster nodes:

Start and enable the linstor-controller service on the host where it has been installed:

If you are sure the linstor-controller service gets automatically enabled on installation you can use the following command as well:

Whenever you run the LINSTOR command line client, it needs to know where your linstor-controller runs. If you do not specify it, it will try to reach a locally running linstor-controller listening on IP 127.0.0.1 port 3376. Therefore we will use the linstor-client on the same host as the linstor-controller.

should give you an empty list and not an error message.

You can use the linstor command on any other machine, but then you need to tell the client how to find the linstor-controller. As shown, this can be specified as a command line option, an environment variable, or in a global file:

Alternatively you can create the /etc/linstor/linstor-client.conf file and populate it like below.

If you have multiple linstor-controllers configured you can simply specify them all in a comma separated list. The linstor-client will simply try them in the order listed.

The next step is to add nodes to your LINSTOR cluster. You need to provide:

When you use linstor node list you will see that the new node is marked as offline. Now start and enable the linstor-satellite on that node so that the service comes up on reboot as well:

You can also use systemctl start linstor-satellite if you are sure that the service is already enabled as default and comes up on reboot.

About 10 seconds later you will see the status in linstor node list becoming online. Of course the satellite process may be started before the controller knows about the existence of the satellite node.

StoragePools identify storage in the context of LINSTOR. To group storage pools from multiple nodes, simply use the same name on each node. For example, one valid approach is to give all SSDs one name and all HDDs another.

On each host contributing storage, you need to create either an LVM VG or a ZFS zPool. The VGs and zPools identified with one LINSTOR storage pool name may have different VG or zPool names on the hosts, but do yourself a favor and use the same VG or zPool name on all nodes.

These then need to be registered with LINSTOR:

To list your storage-pools you can use:

or using the short version

In case anything goes wrong with the storage pool’s VG/zPool, e.g. the VG having been renamed or somehow became invalid you can delete the storage pool in LINSTOR with the following command, given that only resources with all their volumes in the so-called ‘lost’ storage pool are attached. This feature is available since LINSTOR v0.9.13.

or using the short version

Should the deletion of the storage pool be prevented due to attached resources or snapshots with some of its volumes in another still functional storage pool, hints will be given in the ‘status’ column of the corresponding list-command (e.g. linstor resource list). After deleting the LINSTOR-objects in the lost storage pool manually, the lost-command can be executed again to ensure a complete deletion of the storage pool and its remaining objects.

A resource group is a parent object of a resource definition where all property changes made on a resource group will be inherited by it’s resource definition children. The resource group also stores settings for automatic placement rules and can spawn a resource definition depending on the stored rules.

In simpler terms, resource groups are like templates that define characteristics of resources created from them. Changes to these pseudo templates will be applied to all resources that were created from the resource group, retroactively.

A simple pattern for deploying resources using resource groups would look like this:

The commands above would result in a resource named ‘my_ssd_res’ with a 20GB volume replicated twice being automatically provisioned from nodes who participate in the storage pool named ‘pool_ssd’.

A more useful pattern could be to create a resource group with settings you’ve determined are optimal for your use case. Perhaps you have to run nightly online verifications of your volumes’ consistency, in that case, you could create a resource group with the ‘verify-alg’ of your choice already set so that resources spawned from the group are pre-configured with ‘verify-alg’ set:

The commands above result in twenty 10GiB resources being created each with the ‘crc32c’ ‘verify-alg’ pre-configured.

You can tune the settings of individual resources or volumes spawned from resource groups by setting options on the respective resource-definition or volume-definition. For example, if ‘res11’ from the example above is used by a very active database receiving lots of small random writes, you might want to increase the ‘al-extents’ for that specific resource:

If you configure a setting in a resource-definition that is already configured on the resource-group it was spawned from, the value set in the resource-definition will override the value set on the parent resource-group. For example, if the same ‘res11’ was required to use the slower but more secure ‘sha256’ hash algorithm in its verifications, setting the ‘verify-alg’ on the resource-definition for ‘res11’ would override the value set on the resource-group:

In the following scenario we assume that the goal is to create a resource ‘backups’ with a size of ‘500 GB’ that is replicated among three cluster nodes.

First, we create a new resource definition:

Second, we create a new volume definition within that resource definition:

If you want to change the size of the volume-definition you can simply do that by:

The parameter 0 is the number of the volume in the resource backups. You have to provide this parameter , because resources can have multiple volumes and they are identified by a so called volume-number. This number can be found by listing the volume-definitions.

So far we have only created objects in LINSTOR’s database, not a single LV was created on the storage nodes. Now you have the choice of delegating the task of placement to LINSTOR or doing it yourself.

By using the --drbd-diskless option instead of --storage-pool you can have a permanently diskless DRBD device on a node. This means that the resource will appear as block device and can be mounted to the filesystem without an existing storage-device. The data of the resource is accessed over the network on another nodes with the same resource.

The so called consistency group is a feature from DRBD. It is mentioned in this user-guide, due to the fact that one of LINSTOR’s main functions is to manage storage-clusters with DRBD. Multiple volumes in one resource are a consistency group.

This means that changes on different volumes from one resource are getting replicated in the same chronological order on the other Satellites.

Therefore you don’t have to worry about the timing if you have interdependent data on different volumes in a resource.

To deploy more than one volume in a LINSTOR-resource you have to create two volume-definitions with the same name.

This can be achieved by setting the StorPoolName property to the volume definitions before the resource is deployed to the nodes:

Here the ‘resource create’ commands do not need a --storage-pool option. In this case LINSTOR uses a ‘fallback’ storage pool. Finding that storage pool, LINSTOR queries the properties of the following objects in the following order:

If none of those objects contain a StorPoolName property, the controller falls back to a hard-coded ‘DfltStorPool’ string as a storage pool.

This also means that if you forgot to define a storage pool prior deploying a resource, you will get an error message that LINSTOR could not find the storage pool named ‘DfltStorPool’.

LINSTOR can be used without DRBD as well. Without DRBD, LINSTOR is able to provision volumes from LVM and ZFS backed storage pools, and create those volumes on individual nodes in your LINSTOR cluster.

Currently LINSTOR supports the creation of LVM and ZFS volumes with the option of layering some combinations of LUKS, DRBD, and/or NVMe-oF/NVMe-TCP on top of those volumes.

For example, assume we have a Thin LVM backed storage pool defined in our LINSTOR cluster named, thin-lvm:

We could use LINSTOR to create a Thin LVM on linstor-d that’s 100GiB in size using the following commands:

You should then see you have a new Thin LVM on linstor-d. You can extract the device path from LINSTOR by listing your linstor resources with the --machine-readable flag set:

If you wanted to layer DRBD on top of this volume, which is the default --layer-list option in LINSTOR for ZFS or LVM backed volumes, you would use the following resource creation pattern instead:

You would then see that you have a new Thin LVM backing a DRBD volume on linstor-d:

The following table shows which layer can be followed by which child-layer:

LINSTOR can deal with multiple network interface cards (NICs) in a machine, they are called netif in LINSTOR speak.

Additional NICs are created like this:

NICs are identified by the IP address only, the name is arbitrary and is not related to the interface name used by Linux. The NICs can be assigned to storage pools so that whenever a resource is created in such a storage pool, the DRBD traffic will be routed through the specified NIC.

FIXME describe how to route the controller <-> client communication through a specific netif.

LINSTOR can handle transparent encryption of drbd volumes. dm-crypt is used to encrypt the provided storage from the storage device.

Basic steps to use encryption:

LINSTOR provides various commands to check the state of your cluster. These commands start with a ‘list-‘ prefix and provide various filtering and sorting options. The ‘–groupby’ option can be used to group and sort the output in multiple dimensions.

Snapshots are supported with thin LVM and ZFS storage pools.

DRBD options are set using LINSTOR commands. Configuration in files such as /etc/drbd.d/global_common.conf that are not managed by LINSTOR will be ignored. The following commands show the usage and available options:

For instance, it is easy to set the DRBD protocol for a resource named backups:

LINSTOR can convert resources between diskless and having a disk. This is achieved with the resource toggle-disk command, which has syntax similar to resource create.

For instance, add a disk to the diskless resource backups on ‘alpha’:

Remove this disk again:

LINSTOR expects DRBD Proxy to be running on the nodes which are involved in the relevant connections. It does not currently support connections via DRBD Proxy on a separate node.

Suppose our cluster consists of nodes ‘alpha’ and ‘bravo’ in a local network and ‘charlie’ at a remote site, with a resource definition named backups deployed to each of the nodes. Then DRBD Proxy can be enabled for the connections to ‘charlie’ as follows:

The DRBD Proxy configuration can be tailored with commands such as:

LINSTOR does not automatically optimize the DRBD configuration for long-distance replication, so you will probably want to set some configuration options such as the protocol:

Please contact LINBIT for assistance optimizing your configuration.

It is possible to have LINSTOR working with an external database provider like Postgresql, MariaDB and since version 1.1.0 even ETCD key value store is supported.

To use an external database there are a few additional steps to configure. You have to create a DB/Schema and user to use for linstor, and configure this in the /etc/linstor/linstor.toml.

To make LINSTOR’s administrative tasks more accessible and also available for web-frontends a REST-API has been created. The REST-API is embedded in the linstor-controller and since LINSTOR 0.9.13 configured via the linstor.toml configuration file.

If you want to use the REST-API the current documentation can be found on the following link: https://app.swaggerhub.com/apis-docs/Linstor/Linstor/

Linstor uses SLF4J with Logback as binding. This gives Linstor the possibility to distinguish between the log levels ERROR, WARN, INFO, DEBUG and TRACE (in order of increasing verbosity). In the current linstor version (1.1.2) the user has the following four methods to control the logging level, ordered by priority (first has highest priority):

See the Logback Manual to find more details about logback.xml.

When none of the configuration methods above is used Linstor will default to INFO log level.

It is possible to have the LINSTOR use SSL secure TCP connection between controller and satellite connections. Without going into further details on how java’s SSL engine works we will give you command line snippets using the keytool from java’s runtime environment on how to configure a 3 node setup using secure connections. The node setup looks like this:

Node alpha is the just the controller. Node bravo and node charlie are just satellites.

Here are the commands to generate such a keystore setup, values should of course be edited for your environment.

Now just start controller and satellites and add the nodes with --communication-type SSL.

LINSTOR automatically configures quorum policies on resources when quorum is achievable. This means, whenever you have at least two diskful and one or more diskless resource assignments, or three or more diskful resource assignments, LINSTOR will enable quorum policies for your resources automatically.

Inversely, LINSTOR will automatically disable quorum policies whenever there are less than the minimum required resource assignments to achieve quorum.

This is controlled via the, DrbdOptions/auto-quorum, property which can be applied to the linstor-controller, resource-group, and resource-definition. Accepted values for the DrbdOptions/auto-quorum property are disabled, suspend-io, and io-error.

Setting the DrbdOptions/auto-quorum property to disabled will allow you to manually, or more granularly, control the quorum policies of your resources should you so desire.

For example, to manually set the quorum policies of a resource-group named my_ssd_group, you would use the following commands:

You may wish to disable DRBD’s quorum features completely. To do that, you would need to first disable DrbdOptions/auto-quorum on the appropriate LINSTOR object, and then set the DRBD quorum features accordingly. For example, use the following commands to disable quorum entirely on the my_ssd_group resource-group:

Kubernetes is a container orchestrator. Kubernetes defines the behavior of containers and related services via declarative specifications. In this guide, we’ll focus on using kubectl to manipulate .yaml files that define the specifications of Kubernetes objects.

The Controller pod includes a LINSTOR Client, making it easy to interact directly with LINSTOR. For instance:

This should only be necessary for investigating problems and accessing advanced functionality. Regular operation such as creating volumes should be achieved via the Kubernetes integration.

The operator Helm chart deploys the LINSTOR CSI plugin for you so if you used that, you can skip this section.

If you are integrating LINSTOR using a different method, you will need to install the LINSTOR CSI plugin. Instructions for deploying the CSI plugin can be found on the project’s github. This will result in a linstor-csi-controller StatefulSet and a linstor-csi-node DaemonSet running in the kube-system namespace.

Once all linstor-csi Pods are up and running, we can provision volumes using the usual Kubernetes workflows.

Configuring the behavior and properties of LINSTOR volumes deployed via Kubernetes is accomplished via the use of StorageClasses.

Here below is the simplest practical StorageClass that can be used to deploy volumes:

DRBD options can be set as well in the parameters section. Valid keys are defined in the LINSTOR REST-API (e.g., DrbdOptions/Net/allow-two-primaries: "yes").

We can create the StorageClass with the following command:

Now that our StorageClass is created, we can now create a PersistentVolumeClaim which can be used to provision volumes known both to Kubernetes and LINSTOR:

We can create the PersistentVolumeClaim with the following command:

This will create a PersistentVolumeClaim known to Kubernetes, which will have a PersistentVolume bound to it, additionally LINSTOR will now create this volume according to the configuration defined in the linstor-basic-storage-class StorageClass. The LINSTOR volume’s name will be a UUID prefixed with csi- This volume can be observed with the usual linstor resource list. Once that volume is created, we can attach it to a pod. The following Pod spec will spawn a Fedora container with our volume attached that busy waits so it is not unscheduled before we can interact with it:

We can create the Pod with the following command:

Running kubectl describe pod fedora can be used to confirm that Pod scheduling and volume attachment succeeded.

To remove a volume, please ensure that no pod is using it and then delete the PersistentVolumeClaim via kubectl. For example, to remove the volume that we just made, run the following two commands, noting that the Pod must be unscheduled before the PersistentVolumeClaim will be removed:

Creating snapshots and creating new volumes from snapshots is done via the use of VolumeSnapshots, VolumeSnapshotClasses, and PVCs. First, add the optional CSI snapshotter to your cluster:

More detailed installation instructions can be found on the project site

Then we can create our VolumeSnapshotClass:

Create the VolumeSnapshotClass with kubectl:

Now we will create a volume snapshot for the volume that we created above. This is done with a VolumeSnapshot:

Create the VolumeSnapshot with kubectl:

You can check that the snapshot creation was successful

Finally, we’ll create a new volume from the snapshot with a PVC.

Create the PVC with kubectl:

LINSTOR volumes are typically accessible both locally and over the network.

By default, the CSI plugin will attach volumes directly if the Pod happens to be scheduled on a kubelet where its underlying storage is present. However, Pod scheduling does not currently take volume locality into account. The replicasOnSame parameter can be used to restrict where the underlying storage may be provisioned, if locally attached volumes are desired.

See localStoragePolicy to see how this default behavior can be modified.

Stork is a scheduler extender plugin for Kubernetes which allows a storage driver to give the Kubernetes scheduler hints about where to place a new pod so that it is optimally located for storage performance. You can learn more about the project on its GitHub page.

We are currently working with the maintainers behind Stork to have a LINSTOR driver shipped with it by default. In the meantime, you can use a custom-built Stork container by LINBIT which includes a LINSTOR driver, available on Docker Hub

In general, all configuration for LINSTOR volumes in Kubernetes should be done via the StorageClass parameters, as seen with the storagePool in the basic example above. We’ll give all the available options an in-depth treatment here.

Proxmox VE is an easy to use, complete server virtualization environment with KVM, Linux Containers and HA.

‘linstor-proxmox’ is a Perl plugin for Proxmox that, in combination with LINSTOR, allows to replicate VM disks on several Proxmox VE nodes. This allows to live-migrate active VMs within a few seconds and with no downtime without needing a central SAN, as the data is already replicated to multiple nodes.

If this is a fresh installation, skip this section and continue with Proxmox Plugin Installation.

LINBIT provides a dedicated public repository for Proxmox VE users. This repository not only contains the Proxmox plugin, but the whole DRBD SDS stack including a DRBD SDS kernel module and user space utilities.

The DRBD9 kernel module is installed as a dkms package (i.e., drbd-dkms), therefore you’ll have to install pve-headers package, before you set up/install the software packages from LINBIT’s repositories. Following that order, ensures that the kernel module will build properly for your kernel. If you don’t plan to install the latest Proxmox kernel, you have to install kernel headers matching your current running kernel (e.g., pve-headers-$(uname -r)). If you missed this step, then still you can rebuild the dkms package against your current kernel, (kernel headers have to be installed in advance), by issuing apt install --reinstall drbd-dkms command.

LINBIT’s repository can be enabled as follows, where “$PVERS” should be set to your Proxmox VE major version (e.g., “6”, not “6.1”):

For the rest of this guide we assume that you have a LINSTOR cluster configured as described in Initializing your cluster. Also make sure to setup each node as a “Combined” node. Start the “linstor-controller” on one node, and the “linstor-satellite” on all nodes. The preferred way to use the plugin, starting from version 4.1.0, is via LINSTOR resource groups and a single volume group within every resource group. LINSTOR resource groups are described in Resource groups. All the required LINSTOR configuration (e.g., redundancy count) has to be set on the resource group.

The final step is to provide a configuration for Proxmox itself. This can be done by adding an entry in the /etc/pve/storage.cfg file, with a content similar to the following.

The “drbd” entry is fixed and you are not allowed to modify it, as it tells to Proxmox to use DRBD as storage backend. The “drbdstorage” entry can be modified and is used as a friendly name that will be shown in the PVE web GUI to locate the DRBD storage. The “content” entry is also fixed, so do not change it. The redundancy (specified in the resource group) specifies how many replicas of the data will be stored in the cluster. The recommendation is to set it to 2 or 3 depending on your setup. The data is accessible from all nodes, even if some of them do not have local copies of the data. For example, in a 5 node cluster, all nodes will be able to access 3 copies of the data, no matter where they are stored in. The “controller” parameter must be set to the IP of the node that runs the LINSTOR controller service. Only one node can be set to run as LINSTOR controller at the same time. If that node fails, start the LINSTOR controller on another node and change that value to its IP address. There are more elegant ways to deal with this problem. For more, see later in this chapter how to setup a highly available LINSTOR controller VM in Proxmox.

Recent versions of the plugin allow to define multiple different storage pools. Such a configuration would look like this:

By now, you should be able to create VMs via Proxmox’s web GUI by selecting “drbdstorage“, or any other of the defined pools as storage location.

Starting from version 5 of the plugin one can set the option “preferlocal yes”. If it is set, the plugin tries to create a diskful assignment on the node that issued the storage create command. With this option one can make sure the VM gets local storage if possible. Without that option LINSTOR might place the storage on nodes ‘B’ and ‘C’, while the VM is initially started on node ‘A’. This would still work as node ‘A’ then would get a diskless assignment, but having local storage might be preferred.

At this point you can try to live migrate the VM – as all data is accessible on all nodes (even on Diskless nodes) – it will take just a few seconds. The overall process might take a bit longer if the VM is under load and if there is a lot of RAM being dirtied all the time. But in any case, the downtime should be minimal and you will see no interruption at all.

For the rest of this guide we assume that you installed LINSTOR and the Proxmox Plugin as described in LINSTOR Configuration.

The basic idea is to execute the LINSTOR controller within a VM that is controlled by Proxmox and its HA features, where the storage resides on DRBD managed by LINSTOR itself.

The first step is to allocate storage for the VM: Create a VM as usual and select “Do not use any media” on the “OS” section. The hard disk should of course reside on DRBD (e.g., “drbdstorage”). 2GB disk space should be enough, and for RAM we chose 1GB. These are the minimum requirements for the appliance LINBIT provides to its customers (see below). If you wish to set up your own controller VM, and you have enough hardware resources available, you can increase these minimum values. In the following use case, we assume that the controller VM was created with ID 100, but it is fine if this VM was created at a later time and has a different ID.

LINBIT provides an appliance for its customers that can be used to populate the created storage. For the appliance to work, we first create a “Serial Port”. First click on “Hardware” and then on “Add” and finally on “Serial Port”:

If everything worked as expected the VM definition should then look like this:

The next step is to copy the VM appliance to the VM disk storage. This can be done with qemu-img.

Once completed you can start the VM and connect to it via the Proxmox VNC viewer. The default user name and password are both “linbit”. Note that we kept the default configuration for the ssh server, so you will not be able to log in to the VM via ssh and username/password. If you want to enable that (and/or “root” login), enable these settings in /etc/ssh/sshd_config and restart the ssh service. As this VM is based on “Ubuntu Bionic”, you should change your network settings (e.g., static IP) in /etc/netplan/config.yaml. After that you should be able to ssh to the VM:

In the next step you add the controller VM to the existing cluster:

In our test cluster the Controller VM disk was created in DRBD storage and it was initially assigned to one host (use linstor resource list to check the assignments). Then, we used linstor resource create command to create additional resource assignments to the other nodes of the cluster for this VM. In our lab consisting of four nodes, we created all resource assignments as diskful, but diskless assignments are fine as well. As a rule of thumb keep the redundancy count at “3” (more usually does not make sense), and assign the rest as diskless.

As the storage for the Controller VM must be made available on all PVE hosts in some way, we must make sure to enable the drbd.service on all hosts (given that it is not controlled by LINSTOR at this stage):

By default, at startup the linstor-satellite service deletes all of its resource files (.res) and regenerates them. This conflicts with the drbd services that needs these resource files to start the controller VM. It is good enough to first bring up the resources via drbd.service and ensure that the linstor-satellite.service, which brings up the controller resource never deletes the according res file. To make the necessary changes, you need to create a drop-in for the linstor-satellite.service via systemctl (do *not edit the file directly).

Of course adapt the name of the controller VM in the LS_KEEP_RES variable. Note that the value given is interpreted as regex, so you don’t need to specify the exact name.

Don’t forget to restart the linstor-satellite.service.

After that, it is time for the final steps, namely switching from the existing controller (residing on the physical host) to the new one in the VM. So let’s stop the old controller service on the physical host, and copy the LINSTOR controller database to the VM host:

Finally, we can enable the controller in the VM:

To check if everything worked as expected, you can query the cluster nodes on a physical PVE host by asking the controller in the VM: linstor --controllers=10.43.7.254 node list. It is perfectly fine that the controller (which is just a Controller and not a “Combined” host) is shown as “OFFLINE”. This might change in the future to something more reasonable.

As the last — but crucial — step, you need to add the “controllervm” option to /etc/pve/storage.cfg, and change the controller IP address to the IP address of the Controller VM:

Please note the additional setting “controllervm”. This setting is very important, as it tells to PVE to handle the Controller VM differently than the rest of VMs stored in the DRBD storage. In specific, it will instruct PVE to NOT use LINSTOR storage plugin for handling the Controller VM, but to use other methods instead. The reason for this, is that simply LINSTOR backend is not available at this stage. Once the Controller VM is up and running (and the associated LINSTOR controller service inside the VM), then the PVE hosts will be able to start the rest of virtual machines which are stored in the DRBD storage by using LINSTOR storage plugin. Please make sure to set the correct VM ID in the “controllervm” setting. In this case is set to “100”, which represents the ID assigned to our Controller VM.

It is very important to make sure that the Controller VM is up and running at all times and that you are backing it up at regular times(mostly when you do modifications to the LINSTOR cluster). Once the VM is gone, and there are no backups, the LINSTOR cluster must be recreated from scratch.

To prevent accidental deletion of the VM, you can go to the “Options” tab of the VM, in the PVE GUI and enable the “Protection” option. If however you accidentally deleted the VM, such requests are ignored by our storage plugin, so the VM disk will NOT be deleted from the LINSTOR cluster. Therefore, it is possible to recreate the VM with the same ID as before(simply recreate the VM configuration file in PVE and assign the same DRBD storage device used by the old VM). The plugin will just return “OK”, and the old VM with the old data can be used again. In general, be careful to not delete the controller VM and “protect” it accordingly.

Currently, we have the controller executed as VM, but we should make sure that one instance of the VM is started at all times. For that we use Proxmox’s HA feature. Click on the VM, then on “More”, and then on “Manage HA”. We set the following parameters for our controller VM:

As long as there are surviving nodes in your Proxmox cluster, everything should be fine and in case the node hosting the Controller VM is shut down or lost, Proxmox HA will make sure the controller is started on another host. Obviously the IP of the controller VM should not change. It is up to you as an administrator to make sure this is the case (e.g., setting a static IP, or always providing the same IP via dhcp on the bridged interface).

It is important to mention at this point that in the case that you are using a dedicated network for the LINSTOR cluster, you must make sure that the network interfaces configured for the cluster traffic, are configured as bridges (i.e vmb1,vmbr2 etc) on the PVE hosts. If they are setup as direct interfaces (i.e eth0,eth1 etc), then you will not be able to setup the Controller VM vNIC to communicate with the rest of LINSTOR nodes in the cluster, as you cannot assign direct network interfaces to the VM, but only bridged interfaces.

One limitation that is not fully handled with this setup is a total cluster outage (e.g., common power supply failure) with a restart of all cluster nodes. Proxmox is unfortunately pretty limited in this regard. You can enable the “HA Feature” for a VM, and you can define “Start and Shutdown Order” constraints. But both are completely separated from each other. Therefore it is hard/impossible to guarantee that the Controller VM will be up and running, before all other VMs are started.

It might be possible to work around that by delaying VM startup in the Proxmox plugin itself until the controller VM is up (i.e., if the plugin is asked to start the controller VM it does it, otherwise it waits and pings the controller). While a nice idea, this would horribly fail in a serialized, non-concurrent VM start/plugin call event stream where some VM should be started (which then are blocked) before the Controller VM is scheduled to be started. That would obviously result in a deadlock.

We will discuss these options with Proxmox, but we think the current solution is valuable in most typical use cases, as is. Especially, compared to the complexity of a pacemaker setup. Use cases where one can expect that not the whole cluster goes down at the same time are covered. And even if that is the case, only automatic startup of the VMs would not work when the whole cluster is started. In such a scenario the admin just has to wait until the Proxmox HA service starts the controller VM. After that all VMs can be started manually/scripted on the command line.

OpenNebula is a flexible and open source cloud management platform which allows its functionality to be extended via the use of addons.

The LINSTOR addon allows the deployment of virtual machines with highly available images backed by DRBD and attached across the network via DRBD’s own transport protocol.

Installation of the LINSTOR storage addon for OpenNebula requires a working OpenNebula cluster as well as a working LINSTOR cluster.

With access to LINBIT’s customer repositories you can install the linstor-opennebula with

or

Without access to LINBIT’s prepared packages you need to fall back to instructions on it’s GitHub page.

A DRBD cluster with LINSTOR can be installed and configured by following the instructions in this guide, see Initializing your cluster.

The OpenNebula and DRBD clusters can be somewhat independent of one another with the following exception: OpenNebula’s Front-End and Host nodes must be included in both clusters.

Host nodes do not need a local LINSTOR storage pools, as virtual machine images are attached to them across the network ^[1].

It is recommended to use LINSTOR resource groups to configure the deployment how you like it, see OpenNebula resource group. Previous auto-place and deployment nodes modes are deprecated.

Live migration is supported even with the use of the ssh system datastore, as well as the nfs shared system datastore.

Free space is calculated differently depending on whether resources are deployed automatically or on a per node basis.

For datastores which place per node, free space is reported based on the most restrictive storage pools from all nodes where resources are being deployed. For example, the capacity of the node with the smallest amount of total storage space is used to determine the total size of the datastore and the node with the least free space is used to determine the remaining space in the datastore.

For a datastore which uses automatic placement, size and remaining space are determined based on the aggregate storage pool used by the datastore as reported by LINSTOR.

Openstack consists of a wide range of individual services; the two that are mostly relevant to DRBD are Cinder and Nova. Cinder is the block storage service, while Nova is the compute node service that’s responsible for making the volumes available for the VMs.

The LINSTOR driver for OpenStack manages DRBD/LINSTOR clusters and makes them available within the OpenStack environment, especially within Nova compute instances. LINSTOR-backed Cinder volumes will seamlessly provide all the features of DRBD/LINSTOR while allowing OpenStack to manage all their deployment and management. The driver will allow OpenStack to create and delete persistent LINSTOR volumes as well as managing and deploying volume snapshots and raw volume images.

Aside from using the kernel-native DRBD protocols for replication, the LINSTOR driver also allows using iSCSI with LINSTOR cluster(s) to provide maximum compatibility. For more information on these two options, please see Choosing the Transport Protocol.

An initial installation and configuration of DRBD and LINSTOR must be completed prior to installing OpenStack driver. Each LINSTOR node in a cluster should also have a Storage Pool defined as well. Details about LINSTOR installation can be found here.

There are two main ways to run DRBD/LINSTOR with Cinder:

These are not exclusive; you can define multiple backends, have some of them use iSCSI, and others the DRBD protocol.

Docker is a platform for developing, shipping, and running applications in the form of Linux containers. For stateful applications that require data persistence, Docker supports the use of persistent volumes and volume_drivers.

The LINSTOR Docker Volume Plugin is a volume driver that provisions persistent volumes from a LINSTOR cluster for Docker containers.

To install the linstor-docker-volume plugin provided by LINBIT, you’ll need to have a working LINSTOR cluster. After that the plugin can be installed from the public docker hub.

As the plugin has to communicate to the LINSTOR controller via the LINSTOR python library, we must tell the plugin where to find the LINSTOR Controller node in its configuration file:

A more extensive example could look like this:

The following are some examples of how you might use the LINSTOR Docker Volume Plugin. In the following we expect a cluster consisting of three nodes (alpha, bravo, and charlie).

The LINSTOR client contains a sub-command that can generate a migration script that adds existing DRBDManage nodes and resources to a LINSTOR cluster. Migration can be done without downtime. If you do not plan to migrate existing resources, continue with the next section.

The first thing to check is if the DRBDManage cluster is in a healthy state. If the output of drbdmanage assignments looks good, you can export the existing cluster database via drbdmanage export-ctrlvol > ctrlvol.json. You can then use that as input for the LINSTOR client. The client does not immediately migrate your resources, it just generates a shell script. Therefore, you can run the migration assistant multiple times and review/modify the generated shell script before actually executing it. Migration script generation is started via linstor dm-migrate ctrlvol.json dmmmigrate.sh. The script will ask a few questions and then generate the shell script. After carefully reading the script, you can then shutdown DRBDManage, and rename the following files. If you do not rename them, the lower-level drbd-utils pick up both kinds of resource files, the ones from DRBDManage and the ones from LINSTOR.

Obviously, you need the linstor-controller service started on one node and the linstor-satellite service on all nodes.

We assume that the following steps are accomplished on all cluster nodes:

Note that drbdmanage uses dbus-activation to start its server component when necessary, do not start the server manually.

The first step is to review the configuration file of drbdmanage (/etc/drbdmanaged.cfg) and to create an LVM volume group with the name specified in the configuration. In the following we use the default name, which is drbdpool, and assume that the volume group consists of /dev/sda6 and /dev/sda7. Creating the volume group is a step that has to be executed on every cluster node:

The second step is to initialize the so called control volume, which is then used to redundantly store your cluster configuration. If the node has multiple interfaces, you have to specify the IP address of the network interface that DRBD should use to communicate with other nodes in the cluster, otherwise the IP is optional. This step must only be done on exactly one cluster node.

We recommend using ‘drbdpool’ as the name of your LVM volume group as it is the default value and makes your administration life easier. If, for whatever reason, you decide to use a different name, make sure that the option drbdctrl-vg is set accordingly in /etc/drbdmanaged.cfg. Configuration will be discussed in Cluster configuration.

Adding nodes to your cluster is easy and requires a single command with two parameters:

If DNS is configured properly, the tab-completion of drbdmanage is able to complete the IP of the given node name.

Here we assume that the command was executed on node ‘alpha’. If the ‘root’ user is allowed to execute commands as ‘root’ on ‘bravo’ via ssh, then the node ‘bravo’ will automatically join your cluster.

If ssh access with public-key authentication is not possible, drbdmanage will print a join command that has to be executed on node ‘bravo’. You can always query drbdmanage to output the join command for a specific node:

Drbdmanage knows many configuration settings like the log-level or the storage plugin that should be used (i.e., LVM, ThinLV, ThinPool, ZPool, or ThinZpool). Executing drbdmanage modify-config starts an editor that is used to specify these settings. The configuration is split in several sections. If an option is specified in the [GLOBAL] section, this setting is used in the entire cluster. Additionally, it is possible to specify settings per node and per site. Node sections follow a syntax of [Node:nodename]. If an option is set globally and per node, the node setting overrules the global setting.

It is also possible to group nodes into sites. In order to make node ‘alpha’ part of site ‘mysite’, you have to specify the ‘site’ option in alpha’s node section:

It is then also possible to specify drbdmanage settings per site using [Site:] sections. Lets assume that you want to set the ‘loglevel’ option in general to ‘INFO’, for site ‘mysite’ to ‘WARN’ and for node alpha, which is also part of site ‘mysite’ to DEBUG. This would result in the following configuration:

By executing drbdmanage modify-config without any options, you can edit global, per site and per node settings. It is also possible to execute ‘modify-config’ for a specific node. In this per-node view, it is possible to set further per-node specific settings like the storage plugin discussed in Configuring storage plugins.

Storage plugins are per node settings that are set with the help of the ‘modify-config’ sub command.

Lets assume you want to use the ‘ThinLV’ plugin for node ‘bravo’, where you want to set the ‘pool-name’ option to ‘mythinpool’:

In the following scenario we assume that the goal is to create a resource ‘backups’ with a size of ‘500 GB’ that is replicated among 3 cluster nodes. First we show how to achieve the goal in individual steps, and then show a short-cut how to achieve it in a single step:

First, we create a new resource:

Second, we create a new volume within that resource:

In case we would not have used ‘add-resource’ in the first step, drbdmanage would have known that the resource did not exist and it would have created it.

The third step is to deploy the resource to 3 cluster nodes:

In this case drbdmanage chooses 3 nodes that fit all requirements best, which is by default the set of nodes with the most free space in the drbdpool volume group. We will see how to manually assign resources to specific nodes in a moment.

As deploying a new resource/volume to a set of nodes is a very common task, drbdmanage provides the following short-cut:

Manual deployment can be achieved by assigning a resource to specific nodes. For example if you decide to assign the ‘backups’ resource to ‘bravo’ and ‘charlie’, you should execute the following steps:

In the following we assume that the ThinLV plugin is used on all nodes that have deployed resources from which snapshots should be taken. For further information on how to configure the storage plugin, please refer to Cluster configuration.

Drbdmanage provides various commands to check the state of your cluster. These commands start with a ‘list-‘ prefix and provide various filtering and sorting options. The ‘–groupby’ option can be used to group and sort the output in multiple dimensions. Additional output can be turned on by using the ‘–show’ option. In the following we show some typical examples:

Currently, it is possible to set the following drbdsetup options:

Additionally, it is possible to set DRBD event handler.

As for example net-options are allowed in the ‘common’ section as well as per resource, these commands then provide the according switches.

Setting max-buffers for a resource ‘backups’ looks like this:

Setting this option in the common section looks like this:

Additionally, there is always an ‘–unset-‘ option for every option that can be specified. So, unsetting max-buffers for a resource ‘backups’ looks like this:

It is possible to visualize currently set options with the ‘show-options’ subcommand.

Setting net-options per site is also supported. Lets assume ‘alpha’ and ‘bravo’ should be part of site ‘first’ and ‘charlie’ and ‘delta’ should be part of site ‘second’. Further, we want to use DRBD protocol ‘C’ within the two sites, and protocol ‘A’ between the sites ‘first’ and ‘second’. This would be set up as follows:

The ‘–sites’ parameter follows a ‘from:to’ syntax, where currently ‘from’ and ‘to’ have a symmetric semantic. Setting an option from ‘first:second’ also sets this option from ‘second:first’.

DRBD event handler can be set in the ‘common’ section and per resource:

Rebalancing data means moving some assignments around, to make better use of the available resources. We’ll discuss the same example as for the manual workflow.

Given is an example policy that data needs to be available on 3 nodes, so you need at least 3 servers for your setup.

Now, as your storage demands grow, you will encounter the need for additional servers. Rather than having to buy 3 more servers at the same time, you can rebalance your data across a single additional node.

First, you need to add the new machine to the cluster; see Adding nodes to your cluster for the commands.

The next step is to add the assignment:

Now you need to wait for the (initial) sync to finish; you can eg. use the command drbdadm status with (optionally) the resource name.

One of the nodes that still has the data will show a status like

while the target node will have a state of SyncTarget.

When the target assignment reaches a state of UpToDate, you have a full additional copy of your data on this node; now it is safe to remove the assignment from another node:

And voilà – you moved one assignment, in two^[4] easy steps!

The easiest way to get an overview about drbdmanage’s subcommands is to read the main man-page (man drbdmanage).

A quick way to list available commands on the command line is to type drbdmanage list.

Further information on subcommands (e.g., list-nodes) can be retrieved in three ways:

Using the ‘help’ subcommand is especially helpful when drbdmanage is executed in interactive mode (drbdmanage interactive).

One of the most helpful features of drbdmanage is its rich tab-completion, which can be used to complete basically every object drbdmanage knows about (e.g., node names, IP addresses, resource names, …​). In the following we show some possible completions, and their results:

If tab-completion does not work out of the box, please try to source the according file: