This section goes over core concepts and terms that you will need to familiarize yourself with to understand how LINSTOR works and deploys storage. The section is laid out in a “ground up” approach.

While LINSTOR might be used to make the management of DRBD more convenient, it is often integrated with software stacks higher up. Such integration exist already for Kubernetes, OpenStack, OpenNebula and Proxmox. Chapters specific to deploying LINSTOR in these environments are included in this guide.

The southbound drivers used by LINSTOR are LVM, thinLVM and ZFS.

LINSTOR is packaged in both the .rpm and the .deb variants:

For further detail about these packages see the Installable Components section above.

LINSTOR doesn’t support rolling upgrade, controller and satellites must have the same version, otherwise the controller with discard the satellite with a VERSION_MISMATCH. But this isn’t a problem, as the satellite won’t do any actions as long it isn’t connected to a controller and DRBD will not be disrupted by any means.

If you are using the embedded default H2 database and the linstor-controller package is upgraded an automatic backup file of the database will be created in the default /var/lib/linstor directory. This file is a good restore point if for any reason a linstor-controller database migration should fail, than it is recommended to report the error to Linbit and restore the old database file and downgrade to your previous controller version.

If you use any external database or etcd, it is recommended to do a manually backup of your current database to have a restore point.

So first upgrade the linstor-controller, linstor-client package on you controller host and restart the linstor-controller, the controller should start and all of it’s client should show OFFLINE(VERSION_MISMATCH). After that you can continue upgrading linstor-satellite on all satellite nodes and restart them, after a short reconnection time they should all show ONLINE again and your upgrade is finished.

LINSTOR is also available as containers. The base images are available in LINBIT’s container registry, drbd.io.

In order to access the images, you first have to login to the registry (reach out to sales@linbit.com for credentials):

The containers available in this repo are:

An up to date list of available images with versions can be retrieved by opening http://drbd.io in your browser. Make sure to access the host via “http”, as the registry’s images themselves are served via “https”.

To load the kernel module, needed only for LINSTOR satellites, you’ll need to run a drbd9-$dist container in privileged mode. The kernel module containers either retrieve an official LINBIT package from a customer repository, use shipped packages, or they try to build the kernel modules from source. If you intend to build from source, you need to have the according kernel headers (e.g., kernel-devel) installed on the host. There are 4 ways to execute such a module load container:

Example building from shipped source (RHEL based):

Example using a module shipped with the container, which is enabled by not bind-mounting /usr/src:

Example using a hash and a distribution (rarely used):

Example using an existing repo config (rarely used):

Then run the LINSTOR satellite container, also privileged, as a daemon:

To run the LINSTOR controller container as a daemon, mapping ports 3370, 3376 and 3377 on the host to the container:

To interact with the containerized LINSTOR cluster, you can either use a LINSTOR client installed on a system via packages, or via the containerized LINSTOR client. To use the LINSTOR client container:

From this point you would use the LINSTOR client to initialize your cluster and begin creating resources using the typical LINSTOR patterns.

To stop and remove a daemonized container and image:

We assume that the following steps are accomplished on all cluster nodes:

Start and enable the linstor-controller service on the host where it has been installed:

If you are sure the linstor-controller service gets automatically enabled on installation you can use the following command as well:

Whenever you run the LINSTOR command line client, it needs to know where your linstor-controller runs. If you do not specify it, it will try to reach a locally running linstor-controller listening on IP 127.0.0.1 port 3376. Therefore we will use the linstor-client on the same host as the linstor-controller.

should give you an empty list and not an error message.

You can use the linstor command on any other machine, but then you need to tell the client how to find the linstor-controller. As shown, this can be specified as a command line option, an environment variable, or in a global file:

Alternatively you can create the /etc/linstor/linstor-client.conf file and populate it like below.

If you have multiple linstor-controllers configured you can simply specify them all in a comma separated list. The linstor-client will simply try them in the order listed.

The next step is to add nodes to your LINSTOR cluster.

If the IP is omitted, the client will try to resolve the given node-name as host-name by itself.

Linstor will automatically detect the node’s local uname -n which is later used for the DRBD-resource.

When you use linstor node list you will see that the new node is marked as offline. Now start and enable the linstor-satellite on that node so that the service comes up on reboot as well:

You can also use systemctl start linstor-satellite if you are sure that the service is already enabled as default and comes up on reboot.

About 10 seconds later you will see the status in linstor node list becoming online. Of course the satellite process may be started before the controller knows about the existence of the satellite node.

If you want to have other services wait until the linstor-satellite had a chance to create the necessary devices (i.e. after a boot), you can update the corresponding .service file and change Type=simple to Type=notify.

This will cause the satellite to delay sending the READY=1 message to systemd until the controller connects, sends all required data to the satellite and the satellite at least tried once to get the devices up and running.

StoragePools identify storage in the context of LINSTOR. To group storage pools from multiple nodes, simply use the same name on each node. For example, one valid approach is to give all SSDs one name and all HDDs another.

On each host contributing storage, you need to create either an LVM VG or a ZFS zPool. The VGs and zPools identified with one LINSTOR storage pool name may have different VG or zPool names on the hosts, but do yourself a favor and use the same VG or zPool name on all nodes.

These then need to be registered with LINSTOR:

To list your storage-pools you can use:

or using the short version

Should the deletion of the storage pool be prevented due to attached resources or snapshots with some of its volumes in another still functional storage pool, hints will be given in the ‘status’ column of the corresponding list-command (e.g. linstor resource list). After deleting the LINSTOR-objects in the lost storage pool manually, the lost-command can be executed again to ensure a complete deletion of the storage pool and its remaining objects.

A resource group is a parent object of a resource definition where all property changes made on a resource group will be inherited by it’s resource definition children. The resource group also stores settings for automatic placement rules and can spawn a resource definition depending on the stored rules.

In simpler terms, resource groups are like templates that define characteristics of resources created from them. Changes to these pseudo templates will be applied to all resources that were created from the resource group, retroactively.

A simple pattern for deploying resources using resource groups would look like this:

The commands above would result in a resource named ‘my_ssd_res’ with a 20GB volume replicated twice being automatically provisioned from nodes who participate in the storage pool named ‘pool_ssd’.

A more useful pattern could be to create a resource group with settings you’ve determined are optimal for your use case. Perhaps you have to run nightly online verifications of your volumes’ consistency, in that case, you could create a resource group with the ‘verify-alg’ of your choice already set so that resources spawned from the group are pre-configured with ‘verify-alg’ set:

The commands above result in twenty 10GiB resources being created each with the ‘crc32c’ ‘verify-alg’ pre-configured.

You can tune the settings of individual resources or volumes spawned from resource groups by setting options on the respective resource-definition or volume-definition. For example, if ‘res11’ from the example above is used by a very active database receiving lots of small random writes, you might want to increase the ‘al-extents’ for that specific resource:

If you configure a setting in a resource-definition that is already configured on the resource-group it was spawned from, the value set in the resource-definition will override the value set on the parent resource-group. For example, if the same ‘res11’ was required to use the slower but more secure ‘sha256’ hash algorithm in its verifications, setting the ‘verify-alg’ on the resource-definition for ‘res11’ would override the value set on the resource-group:

In the following scenario we assume that the goal is to create a resource ‘backups’ with a size of ‘500 GB’ that is replicated among three cluster nodes.

First, we create a new resource definition:

Second, we create a new volume definition within that resource definition:

If you want to change the size of the volume-definition you can simply do that by:

The parameter 0 is the number of the volume in the resource backups. You have to provide this parameter , because resources can have multiple volumes and they are identified by a so called volume-number. This number can be found by listing the volume-definitions.

So far we have only created objects in LINSTOR’s database, not a single LV was created on the storage nodes. Now you have the choice of delegating the task of placement to LINSTOR or doing it yourself.

By using the --drbd-diskless option instead of --storage-pool you can have a permanently diskless DRBD device on a node. This means that the resource will appear as block device and can be mounted to the filesystem without an existing storage-device. The data of the resource is accessed over the network on another nodes with the same resource.

The so called consistency group is a feature from DRBD. It is mentioned in this user-guide, due to the fact that one of LINSTOR’s main functions is to manage storage-clusters with DRBD. Multiple volumes in one resource are a consistency group.

This means that changes on different volumes from one resource are getting replicated in the same chronological order on the other Satellites.

Therefore you don’t have to worry about the timing if you have interdependent data on different volumes in a resource.

To deploy more than one volume in a LINSTOR-resource you have to create two volume-definitions with the same name.

This can be achieved by setting the StorPoolName property to the volume definitions before the resource is deployed to the nodes:

Here the ‘resource create’ commands do not need a --storage-pool option. In this case LINSTOR uses a ‘fallback’ storage pool. Finding that storage pool, LINSTOR queries the properties of the following objects in the following order:

If none of those objects contain a StorPoolName property, the controller falls back to a hard-coded ‘DfltStorPool’ string as a storage pool.

This also means that if you forgot to define a storage pool prior deploying a resource, you will get an error message that LINSTOR could not find the storage pool named ‘DfltStorPool’.

LINSTOR can be used without DRBD as well. Without DRBD, LINSTOR is able to provision volumes from LVM and ZFS backed storage pools, and create those volumes on individual nodes in your LINSTOR cluster.

Currently LINSTOR supports the creation of LVM and ZFS volumes with the option of layering some combinations of LUKS, DRBD, and/or NVMe-oF/NVMe-TCP on top of those volumes.

For example, assume we have a Thin LVM backed storage pool defined in our LINSTOR cluster named, thin-lvm:

We could use LINSTOR to create a Thin LVM on linstor-d that’s 100GiB in size using the following commands:

You should then see you have a new Thin LVM on linstor-d. You can extract the device path from LINSTOR by listing your linstor resources with the --machine-readable flag set:

If you wanted to layer DRBD on top of this volume, which is the default --layer-list option in LINSTOR for ZFS or LVM backed volumes, you would use the following resource creation pattern instead:

You would then see that you have a new Thin LVM backing a DRBD volume on linstor-d:

The following table shows which layer can be followed by which child-layer:

LINSTOR can deal with multiple network interface cards (NICs) in a machine, they are called netif in LINSTOR speak.

Additional NICs are created like this:

NICs are identified by the IP address only, the name is arbitrary and is not related to the interface name used by Linux. The NICs can be assigned to storage pools so that whenever a resource is created in such a storage pool, the DRBD traffic will be routed through the specified NIC.

FIXME describe how to route the controller <-> client communication through a specific netif.

LINSTOR can handle transparent encryption of drbd volumes. dm-crypt is used to encrypt the provided storage from the storage device.

Basic steps to use encryption:

LINSTOR provides various commands to check the state of your cluster. These commands start with a ‘list-‘ prefix and provide various filtering and sorting options. The ‘–groupby’ option can be used to group and sort the output in multiple dimensions.

Snapshots are supported with thin LVM and ZFS storage pools.

DRBD options are set using LINSTOR commands. Configuration in files such as /etc/drbd.d/global_common.conf that are not managed by LINSTOR will be ignored. The following commands show the usage and available options:

For instance, it is easy to set the DRBD protocol for a resource named backups:

LINSTOR can convert resources between diskless and having a disk. This is achieved with the resource toggle-disk command, which has syntax similar to resource create.

For instance, add a disk to the diskless resource backups on ‘alpha’:

Remove this disk again:

LINSTOR expects DRBD Proxy to be running on the nodes which are involved in the relevant connections. It does not currently support connections via DRBD Proxy on a separate node.

Suppose our cluster consists of nodes ‘alpha’ and ‘bravo’ in a local network and ‘charlie’ at a remote site, with a resource definition named backups deployed to each of the nodes. Then DRBD Proxy can be enabled for the connections to ‘charlie’ as follows:

The DRBD Proxy configuration can be tailored with commands such as:

LINSTOR does not automatically optimize the DRBD configuration for long-distance replication, so you will probably want to set some configuration options such as the protocol:

Please contact LINBIT for assistance optimizing your configuration.

It is possible to have LINSTOR working with an external database provider like Postgresql, MariaDB and since version 1.1.0 even ETCD key value store is supported.

To use an external database there are a few additional steps to configure. You have to create a DB/Schema and user to use for linstor, and configure this in the /etc/linstor/linstor.toml.

To make LINSTOR’s administrative tasks more accessible and also available for web-frontends a REST-API has been created. The REST-API is embedded in the linstor-controller and since LINSTOR 0.9.13 configured via the linstor.toml configuration file.

If you want to use the REST-API the current documentation can be found on the following link: https://app.swaggerhub.com/apis-docs/Linstor/Linstor/

Linstor uses SLF4J with Logback as binding. This gives Linstor the possibility to distinguish between the log levels ERROR, WARN, INFO, DEBUG and TRACE (in order of increasing verbosity). In the current linstor version (1.1.2) the user has the following four methods to control the logging level, ordered by priority (first has highest priority):

See the Logback Manual to find more details about logback.xml.

When none of the configuration methods above is used Linstor will default to INFO log level.

Since LINSTOR 1.8.0, a Prometheus /metrics HTTP path is provided with LINSTOR and JVM specific exports.

The /metrics path also supports 3 GET arguments to reduce LINSTOR’s reported data:

These are all default true, to disabled e.g. error-report data: http://localhost:3070/metrics?error_reports=false

It is possible to have the LINSTOR use SSL secure TCP connection between controller and satellite connections. Without going into further details on how java’s SSL engine works we will give you command line snippets using the keytool from java’s runtime environment on how to configure a 3 node setup using secure connections. The node setup looks like this:

Node alpha is the just the controller. Node bravo and node charlie are just satellites.

Here are the commands to generate such a keystore setup, values should of course be edited for your environment.

Now just start controller and satellites and add the nodes with --communication-type SSL.

Kubernetes is a container orchestrator. Kubernetes defines the behavior of containers and related services via declarative specifications. In this guide, we’ll focus on using kubectl to manipulate .yaml files that define the specifications of Kubernetes objects.

The Controller pod includes a LINSTOR Client, making it easy to interact directly with LINSTOR. For instance:

This should only be necessary for investigating problems and accessing advanced functionality. Regular operation such as creating volumes should be achieved via the Kubernetes integration.

The operator Helm chart deploys the LINSTOR CSI plugin for you so if you used that, you can skip this section.

If you are integrating LINSTOR using a different method, you will need to install the LINSTOR CSI plugin. Instructions for deploying the CSI plugin can be found on the project’s github. This will result in a linstor-csi-controller Deployment and a linstor-csi-node DaemonSet running in the kube-system namespace.

Once all linstor-csi Pods are up and running, we can provision volumes using the usual Kubernetes workflows.

Configuring the behavior and properties of LINSTOR volumes deployed via Kubernetes is accomplished via the use of StorageClasses.

Here below is the simplest practical StorageClass that can be used to deploy volumes:

DRBD options can be set as well in the parameters section. Valid keys are defined in the LINSTOR REST-API (e.g., DrbdOptions/Net/allow-two-primaries: "yes").

We can create the StorageClass with the following command:

Now that our StorageClass is created, we can now create a PersistentVolumeClaim which can be used to provision volumes known both to Kubernetes and LINSTOR:

We can create the PersistentVolumeClaim with the following command:

This will create a PersistentVolumeClaim known to Kubernetes, which will have a PersistentVolume bound to it, additionally LINSTOR will now create this volume according to the configuration defined in the linstor-basic-storage-class StorageClass. The LINSTOR volume’s name will be a UUID prefixed with csi- This volume can be observed with the usual linstor resource list. Once that volume is created, we can attach it to a pod. The following Pod spec will spawn a Fedora container with our volume attached that busy waits so it is not unscheduled before we can interact with it:

We can create the Pod with the following command:

Running kubectl describe pod fedora can be used to confirm that Pod scheduling and volume attachment succeeded.

To remove a volume, please ensure that no pod is using it and then delete the PersistentVolumeClaim via kubectl. For example, to remove the volume that we just made, run the following two commands, noting that the Pod must be unscheduled before the PersistentVolumeClaim will be removed:

Creating snapshots and creating new volumes from snapshots is done via the use of VolumeSnapshots, VolumeSnapshotClasses, and PVCs.

LINSTOR volumes are typically accessible both locally and over the network.

By default, the CSI plugin will attach volumes directly if the Pod happens to be scheduled on a kubelet where its underlying storage is present. However, Pod scheduling does not currently take volume locality into account. The replicasOnSame parameter can be used to restrict where the underlying storage may be provisioned, if locally attached volumes are desired.

See placementPolicy to see how this default behavior can be modified.

Stork is a scheduler extender plugin for Kubernetes which allows a storage driver to give the Kubernetes scheduler hints about where to place a new pod so that it is optimally located for storage performance. You can learn more about the project on its GitHub page.

The next Stork release will include the LINSTOR driver by default. In the meantime, you can use a custom-built Stork container by LINBIT which includes a LINSTOR driver, available on Docker Hub

The LINSTOR High Availability Controller (HA Controller) will speed up the fail over process for stateful workloads using LINSTOR for storage. It is deployed by default, and can be scaled to multiple replicas:

In the event of node failures, Kubernetes is very conservative in rescheduling stateful workloads. This means it can take more than 15 minutes for Pods to be moved from unreachable nodes. With the information available to DRBD and LINSTOR, this process can be sped up significantly.

The HA Controller enables fast fail over for:

A LINSTOR Deployment on Kubernets can be upgraded to a new release using Helm.

Before upgrading to a new release, you should ensure you have an up-to-date backup of the LINSTOR database. If you are using the Etcd database packaged in the LINSTOR Chart, see here

Upgrades will update to new versions of the following components:

Some versions require special steps, please take a look here The main command to upgrade to a new LINSTOR operator version is:

If you used any customizations on the initial install, pass the same options to helm upgrade. For example:

would become

This triggers the rollout of new pods. After a short wait, all pods should be running and ready. Check that no errors are listed in the status section of LinstorControllers, LinstorSatelliteSets and LinstorCSIDrivers.

OpenShift is the official Red Hat developed and supported distribution of Kubernetes. As such, you can easily deploy Piraeus or the LINSTOR operator using Helm or via example yamls as mentioned in the previous chapter, LINSTOR Volumes in Kubernetes.

Some of the value of Red Hat’s Openshift is that it includes its own registry of supported and certified images and operators, in addition to a default and standard web console. This chapter describes how to install the Certified LINSTOR operator via these tools.

The Controller pod includes a LINSTOR Client, making it easy to interact directly with LINSTOR. For instance:

This should only be necessary for investigating problems and accessing advanced functionality. Regular operation such as creating volumes should be achieved via the Kubernetes integration.

Once the operator and all the needed pods are deployed, provisioning volumes simply follows the usual Kubernetes workflows.

As such, please see the previous chapter’s section on Basic Configuration and Deployment.

Some additional components are not included in the OperatorHub version of the LINSTOR Operator when compared to the Helm deployment. Most notably, this includes setting up Etcd and deploying the STORK integration.

Etcd can be deployed by using the Etcd Operator available in the OperatorHub.

Proxmox VE is an easy to use, complete server virtualization environment with KVM, Linux Containers and HA.

‘linstor-proxmox’ is a Perl plugin for Proxmox that, in combination with LINSTOR, allows to replicate VM disks on several Proxmox VE nodes. This allows to live-migrate active VMs within a few seconds and with no downtime without needing a central SAN, as the data is already replicated to multiple nodes.

If this is a fresh installation, skip this section and continue with Proxmox Plugin Installation.

LINBIT provides a dedicated public repository for Proxmox VE users. This repository not only contains the Proxmox plugin, but the whole DRBD SDS stack including a DRBD SDS kernel module and user space utilities.

The DRBD9 kernel module is installed as a dkms package (i.e., drbd-dkms), therefore you’ll have to install pve-headers package, before you set up/install the software packages from LINBIT’s repositories. Following that order, ensures that the kernel module will build properly for your kernel. If you don’t plan to install the latest Proxmox kernel, you have to install kernel headers matching your current running kernel (e.g., pve-headers-$(uname -r)). If you missed this step, then still you can rebuild the dkms package against your current kernel, (kernel headers have to be installed in advance), by issuing apt install --reinstall drbd-dkms command.

LINBIT’s repository can be enabled as follows, where “$PVERS” should be set to your Proxmox VE major version (e.g., “6”, not “6.1”):

For the rest of this guide we assume that you have a LINSTOR cluster configured as described in Initializing your cluster. Also make sure to setup each node as a “Combined” node. Start the “linstor-controller” on one node, and the “linstor-satellite” on all nodes. The preferred way to use the plugin, starting from version 4.1.0, is via LINSTOR resource groups and a single volume group within every resource group. LINSTOR resource groups are described in Resource groups. All the required LINSTOR configuration (e.g., redundancy count) has to be set on the resource group.

The final step is to provide a configuration for Proxmox itself. This can be done by adding an entry in the /etc/pve/storage.cfg file, with a content similar to the following.

The “drbd” entry is fixed and you are not allowed to modify it, as it tells to Proxmox to use DRBD as storage backend. The “drbdstorage” entry can be modified and is used as a friendly name that will be shown in the PVE web GUI to locate the DRBD storage. The “content” entry is also fixed, so do not change it. The redundancy (specified in the resource group) specifies how many replicas of the data will be stored in the cluster. The recommendation is to set it to 2 or 3 depending on your setup. The data is accessible from all nodes, even if some of them do not have local copies of the data. For example, in a 5 node cluster, all nodes will be able to access 3 copies of the data, no matter where they are stored in. The “controller” parameter must be set to the IP of the node that runs the LINSTOR controller service. Only one node can be set to run as LINSTOR controller at the same time. If that node fails, start the LINSTOR controller on another node and change that value to its IP address.

Recent versions of the plugin allow to define multiple different storage pools. Such a configuration would look like this:

By now, you should be able to create VMs via Proxmox’s web GUI by selecting “drbdstorage“, or any other of the defined pools as storage location.

Starting from version 5 of the plugin one can set the option “preferlocal yes”. If it is set, the plugin tries to create a diskful assignment on the node that issued the storage create command. With this option one can make sure the VM gets local storage if possible. Without that option LINSTOR might place the storage on nodes ‘B’ and ‘C’, while the VM is initially started on node ‘A’. This would still work as node ‘A’ then would get a diskless assignment, but having local storage might be preferred.

At this point you can try to live migrate the VM – as all data is accessible on all nodes (even on Diskless nodes) – it will take just a few seconds. The overall process might take a bit longer if the VM is under load and if there is a lot of RAM being dirtied all the time. But in any case, the downtime should be minimal and you will see no interruption at all.

This section describes how the controller can be made highly available, but this is not a must. Please read the entire section before you start, and then decide if the increased complexity, and the limitations are worth it. Or if you are better off by taking regular backups of the LINSTOR controller database and starting a (temporary) controller with the database backup on one of the remaining satellites if the current controller is beyond repair.

For the rest of this guide we assume that you installed LINSTOR and the Proxmox Plugin as described in LINSTOR Configuration.

The basic idea is to execute the LINSTOR controller within a VM that is controlled by Proxmox and its HA features, where the storage resides on DRBD managed by LINSTOR itself.

The first step is to allocate storage for the VM: Create a VM as usual and select “Do not use any media” on the “OS” section. The hard disk should of course reside on DRBD (e.g., “drbdstorage”). 2GB disk space should be enough, and for RAM we chose 1GB. These are the minimum requirements for the appliance LINBIT provides to its customers (see below). If you wish to set up your own controller VM, and you have enough hardware resources available, you can increase these minimum values. In the following use case, we assume that the controller VM was created with ID 100, but it is fine if this VM was created at a later time and has a different ID.

LINBIT provides an appliance for its customers that can be used to populate the created storage. For the appliance to work, we first create a “Serial Port”. First click on “Hardware” and then on “Add” and finally on “Serial Port”:

If everything worked as expected the VM definition should then look like this:

The next step is to copy the VM appliance to the VM disk storage. This can be done with qemu-img.

Once completed you can start the VM and connect to it via the Proxmox VNC viewer. The default user name and password are both “linbit”. Note that we kept the default configuration for the ssh server, so you will not be able to log in to the VM via ssh and username/password. If you want to enable that (and/or “root” login), enable these settings in /etc/ssh/sshd_config and restart the ssh service. As this VM is based on “Ubuntu Bionic”, you should change your network settings (e.g., static IP) in /etc/netplan/config.yaml. After that you should be able to ssh to the VM:

In the next step you add the controller VM to the existing cluster:

In our test cluster the Controller VM disk was created in DRBD storage and it was initially assigned to one host (use linstor resource list to check the assignments). Then, we used linstor resource create command to create additional resource assignments to the other nodes of the cluster for this VM. In our lab consisting of four nodes, we created all resource assignments as diskful, but diskless assignments are fine as well. As a rule of thumb keep the redundancy count at “3” (more usually does not make sense), and assign the rest as diskless.

As the storage for the Controller VM must be made available on all PVE hosts in some way, we must make sure to enable the drbd.service on all hosts (given that it is not controlled by LINSTOR at this stage):

By default, at startup the linstor-satellite service deletes all of its resource files (.res) and regenerates them. This conflicts with the drbd services that needs these resource files to start the controller VM. It is good enough to first bring up the resources via drbd.service and ensure that the linstor-satellite.service, which brings up the controller resource never deletes the according res file. To make the necessary changes, you need to create a drop-in for the linstor-satellite.service via systemctl (do *not edit the file directly).

Of course adapt the name of the controller VM in the LS_KEEP_RES variable. Note that the value given is interpreted as regex, so you don’t need to specify the exact name.

Don’t forget to restart the linstor-satellite.service.

After that, it is time for the final steps, namely switching from the existing controller (residing on the physical host) to the new one in the VM. So let’s stop the old controller service on the physical host, and copy the LINSTOR controller database to the VM host:

Finally, we can enable the controller in the VM:

To check if everything worked as expected, you can query the cluster nodes on a physical PVE host by asking the controller in the VM: linstor --controllers=10.43.7.254 node list. It is perfectly fine that the controller (which is just a Controller and not a “Combined” host) is shown as “OFFLINE”. This might change in the future to something more reasonable.

As the last — but crucial — step, you need to add the “controllervm” option to /etc/pve/storage.cfg, and change the controller IP address to the IP address of the Controller VM:

Please note the additional setting “controllervm”. This setting is very important, as it tells to PVE to handle the Controller VM differently than the rest of VMs stored in the DRBD storage. In specific, it will instruct PVE to NOT use LINSTOR storage plugin for handling the Controller VM, but to use other methods instead. The reason for this, is that simply LINSTOR backend is not available at this stage. Once the Controller VM is up and running (and the associated LINSTOR controller service inside the VM), then the PVE hosts will be able to start the rest of virtual machines which are stored in the DRBD storage by using LINSTOR storage plugin. Please make sure to set the correct VM ID in the “controllervm” setting. In this case is set to “100”, which represents the ID assigned to our Controller VM.

It is very important to make sure that the Controller VM is up and running at all times and that you are backing it up at regular times(mostly when you do modifications to the LINSTOR cluster). Once the VM is gone, and there are no backups, the LINSTOR cluster must be recreated from scratch.

To prevent accidental deletion of the VM, you can go to the “Options” tab of the VM, in the PVE GUI and enable the “Protection” option. If however you accidentally deleted the VM, such requests are ignored by our storage plugin, so the VM disk will NOT be deleted from the LINSTOR cluster. Therefore, it is possible to recreate the VM with the same ID as before(simply recreate the VM configuration file in PVE and assign the same DRBD storage device used by the old VM). The plugin will just return “OK”, and the old VM with the old data can be used again. In general, be careful to not delete the controller VM and “protect” it accordingly.

Currently, we have the controller executed as VM, but we should make sure that one instance of the VM is started at all times. For that we use Proxmox’s HA feature. Click on the VM, then on “More”, and then on “Manage HA”. We set the following parameters for our controller VM:

As long as there are surviving nodes in your Proxmox cluster, everything should be fine and in case the node hosting the Controller VM is shut down or lost, Proxmox HA will make sure the controller is started on another host. Obviously the IP of the controller VM should not change. It is up to you as an administrator to make sure this is the case (e.g., setting a static IP, or always providing the same IP via dhcp on the bridged interface).

It is important to mention at this point that in the case that you are using a dedicated network for the LINSTOR cluster, you must make sure that the network interfaces configured for the cluster traffic, are configured as bridges (i.e vmb1,vmbr2 etc) on the PVE hosts. If they are setup as direct interfaces (i.e eth0,eth1 etc), then you will not be able to setup the Controller VM vNIC to communicate with the rest of LINSTOR nodes in the cluster, as you cannot assign direct network interfaces to the VM, but only bridged interfaces.

One limitation that is not fully handled with this setup is a total cluster outage (e.g., common power supply failure) with a restart of all cluster nodes. Proxmox is unfortunately pretty limited in this regard. You can enable the “HA Feature” for a VM, and you can define “Start and Shutdown Order” constraints. But both are completely separated from each other. Therefore it is hard/impossible to guarantee that the Controller VM will be up and running, before all other VMs are started.

It might be possible to work around that by delaying VM startup in the Proxmox plugin itself until the controller VM is up (i.e., if the plugin is asked to start the controller VM it does it, otherwise it waits and pings the controller). While a nice idea, this would horribly fail in a serialized, non-concurrent VM start/plugin call event stream where some VM should be started (which then are blocked) before the Controller VM is scheduled to be started. That would obviously result in a deadlock.

We will discuss these options with Proxmox, but we think the current solution is valuable in most typical use cases, as is. Especially, compared to the complexity of a pacemaker setup. Use cases where one can expect that not the whole cluster goes down at the same time are covered. And even if that is the case, only automatic startup of the VMs would not work when the whole cluster is started. In such a scenario the admin just has to wait until the Proxmox HA service starts the controller VM. After that all VMs can be started manually/scripted on the command line.

OpenNebula is a flexible and open source cloud management platform which allows its functionality to be extended via the use of addons.

The LINSTOR addon allows the deployment of virtual machines with highly available images backed by DRBD and attached across the network via DRBD’s own transport protocol.

Installation of the LINSTOR storage addon for OpenNebula requires a working OpenNebula cluster as well as a working LINSTOR cluster.

With access to LINBIT’s customer repositories you can install the linstor-opennebula with

or

Without access to LINBIT’s prepared packages you need to fall back to instructions on it’s GitHub page.

A DRBD cluster with LINSTOR can be installed and configured by following the instructions in this guide, see Initializing your cluster.

The OpenNebula and DRBD clusters can be somewhat independent of one another with the following exception: OpenNebula’s Front-End and Host nodes must be included in both clusters.

Host nodes do not need a local LINSTOR storage pools, as virtual machine images are attached to them across the network ^[1].

It is recommended to use LINSTOR resource groups to configure the deployment how you like it, see OpenNebula resource group. Previous auto-place and deployment nodes modes are deprecated.

Live migration is supported even with the use of the ssh system datastore, as well as the nfs shared system datastore.

Free space is calculated differently depending on whether resources are deployed automatically or on a per node basis.

For datastores which place per node, free space is reported based on the most restrictive storage pools from all nodes where resources are being deployed. For example, the capacity of the node with the smallest amount of total storage space is used to determine the total size of the datastore and the node with the least free space is used to determine the remaining space in the datastore.

For a datastore which uses automatic placement, size and remaining space are determined based on the aggregate storage pool used by the datastore as reported by LINSTOR.

Openstack consists of a wide range of individual services; the two that are mostly relevant to DRBD are Cinder and Nova. Cinder is the block storage service, while Nova is the compute node service that’s responsible for making the volumes available for the VMs.

The LINSTOR driver for OpenStack manages DRBD/LINSTOR clusters and makes them available within the OpenStack environment, especially within Nova compute instances. LINSTOR-backed Cinder volumes will seamlessly provide all the features of DRBD/LINSTOR while allowing OpenStack to manage all their deployment and management. The driver will allow OpenStack to create and delete persistent LINSTOR volumes as well as managing and deploying volume snapshots and raw volume images.

Aside from using the kernel-native DRBD protocols for replication, the LINSTOR driver also allows using iSCSI with LINSTOR cluster(s) to provide maximum compatibility. For more information on these two options, please see Choosing the Transport Protocol.

An initial installation and configuration of DRBD and LINSTOR must be completed prior to installing OpenStack driver. Each LINSTOR node in a cluster should also have a Storage Pool defined as well. Details about LINSTOR installation can be found here.

There are two main ways to run DRBD/LINSTOR with Cinder:

These are not exclusive; you can define multiple backends, have some of them use iSCSI, and others the DRBD protocol.

Docker is a platform for developing, shipping, and running applications in the form of Linux containers. For stateful applications that require data persistence, Docker supports the use of persistent volumes and volume_drivers.

The LINSTOR Docker Volume Plugin is a volume driver that provisions persistent volumes from a LINSTOR cluster for Docker containers.

To install the linstor-docker-volume plugin provided by LINBIT, you’ll need to have a working LINSTOR cluster. After that the plugin can be installed from the public docker hub.

As the plugin has to communicate to the LINSTOR controller via the LINSTOR python library, we must tell the plugin where to find the LINSTOR Controller node in its configuration file:

A more extensive example could look like this:

The following are some examples of how you might use the LINSTOR Docker Volume Plugin. In the following we expect a cluster consisting of three nodes (alpha, bravo, and charlie).