对象是LINSTOR呈现给最终用户或应用程序的最终结果，例如Kubernetes/OpenShift、复制块设备（DRBD）、NVMeOF目标等。

如果您想选择使用DRBD创建复制存储，则需要安装 drbd-dkms 和 drbd-utils 。这些包需要安装在所有节点上。您还需要选择一个卷管理器，ZFS或LVM，在本例中我们使用的是LVM。

根据节点是LINSTOR controller、satellite还是两者（组合）将确定该节点上需要哪些包。对于组合型节点，我们需要controller和satellite 的LINSTOR包。

组合节点：

这将使我们剩余的节点成为我们的Satellites，因此我们需要在它们上安装以下软件包：

SLES高可用性扩展（HAE）包括DRBD。

On SLES, DRBD is normally installed via the software installation component of YaST2. It comes bundled with the High Availability package selection.

当我们下载DRBD的最新模块时，我们可以检查LVM工具是否也是最新的。喜欢命令行安装的用户可用以下命令获得最新的DRBD和LVM版本：

根据节点是LINSTOR controller、satellite还是两者（组合）将确定该节点上需要哪些包。对于组合型节点，我们需要controller和satellite 的LINSTOR包。

组合节点：

这将使我们剩余的节点成为我们的Satellites，因此我们需要在它们上安装以下软件包：

CentOS从第5版开始就有了DRBD 8。对于DRBD 9，您需要查看EPEL和类似的源代码。或者，如果您与LINBIT有签订支持合同，您可以利用我们的RHEL 8存储库。可以使用 yum 安装DRBD。我们也可以检查最新版本的LVM工具。

根据节点是LINSTOR controller、satellite还是两者（组合）将确定该节点上需要哪些包。对于组合型节点，我们需要controller和satellite 的LINSTOR包。

组合节点：

这将使我们剩余的节点成为我们的Satellites，因此我们需要在它们上安装以下软件包：

在仅只有一种存储和热修复存储设备功能的集群中，可以选择一种模型，在该模型中为每个物理备份设备创建一个存储池。此模型的优点是将故障域限制在单个存储设备上。

Since linstor-server 1.5.2 and a recent linstor-client, LINSTOR can create LVM/ZFS pools on a satellite for you. The linstor-client has the following commands to list possible disks and create storage pools, but such LVM/ZFS pools are not managed by LINSTOR and there is no delete command, so such action must be done manually on the nodes.

Will give you a list of available disks grouped by size and rotational(SSD/Magnetic Disk).

It will only show disks that pass the following filters:

With the create-device-pool command you can create a LVM pool on a disk and also directly add it as a storage-pool in LINSTOR.

If the --storage-pool option was provided, LINSTOR will create a storage-pool with the given name.

For more options and exact command usage please check the linstor-client help.

LINSTOR支持如下的存储插件：

使用 resource create 命令，可以将资源定义显式分配给命名节点。

autoplace之后的值告诉LINSTOR您想要多少个副本。storage-pool 选项应该很明显。

可能不太明显的是，您可以省略 --storage-pool 选项，然后LINSTOR可以自己选择一个存储池。选择遵循以下规则：

The remaining storage pools will be rated by different strategies. LINSTOR has currently three strategies:

MaxFreeSpace: This strategy maps the rating 1:1 to the remaining free space of the storage pool. However, this strategy only considers the actually allocated space (in case of thinly provisioned storage pool this might grow with time without creating new resources) MinReservedSpace: Unlink the “MaxFreeSpace”, this strategy considers the reserved spaced. That is the space that a thin volume can grow to before reaching its limit. The sum of reserved spaces might exceed the storage pools capacity, which is as overprovisioning. MinRscCount: Simply the count of resources already deployed in a given storage pool MaxThroughput: For this strategy, the storage pool’s Autoplacer/MaxThroughput property is the base of the score, or 0 if the property is not present. Every Volume deployed in the given storage pool will subtract its defined sys/fs/blkio_throttle_read and sys/fs/blkio_throttle_write property- value from the storage pool’s max throughput. The resulting score might be negative.

The scores of the strategies will be normalized, weighted and summed up, where the scores of minimizing strategies will be converted first to allow an overall maximization of the resulting score.

The weights of the strategies can be configured with

whereas the strategy-names are listed above and the weight can be an arbitrary decimal.

Finally LINSTOR tries to find the best matching group of storage pools meeting all requirements. This step also considers other autoplacement restrictions as --replicas-on-same, --replicas-on-different and others.

These two arguments, --replicas-on-same and --replicas-on-different expect the name of a property within the Aux/ namespace. The following example shows that the client automatically prefixes the testProperty with the Aux/ namespace.

现在我们应该可以加载我们资源的块设备并开始使用LINSTOR。

NVMe-oF/NVMe-TCP允许LINSTOR将无盘资源连接到具有相同资源的节点，数据存储在NVMe结构上。这就带来了这样一个优势：通过网络访问数据，无需使用本地存储就可以装载资源。在这种情况下，LINSTOR不使用DRBD，因此不会复制LINSTOR提供的NVMe资源，数据存储在一个节点上。

要将NVMe-oF/NVMe-TCP与LINSTOR一起使用，需要在充当Satellite的每个节点上安装包 nvme-cli，并将NVMe-oF/NVMe-TCP用于资源：

要使资源使用NVMe-oF/NVMe-TCP，必须在创建资源定义时提供一个附加参数：

为我们的资源创建卷定义：

在节点上创建资源之前，必须知道数据将在本地存储在哪里，以及哪个节点通过网络访问它。

首先，我们在存储数据的节点上创建资源：

在将通过网络访问资源数据的节点上，必须将资源定义为无盘：

-d 参数将此节点上的资源创建为无盘。

现在，您可以在一个节点上装载资源 nvmedata。

Since version 1.5.0 the additional Layer openflex can be used in LINSTOR. From LINSTOR’s perspective, the OpenFlex Composable Infrastructure takes the role of a combined layer acting as a storage layer (like LVM) and also providing the allocated space as an NVMe target. OpenFlex has a REST API which is also used by LINSTOR to operate with.

As OpenFlex combines concepts of LINSTORs storage as well as NVMe-layer, LINSTOR was added both, a new storage driver for the storage pools as well as a dedicated openflex layer which uses the mentioned REST API.

In order for LINSTOR to communicate with the OpenFlex-API, LINSTOR needs some additional properties, which can be set once on controller level to take LINSTOR-cluster wide effect:

Once that is configured, we can now create LINSTOR objects to represent the OpenFlex architecture. The theoretical mapping of LINSTOR objects to OpenFlex objects are as follows: Obviously an OpenFlex storage pool is represented by a LINSTOR storage pool. As the next thing above a LINSTOR storage pool is already the node, a LINSTOR node represents an OpenFlex storage device. The OpenFlex objects above storage device are not mapped by LINSTOR.

When using NVMe, LINSTOR was designed to run on both sides, the NVMe target as well as on the NVMe initiator side. In the case of OpenFlex, LINSTOR cannot (or even should not) run on the NVMe target side as that is completely managed by OpenFlex. As LINSTOR still needs nodes and storage pools to represent the OpenFlex counterparts, the LINSTOR client was extended with special node create commands since 1.0.14. These commands not only accept additionally needed configuration data, but also starts a “special satellite” besides the already running controller instance. This special satellites are completely LINSTOR managed, they will shutdown when the controller shuts down and will be started again when the controller starts. The new client command for creating a “special satellite” representing an OpenFlex storage device is:

The arguments are as follows:

The last step of the configuration is the creation of LINSTOR storage pools:

Once all necessary storage pools are created in LINSTOR, the next steps are similar to the usage of using an NVMe resource with LINSTOR. Here is a complete example:

一个 DM writecache 设备由两个设备、即一个存储设备和一个缓存设备组成。LINSTOR可以设置这样一个writecache设备，但是需要一些额外的信息，比如存储池和缓存设备的大小。

The two properties set in the examples are mandatory, but can also be set on controller level which would act as a default for all resources with WRITECACHE in their --layer-list. However, please note that the Writecache/PoolName refers to the corresponding node. If the node does not have a storage-pool named pmempool you will get an error message.

在 DM writecache 中所需的4个必需参数要么通过属性配置，要么由LINSTOR计算。上述链接中列出的可选属性也可以通过属性进行设置。有关 Writecache/* 属性键的列表，请参见 linstor controller set property—​help 。

使用 --layer-list DRBD,WRITECACHE,STORAGE 将DRBD配置为使用外部元数据时，只有备份设备将使用writecache，而不是保存外部元数据的设备。

LINSTOR can also setup a DM-Cache device, which is very similar to the DM-Writecache from the previous section. The major difference is that a cache device is composed by three devices: one storage device, one cache device and one meta device. The LINSTOR properties are quite similar to those of the writecache but are located in the Cache namespace:

Please see linstor controller set-property --help for a list of Cache/* property-keys and default values for omitted properties.

Using --layer-list DRBD,CACHE,STORAGE while having DRBD configured to use external metadata, only the backing device will use a cache, not the device holding the external metadata.

For some storage providers LINSTOR has special properties:

在 Linstor controller上禁用用户安全性是一次操作，之后会被持久化。

下面是有关命令的详细信息。

在LINSTOR可以加密任何卷之前，需要创建主密码。这可以通过linstor客户端完成。

`crypt create-passphrase`将等待用户输入初始主密码（因为所有其他crypt命令都没有参数）。

如果您想更改主密码，可以使用以下方法：

luks 层可以在创建资源定义或资源本身时添加，而建议使用前一种方法，因为它将自动应用于从该资源定义创建的所有资源。

要输入主密码（在controller重新启动后），请使用以下命令：

It is possible to automate the process of creating and re-entering the master passphrase.

To use this, either an environment variable called MASTER_PASSPHRASE or an entry in /etc/linstor/linstor.toml containing the master passphrase has to be created.

The required linstor.toml looks like this:

If either one of these is set, then every time the controller starts it will check whether a master passphrase already exists. If there is none, it will create a new master passphrase as specified. Otherwise, the controller enters the passphrase.

假设在某些节点上放置了名为 resource1 的资源定义，则可以按如下方式创建快照：

这将在资源所在的所有节点上创建快照。LINSTOR将确保即使在资源处于活动使用状态时也创建一致的快照。

Setting the resource-definition property AutoSnapshot/RunEvery LINSTOR will automatically create snapshots every X minute. The optional property AutoSnapshot/Keep can be used to clean-up old snapshots which were created automatically. No manually created snapshot will be cleaned-up / deleted. If AutoSnapshot/Keep is omitted (or ⇐ 0), LINSTOR will keep the last 10 snapshots by default.

以下步骤将快照还原到新资源。即使原始资源已从创建快照的节点中删除，也可能发生这种情况。

首先使用与快照中的卷匹配的卷定义新资源：

此时，如果需要，可以应用其他配置。然后，准备好后，根据快照创建资源：

这将在快照所在的所有节点上放置新资源。也可以显式选择放置资源的节点；请参阅帮助（ linstor snapshot resource restore -h ).

LINSTOR可以将资源回滚到快照状态。回滚时资源不能被使用。也就是说，它不能安装在任何节点上。如果资源正在使用中，请考虑是否可以通过restoring the snapshot来实现您的目标。

回滚执行如下：

资源只能回滚到最近的快照。要回滚到旧快照，请首先删除中间快照。

可以按如下方式删除现有快照：

Both, the source as well as the target node have to have the resource for snapshot shipping deployed. Additionally, the target resource has to be deactivated.

To manually start the snapshot-shipping, use:

By default, the snapshot-shipping uses tcp ports from the range 12000-12999. To change this range, the property SnapshotShipping/TcpPortRange, which accepts a to-from range, can be set on the controller:

A resource can also be periodically shipped. To accomplish this, it is mandatory to set the properties SnapshotShipping/TargetNode as well as SnapshotShipping/RunEvery on the resource-definition. SnapshotShipping/SourceNode can also be set, but if omitted LINSTOR will choose an active resource of the same resource-definition.

To allow incremental snapshot-shipping, LINSTOR has to keep at least the last shipped snapshot on the target node. The property SnapshotShipping/Keep can be used to specify how many snapshots LINSTOR should keep. If the property is not set (or ⇐ 0) LINSTOR will keep the last 10 shipped snapshots by default.

为了在节点之间移动资源而不减少任何点的冗余，可以使用LINSTOR的磁盘迁移功能。首先在目标节点上创建一个无盘资源，然后使用 --migrate from 选项添加一个磁盘。这将等到数据已同步到新磁盘，然后删除源磁盘。

例如，要将资源 backups 从 alpha 迁移到 bravo ：

LINSTOR can also be configured to automatically enable the above mentioned Proxy connection between two nodes. For this automation, LINSTOR first needs to know on which site each node is.

As the Site property might also be used for other site-based decisions in future features, the DrbdProxy/AutoEnable also has to be set to true:

This property can also be set on node, resource-definition, resource and resource-connection level (from left to right in increasing priority, whereas the controller is the left-most, i.e. least prioritized level)

Once this initialization steps are completed, every newly created resource will automatically check if it has to enable DRBD proxy to any of its peer-resources.

Postgresql linstor.toml 示例如下：

MariaDB linstor.toml 示例如下：

ETCD是一个分布式键值存储，它使得在HA设置中保持LINSTOR数据库的分布式变得容易。ETCD驱动程序已经包含在LINSTOR控制器包中，只需要在 LINSTOR.toml 中进行配置。

有关如何安装和配置ETCD的更多信息，请访问：https://etcd.io/docs[ETCD docs]

下面是 linstor.toml 中的[db]部分示例：

HTTP REST-API也可以通过HTTPS运行，如果您使用任何需要授权的功能，则强烈建议您使用它。所以你必须用一个有效的证书创建一个java密钥库文件，这个证书将用于加密所有的HTTPS通信。

下面是一个简单的示例，说明如何使用java运行时中包含的 keytool 创建自签名证书：

keytool 将要求密码来保护生成的密钥库文件，这是LINSTOR controller配置所需的。在 linstor.toml 文件中，必须添加以下部分：

现在（重新）启动 linstor controller ，端口3371上应该可以使用HTTPS REST-API。

有关如何导入其他证书的更多信息，请访问：https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html

The LINSTOR-Controller also provides a /health HTTP path that will simply return HTTP-Status 200 if the controller can access its database and all services are up and running. Otherwise it will return HTTP error status code 500 Internal Server Error.

LINSTOR在资源上自动配置仲裁策略 当仲裁可实现时 。这意味着，每当您有至少两个磁盘和一个或多个无磁盘资源分配，或三个或多个磁盘资源分配时，LINSTOR将自动为您的资源启用仲裁策略。

相反，当达到仲裁所需的资源分配少于最小值时，LINSTOR将自动禁用仲裁策略。

这是通过 DrbdOptions/auto-quorum 属性控制的，该属性可应用于 linstor-controller 、 resource-group 和 resource-definition 。 DrbdOptions/auto-quorum, 属性的接受值为 disabled 、 suspend io 和 io error 。

将 DrbdOptions/auto-quorum 属性设置为 disabled 将允许您手动或更细粒度地控制资源的仲裁策略（如果您愿意）。

例如，要手动设置名为 my_ssd_group 的 resource-group 的仲裁策略，可以使用以下命令：

您可能希望完全禁用DRBD的仲裁功能。要做到这一点，首先需要在适当的LINSTOR对象上禁用’DrbdOptions/auto-quorum’，然后相应地设置DRBD quorum特性。例如，使用以下命令完全禁用 my_ssd_group 的 resource-group 上的仲裁:

If a satellite is offline for a prolonged period of time, LINSTOR can be configured to declare that node as evicted. This triggers an automated reassignment of the affected DRBD-resources to other nodes to ensure a minimum replica count is kept.

This feature uses the following properties to adapt the behaviour.

DrbdOptions/AutoEvictMinReplicaCount sets the number of replicas that should always be present. You can set this property on the controller to change a global default, or on a specific resource-definition or resource-group to change it only for that resource-definiton or resource-group. If this property is left empty, the place-count set for the auto-placer of the corresponding resource-group will be used.

DrbdOptions/AutoEvictAfterTime describes how long a node can be offline in minutes before the eviction is triggered. You can set this property on the controller to change a global default, or on a single node to give it a different behavior. The default value for this property is 60 minutes.

DrbdOptions/AutoEvictMaxDisconnectedNodes sets the percentage of nodes that can be not reachable (for whatever reason) at the same time. If more than the given percent of nodes are offline at the same time, the auto-evict will not be triggered for any node , since in this case LINSTOR assumes connection problems from the controller. This property can only be set for the controller, and only accepts a value between 0 and 100. The default value is 34. If you wish to turn the auto-evict-feature off, simply set this property to 0. If you want to always trigger the auto-evict, regardless of how many satellites are unreachable, set it to 100.

DrbdOptions/AutoEvictAllowEviction is an additional property that can stop a node from being evicted. This can be useful for various cases, for example if you need to shut down a node for maintenance. You can set this property on the controller to change a global default, or on a single node to give it a different behavior. It accepts true and false as values and per default is set to true on the controller. You can use this property to turn the auto-evict feature off by setting it to false on the controller, although this might not work completely if you already set different values for individual nodes, since those values take precedence over the global default.

Afer the linstor-controller loses the connection to a satellite, aside from trying to reconnect, it starts a timer for that satellite. As soon as that timer exceeds DrbdOptions/AutoEvictAfterTime and all of the DRBD-connections to the DRBD-resources on that satellite are broken, the controller will check whether or not DrbdOptions/AutoEvictMaxDisconnectedNodes has been met. If it hasn’t, and DrbdOptions/AutoEvictAllowEviction is true for the node in question, the satellite will be marked as EVICTED. At the same time, the controller will check for every DRBD-resource whether the number of resources is still above DrbdOptions/AutoEvictMinReplicaCount. If it is, the resource in question will be marked as DELETED. If it isn’t, an auto-place with the settings from the corresponding resource-group will be started. Should the auto-place fail, the controller will try again later when changes that might allow a different result, such as adding a new node, have happened. Resources where an auto-place is necessary will only be marked as DELETED if the corresponding auto-place was successful.

The evicted satellite itself will not be able to reestablish connection with the controller. Even if the node is up and running, a manual reconnect will fail. It is also not possible to delete the satellite, even if it is working as it should be. Should you wish to get rid of an evicted node, you need to use the node lost command. The satellite can, however, be restored. This will remove the EVICTED-flag from the satellite and allow you to use it again. Previously configured network interfaces, storage pools, properties and similar entities as well as non-DRBD-related resources and resources that could not be autoplaced somewhere else will still be on the satellite. To restore a satellite, use

LINSTOR能够设置以下Sysfs设置：

If a LINSTOR volume is composed of multiple “stacked” volume (for example DRBD with external metadata will have 3 devices: backing (storage) device, metadata device and the resulting DRBD device), setting a sys/fs/\* property for a Volume, only the bottom-most local “data”-device will receive the corresponding /sys/fs/cgroup/…​ setting. That means, in case of the example above only the backing device will receive the setting. In case a resource-definition has an nvme-target as well as an nvme-initiator resource, both bottom-most devices of each node will receive the setting. In case of the target the bottom-most device will be the volume of LVM or ZFS, whereas in case of the initiator the bottom-most device will be the connected nvme-device, regardless which other layers are stacked ontop of that.

在命令行中列出可用命令的一种快速方法是键入 linstor 。

有关子命令（例如列表节点）的更多信息可以通过两种方式检索：

当LINSTOR以交互模式（LINSTOR interactive）执行时，使用 help 子命令尤其有用。

LINSTOR最有用的特性之一是它丰富的tab-completion，它基本上可以用来完成LINSTOR所知道的每个对象（例如，节点名、IP地址、资源名等等）。在下面的示例中，我们将展示一些可能的完成及其结果：

如果制表符完成不正常，请尝试获取相应的文件：

对于zsh shell用户，linstor客户机可以生成一个zsh编译文件，该文件基本上支持命令和参数完成。

If something goes wrong and you need help finding the cause of the issue, you can use

The command above will create a new sos-report in /var/log/linstor/controller/ on the controller node. Alternatively you can use

which will create a new sos-report and additionally downloads that report to the local machine into your current working directory.

This sos-report contains logs and useful debug-information from several sources (Linstor-logs, dmesg, versions of external tools used by Linstor, ip a, database dump and many more). These information are stored for each node in plaintext in the resulting .tar.gz file.

如需社区帮助，请订阅我们的邮件列表： https://lists.linbit.com/listinfo/drbd-user

要提交bug或功能请求，请查看我们的GitHub页面 https://GitHub.com/linbit

或者，如果您希望购买远程安装服务、24/7支持、访问认证存储库或功能开发，请联系我们：+1-877-454-6248（1-877-4LINBIT），国际电话：+43-1-8178292-0 | sales@linbit.com

LINBIT为商业支持客户提供LINSTOR Operator。Operator通过安装DRBD、管理Satellite/Controller Pods以及其他相关功能，简化了LINSTOR在Kubernetes上的部署。

Operator用Helm v3安装，如下所示：

The LINSTOR operator can automate some basic storage set up for LINSTOR.

This section describes the different options for enabling security features available when using this operator. The following guides assume the operator is installed using Helm

The helm charts provide a set of further customization options for advanced use cases.

The operator can configure the satellites and CSI plugin to use an existing LINSTOR setup. This can be useful in cases where the storage infrastructure is separate from the Kubernetes cluster. Volumes can be provisioned in diskless mode on the Kubernetes nodes while the storage nodes will provide the backing disk storage.

To skip the creation of a LINSTOR Controller deployment and configure the other components to use your existing LINSTOR Controller, use the following options when running helm install:

After all pods are ready, you should see the Kubernetes cluster nodes as satellites in your LINSTOR setup.

Create a storage class referencing an existing storage pool on your storage nodes.

You can provision new volumes by creating PVCs using your storage class. The volumes will first be placed only on nodes with the given storage pool, i.e. your storage infrastructure. Once you want to use the volume in a pod, LINSTOR CSI will create a diskless resource on the Kubernetes node and attach over the network to the diskfull resource.

在Kubernetes中由社区支持的LINSTOR部署版本称为Piraeus。Piraeus项目提供了一个 an operator 进行部署。

The following storage class contains all currently available parameters to configure the provisioned storage

Sets the file system type to create for volumeMode: FileSystem PVCs. Currently supported are:

autoPlace is an integer that determines the amount of replicas a volume of this StorageClass will have. For instance, autoPlace: "3" will produce volumes with three-way replication. If neither autoPlace nor nodeList are set, volumes will be automatically placed on one node.

placementCount is an alias for autoPlace

The LINSTOR Resource Group (RG) to associate with this StorageClass. If not set, a new RG will be created for each new PVC.

storagePool 是用于为新创建的卷提供存储的LINSTOR storage pool的名称。

disklessStoragePool 是一个可选参数，它只影响作为客户端无磁盘分配给 kubelets 的LINSTOR卷。如果在LINSTOR中定义了自定义 diskless storage pool，请在此处指定。

A comma-seperated list of layers to use for the created volumes. The available layers and their order are described towards the end of this section. Defaults to drbd,storage

Select from one of the available volume schedulers:

Disable remote access to volumes. This implies that volumes can only be accessed from the initial set of nodes selected on creation. CSI Topology processing is required to place pods on the correct nodes.

encryption is an optional parameter that determines whether to encrypt volumes. LINSTOR must be configured for encryption for this to work properly.

nodeList 是要分配给卷的节点列表。这将把卷分配给每个节点，并在所有节点之间进行复制。这也可以用于按主机名选择单个节点，但使用replicasOnSame选择单个节点更灵活。

clientList is a list of nodes for diskless volumes to be assigned to. Use in conjunction with 节点列表.

replicasOnSame is a list of key or key=value items used as autoplacement selection labels when autoplace is used to determine where to provision storage. These labels correspond to LINSTOR node properties.

Let’s explore this behavior with examples assuming a LINSTOR cluster such that node-a is configured with the following auxiliary property zone=z1 and role=backups, while node-b is configured with only zone=z1.

如果我们使用 autoPlace: "1" ` 和 `replicasOnSame: "zone=z1 role=backups"`来配置一个 StorageClass ，那么从该 StorageClass 创建的所有卷都将配置在 node-a 上，因为这是LINSTOR集群中唯一具有所有正确的key=value对的节点。这是选择单个节点进行资源调配的最灵活方式。

如果我们使用 autoPlace: "1" ` 和 `replicasOnSame: "zone=z1" ` 来配置一个 StorageClass ，那么卷将在 `node-a 或 node-b 上配置，因为它们都有 zone=z1 辅助属性。

If we configure a StorageClass with autoPlace: "2" and replicasOnSame: "zone=z1 role=backups", then provisioning will fail, as there are not two or more nodes that have the appropriate auxiliary properties.

如果我们用 autoPlace: "2" 和 replicasOnSame: "zone=z1" 配置一个存储类，那么卷将同时在 node-a 和 node-b 上配置，因为它们都有 zone=z1 辅助属性。

You can also use a property key without providing a value to ensure all replicas are placed on nodes with the same property value, with caring about the particular value. Assuming there are 4 nodes, node-a1 and node-a2 are configured with zone=a. node-b1 and node-b2 are configured with zone=b. Using autoPlace: "2" and replicasOnSame: "zone" will place on either node-a1 and node-a2 OR on node-b1 and node-b2.

replicasOnDifferent takes a list of properties to consider, same as replicasOnSame. There are two modes of using replicasOnDifferent:

Create a diskless resource on all nodes that were not assigned a diskful resource.

Do not place the resource on a node which has a resource with a name matching the regex.

fsOpts 是一个可选参数，在创建时将选项传递给卷的文件系统。

mountOpts 是一个可选参数，在装载时将选项传递给卷的文件系统。

Extra arguments to pass to xfs_io, which gets called before right before first use of the volume.

Advanced DRBD options to pass to LINSTOR. For example, to change the replication protocol, use DrbdOptions/Net/protocol: "A".

The full list of options is available here

LINSTOR supports the volume snapshot feature, which is currently in beta. To use it, you need to install a cluster wide snapshot controller. This is done either by the cluster provider, or you can use the LINSTOR chart.

By default, the LINSTOR chart will install its own snapshot controller. This can lead to conflict in some cases:

In such a case, installation of the snapshot controller can be disabled:

Then we can create our VolumeSnapshotClass:

使用 kubectl 创建 VolumeSnapshotClass ：

现在，我们将为上面创建的卷创建卷快照。这是用 VolumeSnapshot 完成的：

使用 kubectl 创建 VolumeSnapshot ：

You can check that the snapshot creation was successful

最后，我们将使用 PVC 从快照创建一个新卷。

用 kubectl 创建 PVC ：

By default, the operator will install the components required for Stork, and register a new scheduler called stork with Kubernetes. This new scheduler can be used to place pods near to their volumes.

Deployment of the scheduler can be disabled using

The following StatefulSet uses the HA Controller to manage fail over of a pod.

Deploy the set and wait for the pod to start

Then one of the nodes becomes unreachable. Shortly after, Kubernetes will mark the node as NotReady

After about 45 seconds, the Pod will be removed by the HA Controller and re-created by the StatefulSet

Some versions require special steps, see below.

To create a backup of the Etcd database and store it on your control host, run:

These commands will create a file save.db on the machine you are running kubectl from.

LINBIT provides a certified LINSTOR operator via the RedHat marketplace. The operator eases deployment of LINSTOR on Kubernetes by installing DRBD, managing Satellite and Controller pods, and other related functions.

The operator itself is available from the Red Hat Marketplace.

Unlike deployment via the helm chart, the certified Openshift operator does not deploy the needed etcd cluster. You must deploy this yourself ahead of time. We do this via the etcd operator available on operatorhub.io.

Read the storage guide and configure a basic storage setup for LINSTOR.

Read the section on securing the deployment and configure as needed.

Once etcd and storage has been configured, we are now ready to install the LINSTOR operator. You can find the LINSTOR operator via the left-hand control pane of Openshift Web Console. Expand the “Operators” section and select “OperatorHub”. From here you need to find the LINSTOR operator. Either search for the term “LINSTOR” or filter only by “Marketplace” operators.

Once you have located the LINSTOR operator in the Marketplace, click the “Install” button and install it as you would any other operator.

At this point you should have just one pod, the operator pod, running.

Next we needs to configure the remaining provided APIs.

Again, navigate to the left-hand control pane of the Openshift Web Console. Expand the “Operators” section, but this time select “Installed Operators”. Find the entry for the “Linstor Operator”, then select the “LinstorController” from the “Provided APIs” column on the right.

From here you should see a page that says “No Operands Found” and will feature a large button on the right which says “Create LinstorController”. Click the “Create LinstorController” button.

Here you will be presented with options to configure the LINSTOR Controller. Either via the web-form view or the YAML View. Regardless of which view you select, make sure that the dbConnectionURL matches the endpoint provided from your etcd deployment. Otherwise, the defaults are usually fine for most purposes.

Lastly hit “Create”, you should now see a linstor-controller pod running.

Next we need to deploy the Satellites Set. Just as before navigate to the left-hand control pane of the Openshift Web Console. Expand the “Operators” section, but this time select “Installed Operators”. Find the entry for the “Linstor Operator”, then select the “LinstorSatelliteSet” from the “Provided APIs” column on the right.

From here you should see a page that says “No Operands Found” and will feature a large button on the right which says “Create LinstorSatelliteSet”. Click the “Create LinstorSatelliteSet” button.

Here you will be presented with the options to configure the LINSTOR Satellites. Either via the web-form view or the YAML View. One of the first options you’ll notice is the automaticStorageType. If set to “NONE” then you’ll need to remember to configure the storage pools yourself at a later step.

Another option you’ll notice is kernelModuleInjectionMode. I usually select “Compile” for portability sake, but selecting “ShippedModules” will be faster as it will install pre-compiled kernel modules on all the worker nodes.

Make sure the controllerEndpoint matches what is available in the kubernetes endpoints. The default is usually correct here.

Below is an example manifest:

Lastly hit “Create”, you should now see a linstor-node pod running on every worker node.

Last bit left is the CSI pods to bridge the layer between the CSI and LINSTOR. Just as before navigate to the left-hand control pane of the Openshift Web Console. Expand the “Operators” section, but this time select “Installed Operators”. Find the entry for the “Linstor Operator”, then select the “LinstorCSIDriver” from the “Provided APIs” column on the right.

From here you should see a page that says “No Operands Found” and will feature a large button on the right which says “Create LinstorCSIDriver”. Click the “Create LinstorCSIDriver” button.

Again, you will be presented with the options. Make sure that the controllerEnpoint is correct. Otherwise the defaults are fine for most use cases.

Lastly hit “Create”. You will now see a single “linstor-csi-controller” pod, as well as a “linstor-csi-node” pod on all worker nodes.

To deploy STORK, you can use the single YAML deployment available at: https://charts.linstor.io/deploy/stork.yaml Download the YAML and replace every instance of MY-STORK-NAMESPACE with your desired namespace for STORK. You also need to replace MY-LINSTOR-URL with the URL of your controller. This value depends on the name you chose when creating the LinstorController resource. By default this would be http://linstor.<operator-namespace>.svc:3370

To apply the YAML to Openshift, either use oc apply -f <filename> from the command line or find the “Import YAML” option in the top right of the Openshift Web Console.

To deploy our High Availability Controller, you can use the single YAML deployment available at: https://charts.linstor.io/deploy/ha-controller.yaml

Download the YAML and replace:

To apply the YAML to Openshift, either use oc apply -f <filename> from the command line or find the “Import YAML” option in the top right of the Openshift Web Console.

Alternatively, you can deploy the LINSTOR Operator using Helm instead. Take a look at the Kubernetes guide. Openshift requires changing some of the default values in our Helm chart.

If you chose to use Etcd with hostpath volumes for persistence (see here), you need to enable selinux relabelling. To do this pass --set selinux=true to the pv-hostpath install command.

For the LINSTOR Operator chart itself, you should change the following values:

Other overrides, such as storage pool configuration, HA deployments and more, are available and documented in the Kubernetes guide.

Version 5 of the plugin drops compatibility with the legacy configuration options “storagepool” and “redundancy”. Version 5 requires a “resourcegroup” option, and obviously a LINSTOR resource group. The old options should be removed from the config.

Configuring LINSTOR is described in Section LINSTOR配置, a typical example follows: Let’s assume the pool was set to “mypool”, and redundancy to 3.

修改 /etc/one/oned.conf 的以下部分`

将linstor添加到 TM MAD 和 DATASTORE MAD 部分的驱动程序列表中：

添加新的TM_MAD_CONF和DS_MAD_CONF部分：

完成这些更改后，重新启动opennebula服务。

前端节点通过Linstor向存储和主机节点发出命令

存储节点在本地保存vm的磁盘映像。

主机节点负责运行实例化的vm，通常通过Linstor无盘模式在网络上存储它们需要连接的映像。

所有节点都必须安装DRBD9和Linstor。这个过程在 DRBD9用户指南 中有详细说明

只要前端和主机节点满足这两个角色的所有要求，它们就可以作为其主要角色之外的存储节点。

oneadmin用户必须具有对存储节点上的 mkfs 命令的无密码sudo访问权限

创建一个名为ds.conf的数据存储配置文件，并使用 onedatastore 工具基于该配置创建一个新的数据存储。有两个相互排斥的部署选项：LINSTOR_AUTO_PLACE和LINSTOR_deployment_NODES。如果两者都已配置，则忽略LINSTOR_AUTO_PLACE。对于这两个选项，BRIDGE_LIST必须是Linstor集群中所有存储节点的空间分隔列表。

因为1.0.0版LINSTOR支持资源组。资源组是所有链接到该资源组的资源共享的设置的集中点。

Create a resource group and volume group for your datastore, it is mandatory to specify a storage-pool within the resource group, otherwise monitoring space for opennebula will not work. Here we create one with 2 node redundancy and use a created opennebula-storagepool:

现在使用LINSTOR插件添加OpenNebula数据存储：

以下属性已弃用，并将在1.0.0版之后的版本中删除。

Linstor驱动程序也可以用作系统数据存储，配置与普通数据存储非常相似，但有一些更改：

还要将新的sys datastore id添加到镜像数据存储的 COMPATIBLE_SYS_DS （以逗号分隔），否则调度程序将忽略它们。

如果要使用易失性磁盘进行实时迁移，则需要为KVM启用 --unsafe 选项，请参见： opennebula doc

linstor驱动程序将从OpenStack Stein版本开始正式提供。最新版本位于 LINBIT OpenStack Repo. 它是一个名为 linstordrv.py 的Python文件。根据OpenStack的安装，其目标可能会有所不同。

将驱动程序（linstordrv.py）放置在OpenStack Cinder节点中的适当位置。

对于Devstack：

对于Ubuntu：

对于RDO Packstack：

为OpenStack命令行操作配置环境变量后，从Cinder节点运行这些命令。

对于Devstack：

对于RDO Packstack：

对于完整的OpenStack：

一旦Cinder服务重新启动，可以使用Horizon图形用户界面或命令行创建带有LINSTOR后端的新Cinder卷。使用以下命令行创建卷时，请参考。

待补充

导出Cinder卷的默认方法是通过iSCSI。这带来了最大兼容性的优势 – 无论是VMWare、Xen、HyperV还是KVM，iSCSI都可以用于每个虚拟机监控程序。

缺点是，所有数据都必须发送到Cinder节点，由（userspace）iSCSI守护进程处理；这意味着数据需要通过内核/userspace边界，这些转换将消耗一些性能。

另一种方法是使用DRBD作为传输协议将数据传输到VMs。这意味着DRBD 9 ^[2]也需要安装在Cinder节点上。

该解决方案的一个优点是，vm的存储访问请求可以通过DRBD内核模块发送到存储节点，然后存储节点可以直接访问分配的lv；这意味着数据路径上没有内核/用户空间转换，因此性能更好。结合支持RDMA的硬件，您应该获得与VMs直接访问FC后端相同的性能。

另一个优点是，您将从DRBD的HA背景中隐式获益：使用多个存储节点（可能在不同的网络连接上可用）意味着冗余并避免单点故障。

在 cinder.conf 的LINSTOR部分中，可以定义要使用的传输协议。本章开头描述的初始设置被设置为使用DRBD传输。您可以根据需要进行如下配置。那么Horizon^[3]应该在卷创建时提供这些存储后端。

由于兼容性原因，旧类名 DrbdManageDriver 暂时保留；它只是iSCSI驱动程序的别名。

总结一下：

在节点alpha上：

在bravo节点上：