LINSTOR ユーザーズガイド

2024-03-14 07:38:01 UTC

Please Read This First

このガイドは、software-defined storage (SDS) ソリューションである LINSTOR® のリファレンスガイドおよびハンドブックとして使用することを目的としています。

このガイドは全体を通して、あなたがLINSTORと関連ツールの最新版を使っていると仮定します。

This guide is organized as follows:

  • An Introduction to LINSTOR is a foundational overview of LINSTOR and provides explanations for LINSTOR concepts and terms.

  • 基本管理タスクとシステム設定 ではLINSTORの基本的な機能を扱い、一般的な管理作業を使用するための方法を提供します。この章ではそれ以外に、LINSTORの基本設定を配備するためのステップバイステップのガイドとして使用できます。

  • LINSTOR 応用タスク shows a variety of advanced and important LINSTOR tasks and configurations, so that you can use LINSTOR in more complex ways.

  • Administering LINSTOR by GUI deals with the graphical client approach to managing LINSTOR clusters that is available to LINBIT® customers.

  • LINSTOR Integrations has chapters that deal with how to implement a LINSTOR based storage solution with various platforms and technologies, such as Kubernetes, Proxmox VE, OpenNebula, Docker, OpenStack, and others, by using the LINSTOR API.

Introduction to LINSTOR

1. An Introduction to LINSTOR

To use LINSTOR® effectively, this “Introduction to LINSTOR” chapter provides an overview of the software, explains how it works and deploys storage, and introduces and explains important concepts and terms to help your understanding.

1.1. An Overview of LINSTOR

LINSTOR is an open source configuration management system, developed by LINBIT® for storage on Linux systems. It manages LVM logical volumes, ZFS ZVOLs, or both, on a cluster of nodes. It uses DRBD® for replication between different nodes and to provide block storage devices to users and applications. Some of its features include snapshots, encryption, and caching of HDD backed data in SSDs.

1.1.1. Where LINSTOR is Used

LINSTOR was originally developed to manage DRBD resources. While you can still use LINSTOR to make managing DRBD more convenient, LINSTOR has evolved and it is often integrated with software stacks higher up to provide persistent storage more easily and more flexibly than would otherwise be possible within those stacks.

LINSTOR can be used on its own or you can integrate it with other platforms, such as Kubernetes, OpenShift, OpenNebula, OpenStack, Proxmox VE, and others. LINSTOR runs on bare-metal on-premise hardware, or you can use it within virtual machines (VMs), containers, clouds, or hybrid environments.

1.1.2. LINSTOR Supported Storage and Related Technologies

LINSTOR can work with the following storage provider and related technologies:

  • LVM and LVM thin volumes

  • ZFS and ZFS thin volumes

  • File and FileThin (loop devices)

  • Diskless

  • Exos [DEPRECATED]

  • SPDK (remote)

  • Microsoft Windows Storage Spaces and thin Storage Spaces

  • EBS (target and initiator)

  • Device mapper cache (dm-cache) and writecache (dm-writecache)

  • bcache

  • LUKS

  • DRBD

By using LINSTOR, you can work with these technologies on their own or else in various meaningful combinations.

1.2. How LINSTOR Works

A working LINSTOR setup requires one active controller node that runs the LINSTOR controller software as a systemd service, linstor-controller.service. This is the LINSTOR control plane, where the LINSTOR controller node communicates with LINSTOR satellite nodes.

The setup also requires one or more satellite nodes that run the LINSTOR satellite software as a systemd service, linstor-satellite.service. The LINSTOR satellite service facilitates storage and related actions on the node, for example creating storage volumes to provide data storage to users and applications. However, satellite nodes do not have to provide physical storage to the cluster. For example, you can have diskless satellite nodes that participate in the LINSTOR cluster for DRBD quorum purposes.

It is also possible for a node to run both the LINSTOR controller and satellite services and act in a Combined role.

You can think of the storage technologies as implemented on LINSTOR satellite nodes, for example, DRBD replication, as the data plane. With LINSTOR, the control and data planes are separate and can function independently. This means, for example, that you can update the LINSTOR controller node or the LINSTOR controller software while your LINSTOR satellite nodes continue to provide (and replicate if using DRBD) storage to users and applications without interruption.

For convenience, a LINSTOR setup is often called a LINSTOR cluster in this guide, even though a valid LINSTOR setup can exist as an integration within another platform, such as Kubernetes.

Users can interact with LINSTOR by using either a CLI-based client or a graphical user interface (GUI). Both of these interfaces make use of the LINSTOR REST API. LINSTOR can integrate with other platforms and applications by using plugins or drivers that also make use of this API.

Communication between the LINSTOR controller and the REST API happens via TCP/IP and can be secured by using SSL/TLS.

The southbound drivers that LINSTOR uses to interface with physical storage are LVM, thinLVM and ZFS.

diagram of how LINSTOR works
Figure 1. How LINSTOR works

1.3. インストール可能コンポーネント

A LINSTOR setup has three installable components:

  • LINSTOR controller

  • LINSTOR satellite

  • LINSTOR user interfaces (LINSTOR client and LINBIT SDS GUI)

These installable components are either source code that you can compile, or else prebuilt packages, that you can use to install and run the software.

1.3.1. LINSTOR コントローラー

The linstor-controller service relies on a database that holds all the configuration information for the whole cluster. A node or container running the LINSTOR controller software is responsible for resource placement, resource configuration, and orchestration of any operational processes that require a view of the whole cluster.

Multiple controllers can be used for LINSTOR, for example, when setting up a highly available LINSTOR cluster, but only one controller can be active.

As mentioned earlier, the LINSTOR controller operates on a separate plane from the data plane that it manages. You can stop the controller service, update or reboot the controller node, and still have access to your data hosted on the LINSTOR satellite nodes. While you can still access and serve the data on your LINSTOR satellite nodes, without a running controller node, you will not be able to perform any LINSTOR status or management tasks on the satellite nodes.

1.3.2. LINSTOR サテライト

The linstor-satellite service runs on each node where LINSTOR consumes local storage or provides storage to services. It is stateless and receives all the information it needs from the node or container running the LINSTOR controller service. The LINSTOR satellite service runs programs like lvcreate and drbdadm. It acts like an agent on a node or in a container that carries out instructions that it receives from the LINSTOR controller node or container.

1.3.3. LINSTOR User Interfaces

When you need to interface with LINSTOR, you can send instructions to the active LINSTOR controller by using one of its user interfaces (UIs): the LINSTOR client, or the LINBIT SDS GUI.

Both of these UIs rely on the LINSTOR REST API.

LINSTOR Client

The LINSTOR client, linstor, is a command line utility that you can use to issue commands to the active LINSTOR controller node. These commands can be action-oriented, such as commands that create or modify storage resources in your cluster, or they can be status commands to glean information about the current state of your LINSTOR cluster.

You can use the LINSTOR client either by entering linstor followed by valid commands and arguments, or in the client’s interactive mode, by entering linstor on its own.

You can find more information about using the LINSTOR client in the LINSTORクライアントの使用 section in this user’s guide.

Listing 1. The LINSTOR client in interactive mode
# linstor
Use "help <command>" to get help for a specific command.

Available commands:
- advise (adv)
- backup (b)
- controller (c)
- drbd-proxy (proxy)
- encryption (e)
- error-reports (err)
- exos
- file (f)
- help
- interactive
- key-value-store (kv)
- list-commands (commands, list)
- node (n)
- node-connection (nc)
- physical-storage (ps)
- remote
- resource (r)
- resource-connection (rc)
- resource-definition (rd)
- resource-group (rg)
- schedule (sched)
- snapshot (s)
- sos-report (sos)
- space-reporting (spr)
- storage-pool (sp)
- volume (v)
- volume-definition (vd)
- volume-group (vg)
LINSTOR ==>
LINBIT SDS Graphical User Interface

The LINBIT SDS graphical user interface (GUI) is a web-based GUI that you can use to work with LINSTOR. It can be a convenient way to navigate and get overview information about a LINSTOR cluster, or add, modify, or delete LINSTOR objects within a cluster. For example, you can add nodes, add or delete resources, or do other tasks.

You can find more information about using the GUI interface in the LINBIT SDS GUI chapter in this user’s guide.

image of the LINBIT SDS GUI dashboard within a web browser
Figure 2. The LINBIT SDS GUI dashboard

1.4. Internal Components

The internal components of LINSTOR are abstractions of the software code that are used to describe how LINSTOR works and how you use it. Examples of internal components would be LINSTOR objects, such as resources or storage pools. Although these are abstractions, you will interact with them in a very real way as you use either the LINSTOR client or GUI to deploy and manage storage.

Along the way, this section also introduces and explains core concepts and terms that you will need to familiarize yourself with to understand how LINSTOR works and how to use it.

1.4.1. LINSTOR Objects

LINSTOR takes an object-oriented approach to software-defined storage (SDS). LINSTOR objects are the end result that LINSTOR presents to the user or application to consume or build upon.

The most commonly used LINSTOR objects are explained below and a full list of objects follows.

リソース

A resource is the LINSTOR object that represents consumable storage that is presented to applications and end users. If LINSTOR is a factory, then a resource is the finished product that it produces. Often, a resource is a DRBD replicated block device but it does not have to be. For example, a resource could be diskless to satisfy DRBD quorum requirements, or it could be an NVMe-oF or EBS initiator.

A resource has the following attributes:

  • The name of the node that the resource exists on

  • The resource definition that the resource belongs to

  • Configuration properties of the resource

ボリューム

A volume is the closest LINSTOR internal component to physical storage and is a subset of a resource. A resource can have multiple volumes. For example, you might want to have a database stored on slower storage than its transaction log in a MySQL cluster. To accomplish this by using LINSTOR, you could have one volume for the faster transaction log storage media and another for the slower database storage media, and have both under a single “MySQL” resource. By keeping multiple volumes under a single resource you are essentially creating a consistency group.

An attribute that you specify for a volume takes precedence over the same attribute if it is also specified “higher up” in the LINSTOR object hierarchy. This is because, again, a volume is the closest internal LINSTOR object to physical storage.

ノード

A Node is a server or container that participates in a LINSTOR cluster. The node object has the following attributes:

  • Name

  • IP address

  • TCP port

  • Node type (controller, satellite, combined, auxiliary)

  • Communication type (plain or SSL/TLS)

  • Network interface type

  • Network interface name

Storage Pool

A storage pool identifies storage that is assignable to other LINSTOR objects, such as LINSTOR resources, resource definitions, or resource groups, and can be consumed by LINSTOR volumes.

A storage pool defines:

  • The storage back-end driver to use for the storage pool on the cluster node, for example, LVM, thin-provisioned LVM, ZFS, and others

  • The node that the storage pool exists on

  • The storage pool name

  • Configuration properties of the storage pool

  • Additional parameters to pass to the storage pool’s back-end driver (LVM, ZFS, and others)

A List of LINSTOR Objects

LINSTOR has the following core objects:

EbsRemote

ResourceConnection

SnapshotVolumeDefinition

ExternalFile

ResourceDefinition

StorPool

KeyValueStore

ResourceGroup

StorPoolDefinition

LinstorRemote

S3Remote

Volume

NetInterface

Schedule

VolumeConnection

Node

Snapshot

VolumeDefinition

NodeConnection

SnapshotDefinition

VolumeGroup

1.4.2. Definition and Group Objects

While definitions and groups are also LINSTOR objects, they are a special kind. Definition and group objects can be thought of as profiles or templates. These template objects are used to create child objects that will inherit their parent object’s attributes. They might also have attributes that can affect child objects but are not attributes of the child objects themselves. These attributes could be things such as the TCP port to use for DRBD replication or the volume number that a DRBD resource should use.

Definitions

Definitions define the attributes of an object. Objects created from a definition will inherit the configuration attributes defined in the definition. A definition must be created before you can create an associated child object. For example, you must create a resource definition prior to creating the corresponding resource.

There are two LINSTOR definition objects that you can create directly, by using the LINSTOR client: resource definitions and volume definitions.

Resource definition

Resource definitions can define the following attributes of a resource:

  • The resource group that the resource definition belongs to

  • The name of a resource (implicitly, by virtue of the resource definition’s name)

  • The TCP port to use for the resource’s connection, for example, when DRBD is replicating data

  • Other attributes such as a resource’s storage layers, peer slots, and external name.

Volume definition

Volume definitions can define the following:

  • The size of the storage volume

  • The volume number of the storage volume (because a resource can have multiple volumes)

  • ボリュームのメタデータプロパティ

  • The minor number to use for the DRBD device, if the volume is associated DRBD replicated storage

In addition to these definitions, LINSTOR has some indirect definitions: the storage pool definition, the snapshot definition, and the snapshot volume definition. LINSTOR creates these automatically when you create the respective object.

Groups

Groups are similar to definitions in that they are like profiles or templates. Where definitions apply to LINSTOR object instances, groups apply to object definitions. As the name implies, a group can apply to multiple object definitions, just as a definition can apply to multiple object instances. For example, you can have a resource group that defines resource attributes for a frequently needed storage use case. You can then use the resource group to easily spawn (create) multiple resources that need to have those attributes, without having to specify the attributes every time you create a resource.

Resource group

A resource group is a parent object of a resource definition where all property changes made on a resource group will be inherited by its resource definition children[1]. The resource group also stores settings for automatic placement rules and can spawn a resource definition depending on the stored rules.

A resource group defines characteristics of its resource definition child objects. A resource spawned from the resource group, or created from a resource definition that belongs to the resource group, will be a “grandchild” object of a resource group and the “child” of a resource definition. Every resource definition that you create will be a member of the default LINSTOR resource group, DfltRscGrp, unless you specify another resource group when creating the resource definition.

Changes to a resource group will be applied to all resources or resource definitions that were created from the resource group, retroactively, unless the same characteristic has been set on a child object, for example, a resource definition or resource that was created from the resource group.

All of this makes using resource groups a powerful tool to efficiently manage a large number of storage resources. Rather than creating or modifying individual resources, you can simply configure a resource group parent, and all the child resource objects will receive the configuration.

Volume group

Similarly, volume groups are like profiles or templates for volume definitions. A volume group must always reference a specific resource group. In addition, a volume group can define a volume number, and a “gross” volume size.

1.5. LINSTOR Object Hierarchy

As alluded to in previous subsections of this chapter, there is a concept of hierarchy among LINSTOR objects. Depending on the context, this can be described either as a parent-child relationship, or else as a higher-lower relationship where lower means closer to the physical storage layer[2].

A child object will inherit attributes that are defined on its parent objects, if the same attributes are not already defined on the child object. Similarly, a lower object will receive attributes that are set on higher objects, if the same attributes are not already defined on the lower object.

1.5.1. General Rules for Object Hierarchy in LINSTOR

The following are some general rules for object hierarchy in LINSTOR:

  • A LINSTOR object can only receive or inherit attributes that can be set on that object.

  • Lower objects receive attributes set on higher objects.

  • An attribute set on a lower object takes precedence over the same attribute set on higher objects.

  • Child objects inherit attributes set on parent objects.

  • An attribute set on a child object takes precedence over the same attribute set on parent objects.

1.5.2. Using Diagrams to Show Relationships Between LINSTOR Objects

This section uses diagrams to represent the hierarchical relationships between some of the most frequently used LINSTOR objects. Because of the number of LINSTOR objects and their interconnectedness, multiple diagrams are shown first rather than a single diagram, to simplify the conceptualization.

Object hierarchy between controller
Figure 3. The hierarchical relationships between common LINSTOR objects

The next diagram shows the relationships between LINSTOR group objects on a single satellite node.

Object hierarchy between controller
Figure 4. The hierarchical relationships between common LINSTOR group objects on a controller node

While the two preceding diagrams show higher-lower relationships between common LINSTOR objects, you can also think of some LINSTOR objects as having parent-child relationships. The next diagram introduces this kind of relationship between LINSTOR objects by using a storage pool definition (parent object) and a storage pool (child object) as an example. A parent object can have multiple child objects, as shown in the following diagram.

Diagram 3
Figure 5. Higher-lower and parent-child relationships between LINSTOR objects

Having introduced the concept of parent-child relationships in a conceptual diagram, the next diagram is a modified version of the second diagram with some of those relationships added for groups and definitions. This modified diagram also incorporates some of the higher-lower relationships that were shown in the first diagram.

Diagram 4
Figure 6. The hierarchical relationships between LINSTOR group and definition objects

The next diagram synthesizes the relationship concepts of the preceding diagrams while also introducing new LINSTOR objects related to snapshots and connections. With the many objects and criss-crossing lines, the reason for building up to this diagram should be apparent.

diagram of LINSTOR hierarchy and inheritance relationships
Figure 7. The hierarchy and inheritance relationships of common LINSTOR objects

Even with its seeming complexity, the preceding diagram is still a simplification and not an all-encompassing representation of the possible relationships between LINSTOR objects. As listed earlier, there are more LINSTOR objects than are shown in the diagram[3].

The good news is that you do not need to memorize the preceding diagram to work with LINSTOR. It could be useful to refer to though if you are trying to troubleshoot attributes that you have set on LINSTOR objects and their inheritance and effects on other LINSTOR objects in your cluster.

1.6. Using LINSTOR

After introducing LINSTOR and explaining its basic concepts and functions, the next chapters in this guide are how-to oriented and task-based. They provide instructions for using LINSTOR and its various components to solve meaningful real-world data storage problems.

Administering LINSTOR

2. 基本管理タスクとシステム設定

This is a how-to style chapter that covers basic LINSTOR® administrative tasks, including installing LINSTOR and how to get started using LINSTOR.

2.1. Before Installing LINSTOR

Before you install LINSTOR, there are a few things that you should be aware of that might affect how you install LINSTOR.

2.1.1. パッケージ

LINSTOR は RPM と DEB 形式でパッケージングされています。

  1. linstor-client contains the command line client program. It depends on Python which is usually already installed. In RHEL 8 systems you will need to symlink python.

  2. linstor-controller and linstor-satellite Both contain systemd unit files for the services. They depend on Java runtime environment (JRE) version 1.8 (headless) or higher.

For further details about these packages see the Installable Components section above.

If you have a LINBIT® support subscription, you will have access to certified binaries through LINBIT customer-only repositories.

2.1.2. FIPS Compliance

This standard shall be used in designing and implementing cryptographic modules…​

You can configure LINSTOR to encrypt storage volumes, by using LUKS (dm-crypt), as detailed in the Encrypted Volumes section of this user’s guide. Refer to the LUKS and dm-crypt projects for FIPS compliance status.

You can also configure LINSTOR to encrypt communication traffic between a LINSTOR satellite and a LINSTOR controller, by using SSL/TLS, as detailed in the Secure Satellite Connections section of this user’s guide.

LINSTOR can also interface with Self-Encrypting Drives (SEDs) and you can use LINSTOR to initialize an SED drive. LINSTOR stores the drive’s password as a property that applies to the storage pool associated with the drive. LINSTOR encrypts the SED drive password by using the LINSTOR master passphrase that you must create first.

By default, LINSTOR uses the following cryptographic algorithms:

  • HMAC-SHA2-512

  • PBKDF2

  • AES-128

A FIPS compliant version of LINSTOR is available for the use cases mentioned in this section. If you or your organization require FIPS compliance at this level, contact [email protected] for details.

2.2. インストール

コンテナで LINSTOR を使用する場合は、このセクションをスキップして、以下の Containers セクションを使用してインストールしてください。

2.2.1. ボリュームマネージャーのインストール

LINSTOR を使用してストレージボリュームを作成するには、LVM または ZFS のいずれかのボリュームマネージャーをシステムにインストールする必要があります。

2.2.2. LINBIT クラスターノードを管理するスクリプトの使用

If you are a LINBIT® customer, you can download a LINBIT created helper script and run it on your nodes to:

  • クラスターノードをLINBITに登録する。

  • ノードを既存の LINBIT クラスターに参加させる。

  • ノードで LINBIT パッケージ リポジトリを有効にする。

Enabling LINBIT package repositories will give you access to LINBIT software packages, DRBD® kernel modules, and other related software such as cluster managers and OCF scripts. You can then use a package manager to fetch, install, and manage updating installed packages.

Downloading the LINBIT Manage Node Script

To register your cluster nodes with LINBIT, and configure LINBIT’s repositories, first download and then run the manage node helper script by entering the following commands on all cluster nodes:

# curl -O https://my.linbit.com/linbit-manage-node.py
# chmod +x ./linbit-manage-node.py
# ./linbit-manage-node.py
root ユーザーとしてスクリプトを実行する必要があります。

このスクリプトは、https://my.linbit.com/[LINBIT カスタマーポータル] のユーザー名とパスワードの入力を求めます。資格情報を入力すると、スクリプトはアカウントに関連付けられたクラスターノードを一覧表示します (最初は何もありません)。

LINBIT パッケージ リポジトリの有効化

ノードを登録するクラスターを指定した後、プロンプトが表示されたら、スクリプトで登録データを JSON ファイルに書き込みます。次に、スクリプトは、有効または無効にできる LINBIT リポジトリのリストを表示します。 LINSTOR およびその他の関連パッケージは、drbd-9 リポジトリにあります。別の DRBD バージョンブランチを使用する必要がない限り、少なくともこのリポジトリを有効にする必要があります。

ノード管理スクリプト内の最終タスク

リポジトリの選択が完了したら、スクリプトのプロンプトに従って構成をファイルに書き込むことができます。次に、LINBIT の公開署名鍵をノードのキーリングにインストールすることに関する質問には必ず yes と答えてください。

終了する前に、スクリプトは、さまざまなユースケースにインストールできるパッケージ案に関するメッセージを表示します。

On DEB based systems you can install a precompiled DRBD kernel module package, drbd-module-$(uname -r), or a source version of the kernel module, drbd-dkms. Install one or the other package but not both.

2.2.3. パッケージマネージャーを使用して LINSTOR をインストール

ノードを登録し drbd-9 LINBIT パッケージリポジトリを有効にすると、DEB、RPM、または YaST2 ベースのパッケージマネージャーを使用して LINSTOR と関連コンポーネントをインストールできます。

DEB ベースのパッケージマネージャーを使用している場合は、続行する前に apt update と入力してパッケージリポジトリリストを更新します。
LINSTOR ストレージ用の DRBD パッケージのインストール
If you will be using LINSTOR without DRBD, you can skip installing these packages.

LINSTOR を使用して DRBD 複製ストレージを作成できるようにする場合は、必要な DRBD パッケージをインストールする必要があります。ノードで実行している Linux ディストリビューションに応じて、ヘルパースクリプトが提案する DRBD 関連のパッケージをインストールします。スクリプトの推奨パッケージとインストールコマンドを確認する必要がある場合は、次のように入力できます。

# ./linbit-manage-node.py --hints
LINSTOR パッケージのインストール

LINSTOR をコントローラーノードにインストールするには、パッケージマネージャーを使用して linbit-sds-controller パッケージをインストールします。

LINSTOR をサテライトノードにインストールするには、パッケージマネージャーを使用して linbit-sds-satellite パッケージをインストールします。

ノードがサテライトとコントローラー (Combined ロール) の両方になる場合は、両方のパッケージをインストールします。

2.2.4. ソースコードから LINSTOR をインストール

LINSTOR プロジェクトの GitHub ページは https://github.com/LINBIT/linstor-server です。

LINBIT には、LINSTOR、DRBD などのソースコードのダウンロード可能なアーカイブファイルもあります。 https://linbit.com/linbit-software-download-page-for-linstor-and-drbd-linux-driver/ から利用可能です。

2.3. LINSTOR のアップグレード

LINSTOR doesn’t support rolling upgrades. Controller and satellites must have the same version, otherwise the controller will discard the satellite with a VERSION_MISMATCH. But this isn’t a problem, as the satellite won’t do any actions as long it isn’t connected to a controller and DRBD will not be disrupted by any means.

If you are using the embedded default H2 database and the linstor-controller package is upgraded an automatic backup file of the database will be created in the default /var/lib/linstor directory. This file is a good restore point if for any reason a linstor-controller database migration should fail, then it is recommended to report the error to LINBIT and restore the old database file and roll back to your previous controller version.

外部データベースまたはetcdを使用する場合は、現在のデータベースを手動でバックアップして復元ポイントを作成することをお勧めします。

First, upgrade the linstor-controller and linstor-client packages on your controller host and then restart the linstor-controller. The controller should start and all of its clients should show OFFLINE(VERSION_MISMATCH). After that you can continue upgrading the linstor-satellite package on all satellite nodes and restart them, after a short reconnection time they should all show ONLINE again and your upgrade is finished.

2.4. コンテナ

LINSTORはコンテナとしても利用可能です。ベースイメージはLINBITのコンテナレジストリ drbd.io にあります。

LINBIT’s container image repository (http://drbd.io) is only available to LINBIT customers or through LINBIT customer trial accounts. Contact LINBIT for information on pricing or to begin a trial. Alternatively, you can use LINSTOR SDS’ upstream project named Piraeus, without being a LINBIT customer.

画像にアクセスするには、まずLINBITカスタマーポータルの認証情報を使ってレジストリにログインする必要があります。

# docker login drbd.io

このリポジトリで利用可能なコンテナは以下です。

  • drbd.io/drbd9-rhel8

  • drbd.io/drbd9-rhel7

  • drbd.io/drbd9-sles15sp1

  • drbd.io/drbd9-bionic

  • drbd.io/drbd9-focal

  • drbd.io/linstor-csi

  • drbd.io/linstor-controller

  • drbd.io/linstor-satellite

  • drbd.io/linstor-client

An up-to-date list of available images with versions can be retrieved by opening http://drbd.io in your browser. Be sure to browse the image repository through HTTP, although the registry’s images themselves are pulled through HTTPS, using the associated docker pull command.

To load the kernel module, needed only for LINSTOR satellites, you’ll need to run a drbd9-$dist container in privileged mode. The kernel module containers either retrieve an official LINBIT package from a customer repository, use shipped packages, or they try to build the kernel modules from source. If you intend to build from source, you need to have the according kernel headers (for example, kernel-devel) installed on the host. There are four ways to execute such a module load container:

  • 出荷済みソースからビルドする。

  • 出荷済み/ビルド済みのカーネルモジュールの使用する。

  • LINBITノードのハッシュとディストリビューションを指定する。

  • 既存のリポジトリ設定をバインドマウントする。

ソースからビルドする場合(RHEL ベース):

# docker run -it --rm --privileged -v /lib/modules:/lib/modules \
  -v /usr/src:/usr/src:ro \
  drbd.io/drbd9-rhel7

コンテナに同梱されているモジュールを使用した例、 /usr/srcバインドマウントしません :

# docker run -it --rm --privileged -v /lib/modules:/lib/modules \
  drbd.io/drbd9-rhel8

ハッシュとディストリビューションを指定する場合: (あまり使わない):

# docker run -it --rm --privileged -v /lib/modules:/lib/modules \
  -e LB_DIST=rhel7.7 -e LB_HASH=ThisIsMyNodeHash \
  drbd.io/drbd9-rhel7

Example using an existing repository configuration (rarely used):

# docker run -it --rm --privileged -v /lib/modules:/lib/modules \
  -v /etc/yum.repos.d/linbit.repo:/etc/yum.repos.d/linbit.repo:ro \
  drbd.io/drbd9-rhel7
In both cases (hash + distribution, and bind-mounting a repository) the hash or repository configuration has to be from a node that has a special property set. Contact LINBIT customer support for help setting this property.
For now (that is, pre DRBD 9 version “9.0.17”), you must use the containerized DRBD kernel module, as opposed to loading a kernel module onto the host system. If you intend to use the containers you should not install the DRBD kernel module on your host systems. For DRBD version 9.0.17 or greater, you can install the kernel module as usual on the host system, but you need to load the module with the usermode_helper=disabled parameter (for example, modprobe drbd usermode_helper=disabled).

次に、デーモンとして特権モードで LINSTOR サテライトコンテナを実行します。

# docker run -d --name=linstor-satellite --net=host -v /dev:/dev \
  --privileged drbd.io/linstor-satellite
コンテナ化された drbd-utils がネットワークを通してホストカーネルと通信できるようにするには net=host が必要です。

LINSTOR コントローラコンテナをデーモンとして実行するには、ホスト上のポート 3370, 3376, 3377 をコンテナにマッピングします。

# docker run -d --name=linstor-controller -p 3370:3370 drbd.io/linstor-controller

コンテナ化された LINSTOR クラスタと対話するには、パッケージを介してシステムにインストールされた LINSTOR クライアント、またはコンテナ化された LINSTOR クライアントを使用することができます。 LINSTOR クライアントコンテナを使用するには

# docker run -it --rm -e LS_CONTROLLERS=<controller-host-IP-address> \
  drbd.io/linstor-client node list

これ以降は、LINSTOR クライアントを使用してクラスタを初期化し、一般的な LINSTOR パターンを使用してリソースの作成を開始します。

デーモン化されたコンテナとイメージを停止して削除します。

# docker stop linstor-controller
# docker rm linstor-controller

2.5. クラスタの初期化

Before initializing your LINSTOR cluster, you must meet the following prerequisites on all cluster nodes:

  1. DRBD9カーネルモジュールがインストールされ、ロードされている。

  2. The drbd-utils package is installed.

  3. LVM` ツールがインストールされている。

  4. linstor-controller or linstor-satellite packages and their dependencies are installed on appropriate nodes.

  5. linstor-clientlinstor-controller ノードにインストールされている。

コントローラがインストールされているホストで linstor-controller サービスを起動して有効にします。

# systemctl enable --now linstor-controller

2.6. LINSTORクライアントの使用

Whenever you run the LINSTOR command line client, it needs to know on which cluster node the linstor-controller service is running. If you do not specify this, the client will try to reach a locally running linstor-controller service listening on IP address 127.0.0.1 port 3370. Therefore use the linstor-client on the same host as the linstor-controller.

The linstor-satellite service requires TCP ports 3366 and 3367. The linstor-controller service requires TCP port 3370. Verify that you have this port allowed on your firewall.
# linstor node list

Output from this command should show you an empty list and not an error message.

linstor コマンドはどのマシンでも実行できます。ただし、クライアントにどのようにlinstor-controllerを探すか指定する必要があります。これはコマンドラインのオプションか、環境変数、またはグローバルファイルで指定できます。

# linstor --controllers=alice node list
# LS_CONTROLLERS=alice linstor node list

あるいは、 /etc/linstor/linstor-client.conf ファイルを作成して、以下のように追加することもできます。

[global]
controllers=alice

If you have multiple LINSTOR controllers configured you can simply specify them all in a comma-separated list. The LINSTOR client will try them in the order listed.

The LINSTOR client commands can also be used in a much faster and convenient way by only writing the starting letters of the parameters. For example, rather than entering linstor node list you can enter the LINSTOR short notation command linstor n l.

2.7. ノードをクラスタに追加する

次の手順ではノードを LINSTOR クラスタに追加します。

# linstor node create bravo 10.43.70.3

If you omit the IP address, the LINSTOR client will try to resolve the specified node name, bravo in the preceding example, as a hostname. If the hostname does not resolve to a host on the network from the system where the LINSTOR controller service is running, then LINSTOR will show an error message when you try to create the node:

Unable to resolve ip address for 'bravo': [Errno -3] Temporary failure in name resolution

2.7.1. Naming LINSTOR Nodes

If you specify an IP address when you create a LINSTOR node, you can give your node an arbitrary name. The LINSTOR client will show an INFO message about this when you create the node:

    [...] 'arbitrary-name' and hostname 'node-1' doesn't match.

LINSTOR will automatically detect the created node’s local uname --nodename which will be later used for DRBD resource configurations, rather than the arbitrary node name. To avoid confusing yourself and possibly others, in most cases, it would make sense to just use a node’s hostname when creating a LINSTOR node.

2.7.2. Starting and Enabling a LINSTOR Satellite Node

When you use linstor node list LINSTOR will show that the new node is marked as offline. Now start and enable the LINSTOR satellite service on the new node so that the service comes up on reboot as well:

# systemctl enable --now  linstor-satellite

About 10 seconds later you will see the status in linstor node list becoming online. Of course the satellite process might be started before the controller knows about the existence of the satellite node.

In case the node which hosts your controller should also contribute storage to the LINSTOR cluster, you have to add it as a node and also start the linstor-satellite service.

linstor-satellite が必要なデバイスを作成する機会が得られるまで(ブート後など)他のサービスを待機させたい場合は、対応する .service ファイルを更新し、Type=simpleType=notify に変更します。

これにより、サテライトはコントローラーが接続し、必要なすべてのデータをサテライトに送信し、サテライトがデバイスの起動と実行を少なくとも1回試行するまで、systemd への READY=1 メッセージの送信を遅らせます。

2.7.3. Specifying LINSTOR Node Types

When you create a LINSTOR node, you can also specify a node type. Node type is a label that indicates the role that the node serves within your LINSTOR cluster. Node type can be one of controller, auxiliary, combined, or satellite. For example to create a LINSTOR node and label it as a controller and a satellite node, enter the following command:

# linstor node create bravo 10.43.70.3 --node-type combined

The --node-type argument is optional. If you do not specify a node type when you create a node, LINSTOR will use a default type of satellite.

If you want to change a LINSTOR node’s assigned type after creating the node, you can enter a linstor node modify --node-type command.

2.8. ストレージプール

StoragePools identify storage in the context of LINSTOR. To group storage pools from multiple nodes, simply use the same name on each node. For example, one valid approach is to give all SSDs one name and all HDDs another.

2.8.1. ストレージプールの作成

On each host contributing storage, you need to create either an LVM volume group (VG) or a ZFS zPool. The VGs and zPools identified with one LINSTOR storage pool name might have different VG or zPool names on the hosts, but do yourself a favor, for coherency, use the same VG or zPool name on all nodes.

# vgcreate vg_ssd /dev/nvme0n1 /dev/nvme1n1 [...]

After creating a volume group on each of your nodes, you can create a storage pool that is backed by the volume group on each of your nodes, by entering the following commands:

# linstor storage-pool create lvm alpha pool_ssd vg_ssd
# linstor storage-pool create lvm bravo pool_ssd vg_ssd

ストレージプールを表示するには、次のようにします。

# linstor storage-pool list

or using LINSTOR abbreviated notation:

# linstor sp l

2.8.2. Using Storage Pools To Confine Failure Domains to a Single Back-end Device

In clusters where you have only one kind of storage and the capability to hot swap storage devices, you might choose a model where you create one storage pool per physical backing device. The advantage of this model is to confine failure domains to a single storage device.

2.8.3. 複数のノードでストレージプールを共有する

Both the Exos and LVM2 storage providers offer the option of multiple server nodes directly connected to the storage array and drives. With LVM2 the external locking service (lvmlockd) manages volume groups created with the –shared options with vgcreate. The --shared-space can be used when configuring a LINSTOR pool to use the same LVM2 volume group accessible by two or more nodes. The example below shows using the LVM2 volume group UUID as the shared space identifier for a pool accessible by nodes alpha and bravo:

# linstor storage-pool create lvm --external-locking \
  --shared-space O1btSy-UO1n-lOAo-4umW-ETZM-sxQD-qT4V87 \
  alpha pool_ssd shared_vg_ssd
# linstor storage-pool create lvm --external-locking \
  --shared-space O1btSy-UO1n-lOAo-4umW-ETZM-sxQD-qT4V87 \
  bravo pool_ssd shared_vg_ssd

Exos プールは、共有スペース識別子にデフォルトで Exos プールのシリアル番号を使用します。

As of the release of linstor-server v1.26.0, the Exos integration for LINSTOR is deprecated.

2.8.4. Creating Storage Pools by Using the Physical Storage Command

Since linstor-server 1.5.2 and a recent linstor-client, LINSTOR can create LVM/ZFS pools on a satellite for you. The LINSTOR client has the following commands to list possible disks and create storage pools, but such LVM/ZFS pools are not managed by LINSTOR and there is no delete command, so such action must be done manually on the nodes.

# linstor physical-storage list

これはサイズと回転方式 (SSD/磁気ディスク) でグループ化された使用可能なディスクのリストを表示します。

次のフィルターを通過するディスクのみが表示されます。

  • デバイスサイズは 1GiB より大きい

  • デバイスはルートデバイス(子はなし)例: /dev/vda, /dev/sda

  • The device does not have any file system or other blkid marker (wipefs -a might be needed).

  • デバイスは DRBD デバイスでない

create-device-pool コマンドを使用して、ディスク上に LVM プールを作成し、LINSTOR のストレージプールとして直接追加することもできます。

# linstor physical-storage create-device-pool --pool-name lv_my_pool \
  LVMTHIN node_alpha /dev/vdc --storage-pool newpool

If the --storage-pool option was provided, LINSTOR will create a storage pool with the given name.

For more options and exact command usage refer to the LINSTOR client --help text.

2.9. リソースグループを使用した LINSTOR プロビジョニングされたボリュームのデプロイ

Using resource groups to define how you want your resources provisioned should be considered the de facto method for deploying volumes provisioned by LINSTOR. Chapters that follow which describe creating each resource from a resource definition and volume definition should only be used in special scenarios.

Even if you choose not to create and use resource groups in your LINSTOR cluster, all resources created from resource definitions and volume definitions will exist in the ‘DfltRscGrp’ resource group.

リソースグループを使用してリソースをデプロイする簡単なパターンは次のようになります。

# linstor resource-group create my_ssd_group --storage-pool pool_ssd --place-count 2
# linstor volume-group create my_ssd_group
# linstor resource-group spawn-resources my_ssd_group my_ssd_res 20G

上記のコマンドを実行すると ‘pool_ssd’ という名前のストレージプールに参加するノードから、2つに複製された 20GB ボリュームを持つ ‘my_ssd_res’ という名前のリソースが自動的にプロビジョニングされます。

より有効なパターンは、ユースケースが最適であると判断した設定でリソースグループを作成することです。もし一貫性のためボリュームの夜間オンライン検証を実行する必要があるというケースの場合、’verify-alg’ がすでに設定されたリソースグループを作成することで、グループから生成されるリソースは ‘verify-alg’ が事前設定されます。

# linstor resource-group create my_verify_group --storage-pool pool_ssd --place-count 2
# linstor resource-group drbd-options --verify-alg crc32c my_verify_group
# linstor volume-group create my_verify_group
# for i in {00..19}; do
    linstor resource-group spawn-resources my_verify_group res$i 10G
  done

The commands above result in twenty 10GiB resources being created each with the crc32c verify-alg pre-configured.

You can tune the settings of individual resources or volumes spawned from resource groups by setting options on the respective resource definition or volume definition LINSTOR objects. For example, if res11 from the preceding example is used by a very active database receiving many small random writes, you might want to increase the al-extents for that specific resource:

# linstor resource-definition drbd-options --al-extents 6007 res11

If you configure a setting in a resource definition that is already configured on the resource group it was spawned from, the value set in the resource definition will override the value set on the parent resource group. For example, if the same res11 was required to use the slower but more secure sha256 hash algorithm in its verifications, setting the verify-alg on the resource definition for res11 would override the value set on the resource group:

# linstor resource-definition drbd-options --verify-alg sha256 res11
A guiding rule for the hierarchy in which settings are inherited is that the value “closer” to the resource or volume wins. Volume definition settings take precedence over volume group settings, and resource definition settings take precedence over resource group settings.

2.10. クラスターの構成

2.10.1. Available Storage Plugins

LINSTOR has the following supported storage plugins at the time of writing:

  • Thick LVM

  • 単一の thin プールの Thin LVM

  • Thick ZFS

  • Thin ZFS

2.11. リソース、ボリュームの作成と配備

You can use the LINSTOR create command to create various LINSTOR objects, such as resource definitions, volume definitions, and resources. Some of these commands are shown below.

In the following example scenario, assume that you have a goal of creating a resource named backups with a size of 500GiB that is replicated among three cluster nodes.

First, create a new resource definition:

# linstor resource-definition create backups

Second, create a new volume definition within that resource definition:

# linstor volume-definition create backups 500G

If you want to resize (grow or shrink) the volume definition you can do that by specifying a new size with the set-size command:

# linstor volume-definition set-size backups 0 100G
The size of a volume definition can only be decreased if it has no associated resource. However, you can freely increase the size of a volume definition, even one having a deployed resource.

The parameter 0 is the number of the volume in the resource backups. You have to provide this parameter because resources can have multiple volumes that are identified by a so-called volume number. You can find this number by listing the volume definitions (linstor vd l). The list table will show volume numbers for the listed resources.

So far you have only created definition objects in LINSTOR’s database. However, not a single logical volume (LV) has been created on the satellite nodes. Now you have the choice of delegating the task of deploying resources to LINSTOR or else doing it yourself.

2.11.1. リソースを手動で配置する

With the resource create command you can assign a resource definition to named nodes explicitly.

# linstor resource create alpha backups --storage-pool pool_hdd
# linstor resource create bravo backups --storage-pool pool_hdd
# linstor resource create charlie backups --storage-pool pool_hdd

2.11.2. DRBD リソースの自動化

When you create (spawn) a resource from a resource group, it is possible to have LINSTOR automatically select nodes and storage pools to deploy the resource to. You can use the arguments mentioned in this section to specify constraints when you create or modify a resource group. These constraints will affect how LINSTOR automatically places resources that are deployed from the resource group.

Automatically Maintaining Resource Group Placement Count

Starting with LINSTOR version 1.26.0, there is a reoccurring LINSTOR task that tries to maintain the placement count set on a resource group for all deployed LINSTOR resources that belong to that resource group. This includes the default LINSTOR resource group and its placement count.

If you want to disable this behavior, set the BalanceResourcesEnabled property to false on the the LINSTOR controller, LINSTOR resource groups that your resources belong to, or the resource definitions themselves. Due to LINSTOR object hierarchy, if you set a property on a resource group, it will override the property value on the LINSTOR controller. Likewise, if you set a property on a resource definition, it will override the property value on the resource group that the resource definition belongs to.

There are other additional properties related to this feature that you can set on the LINSTOR controller:

BalanceResourcesInterval

The interval in seconds, at the LINSTOR controller level, that the balance resource placement task is triggered. By default, the interval is 3600 seconds (one hour).

BalanceResourcesGracePeriod

The period in seconds for how long new resources (after being created or spawned) are ignored for balancing. By default, the grace period is 3600 seconds (one hour).

Placement Count

By using the --place-count <replica_count> argument when you create or modify a resource group, you can specify on how many nodes in your cluster LINSTOR should place diskful resources created from the resource group.

Creating a resource group with impossible placement constraints

You can create or modify a resource group and specify a placement count or other contraint that would be impossible for LINSTOR to fulfill. For example, you could specify a placement count of 7 when you only have three nodes in your cluster. LINSTOR will create such a resource group without complaining. However, LINSTOR will display an error message when you try to spawn a resource from the resource group. For example:

ERROR:
Description:
    Not enough available nodes
[...]
Storage Pool Placement

In the following example, the value after the --place-count option tells LINSTOR how many replicas you want to have. The --storage-pool option should be obvious.

# linstor resource-group create backups --place-count 3 --storage-pool pool_hdd

What might not be obvious is that you can omit the --storage-pool option. If you do this, then LINSTOR can select a storage pool on its own when you create (spawn) resources from the resource group. The selection follows these rules:

  • 現在のユーザがアクセスできないすべてのノードとストレージプールは無視する。

  • すべてのディスクレスストレージプールは無視する

  • 十分な空き領域がないストレージプールは無視する

The remaining storage pools will be rated by different strategies.

MaxFreeSpace

This strategy maps the rating 1:1 to the remaining free space of the storage pool. However, this strategy only considers the actually allocated space (in case of thin-provisioned storage pool this might grow with time without creating new resources)

MinReservedSpace

Unlike the “MaxFreeSpace”, this strategy considers the reserved space. That is the space that a thin volume can grow to before reaching its limit. The sum of reserved spaces might exceed the storage pool’s capacity, which is as overprovisioning.

MinRscCount

Simply the count of resources already deployed in a given storage pool

MaxThroughput

For this strategy, the storage pool’s Autoplacer/MaxThroughput property is the base of the score, or 0 if the property is not present. Every Volume deployed in the given storage pool will subtract its defined sys/fs/blkio_throttle_read and sys/fs/blkio_throttle_write property- value from the storage pool’s max throughput. The resulting score might be negative.

各方針のスコアは正規化され、重み付けされ、合計されます。この場合、最小化方針のスコアが最初に変換され、結果として得られるスコアの全体的な最大化が可能になります。

You can configure the weights of the strategies to affect how LINSTOR selects a storage pool for resource placement when creating (spawning) resources for which you did not specify a storage pool. You do this by setting the following properties on the LINSTOR controller object. The weight can be an arbitrary decimal value.

linstor controller set-property Autoplacer/Weights/MaxFreeSpace <weight>
linstor controller set-property Autoplacer/Weights/MinReservedSpace <weight>
linstor controller set-property Autoplacer/Weights/MinRscCount <weight>
linstor controller set-property Autoplacer/Weights/MaxThroughput <weight>
Autoplacerの動作を以前のLINSTORバージョンと互換性を保つために、重みが 1 の MaxFreeSpace を除いて、すべてのストラテジーのデフォルトの重みは0です。
0点でもマイナス点でも、ストレージプールが選択されるのを妨げることはありません。これらのスコアを持つストレージプールは、後で検討されるだけです。

最後に、LINSTORはすべての要件を満たすストレージプールのベストマッチングを見つけようとします。このステップでは、 --replicas-on-same, --replicas-on-different, --do-not-place-with, --do-not-place-with-regex, --layer-list, --providers などの自動配置に関する制限も考慮されます。

リソースを自動的に配置するときのリソースの併置の回避

The --do-not-place-with <resource_name_to_avoid> argument specifies that LINSTOR should try to avoid placing a resource on nodes that already have the specified, resource_name_to_avoid resource deployed.

By using the --do-not-place-with-regex <regular_expression> argument, you can specify that LINSTOR should try to avoid placing a resource on nodes that already have a resource deployed whose name matches the regular expression that you provide with the argument. In this way, you can specify multiple resources to try to avoid placing your resource with.

補助ノードプロパティを用いたリソース自動配置の制約付け

リソースの自動配置は、指定した補助ノードプロパティを持つノードにリソースを配置する(または配置しない)ように制約することができます。

この機能は、LINSTORマネージドストレージを使用するKubernetes環境内でリソースの配置を制限しようとしている場合に、特に有用となります。例えば、Kubernetesのラベルに対応する補助ノードのプロパティを設定することができます。このユースケースの詳細については、「LINSTOR Volumes in Kubernetes」LINSTOR User’s Guide の章にある “replicasOnSame” sectionをご覧ください。

引数の --replicas-on-same--replicas-on-different は、 Aux/ 名前空間内のプロパティの名前を指定します。

The following example shows setting an auxiliary node property, testProperty, on three LINSTOR satellite nodes. Next, you create a resource group, testRscGrp, with a placement count of two and a constraint to place spawned resources on nodes that have a testProperty value of 1. After creating a volume group, you can spawn a resource from the resource group. For simplicity, output from the following commands is not shown.

# for i in {0,2}; do linstor node set-property --aux node-$i testProperty 1; done
# linstor node set-property --aux node-1 testProperty 0
# linstor resource-group create testRscGrp --place-count 2 --replicas-on-same testProperty=1
# linstor volume-group create testRscGrp
# linstor resource-group spawn-resources testRscGrp testResource 100M

以下のコマンドで、生成されたリソースの配置を確認することができます。

|=====================================================================================|
# linstor resource list
+-------------------------------------------------------------------------------------+
| ResourceName      | Node   | Port | Usage  | Conns |    State | CreatedOn           |
| testResource      | node-0 | 7000 | Unused | Ok    | UpToDate | 2022-07-27 16:14:16 |
| testResource      | node-2 | 7000 | Unused | Ok    | UpToDate | 2022-07-27 16:14:16 |
+-------------------------------------------------------------------------------------+

Because of the --replicas-on-same constraint, LINSTOR did not place the spawned resource on satellite node node-1, because the value of its auxiliary node property, testProperty was 0 and not 1.

You can verify the node properties of node-1, by using the list-properties command:

# linstor node list-properties node-1
+----------------------------+
| Key              | Value   |
|============================|
| Aux/testProperty | 0       |
| CurStltConnName  | default |
| NodeUname        | node-1  |
+----------------------------+
LINSTORレイヤーまたはストレージプールプロバイダによるリソースの自動配置の制約

You can specify the --layer-list or --providers arguments, followed by a comma-separated values (CSV) list of LINSTOR layers or storage pool providers, to influence where LINSTOR places resources. The possible layers and storage pool providers that you can specify in your CSV list can be shown by using the --help option with the --auto-place option. A CSV list of layers would constrain automatic resource placement for a specified resource group to nodes that have storage that conformed with your list. Consider the following command:

# linstor resource-group create my_luks_rg --place-count 3 --layer-list drbd,luks

Resources that you might later create (spawn) from this resource group would be deployed across three nodes having storage pools backed by a DRBD layer backed by a LUKS layer (and implicitly backed by a “storage” layer). The order of layers that you specify in your CSV list is “top-down”, where a layer on the left in the list is above a layer on its right.

The --providers argument can be used to constrain automatic resource placement to only storage pools that match those in a specified CSV list. You can use this argument to have explicit control over which storage pools will back your deployed resource. If for example, you had a mixed environment of ZFS, LVM, and LVM_THIN storage pools in your cluster, by using the --providers LVM,LVM_THIN argument, you can specify that a resource only gets backed by either an LVM or LVM_THIN storage pool, when using the --place-count option.

The --providers argument’s CSV list does not specify an order of priority for the list elements. Instead, LINSTOR will use factors like additional placement constraints, available free space, and LINSTOR’s storage pool selection strategies that were previously described, when placing a resource.
Automatically Placing Resources When Creating Them

While using resource groups to create templates from which you can create (spawn) resources from is the standard way to create resources, you can also create resources directly by using the resource create command. When you use this command, it is also possible to specify arguments that affect how LINSTOR will place the resource in your storage cluster.

With the exception of the placement count argument, the arguments that you can specify when you use the resource create command that affect where LINSTOR places the resource are the same as those for the resource-group create command. Specifying an --auto-place <replica_count> argument with a resource create command is the same as specifying a --place-count <replica_count> argument with a resource-group create command.

Using Auto-place to Extend Existing Resource Deployments

Besides the argument name, there is another difference between the placement count argument for the resource group and resource create commands. With the resource create command, you can also specify a value of +1 with the --auto-place argument, if you want to extend existing resource deployments.

By using this value, LINSTOR will create an additional replica, no matter what the --place-count is configured for on the resource group that the resource was created from.

For example, you can use an --auto-place +1 argument to deploy an additional replica of the testResource resource used in a previous example. You will first need to set the auxiliary node property, testProperty to 1 on node-1. Otherwise, LINSTOR will not be able to deploy the replica because of the previously configured --replicas-on-same constraint. For simplicity, not all output from the commands below is shown.

# linstor node set-property --aux node-1 testProperty 1
# linstor resource create --auto-place +1 testResource
# linstor resource list
+-------------------------------------------------------------------------------------+
| ResourceName      | Node   | Port | Usage  | Conns |    State | CreatedOn           |
|=====================================================================================|
| testResource      | node-0 | 7000 | Unused | Ok    | UpToDate | 2022-07-27 16:14:16 |
| testResource      | node-1 | 7000 | Unused | Ok    | UpToDate | 2022-07-28 19:27:30 |
| testResource      | node-2 | 7000 | Unused | Ok    | UpToDate | 2022-07-27 16:14:16 |
+-------------------------------------------------------------------------------------+

警告: +1 という値は resource-group create --place-count コマンドでは有効ではありません。 これは、このコマンドはリソースをデプロイせず、後でデプロイするためのテンプレートを作成する だけだからです。

2.12. リソース、リソース定義、およびリソース グループの削除

削除したい LINSTOR オブジェクトの後に delete コマンドを使用して、LINSTOR リソース、リソース定義、およびリソース グループを削除できます。どのオブジェクトを削除するかによって、LINSTOR クラスターおよびその他の関連する LINSTOR オブジェクトに異なる影響があります。

2.12.1. リソース定義の削除

次のコマンドを使用してリソース定義を削除できます:

# linstor resource-definition delete <resource_definition_name>

これにより、指定されたリソース定義が LINSTOR クラスター全体から削除されます。リソースはすべてのノードから削除され、リソース エントリは LINSTOR のデータベース テーブルから削除するようにマークされます。LINSTOR がすべてのノードからリソースを削除した後、リソース エントリは LINSTOR のデータベース テーブルから削除されます。

If your resource definition has existing snapshots, you will not be able to delete the resource definition until you delete its snapshots. See the スナップショットの削除 section in this guide.

2.12.2. リソースの削除

次のコマンドを使用してリソースを削除できます:

# linstor resource delete <node_name> <resource_name>

リソース定義の削除とは異なり、このコマンドは、指定したノード (複数可) から LINSTOR リソースのみを削除します。リソースはノードから削除され、リソースエントリは LINSTOR のデータベーステーブルから削除するようにマークされます。LINSTOR がノードからリソースを削除した後、リソースエントリは LINSTOR のデータベーステーブルから削除されます。

Deleting a LINSTOR resource might have implications for a cluster, beyond just removing the resource. For example, if the resource is backed by a DRBD layer, removing a resource from one node in a three node cluster could also remove certain quorum related DRBD options, if any existed for the resource. After removing such a resource from a node in a three node cluster, the resource would no longer have quorum as it would now only be deployed on two nodes in the three node cluster.

「linstor resource delete」コマンドを実行して単一ノードからリソースを削除すると、次のような情報メッセージが表示される場合があります。

INFO:
    Resource-definition property 'DrbdOptions/Resource/quorum' was removed as there are not enough resources for quorum
INFO:
    Resource-definition property 'DrbdOptions/Resource/on-no-quorum' was removed as there are not enough resources for quorum

また、リソース定義の削除とは異なり、リソースのストレージプールのスナップショットが存在している間にリソースを削除できます。リソースのストレージプールの既存のスナップショットは保持されます。

2.12.3. リソースグループの削除

次のコマンドを使用して、リソースグループを削除できます。

# linstor resource-group delete <resource_group_name>

このコマンドは指定されたリソースグループを削除します。リソース定義が関連付けられていない場合のみ、リソースグループを削除できます。削除しない場合、LINSTOR は次のようなエラー メッセージを表示します。

ERROR:
Description:
    Cannot delete resource group 'my_rg' because it has existing resource definitions.

リソースグループを削除できるようにこのエラーを解決するには、関連するリソース定義を削除するか、リソース定義を別の (既存の) リソースグループに移動します。

# linstor resource-definition modify <resource_definition_name> \ --resource-group <another_resource_group_name>

次のコマンドを入力すると、リソース グループに関連付けられているリソース定義を見つけることができます。

# linstor resource-definition list

2.13. Backup and Restore Database

Since version 1.24.0, LINSTOR has a tool that you can use to export and import a LINSTOR database.

This tool has an executable file called /usr/share/linstor-server/bin/linstor-database. This executable has two subcommands, export-db and import-db. Both subcommands accept an optional --config-directory argument that you can use to specify the directory containing the linstor.toml configuration file.

To ensure a consistent database backup, take the controller offline by stopping the controller service as shown in the commands below, before creating a backup of the LINSTOR database.

2.13.1. Backing Up the Database

To backup the LINSTOR database to a new file named db_export in your home directory, enter the following commands:

# systemctl stop linstor-controller # /usr/share/linstor-server/bin/linstor-database export-db ~/db_export # systemctl start linstor-controller
You can use the --config-directory argument with the linstor-database utility to specify a LINSTOR configuration directory if needed. If you omit this argument, the utility uses the /etc/linstor directory by default.

After backing up the database, you can copy the backup file to a safe place.

# cp ~/db_export <somewhere safe>

The resulting database backup is a plain JSON document, containing not just the actual data, but also some metadata about when the backup was created, from which database, and other information.

2.13.2. Restoring the Database From a Backup

Restoring the database from a previously made backup is similar to export-db from the previous section.

For example, to restore the previously made backup from the db_export file, enter the following commands:

# systemctl stop linstor-controller # /usr/share/linstor-server/bin/linstor-database import-db ~/db_export # systemctl start linstor-controller

You can only import a database from a previous backup if the currently installed version of LINSTOR is the same (or higher) version as the version that you created the backup from. If the currently installed LINSTOR version is higher than the version that the database backup was created from, when you import the backup the data will be restored with the same database scheme of the version used during the export. Then, the next time that the controller starts, the controller will detect that the database has an old scheme and it will automatically migrate the data to the scheme of the current version.

2.13.3. Converting Databases

Since the exported database file contains some metadata, an exported database file can be imported into a different database type than it was exported from.

This allows the user to convert, for example, from an etcd setup to an SQL based setup. There is no special command for converting the database format. You only have to specify the correct linstor.toml configuration file by using the --config-directory argument (or updating the default /etc/linstor/linstor.toml and specifying the database type that you want to use before importing). See the LINSTOR User’s Guide for more information about specifying a database type. Regardless of the type of database that the backup was created from, it will be imported in the database type that is specified in the linstor.toml configuration file.

3. LINSTOR 応用タスク

3.1. 高可用性 LINSTOR クラスターの作成

By default a LINSTOR cluster consists of exactly one active LINSTOR controller node. Making LINSTOR highly available involves providing replicated storage for the controller database, multiple LINSTOR controller nodes where only one is active at a time, and a service manager (here DRBD Reactor) that takes care of mounting and unmounting the highly available storage as well as starting and stopping the LINSTOR controller service on nodes.

3.1.1. 高可用性ストレージの構成

For configuring the highly available (HA) storage, you can use LINSTOR itself. One of the benefits of having the storage under LINSTOR control is that you can easily extend the HA storage to new cluster nodes.

First, create a new resource that is 200MiB in size and set the necessary DRBD options, as shown in the example commands below. You will need to adapt the storage pool name to match an existing storage pool in your environment.

# linstor resource-definition create linstor_db # linstor rd drbd-options --auto-promote=no linstor_db # linstor rd drbd-options --quorum=majority linstor_db # linstor rd drbd-options --on-suspended-primary-outdated=force-secondary linstor_db # linstor rd drbd-options --on-no-quorum=io-error linstor_db # linstor rd drbd-options --on-no-data-accessible=io-error linstor_db # linstor rd drbd-options --rr-conflict=retry-connect linstor_db # linstor volume-definition create linstor_db 200M # linstor resource create linstor_db --storage-pool pool1 --auto-place 3

It is crucial that your cluster qualifies for auto-quorum and uses the io-error policy (see Section [s-linstor-auto-quorum]), and that auto-promote is disabled.

From now on it is assumed that the resource’s name is linstor_db. After creating the linstor_db resource, you can move the LINSTOR database to the new storage and create a systemd mount service. First, stop the current controller service and disable it, as it will be managed by DRBD Reactor later.

# systemctl disable --now linstor-controller

Next, create the systemd mount service.

# cat << EOF > /etc/systemd/system/var-lib-linstor.mount
[Unit]
Description=Filesystem for the LINSTOR controller

[Mount]
# you can use the minor like /dev/drbdX or the udev symlink What=/dev/drbd/by-res/linstor_db/0 Where=/var/lib/linstor EOF

# mv /var/lib/linstor{,.orig} # mkdir /var/lib/linstor # chattr +i /var/lib/linstor # only if on LINSTOR >= 1.14.0 # drbdadm primary linstor_db # mkfs.ext4 /dev/drbd/by-res/linstor_db/0 # systemctl start var-lib-linstor.mount # cp -r /var/lib/linstor.orig/* /var/lib/linstor # systemctl start linstor-controller

Copy the /etc/systemd/system/var-lib-linstor.mount mount file to all the cluster nodes that you want to have the potential to run the LINSTOR controller service (standby controller nodes). Again, do not systemctl enable any of these services because DRBD Reactor will manage them.

3.1.2. 複数のLINSTORコントローラー

The next step is to install LINSTOR controllers on all nodes that have access to the linstor_db DRBD resource (as they need to mount the DRBD volume) and which you want to become a possible LINSTOR controller. It is important that the controllers are manged by drbd-reactor, so verify that the linstor-controller.service is disabled on all nodes! To be sure, execute systemctl disable linstor-controller on all cluster nodes and systemctl stop linstor-controller on all nodes except the one it is currently running from the previous step. Also verify that you have set chattr +i /var/lib/linstor on all potential controller nodes if you use LINSTOR version equal or greater to 1.14.0.

3.1.3. サービスの管理

For starting and stopping the mount service and the linstor-controller service, use DRBD Reactor. Install this component on all nodes that could become a LINSTOR controller and edit their /etc/drbd-reactor.d/linstor_db.toml configuration file. It should contain an enabled promoter plugin section like this:

[[promoter]]
id = "linstor_db"
[promoter.resources.linstor_db]
start = ["var-lib-linstor.mount", "linstor-controller.service"]

Depending on your requirements you might also want to set an on-stop-failure action and set stop-services-on-exit.

その後 drbd-reactor を再起動し、構成したすべてのノードで有効にします。

# systemctl restart drbd-reactor # systemctl enable drbd-reactor

Check that there are no warnings from drbd-reactor service in the logs by running systemctl status drbd-reactor. As there is already an active LINSTOR controller things will just stay the way they are. Run drbd-reactorctl status linstor_db to check the health of the linstor_db target unit.

The last but nevertheless important step is to configure the LINSTOR satellite services to not delete (and then regenerate) the resource file for the LINSTOR controller DB at its startup. Do not edit the service files directly, but use systemctl edit. Edit the service file on all nodes that could become a LINSTOR controller and that are also LINSTOR satellites.

# systemctl edit linstor-satellite
[Service]
Environment=LS_KEEP_RES=linstor_db

After this change you should execute systemctl restart linstor-satellite on all satellite nodes.

Be sure to configure your LINSTOR client for use with multiple controllers as described in the section titled, LINSTORクライアントの使用 and verify that you also configured your integration plugins (for example, the Proxmox plugin) to be ready for multiple LINSTOR controllers.

3.2. DRBDクライアント

By using the --drbd-diskless option instead of --storage-pool you can have a permanently diskless DRBD device on a node. This means that the resource will appear as block device and can be mounted to the file system without an existing storage-device. The data of the resource is accessed over the network on another node with the same resource.

# linstor resource create delta backups --drbd-diskless
The option --diskless is deprecated. Use the --drbd-diskless or --nvme-initiator commands instead.

3.3. LINSTOR – DRBDコンシステンシグループ/マルチボリューム

The so called consistency group is a feature from DRBD. It is mentioned in this user’s guide, due to the fact that one of LINSTOR’s main functions is to manage storage-clusters with DRBD. Multiple volumes in one resource are a consistency group.

これは、1つのリソースで異なるボリュームに対する変更が他のサテライトでも同じ順番で複製されることを意味します。

したがって、リソース内の異なるボリュームに相互依存データがある場合は、タイミングを気にする必要はありません。

LINSTORリソースに複数のボリュームを配備するには、同じ名前の volume-definitions を2つ作成する必要があります。

# linstor volume-definition create backups 500G # linstor volume-definition create backups 100G

3.4. 1つのリソースのボリュームを異なるストレージプールに配置

リソースをノードに配備する前のボリューム定義で StorPoolName プロパティを使うことで、1つのリソースから異なるストレージプールへのボリュームを作成できます。

# linstor resource-definition create backups # linstor volume-definition create backups 500G # linstor volume-definition create backups 100G # linstor volume-definition set-property backups 0 StorPoolName pool_hdd # linstor volume-definition set-property backups 1 StorPoolName pool_ssd # linstor resource create alpha backups # linstor resource create bravo backups # linstor resource create charlie backups
Since the volume-definition create command is used without the --vlmnr option LINSTOR assigned the volume numbers starting at 0. In the following two lines the 0 and 1 refer to these automatically assigned volume numbers.

Here the ‘resource create’ commands do not need a --storage-pool option. In this case LINSTOR uses a ‘fallback’ storage pool. Finding that storage pool, LINSTOR queries the properties of the following objects in the following order:

  • Volume definition

  • Resource

  • Resource definition

  • Node

If none of those objects contain a StorPoolName property, the controller falls back to a hard-coded ‘DfltStorPool’ string as a storage pool.

This also means that if you forgot to define a storage pool prior deploying a resource, you will get an error message that LINSTOR could not find the storage pool named ‘DfltStorPool’.

3.5. DRBDを使わないLINSTOR

LINSTORはDRBDを使わずとも使用できます。 DRBDがなくても、LINSTORはLVMおよびZFS 下位ストレージプールからボリュームをプロビジョニングし、LINSTORクラスタ内の個々のノードにそれらのボリュームを作成できます。

現在LINSTORはLVMとZFSボリュームの作成をサポートしており、LUKS、DRBD、または NVMe-oF/NVMe-TCP の組み合わせをそれらのボリュームの上に重ねることもできます。

For example, assume you have a Thin LVM backed storage pool defined in your LINSTOR cluster named, thin-lvm:

# linstor --no-utf8 storage-pool list
+--------------------------------------------------------------+
| StoragePool | Node      | Driver   | PoolName          | ... |
|--------------------------------------------------------------|
| thin-lvm    | linstor-a | LVM_THIN | drbdpool/thinpool | ... |
| thin-lvm    | linstor-b | LVM_THIN | drbdpool/thinpool | ... |
| thin-lvm    | linstor-c | LVM_THIN | drbdpool/thinpool | ... |
| thin-lvm    | linstor-d | LVM_THIN | drbdpool/thinpool | ... |
+--------------------------------------------------------------+

You could use LINSTOR to create a Thin LVM on linstor-d that’s 100GiB in size using the following commands:

# linstor resource-definition create rsc-1
# linstor volume-definition create rsc-1 100GiB
# linstor resource create --layer-list storage \
          --storage-pool thin-lvm linstor-d rsc-1

You should then see you have a new Thin LVM on linstor-d. You can extract the device path from LINSTOR by listing your linstor resources with the --machine-readable flag set:

# linstor --machine-readable resource list | grep device_path
            "device_path": "/dev/drbdpool/rsc-1_00000",

If you wanted to layer DRBD on top of this volume, which is the default --layer-list option in LINSTOR for ZFS or LVM backed volumes, you would use the following resource creation pattern instead:

# linstor resource-definition create rsc-1
# linstor volume-definition create rsc-1 100GiB
# linstor resource create --layer-list drbd,storage \
          --storage-pool thin-lvm linstor-d rsc-1

You would then see that you have a new Thin LVM backing a DRBD volume on linstor-d:

# linstor --machine-readable resource list | grep -e device_path -e backing_disk
            "device_path": "/dev/drbd1000",
            "backing_disk": "/dev/drbdpool/rsc-1_00000",

次の表は、どのレイヤーの後にどの子レイヤーが続くかを示しています。

[cols=”>1,<5″]

Layer

Child layer

DRBD

CACHE, WRITECACHE, NVME, LUKS, STORAGE

CACHE

WRITECACHE, NVME, LUKS, STORAGE

WRITECACHE

CACHE, NVME, LUKS, STORAGE

NVME

CACHE, WRITECACHE, LUKS, STORAGE

LUKS

STORAGE

STORAGE

One layer can only occur once in the layer list.
luks レイヤの必要条件についての情報はこのユーザガイドの暗号化ボリュームの節を参照ください。

3.5.1. NVMe-oF/NVMe-TCP LINSTOR レイヤ

NVMe-oF/NVMe-TCP allows LINSTOR to connect diskless resources to a node with the same resource where the data is stored over NVMe fabrics. This leads to the advantage that resources can be mounted without using local storage by accessing the data over the network. LINSTOR is not using DRBD in this case, and therefore NVMe resources provisioned by LINSTOR are not replicated, the data is stored on one node.

NVMe-oF only works on RDMA-capable networks and NVMe-TCP on networks that can carry IP traffic. You can use tools such as lshw or ethtool to verify the capabilities of your network adapters.

To use NVMe-oF/NVMe-TCP with LINSTOR the package nvme-cli needs to be installed on every node which acts as a satellite and will use NVMe-oF/NVMe-TCP for a resource. For example, on a DEB-based system, to install the package, enter the following command:

# apt install nvme-cli
If you are not on a DEB-based system, use the suitable command for installing packages on your operating system, for example, on SLES: zypper; on RPM-based systems: dnf.

NVMe-oF/NVMe-TCP を使用するリソースを作成するには、resource-definition を作成するときに追加のパラメータを指定する必要があります。

# linstor resource-definition create nvmedata -l nvme,storage
DRBDが使用されている場合、デフォルトでは -l(layer-stack) パラメータは drbd,storage に設定されています。 NVMeもDRBDも使用せずにLINSTORリソースを作成したい場合は、 -l パラメータを storage だけに設定する必要があります。

デフォルトの NVMe-oF の代わりに NVMe-TCP を使用するには、次のプロパティを設定する必要があります。

# linstor resource-definition set-property nvmedata NVMe/TRType tcp

プロパティ NVMe/TRType は、リソースグループまたはコントローラーレベルで設定することもできます。

Next, create the volume-definition for your resource:

# linstor volume-definition create nvmedata 500G

ノード上にリソースを作成する前に、データがローカルに格納される場所と、どのノードがネットワーク経由でそれにアクセスするかを確認します。

First, create the resource on the node where your data will be stored:

# linstor resource create alpha nvmedata --storage-pool pool_ssd

ネットワーク上でリソースデータにアクセスするノードでは、リソースをディスクレスとして定義する必要があります。

# linstor resource create beta nvmedata --nvme-initiator

これでリソース nvmedata をノードの1つにマウントできます。

ノードに複数のNICがある場合は、NVMe-of/NVME-TCP に対してそれらの間の経路を強制する必要があります。そうしないと、複数のNICで問題が発生する可能性があります。

3.5.2. 書き込みキャッシュレイヤー

DM-Writecache デバイスは、ストレージデバイスとキャッシュデバイスの2つのデバイスで構成されます。 LINSTORはそのような書き込みキャッシュデバイスをセットアップできますが、ストレージプールやキャッシュデバイスのサイズなどの追加情報が必要です。

# linstor storage-pool create lvm node1 lvmpool drbdpool
# linstor storage-pool create lvm node1 pmempool pmempool

# linstor resource-definition create r1
# linstor volume-definition create r1 100G

# linstor volume-definition set-property r1 0 Writecache/PoolName pmempool
# linstor volume-definition set-property r1 0 Writecache/Size 1%

# linstor resource create node1 r1 --storage-pool lvmpool --layer-list WRITECACHE,STORAGE

The two properties set in the examples are mandatory, but can also be set on controller level which would act as a default for all resources with WRITECACHE in their --layer-list. However, note that the Writecache/PoolName refers to the corresponding node. If the node does not have a storage pool named pmempool you will get an error message.

The four mandatory parameters required by DM-Writecache are either configured through a property or figured out by LINSTOR. The optional properties listed in the mentioned link can also be set through a property. Refer to linstor controller set-property --help for a list of Writecache/* property keys.

外部メタデータを使用するようにDRBDを設定しているときに --layer-list DRBD、WRITECACHE、STORAGE を使用すると、外部メタデータを保持するデバイスではなく、下位デバイスのみが書き込みキャッシュを使用します。

3.5.3. キャッシュレイヤー

LINSTOR は DM-Cache デバイスをセットアップすることもできます。これは前のセクションの DM-Writecache によく似ています。主な違いは、キャッシュデバイスが 3 つのデバイスで構成されていることです。1 つのストレージデバイス、1 つのキャッシュデバイス、1 つのメタデバイスです。LINSTOR プロパティは書き込みキャッシュのプロパティと非常に似ていますが、Cache 名前空間にあります。

# linstor storage-pool create lvm node1 lvmpool drbdpool
# linstor storage-pool create lvm node1 pmempool pmempool

# linstor resource-definition create r1
# linstor volume-definition create r1 100G

# linstor volume-definition set-property r1 0 Cache/CachePool pmempool
# linstor volume-definition set-property r1 0 Cache/Cachesize 1%

# linstor resource create node1 r1 --storage-pool lvmpool --layer-list CACHE,STORAGE
Writecache レイヤーを構成する Writecache/PoolName と異なり、Cache レイヤーの唯一の必須プロパティは Cache/CachePool です。これは、Cache レイヤーの Cache/MetaPool は個別に設定するか、またはデフォルトで Cache/CachePool の値を設定するからです。

Refer to linstor controller set-property --help for a list of Cache/* property keys and default values for omitted properties.

外部メタデータを使用するようにDRBDを設定しているときに --layer-list DRBD,CACHE,STORAGE を使用すると、外部メタデータを保持するデバイスではなく、下位デバイスのみがキャッシュを使用します。

3.5.4. ストレージレイヤー

ストレージレイヤーは、LVM、ZFS などのよく知られたボリュームマネージャーからの新しいデバイスを提供します。リソースがディスクレスである必要がある場合(そのタイプは専用の「ディスクレス」プロバイダータイプ)でも、すべてのレイヤーの組み合わせはストレージレイヤーに基づく必要があります。

For a list of providers with their properties refer to Storage Providers.

一部のストレージプロバイダー用に、LINSTOR には特別なプロパティがあります。

StorDriver/WaitTimeoutAfterCreate

If LINSTOR expects a device to appear after creation (for example after calls of lvcreate, zfs create,…​), LINSTOR waits per default 500ms for the device to appear. These 500ms can be overridden by this property.

StorDriver/dm_stats

If set to true LINSTOR calls dmstats create $device after creation and dmstats delete $device --allregions after deletion of a volume. Currently only enabled for LVM and LVM_THIN storage providers.

3.6. ストレージプロバイダー

LINSTOR にはいくつかのストレージプロバイダーがあります。最も使用されているのは LVM と ZFS です。しかし、これら 2 つのプロバイダーについても、シンプロビジョニングされた異なるサブタイプがすでに存在します。

  • Diskless: このプロバイダータイプには、Managing Network Interface Cards で説明されているように PrefNic などの LINSTOR プロパティで構成できるストレージプールがほとんどの場合必要です。

  • LVM / LVM-Thin: The administrator is expected to specify the LVM volume group or the thin-pool (in form of “LV/thinpool”) to use the corresponding storage type. These drivers support following properties for fine-tuning:

    • StorDriver/LvcreateOptions: このプロパティの値は、LINSTOR が実行するすべての lvcreate …​ 呼び出しの LINSTOR 実行に追加されます。

  • ZFS / ZFS-Thin: The administrator is expected to specify the ZPool that LINSTOR should use. These drivers support following properties for fine-tuning:

    • StorDriver/ZfscreateOptions: このプロパティの値は、LINSTOR が実行するすべての zfs create …​ 呼び出しの LINSTOR 実行に追加されます。

  • File / FileThin: Mostly used for demonstration or experiments. LINSTOR will reserve a file in a given directory and will configure a loop device on top of that file.

  • Exos [DEPRECATED]: This special storage provider is currently required to be run on a “special satellite”. Refer to the EXOS Integration chapter.

  • SPDK: 管理者は、LINSTOR が使用する論理ボリュームストアを指定する必要があります。このストレージプロバイダーの使用法は NVME Layer 使用を意味します。

    • Remote-SPDK: This special storage provider currently requires to be run on a “special satellite”. Refer to Remote SPDK Provider for more details.

3.6.1. リモート SPDK プロバイダー

リモート SPDK のストレージプールは、特別なサテライトでのみ作成できます。最初に次のコマンドを使用して新しいサテライトを開始する必要があります。

$ linstor node create-remote-spdk-target nodeName 192.168.1.110

これにより、コントローラーと同じマシンで実行されている新しいサテライトインスタンスが開始されます。この特別なサテライトは、リモート SPDK プロキシに対してすべての REST ベースの RPC 通信を実行します。LINSTOR コマンドのヘルプメッセージが示すように、管理者はこの特別なサテライトを作成するときに設定を追加したい場合があります。

$ linstor node create-remote-spdk-target -h
usage: linstor node create-remote-spdk-target [-h] [--api-port API_PORT]
                                              [--api-user API_USER]
                                              [--api-user-env API_USER_ENV]
                                              [--api-pw [API_PW]]
                                              [--api-pw-env API_PW_ENV]
                                              node_name api_host

The difference between the --api-* and their corresponding --api-\*-env versions is that the version with the -env ending will look for an environment variable containing the actual value to use whereas the --api-\* version directly take the value which is stored in the LINSTOR property. Administrators might not want to save the --api-pw in plain text, which would be clearly visible using commands like linstor node list-property <nodeName>.

特別なサテライトが稼働し始めたら、実際のストレージプールを作成できます。

$ linstor storage-pool create remotespdk -h
usage: linstor storage-pool create remotespdk [-h]
                                              [--shared-space SHARED_SPACE]
                                              [--external-locking]
                                              node_name name driver_pool_name

node_name は自明ですが、name は LINSTOR ストレージプールの名前であり、 driver_pool_name は SPDK 論理ボリュームストアを指定します。

Once this remotespdk storage pool is created the remaining procedure is quite similar as using NVMe: First the target has to be created by creating a simple “diskful” resource followed by a second resource having the --nvme-initiator option enabled.

3.7. ネットワークインターフェイスカードの管理

LINSTOR can deal with multiple network interface cards (NICs) in a machine. They are called “net interfaces” in LINSTOR speak.

サテライト ノードが作成されると、最初のネット インターフェイスが「default」という名前で暗黙的に作成されます。サテライトノードを作成するときに、node create コマンドの –interface-name オプションを使用して別の名前を付けることができます。

既存のノードの場合、次のように追加のネットインターフェイスが作成されます。

# linstor node interface create node-0 10G_nic 192.168.43.231

ネットインターフェイスは IP アドレスのみで識別されます。名前は任意であり、Linux で使用される NIC 名とは関係ありません。次に、ネットインターフェイスをノードに割り当てて、ノードの DRBD トラフィックが対応する NIC を介してルーティングされるようにします。

# linstor node set-property node-0 PrefNic 10G_nic
ストレージプールに「PrefNic」プロパティを設定することもできます。ストレージプールを使用するリソースからの DRBD トラフィックは、対応する NIC 経由でルーティングされます。ただし、ここで注意が必要です。ディスクレスストレージを必要とする DRBD リソース (たとえば、DRBD クォーラムの目的でタイブレーカーの役割を果たしているディスクレス ストレージ) は、「default」ネットインターフェイスの「PrefNic」プロパティを設定するまで、デフォルトのサテライトノードのネットインターフェイスを通過します。セットアップが複雑になる可能性があります。ノードレベルで「PrefNic」プロパティを設定する方がはるかに簡単で安全です。これにより、ディスクレスストレージプールを含む、ノード上のすべてのストレージプールが優先 NIC を使用します。

If you need to add an interface for only controller-satellite traffic, you can add an interface using the above node interface create command. Then you modify the connection to make it the active controller-satellite connection. For example, if you added an interface named 1G-satconn on all nodes, after adding the interface, you can then tell LINSTOR to use this interface for controller-satellite traffic by entering the following command:

# linstor node interface modify node-0 1G-satconn --active

You can verify this change by using the linstor node interface list node-0 command. Output from the command should show that the StltCon label applies to the 1G-satconn interface.

While this method routes DRBD traffic through a specified NIC, it is not possible through linstor commands only, to route LINSTOR controller-client traffic through a specific NIC, for example, commands that you issue from a LINSTOR client to the controller. To achieve this, you can either:

  • Specify a LINSTOR controller by using methods outlined in LINSTORクライアントの使用 and have the only route to the controller as specified be through the NIC that you want to use for controller-client traffic.

  • ip routeiptables などの Linux ツールを使用して、LINSTOR クライアントコントローラートラフィックのポート番号 3370 をフィルタリングし、特定の NIC を介してルーティングする。

3.7.1. LINSTORによる複数のDRBDパスの作成

DRBDのセットアップに 複数のネットワークパスを使用するには、PrefNic プロパティは十分ではありません。代わりに、次のように linstor node interfacelinstor resource-connection path コマンドを使用する必要があります。

# linstor node interface create alpha nic1 192.168.43.221
# linstor node interface create alpha nic2 192.168.44.221
# linstor node interface create bravo nic1 192.168.43.222
# linstor node interface create bravo nic2 192.168.44.222

# linstor resource-connection path create alpha bravo myResource path1 nic1 nic1
# linstor resource-connection path create alpha bravo myResource path2 nic2 nic2

The first four commands in the example define a network interface (nic1 and nic2) for each node (alpha and bravo) by specifying the network interface’s IP address. The last two commands create network path entries in the DRBD .res file that LINSTOR generates. This is the relevant part of the resulting .res file:

resource myResource {
  ...
  connection {
    path {
      host alpha address 192.168.43.221:7000;
      host bravo address 192.168.43.222:7000;
    }
    path {
      host alpha address 192.168.44.221:7000;
      host bravo address 192.168.44.222:7000;
    }
  }
}
ノードインタフェースの作成時にLINSTOR衛星トラフィックに使用するポート番号を指定できますが、DRBDリソース接続パスの作成時には、このポート番号は無視されます。代わりに、コマンドはポート番号を動的に割り当てます。
How Adding a New DRBD Path Affects the Default Path

The NIC that is first in order on a LINSTOR satellite node is named the default net interface. DRBD traffic traveling between two nodes that do not have an explicitly configured resource connection path will take an implicit path that uses the two nodes’ default net interfaces.

When you add a resource connection path between two nodes for a DRBD-backed resource, DRBD traffic between the two nodes will use this new path only, although a default network interface will still exist on each node. This can be significant if your new path uses different NICs than the implicit default path.

To use the default path again, in addition to any new paths, you will need to explicitly add it. For example:

# linstor resource-connection path create alpha bravo myResource path3 default default

Although the newly created path3 uses net interfaces that are named default on the two nodes, the path itself is not a default path because other paths exist, namely path1 and path2. The new path, path3, will just act as a third possible path, and DRBD traffic and path selection behavior will be as described in the next section.

Multiple DRBD Paths Behavior

The behavior of a multiple DRBD paths configuration will be different depending on the DRBD transport type. From the DRBD User’s Guide[4]:

“The TCP transport uses one path at a time. If the backing TCP connections get dropped, or show timeouts, the TCP transport implementation tries to establish a connection over the next path. It goes over all paths in a round-robin fashion until a connection gets established.

“The RDMA transport uses all paths of a connection concurrently and it balances the network traffic between the paths evenly.”

3.8. 暗号化ボリューム

LINSTORはDRBDボリュームの暗号化を透過的に扱うことができます。dm-cryptがストレージデバイスから提供されるストレージを暗号化するのに使われます。

To use dm-crypt, verify that cryptsetup is installed before you start the satellite.

暗号化の基本的な手順は以下になります。

  1. マスターパスフレーズを作成する

  2. Add luks to the layer-list. Note that all plugins (e.g., Proxmox) require a DRBD layer as the top most layer if they do not explicitly state otherwise.

  3. コントローラが再起動した後はマスターパスフレーズを再入力する

3.8.1. 暗号化のコマンド

以下にコマンドの詳細を示します。

Before LINSTOR can encrypt any volume a master passphrase needs to be created. This can be done with the LINSTOR client.

# linstor encryption create-passphrase

crypt-create-passphrase はマスターパスフレーズを初期化するためにユーザの入力を待ちます。

マスターパスフレーズを変更したい場合は以下のコマンドで行います。

# linstor encryption modify-passphrase

resource-definition またはリソース自体を作成するときに luks レイヤーを追加できますが、前者の方法は、resource-definition から作成されたすべてのリソースに自動的に適用されるため、推奨されます。

# linstor resource-definition create crypt_rsc --layer-list luks,storage

マスターパスフェーズを入力する(コントローラを再起動した後)には以下を使用します。

# linstor encryption enter-passphrase
linstor-controllerが再起動したときは、コントローラにマスターパスフェーズを常に入力する必要があります。そうしないとLINSTORは暗号化されたボリュームをオープンしたり作成したりできません。

3.8.2. 自動パスフレーズ

マスターパスフレーズの作成と再入力のプロセスを自動化することは可能です。

これを使用するには、 MASTER_PASSPHRASE という名前の環境変数、またはマスターパスフレーズを含む /etc/linstor/linstor.toml のエントリを作成する必要があります。

linstor.toml は以下のようになります。

[encrypt]
passphrase="example"

If either one of these is set, then every time the controller starts it will check whether a master passphrase already exists. If there is none, it will create a new master passphrase as specified. Otherwise, the controller enters the passphrase.

If a master passphrase is already configured, and it is not the same one as specified in the environment variable or linstor.toml, the controller will be unable to re-enter the master passphrase and react as if the user had entered a wrong passphrase. This can only be resolved through manual input from the user, using the same commands as if the controller was started without the automatic passphrase.
マスターパスフレーズが環境変数と linstor.toml の両方に設定されている場合、linstor.toml からのマスターパスフレーズのみが使用されます。

3.9. クラスター状態の確認

LINSTOR provides various commands to check the state of your cluster. These commands start with a ‘list’ precursor, after which, various filtering and sorting options can be used. The ‘–groupby’ option can be used to group and sort the output in multiple dimensions.

# linstor node list
# linstor storage-pool list --groupby Size

3.10. ノードの退避

LINSTOR コマンド node evacuate を使用して、ノードのリソースを退避することができます。たとえば、クラスターからノードを削除する準備をしていて、ノードのリソースをクラスター内の他のノードに移動する必要がある場合です。ノードの退避に成功すると、ノードの LINSTOR ステータスは “Online” ではなく “EVACUATE” と表示され、LINSTOR リソースを持ちません。

LINSTOR が Kubernetes や OpenNebula などの別の環境内にデプロイされているノードを退避する場合は、リソースを退避する前に、ノードの LINSTOR でサポートされているワークロードをクラスター内の別のノードに移動する必要があります。Kubernetes 環境については Kubernetes でのノードの退避 を参照してください。OpenNebula の LINSTOR ノードの場合、ノードのリソースを退避する前に、ノードがホストする OpenNebula LINSTOR ベースの仮想マシンをクラスター内の別のノードにライブマイグレードする必要があります。

次の手順でノードを退避します。

  1. 退避するノード上のリソースが “InUse” であるかどうかを確認します。 “InUse” ステータスは、リソースが DRBD Primary 状態です。ノードを正常に退避させるには、ノード上のリソースを “InUse” にしないでください。そうしないと、LINSTOR は退避中にノードから “InUse” リソースを削除できません。

  2. linstor node evacuate <node_name> を実行します。退避ノード上のリソースに適した代替ノードがない場合は、警告が表示されます。たとえば、3 つのノードがあり、1 つを退避したいが、リソースグループで配置数が 3 に設定されている場合、ノードが退避ノードからリソースを削除できないという警告が表示されます。

  3. ノードの linstor node list のステータスが、 “Online” ではなく、 “EVACUATE” であることを確認してください。

  4. linstor resource list コマンドを使用して、ノード上のリソースの “State” ステータスを確認します。ノードのリソース内のデータセットのサイズに応じて、しばらく同期アクティビティが表示されるはずです。

  5. linstor resource list --nodes <node_name> コマンドを使用して、ノード上の残りのリソースを一覧表示します。残っている場合は、同期が完了するのを待っているだけかどうかを確認します。

  6. linstor resource list コマンドを使用して、ノードにリソースがないことを確認します。

  7. linstor node delete node_name> コマンドを使用して、クラスタからノードを削除します。

3.10.1. 複数のノードの退避

Some evacuation cases might need special planning. For example, if you are evacuating more than one node, you can exclude the nodes from participating in LINSTOR’s resource autoplacer. You can do this by using the following command on each node that you want to evacuate:

# linstor node set-property <node_name> AutoplaceTarget false

これにより、退避しようとしているノードから、退避を計画している別のノードに LINSTOR がリソースを配置することがなくなります。

3.10.2. 退避ノードの復元

If you already ran a node evacuate command that has either completed or still has resources in an “Evacuating” state, you can remove the “Evacuating” state from a node by using the node restore command. This will work if you have not yet run a node delete command.

After restoring the node, you should use the node set-property <node_name> AutoplaceTarget true command, if you previously set the AutoplaceTarget property to “false”. This way, LINSTOR can again place resources onto the node automatically, to fulfill placement count properties that you might have set for resources in your cluster.

If LINSTOR has already evacuated resources when running a node restore command, evacuated resources will not automatically return to the node. If LINSTOR is still in the process of evacuating resources, this process will continue until LINSTOR has placed the resources on other nodes. You will need to manually “move” the resources that were formerly on the restored node. You can do this by first creating the resources on the restored node and then deleting the resources from another node where LINSTOR might have placed them. You can use the resource list command to show you on which nodes your resources are placed.

3.11. Working With Resource Snapshots and Backups

LINSTOR supports taking snapshots of resources that are backed by thin LVM or ZFS storage pools. By creating and shipping snapshots, you can back up LINSTOR resources to other storage: either to S3 storage or to storage in another (or the same!) LINSTOR cluster. The following sub-sections describe various aspects of working with snapshots and backups.

3.11.1. スナップショットの作成

Assuming a resource definition named ‘resource1’ which has been placed on some nodes, you can create a snapshot of the resource by entering the following command:

# linstor snapshot create resource1 snap1

This will create a snapshot on all nodes where the resource is present. LINSTOR will create consistent snapshots, even when the resource is in active use.

Setting the resource definition property AutoSnapshot/RunEvery LINSTOR will automatically create snapshots every X minutes. The optional property AutoSnapshot/Keep can be used to clean up old snapshots which were created automatically. No manually created snapshot will be cleaned up or deleted. If AutoSnapshot/Keep is omitted (or ⇐ 0), LINSTOR will keep the last 10 snapshots by default.

# linstor resource-definition set-property AutoSnapshot/RunEvery 15
# linstor resource-definition set-property AutoSnapshot/Keep 5

3.11.2. スナップショットの復元

The following steps restore a snapshot to a new resource. This is possible even when the original resource has been removed from the nodes where the snapshots were taken.

最初にスナップショットに一致するボリュームをもつリソースを定義します。

# linstor resource-definition create resource2
# linstor snapshot volume-definition restore --from-resource resource1 \
  --from-snapshot snap1 --to-resource resource2

At this point, additional configuration can be applied if necessary. Then, when ready, create resources based on the snapshots:

# linstor snapshot resource restore --from-resource resource1 \
  --from-snapshot snap1 --to-resource resource2

This will place the new resource on all nodes where the snapshot is present. The nodes on which to place the resource can also be selected explicitly; see the help (linstor snapshot resource restore --help).

3.11.3. スナップショットへのロールバック

LINSTOR can roll a resource back to a snapshot state. The resource must not be in use. That is, the resource must not be mounted on any nodes. If the resource is in use, consider whether you can achieve your goal by restoring the snapshot instead.

ロールバックは次のようにして実行します。

# linstor snapshot rollback resource1 snap1

A resource can only be rolled back to the most recent snapshot. To roll back to an older snapshot, first delete the intermediate snapshots.

3.11.4. スナップショットの削除

スナップショットの削除は以下のコマンドで行います。

# linstor snapshot delete resource1 snap1

3.12. Shipping Snapshots

Snapshots can be shipped between nodes in the same LINSTOR cluster, between different LINSTOR clusters, or to S3 storage such as Amazon S3 or min.io.

スナップショットを送受信するサテライトには、次のツールをインストールします。

  • zstd はデータが転送される前に圧縮するために必要です。

  • thin-send-recv is needed to ship data when using LVM thin-provisioned volumes

You need to restart the satellite node (or nodes) after installing these tools, otherwise LINSTOR will not be able to use them.

3.12.1. Working With Snapshot Shipping Remotes

In a LINSTOR cluster, a snapshot shipping target is called a remote. Currently, there are two different types of remotes that you can use when shipping snapshots: LINSTOR remotes and S3 remotes. LINSTOR remotes are used to ship snapshots to a different LINSTOR cluster or within the same LINSTOR cluster. S3 remotes are used to ship snapshots to AWS S3, min.io or any other service using S3 compatible object storage. A shipped snapshot on a remote is also called a backup.

Since a remote needs to store sensitive data, such as passwords, it is necessary to have encryption enabled whenever you want to use a remote in any way. How to set up LINSTOR’s encryption is described here.
Creating an S3 Remote

To create an S3 remote, LINSTOR will need to know the endpoint (that is, the URL of the target S3 server), the name of the target bucket, the region the S3 server is in, and the access-key and secret-key used to access the bucket. If the command is sent without adding the secret-key, a prompt will pop up to enter it in. The command should look like this:

# linstor remote create s3 myRemote s3.us-west-2.amazonaws.com \
  my-bucket us-west-2 admin password
Usually, LINSTOR uses the endpoint and bucket to create an URL using the virtual-hosted-style for its access to the given bucket (for example my-bucket.s3.us-west-2.amazonaws.com). Should your setup not allow access this way, change the remote to path-style access (for example s3.us-west-2.amazonaws.com/my-bucket) by adding the --use-path-style argument to make LINSTOR combine the parameters accordingly.
Creating a LINSTOR Remote

To create a LINSTOR remote, you only need to specify a name for the remote and the URL or IP address of the controller of the target cluster. An example command is as follows:

# linstor remote create linstor myRemote 192.168.0.15

To ship a snapshot between two LINSTOR clusters, or within the same LINSTOR cluster, besides creating a remote on the source cluster that points to the target cluster, on the target cluster, you also need to create a LINSTOR remote that points to the source cluster. This is to prevent your target cluster from accepting backup shipments from unknown sources. You can create such a remote on your target cluster by specifying the cluster ID of the source cluster in an additional argument to the remote create command. Refer to Shipping a Snapshot of a Resource to a LINSTOR Remote for details.

Listing, Modifying, and Deleting Remotes

To see all the remotes known to the local cluster, use linstor remote list. To delete a remote, use linstor remote delete myRemoteName. If you need to modify an existing remote, use linstor remote modify to change it.

Specifying a LINSTOR Passphrase When Creating a Remote

When the snapshot that you want to ship contains a LUKS layer, the remote on the target cluster also needs the passphrase of the source cluster set when you create the remote. This is because the LINSTOR passphrase is used to encrypt the LUKS passphrase. To specify the source cluster’s LINSTOR passphrase when you create a LINSTOR remote on the target cluster, enter:

$ linstor --controllers <TARGET_CONTROLLER> remote create linstor \
--cluster-id <SOURCE_CLUSTER_ID> --passphrase <SOURCE_CONTROLLER_PASSPHRASE> <NAME> <URL>

For LINSTOR to LINSTOR snapshot shipping, you must also create a LINSTOR remote on the source cluster. For simplicity sake, although not strictly necessary, you can specify the target cluster’s LINSTOR passphrase when you create a LINSTOR remote for the target cluster on the source cluster, before you ship backups or snapshots. On the source cluster, enter:

$ linstor --controllers <SOURCE_CONTROLLER> remote create linstor \
--cluster-id <TARGET_CLUSTER_ID> --passphrase <TARGET_CONTROLLER_PASSPHRASE> <NAME> <URL>
If you are specifying a LINSTOR controller node (perhaps because you have a highly available controller), when creating a remote, you can specify the controller either by an IP address or a resolvable hostname.

3.12.2. Shipping Snapshots to an S3 Remote

To ship a snapshot to an S3 remote, that is, to create a backup of a resource on an S3 remote, all you need to do is to specify an S3 remote that the current cluster can reach and then specify the resource that should be shipped. The following command will create a snapshot of a resource, myRsc, and ship it to the given S3 remote, myRemote:

# linstor backup create myRemote myRsc

If this isn’t the first time you shipped a backup of this resource (to that remote) and the snapshot of the previous backup hasn’t been deleted yet, an incremental backup will be shipped. To force the creation of a full backup, add the --full argument to the command. Getting a specific node to ship the backup is also possible by using --node myNode, but if the specified node is not available or only has the resource diskless, a different node will be chosen.

3.12.3. Shipping Snapshots to a LINSTOR Remote

You can ship a snapshot between two LINSTOR clusters by specifying a LINSTOR remote. Snapshot shipping to a LINSTOR remote also requires at least one diskful resource on the source side (where you issue the shipping command).

Creating a Remote for a LINSTOR Target Cluster

Before you can ship a snapshot to a LINSTOR target cluster, on the target side, you need to create a LINSTOR remote and specify the cluster ID of the source cluster as the remote:

$ linstor remote create linstor --cluster-id <SOURCE_CLUSTER_ID> <NAME> <URL>
If you do not specify the cluster ID of your source cluster when you create a LINSTOR remote on your target cluster, you will receive an “Unknown Cluster” error when you try to ship a backup. To get the cluster ID of your source cluster, you can enter the command linstor controller list-properties|grep -i cluster from the source cluster.
If you might be creating and shipping snapshots of resources that have a LUKS layer, then you also need to specify a passphrase when creating a remote, as described in the Specifying a LINSTOR Passphrase When Creating a Remote section.

In the remote create command shown above, <NAME> is an arbitrary name that you specify to identify the remote. <URL> is either the IP address of the source (remote) LINSTOR controller or its resolvable hostname. If you have configured a highly available LINSTOR controller, use its virtual IP address (VIP) or the VIP’s resolvable name.

Shipping a Snapshot of a Resource to a LINSTOR Remote

To create a snapshot of a resource from your source LINSTOR cluster and ship it to your target LINSTOR cluster, enter the following command:

# linstor backup ship myRemote localRsc targetRsc

This command essentially creates a backup of your local resource on your target LINSTOR remote.

Additionally, you can use --source-node to specify which node should send and --target-node to specify which node should receive the backup. In case those nodes are not available, the LINSTOR controller will choose different nodes automatically.

targetRsc がすでにリモートクラスタにデプロイされているリソースである場合、 localRsc のバックアップ転送のスナップショットはリモートクラスタに転送されますが、リモートクラスタでは復元されません。 linstor backup ship コマンドで --download-only オプションを指定した場合も同様です。

3.12.4. Snapshot Shipping Within a Single LINSTOR Cluster

If you want to ship a snapshot inside the same cluster, you just need to create a LINSTOR remote that points to the local controller. To do this if you are logged into your LINSTOR controller, for example, enter the following command:

# linstor remote create linstor --cluster-id <CLUSTER_ID> <NAME> localhost

You can then follow the instructions to ship a snapshot to a LINSTOR remote.

3.12.5. Listing Backups on a Remote

To show which backups exist on a specific S3 remote, use linstor backup list myS3Remote. A resource-name can be added to the command as a filter to only show backups of that specific resource by using the argument --resource myRsc. If you use the --other argument, only entries in the bucket that LINSTOR does not recognize as a backup will be shown. LINSTOR always names backups in a certain way, and if an item in the remote is named according to this schema, it is assumed that it is a backup created by LINSTOR – so this list will show everything else.

To show which backups exist on a LINSTOR remote, use the linstor snapshot list command on your LINSTOR target cluster.

3.12.6. Deleting Backups on a Remote

バックアップの削除に関しては、いくつかのオプションがあります。

  • linstor backup delete all myRemote : このコマンドは、バックアップであると認識されている、つまり期待されるネーミングスキーマに適合している限り、指定されたリモート上のすべての S3 オブジェクトを削除します。現在のクラスターによって作成されたバックアップのみを削除するオプション --cluster があります。

  • linstor backup delete id myRemote my-rsc_back_20210824_072543: This command deletes a single backup from the given remote – namely the one with the given id, which consists of the resource-name, the automatically generated snapshot-name (back_timestamp) and, if set, the backup-suffix. The option --prefix lets you delete all backups starting with the given id. The option --cascade deletes not only the specified backup, but all other incremental backups depending on it.

  • linstor backup delete filter myRemote …​: This command has a few different arguments to specify a selection of backups to delete. -t 20210914_120000 will delete all backups made before 12 o’clock on the 14th of September, 2021. -n myNode will delete all backups uploaded by the given node. -r myRsc will delete all backups with the given resource name. These filters can be combined as needed. Finally, --cascade deletes not only the selected backup(s), but all other incremental backups depending on any of the selected backups.

  • linstor backup delete s3key myRemote randomPictureInWrongBucket: This command will find the object with the given S3-key and delete it – without considering anything else. This should only be used to either delete non-backup items from the remote, or to clean up a broken backup that you are no longer able to delete by other means. Using this command to delete a regular, working backup will break that backup, so beware!

--cascade オプションを持つすべてのコマンドは、明示的に指定しない限り、それに依存する増分バックアップを削除しません。
すべての linstor backup delete …​ コマンドには --dry-run オプションがあり、削除されるすべての S3 オブジェクトのリストが表示されます。これを使用して、削除してはならないものが誤って削除されることを防ぐことができます。

3.12.7. Restoring Backups From a Remote

Maybe the most important task after creating a backup is restoring it. To restore a backup from an S3 remote, you can use the linstor backup restore command.

With this command, you specify the name of the S3 remote to restore a backup from, the name of the target node, and a target resource to restore to on the node. If the resource name does not match an existing resource definition, LINSTOR will create the resource definition and resource.

Additionally, you need to specify the name of the resource on the remote that has the backup. You do this by using either the --resource or --id arguments but not both.

By using the --resource option, you can restore the latest backup of the resource that you specify, for example:

# linstor backup restore myRemote myNode targetRsc --resource sourceRsc

By using the --id option, you can restore the exact backup that you specify, for example, to restore a backup of a resource other than the most recent. To get backup IDs, refer to Listing Backups on a Remote.

# linstor backup restore myRemote myNode targetRsc --id sourceRsc_back_20210824_072543
When shipping a snapshot to a LINSTOR (not S3) remote, the snapshot is restored immediately to the specified resource on the target (remote) cluster, unless you use the --download-only option or else if the target resource has at least one replica.

When you restore a backup to a resource, LINSTOR will download all the snapshots from the last full backup of the resource, up to the backup that you specified (when using the --id option) or up to the latest backup (when using the --resource option). After downloading these snapshots, LINSTOR restores the snapshots into a new resource. You can skip restoring the snapshots into a new resources by adding the --download-only option to your backup restore command.

LINSTOR can download backups to restore from any cluster, not just the one that uploaded them, provided that the setup is correct. Specifically, the target resource cannot have any existing resources or snapshots, and the storage pool(s) used need to have the same storage providers. If the storage pool(s) on the target node has the exact same name(s) as on the cluster the backup was created on, no extra action is necessary. If the nodes have different storage pool names, then you need to use the --storpool-rename option with your backup restore command. This option expects at least one oldname=newname pair. For every storage pool of the original backup that is not named in that list, LINSTOR assumes that the storage pool name is exactly the same on the target node.

To find out exactly which storage pools you will need rename, and how big the download and the restored resource will be, you can use the command linstor backup info myRemote …​. Similar to the restore command, you need to specify either the --resource or --id option. When used with the backup info command, these options have the same restrictions as when used with the backup restore command. To show how much space will be left over in the local storage pools after a restore, you can add the argument -n myNode. The same as with an actual restore operation, the backup info command assumes that the storage pool names are exactly the same on the target and source nodes. If that is not the case, you can use the --storpool-rename option, just as with the restore command.

Restoring Backups That Have a LUKS Layer

If the backup to be restored includes a LUKS layer, the --passphrase argument is required. With it, the passphrase of the original cluster of the backup needs to be set so that LINSTOR can decrypt the volumes after download and re-encrypt them with the local passphrase.

Controlling How Many Shipments Are Active at the Same Time

There might be cases where an automated task (be it LINSTOR’s scheduled shipping or an external tool) starts too many shipments at once, leading to an overload of the network or some of the nodes sending the backups.

In a case such as this, the solution is to reduce the amount of shipments that can happen at the same time on the same node. This is done by using the property BackupShipping/MaxConcurrentBackupsPerNode. This property can be set either on the controller or on a specific node.

The expected value for this property is a number. Setting it to any negative number will be interpreted as “no limit”, while setting it to zero will result in this specific node not being eligible to ship any backups – or completely disabling backup shipping if the property is set to 0 on the controller.

Any other positive number is treated as a limit of concurrently active shipments per node. To determine which node will send a backup shipment, LINSTOR uses the following logic in the order shown:

  1. The node specified in the command (--source-node for shipping to another LINSTOR cluster, --node for shipping to S3 compatible storage) will ship the backup.

  2. The node that has the most available backup slots will ship the backup.

  3. If no node has an available backup slot, the shipment will be added to a queue and started as soon as a different shipment has finished which leads to a backup slot becoming available.

同じクラスター内へのスナップショットの転送

Before you can ship a snapshot to a different node within the same LINSTOR cluster, you need to create a LINSTOR remote object that specifies your LINSTOR cluster’s ID. Refer to the [s-shipping_snapshots-l2l] section for instructions on how to get your LINSTOR cluster’s ID and create such a remote. An example command would be:

# linstor remote create linstor self 127.0.0.1 --cluster-id <LINSTOR_CLUSTER_ID>
self is a user-specified, arbitrary name for your LINSTOR remote. You can specify a different name if you want.

Both, the source and the target node have to have the resource for snapshot shipping deployed. Additionally, the target resource has to be deactivated.

# linstor resource deactivate nodeTarget resource1
You cannot reactivate a resource with DRBD in its layer-list after deactivating such a resource. However, a successfully shipped snapshot of a DRBD resource can still be restored into a new resource. To manually start the snapshot shipping, use:
# linstor backup ship self localRsc targetRsc
The snapshot ship command is considered deprecated and any bugs found with it will not be fixed. Instead, to ship a snapshot in the same LINSTOR cluster, use the backup ship command, as shown in the example above, with a remote pointing to your local controller. For more details about configuring a LINSTOR cluster as a remote, see the Shipping Snapshots to a LINSTOR Remote section.

By default, the snapshot shipping feature uses TCP ports from the range 12000-12999. To change this range, the property SnapshotShipping/TcpPortRange, which accepts a to-from range, can be set on the controller:

# linstor controller set-property SnapshotShipping/TcpPortRange 10000-12000

A resource can also be periodically shipped. To accomplish this, it is mandatory to set the properties SnapshotShipping/TargetNode and SnapshotShipping/RunEvery on the resource-definition. SnapshotShipping/SourceNode can also be set, but if omitted LINSTOR will choose an active resource of the same resource-definition.

増分スナップショット転送を可能にするには、LINSTOR は少なくとも最後に転送されたスナップショットをターゲットノードに保持する必要があります。プロパティ SnapshotShipping/Keep は、LINSTOR が保持する必要があるスナップショットの数を指定するために使用できます。プロパティが設定されていない(または 0 以下)の場合、LINSTOR はデフォルトで最新の10個のスナップショットを保持します。

# linstor resource-definition set-property resource1 SnapshotShipping/TargetNode nodeTarget
# linstor resource-definition set-property resource1 SnapshotShipping/SourceNode nodeSource
# linstor resource-definition set-property resource1 SnapshotShipping/RunEvery 15
# linstor resource-definition set-property resource1 SnapshotShipping/Keep 5

3.13. 計画されたバックアップの配信

LINSTOR Controller バージョン 1.19.0 以降、LINSTOR クライアントバージョン 1.14.0 以降では、配置された LINSTOR リソースのスケジュールバックアップの配信を設定することができます。

定期的なバックアップの配信は、3つのパートで構成されています。

  • バックアップと配信を行いたい1つまたは複数のLINSTORリソースからなるデータセット

  • バックアップの配信先(他のLINSTORクラスタまたはS3インスタンス)

  • バックアップを発行するタイミングを定義するスケジュール

LINSTORバックアップの配信は、LVMとZFSストレージプールにバックアップされたLINSTORリソースに対してのみ機能します。これは、LINSTORでスナップショットをサポートするストレージプールタイプであるためです。

3.13.1. バックアップの配信スケジュールを作成する

LINSTORクライアントの schedule create コマンドを使用してバックアップ出荷スケジュールを作成し、cron 構文を使用してバックアップ配信の頻度を定義します。また、スケジュールに名前を付けるオプションを設定し、障害発生時の対応、ローカルとリモートのバックアップコピーの数、増分バックアップの配信をスケジュールするかどうかなど、バックアップ配信の様々な側面を定義する必要があります。

最低限、このコマンドはバックアップ出荷スケジュールを作成するために、スケジュール名とフルバックアップcronスキーマを必要とします。コマンドの例は次のようになります。

# linstor schedule create \
  --incremental-cron '* * * * *' \ (1)
  --keep-local 5 \ (2)
  --keep-remote 4 \ (3)
  --on-failure RETRY \ (4)
  --max-retries 10 \ (5)
  <schedule_name> \ (6)
  '* * * * *' # full backup cron schema (7)
cronスキーマは一重引用符または二重引用符で囲んでください。
1 指定された場合、増分cronスキーマは、増分バックアップを作成し、出荷する頻度を記述します。新しいインクリメンタルバックアップは、最新のフルバックアップに基づいて行われます。
2 The --keep-local option allows you to specify how many snapshots that a full backup is based upon should be kept at the local backup source. If unspecified, all snapshots will be kept. [OPTIONAL]
3 --keep-remote オプションは、リモートバックアップ先で保持するフルバックアップの数を指定することができます。このオプションはS3リモートバックアップ先でのみ機能します。なぜなら、クラスタノードが他のクラスタ内のノードからバックアップを削除することを許可したくないからです。削除されたフルバックアップに基づくすべての増分バックアップも、リモートディスティネーションで削除されます。指定しない場合、--keep-remote オプションはデフォルトで “all” になります。[OPTIONAL]
4 スケジュールされたバックアップの出荷が失敗した場合に、”RETRY “または “SKIP “のどちらを行うかを指定します。SKIP” が指定された場合、LINSTORは失敗を無視し、次に予定されているバックアップの配送を続行します。RETRY” が指定された場合、LINSTORは60秒待って、再度バックアップの配送を試みます。LINSTOR の schedule create コマンドは --on-failure オプションが与えられないと、デフォルトで “SKIP” になります。[OPTIONAL]
5 バックアップの出荷が失敗し、--on-failure RETRY オプションが指定された場合、バックアップの出荷を再試行する回数です。このオプションがない場合、LINSTORコントローラはスケジュールされたバックアップの出荷を、成功するまで無期限に再試行します。[OPTIONAL]
6 スケジュールリスト、修正、削除、有効、無効コマンドで後で参照できるように、バックアップスケジュールに付ける名前です。[REQUIRED]
7 このcronスキーマは、LINSTORがどの程度の頻度でスナップショットを作成し、フルバックアップを出荷するかを記述しています。
増分cronスキーマを指定した場合、フルcronスキーマと重複しているため、両方のバックアップが同時に行われる場合、LINSTORはフルバックアップのみを作成し、出荷します。例えば、3時間ごとにフルバックアップを行い、1時間ごとに増分バックアップを行うように指定した場合、3時間ごとにLINSTORはフルバックアップのみを作成し出荷します。この理由から、増分バックアップとフルバックアップの出荷スケジュールの両方に同じcronスキーマを指定しても、増分バックアップは決して作成されないため、意味がありません。

3.13.2. バックアップの配信スケジュールを変更する

LINSTORクライアントの schedule modify コマンドを使用すると、バックアップの出荷スケジュールを変更することができます。 このコマンドの構文は schedule create コマンドの構文と同じです。schedule modify` コマンドで指定する名前は、すでに存在するバックアップスケジュールでなければなりません。コマンドのオプションで指定しないものは、既存の値を保持します。keep-local` または keep-remote オプションをデフォルト値に戻したい場合は、”all “に設定します。max-retries` オプションをデフォルトの値に戻したい場合は、”forever” に設定します。

3.13.3. Configuring the Number of Local Snapshots and Remote Backups to Keep

物理ストレージは無限ではなく、リモートストレージにはコストがかかるため、スナップショットやバックアップの保存数に制限を設けるとよいでしょう。

Both the --keep-remote and --keep-local options deserve special mention as they have implications beyond what might be obvious. Using these options, you specify how many snapshots or full backups should be kept, either on the local source or the remote destination.

Configuring the Keep-Local Option

例えば、--keep-local=2 オプションが設定されている場合、バックアップの出荷スケジュールは、最初の実行時に、フルバックアップのためのスナップショットを作成することになります。次にスケジュールされたフルバックアップの出荷では、フルバックアップのための2番目のスナップショットを作成します。次のスケジュールされたフルバックアップの出荷では、フルバックアップのための3番目のスナップショットが作成されます。しかし今回は、正常に完了すると、LINSTORは最初の(最も古い)フルバックアップの出荷用スナップショットを削除します。もしスナップショットがこのフルバックアップに基づく増分バックアップのために作成された場合、それらはローカルソースノードから削除されます。次のフルバックアップの出荷が成功すると、LINSTORは2番目のフルバックアップのスナップショットとそれに基づくすべての増分スナップショットを削除し、これを繰り返すことで各バックアップの出荷が成功します。

If there are local snapshots remaining from failed shipments, these will be deleted first, even if they were created later.

If you have enabled a backup shipping schedule and then later manually delete a LINSTOR snapshot, LINSTOR might not be able to delete everything it was supposed to. For example, if you delete a full backup snapshot definition, on a later full backup scheduled shipping, there might be incremental snapshots based on the manually deleted full backup snapshot that will not be deleted.

Configuring the Keep-Remote Option

上記の linstor schedule create コマンドの例のコールアウトで述べたように、 keep-remote オプションはS3リモートデスティネーションに対してのみ機能します。ここでは、このオプションがどのように機能するかの例を示します。keep-remote=2` オプションが設定されている場合、バックアップ出荷スケジュールは、最初の実行時に、フルバックアップ用のスナップショットを作成し、それをリモートデスティネーションに出荷します。次にスケジュールされたフルバックアップの出荷では、2番目のスナップショットが作成され、フルバックアップがリモートデスティネーションに出荷されます。次にスケジュールされたフルバックアップの出荷時に、3つ目のスナップショットが作成され、フルバックアップがリモートディスティネーションに出荷されます。今回はさらに、3つ目のスナップショットが正常に出荷された後、最初のフルバックアップがリモートディスティネーションから削除されます。フルバックアップの間に増分バックアップがスケジュールされていた場合、最初のフルバックアップから作成されたものはフルバックアップと一緒に削除されます。

このオプションは、リモート宛先のバックアップを削除するだけです。ローカル ソース ノードでのフル バックアップの基となったスナップショットは削除されません。

3.13.4. バックアップの配信スケジュールを記載する

linstor schedule list` コマンドを使用すると、バックアップの配信スケジュールを一覧することができます。

For example:

# linstor schedule list
╭──────────────────────────────────────────────────────────────────────────────────────╮
┊ Name                ┊ Full        ┊ Incremental ┊ KeepLocal ┊ KeepRemote ┊ OnFailure ┊
╞══════════════════════════════════════════════════════════════════════════════════════╡
┊ my-bu-schedule      ┊ 2 * * * *   ┊             ┊ 3         ┊ 2          ┊ SKIP      ┊
╰──────────────────────────────────────────────────────────────────────────────────────╯

3.13.5. バックアップした配信予定を削除する

LINSTORクライアントの schedule delete コマンドはバックアップされた配信予定LINSTORオブジェクトを完全に削除するコマンドです。このコマンドの唯一の引数は削除したいスケジュール名です。削除されたスケジュールが現在バックアップを作成または配信している場合、スケジュールされた出荷処理は停止されます。処理が停止した時点によっては、スナップショット、バックアップ、またはその両方が作成および配信されない可能性があります。

このコマンドは、以前に作成されたスナップショットや正常に配信されたバックアップには影響しません。これらは、手動で削除されるまで保持されます。

3.13.6. バックアップの定期的な配信を可能にする

LINSTORクライアントの backup schedule enable コマンドを使用すると、以前に作成したバックアップ配信スケジュールを有効にすることができます。このコマンドは次のような構文になっています。

# linstor backup schedule enable \
  [--node source_node] \ (1)
  [--rg resource_group_name | --rd resource_definition_name] \ (2)
  remote_name \ (3)
  schedule_name (4)
1 これは特別なオプションで、可能であればスケジュールされたバックアップ配信のソースとして使用されるコントローラノードを指定することができます。もしこのオプションをコマンドから省略した場合、LINSTORはスケジュールされた配信が行われる時にソースノードを選択することになります。[OPTIONAL]
2 ここでは、バックアップ配信スケジュールを有効にするリソースグループまたはリソース定義のいずれか(ただし両方は不可)を設定できます。このオプションを省略すると、スナップショットを作成できるすべてのLINSTORリソースに対して、バックアップ配信スケジュールが有効になります。[OPTIONAL]
3 バックアップを配信するリモート宛先の名前です。[REQUIRED]
4 以前に作成されたバックアップ配信スケジュールの名前です。[REQUIRED]

3.13.7. バックアップの配信スケジュールを無効にする

以前に有効にしたバックアップ配信スケジュールを無効にするには、LINSTORクライアントの backup schedule disable コマンドを使用します。このコマンドは次のような構文になります。

# linstor backup schedule disable \
  [--rg resource_group_name | --rd resource_definition_name] \
  remote_name \ (3)
  schedule_name (4)

もし、上記の backup schedule enable コマンドの例のように、リソースグループかリソース定義のどちらかを指定するオプションを含めると、そのリソースグループかリソース定義に対してのみ、スケジュールを無効にします。

例えば、以前の backup schedule enable コマンドでリソースグループやリソース定義の指定を省略した場合、LINSTORはスナップショットを作成することができるすべてのリソースのバックアップ出荷をスケジュールすることになります。disableコマンドは、そのコマンドで指定されたリソースグループやリソース定義にのみ影響します。バックアップ出荷のスケジュールは、指定されたリソースグループまたはリソース定義以外のすべてのLINSTORリソースに適用されます。

バックアップスケジュール有効化コマンドと同様に、リソースグループもリソース定義も指定しない場合、LINSTORは配備されたすべてのLINSTORリソースのコントローラレベルでのバックアップ出荷スケジュールを無効化します。

3.13.8. バックアップされた出荷予定のアスペクトを削除する

linstor backup schedule delete` コマンドを使うと、スケジュール自体を削除することなく、指定したリソース定義やリソースグループをバックアップ出荷スケジュールからきめ細かく削除することができます。このコマンドは backup schedule disable コマンドと同じシンタックスと引数を持っています。リソースグループもリソース定義も指定しない場合、指定したバックアップ出荷スケジュールはコントローラーレベルで削除されます。

It might be helpful to think about the backup schedule delete command as a way that you can remove a backup shipping schedule-remote pair from a specified LINSTOR object level, either a resource definition, a resource group, or at the controller level if neither is specified.

backup schedule delete` コマンドは以前に作成されたスナップショットや正常にシッピングされたバックアップには影響を与えません。これらは手動で削除されるか、まだ適用可能な keep-local または keep-remote オプションの効果によって削除されるまで保持されます。

このコマンドは、複数のLINSTORオブジェクトレベルのバックアップスケジュールを無効にした後、粒度の細かい変更を行いたい場合に使用します。この場合、backup schedule enable コマンドは意図しない結果をもたらすかもしれません。

For example, consider a scenario where you have a backup schedule-remote pair that you enabled at a controller level. This controller has a resource group, myresgroup that has several resource definitions, resdef1 through resdef9, under it. For maintenance reasons perhaps, you disable the schedule for two resource definitions, resdef1 and resdef2. You then realize that further maintenance requires that you disable the backup shipping schedule at the resource group level, for your myresgroup resource group.

After completing some maintenance, you are able to enable the backup shipping schedule for resdef3 through resdef9, but you are not yet ready to resume (enable) backup shipping for resdef1 and resdef2. You can enable backup shipping for each resource definition individually, resdef3 through resdef9, or you can use the backup schedule delete command to delete the backup shipping schedule from the resource group, myresgroup. If you use the backup schedule delete command, backups of resdef3 through resdef9 will ship again because the backup shipping schedule is enabled at the controller level, but resdef1 and resdef2 will not ship because the backup shipping schedule is still disabled for them at the resource definition level.

When you complete your maintenance and are again ready to ship backups for resdef1 and resdef2, you can delete the backup shipping schedule for those two resource definitions to return to your starting state: backup shipping scheduled for all LINSTOR deployed resources at the controller level. To understand this it might be helpful to refer to the decision tree diagram for how LINSTOR decides whether or not to ship a backup in the LINSTORコントローラがスケジュールされたバックアップの出荷を決定する方法 subsection.

In the example scenario above, you might have enabled backup shipping on the resource group, after completing some maintenance. In this case, backup shipping would resume for resource definitions resdef3 through resdef9 but continue not to ship for resource definitions resdef1 and resdef2 because backup shipping was still disabled for those resource definitions. After you completed all maintenance, you could delete the backup shipping schedule on resdef1 and resdef2. Then all of your resource definitions would be shipping backups, as they were before your maintenance, because the schedule-remote pair was enabled at the resource group level. However, this would remove your option to globally stop all scheduled shipping at some later point in time at the controller level because the enabled schedule at the resource group level would override any schedule disable command applied at the controller level.

3.13.9. リソース別バックアップ出荷スケジュール一覧表

LINSTORクライアントの schedule list-by-resource コマンドを使えば、リソースごとのバックアップスケジュールをリストアップすることができます。このコマンドはLINSTORリソースを表示し、バックアップの出荷スケジュールがどのように適用され、どのリモートに出荷されるかを表示します。もしリソースが出荷されていない場合は、このコマンドは表示されます。

  • リソースにschedule-remote-pairのエントリーがないかどうか(空白のセル)

  • schedule-remote-pairの項目があるが、無効になっているかどうか(”disabled”)。

  • リソースがないため、schedule-remote-pairのエントリーが有効かどうかに関わらず、バックアップ出荷ができないかどうか(”undeployed”)

リソースにschedule-remote-pairがあり、出荷されている場合、コマンド出力には、最後のバックアップがいつ出荷されたか、次のバックアップがいつ出荷される予定であるかが表示されます。また、次回のバックアップと前回のバックアップの出荷がフルバックアップかインクリメンタルバックアップかが表示されます。最後に、このコマンドは、増分バックアップ(もしあれば)およびフルバックアップの次の出荷予定日を表示します。

スケジュールリストバイリソースコマンドで --active-only フラグを使用すると、出荷されていないリソースをすべてフィルタリングすることができます。

3.13.10. LINSTORコントローラがスケジュールされたバックアップの出荷を決定する方法

LINSTORコントローラは、デプロイされたLINSTORリソースが特定のリモート宛先に特定のバックアップスケジュールで出荷されるかどうかを判断するために、以下のロジックを使用します。

linstor controller backup schedule decision flowchart

図に示すように、有効または無効なバックアップ出荷スケジュールは、次の順序で効果を発揮します。

  1. リソース定義レベル

  2. リソースグループレベル

  3. コントローラーレベル

先行するレベルで有効または無効にされたバックアップ出荷スケジュールと遠隔地のペアは、後のレベルで同じスケジュールと遠隔地のペアの有効または無効の状態を上書きします。

3.13.11. スケジュールされたバックアップの出荷がリソースに与える影響の確認

LINSTORリソースがスケジュールされたバックアップ出荷によってどのような影響を受けるかを知るには、LINSTORクライアントの schedule list-by-resource-details コマンドを使用して指定したLINSTORリソースに対して行うことができます。

このコマンドは、どのLINSTORオブジェクト・レベルにおいて、バックアップ出荷スケジュールが設定されていない(空のセル)、有効、または無効であるかを示すテーブルを出力します。

このコマンドを使用すると、リソースのスケジュールバックアップの有効化、無効化、削除をどのレベルで変更する必要があるかを判断することができます。

出力例は次のようになります。

# linstor schedule list-by-resource-details my_linstor_resource_name
╭───────────────────────────────────────────────────────────────────────────╮
┊ Remote   ┊ Schedule   ┊ Resource-Definition ┊ Resource-Group ┊ Controller ┊
╞═══════════════════════════════════════════════════════════════════════════╡
┊ rem1     ┊ sch1       ┊ Disabled            ┊                ┊ Enabled    ┊
┊ rem1     ┊ sch2       ┊                     ┊ Enabled        ┊            ┊
┊ rem2     ┊ sch1       ┊ Enabled             ┊                ┊            ┊
┊ rem2     ┊ sch5       ┊                     ┊ Enabled        ┊            ┊
┊ rem3     ┊ sch4       ┊                     ┊ Disabled       ┊ Enabled    ┊
╰───────────────────────────────────────────────────────────────────────────╯

3.14. Setting DRBD Options for LINSTOR Objects

You can use LINSTOR commands to set DRBD options. Configurations in files that are not managed by LINSTOR, such as /etc/drbd.d/global_common.conf, will be ignored. The syntax for this command is generally:

# linstor <LINSTOR_object> drbd-options --<DRBD_option> <value> <LINSTOR_object_identifiers>

In the syntax above, <LINSTOR_ object_identifiers> is a placeholder for identifiers such as a node name, node names, or a resource name, or a combination of these identifiers.

For example, to set the DRBD replication protocol for a resource definition named backups, enter:

# linstor resource-definition drbd-options --protocol C backups

You can enter a LINSTOR object along with drbd-options and the --help, or -h, flag to show the command usage, available options, and the default value for each option. For example:

# linstor controller drbd-options -h

3.14.1. Setting DRBD Peer Options for LINSTOR Resources or Resource Connections

The LINSTOR resource object is an exception to the general syntax for setting DRBD options for LINSTOR objects. With the LINSTOR resource object, you can use the drbd-peer-options to set DRBD options at the connection level between the two nodes that you specify. Specifying drbd-peer-options for a LINSTOR resource object between two nodes is equivalent to using the`linstor resource-connection drbd-peer-options` for a resource between two nodes.

For example, to set the DRBD maximum buffer size to 8192 at a connection level, for a resource named backups, between two nodes, node-0 and node-1, enter:

# linstor resource drbd-peer-options --max-buffers 8192 node-0 node-1 backups

The command above is equivalent to the following:

# linstor resource-connection drbd-peer-options --max-buffers 8192 node-0 node-1 backups

Indeed, when using the linstor --curl command to examine the two commands actions on the LINSTOR REST API, the output is identical:

# linstor --curl resource drbd-peer-options --max-buffers 8192 node-0 node-1 backups
curl -X PUT -H "Content-Type: application/json" -d '{"override_props": {"DrbdOptions/Net/max-buffers": "8192"}}' http://localhost:3370/v1/resource-definitions/backups/resource-connections/node-0/node-1

# linstor --curl resource-connection drbd-peer-options --max-buffers 8192 node-0 node-1 backups
curl -X PUT -H "Content-Type: application/json" -d '{"override_props": {"DrbdOptions/Net/max-buffers": "8192"}}' http://localhost:3370/v1/resource-definitions/backups/resource-connections/node-0/node-1

The connection section of the LINSTOR-generated resource file backups.res on node-0 will look something like this:

connection {
        _peer_node_id 1;
        path {
            _this_host ipv4 192.168.222.10:7000;
            _remote_host ipv4 192.168.222.11:7000;
        }
        path {
            _this_host ipv4 192.168.121.46:7000;
            _remote_host ipv4 192.168.121.220:7000;
        }
        net {
			[...]
            max-buffers         8192;
            _name               "node-1";
        }
    }
If there are multiple paths between the two nodes, as in the example above, DRBD options that you set using the resource drbd-peer-options command will apply to all of them.

3.14.2. Setting DRBD Options for Node Connections

You can use the drbd-peer-options argument to set DRBD options at a connection level, between two nodes, for example:

# linstor node-connection drbd-peer-options --ping-timeout 299 node-0 node-1

The preceding command would set the DRBD ping-timeout option to 29.9 seconds at a connection level between two nodes, node-0 and node-1.

3.14.3. Verifying Options for LINSTOR Objects

You can verify a LINSTOR object’s set properties by using the list-properties command, for example:

|======================================================|
# linstor resource-definition list-properties backups
+------------------------------------------------------+
| Key                               | Value            |
| DrbdOptions/Net/protocol          | C                |
[...]

3.14.4. Removing DRBD Options from LINSTOR Objects

To remove a previously set DRBD option, prefix the option name with unset-. For example:

# linstor resource-definition drbd-options --unset-protocol backups

The same syntax applies to any drbd-peer-options set either on a LINSTOR resource, resource connection, or node connection. For example:

# linstor resource-connection drbd-peer-options --unset-max-buffers node-0 node-1 backups

Removing a DRBD option or DRBD peer option will return the option to its default value. Refer to the linstor <LINSTOR_object> drbd-options --help (or drbd-peer-options --help) command output for the default values of options. You can also refer to the drbd.conf-9.0 man page to get information about DRBD options. === ディスクの追加と削除

LINSTOR can convert resources between diskless and having a disk. This is achieved with the resource toggle-disk command, which has syntax similar to resource create.

For instance, add a disk to the diskless resource backups on ‘alpha’:

# linstor resource toggle-disk alpha backups --storage-pool pool_ssd

Remove this disk again:

# linstor resource toggle-disk alpha backups --diskless

3.14.5. ノード間でのディスクの移行

To move a resource between nodes without reducing redundancy at any point, LINSTOR’s disk migrate feature can be used. First create a diskless resource on the target node, and then add a disk using the --migrate-from option. This will wait until the data has been synced to the new disk and then remove the source disk.

For example, to migrate a resource backups from ‘alpha’ to ‘bravo’:

# linstor resource create bravo backups --drbd-diskless
# linstor resource toggle-disk bravo backups --storage-pool pool_ssd --migrate-from alpha

3.15. Configuring DRBD Proxy Using LINSTOR

LINSTOR expects DRBD Proxy to be running on the nodes which are involved in the relevant connections. It does not currently support connections through DRBD Proxy on a separate node.

Suppose your cluster consists of nodes ‘alpha’ and ‘bravo’ in a local network and ‘charlie’ at a remote site, with a resource definition named backups deployed to each of the nodes. Then DRBD Proxy can be enabled for the connections to ‘charlie’ as follows:

# linstor drbd-proxy enable alpha charlie backups
# linstor drbd-proxy enable bravo charlie backups

The DRBD Proxy configuration can be tailored with commands such as:

# linstor drbd-proxy options backups --memlimit 100000000
# linstor drbd-proxy compression zlib backups --level 9

LINSTOR does not automatically optimize the DRBD configuration for long-distance replication, so you will probably want to set some configuration options such as the protocol:

# linstor resource-connection drbd-options alpha charlie backups --protocol A
# linstor resource-connection drbd-options bravo charlie backups --protocol A

Contact LINBIT for assistance optimizing your configuration.

3.15.1. DRBD プロキシを自動的に有効にする

LINSTOR can also be configured to automatically enable the above mentioned Proxy connection between two nodes. For this automation, LINSTOR first needs to know on which site each node is.

# linstor node set-property alpha Site A
# linstor node set-property bravo Site A
# linstor node set-property charlie Site B

As the Site property might also be used for other site-based decisions in future features, the DrbdProxy/AutoEnable also has to be set to true:

# linstor controller set-property DrbdProxy/AutoEnable true

This property can also be set on node, resource-definition, resource and resource-connection level (from left to right in increasing priority, whereas the controller is the leftmost, that is, the least prioritized level). Once this initialization steps are completed, every newly created resource will automatically check if it has to enable DRBD Proxy to any of its peer-resources. === 外部データベースプロバイダー

It is possible to have LINSTOR working with an external database provider like PostgreSQL, MariaDB and since version 1.1.0 even etcd key value store is supported. To use an external database there are a few additional steps to configure. You have to create a DB/Schema and user to use for linstor, and configure this in the /etc/linstor/linstor.toml. ==== PostgreSQL

A sample PostgreSQL linstor.toml looks like this:

[db]
user = "linstor"
password = "linstor"
connection_url = "jdbc:postgresql://localhost/linstor"

3.15.2. MariaDB and MySQL

A sample MariaDB linstor.toml looks like this:

[db]
user = "linstor"
password = "linstor"
connection_url = "jdbc:mariadb://localhost/LINSTOR?createDatabaseIfNotExist=true"
The LINSTOR schema/database is created as LINSTOR so verify that the MariaDB connection string refers to the LINSTOR schema, as in the example above. ==== etcd

etcd is a distributed key-value store that makes it easy to keep your LINSTOR database distributed in a HA-setup. The etcd driver is already included in the LINSTOR-controller package and only needs to be configured in the linstor.toml.

More information about how to install and configure etcd can be found here: etcd docs

And here is a sample [db] section from the linstor.toml:

[db]
## only set user/password if you want to use authentication, only since LINSTOR 1.2.1
# user = "linstor"
# password = "linstor"

## for etcd
## do not set user field if no authentication required
connection_url = "etcd://etcdhost1:2379,etcdhost2:2379,etcdhost3:2379"

## if you want to use TLS, only since LINSTOR 1.2.1
# ca_certificate = "ca.pem"
# client_certificate = "client.pem"

## if you want to use client TLS authentication too, only since LINSTOR 1.2.1
# client_key_pkcs8_pem = "client-key.pkcs8"
## set client_key_password if private key has a password
# client_key_password = "mysecret"

3.16. 複数のLINSTORコントローラー

The LINSTOR Controller has a configuration file that is and has to be placed into the following path: /etc/linstor/linstor.toml. A recent configuration example can be found here: linstor.toml-example ==== LINSTOR REST API

To make LINSTOR’s administrative tasks more accessible and also available for web-frontends a REST API has been created. The REST API is embedded in the LINSTOR controller and since LINSTOR 0.9.13 configured through the linstor.toml configuration file.

[http]
  enabled = true
  port = 3370
  listen_addr = "127.0.0.1"  # to disable remote access

If you want to use the REST API the current documentation can be found on the following link: https://app.swaggerhub.com/apis-docs/Linstor/Linstor/ ==== LINSTOR REST API HTTPS

The HTTP REST API can also run secured by HTTPS and is highly recommended if you use any features that require authorization. To do so you have to create a Java keystore file with a valid certificate that will be used to encrypt all HTTPS traffic.

Here is a simple example on how you can create a self signed certificate with the keytool that is included in the Java Runtime:

keytool -keyalg rsa -keysize 2048 -genkey -keystore ./keystore_linstor.jks\
 -alias linstor_controller\
 -dname "CN=localhost, OU=SecureUnit, O=ExampleOrg, L=Vienna, ST=Austria, C=AT"

keytool will ask for a password to secure the generated keystore file and is needed for the LINSTOR Controller configuration. In your linstor.toml file you have to add the following section:

[https]
  keystore = "/path/to/keystore_linstor.jks"
  keystore_password = "linstor"

Now (re)start the linstor-controller and the HTTPS REST API should be available on port 3371. More information about how to import other certificates can be found here: https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html NOTE: When HTTPS is enabled, all requests to the HTTP /v1/ REST API will be redirected to the HTTPS redirect. ===== LINSTOR REST-API HTTPS 制限付きクライアントアクセス

Client access can be restricted by using a SSL/TLS truststore on the Controller. Basically you create a certificate for your client and add it to your truststore and the client then uses this certificate for authentication.

First create a client certificate:

keytool -keyalg rsa -keysize 2048 -genkey -keystore client.jks\
 -storepass linstor -keypass linstor\
 -alias client1\
 -dname "CN=Client Cert, OU=client, O=Example, L=Vienna, ST=Austria, C=AT"

Next, import this certificate to your controller truststore:

keytool -importkeystore\
 -srcstorepass linstor -deststorepass linstor -keypass linstor\
 -srckeystore client.jks -destkeystore trustore_client.jks

And enable the truststore in the linstor.toml configuration file:

[https]
  keystore = "/path/to/keystore_linstor.jks"
  keystore_password = "linstor"
  truststore = "/path/to/trustore_client.jks"
  truststore_password = "linstor"

Now restart the Controller and it will no longer be possible to access the controller API without a correct certificate.

The LINSTOR client needs the certificate in PEM format, so before you can use it, you have to convert the java keystore certificate to the PEM format.

# Convert to pkcs12
keytool -importkeystore -srckeystore client.jks -destkeystore client.p12\
 -storepass linstor -keypass linstor\
 -srcalias client1 -srcstoretype jks -deststoretype pkcs12

# use openssl to convert to PEM
openssl pkcs12 -in client.p12 -out client_with_pass.pem

To avoid entering the PEM file password all the time it might be convenient to remove the password.

openssl rsa -in client_with_pass.pem -out client1.pem
openssl x509 -in client_with_pass.pem >> client1.pem

Now this PEM file can easily be used in the client:

linstor --certfile client1.pem node list

The --certfile parameter can also added to the client configuration file, see LINSTORクライアントの使用 for more details. === LINSTOR Satellite の設定

The LINSTOR Satellite software has an optional configuration file that uses the TOML file syntax and has to be put into the following path /etc/linstor/linstor_satellite.toml. A recent configuration example can be found here: linstor_satellite.toml-example === ロギング

LINSTOR uses SLF4J with logback as binding. This gives LINSTOR the possibility to distinguish between the log levels ERROR, WARN, INFO, DEBUG and TRACE (in order of increasing verbosity). The following are the different ways that you can set the logging level, ordered by priority (first has highest priority):

  1. Since LINSTOR client version 1.20.1, you can use the command controller set-log-level to change the log level used by LINSTOR’s running configuration. Various arguments can be used with this command. Refer to the command’s --help text for details. For example, to set the log level to TRACE on the LINSTOR controller and all satellites, enter the following command:

    $ linstor controller set-log-level --global TRACE

    To change the LINSTOR log level on a particular node, you can use the LINSTOR client (since version 1.20.1) command node set-log-level.

    Changes that you make to the log level by using the LINSTOR client will not persist LINSTOR service restarts, for example, if a node reboots.
  2. TRACE mode can be enabled or disabled using the debug console:

    Command ==> SetTrcMode MODE(enabled)
    SetTrcMode           Set TRACE level logging mode
    New TRACE level logging mode: ENABLED
  3. When starting the controller or satellite a command line argument can be passed:

    java ... com.linbit.linstor.core.Controller ... --log-level TRACE
    java ... com.linbit.linstor.core.Satellite  ... --log-level TRACE
  4. The recommended place is the logging section in the configuration file. The default configuration file location is /etc/linstor/linstor.toml for the controller and /etc/linstor/linstor_satellite.toml for the satellite. Configure the logging level as follows:

    [logging]
       level="TRACE"
  5. As LINSTOR is using logback as an implementation, /usr/share/linstor-server/lib/logback.xml can also be used. Currently only this approach supports different log levels for different components, like shown in the example below:

    <?xml version="1.0" encoding="UTF-8"?>
    <configuration scan="false" scanPeriod="60 seconds">
    <!--
     Values for scanPeriod can be specified in units of milliseconds, seconds, minutes or hours
     https://logback.qos.ch/manual/configuration.html
    -->
     <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
       <!-- encoders are assigned the type
            ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
       <encoder>
         <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger - %msg%n</pattern>
       </encoder>
     </appender>
     <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
       <file>${log.directory}/linstor-${log.module}.log</file>
       <append>true</append>
       <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
         <Pattern>%d{yyyy_MM_dd HH:mm:ss.SSS} [%thread] %-5level %logger - %msg%n</Pattern>
       </encoder>
       <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
         <FileNamePattern>logs/linstor-${log.module}.%i.log.zip</FileNamePattern>
         <MinIndex>1</MinIndex>
         <MaxIndex>10</MaxIndex>
       </rollingPolicy>
       <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
         <MaxFileSize>2MB</MaxFileSize>
       </triggeringPolicy>
     </appender>
     <logger name="LINSTOR/Controller" level="TRACE" additivity="false">
       <appender-ref ref="STDOUT" />
       <!-- <appender-ref ref="FILE" /> -->
     </logger>
     <logger name="LINSTOR/Satellite" level="TRACE" additivity="false">
       <appender-ref ref="STDOUT" />
       <!-- <appender-ref ref="FILE" /> -->
     </logger>
     <root level="WARN">
       <appender-ref ref="STDOUT" />
       <!-- <appender-ref ref="FILE" /> -->
     </root>
    </configuration>

See the logback manual to find more details about logback.xml. When none of the configuration methods above is used LINSTOR will default to INFO log level. === 監視

Since LINSTOR 1.8.0, a Prometheus /metrics HTTP path is provided with LINSTOR and JVM specific exports.

The /metrics path also supports three GET arguments to reduce LINSTOR’s reported data:

  • resource

  • storage_pools

  • error_reports

These all default to true. To disable, for example, error report data use: http://localhost:3370/metrics?error_reports=false.

3.16.1. ヘルスチェック

The LINSTOR-Controller also provides a /health HTTP path that will simply return HTTP-Status 200 if the controller can access its database and all services are up and running. Otherwise it will return HTTP error status code 500 Internal Server Error. === サテライトへの安全な接続

It is possible to have the LINSTOR use SSL/TLS secure TCP connection between controller and satellite connections. Without going into further details on how Java’s SSL/TLS engine works we will give you command line snippets using the keytool from Java’s runtime environment on how to configure a three node setup using secure connections.

The node setup looks like this:

Node alpha is the just the controller. Node bravo and node charlie are just satellites.

Here are the commands to generate such a keystore setup, values should of course be edited for your environment.

# create directories to hold the key files
mkdir -p /tmp/linstor-ssl
cd /tmp/linstor-ssl
mkdir alpha bravo charlie


# create private keys for all nodes
keytool -keyalg rsa -keysize 2048 -genkey -keystore alpha/keystore.jks\
 -storepass linstor -keypass linstor\
 -alias alpha\
 -dname "CN=Max Mustermann, OU=alpha, O=Example, L=Vienna, ST=Austria, C=AT"

keytool -keyalg rsa -keysize 2048 -genkey -keystore bravo/keystore.jks\
 -storepass linstor -keypass linstor\
 -alias bravo\
 -dname "CN=Max Mustermann, OU=bravo, O=Example, L=Vienna, ST=Austria, C=AT"

keytool -keyalg rsa -keysize 2048 -genkey -keystore charlie/keystore.jks\
 -storepass linstor -keypass linstor\
 -alias charlie\
 -dname "CN=Max Mustermann, OU=charlie, O=Example, L=Vienna, ST=Austria, C=AT"

# import truststore certificates for alpha (needs all satellite certificates)
keytool -importkeystore\
 -srcstorepass linstor -deststorepass linstor -keypass linstor\
 -srckeystore bravo/keystore.jks -destkeystore alpha/certificates.jks

keytool -importkeystore\
 -srcstorepass linstor -deststorepass linstor -keypass linstor\
 -srckeystore charlie/keystore.jks -destkeystore alpha/certificates.jks

# import controller certificate into satellite truststores
keytool -importkeystore\
 -srcstorepass linstor -deststorepass linstor -keypass linstor\
 -srckeystore alpha/keystore.jks -destkeystore bravo/certificates.jks

keytool -importkeystore\
 -srcstorepass linstor -deststorepass linstor -keypass linstor\
 -srckeystore alpha/keystore.jks -destkeystore charlie/certificates.jks

# now copy the keystore files to their host destinations
ssh root@alpha mkdir /etc/linstor/ssl
scp alpha/* root@alpha:/etc/linstor/ssl/
ssh root@bravo mkdir /etc/linstor/ssl
scp bravo/* root@bravo:/etc/linstor/ssl/
ssh root@charlie mkdir /etc/linstor/ssl
scp charlie/* root@charlie:/etc/linstor/ssl/

# generate the satellite ssl config entry
echo '[netcom]
  type="ssl"
  port=3367
  server_certificate="ssl/keystore.jks"
  trusted_certificates="ssl/certificates.jks"
  key_password="linstor"
  keystore_password="linstor"
  truststore_password="linstor"
  ssl_protocol="TLSv1.2"
' | ssh root@bravo "cat > /etc/linstor/linstor_satellite.toml"

echo '[netcom]
  type="ssl"
  port=3367
  server_certificate="ssl/keystore.jks"
  trusted_certificates="ssl/certificates.jks"
  key_password="linstor"
  keystore_password="linstor"
  truststore_password="linstor"
  ssl_protocol="TLSv1.2"
' | ssh root@charlie "cat > /etc/linstor/linstor_satellite.toml"

Now just start controller and satellites and add the nodes with --communication-type SSL. === LDAP 認証の設定

You can configure LINSTOR to use LDAP authentication to limit access to the LINSTOR Controller. This feature is disabled by default but you can enable and configure it by editing the LINSTOR configuration TOML file. After editing the configuration file, you will need to restart the linstor-controller.service. An example LDAP section within the configuration file looks like this:

[ldap]
  enabled = true (1)

  # allow_public_access: if no authorization fields are given allow
  # users to work with the public context
  allow_public_access = false (2)

  # uniform resource identifier: LDAP URI to use
  # for example, "ldaps://hostname" (LDAPS) or "ldap://hostname" (LDAP)
  uri = "ldaps://ldap.example.com"

  # distinguished name: {user} can be used as template for the user name
  dn = "uid={user}" (3)

  # search base for the search_filter field
  search_base = "dc=example,dc=com" (4)

  # search_filter: ldap filter to restrict users on memberships
  search_filter = "(&(uid={user})(memberof=ou=storage-services,dc=example,dc=com))" (5)
1 enabled is a Boolean value. Authentication is disabled by default.
2 allow_public_access is a Boolean value. If set to true, and LDAP authentication is enabled, then users will be allowed to work with the LINSTOR Controller’s public context. If set to false and LDAP authentication is enabled, then users without LDAP authenticating credentials will be unable to access the LINSTOR Controller for all but the most trivial tasks, such as displaying version or help information.
3 dn is a string value where you can specify the LDAP distinguished name to query the LDAP directory. Besides the user ID (uid), the string can consist of other distinguished name attributes, for example:
dn = "uid={user},ou=storage-services,o=ha,dc=example"
4 search_base is a string value where you can specify the starting point in the LDAP directory tree for the authentication query, for example:
search_base = "ou=storage-services"
5 search_filter is a string value where you can specify an LDAP object restriction for authentication, such as user and group membership, for example:
search_filter = "(&(uid={user})(memberof=ou=storage-services,dc=example,dc=com))"
It is highly recommended that you configure LINSTOR REST API HTTPS and LDAPS to protect potentially sensitive traffic passing between the LINSTOR Controller and an LDAP server.

3.16.2. 認証ユーザとしてのLINSTORコマンドの実行

After configuring the LINSTOR Controller to authenticate users through LDAP (or LDAPS), and the LINSTOR REST API HTTPS, you will need to enter LINSTOR commands as follows:

$ linstor --user <LDAP_user_name> <command>

If you have configured LDAP authentication without also configuring LINSTOR REST API HTTPS, you will need to explicitly enable password authentication over HTTP, by using the --allow-insecure-path flag with your linstor commands. This is not recommended outside of a secured and isolated LAN, as you will be sending credentials in plain text.

$ linstor --allow-insecure-auth --user <LDAP_user_name> <command>

The LINSTOR Controller will prompt you for the user’s password, in each of the above examples. You can optionally use the --password argument to supply the user’s password on the command line, with all the warnings of caution that would go along with doing so. === Automatisms for DRBD Resources

This section details some of LINSTOR’s automatisms for DRBD resources. ==== 自動クォーラムポリシー

LINSTOR automatically configures quorum policies on resources when quorum is achievable. This means, whenever you have at least two diskful and one or more diskless resource assignments, or three or more diskful resource assignments, LINSTOR will enable quorum policies for your resources automatically.

Inversely, LINSTOR will automatically disable quorum policies whenever there are less than the minimum required resource assignments to achieve quorum.

This is controlled through the, DrbdOptions/auto-quorum, property which can be applied to the linstor-controller, resource-group, and resource-definition LINSTOR objects. Accepted values for the DrbdOptions/auto-quorum property are disabled, suspend-io, and io-error.

Setting the DrbdOptions/auto-quorum property to disabled will allow you to manually, or more granularly, control the quorum policies of your resources should you want to.

The default policies for DrbdOptions/auto-quorum are quorum majority, and on-no-quorum io-error. For more information about DRBD’s quorum features and their behavior, refer to the quorum section of the DRBD User’s Guide.
The DrbdOptions/auto-quorum policies will override any manually configured properties if DrbdOptions/auto-quorum is not disabled.

For example, to manually set the quorum policies of a resource group named my_ssd_group, you would use the following commands:

# linstor resource-group set-property my_ssd_group DrbdOptions/auto-quorum disabled
# linstor resource-group set-property my_ssd_group DrbdOptions/Resource/quorum majority
# linstor resource-group set-property my_ssd_group DrbdOptions/Resource/on-no-quorum suspend-io

You might want to disable DRBD’s quorum features completely. To do that, you would need to first disable DrbdOptions/auto-quorum on the appropriate LINSTOR object, and then set the DRBD quorum features accordingly. For example, use the following commands to disable quorum entirely on the my_ssd_group resource group:

# linstor resource-group set-property my_ssd_group DrbdOptions/auto-quorum disabled
# linstor resource-group set-property my_ssd_group DrbdOptions/Resource/quorum off
# linstor resource-group set-property my_ssd_group DrbdOptions/Resource/on-no-quorum
Setting DrbdOptions/Resource/on-no-quorum to an empty value in the commands above deletes the property from the object entirely. ==== 自動取り除き

If a satellite is offline for a prolonged period of time, LINSTOR can be configured to declare that node as evicted. This triggers an automated reassignment of the affected DRBD resources to other nodes to ensure a minimum replica count is kept.

This feature uses the following properties to adapt the behavior.

  • DrbdOptions/AutoEvictMinReplicaCount sets the number of replicas that should always be present. You can set this property on the controller to change a global default, or on a specific resource definition or resource group to change it only for that resource definition or resource group. If this property is left empty, the place count set for the Autoplacer of the corresponding resource group will be used.

  • DrbdOptions/AutoEvictAfterTime describes how long a node can be offline in minutes before the eviction is triggered. You can set this property on the controller to change a global default, or on a single node to give it a different behavior. The default value for this property is 60 minutes.

  • DrbdOptions/AutoEvictMaxDisconnectedNodes sets the percentage of nodes that can be not reachable (for whatever reason) at the same time. If more than the given percent of nodes are offline at the same time, the auto-evict will not be triggered for any node , since in this case LINSTOR assumes connection problems from the controller. This property can only be set for the controller, and only accepts a value between 0 and 100. The default value is 34. If you want to turn the auto-evict-feature off, simply set this property to 0. If you want to always trigger the auto-evict, regardless of how many satellites are unreachable, set it to 100.

  • DrbdOptions/AutoEvictAllowEviction is an additional property that can stop a node from being evicted. This can be useful for various cases, for example if you need to shut down a node for maintenance. You can set this property on the controller to change a global default, or on a single node to give it a different behavior. It accepts true and false as values and per default is set to true on the controller. You can use this property to turn the auto-evict feature off by setting it to false on the controller, although this might not work completely if you already set different values for individual nodes, since those values take precedence over the global default.

After the LINSTOR controller loses the connection to a satellite, aside from trying to reconnect, it starts a timer for that satellite. As soon as that timer exceeds DrbdOptions/AutoEvictAfterTime and all of the DRBD connections to the DRBD resources on that satellite are broken, the controller will check whether or not DrbdOptions/AutoEvictMaxDisconnectedNodes has been met. If it has not, and DrbdOptions/AutoEvictAllowEviction is true for the node in question, the satellite will be marked as EVICTED. At the same time, the controller will check for every DRBD-resource whether the number of resources is still above DrbdOptions/AutoEvictMinReplicaCount. If it is, the resource in question will be marked as DELETED. If it isn’t, an auto-place with the settings from the corresponding resource-group will be started. Should the auto-place fail, the controller will try again later when changes that might allow a different result, such as adding a new node, have happened. Resources where an auto-place is necessary will only be marked as DELETED if the corresponding auto-place was successful.

The evicted satellite itself will not be able to reestablish connection with the controller. Even if the node is up and running, a manual reconnect will fail. It is also not possible to delete the satellite, even if it is working as it should be. The satellite can, however, be restored. This will remove the EVICTED-flag from the satellite and allow you to use it again. Previously configured network interfaces, storage pools, properties and similar entities, non-DRBD-related resources, and resources that LINSTOR could not autoplace somewhere else will still be on the satellite. To restore a satellite, use the following command:

# linstor node restore [nodename]

Should you want to instead throw everything that once was on that node, including the node itself, away, you need to use the node lost command instead. ==== Auto-Diskful and Related Options

You can set the LINSTOR auto-diskful and auto-diskful-allow-cleanup properties for various LINSTOR objects, for example, a resource definition, to have LINSTOR help automatically make a Diskless node Diskful and perform appropriate cleanup actions afterwards. This is useful when a Diskless node has been in a Primary state for a DRBD resource for more than a specified number of minutes. This could happen in cases where you integrate LINSTOR managed storage with other orchestrating and scheduling platforms, such as OpenStack, OpenNebula, and others. On some platforms that you integrate LINSTOR with, you might not have a way to influence where in your cluster a storage volume will be used. The auto-diskful options give you a way to use LINSTOR to sensibly delegate the roles of your storage nodes in response to an integrated platform’s actions that are beyond your control. ===== Setting the Auto-Diskful Option

By setting the DrbdOptions/auto-diskful option on a LINSTOR resource definition , you are configuring the LINSTOR controller to make a Diskless DRBD resource Diskful if the resource has been in a DRBD Primary state for more than the specified number of minutes. After the specified number of minutes, LINSTOR will automatically use the resource toggle-disk command to toggle the resource state on the Diskless node, for the given resource.

To set this property, for example, on a LINSTOR resource definition named myres with a threshold of five minutes, enter the command:

# linstor resource-definition set-property myres DrbdOptions/auto-diskful 5
Setting the Auto-Diskful Option on a Resource Group or Controller

You can also set the DrbdOptions/auto-diskful option on LINSTOR controller or resource-group objects. By setting the option at the controller level, the option will affect all LINSTOR resource definitions in your LINSTOR cluster that do not have this option set, either on the resource definition itself, or else on the resource group that you might have created the resource from. Setting the option on a LINSTOR resource group will affect all resource definitions that are spawned from the group, unless a resource definition has the option set on it. The order of priority, from highest to lowest, for the effect of setting the auto-diskful option is: – Resource definition – Resource group – Controller ===== Unsetting the Auto-Diskful Option

To unset the LINSTOR auto-diskful option, enter:

# linstor <controller|resource-definition|resource-group> set-property DrbdOptions/auto-diskful
Setting the Auto-Diskful-Allow-Cleanup Option

A companion option to the LINSTOR auto-diskful option is the DrbdOptions/auto-diskful-allow-cleanup option. You can set this option on the following LINSTOR objects: node, resource, resource definition, or resource group. The default value for this option is True, but the option has no effect unless the auto-diskful option has also been set. After LINSTOR has toggled a resource to Diskful, because the threshold number of minutes has passed where a Diskless node was in the Primary role for a resource, and after DRBD has synchronized the data to this previously Diskless and now Primary node, LINSTOR will remove the resource from any Secondary nodes when that action is necessary to fulfill a replica count constraint that the resource might have. This could be the case, for example, if you have specified number of replicas for a resource by using the --auto-place option. [s-linstor-qos]] === QoS設定

LINSTOR implements QoS for managed resources by using sysfs properties that correspond to kernel variables related to block I/O operations. These sysfs properties can be limits on either bandwidth (bytes per second), or IOPS, or both. The sysfs files and their corresponding LINSTOR properties are as follows: [cols=”3,2″]

sysfs (/sys/fs/) LINSTOR Property

cgroup/blkio/blkio.throttle.read_bps_device

sys/fs/blkio_throttle_read

cgroup/blkio/blkio.throttle.write_bps_device

sys/fs/blkio_throttle_write

cgroup/blkio/blkio.throttle.read_iops_device

sys/fs/blkio_throttle_read_iops

cgroup/blkio/blkio.throttle.write_iops_device

sys/fs/blkio_throttle_write_iops

3.16.3. LINSTOR sysfsのプロパティを使ったQoSの設定

These LINSTOR properties can be set by using the set-property command and can be set on the following objects: volume, storage pool, resource, controller, or node. You can also set these QoS properties on resource groups, volume groups, resource definitions, or volume definitions. When you set a QoS property on a group or definition, resources created from the group or definition will inherit the QoS settings.

グループまたは定義に対して行われた設定は、グループまたは定義から作成された既存のリソースと新しいリソースの両方に影響します。

次の例では、リソースグループを作成し、ボリュームグループを作成し、ボリュームグループにQoS設定を適用し、リソースグループからリソースをスポーンしています。検証コマンドを実行すると、スポーンされたリソースがQoS設定を継承していることがわかります。 この例では、_pool1_という名前の以前に作成されたLINSTORストレージプールを想定して使用します。この名前は、あなたの環境に存在するストレージプール名に置き換える必要があります。

# linstor resource-group create qos_limited --storage-pool pool1 --place-count 3
# linstor volume-group create qos_limited
# linstor volume-group set-property qos_limited 0 sys/fs/blkio_throttle_write 1048576
# linstor resource-group spawn-resources qos_limited qos_limited_res 200M

生成されたリソースがQoS設定を継承していることを確認するには、ストレージプールにストレージを提供するノードで、対応するsysfsファイルの内容を表示します。

# cat /sys/fs/cgroup/blkio/blkio.throttle.write_bps_device
252:4 1048576
QoS プロパティは継承されコピーされないため、 “parent” グループまたは定義から生成された “child” オブジェクトにはプロパティが表示されません。

3.16.4. 複数の DRBD デバイスを持つ LINSTOR ボリュームの QoS 設定

単一の LINSTOR ボリュームは、複数の DRBD デバイスで構成できます。たとえば、外部メタデータを含む DRBD には、データ (ストレージ) デバイス、メタデータ デバイス、および LINSTOR に提供される複合 DRBD デバイス (ボリューム) の 3 つの下位デバイスがあります。データデバイスとメタデータデバイスが異なる下位ディスクに対応している場合、そのような LINSTOR ボリュームに sysfs プロパティを設定すると、ローカルデータ (ストレージ) 下位デバイスのみが対応する /sys/fs/cgroup/blkio/ ファイルのプロパティ値を受け取ります。DRBD メタデータの下位デバイスや、LINSTOR で提供される複合下位デバイスは値を受け取りません。ただし、DRBD のデータとそのメタデータが同じ下位ディスクを共有する場合、QoS 設定はデータとメタデータ操作の両方のパフォーマンスに影響します。

3.16.5. NVMe の QoS設定

In case a LINSTOR resource definition has an nvme-target and an nvme-initiator resource, both data (storage) backing devices of each node will receive the sysfs property value. In case of the target, the data backing device will be the volume of either LVM or ZFS, whereas in case of the initiator, the data backing device will be the connected nvme-device, regardless of which other LINSTOR layers, such as LUKS, NVMe, DRBD, and others (see DRBDを使わないLINSTOR), are above that.

3.17. ヘルプの利用

3.17.1. コマンドラインから確認

A quick way to list available commands on the command line is to enter linstor.

Further information about subcommands, for example, listing nodes) can be retrieved in two ways:

# linstor node list -h
# linstor help node list

LINSTOR がインタラクティブモード( linstor interactive )で動作しているときには ‘help’ サブコマンドはとても役にたちます。

One of the most helpful features of LINSTOR is its rich tab-completion, which can be used to complete nearly every object that LINSTOR knows about, for example, node names, IP addresses, resource names, and others. The following examples show some possible completions, and their results:

# linstor node create alpha 1<tab> # ホスト名が解決できるとIPアドレスを補完します
# linstor resource create b<tab> c<tab> # linstorは backups, charlieリソースを補完します

If tab-completion does not work upon installing the LINSTOR client, try to source the appropriate file:

# source /etc/bash_completion.d/linstor # または
# source /usr/share/bash_completion/completions/linstor

For Z shell users, the linstor-client command can generate a Z shell compilation file, that has basic support for command and argument completion.

# linstor gen-zsh-completer > /usr/share/zsh/functions/Completion/Linux/_linstor

3.17.2. Generating SOS Reports

If something goes wrong and you need help finding the cause of the issue, you can enter the following command:

# linstor sos-report create

The command above will create a new sos-report in the /var/log/linstor/controller directory on the controller node. Alternatively you can enter the following command:

# linstor sos-report download

This command will create a new SOS report and download that report to the current working directory on your local machine.

This SOS report contains logs and useful debug-information from several sources (LINSTOR logs, dmesg, versions of external tools used by LINSTOR, ip a, database dump and many more). These information are stored for each node in plain text in the resulting .tar.gz file.

3.17.3. コミュニティの助けを借りる

For help from the community, subscribe to the DRBD user mailing list located here: https://lists.linbit.com/listinfo/drbd-user

3.17.4. GitHub

To file bug or feature request, check out the LINBIT GitHub page https://github.com/linbit

3.17.5. 有料のサポートと開発

Alternatively, if you need to purchase remote installation services, 24×7 support, access to certified repositories, or feature development, contact us: +1-877-454-6248 (1-877-4LINBIT) , International: +43-1-8178292-0 | [email protected]

Administering LINSTOR by GUI

4. LINBIT SDS GUI

LINBIT SDS GUI is a LINSTOR® client alternative, currently in Technology Preview phase. This component is proprietary and users need to have access to LINBIT® customer-only repositories to be able to use it.

4.1. 前提条件

  • LINBIT の顧客リポジトリへのアクセス

  • LINSTOR コントローラーインスタンスの実行と動作環境

4.2. Installing the LINBIT SDS GUI

LINSTOR コントローラーと同じノードに LINSTOR GUI パッケージをインストールし linstor-controller サービスを再起動します。

yum/dnf ベースのディストリビューションでは、次の方法でソフトウェアをインストールできます。

yum install linstor-gui

apt ベースのディストリビューションでは、次の方法でソフトウェアをインストールします。

apt install linstor-gui

On Kubernetes, LINBIT SDS GUI is a built-in feature since linstor-controller v1.15.0.

4.3. Administering the LINSTOR Cluster by Using the LINBIT SDS GUI

You can access the LINBIT SDS GUI by opening an HTTP connection with the active LINSTOR controller node via TCP port 3370. For example, if your LINSTOR controller’s IP address is 192.168.222.250, you would enter http://192.168.222.250:3370 into your web browser’s address bar to use the LINBIT SDS GUI.

LINSTOR Integrations

5. Kubernetes で LINSTOR ボリューム

This chapter describes the usage of LINSTOR® in Kubernetes (K8s) as managed by the operator and with volumes provisioned using the LINSTOR CSI plugin.

This chapter goes into great detail regarding all the install time options and various configurations possible with LINSTOR and Kubernetes. The chapter begins with some explanatory remarks and then moves onto deployment instructions. After that, there are instructions for getting started with LINSTOR to configure storage within a Kubernetes deployment. Following that, more advanced topics and configurations, such as snapshots and monitoring, are covered.

5.1. Kubernetesの概要

Kubernetes_は、コンテナオーケストレーターです。Kubernetes は、宣言した仕様に基づいてコンテナと関連サービスの動作を定義します。このガイドでは、Kubernetes オブジェクトの仕様を定義する AML ファイルを介して kubectl を使用することに焦点を当てます。

5.2. Kubernetes への LINSTOR のデプロイ

LINBIT® provides a LINSTOR Operator to commercial support customers. The Operator eases deployment of LINSTOR on Kubernetes by installing DRBD®, managing satellite and controller pods, and other related functions.

LINBIT’s container image repository (https://drbd.io), used by LINSTOR Operator, is only available to LINBIT customers or through LINBIT customer trial accounts. Contact LINBIT for information on pricing or to begin a trial. Alternatively, you can use the LINSTOR SDS upstream project named Piraeus, without being a LINBIT customer.

LINSTOR Operator v2 is the recommended way of deploying LINBIT SDS for Kubernetes on new clusters. Users of existing Operator v1 deployments should continue to use their Helm deployments and skip to the, Operator v1 deployment instructions.

5.3. Deploying LINSTOR Operator v2

You can deploy the LINSTOR Operator v2 by using either the Kustomize tool, integrated with kubectl, or else by using Helm and a LINBIT Helm chart.

If you have already deployed LINSTOR Operator v1 into your cluster, you can upgrade your LINBIT SDS deployment in Kubernetes to Operator v2 by following the instructions at charts.linstor.io.

5.3.1. Creating the Operator v2 by Using Kustomize

To deploy the Operator, create a kustomization.yaml file. This will declare your pull secret for drbd.io and allow you to pull in the Operator deployment. The Operator will be deployed in a new namespace linbit-sds. Make sure to replace MY_LINBIT_USER and MY_LINBIT_PASSWORD with your own credentials. You can find the latest releases on charts.linstor.io.

Listing 2. kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: linbit-sds
resources:
  - https://charts.linstor.io/static/v2.4.0.yaml (1)
generatorOptions:
  disableNameSuffixHash: true
secretGenerator:
  - name: drbdio-pull-secret
    type: kubernetes.io/dockerconfigjson
    literals:
      - .dockerconfigjson={"auths":{"drbd.io":{"username":"MY_LINBIT_USER","password":"MY_LINBIT_PASSWORD"}}} (2)
1 Replace with the latest release manifest from charts.linstor.io.
2 Replace MY_LINBIT_USER and MY_LINBIT_PASSWORD with your my.linbit.com credentials.

Then, apply the kustomization.yaml file, by using kubectl command, and wait for the Operator to start:

$ kubectl apply -k .
namespace/linbit-sds created
...
$ kubectl -n linbit-sds  wait pod --for=condition=Ready --all
pod/linstor-operator-controller-manager-6d9847d857-zc985 condition met

The Operator is now ready to deploy LINBIT SDS for Kubernetes.

5.3.2. Deploying LINBIT SDS for Kubernetes by Using the Command Line Tool

Deploying LINBIT SDS for Kubernetes with the Operator v2 is as simple as creating a new LinstorCluster resource and waiting for the Operator to complete the setup:

$ kubectl create -f - <<EOF
apiVersion: piraeus.io/v1
kind: LinstorCluster
metadata:
  name: linstorcluster
spec: {}
EOF
$ kubectl wait pod --for=condition=Ready -n linbit-sds --timeout=3m --all

Output should eventually show that the wait condition has been met and the LINBIT SDS pods are up and running.

pod/ha-controller-4tgcg condition met
pod/k8s-1-26-10.test condition met
pod/linstor-controller-76459dc6b6-tst8p condition met
pod/linstor-csi-controller-75dfdc967d-dwdx6 condition met
pod/linstor-csi-node-9gcwj condition met
pod/linstor-operator-controller-manager-6d9847d857-zc985 condition met

5.3.3. Creating the Operator v2 by Using Helm

To create the LINSTOR Operator v2 by Using Helm, first enter the following commands to add the linstor Helm chart repository:

$ MY_LINBIT_USER=<my-linbit-customer-username>
$ MY_LINBIT_PASSWORD=<my-linbit-customer-password>
$ helm repo add linstor https://charts.linstor.io

Next, enter the following command to install the LINSTOR Operator v2:

$ helm install linstor-operator linstor/linstor-operator \
  --set imageCredentials.username=$MY_LINBIT_USER \
  --set imageCredentials.password=$MY_LINBIT_PASSWORD \
  --wait

5.3.4. Deploying LINBIT SDS for Kubernetes by Using Helm

After output from the command shows that the Operator v2 was installed, you can use Helm to deploy LINBIT SDS by installing the linbit-sds chart:

$ helm install linbit-sds linstor/linbit-sds

Output from this final helm install command should show a success message.

[...]
LinstorCluster: linbit-sds

Successfully deployed!
[...]

5.3.5. Configuring Storage with Operator v2

By default, LINBIT SDS for Kubernetes does not configure any storage. To add storage, you can configure a LinstorSatelliteConfiguration, which the Operator uses to configure one or more satellites.

The following example creates a simple FILE_THIN pool and it does not require any additional set up on the host:

$ kubectl apply -f - <<EOF
apiVersion: piraeus.io/v1
kind: LinstorSatelliteConfiguration
metadata:
  name: storage-pool
spec:
  storagePools:
    - name: pool1
      fileThinPool:
        directory: /var/lib/linbit-sds/pool1
EOF

Other types of storage pools can be configured as well. Refer to the examples upstream.

5.3.6. Securing Operator v2 Deployment

By configuring key and certificate based encryption, you can make communication between certain LINSTOR components, for example, between LINSTOR satellite nodes and a LINSTOR controller node, or between the LINSTOR client and the LINSTOR API, more secure.

Configuring TLS Between the LINSTOR Controller and Satellite

To secure traffic between the LINSTOR controller and satellite nodes, you can configure TLS, either by using cert-manager or OpenSSL to create TLS certificates to encrypt the traffic.

Provisioning Keys and Certificates By Using cert-manager

This method requires a working cert-manager deployment in your cluster. For an alternative way to provision keys and certificates, see the OpenSSL section below.

The LINSTOR controller and satellite only need to trust each other. For that reason, you should only have a certificate authority (CA) for those components. Apply the following YAML configuration to your deployment to create a new cert-manager Issuer resource:

Listing 3. linstor-cert-manager.yaml
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: ca-bootstrapper
  namespace: linbit-sds
spec:
  selfSigned: { }
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: linstor-internal-ca
  namespace: linbit-sds
spec:
  commonName: linstor-internal-ca
  secretName: linstor-internal-ca
  duration: 87600h # 10 years
  isCA: true
  usages:
    - signing
    - key encipherment
    - cert sign
  issuerRef:
    name: ca-bootstrapper
    kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: linstor-internal-ca
  namespace: linbit-sds
spec:
  ca:
    secretName: linstor-internal-ca

Next, configure the new issuer resource to let the LINSTOR Operator provision the certificates needed to encrypt the controller and satellite traffic, by applying the following YAML configuration:

Listing 4. linstor-ca-issuer.yaml
---
apiVersion: piraeus.io/v1
kind: LinstorCluster
metadata:
  name: linstorcluster
spec:
  internalTLS:
    certManager:
      name: linstor-internal-ca
      kind: Issuer
---
apiVersion: piraeus.io/v1
kind: LinstorSatelliteConfiguration
metadata:
  name: internal-tls
spec:
  internalTLS:
    certManager:
      name: linstor-internal-ca
      kind: Issuer

After applying the configurations above to your deployment, you can verify that TLS traffic encryption is working.

Provisioning Keys and Certificates By Using OpenSSL

If you completed the Provisioning Keys and Certificates By Using cert-manager section above, you can skip this section and go to the Verifying TLS Configuration section.

This method requires the openssl program on the command line.

First, create a new CA by using a new key and a self-signed certificate. You can change options such as the encryption algorithm and expiry time to suit the requirements of your deployment.

# openssl req -new -newkey rsa:4096 -days 3650 -nodes -x509 \
-subj "/CN=linstor-internal-ca" \
-keyout ca.key -out ca.crt

Next, create two new keys, one for the LINSTOR controller, one for all satellites:

# openssl genrsa -out controller.key 4096
# openssl genrsa -out satellite.key 4096

Next, create a certificate for each key, valid for 10 years, signed by the CA that you created earlier:

# openssl req -new -sha256 -key controller.key -subj "/CN=linstor-controller" -out controller.csr
# openssl req -new -sha256 -key satellite.key -subj "/CN=linstor-satellite" -out satellite.csr
# openssl x509 -req -in controller.csr -CA ca.crt -CAkey ca.key \
-CAcreateserial -out controller.crt -days 3650 -sha256
# openssl x509 -req -in satellite.csr -CA ca.crt -CAkey ca.key \
-CAcreateserial -out satellite.crt -days 3650 -sha256

Next, create Kubernetes secrets from the created keys and certificates:

# kubectl create secret generic linstor-controller-internal-tls -n linbit-sds \
--type=kubernetes.io/tls --from-file=ca.crt=ca.crt --from-file=tls.crt=controller.crt \
--from-file=tls.key=controller.key
# kubectl create secret generic linstor-satellite-internal-tls -n linbit-sds \
--type=kubernetes.io/tls --from-file=ca.crt=ca.crt --from-file=tls.crt=satellite.crt \
--from-file=tls.key=satellite.key

Finally, configure the Operator resources to reference the newly created secrets, by applying the following YAML configuration to your deployment:

Listing 5. linstor-internal-tls-secret.yaml
---
apiVersion: piraeus.io/v1
kind: LinstorCluster
metadata:
  name: linstorcluster
spec:
  internalTLS:
    secretName: linstor-controller-internal-tls
---
apiVersion: piraeus.io/v1
kind: LinstorSatelliteConfiguration
metadata:
  name: internal-tls
spec:
  internalTLS:
    secretName: linstor-satellite-internal-tls
Verifying TLS Configuration

After configuring LINSTOR controller and satellite traffic encryption, you can next verify the secure TLS connection between the LINSTOR controller and a satellite by examining the output of a kubectl linstor node list command. If TLS is enabled, the output will show (SSL) next to an active satellite address.

|=====================================================================|
# kubectl linstor node list
+---------------------------------------------------------------------+
| Node               | NodeType  | Addresses                 | State  |
| node01.example.com | SATELLITE | 10.116.72.142:3367 (SSL)  | Online |
| node02.example.com | SATELLITE | 10.127.183.140:3367 (SSL) | Online |
| node03.example.com | SATELLITE | 10.125.97.50:3367 (SSL)   | Online |
+---------------------------------------------------------------------+
The above command relies on the kubectl-linstor command to simplify entering LINSTOR client commands in Kubernetes. You can install the tool by following the instructions in Simplifying LINSTOR Client Command Entry.

If the output shows (PLAIN) rather than (SSL), this indicates that the TLS configuration was not applied successfully. Check the status of the LinstorCluster and LinstorSatellite resources.

If the output shows (SSL), but the node remains offline, this usually indicates that a certificate is not trusted by the other party. Verify that the controller’s tls.crt is trusted by the satellite’s ca.crt and vice versa. The following shell function provides a quick way to verify that one TLS certificate is trusted by another:

function k8s_secret_trusted_by() {
	kubectl get secret -n linbit-sds \
    -ogo-template='{{ index .data "tls.crt" | base64decode }}' \
    "$1" > $1.tls.crt
	kubectl get secret -n linbit-sds \
    -ogo-template='{{ index .data "ca.crt" | base64decode }}' \
    "$2" > $2.ca.crt
	openssl verify -CAfile $2.ca.crt $1.tls.crt
}
# k8s_secret_trusted_by satellite-tls controller-tls

If TLS encryption was properly configured, output from running the above function should be:

satellite-tls.tls.crt: OK

The upstream Piraeus project’s reference documentation shows all available LinstorCluster and LinstorSatelliteConfiguration resources options related to TLS.

Configuring TLS for the LINSTOR API

This section describes how to set up TLS for the LINSTOR API. The API, served by the LINSTOR controller, is used by clients such as the CSI Driver and the Operator itself to control the LINSTOR cluster.

To follow the instructions in this section, you should be familiar with:

  • Editing LinstorCluster resources

  • Using either cert-manager or OpenSSL to create TLS certificates

Provisioning Keys and Certificates By Using cert-manager

This method requires a working cert-manager deployment in your cluster. For an alternative way to provision keys and certificates, see the OpenSSL section below.

When using TLS, the LINSTOR API uses client certificates for authentication. It is good practice to have a separate CA just for these certificates. To do this, first apply the following YAML configuration to your deployment to create a certificate issuer.

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: ca-bootstrapper
  namespace: linbit-sds
spec:
  selfSigned: { }
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: linstor-api-ca
  namespace: linbit-sds
spec:
  commonName: linstor-api-ca
  secretName: linstor-api-ca
  duration: 87600h # 10 years
  isCA: true
  usages:
    - signing
    - key encipherment
    - cert sign
  issuerRef:
    name: ca-bootstrapper
    kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: linstor-api-ca
  namespace: linbit-sds
spec:
  ca:
    secretName: linstor-api-ca

Next, configure this issuer to let the Operator provision the needed certificates, by applying the following configuration.

---
apiVersion: piraeus.io/v1
kind: LinstorCluster
metadata:
  name: linstorcluster
spec:
  apiTLS:
    certManager:
      name: linstor-api-ca
      kind: Issuer

This completes the necessary steps for securing the LINSTOR API with TLS by using cert-manager. Skip to the Verifying LINSTOR API TLS Configuration section to verify that TLS is working.

Provisioning Keys and Certificates By Using OpenSSL

This method requires the openssl program on the command line. For an alternative way to provision keys and certificates, see the cert-manager section above.

First, create a new certificate authority (CA) by using a new key and a self-signed certificate. You can change options such as the encryption algorithm and expiry time to suit the requirements of your deployment.

# openssl req -new -newkey rsa:4096 -days 3650 -nodes -x509 \
-subj "/CN=linstor-api-ca" \
-keyout ca.key -out ca.crt

Next, create two new keys, one for the LINSTOR API server, and one for all LINSTOR API clients:

# openssl genrsa -out api-server.key 4096
# openssl genrsa -out api-client.key 4096

Next, create a certificate for the server. Because the clients might use different shortened service names, you need to specify multiple subject names:

# cat /etc/ssl/openssl.cnf > api-csr.cnf
# cat >> api-csr.cnf <<EOF
[ v3_req ]
subjectAltName = @alt_names
[ alt_names ]
DNS.0 = linstor-controller.linbit-sds.svc.cluster.local
DNS.1 = linstor-controller.linbit-sds.svc
DNS.2 = linstor-controller
EOF
# openssl req -new -sha256 -key api-server.key \
-subj "/CN=linstor-controller" -config api-csr.cnf \
-extensions v3_req -out api-server.csr
# openssl x509 -req -in api-server.csr -CA ca.crt -CAkey ca.key \
-CAcreateserial -config api-csr.cnf \
-extensions v3_req -out api-server.crt \
-days 3650 -sha256

For the client certificate, setting one subject name is enough.

# openssl req -new -sha256 -key api-client.key \
-subj "/CN=linstor-client" -out api-client.csr
# openssl x509 -req -in api-client.csr \
-CA ca.crt -CAkey ca.key -CAcreateserial \
-out api-client.crt \
-days 3650 -sha256

Next, create Kubernetes secrets from the created keys and certificates.

# kubectl create secret generic linstor-api-tls -n linbit-sds \
--type=kubernetes.io/tls --from-file=ca.crt=ca.crt --from-file=tls.crt=api-server.crt \
--from-file=tls.key=api-server.key
# kubectl create secret generic linstor-client-tls -n linbit-sds \
--type=kubernetes.io/tls --from-file=ca.crt=ca.crt --from-file=tls.crt=api-client.crt \
--from-file=tls.key=api-client.key

Finally, configure the Operator resources to reference the newly created secrets. For simplicity, you can configure the same client secret for all components.

apiVersion: piraeus.io/v1
kind: LinstorCluster
metadata:
  name: linstorcluster
spec:
  apiTLS:
    apiSecretName: linstor-api-tls
    clientSecretName: linstor-client-tls
    csiControllerSecretName: linstor-client-tls
    csiNodeSecretName: linstor-client-tls
Verifying LINSTOR API TLS Configuration

You can verify that the API is running, secured by TLS, by manually connecting to the HTTPS endpoint using a curl command.

# kubectl exec -n linbit-sds deploy/linstor-controller -- \
curl --key /etc/linstor/client/tls.key \
--cert /etc/linstor/client/tls.crt \
--cacert /etc/linstor/client/ca.crt \
https://linstor-controller.linbit-sds.svc:3371/v1/controller/version

If the command is successful, the API is using HTTPS, clients are able to connect to the controller with their certificates, and the command output should show something similar to this:

{"version":"1.20.2","git_hash":"58a983a5c2f49eb8d22c89b277272e6c4299457a","build_time":"2022-12-14T14:21:28+00:00","rest_api_version":"1.16.0"}%

If the command output shows an error, verify that the client certificates are trusted by the API secret, and vice versa. The following shell function provides a quick way to verify that one TLS certificate is trusted by another:

function k8s_secret_trusted_by() {
    kubectl get secret -n linbit-sds \
    -ogo-template='{{ index .data "tls.crt" | base64decode }}' \
    "$1" > $1.tls.crt
    kubectl get secret -n linbit-sds \
    -ogo-template='{{ index .data "ca.crt" | base64decode }}' \
    "$2" > $2.ca.crt
    openssl verify -CAfile $2.ca.crt $1.tls.crt
}
# k8s_secret_trusted_by satellite-tls controller-tls

If TLS encryption was properly configured, output from running the above function should be:

satellite-tls.tls.crt: OK

Another issue might be the API endpoint using a certificate that is not using the expected service name. A typical error message for this issue would be:

curl: (60) SSL: no alternative certificate subject name matches target host name 'linstor-controller.piraeus-datastore.svc'

In this case, make sure you have specified the right subject names when provisioning the certificates. All available options are documented in the upstream Piraeus project’s reference documentation for LinstorCluster.

Creating a Passphrase For LINSTOR

LINSTOR can use a passphrase for operations such as encrypting volumes and storing access credentials for backups.

To configure a LINSTOR passphrase in a Kubernetes deployment, the referenced secret must exist in the same namespace as the operator (by default linbit-sds), and have a MASTER_PASSPHRASE entry.

The following example YAML configuration for the .spec.linstorPassphraseSecret configures a passphrase example-passphrase.

Choose a different passphrase for your deployment.
---
apiVersion: v1
kind: Secret
metadata:
  name: linstor-passphrase
  namespace: linbit-sds
data:
  # CHANGE THIS TO USE YOUR OWN PASSPHRASE!
  # Created by: echo -n "example-passphrase" | base64
  MASTER_PASSPHRASE: ZXhhbXBsZS1wYXNzcGhyYXNl
---
apiVersion: piraeus.io/v1
kind: LinstorCluster
metadata:
  name: linstorcluster
spec:
  linstorPassphraseSecret: linstor-passphrase

5.3.7. Using CustomResourceDefinitions in Operator v2 Deployments

Within LINSTOR Operator v2 deployments, you can change the cluster state by modifying LINSTOR related Kubernetes CustomResourceDefinitions (CRDs) or check the status of a resource. An overview list of these resources follows. Refer to the upstream Piraeus project’s API reference (linked for each resource below) for more details. LinstorCluster

This resource controls the state of the LINSTOR cluster and integration with Kubernetes. LinstorSatelliteConfiguration:: This resource controls the state of the LINSTOR satellites, optionally applying it to only a subset of nodes. LinstorSatellite:: This resource controls the state of a single LINSTOR satellite. This resource is not intended to be changed directly, rather it is created by the LINSTOR Operator by merging all matching LinstorSatelliteConfiguration resources. LinstorNodeConnection:: This resource controls the state of the LINSTOR node connections. ==== Next Steps After Deploying LINSTOR Operator v2

After deploying LINBIT SDS for Kubernetes, you can continue with the [s-kubernetes-basic-configuration-and-deployment], Configuring the DRBD Module Loader in Operator v2 Deployments, Using the Host Network for DRBD Replication in Operator v2 Deployments sections in this chapter, or refer to the available tutorials in the upstream Piraeus project. === Deploying LINSTOR Operator v1

If you plan to deploy LINSTOR Operator on a new cluster, you should use Operator v2. If you have already deployed the LINSTOR Operator v2, you can skip this section and proceed to other topics in the chapter, beginning with [s-kubernetes-deploy-external-controller].

The Operator v1 is installed using a Helm v3 chart as follows:

  • Create a Kubernetes secret containing your my.linbit.com credentials:

    kubectl create secret docker-registry drbdiocred --docker-server=drbd.io \
      --docker-username=<YOUR_LOGIN> --docker-email=<YOUR_EMAIL> --docker-password=<YOUR_PASSWORD>

    The name of this secret must match the one specified in the Helm values, by default drbdiocred.

  • Configure the LINSTOR database back end. By default, the chart configures etcd as database back end. The Operator can also configure LINSTOR to use Kubernetes as datastore directly. If you go the etcd route, you should configure persistent storage for it:

    • Use an existing storage provisioner with a default StorageClass.

    • Use hostPath volumes.

    • Disable persistence, for basic testing only. This can be done by adding --set etcd.persistentVolume.enabled=false to the helm install command below.

  • Read the storage guide and configure a basic storage setup for LINSTOR

  • Read the section on securing the deployment and configure as needed.

  • Select the appropriate kernel module injector using --set with the helm install command in the final step.

    • Choose the injector according to the distribution you are using. Select the latest version from one of drbd9-rhel7, drbd9-rhel8,…​ from http://drbd.io/ as appropriate. The drbd9-rhel8 image should also be used for RHCOS (OpenShift). For the SUSE CaaS Platform use the SLES injector that matches the base system of the CaaS Platform you are using (e.g., drbd9-sles15sp1). For example:

      operator.satelliteSet.kernelModuleInjectionImage=drbd.io/drbd9-rhel8:v9.1.8
    • Only inject modules that are already present on the host machine. If a module is not found, it will be skipped.

      operator.satelliteSet.kernelModuleInjectionMode=DepsOnly
    • Disable kernel module injection if you are installing DRBD by other means. Deprecated by DepsOnly

      operator.satelliteSet.kernelModuleInjectionMode=None
  • Finally create a Helm deployment named linstor-op that will set up everything.

    helm repo add linstor https://charts.linstor.io
    helm install linstor-op linstor/linstor

    Further deployment customization is discussed in the advanced deployment section ==== LINSTOR向けKubernetesバックエンド

The LINSTOR controller can use the Kubernetes API directly to persist its cluster state. To enable this back end, use the following override file during the chart installation:

Listing 6. k8s-backend.yaml
etcd:
  enabled: false
operator:
  controller:
    dbConnectionURL: k8s
It is possible to migrate an existing cluster that uses an etcd back end to a Kubernetes API back end, by following the migration instructions at charts.linstor.io. ==== 永続ストレージボリュームの作成

You can use the pv-hostpath Helm templates to create hostPath persistent volumes. Create as many PVs as needed to satisfy your configured etcd replicas (default 1).

Create the hostPath persistent volumes, substituting cluster node names accordingly in the nodes= option:

helm repo add linstor https://charts.linstor.io
helm install linstor-etcd linstor/pv-hostpath

By default, a PV is created on every control-plane node. You can manually select the storage nodes by passing --set "nodes={<NODE0>,<NODE1>,<NODE2>}" to the install command. NOTE: The correct value to reference the node is the value of the kubernetes.io/hostname label. You can list the value for all nodes by running kubectl get nodes -o custom-columns="Name:{.metadata.name},NodeName:{.metadata.labels['kubernetes\.io/hostname']}" ==== 既存のデータベースの使用

LINSTOR can connect to an existing PostgreSQL, MariaDB or etcd database. For instance, for a PostgreSQL instance with the following configuration:

POSTGRES_DB: postgresdb
POSTGRES_USER: postgresadmin
POSTGRES_PASSWORD: admin123

The Helm chart can be configured to use this database rather than deploying an etcd cluster, by adding the following to the Helm install command:

--set etcd.enabled=false --set "operator.controller.dbConnectionURL=jdbc:postgresql://postgres/postgresdb?user=postgresadmin&password=admin123"

5.3.8. Configuring Storage With Operator v1

The LINSTOR Operator v1 can automate some basic storage set up for LINSTOR.

ストレージプール作成の構成

The LINSTOR Operator can be used to create LINSTOR storage pools. Creation is under control of the LinstorSatelliteSet resource:

$ kubectl get LinstorSatelliteSet.linstor.linbit.com linstor-op-ns -o yaml
kind: LinstorSatelliteSet
metadata:
[...]
spec:
  [...]
  storagePools:
    lvmPools:
    - name: lvm-thick
      volumeGroup: drbdpool
    lvmThinPools:
    - name: lvm-thin
      thinVolume: thinpool
      volumeGroup: ""
    zfsPools:
    - name: my-linstor-zpool
      zPool: for-linstor
      thin: true
インストール時のストレージプール作成

At installation time, by setting the value of operator.satelliteSet.storagePools when running the helm install command.

First create a file with the storage configuration such as:

operator:
  satelliteSet:
    storagePools:
      lvmPools:
      - name: lvm-thick
        volumeGroup: drbdpool

This file can be passed to the Helm installation by entering the following command:

helm install -f <file> linstor-op linstor/linstor
ストレージプール作成の構成

On a cluster with the operator already configured (that is, after helm install), you can edit the LinstorSatelliteSet configuration by entering the following command:

$ kubectl edit LinstorSatelliteSet.linstor.linbit.com <satellitesetname>

The storage pool configuration can be updated as in the example above.

物理デバイスの準備

By default, LINSTOR expects the referenced VolumeGroups, ThinPools and so on to be present. You can use the devicePaths: [] option to let LINSTOR automatically prepare devices for the pool. Eligible for automatic configuration are block devices that:

  • Are a root device (no partition)

  • do not contain partition information

  • have more than 1 GiB

To enable automatic configuration of devices, set the devicePaths key on storagePools entries:

  storagePools:
    lvmPools:
    - name: lvm-thick
      volumeGroup: drbdpool
      devicePaths:
      - /dev/vdb
    lvmThinPools:
    - name: lvm-thin
      thinVolume: thinpool
      volumeGroup: linstor_thinpool
      devicePaths:
      - /dev/vdc
      - /dev/vdd

現在、このメソッドは LVM および LVMTHIN ストレージプールの作成をサポートしています。

LVM ストレージプールの設定

The available keys for lvmPools entries are: * name name of the LINSTOR storage pool. [Required] * volumeGroup name of the VG to create. [Required] * devicePaths devices to configure for this pool. Must be empty and >= 1GiB to be recognized. [Optional] * raidLevel LVM raid level. [Optional] * vdo Enable [VDO] (requires VDO tools in the satellite). [Optional] * vdoLogicalSizeKib Size of the created VG (expected to be bigger than the backing devices by using VDO). [Optional] * vdoSlabSizeKib Slab size for VDO. [Optional] [VDO]: https://www.redhat.com/en/blog/look-vdo-new-linux-compression-layer

LVM Thin Poolの設定
  • name name of the LINSTOR storage pool. [Required]

  • volumeGroup VG to use for the thin pool. If you want to use devicePaths, you must set this to "". This is required because LINSTOR does not allow configuration of the VG name when preparing devices. thinVolume name of the thin pool. [Required]

  • devicePaths devices to configure for this pool. Must be empty and >= 1GiB to be recognized. [Optional]

  • raidLevel LVM raid level. [Optional]

The volume group created by LINSTOR for LVM thin pools will always follow the scheme “linstor_$THINPOOL”.
ZFS ストレージプールの設定
  • name name of the LINSTOR storage pool. [Required]

  • zPool name of the zpool to use. Must already be present on all machines. [Required]

  • thin true to use thin provisioning, false otherwise. [Required]

automaticStorageType の使用(非推奨)

ALL eligible devices will be prepared according to the value of operator.satelliteSet.automaticStorageType, unless they are already prepared using the storagePools section. Devices are added to a storage pool based on the device name (that is, all /dev/nvme1 devices will be part of the pool autopool-nvme1) The possible values for operator.satelliteSet.automaticStorageType: * None no automatic set up (default) * LVM create a LVM (thick) storage pool * LVMTHIN create a LVM thin storage pool * ZFS create a ZFS based storage pool (UNTESTED) ==== Securing Operator v1 Deployment

This section describes the different options for enabling security features available when using a LINSTOR Operator v1 deployment (using Helm) in Kubernetes.

既存の etcd インスタンスとの安全な通信

Secure communication to an etcd instance can be enabled by providing a CA certificate to the operator in form of a Kubernetes secret. The secret has to contain the key ca.pem with the PEM encoded CA certificate as value.

The secret can then be passed to the controller by passing the following argument to helm install

--set operator.controller.dbCertSecret=<secret name>
証明書を使用した etcd での認証

If you want to use TLS certificates to authenticate with an etcd database, you need to set the following option on Helm install:

--set operator.controller.dbUseClientCert=true

If this option is active, the secret specified in the above section must contain two additional keys:

  • client.cert PEM formatted certificate presented to etcd for authentication

  • client.key private key in PKCS8 format, matching the above client certificate.

Keys can be converted into PKCS8 format using openssl:

openssl pkcs8 -topk8 -nocrypt -in client-key.pem -out client-key.pkcs8

5.3.9. Configuring Secure Communication Between LINSTOR Components in Operator v1 Deployments

The default communication between LINSTOR components is not secured by TLS. If this is needed for your setup, choose one of three methods: // “cert-manager” is a product name so keep the original case

cert-managerを用いた鍵、証明書の生成

Requires cert-manager to be installed in your cluster.

Set the following options in your Helm override file:

linstorSslMethod: cert-manager
linstorHttpsMethod: cert-manager
Generate Keys and Certificates Using Helm

Set the following options in your Helm override file:

linstorSslMethod: helm
linstorHttpsMethod: helm
鍵や証明書の手動生成

Create a private key and self-signed certificate for your certificate authorities:

openssl req -new -newkey rsa:2048 -days 5000 -nodes -x509 -keyout ca.key \
  -out ca.crt -subj "/CN=linstor-system"
openssl req -new -newkey rsa:2048 -days 5000 -nodes -x509 -keyout client-ca.key \
  -out client-ca.crt -subj "/CN=linstor-client-ca"

Create private keys, two for the controller, one for all nodes and one for all clients:

openssl genrsa -out linstor-control.key 2048
openssl genrsa -out linstor-satellite.key 2048
openssl genrsa -out linstor-client.key 2048
openssl genrsa -out linstor-api.key 2048

Create trusted certificates for controller and nodes:

openssl req -new -sha256 -key linstor-control.key -subj "/CN=system:control" \
  -out linstor-control.csr
openssl req -new -sha256 -key linstor-satellite.key -subj "/CN=system:node" \
  -out linstor-satellite.csr
openssl req -new -sha256 -key linstor-client.key -subj "/CN=linstor-client" \
  -out linstor-client.csr
openssl req -new -sha256 -key linstor-api.key -subj "/CN=linstor-controller" \
  -out  linstor-api.csr
openssl x509 -req -in linstor-control.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out linstor-control.crt -days 5000 -sha256
openssl x509 -req -in linstor-satellite.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out linstor-satellite.crt -days 5000 -sha256
openssl x509 -req -in linstor-client.csr -CA client-ca.crt -CAkey client-ca.key \
  -CAcreateserial -out linstor-client.crt -days 5000 -sha256
openssl x509 -req -in linstor-api.csr -CA client-ca.crt -CAkey client-ca.key \
  -CAcreateserial -out linstor-api.crt -days 5000 -sha256 -extensions 'v3_req' \
  -extfile <(printf '%s\n' '[v3_req]' extendedKeyUsage=serverAuth \
  subjectAltName=DNS:linstor-op-cs.default.svc)
linstor-op-cs.default.svc in the last command needs to match create service name. With Helm, this is always <release-name>-cs.<namespace>.svc.

Create Kubernetes secrets that can be passed to the controller and node pods:

kubectl create secret generic linstor-control --type=kubernetes.io/tls \
  --from-file=ca.crt=ca.crt --from-file=tls.crt=linstor-control.crt \
  --from-file=tls.key=linstor-control.key
kubectl create secret generic linstor-satellite --type=kubernetes.io/tls \
  --from-file=ca.crt=ca.crt --from-file=tls.crt=linstor-satellite.crt \
  --from-file=tls.key=linstor-satellite.key
kubectl create secret generic linstor-api --type=kubernetes.io/tls \
  --from-file=ca.crt=client-ca.crt --from-file=tls.crt=linstor-api.crt \
  --from-file=tls.key=linstor-api.key
kubectl create secret generic linstor-client --type=kubernetes.io/tls \
  --from-file=ca.crt=client-ca.crt --from-file=tls.crt=linstor-client.crt \
  --from-file=tls.key=linstor-client.key

Pass the names of the created secrets to helm install:

linstorHttpsControllerSecret: linstor-api
linstorHttpsClientSecret: linstor-client
operator:
  controller:
    sslSecret: linstor-control
  satelliteSet:
    sslSecret: linstor-satellite
LINSTOR のパスフレーズを自動設定

LINSTOR needs to store confidential data to support encrypted information. This data is protected by a master passphrase. A passphrase is automatically generated on the first chart install.

If you want to use a custom passphrase, store it in a secret:

kubectl create secret generic linstor-pass --from-literal=MASTER_PASSPHRASE=<password>

On install, add the following arguments to the Helm command:

--set operator.controller.luksSecret=linstor-pass

5.3.10. Helm Installation Examples for Operator v1

All the below examples use the following sp-values.yaml file. Feel free to adjust this for your uses and environment. See [Configuring storage pool creation] for further details.

operator:
  satelliteSet:
    storagePools:
      lvmThinPools:
      - name: lvm-thin
        thinVolume: thinpool
        volumeGroup: ""
        devicePaths:
        - /dev/sdb
Default install. This does not setup any persistence for the backing etcd key-value store.
This is not suggested for any use outside of testing.
kubectl create secret docker-registry drbdiocred --docker-server=drbd.io \
  --docker-username=<YOUR_LOGIN> --docker-password=<YOUR_PASSWORD>
helm repo add linstor https://charts.linstor.io
helm install linstor-op linstor/linstor
LINBIT’s container image repository (http://drbd.io), used in the previous and upcoming kubectl create commands, is only available to LINBIT customers or through LINBIT customer trial accounts. Contact LINBIT for information on pricing or to begin a trial. Alternatively, you can use the LINSTOR SDS upstream project named Piraeus, without being a LINBIT customer.

Install with LINSTOR storage-pools defined at install through sp-values.yaml, persistent hostPath volumes, three etcd replicas, and by compiling the DRBD kernel modules for the host kernels.

This should be adequate for most basic deployments. Note that this deployment is not using the pre-compiled DRBD kernel modules just to make this command more portable. Using the pre-compiled binaries will make for a much faster install and deployment. Using the Compile option would not be suggested for use in a large Kubernetes clusters.

kubectl create secret docker-registry drbdiocred --docker-server=drbd.io \
  --docker-username=<YOUR_LOGIN> --docker-password=<YOUR_PASSWORD>
helm repo add linstor https://charts.linstor.io
helm install linstor-etcd linstor/pv-hostpath --set "nodes={<NODE0>,<NODE1>,<NODE2>}"
helm install -f sp-values.yaml linstor-op linstor/linstor --set etcd.replicas=3 \
  --set operator.satelliteSet.kernelModuleInjectionMode=Compile

Install with LINSTOR storage-pools defined at install through sp-values.yaml, use an already created PostgreSQL DB (preferably clustered), rather than etcd, and use already compiled kernel modules for DRBD.

The PostgreSQL database in this particular example is reachable through a service endpoint named postgres. PostgreSQL itself is configured with POSTGRES_DB=postgresdb, POSTGRES_USER=postgresadmin, and POSTGRES_PASSWORD=admin123

kubectl create secret docker-registry drbdiocred --docker-server=drbd.io \
  --docker-username=<YOUR_LOGIN> --docker-email=<YOUR_EMAIL> --docker-password=<YOUR_PASSWORD>
helm repo add linstor https://charts.linstor.io
helm install -f sp-values.yaml linstor-op linstor/linstor --set etcd.enabled=false \
  --set "operator.controller.dbConnectionURL=jdbc:postgresql://postgres/postgresdb?user=postgresadmin&password=admin123"

5.3.11. Helm デプロイメントの終了

To protect the storage infrastructure of the cluster from accidentally deleting vital components, it is necessary to perform some manual steps before deleting a Helm deployment.

  1. Delete all volume claims managed by LINSTOR components. You can use the following command to get a list of volume claims managed by LINSTOR. After checking that none of the listed volumes still hold needed data, you can delete them using the generated kubectl delete command.

    $ kubectl get pvc --all-namespaces -o=jsonpath='{range .items[?(@.metadata.annotations.volume\.beta\.kubernetes\.io/storage-provisioner=="linstor.csi.linbit.com")]}kubectl delete pvc --namespace {.metadata.namespace} {.metadata.name}{"\n"}{end}'
    kubectl delete pvc --namespace default data-mysql-0
    kubectl delete pvc --namespace default data-mysql-1
    kubectl delete pvc --namespace default data-mysql-2
    These volumes, once deleted, cannot be recovered.
  2. Delete the LINSTOR controller and satellite resources.

    Deployment of LINSTOR satellite and controller is controlled by the LinstorSatelliteSet and LinstorController resources. You can delete the resources associated with your deployment by using kubectl

    kubectl delete linstorcontroller <helm-deploy-name>-cs
    kubectl delete linstorsatelliteset <helm-deploy-name>-ns

    After a short wait, the controller and satellite pods should terminate. If they continue to run, you can check the above resources for errors (they are only removed after all associated pods have terminated).

  3. Delete the Helm deployment.

    If you removed all PVCs and all LINSTOR pods have terminated, you can uninstall the Helm deployment

    helm uninstall linstor-op
    Due to the Helm’s current policy, the Custom Resource Definitions named LinstorController and LinstorSatelliteSet will not be deleted by the command. More information regarding Helm’s current position on CRDs can be found here.

5.3.12. Advanced Deployment Options for Operator v1

The Helm charts provide a set of further customization options for advanced use cases.

LINBIT’s container image repository (http://drbd.io), used in the Helm chart below, is only available to LINBIT customers or through LINBIT customer trial accounts. Contact LINBIT for information on pricing or to begin a trial. Alternatively, you can use the LINSTOR SDS upstream project named Piraeus, without being a LINBIT customer.
global:
  imagePullPolicy: IfNotPresent # empty pull policy means k8s default is used ("always" if tag == ":latest", "ifnotpresent" else) (1)
  setSecurityContext: true # Force non-privileged containers to run as non-root users
# Dependency charts
etcd:
  enabled: true
  persistentVolume:
    enabled: true
    storage: 1Gi
  replicas: 1 # How many instances of etcd will be added to the initial cluster. (2)
  resources: {} # resource requirements for etcd containers (3)
  image:
    repository: gcr.io/etcd-development/etcd
    tag: v3.4.15
stork:
  enabled: false
  storkImage: docker.io/openstorage/stork:2.8.2
  schedulerImage: registry.k8s.io/kube-scheduler
  schedulerTag: ""
  replicas: 1 (2)
  storkResources: {} # resources requirements for the stork plugin containers (3)
  schedulerResources: {} # resource requirements for the kube-scheduler containers (3)
  podsecuritycontext: {}
csi:
  enabled: true
  pluginImage: "drbd.io/linstor-csi:v1.1.0"
  csiAttacherImage: registry.k8s.io/sig-storage/csi-attacher:v4.3.0
  csiLivenessProbeImage: registry.k8s.io/sig-storage/livenessprobe:v2.10.0
  csiNodeDriverRegistrarImage: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.8.0
  csiProvisionerImage: registry.k8s.io/sig-storage/csi-provisioner:v3.5.0
  csiSnapshotterImage: registry.k8s.io/sig-storage/csi-snapshotter:v6.2.1
  csiResizerImage: registry.k8s.io/sig-storage/csi-resizer:v1.8.0
  csiAttacherWorkerThreads: 10 (9)
  csiProvisionerWorkerThreads: 10 (9)
  csiSnapshotterWorkerThreads: 10 (9)
  csiResizerWorkerThreads: 10 (9)
  controllerReplicas: 1 (2)
  nodeAffinity: {} (4)
  nodeTolerations: [] (4)
  controllerAffinity: {} (4)
  controllerTolerations: [] (4)
  enableTopology: true
  resources: {} (3)
  customLabels: {}
  customAnnotations: {}
  kubeletPath: /var/lib/kubelet (7)
  controllerSidecars: []
  controllerExtraVolumes: []
  nodeSidecars: []
  nodeExtraVolumes: []
priorityClassName: ""
drbdRepoCred: drbdiocred
linstorSslMethod: "manual" # <- If set to 'helm' or 'cert-manager' the certificates will be generated automatically
linstorHttpsMethod: "manual" # <- If set to 'helm' or 'cert-manager' the certificates will be generated automatically
linstorHttpsControllerSecret: "" # <- name of secret containing linstor server certificates+key. See docs/security.md
linstorHttpsClientSecret: "" # <- name of secret containing linstor client certificates+key. See docs/security.md
controllerEndpoint: "" # <- override to the generated controller endpoint. use if controller is not deployed via operator
psp:
  privilegedRole: ""
  unprivilegedRole: ""
operator:
  replicas: 1 # <- number of replicas for the operator deployment (2)
  image: "drbd.io/linstor-operator:v1.10.4"
  affinity: {} (4)
  tolerations: [] (4)
  resources: {} (3)
  customLabels: {}
  customAnnotations: {}
  podsecuritycontext: {}
  args:
    createBackups: true
    createMonitoring: true
  sidecars: []
  extraVolumes: []
  controller:
    enabled: true
    controllerImage: "drbd.io/linstor-controller:v1.23.0"
    dbConnectionURL: ""
    luksSecret: ""
    dbCertSecret: ""
    dbUseClientCert: false
    sslSecret: ""
    affinity: {} (4)
    httpBindAddress: ""
    httpsBindAddress: ""
    tolerations: (4)
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
    resources: {} (3)
    replicas: 1 (2)
    additionalEnv: [] (5)
    additionalProperties: {} (6)
    sidecars: []
    extraVolumes: []
    customLabels: {}
    customAnnotations: {}
  satelliteSet:
    enabled: true
    satelliteImage: "drbd.io/linstor-satellite:v1.23.0"
    storagePools: {}
    sslSecret: ""
    automaticStorageType: None
    affinity: {} (4)
    tolerations: [] (4)
    resources: {} (3)
    monitoringImage: "drbd.io/drbd-reactor:v1.2.0"
    monitoringBindAddress: ""
    kernelModuleInjectionImage: "drbd.io/drbd9-rhel7:v9.1.14"
    kernelModuleInjectionMode: ShippedModules
    kernelModuleInjectionAdditionalSourceDirectory: "" (8)
    kernelModuleInjectionResources: {} (3)
    kernelModuleInjectionExtraVolumeMounts: []
    mountDrbdResourceDirectoriesFromHost: "" (10)
    additionalEnv: [] (5)
    sidecars: []
    extraVolumes: []
    customLabels: {}
    customAnnotations: {}
haController:
  enabled: false
  image: drbd.io/linstor-k8s-ha-controller:v0.3.0
  affinity: {} (4)
  tolerations: [] (4)
  resources: {} (3)
  replicas: 1 (2)
  customLabels: {}
  customAnnotations: {}
1 Sets the pull policy for all images.
2 Controls the number of replicas for each component.
3 Set container resource requests and limits. See the Kubernetes docs. Most containers need a minimal amount of resources, except for:
  • etcd.resources See the etcd docs

  • operator.controller.resources Around 700MiB memory is required

  • operater.satelliteSet.resources Around 700MiB memory is required

  • operator.satelliteSet.kernelModuleInjectionResources If kernel modules are compiled, 1GiB of memory is required.

4 Affinity and toleration determine where pods are scheduled on the cluster. See the Kubernetes docs on affinity and toleration. This might be especially important for the operator.satelliteSet and csi.node* values. To schedule a pod using a LINSTOR persistent volume, the node requires a running LINSTOR satellite and LINSTOR CSI pod.
5 Sets additional environments variables to pass to the LINSTOR controller and satellites. Uses the same format as the env value of a container
6 Sets additional properties on the LINSTOR controller. Expects a simple mapping of <property-key>: <value>.
7 kubelet expects every CSI plugin to mount volumes under a specific subdirectory of its own state directory. By default, this state directory is /var/lib/kubelet. Some Kubernetes distributions use a different directory:
  • microk8s: /var/snap/microk8s/common/var/lib/kubelet

8 Directory on the host that is required for building kernel modules. Only needed if using the Compile injection method. Defaults to /usr/src, which is where the actual kernel sources are stored on most distributions. Use "none" to not mount any additional directories.
9 Set the number of worker threads used by the CSI driver. Higher values put more load on the LINSTOR controller, which might lead to instability when creating many volumes at once.
10 If set to true, the satellite containers will have the following files and directories mounted from the host operating system:
  • /etc/drbd/drbd.conf (file)

  • /etc/drbd.d (directory)

  • /var/lib/drbd (directory)

  • /var/lib/linstor.d (directory)

    All files and directories must already exist on the host.

5.3.13. High-Availability Deployment in Operator v1

To create a high-availability deployment of all components within a LINSTOR Operator v1 deployment, consult the upstream guide The default values are chosen so that scaling the components to multiple replicas ensures that the replicas are placed on different nodes. This ensures that a single node failures will not interrupt the service. NOTE: If you have deployed LINBIT SDS in Kubernetes by using the LINSTOR Operator v2, high availability is built into the deployment by default. ===== 高可用性コントローラーを使用した高速ワークロードフェイルオーバー

When node failures occur, Kubernetes is very conservative in rescheduling stateful workloads. This means it can take more than 15 minutes for Pods to be moved from unreachable nodes. With the information available to DRBD and LINSTOR, this process can be sped up significantly.

The LINSTOR High Availability Controller (HA Controller) speeds up the failover process for stateful workloads using LINSTOR for storage. It monitors and manages any Pod that is attached to at least one DRBD resource.

For the HA Controller to work properly, you need quorum, that is at least three replicas (or two replicas + one diskless tiebreaker). If using lower replica counts, attached Pods will be ignored and are not eligible for faster failover.

The HA Controller is packaged as a Helm chart, and can be deployed using:

$ helm repo update
$ helm install linstor-ha-controller linstor/linstor-ha-controller

If you are using the HA Controller in your cluster you can set additional parameters in all StorageClasses. These parameters ensure that the volume is not accidentally remounted as read-only, leading to degraded Pods.

parameters:
  property.linstor.csi.linbit.com/DrbdOptions/auto-quorum: suspend-io
  property.linstor.csi.linbit.com/DrbdOptions/Resource/on-no-data-accessible: suspend-io
  property.linstor.csi.linbit.com/DrbdOptions/Resource/on-suspended-primary-outdated: force-secondary
  property.linstor.csi.linbit.com/DrbdOptions/Net/rr-conflict: retry-connect

To exempt a Pod from management by the HA Controller, add the following annotation to the Pod:

$ kubectl annotate pod <podname> drbd.linbit.com/ignore-fail-over=""

5.3.14. Backing up the etcd Database

To create a backup of the etcd database (in LINSTOR Operator v1 deployments) and store it on your control host, enter the following commands:

kubectl exec linstor-op-etcd-0 -- etcdctl snapshot save /tmp/save.db
kubectl cp linstor-op-etcd-0:/tmp/save.db save.db

These commands will create a file save.db on the machine you are running kubectl from. === LINSTORオペレーターを使用したデプロイ

The Operator can configure the satellites and CSI plugin to use an existing LINSTOR setup. This can be useful in cases where the storage infrastructure is separate from the Kubernetes cluster. Volumes can be provisioned in diskless mode on the Kubernetes nodes while the storage nodes will provide the backing disk storage. ==== Operator v2 Deployment with an External LINSTOR Controller ===== Configuring the LinstorCluster Resource

To use an externally managed LINSTOR cluster, specify the URL of the LINSTOR controller in the LinstorCluster resource in a YAML configuration and apply it to your deployment. In the following example, the LINSTOR controller is reachable at http://linstor-controller.example.com:3370.

apiVersion: piraeus.io/v1
kind: LinstorCluster
metadata:
  name: linstorcluster
spec:
  externalController:
    url: http://linstor-controller.example.com:3370
You can also specify an IP address rather than a hostname and domain for the controller. ===== Configuring Host Networking for LINSTOR Satellites

Normally the pod network is not reachable from outside the Kubernetes cluster. In this case the external LINSTOR controller would not be able to communicate with the satellites in the Kubernetes cluster. For this reason, you need to configure your satellites to use host networking.

To use host networking, deploy a LinstorSatelliteConfiguration resource by applying the following YAML configuration to your deployment:

apiVersion: piraeus.io/v1
kind: LinstorSatelliteConfiguration
metadata:
  name: host-network
spec:
  patches:
    - target:
        kind: Pod
        name: satellite
      patch: |
        apiVersion: v1
        kind: Pod
        metadata:
          name: satellite
        spec:
          hostNetwork: true
Verifying an External LINSTOR Controller Configuration

You can verify that you have correctly configured your Kubernetes deployment to use an external LINSTOR controller by verifying the following:

  • The Available condition on the LinstorCluster resource reports the expected URL for the external LINSTOR controller:

    $ kubectl get LinstorCluster -ojsonpath='{.items[].status.conditions[?(@.type=="Available")].message}{"\n"}'
    Controller 1.20.3 (API: 1.16.0, Git: 8d19a891df018f6e3d40538d809904f024bfe361) reachable at 'http://linstor-controller.example.com:3370'
  • The linstor-csi-controller deployment uses the expected URL:

    $ kubectl get -n linbit-sds deployment linstor-csi-controller -ojsonpath='{.spec.template.spec.containers[?(@.name=="linstor-csi")].env[?(@.name=="LS_CONTROLLERS")].value}{"\n"}'
    http://linstor-controller.example.com:3370
  • The linstor-csi-node deployment uses the expected URL:

    $ kubectl get -n linbit-sds daemonset linstor-csi-node -ojsonpath='{.spec.template.spec.containers[?(@.name=="linstor-csi")].env[?(@.name=="LS_CONTROLLERS")].value}{"\n"}'
    http://linstor-controller.example.com:3370
  • The Kubernetes nodes are registered as satellite nodes on the LINSTOR controller:

    $ kubectl get nodes -owide
    NAME               STATUS   ROLES           AGE   VERSION   INTERNAL-IP      [...]
    k8s-1-26-10.test   Ready    control-plane   22m   v1.26.3   192.168.122.10   [...]
    [...]

    After getting the node names from the output of the above command, verify that the node names are also LINSTOR satellites by entering a LINSTOR node list command on your LINSTOR controller node.

    $ linstor node list
    ╭─────────────────────────────────────────────────────────────────────╮
    ┊ Node             ┊ NodeType  ┊ Addresses                   ┊ State  ┊
    ╞═════════════════════════════════════════════════════════════════════╡
    ┊ k8s-1-26-10.test ┊ SATELLITE ┊ 192.168.122.10:3366 (PLAIN) ┊ Online ┊
    [...]

5.3.15. Operator v1 Deployment with an External LINSTOR Controller

To skip the creation of a LINSTOR controller deployment and configure the other components to use your existing LINSTOR controller, use the following options when running helm install:

  • operator.controller.enabled=false This disables creation of the LinstorController resource

  • operator.etcd.enabled=false Since no LINSTOR controller will run on Kubernetes, no database is required.

  • controllerEndpoint=<url-of-linstor-controller> The HTTP endpoint of the existing LINSTOR controller. For example: http://linstor.storage.cluster:3370/

After all pods are ready, you should see the Kubernetes cluster nodes as satellites in your LINSTOR setup.

Your Kubernetes nodes must be reachable using their IP by the controller and storage nodes.

Create a storage class referencing an existing storage pool on your storage nodes.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-on-k8s
provisioner: linstor.csi.linbit.com
parameters:
  autoPlace: "3"
  storagePool: existing-storage-pool
  resourceGroup: linstor-on-k8s

You can provision new volumes by creating PVCs using your storage class. The volumes will first be placed only on nodes with the given storage pool, that is, your storage infrastructure. Once you want to use the volume in a pod, LINSTOR CSI will create a diskless resource on the Kubernetes node and attach over the network to the diskful resource. === Kubernetes で LINSTOR の操作

The controller pod includes a LINSTOR Client, making it easy to interact directly with LINSTOR. For instance:

kubectl exec deployment/linstor-op-cs-controller -- linstor storage-pool list

5.3.16. Simplifying LINSTOR Client Command Entry

To simplify entering LINSTOR client commands within a Kubernetes deployment, you can use the kubectl-linstor utility. This utility is available from the upstream Piraeus datastore project. To download it, enter the following commands on your Kubernetes control plane node:

# KL_VERS=0.3.0 (1)
# KL_ARCH=linux_amd64 (2)
# curl -L -O \
https://github.com/piraeusdatastore/kubectl-linstor/releases/download/v$KL_VERS/kubectl-linstor_v"$KL_VERS"_$KL_ARCH.tar.gz
1 Set the shell variable KL_VERS to the latest release version of the kubectl-linstor utility, as shown on the kubectl-linstor releases page.
2 Set the shell variable KL_ARCH to the architecture appropriate to your deployment and supported by the utility’s available releases.
If your deployment uses the LINSTOR Operator v2, you must use version 0.2.0 or higher of the kubectl-linstor utility.
It is possible that the tar archive asset name could change over time. If you have issues downloading the asset by using the commands shown above, verify the naming convention of the asset that you want on the kubectl-linstor releases page or else manually download the asset that you want from the releases page.

To install the utility, first extract it and then move the extracted executable file to a directory in your $PATH, for example, /usr/bin. Then you can use kubectl-linstor to get access to the complete LINSTOR CLI.

$ kubectl linstor node list
╭────────────────────────────────────────────────────────────────────────────────────╮
┊ Node                           ┊ NodeType   ┊ Addresses                   ┊ State  ┊
╞════════════════════════════════════════════════════════════════════════════════════╡
┊ kube-node-01.test              ┊ SATELLITE  ┊ 10.43.224.26:3366 (PLAIN)   ┊ Online ┊
┊ kube-node-02.test              ┊ SATELLITE  ┊ 10.43.224.27:3366 (PLAIN)   ┊ Online ┊
┊ kube-node-03.test              ┊ SATELLITE  ┊ 10.43.224.28:3366 (PLAIN)   ┊ Online ┊
┊ linstor-op-cs-controller-[...] ┊ CONTROLLER ┊ 172.24.116.114:3366 (PLAIN) ┊ Online ┊
╰────────────────────────────────────────────────────────────────────────────────────╯

It also expands references to PVCs to the matching LINSTOR resource.

$ kubectl linstor resource list -r pvc:my-namespace/demo-pvc-1 --all
pvc:my-namespace/demo-pvc-1 -> pvc-2f982fb4-bc05-4ee5-b15b-688b696c8526
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node              ┊ Port ┊ Usage  ┊ Conns ┊    State   ┊ CreatedOn           ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════╡
┊ pvc-[...]    ┊ kube-node-01.test ┊ 7000 ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2021-02-05 09:16:09 ┊
┊ pvc-[...]    ┊ kube-node-02.test ┊ 7000 ┊ Unused ┊ Ok    ┊ TieBreaker ┊ 2021-02-05 09:16:08 ┊
┊ pvc-[...]    ┊ kube-node-03.test ┊ 7000 ┊ InUse  ┊ Ok    ┊   UpToDate ┊ 2021-02-05 09:16:09 ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────╯

It also expands references of the form pod:[<namespace>/]<podname> into a list resources in use by the pod. This should only be necessary for investigating problems and accessing advanced functionality. Regular operation such as creating volumes should be achieved through the Kubernetes integration. === Getting Started with LINBIT SDS Storage in Kubernetes

Once all linstor-csi Pods are up and running, you can provision volumes using the usual Kubernetes workflows.

Configuring the behavior and properties of LINSTOR volumes deployed through Kubernetes is accomplished using Kubernetes StorageClass objects.

The resourceGroup parameter is mandatory. Usually you want it to be unique and the same as the storage class name.

Here below is the simplest practical StorageClass that can be used to deploy volumes:

Listing 7. linstor-basic-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  # The name used to identify this StorageClass.
  name: linstor-basic-storage-class
  # The name used to match this StorageClass with a provisioner.
  # linstor.csi.linbit.com is the name that the LINSTOR CSI plugin uses to identify itself
provisioner: linstor.csi.linbit.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  # LINSTOR will provision volumes from the drbdpool storage pool configured
  # On the satellite nodes in the LINSTOR cluster specified in the plugin's deployment
  storagePool: "lvm-thin"
  resourceGroup: "linstor-basic-storage-class"
  # Setting a fstype is required for "fsGroup" permissions to work correctly.
  # Currently supported: xfs/ext4
  csi.storage.k8s.io/fstype: xfs
The storagePool value, lvm-thin in the example YAML configuration file above, must match an available LINSTOR StoragePool. You can list storage pool information using the linstor storage-pool list command, executed within the running linstor-op-cs-controller pod, or by using the kubectl linstor storage-pool list command if you have installed the kubectl-linstor utility.

You can create the storage class with the following command:

kubectl create -f linstor-basic-sc.yaml

Now that your storage class is created, you can now create a persistent volume claim (PVC) which can be used to provision volumes known both to Kubernetes and LINSTOR:

Listing 8. my-first-linstor-volume-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: my-first-linstor-volume
spec:
  storageClassName: linstor-basic-storage-class
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi

You can create the PersistentVolumeClaim with the following command:

kubectl create -f my-first-linstor-volume-pvc.yaml

This will create a PersistentVolumeClaim, but no volume will be created just yet. The storage class we used specified volumeBindingMode: WaitForFirstConsumer, which means that the volume is only created once a workload starts using it. This ensures that the volume is placed on the same node as the workload.

For our example, we create a simple Pod, which mounts or volume by referencing the PersistentVolumeClaim. .my-first-linstor-volume-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: fedora
  namespace: default
spec:
  containers:
  - name: fedora
    image: fedora
    command: [/bin/bash]
    args: ["-c", "while true; do sleep 10; done"]
    volumeMounts:
    - name: my-first-linstor-volume
      mountPath: /data
    ports:
    - containerPort: 80
  volumes:
  - name: my-first-linstor-volume
    persistentVolumeClaim:
      claimName: "my-first-linstor-volume"

You can create the Pod with the following command:

kubectl create -f my-first-linstor-volume-pod.yaml

Running kubectl describe pod fedora can be used to confirm that Pod scheduling and volume attachment succeeded. Examining the PersistentVolumeClaim, we can see that it is now bound to a volume.

To remove a volume, verify that no pod is using it and then delete the PersistentVolumeClaim using the kubectl command. For example, to remove the volume that we just made, run the following two commands, noting that the Pod must be unscheduled before the PersistentVolumeClaim will be removed:

kubectl delete pod fedora # unschedule the pod.

kubectl get pod -w # wait for pod to be unscheduled

kubectl delete pvc my-first-linstor-volume # remove the PersistentVolumeClaim, the PersistentVolume, and the LINSTOR Volume.

5.3.17. StorageClass で使用可能なパラメータ

The following storage class contains all currently available parameters to configure the provisioned storage.

linstor.csi.linbit.com/ is an optional, but recommended prefix for LINSTOR CSI specific parameters.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: full-example
provisioner: linstor.csi.linbit.com
parameters:
  # CSI related parameters
  csi.storage.k8s.io/fstype: xfs
  # LINSTOR parameters
  linstor.csi.linbit.com/autoPlace: "2"
  linstor.csi.linbit.com/placementCount: "2"
  linstor.csi.linbit.com/resourceGroup: "full-example"
  linstor.csi.linbit.com/storagePool: "my-storage-pool"
  linstor.csi.linbit.com/disklessStoragePool: "DfltDisklessStorPool"
  linstor.csi.linbit.com/layerList: "drbd storage"
  linstor.csi.linbit.com/placementPolicy: "AutoPlaceTopology"
  linstor.csi.linbit.com/allowRemoteVolumeAccess: "true"
  linstor.csi.linbit.com/encryption: "true"
  linstor.csi.linbit.com/nodeList: "diskful-a diskful-b"
  linstor.csi.linbit.com/clientList: "diskless-a diskless-b"
  linstor.csi.linbit.com/replicasOnSame: "zone=a"
  linstor.csi.linbit.com/replicasOnDifferent: "rack"
  linstor.csi.linbit.com/disklessOnRemaining: "false"
  linstor.csi.linbit.com/doNotPlaceWithRegex: "tainted.*"
  linstor.csi.linbit.com/fsOpts: "-E nodiscard"
  linstor.csi.linbit.com/mountOpts: "noatime"
  linstor.csi.linbit.com/postMountXfsOpts: "extsize 2m"
  # Linstor properties
  property.linstor.csi.linbit.com/*: <x>
  # DRBD parameters
  DrbdOptions/*: <x>

5.3.18. csi.storage.k8s.io/fstype

The csi.storage.k8s.io/fstype parameter sets the file system type to create for volumeMode: FileSystem PVCs. Currently supported are: * ext4 (default) * xfs ==== autoPlace

autoPlace is an integer that determines the amount of replicas a volume of this StorageClass will have. For instance, autoPlace: "3" will produce volumes with three-way replication. If neither autoPlace nor nodeList are set, volumes will be automatically placed on one node. IMPORTANT: If you use this option, you must not use nodeList. IMPORTANT: You have to use quotes, otherwise Kubernetes will complain about a malformed StorageClass. TIP: This option (and all options which affect auto-placement behavior) modifies the number of LINSTOR nodes on which the underlying storage for volumes will be provisioned and is orthogonal to which kubelets those volumes will be accessible from.

5.3.19. placementCount

placementCountautoPlace のエイリアスです。

5.3.20. resourceGroup

The LINSTOR Resource Group (RG) to associate with this StorageClass. If not set, a new RG will be created for each new PVC. ==== storagePool

storagePool is the name of the LINSTOR storage pool that will be used to provide storage to the newly-created volumes. CAUTION: Only nodes configured with this same storage pool with be considered for auto-placement. Likewise, for StorageClasses using nodeList all nodes specified in that list must have this storage pool configured on them. ==== disklessStoragePool

disklessStoragePool is an optional parameter that only affects LINSTOR volumes that are assigned as “diskless” to kubelets, that is, as clients. If you have a custom diskless storage pool defined in LINSTOR, you will specify that here.

5.3.21. layerList

A comma-separated list of layers to use for the created volumes. The available layers and their order are described towards the end of this section. Defaults to drbd,storage ==== placementPolicy

Select from one of the available volume schedulers: * AutoPlaceTopology, the default: Use topology information from Kubernetes together with user provided constraints (see [s-kubernetes-replicasonsame] and [s-kubernetes-replicasondifferent]). * AutoPlace Use the LINSTOR auto-placement feature, influenced by [s-kubernetes-replicasonsame] and [s-kubernetes-replicasondifferent] * FollowTopology: Use CSI Topology information to place at least one volume in each “preferred” zone. Only usable if CSI Topology is enabled. * Manual: Use only the nodes listed in nodeList and clientList. * Balanced: EXPERIMENTAL Place volumes across failure domains, using the least used storage pool on each selected node. ==== allowRemoteVolumeAccess

Control on which nodes a volume is accessible. The value for this option can take two different forms:

  • A simple "true" or "false" allows access from all nodes, or only those nodes with diskful resources.

  • Advanced rules, which allow more granular rules on which nodes can access the volume.

    The current implementation can grant access to the volume for nodes that share the same labels. For example, if you want to allow access from all nodes in the same region and zone as a diskful resource, you could use:

    parameters:
      linstor.csi.linbit.com/allowRemoteVolumeAccess: |
        - fromSame:
          - topology.kubernetes.io/region
          - topology.kubernetes.io/zone

    + You can specify multiple rules. The rules are additive, a node only need to match one rule to be assignable. ==== encryption

encryption is an optional parameter that determines whether to encrypt volumes. LINSTOR must be configured for encryption for this to work properly. ==== nodeList

nodeList is a list of nodes for volumes to be assigned to. This will assign the volume to each node and it will be replicated among all of them. This can also be used to select a single node by hostname, but it’s more flexible to use replicasOnSame to select a single node. IMPORTANT: If you use this option, you must not use autoPlace. TIP: This option determines on which LINSTOR nodes the underlying storage for volumes will be provisioned and is orthogonal from which kubelets these volumes will be accessible.

5.3.22. clientList

clientList is a list of nodes for diskless volumes to be assigned to. Use in conjunction with [s-kubernetes-nodelist]. ==== replicasOnSame

5.3.23. replicasOnDifferent

replicasOnDifferent takes a list of properties to consider, same as replicasOnSame. There are two modes of using replicasOnDifferent: * Preventing volume placement on specific nodes: + If a value is given for the property, the nodes which have that property-value pair assigned will be considered last. + Example: replicasOnDifferent: "no-csi-volumes=true" will place no volume on any node with property no-csi-volumes=true unless there are not enough other nodes to fulfill the autoPlace setting. * Distribute volumes across nodes with different values for the same key: + If no property value is given, LINSTOR will place the volumes across nodes with different values for that property if possible. + Example: Assuming there are 4 nodes, node-a1 and node-a2 are configured with zone=a. node-b1 and node-b2 are configured with zone=b. Using a StorageClass with autoPlace: "2" and replicasOnDifferent: "zone", LINSTOR will create one replica on either node-a1 or node-a2 and one replica on either node-b1 or node-b2.

5.3.24. disklessOnRemaining

ディスクフルリソースが割り当てられていないすべてのノードにディスクレスリソースを作成します。

5.3.25. doNotPlaceWithRegex

Do not place the resource on a node which has a resource with a name matching the regular expression. ==== fsOpts fsOpts is an optional parameter that passes options to the volume’s file system at creation time. IMPORTANT: These values are specific to your chosen file system. ==== mountOpts mountOpts is an optional parameter that passes options to the volume’s file system at mount time.

5.3.26. postMountXfsOpts

Extra arguments to pass to xfs_io, which gets called before right before first use of the volume. ==== property.linstor.csi.linbit.com/*

Parameters starting with property.linstor.csi.linbit.com/ are translated to LINSTOR properties that are set on the Resource Group associated with the StorageClass.

For example, to set DrbdOptions/auto-quorum to disabled, use:

property.linstor.csi.linbit.com/DrbdOptions/auto-quorum: disabled

オプションの完全なリストは こちら で入手できます。

5.3.27. DrbdOptions/*: <x>

This option is deprecated, use the more general property.linstor.csi.linbit.com/* form. Advanced DRBD options to pass to LINSTOR. For example, to change the replication protocol, use DrbdOptions/Net/protocol: "A". === スナップショット

Snapshots create a copy of the volume content at a particular point in time. This copy remains untouched when you make modifications to the volume content. This, for example, enables you to create backups of your data before performing modifications or deletions on your data. Because a backup is useless unless you have a way to restore it, this section describes how to create a snapshot, and how to restore it, for example, in the case of accidental deletion of your data. The next subsection contains instructions around snapshots within Operator v2 deployments. If you have deployed LINBIT SDS in Kubernetes by using Operator v1, skip ahead to the [s-kubernetes-add-snaphot-support-v1] subsection. ==== Working With Snapshots

Before you can add snapshot support within a LINBIT SDS deployment, you need to meet the following environment prerequisites:

  • Your cluster has a storage pool supporting snapshots. LINSTOR supports snapshots for LVM_THIN, FILE_THIN, ZFS and ZFS_THIN pools.

  • You have a StorageClass, PersistentVolumeClaim, and Deployment that uses a storage pool that supports snapshots.

  • Your cluster has a CSI snapshotter (snapshot-controller) deployed. To verify if it is already deployed, you can enter the following command:

    $ kubectl api-resources --api-group=snapshot.storage.k8s.io -oname

    Output should be similar to the following if a snapshot controller is already deployed:

    volumesnapshotclasses.snapshot.storage.k8s.io
    volumesnapshotcontents.snapshot.storage.k8s.io
    volumesnapshots.snapshot.storage.k8s.io

    If output from the command is empty, you can deploy a snapshot controller by entering the following commands:

    $ kubectl apply -k https://github.com/kubernetes-csi/external-snapshotter//client/config/crd
    $ kubectl apply -k https://github.com/kubernetes-csi/external-snapshotter//deploy/kubernetes/snapshot-controller
スナップショットの作成

To create a volume snapshot, you first need to create a volume snapshot class (VolumeSnapshotClass). This volume snapshot class will specify the linstor.csi.linbit.com provisioner, and sets the clean-up policy for the snapshots to Delete. This means that deleting the Kubernetes resources will also delete the snapshots in LINSTOR.

You can create a volume snapshot class by entering the following command:

$ kubectl apply -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: linbit-sds-snapshots
driver: linstor.csi.linbit.com
deletionPolicy: Delete
EOF

To create a snapshot, you create a VolumeSnapshot resource. The VolumeSnapshot resource needs to reference a snapshot-compatible PersistentVolumeClaim resource, and the VolumeSnapshotClass that you just created. For example, you could create a snapshot (named data-volume-snapshot-1) of a PVC named data-volume by entering the following command:

$ kubectl apply -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: data-volume-snapshot-1
spec:
  volumeSnapshotClassName: linbit-sds-snapshots
  source:
    persistentVolumeClaimName: data-volume
EOF
Verifying Snapshot Creation

You can verify the creation of a snapshot by entering the following commands:

$ kubectl wait volumesnapshot --for=jsonpath='{.status.readyToUse}'=true data-volume-snapshot-1
volumesnapshot.snapshot.storage.k8s.io/data-volume-snapshot-1 condition met
$ kubectl get volumesnapshot data-volume-snapshot-1

Output should show a table of information about the volume snapshot resource, similar to the following:

NAME                     READYTOUSE   SOURCEPVC     SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS
data-volume-snapshot-1   true         data-volume                           1Gi           linbit-sds-snapshots

You can further verify the snapshot in LINSTOR, by entering the following command:

$ kubectl -n linbit-sds exec deploy/linstor-controller -- linstor snapshot list

Output should show a table similar to the following:

+-----------------------------------------------------------------------------------------+
| ResourceName | SnapshotName   | NodeNames | Volumes  | CreatedOn           | State      |
|=========================================================================================|
| pvc-[...]    | snapshot-[...] | kube-0    | 0: 1 GiB | 2023-02-13 15:36:18 | Successful |
+-----------------------------------------------------------------------------------------+
スナップショットの復元

To restore a snapshot, you will need to create a new PVC to recover the volume snapshot to. You will replace the existing PVC, named data-volume in this example, with a new version based on the snapshot.

First, stop the deployment that uses the data-volume PVC. In this example, the deployment is named volume-logger.

$ kubectl scale deploy/volume-logger --replicas=0 deployment.apps "volume-logger" deleted $ kubectl rollout status deploy/volume-logger deployment "volume-logger" successfully rolled out

Next, remove the PVC. You still have the snapshot resource, so this is a safe operation.

$ kubectl delete pvc/data-volume persistentvolumeclaim "data-volume" deleted

Next, create a new PVC by referencing a previously created snapshot. This will create a volume which uses the data from the referenced snapshot.

kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-volume
spec:
  storageClassName: linbit-sds-storage
  resources:
    requests:
      storage: 1Gi
  dataSource:
    apiGroup: snapshot.storage.k8s.io
    kind: VolumeSnapshot
    name: data-volume-snapshot-1
  accessModes:
ReadWriteOnce
EOF

Because you named the new volume, data-volume, the same as the previous volume, you can just scale up the Deployment again, and the new pod will start using the restored volume.

$ kubectl scale deploy/volume-logger --replicas=1 deployment.apps/volume-logger scaled

5.3.28. スナップショットのS3ストレージへの保存

LINSTOR can store snapshots on S3 compatible storage for disaster recovery. This is integrated in Kubernetes using a special VolumeSnapshotClass:

---
kind: VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1
metadata:
  name: linstor-csi-snapshot-class-s3
driver: linstor.csi.linbit.com
deletionPolicy: Retain
parameters:
  snap.linstor.csi.linbit.com/type: S3
  snap.linstor.csi.linbit.com/remote-name: backup-remote
  snap.linstor.csi.linbit.com/allow-incremental: "false"
  snap.linstor.csi.linbit.com/s3-bucket: snapshot-bucket
  snap.linstor.csi.linbit.com/s3-endpoint: s3.us-west-1.amazonaws.com
  snap.linstor.csi.linbit.com/s3-signing-region: us-west-1
  snap.linstor.csi.linbit.com/s3-use-path-style: "false"
  # Refer here to the secret that holds access and secret key for the S3 endpoint.
  # See below for an example.
  csi.storage.k8s.io/snapshotter-secret-name: linstor-csi-s3-access
  csi.storage.k8s.io/snapshotter-secret-namespace: storage
---
kind: Secret
apiVersion: v1
metadata:
  name: linstor-csi-s3-access
  namespace: storage
immutable: true
type: linstor.csi.linbit.com/s3-credentials.v1
stringData:
  access-key: access-key
  secret-key: secret-key

Refer to the instructions in the Shipping Snapshots section for the exact meaning of the snap.linstor.csi.linbit.com/ parameters. The credentials used to log in are stored in a separate secret, as show in the example above.

Referencing the above storage class when creating snapshots causes the snapshots to be automatically uploaded to the configured S3 storage.

Restoring From Remote Snapshots

Restoring from remote snapshots is an important step in disaster recovery. A snapshot needs to be registered with Kubernetes before it can be used to restore.

復元すべきスナップショットがS3へのバックアップの一部である場合、LINSTOR “remote” を最初に設定します。

linstor remote create s3 backup-remote s3.us-west-1.amazonaws.com \
  snapshot-bucket us-west-1 access-key secret-key
linstor backup list backup-remote

The snapshot you want to register needs to be one of the listed snapshots.

To register the snapshot with Kubernetes, you need to create two resources, one VolumeSnapshotContent referencing the ID of the snapshot and one VolumeSnapshot, referencing the content.

---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: example-backup-from-s3
  namespace: project
spec:
  source:
    volumeSnapshotContentName: restored-snap-content-from-s3
  volumeSnapshotClassName: linstor-csi-snapshot-class-s3
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotContent
metadata:
  name: restored-snap-content-from-s3
spec:
  deletionPolicy: Delete
  driver: linstor.csi.linbit.com
  source:
    snapshotHandle: snapshot-id
  volumeSnapshotClassName: linstor-csi-snapshot-class-s3
  volumeSnapshotRef:
    apiVersion: snapshot.storage.k8s.io/v1
    kind: VolumeSnapshot
    name: example-backup-from-s3
    namespace: project

Once applied, the VolumeSnapshot should be shown as ready, at which point you can reference it as a dataSource in a PVC.

5.4. ボリュームのアクセスしやすさと局所性

ボリュームの局所性は、ストレージクラスで volumeBindingMode: WaitForFirstConsumer を設定することで実現されます。これにより、Kubernetes と CSI ドライバーは、PVC を参照する最初のコンシューマー(ポッド)がスケジュールされるまで待機するように指示されます。次に、CSI ドライバーは、コンシューマーと同じノード上の下位データを使用してボリュームをプロビジョニングします。適切なストレージプールのないノードが選択された場合、アクセス可能なノードのセットから代替ノードが選択されます(以下を参照)。

Volume accessibility is controlled by the allowRemoteVolumeAccess parameter. Whenever the CSI plugin needs to place a volume, this parameter is consulted to get the set of “accessible” nodes. This means they can share volumes placed on them through the network. This information is also propagated to Kubernetes using label selectors on the PV.

5.4.1. ボリュームのアクセスしやすさと局所性の例

次の例は、ボリュームのアクセスしやすさと局所性を最適化する一般的なシナリオを示しています。また、クラスター内のゾーン間でボリュームレプリカを分散する方法の例も含まれています。

Single-Zone Homogeneous Clusters

クラスタは単一のゾーンにのみまたがっています。つまり、ノード間の遅延は低くなっています。クラスタは同種です。つまり、すべてのノードが同様に構成され、すべてのノードに独自のローカルストレージプールがあります。

Listing 9. example-storage-class.yaml [source,yaml]
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-storage
provisioner: linstor.csi.linbit.com
volumeBindingMode: WaitForFirstConsumer (1)
parameters:
  linstor.csi.linbit.com/storagePool: linstor-pool (2)
  linstor.csi.linbit.com/placementCount: "2" (3)
  linstor.csi.linbit.com/allowRemoteVolumeAccess: "true" (4)
1 Enable late volume binding. This places one replica on the same node as the first consuming pod, if possible.
2 Set the storage pool(s) to use.
3 Ensure that the data is replicated, so that at least 2 nodes store the data.
4 Allow using the volume even on nodes without replica. Since all nodes are connected equally, performance impact should be manageable.
Multi-Zonal Homogeneous Clusters

As before, in our homogeneous cluster all nodes are configured similarly with their own local storage pool. The cluster spans now multiple zones, with increased latency across nodes in different zones. To ensure low latency, we want to restrict access to the volume with a local replica to only those zones that do have a replica. At the same time, we want to spread our data across multiple zones.

Listing 10. example-storage-class.yaml [source,yaml]
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-storage
provisioner: linstor.csi.linbit.com
volumeBindingMode: WaitForFirstConsumer (1)
parameters:
  linstor.csi.linbit.com/storagePool: linstor-pool (2)
  linstor.csi.linbit.com/placementCount: "2" (3)
  linstor.csi.linbit.com/allowRemoteVolumeAccess: | (4)
    - fromSame:
      - topology.kubernetes.io/zone
  linstor.csi.linbit.com/replicasOnDifferent: topology.kubernetes.io/zone (5)
1 Enable late volume binding. This places one replica on the same node as the first consuming pod, if possible.
2 Set the storage pool(s) to use.
3 Ensure that the data is replicated, so that at least 2 nodes store the data.
4 Allow using the volume on nodes in the same zone as a replica, under the assumption that zone internal networking is fast and low latency.
5 Spread the replicas across different zones.
Multi-Region Clusters

If your cluster spans multiple regions, you do not want to incur the latency penalty to replicate your data across regions. To accomplish this, you can configure your storage class to just replicate data in the same zone.

Listing 11. example-storage-class.yaml [source,yaml]
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-storage
provisioner: linstor.csi.linbit.com
volumeBindingMode: WaitForFirstConsumer (1)
parameters:
  linstor.csi.linbit.com/storagePool: linstor-pool (2)
  linstor.csi.linbit.com/placementCount: "2" (3)
  linstor.csi.linbit.com/allowRemoteVolumeAccess: | (4)
    - fromSame:
      - topology.kubernetes.io/zone
  linstor.csi.linbit.com/replicasOnSame: topology.kubernetes.io/region (5)
1 Enable late volume binding. This places one replica on the same node as the first consuming pod, if possible.
2 Set the storage pool(s) to use.
3 Ensure that the data is replicated, so that at least 2 nodes store the data.
4 Allow using the volume on nodes in the same zone as a replica, under the assumption that zone internal networking is fast and low latency.
5 Restrict replicas to only a single region.
外部ストレージを備えたクラスター

クラスターはローカルストレージのないノードのみで構成されています。ボリュームアクセスは、リモートボリュームアクセスを介して行う必要があります。

Listing 12. example-storage-class.yaml [source,yaml]
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-storage
provisioner: linstor.csi.linbit.com
parameters:
  linstor.csi.linbit.com/storagePool: linstor-pool (1)
  linstor.csi.linbit.com/placementCount: "1" (2)
  linstor.csi.linbit.com/allowRemoteVolumeAccess: "true" (3)
1 Set the storage pool(s) to use.
2 Assuming we only have one storage host, we can only place a single volume without additional replicas.
3 Our worker nodes need to be allowed to connect to the external storage host.

5.5. LINSTOR アフィニティ コントローラ

ボリュームのアクセシビリティは、PersistentVolume (PV) の node affinity によって制御されます。このアフィニティは静的です。つまり、一度定義すると変更できません。

This can be an issue if you want to use a strict affinity: Your PV is pinned to specific nodes, but you might want to remove or add nodes. While LINSTOR can move the volume (for example: this happens automatically if you remove a node in Kubernetes), the PV affinity is not updated to reflect this.

ここで、LINSTOR Affinity Controller の出番です。PV を監視し、それらのアフィニティを LINSTOR のボリュームの状態と比較します。それらが同期しなくなると、PV は更新されたバージョンに置き換えられます。

LINSTOR Affinity Controller は、Helm チャートにパッケージ化されています。 Operator と同じ名前空間にインストールする場合は、以下を実行するだけです。

$ helm repo update $ helm install linstor-affinity-controller linstor/linstor-affinity-controller

Additional options for the chart are available at the upstream project.

5.6. LINSTOR スケジューラーを使用したボリューム局所性の最適化

LINBIT maintains an open source plugin for the Kubernetes scheduler. The scheduler will take the current placement of volumes into account and optimize for data locality. If possible, the pod will be assigned to a node that also hosts replicas of attached volumes, reducing latency for read operations.

スケジューラーは別のチャート artifacthub.io から利用できます。チャートは、後で Pod リソースを作成するときに使用できる新しいスケジューラーをデプロイします。

apiVersion: v1
kind: Pod
metadata:
  name: busybox
spec:
  schedulerName: linstor-scheduler (1)
  containers:
  - name: busybox
    image: busybox
    command: ["tail", "-f", "/dev/null"]
    volumeMounts:
    - name: my-first-linstor-volume
      mountPath: /data
    ports:
    - containerPort: 80
  volumes:
  - name: my-first-linstor-volume
    persistentVolumeClaim:
      claimName: "test-volume"
1 Add the name of the scheduler to your pod.

5.7. Configuring the DRBD Module Loader in Operator v2 Deployments

To follow the steps in this section, you should be familiar with editing LinstorSatelliteConfiguration resources.

The DRBD module loader is the component responsible for making the DRBD kernel module available, in addition to loading other useful kernel modules for LINBIT SDS in Kubernetes. This section describes how you can configure various aspects of the DRBD kernel module loader, within a LINSTOR Operator v2 deployment.

Besides the DRBD kernel module, these modules are also loaded if available:

[cols=”1,1″]

Module

Purpose

libcrc32c

dependency for DRBD

nvmet_rdma, nvme_rdma

LINSTOR NVME layer

loop

LINSTOR when using loop devices as backing disks

dm_writecache

LINSTOR writecache layer

dm_cache

LINSTOR cache layer

dm_thin_pool

LINSTOR thin-provisioned storage

dm_snapshot

LINSTOR Snapshots

dm_crypt

LINSTOR encrypted volumes

5.7.1. Disabling the DRBD Module Loader

In some circumstances it might be necessary to disable the DRBD module loader entirely. For example, if you are using an immutable operating system, and DRBD and other modules are loaded as part of the host configuration.

To disable the DRBD module loader completely, apply the following YAML configuration to your deployment:

apiVersion: piraeus.io/v1
kind: LinstorSatelliteConfiguration
metadata:
  name: no-loader
spec:
  patches:
    - target:
        kind: Pod
        name: satellite
      patch: |
        apiVersion: v1
        kind: Pod
        metadata:
          name: satellite
        spec:
          initContainers:
          - name: drbd-module-loader
            $patch: delete

5.7.2. Selecting a Different DRBD Module Loader Version

By default, the Operator will try to find a DRBD module loader that matches the host operating system. The Operator determines the host distribution by inspecting the .status.nodeInfo.osImage field of the Kubernetes Node resource. A user-defined image can be used if the automatic mapping does not succeed or if you have different module loading requirements.

The following YAML configuration overrides the chosen DRBD module loader image with a user-defined image example.com/drbd-loader:v9:

apiVersion: piraeus.io/v1
kind: LinstorSatelliteConfiguration
metadata:
  name: custom-drbd-module-loader-image
spec:
  patches:
    - target:
        kind: Pod
        name: satellite
      patch: |
        apiVersion: v1
        kind: Pod
        metadata:
          name: satellite
        spec:
          initContainers:
          - name: drbd-module-loader
            image: example.com/drbd-loader:v9

drbd.io, available to LINBIT customers only, maintains the following module loader container images:

Image Distribution

drbd.io/drbd9-amzn2:v9.2.5

Amazon Linux 2

drbd.io/drbd9-bionic:v9.2.5

Ubuntu 18.04

drbd.io/drbd9-focal:v9.2.5

Ubuntu 20.04

drbd.io/drbd9-jammy:v9.2.5

Ubuntu 22.04

drbd.io/drbd9-rhel7:v9.2.5

Red Hat Enterprise Linux 7

drbd.io/drbd9-rhel8:v9.2.5

Red Hat Enterprise Linux 8

drbd.io/drbd9-rhel9:v9.2.5

Red Hat Enterprise Linux 9

If you need to create a module loader image for your own distribution, you can refer to the container source files which are available in the upstream Piraeus project.

5.7.3. Changing How the Module Loader Loads the DRBD Kernel Module

By default, the DRBD module loader will try to build the kernel module from source. The module loader can also be configured to load the module from a DEB or RPM package included in the image, or skip loading DRBD entirely.

To change the behavior of the DRBD module loader, set the LB_HOW environment variable to an appropriate value shown in the following table:

LB_HOW Module Loader Behavior

compile

The default value. Builds the DRBD module from source and tries to load all optional modules from the host.

shipped_modules

Searches for .rpm or .deb packages at /pkgs and inserts contained the DRBD modules. Optional modules are loaded from the host if available.

deps_only

Only tries to load the optional modules. No DRBD module will be loaded.

After setting the LB_HOW environment variable, apply the following YAML configuration to your deployment. Based on the name within the metadata section, the example below would be used with an LB_HOW environment variable that was set to deps_only.

apiVersion: piraeus.io/v1
kind: LinstorSatelliteConfiguration
metadata:
  name: no-drbd-module-loader
spec:
  patches:
    - target:
        kind: Pod
        name: satellite
      patch: |
        apiVersion: v1
        kind: Pod
        metadata:
          name: satellite
        spec:
          initContainers:
          - name: drbd-module-loader
            env:
            - name: LB_HOW
              value: deps_only

5.8. Using the Host Network for DRBD Replication in Operator v2 Deployments

Instructions in this section will describe how you can use the host network for DRBD replication traffic.

By default, DRBD will use the container network to replicate volume data. This ensures replication works on a wide range of clusters without further configuration. It also enables use of NetworkPolicy to block unauthorized access to DRBD traffic. Since the network interface of the pod is tied to the lifecycle of the pod, it also means DRBD will temporarily disrupt replication when the LINSTOR satellite pod is restarted.

In contrast, using the host network for DRBD replication will cause replication to work independently of the LINSTOR satellite pod. The host network might also offer better performance than the container network. As a downside, you will have to manually ensure connectivity between nodes on the relevant ports.

To follow the steps in this section, you should be familiar with editing LinstorSatelliteConfiguration resources.

5.8.1. Configuring DRBD Replication to Use the Host Network

Switching from the default container network to the host network for DRBD replication is possible at any time. Existing DRBD resources will then be reconfigured to use the host network interface.

To configure the host network for the LINSTOR satellite, apply the following YAML configuration to your deployment:

apiVersion: piraeus.io/v1
kind: LinstorSatelliteConfiguration
metadata:
  name: host-network
spec:
  patches:
    - target:
        kind: Pod
        name: satellite
      patch: |
        apiVersion: v1
        kind: Pod
        metadata:
          name: satellite
        spec:
          hostNetwork: true

After the satellite pods are recreated, they will use the host network. Any existing DRBD resources are reconfigured to use a new IP address on the host network rather than an IP address on the container network.

5.8.2. Configuring DRBD Replication to Use the Container Network

Switching back from host network to container network involves manually resetting the configured peer addresses used by DRBD. You can do this by rebooting every node, or by manually resetting the addresses by using the drbdadm CLI command on each node. Each method is described below.

[[s-kubernetes-drbd-replication-switching-from-host-to-container-network-node-rebooting-v2 ===== Rebooting Nodes to Switch DRBD Replication from the Host to the Container Network

First, you need to remove the LinstorSatelliteConfiguration that set hostNetwork: true. You can do this by entering the following kubectl command:

$ kubectl delete linstorsatelliteconfigurations.piraeus.io host-network
linstorsatelliteconfiguration.piraeus.io "host-network" deleted

Next, reboot each cluster node, either serially, one by one, or else all at once. In general, replication will not work between rebooted nodes and non-rebooted nodes. The non-rebooted nodes will continue to use the host network addresses, which are generally not reachable from the container network.

After all nodes have restarted, all resources will be configured to use the container network, and all DRBD connections should be connected again.

[[s-kubernetes-drbd-replication-switching-from-host-to-container-network-node-drbdadm-v2 ===== Using the DRBD Administration Tool to Switch DRBD Replication from the Host to the Container Network

During this procedure, ensure that no new volumes or snapshots are created, otherwise the migration to the container network might not be applied to all resources.

First, you need to temporarily stop all DRBD replication and suspend all DRBD volume I/O operations by using the drbdadm suspend-io all command. Enter the command once on each LINSTOR satellite pod.

$ kubectl exec ds/linstor-satellite.node1.example.com -- drbdadm suspend-io all
$ kubectl exec ds/linstor-satellite.node2.example.com -- drbdadm suspend-io all
$ kubectl exec ds/linstor-satellite.node3.example.com -- drbdadm suspend-io all

Next, disconnect all DRBD connections on all nodes.

$ kubectl exec ds/linstor-satellite.node1.example.com -- drbdadm disconnect --force all
$ kubectl exec ds/linstor-satellite.node2.example.com -- drbdadm disconnect --force all
$ kubectl exec ds/linstor-satellite.node3.example.com -- drbdadm disconnect --force all

Next, you can safely reset all DRBD connection paths. This frees the connection on each node to be moved to the container network.

$ kubectl exec ds/linstor-satellite.node1.example.com -- drbdadm del-path all
$ kubectl exec ds/linstor-satellite.node2.example.com -- drbdadm del-path all
$ kubectl exec ds/linstor-satellite.node3.example.com -- drbdadm del-path all

Finally, remove the LinstorSatelliteConfiguration resource configuration that set hostNetwork: true. This will result in the creation of new LINSTOR satellite pods that use the container network.

$ kubectl delete linstorsatelliteconfigurations.piraeus.io host-network
linstorsatelliteconfiguration.piraeus.io "host-network" deleted

After the pods are recreated and the LINSTOR satellites are Online, the DRBD resource will be reconfigured and resume I/O operations.

5.9. Kubernetes でのノードの退避

リソースの LINSTOR ノードを退避させて、クラスター内の他のノードに配置する場合、そのプロセスは ノードの退避 で詳しく説明されています。 ただし、Kubernetes で LINSTOR ノードを退避させる前に、追加のアクションを実行する必要があります。

First, prevent new Kubernetes workloads from being scheduled to the node and then move the node’s workload to another node. You can do this by entering the following commands:

# kubectl cordon <node_name>
# kubectl drain --ignore-daemonsets <node_name>

After verifying that your cluster is running as expected, you can continue to follow the steps in ノードの退避.

複数のノードの退避を計画している場合は、退避するすべてのノードで次のコマンドを入力します。

# linstor node set-property n1.k8s-mwa.at.linbit.com AutoplaceTarget false

これにより、退避しようとしているノードから、退避を計画している別のノードに LINSTOR がリソースを配置することがなくなります。

5.10. Deleting a LINSTOR Node in Kubernetes

Before you can delete a LINSTOR storage node from a Kubernetes cluster, the node must have no deployed LINSTOR resources or Kubernetes workloads. To remove resources and workloads from a node, you can follow the instructions in the Kubernetes でのノードの退避 section.

Alternatively, let the LINSTOR Operator handle the node evacuation for you. To do this, you need to take a few preparatory steps.

First, use kubectl to apply a label, marked-for-deletion in the following example, to the node that you want to delete from your Kubernetes cluster.

# kubectl label nodes <node-name> marked-for-deletion=
You must add an equals sign (=) to the end of your label name.

Next, configure the linstorcluster Kubernetes resource so that it does not deploy LINSTOR resources to any node with the label that you have applied. To do this, edit the resource and replace the contents with the lines shown in the following example, including the …​ line. Change the key to match the label that you previously applied to your node.

# kubectl edit linstorclusters linstorcluster
...
spec:
  nodeAffinity:
    nodeSelectorTerms:
    - matchExpressions:
      - key: marked-for-deletion
        operator: DoesNotExist

After taking these steps, the LINSTOR Operator will automatically delete the corresponding LinstorSatellite resource. That will stop the LINSTOR satellite service on that node, but only after LINSTOR runs a node evacuate operation and there are no more resources on the node. Enter the following command to wait for the Operator to finish evacuating the node:

# kubectl wait linstorsatellite/<node> --for=condition=EvacuationCompleted

This can take some time. You can also use kubectl describe linstorsatellite/<node> to get a status update on the evacuation process.

After the Operator finishes evacuating the node, you can delete the node by entering the following command:

# kubectl delete node <node-name>

5.11. Monitoring With Prometheus

A LINSTOR deployment in Kubernetes offers the possibility of integrating with the Prometheus monitoring stack. You can use Prometheus to monitor LINSTOR and related components in Kubernetes. This monitoring solution is configurable to suit your purposes and includes features such as a Grafana dashboard for Prometheus scraped metrics, and the ability to set up alert rules and get alerts for events.

The Prometheus monitoring integration with LINSTOR configures:

  • Metrics scraping for the LINSTOR and DRBD state.

  • Alerts based on the cluster state

  • A Grafana dashboard

To configure monitoring for your LINSTOR in Kubernetes deployment, you should be familiar with:

  • Deploying workloads in Kubernetes using Helm

  • Deploying resources using kubectl

5.11.1. Configuring Monitoring with Prometheus in Operator v2 Deployments

This section describes configuring monitoring with Prometheus for LINSTOR Operator v2 deployments in Kubernetes. If you need to configure monitoring with Prometheus for a LINSTOR Operator v1 deployment, refer to the instructions in the Monitoring with Prometheus in Operator v1 Deployments section.

Deploying the Prometheus Operator for LINSTOR Operator v2 Deployments
If you already have a working Prometheus Operator deployment, skip the steps in this section.

To deploy the Prometheus monitoring integration in Kubernetes, you need to deploy the Prometheus Operator. A simple way to do this is to use the Helm chart provided by the Prometheus Community.

First, add the Helm chart repository to your local Helm configuration, by entering the following command:

# helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Then, deploy the kube-prometheus-stack chart. This chart will set up Prometheus, Alertmanager, and Grafana for your cluster. Configure it to search for monitoring and alerting rules in all namespaces:

# helm install \
--create-namespace -n monitoring prometheus prometheus-community/kube-prometheus-stack \
--set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
--set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \
--set prometheus.prometheusSpec.ruleSelectorNilUsesHelmValues=false
By default, the deployment will only monitor resources in the kube-system and its own namespace. LINSTOR is usually deployed in a different namespace, linbit-sds, so you need to configure Prometheus to watch this namespace. In the example above, this is achieved by setting the various *NilUsesHelmValues parameters to false.
Deploying Prometheus Monitoring and Alerting Rules for LINSTOR

After creating a Prometheus Operator deployment and configuring it to watch all namespaces, apply the monitoring and alerting resources for LINSTOR. You can do this by either using the kustomize or helm utilities.

Using Kustomize to Deploy Prometheus Monitoring and Alerting Rules

To deploy Prometheus monitoring and alerting rules for LINSTOR by using the kustomize utility, you can apply a LINBIT GitHub-hosted kustomization configuration, by entering the following command:

# kubectl apply -k \
"https://github.com/linbit/linstor-operator-builder//config/monitoring?ref=v2"

If you have configured SSL/TLS for your LINSTOR deployment in Kubernetes, you will need to apply a different version of the monitoring configuration. You can do this by entering the following command:

kubectl apply -k \
"https://github.com/linbit/linstor-operator-builder//config/monitoring-with-api-tls?ref=v2"

Output from applying the monitoring configuration to your deployment should show the following:

configmap/linbit-sds-dashboard created
podmonitor.monitoring.coreos.com/linstor-satellite created
prometheusrule.monitoring.coreos.com/linbit-sds created
servicemonitor.monitoring.coreos.com/linstor-controller created
Using Helm to Deploy Prometheus Monitoring and Alerting Rules

If you are using Helm, you can deploy Prometheus monitoring for your LINSTOR in Kubernetes deployment by enabling monitoring in the linbit-sds chart. Enter the following command to do this:

# helm repo update linstor && \
helm install linbit-sds linstor/linbit-sds \
--set monitoring.enabled=true \
Unlike the kustomize deployment method, you can use the same helm install command for a regular and an SSL/TLS-enabled LINSTOR deployment. The Helm deployment configures the correct endpoint automatically.

Output from the command above should show that the linbit-sds chart was successfully deployed.

Verifying a Monitoring Deployment

You can verify that the monitoring configuration is working in your deployment by checking the local Prometheus, Alertmanager, and Grafana web consoles.

Verifying the Prometheus Web Console Deployment

First, get access to the Prometheus web console from your local browser by forwarding the Prometheus web console service to local port 9090:

# kubectl port-forward -n monitoring services/prometheus-kube-prometheus-prometheus 9090:9090
If you need to access the Prometheus instance from a system other than localhost, you might need to add the --address 0.0.0.0 argument to the previous command.

Next, in a web browser, open http://localhost:9090/graph and display the linstor_info and drbd_version metrics, for example, by entering each metric into the search field, and clicking the Execute button.

the `linstor_info` Prometheus metric
Figure 8. The linstor_info Prometheus metric
the `drbd_version` Prometheus metric
Figure 9. The drbd_version Prometheus metric
Verifying the Prometheus Alertmanager Web Console Deployment

To view the Alertmanager console, forward the Prometheus Alertmanager service to local port 9093, by entering the following command:

# kubectl port-forward -n monitoring services/prometheus-kube-prometheus-alertmanager 9093:9093
If you need to access the Prometheus Alertmanager instance from a system other than localhost, you might need to add the --address 0.0.0.0 argument to the previous command.

Next, in a web browser, open http://localhost:9093. The Alertmanager console should show an itemized list of alert groups for your deployment, including an alert group for the linbit-sds namespace.

You can verify that an alert will fire and be shown in your Alertmanager console by running a command that will cause an event applicable to an alert that you want to test. For example, you can disconnect a DRBD resource. This will cause a drbdResourceSuspended alert. To test this, assuming that you have a LINSTOR satellite pod named kube-0 that is running in the linbit-sds namespace and that the satellite is in a secondary role for a DRBD resource named my-res, enter the following command:

# kubectl exec -it -n linbit-sds kube-0 -- drbdadm disconnect --force my-res
Use caution with the types of events that you use to test alerts, especially on a production system. The above drbdadm disconnect command should be safe if you run it on a satellite node pod that is in a secondary role for the resource you disconnect. You can verify the resource role state that the satellite node pod is in by using a drbdadm status <resource-name> command.
alerts for the `my-res` DRBD resource in a disconnected state
Figure 10. Alerts for the my-res DRBD resource in a disconnected state

After verifying that the Prometheus Alertmanager web console shows new alerts that relate to the event you caused, you should revert your deployment to its previous state. To do this for the previous drbdadm disconnect example, enter the following command:

# kubectl exec -it -n linbit-sds kube-0 -- drbdadm connect my-res

You can verify that the kube-0 satellite node pod is again connected to the my-res DRBD resource by entering a drbdadm status command:

# kubectl exec -it -n linbit-sds kube-0 -- drbdadm status my-res

Output from the command should show that the resource is in an up-to-date state on all diskful and connected nodes.

my-res role:Secondary
  disk:UpToDate
  kube-1 role:Secondary
    peer-disk:Diskless
  kube-2 role:Secondary
    peer-disk:UpToDate
Verifying the Grafana Web Console and LINBIT SDS Dashboard Deployment

To view the Grafana console and LINBIT SDS dashboard, first forward the Grafana service to local port 3000, by entering the following command:

# kubectl port-forward -n monitoring services/prometheus-grafana 3000:http-web
If you need to access the Grafana instance from a system other than the localhost, you might need to add the --address 0.0.0.0 argument to the previous command.

Next, in a web browser, open http://localhost:3000 and log in. If you are using the example deployment from above, enter username admin and password prom-operator to log in to the Grafana instance. After logging in, change your password (http://192.168.121.20:3000/profile/password) to something other than the default password.

Select “LINBIT SDS” from the available dashboards (http://localhost:3000/dashboards) to show an overview of the health status of your LINSTOR deployment, including various metrics and statistics.

the LINBIT SDS Grafana dashboard
Figure 11. The LINBIT SDS Grafana dashboard

5.11.2. Monitoring with Prometheus in Operator v1 Deployments

In Operator v1 deployments, the operator will set up monitoring containers along the existing components and make them available as a Service.

Prometheus Operator を使用する場合、Linstor オペレーターは ServiceMonitor インスタンスも設定します。 Piraeus 名前空間が有効 と仮定して、メトリックはオペレーターに関連付けられた Prometheus インスタンスによって自動的に収集されます。

メトリックのエクスポートを無効にするには operator.satelliteSet.monitoringImage を空の値に設定します。

LINSTOR Controller Monitoring in Operator v1 Deployments

The LINSTOR controller exports cluster-wide metrics. Metrics are exported on the existing controller service, using the path /metrics.

DRBD Resource Monitoring in Operator v1 Deployments

すべてのサテライトは drbd-reactor を使用して DRBD から直接メトリックをエクスポートするセカンダリコンテナにバンドルされています。メトリックはポート 9942 で、ヘッドレスサービス <linstorsatelliteset-name>-monitoring という名前で提供されています。

If you want to disable the monitoring container, set monitoringImage to "" in your LinstorSatelliteSet resource.

5.12. Kubernetes での LINSTOR デプロイメントのアップグレード

A LINSTOR deployment in Kubernetes can be upgraded to a new release. It is also possible to upgrade an existing LINSTOR Operator v1 deployment in Kubernetes to an Operator v2 deployment.

During the upgrade process, attaching, detaching or provisioning of volumes will be paused. Existing volumes and volumes already in use by a pod will continue to work.

5.12.1. Upgrading LINSTOR Operator v2

If you deployed LINSTOR using Operator v2, you can upgrade LINBIT SDS by deploying the newer version of LINSTOR Operator.

During the upgrade process, the LINSTOR satellite pods will restart. This will stop replication of DRBD devices, freezing any writes on volumes, until the satellite pods are online again.

How you upgrade LINBIT SDS depends on your original deployment method.

Upgrading LINSTOR Operator v2 by Using kustomize

If you deployed LINBIT SDS by using kustomize, upgrade to a new LINSTOR Operator version by changing the resource link in your kustomization.yaml file.

Listing 13. kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: linbit-sds
resources:
  - https://charts.linstor.io/static/v2.4.0.yaml (1)
generatorOptions:
  disableNameSuffixHash: true
secretGenerator:
  - name: drbdio-pull-secret
    type: kubernetes.io/dockerconfigjson
    literals:
      - .dockerconfigjson={"auths":{"drbd.io":{"username":"MY_LINBIT_USER","password":"MY_LINBIT_PASSWORD"}}} (2)
1 Replace with the latest release manifest from charts.linstor.io.
2 Replace MY_LINBIT_USER and MY_LINBIT_PASSWORD with your my.linbit.com credentials.

Then, apply the kustomization.yaml file, by using the following kubectl command, and wait for the upgrade to complete:

$ kubectl apply -k .
namespace/linbit-sds applied
[...]
$ kubectl -n linbit-sds  wait pod --for=condition=Ready --all
pod/linstor-operator-controller-manager-6d9847d857-zc985 condition met
[...]
Upgrading LINSTOR Operator v2 by Using Helm

If you used Helm, to deploy LINBIT SDS, upgrade to a new Operator version by upgrading the linstor-operator Helm chart.

$ helm repo update
[...]
Update Complete. ⎈Happy Helming!⎈
$ helm install linstor-operator linstor/linstor-operator --wait
Release "linstor-operator" has been upgraded. Happy Helming!
[...]

5.12.2. Upgrading LINSTOR Operator v1

Before upgrading to a new release, you should ensure you have an up-to-date backup of the LINSTOR database. If you are using the etcd database packaged in the LINSTOR Chart, see here.

Upgrades using the LINSTOR etcd deployment require etcd to use persistent storage. Only follow these steps if etcd was deployed using etcd.persistentVolume.enabled=true

アップグレードは、次のコンポーネントを新しいバージョンに更新します。

  • LINSTOR オペレーターのデプロイメント

  • LINSTOR コントローラー

  • LINSTOR サテライト

  • LINSTOR CSI Driver

  • etcd

  • Stork

Some versions require special steps, refer to instructions here. The main command to upgrade to a new LINSTOR Operator version is:

helm repo update
helm upgrade linstor-op linstor/linstor

If you used any customizations on the initial install, pass the same options to helm upgrade. The options currently in use can be retrieved from Helm.

# Retrieve the currently set options
$ helm get values linstor-op
USER-SUPPLIED VALUES:
USER-SUPPLIED VALUES: null
drbdRepoCred: drbdiocred
operator:
  satelliteSet:
    kernelModuleInjectionImage: drbd.io/drbd9-rhel8:v9.0.28
    storagePools:
      lvmThinPools:
      - devicePaths:
        - /dev/vdb
        name: thinpool
        thinVolume: thinpool
        volumeGroup: ""
# Save current options
$ helm get values linstor-op > orig.yaml
# modify values here as needed. for example selecting a newer DRBD image
$ vim orig.yaml
# Start the upgrade
$ helm upgrade linstor-op linstor/linstor -f orig.yaml

これにより、新しい pod のロールアウトがトリガーされます。少し待つと、すべての pod が実行され、準備が整います。LinstorControllers、LinstorSatelliteSets、および LinstorCSIDrivers のステータスセクションにエラーがリストされていないことを確認してください。

5.12.3. Upgrading LINSTOR Operator v1 to LINSTOR Operator v2

To upgrade an existing LINBIT SDS deployment in Kubernetes from a LINSTOR Operator v1 deployment to an Operator v2 deployment, you can follow the migration instructions at charts.linstor.io.

It is possible to do this upgrade without affecting existing storage volumes that are attached to your LINSTOR satellite pods. However, during the migration to an Operator v2 deployment, you will not be able to create new volumes, nor attach or detach existing volumes, until you roll out your new deployment.

5.12.4. Kubernetes バックエンドを使用した LINSTOR クラスターのアップグレード

LINSTOR オペレーターは、アップグレードする前にデータベースのバックアップを作成します。これにより、アップグレード中に問題が発生した場合に、以前の状態にロールバックできます。

オペレーターがバックアップを自動的に作成できない状況があります。これは、次の理由が考えられます。

  • The base version of the chart or operator is too old. Automatic backups are available starting with version 1.8.0 If upgrading from a version before 1.8.0, follow the manual steps in the next section.

  • バックアップが大きすぎて、Kubernetes のシークレットに収まりません。この場合、LinstorController リソースのステータスフィールドにエラーが報告されます。作成したバックアップを安全な場所にコピーし、必要なシークレットを作成して、指示に従います。

    kubectl cp <linstor-operator-pod>:/run/linstor-backups/linstor-backup-<some-hash>.tar.gz <destination-path>
    kubectl create secret linstor-backup-<same-hash>
LINSTOR データベースのバックアップを作成

Follow these instructions if you need to manually create a backup of the LINSTOR Kubernetes database.

  1. 現在のコントローラーを停止します。

    $ kubectl patch linstorcontroller linstor-op-cs '{"spec":{"replicas": 0}}'
    $ kubectl rollout status --watch deployment/linstor-op-cs-controller
  2. 次のコマンドは、すべての LINSTOR カスタムリソース定義の現在の状態を保存するファイル crds.yaml を作成します。

    $ kubectl get crds | grep -o ".*.internal.linstor.linbit.com" | \
      xargs kubectl get crds -oyaml > crds.yaml
  3. 定義に加えて、実際のリソースもバックアップする必要があります。

    $ kubectl get crds | grep -o ".*.internal.linstor.linbit.com" | \
      xargs -i{} sh -c "kubectl get {} -oyaml > {}.yaml"
  4. --set IHaveBackedUpAllMyLinstorResources=true を使用して chart のアップグレードを実行し、上記の手順を実行したことを確認します。

Restoring From a LINSTOR Database Backup

LINSTOR のアップグレード中に障害から回復する必要がある場合は、次の手順に従ってください。

  1. バックアップを準備します(バックアップがローカルマシンですでに利用可能な場合はスキップ)。

    $ # List the available backups
    $ kubectl get secret '-ocustom-columns=NAME:.metadata.name,FROM:metadata.annotations.linstor\.linbit\.com/backup-previous-version,CREATED-AT:.metadata.creationTimestamp'
    $ kubectl get secret linstor-backup-<backup-specific-hash> '-ogo-template=go-template={{index .data ".binaryData.backup.tar.gz" | base64decode}}' > linstor-backup.tar.gz
  2. バックアップを解凍します

    $ tar xvf linstor-backup.tar.gz
    crds.yaml
    ....
  3. 現在のコントローラーを停止します。

    $ kubectl patch linstorcontroller linstor-op-cs "{"spec":{"replicas": 0}}"
    $ kubectl rollout status --watch deployment/piraeus-op-cs-controller
  4. 既存のリソースを削除します

    $ kubectl get crds | grep -o ".*.internal.linstor.linbit.com" | xargs --no-run-if-empty kubectl delete crds
  5. 古い LINSTOR CRD を適用します

    $ kubectl apply -f crds.yaml
  6. 古い LINSTOR リソースの状態を適用します

    $ kubectl apply -f *.internal.linstor.linbit.com.yaml
  7. 古い LINSTOR バージョンを使用してヘルムチャートを再適用します

    $ helm upgrade linstor-op charts/piraeus --set operator.controller.controllerImage=... --set operator.satelliteSet.satelliteImage=...

5.12.5. 特定のバージョンのアップグレード手順

一部のバージョンでは特別な手順が必要です。以下を参照ください。

Upgrading to v2.4

追加の手順は必要ありません。

LINSTOR Satellites are now managed via DaemonSet resources. Any patch targeting a satellite Pod resources is automatically converted to the equivalent DaemonSet resource patch. In the Pod list, you will see these Pods using a new linstor-satellite prefix.

Upgrading to v2.3

Version v2.3 removed the NetworkPolicy resource from default deployment. To clean up existing NetworkPolicy resources, run the following command:

$ kubectl delete networkpolicy -n linbit-sds satellite
Upgrading to v2.2

Version v2.2 removed the dependency on cert-manager for the initial deployment. To clean up existing Certificate resource, run the following command:

$ kubectl delete certificate -n linbit-sds linstor-operator-serving-cert
Upgrading to v2.1

追加の手順は必要ありません。

v1.10 へのアップグレード

Version 1.10 introduces an option to share DRBD configuration between host and container. If you need this option, you have to update the CRDs. Because Helm does not upgrade CRDs on chart upgrade, instead enter the following commands:

$ helm repo update
$ helm pull linstor/linstor --untar
$ kubectl replace -f linstor/crds/
customresourcedefinition.apiextensions.k8s.io/linstorcontrollers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorcsidrivers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorsatellitesets.linstor.linbit.com replaced
v1.9 へのアップグレード

バージョン 1.9 では、デフォルトで <s-kubernetes-ha-controller,LINSTOR HA Controller>> デプロイメントが無効になっています。デプロイメントは、LINSTOR Operator チャートから移動しました。古いバージョンを引き続き使用する場合は、次の Helm コマンドを使用して再度有効にします。

helm upgrade linstor-op linstor/linstor ... --set haController.enabled=true

v1.6 以前から v1.9 にアップグレードする場合は、次のいずれかを行う必要があります。

  1. アップグレードする前に、マスターパスフレーズを作成する。

    $ kubectl create secret generic linstor-pass --from-literal=MASTER_PASSPHRASE=<password>
  2. または、最初に v1.7 にアップグレードすると、Helm によってマスターパスフレーズが自動的に作成されます。このパスフレーズは、次のように入力して後で表示できます。

    $ kubectl get secret linstor-op-passphrase \
    -ogo-template='{{ .data.MASTER_PASSPHRASE | base64decode }}'
v1.8 へのアップグレード
このアップグレードには K8s データベースの完全な再構築が必要なため、アップグレードに通常よりも時間がかかる場合があります。

Version 1.8 introduces new options to centrally set the log level and number of worker threads for the CSI driver. If you need these options, you have to update the CRDs. As Helm does not upgrade CRDs on chart upgrade, instead enter the following commands:

$ helm repo update
$ helm pull linstor/linstor --untar
$ kubectl replace -f linstor/crds/
customresourcedefinition.apiextensions.k8s.io/linstorcontrollers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorcsidrivers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorsatellitesets.linstor.linbit.com replaced

In addition, 1.8 reworks the way SSL/TLS setups work. Refer to the [s-kubernetes-securing-deployment-v1] section and work through these steps again.

v1.6 以前から v1.8 にアップグレードする場合は、次のいずれかを行う必要があります。

  1. アップグレードする前に、マスターパスフレーズを作成する。

    $ kubectl create secret generic linstor-pass --from-literal=MASTER_PASSPHRASE=<password>
  2. または、最初に v1.7 にアップグレードすると、Helm によってマスターパスフレーズが自動的に作成されます。このパスフレーズは、次のように入力して後で表示できます。

    $ kubectl get secret linstor-op-passphrase \
    -ogo-template='{{ .data.MASTER_PASSPHRASE | base64decode }}'
v1.7 へのアップグレード

追加の手順は必要ありません。

v1.6 へのアップグレード

This versions introduces a new option to support Kubernetes distributions which use different state directories than the default of /var/lib/kubelet. A notable example is microk8s, which uses /var/snap/microk8s/common/var/lib/kubelet. To support this, a small addition to the LinstorCSIDriver CRD was necessary. As Helm does not upgrade CRDs on chart upgrade, instead enter the following commands:

$ helm repo update
$ helm pull linstor/linstor --untar
$ kubectl replace -f linstor/crds/
customresourcedefinition.apiextensions.k8s.io/linstorcontrollers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorcsidrivers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorsatellitesets.linstor.linbit.com replaced

If you do not apply the new CRDs, you will get errors such as the following:

Error: UPGRADE FAILED: error validating "": error validating data: ValidationError(LinstorCSIDriver.spec): unknown field "kubeletPath" in com.linbit.linstor.v1.LinstorCSIDriver.spec

以前に付属のスナップショットコントローラーを使用して VolumeSnapshot リソースを処理したことがある場合は、それを Piraeus プロジェクトによって提供される新しい chart に置き換える必要があります。 スナップショットセクション が更新され、スナップショットコントローラーをクラスターに追加する方法e 含まれるようになりました。

v1.5 へのアップグレード

This version introduces a monitoring component for DRBD resources. This requires a new image and a replacement of the existing LinstorSatelliteSet CRD. Helm does not upgrade the CRDs on a chart upgrade, instead enter the following commands:

$ helm repo update
$ helm pull linstor/linstor --untar
$ kubectl replace -f linstor/crds/
customresourcedefinition.apiextensions.k8s.io/linstorcontrollers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorcsidrivers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorsatellitesets.linstor.linbit.com replaced

If you do not plan to use the provided monitoring you still need to apply the above steps, otherwise you will get an error such as the following:

Error: UPGRADE FAILED: error validating "": error validating data: ValidationError(LinstorSatelliteSet.spec): unknown field "monitoringImage" in com.linbit.linstor.v1.LinstorSatelliteSet.spec
一部の Helm バージョンでは、CRD を交換した後でも監視イメージを設定できません。その場合、クラスター内の LinstorSatelliteSet は空の monitoringImage 値を表示します。 kubectl edit linstorsatellitesets を使用してリソースを編集し、値を drbd.io/drbd-reactor:v0.3.0 に設定して、監視を有効にします。
v1.4 へのアップグレード

このバージョンでは、etcd イメージの新しいデフォルトバージョンが導入されているため、etcd が永続ストレージを使用していることに特に注意してください。永続ストレージなしで etcd イメージをアップグレードすると、クラスターが壊れるので注意してください。

新しい Helm オプションを使用せずに既存のクラスターをアップグレードする場合、追加の手順は必要ありません。

新しく導入された additionalProperties および additionalEnv 設定を使用する場合は、インストールされている CustomResourceDefinitions を新しいバージョンに置き換える必要があります。Helm は chart upgrade で CRD をアップグレードしません。

$ helm pull linstor/linstor --untar
$ kubectl replace -f linstor/crds/
customresourcedefinition.apiextensions.k8s.io/linstorcontrollers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorcsidrivers.linstor.linbit.com replaced
customresourcedefinition.apiextensions.k8s.io/linstorsatellitesets.linstor.linbit.com replaced
v1.3 へのアップグレード

追加の手順は必要ありません。

v1.2 へのアップグレード

LINSTOR Operator v1.2 is supported on Kubernetes 1.17+. If you are using an older Kubernetes distribution, you might need to change the default settings, for example [the CSI provisioner](https://kubernetes-csi.github.io/docs/external-provisioner.html).

CSI コンポーネントの更新時に既知の問題があります。pod は最新のイメージに更新されず、LinstorCSIDrivers リソースの errors セクションに DaemonSet の更新エラーが表示されます。この場合、 deployment/linstor-op-csi-controllerdaemonset/linstor-op-csi-node を手動で削除してください。それらはオペレーターによって再作成されます。

6. Openshift での LINSTOR ボリューム

This chapter describes the usage of LINBIT SDS in OpenShift as managed by the LINSTOR® Operator and with volumes provisioned using the LINSTOR CSI plugin.

OpenShift is the official Red Hat developed and supported distribution of Kubernetes. The value of OpenShift is the strong integration of otherwise optional components, such as network ingress and monitoring, all tied together with a Web UI to manage it all. LINSTOR Operator integrates with these components where possible.

6.1. Openshift での LINSTOR のデプロイ

OpenShiftへのLINBIT SDSのデプロイは、他のKubernetesクラスタへのLINBIT SDSのデプロイと同様です。前提条件として、以下のものが必要です。

  • oc ユーティリティで OpenShift クラスターにアクセスできる。

  • Your LINBIT® Customer Portal credentials for accessing drbd.io.

LINBIT’s container image repository (http://drbd.io) is only available to LINBIT customers or through LINBIT customer trial accounts. Contact LINBIT for information on pricing or to begin a trial. Alternatively, you can use the LINSTOR SDS upstream project, named Piraeus, without being a LINBIT customer.

First, create a new OpenShift project for LINBIT SDS which will also create a namespace for LINBIT SDS deployment:

$ oc new-project linbit-sds
Now using project "linbit-sds" on server ...

Next, create a file named kustomization.yaml to customize some of the deployment’s default values:

Listing 14. kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: linbit-sds
resources:
  - https://charts.linstor.io/static/v2.1.1.yaml
generatorOptions:
  disableNameSuffixHash: true
secretGenerator:
  - name: drbdio-pull-secret
    type: kubernetes.io/dockerconfigjson
    literals:
      - .dockerconfigjson={"auths":{"drbd.io":{"username":"MY_LINBIT_USER","password":"MY_LINBIT_PASSWORD"}}} (1)
1 Replace MY_LINBIT_USER and MY_LINBIT_PASSWORD with your LINBIT customer portal credentials.
The YAML configuration manifest above is current at the time of writing. Refer to https://charts.linstor.io for the most up-to-date version or previous versions if needed.

You can make additional modifications to the kustomization.yaml file based on your preferences and needs. Possible options are discussed in the Kubernetes advanced deployment section.

Finally, deploy LINBIT SDS by applying the customized configuration, and wait until all pods are ready, by entering the following commands in the same directory as your kustomization.yaml:

$ oc apply -k . && \
oc wait --for=condition=ready pod --all -n linbit-sds --timeout=5m && \
oc get pods -n linbit-sds

Output should eventually show that a LINSTOR controller pod is up and running.

NAME                                                   READY   STATUS    RESTARTS   AGE
linstor-operator-controller-manager-59586c7bb5-qlfwb   1/1     Running   0          11s

After deploying the LINSTOR controller pod, enter the following command to complete the deployment of your LINBIT SDS in OpenShift cluster:

# oc apply -f - <<EOF
apiVersion: piraeus.io/v1
kind: LinstorCluster
metadata:
  name: linstorcluster
spec: {}
EOF
# oc wait pod --for=condition=Ready -n linbit-sds --timeout=5m --all

Output should eventually show that your LINBIT SDS cluster pods are up and running.

NAME                                                   READY   STATUS    RESTARTS   AGE
ha-controller-6fl6b                                    1/1     Running   0          60s
ha-controller-c955w                                    1/1     Running   0          60s
ha-controller-v4mdr                                    1/1     Running   0          60s
kube-0                                                 2/2     Running   0          56s (1)
kube-1                                                 2/2     Running   0          56s (1)
kube-2                                                 2/2     Running   0          55s (1)
linstor-controller-779bffbc59-6jjzd                    1/1     Running   0          61s
linstor-csi-controller-8cd45658f-6f9t6                 7/7     Running   0          61s
linstor-csi-node-blgk8                                 3/3     Running   0          61s
linstor-csi-node-mn8p6                                 3/3     Running   0          61s
linstor-csi-node-pncpz                                 3/3     Running   0          61s
linstor-operator-controller-manager-59586c7bb5-qlfwb   1/1     Running   0          4m2s
1 These pods are the LINSTOR satellite node pods. The pod name reflects each node’s hostname.

6.1.1. OpenShiftでLINBIT SDSを利用する

次の手順は、Kubernetesと全く同じです。

6.1.2. LINBIT GUIアクセスの設定

LINSTORコンテナイメージには、LINBIT GUIがプリインストールされています。これをクラスタ上で公開するには、OpenShift Routeリソースを設定します。

$ oc create route edge --service linstor-op-cs
$ oc get route
NAME           HOST/PORT                                  PATH  SERVICES       PORT           TERMINATION  WILDCARD
linstor-op-cs  linstor-op-cs-storage.apps.oc.example.com        linstor-op-cs  linstor-op-cs  edge         None

GUIは https://linstor-op-cs-storage.apps.oc.example.com/ui/ でアクセスできるようになりました。

This might enable external access to LINBIT GUI and LINSTOR API. Ensure that only authorized users can access it, for example, by requiring client side TLS certificates on the route.

6.1.3. クラスタモニタリングの設定

OpenShift includes a fully configured monitoring stack. While most of the monitoring stack is only intended for OpenStack infrastructure, basic monitoring for LINBIT SDS is possible.

まず、 Red Hatドキュメント の手順に従って、OpenShiftでユーザー定義プロジェクトの監視が有効になっていることを確認します。

Once monitoring for user-defined projects is enabled, the LINSTOR Operator automatically detects the presence of OpenShift’s Prometheus-based monitoring stack and configures the ServiceMonitor resources. The Prometheus instance will scrape metrics for DRBD® and LINSTOR from all Cluster nodes.

6.2. OpenShiftでLINBIT SDS 連携

LINSTOR Controller podにはLINSTOR Clientが含まれており、LINSTORの対話型モードに簡単にアクセスすることができます。例えば:

$ oc exec -it deployment/linstor-op-cs-controller -- linstor interactive
LINSTOR ==> ...

This should only be necessary for investigating problems and accessing advanced functionality. Regular operation such as creating volumes should be achieved through the Kubernetes integration.

7. NomadのLINSTORボリューム

This chapter describes using LINSTOR® and DRBD® to provision volumes in Nomad.

7.1. Nomad の紹介

Nomad は、コンテナとコンテナ化されていないアプリケーションをオンプレミスとクラウド上にデプロイして管理するための、シンプルで柔軟性のあるワークロードオーケストレーターです。

Nomad は Container Storage Interface(CSI) に準拠した plug-ins 経由のストレージボリュームのプロビジョニングをサポートします。

LINBIT® distributes a CSI plug-in in the form of container images from drbd.io. The plug-in can be configured to work with a LINSTOR cluster that is deployed along or inside a Nomad cluster.

LINBIT’s container image repository (http://drbd.io) is only available to LINBIT customers or through LINBIT customer trial accounts. Contact LINBIT for information on pricing or to begin a trial. Alternatively, you may use LINSTOR SDS’ upstream project named Piraeus, without being a LINBIT customer.

7.2. NomadにLINSTORの展開

この項では、Nomadで新しいLINSTORクラスタを展開および設定する方法について説明します。

LINSTORをノードに直接インストールする場合は、 LINSTOR のインストール を参照してください。このセクションをスキップして、 CSI ドライバのデプロイ に直接移動できます。

7.2.1. Nomadを準備する

LINSTORを実行するには、すべてのNomadエージェントを次のように設定する必要があります。

  • docker driverをサポートし、特権コンテナの実行を許可します。

    特権コンテナーの実行を許可するには、以下のスニペットをNomadエージェント構成に追加して、Nomadを再起動します。

    Listing 15. /etc/nomad.d/docker-privileged.hcl
    plugin "docker" {
      config {
        allow_privileged = true
      }
    }
  • コンテナネットワークのサポート。 Container Network Interface プラグインがインストールされていない場合は、ジョブネットワークで mode = "host" のみを使用できます。ほとんどの本番環境のセットアップでは、デフォルトのプラグインをインストールすることをお勧めします。

    plug-in release page に移動し、ディストリビューションに適したリリースアーカイブを選択して、/opt/cni/bin に解凍します。解凍する前にディレクトリを作成する必要があるかもしれません。

  • ホストボリュームを提供して、ホストの /dev ディレクトリへのコンテナアクセスを許可する

    ホストボリュームを作成するには、以下のスニペットをNomadエージェント構成に追加し、Nomadを再起動します。

    Listing 16. /etc/nomad.d/host-volume-dev.hcl
    client {
      host_volume "dev" {
        path = "/dev"
      }
    }

7.2.2. LINSTORコントローラジョブの作成

LINSTOR コントローラは、レプリカのないサービスとして導入されます。どの時点においても、1つのクラスタ内で実行可能なLINSTOR コントローラは1つのみです。データベースへのアクセス権がある場合は、新しいノードでコントローラを再起動できます。詳細は 高可用性 LINSTOR クラスターの作成 を参照してください。

The following example will create a Nomad job starting a single LINSTOR Controller in data center dc1 and connect to an external database.

Listing 17. linstor-controller.hcl
job "linstor-controller" {
  datacenters = ["dc1"] (1)
  type = "service"

  group "linstor-controller" {
    network {
      mode = "bridge"
      # port "linstor-api" { (2)
      #   static = 3370
      #   to = 3370
      # }
    }

    service { (3)
      name = "linstor-api"
      port = "3370"

      connect {
        sidecar_service {}
      }

      check {
        expose = true
        type = "http"
        name = "api-health"
        path = "/health"
        interval = "30s"
        timeout = "5s"
      }
    }

    task "linstor-controller" {
      driver = "docker"
      config {
        image = "drbd.io/linstor-controller:v1.13.0" (4)

        auth { (5)
          username = "example"
          password = "example"
          server_address = "drbd.io"
        }

        mount {
          type = "bind"
          source = "local"
          target = "/etc/linstor"
        }
      }

      # template { (6)
      #  destination = "local/linstor.toml"
      #  data = <<EOH
      #    [db]
      #    user = "example"
      #    password = "example"
      #    connection_url = "jdbc:postgresql://postgres.internal.example.com/linstor"
      #  EOH
      # }

      resources {
        cpu    = 500 # 500 MHz
        memory = 700 # 700MB
      }
    }
  }
}
1 dc1 はデータセンタ名に置き換えてください
2 これにより、ホストでポート 3370 の LINSTOR API が公開されます。
クラスタが Consul Connect で構成されていない場合は、このセクションのコメントを削除してください。
3 service ブロックは、サービスメッシュを介してL INSTOR API を他のジョブに公開するために使用されます。
クラスタが Consul Connect で構成されていない場合は、このセクションを削除してください。
4 This sets the LINSTOR Controller image to run. The latest images are available from drbd.io.
:latest タグの使用は、バージョンの不一致や意図しないアップグレードにすぐにつながる可能性があるため、お勧めしません。
5 イメージを pull するときに使用する認証を設定します。 drbd.io から pull する場合は、ここでLINBITの顧客ログインを使用する必要があります。プライベートレポジトリからの pull に関しては こちら を参照ください。
6 このテンプレートは、LINSTORの任意の構成オプションを設定するために使用できます。この例では、LINSTORの外部データベースを構成します。詳細は LINSTOR データベースオプション こちら およびNomadテンプレート こちら を参照ください。

次のコマンドを実行して、ジョブを適用します。

$ nomad job run linstor-controller.hcl
==> Monitoring evaluation "7d8911a7"
    Evaluation triggered by job "linstor-controller"
==> Monitoring evaluation "7d8911a7"
    Evaluation within deployment: "436f4b2d"
    Allocation "0b564c73" created: node "07754224", group "controller"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "7d8911a7" finished with status "complete"
LINSTOR データベースにホストボリュームを使用

If you want to try LINSTOR without setting up an external database, you can make use of LINSTOR’s built-in file system based database. To make the database persistent, you need to ensure it is placed on a host volume.

ホストボリュームを使用すると、1つのノードのみでLINSTOR コントローラを実行できます。ノードが使用不可の場合は、LINSTOR クラスタも使用不可になります。代替策として、外部(高可用性)データベースを使用するか、 LINSTORクラスタをホストに直接導入 します。

LINSTOR データベース用のホストボリュームを作成するには、まず、必要な権限を持つディレクトリをホスト上に作成します。

$ mkdir -p /var/lib/linstor
$ chown -R 1000:0 /var/lib/linstor

次に、以下のスニペットをNomadエージェント構成に追加し、Nomadを再起動します。

Listing 18. /etc/nomad.d/host-volume-linstor-db.hcl
client {
  host_volume "linstor-db" {
    path = "/var/lib/linstor"
  }
}

次に、上記の例の linstor-controller.hcl に次のスニペットを追加し、構成テンプレートの connection_url オプションを変更します。

Listing 19. job > group
volume "linstor-db" {
  type = "host"
  source = "linstor-db"
}
Listing 20. job > group > task
volume_mount {
  volume = "linstor-db"
  destination = "/var/lib/linstor"
}

template {
  destination = "local/linstor.toml"
  data = <<EOH
    [db]
    user = "linstor"
    password = "linstor"
    connection_url = "jdbc:h2:/var/lib/linstor/linstordb"
  EOH
}

7.2.3. LINSTOR サテライトジョブの作成

Nomad で LINSTORサテライトは、Nomadのシステムジョブとしてデプロイされ、特権コンテナー内で実行されます。サテライトに加えて、このジョブはLINSTORで使用される他のカーネルモジュールとともにDRBDモジュールもロードします。

次の例では、Nomadジョブを作成して、データセンター dc1 のすべてのノードでLINSTORサテライトを開始します。

Listing 21. linstor-satellite.hcl
job "linstor-satellite" {
  datacenters = ["dc1"] (1)
  type = "system"

  group "satellite" {
    network {
      mode = "host"
    }

    volume "dev" { (2)
      type = "host"
      source = "dev"
    }

    task "linstor-satellite" {
      driver = "docker"

      config {
        image = "drbd.io/linstor-satellite:v1.13.0" (3)

        auth { (4)
          username = "example"
          password = "example"
          server_address = "drbd.io"
        }

        privileged = true (5)
        network_mode = "host" (6)
      }

      volume_mount { (2)
        volume = "dev"
        destination = "/dev"
      }

      resources {
        cpu    = 500 # 500 MHz
        memory = 500 # 500MB
      }
    }

    task "drbd-loader" {
      driver = "docker"
      lifecycle {
        hook = "prestart" (7)
      }

      config {
        image = "drbd.io/drbd9-rhel8:v9.0.29" (8)

        privileged = true (5)
        auth { (4)
          username = "example"
          password = "example"
          server_address = "drbd.io"
        }
      }

      env {
        LB_HOW = "shipped_modules" (9)
      }

      volume_mount { (10)
        volume = "kernel-src"
        destination = "/usr/src"
      }
      volume_mount { (10)
        volume = "modules"
        destination = "/lib/modules"
      }
    }

    volume "modules" { (10)
      type = "host"
      source = "modules"
      read_only = true
    }

    volume "kernel-src" { (10)
      type = "host"
      source = "kernel-src"
      read_only = true
    }
  }
}
1 dc1 はデータセンタ名に置き換えてください
2 dev ホストボリュームは Nomadを準備する で作成されたボリュームであり、サテライトがホストブロックデバイスを管理できるようにします。
3 This sets the LINSTOR Satellite image to run. The latest images are available from drbd.io. The satellite image version has to match the version of the controller image.
:latest タグの使用は、バージョンの不一致や意図しないアップグレードにすぐにつながる可能性があるため、お勧めしません。
4 イメージを pull するときに使用する認証を設定します。 drbd.io から pull する場合は、ここでLINBITの顧客ログインを使用する必要があります。プライベートレポジトリからの pull に関しては こちら を参照ください。
5 ストレージデバイスやDRBDを構成し、カーネルモジュールをロードするためには、コンテナーを特権モードで実行する必要があります。
6 サテライトはDRBDと通信する必要があるため、ホストネットワークで実行されているnetlinkインターフェースにアクセスする必要があります。
7 drbd-loader タスクはサテライトの開始時に1回実行され、DRBDやその他の有用なカーネルモジュールをロードします。
8 drbd-loader は、使用しているディストリビューションに固有のものです。使用可能なオプションは次のとおりです。
  • Ubuntu18.04(Bionic Beaver) 用の drbd.io/drbd9-bionic

  • Ubuntu20.04(Focal Fossa) 用の drbd.io/drbd9-focal

  • RHEL8 用の drbd.io/drbd9-rhel8

  • RHEL7 用の drbd.io/drbd9-rhel7

9 drbd-loader コンテナは環境変数で設定できます。 LB_HOW はコンテナにDRBDカーネルモジュールの挿入方法を指示します。使用可能なオプションは次のとおりです。
shipped_modules

コンテナに同梱されているパッケージ済みのRPMまたはDEBを使用します。

コンパイル

ソースからDRBDをコンパイルします。カーネルヘッダーへのアクセスが必要です(下記参照)。

deps_only

LINSTOR サテライトで使用される既存のモジュール(例えば dm_thin_pooldm_cache など)のみをロードするようにします。

10 drbd-loader コンテナがDRBDを構築したり、既存のモジュールをロードしたりするためには、ホストの /usr/src/lib/modules にアクセスできる必要があります。

これには、すべてのノードに追加のホストボリュームを設定する必要があります。次のスニペットをすべての Nomad エージェント構成に追加してから、Nomad を再起動する必要があります。

Listing 22. /etc/nomad.d/drbd-loader-volumes.hcl
client {
  host_volume "modules" {
    path = "/lib/modules"
    read_only = true
  }
  host_volume "kernel-src" {
    path = "/usr/src"
    read_only = true
  }
}

次のコマンドを実行して、ジョブを適用します。

$ nomad job run linstor-satellite.hcl
==> Monitoring evaluation "0c07469d"
    Evaluation triggered by job "linstor-satellite"
==> Monitoring evaluation "0c07469d"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "0c07469d" finished with status "complete"

7.2.4. NomadでのLINSTORの設定

linstor-controller および linstor-satellite ジョブが実行されたら、 linstor コマンドラインツールを使用してクラスタの設定を開始できます。

これは次のように実行できます。

  • コンテナに入って nomad exec で直接 linstor-controller を実行する。

  • linstor-controller を実行しているホスト上で drbd.io/linstor-client コンテナを使用する。

    docker run -it --rm --net=host drbd.io/linstor-client node create
  • linstor-client パッケージを linstor-controller を実行しているホストにインストールします。

いずれの場合も、 サテライトをクラスタに追加 および ストレージプールを作成 を実行する必要があります。たとえば、ノード nomad-01.example.com を追加してLVMシンストレージプールを構成するには、次のコマンドを実行します。

$ linstor node create nomad-01.example.com
$ linstor storage-pool create lvmthin nomad-01.example.com thinpool linstor_vg/thinpool
CSIドライバでは、サテライトにホスト名に基づいた名前を付ける必要があります。正確に言うと、サテライト名はノードのNomads attr.unique.hostname 属性と一致する必要があります。

7.3. LINSTOR CSIドライバのNomadへの配備

CSIドライバはシステムジョブとしてデプロイされます。つまり、クラスタ内のすべてのノードで実行されます。

次の例では、データセンター dc1 内のすべてのノードでLINSTOR CSIドライバを起動するNomadジョブを作成します。

Listing 23. linstor-csi.hcl
job "linstor-csi" {
  datacenters = ["dc1"] (1)
  type = "system"

  group "csi" {
    network {
      mode = "bridge"
    }

    service {
      connect {
        sidecar_service { (2)
          proxy {
            upstreams {
              destination_name = "linstor-api"
              local_bind_port  = 8080
            }
          }
        }
      }
    }

    task "csi-plugin" {
      driver = "docker"
      config {
        image = "drbd.io/linstor-csi:v0.13.1" (3)

        auth { (4)
          username = "example"
          password = "example"
          server_address = "drbd.io"
        }

        args = [
          "--csi-endpoint=unix://csi/csi.sock",
          "--node=${attr.unique.hostname}", (5)
          "--linstor-endpoint=http://${NOMAD_UPSTREAM_ADDR_linstor_api}", (6)
          "--log-level=info"
        ]

        privileged = true (7)
      }

      csi_plugin { (8)
        id = "linstor.csi.linbit.com"
        type = "monolith"
        mount_dir = "/csi"
      }

      resources {
        cpu    = 100 # 100 MHz
        memory = 200 # 200MB
      }
    }
  }
}
1 dc1 はデータセンタ名に置き換えてください
2 sidecar_service スタンザは、 Consul Connect を使用して生成されたサービスメッシュの使用を可能にします。Nomadでこの機能を構成していない場合、または外部 LINSTOR コントローラーを使用している場合は、この構成をスキップできます。
3 This sets the LINSTOR CSI Driver image to run. The latest images are available from drbd.io.
:latest タグの使用は、バージョンの不一致や意図しないアップグレードにすぐにつながる可能性があるため、お勧めしません。
4 イメージを pull するときに使用する認証を設定します。 drbd.io から pull する場合は、ここでLINBITの顧客ログインを使用する必要があります。プライベートレポジトリからの pull に関しては こちら を参照ください。
5 この引数は、CSIドライバがLINSTOR APIで自身を識別するために使用するノード名を設定します。デフォルトでは、これはノードのホスト名に設定されます。
6 この引数は、LINSTOR APIエンドポイントを設定します。consulサービスメッシュ(前述のNr.2を参照)を使用していない場合は、これをControllers APIエンドポイントに設定する必要があります。エンドポイントは、デプロイされているすべてのノードから到達可能である必要があります。
7 CSIドライバはマウントコマンドを実行する必要があるため、特権付きコンテナが必要です。
8 csi_plugin スタンザは、このタスクがCSIプラグインであることをNomadに通知します。Nomadエージェントは、ボリュームに対する要求をジョブコンテナーの1つに転送します。

次のコマンドを実行して、ジョブを適用します。

$ nomad job run linstor-csi.hcl
==> Monitoring evaluation "0119f19c"
    Evaluation triggered by job "linstor-csi"
==> Monitoring evaluation "0119f19c"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "0119f19c" finished with status "complete"

7.4. NomadでのLINSTORボリュームの使用

Nomadのボリュームは、 volume仕様 を使用して作成されます。

例として、次の指定では、2つのレプリカを持つ1GiBボリュームがLINSTORストレージプール thinpool から要求されます。

Listing 24. vol1.hcl
id = "vol1" (1)
name = "vol1" (2)

type = "csi"
plugin_id = "linstor.csi.linbit.com"

capacity_min = "1GiB"
capacity_max = "1GiB"

capability {
  access_mode = "single-node-writer" (3)
  attachment_mode = "file-system" (4)
}

mount_options {
  fs_type = "ext4" (5)
}

parameters { (6)
  "resourceGroup" = "default-resource",
  "storagePool" = "thinpool",
  "autoPlace" = "2"
}
1 id は、Nomad でこのボリュームを参照するために使用されます。ジョブ指定の volume.source フィールドで使用されます。
2 name は、バックエンド(LINSTORなど)でボリュームを作成するときに使用されます。理想的には、これは id と一致し、有効なLINSTORリソース名です。名前が有効でない場合は、LINSTOR CSIによって互換性のあるランダムな名前が生成されます。
3 ボリュームがサポートする必要があるアクセスの種類。LINSTOR CSIは次のものをサポートします。
single-node-reader-only

一度に 1 つのノードに対してのみ読み取りアクセスを許可する

single-node-writer

一度に 1 つのノードに対してのみ読み取り/書き込みアクセスを許可する

multi-node-reader-only

複数のノードからの読み取り専用アクセスを許可する

4 file-system または block-device のいずれかです。
5 使用するファイルシステムを指定します。LINSTOR CSI は ext4xfs をサポートします。
6 LINSTOR CSI に渡す追加パラメータ。上記の例では、リソースが default-resource resource group の一部であることを要求しており、2つのレプリカをデプロイする必要があります。

使用可能なパラメータの完全なリストについては、 ストレージクラスで使用可能なパラメータ を参照ください。Kubernetes は Nomad と同様に CSI プラグインを使用します。

ボリュームを作成するには、次のコマンドを実行します。

$ nomad volume create vol1.hcl
Created external volume vol1 with ID vol1
$ nomad volume status
Container Storage Interface
ID    Name  Plugin ID               Schedulable  Access Mode
vol1  vol1  linstor.csi.linbit.com  true         <none>
$ linstor resource list
╭──────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node                 ┊ Port ┊ Usage  ┊ Conns ┊    State ┊ CreatedOn           ┊
╞══════════════════════════════════════════════════════════════════════════════════════════════╡
┊ vol1         ┊ nomad-01.example.com ┊ 7000 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2021-06-15 14:56:32 ┊
┊ vol1         ┊ nomad-02.example.com ┊ 7000 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2021-06-15 14:56:32 ┊
╰──────────────────────────────────────────────────────────────────────────────────────────────╯

7.4.1. ジョブでのボリュームの使用

ジョブでボリュームを使用するには、ジョブ指定に volume および volume_mount スタンザを追加します。

job "example" {
  ...

  group "example" {
    volume "example-vol" {
      type = "csi"
      source = "vol1"
      attachment_mode = "file-system"
      access_mode = "single-node-writer"
    }

    task "mount-example" {
      volume_mount {
        volume = "example-vol"
        destination = "/data"
      }

      ...
    }
  }
}

7.4.2. ボリュームのスナップショットの作成

基礎となるストレージプールドライバがスナップショットをサポートしていれば、LINSTORは既存のボリュームのスナップショットを作成できます。

次のコマンドは、ボリューム vol1snap1 という名前のスナップショットを作成します。

$ nomad volume snapshot create vol1 snap1
Snapshot ID  Volume ID  Size     Create Time  Ready?
snap1        vol1       1.0 GiB  None         true
$ linstor s l
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ SnapshotName ┊ NodeNames                                  ┊ Volumes  ┊ CreatedOn           ┊ State      ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ vol1         ┊ snap1        ┊ nomad-01.example.com, nomad-02.example.com ┊ 0: 1 GiB ┊ 2021-06-15 15:04:10 ┊ Successful ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

スナップショットを使用して、既存のボリュームにスナップショットのデータを事前に設定できます。

$ cat vol2.hcl
id = "vol2"
name = "vol2"
snapshot_id = "snap1"

type = "csi"
plugin_id = "linstor.csi.linbit.com"
...

$ nomad volume create vol2.hcl
Created external volume vol2 with ID vol2

8. Proxmox VE での LINSTOR ボリューム

This chapter describes DRBD® in Proxmox Virtual Environment (VE) using the LINSTOR® Proxmox Plugin.

8.1. Proxmox VE の紹介

Proxmox VE はKVM、Linuxコンテナ、HAとともに使われる簡単で、完全なサーバ仮想化環境を提供します。

linstor-proxmox is a Perl plugin for Proxmox that, in combination with LINSTOR, allows you to replicate VM disks on several Proxmox VE nodes. This allows to live-migrate active VMs within a few seconds and with no downtime without needing a central SAN, as the data is already replicated to multiple nodes.

8.2. Proxmox のアップグレード

新規インストールの場合はこのセクションをスキップして Installing the LINSTOR Proxmox Plugin に進んでください。

8.2.1. Upgrading Plugin to 7.x

Version 7 of the plugin uses a LINSTOR controller API that is available from LINSTOR version 1.21.1 onwards. Make sure that your LINSTOR setup (controller and satellites) use at least that version.

8.2.2. Upgrading Plugin from 4.x to 5.x

Version 5 of the plugin drops compatibility with the legacy configuration options “storagepool” and “redundancy”. Version 5 requires a “resourcegroup” option, and obviously a LINSTOR resource group. The old options should be removed from the configuration.

LINSTOR の構成に関しては LINSTOR の構成 を参照してください。以下に典型的は例を示します。プールが “mypool” 、冗長性が 3 に設定される例です。

プールが “mypool” に、冗長性が 3 に設定されていると仮定します。

# linstor resource-group create --storage-pool=mypool --place-count=3 drbdMypoolThree
# linstor volume-group create drbdMypoolThree
# vi /etc/pve/storage.cfg
drbd: drbdstorage
   content images,rootdir
   controller 10.11.12.13
   resourcegroup drbdMypoolThree

8.2.3. Upgrading Plugin from 5.x to 6.x

Version 6.0.0 of the plugin drops all code related to the redundancy setting. This is handled by LINSTOR resource groups (resourcegroup setting) for a very long time. No change should be required.

LINSTOR によって管理される VM で LINSTOR コントローラを実行するための controllervm 設定はなくなりました。 drbd-reactor を使用して高可用性 LINSTOR コントローラを実現することをお勧めします。

設定 statuscachepreferlocal がデフォルトで有効になりました。

8.2.4. PVE のバージョン 5.x から 6.x へアップグレード

With version 6 PVE added additional parameters to some functions and rightfully reset their “APIAGE”. This means that old plugins, while actually usable as they don’t use any of these changed functions do not work anymore. Please upgrade to plugin version 5.2.1 at least.

8.3. Installing the LINSTOR Proxmox Plugin

To use LINSTOR in Proxmox, you will need to install the LINSTOR Proxmox plugin.

8.3.1. Proxmox VE カーネル ヘッダーのインストール

To use LINSTOR in Proxmox, you will need to install the DRBD kernel module. The DRBD 9 kernel module is installed as a kernel module source package (drbd-dkms). Therefore, you will have to install the Proxmox VE kernel headers package, pve-headers, before you install the DRBD kernel module from the LINBIT® repositories. Following that order ensures that the kernel module will build properly for your kernel.

最新の Proxmox カーネルをインストールする予定がない場合は、現在実行中のカーネルに一致するカーネル ヘッダーをインストールする必要があります (たとえば pve-headers-$(uname -r) )。この手順に従わなかった場合でも apt install --reinstall drbd-dkms コマンドで、現在のカーネルに対して drbd-dkms パッケージを再構築できます (事前にカーネル ヘッダーをインストールしている場合)。

pve-headers パッケージをインストールする前に、Proxmox PVE リポジトリを APT ソースリスト /etc/apt/sources.list に追加し apt update と入力します。 the Proxmox wiki を参照ください。

8.3.2. Installing the Proxmox Plugin By Using LINBIT Customer Repositories

LINBIT のお客様または評価アカウントをお持ちの場合は、ノードで LINBIT の drbd-9 リポジトリを有効にしてから、apt update コマンドを使用してリポジトリを更新できます。

You can then install the DRBD kernel module, DRBD utilities, and the LINSTOR Proxmox plugin by entering:

# apt install drbd-utils linstor-proxmox drbd-dkms
ノードを LINBIT に登録し、LINBIT リポジトリを有効にする手順については LINBIT クラスターノードを管理するスクリプトの使用 を参照してください。

8.3.3. Installing the Proxmox Plugin By Using LINBIT Public Repositories

LINBIT provides a dedicated public repository for Proxmox VE users. This repository not only contains the Proxmox plugin, but the whole LINBIT SDS stack including a DRBD kernel module and user-space utilities.

You can add LINBIT’s public repository by entering the commands below, setting $PVERS to your Proxmox VE major version (for example, “8”, not “8.1”):

# wget -O /tmp/package-signing-pubkey.asc \
https://packages.linbit.com/package-signing-pubkey.asc
# gpg --yes -o /etc/apt/trusted.gpg.d/linbit-keyring.gpg --dearmor \
/tmp/package-signing-pubkey.asc
# PVERS=8 && echo "deb [signed-by=/etc/apt/trusted.gpg.d/linbit-keyring.gpg] \
http://packages.linbit.com/public/ proxmox-$PVERS drbd-9" > /etc/apt/sources.list.d/linbit.list

After adding the LINBIT package repository, you can install the Proxmox plugin and other necessary components (DRBD kernel module and utilities), by entering the following command:

# apt update && apt -y install drbd-dkms drbd-utils linstor-proxmox

8.4. LINSTOR の構成

このガイドの残りの部分では クラスタの初期化 に基づいて LINSTOR クラスタが構成されていると仮定します。1 つのノードで “linstor-controller” を起動し、すべてのノードで “linstor-satellite” を起動します。”linstor-satellite” サービスには systemctl edit linstor-satellite.service で追加構成が必要です。

[Service]
Type=notify
TimeoutStartSec=infinity

The preferred way to use the plugin, starting from version 4.1.0, is through LINSTOR resource groups and a single volume group within every resource group. LINSTOR resource groups are described in リソースグループを使用した LINSTOR プロビジョニングされたボリュームのデプロイ. All the required LINSTOR configuration (e.g., redundancy count) has to be set on the resource group.

8.5. Configuring the Proxmox Plugin

最後の手順ではProxmox自身を設定します。これは /etc/pve/storage.cfg に以下のような設定を加えることで行います。

drbd: drbdstorage
   content images,rootdir
   controller 10.11.12.13
   resourcegroup defaultpool

The drbd entry is fixed and you are not allowed to modify it, as it tells to Proxmox to use DRBD as storage back end. The “drbdstorage” entry can be modified and is used as a friendly name that will be shown in the PVE web GUI to locate the DRBD storage. The “content” entry is also fixed, so do not change it. The redundancy (specified in the resource group) specifies how many replicas of the data will be stored in the cluster. The recommendation is to set it to 2 or 3 depending on your setup. The data is accessible from all nodes, even if some of them do not have local copies of the data. For example, in a 5 node cluster, all nodes will be able to access 3 copies of the data, no matter where they are stored in. The “controller” parameter must be set to the IP of the node that runs the LINSTOR controller service. Only one node can be set to run as LINSTOR controller at the same time. If that node fails, start the LINSTOR controller on another node and change that value to its IP address.

異なるリソースグループで異なるストレージプールを使用する構成は、次のようになります。

drbd: drbdstorage
   content images,rootdir
   controller 10.11.12.13
   resourcegroup defaultpool

drbd: fastdrbd
   content images,rootdir
   controller 10.11.12.13
   resourcegroup ssd

drbd: slowdrbd
   content images,rootdir
   controller 10.11.12.13
   resourcegroup backup

これらの設定後、Web GUIでVMを作成し、ストレージとして “drbdstorage” もしくは他の定義されているプールを選択することでDRBDボリュームを使用できます。

Starting from version 5 of the plugin, you can set the option “preferlocal yes”. If it is set, the plugin tries to create a diskful assignment on the node that issued the storage create command. With this option you can ensure that the VM gets local storage if possible. Without that option LINSTOR might place the storage on nodes ‘B’ and ‘C’, while the VM is initially started on node ‘A’. This would still work as node ‘A’ then would get a diskless assignment, but having local storage might be preferred.

注意: DRBDは現時点で raw ディスクフォーマットのみをサポートします。

この時点で、VMのライブマイグレーションができます。すべてのノード(ディスクレスノードでも)ですべてのデータにアクセスできるため、わずか数秒で完了します。 VMに負荷がかかっていて、ダーティメモリが常に多い場合は、全体的な処理に少し時間がかかることがあります。しかし、いずれの場合でも、ダウンタイムは最小限で、中断は全く見られません。

Table 1. テーブル構成オプション
Option Meaning

controller

The IP of the LINSTOR controller (‘,’ separated list allowed)

resourcegroup

The name of a LINSTOR resource group which defines the deployment of new VMs. As described above

preferlocal

Prefer to create local storage (yes/no). As decribed above

statuscache

Time in seconds status information is cached, 0 means no extra cache. Relevant on huge clusters with hundreds of resources. This has to be set on all drbd storages in /etc/pve/storage.cfg to take effect.

apicrt

Path to the client certificate

apikey

Path to the client private key

apica

Path to the CA certificate

8.6. コントローラの高可用性 (オプション構成)

LINSTOR を高可用性にするということは、LINSTOR コントローラーを高可用性にすることです。このステップはセクション 高可用性 LINSTOR クラスターの作成 で説明されています。

The last — but crucial — step is to configure the Proxmox plugin to be able to connect to multiple LINSTOR controllers. It will use the first one it receives an answer from. This is done by adding a comma-separated list of controllers in the controller section of the plugin like this:

drbd: drbdstorage
   content images,rootdir
   controller 10.11.12.13,10.11.12.14,10.11.12.15
   resourcegroup defaultpool

9. OpenNebulaのLINSTORボリューム

This chapter describes DRBD® in OpenNebula using the LINSTOR® storage driver add-on.

詳しいインストールと設定の手順は、ドライバのソースの README.md を参照ください。

9.1. OpenNebula の紹介

OpenNebula は、アドオンを使用してその機能を拡張できる、柔軟でオープンソースのクラウド管理プラットフォームです。

LINSTOR アドオンを使用すると、DRBD によってバックアップされ、DRBD プロトコルを介してネットワークに接続された高可用性イメージを持つ仮想マシンを配備できます。

9.2. OpenNebula アドオンのインストール

OpenNebula 用の LINSTOR ストレージアドオンのインストールには、動作中の OpenNebula クラスタと動作中の LINSTOR クラスタが必要です。

With access to the LINBIT® customer repositories you can install the linstor-opennebula package with:

# apt install linstor-opennebula

または

# yum install linstor-opennebula

LINBITのパッケージにアクセスすることができない場合は、 OpenNebula LINSTOR アドオン GitHub ページ を確認してください。

LINSTORを使用したDRBDクラスタは、このマニュアルの指示に従ってインストールおよび設定できます。詳細は クラスタの初期化 を参照してください。

OpenNebulaとDRBDクラスタは、OpenNebulaのフロントエンドノードとホストノードを両方のクラスタに含める必要がありますが、これ以外は互いに独立にできます。

ホストノードは、仮想マシン・イメージがネットワークを介して接続されるためローカルの LINSTOR ストレージプールを必要としません [5]

9.3. 配備オプション

LINSTOR リソースグループ リソースグループの作成 を使用して構成することをお勧めします。以前の自動配置およびデプロイメントノードモードは非推奨です。

9.4. OpenNebula アドオンの構成

9.4.1. OpenNebula へのドライバーの追加

/etc/one/oned.conf の以下のセクションを変更します。

Add linstor to the list of drivers in the TM_MAD and DATASTORE_MAD sections:

TM_MAD = [
    EXECUTABLE = "one_tm",
    ARGUMENTS = "-t 15 -d dummy,lvm,shared,fs_lvm,qcow2,ssh,vmfs,ceph,linstor"
]
DATASTORE_MAD = [
    EXECUTABLE = "one_datastore",
    ARGUMENTS  = "-t 15 -d dummy,fs,lvm,ceph,dev,iscsi_libvirt,vcenter,linstor -s shared,ssh,ceph,fs_lvm,qcow2,linstor"
]
linstorDATASTORE_MAD 引数で 2 回指定されます。

TM_MAD_CONF および DS_MAD_CONF セクションを新たに追加します。

TM_MAD_CONF = [
    NAME = "linstor", LN_TARGET = "NONE", CLONE_TARGET = "SELF", SHARED = "yes", ALLOW_ORPHANS="yes",
    TM_MAD_SYSTEM = "ssh,shared", LN_TARGET_SSH = "NONE", CLONE_TARGET_SSH = "SELF", DISK_TYPE_SSH = "BLOCK",
    LN_TARGET_SHARED = "NONE", CLONE_TARGET_SHARED = "SELF", DISK_TYPE_SHARED = "BLOCK"
]
DS_MAD_CONF = [
    NAME = "linstor", REQUIRED_ATTRS = "BRIDGE_LIST", PERSISTENT_ONLY = "NO",
    MARKETPLACE_ACTIONS = "export"
]

これらの変更を行った後、OpenNebula サービスを再起動します。

9.4.2. ノードの構成

フロントエンドノードは、LINSTOR を介してストレージノードおよびホストノードにコマンドを発行します。

ストレージノードは、VM のディスクイメージをローカルに保持します。

ホストノードは、インスタンス化された VM の実行を担当し、通常、必要なイメージ用のストレージを LINSTOR ディスクレスモードを介してネットワーク経由で接続します。

すべてのノードに DRBD9 と LINSTOR がインストールされている必要があります。このプロセスの詳細は DRBD9 のユーザーズガイド を参照ください。

両方の役割のすべての要件を満たす限り、フロントエンドノードとホストノードをプライマリロールに加えてストレージノードとして機能させることができます。

フロントエンドノードの構成

使用するコントロールノードがフロントエンドノードから到達可能であることを確認してください。ローカルで実行される LINSTOR コントローラーでは linstor node list を、リモートで実行する LINSTOR コントローラーでは linstor --controllers "<IP:PORT>" node list を使用することで簡単にテストできます。

ホストノードの構成

ホストノードでは、LINSTOR サテライトプロセスが実行され、フロントエンドノードおよびストレージノードと同じ LINSTOR クラスターのメンバーである必要があります。また、オプションでローカルにストレージを持つことができます。 oneadmin ユーザーがパスワードなしでホスト間で SSH できる場合、SSH システムデータストアでもライブマイグレーションを使用できます。

ストレージノードの構成

フロントエンドノードとホストノードのみが OpenNebula のインストールを必要としますが、oneadmin ユーザーはパスワードなしでストレージノードにアクセスできる必要があります。 oneadmin ユーザーアカウントを手動で構成する方法については、ディストリビューションの OpenNebula インストールガイドを参照してください。

The storage nodes must use storage pools created with a driver that’s capable of making snapshots, such as the thin LVM plugin.

Given two physical devices (/dev/sdX and /dev/sdY), the following example shows how you would prepare thin-provisioned storage, by using LVM commands, to back a LINSTOR storage pool or pools. The example uses generic names for the volume group and thin pool.

pvcreate /dev/sdX /dev/sdY
vgcreate drbdpool /dev/sdX /dev/sdY
lvcreate -l 95%VG --poolmetadatasize 8g -T /dev/drbdpool/drbdthinpool
Set the thin-provisioned logical volume’s metadata space to a reasonable size because if it becomes full it can be difficult to resize. For this reason, you might also want to configure monitoring or automatic extension of your LVM thin-provisioned logical volumes. Refer to the “Size of pool metadata LV” section in man lvmthin for more details.

Next, you would create a LINSTOR storage pool or pools using the /dev/drbdpool/drbdthinpool device as the backing storage.

9.4.3. 管理アカウントの権限

oneadmin ユーザーには、ストレージノードで mkfs コマンドへのパスワードなしsudoアクセス権が必要です。

oneadmin ALL=(root) NOPASSWD: /sbin/mkfs
Groups

Be sure to consider the groups that you need to add the oneadmin user to so that the oneadmin user can access the devices and programs needed to access storage and instantiate VMs. For this add-on, the oneadmin user must belong to the disk group on all nodes to access the DRBD devices where images are held.

usermod -a -G disk oneadmin

9.4.4. 新しい LINSTOR データストアの作成

ds.conf という名前のデータストア設定ファイルを作成し、 onedatastore ツールを使用して、その設定に基づいて新しいデータストアを作成します。相互に排他的な2つの配備オプション LINSTOR_AUTO_PLACE と LINSTOR_DEPLOYMENT_NODES があります。両方が設定されている場合、LINSTOR_AUTO_PLACE は無視されます。どちらのオプションでも、BRIDGE_LIST は LINSTOR クラスター内のすべてのストレージノードのスペース区切りリストである必要があります。

9.4.5. リソースグループの作成

バージョン1.0.0以降、LINSTORはリソースグループをサポートしています。リソースグループは、そのリソースグループにリンクされているすべてのリソースが共有する設定の一元化されたポイントです。

データストアのリソースグループとボリュームグループを作成します。リソースグループ内でストレージプールを指定する必要があります。指定しないと、OpenNebula のスペースの監視が機能しません。ここでは2ノードの冗長性を持つものを作成し、作成された opennebula-storagepool を使用します。

linstor resource-group create OneRscGrp --place-count 2 --storage-pool opennebula-storagepool
linstor volume-group create OneRscGrp

LINSTOR プラグインを使用して OpenNebula データストアを追加します。

cat >ds.conf <<EOI
NAME = linstor_datastore
DS_MAD = linstor
TM_MAD = linstor
TYPE = IMAGE_DS
DISK_TYPE = BLOCK
LINSTOR_RESOURCE_GROUP = "OneRscGrp"
COMPATIBLE_SYS_DS = 0
BRIDGE_LIST = "alice bob charlie"  #node names
EOI

onedatastore create ds.conf

9.4.6. プラグインの属性

LINSTOR_CONTROLLERS

LINSTOR コントローラープロセスがローカルで実行されていない場合、 LINSTOR_CONTROLLERS を使用して、コントローラーIPとポートのコンマ区切りリストを LINSTOR クライアントに渡すことができます。

LINSTOR_CONTROLLERS = "192.168.1.10:8080,192.168.1.11:6000"

LINSTOR_RESOURCE_GROUP

LINSTOR_RESOURCE_GROUP 属性は、OpenNebula データストアを LINSTOR リソースグループに関連付けるために使用されます。

9.4.7. 廃止された属性

次の属性は非推奨であり、バージョン 2.0 で削除されました。

LINSTOR_CLONE_MODE

LINSTOR は、使用するクローンモードを自動的に決定するようになりました。

LINSTOR は snapshotcopy の2つの異なるクローンモードをサポートしています。これらのモードは LINSTOR_CLONE_MODE 属性で設定されます。

The default mode is snapshot. It uses a LINSTOR snapshot and restores a new resource from this snapshot, which is then a clone of the image. This mode is usually faster than using the copy mode as snapshots are cheap copies.

2番目のモードは copy モードで、元のサイズと同じサイズの新しいリソースを作成し、 dd でデータを新しいリソースにコピーします。このモードは snapshot よりも遅くなりますが、スナップショットメカニズムに依存しないため、より堅牢です。別のlinstorデータストアにイメージをクローンする場合にも使用されます。

LINSTOR_STORAGE_POOL

LINSTOR_STORAGE_POOL 属性は、データストアが使用するLINSTORストレージプールを選択するために使用されます。リソースグループが使用される場合、ストレージプールは自動選択フィルターオプションで選択できるため、この属性は必要ありません。

LINSTOR_AUTO_PLACE または LINSTOR_DEPLOYMENT_NODES が使用され、LINSTOR_STORAGE_POOL が設定されていない場合、LINSTORで DfltStorPool にフォールバックします。

LINSTOR_AUTO_PLACE

LINSTOR_AUTO_PLACE オプションは、冗長ストレージの個数を表し、1からストレージノードの総数の間の数値です。リソースは、冗長性の個数に基づいてストレージノードに自動的に割り当てられます。

LINSTOR_DEPLOYMENT_NODES

LINSTOR_DEPLOYMENT_NODES を使用すると、リソースが常に割り当てられるノードのグループを選択できます。ブリッジリストには、LINSTOR クラスター内のすべてのストレージノードがまだ含まれていることに注意してください。

9.4.8. システムデータストアとしての LINSTOR を構成

LINSTOR ドライバーはシステムデータストアとしても使用できます。構成は通常のデータストアとかなり似ていますが、いくつか相違があります。

cat >system_ds.conf <<EOI
NAME = linstor_system_datastore
TM_MAD = linstor
TYPE = SYSTEM_DS
DISK_TYPE = BLOCK
LINSTOR_RESOURCE_GROUP = "OneSysRscGrp"
BRIDGE_LIST = "alice bob charlie"  # node names
EOI

onedatastore create system_ds.conf

また、新しいsysデータストアIDをイメージデータストア(COMMA区切り)の COMPATIBLE_SYS_DS に追加します。そうしないと、スケジューラはそれらを無視します。

揮発性ディスクを使用したライブマイグレーションが必要な場合は、KVMの --unsafe オプションを有効にする必要があります。https://docs.opennebula.org/5.8/deployment/open_cloud_host_setup/kvm_driver.html を参照してください。

9.5. ライブマイグレーション

ライブマイグレーションは、SSH システムデータストアとnfs共有システムデータストアを使用してもサポートされます。

9.6. 空き容量の計算

空き容量は、リソースが自動的に配備されるか、ノードごとに配備されるかに応じて異なって計算されます。

ノードごとに配置されるデータストアでは、リソースが配備されているすべてのノードの最も制限的なストレージプールに基づいて空き領域がレポートされます。たとえば、総ストレージ容量の最小値を持つノードの容量を使用してデータストアの合計サイズを決定し、空き容量が最小のノードを使用してデータストアの残りの容量を決定します。

自動配置を使用するデータストアの場合、サイズと残りの領域は、LINSTORによって報告されたデータストアで使用される集約ストレージプールに基づいて決定されます。

10. OpenStackでのLINSTORボリューム

This chapter describes using LINSTOR® to provision persistent, replicated, and high-performance block storage for OpenStack.

10.1. OpenStack の紹介

OpenStack consists of a wide range of individual services; The service responsible for provisioning and managing block storage is called Cinder. Other OpenStack services such as the compute instance service Nova can request volumes from Cinder. Cinder will then make a volume accessible to the requesting service.

LINSTOR は、ボリュームドライバーを使用して Cinder と統合できます。ボリュームドライバーは、CinderAPI への呼び出しを LINSTOR コマンドに変換します。例えば Cinderにボリュームを要求すると、LINSTOR に新しいリソースが作成され、Cinder ボリュームのスナップショットは LINSTOR のスナップショットに変換されます。

10.2. OpenStack 用の LINSTOR のインストール

An initial installation and configuration of DRBD® and LINSTOR must be completed before using the OpenStack driver.

この時点で、LINSTOR クライアントを使用してストレージクラスターノードを一覧表示できるはずです。

╭────────────────────────────────────────────────────────────────────────────╮
┊ Node                      ┊ NodeType  ┊ Addresses                 ┊ State  ┊
╞════════════════════════════════════════════════════════════════════════════╡
┊ cinder-01.openstack.test  ┊ COMBINED  ┊ 10.43.224.21:3366 (PLAIN) ┊ Online ┊
┊ cinder-02.openstack.test  ┊ COMBINED  ┊ 10.43.224.22:3366 (PLAIN) ┊ Online ┊
┊ storage-01.openstack.test ┊ SATELLITE ┊ 10.43.224.11:3366 (PLAIN) ┊ Online ┊
┊ storage-02.openstack.test ┊ SATELLITE ┊ 10.43.224.12:3366 (PLAIN) ┊ Online ┊
┊ storage-03.openstack.test ┊ SATELLITE ┊ 10.43.224.13:3366 (PLAIN) ┊ Online ┊
╰────────────────────────────────────────────────────────────────────────────╯

また、ノードごとに 1 つ以上の ストレージプール を構成する必要があります。このガイドでは、ストレージプールの名前が cinderpool であると仮定しています。LINSTOR は デフォルトで作成されたディスクレスストレージプールを含む、各ノードのストレージプールを一覧表示します。

$ linstor storage-pool list
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node                      ┊ Driver   ┊ PoolName        ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ cinder-01.openstack.test  ┊ DISKLESS ┊                 ┊              ┊               ┊ False        ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ cinder-02.openstack.test  ┊ DISKLESS ┊                 ┊              ┊               ┊ False        ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ storage-01.openstack.test ┊ DISKLESS ┊                 ┊              ┊               ┊ False        ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ storage-02.openstack.test ┊ DISKLESS ┊                 ┊              ┊               ┊ False        ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ storage-03.openstack.test ┊ DISKLESS ┊                 ┊              ┊               ┊ False        ┊ Ok    ┊
┊ cinderpool           ┊ storage-01.openstack.test ┊ LVM_THIN ┊ ssds/cinderpool ┊      100 GiB ┊       100 GiB ┊ True         ┊ Ok    ┊
┊ cinderpool           ┊ storage-02.openstack.test ┊ LVM_THIN ┊ ssds/cinderpool ┊      100 GiB ┊       100 GiB ┊ True         ┊ Ok    ┊
┊ cinderpool           ┊ storage-03.openstack.test ┊ LVM_THIN ┊ ssds/cinderpool ┊      100 GiB ┊       100 GiB ┊ True         ┊ Ok    ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

10.2.1. LINSTOR ドライバをアップグレード

新規インストールの場合、このセクションをスキップして LINSTOR ドライバをインストール に進んでください。

1.x から 2.x へのアップグレード

ドライバーバージョン 2 は、実行時に ボリュームタイプ でこれらのオプションを管理するため、いくつかの静的構成オプションを削除しました。

Option in cinder.conf Status Replace with

linstor_autoplace_count

削除

ボリュームタイプで linstor:redundancy プロパティを使用します。フルクラスターレプリケーションで値 0 を使用することはサポートされていません。the LINSTOR autoplacer オプションを使用してください

linstor_controller_diskless

削除

置き換えは必要ありません。ドライバーは必要に応じて cinder ホスト上にディスクレスリソースを作成します

linstor_default_blocksize

削除

この設定は効果がありませんでした

linstor_default_storage_pool_name

非推奨

この設定は、将来のバージョンで削除するために非推奨になりました。代わりに、ボリュームタイプで linstor:storage_pool プロパティを使用してください

linstor_default_uri

非推奨

より適切な名前の linstor_uris に置き換えられました

linstor_default_volume_group_name

削除

ノードとストレージプールの作成は、ドライバーバージョン 2 で削除されました。OpenStack 用の LINSTOR のインストール を参照してください

linstor_volume_downsize_factor

削除

この設定は目的を果たさず、置き換えなしで削除されました

10.2.2. LINSTOR ドライバをインストール

OpenStack Stein 以降、LINSTOR ドライバーは Cinder プロジェクトの一部です。ドライバはそのまま使用できますが、新しいバージョンで利用できる機能や修正が不足している可能性があります。安定バージョンの OpenStacks 更新ポリシーにより、ドライバーに対するほとんどの改善は、古い安定リリースにバックポートされません。

LINBIT® maintains a fork of the Cinder repository with all improvements to the LINSTOR driver backported to the supported stable versions. Currently, these are:

OpenStack Release Included Version LINBIT Version LINBIT Branch

yoga

1.0.1

2.0.0

linstor/stable/yoga

xena

1.0.1

2.0.0

linstor/stable/xena

wallaby

1.0.1

2.0.0

linstor/stable/wallaby

victoria

1.0.1

2.0.0

linstor/stable/victoria

ussuri

1.0.1

2.0.0

linstor/stable/ussuri

train

1.0.0

2.0.0

linstor/stable/train

stein

1.0.0

2.0.0

linstor/stable/stein

LINSTOR Driver を有効にする正しい手順は、OpenStack ディストリビューションによって異なります。一般に python-linstor パッケージは、Cinder ボリュームサービスを実行しているすべてのホストにインストールする必要があります。次のセクションでは、一般的な OpenStack ディストリビューションへのインストール手順について説明します。

DevStack でインストール

DevStack は、ラボ環境で OpenStack を試すのに最適な方法です。最新のドライバーを使用するには、次の DevStack 構成を使用します。

Listing 25. local.conf
# This ensures the LINSTOR Driver has access to the 'python-linstor' package.
#
# This is needed even if using the included driver!
USE_VENV=True
ADDITIONAL_VENV_PACKAGES=python-linstor

# This is required to select the LINBIT version of the driver
CINDER_REPO=https://github.com/LINBIT/openstack-cinder.git
# Replace linstor/stable/victoria with the reference matching your OpenStack release.
CINDER_BRANCH=linstor/stable/victoria
Kolla でインストール

Kolla は OpenStack コンポーネントをコンテナーにパッケージ化します。次に、例えば Kolla Ansible を使用してデプロイできます。kolla コンテナーで使用可能なカスタマイズオプションを利用して、LINSTOR ドライバーをセットアップできます。

必要な python-linstor パッケージがインストールされるようにするには、次のオーバーライドファイルを使用します。

Listing 26. template-override.j2
{% extends parent_template %}

# Cinder
{% set cinder_base_pip_packages_append = ['python-linstor'] %}

LINBIT バージョンのドライバーをインストールするには kolla-build.conf を更新します。

Listing 27. /etc/kolla/kolla-build.conf
[cinder-base]
type = git
location = https://github.com/LINBIT/openstack-cinder.git
# Replace linstor/stable/victoria with the reference matching your OpenStack release.
reference = linstor/stable/victoria

Cinder コンテナを再構築するには、次のコマンドを実行します。

# A private registry used to store the kolla container images
REGISTRY=deployment-registry.example.com
# The image namespace in the registry
NAMESPACE=kolla
# The tag to apply to all images. Use the release name for compatibility with kolla-ansible
TAG=victoria
kolla-build -t source --template-override template-override.j2 cinder --registry $REGISTRY --namespace $NAMESPACE --tag $TAG
Kolla Ansible デプロイメント

Kolla Ansible を使用して OpenStack をデプロイする場合は、次のことを確認する必要があります。

  • 上記のセクションで作成されたカスタム Cinder イメージは、Cinder サービスのデプロイメントが有効になっている場合に使用されます。

Listing 28. /etc/kolla/globals.yml
# use "source" images
kolla_install_type: source
# use the same registry as for running kolla-build above
docker_registry: deployment-registry.example.com
# use the same namespace as for running kolla-build above
docker_namespace: kolla
# deploy cinder block storage service
enable_cinder: "yes"
# disable verification of cinder back ends, kolla-ansible only supports a small subset of available back ends for this
skip_cinder_backend_check: True
# add the LINSTOR back end to the enabled back ends. For back end configuration see below
cinder_enabled_backends:
  - name: linstor-drbd

LINSTOR ドライバー構成は kolla-ansible のオーバーライドディレクトリの 1 つに配置できます。使用可能な構成オプションの詳細については、以下のセクションを参照してください。

Listing 29. /etc/kolla/config/cinder/cinder-volume.conf
[linstor-drbd]
volume_backend_name = linstor-drbd
volume_driver = cinder.volume.drivers.linstordrv.LinstorDrbdDriver
linstor_uris = linstor://cinder-01.openstack.test,linstor://cinder-02.openstack.test
OpenStack Ansible デプロイメント

OpenStack Ansible は、OpenStack 環境を構成およびデプロイするための Ansible プレイブックを提供します。これにより、デプロイメントをきめ細かくカスタマイズできるため、LINSTOR ドライバーを直接セットアップできます。

Listing 30. /etc/openstack_ansile/user_variables.yml
cinder_git_repo: https://github.com/LINBIT/openstack-cinder.git
cinder_git_install_branch: linstor/stable/victoria

cinder_user_pip_packages:
  - python-linstor

cinder_backends: (1)
  linstor-drbd:
   volume_backend_name: linstor-drbd
   volume_driver: cinder.volume.drivers.linstordrv.LinstorDrbdDriver
   linstor_uris: linstor://cinder-01.openstack.test,linstor://cinder-02.openstack.test
1 使用可能なバックエンドパラメータの詳細な説明は、以下のセクションにあります。
汎用 Cinder デプロイメント

他の形式の OpenStack デプロイメントの場合、このガイドでは一般的なヒントのみを提供できます。

LINSTOR ドライバーのバージョンを更新するには、インストールされた cinder ディレクトリを見つけてください。考えられるパスは次のとおりです。

/usr/lib/python*/dist-packages/cinder/
/usr/lib/python*/site-packages/cinder/

LINSTOR ドライバーは、cinder ディレクトリにある linstordrv.py というファイルです。

$CINDER_PATH/volume/drivers/linstordrv.py

ドライバを更新するには、ファイルを LINBIT リポジトリのファイルに置き換えます

RELEASE=linstor/stable/victoria
curl -fL "https://raw.githubusercontent.com/LINBIT/openstack-cinder/$RELEASE/cinder/volume/drivers/linstordrv.py" > $CINDER_PATH/volume/drivers/linstordrv.py

登録されたものを更新するには、Python キャッシュを削除する必要がある場合もあります。

rm -rf $CINDER_PATH/volume/drivers/__pycache__

10.3. Cinder 用の LINSTOR バックエンドを構成

LINSTOR ドライバーを使用するには、Cinder ボリュームサービスを構成します。Cinder 構成ファイルを編集してから、Cinder ボリュームサービスを再起動することで構成できます。

ほとんどの場合、Cinder 構成ファイルは /etc/cinder/cinder.conf にあります。一部のデプロイメントでは、このファイルを事前に編集できます。詳細は上記のセクションを参照してください。

LINSTOR を使用して新しいボリュームバックエンドを構成するには、次のセクションを cinder.conf に追加します。

[linstor-drbd]
volume_backend_name = linstor-drbd (1)
volume_driver = cinder.volume.drivers.linstordrv.LinstorDrbdDriver (2)
linstor_uris = linstor://cinder-01.openstack.test,linstor://cinder-02.openstack.test (3)
linstor_trusted_ca = /path/to/trusted/ca.cert (4)
linstor_client_key = /path/to/client.key (5)
linstor_client_cert = /path/to/client.cert (5)
# Deprecated or removed in 2.0.0
linstor_default_storage_pool_name = cinderpool (6)
linstor_autoplace_count = 2 (7)
linstor_controller_diskless = true (8)
# non-linstor-specific options
... (9)
ここで説明するパラメーターは、Linbit が提供する最新のリリースに基づいています。 OpenStack に含まれているドライバーは、これらのパラメーターのすべてをサポートしているとは限りません。詳細は OpenStack ドライバーのドキュメント を参照ください。
1 ボリュームバックエンド名。 Cinder 構成で一意である必要があります。セクション全体で同じ名前を共有する必要があります。この名前は enabled_backends 設定の cinder.conf で、および新しいボリュームタイプを作成するときに再び参照されます。
2 使用する LINSTOR ドライバーのバージョン。2 つのオプションがあります。
  • cinder.volume.drivers.linstordrv.LinstorDrbdDriver

  • cinder.volume.drivers.linstordrv.LinstorIscsiDriver

    どのドライバーを使用するかは LINSTOR のセットアップと要件によって異なります。各選択肢の詳細は 以下のセクション を参照ください。

3 The URL(s) of the LINSTOR Controller(s). Multiple Controllers can be specified to make use of LINSTOR high availability. If not set, defaults to linstor://localhost.
In driver versions before 2.0.0, this option is called linstor_default_uri
4 HTTPS が有効の場合、 指定された証明書を使用して LINSTOR コントローラーの信頼性を検証します。
5 HTTPS が有効の場合、 指定されたキーと証明書が、認証のために LINSTOR コントローラーで使用されます。
6 Deprecated in 2.0.0, use volume types instead. The storage pools to use when placing resources. Applies to all diskful resources created. Defaults to DfltStorPool.
7 2.0.0 で非推奨になったので、代わりに volume types を使用してください。特定のボリュームに対して作成する複製数。値が 0 の場合、すべてのノードに複製が作成されます。デフォルトは 0 です。
8 Removed in 2.0.0, volumes are created on-demand by the driver. If set to true, ensures that at least one (diskless) replica is deployed on the Cinder Controller host. This is useful for ISCSI transports. Defaults to true.
9 ここでは、より一般的な Cinder オプションを指定できます。たとえば、iSCSI コネクタの場合は target_helper = tgtadm です。
複数の LINSTOR バックエンドを構成して、それぞれに異なる名前と構成オプションを選択することもできます。

LINSTOR バックエンドを構成した後、それも有効にする必要があります。これを cinder.conf の有効なバックエンドのリストに追加し、オプションでデフォルトのバックエンドとして設定します。

[DEFAULT]
...
default_volume_type = linstor-drbd-volume
enabled_backends = lvm,linstor-drbd
...

最後のステップとして、Cinder 構成を変更した場合、またはドライバー自体を更新した場合は、必ず Cinder サービスを再起動してください。サービスを再起動する方法については、OpenStack ディストリビューションのドキュメントを確認してください。

10.3.1. トランスポートプロトコルの選択

Cinder のトランスポートプロトコルは、クライアント(たとえば、nova-compute)が実際のボリュームにアクセスする方法です。LINSTOR を使用すると、異なるトランスポートを使用する 2 つの異なるドライバーを選択できます。

  • cinder.volume.drivers.linstordrv.LinstorDrbdDriver : トランスポートプロトコルとして DRBD を使用

  • cinder.volume.drivers.linstordrv.LinstorIscsiDriver : トランスポートプロトコルとして iSCSI を使用

トランスポートプロトコルとして DRBD を使用

LinstorDrbdDriver は、クライアント(つまり、nova-compute)がリクエストを発行したノードでボリュームの複製がローカルで利用できるようにすることで機能します。これはすべての compute ノードが同じ LINSTOR クラスターの一部である LINSTOR サテライトを実行している場合にのみ機能します。

このオプションの利点は次のとおりです。

  • セットアップが完了すると、Cinder ホストはデータパスに関与しなくなります。ボリュームへのすべての読み書きは、構成された対向ノード間でのレプリケーションを処理するローカル DRBD モジュールによって処理されます。

  • Cinder ホストはデータパスに関与していないため、Cinder サービスが中断しても、すでに接続されているボリュームには影響しません。

既知の制限:

  • すべてのホストとハイパーバイザーが DRBD ボリュームの使用をサポートしているわけではありません。Linux ホストと kvm ハイパーバイザーがデプロイメントの対象として制限されます。

  • 接続しているボリュームと使用中のボリュームのサイズ変更は完全には機能しません。サイズ変更自体は成功していますが、サービスは再起動するまで VM にサイズ変更を伝えません。

  • マルチアタッチ(複数の VM に同じボリュームをアタッチする)はサポートされていません。

  • 暗号化されたボリューム は、DRBD デバイスの udev ルールが設定されている場合にのみ機能します。

    udev ルールは drbd-utils パッケージの一部であるか、独自の drbd-udev パッケージの中にあります。
iSCSI トランスポートプロトコルを使用

Cinder ボリュームをエクスポートするデフォルトの方法は、iSCSI 経由です。これにより最大の互換性が得られます。iSCSI は、VMWare、Xen、HyperV、または KVM などのあらゆるハイパーバイザーで使用できます。

欠点は、すべてのデータを(ユーザースペースの)iSCSI デーモンで処理するためにCinder ノードに送信する必要があることです。これは、データがカーネル/ユーザスペース境界を通過する必要があることを意味し、パフォーマンスに影響を与えます。

もう1つの欠点は、単一障害点になりうる点です。iSCSI デーモンを実行している Cinder ノードがクラッシュすると、他のノードはボリュームにアクセスできなくなります。これを軽減するために自動フェイルオーバー用に Cinder を構成する方法はいくつかありますが、かなりの労力が必要です。

In driver versions before 2.0.0, the Cinder host needs access to a local replica of every volume. This can be achieved by either setting linstor_controller_diskless=True or using linstor_autoplace_count=0. Newer driver versions will create such a volume on-demand.

10.3.2. LINSTOR バックエンドのステータスを確認

すべてのバックエンドが稼働していることを確認するには、OpenStack コマンドラインを使用できます。

$ openstack volume service list
+------------------+----------------------------------------+------+---------+-------+----------------------------+
| Binary           | Host                                   | Zone | Status  | State | Updated At                 |
+------------------+----------------------------------------+------+---------+-------+----------------------------+
| cinder-scheduler | cinder-01.openstack.test               | nova | enabled | up    | 2021-03-10T12:24:37.000000 |
| cinder-volume    | cinder-01.openstack.test@linstor-drbd  | nova | enabled | up    | 2021-03-10T12:24:34.000000 |
| cinder-volume    | cinder-01.openstack.test@linstor-iscsi | nova | enabled | up    | 2021-03-10T12:24:35.000000 |
+------------------+----------------------------------------+------+---------+-------+----------------------------+

Horizon GUI をデプロイしている場合は、代わりに Admin > System Information > Block Storage Service を確認してください。

In the above example all configured services are enabled and up. If there are any issues, examine the logs of the Cinder Volume service.

10.4. LINSTOR の新しいボリュームタイプを作成

Cinder を使用してボリュームを作成する前に、ボリュームタイプを作成する必要があります。これは、コマンドラインから実行できます。

# Create a volume using the default back end
$ openstack volume type create default
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| description | None                                 |
| id          | 58365ffb-959a-4d91-8821-5d72e5c39c26 |
| is_public   | True                                 |
| name        | default                              |
+-------------+--------------------------------------+
# Create a volume using a specific back end
$ openstack volume type create --property volume_backend_name=linstor-drbd linstor-drbd-volume
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| description | None                                 |
| id          | 08562ea8-e90b-4f95-87c8-821ac64630a5 |
| is_public   | True                                 |
| name        | linstor-drbd-volume                  |
| properties  | volume_backend_name='linstor-drbd'   |
+-------------+--------------------------------------+

あるいは Horizon GUI でボリュームタイプを作成することもできます。Admin > Volume > Volume Types から “Create Volume Type” をクリックします。 volume_backend_name を “Extra Specs” として追加することで、バックエンドを割り当てることができます。

10.4.1. ボリュームタイプの詳細設定

各ボリュームタイプは、Horizon GUI で呼び出されるプロパティまたは “Extra Specs” を追加することでカスタマイズできます。

コマンドラインでボリュームタイプにプロパティを追加するには、次を使用します。

openstack volume type set linstor_drbd_b --property linstor:redundancy=5

または GUI で Admin > Volume > Volume Types から プロパティを設定することもできます。 Actions 列で、ドロップダウンメニューを開き View Extra Specs ボタンをクリックします。これにより、プロパティの作成、編集、削除に使用できるダイアログが開きます。

使用可能なボリュームタイプのプロパティ
linstor:diskless_on_remaining

自動配置後、選択されていないノードにディスクレスレプリカを作成する。

linstor:do_not_place_with_regex

Do not place the resource on a node which has a resource with a name matching the regular expression.

linstor:layer_list

リソースを適用するためのレイヤーのコンマ区切りリスト。空の場合、デフォルトはDRBD、Storageです。

linstor:provider_list

使用するプロバイダーのコンマ区切りリスト。空の場合、LINSTOR は適切なプロバイダーを自動的に選択します。

linstor:redundancy

作成するレプリカの数。デフォルトは 2 です。

linstor:replicas_on_different

自動配置を使用してストレージをプロビジョニングする場所を決定するときに、自動配置選択ラベルとして使用される key または key=value アイテムのコンマ区切りのリスト。

linstor:replicas_on_same

自動配置を使用してストレージをプロビジョニングする場所を決定するときに、自動配置選択ラベルとして使用される key または key=value アイテムのコンマ区切りのリスト。

linstor:storage_pool

自動配置時に使用するストレージプールのコンマ区切りリスト。

linstor:property:<key>

キーの前に linstor:property: が付いている場合、そのキーは LINSTOR プロパティとして解釈されます。プロパティはボリュームタイプ用に作成された Resource Group に設定されます。

OpenStack では、プロパティ名に / を使用できません。LINSTOR プロパティ名に / が含まれている場合は : に置き換えます。

例: quorum policy を変更するには DrbdOptions/auto-quorum を設定する必要があります。これは OpenStack の linstor:property:DrbdOptions/auto-quorum プロパティを設定することで実行できます。

10.5. ボリュームの使用

ボリュームタイプを構成したら、それを使用して新しいボリュームのプロビジョニングを開始できます。

たとえば、コマンドラインで単純な 1Gb ボリュームを作成するには、次のコマンドを使用できます。

openstack volume create --type linstor-drbd-volume --size 1 \
  --availability-zone nova linstor-test-vol
openstack volume list
/etc/cinder/cinder.confdefault_volume_type = linstor-drbd-volume を設定した場合、上記の openstack volume create …​ コマンドから --type linstor-drbd-volume を省略できます。

10.6. トラブルシューティング

このセクションでは、LINSTOR ボリュームとスナップショットの使用で問題が発生した場合の対処方法について説明します。

10.6.1. Horizon でエラーメッセージを確認する

Every volume and snapshot has a Messages tab in the Horizon dashboard. In case of errors, you can use the list of messages as a starting point for further investigation. Some common messages in case of errors:

create volume from backend storage:Driver failed to create the volume.

This message shows that there was an error creating a new volume. Check the Cinder Volume service logs for more details.

schedule allocate volume:Could not find any available weighted backend.

If this is the only error message, this means that the Cinder Scheduler could not find a volume back end suitable for creating the volume. This is most likely because:

  • The volume back end is offline. See LINSTOR バックエンドのステータスを確認.

  • The volume back end does not have enough free capacity to fulfil the request. Check the output of cinder get-pools --detail and linstor storage-pool list to verify that the requested capacity is available.

10.6.2. Cinder ボリュームサービスの確認

LINSTOR ドライバーは、Cinder ボリュームサービスの一部として呼び出されます。

Distribution Log location or command

DevStack

journalctl -u devstack@c-vol

10.6.3. Compute サービスログの確認

一部の問題は Cinder サービスログに記録されませんが、ボリュームの実際のコンシューマー、おそらく Compute サービス(Nova)に記録されます。ボリュームサービスと同様に、確認するホストと正確な場所は、Openstack ディストリビューションによって異なります。

Distribution Log location or command

DevStack

journalctl -u devstack@n-cpu

11. DockerのLINSTORボリューム

This chapter describes LINSTOR® volumes in Docker as managed by the LINSTOR Docker Volume Plugin.

11.1. Docker の概要

Docker is a platform for developing, shipping, and running applications in the form of Linux containers. For stateful applications that require data persistence, Docker supports the use of persistent volumes and volume drivers.

LINSTOR Docker Volume Plugin は、LINSTORクラスタから永続的ボリュームをプロビジョニングするDockerコンテナ用のボリュームドライバです。

11.2. Installing the LINSTOR Plugin for Docker

To install the linstor-docker-volume plugin provided by LINBIT®, you will need to have a working LINSTOR cluster. After that the plugin can be installed from the public docker hub.

# docker plugin install linbit/linstor-docker-volume --grant-all-permissions
The --grant-all-permissions flag will automatically grant all permissions needed to successfully install the plugin. If you want to manually accept these, omit the flag from the command above.

The implicit :latest tag is the latest amd64 version. We currently also build for arm64 with the according tag. To installing the arm64 plugin, enter the following command:

# docker plugin install linbit/linstor-docker-volume:arm64 --grant-all-permissions

11.3. Configuring the LINSTOR Plugin for Docker

As the plugin has to communicate to the LINSTOR Controller software using the LINSTOR Python library, we must tell the plugin where to find the LINSTOR controller node in its configuration file:

# cat /etc/linstor/docker-volume.conf
[global]
controllers = linstor://hostnameofcontroller

A more extensive example is this:

# cat /etc/linstor/docker-volume.conf
[global]
storagepool = thin-lvm
fs = ext4
fsopts = -E discard
size = 100MB
replicas = 2

11.4. Using the LINSTOR Plugin for Docker

The following are some examples of how you might use the LINSTOR Docker Volume plugin. In the following we expect a cluster consisting of three nodes (alpha, bravo, and charlie).

11.4.1. 例1 – 典型的なdockerパターン

ノードalpha上:

$ docker volume create -d linbit/linstor-docker-volume \
        --opt fs=xfs --opt size=200 lsvol
$ docker run -it --rm --name=cont \
        -v lsvol:/data --volume-driver=linbit/linstor-docker-volume busybox sh
$ root@cont: echo "foo" > /data/test.txt
$ root@cont: exit

ノードbravo上:

$ docker run -it --rm --name=cont \
        -v lsvol:/data --volume-driver=linbit/linstor-docker-volume busybox sh
$ root@cont: cat /data/test.txt
  foo
$ root@cont: exit
$ docker volume rm lsvol

11.4.2. 例2 – ホスト指定で1つのディスクフル割り当て、2つのディスクレスノード

$ docker volume create -d linbit/linstor-docker-volume --opt nodes=bravo lsvol

11.4.3. 例3 – どこかに1つのディスクフル割り当て、2つのディスクレスノード

$ docker volume create -d linbit/linstor-docker-volume --opt replicas=1 lsvol

11.4.4. 例4 – ホスト指定で2つのディスクフル割り当て、charlie はディスクレス

$ docker volume create -d linbit/linstor-docker-volume --opt nodes=alpha,bravo lsvol

11.4.5. 例5 – どこかに2つのディスクフル割り当て、1つのディスクレスノード

$ docker volume create -d linbit/linstor-docker-volume --opt replicas=2 lsvol

11.4.6. 例6 – Docker swarm manager ノードからのサービスで LINSTOR ボリュームを使用する

$ docker service create \
        --mount type=volume,src=lsvol,dst=/data,volume-driver=linbit/linstor-docker-volume \
        --name swarmsrvc busybox sh -c "while true; do sleep 1000s; done"
Docker services do not accept the -v or --volume syntax, you must use the --mount syntax. Docker run will accept either syntax.

12. LINSTOR Gateway for Highly Available NFS/iSCSI/NVMe-oF Storage

LINSTOR Gateway manages highly available NVMe-oF targets, iSCSI targets, and NFS exports by leveraging both LINSTOR and DRBD Reactor.

12.1. LINSTOR Gateway の要件

Before you can use LINSTOR Gateway, you will need to have an initialized LINSTOR cluster, with DRBD Reactor, as well as NVMe-oF, NFS, iSCSI utilities installed and configured. The following sections detail how to meet these requirements.

12.1.1. LINSTOR クラスター

LINSTOR Gatewayを使用する前提として、LINSTORクラスタをセットアップする必要があります。LINSTORの設定オプションの詳細については、 LINSTORユーザーズガイド を参照してください。

For NVMe-oF, iSCSI, and NFS, a LINSTOR storage-pool, resource-group, and volume-group for LINSTOR Gateway need to be created before use. The following section provides example commands for setting up the prerequisites in a three-node LINSTOR cluster.

各ノードで、物理デバイス /dev/sdb を使用して、LVMで管理されたストレージプールを作成します。

# linstor physical-storage create-device-pool
          --pool-name lvpool LVM LINSTOR1 /dev/sdb --storage-pool lvmpool
# linstor physical-storage create-device-pool
          --pool-name lvpool LVM LINSTOR2 /dev/sdb --storage-pool lvmpool
# linstor physical-storage create-device-pool
          --pool-name lvpool LVM LINSTOR3 /dev/sdb --storage-pool lvmpool

前のコマンドで作成したストレージプールをバックアップとするリソースグループとボリュームグループを作成します。

# linstor resource-group create iscsi_group --storage-pool lvmpool --place-count 2
# linstor resource-group create nfs_group --storage-pool lvmpool --place-count 3
# linstor resource-group create nvme_group --storage-pool lvm --place-count 2
# linstor volume-group create iscsi_group
# linstor volume-group create nfs_group
# linstor volume-group create nvmeof_group

LINSTOR Gatewayを使用するには、各SatelliteノードでLINSTOR Satelliteのコンフィギュレーションを変更する必要があります。 以下のファイルをお好みのテキスト・エディターで編集または作成してください。

# vim /etc/linstor/linstor_satellite.toml

Satelliteの設定に次の内容を追加します。

[files]
  allowExtFiles = [
    "/etc/systemd/system",
    "/etc/systemd/system/linstor-satellite.service.d",
    "/etc/drbd-reactor.d"
  ]

Satellite設定の変更を保存し、全ノードでSatelliteサービスを再起動して変更を読み込みます。

# systemctl restart linstor-satellite

12.1.2. DRBD Reactor

DRBD Reactorは、クラスタ内のiSCSI、NFS、またはNVMe-oFリソースをオーケストレーションするためのデーモンです。クラスタ内のすべてのノードにインストールし、サービスを有効にする必要があります。

DRBD Reactorの主な機能は次のようにまとめられます。各DRBD Reactorデーモンは、DRBD Reactorが管理するサービスの引き継ぎを試行します。競争に勝ったデーモンは、最初の勝者ができなくなるまで、他のノードがサービスをアクティブにするのを抑えます。新しい勝者がDRBD Reactorの管理サービスを引き継ぐため、高可用性を実現します。

DRBD Reactorのインストールと設定オプションの詳細については、 DRBD Reactor GitHubページ を参照してください。

すべてのノードでDRBD Reactorサービスを開始し、有効にします。

# systemctl enable --now drbd-reactor

これに加えて、DRBD Reactorの構成が変更されたときに自動的にリロードするように設定する必要があります。 DRBD Reactor README に記載されているように、すべてのノードでこのスニペットを実行します。

# cp examples/drbd-reactor-reload.{path,service} /etc/systemd/system/
# systemctl enable --now drbd-reactor-reload.path

DRBD Reactorは、LINSTOR Gatewayと統合する際にPacemakerのリソースエージェントを使用します。また、クラスタ内のすべてのノードにリソースエージェントをインストールする必要があります。このパッケージは、RPMおよびDEBベースのディストリビューションでは、resource-agents という名前です。

# dnf -y install resource-agents

12.1.3. NVME-oF, iSCSI, and NFS Utilities

To configure NVMe-oF targets for LINSTOR Gateway, you will need to install the NVMe-of command line interface on all nodes.

The nvmetcli package can be installed on RPM based systems using the following command:

# dnf -y install nvmetcli

LINSTOR Gatewayは、すべてのノードにLIO iSCSIの実装をインストールする必要があります。

targetcli パッケージはRPMベースのシステムに以下のコマンドでインストールすることができます。

# dnf -y install targetcli

LINSTOR GatewayでNFSをサポートするためには、各ノードにnfs-serverユーティリティがインストールされている必要があります。

nfs-utils パッケージは、RPM ベースのシステムで、以下のコマンドでインストールできます。

# dnf -y install nfs-utils

NFSサーバをsystemdで有効にすると、DRBD Reactorsのサービス管理機能と競合するため、有効にしないでください。nfs-serverサービスを無効にし、次のコマンドを使用して無効になっていることを確認します。

# systemctl disable nfs-server --now
# systemctl status nfs-server

上記のコマンドの出力で、サービスが inactivedisabled としてリストアップされていることを確認します。

● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; ..snip..)
   Active: inactive (dead)

12.2. 要求事項を満たしているかどうかの検証

LINSTOR Gatewayを使い始める前にできる最後のことは、これまでのセクションで説明した前提条件を満たしているかどうかを確認することです。

12.2.1. コンポーネントがインストールされていることを確認

必要なコンポーネントがすべて揃っていることを確認しましょう。このガイドでは、linstor-gatewayを使う前に、ストレージ・プール、リソース・グループ、ボリューム・グループを備えたLINSTORクラスタをすでにインストールし、構成していることを想定しています。

初期化されたLINSTORクラスタに加えて、以下のツールが全ノード上に存在する必要があります。

  • linstor-client

  • drbd-reactor

  • nvmetcli

  • targetcli (iSCSI 用)

  • nfs-utils (RPM) or nfs-common (DEB)

  • nfs-server (RPM) or nfs-kernel-server (DEB)

  • resource-agents

LINSTOR Gatewayは、前提条件となるツールが存在するかどうかをチェックするユーティリティを提供します。

# linstor-gateway check-health

このコマンドは、必要なコンポーネントをすべてインストールした場合、以下のような出力が表示されるはずです。エラーが報告された場合、先に進む前にそのエラーを解決する必要があります。

[✓] LINSTOR
[✓] drbd-reactor
[✓] Resource Agents
[✓] iSCSI
[✓] NVMe-oF
[✓] NFS

12.2.2. LINSTORクラスタ初期化の確認

LINSTORクラスタが正しく初期化されていることを、以下のコマンドの出力と比較することで確認してください。

すべてのLINSTORノードがSatelliteまたはCombinedタイプとしてリストアップされていること、クォーラムをサポートするために3台(またはそれ以上)あることを確認してください。

# linstor node list
╭────────────────────────────────────────────────────────────╮
┊ Node     ┊ NodeType  ┊ Addresses                  ┊ State  ┊
╞════════════════════════════════════════════════════════════╡
┊ LINSTOR1 ┊ COMBINED  ┊ 172.16.16.111:3366 (PLAIN) ┊ Online ┊
┊ LINSTOR2 ┊ SATELLITE ┊ 172.16.16.112:3366 (PLAIN) ┊ Online ┊
┊ LINSTOR3 ┊ SATELLITE ┊ 172.16.16.113:3366 (PLAIN) ┊ Online ┊
╰────────────────────────────────────────────────────────────╯

LINSTORのストレージプールリストにLVMまたはZFSでバックアップされたストレージプールが含まれていることを確認します。

# linstor storage-pool list
╭─────────────────────────────────────────────────────────..snip..─────────╮
┊ StoragePool          ┊ Node     ┊ Driver   ┊ PoolName ┊ ..snip.. ┊ State ┊
╞═════════════════════════════════════════════════════════..snip..═════════╡
┊ DfltDisklessStorPool ┊ LINSTOR1 ┊ DISKLESS ┊          ┊ ..snip.. ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ LINSTOR2 ┊ DISKLESS ┊          ┊ ..snip.. ┊ Ok    ┊
┊ DfltDisklessStorPool ┊ LINSTOR3 ┊ DISKLESS ┊          ┊ ..snip.. ┊ Ok    ┊
┊ lvmpool              ┊ LINSTOR1 ┊ LVM      ┊ lvpool   ┊ ..snip.. ┊ Ok    ┊
┊ lvmpool              ┊ LINSTOR2 ┊ LVM      ┊ lvpool   ┊ ..snip.. ┊ Ok    ┊
┊ lvmpool              ┊ LINSTOR3 ┊ LVM      ┊ lvpool   ┊ ..snip.. ┊ Ok    ┊
╰─────────────────────────────────────────────────────────..snip..─────────╯

ストレージプールを使用するLINSTORリソースグループが少なくとも1つ作成されていることを確認してください。また、各リソース-グループが対応するボリューム-グループを持っていることを確認してください。

# linstor resource-group list
╭────────────────────────────────────────────────────────────────╮
┊ ResourceGroup ┊ SelectFilter            ┊ VlmNrs ┊ Description ┊
╞════════════════════════════════════════════════════════════════╡
┊ DfltRscGrp    ┊ PlaceCount: 2           ┊        ┊             ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ iscsi_group   ┊ PlaceCount: 2           ┊ 0      ┊             ┊
┊               ┊ StoragePool(s): lvmpool ┊        ┊             ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ nvmeof_group  ┊ PlaceCount: 2           ┊ 0      ┊             ┊
┊               ┊ StoragePool(s): lvmpool ┊        ┊             ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ nfs_group     ┊ PlaceCount: 3           ┊ 0      ┊             ┊
┊               ┊ StoragePool(s): lvmpool ┊        ┊             ┊
╰────────────────────────────────────────────────────────────────╯
# linstor volume-group list iscsi_group
╭──────────────────╮
┊ VolumeNr ┊ Flags ┊
╞══════════════════╡
┊ 0        ┊       ┊
╰──────────────────╯
# linstor volume-group list nfs_group
╭──────────────────╮
┊ VolumeNr ┊ Flags ┊
╞══════════════════╡
┊ 0        ┊       ┊
╰──────────────────╯
# linstor volume-group list nvmeof_group
╭──────────────────╮
┊ VolumeNr ┊ Flags ┊
╞══════════════════╡
┊ 0        ┊       ┊
╰──────────────────╯

12.3. iSCSIターゲットの作成

準備が完了したら、iSCSI LUN の作成を始めることができます。 linstor-gateway コマンドラインユーティリティは iSCSI に関連する全ての動作を管理するのに使用されます。

サブコマンド iscsi に関する詳細な情報は linstor-gateway iscsi help を使ってください。

次のコマンドを実行すると、LINSTORクラスタに指定した名前とリソースグループで新しいDRBDリソースが作成されます。 このコマンドは、iSCSIターゲットの高可用性を可能にするためのDRBD Reactor設定ファイルも作成します。

# linstor-gateway iscsi create iqn.2019-08.com.linbit:example 192.168.122.181/24 1G \
                        --username=foo --password=bar --resource-group=iscsi_group

上記のコマンドを実行すると、提供されたユーザー名とパスワードを使用した CHAP 認証が有効な 1GiB の iSCSI ターゲットが作成されます。このターゲットは、コマンドで指定されたIPアドレスで検出可能です。ターゲットはLINSTORが管理するDRBDデバイスによってバックアップされます。DRBDリソースはLINSTORによって iscsi_group リソースグループ内に作成されます。上記のコマンドで作成されたDRBD Reactorの設定ファイルは /etc/drbd-reactor.d/ にあります。

LINSTOR Gateway が作成した iSCSI リソースは linstor-gateway iscsi list コマンドでリストアップすることができます。

# linstor-gateway iscsi list
+--------------------------------+--------------------+---------------+-----+---------------+
|              IQN               |     Service IP     | Service state | LUN | LINSTOR state |
+--------------------------------+--------------------+---------------+-----+---------------+
| iqn.2019-08.com.linbit:example | 192.168.122.181/24 | Started       |   1 | OK            |
+--------------------------------+--------------------+---------------+-----+---------------+
DRBD Reactorの状態は、drbd-reactorctl status コマンドで確認することができます。

12.4. iSCSIターゲットの削除

次のコマンドは、LINSTORクラスタと同様にDRBD ReactorからiSCSIターゲットを削除します。

# linstor-gateway delete -i iqn.2021-04.com.linbit:lun4 -l 4

12.5. NFS エクスポートの作成

NFSエクスポートを作成する前に、DRBDリソースがどのファイルシステムでフォーマットされるかをLINSTORに通知する必要があります。 これはNFSエクスポート用に作成したリソースグループに FileSystem/Type プロパティを設定することで行います。次のLINSTORコマンドを使用してください。

# linstor resource-group set-property nfs_group FileSystem/Type ext4
この設定は、リソースグループごとに一度だけ、LINSTOR GatewayのNFSエクスポート専用に作成されたリソースグループに対してのみ必要です。

最後に、次のコマンドはクラスタ内にHA NFSエクスポートを作成します。このコマンドは、LINSTORクラスタ内に指定された名前とリソースグループを使用して新しいリソースを作成します。このコマンドは、NFSエクスポートの高可用性を実現するためのDRBD Reactor設定ファイルも作成します。

# linstor-gateway nfs create nfstest 172.16.16.102/32 1G \
                  --allowed-ips=172.16.16.0/24 --resource-group=nfs_group

上記のコマンドを実行すると、1GiBのNFSエクスポートが、--service-ip として指定されたIPアドレスを使用してマウントできるようになります。エクスポートはLINSTORが管理するDRBDデバイスによってバックアップされます。 DRBDリソースはLINSTORによって nfs_group リソースグループ内に作成されます。上記のコマンドで作成されたDRBD Reactorの設定ファイルは /etc/drbd-reactor.d/ にあります。

LINSTOR Gatewayで作成されたNFSリソースを一覧するには、linstor-gateway nfs list コマンドを使用します。

# LINSTOR-gateway nfs list
+----------+------------------+---------------+------------------------------+---------------+
| Resource |    Service IP    | Service state |          NFS export          | LINSTOR state |
+----------+------------------+---------------+------------------------------+---------------+
| nfstest  | 172.16.16.102/32 | Started       | /srv/gateway-exports/nfstest | OK            |
+----------+------------------+---------------+------------------------------+---------------+
DRBD Reactorの状態は、drbd-reactorctl status コマンドで確認することができます。

12.6. NFS エクスポートの削除

次のコマンドは、LINSTORクラスタと同様にDRBD ReactorからNFSエクスポートを削除します。

# linstor-gateway nfs delete -r nfstest

12.7. Creating NVMe-oF Targets

The linstor-gateway command line utility will be used to manage all NVMe-oF target related actions.

Use linstor-gateway nvme help for detailed information regarding the nvme subcommand.

The following command will create a new DRBD resource in the LINSTOR cluster with the specified name and resource group. This command also creates the DRBD Reactor configuration files to enable high availability of the NVMe-oF target.

# linstor-gateway nvme create linbit:nvme:vol0 192.168.221.69/24 2G

After running the command above, you will have a highly available 2GiB NVMe-oF target created in your cluster that is discoverable on the IP address used in the command. The target will be backed by a DRBD device managed by LINSTOR. The DRBD resource was created by LINSTOR in the nvmeof_group resource group. The DRBD Reactor configuration files created by the above command can be found in /etc/drbd-reactor.d/.

You can list the NVMe-oF resources that you created by using LINSTOR Gateway by entering the linstor-gateway nvme list command:

# linstor-gateway nvme list
+------------------+-------------------+---------------+-----------+---------------+
|       NQN        |    Service IP     | Service state | Namespace | LINSTOR state |
+------------------+-------------------+---------------+-----------+---------------+
| linbit:nvme:vol0 | 192.168.221.69/24 | Started       |         1 | OK            |
+------------------+-------------------+---------------+-----------+---------------+
DRBD Reactorの状態は、drbd-reactorctl status コマンドで確認することができます。

12.8. Deleting NVMe-oF Targets

The following command will delete the NVMe-oF target from DRBD Reactor as well as the LINSTOR Cluster:

# linstor-gateway nvme delete linbit:nvme:vol0

13. LINSTOR Exosの統合

As of the release of linstor-server v1.26.0, the Exos integration for LINSTOR is deprecated.

The Exos storage manager from Seagate could be configured as one large block device managed by LINSTOR® such as a local drive, but this would prevent concurrent sharing of LINSTOR resources between multiple servers out of the same pool.

LINSTORとExosの統合により、複数のサーバーノードが同一のExosプールで提供されるLINSTORリソースを割り当て、接続することができます。 そのため、SSD/HDD階層化、SSDキャッシング、スナップショット、シンプロビジョニングなど、Exosのストレージ管理機能のすべてがLINSTORリソースやKubernetes Storage Classで利用可能になります。

設定後、LINSTORは2つのExosコントローラのうち1つを介してサーバーノードに提示されるLUNとしてResourceレプリカを動的にマッピングします。

Since the Exos controllers are managed by a secure network API, LINSTOR must be configured with proper networking and username and password combination. The diagram below is showing the relationship between LINSTOR cluster and Exos Enclosures.

Exos統合
マルチホストの設定では、最大8台のLINSTORノードを48Gbit SASリンクで直接接続し、低レイテンシーと高スループットを実現します。

ロードバランシングとサーバーのフェイルオーバーはLINSTORによって管理、有効化され、ボリューム作成はExosハードウェアRAIDエンジンによって処理されます。

LINSTORのExosストレージプロバイダは、Exos REST-APIとのネイティブな統合を提供しています。

このセクションでは、Exosの統合を有効にし、LINSTORがExosエンクロージャーでバックアップされたストレージを管理するための設定方法を説明します。

Exosストレージシステムには、あらゆるエンタープライズストレージのニーズに対応する豊富な構成オプションが用意されています。使いやすさを最大限に高めるために、このガイドは次のデフォルトおよび前提条件に基づいています。

  1. デュアルコントローラ:Exosシステムコントローラはアクティブ/アクティブであり、自動フェイルオーバー機能を備えています。完全にサポートするには、両方のコントローラのIPアドレスをLINSTORプロパティでも構成する必要があります。

  2. デュアルExosプール:プールAのデータがコントローラAを介してアクセスされる場合、最適なパフォーマンスが得られます。ノードが同じコントローラのコントローラAとコントローラBの両方に接続されている場合、LINSTORはLinuxマルチパスを構成し、最適なルートを検出します。

  3. Exos Pool Serial Numbers:Exosプールが作成されると、一意のシリアル番号が受信されます。各プールは、ExosエンクロージャとLINSTOR間のリンクを作成するために、LINSTORの下位ストレージとして構成する必要があります。この情報により、LINSTORはExosプールAまたはプールBを参照しているかどうかを把握できます。

  4. Creating Exos Pools – The administrator is required to create Exos Pools A and B before configuring LINSTOR. Exos features such as thin provisioning, auto tiering, and snapshot options are selected at this time.

  5. Replicas Within Enclosures – Exos system have redundant controllers, power supplies and communication paths to the drives. Some administrators might require that resource replicas are not stored in the same enclosure. In this case the administrator must create multiple LINSTOR Pools configured with only one Exos pool member from each enclosure.

13.1. LINSTORのストレージプロバイダとしてのExosプロパティ

LINSTORのExosとのネイティブな統合は、LINSTORコントローラにいくつかのプロパティを設定し、以下のセクションで説明するように、Exosエンクロージャに固有の適切なLINSTORオブジェクトを作成することによって構成されます。

次の表の情報は、Exosエンクロージャから取得する必要があります。この情報は、後続のサブセクションで適切なLINSTORコントローラのプロパティおよびLINSTORオブジェクトを移入するために使用されます。

Table 2. 必要な情報
Exos Information Description Placeholder in Command Examples

Exos Enclosure Name

管理者によって一意に選択された特定のExosエンクロージャー

<exos_encl_name>

Controller Hostname

コントローラーの1つのDNS解決可能なホスト名

<exos_ctrlr_name>

Controller IP

コントローラのIPアドレス

<exos_ctrlr_ip>

REST-API Username

指定されたエンクロージャーの下にあるすべてのExosコントローラーのREST-APIのユーザー名

<exos_rest_user>

REST-API Password

特定のエンクロージャーの下にあるすべてのExosコントローラーのREST-APIのパスワード

<exos_rest_pass>

Exos Pool Serial Number

LINSTORプールのメンバーになるExosプールのシリアル番号

<exos_pool_sn>

13.2. LINSTOR Exosの統合

LINSTORのサーバノードおよび複数のExosストレージシステムのトポロジーの構成手順は、次のとおりです。

  1. グローバルまたはユニークなExosコントローラのユーザ名とパスワードを設定します。

  2. ExosエンクロージャーおよびコントローラーネットワークIDを定義します。

  3. 物理SASケーブル接続に一致するノードとエンクロージャ間のプールマッピングを作成します。

13.2.1. ステップ1:Exosユーザ名とパスワード

ユーザー名とパスワードは、システム管理者がExosシステムをどのように展開したかに応じて、各Exosエンクロージャに対して一意にするか、またはすべてのエンクロージャに対して共通にすることができます。特定のExosコントローラに対して設定されていない場合は、デフォルトのExosユーザー名とパスワードが使用されます。デフォルトは次のように設定されています。

デフォルトは次のように設定されています。

# linstor exos set-defaults --username <exos_rest_name>
# linstor exos set-defaults --password <exos_rest_pass>

一意のユーザー名とパスワードExosコントローラの場合、次のように設定されます。

# linstor controller set-property
    StorDriver/Exos/<exos_encl_name>/username <exos_rest_name>
# linstor controller set-property
    StorDriver/Exos/<exos_encl_name>/Password <exos_rest_pass>
この方法で入力されたパスワードは、 get-defaults を使用するとプレーンテキストとして表示されます。

With the above command, LINSTOR will store your password in plain text in the LINSTOR properties and visible by a simple linstor controller list-properties command. You can hide it under an environment variable, and use the UsernameEnv or PasswordEnv (or both) properties. This tells LINSTOR to look in environment variable for the actual username and password, as shown in the following example:

LINSTORは環境変数を変更せず、読み取りのみを行います。ストレージ管理者は、env-varsが正しく設定されていることを確認する必要があります。
# echo $EXOS_PW
mySecretPassword
# linstor controller set-property \
    StorDriver/Exos/<exos_encl_name>/PasswordEnv EXOS_PW

両方のプロパティバージョン(つまり PasswordPasswordEnv )が設定されている場合は、環境以外のバージョンが優先されます。

環境変数が設定される前にサテライトが開始された場合、新しい環境変数を表示するにはサテライトを再起動する必要があります。

13.2.2. ExosエンクロージャーおよびコントローラーネットワークIDを定義します。

LINSTORでExosエンクロージャを登録するには、次のように create コマンドを使います。

# linstor exos create <exos_encl_name> <exos_ctrl_a_ip> [<exos_ctrl_b_ip>]

特別な --username または --password が指定されていない場合は、上記のデフォルトが使用されます。

The Controller’s DNS name and IP address can be used interchangeably.

If you want to use a hostname that is not DNS resolvable to reference your Exos enclosure within LINSTOR, you can use any name in place of <exos_hostname>, but you will also have to supply the enclosure’s IP address: linstor node create <desired_name> <enclosure_ip>

次の例を使用して、現在のコントローラ設定を作成および検査します。

|==================================================================|
# linstor exos create Alpha 172.16.16.12 172.16.16.13
# linstor exos list
+------------------------------------------------------------------+
| Enclosure | Ctrl A IP    | Ctrl B IP    | Health | Health Reason |
| Alpha     | 172.16.16.12 | 172.16.16.13 | OK     |               |
+------------------------------------------------------------------+

For a more in-depth view, you can always ask the LINSTOR controller or the LINSTOR nodes for the Exos-related properties:

# linstor controller list-properties | grep Exos
| StorDriver/Exos/Alpha/A/IP                | 172.16.16.12         |
| StorDriver/Exos/Alpha/B/IP                | 172.16.16.13         |

13.2.3. ステップ3: ノードにエンクロージャとプールのマッピングの作成

A LINSTOR Satellite node can be created as usual.

# linstor node create <satellite_hostname>

The storage pool can also be created as usual in LINSTOR. Only the name of the previously registered Exos enclosure and the serial number of the Exos pool needs to be specified:

# linstor storage-pool create exos \
  <satellite_hostname> <linstor_pool_name> <exos_encl_name> <exos_pool_sn>

the linstor_pool_name can be set to (almost) any unique string for the LINSTOR deployment.

Here is an example of mapping an Exos Pool in Exos enclosure Alpha to two Satellite nodes:

# linstor storage-pool create exos \
   node1 poolA Alpha 00c0ff29a5f5000095a2075d01000000
# linstor storage-pool create exos \
   node2 poolA Alpha 00c0ff29a5f5000095a2075d01000000

After creating an exos storage pool the LINSTOR Satellite will scan the given Exos enclosure for connected ports. If cabled, these ports will be listed in the following command:

# linstor exos map -p +----------------------------------------------+ | Node Name | Enclosure Name | Connected Ports |
|==============================================|
| node1     | Alpha          | A0, B0          |
| node2     | Alpha          | A1, B1          |
+----------------------------------------------+

プール構成は次のように表示します。

hr01u09:~ # linstor sp list -s poolA -p
+----------------------------------------------------------------------------------------------+
| StoragePool | Node  | Driver   | PoolName                               | FreeCapacity | ... |
|==============================================================================================|
| poolA       | node1 | EXOS     | Alpha_00c0ff29a5f5000095a2075d01000000 |      581 TiB | ... |
| poolA       | node2 | EXOS     | Alpha_00c0ff29a5f5000095a2075d01000000 |      581 TiB | ... |
+----------------------------------------------------------------------------------------------+

使用可能なすべてのExosコマンドの詳細な説明は、組み込みヘルプで参照できます。

# linstor exos -h

13.3. Exosストレージプールによってバックアップされるリソースの作成

Exos でバックアップされたストレージプールからの LINSTOR リソースの作成は LINSTOR resource groups またはより詳細な resource-definition, volume-definition, resource creation ワークフローを説明する各セクションなど、LINSTORユーザーガイドの他のセクションで説明されている通常の LINSTOR 使用パターンに従います。

14. LINSTOR Volumes in CloudStack

This chapter describes using LINSTOR to provision volumes that can be used to back primary storage in Apache CloudStack. CloudStack primary storage stores the virtual disks for virtual machines (VMs) running on hosts in CloudStack. A LINBIT-developed CloudStack plug-in integrates LINSTOR with CloudStack. A benefit to integrating LINSTOR with CloudStack is that it can be a way to provide highly available primary storage that is also flexible to manage.

Currently, the LINSTOR plug-in for CloudStack can only be used to provision volumes for use with KVM hypervisors.

Setting up and deploying CloudStack can be a complex task. A production-ready deployment can take several weeks to months before it is ready for users. A basic test deployment in a virtual environment can be set up in a few hours perhaps. This chapter will deal only with aspects related to integrating LINSTOR in CloudStack and should be considered a general overview. You should supplement instructions in this chapter with instructions and best practice recommendations from the CloudStack documentation.

Attention should be paid to security, firewall, and resource provisioning instructions in the CloudStack documentation, and in other chapters in the LINSTOR User’s Guide, before production deployment.
In this chapter, as in other areas of the LINSTOR User’s Guide, the word node is used. In most cases, you can think of a node as equivalent to a CloudStack host.

14.1. Introduction to CloudStack

From the CloudStack documentation: “Apache CloudStack is an open source Infrastructure-as-a-Service (IaaS) platform that manages and orchestrates pools of storage, network, and computer resources to build a public or private IaaS compute cloud.”

14.2. Preparing Your Environment For CloudStack and LINSTOR Deployment

You will need to make a few preparatory steps before deploying LINSTOR for use with CloudStack.

14.2.1. Configuring Time Synchronization

It is important that the nodes in your cluster are time synchronized. To do this, you can install and configure a tool such as Chrony or OpenNTPD.

14.2.2. Adding Node IP Addresses and Host Names

Add your cluster nodes’ IP addresses and host names to each node’s /etc/hosts file.

14.2.3. Setting a Root User Password and Permitting SSH Login

Set a root password by using the passwd root command. Then edit your SSH configuration file (/etc/ssh/sshd_config) to permit root login. The line in your file should look like this:

PermitRootLogin yes

14.2.4. Configuring SELinux

For CloudStack to work properly, SELinux must be set to permissive or disabled (not recommended in production).

14.3. Installing and Preparing LINSTOR for CloudStack

The LINSTOR plug-in is included in Apache CloudStack versions 4.16.1 and later. You do not have to install anything else to support LINSTOR. CloudStack versions 4.16.0 and earlier do not support the LINSTOR plug-in.

One of the pull requests did not merge properly in CloudStack v4.17.0 which caused a CloudStack UI bug in the CloudStack initialization wizard. More details are available here. If you need to be on the v4.17 branch, it is recommended (at time of writing) that you install v4.17.1.

Follow the installation instructions in the LINSTOR User’s Guide to install LINSTOR on the storage providing nodes in your cluster.

A basic outline of installation steps is:

  1. Install necessary packages for storage layers that you will be using, for example, ZFS or LVM. The steps below use ZFS as a backing storage layer for LINSTOR.

  2. Install the necessary LINSTOR packages (DRBD kernel module, linbit-sds-controller, and linbit-sds-satellite packages) from LINBIT repositories if you are a LINBIT customer, otherwise, you will need to build from source.

  3. Restart the multipathd daemon.

    systemctl restart multipathd
  4. Enable and start the LINSTOR Controller and LINSTOR Satellite services on your nodes.

    # systemctl enable --now linstor-controller
    # systemctl enable --now linstor-satellite
  5. Add your nodes to LINSTOR.

    linstor node create <node_host_name>
  6. Create a new LINSTOR storage pool on all of your participating nodes. For example, given a ZFS pool named zfs_storage, enter the following to create a storage pool named DfltStorPool:

    # linstor storage-pool create zfs <node_host_name> DfltStorPool zfs_storage
  7. Create a LINSTOR resource group to be used for CloudStack. To create a resource group named cloudstack, to be placed on two of your cluster nodes, enter:

    # linstor resource-group create cloudstack --place-count 2 --storage-pool DfltStorPool
  8. Create a LINSTOR volume group from your resource group, by entering the command:

    # linstor volume-group create cloudstack

14.3.1. Verifying Creating Resources

After installing LINSTOR and creating a resource group backed by a storage pool and storage layer, test that you can create storage resources. You can do this by spawning resources from the resource group that you created.

The CloudStack setup best practices recommend that a primary storage mount point (and therefore the LINSTOR resource that backs it) “should not exceed 6TB in size.”
# linstor resource-group spawn cloudstack testres 1GiB

Verify that LINSTOR created your resources by using a resource list command.

# linstor resource list
+----------------------------------------------------------------------------------+
| ResourceName | Node   | Port | Usage  | Conns |      State | CreatedOn           |
|-=================================================================================|
| testres      | node-0 | 7000 | Unused | Ok    |   UpToDate | 2022-11-10 20:12:30 |
| testres      | node-1 | 7000 | Unused | Ok    |   UpToDate | 2022-11-10 20:12:30 |
| testres      | node-2 | 7000 | Unused | Ok    | TieBreaker | 2022-11-10 20:12:29 |
+----------------------------------------------------------------------------------+

14.3.2. Cleaning Up After Verifying Resource Creation

After verifying resource creation, you can delete the resource, here testres, in your cluster by entering:

# linstor resource-definition delete testres

14.4. Installing CloudStack

After installing and preparing LINSTOR, you can install and configure CloudStack. As disclaimed previously, you should take these instructions as a way to setup CloudStack quickly for testing and illustrative purposes. Refer to CloudStack documentation for detailed instructions and best practice recommendations, before deploying into production.

14.4.1. Installing MySQL

First, install a MySQL server instance that is necessary for CloudStack’s database.

On Ubuntu, enter:

# apt install -y mysql-server

On RHEL, enter:

# dnf install -y mysql-server

14.4.2. Configuring the CloudStack Database

After installing the MySQL server package, create a CloudStack database configuration file named /etc/mysql/conf.d/cloudstack.cnf with the following contents:

[mysqld]
innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=350 (1)
log-bin=mysql-bin
binlog-format = 'ROW'
1 350 is the max_connections value specified in the CloudStack installation guide. You can change this value depending on your needs.

If you are on an Ubuntu 16.04 or later system, for binary logging, you need to specify a server_id in your .cnf database configuration file, for example:

[mysqld]
server_id = 1
innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=350
log-bin=mysql-bin
binlog-format = 'ROW'

Then restart the MySQL service by entering systemctl restart mysql.

14.4.3. Installing NFS for Secondary Storage

Next, install and configure NFS for CloudStack’s secondary storage. You only need to do this on the node that will be your CloudStack management node. CloudStack uses secondary storage to store such things as operating system images for VMs and snapshots of VM data.

To install NFS, on Ubuntu, enter:

# apt install -y nfs-kernel-server

On RHEL, enter:

# dnf install -y nfs-utils

After installing the NFS server, create an NFS export for CloudStack’s secondary storage by entering the following commands:

# mkdir -p /export/secondary
# echo "/export *(rw,async,no_root_squash,no_subtree_check)" >> /etc/exports
# exportfs -a

Next, enable and start the NFS server service.

# systemctl enable --now nfs-server

14.5. Installing and Configuring CloudStack

General CloudStack installation and configuration instructions follow. As your environment may have specific needs or variations, you should also reference the instructions in the CloudStack Installation Guide.

14.5.1. Installing CloudStack

While official CloudStack releases are “always in source code form,” for convenience, there are community generated DEB and RPM packages available at cloudstack.org:

You can follow the links above to find and download the packages that you need for your installation. Be sure to verify the integrity of downloaded packages against CloudStack’s signing keys, as outlined in the instructions here.

Alternatively, you can follow instructions here to configure the CloudStack repository appropriate to your Linux distribution and then pull and install packages by using your distribution’s package manager.

After adding the CloudStack repository, you may need to update the package manager’s repository list, before you can install packages.

For your CloudStack management node, install these packages:

  • cloudstack-management

  • cloudstack-common

  • cloudstack-ui

For your other cluster nodes that will be hosting VMs, install the cloudstack-agent package.

14.5.2. Initializing the CloudStack Database

After installing the necessary CloudStack packages, initialize the CloudStack database.

For testing purposes, you can enter the following command on your management node:

# cloudstack-setup-databases cloud:cloud --deploy-as=root:nonsense -i <node_name>

Here, the cloud after cloud: and nonsense are passwords that you can change as you see fit.

For production deployments, follow the more detailed instructions in the CloudStack Installation Guide.

14.6. Installing the CloudStack System Virtual Machine Image Template

CloudStack needs to run some system VMs for some of its functionality. You can download a CloudStack VM template image and then run a CloudStack script that will prepare the image for various system VMs in deployment. On the CloudStack management node, enter the following commands:

# CS_VERSION=4.17
# CS_VERSION_PATCH=4.17.1
# wget https://download.cloudstack.org/systemvm/$CS_VERSION/systemvmtemplate-$CS_VERSION_PATCH-kvm.qcow2.bz2
# /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt \
-m /export/secondary \
-f systemvmtemplate-$CS_VERSION_PATCH=-kvm.qcow2.bz2 \
-h kvm -o localhost -r cloud -d cloud

14.7. Configuring KVM Hypervisor Hosts for Use in CloudStack

Currently, the LINSTOR CloudStack plug-in only supports KVM hypervisor hosts. The instructions that follow are for configuring your CloudStack installation with KVM hypervisor hosts.

Enter the following command to add libvirt configurations to every node in your cluster that will host CloudStack VMs:

# cat << EOF >> /etc/libvirt/libvirtd.conf
listen_tls = 0
listen_tcp = 1
tcp_port = "16509"
auth_tcp = "none" # not suitable for production
mdns_adv = 0
EOF

Restart the libvirtd service on all hypervisor nodes.

# systemctl restart libvirtd

14.7.1. Configuring AppArmor

If you are running CloudStack on Ubuntu Linux and if AppArmor is enabled, enter the following:

# ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/
# ln -s /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper /etc/apparmor.d/disable/
# apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd
# apparmor_parser -R /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper

14.7.2. Restarting the CloudStack Management Service

After making the necessary setup and preparatory configurations, restart the cloudstack-management service.

# systemctl restart cloudstack-management

You can follow the progress of CloudStack’s initial database setup by entering:

# journalctl -u cloudstack-management -f

14.7.3. Logging into the CloudStack UI

After some time, you should be able to log into the CloudStack management UI. Given a management node resolvable host name of node-0, enter the following URL into a web browser on a computer in your cluster’s network: http://node-0:8080/client.

Once you are greeted by the CloudStack UI’s portal login page, log into the portal by using the default user name admin and the default password password.

After successfully logging in, the CloudStack UI will display the “Hello and Welcome to CloudStack” page.

14.7.4. Running the CloudStack Initialization Wizard

You can continue to set up CloudStack by launching an initialization wizard. Click on the “Continue with installation” button to launch the wizard.

The wizard will first prompt you to change the default password for the administrator user. After changing the password, you can continue through the wizard steps to configure a zone, network, and resources details. Complete the fields in each setup step according to your environment and needs. More details about initializing CloudStack can be found here.

The following fields will be common to all LINSTOR use cases in CloudStack:

  • Zone details:

    • Hypervisor: KVM

  • Add resources, IP Address step:

    • Host Name: <host_name_of_cluster_node_that_will_host_VMs>

    • Username: root

    • Password: <root_password_that_you_configured_previously_for_the_host>

  • Add resources, Primary Storage step:

    • Protocol: Linstor

    • Server: <IP_address_of_LINSTOR_controller_node>

    • Resource Group: <LINSTOR_resource_group_name_that_you_configured_previously>

Based on configuring an NFS export for secondary storage earlier, complete the fields presented during the “Add resources, Secondary Storage” step as follows:

  • Provider: NFS

  • IP Address: <IP_address_of_NFS_server> # should be the CloudStack management node

  • Path: <NFS_mount_point> # /export/secondary, as configured previously

After completing entry fields in the “Add resources” fields and clicking the “Next” button, the wizard will display a message indicating the “Zone is ready to launch.” Click on the “Launch Zone” button.

The “Adding Host” step of the “Launch Zone” process may take a while.

After the zone is added, the wizard will show a “Zone creation complete” message. You can then click on the “Enable Zone” button. After another “Success” notification you will be returned to the CloudStack UI dashboard.

14.7.5. Verifying Primary Storage in CloudStack

Click on the “Infrastructure” icon on the left of the screen, then click on “Primary Storage”. The “Primary Storage” screen should show your LINSTOR-backed primary storage pool in an “Up” state.

14.7.6. Verifying Secondary Storage in CloudStack

Click on the “Infrastructure” icon on the left of the screen, then click on the “Secondary Storage” link. The “Secondary Storage” screen should show your NFS-backed secondary storage in a “ReadWrite” access state.

14.8. Taking Next Steps in CloudStack

After configuring LINSTOR for use in CloudStack you can move onto other tasks, such as adding hosts to host your CloudStack VMs.

LINBIT has also made available a video demonstrating deploying LINSTOR and CloudStack into a three-node VM cluster. You can view the video here.

15. LINSTOR Volumes in Oracle Linux Virtualization Manager

This chapter describes using LINSTOR® to provision persistent, replicated, and high-performance block storage for Oracle Linux Virtualization Manager.

15.1. Introduction to Oracle Linux Virtualization Manager

Oracle Linux Virtualization Manager (OLVM) is a server virtualization management platform based on oVirt. It can configure, manage and monitor a Kernel-based Virtual Machine (KVM) environment based on Oracle Linux.

OLVM supports multiple storage technologies that can be integrated with LINSTOR:

  • iSCSI or NFS storage are used for shared storage. In this setup, all OLVM nodes connect to a service host that exports the storage volume, by using iSCSI or NFS. Here, LINSTOR Gateway can help you create and manage iSCSI targets and NFS exports based on replicated DRBD volumes and make them highly available.

  • Managed Block Storage is another feature available in OLVM. Disks for VMs are managed as separate volumes, rather than from within a pool of storage on a service host. This makes it possible to directly attach a volume to a VM while avoiding the performance overhead that a service host adds.

15.2. Using LINSTOR Gateway for OLVM Storage

LINSTOR Gateway creates highly available iSCSI targets and NFS exports, which OLVM can consume as data domains. All volume access is routed through the LINSTOR Gateway host exporting the volume. Volume access includes reads and writes by any VM using the storage domain.

As prerequisites, you need:

LINSTOR and LINSTOR Gateway don’t need to run on the same nodes as OLVM, provided that the Gateway nodes are reachable from the OLVM hosts.

15.2.1. Using LINSTOR Gateway for Data Domains

Data Domains are the primary storage for your VMs in OLVM. With LINSTOR Gateway, you can create both iSCSI targets and NFS exports, to use as Data Domains in OLVM.

Use the linstor-gateway command to create iSCSI or NFS exports. Choose a service IP that is reachable from the OLVM nodes. The following example creates an iSCSI export of 60GB reachable under 192.168.0.100 and a NFS export of 50GB, available at 192.168.0.101:/srv/gateway-exports/nfs-data

$ linstor-gateway iscsi create iqn.2019-08.com.linbit:data-domain 192.168.0.100/24 60G
Created iSCSI target 'iqn.2019-08.com.linbit:data-domain'
$ linstor-gateway nfs create nfs-data 192.168.0.101/24 50G
Created export 'nfs-data' at 192.168.0.101:/srv/gateway-exports/nfs-data

To configure the storage domains, navigate to the OLVM Administration Portal and open the Storage > Domains page. Click New Domain and choose a name for the new storage domain. Then, select the Domain function (either Data or ISO) and the matching storage type (either iSCSI or NFS). Enter the required connection parameters to complete the configuration.

15.2.2. Using LINSTOR Gateway to Deploy the OLVM Self-Hosted Engine

LINSTOR Gateway iSCSI Targets can be used to create the initial data domain used when deploying the OLVM self-hosted engine.

Use a separate data domain for the self-hosted engine to reduce the risk of adverse interference with your VMs. Create a iSCSI target with at least 60GB to use for the management VM. The following example creates a 60GB volume exported as iqn.2019-08.com.linbit:olvm-engine, available at the IP 192.168.0.200. Change the IP and iSCSI target name to appropriate values for your setup.

$ linstor-gateway iscsi create iqn.2019-08.com.linbit:engine-data 192.168.0.200/24 60G

While setting up the OLVM self-hosted engine, you will be asked to provide details for the storage of the Manager virtual machine. You only need to provide the storage type iscsi and the IP address 192.168.0.200. All other information will be discovered automatically.

Please specify the storage you would like to use (glusterfs, iscsi, fc,
nfs)[nfs]: iscsi
Please specify the iSCSI portal IP address: 192.168.0.200
  Please specify the iSCSI portal port [3260]:
  Please specify the iSCSI discover user:
  Please specify the iSCSI discover password:
  Please specify the iSCSI portal login user:
  Please specify the iSCSI portal login password:
  The following targets have been found:
    [1] iqn.2019-08.com.linbit:engine-data
        TPGT: 1, portals:
            192.168.0.200:3260
  Please select a target (1) [1]: 1
  Please select the destination LUN (1) [1]:

After the setup completes, the iSCSI target is added as a data domain to OLVM. Use separate data domains for your VMs to avoid interference with the self-hosted engine’s storage.

15.3. Using LINSTOR Cinder for Managed Block Storage

OLVM can be configured to use the LINSTOR Cinder driver to attach LINSTOR managed volumes directly on the VM host. In contrast to using LINSTOR Gateway, storage access and replication happens directly from the host the VM is running on.

Using LINSTOR Cinder requires several additional steps to set up for OLVM:

  • All OLVM hosts, including the engine host, need to be registered with LINBIT and the ovirt repository enabled:

    # curl -O https://my.linbit.com/linbit-manage-node.py
    # chmod +x ./linbit-manage-node.py
    # ./linbit-manage-node.py
    ...
        1) pacemaker-2(Disabled)
        2) ovirt(Enabled)
        3) drbd-9.0(Disabled)
        4) drbd-9(Enabled)
    ...
  • OLVM hosts as well as the management engine need to be part of the LINSTOR cluster. Ensure that the linstor-satellite service is installed and configured on all hosts.

  • OLVM needs to have CinderLib support enabled. It is disabled by default. To enable it, run engine-setup on the engine host:

    $ engine-setup --reconfigure-optional-components
    ...
    --== PRODUCT OPTIONS ==--
    
    Configure Cinderlib integration (Currently in tech preview) (Yes, No) [No]: Yes
    ...
  • The engine hosts need to have the LINBIT version of python3-cinder-common installed, which can be identified by the linbit1 string in the version number:

    $ dnf install --enablerepo=ovirt python3-cinder-common
    $ dnf list --installed python3-cinder-common
    Installed Packages
    python3-cinder-common.noarch   1:17.2.0-1.el8.linbit1   @linbit-ovirt
  • The engine hosts need to have the linstor-cinder package installed:

    $ dnf install --enablerepo=ovirt linstor-cinder
  • All OLVM hosts need to have the LINBIT version of vdsm and python3-osbrick installed, which can be identified by the linbit1 string in the version number:

    $ dnf install --enablerepo=ovirt vdsm python3-os-brick
    $ dnf list --installed vdsm python3-os-brick
    Installed Packages
    python3-os-brick.noarch   4.0.4-1.el8.linbit1             @linbit-ovirt
    vdsm.x86_64               4.40.100.2-1.0.13.el8.linbit1   @linbit-ovirt

To configure LINSTOR Cinder for Managed Block Storage, navigate to the OLVM Administration Portal and open the Storage > Domains page. Click New Domain and select Domain Function “Managed Block Storage”. Choose a name, and set the following driver options:

Driver Option

Value

volume_driver

linstor_cinder.LinstorDrbdDriver

linstor_uris

URL of the LINSTOR Controller Endpoint(s). Separate multiple endpoints by using a comma (,).

linstor_default_resource_group_name

LINSTOR resource group to use. Volumes created in this data domain will inherit all settings on the resource group.

OLVM 4.4 does not support creating VMs using Managed Block Storage from the VM Portal, only through the Administration Portal.

1. Property changes made on a resource group will be inherited by its resource definition children and you will see the changes reflected in the corresponding .res resource files. However, property changes made on the parent are not copied to the child objects (either the resource definition or resource LINSTOR objects), that is, the child objects do not carry the property themselves. The change affects the object children, but the property itself remains on the parent.
2. Physical storage might not exist on a particular node, for example, a diskless node. Here, the “physical storage” layer is imaginary, for the purposes of conceptualizing object hierarchy in LINSTOR.
3. Also, because LINSTOR is evolving software, for some special use cases and contexts, it might be the case that certain object property hierarchies are different from the general rules. When these cases occur in documentation, there will be mention of any exceptions to the general rules.
5. ホストがストレージノードでもある場合は、イメージのローカルコピーを使用できます