settings-content-linux-control-linstor

The Technology Inside LINSTOR (Part II)

In our first look into LINSTOR you learned a lot about the single communication protocol, transaction-safety and modularity features. In the next chapter you can dive deeper into the construction.

Fault Tolerance

Keeping the software responsive is one of the more difficult problems that we have to deal with in LINSTOR’s design and implementation. The Controller/Satellite split is one fundamental part of LINSTOR’s design toward fault tolerance, but there are many other design and implementation details that improve the software’s robustness, and many of them are virtually invisible to the user.

On the Controller side, communication and persistence are the two main areas that can lead to the software becoming unresponsive. The following problems could lead to an unusable network communication service on the Controller side:

  • Stopping or reconfiguring a network interface
  • Address conflicts
  • In-use TCP/IP ports

All network I/O in LINSTOR is non-blocking, so that unresponsive network peers do not lead to a lockup of LINSTOR’s network communication service. While the network communication service has been designed to recover from many kinds of problems, it additionally allows the use of multiple independent network connectors, so that the system remains accessible even in the case where a network connector requires reconfiguration to recover. The network connectors can also stop and start independently, allowing reinitialization of failed connectors.

The Controller can obviously not continue normal operation while the database service is inoperative, which could of course happen if an external database is used, for example, due to a downtime of the database server or due to a network problem. Once the database service becomes available again, the Controller will recover automatically, without requiring any operator intervention.

Satellites in LINSTOR

The Satellite side of LINSTOR does not run a database, and a single unresponsive Satellite is less critical for the system as a whole than an unresponsive Controller. Nonetheless, if a Satellite satellite-linstorencounters a failure during the configuration of one storage resource, that should still not temporarily prevent it from being able to service requests for the configuration of other resources.

The biggest challenge regarding fault tolerance on the Satellite side is the fact that the Satellite interacts with lots of external programs and processes that are neither part of LINSTOR nor under the direct control of the Satellite process. These external components include system utilities required for the configuration of backend storage, such as LVM or ZFS commands, processes observing events generated by the DRBD kernel module whenever the state of a resource changes, block device files that appear or disappear when storage devices are reconfigured, and similar kinds of objects.

To achieve fault tolerance on the Satellite side, the software has been designed to deal with many possible kinds of malfunctions of the external environment that LINSTOR interacts with. This includes the time-boxing and the enforcement of size limits on the amount of data that is read back when executing external processes, as well as recovery procedures that attempt to abort external processes that have become unresponsive. There is even a fallback that reports a malfunctioning operating system kernel if the operating system is unable to end an unresponsive process. The LINSTOR code also contains a mechanism that can run critical operations, such as the attempt to open a device file ( which may block forever due to faulty operating system drivers) asynchronously, so that even if the operation blocks, LINSTOR would normally at least be able to detect and report the problem.

Usability

With feature richness, customizability and flexibility, also comes complexity. The only thing that can be done to make the system as easy to understand and use as possible is to attempt to make the system intuitive, self-explaining and unambiguous.

Clarity in the naming scheme of objects turned out to be an important factor for a user’s ability to use the software intuitively. In our previous product, drbdmanage, users would typically look for commands to either create a “resource” or a “volume.” However, the corresponding commands, “new-resource” and “new-volume”, only define a resource and its volumes, but do not actually create storage resources on any of the cluster nodes. Another command, “assign”, was required to assign the resource to cluster nodes, thereby creating the actual storage resource, and users sometimes had a hard time finding this command.

For this reason, the naming of objects was changed in LINSTOR. A user looking for a command to create a resource will find the command that actually creates a storage resource, and one of the required parameters for this command is the so-called resource definition. It is quite obvious that the next step would be to look for a command that creates a resource definition. This kind of naming convention is supposed to make it easier for users to figure out how to intuitively use the application.

LINSTOR is also explicit with replies to user commands, as well as with return codes for API calls. The software typically replies with a message that describes whether or not the command was successful, what the software did, and to which objects the message refers. Error messages that include a description of the problem cause or hints for possible correction measures also follow a uniform structure.

Similar ideas also applies to return codes, which include not only the error code (e.g., Object exists), but also information on what objects the error refers to (e.g., the type of object and the identifier specified by the user).

Reporting System

To make diagnosing errors easier, LINSTOR also generates a unique identifier for every error that is logged. The traditional logging and error reporting on Unix/Linux systems basically consists of single text lines logged to one large logfile, sometimes even a single logfile for many different applications. An application could log multiple lines for each error, but support for logging multiple lines atomically (instead of interleaved with log lines for other errors, possibly from other applications) is virtually nonexistent.

For this reason, LINSTOR logs a single-line short description of the error, including the error identifier, to the system log, but also logs the details of the error to a report file that can be found using the error identifier. The detailed log report also contains information such as the component where the error occured, the exact version of the software that was used, debug information, nested errors, and many other details that may help with problem mitigation.

Implementation Quality

While the various design characteristics are important factors for creating a powerful and robust software system, even the best design cannot produce a reliable application if it is not implemented with high quality.

The first step, even before we wrote the code, was to choose a programming language that would be suitable for the task. While our previous product, drbdmanage, and the current LINSTOR client are implemented in Python, the LINSTOR server-side components (the Controller and Satellite) are implemented in Java. A server application that manages highly available storage systems should obviously be designed and implemented much more carefully than the typical single-user desktop application. Java is a very strict programming language that provides strong static typing, checked exceptions and allows only few implicit type conversions – which are all features that also enable IDEs to perform static checking of the code while it is being written.

Obviously, while it can make writing high quality code easier, the choice of programming language alone does not automatically lead to better code. To keep LINSTOR’s code clean, readable, self-explaining and maintainable, we apply many of the best practices that have proven successful in the creation of mission-critical software systems. This includes more important things like choosing descriptive variable names or maintaining a clear and logical control flow, but even extends to less technical details like consistent formatting of the source code. The coding standard that we apply to produce high-quality code is based on standards from the aviation industry and is among the strictest coding standards that exist today.

checking-checklist-linstorEasy Validity Checks

There is also a strong focus on correctness and strict checking in the way LINSTOR is implemented. As an example, the name of objects like nodes, resources or storage pools is not simply a String, but an object that can only be constructed with a name that is valid for that kind of object. It is impossible to create a resource name object that contains invalid characters, or to accidentally use a resource name object as the identifier for the creation of a storage pool. As a result, developers cannot forget to perform a validity check on a node name or on a volume number, and they also cannot apply the wrong check by accident.

All those considerations, design characteristics and implementation methods are important factors that helped us create a dependable and user friendly software that we hope will prove useful and valuable to its users like you.

 

If you have any questions or suggestions concerning LINSTOR, please leave a comment or write email to [email protected] .

control room linstor management

The technology inside LINSTOR (Part I)

Spotlight on LINSTOR’s design and technology: What we do and how we do it to create a powerful, flexible and robust storage cluster management software

LINSTOR is an application that is typically integrated with highly automated systems, such as software defined storage systems or virtualization environments. Users often interact with the management interface of some other application that uses LINSTOR to manage the storage required for that application’s use case, which also means that the users may not have direct access to the storage systems or to the LINSTOR user interface.

A single storage cluster can be the backend of multiple independent application systems, so the biggest challenge for a software like LINSTOR is to remain responsive even if some actions or components of the cluster fail. At the same time, the software should be flexible enough to cover all use cases, to enable future extension or modification, and despite all the complexity that is the result of these requirements, it should at the same time be easy to understand and easy to use for the administrators who are tasked with installing and maintaining the storage system.

It is quite clear to anyone who has worked on a bigger software project as a developer that many of those requirements work against each other. Customizability, flexibility, an abundance of features cause complexity, but complexity is the natural enemy of usability, reliability and maintainability. When we started the development of LINSTOR, our challenge was to design and implement the software so that it would achieve our goals with regards to feature richness and flexibility while at the same time remaining reliable and easy to use.

Modularity

One of the most important aspects of LINSTOR’s design is its modularity. We divided the system into two components, the Controller and the Satellite, so that the Controller component could remain as independent as possible from the Satellite component – and vice versa.

Even inside those two components, many parts of the software are exchangeable – the communication layer, the serialization protocol, the database layer, all of its API calls, even all of the debug commands that we use for internal development, as well as many other implementation details are exchangeable parts of the software. This provides not only a maximum of flexibility for future extensions, it also acts as a sort of safety net. For example, if support for the database or the serialization protocol that we use currently were dropped by their maintainers, we could simply exchange those parts without having to modify every single source code file of the project, because implementation details are hidden behind generic interfaces that connect various parts of our software.

Another positive side effect is that many of those components, being modular, are naturally able to run multiple differently configured instances. For example, it is possible to configure multiple network connectors in LINSTOR, each bound to different network interfaces or ports.

Linstor Linbit Opennebula openstack

A single communication protocol

As a cluster software, LINSTOR must of course have some mechanism to communicate with all of the nodes that are part of the cluster. Integration with other applications also requires some means of communication between those applications and the LINSTOR processes, and the same applies to any kind of user interface for LINSTOR.

There are lots of different technologies available, but many of them are only suitable for certain kinds of communication. Some clusters use distributed key/value stores like etcd for managing their configuration, but use D-Bus for command line utilities and a REST interface for connecting other applications.

Instead of using many different technologies, LINSTOR uses a single versatile network protocol for communication with all peers. The protocol used for communication between the Controller and the Satellites is the same as the one used for communication between the Controller and the command line interface or any other application. Since this protocol is implemented on top of standard TCP/IP connections, it also made all aspects of LINSTOR’s communication network transparent. An optional SSL layer can provide secure encrypted communication. Using a single mechanism for communication also means less complexity, as the same code can be used for implementing different communication channels.

Transaction-safety

Even though LINSTOR keeps its configuration objects in memory, there is an obvious need for some kind of persistence. Ideally, what is kept in memory should match what is persisted, which means that any change should be a transaction, both in memory and on persistent storage.

Most Unix/Linux applications have traditionally favored line-based text files for the configuration of the software and for persisting its state, whereas LINSTOR keeps its configuration in a database. Apart from the fact that a fully ACID-compliant database is an ideal foundation for a building a transaction-safe application, using a database also has other advantages. For example, if an upgrade of the software requires changes to the persistent data structures, the upgrade of the data can be performed as a single transaction, so that the result is either the old version or the new version of the data, but not some broken state in between. Database constraints also provide an additional safeguard that helps ensuring the consistency of the data. Assuming that there were a bug in our software, so that it would fail to detect duplicate volume numbers being assigned to storage volumes, the database would abort the transaction for creating the volume due to constraint violations, thereby preventing inconsistencies in the corresponding data structures.

To avoid requiring users to set up and maintain a database server, LINSTOR uses its own integrated database by default – it is simply started as an integral part of the Controller component. Optionally, the Controller can also access a centralized database by means of a JDBC driver.

Read more in the second blog post! 

Find LINSTOR in our Github repository

container native storage

Container-Native Software Defined Storage (SDS) from LINBIT

Container-native technologies represent the next wave of efficiency for data centers. Just like previous waves, a new technology points to new levels of efficiency and delivers some of it, but as you get deeper into it, you see opportunities to optimize how it works. And after a while longer, you see it. You see how the whole thing could have been done in a “native” way to get the most out of it. This is pretty typical. Technology often hits its mark only after the first few revisions. When it comes to containers, we are now getting to that stage across the industry. “Container-native xyz” is now possible and can bring maturity and speed.

At LINBIT, we have seen the needs of the DRBD community evolve over time. Starting with data replication in clusters, customers welcomed an open source alternative to proprietary paths towards high availability (HA): always-on applications and always-available data. That model was later expanded to provide geo-clustering: geographically distributed resources that provide disaster recovery (DR).

Market feedback: Need for flexible storage management

HA and DR are environments that, by necessity, pay special attention to data and storage. They provide a front-row seat to how data management and storage management technologies are changing. We saw early on that LINBIT users have growing needs for flexible provisioning, configuration management, and cluster management, without sacrificing data availability and application up-time. LINBIT’s entry into the SDS world was natural and motivated by addressing that need: flexible storage management with built-in HA and DR.

LINBIT SDS provisions, replicates, and manages data storage independent of hardware, enabling the use of commodity hardware in addition to open source software, an important objective for many data centers. Today, LINBIT SDS is a successful player in the SDS market. Additionally, LINBIT is an enthusiastic member of the open source OpenSDS Project, making it easier and more cost-effective for the millions of DRBD users to implement SDS solutions.

Putting all of that in the context of container technologies, it became clear a couple of years ago that a container-native implementation of LINBIT SDS would be an emerging need. That led to the announcement we made recently focused on a major addition to the LINBIT product suite: LINSTOR, which adds support for Kubernetes and OpenShift environments.

Reduced costs for cluster storage

LINSTOR fills a significant gap in the market by providing container-native block storage, which as you may know, is especially important for enterprise/database applications. LINSTOR can also provide data persistence for elastic applications, which dynamically create or remove containers depending on the load.

With LINSTOR, enterprise and database applications can benefit from container technologies while preserving data even when a container is eliminated. This can reduce the cost and complexity of deploying scale-out cluster storage.

Just as virtualization changed how storage can be configured and managed, container technologies demand a new approach. LINBIT SDS and LINSTOR provide that.

By simplifying storage cluster configuration and ongoing management, then plugging into cloud and container front-ends, users get the resilient infrastructure they need while retaining flexibility to choose vendors. For example, using a pool of Linux storage, system administrators can set the number of nodes, the number and size of storage volumes, and number of replicas. LINSTOR does the rest and builds the environment, eliminating the complex and error-prone process of manually setting up cluster configuration files for each individual server.

As always, we’d love to hear from you if you are using or are interested in using LINSTOR and LINBIT SDS. We are proud that LINBIT users scale from small shops to  large enterprises; to universities and government agencies . We’re committed to delivering always-on/always-available/always-efficient technologies.

Cluster-wide management of replicated storage with LINSTOR

The new generation of LINBIT’s storage management system focuses on ease-of-use, resilience and scalability.

Todays IT installations often consist of many individual servers, each running some part of the software infrastructure that together form the kind of service that the installation is supposed to provide. Software processes rely on data, and high availability or disaster recovery solutions, which have typically included replication of the data to one or multiple other physically independent systems.

LINSTOR is the new generation of the software component that implements the automatic management of replicated storage resources in the LINBIT SDS system. Besides adding new features that users have been previously requested, such as the ability to make use of multiple-tier storage, LINBIT has also improved the existing features.

Linstor Linbit Opennebula openstack

Linstor features

Ease-of-use

Our experience has shown that administrators of complex IT environments typically struggle with two things: figuring out how to make the system do what they want, and determining the cause of the problem when a system fails to do what the administrators expect. Creating a new product has given us the opportunity to consider these issues during the design phase and to focus on making the new software easier to use and troubleshoot. Two examples of related enhancements are the more consistent and logical naming of LINSTOR objects and commands, and the greatly enhanced logging and problem reporting.

Resilience

Another area of improvement that we focused on is the resilience of the system as a whole, which depends not only on the LINSTOR software, but also on the entire external environment. For this reason, we designed LINSTOR to manage unexpected changes and to recover from many different types of failures of external components.

Scalability

LINSTOR greatly increases scalability by its ability to perform changes on multiple resources and on multiple nodes concurrently, while still remaining responsive to new requests.

Multi tier storage

Many users have requested the support of multiple-tier storage, and we are pleased to announce that, by adding the concept of storage pools, it has been implemented in LINSTOR. We made this a flexible feature, so that multiple storage pools can be configured, even using different storage backend drivers per storage pool and/or per node if necessary.

The new software is also capable of dealing with multiple network interface cards, each of which can be used as the replication link for DRBD resources or as the communication link for LINSTOR. This feature enables splitting the control network (providing LINSTOR communication) from the data network (providing DRBD replication link communication). IPv6 is  supported for both, LINSTOR communication and DRBD replication links.

RoadmapProduction release roadmap

The roadmap for the production release includes support for:

  • taking snapshots of replicated resources
  • thinly provisioned LVM storage
  • ZFS storage
  • encrypted and authenticated network communication within LINSTOR
  • taking advantage of LINSTOR’s multi-user-capability
Robert Altnoeder on Linkedin
Robert Altnoeder
Robert joined the LINBIT development team in 2013. He had worked with
DRBD at a startup company in the SaaS field before joining LINBIT. His
current primary field of work is the architecture and implementation of
LINSTOR, the cluster management component of LINBIT's SDS software.

LINBIT Introduces Container-Native Storage for Enterprise Applications

New Open-Source Software-Defined Storage Available for Kubernetes and OpenShift Environments.

“LINSTOR is designed to be container-native while consolidating storage and data management which saves time and money for IT departments,” said Brian Hellman, COO of LINBIT.

LINBIT, the de facto standard in open source High Availability (HA), Disaster Recovery (DR), Software-Defined Storage (SDS), and the force behind the renowned DRBD® open source software, today announced the public beta release of LINSTOR, a major new addition to its product portfolio. Serving the rapidly growing containerized applications market, LINSTOR fills a significant gap in the market by providing container-native block storage, a common data access model used by enterprise applications. LINSTOR can also provide data persistence for elastic applications, which dynamically create or remove containers depending on the load.

With LINSTOR, enterprise and database applications can benefit from container technologies while preserving data even when a container is eliminated; data centers can reduce cost and complexity when deploying scale-out cluster storage.

LINSTOR is part of LINBIT SDS, the industry’s fastest software defined storage solution for enterprise, cloud, and container environments. LINBIT SDS provisions, replicates, and manages data storage independent of hardware, thus enabling the use of commodity hardware and open source software, an important objective for many data centers. LINSTOR adds support for the popular Kubernetes and OpenShift environments.

LINBIT Introduces LINSTOR

“LINSTOR is designed to be container-native while consolidating storage and data management which saves time and money for IT departments,” said Brian Hellman, COO of LINBIT.

“Just as virtualization changed how storage must be configured and managed, container technologies demand a new approach,” said Brian Hellman, COO of LINBIT.

LINSTOR takes advantage of DRBD, a part of the Linux kernel for nearly a decade, to deliver fast and reliable data replication. By simplifying storage cluster configuration and ongoing management, then plugging into cloud and container front-ends, users get the resilient infrastructure they need while retaining flexibility to choose vendors. For example, using a pool of Linux storage, system administrators simply set the number of nodes, the number and size of storage volumes, and number of replicas, and LINSTOR identifies the servers with the proper space to build the environment.

“LINSTOR drastically simplifies storage provisioning and data replication without compromising speed and reliability,” said Philipp Reisner, CEO of LINBIT. “LINSTOR eliminates the long and complicated process of manually setting up cluster configuration files for each individual server.”

LINBIT will showcase its product portfolio with a special emphasis on container-native storage at the upcoming Red Hat Summit in San Francisco, California on May 8-10, 2018, Booth 432.

About LINBIT
LINBIT is the force behind DRBD and the de facto open standard for High Availability (HA) software for enterprise and cloud computing. The LINBIT DRBD software is deployed in millions of mission-critical environments worldwide to provide High Availability (HA), Geo Clustering for Disaster Recovery (DR), and Software Defined Storage (SDS) for OpenStack and OpenNebula based clouds.

Sebastian Schinhammer
Sebastian brings his experience in marketing to the table. As Marketing Manager at LINBIT he cares especially about content marketing, brand appearance and public relations. He fills brands with life telling engaging stories. Sebastian strives for his marketing efforts being helpful information or great entertainment rather than all bark but no bite.