Spotlight on LINSTOR’s design and technology: What we do and how we do it to create a powerful, flexible and robust storage cluster management software
LINSTOR is an application that is typically integrated with highly automated systems, such as software defined storage systems or virtualization environments. Users often interact with the management interface of some other application that uses LINSTOR to manage the storage required for that application’s use case, which also means that the users may not have direct access to the storage systems or to the LINSTOR user interface.
A single storage cluster can be the backend of multiple independent application systems, so the biggest challenge for a software like LINSTOR is to remain responsive even if some actions or components of the cluster fail. At the same time, the software should be flexible enough to cover all use cases, to enable future extension or modification, and despite all the complexity that is the result of these requirements, it should at the same time be easy to understand and easy to use for the administrators who are tasked with installing and maintaining the storage system.
It is quite clear to anyone who has worked on a bigger software project as a developer that many of those requirements work against each other. Customizability, flexibility, an abundance of features cause complexity, but complexity is the natural enemy of usability, reliability and maintainability. When we started the development of LINSTOR, our challenge was to design and implement the software so that it would achieve our goals with regards to feature richness and flexibility while at the same time remaining reliable and easy to use.
One of the most important aspects of LINSTOR’s design is its modularity. We divided the system into two components, the Controller and the Satellite, so that the Controller component could remain as independent as possible from the Satellite component – and vice versa.
Even inside those two components, many parts of the software are exchangeable – the communication layer, the serialization protocol, the database layer, all of its API calls, even all of the debug commands that we use for internal development, as well as many other implementation details are exchangeable parts of the software. This provides not only a maximum of flexibility for future extensions, it also acts as a sort of safety net. For example, if support for the database or the serialization protocol that we use currently were dropped by their maintainers, we could simply exchange those parts without having to modify every single source code file of the project, because implementation details are hidden behind generic interfaces that connect various parts of our software.
Another positive side effect is that many of those components, being modular, are naturally able to run multiple differently configured instances. For example, it is possible to configure multiple network connectors in LINSTOR, each bound to different network interfaces or ports.
A single communication protocol
As a cluster software, LINSTOR must of course have some mechanism to communicate with all of the nodes that are part of the cluster. Integration with other applications also requires some means of communication between those applications and the LINSTOR processes, and the same applies to any kind of user interface for LINSTOR.
There are lots of different technologies available, but many of them are only suitable for certain kinds of communication. Some clusters use distributed key/value stores like etcd for managing their configuration, but use D-Bus for command line utilities and a REST interface for connecting other applications.
Instead of using many different technologies, LINSTOR uses a single versatile network protocol for communication with all peers. The protocol used for communication between the Controller and the Satellites is the same as the one used for communication between the Controller and the command line interface or any other application. Since this protocol is implemented on top of standard TCP/IP connections, it also made all aspects of LINSTOR’s communication network transparent. An optional SSL layer can provide secure encrypted communication. Using a single mechanism for communication also means less complexity, as the same code can be used for implementing different communication channels.
Even though LINSTOR keeps its configuration objects in memory, there is an obvious need for some kind of persistence. Ideally, what is kept in memory should match what is persisted, which means that any change should be a transaction, both in memory and on persistent storage.
Most Unix/Linux applications have traditionally favored line-based text files for the configuration of the software and for persisting its state, whereas LINSTOR keeps its configuration in a database. Apart from the fact that a fully ACID-compliant database is an ideal foundation for a building a transaction-safe application, using a database also has other advantages. For example, if an upgrade of the software requires changes to the persistent data structures, the upgrade of the data can be performed as a single transaction, so that the result is either the old version or the new version of the data, but not some broken state in between. Database constraints also provide an additional safeguard that helps ensuring the consistency of the data. Assuming that there were a bug in our software, so that it would fail to detect duplicate volume numbers being assigned to storage volumes, the database would abort the transaction for creating the volume due to constraint violations, thereby preventing inconsistencies in the corresponding data structures.
To avoid requiring users to set up and maintain a database server, LINSTOR uses its own integrated database by default – it is simply started as an integral part of the Controller component. Optionally, the Controller can also access a centralized database by means of a JDBC driver.