Block storage drbd compression deduplication

Linux Data Deduplication and Compression: One more reason to use block level data replication.

Having recently returned from my 6th Red Hat Summit (RHS), I’m writing this blog to answer a common question: “why replicate at the block level?” Using block-level replication, we can easily add high availability or disaster recovery features to any application that doesn’t natively support them.

The most frequently asked question we heard at RHS was, “how do you compare to [insert application replication OR filesystem here?]”. In most cases, the answer was, “LINBIT’s replication software, DRBD, replicates data at the block level.” It would be an extreme task to run performance comparisons vs all of the other replication technologies on the market, so generally we provide background information, including:

Block storage drbd compression deduplication

  • DRBD can usually replicate with 1-3 percent overhead to the cluster’s backing disks, as measured by FIO
  • In dual-primary mode, overhead increases to 15-20 percent
  • DRBD is compatible with any application or Linux filesystem, and is effective at replicating multiple applications simultaneously.
  • DRBD has a read-balancing feature. If you are running a read intensive application, DRBD will pass through reads to secondary nodes once the primary is running at maximum capacity, enabling you to leverage all of your replicated systems. One test showed 1.7x the read performance compared to the advertised speed of the drive.

Deduplication and Compression

Generally, it comes down to efficiency. EMC, NetApp, and the other big storage players use block level replication in their appliances because this way the replication doesn’t need to go “all the way up the stack.” It enables flexibility, stability, and performance. And now, Red Hat has given us one more reason to replicate at the block level: Deduplication and Compression.

In the most recent Red Hat Enterprise Linux 7.5 release, Red Hat announced integration of Red Hat VDO, or Virtual Data Optimizer. VDO is used for deduplication and compression of Linux environments. Though it can be paired with other replication technologies, it can only be fully leveraged when the replication sits underneath the VDO device. Why? You want to deduplicate and compress your data before replicating it for efficiency gains.

Effective transfer times

According to Louis Imershein, Red Hat’s Principal Product Manager for data reduction technologies, “Solutions like LINBIT’s DRBD are able to capture data below the VDO layer.  This means that datasets that benefit from deduplication and compression get replicated in their dehydrated form. With less data to move, Red Hat Enterprise Linux customers with LINBIT DRBD can benefit from faster effective transfer times and reduced bandwidth requirements.”

So, as you’re thinking about underlying storage for your applications, ensure you are using a solution which allows you to maximize the benefit of the existing Linux utilities built in, and around, your Operating System. Thanks to Red Hat, block level replication is now more important than ever.

 

Greg Eckert on Linkedin
Greg Eckert
In his role as the Director of Business Development for LINBIT America and Australia, Greg is responsible for building international relations, both in terms of technology and business collaboration. Since 2013, Greg has connected potential technology partners, collaborated with businesses in new territories, and explored opportunities for new joint ventures.

LINBIT joins the Open Source OpenSDS Project

(Press release originally posted at www.linuxfoundation.org.)

With 1.7 million downloads, LINBIT Represents the Largest Community of Open Source Mission Critical Workloads

COPENHAGEN, Denmark (KubeCon + CloudNativeCon Europe 2018), May 1, 2018 – The OpenSDS Project, an open source community addressing software-defined storage integration challenges with the goal of driving enterprise adoption of open standards, today announced that LINBIT, a longtime leader of open source technologies, has joined OpenSDS. OpenSDS is a Linux Foundation hosted project.

The OpenSDS Project is comprised of storage users and vendors, including Dell EMC, Fujitsu, Hitachi, Huawei, IBM, Western Digital, Vodafone, NTT Communications, Toyota ITC, Yahoo! Japan, and Oregon State University. The project seeks to collaborate with other upstream open source communities such as Cloud Native Computing Foundation and OpenStack, as well as industry organizations such as Storage Networking Industry Association (SNIA). LINBIT’s contributions to OpenSDS will offer seamless integration of DRBD remote data replication in OpenSDS.

“Being responsible for mission critical applications, the LINBIT community has always defined the frontline of the open source momentum in the enterprise,” said Steven Tan, chair of the OpenSDS Technical Steering Committee and VP and CTO Cloud Storage Solutions at Huawei. “We welcome LINBIT to the OpenSDS Project, and look forward to offering its trusted Linux technologies to the OpenSDS community”

“Cloud and container technologies present new challenges for efficient storage management, as do digital transformation and the increasing value of data assets,” said Philipp Reisner, CEO of LINBIT. “LINBIT is ideally positioned to address these challenges, building on its decades of experience with always-on, always-available technologies and leveraging the advanced development work that we are doing with erasure coding, highly available message queuing (HA-MQ), and geo-clustering via highly available network address lookup (HA-DNS).”

In a related announcement, LINBIT launched the public beta release of LINSTOR, a major new addition to its SDS product portfolio for the growing containerized applications space. With container-native block storage, LINSTOR fills a significant gap in the market to provide data persistence for elastic applications with support for Kubernetes and OpenShift environments.

The OpenSDS technical community hosts discussions on a dedicated mailing list: [email protected] For more information about OpenSDS, please email [email protected].

About The Linux Foundation

The Linux Foundation is the organization of choice for the world’s top developers and companies to build ecosystems that accelerate open technology development and industry adoption. Together with the worldwide open source community, it is solving the hardest technology problems by creating the largest shared technology investment in history. Founded in 2000, The Linux Foundation today provides tools, training and events to scale any open source project, which together deliver an economic impact not achievable by any one company. More information at www.linuxfoundation.org

Sebastian Schinhammer
Sebastian brings his experience in marketing to the table. As Marketing Manager at LINBIT he cares especially about content marketing, brand appearance and public relations. He fills brands with life telling engaging stories. Sebastian strives for his marketing efforts being helpful information or great entertainment rather than all bark but no bite.

 

Cluster-wide management of replicated storage with LINSTOR

The new generation of LINBIT’s storage management system focuses on ease-of-use, resilience and scalability.

Todays IT installations often consist of many individual servers, each running some part of the software infrastructure that together form the kind of service that the installation is supposed to provide. Software processes rely on data, and high availability or disaster recovery solutions, which have typically included replication of the data to one or multiple other physically independent systems.

LINSTOR is the new generation of the software component that implements the automatic management of replicated storage resources in the LINBIT SDS system. Besides adding new features that users have been previously requested, such as the ability to make use of multiple-tier storage, LINBIT has also improved the existing features.

Linstor Linbit Opennebula openstack

Linstor features

Ease-of-use

Our experience has shown that administrators of complex IT environments typically struggle with two things: figuring out how to make the system do what they want, and determining the cause of the problem when a system fails to do what the administrators expect. Creating a new product has given us the opportunity to consider these issues during the design phase and to focus on making the new software easier to use and troubleshoot. Two examples of related enhancements are the more consistent and logical naming of LINSTOR objects and commands, and the greatly enhanced logging and problem reporting.

Resilience

Another area of improvement that we focused on is the resilience of the system as a whole, which depends not only on the LINSTOR software, but also on the entire external environment. For this reason, we designed LINSTOR to manage unexpected changes and to recover from many different types of failures of external components.

Scalability

LINSTOR greatly increases scalability by its ability to perform changes on multiple resources and on multiple nodes concurrently, while still remaining responsive to new requests.

Multi tier storage

Many users have requested the support of multiple-tier storage, and we are pleased to announce that, by adding the concept of storage pools, it has been implemented in LINSTOR. We made this a flexible feature, so that multiple storage pools can be configured, even using different storage backend drivers per storage pool and/or per node if necessary.

The new software is also capable of dealing with multiple network interface cards, each of which can be used as the replication link for DRBD resources or as the communication link for LINSTOR. This feature enables splitting the control network (providing LINSTOR communication) from the data network (providing DRBD replication link communication). IPv6 is  supported for both, LINSTOR communication and DRBD replication links.

RoadmapProduction release roadmap

The roadmap for the production release includes support for:

  • taking snapshots of replicated resources
  • thinly provisioned LVM storage
  • ZFS storage
  • encrypted and authenticated network communication within LINSTOR
  • taking advantage of LINSTOR’s multi-user-capability
Robert Altnoeder on Linkedin
Robert Altnoeder
Robert joined the LINBIT development team in 2013. He had worked with
DRBD at a startup company in the SaaS field before joining LINBIT. His
current primary field of work is the architecture and implementation of
LINSTOR, the cluster management component of LINBIT's SDS software.