Secure Linux: Atomicorp includes DRBD for replication

Every so often we get a chance to test new¹ software. Usually this opportunity is driven by the question: Does DRBD play nicely with it?

At HostingCon this year, we met a team from Atomicorp and decided that it would be interesting to see if we could get DRBD running on this hardened version of Linux. Overall, LINBIT’s broad client-base loosly includes “security” since “Availability” is one of the 3 Security pillars of the CIA triad.

 

Image Source: Panmore Institute

Security certainly fits with Atomicorp since they focus on clients in the federal, financial, healthcare, and hosting space. Their HQ is based in the same business park as Raytheon, Boeing, and Booz Allen Hamilton, if that tells you anything about their market.

We frequently take on the challenge of seeing if we can get DRBD compiled and working correctly, like that time we installed it on 2 raspberry pi’s, and this case was no different. While we were confident that there wouldn’t be issues with installation, — after all, it’s Linux — we needed to verify compatibility with the ASL (Atomic Secured Linux™) hardened kernel before announcing that it works.

After speaking with the Atomicorp team, they let us know that some of their clients were already running DRBD and Pacemaker for High Availability within their data centers. That’s great news! We anticipated that the testing would go quickly since we already had verified users.

Upon installing DRBD on a pair of RHEL 7 systems, we found something unexpected. DRBD is already included in the ASL kernel. This means Atomicorp is hardening and packaging a newer mainline kernel instead of hardening that which the distribution supplies. Nice work Atomicorp! The DRBD 8.4.5 version in the ASL kernel is pretty recent too.

It’s funny. Clients often ask us if we have seen DRBD used for their specific use case. DRBD is so versatile that we’re not always familiar with every situation. If we had been asked if anyone was using DRBD with Atomicorp’s ASL product, we would have said “I don’t know.” The irony here is that when you install the ASL hardened kernel, you may automatically get DRBD on a distribution where you otherwise may have not. It is available for everyone who runs Atomicorp’s ASL kernel whether the end user leverages the replication functionality or not².

This isn’t just a fun, internal office story; this is the essence of how Open Source Software works. We now know that there is a connection between ASL and DRBD, and are delighted to work with Atomicorp moving forward. It just makes sense since end-clients of both Atomicorp and LINBIT achieve feature-sets that they wouldn’t have otherwise. Altogether, our partners help advocate for our open source software and when our solutions are combined, everyone keeps inching toward bigger and better solutions, while maintaining focus on their core competencies.

So does the DRBD software work with Atomicorp and the Atomic Secured Linux™ kernel? Of course it does; and now, for the next few weeks, I get to be mocked by my coworkers for having our engineers test something which already had our software baked into it. 😉

 

1: New to us.
2: You’ll still need the userland utilities to manage and initialize DRBD, but that’s less of security concern than compiling and inserting a kernel module.

Configuring an HA+DR Apache ActiveMQ™ Cluster

To quote the Apache Software Foundation:

Apache ActiveMQ™ is the most popular and powerful open source messaging and Integration Patterns server. Apache ActiveMQ is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License.

Deploying a synchronously replicated shared-nothing storage cluster (DRBD) as outlined in this guide, is a supported method for achieving HA without requiring a clustered filesystem or shared database. This method also mitigates the risk of a SAN, clustered filesystem, or shared database being a single point of failure in our persistent storage layer. Read more

DRBD Top is HERE!

DRBD Top is about to make your life easier! Once you install this utility, you can expect to see an easy to read, simplified overview of your resources and their status. This is especially useful if you have a large number of resources and don’t want to be overwhelmed with information, yet still want key details. Read more

True Cost of Data Loss

When you hear the word ‘data,’ what does it make you think?  Something interesting and exciting?  Probably not.  But, what if you lost your customer database, your employee healthcare information, or your organization’s website transactions?  Now, that might make you think twice.  

Read more

Oracle DynDNS with Booth Geo-Clustering

Pacemaker was never designed to operate across the WAN, or any high latency networks. However, there has always been a need and desire to orchestrate active/passive failovers between data centers and across long distances. To address this issue the Booth Pacemaker add-on was conceived back in late 2011. LINBIT has been involved in the development of Booth since 2013, and has been offering it as a supported solution since 2015.

Booth addresses the shortcomings of Pacemaker by introducing the concept of “tickets”. We constrain particular resources to tickets, and only the site which holds the ticket may start the particular resources. This can be thought of like the old token ring networks of days past. In order for Booth to ensure there is no cluster split, and two sites never possess the ticket at the same time, we utilize arbitration nodes to achieve quorum, and set an expiration period upon the tickets. If a site loses communication with the rest of the Booth cluster its ticket will not renew and it will stop resources within the expected time frame.

While Pacemaker with Booth addresses the issues of High Availability across the WAN, one issue which has always proven difficult is redirecting client traffic to the new site. In most of our demonstrations of Booth we have simply used a round-robin DNS (such as in my demonstration here: Booth Geo Cluster Demo). While round-robin DNS is easy to configure and simple, it is quite inefficient as every other request is discarded.

LINBIT has recently been working with Oracle DynDNS in order to find a more efficient and better solution. Fortunately, Oracle DynDNS offers a Managed DNS service toting a feature aptly named, “Active Failover”. The Active Failover feature can be configured to monitor several things for health. The managed Oracle DynDNS servers can monitor an IP address via ping, SMTP, HTTP(S) or a particular listening TCP port, and then update the DNS destinations only when the service fails and Pacemaker switches the sites. This makes it much more efficient and a perfect match for Pacemaker clusters utilizing Booth.

To demonstrate this solution in detail we have developed a tech-guide which outlines, step-by-step, how to configure this using RHEL 7, Pacemaker, Booth, and Oracle DynDNS Managed DNS, to provide a Highly Available, Geo-Clustered, MariaDB service. This document can be found in the documentation section of our website.

Would you want to be your own car mechanic?

Data seems to be on everyone’s mind these days.  From employee to financial data, your company has to keep it available through seamless replication — without downtime. LINBIT DRBD is the open source software that ensures High Availability for your enterprise.

Read more

Red Hat Summit KeyNote: How to survive rapid change – Jim Whitehurst

In Jim Whitehurst’s Red Hat Summit 2017 keynote, he opened by making a bold claim: planning is dead. He followed up his statement with predictions from the past that have turned out to be grossly inaccurate. He referred to a prominent example from 1899 when the US Patent office stated that “Everything that can be invented has been invented,” a quote that received soft laughs from the audience of 10,000.

Following the claim that traditional planning is dead, Jim backed it up with a statement of fact when he stated that “cognitive bias makes us terrible at predicting the future. The faster the rate of change, the worse we are at planning.” So, how do companies succeed if they can’t plan linearly? Both companies and people are fighting to keep pace with change, and change is coming from every direction. For example: how would car manufacturers have planned for Uber? Uber came out of nowhere with a few dollars of venture funding. Now, car companies are building their businesses around providing cars for new-age taxi services like Uber. No doubt that your industry has a similar analogy.

Because things are no longer planned, but rather “emerge,” the key is to participate in the communities who are initiating change, and help drive the forward-moving innovation.  Think about the “Big Data” market where tons of companies making small steps forward are creating massive market changes. The same is happening with AI.
Organizing humans is now about creating context for individual action. The new model for industry success is “Try, Learn, & Modify.” Organizations who figure out how to reward fast failure, continually adapt to changing plans, and participate inside their respective market communities at scale will thrive. If you embrace, rather than fear the rapid change in the world, you will have the chance to be apart of it.

DRBD 9 Now Supports Fencing In Pacemaker

Fencing is the process of isolating a node from a computer cluster or protecting shared resources when a node appears to be malfunctioning. As the number of nodes in a cluster increases, so does the likelihood of failure.1

Read more

Albireo Virtual Data Optimizer (VDO) on DRBD

TL;DR: Pairing DRBD with VDO reduces the replication network and storage utilization by ~85% while increasing load by ~0.8.

VDO (Virtual Data Optimizer)[1] is a ready-to-run software package that delivers block-level deduplication, compression, and thin provisioning capabilities to Linux. VDO operates inline at a 4 KB granularity, delivering the best possible balance of performance and data reduction rates.

Read more

Persistent and Replicated Docker Volumes with DRBD9 and DRBD Manage

drbdmanage has been replaced by LINSTOR.

You find more information about LINSTOR:

https://www.linbit.com/en/products-and-services/linstor/

 

Thank you and have a fun read!

 

 

Nowadays, Docker has support for plugins; for LINBIT, volume plugins are certainly the most interesting feature. Volume plugins open the way for storing content residing in usual Docker volumes on DRBD backed storage.

In this blog post we show a simple example of using our new Docker volume plugin to create a WordPress powered blog with a MariaDB database, where both the content of the blog and the database is replicated among two cluster nodes. Read more