Pacemaker was never designed to operate across the WAN, or any high latency networks. However, there has always been a need and desire to orchestrate active/passive failovers between data centers and across long distances. To address this issue the Booth Pacemaker add-on was conceived back in late 2011. LINBIT has been involved in the development of Booth since 2013, and has been offering it as a supported solution since 2015.
Booth addresses the shortcomings of Pacemaker by introducing the concept of “tickets”. We constrain particular resources to tickets, and only the site which holds the ticket may start the particular resources. This can be thought of like the old token ring networks of days past. In order for Booth to ensure there is no cluster split, and two sites never possess the ticket at the same time, we utilize arbitration nodes to achieve quorum, and set an expiration period upon the tickets. If a site loses communication with the rest of the Booth cluster its ticket will not renew and it will stop resources within the expected time frame.
While Pacemaker with Booth addresses the issues of High Availability across the WAN, one issue which has always proven difficult is redirecting client traffic to the new site. In most of our demonstrations of Booth we have simply used a round-robin DNS (such as in my demonstration here: Booth Geo Cluster Demo). While round-robin DNS is easy to configure and simple, it is quite inefficient as every other request is discarded.
LINBIT has recently been working with Oracle DynDNS in order to find a more efficient and better solution. Fortunately, Oracle DynDNS offers a Managed DNS service toting a feature aptly named, “Active Failover”. The Active Failover feature can be configured to monitor several things for health. The managed Oracle DynDNS servers can monitor an IP address via ping, SMTP, HTTP(S) or a particular listening TCP port, and then update the DNS destinations only when the service fails and Pacemaker switches the sites. This makes it much more efficient and a perfect match for Pacemaker clusters utilizing Booth.
To demonstrate this solution in detail we have developed a tech-guide which outlines, step-by-step, how to configure this using RHEL 7, Pacemaker, Booth, and Oracle DynDNS Managed DNS, to provide a Highly Available, Geo-Clustered, MariaDB service. This document can be found in the documentation section of our website.