In the “Root-on-DRBD” Tech-Guide we showed how to cleanly get DRBD below the Root filesystem, how to use it, and a few advantages and disadvantages. Now, if there’s a complete, live, backup of a machine available, a few more use-cases become available; here we want to discuss testing upgrades of production servers.
Everybody knows that upgrading production servers can be risky business. Even for the simplest changes (like upgrading DRBD on a Secondary) things can go wrong. If you have an HA Cluster in place, you can at least avoid a lot of pressure: the active cluster member is still running normally, so you don’t have to hurry the upgrade as if you had only a single production server.
Now, in a perfect world, all changes would have to go through a staging server first, perhaps several times, until all necessary changes are documented and the affected people know exactly what to do. However, that means having a staging server that is as identical to the production machine as possible: exactly the same package versions, using production data during schema changes (helps to assess the DB load [queue your most famous TheDailyWTF article about that here]), and so on.
That’s quite some work.
Well, no, wait, it isn’t that much … if you have a simple process to copy the production server.
That might be fairly easy if the server is virtualized – a few clicks are sufficient; but on physical hardware you will need DRBD to quickly get the staging machine up-to-date after a failed attempt – and that’s exactly what DRBD can give you.
The trick is to “shutdown” the machine in a way that makes the root filesystem unused; then resync DRBD from the production server, to reboot into the freshly updated “installation”.
(Yes, the data will have to be done in a similar way – but that’s possible with DRBD8, and will get even easier with DRBD9.)
A sample script that shows a basic outline is presented in the resync-root branch in the Root-on-DRBD github Repository. It should be run on the staging server only.
Please note that this is a barely-tested draft – you’ll need to put quite some installation-specific things in there, like other DRBD resources to resynchronize at that time and so on!
Feedback is very welcome; Pull-requests even more so 😉