Synchronizing Disks for HA

What is this document?
This is only ONE way to keeps nodes in synchronization for a high availability
cluster.  It suits our needs and may be a good
starting block for others so I'm sharing it.  Your mileage may vary.

We used this on a web server to update ~1GB worth of documents in sync every 10
minutes.  Depending on how quickly your files change, you may be able to update
more data, more frequently.  I would not recommend using this without a private
fast ethernet channel, i.e. an extra NIC in each node connected via a crossover
cable.

To determine whether this method would be appropriate for you, you may wish to
do this:

 1. Set the minimum update rate you find acceptable (say, 3 minutes).
 2. Conservatively approximate the amount of data you would expect to change in
    3 minutes (200MB)
 3. Assuming you can count on 6MB/sec from your fast ethernet, multiply this by
    1/3 of your update time.  Why?  You need to account for rsync (relatively
    slow executing) to determine what needs updating and disk write time.  In
    our example, this would give us 360MB worth of updates per cycle.

Of course, you can always do what I did - set it up and see if it updates fast
enough.  The scripts I've provided will notify you if it kicks off again before
the previous one
finished.......

BTW, for lots of data, or rapidly changing data, you'll want a shared disk or
use drbd.  However, we find rsync is still very useful to keep config files,
passwd files, etc. consistent and use it in conjuction with drbd.

One last item:  rsync with the --delete option is dangerous!  Make sure your
command options point to the proper destination!  Test this on non-critical
data.  And then test it again.
 

What you'll need:

    1.  OpenSSH
                You can get it at:  http://www.openssh.com/portable.html
    2.  A copy of rsync
                Found at http://rsync.samba.org/ftp/rsync/binaries/    -OR-  
http://rsync.samba.org/ftp/rsync
   3.   Cron and stuff to mirror.

    In actuality, you don't even need SSH.  You could use rsh instead, if your
security needs permit it.

Installing SSH

START HERE for source distribution:
Untar your openssh distribution.  You might want to read the INSTALL file or
HOWTO, but most can get by with the following:
           ./configure
     make
     make install

START HERE for rpm distribution after running "rpm -ivh openssh_<version
#>.rpm":
Once this is done, make sure sshd is started on bootup.  This could be done via
init.d scripts or by placing (if ssh is installed to the default location) "/
usr/local/sbin/sshd" in you /etc/rc.d/rc.local file.  Type this in now to start
it.

NOTE:      YOU SHOULD READ THE OPENSSH DOCUMENTATION.  THE FOLLOWING
INSTRUCTIONS WILL INSTRUCT YOU
ON ONE WAY TO SET UP AUTO-AUTHENTICATION BETWEEN YOUR CLUSTER NODES....BUT YOU
SHOULD UNDERSTAND WHAT
YOU ARE DOING!!!

Make sure that /usr/local/bin is in your path and type:  "ssh-keygen -d"   
This will create your ssh "key".   Do not enter a passphrase (hit return).

In your ~/.ssh directory, there will be two files:  "id_dsa" and "id_dsa.pub".
 "id_dsa" is your private key, "id_dsa.pub" is your public key (please refer to
your
ssh documentation for explanations).  Run the following commands:
cp id_dsa.pub authorized_keys2
chmod 400 authorized_keys2
Now copy the entire .ssh directory (preserving permissions) to your home
directory on the other node in the cluster.

You should be all set.  Try it out by typing on node A:  "ssh nodeB".  You
should be logged in to nodeB without having to type anything (except "yes" to
accept
the host key the very first time you connect....)
 

Installing Rsync

Well, I'm not really going to tell you how to do this.  I just used the rpm. 
If that's not possible for you, I'm sure the good folks at samba will have a
nice README.Install for you to follow.  However, the binary link in the "What
you'll need" section has binaries for just about all flavors.  Here's the link
for the rpm:   http://rsync.samba.org/ftp/rsync/binaries/redhat

Determining your Rsync command

For our example, let's say you have a web server cluster.  As a result, you
need the directory tree "/html" to be current on both nodes.  Assuming node A
is the master, I would use the command from node A:

    rsync --rsh=/usr/local/bin/ssh -naurvl --delete /html/ localnetB:/html

Let's note a few things.  First, since this will be used with cron, you want to
be sure that you use the full path for the ssh executable (and the rsync
executable for that matter).  Also, note the "/html/" syntax.  This last "/" is
necessary - otherwise you'll have the tree "/html/html" on your slave.  Lastly,
I used the "-n" option.  This is for a dry run.  You want to do this to make
sure everything is copied/deleted as you would expect it.  When you put this in
your crontab file, you'll leave off the "-n" option.  Similarly, the "-v"
verbosity option is only for this test.  In your crontab entry, it will be
replaced with the "-q" option for quiet.  Test the command now and make sure it
does what you want - the "--delete" option can be dangerous!

Create your sync script and crontab entry

You now want to create the script which will run rsync every X minutes via
cron.  Our solution has the same script running on both nodes, but it checks
whether it is running on the current primary (the one holding the services) or
not.  If it's running on the secondary, it quits immediately.  I use the
following perl script, called "synch_all.pl":

┌─────────────────────────────────────────────────────────────────────────────┐
│#!/usr/bin/perl                                                              │
│                                                                             │
│use strict;                                                                  │
│use diagnostics;                                                             │
│                                                                             │
│use lib "/root/scripts";                                                     │
│use whohostlib;                                                              │
│use EnvConfig;                                                               │
│use Mail::Sendmail;                                                          │
│                                                                             │
│#Take care of two-way syncing case during shutdown of one server.  Skips the │
│first try to sync after failover.                                            │
│if ( -e "/var/lock/subsys/mirrorstop" ) {                                    │
│    unlink "/var/lock/subsys/mirrorstop";                                    │
│    exit;                                                                    │
│}                                                                            │
│                                                                             │
│# Only sync if serving IP                                                    │
│my $dirname = "";                                                            │
│my $filename = "";                                                           │
│                                                                             │
│#Get server that this is running on...                                       │
│my $me = "";                                                                 │
│my $other = "";                                                              │
│($me,$other) = whohostlib::whohost;                                          │
│                                                                             │
│if (&whohostlib::whostatus() == 1)                                           │
│{                                                                            │
│    #I'm serving!  You want fries with that?                                 │
│                                                                             │
│    #Only sync if other node is running heartbeat:                           │
│    my $tst = `$EnvConfig::sshpath local$other /etc/rc.d/init.d/heartbeat    │
│status`;                                                                     │
│    # For 0.4.9.2 and earlier use  --->    if ($tst =~ /running.../) {       │
│    # For 0.4.9 "beta" versions...                                           │
│    if ($tst =~ /heartbeat OK/) {                                            │
│        #make sure previous mirror has completed...                          │
│                                                                             │
│        #To avoid hard link probs over partitions, created /var/lock/subsys  │
│/.DONOTREMOVE file                                                           │
│        if (link("/var/lock/subsys/.DONOTREMOVE", "/var/lock/subsys/mirror"))│
│{                                                                            │
│            # /home/ -  Home Directories (all users)                         │
│            # /root/scripts/ - Root scripts/system scripts - like where we   │
│put this script...                                                           │
│            # /var/spool/cron/ - System crontabs                             │
│                my @dirlist = ( "/home/", "/root/scripts/", "/var/spool/cron/│
│" );                                                                         │
│                foreach $dirname (@dirlist) {                                │
│                    system "/usr/bin/rsync --rsh=$EnvConfig::sshpath         │
│$EnvConfig::rsyncoptions $dirname local$other:$dirname";                     │
│            }                                                                │
│                                                                             │
│            # /etc/password - System Password File                           │
│            # /etc/shadow   - Actual Encrypted passwords.                    │
│            # /etc/group    - System Group File                              │
│            # /etc/ld.so.conf - System Linked Libraries.                     │
│            # /etc/shells     - Valid Login Shells                           │
│                my @filelist = ( "/etc/passwd", "/etc/shadow", "/etc/group", │
│"/etc/ld.so.conf", "/etc/shells" );                                          │
│                foreach $filename (@filelist) {                              │
│                    system "/usr/bin/rsync --rsh=$EnvConfig::sshpath         │
│$EnvConfig::rsyncoptions $filename local$other:$filename";                   │
│                }                                                            │
│                                                                             │
│            # release rsync process                                          │
│            unlink "/var/lock/subsys/mirror";                                │
│        } else {                                                             │
│            my $ddd = `date`;                                                │
│            my $subject = "Next RSYNC process starting before previous has   │
│completed!\n";                                                               │
│            my $message = $subject.$ddd;                                     │
│                                                                             │
│            sendmail ('smtp' => $EnvConfig::smtphost,                        │
│                'To' => 'admin@domain.com,admin2@domain.com',                │
│                'From' => 'cluster@domain.com',                              │
│                'Reply-To' => 'cluster@domain.com',                          │
│                'Subject' => $subject,                                       │
│                'Message' => $message)                                       │
│            or warn "Next RSYNC process starting before previous has         │
│completed (Couldn't send email)";                                            │
│                                                                             │
│            #Try to get to known state by killing all synch procs and        │
│removing link...                                                             │
│            unlink "/var/lock/subsys/mirror";                                │
│            exec "/root/scripts/kill_synchs.sh";                             │
│            #"kill_synchs.sh is a one liner:  ps ax | grep synch_all | awk   │
│{'print $1'} | xargs -n1 kill -9                                             │
│            exit;                                                            │
│        }                                                                    │
│    }                                                                        │
│}                                                                            │
└─────────────────────────────────────────────────────────────────────────────┘

Mail::Sendmail is a standard Perl module you can get from CPAN, but you'll
notice two homemade ones.  There's no magic here, just decided to use them for
portability and readability (yes, I know, even though it's perl ;-)).  The
"EnvConfig" one lets us set the paths and options we're using external to the
script so that the
actual synch_all.pl script doesn't change.  You could even add the file and
directory lists to this, but ours doesn't change much cluster to cluster.  The
"whohostlib" has two functions used often.  One checks which host the script is
running on, and the other (whostatus) checks whether the node running the
script is holding the services.

┌────────────────────────────────────────────────────────────────────────┐
│ # Configuration file for all scripts, including host definitions, etc. │
│                                                                        │
│ package EnvConfig;                                                     │
│                                                                        │
│ #www_host = "wwwdev.domain.com";                                       │
│ $www_host = "wwwint.domain.com";                                       │
│ #www_host = "wwwprod.domain.com";                                      │
│                                                                        │
│ $sshpath = "/usr/bin/ssh";                                             │
│                                                                        │
│ $rsyncoptions = "-aurlq --delete";                                     │
│ #$rsyncoptions = "-aurlv";                                             │
│                                                                        │
│ $smtphost = 'mail.domain.com';                                         │
│                                                                        │
│ $Debug = 0;                                                            │
│ 1;                                                                     │
└────────────────────────────────────────────────────────────────────────┘



┌─────────────────────────────────────────────────────────────────────────────┐
│ #!/usr/bin/perl                                                             │
│                                                                             │
│ use strict;                                                                 │
│ use diagnostics;                                                            │
│                                                                             │
│ package whohostlib;                                                         │
│ sub whohost {                                                               │
│     my $other;                                                              │
│     my $me;                                                                 │
│                                                                             │
│     #Who am I?                                                              │
│     $me = `hostname -s`;                                                    │
│     chomp $me;                                                              │
│     $me =~ s/[a-zA-Z]//g;                                                   │
│                                                                             │
│     if ($me) {                                                              │
│        $other = 0;                                                          │
│     } else {                                                                │
│        $other = 1;                                                          │
│     }                                                                       │
│                                                                             │
│     return($me,$other);                                                     │
│ }                                                                           │
│                                                                             │
│ #By doing "grep mirror haresources", we peel off the line with the info we  │
│ want...the cluster IP.  Mirror will be added to you haresources later...    │
│ sub whostatus {                                                             │
│     my $status = 0;                                                         │
│     my $x = `grep mirror /etc/ha.d/haresources`;                            │
│     my @p = split(' ',$x);                                                  │
│     my $ip = $p[1];                                                         │
│     $ip =~ s/IPaddr:://g;                                                   │
│     $x = `/etc/ha.d/resource.d/IPaddr $ip status`;                          │
│                                                                             │
│     # If I am primary, return 1 else return 0                               │
│     if ($x =~ /running/) {                                                  │
│        $status = 1;                                                         │
│     } else {                                                                │
│        $status = 0;                                                         │
│     }                                                                       │
│                                                                             │
│     return $status;                                                         │
│ }                                                                           │
│                                                                             │
│                                                                             │
│ 1; #Return True Value                                                       │
└─────────────────────────────────────────────────────────────────────────────┘


NOTE:  We use a convention for our clusters that if the clustername is "foo"
the two nodes that make up the cluster will be named "foo0" and "foo1".  The
private data interfaces are given the hostnames "local0" and "local1".  This
makes stuff easier with scripting, but you don't have to do it.

NOTE #2:  You may not want a mail fired off.  You may just want a log entry. 
You might want both.  If you're unfamiliar with perl, to add a log entry,
substitute all the commands with "MAIL" in them above with the following:
 ┌───────────────────────────────────────────────────────────────────────────┐
 │open(LOG,">>/var/log/ha-log");                                             │
 │$dstr = `date +%Y/%m/%d_%T`;                                               │
 │chomp $dstr;                                                               │
 │print LOG "$dstr RSYNC: Process starting before previous one completes!\n";│
 │close LOG;                                                                 │
 └───────────────────────────────────────────────────────────────────────────┘

At this point you want to create your crontab entry.  The hard link prevents
more than one sync process from running at the same time, but you want to have
a decent idea of how often you need to synchronize and how long it will take to
synchronize. 

SO, once you determine how often you'll be synchronizing, type "crontab -e" to
modify your crontab entry.  If you don't like vi, try using "setenv EDITOR /usr
/local/bin/emacs" (or export for bash users) to select emacs or a different
editor.  If you want to synchronize every 10 minutes, your entry would look
like this:

    */10 * * * * /script_directory/sync.pl &> /dev/null


You could also redirect output to some logfile if you desire, but keep in mind
how often it runs.

Dealing with Failover

We're just about there now.  The last thing you need to consider is when you
shutdown.  You want to do one last synchronization before you shut down.  I
accomplished this with a heartbeat service called "mirror".  On startup, we do
nothing other than prevent the very first synch_all (as a precaution) and
notify the necessary admins about the failover.

NOTE:   You will want to be sure that any applications writing to the synch'ed
areas stop before your last sync.  If these applications are controlled via the
ipresources configuration file, you can ensure this by listing the "mirror"
script right after your IP address.  However, if they are not, you may want to
add an application exit to the beginning of your mirror script.

┌─────────────────────────────────────────────────────────────────────────────┐
│#!/usr/bin/perl                                                              │
│                                                                             │
│use lib "/root/scripts";                                                     │
│use whohostlib;                                                              │
│                                                                             │
│# See how we were called.                                                    │
│if ($#ARGV == 0) {                                                           │
│    $switch = $ARGV[0];                                                      │
│} else {                                                                     │
│    print "Usage: mirror {start|stop|status|restart}\n";                     │
│    exit -1;                                                                 │
│}                                                                            │
│                                                                             │
│#  start)                                                                    │
│                                                                             │
│if ($switch eq "start") {                                                    │
│    #Make sure to skip first sync.....                                       │
│    ($me,$other) = whohostlib::whohost;                                      │
│    system ("/usr/bin/ssh local$me /bin/touch /var/lock/subsys/mirrorstop"); │
│    #Notify admins of startup....                                            │
│    open (XXX,"/root/scripts/.pagelist");                                    │
│    $hh = `hostname`;                                                        │
│    chomp $hh;                                                               │
│    while (<XXX>) {                                                          │
│    chomp;                                                                   │
│    system "/root/scripts/pagescript.pl $_ \"Starting up heartbeat services  │
│on node $hh\"";                                                              │
│    }                                                                        │
│    close XXX;                                                               │
│                                                                             │
│} elsif ($switch eq "stop") {                                                │
│    #stop)                                                                   │
│                                                                             │
│    print "Mirror stop: \n";                                                 │
│                                                                             │
│    #Am I serving???                                                         │
│    #We'll consider mirror to be "running" if the IP is on this box...       │
│    $x = `grep mirror /etc/ha.d/haresources`;                                │
│    @p = split(' ',$x);                                                      │
│    $ip = $p[1];                                                             │
│    $x = `/etc/ha.d/resource.d/IPaddr $ip status`;                           │
│    if ($x =~ /running/) {                                                   │
│    #Notify admins of shutdown....                                           │
│    open (XXX,"/root/scripts/.pagelist");                                    │
│    $hh = `hostname`;                                                        │
│    chomp $hh;                                                               │
│    while (<XXX>) {                                                          │
│        chomp;                                                               │
│        system "/root/scripts/pagescript.pl $_ \"Shutting down heartbeat     │
│services on node $hh\"";                                                     │
│    }                                                                        │
│    close XXX;                                                               │
│                                                                             │
│    #You want fries with that?  One last sync to other box, if not in the    │
│middle of one now...                                                         │
│    if ( -e "/var/lock/subsys/mirror" ) {                                    │
│        print "Not syncing, already doing so...\n";                          │
│    } else {                                                                 │
│        print "Synchronizing data on standby node: \n";                      │
│        system "/root/scripts/synch_all.pl";                                 │
│    }                                                                        │
│    # To prevent NEW master from syncing back to us too soon, lock their     │
│first synch_all...                                                           │
│    ($me,$other) = whohostlib::whohost;                                      │
│    system ("/usr/bin/ssh local$other /bin/touch /var/lock/subsys/           │
│mirrorstop");                                                                │
│    }                                                                        │
│                                                                             │
│} elsif ($switch eq "status") {                                              │
│    #status)                                                                 │
│    #We'll consider mirror to be "running" if the IP is on this box...       │
│    $y = `grep mirror /etc/ha.d/haresources`;                                │
│    @p = split(' ',$y);                                                      │
│    $ip = $p[1];                                                             │
│                                                                             │
│    $x = `/etc/ha.d/resource.d/IPaddr $ip status`;                           │
│    print $x;                                                                │
│} elsif ($switch eq "restart") {                                             │
│    #restart)                                                                │
│                                                                             │
│    system "/etc/ha.d/resource.d/mirror stop";                               │
│    system "/etc/ha.d/resource.d/mirror start";                              │
│                                                                             │
│} else {                                                                     │
│    print "Usage: mirror {start|stop|status|restart}\n";                     │
│}                                                                            │
│exit;                                                                        │
└─────────────────────────────────────────────────────────────────────────────┘

Finally, you need to install the script in your ipresources on both nodes. 
You'll want mirror to be the first service listed (after IP).  For our
webserver, it would read:

    nodeA 192.168.85.1 mirror httpd

 

Be Careful

Please test your setup on non-critical data first.  There could be a bad typo
above or whatever.  The "--delete" option can be dangerous.  You've been
warned.
 

Rev 1.0.0
Rudy Pawul
rpawul@iso-ne.com

