In the previous post, I told you about the setup of making high-available cluster for Jira. In this post I will continue the same thing and tell you how to configure GlusterFS & HeartBeat to make that setup happen.

How it works

So we start with installing GlusterFS first. GlusterFs is a clustered File system which can be used for scaling the storage capacity by having distributed volumes or we can just use GlusterFS for providing high availability of storage using replicated volumes feature. In this we use multiple servers to maintain a replicated view of the main storage, so in case any one of them is down the other storage sever/node can be used.

So for our use case we would use the replicated volume to store JIRA_HOME directory and the tomcat application, which would imply that all the data is always present on more than one server at any given instance.

Before we can create replicated volumes, we should have GlusterFS server and the client which would be used by Jira to access GlusterFS setup on both the servers. In this setup we would use two servers of same configurations (memory,cpu), so that each of them can be used interchangebly as the master in case the original master crashes.

GlusterFS setup

1. Install GlusterFS on both the servers. Download latest debian package from http://www.gluster.org/download/ and run following commands:

   sudo apt-get install openssh-server wget nfs-common
   sudo dpkg -i glusterfs-3.2.0.deb

2. Install Gluster Native Client, because it provides high concurrency, performance and transparent failover compared to NFS or CIFS clients. For this we should have the fuse-utils packages installed

   sudo apt-get install fuse-utils

1. Configure glusterfs to start automatically in debian every time the system boots and then start it manually:

   update-rc.d glusterd defaults
   /etc/init.d/glusterd start

2. Add server1 and server2 to a trusted Storage Pool. For this you should probe the server2 on server1. Ensure the hostnames used are properly resolved by both of the servers.

   gluster peer probe server2

3. Once server2 has been added to storage pool, then create a replicated volume on server1 using the command

   gluster volume create data replica 2 transport tcp server1:/test-volume server2:/test-volume

4. Finally start the volume

   gluster volume start data

5. Mouting new GlusterFS volume using native client so that JIRA can use that directory as a replicated directory which would synchronize data on both the machines.

   server1$: mount -t glusterfs server1:/test-volume /mnt/glusterfs
   server2$: mount -t glusterfs server2:/test-volume /mnt/glusterfs

6. Automatically mouting GlusterFS on startup, edit /etc/fstab and add the following line on both the machines

   server1:/test-volume /mnt/glusterfs glusterfs defaults,_netdev 0 0
   server2:/test-volume /mnt/glusterfs glusterfs defaults,_netdev 0 0

7. Now the GlusterFS is ready and we can use /mnt/glusterfs as the replicated volumne for JIRA to store the JIRA_HOME directory and tomcat application.

So once we have a GlusterFS setup done, we have a single directory which is replicated on both the machines and now we have to install Jira on one of them and it would be installed on the other one two.

But we can’t have two Jira instances running on the same JIRA_HOME directory, so we have to create a setup in which we have Jira on both the machines but only one of them is running at any instance. For that we will install Heartbeat to manage Jira on both the machines, so Jira on server2 will only start when server1 is down( which implies Jira is down on that machine).

Heartbeat Setup

Now we will install Heartbeat, which will monitor the uptime of the both the servers and in case the master server goes down it will start the JIRA on the second server and also assign VIP (Virtual Ip address) to that server.

1. Install Heartbeat on both the servers

   sudo apt-get install heartbeat

2. For configuring Heartbeat there are three files which needs to be changed – /etc/ha.d/ha.cf, /etc/ha.d/authkeys and /etc/ha.d/haresources files
3. The ha.cf file should be same on both the nodes and you can read about supported options in ha.cf using command “man ha.cf” on linux. Sample ha.cf file looks like which we will use for our setup.

   keepalive 1
   deadtime 5
   udpport 694
   bcast eth0
   mcast eth0 225.0.0.1 694 1 0
   udp     eth0
   logfacility     local0
   logfile /var/log/ha-log
   node server1
   node server2
   # The server1,server2 specified in node option should exactly match the hostname of servers given by "uname -n".

4. Authkeys file define which authentication method will be used for heartbeat communication between the nodes. This should also be same on both the nodes.

   auth 1
   1 md5 somerandomstring

5. Give corrent permissions to authkeys file

   chmod 600 /etc/ha.d/authkeys

6. Finally the haresources file. This file is used when heartbeat starts to configure the services managed by heartbeat. And in our use case we have to assign Virtual Ip Address(192.168.1.9) to the current master, start tomcat application and also apache if tomcat is running behind a apache service. So our haresources file looks like

   on server1 file content "server1 IPaddr::192.168.1.9/24 tomcat6 apache"
   on server2 file content "server2 IPaddr::192.168.1.9/24 tomcat6 apache"

7. Start Heartbeat on both the servers

   /etc/init.d/heartbeat start

8. After starting make sure the server1 get the VIP 192.168.1.9 and the tomcat is running only on server1.

And now you shutdown server1, the Heartbeat will notice that and start the tomcat6, apache services on server2 and assign it the the VIP 192.168.1.9 and your Jira will be back up within minutes.

Other Solutions

DRBD instead of GlusterFS

I haven’t tried DRBD, but it provides the same funtionality of replication as GlusterFS but we can also try similiar stuff using DRBD and HeartBeat. I would love to analysis the differences between the two setup but would explore that later.

Links

1. http://linux-ha.org/wiki/Main_Page
2. GlusterFS installation
3. http://www.linuxvirtualserver.org/HighAvailability.html