perform cron jobs redundancy using rcron on centos rhel debian ubuntu fedora

Overview

rcron is a powerful tool that helps system administrators in setting up cron jobs redundancy and failover over groups of machines. RCRON ensure that a job installed on several machines will only run on the active one at any time. Source https://code.google.com/p/rcron/

perform cron jobs redundancy using rcron on centos rhel debian ubuntu fedora

In our scenario we are using two web servers in HA with a database server on the backend, we have a cron job that is required to run daily once on the web server and update some feilds in the database, we could have the a cron job setup on a web server to run daily but what if this server goes down, the application would work just fine as the second web server would take over but what about that cron job. For scenarios like these you can use rcron to run the cron job for redundancy and failover over groups of machines. In our scenario one web server can act as active node for rcron and the second as the passive node, the active would run the cronjob daily but of it goes down the passive would perform this job. Do keep in mind though that if an active server halt or crash while running a cron job, rcron won’t transfer this job elsewhere nor relaunch it. But it does try to log enough informations so you can see which jobs got interrupted.

Now we would require some type of HA daemon for rcon to know machine’s states like keepalived or heartbeat or wackamole, for this guide we would be using keepalived

First and foremost enable IP forwarding on both the nodes. Open /etc/sysctl.conf in an editor of choice

Set net.ipv4.ip_forward = 1 and remove # in front to uncomment it

Load in sysctl settings

Install the prerequistes

On Fedora, CentOS/RHEL

On Debian, Ubuntu

Pull the source code for rcron and run configure, make and make install, this has to be done on both nodes

Create the directory and the conf file on both nodes

On Active Node

Append the following to the file

On Passive Node

Append the following to it

On the active node execute

On the passive node execute

Next configure keepalived

On the active node

Make sure keepalived conf look like this, do replace the ip address below with the address of your nodes

On the passive node

Make sure keepalived conf look like this, do replace the ip address below with the address of your nodes

Finally add the cron entry to crontab on both machines, replace myjob with your cronjob name

Start the Keepalive service on both the nodes

Another thing that I would like to perform is to make sure that the keepalive daemon stays up and running, in case keepalive daemon gets killed we would like to restart it for which we are going to create a script on both the nodes.

Append the following

Make it executable

Add this to crontab on both the nodes

What this script essentially does is to check keepalive daemon is running all the time, if for some reason it decides to crash the other node would automatically become active thus the script would set this node to passive and start keepalived daemon again