Centralized Log Setup awesant elasticsearch logstash and kibana3 part2

Overview

This guide will is the continuation of part 1 where we installed logstash, redis and elasticsearch on the logger server. In this guide we will be installing shippers to ship the logs to the logger server. Logstash is compatible with many shippers, infact you can use logstash as shipper it self however I do not like doing it on machines running something other than java applications such as apache tomcat or JBoss. For everything that is not running java I normally use a very lightweight shipper called awesant. Logstash require java to be installed and is memory hungry, so if you have a light machine with few gigs of RAM available you would not want to use logstash as shipper. Also, assuming the machine running java apps like JBoss will be beefier you can take out about 256Mb for logstash to run. Another reason I like running logstash as shipper for java apps is to tackle multiline log output which is a mess and can be corrected easily using logstash filter multiline.

Centralized Log Setup awesant elasticsearch logstash and kibana3 part2

Install Awesant Log Shipper

On CentOS/RHEL

Append the following to it

Install GPG Key

Install the package

On Debian/Ubuntu

Note: If you are on Ubuntu go ahead and use Debian Wheezy repo

Add GPG key

Edit /etc/apt/sources.list file

Add the repository according to your distro version

Install https transport for apt, resynchronize the package index files and install awesant

Next configure awesant-agent by editing its conf file.

The conf file is easy to understand, I am shipping four log files and adding tags to them, you can add multiple tags by comma separating them. The tags will help us add tag based authorization to kibana later. Just in case you want your developers to access apache logs but not to have access on something like secure or message logs and similarly the administrators to have access to secure and messages log but not apache, tags will come to rescue.

Start the shipper

Now if you are running a Java App such as Tomcat or JBoss you can use logstash as a shipper

Download logstash

Create the init file

Append the following to it

Create the conf file. Make changes to the log file path and redis server ip address as per your environment.

Start logstash

This concludes the second part of the three part series, we will install kibana3 to view our logs next and have authentication and authorization going as well.

Screenshot from 2014 01 18 171734 300x158 Centralized Log Setup awesant elasticsearch logstash and kibana3 part2