Log Management using logstash, kibana, graylog2 on CentOS, RHEL, Fedora Part-3

Overview

Update: Looking for the latest graylog2 guide? click here

My initial plan was to make this a 2 part series but then graylog2 happened and extended this for another one. So here is graylog2 another web interface that you may use instead of kibana. The problem with graylog2 is that it only understands gelf (Graylog Extended Log Format) or syslog log formats. I am content with kibana running in my production env, if you are not then just have to make few tweaks to run graylog2. The advantages of using graylog2 over kibana is in build authentication method and streams setup. Creating streams let you give access to a user on a limited set of logs, for example a developer doesn’t need to know what is happening on the routers and switches. So create a stream with only the logs for application the developer requires access on.

Log Management using logstash, kibana, graylog2 on CentOS, RHEL, Fedora Part-3

Now in the previous post we had logstash send logs to elasticsearch but this time we would like it to send it to the graylog2 server which will pick and store them to elasticsearch anyhow but in gelf format. Also mongodb is used to store session and cache information. The graylog2 web interface then will pick them up and show it to you nicely.

Lets start then, first and foremost as discussed above we would like logstash to send the logs to graylog2 server. Open the logstash conf file in an editor of choice


# vi /etc/logstash/logstash.conf

Edit the output section to send the logs to port 12201, the graylog2-server will be listening on this port.


output {
# elasticsearch {}
gelf {
chunksize => 1420
facility => "logstash-gelf"
host => "127.0.0.1"
level => "INFO"
port => 12201
sender => "%{@source_host}"
}
}

Add 10gen Repository for mongodb


# vi /etc/yum.repos.d/mongodb.repo

Append the following to it
For x86_64 bit CentOS


[10gen]
name=10gen Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64
gpgcheck=0

For i386 32 bit CentOS


[10gen]
name=10gen Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/i686
gpgcheck=0

Install mongodb and other dependencies


# yum install mongo-10gen-server mongo-10gen httpd httpd-devel readline-devel

Open the mongod.conf file in an editor of choice


# vi /etc/mongod.conf

Enable authentication by uncommenting the line


auth = true

Start mongodb


# /etc/init.d/mongod start
# chkconfig mongod on

Login to mongo db and add user and database for graylog2


# mongo
>use admin
>db.addUser('admin', 'password')
>db.auth('admin', 'password')
>use graylog2
>db.addUser('grayloguser', 'gray123')
>exit

Download graylog2-server


# wget https://github.com/downloads/Graylog2/graylog2-server/graylog2-server-0.9.6.tar.gz
# tar zxvf graylog2-server-0.9.6.tar.gz
# mv graylog2-server-0.9.6 graylog2-server
# cd graylog2-server
# cp graylog2.conf.example /etc/graylog2.conf

Open the conf file in an editor of choice


# vi /etc/graylog2.conf

Add the username, db name and password we created above


mongodb_useauth = true
mongodb_user = grayloguser
mongodb_password = gray123
mongodb_host = localhost
#mongodb_replica_set = localhost:27017,localhost:27018,localhost:27019
mongodb_database = graylog2
mongodb_port = 27017

# Graylog Extended Log Format (GELF)
use_gelf = true
gelf_listen_address = 0.0.0.0
gelf_listen_port = 12201

Create a startup script for graylog2-server


# vi /etc/init.d/graylog2-server

Append the following to the file


#!/bin/sh
# graylog2-server: graylog2 message collector
# chkconfig: - 98 02
# description: This daemon listens for syslog and GELF messages and stores them in mongodb
#
CMD=$1
NOHUP=/usr/bin/nohup
JAVA_CMD=/usr/bin/java

GRAYLOG2_SERVER_HOME=/opt/graylog2-server

start() {
echo "Starting graylog2-server ..."
$NOHUP $JAVA_CMD -jar $GRAYLOG2_SERVER_HOME/graylog2-server.jar > /var/log/graylog2.log 2>&1 &
}

stop() {
PID=`cat /tmp/graylog2.pid`
echo "Stopping graylog2-server ($PID) ..."
kill $PID
}

restart() {
echo "Restarting graylog2-server ..."
stop
start
}

case "$CMD" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
*)
echo "Usage $0 {start|stop|restart}"
RETVAL=1
esac

Create graylog2-server logrotate entries


# vi /etc/logrotate.d/graylog2-server

/var/log/graylog2.log {
daily
rotate 90
copytruncate
delaycompress
compress
notifempty
missingok
}

Register graylog2-server init script


# chmod +x /etc/init.d/graylog2-server
# chkconfig --add graylog2-server
# chkconfig graylog2-server on

Start graylog2-server and verify it starts listening on port 12201


# service graylog2-server start
# netstat -tunlp | grep 12201

Download the graylog2-web and install it


# wget https://github.com/downloads/Graylog2/graylog2-web-interface/graylog2-web-interface-0.9.6.tar.gz
# tar zxvf graylog2-web-interface-0.9.6.tar.gz
# mv graylog2-web-interface-0.9.6 /var/www/html/graylog2
# chown -R apache:apache /var/www/html/graylog2
# cd /var/www/html/graylog2
# gem update
# gem install git rake bundler
# bundle install

Update the mongoid,yml file


# cd /var/www/html/graylog2/config
# vi mongoid.yml

Add the username and password


# Use environment variables
#production:
# host: <%= ENV['MONGOID_HOST'] %>
# port: <%= ENV['MONGOID_PORT'] %>
# username: <%= ENV['MONGOID_USERNAME'] %>
# password: <%= ENV['MONGOID_PASSWORD'] %>
# database: <%= ENV['MONGOID_DATABASE'] %>

# or specify values manually
production:
host: localhost
port: 27017
username: grayloguser
password: gray123
database: graylog2

Create the indexes


# cd /var/www/html/graylog2
# bundle exec rake db:mongoid:create_indexes RAILS_ENV=production --trace

Install apache passenger module


# passenger-install-apache2-module

passenger.so  300x176 Log Management using logstash, kibana, graylog2 on CentOS, RHEL, Fedora Part 3

Create a new conf file to load passenger module


# vi /etc/httpd/conf.d/passenger.conf

Append the following to the file as instructed by your apache passenger module installer(Refer the screenshot above)


LoadModule passenger_module /usr/lib/ruby/gems/2.8/gems/passenger-3.0.19/ext/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.19
PassengerRuby /usr/bin/ruby

Create virtual hosts


# vi /etc/httpd/conf/httpd.conf

Append the following code to the file


<VirtualHost *:80>
ServerName graylog.linuxdrops.com
DocumentRoot /var/www/html/graylog2/public
RailsEnv production
ServerAlias gray.logger
ErrorLog logs/graylog2-error_log
CustomLog logs/graylog2-access_log custom
<Directory /var/www/html/graylog2/public>
Allow from all
Options -MultiViews
</Directory>
</VirtualHost>

Restart apache


# /etc/init.d/httpd restart

Point your browser to

http://ipaddress-or-domainname

graylog007 300x176 Log Management using logstash, kibana, graylog2 on CentOS, RHEL, Fedora Part 3

graylog009 300x176 Log Management using logstash, kibana, graylog2 on CentOS, RHEL, Fedora Part 3