Log Management using logstash, kibana, graylog2 on CentOS, RHEL, Fedora Part-3


Update: Looking for the latest graylog2 guide? click here

My initial plan was to make this a 2 part series but then graylog2 happened and extended this for another one. So here is graylog2 another web interface that you may use instead of kibana. The problem with graylog2 is that it only understands gelf (Graylog Extended Log Format) or syslog log formats. I am content with kibana running in my production env, if you are not then just have to make few tweaks to run graylog2. The advantages of using graylog2 over kibana is in build authentication method and streams setup. Creating streams let you give access to a user on a limited set of logs, for example a developer doesn’t need to know what is happening on the routers and switches. So create a stream with only the logs for application the developer requires access on.

Log Management using logstash, kibana, graylog2 on CentOS, RHEL, Fedora Part-3

Now in the previous post we had logstash send logs to elasticsearch but this time we would like it to send it to the graylog2 server which will pick and store them to elasticsearch anyhow but in gelf format. Also mongodb is used to store session and cache information. The graylog2 web interface then will pick them up and show it to you nicely.

Lets start then, first and foremost as discussed above we would like logstash to send the logs to graylog2 server. Open the logstash conf file in an editor of choice

# vi /etc/logstash/logstash.conf

Edit the output section to send the logs to port 12201, the graylog2-server will be listening on this port.

output {
# elasticsearch {}
gelf {
chunksize => 1420
facility => "logstash-gelf"
host => ""
level => "INFO"
port => 12201
sender => "%{@source_host}"

Add 10gen Repository for mongodb

# vi /etc/yum.repos.d/mongodb.repo

Append the following to it
For x86_64 bit CentOS

name=10gen Repository

For i386 32 bit CentOS

name=10gen Repository

Install mongodb and other dependencies

# yum install mongo-10gen-server mongo-10gen httpd httpd-devel readline-devel

Open the mongod.conf file in an editor of choice

# vi /etc/mongod.conf

Enable authentication by uncommenting the line

auth = true

Start mongodb

# /etc/init.d/mongod start
# chkconfig mongod on

Login to mongo db and add user and database for graylog2

# mongo
>use admin
>db.addUser('admin', 'password')
>db.auth('admin', 'password')
>use graylog2
>db.addUser('grayloguser', 'gray123')

Download graylog2-server

# wget https://github.com/downloads/Graylog2/graylog2-server/graylog2-server-0.9.6.tar.gz
# tar zxvf graylog2-server-0.9.6.tar.gz
# mv graylog2-server-0.9.6 graylog2-server
# cd graylog2-server
# cp graylog2.conf.example /etc/graylog2.conf

Open the conf file in an editor of choice

# vi /etc/graylog2.conf

Add the username, db name and password we created above

mongodb_useauth = true
mongodb_user = grayloguser
mongodb_password = gray123
mongodb_host = localhost
#mongodb_replica_set = localhost:27017,localhost:27018,localhost:27019
mongodb_database = graylog2
mongodb_port = 27017

# Graylog Extended Log Format (GELF)
use_gelf = true
gelf_listen_address =
gelf_listen_port = 12201

Create a startup script for graylog2-server

# vi /etc/init.d/graylog2-server

Append the following to the file

# graylog2-server: graylog2 message collector
# chkconfig: - 98 02
# description: This daemon listens for syslog and GELF messages and stores them in mongodb


start() {
echo "Starting graylog2-server ..."
$NOHUP $JAVA_CMD -jar $GRAYLOG2_SERVER_HOME/graylog2-server.jar > /var/log/graylog2.log 2>&1 &

stop() {
PID=`cat /tmp/graylog2.pid`
echo "Stopping graylog2-server ($PID) ..."
kill $PID

restart() {
echo "Restarting graylog2-server ..."

case "$CMD" in
echo "Usage $0 {start|stop|restart}"

Create graylog2-server logrotate entries

# vi /etc/logrotate.d/graylog2-server

/var/log/graylog2.log {
rotate 90

Register graylog2-server init script

# chmod +x /etc/init.d/graylog2-server
# chkconfig --add graylog2-server
# chkconfig graylog2-server on

Start graylog2-server and verify it starts listening on port 12201

# service graylog2-server start
# netstat -tunlp | grep 12201

Download the graylog2-web and install it

# wget https://github.com/downloads/Graylog2/graylog2-web-interface/graylog2-web-interface-0.9.6.tar.gz
# tar zxvf graylog2-web-interface-0.9.6.tar.gz
# mv graylog2-web-interface-0.9.6 /var/www/html/graylog2
# chown -R apache:apache /var/www/html/graylog2
# cd /var/www/html/graylog2
# gem update
# gem install git rake bundler
# bundle install

Update the mongoid,yml file

# cd /var/www/html/graylog2/config
# vi mongoid.yml

Add the username and password

# Use environment variables
# host: <%= ENV['MONGOID_HOST'] %>
# port: <%= ENV['MONGOID_PORT'] %>
# username: <%= ENV['MONGOID_USERNAME'] %>
# password: <%= ENV['MONGOID_PASSWORD'] %>
# database: <%= ENV['MONGOID_DATABASE'] %>

# or specify values manually
host: localhost
port: 27017
username: grayloguser
password: gray123
database: graylog2

Create the indexes

# cd /var/www/html/graylog2
# bundle exec rake db:mongoid:create_indexes RAILS_ENV=production --trace

Install apache passenger module

# passenger-install-apache2-module

passenger.so  300x176 Log Management using logstash, kibana, graylog2 on CentOS, RHEL, Fedora Part 3

Create a new conf file to load passenger module

# vi /etc/httpd/conf.d/passenger.conf

Append the following to the file as instructed by your apache passenger module installer(Refer the screenshot above)

LoadModule passenger_module /usr/lib/ruby/gems/2.8/gems/passenger-3.0.19/ext/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.19
PassengerRuby /usr/bin/ruby

Create virtual hosts

# vi /etc/httpd/conf/httpd.conf

Append the following code to the file

<VirtualHost *:80>
ServerName graylog.linuxdrops.com
DocumentRoot /var/www/html/graylog2/public
RailsEnv production
ServerAlias gray.logger
ErrorLog logs/graylog2-error_log
CustomLog logs/graylog2-access_log custom
<Directory /var/www/html/graylog2/public>
Allow from all
Options -MultiViews

Restart apache

# /etc/init.d/httpd restart

Point your browser to


graylog007 300x176 Log Management using logstash, kibana, graylog2 on CentOS, RHEL, Fedora Part 3

graylog009 300x176 Log Management using logstash, kibana, graylog2 on CentOS, RHEL, Fedora Part 3