Setting Up Graylog Server

I have been looking around for an easy to use and reasonable priced solution for managing logs distributed among many servers and system metrics for these servers. I had a brief look into setting up an ELK system but I found that looked quite cumbersome. Recently I came across Graylog and I found it looked quite promising. I thus set up a little sample system.

While the documentation for Graylog is generally quite good, I found it a bit difficult to piece together the various steps in setting up a minimal working system. Thus I have documented these steps below!

Installing Graylog and Dependencies

Just follow the excellent CentOS installation instructions from the Graylog documetation.

Make sure to provide details for sending emails under the header # Email transport.

If you are using a firewall, open ports 9000 for TCP and 51400 for UPD. For instance, by assuring the following lines are in /etc/sysconfig/iptables.


-A INPUT -p tcp -m state --state NEW -m tcp --dport 9000 -j ACCEPT
-A INPUT -p udp -m state --state NEW -m udp --dport 51400 -j ACCEPT

Don’t forget to restart the iptables service: sudo systemctl restart iptables.

Collecting the Logs from Another CentOS System

  • Install rsyslog on the system

sudo yum install rsyslog

  • Enable and start rsyslog service (also see this guide)

sudo systemctl enable rsyslog

sudo systemctl start rsyslog

  • Edit the file /etc/rsyslog.conf and put the following line at the end, into the section marked as # ### begin forwarding rule ### (replace yourserver.com with your graylog server address.

*.* @yourserver.com:51400;RSYSLOG_SyslogProtocol23Format

  • Restart rsyslog

sudo systemctl restart rsyslog

The rsyslog log messages should now be getting send to your server. Give it a few minutes if you don’t see the messages in graylog immediately. Otherwise, check the system log for any errors (sudo cat /var/log/messages).

Also, you can test the connection by entering the following on the monitored system:


nc -u yourserver.com 51400
Hi

This should result in the message Hi being received by graylog.

Analysing Logs

The next steps are quite easy to to since they can be done in the excellent graylog user interface.

critical errors

  • Create an alert. Trigger it when there is ‘more than 0’ messages in the stream you have just created.

Done! You are now collecting logs from a server and you will receive an email notification whenever there is a serious issue reported on the server!

Free Cloud-based Log and Metrics Management Solutions

I have been looking around for a while for a cloud-based service which allows collecting logs and metrics and analysing them. I am particularly interested in a solution which can be deployed for free for smaller applications/amounts of data.

Here are some of the solutions I came across:

Loggly

loggly

  • Store and analyse logs and metrics
  • Free plan available: For 200MB/day, 7 day data retention
  • It looks to me like the Free plan does not allow using the Loggly API!

Splunk

splunk

  • One of the first and most popular solutions in the space
  • Store and analyse logs and metrics
  • Free version available; allows storing up to 500MB/day; I don’t think there is any limitation on data retention
  • Note that free version requires to install the Splunk server on your own server

Sematext

sematext

  • Based on popular open source ELK stack
  • Free plan allows monitoring up to 5 hosts, but only comes with 30 min data retention

Logz.io

logz

  • Also based on ELK stack
  • Free plan allows 3GB upload per day; data retention limited to 3 days.

Overall, I am not too happy with these offerings. In particular, the short data retention periods seem to make some of these offerings too limited to be useful.

Maybe the best option here would be to install your own ELK stack or Graylog. Here are some guides for that: