Graphite Archive

In this post we describe how to integrate Icinga2 with Graphite and Grafana on Debian stable (jessie).

What is Graphite?

Graphite stores performance data on a configurable period. Via a defined interface, services can send metrics to Graphite, which are then stored for the designated period. Possible examples of the type of metrics that it might be useful for are CPU utilization or traffic of a web server. Graphs can be generated from the various metrics via the integrated web interface in Graphite. This allows changes in values to be observed across different time periods. A good example of where this type of trend analysis could be useful is monitoring levels on a hard drive. By using a Trend Graph, it’s clear to see the growth rate of the footprint and anticipate when an exchange of storage will become necessary.

What is Grafana?

Graphite does have a private web interface but it is neither attractive nor flexible. This is where Grafana comes in.

Grafana is a frontend for various storage metrics, supporting Graphite, InfluxDB and OpenTSDB, among others. Grafana offers an intuitive interface through which you can create graphs to represent the metrics and a variety of functions to optimise their appearance and presentation. The graphs can then be summarized in dashboards; you can also opt to display one from a specific host only, through parameterization.

Installation of Icinga2

Here we describe the only installation which is essential for Graphite. Icinga2 packages in the current version of Debian can be obtained directly from Debmon projekt.
At the Debmon project up to date versions of the different monitoring tools software for the different Debian releases are provided by the official Debian package maintainers. To include these packages, use the following commands:

# add debmon
cat <<EOF >/etc/apt/sources.list.d/debmon.list
deb http://debmon.org/debmon debmon-jessie main
EOF

# add debmon key
wget -O - http://debmon.org/debmon/repo.key 2>/dev/null | apt-key add -

# update repos
apt-get update

Finally we can install Icinga2:

apt-get install icinga2

Installation of Graphite and Graphite-Web

Once Icinga2 is installed, Graphite and Graphite-Web can also be installed.

# install packages for icinga2 and graphite-web and carbon

apt-get install icinga2 graphite-web graphite-carbon libapache2-mod-wsgi apache2

Configuration of Icinga2 with Graphit

Icinga2 must be configured so that all defined metrics are exported to Graphite. The Graphite component receives this data called “Carbon”. In our sample installation Carbon runs on the same host as Icinga2 and uses the default port, meaning no further configuration of Icinga2 is necessary – it is enough to export on.

To do this, simply enter the command: icinga2 feature enable graphite
Subsequently Icinga2 must be restarted: service icinga2 restart

If the Carbon Server is running on a different host or a different port, the configuration of the file /etc/icinga2/features-enabled/graphite.conf Icinga2 should be adjusted. Details can be found in the Icinga2 documentation.

If the configuration was successful, then after a short time a number of files should appear in “/var/lib/graphite/whisper/icinga“. If they don’t, then you should take a look in the log file of Icinga2 (located in “/var/log/icinga2/icinga2.log“)

Configuration of Graphite – Web

Grafana uses the frontend of Graphite as an interface for the stored metrics of Graphite so correct configuration of Graphite-web is very important. For performance reasons, we operate Graphite-web as a WSGI module. A number of configuration steps are necessary:

First, we create a user database for Graphite-web. Since we will not have many users, at this point we use SQLite as the backend for our user data. We do this using the following commands which initialize the user database and transfer ownership to the user under which the Web front-end runs:
```
graphite-manage syncdb
chown _graphite:_graphite /var/lib/graphite/graphite.db
```
Then we activate the WSGI module in Apache: a2enmod wsgi
For simplicity the web interface should run in a separate virtual host and on its own port. So that Apache listens to this port, we add the line “Listen 8000” to the file “/etc/apache2/ports.conf“
The Graphite Debian package already provides a configuration file for Apache, which is fine to use with slight adaptations. cp /usr/share/graphite-web/apache2- graphite.conf /etc/apache2/sites-available/graphite.conf. In order for the virtual host to also use port 8000, we replace the line:
```
<VirtualHost *:80>
```
with:
```
<VirtualHost *:8000>
```
We can now activate the new virtual host via a2ensite graphite and restart Apache: systemctl restart apache2
You should now be able to reach Graphite-web at http://YOURIP:8000/. If you cannot, the Apache log files in “/var/log/apache2/” can provide valuable information.

Configuration of Grafana

Grafana is not currently included in Debian. However, the author offers an Apt repository from which you can install Grafana. Even if the repository points to Wheezy, the packages should still function under Debian Jessie.

The repository is only accessible via https – so first, you need to install https support for apt: apt-get install apt-transport-https

Next, the repository can be integrated.

# add repo (package for wheezy works on jessie)
cat </etc/apt/sources.list.d/grafana.list
deb https://packagecloud.io/grafana/stable/debian/ wheezy main
EOF
 
# add key
curl -s https://packagecloud.io/gpg.key | sudo apt-key add -
 
# update repos
apt-get update

After this, the package can be installed: apt-get install grafana For Grafana to run we still need to activate the service systemctl enable grafana-server.service and start systemctl start grafana-server.

Grafana is now accessible at http://YOURIP:3000/. The default user name and password in our example is ‘admin’. This password should, of course, be replaced by a secure password at the earliest opportunity.

Grafana must then be configured so that it uses Graphite as a data source. For simplicity, the configuration is explained in the following screencast:

After setting up Graphite as the data source, we can create our first graph. Here is another short screencast to illustrate this:

Congratulations! You have successfully installed and configured Icinga2, Graphite and Grafana. For subsequent steps, please refer to the documentation for the specific projects:

To read more about credativ’s work with Debian, please read our Debian blogs.