Nagios Performance Tuning – Tech Tips: Understanding Disk I\O

We often get questions about the kind of hardware requirements needed for a particular Nagios installation.  As covered in a previous article, this is often a very difficult question to answer since monitoring environments differ so much.  Most people assume that for a large Nagios installation, it’s a matter of simply adding enough CPU’s to the machine to handle the workload that it’s given.  Although having enough CPU power is important, I’ve found that it’s ultimately not the biggest hardware limitation to the system.  A large Nagios installation creates an enormous amount of disk activity, and if the hard disk can’t keep up with the constant traffic flow that needs to happen, all of those precious CPU’s are simply going to wait in line to be able to do what they need to do on the system.  I’ve talked to some users who have spent some serious money on hardware to have insanely fast disks to handle their workload, but I wanted to do some experiments in-house for those users who may need to have better performance on a budget.  I want to give special thanks to Nagios community members Dan Wittenberg and Max Schubert for documenting some of the tricks that you guys pioneered on this topic.

Lets look at some of the key files that create disk activity on a Nagios install.  The examples below match a typical source install of Nagios Core, and any vanilla Nagios XI install.

/usr/local/nagios/var/status.dat – This is the bread and butter file of all of the “live” information on the monitoring environment.  This file gets updated every  10-20 seconds (as specified in nagios.cfg) with all current status information.

/usr/local/nagios/var/objects.cache – This file stores all of the object configuration data for Nagios.  This file only gets updated upon a restart of the Nagios process.

/usr/local/nagios/var/host-perfdata && service-perfdata – These files may be in a different location for  Core install, but this file functions as an intermediary file for PNP’s NPCD daemon that processes performance data results.  This file gets updated about every 10-15 seconds.

/usr/local/nagios/var/spool – This directory tree acts as a dropbox for all incoming check results.  The disk activity in this directory is almost constant, since both Nagios and NPCD are continually creating result files, and then reaping the results every X number of seconds.

/usr/local/nagios/var/*.log – I thought the log files were worth mentioning.  Logging takes both CPU and disk activity, so minimize all unnecessary logging if you need to scale.

Test Case: My old desktop workstation as an XI Server.  This machine is not especially powerful, but one of my side projects is seeing how much I can load it up purely with configuration changes:

Nagios XI Server .

• Intel Dual Core CPU 3gz
• 2gb of RAM
• 140gb SATA HD, probably 7200 RPM
• Offloaded MySQL to a VM with 1gb of RAM, and a single CPU
• 1823 Hosts, 7379 Services.  All active checks running on a 5mn check interval.
• 9202 checks in 5mn
• Around 40 checks per second
• All active checks are being executed from the XI server, mostly running PING, HTTP, DNS To IP, and DNS Resolution

Pre-Existing Tweaks:

• Offloaded MySQL to a 2nd machine
• RAM Disk for status.dat and object.cache
• rrdcached in use
• Minimal Logging
• Minimized Notifications – Don’t send more notifications than you need to. This holds up the main Nagios execution loop and uses more CPU.

The Problem:

• In this experiment I started with around 5000 checks, and tried increasing to around 8000.  I found that the CPU load was will fine (around 2.0), but my host and service latencies were continually growing with time, and Nagios couldn’t maintain the check schedule.  This wasn’t an overtaxing of the CPU, but Nagios simply couldn’t write information to the disk fast enough to keep up with the check schedule.  It was essentially constantly waiting in line to access the disk.

The Solution:

• Although this won’t work for every environment, I basically moved every file with constant disk activity to the RAM disk.  The danger of this is that if the server experienced a hard power-down, then all of those check results and performance data results would get lost.  However, if your monitoring server has a hard shutdown, you’ve probably got bigger problems than just a few lost check results.
• All performance data and check results in the spool directory are now written to the RAM disk, as well as the intermediary perfdata files.

/usr/local/nagios/etc/nagios.cfg Modifications:

/usr/local/nagios/etc/pnp/npcd.cfg Modifications:

Modified Nagios XI Command Definitions:

For some long time Core users, this may not be new information.  But what I’ve found in researching this topic is that a lot of people have done modifications to their system like this, but there is very little public documentation on how people actually did it.  I’m hoping this information will be useful to some users and maybe even save somebody some money on hardware. : )