Archive for the 'Visualizations' Category

Nagios Log Server vs. Elasticsearch – Logstash – Kibana

Recently I was asked the following questions via email and thought it would make a great post to explain the differences between deploying Nagios Log Server or just the Elasticsearch, Logstash, Kibana Stack (ELK).

The question was as follows:

In the company I currently work with, we were thinking about deploying ElasticSearch and Logstash along with Kibana, in order to further facilitate log processing and visualization.

What would the added value be if we went for Nagios Log Server instead of ElasticSearch, Logstash and Kibana?

Is there any downside in choosing to install ElasticSearch, Logstash and Kibana on our own instead of installing Nagios Log Server?

Nagios Log Server DashboardOn the surface this is a really straight forward question, and was also asked right away in the Log Monitoring and Log Management with Nagios presentation I gave at the Nagios World Conference.  Nagios Log Server does in fact use the ELK stack, and we are surely glad we chose the stack we did because of the outstanding performance, reliability, redundancy and expandability that it allows Log Server to take advantage of to build this spectacular product.

While both options allow a platform that will give the ability to index and analyze logs from various systems such as syslog, Windows Event Log, text based logs and many many more, Nagios Log Server was designed to be a full featured Log Management product, taking into account the needs of enterprise customers that require important items such as security and role based authentication.

So what makes Nagios Log Server stand out above the competition?  Usually, it all comes down to cost.  While other solutions may be “free” there is no such thing as free lunch, and the man hours learning about “free” technology, as well as the man hours configuring and maintaining such a system must be accounted for.  Additionally, once the “free” system is deployed, who do you contact when something goes wrong, and what is the associated cost?

Added Value

To the point of added value I will list below the extra / added functionality that Nagios Log Server brings to the table over the standard ELK stack.  For the most part, Nagios Log Server simply delivers the missing pieces expected in an enterprise solution, and at the same time provides commercial support for the product as well as saving many organization a ton of money, simply because we at Nagios have done the work figuring out all of the complex features, instead of you having to roll your own system out so to speak. Below is a short list of some of the value added features:

  • Commercial Support – This one item alone makes Log Server stand out.  All licenses come with customer only support.
  • Easy installation – Setup is incredibly easy, either start with a pre-created VM or run a simple install script and your Log Server will be online in a few minutes.  Setting up ELK for production does take a fair amount of knowledge for best practices, although they do make it pretty easy to get going in development environment.
  • Easy cluster formation – Log Server makes sure every member of the cluster knows which IP’s/hostnames it should communicate with and constantly keeps the list current.  While ELK does uses multicast discovery by default, this is almost never recommended in production.
  • Authenticated UI and API – Believe it or not, the ELK stack does not come with any semblance of authentication or authorization, which means anyone that can access the ELK system on the network can not only read, but Delete or Modify your sensitive log data. Log Server has full authentication and authorization to all difference users access to different information, as well as an API that is secured with keyed access.
  • Easy Log Source Wizards and Scripts – Built into Log Server are many easy setup instruction and scripts to make setting up various systems such as Windows Event Logs, or rsyslog a breeze to start sending logs into log server.  Additionally, we have built in easy import functionality to get historical logs into Log Server.
  • GUI based logstash configuration – I believe Log Server has the only GUI based logstash configuration management system in existence.  Easily add logstash configuration inputs, filters, and outputs, with full drag and drop functionality.  On top of that, from one central interface you can add, edit, modify and deploy the configuration files to ALL of the servers in your cluster instead of manually editing configuration files via text editor on each system manually.
  • Per user savable Dashboards – Users can save their custom dashboards that represent the log data the way they like to visualize it.  Each user can have any number of custom dashboards.
  • Per user savable Queries – Queries can be saved separate from dashboards, and you can apply different queries to be viewed in different dashboards.
  • Global Dashboards and Queries – Both queries and dashboards can be saved as Global by administrators so other individuals can use them.
  • Alerting based on Queries – Log server adds the ability to get alerts based on any query.  alerts can be sent via email, sent to a Nagios Monitoring server, sent to an SNMP Trap Receiver, or passed to a custom script for execution.
  • Automated Backup and Maintenance – Automated backup management is part of Log Server, and is basically set it and forget it function.  Once you have set where you want your backup information stored, it will keep all of your precious logs safe and secure there in case you need to retrieve them in the future.
  • GUI based Cluster Management – At a glance view and management of the Log Server cluster status right through the GUI.
  • GUI based Instance Management – Granular view of every member of the cluster, including about 60 metrics such as, disk utilization, memory usage, system load, and so much more.
  • GUI based Index Management – Detailed view (another 25 metrics per index) and actions on every index in the cluster, such as document count, size, and ability to open close, and delete indexes.

Any Downside to Log Server?

This is somewhat a loaded question, I’ll try to be as objective as I can.  I can really only think of two.

  • Not Always Free – While Log Server does offer a free version for a single instance up to an average of 500MB/day, Log Server is commercial software and isn’t free when scaled out to multiple instances, however, with an introductory price of $995, almost all organizations would have spent 10X that much in man hours alone just having their technical staff learn how to install and configure all of the open source components properly.  Once your team has figured it all out, you would have to create any of the above items if they are of value to your organization.
  • Currently Requires CentOS or RHEL – Currently Nagios Log Server is only supported on CentOS or RHEL operating systems, however we are working to get distributions on other operating systems available, and it can be run in a VM on virtually any OS.

We welcome additional questions in the comments below.  Feel free to take Nagios Log Server for a fully functional 90 day free trial.

Exploring the New JSON CGIs in Nagios Core 4.0.7! (Part 1)

The JSON CGIs, from the JSON branch of core, have been officially released with Nagios Core 4.0.7!

The original design goals were:

  1. To provide all information available in current CGIs in JSON format.
  2. Place the presentation responsibility on the client to minimize network traffic and server load.
  3. Perform operations on the server side that are significantly more easily done there.
  4. Spark community developers to create new Nagios Core UI’s from the easy to work with JSON from the CGIs.

The CGIs provide an API to query object, status, and historical information through GET requests.  They use the same authentication as other CGIs.  Once queried, they return valid JSON that can be parsed into JavaScript objects for client side models and processing.  The API is very robust, providing multiple ways to limit queries – name/descriptions, host/service groups,  update/changes times, among many others.

The three new CGIs are:

  1. objectjson.cgi  (object configuration)
  2. statusjson.cgi  (status information)
  3. archivejson.cgi  (historical logs)

Additionally, a new web app is included – jsonquery.html & jsonquery.js.  This is a small UI for crafting GET requests, it can be used to trial specific parameters for GET requests, or to just explore the api.  It is also the easiest way to get acquainted with the new CGIs.

Continue reading ‘Exploring the New JSON CGIs in Nagios Core 4.0.7! (Part 1)’

Keeping an Eye on Problematic Services with the Status Info Dashlet

The Status Info Dashlet has been available on the Nagios Exchange website for about a year now. It is a very cool dashlet, that in my opinion, doesn’t get the attention it deserves. The Status Info Dashlet allows you to display the current status of a service as a dashlet on a dashboard in nice big numbers, and is especially useful in cases when you want to keep an eye on a problematic service.

Setup and Installation of the Status Info Dashlet is quite easy:

First, you need to download the dashlet from the Nagios Exchange site: Status Info Dashlet
Direct Download Link: statusinfo.zip

Next, add the dashlet from the Nagios XI web interface:
Admin -> System Extensions -> Manage Dashlets -> Browse -> statusinfo.zip -> Open -> Upload Dashlet

Once the dashlet has been successfully installed, you can add it to a dashboard of choice:
Dashboards -> Add Dashlets -> Available Dashlets
and clicking on the “dashify” icon in the upper left corner of the dashlet to add it to a Dashboard .

You will see the “Add To Dashboard” interface, which has many different options, allowing you to customize your dashlet. The first three options on the top allow you to set the dashlet’s title, the dashboard, where you would like the dashlet to be added to, and the refresh rate. Below these three options, you will see eight tabs for even more customizations. I will go through each one briefly, and describe the most basic choices.

1) Object-1

Here you can select the object (service), that you want your dashlet to use. You also have an option to show the current service status as text, set the background color, “trim” the output (the beginning or the end), in order to discard the information that you don’t need. You can also show the last refresh time and the refresh interval underneath the object.

2) Object-2

This is NOT a place, where you can select a second object (service) – the name is a bit misleading. Here you can change the object text formatting (text size, weight, style, color, etc.)

3) Text

From here, you can add additional text to your dashlet and style it how you want.

4) Name

The menu allows you to select the name format that you want to appear in the dashlet. The available options are:

– Host
– Host – Service Name
– Service Name
– Service Name – Host

You can also format the text (size, style, color, etc.)

5) Layout

This allows you to select how you want the dashlet displayed. There is a drop-down menu with a various combinations for displaying “Text”, “Object”, and “Name”.

6) Preview

When you click on this tab, you will see a preview of what your dashlet will look like.

7) Help

The help menu explains in details the various options that exist in each section/tab.

8) License

Here you can read the license for this dashlet.

After you selected all of the options you want, just click on the “Add it” button on the bottom. Then you can go to the dashboard, where you added that dashlet and view status info for your service.  You’ve now successfully added the Status Info Dashlet to your Nagios XI views.  It’s as easy as that.

Nagios Fusion 2012 Preview

Need a simpler solution for scaling Nagios?  Distributed monitoring environments often contain several Nagios servers in order to cover multiple geographic or network locations, or sometimes just to scale large enough on a single network.  Nagios Fusion 2012 is a central dashboard and data aggregation for all of your Nagios installations.  Fusion 2012 will integrate seamlessly with Nagios XI and Nagios Core 3.x installs, and requires no additional configuration changes on any of your Nagios servers.  Here’s a highlight of the current feature list:

 

  • Unified authentication for all Nagios XI servers
  • User-defined, customizable dashboards and menus
  • Easily drill down to any Nagios server to find problems
  • Fused Tactical Overview information
  • Fused Health Summaries for Nagios servers
  • Fused  Alert Summary
  • Fused Alert Histogram
  • Fused Top Alert Producers
  • Several new data visualizations

 

The power exists in Nagios Fusion to aggregate almost any information across multiple Nagios installs.  The main question we’re looking at from here is: “What do users want to see in their central Nagios dashboard?”  We’re interested in getting some user feedback for ideas on this project as well as some beta testers for the upcoming release.  Here are some screenshots to give an idea as to what is to come.