With the likes of cloud computing and virtualisation starting to become staples for today’s business, IT environments are continuing to grow in complexity. Furthermore, there is growing pressure on many organisations to reduce the environmental impact of their IT systems.

In response to these developments, organisations need to change the way they manage and therefore monitor their IT infrastructure. For example, with the advent of virtualisation, organisations now have a fundamentally different infrastructure platform from which they are running business systems. This in turn requires a different monitoring approach.

If organisations can’t adapt their physical world approaches to monitoring, they could find that they aren’t made aware of systems problems until users start complaining about downtime.

When it comes to cloud they need to consider how they are going to monitor both the performance and use of cloud services. Finally, with it being unlikely that organisations will move all services to the cloud they need to find ways to monitor environmental factors such as the temperature of data centres and the CO2 emissions of IT.

This data will not only allow them to find ways to reduce energy costs, but also report back up the organisation when it comes to the environmental targets that have been set.

At the same time as being presented with an increasingly complex environment to monitor, IT departments are under pressure to dramatically reduce the cost of day to day operations. The result is a conundrum that requires new ways of thinking to solve.

Taking The Pain Out Of Virtualisation

Virtualisation has been utilised by many organisations looking to improve operational efficiency. Thus, with physical and virtual machines now present in many businesses, monitoring and managing both effectively is vital.

When implementing and monitoring virtualisation, businesses face a number of challenges – all of which add to the complexity of IT monitoring. Virtualisation allows you to improve server utilisation. Yet, when you increase the utilisation of a previously under-used server, it can be difficult to know how that server will manage the increased load.

Therefore, monitoring the performance of the VM host and all the virtual machines running on it is imperative otherwise organisations could find they suffer from performance slow downs or worse still, downtime. The problem is virtualisation doesn’t behave like, or conform to the same rules as physical hardware and therefore often traditional approaches to monitoring both infrastructure and applications don’t meet the grade.

You often find that the VM itself can be monitored, but there is no insight into what’s going on within it or into the applications it’s hosting.

A further challenge for a lot of organisations is staying on top of the growing sprawl of virtualisation, due to the ease of creating virtual machines. To date, this is something that many have struggled with using older proprietary IT monitoring and management tools.

To combat these potential issues, organisations need to update their approaches to IT monitoring. They need to use tools that can provide insight into virtualisation and that enables them to understand how every application on every virtual machine is running, what problems could occur and how they can be remedied.

Without this investment in IT monitoring, virtualisation will not bring the ROI expected and the increased complexity could in fact result in ongoing performance issues.

Up In The Air – The Cloud Effect

The acceptance of cloud computing has increased over the past few years. Now, the terms public, private and hybrid are understood by the majority of organisations, with data centre association Afcom citing that more than 70 per cent of UK businesses are already implementing cloud, or seriously considering it.

Adopting a cloud model allows businesses to move some applications and services off-premise, resulting in reduced costs, with less IT equipment required on-premise. However, cloud does potentially present a number of challenges when it comes to monitoring the overall performance of an organisation’s IT infrastructure.

With this in mind, organisations should start thinking about how they are going to monitor and assess cloud performance in the future, in order to guarantee IT performance. The difficulty at the moment is that there are not many standards around cloud when it comes to moving applications between public and private clouds, and consequently monitoring them.

Therefore, before choosing a monitoring tool, organisations need to ensure that the solution has the flexibility to adapt to any future changes once standards do eventually emerge.

Another challenge organisations should be looking out for as they begin to use cloud-based services, is that they could experience cloud sprawl. Although one of the benefits of the cloud model is that users can buy services on a pay-as-you-go basis, organisations will need to make sure they are controlling cloud deployments, as it can be very easy to continue paying for cloud services long after they have finished using them.

This is for the simple reason that people often forget to tell the cloud provider that they have finished using the service and therefore keep getting billed. Ultimately, businesses will need to have a consolidated view of all cloud services meaning they will then be able to monitor how much these are being used. This way, they can make sure they only pay for what they are actually using rather than paying for what they have deployed.

Going Green Without Feeling Blue – Green IT Made Simple

In addition to adopting new technologies or IT models to reduce costs, organisations are also under pressure to become greener. This pressure comes not just from the board as they look to achieve cost savings, but also through growing public and legislative pressure on them to reduce carbon emissions.

The UK Government has set a target of cutting CO2 emissions by 34% of 1990 levels by 2020. Businesses play a key part in this and the pressure is on to adhere to the new rules and regulations that aim to ensure we hit those targets.

The main challenge for many organisations however, is that they don’t know how much power is being consumed by their IT, especially as data centres are now often located away from an organisation’s main site. Organisations must be able to construct a better picture of energy usage in their data centres – building environmental factors into monitoring to identify areas where further energy savings can be made.

For example, organisations need to be able to tell when and where they can power down servers that are not in use. They also need to monitor data centre temperature very carefully – ensuring cooling systems are working efficiently and keeping their servers at optimum temperature.

Furthermore, IT monitoring can also help organisations build a picture to help them decide which old systems can be decommissioned if they are not performing sufficiently. Being green does not need to be difficult. The right tools will indicate the right processes to improve your environmental credentials without the need for more investment or even more IT complexity.

As the IT landscape continues to change, it is becoming increasingly clear that the monitoring tools and techniques that worked in the past simply don’t suit the modern environment. More flexibility and agility is needed in IT monitoring, while reducing the overall cost. Overall, IT monitoring can help organisations improve their business and environmental performance. The challenge is now to ensure they have the right tools and techniques in place to do this.

Apache Solr is an open source enterprise search service from the Lucene project. Solr is written in Java and runs as a standalone full-text search server within a servlet container such as Tomcat.

Like any service or component in your architecture, you’ll want to monitor it to ensure that it’s available and gather performance data to help with tuning.

In this post, we’ll look at how we can monitor Solr, what performance metrics we might want to gather and how we can easily achieve this with Opsview.

A check list for service checks

Solr is built on Lucene so follows the same layout, an index contains documents that are comprised of fields. As part of the search service value add over Lucene, Solr provides a number of useful ways of obtaining health status / monitoring metrics:

1. Health-check status using the /admin/ping handler
2. The admin statistics page /admin/stats.jsp (XML styled with XSL)
3. JMX MBeans

The list of applicable checks could be defined by whether it is a health check or a data gathering check – but this would lead to a lot of overlap. Instead the list is divided into the checks that can be performed remotely (without an installed agent on the server) and those that are best performed locally to the Solr server.

Remote (agent-less) checks

What should we look for over the network?

Firstly we can have a host-level check which may perform a network level ping. Next we can check TCP connectivity to the servlet container port and then make an HTTP GET request to the Solr ‘front page’ and check for a known string (e.g. Welcome to Solr).

Now we’ve made it up to the application layer so can start to perform Solr specific checks.

Items to monitor may include (delete as applicable):

1. Ping status
2. Number of docs
3. Number of queries / queries per second
4. Average response time
5. Number of updates
6. Cache hit ratios
7. Replication status
8. Synthetic queries

Agent-based checks

Installing an Opsview agent on the Solr server means we can run additional checks over NRPE (Nagios Remote Plugin Executor). This could be operating system level checks such as memory/disk utilisation or CPU load, or the following:

1. Java servlet container process is running
2. JMX checks e.g. heap memory or custom MBeans
3. File age
4. Log parsing for exceptions

The Solr wiki describes how to configure JMX support: http://wiki.apache.org/solr/SolrJmx.

Opsview configuration

For the rest of this article you’ll need to have Opsview installed (or the Opsview VMWare appliance) and have completed the Quick Start.

Solr-specific Plugin

Install the Solr plugin at https://github.com/rbramley/Opsview-solr-checks into /usr/local/nagios/libexec/

The check_solr plugin was developed using Perl, so that it could be contributed back to Opsview. It requires the CPAN XML::XPath module (sudo cpan -i XML::XPath).

The plugin includes usage instructions, check_solr -h which can also be viewed in Opsview by selecting the ‘Show Plugin Help‘ link beneath the Plugin drop down (see Figure 1). The -u option can be used to specify the URL path for multi-core set-ups.

Service check setup

Figure 1 gives an example of a service check configuration.

Figure 2 shows the agentless service check group with plugins and their arguments.

Host configuration

Figure 3 shows a simplistic host setup with a ping check.

Figure 4 is an extract from the Monitors tab, where we select the checks we want performed for the current host.

Viewing output

The check results shown in Figure 5 are visible by navigating through the host group hierarchy.

If you click on the graph icon of Solr Cache Hit Ratios this will drill down onto the graph shown in Figure 6.

Clicking on the graph icon for Solr Avg Response Time – standard will take you to the graphs in Figure 7.

If you shutdown Solr, then the check results will start to turn critical and show in red as per Figure 8.

Alternatives

There are a few other plugins available for monitoring Solr from Opsview, depending on your needs:

• http://code.google.com/p/nagios-plugins-shamil – provides ping, replication status and num docs
• http://code.google.com/p/solr-nagios-check – provides QPS, response time and num docs

Also, chapter 8 of the recently published Apache Solr 3 Enterprise Search Server book includes a section on Monitoring Solr Performance.

Summary

Using check_solr in conjunction with Opsview allows you to ensure that your Solr server is available and provides you with metrics that can help you tune your Solr configuration.

This can be complemented with additional agent-based operating system and JMX checks to give you a full picture view.

Alerts happen. They are the reason why monitoring applications were created: to alert us when servers need attention. The difference between an effective network monitoring system and an annoying one is a fine line between information and noise. Alerts should be descriptive and prompt an administrative action, not elicit a huff of frustration. Here are a few ways to keep your Opsview installation (and you) effective and relevant in your company.

Use a Smartphone

A smartphone should be a tool on every system administrator’s bat belt. The more mobile you are, the more time you spend away from the Operations Center. Why not take Opsview with you? With Opsview Mobile for Android, you can do just that. There is an ambitious roadmap to the mobile app including support for other devices, but if you have an Android there is no reason to wait getting it installed.

The app handles basic needs very well, including a real-time overview of all hosts and services and alert acknowledgement. If you are away from the office (like at the beach!) and get an alert on your phone, acknowledge it and then make a call to your backup (hopefully you have one!) who can begin corrective action. If you don’t have a backup, you at least have a heads-up to an issue at work and can go back to sipping a drink from a coconut.

Use a Real Email Address

Create an email address that can be dedicated to your mobile. For example, create a Gmail address and configure your smartphone for an audible notification on new messages to that address. Smartphone text messages don’t give you the entire story, only a few characters to let you know a host or service is having a problem. There may be more to it that is detailed in the Additional Information section of the alert. A disk utilization error of 95% may be something that can wait until you get back to your office to debug where as 100% would prompt you to boot your laptop to resolve as soon as possible. The only way to know is to have all the alert information in hand (literally).

Modify Alert Templates

The more information you put in the alert, the better chance you can delegate action. Modify the default alert templates to include more information that can help other people, such as a help desk, route tickets more effectively. Since Opsview has Nagios under the hood, all Nagios macros are available. (A complete macro list can be found on Sourceforge.

An example would be inserting comments using the Opsview UI on a host group or individual host, then changing the template to include the macro output for $HOSTGROUPNOTES$ or $HOSTNOTES$. Comments could include where to route tickets or links to documents to solve common problems that first level support can handle. If the issue to too complex, level one support will know which direction to escalate the ticket. The default template is located in /usr/local/nagios/libexec/notifications/com.opsview.notificationmethods.email.tt.

It’s a good idea to keep a backup of any changes you make since the file will be overwritten with each Opsview upgrade.

Set up Layered Email Profiles with Time Periods

The rub with any server monitoring system is no one wants a critical alert at three in the morning, but proper administration can’t be done without notifications. Administrators should embrace alerts, specifically warning alerts since they allow for proactive work to be done preventing critical alerts. That being said, no one wants a warning alert at three in the morning. Fortunately, each Contact in Opsview can have multiple Profiles which can have different layers of alerts. For your work email, create profiles for warning and critical to be sent 24×7. For your Gmail that your phone accesses, create profiles for warning alerts 8×5 and another profile for critical alerts 24×7. Be sure to name your profiles logically for easier administration, such as EmailPhoneWarning or EmailWorkCritical.

Send Alerts to Host and Service Administrators

An IT shop may have specific administrators, such as Windows or Linux admins. Windows administrators may not care to get alerts when Apache is down and Linux admins may not want to be woken up because a Windows server blue-screened.

Digging deeper into the Contact Profile, notifications can be set up for Host Groups and Service Groups. Configure each user to get alerts for their responsible services. Anytime someone gets an alert that they ignore because it is someone else’s responsibility, noise is created and alerts are assumed and disregarded, lowering the value of the entire monitoring system. If you want people to feel your pain, correct the issue and send out an email to work addresses that you were up all night dealing with your problem. It’s not a bad idea to show people that the system is working as it should, notifying only the responsible parties of critical issues (plus it gets you off the hook for coming in late the next morning).

Test New Checks Before Enabling Notifications

New checks are rolled out constantly in a changing environment. But new checks put immediately in production may produce false alarms that annoy other administrators and the help desk. Since Opsview includes built-in features to help monitor and trend check results, every addition should go through a testing period with notifications disabled. After a week, check the Service Graph to find the highest and lowest values to appropriately tune warning and critical thresholds.

Using the Alert Summary, you can determine if time periods should be used for a check. For example, a service may become unavailable during a nightly backup. The check_interval must remain the same, but checks need to be suspended for two hours each night while the backup occurs. You will be able to confidently tune the time period rather than make an uninformed guess at a black out range. Making accurate adjustments before a check “goes live” with notifications enabled greatly reduces unnecessary alerts and allows administrators to maintain faith in the system.

Find out more about Opsview alerts.

Freeware IT monitoring tools are used by thousands of organisation worldwide however using them to monitor complex network, server and application installations can be quite a challenge. This blog post takes the basic capabilities of one such tool, Nagios® Core, and shows how you can scale it with Opsview for use in enterprise environments.

Distributed monitoring

Building and managing a complex distributed monitoring environment with Nagios Core is no mean feat. With Opsview you get distributed monitoring that’s easy to setup and simple to maintain. You can monitor your devices and applications from a central location and grow the system without growing the monitoring complexity.

Slave server clustering

Opsview can automatically load-balance across multiple slaves and reallocate monitoring duties if a slave server fails, giving you high availability and scalability without additional overhead.

Example Clustering Model

Master server clustering

Management of Opsview is performed on a single master server, however master servers can be clustered giving you the high availability and redundancy needed for mission critical monitoring.

Separate database server

Opsview can be run on a separate database server so you can move intensive reporting activity to a dedicated machine and fine tune the server for better performance.

Efficient configuration UI

Nagios Core is capable of monitoring thousands of devices, but maintaining configuration on expanding systems can quickly become a problem. Opsview handles this with an easy to use interface and middleware layer which tackles the complexity of configuring individual software components.

‘Single pane of glass’ monitoring

Unlike Nagios Core where data may be gathered from a number of systems and presented in different ways, Opsview’s intuitive web interface displays all your monitoring information in one place, with a top down view on system status. Devices and applications can be easily grouped by business process and their status displayed using simple ‘traffic lights’ so you can easily see the health of critical and non-critical groups. This makes monitoring and maintaining large, complex systems less time consuming and more efficient with a scalable architecture to cover all your systems and locations.

Opsview's Host Group Hierarchy View

Distributed alerting

Slave servers monitored by Opsview can handle their own notifications, allowing autonomy if communication is lost between master and slave servers. Alerts can be sent by the Master server or slave server by email / sms so you’re always in touch with the health of your system, no matter the location or your systems.

Automated APIs

Opsview APIs speed up system configuration by automatically populating and updating host information saving you time and effort as your system grows.

Example use cases for Opsview's RESTful API

SNMP trap processing

Nagios Core has no native support for SNMP trap processing. Opsview’s SNMP engine accepts incoming traps, analyses the data and decides how to handle them. In-built SNMP discovery allows SNMP objects to be detected and monitored easily and rules can be configured through the management UI.

Notification profiles

With Nagios Core you can be inundated with monitoring information, not all of it useful. In Opsview you can set-up notification profiles so the right people get the right information at the right time. Only want to know about email server status during business hours? No problem. Need SMS alerts about your webstore? It’s covered. Notification profiles can also combined with Opsview’s service desk module to automatically assign support tasks to engineers, helping streamline incident management.

Many freeware IT monitoring tools are great but using them to manage complex systems can be a real challenge. It can also be unforgiving on anyone less than expert in configuring the system with mistakes being punished by a complete stop in monitoring activity.

Distributed monitoring

Opsview takes the complexities of its core engine, Nagios® Core, and makes distributed monitoring simple. All management is performed on a single master server and communication with slaves is handled by Opsview’s middleware layer. Provision is included for geographically diverse monitoring and to cope with potentially unreliable WAN connections between servers.

Host attributes

A feature you won’t find in Nagios Core, host attributes help simplify configurations by allowing you to create multiple services based on a set of pre-defined attributes. You can assign one or many attributes to a host and set service checks to use the attributes for a host to then create multiple services for monitoring.

Keywords

Opsview’s keyword function gives you a flexible way of grouping hosts and services. You can tag devices, business processes and applications giving you a convenient way of seeing the status of the groups, e.g. critical IT systems, network circuits or business users and customers.

Cloning capability

Chances are when you’re configuring or adding devices and services to your network many of them will be quite similar. To save time you can simply choose to clone an existing device or service monitored with Opsview and add it to the network.

SNMP discovery

Nagios Core provides support for SNMP via its plugin project, but it doesn’t provide support for processing SNMP traps. Opsview does this automatically. A powerful processing engine accepts incoming traps, analyses the data and then decides how they should be processed. In-built SNMP discovery also means SNMP objects can be detected and monitored with ease without the need for human intervention.

Notification profiles

Opsview helps you avoid information overload by easily creating complex business rules that define who gets alerts, how they get them and why. Combined with the Opsview service desk module you get a powerful notification tool that helps speed up mean time to repair and streamline workflows.

Configuration UI

Configuring and maintaining a system with Nagios Core can become difficult the bigger and more complex the monitoring environment gets. Opsview’s configuration UI means you don’t need to be a Nagios Core expert to get your monitoring up and running. All the software processes are kept ‘under the hood’ and presented via an intuitive interface so you can see the information that makes a difference to your business without getting caught up with software.

APIs for Automation

Opsview includes automated APIs for configuration, monitoring and notification which makes system set-up pain-free and scaling simple. The APIs also make integrating with other IT Management tools easy.

SLA reporting

Opsview’s Reports Module can automatically generate custom reports in line with business requirements. If you have to produce regular reports for your management or customers, this module will save you hours by generating the reports you need when you need them. The reports can be sent out automatically on email in PDF, HTML, Excel, ODT or XML to your chosen distribution list.

Service desk integration

When you integrate your service desk with Opsview you get a powerful tool for automating incident reporting. Tickets can be created in your system based on alerts generated by Opsview, saving time and freeing up resources. Out of-the-box support is included for Service-Now.com, Bestpractical’s Request Tracker and Atlassian JIRA.

Opsview Community 3.11 - Free distributed monitoring

One of Opsview’s great features is distributed monitoring, which we’ve had for over 5 years now. From the web user interface, you can assign hosts to a slave system and Opsview will take care of all the configuration work for you: from the slave configuration files, to the slave results sent to the master, to the master configuration with freshness checking.

We do all the system integration work, so you don’t have to.

However, there are some limitations in our chosen technologies. We use NSCA, which is the most common method in the Nagios® world, and while we’ve made improvements to it that have gone back upstream, there are some baked-in limitations:

• Only the first 511 bytes of plugin output was returned to the master, limiting the usefulness of the information you could display
• Only the 1st line of data was returned, meaning you had to cram output together
• NSCA communication used fixed size packets which were inefficient
• While results were sent, Nagios would wait for completion, introducing a bottleneck
• If there was a communication problem with the master, results were dropped

Sometimes to move forward, you have to leave the past behind.

So we did that – we ripped out NSCA from Opsview’s slave communications and we’ve addressed every one of these limitations – and added a few nice extras too!

We’ve chosen NRD (Nagios Result Distributor) as our core technology. This is a library, written by one of our partners, CAPSiDE. There are many reasons we chose this, but the top four are:

• It is based on perl, which is our language of choice
• It has taken the test suite we developed for NSCA and enhanced it, demonstrating a mature approach to code development
• The client and server code is a thin shim over the libraries, which means you can easily create your own clients
• We have a good relationship with CAPSiDE and they have given us access to their code repository

We’ve spent some time understanding the core NRD code, enhancing it, fixing some issues and adding in some great new features. CAPSiDE have also released it on CPAN for wider consumption.

So Opsview’s new process for sending results from a slave is:

A couple of other amazing features we’ve squeezed in:

• A known Nagios limitation is the named pipe to submit results. We’ve overcome this by writing directly to the checkresults spool directory – this reduces a Nagios processing cycle on the Opsview master
• We’ve implemented transactions in the results, so if the client has a failure communicating to the server, the client will back off and retry again in 5 seconds. This guarantees you do not have duplicated results
• The nrd daemon on the master will dynamically add more servers as workload increases, thanks to the features of Net::Server
• As all communication between master and slaves is over a tunnelled SSH session, we’ve updated our Opsview check scripts to restart these tunnels if the slave is exhibiting communication errors

With all this extra capabilities, you would think there is a cost in performance. But in fact, our testing shows that performance has got better!

(Based on sending 2016 results in a single transaction over an SSH tunnel from a slave to a master. Times measured on the client.)

This shows that we are getting an average 62% improvement in all aspects of slave communication back to the master!

We are thrilled we’ve added this major new functionality into Opsview and have taken distributed monitoring another huge step further over any of our competitors.

But the best thing is: this is available immediately with our Opsview Community 3.11 release. Install the VM, add a slave and you will get this new architecture setup as part of the process. And if you are an existing Opsview user, you get a silky smooth switch-over. We’ve done a lot of testing to ensure that as part of the upgrade, Opsview will automatically switch any slaves to this architecture and start sending results in the new NRD way.

We’ve just fixed a bug in Nagios® which an Opsview user had raised to us. A change made to Nagios in version 3.2.2 caused an issue where service alerts were being raised in the nagios.log file for every result that came back from a host that was down. This had the impact of adding lots of extra alerts that were overwhelming Opsview’s event views.

To reproduce the problem in Nagios 3.2.3:

1. Create a host with 2 service checks
2. Let this run normally
3. Shutdown the host
4. The first service check will notice the state change and set the host to be checked. It will go into a SOFT state and the service will go into a check attempt of 2 and continue into a hard state correctly
5. The 2nd service check will see that the host is DOWN and force a hard state failure with check attempt 1 of a maximum 4. However, this hard state change did not set the last_hard_state flag correctly, which meant every subsequent check was considered to be a new hard state failure and hence a SERVICE ALERT was raised every time in nagios.log

This took a long time to track down, but we’ve found the problem and fixed it. Our fix is pushed to Nagios already.

While this bug is annoying, we’re upset that this had an impact on a customer system. We make it our principle to keep as up to date with Nagios as possible because Opsview is a shallow fork of Nagios – we make only the changes that are necessary to support our customers and we push our changes back upstream where we can.

We’ve developed a lot of trust with our users – we make the upgrade process for Opsview as easy as possible because we want all our users to get to the latest version (in fact, we’ve just had one user update their Opsview from 4 years ago, right up to the latest version, going through over a hundred database changes!).

One thing we do to make sure our systems work as expected, is to continuously test our latest versions of Opsview. We use Hudson to test Opsview on every change – currently this runs 5269 individual tests, taking 1 hour 46 minutes.

We want to bring this level of quality assurance to Nagios – included in our fix is a test case that checks exactly this issue. Running tests on Nagios will now show that this problem is fixed forever and our nightly builds of Opsview includes these too.

So now everyone can sleep easier knowing that this problem is never going to happen again.

Please note: Nagios, the Nagios logo, and Nagios graphics are the servicemarks, trademarks, or registered trademarks owned by Nagios Enterprises

The JNRPE server provides an open source Java implementation of the Nagios Remote Plugin Executor (NRPE). This is much more efficient for performing JMX checks than regular NRPE as you only need to start one JVM rather than a JVM instantiation per check (as performed by check_jmx invoking java -jar JMXQuery.jar).

However this efficiency gain comes with a few compromises, it doesn’t handle composite data as well as other variants of check_jmx and doesn’t support performance data. It would be nice to see some community action to produce a definitive check_jmx.

Luckily with Opsview, you can work around the lack of performance data using the map.local file so you can get performance graphs to assist with correlation or view rate of change over time.

In JNRPE version 0.6.3 there is also an integer overflow bug in the check_jmx base plugin.

The maximum value for a Java Integer is 2^31 – 1 (or 2,147,483,647) – if a number exceeds this, as an Integer is a signed number, it will wrap around to a negative number. For example on a JVM set with a 5GB heap, we saw the following output:

JMX OK - HeapMemoryUsage.used is -2037300184

Amusingly with the initial warning/critical threshold values set too low, this caused the state to flap between OK (when the heap was larger than a 32 bit number and hence negative) and CRITICAL after garbage collection had brought it beneath 2GB.

This has been raised with the JNRPE project as issue 3131380 – though unfortunately having raised it without signing in, I now can’t attach the following simple patch to the parseData method.

Index: CCheckJMX.java
===================================================================
--- CCheckJMX.java (revision 243)
+++ CCheckJMX.java (working copy)
@@ -334,7 +334,7 @@
- private int parseData(Object o)
+ private long parseData(Object o)
{
if (o instanceof Number)
- return ((Number) o).intValue();
+ return ((Number) o).longValue();
else
- return Integer.parseInt(o.toString());
+ return Long.parseLong(o.toString());
}

NRPE is great for getting plugin information from a remote host. We wanted to use it to get passive data regarding events, such as syslog entries that SEC had highlighted. This meant we needed two things: multi-line support and larger amounts of output.

Multi-line is already in NRPE 2.12 – this was added by Matthias Flacke last year. However, the limit for data is 1K.

We wanted to be able to bump that figure up to 16K. There’s a common.h variable which is called MAX_PACKETBUFFER_LENGTH which is set to 1024. We found we could increase this value and then more data was returned. But there were two problems with it:

• it broke backwards compatibility
• it increased the size of each packetk

The 2nd had an impact on the network. Instead of 1K packets being sent between client and server, we now got 16K packets sent, even if the data contained was small.

The first was worst: it meant you needed to update the client (check_nrpe) with the server (nrpe) at the same time, otherwise you’d get lots of NRPE errors in Nagios with only one change.

So we’ve designed a compatible way: we’ve added a new packet type called RESPONSE_PACKET_WITH_MORE.

The idea is that check_nrpe will see if the packet returned is of the type RESPONSE_PACKET_WITH_MORE. If so, it will read subsequent packets and append that to the existing data, until it gets a RESPONSE_PACKET. So to read 16K worth of data, check_nrpe reads 16 x 1K packets. Of course, only updated nrpe daemons will send this, so this remains fully backwards compatible with existing nrpe daemons.

The patch is here. We’ve also cleanup up some of the graceful_close calls.

Now the process to update your NRPE agents would be:

1. update the central check_nrpe, then
2. update your agents at your leisure

And you won’t get any alerts during this period!

Note: during testing, we found that the limit for returned data from some linux kernels was 4K, even though nrpe was coded with 16K as the limit. This is due to kernel limitations in using pipe() for the interprocess communication.

In our continual task to try and speed up Opsview, we found a bug in NSCA’s handling of aggregate writes when run in –single mode.

The specific failure scenario is this:

1. NSCA and Nagios are told to start up
2. A send_nsca request is received by NSCA before Nagios has created the nagios.cmd command pipe
3. NSCA tries to write to open the command file, but sees it is not there
4. NSCA opens the alternate dump file instead

Now when Nagios does create the nagios.cmd file, NSCA uses that … unless aggregate mode is on and daemon mode is –single. In this case, it continues to use the alternate dump file, thus Nagios doesn’t see the results from the slaves.

Here’s the patch, which we’ve also added into our source for Opsview.

As we are very keen on good testing, we’ve managed to recreate the failing behaviour in a test script. You also need a test configuration file and a patch to the test framework. If you run this test, it will show the error and then after the patch is applied, the test should pass.