With the likes of cloud computing and virtualisation starting to become staples for today’s business, IT environments are continuing to grow in complexity. Furthermore, there is growing pressure on many organisations to reduce the environmental impact of their IT systems.

In response to these developments, organisations need to change the way they manage and therefore monitor their IT infrastructure. For example, with the advent of virtualisation, organisations now have a fundamentally different infrastructure platform from which they are running business systems. This in turn requires a different monitoring approach.

If organisations can’t adapt their physical world approaches to monitoring, they could find that they aren’t made aware of systems problems until users start complaining about downtime.

When it comes to cloud they need to consider how they are going to monitor both the performance and use of cloud services. Finally, with it being unlikely that organisations will move all services to the cloud they need to find ways to monitor environmental factors such as the temperature of data centres and the CO2 emissions of IT.

This data will not only allow them to find ways to reduce energy costs, but also report back up the organisation when it comes to the environmental targets that have been set.

At the same time as being presented with an increasingly complex environment to monitor, IT departments are under pressure to dramatically reduce the cost of day to day operations. The result is a conundrum that requires new ways of thinking to solve.

Taking The Pain Out Of Virtualisation

Virtualisation has been utilised by many organisations looking to improve operational efficiency. Thus, with physical and virtual machines now present in many businesses, monitoring and managing both effectively is vital.

When implementing and monitoring virtualisation, businesses face a number of challenges – all of which add to the complexity of IT monitoring. Virtualisation allows you to improve server utilisation. Yet, when you increase the utilisation of a previously under-used server, it can be difficult to know how that server will manage the increased load.

Therefore, monitoring the performance of the VM host and all the virtual machines running on it is imperative otherwise organisations could find they suffer from performance slow downs or worse still, downtime. The problem is virtualisation doesn’t behave like, or conform to the same rules as physical hardware and therefore often traditional approaches to monitoring both infrastructure and applications don’t meet the grade.

You often find that the VM itself can be monitored, but there is no insight into what’s going on within it or into the applications it’s hosting.

A further challenge for a lot of organisations is staying on top of the growing sprawl of virtualisation, due to the ease of creating virtual machines. To date, this is something that many have struggled with using older proprietary IT monitoring and management tools.

To combat these potential issues, organisations need to update their approaches to IT monitoring. They need to use tools that can provide insight into virtualisation and that enables them to understand how every application on every virtual machine is running, what problems could occur and how they can be remedied.

Without this investment in IT monitoring, virtualisation will not bring the ROI expected and the increased complexity could in fact result in ongoing performance issues.

Up In The Air – The Cloud Effect

The acceptance of cloud computing has increased over the past few years. Now, the terms public, private and hybrid are understood by the majority of organisations, with data centre association Afcom citing that more than 70 per cent of UK businesses are already implementing cloud, or seriously considering it.

Adopting a cloud model allows businesses to move some applications and services off-premise, resulting in reduced costs, with less IT equipment required on-premise. However, cloud does potentially present a number of challenges when it comes to monitoring the overall performance of an organisation’s IT infrastructure.

With this in mind, organisations should start thinking about how they are going to monitor and assess cloud performance in the future, in order to guarantee IT performance. The difficulty at the moment is that there are not many standards around cloud when it comes to moving applications between public and private clouds, and consequently monitoring them.

Therefore, before choosing a monitoring tool, organisations need to ensure that the solution has the flexibility to adapt to any future changes once standards do eventually emerge.

Another challenge organisations should be looking out for as they begin to use cloud-based services, is that they could experience cloud sprawl. Although one of the benefits of the cloud model is that users can buy services on a pay-as-you-go basis, organisations will need to make sure they are controlling cloud deployments, as it can be very easy to continue paying for cloud services long after they have finished using them.

This is for the simple reason that people often forget to tell the cloud provider that they have finished using the service and therefore keep getting billed. Ultimately, businesses will need to have a consolidated view of all cloud services meaning they will then be able to monitor how much these are being used. This way, they can make sure they only pay for what they are actually using rather than paying for what they have deployed.

Going Green Without Feeling Blue – Green IT Made Simple

In addition to adopting new technologies or IT models to reduce costs, organisations are also under pressure to become greener. This pressure comes not just from the board as they look to achieve cost savings, but also through growing public and legislative pressure on them to reduce carbon emissions.

The UK Government has set a target of cutting CO2 emissions by 34% of 1990 levels by 2020. Businesses play a key part in this and the pressure is on to adhere to the new rules and regulations that aim to ensure we hit those targets.

The main challenge for many organisations however, is that they don’t know how much power is being consumed by their IT, especially as data centres are now often located away from an organisation’s main site. Organisations must be able to construct a better picture of energy usage in their data centres – building environmental factors into monitoring to identify areas where further energy savings can be made.

For example, organisations need to be able to tell when and where they can power down servers that are not in use. They also need to monitor data centre temperature very carefully – ensuring cooling systems are working efficiently and keeping their servers at optimum temperature.

Furthermore, IT monitoring can also help organisations build a picture to help them decide which old systems can be decommissioned if they are not performing sufficiently. Being green does not need to be difficult. The right tools will indicate the right processes to improve your environmental credentials without the need for more investment or even more IT complexity.

As the IT landscape continues to change, it is becoming increasingly clear that the monitoring tools and techniques that worked in the past simply don’t suit the modern environment. More flexibility and agility is needed in IT monitoring, while reducing the overall cost. Overall, IT monitoring can help organisations improve their business and environmental performance. The challenge is now to ensure they have the right tools and techniques in place to do this.

Apache Solr is an open source enterprise search service from the Lucene project. Solr is written in Java and runs as a standalone full-text search server within a servlet container such as Tomcat.

Like any service or component in your architecture, you’ll want to monitor it to ensure that it’s available and gather performance data to help with tuning.

In this post, we’ll look at how we can monitor Solr, what performance metrics we might want to gather and how we can easily achieve this with Opsview.

A check list for service checks

Solr is built on Lucene so follows the same layout, an index contains documents that are comprised of fields. As part of the search service value add over Lucene, Solr provides a number of useful ways of obtaining health status / monitoring metrics:

1. Health-check status using the /admin/ping handler
2. The admin statistics page /admin/stats.jsp (XML styled with XSL)
3. JMX MBeans

The list of applicable checks could be defined by whether it is a health check or a data gathering check – but this would lead to a lot of overlap. Instead the list is divided into the checks that can be performed remotely (without an installed agent on the server) and those that are best performed locally to the Solr server.

Remote (agent-less) checks

What should we look for over the network?

Firstly we can have a host-level check which may perform a network level ping. Next we can check TCP connectivity to the servlet container port and then make an HTTP GET request to the Solr ‘front page’ and check for a known string (e.g. Welcome to Solr).

Now we’ve made it up to the application layer so can start to perform Solr specific checks.

Items to monitor may include (delete as applicable):

1. Ping status
2. Number of docs
3. Number of queries / queries per second
4. Average response time
5. Number of updates
6. Cache hit ratios
7. Replication status
8. Synthetic queries

Agent-based checks

Installing an Opsview agent on the Solr server means we can run additional checks over NRPE (Nagios Remote Plugin Executor). This could be operating system level checks such as memory/disk utilisation or CPU load, or the following:

1. Java servlet container process is running
2. JMX checks e.g. heap memory or custom MBeans
3. File age
4. Log parsing for exceptions

The Solr wiki describes how to configure JMX support: http://wiki.apache.org/solr/SolrJmx.

Opsview configuration

For the rest of this article you’ll need to have Opsview installed (or the Opsview VMWare appliance) and have completed the Quick Start.

Solr-specific Plugin

Install the Solr plugin at https://github.com/rbramley/Opsview-solr-checks into /usr/local/nagios/libexec/

The check_solr plugin was developed using Perl, so that it could be contributed back to Opsview. It requires the CPAN XML::XPath module (sudo cpan -i XML::XPath).

The plugin includes usage instructions, check_solr -h which can also be viewed in Opsview by selecting the ‘Show Plugin Help‘ link beneath the Plugin drop down (see Figure 1). The -u option can be used to specify the URL path for multi-core set-ups.

Service check setup

Figure 1 gives an example of a service check configuration.

Figure 2 shows the agentless service check group with plugins and their arguments.

Host configuration

Figure 3 shows a simplistic host setup with a ping check.

Figure 4 is an extract from the Monitors tab, where we select the checks we want performed for the current host.

Viewing output

The check results shown in Figure 5 are visible by navigating through the host group hierarchy.

If you click on the graph icon of Solr Cache Hit Ratios this will drill down onto the graph shown in Figure 6.

Clicking on the graph icon for Solr Avg Response Time – standard will take you to the graphs in Figure 7.

If you shutdown Solr, then the check results will start to turn critical and show in red as per Figure 8.

Alternatives

There are a few other plugins available for monitoring Solr from Opsview, depending on your needs:

• http://code.google.com/p/nagios-plugins-shamil – provides ping, replication status and num docs
• http://code.google.com/p/solr-nagios-check – provides QPS, response time and num docs

Also, chapter 8 of the recently published Apache Solr 3 Enterprise Search Server book includes a section on Monitoring Solr Performance.

Summary

Using check_solr in conjunction with Opsview allows you to ensure that your Solr server is available and provides you with metrics that can help you tune your Solr configuration.

This can be complemented with additional agent-based operating system and JMX checks to give you a full picture view.

Opsview gives you lots of standard options for receiving status notifications, but with a bit of custom scripting you can also get status messages from Opsview and post them to a Twitter account. This could be useful if you want to keep all your status messages in one place with a timeline. Plus you can also set up an account for other members of your team to follow so they can also see the updates. However it’s probably best that the Twitter account is kept within your organisation unless you want the world to see the status of your servers!

This article tells you how to configure Opsview to send messages to Twitter. For configuring this you need to have access to the shell of your Opsview server.

To access Twitter from Opsview, you’ll need to install Tweepy. Tweepy is a Python library for accessing the Twitter API. It requires Python 2.5 or later. If you need to install Python 2.6 you can follow the instructions on this blog.

# wget http://dagobah.ftphosting.net/yum/smartfile.repo -O /etc/yum.repos.d/smartfile.repo

Then you can use YUM to install python26 package:

# yum install python26 python26-distribute

To get a list of installed packages, do:

# yum list *python26* | grep -i installed

Installed Packages
python26.x86_64             2.6.5-6.el5   installed
python26-distribute.noarch  0.6.10-4.el5  installed
python26-libs.x86_64        2.6.5-6.el5   installed

Install Tweepy

/usr/bin/easy_install-2.6 tweepy

Next you’ll need to authenticate Twitter from the command line. This blog tells you how.

At this point you’ll need to create a new Twitter account which will be used to post the notifications.

Make sure you enable Tweet Privacy under Settings, in the Twitter account, so that posts are not made public (unless you want them to be!)

After creating the Twitter account, you’ll need to register a new client app with Twitter.

Select “Connect the app to your Twitter account” with a minor change in the first line.. write python26 instead of python

#!/usr/bin/env python26

Import tweepy

CONSUMER_KEY = 'paste your Consumer Key here'
CONSUMER_SECRET = 'paste your Consumer Secret here'
auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth_url = auth.get_authorization_url()
print 'Please authorize: ' + auth_url
verifier = raw_input('PIN: ').strip()
auth.get_access_token(verifier)
print "ACCESS_KEY = '%s'" % auth.access_token.key
print "ACCESS_SECRET = '%s'" % auth.access_token.secret

Again, in  “Create the command line script”, change to python26. Don’t forget to add your own keys.

#!/usr/bin/env python26

import sys
import tweepy

CONSUMER_KEY = 'paste your Consumer Key here'
CONSUMER_SECRET = 'paste your Consumer Secret here'
ACCESS_KEY = 'paste your Access Key here'
ACCESS_SECRET = 'paste your Access Secret here'
auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_KEY, ACCESS_SECRET)
api = tweepy.API(auth)
api.update_status(sys.argv[1])

Then “Send a test tweet from the command line” and if the test is successful, do the following:

1.	Move the script to /usr/local/nagios/libexec/notifications/.

(call the script ‘connect_twitter’, hence script is at /usr/local/nagios/libexec/notifications/connect_twitter)

2.	chown nagios:nagios /usr/local/nagios/libexec/notifications/connect_twitter
3.	chmod +x /usr/local/nagios/libexec/notifications/connect_twitter

Create another script to be used by Opsview:

/usr/local/nagios/libexec/notifications/notify-by-twitter

The script:

dt=`date '+%H:%M %d%b%y'`
if [ -n "$NAGIOS_SERVICEDESC" ]
then
msg=`echo @$NAGIOS__CONTACTTWITTERID Time:$dt. $NAGIOS_SERVICEDESC @ $NAGIOS_HOSTNAME,$NAGIOS_HOSTADDRESS is $NAGIOS_SERVICESTATE. $NAGIOS_SERVICEOUTPUT | cut -c 0-140`
/usr/local/nagios/libexec/notifications/connect_twitter "$msg"
unset msg
else
msg=`echo @$NAGIOS__CONTACTTWITTERID Time:$dt. $NAGIOS_HOSTNAME,$NAGIOS_HOSTADDRESS is $NAGIOS_HOSTSTATE. $NAGIOS_HOSTOUTPUT| cut -c 0-140`
/usr/local/nagios/libexec/notifications/connect_twitter "$msg"
unset msg
fi

Again,

chown nagios:nagios /usr/local/nagios/libexec/notifications/notify-by-twitter
chmod +x /usr/local/nagios/libexec/notifications/notify-by-twitter

Log in to Opsview as a user with admin privileges.

Go to Advanced then Notification Methods from the drop down menu.

On the top right, click Actions and then Create new Notification Method as following:

Next, go to Configuration on the Opsview Menu bar, then Contacts. Click on the contact you want to set Twitter alerts for. Click the Notifications button. Add your Twitter ID (this ID will receive the Tweets).

Click on Submit and edit Notification Profiles.

Click Actions, and create new Notification Profile for user.

Create the profile and submit changes.

Reload your configuration (click on Configuration Status, top right, then Reload Configuration).

Reboot one of your servers or stop a service which is added in Opsview. Note that Twitter allows 250 direct messages per day, do not cross this limit as tweets won’t come in after you exceed this number.

The Tweets should now start coming in!

Try this out for yourself by downloading Opsview Community edition and get our latest tips and updates by following us on Twitter!

It is often the case that hosts on your network will be similar, but differ in subtle ways depending on their exact purpose. For example some servers may have two internal disks whilst some may have just one, and some servers may have multiple ethernet cards where others do not. Opsview provides attributes to deal with situations like this, as they are a way of associating metadata with a host – here is an example of how to configure Opsview to use them.

Let’s assume we have a Linux host that has three ethernet cards – eth0, eth1 and eth2. Rather than create three service checks which reference these interfaces explicitly, we’ll configure a single Interface Throughput check that will be duplicated for each network card that a host has.

1. Create the attribute

First we’ll create an attribute called “NIC” so that we can store this metadata against the host. Go to Advanced -> Attributes -> Actions -> Create new attribute and enter a name of NIC, then save it.

2. Define the attribute for one or more hosts

Now that the attribute is known to the system, it can be applied to some hosts. In this example, we edit the Linux host, and under the “Attributes” tab, add three entries all with a label of NIC, with the values eth0, eth1 and eth2.

3. Create a service check which uses the attribute

Now that we have some metadata about the host, we can create a service check that uses it. The screenshot shows the configured service check – the “Multiple” prefix indicates that we have selected “NIC” as the value for Create Multiple Services under the Advanced section of the check configuration, and so at reload time Opsview will create three service checks, replacing %NIC% with each of the three values.

4. Apply the new check to the host

The check can either be applied directly on the Monitors tab of the host configuration, or using a Host Template. Once the check has been applied, reload Opsview and navigate to the host status. You should now see three services, one for each ethernet card that you have defined for the host.

5. Apply to more hosts

This new service check can now be applied to any number of hosts you wish, and you simply need to define as many NIC attributes as you need for that particular host.

This example really just scratches the surface of how host attributes can help you better organise your server monitoring. You can find out more about attributes on our documentation site and try it out for yourself by downloading Opsview Community Edition.

So you followed the steps in the previous post about enabling SNMP traps on ESX4. Now you probably want to pick those up by something useful. Opsview can be configured to handle the traps quite easily. Just follow the steps below and your server will be listening to those pesky traps. After that, you’ll need to write a couple of service check handlers in Opsview to make sense of the traps. More on that later. This post is just about picking them up.

This was done on an Ubuntu 10.04 LTS server. The steps are probably the same on Debian systems.

1. Make sure you have snmpd installed. If you don’t, install it! Easy as pie.

aptitude install snmpd

2. Edit /etc/snmp/snmpd.conf and uncomment “master agentx”.

3. Edit /etc/default/snmpd (or /etc/snmp/snmptrapd.conf on newer systems):

TRAPDRUN=yes
TRAPDOPTS='-t -m ALL -M /usr/share/snmp/mibs:/usr/local/nagios/snmp/load -p /var/run/snmptrapd.pid'
SNMPDOPTS='-u nagios -Lsd -Lf /dev/null -p/var/run/snmpd.pid'

4. Edit /etc/snmp/snmptrapd.conf and add the following lines:

traphandle default /usr/local/nagios/bin/snmptrap2nagios
disableAuthorization yes

Please note that this will make the server listen to and handle any SNMP traps it receives, regardless of source.

5. Restart snmpd and snmptrapd:

/etc/init.d/snmpd restart

6. Edit the /etc/sudoers file to allow Opsview to restart snmpd and snmptrapd:

nagios ALL=NOPASSWD:/usr/local/nagios/bin/snmpd reload

7. Test the permissions:

su - nagios
sudo /usr/local/nagios/bin/snmpd reload

8. Exit back to the root user and restart opsview-web:

/etc/init.d/opsview-web restart

That’s all. Opsview should now be able to handle traps sent to it.

This post summarises the work involved to successfully monitor your ESX environment using Opsview and the vSphere SDK for Perl.

Prerequisites

Ensure you have the following:

• 1 x Opsview Master
• 1 x support account http://www.vmware.com/support/
• Installation of library libssl-devel

Step 1

Navigate and download the SDK Libraries that correspond to your OS Archictecture from http://communities.vmware.com/community/vmtn/developer/forums/vsphere_sdk_perl

Step 2

On the Opsview master run the following commands:

opsview@opsmstpr01:~$ sudo tar -zxvf VMware-vSphere-Perl-SDK-4.1.0-254719.x86_64.tar.gz
opsview@opsmstpr01:~$ sudo vmware-vsphere-cli-distrib/vmware-install.pl
Creating a new vSphere CLI installer database using the tar4 format.
Installing vSphere CLI.
Installing version 254719 of vSphere CLI
You must read and accept the vSphere CLI End User License Agreement to continue.
Press enter to display it.

Accept the licence agreement and the default location to install the executable files [/usr/bin]

Step 3

Download and Install the nagios plugin check_esx3 from http://exchange.nagios.org/directory/Plugins/Operating-Systems/*-Virtual-Environments/VMWare/check_esx3/details to /tmp

Step 4

On the Opsview master as user nagios run the following commands:

opsview@opsmstpr01:/tmp$ sudo chown nagios:nagios /tmp/check_esx3
opsview@opsmstpr01:/tmp$ cp /tmp/check_esx3 /usr/local/nagios/lib
opsview@opsmstpr01:/tmp$ sudo cp /tmp/check_esx3 /usr/local/nagios/libexec/

Examples

On the Opsview master as the nagios user run the following tests for the VM server

./check_esx3 -H $HOSTADDRESS$ -u root -p password -l cpu
./check_esx3 -H $HOSTADDRESS$ -u root -p password -l runtime -s list
./check_esx3 -H $HOSTADDRESS$ -u root -p password -l io
./check_esx3 -H $HOSTADDRESS$ -u root -p password -l mem -s usage
./check_esx3 -H $HOSTADDRESS$ -u root -p password -l net
./check_esx3 -H $HOSTADDRESS$ -u root -p password -l runtime -s issues
./check_esx3 -H $HOSTADDRESS$ -u root -p password -l runtime -s status
./check_esx3 -H $HOSTADDRESS$ -u root -p password -l service
./check_esx3 -H $HOSTADDRESS$ -u root -p password -l storage -s path
./check_esx3 -H $HOSTADDRESS$ -u root -p password -l mem -s swap
./check_esx3 -H $HOSTADDRESS$ -u root -p password -l vmfs

On the Opsview master as the nagios user run the following tests for any Guest VM on the server

./check_esx3 -H [IP of ESX Server] -u root -p [Password] -N [Guest VM] -l runtime -s con
./check_esx3 -H [IP of ESX Server] -u root -p [Password] -N [Guest VM] -l cpu
./check_esx3 -H [IP of ESX Server] -u root -p [Password] -N [Guest VM] -l io
./check_esx3 -H [IP of ESX Server] -u root -p [Password] -N [Guest VM] -l mem -s usage
./check_esx3 -H [IP of ESX Server] -u root -p [Password] -N [Guest VM] -l net
./check_esx3 -H [IP of ESX Server] -u root -p [Password] -N [Guest VM] -l runtime -s cpu
./check_esx3 -H [IP of ESX Server] -u root -p [Password] -N [Guest VM] -l runtime -s issues
./check_esx3 -H [IP of ESX Server] -u root -p [Password] -N [Guest VM] -l runtime -s mem
./check_esx3 -H [IP of ESX Server] -u root -p [Password] -N [Guest VM] -l runtime -s status
./check_esx3 -H [IP of ESX Server] -u root -p [Password] -N [Guest VM] -l mem -s swap
./check_esx3 -H [IP of ESX Server] -u root -p [Password] -N [Guest VM] -l runtime -s tools

Step 5

Create an Opsview service check and host templates. For more information see our online documentation.

If you’re managing any web sites or web applications Selenium can be used with Opsview’s monitoring platform to ensure your customers get the very best user experience and increase your online conversion rates.

What is Selenium?

Selenium is a suite of tools to automate web application testing, it is also the 34th element in the periodic table. You can use the Selenium IDE to record web browser actions in the form of test cases. These test cases can be saved in a number of different formats and replayed when needed. The Selenium IDE is implemented as a Firefox extension.

What is Opsview?

Opsview provides comprehensive monitoring and management capabilities for a very wide range of web applications. Our Community Edition is free, easy to install and simple to use. More advanced features are available with Opsview Enterprise. Check out our screencasts

User Experience monitoring

In 2001, Zona Research reported that 30% of surfers would abandon a Web site if it took more than 8 seconds to load, creating the widely-quoted ´8-second´ standard. Ten years later, with wide adoption of broadband services users become frustrated if sites take more than a couple of seconds to load.

Monitoring your web applications from an user’s perspective is vital for ensuring they get the best experience and allows you to get early warning of any application performance issues. You can use the Selenium IDE to record common paths through your web applications and the have Opsview replay these paths on a regular basis. Opsview will alert if any URLs fail to load in a pre-defined timescale or if the entire path takes too long to complete. These checks can be replayed from multiple locations (Opsview Slaves) to give you a complete picture of how your customers are experiencing the applications.

End-to-end testing

Using Selenium to perform end-to-end web application testing ensures that the whole system is being monitored in concert. This can be combined with individual component checks to provide a coherent overview of system performance and health. The Selenium IDE can be used to record transactions performed through the web applications. These are then replayed on a regular cycle to ensure the transaction completes successfully in a given timeframe. By replaying key transactions all components and sub-systems that comprise the application are exercised to ensure correct operation.

Selenium in action

Ten uses for Selenium with Opsview

1. Measure how your applications are performing for end-users
2. Keep tabs on your shopping cart / order processing application
3. Check that your landing page forms are working correctly
4. Confirm that web site authentication is working
5. Check that you’re not getting any pesky database errors
6. Track your cloud hosting provider’s performance against their SLA
7. Make sure your boss can get to his web mail account
8. Be the first to know that Opsview’s web UI is offline
9. Ensure your load balancer is working correctly
10. Get alerted if your homepage has been hacked

What next?

Installation and configuration instructions can be found in Opsview’s documentation along with instructions for downloading the selenium plugin.

Our recommended approach is to configure Selenium on an Opsview slave server, either by updating an existing slave or by deploying a new one. Selenium can place an appreciable load on your server so we recommend allocating another 1GB RAM. For heavy Selenium usage you should consider allocating an additional CPU core.

Thanks

Thanks to Alan Wijntje @ Ziggo, Rohit Deshmukh @ Opsview and Steve Burt @ Opsview for their work on the Selenium plugin for Opsview and related documentation.

Hindsight is a wonderful thing.

In hindsight, Opsview would have always had access controls for objects at the role level – since roles also define which parts of the Opsview application you can get to, it would make sense to also put all the host and service objects into this definition.

(In our defence, we wanted to make it as obvious as possible for a contact when you were changing access information.)

The downside of our design decision many years ago is that Opsview administrators who have lots of their users – or contacts in Opsview terms – with the same sort of access and were having to change each user individually. This was painful and error prone if you had 40 “similar” users.

So Opsview Enterprise 3.12.0 now has access information in the role definition. You change access at the role level and it automatically affects all users of that role. Even better, we ensure that a user’s notification profile only has references to the objects they are allowed – change the role definitions and object references will be automatically removed from all aspects of that contact.

Now you can administer users in a much simpler way.

Hindsight – we wish we had more of it.

But the next best thing to hindsight – migratability.

Migratability?

The Opsview configuration database is normalised and models the data as it is. Some application designers like to use key-value pairs to describe their data, which is nice from an expandable point of view, but rubbish when it comes to actually accessing that data in a meaningful way. When it comes to modelling our data, we model it – we don’t “meta-model” it.

We care a lot about people’s data because we build up a trust with our users that they can upgrade and bring their platform right up to date with the latest versions of Opsview. So we spend a lot of time getting the upgrade scripts just right.

If you are upgrading to Opsview 3.12, here’s what the upgrade scripts do:

• for each contact, it will see if this contact’s role has access information applied
• if it hasn’t, it will use this contact’s access information to populate the role
• if the role already has access information (from another contact), then it will compare all the current roles with their access information and if there is a match, this role is used
• if it isn’t exactly the same, then a new role is created called “old role name – contact”

So for example, let’s say you have 4 contacts – John, Paul, George and Ringo – all using the “Liverpool administrators” role.

The first three have their access information the same, but Ringo has missed out the Production – Slicehost Servers host group he should have had access to.

After the upgrade, the first three would still have the role of Liverpool administrators, but Ringo would have the role of a newly created Liverpool administrators – ringo. You could easily tell from this that Ringo should belong to the Liverpool administrators, so you can set him to this. Otherwise, maybe he is different after all, so you can rename the role to Liverpool drummers.

This means that most of the configuration for this new functionality is done for you automatically and you can think of your users in their role groups, rather than individually. Another step to making Opsview easier to use.

This post is going to explore how we can raise application alarms from Grails applications into an enterprise monitoring solution such as Opsview. These are known as Passive checks, as opposed to Active checks where Opsview is actively polling to check the state of a service.

Let’s start off by describing the main building blocks that we’ll be using (from source to sink):

1. Grails – a modern agile Open Source web framework with a rich plugin ecosystem that uses Groovy, the Spring Framework, Spring MVC and Hibernate.

2. JAlarms – an Open Source framework for allowing Java developers to raise alerts from server applications (also available as a Grails plugin)

3. JSend NSCA – an Open Source Java API that shields us from the underlying NSCA data format – described here

4. NSCA daemon – the Nagios Service Check Acceptor, which allows applications to submit service check results to Nagios

5. Opsview – an Open Source enterprise monitoring solution built on top of Nagios.

This post assumes you already have Opsview set up, with a host & service check defined and that the service check can accept passive check results.

If not – see the Opsview quick start guide.

Opsview configuration

We’ll start by ensuring that Opsview is configured so that it will accept remote NSCA service checks (I strongly suggest you use firewall rules to restrict access) and the encryption method is compatible with JSend NSCA (as it currently only supports NONE, XOR and TRIPLE_DES; Opsview supports about 20 algorithms with DES as the default).

1. Edit /usr/local/nagios/bin/nagconfgen.pl (as this script recreates the nsca.cfg when Opsview reloads configuration). The following diff shows the changes between the new file and the original file (diff nagconfgen.pl nagconfgen.pl.bak):

3357,3358c3357
< #server_address=127.0.0.1
< server_address=0.0.0.0
---

> server_address=127.0.0.1
3518c3517
< decryption_method=3
---
> decryption_method=2
3593c3592
< encryption_method=3
---
> encryption_method=2

If you make Opsview reload the config – you’ll then see the new values in /usr/local/nagios/etc/nsca.cfg

2. Restart opsview (as we’re binding differently):

/etc/init.d/opsview restart

JAlarms

JAlarms 1.5.2 features the addition of the NagiosPassiveCheckChannel contribution (https://github.com/rbramley/jalarms-opsview) that I developed to integrate JAlarms with Opsview using JSend NSCA. Note that the channel was downgraded from JSend NSCA 2.0.1 (as used in the Github code) to 1.3.1_2 so that the dependency could be sourced from the Maven central repository.

NagiosPassiveCheckChannel requires the following settings:

• settings – JSend NSCA NagiosSettings class
• hostname – this must match the hostname as configured in Opsview
• sources – a map of JAlarms source names to Opsview service check names (to allow for multiple jAlarms channels).

Setting up Grails jAlarms for Opsview

Using the recently launched JAlarms plugin (courtesy of Hackergarten Mexico and Andres Almiray) – at the time of writing we’ll need to upgrade the version of jalarms from 1.5.1 to 1.5.2 and declare the JSend NSCA dependency.

1. “grails install-plugin jalarms“

2. Upgrade the jar to 1.5.2 (I built it from source)

3. Add the dependencies in grails-app/conf/BuildConfig.groovy:

Uncomment the repository:

mavenCentral()

Add the following dependencies: compile 'org.apache.servicemix.bundles:org.apache.servicemix.bundles.jsendnsca-core:1.3.1_2'

runtime 'org.apache.servicemix.bundles:org.apache.servicemix.bundles.jsendnsca-core:1.3.1_2'

4. Set up the Spring beans & configuration

For speed, I’ve included the configuration in grails-app/conf/spring/resources.groovy (ideally these would be per-environment in Config.groovy):

import com.solab.alarms.channels.*
import com.googlecode.jsendnsca.core.*

beans = {
  nagiosSettings(NagiosSettings) {
    nagiosHost = '192.168.8.8'
    password = 'changeme'
    encryptionMethod = NagiosSettings.TRIPLE_DES_ENCRYPTION
  }

  channel(NagiosPassiveCheckChannel) {
    settings = ref(nagiosSettings)
    hostname = '192.168.8.16'
    sources = [Test:'Alfresco Share connection']
  }
}

Adding an alarm call

The jAlarms Grails plugin injects a sendAlarm method into controllers & services.

For ease of demonstration we’ll create a controller that always sends an alarm message:

def index = {
  sendAlarm('Test application alert', 'Test') // msg, source
  flash.message = "Alarm sent"
  return
}

Combined with a simple GSP and we get:

If we look in Opsview, we see that the passive check has resulted in a critical state:

Hopefully that’s given you enough information to get started!

About the author

Robin Bramley is a Technical Architect at Ixxus and author of the LeanJavaEngineering blog

Opsview Community 3.11 - Free distributed monitoring

One of Opsview’s great features is distributed monitoring, which we’ve had for over 5 years now. From the web user interface, you can assign hosts to a slave system and Opsview will take care of all the configuration work for you: from the slave configuration files, to the slave results sent to the master, to the master configuration with freshness checking.

We do all the system integration work, so you don’t have to.

However, there are some limitations in our chosen technologies. We use NSCA, which is the most common method in the Nagios® world, and while we’ve made improvements to it that have gone back upstream, there are some baked-in limitations:

• Only the first 511 bytes of plugin output was returned to the master, limiting the usefulness of the information you could display
• Only the 1st line of data was returned, meaning you had to cram output together
• NSCA communication used fixed size packets which were inefficient
• While results were sent, Nagios would wait for completion, introducing a bottleneck
• If there was a communication problem with the master, results were dropped

Sometimes to move forward, you have to leave the past behind.

So we did that – we ripped out NSCA from Opsview’s slave communications and we’ve addressed every one of these limitations – and added a few nice extras too!

We’ve chosen NRD (Nagios Result Distributor) as our core technology. This is a library, written by one of our partners, CAPSiDE. There are many reasons we chose this, but the top four are:

• It is based on perl, which is our language of choice
• It has taken the test suite we developed for NSCA and enhanced it, demonstrating a mature approach to code development
• The client and server code is a thin shim over the libraries, which means you can easily create your own clients
• We have a good relationship with CAPSiDE and they have given us access to their code repository

We’ve spent some time understanding the core NRD code, enhancing it, fixing some issues and adding in some great new features. CAPSiDE have also released it on CPAN for wider consumption.

So Opsview’s new process for sending results from a slave is:

A couple of other amazing features we’ve squeezed in:

• A known Nagios limitation is the named pipe to submit results. We’ve overcome this by writing directly to the checkresults spool directory – this reduces a Nagios processing cycle on the Opsview master
• We’ve implemented transactions in the results, so if the client has a failure communicating to the server, the client will back off and retry again in 5 seconds. This guarantees you do not have duplicated results
• The nrd daemon on the master will dynamically add more servers as workload increases, thanks to the features of Net::Server
• As all communication between master and slaves is over a tunnelled SSH session, we’ve updated our Opsview check scripts to restart these tunnels if the slave is exhibiting communication errors

With all this extra capabilities, you would think there is a cost in performance. But in fact, our testing shows that performance has got better!

(Based on sending 2016 results in a single transaction over an SSH tunnel from a slave to a master. Times measured on the client.)

This shows that we are getting an average 62% improvement in all aspects of slave communication back to the master!

We are thrilled we’ve added this major new functionality into Opsview and have taken distributed monitoring another huge step further over any of our competitors.

But the best thing is: this is available immediately with our Opsview Community 3.11 release. Install the VM, add a slave and you will get this new architecture setup as part of the process. And if you are an existing Opsview user, you get a silky smooth switch-over. We’ve done a lot of testing to ensure that as part of the upgrade, Opsview will automatically switch any slaves to this architecture and start sending results in the new NRD way.