The first part of a series of posts on Grails and Hudson leading up to a presentation at the London Groovy & Grails User Group. Subsequent instalments will include testing (unit, integration, functional), test coverage, automatic war deployment and monitoring Hudson with Opsview Enterprise.

Grails has a rich plugin eco-system with over 400 hundred plugins – so it’s easy to miss something useful. If you’re serious about software craftsmanship, then using static code analysis tools should be part of your quality regime as it gives further insight into the code base (and if you insist, yes it’ll help with your Technical Debt management).

CodeNarc provides static code analysis for Groovy and the CodeNarc plugin for Grails allows you to perform this analysis with the “grails codenarc” script. Behind the scenes this uses the CodeNarc ant task and settings from grails-app/conf/Config.groovy and produces an HTML report by default.

Until recently, if you used the codenarc target within a continuous integration server such as Hudson – then the HTML report would be generated and sit in the workspace waiting for a diligent developer to check it. You can imagine how often that happens in practice with all the other demands of a project!

However, I’ve now integrated the CodeNarc XML output with the Hudson Violations plugin so that an overview trend line is shown against the Hudson job. Then the team quickly fixed the violations…

You can also get a breakdown by priority:

And in-context views of the violations so you know what to fix:

This is how you do it…

Grails config.groovy
codenarc {
reportName = 'target/test-reports/CodeNarcReport.xml'
reportType = 'xml'
// any further settings like maxPriority1Violations=0
}

Hudson

Set up your Grails build step – normally you’d add the ‘codenarc’ target:

The codenarc.properties file can be used to configure specific exclusions, the location of this file can be passed in as a system property (shown above).

You need to have installed the violations plugin (Manage Hudson > Manage Plugins > Available and search for Violations). As this is a recent addition, I’m using a patched version of the Violations plugin, though the patch has been integrated into trunk (I’ll update when it is released). Configure violations:

Note the ‘Faux project path’ – you may need to set this to get an in-context view working properly due to path (e.g. if your code is checked out to workspace/trunk)

I also had a contribution to CodeNarc accepted at the weekend to add aninlineXml report type – this will, with a minor tweak to the CodeNarc parser, allow the Hudson Violations plugin to give the rule description on the pop-up message.

The Nagios Plugins project recently released a new version. Amongst the changes is a new feature which we added for a customer. The requirement was to measure the rate of change for SNMP counters. The standard check_snmp plugin is great at getting information, but only at a specific moment in time. For some things, you want to check and alert on the rate of change. There’s a lot of interesting metrics that you can get from SNMP which are Counter32 or Counter64 values. An example is IP-MIB::ipInAddrErrors.0. This counts the number of packets that are discarded due to invalid IP addresses – probably due to network errors or an attempt at infiltrating your network.

We could have met the requirements by writing a new plugin. But we wanted to this in the core plugins because it is useful functionality for everyone. To enable this feature, we had to introduce some library functions so that plugins could save state information. After some discussions with the Nagios Plugins core team, we designed the library functions for inclusion in the Nagios Plugins C library. There have been attempts before, but the interface was unnecessarily complex. We believe in designing things to work yet keeping it simple – we make these design decisions every day!

But there’s more! Having the plugin is only half the story because we wanted people to easily configure it using Opsview’s web interface. Opsview has the concept of a “service check” which defines the things you care about and how to check them. On this page, you can select SNMP polling, do an SNMP walk to gather all the available SNMP metrics and then you just click to select it. Nice!

You can see the configuration in our screencast.

Even if you don’t use Opsview, you can take advantage of this new feature by downloading the latest Nagios Plugins. We continue to work to improve our open source base. Enjoy!

All enterprises depend on reliable servers, network devices and business applications. Any downtime hits your bottom line. To ensure maximum IT performance across your business, you have to identify and resolve problems before they impact the user experience or security of your data.

To diagnose reliability and performance problems on complex networks you need visibility of system events combined with the ability to spot trends and exceptions. For example, without proactive IT monitoring how can you tell whether poor application performance is related to network congestion, database performance or a high number of users?

Monitoring network events as they happen!

Powerful graphing tools

Here are five reasons why you need better network and application visibility:

1.    To maximise your return on network investments for business application delivery
All businesses invest heavily in setting up and maintaining their IT infrastructure whether it be physical, virtualised or in the Cloud. Effective capacity planning allows you to target your spending wisely and track your return on investment. IT monitoring gives you instant access to the data you need for trend analysis and provides data warehouse capability suitable for enterprise reporting and capacity planning.

2.    To anticipate and resolve issues before they’re problems
Knowing when there is a problem with your IT systems can be useful but anticipating problems before they occur is the real value of system monitoring. IT monitoring gives you the visibility needed to spot performance and capacity issues before they have an impact on your systems. It can also process alarms from network, storage, power and cooling infrastructure ensuring you can mitigate the effects of failing hardware.

3.    Make your staff more effective
The biggest asset for any business is its staff, also usually this is the biggest cost. Adoption of effective IT monitoring tools shifts the focus of your staff from fire-fighting to forward-planning. A project-based approach is more time efficient and allows you to focus the talents of your staff more effectively.

4.    To ensure quality and service levels
Maintaining quality of service requires visibility. Performance issues can remain hidden to the business while quietly eroding the confidence of customers and partners. IT monitoring exposes current problems and gives you the tools for long terms analysis and reporting. It provides service level reporting, providing clear evidence that you’re meeting the required service levels.

5.    Move things forward!
System monitoring is the cornerstone of IT management best practice. By adopting IT monitoring you’re making immediate efficiency gains and you’re laying the foundation for even greater levels of automation, scalability and resiliency.

This post is based on research we’ve undertaken to develop a pilot mash-up style, charting dashboard for our monitoring solution, Opsview Enterprise. However the concepts we discuss could be used when building a dashboard that displays information from many other enterprise solutions. The assumption we make is that the reader is familiar with Java and the Google Web Toolkit (GWT). For more information about GWT and the other libraries used in this blog please see the Resources section at the end.

Introduction

The Opsview Enterprise platform is very flexible and allows us to extract monitoring information via a REST based API. This API returns monitoring data as JSON objects or XML so it is easy to integrate with many UI frameworks. In this example we will develop a mini portal-like application that displays status for different host groups in one of its portlets.

The client will be using the ‘GXT’ framework that is based on GWT for the user interface and some Java libraries for HTTP communication and JSON parsing.

Using the REST based API to talk to Opsview Enterprise

Let’s first have a look at how you can call Opsview Enterprise from a browser to collect monitoring data. In this case we can use the following URL to fetch host groups:

http://<hostname>/opsview/api/status/hostgroup

If we try using this URL directly from a browser it will not work, a 403 access denied will be returned even if we login with correct credentials. A couple of extra headers are needed for the authentication to work; we will look at them later on.

To get around this problem we must first login to Opsview Enterprise the normal way via the Opsview Enterprise UI and then hit the URL:

https://<hostname>/opsview/status/hostgroup

This is basically the first URL you get to. Now you will have a session in the browser so you can test the API URL again.

It will return JSON looking something like this:

{"hostgroup":{
"summary":{"handled":782,"unhandled":60,"total":842,
   "service":{"ok":558,"critical":206,"handled":747,"unknown":29,
"unhandled":58,"warning":12,"total":805},
  		   "host":{"handled":35,"unhandled":2,"down":7,"up":30,"total":37}},
"list":[
        {
            "hostgroup_id":"7",
            "hosts":{ "handled":2, "unhandled":0, "down":{"handled":1},
"up":{"handled":1}, "total":2},
"services":{"ok":{"handled":24},"critical":{"handled":21,"unhandled":3},
"handled":50,"highest":"critical","unknown":{"handled":5}, "unhandled":3,"total":53},
            "name":"ProjectX",
            "downtime":null
        },
        {
            "hostgroup_id":"13",
            "hosts":{"handled":2,"unhandled":1,"down":{"handled":2,"unhandled":1},"total":3},
            "services":{"critical":{"handled":73},"handled":86,"highest":"critical",
			"unknown":{"handled":13},"unhandled":0,"total":86},
            "name":"Internal",
            "downtime":"2"
        },
        {
            "hostgroup_id":"14",
            "hosts":{"handled":31,"unhandled":1,"down":{"handled":2,"unhandled":1},
"up":{"handled":29},"total":32},
            "services":{"ok":{"handled":534},"critical":{"handled":66,"unhandled":43},
"handled":611,"highest":"critical","unknown":{"handled":9,
"unhandled":2},"unhandled":55,"warning":{"handled":2,"unhandled":10},
"total":666},
            "name":"Hosting",
            "downtime":"2"
        }
    ]}}

So we can see by looking at it that it is nested JSON and in this case it returned the current status for 3 host groups: ProjectX, Internal, and Hosting.

Because the JSON is nested we will benefit from doing some processing on it before giving it to the UI routines that in GXT which only handles flat JSON.

The Architecture for the Mini-Dashboard Application

The following picture shows an overview of the architecture and the different components that will be used in this solution:

The GXT client will not call the Opsview Enterprise server directly but instead go via a proxy server. This has the benefit that the JSON response from the Opsview Enterprise server can be processed and converted into Serializable Java objects. In this case the JSON is turned into a Java object called HostGroup, which in turn contains a HostStatus object. The HostGroup object will contain enough information to be able to draw a pie chart showing the status for the different host groups.

The HostStatus object will be used when the user clicks on one of the host group pie slices and wants to drill down to see more detailed status for a particular host group. Because we send the HostStatus object at the same time as the HostGroup object we do not have to make another remote call when the user drills down to see individual host group status.

Another benefit of having the proxy server is that it enables us to have the Mini Dashboard client web app and the Opsview Enterprise server web app on different hosts in different domains. If we did not have the proxy server it would not be possible to access content in Opsview Enterprise as for security reasons you cannot open URL connections to a different host to the host from where the client is downloaded. Instead, you would have to install the Dashboard web app on the same host as Opsview Enterprise is running on.

The Mini-Dashboard Proxy Server

Let’s start implementing the proxy server using the Apache Commons HTTP Client and the Jackson JSON parser. For the remote calls between the Mini-Dashboard client and the proxy server (i.e. AJAX based client calls) we will implement a com.google.gwt.user.client.rpc. RemoteService as provided by Google Web Toolkit. It will handle all marshalling of calls for us.

Here is a UML Diagram giving you an overview of the involved proxy server classes:

First setup the OpsviewServerProxyService interface as follows:

/**
  *
  * The client side stub for the Opsview Proxy Service RPC service.
  * @author Martin Bergljung, Opsera Ltd.
  */
@RemoteServiceRelativePath("opsviewproxy")
public interface OpsviewServerProxyService extends RemoteService {
    public static final String SERVICE_NAME = "OpsviewServerProxyService";

    public List getHostGroups();
}

The RemoteServiceRelativePath annotation associates a RemoteService with a relative path. This annotation will cause the client-side proxy to automatically invoke the ServiceDefTarget.setServiceEntryPoint method with
GWT.getModuleBaseURL() + value() as its argument. Subsequent calls to ServiceDefTarget.setServiceEntryPoint will override this default path.

We define one method called getHostGroups that will return a list of HostGroup objects. Create a HostGroup object as follows:

/**
 * Host Group (method level javadoc omitted for simplicity).
 *
 *  {
 *   "hostgroup_id":"14",
 *   "hosts":{"handled":31,"unhandled":1,"down":{"handled":2,"unhandled":1},
 *            "up":{"handled":29},"total":32},
 *   "services":{"ok":{"handled":534},"critical":{"handled":66,"unhandled":43},
 *              "handled":611,"highest":"critical","unknown":{"handled":9,"unhandled":2},
 *              "unhandled":55,"warning":{"handled":2,"unhandled":10},"total":666},
 *   "name":"Internal",
 *   "downtime":"2"
 *   }
 *
 * @author Martin Bergljung, Opsera Ltd.
 */
public class HostGroup implements Serializable {
    private int m_downtime;
    private String m_hostgroupId;
    private HostStatus m_hostStatus;
    private String m_name;
    private String m_services; // not used

    public HostGroup() {}

    //Getters, Setters, equals, hashCode, and toString have intentionally been left out to save space 

}

Next step is to create an asynchronous version of the interface as AJAX calls are asynchronous by nature. This is the actual interface that the client will come in contact with:

/**
 * The async counterpart of OpsviewServerProxyService.
 */
public interface OpsviewServerProxyServiceAsync {
    public void getHostGroups(AsyncCallback
> callback);
}       

The extra element here is the callback parameter that the client will have to implement. Now let’s move on to the actual implementation of this interface. Create the following class that implements the getHostGroups method:

/**
 * The server side implementation of the Opsview Proxy Server RPC service.
*/
public class OpsviewServerProxyServiceImpl extends RemoteServiceServlet
implements OpsviewServerProxyService {
   public static final String OPSVIEW_API_BASE_URI =
            "https:///opsview/api";
   public static final String OPSVIEW_API_HOSTGROUP_STATUS_URI =
            OPSVIEW_API_BASE_URI + "/status/hostgroup";

   public List getHostGroups() {
        String json = callOpsview(OPSVIEW_API_HOSTGROUP_STATUS_URI);

        if (json != null && json.length() > 0) {
            return readHostGroupsJSON(json);
        }

        return new ArrayList();
   }

Here you need to update the hostname in the OPSVIEW_API_BASE_URI constant to reflect your installation. Also, if you are not using SSL then change to http.

Then add the readHostGroupsJSON method that takes JSON returned from Opsview Enterprise and turns it into a list of HostGroup objects:

    private List readHostGroupsJSON(String json) {
        List hostGroups = new ArrayList();

        try {
            ObjectMapper mapper = new ObjectMapper(); // can reuse, share globally
            JsonNode rootNode = mapper.readValue(json, JsonNode.class);
            JsonNode hostGroupNode = rootNode.path("hostgroup");
            JsonNode listNode = hostGroupNode.path("list");

            for (JsonNode hostGroup : listNode) {
                HostGroup hg = new HostGroup();
                hg.setName(hostGroup.path("name").getTextValue());
                hg.setDowntime(hostGroup.path("downtime").getIntValue());
                String hostGroupId = hostGroup.path("hostgroup_id").getTextValue();
                hg.setHostGroupId(hostGroupId);
                hg.setServices("0"); // not used at the moment

                JsonNode hostStatus = hostGroup.path("hosts");
                HostStatus hs = new HostStatus();
                hs.setHostGroupId(hostGroupId);
                hs.setHandled(hostStatus.path("handled").getIntValue());
                hs.setUnhandled(hostStatus.path("unhandled").getIntValue());
                hs.setTotal(hostStatus.path("total").getIntValue()); // only one needed now

                JsonNode down = hostStatus.path("down");
                JsonNode up = hostStatus.path("up");
                hs.setDownHandled(down.path("handled").getIntValue());
                hs.setDownUnhandled(down.path("unhandled").getIntValue());
                hs.setUpHandled(up.path("handled").getIntValue());
                hs.setUpUnhandled(up.path("unhandled").getIntValue());

                hg.setHosts(hs);
                hostGroups.add(hg);
            }
        } catch (IOException e) {
            System.err.println("Fatal JSON Parsing error: " + e.getMessage());
            e.printStackTrace();
        }

        return hostGroups;
    }

What we do here is use Jackson JSON parser (org.codehaus.jackson) to first lookup the hostgroup node in the JSON response and then with this node we go on to lookup the list node. When we have the list node we can step through all the host groups and setup corresponding HostGroup objects. The HostStatus object is also setup by looking up the hosts node for each hostgroup node.

Here is a snippet of the JSON structure that we are talking about:

{"hostgroup":{
...
"list":[
        {
            "hostgroup_id":"7",
            "hosts":{

Finally we also need to implement the callOpsview method that will use Apache Commons HTTPClient to call Opsview Enterprise. Here is how that is done:

private String callOpsview(String url) {
        String username = ;
        String password =
;

        HttpClient client = new HttpClient();
	// Apache is adding basic authentication in Opsera’s installation so
// the following 2 lines are necessary
        Credentials defaultcreds = new UsernamePasswordCredentials(username, password);
        client.getState().setCredentials(AuthScope.ANY, defaultcreds);
        GetMethod getMethod = new GetMethod(url);
        getMethod.setRequestHeader("X-Username", username);
        getMethod.setRequestHeader("X-Password", password);
        getMethod.setRequestHeader("Accept", "application/json");

        try {
            int statusCode = client.executeMethod(getMethod);
            if (statusCode == HttpStatus.SC_OK) {
                String contents = getMethod.getResponseBodyAsString();
                System.out.println(contents);
                return contents;
            } else {
                System.err.println("Got Error " + statusCode +
                        " when calling Opsview API: " +url);
            }
        } catch (Exception e) {
            System.err.println("Fatal transport error: " + e.getMessage());
            e.printStackTrace();
        } finally {
            // Release the connection.
            getMethod.releaseConnection();
        }

        return "";
    }

When you implement this the username and password need to be specified according to the user account in your Opsview Enterprise installation. Here we can also see that we are setting the following headers:

• X-Username – Username for Opsview user account
• X-Password – Password for Opsview user account
• Accept – indicates that we will accept and handle JSON responses (can also be set to accept XML)

The Opsview Enterprise proxy server implementation also needs to be registered to a URL. We do this in the GWT module definition file as follows for testing purposes:

  <!— Setup the Opsview Server Proxy Service for hosted mode testing -->
    <servlet path="/opsviewproxy" class=
			"com..server.OpsviewServerProxyServiceImpl" />

And as follows for the real web application, open up web.xml and add:

      <servlet>
      <servlet-name>opsviewServerProxyServiceServlet</servlet-name>
      <servlet-class>com.<your package>.server.OpsviewServerProxyServiceImpl</servlet-class>
    </servlet>

    <servlet-mapping>
      <servlet-name>opsviewServerProxyServiceServlet</servlet-name>
      <url-pattern>/dashboardApp/opsviewproxy</url-pattern>
    </servlet-mapping>

The Mini-Dashboard Client

The client is implemented as Google Web Toolkit application. So first create the main entry point / class for the application:

public class DashboardApp implements EntryPoint {

    public void onModuleLoad() {
        OpsviewServerProxyServiceAsync service = (OpsviewServerProxyServiceAsync)
                GWT.create(OpsviewServerProxyService.class);
        Registry.register(OpsviewServerProxyService.SERVICE_NAME, service);

        LayoutContainer container = new LayoutContainer();
        final BorderLayout layout = new BorderLayout();
        container.setLayout(layout);
        container.setWidth(1024);
        container.setHeight(768);
        container.add(createMainMenuAndToolbarPanel(), createMainMenuAndToolbarLayoutData());
        container.add(new Dashboard(2), createDashboardLayoutData());
        container.add(createBottomStatusBar(), createBottomStatusBarLayoutData());

        RootPanel.get().add(container);
    }

This creates the stub that handles communication with the Opsview Enterprise proxy server and registers this stub object locally. Then the layout for the GXT application is setup. Now create the individual panels and layout data:

    private ContentPanel createMainMenuAndToolbarPanel() {
        ContentPanel mainMenuAndToolbarPanel = new ContentPanel();
        mainMenuAndToolbarPanel.setTopComponent(new MainMenu());
        mainMenuAndToolbarPanel.setHeading("Opsview Dashboard...");
        return mainMenuAndToolbarPanel;
    }

    private ContentPanel createBottomStatusBar() {
        ContentPanel panel = new ContentPanel();
        ToolBar toolbar = new ToolBar();
        panel.setHeaderVisible(false);
        Label opsviewVersion = new Label();
        opsviewVersion.setStyleName("Arial");
        opsviewVersion.setText("Opsview Dashboard Example");
        Label loggedInInfo = new Label();
        loggedInInfo.setText("Logged in as Opsview Admin");
        toolbar.add(opsviewVersion);
        toolbar.add(new FillToolItem());
        toolbar.add(loggedInInfo);
        toolbar.setHeight(25);
        panel.add(toolbar);
        return panel;
    }

    private BorderLayoutData createMainMenuAndToolbarLayoutData() {
        BorderLayoutData breadcrumbsLayoutData =
new BorderLayoutData(Style.LayoutRegion.NORTH, 50);
        breadcrumbsLayoutData.setHideCollapseTool(true);
        breadcrumbsLayoutData.setMargins(new Margins(5, 5, 0, 5));
        return breadcrumbsLayoutData;
    }

    private BorderLayoutData createDashboardLayoutData() {
        BorderLayoutData data = new BorderLayoutData(Style.LayoutRegion.CENTER);
        data.setMargins(new Margins(0, 5, 0, 5));
        return data;
    }

    private BorderLayoutData createBottomStatusBarLayoutData() {
        BorderLayoutData breadcrumbsLayoutData =
new BorderLayoutData(Style.LayoutRegion.SOUTH, 25);
        breadcrumbsLayoutData.setHideCollapseTool(true);
        breadcrumbsLayoutData.setMargins(new Margins(0, 5, 0, 5));
        return breadcrumbsLayoutData;
    }

We are also going to need a menu and you can implement it if you like as it is not going to be used in this example. Here is a starting point:

public class MainMenu extends ToolBar {
   public MainMenu() {....
   }
}

Last thing we need for the client is the actual Dashboard class that will call the Opsview Enterprise proxy and draw the pie chart with host group status:

public class Dashboard extends Portal {
   public static final String OFC_FLASH_LOCATION = "dashboardApp/chart/open-flash-chart.swf";

   private Chart m_hostGroupsChart;
   private List m_currentHostGroups;

   public Dashboard(int columns) {
        super(columns);
        setBorders(true);
        setStyleAttribute("backgroundColor", "white");
        setColumnWidth(0, .50);
        setColumnWidth(1, .50);

        add(createHostGroupsChartPortlet(), 1);
        loadHostGroupsChartPortlet();
   }

Here we start off by implementing the Dashboard as a GXT Portal. We add one Portlet that will show the Host Group status pie chart. Then we call a method that will load the pie chart.

Next implement the method that creates the Host Group chart portlet as follows:

   private Portlet createHostGroupsChartPortlet() {
        Portlet portlet = new Portlet();
        portlet.setHeading("Status - Host Groups Hierarchy");
        portlet.setLayout(new FitLayout());
        portlet.setHeight(400);

        String url = !isExplorer() ? "../../" : "";
        url += OFC_FLASH_LOCATION;
        m_hostGroupsChart = new Chart(url);
        m_hostGroupsChart.setBorders(true);

        //portlet.setIcon(ICONS.hostgroup());
        portlet.setTopComponent(createChartPortletToolBar());
        portlet.add(m_hostGroupsChart);

        configurePortlet(portlet);

        return portlet;
    }

First we create the portlet and set its layout and heading etc. We then create a chart component and add it to the portlet. The chart component uses Flash to draw the chart so we need to supply the chart component with the URL for the Flash component. We also create a little toolbar for the portlet to show that you could implement configuration of what you want to see this way:

    private ToolBar createChartPortletToolBar() {
        ToolBar toolbar = new ToolBar();

        Button searchButton = new Button("Config");
        //searchButton.setIcon(ICONS.properties());
        searchButton.setBorders(true);
        searchButton.setToolTip("Click here to configure the host group status chart");

        toolbar.setBorders(true);
        toolbar.setHeight(25);
        toolbar.add(new FillToolItem());
        toolbar.add(searchButton);

        return toolbar;
    }

The loading of the Chart Portlet is done as follows by calling the getHostGroups method of the Opsview Enterprise proxy server stub and implementing the onSuccess and onFailure methods:

    private void loadHostGroupsChartPortlet() {
        // Get the Opsview Server Proxy Service from the Registry
        final OpsviewServerProxyServiceAsync opsviewServerProxyService =
		(OpsviewServerProxyServiceAsync)
                Registry.get(OpsviewServerProxyService.SERVICE_NAME);

        // Make sure we got the service
        if (opsviewServerProxyService == null) {
            showInfoMsg("Opsview Server Proxy service could not be detected!");
        } else {
            opsviewServerProxyService.getHostGroups(new AsyncCallback
>() {
                public void onSuccess(List hostGroups) {
                    m_currentHostGroups = hostGroups;

                    ChartModel cm = new ChartModel("Hosts by Host Group",
                            "font-size: 14px; font-family: Verdana; text-align: center;");
                    cm.setBackgroundColour("#fffff5");

                    PieChart pie = new PieChart();
                    pie.addChartListener(listener);
                    pie.setAlpha(0.5f);
                    pie.setTooltip("#label#
#percent#");
                    pie.setColours("#ff0000", "#00aa00", "#0000ff", "#ff9900", "#ff00ff");

                    for (HostGroup hostGroup : hostGroups) {
                        int totalHosts = hostGroup.getHosts().getTotal();
                        String name = hostGroup.getName();
                        pie.addSlices(new PieChart.Slice(totalHosts, name + " (" +
totalHosts + ")", name));
                    }

                    cm.addChartConfig(pie);

                    m_hostGroupsChart.setChartModel(cm);
                }

                public void onFailure(Throwable throwable) {
			throw new RuntimeException(throwable);
                    showInfoMsg("Opsview Server Proxy
getHostGroups call failed: " + throwable.getMessage());
                }
            });
        }
    }

When the Opsview Enterprise server proxy responds we store the current host groups in a global variable so we can access it later on when the user clicks on one of the slices. We then do not have to call the server proxy as we have the HostStatus for the clicked on host group slice. We then create the PieChart and add a slice for each host group that was returned. A listener is also set for the pie chart and it will load the host status pie chart. The listener implementation looks like this:

    private ChartListener listener = new ChartListener() {
        public void chartClick(ChartEvent ce) {
            PieChart.Slice hostGroup = (PieChart.Slice) ce.getDataType();
            loadHostStatusChartPortlet(hostGroup.getText());
            Info.display("Chart Clicked", "You selected {0}.", "" + ce.getValue() +
", " + hostGroup.getLabel());
        }
    };

The Host Status Chart is loaded like this:

    private void loadHostStatusChartPortlet(String hostGroupName) {
        ChartModel cm = new ChartModel("Host Status for Host Group " + hostGroupName,
                "font-size: 14px; font-family: Verdana; text-align: center;");
        cm.setBackgroundColour("#fffff5");

        PieChart pie = new PieChart();
        pie.addChartListener(listener);
        pie.setAlpha(0.5f);
        pie.setTooltip("#label#
#percent#");
        pie.setColours("#ff0000", "#00aa00", "#0000ff", "#ff9900", "#ff00ff");

        for (HostGroup hostGroup : m_currentHostGroups) {
            if (hostGroupName.equals(hostGroup.getName())) {
                HostStatus hostStatus = hostGroup.getHosts();
                pie.addSlices(new PieChart.Slice(hostStatus.getDownUnhandled(),
"Down - unhandled (" + hostStatus.getDownUnhandled() + ")", "Down - unhandled"));
                pie.addSlices(new PieChart.Slice(hostStatus.getUpHandled(),
"Up - handled (" + hostStatus.getUpHandled() + ")", "Up - handled"));
                pie.addSlices(new PieChart.Slice(hostStatus.getDownHandled(),
"Down - handled (" + hostStatus.getDownHandled() + ")", "Down - handled"));
                pie.addSlices(new PieChart.Slice(hostStatus.getUpUnhandled(),
"Up - unhandled (" + hostStatus.getUpUnhandled() + ")", "Up - unhandled"));
            }
        }

        cm.addChartConfig(pie);

        m_hostGroupsChart.setChartModel(cm);
    }

When the Host Status chart is loaded it is not possible to go back to the Host Groups chart. There is no menu item for that or any implemented code to do that. If you like go ahead and implement this solution by yourself.

The little toolbar in the Chart Portlet is implemented like this:

    private void configurePortlet(final ContentPanel panel) {
        panel.setCollapsible(true);
        panel.setAnimCollapse(false);
        panel.getHeader().addTool(new ToolButton("x-tool-gear"));
        panel.getHeader().addTool(
                new ToolButton("x-tool-close", new SelectionListener() {
                    @Override
                    public void componentSelected(IconButtonEvent ce) {
                        panel.removeFromParent();
                    }
                }));
    }

The last couple of methods are helpers:

    public static boolean isExplorer() {
        String test = Window.Location.getPath();
        if (test.indexOf("pages") != -1) {
            return false;
        }
        return true;
    }

    public static void showInfoMsg(String msg) {
        MessageBox box = new MessageBox();
        box.setButtons(MessageBox.OK);
        box.setIcon(MessageBox.INFO);
        box.setTitle("Information");
        box.setMessage(msg);
        box.show();
    }
}

Running the Mini-Dashboard Client

To run this application after installing it under for example Tomcat we use a URL like this:

http://localhost:8080/opsviewui/dashboardApp.html

When we run this we should see a Host Groups Chart looking something like this:

Clicking on one of the Host Groups will show Host status as follows:

Resources

The Java libraries that were used in this example can be found here:

• GWT – http://code.google.com/webtoolkit/
• GXT – http://www.extjs.com/products/gwt/
• Apache Commons HTTP Client – http://hc.apache.org/httpclient-3.x/
• Jackson JSON Parser – http://jackson.codehaus.org/

In the light of the recent events at a BT network centre in Paddington (London, UK), where a series of compound failures caused a massive outage with huge knock-on effects, I’m sure many businesses are taking another look at their own (and their suppliers) availability with a view to beefing up business continuity.

Within the spirit of continuous improvement this should be taken as an opportunity to improve the overall ’system’ rather than finger pointing.

What is business continuity?

Quite simply, business continuity is how you can stay in business (and meet your customers demands) in the wake of a disaster (whether a localised incident such as flooding or a further reaching issue like a terrorist attack). Your plan will typically cover all the business critical functions, systems and data.

So what is high availability?

High availability (or HA) is the way that system designers ensure ‘operational continuity’ of a system. This will typically involve ensuring that the system has no single point of failure (and for really fault tolerant systems there should also be no single point of recovery).

One of the common mistakes that people often make is getting confused between availability and scalability. Scaling is usually described in terms of vertical (bigger boxes) or horizontal (lots of boxes) scaling – horizontally scaled systems often have a degree of high availability whereas vertical scaling is potentially a bigger risk as you lose more capability when you lose a bigger box.

How do redundancy and diversity fit into the picture?

The phrase ‘belt & braces’ describes this nicely – two different ways of achieving the same goal.

One example might be having multiple (diverse) suppliers provide network connections to a building; in data centres it is common to have a (redundant) backup generator – though this is usually coupled with a battery-based UPS (uninterruptible power supply) to provide seamless failover.

But we’ve got this new fangled Cloud thing…

Let’s attempt to clarify what ‘cloud computing’ means – there are a number of different types of on-demand services that run ‘in the Cloud’:

• IaaS – Infrastructure as a Service
• PaaS – Platform as a Service
• SaaS – Software as a Service
• IaaS – Infrastructure as a Service

This category splits into:

• Virtual Private Server (VPS) providers such as Slicehost allow businesses to rent a fixed size virtual server on demand for a set monthly fee
• Elastic computing providers such as Amazon EC2 allow businesses to have a virtual machine image that can grow to meet requested demand (vertical scaling).

PaaS – Platform as a Service

Platform as a Service providers offer application-hosting platforms (again these can be of fixed-size or scalable) so that application developers can focus on adding business value rather than needing to worry about the underlying infrastructure. Google AppEngine is one example of this approach for Java & Python applications.

SaaS – Software-as-a-Service

Software-as-a-Service provides access to business software over the Internet for a set monthly fee. This can either be multi-tenanted, where multiple companies share a system, or with separate installations per customer. One of the major SaaS success stories is SalesForce.com who started with a CRM on-demand offering.

Public vs. private vs. hybrid clouds

We’ve already discussed public cloud offerings above; a private cloud is typically an on-premise or a dedicated outsourced managed cloud platform (e.g. using the open source Eucalyptus or VMWare vCloud). A hybrid cloud is where a private cloud is used in conjunction with one or more external cloud providers.

There is also the notion of community clouds with varying definitions:

1. whereby similar organisations pool resources into a shared multi-tenant cloud (though I prefer to describe this as a shared private cloud or a restricted cloud e.g. Google’s ‘GovCloud’)
2. a decentralised peer-to-peer cloud utilising spare computing power (and bandwidth) of internet-connected computers.

Deployment models

Assuming that your organisation can operate its business critical systems and store business critical data in the Cloud, then there are a number of possible deployment models to consider, chiefly:

1. Backup
2. Failover
3. Active-active
4. Cloud as a backup

Cloud as a backup

In this model, data plus the necessary software packages and configuration data (e.g. CMDB configuration data) for critical systems are backed up to 1 (or ideally more) hosts in the cloud. In the event of a disaster at the primary operating location, new virtual servers are commissioned and the configuration management tool (e.g. Puppet) is used to provision the servers with the appropriate software.

Recovery time is dependent upon the time taken to commission/provision the new virtual servers and perform any data transfer (or decryption). Database/file replication techniques can help to reduce the time to recover.

Failover to the cloud

For this model, there are pre-configured server instances in the cloud running the business critical systems combined with data replication. In the event of a disaster at the primary operating location, the systems are failed over to the cloud instance (this can be manual or use an automated ‘global load balancer’).

Active-active

This form of hybrid cloud is fundamentally the same as the failover option, however there are more complexities involved in the data synchronisation, session failover etc.

The Cloud

So how can the Cloud help with planning Business Continuity activities?

1 & 2: People & premises

For knowledge worker businesses, the Internet and widespread availability of broadband has increased the prevalence of distributed home workers. With a distributed workforce and cloud-based systems these two are items of less concern; it remains a practical option to have staff work from home (or indeed a temporary serviced office) in the event of a disaster accessing systems running in the cloud.

For a true ‘belt and braces’ approach, 3G/HSDPA mobile broadband dongles can be used to provide a secondary Internet connection for home workers should their main internet connection be unavailable.

3: Technology

Infrastructure-as-a-Service is very compelling for providing a Business Continuity strategy for data centre(s) using the deployment models outlined above. Furthermore, VoIP services can provide sufficient telephony cover for small-medium businesses.

4: Information

Cloud-based services (whether computing based or dedicated storage solutions e.g. Amazon S3) can aid your business in having current data stored confidentially and readily available in the event of a disaster. Some regulated organisations may have to consider whether the service provider can store data within the appropriate territory/jurisdiction. Data integrity is a further consideration for more complex system environments (particularly with the backup approach) – this needs to be taken into account for solution design / recovery procedures.

Conclusion

So we’ve established that cloud computing can alleviate some of the business continuity execution effort from your business, however you still need to plan properly (what happens if key personnel are unavailable; how will equipment & supplies be sourced) and perform due diligence on service providers to ensure that their SLAs and DR plans align with your needs.

We regularly get customer requests to write snmp checks for their devices but a problem we have is we don’t necessarily get access to those devices – in the past we have used output from an snmpwalk command and written the check scripts as best we can, and then shipped the new check to the customer for testing.

This doesn’t sit very well with me as I don’t think this gives a good customer experience.  Why should the customer do the testing when they have paid us to write the code?  How much time and effort has the customer got to put in to then get all debug data back to us to improve the check?

Well, there is a better way, by using SNMP::Persist

The snmpd agent on most unix type systems can be extended such that a script can be used to respond to all snmp requests for a particular oid tree.  This can be done by creating a script such as this:

#!/usr/bin/perl
use strict;
use warnings;
use SNMP::Persist qw(&define_oid &start_persister &define_subtree);
define_oid(".1.3.6.1.4.1.123.123"); # base of OID tree
start_persister();
while (1) {
 # define all sub OIDS within the base to respond to
 define_subtree(
 { '1.1' => [ 'STRING', 'string value' ],
 '1.2' => [ 'INTEGER', 9 ],
 '1.3' => [ 'Counter32', 342.3 ],
 }
 );
 sleep 300;
}

and then add an entry to your snmpd.conf such as

pass_persist .1.3.6.1.4.1.123.123 /path/to/perl/script

After restarting your snmpd agent you should now be able to query the oids

$ snmpwalk -v2c -c public localhost .1.3.6.1.4.1.123.123

While the above isn’t in a fit state to use for TAP tests, it shouldn’t take too much imagination to extend it by using ‘pass’ instead of ‘pass_persist’ in NET::SNMP in a small server started when running the project tests.

The great thing about open source software is the unexpected contributions you may suddenly receive. Jonathan Kamens of Advent Software sent a patch into the nagios-devel mailing list about a speed up to the status CGIs. He identified, using gprof, that status.cgi was taking the most time in the sorting routine of downtimes and comments.

His patch was simple and effective – instead of trying to sort the order of the objects every time, just do it once at the end.

We’ve been looking at the patch and produced some statistics of the old behaviour versus the new behaviour. The times can be found from running the t/660status-downtimes-comments.t perl script within the Nagios code.

For 100000 comments, the status.cgi was taking 28 minutes just reading the current status data – and in the test it didn’t actually finish! With Jonathan’s patch, this reduced down to 53 seconds.

There’s probably still some work to do as an increase of 10 from 10000 to 100000 increases the execution time by a factor of 53, but this is a good start!

There seem to be some work in looking at alternative ways of getting status data, including NDOutils database backend, or mklivestatus to get the information directly from the Nagios daemon.

For Opsview, we’re betting on the database backend, because there are other advantages (arbitrarily complex search criteria, historical information, separation of work from Nagios instance), although we like to provide the old style Nagios CGIs as well, so we’ll be adding this patch to Opsview. Funnily enough, the Nagios users get the patch earlier than the Opsview users do! But that’s all part of working with open source software and improving the tools for everyone.

This Monday morning, we got lots of calls from our users where Opsview slave systems running Nagios were raising freshness alerts because checks weren’t being run within their specified period.

Suspiciously, this was also the weekend where clocks went back one hour due to daylight savings changes.

Hosts and services were meant to be scheduled in 5 minute intervals on our internal system, were marked to be run on 26th October at 23:00 instead of 25th October.

So there seems to be a problem within Nagios where the rescheduling has been adversely affected by the clock changes. We expect this will affect any countries still using Daylight Saving Time when they revert over the next few weeks.

This affects all active checks on all Nagios systems, not just a distributed environment.

We’re going to investigate this more deeply within Nagios because this definitely worked fine over the last few years. From the Nagios mailing lists, it seems to be in Nagios 3.2.0, but not Nagios 3.0.6. However, it has affected Opsview systems that run Nagios 3.0.6 and I think it is due to this patch which we brought forward from Nagios 3.2.0.

The workaround at the moment is to recheck all your hosts and services again. In Opsview 3.3.2, you can use the Mass recheck functionality.

Alternatively, you can use this script which we’ve written. Download it, put it in /usr/local/nagios/bin and run it. It will submit a SCHEDULE_HOST_CHECK and SCHEDULE_HOST_SVC_CHECKS for all hosts in your system, using the objects.cache file to get all the host names. There is a random 5 minute difference applied, so that not all the checks will run at the same time.

You may need to adapt the script if you use different paths, but it will work on all Opsview systems.

In a distributed environment, you only need to run this on the Opsview master server and the requests will be sent to slaves automatically.

Update: We’ve been testing this and have found the following results:

• If you use a timezone such as Europe/London, then the bug was triggered between 2009-10-25 23:00:00 and 2009-10-26 00:00:00
• If you use a timezone such as America/New_York, then the bug will be triggered between 2009-11-01 23:00:00 and 2009-10-02 00:00:00
• If you use UTC, then the bug will not be triggered

Basically, it occurs on the day when the time for the nagios daemon goes back an hour.

We’ve managed to recreate the bug in a libtap test, so we’re working on a fix to Nagios.

Update: We’ve applied a patch to core Nagios for this fix. Using our tests, we’ve found that the bug happens only during the 23:00 to 00:00 on the day when clocks go back in time.

Since it is an automated test, we ran the test checking every half an hour, a million times and … it started to fail … in 2038. This is a known Year 2038 problem. I think there will be changes to Nagios before then

At the heart of Opsview is the Nagios monitoring engine. One of the policies we have with Opsview is to keep the number of changes of our dependent software as low as possible. We do this by keeping track of all the patches we apply and pushing these back upstream (though recently we haven’t had as much time as we’d like…).

Looking back over the last year, we’ve compiled a report of the number of patches to Nagios.

Or if you prefer a graph:

So you can see that we’ve been working hard on reducing the number of changes to Nagios. Are we stripping functionality out of Opsview? Not at all – we’re trying to push as much back to the main project as we can. There was a big drop down from 40 to 30 when we upgraded Opsview to Nagios 3, mainly due to all the patches we already informed the project about.

There was a small increase between February to September 2009 – this was due to customer changes that we needed to apply. But there was another major drop again when we upgraded to Nagios 3.2.0 – this time due to my involvement in the core Nagios team. We’ll do a similar review on NDOutils since my colleague Duncan has been updating code there.

So by definition, Opsview uses a forked version of Nagios. But we call this a shallow fork, because we’re keeping our changes to a minimum and trying to make sure that the core project benefits from the enhancements and fixes and testing procedures that we use.

We’re proud of our involvement with the Nagios project and we’re committed to making Nagios better for everyone, driving standards higher and improving core technologies.