Monitoring NetApp Ontap

Introduction

There are of course numerous way to monitor your NetApp Ontap storage, but this post focusses for now on how to achieve quality monitoring with the help of a Nagios plugin, which was originally developed by John Murphy. The plugin definitely has some flaws, so all help is welcome to improve it. Read the post about debugging Perl scripts, make a fork of the project on Github and start experimenting.

The plugin is able monitor multiple critical NetApp Ontap components, from disk to aggregates to volumes. It can also alert you if it finds any unhealthy components.

NetApp Ontap Logical View

How to monitor Netapp Ontap with Nagios?

  • Download the latest release from GitHub to a temp directory and then navigate to it.
  • Copy the contents of NetApp/* to your /usr/lib/perl5 or /usr/lib64/perl5 directory to install the required version of the NetApp Perl SDK. (confirmed to work with SDK 5.1 and 5.2)
  • Copy check_netapp_ontap.pl script to your nagios libexec folder and configure the correct permissions

Parameters:

–hostname, -H => Hostname or address of the cluster administrative interface.

–node, -n => Name of a vhost or cluster-node to restrict this query to.

–user, -u => Username of a Netapp Ontapi enabled user.

–password, -p => Password for the netapp Ontapi enabled user.

–option, -o => The name of the option you want to check. See the option and threshold list at the bottom of this help text.

–warning, -w => A custom warning threshold value. See the option and threshold list at the bottom of this help text.

–critical, -c => A custom warning threshold value. See the option and threshold list at the bottom of this help text.

–modifier, -m => This modifier is used to set an inclusive or exclusive filter on what you want to monitor.

–help, -h => Display this help text.

Option list:

volume_health:

Check the space and inode health of a vServer volume on a NetApp Ontap cluster. If space % and space in *B are both defined the smaller value of the two will be used when deciding if the volume is in a warning or critical state. This allows you to accomodate large volume monitoring better. thresh: space % used, space in *B (i.e MB) remaining, inode count remaining, inode % used (Usage example: 80%i), “offline” keyword node: The node option restricts this check by vserver name.

aggregate_health:

Check the space and inode health of a cluster aggregate on a NetApp Ontap cluster. If space % and space in *B are both defined the smaller value of the two will be used when deciding if the volume is in a warning or critical state. This allows you to better accomodate large aggregate monitoring. thresh: space % used, space in *B (i.e MB) remaining, inode count remaining, inode % used (Usage example: 80%i), “offline” keyword, “is-home” keyword node: The node option restricts this check by cluster-node name.

snapshot_health:

Check the space and inode health of a vServer snapshot. If space % and space in *B are both defined the smaller value of the two will be used when deciding if the volume is in a warning or critical state. This allows you to better accomodate large snapshot monitoring. thresh: space % used, space in *B (i.e MB) remaining, inode count remaining, inode % used (Usage example: 80%i), “offline” keyword node: The node option restricts this check by vserver name.

quota_health:

Check that the space and file thresholds have not been crossed on a quota. thresh: N/A storage defined. node: The node option restricts this check by vserver name. snapmirror_health: Check the lag time and health flag of the snapmirror relationships. thresh: snapmirror lag time (valid intervals are s, m, h, d). node: The node options restricts this check by snapmirror destination cluster-node name.

filer_hardware_health:

Check the environment hardware health of the filers (fan, psu, temperature, battery). thresh: component name (fan, psu, temperature, battery). There is no default alert level they MUST be defined. node: The node option restricts this check by cluster-node name. port_health: Checks the state of a physical network port. thresh: N/A not customizable. node: The node option restricts this check by cluster-node name.

interface_health desc:

Check that a LIF is in the correctly configured state and that it is on its home node and port. Additionally checks the state of a physical port. thresh: N/A not customizable. node: The node option restricts this check by vserver name.

netapp_alarms:

Check for Netapp console alarms. thresh: N/A not customizable. node: The node option restricts this check by cluster-node name. cluster_health desc: Check the cluster disks for failure or other potentially undesirable states. thresh: N/A not customizable. node: The node option restricts this check by cluster-node name. disk_health: Check the health of the disks in the cluster. thresh: Not customizable yet. node: The node option restricts this check by cluster-node name. For keyword thresholds, if you want to ignore alerts for that particular keyword you set it at the same threshold that the alert defaults to.  

check-ms-win-disk-load-graph-01

Monitoring Windows Disk Load

Introduction

Monitoring disk load is one of the harder things to monitor, but also one of the most crucial things you should monitor. Disk load problems can really give your applications a hard time, slowing them down or crippling them completely. On Linux servers it’s easy, as the CPU wait counter gives clear hints of issues with your disk io.

I rolled out check_diskstat on our Linux servers in September 2014  and really missed a similar plugin for monitoring disk load on Windows servers. Hence, I started thinking about a new Powershell script, which would use the Powershell command ‘get-counter’, to gather all disk related information from the Performance Monitor. I started with making a list of the requirements:

  • The main requirement was that it had to be multilingual, as I work on English and Dutch versions of Windows Server 2003, 2003 R2, 2008 and 2008 R2. 
  • Another requirement was that the script had to allow an argument that specifies the amount of samples over which an average could be calculated.
  • The perfdata output should be outputted in a way where all disk load related values had to be visible in a graph. I had to deal with very high values, eg 8763098004 and very small decimals, eg 0,00014. This implied I had to find some way to make it visually attractive and correct in Highcharts, for example by outputting in milliseconds instead of seconds or megabytes instead of bytes.
  • The plugin also had to work culture independent. Some culture use ‘,’ and other use ‘.’ as decimal. I solved this by replacing [System.Threading.Thread]::CurrentThread.CurrentCulture with ‘en-US’ ans setting it back to the original value once I’m done.

Monitoring disk load may be useful in finding the cause of performance issues. If a component of an application starts writing huge logs or big amounts of data in a database on your Windows disks, a bottleneck could be created in your application’s flow. This bottleneck could quickly result in any kind of lag, latency or slowness for end-users, resulting in more incidents, calls or complaints. An integral part of the job as monitoring engineer, is to avoid  situations as described above. Here Nagios can help you, by alerting you before applications start getting slow. Up until now, the only way to monitor performance counters for Windows servers, was using an agent like NSClient++ (or NCPA?) to retrieve one performance counter. My check_ms_windows_disk_load plugin enables you to combine several disk load related performance counters with only one service. This method has several advantages:

  • You don’t need to worry what counters to monitor. The plugin will do that for you.
  • As the plugin monitors 8 performance counters, and you only need one service, this would save you 7 services for each disk. So your Nagios server has less work, which enables you to monitor other stuff instead or increase the monitor interval on your checks.
  • As you can pass maxsamples (-ms or –MaxSamples) as a parameter, you can choose yourself how long you want the plugin to run before calculating averages. Each sample should be one second.

You could also prove to your application engineers that the storage is or is not the cause of their application’s performance. You can use comprehensive graphs visualizing a collection of disk performance related information. You also need knowledge about your disk load in order to choose the right disk type for the job. Are your 3TB SATA disks strong enough to handle the job or will you have to buy more expensive SSD’s to achieve the performance you need?

How to monitor your disk load?

  1. Put the script in the NSClient++ scripts folder, preferably in a subfolder Powershell.
  2. In the nsclient.ini configuration file, define the script like this:

  3. Make a command in Nagios like this:

  4. Configure your service in Nagios. Make use of the above created command. Configure something similar like this as $ARG1$:

Examples:

One day after everything is configured correctly, your Highcharts graphs should look like this:

disk load graph 01

If you want to test the load on your Windows disks, you can use this Storage Load Generator DiskSPD from Microsoft to play. (Yes Microsoft has a GitHub account!!)

I hope this plugin can help you monitor the disk load on your Windows hosts. Please rate it on the Nagios Exchange if you like my work.

Monitoring MS SharePoint Health

 

 


Error: Your Requested widget " wp-github-commits-19" is not in the widget list.
  • [do_widget_area sidebar-1]
    • [do_widget_area sidebar-2]
      • [do_widget id="tag_cloud-2"]
    • [do_widget_area widgets_for_shortcodes]
      • [do_widget id="wp-github-commits-14"]
      • [do_widget id="wp-github-commits-17"]
      • [do_widget id="wp-github-commits-15"]
      • [do_widget id="wp-github-commits-25"]
      • [do_widget id="wp-github-commits-3"]
      • [do_widget id="wp-github-commits-13"]
      • [do_widget id="wp-github-commits-20"]
      • [do_widget id="wp-github-commits-16"]
      • [do_widget id="wp-github-commits-2"]
      • [do_widget id="wp-github-commits-24"]
      • [do_widget id="wp-github-commits-23"]
      • [do_widget id="wp-github-commits-21"]
      • [do_widget id="wp-github-commits-5"]
      • [do_widget id="wp-github-commits-6"]
      • [do_widget id="wp-github-commits-26"]
      • [do_widget id="wp-github-commits-9"]
      • [do_widget id="wp-github-commits-22"]
      • [do_widget id="wp-github-commits-19"]
      • [do_widget id="wp-github-commits-8"]
      • [do_widget id="wp-github-commits-7"]
      • [do_widget id="wp-github-commits-18"]
      • [do_widget id="wp-github-commits-11"]
      • [do_widget id="wp-github-commits-4"]
      • [do_widget id="wp-github-commits-12"]
      • [do_widget id="wp-github-commits-27"]
    • [do_widget_area wp_inactive_widgets]
      • [do_widget id="recent-posts-2"]
      • [do_widget id="recent-comments-2"]

    Introduction

    SharePoint is a web application platform in the Microsoft Office server suite. Launched in 2001, SharePoint combines various functions which are traditionally separate applications: intranet, extranet, content management, document management, personal cloud, enterprise social networking, enterprise search, business intelligence, workflow management, web content management, and an enterprise application store.
    SharePoint Health Analyzer is a feature in Microsoft SharePoint Foundation 2010 that enables administrators to schedule regular, automatic checks for potential configuration, performance, and usage problems in the server farm. Any errors that SharePoint Health Analyzer finds are identified in status reports that are made available to farm administrators in Central Administration. Status reports explain each issue, list the servers where the problem exists, and outline the steps that an administrator can take to remedy the problem.
    SharePoint Health Analyzer monitors the farm by applying a set of health rules. A number of these rules ship with SharePoint Foundation. You can create and deploy additional rules by writing code that uses the SharePoint Foundation object model. When a health rule executes, SharePoint Health Analyzer creates a status report and adds it to the Health Analyzer Reports list in the Monitoring section of Central Administration.
    This plugin will create a PSObject for each item in the status report that has no ‘Success’ severity and return a critical state if any problems are found, together with information about each problem in the status report, such as the failed service and date modified. 

    SharePoint_2010

    How to use check_ms_sharepoint_health?

    1. Put the script in the NSClient++ scripts folder, preferably in a subfolder Powershell, enabling you to use this Reactor action to update your plugins folder without having to edit the script.
    2. In the nsclient.ini configuration file, define the script like this:
      check_ms_sharepoint health=cmd /c echo scripts/powershell/check_ms_sharepoint_health.ps1; exit $LastExitCode | powershell.exe -command –
    3. Make a command in Nagios like this:
      check_ms_sharepoint_health => $USER1$/check_nrpe -H $HOSTADDRESS$ -p 5666 -t 60 -c check_ms_sharepoint_health 
    4. Configure your service in Nagios, make use of the above created command. 

    Monitoring Microsoft Failover Cluster Preferred Node

     

     


    Error: Your Requested widget " wp-github-commits-5" is not in the widget list.
    • [do_widget_area sidebar-1]
      • [do_widget_area sidebar-2]
        • [do_widget id="tag_cloud-2"]
      • [do_widget_area widgets_for_shortcodes]
        • [do_widget id="wp-github-commits-14"]
        • [do_widget id="wp-github-commits-17"]
        • [do_widget id="wp-github-commits-15"]
        • [do_widget id="wp-github-commits-25"]
        • [do_widget id="wp-github-commits-3"]
        • [do_widget id="wp-github-commits-13"]
        • [do_widget id="wp-github-commits-20"]
        • [do_widget id="wp-github-commits-16"]
        • [do_widget id="wp-github-commits-2"]
        • [do_widget id="wp-github-commits-24"]
        • [do_widget id="wp-github-commits-23"]
        • [do_widget id="wp-github-commits-21"]
        • [do_widget id="wp-github-commits-5"]
        • [do_widget id="wp-github-commits-6"]
        • [do_widget id="wp-github-commits-26"]
        • [do_widget id="wp-github-commits-9"]
        • [do_widget id="wp-github-commits-22"]
        • [do_widget id="wp-github-commits-19"]
        • [do_widget id="wp-github-commits-8"]
        • [do_widget id="wp-github-commits-7"]
        • [do_widget id="wp-github-commits-18"]
        • [do_widget id="wp-github-commits-11"]
        • [do_widget id="wp-github-commits-4"]
        • [do_widget id="wp-github-commits-12"]
        • [do_widget id="wp-github-commits-27"]
      • [do_widget_area wp_inactive_widgets]
        • [do_widget id="recent-posts-2"]
        • [do_widget id="recent-comments-2"]

      Introduction

      Clustering is a very important technology to ensure application availability and performance. Accurate monitoring of your clusters is crucial to keep your applications stable. Not monitoring is not an option, as you might never know when a failover has taken place and waste valuable time looking in the wrong places for solutions to your problems. There are different options and levels of monitoring you can choose for monitoring Microsoft Windows failover clusters.

      Preferred Node Option

      One of the easiest is checking each cluster service if it is still running on it’s preferred node. The plugin from Nedstars (link) to check if MS cluster services are running on their preferred node only works for Windows 2008 R2 or later versions. Apart from that I noticed that there seemed to be a bug in Nedstars script. If a MS cluster contained more then one cluster service, it seemed to only check one of them. I did not investigate this further, so forgive me if I’m wrong here. As we are still using multiple Windows 2003 failover clusters,  I decided to write a completely new plugin that uses WMI to get information about failover cluster services on Microsoft Windows 2003 failover clusters. Windows Server 2008 R2 Failover Cluster Preferred Node: preferred node on 2008 Windows Server 2003 R2 Failover Cluster Preferred Node: check_ms_cluster_preferrede_node_2003R2 The plugin starts by checking the version of the Windows server where it is running on with WMI. If the version is lesser then 6.1 (which is the version number of Windows Server 2008 R2), the script will continue to use WMI to get all cluster services and then check each cluster service individually to see where it is running and compare that with it’s preferred node. If the OS version is not lesser then 6.1, the plugin will import the module ‘failoverclusters’ and start using module commands instead of WMI. I did not check if the script works on a cluster with more then two nodes, as we don’t have clusters with three or more nodes. If anyone could test this for me and let me know… 

      How to use the check_ms_cluster_preferred_node?

      1. Copy the ‘check_ms_cluster_preferred_node.ps1’ Powershell script to the NSClient++ scripts folder, preferably in a sub-folder ‘Powershell’ on all the failover cluster nodes you wish to monitor.
      2. In the nsclient.ini configuration file, define the external command like this (and restart the NSClient++ service (nscp) afterwards):
      3. Make a command in Nagios like this:
      4. Configure your service in Nagios. Use the command you previously made. A healthy check in Nagios XI would look like this:check_ms_cluster_preferred_node

      Other Microsoft Failover Cluster monitoring options?

      So is monitoring the preferred node the best way to monitor clusters. Definitely not. A cluster might have no or multiple preferred nodes for some reason. If you happen to own some MS clusters for critical applications, you will most likely want to be alerted for more issues then only a cluster service failover. I’ve seen multiple clusters with issues that did not consist of a failover. Instead one or more cluster service just went offline and online on the same node. In order to monitor this, I wrote another Powershell script that checks for certain event id’s in the Windows application eventlog and alerts when these events contain information about failing cluster services.

      Monitoring Microsoft Exchange 2010 Mailbox

       
       

      Error: Your Requested widget " wp-github-commits-6" is not in the widget list.
      • [do_widget_area sidebar-1]
        • [do_widget_area sidebar-2]
          • [do_widget id="tag_cloud-2"]
        • [do_widget_area widgets_for_shortcodes]
          • [do_widget id="wp-github-commits-14"]
          • [do_widget id="wp-github-commits-17"]
          • [do_widget id="wp-github-commits-15"]
          • [do_widget id="wp-github-commits-25"]
          • [do_widget id="wp-github-commits-3"]
          • [do_widget id="wp-github-commits-13"]
          • [do_widget id="wp-github-commits-20"]
          • [do_widget id="wp-github-commits-16"]
          • [do_widget id="wp-github-commits-2"]
          • [do_widget id="wp-github-commits-24"]
          • [do_widget id="wp-github-commits-23"]
          • [do_widget id="wp-github-commits-21"]
          • [do_widget id="wp-github-commits-5"]
          • [do_widget id="wp-github-commits-6"]
          • [do_widget id="wp-github-commits-26"]
          • [do_widget id="wp-github-commits-9"]
          • [do_widget id="wp-github-commits-22"]
          • [do_widget id="wp-github-commits-19"]
          • [do_widget id="wp-github-commits-8"]
          • [do_widget id="wp-github-commits-7"]
          • [do_widget id="wp-github-commits-18"]
          • [do_widget id="wp-github-commits-11"]
          • [do_widget id="wp-github-commits-4"]
          • [do_widget id="wp-github-commits-12"]
          • [do_widget id="wp-github-commits-27"]
        • [do_widget_area wp_inactive_widgets]
          • [do_widget id="recent-posts-2"]
          • [do_widget id="recent-comments-2"]
         

        Introduction

        I based this scipt on the MS Exchange 2010 DAG Mailbox check by Matt Haynes, but made the following changes:

        • Recovery mailbox databases are excluded. This way we don’t get any false alerts when colleagues are restoring a backup to a temporary recovery database.
        • For MS Exchange, backups are crucial, as your logs will grow very fast when they don’t get backupped. Therefore a warning is generated if the last backup date is over 26 hours and an error is generated if it was over 50 hours. (these time periods can be altered, depending on the total time your backup software requires to backup all your mailbox databases)
        • Performance data is gathered from the amount of mounted, healthy, and unhealthy mailbox databases.
        • If no mailbox databases are found on the Exchange server, an error is also generated.

        I would like to thank my colleague, De Clerck John, who created the exclusion for the recovery mailbox database and the check for last backup. His knowledge of MS Exchange and Powershell seems to be unlimited. As previously said, the script will count all healthy, failed and mounted mailbox databases and output performance data. This way, you can easily see when exactly something went wrong with your mailbox databases:

        check-ms-exchange-2010-health-graph-01

        In this screenshot it appears we had an issue on 28 April. In fact our Exchange administrators are busy migrating the mailbox databases to different MS Exchange servers, which is also clear in the next days of the graph.

        exchange

        How to monitor your MS Exchange databases?

        1. Copy the ‘check_ms_exchange_2010_health.ps1’ Powershell script to the NSClient++ scripts folder, preferably in a sub-folder ‘Powershell’ on all the Exchange 2010 servers you wish to monitor.
        2. In the nsclient.ini configuration file, define the external command like this (and restart the NSClient++ service (nscp) afterwards):
        3. Make a command in Nagios like this:
        4. Configure your service in Nagios. Use the command you previously made. Argument 1 should be the hostname of the MS Exchange server

        Grtz Willem