Monitoring Microsoft Windows Updates

Introduction

Monitoring WSUS updates on Microsoft Windows Server is critical to ensure you get alerted when your systems need to be patched. The process to update Windows Updates on high priority servers implies proper planning to ensure no post-installation problems. If we could trust Microsoft patches for 100 %, installing WSUS updates on a system would be done the moment a maintenance schedule could be created for this system. Unfortunately in my personal experience, WSUS updates are more a cause of problems instead of a solution. That’s why we prefer to not install them too fast, as you might experience major issues with your production systems or with the software that is running on it. A recent example, a colleague accidentally patched some production SharePoint servers, which prohibited the creation of new sitecollections and caused issues with some icons. The only solution was to restore a backup…

Ideally the updates would first need to get tested on QA systems. If the QA servers are running for some times without issues, the production systems can get patched. The above is one of the reasons I spent some time combining the best features from the available Windows Update plugins on the Nagios Exchange.
Such as Christian Kaufmann’s idea to cache the list of Windows Updates into a file. This results in a much lower performance impact of the plugin on the servers you are monitoring. If you have any experience with WSUS updates, you will have noticed that the ‘TrustedInstaller.exe” process which is a MS Windows system process that takes care of querying the WSUS server and installing updates if requested. 

The plugin will count all available WSUS updates and output the count in every possible state. However it will only alert in case a set number of days have passed since the last successful update was installed. By using this method, you can then define a policy and agree to patch all systems which had no updates for a certain time. You could use different policies for QA and PR (production) systems to prevent problems. 

WSUS

 

Details

Some things you need to know about Windows Updates. Microsoft saves the date of the ‘last successful update’ in the registry. The location of the String Value is:

This date however is saved in the Greenwich Mean Time (GMT) or the Coordinated Universal Time (UTC) format. My plugin will try to translate this time to the local time format with the help of a function called Get-LocalTime. This function uses the [System.TimeZoneInfo] .NET class which is only usable if you have .NET 3.5 or higher. So keep in mind the ‘Last Successful Update’ date is in UTC format for servers where .NET 3.5 or higher is not installed.

The plugin will also check this registry key:

And give a warning if the system has a required reboot pending.

PSWindowsUpdate

Starting from Windows 10, Microsoft apparently decided to no longer make use of the above registry key. The only way I found to retrieve the last successful update date and time is with the help of the PSWindowsUpdate module. So I added another argument which allows you to select a different method named ‘PSWindowsUpdate’ to retrieve the necessary information. Please not that the default method is still the original method, I called ‘UpdateSearcher”

In order for this method to work, you will need to install the PSWindowsUpdate module in this location: C:\Windows\System32\WindowsPowerShell\v1.0\Modules. If you are using Powershell 5 you can just do:

I’ve included the 1.5.1.11 and 1.5.2 version of the module in the GitHub repository. Or you can download it on the Microsoft Script Center Repository.

How to monitor your WSUS updates?

  1. Please note that the default DaysBeforeWarning and DaysBeforeCritical parameters are set to 120 and 150. Feel free to adjust them as required or pass them as an argument.
  2. Put the script in the NSClient++ scripts folder, preferably in a subfolder Powershell.
  3. In the nsclient.ini configuration file, define the script like this:
  4. Make a command in Nagios like this:
  5. Configure your service in Nagios. Make use of the above created command. Configure something similar like this as $ARG1$:
    QA servers =>

    PR servers =>

  6. If you want to make use of the new ‘PSWindowsUpdate’ method you will need to have an argument like this:

(Almost) Final words

So why did I create another pluging to check WSUS updates? Because I’m using a system which completely automates Windows Update installation with the help of Nagios XI and Rundeck. The existing plugins did not meet my requirements.

Please note that there are several known issues with WSUS on some operating systems. It’s recommended to always update to the latest ‘Windows Update Client’. Please check Windows 8.1 and Windows Server 2012 R2 update history for more information. More specific, when using WIndows Server 2012 R2, you will really want the following KB’s:

  • KB3172614 => “July 2016 update rollup for Windows 8.1 and Windows Server 2012 R2”
  • KB3179574 => “August 2016 update rollup for Windows 8.1 and Windows Server 2012 R2”
  • KB3185279 => “September 2016 update rollup for Windows 8.1 and Windows Server 2012 R2”

When you don’t have these update rollup’s, checking  for updates and updating your Windows 2012 R2 systems could go very slow. In our case an update check could take up to 40 minutes instead of 10 seconds. 

Let me know on the Nagios Exchange what you think of my plugin by rating it or submitting a review. Please also consider starring the project on GitHub.

How to Debug Perl Scripts with EPIC Eclipse

Introduction

As about a year ago I took over development of John Murphy’s NetApp Ontap Cluster monitoring plugin, I was in need of some way to debug Perl scripts in Windows. After some online research, it seemed Eclipse with the EPIC plugin was the way to go. One small notice is that Eclipse is built with Java and hence consumes quite a bit of RAM. I wouldn’t recommend using it with less then 4 GB of RAM.

Debug Perl with EPIC Eclipse

 

To make it easier for people to debug Perl scripts from a Wndows client, I’ll list the steps here how to get things running smoothly.

How to debug Perl scripts?

  1. Download and install Eclipse.
  2. Download and install ActivePerl
  3. Verify installation by opening cmd.exe and type perl -v
  4. Open Eclipse, go to Help menu and select Eclipse Marketplace. Search for EPIC or Eclipse Perl Integration and install the EPIC components. You can find more info about EPIC on their website.
  5. In order to see local variables, PadWalker needs to be installed. Before you can use the Perl Package Manager, you client needs a reboot (after installation of ActivePerl).  Open command windows (cmd.exe) and type ‘ppm install PadWalker’, which would result in something similar like this output:
  6. Next thing would be to show line numbers, as looking for line 1085 in a 2000 line script is quite hard without it. Go to the Window menu, and choose Preferences.  Next, choose Perl EPIC in the left column and enable the checkbox left of “Show line numbers”.

Enjoy your debug Perl hunt. I hope it can help you finding and solving issues in the check_netapp_ontap script.

Let me know if there are better free Perl debugging methods on Windows in a comment!

Willem

check-ms-win-disk-load-graph-01

Monitoring Windows Disk Load

Introduction

Monitoring disk load is one of the harder things to monitor, but also one of the most crucial things you should monitor. Disk load problems can really give your applications a hard time, slowing them down or crippling them completely. On Linux servers it’s easy, as the CPU wait counter gives clear hints of issues with your disk io.

I rolled out check_diskstat on our Linux servers in September 2014  and really missed a similar plugin for monitoring disk load on Windows servers. Hence, I started thinking about a new Powershell script, which would use the Powershell command ‘get-counter’, to gather all disk related information from the Performance Monitor. I started with making a list of the requirements:

  • The main requirement was that it had to be multilingual, as I work on English and Dutch versions of Windows Server 2003, 2003 R2, 2008 and 2008 R2. 
  • Another requirement was that the script had to allow an argument that specifies the amount of samples over which an average could be calculated.
  • The perfdata output should be outputted in a way where all disk load related values had to be visible in a graph. I had to deal with very high values, eg 8763098004 and very small decimals, eg 0,00014. This implied I had to find some way to make it visually attractive and correct in Highcharts, for example by outputting in milliseconds instead of seconds or megabytes instead of bytes.
  • The plugin also had to work culture independent. Some culture use ‘,’ and other use ‘.’ as decimal. I solved this by replacing [System.Threading.Thread]::CurrentThread.CurrentCulture with ‘en-US’ ans setting it back to the original value once I’m done.

Monitoring disk load may be useful in finding the cause of performance issues. If a component of an application starts writing huge logs or big amounts of data in a database on your Windows disks, a bottleneck could be created in your application’s flow. This bottleneck could quickly result in any kind of lag, latency or slowness for end-users, resulting in more incidents, calls or complaints. An integral part of the job as monitoring engineer, is to avoid  situations as described above. Here Nagios can help you, by alerting you before applications start getting slow. Up until now, the only way to monitor performance counters for Windows servers, was using an agent like NSClient++ (or NCPA?) to retrieve one performance counter. My check_ms_windows_disk_load plugin enables you to combine several disk load related performance counters with only one service. This method has several advantages:

  • You don’t need to worry what counters to monitor. The plugin will do that for you.
  • As the plugin monitors 8 performance counters, and you only need one service, this would save you 7 services for each disk. So your Nagios server has less work, which enables you to monitor other stuff instead or increase the monitor interval on your checks.
  • As you can pass maxsamples (-ms or –MaxSamples) as a parameter, you can choose yourself how long you want the plugin to run before calculating averages. Each sample should be one second.

You could also prove to your application engineers that the storage is or is not the cause of their application’s performance. You can use comprehensive graphs visualizing a collection of disk performance related information. You also need knowledge about your disk load in order to choose the right disk type for the job. Are your 3TB SATA disks strong enough to handle the job or will you have to buy more expensive SSD’s to achieve the performance you need?

How to monitor your disk load?

  1. Put the script in the NSClient++ scripts folder, preferably in a subfolder Powershell.
  2. In the nsclient.ini configuration file, define the script like this:

  3. Make a command in Nagios like this:

  4. Configure your service in Nagios. Make use of the above created command. Configure something similar like this as $ARG1$:

Examples:

One day after everything is configured correctly, your Highcharts graphs should look like this:

disk load graph 01

If you want to test the load on your Windows disks, you can use this Storage Load Generator DiskSPD from Microsoft to play. (Yes Microsoft has a GitHub account!!)

I hope this plugin can help you monitor the disk load on your Windows hosts. Please rate it on the Nagios Exchange if you like my work.

Monitoring Microsoft Failover Cluster Preferred Node

 

 


Error: Your Requested widget " wp-github-commits-5" is not in the widget list.
  • [do_widget_area sidebar-1]
    • [do_widget_area sidebar-2]
      • [do_widget id="tag_cloud-2"]
    • [do_widget_area widgets_for_shortcodes]
      • [do_widget id="wp-github-commits-14"]
      • [do_widget id="wp-github-commits-17"]
      • [do_widget id="wp-github-commits-15"]
      • [do_widget id="wp-github-commits-25"]
      • [do_widget id="wp-github-commits-3"]
      • [do_widget id="wp-github-commits-13"]
      • [do_widget id="wp-github-commits-20"]
      • [do_widget id="wp-github-commits-16"]
      • [do_widget id="wp-github-commits-2"]
      • [do_widget id="wp-github-commits-24"]
      • [do_widget id="wp-github-commits-23"]
      • [do_widget id="wp-github-commits-21"]
      • [do_widget id="wp-github-commits-5"]
      • [do_widget id="wp-github-commits-6"]
      • [do_widget id="wp-github-commits-26"]
      • [do_widget id="wp-github-commits-9"]
      • [do_widget id="wp-github-commits-22"]
      • [do_widget id="wp-github-commits-19"]
      • [do_widget id="wp-github-commits-8"]
      • [do_widget id="wp-github-commits-7"]
      • [do_widget id="wp-github-commits-18"]
      • [do_widget id="wp-github-commits-11"]
      • [do_widget id="wp-github-commits-4"]
      • [do_widget id="wp-github-commits-12"]
      • [do_widget id="wp-github-commits-27"]
    • [do_widget_area wp_inactive_widgets]
      • [do_widget id="recent-posts-2"]
      • [do_widget id="recent-comments-2"]

    Introduction

    Clustering is a very important technology to ensure application availability and performance. Accurate monitoring of your clusters is crucial to keep your applications stable. Not monitoring is not an option, as you might never know when a failover has taken place and waste valuable time looking in the wrong places for solutions to your problems. There are different options and levels of monitoring you can choose for monitoring Microsoft Windows failover clusters.

    Preferred Node Option

    One of the easiest is checking each cluster service if it is still running on it’s preferred node. The plugin from Nedstars (link) to check if MS cluster services are running on their preferred node only works for Windows 2008 R2 or later versions. Apart from that I noticed that there seemed to be a bug in Nedstars script. If a MS cluster contained more then one cluster service, it seemed to only check one of them. I did not investigate this further, so forgive me if I’m wrong here. As we are still using multiple Windows 2003 failover clusters,  I decided to write a completely new plugin that uses WMI to get information about failover cluster services on Microsoft Windows 2003 failover clusters. Windows Server 2008 R2 Failover Cluster Preferred Node: preferred node on 2008 Windows Server 2003 R2 Failover Cluster Preferred Node: check_ms_cluster_preferrede_node_2003R2 The plugin starts by checking the version of the Windows server where it is running on with WMI. If the version is lesser then 6.1 (which is the version number of Windows Server 2008 R2), the script will continue to use WMI to get all cluster services and then check each cluster service individually to see where it is running and compare that with it’s preferred node. If the OS version is not lesser then 6.1, the plugin will import the module ‘failoverclusters’ and start using module commands instead of WMI. I did not check if the script works on a cluster with more then two nodes, as we don’t have clusters with three or more nodes. If anyone could test this for me and let me know… 

    How to use the check_ms_cluster_preferred_node?

    1. Copy the ‘check_ms_cluster_preferred_node.ps1’ Powershell script to the NSClient++ scripts folder, preferably in a sub-folder ‘Powershell’ on all the failover cluster nodes you wish to monitor.
    2. In the nsclient.ini configuration file, define the external command like this (and restart the NSClient++ service (nscp) afterwards):
    3. Make a command in Nagios like this:
    4. Configure your service in Nagios. Use the command you previously made. A healthy check in Nagios XI would look like this:check_ms_cluster_preferred_node

    Other Microsoft Failover Cluster monitoring options?

    So is monitoring the preferred node the best way to monitor clusters. Definitely not. A cluster might have no or multiple preferred nodes for some reason. If you happen to own some MS clusters for critical applications, you will most likely want to be alerted for more issues then only a cluster service failover. I’ve seen multiple clusters with issues that did not consist of a failover. Instead one or more cluster service just went offline and online on the same node. In order to monitor this, I wrote another Powershell script that checks for certain event id’s in the Windows application eventlog and alerts when these events contain information about failing cluster services.

    Monitoring Microsoft Exchange 2010 Mailbox

     
     

    Error: Your Requested widget " wp-github-commits-6" is not in the widget list.
    • [do_widget_area sidebar-1]
      • [do_widget_area sidebar-2]
        • [do_widget id="tag_cloud-2"]
      • [do_widget_area widgets_for_shortcodes]
        • [do_widget id="wp-github-commits-14"]
        • [do_widget id="wp-github-commits-17"]
        • [do_widget id="wp-github-commits-15"]
        • [do_widget id="wp-github-commits-25"]
        • [do_widget id="wp-github-commits-3"]
        • [do_widget id="wp-github-commits-13"]
        • [do_widget id="wp-github-commits-20"]
        • [do_widget id="wp-github-commits-16"]
        • [do_widget id="wp-github-commits-2"]
        • [do_widget id="wp-github-commits-24"]
        • [do_widget id="wp-github-commits-23"]
        • [do_widget id="wp-github-commits-21"]
        • [do_widget id="wp-github-commits-5"]
        • [do_widget id="wp-github-commits-6"]
        • [do_widget id="wp-github-commits-26"]
        • [do_widget id="wp-github-commits-9"]
        • [do_widget id="wp-github-commits-22"]
        • [do_widget id="wp-github-commits-19"]
        • [do_widget id="wp-github-commits-8"]
        • [do_widget id="wp-github-commits-7"]
        • [do_widget id="wp-github-commits-18"]
        • [do_widget id="wp-github-commits-11"]
        • [do_widget id="wp-github-commits-4"]
        • [do_widget id="wp-github-commits-12"]
        • [do_widget id="wp-github-commits-27"]
      • [do_widget_area wp_inactive_widgets]
        • [do_widget id="recent-posts-2"]
        • [do_widget id="recent-comments-2"]
       

      Introduction

      I based this scipt on the MS Exchange 2010 DAG Mailbox check by Matt Haynes, but made the following changes:

      • Recovery mailbox databases are excluded. This way we don’t get any false alerts when colleagues are restoring a backup to a temporary recovery database.
      • For MS Exchange, backups are crucial, as your logs will grow very fast when they don’t get backupped. Therefore a warning is generated if the last backup date is over 26 hours and an error is generated if it was over 50 hours. (these time periods can be altered, depending on the total time your backup software requires to backup all your mailbox databases)
      • Performance data is gathered from the amount of mounted, healthy, and unhealthy mailbox databases.
      • If no mailbox databases are found on the Exchange server, an error is also generated.

      I would like to thank my colleague, De Clerck John, who created the exclusion for the recovery mailbox database and the check for last backup. His knowledge of MS Exchange and Powershell seems to be unlimited. As previously said, the script will count all healthy, failed and mounted mailbox databases and output performance data. This way, you can easily see when exactly something went wrong with your mailbox databases:

      check-ms-exchange-2010-health-graph-01

      In this screenshot it appears we had an issue on 28 April. In fact our Exchange administrators are busy migrating the mailbox databases to different MS Exchange servers, which is also clear in the next days of the graph.

      exchange

      How to monitor your MS Exchange databases?

      1. Copy the ‘check_ms_exchange_2010_health.ps1’ Powershell script to the NSClient++ scripts folder, preferably in a sub-folder ‘Powershell’ on all the Exchange 2010 servers you wish to monitor.
      2. In the nsclient.ini configuration file, define the external command like this (and restart the NSClient++ service (nscp) afterwards):
      3. Make a command in Nagios like this:
      4. Configure your service in Nagios. Use the command you previously made. Argument 1 should be the hostname of the MS Exchange server

      Grtz Willem