Monitoring @ FOSDEM 2018 => Observability, Tracing and the RED method


Yesterday 03/02/18, I went to FOSDEM in Brussels. FOSDEM is a two-day non-commercial event organised by volunteers to promote the widespread use of free and open source software. The goal is to provide free and open source software developers and communities a place to meet in order to:

  • get in touch with other developers and projects
  • be informed about the latest developments in the free software world
  • be informed about the latest developments in the open source world
  • attend interesting talks and presentations on various topics by project leaders and committers
  • to promote the development and benefits of free software and open source solutions

It seems like FOSDEM is getting more popular each year, resulting in rooms and aula’s filling up fast.


Take a look at the Saturday schedule to get an idea of the wide range of topics that were discussed. I’ll try to give a quick overview of the most interesting monitoring related topics I learned about.


Christiny Yen held an interesting lecture about Observability, which is an upcoming term that’s very trending in the devops monitoring world. Wikipedia describes observability as a measure of how well internal states of a system can be inferred from knowledge of its external outputs.

But is there a significant difference between monitoring and observability? It seems like monitoring vendors are quickly updating their websites and promise observability while a few months back, the word was nowhere to be found.

It seems to me like observabiltiy could be in the same list as other system attributes, such as:

  • functionality
  • performance
  • testability
  • debuggability
  • operability
  • maintainability
  • efficiency
  • teachability
  • usability

By making systems observable, we can make it easier to come to decisions and take action, making sure the OODA loop is actually a loop, instead of having a break in it.

Observability OODA Loop

Observability isn’t a substitute for monitoring, nor does it obviate the need for monitoring, they are complementary. Monitoring is best suited to report the overall health of systems. Monitoring, as such, is best limited to key business and systems metrics derived from time-series based instrumentation, known failure modes as well as blackbox tests. Observability, on the other hand, aims to provide highly granular insights into the behavior of systems along with rich context, perfect for debugging purposes. It can be used as a way to better understand system performance and behavior, even during the what can be perceived as “normal” operation of a system. Since it’s impossible to predict every single failure mode a system could potentially run into or predict every possible way in which a system could misbehave, it becomes important that we build systems that can be debugged armed with evidence and not conjecture. 

How can we make microservices observable?

  • Observable systems should emit events, metrics, logs and traces
  • All components (also non-critical) should be instrumented
  • Instrumentation should not be ‘opt-in’, manual or ‘hard to Do’
  • Need to find the right balance of metrics, logs and traces for a given service
  • Using advanced analytics for diagnosing services, such as anomaly detection and forecasting


Another interesting topic, trace buffering with LLTng, was discussed by Julien Desfossez.  Some LLTng features:

  • System-wide insights: LLTng allows to understand the interactions between multiple components of a given system. By producing a unified log of events, LLTng provides great insight into the system’s behavior.
  • High performance: Tracers achieve great performance through a combination of essential techniques such as per-CPU buffering, RCU data structures, a compact and efficient binary trace format, and more.
  • Flexibility by offering multiple modes:
    • Post-processing mode: Gather as much data as possible and take the time to investigate
    • Snapshot mode: When an error is detected, extract the last trace buffer from memory
    • Live mode: Continuously monitor a stream of events
    • Rotation mode: Periodically, or on a specific trigger, process a chunk of trace without stopping the tracing

Tracing tends to be quite expensive. Think long and hard whether the added complexity is warranted. You might be falling into the trap of premature optimisation? Is optimisation that important when you could just scale horizontally?

Tracing to disk with all kernel events enabled can quickly generate huge traces:

  • 54k events/sec on an idle 4-cores laptop, 2.2 MB/sec
  • 2.7M events/sec on a busy 8-cores server, 95 MB/sec

So make sure you watch you available storage closely..

Some service instrumentations Methods

Tom Wilkie gave a presentation about the RED Method, while also talking about the USE Method and the Four Golden Signals. He explained why consisteny is an important approach for reducing cognitive load.

The USE Method

For every resource, monitor

  • Utilisation: % time the resource was busy
  • Saturation: Amount of work resource has to do (often queue length)
  • Errors: Count of error events

The RED Method

For every service, monitor request

  • Rate: Number of requests per second
  • Errors: The number of those requests that are failing
  • Duration: The amount of time those requests take

Google’s Four Golden Signals

  • Latency: The time it takes to service a request, with a focus on distinguishing between the latency of successful requests and the latency of failed requests.
  • Traffic: A measure of how much demand is being placed on the service. This is measured using a high-level service-specific metric, like HTTP requests per second in the case of an HTTP REST API.
  • Errors: The rate of requests that fail. The failures can be explicit (e.g., HTTP 500 errors) or implicit (e.g., an HTTP 200 OK response with a response body having too few items).
  • Saturation: How “full” is the service. This is a measure of the system utilization, emphasizing the resources that are most constrained (e.g., memory, I/O or CPU). Services degrade in performance as they approach high saturation.


FOSDEM 2018 seems to be a very popular event which covers a wide spectrum of free and open source software projects. I learned some new things and met several interesting people. Hopefully I’m able to return next year. If open source software peakes your interest, I can certainly recommend attending this event. You will not be dissapointed. 

Monitoring Windows Scheduled Tasks


Tasks scheduler is a Microsoft Windows component that allows you to schedule programs or scripts to start at pre-defined intervals. There are two major versions of the task scheduler: In version 1.0, definitions and schedules are stored in binary .job files. Every task corresponds to a single action. This plugin will not work on version 1.0 of the task scheduler, which is running on Windows Server 2000 and 2003. In version 2.0, the Windows task scheduler got a redesigned user interface based on Management console. Version 2.0 also supports calendar and event-based triggers, such as starting a task when a particular event is logged to the event log, or when a combination of events has occurred. Also, several tasks that are triggered by the same event can be configured to run either simultaneously or in a pre-determined chained sequence of a series of actions.

Tasks can also be configured to run based on system status such as being idle for a pre-configured amount of time, on startup, logoff, or only during or for a specified time. Other new features are a credential manager to store passwords so they cannot be retrieved easily. Also, scheduled tasks are executed in their own session, instead of the same session as system services or the current user. You can find a list of all task scheduler 2.0 interfaces here.


Starting from Windows Powershell 4.0, you can use a whole range of Powershell cmdlets to manage your scheduled tasks with Powershell. This plugin for Nagios does not use these cmdlets, as it has to be Powershell 2.0 compatible. Maybe in a few years, when Powershell 2.0 becomes obsolete, I’ll patch the script to make use of the new cmdlets. You can find the complete list of cmdlets here. Failing tasks will always end with some sort of error code. You can find the complete list of error codes here. This plugin will output the exitcodes for failing tasks in the Nagios service description. Output will also notify you on tasks that are still running. We have multiple Windows servers at work with a growing amount of scheduled tasks and each scheduled task needs to be monitored. With the help of Nagios and this plugin you can find out:

  • How many are running at the same time?
  • How many are failing?
  • How long are they running?
  • Who created them?


Disabled scheduled tasks are excluded by default from In earlier versions, you had to manually exclude them by excluding them with -EF or -ET. It seemed like a logical decision to exclude disabled tasks by default and was suggested by someone on the Nagios Exchange reviewing the plugin.. Maybe one day I’ll make a switch to include them again if specified. As some scheduled tasks do not need to be monitored, the script enables you to exclude complete folders.

Since v5.13.160614 it is possible to include hidden tasks. Just add the ‘–Hidden 1’ switch to your parameters and your hidden tasks will be monitored.

One of the folders I tend to exclude almost all the time is the “Microsoft” folder. It seems like several tasks in the Microsoft folder tend to fail sometimes. So unless you absolutely need to know the state of every single scheduled task running on your Windows Server, I can advise you to exclude it too. You can find the folder and tasks in this locations: C:\Windows\System32\Tasks
It is possible to include tasks or task folders with the ‘–InclFolders’ and ‘–InclTasks’ parameters. This filter will get applied after the exclude parameter. Please note that including a folder is not recursive. Only tasks in the root of the folder will be included.


This is the help of the plugin, which lists all valid parameters:

You could put every scheduled task  you don’t want to monitor in a separate  folder and exclude it with the -EF parameter. Alternatvely, you can use the -ET parameter to exclude based on name patterns. One quite important thing to know is that in order to exclude or include the root folder, you need to escape the backslash, like this: “\\”.

How to monitor your scheduled tasks?

  1. Put the script in the NSClient++ scripts folder, preferably in a subfolder Powershell.
  2. In the nsclient.ini configuration file, define the script like this:

    For more information about external scripts configuration, please review the NSClient documentation. You can also consider defining a wrapped script in nsclient.ini to simplify configuration.

  3. Make a command in Nagios like this:

  4. Configure your service in Nagios. Make use of the above created command. Configure something similar like this as $ARG1$:

Some things to consider to make it work:

  • “set-exectionpolicy remotesigned”
  • Nscp service account permissions => Running with local system should suffice, but I had users telling me it only worked with a local admin. I found out that on some NSClient++ versions, more specific version and probably some earlier versions too, the following error occured when running nscp service as local system: “CHECK_NRPE: Invalid packet type received from server”. After filing an issue on the GitHub project page of NSClient++, Michael Medin quickly acknowledged the issue and solved it from version, so the plugin should work again as local system.


If you would run the script in cli from you Nagios plugin folder, this would be the command:

If you would want to exclude one noisy unimportant scheduled task, the command used in cli would look like this:

If you only want the scheduled tasks in the root to be monitored, you can use this command:

This would only give you the scheduled tasks available in the root folder. The output look like this now.

Final Words

It seems the perfdata in the Highcharts graphs sometimes contains decimal numbers (see screenshot), which is kind of strange as I’m sure I only pass rounded numbers. Seems this is related to the way RRD files are working. To reduce the amount of storage space used, NPCD and RRD while average out the data, resulting in decimals, even when you don’t expect them.

This is a small to do list:

  • Add switches to change returned values and output.
  • Add array parameter with exit codes that should be excluded.
  • Test remote execution. In some cases it might be useful to be able to check remotely for failed windows tasks.
  • Include a warning / critical threshold when discovered tasks exceed a certain duration.
  • I was hoping to add some more exit codes to check, which would make failed tasks easier to troubleshoot. You can find the list of scheduled task exit codes here. The constants that begin with SCHED_S_ are success constants, and the constants that begin with SCHED_E_ are error constants.


These are some screenshots of the Nagios XI Graph Explorer for two of our servers making use of the plugin to monitor scheduled tasks: Tasks 01 check_ms_win_tasks_graph_02 Let me know on the Nagios Exchange what you think of my plugin by rating it or submitting a review. Please also consider starring the project on GitHub.


Monitoring Linux Processes


As I had some issues with my Linode server related to mistuned MariaDB settings, I was forced to find a way to monitor a Linux process, such as httpd, mysqld and php. Not only did I need to know if they were running, how many of them were running, but also their cpu and memory usage, so I could tune my Apache settings (located at /etc/httpd/conf/httpd.conf). I hoped to find a plugin which did all of the above, but couldn’t find one. The plugin that came closest to what I needed, was this one written bij Eli Keimig. 

As the last release date was 08/11/2010 and it missed some crucial features, I decided to make it better. At the moment I added the following features:

  • Performance data for Linux process CPU usage.
  • Performance data for Linux process Memory usage.
  • Added Linux process count with performance data.
  • Improved the plugin output.
  • Added minimum and maximum Linux process count.

How to monitor a Linux process?

The plugin uses ‘ps’ to retrieve the Linux process information. Logged in as root, type the following in your terminal to show active processes on the server:

The a option tells ps to list the processes of all users on the system rather than just those of the current user, with the exception of group leaders and processes not associated with a terminal. A group leader is the first member of a group of related processes.

The u option tells ps to provide detailed information about each process.

The x option adds to the list processes that have no controlling terminal, such as daemons, which are programs that are launched during boot and run unobtrusively in the background until they are activated by a particular event or condition.

As the list of processes can be quite long and occupy more than a single screen, the output of ps aux can be piped (transferred) to the less command, which lets it be viewed one screen full at a time. The output can be advanced one screen forward by pressing the SPACE bar and one screen backward by pressing the b key.

With the -C parameter you can specify the Linux process for which to show information.

And you can specify what specific information to show with the -o parameter:

After joining the results with paste and making the sum with bc, we get the result we want.

Check out this screenshot which shows information about the httpd, mysqld, nagios and php processes.

Linux process

This information can really help troubleshoot LAMP configuration issues. I haven’t got a lot of time to produce a decent post, but I’ll extend this post when I find some more time. As it’s a Bash script I’m guessing it doesn’t need to much explanation to get it working in Nagios.


Monitoring Windows Disk Load


Monitoring disk load is one of the harder things to monitor, but also one of the most crucial things you should monitor. Disk load problems can really give your applications a hard time, slowing them down or crippling them completely. On Linux servers it’s easy, as the CPU wait counter gives clear hints of issues with your disk io.

I rolled out check_diskstat on our Linux servers in September 2014  and really missed a similar plugin for monitoring disk load on Windows servers. Hence, I started thinking about a new Powershell script, which would use the Powershell command ‘get-counter’, to gather all disk related information from the Performance Monitor. I started with making a list of the requirements:

  • The main requirement was that it had to be multilingual, as I work on English and Dutch versions of Windows Server 2003, 2003 R2, 2008 and 2008 R2. 
  • Another requirement was that the script had to allow an argument that specifies the amount of samples over which an average could be calculated.
  • The perfdata output should be outputted in a way where all disk load related values had to be visible in a graph. I had to deal with very high values, eg 8763098004 and very small decimals, eg 0,00014. This implied I had to find some way to make it visually attractive and correct in Highcharts, for example by outputting in milliseconds instead of seconds or megabytes instead of bytes.
  • The plugin also had to work culture independent. Some culture use ‘,’ and other use ‘.’ as decimal. I solved this by replacing [System.Threading.Thread]::CurrentThread.CurrentCulture with ‘en-US’ ans setting it back to the original value once I’m done.

Monitoring disk load may be useful in finding the cause of performance issues. If a component of an application starts writing huge logs or big amounts of data in a database on your Windows disks, a bottleneck could be created in your application’s flow. This bottleneck could quickly result in any kind of lag, latency or slowness for end-users, resulting in more incidents, calls or complaints. An integral part of the job as monitoring engineer, is to avoid  situations as described above. Here Nagios can help you, by alerting you before applications start getting slow. Up until now, the only way to monitor performance counters for Windows servers, was using an agent like NSClient++ (or NCPA?) to retrieve one performance counter. My check_ms_windows_disk_load plugin enables you to combine several disk load related performance counters with only one service. This method has several advantages:

  • You don’t need to worry what counters to monitor. The plugin will do that for you.
  • As the plugin monitors 8 performance counters, and you only need one service, this would save you 7 services for each disk. So your Nagios server has less work, which enables you to monitor other stuff instead or increase the monitor interval on your checks.
  • As you can pass maxsamples (-ms or –MaxSamples) as a parameter, you can choose yourself how long you want the plugin to run before calculating averages. Each sample should be one second.

You could also prove to your application engineers that the storage is or is not the cause of their application’s performance. You can use comprehensive graphs visualizing a collection of disk performance related information. You also need knowledge about your disk load in order to choose the right disk type for the job. Are your 3TB SATA disks strong enough to handle the job or will you have to buy more expensive SSD’s to achieve the performance you need?

How to monitor your disk load?

  1. Put the script in the NSClient++ scripts folder, preferably in a subfolder Powershell.
  2. In the nsclient.ini configuration file, define the script like this:

  3. Make a command in Nagios like this:

  4. Configure your service in Nagios. Make use of the above created command. Configure something similar like this as $ARG1$:


One day after everything is configured correctly, your Highcharts graphs should look like this:

disk load graph 01

If you want to test the load on your Windows disks, you can use this Storage Load Generator DiskSPD from Microsoft to play. (Yes Microsoft has a GitHub account!!)

I hope this plugin can help you monitor the disk load on your Windows hosts. Please rate it on the Nagios Exchange if you like my work.