threshold Archives

FireMotD – Dynamic MotD Generator for Linux Systems

23 April 201723/04/2017willemdh10 Comments

Introduction

While developing, playing or working on Linux systems, a dynamic MotD generator script can quickly give you an overview of all used components of your Linux systems. FireMotD can show you this information in a sanitized and colorful way when you are logging in with SSH or console.
Over the past years I’ve been working on the ultimate MotD generator which can be tuned with themes and easily extended. Not having to manually check for available updates, check the server load or how many users are logged on can really save yourself some valuable time. Depending of the chosen theme FireMotD will output all information defined in the theme on your server.

You can find the latest release on GitHub.

Themes

At this moment there are 5 themes you can choose.

– Three 256 color themes: ‘Gray’, ‘Invader’, ‘Modern’, ‘Original’
– Three 16 color themes: ‘Clean’, ‘Blue’, ‘Red’
– One Html theme ‘Html’

256 Colors

Modern

Original

Orange

16 Colors

Blue

Red

Gray

Html

Bash Colors

Most terminals are not just able to display black and white, they can also display colors thanks to escape sequences which are composed of an escape character (in Bash backslash \ ) followed by some other characters. For using one of the 256 colors on the foreground or text color, the control sequence is: \e[0;<colornumber>m
If you want to edit one of the 256 colors on the background, the control sequence is: \e]48;5<colornumber>m

FireMotD contains a nice and useful colortest:

FireMotD -C

1	FireMotD -C

And a colormap with the numbers for the control sequences:

FireMotD -M

1	FireMotD -M

You can find more information about terminals and colors here.

Terminal Emulators

It took me some time to figure out the best console emulator which supports 256 colors. With the help of some Cmder power users in this GitHub issue, I was pointed out that the development version of Cmder has an integrated mintty thanks to Babun. At the moment I’m still using this task, which launches a tab in Conemu or Cmder with a 256 color SSH session.

"%userprofile%\.babun\cygwin\bin\mintty.exe" /usr/bin/ssh root@server -new_console:t:server-root

1	"%userprofile%\.babun\cygwin\bin\mintty.exe" /usr/bin/ssh root@server -new_console:t:server-root

An alternative to using mintty, would be to use the msys connector, created by Maximus. As Maximums points out

Running bash from mintty in ConEmu looks like running bash inside Ubuntu inside VirtualBox in Windows. Check cygwin/msys connector developed specially for ConEmu.

So I gave his msys connector a try and after some small issues and some help of Maximum, I got it working. It works fine, but still has some minor flaws imho:

No right-click paste
Weird output with some commands such as top
When the console windows is resized, colors get scrambled.

You need to copy the executable of the Connector into the bin folder of your cygwin installation and create a task which executes the correct Connector. Please read through this ConEmu documentation. The task I’m using with the Connector looks like this:

conemu-cyg-64.exe -cur_console:h0 /usr/bin/ssh user@server

1	conemu-cyg-64.exe -cur_console:h0 /usr/bin/ssh user@server

How to use MotD generator?

System Install

You need to have make installed on the system, if you want to use the Makefile.

To install to /usr/local/bin/FireMotD

sudo make install

				1

						sudo make install

With this you can probably run FireMotD from anywhere in your system. If not, you need to add /usr/local/bin to your$PATH variable. To adjust the installation path, change the var IDIR=/usr/local/bin in the Makefile to the path you want.

To install bash autocompletion support

sudo make bash_completion

				1

						sudo make bash_completion

With this you can use TAB to autocomplete parameters and options with FireMotD. Does not require the sudo make install above (system install), but requires the bash-completion package to be installed and working. Then you should logout-login or source the bash completion file, eg. $ . /etc/bash_completion.d/FireMotD

If you don’t have root access, just install everything on your user’s folder and source the file from your user’s .profile file

Crontab to get system updates count

This is an example on how to record the system update package count daily.
This will update the file /var/tmp/updatecount.txt for later access.
Root privilege is required for this operation.

To edit root’s crontab

sudo crontab -e

				1

						sudo crontab -e

Then add this line (updates everyday at 3:03am)

3 3 * * * /usr/local/bin/FireMotD -S > /dev/null

				1

						3 3 * * * /usr/local/bin/FireMotD -S > /dev/null

Or using the old way

3 3 * * * /usr/local/bin/FireMotD -U > /var/tmp/updatecount.txt 

				1

						3 3 * * * /usr/local/bin/FireMotD -U > /var/tmp/updatecount.txt

Adding FireMotD to run on login

Choosing where to run your script is kind of situational. Some files will only run on remote logins, other local logins, or even both. You should find out what suits best your needs on each case.

To add FireMotD to a single user

Edit the user’s ~/.profile file, ~/.bash_profile file, or the ~/.bashrc file

nano ~/.profile

				1

						nano ~/.profile

Add the FireMotD call at the end of the file (choose your theme)

/usr/local/bin/FireMotD -t red

				1

						/usr/local/bin/FireMotD -t red

To add FireMotD to all users

You may call FireMotD from a few different locations for running globally.
Eg / etc/bash.bashrc and /etc/profile .

You may also create a initialization script, for example init.sh which will call the FireMotD script in /etc/profile.d when logging in. You can put whatever you like in this init.sh script. Everything in it will be executed at the moment someone logs in your system. Example:

#!/bin/bash
/usr/local/bin/FireMotD --Theme Red

				1
2

						#!/bin/bash
/usr/local/bin/FireMotD --Theme Red

MotD Generator Components

Script Version

To make it easy to see what version of generate_motd.sh you are using, I integrated the version in the frame of the MotD.

ScriptName="`readlink -e $0`"
ScriptVersion=" `cat $ScriptName | grep "# Version:" | awk {'print $3'} | tr -cd '[[:digit:].-]' | sed 's/.\{2\}$//'` "

1 2	ScriptName="`readlink -e $0`" ScriptVersion=" `cat $ScriptName \| grep "# Version:" \| awk {'print $3'} \| tr -cd '[[:digit:].-]' \| sed 's/.\{2\}$//'` "

Ip

I changed the method of retrieving the ip address. As CentOS 7 has different naming conventions, it seem a more solid method to use ip route.

ip route get 8.8.8.8 | head -1 | cut -d' ' -f8

1	ip route get 8.8.8.8 \| head -1 \| cut -d' ' -f8

Release

It’s always useful to know what Linux release your are working on.

cat /etc/*release | head -n 1

1	cat /etc/*release \| head -n 1

Kernel

Knowing the exact kernel version can also contribute to learning to know your servers

uname -rs

uname -rs

Uptime

I found this command to be working best on most systems. It is also capable of showing the number of days, hours, minutes and seconds. I also found it more useful then just using ‘uptime‘.

awk '{print int($1/86400)" day(s) "int($1%86400/3600)":"int(($1%3600)/60)":"int($1%60)}' /proc/uptime

1	awk '{print int($1/86400)" day(s) "int($1%86400/3600)":"int(($1%3600)/60)":"int($1%60)}' /proc/uptime

Average CPU Utilization

CPU utilization is a little harder as you prefer to get the average CPU utilization. To achieve this I’m using mpstat.

CpuUtil=`LANG=en_GB.UTF-8 mpstat 1 1 | awk '$2 ~ /CPU/ { for(i=1;i<=NF;i++) { if ($i ~ /%idle/) field=i } } $2 ~ /all/ { print 100 - $field}' | tail -1`

1	CpuUtil=`LANG=en_GB.UTF-8 mpstat 1 1 \| awk '$2 ~ /CPU/ { for(i=1;i<=NF;i++) { if ($i ~ /%idle/) field=i } } $2 ~ /all/ { print 100 - $field}' \| tail -1`

I also prefer to see the number of cores in the output:

CpuProc="`cat /proc/cpuinfo | grep processor | wc -l` core(s)."

1	CpuProc="`cat /proc/cpuinfo \| grep processor \| wc -l` core(s)."

Detect virtualization technique or hardware platform

‘dmesg’ is a command on most Unix-like operating systems that prints the message buffer of the kernel. It’s a useful command for the MotD generator as it doesn’t require any special permissions.

Qemu with KVM

Platform=`dmesg | grep "DMI:" | sed 's/^.*QEMU/QEMU/' | sed 's/, B.*//'`

1
2

Platform=`dmesg | grep "DMI:" | sed 's/^.*QEMU/QEMU/' | sed 's/, B.*//'`
VMware

Platform=`dmesg | grep "DMI:" | sed 's/^.*VMware/VMware/' | sed 's/, B.*//'`

1

Platform=`dmesg | grep "DMI:" | sed 's/^.*VMware/VMware/' | sed 's/, B.*//'`

Fujitsu Primergy

Platform=`dmesg | grep "DMI:" | sed 's/^.*FUJITSU PRIMERGY/Fujitsu Primergy/' | sed 's/, B.*//'`

1

Platform=`dmesg | grep "DMI:" | sed 's/^.*FUJITSU PRIMERGY/Fujitsu Primergy/' | sed 's/, B.*//'`

On some systems, you could get an ‘Unknown’ result. Please let me know on GitHub by making an issue. You should be able to find the correct platform with dmesg, cut and sed.

Cowsay and other ASCII generators

Something old school that exists for a really long time is Cowsay, which is a configurable speaking ASCII cow. I’ve seen several people integrate it in their MotD, but personally I think it does not fit into an enterprise. Something to play with at home though. You can even make it say a random expression every time you log in. Here you can find an implementation of Cowsay where you can output to html, text or even Json. And here you can find a Python implmentation of Cowsay. I also stumbled on Christopher Johnson’s ASCII Art Collection , an ascii archive containing a massive amount of ASCII art examples. The website is kind of outdated, but check it out if you like ascii art or if you are looking for something to integrate into your MotD. If you want convert some text to ASCII, check out this website or this one where you can enter some text, choose a font, character width and height and see the result immediately. If you want to convert an image to ASCII, check out this website.

(Almost) Final words

It might seem like a small tool, but my MotD generator can really help you get a quick overview of your Linux server state. I’ll try to extend the tool when I find the time, as I still have some ideas left which can improve it. Let me know on the Nagios Exchange what you think of my plugin by rating it or submitting a review. Please also consider starring the project on GitHub.

The last releases are a little slower, as querying the PHP, Apache and MariaDB versions take some time. In future releases, this info will be gathered with a cron job, a bit like the yum update count. Feel free to comment unneeded code to speed things up.

Thanks to all contributors to FireMotD!

Monitoring Windows Scheduled Tasks

22 April 201722/04/2017willemdh74 Comments

Download Latest

Introduction

Tasks scheduler is a Microsoft Windows component that allows you to schedule programs or scripts to start at pre-defined intervals. There are two major versions of the task scheduler: In version 1.0, definitions and schedules are stored in binary .job files. Every task corresponds to a single action. This plugin will not work on version 1.0 of the task scheduler, which is running on Windows Server 2000 and 2003. In version 2.0, the Windows task scheduler got a redesigned user interface based on Management console. Version 2.0 also supports calendar and event-based triggers, such as starting a task when a particular event is logged to the event log, or when a combination of events has occurred. Also, several tasks that are triggered by the same event can be configured to run either simultaneously or in a pre-determined chained sequence of a series of actions.

Tasks can also be configured to run based on system status such as being idle for a pre-configured amount of time, on startup, logoff, or only during or for a specified time. Other new features are a credential manager to store passwords so they cannot be retrieved easily. Also, scheduled tasks are executed in their own session, instead of the same session as system services or the current user. You can find a list of all task scheduler 2.0 interfaces here.

Requirements

Starting from Windows Powershell 4.0, you can use a whole range of Powershell cmdlets to manage your scheduled tasks with Powershell. This plugin for Nagios does not use these cmdlets, as it has to be Powershell 2.0 compatible. Maybe in a few years, when Powershell 2.0 becomes obsolete, I’ll patch the script to make use of the new cmdlets. You can find the complete list of cmdlets here. Failing tasks will always end with some sort of error code. You can find the complete list of error codes here. This plugin will output the exitcodes for failing tasks in the Nagios service description. Output will also notify you on tasks that are still running. We have multiple Windows servers at work with a growing amount of scheduled tasks and each scheduled task needs to be monitored. With the help of Nagios and this plugin you can find out:

How many are running at the same time?
How many are failing?
How long are they running?
Who created them?

Versions

Disabled scheduled tasks are excluded by default from 3.14.12.06. In earlier versions, you had to manually exclude them by excluding them with -EF or -ET. It seemed like a logical decision to exclude disabled tasks by default and was suggested by someone on the Nagios Exchange reviewing the plugin.. Maybe one day I’ll make a switch to include them again if specified. As some scheduled tasks do not need to be monitored, the script enables you to exclude complete folders.

Since v5.13.160614 it is possible to include hidden tasks. Just add the ‘–Hidden 1’ switch to your parameters and your hidden tasks will be monitored.

One of the folders I tend to exclude almost all the time is the “Microsoft” folder. It seems like several tasks in the Microsoft folder tend to fail sometimes. So unless you absolutely need to know the state of every single scheduled task running on your Windows Server, I can advise you to exclude it too. You can find the folder and tasks in this locations: C:\Windows\System32\Tasks
It is possible to include tasks or task folders with the ‘–InclFolders’ and ‘–InclTasks’ parameters. This filter will get applied after the exclude parameter. Please note that including a folder is not recursive. Only tasks in the root of the folder will be included.

Help

This is the help of the plugin, which lists all valid parameters:

check_ms_win_tasks.ps1: This script is designed to check Windows 2008 or higher scheduled tasks and alert in case tasks
 failed in Nagios style output.
Arguments:
 -H   | --Hostname    => Optional hostname of remote system, default is localhost, not yet tested on remote host.
 -EF  | --ExclFolders => Name of folders to exclude from monitoring.
 -IF  | --InclFolders => Name of folders to include in monitoring.
 -ET  | --ExclTasks   => Name of task patterns to exclude from monitoring.
 -IT  | --InclTasks   => Name of task patterns to include in monitoring.
 -Hid | --Hidden      => Switch to determine if hidden tasks need to be excluded.
 -h   | --Help        => Print this help output.

check_ms_win_tasks.ps1: This script is designed to check Windows 2008 or higher scheduled tasks and alert in case tasks

failed in Nagios style output.

Arguments:

-H | --Hostname => Optional hostname of remote system, default is localhost, not yet tested on remote host.

-EF | --ExclFolders => Name of folders to exclude from monitoring.

-IF | --InclFolders => Name of folders to include in monitoring.

-ET | --ExclTasks => Name of task patterns to exclude from monitoring.

-IT | --InclTasks => Name of task patterns to include in monitoring.

-Hid | --Hidden => Switch to determine if hidden tasks need to be excluded.

-h | --Help => Print this help output.

You could put every scheduled task you don’t want to monitor in a separate folder and exclude it with the -EF parameter. Alternatvely, you can use the -ET parameter to exclude based on name patterns. One quite important thing to know is that in order to exclude or include the root folder, you need to escape the backslash, like this: “\\”.

How to monitor your scheduled tasks?

Put the script in the NSClient++ scripts folder, preferably in a subfolder Powershell.

In the nsclient.ini configuration file, define the script like this:

[/settings/external scripts/scripts]
check_ms_win_tasks=cmd /c echo scripts/powershell/check_ms_win_tasks.ps1 $ARG1$; exit $LastExitCode | powershell.exe /noprofile -command -

1 2	[/settings/external scripts/scripts] check_ms_win_tasks=cmd /c echo scripts/powershell/check_ms_win_tasks.ps1 $ARG1$; exit $LastExitCode \| powershell.exe /noprofile -command -

For more information about external scripts configuration, please review the NSClient documentation. You can also consider defining a wrapped script in nsclient.ini to simplify configuration.

Make a command in Nagios like this:

check_ms_win_tasks => $USER1$/check_nrpe -H $HOSTADDRESS$ -p 5666 -t 60 -c check_ms_win_tasks $ARG1$

1

check_ms_win_tasks => $USER1$/check_nrpe -H $HOSTADDRESS$ -p 5666 -t 60 -c check_ms_win_tasks $ARG1$
Configure your service in Nagios. Make use of the above created command. Configure something similar like this as $ARG1$:

-a '-H localhost -EF Microsoft, Backup_Maintenance -ET Jeff, Fimacs'

1

-a '-H localhost -EF Microsoft, Backup_Maintenance -ET Jeff, Fimacs'

Some things to consider to make it work:

“set-exectionpolicy remotesigned”
Nscp service account permissions => Running with local system should suffice, but I had users telling me it only worked with a local admin. I found out that on some NSClient++ versions, more specific version 0.4.3.88 and probably some earlier versions too, the following error occured when running nscp service as local system: “CHECK_NRPE: Invalid packet type received from server”. After filing an issue on the GitHub project page of NSClient++, Michael Medin quickly acknowledged the issue and solved it from version 0.4.3.102, so the plugin should work again as local system.

Examples

If you would run the script in cli from you Nagios plugin folder, this would be the command:

./check_nrpe -H x.x.x.x -p 5666 -c check_ms_win_tasks -a '-H localhost -EF Microsoft'

1	./check_nrpe -H x.x.x.x -p 5666 -c check_ms_win_tasks -a '-H localhost -EF Microsoft'

If you would want to exclude one noisy unimportant scheduled task, the command used in cli would look like this:

./check_nrpe -H x.x.x.x -p 5666 -c check_ms_win_tasks -a '-H localhost -ET scheduled_task_to_exclude'

1	./check_nrpe -H x.x.x.x -p 5666 -c check_ms_win_tasks -a '-H localhost -ET scheduled_task_to_exclude'

If you only want the scheduled tasks in the root to be monitored, you can use this command:

./check_nrpe -H x.x.x.x -p 5666 -c check_ms_win_tasks -a '-H localhost -IF "\\"'

1	./check_nrpe -H x.x.x.x -p 5666 -c check_ms_win_tasks -a '-H localhost -IF "\\"'

This would only give you the scheduled tasks available in the root folder. The output look like this now.

1 / 11 tasks failed! {Taskname: "check_ms_win_tasks_subfolder_01_task" (Author: F1REBASE\Willem)(Exitcode: -2147023673)(Last runtime: 12/06/2014 11:35:56)} 1 / 11 tasks still running! {Taskname: "SamsungMagician" (Author: SamsungMagician)(Exitcode: 267009)(Last runtime: 06/21/2015 11:37:55)} | 'Total Tasks'=11, 'OK Tasks'=9, 'Failed Tasks'=1, 'Running Tasks'=1

1 / 11 tasks failed! {Taskname: "check_ms_win_tasks_subfolder_01_task" (Author: F1REBASE\Willem)(Exitcode: -2147023673)(Last runtime: 12/06/2014 11:35:56)} 1 / 11 tasks still running! {Taskname: "SamsungMagician" (Author: SamsungMagician)(Exitcode: 267009)(Last runtime: 06/21/2015 11:37:55)} | 'Total Tasks'=11, 'OK Tasks'=9, 'Failed Tasks'=1, 'Running Tasks'=1

Final Words

It seems the perfdata in the Highcharts graphs sometimes contains decimal numbers (see screenshot), which is kind of strange as I’m sure I only pass rounded numbers. Seems this is related to the way RRD files are working. To reduce the amount of storage space used, NPCD and RRD while average out the data, resulting in decimals, even when you don’t expect them.

This is a small to do list:

Add switches to change returned values and output.
Add array parameter with exit codes that should be excluded.
Test remote execution. In some cases it might be useful to be able to check remotely for failed windows tasks.
Include a warning / critical threshold when discovered tasks exceed a certain duration.
I was hoping to add some more exit codes to check, which would make failed tasks easier to troubleshoot. You can find the list of scheduled task exit codes here. The constants that begin with SCHED_S_ are success constants, and the constants that begin with SCHED_E_ are error constants.

Screenshots:

These are some screenshots of the Nagios XI Graph Explorer for two of our servers making use of the plugin to monitor scheduled tasks: Let me know on the Nagios Exchange what you think of my plugin by rating it or submitting a review. Please also consider starring the project on GitHub.

Willem

Monitoring NetApp Ontap

1 May 201612/02/2017willemdh

Download Latest

Introduction

I’d like to start with thanking the original developer, John Murphy. Thanks to his plugin to monitor NetApp Ontap storage, we don’t need to buy the expensive NetApp plugin from Quorum. He also inspired me to continue developing Nagios plugins in my free time. So if you want to monitor your NetApp Ontap Cluster, this plugin could help you do that. It is written in Perl and is not being actively developed, like my other Powershell plugins, as my Perl knowledge is less evolved then my Powershell knowledge. Somestimes user send me a piece of code to add, if you want to test the latest additions, give the dev branch a try. All help is welcome to improve this plugin. Read the post about debugging Perl scripts, make a fork of the project on Github and start experimenting.

The plugin is able monitor NetApp Ontap components, from disk to aggregates to volumes and alert you if if finds any unhealthy components.

How to monitor your Netapp Ontap?

Download the latest release from GitHub to a temp directory and then navigate to it.
Copy the contents of NetApp/* to your /usr/lib/perl5 or /usr/lib64/perl5 directory to install the required version of the NetApp Perl SDK. (confirmed to work with SDK 5.1 and 5.2)
Copy check_netapp_ontap.pl script to your nagios libexec folder and configure the correct permissions

Parameters:

–hostname, -H => Hostname or address of the cluster administrative interface.

–node, -n => Name of a vhost or cluster-node to restrict this query to.

–user, -u => Username of a Netapp Ontapi enabled user.

–password, -p => Password for the netapp Ontapi enabled user.

–option, -o => The name of the option you want to check. See the option and threshold list at the bottom of this help text.

–warning, -w => A custom warning threshold value. See the option and threshold list at the bottom of this help text.

–critical, -c => A custom warning threshold value. See the option and threshold list at the bottom of this help text.

–modifier, -m => This modifier is used to set an inclusive or exclusive filter on what you want to monitor.

–help, -h => Display this help text.

Option list:

volume_health: Check the space and inode health of a vServer volume on a NetApp Ontap cluster. If space % and space in *B are both defined the smaller value of the two will be used when deciding if the volume is in a warning or critical state. This allows you to accomodate large volume monitoring better. thresh: space % used, space in *B (i.e MB) remaining, inode count remaining, inode % used (Usage example: 80%i), “offline” keyword node: The node option restricts this check by vserver name.

aggregate_health: Check the space and inode health of a cluster aggregate on a NetApp Ontap cluster. If space % and space in *B are both defined the smaller value of the two will be used when deciding if the volume is in a warning or critical state. This allows you to better accomodate large aggregate monitoring. thresh: space % used, space in *B (i.e MB) remaining, inode count remaining, inode % used (Usage example: 80%i), “offline” keyword, “is-home” keyword node: The node option restricts this check by cluster-node name.

snapshot_health: Check the space and inode health of a vServer snapshot. If space % and space in *B are both defined the smaller value of the two will be used when deciding if the volume is in a warning or critical state. This allows you to better accomodate large snapshot monitoring. thresh: space % used, space in *B (i.e MB) remaining, inode count remaining, inode % used (Usage example: 80%i), “offline” keyword node: The node option restricts this check by vserver name.

quota_health: Check that the space and file thresholds have not been crossed on a quota. thresh: N/A storage defined. node: The node option restricts this check by vserver name. snapmirror_health: Check the lag time and health flag of the snapmirror relationships. thresh: snapmirror lag time (valid intervals are s, m, h, d). node: The node options restricts this check by snapmirror destination cluster-node name.

filer_hardware_health: Check the environment hardware health of the filers (fan, psu, temperature, battery). thresh: component name (fan, psu, temperature, battery). There is no default alert level they MUST be defined. node: The node option restricts this check by cluster-node name. port_health: Checks the state of a physical network port. thresh: N/A not customizable. node: The node option restricts this check by cluster-node name.

interface_health desc: Check that a LIF is in the correctly configured state and that it is on its home node and port. Additionally checks the state of a physical port. thresh: N/A not customizable. node: The node option restricts this check by vserver name.

netapp_alarms: Check for Netapp console alarms. thresh: N/A not customizable. node: The node option restricts this check by cluster-node name. cluster_health desc: Check the cluster disks for failure or other potentially undesirable states. thresh: N/A not customizable. node: The node option restricts this check by cluster-node name. disk_health: Check the health of the disks in the cluster. thresh: Not customizable yet. node: The node option restricts this check by cluster-node name. For keyword thresholds, if you want to ignore alerts for that particular keyword you set it at the same threshold that the alert defaults to.

OutsideIT

Monitoring and Automation to Invent Your Own Future.

Tag: threshold

Monitoring Windows Scheduled Tasks

Download Latest

Introduction

Requirements

Versions

Help

How to monitor your scheduled tasks?

Examples

Final Words

Screenshots:

Monitoring NetApp Ontap

Download Latest

Introduction

How to monitor your Netapp Ontap?