Monitoring Microsoft IIS Application Pools

Introduction

For those who are not aware, IIS is a HTTP web server from Microsoft which can host both static and dynamic content. This is done by a Windows kernel-mode driver named http.sys. It listens for incoming TCP requests on a configured port, performs some basic security checks and passes the request to a user-mode process. The worker fulfills the request and sends the response back to the requester. Web application are grouped into IIS application pools which has it’s own process assigned to it.

As we are migrated al our IIS applications to a new IIS 8.5 farm on Windows 2012 R2 servers, we needed a way to reliably monitor the state of our most critical IIS application pools. So I created a Powershell script which is able to check the state of an application pool and count the number of web application using it. As each IIS application pool has one w3wp.exe IIS worker process assigned, I added the % processor usage and memory usage to the perfdata.

The latest version also contains a new method to retrieve the IIS application pool information. As Get-ChildItem IIS:\AppPools has a weird bug where the command hangs sometimes I had to look for an alternative. This method uses C:\Windows\system32\inetsrv\appcmd.exe   instead, which seems much more performant.  

How to monitor your MS IIS Application Pools with Nagios?

  • Put the script in the NSClient++ scripts folder, preferably in a subfolder Powershell.
  • In the nsclient.ini configuration file, define the script like this:
  • Make a command in Nagios like this:
  • Configure your service in Nagios. Make use of the above created command. Configure something similar like this as $ARG1$:

    Or if you want to monitor an application pool which has OnDemand startmode where there is no IIS worker process when it isn’t used.

    IIS application pools OnDemand Startmode
    When you want to use the AppCmd.exe method:

Final Words

I only had the chance to test this on a Windows Server 2012 R2. It’s very possible you will experience issues on lower IIS versions. You need to install the IIS Management Scripts and Tools feature for the script to work properly.

IIS Application Pool

When you got it up and running your Nagios server should look like this:

monitoring iis application pools

 

Safer Internet Day

What is Safer Internet Day?

Today, Tuesday 07 / 02 17 is Safer Internet Day! This initiative debuted in 2005 to raise awareness of emerging online issues.

This year’s theme is:

Be the change: Unite for a better Internet

All over the world, events and activities are taking place to ‘celebrate’. Register here for detailed information on this and future ‘Safer Internet Day’ events. Or follow #SaferInternetDay on Twitter or Facebook and support this cool and necessary initiative!

Safer Internet Day

Basic Security Guidelines

Strong Password Policy

  • Use strong passwords
  • Use different passwords on every website
  • Use a password manager such as KeePass to securely store your passwords

Updated Software

Always update the software you are using to the latest version as soon as possible. It doesn’t really matter which operating system you are using, those updates are released for a reason. If possible, configure your systems to update automatically. That way you won’t forget it!

Sensitive Information

Do not enter sensitive information when you are not browsing on an encrypted website. Always check the url you are browsing. Is it ‘green’ and does it starts with https? That means all the traffic from and to this website is encrypted and can’t be sniffed. Make sure you entered the correct url. Hackers are able to create scam websites that look exactly like the real thing. 

General Internet Guidelines

Be the change

Make the Internet a great place for all. 

Be kind

Think carefully about the impact on others before sharing something online. Make sure you have a positive impact!

Be you

Think before you share something online. What you share online could be there forever, can be misinterpreted or could reveal personal information about you. 

Be a digital citizen

Report anything you see online, including images and videos, which are offensive, upsetting or inappropriate.

Be a critical thinker

Seeing is not believing… When you see something online take a moment to see the full picture. Not everything or everyone online can be trusted.

Be safe

Wherever you go, make sure you are browsing the Internet in a secure way.

SSLLabs A+ Rating for Let’s Encrypt on CentOS 7

Introduction

This is a small groowing blog post about obtaining an A+ rating with Apache 2.4.6 or higher on a freshly installed CentOS 7 with Let’s Encrypt certificates.  Several blog posts claim getting  an A+ rating on SSLLabs isn’t possible without HPKP (HTTP Public Key Pinning). This actually isn’t true. It’s perfectly possible to get an A+ rating by just enabling HSTS and OCSP Stapling, which are both easy to implement (compared to HPKP). 

chrome_2016-07-22_19-22-54

rating

Contrary to what some blogs are posting, you also don’t need to use a 4096 bit key. This will definitely save you some resources, which is proved in this blog post from CertSimple:

4096 bit handshakes are indeed significantly slower in terms of CPU usage than 2048 bit handshakes.

HTTP Public key pinning (HPKP) is a security mechanism which only protects against a relatively rare MitM attack that’s very hard to pull off. Someone would have to impersonate you with a fraudulent certificate generated via a Certificate Authority that your browser already trusts.

But if misconfigured, it can really brick your website.  And it is not supported by Internet Exploter and Edge as you can see here in the Microsoft Edge Platform Status. HPKP is alsonot recommend with Let’s Encrypt and has a lot of additional requirements. This is explained by Peter Eckersley from the EFF in this blog post.

Let’s Encrypt

These days, encryption is a necessary part of any self-respecting IT organization. If your traffic is unencrypted, you (or your visitors) are almost certainly being monitored by someone in some government somewhere on our little planet. If not by the so-called “Five Eyes” (Australia – Canada – New Zealand – United Kingdom – United States), then some other government organization owned by the Russians, the Chinese or an other less known security agency or hacker group might have you on their radar. Luckily since 3 December 2015 Let’s Encrypt entered public beta. (not that this means you are suddenly safe 😀 )

The Law

If you want information about cryptography laws in your country, please consult http://www.cryptolaw.org. It has a very detailed list of existing and proposed laws and regulations on cryptography for almost any country.

Owned by?

Let’s Encrypt is a free, automated, and open certificate authority (CA), run for the public’s benefit. Let’s Encrypt is a service provided by the Internet Security Research Group (ISRG). ISRG is a California public benefit corporation. Major sponsors are the Electronic Frontier Foundation (EFF), the Mozilla Foundation, Akamai and Cisco Systems. Other partners include the certificate authority IdenTrust, the University of Michigan, the Stanford Law School and the Linux Foundation. Not the smallest organizations aren’t they?

Principles

The key principles behind Let’s Encrypt are:

  • Free: Anyone who owns a domain name can use Let’s Encrypt to obtain a trusted certificate at zero cost.
  • Automatic: Software running on a web server can interact with Let’s Encrypt to painlessly obtain a certificate, securely configure it for use, and automatically take care of renewal.
  • Secure: Let’s Encrypt will serve as a platform for advancing TLS security best practices, both on the CA side and by helping site operators properly secure their servers.
  • Transparent: All certificates issued or revoked will be publicly recorded and available for anyone to inspect.
  • Open: The automatic issuance and renewal protocol will be published as an open standard that others can adopt.
  • Cooperative: Much like the underlying Internet protocols themselves, Let’s Encrypt is a joint effort to benefit the community, beyond the control of any one organization.

Do you need any more convincing? For years people have been paying far too much for their SSL certificates to security companies such as Comodo, GlobalSign, Godaddy, Thawte and others. I have never really understood why they cost so much. And why would we trust them? Remember DigiNotar?

DigiNotar was a Dutch certificate authority owned by VASCO Data Security International. On September 3, 2011, after it had become clear that a security breach had resulted in the fraudulent issuing of certificates, the Dutch government took over operational management of DigiNotar’s systems. That same month, the company was declared bankrupt. After more than 500 fake DigiNotar certificates were found, major web browser makers reacted by blacklisting all DigiNotar certificates. The scale of the incident was used by some organizations like ENISA andAccessNow.org to call for a deeper reform of HTTPS in order to remove the weakest link possibility that a single compromised CA can affect that many users.

Support

An initiative such as Let’s Encrypt, carried and supported by a huge number of people and organizations is in my humble opinion a much safer option then trusting your certificates in the hand of ‘relatively small companies’. All those so called security companies claim to be secure, but in the meantime fail to deliver transparent and open protocols. Your security might be in compromised  by just one overworked individual failing to do his job properly.

But what can other Certificate Authorities offer that Let’s Encrypt can’t?

There are three types of SSL certificates: Domain Validated (DV), Organization Validated (OV) and Extended Validation (EV). To get a DV cert you only need prove that you control the domain for which the certificate is assigned. For an OV cert the CA checks with third parties to ensure that the name of the applying organization is the same as that which owns the domain. For an EV cert, the kind that turn your browser address bar green, you need to provide much more extensive documentation, and there are no personal EV certs. The very fact that the Let’s Encrypt process is automated means that they will not be able to offer anything other than DV certificates. To many companies this isn’t enough. You should use Let’s Encrypt:

  • If you are running your own web server
  • If you have a registered publically accessible domain name

You should not use Let’s Encrypt with:

  • Shared web hosting
  •  You want to keep the existence of your certificate a secret
  • Wildcard certificates
  • Long-lived certificates
  • Extended Validation
  • If you want your certificate to be trusted by older software

SSLLabs Rating Methodology

SSLLab’s approach for calculating the rating consists of four steps.

Certificate Inspection

They look at the certificate to verify that it is valid and trusted. Server certificates are often the weakest point of an SSL server configuration. Certificates that aren’t trusted fail to prevent MITM attacks.

Any of the following certificate issues immediately result in a zero score:

  • Domain name mismatch
  • Certificate not yet valid
  • Certificate expired
  • Self-signed certificate
  • Use of a certificate that is not trusted (unknown CA or some other validation error)
  • Revoked certificate
  • Insecure certificate signature (MD2 or MD5)
  • Insecure key

So how are the Let’s Encrypt certificates doing? Let’s Encrypt’s intermediate is signed by ISRG Root X1. However, since they are a very new certificate authority, ISRG Root X1 is not yet trusted in most browsers. In order to be broadly trusted right away, their intermediate is also cross-signed by another certificate authority, IdenTrust, whose root is already trusted in all major browsers. Specifically, IdenTrust has cross-signed our intermediate using their DST Root CA X3.

Scoring

They inspect the server configuration in three categories. The category scores are combined into an overall score expressed as a number between 0 and 100. A zero in any category will push the overall score to zero. 
They then apply a series of rules to handle some aspects of server configuration that cannot be expressed via numerical scoring. Most rules will reduce the grade (to A-, B, C, D, E, or F) if they encounter an unwanted feature. Some rules will increase the grade (to A+), to reward exceptional configurations.

Protocol support

SSLLab will look  at the protocols supported by an SSL server. For example, both SSL 2.0 and SSL 3.0 have known weaknesses. Because a server can support several protocols, they use the following algorithm to arrive to the final score:

  1. Start with the score of the best protocol
  2. Add the score of the worst protocol
  3. Divide the total by 2
ProtocolScore
SSL 2.00%
SSL 3.080%
TLS 1.090%
TLS 1.195%
TLS 1.2100%

Key exchange support

The key exchange phase serves two functions. The first phase is done with asymetric encryption. It performs authentication, allowing at least one party to verify the identity of the other party. The other is to ensure the safe generation and exchange of the secret keys that will be used during the remainder of the session. The weaknesses in the key exchange phase affect the session in two ways:

  • Key exchange without authentication allows an active attacker to perform a MITM attack, gaining access to the complete communication channel.
  • Most servers also rely on public cryptography for the key exchange. Thus. the stronger the server’s private key, the more difficult it is to break the key exchange phase.
Key exchange aspect Score
Weak key (Debian OpenSSL flaw) 0%
Anonymous key exchange (no authentication) 0%
Key or DH parameter strength < 512 bits 20%
Exportable key exchange (limited to 512 bits) 40%
Key or DH parameter strength < 1024 bits (e.g., 512)40%
Key or DH parameter strength < 2048 bits (e.g., 1024)80%
Key or DH parameter strength < 4096 bits (e.g., 2048) 990%
Key or DH parameter strength >= 4096 bits (e.g., 4096)100%

Cipher support

To break a communication session, an attacker can attempt to break the symmetric cipher used for the bulk of the communication. A stronger cipher allows for stronger encryption and thus increases the effort needed to break it. Because a server can support 7 ciphers of varying strengths, SSLLab penalizes the use of weak ciphers.

Cipher strength Score
0 bits (no encryption) 0%
< 128 bits (e.g., 40, 56) 20%
< 256 bits (e.g., 128, 168) 80%
>= 256 bits (e.g., 256) 100%

You can find the SSL server rating guide here.

Requirements for getting an A+ rating on SSLLabs

Use any of this code at your own risk. Do no use it on a production webserver. Do not use it if you don’t know what you are doing. 

Please replace any variables such as $Hostname and $Email with values.

OpenSSL

Create SSL Configuration

Let’s Encrypt

I’m assuming you have ran CertBot to generate your Let’s Encrypt certificates. If not, start with installing the python-certbot-apache yum package:

Then run

Apache

Make sure httpd and mod_ssl yum packages are installed on the server.

You can check your Apache version like this:

Enable the httpd service:

Create the webroot:

Set webroot owner:

Furthermore set the webroot permissions:

Create demo page:

Create vhost directory /etc/httpd/sites-available

Equally create vhost directory /etc/httpd/sites-enabled

Add sites-enabled/*.conf to httpd.conf:

Create Apache vhost:

Link Apache vhost:

Restart Apache:

In addition add the firewalld https service:

Finally restart firewalld:

Final Words

First of all, let me know if I forgot something. As much as I tried to make the article as accurate as possible, I might have made a mistake. Qualys is also continuously improving their tests and rating methods. As a result some parts of this article might no longer be valid.

Furthermore, please note that an A+ rating is probably not ideally for every web application. It’s not because you have received and A- or A rating that your web application is suddenly vulnerable for any malicious attack. Different websites have different needs, which means that there is not a ‘perfect’ configuration that works for everyone. 

I would like to suggest this GitHub project from SSLLabs which contains some very useful recommendations about SSL and TLS deployments.

Please let me know if I something I wrote is incorrect.

Rundeck 2.10 – Ultimate Open Source Job scheduler

Rundeck Review

June 2016, Nagios announced they were stopping development on Nagios Reactor. So I had to start looking for a replacement. After playing with Foreman, Jenkins, Rundeck and Stackstorm, I decided the best solution for my needs was definitely Rundeck. In this Rundeck review, I’ll try to go into detail on some of the most useful Rundeck features I’ve been using over the last years.

Rundeck Review

Rundeck was definitely a hidden gem in the open source automation landscape, which has been dominated by configuration management oriented tools, such as Ansible, Chef, Puppet and Salt. But imho we don’t always need full configuration management. Usage of a job scheduler and orchestrator is in a lot of cases a more suitable option. And an added bonus is that Rundeck integrates with Ansible thanks to this plugin.

Rundeck is being very actively developed, meaning they regularely release new features. The nice thing is that they truly listen to their community, by allowing us to vote for popular features in a Trello board. Feel free to create an ccount and vote for the features you think deserve priority development time.

So what if you want professional support? Then you can opt into Rundeck Pro, which has some additional features and pro plugins available. Ok, I hope this Rundeck review helps you take a better informed decision on which automation platform to start using in your digital transformation.

Rundeck Projects and Jobs

Rundeck projects will contain definitions about nodes, as well as a set a jobs that reference these nodes. Using access control policies allows you to choose which teams have access to perform actions on jobs. Each node in the Rundeck project can be customized with tags, allowing you to target each kind of node rather than reference specific hosts names or IP addresses. All these Rundeck features allow you to create job libraries with useful scripts. Integrating The Rundeck access, job and exeecution logs into an Elastic stack gives you full visibility of what’s happening in your Rundeck server.

You can group Rundeck jobs in folders and subfolders. A collapsed view of all jobs in my DAF project:

 

Rundeck Security

Please note I’m just listing a few security related topics in this Rundeck review. Please refer to the official Rundeck documentation for all information you need to setup a secure Rundeck instance.

Active Directory integration

Active Directory integration is a basic requirement for any automation tool. Using Active Directory groups allows you to group users and assign specific permissions to them. Please refer to the official Rundeck documentation if you want more information how to configure this.

Agentless SSH based automation

A critical feature of any automation tool is a way to encrypt it’s traffic. As RunDeck uses SSH for executing commands on nodes, it already has a big advantage over other protocols. SSH is a secure protocol used as the primary means of connecting to Linux servers remotely. When you connect, you will be dropped into a shell session, which is a text-based interface where you can interact with your server. For the duration of your SSH session, any commands that you type into your local terminal are sent through an encrypted tunnel and executed on your server. Clients generally authenticate either using passwords (less secure and not recommended) or SSH keys, which are very secure.

SSL / HTPS

The RunDeck URL also needs to be protected, otherwise attackers could easily sniff your network and extract usernames, passwords, job options and more from api calls or logins. This procedure decribes the steps that need to be taken in order to configure SSL for your RunDeck server. I decided to create my ow version of the official documentation, but it’s only applicable to Microsoft .pfx certificates.

SSL

How to configure SSL for RunDeck?

  • Generate a .pfx server certificate with your private root ca
  • Copy the generated server certificate <servername>.pfx to /etc/rundeck/ssl
  • Create a keystore to hold the server certificate <servername>.pfx

  • Retrieve the alias from the <servername>.pfx file

  • Import the Certificate and Private Key into the Java keystore

  • Create a keystore for the CA certificate

  • Add the CA certificate to the CA keystore

  • Edit /etc/rundeck/ssl/ssl.properties and update all properties with their current values:

  • Edit /etc/rundeck/profile and uncomment:

  • Edit /etc/rundeck/rundeck-config.properties

  • Edit /etc/rundeck/framework.properties

  • Make sure port 4443 is opened in the firewall:

  • Restart the rundeckd daemon

  • Tail the RunDeck logs to make sure everything works fine:

Final words

I’d love to give a big thanks to the Rundeck developers for making Rundeck available to the public. I’m sorry if important stuff is missing in this (basic) Rundeck review, I’ll try to add more information over time. It’s also on my to do to open source my Elastic pipeline configurations, which enable analytics on the access, job and execution logs.

Monitor RaspBerry Pi with Nagios

Introduction

Over the past week, I had multiple questions how to monitor RaspBerry Pi with Nagios. Monitoring is crucial to pro-actively  find out any issues that might come up. There are multiple ways to achieve this. I’ll try to build up this ‘how to’ from the ground, starting with using the standard traditional method, which is using the official Nagios NRPE Agent.

NSClient++ does not yet support Raspbian for now. Michael Medin told me in this forum thread that he is planning to port it once he finds some spare time.

It’s also possible to install Go and Telegraf on your Raspbian, but I haven’t got the time to test that. 

How to Monitor RaspBerry Pi with NRPE Agent?

The code below worked fine for me on Raspbian Jessie

Create nrpe.cfg in /usr/local/nagios/etc

The relevant part of my nrpe.cfg looks like this:

make sure to replace <ip-of-your-Nagios-server-here> with (you never guess) the ip of your Nagios server.

Let me know if you experience any issues.

Grtz

Willem