CentOS 7 – An Enterprise Ready Problemless OS

Introduction

It must be about 8 years now since we choose CentOS as our default operating system for Linux servers. A lot has changed since then and it has always been on my to do to write a blog post about it. Karanbir Singh announced the release of CentOS 7.4.1708 on 13/09/17. As with all CentOS 7 components, this release was built from sources hosted at git.centos.org. It also supersedes all previously released content for CentOS Linux 7, and users are highly encouraged to upgrade all systems running CentOS 7. Make sure to read the release notes before upgrading.

One month later, we were able to patch all our CentOS 7 systems and did not run into a single upgrade problem. I would say that merits a big congratulations to the whole CentOS team, and of course also all Red Hat engineers for producing a problemless and stable distribution.

In this post I’ll try to give a general overview of what CentOS is about and why you should choose for this partcular operating system.

centos 7

CentOS Lifecycle

It’s very important to keep an eye on the lifecycles of the operating systems you are managing. Good planning ensures you have enough time to migrate your applications in time before your operating systems are no longer supported. 

CentOS VersionRelease DateFull UpdatesMaintenance Updates
319 March 200420 July 200631 October 2010
49 March 200531 March 200929 February 2012
512 April 200731 January 201431 March 2017
610 July 201110 May 201730 November 2020
77 July 2014Q4 202030 June 2024

Source: https://en.wikipedia.org/wiki/CentOS 

CentOS 7 Repositories

There are three primary CentOS repositories (also known as channels), containing software packages that make up the main CentOS distribution:

  • base – Contains packages that form CentOS point releases, and gets updated when the actual point release is formally made available in form of ISO images.
  • updates – Contains packages that serve as security, bugfix or enhancement updates, issued between the regular update sets for point releases. 
  • addons – provides packages required for building the packages that make up the main CentOS distribution, but are not provided by the upstream.

CentOS vs Red Hat Enterprise Linux

While CentOS is derived from the Red Hat Enterprise Linux codebase, CentOS and Red Hat Enterprise Linux are distinguished by divergent build environments, QA processes, and, in some editions, different kernels and other open source components. For this reason, the CentOS binaries are not the same as the Red Hat Enterprise Linux binaries. Red Hat Enterprise Linux (RHEL) is actually also open source. But although the code is available for Red Hat users, it is not free to use. Red Hat and the CentOS project announced 7 January 2014 they were actually joining forces.

 CentOSRHEL
License FOSS – GPL and othersCommercial – RedHat EULA
SecuritySELinux, NSS, Linux PAM, firewalld SELinux, NSS, Linux PAM, firewalld
Patches/fixesAs promptly as possible given available project resources.SLA through Red Hat
SupportSelf-support24x7 support through Red Hat
Package managementYumYum
Enterprise package managementSpacewalk / KatelloRed Hat Satellite
ClusteringLinux-HARed Hat Cluster Suite (RHCS)
BootloaderGRUB 2GRUB 2
Graphical user interface (GUI)GNOME 3 / KDE SC 4.10GNOME 3 / KDE SC 4.10
Service managementsystemdsystemd
Storage managementLVM / SSM LVM / SSM
Default file systemXFSXFS
ContainerizationDocker, KubernetesRed Hat OpenShift
Virtual device interface (VDI)SPICESPICE

red hat

There are a lot of advantages in choosing Red Hat 7 over CentOS 7. 

  • Enterprise-level support
  • Access to engineering resources
  • Red Hat’s Customer Portal
  • Certifications
  • Latest features

But choosing Red hat also has some considerable disadvantages:

  • Not free
  • Administration overhead for license management

And yes, I do mention the administration overhead as a problem. This problem might not apply for everyone though. In my case though the process of ordering new Red Hat licenses or prolonging expiring licenses just takes a lot of (unnecessary) time. 

Final words

So I hope my blog post gave you some additional information to make a better informed decision which operating systems are best suited for your use case. If you need professional support, Red Hat is there for you, if you feel comfortable supporting your own Linux servers, follow the CentOS rabbit. 

F5 Logstash Filter (apd, dcc, tmm)

Introduction

It has been some time since I gave my F5 Logstash filter an update. As I learned a lot of new things and techniques over the past six months, it was on my ‘short-term’ to do list to give them a major upgrade. A BIG-IP F5 load balancer is a device that acts as a reverse proxy and distributes network or application traffic across a number of servers. Load balancers are used to increase capacity and reliability of applications. They improve the overall performance of applications by decreasing the burden on servers associated with managing and maintaining application and network sessions, as well as by performing application-specific tasks.F5 Logstash

Load balancers are generally grouped into two categories: Layer 4 and Layer 7. Layer 4 load balancers act upon data found in network and transport layer protocols (IP, TCP, FTP, UDP). Layer 7 load balancers distribute requests based upon data found in application layer protocols such as HTTP. Requests are received by both types of load balancers and they are distributed to a particular server based on a configured algorithm. Some industry standard algorithms are:

  • Round robin
  • Weighted round robin
  • Least connections
  • Least response time

Layer 7 load balancers can further distribute requests based on application specific data such as HTTP headers, cookies, or data within the application message itself, such as the value of a specific parameter. Load balancers ensure reliability and availability by monitoring the “health” of applications and only sending requests to servers and applications that can respond in a timely manner.

You can send your F5 logs to an F5 Logstash filter to get a grip on what’s going on in your load balancer. I’m not a 100 % sure if all F5 Big IP load balancers have the same syslog syntax, but I put the F5 Logstash filters I created on GitHub and give something back to the Elastic community. It’s not finished yet and definitely needs some work, but it’s better then a default syslog filters. It contains one global syslog F5 Logstash filter which parses the first piece of the F5 syslogs which contains things like ‘logsource’ ‘severity_label’ and labels the rest of the message as ‘info’. Thanks to Jesse from Nagios for helping me create the dcc filter in Nagios Log Server and Jens for helping me with F5.

Logstash configuration

F5 Logstash input

F5 Logstash filters

dcc => ASM related messages. BIG-IP Application Security Manager (ASM) enables organizations to protect against OWASP top 10 threats, application vulnerabilities, and zero-day attacks. Leading Layer 7 DDoS defenses, detection and mitigation techniques, virtual patching, and granular attack visibility thwart even the most sophisticated threats before they reach your servers.

apd => Access Policy Demon. The apd process runs a BIG-IP APM access policy for a user session.

tmm => The traffic management microkernel is the process running on the BIG-IP host O/S that performs all of the local / global traffic management for the system.

sshd => The ssh daemon provides remote access to the BIG-IP system command line interface

F5 Logstash custom grok patterns

You will need to add these F5 Logstash custom grok patterns to your Logstash patterns directory. For me it’s located in /etc/logstash/patterns

Elasticsearch configuration

Included in the GitHub project you can find my f5 elasticsearch template, with the correct mappings for each field. This enables you to use your data more efficiently and allow for advanced ip aggregations. You can find more information about mapping types here. If you have ideas about better mappings (I know they need some work), please let me know on GitHub by making an issue.

Greetings

Willem

Infoblox Logstash filter (named, dhcpd and httpd)

Introduction

Infoblox is a DDI (DNS, DHCP, and IP address management solution) which simplifies network management a lot. Over the past 8 years I was able to work with it and never looked into another solution, as it completely fulfills all our DNS and DHCP needs. During that time, I’ve been finetuning my Infoblox Logstash grok patterns and index template mappings. As I didn’t found any existing Infoblox Logstash grok patterns, I decided to make them open source. You can download the Logstash configuration file on GitHub here. There is also a template included with the mappings for Elasticsearch. 

Infoblox Logstash

Infoblox Logging

Thanks to Infoblox, we can:

  • Consolidate DNS, DHCP, IP address management, and other core network services into a single platform, managed from a common console
  • Centrally orchestrate DDI functions across diverse infrastructure
  • Boost IT efficiency and automation by seamlessly integrating with other IT systems (such as Rundeck) through RESTful APIs

Infoblox has integrated reporting & analytics capabilities, but imho DNS and DHCP related logs are on the top priority list for sending to a log aggregator, such as Elasticsearch or NLS. DHCP and DNS logs allow us to link ip addresses to device hostnames and mac addresses. As ip addresses are logged everywhere, this is a vital log source in order to trace what happened by who on your network. A good Logstash filter is able to parse all the relevant fields, so they can be used in aggregations. 

Infoblox Logstash Configuration

Please note hat I’m not using a syslog input, but a tcp input. I’ve had considerable issue with the default syslog patterns used by Elasticsearch.  Apart from that I prefer to apply my own field names for syslog data. Using my own custom syslog grok pattern allows me to match the parsed field to our internally used naming conventions. Feel free to adjust the field names as needed.

 

 

 

 

Realmd and SSSD Active Directory Authentication

Introduction to SSSD and Realmd

Starting from Red Hat 7 and CentOS 7, SSSD or ‘System Security Services Daemon’  and realmd have been introduced. SSSD’s main function is to access a remote identity and authentication resource through a common framework that provides caching and offline support to the system. SSSD provides PAM and NSS integration and a database to store local users, as well as core and extended user data retrieved from a central server. 

The main reason to transition from Winbind to SSSD is that SSSD can be used for both direct and indirect integration and allows to switch from one integration approach to another without significant migration costs. The most convenient way to configure SSSD or Winbind in order to directly integrate a Linux system with AD is to use the realmd service. Because it allows callers to configure network authentication and domain membership in a standard way. The realmd service automatically discovers information about accessible domains and realms and does not require advanced configuration to join a domain or realm.

The realmd system provides a clear and simple way to discover and join identity domains. It does not connect to the domain itself but configures underlying Linux system services, such as SSSD or Winbind, to connect to the domain.

Realmd Pam SSSD

Please read through this Windows integration guide from Red Hat if you want more information. This extensive guide contains a lot of useful information about more complex situations.

Realmd / SSSD Use Cases

How to join an Active Directory domain?

  1. First of all start you will need to install the required packages:
  2. Configure ntp to prevent time sync issues:
  3. Join the server to the domain:
  4. Also add the default domain suffix to the sssd configuration file:

    Add the following beneath [sssd]

  5. Finally move the computer object to an organizational unit in Active Directory.

How to leave an Active Directory domain?

I saw multiple times that although the computer object was created in Active Directory it was still not possible to login with an ad account. The solution was each time to remove the server from the domain and then just add it back.

How to permit only one Active Directory group to logon

As it can be very useful to only allow one Active Directory group. For example a group with Linux system administrators.

 How to give sudo permissions to an Active Directory group

Add

Or

Example sssd.conf Configuration

The following is an example sshd.conf configuration file. I’ve seen it happen once that somehow access_provider was set to ad. I haven’t got the chance to play with that setting, as simple worked almost every time for now.

Required security permissions in AD

A few months ago, we had a problem where some users were no longer able to authenticate. After an extended search we discovered the reason was a hardening change in permissions on some ou’s in our AD. My colleague Jenne and I discovered that the Linux server computer objects need minimal permissions on the ou which contains the users that want to authenticate on your Linux servers. After testing almost all obvious permissions, we came to the conclusions that the computer objects need “Read remote access information”!

sssd-permissions-ras

How to debug SSSD and realmd?

The logfile which contains information about successful or failed login attempts is /var/log/secure. It contains information related to authentication and authorization privileges. For example, sshd logs all the messages there, including unsuccessful login. Be sure to check that logfile if you experience problems logging in with an Active Directory user. 

How to clear the SSSD cache?

As suggested by AP in the comments, you can manage your cache with the sss_cache command.  It can be used to clear the cache and update all records:

The sss_cache command can also clear all cached entries for a particular domain:
If the administrator knows that a specific record (user, group, or netgroup) has been updated, then sss_cachecan purge the records for that specific account and leave the rest of the cache intact:

Please refer to the official documentation for more information.

In case the above doesn’t help, you can also remove the cache ‘hte hard way’:

Just wanted to add this command which also helped me in one case somehow. 

Final Words

I hope this guide helps people towards a better Windows Linux integration. Let me know if you think there is a better way to do the above or if you have some useful information you think I should add to this guide.

Greetings.

Willem