Json Over TCP With Powershell

Introduction

Logging is a critical part of any application written in any language. If you want to catch the anomalies in your scripts, you will need to find a way to write logs efficiently and find a way to query these logs and do some analytics on them. Over the years, I’ve been using a set of functions which make the ‘logging part’ of scripting a lot easier and more consistent.

Over the years, I’ve been scripting in Powershell, Bash, Python, Perl and PHP. The first thing I start with when creating a new script is paste my logging function of the language I’m working in.

As most of my scripts are written in Powershell, I’ll start with blogging about my Write-Log Powershell function. This function allows you to write logs to a variety of different targets. Maybe I’ll post some details about my log functions for other languages later.

Powershell – Write-Log

We were unable to make it work with the Json charset CP1252, as Logstash considered each line in the json file as a separate log entry. Only after stripping each newline, carriage return and space from the json object were we able to get Logstash to recognize it.  Anyway it seems more logical to not make your script log to file anymore when you can use an elk stack or a Nagios Log Server.

Being able to send Json over TCP with this Powershell function kind of makes it a lot easier to test and stress your configurations. It also kind of obsoletes logging to file and querying and filtering your logs with Elasticsearch is much more fun then reading through your logs with notepad…

How to achieve Json Over TCP with Powershell?

The function enables you to send json over tcp to a specific input port on your Nagios Log Server. A small example how you get the above to work:

Together with this input:

Should result in something like this:

Nagios_Log_Server_Json_Over_Tcp_01

This again opens up a lot of new possibilities. I’ll try to expand this post with some use cases.

Grtz

Willem

F5 Logstash Filter for BIG-IP F5 Load Balancer

Introduction

In this post I’ll talk about an F5 Logstash filter I’m working on. A BIG-IP F5 load balancer is a device that acts as a reverse proxy and distributes network or application traffic across a number of servers. Load balancers are used to increase capacity (concurrent users) and reliability of applications. They improve the overall performance of applications by decreasing the burden on servers associated with managing and maintaining application and network sessions, as well as by performing application-specific tasks.

Load balancers are generally grouped into two categories: Layer 4 and Layer 7. Layer 4 load balancers act upon data found in network and transport layer protocols (IP, TCP, FTP, UDP). Layer 7 load balancers distribute requests based upon data found in application layer protocols such as HTTP.

Requests are received by both types of load balancers and they are distributed to a particular server based on a configured algorithm. Some industry standard algorithms are:

  • Round robin
  • Weighted round robin
  • Least connections
  • Least response time

Layer 7 load balancers can further distribute requests based on application specific data such as HTTP headers, cookies, or data within the application message itself, such as the value of a specific parameter.

Load balancers ensure reliability and availability by monitoring the “health” of applications and only sending requests to servers and applications that can respond in a timely manner.

You can send your F5 syslog messages to a F5 Logstash filter to get a grip on what’s exactly going on. I’m not a 100 % sure if all F5 Big IP load balancers have the same syslog syntax, but I’m putting the F5 Logstash filters I created on GitHub, as it might help someone. It’s not finished yet and definitely needs some work, but it’s better then the default filters. It contains one global syslog F5 Logstash filter which parses the first piece of the F5 syslogs which contains things like ‘logsource’ ‘severity_label’ and labels the rest of the message as ‘info’. Thanks to Jesse from Nagios for helping me create the dcc filter in Nagios Log Server and Jens for helping me with F5. F5 logstash filters in Nagios Log Server

Logstash Configuration

Logstash Input

F5 Logstash Filters

F5 Type Filter

F5 Program Filter

dcc => ASM related messages. BIG-IP Application Security Manager (ASM) enables organizations to protect against OWASP top 10 threats, application vulnerabilities, and zero-day attacks. Leading Layer 7 DDoS defenses, detection and mitigation techniques, virtual patching, and granular attack visibility thwart even the most sophisticated threats before they reach your servers.

apd => Access Policy Demon. The apd process runs a BIG-IP APM access policy for a user session.

tmm => The traffic management microkernel is the process running on the BIG-IP host O/S that performs all of the local / global traffic management for the system.

tmm1 => The traffic management microkernel is the process running on the BIG-IP host O/S that performs all of the local / global traffic management for the system.

tmm2 => The traffic management microkernel is the process running on the BIG-IP host O/S that performs all of the local / global traffic management for the system.

tmm3 => The traffic management microkernel is the process running on the BIG-IP host O/S that performs all of the local / global traffic management for the system.

sshd => The ssh daemon provides remote access to the BIG-IP system command line interface

logger

For now I haven’t seen a lot useful information coming from the ‘logger’ program, although it is the second biggest sender of logs. On our F5, 12 % of the logs are from the ‘logger’ program.

After some research I noticed 99 % of the logs in the ‘logger’ program are identical. In this support article from F5, the reason is clearly explained:

If a BIG-IP high-availability redundant pair has the Detect ConfigSync Status feature enabled, each unit in the pair sends periodic iControl queries to its peer to determine if the redundant pair configuration is synchronized. These iControl requests occur approximately every 30 seconds on each unit. Each inbound request generates an entry in both the local /var/log/httpd/ssl_access_log file and the /var/log/httpd/ssl_request_log file, which appears similar to the following examples, where 10.0.0.1 is the management IP address of the peer.

So to save some diskspace you could consider filtering the SSL access and request logs on the F5 or drop them in a custom F5 Logstash filter..

Logstash F5 Custom GROK Patterns

You will need to add these F5 Logstash custom grok patterns to your Logstash patterns directory:

Greetings. Willem