It has been some time since I gave my F5 Logstash filter an update. As I learned a lot of new things and techniques over the past six months, it was on my ‘short-term’ to do list to give them a major upgrade. A BIG-IP F5 load balancer is a device that acts as a reverse proxy and distributes network or application traffic across a number of servers. Load balancers are used to increase capacity and reliability of applications. They improve the overall performance of applications by decreasing the burden on servers associated with managing and maintaining application and network sessions, as well as by performing application-specific tasks.
Load balancers are generally grouped into two categories: Layer 4 and Layer 7. Layer 4 load balancers act upon data found in network and transport layer protocols (IP, TCP, FTP, UDP). Layer 7 load balancers distribute requests based upon data found in application layer protocols such as HTTP. Requests are received by both types of load balancers and they are distributed to a particular server based on a configured algorithm. Some industry standard algorithms are:
- Round robin
- Weighted round robin
- Least connections
- Least response time
Layer 7 load balancers can further distribute requests based on application specific data such as HTTP headers, cookies, or data within the application message itself, such as the value of a specific parameter. Load balancers ensure reliability and availability by monitoring the “health” of applications and only sending requests to servers and applications that can respond in a timely manner.
You can send your F5 logs to an F5 Logstash filter to get a grip on what’s going on in your load balancer. I’m not a 100 % sure if all F5 Big IP load balancers have the same syslog syntax, but I put the F5 Logstash filters I created on GitHub and give something back to the Elastic community. It’s not finished yet and definitely needs some work, but it’s better then a default syslog filters. It contains one global syslog F5 Logstash filter which parses the first piece of the F5 syslogs which contains things like ‘logsource’ ‘severity_label’ and labels the rest of the message as ‘info’. Thanks to Jesse from Nagios for helping me create the dcc filter in Nagios Log Server and Jens for helping me with F5.
F5 Logstash input
type => 'syslog-f5'
port => <portnumber>
F5 Logstash filters
dcc => ASM related messages. BIG-IP Application Security Manager (ASM) enables organizations to protect against OWASP top 10 threats, application vulnerabilities, and zero-day attacks. Leading Layer 7 DDoS defenses, detection and mitigation techniques, virtual patching, and granular attack visibility thwart even the most sophisticated threats before they reach your servers.
apd => Access Policy Demon. The apd process runs a BIG-IP APM access policy for a user session.
tmm => The traffic management microkernel is the process running on the BIG-IP host O/S that performs all of the local / global traffic management for the system.
sshd => The ssh daemon provides remote access to the BIG-IP system command line interface
F5 Logstash custom grok patterns
You will need to add these F5 Logstash custom grok patterns to your Logstash patterns directory. For me it’s located in /etc/logstash/patterns
Included in the GitHub project you can find my f5 elasticsearch template, with the correct mappings for each field. This enables you to use your data more efficiently and allow for advanced ip aggregations. You can find more information about mapping types here. If you have ideas about better mappings (I know they need some work), please let me know on GitHub by making an issue.