How to Configure Filebeat for nginx and ElasticSearch

Here we explain how to set up ElasticSearch to read nginx web server logs and write them to ElasticSearch. We use Filebeat to do that.

Filebeat has an nginx module, meaning it is pre-programmed to convert each line of the nginx web server logs to JSON format, which is the format that ElasticSearch requires. Using JSON is what gives ElasticSearch the ability to make it easier to query and analyze such logs.

Note: you could also add ElasticSearch Logstash to this design, but putting that in between FileBeat and Logstash. But that common practice seems redundant here. We will discuss use cases for when you would want to use Logstash in another post.

• nginx web server (or just download the sample shown below and put the into the corresponding folder)
• Filebeat
• Elastic Cloud account (or set up your own server)
• Ubuntu (or other Linux distro). Here we use Ubuntu 16.04.

Set up Elastic Cloud

You can use your own locally-installed instance of ElasticSearch. But here we use Elastic Cloud.

Follow the instructions we wrote here to set up ElasticSearch in the cloud if you don't already have a system. Note the cloud ID, password, Kibana URL, and Elasticsearch URL as you will need them below.

If you don't already have a web server you can install Linux or just download some sample nginx files into the /var/log/nginx folder.

Make some Data If your web server does not have much data, to get a larger amount of log entries change to the nginx log directly and download these two logs:

Install filebeat Download filebeats and then install it:

Now enable the nginx filebeat module.

List enabled modules and you will see that nginx is listed.

Add the cloud it and your userid and password to the Filebeat config file. This makes it simpler to connect to the instance as it eliminates the need to put IP addresses and ports.

Run this command to push nginx dashboards to Kibana. It will start processing logs too.

For subsequent runs of Filebeat run it like this. The -e option will output the logs to stdout.

Filebeat will process all of the logs in /var/log/nginx. You can verify that by querying ElasticSearch for the indices, replacing the URL below for the URL for you instance of ES.

Note that we have saved the userid:password option in the $pwd environment variable.

The index name will be some combination of the word filebeat and today's date.

Open Kibana If you have never used Kibana before it will ask you to set up an index pattern. If you have then navigate to the Management screen and add one.

To add an index pattern simply means how many letters of existing indexes you want to match when you do queries. That is, if you put filebeat* it would read all indices that start with the letters filebeat. If you add the date it would read today's parsed logs. Of course that won't be useful if you parse other kinds of logs besides nginx. We will illustrate that in another post.

Just start typing the letters f-i-l-e and it will show you which ES document indexes match:

KIbana will ask what field it can use as a timestamp.This is so it can produce a time-series analysis, which is the whole point of gathering logs in the first place. Pick @timestamp for now. It is generated by ES. You can use that when nothing else is in the data itself.

Now, from the Discover screen (i.e., top left button on the nav bar) you can browse records. Kibana will ask you what index pattern you want to use.

Click on a record to expand it. The record below is too long to see in its entirety. So scroll up and down to see all of it.

Now click the Visualize screen, again selecting from the nav bar on the left. There are different nginx dashboards that Filbeat already installed. The one below shows website hits over time.

This visualization (aka dashboard) shows the location of the users who have accessed your web site for the time range selected.

ElasticSearch Guide

1. ElasticSearch Tutorial for Beginners: ElasticSearch Basics
2. Using Apache Hive with ElasticSearch
3. Using ElasticSearch with Apache Spark
4. Curl ElasticSearch Commands Cheat Sheet
5. Spark ElasticSearch Hadoop Update and Upsert Example and Explanation
6. How To Write Apache Spark Data to ElasticSearch Using Python
7. ElasticSearch Search Syntax and Boolean and Aggregation Searches
8. ElasticSearch Nested Queries: How to Search for Embedded Documents
9. ElasticSearch Joins: Has_Child, Has_parent query
10. How to Setup up an Elastic Version 7 Cluster
11. How to Configure Filebeat for nginx and ElasticSearch
12. Using Beats and Logstash to Send Logs to ElasticSearch
13. How to Load CSV File into ElasticSearch with Logstash
14. ElasticSearch Aggregations Explained
15. What is the ELK Stack?
16. Logstash 101: Using Logstash in a Data Processing Pipeline
17. Using Kibana to Execute Queries in ElasticSearch using Lucene and Kibana Query Language
18. ElasticSearch Machine Learning
19. How to Install Elastic Enterprise Search
20. How To Use Elastic Enterprise Search with GitHub

These postings are my own and do not necessarily represent BMC's position, strategies, or opinion.