One of the most used mechanisms for monitoring the systems and/or applications of a company is the use of ELK Stack: Elasticsearch, Logstash and Kibana. At Enimbos we have used ELK Stack for the development of monitoring tools by logs. ELK Stack has the advantage of being able to deploy like SaaS on the main Cloud providers of the market.

Elastic Stack is a set of open source tools developed by Elastics that allows to collect data from any type of source and in any format to carry out searches, analysis and visualization of data in real time.

It consists of the following products:

  • Elasticsearch: distributed search engine built on Apache Lucene. It allows indexing and retrieving JSON documents (without rigid schema) in different formats. It is based on Java and provides an access RESTFUL interface.
  • Logstash: it is an engine that allows to collect data from different sources, normalize them and distribute them. Originally optimized for log files, although it can read data from many types of sources.
  • Kibana: tool that allows the visualization and exploration in real time of large amounts of data. Through the graphic representation, you can easily consult complex data sets.
  • Beats: are data loaders that are installed on servers and function as agents that send operational information to Elasticsearch.

Elastic Stack allows data to be obtained from log files using Logstash and its storage in the Elasticsearch analysis and search engine. In addition, it allows the data visualization, monitoring and management in real time through Kibana.

On the other hand, in Enimbos we propose the inclusion of ElastAlert, an open source tool that is integrated with Elastic Stack and allows to detect anomalies and inconsistencies in Elasticsearch data and to launch alerts.

The Elastic Stack plus ElastAlert toolset allows us to create a monitoring system for the extraction, storage and exploitation of the data of the processes or systems to be monitored. It allows obtaining data from log files through Logstash and its storage in the Elasticsearch search and analysis engine. In addition, it allows the visualization, monitoring and exploitation of data in real time through Kibana and the configuration of alerts with ElastAlert.

To store the data in Elasticsearch, Logstash uses the Elasticseach output plugin that allows data to be sent via http(s). Elasticsearch will be in charge of indexing the data obtained from the logs and allowing the search and analysis of them.

 

Data Integration

In Enimbos we propose the use of the Logstash tool for extracting data from the log files of the systems. Logstash is a server-side tool that allows us to collect data from multiple sources simultaneously, transform them and store them in a search and analysis engine, in this case Elasticsearch. It is developed in JRuby and works on a JVM.

Logstash bases its operation on the definition and integration of inputs, codecs, filters and outputs:

  • The entries are the different data sources. In our case, we will use as input the log files generated by the different components of the Homeserve Integration Services Platform.
  • The codecs are used to convert the different formats of the entries into another accepted by Logstash, which will allow us to unify the format of the logs and have the possibility of reading formats other than plain text.
  • Filters allow us to process different events.
  • The outputs are the different destinations in which the processed data will be stored

Logstash is based on the use of plugins that support the various elements of the mentioned tool: inputs, codecs, filters and outputs.

Normally, this log files loading by Logstash will be done through ssh, although Beats of type FileBeat can also be used to send the log file lines for loading in Elasticsearch. In any case, Logstash has a large set of plugins for data entry:

  • Beats: Elastic Beats data
  • Elasticsearch: query results to Elasticsearch
  • Exec: command output
  • File: files
  • http: data received by http or https
  • jdbc: DB data
  • pipe: data from a pipe command
  • tcp: data received by a tcp socket

On the other hand, it is possible to perform the collection of metrics in order to monitor and visualize the operational data in a scorecard. Enimbos proposes the use of Elastic Beats for the collection of said metrics and their storage in Elasticsearch.

Elastic Beats is a tool that is part of Elastic Stack. These are light agents that send data from machines to Logstash or Elasticsearch. There are different types:

  • Filebeat: allows the collection, parsing and sending of data from log files.
  • Metricbeat: allows the collection and sending of metrics at the system level, such as CPU usage, memory usage, file system, disk access, network access, etc.
  • Packetbeat: allows the monitoring of services and applications in real time, obtaining metrics such as latency, response times, errors, access patterns, transactions, HTTP access, etc.
  • Winlogbeat: allows the collection and sending of events from the Windows system log.
  • Auditbeat: allows the collection and sending of system audit metrics.
  • Heartbeat: allows the monitoring of availability and response time of services.

All data collected by both Logstash and Beats are integrated into Elasticsearch. Elasticsearch is a search engine and distributed analysis based on JSON that has a RESTful interface, which allows us to consult any type of data in real time.

Being a distributed system, it allows deploying in a scalable and highly available architecture. Allows the use of different indexes in a cluster and perform queries on one or more indexes in Full Text Search mode. In addition, it indexes all fields of JSON documents without a rigid schema. It allows searches via the REST API using ElasticSearch Query DSL (Domain Specific Language): multilanguage, geolocation, contextual, autocomplete, etc.

In terms of infrastructure, Elasticsearch is a distributed system that can be deployed in clusters over one or more nodes. There are different types of nodes in Elasticsearch:

  • Master nodes that control the cluster
  • Data nodes that maintain data and perform operations on them
  • Client nodes that are small balancers that intervene in certain processing steps
  • ‘Ingest’ nodes dedicated to heavy data transformation operations
  • ‘Tribe’ nodes that can perform coordination operations between nodes and cluster

 

Data visualization

Enimbos proposes the use of Kibana for the visualization of the data of the monitoring tool by logs in scorecards or dashboards. Kibana is a tool that allows to visualize and navigate interactively by the data stored by Elasticsearch.

Kibana allows you to obtain and represent the following types of information:

  • Histograms, bar graphs, pie charts, etc.
  • Visualize data with geospatial information on maps.
  • Analysis of data series over time.
  • Graphs of data relations.

The Kibana dashboards perform queries to Elasticsearch to obtain in real time the value of the KPIs shown. The following figures show some examples of dashboards developed with Kibana that give us an idea of ​​the possibilities of the tool.

Some examples of infrastructure metrics dashboards are:

 

 

Alerts

The monitoring tool must be able to identify deviations in the normal operation of the platform and launch alerts to notify such deviations. For this, Enimbos proposes to use ElastAlert.

ElastAlert is an open source tool that integrates with Elastic Stack and allows to detect and alert anomalies and inconsistencies in Elasticsearch data. It is based on the use of two components types, rules and alerts. The tool periodically queries Elasticsearch by sending the obtained data to the rules that determine if an anomaly is found. Once found, it is notified by means of one or more alerts.

Some of the rules that can be created with ElastAlert are:

  • Increase or decrease of parameters, which will serve to alert us about the use of CPU, memory, etc.
  • Increase in the frequency of events, which will serve to warn of an increase in the error rate in certain components
  • Concrete values ​​in certain fields, which will serve to alert us of system crashes

On the other hand, ElastAlert has a large number of channels for sending alerts: Email, Jira, commands, OpsGenie, etc. The rules and alerts can be administered directly in Kibana.

 

Indexing and graph example

We are going to use as an example an application that generates a log with the following structure:

2018-11-12 06:00:23         INFO       client_1 server_1               service_1              1564      OK          message

The log consists of a series of service calls that can be on one or more servers. For the example we will suppose that we have to add the information to be able to consult the total time of the different invocations of a client to a same server and to know the result of the operation.

A FileBeat is used to collect and send the log lines generated by the application. The relevant FileBeat configuration is in the filebeat.yml file and would be the following:

  • paths:

    – C:\Formacion\ELK\ejemplo.log

  • output.logstash:

# The Logstash host and port

               hosts: [“host:port”]

The log events would be sent to Logstash, whose configuration would be done in the logstash.conf file. In this file the different plugins needed to process and format the log would be enabled and configured, generating a JSON document that would be indexed in ElasticSearch.

  • First, you have to configure the reception of the events from the FileBeat:

input {

               beats {

                               port => ” port”

                               client_inactivity_timeout => 6000

               }

}

  • To carry out the parsing of the log line that we use as an example, we will use three plugins:
    • Grok, allows us to parse and structure arbitrary text in structured, indexable and searchable data.

grok {

              match => { “message” => “%{TIMESTAMP_ISO8601:requestDate} %{WORD:levelDebug} %{DATA:client} %{DATA:server} %{DATA:service} %                         {NUMBER:requestTime:int}%{DATA:result} (?<message>(.|\r|\n)*)

}

}

As we can see, it consists of combining text patterns to obtain matches with the lines of logs. A text pattern is % {SYNTAX: SEMANTIC}, where syntax is the name of the pattern and semantics the identifier we give to the fragment.

  • Date, which allows us to pause dates and use them as a timestamp of events.

date {

               match => [ ” requestDate”, “YYYY-MM-dd HH:mm:ss” ]

               target => ” requestDate”

               }

  • Aggregate, allows you to add information available in a set of events (log lines) and incorporate aggregate information to a final event.

aggregate {

task_id => “%{client}%{server}”

code => “map[‘requestDate’] = event.get(‘requestTime’);

                   map[‘levelDebug’] = event.get(‘levelDebug’);

                   map[‘client’] = event.get(‘client’);

                   map[‘server’] = event.get(‘server’);

                   map[‘service’] = ‘all’;

   map[‘requestTime’] ||= 0 ; map[‘requestTime’] +=    event.get(‘requestTime’)

                   if map[‘result’] != ‘KO’

                                map[‘result’] = event.get(‘result’);

                   end;

                   map[‘message’] = event.get(‘message’);

                   map[‘lastService’] = event.get(‘service’);

                “

push_map_as_event_on_timeout => true

timeout => 60 # 1 minute timeout

}

Through the attribute task_id we indicate the key of the log lines to be added. In the attribute code of the plugin we give value to the attributes of the aggregate event that will be indexed in elasticsearch. The attributes push_map_as_event_on_timeout and timeout are used to indicate that 60 seconds after receiving the last log entry with the key “% {client}% {server} the aggregate event will be generated and indexed.

Finally, it is necessary to configure the ElasticSearch cluster and the indexing structure in which the logs will be stored.

elasticsearch {

hosts => [ “http://localhost:9200” ]

        index => “ejemplo-%{+YYYY.ww.ee}”

}

With this configuration a daily index will be created in Elasticsearch for documents of logs of the application. To start monitoring the indexed data in kibana, it is necessary to generate an index pattern that includes the generated daily indexes:

From this moment we can start to create graphs and consult the data in kibana:

Finally, we can use the events indexed by the aggregate plugin of logstash to create a graph that allows visualizing aggregate information about the invocations of the different clients to the different servers. For the example we are going to use a bar graph that shows the total time:

  • First of all we will use a filter on the lastService field, which only the aggregated events possess.
  • In the X axis we will use the sum of the requestTime field, which will give us the summation of the invocation time of a client to a server.

  • On the y-axis we use two aggregations, on the one hand for the client field and on the other for the server field.

The result of the graph will be the following:

About Victor García

As a Software Team Leader, Víctor is Computer Engineer by Extremadura University. Certifications: Sun Certified Enterprise Architect. Certified ScrumMaster, OSB 11g: Design & Integrate Services, Liferay Developer Training. He started his professional career in 2004 as Junior Consultant, then as Programmer and Programmer Analyst, after a few years, he started working as Project Manager for 5 years, and then for 3 years he worked as Analyst/Team Manager. He joined Enimbos in 2018 as Software Team Leader.