Tutorial

  1. Elasticsearch 6.3
  2. Elasticsearch 5.1 Download Windows 10
  3. Elasticsearch 5.1 Download Windows 7
  4. Elasticsearch 5.1 Download Windows 10
  5. Elasticsearch 5.1 Download Free

Elasticsearch data is stored in the /var/lib/elasticsearch directory, configuration files are located in /etc/elasticsearch. By default, Elasticsearch is configured to listen on localhost only. If the client connecting to the database is also running on the same host and you are setting up a single node cluster you don’t need to change the.

Note: This tutorial is for an older version of the ELK stack, which is not compatible with the latest version. The latest version of this tutorial is available at How To Install Elasticsearch, Logstash, and Kibana (ELK Stack) on Ubuntu 14.04.

Introduction

In this tutorial, we will go over the installation of the Elasticsearch ELK Stack on Ubuntu 14.04—that is, Elasticsearch 1.7.3, Logstash 1.5.4, and Kibana 4.1.1. We will also show you how to configure it to gather and visualize the syslogs of your systems in a centralized location. Logstash is an open source tool for collecting, parsing, and storing logs for future use. Kibana is a web interface that can be used to search and view the logs that Logstash has indexed. Both of these tools are based on Elasticsearch.

Centralized logging can be very useful when attempting to identify problems with your servers or applications, as it allows you to search through all of your logs in a single place. It is also useful because it allows you to identify issues that span multiple servers by correlating their logs during a specific time frame.

  • Today, Amazon Elasticsearch Service (Amazon ES) announces support for in-place Elasticsearch upgrades for domains that are running version 5.1 or later. This new feature lets you move to the latest release in the same major version (for example, 5.3 to 5.6) or from the latest release in a major version to the latest release in the next major.
  • Security vulnerabilities of Elasticsearch Elasticsearch version 1.5.1 List of cve security vulnerabilities related to this exact version. You can filter results by cvss scores, years and months. This page provides a sortable list of security vulnerabilities.
  • Elasticsearch is a powerful open source search and analytics engine that makes data easy to explore.

It is possible to use Logstash to gather logs of all types, but we will limit the scope of this tutorial to syslog gathering.

Our Goal

The goal of the tutorial is to set up Logstash to gather syslogs of multiple servers, and set up Kibana to visualize the gathered logs.

Our Logstash / Kibana setup has four main components:

  • Logstash: The server component of Logstash that processes incoming logs
  • Elasticsearch: Stores all of the logs
  • Kibana: Web interface for searching and visualizing logs, which will be proxied through Nginx
  • Logstash Forwarder: Installed on servers that will send their logs to Logstash, Logstash Forwarder serves as a log forwarding agent that utilizes the lumberjack networking protocol to communicate with Logstash

We will install the first three components on a single server, which we will refer to as our Logstash Server. The Logstash Forwarder will be installed on all of the client servers that we want to gather logs for, which we will refer to collectively as our Client Servers.

Prerequisites

To complete this tutorial, you will require root access to an Ubuntu 14.04 VPS. Instructions to set that up can be found here (steps 3 and 4): Initial Server Setup with Ubuntu 14.04.

If you would prefer to use CentOS instead, check out this tutorial: How To Install ELK on CentOS 7.

The amount of CPU, RAM, and storage that your Logstash Server will require depends on the volume of logs that you intend to gather. For this tutorial, we will be using a VPS with the following specs for our Logstash Server:

Elasticsearch
  • OS: Ubuntu 14.04
  • RAM: 4GB
  • CPU: 2

In addition to your Logstash Server, you will want to have a few other servers that you will gather logs from.

Let’s get started on setting up our Logstash Server!

Install Java 8

Elasticsearch and Logstash require Java, so we will install that now. We will install a recent version of Oracle Java 8 because that is what Elasticsearch recommends. It should, however, work fine with OpenJDK, if you decide to go that route.

Add the Oracle Java PPA to apt:

Update your apt package database:

Install the latest stable version of Oracle Java 8 with this command (and accept the license agreement that pops up):

Now that Java 8 is installed, let’s install ElasticSearch.

Install Elasticsearch

Elasticsearch can be installed with a package manager by adding Elastic’s package source list.

Run the following command to import the Elasticsearch public GPG key into apt:

If your prompt is just hanging there, it is probably waiting for your user’s password (to authorize the sudo command). If this is the case, enter your password.

Create the Elasticsearch source list:

Update your apt package database:

Install Elasticsearch with this command:

Elasticsearch is now installed. Let’s edit the configuration:

You will want to restrict outside access to your Elasticsearch instance (port 9200), so outsiders can’t read your data or shutdown your Elasticsearch cluster through the HTTP API. Find the line that specifies network.host, uncomment it, and replace its value with “localhost” so it looks like this:

Save and exit elasticsearch.yml.

Now start Elasticsearch:

Then run the following command to start Elasticsearch on boot up:

Now that Elasticsearch is up and running, let’s install Kibana.

Install Kibana

Kibana can be installed with a package manager by adding Elastic’s package source list.

Create the Kibana source list:

Update your apt package database:

Install Kibana with this command:

Kibana is now installed.

Open the Kibana configuration file for editing:

In the Kibana configuration file, find the line that specifies host, and replace the IP address (“0.0.0.0” by default) with “localhost”:

kibana.yml excerpt (updated)

Save and exit. This setting makes it so Kibana will only be accessible to the localhost. This is fine because we will install an Nginx reverse proxy, on the same server, to allow external access.

Now enable the Kibana service, and start it:

Before we can use the Kibana web interface, we have to set up a reverse proxy. Let’s do that now, with Nginx.

Install Nginx

Because we configured Kibana to listen on localhost, we must set up a reverse proxy to allow external access to it. We will use Nginx for this purpose.

Note: If you already have an Nginx instance that you want to use, feel free to use that instead. Just make sure to configure Kibana so it is reachable by your Nginx server (you probably want to change the host value, in /opt/kibana/config/kibana.yml, to your Kibana server’s private IP address or hostname). Also, it is recommended that you enable SSL/TLS.

Use apt to install Nginx and Apache2-utils:

Use htpasswd to create an admin user, called “kibanaadmin” (you should use another name), that can access the Kibana web interface:

Enter a password at the prompt. Remember this login, as you will need it to access the Kibana web interface.

Now open the Nginx default server block in your favorite editor. We will use vi:

Delete the file’s contents, and paste the following code block into the file. Be sure to update the server_name to match your server’s name:

Save and exit. This configures Nginx to direct your server’s HTTP traffic to the Kibana application, which is listening on localhost:5601. Also, Nginx will use the htpasswd.users file, that we created earlier, and require basic authentication.

Now restart Nginx to put our changes into effect:

Elasticsearch 5.1 Download

Kibana is now accessible via your FQDN or the public IP address of your Logstash Server i.e. http://logstash_server_public_ip/. If you go there in a web browser, after entering the “kibanaadmin” credentials, you should see a Kibana welcome page which will ask you to configure an index pattern. Let’s get back to that later, after we install all of the other components.

Install Logstash

The Logstash package is available from the same repository as Elasticsearch, and we already installed that public key, so let’s create the Logstash source list:

Update your apt package database:

Install Logstash with this command:

Logstash is installed but it is not configured yet.

Generate SSL Certificates

Since we are going to use Logstash Forwarder to ship logs from our Servers to our Logstash Server, we need to create an SSL certificate and key pair. The certificate is used by the Logstash Forwarder to verify the identity of Logstash Server. Create the directories that will store the certificate and private key with the following commands:

Now you have two options for generating your SSL certificates. If you have a DNS setup that will allow your client servers to resolve the IP address of the Logstash Server, use Option 2. Otherwise, Option 1 will allow you to use IP addresses.

Option 1: IP Address

If you don’t have a DNS setup—that would allow your servers, that you will gather logs from, to resolve the IP address of your Logstash Server—you will have to add your Logstash Server’s private IP address to the subjectAltName (SAN) field of the SSL certificate that we are about to generate. To do so, open the OpenSSL configuration file:

Find the [ v3_ca ] section in the file, and add this line under it (substituting in the Logstash Server’s private IP address):

openssl.cnf excerpt (updated)

Save and exit.

Now generate the SSL certificate and private key in the appropriate locations (/etc/pki/tls/), with the following commands:

The logstash-forwarder.crt file will be copied to all of the servers that will send logs to Logstash but we will do that a little later. Let’s complete our Logstash configuration. If you went with this option, skip option 2 and move on to Configure Logstash.

Option 2: FQDN (DNS)

If you have a DNS setup with your private networking, you should create an A record that contains the Logstash Server’s private IP address—this domain name will be used in the next command, to generate the SSL certificate. Alternatively, you can use a record that points to the server’s public IP address. Just be sure that your servers (the ones that you will be gathering logs from) will be able to resolve the domain name to your Logstash Server.

Now generate the SSL certificate and private key, in the appropriate locations (/etc/pki/tls/…), with the following command (substitute in the FQDN of the Logstash Server):

The logstash-forwarder.crt file will be copied to all of the servers that will send logs to Logstash but we will do that a little later. Let’s complete our Logstash configuration.

Configure Logstash

Logstash configuration files are in the JSON-format, and reside in /etc/logstash/conf.d. The configuration consists of three sections: inputs, filters, and outputs.

Let’s create a configuration file called 01-lumberjack-input.conf and set up our “lumberjack” input (the protocol that Logstash Forwarder uses):

Insert the following input configuration:

Save and quit. This specifies a lumberjack input that will listen on tcp port 5043, and it will use the SSL certificate and private key that we created earlier.

Now let’s create a configuration file called 10-syslog.conf, where we will add a filter for syslog messages:

Insert the following syslog filter configuration:

10-syslog.conf

Save and quit. This filter looks for logs that are labeled as “syslog” type (by a Logstash Forwarder), and it will try to use “grok” to parse incoming syslog logs to make it structured and query-able.

Lastly, we will create a configuration file called 30-lumberjack-output.conf:

Insert the following output configuration:

Save and exit. This output basically configures Logstash to store the logs in Elasticsearch.

With this configuration, Logstash will also accept logs that do not match the filter, but the data will not be structured (e.g. unfiltered Nginx or Apache logs would appear as flat messages instead of categorizing messages by HTTP response codes, source IP addresses, served files, etc.).

If you want to add filters for other applications that use the Logstash Forwarder input, be sure to name the files so they sort between the input and the output configuration (i.e. between 01- and 30-).

Restart Logstash to put our configuration changes into effect:

Now that our Logstash Server is ready, let’s move onto setting up Logstash Forwarder.

Set Up Logstash Forwarder (Add Client Servers)

Do these steps for each Ubuntu or Debian server that you want to send logs to your Logstash Server. For instructions on installing Logstash Forwarder on Red Hat-based Linux distributions (e.g. RHEL, CentOS, etc.), refer to the Build and Package Logstash Forwarder section of the CentOS variation of this tutorial.

Copy SSL Certificate and Logstash Forwarder Package

On Logstash Server, copy the SSL certificate to Client Server (substitute the client server’s address, and your own login):

Elasticsearch

After providing your login’s credentials, ensure that the certificate copy was successful. It is required for communication between the client servers and the Logstash server.

Install Logstash Forwarder Package

On Client Server, create the Logstash Forwarder source list:

It also uses the same GPG key as Elasticsearch, which can be installed with this command:

Then install the Logstash Forwarder package:

Note: If you are using a 32-bit release of Ubuntu, and are getting an “Unable to locate package logstash-forwarder” error, you will need to install Logstash Forwarder manually.

Now copy the Logstash server’s SSL certificate into the appropriate location (/etc/pki/tls/certs):

Configure Logstash Forwarder

On Client Server, create and edit Logstash Forwarder configuration file, which is in JSON format:

Under the network section, add the following lines into the file, substituting in your Logstash Server’s private address for logstash_server_private_address:

logstash-forwarder.conf excerpt 1 of 2

Under the files section (between the square brackets), add the following lines,

Save and quit. This configures Logstash Forwarder to connect to your Logstash Server on port 5043 (the port that we specified an input for earlier), and uses the SSL certificate that we created earlier. The paths section specifies which log files to send (here we specify syslog and auth.log), and the type section specifies that these logs are of type “syslog* (which is the type that our filter is looking for).

Note that this is where you would add more files/types to configure Logstash Forwarder to other log files to Logstash on port 5043.

Now restart Logstash Forwarder to put our changes into place:

Now Logstash Forwarder is sending syslog and auth.log to your Logstash Server! Repeat this section for all of the other servers that you wish to gather logs for.

Connect to Kibana

Elasticsearch 5.1 download windows 7

When you are finished setting up Logstash Forwarder on all of the servers that you want to gather logs for, let’s look at Kibana, the web interface that we installed earlier.

Elasticsearch 6.3

In a web browser, go to the FQDN or public IP address of your Logstash Server. After entering the 'kibanaadmin” credentials, you should see a page prompting you to configure an index pattern:

Go ahead and select @timestamp from the dropdown menu, then click the Create button to create the first index.

Now click the Discover link in the top navigation bar. By default, this will show you all of the log data over the last 15 minutes. You should see a histogram with log events, with log messages below:

Right now, there won’t be much in there because you are only gathering syslogs from your client servers. Here, you can search and browse through your logs. You can also customize your dashboard.

Try the following things:

Elasticsearch 5.1 Download Windows 10

  • Search for “root” to see if anyone is trying to log into your servers as root
  • Search for a particular hostname (search for host: 'hostname')
  • Change the time frame by selecting an area on the histogram or from the menu above
  • Click on messages below the histogram to see how the data is being filtered

Kibana has many other features, such as graphing and filtering, so feel free to poke around!

Conclusion

Now that your syslogs are centralized via Elasticsearch and Logstash, and you are able to visualize them with Kibana, you should be off to a good start with centralizing all of your important logs. Remember that you can send pretty much any type of log to Logstash, but the data becomes even more useful if it is parsed and structured with grok.

To improve your new ELK stack, you should look into gathering and filtering your other logs with Logstash, and creating Kibana dashboards. These topics are covered in the second and third tutorials in this series. Also, if you are having trouble with your setup, follow our How To Troubleshoot Common ELK Stack Issues tutorial.

What you’ll learn

  • How to build a powerful search engine with Elasticsearch
  • The theory of Elasticsearch and how it works under-the-hood
  • Write complex search queries
  • Be proficient with the concepts and terminology of Elasticsearch

Requirements

  • Knowledge of JSON
  • Basic terminal skills is a plus

Description

Do you want to learn the popular search engine, Elasticsearch, from the beginning and become a professional in no time?

Elasticsearch 5.1 Download Windows 7

This course is an excellent way for you to quickly learn Elasticsearch and to put your knowledge to work in just a few hours! If so, then you have come to the right place, as this is the most comprehensive course on Elasticsearch that you will find online!

This course is a great starting point for anyone who wants to learn the ELK stack and Elastic Stack, as Elasticsearch is at the center of both stacks.

Elasticsearch is an extremely popular search engine and will be an excellent addition to your CV – even if you are already familiar with other search engines or frameworks such as Apache Lucene, Apache Solr, Amazon CloudSearch, etc.

Please note that this course is intended for developers who want to interact with an Elasticsearch cluster in one way or another and not system administrators looking to maintain an Elasticsearch cluster in production. The course focuses on functionality relevant to utilize the capabilities of Elasticsearch as a developer.

The course is a combination of theory and learning by doing. Before giving examples of how to perform certain queries, you will have been equipped with the necessary theory in advance.

This ensures that you not only know how to perform powerful searches with Elasticsearch, but that you also understand the relevant theory; you will get a deep understanding of how Elasticsearch works under the hood.

The course starts from the absolute beginning, and no knowledge or prior experience with Elasticsearch is required. We will walk through all of the most important aspects of Elasticsearch, and at the end of this course, you will be able to build powerful search engines. This could be for a website where you could build Google-like search functionality, for example.

So, join me in this course and learn to build powerful search engines with Elasticsearch today!

Note that this course does not cover Logstash and Kibana.

This is so that I can go into much greater detail with Elasticsearch and focus on that exclusively.

This course is therefore dedicated to Elasticsearch. For courses on Logstash and Kibana, please see my other courses.

Who this course is for:

  • Developers who want to learn Elasticsearch. The course is intended for developers and not system administrators.

Elasticsearch 5.1 Download Windows 10

Created by Bo Andersen
Last updated 5/2020
EnglishEnglish [Auto-generated], French [Auto-generated], 6 more

Elasticsearch 5.1 Download Free

Size: 4.52 GB

https://www.udemy.com/course/elasticsearch-complete-guide/.

Coments are closed

Recent News

Scroll to top