Reading list Switch to dark mode

    How to setup Elasticsearch Fluentd and Kibana on Ubuntu 18.04

    Updated 18 January 2021

    In this blog, we will walk through Elasticsearch, Fluentd and Kibana installation and setup.

    Have you ever been stuck with your application behaving waywardly and, unfortunately, with no significant logs whatsoever? Or having hundreds and thousands of streams of logs bombarding continuously which makes it even harder to comprehend.

    These scenarios, come so often, marks the significance of well ordered & legible logs in development of any application. Even the absence of sorting and filtering features could make debugging and issue tracking a tedious task. Besides, an application could be a flock of microservices, so accessing their logs on different locations could be a hectic approach.

    To the rescue, there are ways of managing logs in a more organised manner. EFK Stack is one of the logging solutions.

    Elasticsearch, Fluentd and Kibana

    efk-1

    Elasticsearch, Fluentd, and Kibana comprises a powerful logging and monitoring stack which allows logs management, visualisation, debugging easier in an interactive and centralised manner. It allows users to understand real-time heavy log data by allowing sorting and filtering by time, category, incident, index etc.

    Elasticsearch is an analytical search engine. It allows textual, numerical, geospatial, structured and unstructured data search. Elasticsearch coupled with Kibana, which is a powerful visualisation tool, provide desirable readability of logs. With an interactive web interface, kibana helps in managing, sorting and visualising elasticsearch data. Third component of this stack, Fluentd, is a data collector and shipper. It collects logs from a source, manipulate it and then ship it to a destination. Destination could be it another fluentd or elasticsearch or any other targets.

    Start your headless eCommerce
    now.
    Find out More

    A few people prefer ELK (Elasticsearch, Logstash and Kibana) over EFK. Both logstash and fluentd are capable of carrying out log shipping job. You can prefer the one more suitable for your application.

    This article will only cover EFK stack installation and setup on Ubuntu 18.04 operating system. EFK configuration to visualise logs will be discussed in subsequent blogs. Here, we will deploy server client two-tier architecture. Fluentd will be installed on server to be monitored (where logs are being stored) and EFK stack will be installed on separate server. Fluentd client will then ship data to EFK server. Client side fluentd will communicate with server side fluentd over tcp port 24224. Then, server side fluentd will transfer data on elasticsearch over 9200 tcp port present on the same server. Server side td-agent uses fluent-plugin-elasticsearch to transfer data to elasticsearch server.

    Let’s begin with Elasticsearch, Fluentd and Kibana installation at server side and fluentd installation at client side:

    Elasticsearch Installation

    sudo su
    apt update
    wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
    sudo apt-get install apt-transport-https
    echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
    sudo apt-get update && sudo apt-get install elasticsearch
    sudo update-rc.d elasticsearch defaults 95 10
    service elasticsearch restart
    service elasticsearch status
    tail -f /var/log/elasticsearch/elasticsearch.log
    curl localhost:9200

    Kibana Installation

    sudo apt-get update && sudo apt-get install kibana
    sudo update-rc.d kibana defaults 95 10
    sudo -i service kibana start
    service kibana status

    Fluentd Installation

    Next, we will install TD Agent which is a stable and much simpler distribution of Fluentd.

    apt update
    curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-bionic-td-agent3.sh | sh
    td-agent-gem install fluent-plugin-elasticsearch
    chown -R td-agent: /var/log/td-agent/
    sudo systemctl restart td-agent.service
    sudo systemctl status td-agent.service
    sudo update-rc.d td-agent defaults 95 10

    sudo /bin/systemctl daemon-reload

    Furthermore, you can repeat the td-agent installation steps, mentioned above, on Client server in the same way.
    Now adding configuration in server side td-agent to listen on port 24224 from everywhere.

    cd /etc/td-agent/
    mv td-agent.conf td-agent.conf-default
    nano td-agent.conf

    Add following lines in td-agent file, save it and restart the td-agent server.

    <source>
     @type forward
     port 24224
     bind 0.0.0.0
    </source>
    @include /etc/td-agent/conf.d/*.conf

    mkdir /etc/td-agent/conf.d
    /etc/init.d/td-agent restart

    EFK stack is now installed on the server and fluentd (td-agent) on client. Make sure all services are running seamlessly and server-client side td-agents are able to communicate with each other over port 24224.

    Not to mention, next step would be to start configuring client fluentd to capture and ship logs from client to server. We will also be configuring elasticsearch and kibana with indexes, password protected . In addition to this, we will setup Nginx as a reverse proxy server to run kibana over http/https protocol. We will cover all these points in our next blog.

    Stay tuned.

    . . .

    Leave a Comment

    Your email address will not be published. Required fields are marked*


    2 comments

  • Sanjay Yadav
    • Subhangi (Moderator)
  • Back to Top

    Message Sent!

    If you have more details or questions, you can reply to the received confirmation email.

    Back to Home