#elasticsearch

Linux

Logstash Elasticsearch data retention

0

If you are using Elasticsearch to centralize your log data, that is a great solution. However, after few months, you will have huge log data store in your server hard disk. You have to clean the old log data that you are sure that you will not use it in future.

To delete the 10-day-ago log data, we can use the script below:

1
2
3
#/bin/bash
d=$(date --date="10 day ago" +%Y.%m.%d)
curl -XDELETE http://localhost:9200/logstash-${d}?pretty

So we should run the script above every night to delete data. So we use cronjob:

1
30 11 * * * bash /home/ubuntu/backend/scripts/es_retention.sh > /dev/null 2>&1

Thanks for your reading

linux shirt

Software is like sex t-shirt

Nguyen Sy Thanh Son

0

Docker, Linux

How to HOT Backup Database (MongoDB, MySQL, ES …) to AWS S3

0

Actually, there are many way to backup your database. You can using RSYNC, MongoDump for Mongo, S3 Backup Plugin for ElasticSearch. However, this post will show you the way I used in my project. Maybe, It is not perfect for all case. But in my case, it is really perfect.

Docker Ironman T-shirt

Docker Ironman T-shirt

I am running a project with Microservice Architecture. All Databases and Services are running in Docker Container.  In my plan, I have to backup all databases every night.

At the beginning, I tried to use tar command to compress the data, and then I use command aws s3 copy to copy backup data to S3. It seems work. But tar command makes MongoDB stop working. I tried to google to solve the problem. I found the solution is rsync command.

The backup process should be implement in three steps:

  • Use rsync command to copy the data to other location
  • Compress the data by tar command
  • Move the compressed data to AWS S3

The script should be:

1
2
3
4
echo Backup MongoDb Data ...
rsync -avz /var/lib/docker/volumes/backend_mongodb/ /tmp/mongodb && \
    tar cvzf mongodb-${dt}.tar.gz /tmp/mongodb/ && \
    aws s3 cp mongodb-${dt}.tar.gz s3://prjbackups/${env}/mongodb/mongodb-${dt}.tar.gz

Thanks for your reading.

Nguyen Sy Thanh Son

0

Linux

Monitor Nginx Response Time with FluentD, Kibana and Elasticsearch

2

In my previous post, I showed you How to centralize Nginx logs.  So now, I will use FluentD, Kibana and ElasticSearch to collect Nginx Response Time.

To implement it, we have to change Nginx Log Format. Because, in the default, Nginx does not store Response Time to access.log file. So we change nginx.conf as below:

1
2
log_format upstreamlog '[$time_local] $remote_addr - $remote_user - $server_name to: $upstream_addr $request $status upstream_response_time $upstream_response_time msec $msec request_time $request_time';
access_log /var/log/nginx/access.log upstreamlog;

After reload Nginx, you can try to tail access.log. The result should be as below:

1
[26/Feb/2016:06:42:49 +0000] 118.70.190.253 - - - example.com to: 172.31.27.195:33687 GET /api/v1/post/31856?access_token=r12dx4whN147sm0YC8IYTyn-bfkyWg52-F08_BI_pfKBOo1JCliMwhI_g8qtffAm9QUoRjNiF8TULGpjw8sgbQ&limit=15 HTTP/1.1 200 upstream_response_time 0.140 msec 1456468969.606 request_time 0.259

Now, we will create a regex string that matches with the log format above:

1
/^\[(?([^ ]* [^\]]*))\] (?[^ ]*) - ([^ ]*) - ([^ ]*) to: (?[^ ]*) (?[^ ]*) (?[^ ]*) ([^ ]*) (?<code>[^ ]*) upstream_response_time (?[^ ]*) msec ([^ ]*) request_time ([^ ]*)/

And Insert to td-agent.conf as below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
<source>
  @type tail
  path /var/log/nginx/access.log #...or where you placed your Apache access log
  pos_file /var/log/td-agent/nginx-access.log.pos # This is where you record file position
  tag nginx.access
  time_format %d/%b/%Y:%H:%M:%S %z
  format /^\[(?<time>([^ ]* [^\]]*))\] (?<remote>[^ ]*) - ([^ ]*) - ([^ ]*) to: (?<host>[^ ]*) (?<method>[^ ]*) (?<path>[^ ]*) ([^ ]*) (?<code>[^ ]*) upstream_response_time (?<response_time>[^ ]*) msec ([^ ]*) request_time ([^ ]*)/
</source>
<source>
  @type tail
  path /var/log/nginx/error.log
  pos_file /var/log/td-agent/nginx-error.log.pos
  tag nginx.error
  format /^(?<time>[^ ]+ [^ ]+) \[(?<log_level>.*)\] (?<pid>\d*).(?<tid>[^:]*): (?<message>.*)$/
</source>
 
<source>
  @type tail
  path /build/clotify/logs/*/*.log
  pos_file /var/log/td-agent/clotify.log.pos
  tag clotify
  format /^(?<time>([^ ]* [^ ]*)) (?<log_level>[^ ]*): (?<service>[^ ]*) (?<message>.*)\[in (?<filename>[^ ]*):(?<lineno>[^ ]*)\]/
</source>
 
<match nginx.*>
  @type elasticsearch
  logstash_format true
  host 172.31.18.133
  port 9200
  index_name fluentd-aws-dev-nginx
  type_name fluentd-aws-dev-nginx
</match>

After restarting td-agent. We can wait a minute and then view kibana. The outfit should be:

2016-02-26_1800

Monitor Nginx Response Time with FluentD, Kibana and Elasticsearch

For basic installation, please refer to https://sonnguyen.ws/centralize-docker-logs-with-fluentd-elasticsearch-and-kibana/

Nguyen Sy Thanh Son

2

Docker, Linux

Centralize Nginx Logs with FluentD, Kibana and Elasticsearch

3

As you know, FluentD is a great tool to collect the logs, ElasticSearch supports store and search the log data and Kibana helps you to view and search logs in web interface.

Nginx Logs is a good thing help you monitor, debug, and troubleshoot your application. So in this post, I will show  you how to Centralize Nginx Logs with FluentD, Kibana and Elasticsearch.

Before reading this post, please read Centralize Docker Logs with FluentD, Kibana and Elasticsearch to know how to install Fluent, Kibana and Elasticsearch in Ubuntu 14.04.

In the next step, we add content below to /etc/td-agent/td-agent.conf file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<source>
  @type tail
  path /var/log/nginx/access.log #...or where you placed your Apache access log
  pos_file /var/log/td-agent/nginx-access.log.pos # This is where you record file position
  tag nginx.access
  format nginx
</source>
<source>
  @type tail
  path /var/log/nginx/error.log
  pos_file /var/log/td-agent/nginx-error.log.pos
  tag nginx.error
  format /^(?<time>[^ ]+ [^ ]+) \[(?<log_level>.*)\] (?<pid>\d*).(?<tid>[^:]*): (?<message>.*)$/
</source>
 
<match nginx.*>
@type elasticsearch
logstash_format true
host 172.31.18.133 # elasticsearch IP
port 9200
index_name fluentd-nginx
type_name fluentd-nginx
</match>

And Restart FluentD:

1
service td-agent restart

For Debugging:

1
tail -f /var/log/td-agent/*log

If you got an error:

1
[error]: Permission denied @ rb_sysopen - /var/log/nginx/access.log

We just need add td-agent user to adm group:

1
sudo usermod -aG adm td-agent

Finally, access Kibana to view the logs as image below:

2016-02-16_1310

Centralize Nginx Logs with FluentD, Kibana and Elasticsearch

 

Nguyen Sy Thanh Son

3

Docker, Linux

Centralize Docker logs with FluentD, ElasticSearch and Kibana

3

Beside monitor topic, Log also is a important issue we need to concern. In this post, I just mention the way how to centralize Docker Logs using FluentD, Elasticsearch and Kibana

4a2ee5ea-f25a-4af5-b56b-4ad90a87a985-medium

Try not to become a man of success, but rather try to become a man of value

Secenario

We will install FluentD, ElasticSearch and Kibana in the same machine.

  • FluentD : Collect and Transfer log data to Elasticseach
  • Elasticsearch: Store and indexing log data to support searching/filtering log data
  • Kibana: A web view supports you search/filter and virtualize the log data

Prerequisites

  • We have a machine installed Ubuntu 14.04 with IP 192.168.1.191
  • We already installed Docker, Wget

Now, I will show you step by step to get stated to centralize the log data with FluentD

Elasticsearch

Download:

1
wget https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.1.1/elasticsearch-2.1.1.tar.gz

You should check the latest version at https://www.elastic.co/downloads/elasticsearch

Uncompress:

1
tar xvxf elasticsearch-2.1.1.tar.gz

Run:

1
2
cd elasticsearch-2.1.1
sudo bin/elasticsearch

Or run as daemon:

1
sudo bin/elasticsearch -d

Now, we have Elasticsearch run on port 9200.

FluentD

Add the lines below to /etc/security/limits file:

1
2
3
4
root soft nofile 65536
root hard nofile 65536
* soft nofile 65536
* hard nofile 65536

Open new terminal and type command below, make sure the output is correct:

1
2
$ ulimit -n
65535

Install FluentD (uses Treasure Data):

1
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh

For other Ubuntu version, please read: http://docs.fluentd.org/articles/install-by-deb

Now, we need to install Elasticsearch Plugin for FluentD:

1
2
3
sudo apt-get install make libcurl4-gnutls-dev --yes
sudo td-agent-gem install fluent-plugin-elasticsearch
sudo td-agent-gem install fluent-plugin-record-reformer

Add the content below to /etc/td-agent/td-agent.conf to setup Fluentd transfer all docker logs to Elasticsearch :

1
2
3
4
5
6
7
8
<match docker.*>
  @type elasticsearch
  logstash_format true
  host 192.168.1.191
  port 9200
  index_name fluentd-docker
  type_name fluentd-docker
</match>

And restart FluentD:

1
sudo /etc/init.d/td-agent restart

Docker

Now, we change docker configuration file to use Fluent as a Log Driver. Open /etc/default/docker, and add the line below:

1
DOCKER_OPTS="--log-driver=fluentd --log-opt fluentd-address=localhost:24224"

Add restart docker to apply the change:

1
sudo service docker restart

Kibana

We will run Kibana in Docker Container with command:

1
docker run --name kibana -e ELASTICSEARCH_URL=http://192.168.1.191:9200 -p 5601:5601 -d kibana:4.1.4

Now, you can access http://192.168.1.191:5601 to see Docker Logs in Kibana.

Tips

Delete a index in Elasticsearch:

1
curl -XDELETE 'http://localhost:9200/twitter/'

List all Indexes in Elasticsearch:

1
curl 'localhost:9200/_cat/indices?v'

Nguyen Sy Thanh Son

3