If you are using Elasticsearch to centralize your log data, that is a great solution. However, after few months, you will have huge log data store in your server hard disk. You have to clean the old log data that you are sure that you will not use it in future.

To delete the 10-day-ago log data, we can use the script below:

#/bin/bash

d=$(date --date="10 day ago" +%Y.%m.%d)

curl -XDELETE http://localhost:9200/logstash-${d}?pretty

So we should run the script above every night to delete data. So we use cronjob:

1	30 11 * * * bash /home/ubuntu/backend/scripts/es_retention.sh > /dev/null 2>&1

Thanks for your reading

Software is like sex t-shirt

April 27, 2017 Nguyen Sy Thanh Son

Python

My First AI Application

April 19, 2017 Nguyen Sy Thanh Son

I am not Data or AI scientist, I am just a Developer. I tried to learn AI in my free time. There are a lot of information from Internet. After reading many many articles and source codes, I still did not find the approach. I confused by many AI algorithms as CNN, Logistic Regresstion, K-Tree, N-Tree … As you know, I am a developer, so I did not understand at all. :).

Finally, I find the solution. The thing I need just Tensorflow.

To start, I try to build AI that can classify Dog/Cat Image. I followed the steps below.

Step 0 – Installation

I installed Tensorflow on my Ubuntu 14.04 Server by commands:

sudo apt-get install python-pip

sudo pip install virtualenv

virtualenv -p /usr/bin/python3 env

. env/bin/activate

pip --version

pip install --upgrade tensorflow

To understand this, you have to understand Python, PIP and Virtualenv

Step 1 – The Samples

I search Dog/Cat Image on Google Image. After that, I resize/save them to my local storage.

Please see link: https://github.com/thanhson1085/Hello-AI/tree/master/dataset/training_set

Step 2 – Training

After having the samples, I started to research to train my AI. I found the solution here: https://github.com/thanhson1085/Hello-AI/blob/master/retrain.py

It was copy from Tensorflow Example: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/image_retraining/retrain.py

Now, I run command to train my AI by using Inception V3 Model and my Training Dataset

1	python retrain.py --bottleneck_dir=tmp/bottlenecks --how_many_training_steps 500 --model_dir=tmp/inception --output_graph=output/retrained_graph.pb --output_labels=output/retrained_labels.txt --image_dir=dataset/training_set/

The most import thing is the output file output/retrained_graph.pd. This is the trained model file, I will use this file for my AI.

Step 3 – AI Service

After having the trained model file. I wrote the AI Service. I used Flask Framework to handle uploading image. When user uploads a image, my AI service will check it is CAT or DOG. Please see the source code:

1 2	file.save(file_path) thisis = app.catordog.run(file_path)

For the details, see file https://github.com/thanhson1085/Hello-AI/blob/master/app.py and https://github.com/thanhson1085/Hello-AI/blob/master/catordog.py

Step 4 – Deployment

Now, everything is reading to deploy. I chose Heroku to run my AI service https://thanhson1085-hello-ai.herokuapp.com/

Finally, we have a simple AI, we understand how it works, how to train it and what is most important thing we need to build a AI.

Thanks for your reading!

Tensorflow T-shirt

April 19, 2017 Nguyen Sy Thanh Son

Docker, Linux

How to HOT Backup Database (MongoDB, MySQL, ES …) to AWS S3

April 17, 2017 Nguyen Sy Thanh Son

Actually, there are many way to backup your database. You can using RSYNC, MongoDump for Mongo, S3 Backup Plugin for ElasticSearch. However, this post will show you the way I used in my project. Maybe, It is not perfect for all case. But in my case, it is really perfect.

Docker Ironman T-shirt

I am running a project with Microservice Architecture. All Databases and Services are running in Docker Container. In my plan, I have to backup all databases every night.

At the beginning, I tried to use tar command to compress the data, and then I use command aws s3 copy to copy backup data to S3. It seems work. But tar command makes MongoDB stop working. I tried to google to solve the problem. I found the solution is rsync command.

The backup process should be implement in three steps:

Use rsync command to copy the data to other location
Compress the data by tar command
Move the compressed data to AWS S3

The script should be:

echo Backup MongoDb Data ...

rsync -avz /var/lib/docker/volumes/backend_mongodb/ /tmp/mongodb && \

tar cvzf mongodb-${dt}.tar.gz /tmp/mongodb/ && \

aws s3 cp mongodb-${dt}.tar.gz s3://prjbackups/${env}/mongodb/mongodb-${dt}.tar.gz

Thanks for your reading.