I find that Ubuntu disk space is eaten up heavily by keeping old kernels around.
Just do this:
|
1 2 |
apt-get autoclean apt-get autoremove |
And:
|
1 |
docker system prune |
(this can take a while, maybe 5 minutes? it has to iterate through all the old kernels and remove all of them.)
If you are planning to build a project with Microservices pattern and DevOps culture, you can use the source code here.
In this source code, I built two services, one for Frontend AngularJS and Bootstrap, one for Backend ExpressJS and Mongoose. All services in this project is dockerized and pushed to Docker Hub. You can read Dockerfile in each service for further details. To create a new service, you just create a new directory, writing source code for that service and update docker-compose.yml file.
The Orchestration of Project is written in docker-compose.yml file. So it is so easily to understand and run the project.
|
1 |
docker-compose up |
We also start from Virtual Machine layer with Vagrant.
|
1 |
vagrant up |
After that, you can access the application via link: http://172.20.20.20
You can read Vagrantfile to understand what we need to prepare for VMs.
With the source code, we will have a system architecture as below
Microservices DevOps Starter Project
This starter project also fully supports monitor by using Telegraf, InfluxDB, Grafana and Kapacitor. Supports centralizing Logs with fluentd, Kibana and Elasticsearch.
| Part | Title | Git Tag |
|---|---|---|
| 1 | Starting with ExpressJS | 1.0 |
| 2 | Logger with Winston | 1.1 |
| 4 | Config Management with Node-Config | 1.2 |
| 5 | Building Create User API | 2.1 |
| 6 | Adding Swagger Documents | 2.2 |
| 7 | Building Login API | 2.3 |
| 8 | Building Get User List/Detail API | 2.4 |
| 9 | Authorization all APIs | 2.5 |
| 10 | Unit Test | 2.6 |
| 11 | Building Config API | 3.0 |
| 12 | Using Cache | 3.1 |
| 13 | Using Queue | 3.2 |
| 14 | Starting AngularJS with Yeoman | 4.0 |
| 15 | Config Management for AngularJS | 4.1 |
| 16 | Building Login Page | 4.2 |
| 17 | Building List User Page | 4.2 |
| 18 | Pagination with AngularJS and Bootstrap | 4.3 |
| 19 | Multiple Languages | 4.4 |
| 20 | AngularJS Unit Test | 4.5 |
| 21 | Dockerize Aplication | 5.0 |
| 22 | Orchestration with Docker Compose | 5.1 |
This tutorial will show you how to install Docker and Docker Swarm on CentOS7. This tutorial is for Docker Version 1.11 and I use Vagrant and VirtualBox to create CentOS 7 in Virtual Machine.
Stop worrying about what you have to loose and start focusing on what you have to gain
Create a Vagrantfile for CentOS7
|
1 2 3 4 5 6 7 8 9 10 11 |
Vagrant.configure(2) do |config| config.vm.box = "centos/7" config.vm.synced_folder ".", "/vagrant", disabled: true config.vm.network "private_network", ip: "172.20.20.21" config.vm.provider "virtualbox" do |vb| vb.gui = false vb.memory = 2048 vb.cpus = 2 end end |
Turn on your Virtual Machine with command:
|
1 |
vagrant up |
After finishing creating CentOS server, you access the server via SSH by command:
|
1 |
vagrant ssh |
Go to root account:
|
1 |
sudo su |
In the next step, we will install Docker Engine following the official tutorial of Docker.
Add Docker Repository into CentOS
|
1 2 3 4 5 6 7 8 |
sudo tee /etc/yum.repos.d/docker.repo <<-'EOF' [dockerrepo] name=Docker Repository baseurl=https://yum.dockerproject.org/repo/main/centos/$releasever/ enabled=1 gpgcheck=1 gpgkey=https://yum.dockerproject.org/gpg EOF |
Running the installation
|
1 |
sudo yum install docker-engine |
Start Docker Service
|
1 |
sudo service docker start |
Enable Docker run on OS Start
|
1 |
sudo service enable docker |
Installing Docker Swarm via Docker Pull
|
1 |
docker pull swarm |
Open Docker Service Configuration file
|
1 |
vi /etc/systemd/system/multi-user.target.wants/docker.service |
In the line contains ExecStart, we add text as below
|
1 |
-H tcp://0.0.0.0:2375 |
And now, reload Docker Service
|
1 2 |
systemctl daemon-reload systemctl restart docker |
Now we can run Swarm Node
|
1 2 3 |
docker run -d --name swarm_joiner swarm join \ --addr=172.20.20.21:2375 \ token://acdb9dfa3ea6da0b0cfb2c819385fcd3 |
And running Swarm Manage
|
1 |
docker run -d -p 12375:2375 swarm manage token://acdb9dfa3ea6da0b0cfb2c819385fcd3 |
Finally, verify your work
|
1 |
docker -H :12375 info |
Thanks for your reading!!!
Beside monitor topic, Log also is a important issue we need to concern. In this post, I just mention the way how to centralize Docker Logs using FluentD, Elasticsearch and Kibana
Try not to become a man of success, but rather try to become a man of value
We will install FluentD, ElasticSearch and Kibana in the same machine.
Now, I will show you step by step to get stated to centralize the log data with FluentD
Download:
|
1 |
wget https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.1.1/elasticsearch-2.1.1.tar.gz |
You should check the latest version at https://www.elastic.co/downloads/elasticsearch
Uncompress:
|
1 |
tar xvxf elasticsearch-2.1.1.tar.gz |
Run:
|
1 2 |
cd elasticsearch-2.1.1 sudo bin/elasticsearch |
Or run as daemon:
|
1 |
sudo bin/elasticsearch -d |
Now, we have Elasticsearch run on port 9200.
Add the lines below to /etc/security/limits file:
|
1 2 3 4 |
root soft nofile 65536 root hard nofile 65536 * soft nofile 65536 * hard nofile 65536 |
Open new terminal and type command below, make sure the output is correct:
|
1 2 |
$ ulimit -n 65535 |
Install FluentD (uses Treasure Data):
|
1 |
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh |
For other Ubuntu version, please read: http://docs.fluentd.org/articles/install-by-deb
Now, we need to install Elasticsearch Plugin for FluentD:
|
1 2 3 |
sudo apt-get install make libcurl4-gnutls-dev --yes sudo td-agent-gem install fluent-plugin-elasticsearch sudo td-agent-gem install fluent-plugin-record-reformer |
Add the content below to /etc/td-agent/td-agent.conf to setup Fluentd transfer all docker logs to Elasticsearch :
|
1 2 3 4 5 6 7 8 |
<match docker.*> @type elasticsearch logstash_format true host 192.168.1.191 port 9200 index_name fluentd-docker type_name fluentd-docker </match> |
And restart FluentD:
|
1 |
sudo /etc/init.d/td-agent restart |
Now, we change docker configuration file to use Fluent as a Log Driver. Open /etc/default/docker, and add the line below:
|
1 |
DOCKER_OPTS="--log-driver=fluentd --log-opt fluentd-address=localhost:24224" |
Add restart docker to apply the change:
|
1 |
sudo service docker restart |
We will run Kibana in Docker Container with command:
|
1 |
docker run --name kibana -e ELASTICSEARCH_URL=http://192.168.1.191:9200 -p 5601:5601 -d kibana:4.1.4 |
Now, you can access http://192.168.1.191:5601 to see Docker Logs in Kibana.
Tips
Delete a index in Elasticsearch:
|
1 |
curl -XDELETE 'http://localhost:9200/twitter/' |
List all Indexes in Elasticsearch:
|
1 |
curl 'localhost:9200/_cat/indices?v' |
Recently, perhaps you hear too much about Microservices includes the concepts, cons, pros … I sure that you will ask yourself that how to implement it for your project? It is not easy to answer your question. In this post, i will tell you a part of my story after six month implement a social network project.
The story I would like to tell you is how to deploy a microservices with Docker Swarm and Consul (step by step instruction). With , my scenario, I have three nodes, installed Ubuntu 14.04 with 2G Ram and 2 CPUs. I used Vagrant and Virtualbox to build it.
Microservice with Docker Swarm and Consul
As the image above, We will install Consul in all nodes. With, Node Gateway, It is installed Nginx to load-balance and used as a API Gateway for your services. And we also install Consul-Template to help you change Nginx configuration once there is an new service is on or down via Consul.
With Node Agent-One and Node Agent Two, we will use it as Swarm Clusters. So we will install Swarm, and run Swarm Join for all of them. Beside that, we have to install Consul and Registrator. Consul is a Service Discovery, and Registrator will help you announce for Consul know if there is any service is up or down.
There are many steps we need to do to build this system. So I deviced my post to three parts:
I also publish all scripts to build this system to Github. You can check out it microservices-swarm-consul
So you just can download it and run:
|
1 |
vagrant up |
Note: Make sure you installed Virtualbox and Vagrant in your machine. And if there is any issue in my code. Please report me. Thank you
Before you reading my series of the posts, you should read about the concept of the tools we will use for deploy microservices
Docker is a platform for developers and sysadmins to develop, ship, and run applications. Docker lets you quickly assemble applications from components and eliminates the friction that can come when shipping code. Docker lets you get your code tested and deployed into production as fast as possible.
Docker Swarm is native clustering for Docker. It allows you create and access to a pool of Docker hosts using the full suite of Docker tools. Because Docker Swarm serves the standard Docker API, any tool that already communicates with a Docker daemon can use Swarm to transparently scale to multiple hosts
Registrator automatically registers and deregisters services for any Docker container by inspecting containers as they come online. Registrator supports pluggable service registries, which currently includes Consul, etcd and SkyDNS 2.
In next step, please go to Install Load Balance, API Gateway and Docker Swarm
If you are building docker images in your machine. I sure that your hard disk is growing up day by day. To stop it, you have to remove unnecessary images to release for your hard disk by command below:
|
1 |
sudo docker images | grep "^<none>" | awk '{print $3}' |
Or:
|
1 |
sudo docker rmi $(sudo docker images --filter "dangling=true" -q --no-trunc) |
If you are learning the way how to work with RABBITMQ + CELERY. This source code may help you. I wrote a small example to upload image to web server and use Celery to generate a thumbnail.
Difficult doesn’t mean impossible. It simply means that you have to work hard
This source code supports 2 ways to run it.
You can check the source code in Github: flask-celery-rabbitmq-generate-thumbnail
And the image at docker: flask-celery-rabbitmq-example
At the first, clone this source code to your local:
|
1 |
git clone https://github.com/thanhson1085/flask-celery-rabbitmq-generate-thumbnail.git |
|
1 2 |
sudo docker build -t thanhson1085/flask-celery-rabbitmq-example . |
Or pull from Docker Repo
|
1 2 |
sudo docker pull thanhson1085/flask-celery-rabbitmq-example |
Run Docker image
|
1 2 |
sudo docker run -i -t -p 5000:5000 thanhson1085/flask-celery-rabbitmq-example |
After running the docker image, you should wait for the output as below:
|
1 |
celery_task stderr | [2015-07-29 06:04:52,580: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672// |
I will show you how to run this source code from scratch. And i am using ubuntu server 14.04, installed virtualenv, pip.
Install RabbitMQ Server:
|
1 2 |
sudo apt-get install rabbitmq-server libjpeg8-dev python-imaging |
Fix the issue of PIL
|
1 2 |
ln -s /usr/lib/x86_64-linux-gnu/libjpeg.so /usr/lib |
Create environment to run the application with virtualenv:
|
1 2 3 4 |
cd flask-celery-rabbitmq-generate-thumbnail virtualenv env source env/bin/activate |
Install all packets required:
|
1 2 3 |
pip install -r requirement.txt pip install PIL --allow-external PIL --allow-unverified PIL |
Run web server to upload files
|
1 2 |
python ./server_celery.py |
Run the “generate thumbnail” task in Celery
|
1 2 |
celery worker -A generate_thumbnail_celery -l INFO |
Now, it is ready for testing. (http://localhost:5000). The page should be as image below:
Generate thumbnail with celery rabbitmq
As you know, Install Hadoop is not easy. And It require a virtual machine with high configuration. In my post, I will give you the shortest way to have Hadoop in your machine. It is installation via Docker. I assume that you had Docker in your Ubuntu 14.04
If your dreams do not scare you. They are too small.
To install the docker-cloudera-quickstart from docker-hub, simply use the following command:
|
1 2 |
docker pull thanhson1085/docker-cloudera-quickstart |
To start an instance in BACKGROUND (as daemon):
|
1 2 |
docker run -i -t -d caioquirino/docker-cloudera-quickstart |
To start an instance in FOREGROUND:
|
1 2 |
docker run -i -t thanhson1085/docker-cloudera-quickstart |
To open more terminal instances for the running instance:
|
1 2 3 |
docker ps docker exec -i -t CONTAINER_ID bash -l hadoop df -ls / |
Refer to:
In the previous article, I showed you how to install Docker and clone first image to local machine. So now, I will show you how to push a Ubuntu image to Docker Repository
It’s not about “having” time. It’s about making time
At first, please make sure that you had a account at docker.com (if not, please create one, it is free)
Lets start with Docker Login command in local machine:
|
1 |
sudo docker login |
For example, you need build a Ubuntu image that contains Apache2 web server. You create a Docker build file firstly.
|
1 2 3 |
mkdir -p docker/webserver cd docker/webserver/ vi Dockerfile |
Add content below to Docker build file:
|
1 2 3 |
FROM ubuntu:14.04 MAINTAINER Nguyen Sy Thanh Son <thanhson1085@gmail.com> RUN apt-get update && apt-get install -y apache2 |
Run build a Docker Ubuntu Image:
|
1 2 |
sudo chmod 755 Dockerfile sudo docker build -t="thanhson1085/webserver" . |
And test and install some packet if you want:
|
1 |
sudo docker run -t -i thanhson1085/webserver /bin/bash |
Finnaly, commit & push your Ubuntu Image to your Docker Repository:
|
1 2 |
sudo docker commit -m='first commit' -a="Nguyen Sy Thanh Son" b734c2178fb4 thanhson1085/webserver sudo docker push thanhson1085/webserver |
Note: thanhson1085/webserver was created in your repositories at docker.com
That is all, love Docker at the first eye.
Docker is a platform of developer and sysadmins . With Docker, you will be able to push whole your virtual machines to cloud. So you could easily use it everywhere and every time, or share to other people. I will share you the details of Docker installation in Ubuntu 14.04
Install Docker via apt-get:
|
1 2 |
sudo apt-get update sudo apt-get install docker.io |
Run bash completion:
|
1 |
source /etc/bash_completion.d/docker.io |
Install Apt Transport Https:
|
1 |
sudo apt-get update && apt-get install apt-transport-https |
Then, add the Docker repository key to your local keychain.
|
1 |
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9 |
Open /etc/apt/sources.list.d/docker.list and add text:
|
1 |
deb https://get.docker.com/ubuntu docker main |
And now, install lxc-docker package:
|
1 2 |
sudo apt-get update sudo apt-get install lxc-docker |
Test:
|
1 |
sudo docker run -i -t ubuntu /bin/bash |
