#Docker

Run Docker Swarm with Upstart

In Ubuntu 14.04, to run a job in background an on startup, you should use Upstart. In this post, I will show you the way to run Swarm with Upstart.  I think that it is more complicated that SystemD used by CoreOS or Ubuntu 15.04.

At the first, we need to open /etc/default/docker file and add a line:

And restart Docker:

We need to do this step so that Docker Daemon run at all interfaces and on port 2375. Now, you ready to run Swarm on the machine.

To run Swarm Join, you create file /etc/init/swarm-join.conf with content:

Now, run Swarm Join job:

To run Swarm Manage, you create file /etc/init/swarm-manage.conf with content:

Now run Swarm Manage job:

Done

How to save a file to HDFS with Python

This source code is a simple example the way how to upload image and save it to HDFS. This program will connect HDFS via webhdfs. Actually, it is easier than you think. The most dificulty is preparing environment to test your source code

Help people, even when you know they can't help you back

Help people, even when you know they can’t help you back

Prerequisites:

  • python3
  • virtualenv (optional)
  • pip3
  • Flask==0.10.1
  • Flask-Swagger
  • Hadoop with webhdfs

Installation:

To install Hadoop, pleas take a look at https://github.com/thanhson1085/docker-cloudera-quickstart

I will show you the detail, just hope that it will save your time.

At the first, using Docker to clone docker-cloudera-quickstart to your local.

And run that image, and please do not forget expose webhdfs port(50070, 50075):

There is 2 ways to run my source code. I recommend you using Docker

Using Docker

Install in normal way

There is a trick here. You have to edit /etc/hosts file to your machine know where is HDFS server. You can use “docker inspect” command to get HDFS server ip address.

Finally, you have a HDFS Server with webhdfs support.

Clone this source code to your local

Run the commands below to install and create environment to run the application

Test

I integrated swagger in this app. So you can use Swagger UI to test this app. And you can find swagger schema at http://localhost:5000/docs

After upload the files. You can use hadoop command to look up in hdfs.

Go to Hadoop server:

List files in hdfs:

The output should be:

Github: https://github.com/thanhson1085/flask-webhdfs

Flask RabbitMQ Celery example

If you are learning the way how to work with RABBITMQ + CELERY. This source code may help you. I wrote a small example to upload image to web server and use Celery to generate a thumbnail.

Difficult doesn’t mean impossible. It simply means that you have to work hard

This source code supports 2 ways to run it.

You can check the source code in Github: flask-celery-rabbitmq-generate-thumbnail

And the image at docker: flask-celery-rabbitmq-example

At the first, clone this source code to your local:

Using Docker

  1. Build from Dockerfile

Or pull from Docker Repo

Run Docker image

After running the docker image, you should wait for the output as below:

Install packets normally (Ubuntu 14.04)

I will show you how to run this source code from scratch. And i am using ubuntu server 14.04, installed virtualenv, pip.

Install RabbitMQ Server:

Fix the issue of PIL

Create environment to run the application with virtualenv:

Install all packets required:

Run web server to upload files

Run the “generate thumbnail” task in Celery

Now, it is ready for testing. (http://localhost:5000). The page should be as image below:

generate thumbnail with celery rabbitmq

Generate thumbnail with celery rabbitmq

Install Scrapy Portia in Ubuntu Server 14.04

There are 2 ways to install Scrapy Portia in Ubuntu Server 14.04: Install via virtualenv/pip or Install Docker. Please see the details below

The key is not to prioritize what's on your schedule, but to schedule your priorities

The key is not to prioritize what’s on your schedule, but to schedule your priorities

Option 1: Install Portia via Virtualenv/pip

Please make sure that virtualenv and pip is ready to use in your server.

Do activate Portia environment:

Download Protia source code:

 

Install libraries, packets:

Check  port 9001, and run Portia:

Test Portia at link: http://localhost:9001/static/main.html

Install Portia via Docker (recommended):

Make sure that Docker was installed in your server and ready to use:

Download source code:

Build Portia Image:

And run Protia Docker container:

Docker Build a Ubuntu Image

In the previous article, I showed you how to install Docker and clone first image to local machine. So now, I will show you how to push a Ubuntu image to Docker Repository

It’s not about “having” time. It’s about making time

 

At first, please make sure that you had a account at docker.com (if not, please create one, it is free)

Lets start with Docker Login command in local machine:

For example, you need build a Ubuntu image that contains Apache2 web server. You create a Docker build file firstly.

Add content below to Docker build file:

Run build a Docker Ubuntu Image:

And test and install some packet if you want:

Finnaly, commit & push your Ubuntu Image to your Docker Repository:

Note:  thanhson1085/webserver was created in your repositories at docker.com

That is all, love Docker at the first eye.

 

Install Docker in Ubuntu 14.04

Docker is a platform of developer and sysadmins .  With Docker, you will be able to push whole your virtual machines  to cloud.  So you could easily use it everywhere and every time, or share to other people. I will share you the details of Docker installation in Ubuntu 14.04

Install Docker via apt-get:

Run bash completion:

Install Apt Transport Https:

Then, add the Docker repository key to your local keychain.

Open /etc/apt/sources.list.d/docker.list and add text:

And now, install lxc-docker package:

Test: