#hdfs

How to save a file to HDFS with Python

This source code is a simple example the way how to upload image and save it to HDFS. This program will connect HDFS via webhdfs. Actually, it is easier than you think. The most dificulty is preparing environment to test your source code

Help people, even when you know they can't help you back

Help people, even when you know they can’t help you back

Prerequisites:

  • python3
  • virtualenv (optional)
  • pip3
  • Flask==0.10.1
  • Flask-Swagger
  • Hadoop with webhdfs

Installation:

To install Hadoop, pleas take a look at https://github.com/thanhson1085/docker-cloudera-quickstart

I will show you the detail, just hope that it will save your time.

At the first, using Docker to clone docker-cloudera-quickstart to your local.

And run that image, and please do not forget expose webhdfs port(50070, 50075):

There is 2 ways to run my source code. I recommend you using Docker

Using Docker

Install in normal way

There is a trick here. You have to edit /etc/hosts file to your machine know where is HDFS server. You can use “docker inspect” command to get HDFS server ip address.

Finally, you have a HDFS Server with webhdfs support.

Clone this source code to your local

Run the commands below to install and create environment to run the application

Test

I integrated swagger in this app. So you can use Swagger UI to test this app. And you can find swagger schema at http://localhost:5000/docs

After upload the files. You can use hadoop command to look up in hdfs.

Go to Hadoop server:

List files in hdfs:

The output should be:

Github: https://github.com/thanhson1085/flask-webhdfs

Install Cloudera Hadoop in Ubuntu 14.04

As you know,  Install Hadoop is not easy. And It require a virtual machine with high configuration.  In my post, I will give you the shortest way to have Hadoop in your machine. It is installation via Docker. I assume that you had Docker in your Ubuntu 14.04

c85fe663-679e-4fe7-9f5f-f413af3f15cc-medium

If your dreams do not scare you. They are too small.

Prerequisites

  • Docker
  • Ubuntu 14.04
  • Ram 4G
  • CPU 2 core

Install

To install the docker-cloudera-quickstart from docker-hub, simply use the following command:

Use

To start an instance in BACKGROUND (as daemon):

To start an instance in FOREGROUND:

To open more terminal instances for the running instance:

Test MapReduce with WordCount example

Refer to: http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html

Refer to: