How to save a file to HDFS with Python

This source code is a simple example the way how to upload image and save it to HDFS. This program will connect HDFS via webhdfs. Actually, it is easier than you think. The most dificulty is preparing environment to test your source code

Help people, even when you know they can't help you back

Help people, even when you know they can’t help you back


  • python3
  • virtualenv (optional)
  • pip3
  • Flask==0.10.1
  • Flask-Swagger
  • Hadoop with webhdfs


To install Hadoop, pleas take a look at https://github.com/thanhson1085/docker-cloudera-quickstart

I will show you the detail, just hope that it will save your time.

At the first, using Docker to clone docker-cloudera-quickstart to your local.

And run that image, and please do not forget expose webhdfs port(50070, 50075):

There is 2 ways to run my source code. I recommend you using Docker

Using Docker

Install in normal way

There is a trick here. You have to edit /etc/hosts file to your machine know where is HDFS server. You can use “docker inspect” command to get HDFS server ip address.

Finally, you have a HDFS Server with webhdfs support.

Clone this source code to your local

Run the commands below to install and create environment to run the application


I integrated swagger in this app. So you can use Swagger UI to test this app. And you can find swagger schema at http://localhost:5000/docs

After upload the files. You can use hadoop command to look up in hdfs.

Go to Hadoop server:

List files in hdfs:

The output should be:

Github: https://github.com/thanhson1085/flask-webhdfs