#linux

Sometimes (e.g You are developing an IoT Project), you need to connect you board to Virtual Machine via USB Port. This post will show you steps to connect USB port from Virtual Machine using Vagrant and Virtual Box.

At the first, you have to make sure that the USB device was connected to your computer with command:

VBoxManage.exe list usbhost

Host USB Devices:

UUID: 7c078866-a1ef-43fc-a67c-df06ae84e4ac

VendorId: 0x04f2 (04F2)

ProductId: 0xb40a (B40A)

Revision: 105.100 (105100)

Port: 0

USB version/speed: 2/High

Manufacturer: Chicony Electronics Co., Ltd

Address: {36fc9e60-c465-11cf-8056-444553540000}\0008

Current State: Busy

UUID: dec2a0f4-33db-4a42-9af7-ee7beb034c77

VendorId: 0x13d3 (13D3)

ProductId: 0x3362 (3362)

Revision: 0.1 (0001)

Port: 0

USB version/speed: 1/Full

Manufacturer: Atheros Communications

Product: Bluetooth USB Host Controller

SerialNumber: Alaska Day 2006

Address: {e0cbf06c-cd8b-4647-bb8a-263b43f0f974}\0000

Current State: Busy

UUID: 2f9068df-3345-477b-8de9-d0272bbbaf15

VendorId: 0x1a86 (1A86)

ProductId: 0x7523 (7523)

Revision: 2.84 (0284)

Port: 0

USB version/speed: 1/Full

Manufacturer: QinHeng Electronics

Product: USB2.0-Serial

Address: {4d36e978-e325-11ce-bfc1-08002be10318}\0001

Current State: Busy

The output above shows the USB Device has VendorID is 0x1a86 and ProductId is 0x7523. So we add the lines below to Vagrantfile:

1 2	vb.customize ["modifyvm", :id, "--usb", "on"] vb.customize ['usbfilter', 'add', '0', '--target', :id, '--name', 'ESP', '--vendorid', '0x1a86', '--productid', '0x7523']

Finally, starting your Virtual Machine:

1	vagrant up

In your Linux Virtual Machine, type lsusb to check the result.

March 6, 2017 Nguyen Sy Thanh Son

Linux

Using Ngrok to publish HTTP and SSH for your Rasperberry Pi

August 9, 2016 Nguyen Sy Thanh Son

I have a Rasperberry Pi 2, and i put it in my home in the silent. Sometimes I need to ssh to my Pi or view a website that I deployed to my Pi. I looked for a solution in the Internet. I found Ngrok. I really fitted my case. So in this post, I will share you the way to setup Ngrok on Rasperberry Pi in step by step.

Download Ngrok

You find the correct link to download at https://ngrok.com/download. You see “Linux ARM” and get the link.

After that, you do some commands to download and decompress:

cd /opt

mkdir ngrok

cd ngrox

wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-arm.zip

unzip ngrok-stable-linux-arm.zip

Creating Configuration File for Ngrok

We need to publish two tunnels (ssh, http). So we have to create a Ngrok configuration file to declare those services.

I create a file named ngrok.yml with content as below

web_addr: 0.0.0.0:4040

tunnels:

ssh:

proto: tcp

addr: 0.0.0.0:22

http:

proto: http

addr: 0.0.0.0:80

auth: username:pass

web_addr: 0.0.0.0:4040 means that I can access Ngrok Webpage on my Pi via http://RASPERBERRY_PI_IP_ADDRESS:4040

I also declare two protocols and address for two tunnels – SSH and HTTP

Installing Supervisor

We need run Ngrok as a daemon (in background). So I use Supervisor – a third-party process manager. You can install Supervisor with command.

1	apt-get install supervisor

After finishing the installation, we create a Supervisor configuration file at /etc/supervisor/config.d/ngrok.conf with content as below

[program:ngrok]

command=/opt/ngrok/ngrok start http ssh -log stdout --authtoken YOUR_NGROK_TOKEN_HERE -config=/opt/ngrok/ngrok.yml

stdout_logfile=/var/log/ngrok.out.log

stderr_logfile=/var/log/ngrok.err.log

Running the tunnels

In this step, we just need to run the program via Supervisor by command:

1 2	supervisord reread supervisor start ngrok

Verifying your works

Now, we open the browser and access http://PI_IP_ADDRESS:4040 to get the information of tunnels and try a test.

August 9, 2016 Nguyen Sy Thanh Son

Linux

Delete files older than X days in Linux

July 1, 2016 Nguyen Sy Thanh Son

When you setup a server, sometimes you need to clear temporary files to save the available hard disk. You also setup a crontab to run this command everyday.

1	find /path/to/files -type f -mtime +5 -exec rm {} \;

July 1, 2016 Nguyen Sy Thanh Son

Linux

Monitor server with CollectD, InfluxDB and Grafana

November 29, 2015 Nguyen Sy Thanh Son

In the software development, Monitor is a big feature we need to concern. It is not only for monitoring the production. But we also need to monitor your testing environment (e.g for performance test, stress test). So in this post, I will show you how to monitor server with CollectD, InfluxDB and Grafana.

At the first, we have to install CollectD in the agent machine. In my case, the agent machine also uses Ubuntu 14.04. And we have to install CollectD version 5.5.0+. Why? Because all the versions easier 5.5.0, Plugin cpu of CollectD only use jitters metric that we can not show properly in Grafana.

Gopher and Gun T-Shirt

My scenario is:

I have 2 servers installed Ubuntu 14.04
One at address 192.168.1.194. And the other one is 192.168.1.192 that we need to monitor
They can ping each others.

We will monitor server 192.168.1.192 with CollectD

So now, Install CollectD to Server 192.168.1.192.

CollectD

I recommend you install CollectD from the source code to make sure that you are using the correct version.

Before running build the packet from source code, we need to install the dependencies:

1	sudo apt-get install libcurl4-gnutls-dev

(libcurl4-gnutls-dev to enable curl and nginx plugin)

cd /tmp/

wget https://collectd.org/files/collectd-5.5.0.tar.bz2

tar -jxf collectd-5.5.0.tar.bz2

cd collectd-5.5.0

sudo ./configure

sudo make all install

sudo wget -O /etc/init.d/collectd https://raw.githubusercontent.com/martin-magakian/collectd-script/master/collectd.init

sudo chmod 744 /etc/init.d/collectd

# Run CollectD

sudo /etc/init.d/collectd start

Edit /opt/collectd/etc/collectd.conf to enable CollectD collect cpu data with percentage instead of jitters:

ReportByCpu true

ReportByState true

ValuesPercentage true

</Plugin>

And enable network plugin to push monitor data to InfluxDB at address 192.168.1.194, port 25826

LoadPlugin network

# # client setup:

Server "192.168.1.194" "25826"

</Plugin>

Restart CollectD

1 2	sudo /etc/init.d/collectd stop sudo /etc/init.d/collectd start

Debug:

1	tcpdump -i eth0 -p -n dst port 25826

And now, we install InfluxDB and Grafana in Server 192.168.1.194

InfluxDB

Install with dpkg command:

wget http://influxdb.s3.amazonaws.com/influxdb_0.9.5_amd64.deb

sudo dpkg -i influxdb_0.9.5_amd64.deb

sudo service influxdb start

InfluxDB Web run on port 8083, so you access web at address http://192.168.1.194:8083 and create collectd database:

1	create database collectd

Edit /etc/influxdb/influxdb.conf to enable collectd. See the content below:

[collectd]

enabled = true

bind-address = ":25826" # the bind address

database = "collectd" # Name of the database that will be written to

retention-policy = ""

batch-size = 5000 # will flush if this many points get buffered

batch-pending = 10 # number of batches that may be pending in memory

batch-timeout = "10s"

read-buffer = 0 # UDP read buffer size, 0 means to use OS default

typesdb = "/usr/share/collectd/types.db"

Please make note that with the configuration above we use port 25826 on UDP to receive monitor data that pushed from CollectD

And the next step, restart InfluxDb:

1	sudo /etc/init.d/influxdb restart

For troubleshoot the issue with InfluxDB, we can view logs:

1	sudo tail -f /var/log/influxdb/influxd.log

Grafana

Download and install via dpkg command:

wget https://grafanarel.s3.amazonaws.com/builds/grafana_2.5.0_amd64.deb

sudo dpkg -i grafana_2.5.0_amd64.deb

sudo service grafana-server start

If you meet an error as below:

/tmp# sudo dpkg -i grafana_2.5.0_amd64.deb

Selecting previously unselected package grafana.

(Reading database ... 61771 files and directories currently installed.)

Preparing to unpack grafana_2.5.0_amd64.deb ...

Unpacking grafana (2.5.0) ...

dpkg: dependency problems prevent configuration of grafana:

grafana depends on libfontconfig; however:

Package libfontconfig is not installed.

dpkg: error processing package grafana (--install):

dependency problems - leaving unconfigured

Processing triggers for ureadahead (0.100.0-16) ...

Errors were encountered while processing:

grafana

You can correct it by the commands:

1 2	apt-get install libfontconfig1 apt-get -f install

Grafana run on port 3000. So you can access via link http://192.168.1.194:3000.

To connect InfluxDb, you follow the path: Grafana => Data Source => Add New => And Input the information as the image below:

Grafana connects to InfluxDB (default account root/root)

After that, create New Dash Board => New Panel => Graph as image below:

Grafana + CollectD CPU (percentage)

I created a new dashboard that help me monitor CPU and RAM:

Grafana, CollectD and InfluxDB

Tips

To collect interface data in bytes per second we should use:

1	SELECT derivative("value") FROM "interface_" WHERE "host" = 'yourhost' AND "instance" = 'eth0' AND "type" = 'if_octets'

November 29, 2015 Nguyen Sy Thanh Son

Linux

Logrotate Nginx Custom Logs

June 16, 2015 Nguyen Sy Thanh Son

As you know, when you create a site in nginx webserver, you will want to add custom logs for that site. So this post is a small tip to help you how to do it perfectly.

In your nginx site config (e.g /etx/nginx/site-availables/example.com), you add custom logs as below:

1 2	access_log /usr/share/nginx/example.com/nginx-access.log combined; error_log /usr/share/nginx/example.com/nginx-error.log warn;

So now, your log files will be bigger day by day. You will have to remember to delete the log files if you do not want to your hard disk be full. What should you do now? Fortunately, Logrotate will help you compress. clean and backup it daily.

You just have to add (append) the content below to /etc/logrotate.d/nginx:

/usr/share/nginx/example.com/*.log {

daily

missingok

rotate 52

compress

delaycompress

copytruncate

notifempty

create 0640 www-data adm

sharedscripts

prerotate

if [ -d /etc/logrotate.d/httpd-prerotate ]; then \

run-parts /etc/logrotate.d/httpd-prerotate; \

fi \

endscript

postrotate

[ -s /run/nginx.pid ] && kill -USR1 `cat /run/nginx.pid`

endscript

}

Finaly, run the command below to force the update.

1	logrotate -f -v /etc/logrotate.d/nginx

It is done.

June 16, 2015 Nguyen Sy Thanh Son

Linux, Others

Install Cloudera Hadoop in Ubuntu 14.04

May 27, 2015 Nguyen Sy Thanh Son

As you know, Install Hadoop is not easy. And It require a virtual machine with high configuration. In my post, I will give you the shortest way to have Hadoop in your machine. It is installation via Docker. I assume that you had Docker in your Ubuntu 14.04

c85fe663-679e-4fe7-9f5f-f413af3f15cc-medium

If your dreams do not scare you. They are too small.

Prerequisites

Docker
Ubuntu 14.04
Ram 4G
CPU 2 core

Install

To install the docker-cloudera-quickstart from docker-hub, simply use the following command:

1 2	docker pull thanhson1085/docker-cloudera-quickstart

Use

To start an instance in BACKGROUND (as daemon):

1 2	docker run -i -t -d caioquirino/docker-cloudera-quickstart

To start an instance in FOREGROUND:

1 2	docker run -i -t thanhson1085/docker-cloudera-quickstart

To open more terminal instances for the running instance:

docker ps

docker exec -i -t CONTAINER_ID bash -l

hadoop df -ls /

Test MapReduce with WordCount example

Refer to: http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html

Refer to:

Docker series posts:https://sonnguyen.ws/tag/docker/
Github: https://github.com/thanhson1085/docker-cloudera-quickstart

May 27, 2015 Nguyen Sy Thanh Son

Linux

Change hostname without reboot Ubuntu 14.04

May 14, 2015 Nguyen Sy Thanh Son

This is a small tutorial to help you change hostname of server without reboot the system. This will require 3 simple steps

Step 1: change hostname by command line

1	sudo hostname [new_host_name]

Step 2: edit hostname file with new hostname

1	sudo vi /etc/hostname

Step 2: edit hosts file

1	sudo vi /etc/hosts

Edit the line below:

1	127.0.1.1 [new_hostname]

It is done.

May 14, 2015 Nguyen Sy Thanh Son

Docker, Linux

Install Scrapy Portia in Ubuntu Server 14.04

March 19, 2015 Nguyen Sy Thanh Son

There are 2 ways to install Scrapy Portia in Ubuntu Server 14.04: Install via virtualenv/pip or Install Docker. Please see the details below

The key is not to prioritize what’s on your schedule, but to schedule your priorities

Option 1: Install Portia via Virtualenv/pip

Please make sure that virtualenv and pip is ready to use in your server.

Do activate Portia environment:

1 2	virtualenv portia --no-site-packages source portia/bin/activate

Download Protia source code:

1 2	cd portia/ git clone https://github.com/scrapinghub/portia.git

Install libraries, packets:

pip install -r requirements.txt

sudo pip install -r requirements.txt

pip install -e ./slybot

sudo apt-get install libxml2-dev libxslt-dev python-dev zlib1g-dev libffi-dev libssl-dev

pip install cryptography

Check port 9001, and run Portia:

1 2	sudo lsof -i :9001 sudo twistd -n slyd

Test Portia at link: http://localhost:9001/static/main.html

Install Portia via Docker (recommended):

Make sure that Docker was installed in your server and ready to use:

Download source code:

1 2	git clone https://github.com/scrapinghub/portia cd portia/

Build Portia Image:

1 2	docker build -t portia . sudo docker build -t portia .

And run Protia Docker container:

docker run -i -t --rm

-v <PROJECT_FOLDER>/data:/app/slyd/data:rw -p 9001:9001 --name portia portia

-v data:/app/slyd/data:rw -p 9001:9001 --name portia portia

March 19, 2015 Nguyen Sy Thanh Son

Linux

Install CoreOS, Vagrant in Ubuntu Server14.04

February 24, 2015 Nguyen Sy Thanh Son

In this article, I will show you 2 steps. The first step is install Vagrant Ubuntu Server 14.04. The second one is install CoreOS in Vagrant. Actually, I run Ubuntu Server in Vmware. So to run CoreOS, I have to turn on Virtualize Intel. Please see image below:

Install VirtualBox in Ubuntu:

1 2	sudo apt-get update sudo apt-get install virtualbox

Install Vagrant

1	sudo apt-get install vagrant

Upgrade Vagrant:

1 2	https://dl.bintray.com/mitchellh/vagrant/vagrant_1.7.2_x86_64.deb sudo dpkg -i vagrant_1.7.2_x86_64.deb

Install Git if needed

1	sudo apt-get install git

Get CoreOS-Vagrant Sample

1 2	git clone https://github.com/coreos/coreos-vagrant.git cd coreos-vagrant

Create config file from sample config file:

cp config.rb.sample config.rb

vi config.rb

cp user-data.sample user-data

vi user-data

Turn Vagrant Machine on:

1	sudo vagrant up

Try to run CoreOS nodes:

1	sudo vagrant ssh core-01 -- -A

February 24, 2015 Nguyen Sy Thanh Son

Linux

How to trigger a Jenkins build from Slack

January 29, 2015 Nguyen Sy Thanh Son

As you know, Slack is a great product for team communication. And Jenkins help you setup a automation build (Continuous Integration). In this article, I will show the way how to use Slack Commands to trigger a Jenkins build.

Buy Golang or Gohome t-shirt

In your Jenkins, you install Build Authorization Token Root Plugin, as image below

In configuration job Jenkins, you check “trigger builds remotely” and add a token (you can get a random token here). See image below

So now, you can trigger a remote build by url:

1	http://[jenkins_url]/buildByToken/buildWithParameters?job=[job_name]&token=[your_token]&[param=value]

For details, you can read https://wiki.jenkins-ci.org/display/JENKINS/Build+Token+Root+Plugin

In your Slack, you create a Slack Commands in Slack (e.g: /build_example).

Command textbox, type “/build_example”

URL textbox, add http://[jenkins_url]/buildByToken/buildWithParameters?job=[job_name]&token=[your_token]&[param=value]

Method combobox, choose GET. Finally, click “Save”

To test your work, go to Slack chat page, you type “/build_example” and check the result.

If you want to build with parameters from Slack (e.g /build [branch]). You need write a proxy script to handle the build. You can take a look to https://github.com/thanhson1085/slack-jenkins as a example

67071ae5-5df4-49f7-b91b-20c40b1a2cd1-medium

January 29, 2015 Nguyen Sy Thanh Son