#devops

AngularJS, Docker, NodeJS

Starting a Miroservices Project with NodeJS and Docker

8

If you are planning to build a project with Microservices pattern and DevOps culture, you can use the source code here.

In this source code, I built two services, one for Frontend AngularJS and Bootstrap, one for Backend ExpressJS and Mongoose. All services in this project is dockerized and pushed to Docker Hub. You can read Dockerfile in each service for further details. To create a new service, you just create a new directory, writing source code for that service and update docker-compose.yml file.

Run

The Orchestration of Project is written in docker-compose.yml file. So it is so easily to understand and run the project.

1
docker-compose up

Using Vagrant and Virtualbox

We also start from Virtual Machine layer with Vagrant.

1
vagrant up

After that, you can access the application via link: http://172.20.20.20

You can read Vagrantfile to understand what we need to prepare for VMs.

System Architecture

With the source code, we will have a system architecture as below

Microservices DevOps Starter Project

Microservices DevOps Starter Project

 

Monitor & Logs

This starter project also fully supports monitor by using Telegraf, InfluxDB, Grafana and Kapacitor. Supports centralizing Logs with fluentd, Kibana and Elasticsearch.

Contents

Part Title Git Tag
1 Starting with ExpressJS 1.0
2 Logger with Winston 1.1
4 Config Management with Node-Config 1.2
5 Building Create User API 2.1
6 Adding Swagger Documents 2.2
7 Building Login API 2.3
8 Building Get User List/Detail API 2.4
9 Authorization all APIs 2.5
10 Unit Test 2.6
11 Building Config API 3.0
12 Using Cache 3.1
13 Using Queue 3.2
14 Starting AngularJS with Yeoman 4.0
15 Config Management for AngularJS 4.1
16 Building Login Page 4.2
17 Building List User Page 4.2
18 Pagination with AngularJS and Bootstrap 4.3
19 Multiple Languages 4.4
20 AngularJS Unit Test 4.5
21 Dockerize Aplication 5.0
22 Orchestration with Docker Compose 5.1

Nguyen Sy Thanh Son

8

Linux

Monitor Nginx Response Time with FluentD, Kibana and Elasticsearch

2

In my previous post, I showed you How to centralize Nginx logs.  So now, I will use FluentD, Kibana and ElasticSearch to collect Nginx Response Time.

To implement it, we have to change Nginx Log Format. Because, in the default, Nginx does not store Response Time to access.log file. So we change nginx.conf as below:

1
2
log_format upstreamlog '[$time_local] $remote_addr - $remote_user - $server_name to: $upstream_addr $request $status upstream_response_time $upstream_response_time msec $msec request_time $request_time';
access_log /var/log/nginx/access.log upstreamlog;

After reload Nginx, you can try to tail access.log. The result should be as below:

1
[26/Feb/2016:06:42:49 +0000] 118.70.190.253 - - - example.com to: 172.31.27.195:33687 GET /api/v1/post/31856?access_token=r12dx4whN147sm0YC8IYTyn-bfkyWg52-F08_BI_pfKBOo1JCliMwhI_g8qtffAm9QUoRjNiF8TULGpjw8sgbQ&limit=15 HTTP/1.1 200 upstream_response_time 0.140 msec 1456468969.606 request_time 0.259

Now, we will create a regex string that matches with the log format above:

1
/^\[(?([^ ]* [^\]]*))\] (?[^ ]*) - ([^ ]*) - ([^ ]*) to: (?[^ ]*) (?[^ ]*) (?[^ ]*) ([^ ]*) (?<code>[^ ]*) upstream_response_time (?[^ ]*) msec ([^ ]*) request_time ([^ ]*)/

And Insert to td-agent.conf as below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
<source>
  @type tail
  path /var/log/nginx/access.log #...or where you placed your Apache access log
  pos_file /var/log/td-agent/nginx-access.log.pos # This is where you record file position
  tag nginx.access
  time_format %d/%b/%Y:%H:%M:%S %z
  format /^\[(?<time>([^ ]* [^\]]*))\] (?<remote>[^ ]*) - ([^ ]*) - ([^ ]*) to: (?<host>[^ ]*) (?<method>[^ ]*) (?<path>[^ ]*) ([^ ]*) (?<code>[^ ]*) upstream_response_time (?<response_time>[^ ]*) msec ([^ ]*) request_time ([^ ]*)/
</source>
<source>
  @type tail
  path /var/log/nginx/error.log
  pos_file /var/log/td-agent/nginx-error.log.pos
  tag nginx.error
  format /^(?<time>[^ ]+ [^ ]+) \[(?<log_level>.*)\] (?<pid>\d*).(?<tid>[^:]*): (?<message>.*)$/
</source>
 
<source>
  @type tail
  path /build/clotify/logs/*/*.log
  pos_file /var/log/td-agent/clotify.log.pos
  tag clotify
  format /^(?<time>([^ ]* [^ ]*)) (?<log_level>[^ ]*): (?<service>[^ ]*) (?<message>.*)\[in (?<filename>[^ ]*):(?<lineno>[^ ]*)\]/
</source>
 
<match nginx.*>
  @type elasticsearch
  logstash_format true
  host 172.31.18.133
  port 9200
  index_name fluentd-aws-dev-nginx
  type_name fluentd-aws-dev-nginx
</match>

After restarting td-agent. We can wait a minute and then view kibana. The outfit should be:

2016-02-26_1800

Monitor Nginx Response Time with FluentD, Kibana and Elasticsearch

For basic installation, please refer to https://sonnguyen.ws/centralize-docker-logs-with-fluentd-elasticsearch-and-kibana/

Nguyen Sy Thanh Son

2