Sunday, 19 June 2016

Configure Logstash to listen to HTTP requests as Input

Prerequisite:
  • Installed Logstash 2.3.1


Step 1: Install http plugin in Logstash




Step 2: Configure “.conf” file to receive http input

Create a “http-message.conf” file in “logstash-2.3.1\bin” folder.

Following shall be the content of the file:




Step 3: Run Logstash with the above “.conf” file



Step 4: Simulate HTTP request from PostMan




Click on Send, then verify message on Logstash server.

Monday, 13 June 2016

Configure JMeter to Load Test Rabbit MQ

Configure JMeter to Load Test Rabbit MQ

Pre-Requisite:
·         Installed RabbitMQ 3.6.1
·         Installed JMeter 3.0
·         Installed Ant 1.9.7

Note: 
I was getting lot of request for JMeterAMQP.jar file, so I have added the same into the following GitHub Repo: 
https://github.com/jatinaneja/jmeter_amqp

With this in place, one can jump to Step 2 directly, download JMeterAMQP.jar from above GitHub Repo and use it for copy operation.

Step 1: Download JMeter-RabbitMQ plugin code from GitHub

Visit GitHub at https://github.com/jlavallee/JMeter-Rabbit-AMQP and download the zip file of source code.

Extract the zip content.

Open terminal, go to the source of the extracted folder(JMeter-Rabbit-AMQP-master) and type ant (to build source code)



Step 2: Copy JMeterAMQP.jar into JMeter

Once step 1 is completed, go to folder “\dist” under “JMeter-Rabbit-AMQP-master\target\dist” folder.

Copy JMeterAMQP.jar into “\apache-jmeter-3.0\lib\ext” folder.

Step 3: Copy amqp-client-3.6.1.jar into JMeter

Download amqp-client-3.6.1.jar from Maven Repo (http://mvnrepository.com/artifact/com.rabbitmq/amqp-client/3.6.1)

Copy this jar into “apache-jmeter-3.0\lib” folder.

Step 4: Prepare Test plan in JMeter

Right click Test Plan and go to Add->Thread(Users)->Thread Group. Give a name to Thread Group





   
Right click your group and go to Add->Sampler->AMQP Publisher



Right click your group and go to Add->Listener->Graph Results



Now save the test plan and execute.



The test case will publish 4000(1000 * 4 time) message into the exchange (for which details were given when Sampler was added).

I would recommend following book from Amazon to have grip on better understanding regarding performance testing with JMeter :



Tuesday, 19 January 2016

Create and Run a Docker Image

Create and Run a Docker Image

Step-by-step guide
If you do not have Docker Toolbox installed follow instructions on the Docker Toolbox download page (at https://www.docker.com/toolbox) to install and setup the Docker tools.

Maven based Java Application Docker

Java Application Environment:

a)      Spring Based Web Application

b)      Maven to build the project

c)      Tomcat as Application Server

d)      MySQL as Database


1.    Open Project Directory - 
      In docker terminal, go to the root of your repository.



2.      Create a Dockerfile  - 
      Type command "touch Dockerfile" as shown below:



3.      Open Dockerfile  - 
      To add docker commands which will execute following tasks:
·         Take JDK 8 and Tomcat 8 base image
·         Copy tomcat conf file into docker – The file “tomcat-users.xml” is present in MyProj root folder.
·         Install Maven
·         Install MySQL Server
·         Create a working dir in Linux VM.
·         Copy source code into working directory – The “src” folder is present in MyProj root folder.
·         Run Maven commands to build the code – The file “pom.xml” is present in MyProj root folder.
·         Move generated WAR file to Tomcat webapps folder
·         Move database schema file to working directory – The file “projddl.sql” is present in MyProj root folder.
·         Run MySQL commands to create a database and import the schema file


To open Dockerfile for edit, type command "notepad Dockerfile&" as shown below:





 Save the Dockerfile. Then press Enter in docker terminal to go to next prompt.

Below is the root folder of MyProject which contain files which will be transferred to Docker VM.



4.      Build an image

 To create an image of application, type command "docker build -t my-proj" as shown below: Where "my-proj" is the name of image that we would like to be created.



This uses the file ./Dockerfile in the root of the repo to make a Docker Image. Once build is successful following screen is displayed.



5.      View Image - 
      When complete check the docker image has been created, within the command shell execute:
      $ docker images



6.      Run the docker image - 
      Within the command shell execute:
      " docker run -d -p 8080:8080 --name backend my-proj"

Here  -p 8080:8080 exposes the docker image's http internal port 8080 as port 8080 on the virtual machine the docker image is contained (and running) within.



7.      Verify the Tomcat is running. - 
      Enter url: http://192.168.99.100:8080 . This will open up the Tomcat Admin console page.



8.      Verify the Application is running - 
      Enter url: http://192.168.99.100:8080/MyProj. This will open up the default landing page in your application.
 


Front-end Application Docker

Frontend Application Environment:

a)      ReactJS based frontend Application

b)      NPM to build the dependency

c)      Gulp to package the application

d)      Nginx as Web Server

1.      Open Project root folder - 
      Go to the root of repository where frontend code is:




2.      Create a Dockerfile.



3.      Open Dockerfile - 
      To add docker command which will execute following tasks:
·         Take Node as the base image
·         Create a working dir in linux vm
·         Copy current directory source code into working directory
·         Run NPM command to download dependencies
·         Install Gulp
·         Install Nginx
·         Run Gulp command to build the project
·         Copy dist folder inside Nginx default html folder
·         Copy Nginx conf into Nginx directory inside linux vm







 Save the Dockerfile. Then press Enter to go to next prompt.





4.      Build Image - 
      To create an image of application, execute following command:
"docker build -t my-frontcode ."
 



 


This uses the file. /Dockerfile in the root of the repo to make a Docker Image.

5.      Verify Image - 
       When complete check the docker image has been created, within the command shell execute:
      $ "docker images"




6.      Run the docker image - 
      Within the command shell execute:
"docker run -p 80:80 --name frontend --link backend:backend my-frontcode"

Here -p 80:80 exposes the docker image's http internal port 80 as port 80 on the virtual machine the docker image is contained (and running) within.





7.      Test the frontend code in browser. Enter URL: http://192.168.99.100:80/


    

Friday, 18 September 2015

Developer’s Guide to Install Elasticsearch, Logstash and Kibana


In this I will talk about how to perform log analytics using Elasticsearch, Logstash and Kibana. To start with, we will see how to install these softwares on Windows.

Prerequisites:
  1. ·         elasticsearch-1.4.4
  2. ·         kibana-4.0.1-windows
  3. ·         logstash-1.5.0.rc2


Install Elasticsearch on Windows
Elasticsearch is a search engine tool/platform which allows us to save the documents to be search in certain format and provides APIs to do full text search capabilities. In the recent times, because of its features like Open Source, Scalability, ease of use, it has become very popular among developer community.

Install Elastic Search is every easy, here are the steps for the same

For this demo, we are going to use “elasticsearch-1.4.4”. Unzip and extract the content to the suitable directory.



This will start the Elasticsearch service at http://localhost:9200.

Install Logstash on Windows

Logstash is useful utility when it comes to playing with Logs. It gives you in built-in features to read from various file formats and perform some operations with it. One of the best feature it has is, you can read the logs in certain format (e.g. Apache Logs, SysLogs etc.) and put them into Elastic search.
Unzip the downloaded “logstash-1.5.0.rc2” in any folder.

To enable use of Logstash from any directory, add the path to system variable using environment variables.

>set LOGSTASH_HOME=D:\ELK\logstash-1.5.0.rc2
>set PATH=%PATH%;D:\ELK\logstash-1.5.0.rc2\bin

And that's it, logstash is ready to use



Install Kibana 4 on Windows

Kibana is a JavaScript library which allows us to create beautiful dashboard reports using elasticsearch data.

Here we are going to use “kibana-4.0.1-windows” as it is compatible with current release of elasticsearch that we are using.
Prior to Kibana 4, we need to have a web server running but with Kibana 4, we get it embeded.
Unzip the “kibana-4.0.1-windows” file at any location.

Kibana configuration is very easy, simply edit config/kibana.yml to add the elasticsearch url and done.

Open config/kibana.yml and update property elasticsearch_url: "http://localhost:9200".

To start Kibana, execute

 

A server would get started and you could see the GUI at http://localhost:5601/



Developers Guide to Analyse Logs Using Elasticsearch, Logstash and Kibana


In this I will talk about how to leverage the analysis of logs using Elasticsearch, Logstash and Kibana.


Prerequisites:
·         Installation of Elasticsearch, Logstash and Kibana is complete as per previous article here.


Load logs data into Elasticsearch using Logstash
We are going to write one Logstash configuration which reads the data from Apache Logs file.


Create a sample log file as below and save it in “D:” directory.

ApacheLogs.log
71.141.244.242 - kurt [18/May/2011:01:48:10 -0700] "GET /admin HTTP/1.1" 301 566 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"
134.39.72.245 - - [18/May/2011:12:40:18 -0700] "GET /favicon.ico HTTP/1.1" 200 1189 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; .NET4.0C; .NET4.0E)"


Next thing is to write logstash conf file as shown below.

Name this file as "logstash-apache.conf" and save this file in bin folder of Logstash installation folder.


input {
  file {
    type => "apache"
    path => [ "D:/ApacheLogs.log" ]
    start_position => "beginning"
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}


The above script first reads the apache logs file on given path, parses it into Apache logs format and add it to Elasticsearch.

Now let's execute this config using logstash and insert records into elasticsearch.



Now it will start reading from the file and also start inserting data into Elasticsearch.

Now go to Google Crome "Sense" extension and execute following command to view above log data.
















Now go to Kibana dashboard and configure indexes and Timestamp attribute as shown in the screenshot below.







Now simply go to Visualize tab to create graphs as you like and save them. Next is to view this saved Visualization in the Dashboard tab.