Jmeter Integration with InfluxDB and Grafana for real time monitoring

JMeter:

JMeter is the most popular and easy to use open source load testing tool.It is designed to load test functional behavior and measure performance. You can use JMeter to analyze and measure the performance of web application or a variety of services.

Need of running JMeter in Non-GUI mode:

In order to test actual performance of the application , it is always recommended to run the JMeter test plan in Non-GUI mode. As running tests in GUI mode can consume a lot of resources and potentially produce unreliable load test results.

How to achieve real time monitoring in Non-GUI mode run:

Running tests in Non-GUI mode means we have to wait until the test is completed to see the performance results. While this might be okay for short test run but for longer running soak tests (3–4 hours performance run) you might want to examine the results while the test is running.

To achieve this there are multiple options but in this article we will see how this can be achieved by integrating JMeter with InfluxDB and Grafana.This allows you to create real-time graphs based on time series data.

Advantages:

• Live real-time results
• Visualized key metrics help understand what is occurring during the test
• Store test results for longer duration
• Ability to compare 2 or more performance tests
• Customize performance dashboards in any way you want

Overview:

InfluxDB is a time-series database built for high-performance handling of time-stamped data. JMeter provides a backend listener InfluxdbBackendListenerClient to write the data to InfluxDB.

Grafana is an open-source metric analytics and visualization suite. It is most commonly used for visualizing time series data for infrastructure and application analytics.

Data Flow:

JMeter write the realtime test results into Influx database using backend listener and Grafana will read the data from influxdb and display it on it’s dashboard.

Data flow from JMeter to Grafana

Below are four steps to integrate Jmeter with Grafana and Influxdb:

1. Download and setup Influxdb
2. Setup influxdb backend listener in JMeter
3. Download and Setup Grafana
4. Configure Grafana dashboard
5. Run Jmeter Test and monitor results

1. Download and setup Influxdb

Installing influxdb through brew is very simple. You just need to run these two commands If brew is installed on your machine.

Note: If installation fails then enter below commands for the access. You should change the ownership of these directories to your user.

sudo chown -R $(whoami) /usr/local/lib/pkgconfig

And make sure that your user has write permission.

chmod u+w /usr/local/lib/pkgconfig

At the end of installation you should see this:

Start InfluxDB

There are two ways to start influx db:

• Using brew command: brew services start influxdb
• If you don’t want a background service : influxd -config /usr/local/etc/influxdb.conf

To verify that InfluxDB is up and running, all you need to do is to open a terminal window and run this command:

If the installation was completed successfully and the database is up and running, you will see an InfluxDB command line interface. This can be used for interacting with the database.

Setup Database:

We will need to create a database to store our Jmeter Results. Open InfluxDB command line interface by executing command “influx” in terminal.

• Run ‘SHOW DATABASES’ command, you can see the list of all existing InfluxDB dabasebases. If you have just installed InfluxDB you should see only one ‘_internal’ database, which is used for keeping different stats about database itself:

• Run the command: “CREATE DATABASE jmeter” — here “jmeter” is the database name

• Run ‘SHOW DATABASES’ command again to see whether the “jmeter” database has been created or not

Configue InfluxDB

Once we have created a database for our metrics, we need to make a few changes to the InfluxDB configuration. Use below command to open your influxdb.conf file

In this configuration file you need to find, uncomment and edit the ‘[[graphite]]’ category appropriately:

After that you need to restart InfluxDB by applying an edited configuration:

To confirm the configuration was applied successfully, you should find this line during the InfluxDB startup:

We have completed the first step to install and configure influx DB. Moving to the next step to setup influx db backend listener in jmeter.

2. Setup Influxdb backend listener in JMeter

I will be using one of my existing JMeter test project and we will see how to configure the backend listener to write metrics to InfluxDB.

• Add a new backend listener to the thread group. Right Click on a Thread Group -> Add -> Listener -> Backend Listner

• Change its name to InfluxDB Backend Listener
• In the Backend Listener Implementation dropdown select org.apache.jmeter.visualizers.backend.influxdb.InfluxdbBackendListenerClient
• Configure backend listener as shown below in image

• InfluxdbMetricsSender — this is the class for sending metrics to InfluxDB. We will keep it as default.
• InfluxdbUrl — this is the URL of InfluxDB is in the following format: http://%5Binfluxdb_host%5D:%5Binfluxdb_port%5D/write?db=%5Bdatabase_name%5D. As we have created the “jmeter” database and we are running it on a local machine with a default port, then, in our case, the URL will be http://localhost:8086/write?db=jmeter.
• application — we can store metrics from different applications under one database.
• measurement — the name of the measurement that will be stored in InfluxDB. Use default ‘jmeter’.
• summaryOnly — if you want to keep summary results in the database and do not want to collect all the detailed metrics during test execution set to ‘true’. We will set this to false
• samplersRegexp — use the regexp expression to filter specific metrics which you want to store in a database. Use the default “.*” value if you want to store all your metrics.
• percentiles — metrics percentiles that should be sent to the database.
• testTitle — give your test a name.
• eventTags — a list of tags which will be stored in the ‘events’ measurement of InfluxDB.

In case the influx db access is user restricted then username and password need to be added in URL. http://localhost:8086/write?db=jmeter&u=username&p=password

Here we are done with our setup of Backend Listener in Jmeter. Now lets move to our next setup of Grafana

3. Download and Setup Grafana

Installing grafana through brew is very simple. You just need to run the below command If brew is installed on your machine.

Start Grafana

There are two ways to start grafana:

• Grafana can be started with the below brew command

• Or can be started using the below command

grafana-server — config=/usr/local/etc/grafana/grafana.ini — homepath /usr/local/share/grafana — packaging=brew cfg:default.paths.logs=/usr/local/var/log/grafana cfg:default.paths.data=/usr/local/var/lib/grafana cfg:default.paths.plugins=/usr/local/var/lib/grafana/plugins

After that, Grafana should be available on http://localhost:3000. Use ‘admin’ as a default username and password to log in.

4. Configure Grafana dashboard

In previous step we are able to start grafana in localhost. Once logged in you will be presented with the Home Dashboard. From here you can create data sources, manage users, create report dashboards etc.

Add Data Source

The first thing we need to do is create a data source to pull data from our InfluxDB. Click Add Data Source and choose InfluxDB as the data source type. Specify the url: http://localhost:8086 and the name of the influxdb database we created earlier: jmeter. Configure minInterval time as 5s. You can also configure other settings for authentication. Click Save & Test hopefully you should see the Data source is working.

Create DashBoard

Instead of creating a fresh Dashboard in Grafana , we will use existing dashboard template available on grafana.com.

In Grafana , click on the “+” icon in top left corner and select Import Dashboard enter the dashboard ID we copied 5496 and click Load. On the import dashboard screen please configure with following values:

• Name — specifiy the dashboard name
• Folder — Grafana allows you to organize dashboards into folders.
• DB name — select the InfluxDB data source we created named as “InfluxDB”
• Measurement name — we will set this to ‘jmeter’ not it is the same measurement we defined when we setup the JMeter Listener and is the table in our InfluxDB database.
• Backend send interval — this should match the JMeter property backend_influxdb.send_interval which default value is 5 seconds.

The Dashboard will look like as follows:

We are done with our Grafana setup. Now its time to run the actual jmeter test and monitor results in grafana.

5. Run Jmeter Test and monitor results

Set below values in thread group to run jmeter test

Number of threads: 3 , Ramp up period: 1 , Loop Count : infinite

Now Run the Test.

We should start to see JMeter metrics appearing in your grafana dashboard after 5 seconds and automatically updating every 5 seconds until our load test completes.

To manage the time ranges of the data being visualized, you can choose your desired options provided in Grafana dashboard(Find it in top right Corner)

This completes our integration setup for Jmeter with InfluxDB and Grafana to view Real time Metrics Results.

I hope you will find this article helpful.

Thanks for reading!!