We have been assigned the work to find the best Load Testing tool from Free Tools domain. We also need to evaluate it against some best paid tools. Requirements were:
- Tool should be free and preferably open source
- It should be very stable and mature
- It should be able to generate any amount of load without any software limit. Hardware can be increased as per requirements.
- It should be able to generate the concurrent load in concurrent running threads
- It should support running JUnit test cases out of the box
- It should support recording of Test Cases for UI, which can be played back later with load
- Generated reports should be extensive in information and preferably in format also
- Tool should be in active development
- Out of the box support for various other interfaces will be a plus
- Apache JMeter - Free and Open Source
- Grinder - Free and Open Source
- HP Load Runner - Paid, just to have a good comparison
- Apache JMeter (http://jmeter.apache.org/) - It is an open source free tool, backed by Apache Foundation and hence should be quite stable for the features mentioned on website. Here are few good things we have found:
- It is good for running load test cases.
- Existing JUnit test cases can be fed to it. Procedure is very simple, where you just need to put your jar file having JUnit test cases in a specific folder. JMeter picks all these test cases automatically and show you the list of test case classes and their methods to pick from.
- For UI test cases, we need to create the test cases first. For that, there are two approaches - first to capture the UI events using event recorder of JMeter and other is to write the script manually using JMeter scripting.
- It is having its own expression language to program the logical flow
- It is having good recommendations and feedback and is one of the best tool in open source domain.
- It can use distributed agents and then data can be collected back from these agents to one desktop to see the reports
- It can run both in GUI and Non-GUI mode.
- It is simple to use.
- Documentation is good
- Report is not that extensive as other commercial tools are providing like Load Runner, but, it is improving with time. Moreover, it is open so you can integrate the outcome with any other tool.
- It is built in java and hence is capable to run on any platform.
- There is quite good online community support.
- Grinder (http://grinder.sourceforge.net/) - It is another open source and is rated quite good as we have for JMeter. It is having same kind of featrure set with very good load generation facilities. However, set back is that it understand the Jython script as test scripting language. So developers need to learn Jython also. Moreover being backed by Apache, JMeter seems to be more promising and having good feedback also. Second place certainly goes to Grinder. However few good features are:
- Supports very powerful and fast development scripting with Jython
- UI is quite simple and shows much of data
- Script can also access Java library and hence you can reuse existing set of methods
- It can also capture the UI scripts to generate the test cases
- It can use any feature of java written code
- If you are comfortable with Jython, it could be in hard competition with JMeter.
- HP Load Runner - It is a commercial tool by HP. It is a good product in commercial product domain having good UI, a good UI test generation facility by recording the test cases. Report functionality is also quite extensive. There are few drawbacks we are listing below. Consider, these drawbacks are evaluated from a setup point of view having small agile teams, average budget, and open source driven development environment.
- It may behave differently on different platform. Like we read about weak monitoring on Solaris environment.
- Pricing is very high
- Licenses are bound to amount of load also. So even after having license, you are still bound for the amount of load and that actually kills the true spirit of testing many times.
- http://blackanvil.blogspot.com/2006/06/shootout-load-runner-vs-grinder-vs.html
- http://shmuels.blogspot.com/2012/01/why-apache-jmeter-is-better-than-hp.html
- http://rrees.wordpress.com/2009/04/09/jmeter-or-the-grinder-so-which-one-is-better-like/
Above exploration finally derive us towards JMeter, which suits to our requirements best.
For JMeter, following links can help:
- Download link is - http://jmeter.apache.org/download_jmeter.cgi
- User Manual - http://jmeter.apache.org/usermanual/index.html
- For JUnit test Cases - http://jmeter.apache.org/usermanual/component_reference.html#JUnit_Request
- A good blog for basics - http://jmeter-expert.blogspot.com/
7 comments:
Here is another good tutorial about JMeter basics - http://javaboutique.internet.com/tutorials/JMeter/
Glad that you've decided to use JMeter.
Can you share what kind of use you do with the JUnit sampler? What is the benefit of running JUnit tests from JMeter?
We have use JUnit sampler to run the existing JUnit test cases using JUnit Request to thread group. Anyhow this is the requirement to run existing JUnit test cases from JMeter. Benefits, we were able to simulate load by calling test cases multiple times, from parallel threads also. Definitely test cases need to be designed accordingly.
Refer to http://jmeter.apache.org/usermanual/junitsampler_tutorial.pdf for more about JUnit sampler.
I can't believe no one mentioned Tsung yet. http://tsung.erlang-projects.org/
I would add 2 more to the list:
Tsung
Gatling
For main features of Grinder, Gatling, Tsung and JMeter highlighted and compared refer to Open Source Load Testing Tools: Which One Should You Use? post.
I wanted to add another one to the list:
WebLoad Which is HP LoadRunner competitor
Post a Comment