How to do performance testing in SoapUI
When it comes to testing the performance of your APIs, various tools are available in the market. Among these tools, SoapUI stands out as a powerful and versatile choice for API testing. This article will delve into API performance testing with SoapUI, explicitly focusing on load and stress testing. We will explore the essential concepts, methodologies, and best practices for ensuring your APIs can handle heavy loads and stressful conditions. Additionally, we will discuss how Ready API (SoapUI’s paid version) can provide some distinct advantages in this context.
Before proceeding with this article, ensure you have installed the SoapUI application successfully. If you still need to install it, please refer to our article on SoapUI installation. Additionally, to enhance your knowledge of SoapUI, refer to our article on SoapUI tips and tricks.
Table of contents
- Understanding API performance testing
- Load testing in SoapUI
- Stress testing in SoapUI
- Best Practices performance testing in SoapUI
- SoapUI vs. JMeter in performance testing
- Common errors
- Conclusion
Understanding API performance testing
During the software development lifecycle, API performance testing is a crucial step that identifies potential bottlenecks and performance issues before they affect users. It involves evaluating an API’s speed, scalability, and stability under different conditions.
Two common types of API performance testing are load testing and stress testing:
What is load testing?
The most important essential part of performance testing, which is used to check the performance of an application by applying some load like less than or equal to the desired load, is known as load testing. And the load is a quantity, which means it only focuses on the number of users.
Key objectives of load testing include:
- Performance Measurement: Gathering response times, throughput, and resource utilization metrics under various loads.
- Identifying Bottlenecks: Discovering performance bottlenecks like slow database queries or inefficient code.
- Scalability Assessment: Determining if the API can scale horizontally or vertically to accommodate increased loads.
- Capacity Planning: Providing insights into the server and infrastructure requirements to handle expected loads.
What is stress testing?
Stress testing checks an application’s behavior by applying a load greater than the desired load. Since it is non-functional testing, we use this testing when the application is functionally stable.
Key objectives of stress testing include:
- Breaking Point Identification: Determining the maximum load an API can handle before it fails or becomes unresponsive.
- Fault Tolerance Assessment: Evaluating how well the API handles unexpected conditions and errors, such as database outages or network failures.
- Resource Exhaustion Testing: Assessing the API’s behavior when depleted system resources like CPU, memory, or disk space.
- Recovery Testing: Verifying if the API can gracefully recover from failures and resume regular operation.
Load testing in SoapUI
Load testing in SoapUI involves simulating concurrent users and requests to assess an API’s performance under load.
Here are the steps to execute load testing with SoapUI:
Create a test case
Open SoapUI and create a new project or open an existing one.
Then, you must create a new test suite and test case within the project that represents the API functionality you want to test under load.
Configure load test
In the test case, right-click and select New Load Test to create a load test configuration.
In the Load Test Configuration window, specify the following parameters:
Threads: Specify the number of threads or concurrent users you want to simulate.
Strategy: Choose the strategy that best fits your testing goals. The options are simple: burst, thread, and variance provide different load patterns.
Limit: Define the limit for the load test, which can be based on seconds, total runs, or runs per thread, depending on your testing requirements.
Add assertions
Within the load test configuration, you can also set assertions to validate the responses generated during the load test. Then, click the Add Assertion button to configure assertion rules based on your testing objectives. Assertions help you to ensure that the API responses meet specific criteria.
Execute load test
You can run the load test by clicking the Run button in the Load Test Configuration window.
SoapUI will simulate the specified number of virtual users making API requests concurrently. It also collects performance metrics such as response times, throughput, and error rates during the test.
Analyze results
After completing the load test, you can analyze the results using the Load Test Log and Load Test Statistics. SoapUI provides detailed reports and graphs to help you visualize the data and identify performance bottlenecks.
Iterate and optimize
Based on the test results, make necessary optimizations to your API or infrastructure. Kindly repeat the load testing process to ensure improvements for your future tests.
Stress testing in SoapUI
Stress testing in SoapUI involves pushing the API to its limits to uncover vulnerabilities and weaknesses.
Here are the steps to perform stress testing in SoapUI:
Create a test case
Open SoapUI and create a new project or open an existing one.
Then, you must create a new test suite and test case within the project that represents the API functionality you want to stress test.
Configure stress test
Right-click and select Add Load Test to create a stress test configuration in the test case.
In the Load Test Configuration window, configure the following parameters based on your version of SoapUI:
Threads:
Specify the maximum number of threads or concurrent users you want to use to apply stress. For example, if your application typically handles 100 concurrent users, set this to 150 to stress it beyond normal capacity.
Strategy:
Choose the strategy that best fits your stress-testing goals such as simple, burst, thread, and variance, which provide different stress patterns. For example:
- Simple: This strategy maintains a consistent load throughout the test. If you’re testing a chat application, use this to simulate a continuous stream of messages.
- Burst: The Burst strategy simulates sudden spikes in traffic. If your e-commerce site experiences flash sales, this strategy helps you simulate those traffic surges.
- Thread: Thread strategy increases load incrementally. It’s useful when you want to gradually scale up traffic, like during a gradual increase in users over time.
- Variance: Variance strategy introduces randomness to the load. It can be helpful for simulating real-world scenarios where user behavior is unpredictable.
Limit:
Set the limit to its maximum value for the stress test based on seconds, total runs, or runs per thread, depending on your testing requirements. For example:
- Seconds: If your application needs to sustain high traffic for 15 minutes, you might set the limit to 900 seconds (15 minutes).
- Total Runs: If you want to execute a fixed number of total test runs, set this to 1000 to simulate 1000 transactions.
- Runs per Thread: If you’re interested in measuring the performance of each thread individually, you could set this to 100, meaning each thread would execute 100 transactions.
Execute stress test
To run the stress test, click the Run button in the Load Test Configuration window. After that, SoapUI will subject the API to a high and stressful load, testing its limits.
Monitor and observe
It is essential to closely monitor the behavior of the API, system resources, and any potential errors during a stress test. Paying attention to how the API handles extreme load conditions is crucial. Once the stress test is completed, it is essential to analyze the results to identify any issues or weaknesses. Evaluating the API’s performance under stress is necessary to determine if it meets the desired criteria.
Best practices for performance testing in SoapUI
Regardless of whether you are using SoapUI or any other tool for API performance testing, there are some best practices to follow to ensure adequate testing:
Realistic test data
Use realistic data inputs that accurately represent usage patterns to simulate real-world scenarios.
Incremental testing
Start with low loads and gradually increase them to identify performance thresholds and bottlenecks.
Resource monitoring
Continuously monitor system resources during tests to detect resource exhaustion and its impact on API performance.
Error handling
Create test scenarios that include error cases to evaluate how well the API handles unexpected situations.
Testing in isolation
Isolate the API being tested from external dependencies and third-party services to focus solely on its performance.
Test environment replication
Ensure the test environment closely replicates the production environment regarding hardware, network conditions, and configurations.
Test data cleanup
Implement data cleanup procedures to maintain a consistent and reliable environment, especially when running stress tests with large data sets.
Regular testing
Perform load and stress testing regularly, especially after code changes, updates, or infrastructure modifications.
Collaboration
Involve developers, testers, and operations teams in performance testing to address issues efficiently and collaboratively.
Documentation
Maintain comprehensive documentation of test scenarios, configurations, and results for future reference.
SoapUI vs. JMeter in performance testing
Regarding performance testing, both SoapUI and Apache JMeter are popular choices among software testers and developers. Each tool has its features, advantages, and use cases. In this section, we will compare performance testing in SoapUI and JMeter, highlighting the strengths and weaknesses of each tool.
SoapUI for performance testing
SoapUI is primarily designed for functional testing of APIs but provides limited support for performance testing, especially in its Pro version.
Here is an overview of performance testing with SoapUI:
Strengths
- API-Centric Testing: SoapUI is well-suited for API performance testing, allowing you to create and execute test scenarios for REST, SOAP, and other web services. Its intuitive interface makes it easy to set up API tests.
- Integration with Functional Testing: If your testing workflow involves functional and performance testing, SoapUI can seamlessly transition from functional to performance testing, leveraging existing test cases.
- Load Testing Capabilities (Pro): Ready API offers basic load testing capabilities, including simulating concurrent users and requests. While it lacks some advanced features of dedicated performance testing tools like JMeter, it can still be effective for smaller-scale performance tests.
- Scripting Support (Pro): Ready API allows you to use Groovy scripting, which can be handy for creating custom test scenarios and data manipulation during performance testing.
Limitations
- Limited Load Testing Features: SoapUI’s load testing capabilities are relatively basic compared to dedicated performance testing tools like JMeter. It may not be suitable for complex and large-scale performance testing scenarios.
- Reporting (Pro): Ready API offers enhanced reporting compared to the free version, but it may provide different reporting and analysis capabilities than specialized performance testing tools.
- Resource Intensive (Pro): Running large-scale performance tests in Ready API can be resource-intensive, which may require substantial hardware resources or cloud-based testing environments.
JMeter for performance testing
Apache JMeter is a robust and open-source tool for load and performance testing. It has a strong focus on simulating high loads and analyzing system performance.
Here is an overview of performance testing with JMeter:
Strengths
- Robust Load Testing: JMeter excels in load testing and can simulate many users and requests, making it suitable for stress testing and performance benchmarking.
- Extensive Protocols Support: JMeter supports a wide range of protocols, including HTTP, HTTPS, FTP, JDBC, SOAP, and more, making it versatile for testing various types of applications.
- Rich Reporting and Analysis: JMeter provides comprehensive reporting and analysis features, including graphical representations of test results, which are crucial for identifying performance bottlenecks.
- Large Community and Plugins: JMeter has a large user community, and numerous plugins are available to extend its functionality, making it adaptable to different testing scenarios.
- Scripting Capabilities: JMeter allows you to write custom scripts using various scripting languages, providing flexibility in test scenario creation and data manipulation.
Limitations
- Steep Learning Curve: JMeter’s extensive capabilities can guide a steep learning curve for new users. Setting up complex test scenarios may require a deeper understanding of the tool.
- Primarily Load Testing: While JMeter can be used for functional testing, its primary strength lies in load and performance testing. It may not be as user-friendly for functional testing as SoapUI.
- Resource Intensive: Running high-load tests in JMeter can consume significant system resources. Efficient test distribution and monitoring are essential for large-scale trials.
Common performance testing errors in SoapUI
Performance testing with SoapUI can be a powerful way to ensure the reliability and scalability of your APIs. However, like any testing process, it is not without its challenges. In this section, we will explore some common errors that testers and developers may encounter when performing performance testing in SoapUI and how to address them effectively.
Insufficient test data
One of the most common errors in performance testing is not using realistic test data. Using synthetic or inadequate test data can lead to unrealistic test scenarios that do not reflect real-world conditions. To address this issue:
Solution: Use representative and diverse test data that mimic user behavior and variability. Consider data-driven testing to simulate various scenarios and inputs.
Incomplete test coverage
Another error is conducting performance tests with incomplete test coverage. This can result in missing potential bottlenecks or issues in your API.
Solution: Ensure your test scenarios cover all critical API endpoints and functionalities. Prioritize test cases based on their impact on overall system performance.
Overlooking scalability
Some testers focus solely on load testing without considering the system’s scalability. Load testing evaluates how the system performs under expected loads but does not necessarily assess its ability to scale to handle higher loads.
Solution: Incorporate scalability testing into your performance testing strategy to determine how well the system can adapt and grow to accommodate increased loads.
Neglecting resource monitoring
Monitoring system resources (CPU, memory, disk space, etc.) is essential for identifying resource-related bottlenecks during performance testing. Neglecting this aspect can lead to incomplete test results.
Solution: Implement resource monitoring tools and techniques to capture resource utilization metrics during performance tests. Analyze these metrics to pinpoint resource-related issues.
Lack of test environment replication
Running performance tests in a test environment that significantly differs from the production environment can lead to inaccurate results. This error may not reveal actual production performance issues.
Solution: Ensure the test environment closely replicates the production environment regarding hardware, network conditions, configurations, and data.
Final thoughts on API performance testing with SoapUI
API performance testing is a critical step in ensuring the reliability and scalability of your applications. SoapUI, whether in its free version or Pro edition, provides a powerful platform for conducting load and stress testing. While the free version offers fundamental load testing capabilities, Ready API has advanced features, scripting support, and cloud integration, making it especially valuable for stress testing.
In conclusion, effective API performance testing with SoapUI involves careful planning, realistic test scenarios, and a thorough analysis of results by following best practices and leveraging the appropriate SoapUI version for your needs.
Follow our blog
Be the first to know when we publish new content.
Related articles
- SoapUI tips & tricks
- API mocking with SoapUI
- How to do SoapUI and Postman integration
- SoapUI security testing
- JMeter load testing
- Postman load testing