Performance testing,Stress Testing ,Load Testing

Performance Objectives

Performance objectives are captured in the requirements phase and early design phase of the application life cycle. All performance objectives, resource budget data, key usage scenarios, and so on, are captured as a part of the performance modeling process. The performance modeling artifact serves as important input to the performance testing process. In fact, performance testing is a part of the performance modeling process; you may update the model depending on the application life cycle phase in which you are executing the performance tests.

The performance objectives may include some or all of the following:

• Workload. If the application block is to be integrated with a server-based application, it will be subject to a certain load of concurrent and simultaneous users. The requirements may explicitly specify the number of concurrent users that should be supported by the application block for a particular operation. For example, the requirements for an application block may be 200 concurrent users for one usage scenario and 300 concurrent users for another usage scenario.
• Response time. If the application block is to be integrated with a server-based application, the response time objective is the time it takes to respond to a request for the peak targeted workload on the server. The response time can be measured in terms of Time to First Byte (TTFB) and Time to Last Byte (TTLB). The response time depends on the load that is on the server and the network bandwidth over which the client makes a request to the server. The response time is specified for different usage scenarios of the application block. For example, a write feature may have a response time of less than 4 seconds; whereas a read scenario may have a response time of less than 2 seconds for the peak load scenario.
• Throughput. Throughput is the number of requests that can be served by the application per unit time. A simple application that integrates the application block is supposed to process requests for the targeted workload within the response time goal. This goal can be translated as the number of requests that should be processed per unit time. For an ASP.NET Web application, you can measure this value by monitoring the ASP.NET\Request/secperformance counter. You can measure the throughput in other units that help you to effectively monitor the performance of the application block; for example, you can measure read operations per second and write operations per second.
• Resource utilization budget. The resource utilization cost is measured in terms of server resources, such as CPU, memory, disk I/O, and network I/O. The resource utilization budget is the amount of resources consumed by the application block at peak load levels. For example, the processor overhead of the application block should not be more than 10 percent.

Stress Testing

Stress testing an application block means subjecting it to load beyond the peak operating capacity and at the same time denying resources that are required to process the load. An example of stress testing is hosting the application that uses the application block on a server that already has a process utilization of more than 75 percent because of existing applications, and subjecting the application block to a concurrent load above the peak operating capacity.

The goal of stress testing is to evaluate how the application block responds under such extreme conditions. Stress testing helps to identify problems that occur only under high load conditions. Stress testing application blocks identifies problems such as memory leaks, resource contentions, and synchronization issues.

Stress testing uses the analysis from load testing. The test scenarios and the maximum operating capacities are obtained from load testing.

The stress testing approach can be broadly classified into two types: sustained testing and maximal testing. The difference is usually the time the stress test is scheduled to run for, because a sustained stress test usually has a longer execution time than a maximal stress test. In fact, stress testing can accomplish its goals by intensity or quantity. A maximal stress test tends to concentrate on intensity; in other words, it sets up much more intense situations than would otherwise be encountered but it attempts to do so in a relatively short period of time. For example, a maximal stress test may have 500 users concurrently initiating a very data-intensive search query. The intensity is much greater than a typical scenario. Conversely, a sustained stress load tends to concentrate on quantity because the goal is to run much more in terms of the number of users or functionality, or both, than would usually be encountered. So, for example, a sustained stress test would be to have 2000 users run an application designed for 1000 users.

Input

The following input is required for stress testing an application block

• Performance model (workload characteristics, performance objectives, key usage scenarios, resource budget allocations)
• Potential problematic scenarios from the performance model and load testing
• Peak load capacity from load testing

Stress Testing Steps

Stress testing includes the following steps:

1. Identify key scenarios. Identify test scenarios that are suspected to have potential bottlenecks or performance problems, using the results of the load-testing process.
2. Identify workload. Identify the workload to be applied to the scenarios identified earlier using the workload characteristics from the performance model, the results of the load testing, and the workload profile used in load testing.
3. Identify metrics. Identify the metrics to be collected when stress testing application blocks. The metrics are now identified to focus on potential performance problems that may be encountered during the testing process.
4. Create test cases. Create the test cases for the key scenarios identified in Step 1.
5. Simulate load. Use load-generating tools to simulate the load to stress test the application block as specified in the test case, and use the performance monitoring and measuring tools and the profilers to capture the metrics.
6. Analyze the results. Analyze the results from the perspective of diagnosing the potential bottlenecks and problems that occur only under continuous extreme load condition and report them in a proper format.

The next sections describe each of these steps.

Step 1: Identify Key Scenarios

Identify scenarios from the test cases used for load testing that may have a performance problem under high load conditions.

To stress test the application block, identify the test scenarios that are critical from the performance perspective. Such scenarios are usually resource-intensive or frequently used. These scenarios may include functionalities such as the following:

• Synchronizing access to particular code that can lead to resource contention and possible deadlocks
• Frequent object allocation in various scenarios, such as developing a custom caching solution, and creating unmanaged objects

For example, in the case of the CMAB, the test scenarios that include caching data and writing to a data store such as file are the potential scenarios that need to be stress tested for memory leaks and synchronization issues, respectively.

Step 2: Identify Workload

Identify the workload for each of the performance-critical scenarios. Choose a workload that stresses the application block sufficiently beyond the peak operating capacity.

You can capture the peak operating capacity for a particular profile from the load testing process and incrementally increase the load and observe the behavior at various load conditions. For example, in the case of the CMAB, if the peak operating capacity for a writing to a file scenario is 150 concurrent users, you can start the stress testing by incrementing the load with a delta of 50 or 100 users and analyze the application block's behavior.

Step 3: Identify Metrics

Identify the metrics that help you to analyze the bottlenecks and the metrics related to your performance objectives. When load testing, you may add a wide range of metrics (during the first or subsequent iterations) to detect any possible performance problems, but when stress testing, the metrics monitored are focused on a single problem. 

Load Testing

Load testing analyzes the behavior of the application block with workload varying from normal to peak load conditions. This allows you to verify that the application block is meeting the desired performance objectives.

Input

The following input is required toload test an application block:

• Performance model( workload characteristics, performance objectives, and resource budget allocations)
• Test plans

Load Testing Steps

Load testing involves six steps:

1. Identify key scenarios. Identify performance-critical scenarios for the application block.
2. Identify workload. Distribute the total load among the various usage scenarios identified in Step 1.
3. Identify metrics. Identify the metrics to be collected when executing load tests.
4. Create test cases. Create the test cases for load testing of the scenarios identified in Step 1.
5. Simulate load. Use the load-generating tools to simulate the load for each test case, and use the performance monitoring tools (and in some cases, the profilers) to capture the metrics.
6. Analyze the results. Analyze the data from the performance objectives as the benchmark. The analysis also identifies potential bottlenecks.

The next sections describe each of these steps.

Step 1: Identify Key Scenarios

Generally, you should start by identifying scenarios that can have a significant performance impact or that have explicit performance goals. In the case of application blocks, you should prepare a prioritized list of usage scenarios, and all of these scenarios should be tested.

In the case of the CMAB, the two major functionalities are reading and writing configuration data. The CMAB functionalities can be extended more based on various scenarios, such as whether caching is enabled or disabled, usage of different data stores, or different encryption providers, and so on. Therefore, the load-testing scenarios for the CMAB are the combinations of all the configuration options. The following are some of the scenarios for the CMAB:

• Read a declared configuration section from a file store with caching disabled and data encryption enabled.
• Write configuration data to a file store with encryption enabled.
• Read configuration data from a SQL store with caching and data encryption enabled.
• Write configuration data to a SQL store with data encryption enabled.
• Initialize the Configuration Manager for the first time when the Configuration Manager is performing user operations.

For the CMAB, performance degradation probability is high in a case where data must be written to a file store, because concurrent write operations are not supported on a file and the response time is expected to be greater in this case.

Step 2: Identify Workload

In this step, you identify the workload for each scenario or distribute the total workload among the scenarios. Workload allocation involves specifying the number of concurrent users that are involved in a particular scenario, the rate of requests, and the pattern of requests. You may have a workload defined for each usage scenario in terms of concurrent users (that is, all users firing requests at a given instant without any sleep time between requests). For example, the CMAB has a targeted workload of 200 concurrent users for a read operation on the SQL store with caching disabled and encryption enabled.

In most real-world scenarios, the application block may be performing parallel execution of multiple operations from different scenarios. You may therefore want to analyze how the application block performs with a particular workload profile that is a mix of various scenarios for a given load of simultaneous users (that is, all users have active connections, and all of them may not be firing requests at same time), with two consecutive requests separated by specific think time (that is, the time spent by the user between the two consecutive requests).

• Identify the maximum number of simultaneous users accessing each of the usage scenarios in isolation. This number is based on the performance objectives identified during performance modeling. For example, in the case of the CMAB, the expected load of users is 1,200 simultaneous users.
• Identify the expected mix of usage scenarios. In most real-world server–based applications, a mix of application block usage scenarios might be accessed by various users. You should identify each mix by naming each one as a unique profile. Identify the number of simultaneous users for each scenario and the pattern in which the users have to be distributed across test scenarios. Distribute the workload based on the requirements that the application blocks have been designed for. For example, the CMAB is optimized for situations where read operations outnumber write operations; it is not meant for online transaction processing (OLTP)–style applications. Therefore, the workload has to be distributed so only a small part of the workload is allocated for write operations. Group users together into user profiles based on the key scenarios they participate in.
  For example, in the case of the CMAB, for any given database store, there will be a read profile and a write profile. Each profile has its respective use case as the dominant one. A sample composition of a read profile for a SQL store is shown in Table 8.1. The table assumes that out of a total workload of 1,000 simultaneous users, 600 users are using the SQL store.