Software Performance and Load Testing / Application Optimization

Testing the Testing Strategy

Formerly the site of Gray Test Consulting, I am now using the site for sharing information about software testing best practices

Testing Blind Spots

After a recent presentation at work on testing fundamentals, I was asked a question about testing that I felt really deserved a full blog post. The question was “What suggestions do you have to identify blind spots during testing? The person added:

“The question comes from the challenge I sometimes face which is: the source code does not always tell the whole story; half of the “context” is set by the system configuration, the business transactional data and the business processes in SAP.  I would like to ensure that a change request does not impact inadvertently other areas (e.g. business or system data).”

This is an area of testing I have dealt with most of my career since I specialized in perf and load testing for so long, but often this is overlooked by agile teams or functional testing environments in complex business apps. Unfortunately, by not accounting for it in testing plans and/or working agreements, the testers are often at the mercy of unknown issues/changes/configs/etc. The answer is simple to explain, but it is often extremely difficult to implement. The first part of this post will explain what I consider to be the best way to approach this issue, and the second part will give you some tips on dealing with potential implementation impediments.

The Responsibilities and Boundaries

To know what a tester is responsible for when executing tests and analyzing results, the tester must know the boundaries where his or her team can affect changes. These boundaries include:

  1. Configurations:
    • Can the tester see and change values used to configure the application? If not, that’s a boundary.
    • Can the tester see and change values used to configure the environment? If not, that’s a boundary.
  2. Parts of the System:
    • If the tester’s team is writing the front-end web app and a set of API calls that go into other systems, chances are that the systems the API is calling are not available to the tester. That’s a boundary.
    • API calls that go to 3rd party vendors are often considered black-box and therefore outside of the control of the testers. We will call them boundaries.
  3. Data in the system: Dev teams are usually NOT responsible for the accuracy of the data that their software acts upon. This should be the responsibility of the business partners. A good example might be items in an eCommerce site. The developers had to create data relationships for Category, Quantity, Price, etc. of an item, but if the system has a category of “Kitchen Appliances” for an umbrella, or the person running inventory forgot to count three boxes in the warehouse, that is not a bug in the application.
  4. Test data in the system: Dev teams and the system owners will often have to coordinate about test data, especially in distributed systems. There are several ways to handle test data, but I will give you a couple of thoughts:
    • Test items: these items belong to a single test team. No one outside of the test team is allowed to touch them. That way, the “current state” of the items is always known. If you have multiple test teams for different areas of the application or system, each team gets a SEPARATE set of data.
    • Shared items: a set of test items is shared among teams, but only one team can use the data at any given time. The tean using the data must first “Check it out” (indicating that it is in use) and, when finished, must reset the data back to a known state (reset quantities, close out orders, etc.
  5. Releases: The proper code must be released to the proper environment, and ALL required dependencies must also be handled.

Once the boundaries are defined, the team can now negotiate with the owners of the other areas and the product team to define how testing will proceed.

Implementation

Let’s look at an example system. We will use an example where four apps are all hitting the same shared SAP and CRM system, all using shared auth. Let’s assume the developers and testers only have access to application 1. How do they handle all the items listed above?

Configuration

Have a known AND RELIABLE change control system in place that makes it easy for dev teams to see (and understand) the changes and their impact on the system overall. Ensure that changes to the system are only rolled out at known times so testers can plan accordingly.

Parts of the System

  1. Ensure the team in charge of accounts and security setup test users appropriately.
  2. Have the owners of the system behind the boundary create a shim (or stub, or mock, or etc.) that mimics the input and the expected output.
  3. If the owners won’t do that, then the dev/test team can create their own (not an easy task).

Data in the System

If the data is not specifically needed to complete a test, then the data should be considered as Not our responsibility and therefore does not need to be considered. If, however, it is needed for test completion, then it should be treated as test data. The next section explains handling test data.

Test Data in the System

Here I am showing four sets of test data, as well as four pieces of system data that are needed for the test data to be used properly. In this case, you have to figure out if this data is going to be shared or will belong to the team.

Test Data 1 is assumed to “belong” to the dev team and therefore does not constitute a boundary. Datum 2 through 4 are shared, so coordination will have to occur.

Releases

When an application is self-contained, it is easier to control the code that exists in various parts of the application. But when things are shared, there is a need to track the versions and/or the interfaces that are exposed in dev, test and prod environments. Below we can see that team 1 cannot test fully, but teams 2-4 can. In this case, team 1 would rely on shims or mocks to test and would align the full testing by coordinating with the broader teams the same way we do in the “Parts of the System” section.

Summary

Here is the final complete diagram, showing all of the potential boundaries that the development team may encounter.

By defining every boundary, and then discussing each with the team(s) that own the items on the other side of the boundaries, you can start to negotiate appropriate ways of dealing with the issues. As you negotiate this, remember that it is important to account for the amount of effort involved in each decision. If your team has to take on more responsibility, then your velocity will slow down. Make sure the product owners are aware of these trade-offs and are willing to accept them, or to find ways to mitigate them.