Data Driven Decisions in Testing

Leveraging the data available to us can give us a whole new perspective on the risks our business and team need to mitigate. It can take our testing approach to a whole new level.

Imagine your team is in the middle of developing an important feature. The feature is worth hundreds of thousands of euros to the company and there’s a hard deadline of when this has to go out to customers. The team is working against the clock with less than a week to deliver. Suddenly a bug comes in. Your product is completely unusable for a customer. The display is totally broken. What do you do? How can you balance the risk of the time needed to investigate this issue with the hard deadline for the feature? Once your team gets that feature out the door, how do they explore how users are interacting with it? How do they decide what part of the feature to build out (or not build out) next? How do they action production bug fixes before the customer finishes giving all the details to customer support? The answer to all of these questions is simple. Data. We released a feature that had taken the team weeks to work on. We included monitoring for the work. After 3 months, we reviewed the data. It revealed that only 6 users from our user base of 500,000 people had used the feature. The team could confidently say that it was a bad idea to invest any more time into developing it further. When my team asked me how risky it was to not action a bug fix, I could confidently say it was incredibly low. I could see from our usage statistics that we hadn’t had a single user access our application in the past two years with the device mentioned in the bug. When we’re testing software, we use a wide variety of tools and skills to drive our decision-making. Often, we forget, don’t know how to use, or don’t have access to the right data to take our decision-making to the next level. Leveraging the data available to us can give us a whole new perspective on the risks our business and team need to mitigate. Of course, data has its flaws too but that’s where a testing or problem-solving mindset comes in. Combining a testing mindset with access to data allows us to unleash the potential power of both. Applying all of the skills we use when testing applications to analyzing the data we have available to us can help us to decide when we should and shouldn’t rely on the data. Even if you’re not yet convinced of the power of data, this talk is for you. You’ll see how data took my decision-making process to the next level. You'll get insights into how leveraging the data available can bring whole new perspectives on the risk your business and team need to mitigate.