Responsive Testing

The Realmac Software QA department is always looking to improve how we test our apps, and constantly aiming to raise the quality of apps we ship.

I’ve previously talked about writing test plans that include test cases for an app’s every known bug or feature – this was to ensure that the app is thoroughly tested. However, I’ll be the first to admit that there’s a problem with this approach; the longer the test plan the longer the testing turnaround time will be. Testing the quality of an app is important in its development but, referring back to Dan’s iron triangle, so too is time!

Due to the scale and evolving nature of our apps, it’s no longer practical to form such in-depth test plans in the same manner that we’ve done before. How does such an incredibly detailed test plan fair when large parts of the app are removed, changed or added at short notice? More likely than not, code changes meant a large number of test cases would need updating with some, possibly, becoming altogether irrelevant. Updating such detailed test plans can quickly become a lengthy and laborious process, far from the agile ideal.

I’m going to use Analog Camera as an example here – the app consists of just a couple of main views; a Camera view with asset selection, a crop view, and a share view that includes the filter grid. Despite the simplicity of Analog Camera’s user interface, our master test plan for the 1.0 release was incredibly detailed, actually reaching more than 5000 words in length! As new test cases arose during development they were added to the test plan, no matter how vital they were to performance, UX or overall product quality; and there lies the problem.

As Dan hinted at in his recent post, the entire user flow was re-worked many times. For each time that a view was added, altered or removed from the app, considerable time was spent maintaining the ever-growing test plan. Prior to shipping 1.0, it’s also fair to say that these kinds of changes happened a lot! That can add up to a fair amount of time spent focussing on editing test plans rather than actually testing or designing better tests in the first place.

Though the end results show that we produced a “rock-solid” app that got plenty of positive reviews, our ways of designing and planning tests is evolving into a quicker process. I’m also placing a greater emphasis on improving the quality of tests carried out rather than increasing them in number.

One of the most important lessons learned in QA is that not only is it impossible to fix every bug but it’s impossible to test everything too. Looking to be more agile from the start of development, testing should ideally be involved as early as the design stage, before any code has been written. We’ve seen many bugs arise during testing in the later stages of an app’s development that could’ve been accounted for had the initial designs been subjected to testing. Experimenting with setting short time limits on test designs early in development can also help to focus on what’s really high priority.

There’s a great little post here on ‘10 minute’ test plans by James Whittaker, formerly of Google, where he talks about Google’s testers setting out to find the 3 things that were most important when designing tests (I’ve paraphrased them below):

• Attributes: the adverbs and adjectives that describe the concepts testing is meant to ensure; such as fast, usable, secure and accessible.
• Components: the nouns that define the major code chunks that comprise the product. These are classes, module names and features of the app.
• Capabilities: the verbs that describe user actions, gestures and behaviours.

Using an initial limit of ~10 minutes to create a mind-map, they called out the key attributes and components of an app or feature, spending another 20 minutes or so to add a certain level of detail to a features’ capabilities as well as concise user stories helps prevent the overall test plan from spiralling out of relevance and effectiveness.

As well as being shorter and more concise, this method leads to a quicker and more straightforward way to form a completely new test plan from scratch. The ‘one-test-plan-to-rule-them-all’ approach could be dropped to create these new, concise, test plans. If a feature is dropped, so too can the test plan related to that feature meaning there’s no great legacy of tests needing to be maintained.

Using the methods mentioned above, the areas of an app that are most vital to test become clearer at a much earlier stage, as do tests that can be easily automated to save time in future. They also highlight which other checks can be left to come out in the soak (or beta testing). This all helps QA to respond faster with more effective tests as our apps continue to grow.

Touching a little on beta testing and being responsive, here’s a little Pro Tip: stage your beta releases while adding new beta testers per release. It’ll likely produce more valuable feedback than building a large pool to begin with then spamming the same group of users with new builds. We’ve seen that some incredibly detailed feedback tends to come from new beta testers. This gives us really valuable and up to date information on how new features or changes within an app are being received. So remember: more testers does not equal more feedback; whereas new beta testers does mean new feedback, more often than not!

That’s it for this post, though I’ll be sure to follow up with another QA-related article soon. If you’ve got any thoughts, suggestions or questions on anything I’ve mentioned above, please feel free to get in touch on Twitter.