What to Test

In my previous post on responsive testing I looked at the need to ensure that testing doesn’t become a process that holds up the release of an app. In this post, I’ll talk more broadly about how we make decisions on what to test before shipping common kinds of updates to our apps.

N.B. Before reading on, it’s important to remember that the role of QA isn’t to just break apps but to be able to assure that they meet an expected level of quality. Here at Realmac Software, our marker is set by our one simple goal; design and build the best apps in the world.

We like to pour over the smallest details to achieve this and the QA team is no different here; I for one love to file a ticket for even the smallest typo, performance glitch or other such annoyances. However, we also need to keep the entire app, the bigger picture, in mind when testing. Focussing mainly on breaking an app would make it all too easy to lose sight of the overall character, feeling or purpose behind what we’re building. With that, I’m going to talk about being realistic with knowing what test or look out for when shipping any particular update.

Depending on the project, we aim to work towards milestones with an agreed set of outcomes that lists bugs to fix or new features to design and build. The level of work involved between different milestones can vary greatly and, as such, the time that’s spent testing is usually directly proportional to the scope of changes per release.

For example, the introduction of a new feature may be the result of reworking many areas of an app. Existing user-flows may have been updated or all-together redesigned to accommodate a particular feature, highlighting the need to be responsive when testing. More often than not, feature releases are given a lot more testing-focus than smaller bug fix releases. Taking the recent introduction of (amongst other features) iCloud support to Ember as an example, the following list touches upon just a few different things to consider testing at a user level:

• Data:
  • Migration of data between local store and iCloud
  • Online/Offline conflict merging of images and related meta and annotations
  • Network condition testing, Documents & Data permissions and errors
  • Upgrading from previous versions
• UX:
  • Updated first-run user flow
  • New menu items, keyboard shortcuts, and share services
• UI:
  • Image assets for (non)HiDPI display modes
  • Localised strings for all new or updated features, alerts, and errors

Even after ensuring a new feature functions as intended, we need to consider whether overall performance and reliability needs to be tested further. New user-interface elements or UX changes also should be tested across each display mode and supported localisations. Such a major update may also introduce many new crashes that need to be triaged and scheduled to be fixed within the current release cycle or moved to a future milestone, if they’re not going to noticeably effect the app’s overall quality and experience.

In comparison, minor bug fix releases tend to require a lot less in-depth attention with a narrower testing scope based on a more ‘per-ticket’ basis. We essentially verify whether the original issue has indeed been resolved at no cost to directly related features while checking that related issues haven’t regressed. In theory, this allows us to move quickly in the milestone without being held back by testing wider areas of the app and logging issues that aren’t likely connected to the changes or goal in the current cycle.

As you can see, we can already begin planning tests by knowing what is to be undertaken during any particular release. However, to assure that our test coverage is even more accurate, I keep a keen eye on emailed commit notifications related to any current milestone. Being able to view just how much behind the scenes has changed tells me what the issue was and how it’s been fixed.

QA is a team effort; working closely with our Engineers, in particular, provides valuable clues as to where weaknesses in an app may lie. Discussions with Engineers can be particularly enlightening when it comes to knowing where to look for regressions. Being able to gauge how confident an Engineer is with a newly implemented fix can hint at how much additional time should be spent verifying whether the code is working as intended without re-introducing any previous issues.

Note: If you happen to be testing a particular app but aren’t able to work closely with the Engineers or don’t have access to commit notifications, you can always fallback on Release Notes for a bare minimum guide highlighting what’s changed and in need of some focussed test coverage.

Though our testing is becoming ever more focussed and agile, we still have the fallback of various test manifests for our apps. Our test manifests are useful to ensure that even when the smallest of bug fix releases is tested, the bigger picture can always be kept in mind!

I hope this post provides a useful little insight into how we decide where to focus our manual testing efforts for each release. If there’s anything you’d like me to write or talk about in a future, or if you’ve any questions on this post you can reach me on Twitter.