Testing Strategies for Mobile Applications
Mobile app testing is time consuming and expensive, yet critical to ensuring your consumers have a positive experience when they use your mobile applications. It is vital that you make sure that the experience is a great one for every consumer every time they use your application, starting with the very first time. If you fail to do a good job app testing, this will result in your customer doing it for you—and unlike your testing team, your customers do not have the tools or the time to report back problems. Also, keep in mind that your customers do not want to be treated like guinea pigs. When they find a flaw, you’ll never hear a word from them and they certainly won’t come back.
For many people, “mobile apps” have become synonymous with native applications (and hybrid applications). Commonly downloaded from an app store, they offer the user a unique experience that maximizes the capabilities of the device and operating systems for which they are developed. The app download is often controlled by the gate-keeping app store, with mechanisms in place to charge potential consumers. This simple and proven monetization model has fueled native app popularity within the development community. Beyond their acceptance in the consumer market, they also allow enterprises to deliver productivity tools to an increasingly mobile workforce..
Testing needs to determine whether the app can be successfully downloaded to the device, executed on the device, and interact with the supporting back-end content infrastructure. When updates are made, you need to be sure that the application can be pushed out to and accepted by the end user. There is a misperception that successful testing of app functionality on one device provides assurance across all others of the same operating system.
Devices: The Biggest Mobile Testing Challenge
The mobile devices used by consumers create the most obvious challenge to mobile testing. Potentially tens of thousands of different client devices could be used to access your mobile app or website, and they must therefore all be considered when testing your mobile applications. Add to this the different versions of operating systems, and the permutations get crazy-big! You can sacrifice coverage across device/OS combinations to an extent, but when you reduce the number of device types that you test against, you are taking a chance that your application might not work for a number of potential customers. To handle the device challenge, you have three options: You can test exclusively using real devices, you can test exclusively with emulated devices, or you can use a combination of both.
Network: A Regional Challenge
There are well over 400 mobile network operators in the world.
Each mobile operator may support multiple network technologies including LTE, CDMA, GSM, and some use less common or local networking standards such as iDEN, FOMA, and TD-SCDMA. Each network has a unique combination of network infrastructure that tunnels the packet-based protocols used by mobile networks into TCP-IP protocols used by the mobile web. Each network operator has implemented systems that behave slightly differently from vendor to vendor in order to perform the required tunneling. Lastly, most network operators have inserted mobile web proxies (that is, the gateway) to dictate how, when, and if you are able to connect to a particular site. When a network operator implements a mobile web proxy, it can restrict the flow of information that travels between your server and the test client. Some proxies limit the sites that can be accessed via a phone to only those approved by the operator in what is often referred to as a “walled garden.” Other proxies might use “transcoding” in an attempt to scale down fixed web content to better fit onto mobile phones. As you can see, the network challenge is quite complex.
When you bypass the network’s lower layers, you use TCP/IP to connect directly to the server and you ignore the GPRS tunneling systems used by network operators. Since most real devices are not capable of doing this, you will need to use a device emulator to perform the bypass. Not all device emulators support this feature, so you may want to look for a device emulator that can perform network bypass via the Internet. Some device emulators also have the ability to access the operator’s proxy (but only if it is exposed to the Internet) to allow a more realistic test. Although the operator’s web proxy is available to only its customers, there are test proxies on the Internet that can be used. Even if you don’t have a test proxy, you will still be able to test directly against your origin web server.
Of course, it is possible to test against real networks. One method is to use real devices at the target location, although you will face many of the problems already discussed. Alternatively, many device emulators support modems that allow you to use your emulated devices on the local network—but again, there is the cost of traveling into range of the network. But there is another option.
One piece of useful test equipment is a real device in the cloud. This type of testing solution consists of a physical handset mounted in a remote box with a remote control unit and a remote antenna. The remote control unit is physically connected to the device’s screen and keypad control circuits and is capable of pressing keys and collecting screen images. Exposed to the Internet, this solution lets a user on a local PC or web client control a device with their mouse and keyboard, thereby seeing what is happening remotely on the screen. These devices provide an elegant solution that can be connected to live networks.
Hopefully, you now understand a lot more about the challenges associated with mobile testing of native and web applications. But what do you do with this information? What should be your testing strategy for mobile application testing?
It is not a matter of choosing one tool or technique because there are simply too many compromises that must be made. Most likely you will need to use a combination of testing tools and techniques to meet your quality requirements.