On Time, On Point, On Budget!
Start a Project Live Chat

Main characteristics of testing applications on mobile devices

Posted On:
“If you think that now your life is too painless and quiet I advise you to test something on mobile phones” (Peter-Paul Koch, developer of mobile applications, teacher)

Features of testing mobile applications in general

Mobile users expect that the applications they install are stable, quickly react on actions, safe, have a simple user interface, work 24/7 (especially in applications that interact with server). To understand the features of testing mobile applications you should also take into consideration the factors that distinguish the mobile applications from the desktop – various and specific OS for mobile platforms, different manufacturers and configurations of components, functionality of devices like PDA and etc. Concerning these features the approach to the development and in particular testing applications on mobile devices is quite different from the desktop. There are many additional important nuances and requirements that must be tested. Let us show the main differences in some types of testing: Testing updates — frequent system updates make it necessary to update the application. It is necessary to test application updates on the server side, check the installation paths of application (WiFi, cable, Bluetooth, SD). Testing internationalization allows on early development stage to ensure in the internationalization support, mainly in the language support. There can be many problems concerning the lack of free space on the screen, as well as such problems as, for example, the formatting of dates. Testing usability allows to identify the parts of the application that are not attractive, or cause difficulties in the navigation or use, make sure that the model of resource consumption by the application meets the target audience. For example, office applications should not cause excessive energy consumption without need. This testing is often done in the form of beta testing. Load testing involves the observation of memory use and system resources, besides, load testing allows to identify “bottlenecks” in the application relating to performance, as well as to detect the dangerous memory leaks. Stress testing (monkey testing) — the application must respond correctly to the occurrence of random and unpredictable events. Mobile devices often get in conditions when they receive chaotic useless information (for example an unlocked communicator in your pocket), that’s why the application must react adequately to such data streams. Multiplatform and multidevice testing – the application should work properly in all configurations of devices, and all devices for which it was developed. Because of the fragmentation of the mobile phones the testing of all the available devices that use different versions of the tool and have screens with different sizes, different functionality, OS architecture and the hardware is just as important as hard to do. Laboratory testing – an imitation of the real conditions of the communication quality. Conformance testing is used to verify whether the application meets standards, licensing agreements and terms of use:

Android

  • The installation file of application (.apk) should be agreed with Program Policies .
  • In case of publication of the updated build it’s better to follow the order of version control (for example, the adopted order of version numbers).
  • The application must not contradict the UIG .
  • Android market semi-automatically scans the application for viruses (their presence may lead to the blocking of publisher account).

iPhone

  • The application must have a unique name
  • The application must refer to certain category.
  • You need to provide a link for feedback with the developer.
  • The application must not contradict the Human Interface Guidelines document.
  • App store will test the application on compatibility.
  • The application certification may be denied for reasons of availability of prohibited materials, unforeseen delays in operation or recurrence of existing functions.

The list of things that need to be tested

Screen size and touch-interface:
  • All elements must be of such size that the user can clearly click on them.
  • The absence of blank screens in the application (either omit these screens, or write the explanatory text for the first opening on them).
  • The elements speed of response should be high enough (use the old devices if supported by the application).
  • All clickable items should have clickable state (response to the action).
Phone resources:
  • Memory leaks. Especially may appear on the windows, with a lot of information (for example, long lists), during tasks with long workflow (when the user doesn’t leave the application for a long time), when caching of images works incorrectly.
  • Lack of space for installation or work of application.
  • Installation on SD card.
Different screen resolutions and OS versions:
  • Retina and not retina screens. On retina screens the interface elements / text will be smaller. Pictures for retina-screen can be caught in not retina version and then they will be very large.
  • OS versions. The application must not be installed on unsupported devices, a mandatory test on all possible of the supported devices (complete tests on retina and not retina versions with the latest firmwares, additional checks on devices with other firmwares).
  • Various functions in the devices: the absence / presence of the camera (AF), the absence / presence of GPS.
  • Support of necessary media files by the current model and OS, because individual developers can cut down the support of some formats.
The reaction of application on external interrupts
  • Incoming and outgoing SMS and MMS
  • Incoming and outgoing calls
  • Removal of the battery
  • Cable connecting and disconnecting
  • Network connecting and disconnecting
  • SD-card connecting and disconnecting
  • Enabling and disabling of player
  • Charging the device
  • Working with a physical keyboard (if there are such models in the list of supported models) – line breaks, moving through them, etc.
In-app purchases:
  • Match of price and content declared in the application to the ones that user gets.
  • Purchase recovery (application update)
Internationalization (check in portrait and landscape mode!):
  • Checking the correctness of the translation
  • Checking that all labels suit the appropriate forms, buttons, etc.
  • Checking date formats
  • Checking separators in numbers (between digits and between the integer and fractional part)
Constant feedback from the user:
  • Reaction of buttons on click
  • Messages while loading the content / progress bar
  • Messages when there is access error to the network
  • Messages when trying to delete important information
  • Screen/message when ending the process/game
  • Availability and timing of sound and vibration notifications with notifications on the screen
Updates:
  • Make sure that the same OS are supported as in previous version (if the new version of the application uses the new features of OS, then for older supported versions of OS there must a cut version of the application).
  • Checking for adequate updates (all user data are saved, etc.)
Multiplayer Games:
  • Correctness of players connections(for example, debiting from balance only after the connection)
  • Time lags
  • Connection using various networks
  • Correct behavior when disconnecting players
Conformance testing

The challenges faced by mobile application tester

The first and the most important thing is that the testing of mobile phones will take more time than you expected, even if you already admit that it takes more time. (Peter-Paul Koch). Mobile devices operate on batteries and therefore have to go into standby mode automatically after a couple of minutes of inactivity. This means that you will have to turn on the mobile phone before each testing that takes a decent amount of time in case of simultaneous testing of multiple phones. Of course, on many devices you can disable the automatic blocking (or at least make the off time quite large), but nevertheless it is desirable to work with the most common among users OS settings. Switching on standby mode is especially unpleasant when using such testing systems as Browserscope, requiring certain time for completion. It is possible that the phone will go into standby mode right in the middle of the test. Also there is a problem of typing of certain text (such as test page address). The typing can take a lot of time on the phone with a numeric keyboard, but even on the phones with alphabet (software or hardware) keyboard you will have to accurately enter long texts in several devices with different interfaces. For testing in conditions of the incoming calls, sms, you will have to switch SIM-card from one device to another, which often requires removing of the battery. More often this problem appears when testing features and services of mobile operators on non-standard and expensive fees. In addition, after inserting the SIM-card you will have to wait until the phone is on. Often it is necessary to transfer the testing widgets via Bluetooth that is also rather tediously in a variety of interfaces. Usually, all the tests are performed on one device, then on another, and etc. This is not as optimal, as testing on all devices at the same time, because it doesn’t make it possible simply to compare the output and perform tests on two devices, but this is the most cost-effective way. Creating screenshots on mobile devices is also not a trivial task, especially if you are testing the phone, disconnected from the computer according to test conditions or for some other reasons.

Tools for testing: list and features

Emulators

Emulator is a program copying functionality and behavior of the other program. Some advantages of using emulators:
  • operational testing of application, when the target mobile phone is not available (or is in short supply);
  • testing of complicated or dangerous scripts which are impossible or not recommended to be tested on real mobile phones (for example, tests which in any way can destroy the phone or break the terms of agreement with mobile operator).
Disadvantages:
  • Emulators are often very demanding of resources because the most qualitative of them emulate the application work from the lowest level;
  • the fact that the application works on emulator still doesn’t mean anything because users will run the applications on real mobile phones, which are always different even from the best emulators.
Testing on the target mobile phone is the most right way to ensure in proper application functioning because you are running the application on the same hardware as your users will have. For all popular mobile OS there are free (for developers) and rather functional emulators. For example, for Android there is an official Android SDK, which includes mobile device emulator that implements all hardware and software features of a typical device. The alternative is Google Android Emulator. It runs on Windows as a standalone application without the need to fully download and install the Android SDK. Also, of course, there are other solutions: MobiOne Developer is a mobile Web IDE for Windows that helps the developer to program, test, debug, package and deploy mobile web applications on the devices. The solution is paid, supports iPhone and Android. TestiPhone – is a based on web browser simulator for quick testing of web applications for iPhone. Works with Internet Explorer 7, Firefox 2 and Safari 3. It is free, but the simulation is very limited.

Cloud platform devices

Allow you to remotely test your product on many different devices, transferring testing results to the developer. The most famous are Perfecto Mobile and Device Everywhere. The main idea of these solutions is that they have a stand with real mobile devices connected by cable and web camera transferring the images from the phone. Inside a phone photo the image from web camera is inserted. It is fully controlled using mouse. Also Perfecto Mobile and Device Anywhere remotely offer the devices for rent. They have many different phones, and one hour of working with the phone costs about $15. Advantages:
  • black box – there is almost no interference in the test application
  • one tool and test script is used for testing on all mobile platforms
  • variety of devices
Disadvantages:
  • automatic testing is expensive
  • delays while the interaction with the phone in your place

Automated generation of script tests

JamoSolution – is one of the most promising platforms, which is now used for several tools (for example, M-eux test and SeeTest). It allows you to test the iPhone, Android, Windows Phone and other platforms. It supports record of tests (record & play) and it is possible to test iPhone applications on Windows. It works by installing an agent application on the device, which exempts the developer from the modification of his application. EggPlant allows you to run your test script on many devices at the same time, defining the output using method of detection the images on the screen. Supports testing on Android and iOS, emulators Android, iOS and Windows Phone.

Load testing

HP Virtual User Generator. Used to load the server using multiplexing of incoming traffic. The tester creates a script, runs it on the device, HP VuGen captures the traffic and simulates requests to the server with similar information, but from a few thousands or millions of devices simultaneously.

Services for beta-testing

uTest – is a community of 45 thousand professional testers from 180 countries. Real users test the application. This is paid. (iPhone, Android, Windows Phone 7, iPad) The Beta Family – is a free service for testing applications. You should create an account, upload the beta version of the application, send out invitations for testing, then you can handle test results. You can select the type of beta-testers: private or public. If public is chosen, then anyone will be able to test the application. (iPhone, iPad, Android) Zubhium. Provides SDK, which allows you to embed the code in your application for automatic collection of information about errors. The developer lays out the beta version, invites testers, and receives information. At the moment it is offered in beta version and therefore is free of charge. (Android) MOTOREADY App Testing: MOTOROLA XOOM – is a paid Motorola service for testing Android applications.

Collectors of statistics

It’s always interesting to know how users work with your application: what features are the most popular, which buttons are pressed, what settings are changed, what mistakes are made, how much time the user spends in the application. It’s useful to know who are these users, what firmware versions are used on their devices, where are they geographically located, etc. The easiest way to collect such statistics is to use a ready system for collecting analytical information. For example:
  • Flurry (free) (iOS, Android, Windows Phone)
  • Apsalar (free) (iOS, Android)
  • BuzzBox (free) (Android)
  • Google Analytics (free) (iOS, Android)
  • Mixpanel (paid) (iPhone, Android)
  • Localitics (paid) (iOS, Android, Windows Phone)
  • Bango (paid) (Android, Windows Phone)
Each system has its own features: real-time statistics update (Localitics), accuracy with tracking the unique ID of each user (Bango), tools for conducting user surveys (Apsalar), tools for regular sending of notifications to users (BuzzBox) and etc. Of course, there are also many differences in: interface, tools of analyses, availability of additional API, price and set of supported platforms and etc.

Complex solutions

TestQuest Pro – is a tool for full automatic testing, developed for companies and enterprises. It supports functional, load, regression testing, testing of quality and interaction. Experitest.com – allows to carry out automatic, manual (remote) and cloud testing on a large range of devices (although, it is expensive – $ 2500 per year for automatic testing, $ 4000 per year for manual testing). It should be noted that there are companies specializing in testing, including testing applications on mobile devices. For example, Qulix QA provides full test coverage – verification of application work concerning OS, platform, languages and other; passing certification for products approving and entering the market; testing applications on real mobile devices.

Testing on real devices

iOS

There are very strict requirements for interface and functionality, a special closed platform, a unique mechanism of applications distribution, all these requirements rather clearly allocate testing software for iOS in a specific separate niche with its own specific problems. Beta-testing features Beta-testing on iPhone is a time-consuming process for tester. This is related to its multistage: first user need to find UDID of his device, and then send it to developers. Only after these user’s actions this device is included to the list of registered devices, and then a working version is built and a letter is sent to his email requesting the installation. User opens iTunes, installs the application and only after all listed above operations it appears on his device. Also user have to send a feedback manually (it’s not possible to do automatically). Therefore, the issue about using of special services for beta testing for iPhone is more actual than for other platforms.

Android

Monkey testing Monkey is a tool of stress testing. It generates pseudorandom user actions. However, Monkey allows to adjust rather flexibly “randomness”, interval between events, their type, etc. For such testing the application’s source code is not required, it simply must be installed on the device or emulator and can be launched in the simplest case from the console. Tools for automatic testing The most popular solution is Robotium. “It’s like Selenium, but for Android” developers say. The test are written on Java. In fact, Robotium is a library for usual Unit tests, there is no ability to run tests on the device (remote control). To test applications you need to compile them with this library. It is possible to test the applications without source code, but the process is non-trivial. TestDroid — is a plugin for Eclipse that allows to record tests (record&play) in Robotium format. The system interacts with the phone through a standard Android debugger. The developers promise to release also TestDroid server, it will allow to create clusters for testing. The solution is paid and is still in beta-testing stage. MonkeyRunner. It allows to perform functional testing of application providing an API for device management. MonkeyRunner is more low-level in comparison with Robotium and doesn’t require the source code of application. MonkeyRunner uses Python and test scripts or scenarios can be written on Python or there is ability to record user’s actions using the recorder. Monkey and MonkeyRunner are delivered together in Android SDK. FoneMonkey for Android – is a free open source tool for testing of interface. The program can record high-level action-based test automation scripts on Java/Java Script, which can be edited (if necessary) and written manually. Sikuli – is another open source tool for automation testing of GUI. An open cross-platform visual environment for creating scenarios and scripts which is focused on programming graphical user interface using images (screenshots). The main thing is that script that defines the sequence of actions allows using screenshots, to give a command for pressing the button you can just place a screenshot of this button into the script. Literature:: About a year ago a book “Android Application Testing Guide” by Diego Torres Milano was published that might be interesting and useful for Android developers and testers. This entry was posted on Monday, July 30th, 2012 at 3:32 am and is filed under QA.