On Time, On Point, On Budget!
Start a Project Live Chat

Tools for mobile application testing: choices and features review

Posted On:
Konstantin Shlykov How to improve the labor quality of mobile apps testing employees and to escape from routine? Obviously this can be done with the help of additional tools: from small applications and extensions of SDK to multifunctional automated complexes that execute testing.

Video capturing of device screen

An opportunity of video recording of a reproduced error can often be very effective, as it helps to describe the bug in detail and save developers time or even to avoid the bug reopening due to misunderstanding. Android screen capture – capture is a tool to transfer video from your Android device screen to a computer monitor. To tell the truth, the tool can’t yet record a video. It automatically takes a series of screenshots if the screen changes. However nothing is against using the desktop screenshots with an option of video record. It requires installed SDK. Reflector – is a paid tool for recording video from iOS-devices with a wireless connection to a desktop. You can run it on Windows or Mac. It has a 10-minutes trial version. There is also the Display Recorder tool for iPad and iPhone, however it appears and disappears from AppStore quite often. At a present moment one can’t find it at AppStore (please, note that Display Recorder HD is a different application without screen recording feature. Different sources give different application prices (ranging from two to ten US dollars).

Emulators

Emulator is a tool for exact or partly imitation of functionality and behavior of the device or any other program. Some advantages of using an emulator:
  • Operational application testing, when the target mobile phone is not available (or is in short supply);
  • Testing of difficult and dangerous scenario, that are impossible or not recommended to be done on real mobile phones (i.e. tests that can crash the phone or break the terms of agreement with mobile carrier);
Minuses:
  • Emulators are often very demanding of resources as the most qualitative of them duplicate application work from the lowest levels;
  • The fact that the application runs on emulator still doesn’t mean anything as users will run the application on a real phone which always differs even from the best emulators.
Testing on a target mobile phone is the most right means to ensure the correct application work as the application runs on the same hardware that users will have. There are free (for developers) and rather functional emulators for all popular mobile OS, i.e. for Android there is an official Android SDK which includes a mobile device emulator that implements all hardware and software features of a typical device. Same “native” tools are available for iOS and Windows Phone. Of course there are some alternatives: MobiOne Developer – is a mobile Web IDE for Windows to help a developer to program, test, debug, pack and deploy mobile web applications on devices. It has a lot of useful features, i.e. a page source code view and code inspector. The tool is paid (the current price is 100$). There is a 15-days trial version. It supports iOS and Android. Electric Mobile Studio 2012 – is an emulator for testing iOS applications on Windows. It supports jQuery Mobile, Mobile Web JavaScript, HTML5. It can be build in Visual Studio 2012, it allows you to emulate work with geo-location, switch between device types on the go, etc. The tool is paid (the current price is about 40$), there is a 7-day trial. You can also find older free lite versions in the internet. Opera Mobile Emulator and Opera Mini Simulator Opera Mobile Emulator and Opera Mini Simulator are tools that allow you to test a web-application in corresponding browser. Both tools are free (the second one works online and doesn’t need installation on your computer). BlueStacks App Player is a tool that allows you to launch an Android application on Windows XP-8 and MacOS. Apparently the tool is a virtual machine which doesn’t emulate the device behavior however it can be used for testing the functionality of the application if other means are not available.

Cloud platforms for devices

These tools allow you to test the application on multiple devices, sending testing data to a developer. The most famous tools are Perfecto Mobile and Device Everywhere. The main idea of the tools is that they have a stand with real mobile devices connected using cable and web-camera which transfers images from the phone. There is a web-camera image inside phone photo. You can operate it using a mouse or with the help of automatic script. Also Perfecto Mobile and Device Anywhere offer you a remote “rental” of the devices. There is a subscription and free trial versions. Pluses:
  • Almost no interference in the tested application;
  • One testing script used for testing on all necessary mobile platforms;
  • Variety of devices (iOS, Android, WP, Blackberry and etc.);
  • No application restrictions from the technological point of view – you can test native applications and HTML5
Minuses:
  • High price;
  • Delays when interracting with phones from most countries.

Automatic running of script tests

When you choose a suitable tool you should bear in mind a principle of its work. There are two most popular ways:
  • Test starts when you refer to the screen without analyzing the screen itself and interface elements. Usually it starts via coordinates of gestures on the screen. The main advantage is that there is no need to modify the application. The main minus is dependence of tests from the screen size, device orientation and application design.
  • Running a test by referring to interface elements of the application. The test has labels for forms, buttons, textblocks and other visual stuff. The main plus is that even big changes in the interface can hardly affect the work of the test. The main minus is that developers will need to prepare application versions with additional libraries.
UIAutomation – is a standard tool by Apple that allows you to run test scenarios written in JavaScript on emulator and on device. It is included in Instruments. You don’tneed to compile the additional libraries. There is a feature of recording tests via recorder starting from Xcode 4.3. Robotium – is probably currently the most famous tool to automate Android application testings.“It’s like Selenium, but for Android” – say developers. Tests are written in Java (there are third-party solutions that allow you to write them in Python). You can launch a test only on emulator, not on devices. You need to add a library to the application build. MonkeyRunner is given in Android SDK. It allows you to execute a functional testing of Android application, providing API for device management. MonkeyRunner is more low-level in comparison with Robotium, it doesn’t require application source code. Tests are written in Python or with the help of recorder, executed on emulator and on real devices connected to computer. The big minus of the tool is that gestures are recorded in coordinates and the result check is done by screenshots comparision, that make the usage of the one testing script on different devices more difficult. Also it makes scripts not suitable for regressive testing in case of application GUI change. TestStudio – is a free application for autotesting on iOS. It is based on addressing to application components, not on screenshots. Here we speak about a recorder and an option to manually write and edit tests. It allows you to test web and native components. It has an interestig feature: You can record and run tests on a device without connection to a computer. AppThwack – is an interesting service for testing on Android devices (iOS support is said to be added soon). You upload your application to some resource, it is installed on a real device (the choice is over 100 items) and becomes available for “investigation” – launch, checks of memory usage and processor load, reveal of errors and problems, load with Monkey test. There is a report with screenshots based on testing results. There is a trial (a week), price starts at 29$ per month for testing on 10 most popular devices. JamoSolution – is one of the very promising platforms where several tools (i.e. M-eux test and SeeTest) are currently developing. It allows you to test iOS, Android, Windows Phone and other platforms. It supports test records (record&play). It works by installation of an agent application on a device. It frees you from modifying your application. There is a trial version. EggPlant by TestPlant studio allows you to run your testing script on a great number of devices at a time by defining incoming data with the help of image recognition method on a screen. It supports testing on Android and iOS devices and emulators, as well as Windows Phone emulator. The application was developed for Windows, Linux, Mac. There is a trial version. Squish – is a paid (2400$) tool for autotesting of Qt, Web, Java, iOS and other applications. It supports test record, understands scripts written in Javascript, Python, Perl or Tcl. There is a 30-days trial version. Sikuli – is an open source tool for automation of GUI Java application (including Android) testings. It is an open cross-platform visual environment for scenario-script creation which is orientated on programming graphical interface with the help of images (screenshots). Its feature is a script that sets a sequence of actions and allows you to use screenshots. To give the “press the button” command, you need to add a screenshot of the button to the script. It supports writting scripts in Java and Python. MonkeyTalk – is a free tool for Android and iOS application testings. It has its own powerful scripting language (you can write scripts in Javascript), it allows you to create and store test projects (test cases, suit cases. There are also an integration with Eclipse and a recorder. It requires inserting its library to the application. Robot Framework – is an open source framework for automation of accepting testing and development via accepting tests (ATDD) with a broad functionality. It supports additional libraries (you may use your own ones written in Python or Java). Due to libraries that have been implemented there is an option to test applications on Android and iOS.

Load testing

HP Virtual User Generator is used for loading server by incoming traffic multiplexing. A tester creates a scripts and launches it on a device, HP VuGen captures the traffic and imitates server requests with similar information, however from several thousand to million devices at a time. Neoload allows you to simulate real-world network conditions (emulation of 3G, 3G+, H+, 4G LTE). It can be set for simulation load from a different list of devices (there are pre-sets for iPhone5, Samsung Galaxy Tab II, Nokia Lumia 800, Blackberry Bold 9900 and etc. devices). The main load script can be recorded with the help of any real device using Wi-Fi. There is a free monthly trial, the price of the cheapest license (for 5 users) is 1200 EUR. It supports iOS, Android, Windows Phone and etc. devices.

Monkey testing

Monkey is a stress testing tool for Android (presented in Android SDK). It generates pseudorandom user actions. It allows to set “chaos”, interval between events, their type, etc. Code modification is not required. Testing can be done on emulator or on the connected device. Anteater – is a monkey testing tool of iOS applications. Its functionality is broader than the functionality of Monkey for Android. Probably it is only for Mac. It is free.

Services for beta-testing

uTest – is a society of 45,000 professional testers from 180 countries. Real users test application work. It is paid (iOS, Android, Windows Phone). The Beta Family – is a free service for application testing. You need to register an account, upload a beta version, send invitations for testing, proceed testing results. You may choose a beta-testers type: private or public. Once publich is choosen, the application can be tested by anyone. It works on iOS, Android, Windows Phone. Zubhium. is an SDK for Android which allows you to embed in your application a code for gathering information about the errors. The developer uploads a beta version, invites testers, and get the information. The subscription price is 10$ per month. There is a 30-days trial.

Statistics collection

It is useful to know how users work with your application: what features are the most popular, what buttons are pressed, what settings are changed, what mistakes are made, how much time a user spends on different application screens. It is not also bad to have a user statistics: their iOS version, their location, etc. The easiest way to gather the data is a usage of a ready system for gathering analytical information. For example: Every system has its own feature: real-time statistics update (Localytics), super accuracy with tracking a unique user ID (Bango), tools for conducting user surveys (Apsalar) and etc. Of course there are also many differences in: interface, analyses tools, additional API availability, price and set of supported platforms.

Other useful tools

Fake GPS – is an application for Android devices which allows you to set an optional data in a geolocation module. Also in the above mentioned Android SDK there is a not bad set of small tools which simplify Android application testing. They are a cantilever options for installation, removal and launch of applications, real-time view and output of logs of device work to a file, application reload, etc. A description of these options you can easily find in the internet. You can find lots of useful information in the book by Diego Torres Milano, a developer of SDK : “Android Application Testing Guide”.

Complex solutions

TestQuest Pro – is a tool for a fool autotesting. It has been developed for companies and enterprises. It supports functional, load, regressive, quality and interraction testing. Experitest.com – allows you to conduct autotesting, manual (remote) and cloud testing on a big set of devices (2500$ a year for autotesting, 4000$ a year for manual testing). It is necessary to say that there are companies that specialize on testing, including mobile application testing. For example, Qulix QA provide an all-around testing coverage: verification of application work considering operation system, platform, languages, etc. passing certification for products signature and their appearence in market, application testing on real devices. This entry was posted on Tuesday, April 30th, 2013 at 10:04 am and is filed under QA.