Mobile app testing has often been not only a necessity but also a pain. The pain is because there are a wide range of OS versions and a huge number of devices that have different screen sizes, features, capabilities, functionality, different firmware. To make sure that the application works correctly, it is necessary to test it on the maximum number of real devices before release. Many companies cannot afford to buy all the necessary devices and then maintain them and buy additional new models. To solve this problem, this is where a real device cloud comes in. In this article, we will understand what it is, how it differs from virtual platforms for testing, and what are the main benefits of the real device cloud.
A real device cloud is a data center that contains many real devices of different brands, with different OS and various firmware versions.
The cloud includes software to connect existing devices with all possible clients, presenting them on a computer or making them available to tools such as Selenium or Appium.
Simply put, the real device cloud is a mobile testing environment that allows developers to test the performance of an application remotely on various devices. The device cloud offers organizations access to the newest and legacy devices. Thus, companies can test the performance, usability, and functionality of the application directly on the devices used by their end-users. In addition, the real device cloud allows organizations to save on the purchase and maintenance of the entire laboratory of the real devices on their premises.
Device clouds differ in their capabilities. Some of them consist of virtual or simulated devices. Others offer access to physical devices. There is no way to say one is better than the other. A cloud of emulator devices allows you to test an application for scalability, you can handle multiple simultaneous requests in one go. The real device сloud provides insights into the way factors such as battery charge will impact app performance. Moreover, the real device cloud is often preferable to testing with a device in hand, as it allows you to take notes, switch devices quickly and easily, perform automated tasks, etc.
Device clouds come in different types and sizes, offer different payment models, can be designed for a specific audience, and have specific functionality settings and use cases.
The virtual testing platform is a software that imitates mobile devices. You can test your mobile app using an emulator or simulator directly from your desktop.
Now let's figure out when to use the real device cloud, and when virtual testing platforms will be a better choice.
User interface accuracy validation
The user interface is a critical aspect of any mobile application because most people prefer to use the mobile version, as the phone is always at hand. The user interface does not need any special hardware for testing. If you need to test how good your interface looks, a virtual testing platform is the best choice.
Reliable network conditions
This is one of the important focus areas in mobile app testing. Impossible to guess the characteristics of different bandwidths. Even if we take into account that many people are today on 4G and above, this does not change the fact that network conditions are unreliable when the device is in motion. Moreover, mobile application testing should also focus on offline data management so that the user experiences less inconvenience due to poor networks.
Today, almost all testing of mobile application options are related to network bandwidth. Virtual testing platforms began to provide options for network throttle, and there is also software such as Clumsy that can help you tighten the bandwidth number for testing. However, all these options are not enough to test the application in real conditions. When it comes to bandwidth, we are talking not only about the reception but also about hardware problems, as well as other reasons. In this case, it is better to use testing on real devices with network options.
Hardware & sensor-related verification
Mobile devices have many sensors, and if your application uses any of them, it is very important to test its performance in this aspect. Here are some of the sensors:
- Light measures the intensity of light.
- The accelerometer measures acceleration.
- Proximity measures the distance from the sensor to the nearest visible surface.
- The barometer is needed for GPS and air pressure.
- The magnetometer is needed for magnetic intensity (applications related to the compass, etc.)
- The gyroscope measures the angle of rotation.
Smartphones also have temperature and relative humidity sensors. Sensors involve hardware, so it's important to test sensors on real devices. Accuracy in testing is critical as we must meet user expectations. For example, certain games may use gyroscope sensors to give users the option to play by tilting the screen. Users expect the game to be responsive to the sensor. Any lag or delay can cause the user to leave the game, and this behavior is the same in any application.
Testers use emulators to test sensors. However, this happens at the initial development and testing stages. Once you've reached the final stages of the development process, real devices will help you gather accurate insights into how your app works in real-world conditions. These insights will help you fix any problems with your sensors.
Performance testing
Application performance refers to how well it performs technically. Performance testing helps to understand where the weak points in the application work. The list of key benchmarks for performance analysis includes loading time on various devices, threshold traffic bandwidth (testing how the application behaves with high traffic), and parameters in various network conditions.
Load times in virtual testing and real device cloud can be less than one second. However, to get a complete picture of the performance of the application, you need data on network conditions and offline application work data. This requires testing the app on real devices.
Maintenance as an additional cost
The virtual testing platform does not require special maintenance. Maintenance of devices is quite expensive and time-consuming. If you want to test on real devices, you need to buy them, and then keep them in working order and replenish your device farm with the latest models. Another thing is if you use the services of a real device cloud, then a third-party organization does all this, but you pay for the rental of devices.
Battery testing
Battery drainage is a serious problem these days as people spend many hours on their phones. When buying a new device, they pay attention to how long its battery holds a charge, and how long it takes to charge. If during use, it turns out that some application significantly drains the battery, most likely the user will delete it without much thought.
Virtual testing platforms are not an option when it comes to battery usage. In this case, you should always use a real device.
CPU usage stress testing
CPU usage on the virtual test platform can also mislead you with lower values. The actual state may be quite different.
Battery and CPU usage are related as more CPU usage generates heat and battery consumption increases. Since there are hundreds of processes going on on a real device, testing on a real device is the only way to get an accurate idea of the CPU usage while an application is running. While you're at it, don't forget to watch how your device's memory and GPU are being used.
Test execution on a real device cloud gives real results that emulators can't achieve. Here is a list of the benefits of using a real device cloud.
Unlimited concurrency. Even if you have automated most of your test cases, you still need to run them on different devices with different specifications. In this case, testing can be accelerated by running tests on multiple devices at the same time. The time saved can be spent on manual tests.
Release speed-up. Testing is a time-consuming process. At the same time, testers often work with limited time. In this case, real device clouds have a super-function to save a huge amount of time, because testing can be carried out simultaneously on all available devices.
Customization to your needs. Client-oriented real device clouds always provide the possibility of customization. Agree, that is convenient when there is an opportunity to change the environment to suit your needs. It is important to remain flexible to quickly adapt to changing business needs. The same applies to the tools we use. When one of the tools becomes an integral part of the processes in the company, we don’t want to migrate to a new solution because the old one no longer meets the needs. Therefore, it is important to immediately choose a flexible tool with customization options.
Reliable testing results. Above, we have already considered two ways of mobile application testing: testing on real devices and emulators. The main difference between the two is that emulators act like real devices, but they run on a computer. Emulators have several undeniable advantages. However, emulators are not ideal for testing your application in real-world conditions, as they cannot, for example, show how using the application affects battery life, CPU usage, and so on. Incorrect settings in the emulator can destroy or falsify your test results. As you can see, testing on real devices is more reliable and efficient.
No need to create, configure and maintain your own device lab. Real device cloud allows you to save time and money that could be spent on purchasing devices and creating and maintaining your device lab. Working with a real device cloud, you allow your team to implement the latest testing practices. Real device cloud comes with all the features you need. You do not need to worry about the maintenance and updating of your devices and related costs.
Increasingly, testing teams are turning to real device clouds. There are many solutions on the market, so how do you choose the one that best suits your team? Here are some simple tips:
1. Be specific about your team's needs. Understand what you expect from a real device cloud, and how it will fit into your processes. Determine how a device farm can help meet your project's testing needs today, in the long and short term.
2. Choose a few real device clouds that suit your needs and use the free versions of them. In this way, you will understand how well the device cloud integrates with existing processes and tools. You will also be able to understand how convenient the device cloud is to use.
3. Chat with the support teams of the device clouds. So you can understand how quickly and clearly the support service responds to requests. Ask for a user demo of the features that interest you the most.
4. Pay attention to whether the real device cloud has the ability to test on virtual devices as well. As already mentioned, the combination of emulators, simulators, and real devices provides teams with optimal testing coverage.
5. Try Zebrunner Device Farm. Full access to devices is provided through ADB, you can do development and debugging remotely. Among the main functions of Zebrunner Device Farm are multithreaded test execution; the ability to use cloud devices for both manual and automated test scenarios; full control over the device, which includes manipulation, logging, and screenshots/ video capturing; integration with several tools (Jenkins, Teamcity, TestRail, JIRA, Slack and many more) and support for automation frameworks (Selenium, Appium, Carina, XCUI Test, UI Automator, Protractor, Custom Test Frameworks).
