Many people have often thought about what is more profitable: to ride your own car or use Uber. A similar question arises for companies who need to test their product on real devices: to set up their own device farm or to use a public device farm. We will clarify it below. Furthermore we will consider two public device farms.
Uber is cheaper than using a private car, if you drive in the city not more than 91 km per month in USA, not more that 98 km per month in UK, not more than 67 km in German. Let’s see how things are with test automation, when you need to choose between a public device farm devices and a creation of your own device farm.
Device Farm is a testing environment that helps to test remotely the performance of web and mobile applications on various real devices. Device Farm includes both modern and outdated devices from multiple manufacturers.
The need for device farms arose when the smartphone market started. Even then testers had to test applications on various physical devices and on various OS to make sure that everything works fine. But every year, more and more new smartphone models enter the market, and now almost every day new models come to market. Moreover, different models have different versions of firmware, which makes manual testing more lengthy. To reduce the amount of manpower involved, some companies started to create device rental services, or private device farms.
Now you can use a specific device from the device farm remotely: if the developer or QA engineer works from home or even from another country. In practice, it works as follows: through remote access a special script is run, which simulates the user’s behavior on a particular device.
Public device farms are farms that provide separate devices for public usage.
Using Public Device Farm can be a good solution for midsize businesses that care about quality assurance of its product. At the same time these businesses can’t afford to maintain a large QA team or large number of devices for testing.
Public device farm allows you to:
- Run tests on the same devices that your customers use.
- Simulate real customer issues that may occur in real life while using your software.
- Access numerous devices from different manufacturers with different features, different versions of OS, and firmwares.
Public device farms work with various types of cloud, such as public, private, on-premise. So you can not worry about the security of your data and customers’ data.
Private device farms are farms that individual companies create for their own use. They use devices that they have and unite them in one system with access through a cloud. Every employee gets access to every device from anywhere.
Private device farms can be useful for customer service. A customer with a faulty device can connect it to a private device farm through the cloud and allow the technical specialist to fix the issue remotely. Once the customer disconnects from the farm, remote access to its device is immediately terminated.
If your company already has all devices necessary for testing, then it’s a great idea to create your own private device farm. If you don’t have all the necessary devices to build your own private device farm, it’s better to use a public farm.
Let’s see how device farms work comparing 2 products.
AWS Device Farm is a service for testing Android, iOS, and Web applications on real devices such as smartphones, tablets, desktop web-browsers, and improving the quality of tested products. The service allows to execute test-cases in parallel, generates logs and video to rapidly fix the bugs identified. AWS Device Farm is used for automation testing and for testing with remote access.
The workflow of automation testing with AWS Device Farm:
- Step 1. The customer selects an app (native, hybrid, or web app) to test.
- Step 2. AWS Device Farm tests the customer’s application against selected real devices (smartphones, tablets). Developers get access to selected devices through local host-machines.
- Step 3. The customer gets results with marked bugs and performance issues.
Features and benefits of AWS Device Farm:
- Testing native, hybrid and web-applications on mobile devices.
- Testing cross-platform applications including apps created with PhoneGap, Titanium, Xamarin, Unity, and other frameworks.
- Automated tests against a wide range of smartphones and OS versions.
- Remote access to screen via browser and interaction with devices (gesturing, swiping and interacting with devices for Android apps).
- Parallel running of tests against a range of devices.
- Using built-in or open-source test frameworks Janet, RLi, others.
- Integration with multiple AWS services, including solutions for different market industries (advertising and marketing, media, entertainment, game tech, IoT, and more) and various business aims (analytics, data storage, machine learning, data security, and more).
- Test environment can be configured by location, language, network connection, application data, pre-installed apps to simulate real-world customer conditions.
Why AWS Device Farm suites not for all customers
- This is an expensive tool when you use it intensively.
- Not very clear reporting.
- UI could be better. Some customers have problems with understanding how to use the functionality of this tool.
- Limits for devices. Almost all devices are public. The customer couldn’t keep one device more than 150 minutes per session for manual and automated testing.
Zebrunner Device Farm is a physical device farm for mobile testing, works with private and on-premise types of cloud. Customers get full access to the devices through ADB, and the ability to develop and debug remotely. Zebrunner Device Farm provides a cross-platform solution, a private mobile cloud for testing apps. A private cloud ensures the highest level of security.
What problems Zebrunner Device Farm solves
Necessity to execute a large amount of automated tests daily (500+). At the same time, using a public device farm for this purpose is expensive.
Complex test flow, where simply test-cases require an interaction with multiple mobile platforms, web, API, and IoT devices.
Ensuring the highest level of data security, mandatory for a number of industries.
Main features of Zebrunner Device Farm
Multithreaded tests execution. You can divide the test-suite into multiple threads, when you need a rapid result of test execution. You also can run a simple thread for deep regression coverage of every device.
Dockerized container. Each device has its own dockerized container. If you need to restart Appium or ADB service, it’s not necessary to restart the entire cloud, it’s enough to restart the container for a separate device.
Full control over the selected device. Each selected device is accessible through the admin panel. The customer has full control for manipulation, logging, screenshots/videos.
Plug-n-play. Easily remove mobile devices, when you need it for manual testing. When you’re done, plug it back, and it will work automatically.
Integrations. Use tools that you prefer and get results from them in one place. Retrieve the results directly where you want to work with them: Jenkins, Teamcity, TestRail, JIRA, Slack and many more.
Automation Frameworks. Run mobile test scripts using Selenium, Appium, Carina, XCUI Test, UI Automator, Protractor, Custom Test.
Zebrunner Device Farm benefits
Pricing. We apply elastic pricing when you pay just for time of using this tool. This price includes access to private cloud, dedicated set of devices, secure access via SSO, parallel test execution, automated and manual test scenarios, permanent access and full control, hardware installation and maintenance.
High speed of test running and test execution. Zebrunner Device Farm allows users to run hundreds of tests in parallel and get the results within minutes.
Security. Support team can configure the entire infrastructure directly in your office or in a remote data center, ensuring compliance with all IPSEC and data protection standards.
Rapid scaling. You can use as many devices as you need.
Uber Business Model is beneficial in automated testing in such cases as:
- You have a small or medium-sized company, and you care about quality assurance, but can’t afford to create your own QA department and purchase a necessary number of devices for testing.
- You are in a hurry to release your apps faster and want to save time on testing and processing of test results. Even if your company has its own testers and devices for testing, the process itself is very time-consuming. Sometimes it’s a good solution to save man-hours and use professional service from third-party device farms. As a rule device farms allow to run multiple threads in parallel and save time on testing and analytics of test results significantly.
- As it was mentioned, farms have a large number of models of devices with different OS systems, firmwares, and features. You get maximum coverage for testing and avoid many problems in real-life using your product.
