Which Tool Is Best For Cross Browser And Cross Device Testing?
As we work on our clients’ projects, we take quality assurance very seriously. But we can’t guarantee there won’t be any minor issues. If someone claims they can, you should probably reconsider your relations with such engineers. As for us, we believe we can minimize bugs by applying software that simplifies the work of our QA engineers and empowers them with smart features.
But as the time passes, things that used to work greatly become outdated or broken, and new solutions arise. That’s why you need to keep your ear to the ground and update frequently. Evgenya Utyumova, Devvela's quality assurance engineer conducted research on existing mobile devices, screen resolution and browser testing tools to find out which ones best meet our requirements. In this article, we sum up this research and its results.
We haven’t been doing Agile Web development from the beginning. One day something just went wrong and the whole system got broken which was the turning point.
So, first, we clarified the requirements:
- All required and customizable screen resolutions, also specified by user, to perform screen resolution tests
- All required OS for responsive design tests
- All required browsers and their versions for cross browser compatibility testing
- All browsers, tablet and smartphone emulators and operation systems in one app
- Realistic emulation of all required devices.
- All-in-one.All browsers, OS and devices in one place.
- Opportunity to customize how long to wait before generating screenshots once a URL has been loaded. This helps automate the process of viewing website layouts by using different operating systems, browsers and screen resolutions. Without this function, there’s a big chance pages won’t have enough time to fully load.
- High performance quality
Then we made a list of browsers and screen resolutions and were all set. With knowing what to look for, we started our search of THE QA tool.
AN IMPORTANT NOTICE FOR THE LAZIEST AND/OR THE BUSIEST! If you want to see the results of our research immediately, here's the table. Green - good, alright, satisfying, red - not so good.
The first web app we tried was Browsershots. It makes screenshots of your web design in different operating systems and browsers by opening the web page on a number of distributed computers.
As you land on the homepage you can submit a web address. But first you have to untick all irrelevant browsers. There are 189 in total (bold choice, Browsershots, we give you that!), 166 of which are ticked on default. Erm… seriously? Ok, so we unticked all the irrelevant browsers one by one and submitted a web page. Then we received a message saying our submission was added to the job queue and that we could either wait, or subscribe to paid priority processing. This came as a surprise because when we landed on the homepage we saw nothing about this system. Unfair and even a bit impolite. Also, we couldn’t find how to customize the waiting time before generating screenshots once a URL has been loaded, which is a clear sign NO to this tool.
Then we switched to CrossBrowserTesting. This web app offers its users features such as live cross browser compatibility testing on various browsers and OS compilations, automated screenshots, comparison tools, selenium tests, local testing and real mobile devices. We found here that all browsers we're interested in are supported. Tests in the same browser were performed in multiple screen resolutions simultaneously.
What irritated us at Cross Browsers Testing was that the web page we viewed was twitching as we scrolled down. This isn’t its main flaw, though. Much worse is that it allows testing only with one browser at once. We couldn’t figure out how to test websites on different browsers simultaneously.
Starting tech business with no money, no technical background and no tech co-founder? Sounds crazy! And yet it can work out.
Then we tried Sauce Labs, an automated testing cloud platform providing a report of the run with metadata, Selenium and console logs. It supports a lot of OS and browsers, as well as allowing you to enter Java, Php, Node.js, Python, Ruby, or C# code. But oh how slowly its emulator responds while testing! Another critical downside is that tested screen resolution appeared not to be customizable, and we could only find three resolutions.
Expression Web SuperPreview
Another fancy tool we thought was worth a test was Expression Web SuperPreview. Developed by Microsoft, it allows you to view your web page’s layout in various browsers installed on your PC. Great features like live and side-by-side testing, as well as all browser views available in one place are negated by one single (and, honestly, unsurprising) feature: only Windows OS and compatible browsers supported. No way, not for us.
Litmus claims to provide optimization tools for email professionals. But we found out it also has a browser testing feature and thought we might have found our ideal QA tool in the place you'd least expect. Alas, it could emulate only four devices, eight browsers/versions, and two operating systems. That’s too few. Litmus might be good at other features and services, but we wouldn’t use it as a QA tool.
Then we came across Browser Sandbox by Spoon.net, a resource which allows for live testing online with popular browsers, such as Opera, Firefox, Chrome and Safari. It supports a variety of browsers but everything works very slowly (if it works, which isn’t always the case). It just appeared slow, unstable and generally unreliable, so… no.
Then someone suggested Blisk, a free browser for developers that provides mobile emulators for different devices online. Honestly speaking, it’s a real handy device emulator! Analytics synchronization, debugging, testing, device emulation and screenshots on one and the same platform make it promising. Why only promising? Because it’s still in beta, and some amazing functions are still under development and don’t work yet. One day, Blisk. But not today.
At some point, TestizeScore engine came up in our search. It validates websites against currently accepted internet standards, isolating issues that may impact the user experience. Results are accompanied by recommendations on how to address these issues and how to ensure successful site rendering. Nevertheless, performance quality and range of both browsers and screen resolutions leave much to be desired.
MobileTest.me was another online tool we tried. It allows you to test web pages with various device emulators. What did we find here? A wide range of mobile device emulators, real time testing, images (not just web pages) viewing on various devices and switching between emulated devices facilitated simply, with a scroll. Yet, it remains unclear which browser and screen resolution are used to view web pages. Together with a subscription price which makes $9,99 monthly, MobileTest.me is far from perfect.
Then we discovered Mobilizer, a mobile optimization platform that allows to inspect, test, monitor and analyze the mobile customer experience on real devices. It offers device-level analytics, automated alerts and sharing feature. However, we discovered that Mobilizer provides no OS emulators, and our decision was made. We continued our search.
And our search led us to the next tool called Browseemall. It allows you to view, edit and create screenshots of a web page both on mobile devices and wide screens. Unfortunately, it appeared to neglect screen resolution and supports only a few browsers and their versions.
Just like in a fairy tale, we had almost lost all hope when we came across BrowserStack. Expecting the worst, we started exploring its features. Well, it appeared to support all the required operating systems, browsers and screen resolutions. Not bad. As we tried emulators, they surprised us every time with high level performance. Everything worked just perfectly. Also, using a mobile device emulator, we were able to switch between browsers and between browser and local device menu. Amazing! And, finally! the time before generating screenshots once a URL has been loaded is CUSTOMIZABLE! Also, it appeared to be a complex tool offering all features including screenshots, all device and OS emulators and wide range of browsers in one place. YAAASSS!
So, we tried out 12 testing apps before we could find THE quality assurance tool. Since then our QA guys have been satisfied BrowserStack users. No need to look for an alternative yet. As we said in the introduction to this post, we cannot give a 100% guarantee there are no bugs because no one can. But we seem to have found a way to greatly minimize them and secure for our clients the highest quality possible.
P.S.: Save the table with the results on your computer or Pinterest!
By Evgeniya Utyumova and Mikhail Chistopolov