Latest UI automation test frameworks Review
As I was selecting the UI test framework for a web project, done a little bit of research depending on Project Requirements. Hope this post will help techies who looking for a quick review of the latest frameworks.
Use Case
- UI Test Automation for Web project written in Angular.
- Need multi-browser support including Chrome, Firefox & Safari
- Efficient Testing and easy integration in CI
- Better if Typescript support as it was primary programming language used for existing projects.
Reviewed Frameworks
I strict with my use case in my research and had to be more selective on frameworks. Still, I will add more points that need to be highlighted in each framework which may help readers out there.
Cypress
End to end test automation framework written on Node.js. The unique architecture of running as the same loop of application creates greater access to testing elements and make fast & efficient tests. Having a daily increasing community of users with the capability of Cypress framework.
Pros:
- The architecture of running the same loop as the testing application. Due to this having better access to simply everything in the application.
- Fast execution
- Higher consistency with tests
- Easy debugging
Cons:
- Not supporting Safari browser
- Multiple tabs on the same browser not supported
- The dashboard will be coming with a paid plan.
- Larger projects may need commercial support
Puppeteer
An open-source library developed by Google for test automation on the Chrome browser. Using the Devtools protocol for communication. As this only focussing on one browser properly, users get the advantage of implementing efficient and easy tests with better access to test modules.
Pros:
- Fast and reliable tests
- Efficient tests with better access to modules
- Higher user ratings on Git hub (70k stars)
- No need of installing a separate server to run tests
Cons:
- Only supporting Chrome browser
- Still evolving (But…. having a great community and many supporting documentations :D)
** Maybe a bit biased with Puppeteer. Used it as a POC for my own project and fallen in love with easy implementation and fast tests. Couldn’t find many cons except, the worst one that made me not to adapt, multiple browser support.
Testproject
Cloud-hosted, open-source web and mobile test framework. This SaaS test framework is more into standard projects looking for direct test and easy management with standard use cases.
Pros:
- Cloud-hosted E2E framework
- Fast and responsive support team
- Test recorder help to generate test scripts easily
- Cloud management of tests
- Unlimited cloud space for tests (As per their support team)
Cons:
- The testing team not having full access and management over test scripts due to this SaaS framework
- More suitable for teams looking for standard use cases
- Use Selenium for UI tests
** I tried testproject locally as a trial with a sample use case. But seen inconsistency with test-project agents. Didn’t dig up deeply as not suited for my purpose. Comments are welcome with real users
Nightwatch
New and evolving node.js framework which is using W3C WebDriver API for browser communication.
Pros:
- Evolving community
- Higher git hub rating
- Good documentation
Cons:
- Not properly supporting typescript
- Having Selenium protocol limitations/issues
- Depending on the browser, need to install a specific web driver
Webdriver I/O + Selenium
Web and mobile test automation framework. GUI tests mainly based on Selenium Protocol when supporting multiple browsers.
Pros:
- Enhancing support with other frameworks like “Puppeteer”
- Proper end to end documentation
- The WDIO Testrunner having synchronous behavior which makes test implementation easier.
Cons:
- Need to have Selenium if it needs multiple browsers support
- Underlying Selenium protocols may cause stability issues in CI
- Lower git hub starts (6k only)
Playwright
Microsoft’s web automation testing framework implemented top of Puppeteer with enhanced support for multiple browsers. Playwright is a Node.js library to automate multiple browsers with a single API. Use WebSocket thin client same as Puppeteer for communication.
Pros:
- Fast and efficient tests
- Implemented with promise methods that make synchronous testing. Easy to write tests
- Developed top of famous and growing “Puppeteer” while overcoming its’ limitations
- Not using underlying Selenium protocols (JSON wire protocol)
- A growing community with higher git hub stars (20k just for two years)
Cons:
- Quite new and still evolving (Method chaining missing yet)
- Can be challenging to adopt due to a lack of documentation
What should I choose?
After the research on these test tools, ended up with Webdriver IO and Playwright. The main concern between the two was Webdriver IO using stable selenium while Playwright is quite challenging as it's really new with its tech stack.
I have already used Selenium for years with my previous projects and familiar with some common issues including stability concerns. POC with Playwright was exciting and finding how easy to write and manage with the framework. Still, I know there is a scary challenge of adopting a new framework. Compared to API tests, UI tests are always going to be the hardest ones to automate and so unpredictable in run time. That's why it will be scary while adopting a new framework that may have missing methods and limitations which is not yet identified.
Still, I was thinking of accepting the challenge and decided to adopt “Playwright”.
Selected Tech Stack:
- Playwright: Automation Framework
- Jest: Test Runner
- Typescript: Programming Language
Note: Jest is a javascript testing framework developed and maintained by Facebook which is more similar to Jasmine. Jest is more advanced and provides more layers on top of Jasmine. Check references to find an interesting session on Jest and efficient test writing.
There will be more posts on the journey with “Playwright” which may help other risk-takers out there. Happy Coding! :D
