Exploring the state of open source software testing – Software Testing News

Overview

Open source tools have revolutionised the software testing profession. Any tester working on web/mobile apps has at least researched if not tried and adopted Selenium, Appium, or a tool that builds upon them. And any testing tool vendor recognises that it must address (and try to surpass) the value of these free and esteemed tools in order to gain any traction in todays highly competitive market. But, exactly how open source tools are impacting testing is not as clear. In this article, Wolfgang Platz, founder and chief strategy officer of Tricentis, explores the findings of his companys research into the state of open source testing

Exploring the state of open source software testing

Wolfgang Platz, founder and chief strategy officer,Tricentis

Open source tools have revolutionised the software testing profession. Any tester working on web or mobile apps has at least researchedif not tried and adoptedSelenium, Appium, or a tool that builds upon them. And any testing tool vendorrecognisesthat it must address and try to surpass the value of these free and esteemed tools in order to gain any traction in todays highly competitive market. But, exactly how open source tools are impacting testing is not as clear. Open source testing tools are used by a lot of different roles, in a lot of different regions, for a lot of different purposes.

The recently publishedState of Open Source Testing report a collaboration betweenTricentis,TestProject,SpecFlow, andFlood was designed to capture a global snapshot of where the software testing industry stands today and track how open source testing trends will evolve in the years to come. The biggest survey of its kind this year, it explores whether there are any universal advantages or challenges to opting for open source testing rather than commercial offerings, and the nuances that surface when you take a more targeted look at open source testing across functional testing versus load testing and BDD. And, while some of the surveys findings emphasised the expected, there were also some surprising statistics that change the general understanding of open source tool adoption in testing, and even the software testing discipline in general.

Skills and support

Unsurprisingly, the need for technical skills and a lack of product support were cited as the top challenges regarding the adoption of open source testing, accounting for almost 45 percent of the total votes. This corresponds with what weve seen in the field, in that open source tool adoption requires both skilled resources and a strong commitment to making it work. Indeed, those who expect it will be simple to deploy and scale open source testing frameworks will typically be disappointed. While open source test automation offers strong support for web and mobile technologies, it isnt easy to achieve end-to-end process coverage when numerous integration needs must be resolved, as in the case of complex enterprise apps, for example.

Given this need for technical skills, it would be reasonable to expect SDETs (Software Development Engineers in Test) and even developers to be the dominant users of open source testing tools. This doesnt appear to be the case, however. In fact, only eight percent of respondents were developers. With functional testing carried out by a distinct QA function 84 percent of the time, its perhaps not surprising that testing still isnt a focus area for developers. That said, despite a prediction by many people that agile development and DevOps would spell the end of testing, its erosion as a discipline simply doesnt appear to be happening.

Cost pressures

Time was cited as the number one impediment to functional testing in general. The primary motivator for selecting open source tools, however, was cost, along with the promised flexibility that having code fully under your control can lend to integration and customisation. Of course, cost pressure on testers has been exacerbated recently by the COVID-19 crisis, with a considerable spike in demand for the free community solution, TestProject, highlighting the extent to which it has become an important consideration when it comes to tool usage.

Its little surprise, therefore, that having selected an open source testing tool, and invested in customising it and integrating it into to their process, testers will become heavily reliant on it 92 percent of respondents said they considered the tool theyd selected as either important or very important. Most use their open source tool as part of a Continuous Testing process, with 75 percent running their functional test automation frequently or as part of the CI/CT/CD toolchain.

BDD

In general, most respondents said they found it useful to specify application behaviour using examples in varying forms, such as specification by example. The report shows that Given-When-Then (GWT), while not yet the sole standard, is becoming increasingly popular. Cucumber and SpecFlow were revealed to be the dominant players in the BDD tool area, accounting for 94 percent of responses between them. And the choice of tools was found to be highly correlated to the programming language used, with Java shops choosing Cucumber, and .NET users opting for SpecFlow.

Encouragingly, while many respondents were just at the beginning of their BDD journey, with 79 percent claiming to be no more than proficient, half of them (49%) are already experiencing significant increases in development efficiency. Despite this, though, scaling BDD is still a huge issue when you get into the region of thousands of tests, maintenance can become tricky. More needs to be done, therefore, to enable BDD at enterprise scale.

Load & performance testing

Surprisingly, the survey also found that QA and testers cover the biggest portion (47%) of load testing, with very few respondents reporting that load and performance testing was conducted by dedicated performance engineers. Whats more, although more than half of (56%) of respondents run test loads frequently, it doesnt form part of the CI/CT/CD toolchain. In other words, continuous load testing isnt yet a reality. Finally, many of the responses to the question of the greatest challenge with open source load testing touched upon the interpretation of results, an issue its hoped that, over time, will be improved through the application of AI.

These findings are just the tip of the iceberg, of course. Theres no doubt that open source testing tools have revolutionised the way testing is performed. Through community collaboration and transparency, tools are developed at a rapid pace with consistent alignment to customer requirements. Furthermore, the major projects such as Selenium and Appium have become a sort of global standard, allowing for interoperability of best of breed tools, free or commercial. And as their development and subsequent adoption progress, so new benefits and challenges will continue to arise. Next years report could make for very different reading.

More:
Exploring the state of open source software testing - Software Testing News