AI in Software Testing Questions

Authors note: On April 15 2025, I was invited to speak at a fireside chat organized by the nice folks from Browserstack on the topic “Unlocking the power of AI in Software Testing” (recording). As part of preparation, there were more questions looked into than there was time on the fireside chat to cover. Since there was interest in the answers to the questions during that session I thought there could be interest in the questions we didn’t have time to cover as well, hence this little ditty of an article. Do note that the area of AI in Software Testing is rapidly evolving, hence what makes sense now could be obsolete in 6–12 months time. I hope folks here will find these ideas and opinions useful. Do feel free to add comments and any questions and I will try to respond.

What areas in traditional test automation are most ready for AI integration, and where do you think AI can provide the most impact right now?

Here are some:

1. PRD to test cases — I know of several tools that do this including what you guys at Browserstack have done with Test Management
2. Test case to test automation — there are already several tools out in the market as well including what Browserstack has
3. There are also tools to make test automation esp UI test automation less flaky
4. Then there’s also Visual Testing — this is great for a company that have a ton of informational pages to maintain. These are essentially static pages so they lend very well for Visual testing. Browserstack’s Percy is one such product.

I also wanted to mention a use case that is not quite in the area of test automation but rather test case management and Bug management. What if you are able to also, when the QA is trying to add a new test case, be able to tell her that it is similar to this and this test case (and similarly for bugs)? That will really help contain the growth of test cases which helps manage the amount of tests the company actually needs to run (and for bugs keep the number of duplicate bugs to a minimum). I took a stab at this in 2021 with a bug similarity engine using word embeddings and Cosine Similarity (see [1]).

What are some best practices for balancing AI-driven automation with traditional methods to create a robust testing strategy?

The driving force will be business. It doesn’t matter what any of us thinks. Business dictates how the balance will be. If business demands shorter and shorter cycles and maybe at the expense of some bugs, the balance will shift towards more bots. If the business demands for more craftsmanship and products that are a pleasure to use, then the human touch will dominate. The example I like to use is the Hermes Jane Birkin vs say a Coach or Kate Spade. One costs as much as a car and a lot of it is hand made by craftspeople. The other is mass produced.

For teams starting to integrate AI into their testing workflows, what foundational steps would you recommend to ensure a successful implementation?

I think it starts with awareness and getting people to accept that AI is a tool, just like any other and not be afraid of it. Maybe start with understanding what is possible and what is not with just simple prompting since prompting is the basic foundation of this wave of AI. There are several really good courses on prompting on Coursera. Some are free. If companies can sponsor their QAs to attend them, that could be a great start.

Next at the organizational level, you need to provide clarity on the company policies around using AI tools like ChatGPT, copilot etc. As a leader, talk to test tool vendors to understand what is possible given the current landscape.

Next is getting your feet wet: start with some POCs with clear success criteria. Work with management to get buy in on what the objectives of these POCs are: is it to reduce effort, is it to have better test cases, is it increasing velocity of releases, is it to to reduce escapees? Also how do you measure these? Then with clear understanding of these you can formulate a change in the testing workflow that leadership is able to be aligned on.

When considering AI-driven testing tools, what criteria do you believe are essential for evaluating their effectiveness and suitability for a given project?

There’s a saying, you can bring a horse to water but you can’t make it drink and I’m sure some of you are parents. You can come up with all arguments about what course of study your kid should take up but at the end of the day it is the kid who decides. So it is the same with AI or any other tool. The QAs or PMs or devs need to see the value in the tools for them for adoption to get traction. As a test leader I see myself as a facilitator and educator. If the tool is successful and we see gains in terms of productivity and efficacy the ones that adopt it become the standard bearer for the company and other people will adopt it.

Which testing activity do you think AI can simplify the most?

I’m a big fan of Test Plans and Test Plan reviews. In an article I wrote sometime back called “The sieve of Eratosthenes approach to bug free code” [2] I wrote about this. But test planning is hard cos you have to essentially figure out the requirements and the high level design and come up with a test plan that holds water. AI can not only help you create a draft of a high level test plan, it can even come up with an initial set of test cases. That saves a lot of writer’s block time.

How can AI help address some of the more challenging aspects of test automation, like handling complex data sets or ensuring comprehensive test Coverage?

There’s a tool called AskRosie that integrates with Excel. I was at a Meetup recently and the creator of that tool was saying he used his own tool to help him with his taxes. Gemini on Google sheets is also able to help you analyze your data. So that’s the part about complex data sets.

As for coverage — well there’s an approach to testing called Model Based Testing correct? And the idea is that you create a model of the system in test so that you can assess the risk and hence prioritize tests. I’ve yet to see a tool do this but I reckon some smart engineer somewhere is already working on this — ingest the PRDs, HLD and build a model that is able to help prioritize what tests to run.

How do you foresee the role of AI in test automation evolving over the next few years? Are there any emerging technologies or trends you’re particularly excited about?

You can’t really “forsee” AI in test automation in a vacuum or AI in testing for that matter. While AI in test automation is advancing, other areas are also advancing eg. AI in coding, in creating product requirements, AI in how businesses are run. I was in Singapore back in Jan and I heard from a friend that there’s a startup working on creating a whole system — FE, BE using AI. How will this affect QA? Will we still need inhouse QA in 5–10 years time? And if we still require inhouse QA. Will it still be 1:4, 1:5? Why not 1:50, 1:100?

RPA showed some hype some time back but essentially RPA was using the same underlying technology as test automation. But with Agentic AI, will it replace otherwise manual work? How will that change what is needed for QA?

And this is the humbling fact: We only need QA cos we can’t trust the upstream processes to deliver a quality product or service. What if AI helps those upstream processes to deliver quality? How will it shape this industry? That’s exciting to me.

Do you believe AI could eventually handle more complex testing tasks autonomously, or will human oversight and expertise remain essential?

I spent some time at a Digital Agriculture company. Did you know that the machines that plant the seeds and harvest the crops can self drive and have been doing so while Waymo and other companies were testing out self-driving cars on roads? The US is the most productive farming country in the world because of all the technology invested in it. We also now have robot vacuums that roam our homes and if you notice with each generation it gets smarter and cheaper. But notice that we still have the farmer in the combine and we still have to check on the robovac. The AI is very smart and will allow us humans to do more with less but I think we will still need some level of supervision but the scale of human to machine will go up.

What AI feature would add the most value to your overall testing strategy?

The QA function in any company is a cost center. Any AI feature that helps contain costs will add the most value. This is a very rapidly evolving space. What I say today will not matter in 6 months. At this point I would say the fact that AI can help any team speed up test planning and help identify gaps is a big leg up because how shift left can you be if you catch misalignment in the requirements and high level design before a single line of code is written?

For QA engineers looking to upskill in AI-based testing, what resources or tools would you recommend to get hands-on experience?

Start with a simple course on prompt engineering. Coursera has a pretty good one created by Google but it costs some money. I think there are some free ones out there too. Then take a course on AI Agents. I think http://deeplearning.ai has some good ones.

Now for those of you who want to get your hands dirty, AI Makerspace has a pretty good one where you actually do the vectoring and create a RAG etc. It costs US$3k and lasts 10 weeks.

Now if you don’t even know what a Jupyter notebook is, take a basic data science course. Coursera has some good ones.

As for AI in testing, there’s a lot of buzz now so attend a meetup group on this topic or look out for summits or conferences. There are also free webinars like this. Just take some time to attend to get a sense of the landscape.

What do you consider the biggest barrier to adopting AI in testing?

Fear. Fear of the unknown. Fear of losing one’s job. Fear of not being able to catch up. Benoit Schilling, CTO of Google X once told me (he was architect fellow at Yahoo where I was) when I was feeling stuck in my career: on the weekends, go get a beer and learn something. Start small, there are lots of resources out there. Don’t be discouraged if you don’t know something, even the experts don’t know everything.

References

[1] “Building a Deployable Jira Bug Similarity Engine using Word Embedding and Cosine Similarity”, Jan 2021, Medium

[2] “The sieve of Eratosthenes approach to bug free code”, Jan 2024, Medium