Do you really need to hire a QA or build a QA team?
The inevitable question facing any software start-up CTO or VPE
How you got here
You’ve worked hard to build your company’s technical team and product past the MVP stage. You have a growing customer base and you are starting to see bugs pop up, some of which are causing serious customer complaints. The mounting bug count and having to roll back releases because of critical bugs is causing frustration by product and business and exhaustion by the development team. You are wondering how you can release without the hassle of rolling back or scrambling to fix forward the P0 and having to deal with irate customers. You remember that in your previous company there was a QA team that the engineering team relied on to certify releases and find bugs prior to releases. It wasn’t perfect but maybe that could be the solution to your problem — just go out and hire one or a bunch of QAs. That could buy your team some time to get enough business momentum to figure out the next stage of growth for the product and maybe get to the next round of funding.
Why I wouldn’t rush into hiring a QA or building a QA team just yet
Hiring QA is just like hiring a developer or project manager. You’ll have to recruit, hire, onboard and then manage them. Unlike you, not all test professionals out there have the owner (vs employee) attitude ie. it’s just a job to them, they don’t have the incentive or motivation to care about the business or your customers. Don’t be surprised when on their first day on the job they tell you that they are flying off to Asia or Africa the next month (because they booked the trip months prior) or they are going to be busy with a home renovation or are getting married and will need to take time off. As a good people manager, you congratulate them and hope that when they’re back they’re motivated and eager to get stuff done. If you’re lucky yes but chances are you’ll be hit with another reason to be away and all reasons given are reasonable. It doesn’t really matter if these resources are remote or not. Each country has their own labor laws that you have to adhere to those wrt time off.
Also in my experience, easily upwards of 50% of test professionals are very weak technically. In [1] I lay down what I expect from a Test Engineer. As a Quality Engineering leader I’ve interviewed a lot of so-called “test engineers”. In my screening interview that covers basics of REST and algorithmic problem solving, easily 80% of candidates do not meet the bar. If these get hired, it leads to the situation where you have QAs that will frustrate your development team. I relate one such case in [1]. From my experience, unless they have some prior experience, most engineering leaders do not know how to separate the wheat from the chaff. I have had to take over QA teams and have to manage people out because of prior hiring mistakes.
What are some of the alternatives to hiring QA?
So what should you do as an engineering leader? You still have the release quality problem to solve. Here are some options:
- Increase unit test coverage
- Increase E2E test coverage
- Crowdsourced Manual Testing
Increase unit test coverage
I once worked with a mobile applications team that had both iOS and Android versions of their app. The iOS team had ~ 90% code coverage for unit tests and the Android team had ~ 60%. The only issues the iOS team had were those that had to do with hardware compatibility and needed my team’s expertise to flush out using our test lab’s set up. The Android team however had very basic functionality issues that could regress from build to build. That demonstrated to me that unit testing was critical in ensuring some level of quality.
Also for a team that just has software developers, this is low hanging fruit. Carve out some time to get your unit test coverage up to snuff. Get your code repositories hooked up to a tool like SonarQube and track it from build to build.
One advantage you have now is that most coding assistants eg. Copilot will assist with unit test generation. That allows your team to speed up getting to some level of coverage.
Increase E2E test coverage
Another thing you can do is to build up your E2E test automation with your dev team. It is advisable to get to some level of unit test coverage so that at least the team has put in place “baby guards” in the code that can aid in overall code quality and code reviews (see [2]).
However I understand that shipping code to validate product hypotheses is key to success in pre-series A and series A companies. Also in series B you need the dev team to be solving customer issues and maturing the platform for scale. Also the code base may be held together with “duct tape and chewing gum” and unit tests may not make enough difference in the timeframe needed. The code will need a major refactor at some point to make it testable but that time is not now. Hence, it may make sense to use limited resources to build up the E2E test coverage even without getting the unit test coverage up to snuff.
One way is to engage the many outsourcing companies to build up a E2E test framework for you. I doubt you will have any trouble finding one — just list yourself as VPE or CTO on Linkedin and they will come calling. The issue is how to assess if they are worth the cost.
Some tools that could either assist you to build and E2E test suite include:
- Autify Genesis — takes in product requirements and generates Playwright code (and test cases too)
- ContextQA — takes in the clickstream of your site and generates a “knowledge graph” and generates test code for the user flows you choose
- Mobot — you provide them with the steps and they create the test automation for Mobile devices
Crowdsourced test coverage
If there isn’t time to set aside for boosting up the unit test coverage or set up E2E tests, a short to medium term solution would be to engage with one of several crowdsourced software testing companies. Some examples are:
- Rainforest QA
- Applause
- Testlio
These companies cover Web, Mobile, some level of performance testing, accessibility and localization testing as well. For a fee (or baked into the cost of the SOW), you can get a project manager to work with you to develop test plans and manage the execution of said tests. They will integrate with your bug tracking system and send over identified bugs with steps to reproduce.
The best thing about these companies is that they operate almost 365 days of the year. If anyone in the team takes time off, it is on them to find a replacement so there is no service disruption. Also being global, they could have their test resources on the other side of the world so that you can adopt a “follow the sun” approach to your build-test cycle.
I’m speaking from experience
I started my career in a dev team of 6 engineers (no QA) and we did everything including ensuring quality. Then later when I joined Yahoo as part of the Mobile team (this was when Mobile was predominantly SMS and WAP) it was just me, the engineer and a product manager. Both of us did testing of our service. I know that at a given scale, it is possible to function without QA or a QA team.
The rest of my career was spent building and running QA teams and orchestrating the combination of inhouse, outsourced and crowdsourced test resources.
So when is the right time to hire QA?
There will come a time when in order to scale, you may need to hire an inhouse QA team. How to tell if you have reached that stage? That’s the subject of a follow up article. Feel free to reach out if you’d like an opinion for your situation.
References
[1] “What I look for in a Test Engineer”, Heemeng Foo, Oct 2024, Medium, https://heemeng.medium.com/what-i-look-for-in-a-test-engineer-dce2504a41fb
[2] “Unit Tests and code reviews: the bedrock of software quality”, Heemeng Foo, Sept 2020, Medium, https://blog.devgenius.io/unit-tests-and-code-reviews-the-bedrock-of-software-quality-9a23cd24558b
