Building up a startup test engineering team
Some words of wisdom for startup CTOs and VPEs
Author’s note: This is a follow up to my previous article Do you really need to hire a QA or build a QA team? [1] In that article I wrapped it by saying that at some point, you may need to hire your first test engineer and start building a quality team. This article talks about when you should consider that and what I consider a good approach to do that and why.
First do you even need a QA or test team?
Engineer-led vs Product/Business-led companies
I was at a roundtable discussion at Plato’s Elevate 2024 conference called “Quality without QA: Creating Great Software Using Proactive Methodology” mostly out of sheer curiosity (naturally). The host presented the approach they took as an engineering organization where all tests were designed and automated by engineers so there was no need for QAs or a QA team.
I asked “What if product or business asks if you can ship the product a month ahead of the plan engineering provided?”
“We’ll say we’re not ready by then” was the response.
“Ok then product or business will come back saying that this needs to hit that date in order to capitalize on the customer’s need”
“We’ll say we are not compromising on our engineering process. We will not compromise on engineering quality just to meet business need”
“Ok then business/product comes back saying that maybe this is not the engineering team this company needs. We need engineering leadership to meet the business need”
“But you will lose your best engineering talent, you will lose domain expertise”
“They may come back with: What can we cut from the schedule? What’re these tasks that say test automation? We just need someone to test the features correct? Why can’t we hire some temporary contractors to test them?”
The host left the panel with the understanding that there are engineer-led companies where engineers call the shots and have a stronger negotiating position wrt when the product gets shipped and business or product led companies where when engineering has less leverage. Make no mistake, I am not against set ups where there are no QA, it works for some teams (and that’s great! think about how much you save in terms of headcount and coordination) but how do you as CTO or VPE know when you need to actually build a test team (or hire your first test engineer)?
Developer exhaustion
This situation is where the dev team has too much on its plate and there is too much context switching happening eg. full stack engineers moving from FE to BE within or between sprints. They need a test partner (or partners) to (a) help do first level troubleshooting for bugs (b) look at how features should be tested and shoulder some of the load of testing and/or writing test automation. The team has tried to increase the unit and E2E test coverage as advised in [1] but business needs just warrant they focus on building features.
Customer Service exhaustion
In this situation, almost all releases (or a good number of them) create customer issues downstream hitting the customer service team with a deluge of irate customers. The approaches spelled out in [1] are inadequate for addressing the release quality issues or there is just too much business need to get features out the door even at the expense of some customer unhappiness.
Too much test orchestration needed
The measures spelled out in [1] do require you the CTO/VPE to spend some time orchestrating the testing. This could divert attention away from more important tasks aligned with business goals. You could delegate this to a senior developer or product manager but that is not their core competence. If it’s delegated to a less experienced engineer/PM they could possibly not have sufficient clout to say no to a release.
The lack of quality of the product is impeding the progress of achieving business goals
This is probably the most important reason to proceed with hiring your first test engineer or building up a test team: even with the measures spelled out in [1] or the team can’t spare the resources to get unit test coverage to a high level or the code is in such bad shape (high coupling, low cohesion) due to compressed dev cycles and/or working with legacy code touched by multiple developers who have come and gone such that writing unit tests are just too painful; that the company can’t get enough traction on the product adoption or users/customers are so unhappy the business can’t progress.
Why is it challenging to build a start up test engineering team?
Owners vs employees
As mentioned in [1], there are owners and employees. Someone with an employee mindset looks at the job as just a job. He is not emotionally (or financially) invested in the company as a CTO/VPE or even the pioneer engineers. He will use up all his time off and does not see that, as a start up, the company is in a race against time to ship the vision the founders have of the product/company.
In my experience hiring for test engineers in a startup, upwards of 50% of candidates have an employee mindset.
On the other hand the founder/CTO/VPE may know someone from his network he trusts and knows has the owner mindset but does not have the experience of evolving the test function in the team. This leads me to my next point.
Experience vs longer career runway
This is probably the most challenging aspect of building up a startup test engineering team. Unlike in development where you have the CTO/VPE who has some experience in running a dev team and has a 5–10 year career runway, finding someone who has the breadth of knowledge and skills of test engineering and yet is able to grow from being the first test engineer (or first group) to director of quality assurance in say 5 years is a very tall order. The very experienced test engineers may not want to do the lower level work or are in the stage of life they want their holidays with the kids. The less experienced test engineers are in the stage of life they want to establish a career but need the guidance and coaching to grow.
Building up the Test Function
The usual path
What I’ve seen happen in most teams is that the CTO/VPE gets the budget and goes out and hires a bunch of QAs and manages them herself or delegates it to a more senior member of her team or have QAs embedded within each Pod or team. Oftentimes the pain is so acute the CTO/VPE over-hires (this is akin to the advice not to do grocery shopping when you’re hungry). After some time, the CTO/VPE realizes she (or delegates and engineering leads) can’t provide the guidance/coaching for this group wrt career progression and also provide a plan for how to evolve this group on tandem with the rest of the development and product teams. She then gets budget to go out and hire a more experienced QA leader to clean up the mess. The QA leader comes on board and realizes that some of those earlier hires will not be able to make the next level of growth of the company and has to have them replaced. This creates discontinuity and some organizational and product history information is lost in the process.
A better way
Here’s what I believe would be a better way to approach this problem:
- Engage in a QA consultant or Fractional QA leader to help chart the direction
- This consultant/fractional-leader should make assessment of the situation (including size of QA team needed), help set up the test orchestration and then tapping on their network, source for and hire the first or first few test engineers.
- These test engineers should be relatively junior who have the potential and the hunger to grow. The plan is that one of them should eventually head up the QA function reporting to the CTO/VPE
- This consultant/fractional-leader should be engaged over the course of a few years and coach the test engineers so that at least one will eventually run the team
With this approach you have continuity. The hires that joined as the company was growing stay and take up leadership and senior positions in the company. They have an intimate understanding of organizational history (I like to say that “they know where all the dead bodies are”). Tapping on a consultant/fractional leader, you also tap on their network and expertise hence reducing the burden on HR/Recruiting (which is likely to be small).
What is a Fractional leader?
This is from the Fractional First website [2]: “A fractional leader is an experienced executive who offers part-time, flexible leadership. This allows organizations to leverage their expertise without the commitment of a full-time hire. Typically, they spearhead strategic initiatives and oversee operational teams.”
How this compares with a FTE, consultant or advisor/coach? Again from [2]:
What should you be looking for for the first test engineer(s)?
First of all, what should you look in a great Test Engineer
In [3], I cover what a great test engineer should be. In summary, the candidate should have:
- An engineer mindset
- Intellectual curiosity
- Incisiveness
- Tenacity
- Technical competency
- Spine
Next comes attitude
We are looking for candidates with an owner mindset but most importantly HUNGER. Startups are hard. It is a slog for at least a few years and there is no guarantee of a profitable exit. The ideal candidate must have a resilient mindset as there will be a lot of challenges along the way. Lastly, the ideal candidate has to have a growth mindset as he has to grow from building and executing tests to charting quality strategy.
Finally, a reminder
The role of the CTO/VPE is to advance the business objectives of the company from the technology area. There is a lot to do and while at the moment the issue of software/release quality is acute and pressing, it is not the only task that needs attention. It may just be worthwhile engaging in a consultant/fractional-leader to help get this set up and run.
References
[1] “Do you really need to hire a QA or build a QA team?” Heemeng Foo, Jan 2025, Medium, https://medium.com/@heemeng/do-you-really-need-to-hire-a-qa-or-build-a-qa-team-d489f709b9c9
[2] Fractional First website, https://www.fractionalfirst.com/
[3] “What I look for in a Test Engineer”, Heemeng Foo, Oct 2024, Medium, https://heemeng.medium.com/what-i-look-for-in-a-test-engineer-dce2504a41fb
