How to Measure Engineering ROI: A Practical Guide for SaaS CTOs. Engineering ROI is elusive — but measurable. This guide breaks down attribution philosophies and provides practical formulas to help CTOs quantify and communicate engineering’s impact.
Very helpful article, Itzy, thank you for sharing your thoughts!
I agree w/ Marek re: repositioning for quarterly board meetings and mature C-suite.
This is all about context, timeframes and valid data. Marek's example works well – development in the prior quarter or two, however, eng investment happens overnight (P1- customer down) and as long as 2+ years (architectural renewal, refactoring, market "pivot", etc.), so there might be a lot of effort being expended with only internal Engineering milestones to indicate progress.
The rationale for longer term investments, even when approved at C- or Board-level, tends to decay over time as those investments are raided in favor of winning another deal this quarter. Point being, there is also a point where projects may be halted or fall outside the ROI time horizon and they tank the numbers.
Whatever is tracked and presented needs to be done within a context, and not just with some numbers. The contextual window needs to span across ALL engineering efforts, not only the recent/ near term ones. Sales is different, it's either this quarter or this year, for the most part, same with marketing leads, campaigns and related.
Good example, Chris. I remember two extreme cases i've participated in. One lasted 5 years(!) and the other was 2 years. The 5 year one developed technology that was truly game-changing for the company and enabled sustained hypergrowth. However, it was so risky that the CEO (an engineer) had to run it himself, because it had such a high likelihood of failure that engineering didn't want to touch it.
These skunkworks projects are extremely risky in at least 2 aspects:
1. Significant amounts are invested (gambled) over a long duration without any commercial result, and with each day the risk of having the budget cut goes up. Luckily, the sunk costs fallacy often helps here.
2. Risk of commercial failure skyrockets because of the lack of market or real-world feedback until the work is released at the end.
I'm not sure whether there's a useful generalisation regarding how to account for these edge cases. They scream significant risk, but also significant reward. I'm thinking this level of long-term risk may need its own tracking category.
Worth a read, thanks, Itzy! My two cents: the situation becomes much easier when:
1. The CTO is not the only adult in the room and understands the dynamic (especially leading/lagging indicators), which is usually series B+ or mature businesses with a fresh and funded C-suite with trust mandate;
2. There is no product/engineering schism and the two blend to iterate on experimentation/betting framework.
In which case, the quarterly board meeting can skip “we shipped 2 features faster by 5 days thanks to the $300k investment” and instead say: 2 of 5 of our bets in Q1 prove 12% increase in retention in segment “X”, which results in $1M increase in mean predicted LTV and $500k competitive edge over competitor “Y. Internal Rate of Return is 36%.
This is what mature C-suite are looking for: in Tech/Product it’s about probability and market context.
1. I purposely made no mention of Product because I didn't want to open a door to a complication I often see, which you aptly named a "schism". But I should have made it explicit that I'm assuming that Product and Engineering are working as a tag-team, i.e. completely aligned.
2. Regarding maturity, which is a function of the right experience rather than time — it's a pleasure when everyone's maturity level is compatible.
3. I love it when talking about bets is tolerated (see maturity above). When a bet doesn't pay off, it is not a personal failing of product or engineering. As long as you manage your portfolio of bets well, and enough if them pay off, things should be good.
That's a must-read for tech leads. A deep article about a topic that so many CTOs struggle with or don't even realize as their problem.
Engineering is CapEx. Wouldn't a discounted cash flow (dcf) method be the most appropriate method here? Reference read: https://www.investopedia.com/terms/d/dcf.asp
Very helpful article, Itzy, thank you for sharing your thoughts!
I agree w/ Marek re: repositioning for quarterly board meetings and mature C-suite.
This is all about context, timeframes and valid data. Marek's example works well – development in the prior quarter or two, however, eng investment happens overnight (P1- customer down) and as long as 2+ years (architectural renewal, refactoring, market "pivot", etc.), so there might be a lot of effort being expended with only internal Engineering milestones to indicate progress.
The rationale for longer term investments, even when approved at C- or Board-level, tends to decay over time as those investments are raided in favor of winning another deal this quarter. Point being, there is also a point where projects may be halted or fall outside the ROI time horizon and they tank the numbers.
Whatever is tracked and presented needs to be done within a context, and not just with some numbers. The contextual window needs to span across ALL engineering efforts, not only the recent/ near term ones. Sales is different, it's either this quarter or this year, for the most part, same with marketing leads, campaigns and related.
Good example, Chris. I remember two extreme cases i've participated in. One lasted 5 years(!) and the other was 2 years. The 5 year one developed technology that was truly game-changing for the company and enabled sustained hypergrowth. However, it was so risky that the CEO (an engineer) had to run it himself, because it had such a high likelihood of failure that engineering didn't want to touch it.
These skunkworks projects are extremely risky in at least 2 aspects:
1. Significant amounts are invested (gambled) over a long duration without any commercial result, and with each day the risk of having the budget cut goes up. Luckily, the sunk costs fallacy often helps here.
2. Risk of commercial failure skyrockets because of the lack of market or real-world feedback until the work is released at the end.
I'm not sure whether there's a useful generalisation regarding how to account for these edge cases. They scream significant risk, but also significant reward. I'm thinking this level of long-term risk may need its own tracking category.
Worth a read, thanks, Itzy! My two cents: the situation becomes much easier when:
1. The CTO is not the only adult in the room and understands the dynamic (especially leading/lagging indicators), which is usually series B+ or mature businesses with a fresh and funded C-suite with trust mandate;
2. There is no product/engineering schism and the two blend to iterate on experimentation/betting framework.
In which case, the quarterly board meeting can skip “we shipped 2 features faster by 5 days thanks to the $300k investment” and instead say: 2 of 5 of our bets in Q1 prove 12% increase in retention in segment “X”, which results in $1M increase in mean predicted LTV and $500k competitive edge over competitor “Y. Internal Rate of Return is 36%.
This is what mature C-suite are looking for: in Tech/Product it’s about probability and market context.
Good points, @marekdulowski.
1. I purposely made no mention of Product because I didn't want to open a door to a complication I often see, which you aptly named a "schism". But I should have made it explicit that I'm assuming that Product and Engineering are working as a tag-team, i.e. completely aligned.
2. Regarding maturity, which is a function of the right experience rather than time — it's a pleasure when everyone's maturity level is compatible.
3. I love it when talking about bets is tolerated (see maturity above). When a bet doesn't pay off, it is not a personal failing of product or engineering. As long as you manage your portfolio of bets well, and enough if them pay off, things should be good.