Your marketing VP may tell you that for every dollar she spends, the company gets back $3 in revenue.

Sales people are measured by the amount of revenue they book. It’s probably the primary determining factor of how much they take home.

In contrast with our VPs of Marketing and Sales above, many CTOs can tell you only this: “Without my department, we wouldn’t have anything to sell.” Correct, but not useful.

The connection between engineering and revenue is not clearly visible, and in most cases is actually quite elusive. This is for two inter-related reasons: the lack of a clear cause-and-effect relationship, and the tendency of results to lag significantly behind the corresponding efforts.

The engineering department’s job is to build and maintain the company’s core revenue engine. Engineering, therefore, obviously plays a central role in the company’s success.

Attempts to attribute a fair portion of the company’s business outcomes to the engineering department are seriously challenged by the tension between engineering’s key role, on one side, and the opaqueness of the chain of causation between engineering work and business outcomes, on the other.

Why Quantify Engineering ROI?

Before we attempt to solve the challenge, let’s examine why we need to do it in the first place.

It’s natural to want to know how well engineering — one of a startup’s most expensive departments — is running. The marketing and sales departments can provide simple numbers, directly tied to revenue, that give a good snapshot of how well they are functioning. It’s only natural to try to get a similarly succinct number from engineering.

When asked, “What is Engineering’s ROI?” the CTO feels pressured to answer it, but often doesn’t have the tools. So the CTO counters with DORA metrics, uptime percentages, bug rates, and developer experience (DX) evaluations. None of these translate into a bottom line revenue number. In extreme cases, CTOs have even been known to claim that engineering creativity and innovation cannot be quantified, and should not be measured.

Even though everyone appreciates the difficulty, the CTO’s inability to account for the high costs of their engineering department is nevertheless perceived as the CTO’s own personal shortcoming.

Even if the CTO manages to present the above metrics impressively, the fundamental lack of ROI data remains, because (for example):

• Efficiency does not imply effectiveness.

• Shipping features does not necessarily bring in revenue.

• Velocity does not indicate value.

A seemingly excellent engineering department can generate a quite low ROI, if it does not pull in the direction of some very specific business outcomes. One infamous archetype that exhibits this behaviour is known as the “feature factory.”

The pressure to understand the engineering department’s ROI is typically an attempt to validate whether the high costs are worthwhile. However, if you do it right, you can turn this process into a tool that will actually generate a higher ROI.

The ROI Mindset

The main difficulty with measuring ROI is that it requires mindset changes from the CTO himself, the CEO, and the other executives.

Outcomes versus Outputs

ROI revolves around business outcomes, yet in many companies the engineering department merely delivers requested outputs, in the form of features. In such cases, the head of engineering can be held accountable only for outputs, and they will be unable to optimise for ROI — even if the CFO actually calculates it using some arbitrary formula. In many such output-driven feature-centric companies, no executive will be explicitly responsible to ensure that the requested outputs plausibly align with the company’s business goals, let alone be truly accountable for the eventual outcomes.

If we want true accountability for outcomes, we need to assign the corresponding responsibility and authority. Of course, we need a CTO who is worthy of these, and can properly fill “a seat at the table.”

Leading Indicators

In order to be able to get any inkling of ROI, we need to make a habit of circling back to measure usage of new features, and how they contribute to acquisition, retention, or whatever their stated aim was.

Business outcomes tend to surface in lagging metrics — calculated well after deployment of the changes that we hoped would drive the results we need.

Therefore, CTOs must ensure that relevant leading indicators are identified, that can be used as early litmus tests. This also highlights the imperative to invest in observability and analytics — something which many output-oriented engineering departments feel they have little time for. They see it as a separate effort, distinct from the main deliverable, and which will just slow them down. A mindset that is more conducive to ROI would be, “a feature isn’t complete without observability.”

Advantages of measuring and monitoring leading indicators:

1. Prediction — Leading indicators allow us to track, predict and report success early.

2. Attribution — Suitable leading indicators make it easier to attribute results to specific efforts and features, compared with lagging metrics, which are aggregate effects of multiple phenomena.

3. Savings of time and resources — By monitoring leading indicators, we can make early course corrections, thus saving time and reducing waste of engineering resources on avenues that we discover to be doomed to fail.

4. Increased likelihood of success — We can increase our focus on the intended results, and therefore our likelihood of attaining them, by clarifying from the outset which observations will indicate how successful a feature is considered, and by selecting appropriate leading indicators. (I teach techniques to identify intended outcomes and relevant leading indicators, such as The 5 Whats and Definition of Success.)

How to Demonstrate ROI

Here are a number of methods that can be used, individually or in combination, to form an impression of the ROI of your engineering department, and determine whether it is “enough.”

A. Compare Reality With Expectations

The most natural method, in use almost everywhere, is to compare reality with your original expectations. Are you getting what you expected from engineering? This amounts to a gut feeling, and it is especially prone to conflation of the following factors:

1. Are you satisfied with engineering’s performance? Did they deliver what was requested, on time, and with good quality? Any such satisfaction has nothing to do with ROI.

2. Are you satisfied with the business outcomes that were enabled by engineering’s efforts? If not, should this low ROI reflect back on engineering’s standing in your eyes? After all, engineering itself may have nevertheless performed outstandingly, even though your bets didn’t win.

B. Compare Yourself Against Benchmarks

Current financial benchmarks will tell you whether your engineering costs as a percentage of revenue are consistent with other companies. Your assumption may be that if your investment follows the benchmark, and you’re satisfied with engineering, all is good. But how good is it exactly?

Such benchmarks have a lot of play in them, depending on your industry, and also the growth stage you’re at. For example, a company that has pulled its acceleration lever to scale rapidly after finding product-market-fit (PMF) might increase engineering spending past 60% of revenue, whereas larger, more stable companies will tend to spend a lower percentage.

It is worth explaining that if you invest 60% of your revenue on engineering, that does not mean that engineering ROI is 100÷60 = 167%. The revenue arriving now is most probably not a result of your current spending.

True ROI is a measure of how well your past engineering investments are influencing current revenue, or how well today’s bets will convert into future payoffs. Startups are notorious for their high failure rate at this game.

Source: Benchmarkit’s 2025 B2B SaaS Performance Metrics Benchmarks

C. Demonstrate ROI via Initiatives that Drive ROI

I have successfully used this model at multiple companies, and it forms the core of the method I teach CTOs that work with me. It’s a 3-stage process that’s simple to grasp, but which requires more than technology skills alone to carry it out.

1. Identify
   Identify business outcomes that the company needs. Ideally, you will derive these directly from your company’s business goals, and agreed upon business strategies that form approaches for achieving the goals. If such goals or strategies don’t exist in coherent form, you will need to play an important part in clarifying and crystallising them, because without a clear business direction, you’ll get nowhere useful.

2. Commit
   Design technology initiatives that will result in the outcomes you identified. You cannot do this in a vacuum — you must align and collaborate with your fellow executive leaders. This collaboration has the potential to position you as a capable and trusted partner who leverages technology, and the centrality of the engineering department, in pursuit of the company’s business goals.

3. Deliver & Report
   Execute the initiatives and ensure you deliver the results. An integral part of this requires proper tracking and reporting of results.

By committing to solve specific business problems and to achieve specific business outcomes, and then honouring those commitments, you are demonstrating your value as a leader, and attributing to your engineering department the credit for those specific outcomes.

Here’s an example to demonstrate:

Business goal:

Increase Annual Recurring Revenue (ARR) by 60%

There are many ways to achieve this goal. Multiple business strategies that are agreed upon, including:

Reduce monthly churn down to 11%, to boost MRR in powerfully compounding way

The CTO defines the following initiative:

Identify and fix our top 3 churn triggers (in order to boost MRR independently of traffic or conversion rates)

Later, even before the full effect has been felt, the CTO is happy to report:

We cut churn from 20% → 10%, by fixing our top 5 churn triggers.

It’s a +$10M (+34%) boost to our projected 12-month revenue.

A CTO who can intentionally deliver highly relevant business outcomes, and who can also demonstrate progress using relevant indicators and metrics, is achieving a “high ROI,” even though such a statement may be based on an unquantified intuition.

In this example, based on a development cost of a few hundred thousand dollars, the company got amazing value for money.

Engineer your pathway to ROI

My next free online workshop for SaaS CTOs and Engineering VPs will get you started immediately with practical steps to deliver, demonstrate and boost engineering ROI.

In 60 minutes, I’ll teach you the framework that has helped 120+ engineering leaders get started on optimising their departments and operations for maximum ROI.

Register Now (Free)

D. Calculate ROI using a Formula

The initiatives approach gives results that are undeniably powerful. However, these results are expressed anecdotally, and cannot necessarily be mathematically aggregated across initiatives to produce a standardised measurement.

If we want to gauge how much leverage the company is getting from its engineering department, we need a method of attributing financial results to all the relevant departments.

Companies tend to come up with their own formulae, or at least their own way of applying them. Anyone doing so should consider the following factors:

Attribution Philosophies

There are three philosophies regarding ROI attribution (excuse my borrowing of marketing terminology here):

1. Full Attribution (a.k.a. “Single-Touch”)
   Attribute 100% of the impact to the department or initiative that is most closely associated to the final conversion.

2. Fractional Attribution (a.k.a “Multi-Touch”)
   Split the attribution across contributing departments, according to their proportional contributions. Some companies will attribute arbitrary percentages of revenue e.g. 30% for marketing, 40% for sales, etc.

3. Causal Attribution (a.k.a. Incremental Attribution)
   When there is an incremental lift (“delta”) in a metric, attribute the resulting effect to the initiative or department that caused it.

There are many drawbacks with these philosophies, for example:

1. The full attribution philosophy ignores the inter-departmental teamwork that underpins any product, crediting only the last department in the chain. This will naturally concentrate most of the perceived value in the marketing or sales departments. Additionally, for each type of revenue-generating conversion, including new ones that arise, you’ll need to discuss and define which department exclusively gets the credit.

2. Fractional attribution seems significantly more fair, yet using arbitrary weightings encourages never-ending negotiations about adjusting the weightings.

3. Causal Attribution can be problematic to implement properly, for a number of reasons.

   1. Revenue will need to be broken down and attributed to many different efforts. It quickly gets impractically nitpicky.

   2. If an increase in the percentage of landing page visitors who registered free accounts eventually turned into more paying customers, was this because engineering streamlined the registration form or because marketing figured out how to bring better quality traffic? Or both? Or neither?

   3. Attributing an intentional improvement in a leading indicator can be practical, but isolating its eventual effect on the lagging revenue metric is often not feasable among the noise generated by many concurrent efforts.

4. There are potentially some damaging second-order effects of full attribution, and to some degree fractional attribution, depending on the split. For example, engineering may have no incentive to prioritise work that sales will get credit for, over work that directly helps engineering’s own ROI score.

Attribution by Activity or Revenue Type

The above attribution philosophies are concerned mainly with who to attribute revenue to.

In many cases, it may make sense to overlay this with an orthogonal dimension that denotes the what — activities that led or contributed to revenue-generating events. Dave McClure’s famous AARRR Pirate Metrics framework provides a convenient list (I’ve tweaked it for my own purposes):

1. Acquisition of new users.

2. Activation of the new users — i.e. getting them to their Aha moments.

3. Revenue — converting free users into paying customers.

4. Retention of paying customers for as long as possible, to maximise their lifetime value (LTV). This will also include upgrading them to higher cost packages, known as “expansions” in accounting language.

5. Referral — getting customers to refer others who become customers.

A simplification of this that might be preferred by some companies could distinguish just between new and pre-existing revenue. For example:

1. New revenue — revenue received from customers who converted during the reporting period. This could be attrbuted to the marketing department, or to sales, or split across both, because these departments’ efforts were the key contributions here.

2. Pre-existing revenue — recurring revenue from customers who converted prior to the beginning of the reporting period. A company might decide to attribute retained revenue to the engineering department, because a high quality user experience that delivers value consistently is much more conducive to retention.

This approach essentially boils down to a breakdown of “Single-Touch” attribution by revenue type.

Another perspective is not to differentiate by revenue type, but by the intended purpose of investment. Most basically, this can be expressed as follows:

1. Keep the lights on (KTLO) — Efforts and expenses that are required to keep the product functioning smoothly with its current feature set and customer base. This maps (roughly) to what the CFO calls operating expenses, or “OpEx” for short.

2. New features and capabilities — Investment in new features and capabilities such as scaling up the infrastructure to support a larger customer base. This is what the finance people call capital expenditure (“CapEx”).

CFOs care about the distinction between OpEx and CapEx because these expenditures are treated differently in financial reports (CapEx are reported as assets, while OpEx are expenses), and have differing effects on taxes owed and even influence the company’s valuation differently.

Distinguishing between OpEx and CapEx is logically understandable, but attempting to attribute incoming revenue to the corresponding type of investment is impractical in many cases.

Additional Factors to Consider

1. Risk Mitigation

   A significant portion of engineering work mitigates risks, to prevent potential losses and other adverse events. Production downtime can have a high direct cost through loss of revenue (e.g. in ecommerce), and also can be accompanied by an indirect cost in the form of reputational damage. The attribution problem here is not the who, but the amount. We can know exactly how much we invested, but any figure we put on the potential damage we prevented (multiplied by a probability percentage we pick) is purely imaginary — because if our effort is successful, the events do not happen.1 Will we factor this type of activity into our ROI formula, and if so, how?

2. Actual vs. Predicted Revenue

   Do we care only about actual revenue accrued during the reporting period, or will we attempt to estimate the net present value (NPV) of all improvements?

3. Time-lag

   Do we care that investment and payback periods could be offset from each other, e.g. investment over a sustained period that starts paying off only once it is complete? The residual effect will persist — for how long should we attribute this to the original investment?

4. Intent

   Do we care whether an effect is due to deliberate effort? For example, if we increased revenue by raising prices, does anyone get credit for that (apart from the decision-maker)? What about if we got more business because a competitor had a serious outage? Do we care to differentiate between internal and external causes?

5. Stability

   How stable is our formula? How prone is it to constant renegotiation attempts? How sensitive is it to changing industry or market realities?

6. Incentive

   Is one of the aims of our ROI formula to serve as an incentive for departments to improve? Might our formula cause territorial disputes and dis-incentivise the departments from collaborating? How can we avoid dis-incentivising investments that do not obviously generate revenue (see risk mitigation above), and therefore lower a department’s ROI score (mathematically)?

7. Fairness

   Are we trying to be fair? If so, why exactly? Is it just to prevent squabbling? How does fairness affect each department’s incentive to improve? Will a fair formula encourage collaboration between departments?

8. Availability, Clarity, Estimatability and Mechanical Simplicity

   How readily available are the up-to-date numbers for calculating the formula? How easy is it for a department head to calculate or estimate their current situation? To what degree are the attributions and amounts subject to a determination by the CFO before they can be relied upon? Can a department head understand clearly how to adjust their activities in order to improve their ROI over time?

What Makes an ROI Formula Useful?

I’m not looking for an academic formula that produces a number that has little practical value in real time. My formula needs to be useful. To me, that means it must do as many of these as possible:

1. Track something meaningful over time.

2. Create a timely feedback loop for departments to improve the score.

3. Help engineering compete for budgets and resources with the other departments.

4. Align the departments’ incentives so they all pull in the same direction.

5. Be comparable between companies, e.g. in a private equity portfolio. Optional.

For my formula to help departments improve, it must be:

1. Actionable — department heads must be able to determine which factors influence their scores, and be able to control or influence the values of as many as possible of these factors.

2. Estimatable — department heads must be able to [at least] estimate their current situation to confirm that they are improving. Therefore, It must also be straightforward to calculate or update, i.e., not too intricately pedantic.

3. Mechanical — application of the formula must be mechanical, i.e. not subject to discussion, nor pending the CEO or CFO’s as yet unknown, or even unpredictable, interpretation of recent happenings.

4. Timely — the raw data to calculate the scores must be available regularly, in a timely manner that allows department heads to iterate and improve.

In order to ensure incentives are aligned, the formula must appear to be fair. Fairness also promotes stability (less squabbling → fewer changes), which is a prerequisite for tracking values over time.

ROI Formulae & Application

In The Hitchhiker’s Guide to the Galaxy by Douglas Adams, the Deep Thought supercomputer calculates the “Answer to the Ultimate Question of Life, the Universe, and Everything” to be the number 42. Unfortunately, after 7.5 million years, no one knew what the actual question was, thus rendering the answer somewhat meaningless.

Similarly, when attempting to calculate the ROI of an engineering department, we must clarify to ourselves the specific questions we want answered. In doing so, it is also helpful to anticipate how these answers might influence our subsequent actions.

It turns out that the accounting world already has a formula for ROI, and in the case of SaaS companies, the answer might as well be 42.

ROI Formula - Formal Accounting Version

This is a simple ROI formula for a company. It quite simply means that your company’s ROI is your gross profit divided by your costs of generating it. Ideally it’s a positive multiple, or you can represent it as a percentage.

The question this formula answers is, “For every dollar we spend on delivering our product or service, how much extra money do we get back?”

Cost of Revenue is a term that has quite a narrow definition. It will typically include only the direct costs associated with keeping your service operational for customers. Not salaries of engineers who are building new features, and not (for example) your cloud CI/CD service. This definition is designed to eliminate noise when comparing the financial performance of different companies.

In a manufacturing scenario, the above question could be stronger: “For every additional dollar we spend on delivering our product or service, how much extra money can we expect to get back?” In the SaaS scenario, where costs do not vary much with the number of customers, the above formula will not answer this question. However, it’s not needed; the answer is zero.

This means that formulae based on Cost of Revenue will fail to capture much of what a software engineering department does.

Marketing Efficiency Ratio

Marketing is another department whose activities are not captured by Cost of Revenue. The Marketing Efficiency Ratio (MER) provides a holistic impression of departmental ROI.

A nuance to note here is that they don’t deduct their direct costs — they are just looking for a simple multiple to track. Also, they don’t include salary costs of operating the marketing department. This could be fine for marketing — their raw materials costs are for advertising — but engineering’s main raw material is human brainpower, and the chief costs are in the form of salaries.

Engineering ROI Questions

A SaaS company’s engineering department does two main things:

1. Operate and maintain the company’s product or service.

2. Improve the product or service and develop new capabilities.

That’s how the engineers may see things. From a more outcomes-based perspective, these are the purposes of the engineering department:

1. Retain existing customers, or at least don’t give them a reason to leave.

2. Attract new customers, or enable efforts to do this.

These two sets don’t map onto each other exactly2, but holistically speaking, they are reasonably close.

Somewhat paradoxically, many SaaS companies spend the bulk of their engineering budget on building new features, while the majority of their revenue comes from serving their existing customers.

This situation encourages two distinct ROI questions:

1. “How much revenue is supported by each dollar we spend on operating and maintaining the service?”

2. “How much new revenue is enabled by each dollar we spend on improving the service and developing new capabilities?”

Engineering Operating Efficiency Ratio

Question: “How much revenue is supported by each dollar we spend on operating and maintaining our product or service at the current level?”

This question produces a simple ratio: the revenue attributed to existing customers, divided by the cost of keeping the system running smoothly.

In order to apply this formula, we need to define the terms, and obtain or calculate their component values.

Revenue from Existing Customers

We are calculating values pertaining to a specific time period — the “reporting period”. Revenue from existing customers consists of total payments received from customers that were already paying customers — not free users — prior to the start of the reporting period. Do not confuse this with the amount of monthly recurring revenue (MRR) at the start of the reporting period.

Revenue from existing customers will naturally be affected by these phenomena:

1. Account upgrades (“expansions”) that existing customers made, from one paid tier to another, e.g. from Pro to Enterprise.

2. Price increases that your company instituted.

3. Churn.

KTLO Costs

If all you had to do was to keep your service operating at current levels, how much would it cost you? What percentage of your current engineering and DevOps capacity does it require?

You’d still probably need cloud environments for development and staging, as well as a CI/CD system, and of course Jira (or equivalent).

Here’s a list of typical expenses:

• Direct expenses of production:

  • Troubleshooting customer issues and fixing production bugs

  • On-call employee compensation

  • Servers or cloud compute instances

  • Databases and object storage

  • Networking services (CDN, API gateways, load balancers, etc.)

  • Security and regulatory compliance tools, services and work

  • Backups and snapshots

  • Logging, alerting and observability

  • Data warehouse

  • Replicated resources for resilience and disaster recovery

  • Any other 3rd party API or service fees (production only)

• Appropriate fractions of indirect expenses, that I nevertheless include (see “Classify Engineering Costs as OpEx or CapEx” below):

  • Development and staging environments

  • Software subscription costs, e.g. GitHub, Cursor, Jira.

  • Any other 3rd party API or service fees (development)

  • Departmental expenses, such as:

    • Salary costs (these are higher than gross salaries, due to employment taxes, health insurance, investment schemes, etc.)

    • Travel, team-building, training expenses, etc.

    • Rent and utilities

    • Shared burden of internal services such as HR, legal, etc.

    • Laptops and other equipment

Examples of “appropriate fractions” (above):

• If KTLO only requires 10 people, attribute the cost of 10 Jira licences to KTLO.

• If 20% of days worked were spent on KTLO, attribute 20% of your salary costs to KTLO.

• If something has a fixed cost (or a minimum cost), and is necessary for KTLO, then attribute 100% of the cost (or the cost of the minimal package) to KTLO.

Engineering Growth Efficiency Ratio

Question: “How much new revenue is enabled by each dollar we spend on improving the service and developing new capabilities?”

This question produces a simple ratio: the amount of revenue attributed to new customers divided by the cost of improving the service and developing new capabilities.

Revenue from New Customers

Revenue from new customers consists of total payments received from entities that became paying customers during the reporting period. Note that this is not the same as the amount of MRR at the end of the period.

Amortisation

Amortisation is a way of spreading the recognition of costs over an expected payback period. See “Classifying Engineering Costs as OpEx or CapEx” below.

Classify Engineering Costs as OpEx or CapEx

To begin to answer these questions, you need to classify all your engineering costs as either operating expenses (OpEx) or capital expenditure (CapEx). Your CFO probably already does this, possibly using rough estimates, or according to “financial engineering” principles that may not reflect your CTO’s version of reality.

The way engineering departments operate, including the tools and systems they use for development and testing, is a critical part of a company’s secret sauce of “manufacturing” its product or delivering its service. It’s akin to a purpose-built manufacturing facility. As such, I want to consider all of engineering’s operating expenses, rather than be limited strictly to what accountants include in Cost of Revenue. This also has the advantage of simplifying things somewhat.

Operating Expenses

In this context, operating expenses (OpEx) include anything you spend on keeping your product or service operating smoothly, in its current state and level of service3.

Capital Expenditures

Capital expenditures (CapEx) include the costs of improvements you make to your product or service, and anything you invest in new capabilities — basically any investments that are intended to increase the value of your product. In this specific context, CapEx will generally be anything that is not OpEx.

CapEx Amortisation

It often takes quite a while for a new feature to start delivering results, and its productive lifetime can last for years. The simple fact is that most of a feature’s payback will not arrive during the reporting period in which the feature was delivered.

Were we to recognise the costs immediately, it would inflate our costs in the current reporting period. These costs are chiefly unrelated to the revenue that arrived during this period. When this feature’s revenue arrives later, it will be as if by magic, because it will not have any corresponding development costs. This would render our ratio misleading.

Therefore, treat all capital investments as though they were spread over 3 years (for example). If our reporting period is a quarter (3 months), include (“recognise”) 3/36 = 8.33% of the total cost in each period. Your CFO calls this “amortisation.”

Restated Formula

Based on the above, we can restate our engineering ROI formulae:

Track Capital Expenditures in Aggregate

Don’t attempt to track launch dates and amortisation periods for individual projects or tasks — it snowballs too quickly into a thankless task.

A little-known, but sad fact is that most capital investments will fail to recoup any business value. When you realise one has failed, you could decide to write it off as an immediate loss. Don’t do this, for the same reason as above. Unless you want to handle an exceptional case, it’s just too nitpicky.

It is sufficient to track your aggregate capital investment for each period, and have a spreadsheet distribute it over the following 3 years.

Classifying and Tracking Costs

You need to track two main things:

1. Direct monetary costs — These are things you forked out money for, and are naturally tracked on a spreadsheet in units of money. Attribute each expense to OpEx or CapEx based on its purpose. Where appropriate, split the attribution to reflect the relative amounts consumed by each category.

2. Effort — Track effort invested in OpEx and CapEx. Translate this into money by expressing effort expended (for example, in terms of man-days worked, or number of people assigned) as a percentage of your total capacity, and then multiplying by your total salary costs. Use Jira’s (or whatever work queue system you’re using) tagging or custom fields to classify epics/stories into appropriate categories for this.

Conclusion

Is calculating engineering ROI the right thing to do? And is it fair to collapse all of engineering’s activities into a pair of numbers? Two short answers:

1. Someone will do this anyway. When done naïvely, the results can be unintentionally destructive. I’d rather you had a framework for doing it constructively.

2. Fairness isn’t the point — usefulness is. Used appropriately — and not as a scorecard to punish offenders — an ROI model can be very useful.

However, engineering ROI is more about the journey than the destination. Figuring out the right questions is of critical importance. The moment you start being explicit about ROI — and have deliberate conversations about it with your CEO, CFO and peers — you will naturally start aligning and optimising for ROI.

I have proposed two complementary formulae for measuring engineering ROI. But beware: the ability to measure ROI does not magically give you the ability to generate ROI — which is arguably far more important. The simplicity of the formulae hides the wealth of strategies, techniques and judgement calls that the CTO or VP of Engineering needs to employ in order to consistently deliver, demonstrate and boost ROI. That’s what I do with clients over a period of months.

Acting on the above ideas will stop “What is engineering’s ROI?” being a trap question that freezes you like a deer in headlights, and convert it into an opportunity for you to align incentives, focus your company’s roadmap, and defend the budgets you genuinely need. It’s how you’ll ensure that the value of your engineering organisation — with you at its head — becomes obvious to everyone in the room.

CTOs and Engineering VPs:

• Come to my next free online workshop that gets you started immediately with practical steps for engineering your pathway to ROI.

• Forward this article to your CEO and/or CFO to start aligning your joint ROI vision.

CEOs and CFOs:

• Bring your CTO and work with me together, to design and implement your pathway to maximise your company’s engineering ROI. Book a meeting, or send me a message containing the phrase “ROI article” via my LinkedIn profile.

At the highest level, your company has a goal, which can be to solve a problem, or a class of problems:

• People need to travel from A to B but don't have a car.

• People want to invest but don't know how to start.

• People want more books to read, but affordably.

• People want/need a life partner.

Your Approach

How will your company achieve this goal?

What is your approach?

As a single phrase.

→ That's your Strategy.

Let's give it a capital S because it's the company's highest level strategy.

Effectively, this Strategy is who you want to be as a company.

• The easiest investment platform.

• The taxi app that gets you there fastest.

• The cheapest bookshop on the internet.

• The only dating app that finds you normal people.

That's not the level of strategy my CTO coachees need help with. We focus on one level below this:

Business Goals & Strategies

Ideally, a company has business goals for the next quarter or year.

• Capture a 10% market share.

• Break-even on unit economics.

• Reach $1M annually recurring revenue (ARR).

• Raise a financing round, based on $150 million valuation.

There are many ways to achieve such goals, and each will have its own trade-offs. So the company's departments need to align on some agreed-upon strategies to achieve them.

In this context, a strategy is the chosen approach to achieving a common goal.

Example business strategies:

• Lower the subscription price (to capture more market share).

• Increase advertising budgets (to increase paying customers).

• Increase conversion and customer lifetime value (to break even).

Technology Initiatives

Once we have a set of business strategies, the CTO's job is to figure out how to leverage the engineering department to support these strategies.

For each business strategy, the CTO may be able to identify and propose strategic technology initiatives that could support it, for example:

• Handle a higher volume of users.

• Measure and improve time-to-activation.

• Increase Google Page Speed Insights score of all landing pages.

These initiatives involve identifying, controlling, and moving technological levers that control business outcomes.

Yes, CTOs can, and should, take responsibility for driving business outcomes!

Free Workshop

I'm hosting a free online workshop on 3 + 4 March to show SaaS CTOs/Engineering VPs how to drive strategic business outcomes, and how to get the C-Suite’s cooperation, too.

Register for Workshop

You lost money on it.

You didn't charge anywhere near enough.

Oh, you didn't even charge for it?!!

Did they they ever even use it?

Are they even still a customer?

Here's why you lost

Three major reasons:

Future maintenance tax

You factored in only the cost of building the feature.

But it's still costing you every sprint — because you have to make sure it didn't break.

You also need to keep it well maintained in line with the rest of your product.

Complexity tax

Complexity increases with each additional feature, so:

Each new feature increases the cost of every future feature.

For your product's lifetime, you've just made it slightly more time consuming and expensive to develop anything new.

Opportunity costs

Saying yes to these extra efforts meant saying no to adding features that will possibly drive high impact outcomes for your business.

Recommendations

1. Charge a LOT more for doing "Customer Specials," or agree only to bring forward features that are already planned — but still charge extra for this.

2. Keep track of the percentage of your effort capacity you spend on "Customer Specials." Agree on a budget that you won't be pressured to exceed.

3. Instrument your product to measure feature usage, and get rid of unused features. (Sorry, I meant "retire" them.)

P.S.
Are you a CTO, VP or Dir. of Engineering? — Can you imagine being able to focus every sprint on delivering high impact business outcomes?

👉 Reply to this email with the word "OUTCOMES".

With recent growth, many functions have been slowing down, and there are even occasional outages of various features. It looks like the system is functioning at maximum capacity, and all the engineering team can recommend is to "increase resources" or "increase the DB instance size" etc.

Basically, the development team cannot squeeze any more capacity from the system. This effectively means that the business cannot grow.

You got hired to rescue the company from this situation. The CEO and the other execs expect you to have all the answers. The clock is ticking ...

How would you approach this problem?

A. Rewrite the system from scratch

The current implementation is clearly and severely flawed from a scalability, reliability and resilience perspective. Despite this, rewriting the system from scratch is the wrong answer — in most situations. 

During the rewrite, business growth, and product growth, will be frozen — making it an expensive option. Functioning businesses do not typically have the patience for such things.

The current solution has been trouble-shooted (trouble-shot?) enough to iron out the most egregious bugs and is functioning well enough to garner revenue. Rewriting it will create a whole new set of bugs and quirks that will have to be ironed out.

Thirdly, you'll need to migrate data from the old platform to the new one — which could be fraught with pain. 

In summary: business growth will be frozen during the rewrite, after which there will be a turbulent period during the shakedown of the replacement platform.

B. Buy time and show some quick wins by easing bottlenecks

Analyse known problems, re-examine prior assumptions, and use your  knowledge and experience to guide the team to implement quick (and sometimes dirty) fixes or "plasters". This way, you will very likely squeeze more capacity out of the system while getting more intimately familiar with it. 

Showing some quick wins will also generate build confidence with the executive team, which you may need to rely on later.

These are examples of quick wins I have used successfully for significant results:

• Database Optimisation:

  • Ensure suitable indexes are used for retrieval. (But not too many, which slows down inserts/updates.)

  • Identify and optimise slow queries.

  • Consider using read replicas for reporting / OLAP.

  • Where suitable, cache expensive query results using an in-memory cache such as Redis.

• Traffic Management:

  • Use a content delivery network (CDN) such as Cloudflare to cache static files e.g. images and scripts.

  • Use a web application firewall (WAF) such as Cloudflare to filter out distributed denial of service (DDoS) bots.

• Networking Optimisation:

  • Ensure that requests from one service to another are not routed through your cloud's public network interface.

  • Ensure that your service-to-service communications use connection pools with keep-alive.

You may also need to stoop to these "dirty" (but apparently effective) solutions:

• The dreaded "Increase resources" (RAM in DB machines, ...)

• Scheduled rebooting of various resources that "get stuck," either prior to crashing, or which then go into "zombie mode."

What other quick win ideas have you used? — please comment below.

Remember to measure the effect of the improvements, and their impact on core business metrics!

This option not a solution on its own. It supports incremental growth and buys you time to make a more informed choice about your next move.

By the way, there is also a non-technical bottleneck you may decide to solve at this stage: the team's skill level. They built the current system, and were failing at scaling it. You will likely want to upgrade the team's abilities by bringing in some more capable and experienced people.

C. Refactor or rewrite pieces of it gradually in stages

The system is functioning and bringing in revenue. You'll have the luxury of being able to get immediate feedback on any modification you make.

You'll need to identify components that can be replaced without affecting the rest of the system. Even parts of existing components can be re-routed and replaced using an approach along the lines of Martin Fowler's strangler fig pattern.

This approach seems slower than a complete rewrite, but it's far less risky, and has the added advantage of delivering improvements continually. It's definitely more "messy" than a rebuild, but in a true engineer's mind, there should be nothing more beautiful than a system that actually delivers — and it will continue delivering at every step of the way.

Conclusion

Freezing growth in favour of a complete rewrite is almost always the wrong answer from a technical viewpoint, and is a complete non-option from a business perspective.

As you've probably guessed by now, my default approach, in situations akin to the idealised dilemma I presented, has been as follows:

1. First: Buy time, and show some quick wins by easing bottlenecks.

2. Next: Refactor or rewrite pieces of it gradually in stages.

Have you been in a similar situation? How did you handle it? Please comment below.

Periodically, "Tom" would have a raging row with the CEO. And in a fit of anger and emotion, Tom would resign.

After calming down, the CEO would concede to one of his demands. Tom would back down, and continue for a little longer.

Eventually the CEO decided this couldn't continue. The next time Tom resigned in anger, it would be final.

Tom headed the company's core engineering group. 40 people. He was a talented technologist, but not such a gifted manager.

Tom was extremely dedicated to his job and his people. They loved him. He was intimately familiar with the minutest of details in every project. Nothing moved without his approval — he was involved in all decisions. 

Extremely hands-on, he often did a lot of the challenging work himself. As you can imagine, he worked around the clock and hardly slept.

Anyhow, as you can guess, he had another row with the CEO. This time, the CEO was ready. He accepted Tom's resignation on the spot. There would be a price to pay, but the bill was due.

The CEO asked me to step in at short notice. My job would be to keep the department running smoothly ... without the only person who had seen the cover of the jigsaw box.

Challenge accepted!

CEO: "Oh, by the way, you need to fire 10 people by the end of the week — the department is too bloated. It costs too much and delivers too little."

Me, to myself: "As if I didn't already have a tough enough problem ..."

I said: "’Too bloated’ might well be true. However, I'll know where any potential bloat is only once I've figured out these 5 things:

1. What business outcomes is the group expected to support?

2. What is the group doing, and why?

3. In order to bring about the desired outcomes, what should the group be working on?

4. What is the appropriate size and structure needed in order to deliver the desired outcomes?

5. Who are the appropriate people with skills and domain knowledge that match the above size and structure?

The CEO turned out to be right! But he needed me to pinpoint the reason, and then to fix it. I even ended up letting a few more than 10 people go.

During the first week I discovered things that blew my mind, such as:

• A 5-person team dedicated to projects that had no commercial benefit.

• Excessively complicated solutions, requiring rare skills to develop and maintain.

• No cross-team communication, coordination or collaboration — it had all gone through Tom.

It took quite a few months to restructure and rebuild the department on a new set of values and behaviours.

Together, we had to:

• Process the change in leadership, as well as understand and learn to accept the reason 13 people were fired.

• Spark an environment of collaboration and teamwork, and boost employee morale. Changing an organisation's culture can be difficult, but in this instance they embraced it.

• Discover and piece together operational knowledge that until then had been only in Tom's head.

• Stabilise the high-maintenance operational environment — we worked systematically to reduce what was a constant stream of incidents, first to a slow trickle, and then down to to rare exceptions. This bought back time for uninterrupted and creative development, and this, too, boosted morale.

Within 6 months, the organisation was unrecognisable. Mainly the same people, but with smiles on their faces — more capable, and shouldering more responsibility. And most importantly, driving outcomes with clear business value.

The group could now function smoothly without me; I had even trained my replacement. Yet I was considered indispensable, but in a positive way — my replacement moved into my spot, and I was promoted to become CTO.

Conclusion

Don't be indispensable. At least, not in the way Tom was. If they simply cannot manage without you, you become a single point of failure.

That can be an exhausting situation, which has further disadvantages:

• If you're indispensable, you can't be promoted.

• You also can't be fired, but you'll be resented for it.

• Nobody is indispensable forever — it's temporary. Single points of failure must be eliminated.

The right way to be indispensable? — Be a catalyst for providing significant business value.

Be a CTO who thinks strategically; leverage your technical knowledge, product mindset and business acumen to collaborate with the other executives, and run engineering as a profit centre instead of a cost centre.

P.S.
Which of Tom's mistakes can you spot?

P.P.S.
As a manager, which of Tom's mistakes have you NOT made?

BTW:
Are you in Tom's predicament? — I'll help you fix it.

I saved a company from a ton of trouble, though.

My potential client, a fintech platform startup, wanted help producing a heavily customised "white label" version of their app for a strategic partner. Two separate partners, actually. Both are prestigious brand names. 

I take my hat off to them — amazing! But the client was blundering into a potentially problematic situation, and it couldn't end well unless I helped them change the parameters. 

They approached me as a CTO, so they were surprised by my non-technical questions:

Me: Are you charging these partners for the customised apps?

Client: Yes, one of them agreed to pay 500K up front.

(So far, so good.)

Me: How much will you be charging for routine maintenance? 

Client: No need. We'll be making money from standard usage fees.

(Hmmm 🤔)

Analysis of the Situation

Me: Let's think this through. You're essentially going to be maintaining a portfolio of customised apps, in addition to your core product.

• Multiple front-end apps to maintain — Each app will need constant maintenance for the lifetime of the contract. You will need to keep the apps in sync with your back-end platform, as well as keep up with phone operating system versions, security standards and other environmental concerns. In addition, you'll need to implement and test all your standard new features on each customised app too.

• Slower evolution and iteration — Oh, but wait! Your product is new! It should naturally evolve fast, and radically. Your partners may insist on approving each change, which could slow you down. 

• Multiple concurrent API versions — Having multiple front end apps could also mean that you'll have to support multiple API versions concurrently, which can add costly complexity.

• Core development costs increase — The cost, in manpower and time, of your core development will increase with each custom app and custom feature. This is because the quirks of each customised version have to be constantly taken into account, so that the core technology does not break them.

My Recommendations

Your partners are marquee customers; they are powerful and influential brands, and will lend you significant sales-boosting credibility. 

The challenge here is to prevent your focus and attention being stolen away from your core product.

Having handled such situations myself, I advise the following:

→ Try and find a way to get your partners to use your standard app — maybe with some extremely light cross-branding. 

But if you must white-label your app:

1. Charge (or earn) enough money to fund dedicated product management, engineering, and support resources for the custom apps. I will estimate the costs for you and help you set it up.

2. Ensure that your contracts allow you plenty of leeway to keep the custom apps in sync with your core platform without approval delays. I'll be happy to work with your lawyers on this.

Client: Hmmm. I'll take this up internally and get back to you.

(A week passes.)

Client: Our partners have agreed to use our standard app.

(👏)

Parting Thoughts

I gave free advice that made a potential client think, and it gave them the confidence to push back against strong partners.

I also saved them from hiring someone who would faithfully carry out their exact wishes. 🤦‍♂️

Am I foolish for evaporating this lucrative opportunity? Maybe — though in my mind it would have been unethical not to alert the client.

It is said that the best line of code is one that you don’t need to write. I am proud to have used my technical experience to recommend a non-technical way to solve a business problem. Examples such as this one capture much of the essence of being a CTO, which is first and foremost a business-oriented position.

Software development is creative and non-deterministic, and is full of:

• known unknowns

• unknown unknowns.

Command-and-control-style ("C2") companies cannot handle that:

• Internal and external deadlines that must be met!

• They must be in control of "What?" and "When?"

• Plans are communicated down the hierarchy.

• Progress is reported back up the ladder.

Agile software development

Agile development follows a set of principles that guide software development to successful outcomes:

• Scope is not fixed.

• Progress is incremental.

• Teams are highly autonomous.

• Experimentation is encouraged —

• Even though most experiments fail.

• Work is iterative; there is no "Do it right first time!"

• Continuous feedback and adjustment eventually wins.

There are at least 2 problems with this approach in C2 companies:

1. Many CTOs fear appearing incapable without predictable control over the development process, which is inherently non-deterministic.

2. C2 companies cannot swallow Agile without choking.

Scrum as a wrapper

Scrum is a framework that can be used as a wrapper for agile processes to make them palatable for C2 companies.

• Crisp, military cadence of sprints.

• Metrics can be generated, tracked & gamed.

• Efficiency can be measured and misinterpreted.

Scrum can make everything appear orderly and under control.

The CTO as a buffer

However, it does not have to be a complete illusion — Scrum is quite a fine method, if you have to use it.

There is an art to managing Scrum successfully to achieve good outcomes in a command-and-control company.

As a CTO or Engineering VP, you need to buffer the dissonance between the top-down diktats of executive management and the creative engineering process.

This means that the way you represent the process outwards is not the way it's experienced from the inside:

• Outwardly, you deliver the outcomes the company needs, representing Engineering as a well-oiled and reliable input-output machine.

• Inwardly, your engineering organisation is an island of agility that is insulated from the command-and-control culture.

How can a CTO or VP of Engineering do this?

• Interop API — Insulate the Engineering department from the command-and-control culture, allowing it to function in a truly agile way. Translate the top-down instructions into Scrum or Agile constructs, and vice-versa (see Bets).

• Alignment — use your leadership and communication skills to ensure that everyone is aligned, both within the Engineering department, and with other departments. Knowing what matters to other departments helps make better decisions at all levels.

• Bets — Call upon your experience to make reasonable bets about 1. the best direction to advance, and 2. what is likely to be delivered and by when.

• Risk Mitigation — Foresee, identify and monitor risks, and act early when things start going wrong.

Practical Advice

For practical ideas about reducing the dissonance and risks in order to succeed with Scrum + Agile in a C2 organisation, see these 3 articles:

✅ 1st key: actually start with a business goal.

Let’s consider this business goal example:

i.e. reach a situation where Customer Acquisition Cost (CAC) becomes lower than Customer Lifetime Value (LTV), i.e.

CAC <= LTV

Strategies that could support this goal:

1. Reduce CAC by 20%.

2. Increase LTV by 20%.

(Why 20% — no reason, but always define what to aim for.) 

2nd key: don't dive immediately into tech stuff; stay on the business side for a while …

Product Metrics

Before we jot down some relevant tech initiatives, let's consider some leading indicators that influence the above-mentioned lagging metrics, and are usually within the CTO's sphere of influence — at least partially.

We can usually influence leading indicators more directly, and they give us faster feedback.

• Bounce Rate — % of visitors who ditch upon arrival.
  Decreasing the bounce rate widens the top of the funnel, which should increase all of the numbers further down, i.e. potentially reduce the CAC.
  
  Of course, the landing page content must speak for itself, and that is usually not a CTO thing. But technical factors can bounce visitors even before they've been exposed to your captivating copy.
  

• Registration Rate — % of visitors who sign up (trial or free version).

  Increasing this value widens the middle of the funnel, and thus gives more people a chance to activate later on, i.e. potentially reduce the CAC.
  

• Activation Rate & Time to Activation — % of registered users who perform a specific meaningful action that indicates they are benefiting from the product, and the average time it takes until they do it. The more people who succeed in using the product and who do so in less time, the lower the CAC (potentially).
  

• Subscription Rate — % of registered users who become paying customers.

  Increasing the subscription rate reduces the CAC, because you're increasing the number of paying customers for the same ad spend.
  

• Sessions per Week/Month — The average number of times a registered user access the product. The subscription rate should rise with "stickiness", and so should lifetime value.
  

• Actions per Session — The average number of actions a registered user performs per session. The more useful the product, the higher the subscription rate, the lower the CAC and the higher the lifetime value.
  

• Referral Rate — % of registered users who refer the product to others. Each referred visitor is one we don't have to pay to bring, i.e. potentially reduces CAC.

Initiatives

With the above metrics in mind, we can now propose some very focused tech initiatives that will move the relevant dials in the right directions.

Initiatives that support "Reduce CAC by 20%"

1. Deploy an analytics tool and configure it to measure bounce rate, registration rate, etc.

2. Rework entry pages to achieve a Google Page Speed score above 90, in order to reduce the bounce rate.

3. Add social sign-up methods to increase registration rate.

4. Analyse, streamline and shorten user journeys that lead to activation.

5. Know who is an activated user and prompt activated users to subscribe.

6. Add a refer-a-friend feature; prompt and incentivise users to bring us more traffic.

7. Deploy a tool/service that automatically collects client-side error logs and reports, and act on them to increase the perception of quality and thus conversion and retention rates.

Initiatives that support "Increase LTV by 20%"

Initiatives 1 and 7 (above) are relevant, as well as these:

1. Email inactive subscribers and give them a compelling reason to log on.

2. Shorten user journeys, simplify actions and generally reduce friction, to increase the number of actions performed per session (and also the number of sessions).

Did you notice?

• Many of the above initiatives overlap with Product.

• A good CTO understands product too.

Wrapping Up

1. We took a business goal,

2. broke it down into strategies,

3. considered appropriate leading indicators (for early feedback),

4. and only then did we derive a list of supporting tech initiative candidates for selection.

These 5 non-technical questions are a litmus test.

They are not exhaustive, but they are designed either to be highly indicative of problems, or to justify, and even increase, your confidence in your CTO.

Note for CEOs: these questions are primarily for you to ask yourself, not the CTO. If you don't know, or cannot get the answer easily — that's a strong signal. These questions may naturally spark discussions with your CTO afterwards.

• Can the CTO articulate their technology strategy?

• Does it align with, and support, your business goals and long term vision?

• Is the CTO aware of your North Star business metrics?

• Can they tell you how engineering contributes to these?

• Can the CTO understand business concerns, and address them from a technical perspective?

• Can they communicate complex technical concepts to non-technical stakeholders?

• The selection of a technology stack can have a significant impact on the business.

  • How was the decision made?

  • Were you consulted?

• What do you know about its architecture? Is it microservices? Why?

• Which cloud services does your platform depend on?

• What languages is it written in?

• Your company's innovation is probably not the tech stack itself, so ensure that it is comprised of widely adopted components.

• How easy is it to hire engineers with suitable expertise?

  • How easy will it be in 3-5 years?

• Do you always know the answer to this question?

• Can you look it up by yourself and get a coherent answer?

• By the way, how does engineering know that these are the right things to work on?

"I have no clue how Engineering is performing." — many CEOs.

• There are lots of metrics, many of them useless or even counterproductive. Does your CTO know how to do it properly?

• Do your CTO report the statistics to you, like Sales and Marketing do?

• How long does it take to get a new feature into production?

• What percentage of deployments break things, requiring immediate remediation?

• What fraction of engineering resources are devoted to creating business value and moving the needle on your core metrics?

  • Do they routinely move the needle as much as expected?

The software engineer heard:

“Build me:

1. An iOS app, written in Swift;

2. An Android app, written in Kotlin;

3. And keep maintaining the web-based version of our app that is hosted on our website.”

This probable miscommunication is underpinned by a fundamental misconception.

What is a “Native” App?

The CEO’s goal is most likely to have a downloadable, self-contained app that can be launched directly from the phone's home screen, in contrast with functionality that has to be accessed via a web browser. This app could be expected to make the user log-in using a face-print or fingerprint, and receive push notifications (for example). Such apps are typically downloadable from Apple's App Store and Google's Play Store — but this can be more of a hindrance than an advantage.

You don't actually need a native app for this — at least not in the strict technical meaning of “native”. To a software engineer, “native” means “integrates tightly and exclusively with a specific operating system and/or specific hardware.” Our CEO almost definitely did not intend this.

There is an additional option, that practically speaking, is just as native. Native to the web. Web-native apps leverage web technologies and can run smoothly and securely on any modern general purpose operating system and device. Such apps can be completely indistinguishable from classic native apps.

I have worked with numerous companies that automatically assume that apps made with web technologies cannot be as good as classic native apps, and they make the costly mistake of fragmenting their front-end technology stack.

In 80% of cases, web technologies are a perfect match for what a business needs. When used correctly, browser engines provide a fast user interface layer, and they have APIs that provide much of the hardware integration functionality that was once reserved exclusively for classic native apps.  

Web technologies have graduated to become a set of standardised APIs provided by general purpose operating systems, that provide streamlined access to lower-level resources and hardware.

Multiple Apps or One App — Which is Better? 

If you take the classically native approach, you’ll likely build and maintain multiple apps — one for each target platform. These apps all do exactly the same thing, but are re-written in different languages, using different tools. You will have multiple codebases to manage, and probably a separate engineer, or team, for each one. Good luck when you try to roll out new features in a synchronised manner. Each one will have its own peculiar customer support issues, too.

Using web technologies, you can use a single family of technologies to build and maintain a single codebase that requires fewer engineers. Not only is it cheaper to build apps using this approach, but it’s also more suited for subsequent maintenance because of the more limited competencies required.

The same web app can be deployed to all these target platforms:

1. Web site

2. Progressive Web App (PWA) — a standalone web app, see below

3. iOS app, available in the App Store

4. Android app, available via the Play Store, or as a downloadable APK file

5. Windows desktop app

6. MacOS desktop app

What is a Progressive Web App?

A Progressive1 Web App (PWA) is a website that can put its launching icon on your phone’s home screen or your desktop’s dock or start menu, like any other app. When launched, it uses your browser’s underlying technology, but you don't necessarily see any of the browser’s “chrome”2 — user interface elements such as the address bar. 

You may not even be able to distinguish between a PWA you installed from a website and an app you installed from the App Store. My desktop Gmail and Google Calendar apps happen to be PWAs. They launch from MacOS’s dock or Windows’ taskbar into their own windows, which have no browser toolbar. Unlike individual browser tabs, they are each distinct entries in the series of apps I can cycle through using Command-Tab (MacOS) or Alt-Tab (Windows).

PWAs are far smaller in download size than corresponding classic native apps, and their updates can be instantaneous, too.

Steve Jobs Makes the Case for Web Apps

When the iPhone was launched in 2007, web apps were the official way to develop apps for the iPhone. Steve Jobs gave the best sales pitch for PWAs you’ll ever hear:

“You can write amazing Web 2.0 and AJAX3 apps that look exactly, and behave exactly like apps on the iPhone. 

And these apps can integrate perfectly with iPhone services; they can make a call; they can send an email; they can look up a location on Google Maps. 

After you write them, you have instant distribution. You don’t have to worry about distribution — just put them on your internet server. And they’re really easy to update — just change the code on your own server, rather than having to go through this really complex update process. 

And they’re secure, with the same kind of security you’d use for transactions with Amazon or a bank. And they run securely on the iPhone, so that they don't compromise its reliability or security. 

And guess what? — There’s no SDK4 that you need! You’ve got everything you need — if you know how to write apps — using the most modern web standards, to write amazing apps for the iPhone today.

Subscribe now

What Can PWAs Do Today?

PWAs can do most of the same things you expect from classic apps, for example:

• Install to the operating system's Home screen.

• Authenticate users using the device’s biometric capabilities.

• Use the device's camera and microphone.

• Detect the device’s location using GPS.

• Sense the device’s motion and orientation.

• Receive push notifications, even when the app is not active.

• Detect when the network is offline, and continue to function anyway.

• Create and access data files that persist between usage sessions.

• Make payments using pre-installed payment apps (Apple Pay / Google Pay).

• Share data with other apps, using the operating system’s Share Sheet.

• Establish real-time peer-to-peer communications (WebRTC).

These important features and enhancements are supported on Android but not (yet) on iOS:

• Enable a first class PWA installation experience from your website, including detecting whether your PWA is already installed and the ability to provide custom prompts to help the user install it.

• Register as a Share Target, i.e., appear on the Share Sheet as an app that can receive content that the user shares from other apps.

• Perform various operations in the background when the app is not even running.

• Register as a protocol handler — an app that can open specific types of links and file types.

• Home screen icon context menus.

• Interact with other devices via Bluetooth.

• Read and write to NFC tags.

• Prevent the screen from locking.

• Speech recognition.

You can find more details here:

• What PWA Can Do Today — install this web app on your device to discover and demonstrate your device’s supported capabilities.

• PWA Capabilities — a Google site for developers that thoroughly describes PWA capabilities.

The good news is that if you need one of the missing capabilities, there is a generally accepted workaround:

Deploying a PWA as a Hybrid App

If your mobile app needs a capability that is not yet implemented, you can easily create what is known as a hybrid app using a “wrapper” such as Capacitor. This creates a classic native app that encapsulates your web app, and also provides the missing functionality via available plugins. This approach will necessitate making your app available via Apple's app Store, and possibly Google's Play store, too.

On the desktop you have a variety of options such as Electron and Tauri.

App Stores are Cash Cows

Since launching the iPhone in 2007, Apple has strayed far from Steve Jobs's vision for standards-based apps. Apple could apparently not resist the lucrative temptation of creating a walled garden in iOS for which Apple itself serves as the gatekeeper. Android follows a similar model, ostensibly because Google would otherwise be leaving money on the table.

Thankfully, the web as a free and standards-based platform predates smartphones. Imagine a world in which Apple and Google decided which websites were “safe” to view on your mobile phone. The site owners would pay registration fees, and the content would be reviewed for compliance with the gatekeepers’ arbitrary policies. Any money changing hands would be automatically taxed by Apple or Google at 30%. You're almost there; Apple does exactly this with mobile apps. Google does this to a slightly lesser extent — Android allows you to permit installation of apps that don’t originate in Google’s Play Store, a method pejoratively termed “side-loading”.

Instead of expanding and deepening support for standards-based APIs, Apple and Google created platform-specific Software Development Kits (SDK) that allowed tight integration with each operating system. There are many benefits to this approach, but one of the trade-offs is reduced security and control. This conveniently necessitated that Apple and Google step in as gatekeepers in order to be able to evict bad actors — something they are notoriously bad at doing. The revenues they command are in return for their efforts to “create a safe environment”. 

Revenue Model

The Apple Developer Program generates hundreds of millions of dollars in annual membership fees of $99 or $299, paid by companies and individuals who host apps in the App Store (>2M apps). Google charges a one-time initial registration fee of $25.

In addition to this, Apple and Google charge up to 30% commission on sales of digital products and services. On iOS, apps must use Apple’s payment mechanism and are not allowed to link to an external website for processing payments. Google’s demands are not as strict, but they are in the same vein.

Out of $1.1 trillion billed by Apple App Store developers in 2022, $109 billion was from in-app advertising and $104 billion from digital goods and services. Apple therefore reaps tens of billions of dollars in annual income from the App Store. Google’s figures are smaller, but are still in the same order of magnitude.

Your Risks as a Vendor

When you use an app store as vendor, you assume significant additional risks to your app distribution operation. You are giving a third-party gatekeeper the power and authority to:

• Decide whether and when you can publish your app.

• Decide whether and when you can update your app.

• Remove your app from the store.

• Impose new rules or policies on your company and/or app.

In order to publish an app, you need to create a vendor account first. This involves submitting personal identification documents and company registration details5, as well as paying a fee.

After your account is approved, you need to request approval for your app. This can take a few weeks, even if there are no problems.

Subsequently, each update also needs to go through the approval process. Many hearts flutter during the waiting period because the different examiners don’t seem to interpret and apply the criteria in a consistent and predictable way, and there is no deadline.

Your app’s approval can also be summarily revoked at any time! The most sensitive time for this is when you request approval for an update, because this triggers a fresh examination of your app.

Additionally, at any time, the gatekeeper can introduce an arbitrary rule that you then have to race to comply with, with little recourse to challenge or appeal it.

In short, you do not control the timeline of releases or updates, and your app’s distribution is constantly at risk. Admittedly, the probability of experiencing a business disruption may be low, but its impact will be very high if disaster strikes.

Legal Challenges and Antitrust Regulation

By keeping PWA support alive after their 180 degree turn towards the App Store business model, Apple was able to say that their App Store is not a monopoly because there is an alternative way to install apps. This is probably why Apple has continued to maintain a passable level of support for PWA features, even if there are some glaringly obvious and plausibly deliberate gaps.

However, the success of this approach seems to be waning in the face of increased legal challenges from corporations and government regulators, who assert that Apple and Google use their app stores to exert illegal monopolistic control over the app market. 

Epic Games, makers of Fortnite, recently sued Apple and Google for operating illegal monopolies, in separate cases in the United States. Epic lost against Apple, but beat Google. In the mid-market, smaller companies such as 37signals, makers of Basecamp and the HEY Email and Calendar products, are periodically embroiled in David vs. Goliath skirmishes against Apple’s monopolistic gatekeeping, but only massive companies such as Epic can afford the legal fees to take this fight to the courts. At the lower end of the market are small companies that feel they have no option other than simply to submit to the gatekeepers’ heavy-handed directives.

Similar to Microsoft in the 1990s, Apple is increasingly falling afoul of antitrust regulators in the USA and Europe. Their strategy seems to be to drag things out for as long as possible, all the while raking in the revenue.

As if Steve Jobs’s sales pitch wasn’t enough, Apple recently signalled that it considers Progressive Web Apps a credible threat to their App Store revenue. In what appears to be malicious compliance with the European Union’s Digital Markets Act that forces gatekeepers to allow alternative app stores as well as alternative browser engines, Apple announced the removal, in the EU, of PWA support from the upcoming iOS 17.4 update. In response to uproar from the app development community, the EU’s European Commission requested clarifications from Apple prior to opening an investigation. This resulted in Apple promptly backing down from these plans.

In another saga, the US Department of Justice, along with 16 individual US states, has sued Apple for illegal monopolisation of smartphone markets, listing a whole slew of anticompetitive behaviours that “impose extraordinary costs on developers, businesses and consumers.”

This battle will take a few years to play out until the parties settle it or let the court decide. But the writing is on the wall: eventually, the anticompetitive behaviours will be stopped, either using current laws, or by enacting new ones, as is the case in Europe.

Conclusion

Whether to build a classic native app or a web-native app is not a purely technical decision — there are also significant business aspects to consider. Web-native apps are eminently viable, both functionally and technologically, as well as being a wise choice economically. 

The app store model imposes numerous risks on app companies’ distribution and revenues. App vendors should not follow the herd that accepts these risks as a given, but should instead apply rational judgement.

Apple’s and Google’s monopolistic behaviours confirm that Progressive Web Apps pose a credible threat to the app store ecosystem. The tide that is turning against gatekeeping monopolies ensures that Progressive Web Apps will be an increasingly viable alternative to classic native apps.

Progressive Web Apps deserve a bright future. They are based on a natural progression of the device- and vendor-independent protocols and standards that power the free web, such as HTTP, SSL, HTML, CSS, and Javascript. The level of support for Progressive Web Apps on the various platforms is allowing these platform-independent apps to become increasingly ubiquitous, if not the norm.

The best app store is the web!

You'll get (FREE):

• Tips that are pragmatic, jargon-free and hard-earned.

• Additional perspectives to those you see from your current position — if you’re technical, I’ll show you how the business side thinks, and vice-versa.

• Explanations and decision rationales I wish I could have used to convince my bosses and stakeholders in the past. 

I'm not starting from scratch, though. For the past year, I've been writing daily on LinkedIn and in addition, I have published quite a few stand-alone articles elsewhere.

These are the most popular articles (I’ve migrated them to this newsletter):

Upcoming Articles

Coming soon will be informative articles for people on both sides of the tech / business divide, on topics such as:

• My "… for CxOs" series, in which I show non-engineering executives exactly what they should know, and why they should care, about: GitHub, JIRA, Agile, technical debt etc.

• Practical tips and advice for CTOs and engineering heads.

• Advice for CEOs about how to get the best from their engineering organisations. (OK, it's mainly about what not to do.)

• How to scale organisations — it's harder than scaling technology.

Thanks!

I should have started this ages ago, and I want to thank , author of , for giving me the final push to accept Substack's invitation to get this newsletter up and running. Anton’s newsletter for engineering managers is packed with practical and thought-provoking tips and advice, and is peppered with personal anecdotes.

In addition, I’d like to recognise the following people for their inspiration, encouragement, and for setting an example I can only try to follow:

• author of

• author of

• author of

• author of

And finally …

Thank you for subscribing 👇

Either way, if you're unprepared, going viral is almost equivalent to suffering a DDoS attack. It will totally paralyse your infrastructure, costing you a lot of money and missed opportunities.

You wanted scalability? RESILIENCE is one of the prerequisites.

DDoS attacks are like the weather. You can't prevent them, but you can prepare. Make sure the roof doesn't leak. Wear multiple layers. And especially don't leave the front door open! Or the windows.

I have seen a pattern of clients who are "sophisticated" enough to use Kubernetes, but who still haven't implemented these simple best practices:

1. Cache static content and serve it from a CDN.

It's a waste to serve images and scripts from your back-end servers. Serve these using your cloud provider's content delivery network (CDN), e.g. Amazon CloudFront, or an external one such as Cloudflare.

Additional benefit: much snappier load-time for the end-users!

Additional details: 

1. You'll need to carefully tweak your HTTP headers to fine-tune the caching behaviour. You can get additional benefits from controlling the client-side browser cache too.

2. If your main HTML pages have to be generated dynamically per request, it's usually possible to rework things to make them static, with the dynamic component being generated on the client side. 

3. Been there, done that: talk to me if you're having trouble with this.

2. Use domain names, not fixed IP addresses.

• You won't be able to change your system's fixed IP addresses under fire. 

• Domain names and subdomains give you much more flexibility.

• Assign different subdomains to customer-facing systems and back-office systems — see next.  

3. Don't route internal requests through your public interface!

When one back-end service makes a call to another, don't route the request through the public interface! Use internal addresses. Apart from being a necessary security practice, it prevents your internal requests from having to compete with incoming traffic.

4. Use distinct infrastructure for handling customer-facing requests vs. back office systems.

If all requests reach your system via a common route that isn't resilient, you won't have access to your back-office systems when your customer-facing systems are overloaded. Ensure that you don't have a single point of failure.

It's not just your access to your back-office: what about things like payment confirmations that arrive in the form of callbacks by 3rd party providers to your webhooks? You need to ensure that these will be totally unaffected by heavy loads on your customer-facing systems.

Also:

• Maintain and enforce a whitelist that controls which parties can trip your webhooks. 

5. Use Cloudflare as your first line of defence.

Pay attention to these features:

• I'm Under Attack Mode

• Caching (see point 1 above)

• Rate limiting (part of Web Application Firewall)

Conclusion

Each of the above measures has a significant and independent impact, and they are almost free if you design them in from the beginning.

💡 Engineering for 99.5% uptime is more cost-effective than 99.99% for most startups!

"But why shouldn’t my engineers strive for excellence?"

Of course they should, but 99.99% guaranteed uptime cannot be achieved merely by doing things perfectly.  For that level of reliability, your engineers need to do things differently. You'll need specialised architecture, redundant infrastructure, and streamlined operational and organisational procedures.

It's unlikely to be cost-effective, unless you do the calculations and prove to yourself that your case warrants it. Most businesses do not need 99.999% ("five-nines") uptime, or even 99.99% ("four-nines"). 99.5% ("two-and-a-half nines") uptime is very decent. 

What's the big deal?

💡 A system with 99.99% guaranteed uptime must be 50 times(!) as reliable as one with "only" 99.5%.

This is easier to see when you compare the amount of downtime allowed: 0.01% vs. 0.50%.

How much does this cost?

The cost of building and operating a system in a way that guarantees 99.99% uptime is several times as expensive as 99.5%. This is in terms of system complexity, the number of engineers required, their specialisations, experience levels, and corresponding salaries, as well as significantly increased operational costs and arrangements.

What level of reliability should I aim for?

It's a business decision, not a technical one. Evaluate the effect of downtime on the business before even starting to discuss the technical implications.

How much does an hour of downtime cost you?

Ask yourself these questions:

• If you have an e-commerce platform, how much revenue (or dare I say it, profit) does it make per hour?

• How much downtime can your customers tolerate?

• How much damage will an hour of downtime do? This can be both due to direct loss of revenue and also  due to eventual customer churn due to loss of trust.

• Bonus question: are all hours of downtime equal, or are some hours less costly, such as weekends or after midnight?

How much downtime can you tolerate?

These are the total downtimes you can expect for corresponding uptime guarantees:

Operational & Organisational Aspects

The limitations on your uptime are most likely operational, not technical. You typically have a higher risk of downtime due to operational issues than due to technical problems. There may be fewer things that can go wrong, but when they do, it takes many hours to fix them.

Let's assume that a system — engineered for 99.99% guaranteed uptime — dropped from the sky into your lap. On a technological level, such a system will have redundancies to ensure that there are no single points of failure. However, single points of failure do not have to be technical; they can also be administrative.

💡 A credit card can be a single point of failure.

I have seen operational accounts frozen because a credit card went over its spending limit. How many hours will it take for you to first notice that there's a serious problem, identify that the cause is not technical, and then actually rectify it? A single such event will destroy your 99.99% uptime record.

• Are you organised to prevent such "unplanned administrative downtime"? It's not trivial, because you need to have cross-departmental plans and procedures both to prevent, and to handle such events.

• Do you have a 24x7 rota of engineers? 

• Do you have monitoring and alerting that will activate your 24x7 support when an error rate threshold is crossed? 

Scheduled downtime for maintenance is an additional operational point to consider. Do you allow this at all? If you do, should it fit within the downtime parameters you allow yourself? What are the risks of it lasting longer than planned? The solution may be technical, but the decision is a business one.

Infrastructure & SaaS Provider Guarantees

"The infrastructure I use guarantees 99.xx% uptime."

Let's unpack this sentence. Consider Amazon Web Services (AWS), for example.

• Redundancy — Amazon's Service Level Agreement ("SLA") for EC2 specifies 99.5% uptime for a single machine instance. By running multiple redundant machines you can increase this, but what if the EC2 failure affects multiple machines?

• Availability zones = better redundancy — The answer is to have a redundant system running in at least one other "availability zone". Each zone is geographically separated, has an independent power supply, independent internet connection etc. Running two machines, one in each of two such independent availability zones, allows you to multiply the probabilities and get 99.75% uptime. In reality, Amazon's SLA actually specifies 99.99% for this combination.1

• Downtime is cumulative across services — You'll use multiple different AWS services (e.g. EC2, ELB, S3, CloudFront). Each service's SLA specifies its own uptime. The downtime is cumulative from a probability standpoint. This means that even if you use four services, each having a 99.99% uptime SLA, you'll have a common uptime expectation of 99.96%.2 This is not "almost 99.99%" — it gives 4 times as much downtime! Therefore, if you are using multiple services, your cloud provider's uptime SLA for any one service should be considered to be an upper limit for your whole system.

What "guarantee" means to you vs. cloud providers

In addition to the back-of-the-envelope maths above, we need to consider what is meant by the term "guarantee". You probably will not find this word in any service level agreement. Cloud providers promise to make "commercially reasonable efforts" to make their services available with a specified uptime percentage. If they don't meet this in a specific month, they agree (typically upon your explicit request) to deduct a pro-rated amount from your bill, corresponding to the duration of the outage or a fixed percentage based on the actual range of uptime they delivered. The cloud providers do not reimburse you for your actual losses.3 So based on your monthly spend, how much is a cloud provider's 99.99% SLA worth to you?

Other providers in your SaaS supply chain

Do you have other providers in your supply chain, such as GitHub, npm, and CircleCI? They might not be critical for production, but any problems there can delay deployment of a fix to production.

Your own code must meet your uptime requirements too!

Your own code runs on top of the infrastructure we discussed above. If you are aiming for 99.99%, your own code has to be part of this. Is your own QA and load testing thorough enough to guarantee 99.99% uptime of your own code, even without considering the reliability of the infrastructure layer?

Wrapping Up

• 99.5% uptime is reasonably achievable without excessive costs and distractions. Anything more costs a lot more.

• The good news is that if your own system is constructed well, your actual uptime will likely be significantly better than this, because the providers try to do better than what they promise. It's the guarantee that's expensive, not the typical result.

• Each layer of your system needs to have a higher reliability, in isolation, than the final uptime you're aiming for, because error rates are cumulative.

• Before spending a lot of money, time, and resources on development, upgrade your operational capabilities and administrative procedures.

• If any individual measures (e.g. 24x7x365 on-call support) are not achievable, that doesn't mean you should give up. Do the best with what you can afford.

• Hopefully you have gained a fresh perspective that will help you decide what you really need, and be conscious of where your real risks are.

(Test your CTO 😉)

1. Real users are unpredictable and full of surprises. 

This is the why of Agile — the reason it exists.

You (CEO) are an expert in your industry.

You have used your unique insights to build a product.

But something unexpected happens when people see or use it.

• They can't work out how to use it.

• It solves a problem they don't have.

• It doesn't solve a problem they do have.

• It solves their problem, but causes another.

• It solves a problem that you didn't know about.

• They don't want to use it the way you intended.

• They love the idea of it, but never actually use it.

• It does its job, but they hate it and can't explain why.

OK, not all of these, but at least one.

Get over it — nobody has solved this yet.

But Agile is an effective workaround.

2. You get more things right, faster, by taking small risks. 

Getting your product right first time is possible, but unlikely.

It's a large investment riding on single spin of a roulette wheel.

Instead, the Agile approach is to develop small slices of functionality, collect feedback, adjust and repeat. You make consistent incremental progress, using tight trial-and-error feedback cycles.

Smaller stakes, but multiple spins.

But unlike roulette, your knowledge of previous spins gives you an edge. 😎

3. Prioritise face-to-face collaboration with stakeholders, and especially, real users. 

It is unlikely that you (CEO) know exactly what to build and how, because of point zero: "Real users are weird and unpredictable".

But you do have a vision. Share it with your product managers and engineers. They will collaborate (not just talk) with stakeholders, including Real Users™, and will figure out the best path to realising your vision — one small step at a time.

4. Changes are expected and welcome at any time!

Agile expects requirements to change constantly. However, I'm not referring to your (CEO) whims — I'm talking about changes that are based on real world feedback.

5. Agile is challenging for many CEOs.

a. Autonomy & Trust

You'll need to entrust product managers and engineers with a high degree of decision-making autonomy (see point 3).

b. Minimalism

Agile has a minimalist perspective.

It can be tough to decide what NOT to do.

Or at least, what not to do YET.

If you just tested your CTO on the above, don't be too hard on them.

They might not have summed it up as casually as this, picking out only the bits that CEOs should know. (If they mentioned roulette, give them full marks, because they must have read this post.)

However, this is what I hope they told you:

6. What problem does Agile solve?

I once worked for a consulting firm that catered to large organizations. We spent weeks analysing a client's needs, then wrote a formal specification — a thick book. They signed it, and we spent months building the system. When development was complete, it slammed into its first real users. They would complain that it was out of date, or even useless. It took many further months to process and deliver formal change requests until the system became somewhat acceptable.

This process was too rigid and was highly resistant to changes in the specification.

Problems were discovered late, making them expensive to fix.

Such a processes have been dubbed "waterfall" — a linear sequence of steps that must be followed in order, to completion. Each step has a beginning and an end, and the output of one step is the input of the next.

In this case (simplifying): analysis → design → development → testing → deployment.

Waterfall works well for construction and manufacturing, but not for software development.

It is too expensive and inefficient in terms of effort, time and money, because we cannot get most software features "right" first time.

We need something that is less formal, more flexible, makes our inevitable mistakes cheaper, and suceeds faster.

In one word: "Agile".

7. What is the essence of Agile?

Agile is an iterative, cyclical process.

We reduce up-front investment to a minimum.

We also progress in small increments, which compound fast.

Agile's feedback loop is tight, measured in days / weeks, not months.

When we get something wrong, we find out early, and it's cheaper to fix.

Agile prefers:

• Individuals and interactions over processes and tools

• Working software over comprehensive documentation

• Customer collaboration over contract negotiation

• Responding to change over following a plan

Key principles in Agile:

• Cadence
  Deliver working software early, frequently, and at a constant, sustainable pace.
  

• Collaboration

  Software developers must collaborate face to face with business people.
  

• Autonomy

  Trust motivated people to get the job done, acting in highly autonomous teams.
  

• Reflection
  To increase the effectiveness of the process, reflect and adjust your behaviour at regular intervals.
  

• Progress
  The primary measure of progress is working software.
  

• Change
  Requirements will change — embrace this.

• Simplicity
  Maximise the work you decide not to do.

• Excellence

  Technical excellence and good design enhance agility.
  

Some of these principles can be hard for management; others are challenging for engineers. But it's worthwhile to overcome the difficulties.

8. Agile Vocabulary that you'll hear

You'll hear these words — let me dispel any mystery that surrounds them:

Scrum, Kanban — agile ways of organising software development work. (If your engineers are too insistent on dogmatic adherence to these mystical ideologies, send them to me.)

Sprint — another name for an iteration or cycle. It's essentially a team-level unit of planning and execution.

Story — describes a *tangible*, incremental unit of functionality that can be implemented and delivered independently.

CI/CD — continuous integration and continuous deployment allow features to be deployed to production as soon as they are completed. (FYI Continuous Deployment is like teenage sex: everyone talks about it, but only a few know what they are doing.)

Feature Flags — allow deployed features to be released in a controlled way to subsets of users, in order to gather feedback. With feature flags, deployed does not necessarily mean released -- that's a good thing!

Imagine that your product planning and software development priorities are tactical and feature-driven, rather than strategic.

You have clever, capable and diligent product managers and software developers. They are sweating away with picks and shovels, steadily chipping away at your mountainous product backlog. Your department works like a well-oiled machine, processing tickets and delivering new features by the dozen, on time and to specification, consistently.

💡 You're running a feature factory; optimising for output, not outcome.

Factory assembly lines crank out identical units, each of which has an identical business value, which is also well-defined. It's a mistake to think of your software development operation in the same way, because:

• An assembly line optimises for process efficiency, not product effectiveness.

• Different features can contribute wildly differing amounts of business value.

• Features are not identical, so you invent arbitrary prioritisation criteria.

• Most of your features will not improve any business metric.

• Effort and profit are not directly related.

The problem with a feature factory is that the focus is on fulfilling requests, rather than improving core business metrics. The constant stream of requests encourages you to optimise for efficiency rather than effectiveness. Insufficient consideration is given to the business value of the features being churned out.

Solution

The solution is to optimise the process to maximise business value. Business value needs to become a thread that runs throughout the whole process. 

The essence of the solution is to clarify what the company wants to achieve, and then to allocate the bulk of the development budget towards that, and little else.

Make business impact the driving force behind everything you do, by adopting these principles:

• Core metrics: tie as much work as possible to the company's core business metrics.

• Roadmaps: instead of detailed lists of features, express business outcomes that will be achieved and/or initiatives that the company will focus on.

• Prioritise work based on expected business impact, using the roadmap or core metrics.

• Budgets: keep different types of work within pre-determined budgets.

• Productivity: base your productivity metrics on business impact.

These principles are simple to grasp. However, implementing them requires organisational finesse, patience and perseverance.

I'll provide a mechanical solution to get you started, but the major breakthrough will happen when you notice that your company's mindset has changed.

How I Introduced Budgeting — the Trojan Horse Method

An evolutionary approach usually beats a revolutionary one. However, when you look back after multiple steps, there's no question that a revolution has taken place. This is the story of how I quietly and gradually introduced a new way of thinking, without declaring a revolution.

We were a sales-led company. Some of our large customers routinely leveraged their size to convince us to implement special features for them. Such behaviour is a fact of life at many software companies.

I started tracking the development costs of these Customer Specials in each development cycle, and I made sure to keep management aware of both the direct costs and the opportunity costs.1

The amount of attention we were devoting to Customer Specials, and diverting away from the general customer base, seemed excessive. After a few cycles, it wasn’t a big leap to get management to agree to limit the effort we could expend on Customer Specials by allocating a specific effort budget.

💡 Yes, I slipped the word budget in quite deliberately. Executives are accustomed to budgets, although this use stretches the concept a little.

OK, I admit it — I've understated the problem somewhat. It wasn't just Customer Specials —  my department had become a feature factory and our way of working needed a serious shake-up.

Once I had wedged open the door using Customer Specials, I widened the gap by sneaking in another 6 categories. Everything we did  from that point onwards was assigned a business impact category.

Later, I took it yet another step further, and with the cooperation and active participation of the rest of the executive team, organised quarterly planning meetings — more on this below.

The 7 Business Impact Categories 

I recommend adding a field to your issue tracking system, and/or tagging your issues using the following business impact categories:

1. Basics
   Features that are so basic that the product is not viable without them. New products will have these features; mature products won't.
   

2. Incrementals
   Continuous improvements to existing features. You need these in order to retain your customers.
   

3. Differentiators
   Unique features which distinguish your product from the competition. Alternatively, improvements which are game-changing by an order of magnitude. These are what the business needs to win!
   

4. Speculative Bets
   Potential differentiators, but without any reliable evidence.
   

5. Technological Foundation
   Infrastructure improvements for scalability, performance, reliability, maintainability, etc. This includes paying down historical2 technical debt.
   

6. Embarrassments
   Fixes for embarrassing mishaps that make the product or company look unprofessional (“broken windows”).
   

7. Customer Specials
   Work done for a specific customer which you would not otherwise do, or at least not now.

How to Use the Business Impact Categories

You can use the business impact categories in two independently useful ways: retrospectively and proactively.

1. Analyse all the work that you've delivered recently.

   
   How was effort distributed across your business impact categories?

   How much effort did you waste on things that don't "move the needle"?

   
   These results show how well your recent output reflects your company's business goals. If the results are not satisfactory, you can use them to get management buy-in to change the planning process so that it aligns with the company's goals — see next.
   

2. Use the business impact categories to align your plan with company business goals.

   
   When planning, allocate an effort budget per business impact category, based on what it will take to achieve or support the company's business goals for that period. This can be done at multiple levels such as quarterly planning down to planning individual sprints.

In How to Deliver Business Value Predictably, I referred to effort budgets as an advanced planning technique. To stretch the aviation metaphor I have been using, the business impact categories would be different "fuel tanks", designated for specific uses.

Initiatives

In addition to the improved focus brought by business impact categories, you can define initiatives as a high level way to drive your planning.

Back to my story. After several months of working with business impact categories, we were ready for the next step: quarterly planning.

Rather than a typically useless presentation-fest, the company's executive management team and other key figures met with the product and engineering organisation for an intensive series of interactive discussions over a short two-day period.

During this time we aligned our departmental plans fully, and produced a list of initiatives that we would work on. Each initiative was allocated an effort budget, based on our teams' capacities, growth plans and hiring forecasts.

😎 For the first time, and from this point onwards, each engineer would be able to know exactly how s/he would be contributing to the company's goals.

Whether or not you do quarterly planning, and regardless of the method you use, you should decide on a periodic basis:

• What business problems does the company want to solve?

• Which business metrics does the company want to improve?

• What is it worth to the company to achieve each of these these goals? 

• How many people and how much time can the company afford to dedicate to each?

Think in terms of initiatives rather than features. Decide how much attention you would like to give to each initiative — this will be its effort budget.3

Features should be invented to serve specific initiatives and their goals. They should then be sliced and worked on within the effort budgets. Later, you'll be able to measure their success against those goals, too.

Initiatives vs. Impact Categories

We started with business impact categories, and then graduated to initiatives. Does this mean we no longer need those business impact categories?

Initiatives and business impact categories can be used independently of each other. They can also both be used at the same time, because they are orthogonal to each other.

If you have graduated to initiatives, I still recommend using business impact categories to ensure that you are balancing the different types of work, and are not starving some types of activity.

Recommendations

1. Allocate as much effort as possible to Differentiators.

2. Next come Incrementals.

3. Labelling your Speculative Bets will help you ensure that you make enough of them to succeed, but also keep them within sane levels. Don't expect more than a 15% success rate for these.

4. Customer Specials and Embarrassments should ideally need zero attention. Good luck with that.

5. Allocate a fixed budget for Tech Foundation. It depends on the maturity of your platform, but I would suggest anchoring your opening bid at 25%.

• Deeper work

• Fewer meetings

• Fewer interruptions

• Increased productivity

• Self-sufficient employees

• High quality decision-making

These are achieved through: 

• Clear, written communications.

Is your time valuable? So is everyone else's.

💡 Writing-centric organisations cultivate a consciousness that everyone's time is valuable.

By prioritising writing, you prioritise thinking.

Clarity — Requires you to think and organise your thoughts.

Consistency — Writing tells everyone *exactly* the same thing.

Thoroughness — Helps communicate while raising fewer questions.

Being a "writing-first" organisation does not mean writing-only.

But it does mean investing in the written word, wherever possible.

And I don't mean monosyllabic one-liners in Slack!

Writing-first organisations tend to look like this:

📆 Fewer meetings

• Meetings are focused and productive. 

• Meetings have written agendas and clear goals.

• No meetings just for “status updates” that should be emails or group posts instead. 

💥 Fewer interruptions 

• Read messages and documents at *your* convenience.

• Give people documents instead of private lessons.

• You don’t have to answer questions continually.

• You don't have to wait constantly for answers.

• Get all the pertinent information in one shot.

👩‍⚖️ Quality decisions

• Decisions are based on thoughtful analyses and written arguments, rather than heat-of-the-moment emotions, or shoot-from-the-hip ideas.

• Decisions are written down, along with the rationale behind them. This makes them easier to communicate clearly. (Some might not view that as a good thing, but this also enables decisions to be revisited later. Many times I have wracked my brains why a decision sounded good at the time.)

🤔 Deep Work

• Fewer meetings + fewer interruptions → Deep Work

• Deep work in a state of flow is where breakthroughs happen. Every interruption or switch of context requires another 20+ minutes of concentration just to get back in "the zone".

📄 Documented procedures 

• Procedures are documented, and don't just exist in some people's heads.

• Everything is searchable and discoverable.

🕺 Independent, self-sufficient employees

• Well written and comprehensive documentation allows present and future employees to become more self-sufficient, quicker.

• Knowledge is democratised across the organisation. This is a double-win, because anyone can contribute too.

Q: Why is this graph exponential and not linear?

A: Because of Aha! moments and breakthroughs that become more common as more people think more deeply, more often.

Things you can do today:

👉 When writing an email or document, consider:

• How will this piece of writing be used? 

• What questions might people ask?

• What are the main takeaways? 

• Who is the target audience?

• What is the main goal? 

👉 If appropriate, share an online document in preference to writing everything in an email. 

• Shared documents are easier to comment on. 

• Additional participants can be added without having to forward them long email chains. 

• New employees will automatically gain access.

👉 When scheduling a meeting, share its purpose and agenda.

👉 Reject meeting invitations that don't make these clear.

If you have influence over the systems you use:

👉 Use a shared knowledge management system, e.g. Notion, Nuclino, Confluence, ...

If you have influence over the hiring process:

👉 Test candidates' written communication skills during the recruiting process.

Most of us are wrong, most of the time.

It's just the nature of the beast.

This makes what I have to say next much worse.

Another rule, which I discovered when managing a feature factory:

💡 Each feature you add to a product increases the cost of every subsequent feature.

Yes, each newer feature will take more people, longer to build. 😱

That feature you added just to close a deal? — It made all future features more expensive, so I doubt you charged enough for it!

You've heard of these 2 types of software development costs:

1. Direct costs (people x time x salaries)

2. Opportunity costs — saying yes to one thing means saying no to other possibilities.

Above, we just met a third cost, and it's hidden:

3. Complexity cost

Complexity cost is NOT another name for technical debt! That's the fourth type of cost, also hidden.

This article goes into detail on tech debt:

As you increase the scope of a product, and the number of features that interact with each other, it becomes more expensive to make any further change. Any change affects more parts of an increasingly large and more complex system. There are more considerations that need to be taken into account when designing, implementing, testing, troubleshooting and deploying the feature.

Operating and supporting a larger, more complex product costs more, too.

Therefore:

1. Be sure about the business value each new feature is supposed to bring.

2. When an experimental feature doesn't deliver results, remove it.

3. In fact, remove all features that don't contribute value.

This is much easier to say than it is to do.

Because of the 85% rule, we don't know anything in advance.

Therefore, we need to create a culture in which we routinely experiment, measure and cut.

To do it right, each experiment should have these two phases:

Phase 1: Experiment

1. Define or select a problem to solve.

2. Design a feature that you hope will solve it.

3. Define how you will measure it, and the success criteria.

4. Implement in a quick-and-dirty way if necessary.

Phase 2: Clean Up

• Failure: remove the feature or tweak it.

• Success: pay back the tech debt incurred in phase 1. If you don't pay down your tech debt, your costs will spiral out of control.

• It can make you bankrupt.

• At least one type of technical debt is good.

Technical Debt is a software engineering concept, and is defined as follows:

Technical debt is is a measure of the amount of duct tape holding your system together, plus the amount of rust1 that it has accumulated.

Duct tape represents improvisations, which are present in every software system, as a result of:

• Pressure to produce too fast, and not doing things properly.

• Quick fixes that were needed to "plug leaks".

• Experimenting with various features before deciding whether to pursue them.

Rust: Even a well designed and implemented software system will not continue working forever.

It will slowly degrade, as if it is rusting. This is because it doesn't exist in a vacuum.

• Business processes change and assumptions break. The world is constantly changing, and businesses must adapt to survive. Software is designed and written based on various assumptions. These assumptions gradually become invalid. The gap between the current situation and the ideal situation will tend to widen steadily.

• Integrations with third parties change over time. Changes are required to keep up with evolving third party APIs, cloud infrastructure and security patches.

• Scaling issues. As the business grows, the system may need to handle significantly more traffic, and perform an order of magnitude more operations. As a result, its performance may drop below acceptable levels.

What does not count as technical debt?

— Bugs.

Any functioning software system starts accumulating technical debt from the moment it is born.

Whether or not you decide to pay back the principal, the interest is deducted automatically on a regular basis, in terms of the following:

• Lost productivity — it takes longer to deliver new features.

• Lost business — due to breakdowns, missing features, bad performance, security issues.

If you let technical debt get out of control, there can be additional consequences:

• Employee churn — It is frustrating to work on an ageing system that is not well maintained, has frequent breakdowns and emergencies, and is generally perceived as being disrespected by the business. In addition, opportunities to learn and grow professionally in such an environment may be perceived as being limited.

• Default or bankruptcy — I was recently asked to help a company that had let their technical debt get so out of control that they were seriously considering rewriting their whole system. In a different language. In the business world, this would be equivalent to defaulting on a loan, or even going bankrupt.

Strategies for controlling technical debt

• Clean up after experiments — It's fine to go into tech debt when experimenting with new features. Not just fine — I actually recommend2 it. However, after the experimental phase you must replace the duct tape with properly engineered solutions, or remove it altogether.

• Budget for it — Allocate a constant, fixed percentage of engineering capacity in each sprint, or planning period, for the express purpose of paying down technical debt. I usually frame it at 25% and negotiate from there.

• Track technical debt — Use your issue tracking system (Jira, Linear, etc.) to tag and track technical debt. 

• Identify technical debt hotspots — Use your source code control system (GitHub, GitLab, etc.) to identify technical debt hotspots. These are modules where there are a lot of frequent fixes that may indicate instability or too much improvisation.

When you address technical debt as a matter of routine, the curve looks like this:

"Just throw it over the wall"

Is that how hand-offs happen in your company?

When your engineers start work, do they find:

😡 Product questions that need to be resolved?

😤 Requirements that are not fully cooked?

The good news is:

😎 Once you've batted the issue back over to the other side, you'll get a few days' quiet until it comes back.

Playing ping-pong with requirements is common in many organisations. It delays delivery, and lengthens the lifespan of issues (lead time or cycle time).

Don't attempt to fix this by instituting formal definitions and templates. This bureaucracy will help in the worst cases, where the problem is a lack of diligence. However, if your product managers are reasonably competent, this will make things worse.

The fundamental problem is usually a lack of collaboration. Requirements should not be prepared in a vacuum and then handed off to the engineering team, waterfall-style. Instead, they should be formulated in close collaboration with engineering. With good collaboration, there is no "other side of the wall" — you're all batting for the same team.

Handing something off between one team and the next should be an overlapping process. Just as the product manager is involved, informed and aware of issues during the implementation process, relevant engineers should be involved during appropriate phases of the requirements process.

Product Managers: 

• Start cooking requirements well in advance.

• Give requirements a longer incubation period.

• No handed-off requirement should surprise engineering.

• Share with engineering well before you write formal requirements.

Engineers:

• Ensure that nothing handed off from product should surprise you.

• Don't expect perfect requirements to fall into your lap, waterfall-style.

• Help product understand the challenges and trade-offs you might face.

• Help product formulate requirements that you can realistically implement.

The way you choose to implement this will depend very much on your existing processes and company culture.

For the more strategic features or complex implementations, I start off with a Brief. That's my fancy name for a brain-dump, and it is therefore a rather informal document.

The 3 Goals of Briefs:

1. Inform people about what will soon be coming down the pipeline.

2.  Serve as a focal point for getting input and especially questions.

3. Attempt to uncover hidden problems before development starts.

Brief Structure:

I use a minimal structure, and modify it based on common sense.

1. Background

2. Problem

3. Proposed Solution

Anyone can comment on a brief, though I typically also designate specific people to reserve time to review it.

Remember, it's an iterative process. You'll need to keep the discussion alive through the iterations.

After a few volleys of comments, revisions, and occasionally synchronous discussions, you may find that the process achieves one of the following results: 

• Great solution! This is how we'll need to implement it.

• Great problem! Not so great a solution. Here's a better one.

• The problem is not the problem. We need to solve a different problem.

Photo by Fleur on Unsplash

“Features are taking far too long to ship — how can I fix it?“ This is one of the most common problems I have encountered. 

It is actually one of my favourites, because there is typically plenty that can be done. Although it can take months to solve, the first results can show up within a few short weeks.

It is rare that the problem is due to a single cause. It is usually a combination of factors.

However, we opened with a subjective feeling, so the first step is to understand whether it is justified.

Behavioural Patterns & Indicators

The ticketing system (e.g., Jira) is a goldmine of behavioural data that shows how the organisation operates.

We are primarily looking for repeating patterns of behaviour, not isolated incidents. These patterns are indicators of possible problems, and are not necessarily the problems themselves.

Here is a breakdown of common indicators that are not difficult to discover. At this stage, I'm trying to be non-judgemental, but as you'll see, some judgement inevitably leaks through.

• Bumped Stories — These are stories that got bumped to the next sprint, sometimes repeatedly. I sort these into three main categories:
  

  • Cliff Hangers — These are stories that were completed, but not deployed, and got bumped to the next sprint. This type of thing borders on the criminal, and almost always comes with an interesting (but sad) story.
    

  • Unfinished Masterpieces — These are stories that were started, but got bumped to the next sprint for seemingly trivial reasons, sometimes repeatedly. This is where most of the meat usually is.

    • Waiting for someone on another team, or in another department, to do something (e.g. design, translate, approve, register, pay, …)

    • Waiting for someone on the same team to do something (e.g., review and approve the code, implement code review feedback, …)

    • Waiting for QA validation — depending on how the engineering department is structured, this could belong under 1 or 2, but it really deserves its own entry.

    • Failed QA validation, but not repaired and/or resubmitted for validation.

    • Stuck because of a dependency on another story.
      

  • Writer’s Block —  These are stories that were not even started, and got bumped to the next sprint, sometimes repeatedly.
    

• Stowaways — These are stories that somehow sneaked into the sprint after it started. There are 3 common types:

  1. Production or support issues.

  2. “The Boss wants it done immediately!”

  3. No apparent reason.
     

• Overweight Baggage — These are stories that are just taking far longer than the original estimates, and were not finished in time.

  1. Unanticipated complexities were uncovered during implementation.

  2. Stories that should have been epics, i.e. too large in scope.

  3. Effort is consistently underestimated.

  4. Small stories have unexpectedly high estimates.

Actionable Causes

The causes of each of the above issues are typically a combination of the following common actionable causes. This is where I'm being very judgemental!

• Overbooking — Too many stories are scheduled for the sprint. This can be because:

  • Engineering throughput is uncalculated, or does not reflect reality. 

  • QA capacity does not match planned engineering throughput, violating the Theory of Constraints.

  • Too much negotiation and pressure on engineering regarding story size estimates.

  • "To make sure the engineers always have plenty of work to do."
    

• Stowaways — High priority support incidents and/or other "urgent" work enters the sprint late and displaces scheduled tasks, because sprint planning did not allow for an average quantity of incidents.
  

• Process not respected by management — The engineering organisation's method of planning and executing has not been communicated to, and fully embraced by, upper management. Upper management is not aware of the importance of respecting the process, and routinely introduce noise in the form of Stowaways. Product management and engineering have not developed the ability to push back on such requests.
  

• Requirements are not ready — Product requirements are prepared in a vacuum, i.e. "thrown over the wall" without collaborating with engineering. This inevitably causes ping-pong while the engineers try to digest the requirements and ask questions that the product manager didn't think of.
  

• Technical debt — There is no deliberate plan to track and manage technical debt, and it keeps accumulating. The effort needed for every engineering task increases with the level of technical debt.
  

• Features are prioritised inconsistently — Work is prioritised based on who shouts the loudest rather than by business impact.
  

• Features or support issues are prioritised ineffectively — Too much work is classified as highest priority. I have seen one case where there were 3 levels of "Highest" priority: Highest-Highest, Highest-High, and Highest-Medium (even they thought that Highest-Low would have been too much of a joke).
  

• Stories are not small enough and/or are sliced incorrectly.
  

• Effort is underestimated — Analysis by engineers is not thorough enough when estimating the effort required, or (far worse) there is undue pressure to reduce the estimates.
  

• Suboptimal ordering of work — The order of execution does not attempt to reduce risk and maximise impact early, and to take advantage of QA throughput.
  

• Code reviews cause delays — Code waits too long for review by tech leads or senior engineers. In some cases it cannot be fixed up and tested in time for release.
  

• QA is a bottleneck — QA can cause delays for a number of reasons, especially: 

  • QA are not part of the product/engineering collaboration that generates the stories.

  • Lack of automation that makes it take a long time to set up complex testing environments.

  • Understaffing.
    

• Cross-team dependencies — Too many cross team dependencies, or even a few dependencies that are not aligned, are problematic. Different teams naturally have a different plan, priorities and cadence. Waiting for someone on another team can cause delays, or at least unpredictable timing.
  

• External bureaucratic processes — Depending on a clumsy bureaucratic process that is managed outside the team and has a mind of its own can drive you crazy, unless you either get it changed or allow for it in your plans.
  

• Faulty implementations — Implementaions can be imperfect for many reasons, especially ineffective definition of requirements by a product manager and lack of engineering professionalism.
  

• No distinction between deployment and release — Deployment of some stories is artificially delayed because deploying a feature necessarily releases it; feature flags are not used.
  

• Migrations cause downtime — Deployment of some stories is delayed for operational reasons, because downtime is required to migrate the database structure.

Root Causes

The above "actionable causes" are easy targets when focusing on fixing the process. However, the real root causes are much more systemic. Some organisations have the culture, maturity, self-awareness and desire to recognise and treat these systemic issues. It is not trivial, but is quite doable with willing partners.

• Lack of alignment between departments, especially:

  • Ineffective communication.

  • Processes not aligned.

  • Insufficient collaboration at relevant stages.

• Management deficiencies, especially:

  • Ineffective planning & delivery management on macro and/or micro levels.

  • Unsuitable organisational structure.

  • Weak management, lacking backbone.

  • Following the bureaucratic steps of the process without understanding its essence.

• Execution deficiencies, e.g.:

  • Professional abilities are lacking, e.g. defining requirements, slicing stories.

  • Professionalism is lacking, i.e. lack of diligence and attention to detail.

  • Lack of appropriate tools, e.g. automated tests (or the time/resources to build them — see ineffective planning).

Solutions

Some of the solutions for the above actionable problems are obvious, and others are far less so. This is the first in a series of upcoming posts in which I will drill down each set of problems and discuss the solutions in detail. 

Photo by Erik Odiin on Unsplash