Building growth processes

The growth team moves fast. You release a new A/B test every week, a lot of them are winning and the impact is huge. 🎉

Let’s imagine you just optimized this funnel and got a whopping 9% overall conversion rate to purchase.

How to build good growth processes? How to ensure this won’t break when you’ll be doing more experiments or other teams will do changes in the code?

Here are my methods:

1. Cover your code with automated tests
2. Do manual regression QA
3. Setup monitoring for critical errors
4. 🥇 Add analytic alerts for events and funnels 🥇

And the latter is the best solution, let me tell you why.

Automated Tests

Automated tests check your application without human interaction. This can be a great tool, or a waste of time – depending on what tests you choose. Automated tests can be run on every commit – to check if this new code doesn’t break anything. If it does – it won’t let adding such code to a project.

I recommend adding only Backend API tests. They bring the most value, actually prevent bugs and are very fast to run (in a matter of seconds). Stay away from Frontend component tests, it’s a lot of effort to maintain this code as the UI constantly changes. If your company has an Automation team – ask them to cover critical flows (sign up, payments) with e2e tests with Playwright/Selenium. But don’t do it yourself, that’s really time consuming.

Automated tests are neat, but they don’t catch all the bugs, only the most obvious ones.

Manual Regression QA

Before every release a QA engineer goes through a list of test cases for existing features and verifies that there is no regression. But that test cases list grows very fast and soon it’ll be impossible to verify all test cases. And you’ll start compromising on QAing some features. Surely these abandoned features will break.

Monitoring

You can set up monitoring with DataDog or a similar tool. Or simply add alerts in your logging solution when a certain message is written to the log. That’s simple and efficient. And it is a must to do for critical error logs. But it won’t catch issues when the feature is working but its discoverability is reduced. Or the new UI is confusing and so no user actually goes through some flow.

Analytic Alerts

That’s my favorite one.

Let’s have a closer look at the funnel we talked about before. What data points are important to us here?

Overall conversion rate

This is the most important one. If it goes below some threshold, you want to be notified. With hourly alerts you will know if that happens in an hour max. If you don’t have enough traffic you can create bi-hourly, multi-hourly or even daily alerts. Gradually you will calibrate it to avoid false alarms. Go ahead and setup such alert in your analytic tool, here’s how you can do it in Mixpanel.

What else you might want to monitor with alerts?

Absolute conversion counts – how many users who did all the steps in the funnel. If you’re not sure which threshold to choose for such an alert you can add an Anomaly alert. Analytic tool will forecast (based on statistics, not AI 🤖) the range based on old values. And if the new value during some time interval will go off that forecasted range – alert will be sent.

These are my methods. And how do you monitor your features, fellow growth hacker?