A simple growth experiment
We added a pop-up to our blog—here’s what happened
👋 Hey, I’m George Chasiotis. Welcome to GrowthWaves, your weekly dose of B2B growth insights—featuring powerful case studies, emerging trends, and unconventional strategies you won’t find anywhere else.
We run a simple experiment on Minuttia’s blog.
It turned out to be (somewhat) successful.
Here’s what we did and what we learned.
Hypothesis
Every growth experiment starts with a hypothesis. Think of the hypothesis as the reason why you’re running this experiment. To form your hypothesis, you need one critical component: Observation
First, you observe something, and then you come up with a hypothesis based on what you have observed. For us, the observation was the following:
Many people visit our blog, read our content, and get value from it but never take an action that is meaningful to us.
In this case, a meaningful action would be visiting our Let’s Talk page and scheduling a call.
So, based on this observation, we formed our hypothesis:
Adding a pop-up that appears after the visitor has scrolled through 30% of the page and disappears before the footer will increase the number of visits to our Let’s Talk page.
Note: We could have quantified the anticipated increase in percentage points (pp) or absolute numbers.
So, we installed this simple pop-up on our blog:
As you can see in the video, the pop-up appears after scrolling through 30% of the page and disappears right before the page's footer.
Structuring our experiment
The first two elements of our experiment are Observation and Hypothesis. We then need to fill out an array of information based on a Notion template we’re using.
Editor’s Note: I’ll be giving away the template with this month’s GrowthWaves Pro note. To access it, please sign up for GrowthWaves Pro.
For this particular experiment, the Score (pre-running) was relatively high (7, with 9 being the highest) because we believed that…
The experiment would be impactful;
It would also be easy to run;
And we were confident that it would be successful.
The (core) metric we were trying to impact was the number of clicks to our Let’s Talk page, and the growth loop within which we were experimenting was the Acquisition one.
All these bits are case-specific, of course. But seeing how other companies structure their experiments can help you understand how you can structure yours.
Running and completing our experiment
After structuring our experiment in the ‘Hypothesis’ phase, it’s time to run it.
Here’s what our experiments look like while in the Active phase:
As you’ll notice, the columns are somewhat different from the previous phase, Hypothesis.
After completing an experiment, we mark it as ‘Complete’ and move it to the final phase of its lifecycle, which looks like this:
In this phase, we evaluate the results and decide whether the experiment was successful.
Every experiment will end up with one of these two outcomes:
Win
Loss
For the record, ‘Loss’ isn’t bad. It just means that this particular experiment (or version of it) wasn’t successful based on the hypothesis, metric, and all other attributes you’ve set.
Rest assured that a ‘failed’ experiment can teach you as much (if not more) as a successful one.
We’ll decide on a deployment (e.g., of the change) or pullback depending on how the experiment went.
So, all that is good, but what about the experiment we ran? Here are the results.
Win or loss?
At first sight, our experiment was a win. I will explain why that may not be true in a minute. Below, you can see a screenshot from Minuttia’s growth model:
Note: I’m sorry I had to blur a big part of the screenshot.
As you can see, November was the first month we introduced the blog pop-up. In November, our pop-up received 8 clicks. Then, it received 28 clicks in December, 40 in January, and 24 in February.
CTA clicks on this pop-up increased by 42.86% between December and January. That could be partially attributed to seasonality, as January is traditionally a stronger month for Minuttia’s services.
To understand the impact of this, you’ve got to compare ‘Blog Pop-up CTA Clicks’ to the ‘Let’s Talk CTA Clicks’ metric, which represents clicks to our Let’s CTA across Minuttia’s website.
Here’s the contribution for a 3-month period:
December 2024: 43.08%
January 2025: 43.48%
February 2025: 37.50%
Note: I’m not considering November because it was the month we ran the experiment.
On average, we can see that clicks to our pop-up account for 41.35% of all clicks back to our Let’s Talk page.
However, an issue that came up is that visits from the pop-up don’t convert as well as those from the rest of the website. This has decreased the form attempt-to-submit ratio.
But does that suggest we need to work on conversion optimization for our Let’s Talk page, or is there something else we’re missing?
The experiment was somewhat successful, but the traffic we send to our Let’s Talk page doesn’t convert. I take two things from this:
The first part of our hypothesis was correct; a pop-up can indeed generate clicks (and traffic) back to a page.
The second part was wrong, which means that we optimized for the wrong metric.
This means that we now have to go back to the drawing board and design another experiment that will allow us to capture as much value as possible from this pop-up.
In that context, our experiment definitely taught us something and was a success.
Final thoughts
Growth folks working in big companies won’t like this, but:
Running growth experiments for big companies with massive user and/or customer bases and huge audiences isn’t the same as running experiments for small companies.
It’s just not.
So, I'm always skeptical whenever I read growth case studies from big companies—however successful or innovative they may be.
Would that work in a small company with little reach and attention?
Was the experiment successful because of the company or because there was substance in designing and running it?
In any case, I hope you now better understand how to set up an experiment. (And realized that even though archaic, pop-ups still work!)
If you do, regardless of the outcome, let me know what you learned.
GrowthWaves News
1. GrowthWaves Live #6
GrowthWaves Live comes back on March 20 and is available to all our GrowthWaves Pro members.
In these one-hour sessions, we discuss trends, case studies, experiments, and lessons learned from our work and keep you up-to-date with everything you need to know about GrowthWaves.
Become a GrowthWaves Pro member to join the next event.
2. SERP Conf. Sofia 2025
On March 25th, I’ll be speaking at the SERP Conf. Sofia in Sofia, Bulgaria.
If you haven’t visited Sofia yet, this is a great opportunity to meet this beautiful city in the heart of the Balkans.
To make the decision easier, use the code ‘george-20’ at checkout to get 20% off your ticket.
The event was sold out last year, and from what I hear, tickets are moving fast.
If you’re in Sofia around March 25th and want to meet up, just DM me on LinkedIn.
See you there!