What data should you expect from a fake door test?
The data you get from fake door testing isn’t the same as anything you can expect from traditional consumer insights methods like surveys. Where those give you opinion-based data through comments or a sliding scale, fake door tests give you multiple different data points based on behaviour.
In this blog we will go through the different data points you can expect to gather from fake door testing, what they mean and how to use them effectively to tell the story from the data.
Advertising performance data
The first collection of data you will be able to see is all about the advertising you are running to direct traffic to your landing page variants. This data is an important starting point to help you understand the initial demand for your product and to forecast go-to-market costs.
Cost-per-click (CPC)
What: The data you receive from the CPC will tell you how much you have to spend on advertising to get someone to click on your ad.
Purpose: This data helps you to understand initial demand for your product. A lower CPC indicates a higher rate of interest in that target group and therefore a lower customer acquisition cost.
How to use it: You can start to see a different between each variants CPC which you can use to give you an indication of which product will have a cheaper acquisition cost. However it isn’t the end cost, as they still need to convert on the landing page. You should think of it as a top-level indicator of success.
Using CPC data when running target group tests is also a great way to quickly test out different audiences and see the varying demands from each.
Cost-per-visit (CPV)
What: This data tells you how much you had to spend to get someone to visit your landing page. This number can be different from your CPC because it is measure from the landing page and not the ad platform, meaning cookie consents, shared links or ad blockers can skew the data between these two.
Purpose: This data shows you a similar thing to the CPC but is measured from a different place, allowing you to align on the most accurate numbers.
How to use it: You can use this data in conjunction with your CPC to gain an understanding of the overall acquisition cost you should forecast for from an advertising perspective.
Never miss expert insights and case studies on market success prediction with our monthly newsletter
Now you have attracted people to visit your landing pages, you need to understand the behaviour they take on those pages and how that relates to the goal of your test. This is where fake door tests start to shine and give reliable behavioural data.
Unique landing page visitors
What: This data tells you how many unique visitors you had to your landing page, it doesn’t count people visiting on more than one occasion.
Purpose: This data helps you to understand the relative trend in visitors between your landing page variants, it will tell you day by day how many visitors each landing page gets.
How to use it: This data is a great way to analyse the overall health of your landing page and drill down into the quantity of visitors so you can quantify the insights for each variant. If one variant has a considerably low traffic volume, the averages will be skewed.
Low quantity of traffic to a specific landing page should only be an issue if the traffic suddenly drops off. This could be because the landing page is down or the advert has been turned off for some reason. Low quantity of traffic to a specific variant generally means that variant is undesirable. Low quantity of traffic overall could mean the product is undesirable, the ads aren’t setup correctly or the ad creative isn’t compelling enough.
Conversion rate on call-to-action
What: This tells you the percentage of users that click on the call-to-action on your landing page. This is generally the button that directs them to a ‘purchase’.
Purpose: The data collected here is very important and will help you understand the behaviour that a user takes on your page. Specifically them clicking on your calls-to-action.
How to use it: The insights here give you your first reliable impression for the demand of your product and specifically the variant you’re testing. The higher the conversion rate, the more demand for that variant.
Conversion rate to lead
What: This tells you the percentage of users that completed the action you wanted them to take on the landing page. Whether that is filling in their email address or another form of skin-in-the-game.
Purpose: The data you collect here really starts to define the actual demand. At this point, users are ‘giving’ something in return for your product.
How to use it: With this data you can validate your hypothesis and find the real demand for your product. This is the behavioural data that will show you real market demand.
Single opt-in to double opt-in conversion rate
What: This stage tells you the percentage of users that received the confirmation email, so converted into a lead, and then confirmed their email address, the double opt-in.
Purpose: This data will give you even more reliable evidence of the products demand as there are multiple steps for a user to take to ‘access’ it. If they go through the double opt-in process you can confidently say that they really wanted this product.
How to use it: This is the most reliable piece of data to show demand as there are a few steps users have to take to get here, which confirms demand for that variant. It’s also a great way to determine which variant to develop if they are close in other metrics, the more double opt-ins the deeper the demand.
Based on the amount you spent to get users to the landing page and how many conversions you had, you can also work out the customer acquisition cost (CAC) as well. This is the best way to forecast the go-to-market costs for this product and variant.
Never miss expert insights and case studies on market success prediction with our monthly newsletter
Now we have an idea of all of the separate pieces of data and their purpose, we can bring it all together to start analysing how a customer gets from the advert to the double opt-in and how many pass through the funnel.
This is a great visual way to match up all of the data and see where users dropped off. Sometimes the obvious product development choice from your advertisements isn’t always the best choice to actually develop and you can only really start to see this when you build the customer walkthrough.
Introducing the Customer Demand Score (CDS)
All of this data can be overwhelming to read and analyse effectively, especially when you are testing multiple variants of a product with little difference between them.
That’s why we at Horizon have developed the Customer Demand Score. It aggregates all of the data gathered from your fake door tests and benchmarks it against our database of test results.
What you get out of this is an easy to read score for each variant, allowing you to quickly see the best performing one overall.
While we think this is the best thing since sliced bread, it doesn’t mean it is should be taken completely at face value. After all, you might be testing for specific data points, you test mission might direct you to look at a specific conversion rate as a sign of success.
So always look into the data deeper, and get a quick overview with the CDS.
Never miss expert insights and case studies on market success prediction with our monthly newsletter
An experienced growth marketer now helping Horizon and it's customers create successful products. Always looking to expand his ideas and take on unique and interesting takes on the world of marketing and product development processes.
.webp){kind=logo}
