In the competitive landscape of product development and innovation, understanding, if consumers will actually buy a new product or a variant of your product, is crucial for accurate demand planning and maximising the return on investment for your product decisions. While behavioural research through fake door tests offers a gateway to consumer-validate product decisions with real-world purchase intent insights, some pitfalls can compromise the quality of your research data, the statistical significance of your insights, and, ultimately, the outcome of your product decisions. Wrong product decisions can have a detrimental impact on both the top and bottom-line results of your company.
In this article, we’ll discuss some of the most prevalent mistakes made in behavioural research with fake door tests that present challenges for teams in innovation, product, and market research.
Fake door tests, also called smoke tests, pretotyping, painted door- or red-eye tests, allow you to uncover the true purchase intent based on behavioural data by running multivariate landing page tests with consumers from social networks or search. If you are curious about the general mechanics of fake door tests, we invite you to explore our comprehensive guide here.
Mistake #1: Not defining a clear research question
A well-defined research question is your compass in the vast sea of consumer behaviour. It narrows your focus to the critical factors required for making informed, consumer-validated product decisions. Alongside this, a test hypothesis anticipates the outcome based on your research question and the structure of your test. These elements should ideally be based on prior qualitative market research, then validated through behavioural research like fake door tests. Research questions may include inquiries into the optimal price point, branding, value propositions, features, or targeting specific consumer demographics, for example:
👉 What price point between my current product price (control variant) and two higher price points performs best, and is there an opportunity to increase product pricing?
👉 Which branding/value proposition/feature shows the highest purchase intent from consumers
👉 Which of the three target groups indicates the highest demand for my new product?
Without a research hypothesis or question, you are without a solid criterion to conclude your research as successful.
Mistake #2: Ending tests prematurely
The duration of a fake door test isn’t set in stone. Whether it runs for 7, 14, or more days should be dictated by the point at which you can answer your research question or hypothesis based on statistically significant insights. This means identifying a winning variant with confidence because you've accumulated enough data for a reliable conclusion. High conversion rates for test variants might be promising per se, but without statistical significance, they're deceptive. They can lead to conclusions that don't hold up against long-term consumer behaviour, resulting in misguided product decisions.
Behavioural consumer research tools like Horizon can provide essential metrics, such as the sampling error per variant of your test, to guide you in knowing when you've reached a statistically valid conclusion.
Mistake #3: Testing too many variables in one test
In setting up a multivariate fake door test, it's imperative to isolate the performance impact of individual variables you modify across variants. This could be different branding elements, value propositions, features, target demographics, or product pricing. For example, if you're assessing which colour variant of a product resonates most with consumers, alter only the colour and its descriptive wording, leaving all other content and branding consistent across all test pages and customer journeys used in the research.
How Bosch consumer-validated a decision on a line extension with behavioural research
Bosch successfully employed Horizon for behavioural research to validate a decision on a colour extension for an existing product line. Using Horizon, Bosch gathered real-life purchase-intent data at scale for different colour variants with their already existing variant as control group.
In this process, Bosch specifically used Facebook and Instagram ads, driving traffic to multiple landing page variants, each showcasing the product in a distinct colourway. The data from consumer actions on these landing pages provided the purchase intent insights Bosch needed to decide on the next colour to introduce to the market.
“Working with Horizon was a remarkable customer experience and enabled Bosch to make an impactful strategic pricing decision.”
E-Lin Tan
Global Head Smart Indoor Gardening - BSH Hausgeräte GmbH (Bosch)
👉 Get the full case study for free here.
________________________________
In conclusion, when executed methodologically accurately, fake door tests are one of the most powerful methods to consumer-validate your product decisions by uncovering purchase intent from real consumers through behavioural data. They can help innovation, product & consumer insight teams to close the say-do gap, validating what consumers say they would do with what they actually do. By avoiding these common mistakes, you ensure that your fake door tests are not only a reflection of genuine consumer behaviour but also a robust foundation for your demand planning and a vehicle to increase the ROI of your product decisions.
.webp){kind=logo}
