This is a complete guide to the smoke testing process for consumer insights, product innovation and product management teams.
In this in-depth guide you will learn:
What a smoke test is and why you should use them
How to set up effective and efficient smoke tests
The advantages of smoke testing in market research
If you're ready to start collecting unbiased behavioural data to make consumer-validated product decisions, this guide is for you.
Key Takeaways:
Smoke tests, originally from software engineering, have been adapted for market research to validate assumptions and reduce uncertainties early in product decision-making by generating consumer insights.
In software development, smoke tests are performed at the start of the testing process, running a basic suite of tests to confirm the software's functionalities. If these tests fail, major issues must be addressed before more detailed testing can proceed.
In market research, smoke tests are used differently, mainly to consumer-validate market and product assumptions. A common method is creating ad campaigns and a landing page for a non-existent product, then measuring consumer behaviour along the entire customer journey. With these insights, teams can confidently proceed with the right product decision or plan for consumer demand.
While smoke testing is well-established in software development, its use in market research is still relatively new. Thus, product, innovation, and market research teams may need to be innovative and flexible in their smoke testing approach, tailoring it to their market and organisation's specific needs and constraints.
In this chapter, we will grasp the fundamentals of what a smoke test is and the basics of how they work in market research vs. their application in software development.
In later chapters, we will examine how to set up your smoke tests in more detail.
What is a Smoke Test?
Originating from software engineering, smoke tests quickly detect fundamental product issues, ensuring critical functions work correctly. They're designed for rapid execution, enabling early identification and rectification of significant failures, paving the way for more detailed testing. If an application fails a smoke test, it requires further development.
Market research has adapted the concept to validate early-stage product development assumptions and minimise uncertainties. It's a quick, efficient method to evaluate if a new product, service, or strategy will engage a market before substantial investment in full-scale development or release.
Market research smoke tests commonly involve creating a rudimentary product representation (usually a landing page) to measure consumer behaviour and assess if consumers demonstrate enough interest—e.g. measured via clicks on a ‘Buy Now’ button—to gain certainty with a product decision. This process enables product developers and market researchers to mitigate failure risks and resource wastage early on.
How Leading Brands Harness the Potential of Smoke Tests before Decision, Development & Go-to-Market
Leading brands across industries have harnessed the potential of smoke tests in their research efforts and go-to-markets.
Dropbox is one of the most well-known cases. Before developing their innovative cloud storage service, the founders created a simple video explaining the idea of Dropbox and gauged people's reactions to the video. The video went viral in the tech community, validating the market demand and prompting the founders to continue building the actual product.
Buffer’s CEO and Co-Founder, Joel Gascoigne, successfully implemented a smoke test to evaluate a) consumer interest in the product and b) their willingness to pay for different pricing plans. He established a pricing page with three distinct levels. Observing a high click-through rate on the paid plans and users following the guided CTAs before providing signing up, he concluded that the smoke test was successful as he identified which price point was possible to set for his product.
Smoke Tests in different Stages of Market Research
Smoke tests in market research offer cost-effective ways to gather consumer insights and make well-informed decisions.
Idea Validation
Smoke tests are widely used in early market research to validate product or feature ideas. They can help consumer insights, innovation, and product teams gauge if an idea appeals to potential customers before investing heavily in development. This pre-emptive step helps avoid creating unwanted products, reducing risks of failure and increasing the likelihood of product success.
Testing Market Demand
Smoke tests can also identify the best market for product launch. Teams can use them to assess target groups’ interest in a product. By promoting a product-focused landing page to various market segments, consumer behaviour can indicate in which segment the product will likely be most successful.
One of our clients, an international beauty player, has identified France as the primary market for their new product. You can get the free case study here.
Evaluating Brand Perception
Smoke tests can be used to evaluate and refine branding strategies before go-to-market. For example, a business could use a smoke test to assess consumer behaviour with a specific brand identity or a value proposition. By presenting the new branding concept on a landing page or through mock ads, companies can gauge public reaction before fully committing to the entire design.
Uncovering Price Uplifts and Margin Potential
A smoke test allows you to evaluate price uplift potential by testing various price points, and this being the only distinction between the variants. You can track conversions by creating landing pages with low, medium, and high prices and directing equal traffic to each. If higher-priced variants achieve a relatively high conversion rate on, for example, the add-to-cart button, it suggests the potential to increase your price without hurting demand.
BSH, one of our customers, discovered a potential 20% uplift in the price of one of their products in a recent smoke test with Horizon.👇
In this chapter, we will look at why you should conduct smoke tests and how the testing process fits into your current consumer insights process.
Why should I run Smoke Tests?
The benefits of running smoke tests are that you are gathering the most unbiased behavioural data possible for your new product quickly and cost-effectively.
Quickly validate Product Ideas
Smoke tests can gather highly valuable insights for decisions on new and existing products. In as little as a week, you can have behavioural insight on a product before developing any MVP, concept or prototype. All you need is some concept imagery, and you can get started. This allows you to quickly set up and run tests, so you can run more tests per product and develop stronger product propositions or discover that the true market demand is lower than expected.
Gather unbiased purchase intent Data
Because of how smoke testing works, it supplies market research, innovation and product teams behavioural data. Throughout the consumer journey, you gather data on interactions with ads, landing pages and emails to fully understand how your product concept would be received in real life.
The unbiased part of smoke testing comes from the fact that consumers don’t know they are in a market research test at the time, as the product appears to exist and is purchasable. Once they have completed the journey, you should let them know it was part of a test. It’s important to disclose that before gathering personal data such as an email address.
Reduce Product Development Risk
Developing new products and extending product lines is an expensive and resource-heavy commitment. While traditional market research tries to alleviate this by ensuring the products in development will meet consumer expectations and sell well, it is based mainly on opinions rather than behaviour and so can only be trusted so far.
With smoke testing, you are gathering behavioural data without developing any MVP, making it cost-effective and reliable in its forecast of market demand, reducing the failure rate of new products and ensuring that only the products with consumer demand are in production.
Improve existing Products and launch Line Extensions with Confidence
By gathering unbiased behavioural insights, you can trust that consumers' behaviour towards this test aligns with their real-world thinking. This means you can better forecast the demand for this product on launch.
You can also validate your current research using smoke testing. Using your insights from your opinion-based research, you can develop smoke tests to find whether the behaviour matches the opinions.
Our client Bosch was able to make a consumer-validated decision on a line extension with behavioural research. You can get the free case study here.
Never miss expert insights and case studies on market success prediction with our monthly newsletter
Through smoke tests, you can nurture a much more innovation focussed culture because the test process is so quick and inexpensive that you can easily test new ideas and concepts reliably within the business. Some companies run hundreds of tests yearly through smoke tests to quickly validate ideas that would otherwise be put on the shelf without consumer demand verification.
Improve customer-centricity
Customer-centricity is at the heart of any great product, and with smoke tests, you can understand closer what consumers want by gathering behavioural data. Smoke tests' main purpose and benefit within customer-centricity is that the consumers show you what they want by putting their behaviour on the ads and landing page presenting your product.
Chapter 3:
Pros and Cons of Smoke Testing
In this chapter, we look at the challenges that the application of smoke testing for market research has and how to overcome them.
Advantages of Smoke Tests vs. Other Market Research Approaches
Smoke tests offer a unique, efficient market research approach compared to traditional surveys. However, the best strategies often combine different approaches, leveraging their respective strengths.
Speed and Cost-Effectiveness
One of the key advantages of smoke tests is their speed and cost-effectiveness. Unlike surveys and focus groups, which can be time-consuming and expensive, smoke tests can be set up quickly and inexpensively, allowing businesses to gather behavioural consumer data quickly.
Smoke tests allow you to act with maximum customer-centricity at low costs.
Real-World Consumer Validation
Smoke tests offer more reliable consumer insights than traditional methods by tracking behaviour along ad campaigns and landing pages presenting a product concept. This real-world validation captures actual behaviour and purchase intent rather than relying on potentially biased self-reported data, enhancing the value of research for product development.
Data-Driven Decision-Making
Smoke tests enable data-driven decision-making by providing quantitative data on key metrics like conversion and click-through rates. This can help businesses make informed decisions about product development, marketing strategies, and more. Qualitative market research methods like surveys and focus groups often provide valuable data that can be harder to analyse and serve as less actionable.
Precise Targeting
Smoke tests can precisely target specific audiences, yielding insights more applicable than traditional methods. Although they aren't a replacement for qualitative research, businesses gain a deeper market understanding and can make better-informed decisions when combined.
Risks of Smoke Tests and how to avoid them
The risks of smoke tests are primarily related to managing consumer expectations because you collect data on a not-yet-built product. You need to be clear to avoid appearing deceitful or fraudulent.
Although seemingly detrimental, these risks can be handled easily with proper safeguards.
Perceived Deception
Smoke tests involve consumers interacting with a product or service proposition without knowing it's a test, which can lead to issues if they discover the truth. They may feel duped and that the brand has been dishonest.
This can be effectively circumvented by ensuring appropriate messaging throughout the test and if a user completes the entire process.
An example could be:
The product is not available.
Placing these messages latest at the resolution stage manages the audience's expectations in a way that doesn’t blatantly say they have been part of a test.
Remember that you must comply with GDPR standards 👇
When performing smoke tests, you mustn’t fetch and process personal data unless you have not informed the consumer that they are part of a test. Make sure you have the right landing pages, privacy policies and process in place to maintain GDPR compliance while testing.
Brand Bias and Reputation
The potential perception by consumers of your test as deceptive might impact the brand's reputation. Therefore it’s crucial to disclose that consumers are taking part in a test before gathering any personal data so that you do not disappoint them and violate GDPR compliance by storing personal data under false conditions.
Another challenge with smoke testing, particularly when testing with landing pages, is brand bias. If you have a well-known brand, results can be influenced because consumers recognise the brand and are predisposed to accept anything the brand offers.
You can navigate both challenges through strategies in off-brand and on-brand testing:
Off-Brand Smoke Testing
This method allows you to run tests using fictitious brands. This ensures that any insights gathered are brand-bias-free and focus solely on the product in question.
You can also leverage brand bias to your advantage through on-brand testing. This approach works perfectly for running a brand test, as you can pit multiple brands against each other to see which meets the highest consumer demand or, ultimately, purchase intent.
On-brand testing is also highly effective for making other strategic product decisions.
One example would be if you already had a successful product in the market, but there's an opportunity to introduce a new variation (different colour, size, etc.). Using on-brand tests would allow you to gauge the demand for this new variant based on consumers already familiar with the existing product. This can help you determine whether this variant should be developed or not
Our client Bosch was able to do exactly that: Introducing a new variant through a smoke test with Horizon, using on-brand testing. You can get the free case study here.
Whether using on or off-brand testing you will always need to have GDPR-compliant privacy policies and imprints. This means, whichever brand you use on the face of the landing page, your real and main company, i.e. the legal entity will always be visible in other areas such as the imprint.
Incognito testing allows you to test without disclosing your company in the imprint or as the provider of the landing pages. With our incognito testing service, Horizon provides the imprint and legal infrastructure for your ads and landing pages so that your company brand or legal entity is not associated with the research project by consumers and competitors. This allows you to get to a consumer-validated product decision even faster.
Not having focused Test Parameters
Smoke tests focused on one variable at a time, effectively gathering consumer demand data. Similar to A/B tests, minimal changes between tests allow clear attribution of results. For example, testing a new headphone type, pricing, features, and colour should each be examined separately to ensure clear results.
The graphic above illustrates how you progress from one test to the next, with one variable per test. For example, you might conduct a series of pricing tests. You ensure clean data by creating multiple landing pages where the price is the only varying element. Then, in the next round, you shift to features, adopting the same pattern - landing pages where the only change is the set of features presented. Repeat this process for any other aspects you wish to test.
Chapter 4:
How to Run Smoke Tests
In this chapter, we will look at the process of running smoke tests and how you can set these up in a few steps.
How do Smoke Tests work?
Test Setup and Design
In a smoke test, the initial step is determining what to test—a product idea, features, or branding. A landing page is then developed, presenting the concept as if it's real, with product benefits and a clear call to action, like a purchase or signup.
Test Execution
Smoke test execution entails directing traffic to the landing page, commonly via ads on platforms like Google or Meta. These platforms can target distinct consumer demographics and behaviours to identify the most promising market for the product pre-development.
Methodologies and Key Metrics
Smoke test methodology, rooted in minimal viable testing, primarily tracks metrics like conversions on CTA (i.e. add-to-cart) clicks. It aligns with Alberto Savoia's pretotyping concept, emphasizing early, real-world behavioural data collection to verify product/market fit assumptions. Savoia argues that most new products are prone to fail due to development based on assumptions rather than market realities.
Analysis
Smoke test analysis uses behavioural consumer data to guide product decision-making. Unlike qualitative surveys, it tracks behaviour such as clicks and actions on ad campaigns and a landing page, representing real-world purchase intent.
Never miss expert insights and case studies on market success prediction with our monthly newsletter
You should use smoke tests before building a new product or making any kind of product decision. They validate your idea, concept or decision by gathering data to help you understand the market demand for that product decision before proceeding to development or go-to-market.
Smoke testing is basically pretotyping, testing a product concept before developing a prototype or MVP to gauge market demand. Unlike traditional consumer research, smoke tests require a focused approach, testing specific aspects individually for strategic decision-making.
Step-by-Step: How to conduct Smoke Tests
Executing effective smoke tests in market research involves carefully considered steps. From formulating hypotheses to analysing results, each stage is critical in garnering consumer insights. It’s a simple process that doesn’t take long to set up and run.
Step 1: Formulate the Hypothesis
The first step involves formulating a test hypothesis about your product decision. This hypothesis is what you aim to test through your smoke test. It could be about the product's market demand, the audience's willingness to pay for a specific price, or any other assumptions you've made about the product.
An example of a hypothesis could be:
‘Launching a red-coloured variant of our product will increase sales by 33%’
Step 2: Design the Test
With our example hypothesis, we would need multiple versions of our landing page and advertisements, one variant for a distinct colour, with the product colour being the only difference between the landing pages. The distinct paths of ads and landing pages are called variants, and you need to create one for each version of the product you are testing.
3 wireframes for distinct variants of ad campaigns and landing pages. Each path would represent a separate colour option.
We advise you to run a minimum of two variants and a maximum of six. Any more than that, the data gets hard to read, and the spread of data is too thin. Any less than two, and you don’t have any valid comparison that can validate your hypothesis.
Step 3: Create Landing Pages
Designing the landing pages is one of the most important areas to put some time and focus on. There are two key areas to consider with designing your landing pages:
Minimum Changes for Maximum Attribution
A common issue is making multiple changes to landing pages between variants, making it impossible to attribute data to specific changes. Ideally, only elements that need to be changed between variants should be altered. For example, when testing for the right product colour, only the colour and related wording should be changed, leaving all other content and branding the same.
The colour of the products is the only distinction between the three variants
Design it once, design it well
Because of the use of variants and keeping changes to a minimum, you don’t need to design three individual landing pages. You prepare one page and replicate it to make minor changes. This will save you a lot of time, but you should spend more time on that first design to ensure a compelling page.
The landing page must look like an actual sales landing page you would create when launching the product. It needs enough detail and data to convince people to purchase the product.
Step 4: Create your Ads
Driving consumers to the landing page quickly, in sufficient numbers, and from the right target group is vital. Digital advertising, like Facebook or Instagram ads, works well for this.
You need to create ads for each test variant, keeping the campaign design consistent except for the variable being tested. The ad should promote the product convincingly with good copy and visuals. Running a campaign with two ad variants can boost performance by 5%, so consider creating different ad styles for maximum impact.
Step 5: Create your Target Audience
Now you can pull it all together in the ad platform of your choice and create the audience to send your ads out to.
Typically you only need to set up one audience based on the current customers for your brand or an idea of who that audience is. You can also run target group tests with smoke testing, meaning you set up multiple audiences and send them to their respective landing page variant to see which performs better.
There are some guides to follow with your audience:
Aim for a target audience size between 1.5-5 million, at least 250,000
Include interests that are relevant to your market
Exclude interests to make sure you aren’t getting outliers
Step 6: Create follow-up Content
After a consumer completes your test journey by clicking the CTA button or filling in your signup form, they usually encounter a confirmation page, email, and resolution landing page, in that order. You can use tools like Unbounce, Mailchimp, Google Analytics, and Facebook Ads to collect the data, which then needs to be consolidated and analyzed.
Shameless plug: Alternatively, you can use platforms like Horizon that streamline this process, connecting all tools in one place and presenting the data in a user-friendly format.
The Customer Demand Score by Horizon allows you to identify the variant with the highest consumer demand in no-time.
Step 7: Run the Test
All that’s left is to press play and get those tests running. Be aware of seasonality, as this can affect your results as well.
You should aim to let the tests run for at least a week to allow for sufficient data collection.
Step 8: Analyse the Test
Once the results have been collected and returned, you need to review the data in whichever platform you have chosen to gather it.
We must refer to our hypothesis statement to analyse the right data and see if the insights have proven or disproven our initial thinking.
Step 9: Derive Actionable Insights
Smoke tests help gather consumer behavioural data to inform product decisions. If they reveal strong consumer demand, teams can make the right product decisions; otherwise, they reevaluate the product concept or market. Overall, smoke tests, when appropriately executed, offer a cost-effective way to validate ideas and make data-driven product decisions.
How to structure Smoke Test Landing Pages:
The layout of your smoke test landing pages needs specific calls to action that drive consumers on your landing page. You also need to tell the consumer what the product is about in enough detail to make it seem like an actual product.
There is a flow that works best for sales landing pages, and smoke door test landing pages are no different. They should be treated as a live sales page. The only difference is that they are converting to demand data instead of converting to actual sales.
Here’s the perfect landing page flow for your smoke tests:
Header
Create an engaging header with appealing visuals and concise content. The content should show the product image and convey its benefits to the consumers, focusing on them, not your brand. Include a call-to-action in the header to prompt users to complete the test quickly.
Value Proposition
Briefly explain the product's benefits in more detail and the impact on the consumers' problem you're solving.
Product Detail and Conversion Goal
The content here should make a consumer want to ‘purchase’ the product and should be convincing. This is also where you would put details about the product such as features, pricing, appealing imagery and the add-to-cart button. This button reflects your conversion goal to measure purchase intent.
Process
Now you have the process section where you tell a consumer how to use the product. It could also relate to one of the main benefits of the product. You will want a mid-page call to action as well here.
Further Benefits and Content
Below the product detail and the process section you could add additional content to the landing page to make it more compelling and appear like an actual sales page.
It's important to include multiple and consistent call-to-actions across the entire page that drive the consumer towards the product detail section. That's where the conversion should happen in case of purchase intent.
Are Smoke Tests Landing Page Tests?
Smoke tests are also called landing page tests, facade tests, fake door tests, painted door tests and even Pretotyping. Smoke tests go by many names, but they all do the same thing.
Chapter 5:
Conclusion & Takeaways
In this chapter, we'll sum it all up for you on why, how, and when successful product, innovation and consumer insights teams integrate smoke tests into their market research process.
Conclusion: Embrace the Power of Smoke Tests in Market Research
Smoke tests offer an invaluable way for those engaged in market research and product development. They provide a cost-effective and efficient way to gauge consumer demand, validate product ideas, and collect crucial insights for product decision-making before significant resources are invested. They are a powerful aid in risk mitigation, helping to avoid costly missteps and optimising products based on real-world demand data.
The value of smoke tests extends across the product development lifecycle, from the early validation of a product concept, through optimisation of features and branding decisions, to final tweaks before go-to-market. They empower teams to make data-driven product decisions, ensuring every step is backed by tangible evidence from the target market.
Embracing smoke tests in product development can pave the way for more informed, confident decision-making and product success.
Takeaways
Smoke tests can be used as a consumer insights method to gather behavioural data through landing pages.
They work by directing people to landing pages and gathering data by tracking their interactions with it.
The benefits of them are being able to gather behavioural data and validate your opinion-based data like surveys.
You should run them before you build a product or are making any strategic product decision, but after your initial consumer insights testing.
The risks of smoke tests are consumers feeling ‘scammed’ if the test hasn’t been communicated well. You shouldn’t be scared of admitting it’s a test to benefit them, but equally you don’t have to let them know.
The process of running a smoke test is to develop what you are wanting to test, or your test mission, designing your content for the test and analysing the data afterwards to validate the reason for testing.
Your smoke test should always include a reason for testing, without it you will not understand what the data means and how to act upon it.
Smoke tests can also be called Pretotyping, landing page tests, façade tests, smoke tests, painted door tests.
.webp){kind=logo}
