8 Frequently Asked Questions About Horizon

When talking with clients about a new testing methodology and tool, there are always going to be certain questions that pop-up, so we wanted to answer all of those questions for you.

‍

1. Is there any brand risk when showing products that are not available yet?

Naturally you might expect there to be a risk to the sentiment around your brand when showing products or product decisions that don’t exist yet, however this is very easily managed with the right user journeys and on or off-brand testing methods.

After running hundreds of tests, we’ve come to use one of three management strategies to make sure the users of Horizon don’t incur any brand damage:

First, you can be clear that it was a market research test once we have resolved the customer journey. This is normally accompanied with a % off available products in your webshop as a reward for participating, and to minimize the risk of the consumer being disappointed.

Second, you don’t mention it was a test. If you have set a test up properly, there shouldn’t be any way for consumers to tell it was a test. At the end of the journey the consumer will be shown a page, stating that the product is unavailable. This has been used in a lot of Pretotyping tests without any issues (we are talking about six-digit number of consumers with zero issues) and with more and more products frequently becoming out-of-stock, this is a normal expectation from consumers.

Third, you can run the test without your brand at all, perfect if you want to test products while removing a consumer’s brand bias. All you need to do is change the branding on your product to something new and you can start testing without brand risk. If you want to guarantee complete brand removal, you can use our Incognito service, where we host all of the legal policies and data management as well. When testing incognito, a) the consumer cannot trace the test back to your company and more importantly b) your life as a corporate innovator, product manager or consumer intelligence manager is getting a lot easier as you don’t need to care about DPAs, data privacy statements, legal approval .. ok, let’s stop here.

‍

2. How does this testing work?

An in depth explanation of the Pretotyping process can be found here. 

At it’s core, you are exposing real-life consumers (usually via paid social) to real-life stimulus and analyze their behaviour with that stimulus.

How that actually looks is very simple:

1. You have a social media ad, which directs consumers to a single landing page with your product on, or, if you would like to increase level of purchase intent, multi-stage landing pages.
2. The consumers then click on defined call-to-actions, like 'add-to-cart’, which are tracked. Depending on the customer journey, the consumer understands that the product is not available.
   Note that, before collecting any email address or other personal data from consumers, you need to inform them on your page (e.g. in a pop-up after clicking "add-to-cart" and before requesting any personal data, such as an email for adding them to your newsletter list) that your product is not available. It’s crucial to make sure they are aware of this before they submit any personal data. We have an additional piece that easily explains on how to tell consumers your product is not available. You can find it here.
3. The consumers, after being informed about the context of the test and made aware of the product not being available, if they want to get notified when the product becomes available, can then input their details into a form, normally name and email address.
4. They are sent an email to confirm their email address.
5. Then are given a verification that the email address confirmation has been concluded and depending on the customer journey, an additional % off of an existing product in your webshop

It’s really that simple. The key is that at every stage, data is tracked, analysed and presented in the Horizon dashboard so you can tell what products, or variations of a product perform the best. 

If you’ve ever been in marketing, you’ll probably know the term ‘A/B testing’. It’s essentially a comparable version of that, only that Horizon allows you to acquire behavioural / quantitative data by running so called Flights before you release any product decision to the market.

3. What insights can I expect?

There are many data points gathered in the Pretotyping process, and the insights gathered from them really depends on the purpose of your test. A full breakdown of the data and insights from Pretotyping can be found in this blog.

Typically with the tests we run, it comes down to conversion rates on the landing page(s) and, optionally, email confirmation. By looking at these data points, the insights gained will be around how high the demand for that product is. 

As an example: if one user journey (from advert to resolution) has a 20% conversion rate and your other variant only has a 10% conversion rate, you can see a significantly better relative performance of the variant with the 20% conversion rate.

‍

4. How reliable are the results?

The results from Pretotyping tests are highly reliable as the testing process is mimicking real-life scenarios with real audiences (no panels!). Audiences can be even sourced hyperlocally, i.e. you can run a test only with a certain age group in LA +10 miles.

The way the tests are setup, with advertising and landing pages make the consumer believe the product is live and ready to buy, because of this, it removes as much of the friction and bias present in consumer surveys and other research methods, whilst improving the quality of insight.

The other reliable side of Pretotyping is that everything is quantifiable and tracked. Any action a consumer takes either on the ads or landing page is tracked, so you can build up the real-world story from the data.

This makes your life as innovator, product manager or consumer insights manager so much easier when it comes to get buy-in from your superiors. This data is real-life and it usually can’t be questioned (if tests are set up correctly - we are here to help, get in touch here).

‍

5. How long does it take to get the data and insights?

The amazing thing about Pretotyping is that data comes in rapidly and can be viewed in real-time.

A typical test will run for 7 days, with data streaming in the entire time, allowing you to analyse on the go and even stop the test early if you feel confident in your results.

With the test-design and development of test stimulus, speaking landing pages and ads, you should expect to add another two week onto the process, depending on how quickly these can be created.

In total, you can expect data within 3 weeks.

6. What is the budget involved?

The budget is entirely up to you, you can run tests within whatever budget you have available. The only difference is the quantity of data, the quality will remain the same with whatever budget, but you should consider the budget you need to reach significance in data.

We see an average  advertising budget of around €1000 per landing page variant. But, it really depends on the product, the audience you are sourcing and the level of sample size you are aiming for. 

‍

7. Do we get access to the leads that are being collected with a Flight?

It depends on how you setup the data management. If you choose our Incognito package, where Horizon is the legal entity processing and managing lead data, then unfortunately not due to privacy regulations. The benefits of incognito testing can be seen in the next section.

If you choose to process the data yourselves, i.e. providing privacy policy and imprint yourself, then you can access and download the lead data generated by these tests.

‍

8. What does “Incognito Testing” mean and why is it important for you?

Horizon provides the imprint and legal infrastructure for your ads and landing pages so that your company brand or legal entity is not associated with the research project by consumers and competitors.