It’s not a secret that Intellivy loves data. One way to retrieve data is by A/B testing. It can be a great tool to learn more about your customers’ preferences. However, Intellivy isn’t a fan. Here’s why and what to do instead.
What is A/B testing?
A/B testing, also known as split testing, is a randomized experiment where two or more variations of a webpage are shown to users. Statistical analysis determines which page performs better based on conversion rates.
It sounds great, doesn’t it? It’s a systematic method that helps you to make business decisions based on data. True. A/B testing isn’t all that bad. It can help you to go from 80% to 90% in optimization. Now, going from 20% to 80% that’s where the real problems are. A/B testing is based on data. This data comes from traffic. When there isn’t enough traffic, the results of the A/B test will tell you less than nothing. You need a certain amount of traffic in order to get reliable results to help you determine which page is the best.
Requirements for reliable A/B testing
In order to complete a reliable A/B test, you need to have at least 100 sessions per day. Plus, at least 10 sales.
Referring back to the percentages in optimization, Intellivy believes you are around 60% optimization when your product is getting 2-3 organic sales per day on Amazon. With this number of organic sales, you’ve been able to convert a potential customer into a buyer. Although this number will not make you an instant bestseller, you did what 70% of products on Amazon will never accomplish: Selling 2-3 units per day on average. You probably won’t settle for this number, and it’s also not enough traffic to run an A/B test.
What Amazon says about A/B testing
According to Amazon Experiments, you are restricted to run an A/B test on a product that doesn’t get enough traffic. Amazon Help states: “An ASIN is eligible if it belongs to your brand and has received enough traffic in recent weeks to be eligible for experimentation. We only let you experiment with high-traffic ASINs to increase the likelihood that you can confidently determine a winner at the end of the experiment. Depending on the category, high-traffic ASINs may get several dozen orders per week or more. When selecting an ASIN to experiment on, MYE (a new Amazon function: Manage Your Experiments) will show the eligibility status of most candidate ASINs, but note that ASINs with very low traffic may not appear at all.”
Without enough traffic, you are unable to A/B test. And without A/B testing, you are unable to get more traffic. Rock, meet hard place.
But even if you are eligible for running Amazon Experiments, it’s a slow process. As Amazon Help states: “The recommended experiment duration is 8-10 weeks.” Clearly, it’s not a quick win. And, if you want the results of your tests to be reliable, you can only test 1 element [i.e. color, angle, title, word choice, etc.] at a time. A/B testing 1 element will take you almost 2 months. Meaning that if you want to test 5 elements with 4 variations (20 A/B tests on Amazon Experiments), it will take you over 3 years until you get results.
What’s Rapid Microtesting?
Intellivy ran into this problem time and again. This is the reason why we started looking for a solution. Something that would give similar results as A/B testing, but lightning-fast and without needing actual traffic. Since a tool like that didn’t exist yet, we decided to build one ourselves. Introducing: Rapid Microtesting. A shortcut where you are able to simulate a similar situation as an A/B test without needing the same amount of traffic fand time. With Rapid Microtesting, you will collect data from an audience that is similar to your customers. They will answer your questions. This information will tell you everything you need to know to improve your product/listing and get more traffic, a higher conversion rate, and more organic sales.
The 5 most important elements of Rapid Microtesting:
1. Frame the mindset of your audience
Before asking your audience a question, the audience needs to believe that they are truly a customer looking to buy a product. Imagine that you sell dog treats to dog owners. In order for your audience to believe that they are a dog owner searching for a treat their dog will love, you need to illustrate this situation in such a way it comes alive in the mind of the audience. Essentially, framing the mindset of your audience is describing the situation of your customer so vividly that the audience becomes your customer.
There are two ways to frame the mindset of your audience:
Conscious framing: The situation of the customer is illustrated in the mind of the audience via writing or speaking.
Subsconcious framing: The situation of the customer is illustrated in the mind of the audience via visuals or images.
Rapid Microtesting uses both methods: you’re able to use conscious framing through the questions you ask the audience. Subconscious framing happens for instance in the Marketplace Poll, where the Amazon marketplace is simulated in detail. This frames the mind of our members to answer as if they were shopping on Amazon.
2. Formulate a clear and concrete question
To collect data, a question needs to be asked. This question will be answered by your audience. In order to receive valuable data from the audience answering your question, the question needs to be clear, concise, and concrete. In other words: Does your question give you the data that you need to understand the preference of your customer.
Formulate your question like this: “(Imagine you have [problem], and you found out there was a solution like [solution], which of these products you would buy and why?)”
For example: You are the owner of an elderly dog with osteoporosis. The vet has recommended dog food enriched with protein and vitamin E. You are searching on Amazon for dog food according to the vet’s advice. These are your search results. Which of these products would you buy and why?
3. Setup a test set to create a baseline and variations
In this step, you will determine where your product currently stands among competing products. This will be your baseline. You will test a set of current product images, titles, or other elements. Variations will be tested and will give you insights into how your product performs compared to competitors. Make sure that your test set is representative of reality. Test against your competitors instead of random products or you will not get data representative of the marketplace.
4. Calculate your perfect sample size
In order to be able to analyze trustworthy data, a certain number of responses is required. Generally speaking, the more responses, the more reliable the results. Please note that this explanation is somewhat heavy on the theoretical side of statics. If you don’t want to go through this calculation anytime you’re going to Rapid Microtest, make sure to include at least 33 respondents for every variation.
A. The number of people in your population
Who is your population [i.e. your customers] and how many are there?
B. Decide your error margin
There’s always a chance that the results are wrong. That’s just the way tests work. Sometimes, they’re incorrect. By deciding on your error margin, you are determining the risk you’re willing to take. Intellivy recommends an error margin of no bigger than 10%. The smaller your error margin, the more responses you need.
Others [i.e. people a lot smarter than us] have already calculated the number of responses you need based on your population size and error margin.
Here are the number of responses you need:
5. Select the right audience
During the previous step, you have calculated the perfect number of responses you need to get reliable results. Now it’s time to use the age and gender selection to get an audience that comes close to your actual audience.
Via Intellivy’s Rapid Microtesting, it’s possible to analyze your product at any time: from its development stage, to optimizing an Amazon listing.
Are you still struggling with Amazon A/B tests that take forever? Not touching your listing for 5-8 weeks, which will result in a lower sales velocity because of an unoptimized listing? Are you letting the precious honeymoon period pass by without making the most of it?
Or do you go all in, fully confident that your listing will crush your competition, based on the proof that Rapid Microtesting has provided to you?
Sincerely,
Mels Terlouw
Intellivy