Conversion 101: When to Survey and When to A/B Test

When I started thinking about conversion, I assumed that the best way to identify conversion improvements is to A/B Test them. A/B Testing is a process where a company will change an element of its site for different users and see which version of the element performs better. For example, 50% of people will see a red button and 50% of people will see a blue button. The company will then see which color button gets more clicks and use that permanently on the site.

This is an accepted best practice for big companies who are trying to maximize sales on their e-commerce platforms.

However, as I've been researching Conversion Rate Optimization, I've found that this is only a best practice for sites that generate large amounts of traffic. The general rule is that A/B Testing is the best course of action if your website has >100,000 visits in 2 - 4 weeks. Getting 100,000 visits or more will give you a sample size large enough to reduce the risk of skewed results, and running a test for 4 weeks or less will prevent other variables (like seasonality, business cycle changes, economic shifts) from impacting your results.

So what should you do if you run a smaller e-commerce site? Since A/B Tests require a lot of time and effort to set up, they are not the best option. For smaller sites you will get a better ROI by conducting interviews with customers, user testing, and surveys. Getting in touch with your customers directly and seeing what they like/don't like about the site or asking friends who have never used your website to try to accomplish basic actions will cheaply get the same results as an A/B Test.

Bennett