I’ve been playing around with Optimizely to run A/B and multivariate testing. I’d thought I share my experience around this optimization efforts.
I came across Optimizely during eMetrics and Conversion summit in San Francisco. I happened to be in the breakout session where Optimizely founder Dan Siroker was presenting. He was the Director of Analytics for the Obama Presidential Campaign and Deputy New Media Director for the Presidential Transition. In his presentation he went through the slides showing the learnings on various tests he ran for Obama’s fund raising.
To make sure the proper page is being tested, I added additional parameter and value to make sure it is not impacting similar URL else where.
Another challenge that I encountered was the swapping dynamic element. For example, some images like Trust-e or Geo Trust logos are generated dynamically. So when you use the “move” feature on Optimizely, it doesn’t move the entire logical element, so the page gets distorted when you push the test.
The customer support from Optimizely team is really fast, technical, and friendly. It was actually pretty surprising given that I am using the 200,000 visitor plan, but getting the survice level like an enterprise customer. I hope they don’t change that aspect of the service, it really helps to have a quick and accurate turn around on issues that could impact the bottom-line.
I have to say the setting up the multi variate piece of the tool is a little clunky, given my experience from using Webtrends Optimize. However, the price and service Optimizely is just so compelling it really doesn’t matter. What matters is really churning out the test and get back the results, and move on to drive the lift in conversions.
What I learned from few tests is that it goes back to basic fundamental thing.
Plan your testing wisely.
Because you can quickly set up tests and run little tests here and there, I felt that I was missing the big opportunities to get huge ROI. In my view, good planning would mean to do your planning due diligence:
- Use your analytics tool measure where the low hanging fruit is in terms of consumer pain points. Find high traffic page with high bounces, traffic from campaign where landing page is not effective, etc.
- Create hypothesis on why the page is not effective. Ask yourself why does that page exist, what does that page needs to do
- A/B testing or MVT. They are two different tactics, and require different amount of resource/efforts to deliver. So think and plan wisely.
- Set the right expectation with business partners/clients that getting results may take time to get accurate and significant read
- Define your conversion and make sure key stakeholders know what you’re optimizing against
Seriously, these things are really important. Optimizely allows you to run adhoc tests really quickly, but it could also disappoint you if you’re not geting the results as expected after waiting few days or maybe weeks.
With Google Anlaytics integrating their Google Website Optimizer, and companies like Visual Website Optimizer (similar to Optimizely) emerging, the barrier to do website optimization is becoming lower and less difficult.
I would say lastly, don’t be afraid and just go testing and learn quickly from your mistakes.