on March 19, 2012
with Comments Off
It is a best practice within the ecommerce and consumer internet industry to constantly “test, learn, and refine,” and one that Nor1 wholeheartedly embraces. The reality of product development is that sometimes small things make a big difference and sometimes big things make no difference at all, or they produce negative results. Experimentation is an excellent tool for finding out which is which.
Our philosophy at Nor1 is that we should constantly be running experiments and measuring the results of subtle changes that are quickly deployed to a subset of hotel properties or guests. We encourage our employees and customers to embrace a data-driven culture such that we remove the opinion from decision-making and learn to trust the evidence. Sometimes we simply confirm what we already thought, but often we are surprised by what we learn, and in this way we open the intrepid doorway to innovation.
Learn More, Upgrade More?
One recently concluded experiment was to test the impact of the words “Learn More” on the eStandby Upgrade® call to action image. The wording had always been on the image and we wanted to verify that it helped drive guest interest and subsequent upgrade requests.
It turns out that by removing the words Learn More from the bottom right hand corner, we were able to increase conversion of upgrade requests by 11%!
Key Experimentation Tenets
There are many factors that influence the design and execution of a successful experiment. Below we highlight many of the key tenets:
1) Understand what you are testing. Start with a clear hypothesis, define the metrics that matter, and the business decision to be made. At the end of the experiment you want to take an action that is based on what you learned.
2) Randomly expose guests to the variants. It is critical to employ random sampling techniques to ensure guests that are exposed to an experiment are representative of the entire population. By doing this, we can be more certain that the results of the experiment are generalizable and are due to what we tested, as opposed to a confounding factor that was not considered.
3) Get statistically significant results as fast as possible while mitigating risk. In order to make decisions quickly, we need to expose enough bookings to our various treatments. We start by first rolling out the experiment to a select number of properties to ensure the mechanics are working as planned. Once we are confident that the experiment is working as intended, we then expose additional properties to ensure we have a representative sample and sufficient volume.
4) Have procedures in place for quick rollbacks if necessary. It’s important to identify any key metric movements that would indicate an immediate need to stop a test. By doing so, we can ensure that there is no major harm done to guest experience, brand image, or revenue metrics.
5) Trust the data. During the planning of the experiment, it is critical to use judgement when deciding what to measure along with any caveats to consider. A simple rule of thumb is to collect the data the helps make a business decision. Once an experiment is completed and statistically valid, it is essential to trust the evidence even if it’s counter to what you thought would happen.
6) Communicate findings to stakeholders. Each successfully completed experiment provides insights about consumer behavior. It is our goal to share relevant information to key stakeholders on a periodic basis to help further their understanding.
Continuous optimization is a core principal and practice at Nor1, and by following these six tenets of experimentation, we will continue to accelerate the pace of innovation on behalf of our customers.