Minimum Detectable Effect (MDE) in Optimizely Web Experimentation is a critical statistical measure. It defines the smallest detectable improvement or change that an experiment should be capable of identifying. MDE is essential for experimenters, guiding decisions about experiment design, cost assessment, and traffic allocation.
It sets the threshold for the smallest meaningful effect an experiment should detect, helping organisations make informed choices about the feasibility and expected impact of their experiments. In essence, MDE ensures that experiments are well-structured and justifiably resource-intensive in terms of traffic, time, and cost.
Understanding Minimum Detectable Effect
Minimum Detectable Effect (MDE) is a statistical calculation in Optimizely Web Experimentation. In simple terms, it represents the smallest relative change in conversion rate that warrants detection. To illustrate, if your baseline conversion rate is 20%, and you set an MDE of 10%, your experiment will identify any conversion rate shifts beyond the absolute range of 18% to 22%. This translates to a 10% relative effect, equivalent to a 2% absolute change in conversion rate.
Minimum Detectable Effect serves various essential purposes within Optimizely Web Experimentation, playing a key role in different areas of experimentation planning and design:
Experiment Sensitivity with Minimum Detectable Effect
Minimum Detectable Effect quantifies an experiment’s sensitivity, determining its ability to detect meaningful changes. By setting an appropriate MDE, you establish a threshold for recognising significant effects and avoid chasing insignificant fluctuations.
Evaluating Costs with Minimum Detectable Effect
MDE plays a pivotal role in cost considerations for experiments. It helps assess whether the expected outcomes justify the investment in terms of traffic volume, time, and resources. Realistic MDE values enable a clear assessment of cost-effectiveness, as it is an input for determining the sample size per variation, thus how long an experiment will need to run.
Experiment Design Considerations with MDE
Experiment design and scope are directly influenced by MDE. It guides decisions about variations, their number, and the extent of differences required for achieving statistical significance. Experimenters can use MDE to fine-tune their experiments efficiently. For example, if you want to detect a large effect within an experiment, then you should consider making a large change to a variation to accommodate this.
Using Minimum Detectable Effect for Estimation
MDE aids in estimating experiment duration by combining additional parameters like baseline conversion rate and statistical significance level. Optimizely’s Sample Size Calculator facilitates this calculation, aiding experimenters in planning effectively by knowing how long an experiment will take to run, based on the calculated required number of visitors per variation.
Experiment Prioritisation with MDE
MDE is instrumental in experiment planning decisions. It helps prioritise experiments by assessing their expected impact and sensitivity. Therefore, when planning your experimentation roadmap, you may want to order you experiments from highest MDE to lowest, so that your most impactful experiments are executed first. This strategic approach ensures that high-impact experiments receive due attention.
Minimum Detectable Effect Best Practice
Effective use of Minimum Detectable Effect in Optimizely Web Experimentation involves adhering to best practices:
Consider Business Impact: Evaluate an experiment’s potential business impact when setting the MDE. Revenue-focused experiments may justify a lower MDE, even with higher traffic requirements.
MDE as a Guide: View MDE as a guiding principle rather than an exact prediction. It helps set boundaries on time investment and expected outcomes.
Focus on Impactful Variations: When dealing with traffic limitations, concentrate variation designs on elements directly affecting the primary conversion event. This focused approach maximises the chances of obtaining meaningful results.
In essence, MDE in Optimizely Web Experimentation is a valuable tool that helps organisations set realistic boundaries for their experiments, ensuring that the investment in terms of traffic, time, and resources is justified by the expected impact and sensitivity of the experiment.