Skip to main content

Experimentation Platform

Accelerating software innovation through trustworthy experimentation

Home: ExP Platform
Journal Survey
Encyclopedia article
Objective Bayesian AB
Rules of Thumb
Two Stage
Large Scale
Puzzling Outcomes
Experiments at Microsoft
Practical Guide (short)
Tracking Users' Clicks an
Seven Pitfalls
Semmelweis Reflex
Power Calculator
What is a HiPPO
Pitfalls of Long Term

Objective Bayesian Two Sample Hypothesis Testing for Online Controlled Experiments


by Alex Deng


WWW 2015 1st Workshop of Online and Offline Evaluation of Web-based Services,

May, 2015, Florence, Italy. Paper, Slides(PDF). 



As A/B testing gains wider adoption in the industry, more people begin to realize the limitations of the traditional frequentist null hypothesis statistical testing (NHST). The large number of search results for the query ``Bayesian A/B testing'' shows just how much the interest in the Bayesian perspective is growing. In recent years there are also voices arguing that Bayesian A/B testing should replace frequentist NHST and is strictly superior in all aspects. Our goal here is to clarify the myth by looking at both advantages and issues of Bayesian methods. In particular, we propose an objective Bayesian A/B testing framework for which we hope to bring the best from Bayesian and frequentist methods together. Unlike traditional methods, this method requires the existence of historical A/B test data to objectively learn a prior. We have successfully applied this method to Bing, using thousands of experiments to establish the priors.