I recently read a research paper entitled Bayesian Statistics and the Efficiency and Ethics of Clinical Trials. While mostly technical, it asks a few important questions about clinical drug trials that are interesting to think about but which also have a direct link to marketing and website experimentation. Let me explain:
But these trials can raise some ethical questions. Is the purpose of a clinical trial to help patients or to advance research? Hopefully both. But when these two purposes start to be at odds with each other, what do you do?
Imagine you are a cardiologist (a heart doctor) and you have an experimental drug to try. You have 100 patients that you divide into two groups. One group will receive the standard treatment, and one group will take the new drug.
You know that the standard treatment works about 50% of the time. As you bring on patients into the new drug group, it starts to look like the success rate is much lower than 50%. About half way through the test you are pretty convinced that the new treatment is not helping your patients. Now what do you do?
If you stop the trial and switch the patients back to the standard treatment, they have a better chance at getting better. But by stopping the experiment early, you are removing any statistical certainty from the experiment. There are pretty strict mathematical rules when it comes to statistical significance in experimentation.
So what do you do? Do you be a good researcher or do you be a good doctor? Do you help your patients or do you advance scientific research?
It is a hard question.
Let’s take it out of life and death and move it to an easier arena… Marketing. Similar trials can also done in marketing. We can divide our audience in half, send one group a tried and true campaign, and send the other a new campaign. If both campaigns are trying to sell the same product, we can measure how many we sell from each and know which campaign is better.
But what if half way through the experiment it starts to look like the new campaign isn’t selling as much as the original campaign? Do you stop the new campaign and send everyone back to the original? You’ll probably make more money that way but you’ll sacrifice any statistical significance in the experiment.
What do you do? Do you be a good researcher or do you be a good marketer?
It’s the same question and it’s just as hard here – even though the stakes aren’t as high.
The paper offers has a few solutions to this problem in the clinical trial world that we can apply to our marketing world:
1. Create a Data Monitoring Committee who is separate from the researchers the are performing the tests. This puts some separation between the patients and the decisions which makes the hard decisions a little easier. For most marketing situations, this is overkill.
2. Create a set of auxiliary endpoints. Most experiments have one main goal that they are trying to maximize but there are often side effects that can be measured too. For example, you may choose product sales as the goal of your experiment but you also care about the bounce rate as well. You can set up some additional endpoints for the experiment that trigger the experiment to stop prematurely if one of these side effects starts to go out of a specified range.
3. Use a Bayesian Statistics model for the experiment instead of a standard experiment. The details are complicated but the difference can be easily explained.
In a standard experiment, you divide your sample into two groups and run each group through the experiment. After all the groups have been tested, you see which one had more successes and that is considered the winner. This model leads to the moral problems that I mentioned above.
On the other hand, instead of waiting until the end, a Bayesian model will continually measure the probability that one group is better than the other. At the beginning, the groups are 50/50 – they both have an equal chance. As you start to run tests, those probabilities start to change. After a while, they may be 70/30. This doesn’t mean that it’s 70% better but that there is a 70% chance that it is better. There is a 30% chance that it’s the same or worse. If one of the groups is actually better, after a while the probabilities will continue to drift towards that one. This method lets you stop the experiment at any time and know the probability that one of the groups is better than the other.
As you can see, the connection between clinical drug trials and your website is not that big of a stretch. I knew this when I wrote Title Experiments and that is why I decided to use Bayesian Statistics at the heart of the plugin. Not only is it used to save lives but it helps you write better titles for your website as well. I call that a win win scenario!