Split Test Calculator & Decision Tool
Lately, I've been writing more about work that's not just academic to me, but very relevant to my current client work, such as split testing. As an independent consultant doing this work strategically for smaller companies, I don't always have access to some of the more expensive tools and techniques, meaning I have to invent a few of my own.Recently, I was looking for a better split test calculator, to help me determine statistical significance on some of my simpler (A/B) split tests. There are a few out there that, from a mathematical standpoint, work perfectly well, but I found that either they weren't a good fit to usability work or they didn't provide much in the way of explanation.
So, I decided to build one of my own. My split test calculator (also available under the "Articles" tab) was designed with a couple of key features in mind: (1) automatic conversion rate calculation, (2) confidence level calculation, and (3) predictions for tests that haven't reached significance. Item (3) was key for me. It's, admittedly, not something that a statistician might completely approve of, but the calculator attempts to project how many more visitors you would need to make your test statistically viable. It's intended primarily as a decision tool to assess whether it's worth continuing a test, purely from a practical standpoint.
I hope others find this tool useful. I'll be refining and hopefully adding features to it as I go, so I'd welcome any comments.
