Conversions don't tell the whole story
Conversion-tracking has recently become a popular tool for both marketing and usability professionals. Although tracking how often a visitor "converts" (into a request, download, purchase, etc.) is certainly a business-oriented question, it's also a reasonably good measure of how effective a website is at helping users to accomplish their goals.Like any metric, though, it's easy to get carried away with looking at conversion percentages out of context, and that can be a mistake, especially if you're testing site changes. Ideally, you should approach any testing scenario like a scientist, and create a hypothesis, thinking through the scenarios and predicting what effect they're going to have. That way, when the numbers come back, you have a way to evaluate them, and, in a perfect world, you'll be able to piece together some causality. Knowing that A is better than B is great, but better still to know why A is better, so that you can inform future decisions.
In addition, without that hypothesis, you may find yourself in a situation where the numbers lead you in the wrong direction. Case in point: one of my clients wanted to test featuring some lower-priced products on their home-page (versus moderately-priced products). We hypothesized that this would drive more conversions, but we also realized that this might drop our average price-point per purchase and ultimately hurt the client's revenues. Without thinking the situation through in advance, we never would've measured the average purchase price during the test. Ultimately, what we expected was exactly what happened. Conversions improved only a tiny bit (0.1%), but the average purchase price dropped almost $100.
Of course, from a pure usability standpoint, you could argue that the small increase in conversion (assuming it was statistically valid) meant that the lower-priced items were what the users wanted to see. That may be true, but our decisions are rarely "pure", and that tiny bit of personal preference wasn't worth a significant dent to the bottom line. Any measurement of site performance has to be viewed in a broader context; data is a valuable thing, but don't let it make your decisions for you.
