I am severely, severely jealous of web developers because of how easy it is to test, measure, and iteratively improve this. Using your judgment is nice, using your data is muuuuuuch better.
I have one screen in my workflow which I realized was costing me 20% of trial users (thank you, Mixpanel). That was clearly unacceptable.
I redid it to introduce more complexity to the interface (it has the same number of total options as before PLUS a way to show/hide several of them) but hide most of it from novice users.
When I checked my stats for yesterday this morning, it looked like I was losing only 10% at that step now, not 20%. I have no idea whether that data is statistically significant, but just the fact that I'm even in a position to collect it, for practically free, amazes the desktop developer in me.
Now I just need to get my A/B testing framework working so that I can get automated statistical significance tests done on it, and then I can start doing that sort of iterative improvement on an industrial scale, rather than a series of ad hoc one-offs as I've been doing for a while.
Is there a way you can incorporate the A/B testing into an actual product?
You can release two pieces that are functionally equivalent but have a slightly different UI on it. If you send stats back to the server you can probably get a good amount of data.
You mean like downloadable software? I did it once.
There are many ways to distribute Java applications to customers. That probably bores you. To make a long story short, the tradeoff is usability versus download size.
I made two functionally identical (or so I thought -- long story) versions of BCC, and randomly redirected half of the people downloading from my site to one (1 MB, required JRE) and half to the other (10 MB, did not).
I was able to check which versions were causing sales by incorporating a URL parameter naming the version of the executable when people clicked from the nag prompts to the website. Set the version they're using in a cookie, associate it with a later purchase, blah blah bob's your uncle.
Early testing suggested that the larger one was statistically significantly more effective at producing sales, but this trend did not continue.
There are numerous downsides with this. It took a LOT of work to set up. I got to maintain two versions of the same executable, which introduced bugs (I'll spare you the details, aside from one: Vista). It cost me in support requests. I will be supporting those executables for years to come (trust me -- I still get people buying from software not on my website in 2+ years). Changing the A/B tests used takes a full release cycle.
On the web, these are all so easily surmountable it isn't even funny.
I have one screen in my workflow which I realized was costing me 20% of trial users (thank you, Mixpanel). That was clearly unacceptable.
I redid it to introduce more complexity to the interface (it has the same number of total options as before PLUS a way to show/hide several of them) but hide most of it from novice users.
When I checked my stats for yesterday this morning, it looked like I was losing only 10% at that step now, not 20%. I have no idea whether that data is statistically significant, but just the fact that I'm even in a position to collect it, for practically free, amazes the desktop developer in me.
Now I just need to get my A/B testing framework working so that I can get automated statistical significance tests done on it, and then I can start doing that sort of iterative improvement on an industrial scale, rather than a series of ad hoc one-offs as I've been doing for a while.