Licenses, Fine Print, and Benchmarks
Posted 10/14/2009 at 7:57 AM by Brent OzarIf you’re like me, you probably don’t bother reading through all the fine print before accepting software licenses. When I’m awake at 3 AM and I need to fall asleep, I’ll install a new piece of software and read the End User Licensing Agreement (EULA) to put myself out. Works every time.
Rock a bye baby, licensed as follows...
Buried in the fine print in arcane legalese, however, is some stuff that really matters to you: you probably can’t publish performance benchmarks or competitive reviews of software without getting approval from the vendor.
For example, here’s a quote from VMware Server’s EULA:
You may use the Software to conduct internal performance testing and benchmarking studies, the results of which you (and not unauthorized third parties) may publish or publicly disseminate; provided that VMware has reviewed and approved of the methodology, assumptions and other parameters of the study. Please contact VMware at benchmark@vmware.com to request such review.
When I first read about this, back before I came to work for Quest, I hit the roof. How dare they stop us, the community, from testing their software and writing about it? People need to know about the exact performance overheads of virtualization before they implement it. We need to work together to share this information, right?
The more underground reviews I read, the more uninformed opinions I got from friends, and the more I saw bad VMware implementations, the more I understood why the license prohibited benchmarks. There are some really lazy users out there and some really shady virtualization implementations – just like there’s bad SQL Server implementations. You know who I’m talking about – the folks who install everything on the C drive and then have to keep shrinking their log files so the OS doesn’t run out of drive space. If those folks published performance benchmarks, they’d be worse than useless.
In VMware’s corporate blog, they talked about why their license prohibits benchmarking in their EULA, and it does a great job of laying out the problems involved with letting just anybody publish reviews.
But I’m Special! I Should Be Allowed To Publish Benchmarks!
Some users take the time to really learn the technology, dive deeply into performance bottlenecks, and set up a specialized test rig to benchmark stuff. I used to do this with SAN gear, and I know how much work is involved. All it takes is one math error, one configuration problem, or one unseen hardware issue, and the results are completely garbage.
Perfect example – I’ve got SQLIO testing code up at SQLServerPedia to help people test their storage subsystems. I had a reader tell me that when he used that code on the latest solid-state gear, he got so many IOPs that my text parsing code broke and chopped off the leading numbers, thereby giving him falsely low results. It’s an awesome problem to have, but the only reason I found out about it was because somebody else took the time to dive into my code. It’s not as if I tossed that code into production without testing, either – I’d tested it against dozens of SANs and hundreds of drive arrays without an issue.
Even if you’re completely qualified, someone else has to double-check your work. The vendor has to:
- Set up an application process
- Vet the candidates
- Babysit the ones who don’t get approved
- Work with the ones who DO get approved
- Double-check their results with internal test gear
The Vendor Has to Reproduce the Results
No matter whether our software came in worst or first, we would want to reproduce it internally and then figure out how to get better. It’s just like carmakers: you’d better believe that before each new car goes to the EPA for testing, the automaker already knows exactly what mileage to expect. Before each car is sent to a car magazine, the maker already knows the 0-60mph times. They know all of the possible tests that each reviewer will run, and they’ve already run ‘em.
Software and hardware testing is just as challenging. Comparing SQL Server backup software performance on a text-filled database hosted on a single-CPU box with 1gb of memory and SATA drives is completely different than testing a binary-filled database on a HP SuperDome with 128gb of memory and an EMC SAN. Before someone publishes a set of competitive benchmarks, we need to know that they’re not performing some obscure edge use case and saying it’s representative of the software as a whole.
This puts me in an odd position. As a performance freak, I want to know exactly how our stuff performs – especially how it stacks up against the competition. But that’s where it gets really crazy, and I’ll talk about that in my next post.
