Does A/B Testing Make You Nervous?

Back in the 1990s, when I owned XLO, we provided, in addition to our products, a large amount of sales and technical training and support information for our dealers, both about the cables, themselves, and about how to sell them. The technical information was, of course, developed "in house", but much of the other information ― about how to sell hi-fi and how to promote and run a successful store ― came directly from the experience of our dealers, themselves, which we asked them to tell us about so that we could share it with others in an extensive series of for-XLO-dealers-and-factory-reps-only "How To" articles called "Hot Tips".

AR-abtest2.jpgThere were "Hot-Tips" about practically any audio-related subject you could think of, but one of the (to me) most surprising, shared information that we had gotten from one of our strongest dealers, but that, because it was SO contrary to conventional belief and practice, we (I and my sales staff) insisted on verifying with nearly two dozen of our other dealers and their top salespeople before I was willing to  release it for distribution.

The subject of so much checking and re-checking was "A-B" comparison testing and, once verified, we thought it so important and so significant that it was also written-up (so they could help dealers with it), in our Factory Rep's training manual and in the manual we provided for training dealer salespeople.

Basically the advice about A-B testing was "Unless you really have to do it, DON'T!". 

What dealers all across the country had found was that A-B comparisons often did more to make customers uncomfortable and to actually drive them out of the store than to help them in making their purchase choices. The reason for this was simple but thoroughly understandable, once you stopped to think about it: Knowing that they were not experts and believing that the salespeople serving them were, a surprising number of customers felt that a comparison test was more of a test of their judgment or hearing ability than of the equipment being auditioned, and they were afraid of picking the "wrong" thing and making themselves look foolish in the eyes of the salesperson.

AR-abtesting4.pngPerhaps it was the idea of "high fidelity" that got them: High fidelity implies a high level of fidelity TO something. And that, in turn, implies some sort of an absolute standard of performance instead of just a simple matter of picking whichever product you like best. Whatever it was, our dealers told us that a significant percentage of their customers were "turned-off" by A-B comparisons and preferred to simply have their "expert" salesperson give them the benefit of his knowledge and tell them which product to buy, instead of making them have to guess at it and not only run the risk of embarrassing themselves, but of buying the wrong thing.

Coming from non-audiophile "man in the street" customers, that sort of thinking is not unreasonable: They're looking, perhaps for the first time, at making a, to them, highly technical and perhaps very expensive, purchase of goods that they know they know nothing about, and it seems only natural that they should have a little trepidation about doing it right. It's sort of like dancing is to some people who do it seldom or never - you go out onto the floor to execute some, to you, complicated moves knowing full well that everybody is watching and that it's altogether too likely that you'll fall down and make a spectacle of yourself - and it becomes a chore instead of a source of fun. 

That's certainly okay for "newbies", but for seasoned audiophiles to have the same kind of fear of making the wrong decision if they trust just their ears and not "scientific" test results or the opinions of self-declared "experts" is simply absurd.

AR-abtest3.jpgHi-fi, like dancing, is supposed to be fun. It's NOT a thing where - except in the very grossest sense (like frequency response or distortion characteristics, for example) - there's any kind of absolute standard by which to judge system performance. Yes, I know that Harry Pearson insisted on comparing everything to "the absolute sound" - the sound of a live music performance, but how can you possibly ever do that? In the first place, the odds are overwhelming that you weren't AT the original recording session, so, except for guessing, based on your own tastes and preferences regarding how you think it OUGHT TO HAVE sounded, you really don't have any real "absolute  sound" to compare anything against. Another thing is that, with modern multi-channel recordings, it's entirely possible (except for certain "live" recordings) that there never was any original performance at all: The artists may all have come in separately or in groups and been recorded separately to be mixed into a complete performance at some later time. Even if there WAS a live performance, and even if you WERE there to hear it, unless you were at all of the locations of all of the microphones, (a neat trick if you can do it!) you still have no idea at all of what the recorded music really sounded like when it was originally performed.

The fact is that there is no "absolute sound". And why does there need to be? Isn't the whole purpose of music to be "ear candy"? And if the purpose of our equipment - even so-called "voodoo" products of whatever kind - is to bring us that candy, why should we need anything other than our own ears and our own tastes and preferences to evaluate it? If product "X" measures well and is loved by the experts and validated by the testers but I HATE it, am I supposed to ignore my own response in favor of theirs? Should I buy something I don't like just because someone else does like it? 

If those customers who are intimidated by A-B comparison tests would just set aside their doubts and trust their own ears, they might not get a "better" (whatever that means) system, but they might get one that they like more. Wouldn't you think that real-live-committed-audiophiles would learn that same lesson and try to grow some ears? And once they have them, learn to trust them instead of the words or charts, graphs, and "approved" equipment lists of all the so-called experts who are either trying to impose their own opinions derived from their own ears or to say that ears can't be trusted at all?

comments powered by Disqus

Audiophile Review Sponsors