It’s the time of year for saving money!
Last week I called up a manufacturer and cancelled a review. I haven’t always had that “luxury.” But, sometimes you have to say “no,” both for reader’s and the manufacturer’s sake…
Back in my past I once wrote for a publication that had an “ironclad” policy – if I received a product slated for review in that publication I HAD to review it, regardless of how good or bad it was. The policy allowed for defective product (get another review sample), but didn’t specify whether the defect had to be mentioned in the course of the review…some reviews did, and others did not…
While this policy does allow a publication to keep its street cred up to snuff. It does not insure a level playing field for all manufacturers. If a newly minted audio firm sent in their gear for review, hoping for that golden ring of recommendation, they could just as easily wind up with a dead company from one bad review…I’ve seen it happen more than once.
Since I write reviews for several different publications, sometimes I really don’t know who a review will be placed with, and who it will be most appropriate for, until I receive the component and have put it through its paces. Also, timing plays a big part in my review choices. The Cary DMS-500, which I reviewed for Home Theater Review (soon to go live), was not ready for a full review when I received it, but after several firmware updates it was, so then I reviewed it. Occasionally a cool product is not really right for any of my other regular outlets but still worthy of your attention, then I write a review for Audiophile Review.
So, what would make me pass on a review of a product that I have received? Several reasons, but the most important one is if I feel I can’t properly review the component. Sometimes that is because I can’t get it perform properly in my listening environment – that’s usually a loudspeaker that just doesn’t work in my room. It happens. Very few rooms, outside of well-designed purpose-built listening rooms, permit any and all speaker designs to all perform optimally.
During the last several years I have gotten several digital products that didn’t make the cut. One was a wonderful sounding DAC that had this bad habit of occasionally sputtering out random noise. Shipping damage was the claimed culprit. The distributor chose not to repair and resend, which was probably a good idea.
This “room doesn’t work” issue can also be applied to earphones – sometimes I come across an earphone design where I can’t get a proper fit. If I can’t, after trying all the alternatives get a good occluded fit, I don’t review the earphones. That’s basically what happened with the earphones I passed on reviewing – they were wireless in-ears that I could not use in active situations because the left ear enclosure would not stay seated correctly for more than a minute or so without slipping and needing adjustment. None of the supplied tips worked any better at delivering a good fit, and since the whole point of wireless in-ears is so you can use them in active situations, the poor fit was strike one, two, and three…
After being an audio journalist for as many years as I have, my own street cred isn’t of much concern to me. In my humble opinion writing a “bad review” just to “keep the edge” is more an exercise in ego than one of consumer support. There are so many great new products from solid companies that consumers have multiple good choices in every product category. So why, pray tell, should ink (or electrons) be wasted on the mediocre ones? And lately finding genuinely BAD products is as hard as finding perfect ones…
I try not to write “good reviews” or “bad reviews” but reviews that outline a products strengths and weaknesses in a way so that the reader can decide for themselves, based on their own requirements and tastes, whether a component would be appropriate for them. And sometimes I come upon a product that either I can’t review because it won’t perform properly, or is not a good match for my room, ears, or my way of using a component to the point that I can’t or won’t review it.
So, that’s why I rejected those earphones…
If there are so many good products why can’t they play Pet Sounds? Or any of my eight other reference albums for that matter? Those get three out four audio components rejected. Then I play my three reference recordings and 60% of the albums that survived my reference albums test are rejected. I’m generally left with about one in ten components that will properly play my reference albums and recordings before we can even get to is this better?
Maybe it’s your room?
Unlikely even Art Dudley has noted banjos are hard to reproduce, Herb Reichert has noted problems with Cajun saw fiddling. I’ll trust my ears on the harmonica reference recordings I make.
Too many rooms over the years in any case to assume its the room.
Over the years I’ve heard more rooms than I could count that turned good-sounding electronics into drek…Using a harmonica as a reference sound is a lot like Peter, of Peter, Paul, and Mary who used the recorded sound of his own voice to EQ rooms…Sounded great unless there was a bass…
It just one piece of the puzzle. And I’m talking about rooms that sound good with the equipment I have in them.
Steve I wonder if this is more of an issue of the products simply not meeting your particular use case and expectations. The way you have explained your concerns, it sounds like it’s your subjective judgement that they didn’t do a good job. Maybe another person would feel differently.
Like Steven Stone, I have personally found that there are a lot more good products on the market. Not perfect, but at least by basic objective and subject measures, very good. Modern technology, engineering, and manufacturing efficiencies have allowed the production of products that are outstanding for what they are. I have an amplifier that costs $50, fits in my pocket, and with a reasonable 8ohm load sounds pretty good. It isn’t a great amp, but it’s $50 and fits in my pocket. This kind of product didn’t exist 20 years ago. I have a $200 integrated amp based around a chip amplifier that has unbelievable measured performance. It’s distortion and S/N are more on par with 5 figure amps. It’s sound is actually quite good. Again, the similar priced mass produced brands aren’t there and have never been there. This is a big sea change.
I’m sure not everyone will feel those products sound good. They won’t meet their use case. However, for what they are, I’m amazed they even exist.
One of my main priorities in a top end system is the ability to provide realistic scale and spaciousness. I want the system to be able to make me feel like this is the real event (even if an artificially created edifice of the event) and not some small scale facsimile of the event. By that definition some 99% of all speakers and amps wouldn’t meet my criteria. While that is important to me for my own system, I am careful not to dismiss products that fail to achieve that because I recognize the technical barriers and how unrealistic it is to expect that from most products.
Maybe the same is true for you?
Interesting comments. I’ve been happy with the components in my office and home for a very longtime. So I’ve found what works. Nothing is or was particularly unusual or expensive. I want Pet Sounds to be a wall of sound and the drum kit on All the Young Dudes not to be stretched so my priorities are more realistic than yours and were easily attainable in the past.
I too am having to say NO more often in my new role as Managing Editor of HomeTheaterReview.com. Simply put, we get hit up with so many review opportunities that we couldn’t possibly say “yes” to everyone.
BROKEN gear is a massive frustration and a total lose-lose. Nobody wins and time and money is wasted. You do your best to work through it but sometimes you just have to say – we are sending it back.
Very interesting topic! I’ve never once thought about a reviewer declining to review a product.
What I like about AR’s & HTR’s reviews is that I’m not reading an editorial. It’s never “this product sucks.” It’s “this speaker isn’t as good at producing bass when playing rock so you may need to add a sub.”
Keep up the great work!
I had a great conversation with Dan Wiggins about why some in-ear headphones fit better than others. Why some work for some people and not others? He had some great thoughts that may actually be important in reviewing. He noted that overall weight and the center of gravity matter. If the weight is too high, they may fatigue the ear and pull themselves out. Further, if that weight is concentrated more toward the outside end of the device, it will pull itself out. In his designs he specifically uses low mass materials and concentrates the weight so that shy naturally want to fall inward toward your head.
Yes, fit is critical for earphones. Personally I’d rather use an earphone with a great fit and average sound than one with great sound that simply does not fit well.
I’ve been getting trained on how to measure headphones. The iem’s Are the hardest to measure. People don’t realize that very tiny differences in insertion and even tip type make HUGE differences in the response. If I get a chance I’ll post an example.
With the Apple EarPods I measured them numerous ways. The Apple engineer I spoke with told me that two of my 6 measurements match two of their internal measurements for different fit scenarios. One shows rolled off bass below 100hz and the other shows a basically flat rising response past 20hz. Both are considered correct. One is for the average ear where a good fit is not achieved. The other is for an average ear that happens to fit around the ear bud shape better. To me that is nuts. Both are correct yet wildly different.
I have recently reviewed a product that gave me real anguish. While I was rewriting the final copy, I happened to run into Michael Fremer. As a newly minted reviewer he had some sage advice, don’t be afraid to call a company out, but don’t waste your time writing negative reviews. I do tell people, if I haven’t reviewed something, it may be for a reason. If asked and I say no, it’s likely that I already know it is flawed and don’t want to write that review. In the case of this particular product, I knew that the product had an intentional design flaw and that there was a plan to secretly fix it without saying anything. The consumer won’t be made aware, it will just happen. I knew that this would happen by the time my review posted and I had heard the change. However, I couldn’t say any of that. What do I do? Write about the product I reviewed? Send it back and say I can’t review this? Break trust and reveal what I learned?
I’ve also sent back products that were sent to me which I felt were utter nonsense. Some of which I tried to be open minded about, but heard exactly what I expected to hear. I won’t name names, but let’s just say that one was nicknamed by a fellow reviewer as a very expensive wood chip. In this case I made an explicit effort to be polite but honest with the maker. I did talk about it publicly, but stated clearly that while I don’t personally believe in the veracity or the claims nor do I believe that I could hear a difference, other should draw their own opinions.
In the bad old days it was routine for some countries’ reviewers to also be consultants. They would get a prototype, rip it to shreds, and the manufacturer made corrections. Then the product was released and reviewed by the same folks who were consultants who delivered rave reviews. The public never saw the flawed products…
I don’t think those bad old days are done. This is still happening.
Geez, so in other words these companies didn’t have employees or focus groups, for lack of a better word, who were willing or capable of providing honest or meaningful evaluations/critiques of their own products? Strange.
Frankly that’s why I suggest sticking to the old Stereo Review format but adding a short note on sound quality by comparison to recent or long-term reference components – IN THE SAME SYSTEM. But look…Stereophile is doing away with the test and measurement stuff….HMMMM….that doesn’t bode well for the consumer.
If you cannot get earphones to fit, and if your goal is to write for consumers rather than manufacturers, I think most of us would want to see that review. Your ears are surely more attuned than mine but I am guessing they are not so far different in size than mine or most other people, and most of us don’t want to waste time ordering earphones that are unlikely to fit us and will either have to be shipped back or the cost eaten. The manufacturer might not like it, but it also might be more likely to go back to the drawing board and offer more fixes/fits.