Audio Science Review

[shadow.clone]

I quite enjoy reading Amir's DAC reviews on ASR. He sticks to his test results and doesn't pollute his reviews with subjectivity, allowing readers to make their own conclusions.

He recently posted a rather shocking review of the Audio-gd NFB28.28 over here: https://www.audiosciencereview.com/forum/index.php?threads/review-and-measurements-of-audio-gd-nfb28-28-dac-and-headphone-amp.5147/. Perhaps there is some truth to the bashing certain Audio-gd products face on SBAF? Admittedly, some of the distortion on the NFB28.28 is quite low down and may not be audible, but I think it's refreshing to have someone call out manufacturers for poor performance in their products (anomalies or otherwise). I feel this benefits the community greatly - SMSL, for example, are sending Amir another review sample of the SU-8 to review, so I presume this unit will have fixed/lessened the distortion he measured on its unbalanced outputs (I doubt cherry picking the review sample will remove this distortion, so its probably firmware/circuit refinements).

I have no reason to doubt the validity/reliability of his results. What do you guys think of his reviews?

PS: I do believe that cables make a difference. I mention it in case you think I listen through an oscilloscope.
PPS: I've posted here as Audio-gd are well known in the headphone community and the NFB28.28 is an all in one unit. Mods, please move to the DAC section if needed. 

[GoodEnoughGear]

Not everything you hear can be measured. Not everything you measure can be heard.

In my very personal opinion he's a supercilious bloody **** who likes most to look like he knows what he's doing and bathe in the adoration of his followers. I suggest you research a little of his history and make up your own mind.

I am a known SBAFer, fair warning .

[shadow.clone]

Not everything you hear can be measured. Not everything you measure can be heard.

Exactly. He doesn't make inferences beyond his test results - you decide what to make of it. I believe it's a valuable reference for DAC measurements. It may even dissuade manufacturers from making outlandish claims (a benefit to the community). I'm not saying it should be the only "data point" used in consumer decision making; his tests may not pick up all issues exhibited by a DAC after all, but it's one way consumers can start to make sense of all the options out there. Imagine if CPU/GPU reviews had no objective data, because "everyone runs different software and ancillary equipment" or the CPU/GPU in question "ran games better than all CPU/GPUs used in the past so tests don't matter".

In a way, I think his reviews are analogous to Tyll's measurements and subjective feedback - Tyll's descriptions may not have been "accurate" to some readers due to differences in preferences/hearing/ancillary equipment, but he was quite "precise"/consistent over time and he served as an important frame of reference.

In my very personal opinion he's a supercilious bloody ****

Irrelevant to me if he provides valid/reliable test results.

...who likes most to look like he knows what he's doing...

An example of him not knowing what he's doing?

[gLer]

An example of him not knowing what he's doing?

Have you ever heard an NFB-28.28? Once you do you'll have your answer.

[kenvanraas]

Strap 'yerself, this is going to be a huge/amazing thread 
Science vs ......
Usb sound quality.Schiit products.Sinad.budget equipment.then the testing equipment.snake oils and what not.

[gLer]

Strap 'yerself, this is going to be a huge/amazing thread 
Science vs ......
Usb sound quality.Schiit products.Sinad.budget equipment.then the testing equipment.snake oils and what not.

Haha, very possibly. Huge can of worms indeed! If you really think a $299 Topping DX7 is better than a $2399 Schiit Yggy because it 'measures better', good for you I say. You just saved upward of R20k on tech you don't need. Go take the family on holiday or something. Maybe that's the point anyway

[capetownwatches]

An example of him not knowing what he's doing?

Right here:
https://www.audiosciencereview.com/forum/index.php?threads/budget-dac-review-schiit-modi-2-99.1649/

Amir unfortunately doesn't know his way around an analyzer, and those measurements have been shown to be erroneous.

Here are some contrasting measurements of the Modi 2:
https://www.superbestaudiofriends.org/index.php?threads/measurements-are-hard.5159/#post-165046

The above post gives some perspective. To quote Marv:

1. Measurements are a pain the in ass, and they are very easy to screw up.
2. Having expensive gear like an AP doesn't mean it's any easier getting them right.
3. Beware of rando measurements on the Internet.

[capetownwatches]

Have you ever heard an NFB-28.28? Once you do you'll have your answer.

+1

Amir stuffed up those measurements as well.

[shadow.clone]

Have you ever heard an NFB-28.28? Once you do you'll have your answer.

I haven't. I'm in the market for a DAC to pair with my Gustard H20. Hence my research into commercial DACs.

Now, your assertion seems to be that since the NFB-28.28 sounds great and Amir published a bad review of this DAC/amp, it therefore follows that he doesn't know what he is doing? Presumably you have an issue with the validity and/or reliability of his results:

• Validity - he's measuring metrics which have no tangible effect on perceived audio quality
• Reliability - his measurements can't be trusted

I'm inclined to believe that his measurements are reliable (i.e. he knows how to use his equipment, he can be trusted to not fake his results, etc.), but I accept that there may be issues around validity when readers use his test results to gauge overall sound quality (not just the performance of the DAC/amp). As I've agreed above, there is definitely a disconnect between measurements and what can be heard. It's pretty obvious - on a simplified level we have the following overall system:

Source -> Headphones/Speakers -> Ears -> Brain -> Perceptions (i.e. "what can be heard")

Measuring one part of the system such as THD/SINAD of the source (input) won't indicate exactly what perceptions will be reached by every individual (outputs). Heck, the ear/brain/perception transfer functions (whatever they may be) are different between individuals and will also be changing continuously (age, mood, etc.). However, you'll have to agree that measurements of the source are still strong indicators of the perceptions that will be reached. To totally discard Amir's measurements due to validity is rather strange to me, especially since it's one of the few parts of the entire system can be measured objectively. My opinion is that the ear/brain/perception transfer functions can compensate for lacking audio quality using factors such as looks, price, expectations ("it has to sound great because XYZ said so"), etc. Over the years I've gotten the sense that "audiophiles" are prone to lots of compensation at the ear/brain/perception level.

Imagine an exhibit of a red painting. Tests are conducted and the light reflecting off it is measured to be 680nm in wavelength. However some people, upon seeing the painting, say "No! Tests are wrong! Painting is actually Grey!". I personally wouldn't argue with them. Perhaps they're colorblind and are really seeing grey? Or, and this is more likely, they're a group of "audiophiles" that happened to stumble into a gallery? /s

[shadow.clone]

Right here:
https://www.audiosciencereview.com/forum/index.php?threads/budget-dac-review-schiit-modi-2-99.1649/

Amir unfortunately doesn't know his way around an analyzer, and those measurements have been shown to be erroneous.

Here are some contrasting measurements of the Modi 2:
https://www.superbestaudiofriends.org/index.php?threads/measurements-are-hard.5159/#post-165046

The above post gives some perspective. To quote Marv:

1. Measurements are a pain the in ass, and they are very easy to screw up.
2. Having expensive gear like an AP doesn't mean it's any easier getting them right.
3. Beware of rando measurements on the Internet.

So there's some issues around validity/reliability in his test results. I personally wouldn't use a "rando's" results to disprove another rando's results, but this is good thanks.

Some further reading is warranted.

[shadow.clone]

Right here:
https://www.audiosciencereview.com/forum/index.php?threads/budget-dac-review-schiit-modi-2-99.1649/

Amir unfortunately doesn't know his way around an analyzer, and those measurements have been shown to be erroneous.

Here are some contrasting measurements of the Modi 2:
https://www.superbestaudiofriends.org/index.php?threads/measurements-are-hard.5159/#post-165046

The above post gives some perspective. To quote Marv:

1. Measurements are a pain the in ass, and they are very easy to screw up.
2. Having expensive gear like an AP doesn't mean it's any easier getting them right.
3. Beware of rando measurements on the Internet.

After a quick skim over the threads, it seems that USB noise leaked into the output? It's not unreasonable to expect owners of the Modi 2 to run into similar issues if Amir didn't run/connect anything out of the norm on/to his PC during his tests. At least now they'll know that a USB decrapifier can improve the DAC's performance.

Testing every DAC in its ideal environment is not easy or quick to do. Still a valuable data point for consumer decision making.

[gLer]

I don't necessarily take issue with his measurements. I'm sure he's well practiced in what he does, and the measurements are what they are. What I do take issue with is how valid those measurements are in determining the overall sound quality/character of the dac/s in question, but more than that, the relative impact of that quality (or lack thereof) on the entire chain (which you described so well above).

From what I've read and from personal experience, the headphones/speakers make by far the largest contribution to the the sound you hear from the chain (pre your ears and brain), in the order of 60%-80%. It's arguable whether amps have more impact than dacs (or vice versa), but either way the amp/dac combo contributes at most 20% to the final sound. Then there's cables (both USB and headphone), interconnects (if you use them), and don't forget the source (Mac/PC/streamer) - which is often the biggest quality culprit - and then the source files themselves (this is all moot if you're going to be playing mp3s on your fancy system).

Once all the above is factored in, you also have to factor in synergy between amps, dacs and headphones. Some headphones are quite picky, and prefer certain amps (or dacs) over others. Low impedance headphones generally don't do well on OTL tube amps, for example. Neither do most power-hungry planars. And some amp/headphone combos just seem to be made for each other (and often are - headphone designers are known to favour certain amps when tuning their headphones).

Now, going back to Mr Audiosciencereview, you can see how measuring one component that has a relatively small effect on the overall sound of a headphone system WITHOUT balancing the results with the real-world impact of those measurements and against different types of headphones (or acknowledging that other parts of the chain - like a dirty source - could well be even more detrimental to what you ultimately hear), is just poor science in my opinion. The actual measuring part is fine, I'll take him at his word, but he almost completely misses the point.

He also makes the (wrong, in my opinion) assumption that something that measures well will necessarily sound good. It's this "measure first, listen second" approach that always gets my goat, especially when it comes to something so subjectively variable as high-fidelity audio.

PS. Don't even get me started on mastering and recording quality - take a well measured dac, amp and headphone (which, if you follow Amir's logic, will be perfectly neutral, distortion-free and transparent) and use it to play some heavily compressed modern music (take your pick, most modern music is compressed for 'loudness'). Trust me when I say your ears won't thank you.

[gLer]

And after you've read and absorbed my words of wisdom above (jk), here's an article that describes my position far more eloquently: http://www.6moons.com/audioreviews/fragilesouls/fragilesouls.html

[Nidri]

Some Audio Designers' views on the Role of Measurement:

http://www.herronaudio.com/images/Measurements.pdf

(It's from 1999 but still a good read. I love Nelson Pass's levelheaded approach.)

RE the broader conversation. 

As with any contentious topic, it's impossible to resolve unless we all operate off the same Base Assumptions (BA's).
So it's always worthwhile to do a Base Assumption Analysis (BAA™).

In this 'hobby' of ours, there are many prevalent BA's.

Faithfully recreating the live musical event.
I don't attend live musical events. I don't like crowds, I don't like noise, I don't like concerts, I don't like dressing up and going to auditoriums, I don't like any of that stuff. If anything, I might be trying to recreate a live studio event. (Actually, also nonsense given the amount of processing, conversion and mixing involved.) I like to listen to Led Zeppelin and Doors albums. Simple as that.

The pursuit of 'high fidelity'.
I'm not pursuing anything. There is nothing lofty or noble about what we do. I like to listen to music. I prefer certain pieces of equipment over others in terms of how that music sounds to me. I accept that this is subjective, but if I'm deceiving myself (or letting myself be deceived) about the role of the specific equipment in all of this, then so be it. I'm more than happy to believe a beautiful lie. And if I actually enjoy the sound of distortion, then so be it, leave me be.

The Practice of 'Critical Listening'.
That doesn't sound like any fun at all. No thanks.

The Pursuit of Neutrality.
Switzerland is neutral. Can you think of any Swiss a) Rock Stars, b) Marvel Superheroes or c) Supercars? Didn't think so.

My point is, most of the disagreement/debate/anger I've encountered RE this most contentious topic, flows from the assumption (haha) that all 'audiophiles' subscribe to some or all of the BA's listed above. As if we need that kind of validation. I certainly don't. I don't susbscribe to any of the above. In fact, those BA's end up being the sticks that objectivists try to beat us with, e.g. Ah, but you said you're after high fidelity, then why do you listen to this tube amp, you're deceiving yourself. Or whatever.

Anyway, I've said too much already.
BAA™ and other topics will be covered in my forthcoming book BAA™ and other topics.
Catch me on the lecture circuit.

 

[shadow.clone]

Now, going back to Mr Audiosciencereview, you can see how measuring one component that has a relatively small effect on the overall sound of a headphone system WITHOUT balancing the results with the real-world impact of those measurements and against different types of headphones (or acknowledging that other parts of the chain - like a dirty source - could well be even more detrimental to what you ultimately hear), is just poor science in my opinion. The actual measuring part is fine, I'll take him at his word, but he almost completely misses the point.

He also makes the (wrong, in my opinion) assumption that something that measures well will necessarily sound good. It's this "measure first, listen second" approach that always gets my goat, especially when it comes to something so subjectively variable as high-fidelity audio.

He doesn't make such claims. He simply tests a given device in the audio chain and, as stated above, leaves it up to the reader to make their own conclusions.

He does state which devices he would/wouldn't buy, but that is based on his personal criteria (i.e. his measurements) and plain to see. I don't rely solely on measurements in my purchasing decisions.

Let's compare his approach to the SBAF thread posted by capetownwatches. At a glance, they seem to have gotten mighty upset with Amir since he posted a bad review of what I presume to be a much loved DAC of theirs. They then proceed to slag him off, rather than Schitt for producing a product in keeping with their namesake. The group think/bias there is rather strong. Heck, maybe some unsuspecting members were living with the same USB noise because the DAC is supposed to be great according to other members (ear/brain/perception compensation)?

There's a place for subjective feedback, why not measurements too, even if its just to keep manufacturers in check? It's not hard to imagine the next iteration of Schitt Modi having "aerospace/NASA-inspired/trickle-down/hocus-pocus USB filtration technologies" which massively reduce USB noise as a result of his measurements. 

NB: I'm definitely not saying all distortion is bad and THD is everything.