AVForums South Africa

Audio/Video Hardware => Headphones => Topic started by: shadow.clone on November 10, 2018, 01:18:55 PM

Title: Audio Science Review
Post by: shadow.clone on November 10, 2018, 01:18:55 PM
I quite enjoy reading Amir's DAC reviews on ASR. He sticks to his test results and doesn't pollute his reviews with subjectivity, allowing readers to make their own conclusions.

He recently posted a rather shocking review of the Audio-gd NFB28.28 over here: https://www.audiosciencereview.com/forum/index.php?threads/review-and-measurements-of-audio-gd-nfb28-28-dac-and-headphone-amp.5147/ (https://www.audiosciencereview.com/forum/index.php?threads/review-and-measurements-of-audio-gd-nfb28-28-dac-and-headphone-amp.5147/). Perhaps there is some truth to the bashing certain Audio-gd products face on SBAF? Admittedly, some of the distortion on the NFB28.28 is quite low down and may not be audible, but I think it's refreshing to have someone call out manufacturers for poor performance in their products (anomalies or otherwise). I feel this benefits the community greatly - SMSL, for example, are sending Amir another review sample of the SU-8 to review, so I presume this unit will have fixed/lessened the distortion he measured on its unbalanced outputs (I doubt cherry picking the review sample will remove this distortion, so its probably firmware/circuit refinements).

I have no reason to doubt the validity/reliability of his results. What do you guys think of his reviews?

PS: I do believe that cables make a difference. I mention it in case you think I listen through an oscilloscope.
PPS: I've posted here as Audio-gd are well known in the headphone community and the NFB28.28 is an all in one unit. Mods, please move to the DAC section if needed. 
Title: Re: Audio Science Review
Post by: GoodEnoughGear on November 10, 2018, 02:09:57 PM
Not everything you hear can be measured. Not everything you measure can be heard.

In my very personal opinion he's a supercilious bloody **** who likes most to look like he knows what he's doing and bathe in the adoration of his followers. I suggest you research a little of his history and make up your own mind.

I am a known SBAFer, fair warning :).
Title: Re: Audio Science Review
Post by: shadow.clone on November 10, 2018, 02:51:21 PM
Not everything you hear can be measured. Not everything you measure can be heard.

Exactly. He doesn't make inferences beyond his test results - you decide what to make of it. I believe it's a valuable reference for DAC measurements. It may even dissuade manufacturers from making outlandish claims (a benefit to the community). I'm not saying it should be the only "data point" used in consumer decision making; his tests may not pick up all issues exhibited by a DAC after all, but it's one way consumers can start to make sense of all the options out there. Imagine if CPU/GPU reviews had no objective data, because "everyone runs different software and ancillary equipment" or the CPU/GPU in question "ran games better than all CPU/GPUs used in the past so tests don't matter".

In a way, I think his reviews are analogous to Tyll's measurements and subjective feedback - Tyll's descriptions may not have been "accurate" to some readers due to differences in preferences/hearing/ancillary equipment, but he was quite "precise"/consistent over time and he served as an important frame of reference.

In my very personal opinion he's a supercilious bloody ****

Irrelevant to me if he provides valid/reliable test results.

...who likes most to look like he knows what he's doing...

An example of him not knowing what he's doing?
Title: Re: Audio Science Review
Post by: gLer on November 10, 2018, 03:43:01 PM
An example of him not knowing what he's doing?
Have you ever heard an NFB-28.28? Once you do you'll have your answer.
Title: Re: Audio Science Review
Post by: kenvanraas on November 10, 2018, 05:00:50 PM
Strap 'yerself, this is going to be a huge/amazing thread  :popcorn:
Science vs ......
Usb sound quality.Schiit products.Sinad.budget equipment.then the testing equipment.snake oils and what not.

(https://i.imgur.com/ZrOOs5T.jpg)
Title: Re: Audio Science Review
Post by: gLer on November 10, 2018, 05:34:07 PM
Strap 'yerself, this is going to be a huge/amazing thread  :popcorn:
Science vs ......
Usb sound quality.Schiit products.Sinad.budget equipment.then the testing equipment.snake oils and what not.
Haha, very possibly. Huge can of worms indeed! If you really think a $299 Topping DX7 is better than a $2399 Schiit Yggy because it 'measures better', good for you I say. You just saved upward of R20k on tech you don't need. Go take the family on holiday or something. Maybe that's the point anyway ;)
Title: Re: Audio Science Review
Post by: capetownwatches on November 10, 2018, 06:27:44 PM
An example of him not knowing what he's doing?

Right here:
https://www.audiosciencereview.com/forum/index.php?threads/budget-dac-review-schiit-modi-2-99.1649/

Amir unfortunately doesn't know his way around an analyzer, and those measurements have been shown to be erroneous.

Here are some contrasting measurements of the Modi 2:
https://www.superbestaudiofriends.org/index.php?threads/measurements-are-hard.5159/#post-165046

The above post gives some perspective. To quote Marv:

1. Measurements are a pain the in ass, and they are very easy to screw up.
2. Having expensive gear like an AP doesn't mean it's any easier getting them right.
3. Beware of rando measurements on the Internet.


Title: Re: Audio Science Review
Post by: capetownwatches on November 10, 2018, 06:33:37 PM
Have you ever heard an NFB-28.28? Once you do you'll have your answer.

+1

Amir stuffed up those measurements as well.
Title: Re: Audio Science Review
Post by: shadow.clone on November 10, 2018, 06:45:41 PM
Have you ever heard an NFB-28.28? Once you do you'll have your answer.

I haven't. I'm in the market for a DAC to pair with my Gustard H20. Hence my research into commercial DACs.

Now, your assertion seems to be that since the NFB-28.28 sounds great and Amir published a bad review of this DAC/amp, it therefore follows that he doesn't know what he is doing? Presumably you have an issue with the validity and/or reliability of his results:
I'm inclined to believe that his measurements are reliable (i.e. he knows how to use his equipment, he can be trusted to not fake his results, etc.), but I accept that there may be issues around validity when readers use his test results to gauge overall sound quality (not just the performance of the DAC/amp). As I've agreed above, there is definitely a disconnect between measurements and what can be heard. It's pretty obvious - on a simplified level we have the following overall system:
Source -> Headphones/Speakers -> Ears -> Brain -> Perceptions (i.e. "what can be heard")
Measuring one part of the system such as THD/SINAD of the source (input) won't indicate exactly what perceptions will be reached by every individual (outputs). Heck, the ear/brain/perception transfer functions (whatever they may be) are different between individuals and will also be changing continuously (age, mood, etc.). However, you'll have to agree that measurements of the source are still strong indicators of the perceptions that will be reached. To totally discard Amir's measurements due to validity is rather strange to me, especially since it's one of the few parts of the entire system can be measured objectively. My opinion is that the ear/brain/perception transfer functions can compensate for lacking audio quality using factors such as looks, price, expectations ("it has to sound great because XYZ said so"), etc. Over the years I've gotten the sense that "audiophiles" are prone to lots of compensation at the ear/brain/perception level.

Imagine an exhibit of a red painting. Tests are conducted and the light reflecting off it is measured to be 680nm in wavelength. However some people, upon seeing the painting, say "No! Tests are wrong! Painting is actually Grey!". I personally wouldn't argue with them. Perhaps they're colorblind and are really seeing grey? Or, and this is more likely, they're a group of "audiophiles" that happened to stumble into a gallery? :tongue: /s
Title: Re: Audio Science Review
Post by: shadow.clone on November 10, 2018, 06:47:21 PM
Right here:
https://www.audiosciencereview.com/forum/index.php?threads/budget-dac-review-schiit-modi-2-99.1649/

Amir unfortunately doesn't know his way around an analyzer, and those measurements have been shown to be erroneous.

Here are some contrasting measurements of the Modi 2:
https://www.superbestaudiofriends.org/index.php?threads/measurements-are-hard.5159/#post-165046

The above post gives some perspective. To quote Marv:

1. Measurements are a pain the in ass, and they are very easy to screw up.
2. Having expensive gear like an AP doesn't mean it's any easier getting them right.
3. Beware of rando measurements on the Internet.


So there's some issues around validity/reliability in his test results. I personally wouldn't use a "rando's" results to disprove another rando's results, but this is good thanks.

Some further reading is warranted.
Title: Re: Audio Science Review
Post by: shadow.clone on November 10, 2018, 08:56:16 PM
Right here:
https://www.audiosciencereview.com/forum/index.php?threads/budget-dac-review-schiit-modi-2-99.1649/

Amir unfortunately doesn't know his way around an analyzer, and those measurements have been shown to be erroneous.

Here are some contrasting measurements of the Modi 2:
https://www.superbestaudiofriends.org/index.php?threads/measurements-are-hard.5159/#post-165046

The above post gives some perspective. To quote Marv:

1. Measurements are a pain the in ass, and they are very easy to screw up.
2. Having expensive gear like an AP doesn't mean it's any easier getting them right.
3. Beware of rando measurements on the Internet.


After a quick skim over the threads, it seems that USB noise leaked into the output? It's not unreasonable to expect owners of the Modi 2 to run into similar issues if Amir didn't run/connect anything out of the norm on/to his PC during his tests. At least now they'll know that a USB decrapifier can improve the DAC's performance.

Testing every DAC in its ideal environment is not easy or quick to do. Still a valuable data point for consumer decision making.
Title: Re: Audio Science Review
Post by: gLer on November 10, 2018, 10:36:14 PM
I don't necessarily take issue with his measurements. I'm sure he's well practiced in what he does, and the measurements are what they are. What I do take issue with is how valid those measurements are in determining the overall sound quality/character of the dac/s in question, but more than that, the relative impact of that quality (or lack thereof) on the entire chain (which you described so well above).

From what I've read and from personal experience, the headphones/speakers make by far the largest contribution to the the sound you hear from the chain (pre your ears and brain), in the order of 60%-80%. It's arguable whether amps have more impact than dacs (or vice versa), but either way the amp/dac combo contributes at most 20% to the final sound. Then there's cables (both USB and headphone), interconnects (if you use them), and don't forget the source (Mac/PC/streamer) - which is often the biggest quality culprit - and then the source files themselves (this is all moot if you're going to be playing mp3s on your fancy system).

Once all the above is factored in, you also have to factor in synergy between amps, dacs and headphones. Some headphones are quite picky, and prefer certain amps (or dacs) over others. Low impedance headphones generally don't do well on OTL tube amps, for example. Neither do most power-hungry planars. And some amp/headphone combos just seem to be made for each other (and often are - headphone designers are known to favour certain amps when tuning their headphones).

Now, going back to Mr Audiosciencereview, you can see how measuring one component that has a relatively small effect on the overall sound of a headphone system WITHOUT balancing the results with the real-world impact of those measurements and against different types of headphones (or acknowledging that other parts of the chain - like a dirty source - could well be even more detrimental to what you ultimately hear), is just poor science in my opinion. The actual measuring part is fine, I'll take him at his word, but he almost completely misses the point.

He also makes the (wrong, in my opinion) assumption that something that measures well will necessarily sound good. It's this "measure first, listen second" approach that always gets my goat, especially when it comes to something so subjectively variable as high-fidelity audio.

PS. Don't even get me started on mastering and recording quality - take a well measured dac, amp and headphone (which, if you follow Amir's logic, will be perfectly neutral, distortion-free and transparent) and use it to play some heavily compressed modern music (take your pick, most modern music is compressed for 'loudness'). Trust me when I say your ears won't thank you.
Title: Re: Audio Science Review
Post by: gLer on November 10, 2018, 10:43:43 PM
And after you've read and absorbed my words of wisdom above (jk), here's an article that describes my position far more eloquently: http://www.6moons.com/audioreviews/fragilesouls/fragilesouls.html
Title: Re: Audio Science Review
Post by: Nidri on November 11, 2018, 09:13:20 AM
Some Audio Designers' views on the Role of Measurement:

http://www.herronaudio.com/images/Measurements.pdf

(It's from 1999 but still a good read. I love Nelson Pass's levelheaded approach.)

RE the broader conversation. 

As with any contentious topic, it's impossible to resolve unless we all operate off the same Base Assumptions (BA's).
So it's always worthwhile to do a Base Assumption Analysis (BAA™).

In this 'hobby' of ours, there are many prevalent BA's.

Faithfully recreating the live musical event.
I don't attend live musical events. I don't like crowds, I don't like noise, I don't like concerts, I don't like dressing up and going to auditoriums, I don't like any of that stuff. If anything, I might be trying to recreate a live studio event. (Actually, also nonsense given the amount of processing, conversion and mixing involved.) I like to listen to Led Zeppelin and Doors albums. Simple as that.

The pursuit of 'high fidelity'.
I'm not pursuing anything. There is nothing lofty or noble about what we do. I like to listen to music. I prefer certain pieces of equipment over others in terms of how that music sounds to me. I accept that this is subjective, but if I'm deceiving myself (or letting myself be deceived) about the role of the specific equipment in all of this, then so be it. I'm more than happy to believe a beautiful lie. And if I actually enjoy the sound of distortion, then so be it, leave me be.

The Practice of 'Critical Listening'.
That doesn't sound like any fun at all. No thanks.

The Pursuit of Neutrality.
Switzerland is neutral. Can you think of any Swiss a) Rock Stars, b) Marvel Superheroes or c) Supercars? Didn't think so.

My point is, most of the disagreement/debate/anger I've encountered RE this most contentious topic, flows from the assumption (haha) that all 'audiophiles' subscribe to some or all of the BA's listed above. As if we need that kind of validation. I certainly don't. I don't susbscribe to any of the above. In fact, those BA's end up being the sticks that objectivists try to beat us with, e.g. Ah, but you said you're after high fidelity, then why do you listen to this tube amp, you're deceiving yourself. Or whatever.

Anyway, I've said too much already.
BAA™ and other topics will be covered in my forthcoming book BAA™ and other topics.
Catch me on the lecture circuit.

 :coffee:
Title: Re: Audio Science Review
Post by: shadow.clone on November 11, 2018, 12:14:44 PM
Now, going back to Mr Audiosciencereview, you can see how measuring one component that has a relatively small effect on the overall sound of a headphone system WITHOUT balancing the results with the real-world impact of those measurements and against different types of headphones (or acknowledging that other parts of the chain - like a dirty source - could well be even more detrimental to what you ultimately hear), is just poor science in my opinion. The actual measuring part is fine, I'll take him at his word, but he almost completely misses the point.

He also makes the (wrong, in my opinion) assumption that something that measures well will necessarily sound good. It's this "measure first, listen second" approach that always gets my goat, especially when it comes to something so subjectively variable as high-fidelity audio.

He doesn't make such claims. He simply tests a given device in the audio chain and, as stated above, leaves it up to the reader to make their own conclusions.

He does state which devices he would/wouldn't buy, but that is based on his personal criteria (i.e. his measurements) and plain to see. I don't rely solely on measurements in my purchasing decisions.

Let's compare his approach to the SBAF thread posted by capetownwatches. At a glance, they seem to have gotten mighty upset with Amir since he posted a bad review of what I presume to be a much loved DAC of theirs. They then proceed to slag him off, rather than Schitt for producing a product in keeping with their namesake. The group think/bias there is rather strong. Heck, maybe some unsuspecting members were living with the same USB noise because the DAC is supposed to be great according to other members (ear/brain/perception compensation)?

There's a place for subjective feedback, why not measurements too, even if its just to keep manufacturers in check? It's not hard to imagine the next iteration of Schitt Modi having "aerospace/NASA-inspired/trickle-down/hocus-pocus USB filtration technologies" which massively reduce USB noise as a result of his measurements. 

NB: I'm definitely not saying all distortion is bad and THD is everything. 
Title: Re: Audio Science Review
Post by: gLer on November 11, 2018, 02:01:49 PM
I completely agree there’s a place for measurements. If something can be measured accurately and relatively, it should be. I’m just saying measurements play a much smaller part in my purchase decisions compared to, say, reliable feedback from experienced users and, if at all possible (which it hardly ever is in South Africa), personal experience. In fact I can honestly say I’ve never perused audiosciencereview for anything other than casual reading, certainly never as a precursor for buying anything.
Title: Re: Audio Science Review
Post by: kamikazi on November 12, 2018, 03:05:40 PM
It really depends on what you test. Normal test wave forms are usually quite simple, there is not too much you can read into them, but if it struggles to output simple wave forms you can bet it will struggle with more complex ones. Then there are trickier measurements that can trip up gear from time to time and they are often worst cases which could have little relation to actual music playback. So it can be difficult to relate to how measurements plays out in real listening tests. Then there are many types or signal errors, distortions and aberrations that can come out of measurements. Some are worse than others and more audible. Some would be inaudible unless you really know what you are looking/listening for. If you don't know what your doing with measuring gear, some of those can even be artifacts created by the measurement device itself.
Title: Re: Audio Science Review
Post by: shadow.clone on November 12, 2018, 11:22:17 PM
As usual it depends, which I do accept.

It's just odd that (bad) measurements sometimes don't matter because the differences are below the threshold of audibility, but in the same breath "audiophiles" can detect the differences between fuses and so on, which also can't be measured. Cuts both ways.
Title: Re: Audio Science Review
Post by: gLer on November 13, 2018, 07:12:32 AM
Don’t underestimate the power of placebo and expectation bias... There are other things as well that affect the way we perceive our gear, like visual impact. I’m much more likely to overlook minor flaws in sound if the gear is well designed, well made, and visually appealing, for example.
Title: Re: Audio Science Review
Post by: shadow.clone on November 13, 2018, 08:02:04 AM
Yup, covered by the ear/brain/perception compensation that I spoke about in post #8.

A more interesting bias is confirmation bias, i.e. ignoring Amir's reviews rather than considering its potential merits, just because the results don't match preexisting perceptions. Case in point: Modi 2 and NFB28.28 reviews.
Title: Re: Audio Science Review
Post by: gLer on November 13, 2018, 07:01:31 PM
I found this quote from him in one of his review discussion threads quite enlightening:

He used the ISORegen (a USB ‘cleaner’ which he panned by the way) to test noise reduction on a Schiit Modi, concluding it does indeed clean up the noise of that dac almost completely because the Schiit is so poorly isolated from PC noise. However, further in the thread he says this, in reply to a question from a user of all this ‘noise’ that he’s measuring is actually audible:

“Good question. Did a quick test with and without and I can't tell the difference. I also played with the front switch on ISO regen and again could not detect much.

That jives with psychoacoustics that says that wide skirt is masked by our main tone. And at any rate, nearly 0 db 12 Khz tone is not something that exists in music.

The other distortion products are too low level to be audible.

So yes, you have a good point that the audible improvements may not be there with ISO Regen. Indeed this may be the reason many have bought the Schiit Modi 2 and not complaining about any obvious fidelity problems.”

So...he slams products he reviews for having excessive noise that no-one can actually hear!?

That makes perfect sense 🤔
Title: Re: Audio Science Review
Post by: legro on November 13, 2018, 09:36:24 PM
The old subjective vs objective debate.

You listen to music and it sounds great - crap.

You phone friends, do measurements/ check others' measurements, read mags, consult experts and they tell you that you're wrong and it actually sounds crap - great.

Who do you believe, but really, who do you believe......???
Title: Re: Audio Science Review
Post by: shadow.clone on November 14, 2018, 08:16:59 PM
I found this quote from him in one of his review discussion threads quite enlightening:

He used the ISORegen (a USB ‘cleaner’ which he panned by the way) to test noise reduction on a Schiit Modi, concluding it does indeed clean up the noise of that dac almost completely because the Schiit is so poorly isolated from PC noise. However, further in the thread he says this, in reply to a question from a user of all this ‘noise’ that he’s measuring is actually audible:

“Good question. Did a quick test with and without and I can't tell the difference. I also played with the front switch on ISO regen and again could not detect much.

That jives with psychoacoustics that says that wide skirt is masked by our main tone. And at any rate, nearly 0 db 12 Khz tone is not something that exists in music.

The other distortion products are too low level to be audible.

So yes, you have a good point that the audible improvements may not be there with ISO Regen. Indeed this may be the reason many have bought the Schiit Modi 2 and not complaining about any obvious fidelity problems.”

So...he slams products he reviews for having excessive noise that no-one can actually hear!?

That makes perfect sense 🤔

You must remember that he caters for "audiophiles" too, i.e. those individuals capable of amazing feats of hearing and auditory memory. Although, rather curiously, many such individuals seem to draw the line at shakti stones :thinking:
Title: Re: Audio Science Review
Post by: shadow.clone on November 14, 2018, 08:25:56 PM
The old subjective vs objective debate.

You listen to music and it sounds great - crap.

You phone friends, do measurements/ check others' measurements, read mags, consult experts and they tell you that you're wrong and it actually sounds crap - great.

Who do you believe, but really, who do you believe......???

A bit of both is needed really. "Measurements don't matter" gives the manufacturers too much bargaining power in my opinion. Perhaps this is changing with the proliferation of consumer reviews, but it can go the other way with group think on forums. As an example, B&W was once the darling child on this forum. Not to say they produce bad products or have dropped off lately, but it seems that forum (to my perception) has shifted towards exploring other brands.
Title: Re: Audio Science Review
Post by: gLer on November 15, 2018, 04:23:09 PM
Read an interesting response we from Zach Merbach (Mr ZMF himself) in response to someone asking him what he thinks the correlation is between those who subjectively love tube amps (like he does) and those who objectively argue they measure terribly:

“Well there's a lot of ways to look at measuremenst, if you look at subjectively what you like and the types of distortion in amps that make a certain kind of euphony and holography, then you can look at those "bad" measurements and realize they are the very thing that make you appreciate X amp and vice versa. It's just about understanding how measurements affect your subjective preference.”

I like the idea of knowing what type of noise or distortion floats your boat and then knowing how to find those ‘flawed’ amps through their ‘flawed’ measurements. Indirectly he’s suggesting that just because something measures well doesn’t mean it’s going to sound good to everyone (and vice versa).
Title: Re: Audio Science Review
Post by: Katji on November 15, 2018, 05:24:57 PM
The old subjective vs objective debate.

You listen to music and it sounds great - crap.

You phone friends, do measurements/ check others' measurements, read mags, consult experts and they tell you that you're wrong and it actually sounds crap - great.

Who do you believe, but really, who do you believe......???
You "believe" your sense perceptions, but your perceptions are continually being influenced.  ...Finished and klaar. Unless you try to be selective/discriminating about the inputs you get.
Title: Re: Audio Science Review
Post by: legro on November 16, 2018, 08:00:05 AM
I guess what it boils down to is that I'm just too damn lazy to trust anything but my own ears.

From the endless discussions on this forum I gather it saves me a heap of non-musical-enjoyment-related trouble.
Title: Re: Audio Science Review
Post by: xenithon on November 16, 2018, 10:29:06 AM
The problem is that there is no reliable, defined definition of the term "something measures well" when in comes to audio. That is, the link between a measurement and the sound it produces.

Thus, they resort to testing audio equipment like they were medical kit or extremely sensitive electrical components, where you are aiming for flat everything - because that is the requirement. Those types of requirements have, for the most part, not been defined in the world of audiophilia.

I say "for the most part" as there are some interesting attempts - take for example the Harman target response curve. It is not ruler flat, at all. Yet it is often strived for in terms of creating the most natural sound, even for use in studios.

The types of measurements used these days and on that site? It's like using the Scoville scale to measure chocolate mousse.




Title: Re: Audio Science Review
Post by: capetownwatches on November 27, 2018, 06:22:11 AM
Here's an informed opinion on ASR and the veracity of the measurements to be found there.

Take a moment and watch this Currawong (Amos Barnett) video from 34:55 to around 36:00

https://m.youtube.com/watch?v=zjF9HnCNJKU (https://m.youtube.com/watch?v=zjF9HnCNJKU)


 :popcorn:
Title: Re: Audio Science Review
Post by: Rotten Johnny on November 27, 2018, 07:51:41 AM
Here's an informed opinion on ASR and the veracity of the measurements to be found there.

Take a moment and watch this Currawong (Amos Barnett) video from 34:55 to around 36:00

https://m.youtube.com/watch?v=zjF9HnCNJKU (https://m.youtube.com/watch?v=zjF9HnCNJKU)


 :popcorn:

 :clap:
Title: Re: Audio Science Review
Post by: Michon on November 27, 2018, 08:31:15 AM
Here's an informed opinion on ASR and the veracity of the measurements to be found there.

Take a moment and watch this Currawong (Amos Barnett) video from 34:55 to around 36:00

https://m.youtube.com/watch?v=zjF9HnCNJKU (https://m.youtube.com/watch?v=zjF9HnCNJKU)


 :popcorn:

I know next to nothing about about ASR and Amir.

Are there any more details about (or reference to) his appearance at an Audio Precision presentation requesting a unit that is easier to operate?
Title: Re: Audio Science Review
Post by: shadow.clone on November 27, 2018, 11:37:04 AM
Here's an informed opinion on ASR and the veracity of the measurements to be found there.

Take a moment and watch this Currawong (Amos Barnett) video from 34:55 to around 36:00

https://m.youtube.com/watch?v=zjF9HnCNJKU (https://m.youtube.com/watch?v=zjF9HnCNJKU)


 :popcorn:

https://en.wikipedia.org/wiki/Ad_hominem

If I was still involved with hardware development, I would also request easier to use test equipment. Pretty sure anyone would do so in any field with any equipment. The context of his question matters, otherwise ad hominem (i.e. the foundation of politics :tongue:).

Title: Re: Audio Science Review
Post by: Michon on November 27, 2018, 12:07:07 PM
https://en.wikipedia.org/wiki/Ad_hominem

If I was still involved with hardware development, I would also request easier to use test equipment. Pretty sure anyone would do so in any field with any equipment. The context of his question matters, otherwise ad hominem (i.e. the foundation of politics :tongue:).

This is partly why I ask. Amos does not go into detail except for saying that Amir is incompetent and mentions different measurement results from others, using a heat gun to heat components and an incident at an Audio Precision presentation as the reasons.

Is there a well written and evidence based critique of the measurements posted by ASR? If so, please link to it. I am not interested in a discussion about the correlation of measurements and perception, but rather the validity of his measurement methodology and the repeatability of its results.

I was made aware ASR earlier this year, but have yet to consume any of the content. If it can be shown that his measurements are fraught with error I will know not to waste my time.
Title: Re: Audio Science Review
Post by: capetownwatches on November 27, 2018, 01:14:44 PM
Is there a well written and evidence based critique of the measurements posted by ASR? If so, please link to it.

Personally I have no axe to grind with Amir at all, and in fact came to his defence when he appeared (briefly) on this very forum some time ago.

I started reading ASR about 2 years ago and liked (and still do, in theory) the science-based position Amir espouses.
That said, there are numerous instances where his measurements have contradicted some very experienced and respected posters' findings.

Here is just one: https://www.superbestaudiofriends.org/index.php?threads/measurements-are-hard.5159/

There are others, but I'm not going to do all the work for you lazy buggers!  :dop:

The problem I have with Amir's testing is that erroneous conclusions can lead to inexperienced folk making poor decisions.
Many don't have the time, inclination or technical knowledge to refute his findings, and might take them at face value and as fact.

How many potential buyers have given Audio-gd amps a wide berth WITHOUT LISTENING TO ONE following his test results, for example?
Title: Re: Audio Science Review
Post by: gLer on November 29, 2018, 04:26:58 PM
I don't know Amir from a bar of soap, and still read the reviews on ASR from time to time (for the sake of completeness of course). But one line from Currawong's video sums up my feelings about ASR (and hardcore objectivists in general): Tone sweeps are NOT music.

Enough said. 
Title: Re: Audio Science Review
Post by: naughty on November 29, 2018, 04:40:17 PM
Tone sweeps are NOT music.

sure .... but music is a collection of mutiple tones and frequencies .... so tone/frequency sweeps and analyzing various individual tones/frequencies are a relevant way of testing equipment for the purpose of music playback .... the problem is that many people do not know how to interpret the results of this analysis properly  :nfi:
Title: Re: Audio Science Review
Post by: amirm on December 17, 2018, 05:32:43 AM
Right here:
https://www.audiosciencereview.com/forum/index.php?threads/budget-dac-review-schiit-modi-2-99.1649/

Amir unfortunately doesn't know his way around an analyzer, and those measurements have been shown to be erroneous.

Here are some contrasting measurements of the Modi 2:
https://www.superbestaudiofriends.org/index.php?threads/measurements-are-hard.5159/#post-165046

The above post gives some perspective. To quote Marv:

1. Measurements are a pain the in ass, and they are very easy to screw up.
2. Having expensive gear like an AP doesn't mean it's any easier getting them right.
3. Beware of rando measurements on the Internet.

Folks on SBAF post plenty of random measurements.  I have found issue after issue that is present in their measurements yet they finish it with, "oh, isn't this a great product?" 

I got so tired of addressing the nonsense they post that i wrote an article on it.  See: https://www.audiosciencereview.com/forum/index.php?threads/are-measurements-of-schiit-yggdrasil-dac-inconsistent.3812/

In case after case, I show that their measurements are actually consistent with mine.  And where they differ is because they were given special versions of the product under the table.

As to my credentials, here they are: https://www.audiosciencereview.com/forum/index.php?threads/a-bit-about-your-host.1906/

Now, post and compare their credentials to mine.  I bet you don't even know what their credentials are.

I know as non-technical people it may be hard for some of you to adjudicate these things.  But please don't put aside your sensibilities.  Look up the print ads from Schiit and see the references to folks you are choosing to believe.   Here is that bit from my thread above:

(https://www.audiosciencereview.com/forum/index.php?attachments/1532317455481-png.14148/)

Does this read as someone who is impartial testing audio gear in dispute?

Let's remember that it is my work that eventually forced Schiit to go and buy a decent audio analyzer and produce good products like Schiit Modi 3. 

Title: Re: Audio Science Review
Post by: amirm on December 17, 2018, 05:37:13 AM
Haha, very possibly. Huge can of worms indeed! If you really think a $299 Topping DX7 is better than a $2399 Schiit Yggy because it 'measures better', good for you I say. You just saved upward of R20k on tech you don't need. Go take the family on holiday or something. Maybe that's the point anyway ;)
Good for him indeed.  Have you done a controlled blind test comparing the two?  I have.  If you can tell the difference between the two, run the test and if you can blindly tell them apart, I will send you USD $500 for your troubles.  Make a youtube video with someone making the AB switch, run a dozen times and we can talk.

Remember, Schiit has gone to the drawing board and fixed major issues like linearity in Yggdrasil as a result of my testing.  For the first time ever, they have started to post Audio Precision measurements for products such as Schiit Modi 3.  If measurements don't matter, why do they post these results now?  Think about that.  You are letting beliefs overcome data, facts and audio science.

A DAC has but one job: to faithfully convert digital samples to analog.  When it starts to do that incorrectly, then it is not a good DAC.  No way can you make a case that screwing up audio samples and generating distortion that is not in your music is "good for you."
Title: Re: Audio Science Review
Post by: amirm on December 17, 2018, 05:44:45 AM
What I do take issue with is how valid those measurements are in determining the overall sound quality/character of the dac/s in question, but more than that, the relative impact of that quality (or lack thereof) on the entire chain (which you described so well above).
That is not what I do.  I measure equipment to see if they are engineered well and do what the manufacturers claim.  Instrumentation together with interpretation of results quickly demonstrates the good from bad.  Bad designs are not there because someone thought those things were good ideas.  They are bad because the people who designed the gear either didn't know what they were doing, or lacked the equipment and knowledge to measure the gear to see if it is working right.

Most DACs today are using a DAC chip internally with superb specifications.  It is the implementation of that DAC chip that either preservers that performance, or screws it up.  Measurements are the absolute tool to show that.

Now, when problems are shown in measurements, I apply science of psychoacoustics as to whether they are audible problems or not.  So there are comments about audibility but based on sound science, not random folklore online.

And I have yet to hear a "character" in any DAC.  What you read about such things is all sourced in improper audio testing resulting in totally incorrect conclusions.  Do a level matched, controlled test of two DACs blind and then tell me one sounds warm and the other doesn't.  I have done this many, many times.  And I have done it sighted to know how wrong those conclusions are.

Ultimately what I bring to ASR Forum is the best of what audio science knows, and actual, concrete, repeatable data about performance of audio products.  This is causing a good change in the industry already, with companies modifying their products to fix things, designing new performant products, and in general taking proper design and verification seriously. 

Without it, we have wild west where any claims about audio fidelity works.  Guy says put a rock on your DAC and it sounds good and many people believe.  In what other area of your life are you this open to suggestions and dismissal of science?
Title: Re: Audio Science Review
Post by: amirm on December 17, 2018, 05:49:59 AM
It really depends on what you test. Normal test wave forms are usually quite simple, there is not too much you can read into them, but if it struggles to output simple wave forms you can bet it will struggle with more complex ones.
Sorry no.  If an audio product distorts one tone, then it will distort many tones worse.  I post 32-tone tests now in my reviews.  They show nothing new other than what was bad, is really bad.  :)  Here is an example from my review of Schiit Budget DACs: https://www.audiosciencereview.com/forum/index.php?threads/review-battle-of-schiit-audio-dacs.5487/

Schiit modi 3:
(https://www.audiosciencereview.com/forum/index.php?attachments/schiit-modi-3-multitone-dac-measurements-png.18292/)

Schiit modi multibit:
(https://www.audiosciencereview.com/forum/index.php?attachments/schiit-modi-2-multibit-dac-multitone-measurements-png.18293/)

See?  Other measurements of Schiit Multi-bit showed it to be much worse than Modi 3.  Multiple tones just multiplies the effect.

A DAC has no idea what it is being fed to it anyway.  It is all digital values to be converted to an analog voltage.  Spectrums like above are done with signal processing.  The DAC is dumb and does what it does to one tone that it does to many.
Title: Re: Audio Science Review
Post by: amirm on December 17, 2018, 05:57:29 AM
I don't know Amir from a bar of soap, and still read the reviews on ASR from time to time (for the sake of completeness of course). But one line from Currawong's video sums up my feelings about ASR (and hardcore objectivists in general): Tone sweeps are NOT music.

Enough said.
You don't know me from a bar of soap?  I need to get my name on some.  :D

Currawong needs to study audio science.  If he did, he would know that every lossy audio codec converts music to pure tones, applies compression there, and then converts it back when you play it.  If individual tones are not music, how come your MP3 sounds like music?  Of course Currawong and countless others who say these things are not technical and have no experience in signal processing which begs the question: why do you guys believe them?  Why not ask him for his credentials and ask him if he knows of any paper at Audio Engineering Society, ASA, IEEE, etc. backs what he says?

Folks, it is time that we all graduate from this folklore.  We can't keep believing lay people just because they do reviews.  There are decades of audio science that have answered all of this.  We don't get to invent new audio science on our own, and spend our money on them as if they are correct.  They are not. 

Let's not keep creating FUD and doubt around good data.  Let's encourage great engineering and design.  Why do we want audio to be excluded this way?
Title: Re: Audio Science Review
Post by: stereosane on December 18, 2018, 07:07:37 AM
But in the end measurements mean nothing if the sound isn’t good, and I’ve got an item that measures very bad yet is the best sounding component I’ve owned. Fair enough good electrical work should measure within specs usually but what if it sounds better by changing some of those parameters, is it still wrong or a bad design I don’t think so.. as long as it sounds good to me, there no one right sound, everyone hears things slightly different so perfect measurements and bad measurements will all sound different to different people..

I truly think that’s why so many people upgrade, bought an item because it reviewed well and specs are the best yet somehow when listening to it you just don’t feel the magic, like my Nad M51 Dac.. very very good test results yet I found it a bit sterile and cold, my Audio-GD Dac sounds like heaven to my ears compared to that Nad yet the Audio-GD measures poorly against it..

Its not about trusting your ears it’s about letting your ears hear how they hear which is very unique to each person :)

Title: Re: Audio Science Review
Post by: gLer on December 18, 2018, 07:44:49 AM
But in the end measurements mean nothing if the sound isn’t good, and I’ve got an item that measures very bad yet is the best sounding component I’ve owned. Fair enough good electrical work should measure within specs usually but what if it sounds better by changing some of those parameters, is it still wrong or a bad design I don’t think so.. as long as it sounds good to me, there no one right sound, everyone hears things slightly different so perfect measurements and bad measurements will all sound different to different people..

I truly think that’s why so many people upgrade, bought an item because it reviewed well and specs are the best yet somehow when listening to it you just don’t feel the magic, like my Nad M51 Dac.. very very good test results yet I found it a bit sterile and cold, my Audio-GD Dac sounds like heaven to my ears compared to that Nad yet the Audio-GD measures poorly against it..

Its not about trusting your ears it’s about letting your ears hear how they hear which is very unique to each person :)
You’ve just hit the nail on the proverbial head. The idea that every amp and dac should just do its job and get out the way of the music is all good and well, but doesn’t even start to explain why two components that supposedly do that sound completely different to each other. And we also tend to forget that the biggest variable of all is the transducer, but a very large margin too. You could have the world’s most ‘perfect’, best measuring amp and it will still sound like cr** with a rubbish transducer.

Bottom line: be guided by others with experience (and by that I mean listening experience, not qualifications - there are no ‘qualifications’ for knowing what to listen for) but only ever trust your own ears.
Title: Re: Audio Science Review
Post by: amirm on December 18, 2018, 09:15:43 PM
But in the end measurements mean nothing if the sound isn’t good, and I’ve got an item that measures very bad yet is the best sounding component I’ve owned.

[...]

Its not about trusting your ears it’s about letting your ears hear how they hear which is very unique to each person :)
You have no way of assessing what your ears received.  What you perceived is the combination of what the ear heard, and your brain decided.  The latter is highly adaptive. When listening to music for example, it is discarding 99% of what the ear is hearing.  Can you imagine it capturing every note, every nuance, every bit of sonic signal?  You would need a brain the size of everyone's combined.  :)

So what happens is that a highly lossy process is applied to your hearing when you enjoy music.  This is called translation from short-term/echoic to long-term memory.

When people perform tests of new product, they act different.  They use their brain to analyze fidelity.  Then they will hear detail, air, etc. that was always there but was missed before.  Quickly then they form an opinion that this new gear must be sounding better.  This is the case whether the sound waves changed one bit, or whether the brain received anything different!

This is why we don't test audio gear in listening tests like you all do.  We perform the test blind.  That way, the above analysis cannot benefit from knowing what is new gear and what is not.

When we perform the correct test as above, then the differences you perceive go out the window.  Folks can't hear the clear, night-and-day differences they thought existed before.

This is an easy test to do: play the same digital file multiple times in a row.  Then focus on differences, in detail, etc.  I guarantee you that you will hear different fidelity in every play even though you know for a fact there is no difference!  Your brain is that good at confusing you.

In sharp contrast, measurements are reliable.  When something distorts, it is guaranteed to screw up the fidelity of music you bought.  I have yet to find a case of a unit distorting yet sounding better in controlled testing. 

Controlled testing by the way also includes matching levels and making sure channels are not swapped.  Time after time, I have heard clear fidelity differences that simply vanish like a fart in the wind when I match levels.  Even when testing sighted these controls go a long way toward producing more reliable results.

Note that what I am writing here is the opinion of professional research community in audio.  It has been developed over decades of research, experimentation, knowledge of our hearing and brain, etc.

As I said, you are welcome to invent your own audio science but please don't put it forward as having any validity.  We (audio scientist and engineers) are not stupid to spend so much extra work to run controlled tests if all we had to do was test things as you do.  We do it because it is *essential* for generating correct results.

As the movie matrix, ultimately you can continue to live in the illusion of matrix or adopt what the real world/science is.  I am here to pursue and advance the latter.
Title: Re: Audio Science Review
Post by: amirm on December 18, 2018, 09:24:43 PM
You’ve just hit the nail on the proverbial head. The idea that every amp and dac should just do its job and get out the way of the music is all good and well, but doesn’t even start to explain why two components that supposedly do that sound completely different to each other.
Once again, to say two things sound the same requires that you only judge the sound.  You have no ability to do that in typical audiophile tests.  Your brain is a key part there and it manufactures conclusions that are hugely wrong and have been shown to be the case numerous times.  I have done this work professionally and I can tell you your family's livelihood depends on knowing the correct facts, you never perform audio tests the way audiophiles do.

There is a reason not one DAC manufacturer has ever posted a blind/controlled test showing their stuff sounds better.  I on the other hand, have done this so many times.  I guarantee you that what you think sounds different, will not at all hold up to scrutiny in a controlled test.  Your conclusions are illusions. 

And the problem with illusions is that they won't last.  This is why audiophiles keep upgrading things.  Every time they evaluate another audio gear, they arrive at a random conclusion not backed by what their hearing is capable of.  So they routinely decided the new gear is better so buy that product.  And advocate it until they see the next thing they think is better.

How many times do we hear they have bought a new gear and it removed a "veil?"  Seemingly infinite amount of veils can be removed.  Surely no logical person can believe that.

Quote
And we also tend to forget that the biggest variable of all is the transducer, but a very large margin too. You could have the world’s most ‘perfect’, best measuring amp and it will still sound like cr** with a rubbish transducer.

Bottom line: be guided by others with experience (and by that I mean listening experience, not qualifications - there are no ‘qualifications’ for knowing what to listen for) but only ever trust your own ears.
Absolutely trust your ears: just figure out how to get the brain and all the other factors it includes out of the way. This is what controlled tests do.  Otherwise you are creating a free-form illusion that is just not grounded in anything real.

Title: Re: Audio Science Review
Post by: amirm on December 18, 2018, 09:40:28 PM
Here's an informed opinion on ASR and the veracity of the measurements to be found there.

Take a moment and watch this Currawong (Amos Barnett) video from 34:55 to around 36:00

https://m.youtube.com/watch?v=zjF9HnCNJKU (https://m.youtube.com/watch?v=zjF9HnCNJKU)


 :popcorn:
There is nothing informed about his opinion. This is the same guy that on head-fi created the conspiracy story that I someone else ("NWAVGUY").  See this post by head-fi owner forum saying it on his behalf:

(https://www.audiosciencereview.com/forum/index.php?attachments/1545161444678-png.19050/)

And look at all the gear he has behind him:

(https://www.audiosciencereview.com/forum/index.php?attachments/1545161507774-png.19051/)

It is audio-gd and Schiit gear which have done poorly in my testing.  So of course he is going to be emotional and irrational in the way he lashes out.

But here is the data.  I tested another audio-gd product: https://www.audiosciencereview.com/forum/index.php?threads/review-and-measurements-of-audio-gd-nfb28-28-dac-and-headphone-amp.5147/

And this is where it finished in ranking:
(https://www.audiosciencereview.com/forum/index.php?attachments/audio-gd-nfb28-28-dac-and-headphone-amplifier-sinad-measurements-png.17412/)

These are the facts.  I run the exact same test file on every dac.  I connect three cables: USB plus audio out.  And what comes out is a number that is in the above graph.  If he wants to explain to me how that can go wrong, I am all ears.

By now I have tested 100+ DACs.  Countless manufacturers have reviewed them, commented on them, etc.  No way would I be able to continue to do what I do, with very strong support from owners loaning me gear to test if my results were not correct and objective.

I feel bad for people who have gone and put their reputation behind these products, made youtube videos to promote them and praise them lavishly in forums.  They didn't learn how to do proper audio testing so here they are.  Real, objective data contradicting their opinion.  This is what happens when you fight science.  You don't have a strong leg to stand on.  You just don't.

Anyway guys, as I said, a lot of good is being done.  We are getting much more data out there, forcing manufacturers to better engineer their products.  Let's not continue the stale FUD debate created by the few and cast doubt on this work.  I could be spending the time testing more products than writing these posts.  :)

Edit: fixed typos.  :)
Title: Re: Audio Science Review
Post by: kenvanraas on December 18, 2018, 09:50:30 PM


(https://i.imgur.com/ZrOOs5T.jpg)
Title: Re: Audio Science Review
Post by: GoodEnoughGear on December 18, 2018, 09:53:42 PM

(https://i.imgur.com/ZrOOs5T.jpg)

Haaahahahaha. Dude you nailed it.  :ROFLMAO:
Title: Re: Audio Science Review
Post by: Katji on December 18, 2018, 10:04:47 PM
But in the end measurements mean nothing if the sound isn’t good,

What?

Quote
...and I’ve got an item that measures very bad yet is the best sounding component I’ve owned. Fair enough good electrical work should measure within specs usually but what if it sounds better by changing some of those parameters, is it still wrong or a bad design I don’t think so.. as long as it sounds good to me, there no one right sound, everyone hears things slightly different so perfect measurements and bad measurements will all sound different to different people..

[...]

Its not about trusting your ears it’s about letting your ears hear how they hear which is very unique to each person :)

god help.

btw, It's not your ears, it's your mind.  The outer ear mechanism is just the first step.

Just get the basics of psychoacoustics.
Title: Re: Audio Science Review
Post by: shadow.clone on December 23, 2018, 08:02:19 AM
@amirm thanks for wading into this thread and responding to allegations/comments. I've purchased a Khadas Tone Board after reading your review and I'm absolutely loving it. Amazing piece of engineering, which I suspect is largely due to a rigorous design cycle and a great PCB layout as a result.

I wonder if the next positive spin off from your measurements will be to get the "ESS hump" sorted...
 
Title: Re: Audio Science Review
Post by: gLer on December 24, 2018, 01:59:33 PM
Interesting response from Kingwa on the official Audio-gd site (doesn’t mentions Amit by name but it’s clearly directed at him):

http://audio-gd.com/En%20audio-gd.htm
    
A message to respond some one

The English is not my advantage so the message must had a lot grammar mistook.
If I was you, I would not say anything how the specs effect the sound quality.
You can say how bad of the specs and the sound quality , but you should not hint the transparency = low distortion.
Did you say the tube devices sound not transparency? Or did you mean the phono sound not transparency ?
You know how to measure , you may know the electric technology, but you may don't know the audio technology.

I have very lot changes to measure the expensive hi-end devices of different brands, because a lot customers brought their units come to our listen room to have the compared with our products.
I have measured the wonderful units to study why their sound is tube like or transparency or sound digital .
Through a lot measured and studied I know how to make the sound like analogue but not digital .
Audio study is large different to electric study . I can not study the audio technology only in book but I must open and study by the hi-end units , a lot practice and listen are necessary.

Did you know the specs effect between OS and NOS ? The same unit, OS and NOS can had 15DB different on the THD.
It is a unit only with NOS design.
We have declared on the web page :THD today is extremely low with Sigma Delta chips; R2R ladders have good THD but nos as good as Delta Sigma .
In your point the most DS design must better than most R2R design.

In the web page we have declared the unit sound has Smooth, coherent and analogue sound quality and has no emphasise on any frequencies in audio spectrum, and more like real oil painting.
It obvious design for the analog or tube like sound .
It must some things like the pure analog design .
There are over 3000 users and I had got a lot feedback from users says sound wonderful, I don't think these users don't know how to listen music .

A few Americians may dislike the Chinese products, like Mr. Donald don't like Huawei .
But music has not limit of countries. We are not only enjoy the Chinese music but we also respect the oversea musicians .
The most people just want to enjoy the music but not enjoy the specs.