Kuala Lumpur International AV ShowHi-Fi
July 31, 2014

KLIAVS 2014, The “Golden Ear Challenge” Statistics

Article by WL Low

 

Our test “Conductor” Lam Seng Fatt.

 

Here are some statistics we have compiled from the final tally of the “Golden Ear Challenge” presented by this website, www.AV2Day.com. From the 10,000-odd KLIAV 2014 visitors, we sampled 294 participants over a three-day period, which means about 3% of total showgoers were tested.

 

Many have personally complained to me that the room was closed and music was playing inside or the room was closed with a sign hanging on the door indicating the time of the next session. The answer is that if you had encountered the first scenario, most likely during the first 2 days of the show, it was because the demo room next door had the ‘common sense’ to play Tchaikovsky’s 1812 Overture, with the 12 salutation cannons at full blast, while our participants in room no.7038 were trying to pick out the little differences between two digital transmission cables! We had to close the door to ensure a more conducive environment for the Golden Ear challenge. I spoke to the people next door on the second day when I met them and they never bothered us again, not with their 12 salutation cannons anyway. The later scenario was that we were either out for lunch or out for some fresh air. Either way, we appologize for not being “there”.

 

From the answers given by the 294 willing participants in the Golden Ear challenge, here is the break down of the scores by numbers:

 

1) Preference: Participants were asked if they preferred the sound of cable A (Co-axial) or B (Toslink)?
48% of participants chose A,
31% of participants chose B, and
21% of participants chose not to indicate their preferred cable.

 

Co-axial is the preferred choice of digital transmission by a good margin.

 

Some of the “Golden Ears” challenge participants. Recognise any of them?

 

Next we look at the breakdown of participants’ score card.

 

2) Scorecard: The correct sequence on all 3 days is the same (although I did indicate that it may NOT be, during the test sessions), meaning if you had participated and had circled the five sequences as:

 

1) A,
2) B,
3) A,
4) B,
5) A,

 
Then congratulations on being blessed with goods ears, or being well trained. There was also a C choice for “Unsure” for those who “heard no differences” whatsoever, which some had also circled on the test sheets.

There are just 8 (just below 3%) participants out of 294 participants who got all 5 sequences correct. We are going to certify these people and reward them handsomely in the near future.

 

There are 34 (about 12%) participants who got 4 out of 5 sequences correct. Well done folks, but we do not certify nor reward second best!

 

There are 78 (almost 27%) participants who got 3 out of 5 sequences correct. Not bad actually, but these folks probably suffered from some form of hearing fatigue by the 4th test, temporary loss of concentration or were just plain lucky! Actually, more than 60% of this group got the first 3 sequences correct! That’s why we decided to do the test in 5 sequences and not 3 to weed out the element of luck!

 

There are 76 (spot on 26%) participants who got 2 out of 5 sequences correct. What can I say about this bunch except to urge them to train harder to improve their listening/concentration skills or with them better luck next time!

 

There are 62 (more than 21%) participants who got 1 out of 5 sequences correct. We either welcome these newbies to the hobby, or they (the more senior audiophiles, that is) ran out of luck after just one strike.

 

There are 36 (more than 12%) participants who got 0 correct, with 2 totally indicating that they heard absolutely no difference whatsoever on any sequences in the test sheet! Ah…! It was not meant to be, don’t spend anymore $$$ on hifi equipment! Just enjoy and be a proud music lover. Ha! Ha!

 

Here’s another interesting fact – 12 of the 36 in this group actually got the sequences all in reverse, meaning that they could clearly hear the difference, but just got confused between the A & B sound.

 

Willy Low, is seen here as co-test “Conductor” when Lam Seng Fatt was out on a toilet break. Veteran audiophile, Erik Goh also conducted the test on the last day. We at AV2Day would also like to thank Ooi Chee Hiean for his help with crowd control

 
I wish to end this article with something rather cheerful which we witnessed during those 3 days.

 

First scenario: During one of the test sessions on the second day, a couple came into the room, and the husband wanted to participate in the “Golden Ear” challenge. Lam Seng Fatt duly started the test session, but the man had trouble deciding which answer to circle on each sequence. Each time the wife looked on and encouraged him to listen again. Something which we obliged for those with a short memory span. After a few repeats, the husband was still rather undecided about the sequences. The wife lost her temper and started saying that it was very easy to hear the difference, and that she should take the challenge alongside her husband. She was very sure of her answers each time, when her hapless husband was still thinking about which to circle. I am also not sure if either of the couple won the challenge, but I did advise the husband that his wife is a good listener and that he should bring her along the next time he wants to buy some hifi equipment. The beaming wife told us that this is the best hifi show yet, as it was boring for her in the last few years, just accompanying her husband to the show! She looks forward to more fun & games next year.

 

The next scenario involved the boss of a hifi dealership who insisted that all his staff and business associates participated in the event as a form of baptism of fire! So over the course of the 3 demo days, the boss and all his staff and business associates came to take the challenge repeatedly, sometimes with very hilarious results!

 

I was actually very surprised that many industry people came to participate in the “Golden Ear” challenge!

 

Stay tuned, as we shall announce the names of all our 8 deserving Golden Ears and details of the prize-giving ceremony on this web site shortly.

 

Article by WL Low

 

 

10 Comments - View all

ST 10:51am on August 10, 2014

Hi Willy Low, Thank you for your explanation. I was just trying to eliminate element of luck before drawing any conclusion based on the results. Unfortunately, the law of probabilty doesn't correlates with some results and while in other cases it perfectly fits to a T if my readings of the results is right. That is good because difference could be audible. But we are not looking for difference, we are looking for which cable sounded better. The way I look at it, when the demo was played for determination, participants chose either A or B and some people who couldn't tell the difference didn't chose anything. The author mentioned that 21% chose not to indicate their preference. Wouldn't it be better that this category be eliminated from the statistics completely since whichever cable made no difference to their ears or they could clearly identify the difference but refused to tell? Their answers to the test were unreliable sample since we have no way to eliminate the elements of luck? Did any of this group get all answers correct? My guess is there must be at least 140 to 160 would have got (1) correct. I forgotten the tolerance formula. I believe the same number would apply to (2), (3), (4) and (5). Am I correct? If the number exceeds more 200 for (1) which is the best indicator for reliably test since it was within a short span of time after the demo then we may have some concrete evidence and all we need is to fine tune the blind test in next year's shoot out - if there is one.   Once again, I am sharing my thoughts on this for sake of discussion and not disputing your findings.  Thank you. Warm regards, ST

10 CommentsX
ST 10:51am on August 10, 2014

Hi Willy Low,

Thank you for your explanation. I was just trying to eliminate element of luck before drawing any conclusion based on the results. Unfortunately, the law of probabilty doesn’t correlates with some results and while in other cases it perfectly fits to a T if my readings of the results is right. That is good because difference could be audible. But we are not looking for difference, we are looking for which cable sounded better.

The way I look at it, when the demo was played for determination, participants chose either A or B and some people who couldn’t tell the difference didn’t chose anything. The author mentioned that 21% chose not to indicate their preference. Wouldn’t it be better that this category be eliminated from the statistics completely since whichever cable made no difference to their ears or they could clearly identify the difference but refused to tell? Their answers to the test were unreliable sample since we have no way to eliminate the elements of luck? Did any of this group get all answers correct?

My guess is there must be at least 140 to 160 would have got (1) correct. I forgotten the tolerance formula. I believe the same number would apply to (2), (3), (4) and (5). Am I correct? If the number exceeds more 200 for (1) which is the best indicator for reliably test since it was within a short span of time after the demo then we may have some concrete evidence and all we need is to fine tune the blind test in next year’s shoot out – if there is one.  

Once again, I am sharing my thoughts on this for sake of discussion and not disputing your findings. 

Thank you.

Warm regards,
ST

willylow 9:26pm on August 8, 2014

Persoanlly, I do not think toslink is lousy, and in the right system, can sound very good indeed. I guess it’s just system synergy and listener preference that dictated the statistic results.

In regards to the question of digital inputs cross bleeding in to each other?

I don’t think that is the case here, as all equipment except for the CD trasnport is brand new. I also took one extra pre-caution when setting up the yest system was to use the co-axial input furthest away from the toslink input, to ensure minimum cross bleeding, if it happened. I have personally used a Bryston BDA-1 DAC as my main digital switching hub between my deigital media player and CD transport, with the same multiple input arrangement for nearly 5 years now, never hearing any digital input cross bleeding.

willylow 9:15pm on August 8, 2014

ST,

Thank you for further defining the statistics. Here are my further clarifictaions to your answers:

1) You wrote “There are 62 (more than 21%) participants who got 1 out of 5 sequences correct. We either welcome these newbies to the hobby, or they (the more senior audiophiles, that is) ran out of luck after just one strike.” Does this mean they got only (1) A correct? And those who got the (1) wrong but got one of the others correct weren’t included in this category?

The above means a total of 62 participants got only 1 out of 5 sequences correct, iregardless of the sequences of A/B choice order. It simply meant that they only manage to get one correct and four wrong answers(if this was a test by objective format, remember those in school, done with a 2B pencil?).

2) I ask this because in category 3 you mentioned “There are 78 (almost 27%) participants who got 3 out of 5 sequences correct. Not bad actually, but these folks probably suffered from some form of hearing fatigue by the 4th test, temporary loss of concentration or were just plain lucky! Actually, more than 60% of this group got the first 3 sequences correct! That’s why we decided to do the test in 5 sequences and not 3 to weed out the element of luck!”, which means here the sequence can start form )1), (2) or (3) to get three correct answers in sequence since you mentioned 60% of the 78 got the first three answers correct.

Can you see the contradiction?

There seem to be some contradiction to the above results, I agree, let me explein. There were a total of 78 participants who got 3/5 sequences correct, in no particular order. When I say 60% of this group got the first 3 sequences right by choice and the last 2 sequences wrong, meant that they could clearly and accurately identify the sound of the two connections in the first 3 sequences in correct order, but missed the remaining 2/5 due to a number of reasons like, short memory span, confusiong set in and lastly listening fatique. The remaining 40% of this group got 3/5 sequences correct in no particular order. That makes it more a luck factor, rather than hearing skills in play.

ST 8:01pm on August 4, 2014

Hi,

I think you guys may have proven something. Statistically, it looks like 95% could tell the difference. But I am having difficulties understanding results due the way the results were written.

You wrote “There are 62 (more than 21%) participants who got 1 out of 5 sequences correct. We either welcome these newbies to the hobby, or they (the more senior audiophiles, that is) ran out of luck after just one strike.” Does this mean they got only (1) A correct? And those who got the (1) wrong but got one of the others correct weren’t included in this category?

I ask this because in category 3 you mentioned “There are 78 (almost 27%) participants who got 3 out of 5 sequences correct. Not bad actually, but these folks probably suffered from some form of hearing fatigue by the 4th test, temporary loss of concentration or were just plain lucky! Actually, more than 60% of this group got the first 3 sequences correct! That’s why we decided to do the test in 5 sequences and not 3 to weed out the element of luck!”, which means here the sequence can start form )1), (2) or (3) to get three correct answers in sequence since you mentioned 60% of the 78 got the first three answers correct.

Can you see the contradiction?

Dear guys, I may be crossing my line here by asking too many questions on what I consider one of well conducted blindtest. It is just you don’t get many participants like these to get useful data. So far, it looks like the preference is split 50:50. It can be due to taste or they are just guessing. However, if 95% picked answer (1) correctly then you guys have proven that the difference is audible to average listener.

Thanks.

sflam 10:14pm on August 3, 2014

hi ST,
regarding your question as to whether the inputs leaking sound to the other source (i.e. crosstalk) masked the differences, i think the answer is no. we used pretty good components – the dac was the bryston bda2. the transport was a sony DVP-9000ES which is a pretty high-end dvd/sacd/cd player.

tq
sflam

sflam 6:15pm on August 3, 2014

hi ST,

AV2day conducted this shootout partly for fun and partly to bust the myth that toslink is lousy.
the reference QED toslink cable we used was made of plastic. QED’s latest generation reference toslink cable is made of glass. there is a difference between plastic and glass toslink – glass is better.
if we had used a glass toslink, i am very sure the results would be different. in my system, the glass toslink sounds better than the coax.

tq
sflam

ST 5:15pm on August 3, 2014

Hi SfLam,

Me too not good at counting :) . Did we get the intended result from this challenge? The choice is split 50/50. The C value for double blind test is 95% of correct answers. Ignoring rest of the 4 tests, how many got the first one correct? We should get 273 correct answers for (A).

I believe there is a difference but the result is shocking and as it means no difference between Toslink and SPDIF. Could the inputs leaking sound to the other source causing the difference to be masked?

sflam 7:59pm on August 2, 2014

Hi ST,
I must say my math is horrible. We never knew the probability of the number of audiophiles who could get it right by chance.
Even with five test samples it was already tiring for the participants.

Perhaps u can suggest some other format which can eliminate chance without exhausting the listeners.

Tq
Sflam

ST 5:38pm on August 2, 2014

You guys deserve our utmost gratitude for conducting this experiment. The test was fun and as expected I have confirmed that I have wooden ears. Having said that, I believe you can tell the difference but I guess you need little bit more time with the system. Maybe next year you guys can come up with a more robust test. The mathematical probability of getting all correct answers is 1 in 32. So out of 294 , 9 would have got it right.

I congratulate the 8 lucky winners.

Adrian4454 11:25am on August 1, 2014

Hi Lam FS and Low,
Thanks a lot for making this event worthwhile. Just some feedbacks over the challenge.

And boy, it was really a challenge, considering the track was not being play on the same time frame on each of the supposedly Toslink and Coaxial switch.

I struggle to tell the differences when track reached percussion works instead of vocal.

Worse still if you are not familiar with the song. So for the worthy Golden Ears winners out there, I salute you!