"John Phillips" wrote in message
...
On 2006-01-21, Iain Churches wrote:
"Jim Lesurf" wrote in message
...
As has recently been posted elsewhere, some 'ABX' / DBT tests *have*
been
reported - *some* of which apparently show some component differences
are
audible.

Indeed. As we found. But only a few participants had
sufficiently good levels of audio perception to reliably and repeatly
pick out the changes. ...

May I point out gently the trap into which Stereophile is reported to
have fallen in this respect?

You may:-)

You can be sure, that however carefully the tests are conducted, and
whatever protocol, there will alway sbe someone (usually on the outside)
who will have something to say (and usually after the event:-)
I cannot speak for Stereophile, but in the test with which I was
associated, all those who took part were sastisfied with the methods.

It has been reported that some years ago they tested about 3,000 people
for their ability to discern whether a CD had been tweaked or not
(green felt pens came into this somewhere as I recall). Apparently,
Stereophile pointed to a small sub-set of the test subjects who had
got statistically significant scores as evidence that some people could
distinguish the tweaked CDs.

The tests in which I was involved had nothing to do with green felt pens,
or anything so exotic.

We were simply interested to know if people
could hear the difference between a silver foil and an industrial standard
output coupling capacitor on a simple mu-follower stage.


However, even with purely random guessing, some of the 3,000 will
score that well from just statistics. So the Stereophile conclusion
was incorrect. They should have re-tested the top scorers and only
if they scored well again should their conclusion have been drawn.
They might alternatively have looked at the size of the sub-set and
checked if it too was statistically significant, but a re-test would
have been more convincing.

Agreed. There was much discussion about how the tests should be
carried out, and we were anxious to involve those on the test panel in
deciding the protocol. Each member was allowed to suggest a piece
of music with which he/she was familiar to be used in the test, and so we
had a good selection of material from which people could choose.
All participants were allowed access to the test material before hand
to become acquainted with the music involved.

And how can anyone make a random guess when they don't know
when the source is being switched? We allowed a window of 5 secs
in which they must indicate.

On a re-test, those with the best scores, got more or less the same
results. Interesting too was the fact that even with the TC sheet in front
of them, telling where the changes were taking place, a large percentage
of those tested could not hear any difference.

I should mention again that we were not concerned about whether one
cap was better than the other, but if people could discern a difference.

I firmly believe that the way the test is organised, has a great
effect upon the number of people who are interested to take part.
In my view, an ostentatious "challenge" with large sums of money is
not the way it should be done.

Iain