In article , Iain Churches
wrote:

"John Phillips" wrote in message
...


May I point out gently the trap into which Stereophile is reported to
have fallen in this respect?

You may:-)

You can be sure, that however carefully the tests are conducted, and
whatever protocol, there will alway sbe someone (usually on the outside)
who will have something to say (and usually after the event:-)

That is a vital part of the scientific method. However their comments can
also be assessed by other since the entire process is one of open scrutiny.

However the interesting thing about John's comment was that it was not
regarding the protocol of the test, but of the error of "selecting
results". This can easily lead to invalid conclusions as he indicated. By
suitable 'selection' you can from a large ensemble of tests get an 'result'
you want.

The point here is one I made a while ago. The test must be suitable for the
hypothesis being tested.

From what John said, the test was of a given population of test suvjects to
see if "people can tend to hear this difference" in a general way.

However "can some specific people hear this difference" is a different
hypothesis. Hence as John indicated, it would then require re-testing
people who seemed to "do well" sic in the first set of tests. Actually it
would *also* require re-testing some of those who "did badly" to see if
their results also changed. IIRC statisticians refer to this problem with
terms like "reversion to the mean".



I firmly believe that the way the test is organised, has a great effect
upon the number of people who are interested to take part. In my view,
an ostentatious "challenge" with large sums of money is not the way it
should be done.

You may be correct. However IIUC the offer of a large sum only appeared
*after* people kept appearing, making the claims, and then refusing to
engage in a suitable test. And that the level of 'ostentation' and emotion
on both sides seems to have grown as a result of people steadily exhibiting
this behaviour of "make the claim, but refuse to put it to a test".

Slainte,

Jim

--
Electronics http://www.st-and.ac.uk/~www_pa/Scot...o/electron.htm
Audio Misc http://www.st-and.demon.co.uk/AudioMisc/index.html
Armstrong Audio http://www.st-and.demon.co.uk/Audio/armstrong.html
Barbirolli Soc. http://www.st-and.demon.co.uk/JBSoc/JBSoc.html