Thus spake Jim Lesurf:
I can't cite examples but my impression FWIW, is that calibrated
variations can be fairly large before becoming significant. If that
is indeed the case, I suggest that using DB testing for testing
auditory differences is largely pointless.

I don't regard it as 'pointless'. Its 'point' is to indicate what the
actual limits of perception may be, regardless of the beliefs or
wishes of the individual. The results show that people often tend not
to be able to hear differences that they believe they can.

If subjects can't hear fairly large differences in a calibration cycle, I
can envisage 2 explanations. Firstly, some/many/most subjects are fairly
insensitive to variations & by definition, would be wasting money by buying
expensive audio equipment for sonic reasons alone. The second, is that
because the way the mind works, comparing sequences such as replaying the
same piece of music is going to confuse the subjects & muddy the results. I
can imagine this explanation being very inconvenient to many because it
throws in hidden variables such as how reliable human memory is & its
effects on the outcome. I only entertain this possibility because my own
experience suggests measuring qualitative stuff can be damned difficult. A
lot of people also state they can hear differences beyond measurability.

I wouldn't be surprised if a lot of research has already been done that
would be relevant to perception in general that can be applied to audio.
Lets take something less controversial like visual acuity. Photographs are
printed out at different resolutions or maybe colour depth & subjects asked
to make comparisons. In one test, the photos are laid out side by side & the
viewer asked which version they prefer & why. The next test, the photos are
viewed singly but in sequence. Both tests can be randomised. My assumption
would be that subject's scoring when comparing photos side by side would be
a lot higher than when viewed singly in sequence. Now, to me, this would
demonstrate how much effect memory has on perception. I can't think of a way
to do the same thing with auditory tests apart from the ludicrous idea of
playing back different criteria in each channel! The difference with audio
is that its coherence happens over time where a still picture's doesn't.
Pause a movie & the picture becomes static but retains some meaning. The
sound just disappears completely or mathematically would become set of a
sine waves of the time slice before the pause with the correct amplitude.
The photo test's relevance to audio is that the perceived differences vary
according to how much memory is involved in making comparisons (or how the
test is conducted.) The differences are still real & measurable with the
photos. I'm suggesting that although testing involving memory may mask
differences, those differences maybe indeed be real & considered to be
worthwhile by some people.

There some weaknesses here such as why doesn't testing using equipment not
track these supposed differences? One comment I can make is that most tests
are perhaps too static & are over-simplified. Another objection I have that
it goes against the grain of simplicity by heaping on more variables. Just
because many people believe in ghosts, doesn't mean they exist. I don't deny
that I may have built myself a house of cards here!

I heartily wish I could suggest alternatives but I can't.

Well, from the POV of the scientific method a hypothesis has to be
testable to have any validity/meaning. So if you/someone can't
propose and carry out an appropriate alternative we have to stick
with hypotheses we *can* test. This is to avoid people simply
believing whatever they choose, regardless of the reality.

But only if the tests are valid & don't end up perpetuating a fallacy. If it
meant going back to the drawing board, so be it.

In article , Paul B
wrote:
Thus spake Jim Lesurf:
I can't cite examples but my impression FWIW, is that calibrated
variations can be fairly large before becoming significant. If that
is indeed the case, I suggest that using DB testing for testing
auditory differences is largely pointless.

I don't regard it as 'pointless'. Its 'point' is to indicate what the
actual limits of perception may be, regardless of the beliefs or
wishes of the individual. The results show that people often tend not
to be able to hear differences that they believe they can.

If subjects can't hear fairly large differences in a calibration cycle,
I can envisage 2 explanations. Firstly, some/many/most subjects are
fairly insensitive to variations & by definition, would be wasting
money by buying expensive audio equipment for sonic reasons alone.

That may be so. However, if so, we would then have to be cautious about
trying to draw specific/individual conclusions from the above as it is a
generalisation. So some people *might* be able to hear *some* differences
when others cannot.

But to see if this is the case, we would first need some test subjects to
demonstrate in a suitable test that *they* *can* hear a given 'difference'
even if (many) others cannot. Otherwise the simplest hypothesis consistent
with results may be that - despite claims to the contrary - *no one* can
hear a given 'difference'.


The second, is that because the way the mind works, comparing sequences
such as replaying the same piece of music is going to confuse the
subjects & muddy the results. I can imagine this explanation being very
inconvenient to many because it throws in hidden variables such as how
reliable human memory is & its effects on the outcome. I only entertain
this possibility because my own experience suggests measuring
qualitative stuff can be damned difficult. A lot of people also state
they can hear differences beyond measurability.

The problem with the above is as follows:

IIUC there is good evidence to the effect that our memory and state of mind
affect what we notice, or how we perceive or judge what we experience.

This may be a reason for saying that 'time serial' comparison tests are
affected by this, so tending to reduce the noticibility of real
differences.

However this may also mean that people hear 'differences' which are due
simply to their change in mental (or physiological) state, etc. Thus they
may be saying that one item sounds different to another when the actual
sounds produced are unchanged.

Thus the same 'mechanism' produced to 'explain' why such tests tend to show
people unable to hear a difference also 'explains' why they may think they
hear differences in situations where none really exists.

The upshot being that we then have no reliable evidence that any such
differences exist. But plus having a reason for saying that what people
claim may be based on an error.

This would also give us grounds to say, "since the perceptions are
variable, there is no real point in worrying about differences so slight at
to fall within these variations".

Thus we end up with "a differences which makes no difference *is* no
difference". (Spok's Rule.) :-)

The advantage of some of the ABX forms of test is that the comparisons can
be done on all sorts of time scales - under the control of the test
subject. So they can switch quickly if worried about 'memory' or drifts in
their physiology, etc. For some kinds of difference this seems IIRC to
produce enhanced sensitivity. But for others it shows no sign of the
subjects being able to hear any difference, on any timescales people have
employed.



I heartily wish I could suggest alternatives but I can't.

Well, from the POV of the scientific method a hypothesis has to be
testable to have any validity/meaning. So if you/someone can't propose
and carry out an appropriate alternative we have to stick with
hypotheses we *can* test. This is to avoid people simply believing
whatever they choose, regardless of the reality.

But only if the tests are valid & don't end up perpetuating a fallacy.
If it meant going back to the drawing board, so be it.

The problem with *if* here is that it is a speculation. That has no real
use in the scientific method *unless* you can then propose a test which
would distinguish you hypothesis from the competing ones...

Thus a given test *might* not be 'valid'. But to decide this would require
a suitable test, ideally also a proposed 'mechanism' for the cause of the
lack of 'validity' which the new test would probe.

Without that, we have to work on the basis of using the hypotheses that are
consistent with the evidence we have, and trying to avoid adding mechanisms
which the evidence does not require, or ideas we cannot test.

Many things *might* be the case. But that does not tell us they *are* the
case. For that we require relevant evidence. Alas, "the evidence does not
agree with my beliefs" is not actually evidence... :-)

Slainte,

Jim

--
Electronics http://www.st-and.ac.uk/~www_pa/Scot...o/electron.htm
Audio Misc http://www.st-and.demon.co.uk/AudioMisc/index.html
Armstrong Audio http://www.st-and.demon.co.uk/Audio/armstrong.html
Barbirolli Soc. http://www.st-and.demon.co.uk/JBSoc/JBSoc.html

Thus spake Jim Lesurf:
snipped
If subjects can't hear fairly large differences in a calibration
cycle, I can envisage 2 explanations. Firstly, some/many/most
subjects are fairly insensitive to variations & by definition, would
be wasting money by buying expensive audio equipment for sonic
reasons alone.

That may be so. However, if so, we would then have to be cautious
about trying to draw specific/individual conclusions from the above
as it is a generalisation. So some people *might* be able to hear
*some* differences when others cannot.

A maddeningly large sample may be needed.

But to see if this is the case, we would first need some test
subjects to demonstrate in a suitable test that *they* *can* hear a
given 'difference' even if (many) others cannot. Otherwise the
simplest hypothesis consistent with results may be that - despite
claims to the contrary - *no one* can hear a given 'difference'.

Indeed. There may well be a large variation in subject's auditory
sensitivity according to age, how critical their listening had been etc. I
would still be very suspicious *if* subjects can't hear large introduced
differences. If a number of subjects *can* hear calibrated differences but
can't hear any during the real part of the test, that would certainly point
towards the effectiveness of DB testing & would by definition, render my
worries regarding memory effects as being irrelevant. *Testing the test by
using a cal cycle is crucial to my mind.*

The second, is that because the way the mind works, comparing
sequences such as replaying the same piece of music is going to
confuse the subjects & muddy the results. I can imagine this
explanation being very inconvenient to many because it throws in
hidden variables such as how reliable human memory is & its effects
on the outcome. I only entertain this possibility because my own
experience suggests measuring qualitative stuff can be damned
difficult. A lot of people also state they can hear differences
beyond measurability.

The problem with the above is as follows:

IIUC there is good evidence to the effect that our memory and state
of mind affect what we notice, or how we perceive or judge what we
experience.

Agreed. Some people even concoct whole events such as alien abduction,
invent the concept ritual child abuse, implant false memories etc.
Attributing differences in audio equipment would be picnic in the park -
hence the need for some kind of testing.

This may be a reason for saying that 'time serial' comparison tests
are affected by this, so tending to reduce the noticibility of real
differences.

Yup. I've repeated a section of music & concluded that I couldn't tell that
both plays sounded the same or not!

However this may also mean that people hear 'differences' which are
due simply to their change in mental (or physiological) state, etc.
Thus they may be saying that one item sounds different to another
when the actual sounds produced are unchanged.

Again yes. Some form of testing would be needed to average out the noise.

Thus the same 'mechanism' produced to 'explain' why such tests tend
to show people unable to hear a difference also 'explains' why they
may think they hear differences in situations where none really
exists.

The upshot being that we then have no reliable evidence that any such
differences exist. But plus having a reason for saying that what
people claim may be based on an error.

This would also give us grounds to say, "since the perceptions are
variable, there is no real point in worrying about differences so
slight at to fall within these variations".

Thus we end up with "a differences which makes no difference *is* no
difference". (Spok's Rule.) :-)

Very much a possibility that occurred to me last night! The memory effect
biting back by masking - even if it could be proved that some differences
not shown in BT tests but in some other one did exist they may well be
fairly pointless in that they simply get drowned within the noise of
perception. I didn't want to throw in too many of my own objections,
otherwise I'll be accused of sock puppetry within the same message!

The advantage of some of the ABX forms of test is that the
comparisons can be done on all sorts of time scales - under the
control of the test subject. So they can switch quickly if worried
about 'memory' or drifts in their physiology, etc. For some kinds of
difference this seems IIRC to produce enhanced sensitivity. But for
others it shows no sign of the subjects being able to hear any
difference, on any timescales people have employed.

Months? I ask because I've either replaced or upgraded equipment, listened &
made a mental note that I could hear no differences then forgotten about it.
Months later, I've played a particular piece of music to be struck by how
different it sounds.

I heartily wish I could suggest alternatives but I can't.

Well, from the POV of the scientific method a hypothesis has to be
testable to have any validity/meaning. So if you/someone can't
propose and carry out an appropriate alternative we have to stick
with hypotheses we *can* test. This is to avoid people simply
believing whatever they choose, regardless of the reality.

But only if the tests are valid & don't end up perpetuating a
fallacy. If it meant going back to the drawing board, so be it.

The problem with *if* here is that it is a speculation. That has no
real use in the scientific method *unless* you can then propose a
test which would distinguish you hypothesis from the competing ones...

Thus a given test *might* not be 'valid'. But to decide this would
require a suitable test, ideally also a proposed 'mechanism' for the
cause of the lack of 'validity' which the new test would probe.

You make it seem that I advocate rolling dice Yes, of course it's more
satisfactory to forward a displacing theory rather than merely suggesting
the existing one is flawed but where would we be if someone was to suggest
that lead in cosmetics was dangerous & others said that they would continue
using it until the doubter came up with a substitute. As for speculation
Jim, much good science has come from it.

Without that, we have to work on the basis of using the hypotheses
that are consistent with the evidence we have, and trying to avoid
adding mechanisms which the evidence does not require, or ideas we
cannot test.

Many things *might* be the case. But that does not tell us they *are*
the case. For that we require relevant evidence. Alas, "the evidence
does not agree with my beliefs" is not actually evidence... :-)

Until someone comes up with a watertight explanation why DB is infallible or
near as dammit so, I'll reserve the right to be sceptical in the same manner
that I've been sceptical of my own hearing. To sum up, I'm not suggesting
that DB testing is completely pointless but IMO, can't be relied upon as the
sole means of testing, especially when some use it as a club to bash people
with the idea that most equipment sounds essentially identical. I feel more
comfortable with folks being cloth-eared than folks having so-called golden
ears!

--
Basically, I hate people who preface most sentences with "Basically...

In article , Paul B
wrote:
Thus spake Jim Lesurf: snipped
If subjects can't hear fairly large differences in a calibration
cycle, I can envisage 2 explanations. Firstly, some/many/most
subjects are fairly insensitive to variations & by definition, would
be wasting money by buying expensive audio equipment for sonic
reasons alone.

That may be so. However, if so, we would then have to be cautious
about trying to draw specific/individual conclusions from the above as
it is a generalisation. So some people *might* be able to hear *some*
differences when others cannot.

A maddeningly large sample may be needed.

That would depend on two factors, currently undetermined.

1) How small the fraction of the general population may be that can
actually detect a given, small, change. The smaller the fraction, them more
people in general you would have to test to get some reliable idea of the
number of people involved.

2) The extent to which such people are 'self identifying'. If it were the
case that those who keep insisting they can hear differences which others
cannon *can* do this, then they have identified themselves from the general
population. Thus if they then demonstrated in a suitable test that they
*can* do what they assert, then we can know they exist - although that in
itself won't tell us what fraction of the general population they
represent.

The snag, though, is that the general result of tests indicates that people
can't hear the small differences that they claim. Hence at present the
evidence is that the fraction of the population involved is between 'small'
and 'nil' according to curcumstances. i.e. a value we can't reliably
distinguish from 'nil'.

[snip]

The advantage of some of the ABX forms of test is that the comparisons
can be done on all sorts of time scales - under the control of the
test subject. So they can switch quickly if worried about 'memory' or
drifts in their physiology, etc. For some kinds of difference this
seems IIRC to produce enhanced sensitivity. But for others it shows no
sign of the subjects being able to hear any difference, on any
timescales people have employed.

Months? I ask because I've either replaced or upgraded equipment,
listened & made a mental note that I could hear no differences then
forgotten about it. Months later, I've played a particular piece of
music to be struck by how different it sounds.

Indeed, that happens to me over timescales from hours to months, using the
same equipment and source material.

However some of the tests I recently read about in the JAES, Audio Amateur,
etc, involved loaning some people a 'black box' for some months, and
inviting them to decide on an 'AB' or 'ABX' basis what it contained. This
was in one case apparently done because those being tested insisted that a
'quick' test was not 'sensitive', but prolonged listening would enable them
to be more discriminating. The reported results showed no foundation for
their belief.

[snip]


But only if the tests are valid & don't end up perpetuating a
fallacy. If it meant going back to the drawing board, so be it.

The problem with *if* here is that it is a speculation. That has no
real use in the scientific method *unless* you can then propose a test
which would distinguish you hypothesis from the competing ones...

Thus a given test *might* not be 'valid'. But to decide this would
require a suitable test, ideally also a proposed 'mechanism' for the
cause of the lack of 'validity' which the new test would probe.

You make it seem that I advocate rolling dice Yes, of course it's
more satisfactory to forward a displacing theory rather than merely
suggesting the existing one is flawed but where would we be if someone
was to suggest that lead in cosmetics was dangerous & others said that
they would continue using it until the doubter came up with a
substitute. As for speculation Jim, much good science has come from it.

Without that, we have to work on the basis of using the hypotheses
that are consistent with the evidence we have, and trying to avoid
adding mechanisms which the evidence does not require, or ideas we
cannot test.

Many things *might* be the case. But that does not tell us they *are*
the case. For that we require relevant evidence. Alas, "the evidence
does not agree with my beliefs" is not actually evidence... :-)

Until someone comes up with a watertight explanation why DB is
infallible or near as dammit so, I'll reserve the right to be sceptical
in the same manner that I've been sceptical of my own hearing. To sum
up,

No test method or experiment is "infallible". I am afraid that science does
not work like that. What it does it gathers evidence so we can use that to
assess how reliable or useful a given idea may (or may not) be. If you wish
for "infallability" then I'm afraid you will have to ask a theologian, not
a scientist or an engineer. :-)

I'm not suggesting that DB testing is completely pointless but IMO,
can't be relied upon as the sole means of testing, especially when some
use it as a club to bash people with the idea that most equipment
sounds essentially identical. I feel more comfortable with folks being
cloth-eared than folks having so-called golden ears!

I agree - but only if the speculation is testable and some evidence to
support it can be gathered and assessed. So if we say a given product is
'dangerous' we migh then regard it with caution, but then expect some
evidence to back up the assertion. If no evidence can be provided, we can
decide to regard the assertion as having no reliable substance.

We may change our minds at a later point if evidence *does* appear. But the
change of understanding should be based on evidence.

Otherwise we would have to work on the basis of never doing anything at all
because it "might be dangerous".

Slainte,

Jim

--
Electronics http://www.st-and.ac.uk/~www_pa/Scot...o/electron.htm
Audio Misc http://www.st-and.demon.co.uk/AudioMisc/index.html
Armstrong Audio http://www.st-and.demon.co.uk/Audio/armstrong.html
Barbirolli Soc. http://www.st-and.demon.co.uk/JBSoc/JBSoc.html

Thus spake Jim Lesurf:
snipped
If subjects can't hear fairly large differences in a calibration
cycle, I can envisage 2 explanations. Firstly, some/many/most
subjects are fairly insensitive to variations & by definition,
would be wasting money by buying expensive audio equipment for
sonic reasons alone.

That may be so. However, if so, we would then have to be cautious
about trying to draw specific/individual conclusions from the above
as it is a generalisation. So some people *might* be able to hear
*some* differences when others cannot.

A maddeningly large sample may be needed.

That would depend on two factors, currently undetermined.

1) How small the fraction of the general population may be that can
actually detect a given, small, change. The smaller the fraction,
them more people in general you would have to test to get some
reliable idea of the number of people involved.

2) The extent to which such people are 'self identifying'. If it were
the case that those who keep insisting they can hear differences
which others cannon *can* do this, then they have identified
themselves from the general population. Thus if they then
demonstrated in a suitable test that they *can* do what they assert,
then we can know they exist - although that in itself won't tell us
what fraction of the general population they represent.

The snag, though, is that the general result of tests indicates that
people can't hear the small differences that they claim. Hence at
present the evidence is that the fraction of the population involved
is between 'small' and 'nil' according to curcumstances. i.e. a value
we can't reliably distinguish from 'nil'.

I subscribe to the idea that most healthy people have reasonably similar
hearing acuity & I wouldn't get too fixated on those hovering above the
'nil' threshold. I can understand that people can be trained to hear better
than average but 1 or 2 in a million people aren't worth designing equipment
for & what format apart from CDs, DVD-As, SACDs or records would they listen
to anyway?

I re-emphasise the need to check the calibration cycle in any DB test as a
fair means of determining its effectiveness. /If/ DB was proved to be
sufficiently effective, it would save a lot of head scratching over devising
different tests.

The advantage of some of the ABX forms of test is that the
comparisons can be done on all sorts of time scales - under the
control of the test subject. So they can switch quickly if worried
about 'memory' or drifts in their physiology, etc. For some kinds
of difference this seems IIRC to produce enhanced sensitivity. But
for others it shows no sign of the subjects being able to hear any
difference, on any timescales people have employed.

Months? I ask because I've either replaced or upgraded equipment,
listened & made a mental note that I could hear no differences then
forgotten about it. Months later, I've played a particular piece of
music to be struck by how different it sounds.

Indeed, that happens to me over timescales from hours to months,
using the same equipment and source material.

However some of the tests I recently read about in the JAES, Audio
Amateur, etc, involved loaning some people a 'black box' for some
months, and inviting them to decide on an 'AB' or 'ABX' basis what it
contained. This was in one case apparently done because those being
tested insisted that a 'quick' test was not 'sensitive', but
prolonged listening would enable them to be more discriminating. The
reported results showed no foundation for their belief.

JAES is members only. I would tend to conclude that /if/ these tests were
carried out correctly & are statically significant, that this form of DB
testing was pretty valid but what was being tested? If interconnect cables,
I would expect as much!

But only if the tests are valid & don't end up perpetuating a
fallacy. If it meant going back to the drawing board, so be it.

The problem with *if* here is that it is a speculation. That has no
real use in the scientific method *unless* you can then propose a
test which would distinguish you hypothesis from the competing
ones...

Thus a given test *might* not be 'valid'. But to decide this would
require a suitable test, ideally also a proposed 'mechanism' for the
cause of the lack of 'validity' which the new test would probe.

You make it seem that I advocate rolling dice Yes, of course it's
more satisfactory to forward a displacing theory rather than merely
suggesting the existing one is flawed but where would we be if
someone was to suggest that lead in cosmetics was dangerous & others
said that they would continue using it until the doubter came up
with a substitute. As for speculation Jim, much good science has
come from it.

Without that, we have to work on the basis of using the hypotheses
that are consistent with the evidence we have, and trying to avoid
adding mechanisms which the evidence does not require, or ideas we
cannot test.

Many things *might* be the case. But that does not tell us they
*are* the case. For that we require relevant evidence. Alas, "the
evidence does not agree with my beliefs" is not actually
evidence... :-)

Until someone comes up with a watertight explanation why DB is
infallible or near as dammit so, I'll reserve the right to be
sceptical in the same manner that I've been sceptical of my own
hearing. To sum up,

No test method or experiment is "infallible". I am afraid that
science does not work like that. What it does it gathers evidence so
we can use that to assess how reliable or useful a given idea may (or
may not) be. If you wish for "infallability" then I'm afraid you will
have to ask a theologian, not a scientist or an engineer. :-)

Hence the qualification of 'near as dammit' to indicate a methodology that
ain't too controversial.

I'm not suggesting that DB testing is completely pointless but IMO,
can't be relied upon as the sole means of testing, especially when
some use it as a club to bash people with the idea that most
equipment sounds essentially identical. I feel more comfortable with
folks being cloth-eared than folks having so-called golden ears!

I agree - but only if the speculation is testable and some evidence to
support it can be gathered and assessed. So if we say a given product
is 'dangerous' we migh then regard it with caution, but then expect
some evidence to back up the assertion. If no evidence can be
provided, we can decide to regard the assertion as having no reliable
substance.

We may change our minds at a later point if evidence *does* appear.
But the change of understanding should be based on evidence.

Otherwise we would have to work on the basis of never doing anything
at all because it "might be dangerous".

I'm having a longish weekend in Edinburgh, so I'll be giving this topic a
well earned rest. I'm hoping to make it to Leith where a friend has a pair
of Art Stiletto's I'm curious to hear.