Thus spake Jim Lesurf:
I can't cite examples but my impression FWIW, is that calibrated
variations can be fairly large before becoming significant. If that
is indeed the case, I suggest that using DB testing for testing
auditory differences is largely pointless.

I don't regard it as 'pointless'. Its 'point' is to indicate what the
actual limits of perception may be, regardless of the beliefs or
wishes of the individual. The results show that people often tend not
to be able to hear differences that they believe they can.

If subjects can't hear fairly large differences in a calibration cycle, I
can envisage 2 explanations. Firstly, some/many/most subjects are fairly
insensitive to variations & by definition, would be wasting money by buying
expensive audio equipment for sonic reasons alone. The second, is that
because the way the mind works, comparing sequences such as replaying the
same piece of music is going to confuse the subjects & muddy the results. I
can imagine this explanation being very inconvenient to many because it
throws in hidden variables such as how reliable human memory is & its
effects on the outcome. I only entertain this possibility because my own
experience suggests measuring qualitative stuff can be damned difficult. A
lot of people also state they can hear differences beyond measurability.

I wouldn't be surprised if a lot of research has already been done that
would be relevant to perception in general that can be applied to audio.
Lets take something less controversial like visual acuity. Photographs are
printed out at different resolutions or maybe colour depth & subjects asked
to make comparisons. In one test, the photos are laid out side by side & the
viewer asked which version they prefer & why. The next test, the photos are
viewed singly but in sequence. Both tests can be randomised. My assumption
would be that subject's scoring when comparing photos side by side would be
a lot higher than when viewed singly in sequence. Now, to me, this would
demonstrate how much effect memory has on perception. I can't think of a way
to do the same thing with auditory tests apart from the ludicrous idea of
playing back different criteria in each channel! The difference with audio
is that its coherence happens over time where a still picture's doesn't.
Pause a movie & the picture becomes static but retains some meaning. The
sound just disappears completely or mathematically would become set of a
sine waves of the time slice before the pause with the correct amplitude.
The photo test's relevance to audio is that the perceived differences vary
according to how much memory is involved in making comparisons (or how the
test is conducted.) The differences are still real & measurable with the
photos. I'm suggesting that although testing involving memory may mask
differences, those differences maybe indeed be real & considered to be
worthwhile by some people.

There some weaknesses here such as why doesn't testing using equipment not
track these supposed differences? One comment I can make is that most tests
are perhaps too static & are over-simplified. Another objection I have that
it goes against the grain of simplicity by heaping on more variables. Just
because many people believe in ghosts, doesn't mean they exist. I don't deny
that I may have built myself a house of cards here!

I heartily wish I could suggest alternatives but I can't.

Well, from the POV of the scientific method a hypothesis has to be
testable to have any validity/meaning. So if you/someone can't
propose and carry out an appropriate alternative we have to stick
with hypotheses we *can* test. This is to avoid people simply
believing whatever they choose, regardless of the reality.

But only if the tests are valid & don't end up perpetuating a fallacy. If it
meant going back to the drawing board, so be it.