Page 1 of 2 12 LastLast
Results 1 to 20 of 36

Thread: The proper evaluation of audio equipment - where do we go from here?

  1. #1
    Join Date
    Jan 2006
    Location
    South of England, UK
    Posts
    4,016

    Default The proper evaluation of audio equipment - where do we go from here?

    As we have seen in the main forum, despite encouraging members to comment on the two definitions I proposed, there has been almost total disinterest in the subject. Clearly, the 'proper evaluation of audio equipment' is not a subject that is of any interest to the wider HUG audience. That's surprising to me as a speaker designer because it suggest that the design standard I set myself of assuming that my designs will be put to the test by the most analytical critic, is perhaps needlessly high. It explains why there is so much mediocrity and worse, why the public are willing fodder for the marketing machine that is consumer electronics.

    None of those more philosophical issues need concern us here. We've done our level best to highlight the way commerce works and of how marginal - or even regressive - features can be spun-up into added value. The removal of tone controls from home amplifiers is an example of denying the user the ability to tune the sound to his room, whilst presenting that as a sonic victory. It's utter bullshit of course as any recording engineer will confirm with a vast array of tone control facilities available at a touch of a button to him.

    Perhaps - although I've been banging on about this for some years now - there will be a tipping point where common sense returns to 'audiophile' consumption. I remain astonished that even during these very challenging economic times for western society that there has not been a shake-out of high-end audio suppliers. That must say something about the profit margins. Nothing wrong with a healthy profit of course, but is the consumer truly getting best value? How can anyone know unless the equipment is properly scrutinised using a test methodology which takes advantage of human hearing, specifically, that human senses are optimised for a fight-or-flight response. That means that we are good at making snap comparative decisions (which tend to have a high factual content, low emotional content), and not optimised for arriving at long-range wishy-washy decisions about preferences.

    Here I proposed two generic approaches to audio equipment analysis ....

    I detect another potential misunderstanding here. As no one has actually admitted to having taken part in an instantaneous A-B listening test, it must indeed seem mysterious. Let me summarise the position as I see it thus far.

    I propose some terminology to describe the two very different ways of listening to and evaluating audio.

    1) Critical listening a.k.a. instantaneous A-B comparison

    Definition: where a serious attempt is made to eliminate confounding variables. Conceptually, a test subject could be picked at random from the general population, literally off the street. The inexperience, health and motivation of the listener is not important. The absolute acuity of his hearing is not important. He/she will be presented with the juxtaposition of audio experience A switching over to B with only a fraction of a seconds delay (using a relay arrangement), and if there is a difference it should be audible. The listener can dwell on A or B as little or as long as he wishes, the important mental comparison is made at the point of switchover and not at any other time during the listening session. There is no requirement for audio memory at all. Prolonging the listening session does not accrue more useful relative information about A or B, because the listener will fatigue. Maximum suggested listening session perhaps 20-40 mins.

    2) Holistic evaluation a.k.a casual listening, long-term listening, recreational listening

    Definition: The listener's experience, health, hearing acuity, motivation and audio memory is of paramount importance. He/she will listen to A in isolation for perhaps several CDs. Then he will take as long as he needs (minutes?) to change over to B. There is no requirement to control replay levels or any other confounding variables. His impression of the sonic character of A or B accrues as he listens and his emotional response develops from what he hears, his prevailing mood, listening fatigue and innumerable internal mental processes. The listener argues that the longer he listens the more of an impression he develops of whatever he is listening to. At the end of the process (which could take days, weeks or months) the listener will pronounce judgement on A or B based solely on his audio recollection of how A or B has 'moved him' i.e. by recalling his audio memory of sonic events A and B separated over time.
    Those of you who replied to the above question have now been invited to this forum. Clearly you understood the question being asked, and were willing to express an opinion. That's just the sort of robust dialogue we appreciate! Shame it only represents 0.34% of total membership. that's life I guess.
    Alan A. Shaw
    Designer, owner
    Harbeth Audio UK

  2. #2
    Join Date
    Jun 2009
    Location
    Canada
    Posts
    841

    Default Objective truths ....'use the force Luke ...'!?

    I can understand your frustration with the small number who took part, but perhaps it seems like weighing in on whether the engineers who designed your car used a correct methodology to test engine performance - in other words, beyond one's ken (I don't think it is, but that's only my view).

    The reason I think it's such an important issue is the reason you expressed in one of your posts in the main thread: without some form of rigorous, objective evaluation, what's to stop someone from putting $3 worth of drivers into a $20 box and charging $10,000? From going for the quick buck instead of lasting quality? If that's an opportunity to be had, someone will take it - that's just how it is.

    For me too, the question is interesting because it deals with the nature of knowledge and the difference between objective and subjective knowledge. I strongly believe that there are real, objective truths about the world that exist despite our beliefs about them. That may seem unremarkable, but I think there are many people nowadays who do not necessarily believe that, strange as it may seem. My wife's an academic (a historian), so I know first hand from her and her friends and colleagues how pervasive is the idea in the social sciences and the humanities that all reality is socially constructed, that there are no objective truths, and so on. That may seem like esoterica, but I think it filters into the wider population, and is reinforced through mass culture, which would far rather deal in mythology and emotion (exciting!) than scientific truth (dull!). My favourite example is Star Wars: "use the Force, Luke" - what the hell is "the Force"?

    Anyway, I don't mean to ramble so I'll suggest one point of clarification. Many people speak of "blind" or "double blind" testing, and associate any form of A-B comparison with "double blind" testing. Now I know the two can go together, but they don't have to. It seems to me that the A-B comparison methodology you're describing is not at all blind, single or double. I.e. that the listener can know which is A and which is B, and still arrive at a valid result. Is this correct? Or does the outcome need to be validated with a blind test? If not, I'm even more perplexed as to why people would think A-B is not a good methodology, especially if the listener is in complete command of switching and timing.

  3. #3
    Join Date
    Jan 2006
    Location
    South of England, UK
    Posts
    4,016

    Default DBT and the solo user?

    Quote Originally Posted by EricW View Post
    ... Many people speak of "blind" or "double blind" testing, and associate any form of A-B comparison with "double blind" testing. Now I know the two can go together, but they don't have to. It seems to me that the A-B comparison methodology you're describing is not at all blind, single or double. I.e. that the listener can know which is A and which is B, and still arrive at a valid result. Is this correct? Or does the outcome need to be validated with a blind test? If not, I'm even more perplexed as to why people would think A-B is not a good methodology, especially if the listener is in complete command of switching and timing.
    Frustrated? Totally! I think this must be vaguely related to an Adult-child cognitive personality whereby it gravely concerns me when I see others deliberately putting their hands in the fire when I (wrongly) believe that I can reason them out of the inevitable outcome. I think it's also magnified when one employs some relatively young staff who have a history of a troubled home life. This modifies your role as an employer into a mentoring/parenting role, which generally works well for both sides.

    Anyway, your question: in honesty I don't actually know what double blind testing is (DBT). It could mean that you literally wear a blindfold in addition to not knowing what you are listening to just to be sure of no bias. Or it could be that the test operator (note: how to be both the sole test subject and sole test operator?) deliberately reintroduces certain items to double-check the repeatability of the listener's score. Or it could be whatever you want it to be!

    As to my setup: I'm going to partially quote myself here ....

    (to follow)

    but it really is important that the solo tester/listener does not know what he is listening to, especially towards the end of an evaluation process where he needs to be cruelly objective, especially if it is his own baby v. an established model, his or not. I work alone, not because I am an (especially) solitary soul, but I don't need or want or benefit from other opinions at any time during the design process. I don't care a jot what others think: either they will be wrong or they will be being kind. I cannot imagine that they could hold both the factual, measurable performance (the lab data) in their head whilst listening (because they were not in the lab and don't know how it measures objectively) and be able to weave a net of correlations between the objective data and how it sounds. The objective is not criticism for the sake of criticism; the objective is to know if shaping or shading more energy in this band or that would amend the subjective opinion, maybe reverse it.

    Unless you can deconstruct what you hear into at least broad frequency bands you cannot answer the $64,000 question: is what I am hearing an insoluble coloration problem (which would bring the design to a screaming dead end) or is it just an energy disposition issue which could be rectified by perhaps small adjustments in the cabinet, drive units or crossover. If you make an incorrect call at that point - and it's a 45/55 issue all too often to identify one or other - you have a 45/55 chance of bringing to market a product which is colored somewhere and somehow. Now, you may get away with that. Maybe nobody will notice. But if they are smart enough to use an A-B test against a product which does not exhibit that issue, then you have wasted a year or two of your life. A long legal case that collapses because of evidential integrity in court perhaps?

    What I do is plug (speakers) A and B into the relay box. I do it almost out of the corner of my eye because I do not want to pay attention to which is A and B. A may (or may not) be a Harbeth. B may or may not be a brand XYZ. Then having wired them up, and given a 1 sec. burst of nose just to check that all four are wired and the switcher is OK, I'll line-up a CD and walk away for an hour or two and busy myself with something unrelated - say typing here. Then, when I return, I settle down and with the relay driving A or B (just how it was left) I start to listen. I do not look at the speakers, but at a point on the floor between them and me. And off we go: flicking backwards and forwards between them. Within just a few seconds opinions are beginning to form in the brain about the relative performance of A v. B. 'Those bells are really bright on A ...'. 'B sounds comparatively thin on cello compared with A'..... 'A has fractionally more presence than B...'

    Then I leave the music playing with the switch in either A or B (doesn't matter) and walk away for minutes.

    Today I drilled-out and assembled a little relay A-B box - picture attached. You can see a nice chunky relay (yellow) which is controlled by a foot switch or hand (override) switch. That's a mono change over, so if you want to switch one stereo amp to A or B speakers, you would need to duplicate this box. It's incredibly simple stuff really!
    Attached Images Attached Images
    Alan A. Shaw
    Designer, owner
    Harbeth Audio UK

  4. #4
    Join Date
    Jun 2011
    Location
    Belgium
    Posts
    115

    Default Actual experience

    It depends on the objective of the test I would say. A simple A-B comparison would suffice if there could not be a bias towards either A or B. For example, in speaker design I can imagine this being done to compare 2 variations of a crossover circuit.

    When there is a bias, e.g. comparing amplifiers, cd players.. a blind test is needed to eliminate the bias. Unsurprisingly all of a sudden it's a lot more difficult to hear differences (in reality, it's impossible).

    I'm going to tell you about a test I was actually a part of, although not with audio equipment, it's very relevant. I'm a musician, a horn player, and I play and teach for a living. We did a test involving 4 instruments; A smaller bore professional model, a large bore professional model, a medium bore student model and a mystery instrument. First all of us were required to answer some questions before the blind test, these were questions about what we thought the sound and playing qualities of these instruments would be (and believe me, our 'community' has some pretty strong convictions about that). The answers were predictable, the pro models scores high on things like intonation, the student model much less so. The smaller bore horn would sound brighter and more upfront, while the larger bore model would sound thicker and fatter.

    When the actual listening test was done, the results were very surprising. They were in fact all over the place and no one seemed to know which instrument was which. What was perhaps most surprising was that the 4th mystery model was in fact the 3rd horn played (the student model) and that it scored a lot differently, even though they were played one after another, by the same player! In fact, when the scores were counted, horn n°3 scored lowest and n°4 highest. The same horn lost and won the test.

    PS: This was done in a medium sized concert hall. The results would probably have been different in a small room, but the objective was to compare the sound of different models in the hall.

  5. #5
    Join Date
    Dec 2010
    Location
    Canada
    Posts
    147

    Default Is the music important?

    I understand the technical use of the comparator, switching back and forth, what I do have a question about is the music that you (or I) would choose; would it be symphonic, a long Beethoven track, say the 9th, or would a rock or pop song or a jazz piece be suitable, (all on CD/hard drive, thus one song after the next)? Or would the type of music not make a difference to what one is trying to hear.

    I still would like to make my own comparator, but aside from a bit of soldering talent, the parts necessary and what connects to what is something that would require very close adult supervision...and clear drawings.

    George

  6. #6
    Join Date
    Jul 2011
    Location
    The Netherlands
    Posts
    11

    Default A-B against reference?

    A question to understand the process. Are you using this method to benchmark a Harbeth speaker against other established models or to develop a new speaker?

    I can imagine that, when developing a new speaker, one had to A-B against one or two reference speakers.

    {Moderator's comment: do you mean compare against Harbeth existing models or competitors?}

  7. #7
    Join Date
    Jul 2011
    Location
    The Netherlands
    Posts
    11

    Default A static reference?

    Quote Originally Posted by espakman View Post
    A question to understand the process. Are you using this method to benchmark a Harbeth speaker against other established models or to develop a new speaker?

    I can imagine that, when developing a new speaker, one had to A-B against one or two reference speakers.

    {Moderator's comment: do you mean compare against Harbeth existing models or competitors?}
    What I mean is when doing a A-B in the development process it seems to me that A (or B) should be a chosen "static" reference speaker (either a Harbeth or competitor), used in the development of every Harbeth speaker. I think that in an A-B comparison you always need one static to compare against, instead of a variable A and B. But I could be totally wrong.

  8. #8
    Join Date
    Jan 2006
    Location
    South of England, UK
    Posts
    4,016

    Default What should the reference be?

    Quote Originally Posted by espakman View Post
    What I mean is when doing a A-B in the development process it seems to me that A (or B) should be a chosen "static" reference speaker (either a Harbeth or competitor), used in the development of every Harbeth speaker. I think that in an A-B comparison you always need one static to compare against, instead of a variable A and B. But I could be totally wrong.
    Absolutely correct. You need something to compare against, as you say, a reference. How to select that reference then? That's an important question.

    The first point is that nothing sounds like real life. No speaker system fills the room with sound as real instruments do. And if I am honest - and I wouldn't admit this on an open forum - when I go for months without listening to any hifi, just live music, bird song, my portable radios, live human voices, it takes me quite a few hours to get back to accept what (any) two boxes can do is even remotely a passing representation of what I hear live. However, the more one listens the more cues you pick-out from the reproduced sound that connote experiences you had live, so your initial hostility slowly breaks down as you become seduced by the reproduced sound until you accept it as unremarkable, and allow yourself to believe that what you are hearing is 'just like the real thing'.

    It's really important to pass through that barrier when listening by ear alone because if you don't (and I speak from experience) you reach for the soldering iron and immediately start to crank-up the tweeter by many dBs to add back the 'sparkle' that is live sound and that isn't reproduced sound. You can fake that gossamer-winged 3D lightness of real sound (I mean, at the venue, live, you present) for a short while by boosting the top. The problems is that as you so so it increases your HF sensitivity, such that an hour later you reverse the boost and that then sounds 'right' and a hell of a lot fatiguing. So you are back to where you were - accepting that reproduced sound and live sound are not at all the same experience. Make sense?

    As for the precise reference. It depends what you are trying to achieve. How valid would a P3 comparison against a M40.1 be? Is that a market-relevant comparison? And which one would you change to sound more like the other? or would a more viable comparison be an M30.1 v. SuperHL5 or Compact 7ES3? Or against a XYZ medium sized model? Or a Quad electrostatic which, aside from the technology difference, is a very different aspect ratio and as a dipole drives the room in a completely different way? Or should one look outwards to the user and ask what sort of comparison he would make, at home/in the dealers?

    And what about types of music (post #5)? Could one argue that if the target customer only plays heavy metal thrash that the primary (or even only) music to be played is that? Or only jazz? Or classical piano? Or if the brand prides itself on being a General Purpose Broadcast Monitor (GPBM) as we do, we better be sure that it handles all types of music (my position, providing not too loud). Or the comparison should be against this month's WhatHiFi 5* winner? Or Stereophiles latest rave? Or a well respected classic speaker from the 60s? Or a well regarded speaker at half (or twice) the price? There is no fixed answer.

    What I can say here is that I ceased to routinely listen to competitor's speakers twenty years ago and have only listened to one of two since under casual and critical conditions as already defined. How? Why? First, under critical (A-B) conditions the competition (I have heard, this cannot be a universal truth) sounds so unnatural, so hopelessly unbalanced and recently, so thin and bright that I have learned nothing whatever beneficial from the effort expended. Second, once you have switched backwards and forwards to the very best pre-RADIAL speakers (e.g. 1970s/80s BBC monitiors from Rogers/Spendor using polypropylene cones, not current ones) on A-B they sound very balanced but so terribly foggy. When switching over to RADIAL you can hear details which are lost on the PP cones due to coloration. So, as no new cone materials have been presented since RADIAL (nearly twenty years ago now) it's most unlikely that I am going to have a eureka moment any time soon. I take this for granted, but unless you actually make the box and try it for yourself, you would, I guess, think that pretty much every A sounds like every B and that you'd be straining to hear the difference even under instantaneous A-B.

    Nothing could be further from the truth: the differences are shocking. They shouldn't be. There has been no real progress since the 70s.
    Alan A. Shaw
    Designer, owner
    Harbeth Audio UK

  9. #9
    Join Date
    Jul 2011
    Location
    The Netherlands
    Posts
    11

    Default 1950s comparisons

    Quote Originally Posted by A.S. View Post
    It's really important to pass through that barrier when listening by ear alone because if you don't (and I speak from experience) you reach for the soldering iron and immediately start to crank-up the tweeter by many dBs to add back the 'sparkle' that is live sound and that isn't reproduced sound. You can fake that gossamer-winged 3D lightness of real sound (I mean, at the venue, live, you present) for a short while by boosting the top. The problems is that as you so so it increases your HF sensitivity, such that an hour later you reverse the boost and that then sounds 'right' and a hell of a lot fatiguing. So you are back to where you were - accepting that reproduced sound and live sound are not at all the same experience. Make sense?
    That absolutely makes sense and is also my experience But what about, this can be totally of topic and if so please moderate, the live versus recorded sound A-B comparison in the 1950s with Wharfedale and Quad. Is the human brain that easily fooled?

    It indeed seems that there is no real progress in speaker technologie (and audio gear in general) the last decades. The opposite is probably true...

    The only way to go forward is find a way to optimize the speaker/room integration.

  10. #10
    Join Date
    Jan 2006
    Location
    South of England, UK
    Posts
    4,016

    Default

    Quote Originally Posted by espakman View Post
    ... But what about...the live versus recorded sound A-B comparison in the 1950s with Wharfedale and Quad. Is the human brain that easily fooled?
    I've wondered about that too. Points to consider ....

    1) How far away were the listeners?
    2) How big was the venue (town hall?)
    3) How hard or soft an acoustic?
    4) How many in the audience? Dozens? Hundreds?
    5) Exactly what type of music was being played?

    My guess is that you're referring to the town-hall type demos (I'll look for a picture). In which case, my approx. answers would be:

    1) Average 10m away
    2) Big town hall, speakers on the stage
    3) With a large audience seated, very well damped acoustic
    4) 200+
    5) Organ music was frequently used to impress. Other solo instruments, all mono (predates stereo).

    Perhaps?

    Just about any speaker will sound great when playing organ. Interesting that voice wasn't used - or was it?

    Also consider that in the 50s, the expectation of the public was for the rolled-off top of AM radio.
    Alan A. Shaw
    Designer, owner
    Harbeth Audio UK

  11. #11
    Join Date
    Dec 2010
    Location
    Canada
    Posts
    147

    Default Using real live sound as a reference during speaker evaluation

    The difference of live music as compared to recorded, even if done by the most expert recording engineer, is strikingly obvious when you hear live music coming from unexpected places. For example, I was walking by a somewhat industrial looking building in Budapest a few months ago, and heard piano music, and almost instantaneously I identified it as being live; it was only when the pianist stopped playing, plunked a few notes as a pause and began again that I was able to confirm my suspicion. Interestingly, it was only then as I walked a bit further on that I noticed the sign over the door identifying the building as a dance school. The piano was playing for a dance practice.

    I have also noted on several occasions that live music is different than recorded, even when you hear a music playing out of a window of a club or someone’s apartment or a band or single musician is playing out on the street, you just know it’s live! Another example was very recently when I was outside on the deck of the house and my wife was talking to my son via FaceTime. Even though he was in Canada and I was in Hungary, the live voice as compared to a pre-recorded podcast for example, was easy to identify.

    So, conclusion, reproduced sound does not sound real, it just sounds really close. And therefore, can A-B testing be done live vs recorded in order to produce a speaker that is almost live, or does it just become, as Alan said, eventually shrill and fatiguing due to pushing the high end…?

  12. #12
    Join Date
    May 2011
    Location
    India
    Posts
    541

    Default Room correction and WAF?

    Quote Originally Posted by espakman View Post
    It indeed seems that there is no real progress in speaker technologie (and audio gear in general) the last decades. The opposite is probably true...

    The only way to go forward is find a way to optimize the speaker/room integration.
    I am not so sure...I suspect that there has been little real progress in the traditional audio industry. Which is the reason for the almost hysterical peddling of high end audio gear, supported by equally hysterical mainstream media reviews.

    The real progress has been in the areas that audiophiles look down upon. The cheap and cheerful iPod that can be a hifi component if well used, and digital streaming of audio as exemplified by some one like Sonos.

    Speaker/room integration is certainly an area to be explored. The traditional means of improving acoustics are very wife/family unfriendly, requiring a dedicated listening room to be earmarked for domestic harmony. I for one have never understood the sense in sitting in one in splendid isolation.

    But equally important these days is the cool factor of how good the system/speakers look. And do not have the clutter of cables and connectors that lead to hifi audio ending up as a hobbyist thing - which the wives put up with as one more toys for boys thing.

  13. #13
    Join Date
    Jan 2006
    Location
    South of England, UK
    Posts
    4,016

    Default The liquid tone of (some) BBC recordings

    Quote Originally Posted by Macjager View Post
    .... I was walking by a somewhat industrial looking building in Budapest a few months ago, and heard piano music, and almost instantaneously I identified it as being live; it was only when the pianist stopped playing, plunked a few notes as a pause and began again that I was able to confirm my suspicion. ...
    I was thinking this over when the BBC Lunchtime Concert started on my car radio (link to follow/clip to follow). Once notable characteristic of a sound recording made by the BBC v. a commercial recording of the type you'd by on a CD is relate to how much ambience there is. Because this recording style differentiation goes back at least a generation, it must be deliberate not accidental. It must be how classical music BBC sound engineers - or perhaps more accurately BBC producers - are trained. Doubtless it's not universal in the BBC, and there will be (many) exceptions but it is a hallmark of a BBC recording/broadcast which you rarely hear elsewhere.

    How and why do they achieve that greater ambience? It can really only be a factor of the recording space plus the position of the performers in that space plus pulling the microphones backwards from the performers (from where they would be placed for a commercial recording) and letting those mics capture more of the reverberation and proportionately less of the direct sound. Recording is always a balance with a fulcrum somewhere between bathroomy and as dead (being on a mountain peak shouting at the top of your voice) with no echo at all. Why would the BBC (and I'd anticipated other western Europe broadcasters too) have a particular view on where this live/reverb balance should be? It surely is related to creating the illusion of a different sort of listening experience to a commercial CD. The essence of a philanthropic, publicly funded broadcast balance is to place the listener in a good but definitely not front row, to be part of an audience with a shared experience, coughs and squeaky chairs and all. Conversely, the commercial recording seems to strip out any pretence of you being amongst others: you've paid your money and the orchestra is performing for you and you alone, and you've a front of house seat.

    How does this relate to that lightness of touch that you mention indicated a real piano? I'd say that the BBC recordings (as a very general best-practice rule) build-in extra ambience to compensate for what is partially lost when we playback at home resulting in a overall more lifelike experience. Not suitable for pop music, but I really do like that liquid tone on piano.
    Alan A. Shaw
    Designer, owner
    Harbeth Audio UK

  14. #14
    Join Date
    Dec 2010
    Location
    Canada
    Posts
    147

    Default Church acoustics

    Whilst reading the above comments from Alan, I was listening to Radio Paradise, an Internet radio station, that was playing a Tom Petty song. As I drifted between reading the article and listening to the song, I noted that the music seemed, as Alan describes, lacking in ambiance; instrument directly to mixing board. I then thought that this sound (?) is present in certain music, and my usual response is to want to turn up the volume...which leads me to think that having the sound bounce around our listening space a bit is desirable (certainly not referring to a huge reflective concrete space with a small carpet and one chair...)..ambiance in sound (sounds reflecting off something as it gets to our ear) is natural and therefore accepted.

    Unless we live on a mountain, there will always be sound reflecting off something. This complexity of sound reproduction therefore would preclude simple solutions in audio system design, ie., no tone controls, strange fixations on cables and interconnects etc..
    So, do we need to have a slider/button that can add or subtract ambiance (echo/reverb/?) to the sound, so the listener feels he is listening in a place that is "live" or closer to it...?

    If you have a chance to listen to the Trinity Sessions by the Cowboy Junkies you will hear an interesting recording with just the right amount of ambiance. They also have a lot of technical information on the web about how they set up in the Church, quite fascinating! http://en.wikipedia.org/wiki/The_Trinity_Session


    Cheers

    George

  15. #15
    Join Date
    Mar 2006
    Location
    USA
    Posts
    149

    Default BBC concerts

    Hope this isn't to much of an aside. But looks like the BBC lunchtime concerts Alan mentioned could be a complete performance of the Beethoven piano Sonatas ( probably my Desert Island choice if only allowed 10 cds!) Performed by three different pianists. Barry Douglas, Llyr Williams and Leonidas Kavakos.

    Started this Tuesday here:
    www.bbc.co.uk/iplayer/radio/bbc_radio_three/20120814
    The concert the following Tuesday 21st has no 15 (Pastorale) my personal favorite. Recommended for all lovers of piano music.

  16. #16
    Join Date
    Oct 2011
    Location
    Canada
    Posts
    97

    Default Science, rationalism and self delusion?

    Quote Originally Posted by EricW View Post
    I can understand your frustration with the small number who took part, but perhaps it seems like weighing in on whether the engineers who designed your car used a correct methodology to test engine performance - in other words, beyond one's ken (I don't think it is, but that's only my view).

    The reason I think it's such an important issue is the reason you expressed in one of your posts in the main thread: without some form of rigorous, objective evaluation, what's to stop someone from putting $3 worth of drivers into a $20 box and charging $10,000? From going for the quick buck instead of lasting quality? If that's an opportunity to be had, someone will take it - that's just how it is.

    For me too, the question is interesting because it deals with the nature of knowledge and the difference between objective and subjective knowledge. I strongly believe that there are real, objective truths about the world that exist despite our beliefs about them. That may seem unremarkable, but I think there are many people nowadays who do not necessarily believe that, strange as it may seem. My wife's an academic (a historian), so I know first hand from her and her friends and colleagues how pervasive is the idea in the social sciences and the humanities that all reality is socially constructed, that there are no objective truths, and so on. That may seem like esoterica, but I think it filters into the wider population, and is reinforced through mass culture, which would far rather deal in mythology and emotion (exciting!) than scientific truth (dull!). My favourite example is Star Wars: "use the Force, Luke" - what the hell is "the Force"?

    Anyway, I don't mean to ramble so I'll suggest one point of clarification. Many people speak of "blind" or "double blind" testing, and associate any form of A-B comparison with "double blind" testing. Now I know the two can go together, but they don't have to. It seems to me that the A-B comparison methodology you're describing is not at all blind, single or double. I.e. that the listener can know which is A and which is B, and still arrive at a valid result. Is this correct? Or does the outcome need to be validated with a blind test? If not, I'm even more perplexed as to why people would think A-B is not a good methodology, especially if the listener is in complete command of switching and timing.
    As a trained historian I have to comment on the above. While historians strive to be as objective as possible they understand that they are dealing with artifacts--traces--which have their own inherent biases and set(s) of problems. That is why good historians actually utilize 'scientific methods' to arrive at reliable knowledge based on close readings of the texts. However, this post-structuralist relativism you refer to pervades literary departments and does exist in certain methodological approaches to history.

    How does this relate to the discussion at hand? Well, the one thing the social sciences has given me are critical thinking skills--something many people in audiophilia apparently lack.

    I wonder, however, if this lack of critical thought is really just that. Or, is it a lack of understanding of basic scientific principles???

    Then again there are real engineers designing amplifiers who genuinely think that their amplifiers sound better than others...

  17. #17
    Join Date
    Jan 2006
    Location
    South of England, UK
    Posts
    4,016

    Default Self-worth and the audiophile

    Quote Originally Posted by Zemlya View Post
    ... the social sciences has given me are critical thinking skills--something many people in audiophilia apparently lack. I wonder, however, if this lack of critical thought is really just that. Or, is it a lack of understanding of basic scientific principles???

    Then again there are real engineers designing amplifiers who genuinely think that their amplifiers sound better than others...
    I find it deeply disturbing that despite all I've written, all the evidence of previous scientific studies of A v B which seem to usually or even always conclude that under controlled conditions, broadly similar amp designs are indistinguishable, I still read - this very week - an individual having the cheek to use this platform to promote the 'extra openness' (or whatever) of his pet amplifier. I am truly sick and tired of it. I'm actually now past the point that I am going to expend my energy to re-educate these misguided individuals. I have also hardened my heart so that I actually don't care if they throw precious money after a marketing fantasy - even in these difficult economic times.

    I think my last contribution will be in the next week or so to record the output as it drives the speakers of, say, a 1960s amp v. a highly respected contemporary brand. I have the amps and just need to make some time (I'm really busy at the moment). Maybe I'll surprise myself and there will be audible differences; I must keep an open mind about that.

    I have away on business this week and taking a stroll around the hotel soon found myself in the entertainment district. Too late for a live show I thought I'd treat myself to Batman: The dark night returns in IMAX. Thin plot; dreadfully written dialogue; far, far too loud throughout; bass not only ridiculously loud but of poor quality and worst of all, the dialogue channel was a borderline unintelligible. And that's state-of-the-art cinema at $20 a seat. On leaving, I overheard two lads (early 20s) behind me and a middle aged couple commenting how they too had struggle with the dialogue audibility. I think I'll write to IMAX UK - not that there will be any result.

    Anyway, your final comment is about amplifier designers believing in their creations. As a class, these chaps are spectacularly incapable of making proper A-B comparisons. I've seen one dither about switching off/on and soldering in one resistor from brand X or brand Y and then passing judgement, which in my view is utterly worthless. But they need to believe that they have a power and influence over events which somehow completes their self-worth. After the film, I went for a night cap in a local club - deserted I was told because of the economic climate - and fell into conversation with a nice young lady (30) from Poland who claims to be studying law there and doing holiday bar work in the UK for pocket money. Sitting around a bar you can always learn something about the human condition; it's easy to fall into conversation with strangers providing that you let them do the talking and have no intention of revealing anything of your work or profession. Slowly the picture unravels: she has an unshakeable belief that she is beautiful. The seed of this idea was (probably) planted by her doting father when she was a small girl and (I'm guessing) he reminded her of this every day. Naturally, she accepted it as a fact. Her long-term boyfriend ran with that baton until, for whatever reason, he stopped saying it. The sad fact is that in that environment, she was comparatively unexceptional. It's an awkward situation when someone who is so desperately hunting for a compliment indicates what they want you to re-affirm; you are torn between telling an untruth which merely perpetuates their delusion or steering the subject away leaving the question hanging in the air when you both know it is unresolved. What is kinder? What is cruel? Can her self-image and the reality ever be re-aligned? Should I try to play along even though my brain just would not allow my mouth to utter those affirmative words because they were just not true (even though it is ungallant not to)? Is it my duty to do anything other than drink and listen? Do I care? Yes I do; I'm reliving the three hour conversation as I write this and it pains me to think that she is there now, days later still desperately needing to prop-up her identity, using the warm words of strangers as positive feedback. Trapped in the moment, unable to move-on, unable to create a real place for herself in the world despite her evidently brilliant legal brain and income potential. Trapped in a self-image loop. What a miserable existence. I suggest that 'audiophilia' has much in common with Jessica from Poland.

    The astute marketing man positions himself as the proud father bouncing his bonny baby consumer on this lap creating respect, then telling him how wonderful he is and then how much more wonderful he would feel if he purchased this and that fantasy. This is the well oiled process of 'setting-up the consumer to fail' and is the normal way to sell lifestyle products these days. I find it deeply upsetting that the consumer cannot see how malleable he is to the marketing expert. There is a way out: self-awareness. Not one consumer in a thousand wants to be more self-aware as the HUG experience has shown ad nauseum.

    Nothing I could have said to Jessica or to the hardened audiophile will make a scrap of difference. They'll plough on just the same!
    Alan A. Shaw
    Designer, owner
    Harbeth Audio UK

  18. #18
    Join Date
    Mar 2006
    Location
    USA
    Posts
    149

    Default AB testing - essay

    Would like to hear Alan's thoughts on this essay/paper. As it seems to question aspects of AB testing with regarding audio.

    I stumbled across it simply because one area I was trying to understand how possibly other designers other than Alan have managed to compare equipments with quick A-B changes whilst the Musical passages listened to would inevitable be different between switching.

    This essay brought that question / problem up.http://www.anstendig.org/ABTesting.html
    Again I have no strong opinion yet on this but thought it may help clarify our position here.

    {Moderator's comment: are you sure that you really understand Alan's method? If you press the switch within about fifteen thousandths of a second you have made the change over. How far can a 'musical passage' proceed in a fifteen thousandth of a second?}

  19. #19
    Join Date
    Jan 2006
    Location
    South of England, UK
    Posts
    4,016

    Default

    Having read the paper the word 'cobblers' comes to mind. Written to support a personal theory and repeats the mantra without example or justification ...

    Direct visual comparison has been accepted for centuries as scientifically accurate. But direct comparison is possible only with sight and impossible with all the other senses. That fact is probably the most pertinent scientifically established fact about all sensory perception.
    Unfortunately, a large part of the audio research that has already been published has utilized AB or similar testing that is simply a misapplication of visual criteria in the realm of sound. All of that research has, therefore, to be considered invalid. If any valid conclusions have been reached by these methods, their acceptance will have to wait until they can be confirmed by means that are scientifically accurate. It is difficult to comprehend the enormity of this situation. Whole edifices of scientific thought, methods, and practice have been built upon this scientifically invalid procedure. No matter how the procedure is refined (as in double-blind AB testing, using two or more blindfolded subjects and comparing components, etc., in such an order that the subjects could not guess their identity), there is no possibility of dependably recognizing subtle differences.
    I profoundly disagree. I note that the author does not indicate that he himself has made a switcher box. The style of the writing doesn't inspire me to confidence in the opinions. We here try wherever possible to use home-made examples to illustrate a point. Yes, it takes my time to conceive and prepare and test run, but I strongly believe that if one makes a claim about audibility issues, it should where possible, include worked examples. Such as my examples of a piano recording here.

    I wonder if you can really visualise in your mind's ear (if you know what I mean) what virtually instantaneous switch-overs sound like. OK, here is a very quick mock-up. Imagine that you are in the hot seat with a hand or foot switch driving a relay change over. We know that the relay needs fifteen thousandths of a second to click from A to B, during which time there will be a fifteen thousandths of a second gap in the music because neither A nor B is receiving a signal. Tell me how many times the switch has been operated in the following musical clip - this is exactly what you would hear if you were comparing speakers A v. B or amp X v. Y. How much (or little) audio memory do you need to make this procedure a valid one? Despite what the author of your paper said, isn't this a close approximation to placing two coloured objects one atop the other? I'd say it is.

    Loading the player ...
    Clip 1

    Can't hear any difference? Good, that's because there isn't any difference between A and B. A and B are the same audio file, just with a minute pause introduced at certain random places (on both channels) to simulate the relays operating. There are in fact 13 simulated switch-overs, 13 gaps in the music each of 0.015 seconds. I'd hope you could hear perhaps 6-10 of those change-overs: some are inaudible because the gap is masked by some feature in the music at that random point. So, Clip 1 is All A. No B at all.

    If we now operate the relay once before the music starts and listen to all B only (no A at all) we hear this:

    Loading the player ...
    Clip 1B

    How does that sound in comparison? Is A the same as B? How would you describe the difference between A and B (if any)? How would you like to play A today, and play B next week and make a reliable mental comparison? Or even after dismantling the listening rig for A, going and making a cup of tea, returning and ten minutes later re-starting the comparison. Absolutely and utterly useless as a piece of science I'm sure that you'll agree.

    Now, leaving the time markers where the A-B switching was applied to Clip 1, I have applied an audio effects which simulate in an extreme way, what we may be presented with in real an A-B comparative session, for example, comparing two loudspeakers. So this is what we hear when we switch between A and B with 15 thousandths of a second gap between them as the relays change-over:

    Loading the player ...
    Clip 2

    Surely it must be obvious to the least able listener (who himself decides when to make the change-over and hence is prepared for a sonic event change which he has triggered) that at switch-over that A does not sound the same as B. It's crucial (in my methodology) to remember that the listener himself is making the decision to switchover. He can do this as frequently as he likes. As fast as his finger will permit or every few minutes. I would encourage him to changeover frequently in the early minutes of the listening session as he builds a mental model of what he is hearing and of the gross differences, rather like priming a mental pump with as much early exposure data as possible. Decoding what he is hearing into frequency bands and resolving more subtle sonic events will certainly need longer focused attention to A or B alone, and then once an effect detected, switch to the other. This is just a 50 second clip: we are only just about becoming familiar with the differences when the clip runs out, but were we playing 30 mins. of music or so, we would have enough data to make some useful comparisons between A and B which would be unlikely to be contested by more exposure. What I notice is that my sensitivity to the sonic differences is enhanced the more switch-over comparisons I hear in Clip 2, so that towards the end of Clip 2, A and B sound distinctly different even iof they didn't sound much different at the outset. This is exactly what we want; increased hearing acuity through training of our own ears, through comparative exposure.

    The approach is one of using musical events to as they pass the listener to pique his curiosity as to how that event (instrument/acoustic space) sounds on A or B and to encourage him to effect a switch-over. No musical events are so transitory that they appear and disappear within a tiny fraction of a second such that they would have ceased by the time the switch-over operates: musical events take seconds. This gives us time to hear the same event on A or B. There is nothing to prevent the listener from rewinding the music or putting the music into a play loop for close listening to a particular feature. In practice, that is a degree of precision not required - the differences between loudspeakers are generally not that subtle.

    What do you experience when listening to these three clips?
    Last edited by A.S.; 20-08-2012 at 05:44 PM. Reason: A-B in practice - examples (revised)
    Alan A. Shaw
    Designer, owner
    Harbeth Audio UK

  20. #20
    Join Date
    Mar 2006
    Location
    USA
    Posts
    149

    Default A-B thoughts

    Thanks Alan for taking the time (again!) to make up those illuminating clips. I'm pretty much now with you on all this.

    The three clips do really help clarify how it would work in practice and you can hear the differences ( at some points clearer than others) in clip 2. In Clips 1 and 1b you can hear the difference (b being less 'bright' for want of a better word) but this was again when listened after a short period. Your point about after longer change over periods would surely make one more uncertain ( opposite to what the essay I attached suggested).

    The authors resume certainly is interesting. especially his mystical work!http://www.anstendig.org/Vita.html

    I would still love to hear a report from you, if you ever get the chance to try a super duper high end amp with your selector. As part of me still hopes it would reveal some small but worth while improvements to the musical stereo illusion we all enjoy. Maybe you already have? I'm just not sure from what I've read in the forums if you have or not. I know you are confident of little difference but would still like confirmation from someone I trust.

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •