This article was featured in the January issue of Bartender magazine.
Blind Tastings…Eyes Not Quite Wide Shut
By Philip Duff
Not long ago, while sailing on the electric Interweb, a wee notice caught my eye. Among the many other internet advertisments promising to enlarge (or reduce) parts of my body I myself consider to be perfectly formed and correctly proportioned, this one was about a spirits competition.
Specifically, this competition touted it’s objectivity: ‘Judges did not know what the spirits were they tasted, hence the results were as copper-bottomed as can be’. It must be said that I cast a cold eye over their results for 2010 – and it was immediately apparent that this claim was what us wordsmiths like to term, ‘a screaming pile of shit‘. I knew that, dear reader, because there was not one single surprise in the results. To a man, the judges rewarded the flavours they are known to favour personally – strong, bitter, challenging – and dumped gloriously on the ones that they do not. There were no surprises, and this is pretty much a guarantee that the test was not blind. But didn’t the organisers promise…..? Yes, they did – but they didn’t didn’t deliver.
“Body language, as you serve samples to the judges, can be an indicator to what is being served.” Philip Duff
A real double-blind test (as used for new pharmaceuticals) is a very different kettle of fish. One team will design the test, one will execute it and a third will analyse the results: these teams will never meet. But Philip, I hear you say, surely that’s a little over the top for a liquor contest? Liquor, though, is a multi-billion-dollar industry, as large as construction or defense. Surely it deserves a quality contest you can trust?
Everything influences the outcome
– Do the people pouring the samples know what brand is which? You should pour from number-coded water bottles and not the original bottles. Also, body language as the samples are served to the judges, can be an indicator to what they are serving.
– Do the judges know which spirit brands are in the test? Can they catch a glimpse of the bottles backstage? Are there cases here and there?
– Are the judges able to see and hear one another? Are they talking? I mean, if David Wondrich is sitting next to you and makes a spewing noise and says “Jesus, that tastes like ass!” about sample #3, would YOU have the balls to rate it as anything but, er, ass?
– How many samples do they taste? 200 in a day isn’t unusual: 90 would be an easy day. 90 fucking samples! Even at a mere 7.5mls per sample swallowed, that adds up to 22.5 ounces of booze, or 90ml per hour for a day just short of 8 hours. Humans can only eliminate 30ml of 40% alcohol per hour, so a 90-sample day guarantees hammered judged. Wouldn’t want to be sample #89, would you?
– Who collates the results? Does he or she know which brand is which?
The Real McCoy
Proper blind tastings always produce surprising results. The most famous one of recent years was held on January 26, 2005, in New York, for an article in the New York Times. The tasting panel was star-studded: Eben Klemm, Simon Difford, Dale de Groff…you get the picture, right? It was a truly blind tasting of high-end vodkas. Just for a laugh, the host slipped Smirnoff into the selection. Rib-tickler, right? But here’s the thing: it emerged as best-tasting. As soon as the result was announced, for some panelists, it was the starting whistle for the Back-Pedalling Olympics, but for others, they graciously pointed out that they’d chosen it based on taste and that’s that.
“No surprises = not a proper blind tasting.” Philip Duff
So why spirits contests? Well, spirits brands find that potential distributors get tight pants when you can mention your delicious dram has won a gold medal in the XYZ Spirits Challenge Tasting Institute Fest-o-rama. And people like organising spirits contests because it can cost from $100 to $400 per spirit, per category, to enter. A big contest like San Francisco World Spirits Competition will have around 1500 entrants. Kaaaaaaaa-ching! Like games at a five-year-old’s birthday party, everyone’s a winner: out of those 1500-odd, around 900 will get a trophy, medal or citation, the grown-up version of a lucky bag. There is no consistency from contest to contest or from year to year.
Don’t believe me? Well, a chap who clearly has fewer saucy ladies in his smartphone’s contacts list than me went to the immense trouble of doing a multi-year statistical study of a large wine competition. His findings – to be found by Googling, ‘An Examination of Judge Reliability at a major U.S. Wine Competition‘ by the aforementioned chap with the pleasantly euphonious name of Mr. Robert T. Hodgson – in essence are that you could ascertain spirit quality more accurately through analysing beaver droppings than by such contests (I’m paraphrasing).
But never mind! Speaking as a vetern (bar contest) competitor: if you always enter but never win, eventually they ask you to present the awards…
Did you know…
Perhaps the greatest upset for a blind tasting was the 1976 ‘Paris Convention’ where wines from California consistently ranked higher than the top French vin on offer. It was up and up for the Californain wine industry after this – and the result is still debated today.
We recently got together with friends for a blind rye tasting. TWENTY-FOUR spirits were presented! Our efforts to keep everything blind probably weren’t quite enough, as some of us did see bottles beforehand, but the effort was there. Nothing was poured from an original bottle — the whiskies were poured into other vessels and numbered by one person, who then took those vessels and poured them into other vessels which he numbered himself.
After the tasting, the two charts of numbers were compared to match up which spirit was which. There were indeed some surprises, and with some results not so much. It was a lot of fun and still a great learning experience. Next time we need a third person to hide all the bottles before anyone shows up.