Double-blind Experiments for Dummies: Wine - Part 2

This is part 2 of yesterday's post entitled Double-blind Experiments for Dummies: Wine - Part 1.


As can be seen from yesterday, trying to conduct a proper double-blind experiment in the home is pretty hard. I identified 5 potential issues with my own lengthy experimental setup procedures (and I haven't yet mentioned how questionable the results would be in an experiment on only 4 participants).

This highlights why it always pays to be sceptical about reported results from scientific studies without first reading all the details, caveats, experimental snafus, etc. It especially highlights why pseudo-science is a load of crap; it relies on anecdote, selection bias, and the placebo effect, rather than rigorous scientific principles.

And now onto the experimental procedures of the wine-tasting experiment.

Experimental Procedure

Results were broken up as follows (example data):

Code	Tasting #	Score out of 10	Guess	Comments
A	1	6	KH	Strong taste, long flavour
AB	5	4	407	Too strong

A code of AB would mean that in the first round (tastings 1-3), the bottle had code A, whereas in the second round (tastings 4-6), the bottle had code B. In the above example data, the two rows show an inconsistent guess as to which wine was being drunk.

At the start of each round, and with my eyes closed, I shuffled the bottles around.
Issue #6: By not also getting others to shuffle, who didn't see me shuffle the bottles, there could be some memory of which bottle was which. I doubt this will have been a problem in this experiment, but that's not up to me to judge.

I poured 2 shots of a wine into each glass, and we drank.
Issue #7: My wife sometimes only had one shot of wine, or poured out some of hers, so she didn't necessarily experience the same amount of pain having to get through the wines that were particularly strong in flavour. This could have affected her scores, and therefore guesses.

We minimised discussion until each person had written their results.
Issue #7: It was a social gathering, so there was still some discussion, especially between rounds, where we let our guesses be known. This is probably the biggest issue with the experiment, as there could be a tendency towards group-think, and this was intended to be a scientific experiment, rather than a focus group.

After each bottle was used in each of the three rounds, it was put aside, in order. At the end of the round, I would pull off the outer paper bag, in order to write down the code that was written on the inner paper bag. That code corresponded to the bottle in the round that had just been completed. After the first round, the code on the bag that was being taken off would be noted down, before the code on the next bag. This would allow a chain of codes to be maintained so that each bottle of wine could be tracked back through the rounds, despite the shuffling of the bottles.
Issue #8: After the first round, the codes that were written on the brown bags in fluoro yellow were technically visible, but in relatively low light, with extremely low contrast writing, I'd be very surprised if someone could see those codes. When I was trying to read them between rounds, I really had to focus.

In between tastes, we each had some water in our wine glass in order to rinse out the previous wine and to cleanse our palates.
Issue #9: We conducted this experiment over dinner, so: 

• Roughly the first round was before eating;
• Roughly the second round was while eating slow-roasted leg of lamb, a garlic-flavoured carrot dip and rice with currants, pine nuts and spices; and
• The whole third round was after eating a dark chocolate and coconut cream dessert.

That'll do for the experimental procedure. As with yesterday, let me know about any issues that I haven't already mentioned.

More to come later today or tomorrow regarding the results and conclusions.