The Noise in the Glass: Why Wine Scores at Our Club Are Messier Than You Think
Wine scoring looks simple. You taste, you feel, you write a number. But decades of research in wine economics tell a more uncomfortable story — one that is worth sitting with honestly, especially if you are the kind of club that takes its scoring seriously.
The 1976 Judgment of Paris — the blind tasting that famously upended the wine world by ranking California wines above French classics — has been picked apart by researchers ever since. What they found was not just a surprising result but a troubling one: the judges disagreed substantially, and some were far more consistent than others. Research published in the Journal of Wine Economics showed that consistent and inconsistent tasters at the Paris tasting produced quite different results, and that findings based on the full panel masked those individual differences entirely.
The problem runs deeper than individual quirks. Tasters do not use the same scale — some are generous, some are stingy — and that divergence in baseline alone can swamp any signal about the wine's actual quality. Two people tasting the same wine may be playing completely different games. Expectations compound the noise further. Price information, prior ratings, a confident neighbour — all of these shift scores, and experienced tasters are not reliably less susceptible than novices.
Now bring this into our setting. We are enthusiastic, curious, and honest. We are not trained tasters. The deviations that researchers document in professional panels — with controlled conditions, calibrated judges, and structured protocols — are almost certainly larger around our table. A high score from one of us and a low score from another may reflect personal preference, mood, familiarity with the style, or simply a different sense of what “85 points” means. I include myself in all of this.
None of this means our scores are worthless. Averaged across enough tasters, noise tends to cancel out, and a genuine consensus can emerge. Research on the Paris tasting found that simply averaging the scores of two or more judges significantly improved the quality of judgments. The wisdom of the crowd is real — but only if we understand what the crowd is actually measuring.
The honest takeaway: treat our collective scores as a rough compass, not a precise instrument. The wine that finishes first in our ranking probably earned it. The difference between second and fourth place is likely noise.
We Taste Blind — and That Is a Genuine Strength
Our club tastes blind. Labels are hidden, bottles are wrapped, and members rank what is in the glass rather than what is on the label. This is not a trivial commitment. Most social wine drinking is saturated with label information — the producer, the vintage, the price, the recommendation — all of which shape perception before the wine reaches the mouth. Removing that information is a genuine methodological achievement, and it is rarer than it should be.
But blind tasting removes the label from the glass, not the taster from the room. The biases documented above — scale divergence, expectation effects, inconsistency — do not require label information to operate. They are features of human perception and judgment, and they travel with us to every tasting regardless of what we can see on the bottle.
Our Method: A Useful Shortcut With Real Weaknesses
Our ranking method is a forced-choice ordinal rank-sum. Each member assigns every wine a rank from 1 (best) to 12 (worst), with no ties permitted. The ranks are then summed across all members, and the wine with the lowest total wins.
This method has real virtues. It is simple, fast, and produces a clear result without requiring members to agree on what a numerical score means. It sidesteps the scale divergence problem by asking for relative judgments rather than absolute ones.
But it carries five weaknesses that are worth being honest about.
First, it discards magnitude. A member who thinks wine A is dramatically better than wine B and a member who thinks they are nearly identical will both record a rank of 1 and 2. The method cannot tell the difference.
Second, it forces discrimination where none may exist. Requiring a strict ordering of twelve wines means that even wines a member cannot meaningfully distinguish must be separated. The ranking looks precise. It may not be.
Third, it produces no measure of agreement. The final scores tell us which wine won, but not whether the group agreed or whether the result was driven by one or two strong opinions pulling in the same direction.
Fourth, ranking twelve wines in a single sitting is a significant cognitive task. Palate fatigue is real, and the reliability of late rankings — wines tasted ninth, tenth, eleventh — is almost certainly lower than the reliability of early ones.
Fifth, and most quietly: a rank of 1 from one member and a rank of 1 from another are treated as equivalent. But if one member is a generous ranker and another is severe — if their internal scales are calibrated differently — then summing those ranks aggregates not judgments but private reference points.
The Host’s Hidden Hand
Sequence matters more than we tend to admit. The order in which wines are tasted shapes how they are perceived, independently of what is in the glass. A lighter wine tasted after a very tannic one will seem fresher than it is. A wine with residual sweetness tasted after a bone-dry one will seem richer. These are not failures of attention — they are features of how palates work.
In a professional tasting, the sequence is carefully managed. In our club, it is managed by the convener, who may or may not be thinking about it. Blind tasting does not reduce this problem. If anything, it amplifies it: without label information to anchor perception, tasters rely more heavily on contrast effects from the wines around the one they are currently tasting.
The implication is uncomfortable. The wine that finishes in the middle of our rankings may have finished there partly because of where it sat in the sequence, not because of what was in the glass.
When the Wines Are Too Similar to Tell Apart
Some of our tastings are horizontal: same grape variety, same vintage, same appellation. The intention is rigour — by holding variables constant, we isolate the differences that matter. But horizontal tastings also push the wines closer together in profile, sometimes below the threshold of reliable discrimination.
Research on expert tasters is unsparing here. In one well-known study, the same wines were presented to expert judges on multiple occasions, and their scores were found to be distributed essentially at random — no more consistent than chance would predict. If trained professionals cannot reliably distinguish wines in a controlled setting, the implications for our club tastings are worth sitting with.
This does not mean horizontal tastings are pointless. A well-defined topic — a single appellation, a single producer across vintages — narrows the criteria gap and gives the evening genuine focus. The conversation tends to be richer when everyone is trying to answer the same question. But the ranking that emerges from a flight of very similar wines should be held lightly.
What Are We Actually Measuring?
Underlying all of this is a question that wine researchers call construct validity: what, exactly, is wine quality? Is it complexity? Balance? Typicity? Longevity? Pleasure? These are not the same thing, and different tasters weigh them differently — often without realising it.
Professional competitions address this, imperfectly, through briefing hierarchies and calibration wines. Judges are told what they are evaluating before they begin, and a reference wine is used to anchor the scale. Our club has none of this. Each member brings their own implicit criteria to the table, which are never made explicit.
The result is that each rank in our system is a private judgment against a private standard. Summing those ranks does not aggregate our judgments — it aggregates twelve separate private conversations, each conducted in a slightly different language.
Research bears this out in an interesting way: expert tasters tend to agree more on wines at the low end of quality than on wines at the high end. Obvious flaws produce consensus. Genuine excellence produces disagreement, because excellence can be achieved in more than one way, and different tasters value different paths to it.
The Conversation That Numbers Cannot Have
The most valuable part of our evening is not the ranking. It is what happens after the scores are in—the conversation the numbers cannot have on their own.
When a wine divides the room, that division is information. When someone names a quality that others had not noticed, that observation changes what everyone else tastes in the next sip. When the producer or region is revealed and the room recalibrates — “of course, that explains the acidity” — something genuinely educational has happened.
But this part of the evening has its own hazards. Dominant voices can pull the room toward a consensus that does not reflect individual experience. Quieter members may not surface their most interesting observations. The discussion can stay at the level of tasting notes — “I got blackcurrant, you got tobacco” — without ever reaching the more interesting question of why the wine is or is not working.
The qualitative discussion is where the real learning happens. It is also where the most can go wrong.
Most Clubs Do Far Less
It is worth pausing here to note that most wine clubs do not do any of this. They pour, they drink, they chat. No blind tasting, no ranking, no attempt to surface disagreement. We have tried harder than that — and I mean that genuinely. By trying harder, we have exposed problems that a more casual approach would never have encountered. That feels like the right kind of trouble to be in.
That is not a reason to abandon the method. It is a reason to be honest about what the method can and cannot tell us. Any scoring system carries these burdens. The question is whether we knowingly carry them.
The Convener
But there is one person in the room for whom preparation is not optional: the convener. Just as a university tutorial rises or falls on the quality of its tutor, a wine tasting rises or falls on the convener. The parallel is exact — and so is the failure mode. The bad tutor is not ignorant. They are often knowledgeable enough to divert every question without ever resolving it: circling, reframing, filling the hour with the appearance of inquiry while the actual problem goes untouched. The bad convener does the same. They talk fluently about what they are sensing, steer the room away from difficult comparisons, and leave everyone feeling vaguely entertained but no clearer about what was in the glass. Knowledge without resolution is its own kind of bore.
The good convener is something else entirely. They have done the reading. They know the producer, the vintage, the appellation. They have thought about the sequence. They have a question they want the evening to answer. In the right hands, a wine tasting is not a social occasion with scoring attached — it is a genuine inquiry, and the scores are just one tool.
Which brings us, finally, to the only questions that truly matter: was the evening enlightening? Do you feel good about it? Did something shift — a new wine discovered, a prejudice gently overturned, a conversation that lingered longer than the glass? If the answer is yes, then the noise in the scores is beside the point. The imperfect method, the undefined criteria, the palate fatigue, the shy member who said nothing — none of it negates an evening well spent among people who care about what is in the glass. That, in the end, is what a wine club is for.
If any of the problems above felt uncomfortably familiar, the companion piece — Seven Things We Could Actually Do Differently — offers some practical responses. None of them requires a new method. Just a little more intention.
I hunt down value wines with soul, the kind that tell stories, spark debates, and challenge your assumptions about what good and "affordable" can taste like. The deal: Hong Kong prices with free delivery in urban areas. Six-bottle minimum. Ready to explore? WhatsApp 852 66236746 or email cf.lau@dunndunn.hk - Kevin K Tang
(AI photo)
The 2010 and 2006 Château Saint-Pierre are classic St. Juliens that punch well above their weight. Despite their age, both wines are built for longevity — structured, concentrated, and with plenty still to give.
The 2010 opens with a woody edge that quickly gives way to dark fruit — currant and blackberry — underpinned by crisp acidity. The 65% Cabernet Sauvignon lends firm structure and intensity, with concentrated black fruit filling the palate. Full-bodied at 14.5% ABV, it is generous and well-balanced with a dense core, though the finish is slightly short by cru standards.
The 2006 is a solid rather than spectacular Saint-Julien — one of the smaller productions in the appellation. At 13% ABV it is medium-bodied, and the nose offers roasted herbs, coal, graphite, and crème de cassis, though it can feel somewhat closed and unyielding. There is power here, but also a certain rigidity that may not fully resolve for some years yet.
The 2010 offers exceptional value: potent, tannic, and texturally inviting. Critics have scored it highly across the board — RP 97, Jeb Dunnuck 97, James Suckling 93, Wine Spectator 91.
On the palate, the 2006 remains somewhat austere — the tannins are firm and the structure unyielding, with the fruit yet to fully emerge from behind the wine’s rigid frame. The finish is decent but not generous. Patience is required, and even then it may never fully shed its angular character. Less compelling than the 2010 at this stage. Critics scored it RP 93, Wine Spectator 90.
Saint-Pierre will not suit every palate. It is a big, dense wine — more about power and structure than finesse — and its relative anonymity means it rarely commands the attention it deserves. But for the drinker who prizes intensity, density, and a formidable tannic backbone, this is the address regardless of vintage. The 2010 delivers this in spades; the 2006, while more austere, remains a serious and fairly priced option for the patient drinker.
Wine Searcher average retail prices (ex-tax, approx. April 2026):
2010 Château Saint-Pierre — HK$848 per bottle
2006 Château Saint-Pierre — HK$622 per bottle
Prices sourced from Wine Searcher, converted from USD at the prevailing HKD peg rate. Retailer pricing varies and is subject to change. Please contact us for our current pricing.
I am, first and foremost, a wine person. My world is built on things I can see, smell, taste, and reason about — terroir, vintage variation, the slow logic of a wine finding its moment. The supernatural has no place in that world. And yet, over the past twenty years, I have had a series of encounters I cannot explain, all circling the same person.
His name is Francis, and he too loves wine — perhaps the most ordinary thing about him. Every one of these encounters involves him, and I often wonder what it is about Francis, and what may have become of him.
The Beijing Encounter
The earliest incident occurred in Beijing, where Francis was working as a manager for a Canadian company. We stayed at the Old Beijing Hotel (then called Beijing Fan Dian), which he used as a room and office—a grand establishment that still exists. We had spent the evening over a bottle of red, the kind of wine-warmed night that makes the world feel entirely rational. I went to bed relaxed, comfortable, without a single thought of anything unusual. Nothing could have prepared me for what happened next.
Then, in the dead of night, I felt it. A weight — enormous, suffocating — pressing down on my entire body. The shape of a woman, large and heavy. I could feel her mass and yet see nothing in the darkness. I tried to move. I could not. I tried to push her off. Nothing. I could barely breathe. In desperation, I cried out: “Francis! There is something on me!”
He switched on the lights and crossed the room. The moment he did, it was gone. Instantly. Completely. As if it had never been there at all.
I lay there, heart pounding, staring at the ceiling. Nothing to see. Nothing to point to. I am a rational person — I deal in things I can taste, touch, and reason about. But in that moment, every certainty dissolved. I had felt her weight. Her presence. And I had been utterly helpless.
That was my first encounter with something I could not explain — the first in my life. Decades later, it remains as vivid as the night it happened.
Queen’s Road West
Another incident occurred in Hong Kong. We had shared a quiet dinner earlier — a glass each, nothing more, the kind of drinking that leaves the mind clear and the senses intact. Francis was driving along Queen’s Road West at under 50 kilometres per hour when, approaching a turn before the junction with Lu Koo Road, he suddenly veered and struck the left side of the road. He had never had an accident before. When I asked why, he told me he had seen someone standing in the road. His hands were shaking, knuckles pale, eyes still fixed on the windshield as if the figure might reappear. We ended up at Queen Mary Hospital for a couple of stitches.
Manhattan Heights
After retiring early, Francis bought a flat in Manhattan Heights, near Mei Foo. The view was lovely, overlooking the water. That evening we had opened a decent bottle together, something white and crisp — the kind of wine that belongs to calm, ordinary nights.
One cold night, all the windows were shut tight. There was no way any wind could have slipped through — and even if it had, it would not have been strong enough to do what happened next. I felt a cool breath of air brush the back of my neck, and something gently stroke my hair.
I said nothing to Francis. He was still living there at the time, and I did not want to alarm him. But I knew what I had felt.
Francis eventually settled in Shenzhen for good. I never got the chance to discuss any of these encounters with him. The questions remain — what draws the supernatural to him, what he may have seen or felt and never shared. I wonder whether the answers would bring comfort, or only deepen the mystery.
A Personal Footnote
I should add that I have had one other experience worth noting — though it belongs to an entirely different category. When my father passed away at Prince of Wales Hospital, the doctors asked me to push his trolley to the mortuary as a final tribute. I did.
Along the corridor to the mortuary, there were three turns. Each time I rounded a corner, I felt faint. The first time, I told myself it was the effort — the weight of the trolley, the weight of grief. But it happened at the second corner, and again at the third.
I have spent many years learning to trust what is in the glass — its colour, its weight, its honesty. A wine is either what it claims to be or it is not. The encounters with Francis were nothing like that. They offered no such clarity, no finish I could name. And perhaps that is what haunts me most: a man who loved wine as I do, standing at the centre of things no glass of reason could ever explain.
