Subject: Digest for the period 5/29/2006 - 5/30/2006 Date: Tue, 30 May 2006 01:04:43 -0400 Table of contents ---------------------------------------------------------------------- 1. RE: Yet more... (John C. Tull) 2. Subjectiveness in judging (Michael Schwartz) 3. RE: Subjectiveness in judging (Jamil Zainasheff) ---------------------------------------------------------------------- From: John C. Tull Date: Sun, 28 May 2006 13:09:52 -0700 Subject: RE: Yet more... There has been a push to avoid using straight commercial beers in exams because it does not generally provide an interesting context for the examinees to point out issues, flaws, etc. (not that there are not significantly flawed commercial beers out there). Your exam administrator did the right thing and created a beer rather than simply pouring one straight from the bottle, probably because a homebrewed example was not readily available and because they were seeking broad feedback on a beer that was obviously not brewed to style. This kind of beer concocting can be tricky as well because you may end up with something completely out in left field, like the famous "umami" beer that was created from an unusual dosing of a beer with worcestershire sauce, not a recommended flavor adjusting protocol in the examination handbook. I believe one of the proctors pulled the umami descriptor out of his war chest of terms, and it has been an interesting joke ever since. Since then, a more strict set of guidelines has been discussed by the examination review committee for flavor alteration, and I believe these guidelines are now provided to exam admins prior to the event so they can avoid overdosing beers. John On May 27, 2006, at 10:04 PM, JudgeNet - the beer judge digest wrote: > John mentioned in passing in his last post: > > Snip.... > The graders are likely not to make any deductions for your guess as > exam beers > are often commercial beers. > ....snip > > The last time I wrote an exam, the proctor blended various bottles > of homebrew > and then had us all judge the result. For instance, taking a light > coloured > sample and adding a bit of a dark beer to it then calling it an > American Brown. > That way, no one really knows what to expect for a sample - > proctors included - > and the sample is closer to what you might expect in a local > competition > compared to judging a commercial example of a style. > > I have also done that with calibration beers before a judging session. > > Denis Barsalo ---------------------------------------------------------------------- From: Michael Schwartz Date: Mon, 29 May 2006 09:34:42 -0500 Subject: Subjectiveness in judging > From: Bev D. Blackwood II Date: Thu, 25 May 2006 08:25:58 -0500 > Subject: Objective Beer Scoring... Not Possible? > > I've also seen the exact same beer (not mine this time and an American =20 > Pale Ale, no less) win Best of Show in a Chicago competition and get > shut out in a Dallas competition on the SAME DAY! So it's VERY > subjective. This is the most common complaint I've heard about homebrew competitions =20 and I've heard it from many, many people who have entered competitions. =20 It's not whether they win or not, because obviously that's based on the other beers that are entered, but the wide disparity in scoring and comments. I had an IPA that scored around 40 at one competition and within two weeks was scored in the mid-twenties in another competition. I've heard many other people relate similar experiences where there was a wide disparity in the scoring of the beer within weeks of each other. Some of these comments coming from very accomplished brewers. One of the judges =20 in our club got into the BJCP program specifically because he perceived this as a problem. Every person that I've heard from who mentioned disparity in scoring for a given beer in a short time frame has also said they believe this is a huge problem and perceives it as a lack of quality in the program or judges. While my experience doesn't make or replace an objective or statistical study of the issue, it does indicate that this subjectivity in judging is perceived as a problem by the people entering homebrew competitions. Perceptions rule the day whether justified or not. You may say settle down, take it easy, it's only beer. Some people might want to punch you =20 in the nose for saying that about their beer. ;) Shouldn't judging and the BJCP program have as part of it's goal, uniformity in scoring as well as high-quality comments about perceptions =20 and what could be improved? It was my understanding that was a prime goal. At every competition I've judged in, the judges try to reconcile their scores to some degree if there is more than about a 5 point disparity. Ideally, this isn't required and in my experience happens on =20 about one or two beers out of a flight. You would think that this level =20 of agreement would extend across competitions and realistically it should. Otherwise, perhaps a disclaimer should be added to the judging forms stating that it's purely subjective and your score depends on which side of the bed the judge rolled out on that morning. While I say that with my tongue firmly planted in my cheek, there is something to it if the participants expect their beers to be judged objectively when in truth they can't be. Meeting expectations is 98% of =20 the battle. If participants in a homebrew competition understand that there is much more subjectivity to the scoring than is indicated by the very formal point system that's used, it would go a long way toward eliminating the perception that this is a problem. -- Mike Schwartz Beer Barons of Milwaukee beerbarons.org ---------------------------------------------------------------------- From: Jamil Zainasheff Date: Mon, 29 May 2006 11:11:14 -0700 Subject: RE: Subjectiveness in judging > > I've also seen the exact same beer (not mine this time and > an American > > Pale Ale, no less) win Best of Show in a Chicago > competition and get > > shut out in a Dallas competition on the SAME DAY! So it's VERY > > subjective. I've been tracking data on entries since my very first competition. With a fairly large data set (albeit, only one brewer) I have a scoring differential from 0 to 25.5 points. The 25.5 points (47 versus 21.5) was a Sweet Stout. A Doppelbock which scored 48.5 in the first round of the NHC this year has also scored 24.5. The average differential has been 12 points over all entries. It appears to me, that the beers most likely to show a large differential are also the ones that are most common to homebrew. The APAs, IPAs, Stouts, Porters, etc. I'm thinking at this point this is where regional difference really comes into play. Or possibly, it is that the given beer is or is not like the version that the judge makes. As some of my BJCP buddies will tell you, I've been fascinated with the whole regional difference idea. I see this as having more impact on a beer than shipping stress. I've been trying to map the whole regional difference thing, but it wasn't until recently that I realized the large difference between "common styles" and other styles in a region. I think that this is the key to figuring this out. In any case, I believe the real value of competition and score sheets is not in winning, but in getting back a good sheet that details what the judge perceived. If they can give me a good analysis of what they found and how it matches their ideal of the style, then I've got something to work with to improve my beer. I also believe an entrant cannot question what it was the judge perceived. The entrant was not there, does not have that same bottle of beer in hand, hasn't had the other beers in the flight, same amount of sleep, didn't drive the same route to the judging, and isn't breathing the same air. There are just too many tiny influences on perception and too few judges that know how to overcome those factors. JZ ---------------------------------------------------------------------- ***** Important Subscriber Information ***** To post a message to JudgeNet, send it to judge`at`synchro.com. Send plain text only, no HTML, MIME, encoded text or attachments. Make sure you use a meaningful subject. Quote only as much material as is needed for context. To manage your subscription, go to http://synchro.com/judge/subscriptions.html or send an email to judge-request`at`synchro.com with the subject: help judge. JudgeNet is also available as an NNTP newsgroup, go to news://news.synchro.com/synchro.judge