Subject: Digest for the period 5/22/2006 - 5/23/2006 Date: Tue, 23 May 2006 01:03:56 -0400 Table of contents ---------------------------------------------------------------------- 1. RE: Accounting for all 50 points (Bill Pierce) 2. RE: Accounting for all 50 points (Jamil Zainasheff) 3. Accounting for points (Jon Tobey) 4. RE: Accounting for Points (Michael Schwartz) 5. Accounting for Points and Quality of Judging (Stephen Johnson) ---------------------------------------------------------------------- From: Bill Pierce Date: Mon, 22 May 2006 11:25:12 -0400 Subject: RE: Accounting for all 50 points John Peed's point about accounting for all of a beer's potential 50 points and providing the brewer with constructive suggestions for improvements is a valid one. I agree that he was not given very much substantive information in the judging of the beer he mentioned. However, I also feel compelled to mention the time constraints and demands on judges during the judging process. The last three flights I have judged in recent competitions contained 15 or more beers, which we were expected to complete in a little more than three hours. That averages only about 12 minutes per beer. It's very difficult to provide as complete feedback and as many suggestions for improvements as one would like under that kind of time pressure. I find myself able to hit the highlights and point out the more obvious strengths, weaknesses and improvements that I note, but not to provide a long detailed evaluation during the brief time I have to experience the beer and fill out the scoresheet before I am required to move on to the next entry. I strive not to short-change the brewer, but there is only so much I can do in a restricted period of time. Bill Pierce Cellar Door Homebrewery Burlington, Ontario ---------------------------------------------------------------------- From: Jamil Zainasheff Date: Mon, 22 May 2006 08:51:46 -0700 Subject: RE: Accounting for all 50 points > The last three flights I have judged in recent competitions > contained 15 or more beers, which we were expected to > complete in a little more than three hours. What competitions are setting up judges to do flights of 15? Those flights should be split somehow either by breaking up subcategories, doing preliminary judging, or doing a mini-BOS as the NHC does. 15 in one flight seems ridiculous to me. JZ ---------------------------------------------------------------------- From: Jon Tobey Date: Mon, 22 May 2006 08:58:53 -0700 (PDT) Subject: Accounting for points I ask this question annually, but nobody ever picks it up and answers it, so I'll post it again: As far as scoring a beer >in competition< I think that our goals are very unclear. Are we trying to pick the best beer, OR are we trying to pick the best beer AND help people make better beer. Just once I would like a distinct answer to this question. If we don't know the goal in evaluating the beers, no wonder we don't please our customers! The answer leads to two very different ways of scoring. If we are just trying to pick the "best" beer, I believe that scoring can be (but not necessarily should be) much more subjective. This is becuase if we are helping people trying to make better beer we spend a lot more time defining and discussing the "flaws" (flaws are afterall only in the context of style, one man's brettanomyces infection is another man's Belgian), and thus are much more concrete. If we are truly trying to help brewers, I also have a hard time with points deduced that are not directly linked to flaws. I can't tell you how many times I've seen a beautiful beer score 1/3 for appearance to make a bottom's up score sheet come out to the score the judge has arbitrarily assigned to the beer. (It's arbitrary if you can't tell the brewer exactly how you got to that number, wouldn't you say?) Among other things, in my experience, top down scoring leads to a more linear scoring. Bottom up score sheets tend to get very nonlinear somewhere between 35 and 40 where each extra point requires a certain "je ne sais quoi" from the judges. Even in this thread the comment was made to the effect "if it's a 40, score it a 40." Folks, that's only an 80% beer and we make it out like that is some rarified realm. I would say for a beer to get 20% of the score deducted you better be able to point to some sort of technical or stylistic flaw and not just "um, it's intangible." We have this weird thing about scoring beers well. A lot of people make very good beer. In fact, I would argue that a well-made homebrew can/should surpass commercial microbrews. Isn't that why we brew? Again, this is much more germane if the goal is to help people make better beer. If we're just waving magic wands and picking the "best" beers based on our superior spiritual intuition and personal tastes, it's completely different and we can dispense with a lot of hard questions! At the point, the rules are unclear and therefore the results are likewise unclear. I would like to see an official statement regarding the whole reason we are doing this, so that everybody is operating from the same frame of reference - judges and competitors. Jon Tobey Ideastream 425-822-8351 "It's like one of those craziass Australian wooden Frisbees." My Name is Earl ---------------------------------------------------------------------- From: Michael Schwartz Date: Mon, 22 May 2006 12:48:47 -0500 Subject: RE: Accounting for Points John has a valid point. The comments he received were appropriate and good but they should have some given indication of why the beer wasn't "perfect" or what needed to be adjusted to improve it. Any beer scoring below 40 should have some flaws or missing style components that can be described and suggestions for improvement made, no matter how minor. Often, this is summed up in the overall impression section. > From: Jamil Zainasheff Date: Sat, 20 May 2006 12:48:34 -0700 > Another task I thought would be good for all judges is to write up a > score sheet on an imaginary 50 point beer. You'd have to describe every > nuance > that made it perfect. Now compare that to the beers in competition. I > think it might make people more flexible in their scoring range. Would you really gain anything that's not in the style description, which is in essence the perfect score sheet? The BJCP style description will also help the contestant understand or evaluate the feedback. What did the judge say about each section compared to what the style description says? Were there required elements in the style description that were lacking? As judges we're constantly told to be positive in our feedback. This means we spend time writing what we thought was right about the beer more than what was wrong, sometimes to a fault. > From: George de Piro Date: Sun, 21 May 2006 09:45:47 -0400 > If the beer is really good, give it a score over 40; yes, it's OK to give > good beers high scores! If the beer is not really good, indicate what =20 > the problems are. If you don't perceive any defects, then you shouldn't > be > randomly deducting points. When scoring, I always refer back to the scale included on the judging form. I was taught to refer to that when I was trained how to judge. It can be somewhat subjective but it clearly shows the range the beer should be scored in based on how well it hit the style and is free from flaws. This guide is more critical regarding off-flavors or aromas than most judges I've worked with. Again - the flaws that reduce the score should be identified preferably in the section where the aspect is discussed. Note that based on this, to score over 37 the beer cannot have any flaws but only require minor adjustment of the recipe or procedure. Despite having judged many beers that had major flaws that dominated, the only time I've scored a beer in the Problematic category was when the off-flavor dominated so completely that the character of the beer couldn't be evaluated. On the converse, I've only scored one or two beers ever over 44. World class examples are few and far between. Most of the judges I've worked with over the last 4 or 5 years have scored similarly. Outstanding (45 - 50): World-class example of style Excellent (38 - 44): Exemplifies style well, requires minor fine-tuning Very Good (30 - 37): Generally within style parameters, some minor flaws Good (21 - 29): Misses the mark on style and/or minor flaws Fair (14 - 20): Off flavors, aromas or major style deficiencies Problematic (0 - 13): Major off flavors and aromas dominate And Stylistic Accuracy Classic Example - Not to Style Technical Merit Flawless - Significant Flaws Intangibles Wonderful - Lifeless This provides an idea of how much the perceived flaws or missing of the style affected the score as well as that thing that's almost impossible to describe "intangibles" affected the score. A beer can be within the style guidelines on all major points and be lifeless. That extra something makes a huge difference between a very good beer and an outstanding one, especially in the more esoteric styles. As David Houseman stated, there are time and space constraints that may make it difficult or impossible to completely convey all of your thoughts about the beer. In most competitions the judges realistically have about 7 - 8 minutes per beer. I always include the email address so the contestant can contact me if they would like further feedback. I have yet to be contacted even though I know some of my descriptions aren't complete and in a couple cases of really troubled beers I said feel free to contact me to discuss it. Perhaps some would disagree with me on points I've made. If nothing else, the BJCP program should be about uniformity in scoring and quality of feedback. -- Mike Schwartz Beer Barons of Milwaukee beerbarons.org ---------------------------------------------------------------------- From: Stephen Johnson Date: Mon, 22 May 2006 20:36:53 -0500 Subject: Accounting for Points and Quality of Judging Thanks to John Peed and others for bringing up a good topic for discussion here. John knows that his fellow club member and BJCP National level judge Tom Karnowski and I have struggled for years here in Tennessee, along with some good folks in Memphis and other clubs in the Southeast, to bring up the quality of judging in our region by trying to encourage the better judges to travel to our competitions, conducting ongoing training in sensory evaluation and BJCP oriented training with our local clubs, and trying to give recognition to good judges who fill out good score sheets at competitions. But it is an uphill battle, and more often than not as competition organizers and judging coordinators, we are left with the hand we are dealt with when it comes to hosting a competition, and sometimes that hand only has a couple of aces and face cards to begin with. We try to pair up our better judges (most of the time BJCP) with our more novice, yet willing to learn but probably inexperienced judges. There is only so much an experienced judge can do to train and educate a novice judge during the course of a 3 hour judging session when you are asked to evaluate a flight of 12 IPA's during that time. At a good pace, that is 15 min. per beer. At the very least, I can give good written feedback independently during that 15 minute time frame under the best of judging conditions. If I have to do some training at the table with a novice judge that might include any of the following: educating about a style he/she is unfamiliar with, encourage positive comments, corral some misinformation, or inform about the source of some perceived defects -- then my time extends more towards 20 or more minutes per beer, and that throws off the entire flight getting done in a reasonable time frame. As an organizer, flights that take a much longer time than allotted can really gum up the works in keeping to a schedule that might be time sensitive. I hope many of you get the gist of my thoughts here. The fact is that we are dealing with an ideal application of recognized judges giving good, quality judging to a hobbyist who has paid a small fee to have his beer evaluated in an impartial and professional manner. The facts in the real world are that we typically need every available person at most of our homebrew competitions to provide some impartial and hopefully helpful feedback to each entrant. Some of those judges may not give the best comments. But, I would hesitate to have to be the "hard-ass" organizer to say to some judges who show up, "Hey, Joe, you wrote some pretty meager score sheets at our competition last year, and to be honest, I don't think I can let you judge today even though you drove down to Nashville from Louisville, where you left in the dark at 5:00 am to come here. Your comments weren't very helpful to those brewers, and we only want the best judges here today." I think the annual MCAB event may be one of the few that is able to claim they have only the best judges evaluating beers entered into that competition. At some point we've got to come up with a system to help educate and give feedback to our judges about the quality of their judging. Here's an idea: Maybe each organizer could submit copies of their score sheets to the BJCP for judge comment evaluations, and those who have written better score sheets get additional experience points added to their points for a given competition. Rather than punishing the mediocre, let's reward the exceptional. Just my 10 cents worth. Thanks for the bandwidth. Steve Johnson BJCP Certified Judge Music City Brewers Nashville, TN ---------------------------------------------------------------------- ***** Important Subscriber Information ***** To post a message to JudgeNet, send it to judge`at`synchro.com. Send plain text only, no HTML, MIME, encoded text or attachments. Make sure you use a meaningful subject. Quote only as much material as is needed for context. To manage your subscription, go to http://synchro.com/judge/subscriptions.html or send an email to judge-request`at`synchro.com with the subject: help judge. JudgeNet is also available as an NNTP newsgroup, go to news://news.synchro.com/synchro.judge