Subject: Digest for the period 5/31/2006 - 6/1/2006 Date: Thu, 01 Jun 2006 01:00:02 -0400 Table of contents ---------------------------------------------------------------------- 1. Local BJCP Reps & Proctor Scoring (Mike Winnie) 2. RE: Subjectiveness in judging (Martin Brungard) 3. Re: disparity (Dion Hollenbeck) 4. RE: Disparity, Proctor scoring, Score sheet epiphany (Houseman, David L) 5. Various John Tobey responses (Al Boyce) 6. Re: Proctor Scoring (Steve Piatz) 7. RE: Subjectiveness in judging (Jamil Zainasheff) 8. RE: disparity (Jamil Zainasheff) 9. Re: Epiphany (Jon Tobey) 10. Plea for Judges B3 Comp (Chris Love) 11. So accounting isn't the problem (Peed, John) ---------------------------------------------------------------------- From: Mike Winnie Date: Wed, 31 May 2006 07:51:33 -0400 Subject: Local BJCP Reps & Proctor Scoring Jon Tobey writes: =20 "When I was having a problem trying to figure out why I couldn't get support from my "local" rep I found out he's in AZ or something equally ludicrous (I'm in Seattle). Puh-lease. I might as well write to the Pope." You can always run for the rep position yourself next time it opens up. Jon Tobey writes: =20 "The only time I proctored an exam, myself and the other judge (an excellent judge and brewer) were 20 Pts. out on EVERY beer. Unlike judges, proctors don't get to rectify their scores. " Currently (for the last year at least), while proctors are required to fill out their own scoresheets in private, they get to come up with a concensus score afterwards. The concesus scores are used by the graders. =20 =20 Mike Winnie BJCP National & Grader ---------------------------------------------------------------------- From: Martin Brungard Date: Wed, 31 May 2006 04:03:23 -0800 Subject: RE: Subjectiveness in judging Jamil pointed out that judging scores do vary quite a bit at times. I contend that its not all judging that is at fault here. Having judged at the Nationals several times, I can say that there is more than judging at fault. A particular case in point was when I judged Bavarian Weizens at the Nationals. Since this was the second round, I knew that these should have all been fairly good beers in order to advance. I was thoroughly disappointed in the samples that were presented. Most were scoring in the 20's, characterless and uninspiring. To have made it to this round, I would have expected that any of these beers would have scored in at least the 30's. That fact that the style is delicate and does not age well helped point out the problem to me. The problem is that transportation and cellaring can have a substantial affect on the vitality of the beer. The score is affected accordingly. So, discrepancy in scoring may not be due to judging personnel alone. The handling and treatment of the bottles between the brewer and judge could also be to blame. This brings another point that contest masters and cellar masters should be aware of, treat the samples as nicely as possible. I know that most contests don't have refrigeration available for storing a large number of entries for weeks. The samples end up unrefrigerated for a while. Do what you can to avoid that. Poor cellaring is going to affect the scores. Don't blame all of the resulting score on the judges, they are just calling them as they taste them. Martin Brungard Tallahassee, FL ---------------------------------------------------------------------- From: Dion Hollenbeck Date: Wed, 31 May 2006 06:50:37 -0600 Subject: Re: disparity Actually, for me, the Overall Impression section is where I am most likely to put comments like, "Wonderful beer, too bad a stout is not golden colored." In So. Cal. we so rarely get stinkers, that most of my comments in this section address issues that make it out of style. This allows me to 1) give the brewer a pat on the back for what was done nicely, refer to shortcomings which I deducted points for above, and to refer to the intangibles that don't go anywhere else per se, like "nice complexity", or "tone down the hop bitterness a bit", or give feedback on how to solve any problems. The little "Stylistic Accuracy", etc. boxes give me a chance to show the brewer where on each scale the beer falls. A very lot of information can be conveyed with just those checkboxes, as long as the descriptions in the other sections is complete and accurate. For example, with a stout that is golden in color, I can state in Color section that it is golden, and I can mark the box to show that it is sytlistically poor, while at the same time marking a box that says that the intangibles are wonderful. Will not need a lot of additional writing to convey these. regards, dion -- Dion Hollenbeck Email: hollen`at`woodsprite.com Home Page: http://www.woodsprite.com Brewing Page: http://hbd.org/hollen Toys: 98 4Runner, 86 4x4 PU ---------------------------------------------------------------------- From: Houseman, David L Date: Wed, 31 May 2006 08:59:37 -0400 Subject: RE: Disparity, Proctor scoring, Score sheet epiphany Jon, You'll be glad to know that the proctor scoring methodology has changed since you proctored. While individual proctors scores remain as scored, there is a separate "consensus scores" section which the proctors create to arrive at a consensus to which the examinees are measured. Surly Jon you didn't mean that you had entered your homebrew in THE GABF (Great American Beer Festival) since that is not a homebrew competition but a professional competition? Must be some other competition using that name? But that does remind me that at THE GABF we do not judge like the AHA/BJCP judging. Every round is similar to a BOS round. However on the first round we do fill out some tasting sheets which do provide some feedback to the brewers. However this feedback is only what we sense, not how to correct a problem. These are professionals who can figure that you for themselves. We do not score the beers. We just select the top three beers from each flight to more forward. The expertise present at these judging sessions is excellent. A wealth of knowledge. But scoring isn't used and the feedback is more on perception and comments on style. So all this discussion about the judging forms, the process of judging and the scoring and feedback is interesting. It can all be used or not. We could judge beer like wine judges do wine. The AHA/BJCP chose this process because it worked in the homebrew community. Those that established this process have seen their handiwork be very hardy, lasting for many years. It has evolved somewhat, and will continue to do so, but there were some very wise people who set up the judging forms, the style guidelines, the exams and the process of homebrew competitions. We should thank them for their insight. Dave Houseman ---------------------------------------------------------------------- From: Al Boyce Date: Wed, 31 May 2006 08:27:54 -0500 Subject: Various John Tobey responses Jon- Re: "far-away" local judges There are only seven regional BJCP judges in the country - it is very likely that your regional judge is going to be far-away from you. That doesn't mean that they can't help you with your BJCP quandry. Email and telephone are powerful tools, and we know how to use them, Jon - give your regional rep a chance! ;-D I live in Minnesota, and I have helped solve BJCP problems from as far away as Edmonton, Calgary and Alaska! Re: Proctor scoring The proctors DO have to come to a consensus on their beers for the BJCP exam - there is a new "CONSENSUS SCORESHEET" that they fill out - after they have independently scored each beer. It is NOT an "AVERAGED" score, it is a consensus, so the proctors fill it out after talking over and rectifying their scores. It is those CONSENSUS scores to which the examinees must approach to do best on the scoring part of the exam. Re: "...so we should at least tell them they make good beer although we can't reward them for it under the present system." You CAN tell them they make good beer under the current system - that's EXACTLY what the "Overall" category is for - to tell them how you enjoyed their beer, whether or not it was not-to-style. Re: "...I've had Helleses scored too malty and Stouts marked too dark." If you get scoresheets back from Recognized or higher BJCP judges that show an apparent misunderstanding of the beer style being judged, if the judge put his email or phone on the scoresheet (a space is now provided on the sheet for that), please contact them directly and RESPECTFULLY discuss it with them. If the email is not provided, or you choose to remain anonymous, please send copies of the offending scoresheet to the competition organizer and/or to your regional BJCP rep, so that they may contact the judge and set them straight, possibly sugggesting a BJCP course refresher for them. It's the only feedback system for judges we have right now, so use it. - Al Boyce North BJCP Rep -----Original Message----- From: JudgeNet - the beer judge digest [mailto:judge`at`synchro.com] Sent: Tuesday, May 30, 2006 11:02 PM To: Digest Recipients Subject: Digest for the period 5/30/2006 - 5/31/2006 Table of contents ---------------------------------------------------------------------- 1. Local BJCP reps (Jon Tobey) 2. Proctor Scoring (Jon Tobey) 3. Score sheet epiphany (Jon Tobey) 4. disparity (Jon Tobey) 5. RE: Score sheet epiphany (Bob Boufford) 6. Bottom-up/Top-down, Objective/Subjective scoring (Rick Garvin) 7. Help! Need judges for BUZZ Off this Sat! (Christopher Clair) ---------------------------------------------------------------------- From: Steve Piatz Date: Wed, 31 May 2006 08:55:58 -0500 Subject: Re: Proctor Scoring > From: Jon Tobey Date: Mon, 29 May 2006 19:31:59 -0700 (PDT) > Subject: Proctor Scoring > > > The only time I proctored an exam, myself and the other judge (an > excellent judge and brewer) were 20 Pts. out on EVERY beer. Unlike > judges, proctors don't get to rectify their scores. So in the case in > this thread, you would've been screwed, as I assume everybody in that > exam was. Interestingly, we had no problem rectifying scores within > 3 points once we talked it over (in fact like in this post one beer > was a commercial example I recognized and considered an archetypical > beer). At that point I actually suggested on this list that proctors > should have to be at least that close, but the answer was something > to the effect that this artificial spread somehow benefitted the > examinees. How an undefined baseline is helpful escaped me then, so I > certainly can't recall the reasoning now. > These days we require the proctors to NOT reconcile their scores to be within 7 points or to adjust their score sheets after discussing the beers. Rather, the proctors have a spearate form that they use to assign their consensus scores (they provide a consensus for each section of the score sheet). We want all the details the proctors find on the beers and we don't want them to adjust their writting after discussing the beer. The following is a direct quote from the instructions given to the proctors: + BJCP Exam Co-Proctor Instructions + + Your purpose as proctor for the BJCP exam is to be the eyes, + nose, and palate of the exam graders who will be grading the + examinees using your perception of the exam beers. While you + should always strive to generate proper and complete score + sheets it is especially important when you are a proctor for + an exam since the graders will not be able to taste the beer + you and the examinees are writing score sheets for. + + There are only two cases where the BJCP officially sees + score sheets you generate; the first is when you take the + exam and the second is when you volunteer to help proctor a + BJCP exam. In return for judging the exam beers you receive + experience points, in return the BJCP expects a very complete + score sheet. The BJCP reserves the right to withhold some or + all of the experience points if you do not provide an adequate + set of score sheets. + + The procedure for producing score sheets when proctoring + the exam is slightly different than in a normal competition + setting. + · First, you should NOT communicate with the other + proctor(s) in any manner while producing your score sheet. + + · Second, you should NOT attempt to reconcile your + scores when you are finished judging the beer. + + · Third, after you and the other proctor(s) have finished + your score sheets you should generate a consensus score + for the beer - to help the graders you should provide + a consensus for each of the subsections on the score + sheet. The consensus scores go on a separate form. + + You as a proctor are allowed to use the style guidelines while + the examinees are not. Use the style guidelines to make sure + you are describing the beer versus the current guidelines not + versus your perhaps faulty recollection of the style. With + exception of using the style guidelines, you are to judge + the beers with the same information the examinees have - + just the beer style and subcategory names. Try not to make + comments or non-verbal expressions while judging the beer + that would influence the other proctor(s). + + You will notice that the proctor score sheet has even more + space available for your comments than a normal competition + score sheet. You are expected to comment on every minor + aspect of the beer. Use the fine print under each subsection + heading to guide you in your comments. Comment on the + presence or absence of each characteristic appropriate for + the subsection. You need to provide information that will + allow the exam graders to virtually perceive the beer via your + descriptions. For example, you need to mention the presence + or absence of hop aroma and if present try to describe the + type of hop aroma - e.g., spicy, floral, resiny, etc. -- Steve Piatz piatz`at`cray.com BJCP Exam Director, West ---------------------------------------------------------------------- From: Jamil Zainasheff Date: Wed, 31 May 2006 08:05:54 -0700 Subject: RE: Subjectiveness in judging > Don't blame all of the resulting score on the judges, they > are just calling > them as they taste them. I agree. There may be a range of scores, but as I said, each beer at that time and place is a different beer. Don't worry so much about the score. Focus on the perceptions and feedback. Overall, I think judging (by BJCP judges) tends to be of a very high quality and consistency. Sure, there are a couple folks out there that could use some improvement. What group doesn't have a few of those? But on the other side there are a number of folks that are just really stunning in their abilities. In my opinion, the whole BJCP effort has been and continues to be an outstanding success. JZ ---------------------------------------------------------------------- From: Jamil Zainasheff Date: Wed, 31 May 2006 08:09:50 -0700 Subject: RE: disparity > Actually, for me, the Overall Impression section is where I am most > likely to put comments like ... > > The little "Stylistic Accuracy", etc. boxes give me a chance to show > the brewer where on each scale the beer falls... > I'm with Dion on the usefulness of the Overall Impression and checkboxes. The boxes convey a lot of information in a small, quick to fill out space. I'm stunned when judges refuse to fill them out. And the Overall Impression is the place where the feedback goes. It gives you a chance to not only "wrap-up" the comments, but to suggest how the beer could be improved. JZ ---------------------------------------------------------------------- From: Jon Tobey Date: Wed, 31 May 2006 09:58:04 -0700 (PDT) Subject: Re: Epiphany From: Bob Boufford Date: Tue, 30 May 2006 06:41:33 -0600 Subject: RE: Score sheet epiphany Jon, It's about juding beer, not the bottle. I would recommend you visit the recent discussions on the BJCP forums and the AHA TechTalk concerning bottles before continuing to promote the idea of assigning points for the bottle. The better option is to take the Bottle Inspection section completly off the form to reduce the bias that is perceived by many, to occur now in competitions. Cheers, Bob Bob, you missed my points entirely here. Sometimes that happens when I try to make two points in one post. I will clarify: First, I'm trying to find some way to justify the minimum scoring and some of the "it was in the bottle" was, uh, tongue-in-cheek. The artificial minimum score is a silly as the artificial maximum score, but I think we are wedded to it. I would agree with you to take it off the score sheet, but I don't see anything that revolutionary happening in this staid institution. Second, I was trying to find some way to make the current system work by having all of the scores relate to the beer's style and making the point that if you take clipping into account, and the fact that a huge portion of the score is then not to stye points, it's no wonder you have subjective, wide-ranging scores. It's simply impossible to say that we can score beer objectively with that system. I think there has been some agreement of that on this thread, but nobody is really offering solutions, ergo I shared my epiphany. I'll run it by my local rep the next time I run into him at the pub. Jon Tobey Ideastream 425-822-8351 "It's like one of those craziass Australian wooden Frisbees." My Name is Earl ---------------------------------------------------------------------- From: Chris Love Date: Wed, 31 May 2006 10:05:56 -0700 Subject: Plea for Judges B3 Comp Greetings: Mark your calendars for Saturday, June 10, 2006 as The 2nd Annual B3 Forum Homebrew Competition is now calling for judges and stewards. Judging will be at the beautiful E.J. Phair Brewing Company and Alehouse in Concord, California. A light breakfast and lunch will be provided. We have almost 100 entries and very few judges, so please if you can find the time to help us out it would be appreciated. Event details: Saturday, June 10th - 9:30 AM to 3:00 PM. Judges and Stewards should arrive by 9:00 AM so judging can start promptly at 9:30. E.J. Phair Brewing Company and Alehouse http://www.ejphair.com/alehouse.htm 2151 Salvio Street Suite L Concord, CA 94520 If you are interested in judging or stewarding for this event please go to http://www.brewingcompetition.com/2006B3/2006B3.html and register now! Thank you and cheers! Chris Love Judge Coordinator chris`at`brew-monkey.com ---------------------------------------------------------------------- From: Peed, John Date: Wed, 31 May 2006 15:09:37 -0700 Subject: So accounting isn't the problem Gordon - point taken; thanks for the info on how to contact my rep. Dave, Bev - points also taken. John Tull's comments on 5/26 make sense to me - at least the part about good judges validating their score in their writing. OK, so we don't need to account for points. All we really need is to have some correlation between comments and the numeric score. Good judges clearly correlate number with comments. Quite a few judges just describe the beer and assign a number - that works when they have anything bad to say about the beer, but it completely breaks down when they note no flaws. It's to style, it's pleasant, I like it and here is your way-less-than-perfect score. The problem isn't the methodology, it's that some judges describe without really critiquing. So are we testing too much on knowledge of beer and not enough on good judging? John Peed ---------------------------------------------------------------------- ***** Important Subscriber Information ***** To post a message to JudgeNet, send it to judge`at`synchro.com. Send plain text only, no HTML, MIME, encoded text or attachments. Make sure you use a meaningful subject. Quote only as much material as is needed for context. To manage your subscription, go to http://synchro.com/judge/subscriptions.html or send an email to judge-request`at`synchro.com with the subject: help judge. JudgeNet is also available as an NNTP newsgroup, go to news://news.synchro.com/synchro.judge