Return-Path: owner-judge at synchro.com Received: from srvr7.engin.umich.edu (root at srvr7.engin.umich.edu [141.212.2.69]) by srvr5.engin.umich.edu (8.8.5/8.8.5) with ESMTP id OAA12213 for ; Sun, 18 May 1997 14:48:13 -0400 (EDT) Received: from redheat.rs.itd.umich.edu (0 at redheat.rs.itd.umich.edu [141.211.83.36]) by srvr7.engin.umich.edu (8.8.4/8.8.4) with ESMTP id OAA28617 for ; Sun, 18 May 1997 14:48:10 -0400 (EDT) Received: by redheat.rs.itd.umich.edu (8.8.5/2.2) with X.500 id OAA20529; Sun, 18 May 1997 14:48:09 -0400 (EDT) Received: from uu6.psi.com by redheat.rs.itd.umich.edu (8.8.5/2.2) with SMTP id OAA20523; Sun, 18 May 1997 14:48:07 -0400 (EDT) Received: by uu6.psi.com (5.65b/4.0.071791-PSI/PSINet) via UUCP; id AA06939 for spencer at umich.edu; Sun, 18 May 97 14:47:58 -0400 Received: (from majordom at localhost) by synchro.com (8.6.12/8.6.12) id OAA14884 for judge-digest-outgoing; Sun, 18 May 1997 14:18:55 -0400 Date: Sun, 18 May 1997 14:18:55 -0400 Message-Id: <199705181818.OAA14884 at synchro.com> From: owner-judge-digest at synchro.com To: judge-digest at synchro.com Subject: judge-digest V1 #1442 Reply-To: judge at synchro.com Errors-To: owner-judge-digest at synchro.com Precedence: bulk judge-digest Sunday, 18 May 1997 Volume 01 : Number 1442 ============================================================================ J u d g e N e t - t h e b e e r j u d g e d i g e s t ============================================================================ Moderator: Chuck Cox Archivist: Spencer Thomas Publisher: SynchroSystems Submissions: judge at synchro.com Subscriptions: judge-request at synchro.com Archive: http://realbeer.com/spencer/judge BJCP info: geninfo at bjcp.synchro.com ============================================================================ contents: 3rd Annual Boneyard Brew-Off Judge Training? Exam feedback Wrong Categories crapshoot Lack of Feedback? Essay and tasting scores on retakes ---------------------------------------------------------------------- From: "Joel Plutchak" Date: Fri, 16 May 1997 09:58:44 -0500 Subject: 3rd Annual Boneyard Brew-Off 3rd Annual B.U.Z.Z. Boneyard Brew-Off, 2nd Announcement The Boneyard Union of Zymurgical Zealots (B.U.Z.Z.) is *still* organizing their third annual homebrew competition. This a short version of the full announcement. Judges and stewards welcome! We plan the usual Friday evening Judge Social, a post-competition dinner, and prize raffle. Please contact the Judge Director or Competition Organizer for more details. Sanctioned by the AHA and BJCP; A Midwest Homebrewer of the Year event. Saturday, June 14, 1997 Any homebrewed beer or mead is eligible. AHA 1997 Style Guidelines; Additional special category: No One Gets Out Alive High Gravity Brew-Off Starting gravity over 1.070, any style or non-style; judged solely on potency and overall drinkability. Not eligible for Best of Show, but a special trophy will be awarded. Send entries to: Boneyard Brew-Off c/o Picadilly Beverage Shop 505 S. Neil Street Champaign IL 61821 *** PLEASE NOTE: Entries accepted May 27 through June 9, 1997 ONLY! *** Fee: $5 per entry; $4 per entry for four or more per brewer. 2 bottles per entry; standard AHA bottle requirements apply. Further information, including rules and instructions, on-line judge registration, and entry forms for downloading can be found at: http://starfire.ne.uiuc.edu/buzz/contest3.html Sponsors ======== The B.U.Z.Z. would like to thank the following sponsor(s): * Picadilly Beverage Shops, Champaign-Urbana IL * Two Brothers Brewing Company, Warrenville IL * Northwestern Extract Co., Brookfield, WI * CYMBA, Darien CT * American Homebrewers Association, Boulder CO * Jack Schmidling Productions, Marengo IL * Liquid Bread, Orlando FL * St. Louis Bread Compamy, Champaign IL * Hamburg Distributing Co., Champaign IL For more information about becoming a sponsor, contact one of the people listed below. - ---------------------------------------------------------------------------- Organizer: Registrar: Joel Plutchak Daniel Juliano email: joel at wave.atmos.uiuc.edu email: dan at starfire.ne.uiuc.edu 916 W. Charles Street Champaign IL 61821 Judge Director: (217) 359-4931 (eves & wkends) Troy Jesse (217) 333-8132 (M-F, 8am-4pm) email: tjesse at students.uiuc.edu ------------------------------ From: Jim Liddil Date: Fri, 16 May 1997 8:17:34 -0700 (MST) Subject: Judge Training? Keith wrote: > I know this may be a bit off topic but I thought I would put it out on the > floor and see what develops. I recently entered a belgian tripel in a > competition and was surprised I didn't win because it was a wonderful beer, > 8.25% abv, very smoothly drinkable, and decieving of the alcohol level, and > know real flaws except a little banana. Well the scoresheet came back with > a 25 or 28 or something. It was judged as a fruit lambic! This indicates that the judges had a COMPLETE lack of understanding of the style. The people who judged this beer should be made aware of their mistake and politely instructed to review the aspects of this style. I find it odd that none of the judges called into question the category this beer was placed in. So much for useful feedback. :-) Jim ------------------------------ From: Scott Bickham Date: Fri, 16 May 1997 11:18:33 -0400 (EDT) Subject: Exam feedback Jeremy wrote: > Is there really a positive BJCP policy not to return exam > materials as opposed to just not having the mechanism/ > resources to do so? And what is wrong with his knowing > about the recipe? I think one of the big flaws with the > exam is the near total lack of feedback. The exam is an > incredible learning experience. But the benefit could > be improved with better feedback. Many people taking the > exam are judging out beers with a score sheet for the first > time, or nearly so. Think how much one could learn with > rapidly-returned copies of the scoresheet of the proctors > and any other examinees who did well. The problem in returning exam materials is both logistics and protecting the BJCP from protests. As you can see from this thread, an examinee protested because he though his scoring of one beer was responsible for his not making the higher level. In fact, only 10% of the total points lost on the entire exam were directly or indirectly associated with how this particular beer was judged. There were similar deductions on the other exam beers and on several of the essay questions. One of the readers of this thread made the astute observation that it is pretty amazing that the two graders and two associate directors all independently assigned scores of 76, 79, 78 and 79 to this exam. This iss all that surprising to me, considering that this group has scored or reviewed a total of 700 BJCP exams. Norm also didn't fully state two other problems with this exam: 1. The original exam was never sent to the exam director; instead after a three week delay, a xerox copy and originals of the other four exams were sent. No explanation was given. 2. The administrator was instructed to give information on the source of the exam beers to the graders. This was not done, however one of the examinees was given that information to help his protest. Before this incident happened, I was leaning towards recommending that the administrators make copies of the exams before sending them off to be graded, but it would be important to do that in a timely manner and send the originals to the BJCP (since pages are often skipped with most document feeders). But how I can I assume that these instructions will be followed when simpler ones such as using 8.5x11 inch paper (not 8x10.5 looseleaf) and asking the examinees to only write on one side are not followed? You tell me. Since it's in the area of judging where many novice judges falter, I think the first step in providing better feedback is to return the judging sheets, along with an example of ones that would receive high marks. This can be done much more easily than returning the entire exam, and would probably be approved by Russ Wigglesworth if I stapled the scoresheets to the Report to Participant form to keep things organized. But with an estimated 400 exams being given this year, any improvements have to be relatively simple. Scott - -- ======================================================================== Naval Research Laboratory, Code 6691 E-mail: bickham at dave.nrl.navy.mil Complex Systems Theory Branch Home or BJCP: 7507 Swan Point Way Washington, DC 20375 Columbia, MD 21045 (202) 404-8632 FAX: (202) 404-7546 (410) 290-7721 BJCP Web Page: http://www.bjcp.org ========================================================================= ------------------------------ From: "Houseman, David L" Date: Fri, 16 May 1997 11:23:12 -0400 Subject: Wrong Categories Keith Chamberlin writes about what to do about entries entered in the wrong categories. As a judge, whenever I get an entry to judge that is obviously out of category, I do ask the organizer to check to ensure that it has been submitted to the judges correctly. This only goes so far as to find out if the entry form indication of category/subcategory matches what was submitted to the judges. Sometimes I've found that in the handling of entries the organizers have made a mistake and it's correctable by then sending the entry to the right judges. But if the entry was in fact for the wrong category/subcategory, I don't think the judges or the organizers should in any way take steps to re-categorize the beer. It's the responsibility of the entrant to ensure the correct category and subcategory has been entered. Last year I entered a saison as a Classic Spice Specialty and the organizers took it upon themselves to re-categorize the beer into the Belgian ale flights but miss categorized the beer themselves as a dubbel whereupon you can image the comments that were received. The judges should have questioned this of the organizers to check for obvious errors. Of course the organizers should NOT have taken it upon themselves to re-categorize any beer although combining categories in flights would have been legit. Dave Houseman ------------------------------ From: Jeremy Bergsman Date: Fri, 16 May 1997 10:25:13 -0700 Subject: crapshoot > From: Scott Bickham > Anyway, the "index of dependability" (which is the index appropriate for > the types of absolute "level" distinctions that we are making) is 0.86 (for > an examinee who responds to 10 items, each graded by two graders). This is > pretty damned good. It indicates that we can have a lot of confidence in > the classifications that we make based on exam scores. > However very > little variance is attributable to the graders, which means that > we're being _relatively_similar in our severity when grading exams and > that we're not being too "biased" by idiosyncracies of individual > examinees. So while the exam is proven to be reliable, we are working > to make it even better. I never impugned the graders or suggested that their grades were variable between or within graders, or that their grades were biased. However, and maybe Ed (Hi Ed) can correct me if I'm wrong, what he has analyzed doesn't indicate whether the exam is measuring the right thing, it only indicates that what it measures is repeatable. It also doesn't address grader bias if all graders have a similar set of biases (an "old boys club" kind of situation could be proposed which would not show up with his analysis). The issue of what the exam measures is an extremely important one. It has been discussed here many times, so I won't start that again except to say that I think there are problems with what it measures vs. what skills a judge needs and that is what I meant when I said it is bogus. That was rather strong language and I apologize if I offended anyone, but I stand by the intent of my statement. - -- Jeremy Bergsman jeremybb at leland.stanford.edu http://www-leland.stanford.edu/~jeremybb ------------------------------ From: pgarofalo at juno.com (Peter M Garofalo) Date: Fri, 16 May 1997 13:01:33 EDT Subject: Lack of Feedback? I have so far taken a back seat in the discussions concerning the BJCP exam, how it is administered, whether or not it is fair, etc. The comment on today's digest concerning "a nearly total lack of feedback" on the BJCP exam really got to me, though. First off, if there isn't a policy preventing the return of exam materials, maybe we should return them. When I grade exams, I add quite a few comments that I later condense into a concise report of strengths and weaknesses on the Report to Participant (RTP) form. Anybody who thinks there is a lack of feedback should be made aware that there is significant, thoughtful, and detailed feedback presented by the *volunteer* graders. Want to do something to improve it? Sign up to grade some exams! I take as much pride in the exams I grade as in the beers I judge. While John Calen may have had a legitimate beef, I am convinced that his protest was handled properly. Bringing up such a disagreement on this forum is poor form, to say the least. Yes, the exam is tough. Too tough? Maybe, but change will be slow (as it should be!). There is evidence that the exam is accurately applied and graded; perhaps the next challenge is to determine just how deep the level of knowledge must be. I am comfortable with several aspects of the exam: the average (about 75) is relatively high, and few who take the exam fail (< 5 %). It is considerably more difficult to score 80 than 70, separating Certified and National levels of achievement. It is indeed a proud achievement to score a 90, and it's pretty rare. Anyone with a Master ranking certainly has earned it. Bottom line: the exam could use some tuning-up, and this is currently underway (another committee). The feedback is thorough and detailed, but maybe we should discuss returning exams--maybe it would save me some time. Peter Garofalo Syracuse, NY ------------------------------ From: Scott Bickham Date: Fri, 16 May 1997 15:37:15 -0400 (EDT) Subject: Essay and tasting scores on retakes As many of you know, the BJCP has been offering retakes of either the essay or tasting portion of the exam. The proctors are given information on how to accomodate these requests, and there have been a handful so far this year. This information has been posted here and will be given in the next issue of the BJCP Reporter. This separation also means that in cases where portion or the entire exam has been retaken, the higher scores are the ones that count. For example, if Joe Judge scored 73 essay and 79 taste on his first attempt and 81 essay and 75 taste on his second attempt, the higher scores matter and his total score is 81*(0.7) + 79*(0.3) = 80.4 total, so Joe qualifies for the National level. Unfortunately, the separate scores have only been entered since 1995, though we have hard copies of most of the Reports to Participant and can refer to those when necessary. If you are retaking the exam or have retaken the exam and feel that your total score needs to be adjusted to take the higher ones on each portion into account, give us as much information and we will set the record straight. Thanks, Scott - -- ======================================================================== Naval Research Laboratory, Code 6691 E-mail: bickham at dave.nrl.navy.mil Complex Systems Theory Branch Home or BJCP: 7507 Swan Point Way Washington, DC 20375 Columbia, MD 21045 (202) 404-8632 FAX: (202) 404-7546 (410) 290-7721 BJCP Web Page: http://www.bjcp.org ========================================================================= ------------------------------ End of judge-digest V1 #1442 **************************** Send subscription cancellations & changes to judge-request at synchro.com. Messages sent to the wrong address will be ignored.