Return-Path: owner-judge at synchro.com Received: from srvr7.engin.umich.edu (root at srvr7.engin.umich.edu [141.212.2.69]) by srvr5.engin.umich.edu (8.8.5/8.8.5) with ESMTP id MAA00499 for ; Mon, 19 May 1997 12:36:17 -0400 (EDT) Received: from judgmentday.rs.itd.umich.edu (0 at judgmentday.rs.itd.umich.edu [141.211.83.37]) by srvr7.engin.umich.edu (8.8.4/8.8.4) with ESMTP id MAA00936 for ; Mon, 19 May 1997 12:36:15 -0400 (EDT) Received: by judgmentday.rs.itd.umich.edu (8.8.5/2.2) with X.500 id MAA20580; Mon, 19 May 1997 12:36:14 -0400 (EDT) Received: from uu6.psi.com by judgmentday.rs.itd.umich.edu (8.8.5/2.2) with SMTP id MAA20565; Mon, 19 May 1997 12:36:12 -0400 (EDT) Received: by uu6.psi.com (5.65b/4.0.071791-PSI/PSINet) via UUCP; id AA03677 for spencer at umich.edu; Mon, 19 May 97 12:35:54 -0400 Received: (from majordom at localhost) by synchro.com (8.6.12/8.6.12) id LAA17945 for judge-digest-outgoing; Mon, 19 May 1997 11:48:38 -0400 Date: Mon, 19 May 1997 11:48:38 -0400 Message-Id: <199705191548.LAA17945 at synchro.com> From: owner-judge-digest at synchro.com To: judge-digest at synchro.com Subject: judge-digest V1 #1443 Reply-To: judge at synchro.com Errors-To: owner-judge-digest at synchro.com Precedence: bulk judge-digest Monday, 19 May 1997 Volume 01 : Number 1443 ============================================================================ J u d g e N e t - t h e b e e r j u d g e d i g e s t ============================================================================ Moderator: Chuck Cox Archivist: Spencer Thomas Publisher: SynchroSystems Submissions: judge at synchro.com Subscriptions: judge-request at synchro.com Archive: http://realbeer.com/spencer/judge BJCP info: geninfo at bjcp.synchro.com ============================================================================ contents: Shooting Crap Hop ID Mis-Categorizing Styles exam feedback ---------------------------------------------------------------------- From: Ed & Carol Wolfe Date: Sun, 18 May 1997 15:35:45 -0400 Subject: Shooting Crap Jeremy wrote: .I never impugned the graders or suggested that their grades were variable .between or within graders, or that their grades were biased. However, .and maybe Ed (Hi Ed) can correct me if I'm wrong, what he has analyzed .doesn't indicate whether the exam is measuring the right thing, it only .indicates that what it measures is repeatable. It also doesn't address .grader bias if all graders have a similar set of biases (an "old boys .club" kind of situation could be proposed which would not show up .with his analysis). .The issue of what the exam measures is an extremely important one. .It has been discussed here many times, so I won't start that again .except to say that I think there are problems with what it measures .vs. what skills a judge needs and that is what I meant when I said .it is bogus. That was rather strong language and I apologize if .I offended anyone, but I stand by the intent of my statement. Yes, Jeremy, you are correct. But, although I didn't see the original posting, Scott's reply is perfectly in line with defending the exam as being a "crap shoot." That phrase implies that the exam scores are greatly influenced by chance--a concern about the exam's reliability (i.e., the extent to which the exam scores are influenced by systematic (e.g., bias) and unsystematic "error" (e.g., chance)). Hence, Scott's reply was right on target. The work we've been doing on the BJCP exam indicates that the exam is very reliable--the scores that are assigned would tend to result in VERY similar rankings of (or, as indicated in Scott's message, categorizations of) examinees regardless of which items or graders are used to produce the examinee's scores. The issue that Jeremy brings up here is a very different one--one of validity. That is, validity is concerned with the extent to which the measures produced by the exam are correlated with the latent trait that the scores are suppose to represent. In our case, the scores are suppose to represent "judging ability." I think that the current exam is a pretty good measure, considering the efficiency of the examination system. And, OF COURSE we could come up with a better measure, but a better measure would involve more cost, time, resources, etc. However, I think that jumping to the conclusion that we NEED to come up with a better exam is a bit premature. The problem is that we simply don't know how valid the BJCP exam is. We don't have any information except anecdotal cases, and that doesn't prove anything. We're working on collecting that information right now, as Scott mentioned in his message, but it will take time to build and evaluate the evidence for the exam's validity. We'll keep you informed as things progress though. Ed Wolfe ------------------------------ From: Paul Edwards Date: Mon, 19 May 1997 07:05:09 -0500 (EST) Subject: Hop ID Fellow Judges: One two different occassions, I've had English-style ales (one an ESB and one an Ord. Bitter) judged as being "out of style because of the Cascade hops". The ESB comment came from a Nat'l judge (and a friend of mine, who shall remain anonymous), the other (the Ord Bitter) came at an informal club judging with "apprentice judges". One person thought the Ord Bitter was SN Pale Ale. OTOH, one club member (who happens to be an assistant brewer for the local Rock Bottom) correctly picked EKG, and remarked that there was "tons of EKG". (Ord. Bitter had a 1/2 ounce plug at knockout and another 1/2 ounce dry hop in keg for 14 days). The thing is, _both_ beers were brewed with Northern Brewer for bittering, and EKG's for late kettle additions and dry-hopping. Both happened to be fermented with Wyeast #1968. Both beers were hopped towards the high end of their respective style parameters, and I'll admit the Ord. Bitter was even "over the top" at about 45 BU's in a 1.040 OG wort (using Tinseth's equations). To me neither beers smelled of Cascade at all, yet they did have a resinous hop flavor, but still to me not that signature grapefruit flavor I associate with Cascade. OTOH, they were _my_ beers and I wasn't tasting them blind. Has anyone else had this experience?. Do aggressively hopped (bitter, flavor and aroma) fruity-estery ales come off tasting like SNPA or SNCA to anyone else, even though there were no Cascade hops in the brew? Perhaps I should try a different bittering hop? Or are these judges mistaken and are just assuming every fruity-estery-bitter ale has Cascade in it? I am going to experiment with only dry hop and no late kettle hops (and vice versa), and of course I'm going to brew another Ord Bitter at something like 30-35 BU's. And probably I'll do a repeat of the recipe _with_ Cascade and see what I get. private e-mail is fine, if you wish. - --Paul Edwards (BJCP Certified and Certified Hophead) pedwards at iquest.net ------------------------------ From: MaltyDog at aol.com Date: Mon, 19 May 1997 09:14:14 -0400 (EDT) Subject: Mis-Categorizing Styles Following up on the thread begun by Keith Chamberlin, I have run into the problem of miscategorized styles, I have run into the problem a couple of times, and after checking with the organizer to make sure that the problem wasn't at that end, I have found it quite appropriate to comment in the "drinkability" section something along the lines of "this beer would have scored higher had it been entered into such-and-such category. It gives the brewer useful information. I don't know how widely done this is, but it seems helpful to me. Sometimes people make mistakes when they enter, sometimes they are not familiar enough with the style, sometimes they beer they made ended up differently then it was planned, but does fit into another style. In those cases, to suggest to them they should try entering in another category seems reasonable. Bill Coleman MaltyDog at aol.com ------------------------------ From: "Bryan L. Gros" Date: Mon, 19 May 1997 08:36:10 -0500 (CDT) Subject: exam feedback Scott writes: >Since it's in the area of judging where many novice judges falter, >I think the first step in providing better feedback is to return the >judging sheets, along with an example of ones that would receive >high marks... This idea would be a great help. Since judges should be trained in scoring beers, after all, and since most exam takers have little experience in judging, getting both the exam score sheet as well as an "ideal" score sheet would be great. Not only would it give you an idea of what the scorers were looking for, but it would give you a good idea of what a proper score sheet should look like. The rest of the exam has little that is open to interpretation. It is mostly book knowlege, so feedback would be less helpful. I'm definately for this kind of feedback. By the way, can people take only one part of the exam these days? I know it was talked about late last year, but I'm not sure if that is the current policy. - Bryan grosbl at ctrvax.vanderbilt.edu Nashville, TN ------------------------------ End of judge-digest V1 #1443 **************************** Send subscription cancellations & changes to judge-request at synchro.com. Messages sent to the wrong address will be ignored.