Subject: Digest for the period 1/19/2004 - 1/20/2004 Date: Tue, 20 Jan 2004 01:02:29 -0500 Table of contents ---------------------------------------------------------------------- 1. More Heresy (Pete) 2. Exam "grader's guide" (Mike Dixon) 3. BCJP Exam Issues (Peter Garofalo) 4. Re: Digest for the period 1/17/2004 - 1/18/2004 (Mark Wilson) ---------------------------------------------------------------------- From: Pete Date: Sun, 18 Jan 2004 08:26:11 -0500 Subject: More Heresy Steve suggests a two (or more, by extension) tierd test system, and I just can't get on board with this one. In the first place, I know people who've worked real hard (I'm NOT one of 'em) and scored in the 80s on the first pop - why force these folks to repeat the process? Also, I see the real possibility for snobbiness, more so than currently exists. "Oh, you got an 85? on WHICH test?" I will concede the points about it being easier, both to take and to grade - and a multiple choice format might even permit the dispensing of scores at the exam scene. But somehow, I don't see that this will benifit the BJCP as a whole - after all, the whole point is that the test is HARD, and we have to KNOW our stuff to score well on it. There's plenty of 'Beer Judge? How hard is THAT??" in the general public already - personally, I'd hate for that statement to get legs. Besides, nothing prevents someone who totally bombs the exam the first time from racking up 20 points as an 'apprentice' and retesting for National at that point... I'm just sayin' Pete Bussa ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * ********************************************************************** ---------------------------------------------------------------------- From: Mike Dixon Date: Sun, 18 Jan 2004 11:13:28 -0500 Subject: Exam "grader's guide" > Mark Wilson wrote: > > Well, this gets at one of MY pet peeves about the exam. There is no > > "grader's guide" availible, so exam-takers don't know how close you > > need to be, and what exactly points are being taken off for. Am I > > penalized for being off spec by 1 point? 3? 5? Since threshold, for > > example, for IBU perception is about 5 IBUs, I hope not. Not to > > mention the fact that many BJCP classic examples do not meet the style > > parameters (e.g. Old Peculiar). Should one concentrate on memorizing > > these sort of details, or more history, geography, etc. in order to > > improve their score? To which Scott Bickham replied: > Mark, you are not a grader, so I do not see how you can make these unfounded > statements. Thumbnail answers to the exam questions have existied for a > number of years and are periodically updated, most recently by John Curtis. > These are distributed to new graders. While there are myriad ways to > present the combination of flavor, technical and background information, the > key provides *some* of the common elements that graders should look for. > However there is not a template for a given score, since there are too many > possibilities and we encourage the graders' individualities. I am not an exam grader either (yet). I took Mark's comment to mean a grader guide is not available to anyone just taking the exam. Meaning it is like the first test you take in college from a teacher and they will only tell you the format, essay, not the basis upon which the essay exam is graded. There is an example of a complete answer in the Study Guide http://www.bjcp.org/study.html#scoresheets but no guide given as to this portion scored a point, that part of the response another point, etc. If that could be added in some format, it would let someone know that giving the specs would earn X, etc. I recently retook the exam. I made a better score, I moved up, but I still will have to retake yet again for the Master level. I now see one of my major failings was specifications for the styles. The comments were more detailed this second time around than the first time and will prove beneficial when the time to retake comes again. Unfortunately, much more complete answers that the first time only netted 4 more points on the written portion of the exam. I believe that is why a grader's guide that is available to potential test takers would be a valuable resource. Cheers, Mike Dixon Wake Forest, NC ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * ********************************************************************** ---------------------------------------------------------------------- From: Peter Garofalo Date: Sun, 18 Jan 2004 21:28:25 -0500 Subject: BCJP Exam Issues I've been reading the commentary on views of the BJCP exam, both real and imagined, with great interest. I don't want to leave my co-director high and dry, and I do appreciate the comments that several of the graders have also tossed in. Here's what I would like to add to the dialogue: we happen to have a very talented and experienced corps of graders, Associate Directors, and yes, Exam Directors. It takes a lot of work, and a lot of exams graded, to reach these positions. After grading a few dozen exams, it becomes crystal clear what is needed to reach each level. Scott and I have each graded or reviewed over 350 exams, and we have both worked extensively to improve the exam. Another point is this: the exam review process (after initial grading) is intended to assess the grading job as well as the examinees. I always send feedback to the lead grader to point out good work, as well as noting areas that need improvement. After initial grading, the Associate Director carefully reviews each exam and adjusts if necessary. Finally, the Exam Director reviews and adjusts grades as necessary. We spend quite a bit of time doing this, and we do not take it lightly. I can truthfully say that the current pool of graders is the most talented ever, and they are constantly improving. I have made it a personal goal to work on the grader pool, mainly through giving feedback. I believe it has worked. Although it is certainly not perfect, the BJCP exam does a great job in separating the relative ranks of judges. I have never gotten a poor scoresheet from a Master-level judge, and that is no accident. It is a hard test, but it is fair, at least in my opinion. Scoring Master is something to be proud of. Cheers, Pete Garofalo BJCP Western Exam Director ---------------------------------------------------------------------- From: Mark Wilson Date: Mon, 19 Jan 2004 11:58:11 -0800 Subject: Re: Digest for the period 1/17/2004 - 1/18/2004 > Mark Wilson wrote: > > > Well, this gets at one of MY pet peeves about the exam. There is no > > "grader's guide" availible, so exam-takers don't know how close you > > need to be, and what exactly points are being taken off for. Am I > > Mark, you are not a grader, so I do not see how you can make these unfounded > statements. Thumbnail answers to the exam questions have existied for a > number of years and are periodically updated, most recently by John Curtis. > These are distributed to new graders. While there are myriad ways to > present the combination of flavor, technical and background information, the > key provides *some* of the common elements that graders should look for. I don't think anything I said was unfounded. The _graders_ may have access to graded exam examples, but as far as I know, rank-and-file test takers do not. I can only find one example of an exam answer on the bjcp site, and it is of a completely correct answer. What does a 5 point, 7 point, 9 point answer look like? > While there is some variability in the level of feedback between exam sets, > most examinees are now given 1.5 - 2 pages of comments in 10 point font. > This adds a lot of time to the grading process, but most of the graders take > this as an opportunity to teach and train new judges. We've reached the Both times I've taken the exam (1996 and 2000) the feedback has consisted of 2 or 3 comments along the lines of "stouts are made with roasted barley not black patent", and that was it. So if graders are providing 1.5-2 pages of feedback now, that's excellent! Getting a marked-up exam back would be great as well. You could see exactly how good/bad your answer really was. Realistically I can see how graders would want to avoid this due to the nit-pick/whine factor that would inevitably result. -Mark ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * ********************************************************************** Subject: Digest for the period 1/19/2004 - 1/20/2004 Date: Tue, 20 Jan 2004 01:02:29 -0500 Table of contents ---------------------------------------------------------------------- 1. More Heresy (Pete) 2. Exam "grader's guide" (Mike Dixon) 3. BCJP Exam Issues (Peter Garofalo) 4. Re: Digest for the period 1/17/2004 - 1/18/2004 (Mark Wilson) ---------------------------------------------------------------------- From: Pete Date: Sun, 18 Jan 2004 08:26:11 -0500 Subject: More Heresy Steve suggests a two (or more, by extension) tierd test system, and I just can't get on board with this one. In the first place, I know people who've worked real hard (I'm NOT one of 'em) and scored in the 80s on the first pop - why force these folks to repeat the process? Also, I see the real possibility for snobbiness, more so than currently exists. "Oh, you got an 85? on WHICH test?" I will concede the points about it being easier, both to take and to grade - and a multiple choice format might even permit the dispensing of scores at the exam scene. But somehow, I don't see that this will benifit the BJCP as a whole - after all, the whole point is that the test is HARD, and we have to KNOW our stuff to score well on it. There's plenty of 'Beer Judge? How hard is THAT??" in the general public already - personally, I'd hate for that statement to get legs. Besides, nothing prevents someone who totally bombs the exam the first time from racking up 20 points as an 'apprentice' and retesting for National at that point... I'm just sayin' Pete Bussa ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * ********************************************************************** ---------------------------------------------------------------------- From: Mike Dixon Date: Sun, 18 Jan 2004 11:13:28 -0500 Subject: Exam "grader's guide" > Mark Wilson wrote: > > Well, this gets at one of MY pet peeves about the exam. There is no > > "grader's guide" availible, so exam-takers don't know how close you > > need to be, and what exactly points are being taken off for. Am I > > penalized for being off spec by 1 point? 3? 5? Since threshold, for > > example, for IBU perception is about 5 IBUs, I hope not. Not to > > mention the fact that many BJCP classic examples do not meet the style > > parameters (e.g. Old Peculiar). Should one concentrate on memorizing > > these sort of details, or more history, geography, etc. in order to > > improve their score? To which Scott Bickham replied: > Mark, you are not a grader, so I do not see how you can make these unfounded > statements. Thumbnail answers to the exam questions have existied for a > number of years and are periodically updated, most recently by John Curtis. > These are distributed to new graders. While there are myriad ways to > present the combination of flavor, technical and background information, the > key provides *some* of the common elements that graders should look for. > However there is not a template for a given score, since there are too many > possibilities and we encourage the graders' individualities. I am not an exam grader either (yet). I took Mark's comment to mean a grader guide is not available to anyone just taking the exam. Meaning it is like the first test you take in college from a teacher and they will only tell you the format, essay, not the basis upon which the essay exam is graded. There is an example of a complete answer in the Study Guide http://www.bjcp.org/study.html#scoresheets but no guide given as to this portion scored a point, that part of the response another point, etc. If that could be added in some format, it would let someone know that giving the specs would earn X, etc. I recently retook the exam. I made a better score, I moved up, but I still will have to retake yet again for the Master level. I now see one of my major failings was specifications for the styles. The comments were more detailed this second time around than the first time and will prove beneficial when the time to retake comes again. Unfortunately, much more complete answers that the first time only netted 4 more points on the written portion of the exam. I believe that is why a grader's guide that is available to potential test takers would be a valuable resource. Cheers, Mike Dixon Wake Forest, NC ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * ********************************************************************** ---------------------------------------------------------------------- From: Peter Garofalo Date: Sun, 18 Jan 2004 21:28:25 -0500 Subject: BCJP Exam Issues I've been reading the commentary on views of the BJCP exam, both real and imagined, with great interest. I don't want to leave my co-director high and dry, and I do appreciate the comments that several of the graders have also tossed in. Here's what I would like to add to the dialogue: we happen to have a very talented and experienced corps of graders, Associate Directors, and yes, Exam Directors. It takes a lot of work, and a lot of exams graded, to reach these positions. After grading a few dozen exams, it becomes crystal clear what is needed to reach each level. Scott and I have each graded or reviewed over 350 exams, and we have both worked extensively to improve the exam. Another point is this: the exam review process (after initial grading) is intended to assess the grading job as well as the examinees. I always send feedback to the lead grader to point out good work, as well as noting areas that need improvement. After initial grading, the Associate Director carefully reviews each exam and adjusts if necessary. Finally, the Exam Director reviews and adjusts grades as necessary. We spend quite a bit of time doing this, and we do not take it lightly. I can truthfully say that the current pool of graders is the most talented ever, and they are constantly improving. I have made it a personal goal to work on the grader pool, mainly through giving feedback. I believe it has worked. Although it is certainly not perfect, the BJCP exam does a great job in separating the relative ranks of judges. I have never gotten a poor scoresheet from a Master-level judge, and that is no accident. It is a hard test, but it is fair, at least in my opinion. Scoring Master is something to be proud of. Cheers, Pete Garofalo BJCP Western Exam Director ---------------------------------------------------------------------- From: Mark Wilson Date: Mon, 19 Jan 2004 11:58:11 -0800 Subject: Re: Digest for the period 1/17/2004 - 1/18/2004 > Mark Wilson wrote: > > > Well, this gets at one of MY pet peeves about the exam. There is no > > "grader's guide" availible, so exam-takers don't know how close you > > need to be, and what exactly points are being taken off for. Am I > > Mark, you are not a grader, so I do not see how you can make these unfounded > statements. Thumbnail answers to the exam questions have existied for a > number of years and are periodically updated, most recently by John Curtis. > These are distributed to new graders. While there are myriad ways to > present the combination of flavor, technical and background information, the > key provides *some* of the common elements that graders should look for. I don't think anything I said was unfounded. The _graders_ may have access to graded exam examples, but as far as I know, rank-and-file test takers do not. I can only find one example of an exam answer on the bjcp site, and it is of a completely correct answer. What does a 5 point, 7 point, 9 point answer look like? > While there is some variability in the level of feedback between exam sets, > most examinees are now given 1.5 - 2 pages of comments in 10 point font. > This adds a lot of time to the grading process, but most of the graders take > this as an opportunity to teach and train new judges. We've reached the Both times I've taken the exam (1996 and 2000) the feedback has consisted of 2 or 3 comments along the lines of "stouts are made with roasted barley not black patent", and that was it. So if graders are providing 1.5-2 pages of feedback now, that's excellent! Getting a marked-up exam back would be great as well. You could see exactly how good/bad your answer really was. Realistically I can see how graders would want to avoid this due to the nit-pick/whine factor that would inevitably result. -Mark ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * **********************************************************************